On Mon, Jul 15, 2024 at 10:31:06AM +0200, Petr Pisar wrote:
Yes. Installability is a very basic test and it should be reliable. My feeling is that broken dependencies are the most common packaging bug.
Or if it's not possible to make them work, disable them? In the current state we just waste everybody's time and teach packagers to ignore CI results.
As far as I know, installability tests are not required to pass gating by default.
Correct.
I think the problem is that the tests are run even if nobody asks for them.
Well, I think it'd be fine if they were enabled by default _if_ they were reliable.
one of them (60-Install-packages.txt) shows: Failed to resolve the transaction: Problem 1: package coreutils-9.5-4.fc41.x86_64 from @commandline conflicts with coreutils-single provided by coreutils-single-9.5-4.fc41.x86_64 from @commandline - conflicting requests
I guess the test does not take RPM Conflicts into account. It's overly optimistic when populating a system by installing all tested packages together instead of creating a new system for each test seperately. Or by adding --allowerasing to "dnf install" invocations if the CI wants to reuse the system.
Yes and no. The test does not look at the package metadata at all, it just tries to install all the packages that were part of the update. In the case above, coreutils.srpm builds coreutils.rpm and coreutils-single.srpm, which have Conflicts on one another, and cannot be installed at the same time.
The test which (I think) we really want is to install the combined set of packages from the update, so we exercise the situation that will occur on end-user systems, but exclude the packages from this set which are known to be not co-installable.
A third problem is that the output is full of things that _look_ like errors:
Warning: Permanently added '3.14.151.22' (ED25519) to the list of known hosts.
egrep: warning: egrep is obsolescent; using grep -E
<string>:36: (WARNING/2) Bullet list ends without a blank line; unexpected unindent.
<string>:9: (WARNING/2) Definition list ends without a blank line; unexpected unindent.
This makes those jobs look semi-abandonded. It also makes it hard to look for the real error, because one has to consider each of those lines and answer the question "is this the error I'm looking for?".
Yes, I've opened many issues regarding Fedora CI. I think the problem is that the CI maintainer is not a regular packager and thus does not observe CI reults in real life. On the other hand, package maintainers do not have accces to the CI system and need to rely on the CI people. Or a CI person?!. The head count is probably another problem.
If there's one "CI person", then that is AdamW. He's doing great work (also in the update I linked in my original post). IIUC, AdamW's focus is on the 'update.*' tests, and those are fine, they generally pass. The bodhi results page says:
For help debugging failed Fedora CI tests (fedora-ci.*), contact the Fedora CI team. For help debugging failed Fedora CoreOS tests (coreos.*), contact the Fedora CoreOS team. For help debugging failed openQA tests (update.*), contact the Fedora Quality team, ...
I added ci@lists.fedoraproject.org in CC.
Zbyszek
On Mon, 2024-07-15 at 08:45 +0000, Zbigniew Jędrzejewski-Szmek wrote:
Yes, I've opened many issues regarding Fedora CI. I think the problem is that the CI maintainer is not a regular packager and thus does not observe CI reults in real life. On the other hand, package maintainers do not have accces to the CI system and need to rely on the CI people. Or a CI person?!. The head count is probably another problem.
If there's one "CI person", then that is AdamW. He's doing great work (also in the update I linked in my original post). IIUC, AdamW's focus is on the 'update.*' tests, and those are fine, they generally pass. The bodhi results page says:
For help debugging failed Fedora CI tests (fedora-ci.*), contact the Fedora CI team. For help debugging failed Fedora CoreOS tests (coreos.*), contact the Fedora CoreOS team. For help debugging failed openQA tests (update.*), contact the Fedora Quality team, ...
I added ci@lists.fedoraproject.org in CC.
Zbyszek
Just to be really clear about this: I am not personally responsible for Fedora CI, and the Fedora QA team is not responsible for it as a group. This is true at both Fedora and Red Hat levels. "Fedora CI" refers to the Jenkins and Zuul pipelines that run the tests reported to resultsdb with 'fedora-ci.' at the start of their name. I and my team *are* responsible for openQA, which runs the tests reported with 'update.' at the start of their name.
To be clear about something else - no "Fedora CI" test is gating for any Fedora update unless a package has configured it to be so in its per-package gating config. There is no project-wide gating on Fedora CI tests. This is because we know none of them is reliable enough to gate project-wide (and it's hard to read the results, and there's no one person whose job it is to look at failures and help resolve them, and so on). The only project-wide gating is on openQA tests, for critical path updates.
Due to various...internal considerations, when the system called "Fedora CI" was set up, it was intentionally set up under a different team which is *not* the team I'm on. Resource constraints since then have led to a situation where there really isn't anyone whose primary full-time job is caring about Fedora CI, which is the reason a lot of things to do with it are the way they are.
I don't love this situation, and I'd love to have the time and access to work on improving the Fedora CI tests, but I just don't. Like the team that is nominally responsible for it, I have a bunch of other stuff that is considered higher priority. I have contributed a few things to Fedora CI where I can (like the 'results dashboard' for the installability test), but it's never the top of my to-do list, and I don't have admin access to any of it. I'm just an outside contributor to that effort like anyone else.
ci@ is indeed the right list for anything to do with Fedora CI.
We are aware of many of the problems with the installability test - there's a ticket tracking them at https://pagure.io/fedora-ci/general/issue/448 . There are lots of ideas and plans there for improving things. The problem is just that nobody ever gets a clear week to sit down and do them as their top priority work. I've spent the last two months trying to clear my plate to work on an idea which would necessarily involve some improvements to it and I've never got there.