CI August 2023

ci@lists.fedoraproject.org

5 participants
4 discussions

Upgrading gerrit 3.5 to 3.7
by Francisco Seruca Salgado 31 Aug '23

31 Aug '23

Hello all, In a few minutes, we will be starting the upgrade procedure for gerrit 3.5 to 3.7. So for a few minutes, Gerrit will be unavailable. We apologize for any inconvenience this may cause. Once the upgrade is complete, we will inform you. Thanks Francisco Seruca Salgado PCInfra Team

1 1

Software Factory services outage fedora.softwarefactory-project.io: Friday Aug 18 4-5pm EST
by Fabien Boucher 24 Aug '23

24 Aug '23

Hello folks, An upgrade operation is scheduled by our cloud provider that could affect Software Factory services. During that time some services might be unavailable. The affected services will be: - Zuul CI not running jobs for pagure.io or src.fedoraproject.org. Regards, Fabien, on behalf of the Software Factory Operation Team

2 2

Failing jobs for Rawhide (and F39?) in Fedora CI due to use of old mock configs
by Adam Williamson 13 Aug '23

13 Aug '23

I've seen several folks report that jobs (usually scratch build tests) are failing in Fedora CI, e.g. this one: https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-buil… the failures all seem to have a common cause. You'll note these lines: DEBUG: The GPG keys listed for the "fedora" repository are already installed but they are not correct for this package. DEBUG: Check that the correct key URLs are configured for this repository.. Failing package is: curl-8.2.1-2.fc40.x86_64 DEBUG: GPG Keys are configured as: file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-39-primary, file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-39-primary, file:///usr/share/distribution-gpg-keys/fedora/RPM-GPG-KEY-fedora-38-primary DEBUG: Public key for fedora-gpg-keys-40-0.1.noarch.rpm is not installed. Failing package is: fedora-gpg-keys-40-0.1.noarch ...for several packages. If you look carefully, you'll see it's using the wrong GPG keys: it's using the keys for F38 and F39, trying to validate a package from Rawhide. Obviously that will fail. This is happening because CI is running EL 8, and the EPEL 8 mock/mock- core-configs update for F39 branching is not yet stable: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2023-a5c155a6c0 the current stable mock-core-configs for EPEL 8 still thinks Rawhide is F39, so that's why it tries to use the F39 and F38 GPG keys to validate Rawhide packages. It needs the newer version of mock-core-configs that knows Rawhide is now F40. I've tried pinging CI folks about this to see if they can pull mock in from updates-testing, but so far no response. I guess it would help if folks running EPEL 8 can test mock, confirm it works, and up-karma the update. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net

2 1

Re: bodhi and testing farm disagrees on test result
by Adam Williamson 05 Aug '23

05 Aug '23

On Fri, 2023-08-04 at 15:40 +0200, Dan Horák wrote: > Hi, > > seems there is an issue with the test results presented in bodhi, please > see https://bodhi.fedoraproject.org/updates/FEDORA-2023-94f22746e1 > > bodhi thinks fedora-ci.koji-build.rpmdeplint.functional failed (it's > "red"), but when I click on the test in bodhi testing farm say "passed". > > What can be wrong? Well, Bodhi just gets the result from resultsdb. And the result there is definitely "failed": https://resultsdb.fedoraproject.org/results/41380875 Fedora CI files that result via ci-resultsdb-listener: https://pagure.io/ci-resultsdb-listener I think the message that triggered that result was this one: https://apps.fedoraproject.org/datagrepper/v2/id?id=0bb07f00-a232-48e9-a64f… which is actually an 'error' message, not a 'failed' message. resultsdb does - as of fairly recently - have an ERROR outcome, at least in the Fedora deployment, but it did not have one for a long time; ci- resultsdb-listener specifically uses the FAILED outcome for .error results, which we could perhaps improve now ERROR is available. The message doesn't tell us exactly what the error was, just "Infrastructure Failure". But this is basically what happened - some kind of system error occurred during/after the test run which somehow caused CI to publish a message indicating the test errored out, even though (as you saw) the logs indicate it passed. From there the details of how we report CI results to resultsdb turned it into a "failed" result. CCing the ci@ list - someone from there should be able to look more closely into what went on here and if anything can be improved (beyond making ci-resultsdb-listener use the ERROR outcome, now we have it). Practically speaking, we should be able to just re-run the tests and it should come out good. I'll bonk that button now. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw@fosstodon.org https://www.happyassassin.net

1 1

2024

2023

2022

2021

2020

2019

2018

2017

CI August 2023