CI November 2023

ci@lists.fedoraproject.org

5 participants
8 discussions

The OSCI team cannot maintain its Fedora CI tests anymore
by Jan Kaluža 24 Nov '23

24 Nov '23

Hi everyone, I'm the Product Owner of the OSCI team. A few years ago, the team set up the Fedora CI tests executed for Pagure PRs and Bodhi updates. Specifically the: - Fedora CI - scratch build - Fedora CI - installability - Fedora CI - dist-git test - Fedora-ci.koji-build.rpminspect.static-analysis - Fedora-ci.koji-build.rpmdeplint.static-analysis These tests run on https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/. Our intent was to run the same tests as we run in RHEL gating also in Fedora gating. That was the main motivation to introduce these tests a few years ago in Fedora. The OSCI team’s focus however changed to CentOS Stream after it was introduced and we never had enough capacity to refocus back to Fedora. I do not see any option currently on how the team could keep maintaining our current Fedora infrastructure and tests. The main reasons for this are simply the amount of work the team can handle in its current head-count and also the differences between the CentOS Stream infrastructure and Fedora infrastructure. We are moving to Gitlab.com CI for CentOS Stream and we cannot do the same move in Fedora. We also cannot maintain both infrastructures at the same time for the reasons mentioned above. I’m therefore asking anyone who wants to keep these tests running to step in and tell us. We are ready to provide a knowledge transfer if it’s needed, but otherwise, we plan to decommission this Jenkins instance and the tests completely by March 2024. Thank you for understanding, Jan Kaluza

2 2

[release] Testing Farm 2023-10.2 with tmt-1.29
by Miroslav Vadkerti 23 Nov '23

23 Nov '23

Hi, Testing Farm release 2023-10.2 <https://docs.testing-farm.io/Testing%20Farm/0.1/releases/2023-10.2.html> is deployed 🎉. This is a release updating to the latest tmt and adding a few bug fixes for issues found after our last release. 🎮 *Upgrades* tmt <https://tmt.readthedocs.io/en/stable/> updated from 1.28.2 to 1.29 <https://tmt.readthedocs.io/en/stable/releases.html#tmt-1-29>. ️🐞 *Bugfixes* The multi-host pipeline was fixed to accept the fields role, hardware, kickstart, watchdog-dispatch-delay, and watchdog-period-delay from the provision step in the tmt plan. (MR <https://gitlab.com/testing-farm/gluetool-modules/-/merge_requests/653>) The request cancellation now correctly cancels resources which did not yet finish provisioning. (MR <https://gitlab.com/testing-farm/gluetool-modules/-/merge_requests/652>) The multi-host pipeline was fixed to have easily adjustable tmt logging options. (MR <https://gitlab.com/testing-farm/gluetool-modules/-/merge_requests/649>) All pipelines were fixed to support the skip result outcome introduced in tmt 1.28. (MR <https://gitlab.com/testing-farm/gluetool-modules/-/merge_requests/640>) 📦️ *Packages* List of important packages bundled in the worker image. ❯ podman run --entrypoint rpm quay.io/testing-farm/worker:2023-10.2 -q tmt standard-test-roles ansible-core podman beakerlib | sort | uniq ansible-core-2.14.11-1.fc38.noarch beakerlib-1.29.3-2.fc38.noarch podman-4.7.2-1.fc38.x86_64 standard-test-roles-4.11-2.fc38.noarch tmt-1.29.0-1.fc38.noarch On the behalf of the Testing Farm Team, /M -- 🧑‍🔧 Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE 🎩 Red Hat Slack :: mvadkert :: #testing-farm #tmt 📱 Fedora Matrix :: mvadkert :: #Fedora CI 🌍️ Remote Czech Republic :: Red Hat Czech s.r.o

1 0

Fedora CI testing affected by systemd/selinux issues
by Miroslav Vadkerti 23 Nov '23

23 Nov '23

Hi, There is an ongoing issue with systemd update breaking plenty of tests in Fedora CI. We are trying to resolve the issue. You can watch for status updates here: https://status.testing-farm.io/issues/2023-11-23-public-ranch-systemd-selin… It is affecting a lot of installability tests, possible also some functional tests might be affected: https://pagure.io/fedora-ci/general/issue/447 Best regards, /M -- 🧑‍🔧 Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE 🎩 Red Hat Slack :: mvadkert :: #testing-farm #tmt 📱 Fedora Matrix :: mvadkert :: #Fedora CI 🌍️ Remote Czech Republic :: Red Hat Czech s.r.o

1 0

Re: The OSCI team cannot maintain its Fedora CI tests anymore
by Martin Cermak 22 Nov '23

22 Nov '23

My group is definitely impacted. I'm trying to reach out to you privately so that we can discuss this surprising news. Thanks, Martin

1 0

Re: The OSCI team cannot maintain its Fedora CI tests anymore
by Martin Cermak 22 Nov '23

22 Nov '23

My group is definitely impacted. I'm trying to reach out to you privately so that we can sync. Martin

1 0

Automatic check for bodhi updates not to break RPM dependencies
by Miro Hrončok 11 Nov '23

11 Nov '23

On 07. 11. 23 12:17, Sandro Mani wrote: > Hi > > Due to an unfortunate oversight of an incorrect branch merge a couple of months > ago, a recently backported security fix caused an unwanted gdal soname bump in > F37, due to an update from the 3.5.x series to the 3.6.x series. > > I'm preparing a gdal-3.6.2-8.really3.5.3.fc37 to address this, please don't > rebuild any dependencies in the meantime, as the new package will bring back > the previous soname. > > Apologies for the troubles. Hello, this is not the first time I saw a bodhi update that breaks dozens of dependencies, goes unnoticed for a week and is automatically pushed stable, only to discover many packages fail to install. How come we don't have an automatic check for this? What can be done? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 1

[release] Testing Farm 2023-10.1 with tmt-1.28.2
by Miroslav Vadkerti 08 Nov '23

08 Nov '23

Testing Farm release 2023-10.1 <https://docs.testing-farm.io/Testing%20Farm/0.1/releases/2023-10.1.html> is deployed 🎉. 🎮 *Upgrades* tmt <https://tmt.readthedocs.io/en/stable/> updated from 1.27.0 to 1.28.2 <https://github.com/teemtee/tmt/releases/tag/1.28.0>. Artemis <https://gitlab.com/testing-farm/artemis> upgraded from v0.0.63 to v0.0.65 <https://gitlab.com/testing-farm/artemis/-/releases/v0.0.65>, see also release notes for v0.0.63 <https://gitlab.com/testing-farm/artemis/-/releases/v0.0.63>, v0.0.64 <https://gitlab.com/testing-farm/artemis/-/releases/v0.0.64>. Testing Farm CLI <https://docs.testing-farm.io/Testing%20Farm/0.1/cli.html> tool updated to v0.0.14, please update your installation for the new features to work. 🎯 *Highlights* The initial version of the multihost testing is now available. The testing is opt-in and has some limitations. See our documentation for details <https://docs.testing-farm.io/Testing%20Farm/0.1/test-request.html#multihost…>. Our results viewer has been updated to properly display <https://artifacts.dev.testing-farm.io/54252652-707d-4a35-aa44-2886033a237b/> the test environments and results. Happy experimenting! (TFT-2165 <https://issues.redhat.com/browse/TFT-2165>) Request cancellation <https://docs.testing-farm.io/Testing%20Farm/0.1/test-request.html#cancel> in Testing Farm is now possible. Use the DELETE method on a request or the testing-farm cancel CLI command to cancel a request easily. (TFT-1696 <https://issues.redhat.com/browse/TFT-1696>) The Staging API is now accessible via api.stage.testing-farm.io, offering a preview of our API reimplementation using the FastAPI framework. (TFT-2237 <https://issues.redhat.com/browse/TFT-2237>) The order of artifact installation <https://docs.testing-farm.io/Testing%20Farm/0.1/test-request.html#artifact-…> can now be altered, addressing some corner cases when installing various builds simultaneously. (TFT-2172 <https://issues.redhat.com/browse/TFT-2172> ) ️🐞 *Bugfixes* A bug that prevented the cleanup of resources during pipeline timeouts has been fixed. This was a RHIVOS blocker. (TFT-2312 <https://issues.redhat.com/browse/TFT-2312>) Repositories on Beaker boxes have been unified with standard testing images, such as rhel-CRB, etc. (TFT-2228 <https://issues.redhat.com/browse/TFT-2228>) An appropriate error message is now returned if a user tries to install a repository artifact that is too large. (TFT-2169 <https://issues.redhat.com/browse/TFT-2169>) The stability of Beaker provisioning for ppc64 has been improved. (TFT-2235 <https://issues.redhat.com/browse/TFT-2235>) Documented reserved directories <https://docs.testing-farm.io/Testing%20Farm/0.1/test-environment.html#_rese…> used by Testing Farm in the test environment. (TFT-1473 <https://issues.redhat.com/browse/TFT-1473>) Non-printable characters are now properly XML-encoded in jUnit XML output. ( TFT-2046 <https://issues.redhat.com/browse/TFT-2046>) The transition to sentry-python for error reporting on the Sentry platform from our workers is complete. (TFT-2227 <https://issues.redhat.com/browse/TFT-2227>) 📦️ *Packages* List of important packages bundled in the worker image. ❯ podman run --entrypoint rpm quay.io/testing-farm/worker:2023-10.1 -q tmt standard-test-roles ansible-core podman beakerlib | sort | uniq ansible-core-2.14.11-1.fc38.noarch beakerlib-1.29.3-2.fc38.noarch podman-4.7.2-1.fc38.x86_64 standard-test-roles-4.11-2.fc38.noarch tmt-1.28.2-1.fc38.noarch 🔍 *Statistics* Testing Farm has passed 700k requests <https://stats.testing-farm.io/d/dpYooDIVk/testing-farm-all-time-stats?orgId…> per year \o/ Error rate ~ 3.45% from start of October <http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?viewPanel=12&orgId=…> ~ 68k testing requests in October <http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?viewPanel=8&orgId=1…> Average queue time 30.4s <http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?viewPanel=26&orgId=…> on Public ranch, 53.5s <http://metrics.osci.redhat.com/d/NnHWU1dnz/testing-farm?viewPanel=27&orgId=…> on Red Hat ranch from the start of October. On the behalf of the Testing Farm Team, /M -- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE Red Hat Slack :: mvadkert :: #testing-farm #tmt Fedora Matrix :: mvadkert :: #Fedora CI Remote Czech Republic :: Red Hat Czech s.r.o

1 0

[status] Fedora CI / 2023-11-03
by Miroslav Vadkerti 06 Nov '23

06 Nov '23

Hi all, Fedora CI is very well alive! We will be sending out bi-weekly updates like this one about some of our plans, interesting information and status updates. Find us on the Fedora CI matrix chat <https://chat.fedoraproject.org/> in case you would like to discuss something. ⏱️ Plans - Close the mailing list and replace it with the Discourse tag - Move Fedora CI Jenkins under Testing Farm infrastructure | fedora-ci/general#440 <https://pagure.io/fedora-ci/general/issue/440> - Deprecate STI tests via Fedora 40 Change Proposal | general/issue#443 <https://pagure.io/fedora-ci/general/issue/443> 💡 Info - It is now easy to do integration testing of Fedora packages hosted on GitHub, see this blog post for details: https://cockpit-project.org/blog/tmt-cross-project-testing.html - Support for testing on AWS instances with dedicated AMD GPUs was deferred to November | fedora-ci/general#422 <https://pagure.io/fedora-ci/general/issue/422> - A prototype for running tmt tests against aarch64 on a pagure.io project was implemented for pagure.io/fedora-asahi/kiwi-descriptions project https://pagure.io/fedora-ci/general/issue/441 https://pagure.io/fedora-asahi/kiwi-descriptions/pull-request/48 <https://pagure.io/fedora-asahi/kiwi-descriptions/pull-request/48#> 🏁 Status - Pull requests created - Installability support for CS8 and CS9 prepared for review | fedora-ci/general#439 <https://pagure.io/fedora-ci/general/issue/439> https://github.com/fedora-ci/mini-tps/pull/27 https://github.com/fedora-ci/installability-pipeline/pull/24 - Closed issues - general/issue#423 <https://pagure.io/fedora-ci/general/issue/423> - moved to upstream Zuul - general/issue#441 <https://pagure.io/fedora-ci/general/issue/441> - prototype for testing on aarch64 for https://pagure.io/fedora-asahi/kiwi-descriptions - Outages - Occasional missing triggers for dist-git testing | general/issue#442 <https://pagure.io/fedora-ci/general/issue/442> - currently, the reason is unknown, please let us know if you encounter this in the above issue 🫂 *Status **Contributors* - Aleksandra Fedorova (bookwar) - Miroslav Vadkerti (mvadkert) Best regards, /M

1 1

2024

2023

2022

2021

2020

2019

2018

2017

CI November 2023