Hi folks!

We've had openQA testing of updates for stable and branched releases, and gating based on those tests, enabled for a while now. I believe this is going quite well, and I think we addressed the issues reported when we first enabled gating - Bodhi's gating status updates work more smoothly now, and openQA respects Bodhi's "re-run tests" button so failed tests can be re-triggered.

A few weeks ago, I enabled testing of Rawhide updates in the openQA lab/stg instance. This was to see how smoothly the tests run, how often we run into unexpected failures or problems, and whether the hardware resources we have are sufficient for the extra load.

So far this has been going more smoothly than I anticipated, if anything. The workers seem to keep up with the test load, even though one out of three worker systems for the stg instance is currently out of commission (we're using it to investigate a bug). We do get occasional failures which seem to be related to Rawhide kernel slowness (e.g. operations timing out that usually don't otherwise time out), but on the whole, the level of false failures is (I would say) acceptably low, enough that my current regime of checking the test results daily and restarting failed ones that don't seem to indicate a real bug should be sufficient.

So, I'd like to propose that we enable Rawhide update testing on the production openQA instance also. This would cause results to appear on the Automated Tests tab in Bodhi, but they would be only informational (and unless the update was gated by a CI test, or somehow otherwise configured not to be pushed automatically, updates would continue to be pushed 'stable' almost immediately on creation, regardless of the openQA results).

More significantly, I'd also propose that we turn on gating on openQA results for Rawhide updates. This would mean Rawhide updates would be held from going 'stable' (and included in the next compose) until the gating openQA tests had run and passed. We may want to do this a bit after turning on the tests; perhaps Fedora 37 branch point would be a natural time to do it.

Currently this would usually mean a wait from update submission to 'stable push' (which really means that the build goes into the buildroot, and will go into the next Rawhide compose when it happens) of somewhere between 45 minutes and a couple of hours. It would also mean that if Rawhide updates for inter-dependent packages are not correctly grouped, the dependent update(s) will fail testing and be gated until the update they depend on has passed testing and been pushed. The tests for the dependent update(s) would then need to be re- run, either by someone hitting the button in Bodhi or an openQA admin noticing and restarting them, before the dependent update(s) could be pushed.

In the worst case, if updated packages A and B both need the other to work correctly but the updates are submitted separately, both updates may fail tests and be blocked. This could only be resolved by waiving the failures, or replacing the separate updates with an update containing both packages.

All of those considerations are already true for stable and branched releases, but people are probably more used to grouping updates for stable and branched than doing it for Rawhide, and the typical flow of going from a build to an update provides more opportunity to create grouped updates for branched/stable. For Rawhide the easiest way to do it if you need to do it is to do the builds in a side tag and use Bodhi's ability to create updates from a side tag.

As with branched/stable, only critical path updates would have the tests run and be gated on the results. Non-critpath updates would be unaffected. (There's a small allowlist of non-critpath packages for which the tests are also run, but they are not currently gated on the results).

I think doing this could really help us keep Rawhide solid and avoid introducing major compose-breaking bugs, at minimal cost. But it's a significant change and I wanted to see what folks think. In particular, if you find the existing gating of updates for stable/branched releases to cause problems in any way, I'd love to hear about it.

Thanks folks!