On Fri, 2020-02-28 at 10:06 +0100, Jan Pazdziora wrote:
On Fri, Feb 28, 2020 at 09:20:16AM +0100, Pierre-Yves Chibon wrote:
Depending on how the pipeline actually *does* testing of updates, you *may* be able to schedule on the update being created or edited instead/as well (this is how the openQA scheduler does it; we don't use the koji-build-group.build.complete topic at all, we use bodhi.update.request.testing (published when an update is created) and bodhi.update.edit (published when an update is edited) instead. However, you can only do that if the pipeline does not rely on the packages actually being in the updates-testing repo, but retrieves them itself. At the point those messages are published, the package files cannot be relied upon to be present anywhere outside of Koji.
Well, when the update is created the RPMs are signed yet and we want to test the signed RPM to make sure that we test what is being pushed to the users.
From reading this, I think things work as designed, it's just that there were no push to update-testing for F32 yet.
The question is, could the CI run based on some other messages, like those that Adam mentioned? Will the test setup be able to pull the bits from koji or does it rely on updates-testing? Bruno was able to schedule the CI job for my Fedora 32 update and it passed so it looks like it was able to get the bits in spite of no update-testing pushes for Fedora 32.
Pierre was answering that question, I think: his answer is that he doesn't want CI to pull direct from Koji because when you do that, you get RPMs which have not been through the signing process, and he thinks it's important to test exactly what actually goes out to users. It's a reasonable point.
In general it seems like a suboptimal user experience from maintainers' point of view when they create an errata and there is an undefined time for which the errata will show failing tests because the gating tests haven't been even scheduled. Any change we can make to get the erratas all ready (with tests run and passed) will increase the chance that maintainers will be willing to add gating tests to their packages.
I mean, it should be possible to fudge this, though it may involve the sort of magic some folks tend to be unhappy with. If we know this is a thing that happens, we can maybe make the UI look less angry and red when the state is 'tests not yet scheduled' as opposed to 'a test actually failed', right? We can even start doing more speculative stuff like having a timer - if all the required tests aren't scheduled *24 hours after the update is submitted*, that's more worrying that if they aren't scheduled 5 minutes after it's submitted. Or we could teach Greenwave or Bodhi or something about updates-testing pushes in some way...