Hi folks!
I am currently rolling out some changes to the Fedora openQA deployment
which enable a new testing workflow. From now on, a subset of openQA
tests should be run automatically on every critpath update, both on
initial submission and on any edit of the update.
For the next little while, at least, this won't be incredibly visible.
openQA sends out fedmsgs for all tests, so you can sign up for FMN
notifications to learn about these results. They'll also be
discoverable from the openQA web UI -
https://openqa.fedoraproject.org
. The results are also being forwarded to ResultsDB, so they'll be
visible via ResultsDB API queries and the ResultsDB web UI. But for
now, that's it...I think.
Our intent is to set up the necessary bits so that these results will
show up in the Bodhi web UI alongside the results for relevant
Taskotron tests. There's an outside possibility that Bodhi is actually
already set up to find these results in ResultsDB, in which case
they'll just suddenly start showing up in Bodhi - we should know about
that soon enough. :) But most likely Bodhi will need a bit of a tweak
to find them.
Let me add a bit of a technical background here. Bodhi web UI now queries ResultsDB for
all available testcases, and then asks for item=$item&type=koji_build for all these
testcases. So the new results won't be visible (unless you start reporting them per
build).
Our depcheck and upgradepath tasks report both type=koji_build (for each SRPM) and
type=bodhi_update (for each bodhi update). Both these tools internally process RPMs or
builds, and then query Bodhi at the end and collate the results to bodhi_update results.
We want to get rid of that collating, because it has numerous issues:
1. It's slow, because Bodhi is very slow to respond
2. It often causes the task to fail, because Bodhi often returns 500 errors
3. It's prone to race conditions. It happens often that a Bodhi update is edited
between the task start and end, changing included builds.
4. It's unnecessary, because Bodhi knows all this information, and can collate the
data itself, without any network issues or race conditions.
So while we still publish type=bodhi_update for compatibility reasons (I'm not sure
whether some part of bodhi backend still might use this data), but want to get rid of it.
So my first thought was to recommend you to also publish just type=koji_build results and
finish this transition. But then I realized that's wrong. OpenQA operates completely
different than the aforementioned tasks do. We operate on builds, and can distinguish
which build is causing issues. Collating them into bodhi_update results is just a
convenience measure for the consumer. But you operate on the whole update as a set. You
can't distinguish which build of the update caused the issues, you just know that some
of them did. So the smallest unit for you is bodhi update, and you should report results
as such.
The way forward is, I believe, to extend Bodhi to query both type=koji_build for all
included builds and collate the results (if needed), and also query type=bodhi_update and
shows those results as well. Because different tasks operate on different type of data,
which influences how they publish the results. And both use cases are valid.
There's one important thing we need to do first, though. Bodhi ID doesn't identify
the thing tested uniquely, because Bodhi updates are mutable (and the ID is kept). So
Bodhi (or any gating tools) can't rely on just retrieving the latest result for a
particular Bodhi ID and trust that result. It might be old and no longer reflect the
current state. We need to extend bodhi_update results with "timestamp" key in
extra data, that will report the "last_modified" time of the Bodhi update
tested. And Bodhi (or any other tool) must not only query for
item=$bodhi_id&type=bodhi_update, but also for ×tamp=$timestamp. Only with
this we can be sure we've really tested particular Bodhi update.