On Fri, 2017-03-03 at 15:12 -0800, Adam Williamson wrote:
If we implement a 'scenario' key, we can then just teach
Bodhi to look
for that and combine it with the item, then maybe if it can't find one,
treat 'testname-arch' as the scenario (as a fallback for old results,
results from Taskotron until we teach it to write 'scenario' items into
its results, and a general fallback for cases where 'scenario' is
missing for some reason). That could be the general convention, I
guess: first look for a 'scenario' key, otherwise try and construct one
from very commonly available keys.
So I come back, a whole two months older and wiser, with a report from
the field on this ;)
In the end we went ahead with more or less my initial plan for the
'scenario' handling: openQA reports a single 'scenario' value in the
extradata for each report it submits, containing a string
representation of the 'scenario' (it does this by smooshing the values
of all the openQA 'scenario keys' together using periods, for the
record).
Then I got to teach Bodhi to handle openQA results. Happily, this
*does* involve dealing with the 'scenario problem' (though the initial
implementation we did in Bodhi ignores it, and winds up omitting some
openQA results). For each update, we run four of the tests twice, once
on a Server image, once on a Workstation image. In openQA's terms, this
is the same test run on a different 'flavor', and 'flavor' is one of
openQA's "scenario keys".
So for reference, here are a couple of sample results:
https://taskotron.fedoraproject.org/resultsdb/results/13573636
https://taskotron.fedoraproject.org/resultsdb/results/13573633
Same 'testcase', same 'item', same 'arch' even...but different
'scenario':
fedora.26.updates-workstation.x86_64.base_selinux.64bit
fedora.26.updates-server.x86_64.base_selinux.64bit
So, first the good news: fundamentally, the concept works. I can tell
Bodhi to query 'all bodhi_update results for the item FEDORA-2017-
4e95959f0d' and then display (only) the latest result for each
'scenario' that appears in the whole set of results.
However, there are two pieces of bad news.
Bad News One: As kparal pointed out in the original thread, the
`/latest` endpoint that was added to ResultsDB 2.0 and intended to help
optimize queries like this doesn't handle 'scenario' cases at all: it
only gives you the single 'latest' result for each test case in your
query, there is no option to ask for 'latest by scenario' rather than
'latest by test case'. As we discussed further in the thread,
implementing this in RDB would be difficult (though not impossible) and
not as efficient as latest-by-testcase, due to details of how RDB
actually stores data. So Bodhi has to either:
1. use `/latest` but then have special knowledge of the cases where it
needs to go and run supplemental queries to handle 'scenario' cases
(which is what it currently does for depcheck, and what my pull request
does for openQA for now)
2. just not use /latest and get *all* the results for the items, then
do the latest-by-scenario work itself. This works, and results in
cleaner Bodhi code, but is slower and makes RDB work harder
Bad News Two: The 'just provide a scenario string' approach fails to
solve a problem I didn't think about until I came to actually work from
the consumer end. Say we get all the openQA results for a given update,
then do the 'latest-by-scenario' filtering using the 'scenario' values.
That works, and now we have all the results we need to display. Great.
But...how do we (Bodhi) *display* them?
Currently Bodhi displays each result as a table row, with four columns
- an icon indicating the outcome; the test case name; the result
'note', if there is one; and the time the result was submitted.
But if we just do that, then you really can't tell the difference
between the results for the 'same test, different flavor' cases. And
the key point is, *RDB doesn't really help Bodhi here*. The ugly,
smooshed-together scenario string does the job I meant it to do - lets
Bodhi identify which results it should display - but it doesn't do a
job it ought to do, but which I didn't think about: it doesn't tell
Bodhi how it can cleanly display the results with the same test case
name so that a reader actually knows what the difference is. Bodhi
*could* just stuff the scenario name in the table somewhere (or use it
instead of the test case name), but it wasn't actually designed to be
read and looks pretty ugly.
So I wound up just having to do ugly special-case stuff again, and
effectively 'teach' Bodhi that the 'scenario' element that can actually
vary between openQA results at present is the 'flavor', and teach it to
parse the flavor out of the scenario value (stupidly, we don't actually
report a 'flavor' key in the extradata...I should fix that) and display
it as a label next to the test case name. (There is also pre-existing
similar 'special casing' to have it display the arch as a label, for
the dist.depcheck case we've talked about before.)
Addressing Bad News One is a bit difficult, as mentioned. But perhaps
we can address Bad News Two with a second try at implementing the
'scenario' concept in results. I think, somewhere back in the thread,
kparal suggested that each result could include each scenario-relevant
value in its extradata, and also indicate which keys were the 'scenario
keys' for the result. I thought that was over-complicated at the time,
but it'd actually help address Bad News Two. Bodhi would then have the
opportunity to, say, compare all the scenario values for the results
for the item, ignore ones that are the same for *every* result, and for
the values that actually do vary across some of the results, display
those values in the result table. That wouldn't require it to have
special knowledge.
Just in case this email isn't long enough, here's my Bodhi PR to
implement 'scenario' handling for openQA results, with a nice long
commit message:
https://github.com/fedora-infra/bodhi/pull/1496
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net