Excerpts from Adam Williamson's message of 2017-05-02 17:45 -07:00:
On Wed, 2017-05-03 at 10:09 +1000, Dan Callaghan wrote:
> Excerpts from Adam Williamson's message of 2017-05-02 16:15 -07:00:
> > So, first the good news: fundamentally, the concept works. I can tell
> > Bodhi to query 'all bodhi_update results for the item FEDORA-2017-
> > 4e95959f0d' and then display (only) the latest result for each
> > 'scenario' that appears in the whole set of results.
>
> I think this might be the fundamental problem here. I think the
> ResultsDB "item" should not be an update, but the set of packages in an
> update. If someone edits the update to change the set of packages in it,
> the previous results are invalidated and new tests are kicked off.
>
> Putting aside the issue of how to actually make the "item" be a set of
> packages -- I think this would eliminate all of these issues with OpenQA
> scenarios.
Uh, I think you're misunderstanding something,
Yes absolutely. :-)
I thought all the "scenario" complexity was because the item
FEDORA-2017-4e95959f0d could have more than one result per test case
*across time* (if an update was edited), but you have already solved
that problem by filtering on update modification timestamp. I didn't
realise that.
But actually the "scenario" is because the item FEDORA-2017-4e95959f0d
could have more than one result per test case because OpenQA tests
different things under the same test case name?
So you are right, I was actually suggesting two totally unrelated
things. Changing the item from being an update to being a set of
packages doesn't actually help anything.
because it just wouldn't
do so at all. Even if openQA tested by package rather than by update,
it would still run the same four tests twice for each package, once
starting from a Server base image, once starting from a Workstation
base image. That change would make no difference whatsoever to this
problem.
> OpenQA would simply report results with test cases like this, with all
> your scenarios and flavours and so forth as different permutations in
> the test case name:
>
> graphical_installation.fedora.26.server.x86_64.blahblah
> graphical_installation.fedora.26.workstation...
This is an entirely different suggestion that has nothing to do with
your other suggestion: you're effectively suggesting that we simply
encode the scenario into the test case name. This was actually one of
the alternatives I floated in my very first mail on this topic, but we
didn't really like it for several reasons.
But unlike your other suggestion this one *would* actually work to help
Bodhi's use case; it would let Bodhi just go ahead and treat the test
case name as the 'scenario' with no need for special casing at all
(assuming we also changed the test case name for Taskotron depcheck
tests to something like 'dist.depcheck.i686' and
'dist.depcheck.x86_64').
Right... so I think what I *actually* wanted to ask was, why not do
this? :-)
I have to admit that I got lost in this thread originally and stopped
following it. Re-reading it all again now, I see that Kamil wrote up
a good summary of the plan here:
https://lists.fedoraproject.org/archives/list/resultsdb-users@lists.fedor...
where option A was to add some new mandatory fields to distinguish the
same test run in different environments. So a ResultsDB result is not
unique on just item+testcase, but rather on
item+testcase+(some other fields that vary per item)+scenario.
Option B was basically what I chimed in with above: jam everything into
the test case name to make it unique.
What you have implemented in your Bodhi patch is actually to just treat
the results as unique on item+testcase+scenario (without arbitrary extra
fields that vary per item). And the different combinations of OpenQA
installation test are smooshed together into the scenario string. Right?
Your Bad News One is that ResultsDB is not efficient at finding the
latest result by item+testcase+scenario. But I am sure we can fix that
in ResultsDB if we want. Worst case, by special-casing "scenario" in the
db schema and promoting it out of extra_data into its own db column with
an index.
Your Bad News Two is that you either display the scenario strings in
Bodhi, and they look kinda ugly and hard to read, or you *don't* display
them in Bodhi and then it looks like there are duplicate test results.
This problem already existed for the test case names though, right?
Those are also ugly, human-unfriendly strings. The solution for the test
cases is that ResultsDB stores some extra metadata about them, with
a link off to a human-friendly page in `ref_url`. (Although I note that
Bodhi right now doesn't show that info anywhere...)
Could you do the same thing for scenarios? Is there an unreasonably
large number of different scenario combinations? Do new ones appear
often?
And if you *could* do that -- maybe it would make sense to just smoosh
it into the test case name, and go back to having results be unique on
item+testcase to keep things simple?
(BTW the reason I am interested in this -- and the reason why I am
trying to help come up with the simplest possible way to model it -- is
because whatever is decided on, the "policy engine" service we are
working on will have to know how to deal with it. Right now we were
assuming that results are unique on item+testcase but there are proving
to be lots of hairy edge cases around that.)
--
Dan Callaghan <dcallagh(a)redhat.com>
Senior Software Engineer, Products & Technologies Operations
Red Hat