> a) query
`results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-
> Rawhide-20170228.n.0.iso` and then go through all the results, make
> them unique by eliminating everything that has the same 'scenario',
> and work with that
This was one of the primary cases I was thinking of, yes. This is the
case that releng would be using to decide whether to release a compose,
for instance.
I'll go slightly off-topic here, but consider this: 'scenario' will help us
identify unique results. But it will not help us decide whether "all" results
have been submitted (maybe some are still running, maybe some have crashed hard and not
sent any results). So there's a good chance that rel-eng tools will need to have deep
knowledge of the testcases anyway. Not just list of all monitored testcases, but also that
this particular test case needs to be performed for bios and uefi (or, that this
particular testcase needs to be present with these two scenarios).
So while having scenarios seems definitely helpful in certain cases, it might not help us
in avoiding having too much knowledge in the gating consumers. Just food for thought.
It's also the case for listing openQA results in Bodhi:
we'll want Bodhi to grab all results for the update, then de-duplicate
by scenario (taking only the most recent result for each scenario).
> b) query `results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-
> Rawhide-20170228.n.0.iso&unique_key=scenario` and have the
> duplication filtering implemented inside resultsdb (this way it seems
> it can be implemented in a generic fashion), work with that
Oo, is that possible? If so, neat. But conceptually it's the same as
a), right? Just a case of where the filtering gets done.
Yes, it would be the same, just more resource-friendly. Not sure if Josef will be inclined
to implement this, he became quite good in saying 'no' :)
> c) query `results?testcases=compose.*&item=Fedora-Server-dvd-x86_64-
> Rawhide-20170228.n.0.iso&scenario=fedora.Rawhide.Server-dvd-
> iso.x86_64.server_realmd_join_kickstart.64bit` and work with that.
> But in this case the consumer would again have to have a deep
> knowledge of how the scenario is constructed, which would miss the
> whole point, I think.
Well, see the above examples - I was kinda envisioning that you'd
*discover* the scenario you're interested in from some kind of existing
result, or just query for all results from two *different* composes,
then use the 'scenario' key to compare the results for the two composes
(so you can say 'ok, scenario A passed in compose 1 but failed in
compose 2'). This is the sort of thing check-compose does - that's how
it produces a list of 'tests that failed today but passed yesterday',
etc. It doesn't *know* what the scenarios are before it queries the
results, but *once it has the results* it calculates the scenarios and
uses them for comparison.
Fair enough, for this use case it's really useful. But it will not be the primary way
of using scenarios, and that was my point, clarifying how it's going to be used
primarily.
> > Note I don't think we should include the 'item' in the scenario for
all
> > the reasons discussed above; it should just be understood that for
> > different purposes you might want to query on "scenario + item" or
just
> > "scenario".
>
> I don't understand why you would ever want to put testcase name into
> the scenario field. Both the item and the scenario are defined
> externally in a well known fields that are mandatory to fill in.
> There's no guessing like with 'arch' or 'firmware' (whether it
> exists, what the allowed values are). So why wouldn't you specify
> only the extra args that make the execution unique in the scenario
> field? If you ever need to search through multiple testcases with the
> same scenario value (I don't know why you would need that, but let's
> assume you do), you can always do that either for a particular
> testcase like this:
> results?testcases=compose.base_selinux&scenario=fedora.Rawhide.Server
> -dvd-iso.x86_64.64bit
> or across many testcases by using a list or a wildcard:
> results?testcases=compose.*&scenario=fedora.Rawhide.Server-dvd-
> iso.x86_64.64bit
>
> I don't think you even want to do this query:
> results?scenario=foobar
> because you can *bet* that you'll eventually receive some results
> from scratch.tmp.testthis.fake namespace or similar. You should
> always specify at least a wildcarded testcase (compose.openqa.*). I'd
> like our system to allow in-development, proof-of-concept and
> experimental tools to run and submit results. There will be errors,
> mistakes, copycats, everything. But they will be constrained to their
> namespaces. And so should you also limit your queries only to those
> namespaces that are known and maintained with the level of quality
> you need for your purposes. You can imagine the concept to be the
> same as e.g. github - you always work in your namespace, and there's
> no global namespace, because that's just asking for trouble.
Well, the funny thing is, for all the same reasons I come to the other
conclusion :) I entirely agree that you'd never want to use the
scenario *without* the test case name...so in that case, saying "you
should always query for the test case name plus the scenario" seems
like pointless make-work, right? If we know that there's never any
point in just using the scenario values *without* the test case name,
why not just stuff the test case name right into the scenario value so
you only have to query for 'scenario', rather than always having to
query for both 'test case name' and 'scenario'?
To be fair, I can think only of a few minor points:
* Data duplication, makes the field longer and harder to read.
* A possibility for errors. Since tasks are allowed to add arbitrary suffixes to their
testcase names, a task can create tens of "sub-testcases" (e.g. look at
dist.rpmgrill*). The scenario field is not automatically generated but fully under
task's control. There might be copy&paste errors or logical errors in the code,
and the scenario might not correctly reflect the used testcase name.
* Consistency (even though that can be arguable). I'd like to be clear that
testcase+item combination is always the default way to consume results. For more complex
tests, it might also require looking at scenario to distinguish unique results. If I write
instructions that say that results are identified by item+scenario if scenario exists,
otherwise as testcase+item, it seems more complex and less obvious to me. (But this is
maybe just about phrasing).
But it's really a pretty minor point. The system would work fine either
way, it just slightly changes what consumers have to do.
Does anyone else have thoughts on this? I'd like to either land the
'scenario' change in the openQA reporter, or definitely decide we want
to do something else, tomorrow or early next week, so I can move
forward with the 'is this compose releasable?' script and displaying
openQA update test results in Bodhi...
Since you're clearly powered by Duracell batteries [1], please go ahead and implement
this. If we decide to implement this later in resultsdb directly, we can always go and
simplify the consumer code.
Btw, if you end up submitting patches to Bodhi and using 'type=bodhi_update'
queries, please also use the 'since=' argument [2] and set it to the update's
'date_modified' timestamp. That doesn't really implement the higher
reliability of such results (as we talked elsewhere), but it doesn't stress resultsdb
so much, which is always good (we can't do the same for type=koji_build easily, but we
can for type=bodhi_update).
Thanks.
[1]
https://www.youtube.com/watch?v=kaF6FxmixJk , am I really that old?
[2]
http://docs.resultsdb20.apiary.io/#reference/0/results/browse-the-results...