On Fri, 2018-12-07 at 10:10 +0100, Pierre-Yves Chibon wrote:
> Yeah, that's what I suggested above - but it seems this wasn't really
> *intended* for that.
It very much was intended for this, back then there was no opting-in via a file
in dist-git so the only way to know for the decision system if a package is
gated and the tests haven't run vs is not gated at all was: have the CI system
tell that it ignores the package.
Oh, OK. From Miroslav and Vit's replies, it sounded like this (the
'ignore' results) was just a thing that no-one had really thought
through much and just kinda 'happened', it wasn't consciously designed
to help with this issue.
> Still, if we do want to tackle that problem, we do need to come
up with
> some ideas. It *is* a real problem, if we want to gate on tests which
> are not run on all packages. I suggested the possibility that Greenwave
> could have a 'test must pass or not have run' type of rule, but you or
> someone else pointed out that comes with obvious holes, like Bodhi will
> consider the update to be passing that rule if the test has been
> *scheduled* but not *finished* yet.
I will not dispute that it can be causing problems, nor that we can improve it,
just stating the history of the current situation :)
OK, I wasn't clear that you were saying it actually *was* intended to
solve this problem. Thanks.
> Sending 'ignore' results or something like that is one
potential way to
> deal with the problem, though I can see it getting messy. For instance,
> openQA runs different sets of tests on different updates. I had to set
> up a whitelist mechanism to get the server tests run on FreeIPA-related
> updates that aren't on the critical path; as part of that whitelisting
> mechanism, I set things up so it runs *only* the server tests on those
> updates, it skips the desktop tests). What do we do if we want to gate
> on the desktop tests? openQA would have to do something like send a
> 'test not run' result for *every individual test it *might* have run*,
> or send something like a 'ignored desktop tests' result and greenwave
> would have to somehow be able to "understand" that.
>
> Conceptually this approach would be sort of mildly abusing ResultsDB as
> a database of test *execution*, and that seems kinda the wrong design
> to me.
>
> We do in fact *have* a database of test execution! It's called ExecDB:
>
>
https://pagure.io/taskotron/execdb
But my understanding is that execdb follows the status of the running test, a
package that is ignored could be approach as a result (something along the lines
of "no results is a result").
At this point we're getting pretty philosophical :P To me, personally,
that doesn't feel right - "no result" *isn't* a result. But I can see
that it might feel correct to someone else. Maybe I'm the minority
here. :P
We would still need to come up with some way to cover the "ran *some*
tests, skipped *some* tests" scenario if we wanted to carry on with
this approach, though.
> Anyone have any thoughts?
Another approach could also be: gate everything except what the packages set in
dist-git to not be gated (basically an opt-out instead of an opt-in), this way
there is no longer a need for a notification from the CI system about ignoring
since the decision maker will know not to look for test results.
This has of course a high cost: make packagers opt-out instead of opting in.
Thinking of it, since packages now opt-in, maybe we do not need the ignore
anymore, since the decision maker will know to gate package only if they have
asked to be gated, which means: if there are no test results, then either the
tests haven't run in which case it should be gated or the tests failed to run,
which should be notified to the CI pipeline admins and the best person to do
this would be the packager here as well.
Does that make sense?
It makes sense for our *current* implementation of *update* gating,
mostly (though it does kinda place a load on the update owner to know
what tests will be run for the update, as opposed to just letting them
say 'well hey that sounds like a good test to gate on, assuming it gets
run' and have The System handle it if that test *doesn't* get run).
But that's only what we're doing right now, and I know some folks want
us to go a lot further with update gating. What if, say, we decided we
wanted to default gate updates on the 'desktop terminal' openQA test,
on the basis that if that test gets run and fails, there's probably a
serious problem somewhere? That test isn't run on *all* updates. We can
hardly tell all packagers "you must go figure out if this test runs on
your package, and opt it out of the gating if it doesn't", that seems
kinda impractical to me.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net