On Mon, 2018-03-19 at 15:57 +0100, Pierre-Yves Chibon wrote:
> On Mon, Mar 19, 2018 at 03:41:15PM +0100, Dan Horák wrote:
> > On Mon, 19 Mar 2018 14:06:56 +0100
> > Dan Horák <dan(a)danny.cz> wrote:
> >
> > > On Sun, 18 Mar 2018 20:25:31 +0100
> > > Fabio Valentini <decathorpe(a)gmail.com> wrote:
> > >
> > > > On Sun, Mar 18, 2018 at 7:13 PM, Michael Cronenworth
> > > > <mike(a)cchtml.com> wrote:
> > > > > On 03/18/2018 01:02 PM, Fabio Valentini wrote:
> > > > > >
> > > > > > I've looked at waiverdb-cli too, but since no tests
seem to have
> > > > > > run at all, it looks like the wrong tool for the job:
> > > > > > I don't want to push an update despite a failed test, I
want to
> > > > > > push my update despite no test data being available ...
> > > > >
> > > > >
> > > > > Randy said the tests refresh every 6 hours and/or every time
the
> > > > > update is edited. Neither seemed to have occurred for you.
> > > >
> > > > Exactly. The "no test results found" status in bodhi
hasn't been
> > > > refreshed in over 10 days now.
> > > >
> > > > Bodhi also displays that all these tests were successful, bit still
> > > > blocks the update because "no test results found", which
is
> > > > obviously just wrong.
> > > >
> > > > A manual lookup in resultdb shows me that the tests have in fact
> > > > been run and have all passed:
> > >
> > >
https://bodhi.fedoraproject.org/updates/FEDORA-2018-200708ae05 is in
> > > the same situation, all tests are green, but "no test results
found"
> > > is reported. It's not very user friendly ...
> >
> > and
https://bodhi.fedoraproject.org/updates/FEDORA-2018-71350d90a7 is
> > even more interesting with "The update can not be pushed: 1 of 2
> > required tests not found", but the listed tests are again all green. No
> > idea what's missing from the output.
>
> All the tests can be green if the "important" ones are missing, they
don't show
> :(
>
> The important ones are the ones defined in the policy that gates packages and
> are listed here:
https://fedoraproject.org/wiki/CI/gating_updates
>
> waiverdb-cli should now support waiving missing results, I'm double-checking it
> and see if we can document it at:
>
https://fedoraproject.org/wiki/Package_update_HOWTO#Handling_feedback_fro...
> next to the other examples.
It is also still the case that unpushing and re-pushing the update
should re-trigger the tests in Taskotron, at the cost of re-setting the
karma and 'wait time' clocks for the update (so you'll need to either
get positive karma or wait 7 or 14 days before being able to push it,
from the time of the re-push).
One obvious easy win here would be to change the "No test results
found" text, as it's clearly confusing. It could be something like "No
results found for blocking tests", perhaps? We could even give Bodhi
the ability to list the names of the 'expected' blocking tests, and
have the text show that somehow, whether as a hyperlink or perhaps just
as a mouseover or something?
Yeah, the text is misleading.
Greenwave supplies that "summary" line, and agreed - it should be
updated to be more informative on its own. This came up in
decathorpe's issue thread here:
FWIW, greenwave does supply the list of missing and/or failed testcase
names in its API response back to Bodhi. Surfacing those more
granular details in the Bodhi UI would be good.