Hello fine fellows,
I hoped somebody else would reply, seems not :( To clarify, I and Dodji already talked
about this on IRC, and I asked him to send an email here to gather more feedback. I'll
summarize my thoughts here now, hopefully somebody else will comment on that as well.
The less good news is nobody got aware of the result of the task
because
the maintainer was most likely not notified. Because by default,
notifications about the result of this task are turned off. And another
message to the thread I mentionned above shows that it's not that easy
to turn those notifications on without making any mistake:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o....
Yes, FMN has a lot of rough edges and you definitely need to know what you're doing in
order to tweak something.
I guess my point is that experience is showing us that if this kind
of
notification is not enabled by default, people are just not aware of it.
I am thus proposing that notifications about the results of
task-abicheck that are either NEEDS_INSPECTION or FAILED be sent to
maintainers of the relevant packages *by default*.
We can extend the default FMN filter called "Critical taskotron tasks on my
packages" to also include dist.abicheck. I can't guarantee it will affect all
Fedora packagers, though. It will definitely not affect anyone who has done modifications
to the default taskotron filter (and they will have a hard time to merge those changes).
Also, we've seen many reports in the past in which certain users didn't have this
default filter present at all, or emails were not delivered. All of that can be surely
fixed by Fedora Infra, I'm just trying to point out that changing a default in FMN
doesn't currently guarantee all people will really receive the messages.
My second concern is about increasing the "spam rate" from our systems. This is
probably more of a general concern, not directly related to abicheck. We currently notify
individually per check and per arch, so if there are 5 "important" checks and 3
arches, you'll receive 15 individual notifications (provided they failed). It's
not that bad, but it does not scale into the future. I'd like to avoid a situation
where people rather turn off the notifications completely rather than receive so many
individual emails. We will definitely need something better in the mid-term future. I
imagine a system which knows that checks X, Y a Z are important for koji builds, waits
until they are all completed (but at most 60 minutes) and then sends a summary to the
user.
The best thing we have right now is Bodhi. I know it's late in the process (for some
checks), but it can tell you something's wrong in a single message ("autopush
disabled because task XYZ failed"). Also allows you to look through all the results
before clicking "push to stable". Yes, it's not ideal, but better than
nothing.
So the question probably is whether to prefer Bodhi, late in the process but less
intrusive, or FMN, early in the process and possibly annoying. Truth be told, a) I
don't like FMN too much at its current state, it's more like a proof-of-concept
than something to be relied on (filter maintenance is a tremendous pain from user POV),
and b) even if FMN was perfectly polished, I don't think it'd be the right
solution to the problem anyway. It's too low-level and can't aggregate messages
into a bigger whole (by design!) as we need it. We used it as a temporary solution to have
at least /something/, at as it usually happens, the temporary solutions grow and grow into
monstrosities. I wonder whether we're heading that way, and whether abicheck should be
considered an acceptable exception.
The good thing about abicheck is that it is well maintained with fast response rates and
its output is reasonably readable. Also, it doesn't seem to produce too many
failures[1][2], which is good because it hints there might be not too many false negatives
(remember that it does run only in critpath packages atm, however). Those could be some of
criteria we should have when deciding whether inform people about some particular check
results by default, and abicheck passes those. A counterexample would be rpmgrill - while
useful, its json output is hardly readable (Matthew recently complained on test list about
that). So I don't have concrete objections to abicheck itself, but whether it is a
good idea to push more things (anything) to FMN and risk the backlash of a subset of
maintainers.
Regardless of the solution (even if we just decide to disable autopush in Bodhi), we
should probably make it much clearer where to report issues with abicheck results, because
it will be the first check in the "release critical" set not directly maintained
by us. That would probably be a topic for a separate thread, but for each task I'd
like to present clear visible instructions with some FAQ and links where to report issues.
Also, should *we* be the ones who decide which failure notifications people receive by
default? Or FESCo, or someone else?
Wdyt people?
[1]
https://taskotron.fedoraproject.org/resultsdb/results?testcase_name=dist....
[2]
https://taskotron.fedoraproject.org/resultsdb/results?testcase_name=dist....