Hi fellow Fedorans.
Recently, AutoQA has been introduced to catch typical problems early
in
the update process. In general, I appreciate that effort, but
currently
I find myself in a phase of conditioning ignorance towards AutoQA,
essentially because it is drowning me in irrelevant information. The
current case why I'm writing is
https://admin.fedoraproject.org/updates/lua-wsapi-1.3.4-4.fc15.
Here are two ideas to make AutoQA relevant, less time-consuming, and
more helpful. In short: good QA is always quiet, only if there is a
problem it communicates.
- Post only errors
It is common, for example, in automated build or continuous
integration
systems to send out emails only on errors. Similar goes for Unix
tools,
which tend to be quiet if everything is ok, and only bother you with
output if something is not. Therefore, I propose to have AutoQA
messages
posted only in case that there has been an error.
How can you then distinguish an update for which the tests have passed from an update for
which the tests haven't yet been executed?
Moreover, currently not all updates are tested. Sometimes our tests simply don't work
properly. Not just the updates are being tested, the whole AutoQA is being tested (and
developed) in this whole effort.
- Accumulate error messages
An email is sent for every single comment to Bodhi. In the case of
AutoQA, it causes one email per platform. It increases the load of
email
tremendously to deal with, which in turn makes me ignore it.
Therefore,
I propose to accumulate messages for all platforms.
We have that in plan, believe me.
Combined with the
earlier proposal, the states for all platforms should be collected by
an
intermediate node, and if and only if a test failed on any of the
platforms, one message with all status messages is posted to the
update.
Sending Bodhi comments is just a quick way how to inform the maintainers. We are working
on a results database with API that other Fedora services (Koji, Bodhi) could query and
use the results as they seem fit. For most tests I expect it will be similar to what you
describe. But that's future. Until that's implemented we can only either send
comments to Bodhi or send no comments at all.
On a related note: it'd be much appreciated if Bodhi would provide an
option to get a daily digest with all comments of all the packages I'm
involved with.
Great idea, you can ask lmacken about that (or create ticket in its Trac).
Or, you can filter your emails and check the relevant folder once a day :)
I hope the fine folks of the AutoQA effort take these proposals into
account when proceeding in the development of the system and help me
to
stop ignorance from taking over.
It will take some time, but we see the deficiencies, same as you do.
We try to improve as fast as possible.
Regards,
Tim
Thanks,
Kamil
PS: We have a special mailing list for AutoQA:
https://fedorahosted.org/mailman/listinfo/autoqa-devel