On Tue, 2011-11-01 at 13:59 +0000, Matthew Garrett wrote:
It's a common complaint that it's too difficult to get
updates to
critpath packages through the update system at the moment. We've been
looking into trying to make that easier without just dropping the
critpath requirements, and one thing we looked at was whether the
requirement for positive karma from proventesters was a net benefit.
Thankfully this is the kind of thing that we can actually generate
numbers for. Luke pulled some statistics which are available at
http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html .
The relevant section here is the set of packages that have (a)
sufficient positive karma to be pushed, but (b) negative proventester
karma - that is, the packages where negative proventester karma
prevented a push.
Straight off, we can see that these amount to 1-2% of all critpath
updates. It's simply not common for proventester to make a difference to
the outcome. If we look at the individual packages, things get even more
interesting. Many of the updates receive a mixture of proventester
karma, so even with the negative the push would still go ahead. As far
as I can tell:
https://admin.fedoraproject.org/updates/xorg-x11-drv-geode-2.11.9-1.fc14
https://admin.fedoraproject.org/updates/system-setup-keyboard-0.8.6-2.fc14
are the only two updates where the proventester karma requirement would
have made a difference, out of 1942 critpath updates that made it to
stable. That doesn't seem like a great hit rate.
So, assuming I'm not grossly misanalysing the data, it seems that we
could drop the proventester requirement from critical path updates with
a negligable change in the quality of the updates. Thoughts?
So, we discussed this at the QA meeting a couple of weeks back.
Initially the plan was for anyone who had concerns to reply to this
mail, but that doesn't seem to have happened, so I will try to summarize
the various responses from that meeting. You can check the full summary
and log of the meeting:
https://fedoraproject.org/wiki/QA/Meetings/20111107
if you want to make sure I'm not misrepresenting anything.
adamw: 1-2% of all critpath updates (where proventester process is seen
to have 'had an impact') is a small number, but it is not 0. It only
takes *one* really bad update to cause big problems.
adamw: it's possible we could use proven tester feedback more
effectively: there have been cases where bad updates went out despite
negative feedback, because we currently don't give negative feedback -
even from proventesters - much significance at all. So there may have
been more cases where proventester input *may have been significant*
with a system where negative karma is more strongly considered. e.g.
updates could be blocked from being auto-pushed if there is any negative
feedback from a proven tester
red_alert (sandro mathys): critpath packages should have detailed test
plans, as currently proventesters often do not know exactly what they
need to verify with some critpath packages: "the process as its done
right now doesn't work (i.e. is no improvement to not having pts) but
having pts with a better process would surely add to the overall
quality"
tflink: we have to keep in mind the reason for the proposal -
maintainers frustrated that packages get stuck in updates-testing for
weeks - is a valid reason and a significant problem. "if we object to
getting rid of the proventester process, do we have any solutions to the
root of their complaints?"
brunowolff: how about keeping the PT karma requirement but adopting the
plan to allow critpath updates to go through without 'required' karma
after a two week wait
adamw: "i assume the stats made the assumption that the proventester
feedback on any update would still have been present but treated it no
different from regular feedback. so, that's not necessarily a safe
assumption: proventesters may feel a stronger 'responsibility to test',
and if you cancel the process, they might stop doing so. but that's hard
to gauge."
red_alert: proposes a meeting during FUDCon NA to try and come up with a
better pt process
That's about all the concrete thoughts / suggestions I can filter out of
the log.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net