On Mon, 2010-11-29 at 10:08 -0800, Adam Williamson wrote:
On Mon, 2010-11-29 at 12:40 -0500, James Laska wrote:
> > * Just drop all the requirements/go back to before we had any updates
> > criteria.
>
> Hmm, certainly an idea. I feel like this is definitely a step backward,
> not forward. Has the initial motivation for an updates policy gone away
> or changed? Have we encountered problems that didn't yet exist, or
> weren't as painful, when the policy was first enabled? Are there other
> problems we need to focus on resolving (I suspect this is the case)?
As I see it, the thing that everyone agrees is problematic is critpath
updates for old releases not getting pushed or taking a very long time
to push. It's also generally agreed that the quality of critpath testing
can be improved by taking some steps we're already looking at
(package-specific test cases).
Things that some people see as problematic are:
1 Having to wait a week to push an update if you can't find testing
2 Testing being required for packages with automated test suites
3 The delay to security updates which is introduced by the testing
requirements
(Note, I've numbered your bullets above to respond to them individually)
#1 - if we don't have testers familiar with the packages, and it's a
popular package (either in terms of # of installed systems, interest or
critpath), we (includes maintainers) should be looking to create/develop
testing interest, right? Or, if no engaged testers were found ... do we
1. Wait X days for exploratory test results
2. Or push immediately
#2 - I'd take this case-by-case ... it depends on what is "automated".
Certainly, I could see a scenario where %check includes the upstream
unittest, and the automated package update acceptance plan passes ...
the update is in good shape and ready for additional functional tests
('updates-testing') or stable.
TODO - draft an autoqa test that confirms whether %check ran during
build-time, and ensures it passed.
#3 - Seems similar to #1 to me. Testing is *always* a delay. In a way,
that's by design. That's not to say we as testers intentionally want to
slow a process down to a crawl. So if there are unnecessary delays
unrelated to the act of testing ... let's work those. It sounds like
the delay here is similar to #1, in that testers aren't providing
feedback on security updates?
TODO - draft general security update test procedure
> > * allow packages with a %check section to go direct to
stable.
>
> Interesting, I like the spirit of this idea, but would like to see if we
> can incorporate this with autoqa karma plans. So, perhaps packages with
> %check get automated karma? Just the same as with packages that pass
> automated tests ... they'll eventually get positive karma of some form.
Yes, I was going to suggest the same thing. I'd suggest packages with a
%check section should get +1 proventester karma.
Note though that the %check needs to also pass. I've seen plenty of
builds that include a %check, but don't fail if the %check fails. There
is an AutoQA test in the making here, anyone interested in helping?
Of course, that relies
on the automated test suite actually testing the things proventester
testing is meant to cover;
http://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan
do we want to audit the test suites in question?
No, but I think we would want to at least ensure they pass. I don't
think it would be horrible to pull together an AutoQA test that checks
for the presence of a %check and whether it passed.
> > * setup a remote test env that people could use to test
things.
>
> I could use more details on this point. Is this talking about setting
> up QA systems hosted in Fedora infrastructure that any tester could
> login and use to test updates?
Yes - it's an idea to make it easier to test older releases, or packages
you don't want to / can't install (or configure) on your active systems.
I don't object to this. We have experience using shared test systems
internally at Red Hat for many years. There are a lot of system state
issues we'd need to work through, but there are certainly benefits to
having public test systems.
I'm inclined to think the more immediate problem though is the lack of
test instructions, not the lack of hardware. Additionally, over time
the lack of hardware will be less of an issue right (e.g. virt)? But
again, no major objections to shared test hardware, other than it does
involve some setup/maint, and doesn't address the lack of test
instructions.
> > * reduced karma requirement on other releases when one has
gone stable
> > * aggregated karma across the releases for the same package version.
>
> I don't have data to indicate how many updates have been released, and
> then reverted/obsoleted on only a subset of releases.
Yeah, I'd really like to see some data here. I did ask Luke on the
-devel thread, but so far no response. Everyone more or less agrees on
the factors here (on the one hand it's hard to get testing for old
releases, on the other hand it *is* possible for the 'same' update to
work fine on one distro but not another), it's hard to balance these
without hard numbers. I think you can possibly draw a distinction
between different types of updates here too: as I wrote on the -devel
list, I can see the argument for a single leaf package update, but
pushing an update to, say, an entire desktop environment and relying on
testing from another release seems scary.
Ah, good point.
> > * allow anon karma to count.
>
> Or maybe it counts, but counts less (.5 karma or something).
Something else to consider here is to make more people login; I suspect
relatively few people are actually doing testing who don't have a FAS
account, but I think we could make the login link more prominent, and
try harder to get people to log in (have a big scare-step when posting
anonymous feedback which says 'your feedback will not count unless you
log in!' and requires you to re-confirm to submit the feedback
anonymously; a nag screen, basically.
Good thinking. Is this something we can get on the bodhi roadmap
(
https://fedorahosted.org/bodhi/roadmap)? I think Luke monitors this
list, perhaps he'll jump in.
> > Security updates:
> >
> > * allow security updates to go direct to stable
>
> Risky
Right. As was pointed out on -devel, the update which caused us to start
thinking about an update testing process in the first place - the
infamous udev update - was a security update.
Yeah.
> > * ask QA to commit to testing security updates
>
> We can't commit to testing without guidance or instructions. Let's
> commit to documenting repeatable procedures that testers can follow and
> expand upon.
>
> For some security updates, they may have already been functionally
> tested upstream. I think it's reasonable to provide proxy karma linking
> to upstream functional tests. Though, I don't think upstream functional
> tests alone can bless a security update.
That seems like it'd be tricky to automate.
Eeew, it certainly would be! I was more just adding some extra flavor
on the karma "by proxy" practice.
> > Non critpath/security:
> >
> > * reduce timeout for non critpath from 7 to 3 days.
> >
> > * change default autokarma to 2 or 1.
>
> No immediate thoughts on these points.
I suggested the default auto-push karma change, though really what
should change is the linking of auto-push and approval. Right now,
whatever you set as the threshold for auto-push is *also* the threshold
for approval, which is more of a hack/unintended consequence than
intentional design.
I see what you mean.
Thanks,
James