----- "Jesse Keating" <jkeating(a)redhat.com> wrote:
On Wed, 2010-04-21 at 10:13 -0400, Kamil Paral wrote:
> Hello,
> this is a final call for comments to our Package Update
> Acceptance Test Plan [1]. This test plan is tightly related
> to Package Update Acceptance Criteria [2] approved by FESCo.
> It basically says how we, the QA team, will decide whether
> a package update should be accepted or rejected. It also
> tries to map the requirements of the policy into an actionable
> test plan.
>
> If you have any comments about the test plan, please post
> them now, so we can include them in the document.
>
>
I'm a bit concerned about the test environment part. We have already
determined in the work that WWoods is doing for depcheck, that we do
need to consider all the pending updates when testing a particular
pending update. It would make sense to me to follow the same logic
here.
The Test Environment section tries to say that we are mainly interested
in testing changes that would happen for users with fully updated
system, because that is the most common case and we can't simply test
all the possibilities. Good example is Package Sanity Test, where
we want to test upgrading from the most recent package in 'updates'
to the package coming to 'updates-testing'. We are not really
interested in upgrading from one package in 'updates-testing' to
another package in 'updates-testing', because that's not what's gonna
happen for most of our users.
Of course some tests have a little different scope. Repo sanity
(the depcheck) tests repository contents rather than packages
on a system. That means that the third bullet point not really
applies to it.
So it really depends on a particular test case, but I think generally
the idea of Test Environment should be valid as it is.
As far as Responsibilities and Permissions, as others have said we
could
collapse the QA and Releng into the proventesters group. Also we
should
add the bodhi ticket author, as the person who owns the package, the
person who did the build, and the person who submitted the update can
all be different people.
Thanks, added.
As far as Schedule we should make it clear at what phase the test
runs.
I think we want it to run post-update-submission, and must be
completed
before the request to push to either -testing or stable is
"accepted".
We talked with Luke a bit about this and how to handle it within
koji/bodhi and have a game plan.
Well I'm not really sure where's the best place to define this.
I'm not even sure the Schedule section should be part of the
test plan. Because things you talk about are also described
here:
https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_policy
(which is not approved yet, but this or something similar will
be in the future).
It seems to me like a duplication of places where the definition
is. Ideas?