On Thu, 2011-08-25 at 14:35 -0600, Stephen John Smoogen wrote:
On Thu, Aug 25, 2011 at 14:25, Chris Lumens
<clumens(a)redhat.com> wrote:
>> kickstart is a very broad area; you can write extremely complex
>> kickstart files that do a lot of stuff. So broadly what we'd need to do
>> is define a subset of kickstart functionality that we expect to work,
>> and then possibly divide that up by release phase (so some stuff must
>> work by Beta, the rest by Final, for e.g.)
>>
>> anaconda devs following this list, do you have any existing expectations
>> as to what level of kickstart functionality ought to be in place for
>> releases, and when you think would be appropriate?
>>
>> So far it seems everyone more or less agrees that it should be possible
>> to do at least a basic unattended kickstart install by Beta.
>
> You're right, kickstart is incredibly broad. I don't think we could
> ever hope to come up with criteria to cover all of it. I guess the best
> we can do is define criteria in terms of something else we already have.
I would go for the classical kickstart test for Alpha:
Does it take a minimal kickstart and build a default system. The
minimal being the exact stuff that would be created if a person just
clicked through a release.
Oh, I like that. That's very good.
For Beta
Take these X broken kickstarts, does it bail at the appropriate places.
Take these Y working kickstarts, does it work.
Where Y is a set of items that can be tested on say a KVM or a
"default" desktop defined somewhere.
I'm not so keen on that; it's a bit specific. One of my fetishes with
the criteria is to keep them generic so they don't have to keep being
changed all the time; I wouldn't want a criterion to rely on some
specific kickstart file we keep lying around somewhere.
For Final
The above and some subset of obscure items that can be tested reliably
somewhere that development and QA can replicate.
again, seems a bit arbitrary; what we really need is the kickstart
functionality we actually believe has to work, not an arbitrary set of
interesting bugs we happen to be able to test.
a minimal answer to this question would be fairly valid, I guess. at
least in the short term. sometimes working up criteria in response to
real-world cases - i.e. someone finds a kickstart problem, files a bug,
proposes it as blocker, we agree it should be one and work up a
criterion in response to it - has proved a decent way of doing things in
the past.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net