On Wed, 2009-12-02 at 15:47 -0500, Peter Jones wrote:
On 12/02/2009 03:16 PM, James Laska wrote:
> If I'm summarizing the pain correctly ... the push back seems to come
> from providing testing before development is ready to accept bug
> Does this only happen during early development?
No. In fact, a part of this *is* due the difficulty in testing loader
changes. Pretty often the (current) workflow goes like:
1) joe writes a loader change that's complex
2) joe compile tests it, tries to unit test code snippets isolated into
their own tiny (transient) test harnesses where possible (which it
often is not).
3) joe fixes whatever problems are found
4) joe submits the change for review on a-d-l
5) a-d-l review finds some problems, misses others
6) if a-d-l review finds problems, joe goes back to step 2
7) else, joe commits the change
8) 0 or more days later a build gets done by joe or somebody else
9) rawhide composes with the new build
10) rawhide is broken because of one character in a string someplace
11) bob comes in in the morning and does a test install with rawhide
12) bob fixes the error, goes through steps 1-7 again (or maybe
just commits a trivial change, but generally not these days)
13) pile-o-bug-reports arrives, almost all of them after the bug is fixed
14) mike spends 8 hours triaging bug reports that are completely
Note how there's no set schedule on step 8.
This happens at all times during development. We can alleviate some of
the problem by writing/improving tools, for example to build/update
anaconda's initrd.img, but I don't honestly believe that will cut down
more than, say, half of this problem. It also doesn't really help
with of the /combined/ tree (that is, with changes other developers are
working on), with arches that aren't what we're using on the desktop,
or with e.g. CD media. For those, there's not much you can do without
just building it in koji and punging up a tree. And we don't really
feel it's worth pushing to testers, no matter what quality of tester
or test methodology, until we've done some simple trials first.
The new way, including some feedback from this thread, is more like this:
1) steps 1-8 from above happen
9) somebody on the anaconda team composes a tree
10) we (the anaconda team) smoke test it and go through steps 11 and 12
13) one of us asks for a tree to be composed for testing
14) we (I guess rel-eng does this bit) put some images somewhere for
testing [details needed here]
15) if they're not catastrophic we replace the LNG images with those
16) we (the anaconda team) go back to step 1.
This is great, thanks for taking the time to outline the detailed steps.
The above procedure sounds like a solid plan. In addition to the above
process, during the initial F-13 rollout of NFR, might I request a few
First, that we continue having rawhide install images available at all
times. However, they aren't images that are pungi'd every night ...
they are simply be the images built and tested from the last time
through step#15 above. So for now in the F-13 cycle, they would be the
LNG (last known good) of F-12. This seems to have the best of both
worlds by allowing us to work the kinks out of this new process, while
keeping up expectations/messaging around rawhide image availability.
Second, that we choose a few dates where we will attempt to walk through
the 16 steps prior to the freeze. My worry with not planning ahead
would be we wait until our pre-alpha verification step ... and it's a
: I'm most likely going to automate this so we automatically
x86_64 and i386 trees composed on cutlet (our local mirror) any
time it sees a new anaconda package in rawhide.
Let's talk @ FUDCon. Will Woods and David Cantrell have some
interesting ideas on how AutoQA might be used to respond to certain
events and provide test feedback for you. Presently, it will detect new
rawhide install images and run through a basic install. The image
detection mechanism and the desired tests can be adjusted as needed.
> Meaning, after Alpha/Beta is having nightly built rawhide
> images still a problem? When in the schedule  would be an
> acceptable time to start providing test feedback?
I think maybe after beta could work if we feel it's really necessary;
it's not wholly unreasonable to expect us (the anaconda team) to be a
bit more rigorous in composing our own trees and testing before (and
after) step 7. But you've got to realize that that's basically just
reverting to the original process above, which still has step 8 with no
set schedule. So it's not like that's really different - you're just
seeing the same installer over and over.
> Another point, I think we can't limit the discussion to just
> anaconda-devel. There are a *lot* of other critical components pulled
> into the install images that are developed outside that group.
We didn't ever limit this discussion to anaconda-devel. Actually,
I don't think we've discussed this there at all.
Sorry, didn't meant to imply that discussion was only taking place on
anaconda-devel-list. Just meant there may be other development teams
who rely on the rawhide install images for test feedback.