#6383: Build a 'test' compose automatically when installer-related packages
change
-----------------------+-----------------------
Reporter: adamwill | Owner: rel-eng@…
Type: task | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
-----------------------+-----------------------
Comment (by dcantrel):
Replying to [comment:4 adamwill]:
yeah, there are definitely obvious overlaps. I'd say right now we
have
testing at two fairly different levels: you're testing down at the level
of individual anaconda commits, we (QA) are testing way up at the level of
entire nightly distribution composes. That's as important a difference as
the difference between anaconda's kickstart-based test mechanism and
openQA's completely different mechanism. With just those two levels, the
separation is pretty clean, for me: I see anaconda's testing as being
almost on the level of unit testing, it's really a part of the anaconda
development process. QA's current testing is very much distribution-level
validation, it's really not answering the question "is this anaconda
broken?" exactly (though sometimes it happens to do that), it's answering
the question "is this Fedora broken?"
There are two parts to what we've been building. The first is what you're
saying... the per-package testing. More like the unit testing. But
there's also the integrated testing where we take a package that passes
unit testing and then do a compose from rawhide (or whatever product we're
trying to test for) using that newly built package. The intent here is to
gate builds we do in to koji based on the outcome of these tests. That
is, not wanting to submit a build that would be render the next real tree
compose by rel-eng unusable.
The overlap I'm thinking about is with regard to the integrated
distribution testing. The questions we need to answer are where can we
get information on what builds of other packages are good to use to bring
in to a test compose and similar.
This would be another level again which kind of comes between the two
-
I wasn't envisaging doing this level of testing on every anaconda commit,
or even every anaconda tag or package build, it seems like too heavy a
process for that. I was envisioning doing it at the level of updates.
(This also avoids the problem of knowing when we need to pull in an
updated blivet or pyparted or whatever along with the updated anaconda,
because the update should include all necessary packages and if it doesn't
that's an error).
Are you thinking something like an updates.img per bug that can then be
tested individually? If so, that won't allow for everything. updates.img
files are useful, but can't easily cover all bases. Plus, the eventual
work would need to go in a real build anyway so I would want to do test
composes using built packages rather than updates.img files.
If the concern is to prevent package builds that contain broken code,
that's what the unit testing framework is for. We trigger a build and
test run from an upstream commit. If it fails, we revert the commit
before making a new release and building in koji. The intent there is
that koji only contains builds with code that passes the unit test suite
*and* the integrated testing suite.
I think there's value in having these different levels of tests
and also
in having different test mechanisms - there are things openQA-style
testing can catch that kickstart testing just can't, and there are things
that are way more unwieldy to test openQA-style than kickstart-style. (In
*theory* we could run all the kickstart tests in openQA, but it'd be very
much a squre peg / round hole kind of thing). But we should definitely
avoid duplication between them as far as possible. So yeah, ideally we
definitely shouldn't have different test processes each with their own
image building mechanism, we should have a single image building mechanism
which produces the images needed for each level of testing.
Just to be clear, I'm not trying to suggest that anyone stop what they are
doing. I definitely think all testing frameworks in discussion are useful
in different capacities. What I want to do is share information and ideas
as much as possible so we don't all struggle solving the same problems
over and over again.
One idea I like about the updates.img testing idea is it lowers the
barrier to entry to let outside contributors either test a fix or
contribute something new and test it.
--
Ticket URL: <
https://fedorahosted.org/rel-eng/ticket/6383#comment:7>
Fedora Release Engineering <
http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project