#6383: Build a 'test' compose automatically when installer-related packages
change
-----------------------+-----------------------
Reporter: adamwill | Owner: rel-eng@…
Type: task | Status: new
Milestone: | Component: other
Resolution: | Keywords:
Blocked By: | Blocking:
-----------------------+-----------------------
Comment (by adamwill):
Replying to [comment:7 dcantrel]:
There are two parts to what we've been building. The first is
what
you're saying... the per-package testing. More like the unit testing.
But there's also the integrated testing where we take a package that
passes unit testing and then do a compose from rawhide (or whatever
product we're trying to test for) using that newly built package. The
intent here is to gate builds we do in to koji based on the outcome of
these tests. That is, not wanting to submit a build that would be render
the next real tree compose by rel-eng unusable.
Aha, thanks. I wasn't aware you were aiming to use it in both these ways.
The overlap I'm thinking about is with regard to the integrated
distribution testing. The questions we need to answer are where can we
get information on what builds of other packages are good to use to bring
in to a test compose and similar.
OK, so this is where I was kind of sidling around the problem: see below.
> This would be another level again which kind of comes between
the two
- I wasn't envisaging doing this level of testing on every anaconda
commit, or even every anaconda tag or package build, it seems like too
heavy a process for that. I was envisioning doing it at the level of
updates. (This also avoids the problem of knowing when we need to pull in
an updated blivet or pyparted or whatever along with the updated anaconda,
because the update should include all necessary packages and if it doesn't
that's an error).
Are you thinking something like an updates.img per bug that can then be
tested
individually? If so, that won't allow for everything. updates.img
files are useful, but can't easily cover all bases. Plus, the eventual
work would need to go in a real build anyway so I would want to do test
composes using built packages rather than updates.img files.
Ah, sorry, no - we are suffering from terminology overloading here ;) I
mean "updates" in the sense of "Bodhi updates", not "anaconda
updates
images". *My* ultimate concern is always "Fedora" (not
"anaconda" or
"GNOME" or any other bit of Fedora). The problem I am trying to solve is
basically [
https://bodhi.fedoraproject.org/updates/FEDORA-2016-d85d9ad927
this] - where a completely broken anaconda package update gets pushed
stable. And more generally, the problem that we just do not have a good
way to test anaconda package updates either manually or automatically.
That's why I say I can "sidle around the problem" of deciding what
packages to include in the test image, because in my scenario, someone
else has already had to do that work in order to create the update. In my
scenario, we'd just have some kind of list of 'trigger packages', and if
*any* of those packages was in an update, we would build a 'test compose'
from stable plus *all* the packages in that update, and test that. If the
update doesn't actually include all the packages it should, it would be
*correct* for the tests to fail, because ultimately the "thing being
tested" in my scenario is the package update: the goal is to say "this
package update is good and if we push it stable the nightlies will keep
working" or "this package update is not good and we should not push it
stable".
Right now when an anaconda update shows up in Bodhi, our choices for
testing it are:
* updates.img (not good for all the reasons you gave)
* boot a live image and 'dnf update' all the relevant packages before
running liveinst (better, but doesn't test the non-live flow, which is a
significant problem)
* Build test images ourselves (I, frankly, haven't been able to build a
boot.iso *or* a live image since the Pungi 4 / livemedia-creator
changeover; even when you *do* know how to make this fly, every tester
doing it themselves is clearly silly and error-prone)
* Go beg releng to build us a 'candidate' compose, which is basically
this, only with lots of silly manual process overhead (we have to file a
ticket, Dennis has to shepherd a candidate compose)
That's the thing I want to improve.
One idea I like about the updates.img testing idea is it lowers the
barrier to entry to let outside contributors either test a fix or
contribute something new and test it.
Since we got off on this tangent, as it happens I actually wrote a thing
for this yesterday, since it seemed like a fun idea and it was easy. With
[
https://phab.qadevel.cloud.fedoraproject.org/D801 these]
[
https://phab.qadevel.cloud.fedoraproject.org/D802 two] diffs for the
Fedora openQA scheduling stuff, it's quite easy to trigger a full openQA
test run for any given compose *plus an arbitrary updates.img* - it
basically adds an arg to the scheduler CLI that lets you say 'stick
inst.updates=(URL) on the end of the kernel cmdline for every test'. You
have to trigger it manually, but that seems kind of appropriate for the
type of testing it's meant for, the type where we're chatting in IRC and
someone says "oh hey, can you try this updates.img?" If it's something
people are interested in using I can figure out a way to hand out
credentials for the staging openQA instance (I'm envisioning people using
it either there or on their own dev instances, never on the prod
instance).
--
Ticket URL: <
https://fedorahosted.org/rel-eng/ticket/6383#comment:8>
Fedora Release Engineering <
http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project