On Fri, 2016-11-25 at 09:27 -0500, Bastien Nocera wrote:
> The problem is that we didn't get around to running the test until the
> day before the go/no-go. There's a lot of stuff to test, and anything
> which only one person is likely to test is a risk. Frankly speaking,
> given how humans work, things that involve digging some piece of
> hardware you never touch out of a pile and hooking it up to a keyboard
> and mouse and a monitor and power and network is quite likely to get
> passed over in favour of something you can run in a VM. Especially if
> it's 4:30. This is why I have an Unused Arm Devices Pile Of Shame on my
> So, partly this is our fault because we could've tested this earlier and
> didn't. But it's also the case that we really need more redundancy in as
> much of the required testing as possible.
Is there any continuous testing done on the images on the installer? Is it
on real hardware? Is it possible to mock hardware setups? Comparing
boot setups on working and non-working installations.
I don't know what you mean by "the images on the installer", and I
don't know precisely what you mean by "mock hardware setups". Mock what
about them, in what way, exactly?
To take the specific example we're starting from here, if you mean 'can
we fake up something like an OS X dual boot install on a virtual
machine', the answer is kinda yes, but it's not entirely
straightforward. I actually did this in order to test my fix for the
bug, but as well as faking up a disk layout that approximated an OS X
install, I had to patch anaconda to think the system was a Mac (I just
sabotaged that check to always return True) and provide that patch as
an anaconda updates.img. Otherwise anaconda would've known the system
wasn't a Mac and wouldn't have hit the right code path.
I think it would be possible to do testing that didn't rely quite
on manual testing, through regression testing on "mock" hardware (a hacked
up VM with a test disk image), comparing the partition types after installation
against a working setup, comparing the file lists in the boot partition,
We do rather a lot of automated testing of the installer, in fact:
is a current Rawhide test run (missing some tests as some images are missing)
is the Fedora 25 Final automated test run (with all tests)
Those tests are run on every 'nightly' Branched and Rawhide compose,
and on all 'candidate' Branched composes. (We also run the Atomic
ostree installer test on the two-week Atomic nightlies).
Reports from these tests are mailed to this list every day, under the
title "Compose check report". I frequently reply to them with details
about the bugs the tests found.
We could, of course, do a lot *more* of this testing. Just personally
I've got a list of about 30 things I want to add to the test set. But
there's only two and a half people working on the openQA tests, and we
have other stuff to do as well. And of course we have to monitor the
tests that *are* run, investigate the bugs they discover, file them,
and follow up on them.
I should probably note some RH inside baseball at this point: there's a
general theme in RH at present that people would like to have less
divergence between Fedora and RHEL testing processes, specifically it
would be nice to have some of the testing that RH's release test team
does on RHEL builds done on Fedora builds too. Most of those tests run
on Beaker; Fedora technically has a Beaker instance but it's not
sufficiently set up for us to be able to actually run the tests RH has
on it yet. That whole 'oh just get Fedora a Beaker setup, shouldn't be
hard right?!' problem is dumped on tflink's lap at present. Like
everyone else, he has a million other things to do and that one isn't
his highest priority, and he also keeps getting roadblocked on it by
things that are out of his control, AIUI.
Once we have a usable Beaker instance we can try importing some of RH's
tests to it and setting them up to run on Fedora, though I would be
*utterly* unsurprised if that turns out to be a lot more work than it
We currently run all openQA tests on virtual machines. openQA *does* in
fact have the necessary bits to run tests on bare metal - SUSE does
this - but we don't do that and haven't investigated the possibility in
detail yet. Fedora's openQA instances are in PHX, so it'd at least
involve putting new hardware in there, which is its own process and
apparently we don't have unlimited physical *space* there for them.
It's really something we just haven't looked at at all yet.
Beaker is rather more focused on metal testing - it's kinda as much a
'allocate resources on machines with specific properties' system as it
is a testing system, really - but again, we're not really in a position
to be doing stuff on Beaker yet.
I'm surprised that the Anaconda, and blivet developers aren't
of this conversation. I'd certainly like them to point out all the ways in
which they're already doing what I mentioned, and showing how we could
add more test cases.
anaconda has its own automated test system and runner, called
the basic concept of kickstart-tests is that a 'test' is a kickstart
which runs an install and then, from %post, decides whether whatever it
was testing worked and writes out either a 'SUCCESS' message or some
kind of details about what went wrong to a canary file. The test runner
spins up a VM, runs an install using the kickstart, sucks the canary
file out of the VM and records the result.
As I understand it, these tests are not fully integrated CI-style into
the anaconda development process yet (though that was/is the goal), but
they are run quite frequently.
I wrote a tool about six months back which actually takes the
kickstart-tests repository and processes the kickstart tests into
openQA tests. It was kind of a proof of concept and took some
shortcuts, but it worked quite well. (I'd link to the results, except
my personal openQA box - which I ran the tests on - dropped dead last
month and I haven't got around to fixing it yet).
We had some discussions with anaconda team about ways in which that
could possibly be officially adopted, but it kinda petered out when I
ran out of time, and the script would probably need a bit of work now
(though likely not much). One constraint with openQA is we have
somewhat limited resources for running the tests - we can run 14 tests
in parallel on the production openQA instance, and 8 on the staging
instance - so we can't just add new tests entirely willy-nilly, we have
to be somewhat strategic, and this was one reason I didn't just throw
all the kickstart-tests into the existing openQA instances.
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net