Hi folks! I thought this might be of interest to the Cockpit community,
so I thought I'd write it up here in case anyone missed it elsewhere.
I work on the Fedora QA team, and we have been using the openQA
automated test system (developed by our friends at SUSE) to run various
functional tests on Fedora composes for the last couple of years.
As Cockpit is considered a critical part of Fedora Server, we run a few
tests that exercise Cockpit. They're nowhere near as extensive as
Cockpit's own test suite, but they run in a somewhat different context:
they run on the Fedora packaged Cockpit, against the rest of a Fedora
environment. They also test in a somewhat different way from Cockpit's
own test suite: openQA tests a lot like a human, by booting up a
virtual machine and then effectively "looking at the screen" and
sending mouse and keyboard input events (via VNC). The test boots a
full Fedora system, runs Firefox, and interacts with Cockpit like a
human being would. For both of these reasons, it can sometimes catch
problems Cockpit's own tests would not (it also sometimes goes wonky in
ways that Selenium-based tests don't, but never mind, that's my job to
deal with :>)
Until recently we ran these tests only on Fedora's nightly development
release distribution composes. Recently, though, we deployed some
enhancements to our openQA setup that let us run tests on Fedora
distribution updates as well, and have the results made visible through
the Fedora update system (Bodhi). I have just made it so that, from now
on, these tests will run on any update that contains the 'cockpit'
package. They also run on any critical path update.
This means that for any Fedora update containing cockpit or any
critical path package, Fedora's openQA cockpit tests should run, and
you should see the results in the Fedora update system (Bodhi). You can
see the results in Bodhi by clicking the Automated Updates tab for any
update. For instance, here's a recent Firefox update for Fedora 25:
If you look at the Automated Tests tab, you can see passes for:
indicating that this Firefox update didn't cause any problems for
Cockpit. Clicking on any test result will take you to the openQA page
for the test, where you can diagnose failures and so on (explaining how
to do this is a bit beyond the scope of this mail, please do ask me if
I hope this stuff will help us avoid shipping updates that break
Cockpit (and other key components). If you have any questions,
concerns, comments, or suggestions, please do ask!
To anticipate one question: you can cause *all* the tests for an update
to be re-run by editing the update in any way (you don't have to change
the package loadout, just changing a single character in the
description or something will do). If you think just one test result is
bogus and want it re-run, currently, you'll have to ask someone with
the necessary power - either me or Jan Sedlak (garretraziel on IRC).
I'm in North America and he's in Europe, so we should have most
timezones covered between us. We're hoping to set up a better mechanism
for this in future.
Note, if you're interested in the results for the nightly Fedora
distribution composes, an email summary of the results for those is
sent each time they're run to the Fedora test@ and devel@ lists, look
for mails with "compose check report" in the subject. Any time any of
the cockpit tests fails, the failure will be listed in the mail (passed
tests are not specifically listed, just a count of them). I usually
keep an eye on those results and analyze failures and file bugs,
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net