The march towards Beaker 1.0
by Nick Coghlan
Beaker has been doing 0.x releases for a long time. We've been talking
about the release later this month being 1.0, but before we declare
we've reached that milestone, I'd really like us to be at the point
where someone could reasonably expect to grab the Beaker RPMs from
beaker-project.org, and using just our documentation, set up and start
running their own Beaker instance, including writing new tests using
either beah or autotest.
We're not there yet, and we're not going to get there this month, so the
next release will be 0.13 and we'll continue with regular 0.x releases
until we're happy we have something we consider worthy of the "Beaker
1.0" title. I don't think we're all that far away - we may need a 0.14,
but I'll be surprised if we see 0.15.
For planning purposes, I'd like to keep marking the backlog as "1.0" -
the 0.x releases will then just be snapshots that make useful chunks of
functionality available to users, rather than having to wait until we're
done with everything we want to include for the 1.0 milestone.
This will throw out the currently predicted releases in
http://beaker-project.org/dev/proposals/handling-large-installations.html and
the associated design proposals, but I'm not really worried about that.
In some respects, it works out better, as it means we can add the
"Effective Job Priorities" proposal to our targets for Beaker 1.0, and
that's the *real* objective of several of the changes we're currently
working on.
Once we're happy we know what we would like to include in 1.0, then I
can go back and adjust those design proposals and the tech roadmap
accordingly.
Based on the off-list discussions, here's what I would like to aim for
as our 1.0 milestone:
* The Effective Job Priorities design proposal (and its dependencies)
* Full Fedora compatibility (including the test suite)
* A stable alternate harness API (ideally with the corresponding
autotest patches merged on their side)
* A harness independent reservesys mechanism
* Moving any remaining hardcoded Red Hat specific settings into
configuration files
* Documentation improvements, including
* an architectural guide that at least explains all of Beaker's
external interfaces
* an explanation of the local watchdog hooks
* more general advice on job design (including effectively using
features like the job matrix)
* Self tests that exercise beah, the legacy harness API and the new
harness API
* Having a public page giving at least basic info on how Beaker is used
at Red Hat
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Test Automation Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
10 years, 9 months
The nose-progressive plugin
by Nick Coghlan
I was recently introduced to the "nose-progressive" plugin, which
changes the way nose displays results to a single updated line, with any
error reports printed immediately and a nice progress bar indicating how
far through the test run is (in terms of number of cases).
Really easy to setup and use with our tests:
sudo yum install python-nose-progressive
./run-tests.sh --with-progressive
And thanks to dcallagh for packaging it for Fedora :)
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Test Automation Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
10 years, 9 months
Nightly RPMs on beaker-project.org
by Dan Callaghan
I've just set up a cron job to build Beaker RHEL6 RPMs from the develop
branch in git each night. They are uploaded to beaker-project.org here:
http://beaker-project.org/nightlies/
This is also usable as a yum repository, so if you like to live on the
wild side you can install directly from there :-)
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Infrastructure Engineering and Development
Red Hat, Inc.
10 years, 10 months
Defining a more useful component list in Bugzilla
by Nick Coghlan
The current component list in Bugzilla has a few limitations, with some
bugs being thrown at a component more or less at random, since they
don't fit any of the existing components. In other cases, the component
is too broad, failing to make appropriate distinctions in the assignments.
Here's the current component list:
command line
Doc
inventory
qpid
lab controller
scheduler
reports
tests
test harness
web UI
Those actually aren't *that* bad, there are just some missing ones and a
couple could do with a name change.
Suggested name/scope changes:
qpid -> notifications (covering both AMQP sending and receiving and
email sending)
tests -> test suite
inventory -> inventory task
test harness -> beah test harness
command line -> bkr CLI
Suggested new components:
beaker-wizard CLI
other tasks (covers all standard tasks other than the inventory task)
harness API (for the new stable harness API)
server XML-RPC (for central issues that aren't related to the web
UI or scheduler)
multiple (problem/feature that spans multiple components)
unknown (for bugs that haven't been triaged yet)
Fedora compatibility
I'm *not* sure those proposals actually make sense, but I wanted to
provide a starting point for discussion.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Test Automation Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
10 years, 10 months
New Beaker API quirks
by Don Zickus
Hi Dan,
I have been enjoying the ease to code against the new Beaker API. The
GET, POST, PUSH stuff keeps things really simple. But during my testing I
noticed a few quirks that Bill P. suggested I email you about for your
opinion.
- When I POST/PUT to a url, if the url has an extra '/' in the path, like
/recipes/(recipe_id)/tasks/(task_id)//status
the command gets rejected. Not sure if that is an http thing or a server
thing. Not to much of a problem, just forces me to clean up my code
(which is good!). Just thought I would mention it.
- when I PUT a zero length file to the server, it fails with a 400 code.
Now I should be filtering those (and will now). But it did take me an
hour before I realized that of my list of files, the rejected ones were of
zero length. Again could be on purpose, not sure.
- Because the server does not have a mechanism to 'push' and abort the
client (other than power off), the client could still be running and
trying to pushing results back to the server.
a POST of a task_result gave me an 'Internal Server Error 500'. Will
that be the expected response? I can trap for that and abort the client.
PUT of logs is still allowed even though the task aborted on the server
side. Bill thought that should be fixed. Though I am not sure what the
expectations are.
That's it so far.
Thanks,
Don
10 years, 10 months
Handling exit status of the commands execution from all the nodes
by shwetha
Hi All ,
In a multi-node testing environment we have to fail a test case when
certain commands on any one of the node fails . For example : if
mounting fails on client node , that test case execution should not be
continued. A message to be sent to all other machines i.e
masternode/node/peer in this case to stop the execution.
How are we going to handle this situation. *This is the basic
requirement for multi-node testing.* We want to check the exit status
after every step. For example in most of our testcases the set-up includes :
1) peer probing
2) volume_create
3) volume_info
4) volume_start
5) volume_status
6) mount
In every step we should check the exit status from all the nodes and if
any one node has non-zero exit status we should stop that test case and
move on to next test case in that task.
-Shwetha
10 years, 10 months
Technical roadmap updated
by Nick Coghlan
I've just published an update [1] to the tech roadmap at
http://beaker-project.org/tech-roadmap.html
Three changes of note:
- I moved the "Live Dashboard" down into the exploration section. It
would still be nice to have it, but it turns out that Graphite's
rendering API plus the ReloadEvery extension for Firefox ([2]) isn't a
bad interim solution :)
- I moved the OpenStack integration up into the planned section. oVirt
Engine is great if you're wanting to run a long lived high availability
core service, but not so good if you're after throwaway VMs to run a
single recipe. By contrast, OpenStack's reputation is pretty much the
exact opposite (great for spinning up instances quickly, less great if
you then want to keep them running forever), so it will hopefully be a
better fit for our use case.
- I added Jenkins integration to the "exploration" section. While you
*can* use Beaker for continuous integration (as Dan's patchbot shows),
Jenkins is really the king of that space. I see a lot of potential value
in an approach where Jenkins buildbots are used for initial smoke tests
on core platforms, and then you can configure a Jenkins plugin to kick
off a more comprehensive set of integration tests in Beaker.
Cheers,
Nick.
[1]
http://git.beaker-project.org/cgit/beaker-project.org/commit/?id=43ad402a...
[2] https://addons.mozilla.org/en-US/firefox/addon/reloadevery/
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Test Automation Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
10 years, 10 months