Defining a harness API
by Dan Callaghan
An important first step towards supporting alternative harnesses and/or
the mythical Beaker Simple Harness is coming up with a stable,
documented API for the harness to interact with Beaker. So I'm starting
this thread now to get the ball rolling.
My first thoughts:
* It should have the smallest possible surface area -- just enough to
expose all of Beaker's functionality and nothing more.
* It should be defined from the point of view of lab controller <-> test
system, not scheduler <-> test system. Corollary: we might need to
start treating the lab controller as a first-class citizen instead of
just a dumb proxy to the scheduler.
* Just because we use XML-RPC now doesn't mean it's the best choice,
and doesn't mean we need to keep using it.
* In particular, the HTTP protocol probably supports everything we need
to build the API (in other words, it can be a "RESTful API" even
though I hate that phrase). For example logs could be uploaded using
HTTP PUT (with Content-Range), which means if we wanted we could
potentially use Apache mod_dav_fs to efficiently write these directly
to the filesystem without any intervening Beaker code.
There are four areas the API needs to cover (that I can think of). The
harness needs to be able to:
* find out what to run
* report results and upload logs
* extend the watchdog time
* synchronize with other recipes in the recipe set
This last area is a new one, it's currently handled entirely in beah and
there is no API on the Beaker side for it. But the dynamic FQDNs in
Beaker 0.10 mean it is now needed, even if it's as simple as having
a call to wait for all other harnesses in the recipe set to check in.
Thoughts?
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Infrastructure Engineering and Development
Red Hat, Inc.
11 years, 1 month
A more transparent design process for Beaker
by Nick Coghlan
One thing I noticed when starting work on Beaker is that a lot of the
design rationale for major architectural decisions is hidden in email
archives (including Red Hat internal ones), Bugzilla entries and code
reviews on Gerrit.
To make it easier for new developers to understand past architectural
changes/additions, to help users to understand changes coming in future
releases, and to help us be confident that our proposed designs are
solid, I'm suggesting we adopt a "Beaker Design Proposal" process for
major changes (such as the scheduler updates I'm working on, the
upcoming integrated dashboard, the "system pools" concept, self-service
user groups and group jobs).
I am unashamedly stealing this concept from Python's PEP process,
because I like it and I think it works pretty well in ensuring that the
*rationale* for a design is captured in addition to forcing us to think
a bit more about the design itself before diving into code. As with
Python, only major changes would need to be written up as a design
proposal - bug fixes and smaller features would be handled through
Bugzilla as usual. In recent times, the integration with oVirt Engine in
0.10 is probably the only change big enough to qualify (although the
Supported Query idea might have been worth writing up as well).
In terms of mechanics, I'd largely follow the Python model for
numbering: reserve BDP 1 for the meta-proposal describing the process,
BDP 2-99 for any other meta-proposals we come up with, and start the
actual design proposals at BDP 100.
Publication would be via Sphinx on beaker-project.org: the design
proposals would just live in another docs directory alongside the
existing User and Admin guides.
BDP approval rights would rest with the Beaker Development Lead (i.e.
currently me) or their delegate (similar to the way PEP approval rights
ultimately rest with Guido van Rossum as Python's BDFL, but one of the
other senior core developers may take on the role of BDFL-delegate for a
particular PEP).
Regards,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
11 years, 1 month
Updating the home page blurb for Beaker
by Nick Coghlan
It has been pointed out [1] that the current home page blurb doesn't do
a great job of describing Beaker's purpose and capabilities. While I
think Ed's suggested wording is an improvement over the status quo, I
have a few additional proposed changes of my own. Rather than trying to
work out the details in patch comments, I figure it makes more sense to
get rough consensus on new wording here, then we can update the home
page and any affected project descriptions in the documentation accordingly.
Here's my current draft
================================
Beaker is a tool to provide and manage globally distributed testing
infrastructure shared by multiple development and quality assurance teams.
With Beaker, users and administrators are able to:
- Manage systems in a globally distributed network
- Maintain an automated inventory of system hardware details
- Provision execution environments on systems
- Schedule tasks to run on one or more systems
- Store and view task results
Beaker enables users to create any task they would like (from testing
changes to kernel memory management to installing their favourite OS and
game server) and run these tasks on any number of machines of any
specification located anywhere in the world, and provides a central
interface where the outcome of these tasks can be reviewed.
Beaker is currently based on:
- Anaconda kickstarts for bare metal system provisioning
- KVM/virsh for guest system provisioning
- oVirt for dynamic virtual system provisioning
- RPM for task definition and distribution
- the Beah framework for task execution
Developed by `Red Hat <http://www.redhat.com>`__, Beaker can be used as
a searchable machine inventory, system provisioning engine, task
scheduler, test automation framework and hardware lab management tool.
================================
Once we have a more accurate description of the status quo, then I
believe it becomes easier to create a description of "things we're
interested in doing differently" (such as supporting execution
frameworks other than Beah, task distribution with Git, image based
provising of guest and dynamic virt systems, etc).
Regards,
Nick.
[1] http://gerrit.beaker-project.org/#/c/1257
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (http://beaker-project.org/)
GlobalSync Development Lead (http://pulpdist.readthedocs.org)
11 years, 2 months
Scheduling recipe sets rather than recipes
by Nick Coghlan
The current scheduler works almost purely at the recipe level. The
extent to which it pays attention to recipesets pretty much amounts to
ensuring all recipes in a recipe set are scheduled on the same lab
controller.
This creates some interesting problems with multi-host testing:
- a recipe set with strict host requirements for only some systems may
hold on to common systems for a long time while waiting for rare ones
(the addition of dynamic virt support opens the door for a recipe set to
hold on to dynamic virt resources while waiting for physical hardware
for other recipes)
- recipe sets scheduled for unique systems may deadlock if a high
priority job is competing with a previously queued low priority job
which has already claimed some resources
To better explain the latter problem, consider a lab with only 2
systems, A and B, containing a particular piece of hardware, and a
multi-host recipe set that needs both of them. Queue a low priority
version of that job while a test is running on system A, and the job
will claim system B immediately for one recipe, while the other will
remain in the queue. If a high priority copy of the job is added before
the test running on system A completes, then system A will be claimed by
Job 2. This leaves the two jobs in a classic ABBA deadlock, as Job 1 has
System B and is waiting for System A, while Job 2 has System A and is
waiting for System B.
Some of the metrics support being added in 0.11 is actually about
measuring the overall impact of the first problem (by seeing what
proportion of their time systems spend in the Scheduled state).
For other reasons to do with being able to effectively partition the
scheduling task between multiple schedulers each handling the systems
managed by a particular lab controller, I've been considering proposing
the inclusion of a "Claimed" state in the recipe lifecycle. The
"Claimed" state would fit between "Queued" and "Scheduled", and indicate
that the recipe had been assigned to a specific lab controller, but not
yet assigned to a specific system (at the moment, this state change is
handled implicitly through setting "recipe.recipeset.lab_controller"
when the first recipe in the recipeset is scheduled).
Furthermore, the scheduler would be updated to work on a *cached* copy
of the System status data. This is needed to avoid the current problem
where there's a race condition with system status changes occurring
during a scheduling pass leading to recipes jumping the queue (I'm
interested in hearing about relatively clean ways to this with SQL
Alchemy, though:
http://stackoverflow.com/questions/13983067/cached-reads-immediate-writes...)
In combination, these two would allow Claimed recipes to be given
priority over Queued recipes on subsequent passes, preventing the
deadlock problem and theoretically also improving system utilitization.
One social challenge with addressing this is that we don't want to
enable/encourage queue jumping for rare systems by scheduling them in a
recipe set with a job that will be scheduled quickly, but I'm not sure
we can solve that at the technical level.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (http://beaker-project.org/)
GlobalSync Development Lead (http://pulpdist.readthedocs.org)
11 years, 2 months