The current scheduler works almost purely at the recipe level. The
extent to which it pays attention to recipesets pretty much amounts to
ensuring all recipes in a recipe set are scheduled on the same lab
controller.
This creates some interesting problems with multi-host testing:
- a recipe set with strict host requirements for only some systems may
hold on to common systems for a long time while waiting for rare ones
(the addition of dynamic virt support opens the door for a recipe set to
hold on to dynamic virt resources while waiting for physical hardware
for other recipes)
- recipe sets scheduled for unique systems may deadlock if a high
priority job is competing with a previously queued low priority job
which has already claimed some resources
To better explain the latter problem, consider a lab with only 2
systems, A and B, containing a particular piece of hardware, and a
multi-host recipe set that needs both of them. Queue a low priority
version of that job while a test is running on system A, and the job
will claim system B immediately for one recipe, while the other will
remain in the queue. If a high priority copy of the job is added before
the test running on system A completes, then system A will be claimed by
Job 2. This leaves the two jobs in a classic ABBA deadlock, as Job 1 has
System B and is waiting for System A, while Job 2 has System A and is
waiting for System B.
Some of the metrics support being added in 0.11 is actually about
measuring the overall impact of the first problem (by seeing what
proportion of their time systems spend in the Scheduled state).
For other reasons to do with being able to effectively partition the
scheduling task between multiple schedulers each handling the systems
managed by a particular lab controller, I've been considering proposing
the inclusion of a "Claimed" state in the recipe lifecycle. The
"Claimed" state would fit between "Queued" and "Scheduled",
and indicate
that the recipe had been assigned to a specific lab controller, but not
yet assigned to a specific system (at the moment, this state change is
handled implicitly through setting "recipe.recipeset.lab_controller"
when the first recipe in the recipeset is scheduled).
Furthermore, the scheduler would be updated to work on a *cached* copy
of the System status data. This is needed to avoid the current problem
where there's a race condition with system status changes occurring
during a scheduling pass leading to recipes jumping the queue (I'm
interested in hearing about relatively clean ways to this with SQL
Alchemy, though:
http://stackoverflow.com/questions/13983067/cached-reads-immediate-writes...)
In combination, these two would allow Claimed recipes to be given
priority over Queued recipes on subsequent passes, preventing the
deadlock problem and theoretically also improving system utilitization.
One social challenge with addressing this is that we don't want to
enable/encourage queue jumping for rare systems by scheduling them in a
recipe set with a job that will be scheduled quickly, but I'm not sure
we can solve that at the technical level.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (
http://beaker-project.org/)
GlobalSync Development Lead (
http://pulpdist.readthedocs.org)