----- Original Message -----
From: "Nick Coghlan" <ncoghlan(a)redhat.com>
Sent: Wednesday, January 2, 2013 12:08:40 PM
Subject: Re: [Beaker-devel] Scheduling recipe sets rather than recipes
On 12/21/2012 09:53 PM, Jaroslav Kortus wrote:
>> Furthermore, the scheduler would be updated to work on a *cached*
>> of the System
>> status data. This is needed to avoid the current problem where
>> a race condition
>> with system status changes occurring during a scheduling pass
>> to recipes jumping
>> the queue (I'm interested in hearing about relatively clean ways
>> this with SQL Alchemy,
> Do you mean https://bugzilla.redhat.com/show_bug.cgi?id=872187
> I've worked around it using the transaction for the whole
> loop (not just per-recipe as it was before). This is sort of
> data, as the transactions reads are consistent no matter what got
> written during the transaction. I'm ordering the recipes by recipe
> priority, recipeset.id and recipe.id (in this order) and then they
> I haven't seen any deadlocks since then (with tasks having the same
Yeah, but lumping everything into one giant transaction has its own
problems (mainly to do with state consistency with external systems
RHEV-M and the filesystem).
What I realised over the Christmas break is that many of these
can be resolved by moving towards a more event based scheduling
with two key scheduling events:
1. When a new recipe is submitted, attempt to assign it to a dynamic
virtual system or to a system from the idle pool
2. When a system completes its current task, attempt to assign it a
recipe from the queue *before* placing it back in the idle pool (in
case of dynamic virt, see if any of the recipes that previously
dynamic virt allocation can now be allocated a dynamic VM)
The current scheduling loop would then become a cleanup loop (e.g.
looking for dead recipes that need to be aborted for various reasons)
With separate scheduling events, different prioritisation rules can
applied to the two kinds of scheduling:
New recipes would use the current recipe based scheduling: filter and
order the available systems according to the preferences of the user
submitting the job and the requirements expressed in the recipe.
Free systems would use system based scheduling: order the queued
according to the preferences of the system owner and the priorities
the queued recipes.
I'd be curious to see what kind of preferences we're talking about.
When it comes to resources used, this would make more sense
on a shared system, where the recipe is running in the background
on a system that is performing other concurrent work. In such a case the system owner
describing their preferences such as memory, disk space and CPU usage makes sense. In
though nothing else is running on a system, so resource usage is a non issue to the
It does make sense to have the recipe run time as a preference (perhaps owners may want to
control of their machine and do not want to accept recipes that run for a week),
but the problem is that we allow recipe owners to dynamically change the run time of a
I guess group/user based preferences make the most sense. The owner may give favour to
over others, which is perfectly legitimate.
The latter scheduling algorithm can deal with the deadlock problem
prioritising recipes that are part of a recipe set that already has
resources allocated on the relevant lab controller over those which
just part of the general queue.
The one downside is that users would be able to exploit this is order
jump the queue for access to rare resources by also scheduling a
in the same recipe set that can run on readily available hardware.
we likely can't prevent such abuse, we should be able to provide
to help detect it and leave it up to organizational "acceptable use"
policies to deal with it.
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (http://beaker-project.org/
GlobalSync Development Lead (http://pulpdist.readthedocs.org
Beaker-devel mailing list