Currently, there's a conflict between the mechanism we use to ensure job
status is tracked correctly (marking jobs as dirty until the entire set
can be updated appropriately), and the mechanism we use to try to avoid
deadlocks between recipe sets that are competing for the same systems
(by always favouring recipe sets that already have some systems assigned).
The problem is that by taking the dirty recipes out of the set of those
considered for scheduling, we recreate the opportunity for the recipe
set deadlock (see [1]). Our instance appears to have enough machines
available that we don't hit it too often, but it looks like it may cause
more problems for smaller instances.
Ray has a suggestion for how we might be able to better cope with that:
I wonder if we can just accept dirty jobs in the
schedule_queued_recipes() method.
As a dirty recipe it will be in the 'Queued' state, and the only state that would
change
to, by making it clean, would be 'Aborted' (or 'Cancelled' if the user
cancels the job).
So if, while it is dirty, the recipe is assigned a system and then later marked
'Aborted',
the recipe will be cleaned up and the system released as would normally happen with a
completed recipe.
After thinking it through a bit, I think we definitely need to do
*something* along these lines. I initially thought getting started on
some aspects of
http://beaker-project.org/dev/proposals/event-driven-scheduler.html
would be a different way to solve the problem, but it turns out that
needs the *exact* same ability to take queued-but-dirty recipes into
account when attempting to schedule recipes on newly available systems.
The one issue I see is that we really *don't* want to be making more
status changes to recipes that are part of a dirty job or we're back
into risking status update conflicts - what we really want is a way to
flag those systems as off limits for the remainder of this pass over the
queue, *without* attempting to schedule the dirty recipe immediately.
Cheers,
Nick.
[1]
https://bugzilla.redhat.com/show_bug.cgi?id=952587
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Test Automation Team Lead
Beaker Development Lead (
http://beaker-project.org/)
PulpDist Development Lead (
http://pulpdist.readthedocs.org)