----- Original Message -----
From: "Jun'ichi Nomura" <j-nomura(a)ce.jp.nec.com>
Sent: Friday, January 6, 2012 3:07:19 PM
Subject: [Beaker-devel] scheduling deadlock
I've observed scheduling deadlock a few times in my Beaker.
It's typical AB-BA deadlock and seems due to Beaker scheduling per
while it should, I think, do per RecipeSet.
What do you think?
It seems like a fine idea to schedule by RecipeSet, and of course it's
the intent of beaker to schedule RecipeSets on a FIFO basis (excluding any priority
I guess the most obvious way to go about this is to wait for every machine to become
before you reserve any of them. Of course this leads to the situation where larger
be waiting forever as smaller RecipeSets continually poach machines that larger ones
I'd expect that you would need an intermediary layer of machines that are in a pre
reserved mode and are managed
such that RecipeSets are run in an expected order.
Reproducible steps are as follows.
Suppose I have 3 jobs below, each contains a recipeset with 2
R:11 requires hostA
R:12 requires hostB
R:21 requires hostB
R:22 requires hostC
job3: RS:3 (with high priority)
R:31 requires hostB
R:32 requires hostC
and I submit them with a certain interval inbetween:
=> R:11 and R:12 take hostA and hostB, respectively.
=> R:22 takes hostC.
R:21 is queued, waiting for hostB.
=> R:31 and R:32 are queued.
When job1 completes and hostB is freed,
R:31 takes hostB, since it has higher priority than R:21.
And now we get AB-BA deadlock between RS:2 and RS:3 ...
What are the actual priorities that you are setting?
The scheduler should bump the priority of any recipe that only matches
one system. So if you submit job2 as High and job3 as Urgent priority, I would
expect you to not have the problem as job2 will be bumped to Urgent as well and then
get hostB before job3.
However if the priorities are Normal and High, then the priority bumping is moot
as they both get a bump.
Interestingly it's not a problem that we've really experienced (that I've
heard of anyway).
Jun'ichi Nomura, NEC Corporation
Beaker-devel mailing list