----- Original Message -----
From: "Jun'ichi Nomura" <j-nomura(a)ce.jp.nec.com>
To: beaker-devel(a)lists.fedorahosted.org
Sent: Friday, January 6, 2012 3:07:19 PM
Subject: [Beaker-devel] scheduling deadlock
Hello,
I've observed scheduling deadlock a few times in my Beaker.
It's typical AB-BA deadlock and seems due to Beaker scheduling per
Recipe,
while it should, I think, do per RecipeSet.
What do you think?
Hi Jun'ichi,
It seems like a fine idea to schedule by RecipeSet, and of course it's
the intent of beaker to schedule RecipeSets on a FIFO basis (excluding any priority
conditions).
I guess the most obvious way to go about this is to wait for every machine to become
available
before you reserve any of them. Of course this leads to the situation where larger
RecipeSets might
be waiting forever as smaller RecipeSets continually poach machines that larger ones
need.
I'd expect that you would need an intermediary layer of machines that are in a pre
reserved mode and are managed
such that RecipeSets are run in an expected order.
Reproducible steps are as follows.
Suppose I have 3 jobs below, each contains a recipeset with 2
recipes:
job1: RS:1
R:11 requires hostA
R:12 requires hostB
job2: RS:2
R:21 requires hostB
R:22 requires hostC
job3: RS:3 (with high priority)
R:31 requires hostB
R:32 requires hostC
and I submit them with a certain interval inbetween:
submit job1
=> R:11 and R:12 take hostA and hostB, respectively.
submit job2
=> R:22 takes hostC.
R:21 is queued, waiting for hostB.
submit job3
=> R:31 and R:32 are queued.
When job1 completes and hostB is freed,
R:31 takes hostB, since it has higher priority than R:21.
And now we get AB-BA deadlock between RS:2 and RS:3 ...
What are the actual priorities that you are setting?
The scheduler should bump the priority of any recipe that only matches
one system. So if you submit job2 as High and job3 as Urgent priority, I would
expect you to not have the problem as job2 will be bumped to Urgent as well and then
get hostB before job3.
However if the priorities are Normal and High, then the priority bumping is moot
as they both get a bump.
Interestingly it's not a problem that we've really experienced (that I've
heard of anyway).
--
Jun'ichi Nomura, NEC Corporation
_______________________________________________
Beaker-devel mailing list
Beaker-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/beaker-devel