On 01/09/2012 09:16 AM, Raymond Mancy wrote:
----- Original Message -----
> From: "Jun'ichi Nomura"<j-nomura(a)ce.jp.nec.com>
> To: beaker-devel(a)lists.fedorahosted.org
> Sent: Friday, January 6, 2012 3:07:19 PM
> Subject: [Beaker-devel] scheduling deadlock
>
> Hello,
>
> I've observed scheduling deadlock a few times in my Beaker.
> It's typical AB-BA deadlock and seems due to Beaker scheduling per
> Recipe,
> while it should, I think, do per RecipeSet.
> What do you think?
>
Hi Jun'ichi,
It seems like a fine idea to schedule by RecipeSet, and of course it's
the intent of beaker to schedule RecipeSets on a FIFO basis (excluding any priority
conditions).
I guess the most obvious way to go about this is to wait for every machine to become
available
before you reserve any of them. Of course this leads to the situation where larger
RecipeSets might
be waiting forever as smaller RecipeSets continually poach machines that larger ones
need.
I'd expect that you would need an intermediary layer of machines that are in a pre
reserved mode and are managed
such that RecipeSets are run in an expected order.
Hi Ray, no new layer is necessary, for example higher priority jobs
should consider machines in "Queued" recipes as Free. (Machines should
be stolen from recipes sorted by priority ASC, queued-time DESC to
prevent starvation of jobs which are in queue for a while already.)
Another problem is the deadlock is not detectable: at least there is no
easy way I would know of to see which machines fit the requirements and
why recipe blocks, but there is this Bug
https://bugzilla.redhat.com/show_bug.cgi?id=607531 (with another two
duplicates - looks like sought after RFE.)
-- Marian
> Reproducible steps are as follows.
>
> Suppose I have 3 jobs below, each contains a recipeset with 2
> recipes:
>
> job1: RS:1
> R:11 requires hostA
> R:12 requires hostB
>
> job2: RS:2
> R:21 requires hostB
> R:22 requires hostC
>
> job3: RS:3 (with high priority)
> R:31 requires hostB
> R:32 requires hostC
>
> and I submit them with a certain interval inbetween:
>
> submit job1
> => R:11 and R:12 take hostA and hostB, respectively.
>
> submit job2
> => R:22 takes hostC.
> R:21 is queued, waiting for hostB.
>
> submit job3
> => R:31 and R:32 are queued.
>
> When job1 completes and hostB is freed,
> R:31 takes hostB, since it has higher priority than R:21.
>
> And now we get AB-BA deadlock between RS:2 and RS:3 ...
>
What are the actual priorities that you are setting?
The scheduler should bump the priority of any recipe that only matches
one system. So if you submit job2 as High and job3 as Urgent priority, I would
expect you to not have the problem as job2 will be bumped to Urgent as well and then
get hostB before job3.
However if the priorities are Normal and High, then the priority bumping is moot
as they both get a bump.
Interestingly it's not a problem that we've really experienced (that I've
heard of anyway).
> --
> Jun'ichi Nomura, NEC Corporation
> _______________________________________________
> Beaker-devel mailing list
> Beaker-devel(a)lists.fedorahosted.org
>
https://fedorahosted.org/mailman/listinfo/beaker-devel
>
_______________________________________________
Beaker-devel mailing list
Beaker-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/beaker-devel