----- Original Message -----
From: "Nick Coghlan" <ncoghlan(a)redhat.com>
Sent: Tuesday, June 11, 2013 11:54:45 AM
Subject: Re: [Beaker-devel] Better handling of pending status updates when scheduling
On 06/11/2013 11:23 AM, Raymond Mancy wrote:
> ----- Original Message -----
>> From: "Nick Coghlan" <ncoghlan(a)redhat.com>
>> To: beaker-devel(a)lists.fedorahosted.org
>> Sent: Tuesday, June 11, 2013 11:16:47 AM
>> Subject: Re: [Beaker-devel] Better handling of pending status updates when
>> scheduling recipes
>> On 06/11/2013 10:06 AM, Raymond Mancy wrote:
>>> ----- Original Message -----
>>>> From: "Nick Coghlan" <ncoghlan(a)redhat.com>
>>>> what we really want is a way to
>>>> flag those systems as off limits for the remainder of this pass over
>>>> queue, *without* attempting to schedule the dirty recipe immediately.
>>> That was attempted with this patch.
>>> In testing I could not get it to run at a reasonable speed. I could take
>>> at it again with refreshed eyes I guess.
>> You could also use a client side system ID cache in the scheduling loop
>> to filter out the off limits systems rather than trying to do it through
>> the database.
> Please Explain :-)
Rather than hitting up the database to figure out which systems are off
limits and try to get *exactly one* suitable system back for each
recipe, maintain a Python set in the scheduling loop that holds the
system IDs for "off limits" systems.
That's not what the above patch does. It tried to determine if there is
exactly one system that could cause the deadlock, and remove it, leaving
all other suitable systems available.
When we get the list of candidate systems back from the DB for a
1. If the recipe is dirty, add the first candidate system that isn't
already in the "off limits" set to the "off limits" set.
We would need to keep counters on the 'off limits' systems, as they
could be 'off limits'.
2. If the recipe is clean, schedule it on the first candidate system
that isn't already in the "off limits" set
If the recipe is clean we would also need to check if it had previously
put any systems, in the 'off limits' set. If it did, it would reduce the
counter by 1, if the counter was zero it could be removed from the 'off limits'
3. If there are no candidate systems that aren't in the "off
set, simply continue on to the next recipe
This way, on each pass through the loop, dirty recipes get a tentative
reservation (by placing the system ID in the scheduling loop's "off
limits" set), while clean recipes get assigned to a system.
The ABBA deadlock is prevented by the fact that when the second recipe
is considered, the contested machine will already be in the "off limits"
set and thus skipped over client side.
I think it could work, I just bet the devil will be in the details.
Red Hat Infrastructure Engineering & Development, Brisbane
Test Automation Team Lead
Beaker Development Lead (http://beaker-project.org/
PulpDist Development Lead (http://pulpdist.readthedocs.org
Beaker-devel mailing list