On Tue, Jun 21, 2011 at 11:21:13AM -0300, Devan Goodwin wrote:
On Tue, Jun 21, 2011 at 11:00 AM, James Bowes
<jbowes(a)redhat.com> wrote:
> On Tue, Jun 21, 2011 at 10:37:07AM -0300, James Bowes wrote:
>> Hey all:
>>
>> For the past few days I've been trying to track down the spec test
>> failure in " Entitlements should bypasses rules for "candlepin"
>> consumers"
>>
>> I've been able to reliably reproduce this failure, but only on a full
>> run of 'buildr spec' (not limiting just to that test or that spec
file),
>> and only right after a full deploy with GENDB=1 and TESTDATA=1. When I
>> run a second time, it passes.
>>
>> The direct cause of the failure is an ISE when attempting to bind by
>> product, from not finding an entitlement that grants the provided
>> product id.
>>
>> Now, there should be an entitlement that grants the requested product,
>> but it never appears because refresh pools is failing. I haven't yet
>> tracked the original cause of this failure, but it never gets reported
>> to the client (who just assumes the refresh passed, and continues
>> along). Even more interestingly, it seems as if the job keeps getting
>> run by pinsetter.
>>
>> I've tracked down the commit where this first starts happening,
>> 1ce24a780... which is just an addition of more test data. I've got no
>> idea why it would cause this.
>>
>> My suspicion is that this is some really nasty bug in refresh pools, so
>> I'm going to keep digging. If anyone has any ideas, let me know :)
>>
>> In addition, it would be good to fix the ISE from bind by product, and
>> to communicate the refresh pools failure in the job status.
>>
>> -James
>
> I've figured it out, now.
> This happens for me because entitlements runs directly after job_status.
> job status queues up a whole bunch of refresh pools jobs during its
> testing. At the end of a spec, created owners are cleaned up, but
> pending jobs are not.
>
> When the entitlement test begins, candlepin is working through ~14
> refresh pools jobs. entitlement creates an owner and gives that owner
> some subscriptions, then adds another refresh pools job.
>
> It's not clear to me if at this point the job queue is run in random
> order, or if failed jobs are readded to the queue, but we get into a
> state where the entitlement spec test refresh pools is run (and does
> what it should), then one of the stale jobs from the job_status spec is
> run.
>
> The stale jobs now refrence owners that don't exist, so when the refresh
> pools job runs, it gets a null owner object. Here's where the fun
> happens: refresh pools looks up subscriptions by owner, and pools by
> owner. An api call for subs by owner with a null owner returns 0
> subscriptions. A call for pools with a null owner returns _all_ pools.
> refresh pools then checks for any pool that does not have a backing sub,
> and deletes it. So now we have no pools or entitlements for anybody. Fun
> times!
>
> I'm going to fix this up just by adding a check to see if the returned
> owner is null, and if so, just quit out. This exposes failures in some
> other tests, that I expect depended on this behaviour as a way to clean
> up old pools before their owner got deleted. I'll fix those up too.
>
> Why is this happening now, and only with test data? I suspect that on my
> machine it's just enough data loaded into the db to slow things down to
> the point where this race condition begins to happen.
>
> Does anyone have any good suggestions for testing race conditions? Let
> me know if so, as otherwise I'm not sure how to unit/spec test this.
>
> -James
>
Not sure how to test the race condition, but we could perhaps cleanup
the jobs during owner deletion (I think that's a cleanup method in
OwnerResource) and test that, combined with the fix you spoke of and a
test on that, that's a decent level of confidence IMO.
What about pools? If an owner is deleted, should their pools be cleaned
up too? or should we prevent the owner from being deleted?
--
Devan Goodwin <dgoodwin(a)rm-rf.ca>
http://rm-rf.ca
_______________________________________________
candlepin mailing list
candlepin(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/candlepin
-James