tricky spec failure

James Bowes jbowes at redhat.com
Tue Jun 21 14:00:50 UTC 2011


On Tue, Jun 21, 2011 at 10:37:07AM -0300, James Bowes wrote:
> Hey all:
> 
> For the past few days I've been trying to track down the spec test
> failure in " Entitlements should bypasses rules for "candlepin"
> consumers"
> 
> I've been able to reliably reproduce this failure, but only on a full
> run of 'buildr spec' (not limiting just to that test or that spec file),
> and only right after a full deploy with GENDB=1 and TESTDATA=1. When I
> run a second time, it passes.
> 
> The direct cause of the failure is an ISE when attempting to bind by
> product, from not finding an entitlement that grants the provided
> product id.
> 
> Now, there should be an entitlement that grants the requested product,
> but it never appears because refresh pools is failing. I haven't yet
> tracked the original cause of this failure, but it never gets reported
> to the client (who just assumes the refresh passed, and continues
> along). Even more interestingly, it seems as if the job keeps getting
> run by pinsetter.
> 
> I've tracked down the commit where this first starts happening,
> 1ce24a780... which is just an addition of more test data. I've got no
> idea why it would cause this.
> 
> My suspicion is that this is some really nasty bug in refresh pools, so
> I'm going to keep digging. If anyone has any ideas, let me know :)
> 
> In addition, it would be good to fix the ISE from bind by product, and
> to communicate the refresh pools failure in the job status.
> 
> -James

I've figured it out, now. 
This happens for me because entitlements runs directly after job_status.
job status queues up a whole bunch of refresh pools jobs during its
testing. At the end of a spec, created owners are cleaned up, but
pending jobs are not.

When the entitlement test begins, candlepin is working through ~14
refresh pools jobs. entitlement creates an owner and gives that owner
some subscriptions, then adds another refresh pools job.

It's not clear to me if at this point the job queue is run in random
order, or if failed jobs are readded to the queue, but we get into a
state where the entitlement spec test refresh pools is run (and does
what it should), then one of the stale jobs from the job_status spec is
run.

The stale jobs now refrence owners that don't exist, so when the refresh
pools job runs, it gets a null owner object. Here's where the fun
happens: refresh pools looks up subscriptions by owner, and pools by
owner. An api call for subs by owner with a null owner returns 0
subscriptions. A call for pools with a null owner returns _all_ pools.
refresh pools then checks for any pool that does not have a backing sub,
and deletes it. So now we have no pools or entitlements for anybody. Fun
times!

I'm going to fix this up just by adding a check to see if the returned
owner is null, and if so, just quit out. This exposes failures in some
other tests, that I expect depended on this behaviour as a way to clean
up old pools before their owner got deleted. I'll fix those up too.

Why is this happening now, and only with test data? I suspect that on my
machine it's just enough data loaded into the db to slow things down to
the point where this race condition begins to happen.

Does anyone have any good suggestions for testing race conditions? Let
me know if so, as otherwise I'm not sure how to unit/spec test this.

-James
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/candlepin/attachments/20110621/6b2a47f0/attachment.sig>


More information about the candlepin mailing list