Ray asked about possible approaches to testing scheduler race conditions for http://gerrit.beaker-project.org/#/c/1718/
Here's my suggestion:
1. Set up 2 systems, A and B 2. Set up a pair of jobs, each containing a recipeset with a pair of recipes targeting the respective systems (so we will have recipes 1A, 1B, 2A, 2B queued in that order) 3. Mark system A unavailable 4. Monkeypatch beakerd.schedule_queued_recipe with a version that, after calling the regular version, marks system A as available, if recipe 1B was just scheduled 5. Call beakerd.schedule_queued_recipes 6. Ensure 1B is scheduled, 1A, 2A and 2B are still queued 7. Call beakerd.schedule_queued_recipes again 8. Ensure 1A and 1B are scheduled, 2A and 2B are still queued 9. Make sure to undo the monkeypatch even if an exception is thrown!
FWIW, Michael Foord's "mock" library makes this kind of temporary replacement of functions in the module under test really easy, and self.addCleanup in unittest2 is really handy as well. The unittest2 feature set hit the stdlib unittest in 2.7, and mock was added in 3.3 as unittest.mock. If we're ever looking for more things to do, adding/upgrading those dependencies could be worthwhile. You can definitely live without them, but they're nice to have.
(The upgraded unittest in 2.7 is actually one of the main benefits of running our automated test suite on Fedora during development: you get much more useful error messages out of self.assertEqual when comparing containers, so even 2.6 compatible tests benefit from the upgrade!)
Cheers, Nick.
On 02/13/2013 03:59 PM, Nick Coghlan wrote:
Ray asked about possible approaches to testing scheduler race conditions for http://gerrit.beaker-project.org/#/c/1718/
Here's my suggestion:
- Set up 2 systems, A and B
- Set up a pair of jobs, each containing a recipeset with a pair of
recipes targeting the respective systems (so we will have recipes 1A, 1B, 2A, 2B queued in that order) 3. Mark system A unavailable 4. Monkeypatch beakerd.schedule_queued_recipe with a version that, after calling the regular version, marks system A as available, if recipe 1B was just scheduled
/me explains this step to Ray offline...
To clarify, the reason this monkeypatching trick works is that the monkeypatched function gets called for every queued recipe, so it lets us keep track of where the scheduling loop in beakerd is up to in processing the queue:
schedule_queued_recipe(1A) # A unavailable, 1A remains queued schedule_queued_recipe(1B) # A marked available in separate session schedule_queued_recipe(2A) # 2A should also remain queued schedule_queued_recipe(2B)
The current bug (since we're not using nested sessions) is that schedule_queued_recipe(2A) will claim system A, resulting in the deadlock we're trying to fix.
Cheers, Nick.
beaker-devel@lists.fedorahosted.org