[PATCH conductor 0/1] #786844 - Reporting of deleted instances

Matt Wagner matt.wagner at redhat.com
Fri Feb 10 17:01:39 UTC 2012


Hi all,

This patch diverges slightly from what we had discussed previously[1], but the aim is roughly the same: if an instance is missing on the backend provider after two consecutive checks, we can surmise that it has been deleted there and should remove it from our database. (Currently, we completely ignore it and just leave it in its previous state.)

I was getting some pretty reliable segfaults related to Nokogiri earlier when running this, though it would *eventually* succeed as it kept retrying. I'm hoping this was a transient error and not something that reliably fails for others. (Of course, more than that, I'm hoping that we get a reliable Nokogiri/Ruby setup soon.)

I had a miserable adventure trying to properly keep a counter of missed checks that persisted between runs, given that we are using this crazy blend of global variables and threads. I decided that the only real fix was to persist this information on the Instance model, but adding a counter column for this felt quite hackish, so I went ahead and just added the "vanished" state I had proposed for a longer-term fix. Currently we only use the vanished state to indicate that it failed the previous check, and if we find it on the next run, we delete the instance.

I wanted to write some RSpec tests for this, but it currently has no tests that I can find, and I don't even know how I would go about writing tests for something that doesn't use classes. We might want to think about refactoring dbomatic down the road. (IMHO)

For what it's worth, a few notes about testing this:
a.) EC2 appears to wait about an hour after termination before it stops reporting the instance in its instance list. You probably shouldn't test there unless you have a high amount of patience. (I also have a Deltacloud patch that fixes a possible error condition there.)
b.) Mock stores its "instances" in /var/tmp/deltacloud-mock-nobody/instances though I think there was some caching going on so you may have to restart deltacloud after deleting an instance there. That's probably the most expedient means of testing this.

-- Matt

[1] http://lists.fedorahosted.org/pipermail/aeolus-devel/2012-February/008735.html



More information about the aeolus-devel mailing list