RFC: #786844 - Detection of deleted instances

Matt Wagner matt.wagner at redhat.com
Thu Feb 9 14:23:22 UTC 2012


On Thu, Feb 09, 2012 at 08:27:34AM -0500, Hugh Brock wrote:
> On Thu, Feb 09, 2012 at 10:58:01AM +0100, Jan Provaznik wrote:
> > On 02/09/2012 01:05 AM, Matt Wagner wrote:
> > >Hi all,
> > >
> > >I was looking at https://bugzilla.redhat.com/show_bug.cgi?id=786844 and
> > >ended up establishing somewhat of a rambling commentary in the comments.
> > >
> > >Basically, the issue is that if you launch an instance in Conductor, and
> > >then go into the cloud provider and *remove* the instance outside of
> > >Conductor, we never pick up any updates on the instance, showing it as
> > >running in perpetuity.
> > >
> > >The reason this happens is that we process these updates by retrieving
> > >the list of instances from the provider via Deltacloud, and updating the
> > >status for each of them. But since cloud providers don't report back
> > >deleted instances, they aren't passed to Deltacloud, or, in turn, us.
> > >
> > >There are a few possible fixes:
> > >
> > >a.) Do nothing, but document that if you remove instances outside of
> > >Conductor, we won't pick up on them. This is a valid course of action, I
> > >think, but probably not the best.
> > >
> > >b.) After retrieving the updates, also go through our database and, for
> > >any that weren't in the list of instances, delete them from our
> > >database. This seems extreme, though, if only because I fear what will
> > >happen if there's some transient error.
> > >
> > >c.) Add a new instance state, "vanished" or something to that effect,
> > >and update the status to that instead of deleting it. I think this is my
> > >favorite option, but I'm not sure what it actually accomplishes. It
> > >seems like there's still a lot of work we'd want to do around this, such
> > >as some sort of special highlight in the UI, and also making sure that
> > >you're able to delete 'vanished' instances without a problem. (But,
> > >again, there's the issue of what happens if delete an instance because
> > >of a transient error you're not aware of.)
> > >
> > >Does anyone else want to weigh in? I'm just worried that we might be
> > >opening a big can of worms here.
> > >
> > >-- Matt
> > 
> > 
> > 
> > $0.02:
> > I'd prefer b) for 1.0 release + adding new instance event "instance
> > vanished" to this instance so a user can check on instance show page
> > what happened.
> > 
> > option c) would be post 1.0 solution (starting with this now sounds
> > like a Rambo mission with many explosions)
> 
> Heh... no Rambo missions at this point, I agree.
> 
> Can we do b., but somehow make it less likely that we would delete
> instances that are temporarily absent? i.e. if an instance hasn't
> shown up in the list 5 times in a row, only then do we delete it?
> Maybe that's just as bad as c. Let me know.

But Rambo is so much fun!

I think implementing (b) but with an event is a good plan for 1.0.

I think the idea of requiring that it be absent in a few sequential
calls is a good one, though it does lengthen the time that we show an
instance in "running" state erroneously. But given that this is a
stopgap fix and that status quo is to leave the instance in "running"
forever, I think that's okay.

-- Matt



More information about the aeolus-devel mailing list