RFC: #786844 - Detection of deleted instances

Hugh Brock hbrock at redhat.com
Thu Feb 9 14:42:57 UTC 2012


On Thu, Feb 09, 2012 at 09:23:22AM -0500, Matt Wagner wrote:
> On Thu, Feb 09, 2012 at 08:27:34AM -0500, Hugh Brock wrote:
> > On Thu, Feb 09, 2012 at 10:58:01AM +0100, Jan Provaznik wrote:
> > > On 02/09/2012 01:05 AM, Matt Wagner wrote:
> > > >Hi all,
> > > >
> > > >I was looking at https://bugzilla.redhat.com/show_bug.cgi?id=786844 and
> > > >ended up establishing somewhat of a rambling commentary in the comments.
> > > >
> > > >Basically, the issue is that if you launch an instance in Conductor, and
> > > >then go into the cloud provider and *remove* the instance outside of
> > > >Conductor, we never pick up any updates on the instance, showing it as
> > > >running in perpetuity.
> > > >
> > > >The reason this happens is that we process these updates by retrieving
> > > >the list of instances from the provider via Deltacloud, and updating the
> > > >status for each of them. But since cloud providers don't report back
> > > >deleted instances, they aren't passed to Deltacloud, or, in turn, us.
> > > >
> > > >There are a few possible fixes:
> > > >
> > > >a.) Do nothing, but document that if you remove instances outside of
> > > >Conductor, we won't pick up on them. This is a valid course of action, I
> > > >think, but probably not the best.
> > > >
> > > >b.) After retrieving the updates, also go through our database and, for
> > > >any that weren't in the list of instances, delete them from our
> > > >database. This seems extreme, though, if only because I fear what will
> > > >happen if there's some transient error.
> > > >
> > > >c.) Add a new instance state, "vanished" or something to that effect,
> > > >and update the status to that instead of deleting it. I think this is my
> > > >favorite option, but I'm not sure what it actually accomplishes. It
> > > >seems like there's still a lot of work we'd want to do around this, such
> > > >as some sort of special highlight in the UI, and also making sure that
> > > >you're able to delete 'vanished' instances without a problem. (But,
> > > >again, there's the issue of what happens if delete an instance because
> > > >of a transient error you're not aware of.)
> > > >
> > > >Does anyone else want to weigh in? I'm just worried that we might be
> > > >opening a big can of worms here.
> > > >
> > > >-- Matt
> > > 
> > > 
> > > 
> > > $0.02:
> > > I'd prefer b) for 1.0 release + adding new instance event "instance
> > > vanished" to this instance so a user can check on instance show page
> > > what happened.
> > > 
> > > option c) would be post 1.0 solution (starting with this now sounds
> > > like a Rambo mission with many explosions)
> > 
> > Heh... no Rambo missions at this point, I agree.
> > 
> > Can we do b., but somehow make it less likely that we would delete
> > instances that are temporarily absent? i.e. if an instance hasn't
> > shown up in the list 5 times in a row, only then do we delete it?
> > Maybe that's just as bad as c. Let me know.
> 
> But Rambo is so much fun!
> 
> I think implementing (b) but with an event is a good plan for 1.0.
> 
> I think the idea of requiring that it be absent in a few sequential
> calls is a good one, though it does lengthen the time that we show an
> instance in "running" state erroneously. But given that this is a
> stopgap fix and that status quo is to leave the instance in "running"
> forever, I think that's okay.

OK, sounds good, let's go this way.

--H

-- 
== Hugh Brock, hbrock at redhat.com                                   ==
== Engineering Manager, Cloud BU                                   ==
== Aeolus Project: Manage virtual infrastructure across clouds.    ==
== http://aeolusproject.org                                        ==

"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey



More information about the aeolus-devel mailing list