RFC: #786844 - Detection of deleted instances

Hugh Brock hbrock at redhat.com
Thu Feb 9 13:27:34 UTC 2012


On Thu, Feb 09, 2012 at 10:58:01AM +0100, Jan Provaznik wrote:
> On 02/09/2012 01:05 AM, Matt Wagner wrote:
> >Hi all,
> >
> >I was looking at https://bugzilla.redhat.com/show_bug.cgi?id=786844 and
> >ended up establishing somewhat of a rambling commentary in the comments.
> >
> >Basically, the issue is that if you launch an instance in Conductor, and
> >then go into the cloud provider and *remove* the instance outside of
> >Conductor, we never pick up any updates on the instance, showing it as
> >running in perpetuity.
> >
> >The reason this happens is that we process these updates by retrieving
> >the list of instances from the provider via Deltacloud, and updating the
> >status for each of them. But since cloud providers don't report back
> >deleted instances, they aren't passed to Deltacloud, or, in turn, us.
> >
> >There are a few possible fixes:
> >
> >a.) Do nothing, but document that if you remove instances outside of
> >Conductor, we won't pick up on them. This is a valid course of action, I
> >think, but probably not the best.
> >
> >b.) After retrieving the updates, also go through our database and, for
> >any that weren't in the list of instances, delete them from our
> >database. This seems extreme, though, if only because I fear what will
> >happen if there's some transient error.
> >
> >c.) Add a new instance state, "vanished" or something to that effect,
> >and update the status to that instead of deleting it. I think this is my
> >favorite option, but I'm not sure what it actually accomplishes. It
> >seems like there's still a lot of work we'd want to do around this, such
> >as some sort of special highlight in the UI, and also making sure that
> >you're able to delete 'vanished' instances without a problem. (But,
> >again, there's the issue of what happens if delete an instance because
> >of a transient error you're not aware of.)
> >
> >Does anyone else want to weigh in? I'm just worried that we might be
> >opening a big can of worms here.
> >
> >-- Matt
> 
> 
> 
> $0.02:
> I'd prefer b) for 1.0 release + adding new instance event "instance
> vanished" to this instance so a user can check on instance show page
> what happened.
> 
> option c) would be post 1.0 solution (starting with this now sounds
> like a Rambo mission with many explosions)

Heh... no Rambo missions at this point, I agree.

Can we do b., but somehow make it less likely that we would delete
instances that are temporarily absent? i.e. if an instance hasn't
shown up in the list 5 times in a row, only then do we delete it?
Maybe that's just as bad as c. Let me know.

--H

-- 
== Hugh Brock, hbrock at redhat.com                                   ==
== Engineering Manager, Cloud BU                                   ==
== Aeolus Project: Manage virtual infrastructure across clouds.    ==
== http://aeolusproject.org                                        ==

"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey



More information about the aeolus-devel mailing list