RFC: #786844 - Detection of deleted instances

Scott Seago sseago at redhat.com
Fri Feb 10 17:05:29 UTC 2012


On 02/09/2012 09:23 AM, Matt Wagner wrote:
> On Thu, Feb 09, 2012 at 08:27:34AM -0500, Hugh Brock wrote:
>> On Thu, Feb 09, 2012 at 10:58:01AM +0100, Jan Provaznik wrote:
>>> On 02/09/2012 01:05 AM, Matt Wagner wrote:
>>>> Hi all,
>>>>
>>>> I was looking at https://bugzilla.redhat.com/show_bug.cgi?id=786844 and
>>>> ended up establishing somewhat of a rambling commentary in the comments.
>>>>
>>>> Basically, the issue is that if you launch an instance in Conductor, and
>>>> then go into the cloud provider and *remove* the instance outside of
>>>> Conductor, we never pick up any updates on the instance, showing it as
>>>> running in perpetuity.
>>>>
>>>> The reason this happens is that we process these updates by retrieving
>>>> the list of instances from the provider via Deltacloud, and updating the
>>>> status for each of them. But since cloud providers don't report back
>>>> deleted instances, they aren't passed to Deltacloud, or, in turn, us.
>>>>
>>>> There are a few possible fixes:
>>>>
>>>> a.) Do nothing, but document that if you remove instances outside of
>>>> Conductor, we won't pick up on them. This is a valid course of action, I
>>>> think, but probably not the best.
>>>>
>>>> b.) After retrieving the updates, also go through our database and, for
>>>> any that weren't in the list of instances, delete them from our
>>>> database. This seems extreme, though, if only because I fear what will
>>>> happen if there's some transient error.
>>>>
>>>> c.) Add a new instance state, "vanished" or something to that effect,
>>>> and update the status to that instead of deleting it. I think this is my
>>>> favorite option, but I'm not sure what it actually accomplishes. It
>>>> seems like there's still a lot of work we'd want to do around this, such
>>>> as some sort of special highlight in the UI, and also making sure that
>>>> you're able to delete 'vanished' instances without a problem. (But,
>>>> again, there's the issue of what happens if delete an instance because
>>>> of a transient error you're not aware of.)
>>>>
>>>> Does anyone else want to weigh in? I'm just worried that we might be
>>>> opening a big can of worms here.
>>>>
>>>> -- Matt
>>>
>>>
>>> $0.02:
>>> I'd prefer b) for 1.0 release + adding new instance event "instance
>>> vanished" to this instance so a user can check on instance show page
>>> what happened.
>>>
>>> option c) would be post 1.0 solution (starting with this now sounds
>>> like a Rambo mission with many explosions)
>> Heh... no Rambo missions at this point, I agree.
>>
>> Can we do b., but somehow make it less likely that we would delete
>> instances that are temporarily absent? i.e. if an instance hasn't
>> shown up in the list 5 times in a row, only then do we delete it?
>> Maybe that's just as bad as c. Let me know.
> But Rambo is so much fun!
>
> I think implementing (b) but with an event is a good plan for 1.0.
>
> I think the idea of requiring that it be absent in a few sequential
> calls is a good one, though it does lengthen the time that we show an
> instance in "running" state erroneously. But given that this is a
> stopgap fix and that status quo is to leave the instance in "running"
> forever, I think that's okay.
>
> -- Matt
How about we change the status the first time we hit a "missing" 
instance to something like "inaccessible (probably terminated)" or 
"missing". If we have a status update field somewhere we can even track 
_when_ the change happened (so we know when "enough time" has elapsed to 
nuke the record).

Scott




More information about the aeolus-devel mailing list