RFC: #786844 - Detection of deleted instances

Mo Morsi mmorsi at redhat.com
Thu Feb 9 09:38:07 UTC 2012


On 02/08/2012 07:05 PM, Matt Wagner wrote:
> Hi all,
>
> I was looking at https://bugzilla.redhat.com/show_bug.cgi?id=786844 and
> ended up establishing somewhat of a rambling commentary in the comments.
>
> Basically, the issue is that if you launch an instance in Conductor, and
> then go into the cloud provider and *remove* the instance outside of
> Conductor, we never pick up any updates on the instance, showing it as
> running in perpetuity.

The option which we go w/ will depend on the semantics of the data we 
keep in the instances table. Ultimately records there represent local / 
persistant cached copies of data running on the cloud.

If we want these to always be kept in sync, then it makes sense to 
delete entries if they are not found on the cloud. If we want to use 
instances to also preserve a record of instance history, it makes sense 
to change the instance state to 'terminated' or 'invisible' as you 
suggested.

I like the former (deleting the record) as its simpler / easier to map 
what is in our db to the cloud but feel that it won't satisfy various 
auditing and other requirements coming down the road so the later option 
might be required. Alternatively we can create a separate table to store 
history but seems a bit more complicated for what its worth atm (at 
least until subsequent reqs come down the line justifying it).



>
> The reason this happens is that we process these updates by retrieving
> the list of instances from the provider via Deltacloud, and updating the
> status for each of them. But since cloud providers don't report back
> deleted instances, they aren't passed to Deltacloud, or, in turn, us.

At what point during the lifetime of conductor we currently query 
deltacloud for updates?


>
> There are a few possible fixes:
>
> a.) Do nothing, but document that if you remove instances outside of
> Conductor, we won't pick up on them. This is a valid course of action, I
> think, but probably not the best.

Agreed, not a long term solution. Would rather do it the right way (tm) 
from the get-go if possible.


>
> b.) After retrieving the updates, also go through our database and, for
> any that weren't in the list of instances, delete them from our
> database. This seems extreme, though, if only because I fear what will
> happen if there's some transient error.

The solution if we want to keep the instances table in sync w/ the cloud 
information we get from Deltacloud (as much as we can). Simpler but 
deleting data has its downsides too.


>
> c.) Add a new instance state, "vanished" or something to that effect,
> and update the status to that instead of deleting it. I think this is my
> favorite option, but I'm not sure what it actually accomplishes. It
> seems like there's still a lot of work we'd want to do around this, such
> as some sort of special highlight in the UI, and also making sure that
> you're able to delete 'vanished' instances without a problem. (But,
> again, there's the issue of what happens if delete an instance because
> of a transient error you're not aware of.)

The solution if we want to preserve more history / data. Problem is, it 
increases the complexity of semantics of the instances table (since data 
there may no longer be represented on the cloud and we will have to add 
checks for this in alot of places).


   -Mo



More information about the aeolus-devel mailing list