[PATCH conductor 1/2] BZ 786844 - Reporting of deleted instances

Matt Wagner matt.wagner at redhat.com
Mon Feb 13 19:19:16 UTC 2012


If an instance goes missing on the backend provider, we no longer ignore it.
Instead, if it's missing in two consecutive checks, we should remove it.

This is an interim solution; there is a discussion about a long-term fix here:
http://lists.fedorahosted.org/pipermail/aeolus-devel/2012-February/008735.html

Resolves https://bugzilla.redhat.com/show_bug.cgi?id=786844
---
 src/app/models/instance.rb |    7 ++++---
 src/dbomatic/dbomatic      |   18 ++++++++++++++++--
 2 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/src/app/models/instance.rb b/src/app/models/instance.rb
index 5ecfa3f..1ef0794 100644
--- a/src/app/models/instance.rb
+++ b/src/app/models/instance.rb
@@ -108,16 +108,17 @@ class Instance < ActiveRecord::Base
   STATE_STOPPED        = "stopped"
   STATE_CREATE_FAILED  = "create_failed"
   STATE_ERROR          = "error"
+  STATE_VANISHED       = "vanished"
 
   STATES = [STATE_NEW, STATE_PENDING, STATE_RUNNING,
              STATE_SHUTTING_DOWN, STATE_STOPPED, STATE_CREATE_FAILED,
-             STATE_ERROR]
+             STATE_ERROR, STATE_VANISHED]
 
   scope :deployed,  :conditions => { :state => [STATE_RUNNING, STATE_SHUTTING_DOWN] }
   # FIXME: "pending" is misleading as it doesn't just cover STATE_PENDING
   scope :pending,   :conditions => { :state => [STATE_NEW, STATE_PENDING] }
   # FIXME: "failed" is misleading too...
-  scope :failed,    :conditions => { :state => [STATE_CREATE_FAILED, STATE_ERROR] }
+  scope :failed,    :conditions => { :state => [STATE_CREATE_FAILED, STATE_ERROR, STATE_VANISHED] }
   scope :stopable,    :conditions => { :state => [STATE_NEW, STATE_PENDING, STATE_RUNNING] }
   scope :ascending_by_name, :order => 'instances.name ASC'
 
@@ -320,7 +321,7 @@ class Instance < ActiveRecord::Base
   end
 
   def destroyable?
-    (state == STATE_CREATE_FAILED) or (state == STATE_STOPPED and not restartable?)
+    (state == STATE_CREATE_FAILED) or (state == STATE_STOPPED and not restartable?) or (state == STATE_VANISHED)
   end
 
   def failed?
diff --git a/src/dbomatic/dbomatic b/src/dbomatic/dbomatic
index e5ac4bc..1719d06 100755
--- a/src/dbomatic/dbomatic
+++ b/src/dbomatic/dbomatic
@@ -30,10 +30,10 @@ help = false
 daemon = true
 dbomatic_log_dir = "/var/log/aeolus-conductor"
 dbomatic_pid_dir = "/var/run/aeolus-conductor"
+# $dbomatic_timeout is also our polling interval between cloud status checks
 $dbomatic_timeout = 60
 $realms_timeout = 300
 $deltacloud_timeout = 30
-
 optparse = OptionParser.new do |opts|
 
   opts.banner = <<BANNER
@@ -200,7 +200,21 @@ def check_one_account(account)
         # Only update the instance / create an event if anything has changed!
         instance.save! if instance.changed?
       else
-        DBomaticLogger.instance.info("ignoring unknown instance #{instance.name} #{instance.external_key}")
+        # We have an instance in our database, but it didn't come back over the API
+        DBomaticLogger.instance.info("known instance missing from provider: #{instance.name} #{instance.external_key}")
+        if instance.state == Instance::STATE_VANISHED
+          # If the instance is found, we automatically update the state, so this means that the
+          # instance was missing on two *consecutive* calls.
+          DBomaticLogger.instance.info("instance was missing on last call as well, so deleting")
+          Event.create(
+            :source => instance, :event_time => Time.now,
+            :status_code => 'vanished', :summary => "Instance disappeared from cloud provider"
+          )
+          instance.destroy
+        else
+          DBomaticLogger.instance.info("instance was present on last call, so setting state to vanished but not deleting")
+          instance.update_attribute(:state, Instance::STATE_VANISHED)
+        end
       end
     end
 
-- 
1.7.6.5




More information about the aeolus-devel mailing list