On 09/22/2011 05:57 AM, Ian Main wrote:
From: Chris Lalancette<clalance(a)redhat.com>
I modified Chris' patch to use a thread instead of fork. This pushes
the communications with deltacloud into a thread which can take as long
as it needs to complete the start.
In the future we may want to put a retry in here too.
Signed-off-by: Ian Main<imain(a)redhat.com>
---
src/app/util/taskomatic.rb | 67 +++++++++++++++++++++++++++----------------
1 files changed, 42 insertions(+), 25 deletions(-)
diff --git a/src/app/util/taskomatic.rb b/src/app/util/taskomatic.rb
index 84dabd0..eae459a 100644
--- a/src/app/util/taskomatic.rb
+++ b/src/app/util/taskomatic.rb
@@ -26,27 +26,15 @@ module Taskomatic
task.state = Task::STATE_PENDING
task.save!
- task.instance.provider_account = match.provider_account
- task.instance.create_auth_key unless task.instance.instance_key
-
- dcloud_instance = create_dcloud_instance(task.instance, match)
-
- handle_dcloud_error(dcloud_instance)
+ create_dcloud_instance(task.instance, match)
task.state = Task::STATE_RUNNING
- task.save!
-
- Rails.logger.info "Task instance create completed with key
#{dcloud_instance.id} and state #{dcloud_instance.state}"
- task.instance.external_key = dcloud_instance.id
- task.instance.state = dcloud_to_instance_state(dcloud_instance.state)
- task.instance.save!
rescue HttpException => ex
task.failure_code = Task::FAILURE_PROVIDER_CONTACT_FAILED
handle_create_instance_error(task, ex)
rescue Exception => ex
handle_create_instance_error(task, ex)
ensure
- task.instance.save!
task.save!
end
end
@@ -146,18 +134,47 @@ module Taskomatic
end
def self.create_dcloud_instance(instance, match)
- client = match.provider_account.connect
-
- overrides =
HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp)
-
- client.create_instance(:image_id => match.provider_image.target_identifier,
- :name => instance.name.tr("/", "-"),
- :hwp_id => match.hwp.external_key,
- :hwp_memory => overrides[:memory],
- :hwp_cpu => overrides[:cpu],
- :hwp_storage => overrides[:storage],
- :realm_id => (match.realm.external_key rescue nil),
- :keyname => (instance.instance_key.name))
+ # because creating an instance can take a potentially long time (and
+ # creating multiple of them just prolongs this), we start a new thread
+ # that does this work. The new thread continues to run in the background
+ # and communicates the status back to the main UI via the database
+
+ Thread.new do
+ begin
+ instance.provider_account = match.provider_account
+ instance.create_auth_key unless instance.instance_key
+
+ client = match.provider_account.connect
+
+ overrides =
HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp)
+
+ dcloud_instance = client.create_instance(:image_id =>
match.provider_image.target_identifier,
+ :name =>
instance.name.tr("/", "-"),
+ :hwp_id => match.hwp.external_key,
+ :hwp_memory => overrides[:memory],
+ :hwp_cpu => overrides[:cpu],
+ :hwp_storage =>
overrides[:storage],
+ :realm_id =>
(match.realm.external_key rescue nil),
+ :keyname =>
(instance.instance_key.name))
+
+ handle_dcloud_error(dcloud_instance)
+
+ Rails.logger.info "Task instance create completed with key
#{dcloud_instance.id} and state #{dcloud_instance.state}"
+ instance.external_key = dcloud_instance.id
+ instance.state = dcloud_to_instance_state(dcloud_instance.state)
+ rescue Exception => ex
+ # any sort of exception causes us to put the instance in CREATE_FAILED
+ # FIXME: if the exception is raised *after* the create_instance, this
+ # isn't true and will result in a rogue instance
+ Rails.logger.error ex.message
+ Rails.logger.error ex.backtrace.join("\n")
+ instance.state = Instance::STATE_CREATE_FAILED
+ raise ex
+ ensure
+ instance.save!
+ Thread.exit
+ end
+ end
end
def self.matches(instance)
Hi,
I'm hitting other error than Chris with this patch:
Instance(#70177876185840) expected, got Instance(#70177877837160)
Deployment Load (0.8ms) SELECT "deployments".* FROM "deployments"
ORDER BY name asc LIMIT 30 OFFSET 0
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.9/lib/active_record/associations/association_proxy.rb:270:in
`raise_on_type_mismatch'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.9/lib/active_record/associations/belongs_to_association.rb:23:in
`replace'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.9/lib/active_record/associations.rb:1465:in
`instance='
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.9/lib/active_record/base.rb:1564:in
`send'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.9/lib/active_record/base.rb:1564:in
`attributes='
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.9/lib/active_record/base.rb:1560:in
`each'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.9/lib/active_record/base.rb:1560:in
`attributes='
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.9/lib/active_record/base.rb:1412:in
`initialize'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.9/lib/active_record/validations.rb:32:in
`new'
/usr/lib/ruby/gems/1.8/gems/activerecord-3.0.9/lib/active_record/validations.rb:32:in
`create!'
/home/jprovazn/devel/conductor/src/app/models/instance.rb:250:in
`create_auth_key'
This occurs only if it runs in thread, if I remove Thread.new block it
works fine. Workaround for this error can be passing instance_id, istead
of instance in InstanceKey.create:
self.instance_key = InstanceKey.create!(:pem => key.pem.first, :name =>
key.id, :instance_id => self.id)
But this doesn't explain why original code is failing - old code should
be correct. For some reason Instance class is loaded twice with
different object_id when "raise_on_type_mismatch" is called.
Jan