As has been discussed in the past, using condor for
instance scheduling in conductor is overkill. We can get
away with a simpler system that is easier to understand
for now, and then look at scaling up in the future.
This patch removes condor from the conductor equation and
replaces it with a combination of inline calls and dbomatic.
Specifically, when an action needs to be performed on an
instance (start, stop, etc), that call is done directly
from the conductor to the appropriate deltacloud instance.
DBomatic is then responsible for updating the database with
information about the instance as it changes state.
To do this, DBomatic has been revamped substantially. Every
60 seconds it wakes up and attempts to update the status of
all instances. It uses one process per provider account for
this task; this ensures that a slow or unresponsive backend
cloud will not prevent the rest of the instances from being
updated. If a backend provider does not respond within 30
seconds, dbomatic gives up on the transaction and tries again
during the next update cycle.
This patch as it stands has been tested by me to work with the
EC2 backend. The original patch was done by markmc with some
contributions by Slow.
Signed-off-by: Chris Lalancette <clalance(a)redhat.com>
---
aeolus-all.spec.in | 1 -
aeolus-conductor.spec.in | 3 +-
src/app/controllers/deployments_controller.rb | 3 +-
src/app/controllers/instances_controller.rb | 3 +-
src/app/models/deployment.rb | 4 +-
src/app/models/instance.rb | 23 +-
src/app/util/condormatic.rb | 171 ----------
src/app/util/taskomatic.rb | 169 ++++++++++
...14114553_remove_condor_job_id_from_instances.rb | 9 +
src/dbomatic/dbomatic | 355 ++++++--------------
src/dutils/active_record_env.rb | 38 --
src/dutils/dutils.rb | 20 --
util/check_services | 18 +-
13 files changed, 303 insertions(+), 514 deletions(-)
delete mode 100644 src/app/util/condormatic.rb
create mode 100644 src/app/util/taskomatic.rb
create mode 100644 src/db/migrate/20110714114553_remove_condor_job_id_from_instances.rb
delete mode 100644 src/dutils/active_record_env.rb
delete mode 100644 src/dutils/dutils.rb
diff --git a/aeolus-all.spec.in b/aeolus-all.spec.in
index 81da990..e7daf00 100644
--- a/aeolus-all.spec.in
+++ b/aeolus-all.spec.in
@@ -9,7 +9,6 @@ URL:
http://aeolusproject.org
Requires: aeolus-conductor-daemons = %{version}-%{release}
Requires: aeolus-conductor-doc = %{version}-%{release}
-Requires: condor-deltacloud-gahp
Requires: iwhd
Requires: aeolus-configure
Requires: imagefactory
diff --git a/aeolus-conductor.spec.in b/aeolus-conductor.spec.in
index 9ec229b..b5ce578 100644
--- a/aeolus-conductor.spec.in
+++ b/aeolus-conductor.spec.in
@@ -153,7 +153,7 @@ png="public/images public/images/icons public/stylesheets/images
\
public/stylesheets/jquery.ui-1.8.1/images"
rake="lib/tasks"
rb="app/models app/controllers app/helpers app/services app/util config \
- config/initializers config/environments db db/migrate dutils \
+ config/initializers config/environments db db/migrate \
features/support features/step_definitions lib spec spec/controllers \
spec/factories spec/helpers spec/models spec/services"
rhtml="app/views/layouts"
@@ -269,7 +269,6 @@ fi
%{app_root}/config.ru
%{app_root}/db
%{app_root}/dbomatic
-%{app_root}/dutils
%{app_root}/lib
%{app_root}/log
%{app_root}/public
diff --git a/src/app/controllers/deployments_controller.rb
b/src/app/controllers/deployments_controller.rb
index bab4670..c0632dd 100644
--- a/src/app/controllers/deployments_controller.rb
+++ b/src/app/controllers/deployments_controller.rb
@@ -226,13 +226,12 @@ class DeploymentsController < ApplicationController
raise ActionError.new("stop is an invalid action.")
end
- # not sure if task is used as everything goes through condor
#permissons check here
@task = instance.queue_action(@current_user, 'stop')
unless @task
raise ActionError.new("stop cannot be performed on this
instance.")
end
- condormatic_instance_stop(@task)
+ Taskomatic.stop_instance(@task)
notices << "Deployment: #{instance.deployment.name}, Instance:
#{instance.name}: stop action was successfully queued.<br/>"
rescue Exception => err
errors << "Deployment: #{instance.deployment.name}, Instance:
#{instance.name}: " + err + "<br/>"
diff --git a/src/app/controllers/instances_controller.rb
b/src/app/controllers/instances_controller.rb
index 09046a1..3225b13 100644
--- a/src/app/controllers/instances_controller.rb
+++ b/src/app/controllers/instances_controller.rb
@@ -127,13 +127,12 @@ class InstancesController < ApplicationController
raise ActionError.new("stop is an invalid action.")
end
- # not sure if task is used as everything goes through condor
#permissons check here
@task = instance.queue_action(@current_user, 'stop')
unless @task
raise ActionError.new("stop cannot be performed on this instance.")
end
- condormatic_instance_stop(@task)
+ Taskomatic.stop_instance(@task)
notices << "#{instance.name}: stop action was successfully
queued.<br/>"
rescue Exception => err
errors << "#{instance.name}: " + err + "<br/>"
diff --git a/src/app/models/deployment.rb b/src/app/models/deployment.rb
index d0ee262..0ea28fa 100644
--- a/src/app/models/deployment.rb
+++ b/src/app/models/deployment.rb
@@ -137,7 +137,7 @@ class Deployment < ActiveRecord::Base
unless @task
raise ActionError.new("stop cannot be performed on this instance.")
end
- condormatic_instance_stop(@task)
+ Taskomatic.stop_instance(@task)
end
else
raise ActionError.new 'all instances must be stopped or running'
@@ -169,7 +169,7 @@ class Deployment < ActiveRecord::Base
:task_target => instance,
:action => InstanceTask::ACTION_CREATE})
end
- condormatic_instance_create(task)
+ Taskomatic.create_instance(task)
if task.state == Task::STATE_FAILED
status[:errors][assembly.name] = 'failed'
else
diff --git a/src/app/models/instance.rb b/src/app/models/instance.rb
index b682c06..55237c8 100644
--- a/src/app/models/instance.rb
+++ b/src/app/models/instance.rb
@@ -14,7 +14,6 @@
# public_addresses :string(255)
# private_addresses :string(255)
# state :string(255)
-# condor_job_id :string(255)
# last_error :text
# lock_version :integer default(0)
# acc_pending_time :integer default(0)
@@ -57,7 +56,6 @@
# Likewise, all the methods added will be available for all controllers.
require 'util/assembly_xml'
-require 'util/condormatic'
class Instance < ActiveRecord::Base
include PermissionedObject
@@ -275,6 +273,18 @@ class Instance < ActiveRecord::Base
:order => (order_field || 'name') +' '+ (order_dir ||
'asc'))
end
+ class Match
+ attr_reader :pool_family, :provider_account, :hwp, :provider_image, :realm
+
+ def initialize(pool_family, provider_account, hwp, provider_image, realm)
+ @pool_family = pool_family
+ @provider_account = provider_account
+ @hwp = hwp
+ @provider_image = provider_image
+ @realm = realm
+ end
+ end
+
def matches
errors = []
if pool.pool_family.provider_accounts.empty?
@@ -287,7 +297,7 @@ class Instance < ActiveRecord::Base
build = image_build || image.latest_build
provider_images = build ? build.provider_images : []
- possibles = []
+ matched = []
pool.pool_family.provider_accounts.each do |account|
# match_provider_hardware_profile returns a single provider
# hardware_profile that can satisfy the input hardware_profile
@@ -314,16 +324,15 @@ class Instance < ActiveRecord::Base
next
end
brealms.each do |brealm_target|
- possibles << Possible.new(pool.pool_family, account, hwp, pi,
- brealm_target.target_realm)
+ matched << Match.new(pool.pool_family, account, hwp, pi,
brealm_target.target_realm)
end
else
- possibles << Possible.new(pool.pool_family, account, hwp, pi, nil)
+ matched << Match.new(pool.pool_family, account, hwp, pi, nil)
end
end
end
- [possibles, errors]
+ [matched, errors]
end
def public_addresses
diff --git a/src/app/util/condormatic.rb b/src/app/util/condormatic.rb
deleted file mode 100644
index 431697a..0000000
--- a/src/app/util/condormatic.rb
+++ /dev/null
@@ -1,171 +0,0 @@
-#
-# Copyright (C) 2010,2011 Red Hat, Inc.
-# Written by Ian Main <imain(a)redhat.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; version 2 of the License.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
-# MA 02110-1301, USA. A copy of the GNU General Public License is
-# also available at
http://www.gnu.org/copyleft/gpl.html.
-
-require 'fileutils'
-require 'tempfile'
-
-class Possible
- attr_reader :pool_family, :account, :hwp, :provider_image, :realm
-
- def initialize(pool_family, account, hwp, provider_image, realm)
- @pool_family = pool_family
- @account = account
- @hwp = hwp
- @provider_image = provider_image
- @realm = realm
- end
-end
-
-def pipe_and_log(pipe, instr)
- pipe.puts instr
- Rails.logger.error instr
-end
-
-def write_pw_file(job_name, pw)
- # here we write out the password file
- # FIXME: should this be configurable?
- pwdir = '/var/lib/aeolus-conductor/jobs'
- FileUtils.mkdir_p(pwdir, options={:mode => 0700})
- FileUtils.chown('aeolus', 'aeolus', pwdir)
-
- # Restrict job names to relatively sane characters only
- job_name.gsub!(/[^a-zA-Z0-9\.\-]/, '_')
-
- pwfilename = File.join(pwdir, job_name)
-
- tmpfile = Tempfile.new(job_name, pwdir)
- tmpfilename = tmpfile.path
- tmpfile.write(pw)
- tmpfile.close
-
- File.rename(tmpfilename, pwfilename)
-
- return pwfilename
-end
-
-def condormatic_instance_create(task)
- instance = task.instance
- matches, errors = instance.matches
- found = matches.first
-
- begin
- if found.nil?
- raise "Could not find a matching backend provider, errors:
#{errors.join(', ')}"
- end
-
- job_name = "job_#{instance.name}_#{instance.id}"
-
- instance.condor_job_id = job_name
-
- overrides =
HardwareProfile.generate_override_property_values(instance.hardware_profile,
- found.hwp)
- pwfilename = write_pw_file(job_name,
- found.account.credentials_hash['password'])
-
- instance.provider_account = found.account
- instance.create_auth_key unless instance.instance_key
- keyname = instance.instance_key ? instance.instance_key.name : ''
-
- # I use the 2>&1 to get stderr and stdout together because popen3 does not
- # support the ability to get the exit value of the command in ruby 1.8.
- pipe = IO.popen("condor_submit 2>&1", "w+")
- pipe_and_log(pipe, "universe = grid\n")
- pipe_and_log(pipe, "executable = #{job_name}\n")
-
- pipe_and_log(pipe,
- "grid_resource = deltacloud #{found.account.provider.url}\n")
- pipe_and_log(pipe, "DeltacloudUsername =
#{found.account.credentials_hash['username']}\n")
- pipe_and_log(pipe, "DeltacloudPasswordFile = #{pwfilename}")
- pipe_and_log(pipe, "DeltacloudImageId =
#{found.provider_image.target_identifier}\n")
- pipe_and_log(pipe,
- "DeltacloudHardwareProfile = #{found.hwp.external_key}\n")
- pipe_and_log(pipe,
- "DeltacloudHardwareProfileMemory = #{overrides[:memory]}\n")
- pipe_and_log(pipe,
- "DeltacloudHardwareProfileCPU = #{overrides[:cpu]}\n")
- pipe_and_log(pipe,
- "DeltacloudHardwareProfileStorage =
#{overrides[:storage]}\n")
- pipe_and_log(pipe, "DeltacloudKeyname = #{keyname}\n")
- pipe_and_log(pipe, "DeltacloudPoolFamily = #{found.pool_family.id}\n")
-
- if found.realm != nil
- pipe_and_log(pipe, "DeltacloudRealmId = #{found.realm.external_key}\n")
- end
-
- pipe_and_log(pipe, "requirements = true\n")
- pipe_and_log(pipe, "notification = never\n")
- pipe_and_log(pipe, "queue\n")
-
- pipe.close_write
- out = pipe.read
- pipe.close
-
- Rails.logger.error "$? (return value?) is #{$?}"
- raise ("Error calling condor_submit: #{out}") if $? != 0
-
- task.state = Task::STATE_PENDING
- instance.state = Instance::STATE_PENDING
- rescue Exception => ex
- Rails.logger.error ex.message
- Rails.logger.error ex.backtrace.join("\n")
- task.state = Task::STATE_FAILED
- instance.state = Instance::STATE_CREATE_FAILED
- # exception is raised after ensure block
- raise ex
- ensure
- instance.save!
- task.save!
- end
-end
-
-def condormatic_instance_stop(task)
- instance = task.instance_of?(InstanceTask) ? task.instance : task
-
- Rails.logger.info("calling condor_rm -constraint 'Cmd ==
\"#{instance.condor_job_id}\"' 2>&1")
- pipe = IO.popen("condor_rm -constraint 'Cmd ==
\"#{instance.condor_job_id}\"' 2>&1")
- out = pipe.read
- pipe.close
-
- Rails.logger.info("condor_rm return status is #{$?}")
- Rails.logger.error("Error calling condor_rm (exit code #{$?}) on job:
#{out}") if $? != 0
-end
-
-def condormatic_instance_reset_error(instance)
-
- condormatic_instance_stop(instance)
- Rails.logger.info("calling condor_rm -forcex -constraint 'Cmd ==
\"#{instance.condor_job_id}\"' 2>&1")
- pipe = IO.popen("condor_rm -forcex -constraint 'Cmd ==
\"#{instance.condor_job_id}\"' 2>&1")
- out = pipe.read
- pipe.close
-
- Rails.logger.info("condor_rm return status is #{$?}")
- Rails.logger.error("Error calling condor_rm (exit code #{$?}) on job:
#{out}") if $? != 0
-end
-
-def condormatic_instance_destroy(task)
- instance = task.instance
-
- Rails.logger.info("calling condor_rm -constraint 'Cmd ==
\"#{instance.condor_job_id}\"' 2>&1")
- pipe = IO.popen("condor_rm -constraint 'Cmd ==
\"#{instance.condor_job_id}\"' 2>&1")
- out = pipe.read
- pipe.close
-
- Rails.logger.info("condor_rm return status is #{$?}")
- Rails.logger.error("Error calling condor_rm (exit code #{$?}) on job:
#{out}") if $? != 0
-end
diff --git a/src/app/util/taskomatic.rb b/src/app/util/taskomatic.rb
new file mode 100644
index 0000000..41018b8
--- /dev/null
+++ b/src/app/util/taskomatic.rb
@@ -0,0 +1,169 @@
+#
+# Copyright (C) 2011 Red Hat, Inc.
+# Written by Ian Main <imain(a)redhat.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA 02110-1301, USA. A copy of the GNU General Public License is
+# also available at
http://www.gnu.org/copyleft/gpl.html.
+
+module Taskomatic
+
+ def self.create_instance(task)
+ begin
+ match = matches(task.instance).first
+
+ task.state = Task::STATE_PENDING
+ task.save!
+
+ dcloud_instance = create_dcloud_instance(task.instance, match)
+
+ handle_dcloud_error(dcloud_instance)
+
+ task.state = Task::STATE_RUNNING
+ task.save!
+
+ Rails.logger.info "Task instance create completed with key
#{dcloud_instance.id} and state #{dcloud_instance.state}"
+ task.instance.provider_account = match.provider_account
+ task.instance.external_key = dcloud_instance.id
+ task.instance.state = dcloud_to_instance_state(dcloud_instance.state)
+ task.instance.save!
+ rescue HttpException => ex
+ task.failure_code = Task::FAILURE_PROVIDER_CONTACT_FAILED
+ handle_create_instance_error(task, ex)
+ rescue Exception => ex
+ handle_create_instance_error(task, ex)
+ ensure
+ task.instance.save!
+ task.save!
+ end
+ end
+
+ def self.do_action(task, action)
+ task.time_started = Time.now
+
+ begin
+ client = task.instance.provider_account.connect
+ dcloud_instance = client.instance(task.instance.external_key)
+
+ task.state = Task::STATE_PENDING
+ task.save!
+
+ dcloud_instance.send(action)
+
+ Rails.logger.info("Task instance '#{action}' complete, now in state
#{dcloud_instance.state}.")
+
+ task.instance.state = dcloud_to_instance_state(dcloud_instance.state)
+ task.instance.save!
+ task.state = Task::STATE_FINISHED
+ rescue Exception => ex
+ task.state = Task::STATE_FAILED
+ task.message = ex.message
+ ensure
+ task.save!
+ end
+ end
+
+ def self.start_instance(task)
+ do_action(task, :start!)
+ end
+
+ def self.stop_instance(task)
+ do_action(task, :stop!)
+ end
+
+ def self.reboot_instance(task)
+ do_action(task, :reboot!)
+ end
+
+ def self.destroy_instance(task)
+ task.time_started = Time.now
+
+ begin
+ client = task.instance.provider_account.connect
+ dcloud_instance = client.instance(task.instance.external_key)
+
+ task.state = Task::STATE_PENDING
+ task.save!
+
+ dcloud_instance.destroy!
+
+ Rails.logger.info("Task Destroy completed.")
+
+ task.state = Task::STATE_FINISHED
+ rescue Exception => ex
+ task.state = Task::STATE_FAILED
+ task.message = ex.message
+ ensure
+ task.save!
+ end
+ end
+
+ def self.dcloud_to_instance_state(state_str)
+ case state_str.upcase
+ when 'PENDING'
+ return Instance::STATE_PENDING
+ when 'RUNNING'
+ return Instance::STATE_RUNNING
+ when 'STOPPED'
+ return Instance::STATE_STOPPED
+ when 'TERMINATED'
+ return Instance::STATE_STOPPED
+ when 'SHUTTING_DOWN'
+ return Instance::STATE_SHUTTING_DOWN
+ else
+ return Instance::STATE_PENDING
+ end
+ end
+
+ private
+
+ class HttpException < Exception
+ end
+
+ def self.handle_dcloud_error(dcloud_instance)
+ raise HttpException, "Error creating dcloud instance, returned internal server
error." if dcloud_instance.class == Net::HTTPInternalServerError
+ end
+
+ def self.handle_create_instance_error(task, ex)
+ Rails.logger.error ex.message
+ Rails.logger.error ex.backtrace.join("\n")
+ task.state = Task::STATE_FAILED
+ task.instance.state = Instance::STATE_CREATE_FAILED
+ raise ex
+ end
+
+ def self.create_dcloud_instance(instance, match)
+ client = match.provider_account.connect
+
+ overrides =
HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp)
+
+ client.create_instance(:image_id => match.provider_image.target_identifier,
+ :name => instance.name,
+ :hwp_id => match.hwp.external_key,
+ :hwp_memory => overrides[:memory],
+ :hwp_cpu => overrides[:cpu],
+ :hwp_storage => overrides[:storage],
+ :realm_id => (match.realm.external_key rescue nil),
+ :keyname => (match.provider_account.instance_key.name
rescue nil))
+ end
+
+ def self.matches(instance)
+ matched, errors = instance.matches
+ if matched.empty?
+ raise "Could not find a matching backend provider, errors:
#{errors.join(', ')}"
+ end
+ matched
+ end
+
+end
diff --git a/src/db/migrate/20110714114553_remove_condor_job_id_from_instances.rb
b/src/db/migrate/20110714114553_remove_condor_job_id_from_instances.rb
new file mode 100644
index 0000000..be9fb6c
--- /dev/null
+++ b/src/db/migrate/20110714114553_remove_condor_job_id_from_instances.rb
@@ -0,0 +1,9 @@
+class RemoveCondorJobIdFromInstances < ActiveRecord::Migration
+ def self.up
+ remove_column :instances, :condor_job_id
+ end
+
+ def self.down
+ add_column :instances, :condor_job_id, :string
+ end
+end
diff --git a/src/dbomatic/dbomatic b/src/dbomatic/dbomatic
index ef8a918..393f09a 100755
--- a/src/dbomatic/dbomatic
+++ b/src/dbomatic/dbomatic
@@ -17,19 +17,19 @@
# MA 02110-1301, USA. A copy of the GNU General Public License is
# also available at
http://www.gnu.org/copyleft/gpl.html.
-$: << File.join(File.dirname(__FILE__), "../dutils")
+$: << File.join(File.dirname(__FILE__), "../app")
require 'rubygems'
-require 'dutils'
-require 'nokogiri'
require 'optparse'
+require 'singleton'
+require 'util/taskomatic'
help = false
daemon = true
-condor_event_log_dir = "/var/log/condor"
-dbomatic_run_dir = "/var/run/aeolus-conductor"
-dbomatic_log_dir = "/var/log/aeolus-conductor"
-dbomatic_pid_dir = "/var/run/aeolus-conductor"
+dbomatic_log_dir = "/var/log/aeolus-conductor"
+dbomatic_pid_dir = "/var/run/aeolus-conductor"
+$dbomatic_timeout = 60
+$deltacloud_timeout = 30
optparse = OptionParser.new do |opts|
@@ -39,6 +39,9 @@ dbomatic [options]
Options:
BANNER
+ opts.on( '-d', '--deltacloud-timeout N', 'Time(in seconds) to wait
for backend clouds to respond (defaults to #{$deltacloud_timeout})', Integer) do
|timeout|
+ $deltacloud_timeout = timeout
+ end
opts.on( '-f', '--pid-file PATH', "Use PATH to the dbomatic pid
directory (defaults to #{dbomatic_pid_dir})") do |newpath|
dbomatic_pid_dir = newpath
end
@@ -47,11 +50,8 @@ BANNER
dbomatic_log_dir = newpath
end
opts.on( '-n', '--nodaemon', 'Do not daemonize (useful in
combination with -l for debugging)') { daemon = false }
- opts.on( '-p', '--path PATH', "Use PATH to the condor log
directory (defaults to #{condor_event_log_dir})") do |newpath|
- condor_event_log_dir = newpath
- end
- opts.on( '-r', '--run PATH', "Use PATH to the dbomatic runtime
directory (defaults to #{dbomatic_run_dir})") do |newpath|
- dbomatic_run_dir = newpath
+ opts.on( '-t', '--timeout N', 'Time out (in seconds) between
refreshes (defaults to #{$dbomatic_timeout})', Integer) do |timeout|
+ $dbomatic_timeout = timeout
end
end
@@ -69,9 +69,9 @@ if help
exit(0)
end
-CONDOR_EVENT_LOG_FILE = "#{condor_event_log_dir}/EventLog"
-CONDOR_EVENT_LOG_FILE_OLD = "#{condor_event_log_dir}/EventLog.old"
-EVENT_LOG_POS_FILE = "#{dbomatic_run_dir}/event_log_position"
+# load in the rails models
+require File.dirname(__FILE__) + '/../config/environment'
+
if dbomatic_log_dir == '-'
DBOMATIC_LOG_FILE = STDOUT
DBOMATIC_PARSER_LOG_FILE = STDOUT
@@ -82,276 +82,127 @@ end
# daemonize
if daemon
- # note that this requires 'active_support', which we get for free from dutils
+ # note that this requires 'active_support'
Process.daemon
end
# Custom Log Format
class DBomaticLogger < Logger
- def format_message(severity, timestamp, progname, msg)
- "#{timestamp.to_formatted_s(:db)} #{severity} #{msg}\n"
- end
-end
-
-# Handle the event log's xml
-class CondorEventLog < Nokogiri::XML::SAX::Document
- attr_accessor :tag, :event_type, :event_cmd, :event_time, :trigger_type,
:grid_resource, :execute_host, :username, :hold_reason, :private_addresses,
:public_addresses
+ include Singleton
def initialize
- @logger = DBomaticLogger.new(DBOMATIC_PARSER_LOG_FILE)
- @logger.level = Logger::DEBUG
- @logger.info "DBOmatic parser starting up"
+ super(DBOMATIC_LOG_FILE)
end
- # Store the name of the event log attribute we're looking at
- def start_element(element, attributes)
- # NOTE: this code is written to work both with Nokogiri 1.4.x and 1.5.x
- # As soon as we stop supporting 1.4.x, we should change this to:
- # @tag = attributes.first[1] if element == "a"
- # and remove the rest.
-
- return unless element == "a"
-
- # Nokogiri 1.5.x passes attributes as [['name1', 'val1'],
['name2,'val2']]
- # while Nokogiri 1.4.x uses a flat array
- @tag = (Array === attributes.first) ? attributes.first[1] : attributes[1]
- end
-
- # Store the value of the event log attribute we're looking at
- def characters(string)
- unless string.strip == ""
- if @tag == "MyType"
- @event_type = string
- elsif @tag == "Cmd"
- @event_cmd = string
- elsif @tag == "EventTime"
- @event_time = string
- elsif @tag == "TriggerEventTypeName"
- @trigger_type = string
- elsif @tag == "GridResource"
- @grid_resource = string
- elsif @tag == "ExecuteHost"
- @execute_host = string
- elsif @tag == "DeltacloudUsername"
- @username = string
- elsif @tag == "HoldReason"
- @hold_reason = string
- elsif @tag == "DeltacloudPublicNetworkAddresses"
- @public_addresses = string
- elsif @tag == "DeltacloudPrivateNetworkAddresses"
- @private_addresses = string
- elsif @tag == "DeltacloudProviderId"
- @provider_instance_id = string
- end
- end
+ def format_message(severity, timestamp, progname, msg)
+ "#{timestamp.to_formatted_s(:db)} #{severity} #{msg}\n"
end
+end
- def update_instance_state_event(inst)
- @logger.info "update_instance_state_event for #{inst}"
+logger = DBomaticLogger.instance
+logger.level = Logger::DEBUG
+logger.datetime_format = "%Y-%m-%d %H:%M " # simplify time output
+logger.info "DBOmatic starting up"
- if @trigger_type == "ULOG_GRID_SUBMIT"
- inst.state = Instance::STATE_PENDING
- elsif @trigger_type == "ULOG_JOB_ABORTED" or @trigger_type ==
"ULOG_JOB_TERMINATED"
- inst.state = Instance::STATE_STOPPED
- elsif @trigger_type == "ULOG_EXECUTE"
- inst.state = Instance::STATE_RUNNING
- elsif @trigger_type == "ULOG_SUBMIT"
- # ULOG_SUBMIT happens when the job is first submitted to condor.
- # However, it's not a state that we care to export to users, but it's
- # also not an error, so we just silently ignore it.
- elsif @trigger_type == "ULOG_JOB_HELD"
- # The job has some error condition.
- #
- # FIXME: we also may want to delete this job from condor, depending
- # on the error. For instance, if you are trying to start an instance
- # with a mismatched image and hardwareprofile architecture, the only
- # reasonable way out is to create a new instance. Needs thought.
- # - I've added a condormatic_instance_reset_error() method to reset
- # error conditions such as this. In the future we should give different
- # options based on the error condition.
- # FIXME: Right now we don't parse out the HoldReason (or HoldReasonCode)
- # so for now I'm going to set this to STATE_ERROR as there are multiple
- # possible reasons for going into the 'held' state.
- #
- # FIXME: This only adds the error to the instance 'last_error' field. We
- # should really be logging this information into the event log but that is not
- # set up at this time so for now this will do.
- inst.last_error = @hold_reason
- inst.state = Instance::STATE_ERROR
- else
- @logger.warn "Unexpected trigger type #{@trigger_type}, not updating instance
state"
- return
- end
+def self.now
+ Time.now.strftime('%Y-%m-%d %H:%M:%S')
+end
- begin
- @logger.info "update_instance_state_event saving instance #{inst}"
- inst.save!
- @logger.debug "updated_instance_state_event saved instance #{inst}, creating
event for state #{inst.state}@#{@event_time}"
- inst.events.create!(:status_code => inst.state,
- :event_time => @event_time)
- rescue Exception => e
- @logger.error "#{e.backtrace.shift}: #{e.message}"
- e.backtrace.each do |step|
- @logger.error "\tfrom #{step}"
+def collect_accounts
+ accounts = []
+ Pool.all.each do |pool|
+ pool.instances.each do |instance|
+ if instance.provider_account and instance.state != Instance::STATE_NEW and not
accounts.include?(instance.provider_account)
+ accounts << instance.provider_account
end
end
-
- @logger.info "update_instance_state_event completed fo #{inst}"
- end
-
- def update_instance_addresses(inst)
- @logger.info "update_instance_addresses for #{inst}, \
- setting public addresses: #{@public_addresses} \
- --- and private addresses #{private_addresses}"
-
- inst.public_addresses = @public_addresses
- inst.private_addresses = @private_addresses
- inst.save!
- @logger.info "update_instance_addresses completed for #{inst}"
end
+ accounts
+end
- def update_provider_instance_id(inst)
- @logger.info "update_provider_instance_id for #{inst}, \
- setting id: #{@provider_instance_id}"
-
- inst.provider_instance_id = @provider_instance_id
- inst.save!
- @logger.info "update_provider_instance_id completed for #{inst}"
- end
+def check_one_account(account)
+ connection = account.connect
- # Create a new entry for events which we have all the necessary data for
- def end_element(element)
- begin
- if element == "c" and @event_type == "JobAdInformationEvent"
and !(a)trigger_type.nil?
+ account.instances.each do |instance|
+ if instance.state != Instance::STATE_NEW
+ api_instance = connection.instance(instance.external_key)
+ if api_instance
+ DBomaticLogger.instance.debug("updating instance state for #{instance.name}:
#{instance.external_key}. #{api_instance}")
+ instance.state = Taskomatic.dcloud_to_instance_state(api_instance.state)
- inst = Instance.find(:first, :conditions => ['condor_job_id = ?',
@event_cmd])
- if inst.nil?
- @logger.warn "Unexpected nil instance, skipping..."
- else
- @logger.info "Instance #{inst} found, running update events"
- update_instance_state_event(inst)
- update_instance_addresses(inst)
- update_provider_instance_id(inst)
- @logger.info "Instance #{inst} update events completed"
+ # only update the public and private addresses if they are not nil.
+ # this prevents us from deleting known information about instances
+ if not api_instance.public_addresses.empty?
+ instance.public_addresses = api_instance.public_addresses
end
- @tag = @event_type = @event_cmd = @event_time = @trigger_type = @grid_resource =
@execute_host = @hold_reason = @public_addresses = @private_addresses =
@provider_instance_id = nil
- end
- rescue Exception => e
- @logger.error "#{e.backtrace.shift}: #{e.message}"
- e.backtrace.each do |step|
- @logger.error "\tfrom #{step}"
+ if not api_instance.private_addresses.empty?
+ instance.private_addresses = api_instance.private_addresses
+ end
+
+ instance.save!
+ instance.events.create!(:status_code => instance.state, :event_time =>
now)
+ else
+ DBomaticLogger.instance.debug("ignoring unknown instance #{instance.name}
#{instance.external_key}")
end
end
end
end
-# Retrieve the current parsing position
-# of the log file as stored in the log
-# file postion tracker
-def get_log_file_pos
- pos = 0
- if File.exists?(EVENT_LOG_POS_FILE)
- File.open(EVENT_LOG_POS_FILE, 'r') { |f| pos = f.read.to_i }
+def refresh_instances
+ accounts = collect_accounts
+
+ # the idea here is that we fork off one process for each provider account.
+ # this is so that one slow or non-responsive deltacloud doesn't hold up the
+ # status for all of them
+ pids = []
+ accounts.each do |account|
+ pid = Process.fork
+ if pid.nil?
+ # child
+ check_one_account(account)
+ Kernel.exit!
+ else
+ # parent
+ pids << pid
+ end
end
- pos
-end
-
-# Set the current parsing position
-# of the log file in the log file
-# position tracker
-def set_log_file_pos(pos)
- File.open(EVENT_LOG_POS_FILE, 'w') { |f| f.write pos.to_s }
-end
-# FIXME we should make sure everything here is done atomically
-def parse_log_file(parser)
- # since the actual log file may be rotated out
- # open a new handle every time we want to parse
- log_file = File.open(CONDOR_EVENT_LOG_FILE)
-
- # persistantly store log position in filesystem
- # incase of dbomatic restarts
- log_file.pos = get_log_file_pos
-
- # if the log has been rotated out
- # FIXME probably can use a better check for this, like tracking
- # the modification times of these files. Also what happens if condor
- # rotates the logs multiple times b4 this executes (is this possible?)
- if log_file.pos > File.size(CONDOR_EVENT_LOG_FILE)
- if File.exists?(CONDOR_EVENT_LOG_FILE_OLD)
- # finish parsing old log file
- old_log_file = File.open(CONDOR_EVENT_LOG_FILE_OLD)
- old_log_file.pos = log_file.pos
- while s = old_log_file.gets
- parser << s
- end
+ # only the parent gets here, as all children exit above
+ start = Time.now.to_i
+ while not pids.empty?
+ pid = Process.wait(pid=-1, flags=Process::WNOHANG)
+ if not pid.nil?
+ pids.delete(pid)
+ next
end
- # reset position
- log_file.pos = 0
- end
+ # before sleeping, see if we have exceeded the timeout. If so, kill off
+ # all remaining children and just hope for the best next time around
+ if (Time.now.to_i - start) > $deltacloud_timeout
+ DBomaticLogger.instance.warn "Connection timeout of #{$deltacloud_timeout}
seconds exceeded, backend clouds could not be contacted. Will try again in
#{$dbomatic_timeout} seconds"
+ pids.each {|pid| Process.kill(9, pid)}
+ end
- while s = log_file.gets
- parser << s
+ sleep 1
end
-
- set_log_file_pos(log_file.pos)
end
-logger = DBomaticLogger.new(DBOMATIC_LOG_FILE)
-logger.level = Logger::DEBUG
-logger.datetime_format = "%Y-%m-%d %H:%M " # simplify time output
-logger.info "DBOmatic starting up"
-
-begin
- DBOMATIC_PID_FILE = "#{dbomatic_pid_dir}/dbomatic.pid"
- FileUtils.mkdir_p File.dirname(DBOMATIC_PID_FILE)
- open(DBOMATIC_PID_FILE, "w") {|f| f.write(Process.pid) }
- File.chmod(0644, DBOMATIC_PID_FILE)
-
- parser = Nokogiri::XML::SAX::PushParser.new CondorEventLog.new
-
- # XXX hack, condor event log doesn't seem to have a top level element
- # enclosing everything else in the doc (as standards conforming xml must).
- # Create one for parsing purposes.
- parser << "<events>"
-
- parse_log_file(parser) if File.exists? CONDOR_EVENT_LOG_FILE
- logger.info "Parsed existing event log file - current postition:
#{get_log_file_pos}"
-
- # hacky fix for BZ #712316 (better dbomatic implementation in progress)
- # wait until condor event log file exists
- sleep 2 until File.exist?(CONDOR_EVENT_LOG_FILE)
-
- # track the modification time of the file for changes
- logfile_timestamp = File.mtime(CONDOR_EVENT_LOG_FILE)
-
- logger.info "Beginning main event loop"
- while true
- sleep 2
- logger.debug "Checking for condor event log modification"
- if File.mtime(CONDOR_EVENT_LOG_FILE) > logfile_timestamp
- logger.info "EventLog modification event triggered, parsing log"
- begin
- logfile_timestamp = File.mtime(CONDOR_EVENT_LOG_FILE)
- parse_log_file parser
- rescue Exception => e
- logger.error "Parser error: #{e.backtrace.shift}: #{e.message}"
- e.backtrace.each do |step|
- logger.error "\tfrom #{step}"
- end
- end
- logger.info "EventLog modification event trigger completed, parsing finished -
current position #{get_log_file_pos}"
+DBOMATIC_PID_FILE = "#{dbomatic_pid_dir}/dbomatic.pid"
+FileUtils.mkdir_p File.dirname(DBOMATIC_PID_FILE)
+open(DBOMATIC_PID_FILE, "w") {|f| f.write(Process.pid) }
+File.chmod(0644, DBOMATIC_PID_FILE)
+
+logger.info "Beginning main event loop"
+while true
+ logger.debug "Deltacloud instances refresh started"
+ begin
+ refresh_instances
+ rescue Exception => e
+ logger.error "#{e.backtrace.shift}: #{e.message}"
+ e.backtrace.each do |step|
+ logger.error "\tfrom #{step}"
end
end
- logger.info "Main event loop completed"
-
- parser << "</events>"
- parser.finish
- logger.info "Finished parsing, now exiting"
-rescue Exception => e
- logger.error "#{e.backtrace.shift}: #{e.message}"
- e.backtrace.each do |step|
- logger.error "\tfrom #{step}"
- end
+ logger.debug "Deltacloud instances refresh completed"
+ sleep $dbomatic_timeout
end
diff --git a/src/dutils/active_record_env.rb b/src/dutils/active_record_env.rb
deleted file mode 100644
index cee3b62..0000000
--- a/src/dutils/active_record_env.rb
+++ /dev/null
@@ -1,38 +0,0 @@
-#
-# Copyright (C) 2008 Red Hat, Inc.
-# Written by Scott Seago <sseago(a)redhat.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; version 2 of the License.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
-# MA 02110-1301, USA. A copy of the GNU General Public License is
-# also available at
http://www.gnu.org/copyleft/gpl.html.
-
-$: << File.join(File.dirname(__FILE__), "../app")
-$: << File.join(File.dirname(__FILE__), "..")
-
-require 'rubygems'
-
-$LOAD_PATH << File.expand_path(File.dirname(__FILE__))
-
-ENV['RAILS_ENV'] = 'development' unless ENV['RAILS_ENV']
-
-require File.dirname(__FILE__) + '/../config/boot'
-require File.dirname(__FILE__) + '/../config/environment'
-
-def database_connect
- conf = YAML::load(File.open(File.dirname(__FILE__) +
'/../config/database.yml'))
- ActiveRecord::Base.establish_connection(conf[ENV['RAILS_ENV']])
-end
-
-# Open ActiveRecord connection
-database_connect
diff --git a/src/dutils/dutils.rb b/src/dutils/dutils.rb
deleted file mode 100644
index c35aeed..0000000
--- a/src/dutils/dutils.rb
+++ /dev/null
@@ -1,20 +0,0 @@
-# Copyright (C) 2008 Red Hat, Inc.
-# Written by Chris Lalancette <clalance(a)redhat.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; version 2 of the License.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
-# MA 02110-1301, USA. A copy of the GNU General Public License is
-# also available at
http://www.gnu.org/copyleft/gpl.html.
-
-require 'active_record_env'
-
diff --git a/util/check_services b/util/check_services
index 8cd51e8..1964487 100755
--- a/util/check_services
+++ b/util/check_services
@@ -1,5 +1,5 @@
#!/usr/bin/ruby
-init_scripts=%w(aeolus-conductor condor deltacloud-core deltacloud-ec2-us-east-1
deltacloud-ec2-us-west-1 deltacloud-mock httpd imagefactory iwhd libvirtd mongod ntpd
postgresql qpidd)
+init_scripts=%w(aeolus-conductor deltacloud-core deltacloud-ec2-us-east-1
deltacloud-ec2-us-west-1 deltacloud-mock httpd imagefactory iwhd libvirtd mongod ntpd
postgresql qpidd)
init_scripts.each do |script|
puts "\nChecking #{script} ..."
@@ -11,19 +11,3 @@ init_scripts.each do |script|
puts " \e[1;31mFAILURE:\e[0m #{out.strip}"
end
end
-
-# Other checks
-commands = [
- {:name => 'condor_q', :command => 'condor_q'},
- {:name => 'condor_status', :command => 'condor_status'}
-]
-commands.each do |cmd|
- puts "\nChecking #{cmd[:name]} ..."
- cmd = "#{cmd[:command]}"
- out = `#{cmd}`
- if $?.to_i == 0
- puts " \e[1;32mSuccess:\e[0m #{out.strip}"
- else
- puts " \e[1;31mFAILURE:\e[0m #{out.strip}"
- end
-end
--
1.7.4.4