As has been discussed in the past, using condor for instance scheduling in conductor is overkill. We can get away with a simpler system that is easier to understand for now, and then look at scaling up in the future.

This patch removes condor from the conductor equation and replaces it with a combination of inline calls and dbomatic. Specifically, when an action needs to be performed on an instance (start, stop, etc), that call is done directly from the conductor to the appropriate deltacloud instance. DBomatic is then responsible for updating the database with information about the instance as it changes state.

To do this, DBomatic has been revamped substantially. Every 60 seconds it wakes up and attempts to update the status of all instances. It uses one process per provider account for this task; this ensures that a slow or unresponsive backend cloud will not prevent the rest of the instances from being updated. If a backend provider does not respond within 30 seconds, dbomatic gives up on the transaction and tries again during the next update cycle.

This patch as it stands has been tested by me to work with the EC2 backend. The original patch was done by markmc with some contributions by Slow.

Signed-off-by: Chris Lalancette clalance@redhat.com --- aeolus-all.spec.in | 1 - aeolus-conductor.spec.in | 3 +- src/app/controllers/deployments_controller.rb | 3 +- src/app/controllers/instances_controller.rb | 3 +- src/app/models/deployment.rb | 4 +- src/app/models/instance.rb | 23 +- src/app/util/condormatic.rb | 171 ---------- src/app/util/taskomatic.rb | 169 ++++++++++ ...14114553_remove_condor_job_id_from_instances.rb | 9 + src/dbomatic/dbomatic | 355 ++++++-------------- src/dutils/active_record_env.rb | 38 -- src/dutils/dutils.rb | 20 -- util/check_services | 18 +- 13 files changed, 303 insertions(+), 514 deletions(-) delete mode 100644 src/app/util/condormatic.rb create mode 100644 src/app/util/taskomatic.rb create mode 100644 src/db/migrate/20110714114553_remove_condor_job_id_from_instances.rb delete mode 100644 src/dutils/active_record_env.rb delete mode 100644 src/dutils/dutils.rb

diff --git a/aeolus-all.spec.in b/aeolus-all.spec.in index 81da990..e7daf00 100644 --- a/aeolus-all.spec.in +++ b/aeolus-all.spec.in @@ -9,7 +9,6 @@ URL: http://aeolusproject.org

Requires: aeolus-conductor-daemons = %{version}-%{release} Requires: aeolus-conductor-doc = %{version}-%{release} -Requires: condor-deltacloud-gahp Requires: iwhd Requires: aeolus-configure Requires: imagefactory diff --git a/aeolus-conductor.spec.in b/aeolus-conductor.spec.in index 9ec229b..b5ce578 100644 --- a/aeolus-conductor.spec.in +++ b/aeolus-conductor.spec.in @@ -153,7 +153,7 @@ png="public/images public/images/icons public/stylesheets/images \ public/stylesheets/jquery.ui-1.8.1/images" rake="lib/tasks" rb="app/models app/controllers app/helpers app/services app/util config \ - config/initializers config/environments db db/migrate dutils \ + config/initializers config/environments db db/migrate \ features/support features/step_definitions lib spec spec/controllers \ spec/factories spec/helpers spec/models spec/services" rhtml="app/views/layouts" @@ -269,7 +269,6 @@ fi %{app_root}/config.ru %{app_root}/db %{app_root}/dbomatic -%{app_root}/dutils %{app_root}/lib %{app_root}/log %{app_root}/public diff --git a/src/app/controllers/deployments_controller.rb b/src/app/controllers/deployments_controller.rb index bab4670..c0632dd 100644 --- a/src/app/controllers/deployments_controller.rb +++ b/src/app/controllers/deployments_controller.rb @@ -226,13 +226,12 @@ class DeploymentsController < ApplicationController raise ActionError.new("stop is an invalid action.") end

- # not sure if task is used as everything goes through condor #permissons check here @task = instance.queue_action(@current_user, 'stop') unless @task raise ActionError.new("stop cannot be performed on this instance.") end - condormatic_instance_stop(@task) + Taskomatic.stop_instance(@task) notices << "Deployment: #{instance.deployment.name}, Instance: #{instance.name}: stop action was successfully queued.<br/>" rescue Exception => err errors << "Deployment: #{instance.deployment.name}, Instance: #{instance.name}: " + err + "<br/>" diff --git a/src/app/controllers/instances_controller.rb b/src/app/controllers/instances_controller.rb index 09046a1..3225b13 100644 --- a/src/app/controllers/instances_controller.rb +++ b/src/app/controllers/instances_controller.rb @@ -127,13 +127,12 @@ class InstancesController < ApplicationController raise ActionError.new("stop is an invalid action.") end

- # not sure if task is used as everything goes through condor #permissons check here @task = instance.queue_action(@current_user, 'stop') unless @task raise ActionError.new("stop cannot be performed on this instance.") end - condormatic_instance_stop(@task) + Taskomatic.stop_instance(@task) notices << "#{instance.name}: stop action was successfully queued.<br/>" rescue Exception => err errors << "#{instance.name}: " + err + "<br/>" diff --git a/src/app/models/deployment.rb b/src/app/models/deployment.rb index d0ee262..0ea28fa 100644 --- a/src/app/models/deployment.rb +++ b/src/app/models/deployment.rb @@ -137,7 +137,7 @@ class Deployment < ActiveRecord::Base unless @task raise ActionError.new("stop cannot be performed on this instance.") end - condormatic_instance_stop(@task) + Taskomatic.stop_instance(@task) end else raise ActionError.new 'all instances must be stopped or running' @@ -169,7 +169,7 @@ class Deployment < ActiveRecord::Base :task_target => instance, :action => InstanceTask::ACTION_CREATE}) end - condormatic_instance_create(task) + Taskomatic.create_instance(task) if task.state == Task::STATE_FAILED status[:errors][assembly.name] = 'failed' else diff --git a/src/app/models/instance.rb b/src/app/models/instance.rb index b682c06..55237c8 100644 --- a/src/app/models/instance.rb +++ b/src/app/models/instance.rb @@ -14,7 +14,6 @@ # public_addresses :string(255) # private_addresses :string(255) # state :string(255) -# condor_job_id :string(255) # last_error :text # lock_version :integer default(0) # acc_pending_time :integer default(0) @@ -57,7 +56,6 @@ # Likewise, all the methods added will be available for all controllers.

require 'util/assembly_xml' -require 'util/condormatic' class Instance < ActiveRecord::Base include PermissionedObject

@@ -275,6 +273,18 @@ class Instance < ActiveRecord::Base :order => (order_field || 'name') +' '+ (order_dir || 'asc')) end

+ class Match + attr_reader :pool_family, :provider_account, :hwp, :provider_image, :realm + + def initialize(pool_family, provider_account, hwp, provider_image, realm) + @pool_family = pool_family + @provider_account = provider_account + @hwp = hwp + @provider_image = provider_image + @realm = realm + end + end + def matches errors = [] if pool.pool_family.provider_accounts.empty? @@ -287,7 +297,7 @@ class Instance < ActiveRecord::Base

build = image_build || image.latest_build provider_images = build ? build.provider_images : [] - possibles = [] + matched = [] pool.pool_family.provider_accounts.each do |account| # match_provider_hardware_profile returns a single provider # hardware_profile that can satisfy the input hardware_profile @@ -314,16 +324,15 @@ class Instance < ActiveRecord::Base next end brealms.each do |brealm_target| - possibles << Possible.new(pool.pool_family, account, hwp, pi, - brealm_target.target_realm) + matched << Match.new(pool.pool_family, account, hwp, pi, brealm_target.target_realm) end else - possibles << Possible.new(pool.pool_family, account, hwp, pi, nil) + matched << Match.new(pool.pool_family, account, hwp, pi, nil) end end end

- [possibles, errors] + [matched, errors] end

def public_addresses diff --git a/src/app/util/condormatic.rb b/src/app/util/condormatic.rb deleted file mode 100644 index 431697a..0000000 --- a/src/app/util/condormatic.rb +++ /dev/null @@ -1,171 +0,0 @@ -# -# Copyright (C) 2010,2011 Red Hat, Inc. -# Written by Ian Main imain@redhat.com -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; version 2 of the License. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, -# MA 02110-1301, USA. A copy of the GNU General Public License is -# also available at http://www.gnu.org/copyleft/gpl.html. - -require 'fileutils' -require 'tempfile' - -class Possible - attr_reader :pool_family, :account, :hwp, :provider_image, :realm - - def initialize(pool_family, account, hwp, provider_image, realm) - @pool_family = pool_family - @account = account - @hwp = hwp - @provider_image = provider_image - @realm = realm - end -end - -def pipe_and_log(pipe, instr) - pipe.puts instr - Rails.logger.error instr -end - -def write_pw_file(job_name, pw) - # here we write out the password file - # FIXME: should this be configurable? - pwdir = '/var/lib/aeolus-conductor/jobs' - FileUtils.mkdir_p(pwdir, options={:mode => 0700}) - FileUtils.chown('aeolus', 'aeolus', pwdir) - - # Restrict job names to relatively sane characters only - job_name.gsub!(/[^a-zA-Z0-9.-]/, '_') - - pwfilename = File.join(pwdir, job_name) - - tmpfile = Tempfile.new(job_name, pwdir) - tmpfilename = tmpfile.path - tmpfile.write(pw) - tmpfile.close - - File.rename(tmpfilename, pwfilename) - - return pwfilename -end - -def condormatic_instance_create(task) - instance = task.instance - matches, errors = instance.matches - found = matches.first - - begin - if found.nil? - raise "Could not find a matching backend provider, errors: #{errors.join(', ')}" - end - - job_name = "job_#{instance.name}_#{instance.id}" - - instance.condor_job_id = job_name - - overrides = HardwareProfile.generate_override_property_values(instance.hardware_profile, - found.hwp) - pwfilename = write_pw_file(job_name, - found.account.credentials_hash['password']) - - instance.provider_account = found.account - instance.create_auth_key unless instance.instance_key - keyname = instance.instance_key ? instance.instance_key.name : '' - - # I use the 2>&1 to get stderr and stdout together because popen3 does not - # support the ability to get the exit value of the command in ruby 1.8. - pipe = IO.popen("condor_submit 2>&1", "w+") - pipe_and_log(pipe, "universe = grid\n") - pipe_and_log(pipe, "executable = #{job_name}\n") - - pipe_and_log(pipe, - "grid_resource = deltacloud #{found.account.provider.url}\n") - pipe_and_log(pipe, "DeltacloudUsername = #{found.account.credentials_hash['username']}\n") - pipe_and_log(pipe, "DeltacloudPasswordFile = #{pwfilename}") - pipe_and_log(pipe, "DeltacloudImageId = #{found.provider_image.target_identifier}\n") - pipe_and_log(pipe, - "DeltacloudHardwareProfile = #{found.hwp.external_key}\n") - pipe_and_log(pipe, - "DeltacloudHardwareProfileMemory = #{overrides[:memory]}\n") - pipe_and_log(pipe, - "DeltacloudHardwareProfileCPU = #{overrides[:cpu]}\n") - pipe_and_log(pipe, - "DeltacloudHardwareProfileStorage = #{overrides[:storage]}\n") - pipe_and_log(pipe, "DeltacloudKeyname = #{keyname}\n") - pipe_and_log(pipe, "DeltacloudPoolFamily = #{found.pool_family.id}\n") - - if found.realm != nil - pipe_and_log(pipe, "DeltacloudRealmId = #{found.realm.external_key}\n") - end - - pipe_and_log(pipe, "requirements = true\n") - pipe_and_log(pipe, "notification = never\n") - pipe_and_log(pipe, "queue\n") - - pipe.close_write - out = pipe.read - pipe.close - - Rails.logger.error "$? (return value?) is #{$?}" - raise ("Error calling condor_submit: #{out}") if $? != 0 - - task.state = Task::STATE_PENDING - instance.state = Instance::STATE_PENDING - rescue Exception => ex - Rails.logger.error ex.message - Rails.logger.error ex.backtrace.join("\n") - task.state = Task::STATE_FAILED - instance.state = Instance::STATE_CREATE_FAILED - # exception is raised after ensure block - raise ex - ensure - instance.save! - task.save! - end -end - -def condormatic_instance_stop(task) - instance = task.instance_of?(InstanceTask) ? task.instance : task - - Rails.logger.info("calling condor_rm -constraint 'Cmd == "#{instance.condor_job_id}"' 2>&1") - pipe = IO.popen("condor_rm -constraint 'Cmd == "#{instance.condor_job_id}"' 2>&1") - out = pipe.read - pipe.close - - Rails.logger.info("condor_rm return status is #{$?}") - Rails.logger.error("Error calling condor_rm (exit code #{$?}) on job: #{out}") if $? != 0 -end - -def condormatic_instance_reset_error(instance) - - condormatic_instance_stop(instance) - Rails.logger.info("calling condor_rm -forcex -constraint 'Cmd == "#{instance.condor_job_id}"' 2>&1") - pipe = IO.popen("condor_rm -forcex -constraint 'Cmd == "#{instance.condor_job_id}"' 2>&1") - out = pipe.read - pipe.close - - Rails.logger.info("condor_rm return status is #{$?}") - Rails.logger.error("Error calling condor_rm (exit code #{$?}) on job: #{out}") if $? != 0 -end - -def condormatic_instance_destroy(task) - instance = task.instance - - Rails.logger.info("calling condor_rm -constraint 'Cmd == "#{instance.condor_job_id}"' 2>&1") - pipe = IO.popen("condor_rm -constraint 'Cmd == "#{instance.condor_job_id}"' 2>&1") - out = pipe.read - pipe.close - - Rails.logger.info("condor_rm return status is #{$?}") - Rails.logger.error("Error calling condor_rm (exit code #{$?}) on job: #{out}") if $? != 0 -end diff --git a/src/app/util/taskomatic.rb b/src/app/util/taskomatic.rb new file mode 100644 index 0000000..41018b8 --- /dev/null +++ b/src/app/util/taskomatic.rb @@ -0,0 +1,169 @@ +# +# Copyright (C) 2011 Red Hat, Inc. +# Written by Ian Main imain@redhat.com +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, +# MA 02110-1301, USA. A copy of the GNU General Public License is +# also available at http://www.gnu.org/copyleft/gpl.html. + +module Taskomatic + + def self.create_instance(task) + begin + match = matches(task.instance).first + + task.state = Task::STATE_PENDING + task.save! + + dcloud_instance = create_dcloud_instance(task.instance, match) + + handle_dcloud_error(dcloud_instance) + + task.state = Task::STATE_RUNNING + task.save! + + Rails.logger.info "Task instance create completed with key #{dcloud_instance.id} and state #{dcloud_instance.state}" + task.instance.provider_account = match.provider_account + task.instance.external_key = dcloud_instance.id + task.instance.state = dcloud_to_instance_state(dcloud_instance.state) + task.instance.save! + rescue HttpException => ex + task.failure_code = Task::FAILURE_PROVIDER_CONTACT_FAILED + handle_create_instance_error(task, ex) + rescue Exception => ex + handle_create_instance_error(task, ex) + ensure + task.instance.save! + task.save! + end + end + + def self.do_action(task, action) + task.time_started = Time.now + + begin + client = task.instance.provider_account.connect + dcloud_instance = client.instance(task.instance.external_key) + + task.state = Task::STATE_PENDING + task.save! + + dcloud_instance.send(action) + + Rails.logger.info("Task instance '#{action}' complete, now in state #{dcloud_instance.state}.") + + task.instance.state = dcloud_to_instance_state(dcloud_instance.state) + task.instance.save! + task.state = Task::STATE_FINISHED + rescue Exception => ex + task.state = Task::STATE_FAILED + task.message = ex.message + ensure + task.save! + end + end + + def self.start_instance(task) + do_action(task, :start!) + end + + def self.stop_instance(task) + do_action(task, :stop!) + end + + def self.reboot_instance(task) + do_action(task, :reboot!) + end + + def self.destroy_instance(task) + task.time_started = Time.now + + begin + client = task.instance.provider_account.connect + dcloud_instance = client.instance(task.instance.external_key) + + task.state = Task::STATE_PENDING + task.save! + + dcloud_instance.destroy! + + Rails.logger.info("Task Destroy completed.") + + task.state = Task::STATE_FINISHED + rescue Exception => ex + task.state = Task::STATE_FAILED + task.message = ex.message + ensure + task.save! + end + end + + def self.dcloud_to_instance_state(state_str) + case state_str.upcase + when 'PENDING' + return Instance::STATE_PENDING + when 'RUNNING' + return Instance::STATE_RUNNING + when 'STOPPED' + return Instance::STATE_STOPPED + when 'TERMINATED' + return Instance::STATE_STOPPED + when 'SHUTTING_DOWN' + return Instance::STATE_SHUTTING_DOWN + else + return Instance::STATE_PENDING + end + end + + private + + class HttpException < Exception + end + + def self.handle_dcloud_error(dcloud_instance) + raise HttpException, "Error creating dcloud instance, returned internal server error." if dcloud_instance.class == Net::HTTPInternalServerError + end + + def self.handle_create_instance_error(task, ex) + Rails.logger.error ex.message + Rails.logger.error ex.backtrace.join("\n") + task.state = Task::STATE_FAILED + task.instance.state = Instance::STATE_CREATE_FAILED + raise ex + end + + def self.create_dcloud_instance(instance, match) + client = match.provider_account.connect + + overrides = HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp) + + client.create_instance(:image_id => match.provider_image.target_identifier, + :name => instance.name, + :hwp_id => match.hwp.external_key, + :hwp_memory => overrides[:memory], + :hwp_cpu => overrides[:cpu], + :hwp_storage => overrides[:storage], + :realm_id => (match.realm.external_key rescue nil), + :keyname => (match.provider_account.instance_key.name rescue nil)) + end + + def self.matches(instance) + matched, errors = instance.matches + if matched.empty? + raise "Could not find a matching backend provider, errors: #{errors.join(', ')}" + end + matched + end + +end diff --git a/src/db/migrate/20110714114553_remove_condor_job_id_from_instances.rb b/src/db/migrate/20110714114553_remove_condor_job_id_from_instances.rb new file mode 100644 index 0000000..be9fb6c --- /dev/null +++ b/src/db/migrate/20110714114553_remove_condor_job_id_from_instances.rb @@ -0,0 +1,9 @@ +class RemoveCondorJobIdFromInstances < ActiveRecord::Migration + def self.up + remove_column :instances, :condor_job_id + end + + def self.down + add_column :instances, :condor_job_id, :string + end +end diff --git a/src/dbomatic/dbomatic b/src/dbomatic/dbomatic index ef8a918..393f09a 100755 --- a/src/dbomatic/dbomatic +++ b/src/dbomatic/dbomatic @@ -17,19 +17,19 @@ # MA 02110-1301, USA. A copy of the GNU General Public License is # also available at http://www.gnu.org/copyleft/gpl.html.

-$: << File.join(File.dirname(__FILE__), "../dutils") +$: << File.join(File.dirname(__FILE__), "../app")

require 'rubygems' -require 'dutils' -require 'nokogiri' require 'optparse' +require 'singleton' +require 'util/taskomatic'

help = false daemon = true -condor_event_log_dir = "/var/log/condor" -dbomatic_run_dir = "/var/run/aeolus-conductor" -dbomatic_log_dir = "/var/log/aeolus-conductor" -dbomatic_pid_dir = "/var/run/aeolus-conductor" +dbomatic_log_dir = "/var/log/aeolus-conductor" +dbomatic_pid_dir = "/var/run/aeolus-conductor" +$dbomatic_timeout = 60 +$deltacloud_timeout = 30

optparse = OptionParser.new do |opts|

@@ -39,6 +39,9 @@ dbomatic [options]

Options: BANNER + opts.on( '-d', '--deltacloud-timeout N', 'Time(in seconds) to wait for backend clouds to respond (defaults to #{$deltacloud_timeout})', Integer) do |timeout| + $deltacloud_timeout = timeout + end opts.on( '-f', '--pid-file PATH', "Use PATH to the dbomatic pid directory (defaults to #{dbomatic_pid_dir})") do |newpath| dbomatic_pid_dir = newpath end @@ -47,11 +50,8 @@ BANNER dbomatic_log_dir = newpath end opts.on( '-n', '--nodaemon', 'Do not daemonize (useful in combination with -l for debugging)') { daemon = false } - opts.on( '-p', '--path PATH', "Use PATH to the condor log directory (defaults to #{condor_event_log_dir})") do |newpath| - condor_event_log_dir = newpath - end - opts.on( '-r', '--run PATH', "Use PATH to the dbomatic runtime directory (defaults to #{dbomatic_run_dir})") do |newpath| - dbomatic_run_dir = newpath + opts.on( '-t', '--timeout N', 'Time out (in seconds) between refreshes (defaults to #{$dbomatic_timeout})', Integer) do |timeout| + $dbomatic_timeout = timeout end end

@@ -69,9 +69,9 @@ if help exit(0) end

-CONDOR_EVENT_LOG_FILE = "#{condor_event_log_dir}/EventLog" -CONDOR_EVENT_LOG_FILE_OLD = "#{condor_event_log_dir}/EventLog.old" -EVENT_LOG_POS_FILE = "#{dbomatic_run_dir}/event_log_position" +# load in the rails models +require File.dirname(__FILE__) + '/../config/environment' + if dbomatic_log_dir == '-' DBOMATIC_LOG_FILE = STDOUT DBOMATIC_PARSER_LOG_FILE = STDOUT @@ -82,276 +82,127 @@ end

# daemonize if daemon - # note that this requires 'active_support', which we get for free from dutils + # note that this requires 'active_support' Process.daemon end

# Custom Log Format class DBomaticLogger < Logger - def format_message(severity, timestamp, progname, msg) - "#{timestamp.to_formatted_s(:db)} #{severity} #{msg}\n" - end -end - -# Handle the event log's xml -class CondorEventLog < Nokogiri::XML::SAX::Document - attr_accessor :tag, :event_type, :event_cmd, :event_time, :trigger_type, :grid_resource, :execute_host, :username, :hold_reason, :private_addresses, :public_addresses + include Singleton

def initialize - @logger = DBomaticLogger.new(DBOMATIC_PARSER_LOG_FILE) - @logger.level = Logger::DEBUG - @logger.info "DBOmatic parser starting up" + super(DBOMATIC_LOG_FILE) end

- # Store the name of the event log attribute we're looking at - def start_element(element, attributes) - # NOTE: this code is written to work both with Nokogiri 1.4.x and 1.5.x - # As soon as we stop supporting 1.4.x, we should change this to: - # @tag = attributes.first[1] if element == "a" - # and remove the rest. - - return unless element == "a" - - # Nokogiri 1.5.x passes attributes as [['name1', 'val1'], ['name2,'val2']] - # while Nokogiri 1.4.x uses a flat array - @tag = (Array === attributes.first) ? attributes.first[1] : attributes[1] - end - - # Store the value of the event log attribute we're looking at - def characters(string) - unless string.strip == "" - if @tag == "MyType" - @event_type = string - elsif @tag == "Cmd" - @event_cmd = string - elsif @tag == "EventTime" - @event_time = string - elsif @tag == "TriggerEventTypeName" - @trigger_type = string - elsif @tag == "GridResource" - @grid_resource = string - elsif @tag == "ExecuteHost" - @execute_host = string - elsif @tag == "DeltacloudUsername" - @username = string - elsif @tag == "HoldReason" - @hold_reason = string - elsif @tag == "DeltacloudPublicNetworkAddresses" - @public_addresses = string - elsif @tag == "DeltacloudPrivateNetworkAddresses" - @private_addresses = string - elsif @tag == "DeltacloudProviderId" - @provider_instance_id = string - end - end + def format_message(severity, timestamp, progname, msg) + "#{timestamp.to_formatted_s(:db)} #{severity} #{msg}\n" end +end

- def update_instance_state_event(inst) - @logger.info "update_instance_state_event for #{inst}" +logger = DBomaticLogger.instance +logger.level = Logger::DEBUG +logger.datetime_format = "%Y-%m-%d %H:%M " # simplify time output +logger.info "DBOmatic starting up"

- if @trigger_type == "ULOG_GRID_SUBMIT" - inst.state = Instance::STATE_PENDING - elsif @trigger_type == "ULOG_JOB_ABORTED" or @trigger_type == "ULOG_JOB_TERMINATED" - inst.state = Instance::STATE_STOPPED - elsif @trigger_type == "ULOG_EXECUTE" - inst.state = Instance::STATE_RUNNING - elsif @trigger_type == "ULOG_SUBMIT" - # ULOG_SUBMIT happens when the job is first submitted to condor. - # However, it's not a state that we care to export to users, but it's - # also not an error, so we just silently ignore it. - elsif @trigger_type == "ULOG_JOB_HELD" - # The job has some error condition. - # - # FIXME: we also may want to delete this job from condor, depending - # on the error. For instance, if you are trying to start an instance - # with a mismatched image and hardwareprofile architecture, the only - # reasonable way out is to create a new instance. Needs thought. - # - I've added a condormatic_instance_reset_error() method to reset - # error conditions such as this. In the future we should give different - # options based on the error condition. - # FIXME: Right now we don't parse out the HoldReason (or HoldReasonCode) - # so for now I'm going to set this to STATE_ERROR as there are multiple - # possible reasons for going into the 'held' state. - # - # FIXME: This only adds the error to the instance 'last_error' field. We - # should really be logging this information into the event log but that is not - # set up at this time so for now this will do. - inst.last_error = @hold_reason - inst.state = Instance::STATE_ERROR - else - @logger.warn "Unexpected trigger type #{@trigger_type}, not updating instance state" - return - end +def self.now + Time.now.strftime('%Y-%m-%d %H:%M:%S') +end

- begin - @logger.info "update_instance_state_event saving instance #{inst}" - inst.save! - @logger.debug "updated_instance_state_event saved instance #{inst}, creating event for state #{inst.state}@#{@event_time}" - inst.events.create!(:status_code => inst.state, - :event_time => @event_time) - rescue Exception => e - @logger.error "#{e.backtrace.shift}: #{e.message}" - e.backtrace.each do |step| - @logger.error "\tfrom #{step}" +def collect_accounts + accounts = [] + Pool.all.each do |pool| + pool.instances.each do |instance| + if instance.provider_account and instance.state != Instance::STATE_NEW and not accounts.include?(instance.provider_account) + accounts << instance.provider_account end end - - @logger.info "update_instance_state_event completed fo #{inst}" - end - - def update_instance_addresses(inst) - @logger.info "update_instance_addresses for #{inst}, \ - setting public addresses: #{@public_addresses} \ - --- and private addresses #{private_addresses}" - - inst.public_addresses = @public_addresses - inst.private_addresses = @private_addresses - inst.save! - @logger.info "update_instance_addresses completed for #{inst}" end + accounts +end

- def update_provider_instance_id(inst) - @logger.info "update_provider_instance_id for #{inst}, \ - setting id: #{@provider_instance_id}" - - inst.provider_instance_id = @provider_instance_id - inst.save! - @logger.info "update_provider_instance_id completed for #{inst}" - end +def check_one_account(account) + connection = account.connect

- # Create a new entry for events which we have all the necessary data for - def end_element(element) - begin - if element == "c" and @event_type == "JobAdInformationEvent" and !@trigger_type.nil? + account.instances.each do |instance| + if instance.state != Instance::STATE_NEW + api_instance = connection.instance(instance.external_key) + if api_instance + DBomaticLogger.instance.debug("updating instance state for #{instance.name}: #{instance.external_key}. #{api_instance}") + instance.state = Taskomatic.dcloud_to_instance_state(api_instance.state)

- inst = Instance.find(:first, :conditions => ['condor_job_id = ?', @event_cmd]) - if inst.nil? - @logger.warn "Unexpected nil instance, skipping..." - else - @logger.info "Instance #{inst} found, running update events" - update_instance_state_event(inst) - update_instance_addresses(inst) - update_provider_instance_id(inst) - @logger.info "Instance #{inst} update events completed" + # only update the public and private addresses if they are not nil. + # this prevents us from deleting known information about instances + if not api_instance.public_addresses.empty? + instance.public_addresses = api_instance.public_addresses end - @tag = @event_type = @event_cmd = @event_time = @trigger_type = @grid_resource = @execute_host = @hold_reason = @public_addresses = @private_addresses = @provider_instance_id = nil - end - rescue Exception => e - @logger.error "#{e.backtrace.shift}: #{e.message}" - e.backtrace.each do |step| - @logger.error "\tfrom #{step}" + if not api_instance.private_addresses.empty? + instance.private_addresses = api_instance.private_addresses + end + + instance.save! + instance.events.create!(:status_code => instance.state, :event_time => now) + else + DBomaticLogger.instance.debug("ignoring unknown instance #{instance.name} #{instance.external_key}") end end end end

-# Retrieve the current parsing position -# of the log file as stored in the log -# file postion tracker -def get_log_file_pos - pos = 0 - if File.exists?(EVENT_LOG_POS_FILE) - File.open(EVENT_LOG_POS_FILE, 'r') { |f| pos = f.read.to_i } +def refresh_instances + accounts = collect_accounts + + # the idea here is that we fork off one process for each provider account. + # this is so that one slow or non-responsive deltacloud doesn't hold up the + # status for all of them + pids = [] + accounts.each do |account| + pid = Process.fork + if pid.nil? + # child + check_one_account(account) + Kernel.exit! + else + # parent + pids << pid + end end - pos -end - -# Set the current parsing position -# of the log file in the log file -# position tracker -def set_log_file_pos(pos) - File.open(EVENT_LOG_POS_FILE, 'w') { |f| f.write pos.to_s } -end

-# FIXME we should make sure everything here is done atomically -def parse_log_file(parser) - # since the actual log file may be rotated out - # open a new handle every time we want to parse - log_file = File.open(CONDOR_EVENT_LOG_FILE) - - # persistantly store log position in filesystem - # incase of dbomatic restarts - log_file.pos = get_log_file_pos - - # if the log has been rotated out - # FIXME probably can use a better check for this, like tracking - # the modification times of these files. Also what happens if condor - # rotates the logs multiple times b4 this executes (is this possible?) - if log_file.pos > File.size(CONDOR_EVENT_LOG_FILE) - if File.exists?(CONDOR_EVENT_LOG_FILE_OLD) - # finish parsing old log file - old_log_file = File.open(CONDOR_EVENT_LOG_FILE_OLD) - old_log_file.pos = log_file.pos - while s = old_log_file.gets - parser << s - end + # only the parent gets here, as all children exit above + start = Time.now.to_i + while not pids.empty? + pid = Process.wait(pid=-1, flags=Process::WNOHANG) + if not pid.nil? + pids.delete(pid) + next end

- # reset position - log_file.pos = 0 - end + # before sleeping, see if we have exceeded the timeout. If so, kill off + # all remaining children and just hope for the best next time around + if (Time.now.to_i - start) > $deltacloud_timeout + DBomaticLogger.instance.warn "Connection timeout of #{$deltacloud_timeout} seconds exceeded, backend clouds could not be contacted. Will try again in #{$dbomatic_timeout} seconds" + pids.each {|pid| Process.kill(9, pid)} + end

- while s = log_file.gets - parser << s + sleep 1 end - - set_log_file_pos(log_file.pos) end

-logger = DBomaticLogger.new(DBOMATIC_LOG_FILE) -logger.level = Logger::DEBUG -logger.datetime_format = "%Y-%m-%d %H:%M " # simplify time output -logger.info "DBOmatic starting up" - -begin - DBOMATIC_PID_FILE = "#{dbomatic_pid_dir}/dbomatic.pid" - FileUtils.mkdir_p File.dirname(DBOMATIC_PID_FILE) - open(DBOMATIC_PID_FILE, "w") {|f| f.write(Process.pid) } - File.chmod(0644, DBOMATIC_PID_FILE) - - parser = Nokogiri::XML::SAX::PushParser.new CondorEventLog.new - - # XXX hack, condor event log doesn't seem to have a top level element - # enclosing everything else in the doc (as standards conforming xml must). - # Create one for parsing purposes. - parser << "<events>" - - parse_log_file(parser) if File.exists? CONDOR_EVENT_LOG_FILE - logger.info "Parsed existing event log file - current postition: #{get_log_file_pos}" - - # hacky fix for BZ #712316 (better dbomatic implementation in progress) - # wait until condor event log file exists - sleep 2 until File.exist?(CONDOR_EVENT_LOG_FILE) - - # track the modification time of the file for changes - logfile_timestamp = File.mtime(CONDOR_EVENT_LOG_FILE) - - logger.info "Beginning main event loop" - while true - sleep 2 - logger.debug "Checking for condor event log modification" - if File.mtime(CONDOR_EVENT_LOG_FILE) > logfile_timestamp - logger.info "EventLog modification event triggered, parsing log" - begin - logfile_timestamp = File.mtime(CONDOR_EVENT_LOG_FILE) - parse_log_file parser - rescue Exception => e - logger.error "Parser error: #{e.backtrace.shift}: #{e.message}" - e.backtrace.each do |step| - logger.error "\tfrom #{step}" - end - end - logger.info "EventLog modification event trigger completed, parsing finished - current position #{get_log_file_pos}" +DBOMATIC_PID_FILE = "#{dbomatic_pid_dir}/dbomatic.pid" +FileUtils.mkdir_p File.dirname(DBOMATIC_PID_FILE) +open(DBOMATIC_PID_FILE, "w") {|f| f.write(Process.pid) } +File.chmod(0644, DBOMATIC_PID_FILE) + +logger.info "Beginning main event loop" +while true + logger.debug "Deltacloud instances refresh started" + begin + refresh_instances + rescue Exception => e + logger.error "#{e.backtrace.shift}: #{e.message}" + e.backtrace.each do |step| + logger.error "\tfrom #{step}" end end - logger.info "Main event loop completed" - - parser << "</events>" - parser.finish - logger.info "Finished parsing, now exiting" -rescue Exception => e - logger.error "#{e.backtrace.shift}: #{e.message}" - e.backtrace.each do |step| - logger.error "\tfrom #{step}" - end + logger.debug "Deltacloud instances refresh completed" + sleep $dbomatic_timeout end diff --git a/src/dutils/active_record_env.rb b/src/dutils/active_record_env.rb deleted file mode 100644 index cee3b62..0000000 --- a/src/dutils/active_record_env.rb +++ /dev/null @@ -1,38 +0,0 @@ -# -# Copyright (C) 2008 Red Hat, Inc. -# Written by Scott Seago sseago@redhat.com -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; version 2 of the License. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, -# MA 02110-1301, USA. A copy of the GNU General Public License is -# also available at http://www.gnu.org/copyleft/gpl.html. - -$: << File.join(File.dirname(__FILE__), "../app") -$: << File.join(File.dirname(__FILE__), "..") - -require 'rubygems' - -$LOAD_PATH << File.expand_path(File.dirname(__FILE__)) - -ENV['RAILS_ENV'] = 'development' unless ENV['RAILS_ENV'] - -require File.dirname(__FILE__) + '/../config/boot' -require File.dirname(__FILE__) + '/../config/environment' - -def database_connect - conf = YAML::load(File.open(File.dirname(__FILE__) + '/../config/database.yml')) - ActiveRecord::Base.establish_connection(conf[ENV['RAILS_ENV']]) -end - -# Open ActiveRecord connection -database_connect diff --git a/src/dutils/dutils.rb b/src/dutils/dutils.rb deleted file mode 100644 index c35aeed..0000000 --- a/src/dutils/dutils.rb +++ /dev/null @@ -1,20 +0,0 @@ -# Copyright (C) 2008 Red Hat, Inc. -# Written by Chris Lalancette clalance@redhat.com -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; version 2 of the License. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, -# MA 02110-1301, USA. A copy of the GNU General Public License is -# also available at http://www.gnu.org/copyleft/gpl.html. - -require 'active_record_env' - diff --git a/util/check_services b/util/check_services index 8cd51e8..1964487 100755 --- a/util/check_services +++ b/util/check_services @@ -1,5 +1,5 @@ #!/usr/bin/ruby -init_scripts=%w(aeolus-conductor condor deltacloud-core deltacloud-ec2-us-east-1 deltacloud-ec2-us-west-1 deltacloud-mock httpd imagefactory iwhd libvirtd mongod ntpd postgresql qpidd) +init_scripts=%w(aeolus-conductor deltacloud-core deltacloud-ec2-us-east-1 deltacloud-ec2-us-west-1 deltacloud-mock httpd imagefactory iwhd libvirtd mongod ntpd postgresql qpidd)

init_scripts.each do |script| puts "\nChecking #{script} ..." @@ -11,19 +11,3 @@ init_scripts.each do |script| puts " \e[1;31mFAILURE:\e[0m #{out.strip}" end end - -# Other checks -commands = [ - {:name => 'condor_q', :command => 'condor_q'}, - {:name => 'condor_status', :command => 'condor_status'} -] -commands.each do |cmd| - puts "\nChecking #{cmd[:name]} ..." - cmd = "#{cmd[:command]}" - out = `#{cmd}` - if $?.to_i == 0 - puts " \e[1;32mSuccess:\e[0m #{out.strip}" - else - puts " \e[1;31mFAILURE:\e[0m #{out.strip}" - end -end

Signed-off-by: Chris Lalancette clalance@redhat.com --- bin/aeolus-check-services | 18 +----------------- recipes/aeolus/manifests/conductor.pp | 21 ++------------------- spec/spec_helper.rb | 2 +- 3 files changed, 4 insertions(+), 37 deletions(-)

diff --git a/bin/aeolus-check-services b/bin/aeolus-check-services index 8cd51e8..1964487 100755 --- a/bin/aeolus-check-services +++ b/bin/aeolus-check-services @@ -1,5 +1,5 @@ #!/usr/bin/ruby -init_scripts=%w(aeolus-conductor condor deltacloud-core deltacloud-ec2-us-east-1 deltacloud-ec2-us-west-1 deltacloud-mock httpd imagefactory iwhd libvirtd mongod ntpd postgresql qpidd) +init_scripts=%w(aeolus-conductor deltacloud-core deltacloud-ec2-us-east-1 deltacloud-ec2-us-west-1 deltacloud-mock httpd imagefactory iwhd libvirtd mongod ntpd postgresql qpidd)

init_scripts.each do |script| puts "\nChecking #{script} ..." @@ -11,19 +11,3 @@ init_scripts.each do |script| puts " \e[1;31mFAILURE:\e[0m #{out.strip}" end end - -# Other checks -commands = [ - {:name => 'condor_q', :command => 'condor_q'}, - {:name => 'condor_status', :command => 'condor_status'} -] -commands.each do |cmd| - puts "\nChecking #{cmd[:name]} ..." - cmd = "#{cmd[:command]}" - out = `#{cmd}` - if $?.to_i == 0 - puts " \e[1;32mSuccess:\e[0m #{out.strip}" - else - puts " \e[1;31mFAILURE:\e[0m #{out.strip}" - end -end diff --git a/recipes/aeolus/manifests/conductor.pp b/recipes/aeolus/manifests/conductor.pp index aded6b3..36b4a8a 100644 --- a/recipes/aeolus/manifests/conductor.pp +++ b/recipes/aeolus/manifests/conductor.pp @@ -5,7 +5,6 @@ class aeolus::conductor inherits aeolus { # specific versions of these two packages are needed and we need to pull the third in package {['aeolus-conductor', 'aeolus-conductor-daemons', - 'condor', 'aeolus-all']: ensure => 'installed', provider => $package_provider } @@ -19,22 +18,6 @@ class aeolus::conductor inherits aeolus { ### Setup selinux for deltacloud selinux::mode{"permissive":}

- ### Start the aeolus services - file {"/etc/condor/config.d/10deltacloud.config": - source => "puppet:///modules/aeolus/condor_config.local", - require => Package['aeolus-conductor-daemons', 'condor'] } - # condor requires an explicit non-localhost hostname - # TODO we can also kill the configure sequence here instead - exec{"/bin/echo 'hostname/domain should be explicitly set and should not be localhost.localdomain'": - logoutput => true, - onlyif => "/usr/bin/test `/bin/hostname` = 'localhost.localdomain'" - } - service { ['condor']: - ensure => 'running', - enable => true, - hasstatus => true, - require => File['/etc/condor/config.d/10deltacloud.config'] } - ### Setup apache for deltacloud include apache if $enable_https { @@ -50,7 +33,7 @@ class aeolus::conductor inherits aeolus { hasstatus => true, require => [Package['aeolus-conductor-daemons'], Rails::Migrate::Db[migrate_aeolus_database], - Service['condor', 'httpd'], + Service['httpd'], Apache::Site[aeolus-conductor], Exec[reload-apache]] }

### Initialize and start the aeolus database @@ -130,7 +113,7 @@ class aeolus::conductor::disabled { }

### Stop the aeolus services - service { ['condor', 'httpd']: + service { ['httpd']: ensure => 'stopped', enable => false, require => Service['aeolus-conductor', diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index 4332150..adf9aa8 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -17,7 +17,7 @@ AEOLUS_DEPENDENCY_PACKAGES = ['rubygem-aws', 'curl', 'java-1.6.0-openjdk', 'open 'appliance-tools', 'livecd-tools', 'python-imgcreate']

# TODO want to include httpd, qpidd here as well but that requires elevated permissions -AEOLUS_DEPENDENCY_SERVICES = ['mongod', 'condor', 'sshd', 'postgresql'] +AEOLUS_DEPENDENCY_SERVICES = ['mongod', 'sshd', 'postgresql']

IWHD_URI='http://localhost:9090/'