This patch is a first try at using condor as a job management system.
This removes the usage of the 'taskomatic' utilities and replaces them
with 'condormatic' calls that use the command line interfaces (no qmf or
gsoap etc) to condor.
On startup of the server (and any changes after running), a pile of
'classads' are created which define each possible startup location for
a given set of image/hardware profiles that exist and are useable as
well as the backend info condor needs to start an instance on the given
provider.
For each instance that you start, a job will be created in condor.
Condor will then match the hardware profile and image to a provider and
can then start an instance on that provider. When you stop or destroy
that instance, the job will be removed (which isn't really how we want
it to go but..).
This patch requires that you have our custom hacked up condor installed.
You can get this at:
http://people.redhat.com/clalance/condor-dcloud
Be sure to read the README. Chris has written up very good instructions
on how to set up condor.
REVISED: This patch plus the new condor fixes a number of the bugs that
were in the previous patch. This patch adds realm matching support and
fixes the start/stop issues we were seeing. So most things basically
work now and I think it's generally useable. Probably the biggest
outstanding bug for useability is that we do not keep long-running jobs
for stateful instances.
The outstanding bugs are now limited to:
- To 'stop' a job in condor we should be using 'hold' instead of
removing the job. This is creating a few different problems.
- We are still reaching directly to the DeltaCloud API to get a list of
available actions for each instance. Maybe this is fine, I'm not
sure.
- Quotas are not yet implemented.
- Classads are sync'd to condor on startup and on any changes to the
hardware profile and image records. However, if you restart condor
you won't have any classads in it to match against and your jobs will
fail.
- We're still using 'on-demand' syncing of states from condor to the
aggregator. eg when you list the instances it updates the states of
each instance from condor at that time. There is no event logging.
- There's no 'reboot' as yet in condor. Not sure how we'll deal with
that just yet.
- We've kept the tasks model and usage but they are quazi-meaningless.
The task table needs to turn into an event/audit log table.
Signed-off-by: Ian Main <imain(a)redhat.com>
---
src/app/controllers/cloud_accounts_controller.rb | 1 +
src/app/controllers/instance_controller.rb | 17 +-
src/app/controllers/pool_controller.rb | 6 +-
src/app/controllers/provider_controller.rb | 1 +
src/app/util/condormatic.rb | 240 +++++++++++++++++++++
src/config/environment.rb | 12 +-
src/db/migrate/20090804142049_create_instances.rb | 1 +
7 files changed, 270 insertions(+), 8 deletions(-)
create mode 100644 src/app/util/condormatic.rb
diff --git a/src/app/controllers/cloud_accounts_controller.rb
b/src/app/controllers/cloud_accounts_controller.rb
index 96eb9af..9343f62 100644
--- a/src/app/controllers/cloud_accounts_controller.rb
+++ b/src/app/controllers/cloud_accounts_controller.rb
@@ -45,6 +45,7 @@ class CloudAccountsController < ApplicationController
else
render :action => "new"
end
+ condormatic_classads_sync
end
def edit
diff --git a/src/app/controllers/instance_controller.rb
b/src/app/controllers/instance_controller.rb
index 039ed3a..5664ec5 100644
--- a/src/app/controllers/instance_controller.rb
+++ b/src/app/controllers/instance_controller.rb
@@ -19,7 +19,7 @@
# Filters added to this controller apply to all controllers in the application.
# Likewise, all the methods added will be available for all controllers.
-require 'util/taskomatic'
+require 'util/condormatic'
class InstanceController < ApplicationController
before_filter :require_user
@@ -96,8 +96,7 @@ class InstanceController < ApplicationController
:task_target => @instance,
:action => InstanceTask::ACTION_CREATE})
if @task.save
- task_impl = Taskomatic.new(@task,logger)
- task_impl.instance_create
+ condormatic_instance_create(@task)
flash[:notice] = "Instance added."
redirect_to :controller => "pool", :action => 'show', :id
=> @instance.pool_id
else
@@ -124,8 +123,16 @@ class InstanceController < ApplicationController
raise ActionError.new("#{action} cannot be performed on this instance.")
end
- task_impl = Taskomatic.new(@task,logger)
- task_impl.send "instance_#{action}"
+ case action
+ when 'stop'
+ condormatic_instance_stop(@task)
+ when 'destroy'
+ condormatic_instance_destroy(@task)
+ when 'start'
+ condormatic_instance_create(@task)
+ else
+ raise ActionError.new("Sorry, action '#{action}' is currently not
supported by condor backend.")
+ end
alert = "#{(a)instance.name}: #{action} was successfully queued."
flash[:notice] = alert
diff --git a/src/app/controllers/pool_controller.rb
b/src/app/controllers/pool_controller.rb
index e687c0b..ad2d42b 100644
--- a/src/app/controllers/pool_controller.rb
+++ b/src/app/controllers/pool_controller.rb
@@ -20,6 +20,7 @@
# Likewise, all the methods added will be available for all controllers.
require 'util/taskomatic'
+require 'util/condormatic'
class PoolController < ApplicationController
before_filter :require_user
@@ -36,8 +37,8 @@ class PoolController < ApplicationController
#FIXME: clean this up, many error cases here
@pool = Pool.find(params[:id])
require_privilege(Privilege::INSTANCE_VIEW,@pool)
- # pass nil into Taskomatic as we're not working off a task here
- Taskomatic.new(nil,logger).pool_refresh(@pool)
+ # Go to condor and sync the database to the real instance states
+ condormatic_instances_sync_states
@pool.reload
@instances = @pool.instances
end
@@ -153,4 +154,5 @@ class PoolController < ApplicationController
end
redirect_to :action => 'show', :id => @pool.id
end
+ condormatic_classads_sync
end
diff --git a/src/app/controllers/provider_controller.rb
b/src/app/controllers/provider_controller.rb
index 8ba84fe..59c9393 100644
--- a/src/app/controllers/provider_controller.rb
+++ b/src/app/controllers/provider_controller.rb
@@ -39,6 +39,7 @@ class ProviderController < ApplicationController
flash[:notice] = "Provider added."
redirect_to :action => "show", :id => @provider
end
+ condormatic_classads_sync
end
def destroy
diff --git a/src/app/util/condormatic.rb b/src/app/util/condormatic.rb
new file mode 100644
index 0000000..e994504
--- /dev/null
+++ b/src/app/util/condormatic.rb
@@ -0,0 +1,240 @@
+#
+# Copyright (C) 2010 Red Hat, Inc.
+# Written by Ian Main <imain(a)redhat.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA 02110-1301, USA. A copy of the GNU General Public License is
+# also available at
http://www.gnu.org/copyleft/gpl.html.
+
+def condormatic_instance_create(task)
+
+ begin
+ instance = task.instance
+ realm = instance.realm rescue nil
+
+ job_name = "job_#{instance.name}_#{instance.id}"
+
+
+ # I use the 2>&1 to get stderr and stdout together because popen3 does not
support
+ # the ability to get the exit value of the command in ruby 1.8.
+ pipe = IO.popen("condor_submit 2>&1", "w+")
+ pipe.puts "universe = grid\n"
+ Rails.logger.info "universe = grid\n"
+ pipe.puts "executable = #{job_name}\n"
+ Rails.logger.info "executable = #{job_name}\n"
+ pipe.puts "grid_resource = dcloud $$(provider_url) $$(username) $$(password)
$$(image_key) #{instance.name} $$(realm_key) $$(hardwareprofile_key)\n"
+ Rails.logger.info "grid_resource = dcloud $$(provider_url) $$(username)
$$(password) $$(image_key) #{instance.name} $$(realm_key) $$(hardwareprofile_key)\n"
+ pipe.puts "log = #{job_name}.log\n"
+ Rails.logger.info "log = #{job_name}.log\n"
+
+ requirements = "requirements = hardwareprofile ==
\"#{instance.hardware_profile.id}\" && image ==
\"#{instance.image.id}\""
+ requirements += " && realm == \"#{realm.name}\"" if realm
!= nil
+ requirements += "\n"
+
+ pipe.puts requirements
+ Rails.logger.info requirements
+
+ pipe.puts "notification = never\n"
+ Rails.logger.info "notification = never\n"
+ pipe.puts "queue\n"
+ Rails.logger.info "queue\n"
+ pipe.close_write
+ out = pipe.read
+ pipe.close
+
+ Rails.logger.info "$? (return value?) is #{$?}"
+ raise ("Error calling condor_submit: #{out}") if $? != 0
+
+ instance.condor_job_id = job_name
+ instance.save!
+
+ rescue Exception => ex
+ task.state = Task::STATE_FAILED
+ Rails.logger.error ex.message
+ Rails.logger.error ex.backtrace
+ else
+ # FIXME: We're kinda lying here.. we don't know the state for the task but I
don't think that matters so much
+ # as we are just going to use the 'task' table as a kind of audit log.
+ task.state = Task::STATE_PENDING
+ end
+ task.instance.save!
+end
+
+# JobStatus for condor jobs:
+#
+# 0 Unexpanded U
+# 1 Idle I
+# 2 Running R
+# 3 Removed X
+# 4 Completed C
+# 5 Held H
+# 6 Submission_err E
+#
+
+def condor_to_instance_state(state_val)
+ case state_val
+ when '0'
+ return Instance::STATE_PENDING
+ when '1'
+ return Instance::STATE_PENDING
+ when '2'
+ return Instance::STATE_RUNNING
+ when '3'
+ return Instance::STATE_STOPPED
+ when '4'
+ return Instance::STATE_STOPPED
+ when '5'
+ return Instance::STATE_CREATE_FAILED
+ when '6'
+ return Instance::STATE_CREATE_FAILED
+ else
+ return Instance::STATE_PENDING
+ end
+end
+
+def condormatic_instances_sync_states
+
+ begin
+ # I'm not going to do the 2&>1 trick here since we are parsing the output
+ # and I'm afraid we'll get a warning or something on stderr and it'll
mess
+ # up the xml parsing.
+ pipe = IO.popen("condor_q -xml")
+ xml = pipe.read
+ pipe.close
+
+ raise ("Error calling condor_q -xml") if $? != 0
+
+ # Set them all to 'stopped' because if they aren't in the condor
+ # queue as jobs then they are not running, pending or anything else.
+ instances = Instance.find(:all)
+ instances.each do |instance|
+ instance.state = Instance::STATE_STOPPED
+ instance.save!
+ end
+
+ def find_value_int(job_ele, attrib)
+ if job_ele.attributes['n'] == attrib
+ cmd = job_ele.elements.each('i') do |i|
+ return i.text
+ end
+ end
+ return nil
+ end
+
+ def find_value_str(job_ele, attrib)
+ if job_ele.attributes['n'] == attrib
+ cmd = job_ele.elements.each('s') do |s|
+ return s.text
+ end
+ end
+ return nil
+ end
+
+ doc = REXML::Document.new(xml)
+ doc.elements.each('classads/c') do |jobs_ele|
+ job_name = nil
+ job_state = nil
+
+ jobs_ele.elements.each('a') do |job_ele|
+ value = find_value_str(job_ele, 'Cmd')
+ job_name = value if value != nil
+ value = find_value_int(job_ele, 'JobStatus')
+ job_state = value if value != nil
+ end
+
+ Rails.logger.info "job name is #{job_name}"
+ Rails.logger.info "job state is #{job_state}"
+
+ instance = Instance.find(:first, :conditions => {:condor_job_id =>
job_name})
+
+ if instance
+ instance.state = condor_to_instance_state(job_state)
+ instance.save!
+ Rails.logger.info "Instance state updated to
#{condor_to_instance_state(job_state)}"
+ end
+ end
+ rescue Exception => ex
+ Rails.logger.error ex.message
+ Rails.logger.error ex.backtrace
+ end
+end
+
+def condormatic_instance_stop(task)
+ instance = task.instance
+
+ Rails.logger.info("calling condor_rm -constraint 'Cmd ==
\"#{instance.condor_job_id}\"' 2>&1")
+ pipe = IO.popen("condor_rm -constraint 'Cmd ==
\"#{instance.condor_job_id}\"' 2>&1")
+ out = pipe.read
+ pipe.close
+
+ Rails.logger.info("condor_rm return status is #{$?}")
+ Rails.logger.error("Error calling condor_rm (exit code #{$?}) on job:
#{out}") if $? != 0
+end
+
+def condormatic_instance_destroy(task)
+ instance = task.instance
+
+ Rails.logger.info("calling condor_rm -constraint 'Cmd ==
\"#{instance.condor_job_id}\"' 2>&1")
+ pipe = IO.popen("condor_rm -constraint 'Cmd ==
\"#{instance.condor_job_id}\"' 2>&1")
+ out = pipe.read
+ pipe.close
+
+ Rails.logger.info("condor_rm return status is #{$?}")
+ Rails.logger.error("Error calling condor_rm (exit code #{$?}) on job:
#{out}") if $? != 0
+end
+
+
+def condormatic_classads_sync
+
+ index = 0
+ providers = Provider.find(:all)
+ Rails.logger.info "Syncing classads.."
+
+ providers.each do |provider|
+ provider.cloud_accounts.each do |account|
+ provider.images.each do |image|
+ provider.hardware_profiles.each do |hwp|
+ provider.realms.each do |realm|
+ pipe = IO.popen("condor_advertise UPDATE_STARTD_AD 2>&1",
"w+")
+
+ pipe.puts "Name=\"provider_combination_#{index}\""
+ pipe.puts 'MyType="Machine"'
+ pipe.puts 'Requirements=true'
+ pipe.puts "\n# Stuff needed to match:"
+ pipe.puts
"hardwareprofile=\"#{hwp.aggregator_hardware_profiles[0].id}\""
+ pipe.puts "image=\"#{image.aggregator_images[0].id}\""
+ pipe.puts "realm=\"#{realm.name}\""
+ pipe.puts "\n# Backend info to complete this job:"
+ pipe.puts "image_key=\"#{image.external_key}\""
+ pipe.puts "hardwareprofile_key=\"#{hwp.external_key}\""
+ pipe.puts "realm_key=\"#{realm.external_key}\""
+ pipe.puts "provider_url=\"#{account.provider.url}\""
+ pipe.puts "username=\"#{account.username}\""
+ pipe.puts "password=\"#{account.password}\""
+ pipe.close_write
+
+ out = pipe.read
+ pipe.close
+
+ Rails.logger.error "Unable to submit condor classad: #{out}" if $?
!= 0
+
+ index += 1
+ end
+ end
+ end
+ end
+
+ Rails.logger.info "done"
+ end
+end
diff --git a/src/config/environment.rb b/src/config/environment.rb
index 919a710..3b49fbc 100644
--- a/src/config/environment.rb
+++ b/src/config/environment.rb
@@ -24,6 +24,7 @@ RAILS_GEM_VERSION = '>= 2.3.2' unless defined?
RAILS_GEM_VERSION
# Bootstrap the Rails environment, frameworks, and default configuration
require File.join(File.dirname(__FILE__), 'boot')
+require 'util/condormatic'
Rails::Initializer.run do |config|
# Settings in config/environments/* take precedence over those specified here
@@ -80,5 +81,14 @@ Rails::Initializer.run do |config|
# config.i18n.load_path += Dir[Rails.root.join('my', 'locales',
'*.{rb,yml}')]
# config.i18n.default_locale = :de
-
+ config.after_initialize do
+ begin
+ # This pulls all the possible classad matches from the database and puts
+ # them on condor on startup. Note that this can fail because this is run on
startup
+ # even for rake db:migrate etc. which won't work since the database doesn't
exist
+ # yet.
+ condormatic_classads_sync
+ rescue Exception => ex
+ end
+ end
end
diff --git a/src/db/migrate/20090804142049_create_instances.rb
b/src/db/migrate/20090804142049_create_instances.rb
index 335b93f..42706e1 100644
--- a/src/db/migrate/20090804142049_create_instances.rb
+++ b/src/db/migrate/20090804142049_create_instances.rb
@@ -32,6 +32,7 @@ class CreateInstances < ActiveRecord::Migration
t.string :public_address
t.string :private_address
t.string :state
+ t.string :condor_job_id
t.integer :lock_version, :default => 0
t.integer :acc_pending_time, :default => 0
t.integer :acc_running_time, :default => 0
--
1.7.0.1