[PATCH conductor] Update Gemfile.lock to fix rake test tasks.
by Jason Guiditta
This should allow a user to run tests from source with:
* rake cucumber
* USE_BUNDLER=yes rake cucumber
* RAILS_ENV=test rake spec
* USE_BUNDLER=yes rake spec
---
src/Gemfile.lock | 17 +++++++++++++----
1 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/src/Gemfile.lock b/src/Gemfile.lock
index 0f3912b..01dfd36 100644
--- a/src/Gemfile.lock
+++ b/src/Gemfile.lock
@@ -28,6 +28,10 @@ GEM
activesupport (= 3.0.9)
activesupport (3.0.9)
addressable (2.2.6)
+ aeolus-image (0.1.0)
+ imagefactory-console (>= 0.4.0)
+ nokogiri (>= 0.4.0)
+ rest-client
arel (2.0.9)
builder (2.1.2)
capybara (1.1.1)
@@ -37,7 +41,7 @@ GEM
rack-test (>= 0.5.4)
selenium-webdriver (~> 2.0)
xpath (~> 0.1.4)
- childprocess (0.2.2)
+ childprocess (0.2.0)
ffi (~> 1.0.6)
crack (0.1.8)
cucumber (1.0.1)
@@ -69,9 +73,9 @@ GEM
json (>= 1.4.6)
haml (3.1.2)
i18n (0.5.0)
+ imagefactory-console (0.5.0)
json (1.4.6)
- json_pure (1.5.4)
- spruz (~> 0.2.8)
+ json_pure (1.4.6)
launchy (2.0.5)
addressable (~> 2.2.6)
mail (2.3.0)
@@ -112,11 +116,16 @@ GEM
rest-client (1.6.1)
mime-types (>= 1.16)
rspec-core (2.6.4)
+ rspec-expectations (2.6.0)
+ diff-lcs (~> 1.1.2)
+ rspec-mocks (2.6.0)
rspec-rails (2.6.1)
actionpack (~> 3.0)
activesupport (~> 3.0)
railties (~> 3.0)
rspec-core (~> 2.6.0)
+ rspec-expectations (~> 2.6.0)
+ rspec-mocks (~> 2.6.0)
ruby-net-ldap (0.0.4)
rubyzip (0.9.4)
sass (3.1.4)
@@ -127,7 +136,6 @@ GEM
rubyzip
simple-navigation (3.0.0)
activesupport (>= 2.3.2)
- spruz (0.2.13)
sqlite3 (1.3.3)
term-ansicolor (1.0.6)
thin (1.2.11)
@@ -155,6 +163,7 @@ PLATFORMS
ruby
DEPENDENCIES
+ aeolus-image
capybara
cucumber
cucumber-rails
--
1.7.6.2
12 years, 6 months
ANNOUNCE: oz 0.7.0 release
by Chris Lalancette
All,
I'm pleased to announce release 0.7.0 of Oz. Oz is a program for doing
automated installation of guest operating systems with limited input from
the user.
Release 0.7.0 is a bugfix and feature release for Oz. Some of the
highlights between Oz 0.6.0 and 0.7.0 are:
- Ability to use the "direct initrd injection" method to install Fedora/RHEL
guests. This is an internal implementation detail, but can significantly speed
up installs for Fedora or RHEL guests. (thanks for the tip from
Kashyap Chamarthy)
- Support for Fedora-16 (thanks to Steve Dake for help in making this work)
- Use the serial port to announce guest boot, rather than a network port. This
makes it so we no longer have to manipulate iptables, and gets us one step
closer to having Oz run as non-root
- (for developers) Re-written unittests in python for speedier execution
- (for developers) Additional methods in the TDL class to merge in external
package lists (thanks to Ian McLeod)
A tarball of this release is available, as well as packages for Fedora-14,
Fedora-15, and RHEL-6. Note that to install the RHEL-6 packages, you must be
running RHEL-6.1 or later. Instructions on how to get and use Oz are available
at http://aeolusproject.org/oz.html
If you have any questions or comments about Oz, please feel free to contact
aeolus-devel(a)lists.fedorahosted.org or me (clalance(a)redhat.com) directly.
Thanks to everyone who contributed to this release through bug reports,
patches, and suggestions for improvement.
--
Chris Lalancette
12 years, 6 months
[PATCHv4 0/5]: Remove condor from the conductor
by Chris Lalancette
All,
This is a patch series to remove condor from the conductor. In short,
condor presents problems for our project because it is an external project, it
is written in C++ (while most of our developers are ruby), and it is too complex
for our current needs.
The new way we do scheduling is described pretty well in patch 1, so I
won't delve into it here. Since the last posting, there have been a few more
bugs fixed:
1) jprovazn finally found an ActiveRecord incantation that works for
re-establishing a connection to a database, so the dbomatic subprocesses don't
trample over each other
2) I had to fix several bugs when running dbomatic under systemd. I can go
into the gory details if you want, just ping me on IRC :)
3) imain discovered that the broken Download Key links are independent of this
patchset, so we don't have to worry about that.
There are also some outstanding TODO items, but which can be postponed until
later:
1) We are still not properly generating the instance name. In condor, we did
this by generating a UUID that would be the instance name, then truncating that
UUID to the maximum length that deltacloud told us an instance name could be
for a particular backend. I'm thinking we want to do the same here.
2) imain believes we can do away with the Task class completely now. That
would be a nice cleanup
3) When launching large deployments, the "Launch Deployment" page can be slow
to return. That's because in the background we are issuing one deltacloud call
at a time to launch each instance in the deployment. We probably want to fork
off a separate process for each deployment so we don't hold up the UI process.
I've personally tested the patches using the EC2 backend, where things seem to
be working well. I was able to launch 5 deployments for a total of 15
instances simultaneously without issue. I also tested up to a 4 assembly
deployable which seemed to work.
At this point, I'm relatively happy with this patchset and I think we should
commit it barring any major problems found in testing. The enhancements above
can be done as additional patches on top of this.
Please review and ACK.
Chris Lalancette
12 years, 6 months
Justin on leave from now to mid October (back on 17th)
by Justin Clift
Hi all,
I'm on leave from now until mid October, back on 17th (Oct).
My intention is to be completely "off the grid" the entire time. *Specifically* away from _all_ computing/connectivity devices in order to get a proper break, to recharge my personal energy levels. (whoo!) :)
NOTE FOR PEOPLE WITH COMMUNITY BUILDING IDEAS :>
*********************************************
I'm far from a control freak. :> So, if you have an idea for doing something Community related and wanting to know if it's a go-er, consider my opinion to be this:
"Go For It & Get It Done"
Try it out, let people know, see what happens, we'll adjust over time. :>
ALTERNATIVE CONTACTS
********************
For anyone needing Community related stuff which can't be asked about on a public mailing list, the contact people during this time are:
+ David Lutterkort <dlutter@redhatcom> (Deltacloud)
+ Francesco Vollero <fvollero(a)redhat.com> (Aeolus in EMEA), Matt Wagner <matt.wagner(a)redhat.com> (Aeolus in US)
+ Bryan Che <bche(a)redhat.com> (anything needing marketing budget, ie. flyers, promo stuff, printing, etc)
+ /dev/null (flames)
Apart from that, have fun guys, and go take some risks. :)
Regards and best wishes,
Justin Clift
--
Aeolus Community Manager
http://www.aeolusproject.org
12 years, 6 months
Aeolus dependency tree
by Chris Lalancette
Just in case anyone is interested in looking at how crazy our dependency
tree is, I generated one on F-15. The commands are:
yum install rpmorphan
rpmdep -dot aeolus.dot aeolus-all
dot -Tsvg aeolus.dot -o aeolus.svg
The result is at http://people.redhat.com/clalance/aeolus.svg
--
Chris Lalancette
12 years, 6 months
[PATCH] Use a thread to do instance creates V3.
by Ian Main
From: Chris Lalancette <clalance(a)redhat.com>
I modified Chris' patch to use a thread instead of fork. This pushes
the communications with deltacloud into a thread which can take as long
as it needs to complete the start.
In the future we may want to put a retry in here too.
You will notice in V2 we now set allow_concurrency = true in the
postgres config. I'm a little worried that this means it may not work
with other DB providers.
Signed-off-by: Ian Main <imain(a)redhat.com>
---
src/app/util/taskomatic.rb | 81 ++++++++++++++++++++++++++++++-------------
src/config/database.pg | 3 ++
2 files changed, 59 insertions(+), 25 deletions(-)
diff --git a/src/app/util/taskomatic.rb b/src/app/util/taskomatic.rb
index 84dabd0..bc9afb6 100644
--- a/src/app/util/taskomatic.rb
+++ b/src/app/util/taskomatic.rb
@@ -26,27 +26,15 @@ module Taskomatic
task.state = Task::STATE_PENDING
task.save!
- task.instance.provider_account = match.provider_account
- task.instance.create_auth_key unless task.instance.instance_key
-
- dcloud_instance = create_dcloud_instance(task.instance, match)
-
- handle_dcloud_error(dcloud_instance)
+ create_dcloud_instance(task.instance, match)
task.state = Task::STATE_RUNNING
- task.save!
-
- Rails.logger.info "Task instance create completed with key #{dcloud_instance.id} and state #{dcloud_instance.state}"
- task.instance.external_key = dcloud_instance.id
- task.instance.state = dcloud_to_instance_state(dcloud_instance.state)
- task.instance.save!
rescue HttpException => ex
task.failure_code = Task::FAILURE_PROVIDER_CONTACT_FAILED
handle_create_instance_error(task, ex)
rescue Exception => ex
handle_create_instance_error(task, ex)
ensure
- task.instance.save!
task.save!
end
end
@@ -146,18 +134,61 @@ module Taskomatic
end
def self.create_dcloud_instance(instance, match)
- client = match.provider_account.connect
-
- overrides = HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp)
-
- client.create_instance(:image_id => match.provider_image.target_identifier,
- :name => instance.name.tr("/", "-"),
- :hwp_id => match.hwp.external_key,
- :hwp_memory => overrides[:memory],
- :hwp_cpu => overrides[:cpu],
- :hwp_storage => overrides[:storage],
- :realm_id => (match.realm.external_key rescue nil),
- :keyname => (instance.instance_key.name))
+ # because creating an instance can take a potentially long time (and
+ # creating multiple of them just prolongs this), we start a new thread
+ # that does this work. The new thread continues to run in the background
+ # and communicates the status back to the main UI via the database
+
+ # This cleans up old DB connections. AR creates a new connection for each
+ # thread. This can leak FDs if you don't clean them up. Basically this
+ # ensures the previously used FD is cleaned up.
+ ActiveRecord::Base.connection_pool.clear_stale_cached_connections!
+
+ Thread.new do
+ begin
+ # These all need to be reloaded because when you create a new thread
+ # AR creates a new connection and these objects become stale.
+ #
+ # Everything used here must be reloaded.
+ instance.reload
+ match.provider_account.reload
+ match.hwp.reload
+ match.realm.reload if match.realm
+
+ instance.provider_account = match.provider_account
+ instance.create_auth_key unless instance.instance_key
+
+ client = match.provider_account.connect
+
+ overrides = HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp)
+
+ dcloud_instance = client.create_instance(:image_id => match.provider_image.target_identifier,
+ :name => instance.name.tr("/", "-"),
+ :hwp_id => match.hwp.external_key,
+ :hwp_memory => overrides[:memory],
+ :hwp_cpu => overrides[:cpu],
+ :hwp_storage => overrides[:storage],
+ :realm_id => (match.realm.external_key rescue nil),
+ :keyname => (instance.instance_key.name))
+
+ handle_dcloud_error(dcloud_instance)
+
+ Rails.logger.info "Task instance create completed with key #{dcloud_instance.id} and state #{dcloud_instance.state}"
+ instance.external_key = dcloud_instance.id
+ instance.state = dcloud_to_instance_state(dcloud_instance.state)
+ rescue Exception => ex
+ # any sort of exception causes us to put the instance in CREATE_FAILED
+ # FIXME: if the exception is raised *after* the create_instance, this
+ # isn't true and will result in a rogue instance
+ Rails.logger.error ex.message
+ Rails.logger.error ex.backtrace.join("\n")
+ instance.state = Instance::STATE_CREATE_FAILED
+ raise ex
+ ensure
+ instance.save!
+ Thread.exit
+ end
+ end
end
def self.matches(instance)
diff --git a/src/config/database.pg b/src/config/database.pg
index 19fe4e6..72a3895 100644
--- a/src/config/database.pg
+++ b/src/config/database.pg
@@ -45,6 +45,7 @@ development:
username: aeolus
password: v23zj59an
host: localhost
+ allow_concurrency: true
min_messages: warning
# Warning: The database defined as 'test' will be erased and
@@ -56,6 +57,7 @@ test: &TEST
username: aeolus
password: v23zj59an
host: localhost
+ allow_concurrency: true
min_messages: warning
production:
@@ -63,6 +65,7 @@ production:
database: conductor
username: aeolus
password: v23zj59an
+ allow_concurrency: true
host: localhost
cucumber:
--
1.7.6.2
12 years, 6 months
[PATCH] Use a thread to do instance creates V2.
by Ian Main
From: Chris Lalancette <clalance(a)redhat.com>
I modified Chris' patch to use a thread instead of fork. This pushes
the communications with deltacloud into a thread which can take as long
as it needs to complete the start.
In the future we may want to put a retry in here too.
You will notice in V2 we now set allow_concurrency = true in the
postgres config. I'm a little worried that this means it may not work
with other DB providers.
Signed-off-by: Ian Main <imain(a)redhat.com>
---
src/app/util/taskomatic.rb | 67 +++++++++++++++++++++++++++----------------
1 files changed, 42 insertions(+), 25 deletions(-)
diff --git a/src/app/util/taskomatic.rb b/src/app/util/taskomatic.rb
index 84dabd0..eae459a 100644
--- a/src/app/util/taskomatic.rb
+++ b/src/app/util/taskomatic.rb
@@ -26,27 +26,15 @@ module Taskomatic
task.state = Task::STATE_PENDING
task.save!
- task.instance.provider_account = match.provider_account
- task.instance.create_auth_key unless task.instance.instance_key
-
- dcloud_instance = create_dcloud_instance(task.instance, match)
-
- handle_dcloud_error(dcloud_instance)
+ create_dcloud_instance(task.instance, match)
task.state = Task::STATE_RUNNING
- task.save!
-
- Rails.logger.info "Task instance create completed with key #{dcloud_instance.id} and state #{dcloud_instance.state}"
- task.instance.external_key = dcloud_instance.id
- task.instance.state = dcloud_to_instance_state(dcloud_instance.state)
- task.instance.save!
rescue HttpException => ex
task.failure_code = Task::FAILURE_PROVIDER_CONTACT_FAILED
handle_create_instance_error(task, ex)
rescue Exception => ex
handle_create_instance_error(task, ex)
ensure
- task.instance.save!
task.save!
end
end
@@ -146,18 +134,47 @@ module Taskomatic
end
def self.create_dcloud_instance(instance, match)
- client = match.provider_account.connect
-
- overrides = HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp)
-
- client.create_instance(:image_id => match.provider_image.target_identifier,
- :name => instance.name.tr("/", "-"),
- :hwp_id => match.hwp.external_key,
- :hwp_memory => overrides[:memory],
- :hwp_cpu => overrides[:cpu],
- :hwp_storage => overrides[:storage],
- :realm_id => (match.realm.external_key rescue nil),
- :keyname => (instance.instance_key.name))
+ # because creating an instance can take a potentially long time (and
+ # creating multiple of them just prolongs this), we start a new thread
+ # that does this work. The new thread continues to run in the background
+ # and communicates the status back to the main UI via the database
+
+ Thread.new do
+ begin
+ instance.provider_account = match.provider_account
+ instance.create_auth_key unless instance.instance_key
+
+ client = match.provider_account.connect
+
+ overrides = HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp)
+
+ dcloud_instance = client.create_instance(:image_id => match.provider_image.target_identifier,
+ :name => instance.name.tr("/", "-"),
+ :hwp_id => match.hwp.external_key,
+ :hwp_memory => overrides[:memory],
+ :hwp_cpu => overrides[:cpu],
+ :hwp_storage => overrides[:storage],
+ :realm_id => (match.realm.external_key rescue nil),
+ :keyname => (instance.instance_key.name))
+
+ handle_dcloud_error(dcloud_instance)
+
+ Rails.logger.info "Task instance create completed with key #{dcloud_instance.id} and state #{dcloud_instance.state}"
+ instance.external_key = dcloud_instance.id
+ instance.state = dcloud_to_instance_state(dcloud_instance.state)
+ rescue Exception => ex
+ # any sort of exception causes us to put the instance in CREATE_FAILED
+ # FIXME: if the exception is raised *after* the create_instance, this
+ # isn't true and will result in a rogue instance
+ Rails.logger.error ex.message
+ Rails.logger.error ex.backtrace.join("\n")
+ instance.state = Instance::STATE_CREATE_FAILED
+ raise ex
+ ensure
+ instance.save!
+ Thread.exit
+ end
+ end
end
def self.matches(instance)
--
1.7.6.2
12 years, 6 months
iwhd security
by Pete Zaitcev
In its present form iwhd has no access controls whatsoever, and it
concerns me. The memcached is like that too, and it is deployed successfully
on many high-traffic, high-profile sites. However, iwhd implies remote
access, which makes the current set-up more hazardous. I would like to
consider options from an attack by rooted systems on the internal
networks, and less privileged insiders.
A short while ago, there was some effort to underpin the whole cloud
with IPA. From that time, a patch exists for iwhd, which implements
Kerberos authentication. There is no authorization in it, however:
anyone who is able to authenticate with Kerberos is considered
authorized. I'm wondering if adding this to mainline iwhd would
help anything. Do Conductor and Factory have an HTTP-Negotiate
client support?
Failing that, I'd like to know if anyone has any suggestions or
requests. For example, would adding an HTTP-Basic (over SSL or not)
help anything?
-- Pete
12 years, 6 months
Need new JEOS build for oz/factory
by wes hayutin
Due to some recent changes w/ RHUI the JEOS images for aeolus need to be
respun or use a new ami. Until then you will get the following when
building/pushing ec2.
'terminated'. [290 of 300 seconds elapsed]
2011-09-22 13:31:21,235 DEBUG
imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(14690) Message:
Instance status (stopped) - waiting for 'terminated'. [295 of 300
seconds elapsed]
2011-09-22 13:31:26,238 DEBUG
imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(14690) Message:
Exception caught in ImageFactory
2011-09-22 13:31:26,301 DEBUG
imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(14690) Message:
Traceback (most recent call last):
File
"/usr/lib/python2.7/site-packages/imgfac/builders/Fedora_ec2_Builder.py", line 425, in push_image
credentials)
File
"/usr/lib/python2.7/site-packages/imgfac/builders/Fedora_ec2_Builder.py", line 581, in push_image_snapshot_ec2
self.install_euca_tools(guestaddr)
File
"/usr/lib/python2.7/site-packages/imgfac/builders/RHEL6_ec2_Builder.py",
line 45, in install_euca_tools
self.guest.guest_execute_command(guestaddr, "yum -y install
euca2ools")
File "/usr/lib/python2.7/site-packages/oz/RedHat.py", line 463, in
guest_execute_command
command, timeout)
File "/usr/lib/python2.7/site-packages/oz/ozutil.py", line 332, in
ssh_execute_command
"root@" + guestaddr, command])
File "/usr/lib/python2.7/site-packages/oz/ozutil.py", line 310, in
subprocess_check_output
raise SubprocessException("'%s' failed(%d): %s" % (cmd, retcode,
stderr), retcode)
SubprocessException: 'ssh -i /tmp/tmpxYCJQd -F /dev/null -o
ServerAliveInterval=30 -o StrictHostKeyChecking=no -o ConnectTimeout=10
-o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no
root(a)ec2-50-17-23-173.compute-1.amazonaws.com yum -y install euca2ools'
failed(1): Warning: Permanently added
'ec2-50-17-23-173.compute-1.amazonaws.com,50.17.23.173' (RSA) to the
list of known hosts.
https://rhui.us-east-1.aws.ce.redhat.com/rhuilb/content/dist/rhel/rhui/se...: [Errno 14] Peer cert cannot be verified or peer cert invalid
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository:
rhui-us-east-rhel-server-releases. Please verify its path and try again
2011-09-22 13:31:26,301 DEBUG imgfac.BuildJob.BuildAdaptor pid(14690)
Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1,
initial)>), changed status from PUSHING to FAILED
12 years, 6 months
How to fix aeolus-image slowness
by Matt Wagner
Hi all,
There's been some discussion about the fact that "aeolus-image list
--images" is extremely slow. It's fine with a handful of images, but the
time required grows exponentially, so that by perhaps 25 images, the
command takes minutes or hours to run.
The problem is that we tried to implement associations like they existed
natively -- an Image has many Builds. But since iwhd stores this
information backwards from how we want it, we need to load ALL builds
and then match the one with our image_id in it.
Where this gets outrageously inefficient is for things like the
"aeolus-image list --images" use case, where we have to walk all
associations many times. (Although in theory we're listing "images," the
reality is that we need several other data types to get all the
requisite data. Otherwise we're just listing UUIDs.) It ends up that,
for each image, we load all builds, and, for each of those builds, we
load all target images, and for each of those, we load all provider
images. This is fine for small sets, but the number of requests grows
exponentially and becomes absurdly impractical very quickly.
The "real" fix that we've kicked around is to find a new way to store
this information. And we should do that at some point. But it's clear
that this isn't going to happen soon, and I think we need to take some
action now.
Here's what I propose: if we know we're going to have to traverse any
associations, load all the objects of that type into memory for the
duration of the request, and do the matching in memory.
So for "aeolus-image list --images", we would instead load all images,
builds, target images, and provider images once, and then do all of the
mapping in memory. This can save a few orders of magnitude of API calls,
and, because it's only for the duration of the request, doesn't leave us
caching stale data.
I haven't fully thought through how to implement this yet, as we don't
want to load these associations if they're not needed at all. But as
soon as we know we're going to want an association, we should load all
of them into memory.
This isn't a perfect plan by any stretch, but I think it makes things
substantially less-bad, and should serve at least as a stop-gap.
What do you all think of this approach?
Best,
Matt
12 years, 6 months