aeolus-devel September 2011

aeolus-devel@lists.fedorahosted.org

47 participants
201 discussions

[PATCH conductor] Update Gemfile.lock to fix rake test tasks.
by Jason Guiditta 23 Sep '11

23 Sep '11

This should allow a user to run tests from source with: * rake cucumber * USE_BUNDLER=yes rake cucumber * RAILS_ENV=test rake spec * USE_BUNDLER=yes rake spec --- src/Gemfile.lock | 17 +++++++++++++---- 1 files changed, 13 insertions(+), 4 deletions(-) diff --git a/src/Gemfile.lock b/src/Gemfile.lock index 0f3912b..01dfd36 100644 --- a/src/Gemfile.lock +++ b/src/Gemfile.lock @@ -28,6 +28,10 @@ GEM activesupport (= 3.0.9) activesupport (3.0.9) addressable (2.2.6) + aeolus-image (0.1.0) + imagefactory-console (>= 0.4.0) + nokogiri (>= 0.4.0) + rest-client arel (2.0.9) builder (2.1.2) capybara (1.1.1) @@ -37,7 +41,7 @@ GEM rack-test (>= 0.5.4) selenium-webdriver (~> 2.0) xpath (~> 0.1.4) - childprocess (0.2.2) + childprocess (0.2.0) ffi (~> 1.0.6) crack (0.1.8) cucumber (1.0.1) @@ -69,9 +73,9 @@ GEM json (>= 1.4.6) haml (3.1.2) i18n (0.5.0) + imagefactory-console (0.5.0) json (1.4.6) - json_pure (1.5.4) - spruz (~> 0.2.8) + json_pure (1.4.6) launchy (2.0.5) addressable (~> 2.2.6) mail (2.3.0) @@ -112,11 +116,16 @@ GEM rest-client (1.6.1) mime-types (>= 1.16) rspec-core (2.6.4) + rspec-expectations (2.6.0) + diff-lcs (~> 1.1.2) + rspec-mocks (2.6.0) rspec-rails (2.6.1) actionpack (~> 3.0) activesupport (~> 3.0) railties (~> 3.0) rspec-core (~> 2.6.0) + rspec-expectations (~> 2.6.0) + rspec-mocks (~> 2.6.0) ruby-net-ldap (0.0.4) rubyzip (0.9.4) sass (3.1.4) @@ -127,7 +136,6 @@ GEM rubyzip simple-navigation (3.0.0) activesupport (>= 2.3.2) - spruz (0.2.13) sqlite3 (1.3.3) term-ansicolor (1.0.6) thin (1.2.11) @@ -155,6 +163,7 @@ PLATFORMS ruby DEPENDENCIES + aeolus-image capybara cucumber cucumber-rails -- 1.7.6.2

2 4

ANNOUNCE: oz 0.7.0 release
by Chris Lalancette 23 Sep '11

23 Sep '11

All, I'm pleased to announce release 0.7.0 of Oz. Oz is a program for doing automated installation of guest operating systems with limited input from the user. Release 0.7.0 is a bugfix and feature release for Oz. Some of the highlights between Oz 0.6.0 and 0.7.0 are: - Ability to use the "direct initrd injection" method to install Fedora/RHEL guests. This is an internal implementation detail, but can significantly speed up installs for Fedora or RHEL guests. (thanks for the tip from Kashyap Chamarthy) - Support for Fedora-16 (thanks to Steve Dake for help in making this work) - Use the serial port to announce guest boot, rather than a network port. This makes it so we no longer have to manipulate iptables, and gets us one step closer to having Oz run as non-root - (for developers) Re-written unittests in python for speedier execution - (for developers) Additional methods in the TDL class to merge in external package lists (thanks to Ian McLeod) A tarball of this release is available, as well as packages for Fedora-14, Fedora-15, and RHEL-6. Note that to install the RHEL-6 packages, you must be running RHEL-6.1 or later. Instructions on how to get and use Oz are available at http://aeolusproject.org/oz.html If you have any questions or comments about Oz, please feel free to contact aeolus-devel(a)lists.fedorahosted.org or me (clalance(a)redhat.com) directly. Thanks to everyone who contributed to this release through bug reports, patches, and suggestions for improvement. -- Chris Lalancette

2 1

[PATCHv4 0/5]: Remove condor from the conductor
by Chris Lalancette 23 Sep '11

23 Sep '11

All, This is a patch series to remove condor from the conductor. In short, condor presents problems for our project because it is an external project, it is written in C++ (while most of our developers are ruby), and it is too complex for our current needs. The new way we do scheduling is described pretty well in patch 1, so I won't delve into it here. Since the last posting, there have been a few more bugs fixed: 1) jprovazn finally found an ActiveRecord incantation that works for re-establishing a connection to a database, so the dbomatic subprocesses don't trample over each other 2) I had to fix several bugs when running dbomatic under systemd. I can go into the gory details if you want, just ping me on IRC :) 3) imain discovered that the broken Download Key links are independent of this patchset, so we don't have to worry about that. There are also some outstanding TODO items, but which can be postponed until later: 1) We are still not properly generating the instance name. In condor, we did this by generating a UUID that would be the instance name, then truncating that UUID to the maximum length that deltacloud told us an instance name could be for a particular backend. I'm thinking we want to do the same here. 2) imain believes we can do away with the Task class completely now. That would be a nice cleanup 3) When launching large deployments, the "Launch Deployment" page can be slow to return. That's because in the background we are issuing one deltacloud call at a time to launch each instance in the deployment. We probably want to fork off a separate process for each deployment so we don't hold up the UI process. I've personally tested the patches using the EC2 backend, where things seem to be working well. I was able to launch 5 deployments for a total of 15 instances simultaneously without issue. I also tested up to a 4 assembly deployable which seemed to work. At this point, I'm relatively happy with this patchset and I think we should commit it barring any major problems found in testing. The enhancements above can be done as additional patches on top of this. Please review and ACK. Chris Lalancette

5 14

Justin on leave from now to mid October (back on 17th)
by Justin Clift 23 Sep '11

23 Sep '11

Hi all, I'm on leave from now until mid October, back on 17th (Oct). My intention is to be completely "off the grid" the entire time. *Specifically* away from _all_ computing/connectivity devices in order to get a proper break, to recharge my personal energy levels. (whoo!) :) NOTE FOR PEOPLE WITH COMMUNITY BUILDING IDEAS :> ********************************************* I'm far from a control freak. :> So, if you have an idea for doing something Community related and wanting to know if it's a go-er, consider my opinion to be this: "Go For It & Get It Done" Try it out, let people know, see what happens, we'll adjust over time. :> ALTERNATIVE CONTACTS ******************** For anyone needing Community related stuff which can't be asked about on a public mailing list, the contact people during this time are: + David Lutterkort <dlutter@redhatcom> (Deltacloud) + Francesco Vollero <fvollero(a)redhat.com> (Aeolus in EMEA), Matt Wagner <matt.wagner(a)redhat.com> (Aeolus in US) + Bryan Che <bche(a)redhat.com> (anything needing marketing budget, ie. flyers, promo stuff, printing, etc) + /dev/null (flames) Apart from that, have fun guys, and go take some risks. :) Regards and best wishes, Justin Clift -- Aeolus Community Manager http://www.aeolusproject.org

1 0

Aeolus dependency tree
by Chris Lalancette 22 Sep '11

22 Sep '11

Just in case anyone is interested in looking at how crazy our dependency tree is, I generated one on F-15. The commands are: yum install rpmorphan rpmdep -dot aeolus.dot aeolus-all dot -Tsvg aeolus.dot -o aeolus.svg The result is at http://people.redhat.com/clalance/aeolus.svg -- Chris Lalancette

1 0

[PATCH] Use a thread to do instance creates V3.
by Ian Main 22 Sep '11

22 Sep '11

From: Chris Lalancette <clalance(a)redhat.com> I modified Chris' patch to use a thread instead of fork. This pushes the communications with deltacloud into a thread which can take as long as it needs to complete the start. In the future we may want to put a retry in here too. You will notice in V2 we now set allow_concurrency = true in the postgres config. I'm a little worried that this means it may not work with other DB providers. Signed-off-by: Ian Main <imain(a)redhat.com> --- src/app/util/taskomatic.rb | 81 ++++++++++++++++++++++++++++++------------- src/config/database.pg | 3 ++ 2 files changed, 59 insertions(+), 25 deletions(-) diff --git a/src/app/util/taskomatic.rb b/src/app/util/taskomatic.rb index 84dabd0..bc9afb6 100644 --- a/src/app/util/taskomatic.rb +++ b/src/app/util/taskomatic.rb @@ -26,27 +26,15 @@ module Taskomatic task.state = Task::STATE_PENDING task.save! - task.instance.provider_account = match.provider_account - task.instance.create_auth_key unless task.instance.instance_key - - dcloud_instance = create_dcloud_instance(task.instance, match) - - handle_dcloud_error(dcloud_instance) + create_dcloud_instance(task.instance, match) task.state = Task::STATE_RUNNING - task.save! - - Rails.logger.info "Task instance create completed with key #{dcloud_instance.id} and state #{dcloud_instance.state}" - task.instance.external_key = dcloud_instance.id - task.instance.state = dcloud_to_instance_state(dcloud_instance.state) - task.instance.save! rescue HttpException => ex task.failure_code = Task::FAILURE_PROVIDER_CONTACT_FAILED handle_create_instance_error(task, ex) rescue Exception => ex handle_create_instance_error(task, ex) ensure - task.instance.save! task.save! end end @@ -146,18 +134,61 @@ module Taskomatic end def self.create_dcloud_instance(instance, match) - client = match.provider_account.connect - - overrides = HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp) - - client.create_instance(:image_id => match.provider_image.target_identifier, - :name => instance.name.tr("/", "-"), - :hwp_id => match.hwp.external_key, - :hwp_memory => overrides[:memory], - :hwp_cpu => overrides[:cpu], - :hwp_storage => overrides[:storage], - :realm_id => (match.realm.external_key rescue nil), - :keyname => (instance.instance_key.name)) + # because creating an instance can take a potentially long time (and + # creating multiple of them just prolongs this), we start a new thread + # that does this work. The new thread continues to run in the background + # and communicates the status back to the main UI via the database + + # This cleans up old DB connections. AR creates a new connection for each + # thread. This can leak FDs if you don't clean them up. Basically this + # ensures the previously used FD is cleaned up. + ActiveRecord::Base.connection_pool.clear_stale_cached_connections! + + Thread.new do + begin + # These all need to be reloaded because when you create a new thread + # AR creates a new connection and these objects become stale. + # + # Everything used here must be reloaded. + instance.reload + match.provider_account.reload + match.hwp.reload + match.realm.reload if match.realm + + instance.provider_account = match.provider_account + instance.create_auth_key unless instance.instance_key + + client = match.provider_account.connect + + overrides = HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp) + + dcloud_instance = client.create_instance(:image_id => match.provider_image.target_identifier, + :name => instance.name.tr("/", "-"), + :hwp_id => match.hwp.external_key, + :hwp_memory => overrides[:memory], + :hwp_cpu => overrides[:cpu], + :hwp_storage => overrides[:storage], + :realm_id => (match.realm.external_key rescue nil), + :keyname => (instance.instance_key.name)) + + handle_dcloud_error(dcloud_instance) + + Rails.logger.info "Task instance create completed with key #{dcloud_instance.id} and state #{dcloud_instance.state}" + instance.external_key = dcloud_instance.id + instance.state = dcloud_to_instance_state(dcloud_instance.state) + rescue Exception => ex + # any sort of exception causes us to put the instance in CREATE_FAILED + # FIXME: if the exception is raised *after* the create_instance, this + # isn't true and will result in a rogue instance + Rails.logger.error ex.message + Rails.logger.error ex.backtrace.join("\n") + instance.state = Instance::STATE_CREATE_FAILED + raise ex + ensure + instance.save! + Thread.exit + end + end end def self.matches(instance) diff --git a/src/config/database.pg b/src/config/database.pg index 19fe4e6..72a3895 100644 --- a/src/config/database.pg +++ b/src/config/database.pg @@ -45,6 +45,7 @@ development: username: aeolus password: v23zj59an host: localhost + allow_concurrency: true min_messages: warning # Warning: The database defined as 'test' will be erased and @@ -56,6 +57,7 @@ test: &TEST username: aeolus password: v23zj59an host: localhost + allow_concurrency: true min_messages: warning production: @@ -63,6 +65,7 @@ production: database: conductor username: aeolus password: v23zj59an + allow_concurrency: true host: localhost cucumber: -- 1.7.6.2

1 0

[PATCH] Use a thread to do instance creates V2.
by Ian Main 22 Sep '11

22 Sep '11

From: Chris Lalancette <clalance(a)redhat.com> I modified Chris' patch to use a thread instead of fork. This pushes the communications with deltacloud into a thread which can take as long as it needs to complete the start. In the future we may want to put a retry in here too. You will notice in V2 we now set allow_concurrency = true in the postgres config. I'm a little worried that this means it may not work with other DB providers. Signed-off-by: Ian Main <imain(a)redhat.com> --- src/app/util/taskomatic.rb | 67 +++++++++++++++++++++++++++---------------- 1 files changed, 42 insertions(+), 25 deletions(-) diff --git a/src/app/util/taskomatic.rb b/src/app/util/taskomatic.rb index 84dabd0..eae459a 100644 --- a/src/app/util/taskomatic.rb +++ b/src/app/util/taskomatic.rb @@ -26,27 +26,15 @@ module Taskomatic task.state = Task::STATE_PENDING task.save! - task.instance.provider_account = match.provider_account - task.instance.create_auth_key unless task.instance.instance_key - - dcloud_instance = create_dcloud_instance(task.instance, match) - - handle_dcloud_error(dcloud_instance) + create_dcloud_instance(task.instance, match) task.state = Task::STATE_RUNNING - task.save! - - Rails.logger.info "Task instance create completed with key #{dcloud_instance.id} and state #{dcloud_instance.state}" - task.instance.external_key = dcloud_instance.id - task.instance.state = dcloud_to_instance_state(dcloud_instance.state) - task.instance.save! rescue HttpException => ex task.failure_code = Task::FAILURE_PROVIDER_CONTACT_FAILED handle_create_instance_error(task, ex) rescue Exception => ex handle_create_instance_error(task, ex) ensure - task.instance.save! task.save! end end @@ -146,18 +134,47 @@ module Taskomatic end def self.create_dcloud_instance(instance, match) - client = match.provider_account.connect - - overrides = HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp) - - client.create_instance(:image_id => match.provider_image.target_identifier, - :name => instance.name.tr("/", "-"), - :hwp_id => match.hwp.external_key, - :hwp_memory => overrides[:memory], - :hwp_cpu => overrides[:cpu], - :hwp_storage => overrides[:storage], - :realm_id => (match.realm.external_key rescue nil), - :keyname => (instance.instance_key.name)) + # because creating an instance can take a potentially long time (and + # creating multiple of them just prolongs this), we start a new thread + # that does this work. The new thread continues to run in the background + # and communicates the status back to the main UI via the database + + Thread.new do + begin + instance.provider_account = match.provider_account + instance.create_auth_key unless instance.instance_key + + client = match.provider_account.connect + + overrides = HardwareProfile.generate_override_property_values(instance.hardware_profile, match.hwp) + + dcloud_instance = client.create_instance(:image_id => match.provider_image.target_identifier, + :name => instance.name.tr("/", "-"), + :hwp_id => match.hwp.external_key, + :hwp_memory => overrides[:memory], + :hwp_cpu => overrides[:cpu], + :hwp_storage => overrides[:storage], + :realm_id => (match.realm.external_key rescue nil), + :keyname => (instance.instance_key.name)) + + handle_dcloud_error(dcloud_instance) + + Rails.logger.info "Task instance create completed with key #{dcloud_instance.id} and state #{dcloud_instance.state}" + instance.external_key = dcloud_instance.id + instance.state = dcloud_to_instance_state(dcloud_instance.state) + rescue Exception => ex + # any sort of exception causes us to put the instance in CREATE_FAILED + # FIXME: if the exception is raised *after* the create_instance, this + # isn't true and will result in a rogue instance + Rails.logger.error ex.message + Rails.logger.error ex.backtrace.join("\n") + instance.state = Instance::STATE_CREATE_FAILED + raise ex + ensure + instance.save! + Thread.exit + end + end end def self.matches(instance) -- 1.7.6.2

1 0

iwhd security
by Pete Zaitcev 22 Sep '11

22 Sep '11

In its present form iwhd has no access controls whatsoever, and it concerns me. The memcached is like that too, and it is deployed successfully on many high-traffic, high-profile sites. However, iwhd implies remote access, which makes the current set-up more hazardous. I would like to consider options from an attack by rooted systems on the internal networks, and less privileged insiders. A short while ago, there was some effort to underpin the whole cloud with IPA. From that time, a patch exists for iwhd, which implements Kerberos authentication. There is no authorization in it, however: anyone who is able to authenticate with Kerberos is considered authorized. I'm wondering if adding this to mainline iwhd would help anything. Do Conductor and Factory have an HTTP-Negotiate client support? Failing that, I'd like to know if anyone has any suggestions or requests. For example, would adding an HTTP-Basic (over SSL or not) help anything? -- Pete

5 12

Need new JEOS build for oz/factory
by wes hayutin 22 Sep '11

22 Sep '11

Due to some recent changes w/ RHUI the JEOS images for aeolus need to be respun or use a new ami. Until then you will get the following when building/pushing ec2. 'terminated'. [290 of 300 seconds elapsed] 2011-09-22 13:31:21,235 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(14690) Message: Instance status (stopped) - waiting for 'terminated'. [295 of 300 seconds elapsed] 2011-09-22 13:31:26,238 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(14690) Message: Exception caught in ImageFactory 2011-09-22 13:31:26,301 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(14690) Message: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/imgfac/builders/Fedora_ec2_Builder.py", line 425, in push_image credentials) File "/usr/lib/python2.7/site-packages/imgfac/builders/Fedora_ec2_Builder.py", line 581, in push_image_snapshot_ec2 self.install_euca_tools(guestaddr) File "/usr/lib/python2.7/site-packages/imgfac/builders/RHEL6_ec2_Builder.py", line 45, in install_euca_tools self.guest.guest_execute_command(guestaddr, "yum -y install euca2ools") File "/usr/lib/python2.7/site-packages/oz/RedHat.py", line 463, in guest_execute_command command, timeout) File "/usr/lib/python2.7/site-packages/oz/ozutil.py", line 332, in ssh_execute_command "root@" + guestaddr, command]) File "/usr/lib/python2.7/site-packages/oz/ozutil.py", line 310, in subprocess_check_output raise SubprocessException("'%s' failed(%d): %s" % (cmd, retcode, stderr), retcode) SubprocessException: 'ssh -i /tmp/tmpxYCJQd -F /dev/null -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no root(a)ec2-50-17-23-173.compute-1.amazonaws.com yum -y install euca2ools' failed(1): Warning: Permanently added 'ec2-50-17-23-173.compute-1.amazonaws.com,50.17.23.173' (RSA) to the list of known hosts. https://rhui.us-east-1.aws.ce.redhat.com/rhuilb/content/dist/rhel/rhui/serv…: [Errno 14] Peer cert cannot be verified or peer cert invalid Trying other mirror. Error: Cannot retrieve repository metadata (repomd.xml) for repository: rhui-us-east-rhel-server-releases. Please verify its path and try again 2011-09-22 13:31:26,301 DEBUG imgfac.BuildJob.BuildAdaptor pid(14690) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from PUSHING to FAILED

3 4

How to fix aeolus-image slowness
by Matt Wagner 22 Sep '11

22 Sep '11

Hi all, There's been some discussion about the fact that "aeolus-image list --images" is extremely slow. It's fine with a handful of images, but the time required grows exponentially, so that by perhaps 25 images, the command takes minutes or hours to run. The problem is that we tried to implement associations like they existed natively -- an Image has many Builds. But since iwhd stores this information backwards from how we want it, we need to load ALL builds and then match the one with our image_id in it. Where this gets outrageously inefficient is for things like the "aeolus-image list --images" use case, where we have to walk all associations many times. (Although in theory we're listing "images," the reality is that we need several other data types to get all the requisite data. Otherwise we're just listing UUIDs.) It ends up that, for each image, we load all builds, and, for each of those builds, we load all target images, and for each of those, we load all provider images. This is fine for small sets, but the number of requests grows exponentially and becomes absurdly impractical very quickly. The "real" fix that we've kicked around is to find a new way to store this information. And we should do that at some point. But it's clear that this isn't going to happen soon, and I think we need to take some action now. Here's what I propose: if we know we're going to have to traverse any associations, load all the objects of that type into memory for the duration of the request, and do the matching in memory. So for "aeolus-image list --images", we would instead load all images, builds, target images, and provider images once, and then do all of the mapping in memory. This can save a few orders of magnitude of API calls, and, because it's only for the duration of the request, doesn't leave us caching stale data. I haven't fully thought through how to implement this yet, as we don't want to load these associations if they're not needed at all. But as soon as we know we're going to want an association, we should load all of them into memory. This isn't a perfect plan by any stretch, but I think it makes things substantially less-bad, and should serve at least as a stop-gap. What do you all think of this approach? Best, Matt

3 3

← Newer
1
...
4
5
6
7
8
9
10
...
21
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

aeolus-devel September 2011