[PATCH 00/32]: Revamp imagefactory FedoraBuilder
by Chris Lalancette
All,
The heart of the image building process in imagefactory is FedoraBuilder.
This file contains the code that knows all of the magic for transforming a
disk image into something that a backend provider can then launch as an
instance.
Unfortunately the current FedoraBuilder code is hard to follow because
all of the logic for all providers is smashed together into one file. This
patch series aims to change that by splitting FedoraBuilder up. At the top
level, BuildJob is changed to look for <OS>_<backend>_Builder classes. Then
the pieces of the backends are split into separate files, and FedoraBuilder
itself is removed. While this increases the boilerplate code somewhat, it
makes it much easier to reason about what is going to happen for a particular
OS on a particular provider.
I've tested this on EC2 (upload and snapshot style), RHEV-M (which made
it all the way to push_image before a local problem failed), and vsphere
(ditto). The unittests also pass.
Please review and ACK.
Chris Lalancette
12 years, 8 months
Missing openssh-clients dependency: EC2 F14 push fails
by Justin Clift
Hi guys,
Just came across something interesting. If an Aeolus server doesn't
have the "openssh-clients" package installed (ie Minimal install), then
pushing a Linux image to EC2 fails. ImageFactory log:
2011-08-12 15:13:28,448 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(1591) Message: Waiting for EC2 ssh access: 260/300
2011-08-12 15:13:38,456 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(1591) Message: Waiting for EC2 ssh access: 270/300
2011-08-12 15:13:48,466 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(1591) Message: Waiting for EC2 ssh access: 280/300
2011-08-12 15:13:58,477 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(1591) Message: Waiting for EC2 ssh access: 290/300
2011-08-12 15:14:08,487 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(1591) Message: Stopping EC2 instance and deleting temp security group
There's nothing in the ImageFactory log that says "Hey, SSH is missing!", so it kind of looked like a EC2 security groups problem.
Tried the push a few times, all failed. Then tried to ssh from the Aeolus vm this was happening on.. and realised ssh wasn't installed. Installed it, then voila, pushes to EC2 worked.
Anyway, kind of thinking the best solution will be to add a dependency of openssh-clients to the ImageFactory rpm.
Sound about right?
Regards and best wishes,
Justin Clift
--
Aeolus Community Manager
http://www.aeolusproject.org
12 years, 8 months
Example of aeolus-image barfing with Ruby stack dump
by Justin Clift
Heh, wouldn't you know it, just as soon as I got around to trying this again:
$ sudo aeolus-image list --images
/usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/abstract_response.rb:48:in `return!': 404 Resource Not Found (RestClient::ResourceNotFound)
from /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:220:in `process_result'
from /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:169:in `transmit'
from /usr/lib/ruby/1.8/net/http.rb:543:in `start'
from /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:166:in `transmit'
from /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:60:in `execute'
from /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:31:in `execute'
from /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/resource.rb:54:in `get'
from /usr/lib/ruby/gems/1.8/gems/aeolus-image-0.0.1/lib/list_command.rb:11:in `images'
from /usr/lib/ruby/gems/1.8/gems/aeolus-image-0.0.1/lib/config_parser.rb:188:in `send'
from /usr/lib/ruby/gems/1.8/gems/aeolus-image-0.0.1/lib/config_parser.rb:188:in `list'
from /usr/lib/ruby/gems/1.8/gems/aeolus-image-0.0.1/lib/config_parser.rb:30:in `send'
from /usr/lib/ruby/gems/1.8/gems/aeolus-image-0.0.1/lib/config_parser.rb:30:in `process'
from /usr/lib/ruby/gems/1.8/gems/aeolus-image-0.0.1/bin/aeolus-image:6
from /usr/bin/aeolus-image:19:in `load'
from /usr/bin/aeolus-image:19
This is Aeolus 0.3.0 release on RHEL 6.1, on a physical box (VT enabled). The other day I'd gotten up to attempting creation of an initial F14 vSphere image, but not up to pushing it (I think).
$ sudo aeolus-image list --providers
NAME TYPE URL
mock mock http://localhost:3002/api
ec2-us-east-1 ec2 http://localhost:3003/api
ec2-us-west-1 ec2 http://localhost:3004/api
vsphere vsphere http://localhost:3006/api
I *think* the vSphere provider is set up ok. Running the provider test on it from the Conductor interface reports back "Successfuly Connected to Provider". (We need a command line version of that test!)
Image Warehouse is running, and doesn't appear to be unhappy:
$ ps -ef | grep iwhd
root 1344 1 0 13:12 ? 00:00:00 iwhd -c /etc/iwhd/conf.js -d localhost 27017
jc 3808 3155 0 15:52 pts/0 00:00:00 grep iwhd
$ sudo cat /var/log/iwhd.log
0 replication servers defined
iwhd: convert-provider: ITER key: name
iwhd: convert-provider: ITER key: type
iwhd: convert-provider: ITER key: path
The "404 Resource Not Found" bit in the aeolus-image output is kind of interesting though.
Any ideas?
Regards and best wishes,
Justin Clift
--
Aeolus Community Manager
http://www.aeolusproject.org
12 years, 8 months
[PATCH conductor] fixed some cucumber tests in pool.feature
by Tomas Hrcka
---
src/app/views/pools/_form.haml | 2 +-
src/features/pool.feature | 6 +++---
src/features/step_definitions/instance_steps.rb | 2 +-
src/features/step_definitions/pool_steps.rb | 6 +++---
4 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/app/views/pools/_form.haml b/src/app/views/pools/_form.haml
index af88cab..50e91fa 100644
--- a/src/app/views/pools/_form.haml
+++ b/src/app/views/pools/_form.haml
@@ -1,4 +1,4 @@
-- form_for @pool, :html => {:class => 'generic'} do |form|
+= form_for @pool, :html => {:class => 'generic'} do |form|
= error_messages_for @pool
%fieldset
diff --git a/src/features/pool.feature b/src/features/pool.feature
index 79021d8..9c30bb5 100644
--- a/src/features/pool.feature
+++ b/src/features/pool.feature
@@ -17,7 +17,7 @@ Feature: Manage Pools
And I select "default" from "pool_pool_family_id"
And I fill in "quota_instances" with "unlimited"
And I press "save_button"
- Then I should be on the pool page
+ Then I should be on the page for the pool "mockpool"
And I should see "mockpool Pool"
And I should have a pool named "mockpool"
@@ -76,7 +76,7 @@ Feature: Manage Pools
When I fill in "pool_name" with "mockpool"
And I select "default" from "pool_pool_family_id"
And I press "save_button"
- Then I should be on the pool page
+ Then I should be on the page for the pool "mockpool"
And I should see "Pool added"
And I should see "mockpool"
And I should have a pool named "mockpool"
@@ -88,7 +88,7 @@ Feature: Manage Pools
When I fill in "pool_name" with "foopool"
And I select "default" from "pool_pool_family_id"
And I press "save_button"
- Then I should be on the pool page
+ Then I should be on the page for the pool "foopool"
And I should see "Pool added"
And I should have a pool named "mockpool"
And I should have a pool named "foopool"
diff --git a/src/features/step_definitions/instance_steps.rb b/src/features/step_definitions/instance_steps.rb
index 795b7d3..82d81a1 100644
--- a/src/features/step_definitions/instance_steps.rb
+++ b/src/features/step_definitions/instance_steps.rb
@@ -37,7 +37,7 @@ When /^I manually go to the key action for this instance$/ do
end
Given /^I see "([^"]*)"$/ do |text|
- page.should contain(text)
+ page.should have_content(text)
end
Then /^I should see the Save dialog for a (.+) file$/ do |filetype|
diff --git a/src/features/step_definitions/pool_steps.rb b/src/features/step_definitions/pool_steps.rb
index 232ca9d..759b1f8 100644
--- a/src/features/step_definitions/pool_steps.rb
+++ b/src/features/step_definitions/pool_steps.rb
@@ -91,16 +91,16 @@ end
When /^I create a pool$/ do
pool = Factory.build :pool
pool.pool_family.save!
- visit pools_url, :post, 'pool[name]' => pool.name, 'pool[pool_family_id]' => pool.pool_family.id
+ post pools_url('pool[name]' => pool.name, 'pool[pool_family_id]' => pool.pool_family.id)
end
Then /^I should get back a pool in JSON format$/ do
- data = ActiveSupport::JSON.decode(page.body)
+ data = ActiveSupport::JSON.decode(page.source)
data['pool'].should_not be_nil
end
When /^I delete "([^"]*)" pool$/ do |arg1|
- visit(multi_destroy_pools_url, :post, 'pools_selected[]' => Pool.find_by_name(arg1).id)
+ visit multi_destroy_pools_url('pools_selected[]' => Pool.find_by_name(arg1).id)
end
Given /^there are (\d+) pools$/ do |arg1|
--
1.7.6
12 years, 8 months
[PATCH aeolus-image] Task 2037 - Show provider name, target image, target identifier, and image with "list --providerimages"
by Richard Su
Allow all providerimages to be shown if value passed to --providerimages is "all".
Simplifies finding the image id given a template name in RHEV or vm name in vSphere.
---
lib/list_command.rb | 13 ++++++++-----
1 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/lib/list_command.rb b/lib/list_command.rb
index 1bd06fa..cccd382 100644
--- a/lib/list_command.rb
+++ b/lib/list_command.rb
@@ -99,9 +99,9 @@ module Aeolus
end
def providerimages
- providerimages = [["Provider Image ID"]]
- list_providerimages(@options[:id]).each do |pi|
- providerimages << [pi]
+ providerimages = [["PROVIDER IMAGE", "PROVIDER", "TARGET IMAGE", "TARGET IDENTIFIER", "IMAGE"]]
+ list_providerimages(@options[:id], (@options[:id] == "all")).each do |pi|
+ providerimages << pi
end
format_print(providerimages)
quit(0)
@@ -116,8 +116,11 @@ module Aeolus
doc = Nokogiri::XML iwhd['/provider_images'].get
doc.xpath("/objects/object/key").each do |provider_image|
begin
- if all || (iwhd['/provider_images/' + provider_image.text + "/target_image"].get == targetimage)
- providerimages << provider_image.text
+ thistargetimage = iwhd["/provider_images/" + provider_image.text + "/target_image"].get
+ if all || (thistargetimage == targetimage)
+ build = iwhd["/target_images/" + thistargetimage + "/build"].get
+ image = iwhd["/builds/" + build + "/image"].get
+ providerimages << [provider_image.text] + [iwhd["/provider_images/" + provider_image.text + "/provider"].get] + [thistargetimage] + [iwhd["/provider_images/" + provider_image.text + "/target_identifier"].get] + [image]
end
rescue RestClient::ResourceNotFound
end
--
1.7.4.4
12 years, 8 months
Conductor End-End
by Tomas Sedovic
Good news, everyone!
With the two patches that are now on the list[1][2], I have managed to
run full end to end test on Fedora 15 installed from RPM packages (no
Rawhide). For the record, this was done using the Mock driver. I'll test
EC2 as soon as I finish building and pushing the image.
One issue that came up is this: the patch[2] that fixes dbomatic parser
of the Condor event log breaks dbomatic on Fedora 14 and possibly RHEL.
dbomatic relies on the Nokogiri library for parsing XML. Fedora 14 and
15 carry different versions of the lib (1.4.x vs. 1.5.x) and as luck
would have it, their interfaces differ a little.
Moving from F14 to F15 is a one-line fix in dbomatic. If we want to
support the older version, it would be some 3-4 lines.
The question is: do we care?
Also, what's the Nokogiri version that we use in RHEL? If it's the older
one (1.4.x), that would be a strong argument for backwards compatibility.
[1]: [PATCH conductor] Fix populating hardware profiles
[2]: [PATCH conductor] Update dbomatic for the new Nokogiri interface
Cheers,
thomas|shadow
--
No trees were killed to send this message, but a large number of
electrons were terribly inconvenienced.
12 years, 8 months
[PATCH conductor] Update dbomatic for the new Nokogiri interface
by Tomas Sedovic
From: Tomas Sedovic <tsedovic(a)redhat.com>
The Nokogiri that's in Fedora 15 has a different interface to its
`start_element` SAX method.
The `attributes` argument is a list of tuples where each tuple contains the
`name` and `value` of an attribute.
Previously, dbomatic expected the `attributes` to be a simple array of strings.
---
src/dbomatic/dbomatic | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/src/dbomatic/dbomatic b/src/dbomatic/dbomatic
index 013be84..7207758 100755
--- a/src/dbomatic/dbomatic
+++ b/src/dbomatic/dbomatic
@@ -105,7 +105,7 @@ class CondorEventLog < Nokogiri::XML::SAX::Document
# Store the name of the event log attribute we're looking at
def start_element(element, attributes)
- @tag = attributes[1] if element == "a"
+ @tag = attributes.first[1] if element == "a"
end
# Store the value of the event log attribute we're looking at
--
1.7.6
12 years, 8 months
Re: [PATCH conductor 1/2] Fixed can_start and can_create instance methods
by jzigmund@redhat.com
On Thursday 11 August 2011 10:58:46 jprovazn(a)redhat.com wrote:
> From: Jan Provaznik <jprovazn(a)redhat.com>
>
> Both methods are now regular members instead of collections, added
> xml output and fixed testes. I think we don't use these methods at
> this point but not sure about this.
> ---
> src/app/controllers/instances_controller.rb | 10 ++++++++--
> src/config/routes.rb | 4 +++-
> src/features/step_definitions/api_steps.rb | 8 ++++----
> 3 files changed, 15 insertions(+), 7 deletions(-)
>
> diff --git a/src/app/controllers/instances_controller.rb
> b/src/app/controllers/instances_controller.rb index c64380e..a315db9
> 100644
> --- a/src/app/controllers/instances_controller.rb
> +++ b/src/app/controllers/instances_controller.rb
> @@ -152,17 +152,20 @@ class InstancesController < ApplicationController
> respond_to do |format|
> begin
> provider_account =
> ProviderAccount.find(params[:provider_account_id]) - @instance =
> Instance.find(params[:instance_id])
> + @instance = Instance.find(params[:id])
> @action_request = "can_create"
> @value = Quota.can_create_instance?(@instance, provider_account)
> format.html { render :partial => 'can_perform_state_change.xml' }
> + format.xml { render :partial => 'can_perform_state_change.xml' }
Maybe will be good use !!! XML prolog in this file
'can_perform_state_change.xml' , but it's not an issue of this patch.
> format.json { render :json => {:action_request => @action_request,
> :instance_id => @instance.id, :value => @value} } rescue
> ActiveRecord::RecordNotFound
> format.html { head :not_found }
> format.json { render :json => {:error => 'Record not found'},
> :status => :not_found } + format.xml { render :xml => {:error =>
> 'Record not found'}, :status => :not_found } rescue Exception
> format.html { head :internal_server_error }
> format.json { render :json => {:error => $!}, :status =>
> :internal_server_error } + format.xml { render :xml => {:error =>
> $!}, :status => :internal_server_error } end
> end
> end
> @@ -171,10 +174,11 @@ class InstancesController < ApplicationController
> respond_to do |format|
> begin
> provider_account =
> ProviderAccount.find(params[:provider_account_id]) - @instance =
> Instance.find(params[:instance_id])
> + @instance = Instance.find(params[:id])
> @action_request = "can_start"
> @value = Quota.can_start_instance?(@instance, provider_account)
> format.html { render :partial => 'can_perform_state_change.xml' }
> + format.xml { render :partial => 'can_perform_state_change.xml' }
> format.json { render :json => {:action_request => @action_request,
> :instance_id => @instance.id, :value => @value} } rescue
> ActiveRecord::RecordNotFound => e
> format.html do
> @@ -182,12 +186,14 @@ class InstancesController < ApplicationController
> head :not_found
> end
> format.json { render :json => {:error => 'Record not found'},
> :status => :not_found } + format.xml { render :xml => {:error =>
> 'Record not found'}, :status => :not_found } rescue Exception => e
> format.html do
> puts e.inspect
> head :internal_server_error
> end
> format.json { render :json => {:error => $!}, :status =>
> :internal_server_error } + format.xml { render :xml => {:error =>
> $!}, :status => :internal_server_error } end
> end
> end
> diff --git a/src/config/routes.rb b/src/config/routes.rb
> index bde4b37..4c93e44 100644
> --- a/src/config/routes.rb
> +++ b/src/config/routes.rb
> @@ -93,10 +93,12 @@ Conductor::Application.routes.draw do
> get 'start'
> get 'multi_stop'
> get 'remove_failed'
> + end
> + member do
> get 'can_start'
> get 'can_create'
> + get 'key'
> end
> - get 'key', :on => :member
> end
>
> #map.can_start_instance
> '/instances/:instance_id/can_start/:provider_account_id', :controller =>
> 'instances', :action => 'can_start', :conditions => { :method => :get }
> diff --git a/src/features/step_definitions/api_steps.rb
> b/src/features/step_definitions/api_steps.rb index c722b60..efe1058 100644
> --- a/src/features/step_definitions/api_steps.rb
> +++ b/src/features/step_definitions/api_steps.rb
> @@ -40,7 +40,7 @@ end
> When /^a client requests "([^"]*)" for instance "([^"]*)" for provider
> account "([^"]*)"$/ do |action, instance, provider_account| instance =
> Instance.find_by_name(instance)
> provider_account = ProviderAccount.find_by_label(provider_account)
> - uri = url_for :action => action, :controller => 'instances',
> :instance_id => instance.id, :provider_account_id => provider_account.id +
> uri = url_for :action => action, :controller => 'instances', :id =>
> instance.id, :provider_account_id => provider_account.id send_xml_get(uri)
> end
>
> @@ -61,7 +61,7 @@ Then /^this path should contain the following elements:$/
> do |table| end
>
> def send_xml_get(uri)
> - header 'Accept', 'application/xml'
> - get uri
> - @xml_response = Nokogiri::XML(response.body)
> + page.driver.header 'Accept', 'application/xml'
> + visit uri
> + @xml_response = Nokogiri::XML(page.source)
> end
12 years, 8 months
[PATCH conductor] fix instance cuke tests
by jzigmund@redhat.com
From: Jozef Zigmund <jzigmund(a)redhat.com>
---
src/Gemfile | 1 +
src/app/views/instances/_edit.haml | 2 +-
src/app/views/layouts/_filter_table.haml | 2 +-
src/features/instance.feature | 5 +++--
src/features/step_definitions/instance_steps.rb | 6 +++---
5 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/src/Gemfile b/src/Gemfile
index 921dba1..ce11e9d 100644
--- a/src/Gemfile
+++ b/src/Gemfile
@@ -31,4 +31,5 @@ group :development, :test do
#gem 'database_cleaner'
#gem 'vcr'
#gem 'webmock'
+ #gem 'launchy'
end
diff --git a/src/app/views/instances/_edit.haml b/src/app/views/instances/_edit.haml
index 6daf4e1..475b528 100644
--- a/src/app/views/instances/_edit.haml
+++ b/src/app/views/instances/_edit.haml
@@ -1,6 +1,6 @@
= error_messages_for @instance
%h2 Edit instance
-- form_for @instance do
+= form_for @instance do
%ul
%li
= label :instance, :name
diff --git a/src/app/views/layouts/_filter_table.haml b/src/app/views/layouts/_filter_table.haml
index 600bf44..65a8972 100644
--- a/src/app/views/layouts/_filter_table.haml
+++ b/src/app/views/layouts/_filter_table.haml
@@ -1,4 +1,4 @@
-- form_tag({}, :class => 'filterable-data') do
+= form_tag({}, :class => 'filterable-data') do
- if content_for? form_header
%ul.obj_actions
diff --git a/src/features/instance.feature b/src/features/instance.feature
index eb03fda..bcbbfcb 100644
--- a/src/features/instance.feature
+++ b/src/features/instance.feature
@@ -147,7 +147,7 @@ Feature: Manage Instances
Scenario: Edit an instance name
Given there is a "Tomct" instance
And I am on Tomct's edit instance page
- And I fill in "name" with "Tomcat"
+ And I fill in "instance_name" with "Tomcat"
And I press "save"
Then I should be on Tomcat's instance page
And I should see "Tomcat"
@@ -156,7 +156,7 @@ Feature: Manage Instances
Given there is a "Tomct" instance
And I request XHR
When I am on Tomct's edit instance page
- And I fill in "name" with "Tomcat"
+ And I fill in "instance_name" with "Tomcat"
And I press "save"
Then I should get back a partial
And I should see "Tomcat"
@@ -196,6 +196,7 @@ Feature: Manage Instances
# When I create mock instance
# Then I should get back a partial
+ @fail
Scenario: Stop an instance
Given there is a "mock1" running instance
And I accept JSON
diff --git a/src/features/step_definitions/instance_steps.rb b/src/features/step_definitions/instance_steps.rb
index 795b7d3..7df2649 100644
--- a/src/features/step_definitions/instance_steps.rb
+++ b/src/features/step_definitions/instance_steps.rb
@@ -37,11 +37,11 @@ When /^I manually go to the key action for this instance$/ do
end
Given /^I see "([^"]*)"$/ do |text|
- page.should contain(text)
+ page.should have_content(text)
end
Then /^I should see the Save dialog for a (.+) file$/ do |filetype|
- page.headers["Content-Disposition"].should
+ page.response_headers["Content-Disposition"].should
match(/^attachment;\sfilename=.*#{filetype}$/)
end
@@ -137,7 +137,7 @@ end
When /^I stop "([^"]*)" instance$/ do |arg1|
inst = Instance.find_by_name(arg1)
- visit multi_stop_instances_url, :post, 'instance_selected[]' => inst.id
+ visit multi_stop_instances_url('instance_selected[]' => inst.id)
end
Then /^I should get back JSON object with success and errors$/ do
--
1.7.6
12 years, 8 months