aeolus-devel April 2012

aeolus-devel@lists.fedorahosted.org

39 participants
89 discussions

[PATCH conductor] BZ 801527: wrong error message when imagefactory is down
by Maros Zatko 06 Sep '12

06 Sep '12

From: Maros Zatko <mzatko(a)redhat.com> https://bugzilla.redhat.com/show_bug.cgi?id=801527 API: * new API exception ServiceUnavailable * rescue for ECONNREFUSED when imagefactory is down --- src/app/controllers/api/images_controller.rb | 14 +++++++++----- src/lib/exceptions.rb | 1 + 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/src/app/controllers/api/images_controller.rb b/src/app/controllers/api/images_controller.rb index 465a20b..2bcbeee 100644 --- a/src/app/controllers/api/images_controller.rb +++ b/src/app/controllers/api/images_controller.rb @@ -89,11 +89,15 @@ module Api :template => @tpl.uuid, :environment => @pool_family.name }) - @image = Aeolus::Image::Factory::Image.new(:id => iwhd_image.id) - @image.targets = req[:params][:targets] - @image.template = req[:params][:template] - @image.save! - respond_with(@image) + begin + @image = Aeolus::Image::Factory::Image.new(:id => iwhd_image.id) + @image.targets = req[:params][:targets] + @image.template = req[:params][:template] + @image.save! + respond_with(@image) + rescue Errno::ECONNREFUSED + raise(Aeolus::Conductor::API::ServiceUnavailable.new(503, 'Imagefactory is dead, Jim!')) + end elsif req[:type] == :import account = ProviderAccount.find_by_label(req[:params][:provider_account_name]) raise(Aeolus::Conductor::API::ProviderAccountNotFound.new(404, t("api.error_messages.provider_account_not_found", diff --git a/src/lib/exceptions.rb b/src/lib/exceptions.rb index d56ddb3..9b1edab 100644 --- a/src/lib/exceptions.rb +++ b/src/lib/exceptions.rb @@ -38,6 +38,7 @@ module Aeolus class ProviderImageDeleteFailure < Error; end class ProviderImageNotFound < Error; end class ProviderImageStatusNotFound < Error; end + class ServiceUnavailable < Error; end class TargetImageDeleteFailure < Error; end class TargetImageNotFound < Error; end class TargetImageStatusNotFound < Error; end -- 1.7.7.6

2 2

RFC: Background Processing
by Richard Su 14 Jun '12

14 Jun '12

Hi, This expands on some of the notes Jan provided in other RFCs. delayed_jobs and resque appears to be the most commonly deployed solution. I listed what I thought should be the requirements for a background processing solution. For each requirement I then added some details on how well delayed_jobs and resque could satisfy it. Resque contains most of the features we need. It requires Redis, which is a open source project sponsored by VMware. Redis is available in Fedora. But I don't see Redis available in RHEL and getting it in for RHEL is the big question mark. https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Background_Proce… --- Background Processing # Summary The two most common solutions are delayed_jobs and resque. There is a good write up on github comparing other background processing solutions and why they eventually steered towards delayed_jobs and then resque, https://github.com/blog/542-introducing-resque. The primary differences between delayed_jobs and resque are: At the moment, delayed_jobs doesn't have support for recurring jobs. Resque does support recuring jobs through the resque-scheduler extension/gem. resque provides a sinatra app to monitor the queue. delayed_job doesn't provide monitoring tools out of the box, but we can potential build something on top of rails or simply look at the contents of the database table. resque requires multiple components and potentially could be more difficult to support. It requries a second gem called resque-scheduler. It also uses Redis as its backend and it is currently not available with RHEL. This may be the deal breaker. # Requirements 1. Bucket jobs into different queues. A long running job to check instance status for 1000 instances should not hold up other jobs. The solution should also support multiple workers which would minimize impact of longer running jobs. But using different queues will offer finer grain control. * delayed_job: supports multiple queues through named queues starting with version 3.0. Can start up multiple workers for all queues or for specific queues. * resque: supports multiple queues and workers. 2. Jobs should persist in some way. If a crash occurs, we should be able to restart the system and continue with processing incomplete jobs in the queue. * delayed_job: Jobs persists as objects stored in activerecord entries. * resque: Jobs persists as json objects in redis entries. Using json objects instead of actual objects which may have advanced to a different version makes updating the application potentially easier. 3. Recurring jobs. * delayed_jobs: Not available, in development. * resque: Through resque-scheduler extension. * whenever: A potential alternative to do cron style scheduling [6]. 4. Alerts. Failures should be presented to the user in some way (email, conductor UI) so that appropriate actions can be taken. * delayed_jobs: Support code hooks for different stages in the process. Hooks can be added for error, failure, success.. By default workers will retry a job 25 times. We should use a lower number. No sense in retrying that number of times and holding up the queue if there is a hard failure somewhere in the system. By default it also deletes failed jobs, but can be configured to leave them in the queue with a flag to indicate failure. * resque: Failed jobs can go through additional processing using different failure backends. redis, syslog, custom, etc.. 5. A mechanism to requeue a failed job once the underlying issue has been resolved. If an instance start job fails and there is a network failure to a provider. Once the network is back online, we should have an ability to requeue those jobs. Not sure if this should be automated or if this should be a button somewhere where a user can manually requeue all or select failed jobs. * custom 6. Monitor job status. We should have some way to see what is in the queue. * delayed_jobs: Can only view queue through activerecord database entries. There is no UI so it is more difficult to see what is going on. * resque: Provides a sinatra app to monitor queues, jobs, and workers. 7. Should not enqueue duplicate jobs. * custom 8. Ability to remove jobs from the queues and to place a pause on the queues or jobs. * custom 9. Supportable in Fedora and RHEL * delayed_jobs: We used it in the past. Will need to carry the gem. * resque: Will need to carry the gem. In addition it requires Redis as the backend. Redis is available in Fedora but not in RHEL. Redis is a open source project sponsored by VMware [4]. # Use Cases 1. Dbomatic replacement for instance and realm checking and RHEV instance start. Each RHEV instance that is created will also lead to a job that is enqueued to start that instance. Create a new job to perform instance status check. Create a status check job for each provider account. Allow status check job to be disabled/enabled per provider account. Create a new job to sync realms for all providers. This can be broken up to a job per provider if needed. Create two queues. One for managing instance lifecycle. And a second queue for all other jobs. Start with two workers per queue. Make the number of workers configurable so that it may be adjusted when needed. 2. ldap syncing 3. Generic instance start and stop # Reference [1] https://github.com/collectiveidea/delayed_job/wiki/Named-Queues-Proposal [2] https://github.com/blog/542-introducing-resque Discusses github's use of different background job solutions [3] https://github.com/bvandenbos/resque-scheduler [4] http://redis.io/ [5] http://blog.railsupgrade.com/2011/08/replace-delayedjob-with-resque.html [6] https://github.com/javan/whenever

4 8

RFC: Feature Planning - Design the model changes for support of stateful instances
by jzigmund＠redhat.com 08 Jun '12

08 Jun '12

Hi, I wrote up current status of instance states, and described a proposal how we could implement this feature. Rather than paste in the entire text, I'm just linking to the wiki writeup. https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Design_the_model… Any thoughts or improvement ideas are welcomed. -- Jozef

3 3

[PATCH conductor 0/1] #787280 - Get rid of fake links
by Matt Wagner 30 May '12

30 May '12

To Whom It May Concern, Whilst I was using the application, I came to the realization that some of the links in the footer served no purpose. They went nowhere, and there wasn't even an apparent place that they *should* go. I entered https://bugzilla.redhat.com/show_bug.cgi?id=787280 and proceeded to remove the links and the associated entries in en.yml. I thank you for your kind consideration of this matter. Good day, Matt

3 8

[PATCH conductor] BZ808338: fix ajax caching for IE
by Imre Farkas 29 May '12

29 May '12

This issue was really hard to debug: first, I was able to reproduce it only in production environment, not in development env. It's still unclear how it affects Internet Explorer's behaviour. As it turned out, IE caches every ajax requests. It means that e.g. on the deployments#show page clicking on the Properties tab, then the Instances tab and then back to the Properties tab does not fire any request to the server to update the content of the Properties tab. Needless to say that it's not broken in FF or Chrome. The solution is quite simple, jQuery provides a 'cache' parameter for the global ajaxSetup config. If it's false, then jQuery will append a timestamp to the url avoiding the IE browser cache.

2 2

[PATCH aeolus-configure] Apache handles web fonts properly
by Matt Wagner 04 May '12

04 May '12

Our Apache config was invalid for /fonts, though the problem wasn't apparent because the current UI doesn't use them. As we work towards a unified UI, we're using a web font and it wasn't showing in RPM builds due to the invalid reference. This fixes the Alias, tells Apache to not proxy those requests to Rails, and also sets Cache-Control headers on the fonts. I had to add a MIME type for *.woff manually to appease Apache. This is not associated with a BZ. There is no need to carry this in the existing product release; it's only useful going forward when we add web fonts.

3 4

Added deployment state
by Jan Provazník 04 May '12

04 May '12

Hi, this patchset adds deployment state attribute. This attribute will be used for tracking deployment's state which will be needed especially for doing rollback when launching a deployment. It also replaces 'status' method which computed deployment's state from states of all instances. This patchset introduces state transitions (IOW now it's not possible to change deployment's state from 'running' to 'pending' or from 'stopped' to 'shutting down'). deployment's state is now mostly changed on a user's action (e.g. a user clicks stop button -> state is changed to 'shutting down'), though some transitions are done automatically ('pending' -> 'running', 'shutting down' -> 'stopped', all rollback state transitions will be automatic too). Drawback of setting state when an action is performed is that deployment's state may not describe real state of all instances properly, for example if a deployment is running and all instances are suddenly changed to 'stopped' (from outside of conductor), then deployment's state will be 'incomplete' until a user stops/deletes the deployment from conductor. Which reminds me I've added another deployment state: 'incomplete', this state is used when a deployment is running and then some (or all) instances unexpectedly change state (wiki page Robust_instance_launching is updated too).

2 7

[PATCH conductor] BZ786535: display failures for instances (rev. 3)
by Imre Farkas 03 May '12

03 May '12

From: Imre Farkas <ifarkas(a)redhat.com> https://bugzilla.redhat.com/show_bug.cgi?id=786535 Resending this patch based on Jozef's suggestion to add autoupdate to the error messages. I also moved the error messages from flash messages to alerts for readabilty reasons. --- src/app/controllers/deployments_controller.rb | 2 +- src/app/models/deployment.rb | 2 +- src/app/util/taskomatic.rb | 8 +++ src/app/views/deployments/_alerts_show.html.haml | 62 ++++++++++++++++----- src/public/javascripts/backbone/views.js | 34 ++++++++++++ 5 files changed, 91 insertions(+), 17 deletions(-) diff --git a/src/app/controllers/deployments_controller.rb b/src/app/controllers/deployments_controller.rb index 504a050..4ef653a 100644 --- a/src/app/controllers/deployments_controller.rb +++ b/src/app/controllers/deployments_controller.rb @@ -183,7 +183,7 @@ class DeploymentsController < ApplicationController require_privilege(Privilege::VIEW, @deployment) init_new_deployment_attrs save_breadcrumb(deployment_path(@deployment, :viewstate => viewstate_id), @deployment.name) - @failed_instances = @deployment.failed_instances + @failed_instances = @deployment.failed_instances.list(sort_column(Instance), sort_direction) if filter_view? @view = 'instances/list' params[:instances_preset_filter] = "" unless params[:instances_preset_filter] diff --git a/src/app/models/deployment.rb b/src/app/models/deployment.rb index 4f80aff..6b0277a 100644 --- a/src/app/models/deployment.rb +++ b/src/app/models/deployment.rb @@ -432,7 +432,7 @@ class Deployment < ActiveRecord::Base end def failed_instances - instances.select {|instance| instance.failed?} + instances.failed end PRESET_FILTERS_OPTIONS = [] diff --git a/src/app/util/taskomatic.rb b/src/app/util/taskomatic.rb index a7181dc..463f521 100644 --- a/src/app/util/taskomatic.rb +++ b/src/app/util/taskomatic.rb @@ -73,6 +73,14 @@ module Taskomatic rescue Exception => ex task.state = Task::STATE_FAILED task.message = ex.message + task.instance.update_attributes(:last_error => ex.message) + + # For RHEV-M, since we need to start up the instance after the vm has been created + # we also have to handle create_failed state events separately + if task.instance.state == Instance::STATE_STOPPED && task.action == InstanceTask::ACTION_START && + task.instance.provider_account.provider.provider_type.deltacloud_driver == 'rhevm' + task.instance.update_attributes(:state => Instance::STATE_CREATE_FAILED) + end ensure task.save! if Task.exists?(task.id) end diff --git a/src/app/views/deployments/_alerts_show.html.haml b/src/app/views/deployments/_alerts_show.html.haml index f850c6c..6bc1519 100644 --- a/src/app/views/deployments/_alerts_show.html.haml +++ b/src/app/views/deployments/_alerts_show.html.haml @@ -1,15 +1,47 @@ -- if @failed_instances.count > 0 - %section.content-section.alerts.collapse_entity - %header - %h2=t 'alerts_label' - %span.label.badge.alert.count= @failed_instances.count - .section-controls - = link_to t("providers.edit.toggle_alerts"), "#", :class => 'collapse alerts' unless @statistics[:instances_failed].blank? - .content.collapsible - %dl.alerts - %ul - - @failed_instances.each do |inst| - %li.alert - %dt.subject.critical= inst.name - %dd.type=t 'alerts.instance_failure' - %dd.desc= "#{inst.last_error.blank? ? inst.state : inst.last_error}" +#deployment-alerts + - if @failed_instances.count > 0 + %section.content-section.alerts.collapse_entity + %header + %h2=t 'alerts_label' + %span.label.badge.alert.count= @failed_instances.count + .section-controls + = link_to t("providers.edit.toggle_alerts"), "#", :class => 'collapse alerts' unless @statistics[:instances_failed].blank? + .content.collapsible + %dl.alerts + %ul + - @failed_instances.each do |inst| + %li.alert + %dt.subject.critical= inst.name + %dd.type=t 'alerts.instance_failure' + %dd.desc= "#{inst.last_error.blank? ? inst.state : inst.last_error}" + +%script#deploymentAlertsHeaderTemplate{ :type => 'text/x-jquery-tmpl' } + :plain + <section class="content-section alerts collapse_entity"> + <header> + <h2>Alerts</h2> + <span class="label badge alert count">${failedCount}</span> + <div class="section-controls"> + <a class="collapse alerts" href="#">#{t('providers.edit.toggle_alerts')}</a> + </div> + </header> + <div class="content collapsible"> + <dl class="alerts"> + <ul></ul> + </dl> + </div> + </section> + +%script#deploymentAlertTemplate{ :type => 'text/x-jquery-tmpl' } + :plain + <li class="alert"> + <dt class="subject critical">${name}</dt> + <dd class="type">#{t('alerts.instance_failure')}</dd> + <dd class="desc"> + {{if last_error}} + ${last_error} + {{else}} + ${state} + {{/if}} + </dd> + </li> diff --git a/src/public/javascripts/backbone/views.js b/src/public/javascripts/backbone/views.js index a28bb1f..8f2b6eb 100644 --- a/src/public/javascripts/backbone/views.js +++ b/src/public/javascripts/backbone/views.js @@ -219,6 +219,40 @@ Conductor.Views.DeploymentsShow = Backbone.View.extend({ $instances.empty(); $('#instanceTemplate').tmpl(this.collection.toJSON()).appendTo($instances); + + var $alerts = this.$('#deployment-alerts'); + var alertsVisible = $alerts.find('.collapsible').length == 0 || $alerts.find('.collapsible').is(":visible") + var failedInstances = _.compact(_.map(this.collection.models, function(model) { + if(model.get('is_failed')) { + return model.toJSON(); + } + else { + return null; + } + })); + + // Render alerts table + $alerts.empty(); + if(failedInstances.length == 0) return; + + $('#deploymentAlertsHeaderTemplate').tmpl({ + 'failedCount' : failedInstances.length + }).appendTo($alerts); + + // Add callback to Toggle link + $alerts.find('a.collapse').click(function(e) { + e.preventDefault(); + $(this).parents('.collapse_entity').find('.collapsible').slideToggle(80); + }); + + // Restore toggle state on alerts table + if(!alertsVisible) { + $alerts.find('.collapsible').hide(); + } + + // Render alerts + var $alertsList = this.$('dl.alerts > ul'); + $('#deploymentAlertTemplate').tmpl(failedInstances).appendTo($alertsList); } }); -- 1.7.6.5

2 1

FWIW: Current OpenStack status
by Matt Wagner 03 May '12

03 May '12

Hi all, One of the tasks I've been working on is #3178, "Assess the viability of importing and launching images from a openstack provider." We don't have many OpenStack implementation tasks in this sprint, but I was kind of roaring to see something working, so I wanted to sort of take inventory of where things stood today. With the patches adding support applied to master a bit ago, I pulled up a Rails console and saw how far I could get. [1] lists the steps I took. The first obstacle I hit was some exception in the Deltacloud logs, raised when I tried to list instances[2]. Marios helped me debug this and indicated that the problem was that I was using an older version of the Deltacloud package; it was using the Rackspace driver versus the OpenStack one. (See the filename in line 20 of the link.) Until this is updated in the next Deltacloud release, I worked around this by building a local gem for Deltacloud and restarting with that. That much worked great. My next issue was that we do not appear to have anything in place to collect any credentials for those setting up an OpenStack provider account. Thus I cheated and used the Rails console for now, talking to Deltacloud directly. And with that, basic Deltacloud API operations work -- I can list images and instances, for example. Complicating things, the credentials we need are semi-variable. If you are running Keystone, we need to collect your username, password, and tenant name. If you're not running Keystone, we don't need your tenant name. (OpenStack also allows authentication via key + secret key, but Deltacloud does not presently support this.) I'm not sure what the most elegant solution is here just yet, nor if there's a good way to proceed without provider-specific logic. As far as Deltacloud is concerned, the tenant name is just added to the username, so we could do the same, except it might not be particularly clear to end users. I attempted to add Credential Definitions and add a provider account, but that resulted in me receiving some exceptions about hardware profiles. I'd like to investigate further, but I think I'm fairly far afield from my stated objective of investigating the viability of importing images. It may be that I'm just around the bend from getting it working, but I'm starting to think that the conclusion here is that, in its present form, the answer to the task is "Not very viable at the moment." I'd be happy to carry on, but only if people think it's a worthwhile endeavor before Factory adds support for building for OpenStack. -- Matt [1] https://gist.github.com/2431166 [2] https://gist.github.com/2431171

6 6

[PATCH 0/1] Cucumber test fix
by Jason Guiditta 03 May '12

03 May '12

Note that you need the newly pushed (to fedora 16) version of deltacloud client to see this error. This patch should not cause 0.4 to fail for those still on that older version (you need to upgrade soon, btw). As Fedora 15 is closing in on EOL, you can always gem install this, or grab the rpm from koji for version 0.5[1]. The exact test to run is in the RM issue, but for your convenience in testing, it is: cucumber features/provider_account.feature:17 [1] http://koji.fedoraproject.org/koji/packageinfo?packageID=11102 -j

2 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

aeolus-devel April 2012