While testing aeolus I've found that aeolus is mainly conceived for
stateless instances (https://www.aeolusproject.org/redmine/issues/1059).
Will additional functionalities for stateful images be added in aeolus?.
I guess support for stateful instances would require:
- Differentiate between stopped and deleted (or terminated).
- Being able to re-start an stopped instance.
- Being able to delete an stopped instance (to destroy the underlying vm)
- Being able to modify an existing instance (for example, assigning a
different hardware profile)
I think statefull instances will be mainly required in private clouds
using RHEV or VMware providers, i.e.
As deltacloud support both start/stop/destroy vms, I guess this is
doable, is it in current roadmap plan?
Providing a formal deffinition of the XML for an API, makes using the API, in particular creating and maintaining client applications that utilise the API much simpler, and far clearer for a developer.
There are an array of tools that provide class generation, and automatic marshalling and unmarshalling of objects, generated straight from a formally defined XML Document, for ruby, this project looks to offer these functions: https://github.com/rubyjedi/soap4r .
In my experience writing Java and Android clients, I have found that the time spent creating an XSD or other formal XML definition, repays 10 fold in development time, and also makes keeping clients up to date alot simpler. For DC Client I ended up writing my own: https://github.com/mtaylor/deltacloud-tools/blob/master/DeltaCloudClient/...
But, I think it would be better as a whole to adopt this approach from the start. It would not only benefit the creation of the CLI but also anyone else who needs to utilise these APIs
deltacloud-devel mailing list
Are the image building methods that aeolus uses documented anywhere?
I recently discovered that one restriction of using Aeolus with Red Hat Enterprise Virtualization (RHEV) as a provider is that the image builder service requires the use of Hardware-assisted virtualization (HVM). I realized this when running Aeolus in a virtual machine in order to make Aeolus fault tolerant. When executing `aeolus-image build --target rhevm --template template.tpl ` I saw the following error in the imagefactory.log:
libvirtError: unknown OS type hvm
I was told that RHEV requires KVM to build images. I would like to know what build methods are supported or plan to be supported for each type of provider.
As we plan the next iteration, I've been asked to organize our work
around the ability to launch an image without having to manually craft
an XML file and host it on a webserver. Instead, Conductor users should
be able to use a tool in Conductor to browse images stored in Image
Warehouse, click one of them, select a hardware profile and maybe a
pool, and launch it.
There's a semi-related feature to be able to import images. I've
proposed treating it as a separate feature so as to not confuse the
discussion, but it's also in scope for this next iteration so I'll
mention it here.
My plan, if I haven't misunderstood the request, is to start a
discussion here, and then create a wiki page once the discussion
stabilizes a bit.
The work required is something like this:
* Figure out how this should interface with the UI. I think the actual
"launch an instance" form can be very simple, but it's not abundantly
clear to me where we should place the listing of images in Warehouse.
We should solve this before we dive in on implementation.
* Interface with Image Warehouse. We're already doing this to some
extent, but two major areas came up on the call today. The first is
that a lot of what we're doing is already supported in the
aeolus-image command-line tool. To avoid code duplication and
reinventing the wheel, we'd like to break out a common library we can
both use. (This also applies to image importation.) Obviously, we'll
want to work with the Infrastructure team around aeolus-image. Second,
we endeavor to switch from our custom-written WarehouseModel code
using rest-client and nokogiri to ActiveResource.
* Actually implement it once we know where it goes, how it should look,
and have a better layer to connect to Warehouse. We'll collect the
user data, probably within the context of a pool, create a deployable
behind the scenes with this image in it, and then launch it like it's
a normal deployable.
This task should convey the following advantages:
* The workflow will be simplified for users who do not need multi-image
deployable. They will not need to create an XML file, and will not be
required to have a webserver running to host the deployable XML. I
know from experience that the whole notion of pasting UUIDs into XML
files and pushing them to webservers was not entirely popular among
* It will provide an interface for browsing Image Warehouse. Right now
(as far as I'm aware), you're resigned to either getting XML back from
Warehouse, or using the command-line tools if you want to know what
images are stored in Conductor. Both are perfectly valid, but we'll
implicitly provide a GUI.
* Ideally, we will incidentally improve the responsiveness of our
interface to Warehouse as we switch to ActiveResource, though this
remains to be seen.
* Although it's not an intentional goal, this may make it easier down
the road if we ever decide that we want to provide a GUI for creating
Let the discussion commence!
One of the proposed features for this iteration is Status Reporting.
There are four main components to it:
* Handling changes that happen on the backend provider
* Storing & displaying user actions
* Generating and exporting reports
* API to notify and get notified by external services (Matahari)
Below, I'll go over each of these and comment on what we have and what
needs to happen. At the end I wrote some features and tasks that I think
should go into Redmine.
Please, do speak your mind and write your suggestions and feedback in
Most of the functionality is already in place: when an instance crashes
or is stopped outside of Conductor, Condor picks it up and updates the
Additionally, we calculate instance's uptime and as far as I could tell,
it's as accurate as we can reasonably get without running an agent
directly on the instance (hello, Matahari).
What's missing here is mostly the UI: Instance's uptime is not displayed
at all. We do show uptime for Deployments, but that one is not
calculated correctly, so that needs to be fixed.
It would also improve the user experience, if we updated these values
automatically -- without having to reload the page.
So when you launched a deployment, you'd get the deployment-details page
that shows its status as 'pending'. Once the deployment is running, the
status would change to 'running' automatically and the uptime would
start to count.
Doing this consistently across the app would be great but isn't trivial.
would soon became a mess. Libraries like Backbone.js should help, but
it takes some time and effort getting them in. Backbone in particular
isn't a drop-in solution. Others may be. Suggestions?
Again, some of it is in place already: we have a model called Event that
is hooked to the `after_save` callback of Instance and it writes a
timestamped entry every time an instance changes.
What needs to be done here:
* Better UI
- Right now we show instance's history in the ugly & forgotten
instance#show page. It would make sense to make it look prettier, and
possibly display the events on the Deployment page for all its
* Make the event log persistent
When an instance gets deleted now, all events get deleted with it. What
we want to do instead is to hide the instance from the UI but keep it
and the events around so that we can still generate correct reports.
Matt Wagner suggested a couple of gems that should handle this
pretty transparently. They're worth looking into as doing this by
ourselves could get tricky.
* More events as needed
Generating And Exporting Status Reports
Right now, this will mean creating a simple CSV file that can be downloaded.
Possibly something like:
plus a way of accessing that from the UI.
The first implementation of this will be for admins only, thus no
permission checking is needed. Same goes for showing a history on a
single object: if you can see the instance, you should be able to see
its history as well.
However, we'll probably end up more granular ways of displaying the
data: for instance a user will want to see what happened with their
resources. Similarly, an admin may want to check history of a particular
user or a pool. Or seeing just the last moth's activity.
For this we'll need to add permission checks and some querying
capability + UI. That's out of this iteration's scope, though.
Matahari And Other External Services
Again, this is outside of this iteration's scope.
As we'll want to get notification from Matahari (and maybe other
services) we'll need a way to get notified about events that should be
logged but that Conductor itself cannot detect.
From talking to Matahari folks, it seems they need Conductor to let
them know when an instance was started and stopped.
If I understand it correctly, the whole notion of Config Server is to
have something that communicates with the in-instance agents (matahari).
Thus, we may want to hijack that and send the messages through Config
Server rather than directly.
This will save us the trouble of creating yet another communication
interface and discovering the matahari agents. Rather, we'll piggyback
on something that's already planned anyway.
As for other services notifying Conductor, this can be as simple as:
So, like, fyi, that thingy over over here has changed.
Possibly with a timestamp in the request body so that we can log when
the event happened vs. when it was received.
However, this must be tied into whatever way we implement cross-module
authentication and authorization, otherwise everyone will be able to
create phoney events.
Features & Tasks
As a user, I want to see the uptime of my instances and deployments
* Display uptime of instances in the UI
* Fix the uptime calculation for deployments
As a user, I want to see Conductor always show the current status
without having to press <F5> every time
* Implement Backbone.js or some other system that will make this easy for us
* Dynamically update the instance/deployment status across Conductor
* Dynamically update the scoreboards in the Monitor section
* Dynamically show/hide errors as they unfold happen on the backend
* Periodically update uptime values so that they're always semi up-to-date
As an admin, I want to see a report of everything that happened to instances
* Generate a report of all the Events in a machine-processable format
* Add a download link for the report into the UI
* Make this available to administrators only
As an admin, I want to get the full reports even when the instances have
* Make the event logs persistent even on instance deletion
* Preserve the data associated with the deleted object for events/reports
As an admin, I want to be able to query the status report to get to the
info I need
- note: this is probably out of scope for the Iteration 4.
* Filter by a specific timeframe
* Filter by a specific user
* Filter by a specific pool
As a user, I want to be notified on changes that happen inside of an
- note: this is out of scope for the Iteration 4.
* Implement a way of communicating with Matahari
* Provide an API for getting event notifications from external services
As a user, I want to be able to see and query events of objects that I
have access to
- note: this is out of scope for the Iteration 4.
* Display the status reports filtered by user's permissions
In order to develop an additional feature for Oz, I need one more Package
Review for python-iptables: https://bugzilla.redhat.com/show_bug.cgi?id=727305
If someone could pick that up and review it, I would owe them another beer :)
Here's a quick rundown for bundler and production WRT development/test dependencies.
At runtime, an application uses:
to have all dependencies loaded. In a Rails application, these lines can be found in the file config/boot.rb.
By default Bundler.setup loads ALL dependencies. In production this doesn't work so well as it will inevitable load in dependencies specific to development and test. There are however, two ways to prevent this.
The first way is to specify which dependency groups to load. For example:
would load only those dependencies specified in the default group. Ie, ignore any dependencies specified in named groups eg, :development, and :test.
The problem with this approach is that config/boot.rb is a Rails provided file and is subject to change between versions. It could of course be patched but aside from the overall ickiness of this, there's no guarantee that the code would remain in this file.
The other option available is designed specifically to handle production environments.
If you execute:
$ bundle install --without development test
Bundler will create a .bundler/config file in the root of the project. This file will look something like:
When Bundler is subsequently run, it will NOT attempt to load the named dependency groups (even though they are still listed in the Gemfile.lock).
Thus if we were to ship the product with this file, there would be no need to have the development/test dependencies installed on a production machine.
Additionally, I consider it good practice to ship a production Rails application that excludes specs, features, test and development environment configurations, and any test/development specific rake files.
P.S. I know it's planned but I'd also like to add my support for committing Gemfile.lock.
Hope this helps,
We just finished pushing the last of the Aeolus components into Fedora.
As soon as they propagate out to the mirrors, you should be able to
install Aeolus directly from Fedora via 'yum install aeolus-all'
As you are aware, the rails 3 port has been going on for some time. I'm
happy to report that I have now pushed the rails 3 port to the master branch.
This is going to cause some short term problems, but I'm hoping to get the
whole team involved in fixing those problems. In particular I know there is
With that said, if you want to run the new code on your existing setups,
you have 2 options:
1) Install a VM with rawhide, which will have all of the "known working"
packages already available.
2) Use gem to install the known working packages.
If you go for option 2), you'll need to know the list of gems. Attached to
this mail is the Gemfile and Gemfile.lock of the dependency list on rawhide.
The Gemfile is already in the git repository; I'll probably also be importing
the Gemfile.lock soon. You can use the attached files to "bundle install" in
Because aeolus-all won't currently work due to unmet dependencies, in addition
to the gems you need to install some packages:
yum install aeolus-configure condor-deltacloud-gahp iwhd imagefactory \
qpid-cpp-server mongodb-sever mod_ssl deltacloud-core ntp \
rubygem-aeolus-image rubygem-aws rubygem-rbvmomi
In the meantime, we'll be working to compile the necessary packages on
Fedora-14/Fedora-15/RHEL-6 so that 'yum install aeolus-all' will work again.
Let me know if you have questions about any of this.