On Mon, Jul 18, 2011 at 09:57:28AM -0400, Matt Wagner wrote:
On Fri, Jul 15, 2011 at 05:25:03PM -0400, Hugh Brock wrote:
> Mark had the very interesting idea of conceptually separating the
> Aeolus components into discrete projects, each responsible for its own
> existence and usefulness. Obviously in the short term this will be a
> conceptual separation only, but I'd like to know what folks think of
> the idea of actually splitting the projects up this way at some time
> in the future.
Conceptually and perhaps behind-the-scenes, I like the idea of having
various components that interact.
Speaking practically, though, it already drives me crazy that we have
various bits and pieces that don't necessarily fit together perfectly.
We have components written in Ruby, Python, and C. Sometimes they
interact over REST APIs, other times they use QMF, and sometimes they
just directly modify each others' databases. (Although hopefully not so
much of that last one these days.)
To give one example of where this is a problem, our interactions with
Warehouse via API can be really slow, because we end up issuing a ton of
queries to get what we need. (Although this problem is far worse in
tests than in actual usage by our app, so it's not a serious problem for
now.)
All this said, I still like the concept of having discrete components,
but only if they are all designed with a common purpose and tightly
integrated with the other components. When they become autonomous
projects and lose sight of the overarching project direction, I think it
becomes frustrating and perhaps detrimental to the project.
This is a very important point of course.
I think what we have now in some ways is the worst of both worlds. We
in fact *have* separate projects, that we are constantly trying to
integrate with all of the associated pain you describe above. The
problem is that because the project goals are so enormous and
overarching, we haven't set out goals that each component could
achieve independently of the other projects. I believe if we do a
better job of setting those goals, then integrating will become a
secondary (if still very important) task, and the folks on each
project will have more latitude to develop real features and gain a
sense of forward progress.
> Aeolus mission: To free users from cloud lock-in by making the
choice
> of cloud provider -- private, public, or hybrid -- a simple
> launch-time option.
>
> Aeolus is: A single, consistent set of tools to build and manage
> organized groups of virtual machines across clouds.
I like having both sections. They complement each other -- the mission
gives the "what" and the "why," and the "Aeolus is"
provides the "how."
> * Aeolus Conductor: A way to provide cloud resources to users, manage
> users' access to and use of those resources, and control users'
> instances in clouds. This lets users make intelligent choices about
> which cloud to use.
I think this section is good, but, in light of the above discussion
about stand-alone components, this reminds me of something I was
thinking about this weekend: if Conductor were removed from the project
and used as a stand-alone app, what exactly is it meant to do?
For example, I have a Rackspace account with some VMs I started by hand.
As things stand today, I can't use Conductor to manage them, because I
didn't start them through Conductor / Factory / Warehouse. I think this
is by design, but should it be? Would it be more useful if I could use
it to consolidate my EC2 and Rackspace dashboards, even for things not
launched through Conductor?
I think you've identified a key missing part of Conductor. There is no
earthly reason why Conductor, independently of any of the other
components in the project, shouldn't be able to connect to all the
provider accounts it knows about and show you all the images that are
available for you to launch there, as well as all the instances that
are available for you to manage there. All this information is already
available from the Deltacloud API, why not just use it?
> * Aeolus Composer: A way to build cloud-specific images from
> generic templates, so that users can choose clouds freely using
> compatible images.
This is good. If we go forward with this naming scheme, could we work on
building a spiffy little table on a website somewhere that shows which
components fit where?
Definitely...
> * Aeolus Orchestrator: A way to manage clumps of instances in
an
> organized way. Users should be able to automatically bring up a set
> of different instances on a single cloud or spanning multiple
> clouds, configure them, and tell them about each other.
On the above note, it's not clear to me what component(s) Orchestrator
maps to. (Or maybe I just haven't had my coffee yet.) It sounds like
this is a blend of Warehouse, Condor, and Conductor, though Conductor is
already mentioned above.
I probably need to make this more specific. What I really meant was
"Deployables" -- i.e. launching and managing multiple instances as a
clump. So this "Orchestrator" component doesn't really exist yet -- a
piece of it is in Conductor, a piece is in config server/Audrey, a
piece (will be in) Matahari, a piece is in Composer.
The Orchestrator as a standalone component should be able to work
*without* Conductor if you point it at a Deltacloud driver and provide
it with cloud credentials. It should be able to work without Composer
if there are already images around (presumably, in a cloud account)
for it to use.
However, if you point Orchestrator at Conductor, then it should be
able to use a Conductor pool as the cloud it's going to launch and
orchestrate things in, via the Deltacloud API Conductor will
eventually expose.
> * Aeolus HA Manager (need a better name): A way to make
instances or
> clumps of instances in the cloud highly available.
Backup Vocals? (This is a bit of a stretch, but I'm not sure that
orchestras typically have stateful hot spares for musicians.)
Heh... they do, but "Freelance Substitute Manager" is not a really
compelling name :)
> All these things are possible because of the Deltacloud
cross-cloud
> abstraction library -- but Deltacloud itself is a separate project.
We might want to give a little more mention of Deltacloud, as a sister
project or whatnot. (Can we call the "abstraction library" an API?)
> What does each Aeolus component do, specifically?
> =================================================
>
> Aeolus Conductor:
>
> * lets a user launch a VM on a cloud, track it through its
> lifecycle, and clean it up when it is no longer needed
>
> * lets a user easily check how much cloud time she has burned up,
> set thresholds for alerting her when interesting things happen
> (a VM crashed, she's about to go over n hours of usage this
> month)
Should this (long-term) support tracking of cost as well? In addition to
showing how much time you've used, we could show how much it's cost you.
(I know this is non-trivial to implement. I'm just wondering whether
it's something we want to list here. It seems like it's something of
interest to some people.)
> * lets an administrator group cloud resources for other users, so
> that they can track and control which resources are used and at what
> level.
>
> * lets a user share access to his VMs with others
Nitpick, but can we say "other users" instead of "others"?
> * lets a user make an informed choice of which of the available
> clouds to use, based on cost and QOS data and other useful
> metrics. Or, lets the user delegate that choice to Conductor
> itself
>
> * lets the user monitor software and services on the guest, and the
> health of the guest
>
> * has a truly superior GUI for doing all this
Overall, this section sounds great.
> Aeolus Composer:
>
> * lets users create a template that defines an image, which we can
> build for each cloud they are interested in
This is much clearer this time around.
> * lets users, as part of building the image, specify arguments to
> pass into the build process to configure software installation,
> and also to specify values that will be passed back into the image
> metadata to be used later
>
> * lets users build images for multiple operating systems, including
> Windows and various Linux flavors
>
> * lets users rebuild images from the original template if needed, to
> update packages for feature enhacements, bug fixes, or security
> issues
>
> Aeolus Orchestrator:
>
> * lets users specify a clump of images to launch and configure
> simultaneously. We want the user's specification to be human
> readable and portable
>
> * lets users specify values to collect at launch-time. The values
> will influence how the instances configure themselves, either by
> being interpolated into config snippets in the clump
> specification, or by being passed directly to the instance
> post-boot. Launch-time values mean the instances in a clump can be
> configured differently each time they are launched
>
> * lets users provide configuration parameters and tooling for each
> instance, to be injected and run at launch time. User-provided
> configuration tooling can do anything including install software
> packages, set config values, or start and stop services. Config
> tooling can be in any form that the user's image supports --
> puppet manifest, bash script, powershell script are all fine.
To keep the overall tone, can we s/fine/supported/ (or "possible")?
> Aeolus HA Manager:
>
> * lets users include instances or clumps of instances in an HA
> infrastructure which provides isolation, recovery, and
> notification of failed applications or instances
>
> * lets users specify policy for maintaining or relaunching services
> within an HA infrastructure
Overall, I think this document is looking really good.
-- Matt
Let me know if the explanation above makes any of this clearer. I
agree that getting the UI coordinated is going to be a sticky
point. But I think Mark would argue that part of the difficulty we've
had thus far is that we've been building UI for something that we
didn't know what it was :).
Take care,
--H
--
== Hugh Brock, hbrock(a)redhat.com ==
== Engineering Manager, Cloud BU ==
== Aeolus Project: Manage virtual infrastructure across clouds. ==
==
http://aeolusproject.org ==
"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey