On Thu, Oct 25, 2012 at 04:32:35PM -0400, Hugh Brock wrote:
Now... let's take a look at Conductor. Conductor:
* Provides a UI for building and pushing images and wrapping them in
deployables
* Provides a UI for launching deployables and setting launchtime
parameters for them
* Provides a UI for managing deployables and instances while they are
running and tracking them over time
* Provides the back end facilities for doing the launching itself,
handling failures and exceptions, and actually setting the config
values post launch
* Provides an extensive and complex "matching" mechanism that, given a
user's privileges, the requirements of the application being launched,
the relative weight of the particular cloud providers that are
available to launch on, and the user's quota on each provider, will
choose a suitable cloud provider to launch on
* Provides all the UI required to manage the inputs to the matching
mechanism
* Provides full RBAC services for all of the entities in the system
* Provides a UI for managing the RBAC system
* Provides an event logging facility so that users can retrieve and
track instance lifecycle events
* Provides graphs of the event logs
Clearly Conductor fails lesson 1. With more than 50 model classes and at
least one pair of controllers that actually do the same thing, it also
fails lesson 2.
Finally, let's look at the umbrella project, Aeolus. Aeolus clearly
fails lesson 3 because historically *I* have provided the governance
with my mighty ><}}}*>. While I find this method of governance quite
satisfying, I am forced to admit that it neither scales nor works for an
upstream project where there might be people who do not work for
me. Sigh. (Scales, get it? The ><}}}*> doesn't *scale*. I kill me!)
Does Aeolus pass Lesson 4? If not, we should all hang it up and go do
something else. Fortunately, I think Aeolus does pass lesson 4,
handily. I think there is real need for a set of cross-cloud tools, from
small shops that have nothing but an EC2 account and a few in-house
machines running VMWare, to individual developers who want to launch a
multi-tier app on whatever the cheapest public cloud happens to be at
the time, to larger organizations that want to provide a fully managed
self-service environment to internal users. I think Aeolus is well
positioned to fill this need, and I've seen other closed source tools
that do so I'm confident the need is real.
What about lesson 5? We have tried to make some headway here but we've
not done well at coming up with a good community marketing plan and
sticking with it. My hope is that improving our score on lesson 3 will
help with this.
So, where does this leave us?
1. We need to fix the governance. This means that you folks, the Aeolus
developers, need to figure out what model we are going to follow for
making architectural decisions, API decisions, coding style decisions,
and so on, put it in place, and stick to it. I would suggest that that
model be more sophisticated than "argue about it on the mailing list and
never come to a conclusion," which is what we've done to date.
2. Once we've fixed the governance, I believe we need to fix Conductor,
and I believe that fixing Conductor requires breaking it into individual
pieces that do one thing, well. One set of suggestions for those pieces,
which you may or may not agree with, is the following:
* A standalone self-service portal (i.e. Winged Monkey). Provides
management of single instances or n-tier apps. Will work all by itself
in front of rhev-m, openstack, vsphere, ec2, rackspace, or will work
connecting to Heat connected to one of those things, or will work
connected to Conductor. Appeals just as much to the individual
developer or guy-with-a-webapp as to the corporation with
self-service users connecting to Conductor pools.
* A standalone catalog manager service. Provides API to retrieve deployables,
resolve the templates they require into actual build images (via
querying Tim presumably), answer the question "what can I launch on
this cloud."
* A standalone app design/catalog manager UI that would provide the app
designer a GUI for Tim and the catalog manager
* A launching/state tracking mechanism -- i.e. Heat. Takes a
CloudFormations template and launches it on a given cloud. Tracks the
state of the instances in the template and manages failures
appropriately (by restarting, auto-scaling, etc.)
* An event tracking service. Provides up-to-date views into instance and
app events collected over time via an API. Clients would include
Winged Monkey and other sophisticated reporting services.
* A cloud abstraction layer or "synthetic cloud" provider (the real meat
of Conductor). Given configuration input by an admin, aggregates a set
of cloud accounts and providers into a single cloud that it presents
to the user via an API. Allows admins to shape provided clouds
according to cost and resource needs, provider track record
("quality"), quota, other factors.
* Other thin UI layers as appropriate. I would prefer to see us move to
the pattern OpenStack and OpenShift and Katello (to some extent) are
using here, which is a set of services that provide APIs only
(probably REST but I don't really care to be honest) and a stateless
UI layer in front -- model code, sure, but no access to storage other
than via the constituent projects that make up the back end. (This is
another reason we need better governance.)
I am happy to read especially the last 3 paragraphs. Actually yesterday
morning (EST) we had a discussion in Brno regarding the inputs we got:
1) your statements about the overall goal from the IRC last week
Actually I was surprised when some of the Brno boys told me that they
did not hear anything like that previously. And that their ideas about
the project where not perfectly aligned with what you said.
2) the Winged Monkey thing which clearly shows what Conductor is not.
In my personal view, that is still an outside view, Conductor
architecture need to be discussed and a top level architecture has to be
agreed upon which should be in many aspects different to what we have
now.
You outlined a vertical division of Conductor -- well yes that is good
but much more important to me seems the horizontal division that you
talk about in the last but one paragraph.
I suggest an architecture where Conductor is split horizontally with a
very rough cut.
I see it (again I am looking from outside so surely I miss many
details):
[cloud provider1] [cloud provider2] ......
[Image Factory] [DC] [another external service/project]
[CONDUCTOR CORE]
[CLI tools] [CONDUCTOR UI]
The conductor is that you call "a cloud abstraction layer" -- it is an
application that provides a set of well defined APIs to the presentation
layers.
It can be seen it the terms of Rails as a set of models and very thin
controllers if you like.
It should be further vertically split lets say into a set of classes
that encapsulate the API: C1, C2, C3... Cn
Like this:
[CONDUCTOR CORE]
C1 C2 C3 ..... Cn
that are consumed by the CLI tools and the Conductor UI
The Contuctor UI has 2 important properties:
1) it is thin in the sense that is is just the user interface
2) is is vertically split into parts like
i) provider account administration UI
ii) template editing UI
iii) something like Winged Monkey launch pages
iv) whatever
In the Rails terminology it could be an Engine that consists of a thin
controller(s) and views.
I want to stress the difference between
1) a controller in the Conductor Core -- that exposes a functionality
implemented by the classes C1...Cn
2) a controller in the Conductor UI whose job is to support the WUI.
it support the browser view all those javascript that make things
in the page change in time (pushes events), implements pagination,
deals with Wizard-like interface where you do things in steps etc.
This clearly separates the cloud abstraction layer logic from the logic
of the web interface.
So we have:
[CONDUCTOR CORE]
C1 C2 C3 ..... Cn
[CLI tools] [engine 1][engine 2][engine 3][engine n]
and from the point of view of a user an App can by made of 1 or more
engines.
each engine might use one or more of the conductor core APIs e.g. engine
1 may need C1 and C3 etc.
e.g. the Winged Monkey like App would be made of the Launch engine and
the View Current State engine etc.
I abstract from the technology that exposes the interfaces of C1,C2...Cn
to both the CLI and the Conductor UI engines.
It might be REST, it might be some RPC (SOAP) whatever. I don't really
care at this point. Although I personally consider using REST internally
inside a process a dull thing to do -- I would prefer direct Ruby method
invocation.
A point on the 3rd paragraph from bottom: One of the question I had when
I saw the architecture picture in the Aeolus wiki was "where is the
message bus"?
The nature of the problem brings us many things that happen
asynchronously, furthermore the componentization adds more complexity to
the communication. I see here the patterns of producer - consumer,
subscribers etc. I see places where you need to poll. And in the code
... I see the dbomatic and delayed job.
Well it might be an unwanted extra external dependency to add a
industrial strength message broker to the project, but you have other
options like the LISTEN/NOTIFY interface in the PostgreSQL and surely
others.
So I hope, and I am sure others hope too, that the overall architecture
(redesign) is going to be a major topic following month in Brno!
Regards.
--martin
--
Martin Povolny <mpovolny(a)redhat.com>
tel. +420 777714458