RFC: Integrating Aeolus with Heat

Scott Seago sseago at redhat.com
Thu Sep 6 20:15:20 UTC 2012


On 08/28/2012 08:41 AM, Tomas Sedovic wrote:
> On 08/27/2012 02:47 PM, Jan Provaznik wrote:
>> On 08/21/2012 06:15 PM, Tomas Sedovic wrote:
>>> Hey Folks,
>>>
>>> I've been contributing to the Heat project lately and I think it's
>>> gotten to a point where we (the Aeolus community) could look at it and
>>> figure out if it's something we want to use.
>>>
>>> Please forgive me the length of this email. I tried to make it as short
>>> but there's a lot to cover. Grab your favourite beverage and sit
>>> comfortably.
>>>
>>>
>>> A little background
>>> -------------------
>>>
>>> Heat is a project that provides orchestration and high availability to
>>> OpenStack. Its API is modelled after Amazon CloudFormation.
>>>
>>> What it does in Aeolus' terms is: you create a deployable and pass 
>>> it to
>>> Heat API along with any launch-time parameters. Heat will decide on the
>>> correct order of the resources (instances, databases, load balancers,
>>> remote storage, floating IP addresses, etc.) and launches and 
>>> configures
>>> the whole thing to produce a running deployment.
>>>
>>> Heat started about five months ago and hopes to become one of the core
>>> OpenStack components. It's written in Python following the OpenStack
>>> development model and tools.
>>>
>>>
>>> What's this to do with Aeolus?
>>> ------------------------------
>>>
>>> We're trying to solve some of the same problems:
>>>
>>> * launching multiple instances in correct order
>>> * passing parameters that are only known when an instance is launched
>>> * configuring and advertising the services running inside the VMs
>>>
>>> If we could outsource this to a project whose sole focus is solving
>>> these issues, we could focus on things that make Aeolus unique: 
>>> building
>>> great UI, tools and APIs for managing cross-cloud deployments.
>>>
>>> As a side benefit, we would get the current and future features of Heat
>>> for free:
>>>
>>> * high availability
>>> * auto-scaling
>>> * load-balancing
>>>
>>>
>>> OpenStack? Huh?
>>> ---------------
>>>
>>> Heat is an OpenStack project and the developers did not put effort into
>>> making sure that it works with anything else.
>>>
>>> However, Ian Main and I looked into the code and it seems that we could
>>> integrate Deltacloud support with little changes in the Heat codebase.
>>>
>>> The Deltacloud backend would be an external package that would not
>>> pollute Heat's code and dependencies. The changes to add supports for a
>>> custom backend would be small so there's a good chance they'd be
>>> accepted.
>>>
>>>
>>> Proof of concept
>>> ----------------
>>>
>>> It would be cool if we had a working prototype to demonstrate the
>>> strengths and weaknesses as a reference for the discussion.
>>>
>>> We're not there yet but Ian and I have came up with a plan that should
>>> produce a basic prototype reasonably quickly.
>>>
>>> The first step is to get Conductor to launch multi-instance deployments
>>> using Heat, nothing else. The plan is at the end of this email.
>>>
>>>
>>> Image management
>>> ----------------
>>>
>>> Heat's image management is very similar to Aeolus'.
>>>
>>> The images are built using Oz and stored in Glance (OpenStack's image
>>> storage tool).
>>>
>>> Heat supports both installing everything at runtime from bare (JEOS)
>>> images and using preconfigured images and just launching those.
>>>
>>> So far the focus has been on the former but both approaches work and
>>> Heat developers are spending more time making the experience with image
>>> prebuilding smoother.
>>>
>>> Since Conductor supports both modes as well (prebuilt images with Image
>>> Factory, post-launch with Audrey), there shouldn't be an impedance
>>> mismatch.d
>>>
>>> We will keep building images in Image Factory, we'll pass the image IDs
>>> from Deployable XMLs to Heat that will in turn pass them to Deltacloud.
>>>
>>>
>>> What's next?
>>> ------------
>>>
>>> You can read the plan for the proof-of-concept and the long-term phases
>>> and I would like your feedback. Please post correction on anything I've
>>> missed or said wrong and anything else that comes in your mind. Do you
>>> think integrating with Heat makes sense?
>>>
>>> I want stress that this is not a done deal. I think that that Heat can
>>> be very useful to Aeolus but it's quite possible that I've missed
>>> something and we still need to see it in action before we commit to
>>> anything.
>>>
>>> There's a real possibility that this just is not a good match in which
>>> case we'll go our separate ways (unless we decide to make it a good
>>> match). Feel free to ask anything that's unclear.
>>>
>>>
>>> Gimme teh links!
>>> ----------------
>>>
>>> If you want to try Heat out, follow these steps:
>>>
>>> https://github.com/heat-api/heat/blob/master/docs/GettingStarted.rst
>>>
>>> Heat developers hang out at #heat on freenode and are all really nice
>>> chaps.
>>>
>>> The repo for experimental integration of Deltacloud into Heat:
>>>
>>> https://github.com/tomassedovic/heat/tree/multiple-backends
>>>
>>> The Deltacloud client for Python lives in the official Deltacloud
>>> repository:
>>>
>>> https://git-wip-us.apache.org/repos/asf?p=deltacloud.git;a=tree;f=clients/python;h=8f12a785cb6102c0959e4dafb73c219917e16b9a;hb=HEAD 
>>>
>>>
>>>
>>>
>>> The client is very barebones at the moment. We'll be adding the
>>> necessary features here:
>>>
>>> https://github.com/tomassedovic/deltacloud/tree/python-client
>>>
>>> and work on getting them accepted upstream.
>>>
>>> The API Heat uses at the moment is based on AWS CloudFormation. You can
>>> read the documentation here:
>>>
>>> http://docs.amazonwebservices.com/AWSCloudFormation/latest/APIReference/Welcome.html?r=2048 
>>>
>>>
>>>
>>>
>>> Here are some Deployable descriptions that Heat accepts and is able to
>>> launch (in Heat and CloudFormation terminology, these are called
>>> "templates", the deployment is called "stack")
>>>
>>> https://github.com/heat-api/heat/tree/master/templates
>>>
>>>
>>> What follows is the short-term and long-term plan on getting this done.
>>>
>>> Take care,
>>> Thomas
>>>
>>>
>>> Proof of concept
>>> ----------------
>>>
>>> ### Getting Deltacloud to work with Python ###
>>>
>>> The Deltacloud API is language-agnostic but we still need Python
>>> bindings.
>>>
>>> Deltacloud ships with a Python client but it hasn't been touched for
>>> over a year. I've sent a few patches to fix it but more work is 
>>> required
>>> for it to be usable.
>>>
>>>
>>> ### Passing the Provider credentials from Heat API to Deltacloud ###
>>>
>>> Currently, the Heat API user can authenticate using either Keystone or
>>> the EC2-style credentials.
>>>
>>> We need a new authentication handler that will receive Deltacloud-style
>>> credentials and pass them to the Deltacloud API.
>>>
>>> Ian has written a patch that does this.
>>>
>>>
>>> ### Integrating Deltacloud client with Heat ###
>>>
>>> Heat is currently using the Nova python client to launch instances, 
>>> etc.
>>> The calls are already reasonably isolated.
>>>
>>> I've written a shim module that provides the same API as the Nova 
>>> client
>>> and we hooked it up to Ian's authentication code.
>>>
>>> It's not connected to Deltacloud yet (that's the next step), but as far
>>> as majority of Heat's codebase is concerned, everything works without
>>> making a single call to OpenStack.
>>>
>>>
>>> ### Getting Aeolus Conductor to generate AWS::CFN Template ###
>>>
>>> Conductor already has all the info it needs to launch the instances
>>> stored in the Deployable XML.
>>>
>>> We'll take the XML, parse it and generate the equivalent JSON template
>>> that Heat accepts.
>>>
>>>
>>> ### Passing the generate template from Conductor to Heat ###
>>>
>>> Once we have the converted deployable, Conductor will send it to Heat
>>> via the current CloudFormation API.
>>>
>>> Heat will launch the deployment/stack via its Deltacloud bindings.
>>>
>>>
>>> ### Querying Heat data from Conductor ###
>>>
>>> Heat doesn't support any callbacks. When Conductor wants to know 
>>> details
>>> about the stack it launched, it will use the CloudFormation API to 
>>> query
>>> the data.
>>>
>>> For the proof of concept stage, we will just issue the query to Heat
>>> upon every relevant UI action: e.g. `ListStacks` when showing
>>> deployables in the UI, `DescribeStackResource` when shoving a 
>>> details of
>>> a single deployable, `DescribeStackEvents` to get deployable events, 
>>> etc.
>>>
>>
>> This is OK for POC, but it would be really nice to have callback support
>> for real integration.
>
> Yeah I agree.
>
>>
>> nit: you probably meant 'deployment' instead of 'deployable' in the
>> paragraph above.
>
> You're right, thanks!
>
>>
>>>
>>>
>>> Long-term
>>> ----------
>>>
>>> ### Dead Nova client dependencies ###
>>>
>>> Parts of the Heat code import Nova and Glance clients but they don't 
>>> use
>>> the code. These should be removed.
>>>
>>>
>>> ### Nova client exceptions ###
>>>
>>> Several libraries import the exception classes from Nova client.
>>>
>>> If we could make the Deltacloud backend to raise the same exceptions
>>> without depending on the `python-novaclient` library, that would be 
>>> cool.
>>>
>>> Barring that, we'll either have to wrap the backend exceptions or have
>>> the Deltacloud code bite the bullet and depend on the novaclient lib
>>> (the Heat package depends on it anyway so it may not be as bad).
>>>
>>>
>>> ### Make the backends swappable at the configuration level ###
>>>
>>> Similarly to how Heat can change between the AMQP implementations for
>>> Fedora and Ubuntu, we need a configuration option that replaces
>>> OpenStack with Deltacloud.
>>>
>>>
>>> ### Make the Heat Deltacloud backend a separate package ###
>>>
>>> Heat doesn't have to ship additional backends if they don't want to. 
>>> The
>>> configuration option from previous task would take a Python package
>>> namespace.
>>>
>>> That way it doesn't care if it's within the main codebase or a
>>> third-party module.
>>>
>>>
>>> ### Template format ###
>>>
>>> Aeolus uses a different format from Heat to describe deployables.
>>>
>>> We need to settle on what's the best thing to use. Options:
>>>
>>> 1) Translate Aeolus Deployable XML to Amazon CloudFormation JSON
>>>
>>> 2) Implement Aeolus Deployable XML format in Heat
>>>
>>> 3) Adopt CloudFormation JSON in Conductor
>>>
>>> 4) Have both Aeouls and Heat adopt another (open standard) format
>>>
>>> With all likelihood Heat will reject 2), we won't switch to 3) and 
>>> since
>>> *both* communities would have to accept and implement 4), we'll 
>>> probably
>>> stick to 1) for some time.
>>>
>>> Still, listing it here as it needs to be decided.
>>>
>>>
>>> ### Heat-Conductor communication ###
>>>
>>> Heat is working on implementing a proper RESTful API that will
>>>
>>> 1) be more consistent with other OpenStack components
>>> 2) suck less
>>>
>>> Once that's done, Conductor should probably adopt this API.
>>>
>>> We then need to figure out how to pass data from Heat back to 
>>> Conductor.
>>> If Conductor's launching instances via Heat, how does that information
>>> get updated?
>>>
>>> 1) Conductor asks Heat every time it needs the data
>>>
>>> 2) Conductor asks Heat periodically and caches the results
>>>
>>> 3) Heat allows Conductor to register a callback -- either via HTTP
>>> (webhooks) or via RPC -- and tells Conductor when stuff happens 
>>> (e.g. an
>>> instance was launched, crashed, etc.)
>>>
>>> 4) Conductor continues to use DBomatic to query the instances directly
>>> via Deltacloud, ignoring any additional information Heat may provide
>>>
>>> For the proof of concept, everything other than 1) is a premature
>>> optimization and then we'll see.
>>>
>>> I think 4) would be rather brittle especially since adopting Heat might
>>> possibly let us drop DBomatic entirely.
>>>
>>>
>>> ### Is Heat going to be the only way to launch deployments in Conductor
>>> or will it be an optional part? ###
>>>
>>> We will need to see how stable and usable Heat is. How easy it is to
>>> deploy and keep up (it would be another dependency, after all) and how
>>> much benefit and code cleanup it brings to Aeolus.
>>>
>>>
>>> ### Missing parts in Deltacloud ###
>>>
>>> Deltacloud provides only a subset of the information OpenStack clients
>>> have access to. A lot of these are provider-specific.
>>>
>>> We need to identify these cases and decide what to do with them:
>>>
>>> 1) Have Deltacloud provide the access for providers that support them
>>> and advertise the support properly
>>>
>>> 2) Don't let Heat provide them for non-Openstack providers
>>>
>>> 3) Implement them in Heat for providers that support them and keep
>>> Deltacloud as the lowest common denominator
>>>
>>> This includes working with the Deltacloud people: we're going to let
>>> them know what's missing and discuss the appropriate solution.
>>>
>>> Since Deltacloud API is already focused towards discoverability (e.g.
>>> when showing an instance, it lists the actions that can be done to the
>>> instance) that may be the best approach if all parties agree.
>>
>> Hi Tomas,
>> I think it's too early to say if Conductor will switch to Heat, but it
>> would be nice to have this option. I suppose that we will extract
>> launching logic out of Conductor anyway and if it will be possible to
>> use Heat as this launching backend, it would be great.
>
> Yup, that sounds great.
>
>>
>> Here is a couple of questions/concerns:
>> - the deployment launch setup becomes more complicated, a request will
>> pass through 3 standalone services:
>> Conductor -> Heat -> Deltacloud -> Provider
>> Having rack-compatible engine on top of dc-api lib mounted directly in
>> Conductor would be easier.
>>
>
> That is a valid point. I think we'll have to see whether the benefits 
> Heat brings in outweigh this.
>
>> - Heat is focused mainly on Openstack and Deltacloud backend will use
>> only subset of possible features, I wonder if it will be possible in
>> future to keep Heat's backend API compatible with Deltacloud?
>
> At least as far as the initial goal is (implement a fully compatible 
> CloudFormation API), I think so.
>
> That API and concepts within are stable. There are and will be things 
> that Deltacloud probably won't support, but since Aeolus doesn't 
> provide them now anyway, that should not be a concern.
>
> As far as the Heat-specific API and features beyond CloudFormation go, 
> I don't know. I think it's too early to tell.
>
> But it's a good think to watch out for.
>
>>
>> - What are Heat's plans about deployment launch atomicity? (what if any
>> of instance launch fails during deployment launch)
>
> CloudFormation's launching is atomic. If any of the instances fails to 
> launch, the whole thing is rolled back. This can optionally be 
> disabled per deployment launch.
>
> Heat currently doesn't support rollbacks but it will:
>
> https://github.com/heat-api/heat/issues/154
>
> The feature does not have very a high priority now, though.
>
>>
>> - Heat's HA would have to be friendly to single-instance actions, it
>> should distinguish between instance failures and actions invoked by a
>> user (if a user stops single instance, Heat shouldn't try to start it)
>
> Good point. HA is optional and enabled on a per-instance basis in the 
> deployable.
>
> If it is on, though, I'm not sure how could Heat distinguish a user 
> action from an actual failure. In both cases the instance would be down.
>
> I guess we could temporarily disable the monitoring. Alternatively, 
> Heat could extend its API to allow users to stop/restart instances 
> manually.
>
Yes, if conductor is going to allow manual start/stop of instances 
within a deployment we should definitely do this through heat. On the HA 
side, as I understand it, the idea is conductor is telling Heat "Launch 
this 3-instance deployment and make sure it stays running" -- in that 
context, the only way we could add/remove instances from the deployment 
is to augment the Heat API to allow this sort of modification. We're 
getting a bit ahead of ourselves here, though, as Conductor doesn't 
support dynamically changing the size of a deployment anyway, but it's 
something we'll need to do eventually. Basically this means we'll need 
to modify the Deployable model at some point to allow non-static numbers 
of instances. Right now, the assumption is that every assembly 
corresponds to exactly one instance. What we'd need to do is to provide 
a way of identifying the valid numbers of insances of a particular 
assembly. I.e. instead of [DB Server, WebFrontend1, WebFrontend2] we'd 
have [DBServer(1), WebFrontend(2;1-5)] -- i.e. 1 DB Server and anywhere 
from 1 to 5 front end servers, with 2 as a default. We could even allow 
"0 or more" or "0 or 1" to support optional components of a Deployable. 
Any API to allow starting/stopping instances in a deployment would need 
to stick within the Deployable's specs for number of running instances 
per assembly.
> However, from the CloudFormation (and therefore Heat) perspective, the 
> deployments are atomic. I thought Aeolus was trying to do the same 
> thing (though we do allow users to stop individual instances now).
>
Yeah -- But with an instance in a deployment stopped, strictly speaking 
the deployment is no longer "running" -- adding a more dynamic sense of 
required/allowed numbers of instances per assembly would allow certain 
instances to stop without affecting the state of the deployment. For the 
above example, you stop a web front end (leaving another), the 
deployment is still running. If you stop the DB server, though, the 
deployment is no longer in a running state.
>>
>> - it's probably out of Heat's scope, but does it check availability of
>> other provider resources like realms or hw profiles? Except periodic
>> checking of deployment/instance status, we need to check also status of
>> providers and provider realms (and maybe also hw profiles in future),
>> this is now done by dbomatic too.
>
> It checks the availability when it's launching a deployment. If you 
> specify an invalid realm or profile, the launch will fail.
>
> Heat's HA monitors individual instances, services within and deployments.
>
> I think the Heat API could be extended to notify other services about 
> these things.
>
> Of course, since we'd still be using Deltacloud, these things could be 
> monitored by their notification extension (once that's available).
>
Yeah, ideally we'd have some callback with deltacloud that would notify 
us when a Realm or HWP was added or removed (or otherwise made unavailable).
>>
>> Jan
>
Scott




More information about the aeolus-devel mailing list