Dear Aeolus community,

I've sent the first pull request integrating Heat with Conductor. Please have a look at the changes here:

https://github.com/aeolusproject/conductor/pull/424

The pull request contains some observations, notes and rambling that should start a discussion on things I missed, did incorrectly, etc. For your convenience, I'm pasting it here as well:

This is the first step towards full Heat integration. This routes deployment launching and querying for status to Heat. Launch-time parameters should just work, ditto for deleting the instances.

This is mostly to give you guys a direction where I think this should go and to get comments and feedback. I don't think it's wise to merge it just yet -- we need to test it more, clean up the code, find the subtle things this breaks, etc.

There are also a few issues that need fixing:

* Heat's API supports atomic launch (i.e. rollback) but as far as I know, it's not been implemented yet * There's a bug in Heat that means the actual instance status isn't reported back correctly. If you launch a deployment and stop the instances out of band, Heat will show them as running. * Heat generates its own wealth of events that we should show in Conductor. They're currently being ignored.

There's also an issue of performance. The way it's implemented now, every time you ask a deployment or an instance for a status, they request the Heat API for an up-to-date value.

This happens a lot of times and it crawls the entire UI into a halt.

I tried caching the value for the lifetime of the object as a simple workaround for rendering, but apparently, we (or Rails) are instantiating the same Deployment and Instance model objects over and over again when rendering the view so this didn't help much.

The same issue is exists with a database backend as well, but Rails transparently caches the DB queries for each controller action.

As a band-aid, I hooked `Heat::heat_request` to the Rails cache which deals with the issue.

Eventually though, we need to think of a better way to solve it -- either by reducing the number of instances we create or by introducing a proper HTTP caching layer.

Heat's API currently doesn't have any querying capabilities, so to list a few deployments, you have to send a separate request for each and then another request per instance to get instance properties as well. We could work with the Heat community to extend the API in a way that would mean issuing fewer requests.

Before we commit to any optimizations though, it's better to have the raw thing so that we can measure where the bottlenecks actually lie. The advantage of using the built-in Rails caching API here is that you can disable it in the app config without changing anything in the code.

Try it out ------------

You need Heat running with Deltacloud as a backend. I'll write a proper guide tomorrow (I need to package a couple of Python libraries to make things easier).

If you're adventurous, you can go ahead and follow the respective guides:

You can get Heat from here: https://github.com/openstack/heat

No need to set up OpenStack, we're not using it. Though Heat does use a few OpenStack libraries for dependencies.

Next, install `deltacloud_heat` which is a Deltacloud backend for Heat. It in turn uses the Deltacloud Python client.

https://github.com/tomassedovic/deltacloud_heat

https://github.com/tomassedovic/deltacloud

There were some bugs fixed in the Deltacloud API that we require, you need at least version 1.1.0. We also need the latest Python client which I didn't get to submitting upstream yet. In short, use the server and the client from my fork above.

To configure Heat to use Deltacloud, open the `/etc/heat/heat-engine.conf` file and add the following line at the end:

cloud_backend=deltacloud_heat.clients

Then open `/etc/heat/heat-engine.conf` and uncomment the last two lines:

[paste_deploy] flavor = custombackend

Start the Heat engine and api services (if you're going to use the Audrey features, start cloudwatch and cfn-api as well).

That should be it.