RFC: Launching deployments with interdependent instances

Jan Provazník jprovazn at redhat.com
Wed Apr 11 13:49:39 UTC 2012


Copied from 
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Launching_deployments_with_interdependent_instances

Summary
It's possible to launch multi-instance deployments in Conductor. A user 
can define parameters for each instance in deployable XML, there are 3 
types of parameters:
* constant values - param value is defined directly in the XML
* user defined - a user passes value through Conductor UI when launching 
a deployment
* output from other instance service - value is generated by other 
instance/service
Things get complicated with the last one - if a service of an instance 
depends on a value produced by other service running on other instance, 
we get dependencies between these instances and we have to make sure 
that services are launched in proper order.

Owner
Owner(s) of the feature - name, possible contact info

Current status
Targeted release:
Last update:
Most of dependency stuff is handled by Audrey. Conductor launches all 
instances at once, Audrey agent takes care of starting services in 
proper order. With the 1.0 release, Audrey takes an "all-or nothing" 
approach to starting services on an instance. This means that if one 
service is still waiting for input parameters (e.g., instance "A" hasn't 
yet report "hostname" to the config server), then none of the 
audrey-ified services will start (this will change in post-1.0 release): 
we're making services start independent from each other. So, in the case 
where B.httpd is awaiting ${A.hostname}, all other services on instance 
"B" can already be started.

Screencast Demo
* attempt to create a deployable with deadlock dependencies between 
instances, the deployable should not be created, error should be displayed
* launch a deployment with instances I1 and I2, each with 2 services S1, 
S2 and this dependencies: I1S1 depends on I2S2, I2S1 depends on I1S2. 
Show that all services are launched.
* show a representation of instance relationships


Implementation tasks
* extend deployable XML format to define what params are returned by 
each service - currently list of returned params is defined per 
instance, not per service
* #3054 - add check if all params in a deployable can be satisfied and 
if there are no deadlocks
* #3056 - Represent the relationships in complex deployables in the 
Conductor UI


Detailed description
Conductor has to check:
*if there are no deadlocks between services in a deployable
*if all params which reference output values can be satisfied (there are 
not references to not existing params)
This check can be done when creating or updating a deployable. We don't 
have to take care about dependencies on Condcutor side when launching a 
deployment, we can launch all instances at once - Audrey will take care 
of launching instances in proper order.

Conductor can use Audrey API 
(https://www.aeolusproject.org/redmine/projects/audrey/wiki/Audrey_Reporting_API) 
to get current state of services. There should be some (configurable) 
timeout for all services startup, if a service is not started at this 
time, the deployment should be considered as failed and 
rollback/relaunch should be (optionally) initiated. TODO: should be 
there be an additional instance state for representing an instance which 
is running but not all services are started or should be there an 
additional instance attribute to keep track of not running services?


High-level implementation details
Deadlock detection
Could be launched from deploayble's before_save callback. It's in fact 
only about finding cycles in an oriented graph.

Reletionships representation
Might be represented by a graph, something like this: 
http://www.adp-gmbh.ch/misc/tools/graphviz/cluster.html (this example is 
generated by graphviz which is packaged in fedora together with graphviz 
ruby lib)


Future plan
It's possible that Heat API will be used in future for launching 
deployments, check "Future plan" on Robust_instance_launching






More information about the aeolus-devel mailing list