RFC feature planning - robust instance launching

Jan Provaznik jprovazn at redhat.com
Wed Mar 28 14:55:04 UTC 2012


On 03/28/2012 03:21 PM, Hugh O. Brock wrote:
> On Wed, Mar 28, 2012 at 01:59:18PM +0200, Jan Provaznik wrote:
>> On 03/27/2012 11:18 PM, Matt Wagner wrote:
>>> On Tue, Mar 27, 2012 at 02:32:58PM +0200, Jan Provaznik wrote:
>>>> Hi, sending proposal for "robust instance launching" scenario. Any
>>>> thoughts or improvement ideas are welcomed.
>>>>
>>>> Cut&paste from https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Robust_instance_launching
>>>>
>>>>
>>>> Summary
>>>> This page describes multi-instances deployment launch process
>>>
>>> I will apologize in advance for what is likely my ignorance below, but
>>> I'm maybe not the only one.
>>>
>>
>> Hi Matt, thanks for reply. Some comments are inline.
>>
>>>> Owner
>>>>
>>>> Jan Provaznik (jprovazn at redhat.com)
>>>>
>>>> Current status
>>>> Targeted release:
>>>> Last update:
>>>> When launching a deployment, deployment object is created and saved,
>>>> Then ‘launch’ method is called on this deployment which creates
>>>> required instances in conductor DB and associates them w/ the
>>>> deployment object. Then it tries to find suitable ‘match’
>>>> (combination of hwp, provider account, realm) where all instances of
>>>> this deployment can be launched. If a match is found, launch params
>>>> are computed for all instances. Finally we iterate through all
>>>> instances and try to launch them. If any instance launch fails, we
>>>> set create_failed state on this instance and continue with next.
>>>>
>>>> All of above steps are not in transaction, IOW if match is not found
>>>> or launch params upload fails or instance launch fails, deployment
>>>> and instances stay created. There is not retry or fallback plan if an
>>>> error occurs (for example the provider of chosen match is not
>>>> accessible).
>>>
>>>
>>> So my understanding of status quo is that multi-instance deployments
>>> work, but with poor error handling and inflexibility on launch ordering?
>>>
>>
>> yes
>>
>>>
>>>> Screencast Demo
>>>> 1) Successful deployment launch
>>>> All instances should be launched in proper order
>>>>
>>>> 2) Launch on first provider account fails, succeeds on second
>>>> provider account
>>>> Launch two deployment's instances, third instance fails to launch
>>>> Launch on the first account should be rolled back -the two launched
>>>> instances should be stopped
>>>> launch should be done on the second account and should be successful
>>>>
>>>> 3) Launch fails on both providers
>>>> Launch two deployment's instances, third instance fails to launch
>>>> Launch on the first account should be rolled back - the two launched
>>>> instances should be stopped
>>>> Same for second account
>>>> Deployment should be destroyed
>>>> Log for this launch should be created in a log
>>>>
>>>>
>>>> Implementation tasks
>>>> Tasks which were already in Redmine cover whole deployment launch
>>>> process, though may be broken into smaller tasks soon:
>>>> #3060 - Refactor the launch process to include better error
>>>> reporting, retries, switching to alternate providers etc.
>>>> #3061 - Ensure that the UI doesn't contain unlaunched instances
>>>> #3062 - Ensure that multi-instance deployments always launch fully or
>>>> not at all. Conductor should automatically clean-up partial
>>>> deployments
>>>>
>>>>
>>>> Detailed description
>>>> whole deployment launch process can be split into 3 phases:
>>>> 1) pre-launch: we prepare deployment and instances objects (in
>>>> conductor db), prepare launch params and compute dependencies between
>>>> instances in this phase - if anything goes wrong, we just call
>>>> rollback, nothing is saved and user stays on launch page
>>>>
>>>> 2)launch of non-blocked instances: send dc-api create instance
>>>> request for each instance which is not blocked. This step is done on
>>>> foreground together with phase 1 when a user presses "launch" button
>>>> (note: it’s possible to do this call from dbomatic too, if we decide
>>>> it’s better).
>>>>
>>>> 3)launch instances on state change: instances which have not been
>>>> launched in phase 2. because they depend on instanceX are launched
>>>> when instanceX is running. This can be done from instance
>>>> after_update callback - when instance’s state is changed to
>>>> ‘running’, get list of instances which becomes unblocked and launch
>>>> them. Phase 3 will be usually executed on background, because
>>>> instances states are usually updated by dbomatic (though not always -
>>>> in some cases instance’s state is updated directly on dc-api request
>>>> call).
>>>> If an instance launch fails for some reason, we try to deploy
>>>> somewhere else: stop all instances which have been already launched,
>>>> then find another match (skipping all matches which failed), reset
>>>> state to NEW for all instances (or drop and recreate them)
>>>
>>>
>>> What is the type of failure we're trying to guard against here? This
>>> seems a bit extreme, but I suspect I'm just not quite understanding.
>>
>> It depends what are most common launch failures. The idea of the
>> proposed solution (rollback+start somewhere else) is that if you get
>> an error (caused by something outside conductor), there is better
>> chance you will success with choosing another account/provider than
>> retrying to launch the failed instance. IOW it expects that failure
>> reasons are mostly not short-term. This expectation might be
>> completely wrong, a problem is that I don't know what are common
>> failures/outages for various providers.
>
> Jan, would it be worth trying to make this configurable? It strikes me
> as a policy question that we probably can't answer correctly for each
> user's situation.
>
> (On the other hand, making things like this configurable also leads to
> nightmares, so think carefully before you agree...)
>
> --Hugh
>

Make it configurable would be nice and probably not so difficult.
Also depends on which level it should be configurable:
1) config file - no additional work
2) provider or provider account attribute - a user could set number of 
retries on provider's page
3) deployment attribute - a user could set number of retries when 
launching a deployment

I think 3rd option would be most useful. I'll send updated version of 
this doc soon.

Jan



More information about the aeolus-devel mailing list