On 03/28/2012 03:21 PM, Hugh O. Brock wrote:
> On Wed, Mar 28, 2012 at 01:59:18PM +0200, Jan Provaznik wrote:
>> On 03/27/2012 11:18 PM, Matt Wagner wrote:
>>> On Tue, Mar 27, 2012 at 02:32:58PM +0200, Jan Provaznik wrote:
>>>> Hi, sending proposal for "robust instance launching" scenario.
Any
>>>> thoughts or improvement ideas are welcomed.
>>>>
>>>> Cut&paste from
>>>>
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Robust_instanc...
>>>>
>>>>
>>>>
>>>> Summary
>>>> This page describes multi-instances deployment launch process
>>>
>>> I will apologize in advance for what is likely my ignorance below, but
>>> I'm maybe not the only one.
>>>
>>
>> Hi Matt, thanks for reply. Some comments are inline.
>>
>>>> Owner
>>>>
>>>> Jan Provaznik (jprovazn(a)redhat.com)
>>>>
>>>> Current status
>>>> Targeted release:
>>>> Last update:
>>>> When launching a deployment, deployment object is created and saved,
>>>> Then ‘launch’ method is called on this deployment which creates
>>>> required instances in conductor DB and associates them w/ the
>>>> deployment object. Then it tries to find suitable ‘match’
>>>> (combination of hwp, provider account, realm) where all instances of
>>>> this deployment can be launched. If a match is found, launch params
>>>> are computed for all instances. Finally we iterate through all
>>>> instances and try to launch them. If any instance launch fails, we
>>>> set create_failed state on this instance and continue with next.
>>>>
>>>> All of above steps are not in transaction, IOW if match is not found
>>>> or launch params upload fails or instance launch fails, deployment
>>>> and instances stay created. There is not retry or fallback plan if an
>>>> error occurs (for example the provider of chosen match is not
>>>> accessible).
>>>
>>>
>>> So my understanding of status quo is that multi-instance deployments
>>> work, but with poor error handling and inflexibility on launch
>>> ordering?
>>>
>>
>> yes
>>
>>>
>>>> Screencast Demo
>>>> 1) Successful deployment launch
>>>> All instances should be launched in proper order
>>>>
>>>> 2) Launch on first provider account fails, succeeds on second
>>>> provider account
>>>> Launch two deployment's instances, third instance fails to launch
>>>> Launch on the first account should be rolled back -the two launched
>>>> instances should be stopped
>>>> launch should be done on the second account and should be successful
>>>>
>>>> 3) Launch fails on both providers
>>>> Launch two deployment's instances, third instance fails to launch
>>>> Launch on the first account should be rolled back - the two launched
>>>> instances should be stopped
>>>> Same for second account
>>>> Deployment should be destroyed
>>>> Log for this launch should be created in a log
>>>>
>>>>
>>>> Implementation tasks
>>>> Tasks which were already in Redmine cover whole deployment launch
>>>> process, though may be broken into smaller tasks soon:
>>>> #3060 - Refactor the launch process to include better error
>>>> reporting, retries, switching to alternate providers etc.
>>>> #3061 - Ensure that the UI doesn't contain unlaunched instances
>>>> #3062 - Ensure that multi-instance deployments always launch fully or
>>>> not at all. Conductor should automatically clean-up partial
>>>> deployments
>>>>
>>>>
>>>> Detailed description
>>>> whole deployment launch process can be split into 3 phases:
>>>> 1) pre-launch: we prepare deployment and instances objects (in
>>>> conductor db), prepare launch params and compute dependencies between
>>>> instances in this phase - if anything goes wrong, we just call
>>>> rollback, nothing is saved and user stays on launch page
>>>>
>>>> 2)launch of non-blocked instances: send dc-api create instance
>>>> request for each instance which is not blocked. This step is done on
>>>> foreground together with phase 1 when a user presses "launch"
button
>>>> (note: it’s possible to do this call from dbomatic too, if we decide
>>>> it’s better).
>>>>
>>>> 3)launch instances on state change: instances which have not been
>>>> launched in phase 2. because they depend on instanceX are launched
>>>> when instanceX is running. This can be done from instance
>>>> after_update callback - when instance’s state is changed to
>>>> ‘running’, get list of instances which becomes unblocked and launch
>>>> them. Phase 3 will be usually executed on background, because
>>>> instances states are usually updated by dbomatic (though not always -
>>>> in some cases instance’s state is updated directly on dc-api request
>>>> call).
>>>> If an instance launch fails for some reason, we try to deploy
>>>> somewhere else: stop all instances which have been already launched,
>>>> then find another match (skipping all matches which failed), reset
>>>> state to NEW for all instances (or drop and recreate them)
>>>
>>>
>>> What is the type of failure we're trying to guard against here? This
>>> seems a bit extreme, but I suspect I'm just not quite understanding.
>>
>> It depends what are most common launch failures. The idea of the
>> proposed solution (rollback+start somewhere else) is that if you get
>> an error (caused by something outside conductor), there is better
>> chance you will success with choosing another account/provider than
>> retrying to launch the failed instance. IOW it expects that failure
>> reasons are mostly not short-term. This expectation might be
>> completely wrong, a problem is that I don't know what are common
>> failures/outages for various providers.
>
> Jan, would it be worth trying to make this configurable? It strikes me
> as a policy question that we probably can't answer correctly for each
> user's situation.
>
> (On the other hand, making things like this configurable also leads to
> nightmares, so think carefully before you agree...)
>
> --Hugh
>
Make it configurable would be nice and probably not so difficult.
Also depends on which level it should be configurable:
1) config file - no additional work
2) provider or provider account attribute - a user could set number of
retries on provider's page
3) deployment attribute - a user could set number of retries when
launching a deployment
I think 3rd option would be most useful. I'll send updated version of
this doc soon.
Jan
Hi, resending only updated part, whole (almost unchanged) text can be
found on the wiki page
(
).
Instance launch retries
If an error occurs when launching an instance, it makes sense to try
re-launch the instance first before doing rollback - especially if it's
a deployment with multiple instances and some of them are already
running. This might not be true in all situations because:
- retry won't help in many error situations
- each retry means another delay in launch process, some users may
prefer to not waste time by retrying especially if they have more than
one provider which they can use
So this retry option would be ideally configurable by user and its
implementation could be split into this tasks:
1) extend deployment model with an attribute to remember number of
remaining retries
2) invoke instance relaunch from dbomatic in a constant interval (1 min)
for instances which have number of remaining retries > 0 (and decrease
this number with each retry)
3) add "number of retries" input field into deployment launch page, a
reasonable default val is used by default (2 retries?)
TODO: maybe it will be handy to add a new instance state for instances
which failed but will be retried, not sure about this yet.
Jan