On 03/27/2012 11:18 PM, Matt Wagner wrote:
>On Tue, Mar 27, 2012 at 02:32:58PM +0200, Jan Provaznik wrote:
>>Hi, sending proposal for "robust instance launching" scenario. Any
>>thoughts or improvement ideas are welcomed.
>>
>>Cut&paste from
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Robust_instanc...
>>
>>
>>Summary
>>This page describes multi-instances deployment launch process
>
>I will apologize in advance for what is likely my ignorance below, but
>I'm maybe not the only one.
>
Hi Matt, thanks for reply. Some comments are inline.
>>Owner
>>
>>Jan Provaznik (jprovazn(a)redhat.com)
>>
>>Current status
>>Targeted release:
>>Last update:
>>When launching a deployment, deployment object is created and saved,
>>Then ‘launch’ method is called on this deployment which creates
>>required instances in conductor DB and associates them w/ the
>>deployment object. Then it tries to find suitable ‘match’
>>(combination of hwp, provider account, realm) where all instances of
>>this deployment can be launched. If a match is found, launch params
>>are computed for all instances. Finally we iterate through all
>>instances and try to launch them. If any instance launch fails, we
>>set create_failed state on this instance and continue with next.
>>
>>All of above steps are not in transaction, IOW if match is not found
>>or launch params upload fails or instance launch fails, deployment
>>and instances stay created. There is not retry or fallback plan if an
>>error occurs (for example the provider of chosen match is not
>>accessible).
>
>
>So my understanding of status quo is that multi-instance deployments
>work, but with poor error handling and inflexibility on launch ordering?
>
yes
>
>>Screencast Demo
>>1) Successful deployment launch
>>All instances should be launched in proper order
>>
>>2) Launch on first provider account fails, succeeds on second
>>provider account
>>Launch two deployment's instances, third instance fails to launch
>>Launch on the first account should be rolled back -the two launched
>>instances should be stopped
>>launch should be done on the second account and should be successful
>>
>>3) Launch fails on both providers
>>Launch two deployment's instances, third instance fails to launch
>>Launch on the first account should be rolled back - the two launched
>>instances should be stopped
>>Same for second account
>>Deployment should be destroyed
>>Log for this launch should be created in a log
>>
>>
>>Implementation tasks
>>Tasks which were already in Redmine cover whole deployment launch
>>process, though may be broken into smaller tasks soon:
>>#3060 - Refactor the launch process to include better error
>>reporting, retries, switching to alternate providers etc.
>>#3061 - Ensure that the UI doesn't contain unlaunched instances
>>#3062 - Ensure that multi-instance deployments always launch fully or
>>not at all. Conductor should automatically clean-up partial
>>deployments
>>
>>
>>Detailed description
>>whole deployment launch process can be split into 3 phases:
>>1) pre-launch: we prepare deployment and instances objects (in
>>conductor db), prepare launch params and compute dependencies between
>>instances in this phase - if anything goes wrong, we just call
>>rollback, nothing is saved and user stays on launch page
>>
>>2)launch of non-blocked instances: send dc-api create instance
>>request for each instance which is not blocked. This step is done on
>>foreground together with phase 1 when a user presses "launch" button
>>(note: it’s possible to do this call from dbomatic too, if we decide
>>it’s better).
>>
>>3)launch instances on state change: instances which have not been
>>launched in phase 2. because they depend on instanceX are launched
>>when instanceX is running. This can be done from instance
>>after_update callback - when instance’s state is changed to
>>‘running’, get list of instances which becomes unblocked and launch
>>them. Phase 3 will be usually executed on background, because
>>instances states are usually updated by dbomatic (though not always -
>>in some cases instance’s state is updated directly on dc-api request
>>call).
>>If an instance launch fails for some reason, we try to deploy
>>somewhere else: stop all instances which have been already launched,
>>then find another match (skipping all matches which failed), reset
>>state to NEW for all instances (or drop and recreate them)
>
>
>What is the type of failure we're trying to guard against here? This
>seems a bit extreme, but I suspect I'm just not quite understanding.
It depends what are most common launch failures. The idea of the
proposed solution (rollback+start somewhere else) is that if you get
an error (caused by something outside conductor), there is better
chance you will success with choosing another account/provider than
retrying to launch the failed instance. IOW it expects that failure
reasons are mostly not short-term. This expectation might be
completely wrong, a problem is that I don't know what are common
failures/outages for various providers.
Jan, would it be worth trying to make this configurable? It strikes me
as a policy question that we probably can't answer correctly for each
user's situation.
(On the other hand, making things like this configurable also leads to
nightmares, so think carefully before you agree...)
--Hugh
--
== Hugh Brock, hbrock(a)redhat.com ==
== Engineering Manager, Cloud BU ==
== Aeolus Project: Manage virtual infrastructure across clouds. ==
==
==
"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey