RFC: Pluggable modules to allow administrators to select their preferred provider selection algorithm

Matt Wagner matt.wagner at redhat.com
Thu Apr 12 19:44:43 UTC 2012


On Thu, Apr 12, 2012 at 02:08:53PM +0200, Martyn Taylor wrote:
> On 04/12/2012 01:23 PM, Jan Provaznik wrote:
> >On 04/05/2012 06:44 PM, Angus Thomas wrote:
> >>We should present administrators, with the ability to configure the
> >>launch-time provider account selection policy for a specific pool, and
> >>to set a global default policy to apply in pools where no custom policy
> >>is defined.
> >>
> >>The policy would be applied after a set of viable provider accounts has
> >>been identified. Those will be the provider accounts to which the
> >>relevant images have been pushed, and for which a set of hardware
> >>profile matches can be made etc..
> Should we not do this selection at the Hardware Profile level since
> this is the lowest level object from which we can make a decision.
> i.e. even after selecting a provider we might still have a list of
> Hardware Profiles to decide from.  If we're to make the decision on
> the hardware profile level we can get all the information such as
> provider and account then make a suitable decision.

Yes, I'm thinking the same thing here -- we should include both provider
and HWP in the equation. What I'm not sure about, and perhaps it's not
something we have to decide just yet, is at what stage to do this.
Should we pick a provider and then pick the best HWP within that
provider? Or should we group them from the start and operate based on a
combination of HWP and Provider as one object? (The latter sounds more
complicated, but also more optimal -- the best provider overall might
have a non-ideal HWP. I'm not positive how to make this work, though.)

> >>The selection policy should work by defining a probability distribution,
> >>stating how likely each provider account is to be selected to host the
> >>new deployment, expressed as a percentage. Once those percentage are
> >>calculated, Conductor should pick a random number between 1 and 100 and
> >>attempt to launch on the lucky provider account.
> I don't get this?  This is adding some randomness into the decision;
> surely a user would want this to be explicit.

This took me a while to come around to agreeing with. And for extremely
small setups, the injection of randomness might not be the best idea.

The problem this is solving (preempting?) is that you would otherwise
always send *all* instances to whichever provider had the best score, so
the next-best would never get anything.

> >>Using a probability range and randomly selecting within that range might
> >>seem counter-intuitive: Having done the maths and assigned a numerical
> >>probability to each provider account, based on its suitability to host
> >>the deployment, why not just launch on the “best” provider? The issue is
> >>one of scale. When considering a single launch, selecting the account
> >>which gathered the highest score makes sense, but once Conductor is
> >>managing a large volume of deployments, the downside of that approach
> >>becomes clear - If one provider account gathered more than 50% of the
> >>probability ranking, it would get 100% of the instances, without the
> >>randomness.
> True, though using another method like below sounds better to me.
> The thing is that a user would have to have 1000s of deployments in
> order for the distribution to be equivilent to the weighting.  Why
> not use the information that is in conductor to decide when to use
> the cloud.  I.e. if we have 2 deployments in provider 1 then deploy
> to provider 2?

I'd be interested in seeing how practical this is. I'm not sure we need
truly thousands of deployments before it makes sense, but I agree that
for, say, 5 instances, it might be erratic. (Though it's only the subset
of providers that are usable, so it's not like we'll ever match you onto
an awful provider.)

If it ends up being fairly inexpensive to look at a bit more of our
data, I'm all for not invoking randomness. But if it becomes complex
(either to implement, or just computationally-expensive), I think a
weighted-random approach is a clever workaround.

> >>Whilst the various policies should be stackable, one of the two
> >>following policies should be the initial basis for the calculation:
> >>
> >>*Round robin, with optional weighting: *
> >>
> >>With this policy, Conductor would use each of the available provider
> >>accounts equally, by assigning the same probability to each of them.
> >>Varying the probabilities, to assign a weighting, would be useful in
> >>instances where the private cloud providers associated with each
> >>provider account are of differing sizes. e.g. Three vSphere clusters,
> >>one of which has double the capacity of the other two. In that
> >>circumstance, the Administrator could adjust the weighting ratios to
> >>more closely reflect the actual capacities of each cluster.
> >>
> >>It is worth noting that this isn't strictly round robin. The provider
> >>accounts wouldn't be selected in strict rotation, though the overall
> >>result is the same.
> >>
> I guess this is similar to what I described earlier.
> >>*Least used, with optional weighting: *
> >>
> >>This policy would make most sense in scenarios where Conductor is the
> >>sole means by which instances are launched on private cloud providers.
> >>Conductor would seek to ensure that the usage of the providers was
> >>balanced, by giving a higher probability to whichever provider accounts
> >>are currently least used. As with round robin, the weightings could be
> >>adjusted to reflect differing capacities between providers.
> >>
> >>Having used on of those two policies to acquire an initial set of
> >>probabilities, administrators could then elect to apply additional
> >>policies, including:
> >>
> >
> >I wonder which policy is better. It might be quite difficult to
> >tune usable penalty/bonus points for "Least used" policy. If an
> >account fails repeatedly, it gets some penalty points for each
> >fail, but also some bonus points because it's used least. So
> >"Punishing failure" described below will not have big effect?
> Rather than trying to invent our own why don't we look at some of the
> standard policies offered by cloud providers.  If we really want to
> ship with a few policies then we're more likely to get an idea of
> what people actually use.

I'm not quite sure what you're referring to about policies offered by
cloud providers. Do they publish something similar to what we're trying
to do?

Jan's point about policies cancelling each other out is interesting, and
it feels like something that could come up in other places.

> >>*Assigned priority: *
> >>
> >>The probability assigned to each provider account would be adjusted
> >>according to the provider accounts' priority, by increasing the
> >>probability ranking percentage of the higher priority provider accounts
> >>at the expense of the lower priority ones
> >>
> >>*Punishing failure: *
> >>
> >>Once the audit history records past failures, for each occurrence of a
> >>launch failure within a configurable period (6 hours feels reasonable),
> >>a provider account would be fined 5% from its probability ranking. This
> >>would serve to reduce the attempts to launch on a provider which is
> >>running out of capacity, or experiencing hardware issues etc.
> >>
> >>*Cost *
> >>
> >>There are three principle cloud uses which can incur costs: consumption
> >>of network bandwidth, consumption of storage and running a VM.
> >>
> >>Happily, only one of these needs to be a factor when Conductor is
> >>selecting a provider account to launch: the cost of running the VM.
> >>
> >>The amount of network bandwidth that a deployment will consume is pretty
> >>much unknowable at launch time. And, if it is known because, for
> >>example, a deployment is for a streaming media server, then
> >>Administrators can minimize costs by only launching that deployment in a
> >>"Low cost bandwith" pool.
> >>
> >>As long as we're not supporting deployments which include the allocation
> >>of additional storage, the costs of storage consumption are an issue to
> >>consider at build & push time, rather than at launch time.
> >>
> >>So, in order to allow cost to be another factor which affects the
> >>probability rankings, all we need is a cost per realm, per hour, for
> >>each provider hardware profile, for each provider account.
> >>
> >>Admins are going to have to enter that data themselves. That's not as
> >>onerous as it sounds, given that, for example, it will often be the case
> >>that costs will not vary across realms, so the UI can help by
> >>pre-filling.
> >>
> >>Clearly, for private clouds, no alternative means for getting pricing
> >>data into Conductor exists. For public providers, it would be beneficial
> >>if their APIs exported list pricing, however:
> >>
> >>- Few organizations which operate on a scale which justifies using
> >>Aeolus are likely to be paying list price
> >>
> >>- Organizations may wish to store and export the adjusted costs that
> >>they'll be assigning to users, rather than the basic costs appearing on
> >>the provider's monthly invoice.
> >>
> >>Once the cost data is in Conductor, adjusting the probabilities of each
> >>provider account to favour whichever provider could more cheaply host
> >>the specific range of hardware profiles would be a relatively simple
> >>matter of increasing the selection probability percentages of cheaper
> >>provider accounts, by a configurable amount, at the expense of the more
> >>costly provider accounts.
> >>
> >>Having completed the stack of modules' calculations, the result is a
> >>final set of probabilities. At this point, Conductor would roll the
> >>loaded dice and attempt to launch on the winner.
> >>
> >>The UI to allow Administrators to enable modules, and to tune the
> >>parameters associated with them, could give a real-time representation
> >>of the effect of the current settings for a specific deployable. A
> >>certain type of Administrator would be very happy, tuning options and
> >>seeing an immediate change in, for example, a pie chart, which showed
> >>the resultant probability ranking percentages.
> >>
> >>
> >>In future, we could provide Administrators with an interface to
> >>implement their own selection modules. They might choose, for example,
> >>to vary the selection probability percentages according to time and
> >>date, to increase usage of private cloud at times when they would
> >>otherwise be relatively idle.
> This is all good information for hwp/provider selection, but we will
> never know what an end user might want to do.  Why don't we design
> the policy to be pluggable from the off, we can ship with some
> defaults, and some documentation on how to write policy components,
> using the stock ones we provider as examples?  We could then later
> add some console to help with this?

Yes, I agree with this part -- I think it's very important that
administrators be able to write their own from the start. I think we can
provide a decently-usable set from the outset, but I think the ability
to write application-specific plugins is important.



More information about the aeolus-devel mailing list