[Beaker-devel] Towards "Grade of Service" metrics for Beaker

Nick Coghlan ncoghlan at redhat.com
Wed Jul 3 05:40:01 UTC 2013


A full Beaker installation is a complex beast - it can be hard to
summarise neatly whether or not it is meeting its goals. While making
the scheduling algorithm more sensible (and a bit closer to the way
Condor works) is already a near term goal, my long term goals are more
ambitious - I want to be able to collect useful metrics from a Beaker
instance and use that to decide when the scheduling algorithm is "good
enough" :)

A large part of my previous experience was working on a HF radio
communications system
(http://www.defence.gov.au/dmo/esd/jp2043/jp2043.cfm). We handled a wide
variety of different traffic types, but there was an overarching notion
called "Grade of Service".

That in turn was a combination of two things: Quality of Service and
Speed of Service.

Quality of Service was a fairly basic yes/no distinction: did the system
do what it was supposed to do (send a message, connect a call, etc)?

In the context of Beaker, the key "Quality of Service" criterion would
be "successfully provisioned and started running a recipe set".

Speed of Service was then about "Was the service provided in a timely
fashion?", and the required SoS varied depending on the traffic type and
priority level.

This concept maps directly to Beaker's priority levels, where instead of
being a vague and nebulous thing (as they are now), priority levels
should be aligned directly with a target speed of service. We would make
this configurable (since it's a per-installation thing, and dependent on
the available hardware and how well it meets the jobs being submitted),
but example values might be:

    Urgent: 4 hours
    High: 12 hours
    Normal: 24 hours
    Medium: 48 hours
    Low: 120 hours

Grade of Service is then about nominating a target *percentage* of
requests for which the Quality of Service is expected to be achieved
within the nominated Speed of Service. This can either vary by priority
level, or else you can use a single percentage for all of them.

If you're not achieving your target GoS, then you need to do one or more
of the following:

* reassess your SoS targets, asking if they're realistic given the
available resources
* make your instance bigger (using statistics on failed GoS to determine
what kinds of machines are needed)
* manage your instance more effectively (are jobs being submitted at too
high a level? Are systems being "camped" so the effective hardware pool
is smaller than it should be?)
* make the Beaker scheduler smarter

The "effective job priorities" concept is aimed mostly at the second
point - by giving system owners more control over the use of their
systems by the scheduler, it makes them more confident they can do so
without impacting their ability to get their own jobs executed in a
timely fashion.

Dynamic virtualisation support is aimed at all three - it is wasteful to
provision bare metal machines from scratch for jobs that could run on a
VM provisioned from an image, and it is wasteful to dedicate too much
generic x86 hardware to Beaker when it could be used to support a larger
IaaS installation that Beaker can access through dynamic virtualisation.
We should get a design proposal written that explains the current state
of the dynamic virt support, the problems we encountered trying to base
it on oVirt Engine, and what we think will be needed to base it on
OpenStack instead.

If we end up getting the data to justify it, another option we may want
to explore is the idea of "tentative reservations", where a 'scheduled'
recipe may have its assigned system taken away (or swapped for another
one) if a higher priority recipe set needs it. Along similar lines is
the idea of aborting a currently running lower priority recipeset (thus
failing QoS for that recipe) in order to release critical hardware to
allow a higher priority recipe set to meet its speed of service target.

Note that this doesn't change the near term road map at all (that's
still the Planned Development stuff on
http://beaker-project.org/dev/tech-roadmap.html plus a few items from
the Exploration list), but I hope it gives some context to what I see as
one of Beaker's biggest current shortcomings.

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane

Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)


More information about the Beaker-devel mailing list