Towards "Grade of Service" metrics for Beaker
by Nick Coghlan
A full Beaker installation is a complex beast - it can be hard to
summarise neatly whether or not it is meeting its goals. While making
the scheduling algorithm more sensible (and a bit closer to the way
Condor works) is already a near term goal, my long term goals are more
ambitious - I want to be able to collect useful metrics from a Beaker
instance and use that to decide when the scheduling algorithm is "good
enough" :)
A large part of my previous experience was working on a HF radio
communications system
(http://www.defence.gov.au/dmo/esd/jp2043/jp2043.cfm). We handled a wide
variety of different traffic types, but there was an overarching notion
called "Grade of Service".
That in turn was a combination of two things: Quality of Service and
Speed of Service.
Quality of Service was a fairly basic yes/no distinction: did the system
do what it was supposed to do (send a message, connect a call, etc)?
In the context of Beaker, the key "Quality of Service" criterion would
be "successfully provisioned and started running a recipe set".
Speed of Service was then about "Was the service provided in a timely
fashion?", and the required SoS varied depending on the traffic type and
priority level.
This concept maps directly to Beaker's priority levels, where instead of
being a vague and nebulous thing (as they are now), priority levels
should be aligned directly with a target speed of service. We would make
this configurable (since it's a per-installation thing, and dependent on
the available hardware and how well it meets the jobs being submitted),
but example values might be:
Urgent: 4 hours
High: 12 hours
Normal: 24 hours
Medium: 48 hours
Low: 120 hours
Grade of Service is then about nominating a target *percentage* of
requests for which the Quality of Service is expected to be achieved
within the nominated Speed of Service. This can either vary by priority
level, or else you can use a single percentage for all of them.
If you're not achieving your target GoS, then you need to do one or more
of the following:
* reassess your SoS targets, asking if they're realistic given the
available resources
* make your instance bigger (using statistics on failed GoS to determine
what kinds of machines are needed)
* manage your instance more effectively (are jobs being submitted at too
high a level? Are systems being "camped" so the effective hardware pool
is smaller than it should be?)
* make the Beaker scheduler smarter
The "effective job priorities" concept is aimed mostly at the second
point - by giving system owners more control over the use of their
systems by the scheduler, it makes them more confident they can do so
without impacting their ability to get their own jobs executed in a
timely fashion.
Dynamic virtualisation support is aimed at all three - it is wasteful to
provision bare metal machines from scratch for jobs that could run on a
VM provisioned from an image, and it is wasteful to dedicate too much
generic x86 hardware to Beaker when it could be used to support a larger
IaaS installation that Beaker can access through dynamic virtualisation.
We should get a design proposal written that explains the current state
of the dynamic virt support, the problems we encountered trying to base
it on oVirt Engine, and what we think will be needed to base it on
OpenStack instead.
If we end up getting the data to justify it, another option we may want
to explore is the idea of "tentative reservations", where a 'scheduled'
recipe may have its assigned system taken away (or swapped for another
one) if a higher priority recipe set needs it. Along similar lines is
the idea of aborting a currently running lower priority recipeset (thus
failing QoS for that recipe) in order to release critical hardware to
allow a higher priority recipe set to meet its speed of service target.
Note that this doesn't change the near term road map at all (that's
still the Planned Development stuff on
http://beaker-project.org/dev/tech-roadmap.html plus a few items from
the Exploration list), but I hope it gives some context to what I see as
one of Beaker's biggest current shortcomings.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 9 months
Role names in documentation
by Luigi Toscano
Hi all,
looking at the beaker documentation about multihost tests[1], I think that the
variables used for roles (CLIENTS, SERVERS and DRIVER) are introduced in a way
that makes them the "only possible choice" as roles.
I know for personal experience that you can use whatever you want as role
(provided that it's a valid name for a shell variable), and I personally rely
on this feature, so I'm pretty sure it's not by accident :)
Would it make sense to document the role names in a more generic way? If yes,
I guess I should at least file a bug...
[1] http://beaker-project.org/docs/user-guide/multihost.html
Regards
--
Luigi
10 years, 9 months
Add result column selector for Jobs table?
by Nick Coghlan
For 0.13, we added the new "Group" column to the jobs table.
https://bugzilla.redhat.com/show_bug.cgi?id=980164 asks for the
Retention Tag to be visible as well.
Should we just add a column selector like the one on the system table?
I know I've said in the past that I wanted to de-emphasise improvements
to the existing web UI in favour of making Beaker more effective as a
web service, but I think there are a few ideas like this that could be
useful near term improvements while we work on the long term plan of a
cleaner web service/web UI split.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 9 months