Restoring the ability to manually reserve Automated systems
by Nick Coghlan
Beaker 0.15 currently removes the Take button from the system UI when
systems are in automated mode.
https://bugzilla.redhat.com/show_bug.cgi?id=1015131 points out that
there's actually a valid use case for manually reserving an automated
system: when a system is loaned to you, you don't have the ability to
change it between Automated and Manual mode.
That means to allow the loanee to run jobs against the system through
the scheduler, the system must be in Automated mode.
However, having it in automated mode and reserving it through the
scheduler means you have to remember to keep extending the watchdog - in
0.15 there's no ability to say "I'll return it when I'm done (or when
the system owner takes the system away from me)".
In 0.14, you could manually reserve a loaned system *even if* it was
nominally in Automated mode.
Given that the "reserve" permission makes no distinction between manual
and automatic reservations, I don't really want to provide the ability
to manually reserve automated systems without some kind of time limit or
other additional access control mechanism.
Accordingly, the solution I am considering is to *require* that a loan
be in place in order to manually reserve an Automated system. The
behaviour of the system UI would be as follows:
System in Manual mode (no current user):
- Take button visible
- Provision tab indicates user must reserve the system first
System in Automated mode (no current loan or user):
- Take button hidden
- Provision tab offers ability to schedule a provision operation
System in Automated mode (current loan, but no current user):
- Take button visible to the user with the loan
- Provision tab offers ability to schedule a provision operation with a
note saying to reserve the system to switch to manual provisioning
This approach means that "reserve" will grant the ability to schedule a
provision on a system in Automated mode, but you'll also need
"loan_self" (or a loan from someone with "loan_any") to switch to manual
provisioning.
Thoughts?
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 6 months
Fixing loan and reservation permissions in 0.15.1
by Nick Coghlan
A couple of bug reports against 0.15 highlighted some issues with the
way loan management was mapped to the new ACL model. The one we kept
open as the tracking BZ was
https://bugzilla.redhat.com/show_bug.cgi?id=1011858
While it results in a slight misnomer, my proposal is to update the
permissions as follows to ensure we're providing feature parity with 0.14:
1. "loan-any" will grant the same power over loans and reservations as
that enjoyed by the system owner. Both the system owners and users and
groups with the loan-any permission will be able to:
- grant a loan to any user
- return a loan for any user
- transfer a loan directly between users
- return a manual reservation for any user
Neither the system owner *nor* those with loan-any permissions can
return automated reservations (the job needs to be cancelled instead).
2. "loan-self" will grant the ability to loan a system that is not
currently loaned to anyone to yourself. It does *not* grant the ability
to return someone else's loan or transfer their loan to you.
3. Anyone with the system loaned to them will always have the necessary
permissions to return the loan, as well as to manually reserve and
return the system.
Linking the ability to return arbitrary manual reservations to
"loan-any" is the part that's a slight misnomer, but it's necessary to
preserve the old behaviour in 0.14 where members of system admin groups
could return each other's reservations.
For 0.16 we could split out a separate "cancel-reservation" permission
for that, but I'm not sure it makes sense to do so - being able to
cancel a manual reservation is a necessary part of being able to
reliably cancel a loan. I think it makes more sense to postpone doing
anything about that until we grant system owners the ability to cancel
arbitrary jobs running on their systems.
Thoughts?
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 6 months
Creating a Beaker 0.14.2 maintenance release?
by Nick Coghlan
While the focus for the next couple of weeks is on 0.15.1 development, I
started pondering the question of what would be involved in preparing a
0.14.2 release.
It seems to me that at least some of the provisioning and watchdog
related fixes could be cherry-picked to the release-0.14 branch without
too much difficulty
- disabling rather than removing ntp (bug 1002928)
- EFI currentboot fix (bug 997629)
- ensure watchdog triggers for anaconda install loop (bug 954219)
- ensure watchdog triggers for panic loop (bug 953543)
- btrfs tech preview compatibility on RHEL6.x (bug 1002261)
- using platform default auth configuration (bug 989924)
Anyway, it's something to keep in mind as a fallback plan if any of the
regressions in 0.15 prove harder to resolve than expected (or any other
factors serve to delay that release).
Regards,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 6 months
*Don't* update to Beaker 0.15
by Nick Coghlan
We've identified some issues in the Beaker 0.15 release and upgrade
instructions that make it inadvisable to deploy in its current form.
We expect to have an updated Beaker 0.15.1 release available before too
long.
Regards,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 6 months
First draft of reservations design proposal
by Nick Coghlan
The first draft of the improved reservations and loans design proposal
is up at:
http://gerrit.beaker-project.org/#/c/2302/1/dev/proposals/improved-reserv...
Very rough sketch at the moment, so I'm mostly looking for feedback on
whether or not the general concepts seem sound.
I've quoted the whole thing below, since I'm not really after
line-by-line review comments at this stage, so email seems like a more
appropriate venue.
Cheers,
Nick.
Improved System Reservations and Loans
======================================
:Author: Nick Coghlan
:Status: Proposed
:Target Release: 0.16
Abstract
--------
This proposal supplements the current timed system reservation mechanism
with a new mechanism that is independent of the lab controller's external
watchdog timer. This new reservation timer is then used to offer time
limited manual reservations and automatic reservation of systems when
using an alternative harness or after a recipe is aborted.
A similar time limiting mechanism is proposed for system loans, as well as
the ability to indicate when reserving a system in Manual mode that the
loan should be returned automatically when the reservation is returned.
Finally, a command line interface will be provided for system loan
management.
.. note::
Writing this out, I suspect it needs to be done in two phases, with phase
one covering the reservation expiry times and automatically returning
loans when the reservation is returned, while phase two covers loan
expiry.
Harness independent time limited reservations
---------------------------------------------
A new "expiry date" attribute will be added to reservation records.
For systems in Manual mode, the web UI will be updated to allow
specification
of a duration when taking the system. The default will continue to be no
time limit, but if a time limit is requested, then the default will be
24 hours (configurable as a global Beaker server setting). If a time limit
is requested, then the reservation expiry date will be set appropriately.
For systems in Automated mode, the reservation expiry date will not be
used by default. Instead, job submitters will be able to opt in to the new
mechanism by specifying a new ``reservesys`` element as part of the recipe
definition.
Ignoring things like making the attributes optional and giving them a
specific data types, the new ``reservesys`` element is defined as::
<element name="reservesys">
<attribute name="onabort"/>
<attribute name="onfail"/>
<attribute name="onwarn"/>
<attribute name="onpass"/>
<attribute name="duration"/>
</element>
By default, the system would be reserved regardless of how the recipe
finishes. The duration setting would control how long the automatic
reservation would be for (defaulting to 24 hours)
onabort/fail/warn/pass control whether or not the system is
actually reserved in the relevant situations:
* onabort = reserve system if a task or the overall recipe aborts
* onfail = reserve system if at least one task reports a failed result
* onwarn = reserve system if at least one task reports a warning
* onpass = reserve system if none of the above occur
All four settings would default to "true", so the reservation is
unconditional by default.
If any of the chosen conditions is triggered, the appropriate expiry time
will be recorded on the reservation once the recipe has completed rather
than returning the system immediately.
If the recipe is explicitly cancelled, then the system will not be reserved.
While a reservation is active, it may be extended. The existing watchdog
extension commands will be updated to also extended the reservation when
needed.
The Beaker scheduling daemon will periodically check for expired
reservations
and automatically return them. For systems in Automated mode, the expiry of
the post-completion reservation will not alter the result reported for the
recipe.
.. note::
Perhaps Beaker should send a warning email 25 hours prior to expiry of
the reservation?
(Also see :issue:`639938`)
Automatically returning loans
-----------------------------
When reserving a system, a user will be able to choose to automatically
return a loan. If this option is specified, then when the reservation is
returned, the active system loan will also be returned.
Time limited loans
------------------
System loans will be separated out to be tracked explicitly (similar to
the handling of reservations, only without the "type" field). When loaning
the system, the user granting the loan will be able to specify a duration
of the loan, which will be recorded as a loan expiry time. The default will
continue to be no time limit, but if a time limit is requested, then the
default will be 7 days (configurable as a global Beaker server setting).
While a loan is active, it may be extended. However, the user extending the
loan must have ``loan-self`` or ``loan-any`` permissions on the system.
If a system has a loan expiry time configured, then all reservations made
by that user must be time limited, defaulting to the loan expiry time.
Attempting to extend reservations beyond the end of the loan period will
fail.
The Beaker scheduling daemon will periodically check for expired loans
and automatically return them. For reserved systems where the reservation
is from a user that is no longer permitted to access the system once the
loan has been returned, it will also return the reservation or cancel the
recipe as appropriate.
.. note::
Perhaps Beaker should send a warning email 25 hours prior to expiry of
the loan?
User interface proposals
------------------------
Web UI
~~~~~~
TBD
Command line
~~~~~~~~~~~~
TBD
Deferred features
-----------------
* Updating the scheduled provisioning mechanism for systems in Automated
mode to use the new harness independent mechanism rather than the
reservesys task
* Allowing the ``reservesys`` element to be specified at the recipe set
level
to reserve all systems in the recipe set whenever one or more of them
encounters a problem.
Rejected features
-----------------
* Allowing the ``reservesys`` element to be specified at the job level,
since it isn't clear how that would work when recipe sets are run at
different times.
* Having "onpass" default to false in the reservesys element. While this is
desirable in some respects, having different defaults for one of the
items is difficult to document clearly.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 6 months