[Beaker-devel] First draft of reservations design proposal

Raymond Mancy rmancy at redhat.com
Fri Sep 27 02:00:41 UTC 2013



----- Original Message -----
> From: "Nick Coghlan" <ncoghlan at redhat.com>
> To: "beaker-devel" <beaker-devel at lists.fedorahosted.org>
> Sent: Thursday, September 26, 2013 6:47:05 PM
> Subject: [Beaker-devel] First draft of reservations design proposal
> 
> The first draft of the improved reservations and loans design proposal
> is up at:
> 
> 
> http://gerrit.beaker-project.org/#/c/2302/1/dev/proposals/improved-reservations-and-loans.rst
> 
> Very rough sketch at the moment, so I'm mostly looking for feedback on
> whether or not the general concepts seem sound.
> 
> I've quoted the whole thing below, since I'm not really after
> line-by-line review comments at this stage, so email seems like a more
> appropriate venue.
> 
> Cheers,
> Nick.
> 
> Improved System Reservations and Loans
> ======================================
> 
> :Author: Nick Coghlan
> :Status: Proposed
> :Target Release: 0.16
> 
> 
> Abstract
> --------
> 
> This proposal supplements the current timed system reservation mechanism
> with a new mechanism that is independent of the lab controller's external
> watchdog timer. This new reservation timer is then used to offer time
> limited manual reservations and automatic reservation of systems when
> using an alternative harness or after a recipe is aborted.
> 

Any reason why we can't use the current watchdog with an alternative harness ?

> A similar time limiting mechanism is proposed for system loans, as well as
> the ability to indicate when reserving a system in Manual mode that the
> loan should be returned automatically when the reservation is returned.
> 
> Finally, a command line interface will be provided for system loan
> management.
> 
> .. note::
> 
>    Writing this out, I suspect it needs to be done in two phases, with phase
>    one covering the reservation expiry times and automatically returning
>    loans when the reservation is returned, while phase two covers loan
>    expiry.
> 
> 
> Harness independent time limited reservations
> ---------------------------------------------
> 
> A new "expiry date" attribute will be added to reservation records.
> 

I think we still want to be using one place to be the authoritative source of when a system has to be returned 
(i.e the external watchdog or some modified schema of it), otherwise it will just add complication.

I'm imagining something where the server gets an abort/fail/warn/pass and then checks the watchdog to 
see if it has any of these flags set, and will then update the watchdog time accordingly.

> For systems in Manual mode, the web UI will be updated to allow
> specification
> of a duration when taking the system. The default will continue to be no
> time limit, but if a time limit is requested, then the default will be
> 24 hours (configurable as a global Beaker server setting). If a time limit
> is requested, then the reservation expiry date will be set appropriately.
> 
> For systems in Automated mode, the reservation expiry date will not be
> used by default. Instead, job submitters will be able to opt in to the new
> mechanism by specifying a new ``reservesys`` element as part of the recipe
> definition.
> 
> Ignoring things like making the attributes optional and giving them a
> specific data types, the new ``reservesys`` element is defined as::
> 
>     <element name="reservesys">
>         <attribute name="onabort"/>
>         <attribute name="onfail"/>
>         <attribute name="onwarn"/>
>         <attribute name="onpass"/>
>         <attribute name="duration"/>
>     </element>
> 
> By default, the system would be reserved regardless of how the recipe
> finishes. The duration setting would control how long the automatic
> reservation would be for (defaulting to 24 hours)
> 
> onabort/fail/warn/pass control whether or not the system is
> actually reserved in the relevant situations:
> 
> * onabort = reserve system if a task or the overall recipe aborts
> * onfail = reserve system if at least one task reports a failed result
> * onwarn = reserve system if at least one task reports a warning
> * onpass = reserve system if none of the above occur
> 
> All four settings would default to "true", so the reservation is
> unconditional by default.
> 
> If any of the chosen conditions is triggered, the appropriate expiry time
> will be recorded on the reservation once the recipe has completed rather
> than returning the system immediately.
> 
> If the recipe is explicitly cancelled, then the system will not be reserved.
> 
> While a reservation is active, it may be extended. The existing watchdog
> extension commands will be updated to also extended the reservation when
> needed.
> 
> The Beaker scheduling daemon will periodically check for expired
> reservations
> and automatically return them. For systems in Automated mode, the expiry of
> the post-completion reservation will not alter the result reported for the
> recipe.
> 
> .. note::
> 
>    Perhaps Beaker should send a warning email 25 hours prior to expiry of
>    the reservation?
> 
> (Also see :issue:`639938`)
> 
> Automatically returning loans
> -----------------------------
> 
> When reserving a system, a user will be able to choose to automatically
> return a loan. If this option is specified, then when the reservation is
> returned, the active system loan will also be returned.
> 
> 
> Time limited loans
> ------------------
> 
> System loans will be separated out to be tracked explicitly (similar to
> the handling of reservations, only without the "type" field). When loaning
> the system, the user granting the loan will be able to specify a duration
> of the loan, which will be recorded as a loan expiry time. The default will
> continue to be no time limit, but if a time limit is requested, then the
> default will be 7 days (configurable as a global Beaker server setting).
> 
> While a loan is active, it may be extended. However, the user extending the
> loan must have ``loan-self`` or ``loan-any`` permissions on the system.
> 
> If a system has a loan expiry time configured, then all reservations made
> by that user must be time limited, defaulting to the loan expiry time.
> Attempting to extend reservations beyond the end of the loan period will
> fail.
> 
> The Beaker scheduling daemon will periodically check for expired loans
> and automatically return them. For reserved systems where the reservation
> is from a user that is no longer permitted to access the system once the
> loan has been returned, it will also return the reservation or cancel the
> recipe as appropriate.
> 
> .. note::
> 
>    Perhaps Beaker should send a warning email 25 hours prior to expiry of
>    the loan?
> 
> User interface proposals
> ------------------------
> 
> Web UI
> ~~~~~~
> 
> TBD
> 
> Command line
> ~~~~~~~~~~~~
> 
> TBD
> 
> Deferred features
> -----------------
> 

How about the ability of setting hard time limits on systems? If we are going to redesign how we 
track the reservation time of systems, perhaps this is something we should think about as well?

> * Updating the scheduled provisioning mechanism for systems in Automated
>   mode to use the new harness independent mechanism rather than the
>   reservesys task
> 
> * Allowing the ``reservesys`` element to be specified at the recipe set
> level
>   to reserve all systems in the recipe set whenever one or more of them
>   encounters a problem.
> 
> 
> Rejected features
> -----------------
> 
> * Allowing the ``reservesys`` element to be specified at the job level,
>   since it isn't clear how that would work when recipe sets are run at
>   different times.
> 
> * Having "onpass" default to false in the reservesys element. While this is
>   desirable in some respects, having different defaults for one of the
>   items is difficult to document clearly.
> 
> --
> Nick Coghlan
> Red Hat Infrastructure Engineering & Development, Brisbane
> 
> Testing Solutions Team Lead
> Beaker Development Lead (http://beaker-project.org/)
> _______________________________________________
> Beaker-devel mailing list
> Beaker-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/beaker-devel
> 


More information about the Beaker-devel mailing list