----- Original Message -----
From: "Bill Peck" <bpeck(a)redhat.com>
To: beaker-devel(a)lists.fedorahosted.org
Sent: Thursday, February 21, 2013 11:12:53 AM
Subject: Re: [Beaker-devel] Harness API proposal
On 02/20/2013 06:40 PM, Dan Callaghan wrote:
> Excerpts from Nick Coghlan's message of 2013-02-20 16:30:56 +1000:
>> One common complaint from folks attempting to use the current
>> private
>> XML-RPC API to create a custom harness is that there's no way for
>> the
>> code running on the system under test to add tasks to the recipe.
>> This
>> means that an entire test run using an alternate harness will be
>> collapsed into a single task result in Beaker. That's ugly, and
>> makes
>> it hard to upload good logs, provide decent progress reporting,
>> granular test results, etc.
> Well, I don't think this is a "common" complaint. We know of at
> least
> one use case for it though: having a single task which points at an
> autotest control file, where the actual list of tests is given.
>
> Also, the entire test run wouldn't need to be reported under one
> result.
> Remember, a task can have zero or more "results". So you can report
> each
> test result separately under the one mega-task. It's not so bad,
> you
> still get granular test results and logs, just no sensible progress
> reporting.
>
> So I don't see this (the harness adding new tasks) as a hugely
> important
> use case.
I've never liked this RFE. I was warming to until I saw your comment
below about cloning. I really think the task list should be
immutable.
What I can see benefit from would be allowing the matrix report to
work
with sub-task results, but I have no idea how to make that work
without
horrible performance.
Yeah to display a matrix of sub task results would most likely be horrible
in every regard.
Perhaps you could show a matrix of sub-tasks for a specific recipe task name,
but like Nick said, it seems we're getting out of the reporting game.
It may not be worth "fixing" in beaker and
saying
you can pull the results out of beaker into your own reporting tool
to
drill deeper and show regressions, etc..
>
>> What would be highly desirable is to have a way for the Beaker job
>> XML
>> to contain *just* enough detail to get an alternate harness up and
>> running, and then the harness tells *Beaker* what the tasks are
>> rather
>> than the other way around.
>>
>> The design goals I think we should be aiming for when it comes to
>> alternate harnesses:
>>
>> 1. Transport arbitrary XML configuration settings for the harness
>> through the recipe definition
> Passing configuration from the recipe to the harness is definitely
> a good idea, although I find using XML for configuration to be
> hateful.
>
>> 2. No required references to the Beaker task library in the recipe
>> at
>> all (unless a particular harness chooses to rely on it)
> You can avoid the task library while still preserving Beaker's
> (existing, proven) data model of "a recipe has a sequence of
> tasks". The
> idea Bill and I were discussing was to simply add a new attribute
> url=""
> on <task/>, which tells the harness where to fetch the task from.
> If the
> url is absent it means install it from Beaker's task library.
>
>> 3. Encourage harnesses to add all their tasks to the recipe before
>> starting execution, but also allow them to add them incrementally
> This sounds fine, although as I said above I don't think it's too
> important.
>
> One thing to beware of here is that we shouldn't allow a recipe's
> state
> to go backwards (that is, from Compeleted to Running). Lots of
> Beaker
> code assumes this invariant, and I think it's a good one to have
> just
> for user expectations etc. But it means the harness has to be
> careful if
> adding new tasks: it can't mark the last task as completed *and
> then*
> add another after it -- instead it has to add the new task while
> the
> last one is still running, then stop it, then start the newly added
> one.
> So the whole thing is a bit messy.
>
>> 4. Allow users to automatically request reservation of the system
>> upon
>> completion, independently of the harness used
> I absolutely agree we need this feature (it's been a long-standing
> RFE)
> but I don't want to tackle it as part of this effort. As you say,
> it
> should be independent of the harness anyway.
>
>> So I think the approach to selecting alternate harnesses in the
>> first
>> draft doesn't quite work, as it doesn't easily support
>> configuration of
>> the harness itself, doesn't eliminate the need to define at least
>> one
>> task for each recipe and doesn't provide a harness independent way
>> to
>> reserve the system at the end. We also need to define an extra API
>> endpoint to submit a new task.
>>
>> I'll cover the latter first, as I don't think it needs to be
>> particularly complicated:
>>
>> POST /recipes/{recipeid}/tasks/
>> Content-type: application/json
>>
>> label=Something meaningful for the harness used
>>
>> -> 302 See Other
>> Location: /recipes/{recipeid}/tasks/{taskid}
>>
>> This would append a new task to the current recipe. New tasks
>> would
>> always be appended, with no mechanism for inserting them into the
>> middle
>> of the existing task list. Harnesses would be encouraged to use
>> this to
>> prepopulate the full list of tasks before they start running
>> anything,
>> but they would also be allowed to interleave the addition of new
>> tasks
>> with the completion of previous ones.
> This API sounds fine to me, apart from my concerns above about
> managing
> the recipe state.
>
>> We may also want to allow harnesses to populate the "role" and
>> "params"
>> parts of the task definition.
> I don't think this serves any purpose, does it?
>
> Although it raises a question: what happens when someone clones a
> job
> like this, where the harness has added extra tasks? Do those tasks
> appear in the cloned job XML? What happens when you submit it? To
> me
> this is suggesting that adding tasks at runtime is a bad idea.
> Really,
> a recipe is supposed to be "a sequence of tasks you want the
> harness to
> run", *not* a sequence of tasks that the harness *did* run. That is
> what
> task results are for.
>
>> Now, on to the more complex part of the idea, the problem of
>> deciding
>> which harness to install and providing it with arbitrary
>> configuration
>> settings without Beaker needing to care what those settings are.
>>
>> The initial draft of the alternate harness design has the XML look
>> like
>> this:
>>
>> <recipe ks_meta="harness=mylittleharness">
>> <repos>
>> <repo name="mylittleharness"
>>
url="http://example.com/mylittleharness/el6/" />
>> </repos>
>> ...
>> </recipe>
>>
>> However, this leaves in place the current requirement that every
>> recipe
>> definition must include at least one task, and doesn't provide a
>> task
>> library independent way to request reservation of the system after
>> the
>> recipe executes.
> In case it's not clear, my proposed way of selecting alternative
> harnesses would be a patch of approximately 5 lines to the
> rhts_post
> template. Everything else is built on existing Beaker features. So
> it
> has the distinct advantage of being easy and non-intrusive to
> implement.
>
>> My alternative suggestion is
> [...]
>> At the harness level, the interesting parts would be:
>>
>> 1. The required "yuminstall" attribute of the harness element
>> replaces
>> the ks_meta="harness=<yum arg>" part of Dan's initial
design
>> The explicit nature of the "yuminstall" name leaves the door
>> open to
>> less packaging system specific alternatives in the future
> Well, as long as ks_meta exists we are tied to yum anyway :-) The
> real
> issue here is not that the variable "harness=package_or_url" is
> tied to
> yum, but that ks_meta itself it.
>
> The way forward I see is that ks_meta ("Kickstart Metadata")
> becomes
> "Installer Template Variables". They become a cross-installer way
> for
> people to customise the installation -- Beaker takes care of
> translating
> those variables into whatever syntax and meaning is appropriate for
> the
> installer. In that case having a variable "harness=something" is
> not
> yum-specific at all.
>
>> 2. The optional "label" attribute gives us something human
>> readable to
>> show in the UI (defaulting to the "yuminstall" value)
> I do like this idea of having some nice way to display "what
> harness ran
> this recipe" in Beaker's UI, but requiring the user to give a label
> in
> every recipe is not great. We could just display the package name
> as is
> (or the basename if it was a full URL).
>
> A nicer idea might be to have the harness "check in" at the start
> of the
> recipe, report its name and version (and its configuration, and
> other
> useful info too?) -- Marian Csontos has suggested something similar
> in
> the past, and you can already get most of the way there by just
> having
> the harness upload a recipe log under a certain well-known name
> ("debug/harness-checkin.txt" or whatever).
>
>> 3. The ability to request reservation of the system after
>> execution of
>> the recipe is separated out from the definition of what the
>> harness
>> should do
>> 4. "reservesys" is an *element* to leave the door open to possible
>> future feature like <reservesys onsuccess="false"> to only
reserve
>> it in
>> some cases.
> +1000 to this, except as mentioned it doesn't need to be tied to
> this
> harness proposal at all.
>
>> 5. The configuration mechanism for a custom harness is a choice
>> between:
>> - a Beaker task list (for the benefit of custom harnesses
>> that were
>> originally built to work within Beaker's current limitations)
>> - a custom harness-specific "config" element which can
>> contain
>> arbitrary XML (such as a Git URL reference).
> As mentioned above, I think this breaks some fundamentals of "what
> is
> a recipe" in Beaker.
>
>
>
> _______________________________________________
> Beaker-devel mailing list
> Beaker-devel(a)lists.fedorahosted.org
>
https://lists.fedorahosted.org/mailman/listinfo/beaker-devel
_______________________________________________
Beaker-devel mailing list
Beaker-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/beaker-devel