On 02/20/2013 06:40 PM, Dan Callaghan wrote:
Excerpts from Nick Coghlan's message of 2013-02-20 16:30:56
+1000:
> One common complaint from folks attempting to use the current private
> XML-RPC API to create a custom harness is that there's no way for the
> code running on the system under test to add tasks to the recipe. This
> means that an entire test run using an alternate harness will be
> collapsed into a single task result in Beaker. That's ugly, and makes
> it hard to upload good logs, provide decent progress reporting,
> granular test results, etc.
Well, I don't think this is a "common" complaint. We know of at least
one use case for it though: having a single task which points at an
autotest control file, where the actual list of tests is given.
Also, the entire test run wouldn't need to be reported under one result.
Remember, a task can have zero or more "results". So you can report each
test result separately under the one mega-task. It's not so bad, you
still get granular test results and logs, just no sensible progress
reporting.
So I don't see this (the harness adding new tasks) as a hugely important
use case.
I've never liked this RFE. I was warming to until I saw your comment
below about cloning. I really think the task list should be immutable.
What I can see benefit from would be allowing the matrix report to work
with sub-task results, but I have no idea how to make that work without
horrible performance. It may not be worth "fixing" in beaker and saying
you can pull the results out of beaker into your own reporting tool to
drill deeper and show regressions, etc..
> What would be highly desirable is to have a way for the Beaker job XML
> to contain *just* enough detail to get an alternate harness up and
> running, and then the harness tells *Beaker* what the tasks are rather
> than the other way around.
>
> The design goals I think we should be aiming for when it comes to
> alternate harnesses:
>
> 1. Transport arbitrary XML configuration settings for the harness
> through the recipe definition
Passing configuration from the recipe to the harness is definitely
a good idea, although I find using XML for configuration to be hateful.
> 2. No required references to the Beaker task library in the recipe at
> all (unless a particular harness chooses to rely on it)
You can avoid the task library while still preserving Beaker's
(existing, proven) data model of "a recipe has a sequence of tasks". The
idea Bill and I were discussing was to simply add a new attribute url=""
on <task/>, which tells the harness where to fetch the task from. If the
url is absent it means install it from Beaker's task library.
> 3. Encourage harnesses to add all their tasks to the recipe before
> starting execution, but also allow them to add them incrementally
This sounds fine, although as I said above I don't think it's too
important.
One thing to beware of here is that we shouldn't allow a recipe's state
to go backwards (that is, from Compeleted to Running). Lots of Beaker
code assumes this invariant, and I think it's a good one to have just
for user expectations etc. But it means the harness has to be careful if
adding new tasks: it can't mark the last task as completed *and then*
add another after it -- instead it has to add the new task while the
last one is still running, then stop it, then start the newly added one.
So the whole thing is a bit messy.
> 4. Allow users to automatically request reservation of the system upon
> completion, independently of the harness used
I absolutely agree we need this feature (it's been a long-standing RFE)
but I don't want to tackle it as part of this effort. As you say, it
should be independent of the harness anyway.
> So I think the approach to selecting alternate harnesses in the first
> draft doesn't quite work, as it doesn't easily support configuration of
> the harness itself, doesn't eliminate the need to define at least one
> task for each recipe and doesn't provide a harness independent way to
> reserve the system at the end. We also need to define an extra API
> endpoint to submit a new task.
>
> I'll cover the latter first, as I don't think it needs to be
> particularly complicated:
>
> POST /recipes/{recipeid}/tasks/
> Content-type: application/json
>
> label=Something meaningful for the harness used
>
> -> 302 See Other
> Location: /recipes/{recipeid}/tasks/{taskid}
>
> This would append a new task to the current recipe. New tasks would
> always be appended, with no mechanism for inserting them into the middle
> of the existing task list. Harnesses would be encouraged to use this to
> prepopulate the full list of tasks before they start running anything,
> but they would also be allowed to interleave the addition of new tasks
> with the completion of previous ones.
This API sounds fine to me, apart from my concerns above about managing
the recipe state.
> We may also want to allow harnesses to populate the "role" and
"params"
> parts of the task definition.
I don't think this serves any purpose, does it?
Although it raises a question: what happens when someone clones a job
like this, where the harness has added extra tasks? Do those tasks
appear in the cloned job XML? What happens when you submit it? To me
this is suggesting that adding tasks at runtime is a bad idea. Really,
a recipe is supposed to be "a sequence of tasks you want the harness to
run", *not* a sequence of tasks that the harness *did* run. That is what
task results are for.
> Now, on to the more complex part of the idea, the problem of deciding
> which harness to install and providing it with arbitrary configuration
> settings without Beaker needing to care what those settings are.
>
> The initial draft of the alternate harness design has the XML look like
> this:
>
> <recipe ks_meta="harness=mylittleharness">
> <repos>
> <repo name="mylittleharness"
>
url="http://example.com/mylittleharness/el6/" />
> </repos>
> ...
> </recipe>
>
> However, this leaves in place the current requirement that every recipe
> definition must include at least one task, and doesn't provide a task
> library independent way to request reservation of the system after the
> recipe executes.
In case it's not clear, my proposed way of selecting alternative
harnesses would be a patch of approximately 5 lines to the rhts_post
template. Everything else is built on existing Beaker features. So it
has the distinct advantage of being easy and non-intrusive to implement.
> My alternative suggestion is
[...]
> At the harness level, the interesting parts would be:
>
> 1. The required "yuminstall" attribute of the harness element replaces
> the ks_meta="harness=<yum arg>" part of Dan's initial design
> The explicit nature of the "yuminstall" name leaves the door open to
> less packaging system specific alternatives in the future
Well, as long as ks_meta exists we are tied to yum anyway :-) The real
issue here is not that the variable "harness=package_or_url" is tied to
yum, but that ks_meta itself it.
The way forward I see is that ks_meta ("Kickstart Metadata") becomes
"Installer Template Variables". They become a cross-installer way for
people to customise the installation -- Beaker takes care of translating
those variables into whatever syntax and meaning is appropriate for the
installer. In that case having a variable "harness=something" is not
yum-specific at all.
> 2. The optional "label" attribute gives us something human readable to
> show in the UI (defaulting to the "yuminstall" value)
I do like this idea of having some nice way to display "what harness ran
this recipe" in Beaker's UI, but requiring the user to give a label in
every recipe is not great. We could just display the package name as is
(or the basename if it was a full URL).
A nicer idea might be to have the harness "check in" at the start of the
recipe, report its name and version (and its configuration, and other
useful info too?) -- Marian Csontos has suggested something similar in
the past, and you can already get most of the way there by just having
the harness upload a recipe log under a certain well-known name
("debug/harness-checkin.txt" or whatever).
> 3. The ability to request reservation of the system after execution of
> the recipe is separated out from the definition of what the harness
> should do
> 4. "reservesys" is an *element* to leave the door open to possible
> future feature like <reservesys onsuccess="false"> to only reserve it
in
> some cases.
+1000 to this, except as mentioned it doesn't need to be tied to this
harness proposal at all.
> 5. The configuration mechanism for a custom harness is a choice between:
> - a Beaker task list (for the benefit of custom harnesses that were
> originally built to work within Beaker's current limitations)
> - a custom harness-specific "config" element which can contain
> arbitrary XML (such as a Git URL reference).
As mentioned above, I think this breaks some fundamentals of "what is
a recipe" in Beaker.
_______________________________________________
Beaker-devel mailing list
Beaker-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/beaker-devel