On 02/20/2013 06:40 PM, Dan Callaghan wrote:
Excerpts from Nick Coghlan's message of 2013-02-20 16:30:56 +1000:
One common complaint from folks attempting to use the current private 
XML-RPC API to create a custom harness is that there's no way for the 
code running on the system under test to add tasks to the recipe. This 
means that an entire test run using an alternate harness will be 
collapsed into a single task result in Beaker. That's ugly, and makes 
it hard to upload good logs, provide decent progress reporting, 
granular test results, etc.
Well, I don't think this is a "common" complaint. We know of at least 
one use case for it though: having a single task which points at an 
autotest control file, where the actual list of tests is given.

Also, the entire test run wouldn't need to be reported under one result. 
Remember, a task can have zero or more "results". So you can report each 
test result separately under the one mega-task. It's not so bad, you 
still get granular test results and logs, just no sensible progress 
reporting.

So I don't see this (the harness adding new tasks) as a hugely important 
use case.

I've never liked this RFE.  I was warming to until I saw your comment below about cloning.  I really think the task list should be immutable.

What I can see benefit from would be allowing the matrix report to work with sub-task results, but I have no idea how to make that work without horrible performance. It may not be worth "fixing" in beaker and saying you can pull the results out of beaker into your own reporting tool to drill deeper and show regressions, etc..


What would be highly desirable is to have a way for the Beaker job XML 
to contain *just* enough detail to get an alternate harness up and 
running, and then the harness tells *Beaker* what the tasks are rather 
than the other way around.

The design goals I think we should be aiming for when it comes to 
alternate harnesses:

1. Transport arbitrary XML configuration settings for the harness 
through the recipe definition
Passing configuration from the recipe to the harness is definitely 
a good idea, although I find using XML for configuration to be hateful.

2. No required references to the Beaker task library in the recipe at 
all (unless a particular harness chooses to rely on it)
You can avoid the task library while still preserving Beaker's 
(existing, proven) data model of "a recipe has a sequence of tasks". The 
idea Bill and I were discussing was to simply add a new attribute url="" 
on <task/>, which tells the harness where to fetch the task from. If the 
url is absent it means install it from Beaker's task library.

3. Encourage harnesses to add all their tasks to the recipe before 
starting execution, but also allow them to add them incrementally
This sounds fine, although as I said above I don't think it's too 
important.

One thing to beware of here is that we shouldn't allow a recipe's state 
to go backwards (that is, from Compeleted to Running). Lots of Beaker 
code assumes this invariant, and I think it's a good one to have just 
for user expectations etc. But it means the harness has to be careful if 
adding new tasks: it can't mark the last task as completed *and then* 
add another after it -- instead it has to add the new task while the 
last one is still running, then stop it, then start the newly added one. 
So the whole thing is a bit messy.

4. Allow users to automatically request reservation of the system upon 
completion, independently of the harness used
I absolutely agree we need this feature (it's been a long-standing RFE) 
but I don't want to tackle it as part of this effort. As you say, it 
should be independent of the harness anyway.

So I think the approach to selecting alternate harnesses in the first 
draft doesn't quite work, as it doesn't easily support configuration of 
the harness itself, doesn't eliminate the need to define at least one 
task for each recipe and doesn't provide a harness independent way to 
reserve the system at the end. We also need to define an extra API 
endpoint to submit a new task.

I'll cover the latter first, as I don't think it needs to be 
particularly complicated:

     POST /recipes/{recipeid}/tasks/
     Content-type: application/json

     label=Something meaningful for the harness used

     ->  302 See Other
       Location: /recipes/{recipeid}/tasks/{taskid}

This would append a new task to the current recipe. New tasks would 
always be appended, with no mechanism for inserting them into the middle 
of the existing task list. Harnesses would be encouraged to use this to 
prepopulate the full list of tasks before they start running anything, 
but they would also be allowed to interleave the addition of new tasks 
with the completion of previous ones.
This API sounds fine to me, apart from my concerns above about managing 
the recipe state.

We may also want to allow harnesses to populate the "role" and "params" 
parts of the task definition.
I don't think this serves any purpose, does it?

Although it raises a question: what happens when someone clones a job 
like this, where the harness has added extra tasks? Do those tasks 
appear in the cloned job XML? What happens when you submit it? To me 
this is suggesting that adding tasks at runtime is a bad idea. Really, 
a recipe is supposed to be "a sequence of tasks you want the harness to 
run", *not* a sequence of tasks that the harness *did* run. That is what 
task results are for.

Now, on to the more complex part of the idea, the problem of deciding 
which harness to install and providing it with arbitrary configuration 
settings without Beaker needing to care what those settings are.

The initial draft of the alternate harness design has the XML look like 
this:

     <recipe ks_meta="harness=mylittleharness">
         <repos>
             <repo name="mylittleharness"
                 url="http://example.com/mylittleharness/el6/" />
         </repos>
         ...
     </recipe>

However, this leaves in place the current requirement that every recipe 
definition must include at least one task, and doesn't provide a task 
library independent way to request reservation of the system after the 
recipe executes.
In case it's not clear, my proposed way of selecting alternative 
harnesses would be a patch of approximately 5 lines to the rhts_post 
template. Everything else is built on existing Beaker features. So it 
has the distinct advantage of being easy and non-intrusive to implement.

My alternative suggestion is
[...]
At the harness level, the interesting parts would be:

1. The required "yuminstall" attribute of the harness element replaces 
the ks_meta="harness=<yum arg>" part of Dan's initial design
    The explicit nature of the "yuminstall" name leaves the door open to 
less packaging system specific alternatives in the future
Well, as long as ks_meta exists we are tied to yum anyway :-) The real 
issue here is not that the variable "harness=package_or_url" is tied to 
yum, but that ks_meta itself it.

The way forward I see is that ks_meta ("Kickstart Metadata") becomes 
"Installer Template Variables". They become a cross-installer way for 
people to customise the installation -- Beaker takes care of translating 
those variables into whatever syntax and meaning is appropriate for the 
installer. In that case having a variable "harness=something" is not 
yum-specific at all.

2. The optional "label" attribute gives us something human readable to 
show in the UI (defaulting to the "yuminstall" value)
I do like this idea of having some nice way to display "what harness ran 
this recipe" in Beaker's UI, but requiring the user to give a label in 
every recipe is not great. We could just display the package name as is 
(or the basename if it was a full URL).

A nicer idea might be to have the harness "check in" at the start of the 
recipe, report its name and version (and its configuration, and other 
useful info too?) -- Marian Csontos has suggested something similar in 
the past, and you can already get most of the way there by just having 
the harness upload a recipe log under a certain well-known name 
("debug/harness-checkin.txt" or whatever).

3. The ability to request reservation of the system after execution of 
the recipe is separated out from the definition of what the harness 
should do
4. "reservesys" is an *element* to leave the door open to possible 
future feature like <reservesys onsuccess="false"> to only reserve it in 
some cases.
+1000 to this, except as mentioned it doesn't need to be tied to this 
harness proposal at all.

5. The configuration mechanism for a custom harness is a choice between:
    - a Beaker task list (for the benefit of custom harnesses that were 
originally built to work within Beaker's current limitations)
    - a custom harness-specific "config" element which can contain 
arbitrary XML (such as a Git URL reference).
As mentioned above, I think this breaks some fundamentals of "what is 
a recipe" in Beaker.



_______________________________________________
Beaker-devel mailing list
Beaker-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/beaker-devel