Excerpts from Nick Coghlan's message of 2013-02-20 16:30:56 +1000:
One common complaint from folks attempting to use the current private
XML-RPC API to create a custom harness is that there's no way for the
code running on the system under test to add tasks to the recipe. This
means that an entire test run using an alternate harness will be
collapsed into a single task result in Beaker. That's ugly, and makes
it hard to upload good logs, provide decent progress reporting,
granular test results, etc.
Well, I don't think this is a "common" complaint. We know of at least
one use case for it though: having a single task which points at an
autotest control file, where the actual list of tests is given.
Also, the entire test run wouldn't need to be reported under one result.
Remember, a task can have zero or more "results". So you can report each
test result separately under the one mega-task. It's not so bad, you
still get granular test results and logs, just no sensible progress
reporting.
So I don't see this (the harness adding new tasks) as a hugely important
use case.
What would be highly desirable is to have a way for the Beaker job XML
to contain *just* enough detail to get an alternate harness up and
running, and then the harness tells *Beaker* what the tasks are rather
than the other way around.
The design goals I think we should be aiming for when it comes to
alternate harnesses:
1. Transport arbitrary XML configuration settings for the harness
through the recipe definition
Passing configuration from the recipe to the harness is definitely
a good idea, although I find using XML for configuration to be hateful.
2. No required references to the Beaker task library in the recipe at
all (unless a particular harness chooses to rely on it)
You can avoid the task library while still preserving Beaker's
(existing, proven) data model of "a recipe has a sequence of tasks". The
idea Bill and I were discussing was to simply add a new attribute url=""
on <task/>, which tells the harness where to fetch the task from. If the
url is absent it means install it from Beaker's task library.
3. Encourage harnesses to add all their tasks to the recipe before
starting execution, but also allow them to add them incrementally
This sounds fine, although as I said above I don't think it's too
important.
One thing to beware of here is that we shouldn't allow a recipe's state
to go backwards (that is, from Compeleted to Running). Lots of Beaker
code assumes this invariant, and I think it's a good one to have just
for user expectations etc. But it means the harness has to be careful if
adding new tasks: it can't mark the last task as completed *and then*
add another after it -- instead it has to add the new task while the
last one is still running, then stop it, then start the newly added one.
So the whole thing is a bit messy.
4. Allow users to automatically request reservation of the system upon
completion, independently of the harness used
I absolutely agree we need this feature (it's been a long-standing RFE)
but I don't want to tackle it as part of this effort. As you say, it
should be independent of the harness anyway.
So I think the approach to selecting alternate harnesses in the first
draft doesn't quite work, as it doesn't easily support configuration of
the harness itself, doesn't eliminate the need to define at least one
task for each recipe and doesn't provide a harness independent way to
reserve the system at the end. We also need to define an extra API
endpoint to submit a new task.
I'll cover the latter first, as I don't think it needs to be
particularly complicated:
POST /recipes/{recipeid}/tasks/
Content-type: application/json
label=Something meaningful for the harness used
-> 302 See Other
Location: /recipes/{recipeid}/tasks/{taskid}
This would append a new task to the current recipe. New tasks would
always be appended, with no mechanism for inserting them into the middle
of the existing task list. Harnesses would be encouraged to use this to
prepopulate the full list of tasks before they start running anything,
but they would also be allowed to interleave the addition of new tasks
with the completion of previous ones.
This API sounds fine to me, apart from my concerns above about managing
the recipe state.
We may also want to allow harnesses to populate the "role" and "params"
parts of the task definition.
I don't think this serves any purpose, does it?
Although it raises a question: what happens when someone clones a job
like this, where the harness has added extra tasks? Do those tasks
appear in the cloned job XML? What happens when you submit it? To me
this is suggesting that adding tasks at runtime is a bad idea. Really,
a recipe is supposed to be "a sequence of tasks you want the harness to
run", *not* a sequence of tasks that the harness *did* run. That is what
task results are for.
Now, on to the more complex part of the idea, the problem of deciding
which harness to install and providing it with arbitrary configuration
settings without Beaker needing to care what those settings are.
The initial draft of the alternate harness design has the XML look like
this:
<recipe ks_meta="harness=mylittleharness">
<repos>
<repo name="mylittleharness"
url="http://example.com/mylittleharness/el6/" />
</repos>
...
</recipe>
However, this leaves in place the current requirement that every recipe
definition must include at least one task, and doesn't provide a task
library independent way to request reservation of the system after the
recipe executes.
In case it's not clear, my proposed way of selecting alternative
harnesses would be a patch of approximately 5 lines to the rhts_post
template. Everything else is built on existing Beaker features. So it
has the distinct advantage of being easy and non-intrusive to implement.
My alternative suggestion is
[...]
At the harness level, the interesting parts would be:
1. The required "yuminstall" attribute of the harness element replaces
the ks_meta="harness=<yum arg>" part of Dan's initial design
The explicit nature of the "yuminstall" name leaves the door open to
less packaging system specific alternatives in the future
Well, as long as ks_meta exists we are tied to yum anyway :-) The real
issue here is not that the variable "harness=package_or_url" is tied to
yum, but that ks_meta itself it.
The way forward I see is that ks_meta ("Kickstart Metadata") becomes
"Installer Template Variables". They become a cross-installer way for
people to customise the installation -- Beaker takes care of translating
those variables into whatever syntax and meaning is appropriate for the
installer. In that case having a variable "harness=something" is not
yum-specific at all.
2. The optional "label" attribute gives us something human readable to
show in the UI (defaulting to the "yuminstall" value)
I do like this idea of having some nice way to display "what harness ran
this recipe" in Beaker's UI, but requiring the user to give a label in
every recipe is not great. We could just display the package name as is
(or the basename if it was a full URL).
A nicer idea might be to have the harness "check in" at the start of the
recipe, report its name and version (and its configuration, and other
useful info too?) -- Marian Csontos has suggested something similar in
the past, and you can already get most of the way there by just having
the harness upload a recipe log under a certain well-known name
("debug/harness-checkin.txt" or whatever).
3. The ability to request reservation of the system after execution of
the recipe is separated out from the definition of what the harness
should do
4. "reservesys" is an *element* to leave the door open to possible
future feature like <reservesys onsuccess="false"> to only reserve it in
some cases.
+1000 to this, except as mentioned it doesn't need to be tied to this
harness proposal at all.
5. The configuration mechanism for a custom harness is a choice between:
- a Beaker task list (for the benefit of custom harnesses that were
originally built to work within Beaker's current limitations)
- a custom harness-specific "config" element which can contain
arbitrary XML (such as a Git URL reference).
As mentioned above, I think this breaks some fundamentals of "what is
a recipe" in Beaker.
_______________________________________________
Beaker-devel mailing list
Beaker-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/beaker-devel