On 02/21/2013 11:12 AM, Bill Peck wrote:
On 02/20/2013 06:40 PM, Dan Callaghan wrote:
> Excerpts from Nick Coghlan's message of 2013-02-20 16:30:56 +1000:
>> One common complaint from folks attempting to use the current private
>> XML-RPC API to create a custom harness is that there's no way for the
>> code running on the system under test to add tasks to the recipe. This
>> means that an entire test run using an alternate harness will be
>> collapsed into a single task result in Beaker. That's ugly, and makes
>> it hard to upload good logs, provide decent progress reporting,
>> granular test results, etc.
> Well, I don't think this is a "common" complaint. We know of at least
> one use case for it though: having a single task which points at an
> autotest control file, where the actual list of tests is given.
>
> Also, the entire test run wouldn't need to be reported under one result.
> Remember, a task can have zero or more "results". So you can report each
> test result separately under the one mega-task. It's not so bad, you
> still get granular test results and logs, just no sensible progress
> reporting.
>
> So I don't see this (the harness adding new tasks) as a hugely important
> use case.
I've never liked this RFE. I was warming to until I saw your comment
below about cloning. I really think the task list should be immutable.
Yeah, I'm thoroughly persuaded it's a bad idea. It would be good for the
design proposal to capture this as an explicit "No, we are not doing
this, for these reasons", and explain what harness authors should do
instead (i.e. reporting multiple results for a single task). That way
we'll have something we can put to if anyone else asks for the feature.
I also realised harness authors are now free to upload an external link
if there's another service that provides progress tracking for that harness.
So, +1 that alternate harnesses still require a single task entry, and
we design any alternate harness related schema changes around the
existing "a recipe has an immutable list of one or more tasks" concept.
What I can see benefit from would be allowing the matrix report to
work
with sub-task results, but I have no idea how to make that work without
horrible performance. It may not be worth "fixing" in beaker and saying
you can pull the results out of beaker into your own reporting tool to
drill deeper and show regressions, etc..
+1. Aside from wanting to provide an integrated Graphite-backed live
dashboard at some point (since that's a great way to get a rough
overview of the current health of a Beaker installation), I'm generally
trying to get Beaker itself out of the metrics display and analysis game
and instead say "here is how you can access the raw data, do what you
want with it".
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (
http://beaker-project.org/)
PulpDist Development Lead (
http://pulpdist.readthedocs.org)