On Mon, Apr 15, 2013 at 05:39:18PM +1000, Dan Callaghan wrote:
Excerpts from Don Zickus's message of 2013-04-13 05:09:48 +1000:
> Hi Folks,
>
> Throwing this over the wall for a new round of reviewing.
Thanks for all your work on these patches, Don!
I had a look through them all this afternoon (I'm sorry it's taken me so
long to get to it). I see there is a fair bit of code in there to
support running existing Beaker/RHTS-style tasks inside autotest
(emulating rhts-* commands, etc). I didn't realise that before.
Yeah. I was trying to leverage all the old tests to help me address all
the kinks of the beaker interface. It would also make it easier to have
devleopers utilize existing tests (as they are not going away any time
soon).
Using autotest to run existing Beaker tasks is interesting and may be
worth pursuing, but what's *more* interesting to us is the ability to
run autotest tests as part of a Beaker recipe. We've had some interest
from people who have existing autotest-based tests who would like to
make use of Beaker's scheduling abilities.
Of course. That part is fairly simple but would require me writing lots
of tests to exercise all the beaker paths. See below.
Given my very limited understanding of autotest, this is what I was
imagining we would need to do:
* Find a way to map native autotest control files to Beaker recipe XML.
The control files are real Python code, which makes them very
powerful, so I don't think it makes sense to try to abstract that away
in Beaker's XML. It might make more sense to allow the control file to
be embedded literally in the recipe XML. For example we could add
support for something like this:
Long ago, Jan and I talked about just grabbing the tests straight from a
git tree. Much simpler and straight forward than embedding scripts in the
XML (ewww!). Bill P. has something like this working with his simple
harness.
The idea is to pass a known type of <task> with a git url pointer that
included the tesname and option control file.
The harness would recoginize the autotest task, download the git tree or
test and then execute the control file embedded in the test.
<recipe>
...
<script><![CDATA[
job.run_test(...)
]]></script>
</recipe>
where <script/> replaces the series of <task/> elements that you would
normally see in a Beaker recipe.
* Write some code for autotest to fetch the Beaker recipe XML and
configure itself accordingly (that's the bootstrap stuff in these
patches).
* Wrote some code for autotest to collect test output and results and
report them to Beaker somehow (I don't see anything for this in these
patches, aside from the rhts-submit-log/rhts-report-result emulation?)
Well, the harness has the recipe/task/result upload functions that allow
this happen. Autotest, I believe, doesn't have a direct result reporting
mechanism. Instead, I think, one would write a sub-task which
automatically reports back to beaker and shows up on the autotest UI
output. So it's a mind shift in developing tests. All output of those
tests are collected in a 'resultdir' which automatically get uploaded to
beaker when the task/sub-task is completed.
It would be great if we could get this working first, and then the
support for running RHTS-style tasks could be an extra patch set on top
of that.
Like I said, I was using the old tests to help with testing of the
harness. It started off with some quick hacking, then I got sucked in a
little further.
My next steps was to fix the reboot stuff because that isn't quite working
correctly. Then I should be able to quickly setup the normal autotest
workflow from beaker.
I'll coordinate with Bill, so we can have similar workflows.
Cheers,
Don