James Laska wrote:
Thanks Jeff!
Comments below ...
On Mon, 2009-02-09 at 10:05 +1000, Jeff Fearn wrote:
> James Laska wrote:
>> Greetings,
>>
>> There's been a lot of "water-cooler" discussion around what test
>> harness to use to manage test execution on a system. The harness
>> that's getting the most discussion now is STAF
>> (
http://staf.sourceforge.net).
>>
>> Is there a set requirements for the test harness?
> Edited from:
https://engineering.redhat.com/trac/rhts/wiki/TestHarnessDesign
The link provided isn't accessible outside of Red Hat ... can this
information be posted to the beaker wiki?
https://fedorahosted.org/beaker/
https://fedorahosted.org/beaker/wiki/TestHarnessRequirements
I edited it when copying, let me know if you think I chopped anything
important.
>> Is this intended as a replacement for the runtest.sh process outlined
>> at the ancient
>>
https://testing.108.redhat.com/wiki/index.php/Rhts/Docs/TestWriting#RHTS_...
>> ?
> Yes, although at it's simplest STAF can be used as a way to run test.sh tests
across the network and view/fetch the logs.
>
> FWIW here is my view of how STAF would fit in to the Beaker world.
>
> STAF is packaged and run as an initV service. The lab controller (LC) installs STAF
on the test machine during provisioning. Tests can be installed on the test machine during
provisioning or after the STAF service has been started on the test machine; it will
probably vary on the individual test requirements.
>
> Once the test machine is up it can either run the tests automatically, or the LC can
use it's local STAF process to control the work flow on the test server according to
the recipe. Running STAF on the LC allows two way communication between the LC and the
test machines. Furthermore if a developer has STAF running locally they can be granted
access to the test machine by the LC, thereafter they can interact directly with the test
machine. This may be useful to track down odd bugs or run tests that require human
intervention. STAF has a mutli-level permission's system that should suffice for
this.
>
> The LC program would sit above STAF on the LC and control STAF either via the CLI or,
more likely, via one of the language specific API's. i.e. if the LC was written in
Python then it would use the PySTAF API to hook directly in to STAF instead of using the
shell access to STAF.
>
> Hopefully this all makes sense :)
It does thanks! A few follow-up questions ...
* How close are you to a demo/proof_of_concept? Are there any tasks
that you could use fedora-qa assistance on?
STAF comes with a Java demo that is a reasonable example of how things
work. Coming up with a demo more specific to beaker really requires
information on how the LC and STAF will coordinate.
Assuming you have STAF on two machines, with existing RHTS tests on M1 and
the developer on M2, then you can run the RHTS tests on M1 from M2 by running:
PySTAF M1 PROCESS START COMMAND "make" RETURNSTDOUT RETURNSTDERR WORKDIR
"/mnt/tests/CoreOS/coreutils/su/notty" WAIT
You have to allow access to M1 from M2 by editing the STAF conf file, and
you have to replace M1 and M2 with the real machines names :)
e.g. on M1
vi /etc/STAF.cfg
(add line) trust machine M2 level 5
This allows M2 full access to M1, levels less than 5 have less access, and
you can have user instead of machine permissions.
PySTAF is a Python script shipped with STAF, there are Perl, TCL and Bash
equivalents.
* Are there packages built into Fedora that we could open up for
feedback?
http://koji.fedoraproject.org/koji/taskinfo?taskID=1116308
* Will we need to migrate testing.108 packages into a new format?
Only if you want to add extra functionality to the tests. i.e. STAF has
logging and time out options that can be controlled locally or remotely,
if you wanted that extra functionality then the test would have to be
rewritten.
* Do you have any thoughts on how we should store tests in a version
controlled repo? THis might be outside the scope of STAF, but currently
our tests our stored in a version controlled repository by package name.
Should this mechanism be used going forward?
This sounds reasonable to me.
Cheers, Jeff.
--
Jeff Fearn <jfearn(a)redhat.com>
Software Engineer
Engineering Operations
Red Hat, Inc
Freedom ... courage ... Commitment ... ACCOUNTABILITY