[Beaker-devel] Docs/overview for using autotest in Beaker?

Nick Coghlan ncoghlan at redhat.com
Tue Nov 5 02:00:59 UTC 2013


On 11/05/2013 07:51 AM, Don Zickus wrote:
> On Tue, Oct 29, 2013 at 01:33:33PM +1000, Nick Coghlan wrote:
>> Ping :)
>>
>> Even a cursory blog post somewhere (or a reply to this email!)
>> explaining to people how to *experiment* with this would be helpful. At
>> the moment I'm having to point people at the merged pull request to say
>> "Yes, in theory, you can use the upcoming version of autotest as a
>> Beaker test harness", but I have zero resources to give them on how to
>> actually run it (because I don't actually know myself).
> 
> So playing with things a little more..
> 
>>
>> 1. How do you get the experimental version of autotest to play with? Is
>> there a pre-built RPM somewhere that people can use, or do they have to
>> build their own?
> 
> Unfortunately, I see autotest-0.15 has not been packaged with Fedora 20
> for some reason.
> 
> Lucas,
> 
> Why is Fedora using a version of autotest from a year ago instead of
> something far more recent????
> 
> 
> So git clone https://github.com/autotest/autotest.git
> 
> will get you the latest autotest.
> 
> python setup.py build
> python setup.py install
> 
> (though I had to disable version=version.get_version in setup.py and
> comment out the BEAKER_RECIPE_ID and BEAKER_LAB_CONTROLLER_URL hacks in
> client/harness_beaker.py)
> 
> will install the autotest pieces
> 
> Then as long as you have in your `env`
> BEAKER_RECIPE_ID and BEAKER_LAB_CONTROLLER_URL defined
> and AUTOTEST_HARNESS=beaker setup as meta data
> 
> autotest-local bootstrap
> 
> will request BEAKER_RECIPE_ID xml from BEAKER_LAB_CONTROLLER_URL, convert
> it into a control file for autotest to consume and start running tests
> (and reporting them to beaker).

OK, that sounds reasonable (especially once the RPM packaging is brought
up to date)

>> 2. What do you need to set as the alternative harness in
>> http://beaker-project.org/docs/alternative-harnesses/#selecting-an-alternative-harness-in-your-recipe
> 
> I need 'AUTOTEST_HARNESS=beaker' set.

OK, in order to get the
(ks_meta="harness=<package-or-URL-to-yum-install>") recipe attribute to
work properly, it looks like we may also need a
"autotest-beaker-harness" RPM that:

1. Depends on autotest itself
2. Sets "AUTOTEST_HARNESS=beaker" in the system environment variables
3. Arranges for "autotest-local bootstrap" to run on the next reboot

Otherwise you'll need to write a Beaker task that installs beah just to
bootstrap autotest, and installing beah is one of the things using an
alternative harness is supposed to avoid.

>> ?
>> 3. What's known to work, what's still quirky? (e.g. reboot)
> 
> reboot doesn't work for some reason.  I don't think running actual
> autotest-tests work.  I have only run with beaker tests through a job.
> And also running locally doesn't work (even though the tests are locally
> installed [unless you hack up some recipe id workarounds..]).

We're going to have to get the reboot glitch sorted in order to
interoperate with the alternative harness selection mechanism.

>> 4. What's different (if anything) compared to normal usage of autotest?
> 
> Normal usage of autotest is 'autotest-local <some control file>' or
> 'autotest-local run <testname>' (where testname is from 'autotest-local
> list')
> 
> But that is running autotest locally/offline.  I can get it to work with
> beaker tests, but I have to tweak some env variables to fake it (not
> ideal).  So run now it has to run automated, which isn't ideal either.

The current recommendation for local beah testing is to assume offline
execution if JOBID isn't set:
http://beaker-project.org/docs/user-guide/multihost.html#standalone-execution-of-multihost-tests

Something similar may work here: if AUTOTEST_HARNESS=beaker is set, but
the BEAKER_* environment variables are missing, perhaps there could be
some way to provide the recipe definition XML file directly rather than
needing to retrieve it from the server?

>> At the moment, it's impossible for even the core Beaker team to help
>> with this (let alone anyone else that may be interested), as we don't
>> even know how to get set up to run it.
> 
> Yeah, I was trying to figure out a workflow that I wanted to run autotest
> with.
> 
> Initially I wanted to run tests locally for developers and through
> automation for QE folks.  I can run autotest tests for developers and
> beaker tests for QE folks.  Ideally I need to mix and match in both
> directions. :-/
> 
> It shouldn't be hard to do, just need to hack up some glue code to detect
> and run things properly.
> 
>>
>> I'd also like to submit a talk on Beaker to the Continuous Integration
>> miniconf at LCA, and that becomes a lot more compelling when I can point
>> to a post about *how* to run autotest inside Beaker, rather than just to
>> the pull request that made it possible.
> 
> Yeah, my stuff needs more work.  And more thought.  Perhaps some feedback
> about workflows you had in mind, might help target this a little more.
> 
> I'll keep poking at it this week.

Hopefully the above helps - it sounds to me like a way to provide an XML
definition directly for local testing + the "autotest-beaker-harness"
RPM to integrate with the alternative harness selection mechanism are
the key next steps.

Then on to the fun part of getting non-Beaker autotest tests running
inside Beaker :)

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane

Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)


More information about the Beaker-devel mailing list