[Beaker-devel] Docs/overview for using autotest in Beaker?
Nick Coghlan
ncoghlan at redhat.com
Tue Nov 5 02:00:59 UTC 2013
On 11/05/2013 07:51 AM, Don Zickus wrote:
> On Tue, Oct 29, 2013 at 01:33:33PM +1000, Nick Coghlan wrote:
>> Ping :)
>>
>> Even a cursory blog post somewhere (or a reply to this email!)
>> explaining to people how to *experiment* with this would be helpful. At
>> the moment I'm having to point people at the merged pull request to say
>> "Yes, in theory, you can use the upcoming version of autotest as a
>> Beaker test harness", but I have zero resources to give them on how to
>> actually run it (because I don't actually know myself).
>
> So playing with things a little more..
>
>>
>> 1. How do you get the experimental version of autotest to play with? Is
>> there a pre-built RPM somewhere that people can use, or do they have to
>> build their own?
>
> Unfortunately, I see autotest-0.15 has not been packaged with Fedora 20
> for some reason.
>
> Lucas,
>
> Why is Fedora using a version of autotest from a year ago instead of
> something far more recent????
>
>
> So git clone https://github.com/autotest/autotest.git
>
> will get you the latest autotest.
>
> python setup.py build
> python setup.py install
>
> (though I had to disable version=version.get_version in setup.py and
> comment out the BEAKER_RECIPE_ID and BEAKER_LAB_CONTROLLER_URL hacks in
> client/harness_beaker.py)
>
> will install the autotest pieces
>
> Then as long as you have in your `env`
> BEAKER_RECIPE_ID and BEAKER_LAB_CONTROLLER_URL defined
> and AUTOTEST_HARNESS=beaker setup as meta data
>
> autotest-local bootstrap
>
> will request BEAKER_RECIPE_ID xml from BEAKER_LAB_CONTROLLER_URL, convert
> it into a control file for autotest to consume and start running tests
> (and reporting them to beaker).
OK, that sounds reasonable (especially once the RPM packaging is brought
up to date)
>> 2. What do you need to set as the alternative harness in
>> http://beaker-project.org/docs/alternative-harnesses/#selecting-an-alternative-harness-in-your-recipe
>
> I need 'AUTOTEST_HARNESS=beaker' set.
OK, in order to get the
(ks_meta="harness=<package-or-URL-to-yum-install>") recipe attribute to
work properly, it looks like we may also need a
"autotest-beaker-harness" RPM that:
1. Depends on autotest itself
2. Sets "AUTOTEST_HARNESS=beaker" in the system environment variables
3. Arranges for "autotest-local bootstrap" to run on the next reboot
Otherwise you'll need to write a Beaker task that installs beah just to
bootstrap autotest, and installing beah is one of the things using an
alternative harness is supposed to avoid.
>> ?
>> 3. What's known to work, what's still quirky? (e.g. reboot)
>
> reboot doesn't work for some reason. I don't think running actual
> autotest-tests work. I have only run with beaker tests through a job.
> And also running locally doesn't work (even though the tests are locally
> installed [unless you hack up some recipe id workarounds..]).
We're going to have to get the reboot glitch sorted in order to
interoperate with the alternative harness selection mechanism.
>> 4. What's different (if anything) compared to normal usage of autotest?
>
> Normal usage of autotest is 'autotest-local <some control file>' or
> 'autotest-local run <testname>' (where testname is from 'autotest-local
> list')
>
> But that is running autotest locally/offline. I can get it to work with
> beaker tests, but I have to tweak some env variables to fake it (not
> ideal). So run now it has to run automated, which isn't ideal either.
The current recommendation for local beah testing is to assume offline
execution if JOBID isn't set:
http://beaker-project.org/docs/user-guide/multihost.html#standalone-execution-of-multihost-tests
Something similar may work here: if AUTOTEST_HARNESS=beaker is set, but
the BEAKER_* environment variables are missing, perhaps there could be
some way to provide the recipe definition XML file directly rather than
needing to retrieve it from the server?
>> At the moment, it's impossible for even the core Beaker team to help
>> with this (let alone anyone else that may be interested), as we don't
>> even know how to get set up to run it.
>
> Yeah, I was trying to figure out a workflow that I wanted to run autotest
> with.
>
> Initially I wanted to run tests locally for developers and through
> automation for QE folks. I can run autotest tests for developers and
> beaker tests for QE folks. Ideally I need to mix and match in both
> directions. :-/
>
> It shouldn't be hard to do, just need to hack up some glue code to detect
> and run things properly.
>
>>
>> I'd also like to submit a talk on Beaker to the Continuous Integration
>> miniconf at LCA, and that becomes a lot more compelling when I can point
>> to a post about *how* to run autotest inside Beaker, rather than just to
>> the pull request that made it possible.
>
> Yeah, my stuff needs more work. And more thought. Perhaps some feedback
> about workflows you had in mind, might help target this a little more.
>
> I'll keep poking at it this week.
Hopefully the above helps - it sounds to me like a way to provide an XML
definition directly for local testing + the "autotest-beaker-harness"
RPM to integrate with the alternative harness selection mechanism are
the key next steps.
Then on to the fun part of getting non-Beaker autotest tests running
inside Beaker :)
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
More information about the Beaker-devel
mailing list