On 11/05/2013 07:51 AM, Don Zickus wrote:
On Tue, Oct 29, 2013 at 01:33:33PM +1000, Nick Coghlan wrote:
> Ping :)
>
> Even a cursory blog post somewhere (or a reply to this email!)
> explaining to people how to *experiment* with this would be helpful. At
> the moment I'm having to point people at the merged pull request to say
> "Yes, in theory, you can use the upcoming version of autotest as a
> Beaker test harness", but I have zero resources to give them on how to
> actually run it (because I don't actually know myself).
So playing with things a little more..
>
> 1. How do you get the experimental version of autotest to play with? Is
> there a pre-built RPM somewhere that people can use, or do they have to
> build their own?
Unfortunately, I see autotest-0.15 has not been packaged with Fedora 20
for some reason.
Lucas,
Why is Fedora using a version of autotest from a year ago instead of
something far more recent????
So git clone
https://github.com/autotest/autotest.git
will get you the latest autotest.
python setup.py build
python setup.py install
(though I had to disable version=version.get_version in setup.py and
comment out the BEAKER_RECIPE_ID and BEAKER_LAB_CONTROLLER_URL hacks in
client/harness_beaker.py)
will install the autotest pieces
Then as long as you have in your `env`
BEAKER_RECIPE_ID and BEAKER_LAB_CONTROLLER_URL defined
and AUTOTEST_HARNESS=beaker setup as meta data
autotest-local bootstrap
will request BEAKER_RECIPE_ID xml from BEAKER_LAB_CONTROLLER_URL, convert
it into a control file for autotest to consume and start running tests
(and reporting them to beaker).
OK, that sounds reasonable (especially once the RPM packaging is brought
up to date)
> 2. What do you need to set as the alternative harness in
>
http://beaker-project.org/docs/alternative-harnesses/#selecting-an-altern...
I need 'AUTOTEST_HARNESS=beaker' set.
OK, in order to get the
(ks_meta="harness=<package-or-URL-to-yum-install>") recipe attribute to
work properly, it looks like we may also need a
"autotest-beaker-harness" RPM that:
1. Depends on autotest itself
2. Sets "AUTOTEST_HARNESS=beaker" in the system environment variables
3. Arranges for "autotest-local bootstrap" to run on the next reboot
Otherwise you'll need to write a Beaker task that installs beah just to
bootstrap autotest, and installing beah is one of the things using an
alternative harness is supposed to avoid.
> ?
> 3. What's known to work, what's still quirky? (e.g. reboot)
reboot doesn't work for some reason. I don't think running actual
autotest-tests work. I have only run with beaker tests through a job.
And also running locally doesn't work (even though the tests are locally
installed [unless you hack up some recipe id workarounds..]).
We're going to have to get the reboot glitch sorted in order to
interoperate with the alternative harness selection mechanism.
> 4. What's different (if anything) compared to normal usage of
autotest?
Normal usage of autotest is 'autotest-local <some control file>' or
'autotest-local run <testname>' (where testname is from
'autotest-local
list')
But that is running autotest locally/offline. I can get it to work with
beaker tests, but I have to tweak some env variables to fake it (not
ideal). So run now it has to run automated, which isn't ideal either.
The current recommendation for local beah testing is to assume offline
execution if JOBID isn't set:
http://beaker-project.org/docs/user-guide/multihost.html#standalone-execu...
Something similar may work here: if AUTOTEST_HARNESS=beaker is set, but
the BEAKER_* environment variables are missing, perhaps there could be
some way to provide the recipe definition XML file directly rather than
needing to retrieve it from the server?
> At the moment, it's impossible for even the core Beaker team
to help
> with this (let alone anyone else that may be interested), as we don't
> even know how to get set up to run it.
Yeah, I was trying to figure out a workflow that I wanted to run autotest
with.
Initially I wanted to run tests locally for developers and through
automation for QE folks. I can run autotest tests for developers and
beaker tests for QE folks. Ideally I need to mix and match in both
directions. :-/
It shouldn't be hard to do, just need to hack up some glue code to detect
and run things properly.
>
> I'd also like to submit a talk on Beaker to the Continuous Integration
> miniconf at LCA, and that becomes a lot more compelling when I can point
> to a post about *how* to run autotest inside Beaker, rather than just to
> the pull request that made it possible.
Yeah, my stuff needs more work. And more thought. Perhaps some feedback
about workflows you had in mind, might help target this a little more.
I'll keep poking at it this week.
Hopefully the above helps - it sounds to me like a way to provide an XML
definition directly for local testing + the "autotest-beaker-harness"
RPM to integrate with the alternative harness selection mechanism are
the key next steps.
Then on to the fun part of getting non-Beaker autotest tests running
inside Beaker :)
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (
http://beaker-project.org/)