----- Original Message -----
I was pondering the question of running tests from Git repos
recently,
and Dan's recent efforts in resurrecting patchbot (to sanity check
patches on Gerrit), which kinda does exactly that for our own dogfood
tests, prompted me to post my ideas for people to poke holes in :)
Goal:
Allow developers to run tests based on the existing results reporting
infrastructure directly from Git, without the need to build a test
RPM
I don't have comments directly RE: your ideas since i haven't thought
about this in a while, but i figured it's worth re-sharing an old
thread where i threw out my own ideas about this topic way back when.
http://post-office.corp.redhat.com/archives/beaker-user-list/2010-June/ms...
-Chris
Benefits:
- eliminates a step in the test development workflow (bkr task-add)
- avoids versioning issues when updating tests
- potentially helps with VM-image-library-based testing (since the
runtest.sh in a "run from Git" task gives us another location for
harness code execution, independent of kickstart %post snippets)
(Deliberate) Limitations:
- retains the dependency on beah/beakerlib for setting up the
environment and reporting results
- thus doesn't help with the cross-platform testing problem (and, to
be
frank, I don't think we *should* ever try to solve that problem
directly
- instead, we eventually need to figure out how to integrate STAF,
autotest or both as alternatives to beah for the components that run
on
the system under test. That's a much harder problem than simply
allowing
ordinary beak/beakerlib tests to be executed from Git, though, since
the
differences in execution and reporting models would need to be
aligned
somehow)
Design Details:
The proposal is fairly simple:
- create a new standard task (maintained in the main beaker repo)
that
accepts parameters defining:
- a Git URL to clone
- a test execution command to be run from the base directory of
the
clone
- the task's runtest.sh would take care of any setup-and-teardown
needed
to clone the repo, run the test and then delete the repo again
- the exact Git changeset id for the checked out repo would be
reported
as part of the test details (for cases where the submitted URL
doesn't
specify a particular tag or revision)
- (What else would we need in the task parameters to make up for the
lack of per-test-case tasks? Probably everything that would otherwise
be
set in runtest.sh)
Possible additions:
- while the standard version of the task would permit arbitrary Git
URLs, it likely wouldn't be hard to create a modified version that
only
allowed URLs from a defined subset of hosts.
Actually implementing this isn't high on my priority at the moment,
but
if the above idea seems workable, then it should be a lot easier to
make
happen than an approach that requires server side changes.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (
http://beaker-project.org/)
GlobalSync Development Lead (
http://pulpdist.readthedocs.org)
_______________________________________________
Beaker-devel mailing list
Beaker-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/beaker-devel
--
Chris Ward <cward(a)redhat.com>
Quality Engineering - Third-Party QE
--
" Quality is planned in, not inspected in."
" The squeaky mouse gets the cheese!"
" Unless."
--