On 08/31/11 - 10:49:36AM, James Laska wrote:
Greetings,
I've been looking into integrating existing oz unittests with jenkins. It
wasn't hard to have them run in jenkins as is, but I was looking give jenkins a
better idea for pass/fail data over time. One way to do this was to convert
the tests to use py.test (or unittest -- but that's a little crusty). Py.test
can emit junit XML test output, which is jenkins friendly, and has much less
stock test class/method bloat. I've modified the existing oz unittests to be
py.test friendly.
Another annoyance with running tests in jenkins is that the nodes need to be
manually setup with appropriate dependencies ahead of time. While this isn't
hard, it's maintenance and one more thing to forget/fail when testing. The new
test driver (runtests.sh) will setup a python virtualenv and install required
dependencies there. This is intended to handle deps installation during
unittest execution.
> .gitignore | 6 +
> Makefile | 6 +-
> tests/dependencies.txt | 2 +
> tests/factory/run.sh | 5 -
> tests/factory/test_factory.py | 311 +++++++++++++++++++++--------------------
> tests/runtests.sh | 90 ++++++++++++
> tests/tdl/run.sh | 128 -----------------
> tests/tdl/test.cfg | 135 ++++++++++++++++++
> tests/tdl/test_tdl.py | 131 +++++++++++++++++-
> 9 files changed, 524 insertions(+), 290 deletions(-)
Arg. I know I said I was willing to entertain these patches, and I was, but
I have to say that I am not a fan. It turns what is simple bash and python
unittests into a morass of config files, scripts, and magic. Plus it adds 3
new dependencies to run the unittests, which is an additional barrier to entry
for new contributors.
So there are 3 main problems that you see with the current unittests (the 2
listed above plus one you mentioned on IRC):
1) It is a sort of odd combination of bash and python, and you need to modify
the code every time you want to add a new test
2) You want JUnit XML output to feed to jenkins
3) You want to have some sort of automatic dependency resolution for jenkins.
I think we can easily handle 1) by re-writing the small portions of this that
are in bash in python, and make it smarter. For instance, take what
you've done for moving the tests in their own directory, and just have the
python code iterate over those tests.
For 2), after seeing the output of these patches, it seems to me that it should
be easy enough to have the existing python code generate JUnit XML as output.
Maybe we have the code take a flag (--junitxml=output/path), and then we can
build it up ourselves (it is, after all, simple XML).
The third problem you have is a little more difficult. I want to keep running
the tests as simple as possible, but you want automatic dependency stuff.
I wonder if we can add a new "virtualenv" target to the Makefile, which only
jenkins will call. Then it could call out to the tests as re-written above.
What do you think about that proposal?
--
Chris Lalancette