On Thu, 2011-09-01 at 09:23 -0400, Chris Lalancette wrote:
On 08/31/11 - 10:49:36AM, James Laska wrote:
>
> Greetings,
>
> I've been looking into integrating existing oz unittests with jenkins. It
> wasn't hard to have them run in jenkins as is, but I was looking give jenkins a
> better idea for pass/fail data over time. One way to do this was to convert
> the tests to use py.test (or unittest -- but that's a little crusty). Py.test
> can emit junit XML test output, which is jenkins friendly, and has much less
> stock test class/method bloat. I've modified the existing oz unittests to be
> py.test friendly.
>
> Another annoyance with running tests in jenkins is that the nodes need to be
> manually setup with appropriate dependencies ahead of time. While this isn't
> hard, it's maintenance and one more thing to forget/fail when testing. The new
> test driver (runtests.sh) will setup a python virtualenv and install required
> dependencies there. This is intended to handle deps installation during
> unittest execution.
>
> > .gitignore | 6 +
> > Makefile | 6 +-
> > tests/dependencies.txt | 2 +
> > tests/factory/run.sh | 5 -
> > tests/factory/test_factory.py | 311
+++++++++++++++++++++--------------------
> > tests/runtests.sh | 90 ++++++++++++
> > tests/tdl/run.sh | 128 -----------------
> > tests/tdl/test.cfg | 135 ++++++++++++++++++
> > tests/tdl/test_tdl.py | 131 +++++++++++++++++-
> > 9 files changed, 524 insertions(+), 290 deletions(-)
Arg. I know I said I was willing to entertain these patches, and I was, but
I have to say that I am not a fan. It turns what is simple bash and python
unittests into a morass of config files, scripts, and magic. Plus it adds 3
new dependencies to run the unittests, which is an additional barrier to entry
for new contributors.
What are the new dependencies, python-virtualenv and py.test?
I'm not a big fan of writing tests in bash, it's easy to write, and
easier to write wrong and hard to debug. Plus there are no libs to rely
on, you have to build the custom scaffolding. But that's just a
personal preference.
I could have gone with unittest, which is built-in to python. But I've
done it plenty of times and it requires too much scaffolding to do basic
tasks. I liked py.test since you just write python scripts as your
tests ... but it does add a new dependency.
So there are 3 main problems that you see with the current unittests
(the 2
listed above plus one you mentioned on IRC):
1) It is a sort of odd combination of bash and python, and you need to modify
the code every time you want to add a new test
The current proposed patchset requires modifying a config file when
adding new tests. I didn't consider that code, more config, but it does
require a commit of sorts whenever adding/modifying tests. A previous
version of the patchset required no modification at all, but offered
less flexibility. Tests were split into two directories, expected_pass/
and expected_fail/ I can return to this if desired. I liked the
current .cfg approach since it allowed for customizing the exact failure
case, and I tried to self-document the use of options in the config
file. If that's overkill, I can definitely return to the simpler
directory-based approach.
2) You want JUnit XML output to feed to jenkins
Yeah, I think that'd be helpful all around (unless the tests always
pass) :)
3) You want to have some sort of automatic dependency resolution for
jenkins.
That'd be nice ... it's also for folks running 'make unittests' from a
git checkout who don't have all the runtime/test dependencies installed.
Rather, the later is how I've used it in the past.
I think we can easily handle 1) by re-writing the small portions of
this that
are in bash in python, and make it smarter. For instance, take what
you've done for moving the tests in their own directory, and just have the
python code iterate over those tests.
In the current approach, I left the .tdl test files where they were, but
used a .cfg to define expected outcomes. Would it be easier to remove
the .cfg and just have tests exist in expected_fail/ expected_pass/ ?
For 2), after seeing the output of these patches, it seems to me that
it should
be easy enough to have the existing python code generate JUnit XML as output.
Maybe we have the code take a flag (--junitxml=output/path), and then we can
build it up ourselves (it is, after all, simple XML).
My preference would be to avoid custom-built XML generation (that just
seems prone to rot) and let the python framework (py.test) do that for
us. I figured that would be less to maintain in the long-run. Let me
know what you think.
The third problem you have is a little more difficult. I want to
keep running
the tests as simple as possible, but you want automatic dependency stuff.
I wonder if we can add a new "virtualenv" target to the Makefile, which only
jenkins will call. Then it could call out to the tests as re-written above.
Oh, I see ... so you'd be required to install the deps manually when
running 'make unittests'. However, if testing with jenkins, you could
do 'make virtualenv unittests' which would alleviate manual dependency
installation? That's a neat idea, I could set that up.
What do you think about that proposal?
Thanks for your feedback! Whatever resolves the 3 problems you
highlighted without introducing too much additional maintenance. I can
get busy re-working based on your feedback.
Thanks,
James