On Fri, 2011-09-02 at 07:12 +0100, Mark McLoughlin wrote:
On Thu, 2011-09-01 at 09:23 -0400, Chris Lalancette wrote:
> On 08/31/11 - 10:49:36AM, James Laska wrote:
> >
> > Greetings,
> >
> > I've been looking into integrating existing oz unittests with jenkins. It
> > wasn't hard to have them run in jenkins as is, but I was looking give
jenkins a
> > better idea for pass/fail data over time. One way to do this was to convert
> > the tests to use py.test (or unittest -- but that's a little crusty).
Py.test
> > can emit junit XML test output, which is jenkins friendly, and has much less
> > stock test class/method bloat. I've modified the existing oz unittests to
be
> > py.test friendly.
> >
> > Another annoyance with running tests in jenkins is that the nodes need to be
> > manually setup with appropriate dependencies ahead of time. While this
isn't
> > hard, it's maintenance and one more thing to forget/fail when testing. The
new
> > test driver (runtests.sh) will setup a python virtualenv and install required
> > dependencies there. This is intended to handle deps installation during
> > unittest execution.
> >
> > > .gitignore | 6 +
> > > Makefile | 6 +-
> > > tests/dependencies.txt | 2 +
> > > tests/factory/run.sh | 5 -
> > > tests/factory/test_factory.py | 311
+++++++++++++++++++++--------------------
> > > tests/runtests.sh | 90 ++++++++++++
> > > tests/tdl/run.sh | 128 -----------------
> > > tests/tdl/test.cfg | 135 ++++++++++++++++++
> > > tests/tdl/test_tdl.py | 131 +++++++++++++++++-
> > > 9 files changed, 524 insertions(+), 290 deletions(-)
>
> Arg. I know I said I was willing to entertain these patches, and I was, but
> I have to say that I am not a fan. It turns what is simple bash and python
> unittests into a morass of config files, scripts, and magic. Plus it adds 3
> new dependencies to run the unittests, which is an additional barrier to entry
> for new contributors.
FWIW, I like what OpenStack's glance has done with their unit tests
using virtualenv and nose. Maybe look at that?
https://github.com/openstack/glance
Thanks for the link Mark. I like their run_tests.sh driver in that it
offers a few additional arguments for managing the virtualenv (--clean,
--force, --no-virtual-env etc...). This might be helpful for Chris'
suggestion of optionally using virtualenv.
They still use a unittest-style tests. I don't mind that structure
since it is well documented, it's just extra scaffolding. All tests
must be importable modules, subclass unittest.TestCase, use unusual (but
documented) assert*() methods for doing normal python things (like
comparisons and asserts) and rely on alphanum sorting to determine test
execution order (which may change as tests mature).
A comparison between nose and py.test ...
https://www.ibm.com/developerworks/aix/library/au-python_test/
Another comparison ...
http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-3-pytes...
Using nose would still require an additional dependency (python-nose).
I guess it really comes down to preference on what folks are comfortable
with maintaining/using. I'm happy to adjust to use unittest or nose if
desired. My current use of py.test was in part because of built-in
support for --junit output (without requiring packaging another
testrunner like HTMLTestRunner.py [1] or unittest-xml-reporting [2] or
pyjunitxml [3]), getting something going faster using the existing code
Chris developed and of course ... to explore/learn something new.
/me standing by
Thanks,
James
[1]
http://tungwaiyip.info/software/HTMLTestRunner.html
[2]
https://github.com/danielfm/unittest-xml-reporting
[3]
https://launchpad.net/pyjunitxml