On Fri, 2011-09-02 at 10:04 -0400, Chris Lalancette wrote:
On 09/01/11 - 09:48:42AM, James Laska wrote:
> > Arg. I know I said I was willing to entertain these patches, and I was, but
> > I have to say that I am not a fan. It turns what is simple bash and python
> > unittests into a morass of config files, scripts, and magic. Plus it adds 3
> > new dependencies to run the unittests, which is an additional barrier to entry
> > for new contributors.
>
> What are the new dependencies, python-virtualenv and py.test?
And python-pip.
Oh I forgot, the only hard requirement is python-virtualenv. Since
anything else is handled magically by virtualenv. Using the nice idea
of an optional virtualenv, those other requirements would be needed.
>
> I'm not a big fan of writing tests in bash, it's easy to write, and
> easier to write wrong and hard to debug. Plus there are no libs to rely
> on, you have to build the custom scaffolding. But that's just a
> personal preference.
Yeah, I'm with you here. bash was just easy for me at the time, but I totally
understand converting to python. Heck, the original version of Oz (before it
was a full project) was all written in bash, and it was a mess :).
Haha, I thought after I wrote this that it might come across that your
bash was bad. Apologies, that was not at all the case. It was actually
pretty easy to read and you could tell it was done by someone with
development experience.
> I could have gone with unittest, which is built-in to python.
But I've
> done it plenty of times and it requires too much scaffolding to do basic
> tasks. I liked py.test since you just write python scripts as your
> tests ... but it does add a new dependency.
>
> > So there are 3 main problems that you see with the current unittests (the 2
> > listed above plus one you mentioned on IRC):
> > 1) It is a sort of odd combination of bash and python, and you need to modify
> > the code every time you want to add a new test
>
> The current proposed patchset requires modifying a config file when
> adding new tests. I didn't consider that code, more config, but it does
> require a commit of sorts whenever adding/modifying tests. A previous
> version of the patchset required no modification at all, but offered
> less flexibility. Tests were split into two directories, expected_pass/
> and expected_fail/ I can return to this if desired. I liked the
> current .cfg approach since it allowed for customizing the exact failure
> case, and I tried to self-document the use of options in the config
> file. If that's overkill, I can definitely return to the simpler
> directory-based approach.
OK, so you don't care too much about modifying the code. I also don't care;
you are going to have to commit a patch to add the new test anyway, so adding
one more line to the test controller (run_test or whatever) doesn't seem like
a huge deal.
Yeah, true.
So the big point on 1) is converting to python. To that end,
I've come up with the attached patch; let me know what you think.
Nice, thank you! I'll play with this shortly and get some feedback.
So with my proposed patch we've addressed 1) by converting
everything to
python. If you can work on 3) to add a virtualenv Makefile target, that would
be great. That leaves us with 2); I'm totally with you on avoiding custom code
where we can. I think it will require a bit more research, but if we can find
something that can generate the junit XML (and only the junit XML), that might
be workable.
I found these two projects that might work ...
https://github.com/danielfm/unittest-xml-reporting
https://launchpad.net/pyjunitxml
I tested unittest-xml-reporting and that seems to work. I haven't tried
pyjunitxml. Either would need to be packaged and included in Fedora.
I'll be glad to work that.
Thanks,
James