On 09/01/11 - 09:48:42AM, James Laska wrote:
> Arg. I know I said I was willing to entertain these patches,
and I was, but
> I have to say that I am not a fan. It turns what is simple bash and python
> unittests into a morass of config files, scripts, and magic. Plus it adds 3
> new dependencies to run the unittests, which is an additional barrier to entry
> for new contributors.
What are the new dependencies, python-virtualenv and py.test?
And python-pip.
I'm not a big fan of writing tests in bash, it's easy to write, and
easier to write wrong and hard to debug. Plus there are no libs to rely
on, you have to build the custom scaffolding. But that's just a
personal preference.
Yeah, I'm with you here. bash was just easy for me at the time, but I totally
understand converting to python. Heck, the original version of Oz (before it
was a full project) was all written in bash, and it was a mess :).
I could have gone with unittest, which is built-in to python. But I've
done it plenty of times and it requires too much scaffolding to do basic
tasks. I liked py.test since you just write python scripts as your
tests ... but it does add a new dependency.
> So there are 3 main problems that you see with the current unittests (the 2
> listed above plus one you mentioned on IRC):
> 1) It is a sort of odd combination of bash and python, and you need to modify
> the code every time you want to add a new test
The current proposed patchset requires modifying a config file when
adding new tests. I didn't consider that code, more config, but it does
require a commit of sorts whenever adding/modifying tests. A previous
version of the patchset required no modification at all, but offered
less flexibility. Tests were split into two directories, expected_pass/
and expected_fail/ I can return to this if desired. I liked the
current .cfg approach since it allowed for customizing the exact failure
case, and I tried to self-document the use of options in the config
file. If that's overkill, I can definitely return to the simpler
directory-based approach.
OK, so you don't care too much about modifying the code. I also don't care;
you are going to have to commit a patch to add the new test anyway, so adding
one more line to the test controller (run_test or whatever) doesn't seem like
a huge deal. So the big point on 1) is converting to python. To that end,
I've come up with the attached patch; let me know what you think.
> 2) You want JUnit XML output to feed to jenkins
Yeah, I think that'd be helpful all around (unless the tests always
pass) :)
> 3) You want to have some sort of automatic dependency resolution for jenkins.
That'd be nice ... it's also for folks running 'make unittests' from a
git checkout who don't have all the runtime/test dependencies installed.
Rather, the later is how I've used it in the past.
> I think we can easily handle 1) by re-writing the small portions of this that
> are in bash in python, and make it smarter. For instance, take what
> you've done for moving the tests in their own directory, and just have the
> python code iterate over those tests.
In the current approach, I left the .tdl test files where they were, but
used a .cfg to define expected outcomes. Would it be easier to remove
the .cfg and just have tests exist in expected_fail/ expected_pass/ ?
> For 2), after seeing the output of these patches, it seems to me that it should
> be easy enough to have the existing python code generate JUnit XML as output.
> Maybe we have the code take a flag (--junitxml=output/path), and then we can
> build it up ourselves (it is, after all, simple XML).
My preference would be to avoid custom-built XML generation (that just
seems prone to rot) and let the python framework (py.test) do that for
us. I figured that would be less to maintain in the long-run. Let me
know what you think.
> The third problem you have is a little more difficult. I want to keep running
> the tests as simple as possible, but you want automatic dependency stuff.
> I wonder if we can add a new "virtualenv" target to the Makefile, which
only
> jenkins will call. Then it could call out to the tests as re-written above.
Oh, I see ... so you'd be required to install the deps manually when
running 'make unittests'. However, if testing with jenkins, you could
do 'make virtualenv unittests' which would alleviate manual dependency
installation? That's a neat idea, I could set that up.
Yeah, if you could look at doing that, that would at least address this part
of the problem.
> What do you think about that proposal?
Thanks for your feedback! Whatever resolves the 3 problems you
highlighted without introducing too much additional maintenance. I can
get busy re-working based on your feedback.
So with my proposed patch we've addressed 1) by converting everything to
python. If you can work on 3) to add a virtualenv Makefile target, that would
be great. That leaves us with 2); I'm totally with you on avoiding custom code
where we can. I think it will require a bit more research, but if we can find
something that can generate the junit XML (and only the junit XML), that might
be workable.
--
Chris Lalancette