On Thu, 31 Oct 2013 07:17:03 -0400 (EDT)
Josef Skladanka <jskladan(a)redhat.com> wrote:
Bah, just realized this only went to Josef the first time I sent it. He
gets two copies!
Tim (and of course the rest of the gang ;)),
During our chat with Tim, we agreed that we'd really like to use some
standardized 'output format' for the tests in Taskbot, to be a bit
more programming-language/results-store-implementation agnostic. We
knew about two options - TAP
<
http://testanything.org/wiki/index.php/Main_Page> and Subunit
<
https://pypi.python.org/pypi/python-subunit>.
= Subunit =
At least in Python is quite tightly coupled with unittest, both
ideologically and practically. I was unable to find a way to just
produce a Subunit stream without the need of actually running a
testsuite.
The format is (basically) just plain PASS/FAIL/INFO/..., and there is
possibility to add some 'details'results. It should also be possible
to add an attachment to the end of the stream, but no result-specific
data can be added (IMO).
Also Subunit is now in the process of transition to new
implementation, which now should fix some issues with concurrency,
add more result-states, etc., but will be binary, so human
readability would quite suffer.
I do not really feel that this is a good match for our needs (at
least without any significant hacking)
Yeah, this matches what I remember finding when I looked into this a
couple of months ago. It looks like a better protocol overall but it's
going to be quite a bit more upfront work to support than TAP. I know
that openstack's test suite uses subunit for nova and neutron [1]
[1]
https://wiki.openstack.org/wiki/Testr
Subunit has been recently added to the fedora repos but I suppose
that's of little utility for us if we end up having to re-implement
parts to get away from UnitTest.
= TAP =
TAP is not unittest-specific, and is human-readable plaintext format.
It also has just PASS/FAIL logic, but there is a possibility to add
YAML 'metadata' to any result (since TAP v. 13).
The real issue with TAP is Python support.
There is a TAP-consumer library created as an example for PyParsing
<
http://pyparsing.wikispaces.com/file/detail/TAP.py>, but it does not
support the v13 protocol, and is quite useless as such. TAP-producer
library for Python <
https://github.com/rjbs/pytap> is also using the
old version (i.e. no YAML extensions), and seems to be dead (2 years
since last update). It is also quite badly written.
Another option for TAP emission is bayeux [2] which was created for a
better, less perl-ish interface to tap [3]. It might be worth asking
mcepl if he has any thoughts on TAP vs. subunit vs. something else.
[2]
http://luther.ceplovi.cz/git/bayeux.git/
[
3]http://lists.idyll.org/pipermail/testing-in-python/2012-March/004881.html
= Result =
Although neither choice is ideal, I feel that TAP would be the better
choice, even though it would mean implementing our own
producer/parser. Also the TAP is really simplistic format, so
creating a TAP output should be quite easy even in any programming
language.
I think that TAP is certainly appears to be the simpler solution for
now and I suspect it's a bit more widely used than subunit. As long as
the amount of work required to have a TAP emitter and parser isn't
crazy, I agree that TAP is a better choice.
It is possible that I somehow utterly misunderstood the Subunit
concept, so it might be useful to contact some QAs currently using it
(I thing Tim mentioned something about OpenStack?), or contact the
developer directly.
I suspect that you found this when searching but there's a direct
comparison of the two by someone with more of a TAP background:
http://www.kinoshita.eti.br/2011/06/04/a-comparison-of-tap-test-anything-...
Thanks for looking into this,
Tim