Tim (and of course the rest of the gang ;)),
During our chat with Tim, we agreed that we'd really like to use some standardized
'output format' for the tests in Taskbot, to be a bit more
programming-language/results-store-implementation agnostic.
We knew about two options - TAP <
http://testanything.org/wiki/index.php/Main_Page>
and Subunit <
https://pypi.python.org/pypi/python-subunit>.
= Subunit =
At least in Python is quite tightly coupled with unittest, both ideologically and
practically. I was unable to find a way to just produce a Subunit stream without the need
of actually running a testsuite.
The format is (basically) just plain PASS/FAIL/INFO/..., and there is possibility to add
some 'details'results. It should also be possible to add an attachment to the end
of the stream, but no result-specific data can be added (IMO).
Also Subunit is now in the process of transition to new implementation, which now should
fix some issues with concurrency, add more result-states, etc., but will be binary, so
human readability would quite suffer.
I do not really feel that this is a good match for our needs (at least without any
significant hacking)
= TAP =
TAP is not unittest-specific, and is human-readable plaintext format.
It also has just PASS/FAIL logic, but there is a possibility to add YAML
'metadata' to any result (since TAP v. 13).
The real issue with TAP is Python support.
There is a TAP-consumer library created as an example for PyParsing
<
http://pyparsing.wikispaces.com/file/detail/TAP.py>, but it does not support the
v13 protocol, and is quite useless as such.
TAP-producer library for Python <
https://github.com/rjbs/pytap> is also using the
old version (i.e. no YAML extensions), and seems to be dead (2 years since last update).
It is also quite badly written.
= Result =
Although neither choice is ideal, I feel that TAP would be the better choice, even though
it would mean implementing our own producer/parser.
Also the TAP is really simplistic format, so creating a TAP output should be quite easy
even in any programming language.
It is possible that I somehow utterly misunderstood the Subunit concept, so it might be
useful to contact some QAs currently using it (I thing Tim mentioned something about
OpenStack?), or contact the developer directly.
J.