-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 22.10.2014 13:43, Honza Horak wrote:
Fedora lacks integration testing (unit testing done during build
is not enough). Taskotron will be able to fill some gaps in the
future, so maintainers will be able to set-up various tasks after
their component is built. But even before this works we can
benefit from having the tests already available (and run them
manually if needed).
Hereby, I'd like to get ideas and figure out answers for how and
where to keep the tests. A similar discussion already took place
before, which I'd like to continue in:
https://lists.fedoraproject.org/pipermail/devel/2014-January/193498.html
And some short discussion already took place here as well:
Some high level requirements: * tests will be written by
maintainers or broader community, not a dedicated team * tests
will be easy to run on anybody's computer (but might be
potentially destructive; some secure environment will not be part
of tests) * tests will be run automatically after related
components get built (probably by Taskotron)
Where to keep tests? a/ in current dist-git for related components
(problem with sharing parts of code, problem where to keep tests
related for more components) b/ in separate git with similar
functionality as dist-git (needs new infrastructure, components
are not directly connected with tests, won't make mess in current
dist-git) c/ in current dist-git but as ordinary components (no
new infrastructure needed but components are not directly
connected with tests)
How to deliver tests? a/ just use them directly from git (we need
to keep some metadata for dependencies anyway) b/ package them as
RPMs (we can keep metadata there; e.g. Taskotron will run only
tests that have "Provides: ci-tests(mariadb)" after mariadb is
built; we also might automate packaging tests to RPMs)
Structure for tests? a/ similar to what components use (branches
for Fedora versions) b/ only one branch Test maintainers should be
allowed to behave the same as package maintainers do -- one likes
keeping branches the same and uses "%if %fedora" macros, someone
else likes specs clean and rather maintain more different
branches) -- we won't find one structure that would fit all, so
allowing both ways seems better.
Which framework to use? People have no time to learn new things,
so we should let them to write the tests in any language and just
define some conventions how to run them.
TAP (Test Anything Protocol) FTW. It really makes sense when you're
trying to combine tests from multiple different languages, testing
frameworks, etc.
Stef
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iEYEARECAAYFAlRJWOMACgkQe/sRCNknZa/ltQCfcTBBPOIl3fISqjm0j3YUw+TU
eSAAoIMo+NSOg/iWf27VQuq0J2rTebT/
=L9uQ
-----END PGP SIGNATURE-----