Some thoughts:
> Where to keep tests? a/ in current dist-git for related
components
> (problem with sharing parts of code, problem where to keep tests
> related for more components) b/ in separate git with similar
> functionality as dist-git (needs new infrastructure, components are
> not directly connected with tests, won't make mess in current
> dist-git) c/ in current dist-git but as ordinary components (no new
> infrastructure needed but components are not directly connected
> with tests)
>
> How to deliver tests? a/ just use them directly from git (we need
> to keep some metadata for dependencies anyway) b/ package them as
> RPMs (we can keep metadata there; e.g. Taskotron will run only
> tests that have "Provides: ci-tests(mariadb)" after mariadb is
> built; we also might automate packaging tests to RPMs)
To answer both of these, the plan is to keep taskotron tasks in their own
git repo; currently this is at (0).
To run the tasks, taskotron sets up a disposable task client and then git
clones the task to be run. This solves the issue of delivery and allows
a continuous integration-like solution.
> Structure for tests? a/ similar to what components use (branches
> for Fedora versions) b/ only one branch Test maintainers should be
> allowed to behave the same as package maintainers do -- one likes
> keeping branches the same and uses "%if %fedora" macros, someone
> else likes specs clean and rather maintain more different branches)
> -- we won't find one structure that would fit all, so allowing both
> ways seems better.
This is something we'll need to figure out, but, I suspect git branches will
be involved.
> Which framework to use? People have no time to learn new things,
so
> we should let them to write the tests in any language and just
> define some conventions how to run them.
You'll need to at least define the task in a yaml file, and output will need to
be TAP. The example task is at (1).
TAP (Test Anything Protocol) FTW. It really makes sense when
you're
trying to combine tests from multiple different languages, testing
frameworks, etc.
Stef
Indeed, which is why we settled on it.
John.
(0) https://bitbucket.org/fedoraqa
(1) https://bitbucket.org/fedoraqa/task-rpmlint