Hi,
The standard test interface defines `tests/tests.yml` as the sole
playbook to be run to test a repository. This makes it impossible to run
multiple independent tests on fresh instances of the same test subjects.
I noticed this while converting the tests for linux-system-roles to this
standard. We're running different ansible playbooks against virtual
machine images as test subjects and would like each playbook to start
from a fresh machine. We've already had a bug where one of the tests
didn't clean up correctly and influenced the result of the next test.
Thus, I suggest changing the "Invocation" section to say that the
testing system must independently run all playbooks `tests/tests*.yml`
in the dist-git repo. This is compatible with existing repositories.
If we decide to go this way, we'll have to decide how to handle
artifacts in this case. I see two reasonable choices:
1. Keep one artifacts directory and let the authors of the playbooks
take care that they don't overwrite each other's result. We couldn't run
the tests in parallel in this case, because tests might want to
concatenate a main`test.log`.
2. Have a separate artifacts directory for each test run.
What do you think?
Lars
Show replies by date