On 18.07.2017 16:38, Merlin Mathesius wrote:
On 07/17/2017 05:34 AM, Stef Walter wrote:
> I've added documentation about launching test subjects using Ansible
> Dynamic inventory scripts. Test subjects are the thing to be tested. For
> example a QCow2, container image, or often a set of RPMs. These get
> launched into a form that Ansible (and thus the test playbooks) can run
> against them:
>
> The tutorial style documentation is here:
>
>
https://fedoraproject.org/wiki/CI/Tests
>
> More documentation will be landing here about writing tests, preparing a
> dist-git repo for tests, and so on.
>
> The information about the spec is here:
>
>
https://fedoraproject.org/wiki/Changes/InvokingTests
>
> Note that older version of the spec used to have a test_local.yml file,
> and test subject specific playbooks. These test subject specific
> playbooks are no longer necessary with the inventory.
>
> In addition test_local.yml may be used in place of tests.yml in the
> documentation. Once more people have looked over this, I'll open some
> pull requests to rename test_local.yml to tests.yml.
>
> For the documentation to work, we do need to package the inventory
> scripts that are already present in standard-test-roles. In addition,
> I've opened a pull request to standard-test-roles for adding support for
> launching docker containers as dynamic inventory:
>
>
https://pagure.io/standard-test-roles/pull-request/22
Hi, Stef,
Thank you for providing the documentation and examples at
https://fedoraproject.org/wiki/CI/Tests.
PR#22 and several other fixe/improvement PRs have been merged.
I was able to experiment with the inventory scripts a bit, and there's a
few improvements I'd like to see:
1. The docker script is not compatible with Fedora 25 due to a
particular improvement in docker (combining "--rm" and "--detach"
for
"docker run ...") that's available on F26 that the script is taking
advantage of. F25 could be accommodated (I think) by eliminating the
"--rm" from "docker run ..." and doing a "docker rm -f ..."
during cleanup.
Good catch thanks. Hope this fixes that:
https://pagure.io/standard-test-roles/pull-request/23
2. The qcow2 script creates a log file in the current directory.
That
should really be in the artifacts directory--or other directory of
choice. What would you think of having the script honor a log directory
environment variable? (eg., "LOGDIR")
Ah yes. It's such a shame that Ansible doesn't pass it's -e variables to
inventory scripts. For this reason the spec mandates also passing the
artifacts directory as an environment variable:
$TEST_ARTIFACTS
https://fedoraproject.org/wiki/Changes/InvokingTests#Invocation
I'll start to make use of it to accomplish this.
3. The docker script should probably create a log file too.
("docker
logs ... > logfile")
Will do along with the above.
I'd be happy to take a shot at implementing the above
improvements.
Then, I think we have something ready to tag and package as v2.0.
Wonderful.
I would also be interested in seeing the example "test.yml"
you refer to
on the wiki page. I couldn't find it checked in anywhere.
See the gzip git repo. But you can typically substitute test_local.yml.
Also, do you have any thoughts on how to have a single test.yml
playbook
cleanly pick and choose amongst which tests to run in which
environments? Some tests simply won't run in a container and/or cloud
image, and so should be skipped when they can't run properly. I also
imagine there could be tests that should be run ONLY in a container
and/or cloud image. I attempted to implement something like this with
the "selector" role
(
https://pagure.io/standard-test-roles/blob/master/f/roles/standard-test-s...),
but it feels rather clumsy to use (eg.,
https://upstreamfirst.fedorainfracloud.org/sed/blob/master/f/test_local.yml).
I believe we're brainstorming this today. We could factor this aspect in.
Stef