On 19.07.2017 22:56, Merlin Mathesius wrote:
On 07/19/2017 07:32 AM, Stef Walter wrote:
> On 18.07.2017 16:38, Merlin Mathesius wrote:
>> On 07/17/2017 05:34 AM, Stef Walter wrote:
>>> I've added documentation about launching test subjects using Ansible
>>> Dynamic inventory scripts. Test subjects are the thing to be tested. For
>>> example a QCow2, container image, or often a set of RPMs. These get
>>> launched into a form that Ansible (and thus the test playbooks) can run
>>> against them:
>>>
>>> The tutorial style documentation is here:
>>>
>>>
https://fedoraproject.org/wiki/CI/Tests
>>>
>>> More documentation will be landing here about writing tests, preparing a
>>> dist-git repo for tests, and so on.
>>>
>>> The information about the spec is here:
>>>
>>>
https://fedoraproject.org/wiki/Changes/InvokingTests
>>>
>>> Note that older version of the spec used to have a test_local.yml file,
>>> and test subject specific playbooks. These test subject specific
>>> playbooks are no longer necessary with the inventory.
>>>
>>> In addition test_local.yml may be used in place of tests.yml in the
>>> documentation. Once more people have looked over this, I'll open some
>>> pull requests to rename test_local.yml to tests.yml.
>>>
>>> For the documentation to work, we do need to package the inventory
>>> scripts that are already present in standard-test-roles. In addition,
>>> I've opened a pull request to standard-test-roles for adding support for
>>> launching docker containers as dynamic inventory:
>>>
>>>
https://pagure.io/standard-test-roles/pull-request/22
>> Hi, Stef,
>>
>> Thank you for providing the documentation and examples at
>>
https://fedoraproject.org/wiki/CI/Tests.
>>
>> PR#22 and several other fixe/improvement PRs have been merged.
>>
>> I was able to experiment with the inventory scripts a bit, and there's a
>> few improvements I'd like to see:
>>
>> 1. The docker script is not compatible with Fedora 25 due to a
>> particular improvement in docker (combining "--rm" and
"--detach" for
>> "docker run ...") that's available on F26 that the script is
taking
>> advantage of. F25 could be accommodated (I think) by eliminating the
>> "--rm" from "docker run ..." and doing a "docker rm -f
..." during cleanup.
> Good catch thanks. Hope this fixes that:
>
>
https://pagure.io/standard-test-roles/pull-request/23
Stef,
Yes it does. It has been merged. Thanks.
>> 2. The qcow2 script creates a log file in the current directory. That
>> should really be in the artifacts directory--or other directory of
>> choice. What would you think of having the script honor a log directory
>> environment variable? (eg., "LOGDIR")
> Ah yes. It's such a shame that Ansible doesn't pass it's -e variables to
> inventory scripts. For this reason the spec mandates also passing the
> artifacts directory as an environment variable:
>
> $TEST_ARTIFACTS
>
>
https://fedoraproject.org/wiki/Changes/InvokingTests#Invocation
>
> I'll start to make use of it to accomplish this.
>
>> 3. The docker script should probably create a log file too. ("docker
>> logs ... > logfile")
> Will do along with the above.
Thanks for
https://pagure.io/standard-test-roles/pull-request/24 which
accomplishes placing the inventory script logfiles in the
$TEST_ARTIFACTS directory. It has also been merged.
To further make use of the $TEST_ARTIFACTS environment variable, I have
raised
https://pagure.io/standard-test-roles/pull-request/25 so the
current roles will take their default artifacts directory from
$TEST_ARTIFACTS--using "./artifacts" if not specified. Thus, specifying
"-e artifacts=..." on the ansible-playbook command line is no longer
necessary--but can still be provided as an override if desired.
Would you please let me know what you think about PR#25, Stef?
I think it is useful for debugging, testing and making this easier to
work with.
However the canonical way for a testing system to invoke a playbook is
to set both the relevent environment variables *and* the ansible
variables. The spec states this clearly:
https://fedoraproject.org/wiki/Changes/InvokingTests
>> I'd be happy to take a shot at implementing the above
improvements.
>> Then, I think we have something ready to tag and package as v2.0.
> Wonderful.
>
>> I would also be interested in seeing the example "test.yml" you refer
to
>> on the wiki page. I couldn't find it checked in anywhere.
> See the gzip git repo. But you can typically substitute test_local.yml.
Got it. I thought maybe you had come up with a new and interesting
standard format for a master test playbook. :)
Heh, no. We really should be reaching the point where less changes, and
not more.
As a matter of fact all the old ways of launching an image such as
test_cloud.yml still work. We should decide whether we want to deprecate
them and when.
Stef
>> Also, do you have any thoughts on how to have a single
test.yml playbook
>> cleanly pick and choose amongst which tests to run in which
>> environments? Some tests simply won't run in a container and/or cloud
>> image, and so should be skipped when they can't run properly. I also
>> imagine there could be tests that should be run ONLY in a container
>> and/or cloud image. I attempted to implement something like this with
>> the "selector" role
>>
(
https://pagure.io/standard-test-roles/blob/master/f/roles/standard-test-s...),
>> but it feels rather clumsy to use (eg.,
>>
https://upstreamfirst.fedorainfracloud.org/sed/blob/master/f/test_local.yml).
> I believe we're brainstorming this today. We could factor this aspect in.
>
> Stef
>
>
>
> _______________________________________________
> CI mailing list -- ci(a)lists.fedoraproject.org
> To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org