On 20.07.2017 16:13, Merlin Mathesius wrote:On 07/20/2017 12:17 AM, Stef Walter wrote:On 19.07.2017 22:56, Merlin Mathesius wrote:...To further make use of the $TEST_ARTIFACTS environment variable, I have raised https://pagure.io/standard-test-roles/pull-request/25 so the current roles will take their default artifacts directory from $TEST_ARTIFACTS--using "./artifacts" if not specified. Thus, specifying "-e artifacts=..." on the ansible-playbook command line is no longer necessary--but can still be provided as an override if desired. Would you please let me know what you think about PR#25, Stef?I think it is useful for debugging, testing and making this easier to work with. However the canonical way for a testing system to invoke a playbook is to set both the relevent environment variables *and* the ansible variables. The spec states this clearly: https://fedoraproject.org/wiki/Changes/InvokingTestsYes, it is stated in the spec. However, it is not show in the examples at https://fedoraproject.org/wiki/CI/Tests. However, it is awkward and redundant to be required to specify the same parameter in multiple ways. While that shouldn't be a problem for an automated testing system, it does makes it more difficult and confusing for test developers. It's probably worth trying to eliminate the test subjects parameter redundancy requirement, too.Indeed. Can we just use environment variables? That would be nice. What would that look like. For example here in this simple example: https://fedoraproject.org/wiki/CI/Tests#Writing_a_new_test
I'd be happy to take a shot at implementing the above improvements. Then, I think we have something ready to tag and package as v2.0.Wonderful.Do you feel the current master branch of standard-test-roles is in good shape to release as v2.0?I think so. Although I believe we're currently backwards compatible, it may be worth tagging v1.1 ... but I'm probably straying into bikeshedding territory at this point :S
https://pagure.io/standard-test-roles/pull-request/26
I would also be interested in seeing the example "test.yml" you refer to on the wiki page. I couldn't find it checked in anywhere.See the gzip git repo. But you can typically substitute test_local.yml.Got it. I thought maybe you had come up with a new and interesting standard format for a master test playbook. :)Heh, no. We really should be reaching the point where less changes, and not more.Agreed. I just wanted to see if you made any changes. As discussed during brainstorming yesterday, we probably will want to make some updates to the "template" tests.yml so only tests applicable to the current environment are attempted to be run...Exactly, the Ansible tags stuff will be another change in the spec that lands. So I guess another round of change to the spec is expected. That should give us the leeway to make the above environment vs. ansible variable changes. But lets not wait on releasing a new standard-test-roles.
Cheers, StefAs a matter of fact all the old ways of launching an image such as test_cloud.yml still work. We should decide whether we want to deprecate them and when.Yup. I agree.StefMerlin