On 07/20/2017 11:59 AM, Stef Walter wrote:
On 20.07.2017 16:13, Merlin Mathesius wrote:

On 07/20/2017 12:17 AM, Stef Walter wrote:
On 19.07.2017 22:56, Merlin Mathesius wrote:

          
...
To further make use of the $TEST_ARTIFACTS environment variable, I have
raised https://pagure.io/standard-test-roles/pull-request/25 so the
current roles will take their default artifacts directory from
$TEST_ARTIFACTS--using "./artifacts" if not specified. Thus, specifying
"-e artifacts=..." on the ansible-playbook command line is no longer
necessary--but can still be provided as an override if desired.

Would you please let me know what you think about PR#25, Stef?
I think it is useful for debugging, testing and making this easier to
work with.

However the canonical way for a testing system to invoke a playbook is
to set both the relevent environment variables *and* the ansible
variables. The spec states this clearly:

https://fedoraproject.org/wiki/Changes/InvokingTests
Yes, it is stated in the spec. However, it is not show in the examples
at https://fedoraproject.org/wiki/CI/Tests.

However, it is awkward and redundant to be required to specify the same
parameter in multiple ways. While that shouldn't be a problem for an
automated testing system, it does makes it more difficult and confusing
for test developers. It's probably worth trying to eliminate the test
subjects parameter redundancy requirement, too.
Indeed. Can we just use environment variables? That would be nice. What
would that look like. For example here in this simple example:

https://fedoraproject.org/wiki/CI/Tests#Writing_a_new_test


Yes, that would be nice. I can take a closer look at how that might play out.

I'd be happy to take a shot at implementing the above improvements.
Then, I think we have something ready to tag and package as v2.0.
Wonderful.
Do you feel the current master branch of standard-test-roles is in good
shape to release as v2.0?
I think so. Although I believe we're currently backwards compatible, it
may be worth tagging v1.1 ... but I'm probably straying into
bikeshedding territory at this point :S

I consider the inclusion of the inventory scripts worthy of bumping to v2.0. :)

Yes, everything should be backwards compatible when using the old calling conventions. However, I discovered that the standard-test-rhts role is not currently compatible with a docker image created using the dynamic inventory script. This is due to the restraint client requiring sshd and restraintd to be spun up--neither of which are currently done. But I can look at that later without holding up the package release.

However, there is one more thing I'd like to see included in the release: the ability to diagnose problems inside the containers/VMs spun up by the inventory scripts. The inventory scripts were doing such a good job of cleaning up after themselves that there was no opportunity to shell in and poke around. (I needed this to investigate the cause of the compatibility problem between the rhts role and docker inventory container I just mentioned.) So, I added hooks to both the qcow2 and docker inventory scripts to pause before cleaning up if TEST_DIAGNOSE_INVENTORY=1 is set in the environment.
https://pagure.io/standard-test-roles/pull-request/26

I would also be interested in seeing the example "test.yml" you refer to
on the wiki page. I couldn't find it checked in anywhere.
See the gzip git repo. But you can typically substitute test_local.yml.
Got it. I thought maybe you had come up with a new and interesting
standard format for a master test playbook. :)
Heh, no. We really should be reaching the point where less changes, and
not more.
Agreed. I just wanted to see if you made any changes.

As discussed during brainstorming yesterday, we probably will want to
make some updates to the "template" tests.yml so only tests applicable
to the current environment are attempted to be run...
Exactly, the Ansible tags stuff will be another change in the spec that
lands. So I guess another round of change to the spec is expected.

That should give us the leeway to make the above environment vs. ansible
variable changes. But lets not wait on releasing a new standard-test-roles.

Indeed.

Cheers,

Stef


As a matter of fact all the old ways of launching an image such as
test_cloud.yml still work. We should decide whether we want to deprecate
them and when.
Yup. I agree.

Stef
Merlin