On 24.05.2017 13:50, Tim Flink wrote:
As we're working more with the test invocation spec [1] and the
ansible implementation [2], there are a few things which aren't 100%
clear to me and I'm hoping to get some clarification on.
Why is "ansible_connection=local" a requirement for the testing system?
The initial invocation of Ansible is meant to act locally. The ansible
scripts are meant to stage their own containers, or virtual machines
(eg: with qemu, or libvirt) and then perform further tasks into them.
Why is obtaining the test subject solely the responsibility of the
testing system? This seems like something that could potentially have
different requirements for different usecases. For example, something
like abicheck needs to grab previous builds in addition to the latest
build of a given package.
It seems like those previous builds are not the test subject. They are
part of the fixtures of the test or test suite. In CI terms, the test
subject is the thing that has changed. The fixtures are things it are
tested against. We didn't include the term 'fixtures' in the terminology
section, but it's usage here is the usual usage within testing.
I don't understand what we're trying to accomplish with all
the
test_<subject>.yml. Is the idea that we'd run test_oci.yml for every
package used in building a container against that container?
When a test is written in such a way that it makes sense to test it as
an OCI image, then we expect that test_oci.yml is present. In some cases
this file will be a boilerplate of a few lines.
What is the point in having test_local.yml and test_rpm.yml? I
remember
the idea of in-situ tests being part of the discussion but I thought
that we were pretty limited on what we could do with those since we're
not going to be modifying rpm like debian was able to get changes made
for .deb packages.
test_rpm.yml installs RPMs and then tests against them. test_local.yml
assumes that everything to be tested has already been installed.
Both test_rpm.yml and test_local.yml operate in-situ, but one performs
an additional action.
Again, as with test_oci.yml if test_rpm.yml is commonly the same
boilerplate, we could find a simple way to reduce that. Do you have
ideas? It does seem to be necessary for packages to be able to deviate
from the default behavior in cases. So such a solution should not be a
blanket thing.
Cheers,
Stef