From: Ondrej Lichtner <olichtne(a)redhat.com>
The following patch series is meant as a proposal to add a new feature and
change smoke test generation. Please read the ideas behind this first before
actually applying it.
The following patch series changes the way smoke tests are generated. I decided
to implement this change because the automatic generation of all combinations
doesn't take into account the different semantics of the resulting recipes.
Every recipe contains the same comman sequence, however for some combinations
the commands are expected to fail. This happens in recipes that have a vlan
interface on one side and a non-vlan interface on the other side. These were
previously reporting a FAIL which was expected, but seeing it in the results
can be confusing and not clear on first look.
The first patch therefore adds an attribute pass_result to the command tag,
this attribute represents the command result that is seen as a PASS state for
the command, the default value is true.
I'm not 100% sure about this change because the boolean value isn't really
descripte enough. The other approach is to have a way to tell the test modules
directly that they should expect a negative result. This could be a better
solution because it could still report an unexpected failure of the test while
providing a PASS result when the test was negative. This however wouldn't be
usable for exec commands, and it would require cooperation from test module
authors which could be unreliable.
The other two patches reimplement the recipe generation script in python. The
resulting recipes are now fully expanded (no source attributes) and can contain
additional varible references that can be dynamically defined in a sepparate
file. This is now used to set a different pass_result value based on the
configuration combination of the currently generated recipe.
Ondrej Lichtner (3):
add attribute "pass_result" to <command> tags
smoke lib: add variable icmp_result to cmd sequences
lnst/Controller/Machine.py | 3 +
lnst/Controller/RecipeParse.py | 7 ++
recipe-schema.rng | 27 +++++++
recipes/smoke/generate-recipes.py | 143 ++++++++++++++++++++++++++++++++++++++
recipes/smoke/lib/seq-bg.xml | 2 +-
recipes/smoke/lib/seq-ping.xml | 2 +-
recipes/smoke/lib/variables.conf | 14 ++++
7 files changed, 196 insertions(+), 2 deletions(-)
create mode 100755 recipes/smoke/generate-recipes.py
create mode 100644 recipes/smoke/lib/variables.conf