Hi, 

> Anyone have any thoughts?

Let me try to summarize mine.

# The org.centos.prod.ci.pipeline.allpackages-build.package.ignored testcase

The problem I see here is that this testcase doesn't fit in any
of designs for Greenwave, CI Messages and/or ResultsDB.  Simply
because "ignored" supposed to be the outcome of the test, not the
testcase itself.

In CI Messages schema we define "not applicable" as a possible value
for the status field [1]. This message supposed to mean

    "We were trying to run a certain testcase named "<testcase_name>,
    but there was a certain logic in the test framework or pipeline
    which led us to skipping it".

This is the result of our testcase, which we want share with all
consumers of the _testcase_, to let them do something about it.  While
"org.centos.prod.ci.pipeline.allpackages-build.package.ignored" is
neither testcase nor a result. It is just an informational message on
the Message Bus, and probably a redundant one, thus it shouldn't be in
the ResultsDB at all.

So for me it is not really a question, rather a work item: we need to
find out how to fix it, and fix it. Most likely by aligning the
pipeline messages to CI Messages format.

# How to use non_applicable result in gating policy

Now given by the testcase result with "not_applicable" as an outcome,
can we use it in Greenwave?

Afaik currently PassingTestCase rule [2] only checks if testcase has
PASS status or waived.  I think it needs to be adjusted and it needs
to be configurable.

We need another rule, something like

    ParametrizedTestCaseRule(testcase=TEST_CASE_NAME, outcomes=[PASS, NOT_APPLICABLE, WAIVED])

or maybe 

    PassingOrSkippedTestCaseRule(testcase=TEST_CASE_NAME)

Or both.

This way we can use all outcomes in our decision making process.

# But should there be the "not applicable" outcome at all and should we treat it like PASS? 

I think it should be possible to report results like that. There are
certain use cases and extensive test suites where you can skip test
results partially or temporarily, or based on certain parameters. And
if certain CI system provides this kind of flexibility, it should be
able to communicate it.

But I think there shouldn't be a "treat non_applicable like PASS"
approach by default. We need to clearly identify those testcases where
it makes sense and use it only for this limited subset.

And the smaller the list - the better.

# How to deal with exceptions for a global Greenwave policy

I think that the better way to treat exceptions is to make them
explicit. We shouldn't try to identify exceptions based on their test
results, but rather have a list of them predefined and stored
somewhere near the global policy itself.

To justify: CI systems, test suites and test runs can be
misconfigured. We can in theory disable a certain feature on a lower
level by mistake and we need an independent source of truth to verify
our results against it.

And it goes again back to Greenwave. It currently provides Remote Rule
which allows reading additional testcases from a dist-git repo on a
per project basis.

We would need a NotReallyRemoteRule which would allow overrides to
global policy on a per project basis, using set of rules configured
additionally on a Greenwave server itself or another centralized
storage.

# ExecDB vs ResultsDB

This topic probably needs the thread on its own.

If I understood correctly, ExecDB is the database of test jobs, while
ResultsDB is the database of test cases.

And we were recently discussing if it is possible to extend ResultsDB
into a Results State Machine:

Imagine that we have several CI systems capable of running the same
test case scenario.  If we extend ResultsDB status field with PENDING
and IN PROGRESS values we can use it as a task tracker.

1) When new artifact get's created, we check with Greenwave which test
cases we need to gate it.

2) Then we create "(artifact_id, testcase_id, pending)" entry in
ResultsDB for each of them.

3) CI system periodically checks if there is a "(artifact_id,
testcase_id, pending)" entry in ResultsDB, which is not overriden by
"(artifact_id, testcase_id, in progress)" result.

4) If it finds one, it triggers the testcase and sends the
"(artifact_id, testcase_id, in progress)" message to ResultsDB.

This idea does go into direction of ExecDB execution tracker, but
using (artifact,testcase) pair as a primary key.

Do you think ExecDB is a better place for it?



-- 
Aleksandra Fedorova
bookwar