Right. That makes sense. Docker wouldn't be a bad idea. Could you use a single image and have a bunch of dockerfiles or run 'docker run' commands for test cases?


On Wed, Sep 3, 2014 at 7:32 AM, Martin Preisler <mpreisle@redhat.com> wrote:
----- Original Message -----
> From: "Gabe Alford" <redhatrises@gmail.com>
> To: "SCAP Security Guide" <scap-security-guide@lists.fedorahosted.org>
> Sent: Wednesday, September 3, 2014 3:13:44 PM
> Subject: Re: Test run of Jenkins (CI tool)
>
> It would be a lot of work, but we could write automated unit testing
> scripts using python unittest, python nose, or something else. Then, run
> those test with a command like 'run-tests'

The test harness is a non issue. There are tons of frameworks that would
work well.

The issue in my opinion is how we run the tests themselves. That's the part
that is hard to solve and we should focus on that first IMO. Unfortunately,
it's hard to evaluation or remediate the system itself without potentially
breaking it. This is simply unacceptable on the developer's workstation.
However we can break docker instances all we want because we just delete
them afterwards :-)

In the future we may have a test runner in upstream repo that would pull
docker images and run all the test cases. This way contributors can run
the tests themselves without having to depend on Jenkins. Because of the
size of the images we shall not keep them in the ssg repo but we could
keep the surrounding test harness, test cases and scripts in there.