<div dir="ltr">It would be a lot of work, but we could write automated unit testing scripts using python unittest, python nose, or something else. Then, run those test with a command like 'run-tests'<br></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Sep 3, 2014 at 6:06 AM, Martin Preisler <span dir="ltr"><<a href="mailto:mpreisle@redhat.com" target="_blank">mpreisle@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="">----- Original Message -----<br>
> From: "Shawn Wells" <<a href="mailto:shawn@redhat.com">shawn@redhat.com</a>><br>
> To: "scap-security-guide" <<a href="mailto:scap-security-guide@lists.fedorahosted.org">scap-security-guide@lists.fedorahosted.org</a>><br>
> Sent: Wednesday, September 3, 2014 6:31:00 AM<br>
> Subject: Test run of Jenkins (CI tool)<br>
><br>
> Also in the vein of automation, went ahead and installed an instance of<br>
> Jenkins and connected it to SSG:<br>
><br>
> <a href="http://jenkins.ssgproject.org:8080/" target="_blank">http://jenkins.ssgproject.org:8080/</a><br>
><br>
> Not meant to be permanent right now, but wanted to get something stood<br>
> up for us to play with. Martin stood one up for the OpenSCAP<br>
> interpreter, which was the source of thinking to stand up a sandbox for<br>
> SSG too. There really isn't much to see in the UI, but one thing it does<br>
> is integrate into GitHub. Whenever someone issues a pull request,<br>
> Jenkins will automagically detect that, apply the patch(s), and run<br>
> "make validate."<br>
<br>
</div>I wanted to create a testing Jenkins instance for all things OpenSCAP.<br>
There are fixed costs in maintaining the master and all the slaves.<br>
You have to take care of the machines, keep them updated, etc... You<br>
have to keep updating Jenkins itself and be on top of plugin updates.<br>
<br>
In my opinion it makes no sense to have 2 Jenkins servers for projects<br>
so closely related. I would prefer to pool resources.<br>
<br>
Unfortunately we can't just host our Jenkins on OpenShift because we need<br>
multiple slaves for RHEL 5, 6, 7 and Fedora. And the storage space available<br>
is not enough for our use-case.<br>
<div class=""><br>
> Check out <a href="https://github.com/OpenSCAP/scap-security-guide/pull/45" target="_blank">https://github.com/OpenSCAP/scap-security-guide/pull/45</a> for an<br>
> example. Or for the lazy :-), see below:<br>
<br>
</div>The integration is really cool. I planned to make the Jenkins instance<br>
public after a testing period and do exactly this.<br>
<div class=""><br>
> Does anyone have experience with Jenkins, thoughts on how we could<br>
> begin building out unit tests, or really any thoughts on usefulness<br>
> beyond "make validate" on pull requests?<br>
<br>
</div>I was experimenting with docker for this purpose. The idea was to start<br>
with EL6 image, put ssg and a test script inside. Then you can call<br>
docker run and get the results. The docker instance is deleted after<br>
each run so there is no risk of side effects.<br>
<br>
The idea is to do the following:<br>
1) pull ssg HEAD<br>
<br>
2) make content<br>
<br>
3) generate a profile with just one Rule selected - the tested rule<br>
(a simple python script can easily do this)<br>
<br>
4) evaluate that profile, compare the results with expected results on a<br>
freshly installed RHEL6<br>
<br>
5) anti-remediate - break the configuration (if applicable)<br>
<br>
6) evaluate again, check that the result is "fail" (if applicable)<br>
<br>
7) remediate - fix the configuration (if applicable)<br>
<br>
8) evaluate for the third time, check that the rule passes<br>
<br>
*) as an added bonus we could run whatever we are testing, let it<br>
parse the config files and do another sanity check (sshd for example)<br>
<br>
This should provide repeatable tests for all the rules.<br>
<br>
Unfortunately we can't do this on OpenShift (AFAIK). We need real<br>
slaves with docker-io installed on them. Another problem is that we can<br>
not test every single rule with this approach. "process test" comes to<br>
mind. Or even systemd tests. However we should be able to test most of<br>
the rules.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Martin Preisler<br>
--<br>
SCAP Security Guide mailing list<br>
<a href="mailto:scap-security-guide@lists.fedorahosted.org">scap-security-guide@lists.fedorahosted.org</a><br>
<a href="https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide" target="_blank">https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide</a><br>
<a href="https://github.com/OpenSCAP/scap-security-guide/" target="_blank">https://github.com/OpenSCAP/scap-security-guide/</a></font></span></blockquote></div><br></div>