----- Original Message -----
From: "Shawn Wells" <shawn(a)redhat.com>
To: "scap-security-guide" <scap-security-guide(a)lists.fedorahosted.org>
Sent: Wednesday, September 3, 2014 6:31:00 AM
Subject: Test run of Jenkins (CI tool)
Also in the vein of automation, went ahead and installed an instance of
Jenkins and connected it to SSG:
Not meant to be permanent right now, but wanted to get something stood
up for us to play with. Martin stood one up for the OpenSCAP
interpreter, which was the source of thinking to stand up a sandbox for
SSG too. There really isn't much to see in the UI, but one thing it does
is integrate into GitHub. Whenever someone issues a pull request,
Jenkins will automagically detect that, apply the patch(s), and run
I wanted to create a testing Jenkins instance for all things OpenSCAP.
There are fixed costs in maintaining the master and all the slaves.
You have to take care of the machines, keep them updated, etc... You
have to keep updating Jenkins itself and be on top of plugin updates.
In my opinion it makes no sense to have 2 Jenkins servers for projects
so closely related. I would prefer to pool resources.
Unfortunately we can't just host our Jenkins on OpenShift because we need
multiple slaves for RHEL 5, 6, 7 and Fedora. And the storage space available
is not enough for our use-case.
Check out https://github.com/OpenSCAP/scap-security-guide/pull/45
example. Or for the lazy :-), see below:
The integration is really cool. I planned to make the Jenkins instance
public after a testing period and do exactly this.
Does anyone have experience with Jenkins, thoughts on how we could
begin building out unit tests, or really any thoughts on usefulness
beyond "make validate" on pull requests?
I was experimenting with docker for this purpose. The idea was to start
with EL6 image, put ssg and a test script inside. Then you can call
docker run and get the results. The docker instance is deleted after
each run so there is no risk of side effects.
The idea is to do the following:
1) pull ssg HEAD
2) make content
3) generate a profile with just one Rule selected - the tested rule
(a simple python script can easily do this)
4) evaluate that profile, compare the results with expected results on a
freshly installed RHEL6
5) anti-remediate - break the configuration (if applicable)
6) evaluate again, check that the result is "fail" (if applicable)
7) remediate - fix the configuration (if applicable)
8) evaluate for the third time, check that the rule passes
*) as an added bonus we could run whatever we are testing, let it
parse the config files and do another sanity check (sshd for example)
This should provide repeatable tests for all the rules.
Unfortunately we can't do this on OpenShift (AFAIK). We need real
slaves with docker-io installed on them. Another problem is that we can
not test every single rule with this approach. "process test" comes to
mind. Or even systemd tests. However we should be able to test most of