----- Original Message -----
From: "Jan Lieskovsky" jlieskov@redhat.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Cc: "open-scap-list" open-scap-list@redhat.com Sent: Wednesday, July 27, 2016 9:34:04 AM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
----- Original Message -----
From: "Gabe Alford" redhatrises@gmail.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Cc: "open-scap-list" open-scap-list@redhat.com Sent: Wednesday, July 27, 2016 3:00:34 PM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
On Tue, Jul 26, 2016 at 3:24 PM, Martin Preisler < mpreisle@redhat.com > wrote:
I found pretty bad inefficiencies in some of our XSLTs.
Check out https://github.com/OpenSCAP/openscap/commit/a65bf27dec4a93e2b87cec8cbcd80bec... or https://github.com/OpenSCAP/openscap/commit/1dd5573b2c964b00af57215cadb7f13b...
Long story short I was able to cut SSG build time on my machine from 2m21.061s to 1m08.771s. In other words my machine now needs roughly half the time to build SSG. I was timing `make -j 4`.
It will take a while for these changes to make it into releases and into distributions but I am writing this email because the savings are significant and perhaps you want to enjoy them sooner. SCAP Security Guide contributors are doing a lot of content builds.
This will only work if you !!! have OPENSCAP 1.2.x !!!
If you want to speed up the build, replace
/usr/share/openscap/xsl/xccdf_1.1_to_1.2.xsl with https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf_1.1_...
/usr/share/openscap/xsl/xccdf-share.xsl with https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf-shar...
and
/usr/share/openscap/xsl/xccdf-guide-impl.xsl with https://github.com/OpenSCAP/openscap/blob/maint-1.2/xsl/xccdf-guide-impl.xsl
Perhaps it would be worth it to deploy this on our Jenkins slaves even before it hits releases? What do you think?
+1
Not like I would want to be the blocker for speedups / SSG Jenkins optimizations, but how would you like to deploy these changes?
Two scenarios come to mind:
- just replace the affected XSLT transformations in recent RHEL openscap builds,
- make a separate openscap RPM (applied on Jenkins slaves outside of RHEL release cycle process)
While the advantage might be more quicker completion of Jenkins SSG build jobs, there are two disadvantages / concerns related with this:
- if modified oscap returns different result than latest oscap available in RHEL - which report should be considered the right one? Which one is valid?
- the concern with maybe even more importance is we wouldn't test on default RHEL / Fedora systems like we do now. The environment would be modified a bit (and it might or might not return same results). So to ensure we "will continue working" on default RHEL / Fedora - should we create another Jenkins CI jobs for default RHEL / Fedora?
Maybe we could use "speed optimized" openscap on 'scap-security-guide-pull-requests' Jenkins CI job, and 'default one' on 'scap-security-guide' Jenkins CI job (the latter one being which actually tells if we don't regress across PRs).
I think that would overcomplicate things. I can relate to your concerns. The only reason why we are even considering this is because the speed-up is so dramatic.
What do you think about me generating RHEL 6 and 7 datastreams and guides and diff-ing them? If the results are the same across the board that would take care of my concerns. Do you agree?