I found pretty bad inefficiencies in some of our XSLTs.
Check out https://github.com/OpenSCAP/openscap/commit/a65bf27dec4a93e2b87cec8cbcd80bec... or https://github.com/OpenSCAP/openscap/commit/1dd5573b2c964b00af57215cadb7f13b...
Long story short I was able to cut SSG build time on my machine from 2m21.061s to 1m08.771s. In other words my machine now needs roughly half the time to build SSG. I was timing `make -j 4`.
It will take a while for these changes to make it into releases and into distributions but I am writing this email because the savings are significant and perhaps you want to enjoy them sooner. SCAP Security Guide contributors are doing a lot of content builds.
This will only work if you !!! have OPENSCAP 1.2.x !!!
If you want to speed up the build, replace
/usr/share/openscap/xsl/xccdf_1.1_to_1.2.xsl with https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf_1.1_...
/usr/share/openscap/xsl/xccdf-share.xsl with https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf-shar...
and
/usr/share/openscap/xsl/xccdf-guide-impl.xsl with https://github.com/OpenSCAP/openscap/blob/maint-1.2/xsl/xccdf-guide-impl.xsl
Perhaps it would be worth it to deploy this on our Jenkins slaves even before it hits releases? What do you think?
On Tue, Jul 26, 2016 at 3:24 PM, Martin Preisler mpreisle@redhat.com wrote:
I found pretty bad inefficiencies in some of our XSLTs.
Check out https://github.com/OpenSCAP/openscap/commit/a65bf27dec4a93e2b87cec8cbcd80bec... or https://github.com/OpenSCAP/openscap/commit/1dd5573b2c964b00af57215cadb7f13b...
Long story short I was able to cut SSG build time on my machine from 2m21.061s to 1m08.771s. In other words my machine now needs roughly half the time to build SSG. I was timing `make -j 4`.
It will take a while for these changes to make it into releases and into distributions but I am writing this email because the savings are significant and perhaps you want to enjoy them sooner. SCAP Security Guide contributors are doing a lot of content builds.
This will only work if you !!! have OPENSCAP 1.2.x !!!
If you want to speed up the build, replace
/usr/share/openscap/xsl/xccdf_1.1_to_1.2.xsl with
https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf_1.1_...
/usr/share/openscap/xsl/xccdf-share.xsl with
https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf-shar...
and
/usr/share/openscap/xsl/xccdf-guide-impl.xsl with
https://github.com/OpenSCAP/openscap/blob/maint-1.2/xsl/xccdf-guide-impl.xsl
Perhaps it would be worth it to deploy this on our Jenkins slaves even before it hits releases? What do you think?
+1
----- Original Message -----
From: "Gabe Alford" redhatrises@gmail.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Cc: "open-scap-list" open-scap-list@redhat.com Sent: Wednesday, July 27, 2016 3:00:34 PM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
On Tue, Jul 26, 2016 at 3:24 PM, Martin Preisler < mpreisle@redhat.com > wrote:
I found pretty bad inefficiencies in some of our XSLTs.
Check out https://github.com/OpenSCAP/openscap/commit/a65bf27dec4a93e2b87cec8cbcd80bec... or https://github.com/OpenSCAP/openscap/commit/1dd5573b2c964b00af57215cadb7f13b...
Long story short I was able to cut SSG build time on my machine from 2m21.061s to 1m08.771s. In other words my machine now needs roughly half the time to build SSG. I was timing `make -j 4`.
It will take a while for these changes to make it into releases and into distributions but I am writing this email because the savings are significant and perhaps you want to enjoy them sooner. SCAP Security Guide contributors are doing a lot of content builds.
This will only work if you !!! have OPENSCAP 1.2.x !!!
If you want to speed up the build, replace
/usr/share/openscap/xsl/xccdf_1.1_to_1.2.xsl with https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf_1.1_...
/usr/share/openscap/xsl/xccdf-share.xsl with https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf-shar...
and
/usr/share/openscap/xsl/xccdf-guide-impl.xsl with https://github.com/OpenSCAP/openscap/blob/maint-1.2/xsl/xccdf-guide-impl.xsl
Perhaps it would be worth it to deploy this on our Jenkins slaves even before it hits releases? What do you think?
+1
Not like I would want to be the blocker for speedups / SSG Jenkins optimizations, but how would you like to deploy these changes?
Two scenarios come to mind: * just replace the affected XSLT transformations in recent RHEL openscap builds, * make a separate openscap RPM (applied on Jenkins slaves outside of RHEL release cycle process)
While the advantage might be more quicker completion of Jenkins SSG build jobs, there are two disadvantages / concerns related with this: * if modified oscap returns different result than latest oscap available in RHEL - which report should be considered the right one? Which one is valid? * the concern with maybe even more importance is we wouldn't test on default RHEL / Fedora systems like we do now. The environment would be modified a bit (and it might or might not return same results). So to ensure we "will continue working" on default RHEL / Fedora - should we create another Jenkins CI jobs for default RHEL / Fedora?
Maybe we could use "speed optimized" openscap on 'scap-security-guide-pull-requests' Jenkins CI job, and 'default one' on 'scap-security-guide' Jenkins CI job (the latter one being which actually tells if we don't regress across PRs).
But even this approach won't solve the issue when the result of 'speed optimized' openscap would differ from 'default' openscap (but maybe the result on the default one would be more authoritative / determinant in this case).
Just my 2cs.
Jan -- Jan iankko Lieskovsky / Red Hat Security Technologies Team
-- SCAP Security Guide mailing list scap-security-guide@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/scap-security-guide@lists.fedorah... https://github.com/OpenSCAP/scap-security-guide/
----- Original Message -----
From: "Jan Lieskovsky" jlieskov@redhat.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Cc: "open-scap-list" open-scap-list@redhat.com Sent: Wednesday, July 27, 2016 9:34:04 AM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
----- Original Message -----
From: "Gabe Alford" redhatrises@gmail.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Cc: "open-scap-list" open-scap-list@redhat.com Sent: Wednesday, July 27, 2016 3:00:34 PM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
On Tue, Jul 26, 2016 at 3:24 PM, Martin Preisler < mpreisle@redhat.com > wrote:
I found pretty bad inefficiencies in some of our XSLTs.
Check out https://github.com/OpenSCAP/openscap/commit/a65bf27dec4a93e2b87cec8cbcd80bec... or https://github.com/OpenSCAP/openscap/commit/1dd5573b2c964b00af57215cadb7f13b...
Long story short I was able to cut SSG build time on my machine from 2m21.061s to 1m08.771s. In other words my machine now needs roughly half the time to build SSG. I was timing `make -j 4`.
It will take a while for these changes to make it into releases and into distributions but I am writing this email because the savings are significant and perhaps you want to enjoy them sooner. SCAP Security Guide contributors are doing a lot of content builds.
This will only work if you !!! have OPENSCAP 1.2.x !!!
If you want to speed up the build, replace
/usr/share/openscap/xsl/xccdf_1.1_to_1.2.xsl with https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf_1.1_...
/usr/share/openscap/xsl/xccdf-share.xsl with https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf-shar...
and
/usr/share/openscap/xsl/xccdf-guide-impl.xsl with https://github.com/OpenSCAP/openscap/blob/maint-1.2/xsl/xccdf-guide-impl.xsl
Perhaps it would be worth it to deploy this on our Jenkins slaves even before it hits releases? What do you think?
+1
Not like I would want to be the blocker for speedups / SSG Jenkins optimizations, but how would you like to deploy these changes?
Two scenarios come to mind:
- just replace the affected XSLT transformations in recent RHEL openscap builds,
- make a separate openscap RPM (applied on Jenkins slaves outside of RHEL release cycle process)
While the advantage might be more quicker completion of Jenkins SSG build jobs, there are two disadvantages / concerns related with this:
- if modified oscap returns different result than latest oscap available in RHEL - which report should be considered the right one? Which one is valid?
- the concern with maybe even more importance is we wouldn't test on default RHEL / Fedora systems like we do now. The environment would be modified a bit (and it might or might not return same results). So to ensure we "will continue working" on default RHEL / Fedora - should we create another Jenkins CI jobs for default RHEL / Fedora?
Maybe we could use "speed optimized" openscap on 'scap-security-guide-pull-requests' Jenkins CI job, and 'default one' on 'scap-security-guide' Jenkins CI job (the latter one being which actually tells if we don't regress across PRs).
I think that would overcomplicate things. I can relate to your concerns. The only reason why we are even considering this is because the speed-up is so dramatic.
What do you think about me generating RHEL 6 and 7 datastreams and guides and diff-ing them? If the results are the same across the board that would take care of my concerns. Do you agree?
----- Original Message -----
From: "Martin Preisler" mpreisle@redhat.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org, "Jan Lieskovsky" jlieskov@redhat.com Cc: "open-scap-list" open-scap-list@redhat.com Sent: Wednesday, July 27, 2016 8:01:19 PM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
----- Original Message -----
From: "Jan Lieskovsky" jlieskov@redhat.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Cc: "open-scap-list" open-scap-list@redhat.com Sent: Wednesday, July 27, 2016 9:34:04 AM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
----- Original Message -----
From: "Gabe Alford" redhatrises@gmail.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org Cc: "open-scap-list" open-scap-list@redhat.com Sent: Wednesday, July 27, 2016 3:00:34 PM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
On Tue, Jul 26, 2016 at 3:24 PM, Martin Preisler < mpreisle@redhat.com > wrote:
I found pretty bad inefficiencies in some of our XSLTs.
Check out https://github.com/OpenSCAP/openscap/commit/a65bf27dec4a93e2b87cec8cbcd80bec... or https://github.com/OpenSCAP/openscap/commit/1dd5573b2c964b00af57215cadb7f13b...
Long story short I was able to cut SSG build time on my machine from 2m21.061s to 1m08.771s. In other words my machine now needs roughly half the time to build SSG. I was timing `make -j 4`.
It will take a while for these changes to make it into releases and into distributions but I am writing this email because the savings are significant and perhaps you want to enjoy them sooner. SCAP Security Guide contributors are doing a lot of content builds.
This will only work if you !!! have OPENSCAP 1.2.x !!!
If you want to speed up the build, replace
/usr/share/openscap/xsl/xccdf_1.1_to_1.2.xsl with https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf_1.1_...
/usr/share/openscap/xsl/xccdf-share.xsl with https://raw.githubusercontent.com/OpenSCAP/openscap/maint-1.2/xsl/xccdf-shar...
and
/usr/share/openscap/xsl/xccdf-guide-impl.xsl with https://github.com/OpenSCAP/openscap/blob/maint-1.2/xsl/xccdf-guide-impl.xsl
Perhaps it would be worth it to deploy this on our Jenkins slaves even before it hits releases? What do you think?
+1
Not like I would want to be the blocker for speedups / SSG Jenkins optimizations, but how would you like to deploy these changes?
Two scenarios come to mind:
- just replace the affected XSLT transformations in recent RHEL openscap builds,
- make a separate openscap RPM (applied on Jenkins slaves outside of RHEL release cycle process)
While the advantage might be more quicker completion of Jenkins SSG build jobs, there are two disadvantages / concerns related with this:
- if modified oscap returns different result than latest oscap available in RHEL - which report should be considered the right one? Which one is valid?
- the concern with maybe even more importance is we wouldn't test on
default RHEL / Fedora systems like we do now. The environment would be modified a bit (and it might or might not return same results). So to ensure we "will continue working" on default RHEL / Fedora - should we create another Jenkins CI jobs for default RHEL / Fedora?
Maybe we could use "speed optimized" openscap on 'scap-security-guide-pull-requests' Jenkins CI job, and 'default one' on 'scap-security-guide' Jenkins CI job (the latter one being which actually tells if we don't regress across PRs).
I think that would overcomplicate things. I can relate to your concerns. The only reason why we are even considering this is because the speed-up is so dramatic.
What do you think about me generating RHEL 6 and 7 datastreams and guides and diff-ing them? If the results are the same across the board that would take care of my concerns. Do you agree?
That would be helpful, yes. Thanks, Martin.
There will be some differences (for example timestamp) for sure. But the point is to ensure there won't be some other inevitable deviations (that could lead e.g. to the reduction of the file size at the end). IOW verification if those XSLT changes are that isolated enough, it won't hurt when they are used also together with old openscap code base (without other patches from upstream being applied).
Thanks, Jan -- Jan iankko Lieskovsky / Red Hat Security Technologies Team
-- Martin Preisler Identity Management and Platform Security | Red Hat, Inc.
----- Original Message -----
From: "Jan Lieskovsky" jlieskov@redhat.com To: "Martin Preisler" mpreisle@redhat.com Cc: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org, "open-scap-list" open-scap-list@redhat.com Sent: Thursday, July 28, 2016 5:05:26 AM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
[snip]
That would be helpful, yes. Thanks, Martin.
There will be some differences (for example timestamp) for sure. But the point is to ensure there won't be some other inevitable deviations (that could lead e.g. to the reduction of the file size at the end). IOW verification if those XSLT changes are that isolated enough, it won't hurt when they are used also together with old openscap code base (without other patches from upstream being applied).
I wrote a blog post about the optimizations and about the test-case I created to verify correctness. Check it out at https://martin.preisler.me/2016/07/openscap-xslt-performance-improvements-fo... and let me know if this addresses your concerns.
Feel free to run the test-case on your machine as well so that we have more data sources.
Hello Martin,
sorry for the delay when processing this request (other issues prevented me from sooner reaction). Anyway, pls see below.
----- Original Message -----
From: "Martin Preisler" mpreisle@redhat.com To: "Jan Lieskovsky" jlieskov@redhat.com Cc: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org, "open-scap-list" open-scap-list@redhat.com Sent: Thursday, July 28, 2016 8:01:47 PM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
----- Original Message -----
From: "Jan Lieskovsky" jlieskov@redhat.com To: "Martin Preisler" mpreisle@redhat.com Cc: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org, "open-scap-list" open-scap-list@redhat.com Sent: Thursday, July 28, 2016 5:05:26 AM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
[snip]
That would be helpful, yes. Thanks, Martin.
There will be some differences (for example timestamp) for sure. But the point is to ensure there won't be some other inevitable deviations (that could lead e.g. to the reduction of the file size at the end). IOW verification if those XSLT changes are that isolated enough, it won't hurt when they are used also together with old openscap code base (without other patches from upstream being applied).
I wrote a blog post about the optimizations and about the test-case I created to verify correctness. Check it out at https://martin.preisler.me/2016/07/openscap-xslt-performance-improvements-fo... and let me know if this addresses your concerns.
Feel free to run the test-case on your machine as well so that we have more data sources.
I have compared the output on two RHEL-7 machines running openscap-1.2.10 vs openscap-1.2.11 (latest 1.2-main master) and diffed both *-xccdf-1.2.xml and selected guides manually, and the results are below:
* for -xccdf-1.2 files, the output is identical:
$ diff -s ssg-rhel7-xccdf-1.2.xml ssg-rhel7-xccdf-1.2.xml-latest Files ssg-rhel7-xccdf-1.2.xml and ssg-rhel7-xccdf-1.2.xml-latest are identical
$ ls -l ssg-rhel7-xccdf-1.2.xml* -rw-r--r--. 1 iankko iankko 1839260 aug 1 18:16 ssg-rhel7-xccdf-1.2.xml -rw-rw-r--. 1 iankko iankko 1839260 aug 1 17:47 ssg-rhel7-xccdf-1.2.xml-latest
* for the modified guides template, the output differs slightly.
The 'guide-tree-leaf-id*' changes are irrelevant (looks like just different hash value) => safe.
When unified all of them to same form in both files, there are some changes in xccdf:Values yet (looks some xccdf:Values got different default selector values, pls see the attached file below. But IMHO this is just demonstration of the bug you have mentioned before) => changes pending confirmation.
The other change is just oscap version change in some element at the benchmark end => expected / safe change too.
To summary, IMHO the changes are either safe, or are demonstration of that 'default selector' xccdf:Value bug, you mentioned earlier =>
I agree this change to be safe to be applied on Jenkins slaves (IOW it's functionally identical to version in previous oscap releases).
Regards, Jan -- Jan iankko Lieskovsky / Red Hat Security Technologies Team
-- Martin Preisler Identity Management and Platform Security | Red Hat, Inc.
----- Original Message -----
From: "Jan Lieskovsky" jlieskov@redhat.com To: "Martin Preisler" mpreisle@redhat.com Cc: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org, "open-scap-list" open-scap-list@redhat.com Sent: Monday, August 1, 2016 12:38:01 PM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
Hello Martin,
[snip]
I agree this change to be safe to be applied on Jenkins slaves (IOW it's functionally identical to version in previous oscap releases).
Thanks for the review, Jan!
I have applied the optimizations on Jenkins slaves and from my initial testing the time required for pull request builds and normal CI builds went down roughly 35-40%. It's not as much as I expected but it is a noticeable improvement.
Old typical el7 SSG build + validate times: 30-40 minutes. New typical el7 SSG build + validate times: 20-30 minutes.
The next big optimization lies in getting better hardware for Jenkins. `make jenkins` takes 2m16.877s on my work laptop and 20+ minutes on the OS1 nodes. That is a huge difference. And my laptop can't be called a powerful machine, it's just a dual core i7 laptop.
On 8/2/16 2:51 PM, Martin Preisler wrote:
----- Original Message -----
From: "Jan Lieskovsky"jlieskov@redhat.com To: "Martin Preisler"mpreisle@redhat.com Cc: "SCAP Security Guide"scap-security-guide@lists.fedorahosted.org, "open-scap-list"open-scap-list@redhat.com Sent: Monday, August 1, 2016 12:38:01 PM Subject: Re: Latest OpenSCAP changes to speed up SSG builds
Hello Martin,
[snip]
I agree this change to be safe to be applied on Jenkins slaves (IOW it's functionally identical to version in previous oscap releases).
Thanks for the review, Jan!
I have applied the optimizations on Jenkins slaves and from my initial testing the time required for pull request builds and normal CI builds went down roughly 35-40%. It's not as much as I expected but it is a noticeable improvement.
Old typical el7 SSG build + validate times: 30-40 minutes. New typical el7 SSG build + validate times: 20-30 minutes.
The next big optimization lies in getting better hardware for Jenkins. `make jenkins` takes 2m16.877s on my work laptop and 20+ minutes on the OS1 nodes. That is a huge difference. And my laptop can't be called a powerful machine, it's just a dual core i7 laptop.
OS1 is being decommissioned over the coming months. I'm not sure what (if any) environment is taking over for that.
What kind of computing power do we need? Perhaps we could look into a Reserved Instance on EC2.
With regards to the build times mentioned below I suspect your laptop has a higher clock speed compared to your server which likely has many cores to host many guest VMs, one of which is Jenkins. Cores are great for supporting many guest VMs but in a build environment they struggle due to most 8-14 core Xeons only having 2.xGHz clock speed. Cheap Dell R220’s offer Intel 4 core Xeons with 3.7ghz and they work great for our build machines. A 1ghz diff (2.x to 3.x GHz) in some builds can double overall build times.
Lee Meinecke, CISSP Research Associate Georgia Tech Research Institute Applied Systems Laboratory
From: Shawn Wells shawn@redhat.com Reply-To: SCAP Security Guide scap-security-guide@lists.fedorahosted.org Date: Tuesday, August 2, 2016 at 5:45 PM To: SCAP Security Guide scap-security-guide@lists.fedorahosted.org Cc: open-scap-list open-scap-list@redhat.com Subject: Re: Latest OpenSCAP changes to speed up SSG builds
On 8/2/16 2:51 PM, Martin Preisler wrote:
----- Original Message -----
From: "Jan Lieskovsky" jlieskov@redhat.commailto:jlieskov@redhat.com
To: "Martin Preisler" mpreisle@redhat.commailto:mpreisle@redhat.com
Cc: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.orgmailto:scap-security-guide@lists.fedorahosted.org, "open-scap-list" open-scap-list@redhat.commailto:open-scap-list@redhat.com
Sent: Monday, August 1, 2016 12:38:01 PM
Subject: Re: Latest OpenSCAP changes to speed up SSG builds
Hello Martin,
[snip]
I agree this change to be safe to be applied on Jenkins slaves
(IOW it's functionally identical to version in previous oscap releases).
Thanks for the review, Jan!
I have applied the optimizations on Jenkins slaves and from my initial
testing the time required for pull request builds and normal CI builds
went down roughly 35-40%. It's not as much as I expected but it is
a noticeable improvement.
Old typical el7 SSG build + validate times: 30-40 minutes.
New typical el7 SSG build + validate times: 20-30 minutes.
The next big optimization lies in getting better hardware for Jenkins.
`make jenkins` takes 2m16.877s on my work laptop and 20+ minutes on
the OS1 nodes. That is a huge difference. And my laptop can't be called
a powerful machine, it's just a dual core i7 laptop.
OS1 is being decommissioned over the coming months. I'm not sure what (if any) environment is taking over for that.
What kind of computing power do we need? Perhaps we could look into a Reserved Instance on EC2.
----- Original Message -----
From: "Martin Preisler" mpreisle@redhat.com To: "SCAP Security Guide" scap-security-guide@lists.fedorahosted.org, "open-scap-list" open-scap-list@redhat.com Sent: Tuesday, July 26, 2016 5:24:17 PM Subject: [Open-scap] Latest OpenSCAP changes to speed up SSG builds
I found pretty bad inefficiencies in some of our XSLTs.
Just for fun I looked at HTML report generation today and also found horrible inefficiencies. I was able to save roughly 65% :-/
Especially https://github.com/OpenSCAP/openscap/commit/9fa05b0335d4b598c77e22d918fc4c7c... is a total gem. What the hell was I thinking...
There is probably more optimization to be had but I am happy that we now typically generate an HTML report in a second and we typically generate an HTML guide in a second. I think that's good enough for now.
scap-security-guide@lists.fedorahosted.org
-
Gabe Alford -
Jan Lieskovsky -
Martin Preisler -
Meinecke, Lee -
Shawn Wells