First of let me say that the statement was not intended to insult any of the contributors such as Simon, Jan, Martin, and so many others on different projects. Their work in other SCAP related projects has been absolutely phenomenal. This thread is about Aqueduct and Red Hat's role in that project as a developer and user.

There is no disputing Red Hat has been a voice for the Aqueduct project since the beginning.  But Linux, OpenSCAP, SSG, RHEL, Docker, and thousands of other Open Source projects were not built by blogs, presentations and podcasts alone.  It was built with a community - a community contributing code.  Something we all (Red Hat included) take great pride in (http://www.redhat.com/infographics/openstack/).

Shawn, you had a tough decision to make and you made it.  Being an Open Source community leader isn’t easy and isn’t always fun.  As leaders in the community we often have to remember that “the needs of the many outweigh the needs of the few” *Star Trek*. 

At the time when you made the decision to push forward with remediation inside SSG Aqueduct code base was too large for the bash one-liners restriction in SSG.  Aqueduct community understood the decision, even if it caused a little heartburn Aqueduct stood by SSG.  But then ‘hatters’ used Aqueduct code to answer a problem Red Hat has for their professional services team with no contributions back into Aqueduct.

Actions need to align with the decisions and the code speaks for itself.

Aqueduct_GIT ---$ grep -rai '@redhat.com' * 

aqueduct.spec:* Mon Sep 10 2012 Ted Brunell <tbrunell@redhat.com>

aqueduct.spec:* Tue May 4 2012 Ted Brunell <tbrunell@redhat.com>

Looking at the Aqueduct code there are 29,136 lines.  A quick search for Red Hat contributions (see above). The basis for this search is that a Red Hat email is tied to the Copy Right or email (as seen in rhel-stig-fix). I excluded findings for generic emails such as security@redhat.com

A git log yields no Red Hat contributions.

The community was fully aware of Red Hat’s position on Aqueduct being a separate project as Red Hat wanted SSG to be the source of remediation content within the <fix> tag statement.  Aqueduct wasn’t able to fit this model simply due to the size (line count) of the Aqueduct scripts.  Aqueduct needed more code to conduct remediation properly and the consensus was that ‘bash one liners’ were the most viable candidates for <fix> tags.  As no other language support existed in the SCAP framework, non-bash languages in Aqueduct such as Puppet and Ansible would be completely unfeasible to ingest, both near and far term [1].  This is why the government pushed Puppet content into Aqueduct (Thank you Maura).

Conversations around linking of SCAP content to external sources was discussed and favored by a number of different sources (Red Hat & Non-Red Hat) [2] [5.1]. Non-Red Hat & Red Hat people even agreed that linking remediation content to SCAP content would be feasible [4-5]. Other contributors voiced concerns of SCAP remediation language openly in the community and decided to push content into Aqueduct [5.1].

The Aqueduct community has stood by SSG and assisted as much as possible.  We even hosted a call specifically to help SSG meet their submission deadline of the RHEL 5 SCAP content [6].

Thanks for the how-to guide [3].  P.S. You have a 404 link broken to one of the Aqueduct postings [7] :( .

Credit where credit is due, Aaron Prayther contributed that code in Aqueduct [8] [9].

Just as the comments previously in the thread are as you put it ‘a dagger to the heart’ of hatters, it is the same dagger to the heart of an open source community watch an organization champion the project, leverage its code to solve its own problems, while not contributing back into the project.  As Open Source community leaders this is not the example we want to set.


[1] https://lists.fedorahosted.org/mailman/private/aqueduct/2012-August/000384.html

[2] https://lists.fedorahosted.org/pipermail/scap-security-guide/2012-September/001729.html

[3] http://blog-shawndwells.rhcloud.com/so-you-wanna-stig-a-rhel5-box-part2/

[4] https://lists.fedorahosted.org/pipermail/scap-security-guide/2012-September/001729.html

[5] https://lists.fedorahosted.org/pipermail/scap-security-guide/2013-March/002855.html

[5.1] https://lists.fedorahosted.org/mailman/private/aqueduct/2012-August/000375.html

[6] https://fedorahosted.org/aqueduct/attachment/wiki/Call/Aqueduct_8-10-2012.mp3?format=raw

[7] http://blog-shawndwells.rhcloud.com/2012/10/so-you-wanna-stig-a-rhel5-box-part2/

[8] https://lists.fedorahosted.org/mailman/private/aqueduct/2011-June/000003.html 

[9] https://lists.fedorahosted.org/mailman/private/aqueduct/2011-June/000002.html 



5 PM, Shawn Wells <shawn.d.wells@gmail.com> wrote:

On 9/3/14, 12:47 PM, Vincent Passaro wrote:

I can relate to you experienced from your previous employers. It’s sometimes very difficult to convince leadership to buy off on pushing code into the Open Source community. 

We (the Aqueduct community) would have really liked to see Red Hat and their Employees set the standard for keeping a community working together, rather than individuals forking existing code in active projects to solve the exact problem.


Many Red Hat staff needed a stable, repeatable process to build a customized RHEL6 STIG installation disk. Code from Aqueduct was needed to accomplish this goal. And from the Tresys CLIP. And from SCAP Security Guide. And from some home-brew gluecode. Once these things were mashed together a installation disk was churned out in a very predictable manner, serving the purposes of those who needed the disks. People noticed the ability to generate custom, pre-STIG'd installation media and wanted to create their own. So we put the code on GitHub.

Independently tracking each underlying project and figuring out what patches/revisions from each are needed is generally to cumbersome to the average user. The stig-fix-el6-kickstart [1] and stig-fix-el6 [2] repositories are not to meant to usurp or undermine any of these communities, but rather snapshot each component and turn out an installable DVD/ISO file. This represents a very healthy upstream/downstream relationship. Every attempt was made to credit upstream sources, and you'll notice upstream sources are called out in the respective README files [3][4].

Hope this clears up the stig-fix-el6-kickstart repos. They are not an attempt at forking any project, or to build a new community. Just to organize all the subprojects in a very consumable fashion (which is to say, generate installation disks).


A community is only as strong as it's contributors and Red Hat isn't contributing anything.

If you were to pose the concept to the Aqueduct mailing list, the community would embrace your ideas of ‘allowing the customer to decide on what’s best for their site/project’ and would help you along the way. As the STIG matures and issues/fixes are committed, having the source Aqueduct code based being reused will save all of us a great deal of time. Rather than forcing people to pick which fork of the day works best for them. Again - a community is only as strong as it's contributors. 

If you’re interested in helping out the Aqueduct community, I’d be more than happy to sponsor your commit access.


Hatters helped migrate the DoDBastile code from forge.mil into Aqueduct [5], Hatters helped create some of the first "how to use Aqueduct" videos [6], Hatters hosted the initial Aqueduct community calls [7][8][9], Hatters helped organize Aqueduct community meetups (such as before the DC Government Symposium [10]), and Hatters invited you to speak at Red Hat Summit 2012 [11].

Hatters also list Aqueduct as a project of interest off the official Red Hat Government webpage:
https://www.redhat.com/en/technologies/industries/government/standards

And I personally wrote a blog on using Aqueduct to STIG a RHEL5 box (albeit, it's now somewhat outdated):
http://blog-shawndwells.rhcloud.com/so-you-wanna-stig-a-rhel5-box-part2/

My prior role at Red Hat was to cover U.S. Intelligence programs. During our 2012 update to Ft Meade -- which had worldwide attendance from staffers and contractors -- I specifically called out Aqueduct. Reference p91-p96:
http://blog-shawndwells.rhcloud.com/wp-content/uploads/2012/07/2012-02-08-MPO-Red-Hat-Update-v2.pdf

Another Hatter called out Aqueduct at Red Hat Summit 2014 as well (ref p17):
http://rhsummit.files.wordpress.com/2014/04/seifried_w_0230_cloud_security.pdf

So, we should perhaps reword slightly:
"A community is only as strong as it's contributors and Red Hat isn't contributing anything to Aqueduct, lately"

And lets now address that reworded statement.

On 7-JULY-2011, Hatters began briefing the Aqueduct on initiatives regarding SCAP [12][13]. Steve Grubb personally gave a webinar on the matter [14]. Then on 28-SEPT-2011, the Aqueduct community was informed the SCAP Security Guide project was starting [15]. A snippet from that original EMail:

It is designed to provide security guidance and checking content in the 
SCAP formats.  Specifically, the project goal is to produce --- from a 
single body of content:

1) an informative prose guide, similar in style to NSA's RHEL 5 guide
2) a desktop settings baseline, similar in style to NIST USGCB
3) a server settings baseline,  similar in style to a STIG
4) additional baselines for other network roles (such as LAMP server)
5) a body of automated checking content in the OVAL format

As reflected in the text above, SSG was initially formed to build out prose and compliance automation content. Incredibly complementary to Aqueduct, which held remediation scripts.

Almost a year later, on 1-AUG-2012, I began the conversation of linking SSG (compliance) with Aqueduct (remediation) [16]. My own words:
Today the scap-security-guide project provides scanning only 
(pass/fail), in the near future we'll start working on remediation. For 
remediation it is envisioned that we'll be able to work together with 
the Aqueduct community to provide a single document (the 
scap-security-guide) which will contain both scanning and remediation 
components.

Functionally you'll deploy RHEL, run "oscap eval --profile STIG-server" 
and receive a report stating what actions you must take to become STIG 
compliant. "oscap fix --profile STIG-server" will perform the 
remediation(s).

Between Sept 2011 and Aug 2012 the SSG project matured greatly. One of the maturation points was the establishment of a very large body of compliance rules which could be logically organized into profiles (STIG, USGCB, NIST, etc). The vision was to have SSG become more of an "engine" -- you run a profile, and the variables of that profile (say, password length) would auto-populate into human-readable prose guides, OVAL, and remediation scripts. This vision would enable multiple groups (DoD for STIG, civilian for NIST) to collaborate in a central location for all aspects of compliance.

To accomplish that vision the bash scripts needed to be embedded inside the SSG content. There was not, nor currently is, any feasible way to link to external sources. Aqueduct decided to maintain their own bash scripts and not integrate into a consolidated prose->scanning->remediation process. And that's totally fine. But because of this decision, SSG began to embedded <fix> tags natively outside of Aqueduct. For the complete workflow to be established there wasn't any other choice.

Personally, I've long known individuals attach themselves to shared visions, shared purposes, shared callings. But I never fully appreciated this until working on the SCAP Security Guide project. As our scope expanded from compliance checking (which is, admit it, boring) to an integrated compliance workflow (which is incredibly useful and spans multiple user communities), our community exponentially expanded. Today we have some 47 code committers who have pushed an average of 50 commits per author across 226,969 lines of code [17]. Our other types of contributors, those working on documentation, reporting bugs, and working across their spheres of influence advocating the use of SSG, are incredibly hard to measure and provide metrics on. But they are there, and they are highly valued. 64.14% of SSG's commits come during 0800-1800, core business hours [18]. Not only are SSG community members working during their personal time, many employers are allowing employees to work on SSG during working hours as it provides a direct business impact. In fact, the business impact has been so overwhelming that SSG will begin shipping natively in RHEL come RHEL 6.6 [19]. Heck, there is even a community college giving homework assignments to use SSG to harden Linux machines (how cool is that?!) [20].

And the SCAP Security Guide project is but one under the umbrella of OpenSCAP family. A core principal of OpenSCAP is that the components and peripherals to enable enterprise SCAP usage should be open source and freely available. To us Red Hatters, that means we should ship them natively in the operating system. And the core components are (or will be shortly) shipping natively.

The OpenSCAP interpreter reflects some 4,799 commits across 46 authors and 1,414,106 lines of code [21]. Even Oracle has submitted patches to OpenSCAP [22]!

The SCAP Workbench project, which provides a GUI to using and customizing SCAP, reflects some 1,114 commits across 10 authors and 12,760 lines of code [23].

The OSCAP Anaconda Plugin, which integrates SCAP directly into RHEL's installer, reflects some 222 commits across 2 authors and 6,308 lines of code [24].

Across the 4 core projects -- OpenSCAP interpretter, SSG, Workbench, and Anaconda -- there are some 1.66 million lines of source code from nearly 80 individual contributors. And that's just the code contributors! The community is so, so much more.


All this verbosity to provide background for the following: The state of technology has allowed us to move from "compliance craftsmanship" to "compliance manufacturing." Under this vein OpenSCAP has become an empowered movement across multiple interrelated communities, all working towards the vision of a holistic workflow between prose guides, scanning engines, remediation content, and natively provided tools. This fundamental shift meant that compliance scripts must be natively integrated into the scanning technologies. A side effect of forging unified workflow meant that Aqueduct would be put in a position of integrating with SSG -- which we welcomed, and continue to welcome, with open arms! -- or, unfortunately, would be left behind as the broader community forged ahead [yes yes, I realize how "borg"-ish this sounds].

When it came to the matter of "integrate or forge ahead," and in order for progress to be made, I had to make a decision. It was to forge ahead. And I beat that drum in a very loud, outspoken manner. So yes, when it came to Red Hat contributions, I lobbied for them to be made to SSG. This was not out of malintent or anger, and certainly not about control. When talking about SSG I proudly refer to the Aqueduct as part of our lineage. I state that many of the remediation scripts originated from Aqueduct, and I talk about the DoDBastile story. I actively state that Aqueduct is a perfect place for things like Puppet, which SSG simply isn't ready for yet.

It's disheartening when you make comments insinuating that Red Hat, or any of our communities, is falling short. Upon becoming a Hatter, everyone is shown this video:
http://www.redhat.com/v/ogg/stories/RHS_RedHatWay.ogg

It reflects our mission statement, which itself was developed in an open source fashion [25]:
To be the catalyst in communities of customers, contributors, and partners creating better technology the open source way.

Vince, and others in the Aqueduct community, I know my decision to forge ahead (effectively split from Aqueduct) gave you heartburn. In hindsight I can certainly identify moments where I could have been far less of, well, a dick. The decision weighed on me, and I often reflect on if I made the right one. I carry an element of personal guilt about it. Vince, I am sorry for how I handled the divergence of Aqueduct and SSG and I am sorry for any perceived role in devaluing the Aqueduct community.

But do not take it out on the community. Telling a Red Hatter they are effectively falling short on the ethos of open source is a dagger to the heart. I really can't think of a more negatively impacting statement that cuts to the core of our ideology. Such comments devalue the work of people like Simon, Jan, and Martin who are professionals that Red Hat employs and who work on OpenSCAP projects as their full time job. This is completely uncalled for.

With this long note coming to a close, it would sincerely be great to work together and re-unify. There's strength in numbers, and strength in a broad base of ideas to accomplish a goal. Constructive dialog on that would be incredibly welcome. Hope hasn't been lost for a vulcan re-unification [26]*;)



[1] https://github.com/RedHatGov/stig-fix-el6-kickstart
[2] https://github.com/RedHatGov/stig-fix-el6
[3] https://github.com/RedHatGov/stig-fix-el6/blob/master/README
[4] https://github.com/RedHatGov/stig-fix-el6-kickstart/blob/master/README
[5] https://lists.fedorahosted.org/mailman/private/aqueduct/2011-June/000000.html
[6] https://lists.fedorahosted.org/mailman/private/aqueduct/2011-June/000004.html
[7] https://lists.fedorahosted.org/mailman/private/aqueduct/2011-June/000012.html
[8] https://lists.fedorahosted.org/mailman/private/aqueduct/2011-September/000034.html
[9] https://lists.fedorahosted.org/mailman/private/aqueduct/2011-November/000080.html
[10] https://lists.fedorahosted.org/mailman/private/aqueduct/2011-November/000062.html
[11] http://www.vincentpassaro.com/2012/04/01/red-hat-summit-2012-2/
[12] https://lists.fedorahosted.org/mailman/private/aqueduct/2011-July/000023.html
[13] https://fedorahosted.org/aqueduct/attachment/wiki/Call/intro-SCAP-tech-talk.pdf?format=raw
[14] https://sas.elluminate.com/p.jnlp?psid=2011-07-15.0937.M.A6EDFFACF1B4203F04043FD1E3A67B.vcr&sid=819
[15] https://lists.fedorahosted.org/mailman/private/aqueduct/2011-September/000033.html
[16] https://lists.fedorahosted.org/mailman/private/aqueduct/2012-August/000373.html
[17] http://people.redhat.com/swells/gitstats/ssg-20140904/
[18] http://people.redhat.com/swells/gitstats/ssg-20140904/activity.html
[19] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6-Beta/html-single/6.6_Release_Notes/index.html#bh-chap-security
[20] http://content.hccfl.edu/pollock/AUnixSec/Proj-C-Harden.htm
[21] http://people.redhat.com/swells/gitstats/20140904-openscap/
[22] http://people.redhat.com/swells/gitstats/20140904-openscap/authors.html
[23] http://people.redhat.com/swells/gitstats/20140904-scap-workbench/
[24] http://people.redhat.com/swells/gitstats/20140904-oscap-anaconda-addon/
[25] http://www.managementexchange.com/blog/how-red-hat-used-open-source-way-develop-company-mission
[26] http://en.memory-alpha.org/wiki/Vulcan_reunification
* I just realized I've been watching way, way to much Star Trek
--
SCAP Security Guide mailing list
scap-security-guide@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide
https://github.com/OpenSCAP/scap-security-guide/