Martin Zember kindly let us torture him a bit about SCAP scanning and
tailoring so we can get a better idea about what we should document
or completely change. We explicitly asked him not to study anything
in advance so that we can figure out what's "easy" and what's not.
And more importantly, where are users looking for information.
I am cross-posting to share the results of this test because it
involves all the SCAP projects.
The test we performed is described at
Please note that this is the first iteration and we will likely
tune the test process in each iteration. Suggestions are of course
Notable general pain points:
* USGCB? What is it? Is it on paper or is it machine readable?
* What does benchmark mean in this context?
* How do I change a guidance? (tailoring confusion)
* What is a profile? A lot of confusion about content vs profile.
* It is easy to find some info about RHEL5 USGCB, the assumption
was that RHEL6 should be similar which is sadly not the case.
provides content for RHEL5, doesn't even mention
anything about RHEL6
* In general it's hard to figure out that SCAP is the way to
automate paper guidances. Perhaps we want to use other keywords.
* A lot of confusion about the difference between openscap and
* xccdf Value vs xccdf Rule, which Rules use which Values.
* openscap doesn't include any content, yet it was expected of it
- user tried to look for "some version of openscap that can do
* if user selects a profile ID that's not in the content, openscap
should either print `oscap info` on the file to help or at
least hint the existence of `oscap info`.
* ssg custom repo in documentation, outdated...
* user didn't notice a report has already been generated using
--report and generated it again
* confusion about OVAL results and check system details
* Upstream web mentions EPEL which is outdated and confusing
- note: it is not actually outdated yet but will be soon
* Upstream doesn't mention profiles that ssg provides
* It is entirely ungooglable
- user expected to find it with: "automate usgcb linux",
"usgcb linux tool", "check linux against usgcb",
"linux security compliance", "usgcb audit linux".
- the only way to find it is to find openscap page first
and then look at related projects, unfortunately openscap
doesn't mention that scap-security-guide has USGCB...
* stig-rhel6-server was renamed to stig-rhel6-server-upstream,
documentation mentions stig-rhel6-server. Why would an ID
like this change?
* user@target placeholder text disappears when you paste IP
there. Then user doesn't realize that user@ can also go there.
* Where does workbench get ssh password? User first searched
for input boxes or settings before starting the scan and
filling out the ssh-askpass box.
* Ctrl+F or / was expected to search in both main window
and tailoring window.
* Enter doesn't confirm search in tailoring window, user had
* No visualization of dependencies between rules and values
* Save content vs Save tailoring, what is the difference,
what do I want to use?
* The user manual was not used unless explicitly talked about.
* Opening tailoring file directly doesn't automatically open
the content and then the tailoring file.
* scap-workbench is not all that much advertised on
scap-security-guide or openscap web pages. It is therefore
quite hard to discover it.