On Mon, Sep 1, 2014 at 5:32 AM, Martin Preisler <mpreisle(a)redhat.com> wrote:
----- Original Message -----
> From: "Greg Elin" <gregelin(a)gitmachines.com>
> To: "SCAP Security Guide"
<scap-security-guide(a)lists.fedorahosted.org>
> Sent: Friday, August 29, 2014 11:18:37 PM
> Subject: Re: New report and guide in openscap 1.1.0
>
> I agree with Trey. The details would be more useful if they contained
more
> information about how something failed. I would expect that in a detail.
As I said in my previous email, we show these details. We always have!
They were not present in one of the previous ! draft ! versions of
XCCDF report. When I released that draft I explicitly said that check
details
are not there but will be there in the final version. Are you sure you are
looking at the new report?
Thanks for the prodding. I looked more closely last night. At least two of
us had some confusion with this.
AFAIK, Oval results only show up if the `--oval-results` is added to `oscap
xccdf eval` command. It seems to me specific errors should be shared by
default and flag used to hide them. Yes, it is in the man page and of
course we all read the man page... (But I guess that is a comment for the
OpenSCAP List?)
Oval details show up on only some failed tests in the sample report (
https://mpreisle.fedorapeople.org/openscap/1.1.0_xslt/report.html)
"package_aide_installed" for example does not show any oval details though
it fails while "service_atd_disabled" does show oval results.
Last night I built the openSCAP from source for my first time to see if the
issue was related to a sample report.
In the report I ran on RHEL6.4 64 bit system with `--oval-results` flag, I
encountered the OVAL results more often than I did in the sample report,
but there were still some without any OVAL details, like
"rsyslog_send_messages_to_logserver"
and "accounts_password_pam_cracklib_dcredit".
It seems OVAL details show up when there are specific "items violating."
That is _incredibly_ helpful for a test like "rpm_verify_permissions". It's
awesome.
It seems OVAL details do not show up if the control just fails. That is
confusing. Didn't an OVAL criterion fail? Which one? Maybe there is more
than one criterion for a given test? The fact sometimes OVAL details show
up and sometimes they don't seems like yet another secret, undocumented
detail that is obvious to those in the know, but confusing to those that
aren't. At the very least, there should always be an explanation of the
OVAL detail even if it is "OVAL failed without any detail. Here is the
criterion that failed: "
> I'd be OK with default report ONLY showing details for the severity=high
> items that I failed and all other details went either into a second
report
We show check system details for all failed rules, always. This works for
both OVAL and SCE checks.
What I was trying to get at is that I think fails that are high severity
should be treated differently in the report than those of medium or high.
For example, they could their own section, or -- as I was musing -- be the
only fails included in the report.
Consider a scan result indicating 100 errors: 5 are high severity, 20 are
medium, and 75 low,
Speaking for myself, I am eager to fix the 5 that are high severity. I'll
glance at the 20 and the 75 to see if there is a relevant pattern (e.g.,
sub groupings), but generally speaking I am going to leave those for later.
The five I wanted listed so I can see the issues right away, what failed,
and how to fix. I could print that report and check items off. I would be
OK with having to add a flag to get details for the medium and low.
(Note: this is also a question of quantity. When I have a small number of
fails, I want to see them. When I have ten or more fails, I want a way of
grouping the issues.)
So, it would be helpful to get a report that by default listed out the
details of the 5 severe errors and just had a link to the others.
BTW - having the report shorter now is helpful! Thanks! I showed a
manageable 25 pages; small if printed double sided.
I'll try to mockup what I mean.
> I could open if I wanted, or the generic detail descriptions were on the
> WEB rather than part of the report.
Are you sure you want detailed reports of vulnerabilities on your
infrastructure to end up on the "WEB"? One of our goals is to keep the
reports and guides self sufficient. We do not want to rely on remote tools.
That's why we bundle even all the JavaScript and CSS.
No, I do not want results of what controls *my* system passed or failed
posted on the web.
I am speaking about the generic content that is a part of each control
being on the web:
rationale, links, related identifiers, remediation scripts, etc.
The most important information in the results of my scan, are the details
about my system and my scan results. Generic information is _generic_. I
only want that generic information co-located with details when it
specifically helps me take an action.
Maybe what I am driving at as a general theme here is the difference
between an "Action/Working Report" optimized to help me get to work
resolving issues and "Complete Report" that represents an artifact
capturing a state that is less transitory.
I'll try to mockup what I mean.
> The report could contain information about the system, but
I'm not sure
if
> there is a lot of value of all the (static) details being in the in
report
> by default.
Not sure what you mean by static details. The IP and MAC addresses? They
help identify the machine if hostname is not set properly. The CPE
platforms
should be there as well IMO as they help explain applicable and
not-applicable
results.
By static details I am referring to information in the SSG that does not
change from system to system or from scan result to scan result.
I feel I should say a bit more about this because the above may seem
contradictory.
It might seem I am one hand suggesting more information be added to the
report to make the report easier for beginners (e.g., include OVAL detail
on all failed tests).
On the other hand, it might seem I am suggesting less information be
included in the report (e.g., removing generic static details, or putting
medium and low control fails details in a second report).
I am arguing for clarity and less cognitively taxing presentation that does
way better than most other scan reports out there.
Only some fails having an OVAL detail section seems ambiguous. Meanwhile
including all explanatory detail for every item often seems extraneous.
If I knew the perfect algorithm, I would share it. I do know what I want to
read most of all in the report is : "The control ____ failed because
__________" and be able to follow a link to get more detail if I happen to
need it.
I think I really want a simple report that lists what failed and why, and
then a second report (or hidden second half of the report) that has way
more details.
If a control's one criterion failed, just tell me it in the summary and
don't make me look it up. If a control's multiple criterion failed, say
multiple items caused violation and link to detail. I can always look up
the full control details on the web since they are public after all. (And
if the controls is a custom, agency-only rule....well, include that detail
in the report since it is specific to the system/environment and not
generic.)
And yes, I'll work on a mockup...
Greg