On Mon, Sep 1, 2014 at 5:32 AM, Martin Preisler <mpreisle@redhat.com> wrote:
----- Original Message -----
> From: "Greg Elin" <gregelin@gitmachines.com>
> To: "SCAP Security Guide" <scap-security-guide@lists.fedorahosted.org>
> Sent: Friday, August 29, 2014 11:18:37 PM
> Subject: Re: New report and guide in openscap 1.1.0
>
> I agree with Trey. The details would be more useful if they contained more
> information about how something failed. I would expect that in a detail.

As I said in my previous email, we show these details. We always have!

They were not present in one of the previous ! draft ! versions of
XCCDF report. When I released that draft I explicitly said that check details
are not there but will be there in the final version. Are you sure you are
looking at the new report?

Thanks for the prodding. I looked more closely last night. At least two of us had some confusion with this. 

AFAIK, Oval results only show up if the `--oval-results` is added to `oscap xccdf eval` command. It seems to me specific errors should be shared by default and flag used to hide them. Yes, it is in the man page and of course we all read the man page... (But I guess that is a comment for the OpenSCAP List?)

Oval details show up on only some failed tests in the sample report (https://mpreisle.fedorapeople.org/openscap/1.1.0_xslt/report.html) "package_aide_installed" for example does not show any oval details though it fails while "service_atd_disabled" does show oval results. 

Last night I built the openSCAP from source for my first time to see if the issue was related to a sample report.  

In the report I ran on RHEL6.4 64 bit system with `--oval-results` flag, I encountered the OVAL results more often than I did in the sample report, but there were still some without any OVAL details, like "rsyslog_send_messages_to_logserver" and "accounts_password_pam_cracklib_dcredit".

It seems OVAL details show up when there are specific "items violating." That is _incredibly_ helpful for a test like "rpm_verify_permissions". It's awesome.

It seems OVAL details do not show up if the control just fails. That is confusing. Didn't an OVAL criterion fail? Which one? Maybe there is more than one criterion for a given test? The fact sometimes OVAL details show up and sometimes they don't seems like yet another secret, undocumented detail that is obvious to those in the know, but confusing to those that aren't.  At the very least, there should always be an explanation of the OVAL detail even if it is "OVAL failed without any detail. Here is the criterion that failed: "

 

> I'd be OK with default report ONLY showing details for the severity=high
> items that I failed and all other details went either into a second report

We show check system details for all failed rules, always. This works for
both OVAL and SCE checks.

What I was trying to get at is that I think fails that are high severity should be treated differently in the report than those of medium or high. For example, they could their own section, or -- as I was musing -- be the only fails included in the report. 

Consider a scan result indicating 100 errors: 5 are high severity, 20 are medium, and 75 low, 

Speaking for myself, I am eager to fix the 5 that are high severity. I'll glance at the 20 and the 75 to see if there is a relevant pattern (e.g., sub groupings), but generally speaking I am going to leave those for later.  

The five I wanted listed so I can see the issues right away, what failed, and how to fix. I could print that report and check items off. I would be OK with having to add a flag to get details for the medium and low. 

(Note: this is also a question of quantity. When I have a small number of fails, I want to see them. When I have ten or more fails, I want a way of grouping the issues.)

So, it would be helpful to get a report that by default listed out the details of the 5 severe errors and just had a link to the others. 

BTW - having the report shorter now is helpful! Thanks! I showed a manageable 25 pages; small if printed double sided.

I'll try to mockup what I mean.

 

> I could open if I wanted, or the generic detail descriptions were on the
> WEB rather than part of the report.

Are you sure you want detailed reports of vulnerabilities on your
infrastructure to end up on the "WEB"? One of our goals is to keep the
reports and guides self sufficient. We do not want to rely on remote tools.
That's why we bundle even all the JavaScript and CSS.


No, I do not want results of what controls *my* system passed or failed posted on the web. 

I am speaking about the generic content that is a part of each control being on the web: 
 rationale, links, related identifiers, remediation scripts, etc. 

The most important information in the results of my scan, are the details about my system and my scan results. Generic information is _generic_. I only want that generic information co-located with details when it specifically helps me take an action. 

Maybe what I am driving at as a general theme here is the difference between an "Action/Working Report" optimized to help me get to work resolving issues and "Complete Report" that represents an artifact capturing a state that is less transitory. 

I'll try to mockup what I mean.

 
> The report could contain information about the system, but I'm not sure if
> there is a lot of value of all the (static) details being in the in report
> by default.

Not sure what you mean by static details. The IP and MAC addresses? They
help identify the machine if hostname is not set properly. The CPE platforms
should be there as well IMO as they help explain applicable and not-applicable
results.

By static details I am referring to information in the SSG that does not change from system to system or from scan result to scan result. 

I feel I should say a bit more about this because the above may seem contradictory.

It might seem I am one hand suggesting more information be added to the report to make the report easier for beginners (e.g., include OVAL detail on all failed tests). 

On the other hand, it might seem I am suggesting less information be included in the report (e.g., removing generic static details, or putting medium and low control fails details in a second report).

I am arguing for clarity and less cognitively taxing presentation that does way better than most other scan reports out there. 

Only some fails having an OVAL detail section seems ambiguous. Meanwhile including all explanatory detail for every item often seems extraneous. 

If I knew the perfect algorithm, I would share it. I do know what I want to read most of all in the report is :  "The control ____ failed because __________" and be able to follow a link to get more detail if I happen to need it. 

I think I really want a simple report that lists what failed and why, and then a second report (or hidden second half of the report) that has way more details. 

If a control's one criterion failed, just tell me it in the summary and don't make me look it up. If a control's multiple criterion failed, say multiple items caused violation and link to detail. I can always look up the full control details on the web since they are public after all. (And if the controls is a custom, agency-only rule....well, include that detail in the report since it is specific to the system/environment and not generic.)

And yes, I'll work on a mockup...

Greg