----- Original Message -----
From: "Jan Lieskovsky" <jlieskov(a)redhat.com>
To: "Jan Cerny" <jcerny(a)redhat.com>
Cc: "SCAP Security Guide" <scap-security-guide(a)lists.fedorahosted.org>
Sent: Friday, September 11, 2015 11:39:40 AM
Subject: Re: A tool for support of SCAP content development
----- Original Message -----
> From: "Jan Cerny" <jcerny(a)redhat.com>
> To: "SCAP Security Guide"
> Sent: Thursday, June 4, 2015 9:15:42 AM
> Subject: A tool for support of SCAP content development
> Nowadays, there is almost no support for OVAL content development.
Agree in this point.
> The developers have to edit the XML files manually, there is no
> analysis tool, no debugger or any other tool that could make
> the content development faster and easier.
> I have started working on a new utility that will be able to
> debug OVAL checks, show step-by-step how they are evaluated,
> browse collected objects and system characteristics, communicate
> with probes, etc.
* is this new tool intended to become part of the openscap-* package,
or rather the plan being it to be a separate tool?,
It was decided to be as part of the openscap.
Official instructions approved by university says "implement this in the OpenSCAP
by form of editing existing source code."
For university instructions, see Description in Czech on this page:
I should definitely put the link on the wiki.
* above you speak about debugger -- is the plan to add also profiler?
It is not in my plan now, but thank your very much for your idea!
> I'm working on design of the utility now. I'm considering many
> alternatives, so I would like to ask you a few questions.
> * Can you describe most common problems that you have to face
> when you create the OVAL content?
Absence of ability to debug a particular OVAL check. Right now
it's possible to use 'oscap oval eval / oscap oval collect'
for particular rule to determine what got actually collected on
the system. This approach is working for simple rules. But it does
not scale properly (if the underlying OVAL object consist of another
OVAL variable, that variable using some OVAL functions on other
variables and so on, it's hard to determine which OVAL object failed
to be collected at the beginning -- which OVAL object is responsible
for the whole rule failure).
Next step is XCCDF evaluation - while the corresponding OVAL check might
work properly, it's possible (due to combination of "check",
"entity_check" attributes, order and / or logical operations between the
different OVAL criterions) the final XCCDF rules is still failing even
in the case "some" OVAL objects have been collected on the system.
Right now oscap provides 'oscap xccdf eval .. --oval-results' combination
alllowing to inspect, why the rule as a whole did not pass (e.g. wrong
"check" / "check_existence" attributes etc.).
But it would be good if the new tool would support XCCDF debugging too
(something like 'oscap xccdf eval ... --oval-results' is allowing now,
I am working now on enhancing the debug infos and introduce "--verbose" option
in oscap tool. I think it is a good first step.
> * How do you usually solve these problems?
See above. By separating the rule intended to be debugged. Then running:
* oscap oval eval --id
* oscap oval collect --id
* oscap xccdf eval .. --oval-results
combination of commands (depending on the level [e.g. if OVAL itself,
or at XCCDF level] the OVAL check is failing).
> * Can you tell me an example when did you run into issues with OVAL?
Basically any a bit more complex OVAL check needs to be designed in
stages (from top to bottom). Consider OVAL criterion requiring presence
just some OVAL object, this object to consist of multiple stages, e.g.:
* you first collect something on the system into object, save the
subexpression of it into <local_variable>,
* apply couple of functions to that variable, finally create new final
object from the last variable. This final object is then used in the
Right now it needs to be done from bottom to the top (at each level, at
each <local_variable> you mostly need to verify it returns appropriate
results). Other issue is OVAL standard reporting functionality. AFAICT
the standard requires the rule to return 'error' also in the case the
particular OVAL rule wasn't able to obtain child results (results for
the child objects compositing the final object). Having seen this 'error'
result for the first time, it's not clear if:
* OVAL check truly has a syntax error,
* or OVAL check is correct, but the engine wasn't able to determine value
for some of the OVAL variables above in the evaluation tree.
Another issue / RFE in 'oscap oval eval / collect' functionality is the
'--id' parameter. IIRC as of right now it supports specification of
an ID just for OVAL objects (therefore having OVAL variable manipulating
some previously collected OVAL object via some OVAL function, it's
impossible to see the retrieved values of that variable directly.
You need to see the values of the original OVAL object, then create
another OVAL object from that OVAL variable using the OVAL function,
to be able to see the true values, that OVAL variable is holding).
Not to speak to abstract. Consider the following:
<ind:textfilecontent54_object id="my_object" ..>
<some_oval_function_here, e.g. "split", "concat", whatever>
<object_component object_ref="my_object" />
Right now it's not possible to perform the following:
oscap oval collect --id "my_variable" --syschar /tmp/out
because "my_variable" doesn't constitute an OVAL object (therefore
to bypass it, it's necessary to define new artificial OVAL object
from that "my_variable" local variable, and try to collect it).
Therefore it would be very handy if 'collect' could be enhanced
to support variables too.
This is a nice use-case.
> * Imagine an ideal tool for an OVAL developer. What should it be
> able to do? What would be its features?
Maybe above being a sketch. From the above it's not clear if the
new tool should be just debugger, or profiler too. If the debugger,
basically it would be able to:
* report values of particular OVAL objects, variables at any time
of OVAL evaluation,
* report the branch OVAL check has been evaluated (like OVAL check
went through criterion #1, the rest of the criteria weren't taken
into account, since criterion #1 already succeeded),
I am going to achieve this by collecting all the possible information
during evaluating the content and processing those information to have
a meaningful output.
* + the requests as formulated above.
If we would talk also about profiling capabilities - it would be
able to report things like CPU usage in %, virtual memory usage,
and or count of system files processed when evaluating particular rule.
I would consider this, it can be useful feature to avoid bugs with
excessive memory consumption caused by badly written content.
This would be handy to optimize current form in many rules.
> I would appreciate any suggestions, remarks or other inputs.
> Thank you very much for your reply.
Should you need any other clarification wrt to some of the points
above, just ask.
Thank you for your input.
Thank you && Regards, Jan
Jan iankko Lieskovsky / Red Hat Security Technologies Team
> Jan Černý
> Security Technologies | Red Hat, Inc.
> SCAP Security Guide mailing list