+1 on work to support OVAL authoring.

Greg Elin

On Mon, Sep 14, 2015 at 3:17 AM, Jan Cerny <jcerny@redhat.com> wrote:

----- Original Message -----
> From: "Jan Lieskovsky" <jlieskov@redhat.com>
> To: "Jan Cerny" <jcerny@redhat.com>
> Cc: "SCAP Security Guide" <scap-security-guide@lists.fedorahosted.org>
> Sent: Friday, September 11, 2015 11:39:40 AM
> Subject: Re: A tool for support of SCAP content development
>  Hello Jan,
> ----- Original Message -----
> > From: "Jan Cerny" <jcerny@redhat.com>
> > To: "SCAP Security Guide" <scap-security-guide@lists.fedorahosted.org>
> > Sent: Thursday, June 4, 2015 9:15:42 AM
> > Subject: A tool for support of SCAP content development
> >
> > Hello,
> >
> > Nowadays, there is almost no support for OVAL content development.
> Agree in this point.
> > The developers have to edit the XML files manually, there is no
> > analysis tool, no debugger or any other tool that could make
> > the content development faster and easier.
> > I have started working on a new utility that will be able to
> > debug OVAL checks, show step-by-step how they are evaluated,
> > browse collected objects and system characteristics, communicate
> > with probes, etc.
> Two questions:
> * is this new tool intended to become part of the openscap-* package,
>   or rather the plan being it to be a separate tool?,

It was decided to be as part of the openscap.
Official instructions approved by university says "implement this in the OpenSCAP project
by form of editing existing source code."
For university instructions, see Description in Czech on this page:
I should definitely put the link on the wiki.

> * above you speak about debugger -- is the plan to add also profiler?

It is not in my plan now, but thank your very much for your idea!

> > I'm working on design of the utility now. I'm considering many
> > alternatives, so I would like to ask you a few questions.
> > * Can you describe most common problems that you have to face
> >   when you create the OVAL content?
> Absence of ability to debug a particular OVAL check. Right now
> it's possible to use 'oscap oval eval / oscap oval collect'
> for particular rule to determine what got actually collected on
> the system. This approach is working for simple rules. But it does
> not scale properly (if the underlying OVAL object consist of another
> OVAL variable, that variable using some OVAL functions on other
> variables and so on, it's hard to determine which OVAL object failed
> to be collected at the beginning -- which OVAL object is responsible
> for the whole rule failure).
> Next step is XCCDF evaluation - while the corresponding OVAL check might
> work properly, it's possible (due to combination of "check",
> "check_existence",
> "entity_check" attributes, order and / or logical operations between the
> different OVAL criterions) the final XCCDF rules is still failing even
> in the case "some" OVAL objects have been collected on the system.
> Right now oscap provides 'oscap xccdf eval .. --oval-results' combination
> alllowing to inspect, why the rule as a whole did not pass (e.g. wrong
> "check" / "check_existence" attributes etc.).
> But it would be good if the new tool would support XCCDF debugging too
> (something like 'oscap xccdf eval ... --oval-results' is allowing now,
> but enhanced).

I am working now on enhancing the debug infos and introduce "--verbose" option
in oscap tool. I think it is a good first step.

> > * How do you usually solve these problems?
> See above. By separating the rule intended to be debugged. Then running:
> * oscap oval eval --id
> * oscap oval collect --id
> * oscap xccdf eval .. --oval-results
> combination of commands (depending on the level [e.g. if OVAL itself,
> or at XCCDF level] the OVAL check is failing).
> > * Can you tell me an example when did you run into issues with OVAL?
> Basically any a bit more complex OVAL check needs to be designed in
> stages (from top to bottom). Consider OVAL criterion requiring presence
> just some OVAL object, this object to consist of multiple stages, e.g.:
> * you first collect something on the system into object, save the
>   subexpression of it into <local_variable>,
> * apply couple of functions to that variable, finally create new final
>   object from the last variable. This final object is then used in the
>   OVAL test.
> Right now it needs to be done from bottom to the top (at each level, at
> each <local_variable> you mostly need to verify it returns appropriate
> results). Other issue is OVAL standard reporting functionality. AFAICT
> the standard requires the rule to return 'error' also in the case the
> particular OVAL rule wasn't able to obtain child results (results for
> the child objects compositing the final object). Having seen this 'error'
> result for the first time, it's not clear if:
> * OVAL check truly has a syntax error,
> * or OVAL check is correct, but the engine wasn't able to determine value
>   for some of the OVAL variables above in the evaluation tree.
> Another issue / RFE in 'oscap oval eval / collect' functionality is the
> '--id' parameter. IIRC as of right now it supports specification of
> an ID just for OVAL objects (therefore having OVAL variable manipulating
> some previously collected OVAL object via some OVAL function, it's
> impossible to see the retrieved values of that variable directly.
> You need to see the values of the original OVAL object, then create
> another OVAL object from that OVAL variable using the OVAL function,
> to be able to see the true values, that OVAL variable is holding).
> Not to speak to abstract. Consider the following:
> <ind:textfilecontent54_object id="my_object" ..>
>   <ind:filepath>..</ind:filepath>
>   <ind:pattern ..>..</ind:pattern>
>   <ind:instance ..>..</ind:instance>
> </ind:textfilecontent54_object>
> <local_variable id="my_variable>
>   <some_oval_function_here, e.g. "split", "concat", whatever>
>     <object_component object_ref="my_object" />
>     ..
>   </some_oval_function_here>
> </local_variable>
> Right now it's not possible to perform the following:
>   oscap oval collect --id "my_variable" --syschar /tmp/out
> because "my_variable" doesn't constitute an OVAL object (therefore
> to bypass it, it's necessary to define new artificial OVAL object
> from that "my_variable" local variable, and try to collect it).
> Therefore it would be very handy if 'collect' could be enhanced
> to support variables too.

This is a nice use-case.

> > * Imagine an ideal tool for an OVAL developer. What should it be
> >   able to do? What would be its features?
> Maybe above being a sketch. From the above it's not clear if the
> new tool should be just debugger, or profiler too. If the debugger,
> basically it would be able to:
> * report values of particular OVAL objects, variables at any time
>   of OVAL evaluation,
> * report the branch OVAL check has been evaluated (like OVAL check
>   went through criterion #1, the rest of the criteria weren't taken
>   into account, since criterion #1 already succeeded),

I am going to achieve this by collecting all the possible information
during evaluating the content and processing those information to have
a meaningful output.

> * + the requests as formulated above.

> If we would talk also about profiling capabilities - it would be
> able to report things like CPU usage in %, virtual memory usage,
> and or count of system files processed when evaluating particular rule.

I would consider this, it can be useful feature to avoid bugs with
excessive memory consumption caused by badly written content.

> This would be handy to optimize current form in many rules.
> > I would appreciate any suggestions, remarks or other inputs.
> > Thank you very much for your reply.
> Should you need any other clarification wrt to some of the points
> above, just ask.

Thank you for your input.

Jan Černý

> Thank you && Regards, Jan
> --
> Jan iankko Lieskovsky / Red Hat Security Technologies Team
> >
> > Regards
> >
> > Jan Černý
> > Security Technologies | Red Hat, Inc.
> > --
> > SCAP Security Guide mailing list
> > scap-security-guide@lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide
> > https://github.com/OpenSCAP/scap-security-guide/
SCAP Security Guide mailing list