We've had very similar conversations on the CEE front.
The issue is is that we want to keep things as flat as possible. For
me, one big problem with XML and JSON is that it is fine when you have
control over both ends. When you give people a framework using XML or
JSON, they may abuse the nesting.
What we wanted to do is to build something flat or with as minimal
nesting as possible, but do not want to invent yet another parsing
language. This gets into issues of library support, encodings, escape
characters, etc. I don't know if the right option is to define a new
language like "min-json" that leverages JSON but forbids JSON objects
from appearing anywhere but the document level...?
The problem with that is that as soon as people see that it is
compatible with JSON/XML, they want to leverage all of the features of
JSON/XML.
There are two features that will impact this decision:
1. Support for multi-valued fields. E.g., "dest_ip":["1.2.3.4","10.0.0.2"]
2. Support for multiple namespaces (see my previous e-mail to this
list on that topic)
In the CEE discussions, we felt that supporting multi-valued fields to
be important and have been kicking around different solutions to the
namespace problem. (The CEE Profile attempts to solve the multiple
namespaces by using externally defined XML documents, keeping the log
structure simple)
On Fri, Mar 9, 2012 at 7:47 AM, Brian Knox <briank(a)talksum.com> wrote:
> "I strongly disagree here. Parsing and constructing nested structures
> isn't all that hard, especially as there are usable JSON parsing and
> constructing libraries around that can already do this."
>
> I'd like to comment on this statement specifically. First off, I understand
> the desire to have nested structures in the logs. With nested structures, a
> log line can carry more semantic information than with a flat list.
> However, I think there's another consideration that needs to be taken into
> account other than the human work that needs to be done to parse nested
> structures - there is also the CPU work that needs to be done.
>
> At one of my previous jobs, we were handling around 6 billion log lines a
> day, and the large majority of those logs were in JSON format. In the
> research we did, we found that the complexity of the JSON structure could
> have a very significant impact on the number of log lines per second we
> could handle in systems that needed to parse the logs. In one case, a
> reworking of our JSON structure was the difference between ~ 50,000 messages
> a second per core and ~ 250,000 messages a second per core.
>
> To be clear: I'm not arguing to always side with performance when making a
> decision between performance and meaningful structure. I am saying that the
> performance impact of decisions made does need to at least be taken into
> consideration, for those of us aggregating high volumes of logs.
>
> Brian
>
>
<snip>