[lumberjack] New Syntax Spec and updated Field list

Gergely Nagy algernon at balabit.hu
Wed May 2 08:21:53 UTC 2012


William Heinbockel <wheinbockel at gmail.com> writes:

> What I did was take the JSON field types and add datetime, ipv4, and ipv6 types.
> Then made a basic conversion from JSON --> XML and tried to align the
> libumberlog syntaxes with Keith's XML.
> I also added some notes about the "structured" vs. "inline" field
> naming.

Sorry for the late reply, I've been swamped as of late. Nevertheless, my
former opinion that the inline representation is a very, very bad thing
still stands:

First of all, JSON is perfectly capable of representing structured data
(and the syntax format already describes that too!), bolting something
like the inline representation on it is both counter-intuitive for those
who worked with JSON, it's incompatible with anything that is not
explicitly aware of it: you can't just forward it to a database that
happens to accept JSON as-is, you'll have to translate the inline
representation into proper JSON structure. (For example, if I had a
producer that spew out CEE-JSON on one end, and an ElasticSearch indexer
on the other end, I'd need to convert inline-syntax, which would pretty
much destroy the whole idea of having a thin layer.)

It adds quite a bit of burden on the implementation side, and I still
can't see what benefit it would bring, apart from saving a few bytes on
the wire. I'm not entirely sure it would save bytes, either, by the
way.. For example:

{"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
{"foo":{"field1":1,"field2":2,"field3":3}}

Proper structuring is shorteer pretty much the moment you have more than
two fields with the same 'object' prefix.

Furthermore, as mentioned above already, anything that speaks JSON,
understands the structures. None of them understand the inline
syntax. It's alien to JSON as much as trying to bolt on new types were.

It complicates both the sender side (do I want to send stuff as proper
JSON, or with inline representation?), it complicates things on the
consumer side too (do I need to expand inline syntax? if so, it will
likely turn into dotted-notation, and then how do I reassemble it back
if it needs to be forwarded?), and adds extra work on every level.

Granted, parsing nested structures is a bit more code, but seriously,
there are hundreds of libraries out there to work with JSON, multiple
libraries for pretty much every single language, and all of them can
handle nested structures just fine. There is hardly any extra work that
needs be done.

I don't buy the 'compatibility between syntaxes' thing, either. That's
what dot-notation is for, and it needs not be present on the wire.

Another thing that I don't like is that field names are much more
limited than JSON's limitations, and while allowing the full range JSON
allows is probably not a good idea, I'd at least add "-" to the allowed
set.

-- 
|8]



More information about the lumberjack-developers mailing list