[lumberjack] New Syntax Spec and updated Field list

david at lang.hm david at lang.hm
Wed May 2 20:18:02 UTC 2012


On Wed, 2 May 2012, Gergely Nagy wrote:

> William Heinbockel <wheinbockel at gmail.com> writes:
>
>> What I did was take the JSON field types and add datetime, ipv4, and ipv6 types.
>> Then made a basic conversion from JSON --> XML and tried to align the
>> libumberlog syntaxes with Keith's XML.
>> I also added some notes about the "structured" vs. "inline" field
>> naming.
>
> Sorry for the late reply, I've been swamped as of late. Nevertheless, my
> former opinion that the inline representation is a very, very bad thing
> still stands:
>
> First of all, JSON is perfectly capable of representing structured data
> (and the syntax format already describes that too!), bolting something
> like the inline representation on it is both counter-intuitive for those
> who worked with JSON, it's incompatible with anything that is not
> explicitly aware of it: you can't just forward it to a database that
> happens to accept JSON as-is, you'll have to translate the inline
> representation into proper JSON structure. (For example, if I had a
> producer that spew out CEE-JSON on one end, and an ElasticSearch indexer
> on the other end, I'd need to convert inline-syntax, which would pretty
> much destroy the whole idea of having a thin layer.)
>
> It adds quite a bit of burden on the implementation side, and I still
> can't see what benefit it would bring, apart from saving a few bytes on
> the wire. I'm not entirely sure it would save bytes, either, by the
> way.. For example:
>
> {"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
> {"foo":{"field1":1,"field2":2,"field3":3}}
>
> Proper structuring is shorteer pretty much the moment you have more than
> two fields with the same 'object' prefix.
>
> Furthermore, as mentioned above already, anything that speaks JSON,
> understands the structures. None of them understand the inline
> syntax. It's alien to JSON as much as trying to bolt on new types were.
>
> It complicates both the sender side (do I want to send stuff as proper
> JSON, or with inline representation?), it complicates things on the
> consumer side too (do I need to expand inline syntax? if so, it will
> likely turn into dotted-notation, and then how do I reassemble it back
> if it needs to be forwarded?), and adds extra work on every level.

I think that the only good reason for doing having the inline stuff is for 
a template based API. Over the wire it would be far better for everything 
to be nested JSON so that all recipients only need to know JSON.

But when creating an API to generate the logs, figuring out how to specify 
structure is non-trivial and I can see the desire to be able to specify 
the fully qualified name of a field.

David Lang


More information about the lumberjack-developers mailing list