[lumberjack] New Syntax Spec and updated Field list

david at lang.hm david at lang.hm
Wed May 2 21:17:25 UTC 2012


On Wed, 2 May 2012, Balazs Scheidler wrote:

> ----- Original message -----
>> On Wed, 2 May 2012, Gergely Nagy wrote:
>>
>>> William Heinbockel <wheinbockel at gmail.com> writes:
>>>
>>>> What I did was take the JSON field types and add datetime, ipv4, and
>>>> ipv6 types. Then made a basic conversion from JSON --> XML and tried
>>>> to align the libumberlog syntaxes with Keith's XML.
>>>> I also added some notes about the "structured" vs. "inline" field
>>>> naming.
>>>
>>> Sorry for the late reply, I've been swamped as of late. Nevertheless,
>>> my former opinion that the inline representation is a very, very bad
>>> thing still stands:
>>>
>>> First of all, JSON is perfectly capable of representing structured data
>>> (and the syntax format already describes that too!), bolting something
>>> like the inline representation on it is both counter-intuitive for
>>> those who worked with JSON, it's incompatible with anything that is not
>>> explicitly aware of it: you can't just forward it to a database that
>>> happens to accept JSON as-is, you'll have to translate the inline
>>> representation into proper JSON structure. (For example, if I had a
>>> producer that spew out CEE-JSON on one end, and an ElasticSearch
>>> indexer on the other end, I'd need to convert inline-syntax, which
>>> would pretty much destroy the whole idea of having a thin layer.)
>>>
>>> It adds quite a bit of burden on the implementation side, and I still
>>> can't see what benefit it would bring, apart from saving a few bytes on
>>> the wire. I'm not entirely sure it would save bytes, either, by the
>>> way.. For example:
>>>
>>> {"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
>>> {"foo":{"field1":1,"field2":2,"field3":3}}
>>>
>>> Proper structuring is shorteer pretty much the moment you have more
>>> than two fields with the same 'object' prefix.
>>>
>>> Furthermore, as mentioned above already, anything that speaks JSON,
>>> understands the structures. None of them understand the inline
>>> syntax. It's alien to JSON as much as trying to bolt on new types were.
>>>
>>> It complicates both the sender side (do I want to send stuff as proper
>>> JSON, or with inline representation?), it complicates things on the
>>> consumer side too (do I need to expand inline syntax? if so, it will
>>> likely turn into dotted-notation, and then how do I reassemble it back
>>> if it needs to be forwarded?), and adds extra work on every level.
>>
>> I think that the only good reason for doing having the inline stuff is
>> for   a template based API. Over the wire it would be far better for
>> everything   to be nested JSON so that all recipients only need to know
>> JSON.
>>
>> But when creating an API to generate the logs, figuring out how to
>> specify   structure is non-trivial and I can see the desire to be able to
>> specify   the fully qualified name of a field.
>>
>
> I can understand this desire, however this is what the dotted notation is good for. E.g. if it's simpler to use a flat namespace in an API, use that. the mapping between the json structure is quite standard.
>
> In syslog-ng I'm planning to use dotted notation when working with fields as syslog-ng has a flat structure in its event model, but then generate nested json at the end.
>
> The same can be done in any logging API.

is the only difference between the two notations the fact that the 
serparator character is '!' instead of '.'?

If so, I don't have any strong preference betwen them, but I would think 
that '.' would be more likley to conflict with other uses.

David Lang


More information about the lumberjack-developers mailing list