[lumberjack] New Syntax Spec and updated Field list

Dmitri Pal dpal at redhat.com
Wed May 2 22:02:01 UTC 2012


On 05/02/2012 05:17 PM, david at lang.hm wrote:
> On Wed, 2 May 2012, Balazs Scheidler wrote:
>
>> ----- Original message -----
>>> On Wed, 2 May 2012, Gergely Nagy wrote:
>>>
>>>> William Heinbockel <wheinbockel at gmail.com> writes:
>>>>
>>>>> What I did was take the JSON field types and add datetime, ipv4, and
>>>>> ipv6 types. Then made a basic conversion from JSON --> XML and tried
>>>>> to align the libumberlog syntaxes with Keith's XML.
>>>>> I also added some notes about the "structured" vs. "inline" field
>>>>> naming.
>>>>
>>>> Sorry for the late reply, I've been swamped as of late. Nevertheless,
>>>> my former opinion that the inline representation is a very, very bad
>>>> thing still stands:
>>>>
>>>> First of all, JSON is perfectly capable of representing structured
>>>> data
>>>> (and the syntax format already describes that too!), bolting something
>>>> like the inline representation on it is both counter-intuitive for
>>>> those who worked with JSON, it's incompatible with anything that is
>>>> not
>>>> explicitly aware of it: you can't just forward it to a database that
>>>> happens to accept JSON as-is, you'll have to translate the inline
>>>> representation into proper JSON structure. (For example, if I had a
>>>> producer that spew out CEE-JSON on one end, and an ElasticSearch
>>>> indexer on the other end, I'd need to convert inline-syntax, which
>>>> would pretty much destroy the whole idea of having a thin layer.)
>>>>
>>>> It adds quite a bit of burden on the implementation side, and I still
>>>> can't see what benefit it would bring, apart from saving a few
>>>> bytes on
>>>> the wire. I'm not entirely sure it would save bytes, either, by the
>>>> way.. For example:
>>>>
>>>> {"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
>>>> {"foo":{"field1":1,"field2":2,"field3":3}}
>>>>
>>>> Proper structuring is shorteer pretty much the moment you have more
>>>> than two fields with the same 'object' prefix.
>>>>
>>>> Furthermore, as mentioned above already, anything that speaks JSON,
>>>> understands the structures. None of them understand the inline
>>>> syntax. It's alien to JSON as much as trying to bolt on new types
>>>> were.
>>>>
>>>> It complicates both the sender side (do I want to send stuff as proper
>>>> JSON, or with inline representation?), it complicates things on the
>>>> consumer side too (do I need to expand inline syntax? if so, it will
>>>> likely turn into dotted-notation, and then how do I reassemble it back
>>>> if it needs to be forwarded?), and adds extra work on every level.
>>>
>>> I think that the only good reason for doing having the inline stuff is
>>> for   a template based API. Over the wire it would be far better for
>>> everything   to be nested JSON so that all recipients only need to know
>>> JSON.
>>>
>>> But when creating an API to generate the logs, figuring out how to
>>> specify   structure is non-trivial and I can see the desire to be
>>> able to
>>> specify   the fully qualified name of a field.
>>>
>>
>> I can understand this desire, however this is what the dotted
>> notation is good for. E.g. if it's simpler to use a flat namespace in
>> an API, use that. the mapping between the json structure is quite
>> standard.
>>
>> In syslog-ng I'm planning to use dotted notation when working with
>> fields as syslog-ng has a flat structure in its event model, but then
>> generate nested json at the end.
>>
>> The same can be done in any logging API.
>
> is the only difference between the two notations the fact that the
> serparator character is '!' instead of '.'?
>
> If so, I don't have any strong preference betwen them, but I would
> think that '.' would be more likley to conflict with other uses.

Yes that was the only reason for picking '!'. I came across conflicts
when field name has to have '.'.
I might envision that host name might become a field name in some case
for example. 
'!" looks much safer to me.

>
> David Lang
>
>
> _______________________________________________
> lumberjack-developers mailing list
> lumberjack-developers at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/lumberjack-developers


-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager IPA project,
Red Hat Inc.


-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://fedorahosted.org/pipermail/lumberjack-developers/attachments/20120502/659d8b3c/attachment-0001.html>


More information about the lumberjack-developers mailing list