[lumberjack] New Syntax Spec and updated Field list

Balazs Scheidler bazsi77 at gmail.com
Wed May 2 21:02:07 UTC 2012


----- Original message -----
> On Wed, 2 May 2012, Gergely Nagy wrote:
> 
> > William Heinbockel <wheinbockel at gmail.com> writes:
> > 
> > > What I did was take the JSON field types and add datetime, ipv4, and
> > > ipv6 types. Then made a basic conversion from JSON --> XML and tried
> > > to align the libumberlog syntaxes with Keith's XML.
> > > I also added some notes about the "structured" vs. "inline" field
> > > naming.
> > 
> > Sorry for the late reply, I've been swamped as of late. Nevertheless,
> > my former opinion that the inline representation is a very, very bad
> > thing still stands:
> > 
> > First of all, JSON is perfectly capable of representing structured data
> > (and the syntax format already describes that too!), bolting something
> > like the inline representation on it is both counter-intuitive for
> > those who worked with JSON, it's incompatible with anything that is not
> > explicitly aware of it: you can't just forward it to a database that
> > happens to accept JSON as-is, you'll have to translate the inline
> > representation into proper JSON structure. (For example, if I had a
> > producer that spew out CEE-JSON on one end, and an ElasticSearch
> > indexer on the other end, I'd need to convert inline-syntax, which
> > would pretty much destroy the whole idea of having a thin layer.)
> > 
> > It adds quite a bit of burden on the implementation side, and I still
> > can't see what benefit it would bring, apart from saving a few bytes on
> > the wire. I'm not entirely sure it would save bytes, either, by the
> > way.. For example:
> > 
> > {"foo!field1":1,"foo!field2":2,"foo!field3":3} vs
> > {"foo":{"field1":1,"field2":2,"field3":3}}
> > 
> > Proper structuring is shorteer pretty much the moment you have more
> > than two fields with the same 'object' prefix.
> > 
> > Furthermore, as mentioned above already, anything that speaks JSON,
> > understands the structures. None of them understand the inline
> > syntax. It's alien to JSON as much as trying to bolt on new types were.
> > 
> > It complicates both the sender side (do I want to send stuff as proper
> > JSON, or with inline representation?), it complicates things on the
> > consumer side too (do I need to expand inline syntax? if so, it will
> > likely turn into dotted-notation, and then how do I reassemble it back
> > if it needs to be forwarded?), and adds extra work on every level.
> 
> I think that the only good reason for doing having the inline stuff is
> for   a template based API. Over the wire it would be far better for
> everything   to be nested JSON so that all recipients only need to know
> JSON.
> 
> But when creating an API to generate the logs, figuring out how to
> specify   structure is non-trivial and I can see the desire to be able to
> specify   the fully qualified name of a field.
> 

I can understand this desire, however this is what the dotted notation is good for. E.g. if it's simpler to use a flat namespace in an API, use that. the mapping between the json structure is quite standard.

In syslog-ng I'm planning to use dotted notation when working with fields as syslog-ng has a flat structure in its event model, but then generate nested json at the end.

The same can be done in any logging API.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://fedorahosted.org/pipermail/lumberjack-developers/attachments/20120502/8abc6019/attachment-0001.html>


More information about the lumberjack-developers mailing list