On Tue, 27 Mar 2012, William Heinbockel wrote:
On Mar 27, 2012 4:17 PM, david@lang.hm wrote:
But does this mean that unless applications are re-written, they will
never send structured logs at all?
There are only two ways to have applications log with structured data:
- Update the app to send name-value pairs
- Use something like patterndb to convert the unstructured data
If the app wants to leverage structured data, they will already have to make some updates. We just offer them a choice to either use the minimal umberlog interface or convert to a more full featured one.
I don't expect the applications that are writing the logs to care about structured data, they generally don't need it (as it's just as easy for them to do their own thing, they know how to understand it)
The big benifit of structured logs is to the people consuming the logs, the organizations running the applications.
If they cannot benefit until the programs are re-written, then I think that a lot of the value of structured logging is lost.
I see there being value in having the cee-syslog replacement library that sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
This is continuing on this subject.
Today the syslog messages have almost no structure, most of the data in the log is generated by the application (and so things like uid, pid, etc cannot be trusted as they are just what the app claims that they are)
going to structured logging without any app changes makes the metadata in the resulting logs more trustworthy, and allows for a lot of additional metadata to be put in the logs with no application changes.
This could be done by configuring things in the openlog call, but that requries changes to every application.
I think it would be better to have the serialization format be determined automatically by the library if possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a capabilities message.
The client can ignore this message and send traditional formatted syslog data to the server
Or, the client can read the capabilities message and then pick which supported format (potentially with compression options) to use to send data to the server.
This approach lets the format be anything that the library supports, under the control of the sysadmin of the system (by configuring what capabilities the syslog daemon advertises) without the application programmer needing to know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a solution into existing/legacy capabilities. We should approach this as a two-pronged solution:
- Backwards-compatible structured log implementation that works with
the syslog API and syslog protocol (this is umberlog). This should be as minimal and compatible as possible. I commend Gergely on his implementation
- Implement a new event API frontend and improved logging
capabilities. This should be able to work with syslogd, but does not have any requirements to be able to work with the existing syslog API, syslog protocol, /dev/log, etc.
Adding a new application API to generate more structured logs is a good idea. But why do we need to create a completely new interface between the logging library and syslog instead of extending the existing interface?
You don't know what you are going to break and you restrict yourself to the limitations of the current interface. It is not that I am against it, just if we want to improve it it might be best to redo it entirely.
Also, remember that there is more than just syslog and user apps as potential customers. There is also the kernel, klogd, and auditd.
these are almost exclusivly additional log sources that generate logs, not consume them.
David Lang
_______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers