On 03/27/2012 06:17 PM, William Heinbockel wrote:


On Mar 27, 2012 4:17 PM, <david@lang.hm> wrote:
>
> On Tue, 27 Mar 2012, William Heinbockel wrote:
>
>> On Tue, Mar 27, 2012 at 3:18 PM,  <david@lang.hm> wrote:
>>>
>>> I posted some of this at the bottom of a long message yesterday that nobody
>>> replied to, so I broke it out to a separate thread
>>>
>>>
>>> The traditional syslog interface has been a write-only interface, with the
>>> only feedback being that the write could block.
>>>
>>> With the new formats and new syslog replacement library, does there need to
>>> be some rethinking of this interface?
>>>
>>> JSON can be pushed through the traditional syslog interface with no changes
>>> (which is one of the attractions of using JSON), but when we support
>>> additional formats, especially ones with type information, we need to have a
>>> way to tell what format to use when writing the log.
>>
>>
>> I don't think this is right approach.
>> The end goal is to support structured log messages.
>> The first step is to introduce the capability to sent structured
>> events via existing capabilities (read: JSON over Syslog)
>>
>> The next steps would be the ELAPI/selog API that would introduce a new
>> logging interface, probably maintaining at least minimal compatibility
>> with the syslog daemon (not the syslog API or syslog protocol). This
>> would allow us to define and support more feature-rich log protocols
>> and other features.
>
>
> But does this mean that unless applications are re-written, they will never send structured logs at all?

There are only two ways to have applications log with structured data:
1. Update the app to send name-value pairs
2. Use something like patterndb to convert the unstructured data

If the app wants to leverage structured data, they will already have to make some updates. We just offer them a choice to either use the minimal umberlog interface or convert to a more full featured one.
>
> I see there being value in having the cee-syslog replacement library that sends the metadata structured around the existing unstructured log data. It's not as good as a full structured log, but it's a lot better than the existing situation.
>
>
>>>
>>> This could be done by configuring things in the openlog call, but that
>>> requries changes to every application.
>>>
>>> I think it would be better to have the serialization format be determined
>>> automatically by the library if possible.
>>>
>>> One way to do this is to have a handshake.
>>>
>>> When a app opens /dev/log, the syslog server sends a capabilities message.
>>>
>>> The client can ignore this message and send traditional formatted syslog
>>> data to the server
>>>
>>> Or, the client can read the capabilities message and then pick which
>>> supported format (potentially with compression options) to use to send data
>>> to the server.
>>>
>>> This approach lets the format be anything that the library supports, under
>>> the control of the sysadmin of the system (by configuring what capabilities
>>> the syslog daemon advertises) without the application programmer needing to
>>> know anything about these formats.
>>>
>>>
>>> thoughts?
>>>
>>
>> Let's not get ahead of ourselves and try to shoehorn a solution into
>> existing/legacy capabilities. We should approach this as a two-pronged
>> solution:
>>
>> 1. Backwards-compatible structured log implementation that works with
>> the syslog API and syslog protocol (this is umberlog). This should be
>> as minimal and compatible as possible. I commend Gergely on his
>> implementation
>>
>> 2. Implement a new event API frontend and improved logging
>> capabilities. This should be able to work with syslogd, but does not
>> have any requirements to be able to work with the existing syslog API,
>> syslog protocol, /dev/log, etc.
>
>
> Adding a new application API to generate more structured logs is a good idea. But why do we need to create a completely new interface between the logging library and syslog instead of extending the existing interface?

You don't know what you are going to break and you restrict yourself to the limitations of the current interface.
It is not that I am against it, just if we want to improve it it might be best to redo it entirely.

Also, remember that there is more than just syslog and user apps as potential customers. There is also the kernel, klogd, and auditd.

William Heinbockel


With the current plan we ask application developers to change things twice. First to new syslog umberlog interface and then later to a better interface. IMO this is wrong. I tried to rise the concerns about this several times but have not been heard.
IMO we need to define one interface that the application developers would use and start to migrate to and then evolve the infrastructure under it.

We should take selog or like,  polish it warp around ul_log or pure syslog and have as an interface. While developers would migrate to it we will evolve the library and syslog implementation under it to serialize and format things into JSON XML etc but developer will have to change things once.
  

_______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers


-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager IPA project,
Red Hat Inc.


-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/