On 03/27/2012 08:49 PM, david@lang.hm wrote:
On Tue, 27 Mar 2012, Dmitri Pal wrote:
On 03/27/2012 06:17 PM, William Heinbockel
wrote:
On Mar 27, 2012 4:17 PM, <david@lang.hm
<mailto:david@lang.hm>> wrote:
On Tue, 27 Mar 2012, William Heinbockel wrote:
On Tue, Mar 27, 2012 at 3:18 PM,
<david@lang.hm
<mailto:david@lang.hm>> wrote:
I posted some of this at the bottom of a long message
yesterday
that nobody
replied to, so I broke it out to a
separate thread
The traditional syslog interface has been a write-only
interface,
with the
only feedback being that the write
could block.
With the new formats and new syslog replacement library,
does
there need to
be some rethinking of this
interface?
JSON can be pushed through the traditional syslog
interface with
no changes
(which is one of the attractions
of using JSON), but when we support
additional formats, especially ones with type
information, we need
to have a
way to tell what format to use
when writing the log.
I don't think this is right approach.
The end goal is to support structured log messages.
The first step is to introduce the capability to sent
structured
events via existing capabilities (read: JSON over Syslog)
The next steps would be the ELAPI/selog API that would
introduce a new
logging interface, probably maintaining at least minimal
compatibility
with the syslog daemon (not the syslog API or syslog
protocol). This
would allow us to define and support more feature-rich log
protocols
and other features.
But does this mean that unless applications are re-written,
they
will never send structured logs at all?
There are only two ways to have applications log with
structured data:
1. Update the app to send name-value pairs
2. Use something like patterndb to convert the unstructured
data
If the app wants to leverage structured data, they will
already have
to make some updates. We just offer them a choice to either
use the
minimal umberlog interface or convert to a more full featured
one.
I see there being value in having the cee-syslog replacement
library
that sends the metadata structured around the existing
unstructured
log data. It's not as good as a full structured log, but it's
a lot
better than the existing situation.
This could be done by configuring things in the openlog
call, but that
requries changes to every application.
I think it would be better to have the serialization
format be
determined
automatically by the library if
possible.
One way to do this is to have a handshake.
When a app opens /dev/log, the syslog server sends a
capabilities
message.
The client can ignore this message and send traditional
formatted
syslog
data to the server
Or, the client can read the capabilities message and
then pick which
supported format (potentially with compression options)
to use to
send data
to the server.
This approach lets the format be anything that the
library
supports, under
the control of the sysadmin of the
system (by configuring what
capabilities
the syslog daemon advertises)
without the application programmer
needing to
know anything about these formats.
thoughts?
Let's not get ahead of ourselves and try to shoehorn a
solution into
existing/legacy capabilities. We should approach this as a
two-pronged
solution:
1. Backwards-compatible structured log implementation that
works with
the syslog API and syslog protocol (this is umberlog).
This should be
as minimal and compatible as possible. I commend Gergely
on his
implementation
2. Implement a new event API frontend and improved logging
capabilities. This should be able to work with syslogd,
but does not
have any requirements to be able to work with the existing
syslog API,
syslog protocol, /dev/log, etc.
Adding a new application API to generate more structured
logs is a
good idea. But why do we need to create a completely new
interface
between the logging library and syslog instead of extending
the
existing interface?
You don't know what you are going to break and you restrict
yourself
to the limitations of the current interface.
It is not that I am against it, just if we want to improve it
it might
be best to redo it entirely.
Also, remember that there is more than just syslog and user
apps as
potential customers. There is also the kernel, klogd, and
auditd.
William Heinbockel
With the current plan we ask application developers to change
things
twice. First to new syslog umberlog interface and then later to
a better
interface. IMO this is wrong. I tried to rise the concerns about
this
several times but have not been heard.
as I am understanding it, the primary value of umberlog is that it
gets us structured metadata (and more metadata) with no
application changes.
I agree that we need to define additional functions to use for
adding application structured data to the log.
IMO we need to define one interface that
the application developers
would use and start to migrate to and then evolve the
infrastructure
under it.
We should take selog or like, polish it warp around ul_log or
pure
syslog and have as an interface. While developers would migrate
to it we
will evolve the library and syslog implementation under it to
serialize
and format things into JSON XML etc but developer will have to
change
things once.
I think we differ in details, but not in overall strategy.
May be I am missing something but I fail to understand how changing
from syslog() function to ul_syslog() is not an application change.
David Lang
_______________________________________________
lumberjack-developers mailing list
lumberjack-developers@lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/lumberjack-developers
--
Thank you,
Dmitri Pal
Sr. Engineering Manager IPA project,
Red Hat Inc.
-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/