Transforming Log Events into Information

Larry O'Leary loleary at redhat.com
Wed Aug 7 22:55:42 UTC 2013


On Thu, 2013-08-08 at 01:05 +0300, Michael Burman wrote:
> Hi,
> 
> 
> Some time ago (July) Heiko wrote a small mail with a link to a log
> management blog. That got me wondering, what's the roadmap with this
> issue and how should it be solved?
> 
> 
> After a small chat in the IRC-channel, I decided I might write my
> thoughts here also, in case we could get some discussion going on, as
> it probably is an issue that's interesting to many people. Currently
> there's Events in the RHQ, which could be parsed from the logs, but
> this isn't clearly enough. The Fuse team has integrated Kibana to
> their Hawtio management tool, so some teams at JBoss are playing with
> the idea.
> 
> 
> There are some issues which come to my head right away about storing
> and querying:
> 
> 
> - Where to store the logs? Cassandra could be used as a place to store
> the logs, but it's horrible in queries.
> - If we store it in something like ElasticSearch, should the storage
> node architect be extended to include ES as well? This is very close
> to what DataStax Enterprise product does (with Solr, and it has static
> schema which ES doesn't usually use).
> 
> 
> The even trickier issue is, how should we support logs? Would
> shippings logs with the agent be enough to some third party service?
> Should we create Events from these as well? And how would we support
> structured logs in this case? Like json formatted logs.
> 
> 
> It's not probably wise use of resources to build something from the
> ground to compete with likes of logstash/splunk/flume/many others,
> that can parse different logformats effectively. The current Events
> format isn't enough for many purposes, and many logs are not just
> plain text.
> 
> 
> And that's just the parsing + shipping part. The other issue is
> viewing the logs, searching the logs, creating metrics, creating
> statistics etc. Should we integrate some third party UI, such as
> Kibana to our tool (Fuse guys have ported some Ruby parts to Java
> servlets) ? I remember there was some discussion on how these could be
> added, but I can't remember if it was finished.
> 
> 
> I'm only throwing around ideas, but to me this is a big part of
> infrastructure. 
> 
> 
>   -  Micke

To your list of possible issues that need to be addressed with events
and capturing log data is that our current polling method is very
fragile. 

 - due to log buffering and disk caching, simply scanning a log file
every few seconds may not reveal those critical events that they user
needs to capture and alert on
 - due to log rotation and rolling services (such as log4j or even
logrotate for Linux) monitored sources can vanish out from underneath
the poller leading lost events sometimes in the order of thousands of
them

Perhaps these issues are no longer relevant if the plan it to ship all
the logs over to a another library to parse/process after the fact. But
what about monitoring? I see the real value in logging is to receive the
events as they happen. Perhaps supporting an event plug-in system that
allows external loggers to be extended to send log messages directly to
the agent in a format that it expects and can use? Granted, not all
logging systems support this mechanism directly but even the system
logger on Linux can use a named pipe so perhaps RHQ could do the heavy
lifting for the most common logging systems?

Once we get the data, storing does become an issue. So perhaps we do
away with log event polling and capturing the raw log events and
instead:

 - give the user the ability to read the raw log file(s) for a resource
by streaming a current snapshot of the log files to the user in the UI
and via the remote APIs
 - support alert definitions that can perform log parsing and
interpretation to produce meaningful alerts

In this case, the agent would act as the conduit to the log data and
support plug-ins that knew how to parse the log information. This
parsing could be as simple as regular expression/grep like requests and
more complicated such as supporting kibana or the like.

Obviously, considering the alert subsystem is only evaluating metrics
returned by the agent, the alert definition + the logging/event plug-in
would introduce pseudo metrics/measurements that would result in the
agent calls.

However, these are just thoughts as they relate to issues users have
experienced with attempting to make sense of logs and events.
-- 
Larry O'Leary
https://plus.google.com/u/0/112645929986009801513



More information about the rhq-devel mailing list