Hi,

Some time ago (July) Heiko wrote a small mail with a link to a log management blog. That got me wondering, what's the roadmap with this issue and how should it be solved?

After a small chat in the IRC-channel, I decided I might write my thoughts here also, in case we could get some discussion going on, as it probably is an issue that's interesting to many people. Currently there's Events in the RHQ, which could be parsed from the logs, but this isn't clearly enough. The Fuse team has integrated Kibana to their Hawtio management tool, so some teams at JBoss are playing with the idea.

There are some issues which come to my head right away about storing and querying:

- Where to store the logs? Cassandra could be used as a place to store the logs, but it's horrible in queries.
- If we store it in something like ElasticSearch, should the storage node architect be extended to include ES as well? This is very close to what DataStax Enterprise product does (with Solr, and it has static schema which ES doesn't usually use).

The even trickier issue is, how should we support logs? Would shippings logs with the agent be enough to some third party service? Should we create Events from these as well? And how would we support structured logs in this case? Like json formatted logs.

It's not probably wise use of resources to build something from the ground to compete with likes of logstash/splunk/flume/many others, that can parse different logformats effectively. The current Events format isn't enough for many purposes, and many logs are not just plain text.

And that's just the parsing + shipping part. The other issue is viewing the logs, searching the logs, creating metrics, creating statistics etc. Should we integrate some third party UI, such as Kibana to our tool (Fuse guys have ported some Ruby parts to Java servlets) ? I remember there was some discussion on how these could be added, but I can't remember if it was finished.

I'm only throwing around ideas, but to me this is a big part of infrastructure. 

  -  Micke