Scalability issues in RHQ 4.9 - A summary

Elias Ross genman at noderunner.net
Wed Mar 12 00:55:36 UTC 2014


On Mon, Mar 10, 2014 at 2:20 AM, James Strachan <jstracha at redhat.com> wrote:
> FWIW we're just using EK in fabric8 ;) We're not using logstash right now; mostly as inside a fabric8 JVM we know what each log event looks like and all its fields; so we just log it directly to ElasticSearch. i.e. we don't have to parse a log file and use some brittle regular expressions configuration to try guess which bit of text is the category, which the timestamp, which the thread name & which the message etc. Given users can configure the logging configuration and layout at any point; we prefer to not have to rely on brittle regex configurations; so prefer to write directly into ElasticSearch from the JVM. Though logstash is cool and for non-Java systems (like syslogs etc) is very useful.
>

I'm guessing logstash would be useful for log file parsing
applications, but yes I can't see it that useful for situations where
you don't need a log parser (or something to deal with configured
settings.)

I can imagine a very simple agent plugin for RHQ that basically took a
configuration string, created sub-resources based on log files, which
then read and converted log messages to RHQ events.

> In fabric8 we can also log camel messages to ElasticSearch; for generating audit logs of message flows and their payloads which we can then use to correlate messages and to reverse engineer sequence flows and the like.
> https://twitter.com/cmoulliard/status/439410019511963649/photo/1/large
> Using MDC / NDC type stuff the same thing could be done with logs/events too really.

Today I worked on a Cassandra 'Gateway' for ElasticSearch, which is
very much like this one:

https://github.com/elasticsearch/elasticsearch-hdfs

I haven't tested it yet outside of the unit tests (where it works
okay) but seems perfectly useable. The speed might not be great, but
it is useable. I do see stuff like this which is a concern:

https://github.com/elasticsearch/elasticsearch/issues/2458

> Kibana and ElasticSearch are great open source technologies BTW - plus they are lots cheaper than buying Splunk ;)


More information about the rhq-devel mailing list