Scalability issues in RHQ 4.9 - A summary

Michael Burman yak at iki.fi
Sat Mar 8 22:40:13 UTC 2014


I personally don't feel like Cassandra is the correct place for events, or
that the parsing / alerting task is simple. Log parsing should be somewhat
decent, and not just half-made component in the monitoring system. I
however agree that it's pretty vital to a monitoring system, since some
systems do not output anything else than some logs when they encounter an
error that should be logged and alerted.

Cassandra lacks some basic querying properties needed for good event
searching, and in RHQ there's currently only very primitive "Events"
handling in that single message is all we get from the event basically.
Logstash and equivalents provide much richer capabilities, because they
parse the logs to a structured format. What we'd need is much more generic
alert definitions to be able to take advantage of those properties. This is
where in my opinion the development should concentrate on, being able to
create rich alert definitions, but stop wasting resources for now in
re-creating yet another UI for showing logs.

My current solution to this has been in creating agent-plugin (sorry, it's
not ready yet to be published) that runs my own log shipper (pure Java
because other tools don't support IBM JDK on AIX too well) that then ships
using AMQP the logs to an ActiveMQ queue. What I'd like to see is using the
RHQ infrastructure more. Being able to ship the logs using the agents to
the RHQ's HornetQ and from there being able to process them somehow would
already be an improvement. Now the only advantage I have is using agent to
run the logshipper (and configure it), but I can't use the agents firewall
connections and RHQ infra to actually do the log shipping. In my other end
I have Logstash for parsing and Kibana for showing logs. I don't thinnk
they're the perfect tool, but I've used them for some time. They're too
disconnected for my liking from the RHQ though, forcing me to take care of
two different monitoring and alerting systems.

  - Micke


On Sat, Mar 8, 2014 at 2:10 PM, Heiko W.Rupp <hrupp at redhat.com> wrote:

>
> Am 08.03.2014 um 02:59 schrieb Elias Ross <genman at noderunner.net>:
> >
> > I wouldn't mind working on some Cassandra features either. I could
> > work on moving events to Cassandra, but I'm not sure how interested I
> > am in dealing with the data migration steps or whatnot.
>
>
> As you probably know is Fuse Fabric8 also part of the Red Hat family and
> we are checking how we can work together so that they can use parts of
> what we have an vice versa.
> Fabric8 is (iirc) using ELK (Elasticsearch, Logstash Kibana) for parts of
> what we call Events (Logs are one kind of events, but also incoming snmp
> traps, JMX notifications etc. could be Events in RHQ). So we are also
> checking if ELK could be something for RHQ.
> And then there is Solandra (https://github.com/tjake/Solandra ) that is
> Solr
> search on top of Cassandra, so that we could perhaps end up with ELCK here.
>
> I just mention that, as we are aware that the current Events have some
> room for improvement and I do not want you to spend energy on something
> that will be changing/replacing soonish anyway. There is no decision made,
> and of course every input is very welcome.
>
>   Heiko
>
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fedorahosted.org/pipermail/rhq-devel/attachments/20140309/b6a23b0c/attachment-0001.html>


More information about the rhq-devel mailing list