Events migration plan

Elias Ross genman at noderunner.net
Mon Apr 14 17:38:10 UTC 2014


Here's an outline for how RHQ might store and handle events in Cassandra.

Part 1 - Use Elasticsearch as storage

The idea is to focus on improving event storage but keep the UI
elements the same and EJB interfaces behaving the same.

1. Create plugin for storing events in Elasticsearch using Cassandra.
I've already done the work for this, but it needs more testing.

The alternative here is to use Elasticsearch natively, similar to how
storage nodes are created and managed, meaning some sort of RHQ-based
installer or wrappers. This would mean additional work on creating
agent plugins and whatever similarly was done with Cassandra.

2. Create a module for JBoss to bootstrap Elasticsearch.

This should be fairly easy to do, but I do foresee some additional
memory usage required. There are quite a lot of jar files that are
'sharded' into it, and I wonder if that could be avoided.

Google Guice is used as a way to wire the system, I'm not sure how
well it may work inside JBoss EAP. Also, things like plugins and
whatever classloader type things may need some coercion.

3. Create a series of classes for storing RHQ events in Elastic Search
using the same log format as Kibana uses. Ensure that at least log
files are stored in this way and are searchable.

This isn't critical as Kibana won't be used yet. But it makes sense
for searching in the future.

4. Create EJB classes for retrieving events from Elasticsearch,
deletion, search, purging, etc., identical to the existing methods.

Do you keep the existing web service API (REST-based) operational and
use that, or use the Java API directly itself?

5. Event migration tools.

6. Testing and release.


Part 2 - UI improvements or Kibana integration

The idea is to integrate Kibana in some form with the native RHQ UI.


Part 3 - Agent event changes

Does the agent continue to use the existing remoting API or something
that may be better suited for high volume traffic?


Part 4 - Log file importation improvements

Log file discovery should happen more easily. For example, all the
logs from JBoss or Tomcat applications should be directly imported if
those applications are in inventory. Also, any system logs
(/var/log/messages etc.) should appear as well.

Anything to do with log parsing (especially parsing dates, log
severity, messages, exceptions as one message) should work
automatically.

Log rotation detection, compression, log tailing, should be handled
automatically.


More information about the rhq-devel mailing list