Cassandra event storage - design ideas

Elias Ross genman at noderunner.net
Sat Mar 8 21:32:33 UTC 2014


I had come up with a simple design (not tested) for moving events to Cassandra.

The goals are:
0) Replace what's there.
1) Deal with high and low volume situations. Since RHQ doesn't handle
a large number of events very well (> 100,000), mostly they are used
for low volume situations today. You would not want to capture debug
messages for all your application servers, currently. In some cases,
though, you might have thousands of event definitions. But eventually
a solution for all log messages is needed.
2) Allow events to be searched in the same ways, e.g. severity, location, etc.
3) Deal with data retention. How to remove data after a certain time.

Some other goals: (probably outside of this...but worth consideration.)
0) Administrator type problems: "Bring me all errors across all systems."
1) Make it easier to add lots of log files for monitoring. For
example, /var/log/*.log
2) Automate log parsing of dates, severity, log category (where available)
3) Query all the events with 'NullPointerException' in them, for
instance. Probably possible by adding extra columns like 'exception'
or 'http_status', and adding an index or something like that based on
what the user wants.
3b) Design to allow full text search to work. (Simple approach: Create
a separate table, with primary key of the hashed token. Map this back
to event(s). Use the TTL of the event.)
4) Basically, make RHQ function more like 'Splunk-lite'.

Mostly this design is taken from:
http://www.ebaytechblog.com/2012/08/14/cassandra-data-modeling-best-practices-part-2/

      CREATE TABLE rhq.events (
        hour int, -- whole hours since 1970
        event_definition int,
        time bigint, -- could be timeuuid
        type varchar,
        location varchar,
        -- category varchar, -- host?
        severity varchar,
       detail varchar,
     PRIMARY KEY ((hour, event_definition), time) -- 'event_id'
      );

If there is a large TTL and you want to select 'every event', maybe
this is needed? This the earliest and latest events. Updated when
events are inserted. (Is this worth it?)

      CREATE TABLE rhq.event_range (
        event_definition int,
        start_hour int,
        end_hour int,
     PRIMARY KEY (event_definition)
      );

Why partition this way? For large date ranges, there are still only a
few hundred rows to query. And even if thousands events are recorded
per hour, Cassandra columns can be fairly wide, so this shouldn't be a
problem.

time - Past-the-hour relative time, with a unique counter. Basically,
the number of milliseconds since the current hour (32 bits), plus 32
bits for an incremented counter. This is so multiple events with the
same millisecond timestamp don't overwrite each other. Since one agent
is talking to one server, the 'time' will be unique.

type - EventDefinition.getType() -- is this useful?
location - Log file path, etc. Indexed by default.
-- category - Log category, subsystem, etc. Indexed by default. (Maybe
not in v1, as it's not captured today.)
severity - INFO, WARN, etc.
detail - Full log text, etc.

The tables RHQ_EVENT and RHQ_EVENT_SOURCE are integrated into this one
table (column family.)

Indexes: (are they useful?)

CREATE INDEX events_location
  ON events(location);

Should identify the log name. You can then query all events for
'/var/log/messages'. Possible problems: The number of locations may
grow, especially if the log file has timestamps in the name or a
counter. Also doesn't really allow for partial selection, e.g.
'/var/log/*'. (Can you query Cassandra for all unique indexed values?
Create a separate table for this?)

CREATE INDEX events_severity
  ON events(severity);

Useful to find all errors, but unclear if DEBUG or INFO (for example)
should be indexed. Might create a separate column for WARN and ERROR
severity only, e.g. 'high_severity'. May be too high volume.

Deletion:
TTL columns are used for event data. This could be assigned per event
definition, but probably makes sense to simply use a global setting
(as it is) for now.

SQL operations:

    @NamedQuery(name = Event.DELETE_BY_RESOURCES, query = "DELETE FROM
Event ev  "
        + " WHERE ev.source IN ( SELECT evs FROM EventSource evs WHERE
evs.resource.id IN ( :resourceIds ) )"),

Maybe not needed? Since events are deleted by TTL anyway. To do this,
would be to find all the event definitions, the first time for events
for this source (by TTL), then delete incrementally by hour.

    @NamedQuery(name = Event.DELETE_BY_EVENT_IDS, query = "DELETE FROM
Event e WHERE e.id IN ( :eventIds )"),

hour,event_definition,time is used here as the key for deleting the column/row.

    @NamedQuery(name = Event.DELETE_ALL_BY_RESOURCE, query = "" //
    @NamedQuery(name = Event.DELETE_ALL_BY_RESOURCE_GROUP, query = "" //

Similar to DELETE_BY_RESOURCES. Just find the event_definition_ids.

    @NamedQuery(name = Event.FIND_EVENTS_FOR_RESOURCE_ID_AND_TIME,
query = "SELECT ev FROM Event ev "
        + " JOIN ev.source evs JOIN evs.resource res WHERE res.id =
:resourceId AND ev.timestamp BETWEEN :start AND :end "),

Find the event_def_ids, then find the applicable hours, query those
rows and filter by exact timestamp.

    @NamedQuery(name =
Event.FIND_EVENTS_FOR_RESOURCE_ID_AND_TIME_SEVERITY, query = "SELECT
ev FROM Event ev "

Find the event_def_ids, then find the applicable hours, query those
rows and filter by exact timestamp and severity. Severity is indexed.

    @NamedQuery(name = Event.GET_DETAILS_FOR_EVENT_IDS, query = "SELECT "
        + " new
org.rhq.core.domain.event.composite.EventComposite(ev.detail, res.id,
res.name, res.ancestry, res.resourceType.id, ev.id, ev.severity,
evs.location, ev.timestamp) "
        + " FROM Event ev JOIN ev.source evs JOIN evs.resource res
WHERE ev.id IN (:eventIds) AND evs.id = ev.source"
        + "  AND res.id = evs.resource "), //

Given a list of event_ids, return the resource details. Since the
event_id will contain the event_def_id as part of its key... Should be
possible, but need to look into this.

    @NamedQuery(name = Event.QUERY_EVENT_COUNTS_BY_SEVERITY, query = "" //

Might need to create a special table... Each severity column has a
list of events. Or simply rely on the index of severity. Counts are
hard to do in Cassandra, so dropping these might be wise.

      CREATE TABLE rhq.event_severity (
        resource_id int,
        severity varchar,
        event_id stuff,
       PRIMARY KEY ((resource_id),severity,event_id)
      );

    @NamedQuery(name = Event.QUERY_EVENT_COUNTS_BY_SEVERITY_GROUP, query = "" //

Use the same approach as above, but with the list of resources.


More information about the rhq-devel mailing list