RHQ trait storage

John Sanda jsanda at redhat.com
Wed Apr 23 21:15:12 UTC 2014


With respect to alerting, I am bit less concerned since the read does not happen on each write. If I understand correctly, it is done async via JMS when the cache(s) need to be reloaded which could be due to a change in trait value. I do not think that partitioning by resource id really helps for the alerting query since we are filtering off of the agent id and not the resource id. I might instead do something like,

CREATE TABLE agent_traits (
    agent_id int,
    schedule_id,
    value varchar,
    time timestamp,
    PRIMARY KEY (agent_id, schedule_id, time)
);

To reload an agent cache, we can easily and efficiently query across a single partition to obtain the latest trait values. I am not too concerned about repeated inserts. I rather treat the data as immutable and maintain a history versus updating in place. Nathan Marz makes good arguments for immutable data in his upcoming book[1]. And of course in RHQ, we already have lots of history tables that essentially mimic behavior we get for free with Cassandra.

To limit or cap the amount of trait data we store, we can periodically run a job that purges old data. If the rows become too wide, we might need to consider further partitioning the agents_traits table. We could partition by day.

CREATE TABLE agent_traits (
    agent_id int,
    day timestamp
    schedule_id,
    value varchar,
    time timestamp,
    PRIMARY KEY ((agent_id, day), schedule_id, time)
);

This approach would make purging old data more efficient.

I do think the measurement report code should be refactored so that the MeasurementData includes resource ids since we have user-facing trait queries that filter by resource id. I would further denormalize the data though to include in measurement_data_trait the additional columns we fetch.

[1] http://www.manning.com/marz/

On Apr 23, 2014, at 3:26 PM, Elias Ross <genman at noderunner.net> wrote:

> On Tue, Apr 22, 2014 at 11:33 AM, Elias Ross <genman at noderunner.net> wrote:
> 
>> The schema:
>> 
>> CREATE TABLE IF NOT EXISTS measurement_data_trait (
>>    schedule_id int,
>>    value varchar,
>>    timestamp int, -- when it was inserted
>> PRIMARY KEY (schedule_id)
>> );
> 
> My inclination was to change this to
> 
> CREATE TABLE IF NOT EXISTS measurement_data_trait (
>     resource_id int,
>     schedule_id int,
>     value varchar,
>     timestamp int, -- when it was inserted
>  PRIMARY KEY (resource,id schedule_id)
> );
> 
> The reason is that all the traits are selectable by resource_id
> directly. They're also stored together for easier caching and
> selecting.
> 
> The downside is you must know the resource_id before you can do any
> insert. This is a problem as the measurement report does not contain
> this piece of information. Which would require a lot of changes to
> 'MeasurementData', the agent, and so forth.
> 
>> 
>> Approach is to:
>> 1) Select the current value. (*)
>> 2) If it is different then insert into the history table. Use TTL
>> columns to delete old records.
>> 3) In any case, update the measurement_data_trait table with the most
>> current value
>> 
> ...
> 
> I know this approach is slow (and can block). But for alerting we do
> need a select at some point, at least for those alerted on traits.
> Oracle tells us this, because the update count lets us know the trait
> changed or not.
> 
> On IRC, the suggestion was to cache the existing trait values to
> determine a trait changed. We can still do that effectively with a
> Cassandra-side row cache. The alternative is to use a clustered cache,
> but RHQ does not support that sort of clustering yet.
> 
> I think my initial implementation will use some sort of thread pool or
> async EJB call though, for the select+update, and I'll have to
> determine if that is effective or not. I do have a 200+ machine
> cluster which will prove if the average cache (assuming my
> installation is average) performs well enough.
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fedorahosted.org/pipermail/rhq-devel/attachments/20140423/d140bb84/attachment-0001.html>


More information about the rhq-devel mailing list