missed metrics aggregations

Stefan Negrea snegrea at redhat.com
Tue Mar 4 16:47:22 UTC 2014



----- Original Message -----
> From: "Larry O'Leary" <loleary at redhat.com>
> To: rhq-devel at lists.fedorahosted.org
> Sent: Wednesday, February 26, 2014 11:37:55 PM
> Subject: Re: missed metrics aggregations
> 
> On Wed, 2014-02-26 at 12:55 -0500, John Sanda wrote:
> > Solutions:
> > * Ignore missed aggregations
> > We already handle the case of server outages. If we choose to ignore
> > the other scenarios, then we only need to make sure that rows in the
> > metrics_index table get purged. We can accomplish this easily by
> > setting TTLs.
> 
> Not sure I like the idea of missing/incomplete data. Especially if it
> can result in incorrect aggregates.

I would not call them incorrect aggregates. They are aggregates that rely on partial data, or aggregates based on sample rather than full data. While this is not as good as using all data available for a slice, it will not be a random number. But I agree that aggregation based on the full data sample is the optimal solution.

> 
> > * Retry missed/failed aggregations
> > There are a couple different ways we could go about doing this. I will
> > save the details for a separate discussion as it can rather involved.
> > Suffice it to say, we can implement functionality to handle the
> > scenarios of late measurement reports and failed runs. This would
> > obviously be more complex that ignoring missed/failed aggregations but
> > arguably more robust.
> 
> This seems like the ideal solution. I am not sure why aggregation has to
> take a one hour chunk. Ideally we expect it to run every hour but in the
> event it runs late or the server was down why can't we just figure out
> what data still needs to be aggregated and start there? Aggregation
> could happen in one hour chunks starting with the oldest hour.
> 

The data model in Cassandra is a little different from traditional SQL database. It takes a little bit of preparation to do what you proposed. It's not as easy as running another time bound query on the data tables; such query is highly inefficient in Cassandra. This is what John is working on, changes to the data model and code to keep the aggregation fast and nimble while having a recovery mechanism for missed, failed, or incomplete aggregations. 



Stefan


More information about the rhq-devel mailing list