----- Original Message -----
From: "Larry O'Leary" <loleary(a)redhat.com>
To: rhq-devel(a)lists.fedorahosted.org
Sent: Wednesday, February 26, 2014 11:37:55 PM
Subject: Re: missed metrics aggregations
On Wed, 2014-02-26 at 12:55 -0500, John Sanda wrote:
> Solutions:
> * Ignore missed aggregations
> We already handle the case of server outages. If we choose to ignore
> the other scenarios, then we only need to make sure that rows in the
> metrics_index table get purged. We can accomplish this easily by
> setting TTLs.
Not sure I like the idea of missing/incomplete data. Especially if it
can result in incorrect aggregates.
I would not call them incorrect aggregates. They are aggregates that rely on partial data,
or aggregates based on sample rather than full data. While this is not as good as using
all data available for a slice, it will not be a random number. But I agree that
aggregation based on the full data sample is the optimal solution.
> * Retry missed/failed aggregations
> There are a couple different ways we could go about doing this. I will
> save the details for a separate discussion as it can rather involved.
> Suffice it to say, we can implement functionality to handle the
> scenarios of late measurement reports and failed runs. This would
> obviously be more complex that ignoring missed/failed aggregations but
> arguably more robust.
This seems like the ideal solution. I am not sure why aggregation has to
take a one hour chunk. Ideally we expect it to run every hour but in the
event it runs late or the server was down why can't we just figure out
what data still needs to be aggregated and start there? Aggregation
could happen in one hour chunks starting with the oldest hour.
The data model in Cassandra is a little different from traditional SQL database. It takes
a little bit of preparation to do what you proposed. It's not as easy as running
another time bound query on the data tables; such query is highly inefficient in
Cassandra. This is what John is working on, changes to the data model and code to keep the
aggregation fast and nimble while having a recovery mechanism for missed, failed, or
incomplete aggregations.
Stefan