missed metrics aggregations

John Sanda jsanda at redhat.com
Wed Feb 26 17:55:40 UTC 2014


Metrics aggregation is kicked off from the DataPurgeJob that runs at the start of every hour. It computes and stores aggregate metrics for the previously completed time slice(s). For instance, if aggregation runs at 10:02, then raw data stored between 09:00 and 10:00 will get rolled up into 1 hour metrics. I will describe scenarios in which missed aggregations can occur followed by possible solutions. Any feedback is welcomed/appreciated.

Missed aggregation scenarios:
* server outage
Suppose the server goes down at 08:46 and does not come back up until 09:45.  We miss the regularly schedule aggregation for the 08:00 - 09:00 time slice.

* failed aggregation
While aggregation runs, suppose the storage cluster goes goes down. We will fail to store aggregate metrics.

* Late measurement reports
Suppose an agent loses its connection to the server at 09:30. The agent will spool measurement data. Then the agent reconnects to the server at 10:15 after aggregation has finished. The agent sends one or more measurement reports with data from the 09:00 hour. That data will not aggregated.


Problems with missed aggregations:
It can lead to skewed or inaccurate aggregate metrics which in turn can affect baselines and OOBs. Another issue is that rows in the metrics_index table which otherwise would have been purged can wind up living on indefinitely.

Solutions:
* Ignore missed aggregations
We already handle the case of server outages. If we choose to ignore the other scenarios, then we only need to make sure that rows in the metrics_index table get purged. We can accomplish this easily by setting TTLs.

* Retry missed/failed aggregations
There are a couple different ways we could go about doing this. I will save the details for a separate discussion as it can rather involved. Suffice it to say, we can implement functionality to handle the scenarios of late measurement reports and failed runs. This would obviously be more complex that ignoring missed/failed aggregations but arguably more robust.


I guess the first question is, do we need to worry about missed/failed aggregations?


- John


More information about the rhq-devel mailing list