Metrics aggregation is kicked off from the DataPurgeJob that runs at the start of every
hour. It computes and stores aggregate metrics for the previously completed time slice(s).
For instance, if aggregation runs at 10:02, then raw data stored between 09:00 and 10:00
will get rolled up into 1 hour metrics. I will describe scenarios in which missed
aggregations can occur followed by possible solutions. Any feedback is
welcomed/appreciated.
Missed aggregation scenarios:
* server outage
Suppose the server goes down at 08:46 and does not come back up until 09:45. We miss the
regularly schedule aggregation for the 08:00 - 09:00 time slice.
* failed aggregation
While aggregation runs, suppose the storage cluster goes goes down. We will fail to store
aggregate metrics.
* Late measurement reports
Suppose an agent loses its connection to the server at 09:30. The agent will spool
measurement data. Then the agent reconnects to the server at 10:15 after aggregation has
finished. The agent sends one or more measurement reports with data from the 09:00 hour.
That data will not aggregated.
Problems with missed aggregations:
It can lead to skewed or inaccurate aggregate metrics which in turn can affect baselines
and OOBs. Another issue is that rows in the metrics_index table which otherwise would have
been purged can wind up living on indefinitely.
Solutions:
* Ignore missed aggregations
We already handle the case of server outages. If we choose to ignore the other scenarios,
then we only need to make sure that rows in the metrics_index table get purged. We can
accomplish this easily by setting TTLs.
* Retry missed/failed aggregations
There are a couple different ways we could go about doing this. I will save the details
for a separate discussion as it can rather involved. Suffice it to say, we can implement
functionality to handle the scenarios of late measurement reports and failed runs. This
would obviously be more complex that ignoring missed/failed aggregations but arguably more
robust.
I guess the first question is, do we need to worry about missed/failed aggregations?
- John