Aggregation Time Slices - BZ 1161806

Stefan Negrea snegrea at redhat.com
Wed Nov 12 17:13:24 UTC 2014


Hello Everybody,

I want to start a discussion about BZ 1161806 [1]. We discovered an issue with a few integration tests [2] in the server-metrics module about a week ago. What seemed just a few bad tests or an environmental problem led to the discovery of a bigger problem with metrics aggregation. This email is just a summary of the BZ contents with the goal to make sure we did not miss anything. A big thank you goes to John Sanda and Jay Shaughnessy for helping with the solution and code fixes.


1) Background
- All metric data is stored with epoch time, not timezone dependent
- Aggregation is done in time slices. For example, the 6 hour aggregation for a single metric is sliced into data aggregates at 00:00, 06:00, 12:00, and 18:00
- 1 hour aggregation is not sensitive to timezones (or DST transitions) because top of the hour is top of the hour regardless of timezone
- 6 and 24 hour aggregations were sensitive to timezones (and DST transitions) because the slices were calculated in local timezone


2) Impact
- Aggregation for 1 hour metrics was minimally impacted - only DST transitions could cause at most 1 hour to not be computed due to clock rolling
- Aggregation for 6 hour metrics and 24 hours metrics was highly impacted:
  - When DST transitions occur, the aggregation indexes would not be queryable until the DST transition was beyond the metrics index query boundaries
  - In an HA environment with RHQ servers in different timezones, when switching servers the aggregation for past data would stop working


3) Solution
- Switch the time slices for 6 and 24 hour aggregates to be based on UTC timezone regardless of the local timezone 
- The metrics indexes updates slated for RHQ 4.13 will rebase the 6 and 24 hour timeslices to UTC time slices
- The BZ has a nice diagram [3] that we used to find a good solution for the time slice rebasing


4) Future
- Will investigate the impact on RHQ 4.12 and prior to see if any fixes are needed
- We are too late to release a fix for RHQ 4.12 and prior just address the DST transitions because most of the countries have switched for over 2 weeks (the query boundary for metrics indexes)
- We strongly recommend everybody to update to RHQ 4.13 once it is released 


All feedback is welcome ... 


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1161806
[2] https://github.com/rhq-project/rhq/blob/master/modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
[3] https://bugzilla.redhat.com/attachment.cgi?id=956797


Thank you,
Stefan Negrea

Software Engineer



More information about the rhq-devel mailing list