Hello Everybody,
I want to start a discussion about BZ 1161806 [1]. We discovered an issue with a few
integration tests [2] in the server-metrics module about a week ago. What seemed just a
few bad tests or an environmental problem led to the discovery of a bigger problem with
metrics aggregation. This email is just a summary of the BZ contents with the goal to make
sure we did not miss anything. A big thank you goes to John Sanda and Jay Shaughnessy for
helping with the solution and code fixes.
1) Background
- All metric data is stored with epoch time, not timezone dependent
- Aggregation is done in time slices. For example, the 6 hour aggregation for a single
metric is sliced into data aggregates at 00:00, 06:00, 12:00, and 18:00
- 1 hour aggregation is not sensitive to timezones (or DST transitions) because top of the
hour is top of the hour regardless of timezone
- 6 and 24 hour aggregations were sensitive to timezones (and DST transitions) because the
slices were calculated in local timezone
2) Impact
- Aggregation for 1 hour metrics was minimally impacted - only DST transitions could cause
at most 1 hour to not be computed due to clock rolling
- Aggregation for 6 hour metrics and 24 hours metrics was highly impacted:
- When DST transitions occur, the aggregation indexes would not be queryable until the
DST transition was beyond the metrics index query boundaries
- In an HA environment with RHQ servers in different timezones, when switching servers
the aggregation for past data would stop working
3) Solution
- Switch the time slices for 6 and 24 hour aggregates to be based on UTC timezone
regardless of the local timezone
- The metrics indexes updates slated for RHQ 4.13 will rebase the 6 and 24 hour timeslices
to UTC time slices
- The BZ has a nice diagram [3] that we used to find a good solution for the time slice
rebasing
4) Future
- Will investigate the impact on RHQ 4.12 and prior to see if any fixes are needed
- We are too late to release a fix for RHQ 4.12 and prior just address the DST transitions
because most of the countries have switched for over 2 weeks (the query boundary for
metrics indexes)
- We strongly recommend everybody to update to RHQ 4.13 once it is released
All feedback is welcome ...
[1]
https://bugzilla.redhat.com/show_bug.cgi?id=1161806
[2]
https://github.com/rhq-project/rhq/blob/master/modules/enterprise/server/...
[3]
https://bugzilla.redhat.com/attachment.cgi?id=956797
Thank you,
Stefan Negrea
Software Engineer