Earlier today the question came up about whether or not if we can force the execution of
the data purge job. You can if you enable debug mode in system settings and then go to
http://localhost:7080/portal/admin/test/control.jsp. Due to implementation changes,
executing the data purge job here will basically result in a no-op for metrics
compression.
I need to give a little background on the C* implementation to explain why this is so.
When the metrics compression runs, we look at the current time, round down to the hour,
and then subtract an hour to get the time slice for raw data to be aggregated. For
example, suppose the data purge job is running during the 20:00 hour. The time slice over
which we aggregate raw data will be 19:00 to 20:00. We use a custom index table to query
for schedule ids that have data to be aggregated. The row keys are timestamps that
correspond to the start of a particular time slice. The index gets updated whenever data
is inserted. So for any raw data inserted during the 19:00 hour, the corresponding
schedule id will have an entry in the 19:00 hour row. After the raw data for the schedules
is aggregated, the row is deleted.
If I go to control.jsp at 20:25 and execute the data purge job, it will attempt to
aggregate raw data for hour 19:00, not 20:00. And because the 19:00 hour row in the index
is deleted, we will get an empty result set when querying for schedule ids with data to
aggregate. If people utilize control.jsp to perform on demand aggregation, then we can
look at making changes to continue supporting that behavior.
- John