Branch: refs/heads/jsanda/metrics-schema
Home: https://github.com/rhq-project/rhq
Commit: 71def90d385545d0e9bcd35c6471690b57dda4bd
https://github.com/rhq-project/rhq/commit/71def90d385545d0e9bcd35c6471690b5…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/common/cassandra-schema/src/main/resources/schema/update/0002.xml
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/SchedulingSynchronizer.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregateCacheMapper.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationType.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchResult.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheMapper.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ComputeMetric.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/RawCacheMapper.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/AggregateNumericMetric.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/AggregateType.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/MetricsTable.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/NumericMetric.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/RawNumericMetric.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsPerfTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/MeasurementAggregator.java
M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/Simulator.java
Log Message:
-----------
first cut at metrics_cache table which replaces metrics_index table
The number of reads performed during aggregation grow linearly in proportion to
the number of schedules with data to be aggregated. In the worst case scenario
when 1 hr and 6 hr data are aggregated along with raw data, the number of reads
is 3 * N where N is the number of schedules. If there are a million schedules
with data, then a million reads have to be performed just to aggregate the raw
data.
The new metrics_cache table allows us to dramatically reduce the number of
reads. When raw data for example is stored it is now written to both the
raw_data table as well as the metrics_cache table. A single partition, i.e.,
row, in metrics_cache contains data for some configurable number of schedules.
Right now it defaults to 100. Each partition in metrics_cache contains data
only for a given time slice, e.g., 18:00 to 19:00. Once aggregation has run for
that time slice, the partition is deleted.
If we have a million schedules with raw data and 100 schedules are being stored
per partition in metrics_cache, then the number of reads drops from 1 million
to 10,000. That is a 99% reduction in the number of reads. And this reduction
will not come with a sharp increase in memory utilization.
Commit: 4f81346dfabcddaadf9260790057431cfaebeb83
https://github.com/rhq-project/rhq/commit/4f81346dfabcddaadf9260790057431cf…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/SchedulingSynchronizer.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/TaskTracker.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator2.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java
Log Message:
-----------
Some refactoring and fixing bug in TaskTracker
Moving code to process batches into Aggregator2 (which will be renamed). Part
of the reason for this refactoring is logging. Info or debug level logging can
be turned on for AggregationManager without flooding the log file with a lot
of log statements.
TaskTracker provides synchronization for schedules tasks. It is used to track
when all of the scheduled tasks have completed. CountDownLatch cannot be used
anymore because it requires you to specify the number of tasks or events up
front. This worked fine with the metrics_index table since we were only
querying against a single row, and we would iterate over the result set to get
the count. It is not know in advance though how many rows there will be in
metrics_cache so a different sychronization mechanism was needed.
Lastly I should mention that I am not referring to CQL rows here but rather
rows at the actual storage layer, i.e., thrift rows.
Commit: 16394d71d3f7276d65261b6e66154f75a1b62d39
https://github.com/rhq-project/rhq/commit/16394d71d3f7276d65261b6e66154f75a…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationState.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator2.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchAggregationScheduler.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchAggregationState.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchAggregator.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Compute1HourData.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Compute24HourData.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Compute6HourData.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/OneHourDataScheduler.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/RawDataScheduler.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/SixHourDataScheduler.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
Log Message:
-----------
more clean up, removing classes that are now unused.
Commit: e0a832aa8b2f13b3209af02fad1aa1b18d759311
https://github.com/rhq-project/rhq/commit/e0a832aa8b2f13b3209af02fad1aa1b18…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/TaskTracker.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregateCacheMapper.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/RawCacheMapper.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/TaskTracker.java
Log Message:
-----------
update logging to output number of schedules aggregated
Commit: 475a5a1210acc5599453b2bafd813d8ba07b50c5
https://github.com/rhq-project/rhq/commit/475a5a1210acc5599453b2bafd813d8ba…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/AggregateCacheMapper.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/CacheMapper.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/RawCacheMapper.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregateCacheMapper.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationType.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheMapper.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/RawCacheMapper.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
Log Message:
-----------
cleaning up and re-enabling some tests
This commit also updates the logic for determine when aggregation last ran
after a server outage. It is a bit more complicated now since with
metrics_cache there can be multiple partitions per time slice; so, we have to
iterate over all possible partitiion keys per time slice, and if we fail to
find any data, only then do we check the previous time slice. It was eaasier
with the metrics_index table because there was only one partition per time
slice.
Commit: 2e61471baf4ee5593d5101b17679ae6ebcec3eb5
https://github.com/rhq-project/rhq/commit/2e61471baf4ee5593d5101b17679ae6eb…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
Log Message:
-----------
re-enabling and updating tests in MetricsServerTest
Commit: 4783571d26fb0799bec74a201fa966b087a17046
https://github.com/rhq-project/rhq/commit/4783571d26fb0799bec74a201fa966b08…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/core/domain/src/main/java/org/rhq/core/domain/measurement/MeasurementSchedule.java
M modules/enterprise/server/itests-2/src/test/java/org/rhq/enterprise/server/measurement/MeasurementDataManagerBeanTest.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/measurement/MeasurementScheduleManagerBean.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/measurement/MeasurementScheduleManagerLocal.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/scheduler/jobs/DataPurgeJob.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/storage/StorageClientManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
Log Message:
-----------
update server code to pass min/max schedule ids to MetricsServer
Commit: e685f431cc0356ad23639b259913114bb1651708
https://github.com/rhq-project/rhq/commit/e685f431cc0356ad23639b259913114bb…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/StorageResultSetFuture.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchException.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchFailureListener.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java
Log Message:
-----------
adding some initial support for failure handling
This commit gives the capability to log stack traces when exceptions occur
while processing a batch of futures. More importantly, this is a step towards
providing fail fast behavior. Suppose we provide a configurable threshold for
failures, and when that threshold is passed, we abort the aggregation so that
it can be retried at some later time.
Commit: 4fb33c8ac0657c81e6b536788cab11e89fe9ba7d
https://github.com/rhq-project/rhq/commit/4fb33c8ac0657c81e6b536788cab11e89…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/TaskTracker.java
Log Message:
-----------
making some changes based on feedback from code review
The wait() notify() calls have been replaced with a CountDownLatch to address
the scenario in which notify() is invoked before wait(). JavaDoc has also been
added to explain usage.
Commit: 0849007d4cecd0d96f345a4a22969c697aea7819
https://github.com/rhq-project/rhq/commit/0849007d4cecd0d96f345a4a22969c697…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/data-migration/src/main/java/org/rhq/server/metrics/migrator/workers/MetricsIndexUpdateAccumulator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/MetricsIndexEntry.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/MetricsIndexEntryMapper.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/MetricsTable.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsPerfTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
Log Message:
-----------
remove references to now obsolete metrics_index table
There is still a reference in the data migration code. That will be addressed
in a separate task.
Commit: 12e9e885962e1a5866c42d0f7114cd6f2d99a652
https://github.com/rhq-project/rhq/commit/12e9e885962e1a5866c42d0f7114cd6f2…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationType.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
Log Message:
-----------
changing naming convention for the bucket column in metrics_cache
Previously when inserting a row into metrics_index, the bucket column would
have the next wider bucket as its value. For example, if we inserted raw data,
then we would update metrics_index with the bucket column having a value of
"one_hour_metrics". With the schema changes to start using metrics_cache in
place of metrics_index, this approach got confusing. The bucket column now
is set to the name of the historical table in which data is being written.
Commit: f69a6d7103c9c259757e0d4617cf7c3ca226ccdc
https://github.com/rhq-project/rhq/commit/f69a6d7103c9c259757e0d4617cf7c3ca…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/common/cassandra-schema/src/main/resources/schema/update/0002.xml
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntry.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntryMapper.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/MetricsTable.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java
Log Message:
-----------
adding metrics_cache_index table to schema along with initial query support
Commit: 24777143a8ea5f20ed133682573272deb37f4773
https://github.com/rhq-project/rhq/commit/24777143a8ea5f20ed133682573272deb…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntryMapper.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/WaitForRawInserts.java
Log Message:
-----------
update metrics_cache_index when inserting raw data
Commit: 65b8e7689163d4e3a7d843ecae9449e73cba9454
https://github.com/rhq-project/rhq/commit/65b8e7689163d4e3a7d843ecae9449e73…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
Log Message:
-----------
updating, refactoring tests
Commit: 3171180cfab1c23169d306e3d10193ef43da619b
https://github.com/rhq-project/rhq/commit/3171180cfab1c23169d306e3d10193ef4…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
Log Message:
-----------
do not insert raw data that is too old
This initial support caps the age at 24 hours. The plan going forward however
is to make that cap configurable.
Commit: e8ee1a995e09a33105b3e0e0c23f469a25b38f05
https://github.com/rhq-project/rhq/commit/e8ee1a995e09a33105b3e0e0c23f469a2…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchResult.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
A modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/InMemoryMetricsDB.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/Simulator.java
Log Message:
-----------
update metrics_cache and metrics_cache_index for 1 hour and 6 hour data
This commit also introduces a big refactoring of AggregationTests. Tracking and
updating expected values was tedious and error prone; consequently, I
introduced InMemoryMetricsDB.
Commit: 3da0567bc69c767ed0d03250108a48cf36a986eb
https://github.com/rhq-project/rhq/commit/3da0567bc69c767ed0d03250108a48cf3…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/InMemoryMetricsDB.java
Log Message:
-----------
re-enabling 24 hour data tests
Commit: 10225eb431426f11145cb449549732c497182c3c
https://github.com/rhq-project/rhq/commit/10225eb431426f11145cb449549732c49…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/common/cassandra-schema/src/main/resources/schema/update/0002.xml
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntryMapper.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/NumericMetric.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/InMemoryMetricsDB.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
Log Message:
-----------
adding initial support for aggregating late/past data
There are two main changes in this commit. The first is the initial suppport
for aggregating late data. That is, raw data that is stored after its time
slice has already passed.
The schema for metrics_cache_index is changed slightly as well. We now order by
the time column before the start_schedule_id column. And we are always storing
the day, i.e., hour zero in the time_slice column. This reduces the number of
queries that need to be performed when looking to see if there is past data to
aggregate.
There is still work to do for handling late data along with a bunch of clean
up and refactoring. I am trying to consolidate the logic as much as possible
for aggregating past data and for current data so as to avoid a lot of the code
duplication that is going on right now.
Commit: 39555d482723798c755967b62f78222a91166b52
https://github.com/rhq-project/rhq/commit/39555d482723798c755967b62f78222a9…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchResult.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheIterator.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ComputeMetric.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/IndexAggregatesPair.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntry.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsPerfTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
Log Message:
-----------
major refactoring to consolidate common aggregation functions
After adding initial support for aggregating old/past data, it was evident that
there was a lot of duplicate functionality around aggregating past and current
data. This commit introduces a new class, BaseAggregator, which contains common
functions. Aggregation has been renamed to CacheAggregator since it only works
with data out of the metrics_cache table.
This is also the initial commit for CacheIterator. The computeAggregates
function, in its previous forms, took a list of result sets as input. The
computations to be performed for past and current data is the same, but the
format of the input differed. Now the input is an Iterable so that the function
can be used with both past and current data.
Commit: cb43520f62388a599fc938d5529b0f69d8937a6f
https://github.com/rhq-project/rhq/commit/cb43520f62388a599fc938d5529b0f69d…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/core/domain/src/main/java/org/rhq/core/domain/measurement/MeasurementSchedule.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/measurement/MeasurementScheduleManagerBean.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/measurement/MeasurementScheduleManagerLocal.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/scheduler/jobs/DataPurgeJob.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
Log Message:
-----------
updating logic for determining most recent raw data after server outage
This commit also remove the logic around the min/max scheudle ids. Previously
we needed the min and max schedule ids in order to determine the possible set
of rows in metrics_cache. This is no longer necessary though since we track the
schedule ids in metrics_cache_index.
Commit: d89d1782af521838bc73364d0b9a83a4548bb1ae
https://github.com/rhq-project/rhq/commit/d89d1782af521838bc73364d0b9a83a45…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/DateTimeServiceTest.java
Log Message:
-----------
adding more util methods and javadoc to DateTimeService
Commit: 4825b758d903b13b51a39bbb411db21cc0d7e532
https://github.com/rhq-project/rhq/commit/4825b758d903b13b51a39bbb411db21cc…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
Log Message:
-----------
add support for aggregating cache blocks in PastDataAggregator
If aggregation for a cache block (i.e., partition in metrics_cache) fails,
neither the partition nor the corresponding row in metrics_cache_index are
deleted. When a subsequent aggregation run is performed, PastDataAggregator
will retry those failed partitions. It can be assumed that all data for the
time slice is present in metrics_cache; therefore, we can pull data from
metrics_cache instead of raw_metrics and potentially reduce query and I/O
overhead.
Commit: e3d81d7ab7e7ae1dda019cddc28a56f37179136a
https://github.com/rhq-project/rhq/commit/e3d81d7ab7e7ae1dda019cddc28a56f37…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CombinedMetricsIterator.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CombinedMetricsPair.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
Log Message:
-----------
when the 6 hour time slice has passed for late data, recompute the 6 hour data
This commit is best explained with an example. Suppose the current time is
06:12. We receive a measurement report with data having a timestamp of 05:50.
Aggregation will run again at 07:00. The 6 hour time slice for the late data is
00:00 to 06:00, and that time slice has already passed. When aggregation runs,
we not only compute the 1 hour aggregate for the data inserted at 05:50, we
also recompute the 6 hour aggregate so that the late data is included.
Commit: 3382bde839e8e56e47838dc3b8759b94e7364b9c
https://github.com/rhq-project/rhq/commit/3382bde839e8e56e47838dc3b8759b94e…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
Log Message:
-----------
update cache and cache index only when necessary for late data
When aggregating late data whose 6 hour time slice has not yet finished, we
should update metrics_cache and metrics_cache_index so that the necessary rows
are present when 6 hour data aggregation runs. If the 6 hour time slice has
already passed, then there is no need to update metrics_cache and
metrics_cache_index because we go ahead and recompute the 6 hour data. The
same idea holds with 24 hour data. If the 24 hour time slice has not yet
finished, then should update metrics_cache and metrics_cache_index so that the
necessary rows are present when 24 hour data aggregation runs; however, if the
24 hour time slice has already passed, then there is no need to perform the
updates because we will (in the next commit) go ahead and recompute 24 hour
data.
Commit: c20ffb33af5f49de66a3440575a17e3d7304bcfe
https://github.com/rhq-project/rhq/commit/c20ffb33af5f49de66a3440575a17e3d7…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CombinedMetricsIterator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CombinedMetricsPair.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/MetricsFuturesPair.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
Log Message:
-----------
adding support for recomputing 24 hour data
Commit: 36b8d40d4e2da818d2b9e8806f1eadc6d6a1ef3b
https://github.com/rhq-project/rhq/commit/36b8d40d4e2da818d2b9e8806f1eadc6d…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PersistFunctions.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/AggregateNumericMetric.java
Log Message:
-----------
refactoring some functions into a utility class
This commit also updates the logic in PastDataAggregator for how it invokes the
processCacheBlock method. If the 6 hour time slice has finished, it no longer
updates the metrics_cache and metrics_cache_index tables. There are additional
changes needed for when PastDataAggregator calls processCacheBlock. It needs to
determine if the 6 hour and 24 hour data need to be recomputed similar to what
is done in the processBatch method.
Commit: aa08c5bd1c6f5a3d409631b349f9a02b00897a23
https://github.com/rhq-project/rhq/commit/aa08c5bd1c6f5a3d409631b349f9a02b0…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
Log Message:
-----------
recompute 6 hour data when aggregating late data out of metrics_cache table
If the 6 hour time slice of the late data has passed, we need to recompute the
6 hour data just as we do when we pull data from raw_metrics.
Commit: 9a1c812190704ed73245a80c52f9717184e7244a
https://github.com/rhq-project/rhq/commit/9a1c812190704ed73245a80c52f971718…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/MeasurementAggregator.java
Log Message:
-----------
updating test for recomputing 24 hr data when pulling data from metrics_cache
Commit: 5fba9e68e76431ef9c06ef972bf2b813c6dd3468
https://github.com/rhq-project/rhq/commit/5fba9e68e76431ef9c06ef972bf2b813c…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
Log Message:
-----------
refactoring task scheduling logic
Task scheduling logic has been pulled up into BaseAggregator which is now an
abstract class. Subclasses are responsible for creating the Runnable tasks.
Commit: d1562c79b5f83fa1d7f22264b7127c05b656956d
https://github.com/rhq-project/rhq/commit/d1562c79b5f83fa1d7f22264b7127c05b…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
Log Message:
-----------
more refactoring to correctly compute the number of schedules processed
Commit: 3671fd69f9f3efe1556ef59335047e8f2e9c61eb
https://github.com/rhq-project/rhq/commit/3671fd69f9f3efe1556ef59335047e8f2…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
Log Message:
-----------
introduce AggregationTask to provide error handling
Commit: 588f47f05e34d93d1900e1f31cd04b163cc1acee
https://github.com/rhq-project/rhq/commit/588f47f05e34d93d1900e1f31cd04b163…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/common/cassandra-schema/src/main/resources/schema/update/0002.xml
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PersistFunctions.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntry.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntryMapper.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
Log Message:
-----------
making schema changes to metrics_cache_index table
Some columns have been renamed and a new column has been added. We now have the
following timestamp columns - day, collection_time_slice, and insert_time_slice.
collection_time_slice is the timestamp provided by the reported data.
insert_time_slice is the raw time slice in which the data is inserted. For
example, if data is inserted at 17:02, insert_time_slice will be 17:00. And
day is the 24 hour time slice of the collection timestamp.
insert_time_slice is the new column. Previously there was a scenario in which
we could fail to aggregate data. Suppose aggregation for a cache partition
failed. Its cache index entry would be left intact., and note that the set of
schedule ids was empty. Then late data is reported in the next time slice. At
that point, the cache index entry is updated with the schedule ids of the late
data. Prior to this commit we would have only aggregated data for the schedule
ids with late data.
Now we always update the schedule ids for the cache index, even for current
data. When we have past/late data, we now compare collection_time_slice with
insert_time_slice to determine if we can pull data from metrics_cache.
Commit: 9254137e0e2e1799a5278e60ea6fe94247a712df
https://github.com/rhq-project/rhq/commit/9254137e0e2e1799a5278e60ea6fe9424…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
Log Message:
-----------
fix scheduling logic for past data aggregation tasks
With the addition of the insert_time_slice column, we can now have multiple
index entries for a {column_time_slice, start_schedule_id} pair. We can reduce
those into a single index entry and submit an aggregation task for that
combined entry.
PastDataAggregator also now has a deleteCacheIndexEntries function to take care
of deleting all index entries for a {column_time_slice, start_schedule_id} pair.
Commit: a96f1667f7f30187d90baf49f751f9a05327d9c1
https://github.com/rhq-project/rhq/commit/a96f1667f7f30187d90baf49f751f9a05…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntry.java
Log Message:
-----------
more refactoring of processing index entries and adding javadocs
Commit: 2f5aef7994df9fcf48e8333337fbf93eeed90650
https://github.com/rhq-project/rhq/commit/2f5aef7994df9fcf48e8333337fbf93ee…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsPerfTests.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java
R modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/SimulatedMetricsServer.java
M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/Simulator.java
Log Message:
-----------
refactoring tests and adding javadocs
Commit: a813f7238fbe30e2a0793b3aae2ac33999b004c1
https://github.com/rhq-project/rhq/commit/a813f7238fbe30e2a0793b3aae2ac3399…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
Log Message:
-----------
use the same insert time slice for each raw data
There is no need to recompute the insert time slice for each raw data. It is
sufficient to use insert time slice timestamp as the starting point at which
we beging inserting raw data.
Commit: 653ed367c97a48b32998c4afe8f0a97564418e80
https://github.com/rhq-project/rhq/commit/653ed367c97a48b32998c4afe8f0a9756…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CombinedMetricsIterator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/AggregateNumericMetricMapper.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntry.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsPerfTests.java
Log Message:
-----------
do not schedule aggregation tasks on I/O thread
I had previously and inadvertently perfoming the scheduling of aggregation
tasks on an I/O thread. Long running tasks should never run on an I/O thread.
I discovered this mistake in a test environment where I was generating
NoHostAvailableExceptions because of this mistake.
This commit also makes a subtle yet significant change in
AggregateNumericMetricMapper. It now checks for nulls which it should have been
doing all along since we use weak consistency for reads and writes. The change
was needed because I was hitting occassional NPEs with past data aggregation
when 1 hr and/or 6 hr aggregates have to be computed. It is easiest to explain
with an example.
Suppose we are aggregating some past data and we also need to aggregate the
1 hour data into 6 hour data. Almost immediately after computing and storing
the 1 hour data for the given time slice, we then fetch all of the 1 hour data
for the 6 hour time slice. Unless the query is against the same node to which
the preceding write went, the query results might not include all of the data.
We do not have to worry about the missing data in the result sets though
because we already have the data in memory.
Commit: 2b1a1f2b53c0dad019ce85bca04b08def98c1b38
https://github.com/rhq-project/rhq/commit/2b1a1f2b53c0dad019ce85bca04b08def…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
Log Message:
-----------
updating logging and adding javadocs
Commit: f5e109dad5d8025ffc22710d67520d0aa79114ba
https://github.com/rhq-project/rhq/commit/f5e109dad5d8025ffc22710d67520d0aa…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
Log Message:
-----------
add support for "decactivating" metrics_cache
Deactivating metrics_cache results in data *not* being pulled from it. This
applies to both past and current data. Once a specified time has been reached,
it is reactived and we start pulling from metrics_cache again. This logic is
necessary for two purrposes. First, we need it simplify and speed up the data
migration that will be necessary when upgrading to RHQ 4.11. Secondly, it is
necessary for the resizing of cache partitions to take effect. For example,
suppose we store data for up to 5 schedules per cache partition. We decide
to store data for 10 schedules per partition. We need to deactivate the cache
while the change takes effect.
Commit: 5e7c2459546fc22e2c04ca5ca3a5620d7a1e3616
https://github.com/rhq-project/rhq/commit/5e7c2459546fc22e2c04ca5ca3a5620d7…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-22 (Tue, 22 Apr 2014)
Changed paths:
M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java
M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/Simulator.java
Log Message:
-----------
fixing merge conflicts from rebase
Commit: 3338fc8dee082423d9cc2c72afa41f2e6e9624f5
https://github.com/rhq-project/rhq/commit/3338fc8dee082423d9cc2c72afa41f2e6…
Author: John Sanda <jsanda(a)redhat.com>
Date: 2014-04-24 (Thu, 24 Apr 2014)
Changed paths:
M modules/common/cassandra-schema/pom.xml
M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/AbstractManager.java
A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/CQLStep.java
A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/DBConnectionFactory.java
A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/PopulateCacheIndex.java
M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/SchemaManager.java
A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/SchemaUpdateThreadFactory.java
A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/Step.java
M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/UpdateFile.java
M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/VersionManager.java
M modules/common/cassandra-schema/src/main/resources/schema/update/0002.xml
A modules/common/cassandra-schema/src/main/resources/schema/update/0003.xml
M modules/common/cassandra-schema/src/test/java/org/rhq/cassandra/schema/UpdateFileTest.java
M modules/core/dbutils/pom.xml
M modules/core/dbutils/src/main/scripts/dbsetup/sysconfig-data.xml
M modules/core/dbutils/src/main/scripts/dbupgrade/db-upgrade.xml
M modules/core/domain/src/main/java/org/rhq/core/domain/common/composite/SystemSetting.java
M modules/enterprise/server/installer/pom.xml
M modules/enterprise/server/installer/src/main/java/org/rhq/enterprise/server/installer/InstallerServiceImpl.java
M modules/enterprise/server/installer/src/main/resources/module/main/module.xml
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/storage/StorageClientManager.java
M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
Log Message:
-----------
Adding data migration support needed during upgrades
The C* schema management tool has been changed such that it now supports
arbitrary tasks in addition to raw CQL statements. An arbitrary task/step must
implement the new Step interface. PopulateCacheIndex is one such custom task
that migrates data out of metrics_index and into metrics_cache_index.
A property has been added to the system settings for the cache activation time.
PopulateCacheIndex sets that property so the server knows when it can start
pulling data from metrics_cache during aggregation. And at server start up,
MetricsServer is initialized with the property value.
Compare: https://github.com/rhq-project/rhq/compare/8211d77fcbc6...3338fc8dee08
Branch: refs/heads/master
Home: https://github.com/rhq-project/rhq
Commit: 6594868123e3486d2d9b1ff4066918b6ef0e9c03
https://github.com/rhq-project/rhq/commit/6594868123e3486d2d9b1ff4066918b6e…
Author: John Mazzitelli <mazz(a)redhat.com>
Date: 2014-04-24 (Thu, 24 Apr 2014)
Changed paths:
M modules/core/domain/src/main/java/org/rhq/core/domain/install/remote/AgentInstallInfo.java
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/admin/agent/install/RemoteAgentInstallView.java
M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages.properties
M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_cs.properties
M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_de.properties
M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_ja.properties
M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_ko.properties
M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_pt.properties
M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_ru.properties
M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_zh.properties
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/core/CoreServerServiceImpl.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/install/remote/RemoteInstallManagerBean.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/install/remote/SSHInstallUtility.java
Log Message:
-----------
BZ 1090453 - try to see if we can make a low level TCP connection to the agent after it has been installed - this helps detect if its behind a firewall and can't be reached by the server
Branch: refs/heads/BZ1069545
Home: https://github.com/rhq-project/rhq
Commit: 2a7cdf37075ae2ad31d29ca144ac5ab9063c090b
https://github.com/rhq-project/rhq/commit/2a7cdf37075ae2ad31d29ca144ac5ab90…
Author: Stefan Negrea <snegrea(a)redhat.com>
Date: 2014-04-24 (Thu, 24 Apr 2014)
Changed paths:
M modules/core/dbutils/src/main/java/org/rhq/core/db/upgrade/SubcategoryUpgradeTask.java
M modules/core/domain/src/main/java/org/rhq/core/domain/criteria/ResourceTypeCriteria.java
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/groups/detail/ResourceGroupContextMenu.java
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/groups/detail/ResourceGroupTreeView.java
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/resource/detail/ResourceTreeDatasource.java
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/resource/detail/ResourceTreeView.java
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/resource/type/ResourceTypeRepository.java
M modules/enterprise/gui/portal-war/src/main/java/org/rhq/enterprise/gui/navigation/resource/ResourceTreeModelUIBean.java
M modules/enterprise/server/itests-2/src/test/java/org/rhq/enterprise/server/resource/metadata/PluginExtensionMetadataTest.java
M modules/enterprise/server/itests-2/src/test/java/org/rhq/enterprise/server/resource/metadata/test/UpdatePluginMetadataTestBase.java
Log Message:
-----------
[BZ 1069545] Removing more undeeded code for fetching subcategories. Subcategories are now just a proper field on the resource type.
Branch: refs/heads/BZ1069545
Home: https://github.com/rhq-project/rhq
Commit: 089a19c2c86889bfc45ffc0ce22b8b3c53487046
https://github.com/rhq-project/rhq/commit/089a19c2c86889bfc45ffc0ce22b8b3c5…
Author: Stefan Negrea <snegrea(a)redhat.com>
Date: 2014-04-24 (Thu, 24 Apr 2014)
Changed paths:
M modules/core/dbutils/pom.xml
M modules/core/dbutils/src/main/scripts/dbsetup/inventory-schema.xml
M modules/core/dbutils/src/main/scripts/dbupgrade/db-upgrade.xml
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/Resource.java
M modules/enterprise/server/itests-2/src/test/java/org/rhq/enterprise/server/resource/metadata/ResourceMetadataManagerBeanTest.java
Log Message:
-----------
[BZ 1069545] Fix queries related to subcategories.
Branch: refs/heads/master
Home: https://github.com/rhq-project/rhq
Commit: 251d60aa898402d28338a43bc844a3d69d4aff7d
https://github.com/rhq-project/rhq/commit/251d60aa898402d28338a43bc844a3d69…
Author: Mike Thompson <mithomps(a)redhat.com>
Date: 2014-04-23 (Wed, 23 Apr 2014)
Changed paths:
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/common/graph/MetricGraphData.java
Log Message:
-----------
[BZ 1089108] File Systems -> Free Space graph off by a factor of 1000. This was due to a recent bug fix introduced for BZ1080470.
Branch: refs/heads/BZ1069545
Home: https://github.com/rhq-project/rhq
Commit: 641598c4b634a863f95536435f3ff41fbe549a8f
https://github.com/rhq-project/rhq/commit/641598c4b634a863f95536435f3ff41fb…
Author: Stefan Negrea <snegrea(a)redhat.com>
Date: 2014-04-23 (Wed, 23 Apr 2014)
Changed paths:
M modules/plugins/jboss-as-7/src/main/resources/META-INF/rhq-plugin.xml
Log Message:
-----------
[BZ 1069545] Add Subsystems and Server Configuration subcategories.
Commit: ca375271ac26c5c83dfffdf9fa18deab54b0a7aa
https://github.com/rhq-project/rhq/commit/ca375271ac26c5c83dfffdf9fa18deab5…
Author: Stefan Negrea <snegrea(a)redhat.com>
Date: 2014-04-23 (Wed, 23 Apr 2014)
Changed paths:
A modules/core/dbutils/src/main/java/org/rhq/core/db/upgrade/SubcategoryUpgradeTask.java
M modules/core/dbutils/src/main/scripts/dbsetup/inventory-schema.xml
M modules/core/dbutils/src/main/scripts/dbupgrade/db-upgrade.xml
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/ResourceType.java
Log Message:
-----------
[BZ 1069545] Add a ManyToMany relationship between resource type and proper subcategories. Also, add new table, upgrade schema, and upgrade tasks to move to the new model.
Commit: 27c31e4d64b61f3a703821b651a632256feaaf3f
https://github.com/rhq-project/rhq/commit/27c31e4d64b61f3a703821b651a632256…
Author: Stefan Negrea <snegrea(a)redhat.com>
Date: 2014-04-23 (Wed, 23 Apr 2014)
Changed paths:
M modules/core/client-api/src/main/java/org/rhq/core/clientapi/agent/metadata/SubCategoriesMetadataParser.java
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/ResourceSubCategory.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/resource/metadata/ResourceMetadataManagerBean.java
Log Message:
-----------
[BZ 1069545] Clean the subcategory class as well the persisting layers.
Commit: 4ca41e2193d10e6de2b1a6663ac01ee027b26859
https://github.com/rhq-project/rhq/commit/4ca41e2193d10e6de2b1a6663ac01ee02…
Author: Stefan Negrea <snegrea(a)redhat.com>
Date: 2014-04-23 (Wed, 23 Apr 2014)
Changed paths:
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/ResourceSubCategory.java
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/ResourceType.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/resource/metadata/PluginManagerBean.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/resource/metadata/ResourceMetadataManagerBean.java
Log Message:
-----------
[BZ 1069545] Add code that indepdently persists the subcategories.
Commit: 28d90bb194ffab0feba918238733a26e4cf4f56a
https://github.com/rhq-project/rhq/commit/28d90bb194ffab0feba918238733a26e4…
Author: Stefan Negrea <snegrea(a)redhat.com>
Date: 2014-04-23 (Wed, 23 Apr 2014)
Changed paths:
M modules/core/client-api/src/main/java/org/rhq/core/clientapi/agent/metadata/PluginMetadataParser.java
M modules/core/client-api/src/main/java/org/rhq/core/clientapi/agent/metadata/SubCategoriesMetadataParser.java
M modules/core/client-api/src/test/java/org/rhq/core/clientapi/agent/metadata/test/ExtensionModelTest.java
M modules/core/client-api/src/test/java/org/rhq/core/clientapi/agent/metadata/test/NestedSubCategoriesMetadataParserTest.java
M modules/core/client-api/src/test/java/org/rhq/core/clientapi/agent/metadata/test/SubCategoriesMetadataParserTest.java
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/ResourceSubCategory.java
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/ResourceType.java
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/flyweight/AutoGroupCompositeFlyweight.java
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/flyweight/FlyweightCache.java
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/flyweight/ResourceSubCategoryFlyweight.java
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/flyweight/ResourceTypeFlyweight.java
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/group/composite/AutoGroupComposite.java
M modules/core/plugindoc/src/main/java/org/rhq/core/tool/plugindoc/PluginDescriptorProcessor.java
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/groups/detail/ResourceGroupTreeView.java
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/resource/detail/ResourceTreeDatasource.java
M modules/enterprise/gui/portal-war/src/main/java/org/rhq/enterprise/gui/navigation/group/ResourceGroupTreeNode.java
M modules/enterprise/gui/portal-war/src/main/java/org/rhq/enterprise/gui/navigation/resource/ResourceTreeModelUIBean.java
M modules/enterprise/server/appserver/src/main/scripts/storage_setup.groovy
M modules/enterprise/server/itests-2/src/test/java/org/rhq/enterprise/server/resource/metadata/PluginExtensionMetadataTest.java
M modules/enterprise/server/itests-2/src/test/java/org/rhq/enterprise/server/resource/metadata/test/UpdateResourceSubsystemTest.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/resource/ResourceManagerBean.java
M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/resource/metadata/ResourceMetadataManagerBean.java
Log Message:
-----------
[BZ 1069545] First cut at removing the predefined subcategories concept. This version has some functionality removed but will be added in later revisions.
Commit: 53973495c6223126cc78d814b9f6abd422f09d8f
https://github.com/rhq-project/rhq/commit/53973495c6223126cc78d814b9f6abd42…
Author: Stefan Negrea <snegrea(a)redhat.com>
Date: 2014-04-23 (Wed, 23 Apr 2014)
Changed paths:
M etc/eclipse-tools/hibernate.cfg.xml
M modules/core/dbutils/pom.xml
M modules/core/domain/src/main/java/org/rhq/core/domain/resource/ResourceType.java
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/groups/detail/ResourceGroupTreeView.java
M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/resource/detail/ResourceTreeDatasource.java
Log Message:
-----------
[BZ 1069545] Gui changes for the subcategory generation. The subcategories are now driven by attributes on the child resource type.
Compare: https://github.com/rhq-project/rhq/compare/6aa9a3a32f2b...53973495c622
Branch: refs/heads/release/jon3.2.x
Home: https://github.com/rhq-project/rhq
Commit: ea6fdbdfefbcbd4c526d36c33fa90dfbe6f55fbd
https://github.com/rhq-project/rhq/commit/ea6fdbdfefbcbd4c526d36c33fa90dfbe…
Author: John Mazzitelli <mazz(a)redhat.com>
Date: 2014-04-23 (Wed, 23 Apr 2014)
Changed paths:
M modules/enterprise/server/appserver/src/main/bin-resources/bin/internal/rhq-server.sh
Log Message:
-----------
BZ 1083557 - take out usage of $() syntax since Bourne shell doesn't like it (which affects Solaris 10 for example)