rhq-commits April 2014

rhq-commits@lists.fedorahosted.org

16 participants
176 discussions

[rhq-project/rhq] 074e64: [665508] rhq:url-archive, rhq:archive should suppo...
by Jay Shaughnessy 29 Apr '14

29 Apr '14

Branch: refs/heads/master Home: https://github.com/rhq-project/rhq Commit: 074e649afda89f314679a0c65676aa41f80f420b https://github.com/rhq-project/rhq/commit/074e649afda89f314679a0c65676aa41f… Author: Jay Shaughnessy <jshaughn(a)redhat.com> Date: 2014-04-29 (Tue, 29 Apr 2014) Changed paths: M modules/common/ant-bundle/src/main/java/org/rhq/bundle/ant/type/DeploymentUnitType.java M modules/core/util/src/main/java/org/rhq/core/util/updater/Deployer.java M modules/core/util/src/main/java/org/rhq/core/util/updater/DeploymentData.java M modules/core/util/src/test/java/org/rhq/core/util/updater/DeployerCanonicalPathTest.java M modules/core/util/src/test/java/org/rhq/core/util/updater/DeployerTest.java M modules/core/util/src/test/java/org/rhq/core/util/updater/ManageRootDirTest.java M modules/core/util/src/test/java/org/rhq/core/util/updater/SimpleDeployerRawFileTest.java M modules/core/util/src/test/java/org/rhq/core/util/updater/SimpleDeployerTest.java Log Message: ----------- [665508] rhq:url-archive, rhq:archive should support destinationFile/destinationDir - Fix API back-compat issues in DeploymentData constructors - Fix ambiguous constructors issue in DeploymentData when using various null params - Fix some related jdoc - Update our code base to not use deprecated calls in DeploymentProperties and DeploymentData.

1 0

[rhq-project/rhq] 7161d4: BZ 1090448 - this got munged - need to encode in U...
by John Mazzitelli 29 Apr '14

29 Apr '14

Branch: refs/heads/master Home: https://github.com/rhq-project/rhq Commit: 7161d475d9a33de058cff57f05f3c5cb8bd0b865 https://github.com/rhq-project/rhq/commit/7161d475d9a33de058cff57f05f3c5cb8… Author: John Mazzitelli <mazz(a)redhat.com> Date: 2014-04-29 (Tue, 29 Apr 2014) Changed paths: M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_cs.properties Log Message: ----------- BZ 1090448 - this got munged - need to encode in UTF-8.

1 0

[rhq-project/rhq] e68678: [BZ 1076104] - Fix the itest failures in plugin pu...
by Lukas Krejci 28 Apr '14

28 Apr '14

Branch: refs/heads/master Home: https://github.com/rhq-project/rhq Commit: e686783f01689b41968e744d5c297cf3bae09cef https://github.com/rhq-project/rhq/commit/e686783f01689b41968e744d5c297cf3b… Author: Lukas Krejci <lkrejci(a)redhat.com> Date: 2014-04-29 (Tue, 29 Apr 2014) Changed paths: M modules/enterprise/server/itests-2/src/test/java/org/rhq/enterprise/server/resource/metadata/PluginManagerBeanTest.java Log Message: ----------- [BZ 1076104] - Fix the itest failures in plugin purge tests.

1 0

[rhq-project/rhq] 4d736f: BZ 1090448 - make the remote agent install UI more...
by John Mazzitelli 28 Apr '14

28 Apr '14

Branch: refs/heads/master Home: https://github.com/rhq-project/rhq Commit: 4d736ffdb35d4e4d7bce7aeeef477dbcc8ef2992 https://github.com/rhq-project/rhq/commit/4d736ffdb35d4e4d7bce7aeeef477dbcc… Author: John Mazzitelli <mazz(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/admin/agent/install/RemoteAgentInstallView.java M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/admin/topology/AgentTableView.java M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages.properties M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_cs.properties M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_de.properties M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_ja.properties M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_ko.properties M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_pt.properties M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_ru.properties M modules/enterprise/gui/coregui/src/main/resources/org/rhq/coregui/client/Messages_zh.properties Log Message: ----------- BZ 1090448 - make the remote agent install UI more pretty

1 0

[rhq-project/rhq] bd59fc: [BZ 1069545] More updates to the tree build struct...
by Stefan Negrea 28 Apr '14

28 Apr '14

Branch: refs/heads/BZ1069545 Home: https://github.com/rhq-project/rhq Commit: bd59fcacf5657f00689da148a35ce87a32eb37cd https://github.com/rhq-project/rhq/commit/bd59fcacf5657f00689da148a35ce87a3… Author: Stefan Negrea <snegrea(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/groups/detail/ResourceGroupTreeView.java Log Message: ----------- [BZ 1069545] More updates to the tree build structure.

1 0

[rhq-project/rhq] e47353: first cut at metrics_cache table which replaces me...
by John Sanda 28 Apr '14

28 Apr '14

Branch: refs/heads/jsanda/metrics-schema Home: https://github.com/rhq-project/rhq Commit: e47353b003ebc11168f7869d9bd28c52bc93f094 https://github.com/rhq-project/rhq/commit/e47353b003ebc11168f7869d9bd28c52b… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/common/cassandra-schema/src/main/resources/schema/update/0002.xml M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/SchedulingSynchronizer.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregateCacheMapper.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationType.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchResult.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheMapper.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ComputeMetric.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/RawCacheMapper.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/AggregateNumericMetric.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/AggregateType.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/MetricsTable.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/NumericMetric.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/RawNumericMetric.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsPerfTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/MeasurementAggregator.java M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/Simulator.java Log Message: ----------- first cut at metrics_cache table which replaces metrics_index table The number of reads performed during aggregation grow linearly in proportion to the number of schedules with data to be aggregated. In the worst case scenario when 1 hr and 6 hr data are aggregated along with raw data, the number of reads is 3 * N where N is the number of schedules. If there are a million schedules with data, then a million reads have to be performed just to aggregate the raw data. The new metrics_cache table allows us to dramatically reduce the number of reads. When raw data for example is stored it is now written to both the raw_data table as well as the metrics_cache table. A single partition, i.e., row, in metrics_cache contains data for some configurable number of schedules. Right now it defaults to 100. Each partition in metrics_cache contains data only for a given time slice, e.g., 18:00 to 19:00. Once aggregation has run for that time slice, the partition is deleted. If we have a million schedules with raw data and 100 schedules are being stored per partition in metrics_cache, then the number of reads drops from 1 million to 10,000. That is a 99% reduction in the number of reads. And this reduction will not come with a sharp increase in memory utilization. Commit: b00b191dafb3dae483006dd3ab591ea798c5e31d https://github.com/rhq-project/rhq/commit/b00b191dafb3dae483006dd3ab591ea79… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/SchedulingSynchronizer.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/TaskTracker.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator2.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java Log Message: ----------- Some refactoring and fixing bug in TaskTracker Moving code to process batches into Aggregator2 (which will be renamed). Part of the reason for this refactoring is logging. Info or debug level logging can be turned on for AggregationManager without flooding the log file with a lot of log statements. TaskTracker provides synchronization for schedules tasks. It is used to track when all of the scheduled tasks have completed. CountDownLatch cannot be used anymore because it requires you to specify the number of tasks or events up front. This worked fine with the metrics_index table since we were only querying against a single row, and we would iterate over the result set to get the count. It is not know in advance though how many rows there will be in metrics_cache so a different sychronization mechanism was needed. Lastly I should mention that I am not referring to CQL rows here but rather rows at the actual storage layer, i.e., thrift rows. Commit: 122e51731dab09d16d80eead4cbbd81d21bafced https://github.com/rhq-project/rhq/commit/122e51731dab09d16d80eead4cbbd81d2… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationState.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator2.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchAggregationScheduler.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchAggregationState.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchAggregator.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Compute1HourData.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Compute24HourData.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Compute6HourData.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/OneHourDataScheduler.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/RawDataScheduler.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/SixHourDataScheduler.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java Log Message: ----------- more clean up, removing classes that are now unused. Commit: 67c7cced5fca21e2e2a63f470499ef51c2203955 https://github.com/rhq-project/rhq/commit/67c7cced5fca21e2e2a63f470499ef51c… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/TaskTracker.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregateCacheMapper.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/RawCacheMapper.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/TaskTracker.java Log Message: ----------- update logging to output number of schedules aggregated Commit: 387078d556cd41d598298c7db03e1861c89888bb https://github.com/rhq-project/rhq/commit/387078d556cd41d598298c7db03e1861c… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/AggregateCacheMapper.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/CacheMapper.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/RawCacheMapper.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregateCacheMapper.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationType.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheMapper.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/RawCacheMapper.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java Log Message: ----------- cleaning up and re-enabling some tests This commit also updates the logic for determine when aggregation last ran after a server outage. It is a bit more complicated now since with metrics_cache there can be multiple partitions per time slice; so, we have to iterate over all possible partitiion keys per time slice, and if we fail to find any data, only then do we check the previous time slice. It was eaasier with the metrics_index table because there was only one partition per time slice. Commit: 7d94956725c0cbb2459bd858211c251f2c67385c https://github.com/rhq-project/rhq/commit/7d94956725c0cbb2459bd858211c251f2… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java Log Message: ----------- re-enabling and updating tests in MetricsServerTest Commit: 5e37a59d4d9a884714e19bc3dd3824fd853cc592 https://github.com/rhq-project/rhq/commit/5e37a59d4d9a884714e19bc3dd3824fd8… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/core/domain/src/main/java/org/rhq/core/domain/measurement/MeasurementSchedule.java M modules/enterprise/server/itests-2/src/test/java/org/rhq/enterprise/server/measurement/MeasurementDataManagerBeanTest.java M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/measurement/MeasurementScheduleManagerBean.java M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/measurement/MeasurementScheduleManagerLocal.java M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/scheduler/jobs/DataPurgeJob.java M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/storage/StorageClientManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java Log Message: ----------- update server code to pass min/max schedule ids to MetricsServer Commit: ff11bf30bd966555e78c106701762f0de396db49 https://github.com/rhq-project/rhq/commit/ff11bf30bd966555e78c106701762f0de… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/StorageResultSetFuture.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchException.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchFailureListener.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java Log Message: ----------- adding some initial support for failure handling This commit gives the capability to log stack traces when exceptions occur while processing a batch of futures. More importantly, this is a step towards providing fail fast behavior. Suppose we provide a configurable threshold for failures, and when that threshold is passed, we abort the aggregation so that it can be retried at some later time. Commit: d5fc0724b727df4633f769201879e5a754fc699c https://github.com/rhq-project/rhq/commit/d5fc0724b727df4633f769201879e5a75… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/TaskTracker.java Log Message: ----------- making some changes based on feedback from code review The wait() notify() calls have been replaced with a CountDownLatch to address the scenario in which notify() is invoked before wait(). JavaDoc has also been added to explain usage. Commit: d88ae4f7fd4910e29a2c2121005532586c278a84 https://github.com/rhq-project/rhq/commit/d88ae4f7fd4910e29a2c2121005532586… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/data-migration/src/main/java/org/rhq/server/metrics/migrator/workers/MetricsIndexUpdateAccumulator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/MetricsIndexEntry.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/MetricsIndexEntryMapper.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/MetricsTable.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsPerfTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java Log Message: ----------- remove references to now obsolete metrics_index table There is still a reference in the data migration code. That will be addressed in a separate task. Commit: 806efb632cf8bbcffcf3c2c912868be3ad30186c https://github.com/rhq-project/rhq/commit/806efb632cf8bbcffcf3c2c912868be3a… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationType.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java Log Message: ----------- changing naming convention for the bucket column in metrics_cache Previously when inserting a row into metrics_index, the bucket column would have the next wider bucket as its value. For example, if we inserted raw data, then we would update metrics_index with the bucket column having a value of "one_hour_metrics". With the schema changes to start using metrics_cache in place of metrics_index, this approach got confusing. The bucket column now is set to the name of the historical table in which data is being written. Commit: 0210edbe55e9914106723fd1843aec6ddb862968 https://github.com/rhq-project/rhq/commit/0210edbe55e9914106723fd1843aec6dd… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/common/cassandra-schema/src/main/resources/schema/update/0002.xml M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntry.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntryMapper.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/MetricsTable.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java Log Message: ----------- adding metrics_cache_index table to schema along with initial query support Commit: e782f303bd088415785880a044e97accba686a1c https://github.com/rhq-project/rhq/commit/e782f303bd088415785880a044e97accb… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntryMapper.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/WaitForRawInserts.java Log Message: ----------- update metrics_cache_index when inserting raw data Commit: 04d318dfdb5ba41d6f4a7cc513dc6d63f7dc446e https://github.com/rhq-project/rhq/commit/04d318dfdb5ba41d6f4a7cc513dc6d63f… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java Log Message: ----------- updating, refactoring tests Commit: b7821c13747a7cf2ba5d34c01673d96b6ff5809a https://github.com/rhq-project/rhq/commit/b7821c13747a7cf2ba5d34c01673d96b6… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java Log Message: ----------- do not insert raw data that is too old This initial support caps the age at 24 hours. The plan going forward however is to make that cap configurable. Commit: 298ceb07347828e12588d79a9e76f84e58aee337 https://github.com/rhq-project/rhq/commit/298ceb07347828e12588d79a9e76f84e5… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchResult.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java A modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/InMemoryMetricsDB.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/Simulator.java Log Message: ----------- update metrics_cache and metrics_cache_index for 1 hour and 6 hour data This commit also introduces a big refactoring of AggregationTests. Tracking and updating expected values was tedious and error prone; consequently, I introduced InMemoryMetricsDB. Commit: 1469e4dbe348006adee92a293a7f38893550a4c2 https://github.com/rhq-project/rhq/commit/1469e4dbe348006adee92a293a7f38893… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/InMemoryMetricsDB.java Log Message: ----------- re-enabling 24 hour data tests Commit: 603120d619c9bf290a4a0d4fdb606e269d10e82e https://github.com/rhq-project/rhq/commit/603120d619c9bf290a4a0d4fdb606e269… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/common/cassandra-schema/src/main/resources/schema/update/0002.xml M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntryMapper.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/NumericMetric.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/InMemoryMetricsDB.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java Log Message: ----------- adding initial support for aggregating late/past data There are two main changes in this commit. The first is the initial suppport for aggregating late data. That is, raw data that is stored after its time slice has already passed. The schema for metrics_cache_index is changed slightly as well. We now order by the time column before the start_schedule_id column. And we are always storing the day, i.e., hour zero in the time_slice column. This reduces the number of queries that need to be performed when looking to see if there is past data to aggregate. There is still work to do for handling late data along with a bunch of clean up and refactoring. I am trying to consolidate the logic as much as possible for aggregating past data and for current data so as to avoid a lot of the code duplication that is going on right now. Commit: d8eca23a787115028214b80a49ae9efa954efc7f https://github.com/rhq-project/rhq/commit/d8eca23a787115028214b80a49ae9efa9… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/Aggregator.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BatchResult.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheIterator.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ComputeMetric.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/IndexAggregatesPair.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java R modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/ProcessBatch.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntry.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsPerfTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java Log Message: ----------- major refactoring to consolidate common aggregation functions After adding initial support for aggregating old/past data, it was evident that there was a lot of duplicate functionality around aggregating past and current data. This commit introduces a new class, BaseAggregator, which contains common functions. Aggregation has been renamed to CacheAggregator since it only works with data out of the metrics_cache table. This is also the initial commit for CacheIterator. The computeAggregates function, in its previous forms, took a list of result sets as input. The computations to be performed for past and current data is the same, but the format of the input differed. Now the input is an Iterable so that the function can be used with both past and current data. Commit: f1afb402d59da8a83b402569121e8559215991c4 https://github.com/rhq-project/rhq/commit/f1afb402d59da8a83b402569121e85592… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/core/domain/src/main/java/org/rhq/core/domain/measurement/MeasurementSchedule.java M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/measurement/MeasurementScheduleManagerBean.java M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/measurement/MeasurementScheduleManagerLocal.java M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/scheduler/jobs/DataPurgeJob.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java Log Message: ----------- updating logic for determining most recent raw data after server outage This commit also remove the logic around the min/max scheudle ids. Previously we needed the min and max schedule ids in order to determine the possible set of rows in metrics_cache. This is no longer necessary though since we track the schedule ids in metrics_cache_index. Commit: 40a76f8ef13f795e59e104a5343c37343c60c312 https://github.com/rhq-project/rhq/commit/40a76f8ef13f795e59e104a5343c37343… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/DateTimeServiceTest.java Log Message: ----------- adding more util methods and javadoc to DateTimeService Commit: 1f76fff8ef95d992b9ff3e2c161f1372864d5dd6 https://github.com/rhq-project/rhq/commit/1f76fff8ef95d992b9ff3e2c161f13728… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java Log Message: ----------- add support for aggregating cache blocks in PastDataAggregator If aggregation for a cache block (i.e., partition in metrics_cache) fails, neither the partition nor the corresponding row in metrics_cache_index are deleted. When a subsequent aggregation run is performed, PastDataAggregator will retry those failed partitions. It can be assumed that all data for the time slice is present in metrics_cache; therefore, we can pull data from metrics_cache instead of raw_metrics and potentially reduce query and I/O overhead. Commit: d70e250e25088c55b397484958cdba8ebdf7c81d https://github.com/rhq-project/rhq/commit/d70e250e25088c55b397484958cdba8eb… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CombinedMetricsIterator.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CombinedMetricsPair.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java Log Message: ----------- when the 6 hour time slice has passed for late data, recompute the 6 hour data This commit is best explained with an example. Suppose the current time is 06:12. We receive a measurement report with data having a timestamp of 05:50. Aggregation will run again at 07:00. The 6 hour time slice for the late data is 00:00 to 06:00, and that time slice has already passed. When aggregation runs, we not only compute the 1 hour aggregate for the data inserted at 05:50, we also recompute the 6 hour aggregate so that the late data is included. Commit: 412ed2c15230be29470419f8ca858e2f40b82ec3 https://github.com/rhq-project/rhq/commit/412ed2c15230be29470419f8ca858e2f4… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java Log Message: ----------- update cache and cache index only when necessary for late data When aggregating late data whose 6 hour time slice has not yet finished, we should update metrics_cache and metrics_cache_index so that the necessary rows are present when 6 hour data aggregation runs. If the 6 hour time slice has already passed, then there is no need to update metrics_cache and metrics_cache_index because we go ahead and recompute the 6 hour data. The same idea holds with 24 hour data. If the 24 hour time slice has not yet finished, then should update metrics_cache and metrics_cache_index so that the necessary rows are present when 24 hour data aggregation runs; however, if the 24 hour time slice has already passed, then there is no need to perform the updates because we will (in the next commit) go ahead and recompute 24 hour data. Commit: 5313028c39a434c3731b44db3b754c3a0241f6dd https://github.com/rhq-project/rhq/commit/5313028c39a434c3731b44db3b754c3a0… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CombinedMetricsIterator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CombinedMetricsPair.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/MetricsFuturesPair.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java Log Message: ----------- adding support for recomputing 24 hour data Commit: 186e838152e92e08c439b8cf8bc287194ed5f0e1 https://github.com/rhq-project/rhq/commit/186e838152e92e08c439b8cf8bc287194… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/DateTimeService.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java A modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PersistFunctions.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/AggregateNumericMetric.java Log Message: ----------- refactoring some functions into a utility class This commit also updates the logic in PastDataAggregator for how it invokes the processCacheBlock method. If the 6 hour time slice has finished, it no longer updates the metrics_cache and metrics_cache_index tables. There are additional changes needed for when PastDataAggregator calls processCacheBlock. It needs to determine if the 6 hour and 24 hour data need to be recomputed similar to what is done in the processBatch method. Commit: 664ce1493c8b11e113cc328686ec7dfd4d7446f3 https://github.com/rhq-project/rhq/commit/664ce1493c8b11e113cc328686ec7dfd4… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java Log Message: ----------- recompute 6 hour data when aggregating late data out of metrics_cache table If the 6 hour time slice of the late data has passed, we need to recompute the 6 hour data just as we do when we pull data from raw_metrics. Commit: bdd8a0c63a79245378bf29f74fc91aed416aa001 https://github.com/rhq-project/rhq/commit/bdd8a0c63a79245378bf29f74fc91aed4… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/MeasurementAggregator.java Log Message: ----------- updating test for recomputing 24 hr data when pulling data from metrics_cache Commit: a9a59caef28a011ab18c6c685d6430c594ebdf11 https://github.com/rhq-project/rhq/commit/a9a59caef28a011ab18c6c685d6430c59… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java Log Message: ----------- refactoring task scheduling logic Task scheduling logic has been pulled up into BaseAggregator which is now an abstract class. Subclasses are responsible for creating the Runnable tasks. Commit: 8374cb7ed96da2a43994484caacadb39c4312b67 https://github.com/rhq-project/rhq/commit/8374cb7ed96da2a43994484caacadb39c… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java Log Message: ----------- more refactoring to correctly compute the number of schedules processed Commit: 209dfeb3a487d1c3803a8a335c90d1f9ac5bb94c https://github.com/rhq-project/rhq/commit/209dfeb3a487d1c3803a8a335c90d1f9a… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java Log Message: ----------- introduce AggregationTask to provide error handling Commit: ab93df669b213f024511d0e89c1177d39d1c894e https://github.com/rhq-project/rhq/commit/ab93df669b213f024511d0e89c1177d39… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/common/cassandra-schema/src/main/resources/schema/update/0002.xml M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PersistFunctions.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntry.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntryMapper.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java Log Message: ----------- making schema changes to metrics_cache_index table Some columns have been renamed and a new column has been added. We now have the following timestamp columns - day, collection_time_slice, and insert_time_slice. collection_time_slice is the timestamp provided by the reported data. insert_time_slice is the raw time slice in which the data is inserted. For example, if data is inserted at 17:02, insert_time_slice will be 17:00. And day is the 24 hour time slice of the collection timestamp. insert_time_slice is the new column. Previously there was a scenario in which we could fail to aggregate data. Suppose aggregation for a cache partition failed. Its cache index entry would be left intact., and note that the set of schedule ids was empty. Then late data is reported in the next time slice. At that point, the cache index entry is updated with the schedule ids of the late data. Prior to this commit we would have only aggregated data for the schedule ids with late data. Now we always update the schedule ids for the cache index, even for current data. When we have past/late data, we now compare collection_time_slice with insert_time_slice to determine if we can pull data from metrics_cache. Commit: 03ba9f46751cabbb5da4e5bb593c62ec1d00da0d https://github.com/rhq-project/rhq/commit/03ba9f46751cabbb5da4e5bb593c62ec1… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsDAOTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java Log Message: ----------- fix scheduling logic for past data aggregation tasks With the addition of the insert_time_slice column, we can now have multiple index entries for a {column_time_slice, start_schedule_id} pair. We can reduce those into a single index entry and submit an aggregation task for that combined entry. PastDataAggregator also now has a deleteCacheIndexEntries function to take care of deleting all index entries for a {column_time_slice, start_schedule_id} pair. Commit: a90ea57481c0576c68c4d76c48ecb4bbec0e64ac https://github.com/rhq-project/rhq/commit/a90ea57481c0576c68c4d76c48ecb4bbe… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntry.java Log Message: ----------- more refactoring of processing index entries and adding javadocs Commit: 5088a4efdbd3b03bd143de07466ee59bcf3ed65d https://github.com/rhq-project/rhq/commit/5088a4efdbd3b03bd143de07466ee59bc… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsPerfTests.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsTest.java R modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/SimulatedMetricsServer.java M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/Simulator.java Log Message: ----------- refactoring tests and adding javadocs Commit: b87f4d8824d72190a9ff4160d057a3ef1b3fce6c https://github.com/rhq-project/rhq/commit/b87f4d8824d72190a9ff4160d057a3ef1… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java Log Message: ----------- use the same insert time slice for each raw data There is no need to recompute the insert time slice for each raw data. It is sufficient to use insert time slice timestamp as the starting point at which we beging inserting raw data. Commit: ca1edcf55afa9856523dc8fce51e2aca9cd9584f https://github.com/rhq-project/rhq/commit/ca1edcf55afa9856523dc8fce51e2aca9… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CombinedMetricsIterator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/AggregateNumericMetricMapper.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/domain/CacheIndexEntry.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsPerfTests.java Log Message: ----------- do not schedule aggregation tasks on I/O thread I had previously and inadvertently perfoming the scheduling of aggregation tasks on an I/O thread. Long running tasks should never run on an I/O thread. I discovered this mistake in a test environment where I was generating NoHostAvailableExceptions because of this mistake. This commit also makes a subtle yet significant change in AggregateNumericMetricMapper. It now checks for nulls which it should have been doing all along since we use weak consistency for reads and writes. The change was needed because I was hitting occassional NPEs with past data aggregation when 1 hr and/or 6 hr aggregates have to be computed. It is easiest to explain with an example. Suppose we are aggregating some past data and we also need to aggregate the 1 hour data into 6 hour data. Almost immediately after computing and storing the 1 hour data for the given time slice, we then fetch all of the 1 hour data for the 6 hour time slice. Unless the query is against the same node to which the preceding write went, the query results might not include all of the data. We do not have to worry about the missing data in the result sets though because we already have the data in memory. Commit: 980afd404731f748c87eaeda151e55542bf8a453 https://github.com/rhq-project/rhq/commit/980afd404731f748c87eaeda151e55542… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java Log Message: ----------- updating logging and adding javadocs Commit: e0315a311dd908df63e2c1d56a03d68cf47a9f90 https://github.com/rhq-project/rhq/commit/e0315a311dd908df63e2c1d56a03d68cf… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/AggregationManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/BaseAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/CacheAggregator.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/aggregation/PastDataAggregator.java M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java Log Message: ----------- add support for "decactivating" metrics_cache Deactivating metrics_cache results in data *not* being pulled from it. This applies to both past and current data. Once a specified time has been reached, it is reactived and we start pulling from metrics_cache again. This logic is necessary for two purrposes. First, we need it simplify and speed up the data migration that will be necessary when upgrading to RHQ 4.11. Secondly, it is necessary for the resizing of cache partitions to take effect. For example, suppose we store data for up to 5 schedules per cache partition. We decide to store data for 10 schedules per partition. We need to deactivate the cache while the change takes effect. Commit: 16ee3c44bb6a4084a6a08358b9328c7eccb5ae24 https://github.com/rhq-project/rhq/commit/16ee3c44bb6a4084a6a08358b9328c7ec… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/MetricsServerTest.java M modules/helpers/metrics-simulator/src/main/java/org/rhq/metrics/simulator/Simulator.java Log Message: ----------- fixing merge conflicts from rebase Commit: 16ffd42907281ed72434fc18e5a8adfd1c41d303 https://github.com/rhq-project/rhq/commit/16ffd42907281ed72434fc18e5a8adfd1… Author: John Sanda <jsanda(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/common/cassandra-schema/pom.xml M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/AbstractManager.java A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/CQLStep.java A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/DBConnectionFactory.java A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/PopulateCacheIndex.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/SchemaManager.java A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/SchemaUpdateThreadFactory.java A modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/Step.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/UpdateFile.java M modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/VersionManager.java M modules/common/cassandra-schema/src/main/resources/schema/update/0002.xml A modules/common/cassandra-schema/src/main/resources/schema/update/0003.xml M modules/common/cassandra-schema/src/test/java/org/rhq/cassandra/schema/UpdateFileTest.java M modules/core/dbutils/src/main/scripts/dbsetup/sysconfig-data.xml M modules/core/dbutils/src/main/scripts/dbupgrade/db-upgrade.xml M modules/core/domain/src/main/java/org/rhq/core/domain/common/composite/SystemSetting.java M modules/enterprise/server/installer/pom.xml M modules/enterprise/server/installer/src/main/java/org/rhq/enterprise/server/installer/InstallerServiceImpl.java M modules/enterprise/server/installer/src/main/resources/module/main/module.xml M modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/storage/StorageClientManager.java M modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java Log Message: ----------- Adding data migration support needed during upgrades The C* schema management tool has been changed such that it now supports arbitrary tasks in addition to raw CQL statements. An arbitrary task/step must implement the new Step interface. PopulateCacheIndex is one such custom task that migrates data out of metrics_index and into metrics_cache_index. A property has been added to the system settings for the cache activation time. PopulateCacheIndex sets that property so the server knows when it can start pulling data from metrics_cache during aggregation. And at server start up, MetricsServer is initialized with the property value. Compare: https://github.com/rhq-project/rhq/compare/3338fc8dee08...16ffd4290728

1 0

[rhq-project/rhq] 8a087d: BZ 1090476 - do not check for status if the agent ...
by John Mazzitelli 28 Apr '14

28 Apr '14

Branch: refs/heads/master Home: https://github.com/rhq-project/rhq Commit: 8a087da45ba10c3a25f0ea4b933a8cf42733e764 https://github.com/rhq-project/rhq/commit/8a087da45ba10c3a25f0ea4b933a8cf42… Author: John Mazzitelli <mazz(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/admin/agent/install/RemoteAgentInstallView.java Log Message: ----------- BZ 1090476 - do not check for status if the agent wasn't found. this makes the UI more responsive. There is no need to check status anyway since we don't know where the agent is

1 0

[rhq-project/rhq] 974167: Bump ems version to 1.3.5-rhq for lighter connecti...
by Heiko W. Rupp 28 Apr '14

28 Apr '14

Branch: refs/heads/master Home: https://github.com/rhq-project/rhq Commit: 974167ac9fcc3b329ad836ed6c126cf2b70a748e https://github.com/rhq-project/rhq/commit/974167ac9fcc3b329ad836ed6c126cf2b… Author: Heiko W. Rupp <hwr(a)pilhuhn.de> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M pom.xml Log Message: ----------- Bump ems version to 1.3.5-rhq for lighter connection checking.

1 0

[rhq-project/rhq] 82bffc: Bug 1077412 - Restart operation on EAP 5 resource ...
by Thomas Segismont 28 Apr '14

28 Apr '14

Branch: refs/heads/release/jon3.2.x Home: https://github.com/rhq-project/rhq Commit: 82bffc4070b6cd879c19d7f9cfd053adf0bfb6ee https://github.com/rhq-project/rhq/commit/82bffc4070b6cd879c19d7f9cfd053adf… Author: Thomas Segismont <tsegismo(a)redhat.com> Date: 2014-04-28 (Mon, 28 Apr 2014) Changed paths: M modules/plugins/jboss-as-5/src/main/java/org/rhq/plugins/jbossas5/ApplicationServerOperationsDelegate.java Log Message: ----------- Bug 1077412 - Restart operation on EAP 5 resource doesn't return failure message when restart fails (cherry-picked from master 145463496ed1b6dcc7d09193965a77c737bc29b4)

1 0

[rhq-project/rhq] 3dd669: [BZ 1089108] File Systems -> Free Space graph off ...
by mike thompson 28 Apr '14

28 Apr '14

Branch: refs/heads/rc/jon3.2.1.GA Home: https://github.com/rhq-project/rhq Commit: 3dd669bc43f741d2010b9ad4415cac1c29f92178 https://github.com/rhq-project/rhq/commit/3dd669bc43f741d2010b9ad4415cac1c2… Author: Mike Thompson <mithomps(a)redhat.com> Date: 2014-04-27 (Sun, 27 Apr 2014) Changed paths: M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/common/graph/MetricGraphData.java Log Message: ----------- [BZ 1089108] File Systems -> Free Space graph off by a factor of 1000. This was due to a recent bug fix introduced for BZ1080470. (cherry picked from commit 251d60aa898402d28338a43bc844a3d69d4aff7d) Commit: b517223f13f94fba88287a3278ecef789bec8af0 https://github.com/rhq-project/rhq/commit/b517223f13f94fba88287a3278ecef789… Author: Mike Thompson <mithomps(a)redhat.com> Date: 2014-04-27 (Sun, 27 Apr 2014) Changed paths: M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/inventory/common/graph/MetricGraphData.java M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/util/MeasurementConverterClient.java M modules/enterprise/gui/coregui/src/main/java/org/rhq/coregui/client/util/MeasurementUtility.java Log Message: ----------- [BZ 1089108] File Systems -> Free Space graph off by a factor of 1000. Refactored to a much cleaner solution with no IF logic. Removed some dead code from Measurement helper classes. (cherry picked from commit a2f0529d82120e069b22c94558d07f9e3a811e44) Compare: https://github.com/rhq-project/rhq/compare/c92c482ef2a3...b517223f13f9

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

rhq-commits April 2014