Branch: refs/heads/spinder/BZ185375
Home:
https://github.com/rhq-project/rhq
Commit: 2c504afc89a03f513d8e0b6ed44963a6bc6cbf6d
https://github.com/rhq-project/rhq/commit/2c504afc89a03f513d8e0b6ed44963a...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
Log Message:
-----------
[BZ 185375] speed up data migration step
Writes are now done in batches. Reads are done in parallel and concurrently in
batches.
Commit: b9cbea1d29c3463caa8db9dfdef8d559355f5b30
https://github.com/rhq-project/rhq/commit/b9cbea1d29c3463caa8db9dfdef8d55...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
A
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/DateUtils.java
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/Replace412Index.java
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/ReplaceRHQ411Index.java
M
modules/common/cassandra-schema/src/test/java/org/rhq/cassandra/schema/ReplaceIndexTest.java
Log Message:
-----------
[BZ 185375] refactor date functions into util class
Commit: 2d3bdab8506510f21982db25dca71972996036b2
https://github.com/rhq-project/rhq/commit/2d3bdab8506510f21982db25dca7197...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M modules/common/cassandra-schema/pom.xml
A
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/AbortedException.java
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
A
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateData.java
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/ReplaceIndex.java
A
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/TaskTracker.java
Log Message:
-----------
[BZ 185375] lots of clean up, refactoring, and perf improvements
The TaskTracker class has been copied into the source tree from the
service-metrics maven module. The reason for copying the class is beause the
changes for this BZ are going into a patch release, and I want to minimize the
number of modules that have to be touched for this.
We no longer query the rdbms to load schedule ids. Instead we use the legacy
thrift APIs via the hector library to scan for schedule ids. This way, we query
for only those schedule ids that have data.
Reads are now throttled using a RateLimiter. This is better than using a
semaphore for a couple reasons. First, we do not have to worry about releasing
permits, which had to be done in a couple different places in the code.
Secondly, it is more expressive about the throughput. With a semaphore, it was
hard to determine the read throughput, whereas with RateLimiter it is directly
specified as querys per second.
The migration code previously tried to abort processing if a write failed. This
made the code overly complicated. Since we are now using batches, we only do a
handful of writes per schedule id. We now just have the failure detection and
handling code in the migration finished callback. This simplifies things a lot.
Commit: 8b3a089caf4bc3e6fd1819b46df0de52cb74d05c
https://github.com/rhq-project/rhq/commit/8b3a089caf4bc3e6fd1819b46df0de5...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
Log Message:
-----------
[BZ 185375] add failure detection
There is a now a failure threshold that if exceeded will cause the migration to
be aborted. The rationale is that the read/write rates will likely have to be
adjusted based on each environment. If we are reading too fast for example,
and start generating lots of failiures, it makes sense to go ahead and
terminate the migration and restart with a lower read rate.
Commit: fc4690a2f708f3e9c280ec03a339b39f826e4e1d
https://github.com/rhq-project/rhq/commit/fc4690a2f708f3e9c280ec03a339b39...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M modules/common/cassandra-schema/pom.xml
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateData.java
Log Message:
-----------
[BZ 185375] remove key scanning code with hector
I went back and did some testing with the original approach of querying against
all schedule ids that are returned from the RDBMS. I was able to achieve the
same performance (if not better) as I did with hector. I do not see any reason
therefore to pull in additional dependencies.
Conflicts:
modules/common/cassandra-schema/pom.xml
Commit: 32de6ed6c65622c0a4edaf506f1fd4c295217c0f
https://github.com/rhq-project/rhq/commit/32de6ed6c65622c0a4edaf506f1fd4c...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M modules/common/cassandra-schema/pom.xml
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
Log Message:
-----------
[BZ 185375] load schedule ids using astyanax
We do not have a good way in C* 1.2 to scan for schedule ids. Instead I am
using netflix's astyanax library which uses the legacy thrift api. It is
using the get_range_slices operation. This should hopefully be faster than
querying every single schedule id returned from the rdbms.
Conflicts:
modules/common/cassandra-schema/pom.xml
Commit: 12b8934dedf33d8e324f4d0f8e0aaf9e1962f33b
https://github.com/rhq-project/rhq/commit/12b8934dedf33d8e324f4d0f8e0aaf9...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
A
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/RateMonitor.java
Log Message:
-----------
[BZ 185375] initial support for dynamic throttling
Commit: 05d6dd82eabe6f847307e2970978ff50d48b305b
https://github.com/rhq-project/rhq/commit/05d6dd82eabe6f847307e2970978ff5...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/RateMonitor.java
Log Message:
-----------
[BZ 185375] retry data migrations on failures
Commit: 014630bc52d8bd41070758d5e904787c28ce7d80
https://github.com/rhq-project/rhq/commit/014630bc52d8bd41070758d5e904787...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/RateMonitor.java
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/ReplaceRHQ411Index.java
Log Message:
-----------
[BZ 185375] use async writes and dynamic throttling for index update
Commit: b279f9431680845a3701ae50a54319fe9acf7d60
https://github.com/rhq-project/rhq/commit/b279f9431680845a3701ae50a54319f...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateAggregateMetrics.java
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/MigrateData.java
M
modules/common/cassandra-schema/src/main/java/org/rhq/cassandra/schema/RateMonitor.java
M
modules/common/cassandra-schema/src/test/java/org/rhq/cassandra/schema/MigrateAggregateMetricsTest.java
M
modules/common/cassandra-schema/src/test/java/org/rhq/cassandra/schema/SchemaUpgradeTest.java
Log Message:
-----------
[BZ 185375] fix regression in MigrateData
Add back check to execute statements when there is not a full batch. Also
updating RateMonitor to include minimum rates.
Commit: d456e79c59ed309ea67171d10caa01a9e3bb2404
https://github.com/rhq-project/rhq/commit/d456e79c59ed309ea67171d10caa01a...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M modules/common/cassandra-schema/pom.xml
M modules/enterprise/server/appserver/src/main/scripts/rhq-container.build.xml
M modules/enterprise/server/installer/src/main/resources/module/main/module.xml
M pom.xml
Log Message:
-----------
[BZ 185375] adding jboss modules for astyanax
Conflicts:
modules/common/cassandra-schema/pom.xml
Commit: 3d766f551a0558d0d4aabb029779a10666ea8f60
https://github.com/rhq-project/rhq/commit/3d766f551a0558d0d4aabb029779a10...
Author: John Sanda <jsanda(a)redhat.com>
Date: 2015-03-02 (Mon, 02 Mar 2015)
Changed paths:
M modules/common/cassandra-schema/pom.xml
M modules/enterprise/server/appserver/src/main/scripts/rhq-container.build.xml
Log Message:
-----------
[BZ 185375] fix astyanax module dependencies
Compare:
https://github.com/rhq-project/rhq/compare/2c504afc89a0^...3d766f551a05