New branch 'jsanda/aggregation' available with the following commits:
commit 6ad96024e571f92d385f5d3ebf355df5c8c6d4af
Author: John Sanda <jsanda(a)redhat.com>
Date: Wed Dec 18 11:05:37 2013 -0500
[BZ 1009945] add support for running async aggregation in simulator
commit ae1b76655665a6fef2ddbba9b2b1f125c6297dcb
Author: John Sanda <jsanda(a)redhat.com>
Date: Wed Oct 23 15:12:15 2013 -0400
[BZ 1009945] make connection pool sizes and rate limits configurable
Connection pools per host and rate limits are configurable via system
properties. While the connection pool sizes are only set at start up, I plan to
make the rate limits self-tuning.
This commit also adds the missing method body to kick off 6 hour data
aggregation in Aggregate1HourData.java.
Conflicts:
modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
commit 7d96099e86e4634d1137197059a1f5ba4f3a187b
Author: John Sanda <jsanda(a)redhat.com>
Date: Tue Oct 22 17:01:50 2013 -0400
[BZ 1009945] refactoring some duplicate code and fixing when tasks are scheduled
Aggregation tasks for 1 and 6 hour data were getting schedule too early. They
cannot get scheduled until both their respective index entries have arrived and
aggregation for the previous buckets has completed.
commit 19bd7da1c72ea245744de25740314bbd55458e45
Author: John Sanda <jsanda(a)redhat.com>
Date: Tue Oct 22 11:51:22 2013 -0400
[BZ 1009945] set counters for remaining data when we fail to get index entries for 1
and 6 hr data
commit 09901ef4a098844bd005507ec1cc0827535d6670
Author: John Sanda <jsanda(a)redhat.com>
Date: Tue Oct 22 07:30:32 2013 -0400
[BZ 1009945] fix logic for determining if aggregation has finished
We check the respective counters to see if raw, 1 hr, and 6 hr data aggregation
has finished. Prior to checking the counters though, we need to make sure that
they have been initialized. They do not get initialized until the respective
index entries arrive. I have refactored the logic into a new method,
isAggregationFinished. It first waits for the arrival of all the index entries
and then checks the counters.
commit 322cdb7a3181c3160f134e0373b510feeaa5f8ec
Author: John Sanda <jsanda(a)redhat.com>
Date: Mon Oct 21 11:06:17 2013 -0400
[BZ 1009945] adding back support for aggregating 6 hour data
commit ba38ad638efe2553911f9452d4affc7398089601
Author: John Sanda <jsanda(a)redhat.com>
Date: Sat Oct 19 21:58:37 2013 -0400
[BZ 1009945] This is a big refactoring of the initial implementation of
Aggregator.java
The overall design is the same in terms of the fan out approach for doing the
calculations, but some changes were necessary after an extensive performance
analysis. Initially the aggregation tasks were to granular. Tasks were
submitted to the thread pool to process a single schedule. Schedules are now
processed in batches of 250. I did a lot of performance testing with various
batch sizes, and 250 seems optimal both in terms of execution time as well as
memory consumption. The other major change involves throttling. The initial
implementation had no throttling in place which immediately caused read and/or
write timeouts. Then I put throttling in place using RateLimiters, but there
was separate throttling that is used in MetricsServer for inserting raw data.
The same throttling is now used by both MetricsServer and Aggregator. There
are actually separate rate limiters for reads and for writes. Both are
configurable as well.
Conflicts:
modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
commit da67ebe5d7b4b9caf3790cc599a973a511851dec
Author: John Sanda <jsanda(a)redhat.com>
Date: Thu Oct 10 13:02:06 2013 -0400
[BZ 1009945] adding initial test for failure scenario
This test is for when we fail to fetch the raw data index.
commit aa715c07ad1051d04632e3730ddc4e284e6da995
Author: John Sanda <jsanda(a)redhat.com>
Date: Wed Oct 9 22:29:19 2013 -0400
[BZ 1009945] finishing test for 24 hour data
Conflicts:
modules/enterprise/server/server-metrics/src/test/java/org/rhq/server/metrics/AggregationTests.java
commit 884dbd89deffdb06dd8614f88603126087408c2d
Author: John Sanda <jsanda(a)redhat.com>
Date: Fri Oct 4 17:12:15 2013 -0400
[BZ 1009945] handle scenarios for when there is no 1 hr and/or 6 hr data
This commit also includes some major test changes. AggregationTests is a more
thorough set (of not yet complete) tests to provide better coverage.
commit b22be76adcd85aa0ad4421789718ead66d00b22c
Author: John Sanda <jsanda(a)redhat.com>
Date: Thu Oct 3 07:50:23 2013 -0400
[BZ 1009945] delete 1 hr index entries when aggregation of time slice data is
finished
commit ef1b470e9e15ec1a9301c84032e2ae3698556f2c
Author: John Sanda <jsanda(a)redhat.com>
Date: Wed Oct 2 22:53:11 2013 -0400
[BZ 1009945] initial support for generating 6 hr and 24 hr data
Conflicts:
modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsDAO.java
commit 7709dfcc778411294c2f4bacb47327837ba5da54
Author: John Sanda <jsanda(a)redhat.com>
Date: Wed Oct 2 17:58:38 2013 -0400
[BZ 1009945] update 6 hr index after inserting 1 hr data
commit 1d061eaf875757537254880ccd4b5d72cacc415a
Author: John Sanda <jsanda(a)redhat.com>
Date: Wed Oct 2 17:36:21 2013 -0400
[BZ 1009945] Initial commit for new Aggregator class
This is a first pass at computing multiple aggregates concurrently.
Conflicts:
modules/enterprise/server/server-metrics/src/main/java/org/rhq/server/metrics/MetricsServer.java
Show replies by date