On Wed, Feb 26, 2014 at 3:18 PM, Steven North <swn(a)ocsystems.com> wrote:
Hello,
I'm looking for some advice and/or experience with the RHQ agent capacity for
handling metrics.
I am considering an environment where there may be on the order of 80,000 metrics might
be collected from a single platform.
I've done that using the Perf plugin:
https://docs.jboss.org/author/display/RHQ/Test+Using+the+Perftest+Plugin
Of course, the metrics gathering loop is just getting random numbers.
If it takes more than, say, 30 seconds or a minute to gather all
80,000 sequentially, there may be an issue.
For availabilities, I mitigated that somewhat by using an execution
pool. I came up with this for metrics gathering, which you can use if
your metrics take longer than a few milliseconds to gather:
https://bugzilla.redhat.com/show_bug.cgi?id=783603
I think if I did this again, I'd create an annotation which would make
metrics gathering work similarly to async availability checking.
Anyway, there are probably some things that could be improved on the
agent side. For example, using smaller batches for metrics reporting.
Could an RHQ agent possibly handle that many metrics per minute? Per 5 minutes? Would
there be a way to start multiple RHQ agents on a platform to distribute the load?
The agent is identified to the server using its IP address. If you
created multiple virtual IPs you could run multiple agents on one
host.
What about the RHQ server? Any experience on the capacity of metrics per minute?
I've gone up to 900,000 metrics per minute. Currently the limitation I
found is mostly on the number of schedules, since Cassandra metrics
aggregation is done by the schedule, and reads are much slower than
writes.