Advice on RHQ agent metrics capacity

Elias Ross genman at noderunner.net
Thu Feb 27 16:27:39 UTC 2014


On Wed, Feb 26, 2014 at 3:18 PM, Steven North <swn at ocsystems.com> wrote:
> Hello,
>
> I'm looking for some advice and/or experience with the RHQ agent capacity for handling metrics.
>  I am considering an environment where there may be on the order of 80,000 metrics might be collected from a single platform.

I've done that using the Perf plugin:
https://docs.jboss.org/author/display/RHQ/Test+Using+the+Perftest+Plugin

Of course, the metrics gathering loop is just getting random numbers.
If it takes more than, say, 30 seconds or a minute to gather all
80,000 sequentially, there may be an issue.

For availabilities, I mitigated that somewhat by using an execution
pool. I came up with this for metrics gathering, which you can use if
your metrics take longer than a few milliseconds to gather:
https://bugzilla.redhat.com/show_bug.cgi?id=783603

I think if I did this again, I'd create an annotation which would make
metrics gathering work similarly to async availability checking.

Anyway, there are probably some things that could be improved on the
agent side. For example, using smaller batches for metrics reporting.

>
> Could an RHQ agent possibly handle that many metrics per minute?  Per 5 minutes?  Would there be a way to start multiple RHQ agents on a platform to distribute the load?

The agent is identified to the server using its IP address. If you
created multiple virtual IPs you could run multiple agents on one
host.

>
> What about the RHQ server?  Any experience on the capacity of metrics  per minute?

I've gone up to 900,000 metrics per minute. Currently the limitation I
found is mostly on the number of schedules, since Cassandra metrics
aggregation is done by the schedule, and reads are much slower than
writes.


More information about the rhq-devel mailing list