Advice on RHQ agent metrics capacity

John Sanda jsanda at redhat.com
Thu Feb 27 16:44:30 UTC 2014


On Feb 27, 2014, at 11:27 AM, Elias Ross <genman at noderunner.net> wrote:

> On Wed, Feb 26, 2014 at 3:18 PM, Steven North <swn at ocsystems.com> wrote:
>> Hello,
>> 
>> I'm looking for some advice and/or experience with the RHQ agent capacity for handling metrics.
>> I am considering an environment where there may be on the order of 80,000 metrics might be collected from a single platform.
> 
> I've done that using the Perf plugin:
> https://docs.jboss.org/author/display/RHQ/Test+Using+the+Perftest+Plugin
> 
> Of course, the metrics gathering loop is just getting random numbers.
> If it takes more than, say, 30 seconds or a minute to gather all
> 80,000 sequentially, there may be an issue.
> 
> For availabilities, I mitigated that somewhat by using an execution
> pool. I came up with this for metrics gathering, which you can use if
> your metrics take longer than a few milliseconds to gather:
> https://bugzilla.redhat.com/show_bug.cgi?id=783603
> 
> I think if I did this again, I'd create an annotation which would make
> metrics gathering work similarly to async availability checking.
> 
> Anyway, there are probably some things that could be improved on the
> agent side. For example, using smaller batches for metrics reporting.
> 
>> 
>> Could an RHQ agent possibly handle that many metrics per minute?  Per 5 minutes?  Would there be a way to start multiple RHQ agents on a platform to distribute the load?
> 
> The agent is identified to the server using its IP address. If you
> created multiple virtual IPs you could run multiple agents on one
> host.
> 
>> 
>> What about the RHQ server?  Any experience on the capacity of metrics  per minute?
> 
> I've gone up to 900,000 metrics per minute. Currently the limitation I
> found is mostly on the number of schedules, since Cassandra metrics
> aggregation is done by the schedule, and reads are much slower than
> writes.

Currently the number of reads executed during aggregation is directly proportional to the number of schedules. I have been working schema changes in the jsanda/metrics-schema branch that will substantially reduce the number of reads; however, these changes probably won't come until 4.11.


More information about the rhq-devel mailing list