Advice on RHQ agent metrics capacity
John Sanda
jsanda at redhat.com
Thu Feb 27 16:44:30 UTC 2014
On Feb 27, 2014, at 11:27 AM, Elias Ross <genman at noderunner.net> wrote:
> On Wed, Feb 26, 2014 at 3:18 PM, Steven North <swn at ocsystems.com> wrote:
>> Hello,
>>
>> I'm looking for some advice and/or experience with the RHQ agent capacity for handling metrics.
>> I am considering an environment where there may be on the order of 80,000 metrics might be collected from a single platform.
>
> I've done that using the Perf plugin:
> https://docs.jboss.org/author/display/RHQ/Test+Using+the+Perftest+Plugin
>
> Of course, the metrics gathering loop is just getting random numbers.
> If it takes more than, say, 30 seconds or a minute to gather all
> 80,000 sequentially, there may be an issue.
>
> For availabilities, I mitigated that somewhat by using an execution
> pool. I came up with this for metrics gathering, which you can use if
> your metrics take longer than a few milliseconds to gather:
> https://bugzilla.redhat.com/show_bug.cgi?id=783603
>
> I think if I did this again, I'd create an annotation which would make
> metrics gathering work similarly to async availability checking.
>
> Anyway, there are probably some things that could be improved on the
> agent side. For example, using smaller batches for metrics reporting.
>
>>
>> Could an RHQ agent possibly handle that many metrics per minute? Per 5 minutes? Would there be a way to start multiple RHQ agents on a platform to distribute the load?
>
> The agent is identified to the server using its IP address. If you
> created multiple virtual IPs you could run multiple agents on one
> host.
>
>>
>> What about the RHQ server? Any experience on the capacity of metrics per minute?
>
> I've gone up to 900,000 metrics per minute. Currently the limitation I
> found is mostly on the number of schedules, since Cassandra metrics
> aggregation is done by the schedule, and reads are much slower than
> writes.
Currently the number of reads executed during aggregation is directly proportional to the number of schedules. I have been working schema changes in the jsanda/metrics-schema branch that will substantially reduce the number of reads; however, these changes probably won't come until 4.11.
More information about the rhq-devel
mailing list