Thanks for the suggestions. It turns out we were only looking at about 10,000 metrics per
minute going through a single agent. The agent was running at about an average CPU load
of 50% (which seems high), with periodic spikes to 100-150% for 1 second (while collection
was happening). I did a local setup which logged about 20,000 metrics per minute and saw
a much lower average CPU load for the agent (<5%) and with periodic 1 second spikes to
90% CPU during collection. I had to run with 512MB heap, but only seemed to be using half
that.
I have a few new questions....
I will have to investigate what the 50% CPU load was doing on the agent, but one concern
about the metrics collection spikes is that it might interfere with availability reports
or the resulting JBoss restart operations which might result from alerts on the
availability reports. Assuming there is adequate CPU reserve on the server VM, heavy work
load on the agent metrics collection thread shouldn't interfere with the availability
thread or the operation execution thread, right?
All my local testing of metrics has filled up the disk on the RHQ storage node VM. Are
there any settings I can use to restrict the amount of metric data kept by RHQ? This is
just a test system so I don't really care if I only have a week's, a day's, an
hours worth of data. I just don't want the disk to fill up. I removed the 5,000
resources used to collect the 20,000 metrics, but I don't see a corresponding drop in
disk usage by rhq-storage.
Thanks in advance,
Steve
On Feb 27, 2014, at 8:44 AM, John Sanda <jsanda(a)redhat.com> wrote:
On Feb 27, 2014, at 11:27 AM, Elias Ross <genman(a)noderunner.net> wrote:
> On Wed, Feb 26, 2014 at 3:18 PM, Steven North <swn(a)ocsystems.com> wrote:
>> Hello,
>>
>> I'm looking for some advice and/or experience with the RHQ agent capacity for
handling metrics.
>> I am considering an environment where there may be on the order of 80,000 metrics
might be collected from a single platform.
>
> I've done that using the Perf plugin:
>
https://docs.jboss.org/author/display/RHQ/Test+Using+the+Perftest+Plugin
>
> Of course, the metrics gathering loop is just getting random numbers.
> If it takes more than, say, 30 seconds or a minute to gather all
> 80,000 sequentially, there may be an issue.
>
> For availabilities, I mitigated that somewhat by using an execution
> pool. I came up with this for metrics gathering, which you can use if
> your metrics take longer than a few milliseconds to gather:
>
https://bugzilla.redhat.com/show_bug.cgi?id=783603
>
> I think if I did this again, I'd create an annotation which would make
> metrics gathering work similarly to async availability checking.
>
> Anyway, there are probably some things that could be improved on the
> agent side. For example, using smaller batches for metrics reporting.
>
>>
>> Could an RHQ agent possibly handle that many metrics per minute? Per 5 minutes?
Would there be a way to start multiple RHQ agents on a platform to distribute the load?
>
> The agent is identified to the server using its IP address. If you
> created multiple virtual IPs you could run multiple agents on one
> host.
>
>>
>> What about the RHQ server? Any experience on the capacity of metrics per
minute?
>
> I've gone up to 900,000 metrics per minute. Currently the limitation I
> found is mostly on the number of schedules, since Cassandra metrics
> aggregation is done by the schedule, and reads are much slower than
> writes.
Currently the number of reads executed during aggregation is directly proportional to the
number of schedules. I have been working schema changes in the jsanda/metrics-schema
branch that will substantially reduce the number of reads; however, these changes probably
won't come until 4.11.
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel