Advice on RHQ agent metrics capacity

Steven North swn at ocsystems.com
Sat Mar 8 19:20:48 UTC 2014


Heiko,

On Mar 8, 2014, at 12:24 AM, Heiko W.Rupp <hrupp at redhat.com> wrote:

> Steven,
> 
> 
> Am 08.03.2014 um 00:29 schrieb Steven North <swn at ocsystems.com>:
> 
>> Thanks for the suggestions.  It turns out we were only looking at about 10,000 metrics per minute going through a single agent.  The agent was running at about an average CPU load of 50% (which seems high), with periodic spikes to 100-150% for 1 second (while collection was happening).  I did a local setup which logged about 20,000 metrics per minute and saw a much lower average CPU load for the agent (<5%) and with periodic 1 second spikes to 90% CPU during collection.  I had to run with 512MB heap, but only seemed to be using half that.
> 
> what version of RHQ are you looking at?
> In 4.10 (hopefully out very soon), we have addressed a good number of issues - especially one where the agent core itself was using a lot of cpu to select which metrics to take. Also the heap usage should be considerably lower, which should also reduce (CPU for) garbage collection and memory pressure.

I think the customer is using version is JON 3.1.2 == RHQ 4.6 and I was using 3.1.0 = RHQ 4.4.  I have also run tests on RHQ 4.9.  My tests on RHQ 4.4 & 4.9 didn't show any real issues, other than having to increase the heap size from the default 128MB to 512MB.  In my monitoring, GC was not much of an issue, only about 1-2% CPU during the 1 second collection interval (after the heap size increase).  I may have an opportunity to gather more information about what was happening at the customer site next week.  They are going to JON 3.2 soon, so events may overtake this investigation.
  
> 
> The other aspect to take into account is the plugin's getValues() method, which also has sometimes potential for improvement.

Yes, I was very aware of that possibility which lead to my investigations.  The messages in the agent log about metrics collection times and CPU monitoring of my examples helped convince me that our RTI plugin was not the culprit.  There must be another factor in the customer environment that I have yet to discover.

> But let us know when you have findings that we can incorporate (pull-requests welcome :-)

Will do.  I hope it's just some configuration item they have overlooked.

Thanks,

Steve

> 
>  Heiko
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel



More information about the rhq-devel mailing list