Advice on RHQ agent metrics capacity

John Sanda jsanda at redhat.com
Sat Mar 8 02:32:00 UTC 2014


On Mar 7, 2014, at 6:29 PM, Steven North <swn at ocsystems.com> wrote:

> Thanks for the suggestions.  It turns out we were only looking at about 10,000 metrics per minute going through a single agent.  The agent was running at about an average CPU load of 50% (which seems high), with periodic spikes to 100-150% for 1 second (while collection was happening).  I did a local setup which logged about 20,000 metrics per minute and saw a much lower average CPU load for the agent (<5%) and with periodic 1 second spikes to 90% CPU during collection.  I had to run with 512MB heap, but only seemed to be using half that.
> 
> I have a few new questions....
> 
> I will have to investigate what the 50% CPU load was doing on the agent, but one concern about the metrics collection spikes is that it might interfere with availability reports or the resulting JBoss restart operations which might result from alerts on the availability reports.  Assuming there is adequate CPU reserve on the server VM, heavy work load on the agent metrics collection thread shouldn't interfere with the availability thread or the operation execution thread, right?
> 
> All my local testing of metrics has filled up the disk on the RHQ storage node VM.  Are there any settings I can use to restrict the amount of metric data kept by RHQ?  This is just a test system so I don't really care if I only have a week's, a day's, an hours worth of data.  I just don't want the disk to fill up.  I removed the 5,000 resources used to collect the 20,000 metrics, but I don't see a corresponding drop in disk usage by rhq-storage.

There is currently no way to adjust the amount of data stored. There has been discussion of and interest in configurable retention periods; however, I am not aware of any RFEs. I think it would be a great enhancement. If you create a bug report, I would be happy to comment on it. In RHQ 4.10, we will have data compression turned back on. It is disabled in 4.9.0. You can deploy additional storage nodes to reduce the load. You might want to review this doc[1] for details on data replication. One other thing you can look at is the amount of space being consumed by snapshots. They can wind up consuming a good bit of space. I don't want to hijack this thread so I will start a separate one discussing snapshots and some options for automated management.

[1] https://docs.jboss.org/author/display/RHQ/Data+Replication+and+Consistency

> 
> Thanks in advance,
> 
> Steve
> 
> On Feb 27, 2014, at 8:44 AM, John Sanda <jsanda at redhat.com> wrote:
> 
>> 
>> On Feb 27, 2014, at 11:27 AM, Elias Ross <genman at noderunner.net> wrote:
>> 
>>> On Wed, Feb 26, 2014 at 3:18 PM, Steven North <swn at ocsystems.com> wrote:
>>>> Hello,
>>>> 
>>>> I'm looking for some advice and/or experience with the RHQ agent capacity for handling metrics.
>>>> I am considering an environment where there may be on the order of 80,000 metrics might be collected from a single platform.
>>> 
>>> I've done that using the Perf plugin:
>>> https://docs.jboss.org/author/display/RHQ/Test+Using+the+Perftest+Plugin
>>> 
>>> Of course, the metrics gathering loop is just getting random numbers.
>>> If it takes more than, say, 30 seconds or a minute to gather all
>>> 80,000 sequentially, there may be an issue.
>>> 
>>> For availabilities, I mitigated that somewhat by using an execution
>>> pool. I came up with this for metrics gathering, which you can use if
>>> your metrics take longer than a few milliseconds to gather:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=783603
>>> 
>>> I think if I did this again, I'd create an annotation which would make
>>> metrics gathering work similarly to async availability checking.
>>> 
>>> Anyway, there are probably some things that could be improved on the
>>> agent side. For example, using smaller batches for metrics reporting.
>>> 
>>>> 
>>>> Could an RHQ agent possibly handle that many metrics per minute?  Per 5 minutes?  Would there be a way to start multiple RHQ agents on a platform to distribute the load?
>>> 
>>> The agent is identified to the server using its IP address. If you
>>> created multiple virtual IPs you could run multiple agents on one
>>> host.
>>> 
>>>> 
>>>> What about the RHQ server?  Any experience on the capacity of metrics  per minute?
>>> 
>>> I've gone up to 900,000 metrics per minute. Currently the limitation I
>>> found is mostly on the number of schedules, since Cassandra metrics
>>> aggregation is done by the schedule, and reads are much slower than
>>> writes.
>> 
>> Currently the number of reads executed during aggregation is directly proportional to the number of schedules. I have been working schema changes in the jsanda/metrics-schema branch that will substantially reduce the number of reads; however, these changes probably won't come until 4.11.
>> _______________________________________________
>> rhq-devel mailing list
>> rhq-devel at lists.fedorahosted.org
>> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fedorahosted.org/pipermail/rhq-devel/attachments/20140307/6f188d6c/attachment-0001.html>


More information about the rhq-devel mailing list