On Mar 7, 2014, at 6:29 PM, Steven North <swn(a)ocsystems.com> wrote:
Thanks for the suggestions. It turns out we were only looking at
about 10,000 metrics per minute going through a single agent. The agent was running at
about an average CPU load of 50% (which seems high), with periodic spikes to 100-150% for
1 second (while collection was happening). I did a local setup which logged about 20,000
metrics per minute and saw a much lower average CPU load for the agent (<5%) and with
periodic 1 second spikes to 90% CPU during collection. I had to run with 512MB heap, but
only seemed to be using half that.
I have a few new questions....
I will have to investigate what the 50% CPU load was doing on the agent, but one concern
about the metrics collection spikes is that it might interfere with availability reports
or the resulting JBoss restart operations which might result from alerts on the
availability reports. Assuming there is adequate CPU reserve on the server VM, heavy work
load on the agent metrics collection thread shouldn't interfere with the availability
thread or the operation execution thread, right?
All my local testing of metrics has filled up the disk on the RHQ storage node VM. Are
there any settings I can use to restrict the amount of metric data kept by RHQ? This is
just a test system so I don't really care if I only have a week's, a day's, an
hours worth of data. I just don't want the disk to fill up. I removed the 5,000
resources used to collect the 20,000 metrics, but I don't see a corresponding drop in
disk usage by rhq-storage.
There is currently no way to adjust the amount of data stored. There has been discussion
of and interest in configurable retention periods; however, I am not aware of any RFEs. I
think it would be a great enhancement. If you create a bug report, I would be happy to
comment on it. In RHQ 4.10, we will have data compression turned back on. It is disabled
in 4.9.0. You can deploy additional storage nodes to reduce the load. You might want to
review this doc[1] for details on data replication. One other thing you can look at is the
amount of space being consumed by snapshots. They can wind up consuming a good bit of
space. I don't want to hijack this thread so I will start a separate one discussing
snapshots and some options for automated management.
[1]
https://docs.jboss.org/author/display/RHQ/Data+Replication+and+Consistency
Thanks in advance,
Steve
On Feb 27, 2014, at 8:44 AM, John Sanda <jsanda(a)redhat.com> wrote:
>
> On Feb 27, 2014, at 11:27 AM, Elias Ross <genman(a)noderunner.net> wrote:
>
>> On Wed, Feb 26, 2014 at 3:18 PM, Steven North <swn(a)ocsystems.com> wrote:
>>> Hello,
>>>
>>> I'm looking for some advice and/or experience with the RHQ agent capacity
for handling metrics.
>>> I am considering an environment where there may be on the order of 80,000
metrics might be collected from a single platform.
>>
>> I've done that using the Perf plugin:
>>
https://docs.jboss.org/author/display/RHQ/Test+Using+the+Perftest+Plugin
>>
>> Of course, the metrics gathering loop is just getting random numbers.
>> If it takes more than, say, 30 seconds or a minute to gather all
>> 80,000 sequentially, there may be an issue.
>>
>> For availabilities, I mitigated that somewhat by using an execution
>> pool. I came up with this for metrics gathering, which you can use if
>> your metrics take longer than a few milliseconds to gather:
>>
https://bugzilla.redhat.com/show_bug.cgi?id=783603
>>
>> I think if I did this again, I'd create an annotation which would make
>> metrics gathering work similarly to async availability checking.
>>
>> Anyway, there are probably some things that could be improved on the
>> agent side. For example, using smaller batches for metrics reporting.
>>
>>>
>>> Could an RHQ agent possibly handle that many metrics per minute? Per 5
minutes? Would there be a way to start multiple RHQ agents on a platform to distribute
the load?
>>
>> The agent is identified to the server using its IP address. If you
>> created multiple virtual IPs you could run multiple agents on one
>> host.
>>
>>>
>>> What about the RHQ server? Any experience on the capacity of metrics per
minute?
>>
>> I've gone up to 900,000 metrics per minute. Currently the limitation I
>> found is mostly on the number of schedules, since Cassandra metrics
>> aggregation is done by the schedule, and reads are much slower than
>> writes.
>
> Currently the number of reads executed during aggregation is directly proportional to
the number of schedules. I have been working schema changes in the jsanda/metrics-schema
branch that will substantially reduce the number of reads; however, these changes probably
won't come until 4.11.
> _______________________________________________
> rhq-devel mailing list
> rhq-devel(a)lists.fedorahosted.org
>
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel