Things seemed to have settled down - we doubled the number of max
connections in postgresql.conf and started up the agents more slowly.
We still saw the connection limit exceeded error, but it stopped after a
while and we seem to be operating normally again. So we have done
scenario (i) listed below and 200 per server is probably the max we want
to run with. Next is to test scenario (iii) - my plan is to model our
plugins with scenarios for the perftest plugin that mimic the number and
frequency of data collection from our real plugins.
Thanks for all the help.
Bala Nair
SeaChange International
On 2/22/11 4:21 PM, Charles Crouch wrote:
As I'm sure you know the global concurrency limits will only
limit the number of agents connecting to a server, not the number of connections made by
the server to the DB. IIRC when
testing this on Oracle we were seeing the # of db connections used> # of agents.
You've also got the all the background jobs and the UI itself using DB connections. So
if your database server can take it, try doubling the max_connections.
Mazz also pointed out docs on this:
http://rhq-project.org/display/JOPR2/Communications+Configuration#Communi...
If you do add another JON node, you'll definitely need to up the max_connections
because you are increasing the potential number of connections made to the db by 33%
As mazz mentioned, you can stress the db in several directions
i) lots of agents, few metrics => lots of connections
ii) few agents, lots of metrics => lots of work in the db
iii) lots of agents, lots of metrics => lots of connections and lots of db work
each of which will manifest themselves in different way. It may be helpful to run
"select * from pg_stat_activity;" if you hit this situation again and see what
the various connections are upto.
Let us know how you get on
Cheers
Charles
----- Original Message -----
> Depending on the backend, 400 agents to a single server may be too
> much.
> That said, I remember about a year ago someone tested using a database
> with SSD storage and it was blazing fast and probably could have
> handled
> more than 400 agents per server. It (mainly) depends on the database -
> how well it is tuned and how fast the disks are. The number of managed
> resources also has an impact - if your 800 agents are only monitoring,
> say, only the operating system and its child services, it wouldn't be
> as
> bad as if all 800 are monitoring a couple JBossAS instances each (the
> difference is the number of measurement reports that each agent would
> be
> sending up to the server).
>
> Anyway, aside from that, global concurrency limits should help. But
> they
> only delay the agents reports getting into the server. Once a
> concurrency limit is hit, the agents are told to "hold" for a few
> seconds (up to I think a minute or so) and then try the message again
> and if there is room, the message goes through (otherwise, the
> concurrency limit kicks in again and the agent waits again). Its
> essentially a throttling mechanism.
>
> If you are starting all your 800 agents at roughly the same time
> (using
> the agentspawn stuff, you normally launch them about 1m apart) then
> the
> server's will get clobbered with all of their synch requests and
> initial
> inventory reports. This causes immense traffic (which, btw, is not
> typically going to be normal when in real production, since it is rare
> you will start 800 machines/agents at all roughly the same time or
> within minutes apart from each other). Anyway, this immense traffic
> will
> cause the concurrency limits to kick in and you probably have to sit
> and
> wait for steady state to come before you see things calm down.
>
> Check your server logs and see what concurrency limits are getting
> triggered. They come out at the DEBUG level, the jboss-log4j.xml
> categories are:
>
> <!-- details on incoming remote pojo invocations; emits concurrency
> limit debug messages -->
> <!--
> <category
>
name="org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService">
> <priority value="DEBUG"/>
> </category>
> -->
>
> <!-- emits global concurrency limit debug messages -->
> <!--
> <category
>
name="org.rhq.enterprise.communications.GlobalConcurrencyLimitCommandListener">
> <priority value="DEBUG"/>
> </category>
> -->
>
> All that said, I think you are hitting your upper limit on the number
> of
> agents per server. It would not surprise me that you simply must
> increase the number of servers in your HA setup.
>