can't get ha failover to work
Bala Nair
bnairtm at comcast.net
Fri Oct 15 16:39:25 UTC 2010
We're trying to set up an rhq HA cloud with 2 servers and 4 agents and
we're having a problem getting the agents to failover to the second
server. When we first start up everything all the agents are connected
to one server (call it server A) with the other server (server B) not
connected to any agents. The failover list on the agent side showing 2
entries (server B and server A in that order). We go to the HA servers
page in the gui and see both servers are in NORMAL mode with server A
having an agent count of 4 and server B a count of 0. There are no
affinity groups. We then set server A to MAINTENANCE mode and wait. I
expect the 4 agents connected to server A to failover to server B and to
see that in the servers list, but nothing changes. Checking the agent
logs I find the following errors:
2010-10-15 11:17:46,653 INFO [RHQ Server Polling Thread]
(enterprise.communications.command.client.JBossRemotingRemoteCommunicator)-
{JBossRemotingRemoteCommunicator.changing-endpoint}Communicator is
changing endpoint from [InvokerLocator
[servlet://mmc-int:7080/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
to [InvokerLocator
[servlet://127.0.0.1/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
2010-10-15 11:17:46,654 WARN [RHQ Server Polling Thread]
(org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed
to failover to another server. Cause:
org.jboss.remoting.CannotConnectException: Can not connect http client
invoker. Connection refused.
2010-10-15 11:17:46,658 INFO [RHQ Server Polling Thread]
(enterprise.communications.command.client.JBossRemotingRemoteCommunicator)-
{JBossRemotingRemoteCommunicator.changing-endpoint}Communicator is
changing endpoint from [InvokerLocator
[servlet://127.0.0.1/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
to [InvokerLocator
[servlet://mmc-int:7080/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
2010-10-15 11:17:46,661 WARN [RHQ Server Polling Thread]
(org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed
to failover to another server. Cause:
org.rhq.enterprise.communications.util.NotProcessedException
2010-10-15 11:17:46,663 WARN [RHQ Server Polling Thread]
(org.rhq.enterprise.agent.FailoverFailureCallback)-
{AgentMain.too-many-failover-attempts}Too many failover attempts have
been made [2]. Exception that triggered the failover:
[org.rhq.enterprise.communications.util.NotProcessedException]
2010-10-15 11:17:46,663 ERROR [RHQ Server Polling Thread]
(enterprise.communications.command.client.JBossRemotingRemoteCommunicator)-
{JBossRemotingRemoteCommunicator.init-callback-failed}The initialize
callback has failed. It will be tried again. Cause:
org.rhq.enterprise.communications.util.NotProcessedException:null.
Cause: org.rhq.enterprise.communications.util.NotProcessedException
In this case mmc-int is server A. I can understand the second series of
errors where it tries to fail back to mmc-int and fails because mmc-int
is in maintenance mode. I don't understand the initial failure though.
First why is it trying to fail over to localhost instead of server B and
second why the connection refused error? There is no rhq server on this
agent box to refuse a connection.
I have looked through all the agent and server configuration properties
and I just don't see how the localhost address is getting set in this
case. Any help would be appreciated. Thanks.
Bala Nair
SeaChange International
More information about the rhq-devel
mailing list