can't get ha failover to work

Jay Shaughnessy jshaughn at redhat.com
Mon Oct 18 13:23:09 UTC 2010


I agree that something does seem amiss.  I would have thought the 
failover lists on the agents would be AB, AB, BA, BA, so that seems 
strange and I'm not sure how that would have happened unless perhaps the 
failover lists were generated when server B was down.  With bother 
servers in NORMAL mode perhaps force a repartition and see if that has 
any effect.

The Agent connections would be split 4/0 if the agents could not connect 
to server B for some reason.    Is 127.0.0.1 actually in the failover 
list for any of the agents?    You could perhaps inspect the partition 
even history for a clue as to what sequence of events led to the current 
situation.


On 10/15/2010 12:39 PM, Bala Nair wrote:
>    We're trying to set up an rhq HA cloud with 2 servers and 4 agents and
> we're having a problem getting the agents to failover to the second
> server.  When we first start up everything all the agents are connected
> to one server (call it server A) with the other server (server B) not
> connected to any agents. The failover list on the agent side showing 2
> entries (server B and server A in that order).  We go to the HA servers
> page in the gui and see both servers are in NORMAL mode with server A
> having an agent count of 4 and server B a count of 0.  There are no
> affinity groups.  We then set server A to MAINTENANCE mode and wait.  I
> expect the 4 agents connected to server A to failover to server B and to
> see that in the servers list, but nothing changes.  Checking the agent
> logs I find the following errors:
>
> 2010-10-15 11:17:46,653 INFO  [RHQ Server Polling Thread]
> (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)-
> {JBossRemotingRemoteCommunicator.changing-endpoint}Communicator is
> changing endpoint from [InvokerLocator
> [servlet://mmc-int:7080/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
> to [InvokerLocator
> [servlet://127.0.0.1/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
>
> 2010-10-15 11:17:46,654 WARN  [RHQ Server Polling Thread]
> (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed
> to failover to another server. Cause:
> org.jboss.remoting.CannotConnectException: Can not connect http client
> invoker. Connection refused.
>
> 2010-10-15 11:17:46,658 INFO  [RHQ Server Polling Thread]
> (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)-
> {JBossRemotingRemoteCommunicator.changing-endpoint}Communicator is
> changing endpoint from [InvokerLocator
> [servlet://127.0.0.1/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
> to [InvokerLocator
> [servlet://mmc-int:7080/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
>
> 2010-10-15 11:17:46,661 WARN  [RHQ Server Polling Thread]
> (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed
> to failover to another server. Cause:
> org.rhq.enterprise.communications.util.NotProcessedException
>
> 2010-10-15 11:17:46,663 WARN  [RHQ Server Polling Thread]
> (org.rhq.enterprise.agent.FailoverFailureCallback)-
> {AgentMain.too-many-failover-attempts}Too many failover attempts have
> been made [2]. Exception that triggered the failover:
> [org.rhq.enterprise.communications.util.NotProcessedException]
>
> 2010-10-15 11:17:46,663 ERROR [RHQ Server Polling Thread]
> (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)-
> {JBossRemotingRemoteCommunicator.init-callback-failed}The initialize
> callback has failed. It will be tried again. Cause:
> org.rhq.enterprise.communications.util.NotProcessedException:null.
> Cause: org.rhq.enterprise.communications.util.NotProcessedException
>
>
> In this case mmc-int is server A.  I can understand the second series of
> errors where it tries to fail back to mmc-int and fails because mmc-int
> is in maintenance mode.  I don't understand the initial failure though.
> First why is it trying to fail over to localhost instead of server B and
> second why the connection refused error?  There is no rhq server on this
> agent box to refuse a connection.
>
> I have looked through all the agent and server configuration properties
> and I just don't see how the localhost address is getting set in this
> case.  Any help would be appreciated.  Thanks.
>
> Bala Nair
> SeaChange International
>
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel
>


More information about the rhq-devel mailing list