can't get ha failover to work
Jay Shaughnessy
jshaughn at redhat.com
Mon Oct 18 13:23:09 UTC 2010
I agree that something does seem amiss. I would have thought the
failover lists on the agents would be AB, AB, BA, BA, so that seems
strange and I'm not sure how that would have happened unless perhaps the
failover lists were generated when server B was down. With bother
servers in NORMAL mode perhaps force a repartition and see if that has
any effect.
The Agent connections would be split 4/0 if the agents could not connect
to server B for some reason. Is 127.0.0.1 actually in the failover
list for any of the agents? You could perhaps inspect the partition
even history for a clue as to what sequence of events led to the current
situation.
On 10/15/2010 12:39 PM, Bala Nair wrote:
> We're trying to set up an rhq HA cloud with 2 servers and 4 agents and
> we're having a problem getting the agents to failover to the second
> server. When we first start up everything all the agents are connected
> to one server (call it server A) with the other server (server B) not
> connected to any agents. The failover list on the agent side showing 2
> entries (server B and server A in that order). We go to the HA servers
> page in the gui and see both servers are in NORMAL mode with server A
> having an agent count of 4 and server B a count of 0. There are no
> affinity groups. We then set server A to MAINTENANCE mode and wait. I
> expect the 4 agents connected to server A to failover to server B and to
> see that in the servers list, but nothing changes. Checking the agent
> logs I find the following errors:
>
> 2010-10-15 11:17:46,653 INFO [RHQ Server Polling Thread]
> (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)-
> {JBossRemotingRemoteCommunicator.changing-endpoint}Communicator is
> changing endpoint from [InvokerLocator
> [servlet://mmc-int:7080/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
> to [InvokerLocator
> [servlet://127.0.0.1/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
>
> 2010-10-15 11:17:46,654 WARN [RHQ Server Polling Thread]
> (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed
> to failover to another server. Cause:
> org.jboss.remoting.CannotConnectException: Can not connect http client
> invoker. Connection refused.
>
> 2010-10-15 11:17:46,658 INFO [RHQ Server Polling Thread]
> (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)-
> {JBossRemotingRemoteCommunicator.changing-endpoint}Communicator is
> changing endpoint from [InvokerLocator
> [servlet://127.0.0.1/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
> to [InvokerLocator
> [servlet://mmc-int:7080/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
>
> 2010-10-15 11:17:46,661 WARN [RHQ Server Polling Thread]
> (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed
> to failover to another server. Cause:
> org.rhq.enterprise.communications.util.NotProcessedException
>
> 2010-10-15 11:17:46,663 WARN [RHQ Server Polling Thread]
> (org.rhq.enterprise.agent.FailoverFailureCallback)-
> {AgentMain.too-many-failover-attempts}Too many failover attempts have
> been made [2]. Exception that triggered the failover:
> [org.rhq.enterprise.communications.util.NotProcessedException]
>
> 2010-10-15 11:17:46,663 ERROR [RHQ Server Polling Thread]
> (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)-
> {JBossRemotingRemoteCommunicator.init-callback-failed}The initialize
> callback has failed. It will be tried again. Cause:
> org.rhq.enterprise.communications.util.NotProcessedException:null.
> Cause: org.rhq.enterprise.communications.util.NotProcessedException
>
>
> In this case mmc-int is server A. I can understand the second series of
> errors where it tries to fail back to mmc-int and fails because mmc-int
> is in maintenance mode. I don't understand the initial failure though.
> First why is it trying to fail over to localhost instead of server B and
> second why the connection refused error? There is no rhq server on this
> agent box to refuse a connection.
>
> I have looked through all the agent and server configuration properties
> and I just don't see how the localhost address is getting set in this
> case. Any help would be appreciated. Thanks.
>
> Bala Nair
> SeaChange International
>
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel
>
More information about the rhq-devel
mailing list