Can you review the server endpoints you have defined? Perhaps you installed one with endpoint address "localhost". If so, you'll want to edit that server entry and put in the correct publicly reachable IP. (well, public with respect to your agents...)
On 10/18/2010 9:23 AM, Jay Shaughnessy wrote:
I agree that something does seem amiss. I would have thought the failover lists on the agents would be AB, AB, BA, BA, so that seems strange and I'm not sure how that would have happened unless perhaps the failover lists were generated when server B was down. With bother servers in NORMAL mode perhaps force a repartition and see if that has any effect.
The Agent connections would be split 4/0 if the agents could not connect to server B for some reason. Is 127.0.0.1 actually in the failover list for any of the agents? You could perhaps inspect the partition even history for a clue as to what sequence of events led to the current situation.
On 10/15/2010 12:39 PM, Bala Nair wrote:
We're trying to set up an rhq HA cloud with 2 servers and 4 agents and
we're having a problem getting the agents to failover to the second server. When we first start up everything all the agents are connected to one server (call it server A) with the other server (server B) not connected to any agents. The failover list on the agent side showing 2 entries (server B and server A in that order). We go to the HA servers page in the gui and see both servers are in NORMAL mode with server A having an agent count of 4 and server B a count of 0. There are no affinity groups. We then set server A to MAINTENANCE mode and wait. I expect the 4 agents connected to server A to failover to server B and to see that in the servers list, but nothing changes. Checking the agent logs I find the following errors:
2010-10-15 11:17:46,653 INFO [RHQ Server Polling Thread] (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)- {JBossRemotingRemoteCommunicator.changing-endpoint}Communicator is changing endpoint from [InvokerLocator [servlet://mmc-int:7080/jboss-remoting-servlet-invoker/ServerInvokerServlet]] to [InvokerLocator [servlet://127.0.0.1/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
2010-10-15 11:17:46,654 WARN [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed to failover to another server. Cause: org.jboss.remoting.CannotConnectException: Can not connect http client invoker. Connection refused.
2010-10-15 11:17:46,658 INFO [RHQ Server Polling Thread] (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)- {JBossRemotingRemoteCommunicator.changing-endpoint}Communicator is changing endpoint from [InvokerLocator [servlet://127.0.0.1/jboss-remoting-servlet-invoker/ServerInvokerServlet]] to [InvokerLocator [servlet://mmc-int:7080/jboss-remoting-servlet-invoker/ServerInvokerServlet]]
2010-10-15 11:17:46,661 WARN [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed to failover to another server. Cause: org.rhq.enterprise.communications.util.NotProcessedException
2010-10-15 11:17:46,663 WARN [RHQ Server Polling Thread] (org.rhq.enterprise.agent.FailoverFailureCallback)- {AgentMain.too-many-failover-attempts}Too many failover attempts have been made [2]. Exception that triggered the failover: [org.rhq.enterprise.communications.util.NotProcessedException]
2010-10-15 11:17:46,663 ERROR [RHQ Server Polling Thread] (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)- {JBossRemotingRemoteCommunicator.init-callback-failed}The initialize callback has failed. It will be tried again. Cause: org.rhq.enterprise.communications.util.NotProcessedException:null. Cause: org.rhq.enterprise.communications.util.NotProcessedException
In this case mmc-int is server A. I can understand the second series of errors where it tries to fail back to mmc-int and fails because mmc-int is in maintenance mode. I don't understand the initial failure though. First why is it trying to fail over to localhost instead of server B and second why the connection refused error? There is no rhq server on this agent box to refuse a connection.
I have looked through all the agent and server configuration properties and I just don't see how the localhost address is getting set in this case. Any help would be appreciated. Thanks.
Bala Nair SeaChange International
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/rhq-devel
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/rhq-devel