Forced server MAINTENANCE mode seems too drastic

Larry O'Leary loleary at redhat.com
Fri Sep 19 14:41:20 UTC 2014


I have recently come across what I believe is an oversight in how the
storage node can force the server into maintenance mode. 

The problem scenario appears to affect multiple users and can lead to
system downtime. It boils down to the fact that if a storage node goes
down for any reason, you can't start it back up or use the alerting
functionality to handle such failures. This is not limited to just the
storage node resource. It can impact all resources in the system as long
as the storage node's state is such that the server gets stuck in
maintenance mode.

What is not clear, why is the entire communication layer being shutdown
just because the storage node is unavailable? I understand that we can
not collect metrics, but the agent's measurement reports could easily be
rejected and told to try again in a minute. In the meantime,
availability and most importantly, operations/management should still be
allowed.

In the current implementation, once the storage node goes offline, the
user must wait for the data center management group to restart the
entire system. This is because the RHQ server and the agent responsible
for managing the storage node, become unavailable for both management
and monitoring. The result is that the entire system goes offline and
can not be recovered from the RHQ server itself.

Are there plans to fix this? This issue can occur:

- during a scheduled restart of the storage node
- when a restart operation of the storage node fails
- when the storage node throws an error that requires a restart


Thanks,
-- 
Larry O'Leary
https://plus.google.com/+LarryOLeary



More information about the rhq-devel mailing list