I agree. It doesn't just happen when a storage node goes down.
If aggregation happens to cause a time out, then the whole
communication layer is shut down. I experienced this yesterday when
increasing the max rate on 4.12.
If you have multiple servers it's usually not a critical issue,
though, as the agents will connect over to the second. But it is
unnecessary.
On Fri, Sep 19, 2014 at 7:41 AM, Larry O'Leary <loleary(a)redhat.com> wrote:
I have recently come across what I believe is an oversight in how
the
storage node can force the server into maintenance mode.
The problem scenario appears to affect multiple users and can lead to
system downtime. It boils down to the fact that if a storage node goes
down for any reason, you can't start it back up or use the alerting
functionality to handle such failures. This is not limited to just the
storage node resource. It can impact all resources in the system as long
as the storage node's state is such that the server gets stuck in
maintenance mode.
What is not clear, why is the entire communication layer being shutdown
just because the storage node is unavailable? I understand that we can
not collect metrics, but the agent's measurement reports could easily be
rejected and told to try again in a minute. In the meantime,
availability and most importantly, operations/management should still be
allowed.
In the current implementation, once the storage node goes offline, the
user must wait for the data center management group to restart the
entire system. This is because the RHQ server and the agent responsible
for managing the storage node, become unavailable for both management
and monitoring. The result is that the entire system goes offline and
can not be recovered from the RHQ server itself.
Are there plans to fix this? This issue can occur:
- during a scheduled restart of the storage node
- when a restart operation of the storage node fails
- when the storage node throws an error that requires a restart
Thanks,
--
Larry O'Leary
https://plus.google.com/+LarryOLeary
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.