Forced server MAINTENANCE mode seems too drastic

Elias Ross genman at noderunner.net
Fri Sep 19 16:43:41 UTC 2014


I agree. It doesn't just happen when a storage node goes down.

If aggregation happens to cause a time out, then the whole
communication layer is shut down. I experienced this yesterday when
increasing the max rate on 4.12.

If you have multiple servers it's usually not a critical issue,
though, as the agents will connect over to the second. But it is
unnecessary.


On Fri, Sep 19, 2014 at 7:41 AM, Larry O'Leary <loleary at redhat.com> wrote:
> I have recently come across what I believe is an oversight in how the
> storage node can force the server into maintenance mode.
>
> The problem scenario appears to affect multiple users and can lead to
> system downtime. It boils down to the fact that if a storage node goes
> down for any reason, you can't start it back up or use the alerting
> functionality to handle such failures. This is not limited to just the
> storage node resource. It can impact all resources in the system as long
> as the storage node's state is such that the server gets stuck in
> maintenance mode.
>
> What is not clear, why is the entire communication layer being shutdown
> just because the storage node is unavailable? I understand that we can
> not collect metrics, but the agent's measurement reports could easily be
> rejected and told to try again in a minute. In the meantime,
> availability and most importantly, operations/management should still be
> allowed.
>
> In the current implementation, once the storage node goes offline, the
> user must wait for the data center management group to restart the
> entire system. This is because the RHQ server and the agent responsible
> for managing the storage node, become unavailable for both management
> and monitoring. The result is that the entire system goes offline and
> can not be recovered from the RHQ server itself.
>
> Are there plans to fix this? This issue can occur:
>
> - during a scheduled restart of the storage node
> - when a restart operation of the storage node fails
> - when the storage node throws an error that requires a restart
>
>
> Thanks,
> --
> Larry O'Leary
> https://plus.google.com/+LarryOLeary
>
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>


More information about the rhq-devel mailing list