When the server is in maintenance mode, it rejects all incoming requests from agents. When
the storage cluster is down, we automatically put the server into maintenance mode. This
came up while I was discussing with Michael Burman the work he is doing for adding support
for changing the location of the rhq-data directory[1].
Changing the data directory will involve copying files and shutting down the storage node.
It could potentially be a long running operation if there is a lot of data and/or slow
disk. This can get a bit tricky when we have just one storage node. If it takes a while to
move the data, there is a good chance that the server will go into maintenance mode. This
could be problematic since the agent won’t be able to report the results of the operation
back to the server. On the other hand, we cannot insert metrics while the node is down;
so, maintenance mode kind of makes sense since we cannot process the measurement reports.
I am not sure about the best way to handle this. While the storage node is down, we could
add the measurement reports to some form of a queue on the server. When the node comes
back up, we would then drain the queue. Alternatively, we could change the behavior in the
agent/server comm layer. As I understand it, if the server throws an exception when the
agent sends a measurement report, the agent will immediately try to resend. We could add a
delay in there so that the agent wouldn’t keep hammering the server.
I have not thought about this enough yet to arrive at an opinion. What do others think?
And are there any other options that we ought to consider?
[1]
https://bugzilla.redhat.com/show_bug.cgi?id=1074636