When the server is in maintenance mode, it rejects all incoming requests from agents. When the storage cluster is down, we automatically put the server into maintenance mode. This came up while I was discussing with Michael Burman the work he is doing for adding support for changing the location of the rhq-data directory[1].

Changing the data directory will involve copying files and shutting down the storage node. It could potentially be a long running operation if there is a lot of data and/or slow disk. This can get a bit tricky when we have just one storage node. If it takes a while to move the data, there is a good chance that the server will go into maintenance mode. This could be problematic since the agent won’t be able to report the results of the operation back to the server. On the other hand, we cannot insert metrics while the node is down; so, maintenance mode kind of makes sense since we cannot process the measurement reports.

I am not sure about the best way to handle this. While the storage node is down, we could add the measurement reports to some form of a queue on the server. When the node comes back up, we would then drain the queue. Alternatively, we could change the behavior in the agent/server comm layer. As I understand it, if the server throws an exception when the agent sends a measurement report, the agent will immediately try to resend. We could add a delay in there so that the agent wouldn’t keep hammering the server. 

I have not thought about this enough yet to arrive at an opinion. What do others think? And are there any other options that we ought to consider?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1074636