On 2/20/2014 10:32 AM, Larry O'Leary wrote:
On Wed, 2014-02-19 at 14:02 -0500, Jay Shaughnessy wrote:
But there already is timeout for operations. Is there a way to get to this value at runtime on agent side within operation code? I am only able to get to default value using ResourceType.
I think there was some work done by Thomas S to use that timeout in other places. If that timeout is not yet available inside the plugin code, we should expose it
Today we specifically hide that information from the plugin code. And the ultimate value used for the operation timeout has 4 potential values: The value set by the user for that operation invocation, the value set in the operation metadata, the setting of rhq.pc.operation-invocation-timeout or finally, the default of 600s (5 minutes). If we did expose it we'd have to pass it as a "special" operation parameter, I think.
This issue right now is that we don't expose it to the plugin code (and I'm still not sure that we should) and we also don't provide any mechanism to kill a process spawned by the plugin's invokeOperation code (I'm assuming the process would be spawned by SystemInfo.executeProcess(ProcessExecution)).
If you really want to tie the spawned processed life to the operations timeout value then I think the correct way to do things would be to have the invokeOperation() impl handle the InterruptedException generated by the plugin container and then destroy the Process in that handling code. Doing this would mean we'd to provide a hook to get the spawned process.
if you want the spawned process to run independently of the operation's timeout, possible living a shorter or longer life, then I think the timeout for the process should be specified as an operation parameter and then supplied to processExecution.setWaitForCompletion();
I think this does help highlight what the intended operation timeout value in its current form means. Right now, I think the expectation is that if I invoke an operation and it "times out" does it just not mean that we stopped waiting for a response? Do we really want to "cancel" or "kill" the operation?
In some cases, I think so. But in others, perhaps not. When I invoke the stop operation for an AS resource it may take longer then 5 minutes to shut down. That doesn't mean that it fails to stop or the stop is canceled. Instead, it just means that we have timed out waiting for a response and the status message to the user indicates that we timed out waiting for a response.
From what I can gather, the behavior of the operation timeout will vary based on the operation's implementation. The operation timeout (or a user-initiated cancel) will basically interrupt the invokeOperation() execution if it hasn't returned. If the operation ignores the interrupt then it basically just keeps executing. If it checks for the interrupt then it can perform any action it wants, and then throw InterruptedException to indicate that it handled the interrupt. The interrupt handling is typically expected to cancel the work associated with the operation. In theory operation implementations are expected to handle interrupts and cancel themselves. In practice I don't think that is often the case and instead, I think most interrupts are ignored.
As for what to do with this cli script operation, there is something to consider that I didn't see before. This whole topic has basically already been addressed by the Script plugin's invokeOperation() implementation. It's doing very much the same thing for it's "execute" operation as we're discussing here for wildfly cli scripts. The approach taken there is to use several operation parameters: "arguments", "waitTime", "captureOutput", "killOnTimeout". The waitTime and killOnTimeout values are passed straight into ProcessExecution and guide its behavior. It does *not* handle interrupts and so neither cancel or operation timeout affect the script execution. But it's straightforward as to how the script execution is handled.
For consistency we may want to just do the same thing. I'd suggest only that we be more clear in the operation description about how it ignores operationTimeout and cancel. We should likely update the script plugin as well.
Perhaps a secondary, user specified timeout is necessary when spawning processes or at a minimum, an option that says "terminate on timeout?"
By providing such a secondary timeout value or an interrupt/terminate option you are allowing the invocation of an operation to provide the user with these questions/options:
- How long do you want to wait for the operation to complete before
reporting its status? Status could be "unknown due to timeout."
- If spawning a process/thread, how long do you want to wait for it to
complete before a failure is reported? In this case, if we timeout before the operation timeout then we would report "may have failed because it took too long."
- If we wait too long for the spawned process/thread to complete,
should it be interrupted/terminated? This simply means that if we have waiting long enough for a response/result, we interrupt the thread hoping that it properly terminates/kills the spawned process/thread.