On 2/20/2014 10:32 AM, Larry O'Leary wrote:
On Wed, 2014-02-19 at 14:02 -0500, Jay Shaughnessy wrote:
>>> But there already is timeout for operations. Is there a way to get to
>>> this value at runtime on agent side within operation code? I am only
>>> able to get to default value using ResourceType.
>> I think there was some work done by Thomas S to use that timeout in
>> other places. If that timeout is not yet
>> available inside the plugin code, we should expose it
> Today we specifically hide that information from the plugin code. And
> the ultimate value used for the operation timeout has 4 potential
> values: The value set by the user for that operation invocation, the
> value set in the operation metadata, the setting of
> rhq.pc.operation-invocation-timeout or finally, the default of 600s (5
> minutes). If we did expose it we'd have to pass it as a "special"
> operation parameter, I think.
>
> This issue right now is that we don't expose it to the plugin code (and
> I'm still not sure that we should) and we also don't provide any
> mechanism to kill a process spawned by the plugin's invokeOperation code
> (I'm assuming the process would be spawned by
> SystemInfo.executeProcess(ProcessExecution)).
>
> If you really want to tie the spawned processed life to the operations
> timeout value then I think the correct way to do things would be to have
> the invokeOperation() impl handle the InterruptedException generated by
> the plugin container and then destroy the Process in that handling
> code. Doing this would mean we'd to provide a hook to get the spawned
> process.
>
> if you want the spawned process to run independently of the operation's
> timeout, possible living a shorter or longer life, then I think the
> timeout for the process should be specified as an operation parameter
> and then supplied to processExecution.setWaitForCompletion();
I think this does help highlight what the intended operation timeout
value in its current form means. Right now, I think the expectation is
that if I invoke an operation and it "times out" does it just not mean
that we stopped waiting for a response? Do we really want to "cancel" or
"kill" the operation?
In some cases, I think so. But in others, perhaps not. When I invoke the
stop operation for an AS resource it may take longer then 5 minutes to
shut down. That doesn't mean that it fails to stop or the stop is
canceled. Instead, it just means that we have timed out waiting for a
response and the status message to the user indicates that we timed out
waiting for a response.
From what I can gather, the behavior of the operation timeout will vary
based on the operation's implementation. The operation timeout (or a
user-initiated cancel) will basically interrupt the invokeOperation()
execution if it hasn't returned. If the operation ignores the interrupt
then it basically just keeps executing. If it checks for the interrupt
then it can perform any action it wants, and then throw
InterruptedException to indicate that it handled the interrupt. The
interrupt handling is typically expected to cancel the work associated
with the operation. In theory operation implementations are expected to
handle interrupts and cancel themselves. In practice I don't think that
is often the case and instead, I think most interrupts are ignored.
As for what to do with this cli script operation, there is something to
consider that I didn't see before. This whole topic has basically
already been addressed by the Script plugin's invokeOperation()
implementation. It's doing very much the same thing for it's "execute"
operation as we're discussing here for wildfly cli scripts. The
approach taken there is to use several operation parameters:
"arguments", "waitTime", "captureOutput",
"killOnTimeout". The waitTime
and killOnTimeout values are passed straight into ProcessExecution and
guide its behavior. It does *not* handle interrupts and so neither
cancel or operation timeout affect the script execution. But it's
straightforward as to how the script execution is handled.
For consistency we may want to just do the same thing. I'd suggest only
that we be more clear in the operation description about how it ignores
operationTimeout and cancel. We should likely update the script plugin
as well.
Perhaps a secondary, user specified timeout is necessary when spawning
processes or at a minimum, an option that says "terminate on timeout?"
By providing such a secondary timeout value or an interrupt/terminate
option you are allowing the invocation of an operation to provide the
user with these questions/options:
- How long do you want to wait for the operation to complete before
reporting its status? Status could be "unknown due to timeout."
- If spawning a process/thread, how long do you want to wait for it to
complete before a failure is reported? In this case, if we timeout
before the operation timeout then we would report "may have failed
because it took too long."
- If we wait too long for the spawned process/thread to complete,
should it be interrupted/terminated? This simply means that if we have
waiting long enough for a response/result, we interrupt the thread
hoping that it properly terminates/kills the spawned process/thread.