AS is killed if the agent is killed (provided it was started/restarted by the agent)

Larry O'Leary loleary at redhat.com
Wed Feb 13 23:19:30 UTC 2013


All this talk about documenting a flaw in an FAQ seems scary to me. 

Essentially, what I am hearing is, "Bring down your AS7 instances and
try to guess that the RHQ agent is to blame. Then, read the FAQ to
discover why you shouldn't have done that in the first place."

This is something that I think needs some more research. This is not
something that other products have an issue with. If pressing CTRL+C in
a shell caused this type of behavior then I would expect that pressing
that in my root terminal would cause all of the processes I started from
there exit. This just isn't the case. I think the issue has to do with
how we are building the execution environment.

If we can't figure a way around this then perhaps the more appropriate
fix is to disable the JVM's signal handling and implement our own in the
agent's main. That way, we can still handle the CTRL+C and interpret it
as a SIGQUIT instead of a SIGINT that gets passed to all of the agent's
threads including the AS7 shell.

Again, putting some text in a document doesn't help solve the issue and
won't gain any points when users realize that an RHQ agent is very
destructive in a production setting. If we are going to take that
approach then we need to prevent CTRL+C in the first place and just live
with the consequence. Then document why we disabled it in the FAQ.

On Wed, 2013-02-13 at 09:05 -0500, John Mazzitelli wrote:
> Note also, there is another solution we can document in the FAQ. The user can write their own "launch-as7-in-background.sh" and point to that script in the AS7's plugin configuration's "start script" property (I forget the actual name of that property, but the AS4 and AS5 server resources had something similar, so I assume the AS7 plugin provides that too). That script can do what we need and when the agent wants to start AS7, it will just launch that script, which does all the "magic" that it needs to isolate itself from the parent agent process.
> 
> 
> ----- Original Message -----
> > > mazz: Why does Java still have a daemon thread running that forked
> > > the AS7 process? Shouldn't that be a fire and forget?
> > > thomas: As far as I know, the child processes we create with
> > > Runtime.exec haveno link with internal JVM threads
> > 
> > you are both right, Runtime.exec() is a non-blocking call creating a
> > new child process by default. In our case two deamon threads are
> > spawned for handling the streams, but it has no impact on the
> > original problem (calling interrupt() oh these two threads doesn't
> > send the signal to the AS process) there is also waitFor() method
> > and for AS7 we wait for 15s but it is irrelevant.
> > 
> > > I still thought
> > > that if you kill a parent thread, any child threads are killed
> > > along
> > > with it - this is true regardless of whether the parent process is
> > > a
> > > Java JVM or not - this is just the way the UNIX operating system
> > > works.
> > 
> > If the child process was run in the background (&), and the parent
> > process is stopped, the child process is still running and it
> > becomes the orphaned process with parent pid set to the init
> > process, that is what we want for AS.
> > 
> > > In our case, the rhq-agent.sh script is run in the foreground of an
> > > interactive shell. So when you hit Ctrl-C, it's the interactive
> > > shell
> > > that sends the SIGINT signal to all processes of the group the
> > > foreground process belongs to.
> > 
> > Right. If I run "./run-in-background.sh ./standalone.sh" from
> > interactive shell and press Ctrl+C, the AS won't get killed even it
> > is in has the same PGID (because it was run as a background process
> > using the & in the ./run-in-background.sh script), so it looks like
> > the SIGINT is resent only to those child processes that was run in
> > the foreground.
> > 
> > 
> > All in all, I am +1 on a FAQ note as Mazz suggested, because adding
> > an '&' in between any two processes may be risky. There are 4 cases
> > where the "run-in-background" operation may be added (either
> > rhq-agent.sh and JVM (1), JVM and standalone.sh (2), standalone.sh
> > and JVM (3), or JVM calling ./run-in-background.sh calling
> > standalone.sh in the background)
> > 
> > case (1) : impossible, we need to handle user input (typing commands
> > to agent's console)
> > case (2) : it is possible, but I had to concatenate "& exit 0" in the
> > Runtime.exec method, I guess we would lose the return code
> > case (3) : we can't modify standalone.sh, or more precisely, it
> > should work for the default impl. of standalone.sh
> > case (4) : it is possible, but we may again lose the return code, I
> > haven't tried it though
> > 
> > Handling the signals manually (trap commands) is another way to
> > "separate" two processes, as Thomas suggested. This applies only for
> > the scenario (1), because 'trap' is a Unix command. However, it
> > seems contradictory to me as I've already written.
> > 
> > jk
> > 
> > 
> > ----- Original Message -----
> > > From: "John Mazzitelli" <mazz at redhat.com>
> > > To: rhq-devel at lists.fedorahosted.org
> > > Sent: Tuesday, February 12, 2013 7:37:42 PM
> > > Subject: Re: AS is killed if the agent is killed (provided it was
> > > 	started/restarted by the agent)
> > > 
> > > Why does Java still have a daemon thread running that forked the
> > > AS7
> > > process? Shouldn't that be a fire and forget? I would think the
> > > child process would get forked/exec'd and then we forget about it,
> > > our thread ends. That seems wrong that we are allowing a daemon
> > > thread to remain around. Are we sure it needs to be like that? to
> > > me, that would be the thing to try to fix (though, I still thought
> > > that if you kill a parent thread, any child threads are killed
> > > along
> > > with it - this is true regardless of whether the parent process is
> > > a
> > > Java JVM or not - this is just the way the UNIX operating system
> > > works. I could be mistaken.
> > > 
> > > ----- Original Message -----
> > > > > BTW, the reason for Ctrl-C killing all processes is that the
> > > > > signal
> > > > > is sent to all processes in the same group.
> > > > 
> > > > We implement it manually in the
> > > > AgentShutdownHook.interruptAllThreads().
> > > > 
> > > > 
> > > > if there is a way to limit the signal to only getting sent to the
> > > > > agent process (and not its child processes such that the child
> > > > > processes remain alive and running even after the parent
> > > > > process
> > > > > (the agent) dies), then that is something we should look into.
> > > > > I
> > > > > didn't think it was possible to do that, but if so, that would
> > > > > be
> > > > > great.
> > > > 
> > > > The problem here is that on the one hand we need to send SIGINT
> > > > to
> > > > the agent to clear its resources, on the other hand we don't want
> > > > to
> > > > send the SIGINT to the angent, because he will relay it to the
> > > > all
> > > > daemon threads (including the running standalone.sh (run by Java
> > > > runtime)). So it is obviously contradictory. Something like "trap
> > > > 'kill -9 myPid' INT" in the rhq.agent.sh would work, but it
> > > > wouldn't
> > > > clear the resources.
> > > > 
> > > > The only way, I am aware of, is using the & operator:
> > > > 
> > > > "nohup ./rhq-agent.sh --daemon &"  works well
> > > > "./run-in-background.sh ./standalone.sh" works well
> > > > or "Runtime.exec("standalone.sh &")" works well
> > > > 
> > > > check the att., I did a sequence diagram for it, because it is
> > > > quite
> > > > complex.
> > > > 
> > > > 
> > > > jk
> > > > 
> > > > 
> > > > 
> > > > ----- Original Message -----
> > > > > From: "John Mazzitelli" <mazz at redhat.com>
> > > > > To: rhq-devel at lists.fedorahosted.org
> > > > > Sent: Tuesday, February 12, 2013 5:45:53 PM
> > > > > Subject: Re: AS is killed if the agent is killed (provided it
> > > > > was
> > > > > 	started/restarted by the agent)
> > > > > 
> > > > > > BTW, the reason for Ctrl-C killing all processes is that the
> > > > > > signal
> > > > > > is sent to all processes in the same group.
> > > > > 
> > > > > if there is a way to limit the signal to only getting sent to
> > > > > the
> > > > > agent process (and not its child processes such that the child
> > > > > processes remain alive and running even after the parent
> > > > > process
> > > > > (the agent) dies), then that is something we should look into.
> > > > > I
> > > > > didn't think it was possible to do that, but if so, that would
> > > > > be
> > > > > great.
> > > > > 
> > > > > But we should not eliminate the ability to control-C the agent
> > > > > itself.
> > > > > _______________________________________________
> > > > > rhq-devel mailing list
> > > > > rhq-devel at lists.fedorahosted.org
> > > > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > > > 
> > > > _______________________________________________
> > > > rhq-devel mailing list
> > > > rhq-devel at lists.fedorahosted.org
> > > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > > 
> > > _______________________________________________
> > > rhq-devel mailing list
> > > rhq-devel at lists.fedorahosted.org
> > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > 
> > _______________________________________________
> > rhq-devel mailing list
> > rhq-devel at lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel




More information about the rhq-devel mailing list