AS is killed if the agent is killed (provided it was started/restarted by the agent)

John Mazzitelli mazz at redhat.com
Thu Feb 14 01:35:40 UTC 2013


Would you agree that production environments do not run the agent in foreground?

I really don't think this is a major issue for production environments - I don't know - do people run the agent's rhq-agent.sh (and not rhq-agent-wrapper.sh) most of the time?

It sounds to me like this is a solution looking for a problem?

----- Original Message -----
> All this talk about documenting a flaw in an FAQ seems scary to me.
> 
> Essentially, what I am hearing is, "Bring down your AS7 instances and
> try to guess that the RHQ agent is to blame. Then, read the FAQ to
> discover why you shouldn't have done that in the first place."
> 
> This is something that I think needs some more research. This is not
> something that other products have an issue with. If pressing CTRL+C
> in
> a shell caused this type of behavior then I would expect that
> pressing
> that in my root terminal would cause all of the processes I started
> from
> there exit. This just isn't the case. I think the issue has to do
> with
> how we are building the execution environment.
> 
> If we can't figure a way around this then perhaps the more
> appropriate
> fix is to disable the JVM's signal handling and implement our own in
> the
> agent's main. That way, we can still handle the CTRL+C and interpret
> it
> as a SIGQUIT instead of a SIGINT that gets passed to all of the
> agent's
> threads including the AS7 shell.
> 
> Again, putting some text in a document doesn't help solve the issue
> and
> won't gain any points when users realize that an RHQ agent is very
> destructive in a production setting. If we are going to take that
> approach then we need to prevent CTRL+C in the first place and just
> live
> with the consequence. Then document why we disabled it in the FAQ.
> 
> On Wed, 2013-02-13 at 09:05 -0500, John Mazzitelli wrote:
> > Note also, there is another solution we can document in the FAQ.
> > The user can write their own "launch-as7-in-background.sh" and
> > point to that script in the AS7's plugin configuration's "start
> > script" property (I forget the actual name of that property, but
> > the AS4 and AS5 server resources had something similar, so I
> > assume the AS7 plugin provides that too). That script can do what
> > we need and when the agent wants to start AS7, it will just launch
> > that script, which does all the "magic" that it needs to isolate
> > itself from the parent agent process.
> > 
> > 
> > ----- Original Message -----
> > > > mazz: Why does Java still have a daemon thread running that
> > > > forked
> > > > the AS7 process? Shouldn't that be a fire and forget?
> > > > thomas: As far as I know, the child processes we create with
> > > > Runtime.exec haveno link with internal JVM threads
> > > 
> > > you are both right, Runtime.exec() is a non-blocking call
> > > creating a
> > > new child process by default. In our case two deamon threads are
> > > spawned for handling the streams, but it has no impact on the
> > > original problem (calling interrupt() oh these two threads
> > > doesn't
> > > send the signal to the AS process) there is also waitFor() method
> > > and for AS7 we wait for 15s but it is irrelevant.
> > > 
> > > > I still thought
> > > > that if you kill a parent thread, any child threads are killed
> > > > along
> > > > with it - this is true regardless of whether the parent process
> > > > is
> > > > a
> > > > Java JVM or not - this is just the way the UNIX operating
> > > > system
> > > > works.
> > > 
> > > If the child process was run in the background (&), and the
> > > parent
> > > process is stopped, the child process is still running and it
> > > becomes the orphaned process with parent pid set to the init
> > > process, that is what we want for AS.
> > > 
> > > > In our case, the rhq-agent.sh script is run in the foreground
> > > > of an
> > > > interactive shell. So when you hit Ctrl-C, it's the interactive
> > > > shell
> > > > that sends the SIGINT signal to all processes of the group the
> > > > foreground process belongs to.
> > > 
> > > Right. If I run "./run-in-background.sh ./standalone.sh" from
> > > interactive shell and press Ctrl+C, the AS won't get killed even
> > > it
> > > is in has the same PGID (because it was run as a background
> > > process
> > > using the & in the ./run-in-background.sh script), so it looks
> > > like
> > > the SIGINT is resent only to those child processes that was run
> > > in
> > > the foreground.
> > > 
> > > 
> > > All in all, I am +1 on a FAQ note as Mazz suggested, because
> > > adding
> > > an '&' in between any two processes may be risky. There are 4
> > > cases
> > > where the "run-in-background" operation may be added (either
> > > rhq-agent.sh and JVM (1), JVM and standalone.sh (2),
> > > standalone.sh
> > > and JVM (3), or JVM calling ./run-in-background.sh calling
> > > standalone.sh in the background)
> > > 
> > > case (1) : impossible, we need to handle user input (typing
> > > commands
> > > to agent's console)
> > > case (2) : it is possible, but I had to concatenate "& exit 0" in
> > > the
> > > Runtime.exec method, I guess we would lose the return code
> > > case (3) : we can't modify standalone.sh, or more precisely, it
> > > should work for the default impl. of standalone.sh
> > > case (4) : it is possible, but we may again lose the return code,
> > > I
> > > haven't tried it though
> > > 
> > > Handling the signals manually (trap commands) is another way to
> > > "separate" two processes, as Thomas suggested. This applies only
> > > for
> > > the scenario (1), because 'trap' is a Unix command. However, it
> > > seems contradictory to me as I've already written.
> > > 
> > > jk
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "John Mazzitelli" <mazz at redhat.com>
> > > > To: rhq-devel at lists.fedorahosted.org
> > > > Sent: Tuesday, February 12, 2013 7:37:42 PM
> > > > Subject: Re: AS is killed if the agent is killed (provided it
> > > > was
> > > > 	started/restarted by the agent)
> > > > 
> > > > Why does Java still have a daemon thread running that forked
> > > > the
> > > > AS7
> > > > process? Shouldn't that be a fire and forget? I would think the
> > > > child process would get forked/exec'd and then we forget about
> > > > it,
> > > > our thread ends. That seems wrong that we are allowing a daemon
> > > > thread to remain around. Are we sure it needs to be like that?
> > > > to
> > > > me, that would be the thing to try to fix (though, I still
> > > > thought
> > > > that if you kill a parent thread, any child threads are killed
> > > > along
> > > > with it - this is true regardless of whether the parent process
> > > > is
> > > > a
> > > > Java JVM or not - this is just the way the UNIX operating
> > > > system
> > > > works. I could be mistaken.
> > > > 
> > > > ----- Original Message -----
> > > > > > BTW, the reason for Ctrl-C killing all processes is that
> > > > > > the
> > > > > > signal
> > > > > > is sent to all processes in the same group.
> > > > > 
> > > > > We implement it manually in the
> > > > > AgentShutdownHook.interruptAllThreads().
> > > > > 
> > > > > 
> > > > > if there is a way to limit the signal to only getting sent to
> > > > > the
> > > > > > agent process (and not its child processes such that the
> > > > > > child
> > > > > > processes remain alive and running even after the parent
> > > > > > process
> > > > > > (the agent) dies), then that is something we should look
> > > > > > into.
> > > > > > I
> > > > > > didn't think it was possible to do that, but if so, that
> > > > > > would
> > > > > > be
> > > > > > great.
> > > > > 
> > > > > The problem here is that on the one hand we need to send
> > > > > SIGINT
> > > > > to
> > > > > the agent to clear its resources, on the other hand we don't
> > > > > want
> > > > > to
> > > > > send the SIGINT to the angent, because he will relay it to
> > > > > the
> > > > > all
> > > > > daemon threads (including the running standalone.sh (run by
> > > > > Java
> > > > > runtime)). So it is obviously contradictory. Something like
> > > > > "trap
> > > > > 'kill -9 myPid' INT" in the rhq.agent.sh would work, but it
> > > > > wouldn't
> > > > > clear the resources.
> > > > > 
> > > > > The only way, I am aware of, is using the & operator:
> > > > > 
> > > > > "nohup ./rhq-agent.sh --daemon &"  works well
> > > > > "./run-in-background.sh ./standalone.sh" works well
> > > > > or "Runtime.exec("standalone.sh &")" works well
> > > > > 
> > > > > check the att., I did a sequence diagram for it, because it
> > > > > is
> > > > > quite
> > > > > complex.
> > > > > 
> > > > > 
> > > > > jk
> > > > > 
> > > > > 
> > > > > 
> > > > > ----- Original Message -----
> > > > > > From: "John Mazzitelli" <mazz at redhat.com>
> > > > > > To: rhq-devel at lists.fedorahosted.org
> > > > > > Sent: Tuesday, February 12, 2013 5:45:53 PM
> > > > > > Subject: Re: AS is killed if the agent is killed (provided
> > > > > > it
> > > > > > was
> > > > > > 	started/restarted by the agent)
> > > > > > 
> > > > > > > BTW, the reason for Ctrl-C killing all processes is that
> > > > > > > the
> > > > > > > signal
> > > > > > > is sent to all processes in the same group.
> > > > > > 
> > > > > > if there is a way to limit the signal to only getting sent
> > > > > > to
> > > > > > the
> > > > > > agent process (and not its child processes such that the
> > > > > > child
> > > > > > processes remain alive and running even after the parent
> > > > > > process
> > > > > > (the agent) dies), then that is something we should look
> > > > > > into.
> > > > > > I
> > > > > > didn't think it was possible to do that, but if so, that
> > > > > > would
> > > > > > be
> > > > > > great.
> > > > > > 
> > > > > > But we should not eliminate the ability to control-C the
> > > > > > agent
> > > > > > itself.
> > > > > > _______________________________________________
> > > > > > rhq-devel mailing list
> > > > > > rhq-devel at lists.fedorahosted.org
> > > > > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > > > > 
> > > > > _______________________________________________
> > > > > rhq-devel mailing list
> > > > > rhq-devel at lists.fedorahosted.org
> > > > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > > > 
> > > > _______________________________________________
> > > > rhq-devel mailing list
> > > > rhq-devel at lists.fedorahosted.org
> > > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > > 
> > > _______________________________________________
> > > rhq-devel mailing list
> > > rhq-devel at lists.fedorahosted.org
> > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > 
> > _______________________________________________
> > rhq-devel mailing list
> > rhq-devel at lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> 
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> 


More information about the rhq-devel mailing list