AS is killed if the agent is killed (provided it was started/restarted by the agent)

Larry O'Leary loleary at redhat.com
Thu Feb 14 15:03:37 UTC 2013


On Wed, 2013-02-13 at 20:35 -0500, John Mazzitelli wrote:
> Would you agree that production environments do not run the agent in foreground?

For the most part yes. However, there are cases when this is not true.
For example, when performing diagnostics or when the user needs to
interact with the running agent. It is very common for the agent to run
in the foreground, even in production, from time to time.

> I really don't think this is a major issue for production environments - I don't know - do people run the agent's rhq-agent.sh (and not rhq-agent-wrapper.sh) most of the time?

How can killing unrelated processes not be considered a major issue.
Within in some environments, this would be considered malicious and the
agent would be embargoed. 

There are other ways this could happen. For example, platform level
monitoring systems may decide to restart this agent by sending it the
SIGINT signal. From the sounds of it, this too would cause the issue
whether the agent is in the foreground or background? 

> It sounds to me like this is a solution looking for a problem?

I think the problem is well defined. Perhaps the feeling is that a
solution is being investigated for an issue that users may not run into
that often. But the severity of the issue is what warrants thorough
thought from my perspective. As a user, dealing with random shut downs
of my AS server would not be a fun thing to deal with.

> ----- Original Message -----
> > All this talk about documenting a flaw in an FAQ seems scary to me.
> > 
> > Essentially, what I am hearing is, "Bring down your AS7 instances and
> > try to guess that the RHQ agent is to blame. Then, read the FAQ to
> > discover why you shouldn't have done that in the first place."
> > 
> > This is something that I think needs some more research. This is not
> > something that other products have an issue with. If pressing CTRL+C
> > in
> > a shell caused this type of behavior then I would expect that
> > pressing
> > that in my root terminal would cause all of the processes I started
> > from
> > there exit. This just isn't the case. I think the issue has to do
> > with
> > how we are building the execution environment.
> > 
> > If we can't figure a way around this then perhaps the more
> > appropriate
> > fix is to disable the JVM's signal handling and implement our own in
> > the
> > agent's main. That way, we can still handle the CTRL+C and interpret
> > it
> > as a SIGQUIT instead of a SIGINT that gets passed to all of the
> > agent's
> > threads including the AS7 shell.
> > 
> > Again, putting some text in a document doesn't help solve the issue
> > and
> > won't gain any points when users realize that an RHQ agent is very
> > destructive in a production setting. If we are going to take that
> > approach then we need to prevent CTRL+C in the first place and just
> > live
> > with the consequence. Then document why we disabled it in the FAQ.
> > 
> > On Wed, 2013-02-13 at 09:05 -0500, John Mazzitelli wrote:
> > > Note also, there is another solution we can document in the FAQ.
> > > The user can write their own "launch-as7-in-background.sh" and
> > > point to that script in the AS7's plugin configuration's "start
> > > script" property (I forget the actual name of that property, but
> > > the AS4 and AS5 server resources had something similar, so I
> > > assume the AS7 plugin provides that too). That script can do what
> > > we need and when the agent wants to start AS7, it will just launch
> > > that script, which does all the "magic" that it needs to isolate
> > > itself from the parent agent process.
> > > 
> > > 
> > > ----- Original Message -----
> > > > > mazz: Why does Java still have a daemon thread running that
> > > > > forked
> > > > > the AS7 process? Shouldn't that be a fire and forget?
> > > > > thomas: As far as I know, the child processes we create with
> > > > > Runtime.exec haveno link with internal JVM threads
> > > > 
> > > > you are both right, Runtime.exec() is a non-blocking call
> > > > creating a
> > > > new child process by default. In our case two deamon threads are
> > > > spawned for handling the streams, but it has no impact on the
> > > > original problem (calling interrupt() oh these two threads
> > > > doesn't
> > > > send the signal to the AS process) there is also waitFor() method
> > > > and for AS7 we wait for 15s but it is irrelevant.
> > > > 
> > > > > I still thought
> > > > > that if you kill a parent thread, any child threads are killed
> > > > > along
> > > > > with it - this is true regardless of whether the parent process
> > > > > is
> > > > > a
> > > > > Java JVM or not - this is just the way the UNIX operating
> > > > > system
> > > > > works.
> > > > 
> > > > If the child process was run in the background (&), and the
> > > > parent
> > > > process is stopped, the child process is still running and it
> > > > becomes the orphaned process with parent pid set to the init
> > > > process, that is what we want for AS.
> > > > 
> > > > > In our case, the rhq-agent.sh script is run in the foreground
> > > > > of an
> > > > > interactive shell. So when you hit Ctrl-C, it's the interactive
> > > > > shell
> > > > > that sends the SIGINT signal to all processes of the group the
> > > > > foreground process belongs to.
> > > > 
> > > > Right. If I run "./run-in-background.sh ./standalone.sh" from
> > > > interactive shell and press Ctrl+C, the AS won't get killed even
> > > > it
> > > > is in has the same PGID (because it was run as a background
> > > > process
> > > > using the & in the ./run-in-background.sh script), so it looks
> > > > like
> > > > the SIGINT is resent only to those child processes that was run
> > > > in
> > > > the foreground.
> > > > 
> > > > 
> > > > All in all, I am +1 on a FAQ note as Mazz suggested, because
> > > > adding
> > > > an '&' in between any two processes may be risky. There are 4
> > > > cases
> > > > where the "run-in-background" operation may be added (either
> > > > rhq-agent.sh and JVM (1), JVM and standalone.sh (2),
> > > > standalone.sh
> > > > and JVM (3), or JVM calling ./run-in-background.sh calling
> > > > standalone.sh in the background)
> > > > 
> > > > case (1) : impossible, we need to handle user input (typing
> > > > commands
> > > > to agent's console)
> > > > case (2) : it is possible, but I had to concatenate "& exit 0" in
> > > > the
> > > > Runtime.exec method, I guess we would lose the return code
> > > > case (3) : we can't modify standalone.sh, or more precisely, it
> > > > should work for the default impl. of standalone.sh
> > > > case (4) : it is possible, but we may again lose the return code,
> > > > I
> > > > haven't tried it though
> > > > 
> > > > Handling the signals manually (trap commands) is another way to
> > > > "separate" two processes, as Thomas suggested. This applies only
> > > > for
> > > > the scenario (1), because 'trap' is a Unix command. However, it
> > > > seems contradictory to me as I've already written.
> > > > 
> > > > jk
> > > > 
> > > > 
> > > > ----- Original Message -----
> > > > > From: "John Mazzitelli" <mazz at redhat.com>
> > > > > To: rhq-devel at lists.fedorahosted.org
> > > > > Sent: Tuesday, February 12, 2013 7:37:42 PM
> > > > > Subject: Re: AS is killed if the agent is killed (provided it
> > > > > was
> > > > > 	started/restarted by the agent)
> > > > > 
> > > > > Why does Java still have a daemon thread running that forked
> > > > > the
> > > > > AS7
> > > > > process? Shouldn't that be a fire and forget? I would think the
> > > > > child process would get forked/exec'd and then we forget about
> > > > > it,
> > > > > our thread ends. That seems wrong that we are allowing a daemon
> > > > > thread to remain around. Are we sure it needs to be like that?
> > > > > to
> > > > > me, that would be the thing to try to fix (though, I still
> > > > > thought
> > > > > that if you kill a parent thread, any child threads are killed
> > > > > along
> > > > > with it - this is true regardless of whether the parent process
> > > > > is
> > > > > a
> > > > > Java JVM or not - this is just the way the UNIX operating
> > > > > system
> > > > > works. I could be mistaken.
> > > > > 
> > > > > ----- Original Message -----
> > > > > > > BTW, the reason for Ctrl-C killing all processes is that
> > > > > > > the
> > > > > > > signal
> > > > > > > is sent to all processes in the same group.
> > > > > > 
> > > > > > We implement it manually in the
> > > > > > AgentShutdownHook.interruptAllThreads().
> > > > > > 
> > > > > > 
> > > > > > if there is a way to limit the signal to only getting sent to
> > > > > > the
> > > > > > > agent process (and not its child processes such that the
> > > > > > > child
> > > > > > > processes remain alive and running even after the parent
> > > > > > > process
> > > > > > > (the agent) dies), then that is something we should look
> > > > > > > into.
> > > > > > > I
> > > > > > > didn't think it was possible to do that, but if so, that
> > > > > > > would
> > > > > > > be
> > > > > > > great.
> > > > > > 
> > > > > > The problem here is that on the one hand we need to send
> > > > > > SIGINT
> > > > > > to
> > > > > > the agent to clear its resources, on the other hand we don't
> > > > > > want
> > > > > > to
> > > > > > send the SIGINT to the angent, because he will relay it to
> > > > > > the
> > > > > > all
> > > > > > daemon threads (including the running standalone.sh (run by
> > > > > > Java
> > > > > > runtime)). So it is obviously contradictory. Something like
> > > > > > "trap
> > > > > > 'kill -9 myPid' INT" in the rhq.agent.sh would work, but it
> > > > > > wouldn't
> > > > > > clear the resources.
> > > > > > 
> > > > > > The only way, I am aware of, is using the & operator:
> > > > > > 
> > > > > > "nohup ./rhq-agent.sh --daemon &"  works well
> > > > > > "./run-in-background.sh ./standalone.sh" works well
> > > > > > or "Runtime.exec("standalone.sh &")" works well
> > > > > > 
> > > > > > check the att., I did a sequence diagram for it, because it
> > > > > > is
> > > > > > quite
> > > > > > complex.
> > > > > > 
> > > > > > 
> > > > > > jk
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > ----- Original Message -----
> > > > > > > From: "John Mazzitelli" <mazz at redhat.com>
> > > > > > > To: rhq-devel at lists.fedorahosted.org
> > > > > > > Sent: Tuesday, February 12, 2013 5:45:53 PM
> > > > > > > Subject: Re: AS is killed if the agent is killed (provided
> > > > > > > it
> > > > > > > was
> > > > > > > 	started/restarted by the agent)
> > > > > > > 
> > > > > > > > BTW, the reason for Ctrl-C killing all processes is that
> > > > > > > > the
> > > > > > > > signal
> > > > > > > > is sent to all processes in the same group.
> > > > > > > 
> > > > > > > if there is a way to limit the signal to only getting sent
> > > > > > > to
> > > > > > > the
> > > > > > > agent process (and not its child processes such that the
> > > > > > > child
> > > > > > > processes remain alive and running even after the parent
> > > > > > > process
> > > > > > > (the agent) dies), then that is something we should look
> > > > > > > into.
> > > > > > > I
> > > > > > > didn't think it was possible to do that, but if so, that
> > > > > > > would
> > > > > > > be
> > > > > > > great.
> > > > > > > 
> > > > > > > But we should not eliminate the ability to control-C the
> > > > > > > agent
> > > > > > > itself.
> > > > > > > _______________________________________________
> > > > > > > rhq-devel mailing list
> > > > > > > rhq-devel at lists.fedorahosted.org
> > > > > > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > > > > > 
> > > > > > _______________________________________________
> > > > > > rhq-devel mailing list
> > > > > > rhq-devel at lists.fedorahosted.org
> > > > > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > > > > 
> > > > > _______________________________________________
> > > > > rhq-devel mailing list
> > > > > rhq-devel at lists.fedorahosted.org
> > > > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > > > 
> > > > _______________________________________________
> > > > rhq-devel mailing list
> > > > rhq-devel at lists.fedorahosted.org
> > > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > > > 
> > > _______________________________________________
> > > rhq-devel mailing list
> > > rhq-devel at lists.fedorahosted.org
> > > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > 
> > 
> > _______________________________________________
> > rhq-devel mailing list
> > rhq-devel at lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> > 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel




More information about the rhq-devel mailing list