If anyone is interested in helping me analyze heap dumps to find where
we are leaking classloaders, let me know. I think I need another pair of
eyes. I can give folks my hprofs if you know how to analyze them and/or
we can review them together. This is the BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=615377
For sure we leaked EMS classloaders due to a SUN JVM bug - that much I
am certain. And I fixed it locally on my box. If you care, this is the
SUN bug -
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6727821 -
I'm still baffled why they are caching this context classloader in a
static data member, but they are and it causes problems and we need code
to workaround it. In addition, our JBossAS 5 plugin and EMS is storing a
JBoss-specific Configuration object inside this javax Configuration
object (see Configuration.setConfiguration()) and thus leaks that object
and its classloader.
Unfortunately, it doesn't solve the entire problem. Running in
JProfiler, I still see perm gen leaking. Doing a "plugins update"
numerous times will eventually result in an OOM-PermGen. This only
happens if you are managing JBossAS instances.
So what I did was I captured 5 hprof heap dumps:
1) the first at agent startup with --nostart --nonative --console=java
option. This is the agent core started, but the PC is not started and
there is no native code enabled. Memory footprint is very small, and
only our input thread is running.
2) the second hprof is taken after I did a "start" - the PC is started
and it has JBossAS instance in inventory.
3) the third is taken after a "shutdown" - all threads are dead but the
input thread. At this point, this hprof should be able to see leaked
classloaders.
4) I started the PC again via "start" and took a fourth hprof
5) Finally, I "shutdown" the agent internals a second time and took
another hprof.
At this point, if we compare hprof #1 vs. #5, they should be very
similar but due to the leakage, we should be able to compare the two and
find the leakages. This is the part I'm having problems with. This is
how I found the javax Configuration leak, but I haven't found anything
else.