classloader hell

mazz at redhat.com mazz at redhat.com
Thu Sep 30 04:48:33 UTC 2010


> Is this an instance of class leakage that was mentioned in a previous
> post?

Yes, it is. Welcome to the jungle :/

> In my reading of LinkageError it sounds like both the bootloader
> and the pluginclassloader have loaded JAXBContext and because of that
> the signatures don't match.

It sounds like you have a bead on what the problem is. I'll explain what I think happened for the benefit of others. Based on the unusually specific and helpful exception message:

1) the plugin classloader loaded "com.sun.xml.internal.ws.client.WSServiceDelegate" (which is what it sounds like you fixed from before - this class is probably now in some library you packaged in your plugin)
2) from looking at what I think is the source for that class, it extends com.sun.xml.ws.api.WSService which in turn extends javax.xml.ws.spi.ServiceDelegate.
3) One or both of those superclasses was loaded by the bootclassloader (the classloader at the very very top of the classloader hierachy) - this is the "leakage" I was saying was a possibility.
4) Both of the superclasses import and use javax.xml.bind.JAXBContext; thus, the bootloader also loaded this JAXBContext class (take special note - this is a JAXB API!)
5) WSServiceDelegate overrides that createDispatch method - but its loaded by the plugin classloader, which means that method needed to load JAXBContext and got the plugin classloader to load it too
6) boom - you have one method overriding another, but the signatures don't "match" because the two methods reference two different classes (the names might be the same, but they are really two different classes because they have two different loaders)

This is one way leaking classes can cause problems - this is what I was talking about when I mentioned to be on the lookout for this. Essentially, JAXBContext leaked from the bootstrap classloader and tried to mix itself with classes loaded by the plugin classloader.

> The other possibility is that the plugin
> was build using a newer version of jaxb than is present in the jdk.

I don't think that's it. The exception seems pretty clear that its this class problem.

As soon as I noticed the leaked class in question was a JAXB class, I asked myself, "How is your agent getting the JAXB classes into the bootstrap classloader I wonder?". I immediately went and looked at how we are deploying the JAXB libraries in the agent. We may have a problem here. But it may be easy to fix.

I'll claim ignorance to this Web Services API and all its dependencies - I've not used that stuff in years. But I believe this javax.xml.ws stuff (and its com.sun.xml.ws impl classes) are built into the JRE. However, JAXB was NOT shipping with the JRE when we were building/running on JRE 5. IT IS SHIPPING NOW WITH JDK6. I will therefore assume you are running the agent on JRE 6.

We actually put our own JAXB API in the agent/lib/endorsed directory because we needed JAXB and we needed to support JRE 5 (which didn't ship with JAXB). We install those JAXB libraries as endorsed libraries via -Djava.endorsed.dirs (see rhq-agent.sh for where we do that). In our root classloader regex, we hide those JAXB classes because we didn't want to clash with other plugins that also want to use their own external JAXB impl. This should allow other plugins to ship their own JAXB libraries and be able to use those (again, because back on JRE5 we didn't have the JRE to supply them to us).

But when you load the Web Services class, they come from the bootstrap classloader - and when their signatures reference that JAXBContext API, its ALSO comes out of the bootstrap classloader because that is now in the the built-in JRE libs! This is probably causing the problem. But we shouldn't be doing this "JAXB installed as an endorsed library" now anymore. We are moving to JRE 6, and since it has JAXB built-in we should not be shipping these JAXB libraries and installing them ourselves (which was only necessary for JRE5 runtimes)

How can you solve this? First, don't have the agent pass in that endorsed dir sysprop. you can do this by:

export RHQ_AGENT_JAVA_ENDORSED_DIRS=/bogus

rhq-agent.sh will use that instead of its own endorsed default and thus its JAXB libraries won't load - it'll fall back to the JRE's built-in JAXB API (and let's cross our fingers we stuff will still work, we were using JAXB 2.1, unsure how compatible that is with the built-in JRE version of JAXB).

Alternatively, if you don't want to set that env var, you can just delete the libraries that you have in agent/lib/endorsed (which I believe are all related to JAXB). Since the JRE 6 should already have them, you won't miss them.

Now we have to tell the agent to no longer hide those javax.xml.bind and related JAXB classes. You have to change the root classloader regex - you need to take the default regex and strip out the JAXB phrases. From the code, those JAXB related regex phrases are:

1200   // Hide our version of JAXB.
1201   // If the plugins want these, they should include their own implementations.
1202   defaultRegex.append("(javax\\.xml\\.bind\\..*)|");
1203   defaultRegex.append("(com\\.sun\\.activation\\..*)|");
1204   defaultRegex.append("(com\\.sun\\.istack\\..*)|");
1205   defaultRegex.append("(com\\.sun\\.xml\\..*)|");

For the record, those line numbers/code come from:

http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/enterprise/agent/src/main/java/org/rhq/enterprise/agent/AgentConfiguration.java;hb=master

That's the class that defines the default regex (see getPluginContainerConfiguration). you can either tweek that class and rebuild the agent jar or set the regex preference with the tweeked regex in agent-configuration.xml and restart the agent clean (--cleanconfig) to pick up the new regex.

Let us know if that works. If so, we can make those changes to the agent (i.e. no longer ship with the jaxb libs) and the changes to the default regex and commit those changes to the master branch.

----- "Bala Nair" <bnairtm at comcast.net> wrote:

> Ok, we are officially in classloader hell. Our original problem was a
> class not found exception when trying to use a class defined in an
> external jar in one of our agent plugins. After some good advice and
> an explanation of how resource classloaders work, we decided to do the
> following.
> 
> 1. declare an instance classloader for our plugin's server class (in
> the rhq-plugin.xml, server element, classloader="instance" attribute)
> 2. implement the classloaderfacet in the server discovery component
> 3. add the relevant jars to the classpath for the server resource
> component
> 
> So now we no longer see the class not found exception. instead we get
> this:
> 
> java.util.concurrent.ExecutionException: java.lang.Exception:
> java.lang.LinkageError: loader constraint violation: when resolving
> overridden method
> "com.sun.xml.internal.ws.client.WSServiceDelegate.createDispatch(Ljavax/xml/namespace/QName;Ljavax/xml/bind/JAXBContext;Ljavax/xml/ws/Service$Mode;)Ljavax/xml/ws/Dispatch;"
> the class loader (instance of
> org/rhq/core/pc/plugin/PluginClassLoader) of the current class,
> com/sun/xml/internal/ws/client/WSServiceDelegate, and its superclass
> loader (instance of <bootloader>), have different Class objects for
> the type javax/xml/bind/JAXBContext used in the signature
> 
> Is this an instance of class leakage that was mentioned in a previous
> post? In my reading of LinkageError it sounds like both the bootloader
> and the pluginclassloader have loaded JAXBContext and because of that
> the signatures don't match. The other possibility is that the plugin
> was build using a newer version of jaxb than is present in the jdk.
> 
> Question: are the steps we took above correct to implement the
> classloaderfacet.
> 
> Bala Nair
> 
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel


More information about the rhq-devel mailing list