classloader issues using jax-ws in a client plugin

John Mazzitelli mazz at redhat.com
Mon Sep 27 14:48:20 UTC 2010


[this is a very long email - my apologies up front :) but its going to 
document some very complex and technical details of the agent 
classloader stuff, so at least it can be useful for future reference for 
others]

> - the resource classloader for a managed resource is used by the
> resource component (correct?), but additional classpaths for it are
> defined in the resource discovery component when it implements
> ClassLoaderFacet - do I understand this correctly?

That is correct. Let me explain deeper, for those that are interested in 
hearing the details.

The main use-case for this resource classloader stuff is the following:

1) Assume you have a JBossAS 5 server and a JBossAS 6 server running on 
your one agent box (I'll call these J5 and J6). The agent will therefore 
have two resources in inventory - one representing the J5 server 
instance and one representing the J6 server instance.

2) J5 and J6 each have their own, incompatible client jars that are 
required to connect to them. That is to say, in order for the agent's J5 
resource component to remotely connect to J5's management connector, it 
needs J5's client-v5.jar - similarly, J6's client-v6.jar is needed to 
connect to J6 instances (that's not the real names of the jars, but you 
get the idea). You can't use client-v5.jar to connect to a J6 instance 
and vice versa.

3) You can't have client-v5.jar and client-v6.jar in the same 
classloader, for one, the names conflict and secondly, they are 
incompatible with each other.

4) Therefore, when the agent creates the resource component that 
represents the J5 instance, it creates one classloader (call it CL5) for 
that J5 resource component and it creates a separate classloader for the 
J6 resource component (call it CL6). This way, client_v5.jar can go in 
classloader CL5 and client_v6.jar can go in classloader CL6; thus they 
are separated from one another avoiding conflicts.

5) How does the agent know where client_v5.jar is? where client_v6.jar 
is? Those are in the install directories of the individual JBoss 
installs - the agent itself has no way of knowing where these are (only 
the product plugins know product specific details like this). The only 
way for the agent to know this is to ask the plugin's discovery 
components - since they know all about finding/discoverying information 
about their individual products they manage. Therefore, the jboss 
plugin's discovery component is asked by the agent "where can I find all 
the client jars that are needed to connect to this specific J5 
instance?" - the discovery component responds with a URL to 
client_v5.jar and it is placed in the classloader CL5 which in turn is 
used by the J5 resource commponent. The same is done when building the 
CL6 for the J6 resource component.

Welcome to classloader hell :)

> - The class not found exception that we're seeing is for a class defined
> in the java runtime itself.  How is this not available without adding
> rt.jar to the classpath?

OK, this is one of the "gotchas" in this classloader stuff that needs 
further explanation. First, let me explain the main use-case that needed 
to be supported. Recall that the agent core itself has jars that it 
needs for its own stuff. Just look in <agent-install>/lib and you'll see 
them all. For example, the agent core itself uses Apache Commons 
HttpClient (you'll find its jar in the lib/ directory of the agent). But 
what if a plugin wants to use that too - and more importantly what if 
the  plugin needs to use a later version of that library compared to the 
version of the library used by the agent? If we don't hide the agent's 
commons-httpclient jar from the plugin's classloader, the plugin will be 
forced to use the agent core's version of that library and would 
therefore break the plugin since the plugin needs a newer version of 
that library to work.

Did I welcome you to classloader hell yet? :-)

What to do? Well, we create what we call a "root plugin classloader" - 
call it RPCL. The agent makes this RPCL the parent classloader to ALL 
plugin classloaders. So, to follow the example from above, the parent 
classloader to CL5 is RPCL. The parent to CL6 is also RPCL. What's the 
parent classloader to RPCL itself? That's the "normal" 
application/bootstrap classloader.  So, the classloader hierarchy for a 
plugin classloader looks like:

Application/Bootstrap Classloader (java.*, <agent-install>/lib jars)
     ^
     |
Root Plugin Classloader
     ^
     |
Plugin Classloader (plugin jar classes)

What's so special about this RPCL? The RPCL "hides" some classes found 
it the classloaders above it - e.g. the agent/lib jars. Because the 
plugin classloaders have access to the main app classloader (since the 
main app classloader is at the very top of the classloader hierarchy, 
above the RPCL), plugins can still see all the core Java classes, 
however, the RPCL acts as a "filter" and hides some classes (like the 
http commons classes as I mentioned earlier) so the plugins can't see them.

What classes does the RPCL hide? This is actually configurable, via the 
agent preference "rhq.agent.plugins.root-plugin-classloader-regex". Its 
value is one big regular expression that, if a class name matches the 
expression, that class will not be loaded by the RPCL and essentially 
hidden. So when the RPCL is asked to load a class, if it matches the 
regex, that class is hidden and the plugin needs to load it itself from 
the lower plugin classloader (so we hide commons-httpclient which 
requires the plugin to have shipped its own version of that library for 
the class to load).

If you don't specify this preference, the agent defaults to a regex that 
you rarely, if ever, have to customize. By default, the agent hides the 
classes that should be hidden, yet still exposes the classes that 
plugins need. If interested in what the default hidden classes are, see :

org.rhq.enterprise.agent.AgentConfiguration.getPluginContainerConfiguration

to see the regex that it builds up for the default value of this preference.

Here are some of the classes that are hidden by default:

javax.xml.bind.**
com.sun.activation.**
com.sun.istack.**
com.sun.xml.**
org.apache.commons.httpclient.**
org.dom4j.**
EDU.oswego.**
gnu.getopt.**
javax.persistence.**
... and more...

Here you can see we hide some javax classes (JAXB and JPA classes 
specifically), we hide some com.sun classes (some of these due to the 
need to hide JAXB implementation classes) We hide JAXB because some 
plugins may want to use JAXB themselves and we didn't want to force them 
to use the version of JAXB that the agent uses (analogous to the 
commons-httpclient example I give above).

Your exception message says:

javax.xml.ws.WebServiceException: Provider 
com.sun.xml.internal.ws.spi.ProviderImpl not found

You'll see we hide all "com.sun.xml" classes. Thus, that provider is 
hidden and this exception message makes sense.

There are two possible solutions:

1) you'll have to customize the root plugin classloader's regex setting 
(in agent-configuration.xml or whereever) so you "unhide" those 
com.sun.xml classes you need. You'll want to "unhide" 
com.sun.xml.internal.ws.** and anything else you need. You'll have to be 
careful, if you "unhide" more classes than you need, you might screw up 
the classloading for other plugins that expect to get their own versions 
of classes that were hidden before.

2) package up the com.sun.xml classes you need in a jar and ship that 
jar inside your plugin's /lib directory. That may or may not work 
because it would mean you'd be loading 
com.sun.xml.internal.ws.spi.ProviderImpl via the plugin classloader but 
other classes may leak to/from the application classloader and, well, 
let's just say you will have entered the gates of classloader hell at 
that point.

#1 is probably the easiest thing to do unless there is a pre-packaged 
jar library that includes the com.sun.xml API that you need.

Can I ask you what version of Java you are using and what API you are 
trying to use? This isn't JAXB right? Its some kind of web services API?


More information about the rhq-devel mailing list