I'll see if I can then do the fix and commit the changes to the master
branch so its in our next build.
My major concern now is what does this do to other plugins.
I don't think this will affect any other plugins. I can't think of any
plugins (that I or others on the RHQ team wrote) that need JAXB and
included the jaxb libs in their plugins. If they DID, we almost
certainly would have used the same JAXB version as that of the agent
(2.1) so functionality would not change or be broken, we'd just have to
remote the jaxb libs from the plugin (but again, I can't think of any
plugins that actually did this). If you want, just go look in all the
plugin jars in your agent's plugins/ directory and look in each of the
jar's /lib directories - if you see any with these jaxb jars, take those
jaxb jars out (and document what jars you found that did this in the
bugzilla so we remove it from the future builds).
In other words, I think you should OK.
Thanks for working on this.
John Mazz
On 10/01/2010 02:05 PM, Bala Nair wrote:
Success!
We removed the jaxb references from the root-plugin-classloader-regex in
agent-configuration.xml, removed the endorsed libs from the agent
startup and changed our instance classloader back to shared. That seems
to fix the issue - we can now connect to the managed resource via
jax-ws. My major concern now is what does this do to other plugins.
This resource is running on a test system so we can disable most of the
other standard plugins, but before we go with this for release we would
need to test it more thoroughly.
Bala Nair
On 9/30/10 11:07 AM, John Mazzitelli wrote:
> well, I found out why we put the JAXB libs in the agent endorsed
> directory (as opposed to just in the agent lib). This is from the ant
> script that builds the agent distro:
>
> <!-- The JAXB jars need to go in endorsed, since we use JAXB 2.1,
> which would cause conflicts with
> JAXB 2.0, which is bundled with Java 6, when running under
> Java 6. JAXB 2.1 depends on
> Activation 1.1, which we also need to include, in order to
> remain compatible with Java 5,
> which bundles Activation 1.0. -->
> <fileset dir="${settings.localRepository}"
> id="endorsed.classpath.fileset">
> <include
>
name="javax/xml/bind/jaxb-api/${jaxb-api.version}/jaxb-api-${jaxb-api.version}.jar"/>
> <include
>
name="com/sun/xml/bind/jaxb-impl/${jaxb-impl.version}/jaxb-impl-${jaxb-impl.version}.jar"/>
> <include
> name="javax/activation/activation/1.1/activation-1.1.jar"/>
> </fileset>
>
> So this problem is probably gonna be harder to fix. We need JAXB 2.1 and
> according to this comment, it says JRE 6 comes with JAXB 2.0.
>
> On 09/30/2010 09:37 AM, John Mazzitelli wrote:
>> Here's a bugzilla for this issue, in case anyone wants to try to come up
>> with a patch for it:
>>
>>
https://bugzilla.redhat.com/show_bug.cgi?id=638956
>>
>>>> Is this an instance of class leakage that was mentioned in a previous
>>>> post?
>>> Yes, it is. Welcome to the jungle :/
>>>
>>>> In my reading of LinkageError it sounds like both the bootloader
>>>> and the pluginclassloader have loaded JAXBContext and because of that
>>>> the signatures don't match.
>>> It sounds like you have a bead on what the problem is. I'll explain what
I think happened for the benefit of others. Based on the unusually specific and helpful
exception message:
>>>
>>> 1) the plugin classloader loaded
"com.sun.xml.internal.ws.client.WSServiceDelegate" (which is what it sounds like
you fixed from before - this class is probably now in some library you packaged in your
plugin)
>>> 2) from looking at what I think is the source for that class, it extends
com.sun.xml.ws.api.WSService which in turn extends javax.xml.ws.spi.ServiceDelegate.
>>> 3) One or both of those superclasses was loaded by the bootclassloader (the
classloader at the very very top of the classloader hierachy) - this is the
"leakage" I was saying was a possibility.
>>> 4) Both of the superclasses import and use javax.xml.bind.JAXBContext; thus,
the bootloader also loaded this JAXBContext class (take special note - this is a JAXB
API!)
>>> 5) WSServiceDelegate overrides that createDispatch method - but its loaded by
the plugin classloader, which means that method needed to load JAXBContext and got the
plugin classloader to load it too
>>> 6) boom - you have one method overriding another, but the signatures
don't "match" because the two methods reference two different classes (the
names might be the same, but they are really two different classes because they have two
different loaders)
>>>
>>> This is one way leaking classes can cause problems - this is what I was
talking about when I mentioned to be on the lookout for this. Essentially, JAXBContext
leaked from the bootstrap classloader and tried to mix itself with classes loaded by the
plugin classloader.
>>>
>>>> The other possibility is that the plugin
>>>> was build using a newer version of jaxb than is present in the jdk.
>>> I don't think that's it. The exception seems pretty clear that its
this class problem.
>>>
>>> As soon as I noticed the leaked class in question was a JAXB class, I asked
myself, "How is your agent getting the JAXB classes into the bootstrap classloader I
wonder?". I immediately went and looked at how we are deploying the JAXB libraries in
the agent. We may have a problem here. But it may be easy to fix.
>>>
>>> I'll claim ignorance to this Web Services API and all its dependencies -
I've not used that stuff in years. But I believe this javax.xml.ws stuff (and its
com.sun.xml.ws impl classes) are built into the JRE. However, JAXB was NOT shipping with
the JRE when we were building/running on JRE 5. IT IS SHIPPING NOW WITH JDK6. I will
therefore assume you are running the agent on JRE 6.
>>>
>>> We actually put our own JAXB API in the agent/lib/endorsed directory because
we needed JAXB and we needed to support JRE 5 (which didn't ship with JAXB). We
install those JAXB libraries as endorsed libraries via -Djava.endorsed.dirs (see
rhq-agent.sh for where we do that). In our root classloader regex, we hide those JAXB
classes because we didn't want to clash with other plugins that also want to use their
own external JAXB impl. This should allow other plugins to ship their own JAXB libraries
and be able to use those (again, because back on JRE5 we didn't have the JRE to supply
them to us).
>>>
>>> But when you load the Web Services class, they come from the bootstrap
classloader - and when their signatures reference that JAXBContext API, its ALSO comes out
of the bootstrap classloader because that is now in the the built-in JRE libs! This is
probably causing the problem. But we shouldn't be doing this "JAXB installed as
an endorsed library" now anymore. We are moving to JRE 6, and since it has JAXB
built-in we should not be shipping these JAXB libraries and installing them ourselves
(which was only necessary for JRE5 runtimes)
>>>
>>> How can you solve this? First, don't have the agent pass in that endorsed
dir sysprop. you can do this by:
>>>
>>> export RHQ_AGENT_JAVA_ENDORSED_DIRS=/bogus
>>>
>>> rhq-agent.sh will use that instead of its own endorsed default and thus its
JAXB libraries won't load - it'll fall back to the JRE's built-in JAXB API
(and let's cross our fingers we stuff will still work, we were using JAXB 2.1, unsure
how compatible that is with the built-in JRE version of JAXB).
>>>
>>> Alternatively, if you don't want to set that env var, you can just delete
the libraries that you have in agent/lib/endorsed (which I believe are all related to
JAXB). Since the JRE 6 should already have them, you won't miss them.
>>>
>>> Now we have to tell the agent to no longer hide those javax.xml.bind and
related JAXB classes. You have to change the root classloader regex - you need to take the
default regex and strip out the JAXB phrases. From the code, those JAXB related regex
phrases are:
>>>
>>> 1200 // Hide our version of JAXB.
>>> 1201 // If the plugins want these, they should include their own
implementations.
>>> 1202 defaultRegex.append("(javax\\.xml\\.bind\\..*)|");
>>> 1203 defaultRegex.append("(com\\.sun\\.activation\\..*)|");
>>> 1204 defaultRegex.append("(com\\.sun\\.istack\\..*)|");
>>> 1205 defaultRegex.append("(com\\.sun\\.xml\\..*)|");
>>>
>>> For the record, those line numbers/code come from:
>>>
>>>
http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/enterpris...
>>>
>>> That's the class that defines the default regex (see
getPluginContainerConfiguration). you can either tweek that class and rebuild the agent
jar or set the regex preference with the tweeked regex in agent-configuration.xml and
restart the agent clean (--cleanconfig) to pick up the new regex.
>>>
>>> Let us know if that works. If so, we can make those changes to the agent
(i.e. no longer ship with the jaxb libs) and the changes to the default regex and commit
those changes to the master branch.
>>>
>>> ----- "Bala Nair"<bnairtm(a)comcast.net> wrote:
>>>
>>>> Ok, we are officially in classloader hell. Our original problem was a
>>>> class not found exception when trying to use a class defined in an
>>>> external jar in one of our agent plugins. After some good advice and
>>>> an explanation of how resource classloaders work, we decided to do the
>>>> following.
>>>>
>>>> 1. declare an instance classloader for our plugin's server class (in
>>>> the rhq-plugin.xml, server element, classloader="instance"
attribute)
>>>> 2. implement the classloaderfacet in the server discovery component
>>>> 3. add the relevant jars to the classpath for the server resource
>>>> component
>>>>
>>>> So now we no longer see the class not found exception. instead we get
>>>> this:
>>>>
>>>> java.util.concurrent.ExecutionException: java.lang.Exception:
>>>> java.lang.LinkageError: loader constraint violation: when resolving
>>>> overridden method
>>>>
"com.sun.xml.internal.ws.client.WSServiceDelegate.createDispatch(Ljavax/xml/namespace/QName;Ljavax/xml/bind/JAXBContext;Ljavax/xml/ws/Service$Mode;)Ljavax/xml/ws/Dispatch;"
>>>> the class loader (instance of
>>>> org/rhq/core/pc/plugin/PluginClassLoader) of the current class,
>>>> com/sun/xml/internal/ws/client/WSServiceDelegate, and its superclass
>>>> loader (instance of<bootloader>), have different Class objects for
>>>> the type javax/xml/bind/JAXBContext used in the signature
>>>>
>>>> Is this an instance of class leakage that was mentioned in a previous
>>>> post? In my reading of LinkageError it sounds like both the bootloader
>>>> and the pluginclassloader have loaded JAXBContext and because of that
>>>> the signatures don't match. The other possibility is that the plugin
>>>> was build using a newer version of jaxb than is present in the jdk.
>>>>
>>>> Question: are the steps we took above correct to implement the
>>>> classloaderfacet.
>>>>
>>>> Bala Nair
>>>>
>>>>
>>>> _______________________________________________
>>>> rhq-devel mailing list
>>>> rhq-devel(a)lists.fedorahosted.org
>>>>
https://fedorahosted.org/mailman/listinfo/rhq-devel
>>> _______________________________________________
>>> rhq-devel mailing list
>>> rhq-devel(a)lists.fedorahosted.org
>>>
https://fedorahosted.org/mailman/listinfo/rhq-devel
>> _______________________________________________
>> rhq-devel mailing list
>> rhq-devel(a)lists.fedorahosted.org
>>
https://fedorahosted.org/mailman/listinfo/rhq-devel
> _______________________________________________
> rhq-devel mailing list
> rhq-devel(a)lists.fedorahosted.org
>
https://fedorahosted.org/mailman/listinfo/rhq-devel
>
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/rhq-devel