Hi all,
I'm writing to you about this bug: https://bugzilla.redhat.com/show_bug.cgi?id=887320
For those not aware of it, an OutOfMemoryError is raised on the agent when you try to add a child resource of type deployment to a managed AS7 instance.
It happens because the AS7 plugin uses HttpUrlConnection to communicate with AS7 HTTP management interface.
Basically, HttpUrlConnection waits for the request payload to be fully loaded in memory before it tries to contact the server. It does that to be able to resend the request payload if the server ask for redirection or start an authentication challenge.
While this is acceptable for small payloads, it's not suitable for large file uploads (in our case a 42M WAR upload triggers the OutOfMemoryError).
For large file uploads, HttpUrlConnection provides a streaming mode. But in streaming mode it does neither support redirection nor authentication. I tried to use streaming mode to make a fix yesterday and it's really not working (BTW, sorry for the patch revert waltz).
I can see two options:
#1 Keep HttpUrlConnection as communication layer In this case, the streaming mode is mandatory. This means we should maintain our own authentication challenge code and set authentication headers manually. #2 Use HttpClient as communication layer
Personally, I prefer #2
Importing HttpClient in AS7 plugin will increase the plugin dependencies list. But I think it's a small price to pay compared to the efforts we would spend maintaining our own authentication and redirection code.
What do you think?
Thanks Thomas
Hi all,
Following my email on BZ 887320 ( Agent raises OutOfMemoryError when adding new Deployment child resource on AS7 ).
A fix has been made and the AS7 plugin in master branch now uses commons httpclient to upload content.
All httpclient code is encapsulated in the ASUploadConnection so if you're a simple user of the class your code should not be broken (but please read the new documentation as some methods are now deprecated).
Some package protected fields which are no longer relevant have been removed but this should not impact your code (unless you extended the class in same package).
This change was very important as RHQ agents generally run with less resources than other Java programs (128M max heap) and we cannot rely on a tool which holds full EARs in memory.
If you have any concern with this changeset please email me.
Thanks and regards Thomas Segismont
Le 18/12/2012 15:13, Thomas Segismont a écrit :
Hi all,
I'm writing to you about this bug: https://bugzilla.redhat.com/show_bug.cgi?id=887320
For those not aware of it, an OutOfMemoryError is raised on the agent when you try to add a child resource of type deployment to a managed AS7 instance.
It happens because the AS7 plugin uses HttpUrlConnection to communicate with AS7 HTTP management interface.
Basically, HttpUrlConnection waits for the request payload to be fully loaded in memory before it tries to contact the server. It does that to be able to resend the request payload if the server ask for redirection or start an authentication challenge.
While this is acceptable for small payloads, it's not suitable for large file uploads (in our case a 42M WAR upload triggers the OutOfMemoryError).
For large file uploads, HttpUrlConnection provides a streaming mode. But in streaming mode it does neither support redirection nor authentication. I tried to use streaming mode to make a fix yesterday and it's really not working (BTW, sorry for the patch revert waltz).
I can see two options:
#1 Keep HttpUrlConnection as communication layer In this case, the streaming mode is mandatory. This means we should maintain our own authentication challenge code and set authentication headers manually. #2 Use HttpClient as communication layer
Personally, I prefer #2
Importing HttpClient in AS7 plugin will increase the plugin dependencies list. But I think it's a small price to pay compared to the efforts we would spend maintaining our own authentication and redirection code.
What do you think?
Thanks Thomas
Just to get meta on this ....
I am curious:
* How was this memory leak detected? was it just the error message? did the error appear instantly....or only after a period of time? * How do you know it is fixed? The error goes away? Or was there some heap analysis with a memory analysis tool? * Do you think this error ...and others like it ... can be detected with a static analysis tool such as Coverity?
Regards,
Michael Foley QE Lead, JBoss Operations Network ----- Original Message -----
From: "Thomas Segismont" tsegismo@redhat.com To: rhq-devel@lists.fedorahosted.org Sent: Monday, January 21, 2013 8:16:21 AM Subject: RHQ AS7 plugin with httpclient (important if you rely on this plugin for your development)
Hi all,
Following my email on BZ 887320 ( Agent raises OutOfMemoryError when adding new Deployment child resource on AS7 ).
A fix has been made and the AS7 plugin in master branch now uses commons httpclient to upload content.
All httpclient code is encapsulated in the ASUploadConnection so if you're a simple user of the class your code should not be broken (but please read the new documentation as some methods are now deprecated).
Some package protected fields which are no longer relevant have been removed but this should not impact your code (unless you extended the class in same package).
This change was very important as RHQ agents generally run with less resources than other Java programs (128M max heap) and we cannot rely on a tool which holds full EARs in memory.
If you have any concern with this changeset please email me.
Thanks and regards Thomas Segismont
Le 18/12/2012 15:13, Thomas Segismont a écrit :
Hi all,
I'm writing to you about this bug: https://bugzilla.redhat.com/show_bug.cgi?id=887320
For those not aware of it, an OutOfMemoryError is raised on the agent when you try to add a child resource of type deployment to a managed AS7 instance.
It happens because the AS7 plugin uses HttpUrlConnection to communicate with AS7 HTTP management interface.
Basically, HttpUrlConnection waits for the request payload to be fully loaded in memory before it tries to contact the server. It does that to be able to resend the request payload if the server ask for redirection or start an authentication challenge.
While this is acceptable for small payloads, it's not suitable for large file uploads (in our case a 42M WAR upload triggers the OutOfMemoryError).
For large file uploads, HttpUrlConnection provides a streaming mode. But in streaming mode it does neither support redirection nor authentication. I tried to use streaming mode to make a fix yesterday and it's really not working (BTW, sorry for the patch revert waltz).
I can see two options:
#1 Keep HttpUrlConnection as communication layer In this case, the streaming mode is mandatory. This means we should maintain our own authentication challenge code and set authentication headers manually. #2 Use HttpClient as communication layer
Personally, I prefer #2
Importing HttpClient in AS7 plugin will increase the plugin dependencies list. But I think it's a small price to pay compared to the efforts we would spend maintaining our own authentication and redirection code.
What do you think?
Thanks Thomas
_______________________________________________ rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
Hi Michael,
Please find my answers below.
Le 21/01/2013 14:21, Michael Foley a écrit :
Just to get meta on this ....
I am curious:
- How was this memory leak detected? was it just the error message? did the error appear instantly....or only after a period of time?
I found it when trying the child resource creation on AS 7 instance. The error message indicates OutOfMemoryError but it is shown in the server log even if the error actually occurred on the agent. It's always reproducible if you upload a non trivial WAR/EAR (say around 40MB).
- How do you know it is fixed? The error goes away? Or was there some heap analysis with a memory analysis tool?
I'm sure it's fixed because it's a reproducible bug, not an error which occurs randomly after some time or some running conditions.
- Do you think this error ...and others like it ... can be detected with a static analysis tool such as Coverity?
I don't know Coverity so I can't say for sure. But I doubt this one would have been detected. That doesn't mean a static analysis tool wouldn't be useful of course :)
Regards,
Michael Foley QE Lead, JBoss Operations Network
*From: *"Thomas Segismont" tsegismo@redhat.com *To: *rhq-devel@lists.fedorahosted.org *Sent: *Monday, January 21, 2013 8:16:21 AM *Subject: *RHQ AS7 plugin with httpclient (important if you rely on this plugin for your development)
Hi all,
Following my email on BZ 887320 ( Agent raises OutOfMemoryError when adding new Deployment child resource on AS7 ).
A fix has been made and the AS7 plugin in master branch now uses commons httpclient to upload content.
All httpclient code is encapsulated in the ASUploadConnection so if you're a simple user of the class your code should not be broken (but please read the new documentation as some methods are now deprecated).
Some package protected fields which are no longer relevant have been removed but this should not impact your code (unless you extended the class in same package).
This change was very important as RHQ agents generally run with less resources than other Java programs (128M max heap) and we cannot rely on a tool which holds full EARs in memory.
If you have any concern with this changeset please email me.
Thanks and regards Thomas Segismont
Le 18/12/2012 15:13, Thomas Segismont a écrit :
Hi all,
I'm writing to you about this bug: https://bugzilla.redhat.com/show_bug.cgi?id=887320
For those not aware of it, an OutOfMemoryError is raised on the agent when you try to add a child resource of type deployment to a managed AS7 instance.
It happens because the AS7 plugin uses HttpUrlConnection to communicate with AS7 HTTP management interface.
Basically, HttpUrlConnection waits for the request payload to be fully loaded in memory before it tries to contact the server. It does that to be able to resend the request payload if the server ask for redirection or start an authentication challenge.
While this is acceptable for small payloads, it's not suitable for large file uploads (in our case a 42M WAR upload triggers the
OutOfMemoryError).
For large file uploads, HttpUrlConnection provides a streaming mode. But in streaming mode it does neither support redirection nor authentication. I tried to use streaming mode to make a fix yesterday and it's really not working (BTW, sorry for the patch revert waltz).
I can see two options:
#1 Keep HttpUrlConnection as communication layer In this case, the streaming mode is mandatory. This means we should maintain our own authentication challenge code and set authentication headers manually. #2 Use HttpClient as communication layer
Personally, I prefer #2
Importing HttpClient in AS7 plugin will increase the plugin dependencies list. But I think it's a small price to pay compared to the efforts we would spend maintaining our own authentication and redirection code.
What do you think?
Thanks Thomas
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
rhq-devel mailing list rhq-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
Cheers Thomas
rhq-devel@lists.fedorahosted.org