Over the past few days I have been working with Ian to do some
performance analysis and testing of the AS 5 plugin to resolve high
memory usage. For a detailed summary of the issue please see
https://bugzilla.redhat.com/show_bug.cgi?id=788638. In short, the
problem stemmed from the plugin caching managed objects coming back from
the profile service that we discovered can be very large. The class
ManagedComponentComponent, which is the base class most of the service
types in the AS 5 plugin, was caching the ManagedComponent instance it
received from the profile service. This caching was introduced to
resolve high CPU utilization issues that were coming availability checks
in ManagedComponentComponent.getAvailability.
The solution involved two changes. First, don't cache the
ManagedComponent objects coming back from the profile service. Secondly,
to avoid the high CPU utilization issues, we will cache the managed
object's RunState which is just a few bytes in size. When do we cache
it? We refresh the RunState any time we fetch the managed object from
the JBoss service. This happens quite frequently, like for collecting
metrics. There is a configurable interval that is reset every time the
RunState info is refreshed. If that interval is exceeded during an
availability check, we will go ahead and reload the managed object at
that point. This two prong solution effectively addresses both the high
memory usage and high CPU utilization issues.
Attached to this email, you will find two zip files -
with_caching_profile.zip and without_caching_profile.zip. These zip
files contain some reports from JProfiler that focus on CPU utilization
and method execution times. with_caching_profile.zip contains profiling
reports when running the plugin as it exists in master. The
without_caching_profile.zip file contains profiling reports when running
the plugin with the changes described above. The focus of the profiling
was to ensure that the changes did not reintroduce the high CPU
utilization issues.
With out the caching, the agent was managed 4 EAP 5.1 servers and a
total of 1200 v3 EJBs. The agent had max heap of 512 MB and min heap of
64 MB. It is worth noting that a smaller max heap of 384 MB or even 256
MB was suitable when not running the profiler agent. When trying to
manage this inventory with the version of the plugin that uses caching
the agent fell flat on its face. I gave up trying to generate profiling
data when I reached a min heap of 4 GB and max heap of 6GB and was still
getting out of memory errors; consequently, the inventory used for the
reports in with_caching_profile.zip consisted of 2 EAP servers and 600
EJBs. I had a min heap of 1 GB and a max heap of 2 GB. Unfortunately I
still wound up with OOM errors but was able to get profiling data.
If you look at the CPU_Load_telemetry.html report, it shows overall CPU
utilization for the duration of the profiling session. For the caching
version it levels out at around 25%. For the non-caching version, it
also levels off a little over 20%, but there are some big spikes in
there. Those spikes are due to discovery and availability scans. The
reason you don't see the spikes in the caching version is because of the
OOM errors. You can extrapolate this by looking at
Method_Statistics.html. For the caching version, search for
AvailabilityExecutor.run(). It has an invocation count of 7 where as the
non-caching version has an invocation count of 16. The numbers for
AvailabilityExecutor.call() are 8 and 24 respectively. You will find
similar numbers for discovery scans. For the non-caching version of
ManagedComponentComponent.getAvailability the median execution time was
1 µs where as for the caching version it was 11 µs.
The execution time for getAvailability is about the same in both
versions. The CPU utilization in the non-caching version is a little
over 20% (excluding spikes from availability and discovery scans) as it
is in the caching version. And of course the non-caching version allows
the agent to scale to handle a much larger inventory.
I wanted to share this in detail because the strategy for availability
checks could likely be employed in many other resource components and
plugins. For heavyweight components, like the EAP resources, the gains
are substantial. Lastly, I would be remiss not to mention that Jay and
Ian in large part came up with the solution.
- John