Should the fix to BZ 794489 be considered a change to plugin API?
by Lukas Krejci
Hi all,
in fix for https://bugzilla.redhat.com/show_bug.cgi?id=794489 I needed to change the ResourceContext class, which is part of our plugin API. The API changes were following:
1) add (yet another) parameter to the constructor of the class
2) add a protected method (this is not necessarily required but good from the code reuse POV).
(the diff is here: http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blobdiff;f=modules/core/...)
I don't think this should be considered a change in our public plugin API, because ResourceContext is not meant to be constructed by plugin code (an instance of ResourceContext is merely provided to a resource component during the ResourceComponent.start() method). The protected method is a) not publicly accessible and b) doesn't clash with any pre-existing method on that class.
Even though I'm reasonably confident about that opinion, I'd like to get an agreement on it from other team members.
Thanks,
Lukas
12 years, 2 months
ability to view details of other users
by Ian Springer
I'm working on https://bugzilla.redhat.com/show_bug.cgi?id=786159.
Currently, a regular user can view the following details about any other
user (via #Administration/Security/Users):
username
first name
last name
department
email
phone
login enabled?
ldap user?
The assigned roles of other users cannot be viewed.
I propose we change it so they can only view the following fields:
username
first name
last name
department
I think it makes sense to keep these basic details public, because it's
useful to know the real name and department for other users in the
system. For example, if I see that user pilhun deleted my WAR a few days
earlier, it would be nice if I could find out that pilhun is Heiko Rupp
from the JON dev team (facts not obvious from the username), without
having to go find an RHQ Admin and ask them. I can see hiding people's
phone number and email address for privacy purposes. I'm up in the air
about the two boolean fields.
Users with MANAGE_SECURITY would of course be able to view and edit all
details for all users, as they can today.
Please let me know if you have an opinion on this, as I'd like to come
up with a plan so I can implement it next week.
Thanks,
Ian
12 years, 2 months
Availability enhancements
by Jay Shaughnessy
Last September the RHQ team had a face to face meeting and one of our
topics of discussion was availability collection and reporting. We did
some brainstorming generated an informal list of weaknesses and possible
enhancements. Recently we've had time to revisit the list, flesh out
some ideas, and implement some changes in a feature branch (jshaughn/avail).
I've put together a wiki page that summarizes the original list and also
the resulting enhancements. Please feel free to review and comment.
http://rhq-project.org/display/RHQ/Design-Availability+Checking
Initial implementation of the changes is complete, but the branch is
still in active development for testing, revision and possibly more
impl. It is not yet scheduled to be merged into master and will not be
in RHQ 4.3.
12 years, 2 months
GWT changes in master (browser changes as well)
by mike thompson
Now that master contains the smartGWT 3.0 and GWT 2.4.0 versions additional browser support has been added for IE9 and IE6 has been deprecated.
Just a note to change your settings.xml (or elsewhere to reflect the GWT browser compiler changes in the build process)
-- Mike Thompson
12 years, 2 months
tips and tools for load/performance testing jboss-as* plugins
by Ian Springer
In order to test the recent fixes to the jboss-as-5 plugin, which John
Sanda summarized in another thread, it was necessary to start up
multiple EAP instances each having hundreds of EJB's, of all four EJB
types, deployed. To get the EAP instances running on my Fedora dev box,
there were some modifications I needed to make to OS config files, as
well as to my RHQ Agent and Server env settings. These are summarized here:
http://rhq-project.org/display/RHQ/Starting+Multiple+JBoss+AS+Instances
That page also contains a link to a script I wrote for starting up n AS
or EAP instances.
To deploy lots of EJB's, I wrote a tool called jeeGen (Java EE
Generator) that can generate a v2.0, v2.1, v3.0, or v3.1 EJB-JAR
containing a specified number of each of the four types of EJB's. To use
it:
git checkout master
cd modules/helpers/jeeGen
mvn install
java -jar target/jeeGen-4.3.0-SNAPSHOT.jar 3.0 100 100 100 100
This produces a v3.0 EJB-JAR named test-ejb.jar in the cwd. The jar will
contain 100 entity beans, 100 SLSB's, 100 SFSB's, and 100 MDB's.
In the future, support for generating WAR's, EAR's, JBoss datasource
deployment descriptors, etc., could be added to JeeGen.
These tips and tools should be useful for future testing of the
jboss-as, jboss-as-5, and jboss-as-7 plugins.
12 years, 2 months
how to deal with globally uncaught exceptions caused by user navigating to view Y before view X has finished rendering
by Ian Springer
I just checked in a fix to master for
https://bugzilla.redhat.com/show_bug.cgi?id=754521. It was a class of
issue we've seen before, where the user goes to view X and then quickly
goes to view Y, before X has completed rendering. Typically, this
results in NPEs, because the code we added to support Selenium locators
destroys view widgets when it thinks we're done with them. My fix for
754521 was as follows:
1) create a new ViewChangedException runtime exception
2) in places where we detect that the user has switched to a different
view than the one we are still rendering, throw a ViewChangedException,
rather than continuing on rendering. Here's an example from
AbstractTwoLevelTabSetView.updateTabContent():
String currentViewPath = History.getToken();
if (!currentViewPath.startsWith(this.baseViewPath)) {
throw new ViewChangedException(this.baseViewPath + "/" +
getSelectedItemId());
}
Places that are good candidates for such code are in onSuccess() methods
of async callbacks, because while we were waiting for the async call to
return, the user could have navigated away to another view.
3) add code to the uncaught exception handler in CoreGUI that gracefully
handles ViewChangedExceptions by simply logging a DEBUG message to the
gwt log:
GWT.setUncaughtExceptionHandler(new
GWT.UncaughtExceptionHandler() {
public void onUncaughtException(Throwable e) {
if (e instanceof ViewChangedException) {
ViewChangedException viewChangedException =
(ViewChangedException) e;
String obsoleteView =
viewChangedException.getObsoleteView();
Log.debug("User navigated to view [" + currentView
+ "] before view [" + obsoleteView
+ "] was done rendering - rendering of view ["
+ obsoleteView + "] has been aborted.");
} else {
getErrorHandler().handleError(MSG.view_core_uncaught(), e);
}
}
});
If you encounter a similar bug in the future, you should be able to
leverage this new mechanism. Essentially, you would just need to find
the right place in the code to do step 2) from above.
12 years, 2 months
performance analysis for jboss-as-5 plugin
by John Sanda
Over the past few days I have been working with Ian to do some
performance analysis and testing of the AS 5 plugin to resolve high
memory usage. For a detailed summary of the issue please see
https://bugzilla.redhat.com/show_bug.cgi?id=788638. In short, the
problem stemmed from the plugin caching managed objects coming back from
the profile service that we discovered can be very large. The class
ManagedComponentComponent, which is the base class most of the service
types in the AS 5 plugin, was caching the ManagedComponent instance it
received from the profile service. This caching was introduced to
resolve high CPU utilization issues that were coming availability checks
in ManagedComponentComponent.getAvailability.
The solution involved two changes. First, don't cache the
ManagedComponent objects coming back from the profile service. Secondly,
to avoid the high CPU utilization issues, we will cache the managed
object's RunState which is just a few bytes in size. When do we cache
it? We refresh the RunState any time we fetch the managed object from
the JBoss service. This happens quite frequently, like for collecting
metrics. There is a configurable interval that is reset every time the
RunState info is refreshed. If that interval is exceeded during an
availability check, we will go ahead and reload the managed object at
that point. This two prong solution effectively addresses both the high
memory usage and high CPU utilization issues.
Attached to this email, you will find two zip files -
with_caching_profile.zip and without_caching_profile.zip. These zip
files contain some reports from JProfiler that focus on CPU utilization
and method execution times. with_caching_profile.zip contains profiling
reports when running the plugin as it exists in master. The
without_caching_profile.zip file contains profiling reports when running
the plugin with the changes described above. The focus of the profiling
was to ensure that the changes did not reintroduce the high CPU
utilization issues.
With out the caching, the agent was managed 4 EAP 5.1 servers and a
total of 1200 v3 EJBs. The agent had max heap of 512 MB and min heap of
64 MB. It is worth noting that a smaller max heap of 384 MB or even 256
MB was suitable when not running the profiler agent. When trying to
manage this inventory with the version of the plugin that uses caching
the agent fell flat on its face. I gave up trying to generate profiling
data when I reached a min heap of 4 GB and max heap of 6GB and was still
getting out of memory errors; consequently, the inventory used for the
reports in with_caching_profile.zip consisted of 2 EAP servers and 600
EJBs. I had a min heap of 1 GB and a max heap of 2 GB. Unfortunately I
still wound up with OOM errors but was able to get profiling data.
If you look at the CPU_Load_telemetry.html report, it shows overall CPU
utilization for the duration of the profiling session. For the caching
version it levels out at around 25%. For the non-caching version, it
also levels off a little over 20%, but there are some big spikes in
there. Those spikes are due to discovery and availability scans. The
reason you don't see the spikes in the caching version is because of the
OOM errors. You can extrapolate this by looking at
Method_Statistics.html. For the caching version, search for
AvailabilityExecutor.run(). It has an invocation count of 7 where as the
non-caching version has an invocation count of 16. The numbers for
AvailabilityExecutor.call() are 8 and 24 respectively. You will find
similar numbers for discovery scans. For the non-caching version of
ManagedComponentComponent.getAvailability the median execution time was
1 µs where as for the caching version it was 11 µs.
The execution time for getAvailability is about the same in both
versions. The CPU utilization in the non-caching version is a little
over 20% (excluding spikes from availability and discovery scans) as it
is in the caching version. And of course the non-caching version allows
the agent to scale to handle a much larger inventory.
I wanted to share this in detail because the strategy for availability
checks could likely be employed in many other resource components and
plugins. For heavyweight components, like the EAP resources, the gains
are substantial. Lastly, I would be remiss not to mention that Jay and
Ian in large part came up with the solution.
- John
12 years, 2 months
CPU usage spikes in RHQ agent
by Lukas Krejci
Hi all,
recently I've been thinking about how we could go about changing the way we
execute different tasks in the plugin container in agent so that we could
avoid or at least limit the CPU spikes we are experiencing with larger
inventories and/or heavy plugins (like AS5 plugin).
First, let's re-cap why are CPU spikes bad on the agent:
- They eat the cpu cycles that could be used by the managed resources
- the agent can actually render the machine unusable if the second spike
in usage kicks in before the first ends. If they are both scheduled (which
they usually are in agent, we never get out of high CPU usage).
The first, and I think the most important realization is that we do the
various scans wrongly from the CPU usage perspective. Currently, there is a
scheduled job for each, that will kick off one huge task that will recursively
traverse the inventory tree and will execute appropriate scan (avail, config,
discovery, ...) on one resource at a time.
Why is this wrong? Let's examine a (fabricated) CPU usage graph over time:
100% <- scan kicks in
100%
100%
0% <- scan finished
0%
0%
0%
100% <- another scan kicks in
100%
100%
You can notice 2 things there:
- we consume as much CPU as possible in as short time as possible to finish
the scan.
- after that there is a period of "silence"
We have this notion of availability, configuration, discovery, etc. scan
periods. These define the periods in which those huge tasks will be executed.
But from the user point of view, there isn't much difference between "all
resources were checked for avail @ 12.00am and then @ 12.05am and then
12.10am" and "resource A was checked @ 12.00am, resource B @ 12.01am,
resource C @ 12.04am, resource A @ 12.05am, resource B @ 12.06am, resource C @
12.09am". Notice that from the POV of the individual resources, the check
interval is 5 minutes in both cases which I think is the crucial part.
The latter approach allows us to "spread" the execution of the individual
resource checks in the defined period. This will immediately improve the CPU
usage situation because the spikes (or rather the amount of work) will be
spread throughout the whole period available, we won't be rushing to get
everything done as quickly as possible and "have a nap" afterwards.
This of course doesn't solve the problem of overlapping schedules - it only
makes the ramp up of the CPU usage slower in that case. It also doesn't solve
the situation where there is simply too much work to be done at any single
time, generating a constant high CPU load. But there are more tricks we can do
to further improve the situation.
To cap the CPU load, we can try to use the ThreadMXBean available in Java.
This can (optionally) provide the CPU time measurements to the java code.
While this might not be implemented by all JVMs, at least the Sun JVM and
OpenJDK on Linux implement it. Using that, we can insert "gaps" in between
individual resource checks during the scan to lower the overall CPU usage (of
course, at the expense of increased scan duration).
The final trick we can use is to provide the agent plugins with a way of
asking "Am I consuming too much CPU? Should I take a break for a while?". This
could be used during computationally intensive tasks inside the plugin code
(this is akin to the Thread.isInterrupted() method in Java - in general it is
not good practice to force some behavior on the thread of execution (i.e.
forcibly kill a thread) - threads should rather cooperate).
So with all these tricks up our the sleeve, what are the consequences for the
plugin container itself, the plugins and last but not least the agent and its
behavior and configurability as a whole?
First of all, we have to forget about guaranteeing any kind of regularity in
the scan periods. All the approaches above are approximate and are not able to
give any hard guarantees - be it for period duration or CPU usage. On the
other hand, we could not promise anything in our current approach either.
I think the key is to provide more visibility of performance problems through
metrics of the agent resource and give the user more control over the
performance tuning of the agent itself. If the user is able to set the target
CPU usage and periods for different scans, we can easily measure the "health"
of the system (and maybe even provide calltime metrics to make the performance
metrics specific to resource types - i.e. the user would be able to see that
it is indeed the AS5 plugin that eats up most of the discovery time and could
update the schedules accordingly). At this point I think I start to cross with
the recent discussions in the team about providing per-resource(-type)
schedules for avail and discovery and I admit that I don't have an opinion on
the compatibility of those ideas with this text.
On the other hand, I went ahead and actually implemented the above ideas in
code https://github.com/metlos/throttling-executors
The code is a bit rough, lacks some tests and has some scheduling issues in
high concurrency (I tried to implement the scheduling lock free, but was a bit
naive at that ). But it does show the point. There is a benchmark in the
tests that runs the different executors in different scenarios and tries to
establish a) the overhead the extra logic for task spreading and CPU
throttling poses over the "normal" ThreadPoolExecutor and b) the behavior of
the executors in different scenarios.
I would make this already too long email unbearably boring if I included the
discussion of the benchmark results here, so I'll leave that for a standalone
followup email.
So what are your thoughts about the above ideas? How does it blend with the
plans for the avail and discovery scheduling changes?
Cheers,
Lukas
12 years, 2 months
RHQ & mockito
by Stefan Negrea
Hello Everybody,
I've done some recent research into mocking frameworks for use with RHQ tests. Out of all the frameworks I stumbled upon, mockito looked the best (http://code.google.com/p/mockito/).
Here are some features that set mockito apart:
- fluid interfaces
- concise api
- very good documentation (eg: http://docs.mockito.googlecode.com/hg/latest/org/mockito/Mockito.html)
- vibrant & active community
- good maven integration, updates promptly published to maven repos
I know that jMock was used in some parts of RHQ however I find mockito superior especially around argument matchers, number of invocations, and call order. Please take a look at this blog post for a detailed comparison: http://www.zsoltfabok.com/blog/2010/08/jmock-versus-mockito/
I started this research in order to improve test coverage for the content system. My plan is to give mockito a try in that context and if successful apply it more broadly.
Any feedback or ideas are more than welcome.
Thank you,
Stefan Negrea
Software Engineer
12 years, 2 months