Fwd: [jboss-as7-dev] IMPORTANT - New Policy Proposal - No More Intermittent Test Failures
by Charles Crouch
We have very similar issues...
Any suggestions?
----- Forwarded Message -----
From: "Jason T. Greene" <jason.greene(a)redhat.com>
To: jboss-as7-dev(a)lists.jboss.org
Sent: Monday, July 9, 2012 1:16:28 PM
Subject: [jboss-as7-dev] IMPORTANT - New Policy Proposal - No More Intermittent Test Failures
We always have the problem of having a set of tests which fail one out
of 10 runs, but we leave the test around hoping one day someone will fix
it. The problem is no one does, and it makes regression catching hard.
Right now people that submit pull requests have to scan through test
results and ask around to figure out if they broke something or not.
So I propose a new policy. Any test which intermittently fails will be
ignored and a JIRA opened to the author for up to a month. If that test
is not passing in one month time, it will be removed from the codebase.
The biggest problem with this policy is that we might completely lose
coverage. A number of the clustering tests for example fail
intermittently, and if we removed them we would have no other coverage.
So for special cases like clustering, I am thinking of relocating them
to a different test run called "broken-clustering", or something like
that. This run would only be monitored by those working on clustering,
and would not be included in the main "all tests" run.
Any other ideas?
--
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
11 years, 9 months
Questions around traits/availability/operations
by Charles Crouch
Say I have a resource with a Enable/Disable operation that also has a Status trait, e.g. a Deployment/application service under AS7.
Here are some scenarios that I've tried out (apologies if I've mis-remembered something here, I've not run the through the whole battery of scenarios after my initial tests, but I think the following are accurate)
1a) Shutting down the deployment (trait checking faster than availability)
---------------------------------------
I leave the deployment with its default availability collection period of 10mins but change its Status trait to be collected every 30seconds.
Deployment starts off green and status = OK
Execute the disable operation. It completes successfully very quickly.
Wait 1min
The status trait now says DOWN, but the availability still says green, but the app is actually not available.
Wait some more time, possibly upto 10mins.
The availability will now show red.
1b) Shutting down the deployment (trait checking slower than availability)
---------------------------------------
I leave the deployment with its default availability collection period of 10mins but change its Status trait to be collected every 20minutes
Deployment starts off green and status = OK
Execute the disable operation. It completes successfully very quickly.
Wait 1min
The status trait says OK and the availability still says green, but the app is actually not available.
Wait some more time, possibly upto 10mins.
The availability will now show red, but (assuming the trait collection still hasn't fired) the status trait still says OK.
The status trait will now remain OK forever or until we start collecting availability again.
2a) Starting up the deployment (trait checking faster than availability)
---------------------------------
I leave the deployment with its default availability collection period of 10mins but change its Status trait to be collected every 30seconds.
Deployment starts off red and status = DOWN
Execute the enable operation. It completes successfully very quickly.
Wait 1min
The status trait says DOWN and the availability still says red, but the app is actually available.
Wait some more time, possibly upto 10mins.
The availability will now show green. Shortly after that the status will go to OK
2b) Starting up the deployment (trait checking slower than availability)
---------------------------------------
I leave the deployment with its default availability collection period of 10mins but change its Status trait to be collected every 20minutes
Deployment starts off red and status = DOWN
Execute the enable operation. It completes successfully very quickly.
Wait 1min
The status trait says DOWN and the availability still says red, but the app is actually available.
Wait some more time, possibly upto 10mins.
The availability will now show green, but (assuming the trait collection still hasn't fired) the status trait still says DOWN.
Wait some more time, possibly upto another 10mins.
The status trait will eventually flip to OK
Summary and thoughts:
i) IMHO its not very intuitive that having various different schedules for availability and trait collection can result in radically different results. I think the general idea around metric/avail/traits should be that if the system ends up in a steady state that the metric/avail/trait should eventually get into that same steady state. The frequency of collection should just determine how quickly its get there.
So looking at the results above I see the following:
2b) is intuitive, slow trait schedule => get to steady state slowly
1a) is intuitive, fast trait schedule => get to steady state quickly
2a) is not intuitive, fast trait schedule doesn't update the trait quickly
1b) looks broken, trait may never get into correct state
I'm not sure what could be done about the last two. 1b) could possibly be mitigated by letting metrics and traits be collected one final time after the avail has gone red. 2a) could perhaps be addressed to a limited extent by updating what happens when an operation executes, as discussed below.
ii) For operations launched from the RHQ CLI/UI there should be a way (if there isnt already) to trigger an availability/trait/metric collection after the operation has completed. [If we already have this, then the AS7 plugin needs to be updated to support it]
That way users will see the very latest state as soon as possible after the operation is done. Its obviously fine for out of band events, e.g. someone manually deleting the application from the filesystem, to get picked up by the scheduled polling which the plugin does.
What does everyone think? BTW I'm not arguing about why we've ended up where we have, e.g. not collecting metrics/traits for down resources makes perfect sense from a performance perspective. I just want us to think about how we can improve going forward.
Cheers
Charles
11 years, 9 months
Pre-tested-commit
by Heiko W.Rupp
Hi,
I was yesterday at the Java Forum Stuttgart conference ( http://www.java-forum-stuttgart.de/ )
and attended a session about pre-tested-commits with git/gerrit/jenkins, which was quite interesting [1].
As preface: gerrit, which is a review system can act as a git server.
They showcased two setups:
1) user pushes a change. Gerrit can trigger a build from master + the changeset
from the user. If this is successful, a reviewer will be informed to review the changes.
Clicking "ok" submits the changes to the real repo
2) user pushes a change. Gerrit can trigger a build from master + the changeset
from the user. If this is successful, Jenkins tells Gerrit that the change has also
been reviewed so that Gerrit can then directly push to the backend repo.
In both cases if the initial build fails, no code is committed to the real repo; the
committer gets feedback and can fix his code, send an update via git amend
and the process can run again.
I think this is a pretty interesting idea especially if 1 and 2 could be combined
for inside committers on master we'd go for 1) and for community members or
pushes to a release branch we'd go for 2). If the Gerrit server would be internal,
we could set up an email alias where contributors can email their patch to.
This patch would then be picked up by Gerrit and 1) will run.
Of course for this to be more effective, we should really change how the
build works (e.g. only building enterprise if a push goes to core or enterprise
or building plugins only if a push goes to core or plugins etc. - more fine grained is better ),
as the current 1:30h build times are just too long.
We do already a little bit of 1) by the pre-compile job catching plain compile errors,
but effectively the bad code (in case of a compile error) would already be in the repo,
causing an issue for all others pulling from that version.
[1] Abstract with link to slides: http://www.java-forum-stuttgart.de/abstracts.html#C7
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschaeftsführer: Mark Hegarty, Charlie Peters, Michael Cunningham, Charles Cachera
11 years, 9 months
Extending availabilityContext to support isEnabled() ?
by Heiko W.Rupp
Hi,
I was looking at https://bugzilla.redhat.com/show_bug.cgi?id=797331#c4
and here it would be handy to have some AvailabilityContext.isEnabled()
method.
Looking at AvailContextImpl I see that the .enable() and .disable() methods
call the server, which is IMO not suited for .isEnabled(), as this may be called
much more often and would be too heavy weight.
I wonder if we should store the enablement in the AvailabilityContext of
each resource and update on .disable() and .enable() calls accordingly,
so that a call to .isEnabled() would be a simple cheap check
Heiko
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschaeftsführer: Mark Hegarty, Charlie Peters, Michael Cunningham, Charles Cachera
11 years, 10 months
arquillian framework addition for descriptor/component sharing
by John Mazzitelli
I added a little bit to our Arquillian framework to make it easier to re-use plugin descriptors.
As it stood before, you had to write your own discovery/resource components and your own XML descriptor for your tests. Even if everything else was the same, but your resource component class needed to do something different (to do things that you need to test if they are working) you had to write another XML descriptor because the <server> class attribute had to point to your own component class.
What I did was add a method to org.rhq.test.shrinkwrap.RhqAgentPluginDescriptorContainer called "setPluginDescriptorFromTemplate".
You pass in the name of the descriptor .xml file (as found in the classloader - this is just the test classloader that Maven gives you - if you put your .xml file in the src/test/resources folder, it can be found - just pass in the .xml filename) and you also pass in a Map<String,String> whose keys are token names and values are replacement strings. This way, you can have one xml descriptor "template" and you just pass in replacement strings that replace the tokens with your tests own custom values.
For example, we now have a single .xml descriptor template called "single-metric-rhq-plugin.xml" - it is simple. It defines one <server> with one <metric>. If you have any tests that need to test metric collections (and you only need a single resource with a single metric to do your test) you can reuse that template. A portion of that .xml template looks like this:
<server name="SingleMetricServer"
discovery="@@@discovery@@@"
class="@@@class@@@">
<metric property="metric1"
dataType="measurement"
defaultOn="true"
displayType="summary"
defaultInterval="30000" />
</server>
See the @@@discovery@@@ and @@@class@@@? Those need to be replaced by your test. You do that by passing in a Map with two keys - one key of "@@@discovery@@@" and one key with "@@@class@@@" - the values being your component class names.
I also included a simple, reusable discovery component that will discover a single resource. Many tests just want to get a single resource into inventory that we can test with - this SingleResourceDiscoveryComponent does that (the full class name is org.rhq.plugins.test.SingleResourceDiscoveryComponent).
So, a test that wants to get into inventory a single resource that has a single metric can do this in its getTestPlugin method:
@Deployment(name = "SingleMetricPlugin")
@TargetsContainer("connected-pc-with-metric-collection")
public static RhqAgentPluginArchive getTestPlugin() {
RhqAgentPluginArchive pluginJar = ShrinkWrap.create(RhqAgentPluginArchive.class, "single-metric-plugin-1.0.jar");
HashMap<String, String> replacements = new HashMap<String, String>();
replacements.put("@@@discovery@@@", SingleResourceDiscoveryComponent.class.getName());
replacements.put("@@@class@@@", BZ834019ResourceComponent.class.getName());
return pluginJar.setPluginDescriptorFromTemplate("single-metric-rhq-plugin.xml", replacements).addClasses(
SingleResourceDiscoveryComponent.class, BZ834019ResourceComponent.class);
}
Notice I put @@@discovery@@@ in the map, with the value as the full classname of that shared SingleResourceDiscoveryComponent. But I put in my test's own custom resource component in the @@@class@@@ entry.
We can use this setPluginDescriptorFromTemplate for any template you want - just create a .xml file that you think could be reusable, templatize it with strings like @@@variable@@@ and use code such as above to create test plugins using your template.
11 years, 10 months
GWT @ IO
by Jiri Kremser
Hi, I've got bad news. It looks like the only presentation at Google IO about GWT is called "Migrating Code from GWT to Dart". What I am trying to say is that it looks like there will be less and less support in the long term, depending on how the Dart will be successful. Unfortunately, Martin Fowler's http://www.thoughtworks.com/radar has basically the same opinion. There is still vibrant community and a lot of projects out there built with GWT, but to me it looks like the beginning of the very slow end.
JK
11 years, 10 months