Back in January I spent several days going through our server jar tests
to try and get them to run more predictably, especially with newer
versions of testng. The major commits were:
24d89dc93ac47178812ce63c213d176fa265e915
68a688bc7bf11d2d6cfe22b08a1ffb1d282eb804
In the commit comments I had dumped what I learned during that
exercise. After some discussion on this earlier today I decided maybe
this information should have been more widely disseminated. Here's an
abridged version of the commit comments, hopefully it helps when writing
future tests, -Jay
---------------------------------
It seems we didn't really understand how testng was executing tests when
we submitted many test classes in a run. For example, when we submit
the server jar tests we assumed that the test classes would be run
sequentially, in perhaps some random order, and that each would have its
Before/AfterClass methods executed at the start and end of each test
class execution.
That is not true.
Although testng seems to run most tests from a test class together, it
does not seem to make that guarantee. From latest traces it seems that
the test classes are, to a degree, pooled and:
* you can not assume that all test methods for a test class will
execute before execution of test methods from another test class.
* you can not assume that multiple BeforeClass methods will not
executeprior to an AfterClass. For example, this can happen:
1. Class1:BeforeClass
2. Class2:BeforeClass
3. Class1:AfterClass
4. Class2:AfterClass
Put another way: You can only assume that BeforeClass and
AfterClassexecute in order, for a single test class. You can not
predict the execution order of all Before|AfterClass methods for all
test classes.
* The same rules as described in 2 apply to Before|AfterGroups
What this means is that you must be very careful with what is done in
Before|AfterClass and BeforeAfter|Groups. The work must not conflict
with work done in other test classes that will execute in the same test
suite.
This is very important when multiple test classes are sharing some sort
of test resource.
In general, it is much safer, although perhaps a bit slower, to use
@BeforeMethod() and @AfterMethod(alwaysRun=true).
This misunderstanding was particularly bad for us in the server jar
tests, where many test classes extend AbstractEJB3Test and utilize the
jboss mbean server under the covers, in the prepareXXX methods. This
caused clashes described above.
So, for server jar testing this reworks things to better share the mbean
server, even among tests in the same class. And migrates all
AbstractEJB3Test.prepare/unprepare calls from Before|AfterClass|Groups
to Before|After|Method.
... and from the second commit (you can look at the commit diffs to see
the referred to changes):
More work to prevent server jar tests from stepping on each other, or
themselves.
* Avoid the use of DatabaseOperation.CLEAN_INSERT. This deletes all
rows in tables used by the data set. This does not play well with
other modules. Instead use DatabaseOperation.REFRESH and also
clean up usingDatabaseOperation.DELETE. These options affect only
the rows in thedata set. (see DeleteAlertsTest)
* Unless absolutely necessary, use Before|AfterMethod for database
setup and teardown. The perf hit is typically minimal but the risk
oftest class conflict is high. (see DatabaseAndFilePluginDeloymentTest)
* Be more diligent to keep the database clean. For example, in
addition toDatabaseOperation.DELETE, there may be additional data
created by thetests. (see DiscoveryBossBeanTest)
* When possible, assertions should be narrowed to deal with only the
datarelevant to the test or test class. Don't assume the database
isnecessarily clean. This helps ensure that other test data does not
causefailures, or is not manipulated accidentally. Queries should
try to bequalified with the ids known to be relevant. (see
InventoryManagerBeanTest)
* Prefer to use slsb calls to do cleanup, these calls are often better
atensuring related data is also cleaned up. Hibernate cascading is
oftennot enough. And straight SQL requires even more diligence.
Cleaning upplugins is very important (see MetadataBeanTest).
* If a test does specific dbsetup make sure it also cleans it up, or
moveit to Before|AfterMethod. (see PluginManagerBeanTest)
* Make sure your tests can execute in any order, or add specific
dependenciesto force ordering. (see ResourceMetadataManagerBean)
Other tips:
* Use unusual/varying ids for canned data. Always starting ids at id
1 is more likely to cause conflicts.
* Always run your test class twice in a row. It should work more than
once. Ifit doesn't it probably changed state of the db or other
shared resource.
* Start your tests with a clean db (dbsetup) and manually inspect
thedb to see if any data was left behind (or better, write a tool to
do it).
Please add anything else you can to this thread and perhaps we can make
a wiki page for this stuff -Jay
Show replies by date