On 12/2/10 11:41 AM, Joseph Marques wrote:


commit 988bafd8d08916e30c93b4fa79d6e24a588a0401
Author: John Sanda <jsanda@redhat.com>
Date: Tue Nov 9 16:33:44 2010 -0500

Fixing tests that were failing due to database clean up running too soon

MetadataManager previously defined @BeforeClass and @AfterClass methods for
setting up and tearing down each plugin meta data test class. The database was
reset in the @BeforeClass method. Turns out that while the first few test
methods of say OperationMetadataManagerBeanTest run, then before the rest of its
test methods have executed the @BeforeClass method is executed for say
ContentMetadataManagerBeanTest which wind up clearing out the same test data
that OperationMetadataManagerBeanTest was using. To remedy the situation, I
have made every configuration/test method in each XXXMetadataManagerBeanTest
part of the plugin.metadata group. Then I changed the @BeforeClass and
@AfterClass methods into @BeforeGroup and @AfterGroup methods respectively,
esnuring that they only run once for the entire group.

* so by virtue of using the strategy of completely wiping out the db, does this prevent
  us from leverage testng to run our tests in parallel now?  how much slower does the
  test suite execute in this branch versus master?

Initially I was not using dbunit or anything else to reset tables that are effected during a test run. I was using the pattern used in other tests which is to rollback the commit and the end of the test run; however, I needed to commit transactions in order to verify the state of things. We have this in other tests as well in other parts of the code base and the catch is, you might get unexpected results if you run those multiple times without restoring the database. So after a test run and prior to the next one I was restoring the database by running dbsetup. And by test run, I don't mean hudson. I am talking about running test as I was writing the code which would be rather frequent. Having to run dbsetup as I was doing was flat out painful and really slowing me down. Not only was it slow, there were plenty of times that I simply forgot to run dbsetup, and tests would subsequently fail as a result.

While I do clear out a substantial number of tables, I do not completely wipe out the database. I was not aware that we are running tests in parallel; however, tests in the delete-agent-plugin branch are running in the same way as they are in master. TestNG is pretty flexible in how it allows you to specify what you want run in parallel. You can specify entire suites be run in parallel, as well as methods and classes. Granted if we start running tests in parallel, some of the set up logic I put in place for these tests might have to be revisited, but I think the same can be said for tests in other parts of the code base as well.

As for how overall execution time, I cannot say. I have not paid too close attention. I do know though that due to the reasons I described above running tests on my box while I am developing is faster and more efficient. I expect the overall execution time to be at least somewhat slower in the delete-agent-plugin branch than in master if for no other reason than the fact that I added a substantial number of tests in the branch. You can look at the coverage report numbers to get a better sense of this.