test execution times

Jay Shaughnessy jshaughn at redhat.com
Tue Dec 18 19:49:04 UTC 2012


On 12/2/2012 9:58 PM, John Sanda wrote:
> On Friday I went to execute MeasurementDataManagerBeanTest, which is 
> now in the server/itests-2 module. I was unpleasantly surprised by the 
> execution time.  Running this,
>
> $ mvn -o clean test -Dtest=MeasurementDataManagerBeanTest
>
> was taking upwards of 50 seconds. I am running on a fast machine as 
> well so I imagine the situation only gets more bleak on slower 
> hardware. There is nothing crazy going on in 
> MeasurementDataManagerBeanTest to cause such a long build time. I need 
> to write more tests, but I consider this is a blocker. The problem has 
> to do with Arquillian, or more specifically, how we are using it. In a 
> local branch I did some prototyping using 
> AuthorizationManagerBeanTest, for reasons I will explain later, and 
> was able to get the execution time down to about 10 seconds. There are 
> a few things I did to bring down the execution time.
>
> 1) Do not redeploy AS 7 every test run.
> Even when not doing a clean, I was still seeing 40+ second build 
> times. Simply commenting out the antrun plugin in pom.xml sped things 
> up a little bit. We could put that plugin in a profile so we are not 
> always executing the Ant script(s) to deploy our AS 7 server. I do 
> something similar with Cassandra tests.
Did you mean deploy or unzip?  Anyway, I just checked something in to 
avoid the unzip if the AS7 is already present in the target directory.  
So, on subsequent runs you won't need to wait for the unzip to plod 
through all of the files checking to see if it needs to add anything.

As for deploying, we do only start the AS7 one time for the test run. 
And we only deploy the test archive one time as well.  So, you pay the 
same overhead for a single test file as you do for many.  In the case of 
running many, locally or like in a CI run, the overhead is actually 
pretty small.  For the single iteration it is slow, but then again this 
is an integration test and it requires a running server.

>
> 2) Use remote container instead of managed container
> Arquillian handles starting and stopping a managed container whereas 
> the lifecycle of a remote container is not handled by Arquillian. I 
> kept my AS 7 instance running. Maybe Jay and/or Mazz have some 
> insights into whether or not there might be issues with using a remote 
> container.
Certainly using a running remote server can speed things up since it 
avoids that setup/teardown, although I suspect it can affect a lot of 
the tests as written today, given that many freely perform transaction 
management and interact with the entitymanager.

I have nothing against doing this when it make sense.  Perhaps you could 
give us some instructions if you have this working for certain tests.

 From the perspective of jenkins CI runs the managed container certainly 
makes sense.

>
> 3) Introduce more modularity
> I studied the timestamps in the log statements of tests runs to get a 
> better feel for where the most time is being spent. While there is not 
> one huge culprit, the lack of modularity is a big problem, and it will 
> only get worse. We have lots of EJBs, and they are split across only 2 
> EJB JAR files. One contains entity beans, and the other contains all 
> of our SLSBs and MDBs. The latter corresponds to the very large 
> server/jar module. When you go to run an Arquillian test, we deploy 
> all of those EJBs along with some other components. Given the sheer 
> number of them, it is going to take some time for the deployment to run.
>
> In order to speed up deployments and execution times, we need to limit 
> what we deploy to test code, the production code under test, and 
> required runtime dependencies. This however is not possible with the 
> source code organized and packaged as it is. In my branch I chose 
> AuthorizationManagerBean because of its minimal dependencies. It 
> minimized the amount of refactoring I had to do to get things running. 
> I created a new module named server-authorization that produces an EJB 
> JAR artifact. My test produces/deploys a JavaArchive much like the 
> example at 
> http://arquillian.org/guides/getting_started_rinse_and_repeat/.

I have nothing against this, either, as people see fit to take on this 
work perhaps they can make our overall environment better.

>
> Modularizing our EJBs makes the build more complex, but it brings a 
> lot of benefits. Prior to the AS7 merge to master, SLSBs along with 
> tests lived in server/jar. I rarely would execute all of the tests in 
> server/jar because of how long they take to run. Even running a single 
> test class took a non-trivial amount time due to the embedded 
> container start up/initialization time. Now consider the process with 
> our Arquillian set up. Suppose I make a change to 
> MeasurementDataManagerBean that I want to test. Here are the steps,
>
>  1. Make code changes to MeasurementDataManagerBean and
>     to MeasurementDataManagerBeanTest
>  2. mvn install server/jar
>  3. mvn install server/ear
>  4. mvn test -Dtest=MeasurementDataManagerBeanTest (from tests-2 module)
>
Certainly this is tedious during iterative SLSB changes.  Perhaps we 
could create an ear option that does not package the wars and sar, just 
the test artifacts, as this takes some time and then gets stripped away 
anyway.

As for the modularity, I'm a little dubious only in that we have so many 
interactions at the SLSB layer.  The number of different dependencies is 
staggering. At least having the full ejb jars and the ear libs covers 
all of the bases.

>
> In addition to the already very long running time for the test, I have 
> to rebuild both server/jar and the EAR. With what I have done, both 
> MeasurementDataManagerBean and MeasurementDataManagerBeanTest might 
> live in a module named server-metrics.  The steps to test some changes 
> would be as follows,
>
>  1. Make code changes to MeasurementDataManagerBean and
>     to MeasurementDataManagerBeanTest
>  2. mvn test -Dtest=MeasurementDataManagerBeanTest
>
>
> There is no need to rebuild any other modules and the test presumably 
> will execute a lot faster as we only deploy the required artifacts 
> instead of the whole EAR. There are more benefits to increases 
> modularity, but I will postpone those to a later email since this has 
> one has already gotten rather length.
>
> - John
>
>
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fedorahosted.org/pipermail/rhq-devel/attachments/20121218/1b6eda20/attachment-0001.html>


More information about the rhq-devel mailing list