On 12/2/2012 9:58 PM, John Sanda wrote:
On Friday I went to execute MeasurementDataManagerBeanTest, which
is now in the server/itests-2 module. I was unpleasantly surprised
by the execution time. Running this,
$ mvn -o clean test -Dtest=MeasurementDataManagerBeanTest
was taking upwards of 50 seconds. I am running on a fast
machine as well so I imagine the situation only gets more bleak
on slower hardware. There is nothing crazy going on in
MeasurementDataManagerBeanTest to cause such a long build time.
I need to write more tests, but I consider this is a blocker.
The problem has to do with Arquillian, or more specifically, how
we are using it. In a local branch I did some prototyping using
AuthorizationManagerBeanTest, for reasons I will explain later,
and was able to get the execution time down to about 10 seconds.
There are a few things I did to bring down the execution time.
1) Do not redeploy AS 7 every test run.
Even when not doing a clean, I was still seeing 40+ second
build times. Simply commenting out the antrun plugin in pom.xml
sped things up a little bit. We could put that plugin in a
profile so we are not always executing the Ant script(s) to
deploy our AS 7 server. I do something similar with Cassandra
tests.
Did you mean deploy or unzip? Anyway, I just checked something in
to avoid the unzip if the AS7 is already present in the target
directory. So, on subsequent runs you won't need to wait for the
unzip to plod through all of the files checking to see if it needs
to add anything.
As for deploying, we do only start the AS7 one time for the test
run. And we only deploy the test archive one time as well. So, you
pay the same overhead for a single test file as you do for many. In
the case of running many, locally or like in a CI run, the overhead
is actually pretty small. For the single iteration it is slow, but
then again this is an integration test and it requires a running
server.
2) Use remote container instead of managed container
Arquillian handles starting and stopping a managed container
whereas the lifecycle of a remote container is not handled by
Arquillian. I kept my AS 7 instance running. Maybe Jay and/or
Mazz have some insights into whether or not there might be
issues with using a remote container.
Certainly using a running remote server can speed things up since it
avoids that setup/teardown, although I suspect it can affect a lot
of the tests as written today, given that many freely perform
transaction management and interact with the entitymanager.
I have nothing against doing this when it make sense. Perhaps you
could give us some instructions if you have this working for certain
tests.
From the perspective of jenkins CI runs the managed container
certainly makes sense.
3) Introduce more modularity
I studied the timestamps in the log statements of tests runs
to get a better feel for where the most time is being spent.
While there is not one huge culprit, the lack of modularity is a
big problem, and it will only get worse. We have lots of EJBs,
and they are split across only 2 EJB JAR files. One contains
entity beans, and the other contains all of our SLSBs and MDBs.
The latter corresponds to the very large server/jar module. When
you go to run an Arquillian test, we deploy all of those EJBs
along with some other components. Given the sheer number of
them, it is going to take some time for the deployment to run.
In order to speed up deployments and execution times, we need
to limit what we deploy to test code, the production code under
test, and required runtime dependencies. This however is not
possible with the source code organized and packaged as it is.
In my branch I chose AuthorizationManagerBean because of its
minimal dependencies. It minimized the amount of refactoring I
had to do to get things running. I created a new module named
server-authorization that produces an EJB JAR artifact. My test
produces/deploys a JavaArchive much like the example at
http://arquillian.org/guides/getting_started_rinse_and_repeat/.
I have nothing against this, either, as people see fit to take on
this work perhaps they can make our overall environment better.
Modularizing our EJBs makes the build more complex, but it
brings a lot of benefits. Prior to the AS7 merge to master,
SLSBs along with tests lived in server/jar. I rarely would
execute all of the tests in server/jar because of how long they
take to run. Even running a single test class took a non-trivial
amount time due to the embedded container start
up/initialization time. Now consider the process with our
Arquillian set up. Suppose I make a change to
MeasurementDataManagerBean that I want to test. Here are the
steps,
- Make code changes to MeasurementDataManagerBean and
to MeasurementDataManagerBeanTest
- mvn install server/jar
- mvn install server/ear
- mvn test -Dtest=MeasurementDataManagerBeanTest (from
tests-2 module)
Certainly this is tedious during iterative SLSB changes. Perhaps we
could create an ear option that does not package the wars and sar,
just the test artifacts, as this takes some time and then gets
stripped away anyway.
As for the modularity, I'm a little dubious only in that we have so
many interactions at the SLSB layer. The number of different
dependencies is staggering. At least having the full ejb jars and
the ear libs covers all of the bases.
In addition to the already very long running time for the
test, I have to rebuild both server/jar and the EAR. With what I
have done, both MeasurementDataManagerBean
and MeasurementDataManagerBeanTest might live in a module named
server-metrics. The steps to test some changes would be as
follows,
- Make code changes to MeasurementDataManagerBean and
to MeasurementDataManagerBeanTest
- mvn test -Dtest=MeasurementDataManagerBeanTest
There is no need to rebuild any other modules and the test
presumably will execute a lot faster as we only deploy the
required artifacts instead of the whole EAR. There are more
benefits to increases modularity, but I will postpone those to a
later email since this has one has already gotten rather
length.
- John
_______________________________________________
rhq-devel mailing list
rhq-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel