RfC: Test group separation / module reorganisation

John Sanda jsanda at redhat.com
Thu Oct 28 18:59:59 UTC 2010


heiko and I were chatting about this, and I remarked that I feel that 
our build is too monolithic. Let's say for example that I make a change 
in some plugin code. As far as our CI process goes, we will build the 
entire source tree. This results in delayed feedback from the CI process 
as it already takes a considerable amount of time to run a full build. 
The situation can actually be worse though. Suppose a build fails for 
some reason before actually building the plugin changes. Now I will have 
to wait for a subsequent build. I think it is worth exploring different 
ways to make the build more modular, which could in turn speed things up.

Let's say we have separate hudson builds for the server and for the 
plugins. Then I make some changes in server/jar. Since the plugins is a 
downstream dependency, they only need to get built if the changes I make 
in server/jar result in a successful server build. This approach would 
not eliminate building the full source tree, but it would provide 
quicker feedback as well as promote greater loose coupling through the 
increased modularity.

Ultimately yes we need to build the whole source tree, but at least 
during development (i.e., for "check-in" builds), we could be more 
efficient by building things more incrementally.

As for the perf tests, it really doesn't make sense to me for them to 
live in server/jar for a few reasons. First, prior to adding those perf 
tests, the tests in server/jar as well as in all our other modules are 
primarily unit/integration tests that verify intended functionality. 
Performance tests are different in nature. Secondly, the server/jar 
build is already painfully slow. Adding long, running performance tests 
only exacerbates the problem. Lastly, the perf test hudson job as it is 
set up needlessly builds (at least compilation) the whole source tree. 
It could instead be set up as a downstream build that just runs tests. 
The job as it is now takes a lot longer than it needs to do, and uses up 
resources on the build machine when it doesn't have to.

- John

On 10/28/10 2:17 PM, Heiko W.Rupp wrote:
> Hi,
>
> currently (nearly) all tests run in the integration.ejb3 group. No matter if those are server side plugins or core domain
> or real biz logic.
>
> Tests should imho be separated out in more distinct groups and only run if relevant
> e.g. only run UI tests when UI code or code the UI depends on is changed, but not
> when e.g. plugin code has changed.
>
> Another aspect to this is that our builds often build a lot of stuff even if no relevant code
> has been changed. THis is e.g. the case when a plugin is changed, the UI code needs
> no rebuild (and vice versa).
>
> It looks like the issue is not entirely trivial for the automatic testing side, but there is hope:
> - http://www.dzone.com/links/sibilla_first_release_experince_a_new_approach_to.html?ref=ps
>   that is by a colleague, Stefano Maestri and seems to analyze the relevant changed path
>   and only run relevant tests
> - JunitMax ($$-ware) for Eclipse sorts tests according to runtime and likelihood of failing, so that
>   the user gets fast feedback and sees failures in the usual error view
>
> Anyway : server side plugins should move out of enterprise/server to modules/server-plugins
> installer should move out of the enterprise/gui into modules/installer and I guess there are
> a few more candidates.
> John Sanda was also proposing to move the performance testing outside of enterprise/server
> into some (more or less) toplevel module.
>



More information about the rhq-devel mailing list