RfC: Test group separation / module reorganisation
John Sanda
jsanda at redhat.com
Thu Oct 28 18:59:59 UTC 2010
heiko and I were chatting about this, and I remarked that I feel that
our build is too monolithic. Let's say for example that I make a change
in some plugin code. As far as our CI process goes, we will build the
entire source tree. This results in delayed feedback from the CI process
as it already takes a considerable amount of time to run a full build.
The situation can actually be worse though. Suppose a build fails for
some reason before actually building the plugin changes. Now I will have
to wait for a subsequent build. I think it is worth exploring different
ways to make the build more modular, which could in turn speed things up.
Let's say we have separate hudson builds for the server and for the
plugins. Then I make some changes in server/jar. Since the plugins is a
downstream dependency, they only need to get built if the changes I make
in server/jar result in a successful server build. This approach would
not eliminate building the full source tree, but it would provide
quicker feedback as well as promote greater loose coupling through the
increased modularity.
Ultimately yes we need to build the whole source tree, but at least
during development (i.e., for "check-in" builds), we could be more
efficient by building things more incrementally.
As for the perf tests, it really doesn't make sense to me for them to
live in server/jar for a few reasons. First, prior to adding those perf
tests, the tests in server/jar as well as in all our other modules are
primarily unit/integration tests that verify intended functionality.
Performance tests are different in nature. Secondly, the server/jar
build is already painfully slow. Adding long, running performance tests
only exacerbates the problem. Lastly, the perf test hudson job as it is
set up needlessly builds (at least compilation) the whole source tree.
It could instead be set up as a downstream build that just runs tests.
The job as it is now takes a lot longer than it needs to do, and uses up
resources on the build machine when it doesn't have to.
- John
On 10/28/10 2:17 PM, Heiko W.Rupp wrote:
> Hi,
>
> currently (nearly) all tests run in the integration.ejb3 group. No matter if those are server side plugins or core domain
> or real biz logic.
>
> Tests should imho be separated out in more distinct groups and only run if relevant
> e.g. only run UI tests when UI code or code the UI depends on is changed, but not
> when e.g. plugin code has changed.
>
> Another aspect to this is that our builds often build a lot of stuff even if no relevant code
> has been changed. THis is e.g. the case when a plugin is changed, the UI code needs
> no rebuild (and vice versa).
>
> It looks like the issue is not entirely trivial for the automatic testing side, but there is hope:
> - http://www.dzone.com/links/sibilla_first_release_experince_a_new_approach_to.html?ref=ps
> that is by a colleague, Stefano Maestri and seems to analyze the relevant changed path
> and only run relevant tests
> - JunitMax ($$-ware) for Eclipse sorts tests according to runtime and likelihood of failing, so that
> the user gets fast feedback and sees failures in the usual error view
>
> Anyway : server side plugins should move out of enterprise/server to modules/server-plugins
> installer should move out of the enterprise/gui into modules/installer and I guess there are
> a few more candidates.
> John Sanda was also proposing to move the performance testing outside of enterprise/server
> into some (more or less) toplevel module.
>
More information about the rhq-devel
mailing list