[Beaker-devel] Integration tests vs unit tests

Dan Callaghan dcallagh at redhat.com
Fri Sep 6 03:22:53 UTC 2013


Excerpts from Nick Coghlan's message of 2013-09-06 12:27:54 +1000:
> Just dumping a transcript of an IRC conversation between rmancy and I
> about unit tests vs integration tests (in the context of the patch that
> makes the createrepo command configurable:
> http://gerrit.beaker-project.org/#/c/2208/).
> 
> At the moment, Beaker's unit testing is fairly minimal, so we don't have
> a good, quick, confidence building set of tests to run before pushing to
> Gerrit, just the full set of integration tests (which can take half an
> hour or more to run, depending on the details of your system).
> 
> It's going to take some time for us to improve Beaker's unit testing
> story, this just struck me as an expedient way to get something on
> record that this is the direction we're likely to be heading :)

One thing I don't see discussed below at all is the maintenance cost of 
the tests, which is something that concerns me a lot. Tests are written 
once but then maintained forevermore. Every time you write a test, you 
are trading the cost of writing it plus maintaining it forever, against 
the benefit of potential bugs it will find in future.

I think at least 75% of my development time has gone towards writing and 
maintaining tests, both on Beaker and earlier projects. You probably 
have a similar experience.

The reason I mention it is that you might think intuitively that unit 
tests have a lower maintenance cost than functional tests, because unit 
tests are faster. But in my experience unit tests have a much *higher* 
maintenance cost because they are invariably tied to implementation 
details, which means that any "refactoring" (changing implementation 
details without affecting the behaviour from an external point of view) 
means rewriting or rearranging huge swathes of unit tests, whereas 
a good functional test should require no changes at all.

Also, I think we should be clear about our terminology here. Beaker has 
"integration tests" (the directory is called IntegrationTests/) but only 
in the sense that they require an external service (database) which 
makes them unsuitable to run during a package build. The actual tests in 
that directory are a mixture of functional tests (tests which interact 
with the system from an external point of view) and unit tests (tests 
which touch internal implementation details at a low level), tending 
towards mainly functional.

In my opinion the speed of the whole test suite is irrelevant because 
they can be run by a robot while humans do other things. What matters to 
me is how much effort I have to expend fixing tests when I want to 
improve something in Beaker. For that reason I would rather have as much 
coverage as possible from small, simple, clear functional tests, 
resorting to "unit tests" (ones which touch internal implementation 
details) only when it is too hard to write a simple functional test for 
it.

Sometimes even *no* test is the right choice, because the maintenance 
cost of the test outweighs the likelihood of it finding a real bug.

Some recent examples of when I think a good functional test is too hard 
to write:

* testing a database constraint that is already enforced in other ways 
  by the application
* testing failure modes that cannot be triggered by interacting with the 
  system in a normal way
* tests which do complicated things that a test is not normally expected 
  to do, like restarting the server process in the middle of the test

-- 
Dan Callaghan <dcallagh at redhat.com>
Software Engineer, Infrastructure Engineering and Development
Red Hat, Inc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/beaker-devel/attachments/20130906/d2ba0a04/attachment.sig>


More information about the Beaker-devel mailing list