----- Original Message -----
I wasn't going to bring up the unit tests for depcheck until
after the
branch had been merged with master, but seeing as it has come up
already
I might as well.
Disclaimer: Some of this is symantics but I think that the concepts
hold
in general. I also don't mean this as a slight to anyone or any of the
depcheck code in any way.
I advocate putting forth a minimal amount of effort on the current
unit
tests. While they do indeed test the code, I assert that they are not
unit tests as currently written.
(From
http://www.artima.com/weblogs/viewpost.jsp?thread=126923 )
A test is not a unit test if:
- It talks to the database
- It communicates across the network
- It touches the file system
- It can't run at the same time as any of your other unit tests
- You have to do special things to your environment (such as editing
config files) to run it.
Unit tests need to run quickly (I've seen guidelines of .1 second per
test - can find a reference if desired) so that developers can run
them
frequently and detect behaviour changes immediately. Longer, more
complicated tests can be valuable and are OK as integration tests, but
wouldn't be run nearly as often and shouldn't be part of the unit test
suite.
From this point of view the depcheck_unittest.py is really rather an
integration test.
Thank you for mentioning those software design guidelines. It makes sense. We can run the
unit tests quickly before commit, and the longer integration tests may be automatically
run after each commit on our server.
I also assert that the current test suite
By "current test suite" you mean the code mkrizek created in the maketest
branch, right?
is not ideal for integration
testing since it is a little more detailed than we would need in order
to test interaction with dependencies.
What dependencies you have exactly in mind?
So, what am I proposing?
- Limit the level of effort put into the current test suite
* No changes to the .spec
* Easy changes to the makefile only for the short term
- Rewrite/refactor the current suite to mock/stub out externals
* I am willing to do this, but it would take me a little while; my
python-fu is still pretty rusty.
- Decide if we want integration tests and keep them separate from the
unit test suite.
We definitely want integration tests (like depcheck_unittest.py). We also want standard
unit tests, although I'm little skeptical how many of them we will create in our
little team. The more complex tests seem to have larger benefit from my view.
* I know that you can keep track of tags in py.test and only run a
specific tag's tests when explicitly called
- I assume that you can do the same with nose
Thoughts?
Tim
If you can propose some standardized way of writing unit tests/integration tests (where to
place them, how to call them, what to contain, which principles to adhere to) and
implement their execution logic, it will be simply great.
Thanks for useful comments,
Kamil