On 02/11/2011 12:50 PM, Kamil Paral wrote:
Forwarding wwoods' replies here.
> One question to wwoods:
> depcheck_unittest.py requires to be run on amd64 because of those
> multilib test cases, right? I haven't managed to get it running on
> i386.
>
> If it is the case, I'll add a short architecture detection to the
> beginning of the test and print out warning if it is executed on a
> wrong arch.
...ick. Yes, the tests that exist are currently correct for x86_64 only.
They used to also work on i386 but not anymore, I guess. We should
probably have different versions of the multilib tests that expect
different results, depending on whether you're on a multiarch or
singlearch system.
>
> One question to sheer public:
(I'm not sure what you mean by "sheer public" here? But anyway..)
> I have found out that depcheck_unittest.py requires a few packages I
> didn't have installed, specifically python-rpmfluff, rpm-build and
> mash. How are we gonna track these dependencies? We use autoqa.spec
> for managing runtime dependencies of autoqa and we use setup() methods
> for managing dependencies of individual tests. But what about
> dependencies of our unittests?
>
> Should we add those to the autoqa.spec? Should we add "make test-deps"
> command to install those deps?
Probably the latter one - the unittests are only interesting to people
who are probably going to be modifying the test and testing its changes,
so we don't want to pull those in for everyone.
It might make sense to add a BuildRequires: line for them, though?
I wasn't going to bring up the unit tests for depcheck until after the
branch had been merged with master, but seeing as it has come up already
I might as well.
Disclaimer: Some of this is symantics but I think that the concepts hold
in general. I also don't mean this as a slight to anyone or any of the
depcheck code in any way.
I advocate putting forth a minimal amount of effort on the current unit
tests. While they do indeed test the code, I assert that they are not
unit tests as currently written.
(From
http://www.artima.com/weblogs/viewpost.jsp?thread=126923 )
A test is not a unit test if:
- It talks to the database
- It communicates across the network
- It touches the file system
- It can't run at the same time as any of your other unit tests
- You have to do special things to your environment (such as editing
config files) to run it.
Unit tests need to run quickly (I've seen guidelines of .1 second per
test - can find a reference if desired) so that developers can run them
frequently and detect behaviour changes immediately. Longer, more
complicated tests can be valuable and are OK as integration tests, but
wouldn't be run nearly as often and shouldn't be part of the unit test
suite.
I also assert that the current test suite is not ideal for integration
testing since it is a little more detailed than we would need in order
to test interaction with dependencies.
So, what am I proposing?
- Limit the level of effort put into the current test suite
* No changes to the .spec
* Easy changes to the makefile only for the short term
- Rewrite/refactor the current suite to mock/stub out externals
* I am willing to do this, but it would take me a little while; my
python-fu is still pretty rusty.
- Decide if we want integration tests and keep them separate from the
unit test suite.
* I know that you can keep track of tags in py.test and only run a
specific tag's tests when explicitly called
- I assume that you can do the same with nose
Thoughts?
Tim