On 02/11/2011 12:50 PM, Kamil Paral wrote:
Forwarding wwoods' replies here.
One question to wwoods: depcheck_unittest.py requires to be run on amd64 because of those multilib test cases, right? I haven't managed to get it running on i386.
If it is the case, I'll add a short architecture detection to the beginning of the test and print out warning if it is executed on a wrong arch.
...ick. Yes, the tests that exist are currently correct for x86_64 only. They used to also work on i386 but not anymore, I guess. We should probably have different versions of the multilib tests that expect different results, depending on whether you're on a multiarch or singlearch system.
One question to sheer public:
(I'm not sure what you mean by "sheer public" here? But anyway..)
I have found out that depcheck_unittest.py requires a few packages I didn't have installed, specifically python-rpmfluff, rpm-build and mash. How are we gonna track these dependencies? We use autoqa.spec for managing runtime dependencies of autoqa and we use setup() methods for managing dependencies of individual tests. But what about dependencies of our unittests?
Should we add those to the autoqa.spec? Should we add "make test-deps" command to install those deps?
Probably the latter one - the unittests are only interesting to people who are probably going to be modifying the test and testing its changes, so we don't want to pull those in for everyone.
It might make sense to add a BuildRequires: line for them, though?
I wasn't going to bring up the unit tests for depcheck until after the branch had been merged with master, but seeing as it has come up already I might as well.
Disclaimer: Some of this is symantics but I think that the concepts hold in general. I also don't mean this as a slight to anyone or any of the depcheck code in any way.
I advocate putting forth a minimal amount of effort on the current unit tests. While they do indeed test the code, I assert that they are not unit tests as currently written.
(From http://www.artima.com/weblogs/viewpost.jsp?thread=126923 ) A test is not a unit test if: - It talks to the database - It communicates across the network - It touches the file system - It can't run at the same time as any of your other unit tests - You have to do special things to your environment (such as editing config files) to run it.
Unit tests need to run quickly (I've seen guidelines of .1 second per test - can find a reference if desired) so that developers can run them frequently and detect behaviour changes immediately. Longer, more complicated tests can be valuable and are OK as integration tests, but wouldn't be run nearly as often and shouldn't be part of the unit test suite.
I also assert that the current test suite is not ideal for integration testing since it is a little more detailed than we would need in order to test interaction with dependencies.
So, what am I proposing? - Limit the level of effort put into the current test suite * No changes to the .spec * Easy changes to the makefile only for the short term
- Rewrite/refactor the current suite to mock/stub out externals * I am willing to do this, but it would take me a little while; my python-fu is still pretty rusty.
- Decide if we want integration tests and keep them separate from the unit test suite. * I know that you can keep track of tags in py.test and only run a specific tag's tests when explicitly called - I assume that you can do the same with nose
Thoughts?
Tim