On 02/12/2011 01:36 AM, Kamil Paral wrote:
> From this point of view the depcheck_unittest.py is really rather an integration
test.
Thank you for mentioning those software design guidelines. It makes sense. We can run the
unit tests quickly before commit, and the longer integration tests may be automatically
run after each commit on our server.
My thoughts, exactly. It would be awesome to
have all of this (and more)
work with a (yet to be implemented) CI system.
>
> I also assert that the current test suite
By "current test suite" you mean the code mkrizek created in the maketest
branch, right?
I was referring to the depcheck unit tests. I haven't spent a
whole lot
of time with the tests that mkrizek created in the maketest branch but I
think that most of the same concepts apply.
> is not ideal for integration
> testing since it is a little more detailed than we would need in order
> to test interaction with dependencies.
What dependencies you have exactly in mind?
In this case, a dependency is anything that isn't written inside the
class/file under test. For example, the dependencies (for this use of
the word) in autoqa.koji_utils are: autoqa.config, koji, rpmUtils,
repoinfo, sys and time.
>
> So, what am I proposing?
> - Limit the level of effort put into the current test suite
> * No changes to the .spec
> * Easy changes to the makefile only for the short term
>
> - Rewrite/refactor the current suite to mock/stub out externals
> * I am willing to do this, but it would take me a little while; my
> python-fu is still pretty rusty.
>
> - Decide if we want integration tests and keep them separate from the
> unit test suite.
We definitely want integration tests (like depcheck_unittest.py). We also want standard
unit tests, although I'm little skeptical how many of them we will create in our
little team. The more complex tests seem to have larger benefit from my view.
I think that it would be a gradual process of adding both; I'm certainly
not proposing that we drop everything to increase our unit test
coverage. Assuming that there is decent coverage for the section of code
that you're modifying, I think of unit tests as a kind of 'safety net'
to quickly alert you if you end up changing some behaviour that you
weren't expecting to change.
Integration/Functional tests could do the same thing but unit tests tend
are run more often and are usually more localized ("there was a failure
in this block of code" versus "there was a failure in one of the methods
in this general area"). At the end of the day, both are valuable - just
for different purposes.
> * I know that you can keep track of tags in py.test and only run
a
> specific tag's tests when explicitly called
> - I assume that you can do the same with nose
>
> Thoughts?
>
> Tim
If you can propose some standardized way of writing unit tests/integration tests (where
to place them, how to call them, what to contain, which principles to adhere to) and
implement their execution logic, it will be simply great.
My current thought would be to use a feature of py.test that uses
decorators to classify tests as integration tests (see
http://blog.devork.be/2009/12/skipping-slow-test-by-default-in-pytest.html
as an example). By default, py.test would run the unit tests and the
integration+unit tests would be run by using an additional command line
parameter.
I'm trying to do this as a proof of concept while I'm working on #265
and figured that I would do the same with Nose/unittest on another
simple ticket. Once that's done, we can have a more detailed discussion
about what approach to take.
Tim