Hello everyone,

I’m having trouble deciding on what is the best way to perform tests in Python packages. Here are my thoughts and the solution I find most correct, but which is kind of verbose and ugly to read. I’d be happy about comments. TL;DR and code at the bottom.

For a Python package test to be reliable and representative of what will be installed on a real Fedora system, I think it should be

I found that running tests can go wrong in subtle ways:

EXAMPLE

I’m assuming this common (for our use case unhelpful) packaging structure where the Python modules and the test module are side-by-side in the main directory:

package/
|- module/
|- tests/
|- setup.py
|- tox.ini
|- …

This example uses tox to call the twisted trial test runner, which can be called as Python module or as a standalone script. It should be similar to other test runners. Here are approaches to running tests which are NOT working properly with this simple %check section:

%check
sed -i 's|python -m twisted\.trial|…|' tox.ini
%tox

Broken 1: tox calling `python -m twisted.trial` (upstream default)

Used Python: .tox/py/bin/python :(
sys.path[0]: cwd
Uses system modules: NO
Finds tests module: YES

Broken 2: tox calling `%{python3} -m twisted.trial`

Used Python: /usr/bin/python3
sys.path[0]: cwd
Uses system modules: NO
Finds tests module: YES

Broken 3: tox calling `trial-3`

Used Python: /usr/bin/python3
sys.path[0]: /usr/bin
Uses system modules: YES
Finds tests module: NO

A first, still fragile, solution is choosing the calling method and PYTHONPATH in a way that the order of module discovery is “first system modules, then cwd”:

%check
# Call test runner by its script, thus using the correct python and *avoiding*
# sys.path[0] being the cwd.
sed -i 's|python -m twisted\.trial|trial-3' tox.ini
# Prepend pwd to have access to the test module.
PYTHONPATH=%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}:$PWD %tox

This works, but the modules in the main directory are only shadowed. Thus, they might interfere with the tests if, e.g., the installation skips modules. This can be fixed by

I arbitrarily chose removing for the following solution.

TL;DR

With all previously said, I’d arrive at using the following lines for running my Python tests:

%check
# Make sure we use the correct python and and sys.path[0] is the cwd (so the
# test module *is* found).
sed -i 's|python|%{python3}' tox.ini
# Remove all Python modules from cwd, except the test module, to avoid their use.
find . -maxdepth 2 -name '__init__.py' -exec dirname {} \; | grep -v ^\./tests | xargs rm -rf
# Handles populating PYTHONPATH with system paths.
%tox

Are my raised concerns valid? Are there already solutions I’m missing? Should there be documentation/rpm macros to help with this?


Thanks for your input!
Kai