On Tue, 2020-01-14 at 01:29 +0100, Miro Hrončok wrote:
On 13. 01. 20 22:54, Adam Williamson wrote:
> On Sun, 2020-01-12 at 23:24 +0100, Miro Hrončok wrote:
> > > Please don't mention it as required or even necessarily recommended,
as
> > > it may not be. I intentionally set up my projects such that tox is
> > > appropriate for upstream CI, not distribution package checking, and
> > > intend distribution packages to run setup.py test.
> >
> > The %tox macro runs the tests in current environment.
> >
> > See
https://github.com/fedora-python/tox-current-env/ for details.
>
> It's not about the environment. It's about what they actually *do*. In
> my projects, tox runs linters, coverage checks and unit tests; setup.py
> test only runs unit tests. This is a convenient separation because I
> find it appropriate to run linters and coverage checks on commits/PRs
> for upstream, but there is no point in running them on package builds.
I agree with everything you said. Taking what we have, ideally we should
convince upstreams to only run tests with default toxenv, not linters.
We also don't have a tox-standard to only run offline testes etc. Maybe one day...
> > At the same time, `setup.py test` is deprecated.
>
> Sigh. Always nice when people deprecate perfectly well-working
> workflows out from underneath you. :/
Right. Two things to consider:
- the deprecation period will most likely last forever
- setup.py files will eventually disappear anyway
So I looked into all of this stuff a bit last night. My conclusions so
far:
* It would be really nice if there were a short Idiot's Guide with a
sample project and RPM spec that puts everything together, as opposed
to me winding up with a browser full of tabs open to
https://src.fedoraproject.org/rpms/pyproject-rpm-macros/tree/master and
https://www.python.org/dev/peps/pep-0517/ and
https://hynek.me/articles/sharing-your-labor-of-love-pypi-quick-and-dirty/
etc. Especially since...
* ...the Shiny New Stuff does not appear to be available on EPEL *at
all* yet - not even EPEL 8. This makes it a bit of a non-starter if you
want to use the same spec file bits across Fedora and EPEL. I realize
it may be impractical/impossible to backport everything to EPEL 7, and
am getting close to the point where I throw in the towel and drop
Python 2 support from my projects' master branches and hang the EPEL 7
package repos out on a branch, but EPEL 8 would be nice and ought to be
possible?
* Unless I'm missing something, what you suggested for tox - "ideally
we should convince upstreams to only run tests with default toxenv" -
actually seems weirdly difficult to implement. AFAIK none of the
official or unofficial tox docs I can find really cover the idea of
having an environment that's *defined* but is not *default*. It seems
to be virtually universal practice with tox that you put every
environment in `envlist`...and if you just run `tox` without any `-e`
argument or special env var, it runs every environment in `envlist`.
People seem to assume all environments will be default, and if you want
to run fewer than 'all of them' you filter with `-e` or whatever.
I *did* figure out a way to do this, but it's something I never really
saw documented anywhere, and I eventually hit a roadblock with using it
in the CI setup I made. You have to make a `tox.ini` like this:
[tox]
envlist = py{27,36,37,38,39}
[testenv]
deps =
-r{toxinidir}/install.requires
-r{toxinidir}/tests.requires
ci: -r{toxinidir}/ci.requires
commands =
py.test
ci: py.test --cov-report term-missing --cov-report xml --cov openqa_client
ci: diff-cover coverage.xml --fail-under=90
ci: diff-quality --violations=pylint --fail-under=90
setenv =
PYTHONPATH = {toxinidir}
by having `ci` directives in `deps` and `commands` like that, you sort
of imply the existence of environments like `py27-ci` without ever
explicitly declaring them, and tox itself is OK with this. You can run
`tox -epy{27,36,37,38,39}-ci` and it'll do what you (maybe) expect -
it'll read ci.requires and run the additional commands. But if you just
run `tox` it'll run the py{27,36,37,38,39} environments without the
additional `ci` bits.
This definitely seems to be a bit 'off the beaten track', though, like
I said - it's not explicitly documented anywhere I could find (most
documentation of the whole generative environments thing assumes all
the environments will be declared up in `envlist`) and commands like
`tox -l` and `tox -a` may not do what you expect.
The showstopper I ultimately hit was in an extension on top of tox,
`tox-gh-actions`, which is a convenience thing for using tox with
GitHub Actions:
https://github.com/ymyzk/tox-gh-actions/issues/11
(the project I'm using as a testbed for this stuff is hosted in github,
and Actions seemed like the easiest way to set up CI). It seems like
this 'implicit non-default environment' thing completely trips up tox-
gh-actions; when I added this section to `tox.ini` to try and make it
use the `ci` environments:
[gh-actions]
python =
2.7: py27
3.6: py36-ci
3.7: py37-ci
3.8: py38-ci
3.9: py39-ci
it would run, but each actual test run seemed not to actually call tox
at all, it just did apparently nothing and then 'passed'.
So in the end I gave up and wound up just making the `-ci` environments
the defaults, explicitly declared in envlist, and figuring I'd have the
spec file use `-e` to explicitly run the non-ci environments. But then
I didn't even do that because of the whole EPEL thing, so so far I've
wound up not using the pyproject macros at all, I'm doing this instead:
%if 0%{?with_python2}
PYTHONPATH=%{buildroot}%{python2_sitelib} py.test
%endif # with_python3
%if 0%{?with_python3}
PYTHONPATH=%{buildroot}%{python3_sitelib} py.test-3
%endif # with_python3
PYTHON!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net