New subject: Draft of New Python Packaging Guidelines - 0.2

30 Apr 2020

      Hello!
Below is a draft of new Packaging Guidelines! It's full of unfinished 
areas (marked with XXX), but it's ready for scrutiny.
A possibly updated version is on 
https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg?view
Generally, for rules marked **SHOULD** we know of cases where they 
should be broken; for things marked **MUST** we don't.
We have tried to only take the Good Existing Things™ (macros, practices) 
and revise the rest without thinking about the past much. Some used 
technology is unfortunately not compatible with current EPELs, but we 
have considered Fedora 31+. Using the current Python guidelines will 
still be an option for people who target EPEL 6/7/8.
The main controversial idea (of many) is synchronizing Fedora's names 
(a.k.a. `python3dist(...)`, a.k.a. `name` in `setup.py`) with the Python 
Package Index (PyPI, pypi.org), which has by now become the de-facto 
canonical namespace for freely redistributable Python packages.
We believe that this will help both Fedora and the greater Python 
ecosystem, but there will definitely be some growing pains.
Most of Fedora Python packages already are on PyPI, but more than 250 
are missing. There is software developed within Fedora (e.g. pagure, 
fedpkg, rust2rpm); projects that aren't primarily Python packages (e.g. 
perf, ceph, xen, setroubleshoot, iotop); and others.
A full list is attached. The names have been temporarily blocked on PyPI 
to keep trolls from taking them while this is discussed.
Over at the Python Discourse we are discussing how to properly handle 
these; once that discussion is settled they should be unblocked: 
https://discuss.python.org/t/pypi-as-a-project-repository-vs-name-registry-a...
Another general idea is that package metadata should be kept upstream; 
the spec file should duplicate as little of it as possible. Any 
adjustments should be done with upstreamable patches.
The draft lives on hackmd.io, which we found easy to collaborate with. 
If you have an account there, we can add you. If you'd like to 
collaborate some other way, let us know.
Petr and Miro
-----------------------
Current draft for inline comments:
...
[IMPORTANT]
This is a DRAFT; it is not in effect yet.
# Python Packaging Guidelines (draft)
...
[IMPORTANT]
This is a *beta* version of the Python Packaging Guidelines and the
associated RPM macros.
...
Packagers that opt in to following this version **MUST** be prepared
to change their packages frequently when the guidelines or macros are 
updated.
...
These packagers **SHOULD** join [Python SIG mailing
list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...) 
and monitor/start conversations there.
These Guidelines represent a major rewrite and paradigm shift, and not 
all packages are updated to reflect this.
Older guidelines are still being kept up to date, and existing packages 
**MAY** use them instead of this document:
* 201x-era Python packaging guidelines (for packages that use e.g. 
`%py3_install` or `setup.py install`)
* Python 2 appendix (for e.g. `%py2_install`) (requires FESCo exception)
...
[NOTE]
These guidelines only support Fedora 31+. For older releases (such as
in EPEL 8), consult the older guidelines (XXX link).
The two "Distro-wide guidelines" below apply to all software in Fedora 
that uses Python at build- or run-time.
These rest of the Guidelines apply to packages that ship code that *can* 
be imported in Python.
Specifically, that is all packages that install files under 
`/usr/lib*/python*/`.
Except for the two "Distro-wide guidelines", these Guidelines do not 
apply to simple one-file scripts or utilities, especially if these are 
included with software not written in Python.
However, if an application (e.g. CLI tool, script or GUI app) needs a 
more complex Python library, the library **SHOULD** be packaged as an 
importable library under these guidelines.
A major goal for Python packaging in Fedora is to *harmonize with the 
wider Python ecosystem*, that is, the [Python Packaging 
Authority](https://pypa.io) (PyPA) standards and the [Python Package 
Index](https://pypi.org) (PyPI).
Packagers **SHOULD** be prepared to get involved with upstream projects 
to establish best practices as outlined here. We wish to improve both 
Fedora and the wider Python ecosystem.
...
[NOTE]
Fedora's Python SIG not only develops these guidelines, but it's also
involved in PyPA standards and Python packaging best practices. Check 
out [the wiki](https://fedoraproject.org/wiki/SIGs/Python) or [mailing 
list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...) 
if you need help or wish to help out.
## Distro-wide guidelines
### BuildRequire python3-devel
**Every** package that uses Python (at runtime and/or build time), 
and/or installs Python modules **MUST** explicitly include 
`BuildRequires: python3-devel` in its `.spec` file, even if Python is 
not actually invoked during build time.
If the package uses an alternate Python interpreter instead of `python3` 
(e.g. `pypy`, `jython`, `python27`), it **MAY** instead require the 
coresponding `*-devel` package.
The `*-devel` package brings in relevant RPM macros. It may also enable 
automated or manual checks: for example, Python maintainers use this 
requirement to list packages that use Python in some way and might be 
affected by planned changes.
### Mandatory macros
The following macros **MUST** be used where applicable.
The expansions in parentheses are provided only as reference/examples.
The macros are defined for you in all supported Fedora and EPEL versions.
* `%{python3}` (`/usr/bin/python3`)
The Python interpreter.
   For example, this macro should be used for invoking Python from a 
`spec` file script, passed to `configure` scripts to select a Python 
executable, or used as `%{python3} -m pytest` to run a Python-based tool.
...
XXX: Use shebang opts for shebang; document pathfix macro. Document
cases where you don't want this.
* `%{python3_version}` (e.g. `3.9`)
Version of the Python interpreter.
* `%{python3_version_nodots}` (e.g. `39`)
Version of the Python interpreter without the dot.
* `%{python3_sitelib}` (e.g. `/usr/lib/python3.9/site-packages`)
Where pure-Python modules are installed.
* `%{python3_sitearch}` (e.g. `/usr/lib64/python3.9/site-packages`)
Where Python3 extension modules (native code, e.g. compiled from C) 
are installed.
The rest of this document uses these macros, along with `%{_bindir}` 
(`/usr/bin/`), instead of the raw path names.
## Python implementation support
Fedora primarily targets *CPython*, the reference implementation of the 
Python language. We generally use “Python” to mean CPython.
Alternate implementations like `pypy` are available, but currently lack 
comprehensive tooling and guidelines. When targetting these, there are 
no hard rules (except the general Fedora packaging guidelines). But 
please try to abide by the *spirit* of these guidelines. When in doubt, 
consider consulting the Python SIG.
## Python version support
Fedora packages **MUST NOT** depend on other versions of the CPython 
interpreter than the current `python3`.
In Fedora, Python libraries are packaged for a single version of Python, 
called `python3`. For example, in Fedora 32, `python3` is Python 3.8.
In the past, there were multiple Python stacks, e.g. `python3.7` and 
`python2.7`, installable together on the same machine. That is also the 
case in some projects that build *on top* of Fedora, like RHEL, EPEL and 
CentOS. Fedora might re-introduce parallell-installable stacks in the 
future (for example if a switch to a new Python version needs a 
transition period, or if enough interested maintainers somehow appear).
...
XXX dots in package names!
Fedora does include alternate interpreter versions, e.g. `python2.7` or 
`python3.5`, but these are meant only for developers that need to test 
upstream code. Bug and security fixes for these interpreters only cover 
this use case.
Packages such as `pip` or `tox`, which enable setting up isolated 
environments and installing third-party packages into them, **MAY**, as 
an exception to the rule above, use these interpreters as long as this 
is coordinated with the maintainers of the relevant Python interpreter.
## BuildRequire pyproject-rpm-macros
While these guidelines are in Beta, each Python package **MUST** have 
`BuildRequires: pyproject-rpm-macros` to access the beta macros.
(When we go out of Beta, listing the dependency won't be necessary: 
we'll make `python3-devel` require it.)
## Naming
Python packages have several different names, which should be kept in 
sync but will sometimes differ for historical or practical reasons. They 
are:
* the Fedora *source package name* (or *component name*, `%{name}`),
* the Fedora *built RPM name*,
* the *project name* used on *PyPI* or by *pip*, and
* the *importable module name* used in Python (a single package may have 
multiple importable modules).
Some examples (both good and worse):
| Fedora component  | Built RPM          | Project name   | Importable 
module    |
| ----------------- | ------------------ | -------------- | 
-------------------- |
| `python-requests` | `python3-requests` | `requests`     | `requests` 
         |
| `PyYAML`          | `python3-pyyaml`   | `pyyaml`       | `yaml` 
         |
| `python-ldap`     | `python3-ldap`     | `python-ldap`  | `ldap`, 
`ldif`, etc. |
| `python-pillow`   | `python3-pillow`   | `pillow`       | `PIL` 
         |
Elsewhere in this text, the metavariables `SRPMNAME`, `RPMNAME`, 
`PROJECTNAME`, `MODNAME` refer to these names, respectively.
### Canonical project name
Most of these names are case-sensitive machine-friendly identifiers, but 
the *project name* has human-friendly semantics: it is case-insensitive 
and treats some sets of characters (like `._-`) specially.
For automated use, it needs to be 
[normalized](https://www.python.org/dev/peps/pep-0503/#normalized-names) 
to a canonical format used by Python tools and services such as 
setuptools, pip and PyPI.
The `%{py_dist_name}` macro implements this normalization.
Elsewhere in this text, the metavariable `DISTNAME` refers to the 
canonical form of the project name.
...
XXX
# XXX specfile
%py_set_name Django
-> %distname django
-> %pojectname Django

XXX rewrite: It is customary to define the macro `%{distname}` as the
canonical project name and use it throughout the `spec` file. Some 
helper macros use `%{distname}` for this purpose by default.
...
XXX in some places, the "original project name" must be used -- e.g.
`%pypi_source` and `%autosetup` need `Django`, not `django`.
...
XXX The following sections should supersede the [Python section on
the Naming 
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_python_mo...).
### Library naming
A built (i.e. non-SRPM) package for a *Python library* **MUST** be named 
with the `python3-` prefix.
A source package containing primarily a *Python library* **MUST** be 
named with the prefix `python-`.
The Fedora package's name **SHOULD** contain the *canonical project 
name*. If possible, the project name **SHOULD** be the same as the name 
of the main importable module, with underscores (`_`) replaced by dashes 
(`-`).
A *Python library* is a package meant to be imported in Python, such as 
with `import requests`.
Tools like *Ansible* or *IDLE*, whose code is importable but not 
primarily meant to be imported, are not considered libraries in this 
sense, so this section does not apply for them.
(See the [general Libraries and Applications 
guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_libraries_and_ap...) 
for details.)
If the importable module name and the project name do not match, users 
frequently end up confused. In this case, packagers *should* ensure that 
upstream is aware of the problem and (especially for new packages where 
renaming is feasible) strive to get the package renamed. The Python SIG 
is available for assistance.
The Fedora package name should be formed by taking the *project name* 
and prepending `python-` if it does not already start with `python-`.
This may leads to conflicts (e.g. between 
[bugzilla](https://pypi.org/project/bugzilla/) and 
[python-bugzilla](https://pypi.org/project/python-bugzilla/)). In that 
case, ensure upstream is aware of the potentially confusing naming and 
apply best judgment.
### Application naming
Packages that primarily provide executables **SHOULD** be named 
according to the general Fedora guidelines (e.g. `ansible`).
Consider adding a virtual provide according to Library naming above 
(e.g. `python3-PROJECTNAME`), if it would help users find the package.
## Files to include
### Source files and bytecode cache
Packages **MUST** include the source file (`*.py`) **AND** the bytecode 
cache (`*.pyc`) for each pure-Python module.
The cache files are found in a `__pycache__` directory and have an 
interpreter-dependent suffix like `.cpython-39.pyc`.
The cache is not necessary to run the software, but if it is not found, 
Python will try to create it. If this succeeds, the file is not tracked 
by RPM and it will linger on the system after uninstallation. If it does 
not succeed, users can get spurious SELinux AVC denials in the logs.
Normally, byte compilation (generating the cache files) is done for you 
by the `brp-python-bytecompile` script (XXX link to BRP guidelines), 
which runs automatically after the `%install` section of the spec file 
has been processed. It byte compiles any `.py` files that it finds in 
`%{python3_sitelib}` or `%{python3_sitearch}`.
You must include these files of your package (i.e. in the `%files` section).
If the code is in a subdirectory (importable package), include the 
entire directory:
```
%files
%{python3_sitelib}/foo/
```
Adding the trailing slash is best practice for directories.
However, this cannot be used for top-level scripts (those directly in 
e.g. `%{python3_sitelib}`), because both `%{python3_sitelib}` and 
`%{python3_sitelib}/__pycache__/` are owned by Python itself.
Here, the `%pycached` macro can help. It expands to the given `*.py` 
source file and its corresponding cache file(s). For example:
```
%files
%pycached %{python3_sitelib}/foo.py
```
expands roughly to:
```
%files
%{python3_sitelib}/foo.py
%{python3_sitelib}/__pycache__/foo.cpython-3X{,.opt-?}.pyc
```
#### Manual byte compilation
If you need to bytecompile stuff outside of 
`%{python3_sitelib}`/`%{python3_sitearch}`, use the `%py_byte_compile` 
macro.
For more details on `%py_byte_compile` and on the internals of bytecode 
compilation, please see the 
[appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#m...).
...
XXX: Copy the section from appendix here
### Dist-info metadata
Each Python package **MUST** include *Package Distribution Metadata* 
conforming to [PyPA 
specifications](https://packaging.python.org/specifications/) 
(specifically, [Recording installed 
distributons](https://packaging.python.org/specifications/recording-installed-packages/)).
This applies to libraries (e.g. `python-requests`) as well as tools 
(e.g. `ansible`).
...
XXX what with splitting into subpackages? 1) dist-info always
installed, 2) dist-info installed trough a metapackage?
...

Ideally, do the split upstream
Consider package split between library & tool (see poetry, fedpkg)

e.g.
When software is split into several subpackages, it is OK to only
ship metadata in one built RPM.
The metadata takes the form of a `dist-info` directory installed in 
`%{python3_sitelib}` or `%{python3_sitearch}`, and contains information 
that tools like 
[`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html) 
use to introspect installed libraries.
...
XXX example %files with manual dist-info entry
Note that some older tools instead put metadata in an `egg-info` 
directory, or even a single file.
This won't happen if you use the `%pyproject_wheel` macro.
If your package uses a build system that generates an `egg-info` 
directory or file, please contact Python SIG.
...
XXX We need a better solution before we go out of beta.
As an exception, the Python standard library **MAY** ship without this 
metadata.
### Explicit lists
Packagers **SHOULD NOT** simply glob everything under a shared directory.
In particular, the following **SHOULD NOT** be used:
* `%{python3_sitelib}/*`
* `%{python3_sitearch}/*`
* `%{python_sitelib}/*`
* `%{python_sitearch}/*`
* `%{_bindir}/*`
* `%pyproject_save_files *`
* `%pyproject_save_files +bindir`
This rule serves as a check against common mistakes which are otherwise 
hard to detect. It does limit some possibilities for automation.
The most common mistake this rule prevents is installing a test suite 
system-wide as an importable module named `test`, which would then 
conflict with other such packages.
## PyPI parity
Every Python package in Fedora **SHOULD** also be available on [the 
Python Package Index](https://pypi.org) (PyPI).
The command `pip install PROJECTNAME` **MUST** install the same package 
(possibly in a different version), install nothing, or fail with a 
reasonable error message.
If this is not the case, the packager **MUST** contact upstream about 
this. The goal is to get the project name registered or blocked on PyPI, 
or to otherwise ensure the rule is followed.
...
XXX Note that project names that were in Fedora but not on PyPI when
these guidelines were proposed are *blocked* as we discuss how they 
should be handled.
...
This prevents potential trolls, but also legitimate owners, from
taking them.
This is necessary to protect users, avoid confusion, and enable 
automation as Fedora and upstream ecosystems grow more integrated.
As always, specific exceptions can be granted by FPC (XXX link 
exceptions rules).
...
XXX Write an automated check for this.
## Provides and requirements
### Provides for importable modules
For any module intended to be used in Python 3 with `import MODNAME`, 
the package that includes it **SHOULD** provide `python3-MODNAME`, with 
underscores (`_`) replaced by dashes (`-`).
This is of course always the case if the package is named 
`python3-MODNAME`. If the subpackage has some other name, then add 
`%py_provides python3-MODNAME` explicitly. See the following section to 
learn about `%py_provides`.
### Automatic unversioned provides
For any `FOO`, a package that provides `python3-FOO` **SHOULD** use 
`%py_provides` or an automatic generator to also provide `python-FOO`.
The provide **SHOULD NOT** be added manually: if the generator or macro 
is not used, the provide shall not be added at all.
On Fedora 33+, this is done automatically for package names by a 
generator. The generator can be disabled by undefining 
`%__pythonname_provides`.
The following macro invocation will provide both `python3-FOO` and 
`python-FOO`:
%py_provides python3-FOO
...
XXX: finalize `%py_provides`
Using the generator or macro is important, because the specific form of 
the provide may change in the future.
### Machine-readable provides
Every Python package **MUST** provide `python3dist(DISTNAME)` **and** 
`python3.Xdist(DISTNAME)`, where `X` is the minor version of the 
interpreter and `DISTNAME` is the *canonical project name* corresponding 
to the *dist-info metadata*, for example `python3.9dist(requests)`. 
(XXX: add links to previous sections)
This is generated automatically from the dist-info metadata.
The provide **SHOULD NOT** be added manually: if the generator fails to 
add it, the metadata **MUST** be fixed.
If necessary, the automatic generator can be disabled by undefining 
`%__pythondist_provides`.
These *Provides* are used for automatically generated *Requires*.
### Dependencies
As mentioned above, each Python package **MUST** explicitly BuildRequire 
`python3-devel`.
Packages **MUST NOT** have dependencies (either build-time or runtime) 
with the unversioned prefix `python-` if the corresponding `python3-` 
dependency can be used instead.
Packages **SHOULD NOT** have explicit dependencies (either build-time or 
runtime) with a minor-version prefix such as `python3.8-` or 
`python3.8dist(`. Such dependencies **SHOULD** instead be automatically 
generated or a macro should be used to get the version.
Packages **SHOULD NOT** have an explicit runtime dependency on `python3`.
Instead of depending on `python3`, packges have an automatic dependency 
on `python(abi) = 3.X` when they install files to `%{python3_sitelib}` 
or `%{python3_sitearch}`, or they have an automatic dependency on 
`/usr/bin/python3` if they have executable Python scripts, or they have 
an automatic dependency on `libpython3.X.so.1.0()` if they embed Python.
These rules help ensure a smooth upgrade path when `python3` is updated 
in new versions of Fedora.
### Automatically generated dependencies
Packages **MUST** use the automatic Python run-time dependency generator.
Packages **SHOULD** use the opt-in build-dependency generator if possible.
Any necessary changes **MUST** be done by patches or modifying the 
source (e.g. with `sed`), rather than disabling the generator. The 
resulting change **SHOULD** be offered to upstream. As an exception, 
[filtering](https://docs.fedoraproject.org/en-US/packaging-guidelines/AutoProvidesAndReq...) 
**MAY** be used for temporary workarounds and bootstrapping.
Dependencies covered by the generators **SHOULD NOT** be repeated in the 
`.spec` file. (For example, if the generator finds a `requests` 
dependency, then `Requires: python3-requests` is redundant.)
The automatically generated requirements are in the form 
`python3.Xdist(DISTNAME)`, potentially augmented with version 
requirements or combined together with [rich 
dependencies](https://rpm.org/user_doc/boolean_dependencies.html).
Note that the generators only cover Python packages. Other dependencies, 
often C libraries like `openssl-devel`, must be specified in the `.spec` 
file manually.
...
XXX When implemented, this goes here: Alternatively, upstream Python
packages may list non-Python dependencies in the 
`[tool.fedora.requires]`/`[tool.fedora.buildrequires]` section of 
`pyproject.toml`. This is non-standard and only recommended for 
Fedora-related packages.
Where the requirements are specified in the source depends on each 
project's build system and preferences. Common locations are 
`pyproject.toml`, `setup.py`, `setup.cfg`, `config.toml`.
#### Run-time dependency generator
The automatic runtime dependency generator uses package metadata (as 
recorded in installed `*.dist-info` directories) to determine what the 
package depends on.
In an emergency, you can opt-out from running the requires generator by 
adding `%{?python_disable_dependency_generator}` to the package 
(usually, just before the main package’s `%description`).
#### Build-time dependency generator
The opt-in (but strongly recommended) build-time dependency generator 
gathers information from [`pyproject.toml` build-system 
information](https://www.python.org/dev/peps/pep-0517/#source-trees) 
(with fallback to `setuptools`) plus a standardized [build-system 
hook](https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel) 
to gather further requirements.  See `%pyproject_buildrequires` (XXX 
link) for more details.
### Test dependencies
See the *Tests* section. (XXX link.)
### Extras
...
XXX No story here so far.
...
XXX Note:
[python-django-storages](https://src.fedoraproject.org/rpms/python-django-storages/blob/master/f/pyth...), 
`drf-yasg` etc. do `%python_provide python3-%{srcname}+%{distname}` and 
`python3dist(%{srcname}/%{extraname})` XXX 
`python3dist(%{srcname}[%{extraname}])`
## Interpreter invocation
### Shebangs
Shebangs lines to invoke Python **SHOULD** be `#!%{python3} 
-%{py3_shebang_flags}` and it **MAY** include extra flags.
...
XXX define py3_shebang_flags
If the default flags from `%{py3_shebang_flags}` are not desirable, 
packages **SHOULD** explicitly redefine the macro to remove them.
Using `#!%{python3}` (`#!/usr/bin/python3`) rather than e.g. 
`#!/usr/bin/env python` ensures that the system-wide Python interpreter 
is used to run the code, even if the user modifies `$PATH` (e.g. by 
activating a virtual environment).
By default, `-%{py3_shebang_flags}` expands to `-s`, which means *don't 
add user site directory to `sys.path`*. That ensures user-installed 
Python packages (e.g. by `pip install --user`) don't interfere with the 
RPM installed software. Sometimes, `pip`-installed content is desirable, 
such as with plugins. Redefining `%{py3_shebang_flags}` to not include 
`s`, rather than not using the macro at all, ensures that existing or 
future automation won't add the flag.
The `%pyproject_install` macro automatically changes all Python shebangs 
in `%{buildroot}%{_bindir}/*` to use `%{python3}` and add 
`%{py3_shebang_flags}` to the existing flags. If you're not using the 
macro or you need to change a shebang in a different directory, you can 
use the `pathfix.py` command as follows:
```
pathfix.py -n -p -k -i %{python3} -a %{py3_shebang_flags} SCRIPTNAME …
```
...
XXX Ouch! Macroize this? Hell yes!
`%py3_shebang_fix <paths>` -- also rename `%{py3_shebang_flags}`
(doesn't include dash)
...
can be used in `%prep` or `%install`
where:
* `-n` disables ceating backups
* `-p` preserves timestamps
* `-k` keeps existing flags
* `-i` specifies the new interpreter
### Invokable Python modules
Every executable `TOOL` for which the current version of Python matters 
**SHOULD** also be invokable by `python3 -m TOOL`.
If the software doesn't provide this functionality, packagers **SHOULD** 
ask the upstream to add it.
This applies to tools that modify the current Python environment (like 
installing or querying packages), use Python for configuration, or use 
Python to run plugins.
It does not apply to tools like GIMP or Bash which support plugins in 
multiple languages and/or have other means to specify the interpreter.
For example, `pip` can be invoked as `python3 -m pip`.
This allows users to accurately specify the Python version used to run 
the software. This convention works across different environments that 
might not always set `$PATH` or install scripts consistently.
## Using Cython
Tightening the general Fedora policy, packages **MUST NOT** use files 
pre-generated by Cython. These **MUST** be deleted in `%prep` and 
regenerated during the build.
As an exception, these sources **MAY** be used temporarily to prevent 
build time circular dependencies by following the 
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping) 
guidelines.
Generated files (the ones that must be deleted) have a generic `.c` or 
`.cpp` extension.
Cython source files (which should stay) usually have the `.pyx` or 
`.pxd` extension.
Cython is a popular tool for writing extension modules for Python. If 
compiles a Python-like language to C, which is then fed to the C compiler.
Historically, Cython was hard to use upstream as a build-time 
dependency. Many projects include pre-generated C files in source 
distribution to avoid users from needing to install the tool.
Cython uses CPython's fast-changing internal API for performance 
reasons. For a new release of Python, Cython generally needs to be 
updated and the C files regenerated. In Fedora, this is frequently 
needed before upstreams release re-generated sources (e.g. for Alpha 
versins of Python).
Since we do not have a problem with build-time dependencies, we always 
want to run the Cython step.
...
XXX example spec snippet
## Tests
### Running tests
If a test suite exists, it **MUST** be run in the `%check` section 
and/or in Fedora CI.
You **MAY** exclude specific failing tests.
You **MUST NOT** disable the entire testsuite or ignore the result to 
solve a build failure.
As an exception, you **MAY** disable tests with an appropriate `%if` 
conditional (e.g. bcond) when 
[bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping).
A popular testing tool, and one which is well integrated in Fedora, is 
`tox`. Upstream, it is commonly used to test against multiple Python 
versions. In a Fedora package, BuildRequire test dependencies (see *Test 
dependencies* below) and run `tox` with:
```
%tox
```
This sets up the environment (`$PATH`, `$PYTHONPATH`, 
`$TOX_TESTENV_PASSENV`) and instructs `tox` to use the current 
environment rather than create new ones.
For more options, see *Build macros* (XXX link to section).
When upstream doesn't use `tox`, the tests need to be run directly 
depending on upstream choice of a test runner. A popular runner is 
`pytest`, which can be run against the package to be installed using:
```
export PATH=%{buildroot}%{_bindir}
export 
PYTHONPATH=%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}
/usr/bin/pytest  # XXX `%{python} -m pytest` doesn't work :( -- sees PWD
```
...
XXX Do we want a macro for that? We do, because not all projects have
tox.
...
Define %__pytest or %_pytest %pytest_command - /usr/bin/pytest
Define %pytest (see above snippet) (not parametric)
Use positional arguments to specify test directory, and `-k` to select 
tests (e.g. `-k "not network"` may deselect all network-related tests).
### Test dependencies
One part of the Python packaging ecosystem that is still not 
standardized is specifying test dependencies (and development 
dependencies in general).
The best practice to specify tests is using an extra (XXX link to 
section above, which should be fleshed out) like `[test]` or `[dev]`. In 
this case, upstream's instructions to install test dependencies might 
look like `pip install -e.[test]`.
Projects using `tox` usually specify test dependencies in a 
`tox`-specific format: a 
[requires](https://tox.readthedocs.io/en/latest/config.html#conf-requires) 
key in the configuration.
Both forms are handled by the `%pyproject_buildrequires` macro, see below.
If upstream does not use either form, list test dependencies as manual 
*BuildRequires* in the `spec` file.
### Linters
In `%check`, packages **SHOULD NOT** run “linters”: code style checkers, 
test coverage checkers and other tools that check code quality rather 
than functionality.
Tools like `black`, `pylint`, `flake8`, or `mypy` are often 
“opinionated” and their “opinions” change frequently enough that they 
are nuisance in Fedora, where the linter is not pinned to an exact version.
Furthermore, some of these tools take a long time to adapt to new Python 
versions, preventing early testing with Aplha and Beta releases of Python.
And they are just not needed: wrongly formatted code is not important 
enough for the Fedora packager to bug the upstream about it.
Making such an issue break a package build is entirely unreasonable.
Linters *do* make sense in upstream CI. But not in Fedora.
If a linter is used, disable it and remove the dependency on it. If that 
is not easy, talk to upstream about making it easy (for example with a 
configuration option or a separate `tox` environment).
For packages that contain such linters, use them at runtime or extend 
them, you will usually need to run the linter in `%check`. Run it to 
test functionality, not code quality of the packaged software.
## Source files from PyPI
Packages **MAY** use sources from PyPI.
However, packages **SHOULD NOT** use an archive that omits test suites, 
licences and/or documentation present in other source archives.
For example, as of this writing `pip` provides a [source tarball 
(“sdist”)](https://pypi.org/project/pip/#files) which omits the 
relatively large `tests` and `docs` directories present in [the source 
on GitHub](https://github.com/pypa/pip). In this case, the tarball from 
GitHub should be used (XXX link to source URL guidelines).
When using sources from PyPI, you can use the `%pypi_source` macro to 
generate the proper URL. See the *Macro reference* (XXX link directly to 
macro) for details.
## Sample spec file
The following is a viable spec file for a hypothetical Python library 
called `pello` that follows packaging best practices.
...
XXX *Need to get `Pello` uploaded as real example and on GH under
fedora-python.
```
%py_set_name    Pello
Name:           python-%{distname}
Version:        1.2.3
Release:        1%{?dist}
Summary:        Example Python library
License:        MIT
URL:            https://github.com/fedora-python/Pello
Source0:        %{pypi_source}
BuildArch:      noarch
BuildRequires:  python3-devel
BuildRequires:  pyproject-rpm-macros
%global _description %{expand:
A python module which provides a convenient example.
This description provides some details.}
%description %_description

%package -n python3-%{distname}
Summary:        %{summary}

%description -n python3-%{distname} %_description
%prep
%autosetup -p1 -n %{projectname}-%{version}
%generate_buildrequires
%pyproject_buildrequires -t
%build
%pyproject_wheel
%install
%pyproject_install
# Here, "pello" is the name of the importable module.
# You may use %%{distname} here if it's the same.
%pyproject_save_files pello
%check
%tox
# Note that there is no %%files section for
# the unversioned python module, python-%%{distname}
%files -n python3-%{distname} -f %{pyproject_files}
%doc README.md
%license LICENSE
%{_bindir}/pello-greeting
```
## Macro Reference
See the *Mandatory macros* section above. (XXX link to section)

* `%{python3}` (`/usr/bin/python3`)
* `%{python3_version}` (e.g. `3.9`)
* `%{python3_version_nodots}` (e.g. `39`)
* `%{python3_sitelib}` (e.g. `/usr/lib/python3.9/site-packages`)
* `%{python3_sitearch}` (e.g. `/usr/lib64/python3.9/site-packages`)
### Name macros
* `%py_set_name PROJECTNAME` (e.g. `%py_set_name Django`)
Sets `%{projectname}` to the given name, and `%{distname}` to the 
canonical version of it. These macros can then be used throughout the 
`spec` file.
   See *Naming* for details and *Sample spec file* for examples. (XXX 
link to sections)
* `%{projectname}` (e.g. `Django`)
Set by `%py_set_name` to the *project name* of the software.
* `%{distname}` (e.g. `django`)
Set by `%py_set_name` to the *canonical project name* of the software.
### Shebang macro
* `%{py3_shebang_flags}` (`s`)
Flags for `%{python3}` to use in shebangs.
   Redefine this macro to use a different set of flags. See *Shebangs* 
[XXX link section] for details.
### Convenience macros
* `%{pypi_source [PROJECTNAME [VERSION [EXT]]]}` (e.g. 
`https://.../Django-3.0.5.tar.gz%60)

   Evaluates to the appropriate URL for source archive hosted on PyPI. 
Accepts up to three optional arguments:
1. The name of the PyPI project. Defaults to `%srcname` if defined, 
or to `%pypi_name` if defined, or to `%name` (the package name). [XXX 
default to `%projectname`]
   2. The version of the PyPI project. Defaults to `%version` (the 
package version).
   3. The file extension to use. Defaults to `tar.gz`.
In most cases it is not necessary to specify any arguments.
* `%{python3_platform}` (e.g. `linux-x86_64`)
The platform name. Used in some Python build systems.
### Build macros
These macros are most useful for packaging Python projects that use the 
`pyproject.toml` file defined in [PEP 
518](https://www.python.org/dev/peps/pep-0518/) and [PEP 
517](https://www.python.org/dev/peps/pep-0517/), which specifies the 
package's build dependencies (including the build system, such as 
setuptools, flit or poetry).
If `pyproject.toml` is not found, the macros automatically fall backs to 
using `setuptools` with configuration in `setup.cfg`/`setup.py`.
A full tutorial and discussion for the macros is available in the 
macros' [README](https://src.fedoraproject.org/rpms/pyproject-rpm-macros/).
* `%pyproject_wheel`
Build the package. Commonly, this is the only macro needed in the 
`%build` section.
This macro needs BuildRequires generated by `%pyproject_buildrequires`.
* `%pyproject_install`
Install the package built by `%pyproject_wheel`.
This macro needs BuildRequires generated by `%pyproject_buildrequires`.
* `%pyproject_buildrequires`
Generate BuildRequires for the package. Used in the 
`%generate_buildrequires` section of the `spec` file. The macro has 
these options:
* `-r`: Include build-time requirements (commonly needed for `%check`).
   * `-x EXTRA`: Include dependencies given by the given *extra* (XXX 
link). [XXX Multiple comma separated values cannot be given yet.]
   * `-t`: Include dependencies for the default *tox* environment. 
Implies `-r`.
   * `-e ENV`: Include dependencies for the given *tox* environment, and 
save the `ENV` name as `%{toxenv}`. Implies `-r`. Multiple comma 
separated values can be given, for example:
%pyproject_buildrequires -e %{toxenv}-unit,%{toxenv}-integration
* `%pyproject_save_files MODNAME …`
Generate a list of files corresponding to the given importable 
modules, and save it as `%{pyproject_files}`.
Note that README and licence files are not included.
    Also, while the macro allows including executable files (using the 
`+bindir` flag), this feature **MUST NOT** be used in Fedora.
The `MODNAME` may be a glob pattern, which should be specific to 
your package. As mentioned in the *Explicit lists* section, expressions 
like `%pyproject_save_files *` are not acceptable.
* `%{pyproject_files}`
Path of the file written by `%pyproject_save_files`, to be used as:
%files -n python3-%{distname} -f %{pyproject_files}
* `%tox`
Run tests using `tox`.
A different environment may be specified with `-e`, for example:
```
    %check
    %tox
    %if %{with integration_test}
    %tox -e %{toxenv}-integration
    %endif
    ```
Flags for the `tox` command can be specified after `--`:
%tox -- --parallel 0
Additional arguments for the test runner may be specified after 
another `--`:
%tox -- --parallel 0 -- --verbose tests/*
* `%{toxenv}`
The *tox* environment used by the `%tox` macro.
    Can be overridden manually or with `%pyproject_buildrequires -t ENV`.
* `%{default_toxenv}` (e.g. `py39`)
The system-wide default value of `%{toxenv}`.
### Manual Generation
The following macros are available for cases where automatic generation 
is turned off.
They can also be useful for handling files in non-standard locations 
where the generators don't look.
* `%pycached MODNAME.py`
Given a Python file, lists the file and the files with its bytecode 
cache. See *Source files and bytecode cache* for more information.
* `%{py_provides python3-MODNAME}`
...
XXX
See *The `%python_provide` macro* for more details.
* `%{py_byte_compile INTERPRETER PATH}`
...
XXX`%{python3_byte_compile PATH}` XXX?
Byte-compile a Python file into a `__pycache__/*.pyc`.
   See [Manual byte 
compilation](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#m...) 
in the Appendix for usage.
* `%{py_dist_name PROJECTNAME}`
Given a *project name* (e.g. `PyYAML`) it will convert it to the 
canonical format (e.g. `pyyaml`). See *Canonical project name* for more 
information.
* `%{py3_dist PROJECTNAME …}`
Given one or more *project names*, it will convert them to the 
canonical format and evaluate to `python3dist(DISTNAME)`, which is 
useful when listing dependencies. See *Machine-readable provides* for 
more information.
### System Settings
The following macros can be redefined for special use cases. Most of 
such cases are unacceptable in Fedora.
* `%{__python}` (`/usr/bin/python`)
Defining this macro changes the meaning of all “unversioned” Python 
macros such as `%{python}` or `%{python_sitelib}`.
   Don’t use these macros without redefining `%{__python}`.
* `%{__python3}` (`/usr/bin/python3`)

   The python 3 interpreter. Redefining this macro changes all the 
`%{python3...}` macros, e.g. `%{python3_sitelib}`.
* `%{python3_pkgversion}` (`3`)
Distro-wide Python version, i.e. the `3` in `python3`.
   Projects that build on top of Fedora may define it to e.g. `3.9` to 
try allowing multiple Python stacks installable in parallel.
### XXX Conditionals?
...
XXX How to properly express: "if python_version >= 3.8"?
The current way is comparing integers from %python3_version_nodots,
but that will break with 3.10/4.0 comparsion.
...
Do a Lua macro that splits the versions and compares them?
This looks more general, is there something in RPM we can use?
### Disabling automation
The following macros can turn off Python-specific automation.
Consider contacting the Python SIG if you need to do this.
* `%{?python_disable_dependency_generator}`
Disables the automatic dependency generator. See *Automatically 
generated dependencies* for details.
* `%__pythonname_provides`
...
XXX undefine to disable %python-provides generator
* `%__pythondist_requires`
...
XXX undefine to disable %python3dist generator
### Deprecated Macros
The following macros are deprecated. See the *201x-era Python Packaging 
guidelines* (XXX link) for how some of them were used.
* `%py3_build`
* `%py3_build_wheel`
* `%py3_build_egg`
* `%py3_install`
* `%py3_install_wheel`
* `%py3_install_egg`
* `%py3dir`
* `%py3_other_build`
* `%py3_other_install`
* `%python_provide` (without `s` at the end)
...
XXX: `%pyX_build` breaks shebang when upstream already uses some
flags (https://bugzilla.redhat.com/show_bug.cgi?id=1335203) -- should we 
document this in the old guidelines?
## Reviewer checklist
...
After the guidelines are done, distill the **MUST**s/**SHOULD**s here.
...
XXX: Do we need a checklist at all?
How do we keep it updated (which hasn't happened in the last N years)