May 2020 - python-devel - Fedora mailing-lists

Draft of New Python Packaging Guidelines
by Petr Viktorin 15 Jun '21

15 Jun '21

Hello! Below is a draft of new Packaging Guidelines! It's full of unfinished areas (marked with XXX), but it's ready for scrutiny. A possibly updated version is on https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg?view Generally, for rules marked **SHOULD** we know of cases where they should be broken; for things marked **MUST** we don't. We have tried to only take the Good Existing Things™ (macros, practices) and revise the rest without thinking about the past much. Some used technology is unfortunately not compatible with current EPELs, but we have considered Fedora 31+. Using the current Python guidelines will still be an option for people who target EPEL 6/7/8. The main controversial idea (of many) is synchronizing Fedora's names (a.k.a. `python3dist(...)`, a.k.a. `name` in `setup.py`) with the Python Package Index (PyPI, pypi.org) which has by now become the de-facto canonical namespace for freely redistributable Python packages. We believe that this will help both Fedora and the greater Python ecosystem, but there will definitely be some growing pains. Most of Fedora Python packages already are on PyPI, but more than 250 are missing. There is software developed within Fedora (e.g. pagure, fedpkg, rust2rpm); projects that aren't primarily Python packages (e.g. perf, ceph, xen, setroubleshoot, iotop); and others. A full list is attached. The names have been temporarily blocked on PyPI to keep trolls from taking them while this is discussed. Over at the Python Discourse we are discussing how to properly handle these; once that discussion is settled they should be unblocked: https://discuss.python.org/t/pypi-as-a-project-repository-vs-name-registry-… Another general idea is that package metadata should be kept upstream; the spec file should duplicate as little of it as possible. Any adjustments should be done with upstreamable patches. The draft lives on hackmd.io, which we found easy to collaborate with. If you have an account there, we can add you. If you'd like to collaborate some other way, let us know. Petr and Miro ----------------------- Current draft for inline comments: > [IMPORTANT] > This is a DRAFT; it is not in effect yet. # Python Packaging Guidelines (draft) > [IMPORTANT] > This is a *beta* version of the Python Packaging Guidelines and the associated RPM macros. > Packagers that opt in to following this version **MUST** be prepared to change their packages frequently when the guidelines or macros are updated. > These packagers **SHOULD** join [Python SIG mailing list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedo… and monitor/start conversations there. These Guidelines represent a major rewrite and paradigm shift, and not all packages are updated to reflect this. Older guidelines are still being kept up to date, and existing packages **MAY** use them instead of this document: * 201x-era Python packaging guidelines (for packages that use e.g. `%py3_install` or `setup.py install`) * Python 2 appendix (for e.g. `%py2_install`) (requires FESCo exception) > [NOTE] > These guidelines only support Fedora 31+. For older releases (such as in EPEL 8), consult the older guidelines (XXX link). The two "Distro-wide guidelines" below apply to all software in Fedora that uses Python at build- or run-time. These rest of the Guidelines apply to packages that ship code that *can* be imported in Python. Specifically, that is all packages that install files under `/usr/lib*/python*/`. Except for the two "Distro-wide guidelines", these Guidelines do not apply to simple one-file scripts or utilities, especially if these are included with software not written in Python. However, if an application (e.g. CLI tool, script or GUI app) needs a more complex Python library, the library **SHOULD** be packaged as an importable library under these guidelines. A major goal for Python packaging in Fedora is to *harmonize with the wider Python ecosystem*, that is, the [Python Packaging Authority](https://pypa.io) (PyPA) standards and the [Python Package Index](https://pypi.org) (PyPI). Packagers **SHOULD** be prepared to get involved with upstream projects to establish best practices as outlined here. We wish to improve both Fedora and the wider Python ecosystem. > [NOTE] > Fedora's Python SIG not only develops these guidelines, but it's also involved in PyPA standards and Python packaging best practices. Check out [the wiki](https://fedoraproject.org/wiki/SIGs/Python) or [mailing list](https://lists.fedoraproject.org/archives/list/python-devel@lists.fedo… if you need help or wish to help out. ## Distro-wide guidelines ### BuildRequire python3-devel **Every** package that uses Python (at runtime and/or build time), and/or installs Python modules **MUST** explicitly include `BuildRequires: python3-devel` in its `.spec` file, even if Python is not actually invoked during build time. If the package uses an alternate Python interpreter instead of `python3` (e.g. `pypy`, `jython`, `python27`), it **MAY** instead require the coresponding `*-devel` package. The `*-devel` package brings in relevant RPM macros. It may also enable automated or manual checks: for example, Python maintainers use this requirement to list packages that use Python in some way and might be affected by planned changes. ### Mandatory macros The following macros **MUST** be used where applicable. The expansions in parentheses are provided only as reference/examples. The macros are defined for you in all supported Fedora and EPEL versions. * `%{python3}` (`/usr/bin/python3`) The Python interpreter. For example, this macro should be used for invoking Python from a `spec` file script, passed to `configure` scripts to select a Python executable, or used as `%{python3} -m pytest` to run a Python-based tool. > XXX: Use shebang opts for shebang; document pathfix macro. Document cases where you don't want this. * `%{python3_version}` (e.g. `3.9`) Version of the Python interpreter. * `%{python3_version_nodots}` (e.g. `39`) Version of the Python interpreter without the dot. * `%{python3_sitelib}` (e.g. `/usr/lib/python3.9/site-packages`) Where pure-Python modules are installed. * `%{python3_sitearch}` (e.g. `/usr/lib64/python3.9/site-packages`) Where Python3 extension modules (native code, e.g. compiled from C) are installed. The rest of this document uses these macros, along with `%{_bindir}` (`/usr/bin/`), instead of the raw path names. ## Python implementation support Fedora primarily targets *CPython*, the reference implementation of the Python language. We generally use “Python” to mean CPython. Alternate implementations like `pypy` are available, but currently lack comprehensive tooling and guidelines. When targetting these, there are no hard rules (except the general Fedora packaging guidelines). But please try to abide by the *spirit* of these guidelines. When in doubt, consider consulting the Python SIG. ## Python version support Fedora packages **MUST NOT** depend on other versions of the CPython interpreter than the current `python3`. In Fedora, Python libraries are packaged for a single version of Python, called `python3`. For example, in Fedora 32, `python3` is Python 3.8. In the past, there were multiple Python stacks, e.g. `python3.7` and `python2.7`, installable together on the same machine. That is also the case in some projects that build *on top* of Fedora, like RHEL, EPEL and CentOS. Fedora might re-introduce parallell-installable stacks in the future (for example if a switch to a new Python version needs a transition period, or if enough interested maintainers somehow appear). > XXX dots in package names! Fedora does include alternate interpreter versions, e.g. `python2.7` or `python3.5`, but these are meant only for developers that need to test upstream code. Bug and security fixes for these interpreters only cover this use case. Packages such as `pip` or `tox`, which enable setting up isolated environments and installing third-party packages into them, **MAY**, as an exception to the rule above, use these interpreters as long as this is coordinated with the maintainers of the relevant Python interpreter. ## BuildRequire pyproject-rpm-macros While these guidelines are in Beta, each Python package **MUST** have `BuildRequires: pyproject-rpm-macros` to access the beta macros. (When we go out of Beta, listing the dependency won't be necessary: we'll make `python3-devel` require it.) ## Naming Python packages have several different names, which should be kept in sync but will sometimes differ for historical or practical reasons. They are: * the Fedora *source package name* (or *component name*, `%{name}`), * the Fedora *built RPM name*, * the *project name* used on *PyPI* or by *pip*, and * the *importable module name* used in Python (a single package may have multiple importable modules). Some examples (both good and worse): | Fedora component | Built RPM | Project name | Importable module | | ----------------- | ------------------ | -------------- | -------------------- | | `python-requests` | `python3-requests` | `requests` | `requests` | | `PyYAML` | `python3-pyyaml` | `pyyaml` | `yaml` | | `python-ldap` | `python3-ldap` | `python-ldap` | `ldap`, `ldif`, etc. | | `python-pillow` | `python3-pillow` | `pillow` | `PIL` | Elsewhere in this text, the metavariables `SRPMNAME`, `RPMNAME`, `PROJECTNAME`, `MODNAME` refer to these names, respectively. ### Canonical project name Most of these names are case-sensitive machine-friendly identifiers, but the *project name* has human-friendly semantics: it is case-insensitive and treats some sets of characters (like `._-`) specially. For automated use, it needs to be [normalized](https://www.python.org/dev/peps/pep-0503/#normalized-names) to a canonical format used by Python tools and services such as setuptools, pip and PyPI. The `%{py_dist_name}` macro implements this normalization. Elsewhere in this text, the metavariable `DISTNAME` refers to the canonical form of the project name. > XXX > ```spec > # XXX specfile > %py_set_name Django > -> %distname django > -> %pojectname Django > ``` > > XXX rewrite: It is customary to define the macro `%{distname}` as the canonical project name and use it throughout the `spec` file. Some helper macros use `%{distname}` for this purpose by default. > > XXX in some places, the "original project name" must be used -- e.g. `%pypi_source` and `%autosetup` need `Django`, not `django`. > XXX The following sections should supersede the [Python section on the Naming guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Namin…. ### Library naming A built (i.e. non-SRPM) package for a *Python library* **MUST** be named with the `python3-` prefix. A source package containing primarily a *Python library* **MUST** be named with the prefix `python-`. The Fedora package's name **SHOULD** contain the *canonical project name*. If possible, the project name **SHOULD** be the same as the name of the main importable module, with underscores (`_`) replaced by dashes (`-`). A *Python library* is a package meant to be imported in Python, such as with `import requests`. Tools like *Ansible* or *IDLE*, whose code is importable but not primarily meant to be imported, are not considered libraries in this sense, so this section does not apply for them. (See the [general Libraries and Applications guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/#_lib… for details.) If the importable module name and the project name do not match, users frequently end up confused. In this case, packagers *should* ensure that upstream is aware of the problem and (especially for new packages where renaming is feasible) strive to get the package renamed. The Python SIG is available for assistance. The Fedora package name should be formed by taking the *project name* and prepending `python-` if it does not already start with `python-`. This may leads to conflicts (e.g. between [bugzilla](https://pypi.org/project/bugzilla/) and [python-bugzilla](https://pypi.org/project/python-bugzilla/)) In that case, ensure upstream is aware of the potentially confusing naming and apply best judgment. ### Application naming Packages that primarily provide executables **SHOULD** be named according to the general Fedora guidelines (e.g. `ansible`). Consider adding a virtual provide according to Library naming above (e.g. `python3-PROJECTNAME`), if it would help users find the package. ## Files to include ### Source files and bytecode cache Packages **MUST** include the source file (`*.py`) **AND** the bytecode cache (`*.pyc`) for each pure-Python module. The cache files are found in a `__pycache__` directory and have an interpreter-dependent suffix like `.cpython-39.pyc`. The cache is not necessary to run the software, but if it is not found, Python will try to create it. If this succeeds, the file is not tracked by RPM and it will linger on the system after uninstallation. If it does not succeed, users can get spurious SELinux AVC denials in the logs. Normally, byte compilation (generating the cache files) is done for you by the `brp-python-bytecompile` script (XXX link to BRP guidelines), which runs automatically after the `%install` section of the spec file has been processed. It byte compiles any `.py` files that it finds in `%{python3_sitelib}` or `%{python3_sitearch}`. You must include these files of your package (i.e. in the `%files` section). If the code is in a subdirectory (importable package), include the entire directory: ``` %files %{python3_sitelib}/foo/ ``` Adding the trailing slash is best practice for directories. However, this cannot be used for top-level scripts (those directly in e.g. `%{python3_sitelib}`), because both `%{python3_sitelib}` and `%{python3_sitelib}/__pycache__/` are owned by Python itself. Here, the `%pycached` macro can help. It expands to the given `*.py` source file and its corresponding cache file(s). For example: ``` %files %pycached %{python3_sitelib}/foo.py ``` expands roughly to: ``` %files %{python3_sitelib}/foo.py %{python3_sitelib}/__pycache__/foo.cpython-3X{,.opt-?}.pyc ``` #### Manual byte compilation If you need to bytecompile stuff outside of `%{python3_sitelib}`/`%{python3_sitearch}`, use the `%py_byte_compile` macro. For more details on `%py_byte_compile` and on the internals of bytecode compilation, please see the [appendix](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_…. > XXX: Copy the section from appendix here ### Dist-info metadata Each Python package **MUST** include *Package Distribution Metadata* conforming to [PyPA specifications](https://packaging.python.org/specifications/) (specifically, [Recording installed distributons](https://packaging.python.org/specifications/recording-install…) This applies to libraries (e.g. `python-requests`) as well as tools (e.g. `ansible`). > XXX what with splitting into subpackages? 1) dist-info always installed, 2) dist-info installed trough a metapackage? > * Ideally, do the split upstream > * Consider package split between library & tool (see poetry, fedpkg) > > e.g. > When software is split into several subpackages, it is OK to only ship metadata in one built RPM. The metadata takes the form of a `dist-info` directory installed in `%{python3_sitelib}` or `%{python3_sitearch}`, and contains information that tools like [`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html) use to introspect installed libraries. > XXX example %files with manual dist-info entry Note that some older tools instead put metadata in an `egg-info` directory, or even a single file. This won't happen if you use the `%pyproject_wheel` macro. If your package uses a build system that generates an `egg-info` directory or file, please contact Python SIG. > XXX We need a better solution before we go out of beta. As an exception, the Python standard library **MAY** ship without this metadata. ### Explicit lists Packagers **SHOULD NOT** simply glob everything under a shared directory. In particular, the following **SHOULD NOT** be used: * `%{python3_sitelib}/*` * `%{python3_sitearch}/*` * `%{python_sitelib}/*` * `%{python_sitearch}/*` * `%{_bindir}/*` * `%pyproject_save_files *` * `%pyproject_save_files +bindir` This rule serves as a check against common mistakes which are otherwise hard to detect. It does limit some possibilities for automation. The most common mistake this rule prevents is installing a test suite system-wide as an importable module named `test`, which would then conflict with other such packages. ## PyPI parity Every Python package in Fedora **SHOULD** also be available on [the Python Package Index](https://pypi.org) (PyPI). The command `pip install PROJECTNAME` **MUST** install the same package (possibly in a different version), install nothing, or fail with a reasonable error message. If this is not the case, the packager **MUST** contact upstream about this. The goal is to get the project name registered or blocked on PyPI, or to otherwise ensure the rule is followed. > XXX Note that project names that were in Fedora but not on PyPI when these guidelines were proposed are *blocked* as we discuss how they should be handled. > This prevents potential trolls, but also legitimate owners, from taking them. This is necessary to protect users, avoid confusion, and enable automation as Fedora and upstream ecosystems grow more integrated. As always, specific exceptions can be granted by FPC (XXX link exceptions rules). > XXX Write an automated check for this. ## Provides and requirements ### Provides for importable modules For any module intended to be used in Python 3 with `import MODNAME`, the package that includes it **SHOULD** provide `python3-MODNAME`, with underscores (`_`) replaced by dashes (`-`). This is of course always the case if the package is named `python3-MODNAME`. If the subpackage has some other name, then add `%py_provides python3-MODNAME` explicitly. See the following section to learn about `%py_provides`. ### Automatic unversioned provides For any `FOO`, a package that provides `python3-FOO` **SHOULD** use `%py_provides` or an automatic generator to also provide `python-FOO`. The provide **SHOULD NOT** be added manually: if the generator or macro is not used, the provide shall not be added at all. On Fedora 33+, this is done automatically for package names by a generator. The generator can be disabled by undefining `%__pythonname_provides`. The following macro invocation will provide both `python3-FOO` and `python-FOO`: %py_provides python3-FOO > XXX: finalize `%py_provides` Using the generator or macro is important, because the specific form of the provide may change in the future. ### Machine-readable provides Every Python package **MUST** provide `python3dist(DISTNAME)` **and** `python3.Xdist(DISTNAME)`, where `X` is the minor version of the interpreter and `DISTNAME` is the *canonical project name* corresponding to the *dist-info metadata*, for example `python3.9dist(requests)`. (XXX: add links to previous sections) This is generated automatically from the dist-info metadata. The provide **SHOULD NOT** be added manually: if the generator fails to add it, the metadata **MUST** be fixed. If necessary, the automatic generator can be disabled by undefining `%__pythondist_provides`. These *Provides* are used for automatically generated *Requires*. ### Dependencies As mentioned above, each Python package **MUST** explicitly BuildRequire `python3-devel`. Packages **MUST NOT** have dependencies (either build-time or runtime) with the unversioned prefix `python-` if the corresponding `python3-` dependency can be used instead. Packages **SHOULD NOT** have explicit dependencies (either build-time or runtime) with a minor-version prefix such as `python3.8-` or `python3.8dist(`. Such dependencies **SHOULD** instead be automatically generated or a macro should be used to get the version. Packages **SHOULD NOT** have an explicit runtime dependency on `python3`. Instead of depending on `python3`, packges have an automatic dependency on `python(abi) = 3.X` when they install files to `%{python3_sitelib}` or `%{python3_sitearch}`, or they have an automatic dependency on `/usr/bin/python3` if they have executable Python scripts, or they have an automatic dependency on `libpython3.X.so.1.0()` if they embed Python. These rules help ensure a smooth upgrade path when `python3` is updated in new versions of Fedora. ### Automatically generated dependencies Packages **MUST** use the automatic Python run-time dependency generator. Packages **SHOULD** use the opt-in build-dependency generator if possible. Any necessary changes **MUST** be done by patches or modifying the source (e.g. with `sed`), rather than disabling the generator. The resulting change **SHOULD** be offered to upstream. As an exception, [filtering](https://docs.fedoraproject.org/en-US/packaging-guidelines/AutoPr… **MAY** be used for temporary workarounds and bootstrapping. Dependencies covered by the generators **SHOULD NOT** be repeated in the `.spec` file. (For example, if the generator finds a `requests` dependency, then `Requires: python3-requests` is redundant.) The automatically generated requirements are in the form `python3.Xdist(DISTNAME)`, potentially augmented with version requirements or combined together with [rich dependencies](https://rpm.org/user_doc/boolean_dependencies.html). Note that the generators only cover Python packages. Other dependencies, often C libraries like `openssl-devel`, must be specified in the `.spec` file manually. > XXX When implemented, this goes here: Alternatively, upstream Python packages may list non-Python dependencies in the `[tool.fedora.requires]`/`[tool.fedora.buildrequires]` section of `pyproject.toml`. This is non-standard and only recommended for Fedora-related packages. Where the requirements are specified in the source depends on each project's build system and preferences. Common locations are `pyproject.toml`, `setup.py`, `setup.cfg`, `config.toml`. #### Run-time dependency generator The automatic runtime dependency generator uses package metadata (as recorded in installed `*.dist-info` directories) to determine what the package depends on. In an emergency, you can opt-out from running the requires generator by adding `%{?python_disable_dependency_generator}` to the package (usually, just before the main package’s `%description`). #### Build-time dependency generator The opt-in (but strongly recommended) build-time dependency generator gathers information from [`pyproject.toml` build-system information](https://www.python.org/dev/peps/pep-0517/#source-trees) (with fallback to `setuptools`) plus a standardized [build-system hook](https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-whee… to gather further requirements. See `%pyproject_buildrequires` (XXX link) for more details. ### Test dependencies See the *Tests* section. (XXX link.) ### Extras > XXX No story here so far. > XXX Note: [python-django-storages](https://src.fedoraproject.org/rpms/python-django-st…, `drf-yasg` etc. do `%python_provide python3-%{srcname}+%{distname}` and `python3dist(%{srcname}/%{extraname})` XXX `python3dist(%{srcname}[%{extraname}])` ## Interpreter invocation ### Shebangs Shebangs lines to invoke Python **SHOULD** be `#!%{python3} -%{py3_shebang_flags}` and it **MAY** include extra flags. > XXX define py3_shebang_flags If the default flags from `%{py3_shebang_flags}` are not desirable, packages **SHOULD** explicitly redefine the macro to remove them. Using `#!%{python3}` (`#!/usr/bin/python3`) rather than e.g. `#!/usr/bin/env python` ensures that the system-wide Python interpreter is used to run the code, even if the user modifies `$PATH` (e.g. by activating a virtual environment). By default, `-%{py3_shebang_flags}` expands to `-s`, which means *don't add user site directory to `sys.path`*. That ensures user-installed Python packages (e.g. by `pip install --user`) don't interfere with the RPM installed software. Sometimes, `pip`-installed content is desirable, such as with plugins. Redefining `%{py3_shebang_flags}` to not include `s`, rather than not using the macro at all, ensures that existing or future automation won't add the flag. The `%pyproject_install` macro automatically changes all Python shebangs in `%{buildroot}%{_bindir}/*` to use `%{python3}` and add `%{py3_shebang_flags}` to the existing flags. If you're not using the macro or you need to change a shebang in a different directory, you can use the `pathfix.py` command as follows: ``` pathfix.py -n -p -k -i %{python3} -a %{py3_shebang_flags} SCRIPTNAME … ``` > XXX Ouch! Macroize this? Hell yes! > > `%py3_shebang_fix <paths>` -- also rename `%{py3_shebang_flags}` (doesn't include dash) > > can be used in `%prep` or `%install` where: * `-n` disables ceating backups * `-p` preserves timestamps * `-k` keeps existing flags * `-i` specifies the new interpreter ### Invokable Python modules Every executable `TOOL` for which the current version of Python matters **SHOULD** also be invokable by `python3 -m TOOL`. If the software doesn't provide this functionality, packagers **SHOULD** ask the upstream to add it. This applies to tools that modify the current Python environment (like installing or querying packages), use Python for configuration, or use Python to run plugins. It does not apply to tools like GIMP or Bash which support plugins in multiple languages and/or have other means to specify the interpreter. For example, `pip` can be invoked as `python3 -m pip`. This allows users to accurately specify the Python version used to run the software. This convention works across different environments that might not always set `$PATH` or install scripts consistently. ## Using Cython Tightening the general Fedora policy, packages **MUST NOT** use files pre-generated by Cython. These **MUST** be deleted in `%prep` and regenerated during the build. As an exception, these sources **MAY** be used temporarily to prevent build time circular dependencies by following the [bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#b… guidelines. Generated files (the ones that must be deleted) have a generic `.c` or `.cpp` extension. Cython source files (which should stay) usually have the `.pyx` or `.pxd` extension. Cython is a popular tool for writing extension modules for Python. If compiles a Python-like language to C, which is then fed to the C compiler. Historically, Cython was hard to use upstream as a build-time dependency. Many projects include pre-generated C files in source distribution to avoid users from needing to install the tool. Cython uses CPython's fast-changing internal API for performance reasons. For a new release of Python, Cython generally needs to be updated and the C files regenerated. In Fedora, this is frequently needed before upstreams release re-generated sources (e.g. for Alpha versins of Python). Since we do not have a problem with build-time dependencies, we always want to run the Cython step. > XXX example spec snippet ## Tests ### Running tests If a test suite exists, it **MUST** be run in the `%check` section and/or in Fedora CI. You **MAY** exclude specific failing tests. You **MUST NOT** disable the entire testsuite or ignore the result to solve a build failure. As an exception, you **MAY** disable tests with an appropriate `%if` conditional (e.g. bcond) when [bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#b…. A popular testing tool, and one which is well integrated in Fedora, is `tox`. Upstream, it is commonly used to test against multiple Python versions. In a Fedora package, BuildRequire test dependencies (see *Test dependencies* below) and run `tox` with: ``` %tox ``` This sets up the environment (`$PATH`, `$PYTHONPATH`, `$TOX_TESTENV_PASSENV`) and instructs `tox` to use the current environment rather than create new ones. For more options, see *Build macros* (XXX link to section). When upstream doesn't use `tox`, the tests need to be run directly depending on upstream choice of a test runner. A popular runner is `pytest`, which can be run against the package to be installed using: ``` export PATH=%{buildroot}%{_bindir} export PYTHONPATH=%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib} /usr/bin/pytest # XXX `%{python} -m pytest` doesn't work :( -- sees PWD ``` > XXX Do we want a macro for that? We do, because not all projects have tox. > > Define %__pytest or %_pytest %pytest_command - /usr/bin/pytest > > Define %pytest (see above snippet) (not parametric) Use positional arguments to specify test directory, and `-k` to select tests (e.g. `-k "not network"` may deselect all network-related tests). ### Test dependencies One part of the Python packaging ecosystem that is still not standardized is specifying test dependencies (and development dependencies in general). The best practice to specify tests is using an extra (XXX link to section above, which should be fleshed out) like `[test]` or `[dev]`. In this case, upstream's instructions to install test dependencies might look like `pip install -e.[test]`. Projects using `tox` usually specify test dependencies in a `tox`-specific format: a [requires](https://tox.readthedocs.io/en/latest/config.html#conf-requires) key in the configuration. Both forms are handled by the `%pyproject_buildrequires` macro, see below. If upstream does not use either form, list test dependencies as manual *BuildRequires* in the `spec` file. ### Linters In `%check`, packages **SHOULD NOT** run “linters”: code style checkers, test coverage checkers and other tools that check code quality rather than functionality. Tools like `black`, `pylint`, `flake8`, or `mypy` are often “opinionated” and their “opinions” change frequently enough that they are nuisance in Fedora, where the linter is not pinned to an exact version. Furthermore, some of these tools take a long time to adapt to new Python versions, preventing early testing with Aplha and Beta releases of Python. And they are just not needed: wrongly formatted code is not important enough for the Fedora packager to bug the upstream about it. Making such an issue break a package build is entirely unreasonable. Linters *do* make sense in upstream CI. But not in Fedora. If a linter is used, disable it and remove the dependency on it. If that is not easy, talk to upstream about making it easy (for example with a configuration option or a separate `tox` environment). For packages that contain such linters, use them at runtime or extend them, you will usually need to run the linter in `%check`. Run it to test functionality, not code quality of the packaged software. ## Source files from PyPI Packages **MAY** use sources from PyPI. However, packages **SHOULD NOT** use an archive that omits test suites, licences and/or documentation present in other source archives. For example, as of this writing `pip` provides a [source tarball (“sdist”)](https://pypi.org/project/pip/#files) which omits the relatively large `tests` and `docs` directories present in [the source on GitHub](https://github.com/pypa/pip). In this case, the tarball from GitHub should be used (XXX link to source URL guidelines). When using sources from PyPI, you can use the `%pypi_source` macro to generate the proper URL. See the *Macro reference* (XXX link directly to macro) for details. ## Sample spec file The following is a viable spec file for a hypothetical Python library called `pello` that follows packaging best practices. > XXX *Need to get `Pello` uploaded as real example and on GH under fedora-python. ``` %py_set_name Pello Name: python-%{distname} Version: 1.2.3 Release: 1%{?dist} Summary: Example Python library License: MIT URL: https://github.com/fedora-python/Pello Source0: %{pypi_source} BuildArch: noarch BuildRequires: python3-devel BuildRequires: pyproject-rpm-macros %global _description %{expand: A python module which provides a convenient example. This description provides some details.} %description %_description %package -n python3-%{distname} Summary: %{summary} %description -n python3-%{distname} %_description %prep %autosetup -p1 -n %{projectname}-%{version} %generate_buildrequires %pyproject_buildrequires -t %build %pyproject_wheel %install %pyproject_install # Here, "pello" is the name of the importable module. # You may use %%{distname} here if it's the same. %pyproject_save_files pello %check %tox # Note that there is no %%files section for # the unversioned python module, python-%%{distname} %files -n python3-%{distname} -f %{pyproject_files} %doc README.md %license LICENSE %{_bindir}/pello-greeting ``` ## Macro Reference See the *Mandatory macros* section above. (XXX link to section)  * `%{python3}` (`/usr/bin/python3`) * `%{python3_version}` (e.g. `3.9`) * `%{python3_version_nodots}` (e.g. `39`) * `%{python3_sitelib}` (e.g. `/usr/lib/python3.9/site-packages`) * `%{python3_sitearch}` (e.g. `/usr/lib64/python3.9/site-packages`) ### Name macros * `%py_set_name PROJECTNAME` (e.g. `%py_set_name Django`) Sets `%{projectname}` to the given name, and `%{distname}` to the canonical version of it. These macros can then be used throughout the `spec` file. See *Naming* for details and *Sample spec file* for examples. (XXX link to sections) * `%{projectname}` (e.g. `Django`) Set by `%py_set_name` to the *project name* of the software. * `%{distname}` (e.g. `django`) Set by `%py_set_name` to the *canonical project name* of the software. ### Shebang macro * `%{py3_shebang_flags}` (`s`) Flags for `%{python3}` to use in shebangs. Redefine this macro to use a different set of flags. See *Shebangs* [XXX link section] for details. ### Convenience macros * `%{pypi_source [PROJECTNAME [VERSION [EXT]]]}` (e.g. `https://.../Django-3.0.5.tar.gz`) Evaluates to the appropriate URL for source archive hosted on PyPI. Accepts up to three optional arguments: 1. The name of the PyPI project. Defaults to `%srcname` if defined, or to `%pypi_name` if defined, or to `%name` (the package name). [XXX default to `%projectname`] 2. The version of the PyPI project. Defaults to `%version` (the package version). 3. The file extension to use. Defaults to `tar.gz`. In most cases it is not necessary to specify any arguments. * `%{python3_platform}` (e.g. `linux-x86_64`) The platform name. Used in some Python build systems. ### Build macros These macros are most useful for packaging Python projects that use the `pyproject.toml` file defined in [PEP 518](https://www.python.org/dev/peps/pep-0518/) and [PEP 517](https://www.python.org/dev/peps/pep-0517/), which specifies the package's build dependencies (including the build system, such as setuptools, flit or poetry). If `pyproject.toml` is not found, the macros automatically fall backs to using `setuptools` with configuration in `setup.cfg`/`setup.py`. A full tutorial and discussion for the macros is available in the macros' [README](https://src.fedoraproject.org/rpms/pyproject-rpm-macros/). * `%pyproject_wheel` Build the package. Commonly, this is the only macro needed in the `%build` section. This macro needs BuildRequires generated by `%pyproject_buildrequires`. * `%pyproject_install` Install the package built by `%pyproject_wheel`. This macro needs BuildRequires generated by `%pyproject_buildrequires`. * `%pyproject_buildrequires` Generate BuildRequires for the package. Used in the `%generate_buildrequires` section of the `spec` file. The macro has these options: * `-r`: Include build-time requirements (commonly needed for `%check`). * `-x EXTRA`: Include dependencies given by the given *extra* (XXX link). [XXX Multiple comma separated values cannot be given yet.] * `-t`: Include dependencies for the default *tox* environment. Implies `-r`. * `-e ENV`: Include dependencies for the given *tox* environment, and save the `ENV` name as `%{toxenv}`. Implies `-r`. Multiple comma separated values can be given, for example: %pyproject_buildrequires -e %{toxenv}-unit,%{toxenv}-integration * `%pyproject_save_files MODNAME …` Generate a list of files corresponding to the given importable modules, and save it as `%{pyproject_files}`. Note that README and licence files are not included. Also, while the macro allows including executable files (using the `+bindir` flag), this feature **MUST NOT** be used in Fedora. The `MODNAME` may be a glob pattern, which should be specific to your package. As mentioned in the *Explicit lists* section, expressions like `%pyproject_save_files *` are not acceptable. * `%{pyproject_files}` Path of the file written by `%pyproject_save_files`, to be used as: %files -n python3-%{distname} -f %{pyproject_files} * `%tox` Run tests using `tox`. A different environment may be specified with `-e`, for example: ``` %check %tox %if %{with integration_test} %tox -e %{toxenv}-integration %endif ``` Flags for the `tox` command can be specified after `--`: %tox -- --parallel 0 Additional arguments for the test runner may be specified after another `--`: %tox -- --parallel 0 -- --verbose tests/* * `%{toxenv}` The *tox* environment used by the `%tox` macro. Can be overridden manually or with `%pyproject_buildrequires -t ENV`. * `%{default_toxenv}` (e.g. `py39`) The system-wide default value of `%{toxenv}`. ### Manual Generation The following macros are available for cases where automatic generation is turned off. They can also be useful for handling files in non-standard locations where the generators don't look. * `%pycached MODNAME.py` Given a Python file, lists the file and the files with its bytecode cache. See *Source files and bytecode cache* for more information. * `%{py_provides python3-MODNAME}` > XXX See *The `%python_provide` macro* for more details. * `%{py_byte_compile INTERPRETER PATH}` > XXX`%{python3_byte_compile PATH}` XXX? Byte-compile a Python file into a `__pycache__/*.pyc`. See [Manual byte compilation](https://docs.fedoraproject.org/en-US/packaging-guidelines/Pyth… in the Appendix for usage. * `%{py_dist_name PROJECTNAME}` Given a *project name* (e.g. `PyYAML`) it will convert it to the canonical format (e.g. `pyyaml`). See *Canonical project name* for more information. * `%{py3_dist PROJECTNAME …}` Given one or more *project names*, it will convert them to the canonical format and evaluate to `python3dist(DISTNAME)`, which is useful when listing dependencies. See *Machine-readable provides* for more information. ### System Settings The following macros can be redefined for special use cases. Most of such cases are unacceptable in Fedora. * `%{__python}` (`/usr/bin/python`) Defining this macro changes the meaning of all “unversioned” Python macros such as `%{python}` or `%{python_sitelib}`. Don’t use these macros without redefining `%{__python}`. * `%{__python3}` (`/usr/bin/python3`) The python 3 interpreter. Redefining this macro changes all the `%{python3...}` macros, e.g. `%{python3_sitelib}`. * `%{python3_pkgversion}` (`3`) Distro-wide Python version, i.e. the `3` in `python3`. Projects that build on top of Fedora may define it to e.g. `3.9` to try allowing multiple Python stacks installable in parallel. ### XXX Conditionals? > XXX How to properly express: "if python_version >= 3.8"? > The current way is comparing integers from %python3_version_nodots, but that will break with 3.10/4.0 comparsion. > Do a Lua macro that splits the versions and compares them? > This looks more general, is there something in RPM we can use? ### Disabling automation The following macros can turn off Python-specific automation. Consider contacting the Python SIG if you need to do this. * `%{?python_disable_dependency_generator}` Disables the automatic dependency generator. See *Automatically generated dependencies* for details. * `%__pythonname_provides` > XXX undefine to disable %python-provides generator * `%__pythondist_requires` > XXX undefine to disable %python3dist generator ### Deprecated Macros The following macros are deprecated. See the *201x-era Python Packaging guidelines* (XXX link) for how some of them were used. * `%py3_build` * `%py3_build_wheel` * `%py3_build_egg` * `%py3_install` * `%py3_install_wheel` * `%py3_install_egg` * `%py3dir` * `%py3_other_build` * `%py3_other_install` * `%python_provide` (without `s` at the end) > XXX: `%pyX_build` breaks shebang when upstream already uses some flags (https://bugzilla.redhat.com/show_bug.cgi?id=1335203) -- should we document this in the old guidelines? ## Reviewer checklist > After the guidelines are done, distill the **MUST**s/**SHOULD**s here. > XXX: Do we need a checklist at all? > How do we keep it updated (which hasn't happened in the last N years)

8 40

Brand new pipenv 2020.5.28 ready for testing in a copr repo
by Tomas Orsava 29 Jun '20

29 Jun '20

Hi, upstream released a brand new version of `pipenv` this morning, and we've packaged it for you to try out in copr: https://copr.fedorainfracloud.org/coprs/g/python/pipenv/ Please report any issues in Fedora's Bugzilla and in the text please indicate that it's regarding this new version. All the best, Tomas Orsava

1 1

Python 3.9 (not so) mini mass rebuilds and when they will/can happen
by Miro Hrončok 11 Jun '20

11 Jun '20

Hello. Kevin and Smooge, you are CCed in case you have some concerns regarding the data centre move. Jitka and Zuzka, you are CCed because you plan mini mass rebuilds of your own (Perl and Node.js). Ben, you are CCed because I considered it a good idea. Python 3.9.0b1 was released earlier this week and we are running test rebuilds in Copr. Things are going suspiciously well so far, except for some critical stuff -- but we have that covered for now. Hopefully nobody decides to upgrade something major right now :P If everything goes extremely well, I expect us to start rebuilding the initial bootstrap sequence (438 srpms) in Koji side tag this week, the remaining packages (2733) the next week. We expect some packages to fail (~200-300) and merge the side tag, so it's not open too long. We can help firefight things if this blocks anybody. There are possible followup rebuilds in play for two particular reasons: 1) The ABI can change between beta releases of Python and all arched Python packages (661) will need a rebuild. This can happen (but doesn't have to) at: - beta 2: 2020-06-08+ - beta 3: 2020-06-29+ - beta 4: 2020-07-20+ Source: https://www.python.org/dev/peps/pep-0596/#schedule Anytime during the release cycle, the format (magic number) of bytecode cache (.pyc files) can change. The problems with that are not rawhide-critical, but we don't want to ship wrong bytecode cache in Fedora 33 GA. Hence we'll check the numbers prior beta freeze (2020-08-25) and rebuild what's needed (can be anything between 0 and 3000). -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 1

PSA: pytest 5 now finally available in rawhide
by Miro Hrončok 31 May '20

31 May '20

Hello, the pytest package (python3-pytest) has been updated to 5.4.2. In case it gives you trouble, you can pin an older version: BuildRrequires: %{py3_dist pytest} < 5 Or: BuildRrequires: python3dist(pytest) < 5 That will give you the python3-pytest4 package (4.6.10 now), not parallely installable with python3-pytest. Note that the python3-pytest4 package is deprecated, but no removal date has been set. Likely, it will get orphaned or retired when no important packages need it. The pytest 4.6.x series is still being occasionally released, but: https://docs.pytest.org/en/latest/py27-py34-deprecation.html#maintenance-of… "From now on, the core team will no longer actively backport patches, but the 4.6.x branch will continue to exist so the community itself can contribute patches." Side note: The Fedora 31/32 python3-pytest package does not provide python3-pytest4, hence I recommend not to BuildRequire python3-pytest4 directly by name, but using python3dist(pytest) or %py3_dist as in the examples above. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 3

Help needed with Fedora Python Classroom Lab container image
by Miro Hrončok 27 May '20

27 May '20

Hello. There is a docker/podman container image part of the Fedora Python Classroom Lab: https://labs.fedoraproject.org/en/python-classroom I need help with two main issues: 1) The container doesn't built for Fedora 32: https://koji.fedoraproject.org/koji/taskinfo?taskID=45001647 But I get an error I don't understand (dnf exited with error, but I see no error message). 2) I don't know how to get the previous images (namely Fedora 31) from the registry. Apparently they have all ever just been in the candidate registry and they have been all deleted... ? https://pagure.io/releng/issue/9473 An actual user asked about the missing images, hence I don't want to stop producing them, OTOH I clearly have no idea what am I doing. If somebody is interested in participating, this would be really appreciated. If not, I think we should remove the thing from Python Classroom's website. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 3

Re: Help needed with Fedora Python Classroom Lab container image
by Clement Verna 27 May '20

27 May '20

On Wed, 27 May 2020 at 02:28, Adam Williamson <adamwill(a)fedoraproject.org> wrote: > On Wed, 2020-05-27 at 01:20 +0200, Miro Hrončok wrote: > > Hello. > > > > There is a docker/podman container image part of the Fedora Python > Classroom Lab: > > > > https://labs.fedoraproject.org/en/python-classroom > > > > I need help with two main issues: > > > > > > 1) The container doesn't built for Fedora 32: > > > > https://koji.fedoraproject.org/koji/taskinfo?taskID=45001647 > > > > But I get an error I don't understand (dnf exited with error, but I see > no error > > message). > > Errors seem to be in x86_64.log: > > 2020-05-26 09:59:39,727 - atomic_reactor.util - DEBUG - Step 6/11 : RUN > dnf -y --setopt=install_weak_deps=false --disablerepo rawhide-modular > install "@python-classroom" nano openssh-clients vim-enhanced wget man > && dnf clean all > 2020-05-26 09:59:40,900 - atomic_reactor.util - DEBUG - ---> Running in > 75ba910cc4cc > 2020-05-26 09:59:42,917 - atomic_reactor.util - DEBUG - [91mNo repository > match: rawhide-modular > 2020-05-26 09:59:42,917 - atomic_reactor.util - DEBUG - [0m > 2020-05-26 09:59:43,110 - atomic_reactor.util - DEBUG - > atomic-reactor-koji-plugin-f32-container-candid 4.1 kB/s | 413 B 00:00 > 2020-05-26 09:59:43,289 - atomic_reactor.util - DEBUG - Fedora 32 openh264 > (From Cisco) - x86_64 34 kB/s | 5.1 kB 00:00 > 2020-05-26 09:59:43,542 - atomic_reactor.util - DEBUG - Fedora Modular 32 > - x86_64 20 MB/s | 4.9 MB 00:00 > 2020-05-26 09:59:44,455 - atomic_reactor.util - DEBUG - Fedora Modular 32 > - x86_64 - Updates 6.4 MB/s | 1.6 MB 00:00 > 2020-05-26 10:03:19,281 - atomic_reactor.util - DEBUG - Fedora 32 - x86_64 > - Updates 0.0 B/s | 0 B 03:34 > 2020-05-26 10:03:19,282 - atomic_reactor.util - DEBUG - [91mErrors during > downloading metadata for repository 'updates': > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > http://mirrors.dotsrc.org/fedora-buffet/fedora/linux/updates/32/Everything/… > [Failed to connect to mirrors.dotsrc.org port 80: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > https://mirror.lzu.edu.cn/fedora/updates/32/Everything/x86_64/repodata/repo… > [Failed to connect to mirror.lzu.edu.cn port 443: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > http://mirrors.ircam.fr/pub/fedora/linux/updates/32/Everything/x86_64/repod… > [Failed to connect to mirrors.ircam.fr port 80: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > http://fedora.cu.be/linux/updates/32/Everything/x86_64/repodata/repomd.xml > [Failed to connect to fedora.cu.be port 80: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > http://mirror.math.princeton.edu/pub/fedora/linux/updates/32/Everything/x86… > [Failed to connect to mirror.math.princeton.edu port 80: Connection > refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > http://mirror.metrocast.net/fedora/linux/updates/32/Everything/x86_64/repod… > [Failed to connect to mirror.metrocast.net port 80: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > http://mirrors.chroot.ro/fedora/linux/updates/32/Everything/x86_64/repodata… > [Failed to connect to mirrors.chroot.ro port 80: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > http://mirror.init7.net/fedora/fedora/linux/updates/32/Everything/x86_64/re… > [Failed to connect to mirror.init7.net port 80: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > http://download-ib01.fedoraproject.org/pub/fedora/linux/updates/32/Everythi… > [Failed to connect to download-ib01.fedoraproject.org port 80: Connection > refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > https://pubmirror1.math.uh.edu/fedora-buffet/fedora/linux/updates/32/Everyt… > [Failed to connect to pubmirror1.math.uh.edu port 443: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > https://fedora.mirror.liteserver.nl/linux/updates/32/Everything/x86_64/repo… > [Failed to connect to fedora.mirror.liteserver.nl port 443: Connection > refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > https://mirror.netcologne.de/fedora/linux/updates/32/Everything/x86_64/repo… > [Failed to connect to mirror.netcologne.de port 443: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > https://mirrors.lug.mtu.edu/fedora/linux/updates/32/Everything/x86_64/repod… > [Failed to connect to mirrors.lug.mtu.edu port 443: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > http://mirrors.lug.mtu.edu/fedora/linux/updates/32/Everything/x86_64/repoda… > [Failed to connect to mirrors.lug.mtu.edu port 80: Connection refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > http://ftp-stud.hs-esslingen.de/pub/fedora/linux/updates/32/Everything/x86_… > [Failed to connect to ftp-stud.hs-esslingen.de port 80: Connection > refused] > 2020-05-26 10:03:19,283 - atomic_reactor.util - DEBUG - - Curl error (7): > Couldn't connect to server for > https://mirror.init7.net/fedora/fedora/linux/updates/32/Everything/x86_64/r… > [Failed to connect to mirror.init7.net port 443: Connection refused] > > etc. etc. > > At first glance you'd think it's a network issue, but note it *did* > successfully retrieve metadata for several other repos first. What I'd > guess is that it works so long as it hits up a mirror within infra, but > it seems to not be doing that and instead trying to connect to external > mirrors, which presumably it's not allowed to route out to. The > question I guess would be why it's getting an infra mirror for the > other repos but apparently not for the updates repo. Not sure why that > would be, off the top of my head. > So this happen if you are unlucky enough to try to build a container image while the update repo is generated. It seems to happen quite often recently (https://pagure.io/releng/issue/9435) but I honestly not sure how we can improve the situation here :(. > -- > Adam Williamson > Fedora QA Community Monkey > IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net > http://www.happyassassin.net > _______________________________________________ > devel mailing list -- devel(a)lists.fedoraproject.org > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org >

1 0

Redesigning the %python_provide macro from scratch
by Miro Hrončok 26 May '20

26 May '20

Hello Python packagers. After touching the %python_provide topic in: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproj… I have realized several things I don't like about %python_provide: 1) It must be used conditionally (it is not defined in python-srpm-macros). That means you always wrap it in %{?python_provide:...} in order to have a "valid" specfile even when the macro is not yet defined (e.g. during SRPM creation in Koji or on a packager's machine without python-rpm-macros installed). 2) You cannot use it with arbitrary versions. Suppose you package python3-foo 1.0 but you want to provide python3-bar 2.0 for some reason -- this is not very common, but it happens. %python_provide only takes the "name" as an argument, always using %{?epoch} %{version} and %{release}. 3) You need to both add a virtual provide + use the macro. Suppose you want to provide python3-pkg_resources from python3-setuptools. Currently, you need to add: Provides: python3-pkg_resources = %{version}-%{release} %{?python_provide:%python_provide python3-pkg_resources} 4) When used with (sub)package name, it generates a duplicate dependency on Fedora 33+ (and an rpmlint error). 5) It produces Obsoletes, but that might no longer be necessary nor desired. 6) Broken expectations about %_isa. It used to add %_isa provides based on wrong data, it no longer does that (except on old releases and EPELs), can be used manually with name%{?_isa}, but not on the old releases. 7) Undocuemnted error handling (e.g. the macro expands to nothing when used with pypy-foo, but errors when used with foo). Hence, I was thinking (for the sake of backwards compatibility) to provide a new mechanism to do this and preserve the old macro as is, deprecating it in Fedora 36-ish, actually maybe removing it once RHEL 9 goes EOL (or never, which is basically the same from today's perspective). The new macro should solve the problems from above, my current (quick, untested) ideas are: 1) Define the macro in python-srpm-macros. No need to use it conditionally. We can backport it to EPEL 8 and define a "stub" macro in EPEL 7 and 6. (An if we start using the macro only after Fedora 30 goes EOL, we can make the macro behave consistently across all Fedora versions.) 2) Accept version identifier as an optional argument, use %{?epoch} %{version} and %{release} as default. (See for example %{pypi_source} on how this can be done.) 3) Make the macro also produce the provide for the given name and document that. E.g. when you call it with python3-pkg_resources, it also provides python3-pkg_resources (not only python-pkg_resources etc.). 4) Make it so that for given arguments, the macro will only expand to something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed. 5) No obsoletes with the new macro. Packagers use manual obsoletes when desired. 6) Document clearly that there is no %{?_isa} (and there is no "backwards compatibility" load to carry). When absolutely desired, packagers can call the macro with %{name}%{?_isa}. 7) Support arbitrary names. Only provide the given name and nothing else if not "recognized". As a bonus, I think the current if-elif-elif-elif-elif code can be replaced with lua patterns (imagine regex). As always, this leaves us with the name problem, but I'd very much like to use %python_provides (note the s). The only problem I see is that it is likely to be mistaken for the old one, but IMHO it shouldn't really hurt that much. Usage example: %package -n python3-setuptools %python_provides python3-pkg_resources Resulting provides: python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 Another example: %package -n python3-pillow %python_provides python3-PIL 1.1.6-100 Resulting provides: python3-pillow = 7.1.1-1.fc33 python-pillow = 7.1.1-1.fc33 python39-pillow = 7.1.1-1.fc33 python3-PIL = 1.1.6-100 python-PIL = 1.1.6-100 python39-PIL = 1.1.6-100 Dummy when used with package name: %package -n python3-pip %python_provides python3-pip Provides: python3-pip = 20.0.2-1.fc33 python-pip = 20.0.2-1.fc33 python39-pip = 20.0.2-1.fc33 (all of them only once) Any name is OK: %python_provides wroom 666 Provides: wroom = 666 What do you think? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

4 9

Re: Python 3.9 (not so) mini mass rebuilds and when they will/can happen
by Miro Hrončok 22 May '20

22 May '20

On 21. 05. 20 19:26, Kevin Fenzi wrote: >>> If everything goes extremely well, I expect us to start rebuilding the >>> initial >>> bootstrap sequence (438 srpms) in Koji side tag this week, the remaining >>> packages (2733) the next week. We expect some packages to fail (~200-300) >>> and >>> merge the side tag, so it's not open too long. We can help firefight >>> things if >>> this blocks anybody. > > I'm a bit worried about the kojira issues you were seeing, but you > should now be able to do newrepos to unblock there right? I'd appreciate > it if you could gather info if you see it being slow to make newrepos > so we can get the koji folks to fix it. :) I have the newrepo powers now, yes. I am also gathering the wait times, so I can dump them in the issue when done. >>> There are possible followup rebuilds in play for two particular reasons: >>> >>> 1) The ABI can change between beta releases of Python and all arched >>> Python >>> packages (661) will need a rebuild. >>> >>> This can happen (but doesn't have to) at: >>> >>> - beta 2: 2020-06-08+ >>> - beta 3: 2020-06-29+ >> >> I am going to say that anything during this time frame is going to be touch >> and go. We have not gotten the new builders up yet and I do not know what >> our build capacity will be. >> >> When we are saying builds will be slow, I have been assuming that we are >> going to be running at 1/8th to 1/4 normal capacity. [AKA if you think >> things are slow now.. multiply by 4 or 8]. However Kevin deals with the >> builders a lot more so may have a better estimate. I would say that your >> best bet for all of these would be in COPR during that time frame to get >> timely results and then schedule rebuilds based on that. > > Yeah, so the week of 2020-06-08 is going to be our migration week. > There WILL be outages of things as we move services to the new > datacenter. Do you have any more idea of _when_ in that week things > might happen? If it's early in the week, I can try and move koji later > in the week and you can hopefully be done. If it's later in the week, > you are going to be going to the new, lower capacity koji. Ideally this > would be a week to avoid any important builds if you can. ;) 2020-06-08 is the target release date of Python 3.9.0b2. From experience, the actual release will likely happen a bit later, so not early in the week. We can hold the update until you give us a signal. If beta1 won't be a "rawhide is on fire" situation and beta2 actually breaks the ABI, we can run test rebuilds in copr and wait for beta3 with the actual Fedora update. Will coordinate with you when that happens. > The week of the 29th should also as smooge notes be lower capacity. > It should churn through the builds fine, but it will take longer. Longer is no problem. This can be done within couple hours normally (excluding some outliers such as libreoffice and ceph), so even with lower capacity shouldn't be a disaster. >>> - beta 4: 2020-07-20+ >>> >> >> I am expecting we will be back to normal capacity around this time. Thanks for the info. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Can we revert numpy 1.19?
by Miro Hrončok 21 May '20

21 May '20

Hello Gwyn, we are about to start rebuilding in Koji for Python 3.9 this or next week. In Copr, we built NumPy 1.19 rc1 from dist git and it causes some friction. I wonder whether you'd be OK to revert the commits and wait until the rebuilt is done? The new version was not built in rawhide yet. Thanks, -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 3

Working with epel7 on Fedora
by Scott Talbert 18 May '20

18 May '20

Any good workarounds for working with epel7 Python packages on Fedora? Mainly dealing with the fact that python3_pkgversion differs from Fedora to EPEL7. Ie, when doing 'fedpkg mockbuild' the SRPM will be generated with the wrong python3_pkgversion. I tried this, but it didn't seem to work: fedpkg --release epel7 srpm --define 'python3_pkgversion 36' Thanks, Scott

3 3

Intent to retire python-blist, but it's "apparently" being used by python-ujson
by Michel Alexandre Salim 12 May '20

12 May '20

Hi all, python-blist currently does not build against Python 3.9: and worryingly, even the patch we use for Python 3.7 compatibility has not been merged after two years; upstream seems to be totally dead: https://github.com/DanielStutzbach/blist/pull/78 As such I'm leaning towards retiring this package, except that a check on what requires it is a bit worrying (ujson requires it, and in turn many packages depend on ujson). Interestingly, the upstream ujson repo does not seem to mention blist at all, so maybe its addition as a build requirement was just a mistake and we can unentangle the two: https://github.com/ultrajson/ultrajson ~ ❯ sudo repoquery --repo rawhide,rawhide-source --whatrequires python3-blist Last metadata expiration check: 0:01:33 ago on Mon 11 May 2020 06:12:37 PM PDT. python-ujson-0:2.0-0.3.20170206git2f1d487.fc32.src ~ ❯ sudo repoquery --repo rawhide,rawhide-source --whatrequires python3-ujson Last metadata expiration check: 0:01:45 ago on Mon 11 May 2020 06:12:37 PM PDT. buildstream-0:1.4.1-2.fc32.noarch buildstream-0:1.4.1-2.fc32.src python-anyjson-0:0.3.3-25.fc32.src python-gnocchiclient-0:7.0.4-6.fc32.src python-jsonrpc-server-0:0.3.4-3.fc33.src python-language-server-0:0.31.10-3.fc33.src python3-autobahn-0:19.10.1-2.fc32.noarch python3-gnocchiclient-0:7.0.4-6.fc32.noarch python3-jsonrpc-server-0:0.3.4-3.fc33.noarch python3-language-server-0:0.31.10-3.fc33.noarch python3-pydantic-0:1.3-4.fc32.noarch -- Michel Alexandre Salim profile: https://keybase.io/michel_slm chat via email: https://delta.chat/ GPG key: 96A7 A6ED FB4D 2113 4056 3257 CAF9 AD10 ACB1 BEF2

2 3

Renaming pythonXY packages to pythonX.Y (e.g. python39 to python3.9)
by Tomas Orsava 08 May '20

08 May '20

Hello everyone. I’m working on a change to rename pythonXY packages to pythonX.Y, e.g. python39 to python3.9. *Motivation:* When you install an additional Python interpreter, the command that runs it contains a dot (e.g. /usr/bin/python3.9) but the package name does not (e.g. dnf install python39). The name without a dot is a technical debt that we carry since (at least) the python26 package in RHEL 5 for consistency. The name with the dot makes much more sense to Red Hat Python maintainers. It’s a minor inconsistency, but we'd like to get it right in RHEL, and since RHEL splits off from Fedora, it will be bugging us still in 2030+ unless we fix it now. Especially with the likely coming version 3.10 the package name python310 is confusing. This will also get us in sync with e.g. Debian package names. *Backwards compatibility:* We plan to create new components in rawhide containing the dot (e.g. python3.9) for all Python interpreters (except soon-to-be-retired python34 and python26) and push the git commits from pythnoXY to them to preserve the history. The packages will Provide and Obsolete the old name without a dot (e.g. python39). The current packages already provide the name with the dot, so this will be just a switch between real name and virtual provide. Therefore any package that depends on these components will continue to work just fine. And in time we’d like to fix all of those to use the new name, which is backwards compatible, because it is already provided now (as a side note, the packages are generally just recommended, not required). *Technical details:* There has been a recent rawhide-only change to the %python_provide macro that acts on packages named `python3-foo` and adds for them a new Provide `python38-foo`. We’d like to change this to Provide `python3.8-foo` instead. Since this has been rawhide-only so far, and because there’s no real reason to depend on these provides yet, we don’t expect any breakage. What do you think? Do you foresee any problems? All the best, Tomas

3 3

How to deal with remaining Python 2 packages in F33?
by Felix Schwarz 08 May '20

08 May '20

What is the Fedora policy regarding Python 2 packages in F33? If there was a Fesco exception for some package last year can we assume that we can/should keep the package also in F33? I did not find anything about the assumed scope of these Fesco exceptions. Specifically this is about bug 1737930: https://bugzilla.redhat.com/show_bug.cgi?id=1737930#c44 We added a conditional in babel so the Python 2 subpackage is not present anymore in master/F33. However trac maintainers ask if we could re-add the Python 2 package for babel in rawhide. Should I do that? Non-issues: - Babel supports Python 2 just fine. I could re-enable/maintain the subpackage without major issues. - For some (non-Fedora) $PROJECTS I'm still using Python 2 so I have sympathy for other who need Python 2 as well. I'm not on a crusade to wipe Python 2 from Fedora. Felix

2 1

Macronize %py3_shebang_fix
by Miro Hrončok 07 May '20

07 May '20

In this change: https://fedoraproject.org/wiki/Changes/Make_ambiguous_python_shebangs_error We have advised the following: pathfix.py -pni "%{__python3} %{py3_shbang_opts}" <paths> To fix the shebangs. -p preserves timestamps -n prevents creating ~backup files -i specifies the interpreter for the shebang Now we have new features and we ultimately want this for the best experience: pathfix.py -pni %{python3} -ka %{py3_shbang_opts_nodash} -k keeps existing flags -a adds our flags (if not already there) %py3_shbang_opts_nodash is derived from %py3_shbang_opts This is very tedious copypasta. I propose we macronize this as follows: We create %py3_shebang_flags. By default, it is the same as %py3_shbang_opts without dash to avoid confusion, but it can be overridden. Note the proper "e" in the name. We create %py3_shebang_fix: pathfix.py -pni %{__python3} -k%{?py3_shebang_flags:a %py3_shebang_flags} Usage: %prep %autosetup %py3_shebang_fix . Or: %install ...install stuff... %py3_shebang_fix %{buildroot}%{python3_sitearch} %{buildroot}%{_bindir}/* As a side not, is it possible to have conditonal expansion not expand on empty macro? E.g. this works: %undefine py3_shebang_flags %{?py3_shebang_flags:a %py3_shebang_flags} -> nothing But this does not: %global py3_shebang_flags %{nil} %{?py3_shebang_flags:a %py3_shebang_flags} -> a If not, we'll advise the users to undefine when they want "empty flags". -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 1

Macronize %pytest
by Miro Hrončok 06 May '20

06 May '20

Hello Python packagers, since there is no upstream supported universal test invocation for Python (`python setup.py test` is deprecated and the de-facto-standard `tox` doesn't always do what we want in RPM's %check and/or is not always used by upstreams) and a lot of upstreams use pytest, I'd like to propose a %pytest macros, for standardizing the way pytest is run. Usage: %check %pytest Or with options passed to pyets: %check %pytest -v -m "not network" tests/ (Here ends the tl;dr version.) Why not just use `pytest` or `%{python3} -m pytest` instead of `%pytest`? We want to *test the code we ship*. In the general case this involves: 1. Prepending sys.path with %{buildroot}%{python3_sitelib} (and/or sitearch) 2. Prepending $PATH with %{buildroot}%{_bindir} 3. Ensuring $PWD is not in sys.path So the idea is (untested): %pytest_cmd /usr/bin/pytest %pytest %{expand: PYTHONPATH="${PYTHONPATH:-%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}}" \ PATH="%{buildroot}%{_bindir}:$PATH" \ %pytest_cmd } Where PYTHONPATH solves (1), PATH solves (2) and using `/usr/bin/pytest` over `%{python3} -m pytest` kinda solves (3). I say "kinda" because using `python -m pytest` explicitly adds $PWD to sys.path, but using `pytest` doesn't always prevent it. https://docs.pytest.org/en/latest/usage.html#calling-pytest-through-python-… I've been trying to completely solve (3) in my head in a general way, but it always involves `cd`ing to a temporary directory before running the tests, but plenty of test suites cannot handle that gracefully. Unfortunately Python interpreter doesn't have a flag that says "don't add current directory to sys.path, but respect $PYTHONPATH". In the long term, we might propose to add it. Where to have the macro defined? (Skip this part if not interested.) I was thinking: a) pytest package b) pytest subpackage (required if rpm-build) c) python3-rpm-macros d) python-rpm-macros e) pyproject-rpm-macros And here is the pros and cons: a+) self contained, can change with upstream if needed a+) always present with pytest a-) pytest must co-own macro dir or depend on rpm-build b+) same as a+, but not a- b-) additional layer of complexity c+) easy c+) the macro doesn't need to require pytest, can be used with arbitrary command c+) can be generalized to non-pytest later if desired c-) when pytest CLI changes significantly, we need to change it in different package (but we are not using pytets CLI now at all, just Python/Shell environment variables) c~) technically not always present with pytest, but always present with python3-devel d+-) same as (c) d-) the macro uses python3 specific things e+) the new fancy macros are there e-) the other macros from there (actually pyproject related) are not required for this e-) not always present with pytest nor python3-devel Hence, I'd go with (c) python3-rpm-macros. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 3

Newbie introduction to Python SIG
by Adrian Adrian 06 May '20

06 May '20

3 2

Self-introduction
by Fabian Affolter 06 May '20

06 May '20

Hi all, I don't want to repeat myself, please take a look at my wiki page [1]. Why I want to join? I have one too many Python package and I started to add the python-sig as admin. But it goes both ways, if the SIG has access to my packages I want to be part of the SIG. My name is in the member list [2] for quite a while but I guess that doesn't come with the right permissions. Thanks for adding me to python-sig. If you have any further questions please replay to this message or write me in private. These days I'm no longer very active on IRC because of my day job. Kind regard, Fabian [1] https://fedoraproject.org/wiki/User:Fab [2] https://fedoraproject.org/wiki/SIGs/Python/Members_list

3 3

Rawhide python-rpm-generators news: small speedup + %python_provide not needed in most cases
by Miro Hrončok 06 May '20

06 May '20

Hello Python packagers. I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to older releases. The python(abi) requirement is now added from a RPM Lua macro, instead of by executing a Shell script. This is a bit faster especially for packages with a lot of files. For 10 000 files in one package, the speedup is roughly from ~80 to ~2 seconds (time of the entire package build incl. creating the files from a Python script). That is a lot, but most of the real packages have much less files, so I am afraid you won't feel it. More importantly, %python_provide is not needed for package names. Until now, we needed to this: %package -n python3-foo ... %{?python_provide:%python_provide python3-foo} This adds automatic provides "python-foo" (and since recently, also "python38-foo" for forwards/backwards compatibility with EL). This now happens automagically. Any package called "python3-..." will have those provides added implicitly by the Python generators when they are in the buildroot (python3-devel brings those in, so for Python packages this means always). This automation was possible due to the speedup mentioned above, doing it the pre-RPM.16 way would be very slow, because the generator is called for every packaged file (which is still the case now, but it can now be a Lua macro). The ~2 seconds from above include this new feature. There are 2 cases where manual %python_provide calls are still needed: 1. Metapackages without files No files => no dependency generator. (I recommend to %ghost the dist-info/egg-info directory in such packages to workaround this -- we plan to add more good stuff regarding Python [extras] that will require this anyway.) 2. Virtual provides The dependency generator can see the (sub)package's name, but not any other virtual provides. When you do: %package -n python3-%{pypi_name} Provides: python3-%{legacy_name} = %{version}-%{release} You are still supposed to add: %{?python_provide:%python_provide python3-%{legacy_name}} (Also note that %python_provide adds obsoletes for older python-foo versions, that was useful when we renamed everything from python-foo to python2-foo and when we changed the "default" from python2- to python3-. We are keeping the Obsoletes in the macro (for now), but I have decided to not automatically generate the Obsoletes based on the package name. A) I don't consider them really needed in Fedora 33 any more and B) an idea of implicit obsoletes doesn'T sound very intriguing to me. This is a decision that may be revisited later if it's bringing unforeseen trouble.) You don't need to to actively remove the macro from your spec files, it does no harm. If you prefer to maintain a single spec, keep it there until Fedora 32 goes EOL (and EPEL 8 if that's your target). The guidelines always recommended using it the safe way (%{?python_provide:...}), so even if it ceases to exist in the future, it can remain in specs. There is no plan to remove it in any near or distant future, as it is still needed for the virtual provides. The generators also use it internally (for DRY and consistent results). If you'll get into trouble because of this, let us know and we'll fix it. I'll make a note for myself to update the Python packaging guidelines. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

4 10

Monthly highlights from the Red Hat's Python-maint team, April 2020
by Tomas Orsava 04 May '20

04 May '20

Hi, we're the Python-maint team from Red Hat, and we'd like to experiment with sharing with you what we are working on from month to month. There are some things we should not disclose (such as embargoed CVEs), but the majority of what we do is out in public. Please let us know if you find this useful and whether you'd like us to continue. So here are the highlights of what we've been working on this April, 2020. *Red Hat Enterprise Linux:* * Updating python38-pip to support manylinux2014 in future RHEL8 * Fixing CVE-2020-10108 and CVE-2020-10109 in python-twisted-web on RHEL 6 and 7 * Fixing CVE-2019-20477 and CVE-2019-20477 in PyYAML on RHEL 8 and RHSCL 3.5 * Fixing CVE-2020-8492 in python3 on RHEL 7 and 8 * For RHEL9 preparations—see the Fedora section *Red Hat Software Collections:* * upcoming rh-python38 collection * Implementing FIPS support * Fixing CVE-2020-8492 * Updating python-pip to 19.3.1 in order to support manylinux2014 *Fedora:* * Updated Python 2 to the ultimate version (2.7.18) * Python RPM generators were reworked and submitted to rpm upstream * https://github.com/rpm-software-management/rpm/pull/1195 * The *last* Python 3.8 FTBFS Fedora 32 package (upm) fixed * Phoronix reported Python is a lot faster in Fedora 32 (most likely due to --no-semantic-interposition): * https://www.phoronix.com/scan.php?page=article&item=fedora-32-benchmarks&nu… * Automation for the %files section in Python packages (pyproject-rpm-macros) now exists * https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/44 * Several small imporvement ideas to make Python packaging easier were shared * %py_provide * https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproj… * %pytest * https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproj… * Proposal to rename pythonXY to pythonX.Y was shared * https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproj… * The rebuilds of the newly-released Python 3.9.0a6 have started in Python 3.9 copr * https://copr.fedorainfracloud.org/coprs/g/python/python3.9/ * The beta version of pip 20.1 was tested in Fedora and preparations have started for shipping the final version * https://src.fedoraproject.org/rpms/python-pip/pull-request/62 * Update to Sphinx 3 is ongoing and testing has started in copr, there are still packages that block this effort * Tracking bugzilla:https://bugzilla.redhat.com/show_bug.cgi?id=1783776 * Testing of Cython 3.0 alpha has started in copr * Proposed a draft of new Python Packaging Guidelines for Fedora * https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproj… *CPython:* * CVE-2020-8492 fixed, denial of service in urllib HTTP basic authentication: * https://bugs.python.org/issue39503 * test_gdb now automatically skips tests when it detects that Python binary has been optimized: * https://bugs.python.org/issue40019 * "PEP: Modify the C API to hide implementation details" proposed to python-dev: * https://mail.python.org/archives/list/python-dev@python.org/thread/HKM774XK… * Victor Stinner's mentee Dong-hee Na is promoted to a core developer! * antigravity module adjusted for FIPS mode: * https://github.com/python/cpython/pull/19520 * https://xkcd.com/353/ * https://xkcd.com/426/ * IDLE has a high resolution icon now, to be included in Fedora * Progress on isolating subinterpreters: the signal handler has been fixed on Windows. * Added a RHEL8 x86_64 FIPS buildbot to the upstream CI fleet * Some aarch64 upstream builders provided by python-main are now officially on the stable list, meaning we can say now that upstream officially supports aarch64 on Fedora, RHEL7 and RHEL8 * https://github.com/python/buildmaster-config/pull/186 *Misc:* * pyp2rpm development/maintenance has been successfully transferred to the community * Delivered an open-source talk at Red Hat Czech Open House All the best, Python-maint

1 0

Package Naming Question
by Ian McInerney 01 May '20

01 May '20

I am working on packaging up a git tool that is written in Python (git-revise: https://github.com/mystor/git-revise) and was wondering if the "python3-%{name}" applies to tools like this, e.g. ones that are written in Python but are designed for use on their own instead of as an importable module (since this creates a /usr/bin/git-revise launcher). Thanks, -Ian

3 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel May 2020