Intent to retire python-blist, but it's "apparently" being used by
python-ujson
by Michel Alexandre Salim
Hi all,
python-blist currently does not build against Python 3.9:
and worryingly, even the patch we use for Python 3.7 compatibility has
not been merged after two years; upstream seems to be totally dead:
https://github.com/DanielStutzbach/blist/pull/78
As such I'm leaning towards retiring this package, except that a check
on what requires it is a bit worrying (ujson requires it, and in turn
many packages depend on ujson).
Interestingly, the upstream ujson repo does not seem to mention blist at
all, so maybe its addition as a build requirement was just a mistake and
we can unentangle the two:
https://github.com/ultrajson/ultrajson
~
❯ sudo repoquery --repo rawhide,rawhide-source --whatrequires python3-blist
Last metadata expiration check: 0:01:33 ago on Mon 11 May 2020 06:12:37
PM PDT.
python-ujson-0:2.0-0.3.20170206git2f1d487.fc32.src
~
❯ sudo repoquery --repo rawhide,rawhide-source --whatrequires python3-ujson
Last metadata expiration check: 0:01:45 ago on Mon 11 May 2020 06:12:37
PM PDT.
buildstream-0:1.4.1-2.fc32.noarch
buildstream-0:1.4.1-2.fc32.src
python-anyjson-0:0.3.3-25.fc32.src
python-gnocchiclient-0:7.0.4-6.fc32.src
python-jsonrpc-server-0:0.3.4-3.fc33.src
python-language-server-0:0.31.10-3.fc33.src
python3-autobahn-0:19.10.1-2.fc32.noarch
python3-gnocchiclient-0:7.0.4-6.fc32.noarch
python3-jsonrpc-server-0:0.3.4-3.fc33.noarch
python3-language-server-0:0.31.10-3.fc33.noarch
python3-pydantic-0:1.3-4.fc32.noarch
--
Michel Alexandre Salim
profile: https://keybase.io/michel_slm
chat via email: https://delta.chat/
GPG key: 96A7 A6ED FB4D 2113 4056 3257 CAF9 AD10 ACB1 BEF2
3 years, 6 months
Renaming pythonXY packages to pythonX.Y (e.g. python39 to python3.9)
by Tomas Orsava
Hello everyone.
I’m working on a change to rename pythonXY packages to pythonX.Y, e.g.
python39 to python3.9.
*Motivation:*
When you install an additional Python interpreter, the command that runs
it contains a dot (e.g. /usr/bin/python3.9) but the package name does
not (e.g. dnf install python39).
The name without a dot is a technical debt that we carry since (at
least) the python26 package in RHEL 5 for consistency. The name with the
dot makes much more sense to Red Hat Python maintainers.
It’s a minor inconsistency, but we'd like to get it right in RHEL, and
since RHEL splits off from Fedora, it will be bugging us still in 2030+
unless we fix it now. Especially with the likely coming version 3.10 the
package name python310 is confusing. This will also get us in sync with
e.g. Debian package names.
*Backwards compatibility:*
We plan to create new components in rawhide containing the dot (e.g.
python3.9) for all Python interpreters (except soon-to-be-retired
python34 and python26) and push the git commits from pythnoXY to them to
preserve the history.
The packages will Provide and Obsolete the old name without a dot (e.g.
python39). The current packages already provide the name with the dot,
so this will be just a switch between real name and virtual provide.
Therefore any package that depends on these components will continue to
work just fine. And in time we’d like to fix all of those to use the new
name, which is backwards compatible, because it is already provided now
(as a side note, the packages are generally just recommended, not required).
*Technical details:*
There has been a recent rawhide-only change to the %python_provide macro
that acts on packages named `python3-foo` and adds for them a new
Provide `python38-foo`.
We’d like to change this to Provide `python3.8-foo` instead.
Since this has been rawhide-only so far, and because there’s no real
reason to depend on these provides yet, we don’t expect any breakage.
What do you think? Do you foresee any problems?
All the best,
Tomas
3 years, 6 months
How to deal with remaining Python 2 packages in F33?
by Felix Schwarz
What is the Fedora policy regarding Python 2 packages in F33?
If there was a Fesco exception for some package last year can we assume that
we can/should keep the package also in F33? I did not find anything about the
assumed scope of these Fesco exceptions.
Specifically this is about bug 1737930:
https://bugzilla.redhat.com/show_bug.cgi?id=1737930#c44
We added a conditional in babel so the Python 2 subpackage is not present
anymore in master/F33. However trac maintainers ask if we could re-add the
Python 2 package for babel in rawhide.
Should I do that?
Non-issues:
- Babel supports Python 2 just fine. I could re-enable/maintain the subpackage
without major issues.
- For some (non-Fedora) $PROJECTS I'm still using Python 2 so I have sympathy
for other who need Python 2 as well. I'm not on a crusade to wipe Python 2
from Fedora.
Felix
3 years, 6 months
Macronize %py3_shebang_fix
by Miro Hrončok
In this change:
https://fedoraproject.org/wiki/Changes/Make_ambiguous_python_shebangs_error
We have advised the following:
pathfix.py -pni "%{__python3} %{py3_shbang_opts}" <paths>
To fix the shebangs.
-p preserves timestamps
-n prevents creating ~backup files
-i specifies the interpreter for the shebang
Now we have new features and we ultimately want this for the best experience:
pathfix.py -pni %{python3} -ka %{py3_shbang_opts_nodash}
-k keeps existing flags
-a adds our flags (if not already there)
%py3_shbang_opts_nodash is derived from %py3_shbang_opts
This is very tedious copypasta.
I propose we macronize this as follows:
We create %py3_shebang_flags. By default, it is the same as %py3_shbang_opts
without dash to avoid confusion, but it can be overridden. Note the proper "e"
in the name.
We create %py3_shebang_fix:
pathfix.py -pni %{__python3} -k%{?py3_shebang_flags:a %py3_shebang_flags}
Usage:
%prep
%autosetup
%py3_shebang_fix .
Or:
%install
...install stuff...
%py3_shebang_fix %{buildroot}%{python3_sitearch} %{buildroot}%{_bindir}/*
As a side not, is it possible to have conditonal expansion not expand on empty
macro?
E.g. this works:
%undefine py3_shebang_flags
%{?py3_shebang_flags:a %py3_shebang_flags} -> nothing
But this does not:
%global py3_shebang_flags %{nil}
%{?py3_shebang_flags:a %py3_shebang_flags} -> a
If not, we'll advise the users to undefine when they want "empty flags".
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 years, 6 months
Macronize %pytest
by Miro Hrončok
Hello Python packagers,
since there is no upstream supported universal test invocation for Python
(`python setup.py test` is deprecated and the de-facto-standard `tox` doesn't
always do what we want in RPM's %check and/or is not always used by upstreams)
and a lot of upstreams use pytest, I'd like to propose a %pytest macros, for
standardizing the way pytest is run.
Usage:
%check
%pytest
Or with options passed to pyets:
%check
%pytest -v -m "not network" tests/
(Here ends the tl;dr version.)
Why not just use `pytest` or `%{python3} -m pytest` instead of `%pytest`?
We want to *test the code we ship*. In the general case this involves:
1. Prepending sys.path with %{buildroot}%{python3_sitelib} (and/or sitearch)
2. Prepending $PATH with %{buildroot}%{_bindir}
3. Ensuring $PWD is not in sys.path
So the idea is (untested):
%pytest_cmd /usr/bin/pytest
%pytest %{expand:
PYTHONPATH="${PYTHONPATH:-%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}}"
\
PATH="%{buildroot}%{_bindir}:$PATH" \
%pytest_cmd
}
Where PYTHONPATH solves (1), PATH solves (2) and using `/usr/bin/pytest` over
`%{python3} -m pytest` kinda solves (3).
I say "kinda" because using `python -m pytest` explicitly adds $PWD to sys.path,
but using `pytest` doesn't always prevent it.
https://docs.pytest.org/en/latest/usage.html#calling-pytest-through-pytho...
I've been trying to completely solve (3) in my head in a general way, but it
always involves `cd`ing to a temporary directory before running the tests, but
plenty of test suites cannot handle that gracefully.
Unfortunately Python interpreter doesn't have a flag that says "don't add
current directory to sys.path, but respect $PYTHONPATH". In the long term, we
might propose to add it.
Where to have the macro defined? (Skip this part if not interested.)
I was thinking:
a) pytest package
b) pytest subpackage (required if rpm-build)
c) python3-rpm-macros
d) python-rpm-macros
e) pyproject-rpm-macros
And here is the pros and cons:
a+) self contained, can change with upstream if needed
a+) always present with pytest
a-) pytest must co-own macro dir or depend on rpm-build
b+) same as a+, but not a-
b-) additional layer of complexity
c+) easy
c+) the macro doesn't need to require pytest, can be used with arbitrary command
c+) can be generalized to non-pytest later if desired
c-) when pytest CLI changes significantly, we need to change it in different
package (but we are not using pytets CLI now at all, just Python/Shell
environment variables)
c~) technically not always present with pytest, but always present with
python3-devel
d+-) same as (c)
d-) the macro uses python3 specific things
e+) the new fancy macros are there
e-) the other macros from there (actually pyproject related) are not required
for this
e-) not always present with pytest nor python3-devel
Hence, I'd go with (c) python3-rpm-macros.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 years, 6 months
Self-introduction
by Fabian Affolter
Hi all,
I don't want to repeat myself, please take a look at my wiki page [1].
Why I want to join? I have one too many Python package and I started to
add the python-sig as admin. But it goes both ways, if the SIG has
access to my packages I want to be part of the SIG. My name is in the
member list [2] for quite a while but I guess that doesn't come with the
right permissions.
Thanks for adding me to python-sig.
If you have any further questions please replay to this message or write
me in private. These days I'm no longer very active on IRC because of my
day job.
Kind regard,
Fabian
[1] https://fedoraproject.org/wiki/User:Fab
[2] https://fedoraproject.org/wiki/SIGs/Python/Members_list
3 years, 6 months
Rawhide python-rpm-generators news: small speedup + %python_provide
not needed in most cases
by Miro Hrončok
Hello Python packagers.
I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in
Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to
older releases.
The python(abi) requirement is now added from a RPM Lua macro, instead of by
executing a Shell script. This is a bit faster especially for packages with a
lot of files. For 10 000 files in one package, the speedup is roughly from ~80
to ~2 seconds (time of the entire package build incl. creating the files from a
Python script). That is a lot, but most of the real packages have much less
files, so I am afraid you won't feel it.
More importantly, %python_provide is not needed for package names.
Until now, we needed to this:
%package -n python3-foo
...
%{?python_provide:%python_provide python3-foo}
This adds automatic provides "python-foo" (and since recently, also
"python38-foo" for forwards/backwards compatibility with EL). This now happens
automagically. Any package called "python3-..." will have those provides added
implicitly by the Python generators when they are in the buildroot
(python3-devel brings those in, so for Python packages this means always).
This automation was possible due to the speedup mentioned above, doing it the
pre-RPM.16 way would be very slow, because the generator is called for every
packaged file (which is still the case now, but it can now be a Lua macro). The
~2 seconds from above include this new feature.
There are 2 cases where manual %python_provide calls are still needed:
1. Metapackages without files
No files => no dependency generator. (I recommend to %ghost the
dist-info/egg-info directory in such packages to workaround this -- we plan to
add more good stuff regarding Python [extras] that will require this anyway.)
2. Virtual provides
The dependency generator can see the (sub)package's name, but not any other
virtual provides. When you do:
%package -n python3-%{pypi_name}
Provides: python3-%{legacy_name} = %{version}-%{release}
You are still supposed to add:
%{?python_provide:%python_provide python3-%{legacy_name}}
(Also note that %python_provide adds obsoletes for older python-foo versions,
that was useful when we renamed everything from python-foo to python2-foo and
when we changed the "default" from python2- to python3-. We are keeping the
Obsoletes in the macro (for now), but I have decided to not automatically
generate the Obsoletes based on the package name. A) I don't consider them
really needed in Fedora 33 any more and B) an idea of implicit obsoletes doesn'T
sound very intriguing to me. This is a decision that may be revisited later if
it's bringing unforeseen trouble.)
You don't need to to actively remove the macro from your spec files, it does no
harm. If you prefer to maintain a single spec, keep it there until Fedora 32
goes EOL (and EPEL 8 if that's your target). The guidelines always recommended
using it the safe way (%{?python_provide:...}), so even if it ceases to exist in
the future, it can remain in specs. There is no plan to remove it in any near or
distant future, as it is still needed for the virtual provides. The generators
also use it internally (for DRY and consistent results).
If you'll get into trouble because of this, let us know and we'll fix it.
I'll make a note for myself to update the Python packaging guidelines.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 years, 7 months
Monthly highlights from the Red Hat's Python-maint team, April 2020
by Tomas Orsava
Hi,
we're the Python-maint team from Red Hat, and we'd like to experiment
with sharing with you what we are working on from month to month. There
are some things we should not disclose (such as embargoed CVEs), but the
majority of what we do is out in public. Please let us know if you find
this useful and whether you'd like us to continue.
So here are the highlights of what we've been working on this April, 2020.
*Red Hat Enterprise Linux:*
* Updating python38-pip to support manylinux2014 in future RHEL8
* Fixing CVE-2020-10108 and CVE-2020-10109 in python-twisted-web on
RHEL 6 and 7
* Fixing CVE-2019-20477 and CVE-2019-20477 in PyYAML on RHEL 8 and
RHSCL 3.5
* Fixing CVE-2020-8492 in python3 on RHEL 7 and 8
* For RHEL9 preparations—see the Fedora section
*Red Hat Software Collections:*
* upcoming rh-python38 collection
* Implementing FIPS support
* Fixing CVE-2020-8492
* Updating python-pip to 19.3.1 in order to support manylinux2014
*Fedora:*
* Updated Python 2 to the ultimate version (2.7.18)
* Python RPM generators were reworked and submitted to rpm upstream
* https://github.com/rpm-software-management/rpm/pull/1195
* The *last* Python 3.8 FTBFS Fedora 32 package (upm) fixed
* Phoronix reported Python is a lot faster in Fedora 32 (most likely
due to --no-semantic-interposition):
* https://www.phoronix.com/scan.php?page=article&item=fedora-32-benchmarks&...
* Automation for the %files section in Python packages
(pyproject-rpm-macros) now exists
* https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/44
* Several small imporvement ideas to make Python packaging easier were
shared
* %py_provide
* https://lists.fedoraproject.org/archives/list/python-devel@lists.fedorapr...
* %pytest
* https://lists.fedoraproject.org/archives/list/python-devel@lists.fedorapr...
* Proposal to rename pythonXY to pythonX.Y was shared
* https://lists.fedoraproject.org/archives/list/python-devel@lists.fedorapr...
* The rebuilds of the newly-released Python 3.9.0a6 have started in
Python 3.9 copr
* https://copr.fedorainfracloud.org/coprs/g/python/python3.9/
* The beta version of pip 20.1 was tested in Fedora and preparations
have started for shipping the final version
* https://src.fedoraproject.org/rpms/python-pip/pull-request/62
* Update to Sphinx 3 is ongoing and testing has started in copr, there
are still packages that block this effort
* Tracking bugzilla:https://bugzilla.redhat.com/show_bug.cgi?id=1783776
* Testing of Cython 3.0 alpha has started in copr
* Proposed a draft of new Python Packaging Guidelines for Fedora
* https://lists.fedoraproject.org/archives/list/python-devel@lists.fedorapr...
*CPython:*
* CVE-2020-8492 fixed, denial of service in urllib HTTP basic
authentication:
* https://bugs.python.org/issue39503
* test_gdb now automatically skips tests when it detects that Python
binary has been optimized:
* https://bugs.python.org/issue40019
* "PEP: Modify the C API to hide implementation details" proposed to
python-dev:
* https://mail.python.org/archives/list/python-dev@python.org/thread/HKM774...
* Victor Stinner's mentee Dong-hee Na is promoted to a core developer!
* antigravity module adjusted for FIPS mode:
* https://github.com/python/cpython/pull/19520
* https://xkcd.com/353/
* https://xkcd.com/426/
* IDLE has a high resolution icon now, to be included in Fedora
* Progress on isolating subinterpreters: the signal handler has been
fixed on Windows.
* Added a RHEL8 x86_64 FIPS buildbot to the upstream CI fleet
* Some aarch64 upstream builders provided by python-main are now
officially on the stable list, meaning we can say now that upstream
officially supports aarch64 on Fedora, RHEL7 and RHEL8
* https://github.com/python/buildmaster-config/pull/186
*Misc:*
* pyp2rpm development/maintenance has been successfully transferred to
the community
* Delivered an open-source talk at Red Hat Czech Open House
All the best,
Python-maint
3 years, 7 months
Package Naming Question
by Ian McInerney
I am working on packaging up a git tool that is written in Python
(git-revise: https://github.com/mystor/git-revise), and was wondering if
the "python3-%{name}" applies to tools like this, e.g. ones that are
written in Python but are designed for use on their own instead of as an
importable module (since this creates a /usr/bin/git-revise launcher).
Thanks,
-Ian
3 years, 7 months