Rawhide python-rpm-generators news: small speedup + %python_provide not needed in most cases
Hello Python packagers.
I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to older releases.
The python(abi) requirement is now added from a RPM Lua macro, instead of by executing a Shell script. This is a bit faster especially for packages with a lot of files. For 10 000 files in one package, the speedup is roughly from ~80 to ~2 seconds (time of the entire package build incl. creating the files from a Python script). That is a lot, but most of the real packages have much less files, so I am afraid you won't feel it.
More importantly, %python_provide is not needed for package names. Until now, we needed to this:
%package -n python3-foo ... %{?python_provide:%python_provide python3-foo}
This adds automatic provides "python-foo" (and since recently, also "python38-foo" for forwards/backwards compatibility with EL). This now happens automagically. Any package called "python3-..." will have those provides added implicitly by the Python generators when they are in the buildroot (python3-devel brings those in, so for Python packages this means always).
This automation was possible due to the speedup mentioned above, doing it the pre-RPM.16 way would be very slow, because the generator is called for every packaged file (which is still the case now, but it can now be a Lua macro). The ~2 seconds from above include this new feature.
There are 2 cases where manual %python_provide calls are still needed:
1. Metapackages without files
No files => no dependency generator. (I recommend to %ghost the dist-info/egg-info directory in such packages to workaround this -- we plan to add more good stuff regarding Python [extras] that will require this anyway.)
2. Virtual provides
The dependency generator can see the (sub)package's name, but not any other virtual provides. When you do:
%package -n python3-%{pypi_name} Provides: python3-%{legacy_name} = %{version}-%{release}
You are still supposed to add:
%{?python_provide:%python_provide python3-%{legacy_name}}
(Also note that %python_provide adds obsoletes for older python-foo versions, that was useful when we renamed everything from python-foo to python2-foo and when we changed the "default" from python2- to python3-. We are keeping the Obsoletes in the macro (for now), but I have decided to not automatically generate the Obsoletes based on the package name. A) I don't consider them really needed in Fedora 33 any more and B) an idea of implicit obsoletes doesn'T sound very intriguing to me. This is a decision that may be revisited later if it's bringing unforeseen trouble.)
You don't need to to actively remove the macro from your spec files, it does no harm. If you prefer to maintain a single spec, keep it there until Fedora 32 goes EOL (and EPEL 8 if that's your target). The guidelines always recommended using it the safe way (%{?python_provide:...}), so even if it ceases to exist in the future, it can remain in specs. There is no plan to remove it in any near or distant future, as it is still needed for the virtual provides. The generators also use it internally (for DRY and consistent results).
If you'll get into trouble because of this, let us know and we'll fix it.
I'll make a note for myself to update the Python packaging guidelines.
On 2020-04-03 20:44, Miro Hrončok wrote:
Hello Python packagers.
I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to older releases.
The python(abi) requirement is now added from a RPM Lua macro, instead of by executing a Shell script. This is a bit faster especially for packages with a lot of files. For 10 000 files in one package, the speedup is roughly from ~80 to ~2 seconds (time of the entire package build incl. creating the files from a Python script). That is a lot, but most of the real packages have much less files, so I am afraid you won't feel it.
Hi, One thing I'm wondering about is: why is python(abi) provided by a generator, anyway? It's provided by a very small set of packages (python2 and python3 in Fedora). Couldn't it just be listed in those specfiles?
On 07. 04. 20 11:06, Petr Viktorin wrote:
Hi, One thing I'm wondering about is: why is python(abi) provided by a generator, anyway? It's provided by a very small set of packages (python2 and python3 in Fedora). Couldn't it just be listed in those specfiles?
BTW It is redundantly also listed in those specfiles.
On 07. 04. 20 11:06, Petr Viktorin wrote:
On 2020-04-03 20:44, Miro Hrončok wrote:
Hello Python packagers.
I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to older releases.
The python(abi) requirement is now added from a RPM Lua macro, instead of by executing a Shell script. This is a bit faster especially for packages with a lot of files. For 10 000 files in one package, the speedup is roughly from ~80 to ~2 seconds (time of the entire package build incl. creating the files from a Python script). That is a lot, but most of the real packages have much less files, so I am afraid you won't feel it.
Hi, One thing I'm wondering about is: why is python(abi) provided by a generator, anyway? It's provided by a very small set of packages (python2 and python3 in Fedora). Couldn't it just be listed in those specfiles?
Just to put this into more context: The time overhead is for the requires. The provides are filtered on a certain path and even if we are automating stuff for 2 packages which indeed might not be needed, it doesn't really hurt anything.
See also https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject.... for me trying to figure out when this provides are useful -- depending on the answer there, we might want to keep the automated provides of python(abi) only and remove the manual ones.
On 2020-04-07 12:40, Miro Hrončok wrote:
On 07. 04. 20 11:06, Petr Viktorin wrote:
On 2020-04-03 20:44, Miro Hrončok wrote:
Hello Python packagers.
I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to older releases.
The python(abi) requirement is now added from a RPM Lua macro, instead of by executing a Shell script. This is a bit faster especially for packages with a lot of files. For 10 000 files in one package, the speedup is roughly from ~80 to ~2 seconds (time of the entire package build incl. creating the files from a Python script). That is a lot, but most of the real packages have much less files, so I am afraid you won't feel it.
Hi, One thing I'm wondering about is: why is python(abi) provided by a generator, anyway? It's provided by a very small set of packages (python2 and python3 in Fedora). Couldn't it just be listed in those specfiles?
Just to put this into more context: The time overhead is for the requires. The provides are filtered on a certain path and even if we are automating stuff for 2 packages which indeed might not be needed, it doesn't really hurt anything.
See also https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject.... for me trying to figure out when this provides are useful -- depending on the answer there, we might want to keep the automated provides of python(abi) only and remove the manual ones.
The part I don't get is: why keep the automated provides over manual ones. Compared to a line in the relevant spec files, the macro looks like it adds complexity for no benefit.
I get the automatic *requires* on python(abi) -- that's used by thousands of packages and easy to forget. But the provides will only ever be in CPython.
On Tue, Apr 7, 2020 at 8:26 AM Petr Viktorin pviktori@redhat.com wrote:
On 2020-04-07 12:40, Miro Hrončok wrote:
On 07. 04. 20 11:06, Petr Viktorin wrote:
On 2020-04-03 20:44, Miro Hrončok wrote:
Hello Python packagers.
I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to older releases.
The python(abi) requirement is now added from a RPM Lua macro, instead of by executing a Shell script. This is a bit faster especially for packages with a lot of files. For 10 000 files in one package, the speedup is roughly from ~80 to ~2 seconds (time of the entire package build incl. creating the files from a Python script). That is a lot, but most of the real packages have much less files, so I am afraid you won't feel it.
Hi, One thing I'm wondering about is: why is python(abi) provided by a generator, anyway? It's provided by a very small set of packages (python2 and python3 in Fedora). Couldn't it just be listed in those specfiles?
Just to put this into more context: The time overhead is for the requires. The provides are filtered on a certain path and even if we are automating stuff for 2 packages which indeed might not be needed, it doesn't really hurt anything.
See also https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject.... for me trying to figure out when this provides are useful -- depending on the answer there, we might want to keep the automated provides of python(abi) only and remove the manual ones.
The part I don't get is: why keep the automated provides over manual ones. Compared to a line in the relevant spec files, the macro looks like it adds complexity for no benefit.
I get the automatic *requires* on python(abi) -- that's used by thousands of packages and easy to forget. But the provides will only ever be in CPython.
I'd use the same argument to get rid of the manual Provides, but the reason for why they exist in both manual and generated form is simple: to prevent human error.
There have been cases in the past where the python RPM was being built in places where generators are disabled or not going to work properly (e.g. in this multi-python case), then the manual Provides would ensure things came out as intended. The automatic Provides generation is used to ensure that if things are using the generator, that everything is _ensured_ to be consistent. It's a very bad idea to have generated requires without generated provides, because you cannot guarantee the names will match up.
On 07. 04. 20 14:25, Petr Viktorin wrote:
On 2020-04-07 12:40, Miro Hrončok wrote:
On 07. 04. 20 11:06, Petr Viktorin wrote:
On 2020-04-03 20:44, Miro Hrončok wrote:
Hello Python packagers.
I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to older releases.
The python(abi) requirement is now added from a RPM Lua macro, instead of by executing a Shell script. This is a bit faster especially for packages with a lot of files. For 10 000 files in one package, the speedup is roughly from ~80 to ~2 seconds (time of the entire package build incl. creating the files from a Python script). That is a lot, but most of the real packages have much less files, so I am afraid you won't feel it.
Hi, One thing I'm wondering about is: why is python(abi) provided by a generator, anyway? It's provided by a very small set of packages (python2 and python3 in Fedora). Couldn't it just be listed in those specfiles?
Just to put this into more context: The time overhead is for the requires. The provides are filtered on a certain path and even if we are automating stuff for 2 packages which indeed might not be needed, it doesn't really hurt anything.
See also https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject.... for me trying to figure out when this provides are useful -- depending on the answer there, we might want to keep the automated provides of python(abi) only and remove the manual ones.
The part I don't get is: why keep the automated provides over manual ones. Compared to a line in the relevant spec files, the macro looks like it adds complexity for no benefit.
I get the automatic *requires* on python(abi) -- that's used by thousands of packages and easy to forget. But the provides will only ever be in CPython.
I see your point and I don't disagree necessarily. I was just trying to provide more context.
One reason to keep this is robustness. Once the python(abi) generator is enabled, it makes sure that both provides and requires are present.
On 03. 04. 20 20:44, Miro Hrončok wrote:
Hello Python packagers.
I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to older releases.
The python(abi) requirement is now added from a RPM Lua macro, instead of by executing a Shell script. This is a bit faster especially for packages with a lot of files. For 10 000 files in one package, the speedup is roughly from ~80 to ~2 seconds (time of the entire package build incl. creating the files from a Python script). That is a lot, but most of the real packages have much less files, so I am afraid you won't feel it.
More importantly, %python_provide is not needed for package names. Until now, we needed to this:
%package -n python3-foo ... %{?python_provide:%python_provide python3-foo}
This adds automatic provides "python-foo" (and since recently, also "python38-foo" for forwards/backwards compatibility with EL). This now happens automagically. Any package called "python3-..." will have those provides added implicitly by the Python generators when they are in the buildroot (python3-devel brings those in, so for Python packages this means always).
This automation was possible due to the speedup mentioned above, doing it the pre-RPM.16 way would be very slow, because the generator is called for every packaged file (which is still the case now, but it can now be a Lua macro). The ~2 seconds from above include this new feature.
There are 2 cases where manual %python_provide calls are still needed:
- Metapackages without files
No files => no dependency generator. (I recommend to %ghost the dist-info/egg-info directory in such packages to workaround this -- we plan to add more good stuff regarding Python [extras] that will require this anyway.)
- Virtual provides
The dependency generator can see the (sub)package's name, but not any other virtual provides. When you do:
%package -n python3-%{pypi_name} Provides: python3-%{legacy_name} = %{version}-%{release}
You are still supposed to add:
%{?python_provide:%python_provide python3-%{legacy_name}}
(Also note that %python_provide adds obsoletes for older python-foo versions, that was useful when we renamed everything from python-foo to python2-foo and when we changed the "default" from python2- to python3-. We are keeping the Obsoletes in the macro (for now), but I have decided to not automatically generate the Obsoletes based on the package name. A) I don't consider them really needed in Fedora 33 any more and B) an idea of implicit obsoletes doesn'T sound very intriguing to me. This is a decision that may be revisited later if it's bringing unforeseen trouble.)
You don't need to to actively remove the macro from your spec files, it does no harm. If you prefer to maintain a single spec, keep it there until Fedora 32 goes EOL (and EPEL 8 if that's your target). The guidelines always recommended using it the safe way (%{?python_provide:...}), so even if it ceases to exist in the future, it can remain in specs. There is no plan to remove it in any near or distant future, as it is still needed for the virtual provides. The generators also use it internally (for DRY and consistent results).
If you'll get into trouble because of this, let us know and we'll fix it.
First problem:
When you sue %python_provide and when you build the package with the new generator, the provides are listed twice:
$ rpm -qp --provides python3-double-provides-0-0.fc33.noarch.rpm python-double-provides = 0-0.fc33 python-double-provides = 0-0.fc33 python3-double-provides = 0-0.fc33 python38-double-provides = 0-0.fc33 python38-double-provides = 0-0.fc33
$ rpmlint python3-double-provides-0-0.fc33.noarch.rpm ... python3-double-provides.noarch: E: useless-provides python-double-provides python3-double-provides.noarch: E: useless-provides python38-double-provides ...
I've reported that as a bug in RPM:
https://github.com/rpm-software-management/rpm/issues/1166
On 07. 04. 20 12:27, Miro Hrončok wrote:
When you use %python_provide and when you build the package with the new generator, the provides are listed twice:
$ rpm -qp --provides python3-double-provides-0-0.fc33.noarch.rpm python-double-provides = 0-0.fc33 python-double-provides = 0-0.fc33 python3-double-provides = 0-0.fc33 python38-double-provides = 0-0.fc33 python38-double-provides = 0-0.fc33
$ rpmlint python3-double-provides-0-0.fc33.noarch.rpm ... python3-double-provides.noarch: E: useless-provides python-double-provides python3-double-provides.noarch: E: useless-provides python38-double-provides ...
I've reported that as a bug in RPM:
Apparently, there is a good reason to not have those merged. People are notified of the uselessness of the manual one. I'll take that into account when amending the guidelines.
I recommend to remove the manual %python_provide calls for package names on rawhide+.
It is "safe" to keep them for "single-spec" experience: The problem does no harm, createrepo_c merges those provides at the end.
If you like to have a single spec and don't want the rpmlint error, you can wrap the call in %{?!__pythonname_provides: }.
On 08. 04. 20 13:28, Miro Hrončok wrote:
On 07. 04. 20 12:27, Miro Hrončok wrote:
When you use %python_provide and when you build the package with the new generator, the provides are listed twice:
$ rpm -qp --provides python3-double-provides-0-0.fc33.noarch.rpm python-double-provides = 0-0.fc33 python-double-provides = 0-0.fc33 python3-double-provides = 0-0.fc33 python38-double-provides = 0-0.fc33 python38-double-provides = 0-0.fc33
$ rpmlint python3-double-provides-0-0.fc33.noarch.rpm ... python3-double-provides.noarch: E: useless-provides python-double-provides python3-double-provides.noarch: E: useless-provides python38-double-provides ...
I've reported that as a bug in RPM:
Apparently, there is a good reason to not have those merged. People are notified of the uselessness of the manual one. I'll take that into account when amending the guidelines.
Good news is that I've managed to workaround the double-generated provide and even if you keep using %python_provide for backwards compatibility, you will now only get the provides once.
Nice job, Miro!
Thanks a lot for working on this.
On Fri, Apr 3, 2020 at 8:45 PM Miro Hrončok mhroncok@redhat.com wrote:
Hello Python packagers.
I have just updated python-rpm-generators to python-rpm-generators-11-1.fc33 in Rawhide. It uses some fresh stuff from RPM 4.16 and will not be backported to older releases.
The python(abi) requirement is now added from a RPM Lua macro, instead of by executing a Shell script. This is a bit faster especially for packages with a lot of files. For 10 000 files in one package, the speedup is roughly from ~80 to ~2 seconds (time of the entire package build incl. creating the files from a Python script). That is a lot, but most of the real packages have much less files, so I am afraid you won't feel it.
More importantly, %python_provide is not needed for package names. Until now, we needed to this:
%package -n python3-foo ... %{?python_provide:%python_provide python3-foo}This adds automatic provides "python-foo" (and since recently, also "python38-foo" for forwards/backwards compatibility with EL). This now happens automagically. Any package called "python3-..." will have those provides added implicitly by the Python generators when they are in the buildroot (python3-devel brings those in, so for Python packages this means always).
This automation was possible due to the speedup mentioned above, doing it the pre-RPM.16 way would be very slow, because the generator is called for every packaged file (which is still the case now, but it can now be a Lua macro). The ~2 seconds from above include this new feature.
There are 2 cases where manual %python_provide calls are still needed:
- Metapackages without files
No files => no dependency generator. (I recommend to %ghost the dist-info/egg-info directory in such packages to workaround this -- we plan to add more good stuff regarding Python [extras] that will require this anyway.)
- Virtual provides
The dependency generator can see the (sub)package's name, but not any other virtual provides. When you do:
%package -n python3-%{pypi_name} Provides: python3-%{legacy_name} = %{version}-%{release}You are still supposed to add:
%{?python_provide:%python_provide python3-%{legacy_name}}(Also note that %python_provide adds obsoletes for older python-foo versions, that was useful when we renamed everything from python-foo to python2-foo and when we changed the "default" from python2- to python3-. We are keeping the Obsoletes in the macro (for now), but I have decided to not automatically generate the Obsoletes based on the package name. A) I don't consider them really needed in Fedora 33 any more and B) an idea of implicit obsoletes doesn'T sound very intriguing to me. This is a decision that may be revisited later if it's bringing unforeseen trouble.)
You don't need to to actively remove the macro from your spec files, it does no harm. If you prefer to maintain a single spec, keep it there until Fedora 32 goes EOL (and EPEL 8 if that's your target). The guidelines always recommended using it the safe way (%{?python_provide:...}), so even if it ceases to exist in the future, it can remain in specs. There is no plan to remove it in any near or distant future, as it is still needed for the virtual provides. The generators also use it internally (for DRY and consistent results).
If you'll get into trouble because of this, let us know and we'll fix it.
I'll make a note for myself to update the Python packaging guidelines.
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
python-devel@lists.fedoraproject.org
-
Igor Gnatenko -
Miro Hrončok -
Neal Gompa -
Petr Viktorin