Hello Python packagers.
After touching the %python_provide topic in:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
I have realized several things I don't like about %python_provide:
1) It must be used conditionally (it is not defined in python-srpm-macros). That means you always wrap it in %{?python_provide:...} in order to have a "valid" specfile even when the macro is not yet defined (e.g. during SRPM creation in Koji or on a packager's machine without python-rpm-macros installed).
2) You cannot use it with arbitrary versions. Suppose you package python3-foo 1.0 but you want to provide python3-bar 2.0 for some reason -- this is not very common, but it happens. %python_provide only takes the "name" as an argument, always using %{?epoch} %{version} and %{release}.
3) You need to both add a virtual provide + use the macro. Suppose you want to provide python3-pkg_resources from python3-setuptools. Currently, you need to add:
Provides: python3-pkg_resources = %{version}-%{release} %{?python_provide:%python_provide python3-pkg_resources}
4) When used with (sub)package name, it generates a duplicate dependency on Fedora 33+ (and an rpmlint error).
5) It produces Obsoletes, but that might no longer be necessary nor desired.
6) Broken expectations about %_isa. It used to add %_isa provides based on wrong data, it no longer does that (except on old releases and EPELs), can be used manually with name%{?_isa}, but not on the old releases.
7) Undocuemnted error handling (e.g. the macro expands to nothing when used with pypy-foo, but errors when used with foo).
Hence, I was thinking (for the sake of backwards compatibility) to provide a new mechanism to do this and preserve the old macro as is, deprecating it in Fedora 36-ish, actually maybe removing it once RHEL 9 goes EOL (or never, which is basically the same from today's perspective).
The new macro should solve the problems from above, my current (quick, untested) ideas are:
1) Define the macro in python-srpm-macros. No need to use it conditionally. We can backport it to EPEL 8 and define a "stub" macro in EPEL 7 and 6. (An if we start using the macro only after Fedora 30 goes EOL, we can make the macro behave consistently across all Fedora versions.)
2) Accept version identifier as an optional argument, use %{?epoch} %{version} and %{release} as default. (See for example %{pypi_source} on how this can be done.)
3) Make the macro also produce the provide for the given name and document that. E.g. when you call it with python3-pkg_resources, it also provides python3-pkg_resources (not only python-pkg_resources etc.).
4) Make it so that for given arguments, the macro will only expand to something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
5) No obsoletes with the new macro. Packagers use manual obsoletes when desired.
6) Document clearly that there is no %{?_isa} (and there is no "backwards compatibility" load to carry). When absolutely desired, packagers can call the macro with %{name}%{?_isa}.
7) Support arbitrary names. Only provide the given name and nothing else if not "recognized".
As a bonus, I think the current if-elif-elif-elif-elif code can be replaced with lua patterns (imagine regex).
As always, this leaves us with the name problem, but I'd very much like to use %python_provides (note the s). The only problem I see is that it is likely to be mistaken for the old one, but IMHO it shouldn't really hurt that much.
Usage example:
%package -n python3-setuptools %python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
Another example:
%package -n python3-pillow %python_provides python3-PIL 1.1.6-100
Resulting provides:
python3-pillow = 7.1.1-1.fc33 python-pillow = 7.1.1-1.fc33 python39-pillow = 7.1.1-1.fc33 python3-PIL = 1.1.6-100 python-PIL = 1.1.6-100 python39-PIL = 1.1.6-100
Dummy when used with package name:
%package -n python3-pip %python_provides python3-pip
Provides:
python3-pip = 20.0.2-1.fc33 python-pip = 20.0.2-1.fc33 python39-pip = 20.0.2-1.fc33 (all of them only once)
Any name is OK:
%python_provides wroom 666
Provides:
wroom = 666
What do you think?
I finally got around to this mail...
On 2020-04-19 16:55, Miro Hrončok wrote:
Hello Python packagers.
After touching the %python_provide topic in:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
I have realized several things I don't like about %python_provide:
- It must be used conditionally (it is not defined in python-srpm-macros).
That means you always wrap it in %{?python_provide:...} in order to have a "valid" specfile even when the macro is not yet defined (e.g. during SRPM creation in Koji or on a packager's machine without python-rpm-macros installed).
- You cannot use it with arbitrary versions. Suppose you package
python3-foo 1.0 but you want to provide python3-bar 2.0 for some reason -- this is not very common, but it happens. %python_provide only takes the "name" as an argument, always using %{?epoch} %{version} and %{release}.
- You need to both add a virtual provide + use the macro. Suppose you
want to provide python3-pkg_resources from python3-setuptools. Currently, you need to add:
Provides: python3-pkg_resources = %{version}-%{release} %{?python_provide:%python_provide python3-pkg_resources}
- When used with (sub)package name, it generates a duplicate dependency
on Fedora 33+ (and an rpmlint error).
- It produces Obsoletes, but that might no longer be necessary nor
desired.
- Broken expectations about %_isa. It used to add %_isa provides based
on wrong data, it no longer does that (except on old releases and EPELs), can be used manually with name%{?_isa}, but not on the old releases.
- Undocuemnted error handling (e.g. the macro expands to nothing when
used with pypy-foo, but errors when used with foo).
Hence, I was thinking (for the sake of backwards compatibility) to provide a new mechanism to do this and preserve the old macro as is, deprecating it in Fedora 36-ish, actually maybe removing it once RHEL 9 goes EOL (or never, which is basically the same from today's perspective).
The new macro should solve the problems from above, my current (quick, untested) ideas are:
- Define the macro in python-srpm-macros. No need to use it
conditionally. We can backport it to EPEL 8 and define a "stub" macro in EPEL 7 and 6. (An if we start using the macro only after Fedora 30 goes EOL, we can make the macro behave consistently across all Fedora versions.)
- Accept version identifier as an optional argument, use %{?epoch}
%{version} and %{release} as default. (See for example %{pypi_source} on how this can be done.)
- Make the macro also produce the provide for the given name and
document that. E.g. when you call it with python3-pkg_resources, it also provides python3-pkg_resources (not only python-pkg_resources etc.).
- Make it so that for given arguments, the macro will only expand to
something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
- No obsoletes with the new macro. Packagers use manual obsoletes when
desired.
- Document clearly that there is no %{?_isa} (and there is no
"backwards compatibility" load to carry). When absolutely desired, packagers can call the macro with %{name}%{?_isa}.
- Support arbitrary names. Only provide the given name and nothing else
if not "recognized".
Is that better than erroring out when something is not recognized?
As a bonus, I think the current if-elif-elif-elif-elif code can be replaced with lua patterns (imagine regex).
As always, this leaves us with the name problem, but I'd very much like to use %python_provides (note the s). The only problem I see is that it is likely to be mistaken for the old one, but IMHO it shouldn't really hurt that much.
I guess something like %py_provides would also work, and maybe fit more nicely with the new generation of macros. Do you like full "python" prefix? But that's bikeshedding.
Usage example:
%package -n python3-setuptools %python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
I assume you meant "python39-setuptools" there.
Another example:
%package -n python3-pillow %python_provides python3-PIL 1.1.6-100
Resulting provides:
python3-pillow = 7.1.1-1.fc33 python-pillow = 7.1.1-1.fc33 python39-pillow = 7.1.1-1.fc33 python3-PIL = 1.1.6-100 python-PIL = 1.1.6-100 python39-PIL = 1.1.6-100
Dummy when used with package name:
%package -n python3-pip %python_provides python3-pip
Provides:
python3-pip = 20.0.2-1.fc33 python-pip = 20.0.2-1.fc33 python39-pip = 20.0.2-1.fc33 (all of them only once)
Any name is OK:
%python_provides wroom 666
Provides:
wroom = 666
What do you think?
Love it! I haven't thought about how feasible it is to implement, but I trust you on that.
On 28. 04. 20 11:00, Petr Viktorin wrote:
I finally got around to this mail...
Thanks.
- Support arbitrary names. Only provide the given name and nothing else if
not "recognized".
Is that better than erroring out when something is not recognized?
The "not recognized" part is hard to define.
With errors, we need to have 3 categories:
- "recognized" and should provide more (e.g. python3-foo -> python3-foo, python-foo) - "recognized" but should not provide more (e.g. python2-foo -> python2-foo) - "not recognized" and should error (e.g. llama -> error)
While with "meh" attitude, this is simpler to implement:
- should provide more (e.g. python3-foo -> python3-foo, python-foo) - should not provide more (e.g. python2-foo -> python2-foo) (e.g. llama -> llama)
This is also simpler with provides generator (when we cannot error out with arbitrary package names, so we currently need to pre-filter the arguments before using them).
And the benefit of erroring out with arbitrary names is very slow. It tells the packager that there is no way to provide additional names for "llama" and hence they SHOULD use regular Provides, but there is no harm when using this macro.
As always, this leaves us with the name problem, but I'd very much like to use %python_provides (note the s). The only problem I see is that it is likely to be mistaken for the old one, but IMHO it shouldn't really hurt that much.
I guess something like %py_provides would also work, and maybe fit more nicely with the new generation of macros. Do you like full "python" prefix? But that's bikeshedding.
%py_provides is actually much better:
- shorter, yet still understandable - easily distinguishable from %python_provide
Thanks.
Usage example:
%package -n python3-setuptools %python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
I assume you meant "python39-setuptools" there.
Copy paste problems. Full list as meant:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-setuptools = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
What do you think?
Love it! I haven't thought about how feasible it is to implement, but I trust you on that.
I will try and we'll see.
On 28. 04. 20 11:35, Miro Hrončok wrote:
%py_provides is actually much better:
- shorter, yet still understandable - easily distinguishable from %python_provide
Thanks.
%py_provides is now available in rawhide for testing.
I will backport it later, so for now using it unconditionalized might break srpm creations on older Fedoras. For that, you can create SRPMS in mock:
$ mock -r fedora-rawhide-x86_64 --spec=xxx.spec --buildsrpm --sources=.
On Tue, Apr 28, 2020 at 5:00 AM Petr Viktorin pviktori@redhat.com wrote:
I finally got around to this mail...
On 2020-04-19 16:55, Miro Hrončok wrote:
As always, this leaves us with the name problem, but I'd very much like to use %python_provides (note the s). The only problem I see is that it is likely to be mistaken for the old one, but IMHO it shouldn't really hurt that much.
I guess something like %py_provides would also work, and maybe fit more nicely with the new generation of macros. Do you like full "python" prefix? But that's bikeshedding.
I know it's a bit of bikeshedding, but I think it's weird how inconsistent we are with using py_* vs python_*. I'd rather us just fully switch over to using python_* prefix everywhere.
It's annoying and I'd rather be verbose and clear rather than terse and confusing.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 4/19/20 4:55 PM, Miro Hrončok wrote:
Hello Python packagers.
After touching the %python_provide topic in:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
I have realized several things I don't like about %python_provide:
- It must be used conditionally (it is not defined in
python-srpm-macros). That means you always wrap it in %{?python_provide:...} in order to have a "valid" specfile even when the macro is not yet defined (e.g. during SRPM creation in Koji or on a packager's machine without python-rpm-macros installed).
- You cannot use it with arbitrary versions. Suppose you package
python3-foo 1.0 but you want to provide python3-bar 2.0 for some reason -- this is not very common, but it happens. %python_provide only takes the "name" as an argument, always using %{?epoch} %{version} and %{release}.
- You need to both add a virtual provide + use the macro. Suppose you
want to provide python3-pkg_resources from python3-setuptools. Currently, you need to add:
Provides: python3-pkg_resources = %{version}-%{release} %{?python_provide:%python_provide python3-pkg_resources}
- When used with (sub)package name, it generates a duplicate
dependency on Fedora 33+ (and an rpmlint error).
- It produces Obsoletes, but that might no longer be necessary nor
desired.
- Broken expectations about %_isa. It used to add %_isa provides
based on wrong data, it no longer does that (except on old releases and EPELs), can be used manually with name%{?_isa}, but not on the old releases.
- Undocuemnted error handling (e.g. the macro expands to nothing when
used with pypy-foo, but errors when used with foo).
Hence, I was thinking (for the sake of backwards compatibility) to provide a new mechanism to do this and preserve the old macro as is, deprecating it in Fedora 36-ish, actually maybe removing it once RHEL 9 goes EOL (or never, which is basically the same from today's perspective).
The new macro should solve the problems from above, my current (quick, untested) ideas are:
- Define the macro in python-srpm-macros. No need to use it
conditionally. We can backport it to EPEL 8 and define a "stub" macro in EPEL 7 and 6. (An if we start using the macro only after Fedora 30 goes EOL, we can make the macro behave consistently across all Fedora versions.)
- Accept version identifier as an optional argument, use %{?epoch}
%{version} and %{release} as default. (See for example %{pypi_source} on how this can be done.)
- Make the macro also produce the provide for the given name and
document that. E.g. when you call it with python3-pkg_resources, it also provides python3-pkg_resources (not only python-pkg_resources etc.).
- Make it so that for given arguments, the macro will only expand to
something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
I thought multiple identical provides were just unified and didn't pose a problem. So what is the reason to focus on this once-per-build expansion?
- No obsoletes with the new macro. Packagers use manual obsoletes
when desired.
- Document clearly that there is no %{?_isa} (and there is no
"backwards compatibility" load to carry). When absolutely desired, packagers can call the macro with %{name}%{?_isa}.
- Support arbitrary names. Only provide the given name and nothing
else if not "recognized".
Given the limitations, I agree with this choice.
As a bonus, I think the current if-elif-elif-elif-elif code can be replaced with lua patterns (imagine regex).
As always, this leaves us with the name problem, but I'd very much like to use %python_provides (note the s). The only problem I see is that it is likely to be mistaken for the old one, but IMHO it shouldn't really hurt that much.
Usage example:
%package -n python3-setuptools %python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
What provides the names `python-setuptools` and `python39-pkg_resources`? From the code [0] it looks like the current rawhide implementation of %py_provides doesn't. On the other hand, it might be a good idea to also provide the names for the given %subpackage name (with an option to disable this). In that case, the macros should also without any parameters at all.
[0] https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.py...
Thanks for a nifty new macro! Tomas
Another example:
%package -n python3-pillow %python_provides python3-PIL 1.1.6-100
Resulting provides:
python3-pillow = 7.1.1-1.fc33 python-pillow = 7.1.1-1.fc33 python39-pillow = 7.1.1-1.fc33 python3-PIL = 1.1.6-100 python-PIL = 1.1.6-100 python39-PIL = 1.1.6-100
Dummy when used with package name:
%package -n python3-pip %python_provides python3-pip
Provides:
python3-pip = 20.0.2-1.fc33 python-pip = 20.0.2-1.fc33 python39-pip = 20.0.2-1.fc33 (all of them only once)
Any name is OK:
%python_provides wroom 666
Provides:
wroom = 666
What do you think?
On 25. 05. 20 18:33, Tomas Orsava wrote:
On 4/19/20 4:55 PM, Miro Hrončok wrote:
- Make it so that for given arguments, the macro will only expand to
something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
I thought multiple identical provides were just unified and didn't pose a problem. So what is the reason to focus on this once-per-build expansion?
Not if one if manual and one from a generator. See:
https://github.com/rpm-software-management/rpm/issues/1166
- Support arbitrary names. Only provide the given name and nothing else if
not "recognized".
Given the limitations, I agree with this choice.
What limitations?
Usage example:
%package -n python3-setuptools %python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
What provides the names `python-setuptools`
The generator.
and `python39-pkg_resources`?
%py_provides python3-pkg_resources
(Note that it is python3.9-pkg_resources now.)
From the code [0] it looks like the current rawhide implementation of %py_provides doesn't. On the other hand, it might be a good idea to also provide the names for the given %subpackage name (with an option to disable this). In that case, the macros should also without any parameters at all.
I don't get this part, sorry.
[0] https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.py...
On 5/25/20 7:42 PM, Miro Hrončok wrote:
On 25. 05. 20 18:33, Tomas Orsava wrote:
On 4/19/20 4:55 PM, Miro Hrončok wrote:
- Make it so that for given arguments, the macro will only expand
to something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
I thought multiple identical provides were just unified and didn't pose a problem. So what is the reason to focus on this once-per-build expansion?
Not if one if manual and one from a generator. See:
Interesting. One thing not mentioned is, does this cause any issues besides the visual issue of the provide being listed twice?
- Support arbitrary names. Only provide the given name and nothing
else if not "recognized".
Given the limitations, I agree with this choice.
What limitations?
I was referring to this bit from the other thread:
we cannot error out with arbitrary package names, so we currently
need to pre-filter the arguments before using them
Did I misunderstand it?
Usage example:
%package -n python3-setuptools %python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
What provides the names `python-setuptools`
The generator.
Oh right, I thought you were only talking about the provides from the macro. Makes sense.
and `python39-pkg_resources`?
%py_provides python3-pkg_resources
(Note that it is python3.9-pkg_resources now.)
From the code [0] it looks like the current rawhide implementation of %py_provides doesn't. On the other hand, it might be a good idea to also provide the names for the given %subpackage name (with an option to disable this). In that case, the macros should also without any parameters at all.
I don't get this part, sorry.
Irrelevant due to my misunderstanding above.
Tomas
[0] https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.py...
On 26. 05. 20 11:29, Tomas Orsava wrote:
On 5/25/20 7:42 PM, Miro Hrončok wrote:
On 25. 05. 20 18:33, Tomas Orsava wrote:
On 4/19/20 4:55 PM, Miro Hrončok wrote:
- Make it so that for given arguments, the macro will only expand to
something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
I thought multiple identical provides were just unified and didn't pose a problem. So what is the reason to focus on this once-per-build expansion?
Not if one if manual and one from a generator. See:
Interesting. One thing not mentioned is, does this cause any issues besides the visual issue of the provide being listed twice?
Technical issues? No. Createrepo merges those, so in the repo, no difference.
But rpmlint is very loud about those. It tells you: This provide you put in is not needed (which is great except it only applies to rawhide and we don't want packagers start removing it and merging the removal to older branches).
Side note: Once Fedora 32 goes EOL, we can change the behavior, so rpmlint starts bugging packagers about this again.
- Support arbitrary names. Only provide the given name and nothing else if
not "recognized".
Given the limitations, I agree with this choice.
What limitations?
I was referring to this bit from the other thread:
we cannot error out with arbitrary package names, so we currently need to
pre-filter the arguments before using them
Did I misunderstand it?
Not really. I did. Nothing to discuss here, these aren't the droids you're looking for... move along.
On 5/26/20 11:52 AM, Miro Hrončok wrote:
On 26. 05. 20 11:29, Tomas Orsava wrote:
On 5/25/20 7:42 PM, Miro Hrončok wrote:
On 25. 05. 20 18:33, Tomas Orsava wrote:
On 4/19/20 4:55 PM, Miro Hrončok wrote:
- Make it so that for given arguments, the macro will only expand
to something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
I thought multiple identical provides were just unified and didn't pose a problem. So what is the reason to focus on this once-per-build expansion?
Not if one if manual and one from a generator. See:
Interesting. One thing not mentioned is, does this cause any issues besides the visual issue of the provide being listed twice?
Technical issues? No. Createrepo merges those, so in the repo, no difference.
But rpmlint is very loud about those. It tells you: This provide you put in is not needed (which is great except it only applies to rawhide and we don't want packagers start removing it and merging the removal to older branches).
Side note: Once Fedora 32 goes EOL, we can change the behavior, so rpmlint starts bugging packagers about this again.
Ah, makes sense.
Overall, good work!
Tomas
- Support arbitrary names. Only provide the given name and
nothing else if not "recognized".
Given the limitations, I agree with this choice.
What limitations?
I was referring to this bit from the other thread:
> we cannot error out with arbitrary package names, so we currently need to pre-filter the arguments before using them
Did I misunderstand it?
Not really. I did. Nothing to discuss here, these aren't the droids you're looking for... move along.
python-devel@lists.fedoraproject.org
-
Miro Hrončok -
Neal Gompa -
Petr Viktorin -
Tomas Orsava