Hello, Here is a message I want to post to devel-announce later. Let me know if you see anything wrong (or would like to take over python2 maintainership).
--- Intent to orphan Python 2
tl;dr: Unless someone steps up to Python 2 after 2020, we need to start dropping python2 packages now.
Python 2.7 will reach end of upstream support on 1st of January, 2020, after almost 10 years (!) of volunteer maintenance.
Fedora still has more than 3000 packages depending on python2 – many more than we can support without upstream help. We (rightly) don't have the authority to say "please drop your unneeded python2 subpackages, or let us drop them for you" [0]. The next best thing we *can* say is: "if Fedora is to keep python2 alive, we won't be the ones doing it – at least not at the current magnitude". Here are the details.
The current maintainers of python2 would like to "orphan" the python2 package in 2020 (~ Fedora 32/33): - Charalampos Stratakis (cstratak) - Tomáš Orsava (torsava) - Miro Hrnočok (churchyard) - Petr Viktorin (pviktori) - Iryna Schcherbina (ishcherb) - Michal Cyprian (mcyprian) - Bohuslav Kabrda (bkabrda) - David Malcolm (dmalcolm) - Thomas Spura (tomspur)
As with any orphaning, that leaves two options: - someone else agrees now to take over in 2020 (keeping in mind this is a security-critical package and will be abandoned upstream), or - dependent packages drop support for Python 2.
Unlike most other orphanings, we have some thousands of dependent packages, so a lot of time and care is required. In case no one steps up, we'd like to start dropping Python 2 support from dependent packages *now*, starting with ported libraries on whose python2 version nothing in Fedora depends. (We keep a list of those at [1].) Of course, we're ready to make various compromises with interested packagers, as long as there's an understanding that we won't just support python2 forever.
If you are a maintainer of anything at [1] we ask you kindly to consider removing the python2 subpackages.
If no one steps up to maintain python2 after 2020, we're prepared to package a "legacy" python27 package, similar to what we do for e.g. python33 [2], to: - help developers that still need to test against this version - support exceptionally important non–security critical applications, if their upstreams don't manage to port to Python 3 in time
[0] https://pagure.io/packaging-committee/issue/753 [1] http://fedora.portingdb.xyz/#legacy-leaf [2] https://src.fedoraproject.org/rpms/python33/
On Tue, Mar 20, 2018 at 10:51:30AM +0100, Petr Viktorin wrote:
Hello, Here is a message I want to post to devel-announce later. Let me know if you see anything wrong (or would like to take over python2 maintainership).
Intent to orphan Python 2
First a disclaimer: I'm all for moving to python3, so I'm not disagreeing with the general idea, but with the transition plan.
tl;dr: Unless someone steps up to Python 2 after 2020, we need to start dropping python2 packages now.
I think this is a contentious statement. "need" is the wrong word. "One of the ways to prepare for the declared EOS from upsteam is to" would be more accurate.
And it's just one of the possible ways. Another would be prepare a conditional in all spec files and simply flip it over before a mass rebuild for F32 (or whatever).
Python 2.7 will reach end of upstream support on 1st of January, 2020, after almost 10 years (!) of volunteer maintenance.
Fedora still has more than 3000 packages depending on python2 – many more than we can support without upstream help.
This is also misleading. Those 3000 packages have different upstreams, and many of those upstreams will support python2 even after python-eol.
We (rightly) don't have the authority to say "please drop your unneeded python2 subpackages, or let us drop them for you" [0]. The next best thing we *can* say is: "if Fedora is to keep python2 alive, we won't be the ones doing it – at least not at the current magnitude".
The way you propose is essentially a mirror of how python3 support was introduced. For a long time it was hard to use python3 for anything serious because there was _always_ one or two more unported dependencies. It took many years to reach this critical mass where everything "important" was available. This period was harsh on users, because python3 was nice and shiny but people were unable to switch and unable to _influence_ this state in any significant way, the way that it is hard to push the whole ecosystem. The proposed model of nipping python2 support at the edges is the same thing, in reverse. First some leaf packages are dropped, and then some somewhat more important packages, and then suddenly it becomes hard to use python2 for anything "serious", even though it's still "available". I think that's a bad way to proceed for our users. Let them have a single moment where python2 support goes away. Announce this moment at least a year in advance. Let them keep running the last release before that for as long as they need.
If we want to prepare for the python2-eol, IMHO we should first finish all the steps that don't impact users: convert our tooling, make sure python2 is not used in build for anything where python3 can be used, rename packages, fix provides and requires, update she-bangs, add conditionals to make dropping python2 support easy. In particular, let's first finish Changes/Avoid_usr_bin_python_in_RPM_Build [1].
[1] https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build
Zbyszek
On 20.3.2018 11:25, Zbigniew Jędrzejewski-Szmek wrote:
to push the whole ecosystem. The proposed model of nipping python2 support at the edges is the same thing, in reverse. First some leaf packages are dropped, and then some somewhat more important packages, and then suddenly it becomes hard to use python2 for anything "serious", even though it's still "available". I think that's a bad way to proceed for our users. Let them have a single moment where python2 support goes away. Announce this moment at least a year in advance. Let them keep running the last release before that for as long as they need.
I think there is a fundamental difference of how I see python RPMs in Fedora and how you do. In my POV, purpose of pythonX-foo packages that only bring libraries is to provide dependency for an application that happens to be written in python, runs on pythonX and need the foo library. If there is no such application packaged in Fedora, I see no purpose of such package. (Except maybe if such app is packaged in another repository used by our users. That's a corner case and if the maintainer of pythonX-foo is aware of that, it can always be used as argument to keep pythonX-foo.)
We live in an era of pip (and rubygems, and cpan, and npm etc.). Trying to mirror those systems with RPM packages just because it's possible is tedious, incomplete and mostly pointless IMHO. An era when upstream projects were eager to get their tool into the distros has passed. Having pythonX-foo packaged where no other tool/app/service needs it is not beneficial (if foo is a library only thing, not a tool itself). I have this opinion even with python3 packages.
Developers who wants to use python2 for whatever reasons (rational or emotional ones) can do that on Fedora. We encourage developers to use virtual environments and pip [1]. Removing leaf python2 packages does not block them here at all.
Let them have a single moment where python2 support goes away.
Removing all python2 things at one point is impossible. We've tried to test this with Django [2] on a much smaller scale. (Tens instead of thousands packages.) It took us too much time and too much energy. At the end, too much packages were just retired because their maintainers are "dead". I cannot imagine doing this on current scale in 2020. We can just retire python2 then and see what breaks. Honestly, everything would break.
We need to communicate loudly that Python 2 is dead. And this is a way to do it. If the maintainrs add conditionals for Fedora 33, let them have it. But every single python2 package that goes away is a good thing longterm both for Fedora and for the Python ecosystem.
If we want to prepare for the python2-eol, IMHO we should first finish all the steps that don't impact users: convert our tooling, make sure python2 is not used in build for anything where python3 can be used, rename packages, fix provides and requires, update she-bangs, add conditionals to make dropping python2 support easy. In particular, let's first finish Changes/Avoid_usr_bin_python_in_RPM_Build.
Yes, we should focus on that. But if we don't do things in parallel, we have very little chance to move forward fast enough.
[1] https://developer.fedoraproject.org/tech/languages/python/python-installatio... [2] https://fedoraproject.org/wiki/Changes/Django20
On Tue, Mar 20, 2018 at 12:23:06PM +0100, Miro Hrončok wrote:
On 20.3.2018 11:25, Zbigniew Jędrzejewski-Szmek wrote:
to push the whole ecosystem. The proposed model of nipping python2 support at the edges is the same thing, in reverse. First some leaf packages are dropped, and then some somewhat more important packages, and then suddenly it becomes hard to use python2 for anything "serious", even though it's still "available". I think that's a bad way to proceed for our users. Let them have a single moment where python2 support goes away. Announce this moment at least a year in advance. Let them keep running the last release before that for as long as they need.
I think there is a fundamental difference of how I see python RPMs in Fedora and how you do. In my POV, purpose of pythonX-foo packages that only bring libraries is to provide dependency for an application that happens to be written in python, runs on pythonX and need the foo library. If there is no such application packaged in Fedora, I see no purpose of such package. (Except maybe if such app is packaged in another repository used by our users. That's a corner case and if the maintainer of pythonX-foo is aware of that, it can always be used as argument to keep pythonX-foo.)
Indeed, I'm using those python packages like a dinosaur ;) But more seriously, I think the distinction between library and app in the python world is neither clear nor particularly significant. Distro packaging provides clear benefits in both cases: easy and quick install, easy and quick uninstall, testing, provenance history, license checks, general reliability.
So instead of losing packages, I'd like to see more automatic packaging and updates of packages from pypi and other "forges". We are making steps in that direction, but not quickly enough.
We live in an era of pip (and rubygems, and cpan, and npm etc.). Trying to mirror those systems with RPM packages just because it's possible is tedious, incomplete and mostly pointless IMHO. An era when upstream projects were eager to get their tool into the distros has passed. Having pythonX-foo packaged where no other tool/app/service needs it is not beneficial (if foo is a library only thing, not a tool itself). I have this opinion even with python3 packages.
Developers who wants to use python2 for whatever reasons (rational or emotional ones) can do that on Fedora. We encourage developers to use virtual environments and pip [1]. Removing leaf python2 packages does not block them here at all.
Let them have a single moment where python2 support goes away.
Removing all python2 things at one point is impossible. We've tried to test this with Django [2] on a much smaller scale. (Tens instead of thousands packages.) It took us too much time and too much energy. At the end, too much packages were just retired because their maintainers are "dead". I cannot imagine doing this on current scale in 2020. We can just retire python2 then and see what breaks. Honestly, everything would break.
I don't think this is entirely comparable. Django20 was about updating packages and switching support from python2 to python3. Just dropping existing support for python2 is something much easier.
Nevertheless, I agree that this still is a very significant chunk of work at this scale. So spreading this out makes a lot of sense, but imho python2 subpackages don't need to go away, they can just be conditionalized.
What about adding conditionals like
%if 0%{?rhel} > 7 || 0%{?fedora} > 31 # Disable python2 build by default %bcond_with python2 %else %bcond_without python2 %endif
starting now?
Zbyszek
On 20.3.2018 14:45, Zbigniew Jędrzejewski-Szmek wrote:
Indeed, I'm using those python packages like a dinosaur ;)
:D
What about adding conditionals like
%if 0%{?rhel} > 7 || 0%{?fedora} > 31 # Disable python2 build by default %bcond_with python2 %else %bcond_without python2 %endif
starting now?
I am not against that. However currently that number (31) is somewhat artificial. it's up to the maintainer to choose if it's 28 or 32 (or anything in between). Should we somehow recommend a specific Fedora release here? Why 31?
Should a Fedora Change for that release be crafted that says "Mass python2 packages removal"?
On Tue, Mar 20, 2018 at 03:28:23PM +0100, Miro Hrončok wrote:
On 20.3.2018 14:45, Zbigniew Jędrzejewski-Szmek wrote:
Indeed, I'm using those python packages like a dinosaur ;)
:D
What about adding conditionals like
%if 0%{?rhel} > 7 || 0%{?fedora} > 31 # Disable python2 build by default %bcond_with python2 %else %bcond_without python2 %endif
starting now?
I am not against that. However currently that number (31) is somewhat artificial. it's up to the maintainer to choose if it's 28 or 32 (or anything in between). Should we somehow recommend a specific Fedora release here? Why 31?
Re 31: my thinking was that python2-eos is at 2010-01-01. If we keep up current biannual release schedule, we'll have 28 and 29 this year, 30 and 31 in 2019, and 32 in early 2020. But this is of course wrong, because we need python2 support for the lifetime of the release, not just at the start. So actually 29 which will be supported until 2018-10-30 + 13 months ≈ 2020-01-01, is the last release overlapped by python2 support. This right conditional would be:
%if 0%{?rhel} > 7 || 0%{?fedora} > 28
but that round around the corner. So it's not even necessary to add the conditional, as long as the patch to drop support is only added in rawhide, not F28.
OK, so I withdraw my objections. Removing python2 support in rawhide is fine.
Should a Fedora Change for that release be crafted that says "Mass python2 packages removal"?
Yeah, I think it should be filed as a F29 change.
Zbyszek
On 20.3.2018 17:11, Zbigniew Jędrzejewski-Szmek wrote:
Should a Fedora Change for that release be crafted that says "Mass python2 packages removal"?
Yeah, I think it should be filed as a F29 change.
I think we should send the e-mail first and do that after, if nobody takes python2.
On Tue, Mar 20, 2018 at 04:11:46PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Mar 20, 2018 at 03:28:23PM +0100, Miro Hrončok wrote:
On 20.3.2018 14:45, Zbigniew Jędrzejewski-Szmek wrote:
Indeed, I'm using those python packages like a dinosaur ;)
:D
What about adding conditionals like
%if 0%{?rhel} > 7 || 0%{?fedora} > 31 # Disable python2 build by default %bcond_with python2 %else %bcond_without python2 %endif
starting now?
I am not against that. However currently that number (31) is somewhat artificial. it's up to the maintainer to choose if it's 28 or 32 (or anything in between). Should we somehow recommend a specific Fedora release here? Why 31?
Re 31: my thinking was that python2-eos is at 2010-01-01. If we keep up current biannual release schedule, we'll have 28 and 29 this year, 30 and 31 in 2019, and 32 in early 2020. But this is of course wrong, because we need python2 support for the lifetime of the release, not just at the start. So actually 29 which will be supported until 2018-10-30 + 13 months ≈ 2020-01-01, is the last release overlapped by python2 support. This right conditional would be:
%if 0%{?rhel} > 7 || 0%{?fedora} > 28
but that round around the corner. So it's not even necessary to add the conditional, as long as the patch to drop support is only added in rawhide, not F28.
OK, so I withdraw my objections. Removing python2 support in rawhide is fine.
Pff, sorry I can't count. The right conditional would be %if 0%{?rhel} > 7 || 0%{?fedora} > 29 so dropping support in rawhide right now would be premature. It seems that adding a conditional like that is the way to go for now.
Should a Fedora Change for that release be crafted that says "Mass python2 packages removal"?
Yeah, I think it should be filed as a F29 change.
F30 might be more appropriate.
Zbyszek
On 03/20/18 21:47, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Mar 20, 2018 at 04:11:46PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Mar 20, 2018 at 03:28:23PM +0100, Miro Hrončok wrote:
On 20.3.2018 14:45, Zbigniew Jędrzejewski-Szmek wrote:
Indeed, I'm using those python packages like a dinosaur ;)
:D
What about adding conditionals like
%if 0%{?rhel} > 7 || 0%{?fedora} > 31 # Disable python2 build by default %bcond_with python2 %else %bcond_without python2 %endif
starting now?
I am not against that. However currently that number (31) is somewhat artificial. it's up to the maintainer to choose if it's 28 or 32 (or anything in between). Should we somehow recommend a specific Fedora release here? Why 31?
Re 31: my thinking was that python2-eos is at 2010-01-01. If we keep up current biannual release schedule, we'll have 28 and 29 this year, 30 and 31 in 2019, and 32 in early 2020. But this is of course wrong, because we need python2 support for the lifetime of the release, not just at the start. So actually 29 which will be supported until 2018-10-30 + 13 months ≈ 2020-01-01, is the last release overlapped by python2 support. This right conditional would be:
%if 0%{?rhel} > 7 || 0%{?fedora} > 28
but that round around the corner. So it's not even necessary to add the conditional, as long as the patch to drop support is only added in rawhide, not F28.
OK, so I withdraw my objections. Removing python2 support in rawhide is fine.
Pff, sorry I can't count. The right conditional would be %if 0%{?rhel} > 7 || 0%{?fedora} > 29 so dropping support in rawhide right now would be premature. It seems that adding a conditional like that is the way to go for now.
Compared to just removing the subpackage, it's a difference of just one release. It may not be worth the complication of conditionalizing everything.
I'd say maintainers can either drop the subpackages now in Rawhide, or add a conditional for Fedora > 29 -- up to the maintainer.
On 03/20/18 14:45, Zbigniew Jędrzejewski-Szmek wrote:
On Tue, Mar 20, 2018 at 12:23:06PM +0100, Miro Hrončok wrote:
On 20.3.2018 11:25, Zbigniew Jędrzejewski-Szmek wrote:
to push the whole ecosystem. The proposed model of nipping python2 support at the edges is the same thing, in reverse. First some leaf packages are dropped, and then some somewhat more important packages, and then suddenly it becomes hard to use python2 for anything "serious", even though it's still "available". I think that's a bad way to proceed for our users. Let them have a single moment where python2 support goes away. Announce this moment at least a year in advance. Let them keep running the last release before that for as long as they need.
I think there is a fundamental difference of how I see python RPMs in Fedora and how you do. In my POV, purpose of pythonX-foo packages that only bring libraries is to provide dependency for an application that happens to be written in python, runs on pythonX and need the foo library. If there is no such application packaged in Fedora, I see no purpose of such package. (Except maybe if such app is packaged in another repository used by our users. That's a corner case and if the maintainer of pythonX-foo is aware of that, it can always be used as argument to keep pythonX-foo.)
Indeed, I'm using those python packages like a dinosaur ;) But more seriously, I think the distinction between library and app in the python world is neither clear nor particularly significant. Distro packaging provides clear benefits in both cases: easy and quick install, easy and quick uninstall, testing, provenance history, license checks, general reliability.
So instead of losing packages, I'd like to see more automatic packaging and updates of packages from pypi and other "forges". We are making steps in that direction, but not quickly enough.
We live in an era of pip (and rubygems, and cpan, and npm etc.). Trying to mirror those systems with RPM packages just because it's possible is tedious, incomplete and mostly pointless IMHO. An era when upstream projects were eager to get their tool into the distros has passed. Having pythonX-foo packaged where no other tool/app/service needs it is not beneficial (if foo is a library only thing, not a tool itself). I have this opinion even with python3 packages.
Developers who wants to use python2 for whatever reasons (rational or emotional ones) can do that on Fedora. We encourage developers to use virtual environments and pip [1]. Removing leaf python2 packages does not block them here at all.
Let them have a single moment where python2 support goes away.
Removing all python2 things at one point is impossible. We've tried to test this with Django [2] on a much smaller scale. (Tens instead of thousands packages.) It took us too much time and too much energy. At the end, too much packages were just retired because their maintainers are "dead". I cannot imagine doing this on current scale in 2020. We can just retire python2 then and see what breaks. Honestly, everything would break.
I don't think this is entirely comparable. Django20 was about updating packages and switching support from python2 to python3. Just dropping existing support for python2 is something much easier.
Maybe it's much easier, but it's still not trivial.
I just saw a package that uses epydoc to generate its documentation, from its py2 code. Since epydoc is not ported, that package would lose its documentation if python2 was just suddenly dropped – and we wouldn't know in advance, because the package was in the "dual py2/py3 support" bucket. But I'm not complaining about epydoc in general. I'm afraid there are many more cases like that – cases where we won't know about the problem until we try to remove the py2 packages. Especially those with unresponsive maintainers, which tend to use the most interesting ancient technology.
I don't see a good way to uncover such problems other than to start removing stuff.
Nevertheless, I agree that this still is a very significant chunk of work at this scale. So spreading this out makes a lot of sense, but imho python2 subpackages don't need to go away, they can just be conditionalized.
What about adding conditionals like
%if 0%{?rhel} > 7 || 0%{?fedora} > 31 # Disable python2 build by default %bcond_with python2 %else %bcond_without python2 %endif
starting now?
Agreed, we just need to bikeshed the "31" number there.
python-devel@lists.fedoraproject.org
-
Miro Hrončok -
Petr Viktorin -
Zbigniew Jędrzejewski-Szmek