python-devel January 2015

python-devel@lists.fedoraproject.org

6 participants
7 discussions

Apps using default Python in Fedora vs. EPEL
by Bohuslav Kabrda 02 Mar '15

02 Mar '15

Hey all, I just wanted to ask for some thoughts on a "problem" (or rather a "hardship") that is starting to show with the Python 3 transition in Fedora. I've been contacted by two maintainers of "applications" in Fedora for advice and have been thinking for some time how to solve this: Note: by "applications" I mean packages that provide end-user benefit and don't need to be packaged for both Python interpreters. They just use some Python (preferably the default one) to deliver functionality to user. Let's take copr-cli as an example - this is a thin CLI wrapper around python-copr. Current state: - Up until F21, maintainers were encouraged to build applications with Python 2, but weren't discouraged from building with Python 3. - From F22 on, packagers will be encouraged to build with Python 3 rather than Python 2. - Lots of packagers want to keep the same specfile for EPEL and Fedora. - Fedora guidelines mandate explicit usage of %__python2 and %__python3 (and all the sitelib/sitearch macros). The Problem: If packagers want to build against Python 3 in Fedora and Python in EPEL *and* keep the same specfile, they have to invent some ugly hacks, since Fedora's guidelines require explicit usage of versioned Python macros. This affects Requires and BuildRequires and %prep, %build, %install, %check sections. People who want to do this either redefine %__python in Fedora to point to Python 3 or something like that - I'm afraid that we could end up with a huge pile of crazy macro redefinitions in tons of packages and I want to avoid that. Proposed Solution: After thinking a few days about this, here's what I propose: - Every specfile will have a minimal header with macro definitions that will look like this: %if 0%{?fedora} %global default_python python3 %else %global default_python python %endif %make_default_python %default_python - The %make_default_python macro function will point all the unversioned macros to proper values for given %default_python: ### In Fedora %{__python} -> /usr/bin/python3 %{python_sitelib} -> /usr/lib/python3.X/site-packages # and other macros... ### In EPEL %{__python} -> /usr/bin/python2 %{python_sitelib} -> /usr/lib/python2.X/site-packages # and other macros... - This means that packagers will be able to use the unversioned macros throughout their specfile. Requires and BuildRequires will look like this: Requires: %{default_python}-flask BuildRequires %{default_python}-setuptools Note: since BuildRequires need to be expanded in the minimal buildroot, it's necessary to define the %default_python macro in the specfile. Other way would be to define it in a macro file that would be added to the minimal buildroot, but that's neither very likely nor very nice :) I think this solution makes sense for *applications* that need to be built both in Fedora and in EPEL. Note that it'd also align with my EPEL-python3 proposal [1], in case such an app was to be moved to python3X in EPEL (%default_python would just be redefined to "python3X"). I also think that this approach should never be allowed for packaging "libraries", e.g. packages that have python-foo and python3-foo subpackages. Thoughts? Thanks, Slavek [1] https://fedoraproject.org/wiki/User:Bkabrda/EPEL7_Python3

6 19

Re: python-sig and retiring python3-dateutil
by Thomas Spura 27 Jan '15

27 Jan '15

Pete Travis <me(a)petetravis.com> schrieb am Mon Jan 26 2015 at 5:24:32 PM: > On 01/26/2015 08:53 AM, Zbigniew Jędrzejewski-Szmek wrote: > > Hi, > > > > since bug #1126521 seems to be progressing nicely, I think it would be > > nice to get in touch with python-sig, the maintainers of > > python3-dateutil, about retiring python3-dateutil and adding a python3 > > subpackage to python-dateutil. They might want to do it the other way > > around, which would be fine too, but either way, something should be > > arranged. > > > > I tried to sign up for the python-sig mailing list, but it is > > "private" and I haven't received any welcome letter, so I think I'm > > stuck in some moderation queue. > > > > Zbyszek > > Yes, I think it's a good time to bring up retiring python3-dateutil. > I've also requested membership for that list, and included the list > owners here, maybe they can expedite our requests. > As python-dateutil is not (yet) under the python-sig group maintainership [1], you could reach the maintainers of python-dateutil under ther -owner mail address (CC'ed). python-sig(a)fp.o is for all group maintained package maintainers and general python related questions are handled on python-devel(a)fp.o (also CC'ed). (Sorry for the confusion, the python-sig groupmaintainership started very recently and is still a work in process.) I'd say it depends, how python3 will be introduced in F22, which package should provide the subpackage and the other one should be retired. Maybe someone from [2] could comment on what they like to see in the future. I don't mind having two separate packages until we approach F22 and the future process is here. Greetings, Tom [1] https://admin.fedoraproject.org/pkgdb/package/python-dateutil/ [2] http://fedoraproject.org/wiki/Changes/Python_3_as_Default

3 2

Re: python-sig and retiring python3-dateutil
by Zbigniew Jędrzejewski-Szmek 27 Jan '15

27 Jan '15

On Mon, Jan 26, 2015 at 10:57:25AM -0700, Pete Travis wrote: > On 01/26/2015 10:45 AM, Thomas Spura wrote: > > Pete Travis <me(a)petetravis.com <mailto:me@petetravis.com>> schrieb am > > Mon Jan 26 2015 at 5:24:32 PM: > > > > On 01/26/2015 08:53 AM, Zbigniew Jędrzejewski-Szmek wrote: > > > Hi, > > > > > > since bug #1126521 seems to be progressing nicely, I think it > > would be > > > nice to get in touch with python-sig, the maintainers of > > > python3-dateutil, about retiring python3-dateutil and adding a > > python3 > > > subpackage to python-dateutil. They might want to do it the > > other way > > > around, which would be fine too, but either way, something should be > > > arranged. > > > > > > I tried to sign up for the python-sig mailing list, but it is > > > "private" and I haven't received any welcome letter, so I think I'm > > > stuck in some moderation queue. > > > > > > Zbyszek > > > > Yes, I think it's a good time to bring up retiring python3-dateutil. > > I've also requested membership for that list, and included the list > > owners here, maybe they can expedite our requests. > > > > > > As python-dateutil is not (yet) under the python-sig group > > maintainership [1], you could reach the maintainers of python-dateutil > > under ther -owner mail address (CC'ed). > > python-sig(a)fp.o is for all group maintained package maintainers and > > general python related questions are handled on python-devel(a)fp.o > > (also CC'ed). (Sorry for the confusion, the python-sig > > groupmaintainership started very recently and is still a work in process.) I applied for the group membership just in case. > > I'd say it depends, how python3 will be introduced in F22, which > > package should provide the subpackage and the other one should be retired. > > Maybe someone from [2] could comment on what they like to see in the > > future. I don't mind having two separate packages until we approach > > F22 and the future process is here. Building python3-du from python-du seems more standard, i.e. less confusing, but of course there's no difference from the technical side. No matter which name is chosen, it'd be nice to kill two birds with one stone, and do the merge together with an update of python3-dateutil to version 2.4. Scratch build is here: http://koji.fedoraproject.org/koji/taskinfo?taskID=8689481 > > [1] https://admin.fedoraproject.org/pkgdb/package/python-dateutil/ > > [2] http://fedoraproject.org/wiki/Changes/Python_3_as_Default > > We are proceeding with a planned Change[0] with Zbyszek acting on behalf > of the python-dateutil owner. He has already adapted > python-dateutil.spec to provide a python3-dateutil and, with the > relevant maintainers, we have been testing[1] dependent packages with a > scratch build of that package. > > The python3-dateutil package is around because upstream did not have > sources that would work with both anymore. That is no longer the case, > so the logical route to us is to retire python3-dateutil as a unique > srpm. The participation of they python-sig as package owners, and > insight on how to best transition, would be very helpful. > > My apologies if this is encroaching on python-sig territory; we saw a > need for change and took initiative. > > [0] https://fedoraproject.org/wiki/Changes/python-dateutil_2.x > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1126521 Zbyszek

1 0

Re: Python 3 as a Default - Status
by Bohuslav Kabrda 22 Jan '15

22 Jan '15

----- Original Message ----- > On 21. 1. 2015 at 01:44:32, Haïkel wrote: > > Thanks for the heads-up. > > > > As for the cloud image, if we switch to yum, python-urlgrabber won't > > be needed anymore. > > Umm, yes, it will be - yum uses python-urlgrabber internally. Also worth > mentioning that yum itself is not Python 3 compatible. I think that the original message should've said "switch to dnf". Was it not? > Thanks > Jan -- Regards, Slavek Kabrda

1 0

Re: Python 3 as a Default - Status
by Bohuslav Kabrda 22 Jan '15

22 Jan '15

----- Original Message ----- > On Tue, Jan 20, 2015 at 08:22:25AM -0500, Bohuslav Kabrda wrote: > > Hi all, > > since the "Python 3 as a Default" change [1] has been accepted a while ago > > and is scheduled for F22, I'd like to share with you the status. > > > > The proposed change [1] mentions several goals that should be reached to > > pronounce python3 the "default": > > 1) DNF is the default package manager instead of Yum, which only works with > > Python 2 > > 2) Python 3 is the only Python implementation in the minimal buildroot > > 3) Python 3 is the only Python implementation on the LiveCD > > 4) Anaconda and all of its dependencies run on Python 3 > > 5) cloud-init and all of its dependencies run on Python 3 > > > 5) cloud-init is a problem. Basically, Python 3 cloud-init is the only > > thing > > blocking the cloud images (*). Other packages are ready or being worked on. > > The problem here is that cloud-init upstream is really unresponsive about > > Python 3 porting (patch is submitted in their bug tracker [3]) - if someone > > knows these people, please help us by pinging them. > > > [3] https://code.launchpad.net/~harlowja/cloud-init/py2-3/+merge/225240 > > I forwarded this request to Red Hat's openstack team to see if anyone > there has contacts with the cloud-init maintainers. I've heard back > that Ubuntu is in the same boat, also wanting a python3 compatible > cloud-init in the near future. The author of the patch you mention > is actually a cloud-init maintainer so could merge stuff himself, > but he really needs others to review his patches before doing that. > None the less, it sounds like there might be a bit of interest and > movement upstream to try to get this porting work finished. Thanks a lot, I really appreciate that you did that! > Regards, > Daniel > -- > |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| > |: http://libvirt.org -o- http://virt-manager.org :| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| > -- > devel mailing list > devel(a)lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/devel > Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct -- Regards, Slavek Kabrda

1 0

Python 3 as a Default - Status
by Bohuslav Kabrda 21 Jan '15

21 Jan '15

Hi all, since the "Python 3 as a Default" change [1] has been accepted a while ago and is scheduled for F22, I'd like to share with you the status. The proposed change [1] mentions several goals that should be reached to pronounce python3 the "default": 1) DNF is the default package manager instead of Yum, which only works with Python 2 2) Python 3 is the only Python implementation in the minimal buildroot 3) Python 3 is the only Python implementation on the LiveCD 4) Anaconda and all of its dependencies run on Python 3 5) cloud-init and all of its dependencies run on Python 3 Before I go through the 5 goals, let me explain the approach that has been taken so far: There are basically two types of packages (from Fedora's point of view) that depend on Python: "libraries" and "applications" (note that the distinction is intentionally not very clear, there are packages that are both) "Libraries" are, simply put, Python extension modules, those that live in %{python2_sitelib} and %{python2_sitearch} in python2 builds. "Libraries" were receiving both upstream and downstream care from the people working on the change - upstream porting and downstream python3- subpackage additions, to also put files in %{python3_sitelib} and %{python3_sitearch}. We could afford to do the subpackage additions downstream during F21 lifecycle, since this doesn't change anything, it's just one more binary RPM that's spit out of the SRPM build process. "Applications" are stuff that users run (an important characteristic is that users don't care under which Python the application runs) - like Anaconda. "Applications" have been receiving some upstream patches, but weren't rebuilt with Python 3 yet. The reason for not rebuilding yet is that we first wanted to make sure that we've ported all "applications" upstream to be able to switch them all to Python 3 in Fedora at once - we wanted to be sure we wouldn't introduce both Python runtimes to livemedia, cloudimages, etc. As of now, Python 2 is still the suggested Python runtime to build "applications" against. We will start filing bugs to rebuild against Python 3 once we're sure that the switch can safely happen (which I think is now according to the points below). Let's go through the 5 goals: 1) DNF will be the default package manager for F22 [2], so everything is ok here. 2) One of our goals was to not make minimal buildroot larger. The only package from minimal buildroot requiring python (python-libs to be precise) is gdb, which is compatible with python3 in upstream (but it's considered to be an "application", so it hasn't been rebuilt yet). So minimal buildroot is fine. 3) As for LiveCD, we now have more of these - Workstation, Server and all the various flavours and spins. Let me go through Workstation and Server: Workstation: Fedora LiveCDs have, since at least Fedora 20, included both Python 2 and Python 3 runtimes (the primary reason for having Python 3 back then was AFAICS speech-dispatcher, which is Python 3 only in upstream). Although we may not be able to port all libraries and applications (but we may, there's still chance!) from workstation livecd to Python 3, the fact that it already ships both runtimes makes me think that we should build as much apps as possible with Python 3. Server: Because of giants like samba and freeipa, I think we won't be able to achieve python2-free server livecd for F22. But as it is with Workstation, I think we should proceed and build as much with Python 3 as possible. 4) Anaconda is work in progress. I'm communicating with Anaconda devs quite regularly and everything looks promising. 5) cloud-init is a problem. Basically, Python 3 cloud-init is the only thing blocking the cloud images (*). Other packages are ready or being worked on. The problem here is that cloud-init upstream is really unresponsive about Python 3 porting (patch is submitted in their bug tracker [3]) - if someone knows these people, please help us by pinging them. Attached to this mail, you'll find 3 files (fedora-cloud-base.txt, fedora-install-server.txt, fedora-live-workstation.txt). Each one of them is a result of automated script that tells with quite a good certainty what Python dependent packages are on a livecd/image generated by a kickstart with the same name. The files have two sections: "Good" and "Bad". "Good" packages are either "libraries" that have python3- subpackage or "applications" that have already been built with Python 3. "Bad" packages are either "libraries" not having python3- subpackage or "applications" built with Python 2. Note that lots of applications are now "bad" even though their code is Python 3 compatible, they just haven't been built with Python 3 yet. Also, "Bad" contains packages that will not be ported at all and will disappear from livecd, e.g. pyliblzma or yum. We're tracking all packages needed for Workstation and Cloud images at [4] - feel free to grab anything unassigned there or help us pushing our patches upstream. All help is welcome! All in all, I think we're in a good shape and I suggest that we move on by building all current "applications" (those that are Python 3 compatbile in upstream) with Python 3. I already suggested a change to Python guidelines that all *new* "applications" should be built with Python 3 if possible [5]. (*) Assuming cloud images are created out of fedora-cloud-base.ks, which I'd like someone to confirm. -- Regards, Slavek Kabrda [1] http://fedoraproject.org/wiki/Changes/Python_3_as_Default [2] http://fedoraproject.org/wiki/Changes/ReplaceYumWithDNF [3] https://code.launchpad.net/~harlowja/cloud-init/py2-3/+merge/225240 [4] https://fedoraproject.org/wiki/User:Churchyard/python3 [5] https://fedorahosted.org/fpc/ticket/475

2 1

python-sig group maintainership working in pkgdb
by Thomas Spura 15 Jan '15

15 Jan '15

Hi, it took a bit longer than expected to get the FAS group sorted [1], yet it should be possible now to add the "group::python-sig" to existing and new python packages [2-3]. So feel free to add packages to: * add group::python-sig to python related packages * add yourself to the python-sig FAS group (see [1] for direct links to it) * subscribe to the python-sig mailing list, where mails about the packages go to. Greetings, Thomas [1] https://fedoraproject.org/wiki/SIGs/Python#Python_SIG_FAS_group [2] https://lists.fedoraproject.org/pipermail/rel-eng/2014-December/019070.html [3] https://bugzilla.redhat.com/show_bug.cgi?id=1167830#c9

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel January 2015