Python 3.4, ensurepip and wheels
by Bohuslav Kabrda
Hi all,
so I started to work on packaging Python 3.4 beta 1 and I wanted to create ensurepip patch also acceptable for upstream - Toshio and Nick talked about that at [1]. I tried to analyze the situation and here is what I have. First some facts:
- with ensurepip, Python 3.4 bundles setuptools and pip wheel archives
- when ensurepip is run, it installs (or upgrades) currently installed pip from the bundled wheels
A way to solve this "properly" in a distro (as mentioned in [1]) would be:
- throw away bundled setuptools and pip wheels when building Python
- making python3 require python-pip
- patching ensurepip (this is the fun part):
-- if ensurepip is run in venv:
--- we will need to reconstruct the wheels and install them into venv; or just grab the RECORD files and use them as a list of files we need to copy into venv
--- to reconstruct the wheels from installed RPMs, it'd be a good idea to package python-{setuptools,pip} as wheels (with the ".dist-info" directory and all the wheel goodies inside) so that we have the RECORD files
--- to package setuptools and pip as wheels, they'll have to BR: python3-pip (I saw Toshio's notes from flock and I agree with the part about how Fedora should use wheels [2])
-- if ensurepip is run systemwide:
--- if "--upgrade" is used, should this run "yum update python3-pip"?
Now, when new X.Y version Python is introduced, we will have to:
- bootstrap the new python3 interpreter to not require python3-pip, since python3-pip will require python(abi)=X.(Y-1)
- then we will need to build setuptools and pip *not as wheels*, because we still won't have pip that would work with our new Python X.Y - that means boostrapping them too and building them using the current "setup.py install" approach
- when we will have the new pip, we will be able to reenable python3 requires for python3-pip
- and we will also be able to rebuild setuptools and pip *as wheels*
There are also some more minor issues, that are worth mentioning:
- We typically won't update pip after Fedora is released - if there are security issues, we will backport patches and bump the release. The problem with this is that "ensurepip --upgrade" is not able to see Fedora releases and wouldn't work in venv, leaving it vulnerable. I think we have three options here:
-- Either we use some Fedora-specific RPM magic, which won't be acceptable for upstream
-- Or we will use a nice feature of wheel RECORD files - they keep file hashes, so we can check venv pip against system pip and see if something has changed when "ensurepip --upgrade" is run
-- Or we can just copy whole pip again into venv, regardless of what is currently there (doesn't seem very nice..., but it's simple and in the end it works as the above method)
- We can't copy the *.pyc and *.pyo files from system to venv, because they have file locations hardcoded in them, that are used in tracebacks. We can solve this by just copying *.py files and running py_compile in our ensurepip.
I hope I covered all the important points. Basically, we can make this work in a way acceptable for upstream, if we package setupttols and pip as wheels. It'll require some extra effort, but I think it's worth it.
Thoughts? Anyone has better/simpler ideas?
Thanks a lot,
Slavek.
[1] https://lists.fedoraproject.org/pipermail/python-devel/2013-November/0005...
[2] https://fedoraproject.org/wiki/User:Toshio/Flock2013_Python_Guidelines#Wh...
9 years, 9 months
Packaging Python 3.4.0
by Bohuslav Kabrda
Hi all,
I started working on updating python3 to 3.4.0. So far, this is only in a private branch "python3.4" in dist-git, since I'll need to get a Change approved for Fedora 21. I already created the Change [1], but I was told that it will get on FESCo schedule when Fedora 21 schedule and future is more clear, so for now I'll just work on Python 3.4.0.
When Python 3.4.0 reaches beta1 (which means feature freeze, which also means that the bytecode format should no longer change) - November 24 [2] - I plan to create a repo in Copr so that everyone has access to built RPMs, and also I want to start rebuilding as many packages as possible to see how they work with Python 3.4.
As for the current state of Python 3.4 in the dist-git branch:
- It still seems a bit unclear what upstream will do with the sha3 implementation (although they will probably keep it), so I didn't rebase the fips patch yet, there is still plenty of time for that.
- pep453 [3] (pip bootstraping) hasn't been implemented yet. The pep also contains recommendation for distro packagers, so here's what we should do.
-- Make a circular dependency between python3 and python3-pip (ugly, will require bootstraping with every new Python minor version).
-- Definitely delete bundled CA certificates and use system certificates instead.
-- Maybe remove the bundled pip (although this goes against the pep) and use python3-pip. E.g. everything would work as expected, but under the wraps, python3-pip package would be used. So when doing security updates, we'd just fix python3-pip. It is however true that if Fedora's pip would be different from the upstream bundled one, users may experience some behaviour differencies. I'd like to hear your opinions on how we should approach this.
- I'll be continuously working in the dist-git branch, so if you try it and something doesn't work, I'm just working on it or I plan to :) Patches are welcome.
Thanks,
Slavek.
[1] https://fedoraproject.org/wiki/Changes/Python_3.4
[2] http://www.python.org/dev/peps/pep-0429/
[3] http://www.python.org/dev/peps/pep-0453/
9 years, 10 months
Changes for F22 -- Python 3 as Default
by Toshio Kuratomi
The Change page for python3 as default has a pretty empty entry for policies
and guidelines. Just, "discuss with FPC"; no list of what to actually change.
We should start working on fleshing that out.
Here's one that should be fairly easy:
* https://fedoraproject.org/wiki/Packaging:Python#Guidelines : Currently
reads:
If the executables provide the same functionality independent of whether
they are run on top of Python 2 or Python 3, then only one version of the
executable should be packaged. Currently it will be the python
2 implementation, but once the Python 3 implementation is proven to work,
the executable can be retired from the python 2 build and enabled in the
python 3 package. Be sure to test the new implementation. Transitioning from
python2 to python3 is left to individual package maintainers except for
packages in Fedora's critical path. For these, we want to port to python3
versions in the same Fedora release if possible.
Change that to:
If the executables provide the same functionality independent of whether
they are run on top of Python 2 or Python 3, then only one version of the
executable should be packaged. In F22 and later, this should be the Python 3
version. In F21 and earlier this is left to the individual package
maintainers '''except''' for packages in Fedora's critical path. Those should
remain on the Python 2 version so that we don't drag in both python2 and
python3 stacks on a minimal system.
I think this change could go into FPC's queue to approve now because the
differences are all per-version so it does not causes problems for packages
which are not yet thinking about F22.
@churchyard, as part of this we should probably come up with another list of
packages that do not have python3 versions for the packages in the critical
path. (btw, I'm much happier with the text of the latest round of bug
reports, thanks). (Note though -- I think fesco still likes us to pass
anything we're going to mass file bugs for to them (or to the devel mailing
list?) to review first because mass filings can affect a lot of people.
Final note: There were three changes to the python guidelines approved by
FPC in this ticket: https://fedorahosted.org/fpc/ticket/327#comment:9
Unfortunately, both bkabrda and I have been pulled away to work on SCL
drafts so the three changes have not been written up. If someone would be
willing to write up those changes as a new version of the guidelines page::
http://fedoraproject.org/wiki/Packaging:Python
I'd be able to make time to review that and merge it into the Guidelines.
Thanks,
-Toshio
9 years, 10 months
python-setuptools update in F20
by Toshio Kuratomi
A new setuptools upstream release (1.2) was made last week that adds support for
subversion 1.7 and 1.8. Subversion 1.7 shipped in Fedora 19 so the
subversion support in setuptools is currently broken in both F19 and F20.
Unless there's objections I'm definitely going to update setuptools in F20.
I'm quite hesitant about updating in F19 due to potential incompatibilities.
In particular, pip, buildout, and virtualenv would likely all need to update
(these were things that we found had minimum versions to work with the F20
setuptools package as per:
https://fedoraproject.org/wiki/Changes/Python_setuptools_0.7
)
Backwards incompatible changes were introduced between 0.9.8 and 1.0:
https://pypi.python.org/pypi/setuptools#id145 but these changes are unlikely
to affect any code that's currently in the wild.
Please speak now if you don't want this update to hit F20 or do want the
update to hit F19. My plan is to update the python-setuptools package in
F20 to 1.3 near the end of this week.
-Toshio
9 years, 10 months
Removing python-setuptools-devel backwards compat
by Toshio Kuratomi
In November of 2009, in time for Fedora 13, we solved a packaging issue in
the main python package that allowed us to remove a split in our
python-setuptools package. Instead of needing a separate
python-setuptools-devel subpackage for easy_install the python-setuptools
package could now contain all of the functionality of python-setuptools.
To ease with package migration we added an Obsoletes and Provides tag to the
python-setuptools package at that time so that packages, kickstarts, etc
which required python-setuptools-devel would not break.
Fast forward to the present day:
It's November of 2013. Rawhide is Fedora 21. I'd like to drop the
backwards compatibility Provides (and Obsoletes) from the python-setuptools
package. However, there are currently 166 packages BuildRequire'ing
python-setuptools-devel. If owners would care to fix them before I remove
the backwards compatibility there shouldn't be any surprises in mass
rebuilds or late-night package updates later. The changes are easy enough
that provenpackagers could step in if owners don't update.
Changes are simply:
-BuildRequires: python-setuptools-devel
+BuildRequires: python-setuptools
(Same for Requires: lines but AFAICT I fixed the last of those today)
Here's the list:
Listing by package:
| PyOpenGL | fcami, cicku
| TurboGears2 | toshio, lmacken, ralph, vicodan
| antlr | mizdebsk, mjakubicek
| babel | fschwarz, jcollie, nphilipp
| bodhi | lmacken
| bzr-fastimport | dcallagh
| catkin | rmattes
| certmaster | ssalevan, alikins, wakko666
| cloud-init | mattdm, apevec, pbrady, gholms
| clusterPy | volter
| cobbler | jimi, shenson
| django-typepad | lbazan
| euca2ools | gholms
| fedmsg | lmacken, ralph
| fedmsg-notify | lmacken, ralph
| firstaidkit | msivak
| func | gnat, ssalevan, alikins, wakko666
| fuse-python | peter
| gaupol | lucilanga
| geome | pwouters
| gflags | peter
| glances | madko
| gnome-shell-search-fedora-packages | ralph
| gnome-shell-search-github-repositories | ralph
| gnome-shell-search-pinboard | ralph
| grin | terjeros
| hgsvn | terjeros
| libconcord | swt2c, silfreed
| libpfm | wcohen
| nml | heffer
| nwsclient | spot
| openerp | leamas
| ovirt-engine-cli | oschreib, jhernand
| phatch | jcapik
| planet | limb
| protobuf | mizdebsk, jlaska, abbot, konradm
| pss | kushal
| pssh | terjeros
| pycolumnize | kushal
| pykickstart | bcl, clumens
| pymodbus | chkr
| pymol | timfenn
| pyrasite | lmacken
| python-Coherence | hadess, thias
| python-Levenshtein | dwayne
| python-PSI | topdog
| python-TurboMail | fschwarz, toshio, lmacken
| python-altgraph | fab
| python-amara | jamatos
| python-atfork | jlaska
| python-batchhttp | puiterwijk
| python-blivet | bcl, dlehman
| python-boto | robert, gholms
| python-bugzilla | dzickus, crobinso, wwoods
| python-clientform | lmacken
| python-confparser | dougsland, aquini
| python-ctags | kushal
| python-daap | jcollie
| python-decoratortools | toshio, lmacken
| python-demjson | thm
| python-di | msivak
| python-dictclient | ricky
| python-django-ajax-selects | lbazan
| python-django-threadedcomments | lbazan
| python-dotconf | mjakubicek
| python-dtopt | ralph, ricky
| python-editdist | pwouters
| python-elfdata | kushal
| python-enum | maxamillion
| python-fastimport | dcallagh
| python-feedcache | lmacken
| python-flup | till
| python-futures | terjeros
| python-gnutls | gnat, peter
| python-googlevoice | jcollie
| python-guppy | peter
| python-html2text | mschwendt, thl
| python-httplib2 | awjb, dchen
| python-iniparse | timlau
| python-isodate | jmatthews, jlaska
| python-jinja | toshio, thm
| python-kid | toshio, till, lmacken
| python-louie | thias
| python-markupsafe | kylev, lmacken
| python-mechanize | lmacken
| python-meh | clumens, vpodzime
| python-mpd | hguemar
| python-musicbrainz2 | jcollie, alexlan
| python-netifaces | rrix
| python-nss | jdennis
| python-numeric | rstrode, rhughes, alexlan, johnp, alexl, caolanm, ssp,
mbarnes, hadess, mclasen, caillon
| python-oauth | sdz
| python-openid | jcollie
| python-pandas | kushal
| python-peak-rules | kylev, lmacken
| python-peak-util-symbols | lmacken
| python-pmw | timfenn
| python-ptrace | terjeros
| python-pyramid | lmacken, ralph, rossdylan
| python-pysctp | nhorman
| python-rdfextras | pingou
| python-rdflib | dmalcolm, pingou
| python-remoteobjects | bkabrda
| python-repoze-lru | lmacken, ralph
| python-repoze-tm2 | lmacken
| python-repoze-what | lmacken
| python-repoze-what-plugins-sql | lmacken
| python-repoze-what-pylons | spot
| python-repoze-what-quickstart | spot
| python-repoze-who-friendlyform | spot
| python-repoze-who-plugins-sa | lmacken
| python-repoze-who-testutil | spot
| python-rosdep | rmattes
| python-rosinstall | rmattes
| python-rospkg | rmattes
| python-rpmfluff | jhutar
| python-ruledispatch | toshio, lmacken
| python-scripttest | mbacovsk
| python-shapely | volter
| python-shove | lmacken
| python-signalfd | jlaska
| python-simpy | sarantis
| python-sippy | peter
| python-spiffgtkwidgets | leamas
| python-sqlamp | mbacovsk
| python-suds | jortel
| python-sudsds | jskarvad
| python-sybase | yaneti
| python-tag | thias
| python-textile | thm
| python-tgext-admin | lmacken
| python-tgext-crud | lmacken
| python-translationstring | lmacken, ralph
| python-turbocheetah | toshio, lmacken
| python-tw-jquery | lmacken
| python-twill | thias
| python-typepad | lbazan
| python-unipath | terjeros
| python-urllib2_kerberos | tdabasin
| python-urwid | lmacken, fabiand, dcantrel
| python-vcstools | rmattes
| python-venusian | lmacken, ralph, rossdylan
| python-weberror | lmacken, ricky
| python-webflash | lmacken
| python-webhelpers | kylev
| python-webob1.1 | lmacken
| python-wsgiproxy | ricky
| python-wsgiref | lmacken
| python-xappy | lmacken
| python-xmpp | jcollie, peter
| python-yenc | konradm
| python-zope-deprecation | lmacken, ralph
| pywbem | miminar, ke4qqq, jsafrane
| pywebdav | sharkcz
| rednotebook | fab, cwickert
| sendKindle | kparal
| snake | jlaska, wwoods
| spambayes | pghmcfc
| squeal | dmalcolm
| supybot | ricky
| trac | limb, fschwarz, lmacken
| trac-tracnav-plugin | thm
| trash-cli | sundaram, ankursinha
| veusz | jsanders
| vhybridize | fab
| zapplet | ke4qqq
Listing by maintainer:
| abbot | protobuf
| alexl | python-numeric
| alexlan | python-numeric, python-musicbrainz2
| alikins | certmaster, func
| ankursinha | trash-cli
| apevec | cloud-init
| aquini | python-confparser
| awjb | python-httplib2
| bcl | pykickstart, python-blivet
| bkabrda | python-remoteobjects
| caillon | python-numeric
| caolanm | python-numeric
| chkr | pymodbus
| cicku | PyOpenGL
| clumens | pykickstart, python-meh
| crobinso | python-bugzilla
| cwickert | rednotebook
| dcallagh | python-fastimport, bzr-fastimport
| dcantrel | python-urwid
| dchen | python-httplib2
| dlehman | python-blivet
| dmalcolm | squeal, python-rdflib
| dougsland | python-confparser
| dwayne | python-Levenshtein
| dzickus | python-bugzilla
| fab | vhybridize, rednotebook, python-altgraph
| fabiand | python-urwid
| fcami | PyOpenGL
| fschwarz | babel, python-TurboMail, trac
| gholms | cloud-init, python-boto, euca2ools
| gnat | python-gnutls, func
| hadess | python-numeric, python-Coherence
| heffer | nml
| hguemar | python-mpd
| jamatos | python-amara
| jcapik | phatch
| jcollie | babel, python-googlevoice, python-xmpp, python-openid,
python-daap, python-musicbrainz2
| jdennis | python-nss
| jhernand | ovirt-engine-cli
| jhutar | python-rpmfluff
| jimi | cobbler
| jlaska | python-isodate, python-atfork, protobuf, python-signalfd, snake
| jmatthews | python-isodate
| johnp | python-numeric
| jortel | python-suds
| jsafrane | pywbem
| jsanders | veusz
| jskarvad | python-sudsds
| ke4qqq | zapplet, pywbem
| konradm | protobuf, python-yenc
| kparal | sendKindle
| kushal | pycolumnize, pss, python-elfdata, python-ctags, python-pandas
| kylev | python-markupsafe, python-webhelpers, python-peak-rules
| lbazan | python-typepad, python-django-ajax-selects,
python-django-threadedcomments, django-typepad
| leamas | openerp, python-spiffgtkwidgets
| limb | planet, trac
| lmacken | python-repoze-what-plugins-sql, python-peak-rules,
python-translationstring, python-ruledispatch, python-pyramid,
python-shove, python-kid, TurboGears2, python-wsgiref,
python-peak-util-symbols, python-weberror,
python-repoze-who-plugins-sa, python-repoze-lru,
python-decoratortools, python-repoze-tm2, python-webflash,
python-webob1.1, pyrasite, python-clientform, python-tgext-crud,
bodhi, python-xappy, python-repoze-what, python-markupsafe,
python-tw-jquery, trac, python-zope-deprecation, fedmsg,
python-tgext-admin, python-TurboMail, fedmsg-notify,
python-mechanize, python-urwid, python-venusian,
python-turbocheetah, python-feedcache
| lucilanga | gaupol
| madko | glances
| mattdm | cloud-init
| maxamillion | python-enum
| mbacovsk | python-sqlamp, python-scripttest
| mbarnes | python-numeric
| mclasen | python-numeric
| miminar | pywbem
| mizdebsk | antlr, protobuf
| mjakubicek | antlr, python-dotconf
| mschwendt | python-html2text
| msivak | python-di, firstaidkit
| nhorman | python-pysctp
| nphilipp | babel
| oschreib | ovirt-engine-cli
| pbrady | cloud-init
| peter | gflags, python-guppy, python-xmpp, python-sippy, python-gnutls,
fuse-python
| pghmcfc | spambayes
| pingou | python-rdfextras, python-rdflib
| puiterwijk | python-batchhttp
| pwouters | geome, python-editdist
| ralph | python-dtopt, python-zope-deprecation,
gnome-shell-search-fedora-packages, fedmsg,
gnome-shell-search-github-repositories, fedmsg-notify,
python-pyramid, python-venusian, python-translationstring,
TurboGears2, python-repoze-lru, gnome-shell-search-pinboard
| rhughes | python-numeric
| ricky | python-weberror, python-dtopt, python-wsgiproxy, supybot,
python-dictclient
| rmattes | python-rospkg, python-rosdep, python-rosinstall, catkin,
python-vcstools
| robert | python-boto
| rossdylan | python-pyramid, python-venusian
| rrix | python-netifaces
| rstrode | python-numeric
| sarantis | python-simpy
| sdz | python-oauth
| sharkcz | pywebdav
| shenson | cobbler
| silfreed | libconcord
| spot | nwsclient, python-repoze-who-friendlyform,
python-repoze-what-pylons, python-repoze-who-testutil,
python-repoze-what-quickstart
| ssalevan | certmaster, func
| ssp | python-numeric
| sundaram | trash-cli
| swt2c | libconcord
| tdabasin | python-urllib2_kerberos
| terjeros | pssh, python-futures, python-unipath, python-ptrace, grin,
hgsvn
| thias | python-tag, python-louie, python-twill, python-Coherence
| thl | python-html2text
| thm | python-textile, python-jinja, trac-tracnav-plugin, python-demjson
| till | python-flup, python-kid
| timfenn | python-pmw, pymol
| timlau | python-iniparse
| topdog | python-PSI
| toshio | python-TurboMail, python-ruledispatch, python-turbocheetah,
python-kid, python-jinja, TurboGears2, python-decoratortools
| vicodan | TurboGears2
| volter | python-shapely, clusterPy
| vpodzime | python-meh
| wakko666 | certmaster, func
| wcohen | libpfm
| wwoods | python-bugzilla, snake
| yaneti | python-sybase
-Toshio
9 years, 11 months