Python 3.7's Deterministic pycs
by Petr Viktorin
Hello!
The first beta for Python 3.7 is out. It will hopefully get into Fedora
soon as python37.
After it comes out of beta, we'll upgrade python3 to it.
The What's New list is at: https://docs.python.org/3.7/whatsnew/3.7.html
One thing that's interesting for packagers is PEP 552: Deterministic
pycs: https://www.python.org/dev/peps/pep-0552/
Let me summarize in my own words.
A new opt-in mode for byte-compilation makes .pyc (bytecode cache) files
depend only on the contents of the corresponding source file.
If we use this, it will slow down imports, because the whole source file
would need to be read and hashed in order to verify if a .pyc file is
valid. (Currently, metadata like the modification time and file size is
used.)
To speed things up, there's an option, UNCHECKED_HASH, which skips cache
validation entirely. Using this would mean that if you modify a .py
source file installed by RPM, the changes wouldn't take effect (the .py
contents would only be shown in tracebacks).
Modifying installed files in production is extremely bad practice, of
course, but it's quite useful for debugging on throw-away systems. If we
adopt UNCHECKED_HASH, anyone doing it will have to remember to remove
the corresponding .pyc file.
Honestly, I'm not sure we want to use this in Fedora. Is anyone here
into reproducible builds, to make a better argument for this?
--
Petr Viktorin
4 years, 4 months
Getting rid of python3- backports packages
by Miro Hrončok
During the 3.7 boostrapping of the interstitial sequence of Python 3
packages, I have noticed it includes a lot of packages that are only
intended as Python 3 stdlib backports for older Pythons.
https://pypi.org/project/contextlib2/ is a backport of the standard
library’s contextlib module to earlier Python versions
https://pypi.org/project/linecache2/ is a backport of linecache to older
supported Pythons.
https://pypi.org/project/pathlib2/ The goal of pathlib2 is to provide a
backport of standard pathlib module which tracks the standard library
module, so all the newest features of the standard pathlib can be used
also on older Python versions.
https://pypi.org/project/traceback2/ is a backport of traceback to older
supported Pythons.
https://pypi.org/project/unittest2/ is a backport of the new features
added to the unittest testing framework in Python 2.7 and onwards.
https://pypi.org/project/simplejson/ is the externally maintained
development version of the json library included with Python 2.6 and
Python 3.0, but maintains backwards compatibility with Python 2.5.
https://pypi.org/project/funcsigs/ is a backport of the PEP 362 function
signature features from Python 3.3
https://pypi.org/project/mock/ is now part of the Python standard
library, available as unittest.mock in Python 3.3 onwards.
Now I see fairly no real reason to have python3-contextlib2,
python3-linecache2 etc. packaged and maintained in Fedora. The only
reason there is that other packages actually depend on them.
I think we should actively change the packages not to depend on those
and eventually get rid of them. This involves changing how upstreams are
using those. Thoughts?
Here is some dependency info:
$ whatrequires python3-contextlib2
python3-futurist
python3-pytest-virtualenv
python3-vcrpy
python-pytest-shutil
python-traceback2
$ whatrequires python3-linecache2
python3-traceback2
$ whatrequires python3-pathlib2
pipenv
python3-spyder
python-testpath
$ whatrequires python3-traceback2
python3-testtools
python3-unittest2
$ whatrequires python3-unittest2
python3-testtools
ProDy
cloud-init
compose-utils
fedpkg
libabigail
pungi
pyephem
python-agate
python-agate-excel
python-autobahn
python-billiard
python-case
python-funcsigs
python-leather
python-linecache2
python-mimerender
python-mock
python-pyclipper
python-pykafka
python-ripozo
python-rsa
python-s3transfer
python-traceback2
python-tw2-core
python-unicodecsv
$ whatrequires python3-simplejson
copr-cli
copr-frontend
copr-rpmbuild
gnome-code-assistance
kismon
python2-pystache (huh?)
python3-canonicaljson
python3-cherrypy
python3-cinderclient
python3-cornice
python3-geojson
python3-jsonpickle
python3-manilaclient
python3-mwclient
python3-neutronclient
python3-novaclient
python3-octaviaclient
python3-osc-lib
python3-oslo-serialization-tests
python3-peewee
python3-rabbitvcs
python3-resultsdb_api
python3-scales
python3-tackerclient
python3-troveclient
python3-tw2-core
python3-tweepy
python3-twitter
python3-uri-templates
python3-x2go
translate-toolkit
pymol
python-anyjson
python-arrow
python-marshmallow
python-mygpoclient
python-openstackclient
python-sphinx
$ whatrequires python3-funcsigs
module-build-service
python3-debtcollector
python3-mock
python3-octaviaclient
python3-oslo-utils
python3-tinyrpc
python3-tinyrpc-tests
odcs
python-ZEO
python-msgpack
python-mwclient
python-ripe-atlas-cousteau
$ whatrequires python3-mock
copr-keygen
module-build-service
python3-ara-tests
python3-certbot
python3-certbot-dns-*
python3-conda
python3-congressclient-tests
python3-keystoneclient-tests
python3-kubernetes-tests
python3-magnumclient-tests
python3-music21
python3-netdiff
python3-octaviaclient-tests
python3-osc-lib-tests
python3-oslo-cache-tests
python3-oslo-log-tests
python3-oslo-middleware-tests
python3-oslo-policy-tests
python3-oslo-rootwrap-tests
python3-oslo-serialization-tests
python3-oslo-service-tests
python3-oslo-versionedobjects
python3-oslo-vmware-tests
python3-oslotest
python3-pytest-spec
python3-pywbem
python3-sphinx
python3-testfixtures-tests
python3-testify
python3-testinfra
python3-tinyrpc-tests
python3-vcrpy
waiverdb
...320 source packages buildrequire this one!
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 5 months
What is the criterion for Python 3.7 side tag merge?
by Miro Hrončok
The Python 3.7 rebuild in the f29-python side tag is currently:
- 2168 built
- ~170 FTBFS
- ~120 blocked by dependencies that FTBFS
Ideally, this would all get fixed before the merge, but that's
unrealistic, there are packages that are FTBFS for unrelated reasons, etc.
What is the criterion to merge the side tag?
The critpath installs fine (except stuff that cannot be installed due to
yum-deprecated vs yum collisions and gnome-software that has some broken
dependency on PackageKit).
Stuff that doesn't build and sounds important:
- ceph (also FTBFS in regular rawhide)
- libreoffice (also FTBFS in regular rawhide)
- orca (blocked by pyatspi, looking at it ATM)
- kernel-tools
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 5 months
Re: Heads up: Python 3.7 rebuild in progress
by Miro Hrončok
On 21.6.2018 06:12, Jerry James wrote:
> On Wed, Jun 20, 2018 at 1:36 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
>> Unfortunately this is blocked by failing:
>>
>> python-manuel
>> https://koji.fedoraproject.org/koji/taskinfo?taskID=27740145
>> https://bugzilla.redhat.com/show_bug.cgi?id=1593132
>
> I had planned to look at that tonight. However, just as I arrived
> home from work tonight, as I was pulling into my driveway, my
> neighbor's propane grill exploded, catching his deck and then his
> house on fire. Fortunately there were no serious injuries.
I'm glad nobody go seriously hurt.
> Is there a way to do a mock build against the packages currently in f29-python?
fedpkg mock-config --target f29-python
This gives you a mock config you can use with mock -r or fedpkg
mockbuild --root. (Untested.)
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 5 months
Re: Intent to drop python2-notebook
by Miro Hrončok
On 5.6.2018 18:21, Jerry James wrote:
> On Tue, Jun 5, 2018 at 9:30 AM, Miro Hrončok <mhroncok(a)redhat.com
> <mailto:mhroncok@redhat.com>> wrote:
>
> On 11.5.2018 17:54, Miro Hrončok wrote:
>
> python2-notebook is a leaf package (nothing in Fedora
> (build)requires it). I'd like to drop it from rawhide. Any
> objections?
>
> Dropped.
>
>
> Uh oh. I just added both a BR and an R from sagemath to
> python2-notebook a couple of days ago, because sagemath does require
> it.
Does it require all of it? It's not a library. Can we see what exactly
is required and why maybe? How is it used - via Python imports or via
the command form /usr/bin?
> Can we bring it back? I can help maintain it if maintenance cycles
> are an issue.
We can, but please check the above first.
The issue is that upstream no longer seem to care about Python 2. Last
time, it had some SyntaxErrors on legacy Python. Fortunatelly, only in
some tests, so I've dropped those from the py2 package, but I just
didn't want to deal with this any more.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 5 months
Intent to drop python2-notebook
by Miro Hrončok
python2-notebook is a leaf package (nothing in Fedora (build)requires
it). I'd like to drop it from rawhide. Any objections?
This **does not** affect the ability to run Jupyter Notebook (via the
jupyter-notebook command) with Python 2 kernel (from the
python2-ipykernel package).
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 5 months