python-devel August 2016

python-devel@lists.fedoraproject.org

22 participants
25 discussions

Python 3.6 and blocking-by-default os.urandom()
by Nick Coghlan 05 Jan '17

05 Jan '17

Hi folks, The long discussions upstream regarding the default behaviour of Python's os.urandom() API in Python 3.6 have come to an end, with Guido opting to make os.urandom() block implicitly waiting for the system RNG to be ready, even on Linux (where /dev/urandom doesn't do that). While I can see his point from a cross-platform language runtime design perspective, I'm still not sure it's the right answer for the Fedora system Python, as it doesn't play nice with ABRT, and goes against Ted T'so's decision to keep /dev/urandom non-blocking at the kernel level (risking confusion for folks that are Linux developers & operators first, and Pythonistas second). If "os.urandom() was called before system RNG is ready" throws an uncaught BlockingIOError, then we'll get a nice ABRT-friendly Python stack trace for people to debug (and perhaps figure out their VM entropy pool isn't being seeded properly, or their ARM hardware design needs a better source of entropy). By contrast, if it just blocks, then folks will be faced with a system hang, which they'll need to trace back to CPython being blocked on a kernel getrandom() call, and then infer from that that the system RNG isn't ready (for whatever reason), and they should probably do something about that. Patching this behaviour should be relatively straightforward - Python 3.5 currently makes this call in non-blocking mode and falls back to reading /dev/urandom directly in that case, so we'd just be forward porting the same logic to 3.6 and raising an exception instead of falling back to the file descriptor. Accordingly, what I propose we do is as follows: 1. Raise the concern in the F26 system-wide change proposal for migrating to Python 3.6 2. Apply the patch when the 3.6 beta releases are added to Fedora Rawhide 3. Decide whether or not to keep the patch based on ABRT results and other feedback on the Rawhide builds. Note that if the feedback on Rawhide shows that the change is helping people to find and diagnose VMs and hardware with improperly seeded entropy pools, that's a *good* thing: this proposed change is replacing Python 3.5's "/dev/urandom and os.urandom() may silently return statistically less-than-fully-random random numbers if the kernel entropy pool isn't seeded properly" with "os.urandom() will fail noisily in those cases, so you can either switch to the random module, or fix your entropy pool seeding". Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 9

Renaming python to python2
by Charalampos Stratakis 06 Sep '16

06 Sep '16

Hello Python-SIG, Currently I am working on some ideas regarding the python package. Nothing is too concrete yet, however the technical side should be fairly simple. The first thing is renaming python to python2. Currently the python packaging guidelines, suggest that we use the python_provides macro, for both python2 and python3 subpackages of whatever python module someone might be packaging. The macro for python3 subacpakages, currently, does nothing however sometime in the future it would be great if everything could switch to python 3. When this is done we can flip the switch in the macros and the python3 subpackage will also provide the python-<module> namespace, meaning essentially that when we speak about python we talk about python3, and the python binary will be a symlink to python3. In order to slowly start the procedure, a first step would be to rename the python package to python2. Here is a github repo with the current work so far: https://github.com/fedora-python/python2-spec/tree/py2rename Would love to hear some feedback on that. Regards, Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat

6 12

Python profile guided optimizations
by Charalampos Stratakis 31 Aug '16

31 Aug '16

Hello again, An new feature that I am currently working on for python is the profile-guided-optimization. Upstream: https://bugs.python.org/issue24915 Original bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=613046 Copr repo with builds: dnf copr enable cstratak/python-profiled-optimizations Supposedly it speeds up the python interpreter's performance by ~20%, however it would require some testing to figure out if that is indeed the case for the python rpm, and some tweaking might be required. It would be great if I could get some help from someone who is involved python's benchmarking. Planning first to try this with Python 3 and then also maybe Python 2. I'll rework the builds at copr later and I'll post a github repo as well with the proposed changes. Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat

1 0

BuildRequires on obsoleted packages provided by Python
by Charalampos Stratakis 31 Aug '16

31 Aug '16

Hello all, While checking out the SPEC file of python, it seems there were some packages that, while separate at some point, they got included in python's stdlib and then obsoleted as standalone packages (thus to cope with the change, python was obsoleting these packages and providing them as well in the SPEC). So every package that currently (Build)Requires any of these packages will essentially drag python with it. I will remove these provides soon, since the packages were orphaned long time ago, but the packages that still require them, will need to be fixed and (Build)Require python instead. Here is a github commit with these changes from a testing repo: https://github.com/fedora-python/python2-spec/commit/dfdd96e653d65ce6835955… And a list of the provided packages and the affected ones Distutils: None python-sqlite: cas yum python-ctypes: drobo-utils glusterfs-extra-xlators glusterfs-geo-replication python-smbios python-hashlib: pyrpkg python-uuid: dpm-server-mysql oz python2-celery python-argparse: R2spec catkin diskimage-builder euca2ools fedora-review feedstail gfal2-util glacier-cli grin hash-slinger imagefactory instack libstoragemgmt nordugrid-arc-nagios-plugins os-apply-config os-cloud-confic os-collect-confic os-net-config pyrpkg python-amqpclt python-catkin_pkg python-catkin_tools python-cloudservers python-gear python-novaclient python-postman python-requestbuilder python-rosdistro python-rospkg python-sparklines python2-oslo-config repo_manager rpkg vdsm Depending on feedback here I will follow (or not) the mass bug filling procedure so maintainer fix their packages. The reasoning behind this change, at the current time, is that I intent to rename python to python2 soon, which will lead to a re-review of python, so at the moment trying to have things as clear and consistent as possible. Plans for that change is only for rawhide (although it would be nice for f25 as well). Regards, Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat

1 1

Review swap python-django-netjsongraph
by Germano Massullo 30 Aug '16

30 Aug '16

Hi, is anybody interested in a review swap? python-django-netjsongraph - Reusable django app for collecting and visualizing network topology https://bugzilla.redhat.com/show_bug.cgi?id=1369213

1 0

Python 3 transition at Facebook
by Craig Rodrigues 27 Aug '16

27 Aug '16

This is not directly related to Fedora, but I thought people on this list might be interested in how the transition to Python 3.5 as default happened at Facebook: http://pyohio.org/schedule/presentation/212/ https://www.youtube.com/watch?v=nRtp9NgtXiA -- Craig

1 0

Python 3.5 for Fedora 23
by Miro Hrončok 27 Aug '16

27 Aug '16

Hi, As a follow up of our Flock discussion, I will build several Python versions in Copr, for development purposes (such as testing your code with tox on multiple Python versions). Those builds will be installable alongside regular python3/python packages and will be normal packages, no software collections etc. One flat package (i.e. no -libs, -devel...) with bundled setuptools and pip. First I've built Python 3.5 for Fedora 23, you can grab it here [1]. If you try to use tox with it, you'll have to use Python 3 version (python3-tox is the package and unfortunately also the command), due to a bug in virtualenv [2]. I would appreciate any feedback on the build, so I can build python34 package for Fedora 24+ in similar manner soon. Also python33 and python26 are planned. If those builds prove themselves useful I'll try to put them in Fedora (with a strict guideline that forbids any other package to depend on them). [1] https://copr.fedorainfracloud.org/coprs/g/python/python35/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=1365941 -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

5 11

Automatic Provides: Discussion summary and plan
by Petr Viktorin 26 Aug '16

26 Aug '16

To help automatic building RPMs from native Python packages, we need an automatic way to record the software's upstream name (dist name, name on PyPI) in Fedora packages. For this, we are using RPM's automatic dependency generator written by Per Øyvind Karlsen and Neal Gompa, which automatically scans ".dist-info" and similar files and generates virtual Provides, currently in the form "pythonX.Ydist(name)". (The generator was originally developed for Mandriva and Mageia; finally it's coming to Fedora as well!) The Fedora Change page for this feature is here: https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Pa… Unfortunately, while implementing this, we failed to understand how the virtual provides work with source RPMs. This leads to the situation that, as currently planned, the automatic provides are only useful in "Requires" lines, not in "BuildRequires". The problem is that if a requirement like "python3.5dist(name)" is present in a SRPM, the SRPM can't be rebuilt on systems with other Python versions, which would provide, say, "python3.6dist(name)". For these systems, the SRPM would need to be rebuilt, which not all tools do. Igor Gnatenko (ignatenkobrain) explained this on IRC on #fedora-python, after which we had a long discussion about the problem and possible solutions. We're sorry for not getting this earlier -- despite several people mentioning it (including Neal who wrote the generator)! The fix is to enable "--majorver-provides" in the dependency generator, so that each package will provide two forms of the virtual provide: python3.5dist(name) python3dist(name) The full version should then be used for runtime Requires, while the one with major version only should be used for BuildRequires. We'll wrap this up in easy-to use (and hopefully future-proof) macros: * One macro for getting the canonical (normalized) dist-name: %{python_dist_name NAME} * Four macros for adding Requires and BuildRequires lines (which use the python_dist_name macro above): %{requires_python3_dist NAME} => Requires: python3.Ydist(name) %{buildrequires_python3_dist NAME} => BuildRequires: python3dist(name) and similarly for Python 2. An alternative would be macros that don't include the keyword "Requires:" or "BuildRequires:". This would result in specfiles with lines like: BuildRequires: %{python3_dist_br name} Requires: %{python3_dist name} This would be susceptible to people copying the Requires line, adding "Build" to the front, and forgetting to change the macro as well, leading to a hard-to-catch failure (working binary RPMs but SRPMs that fail to rebuild on other distro versions). Macros that include the keyword are more robust to copy-pasting. Since the %buildrequires_python3_dist and %python_dist_name macros will be used in the SRPM, they'll need to go in macros.python-srpm to be present in the default buildroot. The %requires_python3_dist macro (and %pythonX_version) can live in macros.pythonX. Following is the new road plan: # Plan for Fedora 25: * The 5 macros will be created and deployed. * The --majorver-provides swich for the Provides generator in Fedora RPM will be enabled * Fedora Packaging Guidelines for Python will be amended: * The addition of the pythonX.Ydist(..) tags will be explained. * The %{python_dist_name} macro will be advertised. * The %{requires_pythonX_dist} macros will be advertised for use in generating Requires tags. * It will be explained that, at this time, it is impossible to generate BuildRequires without the providing package being rebuilt, so the %{buildrequires_pythonX_dist} macros will be discouraged for now. * See if we can get in another targeted mass rebuild. If yes, continue with the f26 plan early. # Plan for Fedora 26: * All Provides will be regenerated in the regular Fedora 26 mass rebuild * Change Python guidelines so the %{buildrequires_pythonX_dist} macros are now encouraged. -- Petr Viktorin

8 20

__pycache__ in python2 directory? O_o
by Igor Gnatenko 25 Aug '16

25 Aug '16

RPM build errors: Installed (but unpackaged) file(s) found: /usr/lib/python2.7/site-packages/__pycache__/site.cpython-35.pyc This is from build of Flask. Looks really weird. Also it has some weird file inside: /usr/lib/python3.5/site-packages/flask/testsuite/test_apps/lib/python2.5/site-packages/SiteEgg.egg Should we skip such things from RPM generator or we should remove that or what? Thoughts? -- -Igor Gnatenko

4 4

Review Request: python35 - Version 3.5 of the Python programming language
by Miro Hrončok 25 Aug '16

25 Aug '16

Hi, can somebody outside our Python Maint team please do the review? https://bugzilla.redhat.com/show_bug.cgi?id=1369688 Thanks -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel August 2016