Python 3.6 and blocking-by-default os.urandom()
by Nick Coghlan
Hi folks,
The long discussions upstream regarding the default behaviour of Python's
os.urandom() API in Python 3.6 have come to an end, with Guido opting to
make os.urandom() block implicitly waiting for the system RNG to be ready,
even on Linux (where /dev/urandom doesn't do that).
While I can see his point from a cross-platform language runtime design
perspective, I'm still not sure it's the right answer for the Fedora system
Python, as it doesn't play nice with ABRT, and goes against Ted T'so's
decision to keep /dev/urandom non-blocking at the kernel level (risking
confusion for folks that are Linux developers & operators first, and
Pythonistas second).
If "os.urandom() was called before system RNG is ready" throws an uncaught
BlockingIOError, then we'll get a nice ABRT-friendly Python stack trace for
people to debug (and perhaps figure out their VM entropy pool isn't being
seeded properly, or their ARM hardware design needs a better source of
entropy). By contrast, if it just blocks, then folks will be faced with a
system hang, which they'll need to trace back to CPython being blocked on a
kernel getrandom() call, and then infer from that that the system RNG isn't
ready (for whatever reason), and they should probably do something about
that.
Patching this behaviour should be relatively straightforward - Python 3.5
currently makes this call in non-blocking mode and falls back to reading
/dev/urandom directly in that case, so we'd just be forward porting the
same logic to 3.6 and raising an exception instead of falling back to the
file descriptor.
Accordingly, what I propose we do is as follows:
1. Raise the concern in the F26 system-wide change proposal for migrating
to Python 3.6
2. Apply the patch when the 3.6 beta releases are added to Fedora Rawhide
3. Decide whether or not to keep the patch based on ABRT results and other
feedback on the Rawhide builds.
Note that if the feedback on Rawhide shows that the change is helping
people to find and diagnose VMs and hardware with improperly seeded entropy
pools, that's a *good* thing: this proposed change is replacing Python
3.5's "/dev/urandom and os.urandom() may silently return statistically
less-than-fully-random random numbers if the kernel entropy pool isn't
seeded properly" with "os.urandom() will fail noisily in those cases, so
you can either switch to the random module, or fix your entropy pool
seeding".
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 8 months
Renaming python to python2
by Charalampos Stratakis
Hello Python-SIG,
Currently I am working on some ideas regarding the python package.
Nothing is too concrete yet, however the technical side should be fairly simple.
The first thing is renaming python to python2.
Currently the python packaging guidelines, suggest that we use the python_provides macro, for both python2 and python3 subpackages of whatever python module someone might be packaging. The macro for python3 subacpakages, currently, does nothing however sometime in the future it would be great if everything could switch to python 3. When this is done we can flip the switch in the macros and the python3 subpackage will also provide the python-<module> namespace, meaning essentially that when we speak about python we talk about python3, and the python binary will be a symlink to python3. In order to slowly start the procedure, a first step would be to rename the python package to python2.
Here is a github repo with the current work so far:
https://github.com/fedora-python/python2-spec/tree/py2rename
Would love to hear some feedback on that.
Regards,
Charalampos Stratakis
Associate Software Engineer
Python Maintenance Team, Red Hat
7 years
Python profile guided optimizations
by Charalampos Stratakis
Hello again,
An new feature that I am currently working on for python is the profile-guided-optimization.
Upstream: https://bugs.python.org/issue24915
Original bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=613046
Copr repo with builds: dnf copr enable cstratak/python-profiled-optimizations
Supposedly it speeds up the python interpreter's performance by ~20%, however it would require some testing to figure out if that is indeed the case for the python rpm, and some tweaking might be required. It would be great if I could get some help from someone who is involved python's benchmarking. Planning first to try this with Python 3 and then also maybe Python 2. I'll rework the builds at copr later and I'll post a github repo as well with the proposed changes.
Charalampos Stratakis
Associate Software Engineer
Python Maintenance Team, Red Hat
7 years
BuildRequires on obsoleted packages provided by Python
by Charalampos Stratakis
Hello all,
While checking out the SPEC file of python, it seems there were some packages that, while separate at some point, they got included in python's stdlib and then obsoleted as standalone packages (thus to cope with the change, python was obsoleting these packages and providing them as well in the SPEC). So every package that currently (Build)Requires any of these packages will essentially drag python with it.
I will remove these provides soon, since the packages were orphaned long time ago, but the packages that still require them, will need to be fixed and (Build)Require python instead.
Here is a github commit with these changes from a testing repo:
https://github.com/fedora-python/python2-spec/commit/dfdd96e653d65ce68359...
And a list of the provided packages and the affected ones
Distutils: None
python-sqlite:
cas
yum
python-ctypes:
drobo-utils
glusterfs-extra-xlators
glusterfs-geo-replication
python-smbios
python-hashlib:
pyrpkg
python-uuid:
dpm-server-mysql
oz
python2-celery
python-argparse:
R2spec
catkin
diskimage-builder
euca2ools
fedora-review
feedstail
gfal2-util
glacier-cli
grin
hash-slinger
imagefactory
instack
libstoragemgmt
nordugrid-arc-nagios-plugins
os-apply-config
os-cloud-confic
os-collect-confic
os-net-config
pyrpkg
python-amqpclt
python-catkin_pkg
python-catkin_tools
python-cloudservers
python-gear
python-novaclient
python-postman
python-requestbuilder
python-rosdistro
python-rospkg
python-sparklines
python2-oslo-config
repo_manager
rpkg
vdsm
Depending on feedback here I will follow (or not) the mass bug filling procedure so maintainer fix their packages.
The reasoning behind this change, at the current time, is that I intent to rename python to python2 soon, which will lead to a re-review of python, so at the moment trying to have things as clear and consistent as possible. Plans for that change is only for rawhide (although it would be nice for f25 as well).
Regards,
Charalampos Stratakis
Associate Software Engineer
Python Maintenance Team, Red Hat
7 years
Python 3.5 for Fedora 23
by Miro Hrončok
Hi,
As a follow up of our Flock discussion, I will build several Python
versions in Copr, for development purposes (such as testing your code
with tox on multiple Python versions).
Those builds will be installable alongside regular python3/python
packages and will be normal packages, no software collections etc. One
flat package (i.e. no -libs, -devel...) with bundled setuptools and pip.
First I've built Python 3.5 for Fedora 23, you can grab it here [1].
If you try to use tox with it, you'll have to use Python 3 version
(python3-tox is the package and unfortunately also the command), due to
a bug in virtualenv [2].
I would appreciate any feedback on the build, so I can build python34
package for Fedora 24+ in similar manner soon.
Also python33 and python26 are planned.
If those builds prove themselves useful I'll try to put them in Fedora
(with a strict guideline that forbids any other package to depend on them).
[1] https://copr.fedorainfracloud.org/coprs/g/python/python35/
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1365941
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
7 years
Automatic Provides: Discussion summary and plan
by Petr Viktorin
To help automatic building RPMs from native Python packages, we need an
automatic way to record the software's upstream name (dist name, name
on PyPI) in Fedora packages.
For this, we are using RPM's automatic dependency generator written by
Per Øyvind Karlsen and Neal Gompa, which automatically scans
".dist-info" and similar files and generates virtual Provides,
currently in the form "pythonX.Ydist(name)".
(The generator was originally developed for Mandriva and Mageia; finally
it's coming to Fedora as well!)
The Fedora Change page for this feature is here:
https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_...
Unfortunately, while implementing this, we failed to understand how the
virtual provides work with source RPMs. This leads to the situation
that, as currently planned, the automatic provides are only useful in
"Requires" lines, not in "BuildRequires".
The problem is that if a requirement like "python3.5dist(name)" is
present in a SRPM, the SRPM can't be rebuilt on systems with other
Python versions, which would provide, say, "python3.6dist(name)".
For these systems, the SRPM would need to be rebuilt, which not all
tools do.
Igor Gnatenko (ignatenkobrain) explained this on IRC on #fedora-python,
after which we had a long discussion about the problem and possible
solutions. We're sorry for not getting this earlier -- despite several
people mentioning it (including Neal who wrote the generator)!
The fix is to enable "--majorver-provides" in the dependency generator,
so that each package will provide two forms of the virtual provide:
python3.5dist(name)
python3dist(name)
The full version should then be used for runtime Requires, while the
one with major version only should be used for BuildRequires.
We'll wrap this up in easy-to use (and hopefully future-proof) macros:
* One macro for getting the canonical (normalized) dist-name:
%{python_dist_name NAME}
* Four macros for adding Requires and BuildRequires lines (which use the
python_dist_name macro above):
%{requires_python3_dist NAME} => Requires: python3.Ydist(name)
%{buildrequires_python3_dist NAME} => BuildRequires: python3dist(name)
and similarly for Python 2.
An alternative would be macros that don't include the keyword "Requires:" or
"BuildRequires:". This would result in specfiles with lines like:
BuildRequires: %{python3_dist_br name}
Requires: %{python3_dist name}
This would be susceptible to people copying the Requires line, adding
"Build" to the front, and forgetting to change the macro as well,
leading to a hard-to-catch failure (working binary RPMs but SRPMs that
fail to rebuild on other distro versions).
Macros that include the keyword are more robust to copy-pasting.
Since the %buildrequires_python3_dist and %python_dist_name macros will
be used in the SRPM, they'll need to go in macros.python-srpm to be
present in the default buildroot.
The %requires_python3_dist macro (and %pythonX_version) can live in
macros.pythonX.
Following is the new road plan:
# Plan for Fedora 25:
* The 5 macros will be created and deployed.
* The --majorver-provides swich for the Provides generator in Fedora RPM
will
be enabled
* Fedora Packaging Guidelines for Python will be amended:
* The addition of the pythonX.Ydist(..) tags will be explained.
* The %{python_dist_name} macro will be advertised.
* The %{requires_pythonX_dist} macros will be advertised for use in
generating Requires tags.
* It will be explained that, at this time, it is impossible to
generate BuildRequires without the providing package being
rebuilt, so the %{buildrequires_pythonX_dist} macros will be
discouraged for now.
* See if we can get in another targeted mass rebuild. If yes, continue
with the f26 plan early.
# Plan for Fedora 26:
* All Provides will be regenerated in the regular Fedora 26 mass rebuild
* Change Python guidelines so the %{buildrequires_pythonX_dist} macros
are now encouraged.
--
Petr Viktorin
7 years, 1 month
__pycache__ in python2 directory? O_o
by Igor Gnatenko
RPM build errors:
Installed (but unpackaged) file(s) found:
/usr/lib/python2.7/site-packages/__pycache__/site.cpython-35.pyc
This is from build of Flask. Looks really weird.
Also it has some weird file inside:
/usr/lib/python3.5/site-packages/flask/testsuite/test_apps/lib/python2.5/site-packages/SiteEgg.egg
Should we skip such things from RPM generator or we should remove that
or what? Thoughts?
--
-Igor Gnatenko
7 years, 1 month