Building the python executable statically with libpython*.a for
better performance
by Miro Hrončok
Hello,
we've been recently approached by a colleague from Red Hat working on
performance (CCed).
According to their testing, Fedora Python performance could be improved by ~15%
by building /usr/bin/python* statically with libpython*.a. That sounds like a
worthy thing to do.
https://bugzilla.redhat.com/show_bug.cgi?id=1749479
Since Python 3.8 Python extension modules are no longer linked to libpython.so
and we can do the following:
* build /usr/bin/python3(.8) statically with libpython*.a
* build and ship libpython3.8.so.1.0 for packages that "embed" Python
The change in the python3 package is trivial:
https://src.fedoraproject.org/rpms/python3/pull-request/133
However it can have serious impact on Python extension modules that are linked
to libpython3.8.so.1.0 by various "nonstandard" build mechanisms or by compiling
code for Python extension module and code that embeds Python into one file.
We will likely propose a Fedora 32 Change for this, however I'm opening this
topic for discussion before we do so.
Testing the proposed Pull Request with your code is also helpful. Let me know
how can we make that easier (e.g. if you want a Copr or a Fedora 30/31 python38
package with this change).
WDYT?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
4 years, 1 month
cython issues on EL8
by Orion Poplawski
I could use some help with some cython issues building pandas for EL8.
First attempt:
https://koji.fedoraproject.org/koji/taskinfo?taskID=37887295
running build_ext
pandas._libs.algos: -> [['pandas/_libs/algos.c']]
Traceback (most recent call last):
File "setup.py", line 838, in <module>
**setuptools_kwargs
File "/usr/lib/python3.6/site-packages/setuptools/__init__.py", line
129, in setup
return distutils.core.setup(**attrs)
File "/usr/lib64/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib64/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib64/python3.6/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib64/python3.6/distutils/command/build_ext.py", line 339,
in run
self.build_extensions()
File "setup.py", line 407, in build_extensions
self.check_cython_extensions(self.extensions)
File "setup.py", line 402, in check_cython_extensions
src=src
Exception: Cython-generated file 'pandas/_libs/algos.c' not found.
Cython is required to compile pandas from a development
branch.
Please install Cython or download a release package of
pandas.
Compared to a rawhide build, it seems like the cython c files are not
being automatically regenerated at the start of the build. What
component is supposed to handle this?
Second attempt to run cython by hand generates another error:
https://koji.fedoraproject.org/koji/taskinfo?taskID=37887587
+ cython ./pandas/_libs/missing.pyx ./pandas/_libs/testing.pyx
./pandas/_libs/interval.pyx ./pandas/_libs/window.pyx
./pandas/_libs/internals.pyx ./pandas/_libs/join.pyx
./pandas/_libs/index.pyx ./pandas/_libs/writers.pyx
./pandas/_libs/groupby.pyx ./pandas/_libs/reduction.pyx
./pandas/_libs/hashtable.pyx ./pandas/_libs/parsers.pyx
./pandas/_libs/tslibs/conversion.pyx
./pandas/_libs/tslibs/np_datetime.pyx ./pandas/_libs/tslibs/period.pyx
./pandas/_libs/tslibs/timedeltas.pyx ./pandas/_libs/tslibs/ccalendar.pyx
./pandas/_libs/tslibs/strptime.pyx ./pandas/_libs/tslibs/resolution.pyx
./pandas/_libs/tslibs/fields.pyx ./pandas/_libs/tslibs/c_timestamp.pyx
./pandas/_libs/tslibs/offsets.pyx ./pandas/_libs/tslibs/timestamps.pyx
./pandas/_libs/tslibs/parsing.pyx ./pandas/_libs/tslibs/tzconversion.pyx
./pandas/_libs/tslibs/timezones.pyx
./pandas/_libs/tslibs/frequencies.pyx ./pandas/_libs/tslibs/nattype.pyx
./pandas/_libs/hashing.pyx ./pandas/_libs/tslib.pyx
./pandas/_libs/indexing.pyx ./pandas/_libs/sparse.pyx
./pandas/_libs/algos.pyx ./pandas/_libs/skiplist.pyx
./pandas/_libs/ops.pyx ./pandas/_libs/reshape.pyx ./pandas/_libs/lib.pyx
./pandas/_libs/properties.pyx ./pandas/io/sas/sas.pyx
./pandas/io/msgpack/_packer.pyx ./pandas/io/msgpack/_unpacker.pyx
Error compiling Cython file:
------------------------------------------------------------
...
cdef int status
with nogil:
status = tokenize_nrows(self.parser, nrows)
if self.parser.warn_msg != NULL:
print(self.parser.warn_msg, file=sys.stderr)
^
------------------------------------------------------------
pandas/_libs/parsers.pyx:932:44: Expected ')', found '='
It this because cython is too old in EL8?
TIA,
Orion
--
Orion Poplawski
Manager of NWRA Technical Systems 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 https://www.nwra.com/
4 years, 2 months
Python 3.8 porting help
by Richard Shaw
I maintain FreeCAD for Fedora and there appears to have been a change in
the API for how Python handles thread management in 3.8, specifically
PyInterpreterState which is giving the the following error when trying to
build FreeCAD:
In file included from
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.cpp:41:
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.inl: In function
'void Swig_python::cleanupSWIG_T(const char*)':
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.inl:75:31: error:
invalid use of incomplete type 'PyInterpreterState' {aka 'struct _is'}
75 | PyObject *modules = interp->modules;
| ^~
In file included from /usr/include/python3.8/genobject.h:11,
from /usr/include/python3.8/Python.h:121,
from
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/PyExport.h:42,
from
/builddir/build/BUILD/FreeCAD-0.18.3/src/Base/swigpyrun.cpp:25:
/usr/include/python3.8/pystate.h:20:8: note: forward declaration of
'PyInterpreterState' {aka 'struct _is'}
20 | struct _is;
| ^~~
Any tips?
I can provide a full build log or setup a COPR but it's somewhat difficult
since I'm building a whole new stack of some dependencies:
Coin3->Coin4 (in review)
- Which SoQt, SIMVoleon, OpenSceneGraph and Pivy need to be ported to.
Pyside->PySide2 (which implies Qt4->Qt5)
Thanks,
Richard
4 years, 2 months
[HEADS UP] Retiring python2 and introducing python27 later this week
by Miro Hrončok
We are retiring python2 and introducing python27 later this week. Rawhide only.
As for now, nothing should break, except python2-debug will exist no more.
Packages (build)requiring python2 or python2-devel should continue to work for
now. If not, let us know.
If you plan to keep a Python 2 package in Fedora 32+, talk to me, I can help you
draft an exception request.
Will post a reply here once it is done.
The following packages build subpackages for python2-debug (and should stop
doing it in Fedora 32+):
- gcc-python-plugin (FTBFS)
- python-mysql (PR exists)
- python-psycopg2 (notified in an existing bugzilla)
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
4 years, 2 months
Python 3.8.0b4 ready in rawhide and updates-testing
by Miro Hrončok
Python 3.8.0b4 is ready in rawhide (as python3).
The python38 package was updated in F31, F30 and F29:
https://bodhi.fedoraproject.org/updates/FEDORA-2019-d11594bf0a
https://bodhi.fedoraproject.org/updates/FEDORA-2019-4954d8773c
https://bodhi.fedoraproject.org/updates/FEDORA-2019-d58eb75449
Please test your code with Python 3.8.0b4 extensively and report issues.
-------- Forwarded Message --------
Subject: [Python-Dev] [RELEASE] ACTION REQUIRED: Python 3.8.0b4 now available
for testing
Date: Fri, 30 Aug 2019 11:09:37 +0200
From: Łukasz Langa <lukasz(a)langa.pl>
To: Python Committers <python-committers(a)python.org>, Python-Dev
<python-dev(a)python.org>, python-list(a)python.org, python-announce(a)python.org
It's time for the *last beta release* of Python 3.8. Go find it at:
https://www.python.org/downloads/release/python-380b4/
This release is the *last* of four planned beta release previews. Beta release
previews are intended to give the wider community the opportunity to test new
features and bug fixes and to prepare their projects to support the new feature
release. The next pre-release of Python 3.8 will be 3.8.0c1, the first release
candidate, currently scheduled for 2019-09-30.
*Call to action*
We strongly encourage maintainers of third-party Python projects to test with
3.8 during the beta phase and report issues found to the Python bug
tracker as soon as possible. Please note this is the last beta release, there
is not much time left to identify and fix issues before the release of 3.8.0. If
you were hesitating trying it out before, now is the time.
While the release is planned to be feature complete entering the beta phase, it
is possible that features may be modified or, in rare cases, deleted up until
the start of the release candidate phase (2019-09-30). Our goal is have no ABI
changes after beta 3 and no code changes after 3.8.0c1, the release candidate.
*To achieve that, it will be extremely important to get as much exposure for 3.8
as possible during the beta phase. That beta phase is coming to an end. Please
test now.*
Please keep in mind that this is a preview release and its use
is not recommended for production environments.
*Acknowledgments*
Many developers worked hard for the past four weeks to squash remaining bugs,
some requiring non-obvious decisions. Many thanks to the most active, namely
Raymond Hettinger, Steve Dower, Victor Stinner, Terry Jan Reedy, Serhiy
Storchaka, Pablo Galindo Salgado, Tal Einat, Zackery Spytz, Ronald Oussoren,
Neil Schemenauer, Inada Naoki, Christian Heimes, and Andrew Svetlov.
3.8.0 would not reach the Last Beta without you. Thank you!
- Ł
4 years, 3 months