Python 3.6, Fedora, and the "C" locale
by Nick Coghlan
Hi folks,
One of the minor irritations with running Python 3 applications in
Fedora containers is that those containers still default to the "C"
local by default - you have to force "C.UTF-8" or "en_US.UTF-8" in
order to tell Python that it should use UTF-8 for communicating with
system interfaces.
However, since 3.5, the CPython start-up sequence has treated claims
from the OS that it should use ASCII more skeptically, and enabled the
"surrogateescape" error handling by default on the standard streams.
Along similar lines, what do folks think of the idea of patching
Python 3.6 in Fedora to assume UTF-8 if it's told that it should use
ASCII to communicate with the OS? That changes the failure mode from
"interface problems happen any time you fail to configure your locale
correctly" to "interface problems happen any time you fail to
configure your locale correctly *and* your default locale uses an
encoding other than UTF-8".
That latter case then further breaks down into the following two situations:
- you use an ASCII-incompatible encoding like Shift-JIS or GB-18030,
in which case the old POSIX default of ASCII was already broken, and
the surrogateescape workaround in 3.5 probably didn't help much
- you use an ASCII-compatible encoding like koi8-r, which means the
surrogateescape workaround in 3.5 probably helped a bit, but libraries
like `click` would still have complained and refused to run
So I think doing this would be a nice usability improvement for users
of the system Python 3 installation on Fedora.
I also have an upstream motive for suggesting this, though: if we do
this in the more constrained environment of "Fedora users" and it
doesn't break the world, then that will help build a case for making
it the default upstream behaviour in Python 3.7.
Regards,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 7 months
Re: Packages FTBFS with Python 3.6
by Zbigniew Jędrzejewski-Szmek
On Tue, Jan 17, 2017 at 08:29:16AM -0700, Jerry James wrote:
> On Tue, Jan 17, 2017 at 7:08 AM, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl> wrote:
> > On Mon, Jan 16, 2017 at 10:27:19PM -0800, Adam Williamson wrote:
> >> On Tue, 2017-01-17 at 05:33 +0000, Zbigniew Jędrzejewski-Szmek wrote:
> >> > Oh, Jerry beat me to it by taking a different route and disabling most
> >> > test. That should work too, but I think it hides real problems with
> >> > the python3 code...
> >>
> >> Yeah. Please don't just disable tests. Tests exist for a reason.
> >
> > We need to disable *some* tests, at least if we hope to build this
> > quickly. It's just a question whether to disable more or less of
> > them.
> >
> > Most tests fail because of sympy/pyglet/xvfb/opengl interaction
> > (#904851), not from sympy error per se — the same tests run fine on a
> > "real" display.
>
> Oh, sorry if I stepped on somebody's toes. i didn't realize others
> were trying to fix sympy, too. I guess I should join the python SIG
> mailing list, but I'm already on far too many mailing lists...
No, no, there were no toes to step on. I didn't announce anywhere
that I'm working on this. And anyway, I think our work is complementary:
I didn't know about the patch you pulled in,
but otoh, I managed to fix some of the tests by passing options to xvfb.
I'll try to push some of my changes too, let's see if that builds.
> But I don't understand what you mean about me disabling tests. I
> haven't disabled any tests, much less "most tests". i did add a BR on
> pyglets, then discovered that that CAUSES test failures due to missing
> GLX support, and so I removed the pyglets BR again. Maybe a BR on
> mesa is needed as well as pyglets? I don't know.
>
> Anyway, I checked in an experimental patch in an attempt at fixing the
> test failures, and now there is a different test failure. :-( So
> either my experimental patch isn't good enough, or it works perfectly
> and has unmasked another problem. I probably won't have time to
> figure that out for a couple of days, so if somebody else is
> interested in looking at it, here is the latest build failure:
>
> https://koji.fedoraproject.org/koji/buildinfo?buildID=828867
Zbyszek
6 years, 8 months
Re: Packages FTBFS with Python 3.6
by Zbigniew Jędrzejewski-Szmek
On Mon, Jan 16, 2017 at 10:27:19PM -0800, Adam Williamson wrote:
> On Tue, 2017-01-17 at 05:33 +0000, Zbigniew Jędrzejewski-Szmek wrote:
> > On Tue, Jan 17, 2017 at 05:29:35AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
> > > On Wed, Jan 11, 2017 at 04:28:18AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
> > > > > python3-sympy-0:1.0-3.fc25.noarch
> > > >
> > > > Pfff, it still fails in the test suite :(
> > >
> > > I'm chipping away at this one. There are some failing python2 tests
> > > which I had to disable, but it seems that things are mostly with python2.
> > > But it seems that the python3 code is plain broken (bad non-absolute
> > > imports).
> >
> > Oh, Jerry beat me to it by taking a different route and disabling most
> > test. That should work too, but I think it hides real problems with
> > the python3 code...
>
> Yeah. Please don't just disable tests. Tests exist for a reason.
We need to disable *some* tests, at least if we hope to build this
quickly. It's just a question whether to disable more or less of
them.
Most tests fail because of sympy/pyglet/xvfb/opengl interaction
(#904851), not from sympy error per se — the same tests run fine on a
"real" display.
Zbyszek
6 years, 8 months
Re: Packages FTBFS with Python 3.6
by Zbigniew Jędrzejewski-Szmek
On Thu, Jan 05, 2017 at 02:45:37PM -0800, Adam Williamson wrote:
> On Wed, 2016-12-21 at 00:18 +0100, Miro Hrončok wrote:
> > python-sphinxcontrib-programoutput
>
> This one was abandoned by its upstream years ago. He actually entirely
> deleted the repository from github, though someone else put up a copy
> of it recently (they didn't pick up maintenance, just put up a
> 'historical copy').
>
> It's only used by certbot and python-acme (which is actually a part of
> certbot that gets its own releases on pypi and which we treat as a
> separate source package).
>
> Both have stopped using it upstream now, so we can quite trivially
> backport those fixes to our certbot and python-acme packages, and
> retire sphinxcontrib-programoutput.
>
> Zbyszek is the owner of the package...do you agree with this plan,
> Zbyszek? I'm fixing up certbot and python-acme, but you'd have to
> retire programoutput.
Hi Adam,
I'm just back from the winter break. It's always great to see other
people take care of my packages, thanks!
You're right that the original repo is gone, but I think that this
extension remains to be useful. There's a bunch of forks on github,
so hopefully somebody will pick up maintenance. If that doesn't happen,
or if a good replacement appears, I'll drop the package. But for now
I prefer to let it live.
I fixed the test suite, it's building in rawhide.
Zbyszek
PS. What's the status of the python3.6 rebuild? Can somebody post
a list of work-needing packages?
6 years, 8 months
To use or not to use: packaged flask
by Alec Leamas
Hi out there!
I'm dipping my toes in flask, completely newbie. Doing so, I see a lot
of fedora flask packages, but no-one anywhere recommends using these -
it's all about pypi.
I "think" I prefer the packaged version, partly because I'm using
another package with native code (which, as I understand it, isn't that
easy to make a linux pypi package of). However, I'm stuck at the very
beginning:
$ ./run_flask
...
ImportError: No module named 'flask.cli'
So, my question: is it possible to run a flask application using the
packaged components similar to run_flask above, in any way? If so, how?
Cheers!
--alec
6 years, 8 months
newbie: Linking python C extensions blues
by Alec Leamas
Still struggling with my first package. Don't know if this belong to
this list (let me know if not)
Anyway, I package the extension and make a 'pip install' which builds
it. Linker command is:
gcc -pthread -shared -Wl,-z,relro
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld
build/temp.linux-x86_64-3.5/lirc/_client.o -L/usr/lib64 -llirc_client
-lpython3.5m -o
build/lib.linux-x86_64-3.5/_client.cpython-35m-x86_64-linux-gnu.so
Still, this module has unresolved references. This can be seen using
ldd. When invoked on a correctly linked variant there is a line
ldd ../lib/.libs/_client.so
liblirc_client.so.0 => /home/mk/tmp/lirc/...........
However, the variant created by setuptools misses this line, and the
corresponding symbols are unresolved. Still, it's linked using
-llirc_client in the linker command above.
Any clue out there? Why isn't my _client.so linked to liblirc_client as
it should?
Cheers!
--alec
6 years, 8 months
Packages FTBFS with Python 3.6
by Miro Hrončok
Hi all,
We've recently tried to rebuild all Python packages with Python 3.6.
However, we currently have bunch of packages that simply fail to build.
As the list contains >200 packages, it would be very helpful, if you
(package maintainers) could help us solve the issues, as we cannot go
one by one to investigate the issue.
I've attached the list of failed packages (failed.txt). You can search
Koji to find out, what went wrong. Sometimes, it's just missing
dependency. That dependency should be on this list as well. If it is
not, maybe we lost it, please tell us if that's the case. Maybe the
dependency is circular and something needs to be bootstrapped. If you
need provenpackager powers, let me know.
Everything currently happens in a side tag. I will notify you when the
side tag gets merged.
To build your package in mock, I've attached a configuration file for it
(fedora-rawhide-x86_64-p36.cfg).
Some useful commands:
fedpkg build --target=f26-python3 # build in Koji from git
fedpkg build --target=f26-python3 --scratch # scratch build in Koji from git
koji build f26-python3 --scratch <srpm> # scratch build in Koji from SRPM
fedpkg srpm && mock -r fedora-rawhide-x86_64-p36 <srpm> # mock build
Thanks for your help.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
6 years, 8 months
Python 3.6 and blocking-by-default os.urandom()
by Nick Coghlan
Hi folks,
The long discussions upstream regarding the default behaviour of Python's
os.urandom() API in Python 3.6 have come to an end, with Guido opting to
make os.urandom() block implicitly waiting for the system RNG to be ready,
even on Linux (where /dev/urandom doesn't do that).
While I can see his point from a cross-platform language runtime design
perspective, I'm still not sure it's the right answer for the Fedora system
Python, as it doesn't play nice with ABRT, and goes against Ted T'so's
decision to keep /dev/urandom non-blocking at the kernel level (risking
confusion for folks that are Linux developers & operators first, and
Pythonistas second).
If "os.urandom() was called before system RNG is ready" throws an uncaught
BlockingIOError, then we'll get a nice ABRT-friendly Python stack trace for
people to debug (and perhaps figure out their VM entropy pool isn't being
seeded properly, or their ARM hardware design needs a better source of
entropy). By contrast, if it just blocks, then folks will be faced with a
system hang, which they'll need to trace back to CPython being blocked on a
kernel getrandom() call, and then infer from that that the system RNG isn't
ready (for whatever reason), and they should probably do something about
that.
Patching this behaviour should be relatively straightforward - Python 3.5
currently makes this call in non-blocking mode and falls back to reading
/dev/urandom directly in that case, so we'd just be forward porting the
same logic to 3.6 and raising an exception instead of falling back to the
file descriptor.
Accordingly, what I propose we do is as follows:
1. Raise the concern in the F26 system-wide change proposal for migrating
to Python 3.6
2. Apply the patch when the 3.6 beta releases are added to Fedora Rawhide
3. Decide whether or not to keep the patch based on ABRT results and other
feedback on the Rawhide builds.
Note that if the feedback on Rawhide shows that the change is helping
people to find and diagnose VMs and hardware with improperly seeded entropy
pools, that's a *good* thing: this proposed change is replacing Python
3.5's "/dev/urandom and os.urandom() may silently return statistically
less-than-fully-random random numbers if the kernel entropy pool isn't
seeded properly" with "os.urandom() will fail noisily in those cases, so
you can either switch to the random module, or fix your entropy pool
seeding".
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 9 months