python-devel September 2018

python-devel@lists.fedoraproject.org

12 participants
17 discussions

Python 3.7's Deterministic pycs
by Petr Viktorin 15 Jul '19

15 Jul '19

Hello! The first beta for Python 3.7 is out. It will hopefully get into Fedora soon as python37. After it comes out of beta, we'll upgrade python3 to it. The What's New list is at: https://docs.python.org/3.7/whatsnew/3.7.html One thing that's interesting for packagers is PEP 552: Deterministic pycs: https://www.python.org/dev/peps/pep-0552/ Let me summarize in my own words. A new opt-in mode for byte-compilation makes .pyc (bytecode cache) files depend only on the contents of the corresponding source file. If we use this, it will slow down imports, because the whole source file would need to be read and hashed in order to verify if a .pyc file is valid. (Currently, metadata like the modification time and file size is used.) To speed things up, there's an option, UNCHECKED_HASH, which skips cache validation entirely. Using this would mean that if you modify a .py source file installed by RPM, the changes wouldn't take effect (the .py contents would only be shown in tracebacks). Modifying installed files in production is extremely bad practice, of course, but it's quite useful for debugging on throw-away systems. If we adopt UNCHECKED_HASH, anyone doing it will have to remember to remove the corresponding .pyc file. Honestly, I'm not sure we want to use this in Fedora. Is anyone here into reproducible builds, to make a better argument for this? -- Petr Viktorin

5 5

EPEL: Python 3.4 will be EOL in March 2019
by Miro Hrončok 23 Oct '18

23 Oct '18

On 13.8.2018 11:49, Larry Hastings wrote on python-dev(a)python.org: > We of the core dev community commit to supporting Python releases for > five years. Releases get eighteen months of active bug fixes, followed > by three and a half years of security fixes. Python 3.4 turns 5 next > March--at which point we'll stop supporting it, and I'll retire as 3.4 > release manager. > > My plan is to make one final release on or around its fifth birthday > containing the last round of security fixes. That's about seven > months from now. See also PEP 429 -- Python 3.4 Release Schedule [0]. We have python34 and python36 in EPEL7. python34 being the "main" one and python36 the "other" one. The original plan [1] was that once the ecosystem is adapted well enough, we can switch what "main" and "other" is and eventually drop python34, creating room for maybe another python3 release to be the "other" next (python38 maybe?). Originally this was supposed to happen for each release [2], but this was later abandoned due to lack of manpower. Do we want to ever switch "main" with "other"? The problem here is: $ dnf repoquery --repo=epel7 --whatrequires 'python(abi) = 3.4' | pkgname | sort | uniq | wc -l 243 $ dnf repoquery --repo=epel7 --whatrequires 'python(abi) = 3.6' | pkgname | sort | uniq | wc -l 15 The original idea is that all packages would have the python3_other macros in them [2] and all we need is rebuild everything in proper order, maybe with some bootstrapping. However that never happened and most of the packages I've seen lack the python3_other bits (no statistics, just my impression). Is this something we want? If so, are the packagers willing to adapt their packages (as much as I'd like to do this, I lack the resources to hack on 228 packages)? The Python Maint team in Red Hat is here to help. However the drive would need to come from the EPEL community, as now we are only guessing what are the needs here. [0] https://www.python.org/dev/peps/pep-0429/ [1] https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraprojec… [2] https://fedoraproject.org/wiki/User:Bkabrda/EPEL7_Python3 -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

3 2

Mass Python 2 packages Removal
by Miro Hrončok 15 Oct '18

15 Oct '18

Hi, this is about: https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal We have ~300 bugz open and blocking https://bugzilla.redhat.com/show_bug.cgi?id=PY2REMOVAL Some of them are > 1 week old and hence anyone can go and remove the mentioned (sub)packages. I've retired the packages that were for retirement and I will retire more once they are unblocked (needs to go to compose -> portingdb before the automation recognizes it). I open more bugs as they are closed (I don't wan to deal with 1000 bugs at once). Retirement is easy. The problematic part is removing subpackages. I go and manually choose the packages that appear not to be just simple "py spec generates py2 and py3 subpackages" combo, but a bit more complicated, and I leave the "simple" ones alone for now. I'm theoretically able to remove a simple py2 subpackage in about a minute, yet it is tedious and after ~20 packages I need to stop (before I go crazy). We have something in between 900 and 2000 packages to remove (there are 900 leaves to remove and we may make another 1100 into leaves as we go). Is anyone willing to look into automating this? Write a spec depython2izer? This usually means: - removing python2- and python- BRs - removing %package -n python2 and it's metadata, description & %files - removing all %py2_build, %py2_install or their older manual variants - removing python2 tests from %check - (sometimes) removing %py3dir or python3 dir pushd/popd dance - (sometimes) removing with_python3 conditionals as they make no sense IIRC Zbyszek said at Flock that he might work on that. Is that still true? If not, shall I draft something? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 8

Plans for Matplotlib 3.0+
by Elliott Sales de Andrade 29 Sep '18

29 Sep '18

Hi all, In case you weren't aware, Matplotlib 3.0 will drop support for Python 2. The rc1 was just released today/yesterday, though I have not yet attempted to see how that update will go. The 2.2.x line will continue to be supported and receive bug fixes until Python 2 is EOL. So I'm wondering what exactly our plans are going forward for when the final 3.0 release is made? -- Elliott

4 16

Re: Python Classroom spin dep issue: scanning
by Miro Hrončok 27 Sep '18

27 Sep '18

On 26.9.2018 02:41, Adam Williamson wrote: > Hi, Miro! Hi Adam. CCing Python Devel, so others can hop in the discussion. > The Python Classroom spin images no longer build in recent Rawhide, > because of a dnf behaviour change that exposes dependency issues as > failures rather than hiding them by silently excluding packages. (This > change will likely get into F29 soon too). > > The issue is this. fedora-live-python-classroom.ks includes > fedora-live-minimization.ks , which does this: > > -sane-backends > > but it also includes fedora-live-workstation.ks, which includes fedora- > workstation-common.ks, which does this: > > @gnome-desktop > > ...and @gnome-desktop includes sane-backends-drivers-scanners and > libsane-hpaio, which both require sane-backends. > > Until recently dnf was simply silently excluding sane-backends-drivers- > scanners and libsane-hpaio from the image, to 'resolve' this. Now it > fails on the problem, and we get to make a choice. > > You basically have three options: > > 1. Just drop fedora-live-minimization.ks from fedora-live-python- > classroom.ks . It's pretty old and random at this point. It's basically > trying to drop printing and scanning stuff to save a bit of space. Do > you actually want printing and scanning not to work on your image? The idea behind this was: Python Classroom (GNOME) is used for (usually one-time of) installations/live on classroom computers for a workshop at a conference or an actual course at school or university, or similar. Most of classrooms I've seen do not have printers and scanners attached to the computers. Hence I decided to save space while removing this. If anybody want's to use the Classroom Lab in an environment other than a classroom (such as a Python developer who finds it a good balance of default packages) or a printer + scanner is connected to a classroom computer, they can install the packages. However I didn't examine how much space this actually saves and if it's not a significant amount, we can simply go with 1. > 2. Keep fedora-live-minimization.ks but explicitly add back `sane- > backends` to %packages in fedora-live-python-classroom.ks . This would > override the exclusion and fix the dep issue, and include the scanning > bits in the image. Having scanning and not having printing seems a bit weird. But again, if printing is huge and scanning is not, we can use 2. > 3. Keep fedora-live-minimization.ks and add excludes for sane-backends- > drivers-scanners and libsane-hpaio to fedora-live-python-classroom.ks . > This would lose scanning support, and save a bit of space. I'd use this if we realize the space savings to be distinguishable. > What would you like to do? Thanks! Before I build 3 images and compare their sizes, do you have any idea about a better way to find this out? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

python3-debug: Should we use -Og rather than -O0?
by Petr Viktorin 24 Sep '18

24 Sep '18

Hi, I've been looking at documentation of these GCC flags: -O0 Reduce compilation time and make debugging produce the expected results. This is the default -Og Optimize debugging experience. -Og enables optimizations that do not interfere with debugging. It should be the optimization level of choice for the standard edit-compile-debug cycle, offering a reasonable level of optimization while maintaining fast compilation and a good debugging experience. (from https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html) Currently, we use -O0 for the debug build of Python, but it seems that -Og would actually be a better option. Is there something I'm missing?

1 0

Future of Python 2.6 and Jython in Fedora
by Miro Hrončok 21 Sep '18

21 Sep '18

In Fedora, we advertise "all the Pythons" are available and developers may use them with venv, virtualenv, tox. https://developer.fedoraproject.org/tech/languages/python/multiple-pythons.… Two of the interpreters we ship are quite problematic in this regard. ### Python 2.6 - virtualenv 16.x dropped support for 2.6 and we dirty patched it back - tox 3 dropped support for 2.6 - pip/setuptools dropped support for 2.6 and our patched virtualenv uses an older version of pip and setuptools to make work However, I know there are developers who use Python 2.6 in Fedora (namely because their software maintains RHEL 6 compatibility). I propose that we indicate on Fedora Developer Portal that Python 2.6 is provided as is and that it may not work with virtualenv/tox. We maintain the current patch of virtualenv at least for the lifetime of Fedora 29, but if an upgrade of virtualenv is done, we ditch that patch from Fedora 30+. We also mark it deprecated and schedule the removal for EPEL 6 EOL (AFAIK late 2020). ### Jython - virtualenv somehow works but is not officially supported - pip is not supported and has hesisenbugs I know nobody who uses Jython with virtualenv and tox. I propose that we indicate on Fedora Developer Portal that Jython is provided as is and that it may not work with virtualenv/tox. (No scheduled removal.) Thoughts? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 3

release-monitoring for python projects
by Martin Bukatovic 21 Sep '18

21 Sep '18

Dear python devel, While I was considering packaging of tmuxp tool, I noticed that dependencies of this package provided by Fedora are bit old. Eg. python-libtmux is 2 years old, which is old enough to prevent packaging any reasonable recent version of tmuxp. Do I read this right that we need to update both python-libtmux and other libraries in rawhide first so that review request for tmuxp can be created, and when it will be completed, the new package will land only in rawhide? I checked release-monitoring.org tool and noticed that wrong backed is selected for both packages. When I fixed this, the tool created "new version is available" bugs for both packages. Considering the information (most?) python packages have in it's spec file, would it make sense to preconfigure monitoring details based on information from the spec file when the package is added or updated? Or maybe just mandate that the release monitoring information are correct (and that the integration works) during review request? Or was there some bulk change some time ago related which could break this? -- Martin Bukatovič

2 1

Python 2 Package Removal and when to use fedora-obsolete-packages
by Scott Talbert 15 Sep '18

15 Sep '18

I'm thinking about removing some of my Python 2 subpackages (that are no longer used). However, I'm wondering about the case where the removed packages should be added to fedora-obsolete-packages. I don't _think_ the removal of my packages would cause any problems upon upgrade. Can someone provide an example of a situation where a removed package could cause a problem on upgrade (and should thus be added to fedora-obsolete-packages). Thanks, Scott

4 6

Re: DNF: "There are following alternatives to this package"
by Tomas Orsava 13 Sep '18

13 Sep '18

On 09/13/2018 06:43 PM, Mathieu Bridon wrote: > On Thu, 2018-09-13 at 18:17 +0200, Tomas Orsava wrote: >> We'd like to propose a new functionality for dnf: When a user tries >> to install a package XYZ and dnf doesn't find it, dnf would recommend >> them alternative packages. These offered packages would advertise >> that they are an alternative for XYZ using a specially formatted >> Provides tag. >> >> For example, packages `python2-requests` and `python3-requests` >> would both have the following tag: >> >> Provides: alternative-for(python-requests) = %{version}- >> %{release} >> >> (Possibly via the already existing and widespread %python_provide >> macro in the Python case.) >> >> And when the user would try to install `python-requests`, dnf would >> look for packages with the relevant Provides tag and display them: >> >> # dnf install python-requests >> * There are following alternatives to this package: >> python2-requests python3-requests >> No match for argument: python-requests >> Error: Unable to find a match >> >> This would be very similar to an already existing functionality that >> searches for lowercase package names: >> >> # dnf install python3-REQUESTS >> * Maybe you meant: python3-requests >> No match for argument: python3-REQUESTS >> Error: Unable to find a match >> >> (That functionality is broken in some versions—RHBZ#1628514—so you >> might not see this result at the moment.) >> >> What are your thoughts? > It's neat, but it doesn't help catch typos, which seems like the most > probably case where I'd want such a feature. > > How about instead of a scheme based on provides, just quickly check if > a package has a "similar" name? > > Basically extend the existing check for lowercase you mention. Catching typos would be useful, and I'd certainly support that addition, but this doesn't really scratch the itch I'm trying to solve. We want to tell the user "these are exactly the alternatives available to you". Not guess, not rely on some typo resolution. We know what the alternatives are, here they are. I've talked with people working on dnf and this looks like an easy mechanism to add, and it will be more reliable for the use cases it covers. Tomas > > $ git stats > git: 'stats' is not a git command. See 'git --help'. > > The most similar command is > status > >

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel September 2018