python-devel March 2010

python-devel@lists.fedoraproject.org

11 participants
13 discussions

Status of Fedora 13 Feature: EasierPythonDebugging
by David Malcolm 26 Mar '10

26 Mar '10

https://fedoraproject.org/wiki/Features/EasierPythonDebugging is currently marked as 95%, but a better answer is "it's complicated". I've done all I can for now to finish it, and it does provide some useful improvements, but it's being heavily impacted by a bug outside of my expertise (see (b) below). I've been testing the feature, building out a test suite, and sending upstream as http://bugs.python.org/issue8032 The torture test for this feature that I've been using is to run "yum update", then hit CTRL-Z as it's running, then: [david@brick ~]$ gdb attach $(pidof -x yum) (gdb) bt and see what you can see. There are two outstanding bugs with the feature: (a) the attach-to-yum testcase found a nasty issue: when there are cycles in the graph of python object references, my pretty-print code would take gdb into an infinite recursion. I've fixed this (and various other bugs) in v4 of the hooks so that with v4 of the hooks it seems to be robust. I've built python/python3 with v4; the relevant updates are: https://admin.fedoraproject.org/updates/python-2.6.4-24.fc13 https://admin.fedoraproject.org/updates/python3-3.1.2-3.fc13 and ought to be installable on a f13 host with: su -c 'yum --enablerepo=updates-testing update python python3' so this is in "testing", though not in the tree yet. (b) the critically important frame information within the heart of the bytecode excution loop ought to tell us what code we're running, but is showing up in gdb as "optimized out". (specifically, the "f" parameter to function PyEval_EvalFrameEx). Unfortunately, this greatly reduces the effectiveness of the feature (it's still somewhat useful without this, but the feature page would need a significant rewrite if this doesn't get fixed). This looks like a regression of rhbz:556975: https://bugzilla.redhat.com/show_bug.cgi?id=556975 I don't know if this is a gcc/gdb/python issue. I've reopened this bug. I haven't yet added it to any of the tracker bugs: http://fedoraproject.org/wiki/BugZappers/HouseKeeping/Trackers Which tracker should this be in? Dave

2 1

Pending updates for Fedora 13
by David Malcolm 24 Mar '10

24 Mar '10

I've been finishing some last-minute work on Fedora 13 features. I've refreshed the gdb debugging hooks [1] in both the python and python3 with the latest version that I submitted upstream. This has various bug fixes, plus the beginnings of some useful new commands "py-list", "py-up" and "py-down", python-level analogs of the regular gdb "list", "up", "down" commands. I've also added a "top"-like systemtap script to both packages, showing all python function calls that happened in the last second across all processes on the system. See: http://fedoraproject.org/wiki/Features/SystemtapStaticProbes#.22top.22_for_… The two updated builds aren't in the Fedora 13 tree yet. I've submitted them as updates here: https://admin.fedoraproject.org/updates/python-2.6.4-23.fc13 https://admin.fedoraproject.org/updates/python3-3.1.2-2.fc13 and they're currently still "pending" i.e. waiting to be synced to the mirrors. Please can people test these (either when they reach the mirrors, or before), and set karma on the updates accordingly, so that the new builds can go into Fedora 13. Thanks! Dave [1] as in https://fedoraproject.org/wiki/Features/EasierPythonDebugging

1 0

Python 2.6.5 and 3.1.2 likely to be released at the end of the week
by David Malcolm 22 Mar '10

22 Mar '10

Python 2.6.5 and 3.1.2 are due at the end of this week. I plan to build 2.6.5 and 3.1.2 into Fedora 14 as soon as they're released (though if any other maintainer wants to do this, feel free!) My feeling is that they're too late for F-13 at this point in the schedule [1]. Having said that, if enough people want to test them, I'd be willing to build them, for some definition of "enough" (the bump from 3.1.1 to 3.1.2 may be safer than that from 2.6.4 to 2.6.5, since the python 3 stack isn't on the critical path [2]) Dave [1] http://fedoraproject.org/wiki/Releases/13/Schedule [2] http://fedoraproject.org/wiki/Critical_Path_Packages

3 6

Automatic Provides and Requires for Python modules
by Aurelien Bompard 22 Mar '10

22 Mar '10

Hi people, At the top of our Python packaging page (https://fedoraproject.org/wiki/Packaging:Python) there's a note which reads 'In theory /usr/lib/rpm/pythondeps.sh would also automatically generate "Provides" lines' This is also true for python modules : distutils and setuptools have a way to specify provides and requires, but those are not reflected in the RPM. It means that they must be entered and maintained by hand in the spec file. In my opinion, this is not optimal. The perl modules have an interesting way of reflecting their dependencies in the rpm: they add a dependency on "perl(Module::Name)". I believe the same system can be applicable to python modules. Currently, there are two main packaging systems in python : distutils and setuptools. Distutils declares the dependencies in an egg-info file, which is RFC-822-formatted. Setuptools turns this file into a PKG-INFO file in a subdirectory, and adds a requires.txt files with additional dependencies, in a different format. The pythondeps.sh script should be able to extract requires from these files. To match the dependencies, pythondeps.sh should create virtual provides like "python(ModuleName) = version". The good news is : I've written it already ;-) It's based on the pythondeps.sh script from Git master (which changed a little bit due to python3, see bug 532118). Also, the script does not try to be too smart with versionned dependencies, because the format is a little bit different in python and in rpm. For those complicated cases, handwritten requirements can still be added to the spec file. The script only covers the usual cases. For reference, the dependency format in distutils is described here: http://www.python.org/dev/peps/pep-0314/ The dependency format in setuptools is described here: http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies A patch would not make much sense due to the size of the addition, so here's the full script: http://aurelien.bompard.org/projects/divers/pythondeps.sh I've tested it with quite a few packages, but to make sure I've written a few unit tests (very simple, bash-based) : http://aurelien.bompard.org/projects/divers/test-pythondeps.sh I do believe it would be a valuable addition to RPM (of course, even if accepted shortly, I don't expect it to land in F-13 since it requires recompiling all the python packages). Do you think it's a good idea ? What about the implementation ? If it looks good to you I'll propose it to rpm.org. Cheers, Aurélien -- http://aurelien.bompard.org ~~~~ Jabber : abompard(a)jabber.fr "Everyone thinks of changing the world, but no one thinks of changing himself." -- Tolstoï

2 2

Re: Python SIG
by Stanley A. Klein 21 Mar '10

21 Mar '10

I think the issue of Python distutils/setuptools is important. I posted a note to the Python Distutils SIG asking if there was any way I could do an RPN package (using bdist_rpm) with results going to my usual rpmbuild directory tree instead of being placed in the same directory tree as the sources and setup.py. I didn't get a response. Meanwhile, everything seems to be eggs and easy_install that conflict with the overall system management aspects (and Linux Standards Base aspects) of RPM. I think that any changes to fix the Python distutils/setuptools for RPMs will probably need to originate in the Fedora community. Stan Klein On Thu, March 18, 2010 8:00 am, David Malcolm <dmalcolm(a)redhat.com> wrote: > Message: 1 > Date: Wed, 17 Mar 2010 15:36:35 -0400 > From: David Malcolm <dmalcolm(a)redhat.com> > Subject: Python SIG > To: Fedora Python SIG <python-devel(a)lists.fedoraproject.org> > Message-ID: <1268854595.30012.2147.camel@brick> > Content-Type: text/plain; charset="UTF-8" > > I noticed that there's an effort to better organize the various Special > Interest Groups in Fedora: > https://fedorahosted.org/fedora-engineering-services/ticket/2 > > I noticed that although https://fedoraproject.org/wiki/SIGs/Python > had "[[Category:Language-specific SIGs]]" we didn't have: > "[[Category:SIGs]]" and so we weren't showing anywhere on that page. > > I've fixed this, and the page is now at least showing on the list at the > bottom of: > https://fedoraproject.org/wiki/Category:SIGs > > However, we're not in the hand-written list on that page, and I feel > that we should be (hey, Ruby gets a mention, why can't we :) ) > > Any ideas what the summary info for the SIG should be? At the risk of > plagiarism, Ruby's says: > "A SIG for people who are interested in improving the state of Ruby in > Fedora. This includes packaging Ruby libraries and applications, setting > and improving standards for packaging them as RPM's and maintaining Ruby > packages for Fedora." > > and that might be a good first start, but I'd also include the runtime > as well as libraries and applications - I don't want people to feel that > the runtime is somehow sacrosanct (just more care required!). > > How about this: > "A SIG for people who are interested in Python on Fedora. This includes > packaging and optimizing the various Python 2 and Python 3 runtimes > (CPython, Jython), packaging libraries and applications, setting and > improving standards for packaging them as RPM's and maintaining Python > packages for Fedora." > (somewhat copied from Ruby) > > I think the various questions on the F-E-S ticket are good ones. I'm > willing to do some work towards some of that ticket, but any help would > be welcome. > > Thoughts? > Dave >

2 1

Re: Python SIG
by Stanley A. Klein 20 Mar '10

20 Mar '10

Some of the Enthought packages are already in yum repositories. However, they all are not named so you can easily see that they are indeed Enthought packages. So if you want to install a collection of packages, you have to go through the list of packages (such as using "yum info python*") and notice the names. Also, over the past few years I have built rpms for the Enthought toolset. However, the last few times I ran into problems. I recently tried to download the latest tagged version recently from the Enthought svn repository and ran into the problem that the version I requested was not recognized. The previous time I tried I ran into the "unpackaged files" problem, because there was some kind of glitch in rpm. I did take the tool for building eggs of all their packages and modified it into a tool for fixing the setup.cfg files. Another problem I ran into in the past was files and directories named appropriately for Windows but not for Linux, i.e., with spaces in the file/directory names. I worked most of them out with the people at Enthought, but I later found that it is a problem that still needs watching. There are several other glitches that you run into trying to build the rpm packages. As I said in a previous posting, it would also be nice if bdist_rpm made use of the same rpmbuild directory tree as other rpm builds and did not build the rpms within the svn repository (or at least provided an option to do so). Stan Klein On Fri, March 19, 2010 8:00 am, Toshio Kuratomi <a.badger(a)gmail.com> and Gael Varoquaux <gael.varoquaux(a)normalesup.org> wrote: > > Message: 3 > Date: Thu, 18 Mar 2010 15:17:23 -0400 > From: Toshio Kuratomi <a.badger(a)gmail.com> > Subject: Re: Python SIG > To: Fedora Python SIG <python-devel(a)lists.fedoraproject.org> > > On Thu, Mar 18, 2010 at 03:11:15PM +0000, Jos? Matos wrote: >> On Wednesday 17 March 2010 19:36:35 David Malcolm wrote: >> > How about this: >> > "A SIG for people who are interested in Python on Fedora. This >> includes >> > packaging and optimizing the various Python 2 and Python 3 runtimes >> > (CPython, Jython), packaging libraries and applications, setting and >> > improving standards for packaging them as RPM's and maintaining Python >> > packages for Fedora." >> > (somewhat copied from Ruby) >> >> I find this a good start (just as you do). :-) >> >> Just to add to the discussion my main interest in this SIG is on >> scientific >> and/or educational python packages. >> >> What we see, in terms of packaging, in projects like sage >> http://www.sagemath.org/ or in other projects (proprietary) like EPD >> (Enthought Python distribution) or Python(x,y) is that put everything >> under >> the sun on those packages. >> >> It would be nice to have the free parts in Fedora... >> >> Such as the above statement is written I see these goals there. >> > WRT enthought, they're very interested in getting their software packaged > for individual distributions but don't have any manpower or inhouse > knowledge on how to do that. If someone were to start working on those > packages, I could pass along contact information and we could potentially > have help from them on changing their packaging over time to address our > needs. > > -Toshio > ------------------------------ > > Message: 4 > Date: Thu, 18 Mar 2010 20:23:32 +0100 > From: Gael Varoquaux <gael.varoquaux(a)normalesup.org> > Subject: Re: Python SIG > To: Fedora Python SIG <python-devel(a)lists.fedoraproject.org> > Message-ID: <20100318192332.GB16916(a)phare.normalesup.org> > Content-Type: text/plain; charset=iso-8859-1 > > On Thu, Mar 18, 2010 at 03:17:23PM -0400, Toshio Kuratomi wrote: >> > Such as the above statement is written I see these goals there. > >> WRT enthought, they're very interested in getting their software >> packaged >> for individual distributions but don't have any manpower or inhouse >> knowledge on how to do that. If someone were to start working on those >> packages, I could pass along contact information and we could >> potentially >> have help from them on changing their packaging over time to address our >> needs. > > AFAIK, that's been partly done (mayavi2 is packaged under Fedora, I > believe). > > I believe that Rakesh Pandit is the one to thank for that work. > > I second that assersion though: Enthought is interested in protecting and > selling only the binaries of EPD. Enthought would like the corresponding > software (all open source) to have as much success as possible, and thus > to have high-quality packages on every platform. > > Disclaimer: I worked for Enthought (although only a very brief amount of > time, as I am pursuing a career in academia), and I have much personal > interests in the success of the various Enthought-related packages, but > little time for packaging work. > > Ga?l

1 0

Python SIG
by David Malcolm 20 Mar '10

20 Mar '10

I noticed that there's an effort to better organize the various Special Interest Groups in Fedora: https://fedorahosted.org/fedora-engineering-services/ticket/2 I noticed that although https://fedoraproject.org/wiki/SIGs/Python had "[[Category:Language-specific SIGs]]" we didn't have: "[[Category:SIGs]]" and so we weren't showing anywhere on that page. I've fixed this, and the page is now at least showing on the list at the bottom of: https://fedoraproject.org/wiki/Category:SIGs However, we're not in the hand-written list on that page, and I feel that we should be (hey, Ruby gets a mention, why can't we :) ) Any ideas what the summary info for the SIG should be? At the risk of plagiarism, Ruby's says: "A SIG for people who are interested in improving the state of Ruby in Fedora. This includes packaging Ruby libraries and applications, setting and improving standards for packaging them as RPM's and maintaining Ruby packages for Fedora." and that might be a good first start, but I'd also include the runtime as well as libraries and applications - I don't want people to feel that the runtime is somehow sacrosanct (just more care required!). How about this: "A SIG for people who are interested in Python on Fedora. This includes packaging and optimizing the various Python 2 and Python 3 runtimes (CPython, Jython), packaging libraries and applications, setting and improving standards for packaging them as RPM's and maintaining Python packages for Fedora." (somewhat copied from Ruby) I think the various questions on the F-E-S ticket are good ones. I'm willing to do some work towards some of that ticket, but any help would be welcome. Thoughts? Dave

4 4

Upstream revamp of byte-compilation
by David Malcolm 17 Mar '10

17 Mar '10

IIRC, we currently jump through some hoops in brp-python-bytecompile; upstream landed some new ways of doing this in this patch for 2.7/3.2: http://bugs.python.org/issue8140 (Not sure if it's helpful as I think Toshio solved this already in F13's rpm-build, but I wanted to at least send it to the SIG in case it's of interest) Dave

2 1

[Fwd: [unladen-swallow] Upgrading to llvm-2.7]
by David Malcolm 16 Mar '10

16 Mar '10

[CCing Fedora Python SIG: context is that unladen-swallow is the optimized branch of python with JIT, and Jeffrey Yasskin is working towards being able to dynamically link Python against LLVM, in the python 3.3 timeframe] Does Fedora's LLVM build have just a single configuration, or would it be easy to support two? (Not sure if it should)

1 0

Trying out a new way of packaging Python modules
by David Malcolm 16 Mar '10

16 Mar '10

= BACKGROUND : Multiple python stacks = For Fedora 13 we've added a parallel-installable Python 3 stack, doubling the number of Python runtimes from one to two. I've ben mulling over things we could do for Python in Fedora 14. An idea I'd like to borrow from Debian is to add debug builds of the runtimes, which would give the developer the option of installing a python stack built with full reference tracking information which is a great boon during development of a C extension, and for analysing memory usage; you can e.g. interate over all live objects. In my experience this roughly halves the speed of the interpreter, and increases memory usage. This would double the number of our runtimes from 2 to 4. (As I understand it Debaian have shipped a standard vs debug pair of python stacks for a couple of years now). There's also the possibility of packaging pre-releases of python27 and python32 within Fedora, and of shipping experimental stacks using the other runtime implementations: Unladen Swallow, PyPy, Jython, Pynie, etc. In EPEL5 I've recently proposed a parallel-installable "python26" alternate stack (see https://www.redhat.com/archives/epel-devel-list/2010-March/msg00064.html ). Although I want a python26 stack, I suspect EPEL5 will also want python27 and python32 stacks in the future. Given that Python add-ons aren't compatible between major or minor releases [1], we need to somehow repackage add-on python modules for the python stacks. We updated Fedora's python packaging guidelines for Fedora 13 in order to cover the "python3" runtime: http://fedoraproject.org/wiki/Packaging:Python I like that we have a single src.rpm per upstream tarball (for those cases where both python 2 and python 3 can be built), and have a single build per-arch within Koji that emits a subpackage per python runtime. However I dislike the copy&paste approach we're using within .spec files. For me, .spec files are source code, and I passionately hate copy&paste of source code in programming: it's too easy for things to get out of sync, tends to leads to bugs, and we end up having to work harder than we should need to. Computers give us many ways of sharing and reusing information, and we should use these to make our lives easier (the computer should serve us, not the other way around!). The way I see this: for every OS release, there would be a set of supported Python runtimes, but I want us to be able to change it from release to release, and I don't want us to be tied down in manual specfile rewriting to achieve this; if we have to rewrite all our specfiles to add a new runtime, we will never want to add a new runtime. For Fedora 12, this was just "python". For Fedora 13 this is "python" and "python3". Similarly for EPEL5: the system "python: is 2.4, but I want to add a python26, and I suspect we will want more in the future. For Fedora 14 I don't know what we should do yet, but I'd love to be able to support more of the Python runtimes, and having a debug stack a "yum install" away would be handy. So I want to move from a "static" model where we hardcode information about the available python runtimes into all of .spec files into a more dynamic model, where the .spec file for a python module queries a tool about what python runtimes it ought to support based on this OS, and acts accordingly. = THE EXPERIMENT = I've implemented a tool, currently named "rpm-pyconfig" which can be used to query information about Python runtimes. Currently I'm hardcoding this data within the tool, but the idea is that every runtime would drop a .conf file (perhaps in .ini format) into a config dir "/etc/rpm-pyconfig.d/", and the tool would query this. (somewhat like pkg-config). One idea is that we would have a meta-package "python-meta-config" or somesuch, which "Requires" all of the rpms holding the .conf files, thus defining what the expected set of runtimes we're targetting for this OS release. We'd change this early during the development cycle for a Fedora release, and then keep it stable. I've used this new tool to rewrite "python-coverage.spec" so that it dynamically queries what python runtimes it should support. You can see the specfile I wrote here: http://fedorapeople.org/gitweb?p=dmalcolm/public_git/rpm-pyconfig.git;a=blo… This specfile _does_ build, and emits subpackages. The tool has its own expression-evaluation language. For example "@confbin" means "the python binary for the current configuration", and is expanded to "python" for the default python2 build, and to "python3" for the default python3 build. The main commands are: --for-each (and --for-each-2 and --for-each-3) which iterates over every python configuration (or just those that are Python 2/Python3) --eval which evaluates an expression then prints the result --exe which evaluates an expression then executes the result You can then use this within a specfile in a various ways. For example, you can write: rpm-pyconfig --foreach-2 --exe \ "cp -a ../pristine @confsrcdir" rpm-pyconfig --foreach-3 --exe \ "cp -a %{py3dir} @confsrcdir" and have each subbuild get its own copy of the source tree ("@confsrcdir" expands differently for each one), the python2 subbuilds getting a pristine copy, the python3 subbuilds getting one that's had 2to3 run on it. You can also use it with rpm's subshell capture syntax: %() This is a little-known rpm construct in which the commands within the parentheses are executed, and the stdout is captured, and then processed further. For example, the following uses it to iterate over all python runtimes and express that there should be a -coverage subpackage for each one, leading to "python-coverage" and "python3-coverage": # Define the metadata for each built package: %(rpm-pyconfig --foreach --eval " %package -n @confpkg-coverage Summary: Code coverage testing module for @confpkg Group: System Environment/Libraries Requires: @confpkg-setuptools %description -n @confpkg-coverage %SHARED_DESC @confdescline ") Similarly we can programatically generate multiple %files stanzas as desired: %(rpm-pyconfig --foreach --eval " %files -n @confpkg-coverage %defattr(-,root,root,-) %doc README.txt %{_bindir}/@confbin-coverage @conf_sitearch/coverage/ @conf_sitearch/coverage*.egg-info/ ") Note how in this world the "%python_sitearch" rpm macro goes away, and we instead use the "@conf_sitearch" rpm-pyconfig macro, which will vary per configuration. The current implementation of the tool is here: http://fedorapeople.org/gitweb?p=dmalcolm/public_git/rpm-pyconfig.git;a=blo… (it's just an experiment at this stage, so is somewhat hackish) This is what I've got so far. It's a very different way of doing python packaging to what we've done in the past. I like it, in that it expresses the stacks programmatically, rather than requiring copy&paste of specfile fragments. I'm sure there are issues with this approach, but I suspect they are fixable, and I think this approach is more sustainable than manually handcoding information about the supported Python runtimes into every specfile: it gives us the freedom to add runtimes. Doing so is a major event, but it at least reduces the effort required. An example of an issue: currently it emits "python2-coverage" for the core python2 package, rather than "python-coverage" (that's fixable). Some more thoughts on the brainstorm that led to this can be seen at: https://fedoraproject.org/wiki/DaveMalcolm/PythonIdeas My biggest concern with this is that %() is little-known, and I don't know how well-supported it is e.g. in RHEL5's rpm, and going forward. Oh, and it's a total change in how we do Python packaging :) The syntax is obviously up for grabs, and is somewhat clunky in places. I used "@" to avoid having to deal with "%" and "$" already having meaning in the rpm and shell worlds (since we have to pass around strings that have functional content for all three langugages). I suspect we'd need an exclusion syntax (e.g. "this module isn't buildable on PyPy", "this module isn't buildable on 2.7 with ppc"); you can perhaps express this using things like: if [ $(rpm-pyconfig --eval @major_version.@minor_version) -eq 2.7 ] %ifarch ppc %endif elif or somesuch Thoughts? Dave [1] both compiled C extensions and compiled .pyc/.pyo bytecode files are incompatible (but in both cases this is detectable, the former via the "NEEDED" field in the ELF metadata, the latter via the magic number in their header); see PEP 384 and PEP 3147 for future ideas towards being able to share more, but they're not done yet.

3 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel March 2010