In general, +1 to this.
On Sun, Mar 21, 2010 at 11:25:18AM +0100, Aurelien Bompard wrote:
Hi people,
At the top of our Python packaging page
(
https://fedoraproject.org/wiki/Packaging:Python), there's a note which
reads 'In theory /usr/lib/rpm/pythondeps.sh would also automatically
generate "Provides" lines'
This is also true for python modules : distutils and setuptools have a way
to specify provides and requires, but those are not reflected in the RPM.
It means that they must be entered and maintained by hand in the spec file.
In my opinion, this is not optimal.
The perl modules have an interesting way of reflecting their dependencies in
the rpm: they add a dependency on "perl(Module::Name)". I believe the same
system can be applicable to python modules.
Currently, there are two main packaging systems in python : distutils and
setuptools. Distutils declares the dependencies in an egg-info file, which
is RFC-822-formatted. Setuptools turns this file into a PKG-INFO file in a
subdirectory, and adds a requires.txt files with additional dependencies, in
a different format.
Note that in some corner cases, the egg-info might not give you what you
need.
For instance, if we were to subpackage pkg_resources separately from
setuptools we would only have one upstream egg-info that referenced
setuptools, nothing referencing pkg_resources.
Like I say, these are corner cases, though.
The pythondeps.sh script should be able to extract requires from
these
files. To match the dependencies, pythondeps.sh should create virtual
provides like "python(ModuleName) = version".
In describing this, module name is probably not the best word. ProjectName
or EggName might be better. (Because there can multiple modules but they
are all described by a single name in the egg-info).
Also versions are more problematic than naming's corner cases.
* If we backport a bugfix or feature to an old version, the EggInfo may
require a newer version than we provide even though it would work fine
with our version.
* Upstreams frequently put a minimum version in that they have tested with
rather than the true minimum version that the package will work with.
That means the setup.py file (and thus egginfo) may say it requires
SqlAlchemy-0.5.5 but it really works with the SQLAlchemy-0.5.2 that we
ship.
The good news is : I've written it already ;-)
It's based on the pythondeps.sh script from Git master (which changed a
little bit due to python3, see bug 532118). Also, the script does not try to
be too smart with versionned dependencies, because the format is a little
bit different in python and in rpm. For those complicated cases, handwritten
requirements can still be added to the spec file. The script only covers the
usual cases.
For reference, the dependency format in distutils is described here:
http://www.python.org/dev/peps/pep-0314/
The dependency format in setuptools is described here:
http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies
Note: PEP about changing the version strings in distutils:
http://www.python.org/dev/peps/pep-0386/
It's accepted but not yet implemented. When it is we will have something
that we can map upstream versions to our version + release format (although,
we'd probably only use the version portion in autodeps).
Also PEPs that change the metadata format in distutils:
http://www.python.org/dev/peps/pep-0376/
http://www.python.org/dev/peps/pep-0390/
A patch would not make much sense due to the size of the addition, so
here's
the full script:
http://aurelien.bompard.org/projects/divers/pythondeps.sh
Some specifics:
# Handle alpha and rc releases: the version comparator will be rpm, not
# python, so 1.0rc1 > 1.0. Deal with it by turning 1.0rc1 into 1.0-0.rc1
# (which is the recommended naming scheme anyway)
echo $pyver | sed -e 's/\([0-9.]\+\)\([a-z].*\)/\1-0.\2/g'
Actually, the naming scheme for Fedora would be:
1.0-0.1.rc1, 1.0-0.2.rc1, etc.
If I'm reading the code correctly, these versions just get put in the
autodeps, though, so that shouldn't matter. However, what does matter is
that this sequence won't be parsed so that the sequence of versions is
correct:
0.9 => 0.9
1.0rc1 => 1.0-0.rc1
1.0 => 1.0
1.0post1=> 1.0-0.post1
That would order as 0.9, 1.0, 1.0-0.rc1, 1.0-0.post1
You want something more like this:
0.9 => 0.9-1
1.0rc1 => 1.0-0.rc1
1.0 => 1.0-1
1.0post1=> 1.0-1.post1
0 and 1 are always added and denote pre release versus post release.
The next problem is deciding which version strings you're going to target.
Current distutils, setuptools, and the distutils PEP have three different
algorithms for comparing versions. You can throw out current distutils
because no one use it. Setuptools is something that the python community is
trying to get rid of but it's the current de facto standard. It's version
algorithm is a huge mess because it makes a lot of guesses about what you
might mean. The distutils PEP is not in a python release yet but it is much
better -- there's a few discreet keywords that mean pre release or post
release and some methods of combining them.
I've tested it with quite a few packages, but to make sure I've written a
few unit tests (very simple, bash-based) :
http://aurelien.bompard.org/projects/divers/test-pythondeps.sh
I do believe it would be a valuable addition to RPM (of course, even if
accepted shortly, I don't expect it to land in F-13 since it requires
recompiling all the python packages).
Do you think it's a good idea ? What about the implementation ?
Idea for Provides: +1
Idea for Requires: I think the versioning landscape in python is pretty
crazy right now. I'd leave off versions in Requires altogether until
(hopefully) PEP-386 is implemented and becomes standard. (But including
Requires: python(SQLAlchemy) will be an improvement by itself)
Implementation: The only bug I saw was how you were changing upstream
version to an rpm orderable version-release string and any changes needed
to remove version from the Requires generation.
-Toshio