(Let's try this again, but with less fail!)
Getting source tarballs from github is a nightmare. See http://lists.fedoraproject.org/pipermail/devel/2011-February/148676.html
The debian tool doesn't help very much because one still needs revision garbage in the specfile. Is there any more recent ways to mitigate this? Perhaps we could "lobby" github to change their ways, at least partially?
On 04/20/2012 08:24 PM, Corey Richardson wrote:
(Let's try this again, but with less fail!)
Getting source tarballs from github is a nightmare. See http://lists.fedoraproject.org/pipermail/devel/2011-February/148676.html
The debian tool doesn't help very much because one still needs revision garbage in the specfile. Is there any more recent ways to mitigate this? Perhaps we could "lobby" github to change their ways, at least partially?
Yeah, I filed a support request with them a while back but they didn't seem that interested. Perhaps if they got lots of complaints though they might pay attention.
Corey Richardson wrote:
Getting source tarballs from github is a nightmare. See http://lists.fedoraproject.org/pipermail/devel/2011-February/148676.html
The debian tool doesn't help very much because one still needs revision garbage in the specfile. Is there any more recent ways to mitigate this? Perhaps we could "lobby" github to change their ways, at least partially?
I don't know if this will be of any use to you, but for tcplay there isn't a tagged release, so I use this script to create an suitable tarball from a github commit with a known hash, by doing a clone then archiving it locally. Certainly this doesn't solve the problem of not having a source URL.
#!/bin/sh # usage: tcplay-get-snapshot.sh <hash> git clone git://github.com/bwalex/tc-play ( cd tc-play && \ git archive --format=tar --prefix=tcplay-$1/ $1 \ ) | xz - >tcplay-$1.tar.xz
Best regards, Eric
On Fri, 2012-04-20 at 20:51 -0700, Eric Smith wrote:
Corey Richardson wrote:
Getting source tarballs from github is a nightmare. See http://lists.fedoraproject.org/pipermail/devel/2011-February/148676.html
The debian tool doesn't help very much because one still needs revision garbage in the specfile. Is there any more recent ways to mitigate this? Perhaps we could "lobby" github to change their ways, at least partially?
I don't know if this will be of any use to you, but for tcplay there isn't a tagged release, so I use this script to create an suitable tarball from a github commit with a known hash, by doing a clone then archiving it locally. Certainly this doesn't solve the problem of not having a source URL.
#!/bin/sh # usage: tcplay-get-snapshot.sh <hash> git clone git://github.com/bwalex/tc-play ( cd tc-play && \ git archive --format=tar --prefix=tcplay-$1/ $1 \ ) | xz - >tcplay-$1.tar.xz
We could probably add something like this to the 'snippets' bit of the packaging guidelines, since github is pretty significant. Just to try and help packagers do it in a uniform and guideline-complaint way.
As a reminder for packagers following the thread - any time you have to 'generate' a source tarball like this (i.e. it is not possible to provide a Source: URL that can be expected to remain valid), you should ensure the tarball is reliably reproducible by others and include all necessary instructions to produce it, using comments in the .spec file and/or a script like Eric's - so if you wanted to use Eric's script, you should ensure you actually check the script itself into git (and probably cite it as a Source itself, so it winds up in the .src.rpm), then have a comment in the .spec file which specifies the invocation of the script that was used to generate the source tarball.
On Mon, 2012-04-23 at 14:27 +0100, Adam Williamson wrote:
On Fri, 2012-04-20 at 20:51 -0700, Eric Smith wrote:
Corey Richardson wrote:
Getting source tarballs from github is a nightmare. See http://lists.fedoraproject.org/pipermail/devel/2011-February/148676.html
I noticed putting what you want after .../tarball/XXXX has no effect, thus I have good results by using URLs like:
https://github.com/user/app/tarball/gittag/user-app-gittag.tar.gz
where user and app identify the repository target and gittag is the hex code of the desired commit. This satisfies rpmbuild and the URL is valid.
The downloaded tar contains everything under directory user-app-gittag.
Of course, this works as long as the target data (i.e.: repository) lives on github :-/
Patrick
On 04/23/2012 06:27 AM, Adam Williamson wrote:
On Fri, 2012-04-20 at 20:51 -0700, Eric Smith wrote:
#!/bin/sh # usage: tcplay-get-snapshot.sh <hash> git clone git://github.com/bwalex/tc-play ( cd tc-play && \ git archive --format=tar --prefix=tcplay-$1/ $1 \ ) | xz - >tcplay-$1.tar.xz
We could probably add something like this to the 'snippets' bit of the packaging guidelines, since github is pretty significant. Just to try and help packagers do it in a uniform and guideline-complaint way.
As a reminder for packagers following the thread - any time you have to 'generate' a source tarball like this (i.e. it is not possible to provide a Source: URL that can be expected to remain valid), you should ensure the tarball is reliably reproducible by others and include all necessary instructions to produce it, using comments in the .spec file and/or a script like Eric's - so if you wanted to use Eric's script, you should ensure you actually check the script itself into git (and probably cite it as a Source itself, so it winds up in the .src.rpm), then have a comment in the .spec file which specifies the invocation of the script that was used to generate the source tarball.
"wget --content-disposition https://github.com/$user/$project/tarball/$tag"
lets you download a tarball named $user-$project-$tag-0-$gitsha1.tar.gz. That saves the maintainer from having to document how to generate the tarball, in exchange for dealing with a tarball name that contains random content (the sha1). The path of files in the tarball also contains the sha1.
Even so, this still seems preferable to me than making packagers generate the tarball each time and document the process, which seems very prone to error.
Regards -- Andy
ps btw that wget w/o --content-disposition gets you a tarball named $tag. pps I both package and maintain upstream code in github, guidelines on the wiki would be great.
On Mon, 2012-04-23 at 10:37 -0700, Andy Grover wrote:
"wget --content-disposition https://github.com/$user/$project/tarball/$tag"
lets you download a tarball named $user-$project-$tag-0-$gitsha1.tar.gz. That saves the maintainer from having to document how to generate the tarball, in exchange for dealing with a tarball name that contains random content (the sha1). The path of files in the tarball also contains the sha1.
Even so, this still seems preferable to me than making packagers generate the tarball each time and document the process, which seems very prone to error.
Yup, if that's reliable, it's certainly superior. Thanks!
On 04/23/2012 11:37 AM, Andy Grover wrote:
"wget --content-disposition https://github.com/$user/$project/tarball/$tag"
lets you download a tarball named $user-$project-$tag-0-$gitsha1.tar.gz. That saves the maintainer from having to document how to generate the tarball, in exchange for dealing with a tarball name that contains random content (the sha1). The path of files in the tarball also contains the sha1.
Even so, this still seems preferable to me than making packagers generate the tarball each time and document the process, which seems very prone to error.
Regards -- Andy
ps btw that wget w/o --content-disposition gets you a tarball named $tag. pps I both package and maintain upstream code in github, guidelines on the wiki would be great.
My problem is I'm wedded to spectool -g and it doesn't use --content-disposition. Would it be safe to have spectool always use that option?
On 04/23/2012 10:48 AM, Adam Williamson wrote:
On Mon, 2012-04-23 at 10:37 -0700, Andy Grover wrote:
"wget --content-disposition https://github.com/$user/$project/tarball/$tag"
lets you download a tarball named $user-$project-$tag-0-$gitsha1.tar.gz. That saves the maintainer from having to document how to generate the tarball, in exchange for dealing with a tarball name that contains random content (the sha1). The path of files in the tarball also contains the sha1.
Even so, this still seems preferable to me than making packagers generate the tarball each time and document the process, which seems very prone to error.
Yup, if that's reliable, it's certainly superior. Thanks!
Here's what I've come up with so far for a pkg I'm working on. This is to deal with the resulting source file name not matching the %{source} URL when using --content-disposition:
Name: python-symmetric-jsonrpc ... Version: 0.1 Release: 1%{?dist} URL: https://github.com/niligulmohar/%%7Bname%7D/ Source: https://github.com/niligulmohar/%%7Bname%7D/tarball/release-%%7Bversion%7D # using wget --content-disposition %{source} yields this filename: Source1: niligulmohar-%{name}-release-%{version}-0-g0599f28.tar.gz
... %prep %setup -q -T -b 1 -n niligulmohar-%{name}-06189d9
Any further recommendations?
Thanks -- Andy
p.s. tried editing the wiki but didn't have privs, so may need to defer to an admin once we figure out the best github hack^Wprocedure.
On 2012-04-23 20:56, Orion Poplawski wrote:
My problem is I'm wedded to spectool -g and it doesn't use --content-disposition. Would it be safe to have spectool always use that option?
No, because the Content-Disposition header from the server may result in changing the downloaded file's name to something else than what its "basename" in the URL is, and that won't work with rpmbuild.
On 04/23/2012 11:21 AM, Patrick Monnerat wrote:
On Mon, 2012-04-23 at 14:27 +0100, Adam Williamson wrote:
On Fri, 2012-04-20 at 20:51 -0700, Eric Smith wrote:
Corey Richardson wrote:
Getting source tarballs from github is a nightmare. See http://lists.fedoraproject.org/pipermail/devel/2011-February/148676.html
I noticed putting what you want after .../tarball/XXXX has no effect, thus I have good results by using URLs like:
https://github.com/user/app/tarball/gittag/user-app-gittag.tar.gz
where user and app identify the repository target and gittag is the hex code of the desired commit. This satisfies rpmbuild and the URL is valid.
The downloaded tar contains everything under directory user-app-gittag.
Of course, this works as long as the target data (i.e.: repository) lives on github :-/
Patrick
Nice! I'll note explicitly that this also works with short git tags, so:
%global commit bd245c9
Source0: https://github.com/jukka/pcfi/tarball/%%7Bcommit%7D/jukka-pcfi-%%7Bcommit%7D...
%setup -q -n jukka-pcfi-%{commit}
works.
On 04/23/2012 01:08 PM, Orion Poplawski wrote:
Nice! I'll note explicitly that this also works with short git tags, so:
%global commit bd245c9
Source0: https://github.com/jukka/pcfi/tarball/%%7Bcommit%7D/jukka-pcfi-%%7Bcommit%7D...
%setup -q -n jukka-pcfi-%{commit}
works.
Even if that works, it has the potential for collision on some future date, when that hash prefix is no longer unique. I see no reason to prefer brevity here.
Josh
On Fri, Apr 20, 2012 at 08:51:29PM -0700, Eric Smith wrote:
Corey Richardson wrote:
Getting source tarballs from github is a nightmare. See http://lists.fedoraproject.org/pipermail/devel/2011-February/148676.html
The debian tool doesn't help very much because one still needs revision garbage in the specfile. Is there any more recent ways to mitigate this? Perhaps we could "lobby" github to change their ways, at least partially?
I don't know if this will be of any use to you, but for tcplay there isn't a tagged release, so I use this script to create an suitable tarball from a github commit with a known hash, by doing a clone then archiving it locally. Certainly this doesn't solve the problem of not having a source URL.
#!/bin/sh # usage: tcplay-get-snapshot.sh <hash> git clone git://github.com/bwalex/tc-play ( cd tc-play && \ git archive --format=tar --prefix=tcplay-$1/ $1 \ ) | xz - >tcplay-$1.tar.xz
I took what you provided here an expanded it a little:
http://mcpierce.blogspot.com/2012/04/creating-source-tarball-from-github.htm...
So I watch the jabberd2 ml. They just released 2.2.15. Someone asked for them to provide a tarball in the downloads section [1]. So the upstream has uploaded one there. Originally the upstream asked what purpose uploading a tarball would provide, the main reason they gave was the inclusion of the autoconf generated configure script. Anyway just a thought. Prodding upstreams to place release tarballs in their similar locations may gain similar traction... Just a thought, up till now I didn't know github had anything to allow for this.
[1] https://github.com/Jabberd2/jabberd2/downloads
On 04/23/2012 07:27 AM, Adam Williamson wrote:
On Fri, 2012-04-20 at 20:51 -0700, Eric Smith wrote:
Corey Richardson wrote:
Getting source tarballs from github is a nightmare. See http://lists.fedoraproject.org/pipermail/devel/2011-February/148676.html
The debian tool doesn't help very much because one still needs revision garbage in the specfile. Is there any more recent ways to mitigate this? Perhaps we could "lobby" github to change their ways, at least partially?
I don't know if this will be of any use to you, but for tcplay there isn't a tagged release, so I use this script to create an suitable tarball from a github commit with a known hash, by doing a clone then archiving it locally. Certainly this doesn't solve the problem of not having a source URL.
#!/bin/sh # usage: tcplay-get-snapshot.sh<hash> git clone git://github.com/bwalex/tc-play ( cd tc-play&& \ git archive --format=tar --prefix=tcplay-$1/ $1 \ ) | xz ->tcplay-$1.tar.xz
We could probably add something like this to the 'snippets' bit of the packaging guidelines, since github is pretty significant. Just to try and help packagers do it in a uniform and guideline-complaint way.
As a reminder for packagers following the thread - any time you have to 'generate' a source tarball like this (i.e. it is not possible to provide a Source: URL that can be expected to remain valid), you should ensure the tarball is reliably reproducible by others and include all necessary instructions to produce it, using comments in the .spec file and/or a script like Eric's - so if you wanted to use Eric's script, you should ensure you actually check the script itself into git (and probably cite it as a Source itself, so it winds up in the .src.rpm), then have a comment in the .spec file which specifies the invocation of the script that was used to generate the source tarball.
On 04/23/2012 11:21 AM, Patrick Monnerat wrote:
On Mon, 2012-04-23 at 14:27 +0100, Adam Williamson wrote:
On Fri, 2012-04-20 at 20:51 -0700, Eric Smith wrote:
Corey Richardson wrote:
Getting source tarballs from github is a nightmare. See http://lists.fedoraproject.org/pipermail/devel/2011-February/148676.html
I noticed putting what you want after .../tarball/XXXX has no effect, thus I have good results by using URLs like:
https://github.com/user/app/tarball/gittag/user-app-gittag.tar.gz
where user and app identify the repository target and gittag is the hex code of the desired commit. This satisfies rpmbuild and the URL is valid.
The downloaded tar contains everything under directory user-app-gittag.
Of course, this works as long as the target data (i.e.: repository) lives on github :-/
Patrick
It wasn't obvious at first to me but this works with tags not just commit hashes. So if a project tags there version numbers you can do something like:
https://github.com/enthought/mayavi/tarball/4.2.0/Mayavi-4.2.0.tar.gz
The contents are still in a directory named user-app-hash
On Mon, 23 Apr 2012 14:08:05 -0600 Orion Poplawski orion@cora.nwra.com wrote:
%global commit bd245c9
Source0: https://github.com/jukka/pcfi/tarball/%%7Bcommit%7D/jukka-pcfi-%%7Bcommit%7D...
%setup -q -n jukka-pcfi-%{commit}
I do not understand how this is supposed to work in the face of "yum update".
Suppose I cut a package last year: swift3-1.0.0-878c23.tag.xz Then I build an RPM: openstack-swift-plugin-swift3-1.0.0-878c23-1.fc17.noarch.rpm
Today I run the same git-archve and get: swift3-1.0.0-5c74ba.tag.xz openstack-swift-plugin-swift3-1.0.0-5c74ba-1.fc18.noarch.rpm
Great, how do I update from the old one to the new one? Please do not suggest epoch.
-- Pete
On Wed, 2012-06-13 at 11:45 -0600, Pete Zaitcev wrote:
Suppose I cut a package last year: swift3-1.0.0-878c23.tag.xz Then I build an RPM: openstack-swift-plugin-swift3-1.0.0-878c23-1.fc17.noarch.rpm
Today I run the same git-archve and get: swift3-1.0.0-5c74ba.tag.xz openstack-swift-plugin-swift3-1.0.0-5c74ba-1.fc18.noarch.rpm
Great, how do I update from the old one to the new one?
That's why the guidelines are saying: 20110102snap 20110102git 20110102git9e88d7e using the date as well as the git hash http://fedoraproject.org/wiki/Packaging:NamingGuidelines#Release_Tag
Pierre
Le mercredi 13 juin 2012 à 19:49 +0200, Pierre-Yves Chibon a écrit :
On Wed, 2012-06-13 at 11:45 -0600, Pete Zaitcev wrote:
Suppose I cut a package last year: swift3-1.0.0-878c23.tag.xz Then I build an RPM: openstack-swift-plugin-swift3-1.0.0-878c23-1.fc17.noarch.rpm
Today I run the same git-archve and get: swift3-1.0.0-5c74ba.tag.xz openstack-swift-plugin-swift3-1.0.0-5c74ba-1.fc18.noarch.rpm
Great, how do I update from the old one to the new one?
That's why the guidelines are saying: 20110102snap 20110102git 20110102git9e88d7e using the date as well as the git hash http://fedoraproject.org/wiki/Packaging:NamingGuidelines#Release_Tag
Pierre
$ git log -r -1 .... log + date .... $ git rev-parse --short HEAD // short hash to use $ git checkout < short hash to use > $ git archive --prefix=<name>-%%{alphatag}/ HEAD | xz > ../<name>-% %{alphatag}.xz
where alphatag is <name><date>git<short hash>
On Wed, 2012-06-13 at 11:45 -0600, Pete Zaitcev wrote:
On Mon, 23 Apr 2012 14:08:05 -0600 Orion Poplawski orion@cora.nwra.com wrote:
%global commit bd245c9
Source0:
https://github.com/jukka/pcfi/tarball/%%7Bcommit%7D/jukka-pcfi-%%7Bcommit%7D...
%setup -q -n jukka-pcfi-%{commit}
I do not understand how this is supposed to work in the face of "yum update".
But if you read his email carefully, Orion isn't speaking about the version. So commit will be bd245c9 but the version might very well be
Version: 20120613git%{commit}
and as long as the date gets updated, yum will be happy with it.
Pierre
On Wed, Jun 13, 2012 at 08:10:14PM +0200, Pierre-Yves Chibon wrote:
On Wed, 2012-06-13 at 11:45 -0600, Pete Zaitcev wrote:
On Mon, 23 Apr 2012 14:08:05 -0600 Orion Poplawski orion@cora.nwra.com wrote:
%global commit bd245c9
Source0:
https://github.com/jukka/pcfi/tarball/%%7Bcommit%7D/jukka-pcfi-%%7Bcommit%7D...
%setup -q -n jukka-pcfi-%{commit}
I do not understand how this is supposed to work in the face of "yum update".
But if you read his email carefully, Orion isn't speaking about the version. So commit will be bd245c9 but the version might very well be
Version: 20120613git%{commit}
and as long as the date gets updated, yum will be happy with it.
Also, probably want to keep all of this out of the version field. The Guidelines say to stick it in release.
Putting it in Version: runs the risk of upstream making a release like 1.0 which would then sort lower than the date (20120613).
-Toshio
On Wed, 2012-06-13 at 20:10 +0200, Pierre-Yves Chibon wrote:
On Wed, 2012-06-13 at 11:45 -0600, Pete Zaitcev wrote:
On Mon, 23 Apr 2012 14:08:05 -0600 Orion Poplawski orion@cora.nwra.com wrote:
%global commit bd245c9
Source0:
https://github.com/jukka/pcfi/tarball/%%7Bcommit%7D/jukka-pcfi-%%7Bcommit%7D...
%setup -q -n jukka-pcfi-%{commit}
I do not understand how this is supposed to work in the face of "yum update".
But if you read his email carefully, Orion isn't speaking about the version. So commit will be bd245c9 but the version might very well be
Version: 20120613git%{commit}
and as long as the date gets updated, yum will be happy with it.
The guidelines actually require an integer before the date/rcs rev part, which should be incremented at every build. Like so:
1.20110102git9e88d7e 2.20110102git9e88d7e 3.20120201git8fg34f6 4.20120201git8fg34f6
It doesn't get reset when you bump the snapshot, to double-plus-ensure that no mess in the snapshot bit of the tag can ever cause problems.
On Wed, 2012-06-13 at 13:22 -0700, Adam Williamson wrote:
On Wed, 2012-06-13 at 20:10 +0200, Pierre-Yves Chibon wrote:
On Wed, 2012-06-13 at 11:45 -0600, Pete Zaitcev wrote:
On Mon, 23 Apr 2012 14:08:05 -0600 Orion Poplawski orion@cora.nwra.com wrote:
%global commit bd245c9
Source0:
https://github.com/jukka/pcfi/tarball/%%7Bcommit%7D/jukka-pcfi-%%7Bcommit%7D...
%setup -q -n jukka-pcfi-%{commit}
I do not understand how this is supposed to work in the face of "yum update".
But if you read his email carefully, Orion isn't speaking about the version. So commit will be bd245c9 but the version might very well be
Version: 20120613git%{commit}
and as long as the date gets updated, yum will be happy with it.
The guidelines actually require an integer before the date/rcs rev part, which should be incremented at every build. Like so:
1.20110102git9e88d7e 2.20110102git9e88d7e 3.20120201git8fg34f6 4.20120201git8fg34f6
It doesn't get reset when you bump the snapshot, to double-plus-ensure that no mess in the snapshot bit of the tag can ever cause problems.
Erf. I forgot that if it's a pre-release snapshot, you make it %{version}-0.X.%{snapshot} .
The rules are a bit complex :) but all make sense, and if you follow them, you should never get in trouble with updates not superseding. You should _always_ include a version, even if it has to be notional, and if it's notional, do everything you can to make sure it's lower than any possible version number upstream might actually release in future. Always include X (post-release) or 0.X (pre-release, so that the first actual release build can always be 1) before the snapshot bit, whatever snapshot naming format you're using.
There are some good examples given on the page right under 'Pre-Release packages', it's worth looking at those.
Pete Zaitcev zaitcev@redhat.com writes:
On Mon, 23 Apr 2012 14:08:05 -0600 Orion Poplawski orion@cora.nwra.com wrote:
%global commit bd245c9
Source0: https://github.com/jukka/pcfi/tarball/%%7Bcommit%7D/jukka-pcfi-%%7Bcommit%7D...
%setup -q -n jukka-pcfi-%{commit}
I do not understand how this is supposed to work in the face of "yum update".
Suppose I cut a package last year: swift3-1.0.0-878c23.tag.xz Then I build an RPM: openstack-swift-plugin-swift3-1.0.0-878c23-1.fc17.noarch.rpm
Today I run the same git-archve and get: swift3-1.0.0-5c74ba.tag.xz openstack-swift-plugin-swift3-1.0.0-5c74ba-1.fc18.noarch.rpm
Great, how do I update from the old one to the new one? Please do not suggest epoch.
DEF_VER='<latest-stable-version>'
if test -d .git -o -f .git && VN=$(git describe --tags --match "[0-9]*" --abbrev=4 HEAD 2>/dev/null) && case "$VN" in *$LF*) (exit 1) ;; [0-9]*) git update-index -q --refresh test -z "$(git diff-index --name-only HEAD --)" || VN="$VN-dirty" esac then VN=$(echo "$VN" | sed -e 's/-/./g'); else VN="$DEF_VER" fi
-
Adam Williamson -
Andy Grover -
Corey Richardson -
Darryl L. Pierce -
Eric Smith -
Jonathan MERCIER -
Josh Stone -
Nathanael D. Noblet -
Nikola Pajkovsky -
Orion Poplawski -
Patrick Monnerat -
Pete Zaitcev -
Pierre-Yves Chibon -
Toshio Kuratomi -
Ville Skyttä