From: Pat Riehecky riehecky@fnal.gov
Over at the CentOS project they are looking into an rpkg based tool, but due to the hard coded filename for external sourecs, a lot of code is duplicated.
The attached patch preserves the existing behavior, but allows a hook into the 'sources' function so that other metadata files can be consulted.
The format of the url in the CentOS structure is also different. However, I've not (yet?) taken a stab at making that customizable.
Pat Riehecky (2): Allow custom 'sources' metadata file Added optional 'valsep' for metadata files not seperated by ' '
src/pyrpkg/__init__.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
From: Pat Riehecky riehecky@fnal.gov
--- src/pyrpkg/__init__.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/pyrpkg/__init__.py b/src/pyrpkg/__init__.py index 31e516c..c4beb20 100644 --- a/src/pyrpkg/__init__.py +++ b/src/pyrpkg/__init__.py @@ -1461,11 +1461,11 @@ class Commands(object): self._run_command(cmd, cwd=self.path) return
- def sources(self, outdir=None): + def sources(self, outdir=None, sourcefile='sources'): """Download source files"""
try: - archives = open(os.path.join(self.path, 'sources'), + archives = open(os.path.join(self.path, sourcefile), 'r').readlines() except IOError, e: raise rpkgError('%s is not a valid repo: %s' % (self.path, e))
From: Pat Riehecky riehecky@fnal.gov
--- src/pyrpkg/__init__.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/pyrpkg/__init__.py b/src/pyrpkg/__init__.py index c4beb20..205208c 100644 --- a/src/pyrpkg/__init__.py +++ b/src/pyrpkg/__init__.py @@ -1461,7 +1461,7 @@ class Commands(object): self._run_command(cmd, cwd=self.path) return
- def sources(self, outdir=None, sourcefile='sources'): + def sources(self, outdir=None, sourcefile='sources', valsep=' '): """Download source files"""
try: @@ -1477,7 +1477,7 @@ class Commands(object): # This strip / split is kind a ugly, but checksums shouldn't have # two spaces in them. sources file might need more structure in the # future - csum, file = archive.strip().split(' ', 1) + csum, file = archive.strip().split(valsep, 1) except ValueError: raise rpkgError('Malformed sources file.') # See if we already have a valid copy downloaded
On 07/02/2014 11:11 AM, Pat Riehecky wrote:
From: Pat Riehecky riehecky@fnal.gov
Over at the CentOS project they are looking into an rpkg based tool, but due to the hard coded filename for external sourecs, a lot of code is duplicated.
The attached patch preserves the existing behavior, but allows a hook into the 'sources' function so that other metadata files can be consulted.
The format of the url in the CentOS structure is also different. However, I've not (yet?) taken a stab at making that customizable.
Pat Riehecky (2): Allow custom 'sources' metadata file Added optional 'valsep' for metadata files not seperated by ' '
src/pyrpkg/__init__.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
For the custom url format, would a config entry or optional arg to Commands.sources be preferred?
Pat
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 2 Jul 2014 11:21:13 -0500 Pat Riehecky riehecky@fnal.gov wrote:
On 07/02/2014 11:11 AM, Pat Riehecky wrote:
From: Pat Riehecky riehecky@fnal.gov
Over at the CentOS project they are looking into an rpkg based tool, but due to the hard coded filename for external sourecs, a lot of code is duplicated.
The attached patch preserves the existing behavior, but allows a hook into the 'sources' function so that other metadata files can be consulted.
The format of the url in the CentOS structure is also different. However, I've not (yet?) taken a stab at making that customizable.
Pat Riehecky (2): Allow custom 'sources' metadata file Added optional 'valsep' for metadata files not seperated by ' '
src/pyrpkg/__init__.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
For the custom url format, would a config entry or optional arg to Commands.sources be preferred?
Pat
what kind of custom url format are you looking at? the url format today is what koji needs.
Dennis
On 07/02/2014 02:40 PM, Dennis Gilmore wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 2 Jul 2014 11:21:13 -0500 Pat Riehecky riehecky@fnal.gov wrote:
On 07/02/2014 11:11 AM, Pat Riehecky wrote:
From: Pat Riehecky riehecky@fnal.gov
Over at the CentOS project they are looking into an rpkg based tool, but due to the hard coded filename for external sourecs, a lot of code is duplicated.
The attached patch preserves the existing behavior, but allows a hook into the 'sources' function so that other metadata files can be consulted.
The format of the url in the CentOS structure is also different. However, I've not (yet?) taken a stab at making that customizable.
Pat Riehecky (2): Allow custom 'sources' metadata file Added optional 'valsep' for metadata files not seperated by ' '
src/pyrpkg/__init__.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
For the custom url format, would a config entry or optional arg to Commands.sources be preferred?
Pat
what kind of custom url format are you looking at? the url format today is what koji needs.
Dennis
The 'sha summed source file(s)' for git.centos.org is hosted under the following pattern:
https://git.centos.org/sources/<packagename>/<branch>/<shasum>
Whereas rpkg's expects (from line 1489 of __init__.py):
https://git.centos.org/sources/<packagename>/<filename>/<shasum>/<filename>
For example, when using the default URL path rpkg attempts to download the following file: https://git.centos.org/sources/a2ps/SOURCES/a2ps-4.14.tar.gz/365abbbe4b7128b...
The path I wish to set for rpkg to use is: https://git.centos.org/sources/a2ps/c7/365abbbe4b7128bf70dad16d06e23c5701874...
Sorry for not being clear earlier,
Pat
On Wed, 2014-07-02 at 15:45 -0500, Pat Riehecky wrote:
The 'sha summed source file(s)' for git.centos.org is hosted under the following pattern:
https://git.centos.org/sources/<packagename>/<branch>/<shasum>
Whereas rpkg's expects (from line 1489 of __init__.py):
https://git.centos.org/sources/<packagename>/<filename>/<shasum>/<filename>
For example, when using the default URL path rpkg attempts to download the following file: https://git.centos.org/sources/a2ps/SOURCES/a2ps-4.14.tar.gz/365abbbe4b7128b...
The path I wish to set for rpkg to use is: https://git.centos.org/sources/a2ps/c7/365abbbe4b7128bf70dad16d06e23c5701874...
[note: I'm not the rpkg upstream maintainer]
Generally I don't think these things are configuration: they should not be modified by a user of the tools, they merely correspond to default internal values of each tool.
As such, I wouldn't be in favour of adding them to the config file. (but then again, I'm not the upstream maintainer :) )
On the other hand, adding a new keyword argument to the Command.sources method implies that a given API-using tool could call it with different urls, which could make sense if your tool can point to different lookaside caches with different url structures. (at my previous job, we had our own pyrpkg-based tool called nbpkg which could interact both with our own infrastructure and the Fedora one. We had the same url structure for lookaside caches though)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 2 Jul 2014 15:45:50 -0500 Pat Riehecky riehecky@fnal.gov wrote:
On 07/02/2014 02:40 PM, Dennis Gilmore wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 2 Jul 2014 11:21:13 -0500 Pat Riehecky riehecky@fnal.gov wrote:
On 07/02/2014 11:11 AM, Pat Riehecky wrote:
From: Pat Riehecky riehecky@fnal.gov
Over at the CentOS project they are looking into an rpkg based tool, but due to the hard coded filename for external sourecs, a lot of code is duplicated.
The attached patch preserves the existing behavior, but allows a hook into the 'sources' function so that other metadata files can be consulted.
The format of the url in the CentOS structure is also different. However, I've not (yet?) taken a stab at making that customizable.
Pat Riehecky (2): Allow custom 'sources' metadata file Added optional 'valsep' for metadata files not seperated by ' '
src/pyrpkg/__init__.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
For the custom url format, would a config entry or optional arg to Commands.sources be preferred?
Pat
what kind of custom url format are you looking at? the url format today is what koji needs.
Dennis
The 'sha summed source file(s)' for git.centos.org is hosted under the following pattern:
https://git.centos.org/sources/<packagename>/<branch>/<shasum>
Whereas rpkg's expects (from line 1489 of __init__.py):
https://git.centos.org/sources/<packagename>/<filename>/<shasum>/<filename>
For example, when using the default URL path rpkg attempts to download the following file: https://git.centos.org/sources/a2ps/SOURCES/a2ps-4.14.tar.gz/365abbbe4b7128b...
The path I wish to set for rpkg to use is: https://git.centos.org/sources/a2ps/c7/365abbbe4b7128bf70dad16d06e23c5701874...
ewww, seriously eww. the tooling would need to rename the file it gets to the name in the spec file. this I think doesn't belong in rpkg, whatever tooling centos people come up with will have to replace rpkgs function with its own version. I would strongly advocate for centos to rethink how they are doing it and follow the standard convention.
Sorry for not being clear earlier,
No Problem
Dennis
On 07/03/2014 09:15 AM, Dennis Gilmore wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 2 Jul 2014 15:45:50 -0500 Pat Riehecky riehecky@fnal.gov wrote:
On 07/02/2014 02:40 PM, Dennis Gilmore wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 2 Jul 2014 11:21:13 -0500 Pat Riehecky riehecky@fnal.gov wrote:
On 07/02/2014 11:11 AM, Pat Riehecky wrote:
From: Pat Riehecky riehecky@fnal.gov
Over at the CentOS project they are looking into an rpkg based tool, but due to the hard coded filename for external sourecs, a lot of code is duplicated.
The attached patch preserves the existing behavior, but allows a hook into the 'sources' function so that other metadata files can be consulted.
The format of the url in the CentOS structure is also different. However, I've not (yet?) taken a stab at making that customizable.
Pat Riehecky (2): Allow custom 'sources' metadata file Added optional 'valsep' for metadata files not seperated by ''
src/pyrpkg/__init__.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)For the custom url format, would a config entry or optional arg to Commands.sources be preferred?
Pat
what kind of custom url format are you looking at? the url format today is what koji needs.
Dennis
The 'sha summed source file(s)' for git.centos.org is hosted under the following pattern:
https://git.centos.org/sources/<packagename>/<branch>/<shasum>
Whereas rpkg's expects (from line 1489 of __init__.py):
https://git.centos.org/sources/<packagename>/<filename>/<shasum>/<filename>
For example, when using the default URL path rpkg attempts to download the following file: https://git.centos.org/sources/a2ps/SOURCES/a2ps-4.14.tar.gz/365abbbe4b7128b...
The path I wish to set for rpkg to use is: https://git.centos.org/sources/a2ps/c7/365abbbe4b7128bf70dad16d06e23c5701874...
ewww, seriously eww. the tooling would need to rename the file it gets to the name in the spec file. this I think doesn't belong in rpkg, whatever tooling centos people come up with will have to replace rpkgs function with its own version. I would strongly advocate for centos to rethink how they are doing it and follow the standard convention.
That kind of change is outside of my purview.
Pat
Rpkg already does the renaming for you:
# https://git.fedorahosted.org/cgit/rpkg.git/tree/src/pyrpkg/__init__.py (line 1513) command = ['curl', '-H', 'Pragma:', '-o', outfile, '-R', '-S', '--fail']
Are there other ways this might break behavior in Fedora? I could easily see the url structure being a configurable option with the Fedora behavior as default.
Brian
-- Brian Stinson bstinson@ksu.edu | IRC: bstinson | Bitbucket/Twitter: bstinsonmhk
On 07/03/2014 03:15 PM, Dennis Gilmore wrote:
For example, when using the default URL path rpkg attempts to download the following file: https://git.centos.org/sources/a2ps/SOURCES/a2ps-4.14.tar.gz/365abbbe4b7128b...
The path I wish to set for rpkg to use is: https://git.centos.org/sources/a2ps/c7/365abbbe4b7128bf70dad16d06e23c5701874...
ewww, seriously eww. the tooling would need to rename the file it gets to the name in the spec file. this I think doesn't belong in rpkg,
yes, so rather than using a curl -O, you get to give it a name to write to - a name that maps to sanity in the source repos, and does the sha validation without needing to waste time doing it down the road, and a process that allows far better and more effecient duplication checks on the cache side.
tell me why we'd want to change to lose all that functionality ? I'm getting assurance on code retreval to match code checkin, for free.
whatever tooling centos people come up with will have to replace rpkgs function with its own version. I would strongly advocate for centos to rethink how they are doing it and follow the standard convention.
'standard convention'... which is exactly what ?
On 05/07/2014 11:55, Karanbir Singh wrote:
On 07/03/2014 03:15 PM, Dennis Gilmore wrote:
For example, when using the default URL path rpkg attempts to download the following file: https://git.centos.org/sources/a2ps/SOURCES/a2ps-4.14.tar.gz/365abbbe4b7128b... The path I wish to set for rpkg to use is: https://git.centos.org/sources/a2ps/c7/365abbbe4b7128bf70dad16d06e23c5701874...
ewww, seriously eww. the tooling would need to rename the file it gets to the name in the spec file. this I think doesn't belong in rpkg,
yes, so rather than using a curl -O, you get to give it a name to write to - a name that maps to sanity in the source repos, and does the sha validation without needing to waste time doing it down the road, and a process that allows far better and more effecient duplication checks on the cache side.
tell me why we'd want to change to lose all that functionality ? I'm getting assurance on code retreval to match code checkin, for free.
whatever tooling centos people come up with will have to replace rpkgs function with its own version. I would strongly advocate for centos to rethink how they are doing it and follow the standard convention.
'standard convention'... which is exactly what ?
I think Dennis is (in general) meaning the dist-git layout that Fedora uses, and fedpkg is designed for. My undertstanding of the history is that Jesse Keating's design for dist-git was inspired by the dist-cvs scheme that Fedora inherited from Red Hat's internal systems when Fedora Core moved out from inside Red Hat's firewall. In turn, I think Jesse then did a very similar conversion inside Red Hat, moving them from their CVS system to a git system based on Fedora's. I've gotten the impression that Red Hat has their own tool internally that, like fedpkg and centpkg, is a layer over rpkg (I'm guessing that's why rpkg exists at all).
CentOS' scheme is conceptually similar (git repo per package, git branches for each OS release, specs, patches etc in git, tarballs and whatnot in a lookaside cache), but the details are a bit different. For example, Fedora's lookaside cache URL scheme is:
http://pkgs.fedoraproject.org/repo/pkgs/<package name>/<filename>/<md5sum>/<filename>
A quick bit of a think doesn't turn up any major differences between the two schemes, other than Fedora's uses the real filename, so doesn't require the file to be renaming during download, so to speak. Oh, and being newer, CentOS is using SHA rather than MD5, but I'm not sure the hash collision issue is that great.
I had a bit of a chat with Jim Perrin in the last couple of weeks and asked why CentOS wasn't using the dist-git scheme (and getting all the existing Fedora tooling for free). He pointed out that the CentOS scheme already existed, and CentOS was being consistant with itself. That's fair enough, really. Changing your own tooling would not doubt be a pain at this point, and making everyone else already consuming git.centos.org's contents change too probably wouldn't go down too well.
Fortunately, the split between the rpkg core and the <vendor>pkg wrapper means that replacing various rpkg core methods is fairly simple, based on my recollection of looking at the code in the past (when implementing another <vendor>pkg wrapper).
On 07/05/2014 01:51 PM, Howard Johnson wrote:
Fortunately, the split between the rpkg core and the <vendor>pkg wrapper means that replacing various rpkg core methods is fairly simple, based on my recollection of looking at the code in the past (when implementing another <vendor>pkg wrapper).
I believe its this functionality that Brian is trying to exploit, plus having the option of multiple backends cant be bad.
My code does some of this... http://github.com/gooseproject/goosepkg.
I altered it to use ssh to push the lookaside stuff. There's a backend piece on the server that limits what can be done for pushing as well, though that's not in the code.
Cheers,
herlo
On Sun, Jul 6, 2014 at 6:33 PM, Karanbir Singh mail-lists@karan.org wrote:
On 07/05/2014 01:51 PM, Howard Johnson wrote:
Fortunately, the split between the rpkg core and the <vendor>pkg wrapper means that replacing various rpkg core methods is fairly simple, based on my recollection of looking at the code in the past (when implementing another <vendor>pkg wrapper).
I believe its this functionality that Brian is trying to exploit, plus having the option of multiple backends cant be bad.
-- Karanbir Singh +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh GnuPG Key : http://www.karan.org/publickey.asc -- buildsys mailing list buildsys@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/buildsys
On Jul 07 01:33, Karanbir Singh wrote:
On 07/05/2014 01:51 PM, Howard Johnson wrote:
Fortunately, the split between the rpkg core and the <vendor>pkg wrapper means that replacing various rpkg core methods is fairly simple, based on my recollection of looking at the code in the past (when implementing another <vendor>pkg wrapper).
I believe its this functionality that Brian is trying to exploit, plus having the option of multiple backends cant be bad.
Definitely. The replacement and redefinition of rpkg methods is really slick. However, it would be nice to push some generality upward into rpkg (without breaking the way it works currently for fedpkg of course). That would let 3rd party sites call rpkg's methods with configurable options rather than copy-pasting rpkg's code into the redefinition.
Brian
-- Brian Stinson bstinson@ksu.edu | IRC: bstinson | Bitbucket/Twitter: bstinsonmhk
buildsys@lists.fedoraproject.org
-
Brian Stinson -
Clint Savage -
Dennis Gilmore -
Howard Johnson -
Karanbir Singh -
Mathieu Bridon -
Pat Riehecky