We should move to something more secure than md5 for the uploaded sources.
This patch series implements the client-side part of this change.
We might want to drop the md5 fallback once we have migrated completely, that is when:
1. all archives on the lookaside have been moved to a stronger hash 2. the "sources" file in all git repos has been updated to the same hash
We are about to move away from md5 for hashes.
However, at the time we move, some source tarballs might still exist only with a md5 hash.
This allows verifying these old archives by falling back on md5.
We might want to just revert this commit eventually, when are confident that md5 is completely gone (i.e that we have migrated the whole lookaside cache to the new hash). --- src/fedpkg/__init__.py | 9 +++++++++ 1 file changed, 9 insertions(+)
diff --git a/src/fedpkg/__init__.py b/src/fedpkg/__init__.py index 910e448..55ce0e6 100644 --- a/src/fedpkg/__init__.py +++ b/src/fedpkg/__init__.py @@ -249,6 +249,15 @@ class Commands(pyrpkg.Commands): def build(self, *args, **kwargs): return(super(Commands, self).build(*args, **kwargs))
+ def _verify_file(self, file, hash, hashtype): + valid = super(Commands, self)._verify_file(file, hash, hashtype) + + if not valid: + # We're moving away from md5, so try falling back on md5 for some time + valid = super(Commands, self)._verify_file(file, hash, "md5") + + return valid + # New functionality def _create_curl(self): """Common curl setup options used for all requests to lookaside."""
Just for the record I'm not going to merge this one as we will work with different approach as outlined in buildsys mailing list https://lists.fedoraproject.org/pipermail/buildsys/2015-February/ (I haven't found the latest e-mail via web archive yet).
On 03/17/2014 01:05 PM, Mathieu Bridon wrote:
We are about to move away from md5 for hashes.
However, at the time we move, some source tarballs might still exist only with a md5 hash.
This allows verifying these old archives by falling back on md5.
We might want to just revert this commit eventually, when are confident that md5 is completely gone (i.e that we have migrated the whole lookaside cache to the new hash).
src/fedpkg/__init__.py | 9 +++++++++ 1 file changed, 9 insertions(+)
diff --git a/src/fedpkg/__init__.py b/src/fedpkg/__init__.py index 910e448..55ce0e6 100644 --- a/src/fedpkg/__init__.py +++ b/src/fedpkg/__init__.py @@ -249,6 +249,15 @@ class Commands(pyrpkg.Commands): def build(self, *args, **kwargs): return(super(Commands, self).build(*args, **kwargs))
- def _verify_file(self, file, hash, hashtype):
valid = super(Commands, self)._verify_file(file, hash, hashtype)
if not valid:
# We're moving away from md5, so try falling back on md5 for some time
valid = super(Commands, self)._verify_file(file, hash, "md5")
return valid
# New functionality def _create_curl(self): """Common curl setup options used for all requests to lookaside."""
On Tue, Feb 10, 2015 at 02:38:12PM +0100, Pavol Babincak wrote:
Just for the record I'm not going to merge this one as we will work with different approach as outlined in buildsys mailing list https://lists.fedoraproject.org/pipermail/buildsys/2015-February/ (I haven't found the latest e-mail via web archive yet).
When you find the email could you send it back this way? I'm curious about which approach you want to take.
Thanks, Pierre
On 03/17/2014 01:05 PM, Mathieu Bridon wrote:
We are about to move away from md5 for hashes.
However, at the time we move, some source tarballs might still exist only with a md5 hash.
This allows verifying these old archives by falling back on md5.
We might want to just revert this commit eventually, when are confident that md5 is completely gone (i.e that we have migrated the whole lookaside cache to the new hash).
src/fedpkg/__init__.py | 9 +++++++++ 1 file changed, 9 insertions(+)
diff --git a/src/fedpkg/__init__.py b/src/fedpkg/__init__.py index 910e448..55ce0e6 100644 --- a/src/fedpkg/__init__.py +++ b/src/fedpkg/__init__.py @@ -249,6 +249,15 @@ class Commands(pyrpkg.Commands): def build(self, *args, **kwargs): return(super(Commands, self).build(*args, **kwargs))
- def _verify_file(self, file, hash, hashtype):
valid = super(Commands, self)._verify_file(file, hash, hashtype)
if not valid:
# We're moving away from md5, so try falling back on md5 for some time
valid = super(Commands, self)._verify_file(file, hash, "md5")
return valid
- # New functionality def _create_curl(self): """Common curl setup options used for all requests to lookaside."""
-- Pavol Babincak Release Engineering, Red Hat _______________________________________________ rel-eng mailing list rel-eng@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/rel-eng
On 02/10/2015 02:59 PM, Pierre-Yves Chibon wrote:
On Tue, Feb 10, 2015 at 02:38:12PM +0100, Pavol Babincak wrote:
Just for the record I'm not going to merge this one as we will work with different approach as outlined in buildsys mailing list https://lists.fedoraproject.org/pipermail/buildsys/2015-February/ (I haven't found the latest e-mail via web archive yet).
When you find the email could you send it back this way? I'm curious about which approach you want to take.
https://lists.fedoraproject.org/pipermail/buildsys/2015-February/004514.html
And Mathieu already sent patches for pyrpkg (which I'm going to review soon):
https://lists.fedoraproject.org/pipermail/buildsys/2015-February/thread.html...
We are moving away from md5 for hashes.
This gets the upload command to just upload the files with the configured lookaside hash.
It only requires a corresponding change to the lookaside cache when we change the actual config to something else than 'md5'.
That will come in a future commit, though. --- src/fedpkg/__init__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/fedpkg/__init__.py b/src/fedpkg/__init__.py index 55ce0e6..1a5c385 100644 --- a/src/fedpkg/__init__.py +++ b/src/fedpkg/__init__.py @@ -286,7 +286,8 @@ class Commands(pyrpkg.Commands): # This is overloaded to add in the fedora user's cert cmd = ['curl', '-k', '--cert', self.cert_file, '--fail', '-o', '/dev/null', '--show-error', '--progress-bar', '-F', - 'name=%s' % self.module_name, '-F', 'md5sum=%s' % file_hash, + 'name=%s' % self.module_name, + '-F', '%ssum=%s' % (self.lookasidehash, file_hash), '-F', 'file=@%s' % file] if self.quiet: cmd.append('-s')
Applied finally. Sorry for the delay.
On 03/17/2014 01:05 PM, Mathieu Bridon wrote:
We are moving away from md5 for hashes.
This gets the upload command to just upload the files with the configured lookaside hash.
It only requires a corresponding change to the lookaside cache when we change the actual config to something else than 'md5'.
That will come in a future commit, though.
src/fedpkg/__init__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/fedpkg/__init__.py b/src/fedpkg/__init__.py index 55ce0e6..1a5c385 100644 --- a/src/fedpkg/__init__.py +++ b/src/fedpkg/__init__.py @@ -286,7 +286,8 @@ class Commands(pyrpkg.Commands): # This is overloaded to add in the fedora user's cert cmd = ['curl', '-k', '--cert', self.cert_file, '--fail', '-o', '/dev/null', '--show-error', '--progress-bar', '-F',
'name=%s' % self.module_name, '-F', 'md5sum=%s' % file_hash,
'name=%s' % self.module_name,
'-F', '%ssum=%s' % (self.lookasidehash, file_hash), '-F', 'file=@%s' % file] if self.quiet: cmd.append('-s')
So, a couple of things came out of the meeting on Monday concerning these changes.
----
First, people seemed to be in agreement that having the hash type in the path on the lookaside cache was a good idea.
That should just be a trivial change to one of the patches: https://lists.fedoraproject.org/pipermail/infrastructure/2014-March/014189.h...
I'll submit a new version of that one soon.
----
However, the discussion showed that we didn't have a clear consensus on whether we would use only one hash or several, and which one(s) we'd use.
To me, one thing is clear: we should keep the existing md5 hashes around for some time, at least as long as some package modules still use a md5 hash in their "sources" file.
Dennis wants us to "convert" all of these to stronger hashes, which we could certainly do. (I'll start looking into writing a script that does that soon)
But the question is about what to do on new uploads.
I would personally prefer that we stored only one hash, and that we'd use the strongest currently available. (that would be sha512?)
But Till made the point that storing multiple hashes makes it easier to compare our sources with other projects or upstreams (as they might have hashes for different algorithms).
At this point, I don't mind implementing that, but it seems others preferred having only one hash...
As I said in the meeting, I'm happy to implement whatever ends up being the consensus. Can we build one? :)
On Wed, Mar 19, 2014 at 13:23:27 +0800, Mathieu Bridon bochecha@fedoraproject.org wrote:
I would personally prefer that we stored only one hash, and that we'd use the strongest currently available. (that would be sha512?)
The sha-3 competition is done (over a year ago) and Keccak was selected. I think it would make more sense to go to that rather than sha512. https://en.wikipedia.org/wiki/SHA-3
On Wed, 19 Mar 2014 09:58:20 -0500 Bruno Wolff III bruno@wolff.to wrote:
On Wed, Mar 19, 2014 at 13:23:27 +0800, Mathieu Bridon bochecha@fedoraproject.org wrote:
I would personally prefer that we stored only one hash, and that we'd use the strongest currently available. (that would be sha512?)
The sha-3 competition is done (over a year ago) and Keccak was selected. I think it would make more sense to go to that rather than sha512. https://en.wikipedia.org/wiki/SHA-3
I did a quick look around, but likely missed it... are there actually any tools that implement this yet?
kevin
On Wed, Mar 19, 2014 at 09:36:52 -0600, Kevin Fenzi kevin@scrye.com wrote:
On Wed, 19 Mar 2014 09:58:20 -0500 Bruno Wolff III bruno@wolff.to wrote:
On Wed, Mar 19, 2014 at 13:23:27 +0800, Mathieu Bridon bochecha@fedoraproject.org wrote:
I would personally prefer that we stored only one hash, and that we'd use the strongest currently available. (that would be sha512?)
The sha-3 competition is done (over a year ago) and Keccak was selected. I think it would make more sense to go to that rather than sha512. https://en.wikipedia.org/wiki/SHA-3
I did a quick look around, but likely missed it... are there actually any tools that implement this yet?
Looking into this further, even though Keccak was picked over a year ago, there is still a standardization process to pick some of the parameters that doesn't look like has happened yet. There appears to be a good project to watch for this: https://github.com/maandree/sha3sum
But I was wrong about this being a good time to switch to it. If we use it now, we might end up needing to recompute the hashes again later. So for now it looks like one of the sha-2 algorthims should be used.
rel-eng@lists.fedoraproject.org