In the light of recent RPM signing intrusions, maybe we should resurrect the RPM feature where multiple signatures are allowed (i.e. --addsign is different to --resign)? With this we could then require N good signatures (and no bad ones) on each package before yum would trust the content.
What I'm getting at with this is distributed package signing, which would make the job of breaking the trust much harder for attackers, as they would have to crack private keys of many people around the world in order to subvert Fedora packages.
For instance, an attacker being in the position of injecting a bad package and signing it with Fedora key would still get nowhere, as he'd need to convince other signatories to sign those packages before them being any threat to Fedora users. Before signing, signatories could require that original contributor that built the package for a particular tag sends a signed e-mail (containing that tag and package checksums - valid only once) to the signatories, therefore requiring yet another compromised private key in order to perform an attack. Signatories could also use alternative build systems with no public access (e.g. their own, Matt's at Dell etc.) to verify package checksums before signing, in order to avoid trusting a compromised Fedora build system.
This would require more distributed resources and would slow the update process down somewhat, but may avoid single point of intrusion as being sufficient to break the distro.
Comments?
On Tue, 2008-08-26 at 11:56 +1000, Bojan Smojver wrote:
In the light of recent RPM signing intrusions, maybe we should resurrect the RPM feature where multiple signatures are allowed (i.e. --addsign is different to --resign)? With this we could then require N good signatures (and no bad ones) on each package before yum would trust the content.
Signatories could also use alternative build systems with no public access (e.g. their own, Matt's at Dell etc.) to verify package checksums before signing, in order to avoid trusting a compromised Fedora build system.
I think the checksums would be the hardest part. Build times, hosts and other details are very often embedded into a build.
Andrew Bartlett
Andrew Bartlett <abartlet <at> samba.org> writes:
I think the checksums would be the hardest part. Build times, hosts and other details are very often embedded into a build.
Yeah, good point. We do have checksums of individual files inside the RPM, right? Maybe we can leverage that in order to provide a build system neutral checksum that can be verified independently?
-- Bojan
Bojan Smojver <bojan <at> rexursive.com> writes:
Yeah, good point. We do have checksums of individual files inside the RPM, right? Maybe we can leverage that in order to provide a build system neutral checksum that can be verified independently?
Or maybe we could even rely on checksums of cpio archives produced by rpm2cpio?
-- Bojan
Once upon a time, Bojan Smojver bojan@rexursive.com said:
Andrew Bartlett <abartlet <at> samba.org> writes:
I think the checksums would be the hardest part. Build times, hosts and other details are very often embedded into a build.
Yeah, good point. We do have checksums of individual files inside the RPM, right? Maybe we can leverage that in order to provide a build system neutral checksum that can be verified independently?
That still doesn't help; some things embed the compile time and info in the files. See for example 'uname -v' (although that one is pretty easily controlled IIRC) and 'perl -V'.
One possible way to handle builds that do this would be to do something like use the timestamp of the spec file or last CVS update time for example and force such builds to use that instead of the current time.
That doesn't help the 'perl -V' example though, since it includes the 'uname -r' and 'uname -v' output in the resulting binary; for example, you can see that the current perl RPM on F9/x86_64 was built on a RHEL5 (or derivative; somebody could tell from the version string) system and what kernel it was running at the time.
Chris Adams <cmadams <at> hiwaay.net> writes:
That still doesn't help; some things embed the compile time and info in the files. See for example 'uname -v' (although that one is pretty easily controlled IIRC) and 'perl -V'.
One possible way to handle builds that do this would be to do something like use the timestamp of the spec file or last CVS update time for example and force such builds to use that instead of the current time.
That doesn't help the 'perl -V' example though, since it includes the 'uname -r' and 'uname -v' output in the resulting binary; for example, you can see that the current perl RPM on F9/x86_64 was built on a RHEL5 (or derivative; somebody could tell from the version string) system and what kernel it was running at the time.
Right. No very good.
Are these things exceptions to the rule or do majority of package have this kind of thing built in? If 95% of packages don't have it, the rest can always be checked by hand by running binary diff or something...
-- Bojan
Bojan Smojver <bojan <at> rexursive.com> writes:
Are these things exceptions to the rule or do majority of package have this kind of thing built in?
Actually, it should be quite easy to verify this. If someone from Red Hat could run 'ls *.rpm | sort | while read pkg; do echo -en "$pkg\t"; rpm2cpio < $pkg | sha1sum; done' for all Fedora packages built in koji of a distro/arch (say F9/i386) and if Matt could do the same on his Dell build farm, we'll clearly see what gives different checksums of cpio archives.
-- Bojan
On Tue, 2008-08-26 at 03:57 +0000, Bojan Smojver wrote:
Bojan Smojver <bojan <at> rexursive.com> writes:
Are these things exceptions to the rule or do majority of package have this kind of thing built in?
Actually, it should be quite easy to verify this. If someone from Red Hat could run 'ls *.rpm | sort | while read pkg; do echo -en "$pkg\t"; rpm2cpio < $pkg | sha1sum; done' for all Fedora packages built in koji of a distro/arch (say F9/i386) and if Matt could do the same on his Dell build farm, we'll clearly see what gives different checksums of cpio archives.
why do you want that?
rpm -qp --dump pkg.rpm
-sv
Seth Vidal <skvidal <at> fedoraproject.org> writes:
why do you want that?
rpm -qp --dump pkg.rpm
Because I didn't read rpm manual page? ;-)
Yeah, that's really useful - thanks for that hint. Makes it really simple for people to compare content of packages.
You reckon this multi-key signing thing could be done in any practical fashion in Fedora?
-- Bojan
On Tue, 2008-08-26 at 05:22 +0000, Bojan Smojver wrote:
Seth Vidal <skvidal <at> fedoraproject.org> writes:
why do you want that?
rpm -qp --dump pkg.rpm
Because I didn't read rpm manual page? ;-)
Yeah, that's really useful - thanks for that hint. Makes it really simple for people to compare content of packages.
You reckon this multi-key signing thing could be done in any practical fashion in Fedora?
I think it will complicate things a lot for users to verify and it's not obvious how much we'll gain in terms of security.
-sv
Seth Vidal <skvidal <at> fedoraproject.org> writes:
I think it will complicate things a lot for users to verify
Users wouldn't actually have to verify anything by hand. The idea was that yum does that for them. I don't see how that would be any more complicated then now.
Say there are 10 signatories in the pool. Yum would check that:
- the package is signed with the Fedora key - the package is signed by at least N (say 2) other keys from the pool - failing the above, it would not accept the package
N could even be configurable in yum for smooth transition from the single key scenario.
and it's not obvious how much we'll gain in terms of security.
It is similar to what a reporter does to confirm a story. One source, not so reliable. Two sources, more reliable. Many sources, most likely reliable.
In terms of attacks, right now if somebody gets a hold of the password of the Fedora key, it's game over. Ditto if someone compromises the build system to start producing bad binaries.
With the multi-key, multi-build system, an attacker would need to get his hands on a lot of private key passwords, break multiple independent build systems etc.
I always think of flight attendants and how they are told by the captain to secure the doors and cross-check. I'm sure there must be a good reason for that cross-check :-)
-- Bojan
On Tue, Aug 26, 2008 at 5:41 PM, Bojan Smojver bojan@rexursive.com wrote:
Seth Vidal <skvidal <at> fedoraproject.org> writes:
I think it will complicate things a lot for users to verify
Users wouldn't actually have to verify anything by hand. The idea was that yum does that for them. I don't see how that would be any more complicated then now.
Say there are 10 signatories in the pool. Yum would check that:
- the package is signed with the Fedora key
- the package is signed by at least N (say 2) other keys from the pool
- failing the above, it would not accept the package
N could even be configurable in yum for smooth transition from the single key scenario.
and it's not obvious how much we'll gain in terms of security.
It is similar to what a reporter does to confirm a story. One source, not so reliable. Two sources, more reliable. Many sources, most likely reliable.
There is a specific "named" fallacy to that logic. I can't remember the mathematical name for it, but basically the issue is that having multiple sources doesn't help if they all get their information from the same top level source. The big issue with multiple signatures is that they are going to be automated somehow to deal with the thousands upon thousands of packages being dealt with... and you are going to have to come up with an additional income source to pay for the extra bureaucracy that is being added.
In terms of attacks, right now if somebody gets a hold of the password of the Fedora key, it's game over. Ditto if someone compromises the build system to start producing bad binaries.
With the multi-key, multi-build system, an attacker would need to get his hands on a lot of private key passwords, break multiple independent build systems etc.
Or just be in the right place somewhere. The build systems will not be completely independent or they would not be able to produce identical binaries..
Stephen John Smoogen <smooge <at> gmail.com> writes:
There is a specific "named" fallacy to that logic. I can't remember the mathematical name for it, but basically the issue is that having multiple sources doesn't help if they all get their information from the same top level source.
Yeah, no kidding.
The point of open source is supposed to be that more eyes see better. That's why we have package reviews, pre-release checks (alpha, beta, rc) and so on. You can never achieve 100% independence, of course. That's why we have bugs :-)
The big issue with multiple signatures is that they are going to be automated somehow to deal with the thousands upon thousands of packages being dealt with...
If there is any chance of this being automated, it cannot work at all and there is no point doing it.
and you are going to have to come up with an additional income source to pay for the extra bureaucracy that is being added.
True. All security has a price.
Or just be in the right place somewhere.
More than one, actually.
The build systems will not be completely independent or they would not be able to produce identical binaries..
Say someone breaks into Fedora build system and subverts the process in such a way that there is their own gcc inserted just at the right time in order to produce the binaries they want. A package is built by the packager and a signed e-mail is sent to the signatories to sign it, because it's an update.
Given this is a new update, another build system, located elsewhere and not publicly accessible, pulls in the package source and builds it. If that other system wasn't broken into, it will produce a different binary for sure. Immediate alarm bells for signatories.
Sure, this is a difficult thing to do right. It doesn't fix all intrusion issues (nothing can). Takes a lot of effort etc. But it does provide at least some checks and balances before packages are swallowed by users out there.
No offence, but right now we have a single point of failure that we already know can be cracked. And that single point of failure is the single point of users' trust. Not a very safe combo, IMHO.
Never mind, it was just an idea. Probably not even a good one. Back to the drawing board... ;-)
-- Bojan
Bojan Smojver <bojan <at> rexursive.com> writes:
Say there are 10 signatories in the pool. Yum would check that:
- the package is signed with the Fedora key
- the package is signed by at least N (say 2) other keys from the pool
- failing the above, it would not accept the package
Just for completeness, yum could alternatively accept say 5 keys from the pool (but no Fedora key), so that any compromise of the central key does not cause the current "change the Fedora key" hoopla. Simply resign by others and continue.
-- Bojan
On 30/08/2008, Bojan Smojver bojan@rexursive.com wrote:
Just for completeness, yum could alternatively accept say 5 keys from the pool (but no Fedora key), so that any compromise of the central key does not cause the current "change the Fedora key" hoopla. Simply resign by others and continue.
What might be good, is only signing packages with one or two keys, but only allowing those keys' public parts to be updated in rpm database (or wherever) if signed by a much larger number of keys, which would be owned by some trusted people from the fedora project. Then automated rollover could be done by simply providing a new "keyring" in updates.
Bill Crawford escreveu:
On 30/08/2008, Bojan Smojver bojan@rexursive.com wrote:
Just for completeness, yum could alternatively accept say 5 keys from the pool (but no Fedora key), so that any compromise of the central key does not cause the current "change the Fedora key" hoopla. Simply resign by others and continue.
What might be good, is only signing packages with one or two keys, but only allowing those keys' public parts to be updated in rpm database (or wherever) if signed by a much larger number of keys, which would be owned by some trusted people from the fedora project. Then automated rollover could be done by simply providing a new "keyring" in updates.
BTW, updates are still frozen. What's the schedule for the normalization of yum services? Will it be necessary a special procedure to be adopted by users?
Best regards,
CdAB
Bill Crawford <billcrawford1970 <at> gmail.com> writes:
What might be good, is only signing packages with one or two keys, but only allowing those keys' public parts to be updated in rpm database (or wherever) if signed by a much larger number of keys, which would be owned by some trusted people from the fedora project. Then automated rollover could be done by simply providing a new "keyring" in updates.
Exactly. Imagine if yum trusted fedora-release package signed by 5 keys of signatories from the pool. This could have been delivered immediately upon (potential) compromise, which would mean:
- seamless transition to new Fedora key without ever trusting the old key - automatic delivery of revocation certificate for the old key - instant suitability of all packages to be resigned by the new key
So, not only would attackers be unable to subvert the packages by stealing Fedora key (because they'd need other signatories to agree to sign bad packages), but any (potential) compromise would be quickly dealt with.
-- Bojan
Bojan Smojver bojan@rexursive.com writes:
For instance, an attacker being in the position of injecting a bad package and signing it with Fedora key would still get nowhere, as he'd need to convince other signatories to sign those packages before them being any threat to Fedora users. Before signing, signatories could require that original contributor that built the package for a particular tag sends a signed e-mail (containing that tag and package checksums - valid only once) to the signatories, therefore requiring yet another compromised private key in order to perform an attack.
Yup, packagers are going to do that, sure...
Most of us are overworked already. We aren't going to jump through any hoops for third-party signatories.
regards, tom lane
Tom Lane <tgl <at> redhat.com> writes:
Yup, packagers are going to do that, sure...
That was the intention, yes. Packagers would notify all signatories (with a signed e-mail) that they've built a new package destined for updates and that signatories should review and sign it. We're still working out the details of making sure packages are are genuine in another thread :-)
I guess from Red Hat's point of view, the only difference would be that Fedora packages would not be valid unless signed and uploaded back to updates by (required number of) other signatories.
-- Bojan
On Tue, Aug 26, 2008 at 03:27:43 +0000, Bojan Smojver bojan@rexursive.com wrote:
I guess from Red Hat's point of view, the only difference would be that Fedora packages would not be valid unless signed and uploaded back to updates by (required number of) other signatories.
I don't think you are really going to gain much from doing that. And there is certainly going to be a lot of pain associated with that. It creates extra work, adds delays, and adds a dependence on third parties. And it doesn't completely prevent people from getting bad code signed.
Bruno Wolff III <bruno <at> wolff.to> writes:
I don't think you are really going to gain much from doing that.
This depends on a particular point of view, of course. If it so happened that Fedora (and/or RHEL) signing key was compromised during the most recent intrusion, it would have been game over for users. Not so if packages had to be signed by multiple keys before being accepted by yum.
and adds a dependence on third parties
I see that as a feature, actually. It eliminates single point of failure.
And it doesn't completely prevent people from getting bad code signed.
I don't think it is possible to design a system that does that completely. But, at least you have more folks looking over the packages (from multiple sources) before signing them - more chance of spotting inconsistencies.
-- Bojan
On Tue, Aug 26, 2008 at 07:38:46 +0000, Bojan Smojver bojan@rexursive.com wrote:
Bruno Wolff III <bruno <at> wolff.to> writes:
and adds a dependence on third parties
I see that as a feature, actually. It eliminates single point of failure.
And adds another. If one of those third parties goes belly up, then Fedora is going to have to take extraordinary measures to get packages signed in a way that will be axxepted again.
Bruno Wolff III <bruno <at> wolff.to> writes:
And adds another. If one of those third parties goes belly up, then Fedora is going to have to take extraordinary measures to get packages signed in a way that will be axxepted again.
Not true. As I mentioned before, the criteria would be that package is signed with N good keys. So, resigning with someone else's key would be sufficient to overcome this.
BTW, third parties do not have to be companies. They can be trusted Fedora contributors, for instance.
-- Bojan
On Tue, 2008-08-26 at 11:56 +1000, Bojan Smojver wrote:
Comments?
I'm not an expert on these things, but I feel some flaws in your scheme.
First, it relies on that building a specifically tagged package will always produce the same result. Others have already pointed out that some software embeds non-deterministic data in files. I'd like to point out additionally that the build dependencies aren't 100% deterministic as well: when you build a new version of a build dependency and build a dependent package soon afterwards, chances are high (enough) that some builders build against the new and some against the old package, which likely has an influence on the produced binaries. Should we ever employ optimization techniques like profile-based compilation, builds between separate build systems (that have different profile data) will inevitably differ.
Second, it makes the update system more vulnerable to DOS attacks, insofar as it only takes an attacker to hack himself into one of the signatories to produce bad signatures on updates he wants to block (if they e.g. contain security fixes). The risk of this is at least proportionally higher with multiple targets to choose from (think RAID-0 and what it does to the probability of errors in such a set of disks).
Third, unless a signatory runs his own build system to verify package builds (and disregarding my first point), his signature doesn't have much value as he has to rely on the checksum data provided by the package maintainer in his signed email -- which relies on the build system not being compromised to begin with.
Then there's the additional burden on maintainers -- yet another bureaucratic hurdle.
Note that I'd appreciate being able to have multiple signatures on a package, but I don't think it helps us with this issue.
Nils
Nils Philippsen <nils <at> redhat.com> writes:
builds between separate build systems (that have different profile data) will inevitably differ.
That is a problem, I admit.
Second, it makes the update system more vulnerable to DOS attacks, insofar as it only takes an attacker to hack himself into one of the signatories to produce bad signatures on updates he wants to block (if they e.g. contain security fixes). The risk of this is at least proportionally higher with multiple targets to choose from (think RAID-0 and what it does to the probability of errors in such a set of disks).
If a signatory is producing bad signatures, just use another one. It is trivial to discard packages signed by bad keys. It is also trivial to ask another signatory from the pool to check and sign instead.
For instance, if there are 10 signatories in the pool and one of them starts signing with a bad key etc., this will affect some number of packages (not all) for a little while, all of which can be resigned by any of the other 9 signatories to make it up to N (which can be say 3 or something like that). That DOS would not really last very long. In RAID terminology, you have plenty of hot spares in the RAID-5 array.
Ditto if someone's key get compromised. Just revoke and let others sign. You still have many eyes going over it.
Compare that to breaking the password on Fedora private key now. Much sweeter for the attacker, as everything becomes wide open. In RAID terminology, you single drive just died.
Third, unless a signatory runs his own build system to verify package builds (and disregarding my first point), his signature doesn't have much value as he has to rely on the checksum data provided by the package maintainer in his signed email -- which relies on the build system not being compromised to begin with.
That is also not true. If the signatory receives a signed e-mail from the packager with checksums in it (i.e. rpm -qp --dump, as per sv), the signatory can verify that against someone else's build system that is not publicly writeable (e.g. Matt's at Dell). Not everyone would need to run their own.
And remember, the attacker would have to break the build system _and_ fake the signed e-mail from the packager AT THE SAME TIME and then get signatories to sign without any checking. Much less likely then just compromising the build system (which is what they can do now).
It is better to check against something then not to check at all, IMHO.
You know, maybe this would be a good opportunity for some cross-distro cooperation. They could build our stuff on their build farms, we build their stuff on ours (not everything - just updates). Then attackers need to compromise all in order to get just one scalp. Quite an effort.
Then there's the additional burden on maintainers -- yet another bureaucratic hurdle.
Yeah, security is always like that - pain in the arse. I still remember some devs in the office chmod-ing everything to 777 because "it's easier" :-)
-- Bojan
On Tue, 2008-08-26 at 11:33 +0000, Bojan Smojver wrote:
Nils Philippsen <nils <at> redhat.com> writes:
builds between separate build systems (that have different profile data) will inevitably differ.
That is a problem, I admit.
Second, it makes the update system more vulnerable to DOS attacks, insofar as it only takes an attacker to hack himself into one of the signatories to produce bad signatures on updates he wants to block (if they e.g. contain security fixes). The risk of this is at least proportionally higher with multiple targets to choose from (think RAID-0 and what it does to the probability of errors in such a set of disks).
If a signatory is producing bad signatures, just use another one. It is trivial to discard packages signed by bad keys. It is also trivial to ask another signatory from the pool to check and sign instead.
[snip]
But that's not what you described in the original posting (emphasis on "no bad ones"):
"""
With this we could then require N good signatures (and no bad ones) on each package before yum would trust the content.
"""
Third, unless a signatory runs his own build system to verify package builds (and disregarding my first point), his signature doesn't have much value as he has to rely on the checksum data provided by the package maintainer in his signed email -- which relies on the build system not being compromised to begin with.
That is also not true. If the signatory receives a signed e-mail from the packager with checksums in it (i.e. rpm -qp --dump, as per sv), the signatory can verify that against someone else's build system that is not publicly writeable (e.g. Matt's at Dell). Not everyone would need to run their own.
The Matt or whoever else has a separate build system are the only valuable signatories. Anybody else can only say "me too" but that doesn't add anything to the trust value.
And remember, the attacker would have to break the build system _and_ fake the signed e-mail from the packager AT THE SAME TIME and then get signatories to sign without any checking. Much less likely then just compromising the build system (which is what they can do now).
No, without independent verification builds, an attacker just has to break the build system because a normal packager has no (sensible) way to tell if e.g. a compromised compiler inserted trojan code into the executables. Thus, a normal packager -- trusting what comes out of the build system -- would just sign off whatever the resulting package is.
As independent verification builds aren't the deterministic thing that you seem to think, that doesn't sound very helpful to me.
It is better to check against something then not to check at all, IMHO.
It's like with spam filtering: bad positives are really bad. Do you think repeating the hoopla of the last days just because of a bad positive caused by a fluke in an independent build system we have no control over is a good thing?
You know, maybe this would be a good opportunity for some cross-distro cooperation. They could build our stuff on their build farms, we build their stuff on ours (not everything - just updates). Then attackers need to compromise all in order to get just one scalp. Quite an effort.
Our packages built in say OpenSUSE's or Ubuntu's build system will differ from what koji spits out, with a probability bordering on certainty. I don't know what that gains us.
Then there's the additional burden on maintainers -- yet another bureaucratic hurdle.
Yeah, security is always like that - pain in the arse. I still remember some devs in the office chmod-ing everything to 777 because "it's easier" :-)
I can live with PITA as long as there's a tangible benefit from it. I don't see that here -- just because something is a pain in the arse, it doesn't necessarily improve security.
Nils
Nils Philippsen <nils <at> redhat.com> writes:
But that's not what you described in the original posting (emphasis on "no bad ones"):
"""
With this we could then require N good signatures (and no bad ones) on each package before yum would trust the content.
"""
I did not say that yum should use another one. The distribution system in charge of accepting signatures would discard bad ones. The package would have to be signed by someone else before distribution and before yum would trust it. Just a slight delay, that's all.
The Matt or whoever else has a separate build system are the only valuable signatories. Anybody else can only say "me too" but that doesn't add anything to the trust value.
I don't think that's right. As long as you have something independent to compare to, it is not just one person that can run the comparison. If you spread the effort to more people, you lower the load.
No, without independent verification builds, an attacker just has to break the build system because a normal packager has no (sensible) way to tell if e.g. a compromised compiler inserted trojan code into the executables. Thus, a normal packager -- trusting what comes out of the build system -- would just sign off whatever the resulting package is.
I did not say that packagers would be signing the packages. I said that signatories would be doing that. Having independent verification builds is a must, because signatories would make sure builds match.
As independent verification builds aren't the deterministic thing that you seem to think, that doesn't sound very helpful to me.
Well, just because we base our build comparisons right now on something as crude as raw checksums, doesn't mean this has to be like that forever. We may find ways of comparing builds differently to determine if they were compromised, by explicitly excluding well known differences within binaries.
It's like with spam filtering: bad positives are really bad. Do you think repeating the hoopla of the last days just because of a bad positive caused by a fluke in an independent build system we have no control over is a good thing?
The hoopla of the last few days would not be repeated. There would be no danger to users, because multiple signatures would be required to inject a bad package into users' machines. That's kinda the point of double checking. Nothing would have to be taken down immediately and reinstalled.
Our packages built in say OpenSUSE's or Ubuntu's build system will differ from what koji spits out, with a probability bordering on certainty. I don't know what that gains us.
I didn't say SUSE or Ubuntu folks would have to build Fedora packages on their build systems, just their build farms (i.e. machines). Isn't koji open source? If it is, they could run that just fine.
But hey - I get it, people don't like the additional hassle. That's OK. Consider my proposal withdrawn until the time someone invents a way of reliably comparing two independent builds.
-- Bojan
On Wed, Aug 27, 2008 at 5:52 AM, Bojan Smojver bojan@rexursive.com wrote:
Well, just because we base our build comparisons right now on something as crude as raw checksums, doesn't mean this has to be like that forever. We may find ways of comparing builds differently to determine if they were compromised, by explicitly excluding well known differences within binaries.
How about you back up and just work on this very specific problem of deterministically doing build comparisons across disparate build systems ..before we even begin to discuss how a multiple sigantory process which relies on that.
I didn't say SUSE or Ubuntu folks would have to build Fedora packages on their build systems, just their build farms (i.e. machines). Isn't koji open source? If it is, they could run that just fine.
They would have to choose to run koji, instead of their own setups. I'm not going to hold my breath on that. If you were going to lobby them to use koji, you should start with OpenSuse and help them integrate the features they need into the koji base. Even if the multiple signatory idea does not pan out, having OpenSuse as koji user and contributor would be great.
-jef
Jeff Spaleta wrote:
On Wed, Aug 27, 2008 at 5:52 AM, Bojan Smojver bojan@rexursive.com wrote:
Well, just because we base our build comparisons right now on something as crude as raw checksums, doesn't mean this has to be like that forever. We may find ways of comparing builds differently to determine if they were compromised, by explicitly excluding well known differences within binaries.
How about you back up and just work on this very specific problem of deterministically doing build comparisons across disparate build systems ..before we even begin to discuss how a multiple sigantory process which relies on that.
Virtual machines, virtually identical as starting points? But what if it is the src rpm that is compromised so the builds will be identical because they both contain the modification? Now you need to have the packager verify again that the source that produced the 2 builds had only the changes he intended.
And if you are really paranoid you have to wonder about the compiler and any existing libraries too: http://cm.bell-labs.com/who/ken/trust.html.
Les Mikesell <lesmikesell <at> gmail.com> writes:
But what if it is the src rpm that is compromised so the builds will be identical because they both contain the modification?
That is not exactly the compromise of the build system and/or Fedora key, now is it? If your own contributors are subverting the system by uploading borked source, the mutli-key system isn't going to help (and I never claimed that).
For people that are not convinced in the usefulness of this (in principle), go the a bank and try to open an account. See if they'll be OK with you producing just one piece of ID.
-- Bojan
On Wed, 2008-08-27 at 21:42 +0000, Bojan Smojver wrote:
Les Mikesell <lesmikesell <at> gmail.com> writes:
But what if it is the src rpm that is compromised so the builds will be identical because they both contain the modification?
That is not exactly the compromise of the build system and/or Fedora key, now is it? If your own contributors are subverting the system by uploading borked source, the mutli-key system isn't going to help (and I never claimed that).
For people that are not convinced in the usefulness of this (in principle), go the a bank and try to open an account. See if they'll be OK with you producing just one piece of ID.
Not to fan the flames, but last time they did just that :-). And I'm pretty sure they'll do it again, possibly because the IDs in question are very hard to fake.
Nils
Nils Philippsen <nils <at> redhat.com> writes:
Not to fan the flames, but last time they did just that . And I'm pretty sure they'll do it again, possibly because the IDs in question are very hard to fake.
May we all know the name of this bank? Just so that we can stay away from such backyard operators.
-- Bojan
We already discussed that in private, but let me rehash this for the record:
On Wed, 2008-08-27 at 22:12 +0000, Bojan Smojver wrote:
Nils Philippsen <nils <at> redhat.com> writes:
Not to fan the flames, but last time they did just that . And I'm pretty sure they'll do it again, possibly because the IDs in question are very hard to fake.
May we all know the name of this bank? Just so that we can stay away from such backyard operators.
Just about every bank in Germany would operate like this. After all, we do have a national ID that is hard enough to fake(*) and if you present it to entities like banks, they usually make a copy and can verify the data on the ID with the "Einwohnermeldeamt" (possibly "residents registration office").
(*): http://www.bundesdruckerei.de/en/products/products_idSystem/idSystem_documen...
Before someone uses terms like "Orwellian", "1984" and so on, there are some very tangible benefits from how this system is employed here: I don't have to jump through hurdles to vote in elections, instead of having to register for voting I'm invited to do that (by my "Einwohnermeldeamt") every time there is one, since the day I turned 18. It's very easy for me to prove (beyond reasonable doubt) that I am who I am where that is necessary. Note that the police don't have the need to know who you are without reasons. And it's quite comforting to know that in situations where they can demand it, the can't make up their own rules what they accept as ID and what not.
Nils
On Thu, 2008-08-28 at 10:49 +0200, Nils Philippsen wrote:
Just about every bank in Germany would operate like this. After all, we do have a national ID that is hard enough to fake(*) and if you present it to entities like banks, they usually make a copy and can verify the data on the ID with the "Einwohnermeldeamt" (possibly "residents registration office").
In other words, they only marginally trust that ID card. Therefore, they cross-check with the office (but most likely NOT the person that issued the card) that what is on that document is ACTUALLY true.
Think of the card as the RPM coming out of Fedora build system, signed with Fedora key (i.e. all the hard to get paraphernalia that the card is made of). Think of the phone call to the office as a comparison to an independently built RPM. Think of the act of opening the bank account as an independent signatory signing the RPM.
On Thu, 2008-08-28 at 20:07 +1000, Bojan Smojver wrote:
On Thu, 2008-08-28 at 10:49 +0200, Nils Philippsen wrote:
Just about every bank in Germany would operate like this. After all, we do have a national ID that is hard enough to fake(*) and if you present it to entities like banks, they usually make a copy and can verify the data on the ID with the "Einwohnermeldeamt" (possibly "residents registration office").
In other words, they only marginally trust that ID card. Therefore, they cross-check with the office (but most likely NOT the person that issued the card) that what is on that document is ACTUALLY true.
I don't know if they do that at all or with everybody, in fact I'm pretty sure they will accept the IDs at face value. Maybe they verify old documents that don't have all the gizmos that make the ID hard to fake today. I just wanted to point out that they can do that.
Think of the card as the RPM coming out of Fedora build system, signed with Fedora key (i.e. all the hard to get paraphernalia that the card is made of). Think of the phone call to the office as a comparison to an independently built RPM. Think of the act of opening the bank account as an independent signatory signing the RPM.
All this doesn't help much if somebody manages to feed the process with corrupt data in the beginning -- be that a fake ID when applying for replacement for an expired ID, or somebody corrupting the repository that keeps the source that is fed to the builders. Besides the difficulties in making distributed build verification work solidly, we have to harden the system/process that feeds the builders in the first place -- which is already much easier than say hacking the builders. Otherwise it's just GIGO, all pain and no gain.
Nils
On Thu, 2008-08-28 at 12:48 +0200, Nils Philippsen wrote:
I don't know if they do that at all or with everybody, in fact I'm pretty sure they will accept the IDs at face value. Maybe they verify old documents that don't have all the gizmos that make the ID hard to fake today. I just wanted to point out that they can do that.
Yeah, it's like customs. You can scrape by, but if they do cross-check, you're history if you have nasties in your bag :-)
All this doesn't help much if somebody manages to feed the process with corrupt data in the beginning
I think I mentioned already that such a thing would be (obviously) beyond the scope of this.
But look, it is obvious that we (and by this I mean myself and the majority of people in the thread) have fundamentally contradicting views about this, so let's agree to disagree, OK? I don't want to drag this forever.
Consider my proposal withdrawn.
Nils Philippsen wrote:
On Thu, 2008-08-28 at 20:07 +1000, Bojan Smojver wrote:
On Thu, 2008-08-28 at 10:49 +0200, Nils Philippsen wrote:
Just about every bank in Germany would operate like this. After all, we do have a national ID that is hard enough to fake(*) and if you present it to entities like banks, they usually make a copy and can verify the data on the ID with the "Einwohnermeldeamt" (possibly "residents registration office").
In other words, they only marginally trust that ID card. Therefore, they cross-check with the office (but most likely NOT the person that issued the card) that what is on that document is ACTUALLY true.
I don't know if they do that at all or with everybody, in fact I'm pretty sure they will accept the IDs at face value. Maybe they verify old documents that don't have all the gizmos that make the ID hard to fake today. I just wanted to point out that they can do that.
Think of the card as the RPM coming out of Fedora build system, signed with Fedora key (i.e. all the hard to get paraphernalia that the card is made of). Think of the phone call to the office as a comparison to an independently built RPM. Think of the act of opening the bank account as an independent signatory signing the RPM.
All this doesn't help much if somebody manages to feed the process with corrupt data in the beginning -- be that a fake ID when applying for replacement for an expired ID, or somebody corrupting the repository that keeps the source that is fed to the builders. Besides the difficulties in making distributed build verification work solidly, we have to harden the system/process that feeds the builders in the first place -- which is already much easier than say hacking the builders. Otherwise it's just GIGO, all pain and no gain.
Nils
It is (almost) all about process. Documented, auditable, verifiable process that is reviewed periodically by someone other than those using it to make sure it is in fact being followed, verified, audited, etc.
In a sense, fedora and rht just had a hostile audit. They failed. Now close the feedback loop and use what you've learned to fix the process. New keys are nice, but ultimately useless w/o a better process to safeguard them, to ensure their correct usage, by the correct people on the correct content, and to enable their revokation if it happens again.
As Schneirer says, good security is hard. This is not a new path, nothing needs to be invented, just some good solid principles (separation of duty, verification of trust, independent review and audit, etc) followed.
Apparently, there was not enough process in place, or the processes that were in place were not followed, or they are (hopefully were) the wrong processes.
Bojan Smojver wrote:
But what if it is the src rpm that is compromised so the builds will be identical because they both contain the modification?
That is not exactly the compromise of the build system and/or Fedora key, now is it?
Is one significantly harder than the other? If it goes unnoticed the end result could be the same.
If your own contributors are subverting the system by uploading borked source, the mutli-key system isn't going to help (and I never claimed that).
I'm not proposing an intentional trojan source submission, but a compromise that modifies it in an unexpected way. I'd think if you go to the trouble to compare builds you'd also want an end-to-end validity check on the input to be sure it wasn't compromised either at the source or in transit.
Les Mikesell <lesmikesell <at> gmail.com> writes:
Is one significantly harder than the other? If it goes unnoticed the end result could be the same.
This depends on many factors and it has no simple answer. Packagers usually rely on checksums and signatures of upstream source before uploading to Fedora CVS (or so I hope :-). But, there is always a possibility that this can be compromised, of course.
The result is, however, not the same. Compromise of Fedora build system and/or key can compromise all packages in Fedora. Compromise of a single package cannot (generally speaking - there are exceptions).
In any event, just because one security measure doesn't help with every possible compromise, doesn't mean it doesn't help at all.
I'm not proposing an intentional trojan source submission, but a compromise that modifies it in an unexpected way. I'd think if you go to the trouble to compare builds you'd also want an end-to-end validity check on the input to be sure it wasn't compromised either at the source or in transit.
See above. But yeah, we may have signatories sign off on source RPMs first, before they are being built by alternative, independent build systems. It's a valid point.
The main idea remains the same: "You can fool some of the people all of the time, and all of the people some of the time, but you can not fool all of the people all of the time."
-- Bojan
Jeff Spaleta <jspaleta <at> gmail.com> writes:
How about you back up and just work on this very specific problem of deterministically doing build comparisons across disparate build systems ..before we even begin to discuss how a multiple sigantory process which relies on that.
We didn't even find out how many packages differ in significant ways by doing raw checksums. But look, I already said that my proposal is withdrawn until we invent a way of doing this correctly.
They would have to choose to run koji, instead of their own setups.
If you re-read the thread, you'll notice that I said that we'd also build stuff for them (i.e. we'd also run their setups). You know, this thing called cooperation. It would not be just one way.
-- Bojan
Jeff Spaleta <jspaleta <at> gmail.com> writes:
They would have to choose to run koji, instead of their own setups. I'm not going to hold my breath on that. If you were going to lobby them to use koji, you should start with OpenSuse and help them integrate the features they need into the koji base. Even if the multiple signatory idea does not pan out, having OpenSuse as koji user and contributor would be great.
I completely missed here that you seem to be under the impression that I'm suggesting that everyone should accept Fedora's build system as their own. Absolutely not (monoculture sucks anyway). This would all be in addition to what they already have and completely unrelated to how they build their own software.
-- Bojan
This is an interesting idea. Cross checking on multiple build mashines with private signed email (containg the checksums of source and binary, stripped away from machine specific data) send to the signatories - this all makes it alot difficult for someone to crack.