On 21 July 2008, Josh Bressers wrote:
On 19 July 2008, "Justin Cappos" wrote:
By the way, did you remove the ability for mirror admins to select a subnet where they'll serve all of the traffic? We're particularly concerned about this issue in the short term. We took our mirror down (mirror1.lockdownhosting.com) quite a while ago so we can't check for ourselves.
I don't know the answer to this, so I'm adding the Fedora Infrastructure list to the CC.
For you Fedora Infrastructure guys, this is regarding this paper: http://www.cs.arizona.edu/people/justin/packagemanagersecurity/
Thanks.
Sadly I'm resending this. The Fedora Infrastructure group doesn't have their act together, so my original message is stuck in a moderator queue nobody has the password for. I've subscribed to the list for the purpose of sending this mail.
Can someone respond to the above question from Justin.
Thanks.
On Fri, 25 Jul 2008, Josh Bressers wrote:
On 21 July 2008, Josh Bressers wrote:
On 19 July 2008, "Justin Cappos" wrote:
By the way, did you remove the ability for mirror admins to select a subnet where they'll serve all of the traffic? We're particularly concerned about this issue in the short term. We took our mirror down (mirror1.lockdownhosting.com) quite a while ago so we can't check for ourselves.
I don't know the answer to this, so I'm adding the Fedora Infrastructure list to the CC.
For you Fedora Infrastructure guys, this is regarding this paper: http://www.cs.arizona.edu/people/justin/packagemanagersecurity/
Thanks.
Sadly I'm resending this. The Fedora Infrastructure group doesn't have their act together, so my original message is stuck in a moderator queue nobody has the password for. I've subscribed to the list for the purpose of sending this mail.
Can someone respond to the above question from Justin.
Wow, what a nice way to ask for help. One wonders why you didn't just file a bug or contact admin@fedoraproject.org.
https://fedorahosted.org/fedora-infrastructure/
Side note, I'm still waiting on RHIT to email me a password for the list.
-Mike
On Fri, 25 Jul 2008, Mike McGrath wrote:
On Fri, 25 Jul 2008, Josh Bressers wrote:
On 21 July 2008, Josh Bressers wrote:
On 19 July 2008, "Justin Cappos" wrote:
By the way, did you remove the ability for mirror admins to select a subnet where they'll serve all of the traffic? We're particularly concerned about this issue in the short term. We took our mirror down (mirror1.lockdownhosting.com) quite a while ago so we can't check for ourselves.
AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror?
-Mike
On Fri, 2008-07-25 at 10:37 -0500, Mike McGrath wrote:
AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror?
More like taking over a subnet and directing all clients at a rouge mirror.
On Fri, 25 Jul 2008, Jesse Keating wrote:
On Fri, 2008-07-25 at 10:37 -0500, Mike McGrath wrote:
AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror?
More like taking over a subnet and directing all clients at a rouge mirror.
<nod> that makes more sense. Domsch?
-Mike
On Fri, Jul 25, 2008 at 10:43:59AM -0500, Mike McGrath wrote:
On Fri, 25 Jul 2008, Jesse Keating wrote:
On Fri, 2008-07-25 at 10:37 -0500, Mike McGrath wrote:
AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror?
More like taking over a subnet and directing all clients at a rouge mirror.
<nod> that makes more sense. Domsch?
Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used.
On Fri, 25 Jul 2008, Matt Domsch wrote:
On Fri, Jul 25, 2008 at 10:43:59AM -0500, Mike McGrath wrote:
On Fri, 25 Jul 2008, Jesse Keating wrote:
On Fri, 2008-07-25 at 10:37 -0500, Mike McGrath wrote:
AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror?
More like taking over a subnet and directing all clients at a rouge mirror.
<nod> that makes more sense. Domsch?
Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used.
Perhaps it might also be a good idea to add a comment to the default yum.conf for gpgcheck explaining what a bad idea it is to set to 0. I could imagine people setting it to 0 not understanding what they're doing. Especially if they're familiar with gpg's encryption bits, but not its signing functionality.
-Mike
On 25 July 2008, Matt Domsch wrote:
Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used.
How does one get this subnet delegation though? Can I request any subnet I want, or do we do some sort of verification?
What happens if the client decided its mirror is bad, I presume it will go off and find a better one, even with delegation?
Thanks.
On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote:
On 25 July 2008, Matt Domsch wrote:
Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used.
How does one get this subnet delegation though? Can I request any subnet I want, or do we do some sort of verification?
At present there is no verification (I'm not at all sure how one _could_ verify except by ARIN & co delegation). However there are limits as to how large a block can be requested. Nothing larger than a IPv4 /16 can be automatically requested. Fedora Infrastructure admins can add larger blocks, and request ARIN & co data when doing so.
What happens if the client decided its mirror is bad, I presume it will go off and find a better one, even with delegation?
Yes, the mirrorlist returned includes quite a few mirrors, in priority order.
On 25 July 2008, Matt Domsch wrote:
On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote:
On 25 July 2008, Matt Domsch wrote:
Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used.
How does one get this subnet delegation though? Can I request any subnet I want, or do we do some sort of verification?
At present there is no verification (I'm not at all sure how one _could_ verify except by ARIN & co delegation). However there are limits as to how large a block can be requested. Nothing larger than a IPv4 /16 can be automatically requested. Fedora Infrastructure admins can add larger blocks, and request ARIN & co data when doing so.
That's a lot of IPs though. Can I request multiple /16s, or only one?
How many mirrors are doing this? Does the mirror have to be part of the /16 to request it?
Thanks for the patience here. I'm trying to understand the risk we're dealing with.
Thanks.
On Fri, Jul 25, 2008 at 01:52:26PM -0400, Josh Bressers wrote:
That's a lot of IPs though. Can I request multiple /16s, or only one?
As many as you like. And recall, such changes are made using your FAS credentials.
How many mirrors are doing this?
374 total Hosts 185 have at least 1 netblock entry 94 of these are "private" - don't serve the public
Does the mirror have to be part of the /16 to request it?
no. Take for example Dell's mirrors. Netblock 143.166/16 is Dell US, but the mirror IPs are located inside the 10/8 private space.
Thanks for the patience here. I'm trying to understand the risk we're dealing with.
I'm happy for the comments and review. Keep it coming!
On 25 July 2008, Matt Domsch wrote:
On Fri, Jul 25, 2008 at 01:52:26PM -0400, Josh Bressers wrote:
That's a lot of IPs though. Can I request multiple /16s, or only one?
As many as you like. And recall, such changes are made using your FAS credentials.
Are these ever checked? Does say a mail get generated every time someone adds one of these? My fear would be that someone could blanket quite a large IP space without anyone noticing. Granted that would no doubt generate a huge volume of traffic, but if they're serving up a frozen repo, they probably won't be pushing all that much data.
How many mirrors are doing this?
374 total Hosts 185 have at least 1 netblock entry 94 of these are "private" - don't serve the public
wow, that's quite a few. I wasn't expecting numbers this high honestly.
Does the mirror have to be part of the /16 to request it?
no. Take for example Dell's mirrors. Netblock 143.166/16 is Dell US, but the mirror IPs are located inside the 10/8 private space.
OK, so here is the problem the way I see it, signing the repository won't fix it. I'll try to explain this clearly, Justin can yell at me if I've gotten any of this wrong.
So let's say Mallory (the bad guy) decides that he wants to host a malicious mirror and wait for a nasty security flaw. He sets up his mirror and even claims some IP subnets to serve. Bob and Alice are happily installing valid updates from him for some period of time. Since Mallory has claimed to serve a specific subnet, he has a rather impressive view of what Bob and Alice have installed.
Now let's say there is a horrible security bug found in a mail server. Mallory knows for a fact that Bob and Alice both have it installed as he's been their mirror for a while. Mallory stops updating his mirror, so none of the users being served will get the mail server updates. Mallory also knows the IP address of the vulnerable clients and can easily break into their systems.
So from what I understand MirrorManager will check on the mirrors to ensure they're not out of date. Mallory knows this and makes sure that when MirrorManager connects to his mirror, it lies and serves up current metadata.
So here is the problem. The repodata was valid. The packages are signed. Even if we sign the repodata, this attack works. Being able to acquire an IP block simply makes this attack easier to do. It's still very possible that a bad mirror will wait for users to connect, serve up old content then use this knowledge to break into their system.
What this problem boils down to, is we need a way for clients to ask MirrorManager what the current valid repo data is. Ideally we want the results to be signed in some manner so it can't be spoofed.
Some thoughts I've had are:
1) Have MirrorManager use https and return some repo verification data. 2) Sign the repo data, and if it's older than X, don't use it (I don't like this solution, but it's probably the easiest, just push out a new signed repo file once a day, even if nothing changes.) 3) Always get repo data from fedoraproject.org (probably not practical due to resource issues) 4) use DNS, have the client query <repodata sha1sum>.repo.fedoraproject.org if the lookup fails, the repo is invalid. (this is really cheap from a resource standpoint, but hard to do technically) 5) ???
Thanks.
Josh Bressers wrote:
On 25 July 2008, Matt Domsch wrote:
On Fri, Jul 25, 2008 at 01:52:26PM -0400, Josh Bressers wrote:
That's a lot of IPs though. Can I request multiple /16s, or only one?
As many as you like. And recall, such changes are made using your FAS credentials.
Are these ever checked? Does say a mail get generated every time someone adds one of these? My fear would be that someone could blanket quite a large IP space without anyone noticing. Granted that would no doubt generate a huge volume of traffic, but if they're serving up a frozen repo, they probably won't be pushing all that much data.
How many mirrors are doing this?
374 total Hosts 185 have at least 1 netblock entry 94 of these are "private" - don't serve the public
wow, that's quite a few. I wasn't expecting numbers this high honestly.
Does the mirror have to be part of the /16 to request it?
no. Take for example Dell's mirrors. Netblock 143.166/16 is Dell US, but the mirror IPs are located inside the 10/8 private space.
OK, so here is the problem the way I see it, signing the repository won't fix it. I'll try to explain this clearly, Justin can yell at me if I've gotten any of this wrong.
So let's say Mallory (the bad guy) decides that he wants to host a malicious mirror and wait for a nasty security flaw. He sets up his mirror and even claims some IP subnets to serve. Bob and Alice are happily installing valid updates from him for some period of time. Since Mallory has claimed to serve a specific subnet, he has a rather impressive view of what Bob and Alice have installed.
Now let's say there is a horrible security bug found in a mail server. Mallory knows for a fact that Bob and Alice both have it installed as he's been their mirror for a while. Mallory stops updating his mirror, so none of the users being served will get the mail server updates. Mallory also knows the IP address of the vulnerable clients and can easily break into their systems.
So from what I understand MirrorManager will check on the mirrors to ensure they're not out of date. Mallory knows this and makes sure that when MirrorManager connects to his mirror, it lies and serves up current metadata.
So here is the problem. The repodata was valid. The packages are signed. Even if we sign the repodata, this attack works. Being able to acquire an IP block simply makes this attack easier to do. It's still very possible that a bad mirror will wait for users to connect, serve up old content then use this knowledge to break into their system.
What this problem boils down to, is we need a way for clients to ask MirrorManager what the current valid repo data is. Ideally we want the results to be signed in some manner so it can't be spoofed.
Some thoughts I've had are:
- Have MirrorManager use https and return some repo verification data.
We'd need to push repo verification data for both the latest and previous repo information. MirrorManager would have to be updated with new data as part of the updates/rawhide push so that it's always up to date with the mirror. We'll have to revise mirrormanager's caching model so that it always has access to the latest repository verification information.
- Sign the repo data, and if it's older than X, don't use it (I don't like this solution, but it's probably the easiest, just push out a new signed repo file once a day, even if nothing changes.)
This is going to have to have some policy attached to it. Do we continue to sign the repo data for EOL releases because people use them even though we tell people they're EOL?
- Always get repo data from fedoraproject.org (probably not practical due to resource issues)
This is the easiest to implement. It means the small repomd.xml file always comes from our server. But the rest of the metadata can come from the individual mirrors. However, it does mean that *very* large swaths of the mirrors will be unable to serve packages to users at any time because their metadata won't match with ours for some period after we have an update pushed. Maybe we could do this with versioned repodata and the client can decide how far back in time or how many past revisions it is willing to accept.
- use DNS, have the client query <repodata sha1sum>.repo.fedoraproject.org if the lookup fails, the repo is invalid. (this is really cheap from a resource standpoint, but hard to do technically)
I don't know enough about implementing this one to comment.
Also, all of these solutions need client-side support. That being the case, it should be generic enough that other repomd consuming clients and distributions will be willing to use it. Otherwise we'll be securing our updates and mirror infrastructure with the default package manager but doing nothing for the ecosystem as a whole.
-Toshio
On Fri, 2008-07-25 at 18:41 -0700, Toshio Kuratomi wrote:
- Always get repo data from fedoraproject.org (probably not practical due to resource issues)
This is the easiest to implement. It means the small repomd.xml file always comes from our server. But the rest of the metadata can come from the individual mirrors. However, it does mean that *very* large swaths of the mirrors will be unable to serve packages to users at any time because their metadata won't match with ours for some period after we have an update pushed. Maybe we could do this with versioned repodata and the client can decide how far back in time or how many past revisions it is willing to accept.
We don't need to version the metadata, we have timestamps in them already.
We can easily do a comparison from this timestamp to now. And we can set this number to be different from the base repo as to the updates repo.
But as you've already mentioned we're stuck with the question of EOL'd releases and how to deal with things deeply out of date.
I can make yum throw out warnings and alerts but at what point does it actually STOP doing anything and does that not open us up to a different kind of DoS?
I don't know enough about implementing this one to comment.
Also, all of these solutions need client-side support. That being the case, it should be generic enough that other repomd consuming clients and distributions will be willing to use it. Otherwise we'll be securing our updates and mirror infrastructure with the default package manager but doing nothing for the ecosystem as a whole.
We need to make sure that whatever we implement is trivially done so by people running a local downstream branch of fedora or centos or what-have-you. Or, as you say, we've saved ourselves and screwed everyone else.
-sv
On 25 July 2008, seth vidal wrote:
But as you've already mentioned we're stuck with the question of EOL'd releases and how to deal with things deeply out of date.
I can make yum throw out warnings and alerts but at what point does it actually STOP doing anything and does that not open us up to a different kind of DoS?
This is of course a policy decision that can be dictated via a configuration file. There is also the issue of what happens when the client keeps getting back "old" data? It need to go find some new data as in our attack scenario, they are running something that needs updating. I would think that the client should try some number of mirrors, if they all return the same old data, it's probably EOL or something similar. If an attacker has enough control over you to prevent you connecting to arbitrary mirrors, you likely have bigger issues than this.
We need to make sure that whatever we implement is trivially done so by people running a local downstream branch of fedora or centos or what-have-you. Or, as you say, we've saved ourselves and screwed everyone else.
Yes, I'm starting to think we should be having this discussion with yum upstream. Now that I have a grasp of what MirrorManager does, I don't think there's really anything to fix there. We need to fix the client.
Seth, which list would you suggest to move this discussion so I can stop pestering the infrastructure folks?
Thanks.
On Sat, 2008-07-26 at 13:06 -0400, Josh Bressers wrote:
This is of course a policy decision that can be dictated via a configuration file.
But our default is what people will use which is what we need to get straight.
There is also the issue of what happens when the client keeps getting back "old" data? It need to go find some new data as in our attack scenario, they are running something that needs updating. I would think that the client should try some number of mirrors, if they all return the same old data, it's probably EOL or something similar.
Except that if they are being given a single netblock mirror then they're sol for finding another mirror.
If an attacker has enough control over you to prevent you connecting to arbitrary mirrors, you likely have bigger issues than this.
They can keep you from knowing about other mirrors, definitely.
Yes, I'm starting to think we should be having this discussion with yum upstream. Now that I have a grasp of what MirrorManager does, I don't think there's really anything to fix there. We need to fix the client.
Unless throwing out warnings, errors and generally exploding when the repodata is too old is good enough, I'm not sure how much more yum is going to be able to do to alert the user that they are not getting any updates applied. And we need to make sure to draw packagekit into this discussion b/c they're going to need to honor the warnings, too.
If something is not broken but only 'worrisome' what is, from a security standpoint, a good enough warning?
We can't seem to rely on users reading logs, or even reading dialog output. At least that's what the gpg-key import discussion around PK seemed to illustrate.
Seth, which list would you suggest to move this discussion so I can stop pestering the infrastructure folks?
yum-devel@lists.linux.duke.edu, thanks,
-sv
Yes, you clearly described one of the attacks we see with MirrorManager.
A few comments:
- Have MirrorManager use https and return some repo verification data.
Is the verification data a signed repomd.xml? Can you expand on this a little?
By the way, before I forget it would be a good idea to avoid using a detached signature for repomd.xml. If you have the signature and data in separate files, you can have false positives for incorrect signatures (for example when the files are updated).
- Sign the repo data, and if it's older than X, don't use it (I don't like
this solution, but it's probably the easiest, just push out a new signed repo file once a day, even if nothing changes.)
You also want to make sure clients don't accept older versions of the metadata than what they've seen. In other words, if they'll accept metadata up to 1 week old, and they've downloaded metadata from yesterday, they shouldn't accept metadata that was signed 5 days ago.
- Always get repo data from fedoraproject.org (probably not practical due to resource issues)
I think this is similar to what openSUSE / SUSE Linux Enterprise do (they actually serve signed metadata from their Download Redirector), so it may be practical for you to do in practice. However, you need to worry about man-in-the-middle attackers...
- use DNS, have the client query
<repodata sha1sum>.repo.fedoraproject.org if the lookup fails, the repo is invalid. (this is really cheap from a resource standpoint, but hard to do technically)
Same comment about man-in-the-middle attacks. DNS-cache poisoning trivially circumvents this protection...
Thanks, Justin
Seth, James Antill, and I met a week ago to discuss. These are the steps we believe are necessary to resolve. I didn't realize this hadn't been posted yet.
1. repomd.xml needs to be signed. Either attached or detached sig (advice sought). If attached, format would be
<repomd></repomd> delimiter / size of above ? signature
2. mirrormanager will start using metalinks or something quite like that, to publish the repomd.xml file pointers on the various mirrors worldwide. This will include typed checksums, a time stamp, and a file size, plus the various URL methods and countries for the mirrors. (I've been coding this on planes this week).
One challenge here is that the metalink XML format doesn't allow for
1 set of attributes for a given file. We would like to include
attributes for repomd.xml for the last several days, because slightly stale mirrors really are OK (pending rsync).
3. mirrormanager requests will use https.
4. yum will enable https cert verification and CRL checking. Right now it secures the stream but doesn't verify the cert.
5. yum will grow repomd.xml signature check
6. yum will grow metalink parsing
7. fedora-release yum.repos.d/* files will point at the new metalink=https://mirrors.fedoraproject.org/metalink?... URL.
Seem reasonable?
On 28 July 2008, Matt Domsch wrote:
Seth, James Antill, and I met a week ago to discuss. These are the steps we believe are necessary to resolve. I didn't realize this hadn't been posted yet.
- repomd.xml needs to be signed. Either attached or detached sig (advice sought). If attached, format would be
<repomd></repomd> delimiter / size of above ? signature
- mirrormanager will start using metalinks or something quite like that, to publish the repomd.xml file pointers on the various mirrors worldwide. This will include typed checksums, a time stamp, and a file size, plus the various URL methods and countries for the mirrors. (I've been coding this on planes this week).
One challenge here is that the metalink XML format doesn't allow for
1 set of attributes for a given file. We would like to include
attributes for repomd.xml for the last several days, because slightly stale mirrors really are OK (pending rsync).
mirrormanager requests will use https.
yum will enable https cert verification and CRL checking. Right now it secures the stream but doesn't verify the cert.
yum will grow repomd.xml signature check
yum will grow metalink parsing
fedora-release yum.repos.d/* files will point at the new metalink=https://mirrors.fedoraproject.org/metalink?... URL.
Seem reasonable?
This does seem reasonable, the only question I have is how often does yum ask MirrorManager for a new repo.xml file?
This strikes me as a good solution to the problems at hand.
Thanks guys, the work is appreciated.
On Mon, 2008-07-28 at 12:07 -0500, Matt Domsch wrote:
- repomd.xml needs to be signed. Either attached or detached sig (advice sought). If attached, format would be
I would prefer a detached sig, so that the checksum of repomd.xml itself doesn't change if the GPG sig for it does. This is important as there are control files in the compose to track consistency of the tree itself, and having the repomd.xml change it's key would invalidate this control file.
On Mon, 2008-07-28 at 14:25 -0400, Jesse Keating wrote:
On Mon, 2008-07-28 at 12:07 -0500, Matt Domsch wrote:
- repomd.xml needs to be signed. Either attached or detached sig (advice sought). If attached, format would be
I would prefer a detached sig, so that the checksum of repomd.xml itself doesn't change if the GPG sig for it does. This is important as there are control files in the compose to track consistency of the tree itself, and having the repomd.xml change it's key would invalidate this control file.
detached sig definitely. Independent of how (or why) this is done we will maintain backward compat. Signing the repomd.xml directly will not allow backward compat (nor cross compat with apt/smart/etc).
I've already written the code for the detached sig - it'll be checked into yum upstream this afternoon.
-sv
On Mon, Jul 28, 2008 at 1:07 PM, Matt Domsch Matt_Domsch@dell.com wrote:
- repomd.xml needs to be signed. Either attached or detached sig
(advice sought). If attached, format would be
I see a number of good ideas to improve the situation, but I don't think I've seen anyone suggest the following.
Would it be feasible to audit the mirror content? We have the list of mirrors, we know what the content should be. I think we'd only need to validate the mirrored repomd.xml, right? Doesn't seem to onerous...
yes, yes, not perfect, malicious mirror could change the content, etc, but at least we'd have some measure of detection.
On Mon, 2008-07-28 at 17:28 -0400, Mike McLean wrote:
On Mon, Jul 28, 2008 at 1:07 PM, Matt Domsch Matt_Domsch@dell.com wrote:
- repomd.xml needs to be signed. Either attached or detached sig
(advice sought). If attached, format would be
I see a number of good ideas to improve the situation, but I don't think I've seen anyone suggest the following.
Would it be feasible to audit the mirror content? We have the list of mirrors, we know what the content should be. I think we'd only need to validate the mirrored repomd.xml, right? Doesn't seem to onerous...
yes, yes, not perfect, malicious mirror could change the content, etc, but at least we'd have some measure of detection.
which is the point. A malicious mirror could safely lie to us and not lie to their targets.
Additionally, mirrormanager DOES check the mirrors.
-sv
On Mon, 2008-07-28 at 17:29 -0400, seth vidal wrote:
On Mon, 2008-07-28 at 17:28 -0400, Mike McLean wrote:
On Mon, Jul 28, 2008 at 1:07 PM, Matt Domsch Matt_Domsch@dell.com wrote:
- repomd.xml needs to be signed. Either attached or detached sig
(advice sought). If attached, format would be
I see a number of good ideas to improve the situation, but I don't think I've seen anyone suggest the following.
Would it be feasible to audit the mirror content? We have the list of mirrors, we know what the content should be. I think we'd only need to validate the mirrored repomd.xml, right? Doesn't seem to onerous...
yes, yes, not perfect, malicious mirror could change the content, etc, but at least we'd have some measure of detection.
which is the point. A malicious mirror could safely lie to us and not lie to their targets.
Additionally, mirrormanager DOES check the mirrors.
Except, of course, for mirrors which are internal to a specific site and thus can't be contacted by MM
Jeremy
On Mon, 2008-07-28 at 17:37 -0400, Jeremy Katz wrote:
On Mon, 2008-07-28 at 17:29 -0400, seth vidal wrote:
On Mon, 2008-07-28 at 17:28 -0400, Mike McLean wrote:
On Mon, Jul 28, 2008 at 1:07 PM, Matt Domsch Matt_Domsch@dell.com wrote:
- repomd.xml needs to be signed. Either attached or detached sig
(advice sought). If attached, format would be
I see a number of good ideas to improve the situation, but I don't think I've seen anyone suggest the following.
Would it be feasible to audit the mirror content? We have the list of mirrors, we know what the content should be. I think we'd only need to validate the mirrored repomd.xml, right? Doesn't seem to onerous...
yes, yes, not perfect, malicious mirror could change the content, etc, but at least we'd have some measure of detection.
which is the point. A malicious mirror could safely lie to us and not lie to their targets.
Additionally, mirrormanager DOES check the mirrors.
Except, of course, for mirrors which are internal to a specific site and thus can't be contacted by MM
and if they're evil then the folks involved are screwed anyway....
which, after all, is why we're in favor of repomd.xml signing
-sv
On Mon, Jul 28, 2008 at 5:38 PM, seth vidal skvidal@fedoraproject.org wrote:
On Mon, 2008-07-28 at 17:37 -0400, Jeremy Katz wrote:
Except, of course, for mirrors which are internal to a specific site and thus can't be contacted by MM
and if they're evil then the folks involved are screwed anyway....
Well, depends. Having an evil system on an internal network is bad, but not as bad as having your servers compromised.
On Mon, Jul 28, 2008 at 5:29 PM, seth vidal skvidal@fedoraproject.org wrote:
On Mon, 2008-07-28 at 17:28 -0400, Mike McLean wrote:
Would it be feasible to audit the mirror content? We have the list of mirrors, we know what the content should be. I think we'd only need to validate the mirrored repomd.xml, right? Doesn't seem to onerous...
yes, yes, not perfect, malicious mirror could change the content, etc, but at least we'd have some measure of detection.
which is the point. A malicious mirror could safely lie to us and not lie to their targets.
Depends on who 'us' is. We could run such checks from a number of different locations.
Anyway, I didn't mean to suggest this as a total solution.
Additionally, mirrormanager DOES check the mirrors.
Good to know.
On Fri, 2008-07-25 at 19:04 -0700, Justin Cappos wrote:
Yes, you clearly described one of the attacks we see with MirrorManager.
A few comments:
- Have MirrorManager use https and return some repo verification data.
Is the verification data a signed repomd.xml? Can you expand on this a little?
By the way, before I forget it would be a good idea to avoid using a detached signature for repomd.xml. If you have the signature and data in separate files, you can have false positives for incorrect signatures (for example when the files are updated).
And if that happens then it regets to another mirror and attempts to find a valid cert. Detached sigs is how it will be b/c otherwise we're breaking backward compat with older clients. And that's just chockful of pain.
- Sign the repo data, and if it's older than X, don't use it (I don't like
this solution, but it's probably the easiest, just push out a new signed repo file once a day, even if nothing changes.)
You also want to make sure clients don't accept older versions of the metadata than what they've seen. In other words, if they'll accept metadata up to 1 week old, and they've downloaded metadata from yesterday, they shouldn't accept metadata that was signed 5 days ago.
And do what if they can't? think about it - at some point every repo will be 'too old' and then what? The client cannot use it at all? If that's the case then we're intentionally putting a sundown into our code, what a nightmare that is.
The best we can do is warn and alert the user that the metadata is old. Stopping working is just preposterous - it's just a different kind of DoS if we do that. If the user cannot read and take warnings then there is really nothing that can be done for them.
-sv
I was wondering if any changes have been made or are planned for MirrorManager (i.e. preventing mirrors from arbitrary grabbing parts of the address space). We're submitting the final version of our paper soon (the version that will appear in print) and I'd like to include any updates about this.
Thanks, Justin
On Fri, Jul 25, 2008 at 6:11 PM, Josh Bressers bressers@redhat.com wrote:
On 25 July 2008, Matt Domsch wrote:
On Fri, Jul 25, 2008 at 01:52:26PM -0400, Josh Bressers wrote:
That's a lot of IPs though. Can I request multiple /16s, or only one?
As many as you like. And recall, such changes are made using your FAS credentials.
Are these ever checked? Does say a mail get generated every time someone adds one of these? My fear would be that someone could blanket quite a large IP space without anyone noticing. Granted that would no doubt generate a huge volume of traffic, but if they're serving up a frozen repo, they probably won't be pushing all that much data.
How many mirrors are doing this?
374 total Hosts 185 have at least 1 netblock entry 94 of these are "private" - don't serve the public
wow, that's quite a few. I wasn't expecting numbers this high honestly.
Does the mirror have to be part of the /16 to request it?
no. Take for example Dell's mirrors. Netblock 143.166/16 is Dell US, but the mirror IPs are located inside the 10/8 private space.
OK, so here is the problem the way I see it, signing the repository won't fix it. I'll try to explain this clearly, Justin can yell at me if I've gotten any of this wrong.
So let's say Mallory (the bad guy) decides that he wants to host a malicious mirror and wait for a nasty security flaw. He sets up his mirror and even claims some IP subnets to serve. Bob and Alice are happily installing valid updates from him for some period of time. Since Mallory has claimed to serve a specific subnet, he has a rather impressive view of what Bob and Alice have installed.
Now let's say there is a horrible security bug found in a mail server. Mallory knows for a fact that Bob and Alice both have it installed as he's been their mirror for a while. Mallory stops updating his mirror, so none of the users being served will get the mail server updates. Mallory also knows the IP address of the vulnerable clients and can easily break into their systems.
So from what I understand MirrorManager will check on the mirrors to ensure they're not out of date. Mallory knows this and makes sure that when MirrorManager connects to his mirror, it lies and serves up current metadata.
So here is the problem. The repodata was valid. The packages are signed. Even if we sign the repodata, this attack works. Being able to acquire an IP block simply makes this attack easier to do. It's still very possible that a bad mirror will wait for users to connect, serve up old content then use this knowledge to break into their system.
What this problem boils down to, is we need a way for clients to ask MirrorManager what the current valid repo data is. Ideally we want the results to be signed in some manner so it can't be spoofed.
Some thoughts I've had are:
- Have MirrorManager use https and return some repo verification data.
- Sign the repo data, and if it's older than X, don't use it (I don't like this solution, but it's probably the easiest, just push out a new signed repo file once a day, even if nothing changes.)
- Always get repo data from fedoraproject.org (probably not practical due to resource issues)
- use DNS, have the client query <repodata sha1sum>.repo.fedoraproject.org if the lookup fails, the repo is invalid. (this is really cheap from a resource standpoint, but hard to do technically)
- ???
Thanks.
-- JB
On Tue, Jul 29, 2008 at 11:35:03AM -0500, Justin Cappos wrote:
I was wondering if any changes have been made or are planned for MirrorManager (i.e. preventing mirrors from arbitrary grabbing parts of the address space). We're submitting the final version of our paper soon (the version that will appear in print) and I'd like to include any updates about this.
Yesterday I sent the long list of steps planned or under way. Some of these involve MM, some yum.
As for "arbitrary grabbing of address space", I'm open to ideas. Perhaps a /16 is too large for "anyone" to be able to grab - e.g. could should limit the auto-granted size by some amount. However, it doesn't eliminate the concern. If Mallory wants to attack specifically Alice, he only need know the addresses Alice is likely to be coming from and add those in, even one-at-a-time.
Restricting to a /16 seemed reasonable to me. A good balance of "big enough to be useful", yet small enough that it can't affect too many people. Larger allocations are available on request, by showing some form of ARIN assignment. Still, one could request such and run a mirror inside that assignment that is still malicious. And I'm not willing to throw out this very useful feature, for fear someone could use it for evil.
On 29 July 2008, "Domsch, Matt" wrote:
On Tue, Jul 29, 2008 at 11:35:03AM -0500, Justin Cappos wrote:
I was wondering if any changes have been made or are planned for MirrorManager (i.e. preventing mirrors from arbitrary grabbing parts of the address space). We're submitting the final version of our paper soon (the version that will appear in print) and I'd like to include any updates about this.
As for "arbitrary grabbing of address space", I'm open to ideas. Perhaps a /16 is too large for "anyone" to be able to grab - e.g. could should limit the auto-granted size by some amount. However, it doesn't eliminate the concern. If Mallory wants to attack specifically Alice, he only need know the addresses Alice is likely to be coming from and add those in, even one-at-a-time.
Restricting to a /16 seemed reasonable to me. A good balance of "big enough to be useful", yet small enough that it can't affect too many people. Larger allocations are available on request, by showing some form of ARIN assignment. Still, one could request such and run a mirror inside that assignment that is still malicious. And I'm not willing to throw out this very useful feature, for fear someone could use it for evil.
I think this is fine, and even desirable in most instances. As long as yum will try the next mirror in the list if the primary one is outdated, this becomes a non issue.
If anything, I'd suggest you advertise this feature more. I had no idea MirrorManager could do this, and I suspect there are a number of organizations that could benefit from knowing this.
Thanks.
You might also think about requiring the mirror's IP address to fall in the subnet (or else they ask for your approval). This might further complicate an attacker using this for evil.
Thank you for providing this information... Justin
On Wed, Jul 30, 2008 at 4:59 AM, Josh Bressers bressers@redhat.com wrote:
On 29 July 2008, "Domsch, Matt" wrote:
On Tue, Jul 29, 2008 at 11:35:03AM -0500, Justin Cappos wrote:
I was wondering if any changes have been made or are planned for MirrorManager (i.e. preventing mirrors from arbitrary grabbing parts of the address space). We're submitting the final version of our paper soon (the version that will appear in print) and I'd like to include any updates about this.
As for "arbitrary grabbing of address space", I'm open to ideas. Perhaps a /16 is too large for "anyone" to be able to grab - e.g. could should limit the auto-granted size by some amount. However, it doesn't eliminate the concern. If Mallory wants to attack specifically Alice, he only need know the addresses Alice is likely to be coming from and add those in, even one-at-a-time.
Restricting to a /16 seemed reasonable to me. A good balance of "big enough to be useful", yet small enough that it can't affect too many people. Larger allocations are available on request, by showing some form of ARIN assignment. Still, one could request such and run a mirror inside that assignment that is still malicious. And I'm not willing to throw out this very useful feature, for fear someone could use it for evil.
I think this is fine, and even desirable in most instances. As long as yum will try the next mirror in the list if the primary one is outdated, this becomes a non issue.
If anything, I'd suggest you advertise this feature more. I had no idea MirrorManager could do this, and I suspect there are a number of organizations that could benefit from knowing this.
Thanks.
-- JB
On Wed, Jul 30, 2008 at 08:42:44AM -0700, Justin Cappos wrote:
You might also think about requiring the mirror's IP address to fall in the subnet (or else they ask for your approval). This might further complicate an attacker using this for evil.
The challenge here is
a) private servers often are on RFC1918 addresses, so they don't fall inside the public-visible netblock assignments. If it's a private server, MM doesn't even crawl it (they're likely unreachable anyhow), relying on them to run report_mirror. This also keeps our crawl times down to 4-6 hours, it only crawls the 50% of listed servers that are public.
b) malicious sysadmins could change their DNS entry after getting the netblock set up by a Fedora sysadmin, so as to no longer be inside the netblock.
I feel the window of opportunity here is small, and we're going to make changes to make it even smaller. Users can't install unsigned packages, the worst a malicious mirror can do is serve "stale" content for a period of time we'll be able to control (it may be ridiculously small, like "never" (which is easy to implement but a PITA for mirrors that sync only once a day), or up to 1 week (I haven't worked out how to do this cleanly, but it's nicer to users of mirrors who are good citizens) which is clearly what I want.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Matt Domsch wrote:
On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote:
On 25 July 2008, Matt Domsch wrote:
Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used.
How does one get this subnet delegation though? Can I request any subnet I want, or do we do some sort of verification?
At present there is no verification (I'm not at all sure how one _could_ verify except by ARIN & co delegation). However there are limits as to how large a block can be requested. Nothing larger than a IPv4 /16 can be automatically requested. Fedora Infrastructure admins can add larger blocks, and request ARIN & co data when doing so.
What happens if the client decided its mirror is bad, I presume it will go off and find a better one, even with delegation?
Yes, the mirrorlist returned includes quite a few mirrors, in priority order.
Our testing showed that when our client was in a MirrorManager-defined CIDR block for a mirror, the returned mirrorlist included only the single mirror. -- It's dangerous either way, of course, but I'm just wondering if our testing was faulty, if this has changed since we tested, or if it might be behaving differently than you expect.
Possibly you tested with a block that was already defined by other mirrors and so multiple entries were returned in the mirrorist? That's just a guess, we didn't test with a block that was defined by more than one mirror (as far as we knew, at least).
- -- Justin Samuel https://www.cs.arizona.edu/~jsamuel/ gpg: 0xDDF1F3EE [66EF 84E2 F184 B140 712B 55A7 2B96 AB8F DDF1 F3EE]
Fedora 7 definitely behaves differently than Fedora 8 and 9. The behavior I describe began with F8. For F7 and earlier, the yum policy would chose any random mirror from the returned list, so having many mirrors on the list, some of which are unreachable from inside an organization, would be bad. The yum default policy was changed to treat the mirrorlist as a priority list, so MM returns a longer list.
-- Matt Domsch Linux Technology Strategist, Dell Office of the CTO linux.dell.com & www.dell.com/linux
-----Original Message----- From: Justin Samuel [mailto:jsamuel.cs.arizona.edu@gmail.com] On Behalf Of Justin Samuel Sent: Friday, July 25, 2008 1:36 PM To: Domsch, Matt Cc: Josh Bressers; Mike McGrath; fedora-infrastructure-list@redhat.com; Justin Cappos; security@fedoraproject.org Subject: Re: YUM security issues...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Matt Domsch wrote:
On Fri, Jul 25, 2008 at 12:46:15PM -0400, Josh Bressers wrote:
On 25 July 2008, Matt Domsch wrote:
Yes, this is a known challenge with subnet delegation in MirrorManager. We're trusting package signing (and soon, repodata signing) to prevent rogue mirrors from issuing unsigned data. In addition, I'm working on adding in a way to prevent stale mirrors (with signed content) from being used.
How does one get this subnet delegation though? Can I request any
subnet I
want, or do we do some sort of verification?
At present there is no verification (I'm not at all sure how one _could_ verify except by ARIN & co delegation). However there are limits as to how large a block can be requested. Nothing larger than a IPv4 /16 can be automatically requested. Fedora Infrastructure admins can add larger blocks, and request ARIN & co data when doing
so.
What happens if the client decided its mirror is bad, I presume it
will go
off and find a better one, even with delegation?
Yes, the mirrorlist returned includes quite a few mirrors, in priority
order.
Our testing showed that when our client was in a MirrorManager-defined CIDR block for a mirror, the returned mirrorlist included only the single mirror. -- It's dangerous either way, of course, but I'm just wondering if our testing was faulty, if this has changed since we tested, or if it might be behaving differently than you expect.
Possibly you tested with a block that was already defined by other mirrors and so multiple entries were returned in the mirrorist? That's just a guess, we didn't test with a block that was defined by more than one mirror (as far as we knew, at least).
- -- Justin Samuel https://www.cs.arizona.edu/~jsamuel/ gpg: 0xDDF1F3EE [66EF 84E2 F184 B140 712B 55A7 2B96 AB8F DDF1 F3EE]
On 25 July 2008, Mike McGrath wrote:
On Fri, 25 Jul 2008, Mike McGrath wrote:
On Fri, 25 Jul 2008, Josh Bressers wrote:
On 21 July 2008, Josh Bressers wrote:
On 19 July 2008, "Justin Cappos" wrote:
By the way, did you remove the ability for mirror admins to select a subnet where they'll serve all of the traffic? We're particularly concerned about this issue in the short term. We took our mirror down (mirror1.lockdownhosting.com) quite a while ago so we can't check for ourselves.
AFAIK, this service is still in place and working fine. Though I am a little confused about the question. It sounds like you'd like to direct all subnet traffic to a specific mirror. But you're also saying you took your mirror down. Are you worried people in your subnet are being directed to a down mirror?
No, the problem is what happens when a malicious mirror claims a subnet? This is currently being viewed as a security issue due to this research: http://www.cs.arizona.edu/people/justin/packagemanagersecurity/attacks-on-pa...
infrastructure@lists.fedoraproject.org
-
Jeremy Katz -
Jesse Keating -
Josh Bressers -
Justin Cappos -
Justin Samuel -
Matt Domsch -
Matt Domsch
-
Matt Domsch -
Mike McGrath -
Mike McLean -
Mike McLean -
Seth Vidal -
Toshio Kuratomi