Lets get this topic started. We've had a lot of requests to have fas authentication with third party groups (both nirik and dgilmore have requested such setups)
We can easily set things up so that public key's can be used. I still have grave security concerns about this though. The obvious fear is compromise of a third party box that allows an unauthorized person to then access our production servers.
The reality is this isn't much different from having an individual contributors machine get hacked and having them then log in to one of our boxes (this has happened once that I am aware of). The main difference though is how to target.
Lets assume an attacker wants to commit something bad to our servers. If they wanted to do it as me, they'd have to attack my workstation and somehow gain root access on the box. At that point they'd be able to take my keys or agent. A difficult task.
Now lets say that one of our third party machines is allowing people to build via mock for PPC (this is one real request). That third party box has the SSH keys of a number of people, lets say one of them is sysadmin-main. The attacker would need to merely create an fas account, request access to the group that gives access to that machine and they'd be able to take the ssh keys as people log in.
Now, I've never actually done this. It's just my understanding that it'd work that way. If you had root on a box and I sshed there with my ssh key, would you not have access to take the key and log in to other boxes as me?
So my question is, is this a real risk or is there a precaution in SSH preventing the attack i'm describing (basically a man in the middle type attack)
I can think of a number of options to prevent this but I'm curious what the rest of you think.
-Mike
On Wed, May 21, 2008 at 9:06 PM, Mike McGrath mmcgrath@redhat.com wrote:
Now lets say that one of our third party machines is allowing people to build via mock for PPC (this is one real request). That third party box has the SSH keys of a number of people, lets say one of them is sysadmin-main. The attacker would need to merely create an fas account, request access to the group that gives access to that machine and they'd be able to take the ssh keys as people log in.
Shouldn't the builds be done through Koji? Why would someone be doing builds with mock, but not through Koji?
Secondly, any attacker can already create a fas account, and use a bit of social engineering to gain enough access to do damage. There are already some obvious attack vectors in the current processes that are much more vulnerable than the configuration of third party systems.
That said, that doesn't mean we should ignore potential risk from blindly trusting third party systems. I would recommend that, at a minimum, third party systems should run with selinux on and enforcing. This would afford some additional assurance of the system's security, but won't solve all problems.
Now, I've never actually done this. It's just my understanding that it'd work that way. If you had root on a box and I sshed there with my ssh key, would you not have access to take the key and log in to other boxes as me?
I'm not a crypto guy by any stretch. But, my understanding is that no, that's not how it public key authentication should work. As I understand it, your private key is never sent across the wire and therefore even with root on the remote system, an attacker could only ever has access to your public key. This is assuming that your private key isn't also in your home directory on the remote system (i.e. the only local file they have is the authorized_keys). Please double-check RFC 4252 for more details.
So my question is, is this a real risk or is there a precaution in SSH preventing the attack i'm describing (basically a man in the middle type attack)
I think the mitm risk is fairly low. I think there are other risks, such as not having direct control over keeping third party systems up to date, or allowing third party systems running non-Fedora or non-RHEL distros (e.g. the latest Debian openssh debacle). Privilege escalation on the remote system is also a concern. Thankfully, selinux can mitigate the last fairly effectively.
I definitely don't think this should be done lightly.
I can think of a number of options to prevent this but I'm curious what the rest of you think.
I think that keeping the keys secure is only a small piece of the puzzle. One option that helps mitigate the keys issue to some extent is if we adopt a "pull" model with third party systems. We give them a public key that allows us to execute commands on the system and pull results from it, but the remote systems never get access into the Fedora production systems.
-Mike
---Brett.
On Wed, 21 May 2008, brett lentz wrote:
On Wed, May 21, 2008 at 9:06 PM, Mike McGrath mmcgrath@redhat.com wrote:
Now lets say that one of our third party machines is allowing people to build via mock for PPC (this is one real request). That third party box has the SSH keys of a number of people, lets say one of them is sysadmin-main. The attacker would need to merely create an fas account, request access to the group that gives access to that machine and they'd be able to take the ssh keys as people log in.
Shouldn't the builds be done through Koji? Why would someone be doing builds with mock, but not through Koji?
Some will want access to the build roots and things to test. There's other reasons third party people would want to setup accounts with us as well that don't involve the account system.
Secondly, any attacker can already create a fas account, and use a bit of social engineering to gain enough access to do damage. There are already some obvious attack vectors in the current processes that are much more vulnerable than the configuration of third party systems.
Maybe, maybe not. If someone wants to gain our trust (which, admittedly would require them to do a lot of work to 'prove' themselves) they could. But thats different then doing the bare minimum to, for example, get in the cvs extras group.
That said, that doesn't mean we should ignore potential risk from blindly trusting third party systems. I would recommend that, at a minimum, third party systems should run with selinux on and enforcing. This would afford some additional assurance of the system's security, but won't solve all problems.
But how would this protect us in any way if the users of that system had root on it? Since all of these 3rd part systems aren't officially monitored by us we have to assume the worst on each of these hosts.
Now, I've never actually done this. It's just my understanding that it'd work that way. If you had root on a box and I sshed there with my ssh key, would you not have access to take the key and log in to other boxes as me?
I'm not a crypto guy by any stretch. But, my understanding is that no, that's not how it public key authentication should work. As I understand it, your private key is never sent across the wire and therefore even with root on the remote system, an attacker could only ever has access to your public key. This is assuming that your private key isn't also in your home directory on the remote system (i.e. the only local file they have is the authorized_keys). Please double-check RFC 4252 for more details.
So my question is, is this a real risk or is there a precaution in SSH preventing the attack i'm describing (basically a man in the middle type attack)
I think the mitm risk is fairly low. I think there are other risks, such as not having direct control over keeping third party systems up to date, or allowing third party systems running non-Fedora or non-RHEL distros (e.g. the latest Debian openssh debacle). Privilege escalation on the remote system is also a concern. Thankfully, selinux can mitigate the last fairly effectively.
I definitely don't think this should be done lightly.
You think mitm is fairly low but is it really? Lets say, for example, you forward your ssh agent to this remote host. What are the implications there?
-Mike
On Thu May 22 2008, Mike McGrath wrote:
You think mitm is fairly low but is it really? Lets say, for example, you forward your ssh agent to this remote host. What are the implications there?
When someone forwards the ssh agent to a machine, the root user of this machine can access it and use it to authenticate to other machines. Afaik, the only way to prevent this is to use "ssh-add -c" when adding the keys to the agent which makes the agent ask the user for permission everytime the key should be used for authentication. But this is a problem that exists even when the FAS is not used by third parties, because an user can still forward his ssh-agent.
Regards, Till
On Thu, May 22, 2008 at 6:31 AM, Mike McGrath mmcgrath@redhat.com wrote:
On Wed, 21 May 2008, brett lentz wrote:
On Wed, May 21, 2008 at 9:06 PM, Mike McGrath mmcgrath@redhat.com wrote: That said, that doesn't mean we should ignore potential risk from blindly trusting third party systems. I would recommend that, at a minimum, third party systems should run with selinux on and enforcing. This would afford some additional assurance of the system's security, but won't solve all problems.
But how would this protect us in any way if the users of that system had root on it? Since all of these 3rd part systems aren't officially monitored by us we have to assume the worst on each of these hosts.
That's why I said it won't solve all problems. The scenarios where selinux helps are when root is not yet compromised, it helps reduce the risk of things like privilege escalation and exploiting holes in a public facing application to gain shell access.
Unfortunately, the current selinux policies will not stop someone accessing an existing shell account and sudoing their way to root.
This is where I think we need to work up some requirements for third party systems. We need some level of assurance that the admins of that system have some sane security policies and are maintaining them. It shouldn't be difficult to work up a script or set of scripts to periodically audit the third party systems to assure that these policies are adhered to.
IMO, a good starting point for those requirements would be:
1. system runs Fedora/RHEL 2. system has selinux enabled and enforcing. 3. system uses an acceptable update schedule. 4. system's admins are known, and willing to be available when we need to contact them (within a reasonable set of hours) 5. the system's admins document their policy for providing root access to their system. this allows us to do some risk analysis. 6. we should be able to quickly and easily revoke the system's access to Fedora.
Now, I've never actually done this. It's just my understanding that it'd work that way. If you had root on a box and I sshed there with my ssh key, would you not have access to take the key and log in to other boxes as me?
I'm not a crypto guy by any stretch. But, my understanding is that no, that's not how it public key authentication should work. As I understand it, your private key is never sent across the wire and therefore even with root on the remote system, an attacker could only ever has access to your public key. This is assuming that your private key isn't also in your home directory on the remote system (i.e. the only local file they have is the authorized_keys). Please double-check RFC 4252 for more details.
So my question is, is this a real risk or is there a precaution in SSH preventing the attack i'm describing (basically a man in the middle type attack)
I think the mitm risk is fairly low. I think there are other risks, such as not having direct control over keeping third party systems up to date, or allowing third party systems running non-Fedora or non-RHEL distros (e.g. the latest Debian openssh debacle). Privilege escalation on the remote system is also a concern. Thankfully, selinux can mitigate the last fairly effectively.
I definitely don't think this should be done lightly.
You think mitm is fairly low but is it really? Lets say, for example, you forward your ssh agent to this remote host. What are the implications there?
The implications for ssh-agent is fairly simple. Your private key still never touches the wire or the remote systems. SSH-Agent forwards the auth challenges to the local system you're logging in from.
Here's a great diagram of the process: http://www.unixwiz.net/techtips/ssh-agent-forwarding.html#fwd
-Mike
---Brett.
On Thu, 22 May 2008, brett lentz wrote:
IMO, a good starting point for those requirements would be:
- system runs Fedora/RHEL
- system has selinux enabled and enforcing.
- system uses an acceptable update schedule.
- system's admins are known, and willing to be available when we need
to contact them (within a reasonable set of hours) 5. the system's admins document their policy for providing root access to their system. this allows us to do some risk analysis. 6. we should be able to quickly and easily revoke the system's access to Fedora.
Thats the problem though, there's no way for us to enforce that in any way without regularly checking in, etc. What if they're not compliant and for how long? I think this policy should be simple or non-existant at all. If we can't reliably say that ssh-key based auth to remote machines is a no-risk operation for us, then we shouldn't do it.
The implications for ssh-agent is fairly simple. Your private key still never touches the wire or the remote systems. SSH-Agent forwards the auth challenges to the local system you're logging in from.
Here's a great diagram of the process: http://www.unixwiz.net/techtips/ssh-agent-forwarding.html#fwd
I know your private key doesn't touch the wire or remote system. But the agent creates a socket in /tmp/ssh-* and I'm worried someone with access to that socket could auth to other machines as the user.
-Mike
On Thu, May 22, 2008 at 8:19 AM, Mike McGrath mmcgrath@redhat.com wrote:
On Thu, 22 May 2008, brett lentz wrote:
The implications for ssh-agent is fairly simple. Your private key still never touches the wire or the remote systems. SSH-Agent forwards the auth challenges to the local system you're logging in from.
Here's a great diagram of the process: http://www.unixwiz.net/techtips/ssh-agent-forwarding.html#fwd
I know your private key doesn't touch the wire or remote system. But the agent creates a socket in /tmp/ssh-* and I'm worried someone with access to that socket could auth to other machines as the user.
Yes, that's a well-known risk. The only protections on that socket are filesystem-level permissions, which root can obviously bypass.
The only mechanism I'm aware of that could revoke root's ability to access that file is selinux. However, current policy still allows root to do whatever he likes.
I don't think it's possible to dictate a "don't use ssh-agent" policy. That seems unenforceable.
This is where I come back to the assurance bit. As you said, that can get a bit complicated and hard to manage on systems that aren't ours.
-Mike
---Brett.
On Thu, 2008-05-22 at 08:41 -0700, brett lentz wrote:
On Thu, May 22, 2008 at 8:19 AM, Mike McGrath mmcgrath@redhat.com wrote:
On Thu, 22 May 2008, brett lentz wrote:
The implications for ssh-agent is fairly simple. Your private key still never touches the wire or the remote systems. SSH-Agent forwards the auth challenges to the local system you're logging in from.
Here's a great diagram of the process: http://www.unixwiz.net/techtips/ssh-agent-forwarding.html#fwd
I know your private key doesn't touch the wire or remote system. But the agent creates a socket in /tmp/ssh-* and I'm worried someone with access to that socket could auth to other machines as the user.
Yes, that's a well-known risk. The only protections on that socket are filesystem-level permissions, which root can obviously bypass.
And the risk isn't increased by us allowing third-party groups to do auth via FAS. This risk is present whenever any user logs in to another machine with agent forwarding. Which is requested by the user/client -- not the machine being logged into
Jeremy
On Thu, 22 May 2008, Jeremy Katz wrote:
On Thu, 2008-05-22 at 08:41 -0700, brett lentz wrote:
On Thu, May 22, 2008 at 8:19 AM, Mike McGrath mmcgrath@redhat.com wrote:
On Thu, 22 May 2008, brett lentz wrote:
The implications for ssh-agent is fairly simple. Your private key still never touches the wire or the remote systems. SSH-Agent forwards the auth challenges to the local system you're logging in from.
Here's a great diagram of the process: http://www.unixwiz.net/techtips/ssh-agent-forwarding.html#fwd
I know your private key doesn't touch the wire or remote system. But the agent creates a socket in /tmp/ssh-* and I'm worried someone with access to that socket could auth to other machines as the user.
Yes, that's a well-known risk. The only protections on that socket are filesystem-level permissions, which root can obviously bypass.
And the risk isn't increased by us allowing third-party groups to do auth via FAS. This risk is present whenever any user logs in to another machine with agent forwarding. Which is requested by the user/client -- not the machine being logged into
The risk does increase as far as targeting goes though. If you were to do this type of attack right now, how would you go about doing it and what machines would you use? If we start allowing third party machines that have basically no barrier to entry it becomes much easier to plan and execute the attack.
-Mike
On Thu May 22 2008, Mike McGrath wrote:
On Thu, 22 May 2008, Jeremy Katz wrote:
And the risk isn't increased by us allowing third-party groups to do auth via FAS. This risk is present whenever any user logs in to another machine with agent forwarding. Which is requested by the user/client -- not the machine being logged into
The risk does increase as far as targeting goes though. If you were to do this type of attack right now, how would you go about doing it and what machines would you use? If we start allowing third party machines that have basically no barrier to entry it becomes much easier to plan and execute the attack.
One can still provide services to Fedora maintainers without using FAS, e.g. a ppc machine that can be used by maintainers to debug their package on that arch. Then the maintainers would send their ssh public key by themself to the administrator of the machine.
Regards, Till
On Thu, May 22, 2008 at 10:19:54AM -0500, Mike McGrath wrote:
I know your private key doesn't touch the wire or remote system. But the agent creates a socket in /tmp/ssh-* and I'm worried someone with access to that socket could auth to other machines as the user.
The agent *isn't* forwarded by default, you need to use either -A in the command line or ForwardAgent yes in the config. Of course nothing stops users from enabling agent forwarding by default but then again nothings stops them from doinf other stupid things with ssh keys, having passwordless keys and keeping a copy of them in some insecure location is one of the worst examples.
For the people having agent forwarding enabled by default you already have a problem with all the other machines that they connect anyway for their daily work/whatever.
Running a kerberos server is a good alternative to ssh pubkey auth, you can enforce centrally non forwardable tickets if you want so you can be sure that other machines that the user authenticates with cannot connect back to the fedora servers.
Kostas Georgiou
On Thu May 22 2008, Mike McGrath wrote:
Now, I've never actually done this. It's just my understanding that it'd work that way. If you had root on a box and I sshed there with my ssh key, would you not have access to take the key and log in to other boxes as me?
So my question is, is this a real risk or is there a precaution in SSH preventing the attack i'm describing (basically a man in the middle type attack)
Afaik this attack is not possible with ssh because a user signs some information that is unique to the current session and contains among other things a hash of the host key that the user wants to login to.
Regards, Till
On Thu, 22 May 2008, Till Maas wrote:
On Thu May 22 2008, Mike McGrath wrote:
Now, I've never actually done this. It's just my understanding that it'd work that way. If you had root on a box and I sshed there with my ssh key, would you not have access to take the key and log in to other boxes as me?
So my question is, is this a real risk or is there a precaution in SSH preventing the attack i'm describing (basically a man in the middle type attack)
Afaik this attack is not possible with ssh because a user signs some information that is unique to the current session and contains among other things a hash of the host key that the user wants to login to.
Yeah, I've been talking with someone in #openssh as well. My understanding of how openssh auth is happens basically like this (sorry I'm not going to take the time to diagram this even though thats what it really needs)
Client tries to ssh to Server A
Server A generates a random number, encrypts it with pub, sends it to the client
The client decrypts this number with private key and sends it back to A.
Bam! Shell.
Here was my concern.
Client tries to ssh to untrusted Server B (with hacked ssh server)
Server B tries to ssh to Server A as the client user.
Server A generates random number, encrypts it with pub, sends it to B.
Server B doesn't know what the number is so it sends that encrypted payload to the client.
The client decrypts this number with private key and sends it back to Server B.
Server B sends this number back to Server A.
Server A allows shell to server B as client user.
Server B allows shell to client user.
client doesn't realize what just happened as it went very quick.
The guys in #openssh are saying this isn't possible but I wasn't convinced with their reason (basically that server B doesn't have server A's host keys). Can someone else explain why the above isn't possible?
-Mike
On Thu May 22 2008, Mike McGrath wrote:
Client tries to ssh to Server A
Server A generates a random number, encrypts it with pub, sends it to the client
The client decrypts this number with private key and sends it back to A.
Bam! Shell.
The public key authentication does not work this way.
The guys in #openssh are saying this isn't possible but I wasn't convinced with their reason (basically that server B doesn't have server A's host keys). Can someone else explain why the above isn't possible?
To authenticate, the client needs to sign a session identifier (and some other information) with his private key and send the signature to the server. The session identifier is a hash of several data that includes the host key.
Regards, Till
On Thu, 22 May 2008, Till Maas wrote:
On Thu May 22 2008, Mike McGrath wrote:
Client tries to ssh to Server A
Server A generates a random number, encrypts it with pub, sends it to the client
The client decrypts this number with private key and sends it back to A.
Bam! Shell.
The public key authentication does not work this way.
The guys in #openssh are saying this isn't possible but I wasn't convinced with their reason (basically that server B doesn't have server A's host keys). Can someone else explain why the above isn't possible?
To authenticate, the client needs to sign a session identifier (and some other information) with his private key and send the signature to the server. The session identifier is a hash of several data that includes the host key.
So what you're saying is it is impossible to do a man in the middle attack with OpenSSH (assuming the host keys of the server haven't been compromised) ?
-Mike
On Thu May 22 2008, Mike McGrath wrote:
So what you're saying is it is impossible to do a man in the middle attack with OpenSSH (assuming the host keys of the server haven't been compromised) ?
I am saying that the information an attackers gets when a user logs in with public-key authentication to a server under the attackers control is not enough to allow the attacker to login into another machine the user can login to.
Regards, Till
On Thu, 22 May 2008, Till Maas wrote:
On Thu May 22 2008, Mike McGrath wrote:
Client tries to ssh to Server A
Server A generates a random number, encrypts it with pub, sends it to the client
The client decrypts this number with private key and sends it back to A.
Bam! Shell.
The public key authentication does not work this way.
Side note about this for my own education. Can you fill in the blanks? Because I know what is there is accurate, just not complete.
-Mike
On Thu May 22 2008, Mike McGrath wrote:
On Thu, 22 May 2008, Till Maas wrote:
On Thu May 22 2008, Mike McGrath wrote:
Client tries to ssh to Server A
Server A generates a random number, encrypts it with pub, sends it to the client
The client decrypts this number with private key and sends it back to A.
Bam! Shell.
The public key authentication does not work this way.
Side note about this for my own education. Can you fill in the blanks? Because I know what is there is accurate, just not complete.
I do not understand what you ask me to do. Do you want me to explain the ssh public-key authentication? I already explained in very short, if you want more detail, you better look into the rfcs, because I would basically copy it to add more detail:
1st phase: create a session encryption key and a uniqe session identifier (hash H in the rfc): http://www.ietf.org/rfc/rfc4253.txt page 22 lists all the information that the hash is computed of which includes the host key
2nd phase: authentication: The clients signs the hash H from above and some other information like the user name and sends it to the server, a full list of the signed information can be found on page 9 of rfc 4252: http://www.ietf.org/rfc/rfc4252.txt
Regards, Till
Till Maas wrote:
On Thu May 22 2008, Mike McGrath wrote:
On Thu, 22 May 2008, Till Maas wrote:
On Thu May 22 2008, Mike McGrath wrote:
Client tries to ssh to Server A
Server A generates a random number, encrypts it with pub, sends it to the client
The client decrypts this number with private key and sends it back to A.
Bam! Shell.
The public key authentication does not work this way.
Side note about this for my own education. Can you fill in the blanks? Because I know what is there is accurate, just not complete.
I do not understand what you ask me to do. Do you want me to explain the ssh public-key authentication? I already explained in very short, if you want more detail, you better look into the rfcs, because I would basically copy it to add more detail:
1st phase: create a session encryption key and a uniqe session identifier (hash H in the rfc): http://www.ietf.org/rfc/rfc4253.txt page 22 lists all the information that the hash is computed of which includes the host key
2nd phase: authentication: The clients signs the hash H from above and some other information like the user name and sends it to the server, a full list of the signed information can be found on page 9 of rfc 4252: http://www.ietf.org/rfc/rfc4252.txt
Note: I have not read the rfc's or openssh source code/docs.
It seems like this would be open to attack in the special case where the user has never logged into 1) The server they think they're connecting to 2) The machine the malicious server is actually trying to authenticate them against. In this scenario the client doesn't have host keys for either of the remote machines so it's unable to verify that the malicious server is lying to it.
I wouldn't think that's a huge issue, just noting that it exists.
-Toshio
On Thu May 22 2008, Toshio Kuratomi wrote:
It seems like this would be open to attack in the special case where the user has never logged into 1) The server they think they're connecting to 2) The machine the malicious server is actually trying to authenticate them against. In this scenario the client doesn't have host keys for either of the remote machines so it's unable to verify that the malicious server is lying to it.
This is also not possible with public key authentication, because the server needs to create a signature with the host key when the session encryption key is generated. In case the attacker forwards the network traffic in this phase to the other server, he will not be able to decrypt the authentication phase. If he uses its own host key, then the signature used for authentication will not be accepted by the other server.
Regards, Till
infrastructure@lists.fedoraproject.org
-
brett lentz -
Jeremy Katz -
Kostas Georgiou -
Mike McGrath -
Till Maas -
Toshio Kuratomi