We run a cluster of directory servers (4 masters, 2 hubs, 14 slaves) behind a set of F5 Bigip load balancers. Our Bigip admins recently decided to switch the boxes to "one-armed" mode and that services would have to use X-Forwarded-For headers or equivalent to get the actual client IP address. Obviously, LDAP has no equivalent.
So, I've hacked something together and I'm posting it looking for feedback. If the stuff is actually usable for other folks, that's good too.
On the load balancer side, the code is specifically for the F5 Bigips. But if other load balancers have similar abilities to trigger on events and can insert binary data into the datastream, it should be adaptable.
Essentially what I've done is defined a new LDAP Extended Operation with a payload that's a string containing the source IP address of the incoming connection. The load balancer sends this LDAP operation as soon as it opens a connection to the LDAP server, before any other traffic gets sent. On the directory server side, I've written an Extended Operation plugin that then logs the string. All we need is logging, so that's good enough for us. There's room for improvement there though, like making the address available for IP based ACls (which we don't use).
In order to insert an LDAP operation into the stream, the code on the load balancer needs to choose an LDAP message-id that hopefully the real client isn't going to use. Going on the assumption that the client will start at 0 and increment up, I had the code insert a message-id of 0x70000000. I initially thought I'd have to have the code on the load balancer look for the response message and throw it away so the client doesn't see it, but I've found that the clients just seem to ignore it with no ill effects, so I haven't bothered filtering the response out. The less the code on the load balancer does, the better.
There's a possibility that the client could send it's own xff extended operation, but since the load balancer always sends first, we can just ignore any subsequent log entries.
On the directory server plugin side, I needed to be able to log this to the access log, not to the error log. The slapi_log_access function isn't declared in the plugin header file, so I had to declare it manually.
Here's what our log entries look like now, for both 636 (SSL) and 389 (cleartext before starttls):
[17/May/2016:15:34:22 -0400] conn=230843 fd=156 slot=156 SSL connection from 130.207.167.12 to 130.207.183.16 [17/May/2016:15:34:22 -0400] conn=230843 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op" [17/May/2016:15:34:22 -0400] conn=230843 op=0 forwarded for 130.207.167.12 [17/May/2016:15:34:22 -0400] conn=230843 op=0 RESULT err=0 tag=120 nentries=0 etime=0
[17/May/2016:15:34:22 -0400] conn=230844 fd=156 slot=156 connection from 130.207.167.12 to 130.207.183.16 [17/May/2016:15:34:22 -0400] conn=230844 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op" [17/May/2016:15:34:22 -0400] conn=230844 op=0 forwarded for 130.207.167.12 [17/May/2016:15:34:22 -0400] conn=230844 op=0 RESULT err=0 tag=120 nentries=0 etime=0
We haven't switched our Bigips yet, so the "connection from" line still shows the actual client IP address.
F5 Bigip code fragments are called "irules" and are written in TCL. The tar file below contains two different irule files, one for cleartext streams and one for SSL streams. By SSL streams, I mean where SSL connections from the client are terminated at the load balancer and then re-SSLed to the ldap server. Sorry, I'm not writing the other kind.
On 05/17/2016 02:04 PM, Robert Viduya wrote:
We run a cluster of directory servers (4 masters, 2 hubs, 14 slaves) behind a set of F5 Bigip load balancers. Our Bigip admins recently decided to switch the boxes to "one-armed" mode and that services would have to use X-Forwarded-For headers or equivalent to get the actual client IP address. Obviously, LDAP has no equivalent.
So, I've hacked something together and I'm posting it looking for feedback. If the stuff is actually usable for other folks, that's good too.
On the load balancer side, the code is specifically for the F5 Bigips. But if other load balancers have similar abilities to trigger on events and can insert binary data into the datastream, it should be adaptable.
Essentially what I've done is defined a new LDAP Extended Operation with a payload that's a string containing the source IP address of the incoming connection. The load balancer sends this LDAP operation as soon as it opens a connection to the LDAP server, before any other traffic gets sent. On the directory server side, I've written an Extended Operation plugin that then logs the string. All we need is logging, so that's good enough for us. There's room for improvement there though, like making the address available for IP based ACls (which we don't use).
In order to insert an LDAP operation into the stream, the code on the load balancer needs to choose an LDAP message-id that hopefully the real client isn't going to use. Going on the assumption that the client will start at 0 and increment up, I had the code insert a message-id of 0x70000000. I initially thought I'd have to have the code on the load balancer look for the response message and throw it away so the client doesn't see it, but I've found that the clients just seem to ignore it with no ill effects, so I haven't bothered filtering the response out. The less the code on the load balancer does, the better.
There's a possibility that the client could send it's own xff extended operation, but since the load balancer always sends first, we can just ignore any subsequent log entries.
On the directory server plugin side, I needed to be able to log this to the access log, not to the error log. The slapi_log_access function isn't declared in the plugin header file, so I had to declare it manually.
Here's what our log entries look like now, for both 636 (SSL) and 389 (cleartext before starttls):
[17/May/2016:15:34:22 -0400] conn=230843 fd=156 slot=156 SSL connection from 130.207.167.12 to 130.207.183.16 [17/May/2016:15:34:22 -0400] conn=230843 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op" [17/May/2016:15:34:22 -0400] conn=230843 op=0 forwarded for 130.207.167.12 [17/May/2016:15:34:22 -0400] conn=230843 op=0 RESULT err=0 tag=120 nentries=0 etime=0 [17/May/2016:15:34:22 -0400] conn=230844 fd=156 slot=156 connection from 130.207.167.12 to 130.207.183.16 [17/May/2016:15:34:22 -0400] conn=230844 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op" [17/May/2016:15:34:22 -0400] conn=230844 op=0 forwarded for 130.207.167.12 [17/May/2016:15:34:22 -0400] conn=230844 op=0 RESULT err=0 tag=120 nentries=0 etime=0We haven't switched our Bigips yet, so the "connection from" line still shows the actual client IP address.
F5 Bigip code fragments are called "irules" and are written in TCL. The tar file below contains two different irule files, one for cleartext streams and one for SSL streams. By SSL streams, I mean where SSL connections from the client are terminated at the load balancer and then re-SSLed to the ldap server. Sorry, I'm not writing the other kind.
This is really nice, and a big help for tracing connections through load balancers and other similar devices. Are you interested in contributing this to 389?
-- 389-users mailing list 389-users@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
I have no issues with that. I’m a little concerned about the ldap message id, however.
On May 18, 2016, at 9:02 AM, Rich Megginson rmeggins@redhat.com wrote:
On 05/17/2016 02:04 PM, Robert Viduya wrote:
We run a cluster of directory servers (4 masters, 2 hubs, 14 slaves) behind a set of F5 Bigip load balancers. Our Bigip admins recently decided to switch the boxes to "one-armed" mode and that services would have to use X-Forwarded-For headers or equivalent to get the actual client IP address. Obviously, LDAP has no equivalent.
So, I've hacked something together and I'm posting it looking for feedback. If the stuff is actually usable for other folks, that's good too.
On the load balancer side, the code is specifically for the F5 Bigips. But if other load balancers have similar abilities to trigger on events and can insert binary data into the datastream, it should be adaptable.
Essentially what I've done is defined a new LDAP Extended Operation with a payload that's a string containing the source IP address of the incoming connection. The load balancer sends this LDAP operation as soon as it opens a connection to the LDAP server, before any other traffic gets sent. On the directory server side, I've written an Extended Operation plugin that then logs the string. All we need is logging, so that's good enough for us. There's room for improvement there though, like making the address available for IP based ACls (which we don't use).
In order to insert an LDAP operation into the stream, the code on the load balancer needs to choose an LDAP message-id that hopefully the real client isn't going to use. Going on the assumption that the client will start at 0 and increment up, I had the code insert a message-id of 0x70000000. I initially thought I'd have to have the code on the load balancer look for the response message and throw it away so the client doesn't see it, but I've found that the clients just seem to ignore it with no ill effects, so I haven't bothered filtering the response out. The less the code on the load balancer does, the better.
There's a possibility that the client could send it's own xff extended operation, but since the load balancer always sends first, we can just ignore any subsequent log entries.
On the directory server plugin side, I needed to be able to log this to the access log, not to the error log. The slapi_log_access function isn't declared in the plugin header file, so I had to declare it manually.
Here's what our log entries look like now, for both 636 (SSL) and 389 (cleartext before starttls):
[17/May/2016:15:34:22 -0400] conn=230843 fd=156 slot=156 SSL connection from 130.207.167.12 to 130.207.183.16 [17/May/2016:15:34:22 -0400] conn=230843 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op" [17/May/2016:15:34:22 -0400] conn=230843 op=0 forwarded for 130.207.167.12 [17/May/2016:15:34:22 -0400] conn=230843 op=0 RESULT err=0 tag=120 nentries=0 etime=0
[17/May/2016:15:34:22 -0400] conn=230844 fd=156 slot=156 connection from 130.207.167.12 to 130.207.183.16 [17/May/2016:15:34:22 -0400] conn=230844 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op" [17/May/2016:15:34:22 -0400] conn=230844 op=0 forwarded for 130.207.167.12 [17/May/2016:15:34:22 -0400] conn=230844 op=0 RESULT err=0 tag=120 nentries=0 etime=0
We haven't switched our Bigips yet, so the "connection from" line still shows the actual client IP address.
F5 Bigip code fragments are called "irules" and are written in TCL. The tar file below contains two different irule files, one for cleartext streams and one for SSL streams. By SSL streams, I mean where SSL connections from the client are terminated at the load balancer and then re-SSLed to the ldap server. Sorry, I'm not writing the other kind.
This is really nice, and a big help for tracing connections through load balancers and other similar devices. Are you interested in contributing this to 389?
-- 389-users mailing list 389-users@lists.fedoraproject.org mailto:389-users@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
-- 389-users mailing list 389-users@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
On 05/18/2016 07:35 AM, Robert Viduya wrote:
I have no issues with that.
Can you open a 389 ticket? https://fedorahosted.org/389/newticket
I’m a little concerned about the ldap message id, however.
Me too. Developers, LDAP SMEs, please feel free to chime in.
On May 18, 2016, at 9:02 AM, Rich Megginson <rmeggins@redhat.com mailto:rmeggins@redhat.com> wrote:
On 05/17/2016 02:04 PM, Robert Viduya wrote:
We run a cluster of directory servers (4 masters, 2 hubs, 14 slaves) behind a set of F5 Bigip load balancers. Our Bigip admins recently decided to switch the boxes to "one-armed" mode and that services would have to use X-Forwarded-For headers or equivalent to get the actual client IP address. Obviously, LDAP has no equivalent.
So, I've hacked something together and I'm posting it looking for feedback. If the stuff is actually usable for other folks, that's good too.
On the load balancer side, the code is specifically for the F5 Bigips. But if other load balancers have similar abilities to trigger on events and can insert binary data into the datastream, it should be adaptable.
Essentially what I've done is defined a new LDAP Extended Operation with a payload that's a string containing the source IP address of the incoming connection. The load balancer sends this LDAP operation as soon as it opens a connection to the LDAP server, before any other traffic gets sent. On the directory server side, I've written an Extended Operation plugin that then logs the string. All we need is logging, so that's good enough for us. There's room for improvement there though, like making the address available for IP based ACls (which we don't use).
In order to insert an LDAP operation into the stream, the code on the load balancer needs to choose an LDAP message-id that hopefully the real client isn't going to use. Going on the assumption that the client will start at 0 and increment up, I had the code insert a message-id of 0x70000000. I initially thought I'd have to have the code on the load balancer look for the response message and throw it away so the client doesn't see it, but I've found that the clients just seem to ignore it with no ill effects, so I haven't bothered filtering the response out. The less the code on the load balancer does, the better.
There's a possibility that the client could send it's own xff extended operation, but since the load balancer always sends first, we can just ignore any subsequent log entries.
On the directory server plugin side, I needed to be able to log this to the access log, not to the error log. The slapi_log_access function isn't declared in the plugin header file, so I had to declare it manually.
Here's what our log entries look like now, for both 636 (SSL) and 389 (cleartext before starttls):
[17/May/2016:15:34:22 -0400] conn=230843 fd=156 slot=156 SSL connection from 130.207.167.12 to 130.207.183.16 [17/May/2016:15:34:22 -0400] conn=230843 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op" [17/May/2016:15:34:22 -0400] conn=230843 op=0 forwarded for 130.207.167.12 [17/May/2016:15:34:22 -0400] conn=230843 op=0 RESULT err=0 tag=120 nentries=0 etime=0 [17/May/2016:15:34:22 -0400] conn=230844 fd=156 slot=156 connection from 130.207.167.12 to 130.207.183.16 [17/May/2016:15:34:22 -0400] conn=230844 op=0 EXT oid="1.3.6.1.4.1.636.2.11.11.1" name="forwarded-for extended op" [17/May/2016:15:34:22 -0400] conn=230844 op=0 forwarded for 130.207.167.12 [17/May/2016:15:34:22 -0400] conn=230844 op=0 RESULT err=0 tag=120 nentries=0 etime=0We haven't switched our Bigips yet, so the "connection from" line still shows the actual client IP address.
F5 Bigip code fragments are called "irules" and are written in TCL. The tar file below contains two different irule files, one for cleartext streams and one for SSL streams. By SSL streams, I mean where SSL connections from the client are terminated at the load balancer and then re-SSLed to the ldap server. Sorry, I'm not writing the other kind.
This is really nice, and a big help for tracing connections through load balancers and other similar devices. Are you interested in contributing this to 389?
-- 389-users mailing list 389-users@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
-- 389-users mailing list 389-users@lists.fedoraproject.org mailto:389-users@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
-- 389-users mailing list 389-users@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
Hi Rich, hi Robert,
On 18/05/16 15:48, Rich Megginson wrote:
On 05/18/2016 07:35 AM, Robert Viduya wrote:
I have no issues with that.
Can you open a 389 ticket? https://fedorahosted.org/389/newticket
I’m a little concerned about the ldap message id, however.
Me too. Developers, LDAP SMEs, please feel free to chime in.
this is quite an interesting approach and it will probably work in the vast majority of cases. But instead of adding a separate LDAP _operation_ to the stream (where we cannot easily ensure uniqueness of the message id) we could implement the "LDAP Session Tracking Control" from http://tools.ietf.org/html/draft-wahl-ldap-session and only insert that _control_ into the first LDAP operation. There is already a 389 ticket for the server side functionality (https://fedorahosted.org/389/ticket/47873). On the F5 there is ASN1::element which makes construction of the control and adding it to the LDAP operation quite feasible.
Regards, J.
On Wed, 2016-05-18 at 23:04 +0200, Jochen Schneider wrote:
Hi Rich, hi Robert,
On 18/05/16 15:48, Rich Megginson wrote:
On 05/18/2016 07:35 AM, Robert Viduya wrote:
I have no issues with that.
Can you open a 389 ticket? https://fedorahosted.org/389/newticket
I’m a little concerned about the ldap message id, however.
Me too. Developers, LDAP SMEs, please feel free to chime in.
this is quite an interesting approach and it will probably work in the vast majority of cases.
I think it sounds a bit fragile, and really relies on the log aggregation techniques of the implementor. I'm not sure I like this approach.
As well, how is this locked down to prevent a fradulent client sending a fake control first, then their real operation? It would certainly cause confusion in the logs ....
But instead of adding a separate LDAP _operation_ to the stream (where we cannot easily ensure uniqueness of the message id) we could implement the "LDAP Session Tracking Control" from http://tools.ietf.org/html/draft-wahl-ldap-session and only insert that _control_ into the first LDAP operation. There is already a 389 ticket for the server side functionality (https://fedorahosted.org/389/ticket/47873). On the F5 there is ASN1::element which makes construction of the control and adding it to the LDAP operation quite feasible.
The issue with this is that this control can be forged. We would need to essentially add another layer of "auth" which states that we allow the control from only certain source ips, ie the load balancer that is allowed to add this control. And even IP layer checks can be forged unless you turn on things like reverse path checking and such on the host.
Both solutions really have shortcomings and complexity, but I think I would prefer to implement the draft version of session-03 over the "loosely coupled" approach that Robert developed. We could at least lock it down to a check on the plugin only allowed some set of IP's to use the control, and put a check in dsktune for rpf. Then the load balancer can inject the control to the operation as needed.
Saying this, there is no faster way to ruin your network than to allow a load balancer to tamper with your traffic. I've seen horrible, horrible things happen because of bigip, netscalers, ace, and others, tampered with the internals of DNS, LDAP, HTTP ... At the point you allow a load balancer to actively look at your traffic you are opening yourself to many subtle, hard to isolate issues.
I really can't recommend that you use one arm mode for LDAP, and highly recomend that you use routed mode, with TLS/SSL termination on the LDAP servers themself.
But that's a discussion for another thread I think .....
Regards, J. -- 389-users mailing list 389-users@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
My code on the load balancer inserts the XFF message before any client data gets passed. So forging can be handled by ignoring any subsequent XFF messages, only the first one is the one from the load balancer.
However, this doesn’t allow for cascading load balancers. HTML’s X-Forwarded-For header handles this by having each load balancer just append to the string. I’m not sure how important this capability is.
On May 18, 2016, at 7:48 PM, William Brown wibrown@redhat.com wrote:
On Wed, 2016-05-18 at 23:04 +0200, Jochen Schneider wrote:
Hi Rich, hi Robert,
On 18/05/16 15:48, Rich Megginson wrote:
On 05/18/2016 07:35 AM, Robert Viduya wrote:
I have no issues with that.
Can you open a 389 ticket? https://fedorahosted.org/389/newticket
I’m a little concerned about the ldap message id, however.
Me too. Developers, LDAP SMEs, please feel free to chime in.
this is quite an interesting approach and it will probably work in the vast majority of cases.
I think it sounds a bit fragile, and really relies on the log aggregation techniques of the implementor. I'm not sure I like this approach.
As well, how is this locked down to prevent a fradulent client sending a fake control first, then their real operation? It would certainly cause confusion in the logs ....
But instead of adding a separate LDAP _operation_ to the stream (where we cannot easily ensure uniqueness of the message id) we could implement the "LDAP Session Tracking Control" from http://tools.ietf.org/html/draft-wahl-ldap-session and only insert that _control_ into the first LDAP operation. There is already a 389 ticket for the server side functionality (https://fedorahosted.org/389/ticket/47873). On the F5 there is ASN1::element which makes construction of the control and adding it to the LDAP operation quite feasible.
The issue with this is that this control can be forged. We would need to essentially add another layer of "auth" which states that we allow the control from only certain source ips, ie the load balancer that is allowed to add this control. And even IP layer checks can be forged unless you turn on things like reverse path checking and such on the host.
Both solutions really have shortcomings and complexity, but I think I would prefer to implement the draft version of session-03 over the "loosely coupled" approach that Robert developed. We could at least lock it down to a check on the plugin only allowed some set of IP's to use the control, and put a check in dsktune for rpf. Then the load balancer can inject the control to the operation as needed.
Saying this, there is no faster way to ruin your network than to allow a load balancer to tamper with your traffic. I've seen horrible, horrible things happen because of bigip, netscalers, ace, and others, tampered with the internals of DNS, LDAP, HTTP ... At the point you allow a load balancer to actively look at your traffic you are opening yourself to many subtle, hard to isolate issues.
I really can't recommend that you use one arm mode for LDAP, and highly recomend that you use routed mode, with TLS/SSL termination on the LDAP servers themself.
But that's a discussion for another thread I think .....
Regards, J. -- 389-users mailing list 389-users@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
-- Sincerely,
William Brown Software Engineer Red Hat, Brisbane
-- 389-users mailing list 389-users@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
389-users@lists.fedoraproject.org
-
Jochen Schneider -
Rich Megginson -
Robert Viduya -
William Brown