[SSSD] Patch to fix LDAP ID backend GSSAPI credential expired messages

Simo Sorce ssorce at redhat.com
Mon Mar 29 13:19:34 UTC 2010


On Mon, 29 Mar 2010 12:42:34 +0400
Eugene Indenbom <eindenbom at gmail.com> wrote:

> Dear SSSD developers,
> 
> I have started to create this patch in order to address the
> reclamation filed on Fedora 12:
> https://bugzilla.redhat.com/show_bug.cgi?id=575187
> 
> The problem shows up as continuous flood of messages to event log (2
> every 10 minutes) about expired Kerberos context.
> Mar 28 06:39:17 node-2 sssd_be: GSSAPI Error: The referenced context
> has expired (Unknown error)
> Mar 28 06:39:17 node-2 sssd_be: GSSAPI Error: The referenced context
> has expired (Unknown error)
> 
> The messages show were caused by expired Kerberos context. The
> assumption made in ldap_child.c that stated that Kerberos ticket is
> only needed during connection setup, so it forcibly set the Kerberos
> ticket lifetime to 5 minutes. The assumption is unfortunately wrong
> as with MIT Kerberos implementation Kerberos encryption and integrity
> is always enforced when supported by both server and client. So after
> 5 minutes the connection Kerberos ticket expired and connection was
> broken polluting event log and putting additional load on DS and KDC
> due to constant reconnects.
> 
> Please, note that Kerberos authentication with LDAP server is the
> default for IPA domain.
> 
> The natural solution for the problem was to increase Kerberos ticket
> lifetime. Farther investigations show that it is necessary also to
> close connection in timely fashion, as even closing connection with
> expired ticket caused message in event log to appear.
> 
> During implementation of the above fixes I have also identified
> several more problems:
> 1. In case of 2 or more requests executed in parallel (all started
> before connection was established) it was possible to begin 2
> connection operations in parallel. One of the established connections
> was finally leaked leaving connection to hang around.
> 2. The were no failover retry login LDAP ID backend itself. More over
> after the first server failure (even on stale cached connection) the
> whole backend was put to OFFLINE.
> 
> The patch attached to this message addresses all the above issues and
> as well as:
> 1. Reduces amount of duplicate code related to LDAP connection and
> retry logic.
> 2. Puts the connection and retry logic on the same architectural
> level in ldap_id.c (now all this logic is handled by sdap_account_
> function family), that should benefit readability of code.
> 
> The patch design is as follows:
> 
> 1. New entity sdap_id_op represents high level LDAP query operation.
> It is responsible of:
>    a) Keeping track of LDAP connection
>    b) Keeping track of number of reconnect retries
> 2. New entity sdap_id_connection represents connection attempt made
> by LDAP ID backend and later tracks connection usage.
> 3. sdap_id_ctx keeps track of all currently open connections and
> ensures: a) That there is only one connection attempt in progress at
> a time. Further operations are queued until connection is completed.
>    b) That cached connection is released in timely fashion
>    c) That all connections are closed when all operations using them
> are complete and they are released from cache.
> 4. The retry logic of all LDAP operations is updated to use new
> facility.
> 
> I have attached to versions of this patch:
> - one against sssd-1.1.0. This version was thoroughly tested and
> currently is in preproduction testing on several systems in my IPA
> domain.
> - the other against git repository. I have only made sure that the
> patch compiled.
> 
> Looking forward for your reply, Eugene Indenbom

Hi Eugene,
it is great to see this kind of contribution but I have to admit the
patch is huge and hard to digest.

When solving multiple problems we tend to prefer multiple separate
patches where each one addresses a different issue. Is there any chance
you can split it in patches along your points above ?

Also a few comments in general.
The goal of SSSD is to never use more than one connection at a time for
account information. So your patch is kind of changing our fundamental
goal by allowing multiple connections. We need to carefully evaluate
that part.

The authentication case is separate, and it doesn't need to be
completely serialized, we can definitely open multiple authentication
connections at the same time. I guess the need to serialize the
connections is connected with acquiring the kerberos credentials.
In that case I would just separate and serialize that part.

I see you started passing around sdap_id_op. The memory hierarchy
around sdap_is_op is very delicate and required a lot of very careful
handling to avoid having it disappear under our feet at the wrong time.
It is meant to represent a single ldap operation tied to a specific ldap
context, any changes to its use should be in a separate patch that I
want to review carefully. But ideally sdap_id_op is opaque to most of
the code and is internal to the processing of replies from the openldap
libraries. It should never be used out of this context.

Let me know if you have any questions, I love to see and
help contributions.
Simo.

-- 
Simo Sorce * Red Hat, Inc * New York



More information about the sssd-devel mailing list