On Mon, 29 Mar 2010 12:42:34 +0400 Eugene Indenbom eindenbom@gmail.com wrote:
Dear SSSD developers,
I have started to create this patch in order to address the reclamation filed on Fedora 12: https://bugzilla.redhat.com/show_bug.cgi?id=575187
The problem shows up as continuous flood of messages to event log (2 every 10 minutes) about expired Kerberos context. Mar 28 06:39:17 node-2 sssd_be: GSSAPI Error: The referenced context has expired (Unknown error) Mar 28 06:39:17 node-2 sssd_be: GSSAPI Error: The referenced context has expired (Unknown error)
The messages show were caused by expired Kerberos context. The assumption made in ldap_child.c that stated that Kerberos ticket is only needed during connection setup, so it forcibly set the Kerberos ticket lifetime to 5 minutes. The assumption is unfortunately wrong as with MIT Kerberos implementation Kerberos encryption and integrity is always enforced when supported by both server and client. So after 5 minutes the connection Kerberos ticket expired and connection was broken polluting event log and putting additional load on DS and KDC due to constant reconnects.
Please, note that Kerberos authentication with LDAP server is the default for IPA domain.
The natural solution for the problem was to increase Kerberos ticket lifetime. Farther investigations show that it is necessary also to close connection in timely fashion, as even closing connection with expired ticket caused message in event log to appear.
During implementation of the above fixes I have also identified several more problems:
- In case of 2 or more requests executed in parallel (all started
before connection was established) it was possible to begin 2 connection operations in parallel. One of the established connections was finally leaked leaving connection to hang around. 2. The were no failover retry login LDAP ID backend itself. More over after the first server failure (even on stale cached connection) the whole backend was put to OFFLINE.
The patch attached to this message addresses all the above issues and as well as:
- Reduces amount of duplicate code related to LDAP connection and
retry logic. 2. Puts the connection and retry logic on the same architectural level in ldap_id.c (now all this logic is handled by sdap_account_ function family), that should benefit readability of code.
The patch design is as follows:
- New entity sdap_id_op represents high level LDAP query operation.
It is responsible of: a) Keeping track of LDAP connection b) Keeping track of number of reconnect retries 2. New entity sdap_id_connection represents connection attempt made by LDAP ID backend and later tracks connection usage. 3. sdap_id_ctx keeps track of all currently open connections and ensures: a) That there is only one connection attempt in progress at a time. Further operations are queued until connection is completed. b) That cached connection is released in timely fashion c) That all connections are closed when all operations using them are complete and they are released from cache. 4. The retry logic of all LDAP operations is updated to use new facility.
I have attached to versions of this patch:
- one against sssd-1.1.0. This version was thoroughly tested and
currently is in preproduction testing on several systems in my IPA domain.
- the other against git repository. I have only made sure that the
patch compiled.
Looking forward for your reply, Eugene Indenbom
Hi Eugene, it is great to see this kind of contribution but I have to admit the patch is huge and hard to digest.
When solving multiple problems we tend to prefer multiple separate patches where each one addresses a different issue. Is there any chance you can split it in patches along your points above ?
Also a few comments in general. The goal of SSSD is to never use more than one connection at a time for account information. So your patch is kind of changing our fundamental goal by allowing multiple connections. We need to carefully evaluate that part.
The authentication case is separate, and it doesn't need to be completely serialized, we can definitely open multiple authentication connections at the same time. I guess the need to serialize the connections is connected with acquiring the kerberos credentials. In that case I would just separate and serialize that part.
I see you started passing around sdap_id_op. The memory hierarchy around sdap_is_op is very delicate and required a lot of very careful handling to avoid having it disappear under our feet at the wrong time. It is meant to represent a single ldap operation tied to a specific ldap context, any changes to its use should be in a separate patch that I want to review carefully. But ideally sdap_id_op is opaque to most of the code and is internal to the processing of replies from the openldap libraries. It should never be used out of this context.
Let me know if you have any questions, I love to see and help contributions. Simo.