[SSSD] Patch to fix LDAP ID backend GSSAPI credential expired messages

Eugene Indenbom eindenbom at gmail.com
Wed Apr 7 14:10:41 UTC 2010


Dear colleagues,

I have finally finished my work on the patch refactoring. All the 
changes made are structured into 4 separate patches:
     0001-GSSAPI-ticket-expiry-time-is-returned-from-ldap_chil.patch
     0002-Added-an-interface-to-query-number-of-configured-fai.patch
     0003-Fixed-recursive-sdap_handle-disconnect-sequence-from.patch
     0004-The-LDAP-ID-backend-connection-logic-has-been-refact.patch

All of them are made by git format-patch with all the patience git has 
managed to find. :)
I have taken into all the notes and suggestion from your previous 
e-mails, but now I want to start from a blank sheet. The work has been 
substantially redone and old quotations take to much room.

--- 0001-GSSAPI-ticket-expiry-time-is-returned-from-ldap_chil.patch

This patch adds expire_time member to sdap_handle. expire_time is the 
time when connection is expected to expire. Currently the time is set 
only for GSSAPI SASL mech. For other currently supported authentication 
types connection never expires and expire_time is set to 0.

Having this time at hand saves from complicated query of kerberos ticket 
cache (see sdap_check_gssapi_reconnect) and is more reliable as kerberos 
ticket cache could be modified since time of connection.

--- 0002-Added-an-interface-to-query-number-of-configured-fai.patch

This patch simply adds an interface to query number of configured 
failover servers. The interface will be really useful when DNS SRV 
record service discovery will be supported. I have added it as a 
provision for forthcoming dynamic failover configuration.

--- 0003-Fixed-recursive-sdap_handle-disconnect-sequence-from.patch

This patch fixes a critical bug of accessing freed memory in 
sdap_handle_release. The problem can be reproduced as follows:
1. Break LDAP connection when LDAP query is in progress (e.g. by 
restarting directory server)
2. sdap_process_result detects communication error and calls 
sdap_handle_release
3. sdap_handle_release calls active sdap_op callback
4. callback deletes sdap_handle
5. sdap_release_handle segfaults or asserts on access to deallocated memory

I have solved the problem by splitting private data from sdap_handle 
into separate structure sdap_handle_data. The deallocation of 
sdap_handle_data is delayed until all enumeration of active ops chain 
are finished (currently only sdap_handle_release enumerates operations). 
The destruction protected region is scoped by 
sdap_handle_data_lock/sdap_handle_data_release calls.

sdap_handle_release contains also other assorted changes to prevent 
double data deallocation.

--- 0004-The-LDAP-ID-backend-connection-logic-has-been-refact.patch

This patch contains refactoring of reconnect retry logic for LDAP 
operations. The ldap_common.c/.h contain reconnect retry API and 
implementation, while changes in ipa_access.c, ldap_id.c and 
ldap_id_enum.c convert existing LDAP queries to new API.

The current reconnect logic does not address the following issues:
  - There is no failover retries after connection breakdown during operation
  - It is possible that 2 connections would be opened in parallel when 2 
operations are executed concurrently

The primary entry points of new reconnect retry framework are: 
sdap_id_op_connect and sdap_id_op_done. The reconnect retry usage can be 
outlined as follows:

1. Operation creates an op handle with sdap_id_op_create
2. When LDAP connection is needed operation call sdap_id_op_connect
3. sdap_id_op_connect:
     - attaches cached connection to sdap_id_op if available
     - or starts a new connection and returns tevent_req for 
synchronization.
4. When asynchronous connection request is completed operation calls 
sdap_id_op_connect_recv to get connection result
5. When operation is done with connection it calls sdap_id_op_done to:
     - notify that LDAP connection is no longer in use
     - check whether reconnect retry is allowed
6. If reconnect retry is allowed operation restarts at step 2.

Therefore sdap_id_op_connect and sdap_id_op_done define a scope, when 
LDAP connection is in use by operation. This allows to organize 
effective and transparent connection caching. More over further changes 
to caching strategy would not affect operation code.

The connection caching strategy employed by patch is simplistic:
1. The connection is established on demand
2. A single connection is cached and discarded from cache when:
     - it is broken or
     - it is about to expire
3. Cached connection is reused (concurrently if needed) by all other 
operations while it is valid
4. The connection is closed when:
     - it is not cached and
     - it is not in use

So typically connection is established on first LDAP query and kept 
until kerberos ticket is expired. Then the connection is reestablished 
with next LDAP query. During normal operation there is strictly one 
connection open.

The number of reconnect retries for each operation is determined as follows:
1. Reconnect retries are stopped and backend put offline when there is 
no more servers to try.
2. There is an absolute limit on retries performed per operation. This 
is required to handle the case when during each retry the connection is 
successfully established, but breaks before any results are returned. In 
such a scenario failover logic will try indefinitely one and the same 
server.
3. The absolute retry limit is calculated as 2 x <number of servers> to 
allow:
     - one established and broken midway connection to each server (e.g. 
reused cached connection)
     - one failed connection attempt to this server, after which 
failover logic moves to next server in the list

And finally it is possible that blackout time for first failed server to 
expire before last failover server is tried. This is also a possibility 
for creation of infinite retry loop.

All changes together has been tested by me as follows:
1. I have tested each operation separately:
     - sdap_account_info: user info, group info,  user groups (see 
ldap_id.c)
     - enum_users and enum_groups (see ldap_id_enum.c)
     - hbac_get_host_info and hbac_get_rules (see ipa_access.c)
2. In the following scenarios:
     - normal operation
     - 1 retry on server failure
     - switch to offline during operation after all trying all servers

Currently patches are being tested during overnight operation.

Eugene
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 0001-GSSAPI-ticket-expiry-time-is-returned-from-ldap_chil.patch
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20100407/178a0036/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 0002-Added-an-interface-to-query-number-of-configured-fai.patch
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20100407/178a0036/attachment-0001.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 0003-Fixed-recursive-sdap_handle-disconnect-sequence-from.patch
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20100407/178a0036/attachment-0002.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 0004-The-LDAP-ID-backend-connection-logic-has-been-refact.patch
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20100407/178a0036/attachment-0003.ksh>


More information about the sssd-devel mailing list