[SSSD] [PATCH] Use dereference to speed up RFC2307bis group processing

Jakub Hrozek jhrozek at redhat.com
Wed May 4 11:19:13 UTC 2011


The attached patch set uses special dereference LDAP controls to speed
up RFC2307bis group processing. In my setup, with the LDAP server being
in Boston and client in Brno, the time for "getent group ipausers"
consisting of 2000 entries went down from 1.5 minute to 4 seconds.

One part of the patchset I'd like to draw attention to is the heuristics
in patch #9. Dereference can potentially provide huge savings, but it's
an all-or-nothing operation. As far as I know, you can't only say "only
dereference entries matching this filter", at least not using the
OpenLDAP dereference.

This could actually slow down the processing - consider if a single IPA
user being created triggered a full dereference of "ipausers" group. In
order to overcome this, the code includes a simple heuristics that
either issues a full dereference or a series of single lookups.

The heuristics itself was pulled out of thin air, really. If there are
any suggestions on a better one, I'll be glad to hear them.

And now the patches:

[PATCH 1/9] sdap_get_generic_ext
Add a private sdap_get_generic_ext_send()/_recv() request that
exposes more of ldap_search_ext options, in particular the server
contols. The existing sdap_generic_search_send()/_recv() request
is now a thin wrapper around the new _ext request.

The other important change is that an entry parsing is a callback now.
That was done in order to allow custom parsing for results such as
OpenLDAP deref or Attribute Scoped Queries.

[PATCH 2/9] Add support for Attribute Scoped Queries
For more details on ASQ, see:
http://msdn.microsoft.com/en-us/library/aa366976%28VS.85%29.aspx
http://msdn.microsoft.com/en-us/library/aa746418%28v=VS.85%29.aspx

[PATCH 3/9] OpenLDAP dereference searches
For more details, see:
http://tools.ietf.org/html/draft-masarati-ldap-deref-00

[PATCH 4/9] Generic dereference search
A generic wrapper around ASQ and OpenLDAP dereference searches.
https://fedorahosted.org/sssd/ticket/635

[PATCH 5/9] append_attrs_to_array allocs on a memory context
A convenience function

[PATCH 6/9] Change sysdb_add_fake_user to add OriginalDN
RFC2307bis code relies heavily on originalDN, so the fake users need to
have an option to store it, too.

[PATCH 7/9] Use fake users during RFC2307bis nested group processing
Instead of downloading complete user data which is potentionally very
slow, only download the necessary minimum information and store the
users as dummy entries.

[PATCH 8/9] Refactor RFC2307bis nested group processing
This patch splits checking cache and hash tables into standalone
functions. This will make it easy to reuse the code in a new branch that
uses dereferencing.

[PATCH 9/9] Use dereference in processing RFC2307bis nested groups
Instead of issuing N LDAP requests when processing a group with N users,
utilize the dereference functionality to pull down all the members in a
single LDAP request.

https://fedorahosted.org/sssd/ticket/799
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-sdap_get_generic_ext.patch
Type: text/x-patch
Size: 15516 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20110504/b4fb06e9/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Add-support-for-Attribute-Scoped-Queries.patch
Type: text/x-patch
Size: 7765 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20110504/b4fb06e9/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-OpenLDAP-dereference-searches.patch
Type: text/x-patch
Size: 13718 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20110504/b4fb06e9/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-Generic-dereference-search.patch
Type: text/x-patch
Size: 6686 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20110504/b4fb06e9/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0005-append_attrs_to_array-allocs-on-a-memory-context.patch
Type: text/x-patch
Size: 1968 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20110504/b4fb06e9/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0006-Change-sysdb_add_fake_user-to-add-OriginalDN.patch
Type: text/x-patch
Size: 2543 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20110504/b4fb06e9/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0007-Use-fake-users-during-RFC2307bis-nested-group-proces.patch
Type: text/x-patch
Size: 9113 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20110504/b4fb06e9/attachment-0006.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0008-Refactor-RFC2307bis-nested-group-processing.patch
Type: text/x-patch
Size: 15606 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20110504/b4fb06e9/attachment-0007.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0009-Use-dereference-in-processing-RFC2307bis-nested-grou.patch
Type: text/x-patch
Size: 18954 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20110504/b4fb06e9/attachment-0008.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <https://lists.fedorahosted.org/pipermail/sssd-devel/attachments/20110504/b4fb06e9/attachment.sig>


More information about the sssd-devel mailing list