On Wed, 2011-05-04 at 13:19 +0200, Jakub Hrozek wrote:
The attached patch set uses special dereference LDAP controls to
speed
up RFC2307bis group processing. In my setup, with the LDAP server being
in Boston and client in Brno, the time for "getent group ipausers"
consisting of 2000 entries went down from 1.5 minute to 4 seconds.
One part of the patchset I'd like to draw attention to is the heuristics
in patch #9. Dereference can potentially provide huge savings, but it's
an all-or-nothing operation. As far as I know, you can't only say "only
dereference entries matching this filter", at least not using the
OpenLDAP dereference.
This could actually slow down the processing - consider if a single IPA
user being created triggered a full dereference of "ipausers" group. In
order to overcome this, the code includes a simple heuristics that
either issues a full dereference or a series of single lookups.
The heuristics itself was pulled out of thin air, really. If there are
any suggestions on a better one, I'll be glad to hear them.
And now the patches:
[PATCH 1/9] sdap_get_generic_ext
Add a private sdap_get_generic_ext_send()/_recv() request that
exposes more of ldap_search_ext options, in particular the server
contols. The existing sdap_generic_search_send()/_recv() request
is now a thin wrapper around the new _ext request.
The other important change is that an entry parsing is a callback now.
That was done in order to allow custom parsing for results such as
OpenLDAP deref or Attribute Scoped Queries.
Ack
Ack
[PATCH 3/9] OpenLDAP dereference searches
For more details, see:
http://tools.ietf.org/html/draft-masarati-ldap-deref-00
Nack
What's the point of derefcount in sdap_x_deref_parse_entry()? It's only
ever assigned and never read.
sdap_parse_deref() assumes that it can determine the mapping to use
based on the objectClass, but this falls down in situations like Magic
Private Groups, where user entries have both the posixAccount and
posixGroup attribute.
[PATCH 4/9] Generic dereference search
A generic wrapper around ASQ and OpenLDAP dereference searches.
https://fedorahosted.org/sssd/ticket/635
Nack.
It would be preferable to have a single sdap_deref_search_done()
function instead of a separate one for each deref type. See
sdap_get_initgr_done() for an example of what I mean. There should be
separate _recv() functions, but only one common _done() function to make
it simpler to follow execution.
[PATCH 5/9] append_attrs_to_array allocs on a memory context
A convenience function
Nack.
We discussed this on IRC. This function serves no real purpose (it was
never used prior to patch 9 in this set) and this modification has zero
net effect, since the mem_ctx attribute has no effect on a
talloc_realloc() for previously-allocated memory.
We decided to remove this function entirely, since it just causes
confusion.
[PATCH 6/9] Change sysdb_add_fake_user to add OriginalDN
RFC2307bis code relies heavily on originalDN, so the fake users need to
have an option to store it, too.
Ack.
Review still pending for the RFC2307bis results.
[PATCH 7/9] Use fake users during RFC2307bis nested group processing
Instead of downloading complete user data which is potentionally very
slow, only download the necessary minimum information and store the
users as dummy entries.
[PATCH 8/9] Refactor RFC2307bis nested group processing
This patch splits checking cache and hash tables into standalone
functions. This will make it easy to reuse the code in a new branch that
uses dereferencing.
[PATCH 9/9] Use dereference in processing RFC2307bis nested groups
Instead of issuing N LDAP requests when processing a group with N users,
utilize the dereference functionality to pull down all the members in a
single LDAP request.