On Wed, 2011-10-05 at 07:56 -0400, Simo Sorce wrote:
On Wed, 2011-10-05 at 07:30 -0400, Stephen Gallagher wrote:
> CCing sssd-devel
>
> On Tue, 2011-10-04 at 22:41 +0000, JR Aquino wrote:
> > So I have been looking through the results of ldbsearch and have been
> > wondering...
> >
> > Why is it that I see cached entries for objects (hbac/ hosts/ etc) that
> > have no correlation to the host itself?
> >
> > Might it be more optimized to search / cache only the data relevant to
> > the client system in question?
> >
> > Surely that would cut down on the exponential costs against an ldap
> > database with grown hosts/users/hbacrules/sudorules ?
>
> Well, the issue is this: rules can specify hostgroups as well as hosts.
> So in order to decide which rules to download, we need to know which
> hostgroups the current machine is a member of. This results in a minimum
> of two ldap searches (one for the hosts, and then one for the
> hostgroups) to get the machine-specific information. We then need to
> perform a rule search for just the rules that apply to that machine.
>
> After we have the rule, we'll need to perform an iterative series of
> lookups for every host and hostgroup that the rules we just acquired
> list as a source host. Now, suppose that one or more of the hostgroups
> specified in source host contains thousands of host entries?
>
> So in other words, doing it this way will result in a net increase in
> the number of back-and-forth LDAP exchanges. In theory this is much
> slower than just grabbing everything in two lookups up front (getting
> all hosts and hostgroups) and doing the processing locally.
>
> In practice, we're hitting a performance issue because of the way we're
> updating the data. So it's better all-around to resolve the issue here,
> rather than deferring the problem by changing the lookup order around
> and potentially increasing the LDAP lookups. And since we still might
> end up with rules that have to handle a lot of hosts, we'll still have
> the same update issue.
>
> The problem in the source is this: I tried to simplify the update
> process by simply wiping out the hosts and hostgroups subtrees in the
> cache and then writing the new values in. The problem with this is that
> deleting entries with memberOf attributes is an expensive operation,
> which I did not take into account.
>
> There are two approaches to resolving this that I'm taking into
> consideration.
>
> 1) Instead of doing a full purge followed by a full write, make only
> changes. This will be a somewhat more complex logic to follow, but if
> little or nothing has changed on the server, then it will save us a LOT
> of memberOf operations.
>
> 2) Add a new feature to the memberOf plugin to allow us to defer
> processing memberOf until all entries have first been written (keeping
> an index of the attributes that will need processing). This is *very*
> complicated and potentially dangerous (though also has the potential for
> the highest performance gain), so I'd prefer to try option 1 first.
3) speed up deletion of entire subtrees using a special control in the
memberof plugin, relatively simple and safe if you ask me.
Could you describe how this would work? Because I was thinking about
this and I couldn't come up with a way to make it work.