[SSSD] Thoughts...

Simo Sorce simo at redhat.com
Wed Oct 5 13:14:31 UTC 2011


On Wed, 2011-10-05 at 14:38 +0200, Jan Zelený wrote:
> > On Wed, 2011-10-05 at 07:58 -0400, Stephen Gallagher wrote:
> > > On Wed, 2011-10-05 at 07:56 -0400, Simo Sorce wrote:
> > > > On Wed, 2011-10-05 at 07:30 -0400, Stephen Gallagher wrote:
> > > > > CCing sssd-devel
> > > > > 
> > > > > On Tue, 2011-10-04 at 22:41 +0000, JR Aquino wrote:
> > > > > > So I have been looking through the results of ldbsearch and have
> > > > > > been wondering...
> > > > > > 
> > > > > > Why is it that I see cached entries for objects (hbac/ hosts/ etc)
> > > > > > that have no correlation to the host itself?
> > > > > > 
> > > > > > Might it be more optimized to search / cache only the data relevant
> > > > > > to the client system in question?
> > > > > > 
> > > > > > Surely that would cut down on the exponential costs against an ldap
> > > > > > database with grown hosts/users/hbacrules/sudorules ?
> > > > > 
> > > > > Well, the issue is this: rules can specify hostgroups as well as
> > > > > hosts. So in order to decide which rules to download, we need to
> > > > > know which hostgroups the current machine is a member of. This
> > > > > results in a minimum of two ldap searches (one for the hosts, and
> > > > > then one for the hostgroups) to get the machine-specific
> > > > > information. We then need to perform a rule search for just the
> > > > > rules that apply to that machine.
> > > > > 
> > > > > After we have the rule, we'll need to perform an iterative series of
> > > > > lookups for every host and hostgroup that the rules we just acquired
> > > > > list as a source host. Now, suppose that one or more of the
> > > > > hostgroups specified in source host contains thousands of host
> > > > > entries?
> > > > > 
> > > > > So in other words, doing it this way will result in a net increase in
> > > > > the number of back-and-forth LDAP exchanges. In theory this is much
> > > > > slower than just grabbing everything in two lookups up front (getting
> > > > > all hosts and hostgroups) and doing the processing locally.
> > > > > 
> > > > > In practice, we're hitting a performance issue because of the way
> > > > > we're updating the data. So it's better all-around to resolve the
> > > > > issue here, rather than deferring the problem by changing the lookup
> > > > > order around and potentially increasing the LDAP lookups. And since
> > > > > we still might end up with rules that have to handle a lot of hosts,
> > > > > we'll still have the same update issue.
> > > > > 
> > > > > The problem in the source is this: I tried to simplify the update
> > > > > process by simply wiping out the hosts and hostgroups subtrees in the
> > > > > cache and then writing the new values in. The problem with this is
> > > > > that deleting entries with memberOf attributes is an expensive
> > > > > operation, which I did not take into account.
> > > > > 
> > > > > There are two approaches to resolving this that I'm taking into
> > > > > consideration.
> > > > > 
> > > > > 1) Instead of doing a full purge followed by a full write, make only
> > > > > changes. This will be a somewhat more complex logic to follow, but if
> > > > > little or nothing has changed on the server, then it will save us a
> > > > > LOT of memberOf operations.
> > > > > 
> > > > > 2) Add a new feature to the memberOf plugin to allow us to defer
> > > > > processing memberOf until all entries have first been written
> > > > > (keeping an index of the attributes that will need processing). This
> > > > > is *very* complicated and potentially dangerous (though also has the
> > > > > potential for the highest performance gain), so I'd prefer to try
> > > > > option 1 first.
> > > > 
> > > > 3) speed up deletion of entire subtrees using a special control in the
> > > > memberof plugin, relatively simple and safe if you ask me.
> > > 
> > > Could you describe how this would work? Because I was thinking about
> > > this and I couldn't come up with a way to make it work.
> > 
> > Right now if you are deleting a subtree you will execute a full memberof
> > delete for each entry. That means doing expensive searches all over the
> > tree and changes to each and every entry repeatedly.
> > 
> > Assume you delete 10 groups and you have 100 users each member of each
> > group. It mean 1000 operations as you change each user in turn to remove
> > the memberof of the group you are currently handling, 10 times, one for
> > each group.
> > 
> > The idea is to add a new control to memberof to tell it: deleting this
> > whole subtree. The control will be passed down and will tell memberof
> > that it should not do the normal delte operation until the final delete
> > of the top entry of the subtree. When that is doen we do a single search
> > for all member/memberof attributes of the form *,<subtree suffix>, and
> > for each result we remove all member/memberof attribute once.
> > If the tree is self-contained wrt members this turns out to be a search
> > that returns 0 entries so no modify operations at all.
> > 
> > If there are members outside of this tree we do one mod operation for
> > each regardless of how many attribute match per entry.
> > 
> > Simo.
> 
> This could certainly work. I don't know about the original design, but the new 
> one should be able to handle it well. One question though. How does one pass a 
> control to the memberof plugin during delete operation? You mean like some 
> special attribute?

No I really mean a control, you can always pass controls to ldb_requets.
It means we need to extend sysdb interface to be able to specify that we
need this control, but shouldn't be too difficult.

check for API with the term "control" in ldb.h to see how to manipulate
controls, or look at samba's ldb modules to get a sense of how controls
are handled there.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York




More information about the sssd-devel mailing list