On Thu, 9 Sep 2010 08:57:37 -0700
Jeff Schroeder <jeffschroeder(a)computer.org> wrote:
On Thu, Sep 9, 2010 at 8:48 AM, Simo Sorce <ssorce(a)redhat.com>
> On Thu, 9 Sep 2010 16:46:35 +0200
> Ralf Haferkamp <rhafer(a)suse.de> wrote:
>> On Thursday 09 September 2010 15:59:46 Simo Sorce wrote:
>> > On Thu, 09 Sep 2010 09:18:12 -0400
>> > Stephen Gallagher <sgallagh(a)redhat.com> wrote:
>> > > -----BEGIN PGP SIGNED MESSAGE-----
>> > > Hash: SHA1
>> > >
>> > > On 09/09/2010 09:14 AM, Ralf Haferkamp wrote:
>> > > > Hi,
>> > > >
>> > > > Is it really the intended behaviour of the sssd LDAP backend
>> > > > (I am running the current code from the master branch) to
>> > > > only return the group members that are already cached in
>> > > > sysdb and to silently ignore everything else? E.g. when I
>> > > > start sssd with empty caches and do a "getent group
>> > > > <random-ldap-group>" I will only get back the group
>> > > > any members. Somehow I think this can't be intended :)
>> > > >
>> > > > I have started working on a patch to let sssd look up the
>> > > > non-cached users via LDAP (and save them into the cache).
>> > > > Find it attached. Note: That patch is not really complete
>> > > > (e.g. it doesn't handle rfc2307 groups correctly). But
>> > > > before putting more effort into this I like to make sure
>> > > > that I am not trying to fix a "feature" here.
>> > >
>> > > No, it is not intentional that groups should be missing users.
>> > > This is definitely a bug. Please file a ticket upstream.
>> > It is intended if enumerations are off.
>> > Thee reason is that you may end up effectively doing full user
>> > enumerations otherwise if you have a big group that contain all
>> > users.
>> Then it should probably be possible to disable that feature
>> separately from enumeration. While, turning enumerations off by
>> default makes sense to me, I think returning incomplete results
>> when resolving groups by default goes a bit too far.
> It depends on the point of view. For the specific logged in user the
> groups he is interested in are fully update wrt his own membership.
> It is true you may not see other members unless they log in, but why
> would that be a problem ? If they do not log in they are not
> affected by group memberships anyway.
>> BTW, I just found out that the behaviour of
>> getgrouplist()/initgroups() is similar currently. It will only
>> return the groups that are already present in the cache. That
>> means in many cases it will return nothing. (Or just the gid you
>> supplied via the group argument).
> Initgroups will always call the LDAP server, and will make sure all
> groups the user is member of are available.
>> > Not only that but it would be an inefficient enumeration as it
>> > would be repeated multiple times for each group.
>> Why is that? Users should of course be saved in the sysdb after
>> being read from LDAP. If you read another group which has that
>> user as a member the result is taken from the cache of course.
>> (That's what my patch tries to do at least, though I admit there
>> is still room for optimization)
> In order to save a user you have to look it up. so if you have a
> group with 50k users in it you will have to do 50k lookups.
> Multiply this for N groups.
> If you need enumeration completeness then you should just have
> enumerations turned on IMO.
> Now thinking a bit more there is probably a way to avoid the
> lookups. You can avoid the 50k lookups by actually *not* fetching
> each member from LDAP but create fake entries that are marked as
> expired so they are immediately refreshed with the right data if
> they are ever requested from the user space.
> But this technique has some pretty sever limits when using
> rfc2307bis and nested groups. (should work just fine for classic
> One limit is that you don't know whether a DN is a user or a Group.
> It is true that we can adopt some heuristics like examining the DN
> components to get a clue, but they would be pretty much schema
> dependent and would fail consistently in some scenarios. So you may
> fail to properly unroll nested groups.
> The other is that the actual user name may not be contained within
> the DN. if the DN includes the uid attribute as part of it does,
> but if the CN is used (like against AD where the samAccoutnName
> attribute is used as the OS user name) then you do not have the
> user name you need to present in the group as member. So, again, it
> would work only in some setups.
> Another concern is about how to properly do caching.
> When we did the initgroups call I used to fully resolve groups, we
> removed that code because it would make initgroups unbearably slow
> as it would end up doing way too many lookups. So we need to make
> sure initgroups can create/manipulate groups and know when a group
> can be considered up to date, and yet at the same time mark it as
> "incomplete" or something like that for getent group <group> calls
> (and vice versa ?)
> I guess we could have a third option to manage this, and allow
> people full group (all member users/groups(recursive)) lookups if
> they really want/need it. But it has a lot of subtle interactions
> we need to care about to avoid perf. side effects.
> May be something like:
> fully_resolve_groups defaulting to false ?
Having enumerate = False but still enumerate the members of each group
(when the group is looked up) would be nice. Some users got really
confused with the output of getent group showed them in the right
groups, but not all of the members.
Remember that while optimizing for hte 50k member group is important,
it is likely not the most common usage scenario.
Ok, I was also wondering we can use a slightly different set of
heuristics depending on what kind of schema is used.
So we have the following scenarios:
1) If we use rfc2307 classic with memberUid attributes, we can just
create the fake/expired users and be done with it.
2) If we use rfc2307bis w/o nested groups we can count the number of
members on the group entry and switch to a full user enumeration if the
number of members is greater than a defined (potentially user defined)
threshold. Assuming a threshold value of 100, if we have 10 members
we just do 10 lookups, while, if we have > 100 member we do a full
enumeration (it's a single ldap search, and if modifyTimestamp is used
also highly optimized after the first search) so we are sure all users
are there with the right name.
3) If we use rfc2307bis with nested groups we do the same as in 2, but
we add an internal user lookup against the cache we just populated to
determine if some members are groups that need further lookups in order
to complete the group unrolling operation.
4) If we use IPA (rfc2307bis + nested groups + unrolled members in
memberOf attribute), we can do a single search with memberOf=groupDN,
as IPa's memberOf fully resolves nested groups. This would return all
and only the users members of the group.
So if the group has 101 members in a directory of 50k users we do not
do a full 50k users enumeration just because we have a group that is 1
above the threshold.
Do I miss any scenario ?
Simo Sorce * Red Hat, Inc * New York