All,

Turns out this particular application connects to kerberos (AD in our env) directly.  So the errors that show up in their app logs:

ddlflhdm201.us.company.com
WARN June 15, 2023 10:08 AM Groups Potential performance problem: getGroups(user=hbase) took 58022 milliseconds
ddlflhdm304.us.company.com
WARN June 15, 2023 8:38 AM Groups Potential performance problem: getGroups(user=mapred) took 39962 milliseconds
ddlplhdm505.us.company.com
WARN June 16, 2023 5:37 AM Groups Potential performance problem: getGroups(user=hdfs) took 62437 milliseconds.
ddlplhdm305.us.company.com
WARN June 15, 2023 9:36 PM Groups Potential performance problem: getGroups(user=hdfs) took 58017 milliseconds
ddlplhdm505.us.company.com
WARN June 10, 2023 8:01 AM Groups Potential performance problem: getGroups(user=yarn) took 58017 milliseconds.


arise from their kerberos client implementation.  Not from sssd.  The insights about filtering on local users might be useful for the app team when communicating back with their app vendor.  But ultimately, it's between this app team and their app vendor.  sssd is in no way involved in this.

Sorry for the confusion, but the app team was blaming sssd for this slowness.  Only on exhaustive research did they determine their app is contacting kerberos directly.

Spike



On Fri, Jun 23, 2023 at 11:07 AM Spike White <spikewhitetx@gmail.com> wrote:
Appreciate the insight.  These are production RHEL7 servers, which I see are based on sssd-1.16.5-xxx.  As in anything production, I am risk-adverse.

I'll just filter and move on.  Appreciate your help.

Spike

On Fri, Jun 23, 2023 at 9:02 AM Alexey Tikhonov <atikhono@redhat.com> wrote:
Hi,


On Thu, Jun 22, 2023 at 7:52 PM Spike White <spikewhitetx@gmail.com> wrote:
Alexey,

Thanks for the suggestion.

This is a commercial application.  Cloudera's hadoop implementation.  No idea if they use getgrouplist() under the hood.  I can ask our Cloudera tech engineer, but I doubt he knows either.

sssd-ad seems to work pretty well on these servers, except for these occasional hiccups.  It's over 150 nodes in this hadoop cluster, with only these occasional hiccups.  Been AD integrated for >1 yr and app team happy (except for one annoying occasional cross-domain caching issue.)

In particular, sssd finds its AD site so it's binding to AD DCs local to this datacenter.  (I checked that first thing.)

These sporadic hiccups I discuss earlier seem limited to local users only, looking up their group memberships.

Would the /etc/nsswitch.conf file syntax be:

     initgroups:  files [SUCCESS=return] systemd sss

It seems this would apply for all accounts that were members of local groups and not just these specific accounts.  (Not sure that's a problem for this particular app team, but I've seen AD accounts added in /etc/group as members of local groups.)

To me, "filter_users" in sssd.conf where I specify the specific accounts would be safer.

Then filtering sounds as a way to go.

If you are courageous you can also try to read SSSD logs to figure out the reason for those "sporadic hiccups".
Latest version, around sssd-2.8+, should make it easier to track a specific lookup across a set of logs.

 

Spike

On Thu, Jun 22, 2023 at 10:44 AM Alexey Tikhonov <atikhono@redhat.com> wrote:
Hi,

On Thu, Jun 22, 2023 at 4:47 PM Spike White <spikewhitetx@gmail.com> wrote:
All,

Successful sssd consumer here.

Have an app team running Hadoop.  They're getting these performance errors in their app.

This is from their app logs.

ddlflhdm201.us.company.com
WARN June 15, 2023 10:08 AM Groups Potential performance problem: getGroups(user=hbase) took 58022 milliseconds


Does it use `getgrouplist()` under the hood?
 

ddlflhdm304.us.company.com
WARN June 15, 2023 8:38 AM Groups Potential performance problem: getGroups(user=mapred) took 39962 milliseconds
ddlplhdm505.us.company.com
WARN June 16, 2023 5:37 AM Groups Potential performance problem: getGroups(user=hdfs) took 62437 milliseconds.
ddlplhdm305.us.company.com
WARN June 15, 2023 9:36 PM Groups Potential performance problem: getGroups(user=hdfs) took 58017 milliseconds

ddlplhdm505.us.company.com
WARN June 10, 2023 8:01 AM Groups Potential performance problem: getGroups(user=yarn) took 58017 milliseconds.


Interestingly, all these users are local users created by the Hadoop package installation.  (hbase, mapred, hdfs, yarn are all local users).


These local users are members only of local groups.  So theoretically the search for group memberships should be lightning quick as the groups are found only in /etc/group.


But since my /etc/nsswitch.conf looks like this:


passwd:     files systemd sss

group:      files systemd sss

netgroup:   files sss

automount:  files sss

services:   files

shadow:     files sss

hosts:      files dns myhostname


You might consider specifying 'initgroups' explicitly and using something like '[SUCCESS=return]' between databases to avoid going to sss altogether (but see below)



It not only searches local files for group membership, it searches AD as well.  And doesn't find it, as there is no such AD user or uid.   Eventually timing out.


In /var/log/messages we see this around the same time frame as 


Jun 15 10:08:09 ddlflhdm201 sssd[be[amer.company.com]]: GSSAPI Error: An invalid name was supplied (Success)


Is sssd-ad properly configured at this host?
Initgroups lookup for unknown users should be fast.

 

Is there a way to restrict these particular local app accounts from having their group memberships looked up in AD?

I.e., my [nss] section looks like this today:

[nss]
#debug_level = 9
filter_groups = root
filter_users = root
#override_homedir = /home/%u

Can I do this?

[nss]
filter_users = root,  hbase, mapred, hdfs, yarn

And would I want to set filter_users_in_groups = false?

Spike White


_______________________________________________
sssd-users mailing list -- sssd-users@lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
sssd-users mailing list -- sssd-users@lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
sssd-users mailing list -- sssd-users@lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
sssd-users mailing list -- sssd-users@lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue