Thanks for response. I re-ran test after adding index as suggested. Which seems to fix the underlying issue and results are inline with older version. So will this index would be added in newer version of sssd ? Is there an easier way to add this index w/o opening an editor ?


older version 
sssd-20201203163147/analysis.out:'Initgroups by name' count: 102 avg: 0.596225137254902 median 0.035176 min: 0.001218 max: 22.037332
sssd-20201203164328/analysis.out:'Initgroups by name' count: 201 avg: 0.313984855721393 median 0.039057 min: 0.000394 max: 23.138423
sssd-20201203165234/analysis.out:'Initgroups by name' count: 100 avg: 0.59339562 median 0.0362155 min: 0.000412 max: 21.559991

With new version + adding index as suggested above.
sssd-20201203145950/analysis.out:'Initgroups by name' count: 103 avg: 0.5026502815533981 median 0.03847 min: 0.000435 max: 21.532709
sssd-20201203152524/analysis.out:'Initgroups by name' count: 90 avg: 1.1182167555555556 median 0.0349585 min: 0.000536 max: 33.604103
sssd-20201203160837/analysis.out:'Initgroups by name' count: 102 avg: 0.4324193431372549 median 0.0338985 min: 0.000479 max: 19.493188

Above output is by parsing the sssd_nss.log with debug_level 9. As you can see from both older/newer versions, we are seeing a very high max value. All these are when request is being sent from nss->be for resolution. Is is possible similar issue is happening at BE side as well when updating ldb ? Please let me know, if you need additional data. 

Thanks



Sanjay Agrawal


On Thursday, December 3, 2020, 01:32:32 PM EST, Sumit Bose <sbose@redhat.com> wrote:


On Thu, Dec 03, 2020 at 05:50:31PM +0000, Sanjay Agrawal wrote:
> resending
> Sanjay Agrawal
>
>    On Wednesday, December 2, 2020, 03:41:36 PM EST, Sanjay Agrawal <sanjayagrawal@yahoo.com> wrote: 

>  Hi, 
>
> we are seeing an issue with newer version of sssd with centos 7.9 sssd version 1.16.5-10.el7_9.5.x86_64, where Initgroups is taking much longer compared to
> previous version. Can you please look into it. Following are details, including a sample program to reproduce the issue with old and new version.
>
> Sample Program: get_user_groups.py
>     - it just call getgrouplist of a user supplied using libc
>
> Old env -
> OS Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Wed Aug 26 11:48:49 BST 2020 x86_64 x86_64 x86_64 GNU/Linux
> SSSD Version    - 1.16.4-37.el7_8.4.x86_64
> Sample log file - sssd_nss-old.log
> Flamegraph      - flamgraph_sssd_nss-old.svg
> ID  879 'Initgroups by name' testuser1@example
>     starttime Tue Dec  1 18:36:42:092541 2020
>     endtime Tue Dec  1 18:36:42:124463 2020
>     lookup_time 0.031922 sec
>
> New env
> OS Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Nov 18 22:40:48 GMT 2020 x86_64 x86_64 x86_64 GNU/Linux
> SSSD Version    - 1.16.5-10.el7_9.5.x86_64
> Sample log file - sssd_nss-new.log
> Flamegraph      - flamgraph_sssd_nss-new.svg
> ID  776 'Initgroups by name' testuser1@example
>     starttime 2020-12-01 17:31:24:328419
>     endtime 2020-12-01 17:31:24:451778
>     lookup_time 0.123359 sec
>
>
> It seem to be due to addtional .1 second taken during following two trace (from sssd_nss-new.log)
> (2020-12-01 17:31:24:355458): [nss] [cache_req_done] (0x0400): CR #776: Finished: Success
> (2020-12-01 17:31:24:451727): [nss] [sysdb_search_group_by_id] (0x0400): No such entry
> (2020-12-01 17:31:24:451757): [nss] [nss_protocol_fill_initgr] (0x0080): Unable to find primary gid [2]: No such file or directory
>
> It may be related to following change, which seems to ref to sysdb_search_group_by_id
> nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71

Hi,

thanks for the analysis and I guess you might be right. The patch adds a
search where one attribute in the filter does not have an index. As a
result the search has to run over the whole cache.

I wonder if you would like to try to add such index? To do this please
stop SSSD and call:

    ldbedit -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST'

which should open an editor showing something similar to


# editing 1 records
# record 1
dn: @INDEXLIST
@IDXATTR: cn
@IDXATTR: objectclass
@IDXATTR: member
@IDXATTR: memberof
@IDXATTR: name
@IDXATTR: uidNumber
@IDXATTR: gidNumber
@IDXATTR: lastUpdate
@IDXATTR: dataExpireTimestamp
@IDXATTR: originalDN
@IDXATTR: nameAlias
@IDXATTR: servicePort
@IDXATTR: serviceProtocol
@IDXATTR: sudoUser
@IDXATTR: sshKnownHostsExpire
@IDXATTR: objectSIDString
@IDXATTR: ghost
@IDXATTR: userPrincipalName
@IDXATTR: canonicalUserPrincipalName
@IDXATTR: uniqueID
@IDXATTR: mail
@IDXATTR: userMappedCertificate
@IDXATTR: ccacheFile
@IDXATTR: ipHostNumber
@IDXATTR: ipNetworkNumber
distinguishedName: @INDEXLIST


Please add in the @IDXATTR list a line

    @IDXATTR: originalADgidNumber

and exit the editor, the databased will be reindexed at this point.
Finally start SSSD again and rerun the test.

HTH

bye,
Sumit

>
> |
> |
> |
> |  |  |
>
>  |
>
>  |
> |
> |  |
> nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71

>
> SYSDB_PRIMARY_GROUP_GIDNUM contains original primary group id from AD because any possible override may not be k...
>  |
>
>  |
>
>  |
>
>
>
>
>
> Sanjay Agrawal 


> _______________________________________________
> sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
_______________________________________________
sssd-users mailing list -- sssd-users@lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org