Thanks Sumit, it worked for us. For some if queries still taking too long. I have collected all the logs. They are pretty large. Is there w site I can upload to ? 

Thanks,
Sanjay Agrawal


On Friday, December 4, 2020, 07:24:31 AM EST, Sumit Bose <sbose@redhat.com> wrote:


On Thu, Dec 03, 2020 at 10:44:09PM +0000, Sanjay Agrawal wrote:
> Thanks for response. I re-ran test after adding index as suggested. Which seems to fix the underlying issue and results are inline with older version. So will this index would be added in newer version of sssd ? Is there an easier way to add this index w/o opening an editor ?

Hi,

something like:

ldbmodify -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST' << END_END
dn: @INDEXLIST
changetype: modify
add: @IDXATTR
@IDXATTR: originalADgidNumber

END_END

should work as well or you put ldif data into a file and add the
filename to the command line.

>
> older version 
> sssd-20201203163147/analysis.out:'Initgroups by name' count: 102 avg: 0.596225137254902 median 0.035176 min: 0.001218 max: 22.037332
> sssd-20201203164328/analysis.out:'Initgroups by name' count: 201 avg: 0.313984855721393 median 0.039057 min: 0.000394 max: 23.138423
> sssd-20201203165234/analysis.out:'Initgroups by name' count: 100 avg: 0.59339562 median 0.0362155 min: 0.000412 max: 21.559991
>
> With new version + adding index as suggested above.
> sssd-20201203145950/analysis.out:'Initgroups by name' count: 103 avg: 0.5026502815533981 median 0.03847 min: 0.000435 max: 21.532709
> sssd-20201203152524/analysis.out:'Initgroups by name' count: 90 avg: 1.1182167555555556 median 0.0349585 min: 0.000536 max: 33.604103
> sssd-20201203160837/analysis.out:'Initgroups by name' count: 102 avg: 0.4324193431372549 median 0.0338985 min: 0.000479 max: 19.493188
>
> Above output is by parsing the sssd_nss.log with debug_level 9. As you can see from both older/newer versions, we are seeing a very high max value. All these are when request is being sent from nss->be for resolution. Is is possible similar issue is happening at BE side as well when updating ldb ? Please let me know, if you need additional data. 

Yes, the max times are most likely related to looking up fresh data from
the server via the backend. It is not unusual the Initgroups request
will take some time, e.g. if you user belongs to many groups and the
cache groups entries are expired as well and must be refreshed as well.
But to better understand what is the reason for the long delay in your
case we would need the backend logs with debug_level=9. There are some
code paths known to be slow, Alexey is e.g. currently working on one of
them. But there might be other reasons for delay as well, e.g. slowly
responding servers, failover to a different server etc.

bye,
Sumit

>
> Thanks
>
>
> Sanjay Agrawal
>
>    On Thursday, December 3, 2020, 01:32:32 PM EST, Sumit Bose <sbose@redhat.com> wrote: 

>  On Thu, Dec 03, 2020 at 05:50:31PM +0000, Sanjay Agrawal wrote:
> > resending
> > Sanjay Agrawal
> >
> >    On Wednesday, December 2, 2020, 03:41:36 PM EST, Sanjay Agrawal <sanjayagrawal@yahoo.com> wrote: 
> > 
> >  Hi, 
> >
> > we are seeing an issue with newer version of sssd with centos 7.9 sssd version 1.16.5-10.el7_9.5.x86_64, where Initgroups is taking much longer compared to
> > previous version. Can you please look into it. Following are details, including a sample program to reproduce the issue with old and new version.
> >
> > Sample Program: get_user_groups.py
> >     - it just call getgrouplist of a user supplied using libc
> >
> > Old env -
> > OS Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Wed Aug 26 11:48:49 BST 2020 x86_64 x86_64 x86_64 GNU/Linux
> > SSSD Version    - 1.16.4-37.el7_8.4.x86_64
> > Sample log file - sssd_nss-old.log
> > Flamegraph      - flamgraph_sssd_nss-old.svg
> > ID  879 'Initgroups by name' testuser1@example
> >     starttime Tue Dec  1 18:36:42:092541 2020
> >     endtime Tue Dec  1 18:36:42:124463 2020
> >     lookup_time 0.031922 sec
> >
> > New env
> > OS Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Nov 18 22:40:48 GMT 2020 x86_64 x86_64 x86_64 GNU/Linux
> > SSSD Version    - 1.16.5-10.el7_9.5.x86_64
> > Sample log file - sssd_nss-new.log
> > Flamegraph      - flamgraph_sssd_nss-new.svg
> > ID  776 'Initgroups by name' testuser1@example
> >     starttime 2020-12-01 17:31:24:328419
> >     endtime 2020-12-01 17:31:24:451778
> >     lookup_time 0.123359 sec
> >
> >
> > It seem to be due to addtional .1 second taken during following two trace (from sssd_nss-new.log)
> > (2020-12-01 17:31:24:355458): [nss] [cache_req_done] (0x0400): CR #776: Finished: Success
> > (2020-12-01 17:31:24:451727): [nss] [sysdb_search_group_by_id] (0x0400): No such entry
> > (2020-12-01 17:31:24:451757): [nss] [nss_protocol_fill_initgr] (0x0080): Unable to find primary gid [2]: No such file or directory
> >
> > It may be related to following change, which seems to ref to sysdb_search_group_by_id
> > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
>
> Hi,
>
> thanks for the analysis and I guess you might be right. The patch adds a
> search where one attribute in the filter does not have an index. As a
> result the search has to run over the whole cache.
>
> I wonder if you would like to try to add such index? To do this please
> stop SSSD and call:
>
>     ldbedit -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b '@INDEXLIST'
>
> which should open an editor showing something similar to
>
>
> # editing 1 records
> # record 1
> dn: @INDEXLIST
> @IDXATTR: cn
> @IDXATTR: objectclass
> @IDXATTR: member
> @IDXATTR: memberof
> @IDXATTR: name
> @IDXATTR: uidNumber
> @IDXATTR: gidNumber
> @IDXATTR: lastUpdate
> @IDXATTR: dataExpireTimestamp
> @IDXATTR: originalDN
> @IDXATTR: nameAlias
> @IDXATTR: servicePort
> @IDXATTR: serviceProtocol
> @IDXATTR: sudoUser
> @IDXATTR: sshKnownHostsExpire
> @IDXATTR: objectSIDString
> @IDXATTR: ghost
> @IDXATTR: userPrincipalName
> @IDXATTR: canonicalUserPrincipalName
> @IDXATTR: uniqueID
> @IDXATTR: mail
> @IDXATTR: userMappedCertificate
> @IDXATTR: ccacheFile
> @IDXATTR: ipHostNumber
> @IDXATTR: ipNetworkNumber
> distinguishedName: @INDEXLIST
>
>
> Please add in the @IDXATTR list a line
>
>     @IDXATTR: originalADgidNumber
>
> and exit the editor, the databased will be reindexed at this point.
> Finally start SSSD again and rerun the test.
>
> HTH
>
> bye,
> Sumit
>
> >
> > |
> > |
> > |
> > |  |  |
> >
> >  |
> >
> >  |
> > |
> > |  |
> > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71
> >
> > SYSDB_PRIMARY_GROUP_GIDNUM contains original primary group id from AD because any possible override may not be k...
> >  |
> >
> >  |
> >
> >  |
> >
> >
> >
> >
> >
> > Sanjay Agrawal 
>
> > _______________________________________________
> > sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> > To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
> _______________________________________________
> sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org