Alexey,

Yes I see that now.  That every time it starts a new LDAP connection, it starts by querying rootDSE.  So I have to look further in the logs.

I think I have discerned a pattern.  It appears that on each hour and half-hour, it's querying the members of the simple_allow_groups line.  I have examined this on 5 different servers in different geographical locations, it holds true for each server. 

for example, in /var/log/sssd dir:

# grep -A 4 'sbus_dispatch.*Dispatching.' sssd_amer.corp.com.log | grep 'name=' | grep BE_REQ_GROUP 

Here's the output.  Each ellipsis is 10 - 20 lines omitted that occurs in the same second.

(2023-10-03 10:07:50): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=oracle@amer.corp.com]
(2023-10-03 10:07:50): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=oracle@emea.corp.com]
...
(2023-10-03 10:30:02): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=zabbix-support@amer.corp.com]
(2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=emeastorageadmins@amer.corp.com]
(2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apacstorageadmins@amer.corp.com]
(2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apacstorageadmins@emea.corp.com]
(2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=amerlnxsvcdelauttfs@amer.corp.com]
(2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=amerlinuxengtfssupt@amer.corp.com]
(2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=amerlinuxeng@amer.corp.com]
...
(2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=oracle@apac.corp.com]
(2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=oracle@japn.corp.com]
(2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=oracle@corp.com]
(2023-10-03 11:00:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=pptadminspac@amer.corp.com]
(2023-10-03 11:00:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=unv_svcgrid_login@amer.corp.com]
(2023-10-03 11:00:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=gbl_storage_admins@amer.corp.com]
...
(2023-10-03 11:30:02): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=zabbix-support@amer.corp.com]
(2023-10-03 11:30:02): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=emeastorageadmins@amer.corp.com]
(2023-10-03 11:30:06): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apacstorageadmins@amer.corp.com]
(2023-10-03 11:30:06): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apacstorageadmins@emea.corp.com]
...
(2023-10-03 12:00:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=pptadminspac@amer.corp.com]
(2023-10-03 12:00:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=unv_svcgrid_login@amer.corp.com]
(2023-10-03 12:00:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=gbl_storage_admins@amer.corp.com]
(2023-10-03 12:00:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=emeastorageadmins@amer.corp.com]
(2023-10-03 12:00:06): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=emeastorageadmins@emea.corp.com]
...
(2023-10-03 12:30:02): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=zabbix-support@amer.corp.com]
(2023-10-03 12:30:02): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=emeastorageadmins@amer.corp.com]
(2023-10-03 12:30:04): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apacstorageadmins@amer.corp.com]
...
(2023-10-03 12:41:16): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=emeastorageadmins@amer.corp.com]
(2023-10-03 12:41:16): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apacstorageadmins@amer.corp.com]
(2023-10-03 12:41:16): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apacstorageadmins@emea.corp.com]
...
(2023-10-03 12:49:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=oracle@amer.corp.com]
(2023-10-03 12:49:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=oracle@emea.corp.com]
(2023-10-03 12:49:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=oracle@apac.corp.com]
(2023-10-03 12:49:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=oracle@japn.corp.com]
(2023-10-03 12:49:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=oracle@corp.com]
...
(2023-10-03 13:00:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=pptadminspac@amer.corp.com]
(2023-10-03 13:00:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=unv_svcgrid_login@amer.corp.com]
(2023-10-03 13:00:01): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=gbl_storage_admins@amer.corp.com]
...
(2023-10-03 13:26:18): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=zabbix-support@amer.corp.com]
(2023-10-03 13:26:18): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=emeastorageadmins@amer.corp.com]
...
(2023-10-03 13:30:02): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apaclinuxsup@amer.corp.com]
(2023-10-03 13:30:02): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apaclinuxsup@emea.corp.com]
(2023-10-03 13:30:02): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apaclinuxeng@amer.corp.com]
(2023-10-03 13:30:02): [be[amer.corp.com]] [dp_get_account_info_send] (0x0200): Got request for [0x2][BE_REQ_GROUP][name=apaclinuxeng@emea.corp.com]
...
and it continues on, each and every half-hour.

So it appears that something is waking up every half-hour and validating memberships in the simple_allow_groups.  I don't claim that's all that's being performed on this half-hour wake-up, but it is clear this is occuring.

Different servers have different simple_allow_group memberships;  it's always the memberships for that specific server that's being queried.

Spike


On Mon, Oct 2, 2023 at 12:23 PM Alexey Tikhonov <atikhono@redhat.com> wrote:
On Mon, Oct 2, 2023 at 7:01 PM Spike White <spikewhitetx@gmail.com> wrote:
>
> So the idea to turn on debug_level = 9 on the client and view the logs was inspired.  We turned on debug level 9 on 4 clients;
>
> 2 in the list (that we got from AD team of servers in that AMERAustin site hitting the non-AMER Austin AD DCs).
> 2 not in their list.  (1 in another AMER site).
>
> Consistently, we see them querying the rootDSE for all these domains on the hour and the half-hour.  Querying the local AMER rootDSE in Austin is not a problem;  they have beaucoup AD DCs in Austin.  Querying the other domains' rootDSEs in Austin is a problem;  they typically only have 2 AD DCs from each region.
>
> Here’s an example from the client logs.  First client:
>
>
>
> (2023-10-03  0:30:06): [be[amer.corp.com]] [sdap_ldap_connect_callback_add] (0x1000): New LDAP connection to [ldap://ausdc16corp05.corp.com:389/??base] with fd [30].
>
> (2023-10-03  0:30:06): [be[amer.corp.com]] [sdap_get_rootdse_send] (0x4000): Getting rootdse
>
> …
>
> (2023-10-03  0:41:18): [be[amer.corp.com]] [sdap_ldap_connect_callback_add] (0x1000): New LDAP connection to [ldap://AUSDC16ROAMER01.amer.corp.com:389/??base] with fd [20].
>
> (2023-10-03  0:41:18): [be[amer.corp.com]] [sdap_get_rootdse_send] (0x4000): Getting rootdse
>
>
>
> Another client:
>
>
>
> (2023-10-02 11:30:02): [be[amer.corp.com]] [sdap_ldap_connect_callback_add] (0x4000): [RID#48] New connection to [ldap://ausdc16amer33.amer.corp.com:3268/??base] with fd [25]
>
> (2023-10-02 11:30:02): [be[amer.corp.com]] [sdap_get_rootdse_send] (0x4000): [RID#48] Getting rootdse
>

What goes next for this 'RID#48', after 'rootdse' is read?

SSSD doesn't query 'rootdse' on its own. It is being read (as a first
operation) when a new connection is established.
You need to see what happens next to figure out *why* this new
connection is established.

As a practical side note, you can also increase
`ldap_connection_expire_timeout` to keep connections longer.
But I would figure out the reason first.


> --
>
> (2023-10-02 11:30:04): [be[amer.corp.com]] [sdap_ldap_connect_callback_add] (0x4000): [RID#49] New connection to [ldap://ausdc16emea05.emea.corp.com:389/??base] with fd [26]
>
> (2023-10-02 11:30:04): [be[amer.corp.com]] [sdap_get_rootdse_send] (0x4000): [RID#49] Getting rootdse
>
> --
>
> (2023-10-02 11:30:05): [be[amer.corp.com]] [sdap_ldap_connect_callback_add] (0x4000): [RID#50] New connection to [ldap://ausdc16apac06.apac.corp.com:389/??base] with fd [27]
>
> (2023-10-02 11:30:05): [be[amer.corp.com]] [sdap_get_rootdse_send] (0x4000): [RID#50] Getting rootdse
>
> --
>
> (2023-10-02 11:30:05): [be[amer.corp.com]] [sdap_ldap_connect_callback_add] (0x4000): [RID#51] New connection to [ldap://AUSDC16JAPN02.japn.corp.com:389/??base] with fd [28]
>
> (2023-10-02 11:30:05): [be[amer.corp.com]] [sdap_get_rootdse_send] (0x4000): [RID#51] Getting rootdse
>
> --
>
> (2023-10-02 11:32:52): [be[amer.corp.com]] [sdap_ldap_connect_callback_add] (0x4000): [RID#84] New connection to [ldap://ausdc16emea05.emea.corp.com:389/??base] with fd [26]
>
> (2023-10-02 11:32:52): [be[amer.corp.com]] [sdap_get_rootdse_send] (0x4000): [RID#84] Getting rootdse
>
> --
>
>
>
> BTW, this seems to occurs on both RHEL7 and RHEL8.  (Haven't looked at our RHEL9 builds yet).   It's occurring on all servers to all rootDSEs, but only a problem for AMERAustin, since Austin is such a heavily-populated.
>
>
> These rootDSEs change almost never.  Any way to have it query not as frequently, or randomize when servers query these rootDSEs.
>
>
> Spike
>
> On Mon, Oct 2, 2023 at 2:37 AM Alexey Tikhonov <atikhono@redhat.com> wrote:
>>
>> Hi,
>>
>> On Mon, Oct 2, 2023 at 6:20 AM Spike White <spikewhitetx@gmail.com> wrote:
>>>
>>> All,
>>>
>>> Is there anything in sssd's RHEL and RHEL-like Linux server OS settings that perform LDAP binds or connections to AD every 30 minutes?
>>>
>>> What our AD team is seeing is all of the DCs in our biggest AMER AD site peak with LDAP sessions for about 10 minutes at the top of the hour then again at the bottom of the hour.  No other AD site in the world appears to see this behavior not even other AD sites in this metro area.
>>>
>>> The reason they noticed is that our non-amer DCs in this biggest AD site hit their 5k LDAP client session limit during those 10 minutes every 30 minutes.  Meaning any clients attempting to establish a LDAP session past 5000 are dropped by the DC.  In their research they see thousands LDAP Binds by RHEL Linux servers against two specific non-AMER AD DCs in a short period of time after digging through some LDAP log samples that they pulled from these DCs.
>>
>>
>> Can they also say what operations are being performed by those connections?
>> Or can you check SSSD logs on the client side?
>>
>> I wonder if this could be `ldap_sudo_smart_refresh_interval`...
>>
>>
>>>
>>>
>>> In this major AD sites, we have dozens and dozens of AMER AD DCs.  So there's enough preferred AD DCs to spread the load.  But typically for the non-AMER regions, the AD team puts 2 of each regions DCs in a site.  For instance, for APAC they would be put two APAC DCs in this AMER major site.  Thus all AMER RHEL servers in this site would randomly hit dozens of AMER DCs, but concentrate on these two preferred APAC DCs.  (preferred because they're in this locatiion).
>>>
>>> I know our older AD integration product used to hit AD every 30 mins to check GPOs, but we're not implementing GPOs with sssd.
>>>
>>> Spike
>>> _______________________________________________
>>> sssd-users mailing list -- sssd-users@lists.fedorahosted.org
>>> To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
>>> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>>
>> _______________________________________________
>> sssd-users mailing list -- sssd-users@lists.fedorahosted.org
>> To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
>> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>
> _______________________________________________
> sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
sssd-users mailing list -- sssd-users@lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue