So the idea to turn on debug_level = 9 on the client and view the logs was
inspired. We turned on debug level 9 on 4 clients;
2 in the list (that we got from AD team of servers in that AMERAustin site
hitting the non-AMER Austin AD DCs).
2 not in their list. (1 in another AMER site).
Consistently, we see them querying the rootDSE for all these domains on the
hour and the half-hour. Querying the local AMER rootDSE in Austin is not a
problem; they have beaucoup AD DCs in Austin. Querying the other domains'
rootDSEs in Austin is a problem; they typically only have 2 AD DCs from
each region.
Here’s an example from the client logs. First client:
(2023-10-03 0:30:06): [be[amer.corp.com]] [sdap_ldap_connect_callback_add]
(0x1000): New LDAP connection to [ldap://ausdc16corp05.corp.com:389/??base]
with fd [30].
(2023-10-03 0:30:06): [be[amer.corp.com]] [sdap_get_rootdse_send]
(0x4000): Getting rootdse
…
(2023-10-03 0:41:18): [be[amer.corp.com]] [sdap_ldap_connect_callback_add]
(0x1000): New LDAP connection to [ldap://
AUSDC16ROAMER01.amer.corp.com:389/??base] with fd [20].
(2023-10-03 0:41:18): [be[amer.corp.com]] [sdap_get_rootdse_send]
(0x4000): Getting rootdse
Another client:
(2023-10-02 11:30:02): [be[amer.corp.com]] [sdap_ldap_connect_callback_add]
(0x4000): [RID#48] New connection to [ldap://
ausdc16amer33.amer.corp.com:3268/??base] with fd [25]
(2023-10-02 11:30:02): [be[amer.corp.com]] [sdap_get_rootdse_send]
(0x4000): [RID#48] Getting rootdse
--
(2023-10-02 11:30:04): [be[amer.corp.com]] [sdap_ldap_connect_callback_add]
(0x4000): [RID#49] New connection to [ldap://
ausdc16emea05.emea.corp.com:389/??base] with fd [26]
(2023-10-02 11:30:04): [be[amer.corp.com]] [sdap_get_rootdse_send]
(0x4000): [RID#49] Getting rootdse
--
(2023-10-02 11:30:05): [be[amer.corp.com]] [sdap_ldap_connect_callback_add]
(0x4000): [RID#50] New connection to [ldap://
ausdc16apac06.apac.corp.com:389/??base] with fd [27]
(2023-10-02 11:30:05): [be[amer.corp.com]] [sdap_get_rootdse_send]
(0x4000): [RID#50] Getting rootdse
--
(2023-10-02 11:30:05): [be[amer.corp.com]] [sdap_ldap_connect_callback_add]
(0x4000): [RID#51] New connection to [ldap://
AUSDC16JAPN02.japn.corp.com:389/??base] with fd [28]
(2023-10-02 11:30:05): [be[amer.corp.com]] [sdap_get_rootdse_send]
(0x4000): [RID#51] Getting rootdse
--
(2023-10-02 11:32:52): [be[amer.corp.com]] [sdap_ldap_connect_callback_add]
(0x4000): [RID#84] New connection to [ldap://
ausdc16emea05.emea.corp.com:389/??base] with fd [26]
(2023-10-02 11:32:52): [be[amer.corp.com]] [sdap_get_rootdse_send]
(0x4000): [RID#84] Getting rootdse
--
BTW, this seems to occurs on both RHEL7 and RHEL8. (Haven't looked at our
RHEL9 builds yet). It's occurring on all servers to all rootDSEs, but
only a problem for AMERAustin, since Austin is such a heavily-populated.
These rootDSEs change almost never. Any way to have it query not as
frequently, or randomize when servers query these rootDSEs.
Spike
On Mon, Oct 2, 2023 at 2:37 AM Alexey Tikhonov <atikhono(a)redhat.com> wrote:
Hi,
On Mon, Oct 2, 2023 at 6:20 AM Spike White <spikewhitetx(a)gmail.com> wrote:
> All,
>
> Is there anything in sssd's RHEL and RHEL-like Linux server OS settings
> that perform LDAP binds or connections to AD every 30 minutes?
>
> What our AD team is seeing is all of the DCs in our biggest AMER AD site
> peak with LDAP sessions for about 10 minutes at the top of the hour then
> again at the bottom of the hour. No other AD site in the world appears to
> see this behavior not even other AD sites in this metro area.
>
> The reason they noticed is that our non-amer DCs in this biggest AD site
> hit their 5k LDAP client session limit during those 10 minutes every 30
> minutes. Meaning any clients attempting to establish a LDAP session past
> 5000 are dropped by the DC. In their research they see thousands LDAP
> Binds by RHEL Linux servers against two specific non-AMER AD DCs in a short
> period of time after digging through some LDAP log samples that they pulled
> from these DCs.
>
Can they also say what operations are being performed by those connections?
Or can you check SSSD logs on the client side?
I wonder if this could be `ldap_sudo_smart_refresh_interval`...
>
> In this major AD sites, we have dozens and dozens of AMER AD DCs. So
> there's enough preferred AD DCs to spread the load. But typically for the
> non-AMER regions, the AD team puts 2 of each regions DCs in a site. For
> instance, for APAC they would be put two APAC DCs in this AMER major site.
> Thus all AMER RHEL servers in this site would randomly hit dozens of AMER
> DCs, but concentrate on these two preferred APAC DCs. (preferred because
> they're in this locatiion).
>
> I know our older AD integration product used to hit AD every 30 mins to
> check GPOs, but we're not implementing GPOs with sssd.
>
> Spike
> _______________________________________________
> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
>
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahoste...
> Do not reply to spam, report it:
>
https://pagure.io/fedora-infrastructure/new_issue
>
_______________________________________________
sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahoste...
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue