Hi all,
Let me introduce the symptoms that triggered this investigation.
First, versions:
- OS: Ubuntu 20.04
- SSSD: 2.2.3-3ubuntu0.7
- Platform: x86_64
Some weeks after starting using a new region/provider we noticed some intermittent issue impacting SSH logins (ie: clients got `Connection closed by UNKNOWN port 65535` on `ssh ...`). When looking into it, I was able to reproduce it somewhat reliably. It happens after some time of inactivity for that user, but if I constantly issue commands (ie: `watch sssctl user-checks <user>` or `watch ssh <host> whoami` from a client) the issue wont trigger. If I stopped trying out those commands and tried again ~30 minutes later, it would fail.
With these symptoms I had these possibilities in mind: some caching issue, some authentication expiring, connections pools issues...
I enabled detailed logging and compared logs from a successful request and a failed one and I found what seems to be the source of the error from sssd logs perspective, but still can't make much sense of it.
Log extracted from sssd_<domain>.log:
```
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [ipa_hbac_rule_info_next] (0x0400): Sending request for next search base: [cn=hbac,dc=ipa,dc=corp,<dc <domain>>][2][(&(objectclass=ipaHBACRule)(ipaenabledflag=TRUE)(accessRuleType=allow)(|(hostCategory=all)(...)))]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_print_server] (0x2000): Searching <ip10.1.10.102:389
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(objectclass=ipaHBACRule)(ipaenabledflag=TRUE)(accessRuleType=allow)(|(hostCategory=all)(...)][cn=hbac,dc=ipa,dc=corp,<dc <domain>>].
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [objectclass]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [cn]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [ipauniqueid]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [ipaenabledflag]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [accessRuleType]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberUser]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [userCategory]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberService]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [serviceCategory]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [sourceHost]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [sourceHostCategory]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [externalHost]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [memberHost]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x1000): Requesting attrs: [hostCategory]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_get_generic_ext_step] (0x2000): ldap_search_ext called, msgid = 18
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_op_add] (0x2000): New operation 18 timeout 60
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_process_result] (0x2000): Trace: sh[0x55d349f96d00], connected[1], ops[0x55d349ff16e0], ldap[0x55d34a10cba0]
(Wed Oct 13 19:40:52 2021) [be[<domain>]] [sdap_process_result] (0x2000): Trace: end of ldap_result list
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [sdap_op_timeout] (0x1000): Issuing timeout for 18
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [sdap_op_destructor] (0x1000): Abandoning operation 18
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [generic_ext_search_handler] (0x0040): sdap_get_generic_ext_recv failed [110]: Connection timed out
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [ipa_hbac_rule_info_done] (0x0080): Could not retrieve HBAC rules
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [ipa_pam_access_handler_done] (0x0020): Unable to fetch HBAC rules [110]: Connection timed out
(Wed Oct 13 19:41:52 2021) [be[<domain>]] [dp_req_done] (0x0400): DP Request [PAM Account #21]: Request handler finished [0]: Success
```
I tried using `ldap_enumeration_search_timeout = 180` which updated the timeout in (`[sdap_op_add] (0x2000): New operation 18 timeout 60 `) but same result (`enumerate` and `subdomain_enumerate` config options are left to their defaults, `false` and `none` respectively).
I looked at the IPA servers LDAP logs and saw now related requests regarding `ipaHBACRules` so I assume the request didn't even left the server.
I tried tweaking some other caching options like `cached_auth_timeout` and `pam_id_timeout` without any impact. Also tried clearing up the caches with `sss_cache -E` or `systemctl stop sssd && rm /var/lib/sss/db/* && systemctl start sssd` and nothing changed.
This is the error returned by `sssctl user-checks` when it fails:
```
$ sudo sssctl user-checks <user>
user: <user>
action: acct
service: system-auth
SSSD nss user lookup result:
- user name: <user>
- user id: <uid>
- group id: <gid>
- gecos: <name>
- home directory: /home/<user>
- shell: /bin/bash
InfoPipe operation failed. Check that SSSD is running and the InfoPipe responder is enabled. Make sure 'ifp' is listed in the 'services' option in sssd.conf.InfoPipe User lookup with [<user>] failed.
testing pam_acct_mgmt
pam_acct_mgmt: System error
PAM Environment:
- no env -
```
Any pointer here would be very useful.
Thank you!
Aitor