Sssd experts,
We have a problem with two RHEL7 servers, where one particular account is
logged into a great many times, fairly frequently. Can be as much as 2-3
concurrent sessions, but generally it’s just a bunch of sequential logins.
All from that one account.
We have just converted this user authentication to sssd (AD back-end) from
a non-Kerberos-based user authentication.
After an hour or so, the account can no longer log in. It reports an
error from pam_sss.so.
It says this in /var/log/secure:
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_sss(sshd:auth): authentication
failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11
user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_sss(sshd:auth): received for
user svc_npesrsnonprd: 4 (System error)
Oct 15 13:53:36 esrmfgstg01 sshd[19126]: pam_unix(sshd:auth):
authentication failure; logname= uid=0 euid=0 tty=ssh ruser=
rhost=10.186.154.11 user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19125]: pam_sss(sshd:auth): authentication
failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.186.154.11
user=svc_npesrsnonprd
Oct 15 13:53:36 esrmfgstg01 sshd[19125]: pam_sss(sshd:auth): received for
user svc_npesrsnonprd: 4 (System error)
In /var/log/sssd/sssd_pam.log file we see:
(Thu Oct 15 13:53:36 2020) [sssd[pam]] [pam_dp_process_reply] (0x0200):
received: [4 (System
error)][amer.dell.com]
In /var/log/sssd/sssd_amer.dell.com.log, we see:
(Thu Oct 15 13:53:36 2020) [sssd[be[amer.dell.com]]] [read_pipe_handler]
(0x0400): EOF received, client finished
(Thu Oct 15 13:53:36 2020) [sssd[be[amer.dell.com]]] [krb5_auth_done]
(0x0040): The krb5_child process returned an error. Please inspect the
krb5_child.log file or the journal for more information
And finally, in /var/log/sssd/krb5_child.log, we see:
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache]
(0x4000): Initializing ccache of type [KEYRING]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache]
(0x4000): CC supports switch
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache]
(0x4000): Match not found
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache]
(0x4000): krb5_cc_cache_match failed
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [create_ccache]
(0x0020): 1016: [122][Disk quota exceeded]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [map_krb5_error]
(0x0020): 1817: [122][Disk quota exceeded]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [k5c_send_data]
(0x0200): Received error code 1432158209
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]]
[pack_response_packet] (0x2000): response packet size: [4]
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [k5c_send_data]
(0x4000): Response sent.
(Thu Oct 15 13:53:36 2020) [[sssd[krb5_child[19139]]]] [main] (0x0400):
krb5_child completed successfully
So we think the problem is that all the keyrings are consumed, or this user
has exceeded its quota of keyrings. Faster than Kerberos garbage collects
them.
In googling, we see a similar problem but it’s specific to GNOME.
https://bugzilla.redhat.com/show_bug.cgi?id=1230750
Our problem does not involve GNOME, just raw SSH connections.
We bumped up */proc/sys/kernel/keys/maxkeys *(from default of 200 to
1000) and it
seemed to help. Just wondering what is the correct fix? And if this
is the correct
fix, some guidance as to best value for maxkeys.
BTW, even when we specify :
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
in /etc/kr5.conf, sssd’s krb5_child.log seems to want to use KEYRING.
What’s up
with that?
Spike