New subject: FreeIPA stops working on nodes ... need help debugging.

25 Oct 2018


      I inherited the management of our FreeIPA instance (master + 2 replicas).  Most of our clients are running Ubuntu 14.04 or greater.   It is becoming an issue where only cached credentials will work and any new users are unable to log in.
So far in all cases, if I unconfigure freeipa ('ipa-client-install --uninstall') and then reinstall (using the --force-join) option, it starts working again.  However, I want to get to the bottom of why it is happening so frequently.  It's hard to pinpoint to a specific time because the cached credentials work for a while.
I've added:
debug_level = 0x3ff0
to my /etc/sssd/sssd.conf file.  Once I restart sssd, it seems to show that it is 'off-line'.  I've tried the following:
- verified DNS is working and all 3 ipa server nodes can be resolved
- verified the client 'hostname' is fully qualified
- verified time is in sync
I see this error a lot in the /var/log/syslog, but I also see it on systems that work:
     Oct 25 17:37:41 server [sssd[ldap_child[26118]]]: Failed to initialize credentials using keytab [default]: Generic error (see e-text). Unable to create GSSAPI-encrypted LDAP connection.
     Oct 25 17:37:41 server [sssd[ldap_child[26118]]]: Generic error (see e-text)
I see this in the /var/log/sssd/ldap_child.log:
    (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]] [ldap_child_get_tgt_sync] (0x2000): Kerberos context initialized
    (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]] [ldap_child_get_tgt_sync] (0x2000): got realm_name: [WP.MYCOMPANY.COM]
    (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]] [ldap_child_get_tgt_sync] (0x0100): Principal name is: [host/server.mycompany.com@WP.MYCOMPANY.COM]
    (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]] [ldap_child_get_tgt_sync] (0x0100): Using keytab [default]
    (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]] [ldap_child_get_tgt_sync] (0x2000): keytab ccname: [FILE:/var/lib/sss/db/ccache_WP.MYCOMPANY.COM_EhCBvz]
    (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]] [ldap_child_get_tgt_sync] (0x0100): Will canonicalize principals
    (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]] [ldap_child_get_tgt_sync] (0x0010): Failed to init credentials: Generic error (see e-text)
The contents of /var/log/sssd/krb5_child.log
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [main] (0x0400): krb5_child started.
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [unpack_buffer] (0x1000): total buffer size: [136]
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [unpack_buffer] (0x0100): cmd [241] uid [718000010] gid [718000010] validate [true] enterprise principal [false] offline [false] UPN [jeff.vincent@WP.MYCOMPANY.COM]
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [unpack_buffer] (0x0100): ccname: [FILE:/tmp/krb5cc_718000010_5IZeI7] keytab: [/etc/krb5.keytab]
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_RENEWABLE_LIFETIME] from environment.
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_LIFETIME] from environment.
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [set_canonicalize_option] (0x0100): SSSD_KRB5_CANONICALIZE is set to [true]
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [k5c_setup_fast] (0x0100): SSSD_KRB5_FAST_PRINCIPAL is set to [host/server.mycompany.com@WP.MYCOMPANY.COM]
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [match_principal] (0x1000): Principal matched to the sample (host/server.mycompany.com@WP.MYCOMPANY.COM).
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [set_canonicalize_option] (0x0100): SSSD_KRB5_CANONICALIZE is set to [true]
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [get_and_save_tgt_with_keytab] (0x0020): 936: [-1765328324][Generic error (see e-text)]
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [check_fast_ccache] (0x0020): get_and_save_tgt_with_keytab failed.
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [k5c_setup_fast] (0x0020): check_fast_ccache failed.
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [k5c_setup_fast] (0x0020): 1812: [-1765328324][Generic error (see e-text)]
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [main] (0x0020): krb5_child_setup failed.
    (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [main] (0x0020): krb5_child failed!
It *seems* to be kerberos related, but I have little working knowledge of kerberos.
Searches hinted at manually trying the kinit:
root@server:/var/log/sssd# kinit -V my.user
    Using default cache: /tmp/krb5cc_718000010_5IZeI7
    Using principal: my.user@WP.MYCOMPANY.COM
    kinit: Generic error (see e-text) while getting initial credentials
So the error seems consistent.  I don't know where to go from here.
Any help is much appreciated.
-Jeff