Jeff wrote:

Thanks for the hint.  The master replica server was having issues.  It's been updated and is running now.  The question I have now is why would it stop working if the other two replicas were still functioning, especially since a reinstall of the client seems to fix it?

It probably depends on how the client was original configured (whether your client was using DNS discovery). I imagine it wasn't so SSSD didn't have, or couldn't find, another master to fall back on.

rob

On Thu, Oct 25, 2018 at 12:13 PM Rob Crittenden <rcritten@redhat.com mailto:rcritten@redhat.com> wrote:

Jeff Vincent via FreeIPA-users wrote:
> I inherited the management of our FreeIPA instance (master + 2
replicas).  Most of our clients are running Ubuntu 14.04 or
greater.   It is becoming an issue where only cached credentials
will work and any new users are unable to log in. 
>
> So far in all cases, if I unconfigure freeipa ('ipa-client-install
--uninstall') and then reinstall (using the --force-join) option, it
starts working again.  However, I want to get to the bottom of why
it is happening so frequently.  It's hard to pinpoint to a specific
time because the cached credentials work for a while.
>
> I've added:
>
>      debug_level = 0x3ff0
>
> to my /etc/sssd/sssd.conf file.  Once I restart sssd, it seems to
show that it is 'off-line'.  I've tried the following:
>
> - verified DNS is working and all 3 ipa server nodes can be resolved
> - verified the client 'hostname' is fully qualified
> - verified time is in sync
>
> I see this error a lot in the /var/log/syslog, but I also see it
on systems that work:
>      Oct 25 17:37:41 server [sssd[ldap_child[26118]]]: Failed to
initialize credentials using keytab [default]: Generic error (see
e-text). Unable to create GSSAPI-encrypted LDAP connection.
>      Oct 25 17:37:41 server [sssd[ldap_child[26118]]]: Generic
error (see e-text)
>
> I see this in the /var/log/sssd/ldap_child.log:
>     (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]]
[ldap_child_get_tgt_sync] (0x2000): Kerberos context initialized
>     (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]]
[ldap_child_get_tgt_sync] (0x2000): got realm_name:
[WP.MYCOMPANY.COM <http://WP.MYCOMPANY.COM>]
>     (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]]
[ldap_child_get_tgt_sync] (0x0100): Principal name is:
[host/server.mycompany.com@WP.MYCOMPANY.COM
<mailto:server.mycompany.com@WP.MYCOMPANY.COM>]
>     (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]]
[ldap_child_get_tgt_sync] (0x0100): Using keytab [default]
>     (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]]
[ldap_child_get_tgt_sync] (0x2000): keytab ccname:
[FILE:/var/lib/sss/db/ccache_WP.MYCOMPANY.COM_EhCBvz]
>     (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]]
[ldap_child_get_tgt_sync] (0x0100): Will canonicalize principals
>     (Thu Oct 25 17:40:22 2018) [[sssd[ldap_child[26163]]]]
[ldap_child_get_tgt_sync] (0x0010): Failed to init credentials:
Generic error (see e-text)
>
> The contents of /var/log/sssd/krb5_child.log
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [main]
(0x0400): krb5_child started.
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[unpack_buffer] (0x1000): total buffer size: [136]
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[unpack_buffer] (0x0100): cmd [241] uid [718000010] gid [718000010]
validate [true] enterprise principal [false] offline [false] UPN
[jeff.vincent@WP.MYCOMPANY.COM <mailto:jeff.vincent@WP.MYCOMPANY.COM>]
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[unpack_buffer] (0x0100): ccname:
[FILE:/tmp/krb5cc_718000010_5IZeI7] keytab: [/etc/krb5.keytab]
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[set_lifetime_options] (0x0100): Cannot read
[SSSD_KRB5_RENEWABLE_LIFETIME] from environment.
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[set_lifetime_options] (0x0100): Cannot read [SSSD_KRB5_LIFETIME]
from environment.
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[set_canonicalize_option] (0x0100): SSSD_KRB5_CANONICALIZE is set to
[true]
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[k5c_setup_fast] (0x0100): SSSD_KRB5_FAST_PRINCIPAL is set to
[host/server.mycompany.com@WP.MYCOMPANY.COM
<mailto:server.mycompany.com@WP.MYCOMPANY.COM>]
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[match_principal] (0x1000): Principal matched to the sample
(host/server.mycompany.com@WP.MYCOMPANY.COM
<mailto:server.mycompany.com@WP.MYCOMPANY.COM>).
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[set_canonicalize_option] (0x0100): SSSD_KRB5_CANONICALIZE is set to
[true]
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[get_and_save_tgt_with_keytab] (0x0020): 936: [-1765328324][Generic
error (see e-text)]
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[check_fast_ccache] (0x0020): get_and_save_tgt_with_keytab failed.
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[k5c_setup_fast] (0x0020): check_fast_ccache failed.
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]]
[k5c_setup_fast] (0x0020): 1812: [-1765328324][Generic error (see
e-text)]
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [main]
(0x0020): krb5_child_setup failed.
>     (Thu Oct 25 17:30:35 2018) [[sssd[krb5_child[26037]]]] [main]
(0x0020): krb5_child failed!
> 
> It *seems* to be kerberos related, but I have little working
knowledge of kerberos.
>
> Searches hinted at manually trying the kinit:
>
>     root@server:/var/log/sssd# kinit -V my.user
>     Using default cache: /tmp/krb5cc_718000010_5IZeI7
>     Using principal: my.user@WP.MYCOMPANY.COM
<mailto:my.user@WP.MYCOMPANY.COM>
>     kinit: Generic error (see e-text) while getting initial
credentials
>
> So the error seems consistent.  I don't know where to go from here.
>
> Any help is much appreciated.

Look in the KDC log on the IPA master(s) (/var/log/krb5kdc.log on
RHEL/Fedora) for clues.

rob