We're seeing problems with some hosts which appears to be caused by the new ad_maximum_machine_account_password_age support in sssd (which notes that a recent adcli is required for this functionality).

For some hosts, there is one set of credentials in krb5.keytab, and AD indicates that the machine account password was changed 30 days after the timestamp on those credentials.  On other hosts, we see a second set of credentials,
30 days newer than the first set, whose timestamp matches AD's PasswordLastSet property for that host.  For the latter set, users report that authentication works some of the time.

It may be relevant that this is a large multi-site domain, and we've also found that some hosts have been set up with "ad_site" specified in sssd.conf, pointing at a site that is incorrect (in that it is not the closest site).

Both sssd and adcli are out of date on the hosts we're troubleshooting, but I don't see any bugzilla entries, mailing list topics, or release notes that indicate that this is a known problem that has been fixed.

Which components should I be looking at when diagnosing this problem further, and what log settings might be most useful as I'm hunting down the problem?



Symptoms for host1:

System keytab contains old credentials.

    [root@host1 ~]# klist -kt /etc/krb5.keytab
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Timestamp           Principal
    ---- ------------------- ------------------------------------------------------
       4 05/04/2018 11:10:30 host/host1.domain.lan@DOMAIN.LAN
       4 05/04/2018 11:10:30 host/HOST1@DOMAIN.LAN
       4 05/04/2018 11:10:30 host/host1.domain.lan@DOMAIN.LAN
       4 05/04/2018 11:10:30 host/HOST1@DOMAIN.LAN
       4 05/04/2018 11:10:30 host/host1.domain.lan@DOMAIN.LAN
       4 05/04/2018 11:10:30 host/HOST1@DOMAIN.LAN
       4 05/04/2018 11:10:30 host/host1.domain.lan@DOMAIN.LAN
       4 05/04/2018 11:10:30 host/HOST1@DOMAIN.LAN
       4 05/04/2018 11:10:30 host/host1.domain.lan@DOMAIN.LAN
       4 05/04/2018 11:10:30 host/HOST1@DOMAIN.LAN
       4 05/04/2018 11:10:30 HOST1$@DOMAIN.LAN
       4 05/04/2018 11:10:30 HOST1$@DOMAIN.LAN
       4 05/04/2018 11:10:30 HOST1$@DOMAIN.LAN
       4 05/04/2018 11:10:30 HOST1$@DOMAIN.LAN
       4 05/04/2018 11:10:30 HOST1$@DOMAIN.LAN

The date of the new credentials matches the “PasswordLastSet” property on that host in AD:

    PS H:\> Get-ADComputer -Identity host1 -Properties name,LastLogonDate,PasswordLastSet,modified,modifyTimeStamp

    DistinguishedName : CN=HOST1,OU=Build,OU=DOMAIN_DevIT,DC=domain,DC=lan
    DNSHostName       : host1.domain.lan
    Enabled           : True
    LastLogonDate     : 6/1/2018 10:41:59 AM
    Modified          : 6/3/2018 3:27:10 PM
    modifyTimeStamp   : 6/3/2018 3:27:10 PM
    Name              : HOST1
    ObjectClass       : computer
    ObjectGUID        : d21d7f72-xxxx-xxxx-xxxx-aac9cf895d31
    PasswordLastSet   : 6/3/2018 3:26:59 PM
    SamAccountName    : HOST1$
    SID               : S-1-5-21-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxx-xxxxxx
    UserPrincipalName :
    UserPrincipalName :

adcli and sssd are out of date on that host:

    [root@host1 ~]# rpm -q adcli sssd
    adcli-0.8.1-3.el7.x86_64
    sssd-1.15.2-50.el7_4.11.x86_64

The current versions are:

    $ yum list adcli sssd
    …
    Installed Packages
    sssd.x86_64                     1.16.0-19.el7                @base
    Available Packages
    adcli.x86_64                    0.8.1-4.el7                  base

Logs include:

    [sssd[ldap_child[1271]]]: Failed to initialize credentials using keytab [MEMORY:/etc/krb5.keytab]: Preauthentication failed. Unable to create GSSAPI-encrypted LDAP connection.
    [sssd[ldap_child[1271]]]: Preauthentication failed



Symptoms for host2:

System keytab contains new credentials, 30 days newer than a previous set.

    [it@host2 ~]$ sudo klist -kt /etc/krb5.keytab
    [sudo] password for it:
    Keytab name: FILE:/etc/krb5.keytab
    KVNO Timestamp           Principal
    ---- ------------------- ------------------------------------------------------
      3 05/21/2018 11:50:41 host/host2.domain.lan@DOMAIN.LAN
      3 05/21/2018 11:50:41 host/host2.domain.lan@DOMAIN.LAN
      3 05/21/2018 11:50:41 host/host2.domain.lan@DOMAIN.LAN
      3 05/21/2018 11:50:41 host/host2.domain.lan@DOMAIN.LAN
      3 05/21/2018 11:50:41 host/host2.domain.lan@DOMAIN.LAN
      3 05/21/2018 11:50:41 host/HOST2@DOMAIN.LAN
      3 05/21/2018 11:50:41 host/HOST2@DOMAIN.LAN
      3 05/21/2018 11:50:41 host/HOST2@DOMAIN.LAN
      3 05/21/2018 11:50:41 host/HOST2@DOMAIN.LAN
      3 05/21/2018 11:50:41 host/HOST2@DOMAIN.LAN
      3 05/21/2018 11:50:41 HOST2$@DOMAIN.LAN
      3 05/21/2018 11:50:41 HOST2$@DOMAIN.LAN
      3 05/21/2018 11:50:41 HOST2$@DOMAIN.LAN
      3 05/21/2018 11:50:41 HOST2$@DOMAIN.LAN
      3 05/21/2018 11:50:41 HOST2$@DOMAIN.LAN
      4 06/21/2018 08:09:02 HOST2$@DOMAIN.LAN
      4 06/21/2018 08:09:02 HOST2$@DOMAIN.LAN
      4 06/21/2018 08:09:02 HOST2$@DOMAIN.LAN
      4 06/21/2018 08:09:02 HOST2$@DOMAIN.LAN
      4 06/21/2018 08:09:02 HOST2$@DOMAIN.LAN
      4 06/21/2018 08:09:02 host/host2.domain.lan@DOMAIN.LAN
      4 06/21/2018 08:09:02 host/host2.domain.lan@DOMAIN.LAN
      4 06/21/2018 08:09:02 host/host2.domain.lan@DOMAIN.LAN
      4 06/21/2018 08:09:02 host/host2.domain.lan@DOMAIN.LAN
      4 06/21/2018 08:09:02 host/host2.domain.lan@DOMAIN.LAN
      4 06/21/2018 08:09:02 host/HOST2@DOMAIN.LAN
      4 06/21/2018 08:09:02 host/HOST2@DOMAIN.LAN
      4 06/21/2018 08:09:02 host/HOST2@DOMAIN.LAN
      4 06/21/2018 08:09:02 host/HOST2@DOMAIN.LAN
      4 06/21/2018 08:09:02 host/HOST2@DOMAIN.LAN


The date of the new credentials matches the “PasswordLastSet” property on that host in AD:

    PS H:\> Get-ADComputer -Identity host2 -Properties name,LastLogonDate,PasswordLastSet,modified,modifyTimeStamp

    DistinguishedName : CN=HOST2,OU=Build,OU=DOMAIN_DevIT,DC=domain,DC=lan
    DNSHostName       : host2.domain.lan
    Enabled           : True
    LastLogonDate     : 7/6/2018 10:51:11 PM
    Modified          : 7/6/2018 10:52:06 PM
    modifyTimeStamp   : 7/6/2018 10:52:06 PM
    Name              : HOST2
    ObjectClass       : computer
    ObjectGUID        : f70a3bab-xxxx-xxxx-xxxx-932457b18768
    PasswordLastSet   : 6/21/2018 8:09:02 AM
    SamAccountName    : HOST2$
    SID               : S-1-5-21-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxx-xxxxxx
    UserPrincipalName :

adcli and sssd are out of date on that host:

    [root@host2 ~]# rpm -q adcli sssd
    adcli-0.8.1-3.el7.x86_64
    sssd-1.14.0-43.el7_3.14.x86_64

The current versions are:

    $ yum list adcli sssd
    …
    Installed Packages
    sssd.x86_64                 1.16.0-19.el7           @base
    Available Packages
    adcli.x86_64                0.8.1-4.el7             base

Logs include:

    [sssd[krb5_child[21580]]]: Preauthentication failed