Full client log below. It turns out that what I was suspecting (issue with regards to the cross-realm TGT) was a step in the right direction. It turns out that my story is a bit more complicated indeed.
My IPA server has a direct trust relationship with another AD realm/domain (say "d2.domain" which has a bidirectional trust relationship with "ad.domain"). So far so good, however I learned this morning that our Windows guys replaced some outdated "ad2.domain" DCs with brand new ones some days ago and, for some obscure reason, the network security stuff had not been updated on my side. Consequently, my IPA server was unable to speak with the new DCs deserving "ad2.domain thus was also unable to process anything related to "ad.domain", including cross-realm TGTs implying that later AD realm/domain. Our network admins should fix that next week and everyting should come back in good working order. In the meantime not bug or a software configuration or a DNS issue. Re-doing the trust with "ad2.domain" would have directly pointed the problem ("Communication error").
==== [Start of /var/log/ipaclient-install.log (w/ sensitive info redacted) ] ====
2021-07-23T14:30:40Z DEBUG IPA version 4.6.8-5.el7.centos.6
2021-07-23T14:30:40Z DEBUG Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
2021-07-23T14:30:40Z DEBUG Starting external process
2021-07-23T14:30:40Z DEBUG args=/usr/sbin/selinuxenabled
2021-07-23T14:30:40Z DEBUG Process finished, return code=0
2021-07-23T14:30:40Z DEBUG stdout=
2021-07-23T14:30:40Z DEBUG stderr=
2021-07-23T14:30:40Z DEBUG [IPA Discovery]
2021-07-23T14:30:40Z DEBUG Starting IPA discovery with domain=ipa.domain, servers=['idmsrv01.ad.domain'], hostname=clientvm01.ad.domain
2021-07-23T14:30:40Z DEBUG Server and domain forced
2021-07-23T14:30:40Z DEBUG [Kerberos realm search]
2021-07-23T14:30:40Z DEBUG Kerberos realm forced
2021-07-23T14:30:40Z DEBUG [LDAP server check]
2021-07-23T14:30:40Z DEBUG Verifying that idmsrv01.ad.domain (realm IPA.DOMAIN) is an IPA server
2021-07-23T14:30:40Z DEBUG Init LDAP connection to: ldap://idmsrv01.ad.domain:389
2021-07-23T14:30:40Z DEBUG Search LDAP server for IPA base DN
2021-07-23T14:30:40Z DEBUG Check if naming context 'dc=ipa,dc=domain' is for IPA
2021-07-23T14:30:40Z DEBUG Naming context 'dc=ipa,dc=domain' is a valid IPA context
2021-07-23T14:30:40Z DEBUG Search for (objectClass=krbRealmContainer) in dc=ipa,dc=domain (sub)
2021-07-23T14:30:40Z DEBUG Found: cn=IPA.DOMAIN,cn=kerberos,dc=ipa,dc=domain
2021-07-23T14:30:40Z DEBUG Discovery result: Success; server=idmsrv01.ad.domain, domain=ipa.domain, kdc=idmsrv01.ad.domain, basedn=dc=ipa,dc=domain
2021-07-23T14:30:40Z DEBUG Validated servers: idmsrv01.ad.domain
2021-07-23T14:30:40Z DEBUG will use discovered domain: ipa.domain
2021-07-23T14:30:40Z DEBUG Using servers from command line, disabling DNS discovery
2021-07-23T14:30:40Z DEBUG will use provided server: idmsrv01.ad.domain
2021-07-23T14:30:40Z INFO Autodiscovery of servers for failover cannot work with this configuration.
2021-07-23T14:30:40Z INFO If you proceed with the installation, services will be configured to always access the discovered server for all operations and will not fail over to other servers in case of failure.
2021-07-23T14:30:41Z DEBUG will use discovered realm: IPA.DOMAIN
2021-07-23T14:30:41Z DEBUG will use discovered basedn: dc=ipa,dc=domain
2021-07-23T14:30:41Z INFO Client hostname: clientvm01.ad.domain
2021-07-23T14:30:41Z DEBUG Hostname source: Machine's FQDN
2021-07-23T14:30:41Z INFO Realm: IPA.DOMAIN
2021-07-23T14:30:41Z DEBUG Realm source: Discovered from LDAP DNS records in idmsrv01.ad.domain
2021-07-23T14:30:41Z INFO DNS Domain: ipa.domain
2021-07-23T14:30:41Z DEBUG DNS Domain source: Forced
2021-07-23T14:30:41Z INFO IPA Server: idmsrv01.ad.domain
2021-07-23T14:30:41Z DEBUG IPA Server source: Provided as option
2021-07-23T14:30:41Z INFO BaseDN: dc=ipa,dc=domain
2021-07-23T14:30:41Z DEBUG BaseDN source: From IPA server ldap://idmsrv01.ad.domain:389
2021-07-23T14:30:48Z DEBUG Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
2021-07-23T14:30:48Z DEBUG Loading StateFile from '/var/lib/ipa-client/sysrestore/sysrestore.state'
2021-07-23T14:30:48Z DEBUG Starting external process
2021-07-23T14:30:48Z DEBUG args=/usr/sbin/ipa-rmkeytab -k /etc/krb5.keytab -r IPA.DOMAIN
2021-07-23T14:30:48Z DEBUG Process finished, return code=5
2021-07-23T14:30:48Z DEBUG stdout=
2021-07-23T14:30:48Z DEBUG stderr=realm not found
2021-07-23T14:30:48Z INFO Skipping synchronizing time with NTP server.
2021-07-23T14:30:48Z DEBUG Starting external process
2021-07-23T14:30:48Z DEBUG args=/usr/bin/keyctl get_persistent @s 0
2021-07-23T14:30:48Z DEBUG Process finished, return code=0
2021-07-23T14:30:48Z DEBUG stdout=96233181
2021-07-23T14:30:48Z DEBUG stderr=
2021-07-23T14:30:48Z DEBUG Enabling persistent keyring CCACHE
2021-07-23T14:30:48Z DEBUG Writing Kerberos configuration to /tmp/tmpO564dk:
2021-07-23T14:30:48Z DEBUG #File modified by ipa-client-install
includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/
[libdefaults]
default_realm = IPA.DOMAIN
dns_lookup_realm = false
dns_lookup_kdc = false
rdns = false
dns_canonicalize_hostname = false
ticket_lifetime = 24h
forwardable = true
udp_preference_limit = 0
default_ccache_name = KEYRING:persistent:%{uid}
[realms]
IPA.DOMAIN = {
kdc = idmsrv01.ad.domain:88
master_kdc = idmsrv01.ad.domain:88
admin_server = idmsrv01.ad.domain:749
kpasswd_server = idmsrv01.ad.domain:464
default_domain = ipa.domain
pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
}
[domain_realm]
.ipa.domain = IPA.DOMAIN
ipa.domain = IPA.DOMAIN
clientvm01.ad.domain = IPA.DOMAIN
.ad.domain = IPA.DOMAIN
ad.domain = IPA.DOMAIN
2021-07-23T14:30:52Z DEBUG Initializing principal enrollmentaccount@IPA.DOMAIN using password
2021-07-23T14:30:52Z DEBUG Starting external process
2021-07-23T14:30:52Z DEBUG args=/usr/bin/kinit enrollmentaccount@IPA.DOMAIN -c /tmp/krbccQTje0m/ccache
2021-07-23T14:30:52Z DEBUG Process finished, return code=0
2021-07-23T14:30:52Z DEBUG stdout=Password for enrollmentaccount@IPA.DOMAIN:
2021-07-23T14:30:52Z DEBUG stderr=
2021-07-23T14:30:52Z DEBUG trying to retrieve CA cert via LDAP from idmsrv01.ad.domain
2021-07-23T14:30:53Z DEBUG get_ca_certs_from_ldap() error: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/AD.DOMAIN@IPA.DOMAIN not found in Kerberos database)
2021-07-23T14:30:53Z DEBUG Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/AD.DOMAIN@IPA.DOMAIN not found in Kerberos database)
2021-07-23T14:30:57Z WARNING Downloading the CA certificate via HTTP, this is INSECURE
2021-07-23T14:30:57Z DEBUG trying to retrieve CA cert via HTTP from
http://idmsrv01.ad.domain/ipa/config/ca.crt2021-07-23T14:30:57Z DEBUG Starting external process
2021-07-23T14:30:57Z DEBUG args=/usr/bin/curl -o -
http://idmsrv01.ad.domain/ipa/config/ca.crt2021-07-23T14:30:57Z DEBUG Process finished, return code=0
2021-07-23T14:30:57Z DEBUG stdout=-----BEGIN CERTIFICATE-----
**[PEM CERTIFICATES REDACTED]**
-----END CERTIFICATE-----
2021-07-23T14:30:57Z DEBUG stderr= % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M100 7754 100 7754 0 0 1056k 0 --:--:-- --:--:-- --:--:-- 1081k
2021-07-23T14:30:57Z INFO Successfully retrieved CA cert
Subject: CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB
Issuer: CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB
Valid From: 2004-01-01 00:00:00
Valid Until: 2028-12-31 23:59:59
Subject: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
Issuer: CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB
Valid From: 2019-03-12 00:00:00
Valid Until: 2028-12-31 23:59:59
Subject: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
Issuer: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
Valid From: 2010-02-01 00:00:00
Valid Until: 2038-01-18 23:59:59
Subject: CN=Sectigo RSA Organization Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB
Issuer: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
Valid From: 2018-11-02 00:00:00
Valid Until: 2030-12-31 23:59:59
2021-07-23T14:30:57Z DEBUG Starting external process
2021-07-23T14:30:57Z DEBUG args=/usr/sbin/ipa-join -s idmsrv01.ad.domain -b dc=ipa,dc=domain -h clientvm01.ad.domain -f
2021-07-23T14:30:57Z DEBUG Process finished, return code=17
2021-07-23T14:30:57Z DEBUG stdout=
2021-07-23T14:30:57Z DEBUG stderr=HTTP response code is 401, not 200
2021-07-23T14:30:57Z ERROR Joining realm failed: HTTP response code is 401, not 200
2021-07-23T14:30:57Z INFO Use ipa-getkeytab to obtain a host principal for this server.
2021-07-23T14:30:57Z DEBUG Starting external process
2021-07-23T14:30:57Z DEBUG args=/usr/bin/kdestroy
2021-07-23T14:30:57Z DEBUG Process finished, return code=0
2021-07-23T14:30:57Z DEBUG stdout=
2021-07-23T14:30:57Z DEBUG stderr=
2021-07-23T14:30:57Z DEBUG Initializing principal host/clientvm01.ad.domain@IPA.DOMAIN using keytab /etc/krb5.keytab
2021-07-23T14:30:57Z DEBUG using ccache /etc/ipa/.dns_ccache
2021-07-23T14:30:57Z INFO Please make sure the following ports are opened in the firewall settings:
TCP: 80, 88, 389
UDP: 88 (at least one of TCP/UDP ports 88 has to be open)
Also note that following ports are necessary for ipa-client working properly after enrollment:
TCP: 464
UDP: 464, 123 (if NTP enabled)
2021-07-23T14:30:57Z ERROR Failed to obtain host TGT: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639107): No credentials cache found
2021-07-23T14:30:57Z WARNING Installation failed. Force set so not rolling back changes.
2021-07-23T14:30:57Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run
return cfgr.run()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 360, in run
return self.execute()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 386, in execute
for rval in self._executor():
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
exc_handler(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner
step()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 655, in _configure
next(executor)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
exc_handler(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 515, in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner
step()
File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 65, in _install
for unused in self._installer(self.parent):
File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line 3670, in main
install(self)
File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line 2391, in install
_install(options)
File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line 2644, in _install
raise ScriptError(rval=CLIENT_INSTALL_ERROR)
2021-07-23T14:30:57Z DEBUG The ipa-client-install command failed, exception: ScriptError:
2021-07-23T14:30:57Z ERROR The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information