Full client log below. It turns out that what I was suspecting (issue with regards to the cross-realm TGT) was a step in the right direction. It turns out that my story is a bit more complicated indeed.

My IPA server has a direct trust relationship with another AD realm/domain (say "d2.domain" which has a bidirectional trust relationship with "ad.domain"). So far so good, however I learned this morning that our Windows guys replaced some outdated "ad2.domain" DCs with brand new ones some days ago and, for some obscure reason, the network security stuff had not been updated on my side. Consequently,  my IPA server was unable to speak with the new DCs deserving "ad2.domain thus was also unable to process anything related to "ad.domain", including cross-realm TGTs implying that later AD realm/domain. Our network admins should fix that next week and everyting should come back in good working order. In the meantime not bug or a software configuration or a DNS issue.  Re-doing the trust with "ad2.domain" would have directly pointed the problem ("Communication error").

Thank you Rob for taking the time to investigate my concern.



==== [Start of /var/log/ipaclient-install.log  (w/ sensitive info redacted) ] ====
2021-07-23T14:30:40Z DEBUG Logging to /var/log/ipaclient-install.log
2021-07-23T14:30:40Z DEBUG ipa-client-install was invoked with arguments [] and options: {'no_dns_sshfp': False, 'force': True, 'verbose': False, 'ip_addresses': None, 'configure_firefox': False, 'realm_name': 'IPA.DOMAIN', 'force_ntpd': False, 'on_master': False, 'no_nisdomain': False, 'ssh_trust_dns': False, 'principal': 'ul-val-s-enroll', 'keytab': None, 'no_ntp': True, 'domain_name': ipa.domain', 'request_cert': False, 'fixed_primary': False, 'no_ac': False, 'no_sudo': False, 'ca_cert_files': None, 'all_ip_addresses': False, 'kinit_attempts': None, 'ntp_servers': None, 'enable_dns_updates': False, 'no_sshd': False, 'no_sssd': False, 'no_krb5_offline_passwords': False, 'servers': ['idmsrv01.ad.domain'], 'no_ssh': False, 'force_join': True, 'firefox_dir': None, 'unattended': False, 'quiet': False, 'nisdomain': None, 'prompt_password': False, 'host_name': None, 'permit': False, 'automount_location': None, 'preserve_sssd': False, 'mkhomedir': True, 'log_file': None, 'uninstall': False}
2021-07-23T14:30:40Z DEBUG IPA version 4.6.8-5.el7.centos.6
2021-07-23T14:30:40Z DEBUG Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
2021-07-23T14:30:40Z DEBUG Starting external process
2021-07-23T14:30:40Z DEBUG args=/usr/sbin/selinuxenabled
2021-07-23T14:30:40Z DEBUG Process finished, return code=0
2021-07-23T14:30:40Z DEBUG stdout=
2021-07-23T14:30:40Z DEBUG stderr=
2021-07-23T14:30:40Z DEBUG [IPA Discovery]
2021-07-23T14:30:40Z DEBUG Starting IPA discovery with domain=ipa.domain, servers=['idmsrv01.ad.domain'], hostname=clientvm01.ad.domain
2021-07-23T14:30:40Z DEBUG Server and domain forced
2021-07-23T14:30:40Z DEBUG [Kerberos realm search]
2021-07-23T14:30:40Z DEBUG Kerberos realm forced
2021-07-23T14:30:40Z DEBUG [LDAP server check]
2021-07-23T14:30:40Z DEBUG Verifying that idmsrv01.ad.domain (realm IPA.DOMAIN) is an IPA server
2021-07-23T14:30:40Z DEBUG Init LDAP connection to: ldap://idmsrv01.ad.domain:389
2021-07-23T14:30:40Z DEBUG Search LDAP server for IPA base DN
2021-07-23T14:30:40Z DEBUG Check if naming context 'dc=ipa,dc=domain' is for IPA
2021-07-23T14:30:40Z DEBUG Naming context 'dc=ipa,dc=domain' is a valid IPA context
2021-07-23T14:30:40Z DEBUG Search for (objectClass=krbRealmContainer) in dc=ipa,dc=domain (sub)
2021-07-23T14:30:40Z DEBUG Found: cn=IPA.DOMAIN,cn=kerberos,dc=ipa,dc=domain
2021-07-23T14:30:40Z DEBUG Discovery result: Success; server=idmsrv01.ad.domain, domain=ipa.domain, kdc=idmsrv01.ad.domain, basedn=dc=ipa,dc=domain
2021-07-23T14:30:40Z DEBUG Validated servers: idmsrv01.ad.domain
2021-07-23T14:30:40Z DEBUG will use discovered domain: ipa.domain
2021-07-23T14:30:40Z DEBUG Using servers from command line, disabling DNS discovery
2021-07-23T14:30:40Z DEBUG will use provided server: idmsrv01.ad.domain
2021-07-23T14:30:40Z INFO Autodiscovery of servers for failover cannot work with this configuration.
2021-07-23T14:30:40Z INFO If you proceed with the installation, services will be configured to always access the discovered server for all operations and will not fail over to other servers in case of failure.
2021-07-23T14:30:41Z DEBUG will use discovered realm: IPA.DOMAIN
2021-07-23T14:30:41Z DEBUG will use discovered basedn: dc=ipa,dc=domain
2021-07-23T14:30:41Z INFO Client hostname: clientvm01.ad.domain
2021-07-23T14:30:41Z DEBUG Hostname source: Machine's FQDN
2021-07-23T14:30:41Z INFO Realm: IPA.DOMAIN
2021-07-23T14:30:41Z DEBUG Realm source: Discovered from LDAP DNS records in idmsrv01.ad.domain
2021-07-23T14:30:41Z INFO DNS Domain: ipa.domain
2021-07-23T14:30:41Z DEBUG DNS Domain source: Forced
2021-07-23T14:30:41Z INFO IPA Server: idmsrv01.ad.domain
2021-07-23T14:30:41Z DEBUG IPA Server source: Provided as option
2021-07-23T14:30:41Z INFO BaseDN: dc=ipa,dc=domain
2021-07-23T14:30:41Z DEBUG BaseDN source: From IPA server ldap://idmsrv01.ad.domain:389
2021-07-23T14:30:48Z DEBUG Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
2021-07-23T14:30:48Z DEBUG Loading StateFile from '/var/lib/ipa-client/sysrestore/sysrestore.state'
2021-07-23T14:30:48Z DEBUG Starting external process
2021-07-23T14:30:48Z DEBUG args=/usr/sbin/ipa-rmkeytab -k /etc/krb5.keytab -r IPA.DOMAIN
2021-07-23T14:30:48Z DEBUG Process finished, return code=5
2021-07-23T14:30:48Z DEBUG stdout=
2021-07-23T14:30:48Z DEBUG stderr=realm not found

2021-07-23T14:30:48Z INFO Skipping synchronizing time with NTP server.
2021-07-23T14:30:48Z DEBUG Starting external process
2021-07-23T14:30:48Z DEBUG args=/usr/bin/keyctl get_persistent @s 0
2021-07-23T14:30:48Z DEBUG Process finished, return code=0
2021-07-23T14:30:48Z DEBUG stdout=96233181

2021-07-23T14:30:48Z DEBUG stderr=
2021-07-23T14:30:48Z DEBUG Enabling persistent keyring CCACHE
2021-07-23T14:30:48Z DEBUG Writing Kerberos configuration to /tmp/tmpO564dk:
2021-07-23T14:30:48Z DEBUG #File modified by ipa-client-install

includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/

[libdefaults]
  default_realm = IPA.DOMAIN
  dns_lookup_realm = false
  dns_lookup_kdc = false
  rdns = false
  dns_canonicalize_hostname = false
  ticket_lifetime = 24h
  forwardable = true
  udp_preference_limit = 0
  default_ccache_name = KEYRING:persistent:%{uid}


[realms]
  IPA.DOMAIN = {
    kdc = idmsrv01.ad.domain:88
    master_kdc = idmsrv01.ad.domain:88
    admin_server = idmsrv01.ad.domain:749
    kpasswd_server = idmsrv01.ad.domain:464
    default_domain = ipa.domain
    pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
    pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem

  }


[domain_realm]
  .ipa.domain = IPA.DOMAIN
  ipa.domain = IPA.DOMAIN
  clientvm01.ad.domain = IPA.DOMAIN
  .ad.domain = IPA.DOMAIN
  ad.domain = IPA.DOMAIN



2021-07-23T14:30:52Z DEBUG Initializing principal enrollmentaccount@IPA.DOMAIN using password
2021-07-23T14:30:52Z DEBUG Starting external process
2021-07-23T14:30:52Z DEBUG args=/usr/bin/kinit enrollmentaccount@IPA.DOMAIN -c /tmp/krbccQTje0m/ccache
2021-07-23T14:30:52Z DEBUG Process finished, return code=0
2021-07-23T14:30:52Z DEBUG stdout=Password for enrollmentaccount@IPA.DOMAIN:

2021-07-23T14:30:52Z DEBUG stderr=
2021-07-23T14:30:52Z DEBUG trying to retrieve CA cert via LDAP from idmsrv01.ad.domain
2021-07-23T14:30:53Z DEBUG get_ca_certs_from_ldap() error: Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/AD.DOMAIN@IPA.DOMAIN not found in Kerberos database)
2021-07-23T14:30:53Z DEBUG Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/AD.DOMAIN@IPA.DOMAIN not found in Kerberos database)
2021-07-23T14:30:57Z WARNING Downloading the CA certificate via HTTP, this is INSECURE
2021-07-23T14:30:57Z DEBUG trying to retrieve CA cert via HTTP from http://idmsrv01.ad.domain/ipa/config/ca.crt
2021-07-23T14:30:57Z DEBUG Starting external process
2021-07-23T14:30:57Z DEBUG args=/usr/bin/curl -o - http://idmsrv01.ad.domain/ipa/config/ca.crt
2021-07-23T14:30:57Z DEBUG Process finished, return code=0
2021-07-23T14:30:57Z DEBUG stdout=-----BEGIN CERTIFICATE-----
**[PEM CERTIFICATES REDACTED]**
-----END CERTIFICATE-----

2021-07-23T14:30:57Z DEBUG stderr=  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M100  7754  100  7754    0     0  1056k      0 --:--:-- --:--:-- --:--:-- 1081k

2021-07-23T14:30:57Z INFO Successfully retrieved CA cert
    Subject:     CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB
    Issuer:      CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB
    Valid From:  2004-01-01 00:00:00
    Valid Until: 2028-12-31 23:59:59

    Subject:     CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
    Issuer:      CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB
    Valid From:  2019-03-12 00:00:00
    Valid Until: 2028-12-31 23:59:59

    Subject:     CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
    Issuer:      CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
    Valid From:  2010-02-01 00:00:00
    Valid Until: 2038-01-18 23:59:59

    Subject:     CN=Sectigo RSA Organization Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB
    Issuer:      CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US
    Valid From:  2018-11-02 00:00:00
    Valid Until: 2030-12-31 23:59:59

2021-07-23T14:30:57Z DEBUG Starting external process
2021-07-23T14:30:57Z DEBUG args=/usr/sbin/ipa-join -s idmsrv01.ad.domain -b dc=ipa,dc=domain -h clientvm01.ad.domain -f
2021-07-23T14:30:57Z DEBUG Process finished, return code=17
2021-07-23T14:30:57Z DEBUG stdout=
2021-07-23T14:30:57Z DEBUG stderr=HTTP response code is 401, not 200

2021-07-23T14:30:57Z ERROR Joining realm failed: HTTP response code is 401, not 200

2021-07-23T14:30:57Z INFO Use ipa-getkeytab to obtain a host principal for this server.
2021-07-23T14:30:57Z DEBUG Starting external process
2021-07-23T14:30:57Z DEBUG args=/usr/bin/kdestroy
2021-07-23T14:30:57Z DEBUG Process finished, return code=0
2021-07-23T14:30:57Z DEBUG stdout=
2021-07-23T14:30:57Z DEBUG stderr=
2021-07-23T14:30:57Z DEBUG Initializing principal host/clientvm01.ad.domain@IPA.DOMAIN using keytab /etc/krb5.keytab
2021-07-23T14:30:57Z DEBUG using ccache /etc/ipa/.dns_ccache
2021-07-23T14:30:57Z INFO Please make sure the following ports are opened in the firewall settings:
     TCP: 80, 88, 389
     UDP: 88 (at least one of TCP/UDP ports 88 has to be open)
Also note that following ports are necessary for ipa-client working properly after enrollment:
     TCP: 464
     UDP: 464, 123 (if NTP enabled)
2021-07-23T14:30:57Z ERROR Failed to obtain host TGT: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639107): No credentials cache found
2021-07-23T14:30:57Z WARNING Installation failed. Force set so not rolling back changes.
2021-07-23T14:30:57Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run
    return cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 360, in run
    return self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 386, in execute
    for rval in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 655, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 65, in _install
    for unused in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line 3670, in main
    install(self)
  File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line 2391, in install
    _install(options)
  File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line 2644, in _install
    raise ScriptError(rval=CLIENT_INSTALL_ERROR)

2021-07-23T14:30:57Z DEBUG The ipa-client-install command failed, exception: ScriptError:
2021-07-23T14:30:57Z ERROR The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information

==== [End of of /var/log/ipaclient-install.log] ====


Le jeu. 22 juil. 2021 à 18:25, Rob Crittenden <rcritten@redhat.com> a écrit :
Adrien Dessemond via FreeIPA-users wrote:
> Hi,
>
> I do have an issue with my latest Linux (CentOS 7) virtual machines
> (names have been redacted):
>
> 1. They start their life in an AD domain (Windows 2019 DCs) and
> everything is working as expected (for the rest of the explanation, I
> will use "AD.DOMAIN" for this realm)
> 2. I have an IPA Server, which, despite having a FQDN ending in
> "ad.domain" is enrolled in its own Kerberos domain (IPA.DOMAIN) =>
> idmsrv01.ad.domain
> 2.On my client machine  I do => realm leave
> 3. Then I try to make it rejoin my IPA domain/realm => 
>
> # export KRB5_TRACE=/dev/stdout
> #  ipa-client-install --server idmsrv01.ad.domain --domain idm.domain
> --realm IPA.DOMAIN --principal enrollmentaccount --mkhomedir --no-ntp
> --force-join
>
> And here is the issue :
>
> (...)
> Using existing certificate '/etc/ipa/ca.crt'.
> Client hostname: myvm01.ad.domain
> Realm: IPA.DOMAIN
> DNS Domain: ipa.domain
> IPA Server: idmsrv01.ad.domain
> BaseDN: dc=ipa,dc=domain
>
> Skipping synchronizing time with NTP server.
> [19420] 1626829415.775243: ccselect can't find appropriate cache for
> server principal ldap/idmsrv01.ad.domain@AD.DOMAIN
> [19420] 1626829415.775244: Getting credentials
> enrollmentaccount@IPA.DOMAIN -> ldap/idmsrv01.ad.domain@AD.DOMAIN using
> ccache FILE:/tmp/krbccoz6hPK/ccache
> [19420] 1626829415.775245: Retrieving enrollmentaccount@IPA.DOMAIN ->
> ldap/idmsrv01.ad.domain@AD.DOMAIN from FILE:/tmp/krbccoz6hPK/ccache with
> result: -1765328243/Matching credential not found (filename:
> /tmp/krbccoz6hPK/ccache)
> [19420] 1626829415.775246: Retrieving enrollmentaccount@IPA.DOMAIN ->
> krbtgt/AD.DOMAIN@AD.DOMAIN from FILE:/tmp/krbccoz6hPK/ccache with
> result: -1765328243/Matching credential not found (filename:
> /tmp/krbccoz6hPK/ccache)
> [19420] 1626829415.775247: Retrieving enrollmentaccount@IPA.DOMAIN ->
> krbtgt/IPA.DOMAIN@IPA.DOMAIN from FILE:/tmp/krbccoz6hPK/ccache with
> result: 0/Success
> [19420] 1626829415.775248: Starting with TGT for client realm:
> enrollmentaccount@IPA.DOMAIN -> krbtgt/IPA.DOMAIN@IPA.DOMAIN
> [19420] 1626829415.775249: Retrieving enrollmentaccount@IPA.DOMAIN ->
> krbtgt/AD.DOMAIN@AD.DOMAIN from FILE:/tmp/krbccoz6hPK/ccache with
> result: -1765328243/Matching credential not found (filename:
> /tmp/krbccoz6hPK/ccache)
> [19420] 1626829415.775250: Requesting TGT krbtgt/AD.DOMAIN@IPA.DOMAIN
> using TGT krbtgt/IPA.DOMAIN@IPA.DOMAIN
> [19420] 1626829415.775251: Generated subkey for TGS request: aes256-cts/3835
> [19420] 1626829415.775252: etypes requested in TGS request: aes256-cts,
> aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac,
> camellia128-cts, camellia256-cts
> [19420] 1626829415.775254: Encoding request body and padata into FAST
> request
> [19420] 1626829415.775255: Sending request (1565 bytes) to IPA.DOMAIN
> [19420] 1626829415.775256: Resolving hostname idmsrv01.ad.domain
> [19420] 1626829415.775257: Initiating TCP connection to stream
> 10.xxx.xxx.xxx:88
> [19420] 1626829415.775258: Sending TCP request to stream 10.xxx.xxx.xxx:88
> [19420] 1626829415.775259: Received answer (482 bytes) from stream
> 10.xxx.xxx.xxx:88
> [19420] 1626829415.775260: Terminating TCP connection to stream
> 10.xxx.xxx.xxx:88
> [19420] 1626829415.775261: Response was from master KDC
> [19420] 1626829415.775262: Decoding FAST response
> [19420] 1626829415.775263: TGS request result: -1765328377/Server
> krbtgt/AD.DOMAIN@IPA.DOMAIN not found in Kerberos database
> Unable to download CA cert from LDAP but found preexisting cert, using it.
>
> Joining realm failed: HTTP response code is 401, not 200
>
> Installation failed. Rolling back changes.
> (...)
>
> So basically my interpretation is : the IPA client is trying  to
> initiate a Kerberos authentication which fails and being unable to do
> that it is unable to get a valid credential for the IPA server which in
> turn results in a 401 error when doing an XML RPC call on it. The
> account used for enrollment is not locked, not expired and is having the
> correct roles especially the enrollment capability.
>
> Understanding "TGS request result: -1765328377/Server
> krbtgt/AD.DOMAIN@IPA.DOMAIN" seems the key to my issue but I have no
> clue on what to do at this point.
>
> - My IPA server has no direct trust to AD.DOMAIN.
> - Client and server are both using IPA 4.6.8
> - Server certificates seem correct and not expired.
>
> Any idea on what is going on / what to look at next?

This is not a recommended DNS configuration [1]. Having the IPA domain
name different from the realm is one thing, but it points to an active
AD realm. Anything that does DNS SRV discovery is going to find the
wrong KDC which is what appears to be happening here.

Can you provide the full client install log? That may provide more
details on what is happening.

rob

[1]
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/planning_identity_management/planning-your-dns-services-and-host-names-planning-dns-and-host-names#guidelines-for-planning-the-dns-domain-name-and-kerberos-realm-name_planning-dns



--
Adrien Dessemond / VE2AKS