there are quite few logs for the various moving pieces.
I am looking at the http related logs

grant@ef-idm03:/var/log/httpd[20220601-13:08][#193]$ sudo more access_log
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:07:57:50 -0700] "POST /ipa/json HTTP/1.1" 200 291
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:07:57:53 -0700] "POST /ipa/json HTTP/1.1" 200 301
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:07:57:54 -0700] "POST /ipa/json HTTP/1.1" 200 307
10.1.132.27 - - [01/Jun/2022:08:03:03 -0700] "POST /ipa/json HTTP/1.1" 401 1300
10.1.132.27 - grant@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:03 -0700] "POST /ipa/json HTTP/1.1" 200 2555
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:07 -0700] "POST /ipa/json HTTP/1.1" 200 291
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:08 -0700] "POST /ipa/json HTTP/1.1" 200 91919
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:11 -0700] "POST /ipa/session/json HTTP/1.1" 200 291
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:11 -0700] "POST /ipa/session/json HTTP/1.1" 200 291
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:11 -0700] "POST /ipa/session/json HTTP/1.1" 200 157
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:11 -0700] "POST /ipa/session/json HTTP/1.1" 200 519
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:12 -0700] "POST /ipa/session/json HTTP/1.1" 200 193
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:50 -0700] "POST /ipa/json HTTP/1.1" 200 291
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:50 -0700] "POST /ipa/json HTTP/1.1" 200 301
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:08:03:50 -0700] "POST /ipa/json HTTP/1.1" 200 307
10.1.132.31 - grant@PRODUCTION.EFILM.COM [01/Jun/2022:08:23:49 -0700] "POST /ipa/json HTTP/1.1" 200 269
10.1.132.31 - grant@PRODUCTION.EFILM.COM [01/Jun/2022:08:23:50 -0700] "POST /ipa/json HTTP/1.1" 200 91895
10.1.132.31 - grant@PRODUCTION.EFILM.COM [01/Jun/2022:08:23:50 -0700] "POST /ipa/session/json HTTP/1.1" 200 269
10.1.132.31 - grant@PRODUCTION.EFILM.COM [01/Jun/2022:08:23:50 -0700] "POST /ipa/session/json HTTP/1.1" 200 114
10.1.132.31 - - [01/Jun/2022:10:06:10 -0700] "POST /ipa/xml HTTP/1.1" 401 1300
10.1.132.31 - - [01/Jun/2022:10:06:10 -0700] "GET /ca/rest/account/login HTTP/1.1" 200 218
10.1.132.31 - - [01/Jun/2022:10:06:11 -0700] "GET /ca/rest/authorities/3dec677a-a6e1-4aa7-8606-45fab060f1e7/cert HTTP/1.1" 200 938
10.1.132.31 - - [01/Jun/2022:10:06:11 -0700] "GET /ca/rest/account/logout HTTP/1.1" 204 -
10.1.132.31 - - [01/Jun/2022:10:06:12 -0700] "POST /ca/rest/certrequests?issuer-id=3dec677a-a6e1-4aa7-8606-45fab060f1e7 HTTP/1.1" 200 321
10.1.132.31 - - [01/Jun/2022:10:06:15 -0700] "POST /ca/agent/ca/displayBySerial HTTP/1.1" 200 12097
10.1.132.31 - host/ef-idm03.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:06:10 -0700] "POST /ipa/xml HTTP/1.1" 200 3468
10.1.132.31 - - [01/Jun/2022:10:11:22 -0700] "POST /ipa/xml HTTP/1.1" 401 1300
10.1.132.31 - - [01/Jun/2022:10:11:23 -0700] "GET /ca/rest/account/login HTTP/1.1" 200 218
10.1.132.31 - - [01/Jun/2022:10:11:23 -0700] "GET /ca/rest/authorities/3dec677a-a6e1-4aa7-8606-45fab060f1e7/cert HTTP/1.1" 200 938
10.1.132.31 - - [01/Jun/2022:10:11:23 -0700] "GET /ca/rest/account/logout HTTP/1.1" 204 -
10.1.132.31 - - [01/Jun/2022:10:11:24 -0700] "POST /ca/rest/certrequests?issuer-id=3dec677a-a6e1-4aa7-8606-45fab060f1e7 HTTP/1.1" 200 321
10.1.132.31 - - [01/Jun/2022:10:11:24 -0700] "POST /ca/agent/ca/displayBySerial HTTP/1.1" 200 12097
10.1.132.31 - host/ef-idm03.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:11:22 -0700] "POST /ipa/xml HTTP/1.1" 200 3468
10.1.132.27 - - [01/Jun/2022:10:20:39 -0700] "POST /ipa/json HTTP/1.1" 401 1300
10.1.132.27 - grant@PRODUCTION.EFILM.COM [01/Jun/2022:10:20:39 -0700] "POST /ipa/json HTTP/1.1" 200 1254
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:20:44 -0700] "POST /ipa/json HTTP/1.1" 200 291
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:20:44 -0700] "POST /ipa/json HTTP/1.1" 200 232
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:20:46 -0700] "POST /ipa/session/json HTTP/1.1" 200 291
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:20:46 -0700] "POST /ipa/session/json HTTP/1.1" 200 291
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:20:46 -0700] "POST /ipa/session/json HTTP/1.1" 200 157
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:20:47 -0700] "POST /ipa/session/json HTTP/1.1" 200 519
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:20:48 -0700] "POST /ipa/session/json HTTP/1.1" 200 1034
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:37:41 -0700] "POST /ipa/json HTTP/1.1" 200 291
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:37:42 -0700] "POST /ipa/json HTTP/1.1" 200 301
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:37:42 -0700] "POST /ipa/json HTTP/1.1" 200 307
10.1.132.27 - admin@PRODUCTION.EFILM.COM [01/Jun/2022:10:41:33 -0700] "POST /ipa/json HTTP/1.1" 200 270
10.1.132.27 - admin@PRODUCTION.EFILM.COM [01/Jun/2022:10:41:34 -0700] "POST /ipa/json HTTP/1.1" 200 148
10.1.132.27 - admin@PRODUCTION.EFILM.COM [01/Jun/2022:10:41:35 -0700] "POST /ipa/json HTTP/1.1" 200 445
10.1.132.27 - - [01/Jun/2022:10:42:24 -0700] "POST /ipa/json HTTP/1.1" 401 1300
10.1.132.31 - - [01/Jun/2022:10:42:24 -0700] "GET /ca/rest/account/login HTTP/1.1" 200 218
10.1.132.31 - - [01/Jun/2022:10:42:25 -0700] "GET /ca/rest/authorities/3dec677a-a6e1-4aa7-8606-45fab060f1e7/cert HTTP/1.1" 200 938
10.1.132.31 - - [01/Jun/2022:10:42:25 -0700] "GET /ca/rest/account/logout HTTP/1.1" 204 -
10.1.132.31 - - [01/Jun/2022:10:42:25 -0700] "POST /ca/rest/certrequests?issuer-id=3dec677a-a6e1-4aa7-8606-45fab060f1e7 HTTP/1.1" 200 321
10.1.132.31 - - [01/Jun/2022:10:42:28 -0700] "POST /ca/agent/ca/displayBySerial HTTP/1.1" 200 12097
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:42:24 -0700] "POST /ipa/json HTTP/1.1" 200 2708
10.1.132.27 - - [01/Jun/2022:10:42:49 -0700] "POST /ipa/json HTTP/1.1" 401 1300
10.1.132.31 - - [01/Jun/2022:10:42:49 -0700] "GET /ca/rest/account/login HTTP/1.1" 200 218
10.1.132.31 - - [01/Jun/2022:10:42:49 -0700] "GET /ca/rest/authorities/3dec677a-a6e1-4aa7-8606-45fab060f1e7/cert HTTP/1.1" 200 938
10.1.132.31 - - [01/Jun/2022:10:42:49 -0700] "GET /ca/rest/account/logout HTTP/1.1" 204 -
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:42:49 -0700] "POST /ipa/json HTTP/1.1" 200 420
10.1.132.27 - - [01/Jun/2022:10:42:53 -0700] "POST /ipa/json HTTP/1.1" 401 1300
10.1.132.31 - - [01/Jun/2022:10:42:54 -0700] "GET /ca/rest/account/login HTTP/1.1" 200 218
10.1.132.31 - - [01/Jun/2022:10:42:54 -0700] "GET /ca/rest/authorities/3dec677a-a6e1-4aa7-8606-45fab060f1e7/cert HTTP/1.1" 200 938
10.1.132.31 - - [01/Jun/2022:10:42:54 -0700] "GET /ca/rest/account/logout HTTP/1.1" 204 -
10.1.132.31 - - [01/Jun/2022:10:42:54 -0700] "POST /ca/rest/certrequests?issuer-id=3dec677a-a6e1-4aa7-8606-45fab060f1e7 HTTP/1.1" 200 321
10.1.132.31 - - [01/Jun/2022:10:42:58 -0700] "POST /ca/agent/ca/displayBySerial HTTP/1.1" 200 12097
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:42:53 -0700] "POST /ipa/json HTTP/1.1" 200 2708
10.1.132.27 - host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM [01/Jun/2022:10:43:13 -0700] "GET /ipa/keys/ca/caSigningCert%20cert-pki-ca?type=kem&value=eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZDQkMtSFM1MTIi
LCJraWQiOm51bGx9.ytyWNFjh3bpO7KG0OBiRVgsFhFvHqjhxq51DkpcBiFfn3ZlUQYSnCjaUhFfhW9tGfe3E9e1RFBayuOFsr-0kGt2In639SYXiel6g7gmnWe_pdKn7EeCbUBwD2HkFelh0yEgnDQX62mciGPWfcCVjkSSh8YGI2XtcoVHhxHpaX8P7bwbkM1fLIbHzyyjrrUap7Hv
B17WMXhx73Nwo_Zwz_1txFcvGFJKVuJ45Oi_Z98vMfbDZtrKJkMcVyYL_kqSc8pfqYUY9LhX2OMPrCSDWviaNMIAVq59dvfYhR1YlWnfH2yEsbojNvflvk8joO5dsfmuddZY_Xw7taVDg2zke7w.dOGMctCsZR1nPx1_-zB3bQ.GHYEFsxLGuD1_X_C8wYgnfo3BRArNn0GRBFIIJAKN
E_Uj3mogkYCciN66MCYS7CYefAJAT1f4tEP2iti5QCRCnaekoDVIAEFpYiKvYt-znJSjJFFzl7TGWLm18U5mK5-lpGQ-vP74RAsvr8AULLMRuNgt1HnYt0pRvELtwgkNK82P0zta4c8X1isYXix_TdqSgg5tTQqWvg6P52qzCAmJK_HsRqkgeb_hjFD7kbKZRigTXHzZX7oN8aarUBcC
CkGJcQAv7zEgo-EgMgWytKsvT8Eyp_7j1O3HuaXpECqh5Tzv53ERW6yDtrDNMskFacxNvYb8B5h90pE1am5Hz5PYsGtgt5k_ECyJRVEZ9GIv8IwSVp3Lyxog_kYcNjiv075YGccTzRuJBSDNUVDmMZOgzgtmu1TZsBYdcgV_Rx-FPaK6lOvBq1_AD4QCMsuAyrUzdURkX53xeOG6AB2R
1s4_KVfIWxR9MMRBIlH4yiklihsF7XUQQhJW2CtD_0LYycDacoy0sZedsWVm6jY0J6hIvWAPsh4kRlbgWTxv05Hmxh4WTXBZukQywTLD2X2aSFHsAekDSWpnxPSNXQ39hcC0h4XPhuhmDUd8WLZVYo3qAc1nhPeP6g5LmV9gdHQmdHh9BC7FXLTmLdXG3WymsWAJ6yiMPGw2f83yfvJ8
QcIkmECQ66XJfvBSYT8Fx59chJC.LoAMvRj2tVYHSLEs-7NfnhCumDM0-HTgzzTSWTiagks HTTP/1.1" 400 226
grant@ef-idm03:/var/log/httpd[20220601-13:08][#194]$

time for the promotion was 10:37

grant@ef-idm03:/var/log/httpd[20220601-13:11][#196]$ sudo tail -40 error_log
[Wed Jun 01 10:11:32.690839 2022] [:error] [pid 22764] ipa: INFO: *** PROCESS START ***
[Wed Jun 01 10:15:19.744857 2022] [mpm_prefork:notice] [pid 22758] AH00170: caught SIGWINCH, shutting down gracefully
[Wed Jun 01 10:15:21.869437 2022] [suexec:notice] [pid 25288] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jun 01 10:15:21.869527 2022] [:warn] [pid 25288] NSSSessionCacheTimeout is deprecated. Ignoring.
[Wed Jun 01 10:15:22.156344 2022] [http2:warn] [pid 25288] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Wed Jun 01 10:15:22.156997 2022] [lbmethod_heartbeat:notice] [pid 25288] AH02282: No slotmem from mod_heartmonitor
[Wed Jun 01 10:15:22.157022 2022] [:warn] [pid 25288] NSSSessionCacheTimeout is deprecated. Ignoring.
[Wed Jun 01 10:15:22.203072 2022] [mpm_prefork:notice] [pid 25288] AH00163: Apache/2.4.35 (IUS) mod_auth_gssapi/1.5.1 mod_nss/1.0.14 NSS/3.28.4 mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Wed Jun 01 10:15:22.203119 2022] [core:notice] [pid 25288] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Jun 01 10:15:25.860535 2022] [:error] [pid 25294] ipa: INFO: *** PROCESS START ***
[Wed Jun 01 10:15:25.945257 2022] [:error] [pid 25295] ipa: INFO: *** PROCESS START ***
[Wed Jun 01 10:20:39.401122 2022] [auth_gssapi:error] [pid 25302] [client 10.1.132.27:49614] NO AUTH DATA Client did not send any authentication headers, referer: https://ef-idm03.production.efilm.com/ipa
[Wed Jun 01 10:20:40.952010 2022] [:error] [pid 25294] ipa: INFO: Host entry for ef-idm04.production.efilm.com already exists, joining may fail on the client side if not forced
[Wed Jun 01 10:20:41.034646 2022] [:error] [pid 25294] ipa: INFO: [jsonserver_kerb] grant@PRODUCTION.EFILM.COM: join(u'ef-idm04.production.efilm.com', nshardwareplatform=u'x86_64', nsosversion=u'4.18.0-372.9.1.el8.x86_64'): SUCCESS
[Wed Jun 01 10:20:44.340949 2022] [:error] [pid 25295] ipa: INFO: [jsonserver_kerb] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: ping(): SUCCESS
[Wed Jun 01 10:20:44.911484 2022] [:error] [pid 25294] ipa: INFO: [jsonserver_kerb] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: schema(known_fingerprints=(u'7a5de3fd', u'70235405'), version=u'2.170'): SchemaUpToDate
[Wed Jun 01 10:20:46.586852 2022] [:warn] [pid 25298] [client 10.1.132.27:49628] failed to set perms (3140) on file (/var/run/ipa/ccaches/host~ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM)!, referer: https://ef-idm03.production.efilm.com/ipa/xml
[Wed Jun 01 10:20:46.665988 2022] [:error] [pid 25295] ipa: INFO: [jsonserver_session] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: ping(): SUCCESS
[Wed Jun 01 10:20:46.672418 2022] [:warn] [pid 25298] [client 10.1.132.27:49628] failed to set perms (3140) on file (/var/run/ipa/ccaches/host~ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM)!, referer: https://ef-idm03.production.efilm.com/ipa/xml
[Wed Jun 01 10:20:46.695819 2022] [:error] [pid 25294] ipa: INFO: [jsonserver_session] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: ping(): SUCCESS
[Wed Jun 01 10:20:46.702320 2022] [:warn] [pid 25298] [client 10.1.132.27:49628] failed to set perms (3140) on file (/var/run/ipa/ccaches/host~ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM)!, referer: https://ef-idm03.production.efilm.com/ipa/xml
[Wed Jun 01 10:20:47.178173 2022] [:error] [pid 25295] ipa: INFO: [jsonserver_session] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: ca_is_enabled(version=u'2.107'): SUCCESS
[Wed Jun 01 10:20:47.184510 2022] [:warn] [pid 25298] [client 10.1.132.27:49628] failed to set perms (3140) on file (/var/run/ipa/ccaches/host~ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM)!, referer: https://ef-idm03.production.efilm.com/ipa/xml
[Wed Jun 01 10:20:47.211175 2022] [:error] [pid 25294] ipa: INFO: [jsonserver_session] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: config_show(raw=True, version=u'2.0'): SUCCESS
[Wed Jun 01 10:20:48.134502 2022] [:warn] [pid 25298] [client 10.1.132.27:49628] failed to set perms (3140) on file (/var/run/ipa/ccaches/host~ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM)!, referer: https://ef-idm03.production.efilm.com/ipa/xml
[Wed Jun 01 10:20:48.205581 2022] [:error] [pid 25295] ipa: INFO: [jsonserver_session] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: host_mod(u'ef-idm04.production.efilm.com', ipasshpubkey=(u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBdFrfakrTiCAmS0oiDYB6Pgj4MvLnG1+B7oyrkrvuoftNNle/EQPuWABGeCCEtuJ63ISb17xv/P19lrEjxFMhM=', u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIFQLTw4Dp3gDT1JZb9/aYpR97e/XizA3tQVKLwIkbf4', u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0lQULgS2oXBSDdmPR6m8sXtYFiZuQSXgnOig9m1J3KJr7H4ldO3eMydsncmm2wL9Lygl6NQepTdJvnaUtxbkbMZ/5lm5NfTzBlWTsmncYQINloazgHJPHI9Vj2UO0Syx4nqlNUn3or5Z/SuwQG73h5Mb+3u9ZhnTNiqVdWdtse6qwCLJYj83O/tIxwQON0S/wNSby7F3MuU///ui/UuwA8ppoUP3zYy6+b7MVT5qH06Hjl6VvDqx9NpmViU+cn5EpiDs1sphn/8pn1wiBBKNp5ouYqvz7YHir4zTSa5g2Onr4CcPBxCnRgTyC1XNX7U3TXMI7BYpJZiBCOJlX+PuNcop/tec6slGKNQG4qo0gSKoy+5q8hH67/u6gWC4QXhf8zwbmXT0TTiAJ4whn5OfnuNPXxymP0lLJs8sy9qJI4M9FvGkqRF+rv9AEBmfVYl42QwL3Bt4qfCRNOLNA6iZR5a7t2DTHHr+GTidk5hE3nXNSvFbHgL7xIxff677Brdk='), updatedns=False, version=u'2.26'): SUCCESS
[Wed Jun 01 10:37:42.075025 2022] [:error] [pid 25294] ipa: INFO: [jsonserver_kerb] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: ping(): SUCCESS
[Wed Jun 01 10:37:42.911059 2022] [:error] [pid 25295] ipa: INFO: [jsonserver_kerb] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: env((u'version',)): SUCCESS
[Wed Jun 01 10:37:43.611988 2022] [:error] [pid 25294] ipa: INFO: [jsonserver_kerb] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: env((u'fips_mode',)): SUCCESS
[Wed Jun 01 10:41:34.576220 2022] [:error] [pid 25295] ipa: INFO: [jsonserver_kerb] admin@PRODUCTION.EFILM.COM: ping(): SUCCESS
[Wed Jun 01 10:41:35.395944 2022] [:error] [pid 25294] ipa: INFO: [jsonserver_kerb] admin@PRODUCTION.EFILM.COM: ping/1(version=u'2.245'): SUCCESS
[Wed Jun 01 10:41:38.372506 2022] [:error] [pid 25295] ipa: INFO: [jsonserver_kerb] admin@PRODUCTION.EFILM.COM: server_conncheck(u'ef-idm03.production.efilm.com', u'ef-idm04.production.efilm.com', version=u'2.162'): SUCCESS
[Wed Jun 01 10:42:24.771287 2022] [auth_gssapi:error] [pid 25297] [client 10.1.132.27:49746] NO AUTH DATA Client did not send any authentication headers, referer: https://ef-idm03.production.efilm.com/ipa/json
[Wed Jun 01 10:42:29.371941 2022] [:error] [pid 25294] ipa: INFO: [jsonserver_kerb] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: cert_request(u'MIID+DCCAuACAQAwRzEdMBsGA1UEChMUUFJPRFVDVElPTi5FRklMTS5DT00xJjAkBgNVBAMTHWVmLWlkbTA0LnByb2R1Y3Rpb24uZWZpbG0uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0kIeItY2TxjIFEfqvJm0Nl/Rz3dmVHVIGQ82KnJVp+JddSFvs9HGhefSCbCwm1/4ehqI6n0dn4oxUj9rvRZbMhEFi/Aac7enWSJeVUJqT3fFH/Y7Lgv1ATD+Tu8m3BQINtidcM2+wXG2ar0Sz7tShA7W74D4xqp7jIoHuECJOrsBrMcdWDlvem3SqVtmy2Yy1Y+xGwpBQIr0MialgdGx17b8cg3QRj6fbVfES2RQclDSjOO+UP3UQvKxMCMsUPrxOpcG8KS7DROnuCeXZ4EE72uKkRXVLLV8hu6uVGS7Iwh6yULgpireJ56PYCxtoHnOkn/UiZOMlDW+mtq/LvzzFwIDAQABoIIBajAlBgkqhkiG9w0BCRQxGB4WAFMAZQByAHYAZQByAC0AQwBlAHIAdDCCAT8GCSqGSIb3DQEJDjGCATAwggEsMIHLBgNVHREEgcMwgcCCHWVmLWlkbTA0LnByb2R1Y3Rpb24uZWZpbG0uY29toEcGCisGAQQBgjcUAgOgOQw3bGRhcC9lZi1pZG0wNC5wcm9kdWN0aW9uLmVmaWxtLmNvbUBQUk9EVUNUSU9OLkVGSUxNLkNPTaBWBgYrBgEFAgKgTDBKoBYbFFBST0RVQ1RJT04uRUZJTE0uQ09NoTAwLqADAgEBoScwJRsEbGRhcBsdZWYtaWRtMDQucHJvZHVjdGlvbi5lZmlsbS5jb20wDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUJua2iKrtCB6C/7qcdhcK4X+5GZgwLwYJKwYBBAGCNxQCBCIeIABjAGEASQBQAEEAcwBlAHIAdgBpAGMAZQBDAGUAcgB0MA0GCSqGSIb3DQEBCwUAA4IBAQCL9BSabUrQCVUaonjt8MgLixdUoVfeyZEx4E6y6EOpNHhkgR1OmugQU48132t1GXfDopobVBruy6kY5AO1MxblPC6W1JVMf1puB4jwfHCy9VSOInnDcViWqPic5LzlP0osbYtOxV7cCMspm5qWaYkhAKpMeDggiUXdQyDqUa8dCXPWLWz3KV68/3V9bjvle+zkPr3Jfl4mZK/BdHmSQJK5ipi9EF12cdlqR9pFg/y0hHw/+Z1zZeKde3NR9RhF/hMZVhxu8mmvuiZGXA8SDOUYyeOMoSFMGfo9vJK+qj/W+AkDozCc+AraFDcfCKPTceDiv8Ye37xQKHBbv7wXPD18', profile_id=u'caIPAserviceCert', principal=u'ldap/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM', add=True): SUCCESS
[Wed Jun 01 10:42:49.295340 2022] [auth_gssapi:error] [pid 7534] [client 10.1.132.27:49770] NO AUTH DATA Client did not send any authentication headers, referer: https://ef-idm03.production.efilm.com/ipa/json
[Wed Jun 01 10:42:49.597101 2022] [:error] [pid 25295] ipa: INFO: [jsonserver_kerb] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: cert_request(u'MIIEGzCCAwMCAQAwRzEdMBsGA1UECgwUUFJPRFVDVElPTi5FRklMTS5DT00xJjAkBgNVBAMTHWVmLWlkbTA0LnByb2R1Y3Rpb24uZWZpbG0uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlVfoLK/9ZErsD8hGCC9Zk22VLURVcxRMntWgJ9XfJcPshB9xkEdRVLZv4qBaaCnVZY1EXHKptG3hG14Xo21k6pfQfnhP6uwe/fgWCJX1I+wskA+6GYkU66uexe+IR1PuynbA2rn8Ls2k52WuFOXSO7ZhuOJXuuZ9L+NaNzZBZ4wia2Yrznn+SeyvS/xnofI0gOGi/sEXEXyHIwJbVg2gjBRpfu+/1cUe+xMnlnrVD1GYRbrR5DKYjz8uCYiOQchmTvRx0lP+xI7eP0aKOSZmbFVJd75Q3exf9Njj8A0C8srK28+y3YxKnu+BTLm92XCiQWXQpDHHnapy3Yii0axCEwIDAQABoIIBjTArBgkqhkiG9w0BCRQxHh4cADIAMAAyADIAMAA2ADAAMQAxADcANAAyADQAOTCCAVwGCSqGSIb3DQEJDjGCAU0wggFJMIHoBgNVHREEgeAwgd2CHWVmLWlkbTA0LnByb2R1Y3Rpb24uZWZpbG0uY29tghtpcGEtY2EucHJvZHVjdGlvbi5lZmlsbS5jb22gRwYKKwYBBAGCNxQCA6A5DDdIVFRQL2VmLWlkbTA0LnByb2R1Y3Rpb24uZWZpbG0uY29tQFBST0RVQ1RJT04uRUZJTE0uQ09NoFYGBisGAQUCAqBMMEqgFhsUUFJPRFVDVElPTi5FRklMTS5DT02hMDAuoAMCAQGhJzAlGwRIVFRQGx1lZi1pZG0wNC5wcm9kdWN0aW9uLmVmaWxtLmNvbTAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBS9xMaNq7PEZPM1/J0jIeSRb8P0DTAvBgkrBgEEAYI3FAIEIh4gAGMAYQBJAFAAQQBzAGUAcgB2AGkAYwBlAEMAZQByAHQwDQYJKoZIhvcNAQELBQADggEBADvqPEEDKnC9I6kS2CFaT85GCJP2qxLRkRLLBk/Qh4wZNdX6bsGLXlcALgrRWrGIfMkOFNbkI6fblWSKDjzOVYcjZkQQVazuQFethuuRcPkADViI3Ikaz9BfWZxGJtIjLOKjASFDdPjVdUHCgNKZyo+i8l0mmaxPDInrzvNN1NCHU7e7KXMtGPkGtM2+e4ozZrrFv3VSBplQktdhLGqNXh6qmJtAl3Mhfw/T1kxlxUO9YXj60ob7xrosrAIQkX9LrJdWYrdfQ48ivIl9LTTQrvYJrUwBLc0xQsHRPffqNo0F074Bhsim0D0nreTPd0wTEfryTwxJm8nz3qST+CQQBBQ=', profile_id=u'caIPAserviceCert', principal=u'HTTP/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM', add=True): NotFound
[Wed Jun 01 10:42:53.041815 2022] [auth_gssapi:error] [pid 25297] [client 10.1.132.27:49790] NO AUTH DATA Client did not send any authentication headers, referer: https://ef-idm03.production.efilm.com/ipa/json
[Wed Jun 01 10:42:58.572629 2022] [:error] [pid 25294] ipa: INFO: [jsonserver_kerb] host/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM: cert_request(u'MIID/jCCAuYCAQAwRzEdMBsGA1UECgwUUFJPRFVDVElPTi5FRklMTS5DT00xJjAkBgNVBAMTHWVmLWlkbTA0LnByb2R1Y3Rpb24uZWZpbG0uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlVfoLK/9ZErsD8hGCC9Zk22VLURVcxRMntWgJ9XfJcPshB9xkEdRVLZv4qBaaCnVZY1EXHKptG3hG14Xo21k6pfQfnhP6uwe/fgWCJX1I+wskA+6GYkU66uexe+IR1PuynbA2rn8Ls2k52WuFOXSO7ZhuOJXuuZ9L+NaNzZBZ4wia2Yrznn+SeyvS/xnofI0gOGi/sEXEXyHIwJbVg2gjBRpfu+/1cUe+xMnlnrVD1GYRbrR5DKYjz8uCYiOQchmTvRx0lP+xI7eP0aKOSZmbFVJd75Q3exf9Njj8A0C8srK28+y3YxKnu+BTLm92XCiQWXQpDHHnapy3Yii0axCEwIDAQABoIIBcDArBgkqhkiG9w0BCRQxHh4cADIAMAAyADIAMAA2ADAAMQAxADcANAAyADUAMjCCAT8GCSqGSIb3DQEJDjGCATAwggEsMIHLBgNVHREEgcMwgcCCHWVmLWlkbTA0LnByb2R1Y3Rpb24uZWZpbG0uY29toEcGCisGAQQBgjcUAgOgOQw3SFRUUC9lZi1pZG0wNC5wcm9kdWN0aW9uLmVmaWxtLmNvbUBQUk9EVUNUSU9OLkVGSUxNLkNPTaBWBgYrBgEFAgKgTDBKoBYbFFBST0RVQ1RJT04uRUZJTE0uQ09NoTAwLqADAgEBoScwJRsESFRUUBsdZWYtaWRtMDQucHJvZHVjdGlvbi5lZmlsbS5jb20wDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUvcTGjauzxGTzNfydIyHkkW/D9A0wLwYJKwYBBAGCNxQCBCIeIABjAGEASQBQAEEAcwBlAHIAdgBpAGMAZQBDAGUAcgB0MA0GCSqGSIb3DQEBCwUAA4IBAQAknz+V+wPg9zUWSGkKoIvG/ETV7/zikjS3FML5M/zsMCz4wzewgLY6r5GS2OyPdTRkmNxG8scPwAl43bwduyagzpD1AcpSVdEXm47hzXnQ9uiQxuFnIRdXeNL5PAEGJz72WRIzVPk0JFybFnPBUu6TXgdy40SBncITz813p1rEKFSHbKnoNSgO4OSTenbzkvKPE53VXcXF8nNruIgXSTo+lqF9Efq3brrr7IhQBtJY1WsjfX1Rv3C+qibxaKZL4Akyim216vAtN0JLDEnZUBdE6MtZ7xf4SETfXVC/tYDB0nvzb9YyyoqCrc80jeU1RNBpdh/gxn6UOj1WfF6+8HZE', profile_id=u'caIPAserviceCert', principal=u'HTTP/ef-idm04.production.efilm.com@PRODUCTION.EFILM.COM', add=True): SUCCESS
[Wed Jun 01 10:44:13.188170 2022] [proxy:error] [pid 25297] (20014)Internal error (specific information not available): [client 10.1.132.27:49818] AH01084: pass request body failed to 0.0.0.0:0 (httpd-UDS)
[Wed Jun 01 10:44:13.188223 2022] [proxy_http:error] [pid 25297] [client 10.1.132.27:49818] AH01097: pass request body failed to 0.0.0.0:0 (httpd-UDS) from 10.1.132.27 ()
grant@ef-idm03:/var/log/httpd[20220601-13:12][#197]$

I could "ipa-server-install —uninstall”, "del-host ef-idm04”, rejoin as a client, then try again to promote.  But it would just be walking in the footprints already trodden.  I wouldn't expect a different outcome.
if there is a particular log, you’d like a glance at - please ask and I’ll fetch it.

thanx

- grant


On Jun 1, 2022, at 12:56, Rob Crittenden <rcritten@redhat.com> wrote:

CAUTION: This email originated outside Company3-Method. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Grant Janssen wrote:
I have attached the ipareplica-install.log
Let me figure out how to add a SAN to the web server certs.

Probably not a priority yet as it isn't failing on a TLS issue.

Was anything logged on the other server side for this request?

rob