I tracked down the source of the myserious "Internal server error 'Link'" message when running this health check. It's caused by having a mixture of both RHEL 8 and RHEL 9 servers.
The error message in context:
# ipa-healthcheck --source=pki.server.healthcheck.clones.connectivity_and_data --check=ClonesConnectivyAndDataCheck --output-type=json --debug [...] stderr= Calling check <pki.server.healthcheck.clones.connectivity_and_data.ClonesConnectivyAndDataCheck object at 0x7f8c87e8cf60> [...] About to check the subsystem clones Starting new HTTPS connection (1): ipa3.example.com:443 https://ipa3.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 431 Cert data successfully obtained from clone. Starting new HTTPS connection (1): ipa5.example.com:443 https://ipa5.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 431 Cert data successfully obtained from clone. Starting new HTTPS connection (1): ipa6.example.com:443 https://ipa6.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 317 Internal server error 'Link' [...] [ { "source": "pki.server.healthcheck.clones.connectivity_and_data", "check": "ClonesConnectivyAndDataCheck", "result": "ERROR", "uuid": "f672f185-6251-47e9-a772-8f9796a34986", "when": "20240312021736Z", "duration": "0.521790", "kw": { "status": "ERROR: pki-tomcat : Internal error testing CA clone. Host: ipa6.example.com Port: 443" } } ]
I edited ClonesConnectivyAndDataCheck.check_ca_clones to call logger.exception in its BaseException handler instead of just logger.error. This logs the traceback of the original exception:
Internal server error 'Link' Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/pki/server/healthcheck/clones/connectivity_and_data.py", line 35, in check_ca_clones certs = cert_client.list_certs(size=3) File "/usr/lib/python3.6/site-packages/pki/__init__.py", line 431, in handler return fn_call(inst, *args, **kwargs) File "/usr/lib/python3.6/site-packages/pki/cert.py", line 674, in list_certs return CertDataInfoCollection.from_json(response.json()) File "/usr/lib/python3.6/site-packages/pki/cert.py", line 179, in from_json links = json_value['Link'] KeyError: 'Link'
I guess the newer version of Dogtag in RHEL 9 doesn't include this "Link" attribute, but pki.cert:CertDataInfoCollection.from_json in RHEL 8 expects it to be present.
-- Sam Morris https://robots.org.uk/ CAAA AA1A CA69 A83A 892B 1855 D20B 4202 5CDA 27B9