Sam Morris via FreeIPA-users wrote:

I tracked down the source of the myserious "Internal server error 'Link'" message when running this health check. It's caused by having a mixture of both RHEL 8 and RHEL 9 servers.

The error message in context:

# ipa-healthcheck --source=pki.server.healthcheck.clones.connectivity_and_data --check=ClonesConnectivyAndDataCheck --output-type=json --debug
[...]
stderr=
Calling check <pki.server.healthcheck.clones.connectivity_and_data.ClonesConnectivyAndDataCheck object at 0x7f8c87e8cf60>
[...]
About to check the subsystem clones
Starting new HTTPS connection (1): ipa3.example.com:443
https://ipa3.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 431
Cert data successfully obtained from clone.
Starting new HTTPS connection (1): ipa5.example.com:443
https://ipa5.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 431
Cert data successfully obtained from clone.
Starting new HTTPS connection (1): ipa6.example.com:443
https://ipa6.example.com:443 "POST /ca/rest/certs/search?size=3 HTTP/1.1" 200 317
Internal server error 'Link'
[...]
[
  {

"source": "pki.server.healthcheck.clones.connectivity_and_data", "check": "ClonesConnectivyAndDataCheck", "result": "ERROR", "uuid": "f672f185-6251-47e9-a772-8f9796a34986", "when": "20240312021736Z", "duration": "0.521790", "kw": { "status": "ERROR: pki-tomcat : Internal error testing CA clone. Host: ipa6.example.com Port: 443" } } ]

I edited ClonesConnectivyAndDataCheck.check_ca_clones to call logger.exception in its BaseException handler instead of just logger.error. This logs the traceback of the original exception:

Internal server error 'Link'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pki/server/healthcheck/clones/connectivity_and_data.py", line 35, in check_ca_clones

certs = cert_client.list_certs(size=3) File "/usr/lib/python3.6/site-packages/pki/__init__.py", line 431, in handler return fn_call(inst, *args, **kwargs) File "/usr/lib/python3.6/site-packages/pki/cert.py", line 674, in list_certs return CertDataInfoCollection.from_json(response.json()) File "/usr/lib/python3.6/site-packages/pki/cert.py", line 179, in from_json links = json_value['Link'] KeyError: 'Link'

I guess the newer version of Dogtag in RHEL 9 doesn't include this "Link" attribute, but pki.cert:CertDataInfoCollection.from_json in RHEL 8 expects it to be present.

Thanks for doing the research, this is great! Any chance you can file a ticket against the "Red Hat Certificate System" project at https://issues.redhat.com/ ? I don't own this particular check.

I'd encourage you to run your entire topology on the same IPA release. We recognize that this isn't always possible, or desirable immediately, but the transition is hopefully kept short. I usually recommend weeks not months.

rob