Hi All,

I've been using freeipa configured as a HA pair on Centos for about 12 months and I've been really impressed, however this morning it has started pumping mud. Any suggestions appreciated.

I did a dnf update of the server which appears to have broken the FreeIPA server and I see the following errors from the ipa start

ipactl start 
IPA version error: data needs to be upgraded (expected version '4.8.7-13.module_el8.3.0+606+1e8766d7', current version '4.8.7-12.module_el8.3.0+511+8a502f20')
Automatically running upgrade, for details see /var/log/ipaupgrade.log
...
[Disabling cert publishing]
[Ensuring CA is using LDAPProfileSubsystem]
[Migrating certificate profiles to LDAP]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information


Some information
The broken system.
CentOS Linux release 8.3.2011
ipa-server-4.8.7-13 (the updated server)

The still operational system
CentOS Linux release 8.3.2011
ipa-server-4.8.7-12

The certificate information based upon the following commands appear to be good.

getcert list -f /var/lib/ipa/ra-agent.pem | grep expires
expires: 2021-12-17 14:43:54 AEDT

ldapsearch -D "cn=directory manager" -W -b o=ipaca "(uid=ipara)"

openssl x509 -text -in /var/lib/ipa/ra-agent.pem

From the /var/log/ipaupgrade.log

2021-01-12T09:51:07Z DEBUG request GET https://groats.ipa.bogus.com.au:8443/ca/rest/account/login
2021-01-12T09:51:07Z DEBUG request body ''
2021-01-12T09:51:07Z DEBUG response status 500
2021-01-12T09:51:07Z DEBUG response headers Content-Type: text/html;charset=utf-8

From the ca debug logs /var/log/pki/pki-tomcat/ca/debug.2021-01-12.log

I'm not sure if the following are relevant

2021-01-12 20:50:49 [main] FINEST: Getting log.instance.SignedAudit.events=ACCESS_SESSION_ESTABLISH,ACCESS_SESSION_TERMINATED,AUTH,AUTHORITY_CONFIG,AUTHZ,CERT_PROFILE_APPROVAL,CERT_REQUEST_PROCESSED,CERT_SIGNING_INFO,CERT_STATUS_CHANGE_REQUEST_PROCESSED,CLIENT_ACCESS_SESSION_ESTABLISH,CLIENT_ACCESS_SESSION_TERMINATED,CMC_REQUEST_RECEIVED,CMC_RESPONSE_SENT,CMC_SIGNED_REQUEST_SIG_VERIFY,CMC_USER_SIGNED_REQUEST_SIG_VERIFY,CONFIG_ACL,CONFIG_AUTH,CONFIG_CERT_PROFILE,CONFIG_CRL_PROFILE,CONFIG_DRM,CONFIG_ENCRYPTION,CONFIG_OCSP_PROFILE,CONFIG_ROLE,CONFIG_SERIAL_NUMBER,CONFIG_SIGNED_AUDIT,CONFIG_TRUSTED_PUBLIC_KEY,CRL_SIGNING_INFO,DELTA_CRL_GENERATION,FULL_CRL_GENERATION,LOG_PATH_CHANGE,OCSP_GENERATION,OCSP_SIGNING_INFO,PROFILE_CERT_REQUEST,PROOF_OF_POSSESSION,RANDOM_GENERATION,ROLE_ASSUME,SECURITY_DOMAIN_UPDATE,SELFTESTS_EXECUTION
2021-01-12 20:50:49 [main] FINEST: Getting log.instance.SignedAudit.events=ACCESS_SESSION_ESTABLISH,ACCESS_SESSION_TERMINATED,AUTH,AUTHORITY_CONFIG,AUTHZ,CERT_PROFILE_APPROVAL,CERT_REQUEST_PROCESSED,CERT_SIGNING_INFO,CERT_STATUS_CHANGE_REQUEST_PROCESSED,CLIENT_ACCESS_SESSION_ESTABLISH,CLIENT_ACCESS_SESSION_TERMINATED,CMC_REQUEST_RECEIVED,CMC_RESPONSE_SENT,CMC_SIGNED_REQUEST_SIG_VERIFY,CMC_USER_SIGNED_REQUEST_SIG_VERIFY,CONFIG_ACL,CONFIG_AUTH,CONFIG_CERT_PROFILE,CONFIG_CRL_PROFILE,CONFIG_DRM,CONFIG_ENCRYPTION,CONFIG_OCSP_PROFILE,CONFIG_ROLE,CONFIG_SERIAL_NUMBER,CONFIG_SIGNED_AUDIT,CONFIG_TRUSTED_PUBLIC_KEY,CRL_SIGNING_INFO,DELTA_CRL_GENERATION,FULL_CRL_GENERATION,LOG_PATH_CHANGE,OCSP_GENERATION,OCSP_SIGNING_INFO,PROFILE_CERT_REQUEST,PROOF_OF_POSSESSION,RANDOM_GENERATION,ROLE_ASSUME,SECURITY_DOMAIN_UPDATE,SELFTESTS_EXECUTION
2021-01-12 20:50:49 [main] FINE: Event filters:
2021-01-12 20:50:49 [main] FINE: - CMC_SIGNED_REQUEST_SIG_VERIFY: (Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - CMC_USER_SIGNED_REQUEST_SIG_VERIFY: (Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - DELTA_CRL_GENERATION: (Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - FULL_CRL_GENERATION: (Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - OCSP_GENERATION: (Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - RANDOM_GENERATION: (Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - SELFTESTS_EXECUTION: (Outcome=Failure)
2021-01-12 20:50:49 [main] FINEST: Property log.instance.SignedAudit.trace not found

However where it dies is
2021-01-12 20:50:50 [main] FINEST: Property internaldb.doCloning not found
2021-01-12 20:50:50 [main] FINEST: Getting internaldb.doCloning=true
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: doCloning: true
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: mininum: 3
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: maximum: 15
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: host: oats.ipa.amnesium.com.au
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: port: 636
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: secure: true
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: authentication: 2
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: makeConnection(true)
2021-01-12 20:50:50 [main] FINEST: Getting internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
2021-01-12 20:50:50 [main] FINEST: Property tcp.keepAlive not found
2021-01-12 20:50:50 [main] FINEST: Getting tcp.keepAlive=true
2021-01-12 20:50:50 [main] FINE: TCP Keep-Alive: true
2021-01-12 20:50:50 [main] FINE: LdapBoundConnection: Connecting to oats.ipa.amnesium.com.au:636 with client cert auth
2021-01-12 20:50:50 [main] FINE: ldapconn/PKISocketFactory.makeSSLSocket: begins
2021-01-12 20:50:50 [main] FINE: SignedAuditLogger: event CLIENT_ACCESS_SESSION_ESTABLISH
2021-01-12 20:50:50 [main] FINEST: Getting pidDir=/var/run/pki/tomcat
2021-01-12 20:50:50 [main] FINEST: Getting pidDir=/var/run/pki/tomcat
2021-01-12 20:50:50 [main] SEVERE: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused)
java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
.....