Hi All,
I've been using freeipa configured as a HA pair on Centos for about 12
months and I've been really impressed, however this morning it has
started pumping mud. Any suggestions appreciated.
I did a dnf update of the server which appears to have broken the
FreeIPA server and I see the following errors from the ipa start
ipactl start
IPA version error: data needs to be upgraded (expected version '4.8.7-
13.module_el8.3.0+606+1e8766d7', current version '4.8.7-
12.module_el8.3.0+511+8a502f20')
Automatically running upgrade, for details see /var/log/ipaupgrade.log
...
[Disabling cert publishing]
[Ensuring CA is using LDAPProfileSubsystem]
[Migrating certificate profiles to LDAP]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
more information
Some information
The broken system.
CentOS Linux release 8.3.2011
ipa-server-4.8.7-13 (the updated server)
The still operational system
CentOS Linux release 8.3.2011
ipa-server-4.8.7-12
The certificate information based upon the following commands appear to
be good.
getcert list -f /var/lib/ipa/ra-agent.pem | grep expires
expires: 2021-12-17 14:43:54 AEDT
ldapsearch -D "cn=directory manager" -W -b o=ipaca "(uid=ipara)"
openssl x509 -text -in /var/lib/ipa/ra-agent.pem
From the /var/log/ipaupgrade.log
2021-01-12T09:51:07Z DEBUG request GET
https://groats.ipa.bogus.com.au:8443/ca/rest/account/login
2021-01-12T09:51:07Z DEBUG request body ''
2021-01-12T09:51:07Z DEBUG response status 500
2021-01-12T09:51:07Z DEBUG response headers Content-Type:
text/html;charset=utf-8
From the ca debug logs /var/log/pki/pki-tomcat/ca/debug.2021-01-12.log
I'm not sure if the following are relevant
2021-01-12 20:50:49 [main] FINEST: Getting
log.instance.SignedAudit.events=ACCESS_SESSION_ESTABLISH,ACCESS_SESSION
_TERMINATED,AUTH,AUTHORITY_CONFIG,AUTHZ,CERT_PROFILE_APPROVAL,CERT_REQU
EST_PROCESSED,CERT_SIGNING_INFO,CERT_STATUS_CHANGE_REQUEST_PROCESSED,CL
IENT_ACCESS_SESSION_ESTABLISH,CLIENT_ACCESS_SESSION_TERMINATED,CMC_REQU
EST_RECEIVED,CMC_RESPONSE_SENT,CMC_SIGNED_REQUEST_SIG_VERIFY,CMC_USER_S
IGNED_REQUEST_SIG_VERIFY,CONFIG_ACL,CONFIG_AUTH,CONFIG_CERT_PROFILE,CON
FIG_CRL_PROFILE,CONFIG_DRM,CONFIG_ENCRYPTION,CONFIG_OCSP_PROFILE,CONFIG
_ROLE,CONFIG_SERIAL_NUMBER,CONFIG_SIGNED_AUDIT,CONFIG_TRUSTED_PUBLIC_KE
Y,CRL_SIGNING_INFO,DELTA_CRL_GENERATION,FULL_CRL_GENERATION,LOG_PATH_CH
ANGE,OCSP_GENERATION,OCSP_SIGNING_INFO,PROFILE_CERT_REQUEST,PROOF_OF_PO
SSESSION,RANDOM_GENERATION,ROLE_ASSUME,SECURITY_DOMAIN_UPDATE,SELFTESTS
_EXECUTION
2021-01-12 20:50:49 [main] FINEST: Getting
log.instance.SignedAudit.events=ACCESS_SESSION_ESTABLISH,ACCESS_SESSION
_TERMINATED,AUTH,AUTHORITY_CONFIG,AUTHZ,CERT_PROFILE_APPROVAL,CERT_REQU
EST_PROCESSED,CERT_SIGNING_INFO,CERT_STATUS_CHANGE_REQUEST_PROCESSED,CL
IENT_ACCESS_SESSION_ESTABLISH,CLIENT_ACCESS_SESSION_TERMINATED,CMC_REQU
EST_RECEIVED,CMC_RESPONSE_SENT,CMC_SIGNED_REQUEST_SIG_VERIFY,CMC_USER_S
IGNED_REQUEST_SIG_VERIFY,CONFIG_ACL,CONFIG_AUTH,CONFIG_CERT_PROFILE,CON
FIG_CRL_PROFILE,CONFIG_DRM,CONFIG_ENCRYPTION,CONFIG_OCSP_PROFILE,CONFIG
_ROLE,CONFIG_SERIAL_NUMBER,CONFIG_SIGNED_AUDIT,CONFIG_TRUSTED_PUBLIC_KE
Y,CRL_SIGNING_INFO,DELTA_CRL_GENERATION,FULL_CRL_GENERATION,LOG_PATH_CH
ANGE,OCSP_GENERATION,OCSP_SIGNING_INFO,PROFILE_CERT_REQUEST,PROOF_OF_PO
SSESSION,RANDOM_GENERATION,ROLE_ASSUME,SECURITY_DOMAIN_UPDATE,SELFTESTS
_EXECUTION
2021-01-12 20:50:49 [main] FINE: Event filters:
2021-01-12 20:50:49 [main] FINE: - CMC_SIGNED_REQUEST_SIG_VERIFY:
(Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - CMC_USER_SIGNED_REQUEST_SIG_VERIFY:
(Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - DELTA_CRL_GENERATION:
(Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - FULL_CRL_GENERATION:
(Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - OCSP_GENERATION: (Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - RANDOM_GENERATION: (Outcome=Failure)
2021-01-12 20:50:49 [main] FINE: - SELFTESTS_EXECUTION:
(Outcome=Failure)
2021-01-12 20:50:49 [main] FINEST: Property
log.instance.SignedAudit.trace not found
However where it dies is
2021-01-12 20:50:50 [main] FINEST: Property internaldb.doCloning not
found
2021-01-12 20:50:50 [main] FINEST: Getting internaldb.doCloning=true
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: doCloning: true
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: mininum: 3
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: maximum: 15
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: host:
oats.ipa.amnesium.com.au
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: port: 636
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: secure: true
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory: authentication:
2
2021-01-12 20:50:50 [main] FINE: LdapBoundConnFactory:
makeConnection(true)
2021-01-12 20:50:50 [main] FINEST: Getting
internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
2021-01-12 20:50:50 [main] FINEST: Property tcp.keepAlive not found
2021-01-12 20:50:50 [main] FINEST: Getting tcp.keepAlive=true
2021-01-12 20:50:50 [main] FINE: TCP Keep-Alive: true
2021-01-12 20:50:50 [main] FINE: LdapBoundConnection: Connecting to
oats.ipa.amnesium.com.au:636 with client cert auth
2021-01-12 20:50:50 [main] FINE:
ldapconn/PKISocketFactory.makeSSLSocket: begins
2021-01-12 20:50:50 [main] FINE: SignedAuditLogger: event
CLIENT_ACCESS_SESSION_ESTABLISH
2021-01-12 20:50:50 [main] FINEST: Getting pidDir=/var/run/pki/tomcat
2021-01-12 20:50:50 [main] FINEST: Getting pidDir=/var/run/pki/tomcat
2021-01-12 20:50:50 [main] SEVERE: Unable to create socket:
java.net.ConnectException: Connection refused (Connection refused)
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
pl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
88)
.....