Hello,
I’m hoping somebody here can help with an issue I’m having with upgrades and ipaca. I have (3) CentOS 7.1.1503
based systems that I’m trying to upgrade from 4.2.0-15.0.1.el7 to ipa-server-4.5.4-10.el7. I’m able to upgrade the “second master” (dirsrv, DNS, ipaca backup) and “third master” (dirsrv, DNS, apace backup) without an issue (replication is good after
3-4 hours). But when I try to upgrade the “first master” (dirsrv, DNS, ipaca primary) the upgrade process completes successfully and starts the services, but the pki-tomcat fails to stay running. Odd thing is that it does run for about 4-5 minutes (I can see
certificate data, and can list certificates from the CLI), but after about 5 minutes the whole IPA system stops (per the systemctl). I can run the IPA services on the “first master” (ipactl start --ignore-service-failures), but eventually the replication for
ipaca fails — I suppose this is expected since pki-tomcat isn’t running (LDAP connection error from the “first master” to the “second/third masters”).
Funny thing is I’m not able to see anything in the logs that point to anything that shows as a fault. In referencing https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/, I wasn’t able to isolate anything
that seems like it was related to the issue of pki-tomcatd terminating.
I eventually reverted all hosts back to a “safe snapshot” and reverted to keep the production systems active and in sync. I’m hoping the wise people here might be able to ID something amiss in the currently running systems before I make another
attempt to get the systems upgraded again, or perhaps suggest superset of logs and data-points I would need to gather if it was to fail again.
Per https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/
—
/var/log/ipaupgrade.log: Completed successfully
/var/log/pki/pki-tomcat/ca/debug: No trace/stack outputs. No strings marked as “error”.
Also checked catalina.out, and didn’t see anything amiss. There was a trace for a missing module (can’t remember the name right now and it isn’t in my notes), but the services and webapps started without it.
Certificate cert-pki-ca:
sudo certutil -L -d /etc/pki/pki-tomcat/alias
-n 'Server-Cert cert-pki-ca' | grep 'Not'
Not Before: Sun Aug 20 22:02:05 2017
Not After : Sat Aug 10 22:02:05 2019
sudo certutil -L -d /etc/dirsrv/slapd-<removed>-<removed 2>-COM/ -n Server-Cert |grep "Not "
Not Before: Thu Aug 31 22:02:18 2017
Not After : Sun Sep 01 22:02:18 2019
sudo certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep "Not "
Not Before: Thu Aug 31 22:02:08 2017
Not After : Sun Sep 01 22:02:08 2019
Was able to read cert using the password in /var/lib/pki/pki-tomcat/conf/password.conf.
In LDAP uid=pkidbuser,ou=people,o=ipaca userCertificate appears to be valid and matches what is in the NSSDB.
There are (8) certificates being monitored, and none have expired —
sudo getcert list
Number of certificates and requests being tracked: 8.
Request ID '20150928161427':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed 2>-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-<removed>-<removed 2>-COM/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed 2>-COM',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM
subject: CN=starfleet.<removed>.<removed 2>.com,O=<removed>.<removed 2>.COM
expires: 2019-09-01 22:02:18 UTC
principal name: ldap/starfleet.<removed>.<removed 2>.com@<removed>.<removed 2>.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv <removed>-<removed 2>-COM
track: yes
auto-renew: yes
Request ID '20150928161756':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM
subject: CN=starfleet.<removed>.<removed 2>.com,O=<removed>.<removed 2>.COM
expires: 2019-09-01 22:02:08 UTC
principal name: HTTP/starfleet.<removed>.<removed 2>.com@<removed>.<removed 2>.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Request ID '20160725201511':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM
subject: CN=CA Audit,O=<removed>.<removed 2>.COM
expires: 2019-08-10 22:04:31 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201512':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM
subject: CN=OCSP Subsystem,O=<removed>.<removed 2>.COM
expires: 2019-08-10 22:02:50 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201513':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM
subject: CN=CA Subsystem,O=<removed>.<removed 2>.COM
expires: 2019-08-10 22:02:11 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201514':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM
subject: CN=Certificate Authority,O=<removed>.<removed 2>.COM
expires: 2035-09-28 16:13:20 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201515':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM
subject: CN=IPA RA,O=<removed>.<removed 2>.COM
expires: 2019-08-10 22:04:11 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/lib64/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20160725201516':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM
subject: CN=starfleet.<removed>.<removed 2>.com,O=<removed>.<removed 2>.COM
expires: 2019-08-10 22:02:05 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Replication notes –
ipa server-role-find
-----------------------
18 server roles matched
-----------------------
Server name: federation.<removed>.<removed 2>.com
Role name: CA server
Role status: enabled
Server name: romulus.<removed>.<removed 2>.com
Role name: CA server
Role status: enabled
Server name: starfleet.<removed>.<removed 2>.com
Role name: CA server
Role status: enabled
Server name: federation.<removed>.<removed 2>.com
Role name: DNS server
Role status: enabled
Server name: romulus.<removed>.<removed 2>.com
Role name: DNS server
Role status: enabled
Server name: starfleet.<removed>.<removed 2>.com
Role name: DNS server
Role status: enabled
Server name: federation.<removed>.<removed 2>.com
Role name: NTP server
Role status: enabled
Server name: romulus.<removed>.<removed 2>.com
Role name: NTP server
Role status: enabled
Server name: starfleet.<removed>.<removed 2>.com
Role name: NTP server
Role status: absent
Server name: federation.<removed>.<removed 2>.com
Role name: AD trust agent
Role status: absent
Server name: romulus.<removed>.<removed 2>.com
Role name: AD trust agent
Role status: absent
Server name: starfleet.<removed>.<removed 2>.com
Role name: AD trust agent
Role status: absent
Server name: federation.<removed>.<removed 2>.com
Role name: KRA server
Role status: absent
Server name: romulus.<removed>.<removed 2>.com
Role name: KRA server
Role status: absent
Server name: starfleet.<removed>.<removed 2>.com
Role name: KRA server
Role status: absent
Server name: federation.<removed>.<removed 2>.com
Role name: AD trust controller
Role status: absent
Server name: romulus.<removed>.<removed 2>.com
Role name: AD trust controller
Role status: absent
Server name: starfleet.<removed>.<removed 2>.com
Role name: AD trust controller
Role status: absent
1) Are there other tests, reports, data that I can perform/provide with the systems in the “pre-upgrade” state that can help prove out the “pre-upgrade” state of the systems? I did run
https://pypi.python.org/pypi/checkipaconsistency and cleaned up some stale RUVs, but they weren’t tried to an active replication agreement (previous “add” failures for the secondary and tertiary
server).
2) Are there other points not discussed in https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ that I should be looking for in the event the service fails to start again?
3) Is there a superset of the list of files I should look for when pki-tomcat fails to start?
3a) Is searching for stack traces and “error” (case insensitive) in the logs sufficient search patterns or are there other searches and data that need to be looked at?
Thank you in advance for the assistance,
Chris