Hello, 

 

I’m hoping somebody here can help with an issue I’m having with upgrades and ipaca. I have (3) CentOS 7.1.1503  based systems that I’m trying to upgrade from 4.2.0-15.0.1.el7 to ipa-server-4.5.4-10.el7. I’m able to upgrade the “second master” (dirsrv, DNS, ipaca backup) and “third master” (dirsrv, DNS, apace backup) without an issue (replication is good after 3-4 hours). But when I try to upgrade the “first master” (dirsrv, DNS, ipaca primary) the upgrade process completes successfully and starts the services, but the pki-tomcat fails to stay running. Odd thing is that it does run for about 4-5 minutes (I can see certificate data, and can list certificates from the CLI), but after about 5 minutes the whole IPA system stops (per the systemctl). I can run the IPA services on the “first master” (ipactl start --ignore-service-failures), but eventually the replication for ipaca fails — I suppose this is expected since pki-tomcat isn’t running (LDAP connection error from the “first master” to the “second/third masters”).

 

Funny thing is I’m not able to see anything in the logs that point to anything that shows as a fault. In referencing https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/, I wasn’t able to isolate anything that seems like it was related to the issue of pki-tomcatd terminating.

 

I eventually reverted all hosts back to a “safe snapshot” and reverted to keep the production systems active and in sync. I’m hoping the wise people here might be able to ID something amiss in the currently running systems before I make another attempt to get the systems upgraded again, or perhaps suggest superset of logs and data-points I would need to gather if it was to fail again.

 

Per https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ 

 

/var/log/ipaupgrade.log: Completed successfully

 

/var/log/pki/pki-tomcat/ca/debug: No trace/stack outputs. No strings marked as “error”.

 

Also checked catalina.out, and didn’t see anything amiss. There was a trace for a missing module (can’t remember the name right now and it isn’t in my notes), but the services and webapps started without it.  

 

Certificate cert-pki-ca: 

sudo certutil -L -d /etc/pki/pki-tomcat/alias  -n 'Server-Cert cert-pki-ca' | grep 'Not'

            Not Before: Sun Aug 20 22:02:05 2017

            Not After : Sat Aug 10 22:02:05 2019

 

sudo certutil -L -d /etc/dirsrv/slapd-<removed>-<removed 2>-COM/ -n Server-Cert |grep "Not "

            Not Before: Thu Aug 31 22:02:18 2017

            Not After : Sun Sep 01 22:02:18 2019

 

sudo certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep "Not "

            Not Before: Thu Aug 31 22:02:08 2017

            Not After : Sun Sep 01 22:02:08 2019

 

Was able to read cert using the password in /var/lib/pki/pki-tomcat/conf/password.conf.

 

In LDAP uid=pkidbuser,ou=people,o=ipaca userCertificate appears to be valid and matches what is in the NSSDB.

 

There are (8) certificates being monitored, and none have expired —

 

sudo getcert list

Number of certificates and requests being tracked: 8.

Request ID '20150928161427':

                status: MONITORING

                stuck: no

                key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed 2>-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-<removed>-<removed 2>-COM/pwdfile.txt'

                certificate: type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed 2>-COM',nickname='Server-Cert',token='NSS Certificate DB'

                CA: IPA

                issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM

                subject: CN=starfleet.<removed>.<removed 2>.com,O=<removed>.<removed 2>.COM

                expires: 2019-09-01 22:02:18 UTC

                principal name: ldap/starfleet.<removed>.<removed 2>.com@<removed>.<removed 2>.COM

                key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

                eku: id-kp-serverAuth,id-kp-clientAuth

                pre-save command: 

                post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv <removed>-<removed 2>-COM

                track: yes

                auto-renew: yes

Request ID '20150928161756':

                status: MONITORING

                stuck: no

                key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'

                certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'

                CA: IPA

                issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM

                subject: CN=starfleet.<removed>.<removed 2>.com,O=<removed>.<removed 2>.COM

                expires: 2019-09-01 22:02:08 UTC

                principal name: HTTP/starfleet.<removed>.<removed 2>.com@<removed>.<removed 2>.COM

                key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

                eku: id-kp-serverAuth,id-kp-clientAuth

                pre-save command: 

                post-save command: /usr/lib64/ipa/certmonger/restart_httpd

                track: yes

                auto-renew: yes

Request ID '20160725201511':

                status: MONITORING

                stuck: no

                key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set

                certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'

                CA: dogtag-ipa-ca-renew-agent

                issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM

                subject: CN=CA Audit,O=<removed>.<removed 2>.COM

                expires: 2019-08-10 22:04:31 UTC

                key usage: digitalSignature,nonRepudiation

                pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad

                post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"

                track: yes

                auto-renew: yes

Request ID '20160725201512':

                status: MONITORING

                stuck: no

                key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set

                certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'

                CA: dogtag-ipa-ca-renew-agent

                issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM

                subject: CN=OCSP Subsystem,O=<removed>.<removed 2>.COM

                expires: 2019-08-10 22:02:50 UTC

                eku: id-kp-OCSPSigning

                pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad

                post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"

                track: yes

                auto-renew: yes

Request ID '20160725201513':

                status: MONITORING

                stuck: no

                key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set

                certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'

                CA: dogtag-ipa-ca-renew-agent

                issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM

                subject: CN=CA Subsystem,O=<removed>.<removed 2>.COM

                expires: 2019-08-10 22:02:11 UTC

                key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

                eku: id-kp-serverAuth,id-kp-clientAuth

                pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad

                post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"

                track: yes

                auto-renew: yes

Request ID '20160725201514':

                status: MONITORING

                stuck: no

                key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set

                certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'

                CA: dogtag-ipa-ca-renew-agent

                issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM

                subject: CN=Certificate Authority,O=<removed>.<removed 2>.COM

                expires: 2035-09-28 16:13:20 UTC

                key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign

                pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad

                post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"

                track: yes

                auto-renew: yes

Request ID '20160725201515':

                status: MONITORING

                stuck: no

                key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'

                certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB'

                CA: dogtag-ipa-ca-renew-agent

                issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM

                subject: CN=IPA RA,O=<removed>.<removed 2>.COM

                expires: 2019-08-10 22:04:11 UTC

                key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

                eku: id-kp-serverAuth,id-kp-clientAuth

                pre-save command: /usr/lib64/ipa/certmonger/renew_ra_cert_pre

                post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert

                track: yes

                auto-renew: yes

Request ID '20160725201516':

                status: MONITORING

                stuck: no

                key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set

                certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'

                CA: dogtag-ipa-renew-agent

                issuer: CN=Certificate Authority,O=<removed>.<removed 2>.COM

                subject: CN=starfleet.<removed>.<removed 2>.com,O=<removed>.<removed 2>.COM

                expires: 2019-08-10 22:02:05 UTC

                key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

                eku: id-kp-serverAuth,id-kp-clientAuth

                pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad

                post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"

                track: yes

                auto-renew: yes

 

Replication notes –

 

ipa server-role-find

-----------------------

18 server roles matched

-----------------------

  Server name: federation.<removed>.<removed 2>.com

  Role name: CA server

  Role status: enabled

 

  Server name: romulus.<removed>.<removed 2>.com

  Role name: CA server

  Role status: enabled

 

  Server name: starfleet.<removed>.<removed 2>.com

  Role name: CA server

  Role status: enabled

 

  Server name: federation.<removed>.<removed 2>.com

  Role name: DNS server

  Role status: enabled

 

  Server name: romulus.<removed>.<removed 2>.com

  Role name: DNS server

  Role status: enabled

 

  Server name: starfleet.<removed>.<removed 2>.com

  Role name: DNS server

  Role status: enabled

 

  Server name: federation.<removed>.<removed 2>.com

  Role name: NTP server

  Role status: enabled

 

  Server name: romulus.<removed>.<removed 2>.com

  Role name: NTP server

  Role status: enabled

 

  Server name: starfleet.<removed>.<removed 2>.com

  Role name: NTP server

  Role status: absent

 

  Server name: federation.<removed>.<removed 2>.com

  Role name: AD trust agent

  Role status: absent

 

  Server name: romulus.<removed>.<removed 2>.com

  Role name: AD trust agent

  Role status: absent

 

  Server name: starfleet.<removed>.<removed 2>.com

  Role name: AD trust agent

  Role status: absent

 

  Server name: federation.<removed>.<removed 2>.com

  Role name: KRA server

  Role status: absent

 

  Server name: romulus.<removed>.<removed 2>.com

  Role name: KRA server

  Role status: absent

 

  Server name: starfleet.<removed>.<removed 2>.com

  Role name: KRA server

  Role status: absent

 

  Server name: federation.<removed>.<removed 2>.com

  Role name: AD trust controller

  Role status: absent

 

  Server name: romulus.<removed>.<removed 2>.com

  Role name: AD trust controller

  Role status: absent

 

  Server name: starfleet.<removed>.<removed 2>.com

  Role name: AD trust controller

  Role status: absent

 

 

1) Are there other tests, reports, data that I can perform/provide with the systems in the “pre-upgrade” state that can help prove out the “pre-upgrade” state of the systems? I did run https://pypi.python.org/pypi/checkipaconsistency and cleaned up some stale RUVs, but they weren’t tried to an active replication agreement (previous “add” failures for the secondary and tertiary server).

2) Are there other points not discussed in https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/ that I should be looking for in the event the service fails to start again?

3) Is there a superset of the list of files I should look for when pki-tomcat fails to start?

3a) Is searching for stack traces and “error” (case insensitive) in the logs sufficient search patterns or are there other searches and data that need to be looked at?

 

 

Thank you in advance for the assistance,

Chris