Hello,
I’m hoping somebody here can help with an issue I’m having with upgrades
and ipaca. I have (3) CentOS 7.1.1503 based systems that I’m trying to
upgrade from 4.2.0-15.0.1.el7 to ipa-server-4.5.4-10.el7. I’m able to
upgrade the “second master” (dirsrv, DNS, ipaca backup) and “third
master” (dirsrv, DNS, apace backup) without an issue (replication is
good after 3-4 hours). But when I try to upgrade the “first master”
(dirsrv, DNS, ipaca primary) the upgrade process completes successfully
and starts the services, but the pki-tomcat fails to stay running. Odd
thing is that it does run for about 4-5 minutes (I can see certificate
data, and can list certificates from the CLI), but after about 5 minutes
the whole IPA system stops (per the systemctl). I can run the IPA
services on the “first master” (ipactl start --ignore-service-failures),
but eventually the replication for ipaca fails — I suppose this is
expected since pki-tomcat isn’t running (LDAP connection error from the
“first master” to the “second/third masters”).
Funny thing is I’m not able to see anything in the logs that point to
anything that shows as a fault. In referencing
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...,
I wasn’t able to isolate anything that seems like it was related to the
issue of pki-tomcatd terminating.
I eventually reverted all hosts back to a “safe snapshot” and reverted
to keep the production systems active and in sync. I’m hoping the wise
people here might be able to ID something amiss in the currently running
systems before I make another attempt to get the systems upgraded again,
or perhaps suggest superset of logs and data-points I would need to
gather if it was to fail again.
Per
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
—
/var/log/ipaupgrade.log: Completed successfully
/var/log/pki/pki-tomcat/ca/debug: No trace/stack outputs. No strings
marked as “error”.
Also checked catalina.out, and didn’t see anything amiss. There was a
trace for a missing module (can’t remember the name right now and it
isn’t in my notes), but the services and webapps started without it.
Certificate cert-pki-ca:
sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert
cert-pki-ca' | grep 'Not'
Not Before: Sun Aug 20 22:02:05 2017
Not After : Sat Aug 10 22:02:05 2019
sudo certutil -L -d /etc/dirsrv/slapd-<removed>-<removed 2>-COM/ -n
Server-Cert |grep "Not "
Not Before: Thu Aug 31 22:02:18 2017
Not After : Sun Sep 01 22:02:18 2019
sudo certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep "Not "
Not Before: Thu Aug 31 22:02:08 2017
Not After : Sun Sep 01 22:02:08 2019
Was able to read cert using the password in
/var/lib/pki/pki-tomcat/conf/password.conf.
In LDAP uid=pkidbuser,ou=people,o=ipaca userCertificate appears to be
valid and matches what is in the NSSDB.
There are (8) certificates being monitored, and none have expired —
sudo getcert list
Number of certificates and requests being tracked: 8.
Request ID '20150928161427':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed
2>-COM',nickname='Server-Cert',token='NSS Certificate
DB',pinfile='/etc/dirsrv/slapd-<removed>-<removed
2>-COM/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed
2>-COM',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=starfleet.<removed>.<removed
2>.com,O=<removed>.<removed
2>.COM
expires: 2019-09-01 22:02:18 UTC
principal name: ldap/starfleet.<removed>.<removed
2>.com(a)<removed>.<removed
2>.COM
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command:
/usr/lib64/ipa/certmonger/restart_dirsrv <removed>-<removed 2>-COM
track: yes
auto-renew: yes
Request ID '20150928161756':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=starfleet.<removed>.<removed
2>.com,O=<removed>.<removed
2>.COM
expires: 2019-09-01 22:02:08 UTC
principal name: HTTP/starfleet.<removed>.<removed
2>.com(a)<removed>.<removed
2>.COM
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Request ID '20160725201511':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=CA Audit,O=<removed>.<removed
2>.COM
expires: 2019-08-10 22:04:31 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command:
/usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201512':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS
Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS
Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=OCSP Subsystem,O=<removed>.<removed
2>.COM
expires: 2019-08-10 22:02:50 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command:
/usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201513':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=CA Subsystem,O=<removed>.<removed
2>.COM
expires: 2019-08-10 22:02:11 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command:
/usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201514':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=Certificate Authority,O=<removed>.<removed
2>.COM
expires: 2035-09-28 16:13:20 UTC
key usage:
digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command:
/usr/lib64/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201515':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=IPA RA,O=<removed>.<removed
2>.COM
expires: 2019-08-10 22:04:11 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
/usr/lib64/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20160725201516':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=starfleet.<removed>.<removed
2>.com,O=<removed>.<removed
2>.COM
expires: 2019-08-10 22:02:05 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command:
/usr/lib64/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Replication notes –
ipa server-role-find
-----------------------
18 server roles matched
-----------------------
Server name: federation.<removed>.<removed
2>.com
Role name: CA server
Role status: enabled
Server name: romulus.<removed>.<removed
2>.com
Role name: CA server
Role status: enabled
Server name: starfleet.<removed>.<removed
2>.com
Role name: CA server
Role status: enabled
Server name: federation.<removed>.<removed
2>.com
Role name: DNS server
Role status: enabled
Server name: romulus.<removed>.<removed
2>.com
Role name: DNS server
Role status: enabled
Server name: starfleet.<removed>.<removed
2>.com
Role name: DNS server
Role status: enabled
Server name: federation.<removed>.<removed
2>.com
Role name: NTP server
Role status: enabled
Server name: romulus.<removed>.<removed
2>.com
Role name: NTP server
Role status: enabled
Server name: starfleet.<removed>.<removed
2>.com
Role name: NTP server
Role status: absent
Server name: federation.<removed>.<removed
2>.com
Role name: AD trust agent
Role status: absent
Server name: romulus.<removed>.<removed
2>.com
Role name: AD trust agent
Role status: absent
Server name: starfleet.<removed>.<removed
2>.com
Role name: AD trust agent
Role status: absent
Server name: federation.<removed>.<removed
2>.com
Role name: KRA server
Role status: absent
Server name: romulus.<removed>.<removed
2>.com
Role name: KRA server
Role status: absent
Server name: starfleet.<removed>.<removed
2>.com
Role name: KRA server
Role status: absent
Server name: federation.<removed>.<removed
2>.com
Role name: AD trust controller
Role status: absent
Server name: romulus.<removed>.<removed
2>.com
Role name: AD trust controller
Role status: absent
Server name: starfleet.<removed>.<removed
2>.com
Role name: AD trust controller
Role status: absent
1) Are there other tests, reports, data that I can perform/provide with
the systems in the “pre-upgrade” state that can help prove out the
“pre-upgrade” state of the systems? I did run
https://pypi.python.org/pypi/checkipaconsistency and cleaned up some
stale RUVs, but they weren’t tried to an active replication agreement
(previous “add” failures for the secondary and tertiary server).
2) Are there other points not discussed in
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
that I should be looking for in the event the service fails to start again?
3) Is there a superset of the list of files I should look for when
pki-tomcat fails to start?
3a) Is searching for stack traces and “error” (case insensitive) in the
logs sufficient search patterns or are there other searches and data
that need to be looked at?
I think the dogtag debug and selftest logs would be what you need to
examine.
rob