Hello,
I’m hoping somebody here can help with an issue I’m having with upgrades and ipaca. I have
(3) CentOS 7.1.1503 based systems that I’m trying to upgrade from 4.2.0-15.0.1.el7 to
ipa-server-4.5.4-10.el7. I’m able to upgrade the “second master” (dirsrv, DNS, ipaca
backup) and “third master” (dirsrv, DNS, apace backup) without an issue (replication is
good after 3-4 hours). But when I try to upgrade the “first master” (dirsrv, DNS, ipaca
primary) the upgrade process completes successfully and starts the services, but the
pki-tomcat fails to stay running. Odd thing is that it does run for about 4-5 minutes (I
can see certificate data, and can list certificates from the CLI), but after about 5
minutes the whole IPA system stops (per the systemctl). I can run the IPA services on the
“first master” (ipactl start --ignore-service-failures), but eventually the replication
for ipaca fails — I suppose this is expected since pki-tomcat isn’t running (LDAP
connection error from the “first master” to the “second/third masters”).
Funny thing is I’m not able to see anything in the logs that point to anything that shows
as a fault. In referencing
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...,
I wasn’t able to isolate anything that seems like it was related to the issue of
pki-tomcatd terminating.
I eventually reverted all hosts back to a “safe snapshot” and reverted to keep the
production systems active and in sync. I’m hoping the wise people here might be able to ID
something amiss in the currently running systems before I make another attempt to get the
systems upgraded again, or perhaps suggest superset of logs and data-points I would need
to gather if it was to fail again.
Per
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
—
/var/log/ipaupgrade.log: Completed successfully
/var/log/pki/pki-tomcat/ca/debug: No trace/stack outputs. No strings marked as “error”.
Also checked catalina.out, and didn’t see anything amiss. There was a trace for a missing
module (can’t remember the name right now and it isn’t in my notes), but the services and
webapps started without it.
Certificate cert-pki-ca:
sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' | grep
'Not'
Not Before: Sun Aug 20 22:02:05 2017
Not After : Sat Aug 10 22:02:05 2019
sudo certutil -L -d /etc/dirsrv/slapd-<removed>-<removed 2>-COM/ -n
Server-Cert |grep "Not "
Not Before: Thu Aug 31 22:02:18 2017
Not After : Sun Sep 01 22:02:18 2019
sudo certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep "Not "
Not Before: Thu Aug 31 22:02:08 2017
Not After : Sun Sep 01 22:02:08 2019
Was able to read cert using the password in /var/lib/pki/pki-tomcat/conf/password.conf.
In LDAP uid=pkidbuser,ou=people,o=ipaca userCertificate appears to be valid and matches
what is in the NSSDB.
There are (8) certificates being monitored, and none have expired —
sudo getcert list
Number of certificates and requests being tracked: 8.
Request ID '20150928161427':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed
2>-COM',nickname='Server-Cert',token='NSS Certificate
DB',pinfile='/etc/dirsrv/slapd-<removed>-<removed
2>-COM/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-<removed>-<removed
2>-COM',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=starfleet.<removed>.<removed
2>.com,O=<removed>.<removed
2>.COM
expires: 2019-09-01 22:02:18 UTC
principal name: ldap/starfleet.<removed>.<removed
2>.com(a)<removed>.<removed
2>.COM
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv
<removed>-<removed 2>-COM
track: yes
auto-renew: yes
Request ID '20150928161756':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=starfleet.<removed>.<removed
2>.com,O=<removed>.<removed
2>.COM
expires: 2019-09-01 22:02:08 UTC
principal name: HTTP/starfleet.<removed>.<removed
2>.com(a)<removed>.<removed
2>.COM
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Request ID '20160725201511':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=CA Audit,O=<removed>.<removed
2>.COM
expires: 2019-08-10 22:04:31 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201512':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=OCSP Subsystem,O=<removed>.<removed
2>.COM
expires: 2019-08-10 22:02:50 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201513':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=CA Subsystem,O=<removed>.<removed
2>.COM
expires: 2019-08-10 22:02:11 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
"subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201514':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=Certificate Authority,O=<removed>.<removed
2>.COM
expires: 2035-09-28 16:13:20 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
"caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160725201515':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=IPA RA,O=<removed>.<removed
2>.COM
expires: 2019-08-10 22:04:11 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/lib64/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20160725201516':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-renew-agent
issuer: CN=Certificate Authority,O=<removed>.<removed
2>.COM
subject: CN=starfleet.<removed>.<removed
2>.com,O=<removed>.<removed
2>.COM
expires: 2019-08-10 22:02:05 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert
"Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Replication notes –
ipa server-role-find
-----------------------
18 server roles matched
-----------------------
Server name: federation.<removed>.<removed
2>.com
Role name: CA server
Role status: enabled
Server name: romulus.<removed>.<removed
2>.com
Role name: CA server
Role status: enabled
Server name: starfleet.<removed>.<removed
2>.com
Role name: CA server
Role status: enabled
Server name: federation.<removed>.<removed
2>.com
Role name: DNS server
Role status: enabled
Server name: romulus.<removed>.<removed
2>.com
Role name: DNS server
Role status: enabled
Server name: starfleet.<removed>.<removed
2>.com
Role name: DNS server
Role status: enabled
Server name: federation.<removed>.<removed
2>.com
Role name: NTP server
Role status: enabled
Server name: romulus.<removed>.<removed
2>.com
Role name: NTP server
Role status: enabled
Server name: starfleet.<removed>.<removed
2>.com
Role name: NTP server
Role status: absent
Server name: federation.<removed>.<removed
2>.com
Role name: AD trust agent
Role status: absent
Server name: romulus.<removed>.<removed
2>.com
Role name: AD trust agent
Role status: absent
Server name: starfleet.<removed>.<removed
2>.com
Role name: AD trust agent
Role status: absent
Server name: federation.<removed>.<removed
2>.com
Role name: KRA server
Role status: absent
Server name: romulus.<removed>.<removed
2>.com
Role name: KRA server
Role status: absent
Server name: starfleet.<removed>.<removed
2>.com
Role name: KRA server
Role status: absent
Server name: federation.<removed>.<removed
2>.com
Role name: AD trust controller
Role status: absent
Server name: romulus.<removed>.<removed
2>.com
Role name: AD trust controller
Role status: absent
Server name: starfleet.<removed>.<removed
2>.com
Role name: AD trust controller
Role status: absent
1) Are there other tests, reports, data that I can perform/provide with the systems in the
“pre-upgrade” state that can help prove out the “pre-upgrade” state of the systems? I did
run
https://pypi.python.org/pypi/checkipaconsistency and cleaned up some stale RUVs, but
they weren’t tried to an active replication agreement (previous “add” failures for the
secondary and tertiary server).
2) Are there other points not discussed in
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
that I should be looking for in the event the service fails to start again?
3) Is there a superset of the list of files I should look for when pki-tomcat fails to
start?
3a) Is searching for stack traces and “error” (case insensitive) in the logs sufficient
search patterns or are there other searches and data that need to be looked at?
Thank you in advance for the assistance,
Chris