After rebooting my CentOS 7 IdM server, pki-tomcatd is failing to start.
I see this (repeated many times) in the journal:
WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@383171f8 background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:748)
getcert list shows a number of expired certificates (which is EXTREMELY frustrating, as I thought that certmonger, which is running, was supposed to take care of these renewals):
Request ID '20170306100908': status: CA_UNREACHABLE ca-error: Error 60 connecting to https://asterisk.penurio.us:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PENURIO.US subject: CN=CA Audit,O=PENURIO.US expires: 2017-06-19 16:27:30 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170306100911': status: CA_UNREACHABLE ca-error: Error 60 connecting to https://asterisk.penurio.us:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PENURIO.US subject: CN=OCSP Subsystem,O=PENURIO.US expires: 2017-06-19 16:26:30 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170306100914': status: CA_UNREACHABLE ca-error: Error 60 connecting to https://asterisk.penurio.us:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PENURIO.US subject: CN=CA Subsystem,O=PENURIO.US expires: 2017-06-19 16:26:30 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes
Is there a published procedure to fix this? (I did find a procedure for RHEL/CentOS 6 and IPA 3, on the Red Hat site, but I am using CentOS 7 with IPA 4.4.)
After rebooting my CentOS 7 IdM server, pki-tomcatd is failing to start.
I see this (repeated many times) in the journal:
WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@383171f8 background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) at java.lang.Thread.run(Thread.java:748)
getcert list shows a number of expired certificates (which is EXTREMELY frustrating, as I thought that certmonger, which is running, was supposed to take care of these renewals):
Request ID '20170306100908': status: CA_UNREACHABLE ca-error: Error 60 connecting to https://asterisk.penurio.us:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PENURIO.US subject: CN=CA Audit,O=PENURIO.US expires: 2017-06-19 16:27:30 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170306100911': status: CA_UNREACHABLE ca-error: Error 60 connecting to https://asterisk.penurio.us:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PENURIO.US subject: CN=OCSP Subsystem,O=PENURIO.US expires: 2017-06-19 16:26:30 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170306100914': status: CA_UNREACHABLE ca-error: Error 60 connecting to https://asterisk.penurio.us:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PENURIO.US subject: CN=CA Subsystem,O=PENURIO.US expires: 2017-06-19 16:26:30 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes
I have tried setting the clock back 48 hours, but certmonger is still unable to renew the certificates -- still with the same error.
I have checked the certificates returned when connecting to asterisk.penurio.us:8443, and they look correct. The CA certificate doesn't expire until 2033, and the server certificate (whose CN is asterisk.penurio.us) expires in 2019.
Ian Pilcher via FreeIPA-users wrote:
After rebooting my CentOS 7 IdM server, pki-tomcatd is failing to start.
I see this (repeated many times) in the journal:
WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@383171f8 background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
at java.lang.Thread.run(Thread.java:748)
getcert list shows a number of expired certificates (which is EXTREMELY frustrating, as I thought that certmonger, which is running, was supposed to take care of these renewals):
Request ID '20170306100908': status: CA_UNREACHABLE ca-error: Error 60 connecting to https://asterisk.penurio.us:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PENURIO.US subject: CN=CA Audit,O=PENURIO.US expires: 2017-06-19 16:27:30 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170306100911': status: CA_UNREACHABLE ca-error: Error 60 connecting to https://asterisk.penurio.us:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PENURIO.US subject: CN=OCSP Subsystem,O=PENURIO.US expires: 2017-06-19 16:26:30 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20170306100914': status: CA_UNREACHABLE ca-error: Error 60 connecting to https://asterisk.penurio.us:8443/ca/agent/ca/profileReview: Peer certificate cannot be authenticated with given CA certificates. stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=PENURIO.US subject: CN=CA Subsystem,O=PENURIO.US expires: 2017-06-19 16:26:30 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes
I have tried setting the clock back 48 hours, but certmonger is still unable to renew the certificates -- still with the same error.
I have checked the certificates returned when connecting to asterisk.penurio.us:8443, and they look correct. The CA certificate doesn't expire until 2033, and the server certificate (whose CN is asterisk.penurio.us) expires in 2019.
Are these three the only expired certs?
What version of IPA?
Did you restart IPA after going back in time? If not, try that, then restart certmonger and it should renew the certs.
Given certmonger didn't fire in the very recent past can you check the syslog for any certmonger-related messages? I assume it renewed some, but not all of the certs?
rob
On 06/20/2017 10:38 PM, Rob Crittenden wrote:
Are these three the only expired certs?
For now ...
What version of IPA?
ipa-server-4.4.0-14.el7.centos.7.x86_64
Did you restart IPA after going back in time? If not, try that, then restart certmonger and it should renew the certs.
Definitely tried that, several times.
Given certmonger didn't fire in the very recent past can you check the syslog for any certmonger-related messages? I assume it renewed some, but not all of the certs?
I did finally managed to get a more useful error message by following the debugging hints here:
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-...
I get this in /var/log/ipa/renew.log:
* Trying 172.31.255.1... * Connected to asterisk.penurio.us (172.31.255.1) port 8443 (#0) * Initializing NSS with certpath: sql:/etc/httpd/alias * CAfile: /etc/ipa/ca.crt CApath: none * Server certificate: * subject: CN=asterisk.penurio.us,O=PENURIO.US * start date: Feb 27 23:37:03 2017 GMT * expire date: Feb 17 23:37:03 2019 GMT * common name: asterisk.penurio.us * issuer: CN=Certificate Authority,O=PENURIO.US * NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER) * Peer's certificate issuer has been marked as not trusted by the user. * Closing connection 0
And, sure enough, I am able to reproduce the behavior with curl:
# SSL_DIR=/etc/httpd/alias curl -v -o /dev/null --cacert /etc/ipa/ca.crt https://asterisk.penurio.us:8443/ca/agent/ca/profileReview
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* About to connect() to asterisk.penurio.us port 8443 (#0) * Trying 172.31.255.1... * Connected to asterisk.penurio.us (172.31.255.1) port 8443 (#0) * Initializing NSS with certpath: sql:/etc/httpd/alias * CAfile: /etc/ipa/ca.crt CApath: none * Server certificate: * subject: CN=asterisk.penurio.us,O=PENURIO.US * start date: Feb 27 23:37:03 2017 GMT * expire date: Feb 17 23:37:03 2019 GMT * common name: asterisk.penurio.us * issuer: CN=Certificate Authority,O=PENURIO.US * NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER) * Peer's certificate issuer has been marked as not trusted by the user. 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 * Closing connection 0 curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
If I don't specify the SSL_DIR, the curl command works, so it definitely seems to be an issue with the NSS database in /etc/httpd/alias. I don't see anything obviously wrong with the trust flags, though:
# certutil -d /etc/httpd/alias -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u ipaCert u,u,u PENURIO.US IPA CA CT,C,C Let's Encrypt Authority X3 - Digital Signature Trust Co. ,, www.penurio.us u,u,u
On 06/20/2017 11:38 PM, Ian Pilcher wrote:
If I don't specify the SSL_DIR, the curl command works, so it definitely seems to be an issue with the NSS database in /etc/httpd/alias. I don't see anything obviously wrong with the trust flags, though:
# certutil -d /etc/httpd/alias -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u ipaCert u,u,u PENURIO.US IPA CA CT,C,C Let's Encrypt Authority X3 - Digital Signature Trust Co. ,, www.penurio.us u,u,u
Trial and error for the win!
It seems as if the NSS database in /etc/httpd/alias had become subtly corrupted, so that the trust flags shown by certutil for the CA certificate were not accurate.
After clearing (-t ',,') and resetting (-t 'C,C,C') the trust flags, curl works, and certmonger has renewed my expired certificates.
That was not fun.
Ian Pilcher via FreeIPA-users wrote:
On 06/20/2017 11:38 PM, Ian Pilcher wrote:
If I don't specify the SSL_DIR, the curl command works, so it definitely seems to be an issue with the NSS database in /etc/httpd/alias. I don't see anything obviously wrong with the trust flags, though:
# certutil -d /etc/httpd/alias -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u ipaCert u,u,u PENURIO.US IPA CA CT,C,C Let's Encrypt Authority X3 - Digital Signature Trust Co. ,, www.penurio.us u,u,u
Trial and error for the win!
It seems as if the NSS database in /etc/httpd/alias had become subtly corrupted, so that the trust flags shown by certutil for the CA certificate were not accurate.
After clearing (-t ',,') and resetting (-t 'C,C,C') the trust flags, curl works, and certmonger has renewed my expired certificates.
That was not fun.
Well, I'm glad it's working, but I'm confused by your setup. Are you still using the Apache Server-Cert or are you using the Let's Encrypt cert? If the latter then you should disable tracking on Server-Cert. Off the top of my head I can't think of any issues it might cause but it is very possible some IPA renewal script dropped the trust on the Let's Encrypt CA since it isn't in the chain of the Server-Cert (or ipaCert).
rob
On 06/21/2017 08:54 AM, Rob Crittenden wrote:
Ian Pilcher via FreeIPA-users wrote:
On 06/20/2017 11:38 PM, Ian Pilcher wrote:
# certutil -d /etc/httpd/alias -L
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert u,u,u ipaCert u,u,u PENURIO.US IPA CA CT,C,C Let's Encrypt Authority X3 - Digital Signature Trust Co. ,, www.penurio.us u,u,u
Well, I'm glad it's working, but I'm confused by your setup. Are you still using the Apache Server-Cert or are you using the Let's Encrypt cert? If the latter then you should disable tracking on Server-Cert. Off the top of my head I can't think of any issues it might cause but it is very possible some IPA renewal script dropped the trust on the Let's Encrypt CA since it isn't in the chain of the Server-Cert (or ipaCert).
The Let's Encrypt intermediate CA certificate and the www.penurio.us certificate (issued by Let's Encrypt) are used only for an Internet- facing reverse proxy virtual host. They are not used for anything IPA- related.
The issue seems to have been the trust flags on the PENURIO.US IPA CA certificate. For whatever reason, it wasn't being trusted even though certutil was showing it as CT,C,C.
I "reset" the trust flags by running:
certutil -d /etc/httpd/alias -M -n 'PENURIO.US IPA CA' -t ',,' certutil -d /etc/httpd/alias -M -n 'PENURIO.US IPA CA' -t 'C,C,C'
And things started working.
I did find it interesting that the trust flags still showed as T,, after I ran the first command, and it's showing as CT,C,C now. It appears that certutil is either not affecting the T flag, or it is not displaying the trusts accurately.
freeipa-users@lists.fedorahosted.org
-
Ian Pilcher -
Rob Crittenden