On Mon, Apr 30, 2018 at 11:49:09AM -0400, Brian Weaver via FreeIPA-users wrote:

After a recent power outage the IPA master server I built a few years ago is having some issues. I've done as much troubleshooting as I can and I think I've tracked down the issue to the certificate database in '/etc/pki/pki-tomcat/alias'. I can use 'certutil' to view a list of certificates. I can also view the key ID of the keys, when no nickname is used to specify a specific key. When I try to look at a specific key it fails.

[root@ipa-server0 alias]# certutil -d $PWD -L

Certificate Nickname Trust Attributes

SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca CTu,Cu,Cu subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu ocspSigningCert cert-pki-ca u,u,u

[root@ipa-server0 alias]# certutil -d $PWD -K -f /tmp/xxx certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa ab76588f20ba1e9d5f4dc4fe6f62dc70dc96484f NSS Certificate DB:auditSigningCert cert-pki-ca < 1> rsa ad2699ef775d3d685d08e6c34b64a02295d6bcef caSigningCert cert-pki-ca < 2> rsa a96b674224d50615416ef25644441887b410db3f (orphan) < 3> rsa 38b6a1d6d1be0dc2f80a2330cf52c73abd22d10d NSS Certificate DB:ocspSigningCert cert-pki-ca < 4> rsa 2beb83b689255e03be47430e204d34067fd873f8 NSS Certificate DB:Server-Cert cert-pki-ca < 5> rsa 0d733da9de0045c502dbb9f20ea8d4ba426afb47 NSS Certificate DB:subsystemCert cert-pki-ca

[root@ipa-server0 alias]# for i in $(certutil -d $PWD -L | grep cert-pki | awk '{print $1}') ; do certutil -d $PWD -K -f /tmp/xxx -n "$i cert-pki-ca" ; done certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" < 0> rsa ad2699ef775d3d685d08e6c34b64a02295d6bcef caSigningCert cert-pki-ca certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.

Does anyone have any suggestions on how to recover from this particular error. It would seem that some of the certificates were recently regenerated by certmonger based on these lines from the logging

Mar 30 07:26:30 ipa-server0.ipa.sunbirddcim.com certmonger[19770]: Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" will no Mar 30 07:26:30 ipa-server0.ipa.sunbirddcim.com certmonger[19769]: Certificate named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" will n Mar 30 07:26:30 ipa-server0.ipa.sunbirddcim.com certmonger[19772]: Certificate named "ipaCert" in token "NSS Certificate DB" in database "/etc/httpd/alias" will not be valid after 201804261702 Mar 30 07:26:30 ipa-server0.ipa.sunbirddcim.com certmonger[19773]: Certificate named "Server-Cert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" will not be Mar 30 07:28:57 ipa-server0.ipa.sunbirddcim.com certmonger[20025]: Certificate named "ipaCert" in token "NSS Certificate DB" in database "/etc/httpd/alias" issued by CA and saved. Mar 30 07:29:48 ipa-server0.ipa.sunbirddcim.com certmonger[20102]: Certificate named "auditSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" issued Mar 30 07:30:03 ipa-server0.ipa.sunbirddcim.com certmonger[20125]: Certificate named "ocspSigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" issued Mar 30 07:30:20 ipa-server0.ipa.sunbirddcim.com certmonger[20148]: Certificate named "subsystemCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-tomcat/alias" issued by Apr 10 07:26:31 ipa-server0.ipa.sunbirddcim.com certmonger[23627]: Certificate named "Server-Cert" in token "NSS Certificate DB" in database "/etc/dirsrv/slapd-IPA-SUNBIRDDCIM-COM" will not be Apr 10 07:27:23 ipa-server0.ipa.sunbirddcim.com certmonger[23724]: Certificate named "Server-Cert" in token "NSS Certificate DB" in database "/etc/httpd/alias" issued by CA and saved. Apr 10 07:27:40 ipa-server0.ipa.sunbirddcim.com certmonger[23783]: Certificate named "Server-Cert" in token "NSS Certificate DB" in database "/etc/dirsrv/slapd-IPA-SUNBIRDDCIM-COM" issued by C

I going to continue to try to muddle my way through it. I'm hoping someone with more knowledge than myself can help me find the correct path.

The result of `ipa --version` is VERSION: 4.3.1, API_VERSION: 2.164. The system is running Fedora 23 and FreeIPA came from a COPR release

name=Copr repo for freeipa-4-3 owned by @freeipa baseurl= https://copr-be.cloud.fedoraproject.org/results/@freeipa/freeipa-4-3/fedora-...

Any help would be greatly appreciated.

Hi Brian,

It is an interesting error. I've never seen it before (in this context).

Is the pki-tomcatd instance functional, e.g. can you execute `ipa cert-find`?

Can you print the certificates individually, i.e. `certutil -L -n "nickname"` for each nickname? What do those certs look like?

(After backing up the NSSDB) can you renew one of the problematic certificates again, using certmonger (`getcert resubmit`)?

Thanks, Fraser