On the failing node, the output of "getcert list" does not show any expired certs.  I have hand-copied the info info this email below (it's interesting to note that while the other IdM servers are tracking 9 certs, the problem server is only tracking 8):

Number of certificates and requests being tracked: 8
Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=<servername>,O=<domain>
subject: CN=<servername>,O=<domain>
expires: 2020-09-12 19:51:34 UTC
principal name: krbtgt/<domain>
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token=NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token=NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<domain>
subject: CN=CA Audit,O=<domain>
expires: 2021-08-10 17:20:21 UTC
key usage: digitialSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token=NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token=NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<domain>
subject: CN=OCSP Subsystem,O=<domain>
expires: 2021-08-10 17:19:42 UTC
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token=NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token=NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<domain>
subject: CN=CA Subsystem,O=<domain>
expires: 2021-08-10 17:19:51 UTC
key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token=NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token=NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<domain>
subject: CN=Certificate Authority,O=<domain>
expires: 2037-09-28 14:29:02 UTC
key usage: digitialSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/var/lib/ipa/ra-agent.key'
certificate: type=NSSDB,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<domain>
subject: CN=IPA RA,,O=<domain>
expires: 2021-08-10 17:20:41 UTC
key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-<domain>/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<domain>
subject: CN=<server>,O=<domain>
expires: 2021-09-09 19:53:33 UTC
principal name: ldap/<serverFQDN@domain>
key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv <domain>
track: yes
auto-renew: yes

Request ID '<###>':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-<domain>',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<domain>
subject: CN=<server>,O=<domain>
expires: 2021-09-09 19:51:45 UTC
principal name: HTTP/<serverFQDN@domain>
key usage: digitialSignature,nonRepudiation,keyEnchipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes

Thank you so much again!
Scot




From: Florence Blanc-Renaud <flo@redhat.com>
Sent: Thursday, August 6, 2020 2:46 AM
To: FreeIPA users list <freeipa-users@lists.fedorahosted.org>
Cc: Scott Z. <sudz28@hotmail.com>
Subject: Re: [Freeipa-users] Re: pki-tomcatd not starting
 
On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
> Thanks much for the assistance.ツ  Here is where I am with your suggestions:
> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n
> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old
> (almost a year old actually, I assume IPA only checks it when it first
> starts up so it didn't care that it was expired until the server was
> rebooted?)

certmonger checks the certificate validity periodically (configurable in
certmonger.conf) and tries multiple times to renew soon-to-expire certs.
The system probably had an issue that was not detected and the cert
reached its expiration date.

>
> 2) ran ipactl start --ignore-service-failures
>  ツ ツ ツ ツ ツ ツ  a. most services started, obviously pki-tomcatd did not
> 3) ran "kinit admin"
>  ツ ツ ツ ツ ツ ツ  a. was forced to change the password, but otherwise nothing happened
> 4) Ran "ipa config-show |grep -i master
>  ツ ツ ツ ツ ツ  a. I see that the IPA CA renewal master is a different idm machine.
> 5) Ran "getcert list | grep -E "Request|certificate:|expires:"
>  ツ ツ ツ ツ ツ  a.I see all certs are currently valid (none expired)
> 6) Ran the command "getcert list" on the problem server, but I cannot
> paste the output here because it's on an airgaped environment so while I
> apologize for this and realize it makes things more difficult, perhaps
> if you tell me what I should be looking for or more specifically what
> you're interested in I can pluck that out and manually include it here?
> So in summary, it is indeed an expired "Server-Cert cert-pki-ca'
> certificate on the problem server, and it can theoretically be renew by
> the Master at this time.
The interesting part is the list of expired certs on the failing node
(is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
instructions are available here:
https://access.redhat.com/solutions/3357331 How do I manually renew
Identity Management (IPA) certificates on RHEL7 after they have expired?
(Replica IPA Server)

flo

> Many thanks!
> Scott
>
> ------------------------------------------------------------------------
> *From:* Florence Blanc-Renaud <flo@redhat.com>
> *Sent:* Monday, August 3, 2020 9:34 PM
> *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>
> *Cc:* Scott Z. <sudz28@hotmail.com>
> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>> Not sure I'm sending this to the right place, but here it goes.テつ  I
>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
>> access) environment that is running into problems.テつ  There are at least 3
>> different IdM servers running in the environment spread out across
>> different geographical areas.テつ  One of those areas suffered an unschedule
>> power outage recently, and ever since we brought everything back up, the
>> IdM server for this region is having an issue.テつ  Please bear with me as I
>> have zero formal experience, training, or real knowledge with IdM.
>>
>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
>> "ipactl status" and it shows "Directory Service: STOPPED".テつ  I then run
>> "ipactl restart", and things go fine until it gets to "Starting
>> pki-tomcatd Service", where it hangs for quite some time before failing
>> to start and killing all the other services.テつ  I check the log at
>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>> (forgive any mistypings, I have to manually type these in as I can't
>> import or screen capure the logs and put them in this message):
>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
>> Invalid certificate: (-8181) Peer's Certificate has expired/"
>> And slightly further down in the same log:
>> "/Cannot reset factory: connections not all returned/"
>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
>> LDAP connection factory because some connections are still outstanding/"
>> ... still further down"
>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>>
>> Assuming I have some weird certificate issue with this server in
>> particular, I try to run a few more commands:
>> "certutil -L -d /etc/httpd/alias"テつ  --> returns a Server-Cert listing
>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
>> for it's attributes.テつ  Comparing to a second IdM server in this
>> environment, it seems to be missing a "Signing-Cert"?
>>
> Hi,
> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
> has the nickname 'Server-Cert cert-pki-ca'. You should check that this
> one is not expired with:
> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
> | grep 'Not '
>
> If the certificate is indeed expired, it will have to be renewed but you
> need first to find which IPA server is the CA renewal master. On your
> server, force a service start and check the CA renewal master:
> # ipactl start --ignore-service-failures
> # kinit admin
> # ipa config-show | grep "renewal master"
>  ツ ツ  IPA CA renewal master: server.domain.com
>
> You need to make sure that all the certificates are valid on the CA
> renewal master:
> (on the CA renewal master)# getcert list | grep -E
> "Request|certificate:|expires:"
>
> - if the CA renewal master is not OK, please post the output of "#
> getcert list" (without the grep) on the CA renewal master. This node
> will have to be repaired first.
> - if the CA renewal master is OK, please post the output of "# getcert
> list" (also without the grep) on the failing node.
>
> We'll be able to help based on this information.
> flo
>
>> I also did a "getcert list", and all certs it has show that they expire
>> in the future (nothing shows as bein currently expired).
>>
>> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
>> but how do I track down which 'peer' the log file is talking about that
>> has an expired cert?テつ  Meanwhile none of the linux clients that point to
>> this IdM server are allowing people to log in/authenticate.
>> Many thanks for any help!
>> Scott
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>