Thanks much for the assistance.  Here is where I am with your suggestions:
1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old (almost a year old actually, I assume IPA only checks it when it first starts up so it didn't care that it was expired until the server was rebooted?)

2) ran ipactl start --ignore-service-failures
       a. most services started, obviously pki-tomcatd did not
3) ran "kinit admin"
       a. was forced to change the password, but otherwise nothing happened
4) Ran "ipa config-show |grep -i master
      a. I see that the IPA CA renewal master is a different idm machine.
5) Ran "getcert list | grep -E "Request|certificate:|expires:"
      a.I see all certs are currently valid (none expired)
6) Ran the command "getcert list" on the problem server, but I cannot paste the output here because it's on an airgaped environment so while I apologize for this and realize it makes things more difficult, perhaps if you tell me what I should be looking for or more specifically what you're interested in I can pluck that out and manually include it here?
So in summary, it is indeed an expired "Server-Cert cert-pki-ca' certificate on the problem server, and it can theoretically be renew by the Master at this time.
Many thanks!
Scott


From: Florence Blanc-Renaud <flo@redhat.com>
Sent: Monday, August 3, 2020 9:34 PM
To: FreeIPA users list <freeipa-users@lists.fedorahosted.org>
Cc: Scott Z. <sudz28@hotmail.com>
Subject: Re: [Freeipa-users] pki-tomcatd not starting
 
On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
> Not sure I'm sending this to the right place, but here it goes.  I
> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
> access) environment that is running into problems.  There are at least 3
> different IdM servers running in the environment spread out across
> different geographical areas.  One of those areas suffered an unschedule
> power outage recently, and ever since we brought everything back up, the
> IdM server for this region is having an issue.  Please bear with me as I
> have zero formal experience, training, or real knowledge with IdM.
>
> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
> "ipactl status" and it shows "Directory Service: STOPPED".  I then run
> "ipactl restart", and things go fine until it gets to "Starting
> pki-tomcatd Service", where it hangs for quite some time before failing
> to start and killing all the other services.  I check the log at
> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
> (forgive any mistypings, I have to manually type these in as I can't
> import or screen capure the logs and put them in this message):
> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
> Invalid certificate: (-8181) Peer's Certificate has expired/"
> And slightly further down in the same log:
> "/Cannot reset factory: connections not all returned/"
> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
> LDAP connection factory because some connections are still outstanding/"
> ... still further down"
> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>
> Assuming I have some weird certificate issue with this server in
> particular, I try to run a few more commands:
> "certutil -L -d /etc/httpd/alias"  --> returns a Server-Cert listing
> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
> for it's attributes.  Comparing to a second IdM server in this
> environment, it seems to be missing a "Signing-Cert"?
>
Hi,
PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
has the nickname 'Server-Cert cert-pki-ca'. You should check that this
one is not expired with:
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
| grep 'Not '

If the certificate is indeed expired, it will have to be renewed but you
need first to find which IPA server is the CA renewal master. On your
server, force a service start and check the CA renewal master:
# ipactl start --ignore-service-failures
# kinit admin
# ipa config-show | grep "renewal master"
   IPA CA renewal master: server.domain.com

You need to make sure that all the certificates are valid on the CA
renewal master:
(on the CA renewal master)# getcert list | grep -E
"Request|certificate:|expires:"

- if the CA renewal master is not OK, please post the output of "#
getcert list" (without the grep) on the CA renewal master. This node
will have to be repaired first.
- if the CA renewal master is OK, please post the output of "# getcert
list" (also without the grep) on the failing node.

We'll be able to help based on this information.
flo

> I also did a "getcert list", and all certs it has show that they expire
> in the future (nothing shows as bein currently expired).
>
> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
> but how do I track down which 'peer' the log file is talking about that
> has an expired cert?  Meanwhile none of the linux clients that point to
> this IdM server are allowing people to log in/authenticate.
> Many thanks for any help!
> Scott
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>