I see some one else opened another thread with similar issue in the last few hours or so, but the error messages are different so I'm going ahead & seeking help on a new thread.
 
I've inherited a FreeIPA installation from somebody used among 5 physical servers with one FreeIPA server (everything CA etc on it) while other 4 physical servers act as clients. Being someone very new at LDAP & FreeIPA, I tried to troublshoot by googling. 
 
System / Server Info:
 
OS - CentOS 7.6, Installed IPA packages version - 4.6.4, Self-Signed CA 
 
Here are the issues & what steps I've taken so far. 
 
1) Before certificates were expired the pki-tomcatd service was failing & I see the following message in /var/log/pki/pki-tomcat/ca/debug:
            Error: netscape.ldap.LDAPException: Authentication failed (48)
 After some googling I've found this link (https://access.redhat.com/solutions/3081821) which asks to check if certificate blob & serial number in pkiuser matches to the 'subsystemCert cert-pki-ca' in our case it does so there was nothing to do but we still get that error.
 
2) Certificates have expired - Now the certificates have expired, they were not auto-renewed, was it because above (pki-tomcatd service failure) not sure. 
 
    2a) For this I've tried to move back the date & tried to renew them through ipa-certupdate, the output says sucessfull but the certificates are not getting renewed. Here is the output of one such output(renamed domain to ourorg.com for privacy).
 
         ipapython.admintool: DEBUG: Not logging to a file 
        ipalib.plugable: DEBUG: importing all plugin modules in ipaclient.remote_plugins.schema$5131ac65... 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.remote_plugins.schema$5131ac65.plugins 
        ipalib.plugable: DEBUG: importing all plugin modules in ipaclient.plugins... 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.automember 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.automount 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.ca 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.cert 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.certmap 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.certprofile 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.csrgen 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.dns 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.hbacrule 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.hbactest 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.host 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.idrange 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.internal 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.location 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.migration 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.misc 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.otptoken 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.otptoken_yubikey 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.passwd 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.permission 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.rpcclient 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.server 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.service 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.sudorule 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.topology 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.trust 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.user 
        ipalib.plugable: DEBUG: importing plugin module ipaclient.plugins.vault 
        ipalib.rpc: INFO: trying https://login1.ourorg.com/ipa/json 
        ipalib.backend: DEBUG: Created connection context.rpcclient_139790894262416 
        ipalib.install.kinit: DEBUG: Initializing principal host/login1.ourorg.com@ourorg.COM using keytab /etc/krb5.keytab 
        ipalib.install.kinit: DEBUG: using ccache /tmp/tmp-O7QeRu/ccache 
        ipalib.install.kinit: DEBUG: Attempt 1/1: success 
        ipalib.frontend: DEBUG: raw: ca_is_enabled(version=u'2.107') 
        ipalib.frontend: DEBUG: ca_is_enabled(version=u'2.107') 
        ipalib.rpc: INFO: [try 1]: Forwarding 'ca_is_enabled/1' to json server 'https://login1.ourorg.com/ipa/json' 
        ipalib.rpc: DEBUG: New HTTP connection (login1.ourorg.com) 
        ipalib.rpc: DEBUG: received Set-Cookie (<type 'list'>)'['ipa_session=MagBearerToken=3bDSVwqoHDuM1MRVLGVRKY2DhplAszGxcdGLUBtRRZTLVV3vj8%2bNHrexIE9KX2JdrFkcYUtCfGkQmUVoYuCUj4DRqwJBoe9Z7i3J14DadLtOVCi2fNwxNR8irDD%2fG2bn4T7ULiLR6b7k1dpS%2bXWo
        iJGHOknn5EYLzi0wEOz88PauUZ7Qh1HioKfddyQhOLl1kQ6LnAsu%2fm2cACveJ8JSe2Mfmqruu8a%2fbQAIXPmRwXnC5oGN8cIk0omO4KuFQaRHWmjSNiLyG1%2bdyPiyWlxKBw%3d%3d;path=/ipa;httponly;secure;']' 
        ipalib.rpc: DEBUG: storing cookie 'ipa_session=MagBearerToken=3bDSVwqoHDuM1MRVLGVRKY2DhplAszGxcdGLUBtRRZTLVV3vj8%2bNHrexIE9KX2JdrFkcYUtCfGkQmUVoYuCUj4DRqwJBoe9Z7i3J14DadLtOVCi2fNwxNR8irDD%2fG2bn4T7ULiLR6b7k1dpS%2bXWoiJGHOknn5EYLzi0wEOz88P
        auUZ7Qh1HioKfddyQhOLl1kQ6LnAsu%2fm2cACveJ8JSe2Mfmqruu8a%2fbQAIXPmRwXnC5oGN8cIk0omO4KuFQaRHWmjSNiLyG1%2bdyPiyWlxKBw%3d%3d;' for principal None 
        ipapython.ipaldap: DEBUG: retrieving schema for SchemaCache url=ldap://login1.ourorg.com:389 conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f239a5242d8> 
        ipalib.frontend: DEBUG: raw: ca_find(None, version=u'2.230') 
        ipalib.frontend: DEBUG: ca_find(None, version=u'2.230') 
        ipalib.rpc: INFO: [try 1]: Forwarding 'ca_find/1' to json server 'https://login1.ourorg.com/ipa/json' 
        ipalib.rpc: DEBUG: HTTP connection keep-alive (login1.ourorg.com) 
        ipalib.rpc: DEBUG: received Set-Cookie (<type 'list'>)'['ipa_session=MagBearerToken=kmtXWE4j%2buLPMXwC6RCOBvqfLCIBziy9XiM7f%2fep%2b7FYBiSPmVPwjf6USK94djhkQ6k0Rleh9KhokFWNf1AWxcH5SyVe5V6QZYLIIGzt%2fF%2f1mHl3uKOLocAauyCAz%2bVxm2FUG%2fR8ORi5
        YghKrOidtRk%2bQvERwvHJKOJ8jjikvPzlWcj1x8CjO1b6ricWSigD3%2bl1UbPEYTOMKxNSL0JEW8Q0ghkPt1bryt9aEuWZVRBU%2f%2fAYnQN6WgYkrvgyBBeYXuceYPKQFtpxUmnl2js%2bDg%3d%3d;path=/ipa;httponly;secure;']' 
        ipalib.rpc: DEBUG: storing cookie 'ipa_session=MagBearerToken=kmtXWE4j%2buLPMXwC6RCOBvqfLCIBziy9XiM7f%2fep%2b7FYBiSPmVPwjf6USK94djhkQ6k0Rleh9KhokFWNf1AWxcH5SyVe5V6QZYLIIGzt%2fF%2f1mHl3uKOLocAauyCAz%2bVxm2FUG%2fR8ORi5YghKrOidtRk%2bQvERwvHJ
        KOJ8jjikvPzlWcj1x8CjO1b6ricWSigD3%2bl1UbPEYTOMKxNSL0JEW8Q0ghkPt1bryt9aEuWZVRBU%2f%2fAYnQN6WgYkrvgyBBeYXuceYPKQFtpxUmnl2js%2bDg%3d%3d;' for principal None 
        ipalib.install.sysrestore: DEBUG: Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-ourorg-COM -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/dirsrv/slapd-ourorg-COM/pwdfile.txt 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/dirsrv/slapd-ourorg-COM -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/dirsrv/slapd-ourorg-COM/pwdfile.txt 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/bin/systemctl is-active dirsrv@ourorg-COM.service 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout=active 
 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/bin/systemctl --system daemon-reload 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/bin/systemctl restart dirsrv@ourorg-COM.service 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/bin/systemctl is-active dirsrv@ourorg-COM.service 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout=active 
 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: wait_for_open_ports: localhost [389] timeout 300 
        ipapython.ipautil: DEBUG: waiting for port: 389 
        ipapython.ipautil: DEBUG: SUCCESS: port: 389 
        ipaplatform.base.services: DEBUG: Restart of dirsrv@ourorg-COM.service complete 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/httpd/alias -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/httpd/alias/pwdfile.txt 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/httpd/alias -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/httpd/alias/pwdfile.txt 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/bin/systemctl is-active httpd.service 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout=active 
 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/bin/systemctl restart httpd.service 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/bin/systemctl is-active httpd.service 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout=active 
 
        ipapython.ipautil: DEBUG: stderr= 
        ipaplatform.base.services: DEBUG: Restart of httpd.service complete 
        ipaclient.install.ipa_certupdate: DEBUG: resubmitting certmonger request '20190129222612' 
        ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'GENERATING_CSR', variant_level=1) 
        ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) 
        ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) 
        ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) 
        ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) 
        ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) 
        ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) 
        ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'POST_SAVED_CERT', variant_level=1) 
        ipalib.install.certmonger: DEBUG: certmonger request is in state dbus.String(u'MONITORING', variant_level=1) 
        ipaclient.install.ipa_certupdate: DEBUG: modifying certmonger request '20190129222612' 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -L -n IPA CA -a -f /etc/ipa/nssdb/pwdfile.txt 
        ipapython.ipautil: DEBUG: Process finished, return code=255 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: IPA CA 
        : PR_FILE_NOT_FOUND_ERROR: File not found 
 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -L -n External CA cert -a -f /etc/ipa/nssdb/pwdfile.txt 
        ipapython.ipautil: DEBUG: Process finished, return code=255 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: External CA cert 
        : PR_FILE_NOT_FOUND_ERROR: File not found 
 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/ipa/nssdb -A -n ourorg.COM IPA CA -t CT,C,C -a -f /etc/ipa/nssdb/pwdfile.txt 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/usr/bin/update-ca-trust 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipaplatform.redhat.tasks: INFO: Systemwide CA database updated. 
        ipapython.ipautil: DEBUG: Starting external process 
        ipapython.ipautil: DEBUG: args=/usr/bin/update-ca-trust 
        ipapython.ipautil: DEBUG: Process finished, return code=0 
        ipapython.ipautil: DEBUG: stdout= 
        ipapython.ipautil: DEBUG: stderr= 
        ipaplatform.redhat.tasks: INFO: Systemwide CA database updated. 
        ipalib.backend: DEBUG: Destroyed connection context.rpcclient_139790894262416 
        ipapython.admintool: INFO: The ipa-certupdate command was successful
 
     In above output there are two occasions where it is mentioned "ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: IPA CA    : PR_FILE_NOT_FOUND_ERROR: File not found" not sure if these are relevant, if so how to debug
 
   2b) I've also used "ipa-cacert-manage renew" following this link https://www.freeipa.org/page/V4/CA_certificate_renewal. Not sure if this was necessary or if doing this caused any more issues.
 
 3) Since certificates have expired kerberos broke, can't do "kinit admin" any longer. Can't change passwords / create users & of course can't access webui. For any of these actions need to move the date back. For now 'sudo' works (without having to move the date back) & general logins work, but not sure how long they continue to work before completely break?.
 
4) This is a production installation with hardly any time to take down FreeIPA let alone physical server. Is there any way to recover from this situation?. 
 
5) If it can't be recovered can we setup another FreeIPA server installation with the same realm / domain(need to procure another system /server) with a new CA & etc from scratch and make all the current 5 physical servers (including current broken FreeIPA server) as clients to the new FreeIPA installation with the same domain / realm?.