I stopped the ntp service with the command "timedatectl set_ntp 0"
I set the new date to be Sept. 1st, 2019 with "timedatectl set-time 2019-09-01"
I waiting a minute and then checked with the "date" command; the problem server believes it is Sept. 1st, 2019.

Now when you say 'restart services', I assume you're only referring to the ipactl services?  In that case I ran "ipactl start --ignore-service-failures".  Interestingly, when I ran this command it not only failed to start pki-tomcatd (which I expected), but actually reset the date back to the present/correct time and date.  Thus, I re-ran the command to set it back to Sept. 1st, 2019.

I then ran the "getcert resubmit -i <reqID> command.  I just now went through these steps again, and it's showing "status: CA_UNREACHABLE" and "ca-error: Internal Error".  Stuck now shows 'no'.
Re-running "certutil -L -d /etc/pki/pki-tomcat/alias -n 'ServerCert cert-pki-ca' now yields a new error message, "certutil: could not find cert: ServerCert cert-pki-ca", and ": PR_FILE_NOT_FOUND_ERROR: File not found"

Many Mahalos for your continued support and patience!
Scott





From: Rob Crittenden <rcritten@redhat.com>
Sent: Monday, August 10, 2020 11:36 AM
To: FreeIPA users list <freeipa-users@lists.fedorahosted.org>; Florence Blanc-Renaud <flo@redhat.com>
Cc: Scott Z. <sudz28@hotmail.com>
Subject: Re: [Freeipa-users] Re: pki-tomcatd not starting
 
Scott Z. via FreeIPA-users wrote:
> Whoops!  Using the additional command to start tracking this paritcular
> cert that you included in a different message, I got it in the "getcert"
> list (with the "getcert start-tracking -n 'Server-Cert cert-pki-ca' -d
> /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B
> /usr/libexec/ipa/certmonger/stop_pkicad -C
> '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -P
> <pin>" command).
>
> I have the date rolled back to Sept. 1st, 2019.  I guess I have 'some'
> progress now at least, but still have an issue;  checking on the cert
> with "getcert list -i <requestID>", it shows "status: CA_REJECTED", and
> "stuck: yes".

How did you roll the date back? Did you restart services? What date did
you pick and does it overlap so that all certs are valid?

rob

>
> Any additional thoughts or help would be greatly appreciated!  And
> thanks for the help so far.
> Scott
>
> ------------------------------------------------------------------------
> *From:* Scott Z. via FreeIPA-users <freeipa-users@lists.fedorahosted.org>
> *Sent:* Monday, August 10, 2020 10:37 AM
> *To:* Florence Blanc-Renaud <flo@redhat.com>
> *Cc:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>; Scott
> Z. <sudz28@hotmail.com>
> *Subject:* [Freeipa-users] Re: pki-tomcatd not starting
>  
> Sorry, I didn't realize I had dropped the mailing list - my mistake!
>
> I backed up the files/directories you mentioned below, then I checked on
> the ra-agent.pem to see if it was still valid (openssl x509 -in
> /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed
> currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not After:  Aug
> 10 17:20:41 2021 GMT).
>
> Based on that information, and knowing that the bad cert is valid from
> Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019
> since all certs will see that date as valid.
>
> The only issue I have now is getting the request ID for the expired
> cert; it doesn't show up in the list of certs when I do "getcert -list",
> I can only see it by running "certutil -L -d
> /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'", and when
> I run that it does not show any Request ID associated for it?
> Scott
>
>
> ------------------------------------------------------------------------
> *From:* Florence Blanc-Renaud <flo@redhat.com>
> *Sent:* Monday, August 10, 2020 8:45 AM
> *To:* Scott Z. <sudz28@hotmail.com>
> *Cc:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>  
> Hi,
>
> re-adding the mailing list as the conversation could also help others.
>
> On 8/8/20 12:06 AM, Scott Z. wrote:
>> I did notice when I compare it to another IdM server in the environment,
>> if I do a "certutil -L -d /etc.httdp/alias" the non-working server has a
>> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that
>> I'm comparing against has a "Signing-Cert" certificate in addition.  Is
>> this because it's the 'Master' or whatever?  Should my 'bad' server have
>> this same Signing-Cert listed?
>
> /etc/httpd/alias only needs its own Server-Cert + IPA CA.
>
>> Scott
>>
>> ------------------------------------------------------------------------
>> *From:* Scott Z. <sudz28@hotmail.com>
>> *Sent:* Friday, August 7, 2020 10:44 AM
>> *To:* Florence Blanc-Renaud <flo@redhat.com>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> /"The interesting part is the list of expired certs on the failing node
>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>> instructions are available here:
>> https://access.redhat.com/solutions/3357331 How do I manually renew
>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>> (Replica IPA Server)"/
>
> Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
> /etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
> the certificates are stored).
>
> If the RA cert is valid, you need to find a time window during which the
> RA cert is already valid (date > notbefore) and the other certs are not
> expired yet (date < notafter). When you have identified a proper date,
> stop ntpd (or chronyd, depending on which service is used for time
> synchronization), move the date back in time to the identified date,
> start all the services except ntpd, then call "getcert resubmit -i
> <request id>" for the expired cert(s).
>
> Check that the cert has been renewed with "getcert list -i <request
> id>", the state should display MONITORING. When all the certs are good,
> you can restart ntpd and the clock will go back to the current date.
>
> It's really important to find a date where all the certs are valid
> because this ensures that the services are able to start and the RA cert
> allows the authentication that is mandatory for certificate renewal.
>
> HTH,
> flo
>>
>> Sadly, after I log in, it's only telling me that it's "Subscriber
>> Exclusive Content".  Not sure what happened with my account, I used to
>> be able to access these docs with no problem but since I took a RHEL
>> class a couple of weeks back now it's not working any more.  I guess
>> they did something to screw up my account when I took the class. Grrrrr!!!
>> Scott
>>
>> ------------------------------------------------------------------------
>> *From:* Florence Blanc-Renaud <flo@redhat.com>
>> *Sent:* Thursday, August 6, 2020 2:46 AM
>> *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>
>> *Cc:* Scott Z. <sudz28@hotmail.com>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>>> Thanks much for the assistance.  Here is where I am with your suggestions:
>>> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n
>>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old
>>> (almost a year old actually, I assume IPA only checks it when it first
>>> starts up so it didn't care that it was expired until the server was
>>> rebooted?)
>>
>> certmonger checks the certificate validity periodically (configurable in
>> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
>> The system probably had an issue that was not detected and the cert
>> reached its expiration date.
>>
>>>
>>> 2) ran ipactl start --ignore-service-failures
>>>         a. most services started, obviously pki-tomcatd did not
>>> 3) ran "kinit admin"
>>>         a. was forced to change the password, but otherwise nothing happened
>>> 4) Ran "ipa config-show |grep -i master
>>>        a. I see that the IPA CA renewal master is a different idm machine.
>>> 5) Ran "getcert list | grep -E "Request|certificate:|expires:"
>>>        a.I see all certs are currently valid (none expired)
>>> 6) Ran the command "getcert list" on the problem server, but I cannot
>>> paste the output here because it's on an airgaped environment so while I
>>> apologize for this and realize it makes things more difficult, perhaps
>>> if you tell me what I should be looking for or more specifically what
>>> you're interested in I can pluck that out and manually include it here?
>>> So in summary, it is indeed an expired "Server-Cert cert-pki-ca'
>>> certificate on the problem server, and it can theoretically be renew by
>>> the Master at this time.
>> The interesting part is the list of expired certs on the failing node
>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>> instructions are available here:
>> https://access.redhat.com/solutions/3357331 How do I manually renew
>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>> (Replica IPA Server)
>>
>> flo
>>
>>> Many thanks!
>>> Scott
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Florence Blanc-Renaud <flo@redhat.com>
>>> *Sent:* Monday, August 3, 2020 9:34 PM
>>> *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>
>>> *Cc:* Scott Z. <sudz28@hotmail.com>
>>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>>> Not sure I'm sending this to the right place, but here it goes.  I
>>>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
>>>> access) environment that is running into problems.  There are at least 3
>>>> different IdM servers running in the environment spread out across
>>>> different geographical areas.  One of those areas suffered an unschedule
>>>> power outage recently, and ever since we brought everything back up, the
>>>> IdM server for this region is having an issue.  Please bear with me as I
>>>> have zero formal experience, training, or real knowledge with IdM.
>>>>
>>>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
>>>> "ipactl status" and it shows "Directory Service: STOPPED".  I then run
>>>> "ipactl restart", and things go fine until it gets to "Starting
>>>> pki-tomcatd Service", where it hangs for quite some time before failing
>>>> to start and killing all the other services.  I check the log at
>>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>>> (forgive any mistypings, I have to manually type these in as I can't
>>>> import or screen capure the logs and put them in this message):
>>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
>>>> Invalid certificate: (-8181) Peer's Certificate has expired/"
>>>> And slightly further down in the same log:
>>>> "/Cannot reset factory: connections not all returned/"
>>>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
>>>> LDAP connection factory because some connections are still outstanding/"
>>>> ... still further down"
>>>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>>>>
>>>> Assuming I have some weird certificate issue with this server in
>>>> particular, I try to run a few more commands:
>>>> "certutil -L -d /etc/httpd/alias"  --> returns a Server-Cert listing
>>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
>>>> for it's attributes.  Comparing to a second IdM server in this
>>>> environment, it seems to be missing a "Signing-Cert"?
>>>>
>>> Hi,
>>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
>>> has the nickname 'Server-Cert cert-pki-ca'. You should check that this
>>> one is not expired with:
>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
>>> | grep 'Not '
>>>
>>> If the certificate is indeed expired, it will have to be renewed but you
>>> need first to find which IPA server is the CA renewal master. On your
>>> server, force a service start and check the CA renewal master:
>>> # ipactl start --ignore-service-failures
>>> # kinit admin
>>> # ipa config-show | grep "renewal master"
>>>     IPA CA renewal master: server.domain.com
>>>
>>> You need to make sure that all the certificates are valid on the CA
>>> renewal master:
>>> (on the CA renewal master)# getcert list | grep -E
>>> "Request|certificate:|expires:"
>>>
>>> - if the CA renewal master is not OK, please post the output of "#
>>> getcert list" (without the grep) on the CA renewal master. This node
>>> will have to be repaired first.
>>> - if the CA renewal master is OK, please post the output of "# getcert
>>> list" (also without the grep) on the failing node.
>>>
>>> We'll be able to help based on this information.
>>> flo
>>>
>>>> I also did a "getcert list", and all certs it has show that they expire
>>>> in the future (nothing shows as bein currently expired).
>>>>
>>>> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
>>>> but how do I track down which 'peer' the log file is talking about that
>>>> has an expired cert?  Meanwhile none of the linux clients that point to
>>>> this IdM server are allowing people to log in/authenticate.
>>>> Many thanks for any help!
>>>> Scott
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
>>>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>>>
>>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>