Adding the "NSSEnforceValidCerts off" definitely got me past the HTTPD error.  It started up and then I ran the systemctl start pki-tomcatd@pki-tomcat which seemed to start up without any errors (it didn't throw any on the command line), but checking the debug log I see I'm still getting the same, original "Peer's Certificate has expired" message for "Server-Cert cert-pki-ca".  I just can't win 🙂 
It's expired, I know it's expired, why does FreeIPA fight me so hard on just trying to renew it?!  LOL!

Just for fun I then ran the "getcert renew -i <reqid>" command.  But per "getcert list", it's still showing as CA_UNREACHABLE and Internal Error.
Scott



From: Rob Crittenden <rcritten@redhat.com>
Sent: Tuesday, August 11, 2020 8:07 AM
To: FreeIPA users list <freeipa-users@lists.fedorahosted.org>; Florence Blanc-Renaud <flo@redhat.com>
Cc: Scott Z. <sudz28@hotmail.com>
Subject: Re: [Freeipa-users] Re: pki-tomcatd not starting
 
Scott Z. via FreeIPA-users wrote:
> Forgot to reply again - ugh!
> Hmmmm, so my domain is actually "idm.project.its.srv2", so I was
> literally typing "systemctl start dirsrv@idm.project.its.srv2"  I see
> what you're saying, I need to put in dashes instead of periods!  DOH! 
> Done.  Moving on...
> 4) Ran systemctl start krb5kdc
> 5) Ran systemctl start kadmin
> 6) Ran systemctl start named-pkcs11
> 7) Ran systemctl start httpd  -  got an error here, nothing really
> useful in the logs or journalctl, it says it's starting the Apache HTTP
> server, then throws "httpd.service: main process exited, code=exited,
> status=1/FAILURE", and "Failed to start The Apache HTTP Server". 
> Finally there is a mention of 'too much time skew'.  I assume the
> problem is that I'm trying to start HTTPD on a system where the date is
> almost a year old. 
> Although now that I'm looking at /var/log/httpd/error_log, I see mention
> of "SSL Library Error: -8181 Certificate has expired".  CERTIFICATES!!!
> "Unable to verify certificate 'Server-Cert'.  Add "NSSEnfroceValideCerts
> off" to nss.conf so the server can start until the problem can be
> resolved", so maybe I'll try that.

That can work, just remember to revert it, but it just bypasses the
start up check. Clients will still require cert validity.

I don't think it will matter either way as the CA certs renew directly
against the CA so Apache not running shouldn't be an issue.

rob

> Scott
>
> ------------------------------------------------------------------------
> *From:* Florence Blanc-Renaud <flo@redhat.com>
> *Sent:* Tuesday, August 11, 2020 6:55 AM
> *To:* Scott Z. <sudz28@hotmail.com>; FreeIPA users list
> <freeipa-users@lists.fedorahosted.org>; Rob Crittenden <rcritten@redhat.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>  
> On 8/11/20 6:39 PM, Scott Z. wrote:
>> First thing I did when I logged in this morning (I'm on Hawaii Standard
>> Time) was run "ipactl status".  The return was "Directory Services:
>> STOPPED", and "Directory Service must running in order to obtain status
>> of other services".
>> 1) Ran "getcert list", and it shows the 9 certs being tracked (all the
>> previous 8 plus the 1 expired guy I added yesterday).  All look good
>> except of course my problem child, who's status is CA_UNREACHABLE and
>> ca-error is Internal error.
>> 2) Ran "ipa stop", looks like all service stopped successfully.
>> 2) Changed date back to Sept. 1, 2019.
>> 3) Ran the "systemctl start dirsrv@<domain> and got back "Job for
>> dirsrv@<domain> failed because a configured resource limit was exceeded."
>>       a. when I looked at "journalctl -xe", I just see a couple of
>> messages that don't tell me much... "Registered Authentication Agent for
>> unix-process:<blahblah>", followed by "Failed to load environment files:
>> no such files or directory".  Then, "dirsrv@<domain> filed to run
>> 'start-pre' task: No such files or directory" and finally "Failed to
>> start 389 Directory Server <domain>".
>>
> If your domain is domain.com, you need to run
> systemctl start dirsrv@DOMAIN-COM
>
> I suspect that you ran instead systemctl start dirsrv@slapd-DOMAIN-COM
> which would produce the error you're seeing.
>
> flo
>
>> Not sure now how to proceed at this point.
>>
>> BTW, I have decided that once I get through this slog and have a working
>> server again, I'm going to donate $50 to the Hawaiian Food Bank or the
>> charity of your choice in appreciation.
>> Scott
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Florence Blanc-Renaud <flo@redhat.com>
>> *Sent:* Monday, August 10, 2020 8:55 PM
>> *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>; Rob
>> Crittenden <rcritten@redhat.com>
>> *Cc:* Scott Z. <sudz28@hotmail.com>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> On 8/10/20 11:46 PM, Scott Z. via FreeIPA-users wrote:
>>> I stopped the ntp service with the command "timedatectl set_ntp 0"
>>> I set the new date to be Sept. 1st, 2019 with "timedatectl set-time
>>> 2019-09-01"
>>> I waiting a minute and then checked with the "date" command; the problem
>>> server believes it is Sept. 1st, 2019.
>>>
>>> Now when you say 'restart services', I assume you're only referring to
>>> the ipactl services?  In that case I ran "ipactl start
>>> --ignore-service-failures".  Interestingly, when I ran this command it
>>> not only failed to start pki-tomcatd (which I expected), but actually
>>> reset the date back to the present/correct time and date.  Thus, I
>>> re-ran the command to set it back to Sept. 1st, 2019.
>>>
>> If the server was configured with ntp, "ipactl start" will also restart
>> ntpd. You need to do the following:
>> ipactl stop
>> change date in the past
>> systemctl start dirsrv@DOMAIN-COM (replace with your domain name)
>> systemctl start krb5kdc
>> systemctl start kadmin
>> systemctl start named-pkcs11 (if IPA is hosting the DNS server)
>> systemctl start httpd
>> systemctl start pki-tomcatd@pki-tomcat
>>
>> Then try getcert resubmit.
>>
>>> I then ran the "getcert resubmit -i <reqID> command.  I just now went
>>> through these steps again, and it's showing "status: CA_UNREACHABLE" and
>>> "ca-error: Internal Error".  Stuck now shows 'no'.
>>> Re-running "certutil -L -d /etc/pki/pki-tomcat/alias -n 'ServerCert
>>> cert-pki-ca' now yields a new error message, "certutil: could not find
>>> cert: ServerCert cert-pki-ca", and ": PR_FILE_NOT_FOUND_ERROR: File not
>>> found"
>> The cert nickname should contain a dash: "Server-Cert cert-pki-ca"
>>
>> HTH,
>> flo
>>>
>>> Many Mahalos for your continued support and patience!
>>> Scott
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Rob Crittenden <rcritten@redhat.com>
>>> *Sent:* Monday, August 10, 2020 11:36 AM
>>> *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>;
>>> Florence Blanc-Renaud <flo@redhat.com>
>>> *Cc:* Scott Z. <sudz28@hotmail.com>
>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>> Scott Z. via FreeIPA-users wrote:
>>>> Whoops!  Using the additional command to start tracking this paritcular
>>>> cert that you included in a different message, I got it in the "getcert"
>>>> list (with the "getcert start-tracking -n 'Server-Cert cert-pki-ca' -d
>>>> /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B
>>>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>>>> '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -P
>>>> <pin>" command).
>>>>
>>>> I have the date rolled back to Sept. 1st, 2019.  I guess I have 'some'
>>>> progress now at least, but still have an issue;  checking on the cert
>>>> with "getcert list -i <requestID>", it shows "status: CA_REJECTED", and
>>>> "stuck: yes".
>>>
>>> How did you roll the date back? Did you restart services? What date did
>>> you pick and does it overlap so that all certs are valid?
>>>
>>> rob
>>>
>>>>
>>>> Any additional thoughts or help would be greatly appreciated!  And
>>>> thanks for the help so far.
>>>> Scott
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Scott Z. via FreeIPA-users <freeipa-users@lists.fedorahosted.org>
>>>> *Sent:* Monday, August 10, 2020 10:37 AM
>>>> *To:* Florence Blanc-Renaud <flo@redhat.com>
>>>> *Cc:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>; Scott
>>>> Z. <sudz28@hotmail.com>
>>>> *Subject:* [Freeipa-users] Re: pki-tomcatd not starting
>>>> Â
>>>> Sorry, I didn't realize I had dropped the mailing list - my mistake!
>>>>
>>>> I backed up the files/directories you mentioned below, then I checked on
>>>> the ra-agent.pem to see if it was still valid (openssl x509 -in
>>>> /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed
>>>> currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not After:  Aug
>>>> 10 17:20:41 2021 GMT).
>>>>
>>>> Based on that information, and knowing that the bad cert is valid from
>>>> Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019
>>>> since all certs will see that date as valid.
>>>>
>>>> The only issue I have now is getting the request ID for the expired
>>>> cert; it doesn't show up in the list of certs when I do "getcert -list",
>>>> I can only see it by running "certutil -L -d
>>>> /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'", and when
>>>> I run that it does not show any Request ID associated for it?
>>>> Scott
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Florence Blanc-Renaud <flo@redhat.com>
>>>> *Sent:* Monday, August 10, 2020 8:45 AM
>>>> *To:* Scott Z. <sudz28@hotmail.com>
>>>> *Cc:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>
>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>> Â
>>>> Hi,
>>>>
>>>> re-adding the mailing list as the conversation could also help others.
>>>>
>>>> On 8/8/20 12:06 AM, Scott Z. wrote:
>>>>> I did notice when I compare it to another IdM server in the environment,
>>>>> if I do a "certutil -L -d /etc.httdp/alias" the non-working server has a
>>>>> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that
>>>>> I'm comparing against has a "Signing-Cert" certificate in addition.  Is
>>>>> this because it's the 'Master' or whatever?  Should my 'bad' server have
>>>>> this same Signing-Cert listed?
>>>>
>>>> /etc/httpd/alias only needs its own Server-Cert + IPA CA.
>>>>
>>>>> Scott
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> *From:* Scott Z. <sudz28@hotmail.com>
>>>>> *Sent:* Friday, August 7, 2020 10:44 AM
>>>>> *To:* Florence Blanc-Renaud <flo@redhat.com>
>>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>>> /"The interesting part is the list of expired certs on the failing node
>>>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>>>> instructions are available here:
>>>>> https://access.redhat.com/solutions/3357331 How do I manually renew
>>>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>>>> (Replica IPA Server)"/
>>>>
>>>> Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
>>>> /etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
>>>> the certificates are stored).
>>>>
>>>> If the RA cert is valid, you need to find a time window during which the
>>>> RA cert is already valid (date > notbefore) and the other certs are not
>>>> expired yet (date < notafter). When you have identified a proper date,
>>>> stop ntpd (or chronyd, depending on which service is used for time
>>>> synchronization), move the date back in time to the identified date,
>>>> start all the services except ntpd, then call "getcert resubmit -i
>>>> <request id>" for the expired cert(s).
>>>>
>>>> Check that the cert has been renewed with "getcert list -i <request
>>>> id>", the state should display MONITORING. When all the certs are good,
>>>> you can restart ntpd and the clock will go back to the current date.
>>>>
>>>> It's really important to find a date where all the certs are valid
>>>> because this ensures that the services are able to start and the RA cert
>>>> allows the authentication that is mandatory for certificate renewal.
>>>>
>>>> HTH,
>>>> flo
>>>>>
>>>>> Sadly, after I log in, it's only telling me that it's "Subscriber
>>>>> Exclusive Content".  Not sure what happened with my account, I used to
>>>>> be able to access these docs with no problem but since I took a RHEL
>>>>> class a couple of weeks back now it's not working any more.  I guess
>>>>> they did something to screw up my account when I took the class. Grrrrr!!!
>>>>> Scott
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> *From:* Florence Blanc-Renaud <flo@redhat.com>
>>>>> *Sent:* Thursday, August 6, 2020 2:46 AM
>>>>> *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>
>>>>> *Cc:* Scott Z. <sudz28@hotmail.com>
>>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>>> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>>>>>> Thanks much for the assistance.  Here is where I am with your suggestions:
>>>>>> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n
>>>>>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old
>>>>>> (almost a year old actually, I assume IPA only checks it when it first
>>>>>> starts up so it didn't care that it was expired until the server was
>>>>>> rebooted?)
>>>>>
>>>>> certmonger checks the certificate validity periodically (configurable in
>>>>> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
>>>>> The system probably had an issue that was not detected and the cert
>>>>> reached its expiration date.
>>>>>
>>>>>>
>>>>>> 2) ran ipactl start --ignore-service-failures
>>>>>>         a. most services started, obviously pki-tomcatd did not
>>>>>> 3) ran "kinit admin"
>>>>>>         a. was forced to change the password, but otherwise nothing happened
>>>>>> 4) Ran "ipa config-show |grep -i master
>>>>>>        a. I see that the IPA CA renewal master is a different idm machine.
>>>>>> 5) Ran "getcert list | grep -E "Request|certificate:|expires:"
>>>>>>        a.I see all certs are currently valid (none expired)
>>>>>> 6) Ran the command "getcert list" on the problem server, but I cannot
>>>>>> paste the output here because it's on an airgaped environment so while I
>>>>>> apologize for this and realize it makes things more difficult, perhaps
>>>>>> if you tell me what I should be looking for or more specifically what
>>>>>> you're interested in I can pluck that out and manually include it here?
>>>>>> So in summary, it is indeed an expired "Server-Cert cert-pki-ca'
>>>>>> certificate on the problem server, and it can theoretically be renew by
>>>>>> the Master at this time.
>>>>> The interesting part is the list of expired certs on the failing node
>>>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>>>> instructions are available here:
>>>>> https://access.redhat.com/solutions/3357331 How do I manually renew
>>>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>>>> (Replica IPA Server)
>>>>>
>>>>> flo
>>>>>
>>>>>> Many thanks!
>>>>>> Scott
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>> *From:* Florence Blanc-Renaud <flo@redhat.com>
>>>>>> *Sent:* Monday, August 3, 2020 9:34 PM
>>>>>> *To:* FreeIPA users list <freeipa-users@lists.fedorahosted.org>
>>>>>> *Cc:* Scott Z. <sudz28@hotmail.com>
>>>>>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>>>>>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>>>>>> Not sure I'm sending this to the right place, but here it goes.  I
>>>>>>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
>>>>>>> access) environment that is running into problems.  There are at least 3
>>>>>>> different IdM servers running in the environment spread out across
>>>>>>> different geographical areas.  One of those areas suffered an unschedule
>>>>>>> power outage recently, and ever since we brought everything back up, the
>>>>>>> IdM server for this region is having an issue.  Please bear with me as I
>>>>>>> have zero formal experience, training, or real knowledge with IdM.
>>>>>>>
>>>>>>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
>>>>>>> "ipactl status" and it shows "Directory Service: STOPPED".  I then run
>>>>>>> "ipactl restart", and things go fine until it gets to "Starting
>>>>>>> pki-tomcatd Service", where it hangs for quite some time before failing
>>>>>>> to start and killing all the other services.  I check the log at
>>>>>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>>>>>> (forgive any mistypings, I have to manually type these in as I can't
>>>>>>> import or screen capure the logs and put them in this message):
>>>>>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
>>>>>>> Invalid certificate: (-8181) Peer's Certificate has expired/"
>>>>>>> And slightly further down in the same log:
>>>>>>> "/Cannot reset factory: connections not all returned/"
>>>>>>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
>>>>>>> LDAP connection factory because some connections are still outstanding/"
>>>>>>> ... still further down"
>>>>>>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>>>>>>>
>>>>>>> Assuming I have some weird certificate issue with this server in
>>>>>>> particular, I try to run a few more commands:
>>>>>>> "certutil -L -d /etc/httpd/alias"  --> returns a Server-Cert listing
>>>>>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
>>>>>>> for it's attributes.  Comparing to a second IdM server in this
>>>>>>> environment, it seems to be missing a "Signing-Cert"?
>>>>>>>
>>>>>> Hi,
>>>>>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
>>>>>> has the nickname 'Server-Cert cert-pki-ca'. You should check that this
>>>>>> one is not expired with:
>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
>>>>>> | grep 'Not '
>>>>>>
>>>>>> If the certificate is indeed expired, it will have to be renewed but you
>>>>>> need first to find which IPA server is the CA renewal master. On your
>>>>>> server, force a service start and check the CA renewal master:
>>>>>> # ipactl start --ignore-service-failures
>>>>>> # kinit admin
>>>>>> # ipa config-show | grep "renewal master"
>>>>>>     IPA CA renewal master: server.domain.com
>>>>>>
>>>>>> You need to make sure that all the certificates are valid on the CA
>>>>>> renewal master:
>>>>>> (on the CA renewal master)# getcert list | grep -E
>>>>>> "Request|certificate:|expires:"
>>>>>>
>>>>>> - if the CA renewal master is not OK, please post the output of "#
>>>>>> getcert list" (without the grep) on the CA renewal master. This node
>>>>>> will have to be repaired first.
>>>>>> - if the CA renewal master is OK, please post the output of "# getcert
>>>>>> list" (also without the grep) on the failing node.
>>>>>>
>>>>>> We'll be able to help based on this information.
>>>>>> flo
>>>>>>
>>>>>>> I also did a "getcert list", and all certs it has show that they expire
>>>>>>> in the future (nothing shows as bein currently expired).
>>>>>>>
>>>>>>> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
>>>>>>> but how do I track down which 'peer' the log file is talking about that
>>>>>>> has an expired cert?  Meanwhile none of the linux clients that point to
>>>>>>> this IdM server are allowing people to log in/authenticate.
>>>>>>> Many thanks for any help!
>>>>>>> Scott
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
>>>>>>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
>>>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
>>>>>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
>>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
>>>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>>>
>>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>