Scott Z. via FreeIPA-users wrote:
Forgot to reply again - ugh!
Hmmmm, so my domain is actually "idm.project.its.srv2", so I was
literally typing "systemctl start dirsrv(a)idm.project.its.xn--srv2" -57a I see
what you're saying, I need to put in dashes instead of periods! DOH!Â
Done. Moving on...
4) Ran systemctl start krb5kdc
5) Ran systemctl start kadmin
6) Ran systemctl start named-pkcs11
7) Ran systemctl start httpd - got an error here, nothing really
useful in the logs or journalctl, it says it's starting the Apache HTTP
server, then throws "httpd.service: main process exited, code=exited,
status=1/FAILURE", and "Failed to start The Apache HTTP Server".Â
Finally there is a mention of 'too much time skew'. I assume the
problem is that I'm trying to start HTTPD on a system where the date is
almost a year old.Â
Although now that I'm looking at /var/log/httpd/error_log, I see mention
of "SSL Library Error: -8181 Certificate has expired". CERTIFICATES!!!
"Unable to verify certificate 'Server-Cert'. Add
"NSSEnfroceValideCerts
off" to nss.conf so the server can start until the problem can be
resolved", so maybe I'll try that.
That can work, just remember to revert it, but it just bypasses the
start up check. Clients will still require cert validity.
I don't think it will matter either way as the CA certs renew directly
against the CA so Apache not running shouldn't be an issue.
rob
Scott
------------------------------------------------------------------------
*From:* Florence Blanc-Renaud <flo(a)redhat.com>
*Sent:* Tuesday, August 11, 2020 6:55 AM
*To:* Scott Z. <sudz28(a)hotmail.com>; FreeIPA users list
<freeipa-users(a)lists.fedorahosted.org>; Rob Crittenden <rcritten(a)redhat.com>
*Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
Â
On 8/11/20 6:39 PM, Scott Z. wrote:
> First thing I did when I logged in this morning (I'm on Hawaii Standard
> Time) was run "ipactl status". The return was "Directory Services:
> STOPPED", and "Directory Service must running in order to obtain status
> of other services".
> 1) Ran "getcert list", and it shows the 9 certs being tracked (all the
> previous 8 plus the 1 expired guy I added yesterday). All look good
> except of course my problem child, who's status is CA_UNREACHABLE and
> ca-error is Internal error.
> 2) Ran "ipa stop", looks like all service stopped successfully.
> 2) Changed date back to Sept. 1, 2019.
> 3) Ran the "systemctl start dirsrv@<domain> and got back "Job for
> dirsrv@<domain> failed because a configured resource limit was exceeded."
> Â Â Â Â a. when I looked at "journalctl -xe", I just see a
couple of
> messages that don't tell me much... "Registered Authentication Agent for
> unix-process:<blahblah>", followed by "Failed to load environment
files:
> no such files or directory". Then, "dirsrv@<domain> filed to run
> 'start-pre' task: No such files or directory" and finally "Failed
to
> start 389 Directory Server <domain>".
>
If your domain is
domain.com, you need to run
systemctl start dirsrv@DOMAIN-COM
I suspect that you ran instead systemctl start dirsrv@slapd-DOMAIN-COM
which would produce the error you're seeing.
flo
> Not sure now how to proceed at this point.
>
> BTW, I have decided that once I get through this slog and have a working
> server again, I'm going to donate $50 to the Hawaiian Food Bank or the
> charity of your choice in appreciation.
> Scott
>
>
> ------------------------------------------------------------------------
> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
> *Sent:* Monday, August 10, 2020 8:55 PM
> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Rob
> Crittenden <rcritten(a)redhat.com>
> *Cc:* Scott Z. <sudz28(a)hotmail.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
> On 8/10/20 11:46 PM, Scott Z. via FreeIPA-users wrote:
>> I stopped the ntp service with the command "timedatectl set_ntp 0"
>> I set the new date to be Sept. 1st, 2019 with "timedatectl set-time
>> 2019-09-01"
>> I waiting a minute and then checked with the "date" command; the
problem
>> server believes it is Sept. 1st, 2019.
>>
>> Now when you say 'restart services', I assume you're only referring
to
>> the ipactl services? In that case I ran "ipactl start
>> --ignore-service-failures". Interestingly, when I ran this command
it
>> not only failed to start pki-tomcatd (which I expected), but actually
>> reset the date back to the present/correct time and date. Thus, I
>> re-ran the command to set it back to Sept. 1st, 2019.
>>
> If the server was configured with ntp, "ipactl start" will also restart
> ntpd. You need to do the following:
> ipactl stop
> change date in the past
> systemctl start dirsrv@DOMAIN-COM (replace with your domain name)
> systemctl start krb5kdc
> systemctl start kadmin
> systemctl start named-pkcs11 (if IPA is hosting the DNS server)
> systemctl start httpd
> systemctl start pki-tomcatd@pki-tomcat
>
> Then try getcert resubmit.
>
>> I then ran the "getcert resubmit -i <reqID> command. I just
now went
>> through these steps again, and it's showing "status:
CA_UNREACHABLE" and
>> "ca-error: Internal Error". Stuck now shows 'no'.
>> Re-running "certutil -L -d /etc/pki/pki-tomcat/alias -n 'ServerCert
>> cert-pki-ca' now yields a new error message, "certutil: could not find
>> cert: ServerCert cert-pki-ca", and ": PR_FILE_NOT_FOUND_ERROR: File not
>> found"
> The cert nickname should contain a dash: "Server-Cert cert-pki-ca"
>
> HTH,
> flo
>>
>> Many Mahalos for your continued support and patience!
>> Scott
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Rob Crittenden <rcritten(a)redhat.com>
>> *Sent:* Monday, August 10, 2020 11:36 AM
>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>;
>> Florence Blanc-Renaud <flo(a)redhat.com>
>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> Scott Z. via FreeIPA-users wrote:
>>> Whoops! Using the additional command to start tracking
this paritcular
>>> cert that you included in a different message, I got it in the
"getcert"
>>> list (with the "getcert start-tracking -n 'Server-Cert
cert-pki-ca' -d
>>> /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B
>>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>>> '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert
cert-pki-ca"' -P
>>> <pin>" command).
>>>
>>> I have the date rolled back to Sept. 1st, 2019. I guess I
have 'some'
>>> progress now at least, but still have an issue; checking
on the cert
>>> with "getcert list -i <requestID>", it shows "status:
CA_REJECTED", and
>>> "stuck: yes".
>>
>> How did you roll the date back? Did you restart services? What date did
>> you pick and does it overlap so that all certs are valid?
>>
>> rob
>>
>>>
>>> Any additional thoughts or help would be greatly
appreciated! And
>>> thanks for the help so far.
>>> Scott
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Scott Z. via FreeIPA-users
<freeipa-users(a)lists.fedorahosted.org>
>>> *Sent:* Monday, August 10, 2020 10:37 AM
>>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Scott
>>> Z. <sudz28(a)hotmail.com>
>>> *Subject:* [Freeipa-users] Re: pki-tomcatd not starting
>>> Â
>>> Sorry, I didn't realize I had dropped the mailing list - my mistake!
>>>
>>> I backed up the files/directories you mentioned below, then I checked on
>>> the ra-agent.pem to see if it was still valid (openssl x509 -in
>>> /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed
>>> currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not
After: Aug
>>> 10 17:20:41 2021 GMT).
>>>
>>> Based on that information, and knowing that the bad cert is valid from
>>> Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019
>>> since all certs will see that date as valid.
>>>
>>> The only issue I have now is getting the request ID for the expired
>>> cert; it doesn't show up in the list of certs when I do "getcert
-list",
>>> I can only see it by running "certutil -L -d
>>> /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'",
and when
>>> I run that it does not show any Request ID associated for it?
>>> Scott
>>>
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Sent:* Monday, August 10, 2020 8:45 AM
>>> *To:* Scott Z. <sudz28(a)hotmail.com>
>>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>> Â
>>> Hi,
>>>
>>> re-adding the mailing list as the conversation could also help others.
>>>
>>> On 8/8/20 12:06 AM, Scott Z. wrote:
>>>> I did notice when I compare it to another IdM server in the environment,
>>>> if I do a "certutil -L -d /etc.httdp/alias" the non-working
server has a
>>>> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one
that
>>>> I'm comparing against has a "Signing-Cert" certificate in
addition. Is
>>>> this because it's the 'Master' or
whatever? Should my 'bad' server have
>>>> this same Signing-Cert listed?
>>>
>>> /etc/httpd/alias only needs its own Server-Cert + IPA CA.
>>>
>>>> Scott
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Scott Z. <sudz28(a)hotmail.com>
>>>> *Sent:* Friday, August 7, 2020 10:44 AM
>>>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>> /"The interesting part is the list of expired certs on the failing
node
>>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>>> instructions are available here:
>>>>
https://access.redhat.com/solutions/3357331 How do I manually renew
>>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>>> (Replica IPA Server)"/
>>>
>>> Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
>>> /etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
>>> the certificates are stored).
>>>
>>> If the RA cert is valid, you need to find a time window during which the
>>> RA cert is already valid (date > notbefore) and the other certs are not
>>> expired yet (date < notafter). When you have identified a proper date,
>>> stop ntpd (or chronyd, depending on which service is used for time
>>> synchronization), move the date back in time to the identified date,
>>> start all the services except ntpd, then call "getcert resubmit -i
>>> <request id>" for the expired cert(s).
>>>
>>> Check that the cert has been renewed with "getcert list -i <request
>>> id>", the state should display MONITORING. When all the certs are
good,
>>> you can restart ntpd and the clock will go back to the current date.
>>>
>>> It's really important to find a date where all the certs are valid
>>> because this ensures that the services are able to start and the RA cert
>>> allows the authentication that is mandatory for certificate renewal.
>>>
>>> HTH,
>>> flo
>>>>
>>>> Sadly, after I log in, it's only telling me that it's
"Subscriber
>>>> Exclusive Content". Not
sure what happened with my account, I used to
>>>> be able to access these docs with no problem but since I took a RHEL
>>>> class a couple of weeks back now it's not working any
more. I guess
>>>> they did something to screw up my account when I took the class.
Grrrrr!!!
>>>> Scott
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>> *Sent:* Thursday, August 6, 2020 2:46 AM
>>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>>>>> Thanks much for the
assistance.ÂÂÂÂÂÂ
Here is where I am with your suggestions:
>>>>> 1) Checked on the cert with "certutil -L -d
/etc/pki/pki-tomcat/alias -n
>>>>> 'Server-Cert cert-pki-ca' and I see that the Validity is
indeed old
>>>>> (almost a year old actually, I assume IPA only checks it when it
first
>>>>> starts up so it didn't care that it was expired until the server
was
>>>>> rebooted?)
>>>>
>>>> certmonger checks the certificate validity periodically (configurable in
>>>> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
>>>> The system probably had an issue that was not detected and the cert
>>>> reached its expiration date.
>>>>
>>>>>
>>>>> 2) ran ipactl start --ignore-service-failures
>>>>>ÂÂÂÂÂ
     ÂÂÂÂÂÂ
a. most services started, obviously pki-tomcatd did not
>>>>> 3) ran "kinit admin"
>>>>>ÂÂÂÂÂ
     ÂÂÂÂÂÂ
a. was forced to change the password, but otherwise nothing happened
>>>>> 4) Ran "ipa config-show |grep -i master
>>>>>ÂÂÂÂÂ
    ÂÂÂÂÂÂ
a. I see that the IPA CA renewal master is a different idm machine.
>>>>> 5) Ran "getcert list | grep -E
"Request|certificate:|expires:"
>>>>>ÂÂÂÂÂ
    ÂÂÂÂÂÂ
a.I see all certs are currently valid (none expired)
>>>>> 6) Ran the command "getcert list" on the problem server,
but I cannot
>>>>> paste the output here because it's on an airgaped environment so
while I
>>>>> apologize for this and realize it makes things more difficult,
perhaps
>>>>> if you tell me what I should be looking for or more specifically what
>>>>> you're interested in I can pluck that out and manually include it
here?
>>>>> So in summary, it is indeed an expired "Server-Cert
cert-pki-ca'
>>>>> certificate on the problem server, and it can theoretically be renew
by
>>>>> the Master at this time.
>>>> The interesting part is the list of expired certs on the failing node
>>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>>> instructions are available here:
>>>>
https://access.redhat.com/solutions/3357331 How do I manually renew
>>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>>> (Replica IPA Server)
>>>>
>>>> flo
>>>>
>>>>> Many thanks!
>>>>> Scott
>>>>>
>>>>>
------------------------------------------------------------------------
>>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>>> *Sent:* Monday, August 3, 2020 9:34 PM
>>>>> *To:* FreeIPA users list
<freeipa-users(a)lists.fedorahosted.org>
>>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>>>>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>>>>> Not sure I'm sending this to the right place, but here it
goes.ÂÂÂÂÂÂÂ
I
>>>>>> inherited a FreeIPA/Identity Manager setup in an enclave (no
internet
>>>>>> access) environment that is running into
problems.ÂÂÂÂÂÂÂ
There are at least 3
>>>>>> different IdM servers running in the environment spread out
across
>>>>>> different geographical
areas.ÂÂÂÂÂÂÂ
One of those areas suffered an unschedule
>>>>>> power outage recently, and ever since we brought everything back
up, the
>>>>>> IdM server for this region is having an
issue.ÂÂÂÂÂÂÂ
Please bear with me as I
>>>>>> have zero formal experience, training, or real knowledge with
IdM.
>>>>>>
>>>>>> Logging in to the serverv (it's a VM server, running Centos
7.5), I run
>>>>>> "ipactl status" and it shows "Directory Service:
STOPPED".ÂÂÂÂÂÂÂ
I then run
>>>>>> "ipactl restart", and things go fine until it gets to
"Starting
>>>>>> pki-tomcatd Service", where it hangs for quite some time
before failing
>>>>>> to start and killing all the other
services.ÂÂÂÂÂÂÂ
I check the log at
>>>>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>>>>> (forgive any mistypings, I have to manually type these in as I
can't
>>>>>> import or screen capure the logs and put them in this message):
>>>>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca
is invalid:
>>>>>> Invalid certificate: (-8181) Peer's Certificate has
expired/"
>>>>>> And slightly further down in the same log:
>>>>>> "/Cannot reset factory: connections not all returned/"
>>>>>> "/CertificateAuthority.shutdown: failed to reset dbFactory:
Cannot reset
>>>>>> LDAP connection factory because some connections are still
outstanding/"
>>>>>> ... still further down"
>>>>>> "/returnConn:mNumConns now 3 Invalid class name
repositorytop/"
>>>>>>
>>>>>> Assuming I have some weird certificate issue with this server in
>>>>>> particular, I try to run a few more commands:
>>>>>> "certutil -L -d
/etc/httpd/alias"ÂÂÂÂÂÂÂ
--> returns a Server-Cert listing
>>>>>> with u,u,u as it's trust attributes, and <IDM.domain>
IPA CA with CT,C,C
>>>>>> for it's
attributes.ÂÂÂÂÂÂÂ
Comparing to a second IdM server in this
>>>>>> environment, it seems to be missing a "Signing-Cert"?
>>>>>>
>>>>> Hi,
>>>>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server
cert
>>>>> has the nickname 'Server-Cert cert-pki-ca'. You should check
that this
>>>>> one is not expired with:
>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert
cert-pki-ca'
>>>>> | grep 'Not '
>>>>>
>>>>> If the certificate is indeed expired, it will have to be renewed but
you
>>>>> need first to find which IPA server is the CA renewal master. On
your
>>>>> server, force a service start and check the CA renewal master:
>>>>> # ipactl start --ignore-service-failures
>>>>> # kinit admin
>>>>> # ipa config-show | grep "renewal master"
>>>>>ÂÂÂÂÂ
 ÂÂÂÂÂÂ
IPA CA renewal master:
server.domain.com
>>>>>
>>>>> You need to make sure that all the certificates are valid on the CA
>>>>> renewal master:
>>>>> (on the CA renewal master)# getcert list | grep -E
>>>>> "Request|certificate:|expires:"
>>>>>
>>>>> - if the CA renewal master is not OK, please post the output of
"#
>>>>> getcert list" (without the grep) on the CA renewal master. This
node
>>>>> will have to be repaired first.
>>>>> - if the CA renewal master is OK, please post the output of "#
getcert
>>>>> list" (also without the grep) on the failing node.
>>>>>
>>>>> We'll be able to help based on this information.
>>>>> flo
>>>>>
>>>>>> I also did a "getcert list", and all certs it has show
that they expire
>>>>>> in the future (nothing shows as bein currently expired).
>>>>>>
>>>>>> I'm confused; it seems to that it is seeing an expired cert
*somewhere*,
>>>>>> but how do I track down which 'peer' the log file is
talking about that
>>>>>> has an expired
cert?ÂÂÂÂÂÂÂ
Meanwhile none of the linux clients that point to
>>>>>> this IdM server are allowing people to log in/authenticate.
>>>>>> Many thanks for any help!
>>>>>> Scott
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>>>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
>>>>>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
>>>>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...