Forgot to reply again - ugh!
Hmmmm, so my domain is actually "idm.project.its.srv2", so I was literally
typing "systemctl start dirsrv(a)idm.project.its.srv2" I see what you're
saying, I need to put in dashes instead of periods! DOH! Done. Moving on...
4) Ran systemctl start krb5kdc
5) Ran systemctl start kadmin
6) Ran systemctl start named-pkcs11
7) Ran systemctl start httpd - got an error here, nothing really useful in the logs or
journalctl, it says it's starting the Apache HTTP server, then throws
"httpd.service: main process exited, code=exited, status=1/FAILURE", and
"Failed to start The Apache HTTP Server". Finally there is a mention of
'too much time skew'. I assume the problem is that I'm trying to start HTTPD
on a system where the date is almost a year old.
Although now that I'm looking at /var/log/httpd/error_log, I see mention of "SSL
Library Error: -8181 Certificate has expired". CERTIFICATES!!!
"Unable to verify certificate 'Server-Cert'. Add "NSSEnfroceValideCerts
off" to nss.conf so the server can start until the problem can be resolved", so
maybe I'll try that.
Scott
________________________________
From: Florence Blanc-Renaud <flo(a)redhat.com>
Sent: Tuesday, August 11, 2020 6:55 AM
To: Scott Z. <sudz28(a)hotmail.com>; FreeIPA users list
<freeipa-users(a)lists.fedorahosted.org>; Rob Crittenden <rcritten(a)redhat.com>
Subject: Re: [Freeipa-users] Re: pki-tomcatd not starting
On 8/11/20 6:39 PM, Scott Z. wrote:
First thing I did when I logged in this morning (I'm on Hawaii
Standard
Time) was run "ipactl status". The return was "Directory Services:
STOPPED", and "Directory Service must running in order to obtain status
of other services".
1) Ran "getcert list", and it shows the 9 certs being tracked (all the
previous 8 plus the 1 expired guy I added yesterday). All look good
except of course my problem child, who's status is CA_UNREACHABLE and
ca-error is Internal error.
2) Ran "ipa stop", looks like all service stopped successfully.
2) Changed date back to Sept. 1, 2019.
3) Ran the "systemctl start dirsrv@<domain> and got back "Job for
dirsrv@<domain> failed because a configured resource limit was exceeded."
    a. when I looked at "journalctl -xe", I just see a couple of
messages that don't tell me much... "Registered Authentication Agent for
unix-process:<blahblah>", followed by "Failed to load environment files:
no such files or directory". Then, "dirsrv@<domain> filed to run
'start-pre' task: No such files or directory" and finally "Failed to
start 389 Directory Server <domain>".
, you need to run
systemctl start dirsrv@DOMAIN-COM
I suspect that you ran instead systemctl start dirsrv@slapd-DOMAIN-COM
which would produce the error you're seeing.
flo
Not sure now how to proceed at this point.
BTW, I have decided that once I get through this slog and have a working
server again, I'm going to donate $50 to the Hawaiian Food Bank or the
charity of your choice in appreciation.
Scott
------------------------------------------------------------------------
*From:* Florence Blanc-Renaud <flo(a)redhat.com>
*Sent:* Monday, August 10, 2020 8:55 PM
*To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Rob
Crittenden <rcritten(a)redhat.com>
*Cc:* Scott Z. <sudz28(a)hotmail.com>
*Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
On 8/10/20 11:46 PM, Scott Z. via FreeIPA-users wrote:
> I stopped the ntp service with the command "timedatectl set_ntp 0"
> I set the new date to be Sept. 1st, 2019 with "timedatectl set-time
> 2019-09-01"
> I waiting a minute and then checked with the "date" command; the problem
> server believes it is Sept. 1st, 2019.
>
> Now when you say 'restart services', I assume you're only referring to
> the ipactl services? In that case I ran "ipactl start
> --ignore-service-failures". Interestingly, when I ran this command it
> not only failed to start pki-tomcatd (which I expected), but actually
> reset the date back to the present/correct time and date. Thus, I
> re-ran the command to set it back to Sept. 1st, 2019.
>
If the server was configured with ntp, "ipactl start" will also restart
ntpd. You need to do the following:
ipactl stop
change date in the past
systemctl start dirsrv@DOMAIN-COM (replace with your domain name)
systemctl start krb5kdc
systemctl start kadmin
systemctl start named-pkcs11 (if IPA is hosting the DNS server)
systemctl start httpd
systemctl start pki-tomcatd@pki-tomcat
Then try getcert resubmit.
> I then ran the "getcert resubmit -i <reqID> command. I just now went
> through these steps again, and it's showing "status: CA_UNREACHABLE"
and
> "ca-error: Internal Error". Stuck now shows 'no'.
> Re-running "certutil -L -d /etc/pki/pki-tomcat/alias -n 'ServerCert
> cert-pki-ca' now yields a new error message, "certutil: could not find
> cert: ServerCert cert-pki-ca", and ": PR_FILE_NOT_FOUND_ERROR: File not
> found"
The cert nickname should contain a dash: "Server-Cert cert-pki-ca"
HTH,
flo
>
> Many Mahalos for your continued support and patience!
> Scott
>
>
>
>
> ------------------------------------------------------------------------
> *From:* Rob Crittenden <rcritten(a)redhat.com>
> *Sent:* Monday, August 10, 2020 11:36 AM
> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>;
> Florence Blanc-Renaud <flo(a)redhat.com>
> *Cc:* Scott Z. <sudz28(a)hotmail.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
> Scott Z. via FreeIPA-users wrote:
>> Whoops! Using the additional command to start tracking this paritcular
>> cert that you included in a different message, I got it in the
"getcert"
>> list (with the "getcert start-tracking -n 'Server-Cert cert-pki-ca'
-d
>> /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B
>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>> '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert
cert-pki-ca"' -P
>> <pin>" command).
>>
>> I have the date rolled back to Sept. 1st, 2019. I guess I have
'some'
>> progress now at least, but still have an issue; checking on the cert
>> with "getcert list -i <requestID>", it shows "status:
CA_REJECTED", and
>> "stuck: yes".
>
> How did you roll the date back? Did you restart services? What date did
> you pick and does it overlap so that all certs are valid?
>
> rob
>
>>
>> Any additional thoughts or help would be greatly appreciated! And
>> thanks for the help so far.
>> Scott
>>
>> ------------------------------------------------------------------------
>> *From:* Scott Z. via FreeIPA-users <freeipa-users(a)lists.fedorahosted.org>
>> *Sent:* Monday, August 10, 2020 10:37 AM
>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Scott
>> Z. <sudz28(a)hotmail.com>
>> *Subject:* [Freeipa-users] Re: pki-tomcatd not starting
>> Â
>> Sorry, I didn't realize I had dropped the mailing list - my mistake!
>>
>> I backed up the files/directories you mentioned below, then I checked on
>> the ra-agent.pem to see if it was still valid (openssl x509 -in
>> /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed
>> currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not After: Aug
>> 10 17:20:41 2021 GMT).
>>
>> Based on that information, and knowing that the bad cert is valid from
>> Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019
>> since all certs will see that date as valid.
>>
>> The only issue I have now is getting the request ID for the expired
>> cert; it doesn't show up in the list of certs when I do "getcert
-list",
>> I can only see it by running "certutil -L -d
>> /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'", and
when
>> I run that it does not show any Request ID associated for it?
>> Scott
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>> *Sent:* Monday, August 10, 2020 8:45 AM
>> *To:* Scott Z. <sudz28(a)hotmail.com>
>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> Â
>> Hi,
>>
>> re-adding the mailing list as the conversation could also help others.
>>
>> On 8/8/20 12:06 AM, Scott Z. wrote:
>>> I did notice when I compare it to another IdM server in the environment,
>>> if I do a "certutil -L -d /etc.httdp/alias" the non-working server
has a
>>> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that
>>> I'm comparing against has a "Signing-Cert" certificate in
addition. Is
>>> this because it's the 'Master' or whatever?ÂÂÂÂ
Should my 'bad' server have
>>> this same Signing-Cert listed?
>>
>> /etc/httpd/alias only needs its own Server-Cert + IPA CA.
>>
>>> Scott
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Scott Z. <sudz28(a)hotmail.com>
>>> *Sent:* Friday, August 7, 2020 10:44 AM
>>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>> /"The interesting part is the list of expired certs on the failing node
>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>> instructions are available here:
>>>
https://access.redhat.com/solutions/3357331 How do I manually renew
>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>> (Replica IPA Server)"/
>>
>> Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
>> /etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
>> the certificates are stored).
>>
>> If the RA cert is valid, you need to find a time window during which the
>> RA cert is already valid (date > notbefore) and the other certs are not
>> expired yet (date < notafter). When you have identified a proper date,
>> stop ntpd (or chronyd, depending on which service is used for time
>> synchronization), move the date back in time to the identified date,
>> start all the services except ntpd, then call "getcert resubmit -i
>> <request id>" for the expired cert(s).
>>
>> Check that the cert has been renewed with "getcert list -i <request
>> id>", the state should display MONITORING. When all the certs are good,
>> you can restart ntpd and the clock will go back to the current date.
>>
>> It's really important to find a date where all the certs are valid
>> because this ensures that the services are able to start and the RA cert
>> allows the authentication that is mandatory for certificate renewal.
>>
>> HTH,
>> flo
>>>
>>> Sadly, after I log in, it's only telling me that it's
"Subscriber
>>> Exclusive Content". Not sure what happened with my
account, I used to
>>> be able to access these docs with no problem but since I took a RHEL
>>> class a couple of weeks back now it's not working any
more. I guess
>>> they did something to screw up my account when I took the class. Grrrrr!!!
>>> Scott
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Sent:* Thursday, August 6, 2020 2:46 AM
>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>>>> Thanks much for the
assistance. Here is where I am with your
suggestions:
>>>> 1) Checked on the cert with "certutil -L -d
/etc/pki/pki-tomcat/alias -n
>>>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed
old
>>>> (almost a year old actually, I assume IPA only checks it when it first
>>>> starts up so it didn't care that it was expired until the server was
>>>> rebooted?)
>>>
>>> certmonger checks the certificate validity periodically (configurable in
>>> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
>>> The system probably had an issue that was not detected and the cert
>>> reached its expiration date.
>>>
>>>>
>>>> 2) ran ipactl start --ignore-service-failures
>>>> ÂÂÂÂÂ
 ÂÂÂÂÂ
 ÂÂÂÂÂ
 a. most services started, obviously
pki-tomcatd did not
>>>> 3) ran "kinit admin"
>>>> ÂÂÂÂÂ
 ÂÂÂÂÂ
 ÂÂÂÂÂ
 a. was forced to change the password, but
otherwise nothing happened
>>>> 4) Ran "ipa config-show |grep -i master
>>>> ÂÂÂÂÂ
 ÂÂÂÂÂ
 ÂÂÂÂÂ
a. I see that the IPA CA renewal master is a different idm machine.
>>>> 5) Ran "getcert list | grep -E
"Request|certificate:|expires:"
>>>> ÂÂÂÂÂ
 ÂÂÂÂÂ
 ÂÂÂÂÂ
a.I see all certs are currently valid (none expired)
>>>> 6) Ran the command "getcert list" on the problem server, but I
cannot
>>>> paste the output here because it's on an airgaped environment so
while I
>>>> apologize for this and realize it makes things more difficult, perhaps
>>>> if you tell me what I should be looking for or more specifically what
>>>> you're interested in I can pluck that out and manually include it
here?
>>>> So in summary, it is indeed an expired "Server-Cert
cert-pki-ca'
>>>> certificate on the problem server, and it can theoretically be renew by
>>>> the Master at this time.
>>> The interesting part is the list of expired certs on the failing node
>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>> instructions are available here:
>>>
https://access.redhat.com/solutions/3357331 How do I manually renew
>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>> (Replica IPA Server)
>>>
>>> flo
>>>
>>>> Many thanks!
>>>> Scott
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>> *Sent:* Monday, August 3, 2020 9:34 PM
>>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>>>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>>>> Not sure I'm sending this to the right place, but here it
goes.ÃÃâ€
’‚ I
>>>>> inherited a FreeIPA/Identity Manager setup in an enclave (no
internet
>>>>> access) environment that is running into problems.ÃÃâ€
’‚ There
are at least 3
>>>>> different IdM servers running in the environment spread out across
>>>>> different geographical areas.ÃÃâ€
’‚ One
of those areas suffered an unschedule
>>>>> power outage recently, and ever since we brought everything back up,
the
>>>>> IdM server for this region is having an issue.ÃÃâ€
’‚ÂÂÂÂÂ
Please bear with me as I
>>>>> have zero formal experience, training, or real knowledge with IdM.
>>>>>
>>>>> Logging in to the serverv (it's a VM server, running Centos 7.5),
I run
>>>>> "ipactl status" and it shows "Directory Service:
STOPPED".ÃÃâ€
’‚ I
then run
>>>>> "ipactl restart", and things go fine until it gets to
"Starting
>>>>> pki-tomcatd Service", where it hangs for quite some time before
failing
>>>>> to start and killing all the other services.ÃÃâ€
’‚ I
check the log at
>>>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>>>> (forgive any mistypings, I have to manually type these in as I
can't
>>>>> import or screen capure the logs and put them in this message):
>>>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is
invalid:
>>>>> Invalid certificate: (-8181) Peer's Certificate has
expired/"
>>>>> And slightly further down in the same log:
>>>>> "/Cannot reset factory: connections not all returned/"
>>>>> "/CertificateAuthority.shutdown: failed to reset dbFactory:
Cannot reset
>>>>> LDAP connection factory because some connections are still
outstanding/"
>>>>> ... still further down"
>>>>> "/returnConn:mNumConns now 3 Invalid class name
repositorytop/"
>>>>>
>>>>> Assuming I have some weird certificate issue with this server in
>>>>> particular, I try to run a few more commands:
>>>>> "certutil -L -d /etc/httpd/alias"ÃÃâ€
’‚ÂÂÂÂÂ
--> returns a Server-Cert listing
>>>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA
CA with CT,C,C
>>>>> for it's attributes.ÃÃâ€
’‚ÂÂÂÂÂ
Comparing to a second IdM server in this
>>>>> environment, it seems to be missing a "Signing-Cert"?
>>>>>
>>>> Hi,
>>>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
>>>> has the nickname 'Server-Cert cert-pki-ca'. You should check that
this
>>>> one is not expired with:
>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert
cert-pki-ca'
>>>> | grep 'Not '
>>>>
>>>> If the certificate is indeed expired, it will have to be renewed but you
>>>> need first to find which IPA server is the CA renewal master. On your
>>>> server, force a service start and check the CA renewal master:
>>>> # ipactl start --ignore-service-failures
>>>> # kinit admin
>>>> # ipa config-show | grep "renewal master"
>>>> ÂÂÂÂÂ
 IPA CA renewal master:
server.domain.com
>>>>
>>>> You need to make sure that all the certificates are valid on the CA
>>>> renewal master:
>>>> (on the CA renewal master)# getcert list | grep -E
>>>> "Request|certificate:|expires:"
>>>>
>>>> - if the CA renewal master is not OK, please post the output of "#
>>>> getcert list" (without the grep) on the CA renewal master. This
node
>>>> will have to be repaired first.
>>>> - if the CA renewal master is OK, please post the output of "#
getcert
>>>> list" (also without the grep) on the failing node.
>>>>
>>>> We'll be able to help based on this information.
>>>> flo
>>>>
>>>>> I also did a "getcert list", and all certs it has show that
they expire
>>>>> in the future (nothing shows as bein currently expired).
>>>>>
>>>>> I'm confused; it seems to that it is seeing an expired cert
*somewhere*,
>>>>> but how do I track down which 'peer' the log file is talking
about that
>>>>> has an expired cert?ÃÃâ€
’‚ÂÂÂÂÂ
Meanwhile none of the linux clients that point to
>>>>> this IdM server are allowing people to log in/authenticate.
>>>>> Many thanks for any help!
>>>>> Scott
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
>>>>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
>>>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>
>>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>