On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
> On 26-10-18 18:20, Florence Blanc-Renaud wrote:
>> On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
>>>
>>> On 26-10-18 18:00, Timo Aaltonen wrote:
>>>> On 26.10.2018 18.59, Kees Bakker wrote:
>>>>> On 26-10-18 14:55, Timo Aaltonen wrote:
>>>>>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
>>>>>>> On 25-10-18 20:46, Timo Aaltonen wrote:
>>>>>>>> On 25.10.2018 21.44, Rob Crittenden wrote:
>>>>>>>>> Kees Bakker wrote:
>>>>>>>>>> On 25-10-18 16:11, Rob Crittenden wrote:
>>>>>>>>>>> Kees Bakker via FreeIPA-users wrote:
>>>>>>>>>>>> On 25-10-18 14:18, Rob Crittenden wrote:
>>>>>>>>>>>>> Kees Bakker via FreeIPA-users wrote:
>>>>>>>>>>>>>> Could it be that this error
already existed since we started? Notice
>>>>>>>>>>>>>> the Request ID of 2016..., and
the expires: 2018-10-24.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # getcert list -n ipaCert | sed
blabla
>>>>>>>>>>>>>> Number of certificates and
requests being tracked: 8.
>>>>>>>>>>>>>> Request ID
'20161103094546':
>>>>>>>>>>>>>> status: CA_UNREACHABLE
>>>>>>>>>>>>>> ca-error: Error 77
connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL
CA cert (path? access rights?).
>>>>>>>>>>>>>> stuck: no
>>>>>>>>>>>>>> key pair storage:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt'
>>>>>>>>>>>>>> certificate:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB'
>>>>>>>>>>>>>> CA:
dogtag-ipa-ca-renew-agent
>>>>>>>>>>>>>> issuer: CN=Certificate
Authority,O=MYDOMAIN
>>>>>>>>>>>>>> subject: CN=IPA
RA,O=MYDOMAIN
>>>>>>>>>>>>>> expires: 2018-10-24 08:45:40
UTC
>>>>>>>>>>>>>> key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>>>>>>>>>>>>> eku:
id-kp-serverAuth,id-kp-clientAuth
>>>>>>>>>>>>>> pre-save command:
/usr/lib/ipa/certmonger/renew_ra_cert_pre
>>>>>>>>>>>>>> post-save command:
/usr/lib/ipa/certmonger/renew_ra_cert
>>>>>>>>>>>>>> track: yes
>>>>>>>>>>>>>> auto-renew: yes
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In other words, is this the same
issue as
https://pagure.io/freeipa/issue/7422 ?
>>>>>>>>>>>>> The problem is your certs expired
yesterday so connections won't work
>>>>>>>>>>>>> (the code and message don't come
from within certmonger).
>>>>>>>>>>>>>
>>>>>>>>>>>>> certmonger _should_ have renewed
them. Try killing ntpd, going back a
>>>>>>>>>>>>> few days, restart krb5kdc, dirsrv,
httpd and the CA then certmonger and
>>>>>>>>>>>>> see what happens.
>>>>>>>>>>>>>
>>>>>>>>>>>> Easy for you to say. You know what
you're doing :-)
>>>>>>>>>>>> For me it's all magic.
>>>>>>>>>>>>
>>>>>>>>>>>> Anyway, I'll try it. I'm just
scared to set the clock back, because there may
>>>>>>>>>>>> be clients in the network that use this
server as a NTP server.
>>>>>>>>>>>>
>>>>>>>>>>>> Another thing I want to mention is that
the error started showing up two days
>>>>>>>>>>>> ago, on Oct 22, while the expiration is
today, Oct 24.
>>>>>>>>>>>>
>>>>>>>>>>> It shouldn't take more than a few minutes
to roll back time, restart
>>>>>>>>>>> services and see what happens. I think your
NTP clients will be able to
>>>>>>>>>>> recover ok if the server is not available for
a few minutes.
>>>>>>>>>>>
>>>>>>>>>>> certmonger logs to syslog so you probably
want to look at that to see if
>>>>>>>>>>> you can find a reason the certs weren't
renewed automatically.
>>>>>>>>>>>
>>>>>>>>>> No, that didn't help.
>>>>>>>>>> And in the syslog there was nothing more than
this. (I had to stop the
>>>>>>>>>> nameserver because it was spitting out lots of
messages.)
>>>>>>>>>>
>>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been
changed
>>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has
been changed
>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping
Certificate monitoring and PKI enrollment...
>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped
Certificate monitoring and PKI enrollment.
>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting
Certificate monitoring and PKI enrollment...
>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started
Certificate monitoring and PKI enrollment.
>>>>>>>>>> Oct 11 06:00:05 ipasrv certmonger[131018]:
2018-10-11 06:00:05 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profile
>>>>>>>>>> Review: Problem with the SSL CA cert (path?
access rights?).
>>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>>> Oct 11 06:00:07 ipasrv certmonger[131018]:
2018-10-11 06:00:07 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>>> Oct 11 06:00:17 ipasrv certmonger[131018]:
2018-10-11 06:00:17 [131018] Error 77 connecting to
https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access
rights?).
>>>>>>>>>>
>>>>>>>>> Ok, I think I know what is going on. This is Ubuntu
which AFAIK still
>>>>>>>>> lacks nss-pem. That is probably why it can't
connect to renew the certs.
>>>>>>>>>
>>>>>>>>> I don't know if there is a workaround. Timo, do
you know?
>>>>>>>> Ubuntu 18.04 and up have libnsspem, and certmonger
depends on it. I've
>>>>>>>> never tested cert renewal though.
>>>>>>>>
>>>>>>> Does that mean, I'm screwed? What options do I have?
>>>>>>> Live with it?
>>>>>>> Migrate to, say Centos?
>>>>>>> Try to upgrade the server to Ubuntu 18.04 (with uncertainty
whether it will work)?
>>>>>>> Something else?
>>>>>> Stock 18.04 has other issues, there's an updated version on
>>>>>> ppa:freeipa/staging which is backported from 18.10 and should be
fine
>>>>>> and hopefully provided as a stable update on 18.04 later on.
>>>>>>
>>>>>> But you could try pulling libnsspem from 18.04, and *then* roll
back time?
>>>>>>
>>>>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
>>>>>
>>>>> Then I stopped ntp (and bind).
>>>>> Set the time back to Oct 11
>>>>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd,
certmonger
>>>>> (in that order).
>>>>>
>>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding
request to dogtag-ipa-renew-agent
>>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit:
dogtag-ipa-renew-agent returned 3
>>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03
[168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview:
Peer certificate cannot be authenticated with given CA certificates.
>>>>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12
[168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview:
Peer certificate cannot be authenticated with given CA certificates.
>>>>>
>>>>> :-(
>>>>>
>>>>> Rob said also to restart CA.
>>>>> "restart krb5kdc, dirsrv, httpd and the CA then
certmonger"
>>>>> I don't know which service that is. Does that matter?
>>>> systemctl restart ipa?
>>>>
>>>>
>>> I'm a bit scared to restart service ipa, because it also restarts several
other services,
>>> link bind, and perhaps ntp. The latter is the one that I want to be
absolutely in control
>>> of not starting.
>> And you're right! The CA is pki-tomcatd, so you already restarted it.
>>
>>> It's getting too late now, time for weekend. I'll give it another try
on Monday.
>>> Meanwhile I want to point at the changed message. In case that rings a bell
for
>>> someone.
>>>
>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error
60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate
cannot be authenticated with given CA certificates.
>>>
>> You can have a look at Rob's blog for additional items to check:
>>
https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-auth...
> Thanks, I just stumbled on it myself. Interesting read, although I don't quite
> understand all details.
>
> I really need some guidance what to do next. I tried the date trick, I installed
> libnsspem (from Ubunu 18.04). The certmonger error message changed from
> Error 77 into Error 60, but the problem remained.
>
> Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
>
> Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess
> WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d
background process
> java.lang.NullPointerException
> at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113)
> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
> at java.lang.Thread.run(Thread.java:748)
>
> I could do the date trick again, but then the question is, why didn't it work
last time?
I tried it once more to set the date back and to restart the renewal process.
I keep getting Error 60 with
Peer certificate cannot be authenticated with given CA certificates.
Also, I had a look at Rob's blog. But I'm lost at what to do with
"... the fix was to reset the NSS trust flags in the Apache NSS database"
The curl command (with the date set back) seems to connect but it gives a
gnutls_handshake() failed: Illegal parameter
And then I don't know what it looks like if
" You should get client certificate not found."
So I didn't try the certutil -M commands.
BTW. While setting the date back I have bind (DNS) switched off because it
gave a lot of messages in my /var/log/syslog.
Right, this is for a previous version of IPA on RHEL/Fedora. The RA
agent cert used to be stored in the Apache NSS database.
curl linked aganst gnu_tls? I have no idea, but it isn't this.
This error message originates in curl.
What is the validity of your CA certificate itself? Is it still valid
back in time?
rob