Hi Rob,
I found this on my second server in /var/log/pki/pki-tomcat/ca/debug:
SSL handshake happened
Could not connect to LDAP server host freeipa-02.corp.mydomain.de port 636
Error netscape.ldap.LDAPException: Authenticatio
n failed (48)
On my primary server I found this:
Internal Database Error encountered: Could not connect to LDAP server host
freeipa-02.corp.mydomain.de port 636 Error netscape.ldap.LDAPException:
Unable to create soc
ket: java.net.UnknownHostException: freeipa-02.corp.mydomain.de: Name or
service not known (-1)
Looks like that it was unable to resolve the name of the second host (why
primary host is connecting to secondary?). I added an entry to hosts file
but still CA does not start.
ср, 12 февр. 2020 г. в 07:58, Dmitri Moudraninets <
dmitry.a.moudraninets(a)gmail.com>:
Hi Rob,
What cat I do to troubleshoot CA?
On Wed 12. Feb 2020 at 01:00, Rob Crittenden <rcritten(a)redhat.com> wrote:
> Dmitri Moudraninets wrote:
> > Hi Rob,
> >
> >
> > It seems that it does not help. I found a backup which was made via
> > ipa-backup this summer. Can I use it somehow for recovery? We did
> > nothing to certificates since that time. We only added
> users/groups/servers.
> >
> > Current situation:
> > I can't update certificates. getcert list shows multiple certificates
> > with CA_UNREACHABLE status:
> > status: CA_UNREACHABLE
> > ca-error: Error 35 connecting to
> >
https://freeipa.corp.mydomain.de:8443/ca/agent/ca/profileReview: SSL
> > connect error.
> >
> >
> > pki-tomcatd is not starting:
> > [root@freeipa ipa]# ipactl start --ignore-service-failures
> > Starting Directory Service
> > Starting krb5kdc Service
> > Starting kadmin Service
> > Starting named Service
> > Starting httpd Service
> > Starting ipa-custodia Service
> > Starting ntpd Service
> > Starting pki-tomcatd Service
> > Failed to start pki-tomcatd Service
> > Forced start, ignoring pki-tomcatd Service, continuing normal operation
> > Starting smb Service
> > Starting winbind Service
> > Starting ipa-otpd Service
> > Starting ipa-dnskeysyncd Service
> > ipa: INFO: The ipactl command was successful
>
> The CA was working previously, what exactly did you do? Changing the RA
> cert would in no way affect the startup of the CA. I'd carefully review
> your shell history to see what you did and check the CA logs to see why
> it won't start up.
>
> Of course the CA is unreachable if it hasn't started, this error is
> expected. You can't debug a CA not starting up via certmonger as it is
> just a client (and in some cases uses the previously broken RA cert for
> communication).
>
> So get the CA starting up first, then tackle the RA cert/key.
>
> rob
> >
> > пн, 25 нояб. 2019 г. в 15:47, Rob Crittenden <rcritten(a)redhat.com
> > <mailto:rcritten@redhat.com>>:
> >
> > Dmitri Moudraninets wrote:
> > > Hi Rob,
> > >
> > > I recovered the key file. Restarted FreeIPA and certmonger. Now
> issue
> > > looks different:
> > > image.png
> > >
> > > Subjects disappeared. If I click on a certificate 29 I see this:
> > > cannot connect to
> > > 'https://freeipa.corp.mydomain.de:443/ca/agent/ca/displayBySerial
> ':
> > > [Errno 13] Permission denied
> >
> > Set the same ownership/permissions on the key as you did the cert
> and
> > run restorecon on it.
> >
> > rob
> >
> > >
> > > пн, 25 нояб. 2019 г. в 13:58, Rob Crittenden <rcritten(a)redhat.com
> > <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>:
> > >
> > > Dmitri Moudraninets wrote:
> > > > Hi Rob,
> > > >
> > > >
> > > >
> > > > I did the following:
> > > > I removed original ra-agent.pem and ra-agent key
> > > > and
> > > > openssl x509 -in /root/debug.cert -out
> /var/lib/ipa/ra-agent.pem
> > > > chown root:ipaapi /var/lib/ipa/ra-agent.pem
> > > > chmod 0440 /var/lib/ipa/ra-agent.pem
> > > > restorecon /var/lib/ipa/ra-agent.pem
> > >
> > > You removed the key!? I sure hope you have a backup of it.
> > >
> > > Put it back and I think that will resolve things.
> > >
> > > >
> > > > Successfully restarted FreeIPA:
> > > > Directory Service: RUNNING
> > > > krb5kdc Service: RUNNING
> > > > kadmin Service: RUNNING
> > > > named Service: RUNNING
> > > > httpd Service: RUNNING
> > > > ipa-custodia Service: RUNNING
> > > > ntpd Service: RUNNING
> > > > pki-tomcatd Service: RUNNING
> > > > smb Service: RUNNING
> > > > winbind Service: RUNNING
> > > > ipa-otpd Service: RUNNING
> > > > ipa-dnskeysyncd Service: RUNNING
> > > > ipa: INFO: The ipactl command was successful
> > >
> > > The agent cert is not required for the CA to operate.
> > >
> > > > Now GUI shows different error:
> > > > cannot connect to
> > > >
> > 'https://freeipa.corp.mydomain.de:443/ca/agent/ca/displayBySerial':
> > > > [Errno 2] No such file or directory
> > > >
> > > >
> > > > [root@freeipa ~]# getcert list -f /var/lib/ipa/ra-agent.pem
> > > > Number of certificates and requests being tracked: 16.
> > > > Request ID '20180912151611':
> > > > status: NEED_CSR
> > > > stuck: no
> > > > key pair storage:
> type=FILE,location='/var/lib/ipa/ra-agent.key'
> > > > certificate:
type=FILE,location='/var/lib/ipa/ra-agent.pem'
> > > > CA: dogtag-ipa-ca-renew-agent
> > > > issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
> > <
http://CORP.MYDOMAIN.DE>
> > > <
http://CORP.MYDOMAIN.DE>
> > > > <
http://CORP.MYDOMAIN.DE>
> > > > subject: CN=IPA RA,O=CORP.MYDOMAIN.DE
> > <
http://CORP.MYDOMAIN.DE> <
http://CORP.MYDOMAIN.DE>
> > > <
http://CORP.MYDOMAIN.DE>
> > > > expires: 2019-11-25 15:32:12 UTC
> > > > key usage:
> > >
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > > > eku: id-kp-serverAuth,id-kp-clientAuth
> > > > pre-save command:
> /usr/libexec/ipa/certmonger/renew_ra_cert_pre
> > > > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
> > > > track: yes
> > > > auto-renew: yes
> > >
> > > This shows that the certificate has the right subject now
> > which is good
> > > but you removed its private key so it won't work.
> > >
> > > rob
> > >
> > > >
> > > > сб, 23 нояб. 2019 г. в 20:26, Rob Crittenden
> > <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
> > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>
> > > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>>:
> > > >
> > > > Dmitri Moudraninets wrote:
> > > > > Hi Rob,
> > > > >
> > > > > ldapsearch -LLL -o ldif-wrap=no -x -D
'cn=directory
> > manager' -W
> > > > > -b uid=ipara,ou=People,o=ipaca usercertificate
> > > > >
> > > > > shows me the following:
> > > > >
> > > > > Issuer: O=CORP.MYDOMAIN.DE
> > <
http://CORP.MYDOMAIN.DE> <
http://CORP.MYDOMAIN.DE>
> > > <
http://CORP.MYDOMAIN.DE>
> > > > <
http://CORP.MYDOMAIN.DE>,
> > > > > CN=Certificate Authority
> > > > > Validity
> > > > > Not Before: Dec 5 15:32:12 2017 GMT
> > > > > Not After : *Nov 25 15:32:12 2019* GMT
> > > > >
> > > > > It's going to expire on Monday. Can it be a
problem?
> > > >
> > > > You didn't provide the cert subject so I can't be
sure
> > this is
> > > the right
> > > > cert. If it contains CN = IPA RA then it is.
> > > >
> > > > And yes, it expires in two days. What you'd need to
do
> is
> > > restore it per
> > > > my previous instruction into /var/lib/ipa/ra-agent.pem
> > on the
> > > renewal
> > > > master (ipa config-show to see which one it is).
> > > >
> > > > Then run:
> > > >
> > > > # getcert resubmit -f /var/lib/ipa/ra-agent.pem
> > > >
> > > > That should renew the cert.
> > > >
> > > > On the other masters I'd run the same command and
that
> > may fix
> > > things
> > > > there as well.
> > > >
> > > > rob
> > > >
> > > > > I tried this command:
> > > > > openssl x509 -text -in /var/lib/ipa/ra-agent.pem
> > > > >
> > > > > and it shows the following:
> > > > > Certificate:
> > > > > Data:
> > > > > Version: 3 (0x2)
> > > > > Serial Number: 28 (0x1c)
> > > > > Signature Algorithm: sha256WithRSAEncryption
> > > > > Issuer: O=CORP.MYDOMAIN.DE
> > <
http://CORP.MYDOMAIN.DE> <
http://CORP.MYDOMAIN.DE>
> > > <
http://CORP.MYDOMAIN.DE>
> > > > <
http://CORP.MYDOMAIN.DE>,
> > > > > CN=Certificate Authority
> > > > > Validity
> > > > > Not Before: Oct 29 10:39:47 2019 GMT
> > > > > Not After : Oct 29 09:39:47 2021 GMT
> > > > > Subject: O=CORP.MYDOMAIN.DE
> > <
http://CORP.MYDOMAIN.DE>
> > > <
http://CORP.MYDOMAIN.DE> <
http://CORP.MYDOMAIN.DE>,
CN=dmud
> > > > > Subject Public Key Info:
> > > > > Public Key Algorithm: rsaEncryption
> > > > > Public-Key: (2048 bit)
> > > > > Modulus:
> > > > >
> > > 00:ba:09:81:99:9b:17:99:07:5a:10:28:c8:7a:03:
> > > > > ...
> > > > >
> > > 18:db:02:ce:b4:66:ce:5a:e9:12:af:d3:da:bf:f7:
> > > > > 66:5f
> > > > > Exponent: 65537 (0x10001)
> > > > > X509v3 extensions:
> > > > > X509v3 Authority Key Identifier:
> > > > > keyid:D2:...70:BF
> > > > >
> > > > > X509v3 Subject Key Identifier:
> > > > > DE:...:51:0A
> > > > > X509v3 Subject Alternative Name:
> > > > > email:dmud@corp.mydomain.de
> > <mailto:email%3Admud@corp.mydomain.de>
> > > <mailto:email%3Admud@corp.mydomain.de
> > <mailto:email%253Admud@corp.mydomain.de>>
> > > > <mailto:email%3Admud@corp.mydomain.de
> > <mailto:email%253Admud@corp.mydomain.de>
> > > <mailto:email%253Admud@corp.mydomain.de
> > <mailto:email%25253Admud@corp.mydomain.de>>>
> > > > > <mailto:email%3Admud@corp.mydomain.de
> > <mailto:email%253Admud@corp.mydomain.de>
> > > <mailto:email%253Admud@corp.mydomain.de
> > <mailto:email%25253Admud@corp.mydomain.de>>
> > > > <mailto:email%253Admud@corp.mydomain.de
> > <mailto:email%25253Admud@corp.mydomain.de>
> > > <mailto:email%25253Admud@corp.mydomain.de
> > <mailto:email%2525253Admud@corp.mydomain.de>>>>
> > > > > Authority Information Access:
> > > > > OCSP -
> > > URI:http://ipa-ca.corp.mydomain.de/ca/ocsp
> > > > >
> > > > >
> > > > > I did nothing to /var/lib/ipa/ra-agent.pem yet.
> > > > >
> > > > >
> > > > > чт, 21 нояб. 2019 г. в 16:54, Rob Crittenden
> > > <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
> > > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com
> >
> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
> > > > > <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
> > <mailto:rcritten@redhat.com>>
> > > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
> > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>>>:
> > > > >
> > > > > Dmitri Moudraninets wrote:
> > > > > > Hi Rob,
> > > > > >
> > > > > > Yes both masters are failing the same way.
> Output
> > > of openssl
> > > > x509
> > > > > -noout
> > > > > > -modulus -in /var/lib/ipa/ra-agent.pem is
the
> > same on both
> > > > masters.
> > > > > > Output of openssl rsa -noout -modulus -in
> > > > /var/lib/ipa/ra-agent.key is
> > > > > > also the same on both masters. But the
output of
> > the first
> > > > command is
> > > > > > not the same as the output of the second
> command.
> > > > > >
> > > > > > I can't remember that I troubleshoot any
other
> > > problems but we
> > > > > tried to
> > > > > > generate some personal certificates for
some
> users.
> > > Also we
> > > > tried to
> > > > > > generate certificates with key files for
some
> of our
> > > internal
> > > > > services.
> > > > > > We did that for the first time and it worked
at
> the
> > > end. Also I
> > > > > changed
> > > > > > the admin password not so long ago.
> > > > > >
> > > > > >
> > > > > > Below you can find the output of the
requested
> > commands:
> > > > > >
> > > > > >
> > > > > > [root@second_master ~]# getcert list -f
> > > > /var/lib/ipa/ra-agent.pem
> > > > > > Number of certificates and requests being
> > tracked: 9.
> > > > > > Request ID '20180912151730':
> > > > > > status: MONITORING
> > > > > > stuck: no
> > > > > > key pair storage:
> > > type=FILE,location='/var/lib/ipa/ra-agent.key'
> > > > > > certificate:
> > > type=FILE,location='/var/lib/ipa/ra-agent.pem'
> > > > > > CA: dogtag-ipa-ca-renew-agent
> > > > > > issuer: CN=Certificate
> > Authority,O=CORP.MYDOMAIN.DE <
http://CORP.MYDOMAIN.DE>
> > > <
http://CORP.MYDOMAIN.DE>
> > > > <
http://CORP.MYDOMAIN.DE>
> > > > > <
http://CORP.MYDOMAIN.DE>
> > > > > > <
http://CORP.MYDOMAIN.DE>
> > > > > > subject: CN=dmud,O=CORP.MYDOMAIN.DE
> > <
http://CORP.MYDOMAIN.DE>
> > > <
http://CORP.MYDOMAIN.DE>
> > > > <
http://CORP.MYDOMAIN.DE>
<
http://CORP.MYDOMAIN.DE>
> > > > > <
http://CORP.MYDOMAIN.DE>
> > > > > >
*<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
I see a
> > username here.
> > > > Does it have
> > > > > > to be like that?*
> > > > > > expires: 2021-10-29 09:39:47 UTC
> > > > > > email: dmud(a)corp.mydomain.de
> > <mailto:dmud@corp.mydomain.de>
> > > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de>>
> > <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>
> > > <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de
> >>>
> > > > <mailto:dmud@corp.mydomain.de
> > <mailto:dmud@corp.mydomain.de> <mailto:dmud@corp.mydomain.de
> > <mailto:dmud@corp.mydomain.de>>
> > > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de>
> > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de>>>>
> > > > > <mailto:dmud@corp.mydomain.de
> > <mailto:dmud@corp.mydomain.de>
> > > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de>>
> > <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>
> > > <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de
> >>>
> > > > <mailto:dmud@corp.mydomain.de
> > <mailto:dmud@corp.mydomain.de> <mailto:dmud@corp.mydomain.de
> > <mailto:dmud@corp.mydomain.de>>
> > > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de>
> > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de>>>>>
> > > > > > key usage:
> > > > >
> > >
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > > > > > eku: id-kp-serverAuth,id-kp-clientAuth
> > > > > > pre-save command:
> > > /usr/libexec/ipa/certmonger/renew_ra_cert_pre
> > > > > > post-save command:
> > > /usr/libexec/ipa/certmonger/renew_ra_cert
> > > > > > track: yes
> > > > > > auto-renew: yes
> > > > >
> > > > > Right, someone overwrote the RA agent
certificate.
> > > > >
> > > > > Look to see if the user entry in the CA has the
> > right cert:
> > > > >
> > > > > $ ldapsearch -LLL -o ldif-wrap=no -x -D
> 'cn=directory
> > > manager'
> > > > -W -b
> > > > > uid=ipara,ou=People,o=ipaca usercertificate
> > > > >
> > > > > Put the base64 value of the usercertificate
> > attribute into a
> > > > file and
> > > > > add a prefix/suffix around it:
> > > > >
> > > > > -----BEGIN CERTIFICATE-----
> > > > > MII....blah=
> > > > > -----END CERTIFICATE-----
> > > > >
> > > > > $ openssl x509 -text -in /path/to/file
> > > > >
> > > > > If the Subject is O = CORP.MYDOMAIN.DE
> > <
http://CORP.MYDOMAIN.DE>
> > > <
http://CORP.MYDOMAIN.DE>
> > > > <
http://CORP.MYDOMAIN.DE>
<
http://CORP.MYDOMAIN.DE>, CN
> > > > > = IPA RA then that's a good
> > > > > start. Also look at the expires date to be sure
> it is
> > > still valid.
> > > > >
> > > > > Assuming that is ok then re-run the openssl
> > modulus commands
> > > > to ensure
> > > > > they are the same.
> > > > >
> > > > > Assuming that too is ok then you have the
proper,
> > valid RA
> > > > agent cert.
> > > > > In that case I'd move the current file out of
the
> > way, who
> > > > knows what it
> > > > > is, then run:
> > > > >
> > > > > # openssl x509 -in /path/to/file -out
> > > > /var/lib/ipa/ra-agent.pem (just to
> > > > > properly format the agent cert)
> > > > > # chown root:ipaapi /var/lib/ipa/ra-agent.pem
> > > > > # chmod 0440 /var/lib/ipa/ra-agent.pem
> > > > > # restorecon /var/lib/ipa/ra-agent.pem
> > > > >
> > > > > Then try something like: ipa cert-show 1
> > > > >
> > > > > This will exercise the RA agent cert and as long
> > as you
> > > don't
> > > > get an
> > > > > error back things are working again.
> > > > >
> > > > > The cert is common among all masters so you can
> > copy the
> > > file
> > > > to your
> > > > > other master(s), ensuring proper ownership,
> > permissions and
> > > > SELinux
> > > > > context.
> > > > >
> > > > > rob
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > WBR
> > > > > Dmitry
> > > >
> > > >
> > > >
> > > > --
> > > > With best regards/Mit freundlichen Grüßen
> > > >
> > > > Moudraninets Dmitry, RHCSA
> > > >
http://www.linkedin.com/in/moudraninets
> > > >
http://www.xing.com/profile/Dmitry_Mudraninets
> > >
> > >
> > >
> > > --
> > > With best regards/Mit freundlichen Grüßen
> > >
> > > Moudraninets Dmitry, RHCSA
> > >
http://www.linkedin.com/in/moudraninets
> > >
http://www.xing.com/profile/Dmitry_Mudraninets
> >
> >
> >
> > --
> > With best regards/Mit freundlichen Grüßen
> >
> > Moudraninets Dmitry, RHCSA
> >
http://www.linkedin.com/in/moudraninets
> >
http://www.xing.com/profile/Dmitry_Mudraninets
>
> --
With best regards/Mit freundlichen Grüßen
Moudraninets Dmitry, RHCSA
http://www.linkedin.com/in/moudraninets
http://www.xing.com/profile/Dmitry_Mudraninets