Hi Rob,

What cat I do to troubleshoot CA?

On Wed 12. Feb 2020 at 01:00, Rob Crittenden <rcritten@redhat.com> wrote:
Dmitri Moudraninets wrote:
> Hi Rob,
>
>
> It seems that it does not help. I found a backup which was made via
> ipa-backup this summer. Can I use it somehow for recovery? We did
> nothing to certificates since that time. We only added users/groups/servers.
>
> Current situation:
> I can't update certificates. getcert list shows multiple certificates
> with CA_UNREACHABLE status:
> status: CA_UNREACHABLE
>         ca-error: Error 35 connecting to
> https://freeipa.corp.mydomain.de:8443/ca/agent/ca/profileReview: SSL
> connect error.
>
>
> pki-tomcatd is not starting:
> [root@freeipa ipa]# ipactl start --ignore-service-failures
> Starting Directory Service
> Starting krb5kdc Service
> Starting kadmin Service
> Starting named Service
> Starting httpd Service
> Starting ipa-custodia Service
> Starting ntpd Service
> Starting pki-tomcatd Service
> Failed to start pki-tomcatd Service
> Forced start, ignoring pki-tomcatd Service, continuing normal operation
> Starting smb Service
> Starting winbind Service
> Starting ipa-otpd Service
> Starting ipa-dnskeysyncd Service
> ipa: INFO: The ipactl command was successful

The CA was working previously, what exactly did you do? Changing the RA
cert would in no way affect the startup of the CA. I'd carefully review
your shell history to see what you did and check the CA logs to see why
it won't start up.

Of course the CA is unreachable if it hasn't started, this error is
expected. You can't debug a CA not starting up via certmonger as it is
just a client (and in some cases uses the previously broken RA cert for
communication).

So get the CA starting up first, then tackle the RA cert/key.

rob
>
> пн, 25 нояб. 2019 г. в 15:47, Rob Crittenden <rcritten@redhat.com
> <mailto:rcritten@redhat.com>>:
>
>     Dmitri Moudraninets wrote:
>     > Hi Rob,
>     >
>     > I recovered the key file. Restarted FreeIPA and certmonger. Now issue
>     > looks different:
>     > image.png
>     >
>     > Subjects disappeared. If I click on a certificate 29 I see this:
>     > cannot connect to
>     > 'https://freeipa.corp.mydomain.de:443/ca/agent/ca/displayBySerial':
>     > [Errno 13] Permission denied
>
>     Set the same ownership/permissions on the key as you did the cert and
>     run restorecon on it.
>
>     rob
>
>     >
>     > пн, 25 нояб. 2019 г. в 13:58, Rob Crittenden <rcritten@redhat.com
>     <mailto:rcritten@redhat.com>
>     > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>:
>     >
>     >     Dmitri Moudraninets wrote:
>     >     > Hi Rob,
>     >     >
>     >     >
>     >     >
>     >     > I did the following:
>     >     > I removed original ra-agent.pem and ra-agent key
>     >     > and
>     >     > openssl x509 -in /root/debug.cert -out /var/lib/ipa/ra-agent.pem
>     >     > chown root:ipaapi /var/lib/ipa/ra-agent.pem
>     >     > chmod 0440 /var/lib/ipa/ra-agent.pem
>     >     > restorecon /var/lib/ipa/ra-agent.pem
>     >
>     >     You removed the key!? I sure hope you have a backup of it.
>     >
>     >     Put it back and I think that will resolve things.
>     >
>     >     >
>     >     > Successfully restarted FreeIPA:
>     >     > Directory Service: RUNNING
>     >     > krb5kdc Service: RUNNING
>     >     > kadmin Service: RUNNING
>     >     > named Service: RUNNING
>     >     > httpd Service: RUNNING
>     >     > ipa-custodia Service: RUNNING
>     >     > ntpd Service: RUNNING
>     >     > pki-tomcatd Service: RUNNING
>     >     > smb Service: RUNNING
>     >     > winbind Service: RUNNING
>     >     > ipa-otpd Service: RUNNING
>     >     > ipa-dnskeysyncd Service: RUNNING
>     >     > ipa: INFO: The ipactl command was successful
>     >
>     >     The agent cert is not required for the CA to operate.
>     >
>     >     > Now GUI shows different error:
>     >     > cannot connect to
>     >     >
>     'https://freeipa.corp.mydomain.de:443/ca/agent/ca/displayBySerial':
>     >     > [Errno 2] No such file or directory
>     >     >
>     >     >
>     >     > [root@freeipa ~]# getcert list -f /var/lib/ipa/ra-agent.pem
>     >     > Number of certificates and requests being tracked: 16.
>     >     > Request ID '20180912151611':
>     >     > status: NEED_CSR
>     >     > stuck: no
>     >     > key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>     >     > certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>     >     > CA: dogtag-ipa-ca-renew-agent
>     >     > issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE>
>     >     > <http://CORP.MYDOMAIN.DE>
>     >     > subject: CN=IPA RA,O=CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE> <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE>
>     >     > expires: 2019-11-25 15:32:12 UTC
>     >     > key usage:
>     >     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>     >     > eku: id-kp-serverAuth,id-kp-clientAuth
>     >     > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>     >     > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>     >     > track: yes
>     >     > auto-renew: yes
>     >
>     >     This shows that the certificate has the right subject now
>     which is good
>     >     but you removed its private key so it won't work.
>     >
>     >     rob
>     >
>     >     >
>     >     > сб, 23 нояб. 2019 г. в 20:26, Rob Crittenden
>     <rcritten@redhat.com <mailto:rcritten@redhat.com>
>     >     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
>     >     > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
>     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>:
>     >     >
>     >     >     Dmitri Moudraninets wrote:
>     >     >     > Hi Rob,
>     >     >     >
>     >     >     > ldapsearch -LLL -o ldif-wrap=no -x -D 'cn=directory
>     manager' -W
>     >     >     > -b uid=ipara,ou=People,o=ipaca usercertificate
>     >     >     >
>     >     >     > shows me the following:
>     >     >     >
>     >     >     >         Issuer: O=CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE> <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE>
>     >     >     <http://CORP.MYDOMAIN.DE>,
>     >     >     > CN=Certificate Authority
>     >     >     >         Validity
>     >     >     >             Not Before: Dec  5 15:32:12 2017 GMT
>     >     >     >             Not After : *Nov 25 15:32:12 2019* GMT
>     >     >     >
>     >     >     > It's going to expire on Monday. Can it be a problem?
>     >     >
>     >     >     You didn't provide the cert subject so I can't be sure
>     this is
>     >     the right
>     >     >     cert. If it contains CN = IPA RA then it is.
>     >     >
>     >     >     And yes, it expires in two days. What you'd need to do is
>     >     restore it per
>     >     >     my previous instruction into /var/lib/ipa/ra-agent.pem
>     on the
>     >     renewal
>     >     >     master (ipa config-show to see which one it is).
>     >     >
>     >     >     Then run:
>     >     >
>     >     >     # getcert resubmit -f /var/lib/ipa/ra-agent.pem
>     >     >
>     >     >     That should renew the cert.
>     >     >
>     >     >     On the other masters I'd run the same command and that
>     may fix
>     >     things
>     >     >     there as well.
>     >     >
>     >     >     rob
>     >     >
>     >     >     > I tried this command:
>     >     >     > openssl x509 -text -in /var/lib/ipa/ra-agent.pem
>     >     >     >
>     >     >     > and it shows the following:
>     >     >     > Certificate:
>     >     >     >     Data:
>     >     >     >         Version: 3 (0x2)
>     >     >     >         Serial Number: 28 (0x1c)
>     >     >     >     Signature Algorithm: sha256WithRSAEncryption
>     >     >     >         Issuer: O=CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE> <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE>
>     >     >     <http://CORP.MYDOMAIN.DE>,
>     >     >     > CN=Certificate Authority
>     >     >     >         Validity
>     >     >     >             Not Before: Oct 29 10:39:47 2019 GMT
>     >     >     >             Not After : Oct 29 09:39:47 2021 GMT
>     >     >     >         Subject: O=CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE> <http://CORP.MYDOMAIN.DE>, CN=dmud
>     >     >     >         Subject Public Key Info:
>     >     >     >             Public Key Algorithm: rsaEncryption
>     >     >     >                 Public-Key: (2048 bit)
>     >     >     >                 Modulus:
>     >     >     >                    
>     >     00:ba:09:81:99:9b:17:99:07:5a:10:28:c8:7a:03:
>     >     >     > ...
>     >     >     >                    
>     >     18:db:02:ce:b4:66:ce:5a:e9:12:af:d3:da:bf:f7:
>     >     >     >                     66:5f
>     >     >     >                 Exponent: 65537 (0x10001)
>     >     >     >         X509v3 extensions:
>     >     >     >             X509v3 Authority Key Identifier:
>     >     >     >                 keyid:D2:...70:BF
>     >     >     >
>     >     >     >             X509v3 Subject Key Identifier:
>     >     >     >                 DE:...:51:0A
>     >     >     >             X509v3 Subject Alternative Name:
>     >     >     >                 email:dmud@corp.mydomain.de
>     <mailto:email%3Admud@corp.mydomain.de>
>     >     <mailto:email%3Admud@corp.mydomain.de
>     <mailto:email%253Admud@corp.mydomain.de>>
>     >     >     <mailto:email%3Admud@corp.mydomain.de
>     <mailto:email%253Admud@corp.mydomain.de>
>     >     <mailto:email%253Admud@corp.mydomain.de
>     <mailto:email%25253Admud@corp.mydomain.de>>>
>     >     >     > <mailto:email%3Admud@corp.mydomain.de
>     <mailto:email%253Admud@corp.mydomain.de>
>     >     <mailto:email%253Admud@corp.mydomain.de
>     <mailto:email%25253Admud@corp.mydomain.de>>
>     >     >     <mailto:email%253Admud@corp.mydomain.de
>     <mailto:email%25253Admud@corp.mydomain.de>
>     >     <mailto:email%25253Admud@corp.mydomain.de
>     <mailto:email%2525253Admud@corp.mydomain.de>>>>
>     >     >     >             Authority Information Access:
>     >     >     >                 OCSP -
>     >     URI:http://ipa-ca.corp.mydomain.de/ca/ocsp
>     >     >     >
>     >     >     >
>     >     >     > I did nothing to /var/lib/ipa/ra-agent.pem yet.
>     >     >     >
>     >     >     >
>     >     >     > чт, 21 нояб. 2019 г. в 16:54, Rob Crittenden
>     >     <rcritten@redhat.com <mailto:rcritten@redhat.com>
>     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
>     >     >     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
>     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
>     >     >     > <mailto:rcritten@redhat.com
>     <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
>     <mailto:rcritten@redhat.com>>
>     >     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
>     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>>:
>     >     >     >
>     >     >     >     Dmitri Moudraninets wrote:
>     >     >     >     > Hi Rob,
>     >     >     >     >
>     >     >     >     > Yes both masters are failing the same way. Output
>     >     of openssl
>     >     >     x509
>     >     >     >     -noout
>     >     >     >     > -modulus -in /var/lib/ipa/ra-agent.pem is the
>     same on both
>     >     >     masters.
>     >     >     >     > Output of openssl rsa -noout -modulus -in
>     >     >     /var/lib/ipa/ra-agent.key is
>     >     >     >     > also the same on both masters. But the output of
>     the first
>     >     >     command is
>     >     >     >     > not the same as the output of the second command.
>     >     >     >     >
>     >     >     >     > I can't remember that I troubleshoot any other
>     >     problems but we
>     >     >     >     tried to
>     >     >     >     > generate some personal certificates for some users.
>     >     Also we
>     >     >     tried to
>     >     >     >     > generate certificates with key files for some of our
>     >     internal
>     >     >     >     services.
>     >     >     >     > We did that for the first time and it worked at the
>     >     end. Also I
>     >     >     >     changed
>     >     >     >     > the admin password not so long ago.
>     >     >     >     >
>     >     >     >     >
>     >     >     >     > Below you can find the output of the requested
>     commands:
>     >     >     >     >
>     >     >     >     >
>     >     >     >     > [root@second_master ~]# getcert list -f
>     >     >     /var/lib/ipa/ra-agent.pem
>     >     >     >     > Number of certificates and requests being
>     tracked: 9.
>     >     >     >     > Request ID '20180912151730':
>     >     >     >     > status: MONITORING
>     >     >     >     > stuck: no
>     >     >     >     > key pair storage:
>     >     type=FILE,location='/var/lib/ipa/ra-agent.key'
>     >     >     >     > certificate:
>     >     type=FILE,location='/var/lib/ipa/ra-agent.pem'
>     >     >     >     > CA: dogtag-ipa-ca-renew-agent
>     >     >     >     > issuer: CN=Certificate
>     Authority,O=CORP.MYDOMAIN.DE <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE>
>     >     >     <http://CORP.MYDOMAIN.DE>
>     >     >     >     <http://CORP.MYDOMAIN.DE>
>     >     >     >     > <http://CORP.MYDOMAIN.DE>
>     >     >     >     > subject: CN=dmud,O=CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE>
>     >     >     <http://CORP.MYDOMAIN.DE> <http://CORP.MYDOMAIN.DE>
>     >     >     >     <http://CORP.MYDOMAIN.DE>
>     >     >     >     > *<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< I see a
>     username here.
>     >     >     Does it have
>     >     >     >     > to be like that?*
>     >     >     >     > expires: 2021-10-29 09:39:47 UTC
>     >     >     >     > email: dmud@corp.mydomain.de
>     <mailto:dmud@corp.mydomain.de>
>     >     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>>
>     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>
>     >     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>>>
>     >     >     <mailto:dmud@corp.mydomain.de
>     <mailto:dmud@corp.mydomain.de> <mailto:dmud@corp.mydomain.de
>     <mailto:dmud@corp.mydomain.de>>
>     >     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>
>     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>>>>
>     >     >     >     <mailto:dmud@corp.mydomain.de
>     <mailto:dmud@corp.mydomain.de>
>     >     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>>
>     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>
>     >     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>>>
>     >     >     <mailto:dmud@corp.mydomain.de
>     <mailto:dmud@corp.mydomain.de> <mailto:dmud@corp.mydomain.de
>     <mailto:dmud@corp.mydomain.de>>
>     >     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>
>     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>>>>>
>     >     >     >     > key usage:
>     >     >     >   
>     >      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>     >     >     >     > eku: id-kp-serverAuth,id-kp-clientAuth
>     >     >     >     > pre-save command:
>     >     /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>     >     >     >     > post-save command:
>     >     /usr/libexec/ipa/certmonger/renew_ra_cert
>     >     >     >     > track: yes
>     >     >     >     > auto-renew: yes
>     >     >     >
>     >     >     >     Right, someone overwrote the RA agent certificate.
>     >     >     >
>     >     >     >     Look to see if the user entry in the CA has the
>     right cert:
>     >     >     >
>     >     >     >     $ ldapsearch -LLL -o ldif-wrap=no -x -D 'cn=directory
>     >     manager'
>     >     >     -W -b
>     >     >     >     uid=ipara,ou=People,o=ipaca usercertificate
>     >     >     >
>     >     >     >     Put the base64 value of the usercertificate
>     attribute into a
>     >     >     file and
>     >     >     >     add a prefix/suffix around it:
>     >     >     >
>     >     >     >     -----BEGIN CERTIFICATE-----
>     >     >     >     MII....blah=
>     >     >     >     -----END CERTIFICATE-----
>     >     >     >
>     >     >     >     $ openssl x509 -text -in /path/to/file
>     >     >     >
>     >     >     >     If the Subject is O = CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE>
>     >     >     <http://CORP.MYDOMAIN.DE> <http://CORP.MYDOMAIN.DE>, CN
>     >     >     >     = IPA RA then that's a good
>     >     >     >     start. Also look at the expires date to be sure it is
>     >     still valid.
>     >     >     >
>     >     >     >     Assuming that is ok then re-run the openssl
>     modulus commands
>     >     >     to ensure
>     >     >     >     they are the same.
>     >     >     >
>     >     >     >     Assuming that too is ok then you have the proper,
>     valid RA
>     >     >     agent cert.
>     >     >     >     In that case I'd move the current file out of the
>     way, who
>     >     >     knows what it
>     >     >     >     is, then run:
>     >     >     >
>     >     >     >     # openssl x509 -in /path/to/file -out
>     >     >     /var/lib/ipa/ra-agent.pem (just to
>     >     >     >     properly format the agent cert)
>     >     >     >     # chown root:ipaapi /var/lib/ipa/ra-agent.pem
>     >     >     >     # chmod 0440 /var/lib/ipa/ra-agent.pem
>     >     >     >     # restorecon /var/lib/ipa/ra-agent.pem
>     >     >     >
>     >     >     >     Then try something like: ipa cert-show 1
>     >     >     >
>     >     >     >     This will exercise the RA agent cert and as long
>     as you
>     >     don't
>     >     >     get an
>     >     >     >     error back things are working again.
>     >     >     >
>     >     >     >     The cert is common among all masters so you can
>     copy the
>     >     file
>     >     >     to your
>     >     >     >     other master(s), ensuring proper ownership,
>     permissions and
>     >     >     SELinux
>     >     >     >     context.
>     >     >     >
>     >     >     >     rob
>     >     >     >
>     >     >     >
>     >     >     >
>     >     >     > --
>     >     >     > WBR
>     >     >     > Dmitry
>     >     >
>     >     >
>     >     >
>     >     > --
>     >     > With best regards/Mit freundlichen Grüßen
>     >     >
>     >     > Moudraninets Dmitry, RHCSA
>     >     > http://www.linkedin.com/in/moudraninets
>     >     > http://www.xing.com/profile/Dmitry_Mudraninets
>     >
>     >
>     >
>     > --
>     > With best regards/Mit freundlichen Grüßen
>     >
>     > Moudraninets Dmitry, RHCSA
>     > http://www.linkedin.com/in/moudraninets
>     > http://www.xing.com/profile/Dmitry_Mudraninets
>
>
>
> --
> With best regards/Mit freundlichen Grüßen
>
> Moudraninets Dmitry, RHCSA
> http://www.linkedin.com/in/moudraninets
> http://www.xing.com/profile/Dmitry_Mudraninets

--
With best regards/Mit freundlichen Grüßen

Moudraninets Dmitry, RHCSA
http://www.linkedin.com/in/moudraninets
http://www.xing.com/profile/Dmitry_Mudraninets