Hi Rob,


It seems that it does not help. I found a backup which was made via ipa-backup this summer. Can I use it somehow for recovery? We did nothing to certificates since that time. We only added users/groups/servers.

Current situation:
I can't update certificates. getcert list shows multiple certificates with CA_UNREACHABLE status:
status: CA_UNREACHABLE
        ca-error: Error 35 connecting to https://freeipa.corp.mydomain.de:8443/ca/agent/ca/profileReview: SSL connect error.


pki-tomcatd is not starting:
[root@freeipa ipa]# ipactl start --ignore-service-failures
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Forced start, ignoring pki-tomcatd Service, continuing normal operation
Starting smb Service
Starting winbind Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

пн, 25 нояб. 2019 г. в 15:47, Rob Crittenden <rcritten@redhat.com>:
Dmitri Moudraninets wrote:
> Hi Rob,
>
> I recovered the key file. Restarted FreeIPA and certmonger. Now issue
> looks different:
> image.png
>
> Subjects disappeared. If I click on a certificate 29 I see this:
> cannot connect to
> 'https://freeipa.corp.mydomain.de:443/ca/agent/ca/displayBySerial':
> [Errno 13] Permission denied

Set the same ownership/permissions on the key as you did the cert and
run restorecon on it.

rob

>
> пн, 25 нояб. 2019 г. в 13:58, Rob Crittenden <rcritten@redhat.com
> <mailto:rcritten@redhat.com>>:
>
>     Dmitri Moudraninets wrote:
>     > Hi Rob,
>     >
>     >
>     >
>     > I did the following:
>     > I removed original ra-agent.pem and ra-agent key
>     > and
>     > openssl x509 -in /root/debug.cert -out /var/lib/ipa/ra-agent.pem
>     > chown root:ipaapi /var/lib/ipa/ra-agent.pem
>     > chmod 0440 /var/lib/ipa/ra-agent.pem
>     > restorecon /var/lib/ipa/ra-agent.pem
>
>     You removed the key!? I sure hope you have a backup of it.
>
>     Put it back and I think that will resolve things.
>
>     >
>     > Successfully restarted FreeIPA:
>     > Directory Service: RUNNING
>     > krb5kdc Service: RUNNING
>     > kadmin Service: RUNNING
>     > named Service: RUNNING
>     > httpd Service: RUNNING
>     > ipa-custodia Service: RUNNING
>     > ntpd Service: RUNNING
>     > pki-tomcatd Service: RUNNING
>     > smb Service: RUNNING
>     > winbind Service: RUNNING
>     > ipa-otpd Service: RUNNING
>     > ipa-dnskeysyncd Service: RUNNING
>     > ipa: INFO: The ipactl command was successful
>
>     The agent cert is not required for the CA to operate.
>
>     > Now GUI shows different error:
>     > cannot connect to
>     > 'https://freeipa.corp.mydomain.de:443/ca/agent/ca/displayBySerial':
>     > [Errno 2] No such file or directory
>     >
>     >
>     > [root@freeipa ~]# getcert list -f /var/lib/ipa/ra-agent.pem
>     > Number of certificates and requests being tracked: 16.
>     > Request ID '20180912151611':
>     > status: NEED_CSR
>     > stuck: no
>     > key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>     > certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>     > CA: dogtag-ipa-ca-renew-agent
>     > issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE>
>     > <http://CORP.MYDOMAIN.DE>
>     > subject: CN=IPA RA,O=CORP.MYDOMAIN.DE <http://CORP.MYDOMAIN.DE>
>     <http://CORP.MYDOMAIN.DE>
>     > expires: 2019-11-25 15:32:12 UTC
>     > key usage:
>     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>     > eku: id-kp-serverAuth,id-kp-clientAuth
>     > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>     > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>     > track: yes
>     > auto-renew: yes
>
>     This shows that the certificate has the right subject now which is good
>     but you removed its private key so it won't work.
>
>     rob
>
>     >
>     > сб, 23 нояб. 2019 г. в 20:26, Rob Crittenden <rcritten@redhat.com
>     <mailto:rcritten@redhat.com>
>     > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>:
>     >
>     >     Dmitri Moudraninets wrote:
>     >     > Hi Rob,
>     >     >
>     >     > ldapsearch -LLL -o ldif-wrap=no -x -D 'cn=directory manager' -W
>     >     > -b uid=ipara,ou=People,o=ipaca usercertificate
>     >     >
>     >     > shows me the following:
>     >     >
>     >     >         Issuer: O=CORP.MYDOMAIN.DE <http://CORP.MYDOMAIN.DE>
>     <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE>,
>     >     > CN=Certificate Authority
>     >     >         Validity
>     >     >             Not Before: Dec  5 15:32:12 2017 GMT
>     >     >             Not After : *Nov 25 15:32:12 2019* GMT
>     >     >
>     >     > It's going to expire on Monday. Can it be a problem?
>     >
>     >     You didn't provide the cert subject so I can't be sure this is
>     the right
>     >     cert. If it contains CN = IPA RA then it is.
>     >
>     >     And yes, it expires in two days. What you'd need to do is
>     restore it per
>     >     my previous instruction into /var/lib/ipa/ra-agent.pem on the
>     renewal
>     >     master (ipa config-show to see which one it is).
>     >
>     >     Then run:
>     >
>     >     # getcert resubmit -f /var/lib/ipa/ra-agent.pem
>     >
>     >     That should renew the cert.
>     >
>     >     On the other masters I'd run the same command and that may fix
>     things
>     >     there as well.
>     >
>     >     rob
>     >
>     >     > I tried this command:
>     >     > openssl x509 -text -in /var/lib/ipa/ra-agent.pem
>     >     >
>     >     > and it shows the following:
>     >     > Certificate:
>     >     >     Data:
>     >     >         Version: 3 (0x2)
>     >     >         Serial Number: 28 (0x1c)
>     >     >     Signature Algorithm: sha256WithRSAEncryption
>     >     >         Issuer: O=CORP.MYDOMAIN.DE <http://CORP.MYDOMAIN.DE>
>     <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE>,
>     >     > CN=Certificate Authority
>     >     >         Validity
>     >     >             Not Before: Oct 29 10:39:47 2019 GMT
>     >     >             Not After : Oct 29 09:39:47 2021 GMT
>     >     >         Subject: O=CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE> <http://CORP.MYDOMAIN.DE>, CN=dmud
>     >     >         Subject Public Key Info:
>     >     >             Public Key Algorithm: rsaEncryption
>     >     >                 Public-Key: (2048 bit)
>     >     >                 Modulus:
>     >     >                    
>     00:ba:09:81:99:9b:17:99:07:5a:10:28:c8:7a:03:
>     >     > ...
>     >     >                    
>     18:db:02:ce:b4:66:ce:5a:e9:12:af:d3:da:bf:f7:
>     >     >                     66:5f
>     >     >                 Exponent: 65537 (0x10001)
>     >     >         X509v3 extensions:
>     >     >             X509v3 Authority Key Identifier:
>     >     >                 keyid:D2:...70:BF
>     >     >
>     >     >             X509v3 Subject Key Identifier:
>     >     >                 DE:...:51:0A
>     >     >             X509v3 Subject Alternative Name:
>     >     >                 email:dmud@corp.mydomain.de
>     <mailto:email%3Admud@corp.mydomain.de>
>     >     <mailto:email%3Admud@corp.mydomain.de
>     <mailto:email%253Admud@corp.mydomain.de>>
>     >     > <mailto:email%3Admud@corp.mydomain.de
>     <mailto:email%253Admud@corp.mydomain.de>
>     >     <mailto:email%253Admud@corp.mydomain.de
>     <mailto:email%25253Admud@corp.mydomain.de>>>
>     >     >             Authority Information Access:
>     >     >                 OCSP -
>     URI:http://ipa-ca.corp.mydomain.de/ca/ocsp
>     >     >
>     >     >
>     >     > I did nothing to /var/lib/ipa/ra-agent.pem yet.
>     >     >
>     >     >
>     >     > чт, 21 нояб. 2019 г. в 16:54, Rob Crittenden
>     <rcritten@redhat.com <mailto:rcritten@redhat.com>
>     >     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
>     >     > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>
>     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>>:
>     >     >
>     >     >     Dmitri Moudraninets wrote:
>     >     >     > Hi Rob,
>     >     >     >
>     >     >     > Yes both masters are failing the same way. Output
>     of openssl
>     >     x509
>     >     >     -noout
>     >     >     > -modulus -in /var/lib/ipa/ra-agent.pem is the same on both
>     >     masters.
>     >     >     > Output of openssl rsa -noout -modulus -in
>     >     /var/lib/ipa/ra-agent.key is
>     >     >     > also the same on both masters. But the output of the first
>     >     command is
>     >     >     > not the same as the output of the second command.
>     >     >     >
>     >     >     > I can't remember that I troubleshoot any other
>     problems but we
>     >     >     tried to
>     >     >     > generate some personal certificates for some users.
>     Also we
>     >     tried to
>     >     >     > generate certificates with key files for some of our
>     internal
>     >     >     services.
>     >     >     > We did that for the first time and it worked at the
>     end. Also I
>     >     >     changed
>     >     >     > the admin password not so long ago.
>     >     >     >
>     >     >     >
>     >     >     > Below you can find the output of the requested commands:
>     >     >     >
>     >     >     >
>     >     >     > [root@second_master ~]# getcert list -f
>     >     /var/lib/ipa/ra-agent.pem
>     >     >     > Number of certificates and requests being tracked: 9.
>     >     >     > Request ID '20180912151730':
>     >     >     > status: MONITORING
>     >     >     > stuck: no
>     >     >     > key pair storage:
>     type=FILE,location='/var/lib/ipa/ra-agent.key'
>     >     >     > certificate:
>     type=FILE,location='/var/lib/ipa/ra-agent.pem'
>     >     >     > CA: dogtag-ipa-ca-renew-agent
>     >     >     > issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE>
>     >     >     <http://CORP.MYDOMAIN.DE>
>     >     >     > <http://CORP.MYDOMAIN.DE>
>     >     >     > subject: CN=dmud,O=CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE> <http://CORP.MYDOMAIN.DE>
>     >     >     <http://CORP.MYDOMAIN.DE>
>     >     >     > *<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< I see a username here.
>     >     Does it have
>     >     >     > to be like that?*
>     >     >     > expires: 2021-10-29 09:39:47 UTC
>     >     >     > email: dmud@corp.mydomain.de
>     <mailto:dmud@corp.mydomain.de> <mailto:dmud@corp.mydomain.de
>     <mailto:dmud@corp.mydomain.de>>
>     >     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>
>     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>>>
>     >     >     <mailto:dmud@corp.mydomain.de
>     <mailto:dmud@corp.mydomain.de> <mailto:dmud@corp.mydomain.de
>     <mailto:dmud@corp.mydomain.de>>
>     >     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>
>     <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>>>>
>     >     >     > key usage:
>     >     >   
>      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>     >     >     > eku: id-kp-serverAuth,id-kp-clientAuth
>     >     >     > pre-save command:
>     /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>     >     >     > post-save command:
>     /usr/libexec/ipa/certmonger/renew_ra_cert
>     >     >     > track: yes
>     >     >     > auto-renew: yes
>     >     >
>     >     >     Right, someone overwrote the RA agent certificate.
>     >     >
>     >     >     Look to see if the user entry in the CA has the right cert:
>     >     >
>     >     >     $ ldapsearch -LLL -o ldif-wrap=no -x -D 'cn=directory
>     manager'
>     >     -W -b
>     >     >     uid=ipara,ou=People,o=ipaca usercertificate
>     >     >
>     >     >     Put the base64 value of the usercertificate attribute into a
>     >     file and
>     >     >     add a prefix/suffix around it:
>     >     >
>     >     >     -----BEGIN CERTIFICATE-----
>     >     >     MII....blah=
>     >     >     -----END CERTIFICATE-----
>     >     >
>     >     >     $ openssl x509 -text -in /path/to/file
>     >     >
>     >     >     If the Subject is O = CORP.MYDOMAIN.DE
>     <http://CORP.MYDOMAIN.DE>
>     >     <http://CORP.MYDOMAIN.DE> <http://CORP.MYDOMAIN.DE>, CN
>     >     >     = IPA RA then that's a good
>     >     >     start. Also look at the expires date to be sure it is
>     still valid.
>     >     >
>     >     >     Assuming that is ok then re-run the openssl modulus commands
>     >     to ensure
>     >     >     they are the same.
>     >     >
>     >     >     Assuming that too is ok then you have the proper, valid RA
>     >     agent cert.
>     >     >     In that case I'd move the current file out of the way, who
>     >     knows what it
>     >     >     is, then run:
>     >     >
>     >     >     # openssl x509 -in /path/to/file -out
>     >     /var/lib/ipa/ra-agent.pem (just to
>     >     >     properly format the agent cert)
>     >     >     # chown root:ipaapi /var/lib/ipa/ra-agent.pem
>     >     >     # chmod 0440 /var/lib/ipa/ra-agent.pem
>     >     >     # restorecon /var/lib/ipa/ra-agent.pem
>     >     >
>     >     >     Then try something like: ipa cert-show 1
>     >     >
>     >     >     This will exercise the RA agent cert and as long as you
>     don't
>     >     get an
>     >     >     error back things are working again.
>     >     >
>     >     >     The cert is common among all masters so you can copy the
>     file
>     >     to your
>     >     >     other master(s), ensuring proper ownership, permissions and
>     >     SELinux
>     >     >     context.
>     >     >
>     >     >     rob
>     >     >
>     >     >
>     >     >
>     >     > --
>     >     > WBR
>     >     > Dmitry
>     >
>     >
>     >
>     > --
>     > With best regards/Mit freundlichen Grüßen
>     >
>     > Moudraninets Dmitry, RHCSA
>     > http://www.linkedin.com/in/moudraninets
>     > http://www.xing.com/profile/Dmitry_Mudraninets
>
>
>
> --
> With best regards/Mit freundlichen Grüßen
>
> Moudraninets Dmitry, RHCSA
> http://www.linkedin.com/in/moudraninets
> http://www.xing.com/profile/Dmitry_Mudraninets



--
With best regards/Mit freundlichen Grüßen

Moudraninets Dmitry, RHCSA
http://www.linkedin.com/in/moudraninets
http://www.xing.com/profile/Dmitry_Mudraninets