This is my first post, so I hope I don’t add too much details to my questions, but we spent so much time on that issue that I feel like I need to provide as much detail as possible 😉

 

We need your help !!!

 

After weeks of reading and attempt to fix our certificates, we can't find what is wrong.

 

Not sure if that is a freeIPA issue or just conflict between ipa-tomcat and openssl on same server. After CA and other certificate renewal, we are having bunch of issues.

 

We used to be able to install client and create replica, but in our attempt to create a new CA replica, and after upgrading to most recent versions and renewing certificates, including CA certificates,

we still truggle to get certificates to be trusted.

 

Both LDAP and HTTPd certificates don't renew with same error in certmonger: "ipa-getcert resubmit -i 20180228054516"

        status: CA_UNREACHABLE

        ca-error: Server at https://ds01.EXAMPLE.com/ipa/xml failed request, will retry: 907 (RPC failed at server.  cannot connect to 'https://ds01.EXAMPLE.com:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1783)).

 

We went through the great tutorial from Flo, trying to debug this.

https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/

 

All certificates looks fine and current, even both Ldapd and HTTPd. We added the ipaCert in the NSS DB as it was gone after upgrade to IPAv4.5, but that did not fix the issue.

 

[root@ds01 ~]# certutil -K -d /etc/httpd/alias/ -f /etc/httpd/alias/pwdfile.txt

certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"

< 0> rsa      ##key1##   ipaCert

< 1> rsa      ##key2##   NSS Certificate DB:Server-Cert

 

Certificate in "certutil -L -d /etc/httpd/alias/ -n ipaCert -a" is different than ca.crt, but same as

dn: uid=ipara,ou=people,o=ipaca

 

ca.crt is registered in the LDAP at:

dn: cn=caSigningCert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=EXAMPLE,dc=com

 

Only thing that we can found with expired certificates is entry:

[root@ds01 ~]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=admin,ou=people,o=ipaca

That shows one userPassword, and 2 different userCertificate that are expired several year ago with description:

description: 2;6;CN=Certificate Authority,O=EXAMPLE.COM;CN=ipa-ca-agent,O=EXAMPLE.COM

description: 2;268304414;CN=Certificate Authority,O=EXAMPLE.COM;E=admin@EXAMPLE.com,CN=CS Administrator,UID=admin,OU=ca,O=EXAMPLE.COM,C=US

 

Not sure if this is an old V3 LDAP entry, not used in V4 or if that is the culprit and how to fix it??

 

We tried 'SSL_DIR=/etc/httpd/alias/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt https://`hostname`:8443/ca/agent/ca/profileReview'

recommended by https://rcritten.wordpress.com/

But still same trust issue.

 

Our issue seems similar than the one discribed in:

https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/XSMWWPJU2VRUIGE6SRAHYAJF7BYBCNOE/

So we tried to reset the trust flag, but that did not fix the issue for us.

 

Looking at the error_log we have:

[:error] [pid 14205] ipa: INFO: [jsonserver_session] admin@EXAMPLE.COM: ca_find(None, version=u'2.228'): SUCCESS

[:error] [pid 14205] ipa: DEBUG: Destroyed connection context.ldap2_94559091634256

[:warn] [pid 14208] [client 192.168.10.217:63438] failed to set perms (3140) on file (/var/run/ipa/ccaches/admin@EXAMPLE.COM)!, referer: https://ds01.EXAMPLE.com/ipa/ui/

 

We have both openldap and pki-tomecat on that master, not sure if that is what is creating issue since renewal of CA or if CRL used to be another server.

 

Any Idea of what could be wrong here?

 

Answer is probably as easy as adding one of the certificate to the NSS trust database, but not sure which one. :-(

 

 

###################

Here is what we get when installing client:

 

[root@ds12 ~]# ipa-client-install

Skip ds12.EXAMPLE.com: LDAP server is not responding, unable to verify if this is an IPA server

Discovery was successful!

Client hostname: ds12.EXAMPLE.com

Realm: EXAMPLE.COM

DNS Domain: EXAMPLE.com

IPA Server: ds01.EXAMPLE.com

BaseDN: dc=EXAMPLE,dc=com

 

Continue to configure the system with these values? [no]: yes

Synchronizing time with KDC...

Attempting to sync time using ntpd.  Will timeout after 15 seconds

User authorized to enroll computers: admin

Password for admin@EXAMPLE.COM:

Successfully retrieved CA cert

    Subject:     CN=Certificate Authority,O=EXAMPLE.COM

    Issuer:      CN=Certificate Authority,O=EXAMPLE.COM

    Valid From:  2014-08-03 19:28:18

    Valid Until: 2034-08-03 19:28:18

 

Joining realm failed: libcurl failed to execute the HTTP POST transaction, explaining:  Peer's certificate issuer has been marked as not trusted by the user.

 

 

[root@ds01 alias]# ipa config-show

  Certificate Subject base: O=EXAMPLE.COM

  IPA masters: ds01.EXAMPLE.com, ds02.EXAMPLE.com, ds03.EXAMPLE.com

  IPA CA servers: ds01.EXAMPLE.com, ds03.EXAMPLE.com

  IPA NTP servers: ds01.EXAMPLE.com, ds02.EXAMPLE.com

  IPA CA renewal master: ds01.EXAMPLE.com

 

[root@ds01 ~]# ipa server-role-find --role "AD trust controller"

----------------------

3 server roles matched

----------------------

  Server name: ds01.EXAMPLE.com

  Role name: AD trust controller

  Role status: absent

 

  Server name: ds02.EXAMPLE.com

  Role name: AD trust controller

  Role status: absent

 

  Server name: ds03.EXAMPLE.com

  Role name: AD trust controller

  Role status: absent

----------------------------

Number of entries returned 3

----------------------------

 

[root@ds01 ~]# getcert list

Number of certificates and requests being tracked: 9.

Request ID '20180228053337':

        status: MONITORING

        stuck: no

        key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'

        certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'

        CA: SelfSign

        issuer: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM

        subject: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM

        expires: 2019-03-07 06:24:12 UTC

        principal name: krbtgt/EXAMPLE.COM@EXAMPLE.COM

        certificate template/profile: KDCs_PKINIT_Certs

        pre-save command:

        post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert

        track: yes

        auto-renew: yes

Request ID '20180228054506':

        status: MONITORING

        stuck: no

        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set

        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'

        CA: dogtag-ipa-ca-renew-agent

        issuer: CN=Certificate Authority,O=EXAMPLE.COM

        subject: CN=CA Audit,O=EXAMPLE.COM

        expires: 2020-02-25 04:27:49 UTC

        key usage: digitalSignature,nonRepudiation

        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"

        track: yes

        auto-renew: yes

Request ID '20180228054507':

        status: MONITORING

        stuck: no

        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set

        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'

        CA: dogtag-ipa-ca-renew-agent

        issuer: CN=Certificate Authority,O=EXAMPLE.COM

        subject: CN=OCSP Subsystem,O=EXAMPLE.COM

        expires: 2020-02-25 04:28:38 UTC

        eku: id-kp-OCSPSigning

        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"

        track: yes

        auto-renew: yes

Request ID '20180228054508':

        status: MONITORING

        stuck: no

        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set

        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'

        CA: dogtag-ipa-ca-renew-agent

        issuer: CN=Certificate Authority,O=EXAMPLE.COM

        subject: CN=CA Subsystem,O=EXAMPLE.COM

        expires: 2020-02-25 04:31:47 UTC

        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

        eku: id-kp-serverAuth,id-kp-clientAuth

        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"

        track: yes

        auto-renew: yes

Request ID '20180228054509':

        status: MONITORING

        stuck: no

        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set

        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'

        CA: dogtag-ipa-ca-renew-agent

        issuer: CN=Certificate Authority,O=EXAMPLE.COM

        subject: CN=Certificate Authority,O=EXAMPLE.COM

        expires: 2038-03-07 03:47:46 UTC

        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign

        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"

        track: yes

        auto-renew: yes

Request ID '20180228054510':

        status: MONITORING

        stuck: no

        key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'

        certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'

        CA: dogtag-ipa-ca-renew-agent

        issuer: CN=Certificate Authority,O=EXAMPLE.COM

        subject: CN=IPA RA,O=EXAMPLE.COM

        expires: 2018-06-15 23:15:23 UTC

        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

        eku: id-kp-serverAuth,id-kp-clientAuth

        pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre

        post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert

        track: yes

        auto-renew: yes

Request ID '20180228054511':

        status: MONITORING

        stuck: no

        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set

        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'

        CA: dogtag-ipa-ca-renew-agent

        issuer: CN=Certificate Authority,O=EXAMPLE.COM

        subject: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM

        expires: 2018-12-16 21:02:44 UTC

        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

        eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection

        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad

        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"

        track: yes

        auto-renew: yes

Request ID '20180228054512':

        status: CA_UNREACHABLE

        ca-error: Server at https://ds01.EXAMPLE.com/ipa/xml failed request, will retry: 907 (RPC failed at server.  cannot connect to 'https://ds01.EXAMPLE.com:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1783)).

        stuck: no

        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'

        certificate: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB'

        CA: IPA

        issuer: CN=Certificate Authority,O=EXAMPLE.COM

        subject: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM

        expires: 2020-03-07 08:49:36 UTC

        principal name: ldap/ds01.EXAMPLE.com@EXAMPLE.COM

        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

        eku: id-kp-serverAuth,id-kp-clientAuth

        pre-save command:

        post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM

        track: yes

        auto-renew: yes

Request ID '20180228054516':

        status: CA_UNREACHABLE

        ca-error: Server at https://ds01.EXAMPLE.com/ipa/xml failed request, will retry: 907 (RPC failed at server.  cannot connect to 'https://ds01.EXAMPLE.com:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1783)).

        stuck: no

        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'

        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'

        CA: IPA

        issuer: CN=Certificate Authority,O=EXAMPLE.COM

        subject: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM

        expires: 2020-03-07 08:49:51 UTC

        principal name: HTTP/ds01.EXAMPLE.com@EXAMPLE.COM

        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment

        eku: id-kp-serverAuth,id-kp-clientAuth

        pre-save command:

        post-save command: /usr/libexec/ipa/certmonger/restart_httpd

        track: yes

        auto-renew: yes

 

[root@ds01 ~]# certutil -L -d /etc/httpd/alias/ -n ipaCert

Certificate:

    Data:

        Version: 3 (0x2)

        Serial Number: 1342111857 (0x4fff0071)

        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption

        Issuer: "CN=Certificate Authority,O=EXAMPLE.COM"

        Validity:

            Not Before: Sat Jun 25 23:15:23 2016

            Not After : Fri Jun 15 23:15:23 2018

        Subject: "CN=IPA RA,O=EXAMPLE.COM"

[..]

 

            Location:

                URI: "http://ds01.EXAMPLE.com:80/ca/ocsp"

[..]

    Mozilla-CA-Policy: false (attribute missing)

    Certificate Trust Flags:

        SSL Flags:

            Valid CA

            Trusted CA

            User

            Trusted Client CA

        Email Flags:

            Valid CA

            Trusted CA

            User

        Object Signing Flags:

            Valid CA

            Trusted CA

            User

 

###################################

 

[root@ds01 ~]# SSL_DIR=/etc/httpd/alias/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt https://`hostname`:8443/ca/agent/ca/profileReview

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to ds01.EXAMPLE.com port 8443 (#0)

*   Trying 192.168.10.146...

* Connected to ds01.EXAMPLE.com (192.168.10.146) port 8443 (#0)

* Initializing NSS with certpath: sql:/etc/httpd/alias/

*   CAfile: /etc/ipa/ca.crt

  CApath: none

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Server certificate:

*       subject: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM

*       start date: Dec 26 21:02:44 2016 GMT

*       expire date: Dec 16 21:02:44 2018 GMT

*       common name: ds01.EXAMPLE.com

*       issuer: CN=Certificate Authority,O=EXAMPLE.COM

* NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)

* Peer's certificate issuer has been marked as not trusted by the user.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

* Closing connection 0

curl: (60) Peer's certificate issuer has been marked as not trusted by the user.

More details here: http://curl.haxx.se/docs/sslcerts.html

 

curl performs SSL certificate verification by default, using a "bundle"

of Certificate Authority (CA) public keys (CA certs). If the default

bundle file isn't adequate, you can specify an alternate file

using the --cacert option.

If this HTTPS server uses a certificate signed by a CA represented in

the bundle, the certificate verification probably failed due to a

problem with the certificate (it might be expired, or the name might

not match the domain name in the URL).

If you'd like to turn off curl's verification of the certificate, use

the -k (or --insecure) option.

 

 

 

Steph