This is my first post, so I hope I don’t add too much details to my questions, but we spent so much time on that issue that I feel like I need to provide as much detail as possible
😉
We need your help !!!
After weeks of reading and attempt to fix our certificates, we can't find what is wrong.
Not sure if that is a freeIPA issue or just conflict between ipa-tomcat and openssl on same server. After CA and other certificate renewal, we are having bunch of issues.
We used to be able to install client and create replica, but in our attempt to create a new CA replica, and after upgrading to most recent versions and renewing certificates, including CA certificates,
we still truggle to get certificates to be trusted.
Both LDAP and HTTPd certificates don't renew with same error in certmonger: "ipa-getcert resubmit -i 20180228054516"
status: CA_UNREACHABLE
ca-error: Server at
https://ds01.EXAMPLE.com/ipa/xml failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://ds01.EXAMPLE.com:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1783)).
We went through the great tutorial from Flo, trying to debug this.
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues-with-freeipa/
All certificates looks fine and current, even both Ldapd and HTTPd. We added the ipaCert in the NSS DB as it was gone after upgrade to IPAv4.5, but that did not fix the issue.
[root@ds01 ~]# certutil -K -d /etc/httpd/alias/ -f /etc/httpd/alias/pwdfile.txt
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa ##key1## ipaCert
< 1> rsa ##key2## NSS Certificate DB:Server-Cert
Certificate in "certutil -L -d /etc/httpd/alias/ -n ipaCert -a" is different than ca.crt, but same as
dn: uid=ipara,ou=people,o=ipaca
ca.crt is registered in the LDAP at:
dn: cn=caSigningCert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=EXAMPLE,dc=com
Only thing that we can found with expired certificates is entry:
[root@ds01 ~]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=admin,ou=people,o=ipaca
That shows one userPassword, and 2 different userCertificate that are expired several year ago with description:
description: 2;6;CN=Certificate Authority,O=EXAMPLE.COM;CN=ipa-ca-agent,O=EXAMPLE.COM
description: 2;268304414;CN=Certificate Authority,O=EXAMPLE.COM;E=admin@EXAMPLE.com,CN=CS Administrator,UID=admin,OU=ca,O=EXAMPLE.COM,C=US
Not sure if this is an old V3 LDAP entry, not used in V4 or if that is the culprit and how to fix it??
We tried 'SSL_DIR=/etc/httpd/alias/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt
https://`hostname`:8443/ca/agent/ca/profileReview'
recommended by https://rcritten.wordpress.com/
But still same trust issue.
Our issue seems similar than the one discribed in:
So we tried to reset the trust flag, but that did not fix the issue for us.
Looking at the error_log we have:
[:error] [pid 14205] ipa: INFO: [jsonserver_session]
admin@EXAMPLE.COM: ca_find(None, version=u'2.228'): SUCCESS
[:error] [pid 14205] ipa: DEBUG: Destroyed connection context.ldap2_94559091634256
[:warn] [pid 14208] [client 192.168.10.217:63438] failed to set perms (3140) on file (/var/run/ipa/ccaches/admin@EXAMPLE.COM)!, referer:
https://ds01.EXAMPLE.com/ipa/ui/
We have both openldap and pki-tomecat on that master, not sure if that is what is creating issue since renewal of CA or if CRL used to be another server.
Any Idea of what could be wrong here?
Answer is probably as easy as adding one of the certificate to the NSS trust database, but not sure which one. :-(
###################
Here is what we get when installing client:
[root@ds12 ~]# ipa-client-install
Skip ds12.EXAMPLE.com: LDAP server is not responding, unable to verify if this is an IPA server
Discovery was successful!
Client hostname: ds12.EXAMPLE.com
Realm: EXAMPLE.COM
DNS Domain: EXAMPLE.com
IPA Server: ds01.EXAMPLE.com
BaseDN: dc=EXAMPLE,dc=com
Continue to configure the system with these values? [no]: yes
Synchronizing time with KDC...
Attempting to sync time using ntpd. Will timeout after 15 seconds
User authorized to enroll computers: admin
Password for admin@EXAMPLE.COM:
Successfully retrieved CA cert
Subject: CN=Certificate Authority,O=EXAMPLE.COM
Issuer: CN=Certificate Authority,O=EXAMPLE.COM
Valid From: 2014-08-03 19:28:18
Valid Until: 2034-08-03 19:28:18
Joining realm failed: libcurl failed to execute the HTTP POST transaction, explaining: Peer's certificate issuer has been marked as not trusted by the user.
[root@ds01 alias]# ipa config-show
Certificate Subject base: O=EXAMPLE.COM
IPA masters: ds01.EXAMPLE.com, ds02.EXAMPLE.com, ds03.EXAMPLE.com
IPA CA servers: ds01.EXAMPLE.com, ds03.EXAMPLE.com
IPA NTP servers: ds01.EXAMPLE.com, ds02.EXAMPLE.com
IPA CA renewal master: ds01.EXAMPLE.com
[root@ds01 ~]# ipa server-role-find --role "AD trust controller"
----------------------
3 server roles matched
----------------------
Server name: ds01.EXAMPLE.com
Role name: AD trust controller
Role status: absent
Server name: ds02.EXAMPLE.com
Role name: AD trust controller
Role status: absent
Server name: ds03.EXAMPLE.com
Role name: AD trust controller
Role status: absent
----------------------------
Number of entries returned 3
----------------------------
[root@ds01 ~]# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20180228053337':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
subject: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
expires: 2019-03-07 06:24:12 UTC
principal name:
krbtgt/EXAMPLE.COM@EXAMPLE.COM
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20180228054506':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=CA Audit,O=EXAMPLE.COM
expires: 2020-02-25 04:27:49 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180228054507':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=OCSP Subsystem,O=EXAMPLE.COM
expires: 2020-02-25 04:28:38 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180228054508':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=CA Subsystem,O=EXAMPLE.COM
expires: 2020-02-25 04:31:47 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180228054509':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=Certificate Authority,O=EXAMPLE.COM
expires: 2038-03-07 03:47:46 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180228054510':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=IPA RA,O=EXAMPLE.COM
expires: 2018-06-15 23:15:23 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20180228054511':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
expires: 2018-12-16 21:02:44 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180228054512':
status: CA_UNREACHABLE
ca-error: Server at
https://ds01.EXAMPLE.com/ipa/xml failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://ds01.EXAMPLE.com:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1783)).
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
expires: 2020-03-07 08:49:36 UTC
principal name:
ldap/ds01.EXAMPLE.com@EXAMPLE.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM
track: yes
auto-renew: yes
Request ID '20180228054516':
status: CA_UNREACHABLE
ca-error: Server at
https://ds01.EXAMPLE.com/ipa/xml failed request, will retry: 907 (RPC failed at server. cannot connect to 'https://ds01.EXAMPLE.com:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1783)).
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=EXAMPLE.COM
subject: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
expires: 2020-03-07 08:49:51 UTC
principal name:
HTTP/ds01.EXAMPLE.com@EXAMPLE.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
[root@ds01 ~]# certutil -L -d /etc/httpd/alias/ -n ipaCert
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1342111857 (0x4fff0071)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: "CN=Certificate Authority,O=EXAMPLE.COM"
Validity:
Not Before: Sat Jun 25 23:15:23 2016
Not After : Fri Jun 15 23:15:23 2018
Subject: "CN=IPA RA,O=EXAMPLE.COM"
[..]
Location:
URI: "http://ds01.EXAMPLE.com:80/ca/ocsp"
[..]
Mozilla-CA-Policy: false (attribute missing)
Certificate Trust Flags:
SSL Flags:
Valid CA
Trusted CA
User
Trusted Client CA
Email Flags:
Valid CA
Trusted CA
User
Object Signing Flags:
Valid CA
Trusted CA
User
###################################
[root@ds01 ~]# SSL_DIR=/etc/httpd/alias/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt
https://`hostname`:8443/ca/agent/ca/profileReview
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to ds01.EXAMPLE.com port 8443 (#0)
* Trying 192.168.10.146...
* Connected to ds01.EXAMPLE.com (192.168.10.146) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/httpd/alias/
* CAfile: /etc/ipa/ca.crt
CApath: none
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Server certificate:
* subject: CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
* start date: Dec 26 21:02:44 2016 GMT
* expire date: Dec 16 21:02:44 2018 GMT
* common name: ds01.EXAMPLE.com
* issuer: CN=Certificate Authority,O=EXAMPLE.COM
* NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
* Peer's certificate issuer has been marked as not trusted by the user.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Closing connection 0
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here:
http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Steph