This is my first post, so I hope I don’t add too much details to my questions, but we
spent so much time on that issue that I feel like I need to provide as much detail as
possible 😉
We need your help !!!
After weeks of reading and attempt to fix our certificates, we can't find what is
wrong.
Not sure if that is a freeIPA issue or just conflict between ipa-tomcat and openssl on
same server. After CA and other certificate renewal, we are having bunch of issues.
We used to be able to install client and create replica, but in our attempt to create a
new CA replica, and after upgrading to most recent versions and renewing certificates,
including CA certificates,
we still truggle to get certificates to be trusted.
Both LDAP and HTTPd certificates don't renew with same error in certmonger:
"ipa-getcert resubmit -i 20180228054516"
status: CA_UNREACHABLE
ca-error: Server at
https://ds01.EXAMPLE.com/ipa/xml failed request, will retry:
907 (RPC failed at server. cannot connect to
'https://ds01.EXAMPLE.com:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE]
ssl handshake failure (_ssl.c:1783)).
We went through the great tutorial from Flo, trying to debug this.
https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issu...
All certificates looks fine and current, even both Ldapd and HTTPd. We added the ipaCert
in the NSS DB as it was gone after upgrade to IPAv4.5, but that did not fix the issue.
[root@ds01 ~]# certutil -K -d /etc/httpd/alias/ -f /etc/httpd/alias/pwdfile.txt
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key
and Certificate Services"
< 0> rsa ##key1## ipaCert
< 1> rsa ##key2## NSS Certificate DB:Server-Cert
Certificate in "certutil -L -d /etc/httpd/alias/ -n ipaCert -a" is different
than ca.crt, but same as
dn: uid=ipara,ou=people,o=ipaca
ca.crt is registered in the LDAP at:
dn: cn=caSigningCert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=EXAMPLE,dc=com
Only thing that we can found with expired certificates is entry:
[root@ds01 ~]# ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=admin,ou=people,o=ipaca
That shows one userPassword, and 2 different userCertificate that are expired several year
ago with description:
description: 2;6;CN=Certificate
Authority,O=EXAMPLE.COM;CN=ipa-ca-agent,O=EXAMPLE.COM
description: 2;268304414;CN=Certificate Authority,O=EXAMPLE.COM;E=admin(a)EXAMPLE.com,CN=CS
Administrator,UID=admin,OU=ca,O=EXAMPLE.COM,C=US
Not sure if this is an old V3 LDAP entry, not used in V4 or if that is the culprit and how
to fix it??
We tried 'SSL_DIR=/etc/httpd/alias/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt
https://`hostname`:8443/ca/agent/ca/profileReview<https://%60hostname%60:8443/ca/agent/ca/profileReview>'
recommended by
https://rcritten.wordpress.com/
But still same trust issue.
Our issue seems similar than the one discribed in:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
So we tried to reset the trust flag, but that did not fix the issue for us.
Looking at the error_log we have:
[:error] [pid 14205] ipa: INFO: [jsonserver_session]
admin@EXAMPLE.COM<mailto:admin@EXAMPLE.COM>: ca_find(None,
version=u'2.228'): SUCCESS
[:error] [pid 14205] ipa: DEBUG: Destroyed connection context.ldap2_94559091634256
[:warn] [pid 14208] [client 192.168.10.217:63438] failed to set perms (3140) on file
(/var/run/ipa/ccaches/admin@EXAMPLE.COM<mailto:/var/run/ipa/ccaches/admin@EXAMPLE.COM>)!,
referer:
https://ds01.EXAMPLE.com/ipa/ui/
We have both openldap and pki-tomecat on that master, not sure if that is what is creating
issue since renewal of CA or if CRL used to be another server.
Any Idea of what could be wrong here?
Answer is probably as easy as adding one of the certificate to the NSS trust database, but
not sure which one. :-(
###################
Here is what we get when installing client:
[root@ds12 ~]# ipa-client-install
Skip
ds12.EXAMPLE.com: LDAP server is not responding, unable to verify if this is an IPA
server
Discovery was successful!
Client hostname:
ds12.EXAMPLE.com
Realm:
EXAMPLE.COM
DNS Domain:
EXAMPLE.com
IPA Server:
ds01.EXAMPLE.com
BaseDN: dc=EXAMPLE,dc=com
Continue to configure the system with these values? [no]: yes
Synchronizing time with KDC...
Attempting to sync time using ntpd. Will timeout after 15 seconds
User authorized to enroll computers: admin
Password for admin@EXAMPLE.COM<mailto:admin@EXAMPLE.COM>:
Successfully retrieved CA cert
Subject: CN=Certificate
Authority,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Valid From: 2014-08-03 19:28:18
Valid Until: 2034-08-03 19:28:18
Joining realm failed: libcurl failed to execute the HTTP POST transaction, explaining:
Peer's certificate issuer has been marked as not trusted by the user.
[root@ds01 alias]# ipa config-show
Certificate Subject base:
O=EXAMPLE.COM
IPA masters:
ds01.EXAMPLE.com,
ds02.EXAMPLE.com,
ds03.EXAMPLE.com
IPA CA servers:
ds01.EXAMPLE.com,
ds03.EXAMPLE.com
IPA NTP servers:
ds01.EXAMPLE.com,
ds02.EXAMPLE.com
IPA CA renewal master:
ds01.EXAMPLE.com
[root@ds01 ~]# ipa server-role-find --role "AD trust controller"
----------------------
3 server roles matched
----------------------
Server name:
ds01.EXAMPLE.com
Role name: AD trust controller
Role status: absent
Server name:
ds02.EXAMPLE.com
Role name: AD trust controller
Role status: absent
Server name:
ds03.EXAMPLE.com
Role name: AD trust controller
Role status: absent
----------------------------
Number of entries returned 3
----------------------------
[root@ds01 ~]# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20180228053337':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: SelfSign
issuer:
CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
subject:
CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
expires: 2019-03-07 06:24:12 UTC
principal name:
krbtgt/EXAMPLE.COM@EXAMPLE.COM<mailto:krbtgt/EXAMPLE.COM@EXAMPLE.COM>
certificate template/profile: KDCs_PKINIT_Certs
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20180228054506':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate
Authority,O=EXAMPLE.COM
subject: CN=CA
Audit,O=EXAMPLE.COM
expires: 2020-02-25 04:27:49 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180228054507':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate
Authority,O=EXAMPLE.COM
subject: CN=OCSP
Subsystem,O=EXAMPLE.COM
expires: 2020-02-25 04:28:38 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180228054508':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate
Authority,O=EXAMPLE.COM
subject: CN=CA
Subsystem,O=EXAMPLE.COM
expires: 2020-02-25 04:31:47 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180228054509':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate
Authority,O=EXAMPLE.COM
subject: CN=Certificate
Authority,O=EXAMPLE.COM
expires: 2038-03-07 03:47:46 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180228054510':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate
Authority,O=EXAMPLE.COM
subject: CN=IPA
RA,O=EXAMPLE.COM
expires: 2018-06-15 23:15:23 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20180228054511':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate
Authority,O=EXAMPLE.COM
subject:
CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
expires: 2018-12-16 21:02:44 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert
cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180228054512':
status: CA_UNREACHABLE
ca-error: Server at
https://ds01.EXAMPLE.com/ipa/xml failed request, will retry:
907 (RPC failed at server. cannot connect to
'https://ds01.EXAMPLE.com:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE]
ssl handshake failure (_ssl.c:1783)).
stuck: no
key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/dirsrv/slapd-EXAMPLE-COM/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-EXAMPLE-COM',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate
Authority,O=EXAMPLE.COM
subject:
CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
expires: 2020-03-07 08:49:36 UTC
principal name:
ldap/ds01.EXAMPLE.com@EXAMPLE.COM<mailto:ldap/ds01.EXAMPLE.com@EXAMPLE.COM>
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-COM
track: yes
auto-renew: yes
Request ID '20180228054516':
status: CA_UNREACHABLE
ca-error: Server at
https://ds01.EXAMPLE.com/ipa/xml failed request, will retry:
907 (RPC failed at server. cannot connect to
'https://ds01.EXAMPLE.com:443/ca/rest/account/login': [SSL: SSL_HANDSHAKE_FAILURE]
ssl handshake failure (_ssl.c:1783)).
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate
Authority,O=EXAMPLE.COM
subject:
CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
expires: 2020-03-07 08:49:51 UTC
principal name:
HTTP/ds01.EXAMPLE.com@EXAMPLE.COM<mailto:HTTP/ds01.EXAMPLE.com@EXAMPLE.COM>
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
[root@ds01 ~]# certutil -L -d /etc/httpd/alias/ -n ipaCert
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1342111857 (0x4fff0071)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: "CN=Certificate Authority,O=EXAMPLE.COM"
Validity:
Not Before: Sat Jun 25 23:15:23 2016
Not After : Fri Jun 15 23:15:23 2018
Subject: "CN=IPA RA,O=EXAMPLE.COM"
[..]
Location:
URI: "http://ds01.EXAMPLE.com:80/ca/ocsp"
[..]
Mozilla-CA-Policy: false (attribute missing)
Certificate Trust Flags:
SSL Flags:
Valid CA
Trusted CA
User
Trusted Client CA
Email Flags:
Valid CA
Trusted CA
User
Object Signing Flags:
Valid CA
Trusted CA
User
###################################
[root@ds01 ~]# SSL_DIR=/etc/httpd/alias/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt
https://`hostname`:8443/ca/agent/ca/profileReview<https://%60hostname%60:8443/ca/agent/ca/profileReview>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to
connect() to
ds01.EXAMPLE.com port 8443 (#0)
* Trying 192.168.10.146...
* Connected to
ds01.EXAMPLE.com (192.168.10.146) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/httpd/alias/
* CAfile: /etc/ipa/ca.crt
CApath: none
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Server
certificate:
* subject:
CN=ds01.EXAMPLE.com,O=EXAMPLE.COM
* start date: Dec 26 21:02:44 2016 GMT
* expire date: Dec 16 21:02:44 2018 GMT
* common name:
ds01.EXAMPLE.com
* issuer: CN=Certificate
Authority,O=EXAMPLE.COM
* NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
* Peer's certificate issuer has been marked as not trusted by the user.
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Closing connection 0
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here:
http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Steph