Finally had a chance to circle back and work on this further.
Based on my prior output of: ===== # ipa-cacert-manage list *IPA.REDACTED.COM IPA CA** **IPA.REDACTED.COM IPA CA* DSTRootCAX3 letsencryptx3 isrgrootx1 lets-encrypt-r3-cross-signed The ipa-cacert-manage command was successful ===== which does show the IPA CA certificate as being recognized and installed...and the manpage for the command references: ...CA certificate of the IPA CA (NSS database nickname: "caSigningCert cert-pki-ca")...
I was also able to see that dogtag implies that the IPA CA component(s) are installed/recognized/not-expired: ========== REDACTED-1:~# getcert list Number of certificates and requests being tracked: 10. Request ID '20200416204629': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=IPA RA,O=IPA.REDACTED.COM expires: 2022-04-06 13:46:30 PDT key usage: digitalSignature,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/lib/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20200416204717': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=CA Audit,O=IPA.REDACTED.COM expires: 2022-04-06 13:45:48 PDT key usage: digitalSignature,nonRepudiation pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20200416204718': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=OCSP Subsystem,O=IPA.REDACTED.COM expires: 2022-04-06 13:45:47 PDT eku: id-kp-OCSPSigning pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20200416204719': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=CA Subsystem,O=IPA.REDACTED.COM expires: 2022-04-06 13:45:47 PDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20200416204720': status: MONITORING stuck: no key pair storage: *type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca'*,token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=Certificate Authority,O=IPA.REDACTED.COM expires: 2040-06-26 14:20:56 PDT key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20200416204721': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=REDACTED-1.ipa.REDACTED.com,O=IPA.REDACTED.COM expires: 2022-04-06 13:45:47 PDT dns: REDACTED-1.ipa.REDACTED.com key usage: digitalSignature,keyEncipherment,dataEncipherment eku: id-kp-serverAuth pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20200416204854': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/lib/ipa/certs/kdc.key' certificate: type=FILE,location='/var/lib/ipa/certs/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=REDACTED-1.ipa.REDACTED.com,O=IPA.REDACTED.COM expires: 2022-04-17 13:48:54 PDT principal name: krbtgt/IPA.REDACTED.COM@IPA.REDACTED.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/lib/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20200416205655': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=KRA Audit,O=IPA.REDACTED.COM expires: 2022-04-06 13:53:14 PDT key usage: digitalSignature,nonRepudiation pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-kra" track: yes auto-renew: yes Request ID '20200416205656': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=KRA Transport Certificate,O=IPA.REDACTED.COM expires: 2022-04-06 13:53:13 PDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "transportCert cert-pki-kra" track: yes auto-renew: yes Request ID '20200416205657': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.REDACTED.COM subject: CN=KRA Storage Certificate,O=IPA.REDACTED.COM expires: 2022-04-06 13:53:13 PDT key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra" track: yes auto-renew: yes =====
Where else should I be looking to try and understand/debug why the server is rejecting itś own connection to itself? From my (albeit limited) understanding thus far, all the requisite components are present and accounted for, no?
Do my apache logs of the following give any hints to anyone as to what isn´t being trusted? ===== [Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid 139703550412544] [client 2604:XXX::36:4001:58500] AH02039: Certificate Verification: Error (19): self signed certificate in certificate chain [Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid 139703550412544] [client 2604:XXX::36:4001:58500] AH02261: Re-negotiation handshake failed [Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid 139703550412544] SSL Library Error: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed =====
-Chris
On 6/16/21 3:32 PM, Chris Moody via FreeIPA-users wrote:
That was kinda my belief thus far as well that the hosts were not trusting themselves - not 100% sure how things got here though. I have a hunch it might be related to the initial deployment and the prior admin using an outdated method to install/manage/renew the LE-certificates.
===== # ipa-cacert-manage list IPA.REDACTED.COM IPA CA IPA.REDACTED.COM IPA CA DSTRootCAX3 letsencryptx3 isrgrootx1 lets-encrypt-r3-cross-signed The ipa-cacert-manage command was successful =====
How would I go about forcing re-installation of the host's own CA certificate to ensure it's trust?
Also, since these nodes are not running on an RPM-based distro, the typical cert-store locations I have seen on other systems are not in the same location(s) so I'm not sure totally sure every location to point certutil to be able to examine each cert-store in depth as well (if that might help diagnose further). I ask because I believe these were initially built and then had the "https://github.com/freeipa/freeipa-letsencrypt/" project used to initially deploy the LE-certs - prior to the ipa-cacert-manage command being the official path toward installing/managing these external certificates. I know because this code had been git-pulled onto these nodes, but it obviously doesn't work properly since this git project manipulates the paths below directly instead of managing via the ipa-cacert-manage command.
ex> /etc/httpd/alias/ (<=== not on these systems) /etc/pki/pki-tomcat/alias/ /etc/ipa/nssdb/ /etc/dirsrv/slapd-IPA-REDACTED-COM/
Checking that project's git page now though, I see their readme now mentions /var/lib/ipa/certs/, where I just noticed cacert.pem.
===== /var/lib/ipa/certs# openssl x509 -text -noout -in cacert.pem Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: O = IPA.REDACTED.COM, CN = Certificate Authority Validity Not Before: Apr 16 20:45:46 2020 GMT Not After : Apr 16 20:45:46 2040 GMT Subject: O = IPA.REDACTED.COM, CN = Certificate Authority ... ===== so I believe I might have just located the IPA CA cert in case I need to re-install it.
To the following question, I have the following LE-related certs installed. And yes, I did run into issues a couple months back when LE moved to the new certs on their end so had to import the new authority certs to get the LE host certs to update & import. The LE certificates are functioning and verify for both slapd and apache/tomcat.
===== DSTRootCAX3.pem LetsEncryptAuthorityX3.pem isrgrootx1.pem lets-encrypt-r3-cross-signed.pem =====
Thank you all so much for the assistance through all this.
-Chris
On 6/16/21 1:26 PM, Rob Crittenden via FreeIPA-users wrote:
The error suggests that your IPA server doesn't trust its own CA certificate.
Does ipa-cacert-manage list include the IPA CA?
BTW the new certificate steps are unrelated. This affects all CA requests.
rob