Hi,

it seems the error happens when you run commands that require communication between IPA framework and the Certificate Server (like ipa ca-show). The workflow is the following:
1. the client (= the command "ipa ca-show") is a python process that communicates with httpd on the secure port. It seems this part is OK (ipa ca-find returns successfully).
2. the IPA framework is a wsgi app running inside httpd. The handling of "ipa ca-show" requires the framework to communicate with Dogtag, which is running inside pki-tomcat.The communication happens over a secure port with authentication based on the RA certificate. This communication is not working, probably because httpd doesn't trust the CA that issued Dogtag's server cert.

I think you need to check where httpd is getting its list of trusted CAs when it's acting as a client of Dogtag server. The code is using api.env.tls_ca_cert which is /etc/ipa/ca.crt on rhel/fedora (you can check with "ipa env tls_ca_cert" to find the value on your server) but may be different on ubuntu.

flo

On Thu, Jun 24, 2021 at 8:49 PM Chris Moody via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
Finally had a chance to circle back and work on this further.

Based on my prior output of:
=====
# ipa-cacert-manage list
IPA.REDACTED.COM IPA CA
IPA.REDACTED.COM IPA CA
DSTRootCAX3
letsencryptx3
isrgrootx1
lets-encrypt-r3-cross-signed
The ipa-cacert-manage command was successful
=====
which does show the IPA CA certificate as being recognized and installed...and the manpage for the command references:
...CA certificate of the IPA CA (NSS database nickname: "caSigningCert cert-pki-ca")...

I was also able to see that dogtag implies that the IPA CA component(s) are installed/recognized/not-expired:
==========
REDACTED-1:~# getcert list
Number of certificates and requests being tracked: 10.
Request ID '20200416204629':
    status: MONITORING
    stuck: no
    key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
    certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
    CA: dogtag-ipa-ca-renew-agent
    issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
    subject: CN=IPA RA,O=IPA.REDACTED.COM
    expires: 2022-04-06 13:46:30 PDT
    key usage: digitalSignature,keyEncipherment,dataEncipherment
    eku: id-kp-clientAuth
    pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre
    post-save command: /usr/lib/ipa/certmonger/renew_ra_cert
    track: yes
    auto-renew: yes
Request ID '20200416204717':
    status: MONITORING
    stuck: no
    key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
    CA: dogtag-ipa-ca-renew-agent
    issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
    subject: CN=CA Audit,O=IPA.REDACTED.COM
    expires: 2022-04-06 13:45:48 PDT
    key usage: digitalSignature,nonRepudiation
    pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
    track: yes
    auto-renew: yes
Request ID '20200416204718':
    status: MONITORING
    stuck: no
    key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
    CA: dogtag-ipa-ca-renew-agent
    issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
    subject: CN=OCSP Subsystem,O=IPA.REDACTED.COM
    expires: 2022-04-06 13:45:47 PDT
    eku: id-kp-OCSPSigning
    pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
    track: yes
    auto-renew: yes
Request ID '20200416204719':
    status: MONITORING
    stuck: no
    key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
    CA: dogtag-ipa-ca-renew-agent
    issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
    subject: CN=CA Subsystem,O=IPA.REDACTED.COM
    expires: 2022-04-06 13:45:47 PDT
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-clientAuth
    pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
    track: yes
    auto-renew: yes
Request ID '20200416204720':
    status: MONITORING
    stuck: no
    key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
    CA: dogtag-ipa-ca-renew-agent
    issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
    subject: CN=Certificate Authority,O=IPA.REDACTED.COM
    expires: 2040-06-26 14:20:56 PDT
    key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
    pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
    track: yes
    auto-renew: yes
Request ID '20200416204721':
    status: MONITORING
    stuck: no
    key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
    CA: dogtag-ipa-ca-renew-agent
    issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
    subject: CN=REDACTED-1.ipa.REDACTED.com,O=IPA.REDACTED.COM
    expires: 2022-04-06 13:45:47 PDT
    dns: REDACTED-1.ipa.REDACTED.com
    key usage: digitalSignature,keyEncipherment,dataEncipherment
    eku: id-kp-serverAuth
    pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
    track: yes
    auto-renew: yes
Request ID '20200416204854':
    status: MONITORING
    stuck: no
    key pair storage: type=FILE,location='/var/lib/ipa/certs/kdc.key'
    certificate: type=FILE,location='/var/lib/ipa/certs/kdc.crt'
    CA: IPA
    issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
    subject: CN=REDACTED-1.ipa.REDACTED.com,O=IPA.REDACTED.COM
    expires: 2022-04-17 13:48:54 PDT
    principal name: krbtgt/IPA.REDACTED.COM@IPA.REDACTED.COM
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-serverAuth,id-pkinit-KPKdc
    pre-save command:
    post-save command: /usr/lib/ipa/certmonger/renew_kdc_cert
    track: yes
    auto-renew: yes
Request ID '20200416205655':
    status: MONITORING
    stuck: no
    key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-kra',token='NSS Certificate DB'
    CA: dogtag-ipa-ca-renew-agent
    issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
    subject: CN=KRA Audit,O=IPA.REDACTED.COM
    expires: 2022-04-06 13:53:14 PDT
    key usage: digitalSignature,nonRepudiation
    pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-kra"
    track: yes
    auto-renew: yes
Request ID '20200416205656':
    status: MONITORING
    stuck: no
    key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='transportCert cert-pki-kra',token='NSS Certificate DB'
    CA: dogtag-ipa-ca-renew-agent
    issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
    subject: CN=KRA Transport Certificate,O=IPA.REDACTED.COM
    expires: 2022-04-06 13:53:13 PDT
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-clientAuth
    pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "transportCert cert-pki-kra"
    track: yes
    auto-renew: yes
Request ID '20200416205657':
    status: MONITORING
    stuck: no
    key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB',pin set
    certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='storageCert cert-pki-kra',token='NSS Certificate DB'
    CA: dogtag-ipa-ca-renew-agent
    issuer: CN=Certificate Authority,O=IPA.REDACTED.COM
    subject: CN=KRA Storage Certificate,O=IPA.REDACTED.COM
    expires: 2022-04-06 13:53:13 PDT
    key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
    eku: id-kp-clientAuth
    pre-save command: /usr/lib/ipa/certmonger/stop_pkicad
    post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "storageCert cert-pki-kra"
    track: yes
    auto-renew: yes
=====

Where else should I be looking to try and understand/debug why the server is rejecting itś own connection to itself?  From my (albeit limited) understanding thus far, all the requisite components are present and accounted for, no?

Do my apache logs of the following give any hints to anyone as to what isn´t being trusted?
=====
[Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid 139703550412544] [client 2604:XXX::36:4001:58500] AH02039: Certificate Verification: Error (19): self signed certificate in certificate chain
[Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid 139703550412544] [client 2604:XXX::36:4001:58500] AH02261: Re-negotiation handshake failed
[Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid 139703550412544] SSL Library Error: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed
=====

-Chris


On 6/16/21 3:32 PM, Chris Moody via FreeIPA-users wrote:
That was kinda my belief thus far as well that the hosts were not trusting themselves - not 100% sure how things got here though.  I have a hunch it might be related to the initial deployment and the prior admin using an outdated method to install/manage/renew the LE-certificates.

=====
# ipa-cacert-manage list
IPA.REDACTED.COM IPA CA
IPA.REDACTED.COM IPA CA
DSTRootCAX3
letsencryptx3
isrgrootx1
lets-encrypt-r3-cross-signed
The ipa-cacert-manage command was successful
=====

How would I go about forcing re-installation of the host's own CA certificate to ensure it's trust? 

Also, since these nodes are not running on an RPM-based distro, the typical cert-store locations I have seen on other systems are not in the same location(s) so I'm not sure totally sure every location to point certutil to be able to examine each cert-store in depth as well (if that might help diagnose further).  I ask because I believe these were initially built and then had the "https://github.com/freeipa/freeipa-letsencrypt/" project used to initially deploy the LE-certs - prior to the ipa-cacert-manage command being the official path toward installing/managing these external certificates.  I know because this code had been git-pulled onto these nodes, but it obviously doesn't work properly since this git project manipulates the paths below directly instead of managing via the ipa-cacert-manage command.

ex>
/etc/httpd/alias/ (<=== not on these systems)
/etc/pki/pki-tomcat/alias/
/etc/ipa/nssdb/
/etc/dirsrv/slapd-IPA-REDACTED-COM/


Checking that project's git page now though, I see their readme now mentions /var/lib/ipa/certs/, where I just noticed cacert.pem.

=====
/var/lib/ipa/certs# openssl x509 -text -noout -in cacert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = IPA.REDACTED.COM, CN = Certificate Authority
        Validity
            Not Before: Apr 16 20:45:46 2020 GMT
            Not After : Apr 16 20:45:46 2040 GMT
        Subject: O = IPA.REDACTED.COM, CN = Certificate Authority
...
=====
so I believe I might have just located the IPA CA cert in case I need to re-install it.



To the following question, I have the following LE-related certs installed.  And yes, I did run into issues a couple months back when LE moved to the new certs on their end so had to import the new authority certs to get the LE host certs to update & import.  The LE certificates are functioning and verify for both slapd and apache/tomcat.

=====
DSTRootCAX3.pem  LetsEncryptAuthorityX3.pem  isrgrootx1.pem  lets-encrypt-r3-cross-signed.pem
=====

Thank you all so much for the assistance through all this.

-Chris


On 6/16/21 1:26 PM, Rob Crittenden via FreeIPA-users wrote:
The error suggests that your 
IPA server doesn't trust its own CA
certificate.

Does ipa-cacert-manage list include the IPA CA?

BTW the new certificate steps are unrelated. This affects all CA requests.

rob

_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure