That was kinda my belief thus far as well that the hosts were not trusting themselves - not 100% sure how things got here though.  I have a hunch it might be related to the initial deployment and the prior admin using an outdated method to install/manage/renew the LE-certificates.

=====
# ipa-cacert-manage list
IPA.XXXYYY.COM IPA CA
IPA.XXXYYY.COM IPA CA
DSTRootCAX3
letsencryptx3
isrgrootx1
lets-encrypt-r3-cross-signed
The ipa-cacert-manage command was successful
=====

How would I go about forcing re-installation of the host's own CA certificate to ensure it's trust? 

Also, since these nodes are not running on an RPM-based distro, the typical cert-store locations I have seen on other systems are not in the same location(s) so I'm not sure totally sure every location to point certutil to be able to examine each cert-store in depth as well (if that might help diagnose further).  I ask because I believe these were initially built and then had the "https://github.com/freeipa/freeipa-letsencrypt/" project used to initially deploy the LE-certs - prior to the ipa-cacert-manage command being the official path toward installing/managing these external certificates.  I know because this code had been git-pulled onto these nodes, but it obviously doesn't work properly since this git project manipulates the paths below directly instead of managing via the ipa-cacert-manage command.

ex>
/etc/httpd/alias/ (<=== not on these systems)
/etc/pki/pki-tomcat/alias/
/etc/ipa/nssdb/
/etc/dirsrv/slapd-IPA-XXXYYY-COM/


Checking that project's git page now though, I see their readme now mentions /var/lib/ipa/certs/, where I just noticed cacert.pem.

=====
/var/lib/ipa/certs# openssl x509 -text -noout -in cacert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = IPA.XXXYYY.COM, CN = Certificate Authority
        Validity
            Not Before: Apr 16 20:45:46 2020 GMT
            Not After : Apr 16 20:45:46 2040 GMT
        Subject: O = IPA.XXXYYY.COM, CN = Certificate Authority
...
=====
so I believe I might have just located the IPA CA cert in case I need to re-install it.



To the following question, I have the following LE-related certs installed.  And yes, I did run into issues a couple months back when LE moved to the new certs on their end so had to import the new authority certs to get the LE host certs to update & import.  The LE certificates are functioning and verify for both slapd and apache/tomcat.

=====
DSTRootCAX3.pem  LetsEncryptAuthorityX3.pem  isrgrootx1.pem  lets-encrypt-r3-cross-signed.pem
=====

Thank you all so much for the assistance through all this.

-Chris


On 6/16/21 1:26 PM, Rob Crittenden via FreeIPA-users wrote:
The error suggests that your 
IPA server doesn't trust its own CA
certificate.

Does ipa-cacert-manage list include the IPA CA?

BTW the new certificate steps are unrelated. This affects all CA requests.

rob

Chris Moody via FreeIPA-users wrote:
Just found some additional 
possible clues in the apache error.log

=====
[Tue Jun 15 17:11:34.636290 2021] [:warn] [pid 31831:tid
139703600768768] [client 2001:470:8af9:255::10:47920] failed to set
perms (3140) on file (/run/ipa/ccaches/chris@IPA.NODE-NINE.COM)!,
referer: https://REDACTED-1.ipa.REDACTED.com/ipa/ui/
[Tue Jun 15 17:11:34.674975 2021] [ssl:error] [pid 31830:tid
139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02039:
Certificate Verification: Error (19): self signed certificate in
certificate chain
[Tue Jun 15 17:11:34.675088 2021] [ssl:error] [pid 31830:tid
139703550412544] [client 2604:a880:2:d1::36:4001:58500] AH02261:
Re-negotiation handshake failed
[Tue Jun 15 17:11:34.675111 2021] [ssl:error] [pid 31830:tid
139703550412544] SSL Library Error: error:1417C086:SSL
routines:tls_process_client_certificate:certificate verify failed
[Tue Jun 15 17:11:34.675884 2021] [wsgi:error] [pid 31828:tid
139703722125056] [remote 2001:470:8af9:255::10:47920] ipa: INFO:
[jsonserver_session] chris@IPA.REDACTED.COM: cert_request('-----BEGIN
NEW CERTIFICATE
REQUEST-----\\nMIIEcTCCAlkCAQAwLDEaMBgGA1UEChMRSVBBLk5PREUtTklORS5DT00xDjAMBgNV\\nBAMTBWNocmlzMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA8QR2JbFR\\nE7S7/XAk9V1bNEvvXxB65MpLnIu6gLnfXr8yz0xcTjvEDmKAukS7u+GhwnVfQvwS\\nRjlexS5leIZfMFU59R5K6ubzM5e3Qy07xQU40+8jLS2o3SUo7CvY+TcgTmfJWiK4\\nlWlG70LcMy6VbBVf+nQNoenryPDK2Y94kG3ydcKjGLZsaJ69s1OaSXxZY4esEhk/\\nFJs+odjMQRguS5sbUmfpSy3FTJsvuy4Sge2X5HsHOCKM2bWMcDqkRGI428toET0i\\n+FIXlvroqJQ7OL/RmwVcYOFMwMWN5Ne3g0ECZUrOWRNGt+TfMmcUxfDaw1kl/QqA\\nIYtv0xBszOC9ECSAak9pUuhabn616jI0teZE1+4trXc1ZLwBKiF+Ev2F7wFLdoPK\\nTyikms8lzQ/GLj9c61NieXsDRJLU3ZMjhQkcfbKKp5R4fp6UPUuJ8MdDzNDiLyuB\\nkgyRLRmF7NBFGvdEV9javdvAuSQz/Wa2ReGRWhI4hj8OgYZaFrNdWbgIRSZUDy3f\\ngDt8NoS7ouoD8vGQ93OvvaUfqgMhs57qZbSZNTgoLKcDFwGyT4oqfA25wzQhzlXU\\nV+q6JLpyz2J+d2CI4B/7/Udh44YT9LiLn9y84QNBaD3qJWrLuYrZYk9hb5HIuJf7\\nXDSh7zHmSed1BrMn/BnzKpxwl201P/Oy/5cCAwEAAaAAMA0GCSqGSIb3DQEBCwUA\\nA4ICAQBhFLpWtZXDhpnmfRu1tkfVQEuGMhG0MdIC1NXAUMs0EoBXKA0/Ak3xCN9c\\nbbvZtBBOktqIV5bxV+d4X0RHLGYh2NGezWHS5gkA92xzmw9gyUKHRQpDmIV9IicH\\nUCSFwL5xO5DFUC2XXshHFgHqpi3kYxCvfcX+wvrmJwjxsbv7raMVsZlN/6w2Z4/p\\nS1VYakUZnXXBl8x3TbK4yZ+mvRX6RjQOU8oCvZMAGns/IgjTL++4O59XThV7VaAa\\nIromeDGnS9klR/hx7BBy7UureQT/aIBpFczjaU5qPBJxkrFi7K+f3vdms9PZOfKv\\n4eowQhanJwoxhkSE11sVmrQQ9+pmrTXHLgO8IFRWAonnM0La31WQ+uBD9vF8iAGS\\neMk+CvEGV6u2UwZph4P6t/KCCGT89BMnJHTRYmyMulx3Ko4jJDMV9v3nm/Ls75ti\\abcdefghijklmnophPfch6I7wJEp+t7egdgrd5jbj4m4lDOT9v9msknlOXoUu\\nibGzaylvac6xhtggDVdz/OIQS7l0jNZE0t0w5ZgEP2fkUlCP6ZpBBL7hloBIzv28...REDACTED\\n-----END
NEW CERTIFICATE REQUEST-----', cacn='ipa', principal='chris',
version='2.233'): NetworkError
=====

-Chris



On 6/15/21 5:09 PM, Chris Moody via FreeIPA-users wrote:
Apologies for the belated response - took me a bit to verify across
all clients.

When I installed the LE certs on each replica/server, I performed the
following:
=====(the privkey & fullchain files provided by LE)=====
ipa-server-certinstall -w -d privkey.pem fullchain.pem
&
/usr/sbin/ipa-certupdate
=====

I have verified via 'openssl s_client -connect' that both https &
ldaps are serving the proper LE certificates across all IPA servers.

I have also now iterated across every ipa-client installation and run
'ipa-certupdate' as well.  All the /etc/ssl/certs/ca-certificates.crt
files on each and every system have the LE certs and are current.

Unfortunately, the 'Actions->New Certificate' process I mentioned is
still giving me identical behavior.

I followed the exact steps in the NewCert dialogue from one of the
validated-current ipa-clients:
=====
# Create a certificate database or use an existing one. To create a new
database:
|# certutil -N -d <database path>|
# Create a CSR with subject /CN=<uid>,O=<realm>/, for example:
|# certutil -R -d <database path> -a -g <key size> -s
'CN=chris,O=IPA.REDACTED.COM'|


=====
IPA Error 907: NetworkError
cannot connect to
'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login': [SSL:
TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
=====

-Chris


On 6/12/21 3:55 AM, Florence Renaud wrote:
Hi,

when the let's encrypt certificates were installed, did you run
ipa-cacert-manage install on one of the nodes + ipa-certupdate on
_all the IPA machines_? It's important to run ipa-certupdate on all
the server/replicas/clients in order to install the CA everywhere.

flo

On Sat, Jun 12, 2021 at 2:19 AM Chris Moody via FreeIPA-users
<freeipa-users@lists.fedorahosted.org
<mailto:freeipa-users@lists.fedorahosted.org>> wrote:

    Hello folks.

    Hopefully I'm just missing something face-palm level obvious, but
    I am running into some trouble when interfacing with my CA
    functionality on an IPA server cluster.  My attempts at scouring
    all my saved prior-comms from the mailing-list as well as several
    search-engines are not enchanting me with much clue.

    It appears that my need for the LetsEncrypt certs for the
    user-facing Web-UI and LDAPs components are causing IPA to
    dis-trust itself.

    ===
    4-node cluster - Ubuntu19.10
    (all nodes currently fully updated/patched via the official
    Ubuntu repos)
    ===
    ipa --version
    VERSION: 4.8.1, API_VERSION: 2.233
    ===
    running letsencrypt certificates successfully for HTTPs & LDAPs
    connectivity
    ===

    These 4-nodes are all happily running and replicating betwixt
    each other.  LDAPs is functioning great and many linux systems
    are able to all join as freeipa-clients.  Users and groups are
    replicating and being used elegantly for many LDAP-based
    authentication/authorization needs.

    Overall, for these nodes, life is good.


    Where I'm running into trouble is in finally wanting to leverage
    certificate issuance on a per-user basis.  End goal is
    integrating things like yubikeys, user-cert auth, and so on.


    In the UI, when I enter a user's account and select Actions->New
    Certificate, I am able to successfully issue the couple prompted
    'certutil' commands to generate the user's CSR.  I then paste in
    the contents of the CSR and hit 'Issue' and run into the
    following error:
    ==========
    IPA Error 907: NetworkError
    cannot connect to
    'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login':
    [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
    ==========

    As I then start digging into cli-mode to attempt to understand
    where things are unhappy, I run into similar troubles with the
    server attempting to talk to itself and not being very happy
    about it.

    ==========
    chris@REDACTED-1:~$ ipa ca-find
    ------------
    1 CA matched
    ------------
      Name: ipa
      Description: IPA CA
      Authority ID: 8acca54b-64d7-44bf-b8f7-59316213cfb6
      Subject DN: CN=Certificate Authority,O=IPA.REDACTED.COM
    <http://IPA.REDACTED.COM>
      Issuer DN: CN=Certificate Authority,O=IPA.REDACTED.COM
    <http://IPA.REDACTED.COM>
    ----------------------------
    Number of entries returned 1
    ----------------------------
    chris@REDACTED-1:~$ ipa ca-show
    Name: ipa
    ipa: ERROR: cannot connect to
    'https://REDACTED-1.ipa.REDACTED.com:443/ca/rest/account/login':
    [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2508)
    ==========

    Verifying with 'openssl s_client' returns the valid and
    non-expired LE cert-chain.

    ==========
    chris@REDACTED-1:~$ openssl s_client
    REDACTED-1.ipa.REDACTED.com:443
    <http://REDACTED-1.ipa.REDACTED.com:443>
    CONNECTED(00000003)
    depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
    verify return:1
    depth=1 C = US, O = Let's Encrypt, CN = R3
    verify return:1
    depth=0 CN = REDACTED-1.ipa.REDACTED.com
    <http://REDACTED-1.ipa.REDACTED.com>
    verify return:1
    ---
    Certificate chain
     0 s:CN = REDACTED-1.ipa.REDACTED.com
    <http://REDACTED-1.ipa.REDACTED.com>
       i:C = US, O = Let's Encrypt, CN = R3
     1 s:C = US, O = Let's Encrypt, CN = R3
       i:O = Digital Signature Trust Co., CN = DST Root CA X3
    ---
    ...<output-truncated>...
    ---
    SSL handshake has read 3046 bytes and written 413 bytes
    Verification: OK
    ---
    New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
    Server public key is 2048 bit
    Secure Renegotiation IS NOT supported
    Compression: NONE
    Expansion: NONE
    No ALPN negotiated
    Early data was not sent
    Verify return code: 0 (ok)
    ...<output-truncated>...
    ==========

    Can anyone please hit me with some clue-bat as to where I can
    read to understand how to get IPA to love itself?  I'm suspecting
    it's likely some certificate inclusion/exception that I need to
    add so that API calls and the ipa command itself will actually
    respect the LE cert-chain?

    Any hints would be greatly appreciated.

    Thanks,
    -Chris

    -- 
    REDACTED, Inc.
    chris@REDACTED.com <mailto:chris@node-nine.com>
    619.354.6463

    _______________________________________________
    FreeIPA-users mailing list --
    freeipa-users@lists.fedorahosted.org
    <mailto:freeipa-users@lists.fedorahosted.org>
    To unsubscribe send an email to
    freeipa-users-leave@lists.fedorahosted.org
    <mailto:freeipa-users-leave@lists.fedorahosted.org>
    Fedora Code of Conduct:
    https://docs.fedoraproject.org/en-US/project/code-of-conduct/
    List Guidelines:
    https://fedoraproject.org/wiki/Mailing_list_guidelines
    List Archives:
    https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
    Do not reply to spam on the list, report it:
    https://pagure.io/fedora-infrastructure

-- 
REDACTED, Inc.
chris@REDACTED.com
619.354.6463

_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- 
REDACTED, Inc.
chris@REDACTED.com
619.354.6463


_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

-- 
Node-Nine, Inc.
chris@node-nine.com
619.354.6463