I did some checking of some of the same stuff on my other IPA server (ipa2; not a CA), and in LDAP it still has the old certificate that we replaced on ipa1.  Could the mismatch between these two servers be what is causing pki-tomcat to fail?  journalctl shows an error that seems to indicate some kind of communication error between the two LDAPs when trying to replicate.  I'm not sure though if that is a symptom of the problem we are trying to fix or part of the cause.

On Tue, Oct 31, 2017 at 8:06 AM, Kristian Petersen <nesretep@chem.byu.edu> wrote:
Unfortunately, this machine is the only CA.  I tried making one of my replicas a CA but because the pki-tomcat stuff was broken, of course that didn't work.  Super bad, I know.  Here is the result of that last command:
sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      53ace5456cb0c07b79d061b7aada366063799089   NSS Certificate DB:subsystemCert cert-pki-ca
< 1> rsa      e9ff606015f8c6a032ee88c51459e1952ba7f901   (orphan)
< 2> rsa      f40b78512366c34f88fa2da4900978a778048d4a   NSS Certificate DB:ocspSigningCert cert-pki-ca
< 3> rsa      8caa824ccc68966582b02dbc14aa422c3d08dee6   NSS Certificate DB:Server-Cert cert-pki-ca
< 4> rsa      6410804f149a562865b616fa3054640b45305ea2   caSigningCert cert-pki-ca
< 5> rsa      13cd3399d4c0734796fee85eca65a2ee05281146   NSS Certificate DB:auditSigningCert cert-pki-ca

On Tue, Oct 31, 2017 at 2:57 AM, Florence Blanc-Renaud <flo@redhat.com> wrote:
On 10/30/2017 05:23 PM, Kristian Petersen via FreeIPA-users wrote:
OK I think  I got the ldapmodify to work.  I reran the commands to check the two certs and they appear to match now.  However, when I run an ipactl restart the system still fails on pki-tomcatd.

Hi,
In this case I think that the next item to investigate is why the key cannot be listed using
sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'

In a previous mail, you wrote that the output of this command was
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.

This tells that
1/ the password is OK (otherwise certutil would display an error message)
2/ the key for 'subsystemCert cert-pki-ca' is missing from the nssdb.

Do you have a backup of the NSS DB /etc/pki/pki-tomcat/alias or was the CA installed on another master, so that we can get the private key?
Can you also list which keys are present in /etc/pki/pki-tomcat/alias with
sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt

Flo

On Mon, Oct 30, 2017 at 3:42 AM, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote:

    On 10/28/2017 01:15 AM, Kristian Petersen via FreeIPA-users wrote:

        I forgot to include the results of the commands in case it is
        helpful:

        -bash-4.2$ ldapsearch -LLL -D 'cn=directory manager' -W -b
        uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
        Enter LDAP Password:
        dn: uid=pkidbuser,ou=people,o=ipaca
        userCertificate::
        MIIDdTCCAl2gAwIBAgIBBDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxD
        SEVNLkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNTEwMTMyMDUwM

        jhaFw0xNzEwMDIyMDUwMjhaMC4xFTATBgNVBAoMDENIRU0uQllVLkVEVTEVMBMGA1UEAwwMQ0EgU3

        Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+

        z8z82YinNVC9YzOejrRqRHST4ZiJIq2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0

        sBM9Aplx/wBu0IIyA4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HX

        Npiocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ETitHkbCCxn+

        8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o+NuoAYaWi/hjpgq0ZXa2zM8

        zIy33h+A+UQIDAQABo4GUMIGRMB8GA1UdIwQYMBaAFB0PNWo+emloojFyMjHrItpaAfVCMD8GCCsG

        AQUFBwEBBDMwMTAvBggrBgEFBQcwAYYjaHR0cDovL2lwYTEuY2hlbS5ieXUuZWR1OjgwL2NhL29jc

        3AwDgYDVR0PAQH/BAQDAgTwMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w

        0BAQsFAAOCAQEAnsZeWq5e0UWJwaJqTiJdm+1jvQJrzOPWRYPfu9MTpfFjyhlNEwMX0azVzTrFbn2

        7+JjQpcxH60zNurhjfavdx3S+/Dmz0dZPgX6AKBeZMfKyyfLeXaoCz3AW9uIbiQZZFdQloGGB82Ek

        M78W6rJVxb5x9Juck4D4GaeqOuHgNPYVnpNkWR4shCnbGdGjrG4kQRO4I91DxYBrKnY8Fmucxq2y1

        4Xi29RT9Plx6p4g4E+LjqdZVAPlK/x3IQDxL2Shp/ycQxGEjfmPX8t3gbyi9e4QvHv5EdmrGpHlIQ

        bicsPmJ3gmDLn+EcIyoxpT7BLmJKPrn0FjF+FTyE/OrzHBkg==
        description: 2;4;CN=Certificate Authority,O=CHEM.BYU.EDU
        <http://CHEM.BYU.EDU> <http://CHEM.BYU.EDU>;CN=CA Subsystem,O=CHE
        M.BYU.EDU <http://M.BYU.EDU> <http://M.BYU.EDU>

        seeAlso: CN=CA Subsystem,O=CHEM.BYU.EDU <http://CHEM.BYU.EDU>
        <http://CHEM.BYU.EDU>

        -bash-4.2$ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n
        'subsystemCert cert-pki-ca' -a
        -----BEGIN CERTIFICATE-----
        MIIDdDCCAlygAwIBAgIBMDANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxDSEVN
        LkJZVS5FRFUxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0xNzA5
        MDQyMDUwNThaFw0xOTA4MjUyMDUwNThaMC4xFTATBgNVBAoMDENIRU0uQllVLkVE
        VTEVMBMGA1UEAwwMQ0EgU3Vic3lzdGVtMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
        MIIBCgKCAQEAtW9NKgtthoustZq+bobtAe+z8z82YinNVC9YzOejrRqRHST4ZiJI
        q2S6pGPUxbDcpit9eBgyjBT5Ale2B1BSN+SfKcBeK+AMjYF0sBM9Aplx/wBu0IIy
        A4owqw0QxhtSpvTFEAPZ15JJEb5Rakgl/Gb19+GIzt7FR2t6xtozPFjlzH5HXNpi
        ocdl7RvF6UjktsnE/0N5T/8aBPQbunECePUakskUjr0Cv1HjIKsERXtTn0HAc5ET
        itHkbCCxn+8oT082PzDmD1gPgtTI86bsuqcJIHVSqVCk3dIRBL0OLeD3tHkfIp4o
        +NuoAYaWi/hjpgq0ZXa2zM8zIy33h+A+UQIDAQABo4GTMIGQMB8GA1UdIwQYMBaA
        FB0PNWo+emloojFyMjHrItpaAfVCMD4GCCsGAQUFBwEBBDIwMDAuBggrBgEFBQcw
        AYYiaHR0cDovL2lwYS1jYS5jaGVtLmJ5dS5lZHUvY2Evb2NzcDAOBgNVHQ8BAf8E
        BAMCBPAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEB
        CwUAA4IBAQC3eGtIqHewdEtW7EagaUGkc4LoCulmhhmTC7lxOYYT+ADBrve6RSOA
        UpXSNCoetQU0QmXQkEXDtaZpjYFV2DaniwoAB6HuyG7do/BYdJoX8vKP/vCoJJCJ
        V64BuCE/uipYclGXbKZPkElbfASIAiNa6X+pSvhIqdTHS0dE7DpHK+m7sIlb1AO0
        yVmCZBIh1OT/sKajOaLA7epksAA1c9M0BSkdgjrIxAKaeHTtadnLPDEGVQor357Z
        yPyQ+vSM6GNI/Z02z+paX7WxuI/uZRHzD2MoprmUCfv03isv66EUu0EVox3wSEBT
        zXGp0EVo/JHfrENJKzszJ4qWGhXJfyII
        -----END CERTIFICATE-----
        -bash-4.2$


    Hi,

    so it looks like the certificate 'subsystemCert cert-pki-ca' has
    been renewed, stored in /etc/pki/pki-tomcat/alias but not copied
    into the LDAP server.

    The most recent version is the one in /etc/pki/pki-tomcat/alias (we
    can see that by comparing the serial numbers) and needs to be put
    into the LDAP entry. You can perform this using ldapmodify tool or a
    graphical LDAP browser.

    With ldapmodify:
    1/ extract the certificate from /etc/pki/pki-tomcat/alias into a
    single line, without the -----BEGIN CERTIFICATE---- and -----END
    CERTIFICATE----- delimiters:
    $ sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
    cert-pki-ca' -a | tail -n +2 | head -n -1 | tr -d '\r\n'
    MIIDdDCC...WGhXJfyII

    (tail -n +2 removes the -----BEGIN CERTIFICATE----- and head -n -1
    removes the -----END CERTIFICATE-----, while tr -d '\r\n' deletes
    new line and return characters).

    2/ Find the certificate serial number
    sudo certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
    cert-pki-ca' | grep Serial
             Serial Number: 48 (0x30)

    2/ perform ldapmodify with the value obtained above:
    ldapmodify -x -D 'cn=directory manager' -W
    dn: uid=pkidbuser,ou=people,o=ipaca
    changetype: modify
    replace: usercertificate
    usercertificate:: <PASTE output from above step 1 here>
    -
    replace: description
    description: 2;48;CN=Certificate Authority,O=CHEM.BYU.EDU
    <http://CHEM.BYU.EDU>,;CN=CA Subsystem,O=CHEM.BYU.EDU
    <http://CHEM.BYU.EDU>

    (do not forget to type return twice to send the modify command).
    In my example, the description field contains "48" as it is the
    serial number of the new subsystemCert cert-pki-ca obtained in the
    step 2.

    After that, you should be able to restart pki-tomcatd. Please tell
    me if you still encounter issues,

    Flo.

        On Fri, Oct 27, 2017 at 5:08 PM, Kristian Petersen
        <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>
        <mailto:nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>>>
        wrote:

             I also found that the certs don't match!  LDAP and certutil
        return
             different certs when you query them.  The blog post didn't
        suggest a
             method for fixing this and I don't want to make the problem
        worse by
             doing it the wrong way.  Suggestions?

             On Fri, Oct 27, 2017 at 1:35 PM, Kristian Petersen
             <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>
        <mailto:nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>>>
        wrote:

                 I followed some of the steps outlined in the blog post
        you liked
                 to and when I got to the part where make sure that the
        private
                 key can be read using the password found in
                 /var/lib/pki/pki-tomcat/conf/password.conf using:
                 sudo certutil -K -d /etc/pki/pki-tomcat/alias -f
                 /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'

                 RESULT:
                 certutil: Checking token "NSS Certificate DB" in slot
        "NSS User
                 Private Key and Certificate Services"
                 certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
                 Unrecognized Object Identifier.

                 So it looks like things aren't associated properly
        anymore. Not
                 sure what my next steps would be though.

                 On Fri, Oct 27, 2017 at 10:27 AM, Florence Blanc-Renaud
                 <flo@redhat.com <mailto:flo@redhat.com>
        <mailto:flo@redhat.com <mailto:flo@redhat.com>>> wrote:

                     On 10/27/2017 12:55 AM, Kristian Petersen via
        FreeIPA-users
                     wrote:

                         I checked the logs that turned up after running
        the find
                         command suggested by Jochen and only a couple
        of them
                         turned up anything that mention pki or pki-tomcat:

                         from /var/log/audit/audit.log:
                         type=SERVICE_START
        msg=audit(1508873851.623:163448):
                         pid=1 uid=0 auid=4294967295 ses=4294967295
                         subj=system_u:system_r:init_t:s0
                         msg='unit=pki-tomcatd@pki-tomcat comm="systemd"
                         exe="/usr/lib/systemd/systemd" hostname=? addr=?
                         terminal=? res=failed'

                         from /var/log/messages:
                         Oct 26 16:01:58 ipa1 ns-slapd:
                         [26/Oct/2017:16:01:58.077129423 -0600] - ERR -
                         slapi_ldap_bind - Error: could not bind id
                         [cn=Replication Manager
                                cloneAgreement1-ipa2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config]
                         authentication mechanism [SIMPLE]: error 32 (No
        such object)
                         Oct 26 16:01:58 ipa1 named-pkcs11[16463]: client
                         192.168.105.11#37937: request has invalid
        signature:
                         TSIG DHCP_UPDATER: tsig verify failure (BADKEY)


                     Hi,

                     just a wild guess, but we saw issues during update
        related
                     either to certificates or IPv6.
                     - Is IPv6 enabled on your server? The server
        doesn't need an
                     IPv6 address but IPv6 should not be disabled.
                     - If selinux is in enforcing mode, there were known
        issues
                     during certificate renewals that could lead to
        pki-tomcat
                     not able to start any more. You can refer to this
        blog post
                     [1] to check that the certificate 'subsystemCert
                     cert-pki-ca' is properly associated to the user
                     uid=pkidbuser,ou=people,o=ipaca. The certificate is
        stored
                     in multiple places (ldap server, nss dbs) and must be
                     consistent.

                     Flo

                     [1]
        https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/
        <https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/>
                            <https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/
        <https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/>>


                         On Thu, Oct 26, 2017 at 2:32 PM, Jochen Hein
                         <jochen@jochen.org <mailto:jochen@jochen.org>
        <mailto:jochen@jochen.org <mailto:jochen@jochen.org>>
                         <mailto:jochen@jochen.org
        <mailto:jochen@jochen.org> <mailto:jochen@jochen.org
        <mailto:jochen@jochen.org>>>>
                         wrote:

                              Kristian Petersen via FreeIPA-users
                              <freeipa-users@lists.fedorahosted.org
        <mailto:freeipa-users@lists.fedorahosted.org>
                         <mailto:freeipa-users@lists.fedorahosted.org
        <mailto:freeipa-users@lists.fedorahosted.org>>
                                     <mailto:freeipa-users@lists.fedorahosted.org
        <mailto:freeipa-users@lists.fedorahosted.org>

                         <mailto:freeipa-users@lists.fedorahosted.org
        <mailto:freeipa-users@lists.fedorahosted.org>>>> writes:

                              > The dirsrv log just shows a bunch of the
        following:
                              > [13/Oct/2017:14:32:07.132312021 -0600] -
        ERR -
                         slapi_ldap_bind - Error:
                              > could not bind id [cn=Replication Manager
                         cloneAgreement1-ipa
                              >
        2.chem.byu.edu-pki-tomcat,ou=csusers,cn=config]
                         authentication mechanism
                              > [SIMPLE]: error 32 (No such object)
                              >
                              > That makes sense though since pki-tomcat
        won't
                         start.  Rob was asking what
                              > was in the logs located at
                         /var/log/pki/pki-tomcat/ca/debug, but that path
                              > doesn't exist on any of my IPA servers.         He said
                         that would normally be the
                              > first place to look.  Hence, I am
        looking for
                         other solutions.

                              Brute force: reproduce the error and run "find
                         /var/log -mmin -1
                              -type f -ls".
                              This finds the files changed in the last
        minute -
                         one of these might
                              help.

                              Jochen

                              --
                              This space is intentionally left blank.




                         --                 Kristian Petersen
                         System Administrator
                         Dept. of Chemistry and Biochemistry


                         _______________________________________________
                         FreeIPA-users mailing list --
        freeipa-users@lists.fedorahosted.org
        <mailto:freeipa-users@lists.fedorahosted.org>
                         <mailto:freeipa-users@lists.fedorahosted.org
        <mailto:freeipa-users@lists.fedorahosted.org>>
                         To unsubscribe send an email to
        freeipa-users-leave@lists.fedorahosted.org
        <mailto:freeipa-users-leave@lists.fedorahosted.org>
                                <mailto:freeipa-users-leave@lists.fedorahosted.org
        <mailto:freeipa-users-leave@lists.fedorahosted.org>>





                 --         Kristian Petersen
                 System Administrator
                 Dept. of Chemistry and Biochemistry




             --     Kristian Petersen
             System Administrator
             Dept. of Chemistry and Biochemistry




        --         Kristian Petersen
        System Administrator
        Dept. of Chemistry and Biochemistry


        _______________________________________________
        FreeIPA-users mailing list --
        freeipa-users@lists.fedorahosted.org
        <mailto:freeipa-users@lists.fedorahosted.org>
        To unsubscribe send an email to
        freeipa-users-leave@lists.fedorahosted.org
        <mailto:freeipa-users-leave@lists.fedorahosted.org>





--
Kristian Petersen
System Administrator
Dept. of Chemistry and Biochemistry


_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org





--
Kristian Petersen
System Administrator
Dept. of Chemistry and Biochemistry



--
Kristian Petersen
System Administrator
Dept. of Chemistry and Biochemistry