Hi Florence,
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/21/18 11:58 AM, dbischof--- via FreeIPA-users wrote:
> thank you very much for your help.
>
> On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
>
>> On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
>>> On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
>>>
>>>> On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
>>>>> my IPA system consists of 2 masters with their own self-signed
CAs,
>>>>> one of
>>>>> them being the certificate renewal master (ipa1). The system has
>>>>> been
>>>>> running for years and has been migrated from an IPA 3 system.
>>>>>
>>>>> Since a while, the Web UI logins on ipa1 don't work anymore
("Login
>>>>> failed
>>>>> due to an unknown reason.").
>>>>>
>>>>> Web UI logins on the other server (ipa2) work and everything else
is
>>>>> working fine, too, ipactl status reports all services running.
>>>>>
>>>>> On login attempt:
>>>>>
>>>>> --- httpd log
>>>>> [...]
>>>>> [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551):
>>>>> Exception
>>>>> occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
>>>>> [...]
>>>>> [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError:
>>>>> Command
>>>>> '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
>>>>> X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
>>>>> X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem'
>>>>> returned
>>>>> non-zero exit status 1
>>>>> ---
>>>>>
>>>>> --- krb5kdc.log
>>>>> [...]
>>>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15517](info): AS_REQ (8
>>>>> etypes
>>>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
>>>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>>>> Additional pre-authentication required
>>>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15517](info): closing
down
>>>>> fd
>>>>> 11
>>>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15518](info): AS_REQ (8
>>>>> etypes
>>>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
>>>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for
krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>>>> Failed
>>>>> to verify own certificate (depth 0): certificate has expired
>>>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15518](info): closing
down
>>>>> fd
>>>>> 11
>>>>> ---
>>>>>
>>>>> --- ipa-checkcerts.py
>>>>> IPA version 4.5.4-10.el7.centos.3
>>>>> Check CA status
>>>>> Check tracking
>>>>> Check NSS trust
>>>>> Check dates
>>>>> Checking certificates in CS.cfg
>>>>> Comparing certificates to requests in LDAP
>>>>> Checking RA certificate
>>>>> Checking authorities
>>>>> Checking host keytab
>>>>> Validating certificates
>>>>> Checking renewal master
>>>>> End-to-end cert API test
>>>>> Checking permissions and ownership
>>>>> Failures:
>>>>> Unable to find request for serial 268304391
>>>>> Unable to find request for serial 268304394
>>>>> Unable to find request for serial 268304393
>>>>> Unable to find request for serial 268304392
>>>>> Subject
O=EXAMPLE.COM,CN=ipa2.example.com and template subject
>>>>> CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
>>>>> ---
>>>>>
>>>>> --- ipa pkinit-status --all
>>>>> -----------------
>>>>> 2 servers matched
>>>>> -----------------
>>>>> Server name:
ipa2.example.com
>>>>> PKINIT status: enabled
>>>>>
>>>>> Server name:
ipa1.example.com
>>>>> PKINIT status: enabled
>>>>> ----------------------------
>>>>> Number of entries returned 2
>>>>> ----------------------------
>>>>>
>>>>> To my understanding, proper certificate exchange between my two
>>>>> servers
>>>>> ceased working at some point. How do i track this down and fix
it?
>>>>>
>>>> your issue looks similar to ticket #6792 [1]. Can you check the result
>>>> of
>>>> upgrade in /var/log/ipaupgrade.log?
>>>> Also check the output of
>>>> $ ipa-pkinit-manage status
>>>> and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
>>>> /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r--
>>>> permissions.
>>>>
>>>> Regarding the certificates, does getcert list show expired certs?
>>>> flo
>>>>
>>>> [1]
https://pagure.io/freeipa/issue/6792
>>>
>>> ---
>>> $ ipa-pkinit-manage status
>>> PKINIT is enabled
>>> ---
>>>
>>> There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem
>>> exist with proper permissions, but I found something in ipaupgrade.log:
>>>
>>> ---
>>> 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias
>>> -L
>>> -f /etc/httpd/alias/pwdfile.txt
>>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
>>> 2018-09-12T13:37:19Z DEBUG stdout=
>>> Certificate Nickname Trust
>>> Attributes
>>>
>>> SSL,S/MIME,JAR/XPI
>>>
>>> Server-Cert u,u,u
>>> EXAMPLE.COM IPA CA CT,C,C
>>>
>>> 2018-09-12T13:37:19Z DEBUG stderr=
>>> 2018-09-12T13:37:19Z DEBUG Starting external process
>>> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias
>>> -L
>>> -n
EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt
>>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
>>> 2018-09-12T13:37:19Z DEBUG stdout=
>>> -----BEGIN CERTIFICATE-----
>>> [...]
>>> -----END CERTIFICATE-----
>>>
>>> 2018-09-12T13:37:19Z DEBUG stderr=
>>> 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin:
>>> update_ra_cert_store
>>> 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store
>>> 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')
>>> 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')
>>> 2018-09-12T13:37:19Z DEBUG Starting external process
>>> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias
>>> -L
>>> -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt
>>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=255
>>> 2018-09-12T13:37:19Z DEBUG stdout=
>>> 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert:
>>> ipaCert
>>> : PR_FILE_NOT_FOUND_ERROR: File not found
>>> [...]
>>> ---
>>>
>> this error can be ignored in most of the cases. The upgrade is trying to
>> move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the
>> files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the
>> upgrade is run a second time, he won't find ipaCert in the NSS database.
>> To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present
>> and contain a certificate with Subject CN=IPA
RA,O=DOMAIN.COM. The files
>> must be readable by root and ipaapi group, and must contain the same cert
>> as the other masters.
>
> checked this, is ok.
>
>> What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
>> ca-bundle.pem? both must contain IPA CA certificate.
>
> True on both servers.
>
>> What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to be
>> readable by everyone. And what is the content of this cert? It should be
>> issued by IPA CA.
>
> Permissions are ok, contents:
>
> --- ipa1
> $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 1 (0x1)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer:
O=EXAMPLE.COM,
CN=ipa1.example.com
> Validity
> Not Before: Nov 28 12:43:05 2017 GMT
> Not After : Nov 28 12:43:05 2018 GMT
> Subject:
O=EXAMPLE.COM,
CN=ipa1.example.com
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> Public-Key: (2048 bit)
> Modulus:
> [...]
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Subject Alternative Name:
> othername:<unsupported>, othername:<unsupported>
> X509v3 Basic Constraints: critical
> CA:FALSE
> X509v3 Subject Key Identifier:
> 86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE
> 1.3.6.1.4.1.311.20.2:
> .".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s
> [...]
> ---
>
> --- ipa2
> $ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 805240833 (0x2fff0001)
> Signature Algorithm: sha256WithRSAEncryption
> Issuer:
O=EXAMPLE.COM, CN=Certificate Authority
> Validity
> Not Before: Jan 18 13:04:17 2018 GMT
> Not After : Jan 19 13:04:17 2020 GMT
> Subject:
O=EXAMPLE.COM,
CN=ipa2.example.com
> Subject Public Key Info:
> Public Key Algorithm: rsaEncryption
> Public-Key: (2048 bit)
> Modulus:
> [...]
> Exponent: 65537 (0x10001)
> X509v3 extensions:
> X509v3 Authority Key Identifier:
> keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
>
> Authority Information Access:
> OCSP -
URI:http://ipa-ca.example.com/ca/ocsp
>
> X509v3 Key Usage: critical
> Digital Signature, Non Repudiation, Key Encipherment,
> Data Encipherment
> X509v3 Extended Key Usage:
> TLS Web Server Authentication, 1.3.6.1.5.2.3.5
> X509v3 CRL Distribution Points:
>
> Full Name:
>
URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin
> CRL Issuer:
> DirName: O = ipaca, CN = Certificate Authority
>
> X509v3 Subject Key Identifier:
> 96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3
> X509v3 Subject Alternative Name:
> othername:<unsupported>, othername:<unsupported>
> [...]
> ---
>
> Additional info: I have DNS separate from IPA, but i (hopefully) made
> proper records as IPA would have done it. In particular, i made an A
> record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not
the
> root cause of my problems, since DNS is not under my control.
>
> Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by
> IPA CA, might this be the actual problem?
>
The cert is expired on ipa1, which is the real root cause.
ipa-pkinit-manage status is reporting that PKINIT is enabled but this
does not match the actual configuration. This is a known issue [1].
If the host ipa1 is running a CA instance, then you can delete
/var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable to
re-generate the KDC cert. Note: if the host doesn't run a CA instance,
then this won't work because of the issue [2].
To check which hosts are running a CA instance, you can use
# ipa server-role-find --role 'CA server'
[1]
https://pagure.io/freeipa/issue/7200
[2]
https://pagure.io/freeipa/issue/7795
removed the expired certificates, re-enabled PKINIT, certificates were
regenerated, Web UI logins working again. Thank you very much.
>> About the errors spotted by ipa-checkcerts.py, what are the
certificates
>> with errors reported? You can find them with ipa cert-show <serial>.
>
> ---
> $ipa cert-show 57
> Issuing CA: ipa
> Certificate: [...]
> Subject:
CN=ipa1.example.com,O=EXAMPLE.COM
> Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
> Not Before: Tue Nov 28 12:42:07 2017 UTC
> Not After: Mon Nov 18 12:42:07 2019 UTC
> Serial number: 57
> Serial number (hex): 0x39
> Revoked: False
>
> $ ipa cert-show 268304391
> Issuing CA: ipa
> Certificate: [...]
> Subject: CN=IPA
RA,O=EXAMPLE.COM
> Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
> Not Before: Wed Jul 11 14:26:35 2018 UTC
> Not After: Tue Jun 30 14:26:35 2020 UTC
> Serial number: 268304391
> Serial number (hex): 0xFFE0007
> Revoked: False
>
> $ ipa cert-show 268304392
> Issuing CA: ipa
> Certificate: [...]
> Subject: CN=CA
Subsystem,O=EXAMPLE.COM
> Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
> Not Before: Wed Jul 11 14:26:45 2018 UTC
> Not After: Tue Jun 30 14:26:45 2020 UTC
> Serial number: 268304392
> Serial number (hex): 0xFFE0008
> Revoked: False
>
> $ ipa cert-show 268304393
> Issuing CA: ipa
> Certificate: [...]
> Subject: CN=OCSP
Subsystem,O=EXAMPLE.COM
> Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
> Not Before: Wed Jul 11 14:27:05 2018 UTC
> Not After: Tue Jun 30 14:27:05 2020 UTC
> Serial number: 268304393
> Serial number (hex): 0xFFE0009
> Revoked: False
>
> $ ipa cert-show 268304394
> Issuing CA: ipa
> Certificate: [...]
> Subject: CN=CA
Audit,O=EXAMPLE.COM
> Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
> Not Before: Wed Jul 11 14:27:25 2018 UTC
> Not After: Tue Jun 30 14:27:25 2020 UTC
> Serial number: 268304394
> Serial number (hex): 0xFFE000A
> Revoked: False
> ---
>
> ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation
> cannot be completed: EXCEPTION (Invalid Credential.)" btw.
>
The errors related to "Unable to find request for serial xxx" mean that the
cert is tracked by certmonger, but there is no corresponding LDAP entry
cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.
Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date certificate in
/var/lib/ipa/ra-agent.{pem|key}. The content should be the same as on the
renewal master, and also the same as in the entry uid=ipara,ou=People,o=ipaca
(which should be replicated and identical on all the masters, except if you
have replication issues).
Replication appeared to be working, however, I did server maintenance
today, ipa1 and ipa2 got upgraded and rebooted. ipa1 works fine, ipa2 does
not come up properly:
--- ipa2
$ ipactl status
Directory Service: RUNNING
krb5kdc Service: STOPPED
kadmin Service: STOPPED
httpd Service: RUNNING
ipa-custodia Service: STOPPED
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: STOPPED
ipa: INFO: The ipactl command was successful
---
--- ipa1
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
[...]
description: 2;268304391;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM
---
Looks good.
--- ipa2
$ openssl x509 -in /var/lib/ipa/ra-agent.pem -text -noout | grep Serial
Serial Number: 268304391 (0xffe0007)
$ ldapsearch -x -b uid=ipara,ou=People,o=ipaca
[...]
description: 2;44;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM
---
Looks bad.
--- /var/log/ipaupgrade.log
[...]
2018-12-28T10:55:53Z DEBUG The ipa-server-upgrade command failed, exception:
RemoteRetrieveError: Failed to authenticate to CA REST API
---
--- /var/log/dirsrv/slapd-EXAMPLE-COM/errors
[...]
[28/Dec/2018:11:55:55.559648469 +0100] - ERR - set_krb5_creds - Could not get initial
credentials for principal [ldap/ipa2.example.com(a)EXAMPLE.COM] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
---
--- ipa-checkcerts.py
[...]
ipa: ERROR: ra.get_certificate(): EXCEPTION (Invalid Credential.)
ra.get_certificate(): EXCEPTION (Invalid Credential.)
ipa: INFO: Checking permissions and ownership
Checking permissions and ownership
ipa: INFO: Failures:
Failures:
ipa: INFO: Unable to find request for serial 268304391
Unable to find request for serial 268304391
ipa: INFO: Unable to find request for serial 268304394
Unable to find request for serial 268304394
ipa: INFO: Unable to find request for serial 268304393
Unable to find request for serial 268304393
ipa: INFO: Unable to find request for serial 268304392
Unable to find request for serial 268304392
ipa: INFO: Unable to find request for serial 268304390
Unable to find request for serial 268304390
ipa: INFO: RA agent description does not match 2;44;CN=Certificate
Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM in LDAP and 2;268304391;CN=Certificate
Authority,O=EXAMPLE.COM;CN=IPA
RA,O=EXAMPLE.COM expected
ipa: INFO: RA agent certificate not found in LDAP RA agent certificate not found in LDAP
[...]
---
The root cause appears to be the wrong IPA RA certificate in ipa2's LDAP,
right? I guess this has to fixed by manually importing the proper
certificate using ldapmodify, similar to the procedure described in [1]?
[1]
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
Mit freundlichen Gruessen/With best regards,
--Daniel.