Hi Florence,
thank you very much for your help.
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
> On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
>> On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
>>
>>> On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
>>>> my IPA system consists of 2 masters with their own self-signed CAs,
>>>> one of
>>>> them being the certificate renewal master (ipa1). The system has
>>>> been
>>>> running for years and has been migrated from an IPA 3 system.
>>>>
>>>> Since a while, the Web UI logins on ipa1 don't work anymore
("Login
>>>> failed
>>>> due to an unknown reason.").
>>>>
>>>> Web UI logins on the other server (ipa2) work and everything else is
>>>> working fine, too, ipactl status reports all services running.
>>>>
>>>> On login attempt:
>>>>
>>>> --- httpd log
>>>> [...]
>>>> [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551):
>>>> Exception
>>>> occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
>>>> [...]
>>>> [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError:
>>>> Command
>>>> '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
>>>> X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
>>>> X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem'
>>>> returned
>>>> non-zero exit status 1
>>>> ---
>>>>
>>>> --- krb5kdc.log
>>>> [...]
>>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15517](info): AS_REQ (8
>>>> etypes
>>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
>>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>>> Additional pre-authentication required
>>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15517](info): closing
>>>> down fd
>>>> 11
>>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15518](info): AS_REQ (8
>>>> etypes
>>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
>>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>>> Failed
>>>> to verify own certificate (depth 0): certificate has expired
>>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15518](info): closing
>>>> down fd
>>>> 11
>>>> ---
>>>>
>>>> --- ipa-checkcerts.py
>>>> IPA version 4.5.4-10.el7.centos.3
>>>> Check CA status
>>>> Check tracking
>>>> Check NSS trust
>>>> Check dates
>>>> Checking certificates in CS.cfg
>>>> Comparing certificates to requests in LDAP
>>>> Checking RA certificate
>>>> Checking authorities
>>>> Checking host keytab
>>>> Validating certificates
>>>> Checking renewal master
>>>> End-to-end cert API test
>>>> Checking permissions and ownership
>>>> Failures:
>>>> Unable to find request for serial 268304391
>>>> Unable to find request for serial 268304394
>>>> Unable to find request for serial 268304393
>>>> Unable to find request for serial 268304392
>>>> Subject
O=EXAMPLE.COM,CN=ipa2.example.com and template subject
>>>> CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
>>>> ---
>>>>
>>>> --- ipa pkinit-status --all
>>>> -----------------
>>>> 2 servers matched
>>>> -----------------
>>>> Server name:
ipa2.example.com
>>>> PKINIT status: enabled
>>>>
>>>> Server name:
ipa1.example.com
>>>> PKINIT status: enabled
>>>> ----------------------------
>>>> Number of entries returned 2
>>>> ----------------------------
>>>>
>>>> To my understanding, proper certificate exchange between my two
>>>> servers
>>>> ceased working at some point. How do i track this down and fix it?
>>>>
>>> your issue looks similar to ticket #6792 [1]. Can you check the
>>> result of
>>> upgrade in /var/log/ipaupgrade.log?
>>> Also check the output of
>>> $ ipa-pkinit-manage status
>>> and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
>>> /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r--
>>> permissions.
>>>
>>> Regarding the certificates, does getcert list show expired certs?
>>> flo
>>>
>>> [1]
https://pagure.io/freeipa/issue/6792
>>
>> ---
>> $ ipa-pkinit-manage status
>> PKINIT is enabled
>> ---
>>
>> There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem
>> exist with proper permissions, but I found something in ipaupgrade.log:
>>
>> ---
>> 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d
>> /etc/httpd/alias -L
>> -f /etc/httpd/alias/pwdfile.txt
>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
>> 2018-09-12T13:37:19Z DEBUG stdout=
>> Certificate Nickname Trust
>> Attributes
>>
>> SSL,S/MIME,JAR/XPI
>>
>> Server-Cert u,u,u
>> EXAMPLE.COM IPA CA CT,C,C
>>
>> 2018-09-12T13:37:19Z DEBUG stderr=
>> 2018-09-12T13:37:19Z DEBUG Starting external process
>> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d
>> /etc/httpd/alias -L
>> -n
EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt
>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
>> 2018-09-12T13:37:19Z DEBUG stdout=
>> -----BEGIN CERTIFICATE-----
>> [...]
>> -----END CERTIFICATE-----
>>
>> 2018-09-12T13:37:19Z DEBUG stderr=
>> 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin:
>> update_ra_cert_store
>> 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store
>> 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')
>> 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')
>> 2018-09-12T13:37:19Z DEBUG Starting external process
>> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d
>> /etc/httpd/alias -L
>> -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt
>> 2018-09-12T13:37:19Z DEBUG Process finished, return code=255
>> 2018-09-12T13:37:19Z DEBUG stdout=
>> 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert:
>> ipaCert
>> : PR_FILE_NOT_FOUND_ERROR: File not found
>> [...]
>> ---
>>
> this error can be ignored in most of the cases. The upgrade is trying
> to move ipaCert (cert+key) from the NSS database /etc/httpd/alias to
> the files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So
> if the upgrade is run a second time, he won't find ipaCert in the NSS
> database. To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key}
> are present and contain a certificate with Subject CN=IPA
>
RA,O=DOMAIN.COM. The files must be readable by root and ipaapi group,
> and must contain the same cert as the other masters.
checked this, is ok.
> What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
> ca-bundle.pem? both must contain IPA CA certificate.
True on both servers.
> What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs to
> be readable by everyone. And what is the content of this cert? It
> should be issued by IPA CA.
Permissions are ok, contents:
--- ipa1
$ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer:
O=EXAMPLE.COM,
CN=ipa1.example.com
Validity
Not Before: Nov 28 12:43:05 2017 GMT
Not After : Nov 28 12:43:05 2018 GMT
Subject:
O=EXAMPLE.COM,
CN=ipa1.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[...]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
othername:<unsupported>, othername:<unsupported>
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE
1.3.6.1.4.1.311.20.2:
.".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s
[...]
---
--- ipa2
$ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 805240833 (0x2fff0001)
Signature Algorithm: sha256WithRSAEncryption
Issuer:
O=EXAMPLE.COM, CN=Certificate Authority
Validity
Not Before: Jan 18 13:04:17 2018 GMT
Not After : Jan 19 13:04:17 2020 GMT
Subject:
O=EXAMPLE.COM,
CN=ipa2.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[...]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
Authority Information Access:
OCSP -
URI:http://ipa-ca.example.com/ca/ocsp
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment,
Data Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, 1.3.6.1.5.2.3.5
X509v3 CRL Distribution Points:
Full Name:
URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin
CRL Issuer:
DirName: O = ipaca, CN = Certificate Authority
X509v3 Subject Key Identifier:
96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3
X509v3 Subject Alternative Name:
othername:<unsupported>, othername:<unsupported>
[...]
---
Additional info: I have DNS separate from IPA, but i (hopefully) made
proper records as IPA would have done it. In particular, i made an A
record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not
the root cause of my problems, since DNS is not under my control.
Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by
IPA CA, might this be the actual problem?
The cert is expired on ipa1, which is the real root cause.
ipa-pkinit-manage status is reporting that PKINIT is enabled but this
does not match the actual configuration. This is a known issue [1].
If the host ipa1 is running a CA instance, then you can delete
/var/kerberos/krb5kdc/kdc.{key,crt} and run ipa-pkinit-manage enable to
re-generate the KDC cert.
Note: if the host doesn't run a CA instance, then this won't work
because of the issue [2].
To check which hosts are running a CA instance, you can use
# ipa server-role-find --role 'CA server'
[1]
> About the errors spotted by ipa-checkcerts.py, what are the
> certificates with errors reported? You can find them with ipa
> cert-show <serial>.
---
$ipa cert-show 57
Issuing CA: ipa
Certificate: [...]
Subject:
CN=ipa1.example.com,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Not Before: Tue Nov 28 12:42:07 2017 UTC
Not After: Mon Nov 18 12:42:07 2019 UTC
Serial number: 57
Serial number (hex): 0x39
Revoked: False
$ ipa cert-show 268304391
Issuing CA: ipa
Certificate: [...]
Subject: CN=IPA
RA,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:26:35 2018 UTC
Not After: Tue Jun 30 14:26:35 2020 UTC
Serial number: 268304391
Serial number (hex): 0xFFE0007
Revoked: False
$ ipa cert-show 268304392
Issuing CA: ipa
Certificate: [...]
Subject: CN=CA
Subsystem,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:26:45 2018 UTC
Not After: Tue Jun 30 14:26:45 2020 UTC
Serial number: 268304392
Serial number (hex): 0xFFE0008
Revoked: False
$ ipa cert-show 268304393
Issuing CA: ipa
Certificate: [...]
Subject: CN=OCSP
Subsystem,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:27:05 2018 UTC
Not After: Tue Jun 30 14:27:05 2020 UTC
Serial number: 268304393
Serial number (hex): 0xFFE0009
Revoked: False
$ ipa cert-show 268304394
Issuing CA: ipa
Certificate: [...]
Subject: CN=CA
Audit,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:27:25 2018 UTC
Not After: Tue Jun 30 14:27:25 2020 UTC
Serial number: 268304394
Serial number (hex): 0xFFE000A
Revoked: False
---
ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate
operation cannot be completed: EXCEPTION (Invalid Credential.)" btw.
The errors related to "Unable to find request for serial xxx" mean that
the cert is tracked by certmonger, but there is no corresponding LDAP
entry cn=xxx,ou=ca,ou=requests,o=ipaca on the local LDAP server.
Is the replication still working between your two masters?
"ipa cert-show" broken on ipa2 often points to an out-of-date
certificate in /var/lib/ipa/ra-agent.{pem|key}. The content should be
the same as on the renewal master, and also the same as in the entry
uid=ipara,ou=People,o=ipaca (which should be replicated and identical on
all the masters, except if you have replication issues).
HTH,
flo