Hi Florence,
thank you very much for your help.
On Fri, 21 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
On 12/20/18 6:52 PM, dbischof--- via FreeIPA-users wrote:
> On Thu, 20 Dec 2018, Florence Blanc-Renaud via FreeIPA-users wrote:
>
>> On 12/20/18 4:22 PM, dbischof--- via FreeIPA-users wrote:
>>> my IPA system consists of 2 masters with their own self-signed CAs,
>>> one of
>>> them being the certificate renewal master (ipa1). The system has been
>>> running for years and has been migrated from an IPA 3 system.
>>>
>>> Since a while, the Web UI logins on ipa1 don't work anymore
("Login
>>> failed
>>> due to an unknown reason.").
>>>
>>> Web UI logins on the other server (ipa2) work and everything else is
>>> working fine, too, ipactl status reports all services running.
>>>
>>> On login attempt:
>>>
>>> --- httpd log
>>> [...]
>>> [:error] [pid 15551] [remote 141.51.X.X:0] mod_wsgi (pid=15551):
>>> Exception
>>> occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
>>> [...]
>>> [:error] [pid 15551] [remote 141.51.X.X:0] CalledProcessError: Command
>>> '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_15551 -X
>>> X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
>>> X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned
>>> non-zero exit status 1
>>> ---
>>>
>>> --- krb5kdc.log
>>> [...]
>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15517](info): AS_REQ (8 etypes
>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: NEEDED_PREAUTH:
>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>> Additional pre-authentication required
>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15517](info): closing down fd
>>> 11
>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15518](info): AS_REQ (8 etypes
>>> {18 17 20 19 16 23 25 26}) 141.51.X.Y: KDC_RETURN_PADATA:
>>> WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM,
>>> Failed
>>> to verify own certificate (depth 0): certificate has expired
>>> Dec 20 16:06:54
ipa1.example.com krb5kdc[15518](info): closing down fd
>>> 11
>>> ---
>>>
>>> --- ipa-checkcerts.py
>>> IPA version 4.5.4-10.el7.centos.3
>>> Check CA status
>>> Check tracking
>>> Check NSS trust
>>> Check dates
>>> Checking certificates in CS.cfg
>>> Comparing certificates to requests in LDAP
>>> Checking RA certificate
>>> Checking authorities
>>> Checking host keytab
>>> Validating certificates
>>> Checking renewal master
>>> End-to-end cert API test
>>> Checking permissions and ownership
>>> Failures:
>>> Unable to find request for serial 268304391
>>> Unable to find request for serial 268304394
>>> Unable to find request for serial 268304393
>>> Unable to find request for serial 268304392
>>> Subject
O=EXAMPLE.COM,CN=ipa2.example.com and template subject
>>> CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
>>> ---
>>>
>>> --- ipa pkinit-status --all
>>> -----------------
>>> 2 servers matched
>>> -----------------
>>> Server name:
ipa2.example.com
>>> PKINIT status: enabled
>>>
>>> Server name:
ipa1.example.com
>>> PKINIT status: enabled
>>> ----------------------------
>>> Number of entries returned 2
>>> ----------------------------
>>>
>>> To my understanding, proper certificate exchange between my two servers
>>> ceased working at some point. How do i track this down and fix it?
>>>
>> your issue looks similar to ticket #6792 [1]. Can you check the result of
>> upgrade in /var/log/ipaupgrade.log?
>> Also check the output of
>> $ ipa-pkinit-manage status
>> and if the files /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
>> /var/lib/ipa-client/pki/ca-bundle.pem exist, with -rw-r--r-- permissions.
>>
>> Regarding the certificates, does getcert list show expired certs?
>> flo
>>
>> [1]
https://pagure.io/freeipa/issue/6792
>
> ---
> $ ipa-pkinit-manage status
> PKINIT is enabled
> ---
>
> There are no expired certificates, kdc-ca-bundle.pem and ca-bundle.pem
> exist with proper permissions, but I found something in ipaupgrade.log:
>
> ---
> 2018-09-12T13:37:18Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L
> -f /etc/httpd/alias/pwdfile.txt
> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
> 2018-09-12T13:37:19Z DEBUG stdout=
> Certificate Nickname Trust
> Attributes
>
> SSL,S/MIME,JAR/XPI
>
> Server-Cert u,u,u
>
EXAMPLE.COM IPA CA CT,C,C
>
> 2018-09-12T13:37:19Z DEBUG stderr=
> 2018-09-12T13:37:19Z DEBUG Starting external process
> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L
> -n
EXAMPLE.COM IPA CA -a -f /etc/httpd/alias/pwdfile.txt
> 2018-09-12T13:37:19Z DEBUG Process finished, return code=0
> 2018-09-12T13:37:19Z DEBUG stdout=
> -----BEGIN CERTIFICATE-----
> [...]
> -----END CERTIFICATE-----
>
> 2018-09-12T13:37:19Z DEBUG stderr=
> 2018-09-12T13:37:19Z DEBUG Executing upgrade plugin: update_ra_cert_store
> 2018-09-12T13:37:19Z DEBUG raw: update_ra_cert_store
> 2018-09-12T13:37:19Z DEBUG raw: ca_is_enabled(version=u'2.228')
> 2018-09-12T13:37:19Z DEBUG ca_is_enabled(version=u'2.228')
> 2018-09-12T13:37:19Z DEBUG Starting external process
> 2018-09-12T13:37:19Z DEBUG args=/usr/bin/certutil -d /etc/httpd/alias -L
> -n ipaCert -a -f /etc/httpd/alias/pwdfile.txt
> 2018-09-12T13:37:19Z DEBUG Process finished, return code=255
> 2018-09-12T13:37:19Z DEBUG stdout=
> 2018-09-12T13:37:19Z DEBUG stderr=certutil: Could not find cert: ipaCert
> : PR_FILE_NOT_FOUND_ERROR: File not found
> [...]
> ---
>
this error can be ignored in most of the cases. The upgrade is trying to
move ipaCert (cert+key) from the NSS database /etc/httpd/alias to the
files /var/lib/ipa/ra-agent.pem and /var/lib/ipa/ra-agent.key. So if the
upgrade is run a second time, he won't find ipaCert in the NSS database.
To be sure, you can check if /var/lib/ipa/ra-agent.{pem|key} are present
and contain a certificate with Subject CN=IPA
RA,O=DOMAIN.COM. The files
must be readable by root and ipaapi group, and must contain the same
cert as the other masters.
checked this, is ok.
What is the content of /var/lib/ipa-client/pki/kdc-ca-bundle.pem and
ca-bundle.pem? both must contain IPA CA certificate.
True on both servers.
What are the permissions of /var/kerberos/krb5kdc/kdc.crt? It needs
to
be readable by everyone. And what is the content of this cert? It should
be issued by IPA CA.
Permissions are ok, contents:
--- ipa1
$ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer:
O=EXAMPLE.COM,
CN=ipa1.example.com
Validity
Not Before: Nov 28 12:43:05 2017 GMT
Not After : Nov 28 12:43:05 2018 GMT
Subject:
O=EXAMPLE.COM,
CN=ipa1.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[...]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
othername:<unsupported>, othername:<unsupported>
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
86:52:EC:A1:C3:FB:EC:CC:6D:F2:09:E7:64:88:D1:80:F4:71:81:AE
1.3.6.1.4.1.311.20.2:
.".K.D.C.s._.P.K.I.N.I.T._.C.e.r.t.s
[...]
---
--- ipa2
$ openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout -fingerprint
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 805240833 (0x2fff0001)
Signature Algorithm: sha256WithRSAEncryption
Issuer:
O=EXAMPLE.COM, CN=Certificate Authority
Validity
Not Before: Jan 18 13:04:17 2018 GMT
Not After : Jan 19 13:04:17 2020 GMT
Subject:
O=EXAMPLE.COM,
CN=ipa2.example.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
[...]
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
keyid:4B:BA:AA:46:F1:29:E4:43:8B:DC:30:B4:90:3E:66:72:DD:F6:C7:FB
Authority Information Access:
OCSP -
URI:http://ipa-ca.example.com/ca/ocsp
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, 1.3.6.1.5.2.3.5
X509v3 CRL Distribution Points:
Full Name:
URI:http://ipa-ca.example.com/ipa/crl/MasterCRL.bin
CRL Issuer:
DirName: O = ipaca, CN = Certificate Authority
X509v3 Subject Key Identifier:
96:C3:94:70:7E:46:77:DB:91:F8:DF:D6:27:FE:73:0A:45:F3:78:F3
X509v3 Subject Alternative Name:
othername:<unsupported>, othername:<unsupported>
[...]
---
Additional info: I have DNS separate from IPA, but i (hopefully) made
proper records as IPA would have done it. In particular, i made an A
record "ipa-ca" that has IPs of both ipa1 and ipa2 - hope, this is not the
root cause of my problems, since DNS is not under my control.
Since /var/kerberos/krb5kdc/kdc.crt on ipa1 appears to be not issued by
IPA CA, might this be the actual problem?
About the errors spotted by ipa-checkcerts.py, what are the
certificates
with errors reported? You can find them with ipa cert-show <serial>.
---
$ipa cert-show 57
Issuing CA: ipa
Certificate: [...]
Subject:
CN=ipa1.example.com,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Not Before: Tue Nov 28 12:42:07 2017 UTC
Not After: Mon Nov 18 12:42:07 2019 UTC
Serial number: 57
Serial number (hex): 0x39
Revoked: False
$ ipa cert-show 268304391
Issuing CA: ipa
Certificate: [...]
Subject: CN=IPA
RA,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:26:35 2018 UTC
Not After: Tue Jun 30 14:26:35 2020 UTC
Serial number: 268304391
Serial number (hex): 0xFFE0007
Revoked: False
$ ipa cert-show 268304392
Issuing CA: ipa
Certificate: [...]
Subject: CN=CA
Subsystem,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:26:45 2018 UTC
Not After: Tue Jun 30 14:26:45 2020 UTC
Serial number: 268304392
Serial number (hex): 0xFFE0008
Revoked: False
$ ipa cert-show 268304393
Issuing CA: ipa
Certificate: [...]
Subject: CN=OCSP
Subsystem,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:27:05 2018 UTC
Not After: Tue Jun 30 14:27:05 2020 UTC
Serial number: 268304393
Serial number (hex): 0xFFE0009
Revoked: False
$ ipa cert-show 268304394
Issuing CA: ipa
Certificate: [...]
Subject: CN=CA
Audit,O=EXAMPLE.COM
Issuer: CN=Certificate
Authority,O=EXAMPLE.COM
Not Before: Wed Jul 11 14:27:25 2018 UTC
Not After: Tue Jun 30 14:27:25 2020 UTC
Serial number: 268304394
Serial number (hex): 0xFFE000A
Revoked: False
---
ipa cert-show on server ipa2 fails with "ipa: ERROR: Certificate operation
cannot be completed: EXCEPTION (Invalid Credential.)" btw.
Mit freundlichen Gruessen/With best regards,
--Daniel.