@abokovoy - Thanks for the heads up, the manual fix helped me solving the issue.

On Mon, Dec 28, 2020 at 1:20 AM Alexander Bokovoy <abokovoy@redhat.com> wrote:
On su, 27 joulu 2020, D R via FreeIPA-users wrote:
>Greetings,
>
>After automatic KDC certificate renewal, I'm no longer able to access the
>UI.
>
>[Sun Dec 27 23:33:20.563064 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72] Traceback (most recent call last):
>[Sun Dec 27 23:33:20.563085 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]   File "/usr/share/ipa/wsgi.py", line 59, in application
>[Sun Dec 27 23:33:20.563121 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]     return api.Backend.wsgi_dispatch(environ,
>start_response)
>[Sun Dec 27 23:33:20.563129 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]   File
>"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 267, in
>__call__
>[Sun Dec 27 23:33:20.563142 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]     return self.route(environ, start_response)
>[Sun Dec 27 23:33:20.563160 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]   File
>"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 279, in
>route
>[Sun Dec 27 23:33:20.563170 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]     return app(environ, start_response)
>[Sun Dec 27 23:33:20.563174 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]   File
>"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 937, in
>__call__
>[Sun Dec 27 23:33:20.563182 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]     self.kinit(user_principal, password, ipa_ccache_name)
>[Sun Dec 27 23:33:20.563194 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]   File
>"/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 973, in
>kinit
>[Sun Dec 27 23:33:20.563201 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]     pkinit_anchors=[paths.KDC_CERT,
>paths.KDC_CA_BUNDLE_PEM],
>[Sun Dec 27 23:33:20.563209 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]   File
>"/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 127, in
>kinit_armor
>[Sun Dec 27 23:33:20.563219 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]     run(args, env=env, raiseonerr=True, capture_error=True)
>[Sun Dec 27 23:33:20.563225 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]   File
>"/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 563, in run
>[Sun Dec 27 23:33:20.563234 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72]     raise CalledProcessError(p.returncode, arg_string,
>str(output))
>[Sun Dec 27 23:33:20.563263 2020] [:error] [pid 6150] [remote
>10.xx.xx.22:72] CalledProcessError: Command '/usr/bin/kinit -n -c
>/var/run/ipa/ccaches/armor_6150 -X
>X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
>X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned
>non-zero exit status 1
>
>---
>
>KRB5_TRACE=/dev/stdout /usr/bin/kinit -n -c
>/var/run/ipa/ccaches/armor_19265 -X
>X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
>X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
>[12904] 1609104974.342210: Getting initial credentials for WELLKNOWN/
>ANONYMOUS@A-LABS.COM
>[12904] 1609104974.342212: Sending unauthenticated request
>[12904] 1609104974.342213: Sending request (184 bytes) to A-LABS.COM
>[12904] 1609104974.342214: Initiating TCP connection to stream
>10.xx.xx.90:88
>[12904] 1609104974.342215: Sending TCP request to stream 10.xx.xx.90:88
>[12904] 1609104974.342216: Received answer (335 bytes) from stream
>10.xx.xx.90:88
>[12904] 1609104974.342217: Terminating TCP connection to stream
>10.xx.xx.90:88
>[12904] 1609104974.342218: Response was from master KDC
>[12904] 1609104974.342219: Received error from KDC: -1765328359/Additional
>pre-authentication required
>[12904] 1609104974.342222: Preauthenticating using KDC method data
>[12904] 1609104974.342223: Processing preauth types: PA-PK-AS-REQ (16),
>PA-PK-AS-REP_OLD (15), PA-PK-AS-REQ_OLD (14), PA-FX-FAST (136),
>PA-ETYPE-INFO2 (19), PA-PKINIT-KX (147), PA-ENC-TIMESTAMP (2), PA-FX-COOKIE
>(133)
>[12904] 1609104974.342224: Selected etype info: etype aes256-cts, salt
>"A-LABS.COMWELLKNOWNANONYMOUS", params ""
>[12904] 1609104974.342225: Received cookie: MIT
>[12904] 1609104974.342226: Preauth module pkinit (147) (info) returned:
>0/Success
>[12904] 1609104974.342227: PKINIT loading CA certs and CRLs from FILE
>[12904] 1609104974.342228: PKINIT loading CA certs and CRLs from FILE
>[12904] 1609104974.342229: PKINIT loading CA certs and CRLs from FILE
>[12904] 1609104974.342230: PKINIT client computed kdc-req-body checksum
>9/D4FAE675E4E8C9664DBE0FAD0EB8C416A639CAF3
>[12904] 1609104974.342232: PKINIT client making DH request
>[12904] 1609104974.342233: Preauth module pkinit (16) (real) returned:
>0/Success
>[12904] 1609104974.342234: Produced preauth for next request: PA-FX-COOKIE
>(133), PA-PK-AS-REQ (16)
>[12904] 1609104974.342235: Sending request (1497 bytes) to A-LABS.COM
>[12904] 1609104974.342236: Initiating TCP connection to stream
>10.xx.xx.90:88
>[12904] 1609104974.342237: Sending TCP request to stream 10.xx.xx.90:88
>[12904] 1609104974.342238: Received answer (1603 bytes) from stream
>10.xx.xx.90:88
>[12904] 1609104974.342239: Terminating TCP connection to stream
>10.xx.xx.90:88
>[12904] 1609104974.342240: Response was from master KDC
>[12904] 1609104974.342241: Processing preauth types: PA-PK-AS-REP (17),
>PA-ETYPE-INFO2 (19), PA-PKINIT-KX (147)
>[12904] 1609104974.342242: Selected etype info: etype aes256-cts, salt
>"A-LABS.COMWELLKNOWNANONYMOUS", params ""
>[12904] 1609104974.342243: Preauth module pkinit (147) (info) returned:
>0/Success
>[12904] 1609104974.342244: PKINIT client verified DH reply
>[12904] 1609104974.342245: Preauth module pkinit (17) (real) returned:
>-1765328308/KDC name mismatch

It says 'KDC name mismatch'.

There are two requirements in the MIT Kerberos PKINIT plugin code on the
client side. After validating signed data and collecting SANs from the
certificate presented by KDC, PKINIT plugin on the client checks:

   - whether list of SANs contains Kerberos principal for
     krbtgt/REALM@REALM, this is enough, no other checks would be needed

   - whether list of SANs contains KDC hostname and whether one of
     EKUs in the certificate match id-pkinit-kdc

See https://pagure.io/freeipa/issue/8532 for a possible manual fix.


>[12904] 1609104974.342246: Produced preauth for next request: (empty)
>[12904] 1609104974.342247: Getting AS key, salt
>"A-LABS.COMWELLKNOWNANONYMOUS", params ""
>Password for WELLKNOWN/ANONYMOUS@A-LABS.COM:
>[12904] 1609104977.873071: AS key obtained from gak_fct: aes256-cts/B8BD
>kinit: Password incorrect while getting initial credentials
>
>--
>
>openssl x509 -in /var/kerberos/krb5kdc/kdc.crt -text -noout
>Certificate:
>    Data:
>        Version: 3 (0x2)
>        Serial Number: 10 (0xa)
>    Signature Algorithm: sha256WithRSAEncryption
>        Issuer: O=DOMAIN.COM, CN=ipa.domain.com

This is a self-issued local certificate, looks like the issue above. The
issuer here should be

Issuer: CN=Certificate Authority,O=DOMAIN.COM

>        Validity
>            Not Before: Dec 27 07:38:54 2020 GMT
>            Not After : Dec 27 07:38:54 2021 GMT
>        Subject: O=DOMAIN.COM, CN=ipa.domain.com
>        Subject Public Key Info:
>            Public Key Algorithm: rsaEncryption
>                Public-Key: (2048 bit)
>                Modulus:
>                    00:cc:6e:b1:b1:2d:05:ab:f1:df:ce:01:43:d5:80:
>                    4a:f6:72:38:3c:50:aa:c7:40:bf:bd:6c:60:5e:8d:
>                    d0:f3:2b:6c:db:fc:8f:48:9f:91:d6:d3:d2:43:f2:
>                    39:35:17:56:37:a8:6f:66:c3:ab:1f:13:8f:d9:48:
>                    c3:be:b9:2b:83:77:78:08:fe:3b:f8:93:83:1c:bb:
>                    d0:e8:eb:49:a5:c1:8c:7f:0c:b5:fa:e7:07:f1:0c:
>                    97:9b:47:e9:a2:a3:ab:9b:c1:70:e3:1b:e9:f2:3d:
>                    2f:96:53:6d:38:eb:57:19:7f:dd:ed:e8:3c:c8:f0:
>                    7c:36:b1:72:03:f1:2f:86:8e:cd:67:fd:fd:85:73:
>                    00:16:60:81:3c:ad:13:4d:19:c0:4d:e7:94:8d:34:
>                    29:99:7a:45:70:db:81:5d:0e:2d:83:7a:9c:19:c7:
>                    ef:0a:79:8d:84:af:74:a3:b9:90:c8:b1:8c:65:d0:
>                    2d:e0:89:98:42:e0:cb:c8:b0:e3:b5:7c:9b:44:01:
>                    a8:31:15:8d:19:79:c5:35:26:4d:3f:e6:83:64:7f:
>                    15:da:50:c1:5e:9c:67:1b:27:e5:35:0c:a8:71:a9:
>                    4e:ee:ef:92:b5:f9:10:f6:31:82:2c:94:04:05:c5:
>                    89:c6:96:1d:48:11:e5:8d:05:92:56:93:99:55:66:
>                    b0:93
>                Exponent: 65537 (0x10001)
>        X509v3 extensions:
>            X509v3 Basic Constraints: critical
>                CA:FALSE
>    Signature Algorithm: sha256WithRSAEncryption
>
>To my understanding, something is wrong with the kdc certificate, it lacks
>some attributes. I'm just not sure how to generate a proper cert.

It would be good to see all extensions and SANs from the cert. You need
to use GnuTLS tools to be able to print Kerberos extensions correctly.

Install gnutls-utils and do
# certtool -i --infile /var/kerberos/krb5kdc/kdc.crt


--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland