Restarting krb5kdc doesn't help, and although it restarts, it complains about /run/krb5kdc.pid.

[ipa01 ~]# systemctl restart krb5kdc
[ipa01 ~]# systemctl status krb5kdc
● krb5kdc.service - Kerberos 5 KDC
     Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; disabled; preset: disabled)
     Active: active (running) since Tue 2024-01-02 16:45:10 CET; 8s ago
    Process: 43349 ExecStart=/usr/sbin/krb5kdc -P /run/krb5kdc.pid $KRB5KDC_ARGS (code=exited, status=0/SUCCESS)
   Main PID: 43351 (krb5kdc)
      Tasks: 3 (limit: 48859)
     Memory: 6.6M
        CPU: 70ms
     CGroup: /system.slice/krb5kdc.service
             ├─43351 /usr/sbin/krb5kdc -P /run/krb5kdc.pid -w 2
             ├─43352 /usr/sbin/krb5kdc -P /run/krb5kdc.pid -w 2
             └─43353 /usr/sbin/krb5kdc -P /run/krb5kdc.pid -w 2

Jan 02 16:45:09 ipa01.hq.spinque.com systemd[1]: Starting Kerberos 5 KDC...
Jan 02 16:45:10 ipa01.hq.spinque.com systemd[1]: krb5kdc.service: Can't open PID file /run/krb5kdc.pid (yet?) after start: Operation not permitted
Jan 02 16:45:10 ipa01.hq.spinque.com systemd[1]: Started Kerberos 5 KDC.

[ipa01 ~]# ll /run/krb5kdc.pid
-rw-r--r--. 1 root root 6 Jan  2 16:45 /run/krb5kdc.pid

[ipa01 ~]# kinit roberto
Password for roberto@HQ.SPINQUE.COM:
kinit: Generic error (see e-text) while getting initial credentials


On Tue, 2 Jan 2024 at 16:19, Roberto Cornacchia <roberto.cornacchia@gmail.com> wrote:
Hi there, clients are having trouble with kerberos authentication:

$ kinit -V user
Using existing cache: xxxxxxxxxx:yyyyy
Using principal: user@SUB.EXAMPLE.COM
Password for user@SUB.EXAMPLE.COM:
kinit: Generic error (see e-text) while getting initial credentials

On the ipa server, /var/log/krb5kdc.log says:

Dec 24 14:40:34 ipa01.sub.example.com krb5kdc[3324](info): AS_REQ (6 etypes {aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26), camellia128-cts-cmac(25)}) <IP>: NEEDED_PREAUTH: user@SUB.EXAMPLE.COM for krbtgt/SUB.EXAMPLE.COM@SUB.EXAMPLE.COM, Additional pre-authentication required
Dec 24 14:40:34 ipa01.sub.example.com krb5kdc[3324](info): closing down fd 11
Dec 24 14:40:51 ipa01.sub.example.com krb5kdc[3324](info): AS_REQ : handle_authdata (2)
Dec 24 14:40:51 ipa01.sub.example.com krb5kdc[3324](info): AS_REQ (6 etypes {aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), camellia256-cts-cmac(26), camellia128-cts-cmac(25)}) 
<IP>: HANDLE_AUTHDATA: user@SUB.EXAMPLE.COM for krbtgt/SUB.EXAMPLE.COM@SUB.EXAMPLE.COM, No such file or directory
Dec 24 14:40:51 ipa01.sub.example.com krb5kdc[3324](info): closing down fd 11
Dec 24 14:40:57 ipa01.sub.example.com krb5kdc[3324](info): AS_REQ (4 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19)}) <IP>: NEEDED_PREAUTH: ldap/ipa01.sub.example.com@SUB.EXAMPLE.COM for krbtgt/SUB.EXAMPLE.COM@SUB.EXAMPLE.COM, Additional pre-authentication required
Dec 24 14:40:57 ipa01.sub.example.com krb5kdc[3324](info): closing down fd 11
Dec 24 14:40:57 ipa01.sub.example.com krb5kdc[3324](info): AS_REQ (4 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19)}) 
<IP>: ISSUE: authtime 1703425257, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, ldap/ipa01.sub.example.com@SUB.EXAMPLE.COM for krbtgt/SUB.EXAMPLE.COM@SUB.EXAMPLE.COM
Dec 24 14:40:57 ipa01.sub.example.com krb5kdc[3324](info): closing down fd 11
Dec 24 14:40:57 ipa01.sub.example.com krb5kdc[3324](info): TGS_REQ (4 etypes {aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17)}) 
<IP>: ISSUE: authtime 1703425257, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, ldap/ipa01.sub.example.com@SUB.EXAMPLE.COM for ldap/ipa02.sub.example.com@SUB.EXAMPLE.COM
Dec 24 14:40:57 ipa01.sub.example.com krb5kdc[3324](info): closing down fd 11

There are 2 ipa servers, ipa01 (Rocky 9.3, ipa 4.10.2) and ipa02 (Rock 9.1, ipa4.10.0), both with CA and DNS. ipa02 is CRL master.
On both, ipa-healthcheck doesn't find any issue.

Also: kinit fails from within ipa01, succeeds from within ipa02.

The issue seems to be in ipa01, and I have already tried to reinstall it from scratch. One thing that is different is the version.

Could you please help me figure out what's wrong?

Best regards,
Roberto