Hi all,
We've been chasing a WebUI login failure for several days and have exhausted our own diagnostic options. Posting the full picture in case anyone recognizes this or can point us at something we've missed.
Environment:
- 3-server multi-master topology (2x RHEL8/Oracle Linux 8 replicas on ipa-server-4.9.13-21.0.1.module+el8.10.0, 1x older replica on ipa-server-4.6.8-5.el7_9.12, since removed from topology — see below)
- Realm: FX.LAN
Symptom:
WebUI login fails on all replicas with "Login failed due to an unknown reason." Root cause traces to ipaserver/rpcserver.py's attempt_kinit/kinit_armor calling kinit -n (anonymous PKINIT armor) as a subprocess, which fails:
kinit: Generic error (see e-text) while getting initial credentials
krb5kdc.log shows:
AS_REQ ... UNKNOWN_REASON: WELLKNOWN/ANONYMOUS(a)FX.LAN for krbtgt/FX.LAN(a)FX.LAN, Generic error (see e-text)
Packet capture confirms the KDC rejects the anonymous AS-REQ on the very first round trip, before any PKINIT certificate exchange begins.
Real password-based kinit, SSH, and the ipa CLI all work perfectly, on every server, throughout. Only this specific anonymous-armor step (used by both the WebUI's password login form and its Kerberos/SPNEGO SSO login path — we tested both) is affected.
What we've ruled out, with evidence:
- PKINIT certs/EKUs/subject — a real but unrelated cert-subject bug was found and fixed along the way; confirmed unrelated via the packet capture above.
- LDAP data/replication — WELLKNOWN/ANONYMOUS entry unmodified since 2017; every LDAP search for it during actual failures returns err=0 success (confirmed via 389-DS access logs correlated to the exact failure timestamps).
- ipa-kdb key-stripping (the class of bug in Pagure #9065) — kadmin.local getprinc WELLKNOWN/ANONYMOUS confirms 2 valid keys present, identical on all servers.
- Local 389-DS index corruption (-30993) on one server — real, but confirmed unrelated (wrong day, and other clean-database servers still fail identically).
- Topology consistency — ipa topologysuffix-verify domain/ca both clean.
- A replica (idmserver-ha) had been left mid-removal (DNS SRV records stripped, server-del never completed) around the time this started — we completed the removal cleanly, no change to the symptom. This rules out what had been our leading hypothesis.
- SPAKE preauth config — production kdc.conf correctly configured.
One static-analysis lead, unconfirmed: in MIT krb5's do_as_req.c, UNKNOWN_REASON is a generic fallback (if (errcode != 0 && state->status == NULL) state->status = "UNKNOWN_REASON";). The anonymous-request handling block contains one status-less error path:
errcode = krb5_copy_principal(context, krb5_anonymous_principal(),
&state->request->client);
if (errcode)
goto errout;
This matches the anonymous-only symptom, but it's a pure in-memory copy of a compile-time constant with no external dependency — we can't explain why it would suddenly start failing, and we haven't been able to confirm live whether this is even the actual branch being hit.
Where we're stuck: we can't get a live gdb trace to confirm/deny the above — our installed krb5-server-1.18.2-34.0.1.el8_10 has an Oracle-specific patch revision (.0.1) with no matching debuginfo available anywhere we could find (checked Oracle's own debuginfo repo and cross-referenced AlmaLinux, which ships a different build 1.18.2-34.el8_10 without that suffix — not a build-id match).
Has anyone seen this exact signature before, or have suggestions for getting further without matching debug symbols? Happy to provide more logs/detail on request.
Thanks,