On la, 05 helmi 2022, Brian J. Murrell via FreeIPA-users wrote:
I am still trying to debug why the webUI on my new replica is not authenticating me.
One difference I have noticed between my two replicas, one working and one not is:
working replica:
# KRB5RCACHEDIR=/var/lib/gssproxy/rcache klist Ticket cache: KCM:0 Default principal: admin@EXAMPLE.COM
Valid starting Expires Service principal 2022-02-04 08:58:59 2022-02-05 08:58:56 krbtgt/EXAMPLE.COM@EXAMPLE.COM 2022-02-04 08:59:03 2022-02-05 08:58:56 HTTP/server-staging.example.com@EXAMPLE.COM
Non-working replica:
# KRB5RCACHEDIR=/var/lib/gssproxy/rcache klist Ticket cache: KEYRING:persistent:0:krb_ccache_AunlIbq Default principal: host/server.example.com@EXAMPLE.COM
Valid starting Expires Service principal 1969-12-31 19:00:00 1969-12-31 19:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
What could cause the latter to not be getting any tickets like the former is?
The above is unrelated completely to IPA server operation.
KRB5RCACHEDIR has nothing to do with credentials caches visible with klist, so specifying that one makes no sense. In both cases you have klist looking into root user's credential cache and the difference there is by what your OS defaults are:
- in one case you have KCM as a credential cache collection - in another case you have kernel keyring as a credential cache collection
FreeIPA server code does not rely on your root user credential caches at all. IPA API in httpd is launched with a special configuration that uses ccaches in pre-defined locations to isolate it from the system-wide settings.
Below are corresponding snippets for 389-ds instance and for httpd:
$ cat install/share/ds-ipa-env.conf.template # Installed and maintained by ipa update tools, please do not modify
[Service] Environment=LC_ALL=C.UTF-8 Environment=KRB5_KTNAME=$KRB5_KTNAME Environment=KRB5_CLIENT_KTNAME=$KRB5_KTNAME Environment=KRB5CCNAME=$KRB5CCNAME
$ cat install/share/ipa-httpd.conf.template # Do not edit. Created by IPA installer.
[Service] Environment=KRB5CCNAME=$KRB5CC_HTTPD Environment=GSS_USE_PROXY=yes Environment=KDCPROXY_CONFIG=$KDCPROXY_CONFIG Environment=LC_ALL=C.UTF-8 ExecStartPre=$IPA_HTTPD_KDCPROXY
Where KRB5CC_HTTPD is set to '/tmp/krb5cc-httpd' within httpd process namespace which uses private tmpfs by systemd and for directory server's instance it is set to '/tmp/krb5cc_<dirsrv_uid>', again, within httpd process namespace which also uses private tmpfs by systemd.
Both of these isolated from what you are showing. And they also not what used to store credentials your gssproxy log attempts to find. ;)
Before I proceed with more explanation, when you see something like below in a ccache with klist:
Valid starting Expires Service principal 1969-12-31 19:00:00 1969-12-31 19:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
it means GSSProxy stored its data in this ccache in encrypted form. Only GSSProxy daemon can decrypt its content.
Now to your logs:
[CID 15][2022/02/05 16:08:59]: [status] Executing request 6 (GSSX_ACQUIRE_CRED) from [0x7fdf7c08a880 (176)] [CID 15][2022/02/05 16:08:59]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "ipa-api", euid: 977,socket: (null) GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: <Null> add_cred: 0 desired_name: { "brian@EXAMPLE.COM" { 1 2 840 113554 1 2 2 1 } [ ] [ ] [ ] } time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) GSSX_RES_ACQUIRE_CRED( status: { 851968 { 1 2 840 113554 1 2 2 } 2529639107 "Unspecified GSS failure. Minor code may provide more information" "No credentials cache found" [ ] } output_cred_handle: <Null> )
In IPA API configuration in httpd we have an arrangement where each user connecting to IPA API endpoint would have a separate ccache stored by httpd daemon with credentials encrypted by GSSProxy. This configuration is enforced through mod_auth_gssapi:
# Protect /ipa and everything below it in webspace with Apache Kerberos auth <Location "/ipa"> AuthType GSSAPI AuthName "Kerberos Login" GssapiUseSessions On Session On SessionCookieName ipa_session path=/ipa;httponly;secure; SessionHeader IPASESSION # Uncomment the following to have shorter sessions, but beware this may break # old IPA client tols that incorrectly parse cookies. # SessionMaxAge 1800 GssapiSessionKey file:$GSSAPI_SESSION_KEY
GssapiImpersonate On GssapiDelegCcacheDir $IPA_CCACHES GssapiDelegCcachePerms mode:0660 GssapiDelegCcacheUnique On GssapiUseS4U2Proxy on GssapiAllowedMech krb5 Require valid-user ErrorDocument 401 /ipa/errors/unauthorized.html Header always append X-Frame-Options DENY Header always append Content-Security-Policy "frame-ancestors 'none'"
# mod_session always sets two copies of the cookie, and this confuses our # legacy clients, the unset here works because it ends up unsetting only one # of the 2 header tables set by mod_session, leaving the other intact Header unset Set-Cookie
# Disable etag http header. Doesn't work well with mod_deflate # https://issues.apache.org/bugzilla/show_bug.cgi?id=45023 # Usage of last-modified header and modified-since validator is sufficient. Header unset ETag FileETag None </Location>
In the configuration above mod_auth_gssapi is responsible to generate ipa_session cookie after a user has been authenticated. Since we rely on Kerberos extension (S4U2Proxy) to communicate further with LDAP and other services on behalf of the user, GSSAPI credentials delegation feature of mod_auth_gssapi is used to create credential caches for each user in $IPA_CCACHES location (/run/ipa/ccaches). In RHEL 8.4+ we set 'GssapiDelegCcacheUnique On', to have unique credential caches for each user's session, not just per-user (see https://pagure.io/freeipa/issue/8589). The stale ccaches then removed by a script that is run periodically as a systemd timer (ipa-ccache-sweep.timer which runs ipa-ccache-sweep.service).
When user connecting to IPA API presents valid ipa_session cookie, its content is unpacked by mod_auth_gssapi to /run/ipa/ccaches/<file> to represent the ccache with that user's tickets. The cookie itself is encrypted by mod_auth_gssapi. The content of the ccache is encrypted by gssproxy.
Basically, this means we have following setup:
- httpd with mod_auth_gssapi runs under 'apache' user - IPA API in httpd with mod_wsgi runs under 'ipaapi' user - GSSProxy manages content of the credential caches accessed from both processes.
--------------- [root@dc ~]# ps axufw|egrep '(httpd|ipaapi)' root 5999 0.0 0.0 6412 2100 pts/0 S+ 10:26 0:00 _ grep -E --color=auto (httpd|ipaapi) root 888 0.0 0.5 26608 16532 ? Ss Feb04 0:07 /usr/sbin/httpd -DFOREGROUND apache 4698 0.0 0.2 38580 8320 ? S 00:00 0:00 _ /usr/sbin/httpd -DFOREGROUND ipaapi 4701 0.0 2.8 207128 84876 ? Sl 00:00 0:03 _ (wsgi:ipa) -DFOREGROUND ipaapi 4702 0.0 2.8 272696 86692 ? Sl 00:00 0:03 _ (wsgi:ipa) -DFOREGROUND ipaapi 4703 0.0 3.0 274744 90128 ? Sl 00:00 0:03 _ (wsgi:ipa) -DFOREGROUND ipaapi 4704 0.0 2.8 272664 84876 ? Sl 00:00 0:03 _ (wsgi:ipa) -DFOREGROUND apache 4705 0.0 0.5 1591628 15912 ? Sl 00:00 0:05 _ /usr/sbin/httpd -DFOREGROUND apache 4706 0.0 0.6 1460492 20444 ? Sl 00:00 0:05 _ /usr/sbin/httpd -DFOREGROUND apache 4707 0.0 0.6 1460488 19756 ? Sl 00:00 0:04 _ /usr/sbin/httpd -DFOREGROUND apache 5909 0.0 0.5 1460488 15908 ? Sl 10:20 0:00 _ /usr/sbin/httpd -DFOREGROUND ---------------
It means ccache generated by GSSAPI operations performed by either mod_auth_gssapi or IPA API is written under corresponding user ('apache' or 'ipaapi') and need to be readable by either process to allow gssproxy interposer to send this data back and forth to gssproxy daemon. To do so, we rely on POSIX ACLs set on /run/ipa/ccaches that allow apache group to read/write to the files in /run/ipa/ccaches and force ownership to ipaapi:ipaapi user and group.
Here is an example: --------------------- [root@dc ~]# ls -la /run/ipa/ccaches/ total 0 drwsrws---+ 2 ipaapi ipaapi 40 Feb 4 11:11 . drwx--x--x. 3 root root 100 Feb 4 11:12 ..
[root@dc ~]# getfacl /run/ipa/ccaches/ getfacl: Removing leading '/' from absolute path names # file: run/ipa/ccaches/ # owner: ipaapi # group: ipaapi # flags: ss- user::rwx group::rwx group:apache:rwx mask::rwx other::---
[root@dc ~]# kinit admin Password for admin@IPA.TEST:
[root@dc ~]# ipa ping ------------------------------------------------------------------------- IPA server version 4.10.0.dev202202031222+git015d5ff89. API version 2.246 -------------------------------------------------------------------------
[root@dc ~]# ls -la /run/ipa/ccaches/ total 12 drwsrws---+ 2 ipaapi ipaapi 60 Feb 6 10:20 . drwx--x--x. 3 root root 100 Feb 4 11:12 .. -rw-------. 1 ipaapi ipaapi 8942 Feb 6 10:20 admin@IPA.TEST-GEIcn3 -----------------------
If you look with klist into that ccache, you'll only see encrypted gssproxy object data:
------------------------ [root@dc ~]# klist -c /run/ipa/ccaches/admin@IPA.TEST-GEIcn3 Ticket cache: FILE:/run/ipa/ccaches/admin@IPA.TEST-GEIcn3 Default principal: admin@IPA.TEST
Valid starting Expires Service principal 010170 00:00:00 010170 00:00:00 Encrypted/Credentials/v1@X-GSSPROXY: ------------------------
And that is correct. When ipaapi or apache user attempts to access the ccache through GSSAPI, it will get the request forwarded to GSSProxy via special gssproxy interposer plugin in GSSAPI. GSSProxy daemon will decide whether this request is correct and will process it, transparent to the caller.
What is important here, is that the original requester (one of httpd daemon processes) still has to have read/write access to the specific ccache in /run/ipa/ccaches that represents current user.
So if you have problems like in the log below, it means one of the possible issues:
- /run/ipa/ccaches has wrong setup, not corresponding to what is shown above
- /run/ipa/ccaches/user@REALM-suffix file is not accesible to httpd processes, or damaged, making its content not decryptable by GSSProxy
- ipa_session cookie generated by mod_auth_gssapi earlier and received by httpd process now is encrypted by a different mod_auth_gssapi session key, rendering its content damaged with regards to what Kerberos 5 library expects for a ccache
- GSSProxy session key is different to what was used to encrypt ccaches in /run/ipa/ccaches. This happens on reboot, for example.
The difference between RHEL 8.3 or later and RHEL 8.4 should be in the name of /run/ipa/ccaches/<ccache> file. Pre-RHEL 8.4 versions used single file to store session per user, making impossible parallel IPA API calls under the same user credentials from different machines.
[CID 15][2022/02/05 16:08:59]: [status] Returned buffer 6 (GSSX_ACQUIRE_CRED) from [0x7fdf7c08a880 (176)]: [0x7fdf7c066260 (176)] [CID 15][2022/02/05 16:08:59]: [status] Handling query output: 0x7fdf7c066260 (176) [2022/02/05 16:08:59]: [status] Handling query reply: 0x7fdf7c066260 (176) [2022/02/05 16:08:59]: [status] Sending data: 0x7fdf7c066260 (176) [2022/02/05 16:08:59]: [status] Sending data [0x7fdf7c066260 (176)]: successful write of 176 [2022/02/05 16:08:59]: Client [2022/02/05 16:08:59]: (/usr/sbin/httpd) [2022/02/05 16:08:59]: connected (fd = 16)[2022/02/05 16:08:59]: (pid = 4268) (uid = 977) (gid = 973)[2022/02/05 16:08:59]: (context = system_u:system_r:httpd_t:s0)[2022/02/05 16:08:59]: [CID 16][2022/02/05 16:08:59]: [status] Handling query input: 0x7fdf7c066260 (176) [CID 16][2022/02/05 16:08:59]: Connection matched service ipa-api [CID 16][2022/02/05 16:08:59]: [status] Processing request [0x7fdf7c066260 (176)] [CID 16][2022/02/05 16:08:59]: [status] Executing request 6 (GSSX_ACQUIRE_CRED) from [0x7fdf7c066260 (176)] [CID 16][2022/02/05 16:08:59]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "ipa-api", euid: 977,socket: (null) GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: <Null> add_cred: 0 desired_name: { "brian@EXAMPLE.COM" { 1 2 840 113554 1 2 2 1 } [ ] [ ] [ ] } time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) GSSX_RES_ACQUIRE_CRED( status: { 851968 { 1 2 840 113554 1 2 2 } 2529639107 "Unspecified GSS failure. Minor code may provide more information" "No credentials cache found" [ ] } output_cred_handle: <Null> ) [CID 16][2022/02/05 16:08:59]: [status] Returned buffer 6 (GSSX_ACQUIRE_CRED) from [0x7fdf7c066260 (176)]: [0x7fdf7c05c9e0 (176)] [CID 16][2022/02/05 16:08:59]: [status] Handling query output: 0x7fdf7c05c9e0 (176) [2022/02/05 16:08:59]: [status] Handling query reply: 0x7fdf7c05c9e0 (176) [2022/02/05 16:08:59]: [status] Sending data: 0x7fdf7c05c9e0 (176) [2022/02/05 16:08:59]: [status] Sending data [0x7fdf7c05c9e0 (176)]: successful write of 176 [CID 16][2022/02/05 16:08:59]: [status] Handling query input: 0x7fdf7c05c9e0 (176) [CID 16][2022/02/05 16:08:59]: Connection matched service ipa-api [CID 16][2022/02/05 16:08:59]: [status] Processing request [0x7fdf7c05c9e0 (176)] [CID 16][2022/02/05 16:08:59]: [status] Executing request 6 (GSSX_ACQUIRE_CRED) from [0x7fdf7c05c9e0 (176)] [CID 16][2022/02/05 16:08:59]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "ipa-api", euid: 977,socket: (null) GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: <Null> add_cred: 0 desired_name: { "brian@EXAMPLE.COM" { 1 2 840 113554 1 2 2 1 } [ ] [ ] [ ] } time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 ) GSSX_RES_ACQUIRE_CRED( status: { 851968 { 1 2 840 113554 1 2 2 } 2529639107 "Unspecified GSS failure. Minor code may provide more information" "No credentials cache found" [ ] } output_cred_handle: <Null> ) [CID 16][2022/02/05 16:08:59]: [status] Returned buffer 6 (GSSX_ACQUIRE_CRED) from [0x7fdf7c05c9e0 (176)]: [0x7fdf7c03f690 (176)] [CID 16][2022/02/05 16:08:59]: [status] Handling query output: 0x7fdf7c03f690 (176) [2022/02/05 16:08:59]: [status] Handling query reply: 0x7fdf7c03f690 (176) [2022/02/05 16:08:59]: [status] Sending data: 0x7fdf7c03f690 (176) [2022/02/05 16:08:59]: [status] Sending data [0x7fdf7c03f690 (176)]: successful write of 176
Cheers, b.
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure