Hi,
Trying to upgrade CentOS 7 tot CentOS 8, following the various hints on the internet. Executing this command fails
# dnf --releasever=8 --allowerasing --setopt=deltarpm=false distro-sync ... Running transaction check Error: transaction check vs depsolve: (ipa-selinux = 4.8.7-12.module_el8.3.0+514+e7703106 if selinux-policy-targeted) is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch rpmlib(RichDependencies) <= 4.12.0-1 is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'. You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue. The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'.
How can I solve this?
On f34, freeipa-server 4.9.3-2: Upon choosing any action using a logged-in UI that has been left idle for some hours, browsers lock a display 'internal server error' (at least on firefox) instead of a log-in page, or the desired page. No actions on the server side will clear it. The only work-around is to delete the browser cookies and cached data, and after navigating to the UI the login page appears normally.
Harry
Kees Bakker via FreeIPA-users wrote:
Hi,
Trying to upgrade CentOS 7 tot CentOS 8, following the various hints on the internet. Executing this command fails
# dnf --releasever=8 --allowerasing --setopt=deltarpm=false distro-sync ... Running transaction check Error: transaction check vs depsolve: (ipa-selinux = 4.8.7-12.module_el8.3.0+514+e7703106 if selinux-policy-targeted) is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch rpmlib(RichDependencies) <= 4.12.0-1 is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'. You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue. The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'.
How can I solve this?
We don't manage the centos repos so I don't know. RHEL uses the leapp utility for in-place upgrades but that isn't supported in centos: https://wiki.centos.org/FAQ/CentOS8#Is_there_a_way_to_upgrade_CentOS-7_to_Ce...
The packages aren't really all that relevant in any case as there is no "upgrade" when upgrading client packages. It just installs new bits.
There are some important differences between 7 and 8 that could affect the resulting upgrade, most importantly that 7 uses authconfig to configure PAM and 8 uses authselect. Even if you managed to install the new bits, manually use authselect to re-do the PAM config, the result will be a failure uninstall because authconfig isn't present to restore the original PAM config.
It is probably easier and safer to unconfigure the IPA client, uninstall the packages, upgrade, re-install then reconfigure. You can for example save the current keytab to re-enroll (see the ipa-client-install man page).
rob
On 10-05-2021 14:45, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
Hi,
Trying to upgrade CentOS 7 tot CentOS 8, following the various hints on the internet. Executing this command fails
# dnf --releasever=8 --allowerasing --setopt=deltarpm=false distro-sync ... Running transaction check Error: transaction check vs depsolve: (ipa-selinux = 4.8.7-12.module_el8.3.0+514+e7703106 if selinux-policy-targeted) is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch rpmlib(RichDependencies) <= 4.12.0-1 is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'. You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue. The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'.
How can I solve this?
We don't manage the centos repos so I don't know. RHEL uses the leapp utility for in-place upgrades but that isn't supported in centos: https://wiki.centos.org/FAQ/CentOS8#Is_there_a_way_to_upgrade_CentOS-7_to_Ce...
The packages aren't really all that relevant in any case as there is no "upgrade" when upgrading client packages. It just installs new bits.
There are some important differences between 7 and 8 that could affect the resulting upgrade, most importantly that 7 uses authconfig to configure PAM and 8 uses authselect. Even if you managed to install the new bits, manually use authselect to re-do the PAM config, the result will be a failure uninstall because authconfig isn't present to restore the original PAM config.
It is probably easier and safer to unconfigure the IPA client, uninstall the packages, upgrade, re-install then reconfigure. You can for example save the current keytab to re-enroll (see the ipa-client-install man page).
rob
Hmm. In this case it's a master that I want to upgrade. I have two installations (two distinct environments). One environment with three masters, and one environment with just one master. In the first environment, I guess your advise would be to disconnect the master, uninstall freeipa, upgrade (or reinstall) centos, install freeipa (master). That, I can do. However, that still leaves me with the problem in my second environment.
On 10-05-2021 15:06, Kees Bakker via FreeIPA-users wrote:
On 10-05-2021 14:45, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
Hi,
Trying to upgrade CentOS 7 tot CentOS 8, following the various hints on the internet. Executing this command fails
# dnf --releasever=8 --allowerasing --setopt=deltarpm=false distro-sync ... Running transaction check Error: transaction check vs depsolve: (ipa-selinux = 4.8.7-12.module_el8.3.0+514+e7703106 if selinux-policy-targeted) is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch rpmlib(RichDependencies) <= 4.12.0-1 is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'. You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue. The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'.
How can I solve this?
We don't manage the centos repos so I don't know. RHEL uses the leapp utility for in-place upgrades but that isn't supported in centos: https://wiki.centos.org/FAQ/CentOS8#Is_there_a_way_to_upgrade_CentOS-7_to_Ce...
The packages aren't really all that relevant in any case as there is no "upgrade" when upgrading client packages. It just installs new bits.
There are some important differences between 7 and 8 that could affect the resulting upgrade, most importantly that 7 uses authconfig to configure PAM and 8 uses authselect. Even if you managed to install the new bits, manually use authselect to re-do the PAM config, the result will be a failure uninstall because authconfig isn't present to restore the original PAM config.
It is probably easier and safer to unconfigure the IPA client, uninstall the packages, upgrade, re-install then reconfigure. You can for example save the current keytab to re-enroll (see the ipa-client-install man page).
rob
Hmm. In this case it's a master that I want to upgrade. I have two installations (two distinct environments). One environment with three masters, and one environment with just one master. In the first environment, I guess your advise would be to disconnect the master, uninstall freeipa, upgrade (or reinstall) centos, install freeipa (master). That, I can do. However, that still leaves me with the problem in my second environment.
I stumbled on these notes at forums.centos.org [1] "There is no 7->8 upgrade. One has to install fresh 8." "There is no officially supported pathway to upgrade CentOS 7 to 8." Wow. I can hardly believe what I'm reading.
[1] https://forums.centos.org/viewtopic.php?f=54&t=73160
On su, 09 touko 2021, Harry G. Coin via FreeIPA-users wrote:
On f34, freeipa-server 4.9.3-2: Upon choosing any action using a logged-in UI that has been left idle for some hours, browsers lock a display 'internal server error' (at least on firefox) instead of a log-in page, or the desired page. No actions on the server side will clear it. The only work-around is to delete the browser cookies and cached data, and after navigating to the UI the login page appears normally.
It looks like the cache entry for user authentication which is stored encrypted on the server side cannot be decrypted anymore. This might happen when you have rebooted a server between authenticating the user and its session's expiration. There are two keys here:
- mod_auth_gssapi uses ipasession.key (in /etc/httpd/alias in Fedora) for encrypting the cookie session
- GSS-Proxy uses own service keytab or an in-memory key to encrypt Kerberos credentials in a ccache generated and stored on the server which corresponds to the content stored in the cookie session
If you'd restart GSS-Proxy or reboot the system, the ccache generated and stored on the server side by GSS-Proxy would not be possible to decrypt in case an ephemeral in-memory key was used.
Can you enable IPA server-side debugging in case this happens to see if we can handle an error from mod_auth_gssapi better?
On ma, 10 touko 2021, Kees Bakker via FreeIPA-users wrote:
On 10-05-2021 14:45, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
Hi,
Trying to upgrade CentOS 7 tot CentOS 8, following the various hints on the internet. Executing this command fails
# dnf --releasever=8 --allowerasing --setopt=deltarpm=false distro-sync ... Running transaction check Error: transaction check vs depsolve: (ipa-selinux = 4.8.7-12.module_el8.3.0+514+e7703106 if selinux-policy-targeted) is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch rpmlib(RichDependencies) <= 4.12.0-1 is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'. You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue. The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'.
How can I solve this?
We don't manage the centos repos so I don't know. RHEL uses the leapp utility for in-place upgrades but that isn't supported in centos: https://wiki.centos.org/FAQ/CentOS8#Is_there_a_way_to_upgrade_CentOS-7_to_Ce...
The packages aren't really all that relevant in any case as there is no "upgrade" when upgrading client packages. It just installs new bits.
There are some important differences between 7 and 8 that could affect the resulting upgrade, most importantly that 7 uses authconfig to configure PAM and 8 uses authselect. Even if you managed to install the new bits, manually use authselect to re-do the PAM config, the result will be a failure uninstall because authconfig isn't present to restore the original PAM config.
It is probably easier and safer to unconfigure the IPA client, uninstall the packages, upgrade, re-install then reconfigure. You can for example save the current keytab to re-enroll (see the ipa-client-install man page).
rob
Hmm. In this case it's a master that I want to upgrade. I have two installations (two distinct environments). One environment with three masters, and one environment with just one master. In the first environment, I guess your advise would be to disconnect the master, uninstall freeipa, upgrade (or reinstall) centos, install freeipa (master). That, I can do. However, that still leaves me with the problem in my second environment.
Standard upgrade process across major versions of RHEL IdM is to deploy a replica on a new RHEL version, transition all services provided by the master-to-be-deleted to it, then cut off the old master and decommission it. It is described in the documentation quite well.
An upgrade from RHEL 7 to RHEL 8 inplace is not supported.
On 10-05-2021 15:35, Alexander Bokovoy wrote:
On ma, 10 touko 2021, Kees Bakker via FreeIPA-users wrote:
On 10-05-2021 14:45, Rob Crittenden wrote:
Kees Bakker via FreeIPA-users wrote:
Hi,
Trying to upgrade CentOS 7 tot CentOS 8, following the various hints on the internet. Executing this command fails
# dnf --releasever=8 --allowerasing --setopt=deltarpm=false distro-sync ... Running transaction check Error: transaction check vs depsolve: (ipa-selinux = 4.8.7-12.module_el8.3.0+514+e7703106 if selinux-policy-targeted) is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch rpmlib(RichDependencies) <= 4.12.0-1 is needed by ipa-common-4.8.7-12.module_el8.3.0+514+e7703106.noarch To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'. You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue. The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'.
How can I solve this?
We don't manage the centos repos so I don't know. RHEL uses the leapp utility for in-place upgrades but that isn't supported in centos: https://wiki.centos.org/FAQ/CentOS8#Is_there_a_way_to_upgrade_CentOS-7_to_Ce...
The packages aren't really all that relevant in any case as there is no "upgrade" when upgrading client packages. It just installs new bits.
There are some important differences between 7 and 8 that could affect the resulting upgrade, most importantly that 7 uses authconfig to configure PAM and 8 uses authselect. Even if you managed to install the new bits, manually use authselect to re-do the PAM config, the result will be a failure uninstall because authconfig isn't present to restore the original PAM config.
It is probably easier and safer to unconfigure the IPA client, uninstall the packages, upgrade, re-install then reconfigure. You can for example save the current keytab to re-enroll (see the ipa-client-install man page).
rob
Hmm. In this case it's a master that I want to upgrade. I have two installations (two distinct environments). One environment with three masters, and one environment with just one master. In the first environment, I guess your advise would be to disconnect the master, uninstall freeipa, upgrade (or reinstall) centos, install freeipa (master). That, I can do. However, that still leaves me with the problem in my second environment.
Standard upgrade process across major versions of RHEL IdM is to deploy a replica on a new RHEL version, transition all services provided by the master-to-be-deleted to it, then cut off the old master and decommission it. It is described in the documentation quite well.
An upgrade from RHEL 7 to RHEL 8 inplace is not supported.
-- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
Thanks for the confirmation, Alexander
On 5/10/21 8:31 AM, Alexander Bokovoy wrote:
On su, 09 touko 2021, Harry G. Coin via FreeIPA-users wrote:
On f34, freeipa-server 4.9.3-2: Upon choosing any action using a logged-in UI that has been left idle for some hours, browsers lock a display 'internal server error' (at least on firefox) instead of a log-in page, or the desired page. No actions on the server side will clear it. The only work-around is to delete the browser cookies and cached data, and after navigating to the UI the login page appears normally.
It looks like the cache entry for user authentication which is stored encrypted on the server side cannot be decrypted anymore. This might happen when you have rebooted a server between authenticating the user and its session's expiration. There are two keys here:
- mod_auth_gssapi uses ipasession.key (in /etc/httpd/alias in Fedora) for encrypting the cookie session
- GSS-Proxy uses own service keytab or an in-memory key to encrypt Kerberos credentials in a ccache generated and stored on the server which corresponds to the content stored in the cookie session
If you'd restart GSS-Proxy or reboot the system, the ccache generated and stored on the server side by GSS-Proxy would not be possible to decrypt in case an ephemeral in-memory key was used.
Can you enable IPA server-side debugging in case this happens to see if we can handle an error from mod_auth_gssapi better?
I think blocking access to the freeipa-ui until all users delete all browser cookies and cached data every time freeipa is rebooted is probably not a good thing.
In my test case, freeipa-server is in a VM using a virtio-fs as the boot/root filesystem. The virtio-fs folks haven't quite managed to solve SELinux problems when their file system is the root device, so for the time being I set SELinux to permissive. Serendipity might have generated a clue, as follows:
# cat /etc/ipa/server.conf
[global] certmonger_wait_timeout=1200 startup_timeout = 720 startup_traceback = True debug=True
# journalctl -f
...
May 10 09:33:40 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:45 registry1.1.quietfountain.com audit[676]: AVC avc: denied { lock } for pid=676 comm="gssproxy" path="/var/lib/ipa/gssproxy/http.keytab" dev="virtiofs" ino=745470 scontext=system_u:system_r:gssproxy_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 ...
It's quite repeatable, every refresh on the browser generates the AVC line above (amid the log being cluttered with 'GSSAPI client step 1'...
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 8:31 AM, Alexander Bokovoy wrote:
On su, 09 touko 2021, Harry G. Coin via FreeIPA-users wrote:
On f34, freeipa-server 4.9.3-2: Upon choosing any action using a logged-in UI that has been left idle for some hours, browsers lock a display 'internal server error' (at least on firefox) instead of a log-in page, or the desired page. No actions on the server side will clear it. The only work-around is to delete the browser cookies and cached data, and after navigating to the UI the login page appears normally.
It looks like the cache entry for user authentication which is stored encrypted on the server side cannot be decrypted anymore. This might happen when you have rebooted a server between authenticating the user and its session's expiration. There are two keys here:
- mod_auth_gssapi uses ipasession.key (in /etc/httpd/alias in Fedora) for encrypting the cookie session
- GSS-Proxy uses own service keytab or an in-memory key to encrypt Kerberos credentials in a ccache generated and stored on the server which corresponds to the content stored in the cookie session
If you'd restart GSS-Proxy or reboot the system, the ccache generated and stored on the server side by GSS-Proxy would not be possible to decrypt in case an ephemeral in-memory key was used.
Can you enable IPA server-side debugging in case this happens to see if we can handle an error from mod_auth_gssapi better?
I think blocking access to the freeipa-ui until all users delete all browser cookies and cached data every time freeipa is rebooted is probably not a good thing.
In my test case, freeipa-server is in a VM using a virtio-fs as the boot/root filesystem. The virtio-fs folks haven't quite managed to solve SELinux problems when their file system is the root device, so for the time being I set SELinux to permissive. Serendipity might have generated a clue, as follows:
# cat /etc/ipa/server.conf
[global] certmonger_wait_timeout=1200 startup_timeout = 720 startup_traceback = True debug=True
# journalctl -f
...
May 10 09:33:40 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:45 registry1.1.quietfountain.com audit[676]: AVC avc: denied { lock } for pid=676 comm="gssproxy" path="/var/lib/ipa/gssproxy/http.keytab" dev="virtiofs" ino=745470 scontext=system_u:system_r:gssproxy_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 ...
It's quite repeatable, every refresh on the browser generates the AVC line above (amid the log being cluttered with 'GSSAPI client step 1'...
Thanks, this is helpful. In your case /var/lib/ipa/gssproxy/http.keytab has unlabeled_t context and that access is obviously not allowed.
# ls -lZ /var/lib/ipa/gssproxy/ total 4 -rw-------. 1 apache apache unconfined_u:object_r:ipa_var_lib_t:s0 178 May 7 09:27 http.keytab
Does restorecon help to relable? Perhaps, this is due to installing in SELinux off?
On 5/10/21 9:55 AM, Alexander Bokovoy wrote:
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 8:31 AM, Alexander Bokovoy wrote:
On su, 09 touko 2021, Harry G. Coin via FreeIPA-users wrote:
On f34, freeipa-server 4.9.3-2: Upon choosing any action using a logged-in UI that has been left idle for some hours, browsers lock a display 'internal server error' (at least on firefox) instead of a log-in page, or the desired page. No actions on the server side will clear it. The only work-around is to delete the browser cookies and cached data, and after navigating to the UI the login page appears normally.
It looks like the cache entry for user authentication which is stored encrypted on the server side cannot be decrypted anymore. This might happen when you have rebooted a server between authenticating the user and its session's expiration. There are two keys here:
- mod_auth_gssapi uses ipasession.key (in /etc/httpd/alias in Fedora) for encrypting the cookie session
- GSS-Proxy uses own service keytab or an in-memory key to encrypt Kerberos credentials in a ccache generated and stored on the server which corresponds to the content stored in the cookie session
If you'd restart GSS-Proxy or reboot the system, the ccache generated and stored on the server side by GSS-Proxy would not be possible to decrypt in case an ephemeral in-memory key was used.
Can you enable IPA server-side debugging in case this happens to see if we can handle an error from mod_auth_gssapi better?
I think blocking access to the freeipa-ui until all users delete all browser cookies and cached data every time freeipa is rebooted is probably not a good thing.
In my test case, freeipa-server is in a VM using a virtio-fs as the boot/root filesystem. The virtio-fs folks haven't quite managed to solve SELinux problems when their file system is the root device, so for the time being I set SELinux to permissive. Serendipity might have generated a clue, as follows:
# cat /etc/ipa/server.conf
[global] certmonger_wait_timeout=1200 startup_timeout = 720 startup_traceback = True debug=True
# journalctl -f
...
May 10 09:33:40 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:45 registry1.1.quietfountain.com audit[676]: AVC avc: denied { lock } for pid=676 comm="gssproxy" path="/var/lib/ipa/gssproxy/http.keytab" dev="virtiofs" ino=745470 scontext=system_u:system_r:gssproxy_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 ...
It's quite repeatable, every refresh on the browser generates the AVC line above (amid the log being cluttered with 'GSSAPI client step 1'...
Thanks, this is helpful. In your case /var/lib/ipa/gssproxy/http.keytab has unlabeled_t context and that access is obviously not allowed.
# ls -lZ /var/lib/ipa/gssproxy/ total 4 -rw-------. 1 apache apache unconfined_u:object_r:ipa_var_lib_t:s0 178 May 7 09:27 http.keytab
Does restorecon help to relable? Perhaps, this is due to installing in SELinux off?
Because selinux is set to permissive, what would not be allowed if SElinux was set to enforcing appears in the log. However, everything is allowed owing to the permissive setting. So in this instance, SELinux didn't block anything, but generated a clue as to what gets attempted upon failed ui accesses.
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 9:55 AM, Alexander Bokovoy wrote:
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 8:31 AM, Alexander Bokovoy wrote:
On su, 09 touko 2021, Harry G. Coin via FreeIPA-users wrote:
On f34, freeipa-server 4.9.3-2: Upon choosing any action using a logged-in UI that has been left idle for some hours, browsers lock a display 'internal server error' (at least on firefox) instead of a log-in page, or the desired page. No actions on the server side will clear it. The only work-around is to delete the browser cookies and cached data, and after navigating to the UI the login page appears normally.
It looks like the cache entry for user authentication which is stored encrypted on the server side cannot be decrypted anymore. This might happen when you have rebooted a server between authenticating the user and its session's expiration. There are two keys here:
- mod_auth_gssapi uses ipasession.key (in /etc/httpd/alias in Fedora) for encrypting the cookie session
- GSS-Proxy uses own service keytab or an in-memory key to encrypt Kerberos credentials in a ccache generated and stored on the server which corresponds to the content stored in the cookie session
If you'd restart GSS-Proxy or reboot the system, the ccache generated and stored on the server side by GSS-Proxy would not be possible to decrypt in case an ephemeral in-memory key was used.
Can you enable IPA server-side debugging in case this happens to see if we can handle an error from mod_auth_gssapi better?
I think blocking access to the freeipa-ui until all users delete all browser cookies and cached data every time freeipa is rebooted is probably not a good thing.
In my test case, freeipa-server is in a VM using a virtio-fs as the boot/root filesystem. The virtio-fs folks haven't quite managed to solve SELinux problems when their file system is the root device, so for the time being I set SELinux to permissive. Serendipity might have generated a clue, as follows:
# cat /etc/ipa/server.conf
[global] certmonger_wait_timeout=1200 startup_timeout = 720 startup_traceback = True debug=True
# journalctl -f
...
May 10 09:33:40 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:45 registry1.1.quietfountain.com audit[676]: AVC avc: denied { lock } for pid=676 comm="gssproxy" path="/var/lib/ipa/gssproxy/http.keytab" dev="virtiofs" ino=745470 scontext=system_u:system_r:gssproxy_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 ...
It's quite repeatable, every refresh on the browser generates the AVC line above (amid the log being cluttered with 'GSSAPI client step 1'...
Thanks, this is helpful. In your case /var/lib/ipa/gssproxy/http.keytab has unlabeled_t context and that access is obviously not allowed.
# ls -lZ /var/lib/ipa/gssproxy/ total 4 -rw-------. 1 apache apache unconfined_u:object_r:ipa_var_lib_t:s0 178 May 7 09:27 http.keytab
Does restorecon help to relable? Perhaps, this is due to installing in SELinux off?
Because selinux is set to permissive, what would not be allowed if SElinux was set to enforcing appears in the log. However, everything is allowed owing to the permissive setting. So in this instance, SELinux didn't block anything, but generated a clue as to what gets attempted upon failed ui accesses.
Sure, I know that. What I am trying to understand is whether you can relabel the path that is located on virtio-fs mount. Since we have proper fcontext associated with /var/lib/ipa/gssproxy, it needs to be labeled to allow proper SELinux work:
# matchpathcon /var/lib/ipa/gssproxy/ /var/lib/ipa/gssproxy system_u:object_r:ipa_var_lib_t:s0
I suspect this is really a use of a file system that is not really compatible with SELinux here. We are not going to add a rule that would allow gssproxy_t context to access unlabeled_t files in a default IPA policy. Either your OS supports SELinux in all components you are using or you are not really benefitting from the SELinux enforcements.
On 5/10/21 10:30 AM, Alexander Bokovoy wrote:
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 9:55 AM, Alexander Bokovoy wrote:
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 8:31 AM, Alexander Bokovoy wrote:
On su, 09 touko 2021, Harry G. Coin via FreeIPA-users wrote:
On f34, freeipa-server 4.9.3-2: Upon choosing any action using a logged-in UI that has been left idle for some hours, browsers lock a display 'internal server error' (at least on firefox) instead of a log-in page, or the desired page. No actions on the server side will clear it. The only work-around is to delete the browser cookies and cached data, and after navigating to the UI the login page appears normally.
It looks like the cache entry for user authentication which is stored encrypted on the server side cannot be decrypted anymore. This might happen when you have rebooted a server between authenticating the user and its session's expiration. There are two keys here:
- mod_auth_gssapi uses ipasession.key (in /etc/httpd/alias in Fedora) for encrypting the cookie session
- GSS-Proxy uses own service keytab or an in-memory key to encrypt Kerberos credentials in a ccache generated and stored on the server which corresponds to the content stored in the cookie session
If you'd restart GSS-Proxy or reboot the system, the ccache generated and stored on the server side by GSS-Proxy would not be possible to decrypt in case an ephemeral in-memory key was used.
Can you enable IPA server-side debugging in case this happens to see if we can handle an error from mod_auth_gssapi better?
I think blocking access to the freeipa-ui until all users delete all browser cookies and cached data every time freeipa is rebooted is probably not a good thing.
In my test case, freeipa-server is in a VM using a virtio-fs as the boot/root filesystem. The virtio-fs folks haven't quite managed to solve SELinux problems when their file system is the root device, so for the time being I set SELinux to permissive. Serendipity might have generated a clue, as follows:
# cat /etc/ipa/server.conf
[global] certmonger_wait_timeout=1200 startup_timeout = 720 startup_traceback = True debug=True
# journalctl -f
...
May 10 09:33:40 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:45 registry1.1.quietfountain.com audit[676]: AVC avc: denied { lock } for pid=676 comm="gssproxy" path="/var/lib/ipa/gssproxy/http.keytab" dev="virtiofs" ino=745470 scontext=system_u:system_r:gssproxy_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 ...
It's quite repeatable, every refresh on the browser generates the AVC line above (amid the log being cluttered with 'GSSAPI client step 1'...
Thanks, this is helpful. In your case /var/lib/ipa/gssproxy/http.keytab has unlabeled_t context and that access is obviously not allowed.
# ls -lZ /var/lib/ipa/gssproxy/ total 4 -rw-------. 1 apache apache unconfined_u:object_r:ipa_var_lib_t:s0 178 May 7 09:27 http.keytab
Does restorecon help to relable? Perhaps, this is due to installing in SELinux off?
Because selinux is set to permissive, what would not be allowed if SElinux was set to enforcing appears in the log. However, everything is allowed owing to the permissive setting. So in this instance, SELinux didn't block anything, but generated a clue as to what gets attempted upon failed ui accesses.
Sure, I know that. What I am trying to understand is whether you can relabel the path that is located on virtio-fs mount. Since we have proper fcontext associated with /var/lib/ipa/gssproxy, it needs to be labeled to allow proper SELinux work:
# matchpathcon /var/lib/ipa/gssproxy/ /var/lib/ipa/gssproxy system_u:object_r:ipa_var_lib_t:s0
I suspect this is really a use of a file system that is not really compatible with SELinux here. We are not going to add a rule that would allow gssproxy_t context to access unlabeled_t files in a default IPA policy. Either your OS supports SELinux in all components you are using or you are not really benefitting from the SELinux enforcements.
OK, I'll try that. However, isn't the entire SELinux question unrelated to the main topic of concern here? Given that at most SELinux complains but doesn't obstruct anything in this instance, what can be done about forcing the deletion of cookies and cached data in order to permit UI usage/login?
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 10:30 AM, Alexander Bokovoy wrote:
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 9:55 AM, Alexander Bokovoy wrote:
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 8:31 AM, Alexander Bokovoy wrote:
On su, 09 touko 2021, Harry G. Coin via FreeIPA-users wrote: > On f34, freeipa-server 4.9.3-2: Upon choosing any action using a > logged-in UI that has been left idle for some hours, browsers lock a > display 'internal server error' (at least on firefox) instead of a > log-in page, or the desired page. No actions on the server side > will > clear it. The only work-around is to delete the browser cookies > and > cached data, and after navigating to the UI the login page appears > normally.
It looks like the cache entry for user authentication which is stored encrypted on the server side cannot be decrypted anymore. This might happen when you have rebooted a server between authenticating the user and its session's expiration. There are two keys here:
- mod_auth_gssapi uses ipasession.key (in /etc/httpd/alias in Fedora) for encrypting the cookie session
- GSS-Proxy uses own service keytab or an in-memory key to encrypt Kerberos credentials in a ccache generated and stored on the server which corresponds to the content stored in the cookie session
If you'd restart GSS-Proxy or reboot the system, the ccache generated and stored on the server side by GSS-Proxy would not be possible to decrypt in case an ephemeral in-memory key was used.
Can you enable IPA server-side debugging in case this happens to see if we can handle an error from mod_auth_gssapi better?
I think blocking access to the freeipa-ui until all users delete all browser cookies and cached data every time freeipa is rebooted is probably not a good thing.
In my test case, freeipa-server is in a VM using a virtio-fs as the boot/root filesystem. The virtio-fs folks haven't quite managed to solve SELinux problems when their file system is the root device, so for the time being I set SELinux to permissive. Serendipity might have generated a clue, as follows:
# cat /etc/ipa/server.conf
[global] certmonger_wait_timeout=1200 startup_timeout = 720 startup_traceback = True debug=True
# journalctl -f
...
May 10 09:33:40 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:45 registry1.1.quietfountain.com audit[676]: AVC avc: denied { lock } for pid=676 comm="gssproxy" path="/var/lib/ipa/gssproxy/http.keytab" dev="virtiofs" ino=745470 scontext=system_u:system_r:gssproxy_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 ...
It's quite repeatable, every refresh on the browser generates the AVC line above (amid the log being cluttered with 'GSSAPI client step 1'...
Thanks, this is helpful. In your case /var/lib/ipa/gssproxy/http.keytab has unlabeled_t context and that access is obviously not allowed.
# ls -lZ /var/lib/ipa/gssproxy/ total 4 -rw-------. 1 apache apache unconfined_u:object_r:ipa_var_lib_t:s0 178 May 7 09:27 http.keytab
Does restorecon help to relable? Perhaps, this is due to installing in SELinux off?
Because selinux is set to permissive, what would not be allowed if SElinux was set to enforcing appears in the log. However, everything is allowed owing to the permissive setting. So in this instance, SELinux didn't block anything, but generated a clue as to what gets attempted upon failed ui accesses.
Sure, I know that. What I am trying to understand is whether you can relabel the path that is located on virtio-fs mount. Since we have proper fcontext associated with /var/lib/ipa/gssproxy, it needs to be labeled to allow proper SELinux work:
# matchpathcon /var/lib/ipa/gssproxy/ /var/lib/ipa/gssproxy system_u:object_r:ipa_var_lib_t:s0
I suspect this is really a use of a file system that is not really compatible with SELinux here. We are not going to add a rule that would allow gssproxy_t context to access unlabeled_t files in a default IPA policy. Either your OS supports SELinux in all components you are using or you are not really benefitting from the SELinux enforcements.
OK, I'll try that. However, isn't the entire SELinux question unrelated to the main topic of concern here? Given that at most SELinux complains but doesn't obstruct anything in this instance, what can be done about forcing the deletion of cookies and cached data in order to permit UI usage/login?
If GSS-Proxy couldn't read the keytab it is supposed to use, it cannot use a stable key for encryption, thus failing to decrypt ccache material from the older sessions. The behavior is fully related.
Are you running this system in permissive from start or did you switch to permissive fater the problems started to occur? If the latter, then chances are that you already had client sessions encrypted with ephemeral keys, thus the problem when you have allowed GSS-Proxy to access the stable key by switching to permissive mode.
On 5/10/21 11:21 AM, Alexander Bokovoy wrote:
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 10:30 AM, Alexander Bokovoy wrote:
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 9:55 AM, Alexander Bokovoy wrote:
On ma, 10 touko 2021, Harry G. Coin wrote:
On 5/10/21 8:31 AM, Alexander Bokovoy wrote: > On su, 09 touko 2021, Harry G. Coin via FreeIPA-users wrote: >> On f34, freeipa-server 4.9.3-2: Upon choosing any action using a >> logged-in UI that has been left idle for some hours, browsers >> lock a >> display 'internal server error' (at least on firefox) instead of a >> log-in page, or the desired page. No actions on the server side >> will >> clear it. The only work-around is to delete the browser cookies >> and >> cached data, and after navigating to the UI the login page appears >> normally. > > It looks like the cache entry for user authentication which is > stored > encrypted on the server side cannot be decrypted anymore. This > might > happen when you have rebooted a server between authenticating the > user > and its session's expiration. There are two keys here: > > - mod_auth_gssapi uses ipasession.key (in /etc/httpd/alias in > Fedora) > for encrypting the cookie session > > - GSS-Proxy uses own service keytab or an in-memory key to encrypt > Kerberos credentials in a ccache generated and stored on the > server > which corresponds to the content stored in the cookie session > > If you'd restart GSS-Proxy or reboot the system, the ccache > generated > and stored on the server side by GSS-Proxy would not be possible to > decrypt in case an ephemeral in-memory key was used. > > Can you enable IPA server-side debugging in case this happens to > see if > we can handle an error from mod_auth_gssapi better? > I think blocking access to the freeipa-ui until all users delete all browser cookies and cached data every time freeipa is rebooted is probably not a good thing.
In my test case, freeipa-server is in a VM using a virtio-fs as the boot/root filesystem. The virtio-fs folks haven't quite managed to solve SELinux problems when their file system is the root device, so for the time being I set SELinux to permissive. Serendipity might have generated a clue, as follows:
# cat /etc/ipa/server.conf
[global] certmonger_wait_timeout=1200 startup_timeout = 720 startup_traceback = True debug=True
# journalctl -f
...
May 10 09:33:40 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:45 registry1.1.quietfountain.com audit[676]: AVC avc: denied { lock } for pid=676 comm="gssproxy" path="/var/lib/ipa/gssproxy/http.keytab" dev="virtiofs" ino=745470 scontext=system_u:system_r:gssproxy_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1
May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 May 10 09:33:46 registry1.1.quietfountain.com [1435]: GSSAPI client step 1 ...
It's quite repeatable, every refresh on the browser generates the AVC line above (amid the log being cluttered with 'GSSAPI client step 1'...
Thanks, this is helpful. In your case /var/lib/ipa/gssproxy/http.keytab has unlabeled_t context and that access is obviously not allowed.
# ls -lZ /var/lib/ipa/gssproxy/ total 4 -rw-------. 1 apache apache unconfined_u:object_r:ipa_var_lib_t:s0 178 May 7 09:27 http.keytab
Does restorecon help to relable? Perhaps, this is due to installing in SELinux off?
Because selinux is set to permissive, what would not be allowed if SElinux was set to enforcing appears in the log. However, everything is allowed owing to the permissive setting. So in this instance, SELinux didn't block anything, but generated a clue as to what gets attempted upon failed ui accesses.
Sure, I know that. What I am trying to understand is whether you can relabel the path that is located on virtio-fs mount. Since we have proper fcontext associated with /var/lib/ipa/gssproxy, it needs to be labeled to allow proper SELinux work:
# matchpathcon /var/lib/ipa/gssproxy/ /var/lib/ipa/gssproxy system_u:object_r:ipa_var_lib_t:s0
I suspect this is really a use of a file system that is not really compatible with SELinux here. We are not going to add a rule that would allow gssproxy_t context to access unlabeled_t files in a default IPA policy. Either your OS supports SELinux in all components you are using or you are not really benefitting from the SELinux enforcements.
OK, I'll try that. However, isn't the entire SELinux question unrelated to the main topic of concern here? Given that at most SELinux complains but doesn't obstruct anything in this instance, what can be done about forcing the deletion of cookies and cached data in order to permit UI usage/login?
If GSS-Proxy couldn't read the keytab it is supposed to use, it cannot use a stable key for encryption, thus failing to decrypt ccache material from the older sessions. The behavior is fully related.
Are you running this system in permissive from start or did you switch to permissive fater the problems started to occur? If the latter, then chances are that you already had client sessions encrypted with ephemeral keys, thus the problem when you have allowed GSS-Proxy to access the stable key by switching to permissive mode.
This was/is the installation sequence, on otherwise stock f34 workstation:
1) #dnf install <all the freeipa world>
2) #setenforce permissive
3) #sed s/^SELINUX=enforcing/SELINUX=permissive/ -i /etc/selinux/config
4) #ipa-server-install <options>
...
on a clean install on f34 of freeipa server with dns:
After enabling dnssec on a zone, to avoid thousands of lines appear in the logs like:
May 10 12:12:45 registry1.1.quietfountain.com named[11774]: File.cpp(94): Could not open the file (Permission denied): /var/lib/ipa/dnssec/tokens/2bf061ad-fdef-a969-a3bb-22d6252d136d/c6824fd2-b21e-95b1-bf02-f7acdfc228a2.object
which continue for many minutes every time named tries to reference the files.
To workaround the situation, I have to run
setfacl -R -m u:named:rwX /var/lib/ipa/dnssec/tokens
after dnssec is enabled on a zone.
It seems some freeipa dnssec permission fixer doesn't get them all or run often enough.
freeipa-users@lists.fedorahosted.org