Kdump kernel might hang if it failed due to initqueue timeout, and failure_action is set to dump_to_rootfs.
That's because dracut initqueue will start it's own emergency handler and bypass kdump emergency handler. And recently dracut will enable initqueue more often so this will more likely to happen.
And we removed the force isolation of kdump error handler in commit a230634 to avoid failing dracut initqueue multiple times (which may cause failure loop in this case), so `systemctl start sysroot.mount` might also hang even if kdump error handler is called.
So two things are fixed in this series: 1. Use a emergency hook to start kdump error handler when initqueue failed. 2. Do a soft reset of pending jobs, so `dead` services won't block sysroot.mount.
Kairui Song (2): module-setup.sh: add kdump-error-handler to emergency hooks Cleanup dead systemd services before start sysroot.mount
dracut-module-setup.sh | 1 + kdump-lib-initramfs.sh | 2 ++ 2 files changed, 3 insertions(+)
Recent version of dracut will enable initqueue more often, and kdump kernel failed in initqueue, initqueue will start it's own error handler, which will block kdump erorr handler.
So insert a emergency hook to start kdump error handler and do kdump failure_actions.
Signed-off-by: Kairui Song kasong@redhat.com --- dracut-module-setup.sh | 1 + 1 file changed, 1 insertion(+)
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 8211d14..3ffb649 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -1054,6 +1054,7 @@ install() { cp "$moddir/kdump-emergency.target" "$initdir/$systemdsystemunitdir/emergency.target" # Also redirect dracut-emergency to kdump error handler ln_r "$systemdsystemunitdir/emergency.service" "$systemdsystemunitdir/dracut-emergency.service" + inst_hook emergency 00 "$moddir/kdump-error-handler.sh"
# Check for all the devices and if any device is iscsi, bring up iscsi # target. Ideally all this should be pushed into dracut iscsi module
Hi,
On Fri, Jul 02, 2021 at 03:52:06AM +0800, Kairui Song wrote:
Recent version of dracut will enable initqueue more often, and
What do you mean by "dracut will enable initqueue more often"? Isn't intiqueue always enabled by dracut?
kdump kernel failed in initqueue, initqueue will start it's own error handler, which will block kdump erorr handler.
According to https://bugzilla.redhat.com/show_bug.cgi?id=1972463, kdump error handler was started when initqueue failed because of timeout. What do you mean by "kdump kernel failed in initque"?
So insert a emergency hook to start kdump error handler and do kdump failure_actions.
Signed-off-by: Kairui Song kasong@redhat.com
dracut-module-setup.sh | 1 + 1 file changed, 1 insertion(+)
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 8211d14..3ffb649 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -1054,6 +1054,7 @@ install() { cp "$moddir/kdump-emergency.target" "$initdir/$systemdsystemunitdir/emergency.target" # Also redirect dracut-emergency to kdump error handler ln_r "$systemdsystemunitdir/emergency.service" "$systemdsystemunitdir/dracut-emergency.service"
inst_hook emergency 00 "$moddir/kdump-error-handler.sh"
# Check for all the devices and if any device is iscsi, bring up iscsi # target. Ideally all this should be pushed into dracut iscsi module
-- 2.31.1 _______________________________________________ kexec mailing list -- kexec@lists.fedoraproject.org To unsubscribe send an email to kexec-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Tue, Jul 6, 2021 at 5:11 PM Coiby Xu coxu@redhat.com wrote:
Hi,
On Fri, Jul 02, 2021 at 03:52:06AM +0800, Kairui Song wrote:
Recent version of dracut will enable initqueue more often, and
Hi Coiby
Dracut only enable initqueue when the target needs a initqueue, in RHEL9 dracut do enable initqueue for more targets, that's depend on dracut modules.
What do you mean by "dracut will enable initqueue more often"? Isn't intiqueue always enabled by dracut?
kdump kernel failed in initqueue, initqueue will start it's own error handler, which will block kdump erorr handler.
According to https://bugzilla.redhat.com/show_bug.cgi?id=1972463, kdump error handler was started when initqueue failed because of timeout. What do you mean by "kdump kernel failed in initque"?
Yes you are right, I guess I accidentally set rd.emergency to some other value so kdump emergency shell was skipped. I'll test again. Maybe only patch 2/2 is needed to fix this issue.
So insert a emergency hook to start kdump error handler and do kdump failure_actions.
Signed-off-by: Kairui Song kasong@redhat.com
dracut-module-setup.sh | 1 + 1 file changed, 1 insertion(+)
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 8211d14..3ffb649 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -1054,6 +1054,7 @@ install() { cp "$moddir/kdump-emergency.target" "$initdir/$systemdsystemunitdir/emergency.target" # Also redirect dracut-emergency to kdump error handler ln_r "$systemdsystemunitdir/emergency.service" "$systemdsystemunitdir/dracut-emergency.service"
inst_hook emergency 00 "$moddir/kdump-error-handler.sh"
# Check for all the devices and if any device is iscsi, bring up iscsi # target. Ideally all this should be pushed into dracut iscsi module
-- 2.31.1 _______________________________________________ kexec mailing list -- kexec@lists.fedoraproject.org To unsubscribe send an email to kexec-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- Best regards, Coiby
-- Best Regards, Kairui Song
On Wed, Jul 07, 2021 at 04:31:12PM +0800, Kairui Song wrote:
On Tue, Jul 6, 2021 at 5:11 PM Coiby Xu coxu@redhat.com wrote:
Hi,
On Fri, Jul 02, 2021 at 03:52:06AM +0800, Kairui Song wrote:
Recent version of dracut will enable initqueue more often, and
Hi Coiby
Dracut only enable initqueue when the target needs a initqueue, in RHEL9 dracut do enable initqueue for more targets, that's depend on dracut modules.
This is new for me, thanks for the explanation!
When kdump failed due to initqueue timeout, the sysroot.mount and other serivces could be in `starting` but `dead` status:
dev-disk-by\x2duuid-530830d1\x2df2c7\x2d4c9a\x2d9a82\x2d148609097521.device loaded inactive dead start <... snip ...> squash-root.mount loaded active mounted /squash/root squash.mount loaded active mounted /squash sysroot.mount loaded inactive dead start /sysroot <... snip ...> dracut-cmdline.service loaded active exited dracut cmdline hook dracut-initqueue.service loaded activating start start dracut initqueue hook dracut-mount.service loaded inactive dead start dracut mount hook
At this point calling `systemctl start sysroot.mount` will just hang. So call `systemctl cancel` here to cancel all pending jobs and have a clean start for mounting sysroot.
Signed-off-by: Kairui Song kasong@redhat.com --- kdump-lib-initramfs.sh | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index ca6a825..4cd18e4 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -227,6 +227,8 @@ dump_to_rootfs() systemctl start dracut-initqueue fi
+ dinfo "Clean up dead systemd services" + systemctl cancel dinfo "Waiting for rootfs mount, will timeout after 90 seconds" systemctl start sysroot.mount
On Fri, Jul 2, 2021 at 3:52 AM Kairui Song kasong@redhat.com wrote:
Kdump kernel might hang if it failed due to initqueue timeout, and failure_action is set to dump_to_rootfs.
That's because dracut initqueue will start it's own emergency handler and bypass kdump emergency handler. And recently dracut will enable initqueue more often so this will more likely to happen.
And we removed the force isolation of kdump error handler in commit a230634 to avoid failing dracut initqueue multiple times (which may cause failure loop in this case), so `systemctl start sysroot.mount` might also hang even if kdump error handler is called.
So two things are fixed in this series:
- Use a emergency hook to start kdump error handler when initqueue
failed. 2. Do a soft reset of pending jobs, so `dead` services won't block sysroot.mount.
Kairui Song (2): module-setup.sh: add kdump-error-handler to emergency hooks Cleanup dead systemd services before start sysroot.mount
dracut-module-setup.sh | 1 + kdump-lib-initramfs.sh | 2 ++ 2 files changed, 3 insertions(+)
-- 2.31.1
Nack this series, PATCH 1/2 is not needed.
-
Coiby Xu -
Kairui Song