On Fri, Apr 07, 2023 at 02:02:34PM +0800, Coiby Xu wrote:
On Fri, Apr 07, 2023 at 01:53:54PM +0800, Coiby Xu wrote:
>Hi Philipp,
>
>This patch fails the nfs-early-kdump test (Fedora 36 cloud image). Since
>"[PATCH v2 5/8] kdump-lib: remove get_nmcli_connection_apath_by_ifname"
>is still good, this is the first bad commit.
>
>I'm yet to figure out how it happens. The following logs may point to a
>clue,
>
> [ 1.469868] dracut-cmdline[252]: Using kernel command line parameters:
rd.driver.pre=btrfs BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.17.5-300.fc36.x86_64
root=UUID=737459fa-eab9-4d8f-9a14-c8c32d403c08 ro rootflags=subvol=root no_timer_check
net.ifnames=0 console=tty1 console=ttyS0,115200n8 systemd.journald.forward_to_console=1
selinux=0 crashkernel=224M test_boot_count=2 rd.earlykdump
> [ 1.484797] dracut-cmdline[252]: /lib/kdump-lib.sh: line 11:
/usr/lib/dracut/dracut-functions.sh: No such file or directory
Another clue is the system somehow doesn't even kexec into the kdump kernel,
[ 45.196201] dracut[3665]: *** Creating initramfs image file
'/boot/initramfs-5.17.5-300.fc36.x86_64.img' done ***
[ 45.588729] init.sh[10924]: kdump: kexec: unloaded kdump kernel
[ 45.588882] init.sh[10924]: kdump: Stopping kdump: [OK]
[ 46.771859] init.sh[10924]: kdump: kexec: loaded kdump kernel
[ 46.771989] init.sh[10924]: kdump: Starting kdump: [OK]
[ 46.925844] systemd[1]: Removed slice system-modprobe.slice - Slice
/system/modprobe.
[[0;32m OK [0m] Removed slice [0;1;39msystem-modpr…lice[0m - Slice
/system/modprobe.
[[0;32m OK [0m] Removed slice [0;1;39msystem-sshd\…e[0m - Slice
/system/sshd-keygen.
[[0;32m OK [0m] Stopped target [0;1;39mmulti-user.target[0m - Multi-User
System.
[[0;32m OK [0m] Stopped target [0;1;39mgetty.target[0m - Login Prompts.
[[0;32m OK [0m] Stopped target [0;1;39mnetwork-online.target[0m - Network is
Online.
[[0;32m OK [0m] Stopped target [0;1;39mnss-lookup.…m - Host and Network Name
Lookups.
[[0;32m OK [0m] Stopped target [0;1;39mrpc_pipefs.target[0m.
[[0;32m OK [0m] Stopped target [0;1;39mtimers.target[0m - Timer Units.
[[0;32m OK [0m] Stopped [0;1;39mdnf-makecache.timer[0m - dnf makecache --timer.
[[0;32m OK [0m] Stopped [0;1;39mfstrim.timer[0m - Discard unused blocks once a
week.
[[0;32m OK [0m] Stopped [0;1;39mraid-check.timer[… - Weekly RAID setup health
check.
[[0;32m OK [0m] Stopped [0;1;39msystemd-tmpfiles-c… Cleanup of Temporary
Directories.
[[0;32m OK [0m] Stopped [0;1;39munbound-anchor.tim… the root trust anchor for
DNSSEC.
[ 46.932655] systemd[1]: Removed slice system-sshd\x2dkeygen.slice - Slice
/system/sshd-keygen.
[ 46.959855] systemd[1]: Stopped target multi-user.target - Multi-User System.
[ 46.961866] systemd[1]: Stopped target getty.target - Login Prompts.
[[0;32m OK [0m] Closed [0;1;39msystemd-coredump.so…et[0m - Process Core Dump
Socket.
[ 46.965732] systemd[1]: Stopped target network-online.target - Network is Online.
Unmounting [0;1;39mvar-lib-nfs-rpc…ount[0m - RPC Pipe File System...
...
By comparison, the console logs for nfs-dump are as follows,
[ 18.709459] dracut[1021]: *** Creating initramfs image file
'/boot/initramfs-5.17.5-300.fc36.x86_64kdump.img' done ***
[ 19.356912] init.sh[865]: kdump: kexec: loaded kdump kernel
[ 19.358967] init.sh[865]: kdump: Starting kdump: [OK]
[ 19.553788] sysrq: Trigger a crash
[ 19.555006] Kernel panic - not syncing: sysrq triggered crash
[ 19.555968] CPU: 1 PID: 672 Comm: sh Kdump: loaded Not tainted
5.17.5-300.fc36.x86_64 #1
[ 19.555968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.16.1-2.fc36 04/01/2014
[ 19.555968] Call Trace:
[ 19.555968] <TASK>
[ 19.555968] dump_stack_lvl+0x47/0x5d
[ 19.555968] panic+0xeb/0x2db
[ 19.555968] sysrq_handle_crash+0x16/0x20
[ 19.555968] __handle_sysrq.cold+0x44/0x116
[ 19.555968] write_sysrq_trigger+0x24/0x40
[ 19.555968] proc_reg_write+0x56/0xa0
[ 19.555968] ? preempt_count_add+0x44/0x90
[ 19.555968] vfs_write+0xb9/0x2a0
[ 19.555968] ksys_write+0x53/0xd0
[ 19.555968] do_syscall_64+0x3a/0x80
[ 19.555968] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 19.555968] RIP: 0033:0x7f94d24f2f17
[ 19.555968] Code: 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e
fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77
51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[ 19.555968] RSP: 002b:00007fff6c5c8b68 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 19.555968] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f94d24f2f17
[ 19.555968] RDX: 0000000000000002 RSI: 000056529f9ed730 RDI: 0000000000000001
[ 19.555968] RBP: 000056529f9ed730 R08: 0000000000000001 R09: 0000000000000073
[ 19.555968] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002
[ 19.555968] R13: 00007f94d25e55a0 R14: 0000000000000002 R15: 00007f94d25e5780
[ 19.555968] </TASK>
[ 0.000000] Linux version 5.17.5-300.fc36.x86_64
(mockbuild(a)bkernel01.iad2.fedoraproject.org) (gcc (GCC) 12.0.1 20220413 (Red Hat
12.0.1-0), GNU ld version 2.37-24.fc36) #1 SMP PREEMPT Thu Apr 28 15:51:30 UTC 2022
Sorry, the above info is misleading as I almost forgot how early-kdump
works. Please ignore it.
I should have compared the logs of a failed early-dump with the logs of a successful
one which shows somehow this patch makes rd.earlykdump have no effect,
# console logs when early-kdump failed
[ 1.460774] systemd[1]: Starting dracut-cmdline.service - dracut cmdline hook...
Starting [0;1;39mdracut-cmdline.service[0m - dracut cmdline hook...
[ 1.468153] dracut-cmdline[252]: dracut-36 (Cloud Edition) dracut-057-3.fc36
[ 1.469868] dracut-cmdline[252]: Using kernel command line parameters:
rd.driver.pre=btrfs BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.17.5-300.fc36.x86_64
root=UUID=737459fa-eab9-4d8f-9a14-c8c32d403c08 ro rootflags=subvol=root no_timer_check
net.ifnames=0 console=tty1 console=ttyS0,115200n8 systemd.journald.forward_to_console=1
selinux=0 crashkernel=224M test_boot_count=2 rd.earlykdump
[ 1.484797] dracut-cmdline[252]: /lib/kdump-lib.sh: line 11:
/usr/lib/dracut/dracut-functions.sh: No such file or directory
[[0;32m OK [0m] Finished [0;1;39mdracut-cmdline.service[0m - dracut cmdline
hook.
# console logs when early-kdump worked
[ 2.100520] systemd[1]: Starting dracut-cmdline.service - dracut cmdline hook...
[ 2.109902] dracut-cmdline[253]: dracut-36 (Cloud Edition) dracut-057-3.fc36
[ 2.114036] dracut-cmdline[253]: Using kernel command line parameters:
rd.driver.pre=btrfs BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.17.5-300.fc36.x86_64
root=UUID=737459fa-eab9-4d8f-9a14-c8c32d403c08 ro rootflags=subvol=root no_timer_check
net.ifnames=0 console=tty1 console=ttyS0,115200n8 systemd.journald.forward_to_console=1
selinux=0 crashkernel=224M test_boot_count=2 rd.earlykdump
[ 2.144734] dracut-cmdline[253]: kdump: early-kdump is enabled.
[ 2.170998] dracut-cmdline[337]: /lib/kdump-lib.sh: line 27: lscpu: command not
found
[ 2.743914] dracut-cmdline[253]: kdump: kexec: loaded early-kdump kernel
[ 2.881411] sysrq: Trigger a crash
[ 2.882604] Kernel panic - not syncing: sysrq triggered crash
[ 2.883581] CPU: 1 PID: 253 Comm: dracut-cmdline Kdump: loaded Not tainted
5.17.5-300.fc36.x86_64 #1
[ 2.883581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.16.1-2.fc36 04/01/2014
[ 2.883581] Call Trace:
[ 2.883581] <TASK>
[ 2.883581] dump_stack_lvl+0x47/0x5d
[ 2.883581] panic+0xeb/0x2db
[ 2.883581] sysrq_handle_crash+0x16/0x20
[ 2.883581] __handle_sysrq.cold+0x44/0x116
[ 2.883581] write_sysrq_trigger+0x24/0x40
[ 2.883581] proc_reg_write+0x56/0xa0
[ 2.883581] ? preempt_count_add+0x44/0x90
[ 2.883581] vfs_write+0xb9/0x2a0
[ 2.883581] ksys_write+0x53/0xd0
...
[ 2.883581] </TASK>
[ 0.000000] Linux version 5.17.5-300.fc36.x86_64
(mockbuild(a)bkernel01.iad2.fedoraproject.org) (gcc (GCC) 12.0.1 20220413 (Red Hat
12.0.1-0), GNU ld version 2.37-24.fc36) #1 SMP PREEMPT Thu Apr 28 15:51:30 UTC 2022
--
Best regards,
Coiby