kexec June 2021

kexec@lists.fedoraproject.org

5 participants
22 discussions

[PATCH] mkdumprd: allow spaces after 'path' config phrase when network dump
by Kazuhito Hagio 27 May '22

27 May '22

Without this patch, when there are two or more spaces after 'path' configuration phrase with ssh or nfs setting, SAVE_PATH is set to '/var/crash' in mkdumprd, and in most cases kdump service fails to start. ssh kdump(a)192.168.122.1 path /kdump ^^ This behavior would be too sensitive and different from the other configurations. With this patch, mkdumprd allows such spaces. Signed-off-by: Kazuhito Hagio <k-hagio(a)ab.jp.nec.com> --- mkdumprd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdumprd b/mkdumprd index a6f7fe8..aa0abfd 100644 --- a/mkdumprd +++ b/mkdumprd @@ -13,7 +13,7 @@ export IN_KDUMP=1 conf_file="/etc/kdump.conf" SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" -SAVE_PATH=$(grep ^path $conf_file| cut -d' ' -f2) +SAVE_PATH=$(awk '/^path/ {print $2}' $conf_file) [ -z "$SAVE_PATH" ] && SAVE_PATH=$DEFAULT_PATH # strip the duplicated "/" SAVE_PATH=$(echo $SAVE_PATH | tr -s /) -- 2.18.0

4 3

Re: [dm-crypt] Kdump with full-disk LUKS encryption
by Kairui Song 05 Nov '21

05 Nov '21

Hi, Thanks a lot, these info are very helpful. Better to keep it for debugging for now, and ask users to use it very carefully. On Tue, Apr 20, 2021 at 3:54 PM Milan Broz <gmazyland(a)gmail.com> wrote: > > Hi, > > TL;DR what you are trying to do is to actually reverse many security measures > we added. It is perhaps acceptable for debugging but hardly for real generic system. > > - using memory-hard function increases cost of dictionary and brute-force > attacks > You can always decrease amount of memory needed, but you should do it only > if you know that security margin is ok (like password is randomly generated > with enough entropy). > > - key is in keyring to remove possibility for normal userspace to receive > the key from kernel. Moreover, there is no need to retain kernel in keyring once > dm-crypt device is activated. (It is still in kernel memory but only in crypto > functions context). (Systemd also uses keyring to cache passphrase but that's > different thing.) > > You can still use old way for activation with --disable-keyring activation, > but then you disable this possibility. > > More comments below. > > On 19/04/2021 12:00, Kairui Song wrote: > > Hi all, > > > > I'm currently trying to add kdump support for systemd with full-disk > > LUKS encryption. vmcores contain sensitive data so they should also be > > protected, and network dumps sometimes are not available. So kdump has > > to open the LUKS encrypted device in the kdump environment. > > > > I'm using systemd/dracut, my work machine is running Fedora 34, and > > there are several problems I'm trying to solve: > > 1. Users have to input the password in the kdump kernel environment. > > But users often don't have shell access to the kdump environment. > > (headless server, graphic card not working after kexec, both are very > > common) > > 2. LUKS2 prefers Argon2 as the key derivation function, designed to > > use a lot of memory. kdump is expected to use a minimal amount of > > memory. Users will have to reserve a huge amount of memory for kdump > > to work (eg. 1G reserve for kdump with 4G total memory which is not > > reasonable). > > When I added Argon2 to LUKS2, I actually expected such issues. Despite > some people beats me that they cannot use arbitrary amount of memory, > we have some hard limits that were selected that it should work on most recent > systems. Maybe kdump can live with it. > > - maximum memory cost limit is 4GB, no LUKS2 device can use more for Argon2 > - we never use more than half of available physical memory > (measured on the host where the device was formatted) > - required amount of memory is visible in LUKS2 metadata (luksDump) > for the particular keyslot (Memory: the value is in kB) > - we use benchmark to calculate memory cost with prefered unlocking > time 2 seconds (again, on the device where LUKS was formatted) > Small systems (like RPi2) the uses much smaller acceptable values. > You can configure all costs (time, memory, threads) during format > or even set them to predefined values. > > I am sorry, but there is really no way around this - and the requeired > memory must be physical memory (otherwise it slows down extremely). > This is a feature, not a bug :-) > > > > To fix these problems, I tried to pass the master key to the second > > kernel directly via initramfs. Kdump will modify the initramfs in > > ramfs to include the key, kexec_load it, and never write to any actual > > back storage. This worked with old LUKS configurations. > > Well, passing volume key this way is quite insecure, but perhaps > acceptable for debugging. > > > > > But LUKS2/cryptsetup now stores the key in the kernel keyring by > > default. The key is accessible from userspace. > > If you are talking about volume key (not passsphrase), it is not > available from userspace. Only reference to it. But you can use > this reference to construct in-kernel dm-crypt device. > Please read https://gitlab.com/cryptsetup/cryptsetup/-/blob/master/docs/Keyring.txt > > > Users can enter the password to start kdump manually and then it will > > work, but usually people expect kdump service to start automatically. > > > > (WIP patch series: > > https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org…) > > > > I've several ideas about how to improve it but not sure which one is > > better, and if this is a good idea after all: > > 1. Simply introduce a config to let systemd-cryptsetup disable kernel > > keyring on setup, there is currently no such option. > > Well, that option could be useful anyway and we have support for it > in cryptsetup (--disable-keyring CLI option) and libcryptsetup, so why not. > Just this should not be a default option. > > This is should be patch for systemd-cryptsetup only as libcryptsetup supports it. > > ... > > > 2. If we can let the key stay in userspace for a little longer, eg. > > for systems booted with dracut/systemd, when > > systemd-cryptsetup(a)%s.service opens the crypt device, keep the key in > > dm-crypt. And later when services like kdump have finished loading, > > cryptsetup can refresh the device and store the key in the kernel > > keyring again. > > We invalidate volume key in keyring after libceyposetup operation > is finished (and kernel removes the reference once keyring garbage collection > is run). > > I can imagine to add some option to keep key inside keyring even after > call is finished, but as said above, this removes some security margin > we intentionally introduced here. I agree with your comments, thanks! These two approaches seem not a good idea now. > > ... > > Milan > How about plan 3 and 4? > 3. Let kdump use some custom helper/service to load all needed > resources in the early initrd boot stage, prior to > systemd-cryptsetup(a)%s.service. It will ask the password / parse the > keyfile and load kdump, then provide info for systemd-cryptsetup or > just do the setup. Or maybe let systemd-cryptsetup support some kind > of "plugins" so other tools can use it. Some details could be changed/improved, but systemd-cryptsetup(a)%s.service will prompt for a password or use a keyfile anyway. So I think at this point, loading kdump with the volume key should be safe? At least long as the kdump kernel/environment itself isn't compromised. Loaded kdump resources can be restricted to be only accessible from the kernel side. After panic, kernel kexec jumps to kdump kernel, and that's an minimized emergency environment that only lives for a very short period. > 4. A better and safer solution seems to keep a consistent key ring > between kexec boots but also more complex and difficult as different > arch implements kexec differently. Maybe plan 4 will be a good idea if doable? Since that keeps the key consistent in the kernel between kexec boots, and cryptsetup can just reuse it. -- Best Regards, Kairui Song

3 4

[PATCH v4] selftest: kill VM reliably by recursively kill children processes
by Coiby Xu 15 Oct '21

15 Oct '21

qemu is launched in nested subprocess and can't be killed by simply killing the job ids, PID Command 2269634 │ ├─ sshd: root [priv] 2269637 │ │ └─ sshd: root@pts/0 2269638 │ │ └─ -bash 2269744 │ │ └─ make test-run V=1 2273117 │ │ └─ /bin/bash /root/kexec-tools-300/tests/scripts/run-test.sh 2273712 │ │ ├─ /bin/bash /root/kexec-tools-300/tests/scripts/run-test.sh 2273714 │ │ │ └─ /bin/bash /root/kexec-tools-300/tests/scripts/run-test.sh 2273737 │ │ │ └─ timeout --foreground 10m /root/kexec-tools-300/tests/scripts/run-qemu -nodefaults -nographic -smp 2 -m 768M -monitor no 2273738 │ │ │ └─ /usr/bin/qemu-system-x86_64 -enable-kvm -cpu host -nodefaults -nographic -smp 2 -m 768M -monitor none -serial stdio 2273746 │ │ │ ├─ /usr/bin/qemu-system-x86_64 -enable-kvm -cpu host -nodefaults -nographic -smp 2 -m 768M -monitor none -serial std 2273797 │ │ ├─ /bin/bash /root/kexec-tools-300/tests/scripts/run-test.sh 2273798 │ │ │ └─ /bin/bash /root/kexec-tools-300/tests/scripts/run-test.sh 2273831 │ │ │ └─ timeout --foreground 10m /root/kexec-tools-300/tests/scripts/run-qemu -nodefaults -nographic -smp 2 -m 768M -monitor no 2273832 │ │ │ └─ /usr/bin/qemu-system-x86_64 -enable-kvm -cpu host -nodefaults -nographic -smp 2 -m 768M -monitor none -serial stdio 2273840 │ │ │ ├─ /usr/bin/qemu-system-x86_64 -enable-kvm -cpu host -nodefaults -nographic -smp 2 -m 768M -monitor none -serial std This led to the error "qemu-system-x86_64: can't bind ip=0.0.0.0 to socket: Address already in use". This patch will kill qemu by killing all the children of the job id. Signed-off-by: Coiby Xu <coxu(a)redhat.com> --- Thanks to Kairui for finding out an error and an issue about previous version, - "$_job" should be used instead "$job" - "Ctrl-c" leads to the failure of parsing pstree result This version stops parsing pstree and use pgrep to get the children of a process and kill them recursively instead. --- tests/scripts/run-test.sh | 37 +++++++++++++++++++++++++++++++++---- tests/scripts/test-lib.sh | 3 +-- 2 files changed, 34 insertions(+), 6 deletions(-) diff --git a/tests/scripts/run-test.sh b/tests/scripts/run-test.sh index a68504d..7113be8 100755 --- a/tests/scripts/run-test.sh +++ b/tests/scripts/run-test.sh @@ -1,9 +1,38 @@ #!/bin/bash -_kill_all_jobs() { - local _jobs=$(jobs -r -p) +_kill_if_valid_pid() { + local _pid="$1" + if ps -p $_pid > /dev/null + then + kill $_pid + fi +} + +_recursive_kill() { + local _pid="$1" + local _children _child + + _children=$(pgrep -P $_pid) + if [ -n "$_children" ]; then + for _child in $_children + do + _recursive_kill $_child + _kill_if_valid_pid $_child + done + fi + _kill_if_valid_pid $_pid +} - [ -n "$_jobs" ] && kill $_jobs +_kill_all_jobs() { + local _jobs=$(jobs -r -p) + local _job + + if [ -n "$_jobs" ]; then + for _job in $_jobs + do + _recursive_kill $_job + done + fi } trap ' @@ -121,7 +150,7 @@ for test_case in $testcases; do [ $? -ne 0 ] && ret=$(expr $ret + 1) results[$test_case]="$res" - + _kill_all_jobs echo -e "-------- Test finished: $test_case $res --------" for script in $scripts; do script="$testdir/$script" diff --git a/tests/scripts/test-lib.sh b/tests/scripts/test-lib.sh index f8a2249..8b24b2a 100644 --- a/tests/scripts/test-lib.sh +++ b/tests/scripts/test-lib.sh @@ -146,8 +146,7 @@ watch_test_outputs() { ret=$? if [ $ret -ne 255 ]; then - # Test finished, kill VMs - kill $(jobs -p) + # Test finished break 2 fi done -- 2.32.0

2 1

[PATCH v2 0/4] Add support for crashkernel.default
by Kairui Song 02 Jul '21

02 Jul '21

In RHEL, kernel supports a `crashkernel=auto` param and when it's used, kernel will use its built-in default value as the crashkernel value. But upstream doesn't like this idea. [1] `crashkernel=auto` was introduced to provide user a default value, and make it possible to update the crashkernel value automatically as kernel package updates if user is using the default `auto` value. After more discussion, `crashkernel=auto` will be dropped. To track the default crashkernel value, kernel package will include a crashkernel.default file with it[2], and kexec-tools can make use of that file, update kernel's boot cmdline accordingly, and help user to reset the crashkernel value to default value. [1]: https://lore.kernel.org/linux-mm/20210507010432.IN24PudKT%25akpm@linux-foun… [2]: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171 Update from V1: - Add support for ostree - Add a crashkernel-howto.txt - Change crashkernel.conf to crashkernel.default Kairui Song (4): Revert "kdump-lib.sh: Remove is_atomic" kdumpctl: Add kdumpctl reset-crashkernel Add a new hook: 92-crashkernel.install Add a crashkernel-howto.conf doc 92-crashkernel.install | 131 +++++++++++++++++++++++++++++++++++++++++ crashkernel-howto.conf | 113 +++++++++++++++++++++++++++++++++++ kdump-lib.sh | 5 ++ kdumpctl | 41 ++++++++++++- kdumpctl.8 | 9 +++ kexec-tools.spec | 7 +++ 6 files changed, 304 insertions(+), 2 deletions(-) create mode 100755 92-crashkernel.install create mode 100644 crashkernel-howto.conf -- 2.31.1

2 6

[PATCH] fadump-init: clean up mount points properly
by Kairui Song 30 Jun '21

30 Jun '21

When running with squash module enabled for both initramfs, /dev and /run are also mounted by squash-init, so move them to newroot as well, else they might leak. Also don't mount /sys since squash-init/systemd will mount it and it's not required here. Also pass `-d` to umount so loop devices (if used) will be force freed. Signed-off-by: Kairui Song <kasong(a)redhat.com> --- dracut-fadump-init-fadump.sh | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/dracut-fadump-init-fadump.sh b/dracut-fadump-init-fadump.sh index 5468d99..cc67d1c 100755 --- a/dracut-fadump-init-fadump.sh +++ b/dracut-fadump-init-fadump.sh @@ -5,9 +5,6 @@ export SYSTEMD_IN_INITRD=lenient [ -e /proc/mounts ] || (mkdir -p /proc && mount -t proc -o nosuid,noexec,nodev proc /proc) -grep -q '^sysfs /sys sysfs' /proc/mounts || - (mkdir -p /sys && mount -t sysfs -o nosuid,noexec,nodev sysfs /sys) - grep -q '^none / ' /proc/mounts || grep -q '^rootfs / ' /proc/mounts && ROOTFS_IS_RAMFS=1 if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [ -f /proc/device-tree/ibm,opal/dump/mpipl-boot ]; then @@ -20,9 +17,13 @@ if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [ -f /proc/device-tree/ibm,o done exec switch_root /newroot /init else - mkdir /newroot/sys /newroot/proc /newroot/oldroot + mkdir /newroot/sys /newroot/proc /newroot/dev /newroot/run /newroot/oldroot + + grep -q '^devtmpfs /dev devtmpfs' /proc/mounts && mount --move /dev /newroot/dev + grep -q '^tmpfs /run tmpfs' /proc/mounts && mount --move /run /newroot/run + grep -q '^sysfs /sys sysfs' /proc/mounts && mount --move /sys /newroot/sys mount --move /proc /newroot/proc - mount --move /sys /newroot/sys + cp --reflink=auto --sparse=auto --preserve=mode,timestamps,links -dfr /fadumproot/. /newroot/ cd /newroot && pivot_root . oldroot @@ -31,11 +32,11 @@ if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [ -f /proc/device-tree/ibm,o unset loop while read -r _ mp _; do case $mp in - /oldroot/*) umount "$mp" && loop=1 ;; + /oldroot/*) umount -d "$mp" && loop=1 ;; esac done </proc/mounts done - umount -l oldroot + umount -d -l oldroot exec /init fi -- 2.31.1

2 3

[PATCH] fadump: kdumpctl should check the modules used by the fadump initramfs
by Kairui Song 30 Jun '21

30 Jun '21

After fadump embedded the fadump initramfs in the normal initramfs, kdumpctl will mistakenly rebuild the initramfs everytime. kdumpctl checks the hostonly-kernel-modules.txt file in initramfs to check if required drivers are included, but the normal initramfs is built in non-hostonly mode, so it doesn't have a hostonly-kernel-modules.txt file. The check will always fail. So let mkfadumprd make a copy of the hostonly-kernel-modules.txt in the fadump initramfs and let kdumpctl check that file instead. Signed-off-by: Kairui Song <kasong(a)redhat.com> --- kdumpctl | 8 ++++++-- mkfadumprd | 8 ++++++-- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/kdumpctl b/kdumpctl index 26247d1..e4334c5 100755 --- a/kdumpctl +++ b/kdumpctl @@ -443,9 +443,13 @@ check_drivers_modified() # Include watchdog drivers if watchdog module is not omitted is_dracut_mod_omitted watchdog || _new_drivers+=" $(get_watchdog_drvs)" - [ -z "$_new_drivers" ] && return 0 - _old_drivers="$(lsinitrd $TARGET_INITRD -f /usr/lib/dracut/hostonly-kernel-modules.txt | tr '\n' ' ')" + + if is_fadump_capable; then + _old_drivers="$(lsinitrd "$TARGET_INITRD" -f /usr/lib/dracut/fadump-kernel-modules.txt | tr '\n' ' ')" + else + _old_drivers="$(lsinitrd "$TARGET_INITRD" -f /usr/lib/dracut/hostonly-kernel-modules.txt | tr '\n' ' ')" + fi ddebug "Modules required for kdump: '$_new_drivers'" ddebug "Modules included in old initramfs: '$_old_drivers'" diff --git a/mkfadumprd b/mkfadumprd index 4af4ae6..aecf2a8 100644 --- a/mkfadumprd +++ b/mkfadumprd @@ -44,14 +44,18 @@ fi ### Unpack the initramfs having dump capture capability mkdir -p "$MKFADUMPRD_TMPDIR/fadumproot" -if ! (pushd "$MKFADUMPRD_TMPDIR/fadumproot" > /dev/null && lsinitrd --unpack "$MKFADUMPRD_TMPDIR/fadump.img" && \ +if ! (pushd "$MKFADUMPRD_TMPDIR/fadumproot" > /dev/null && lsinitrd --unpack "$FADUMP_INITRD" && \ popd > /dev/null); then derror "mkfadumprd: failed to unpack '$MKFADUMPRD_TMPDIR'" exit 1 fi ### Pack it into the normal boot initramfs with zz-fadumpinit module -_dracut_isolate_args="--rebuild $REBUILD_INITRD --add zz-fadumpinit -i $MKFADUMPRD_TMPDIR/fadumproot /fadumproot" +_dracut_isolate_args="--rebuild $REBUILD_INITRD --add zz-fadumpinit \ + -i $MKFADUMPRD_TMPDIR/fadumproot /fadumproot \ + -i $MKFADUMPRD_TMPDIR/fadumproot/usr/lib/dracut/hostonly-kernel-modules.txt + /usr/lib/dracut/fadump-kernel-modules.txt" + if is_squash_available; then _dracut_isolate_args="$_dracut_isolate_args --add squash" fi -- 2.31.1

2 1

[PATCH v3 0/2] eliminate the warning message "find: '/sys/bus/ccwgroup/devices': No such file or directory" for non-s390x machine
by Coiby Xu 30 Jun '21

30 Jun '21

This patch set eliminates the harmless message "find: '/sys/bus/ccwgroup/devices': No such file or directory" for non-s390x machine and fix the format issue caused by mixed usage of tab and spaces. v3: - don't suppress the warning, check the existence of CCWGROUPBUS_DEVICEDIR to eliminate the warning instead [Kairui] v2: - replace spaces with tab to fix alignment issue Coiby Xu (2): check the existence of /sys/bus/ccwgroup/devices before trying to find online network device fix format issue in find_online_znet_device dracut-module-setup.sh | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-) -- 2.32.0

2 4

[PATCH 1/2] suppress the warning message "find: '/sys/bus/ccwgroup/devices': No such file or directory"
by Coiby Xu 28 Jun '21

28 Jun '21

/sys/bus/ccwgroup/devices doesn't exist for non-s390x machines. Suppress this harmless message. Fixes: commit 7d472515688fb330eee76dac1c39ae628398f757 ("Iterate /sys/bus/ccwgroup/devices to tell if we should set up rd.znet") Reported-by: Ruowen Qin <ruqin(a)redhat.com> BugLink: https://bugzilla.redhat.com/show_bug.cgi?id=1974618 Signed-off-by: Coiby Xu <coxu(a)redhat.com> --- dracut-module-setup.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 35a961a..24a6be0 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -467,7 +467,7 @@ kdump_setup_vlan() { find_online_znet_device() { local CCWGROUPBUS_DEVICEDIR="/sys/bus/ccwgroup/devices" local NETWORK_DEVICES d ifname ONLINE - NETWORK_DEVICES=$(find $CCWGROUPBUS_DEVICEDIR -type l) + NETWORK_DEVICES=$(find $CCWGROUPBUS_DEVICEDIR -type l 2>/dev/null) for d in $NETWORK_DEVICES do read ONLINE < $d/online -- 2.32.0

2 3

[PATCH 0/3] Update makedumpfile to latest
by Tao Liu 28 Jun '21

28 Jun '21

This patchset update makedumpfile to the latest (9a6f589d99dcef114c89f), in order to fix the following bugs: PATCH-0001: bz1972464 PATCH-0002: bz1965267 PATCH-0003: bz1974178 Tao Liu (3): Increase SECTION_MAP_LAST_BIT to 5 check for invalid physical address of /proc/kcore when finding max_paddr check for invalid physical address of /proc/kcore when making ELF dumpfile ...e-Increase-SECTION_MAP_LAST_BIT-to-5.patch | 42 +++++++++++++ ...ss-proc-kcore-when-finding-max_paddr.patch | 60 +++++++++++++++++++ ...-proc-kcore-when-making-ELF-dumpfile.patch | 43 +++++++++++++ kexec-tools.spec | 7 +++ 4 files changed, 152 insertions(+) create mode 100644 kexec-tools-2.0.22-makedumpfile-Increase-SECTION_MAP_LAST_BIT-to-5.patch create mode 100644 kexec-tools-2.0.22-makedumpfile-check-for-invalid-physical-address-proc-kcore-when-finding-max_paddr.patch create mode 100644 kexec-tools-2.0.22-makedumpfile-check-for-invalid-physical-address-proc-kcore-when-making-ELF-dumpfile.patch -- 2.29.2

2 4

[PATCH v2 0/2] suppress the warning message "find: '/sys/bus/ccwgroup/devices': No such file or directory" for non-s390x machine
by Coiby Xu 24 Jun '21

24 Jun '21

This patch set suppresses the harmless message "find: '/sys/bus/ccwgroup/devices': No such file or directory" for non-s390x machine and fix the format issue caused by mixed usage of tab and spaces. v2: - replace spaces with tab to fix alignment issue Coiby Xu (2): suppress the warning message "find: '/sys/bus/ccwgroup/devices': No such file or directory" fix format issue in find_online_znet_device dracut-module-setup.sh | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) -- 2.32.0

1 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

kexec June 2021