kexec August 2016

kexec@lists.fedoraproject.org

5 participants
7 discussions

[RFC PATCH 0/3] Implement "systemctl reload kdump.service"
by Xunlei Pang 12 Sep '16

12 Sep '16

On very large modern machines, there are a huge amount of memory deplored, the initialization of the memory is very slow, thus some new technology is introduced in Linux kernel to delay memory(pagetable) initialization after the system is boot up using runtime memory hotplug. Unfortunately, the frequent memory onlining will trigger too many memory online kernel/udev events, thereby causing some trouble to kdump.service which uses udev (98-kexec.rules) to listen to the memory/cpu hotplug events and further trigger kdump.service restart(because kdump must update memory layout for memory bank changes, or note elfcorehdr info for cpu changes). The most annoying one is the cpus are busying doing "kdump.service restart" for a long time. For example, The current rules in 98-kexec.rules for memory online is: SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" There is no way to control the number of kernel events delivered to udev and futher to systemd udev threads to invoke PROGRAM. For systemd service there are several parameters to control the number of service restart: "StartLimitInterval" and "StartLimitBurst". But the service will become unvailable(with some failed status) if the number of service restart is over the limit. They don't help our situation(kdump.service has StartLimitInterval=0, allowing unlimited number of service restart). We here provide a doable solution to mitigate this kdump service restart issue(will hopefull eliminate the side effect of hogging the cpu for some perceivable period): From above-mentioned example, we know udev calls "systemctl try-restart kdump.service" for every event, the whole process of restart is actually very time-consuming, as kdump checks all the possible conditions to decide if the initramfs rebuild is needed or not. However for cpu/memory hotplug scenarios, we don't need any initramfs check, actually we can simply reload the kernel and all set. This patch series, provide a new function reload_dump() to be used by kdump.service reload, and further used by 98-kexec udev PROGRAM. Here is some comparable time data to show the effect of this patch series: 1)/bin/systemctl try-restart kdump.service: 15:07:56 localhost systemd: Stopping Crash recovery kernel arming... 15:07:56 localhost kdumpctl: kexec: unloaded kdump kernel 15:07:56 localhost kdumpctl: Stopping kdump: [OK] 15:07:56 localhost systemd: Starting Crash recovery kernel arming... 15:08:02 localhost kdumpctl: kexec: loaded kdump kernel 15:08:02 localhost kdumpctl: Starting kdump: [OK] 15:08:02 localhost systemd: Started Crash recovery kernel arming. 2)/bin/systemctl reload-or-try-restart kdump.service: 15:09:24 localhost kdumpctl: kexec: loaded kdump kernel 15:09:24 localhost systemd: Reloaded Crash recovery kernel arming. Another test fact is that: 1)/bin/systemctl try-restart kdump.service: I modified the udev/rules.d/98-kexec.rules for cpu subsytem to use: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service" And executed the follow shell commands for seconds which triggers a huge amount of events: echo 0 > /sys/devices/system/cpu/cpu3/online; echo 1 > /sys/devices/system/cpu/cpu3/online; Then observed the cpu utilization using top, the total kdump restart processes lasted more than half an hour(maybe over an hour IIRC). 2)/bin/systemctl reload-or-try-restart kdump.service: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" Executed the same shell commands for seconds to trigger events, and observed the cpu utilization using top, the total kdump reload processes lasted for a couple of minutes. Xunlei Pang (3): kdumpctl: introduce prepare_dump() to refactor some code kdumpctl: provide reload_kdump to be used for memory/cpu hotplug kdump.service: add systemctl "ExecReload" for kdump kdump.service | 1 + kdumpctl | 61 ++++++++++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 49 insertions(+), 13 deletions(-) -- 1.8.3.1

4 11

[PATCH v4 2/4] kexec-kdump-howto: Add doc about the special mount information via "dracut_args"
by Xunlei Pang 26 Aug '16

26 Aug '16

Update "kexec-kdump-howto" to illustrate the usage of special mount information via "dracut_args". Reviewed-by: Pratyush Anand <panand(a)redhat.com> Suggested-by: Dave Young <dyoung(a)redhat.com> Acked-by: Dave Young <dyoung(a)redhat.com> Signed-off-by: Xunlei Pang <xlpang(a)redhat.com> --- kexec-kdump-howto.txt | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/kexec-kdump-howto.txt b/kexec-kdump-howto.txt index 38f831e..f46563f 100644 --- a/kexec-kdump-howto.txt +++ b/kexec-kdump-howto.txt @@ -345,6 +345,24 @@ mount the NFS mount and copy out the vmcore to your NFS server. Restart the kdump service via '/sbin/systemctl restart kdump.service' to commit this change to your kdump initrd. +Special mount via "dracut_args" + +You can utilize "dracut_args" to pass "--mount" to kdump, see dracut manpage +about the format of "--mount" for details. If there is any "--mount" specified +via "dracut_args", kdump will build it as the mount target without doing any +validation (mounting or checking like mount options, fs size, save path, etc), +so you must test it to ensure all the correctness. You cannot use other targets +in /etc/kdump.conf if you use "--mount" in "dracut_args". You also cannot specify +mutliple "--mount" targets via "dracut_args". + +One use case of "--mount" in "dracut_args" is you do not want to mount dump target +before kdump service startup, for example, to reduce the burden of the shared nfs +server. Such as the example below: +dracut_args --mount "192.168.1.1:/share /mnt/test nfs4 defaults" + +NOTE: +- <mountpoint> must be specified as an absolute path. + Remote system via ssh/scp Dumping over ssh/scp requires setting up passwordless ssh keys for every -- 1.8.3.1

1 0

[PATCH v4 0/4] Support special mount via "dracut_args"
by Xunlei Pang 26 Aug '16

26 Aug '16

v1 -> v2: For Patch 1, rename "does_dracut_args_contain_mount" to "is_mount_in_dracut_args". No functional change. v2 -> v3: For Patch 1, remove the double quota requirement for "--mount". For Patch 2, Improved description according to Dave's suggestion. v3 -> v4: For Patch 1, Improved "is_mount_in_dracut_args" using: grep -q "^dracut_args .*\-\-mount" /etc/kdump.conf Improved print: echo "Multiple mount targets specified in one \"dracut_args\"." There were some complaints about nfs kdump that users must mount nfs beforehand, which may cause some overhead to nfs server. For example, there're thounsands of diskless clients deployed with nfs dumping, each time the client is boot up, it will trigger kdump rebuilding so will mount nfs, thus resulting in thousands of nfs request concurrently imposed on the same nfs server. In order to solve this, we introduce a new way of specifying mount information via the already-existent "dracut_args" directive, this is not only for nfs mount, also including other supported filesystems like ext[2-4]/xfs, etc. This feature relys on the latest fedora dracut version: http://koji.fedoraproject.org/koji/taskinfo?taskID=15307615 The last patch is a fix found when designing this feature for nfs, so attached in passing. Xunlei Pang (4): Support special mount information via "dracut_args" kexec-kdump-howto: Add doc about the special mount information via "dracut_args" spec: Update to add dependency on dracut version >= 044-117 mkdumprd: Do not add "nfs" dracut module explicitly dracut-kdump.sh | 4 ++++ dracut-module-setup.sh | 5 +++++ kdump-lib.sh | 27 +++++++++++++++++++++++++-- kdumpctl | 13 ++++++++++++- kexec-kdump-howto.txt | 18 ++++++++++++++++++ kexec-tools.spec | 5 +++-- mkdumprd | 8 -------- 7 files changed, 67 insertions(+), 13 deletions(-) -- 1.8.3.1

1 4

[PATCH v3 0/4] Support special mount via "dracut_args"
by Xunlei Pang 26 Aug '16

26 Aug '16

v1 -> v2: For Patch 1, rename "does_dracut_args_contain_mount" to "is_mount_in_dracut_args". No functional change. v2 -> v3: For Patch 1, remove the double quota requirement for "--mount". For Patch 2, Improved description according to Dave's suggestion. There were some complaints about nfs kdump that users must mount nfs beforehand, which may cause some overhead to nfs server. For example, there're thounsands of diskless clients deployed with nfs dumping, each time the client is boot up, it will trigger kdump rebuilding so will mount nfs, thus resulting in thousands of nfs request concurrently imposed on the same nfs server. In order to solve this, we introduce a new way of specifying mount information via the already-existent "dracut_args" directive, this is not only for nfs mount, also including other supported filesystems like ext[2-4]/xfs, etc. This feature relys on the latest fedora dracut version: http://koji.fedoraproject.org/koji/taskinfo?taskID=15307615 The last patch is a fix found when designing this feature for nfs, so attached in passing. Xunlei Pang (4): Support special mount information via "dracut_args" kexec-kdump-howto: Add doc about the special mount information via "dracut_args" spec: Update to add dependency on dracut version >= 044-117 mkdumprd: Do not add "nfs" dracut module explicitly dracut-kdump.sh | 4 ++++ dracut-module-setup.sh | 5 +++++ kdump-lib.sh | 27 +++++++++++++++++++++++++-- kdumpctl | 13 ++++++++++++- kexec-kdump-howto.txt | 18 ++++++++++++++++++ kexec-tools.spec | 5 +++-- mkdumprd | 8 -------- 7 files changed, 67 insertions(+), 13 deletions(-) -- 1.8.3.1

5 16

[PATCH v3 RESEND 1/4] Support special mount information via "dracut_args"
by Xunlei Pang 25 Aug '16

25 Aug '16

There are some complaints about nfs kdump that users must mount nfs beforehand, which may cause some overhead to nfs server. For example, there're thounsands of diskless clients deployed with nfs dumping, each time the client is boot up, it will trigger kdump rebuilding so will mount nfs, thus resulting in thousands of nfs request concurrently imposed on the same nfs server. We introduce a new way of specifying mount information via the already-existent "dracut_args" directive(so avoid adding extra directives in /etc/kdump.conf), we will skip all the filesystem mounting and checking stuff for it. So it can be used in the above-mentioned nfs scenario to avoid severe nfs server overhead. Specifically, if there is any "--mount" information specified via "dracut_args" in /etc/kdump.conf, always use it as the final mount without any validation(mounting or checking like mount options, fs size, etc), so users are expected to ensure its correctness. NOTE: -Only one mount target is allowed using "dracut_args" globally. -Dracut will create <mountpoint> if it doesn't exist in kdump kernel, <mountpoint> must be specified as an absolute path. -Users should do a test first and ensure it works because kdump does not prepare the mount or check all the validity. Suggested-by: Dave Young <dyoung(a)redhat.com> Signed-off-by: Xunlei Pang <xlpang(a)redhat.com> --- dracut-kdump.sh | 4 ++++ dracut-module-setup.sh | 5 +++++ kdump-lib.sh | 26 ++++++++++++++++++++++++-- kdumpctl | 13 ++++++++++++- 4 files changed, 45 insertions(+), 3 deletions(-) diff --git a/dracut-kdump.sh b/dracut-kdump.sh index 4aab205..42ba37f 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -146,6 +146,10 @@ read_kdump_conf() # remove inline comments after the end of a directive. config_val=$(strip_comments $config_val) case "$config_opt" in + dracut_args) + config_val=$(get_dracut_args_target "$config_val") + [[ -n "$config_val" ]] && add_dump_code "dump_fs $config_val" + ;; ext[234]|xfs|btrfs|minix|nfs) add_dump_code "dump_fs $config_val" ;; diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index f5c0218..68e0ff8 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -450,6 +450,11 @@ kdump_install_conf() { ssh|nfs) kdump_install_net "$config_val" ;; + dracut_args) + if [[ $(get_dracut_args_fstype "$config_val") = nfs* ]] ; then + kdump_install_net "$(get_dracut_args_target "$config_val")" + fi + ;; kdump_pre|kdump_post|extra_bins) dracut_install $config_val ;; diff --git a/kdump-lib.sh b/kdump-lib.sh index 4567a05..6129602 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -23,7 +23,8 @@ is_ssh_dump_target() is_nfs_dump_target() { - grep -q "^nfs" /etc/kdump.conf + grep -q "^nfs" /etc/kdump.conf || \ + [[ $(get_dracut_args_fstype "$(grep "^dracut_args .*\-\-mount" /etc/kdump.conf)") = nfs* ]] } is_raw_dump_target() @@ -45,7 +46,8 @@ is_fs_dump_target() is_user_configured_dump_target() { - return $(is_ssh_dump_target || is_nfs_dump_target || is_raw_dump_target || is_fs_dump_target) + return $(is_mount_in_dracut_args || is_ssh_dump_target || is_nfs_dump_target || \ + is_raw_dump_target || is_fs_dump_target) } strip_comments() @@ -406,3 +408,23 @@ is_wdt_mod_omitted() { return $ret } + +# If "dracut_args" contains "--mount" information, use it +# directly without any check(users are expected to ensure +# its correctness). +is_mount_in_dracut_args() +{ + return $(grep ^dracut_args /etc/kdump.conf | grep -q "\-\-mount") +} + +# If $1 contains dracut_args "--mount", return <filesystem type> +get_dracut_args_fstype() +{ + echo $1 | grep "\-\-mount" | sed "s/.*--mount .$.*$/\1/" | cut -d' ' -f3 +} + +# If $1 contains dracut_args "--mount", return <device> +get_dracut_args_target() +{ + echo $1 | grep "\-\-mount" | sed "s/.*--mount .$.*$/\1/" | cut -d' ' -f1 +} diff --git a/kdumpctl b/kdumpctl index 0436bfc..4d83e69 100755 --- a/kdumpctl +++ b/kdumpctl @@ -236,12 +236,18 @@ check_config() { local nr - nr=$(awk 'BEGIN{cnt=0} /^raw|^ssh[[:blank:]]|^nfs|^ext[234]|^xfs|^btrfs|^minix/{cnt++} END{print cnt}' $KDUMP_CONFIG_FILE) + nr=$(awk 'BEGIN{cnt=0} /^raw|^ssh[[:blank:]]|^nfs|^ext[234]|^xfs|^btrfs|^minix|^dracut_args .*\-\-mount/{cnt++} END{print cnt}' $KDUMP_CONFIG_FILE) [ $nr -gt 1 ] && { echo "More than one dump targets specified." return 1 } + nr=$(grep "^dracut_args .*\-\-mount" $KDUMP_CONFIG_FILE | grep -o "\-\-mount" | wc -l) + [ $nr -gt 1 ] && { + echo "More than one mount targets specified in \"dracut_args\"." + return 1 + } + while read config_opt config_val; do # remove inline comments after the end of a directive. config_val=$(strip_comments $config_val) @@ -365,6 +371,11 @@ check_dump_fs_modified() local _new_dev _new_mntpoint _new_fstype local _target _path _dracut_args + # No need to check in case of mount target specified via "dracut_args". + if is_mount_in_dracut_args; then + return 0 + fi + # No need to check in case of raw target. # Currently we do not check also if ssh/nfs target is specified if is_ssh_dump_target || is_nfs_dump_target || is_raw_dump_target; then -- 1.8.3.1

1 0

[PATCH v2 0/4] Support special mount via "dracut_args"
by Xunlei Pang 25 Aug '16

25 Aug '16

v1 -> v2: For Patch 1, rename "does_dracut_args_contain_mount" to "is_mount_in_dracut_args". No functional change. There were some complaints about nfs kdump that users must mount nfs beforehand, which may cause some overhead to nfs server. For example, there're thounsands of diskless clients deployed with nfs dumping, each time the client is boot up, it will trigger kdump rebuilding so will mount nfs, thus resulting in thousands of nfs request concurrently imposed on the same nfs server. In order to solve this, we introduce a new way of specifying mount information via the already-existent "dracut_args" directive, this is not only for nfs mount, also including other supported filesystems like ext[2-4]/xfs, etc. This feature relys on the latest fedora dracut version: http://koji.fedoraproject.org/koji/taskinfo?taskID=15307615 The last patch is a fix found when designing this feature for nfs, so attached in passing. Xunlei Pang (4): Support special mount information via "dracut_args" kexec-kdump-howto: Add doc about the special mount information via "dracut_args" spec: Update to add dependency on dracut version >= 044-117 mkdumprd: Do not add "nfs" dracut module explicitly dracut-kdump.sh | 4 ++++ dracut-module-setup.sh | 5 +++++ kdump-lib.sh | 27 +++++++++++++++++++++++++-- kdumpctl | 13 ++++++++++++- kexec-kdump-howto.txt | 18 ++++++++++++++++++ kexec-tools.spec | 5 +++-- mkdumprd | 8 -------- 7 files changed, 67 insertions(+), 13 deletions(-) -- 1.8.3.1

1 4

[PATCH 0/4] Support special mount via "dracut_args"
by Xunlei Pang 24 Aug '16

24 Aug '16

There were some complaints about nfs kdump that users must mount nfs beforehand, which may cause some overhead to nfs server. For example, there're thounsands of diskless clients deployed with nfs dumping, each time the client is boot up, it will trigger kdump rebuilding so will mount nfs, thus resulting in thousands of nfs request concurrently imposed on the same nfs server. In order to solve this, we introduce a new way of specifying mount information via the already-existent "dracut_args" directive, this is not only for nfs mount, also including other supported filesystems like ext[2-4]/xfs, etc. This feature relys on the latest fedora dracut version: http://koji.fedoraproject.org/koji/taskinfo?taskID=15307615 The last patch is a fix found when designing this feature for nfs, so attached bypassing. Xunlei Pang (4): Support special mount information via "dracut_args" kexec-kdump-howto: Add doc about the special mount information via "dracut_args" spec: Update to add dependency on dracut version >= 044-117 mkdumprd: Do not add "nfs" dracut module explicitly dracut-kdump.sh | 4 ++++ dracut-module-setup.sh | 5 +++++ kdump-lib.sh | 27 +++++++++++++++++++++++++-- kdumpctl | 13 ++++++++++++- kexec-kdump-howto.txt | 18 ++++++++++++++++++ kexec-tools.spec | 5 +++-- mkdumprd | 8 -------- 7 files changed, 67 insertions(+), 13 deletions(-) -- 1.8.3.1

3 7

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

kexec August 2016