[RFC PATCH 0/3] Implement "systemctl reload kdump.service"
by Xunlei Pang
On very large modern machines, there are a huge amount of memory deplored,
the initialization of the memory is very slow, thus some new technology is
introduced in Linux kernel to delay memory(pagetable) initialization after
the system is boot up using runtime memory hotplug.
Unfortunately, the frequent memory onlining will trigger too many memory online
kernel/udev events, thereby causing some trouble to kdump.service which uses udev
(98-kexec.rules) to listen to the memory/cpu hotplug events and further trigger
kdump.service restart(because kdump must update memory layout for memory bank
changes, or note elfcorehdr info for cpu changes). The most annoying one is the
cpus are busying doing "kdump.service restart" for a long time.
For example,
The current rules in 98-kexec.rules for memory online is:
SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service"
There is no way to control the number of kernel events delivered to udev and
futher to systemd udev threads to invoke PROGRAM.
For systemd service there are several parameters to control the number of service
restart: "StartLimitInterval" and "StartLimitBurst". But the service will become
unvailable(with some failed status) if the number of service restart is over the
limit. They don't help our situation(kdump.service has StartLimitInterval=0, allowing
unlimited number of service restart).
We here provide a doable solution to mitigate this kdump service restart issue(will
hopefull eliminate the side effect of hogging the cpu for some perceivable period):
From above-mentioned example, we know udev calls "systemctl try-restart kdump.service"
for every event, the whole process of restart is actually very time-consuming, as kdump
checks all the possible conditions to decide if the initramfs rebuild is needed or not.
However for cpu/memory hotplug scenarios, we don't need any initramfs check, actually
we can simply reload the kernel and all set.
This patch series, provide a new function reload_dump() to be used by kdump.service reload,
and further used by 98-kexec udev PROGRAM.
Here is some comparable time data to show the effect of this patch series:
1)/bin/systemctl try-restart kdump.service:
15:07:56 localhost systemd: Stopping Crash recovery kernel arming...
15:07:56 localhost kdumpctl: kexec: unloaded kdump kernel
15:07:56 localhost kdumpctl: Stopping kdump: [OK]
15:07:56 localhost systemd: Starting Crash recovery kernel arming...
15:08:02 localhost kdumpctl: kexec: loaded kdump kernel
15:08:02 localhost kdumpctl: Starting kdump: [OK]
15:08:02 localhost systemd: Started Crash recovery kernel arming.
2)/bin/systemctl reload-or-try-restart kdump.service:
15:09:24 localhost kdumpctl: kexec: loaded kdump kernel
15:09:24 localhost systemd: Reloaded Crash recovery kernel arming.
Another test fact is that:
1)/bin/systemctl try-restart kdump.service:
I modified the udev/rules.d/98-kexec.rules for cpu subsytem to use:
SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service"
SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service"
And executed the follow shell commands for seconds which triggers a huge amount of events:
echo 0 > /sys/devices/system/cpu/cpu3/online;
echo 1 > /sys/devices/system/cpu/cpu3/online;
Then observed the cpu utilization using top, the total kdump restart processes
lasted more than half an hour(maybe over an hour IIRC).
2)/bin/systemctl reload-or-try-restart kdump.service:
SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service"
SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service"
Executed the same shell commands for seconds to trigger events, and observed the cpu
utilization using top, the total kdump reload processes lasted for a couple of minutes.
Xunlei Pang (3):
kdumpctl: introduce prepare_dump() to refactor some code
kdumpctl: provide reload_kdump to be used for memory/cpu hotplug
kdump.service: add systemctl "ExecReload" for kdump
kdump.service | 1 +
kdumpctl | 61 ++++++++++++++++++++++++++++++++++++++++++++++-------------
2 files changed, 49 insertions(+), 13 deletions(-)
--
1.8.3.1
7 years
[PATCH v4 2/4] kexec-kdump-howto: Add doc about the special mount information via "dracut_args"
by Xunlei Pang
Update "kexec-kdump-howto" to illustrate the usage of special mount
information via "dracut_args".
Reviewed-by: Pratyush Anand <panand(a)redhat.com>
Suggested-by: Dave Young <dyoung(a)redhat.com>
Acked-by: Dave Young <dyoung(a)redhat.com>
Signed-off-by: Xunlei Pang <xlpang(a)redhat.com>
---
kexec-kdump-howto.txt | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/kexec-kdump-howto.txt b/kexec-kdump-howto.txt
index 38f831e..f46563f 100644
--- a/kexec-kdump-howto.txt
+++ b/kexec-kdump-howto.txt
@@ -345,6 +345,24 @@ mount the NFS mount and copy out the vmcore to your NFS server. Restart the
kdump service via '/sbin/systemctl restart kdump.service' to commit this change
to your kdump initrd.
+Special mount via "dracut_args"
+
+You can utilize "dracut_args" to pass "--mount" to kdump, see dracut manpage
+about the format of "--mount" for details. If there is any "--mount" specified
+via "dracut_args", kdump will build it as the mount target without doing any
+validation (mounting or checking like mount options, fs size, save path, etc),
+so you must test it to ensure all the correctness. You cannot use other targets
+in /etc/kdump.conf if you use "--mount" in "dracut_args". You also cannot specify
+mutliple "--mount" targets via "dracut_args".
+
+One use case of "--mount" in "dracut_args" is you do not want to mount dump target
+before kdump service startup, for example, to reduce the burden of the shared nfs
+server. Such as the example below:
+dracut_args --mount "192.168.1.1:/share /mnt/test nfs4 defaults"
+
+NOTE:
+- <mountpoint> must be specified as an absolute path.
+
Remote system via ssh/scp
Dumping over ssh/scp requires setting up passwordless ssh keys for every
--
1.8.3.1
7 years, 1 month
[PATCH v4 0/4] Support special mount via "dracut_args"
by Xunlei Pang
v1 -> v2:
For Patch 1, rename "does_dracut_args_contain_mount" to "is_mount_in_dracut_args".
No functional change.
v2 -> v3:
For Patch 1, remove the double quota requirement for "--mount".
For Patch 2, Improved description according to Dave's suggestion.
v3 -> v4:
For Patch 1,
Improved "is_mount_in_dracut_args" using:
grep -q "^dracut_args .*\-\-mount" /etc/kdump.conf
Improved print:
echo "Multiple mount targets specified in one \"dracut_args\"."
There were some complaints about nfs kdump that users must mount
nfs beforehand, which may cause some overhead to nfs server.
For example, there're thounsands of diskless clients deployed with
nfs dumping, each time the client is boot up, it will trigger
kdump rebuilding so will mount nfs, thus resulting in thousands
of nfs request concurrently imposed on the same nfs server.
In order to solve this, we introduce a new way of specifying mount
information via the already-existent "dracut_args" directive, this
is not only for nfs mount, also including other supported filesystems
like ext[2-4]/xfs, etc.
This feature relys on the latest fedora dracut version:
http://koji.fedoraproject.org/koji/taskinfo?taskID=15307615
The last patch is a fix found when designing this feature for nfs, so
attached in passing.
Xunlei Pang (4):
Support special mount information via "dracut_args"
kexec-kdump-howto: Add doc about the special mount information via
"dracut_args"
spec: Update to add dependency on dracut version >= 044-117
mkdumprd: Do not add "nfs" dracut module explicitly
dracut-kdump.sh | 4 ++++
dracut-module-setup.sh | 5 +++++
kdump-lib.sh | 27 +++++++++++++++++++++++++--
kdumpctl | 13 ++++++++++++-
kexec-kdump-howto.txt | 18 ++++++++++++++++++
kexec-tools.spec | 5 +++--
mkdumprd | 8 --------
7 files changed, 67 insertions(+), 13 deletions(-)
--
1.8.3.1
7 years, 1 month
[PATCH v3 0/4] Support special mount via "dracut_args"
by Xunlei Pang
v1 -> v2:
For Patch 1, rename "does_dracut_args_contain_mount" to "is_mount_in_dracut_args".
No functional change.
v2 -> v3:
For Patch 1, remove the double quota requirement for "--mount".
For Patch 2, Improved description according to Dave's suggestion.
There were some complaints about nfs kdump that users must mount
nfs beforehand, which may cause some overhead to nfs server.
For example, there're thounsands of diskless clients deployed with
nfs dumping, each time the client is boot up, it will trigger
kdump rebuilding so will mount nfs, thus resulting in thousands
of nfs request concurrently imposed on the same nfs server.
In order to solve this, we introduce a new way of specifying mount
information via the already-existent "dracut_args" directive, this
is not only for nfs mount, also including other supported filesystems
like ext[2-4]/xfs, etc.
This feature relys on the latest fedora dracut version:
http://koji.fedoraproject.org/koji/taskinfo?taskID=15307615
The last patch is a fix found when designing this feature for nfs, so
attached in passing.
Xunlei Pang (4):
Support special mount information via "dracut_args"
kexec-kdump-howto: Add doc about the special mount information via
"dracut_args"
spec: Update to add dependency on dracut version >= 044-117
mkdumprd: Do not add "nfs" dracut module explicitly
dracut-kdump.sh | 4 ++++
dracut-module-setup.sh | 5 +++++
kdump-lib.sh | 27 +++++++++++++++++++++++++--
kdumpctl | 13 ++++++++++++-
kexec-kdump-howto.txt | 18 ++++++++++++++++++
kexec-tools.spec | 5 +++--
mkdumprd | 8 --------
7 files changed, 67 insertions(+), 13 deletions(-)
--
1.8.3.1
7 years, 1 month
[PATCH v3 RESEND 1/4] Support special mount information via "dracut_args"
by Xunlei Pang
There are some complaints about nfs kdump that users must mount
nfs beforehand, which may cause some overhead to nfs server.
For example, there're thounsands of diskless clients deployed with
nfs dumping, each time the client is boot up, it will trigger
kdump rebuilding so will mount nfs, thus resulting in thousands
of nfs request concurrently imposed on the same nfs server.
We introduce a new way of specifying mount information via the
already-existent "dracut_args" directive(so avoid adding extra
directives in /etc/kdump.conf), we will skip all the filesystem
mounting and checking stuff for it. So it can be used in the
above-mentioned nfs scenario to avoid severe nfs server overhead.
Specifically, if there is any "--mount" information specified via
"dracut_args" in /etc/kdump.conf, always use it as the final mount
without any validation(mounting or checking like mount options,
fs size, etc), so users are expected to ensure its correctness.
NOTE:
-Only one mount target is allowed using "dracut_args" globally.
-Dracut will create <mountpoint> if it doesn't exist in kdump kernel,
<mountpoint> must be specified as an absolute path.
-Users should do a test first and ensure it works because kdump does
not prepare the mount or check all the validity.
Suggested-by: Dave Young <dyoung(a)redhat.com>
Signed-off-by: Xunlei Pang <xlpang(a)redhat.com>
---
dracut-kdump.sh | 4 ++++
dracut-module-setup.sh | 5 +++++
kdump-lib.sh | 26 ++++++++++++++++++++++++--
kdumpctl | 13 ++++++++++++-
4 files changed, 45 insertions(+), 3 deletions(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh
index 4aab205..42ba37f 100755
--- a/dracut-kdump.sh
+++ b/dracut-kdump.sh
@@ -146,6 +146,10 @@ read_kdump_conf()
# remove inline comments after the end of a directive.
config_val=$(strip_comments $config_val)
case "$config_opt" in
+ dracut_args)
+ config_val=$(get_dracut_args_target "$config_val")
+ [[ -n "$config_val" ]] && add_dump_code "dump_fs $config_val"
+ ;;
ext[234]|xfs|btrfs|minix|nfs)
add_dump_code "dump_fs $config_val"
;;
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index f5c0218..68e0ff8 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -450,6 +450,11 @@ kdump_install_conf() {
ssh|nfs)
kdump_install_net "$config_val"
;;
+ dracut_args)
+ if [[ $(get_dracut_args_fstype "$config_val") = nfs* ]] ; then
+ kdump_install_net "$(get_dracut_args_target "$config_val")"
+ fi
+ ;;
kdump_pre|kdump_post|extra_bins)
dracut_install $config_val
;;
diff --git a/kdump-lib.sh b/kdump-lib.sh
index 4567a05..6129602 100755
--- a/kdump-lib.sh
+++ b/kdump-lib.sh
@@ -23,7 +23,8 @@ is_ssh_dump_target()
is_nfs_dump_target()
{
- grep -q "^nfs" /etc/kdump.conf
+ grep -q "^nfs" /etc/kdump.conf || \
+ [[ $(get_dracut_args_fstype "$(grep "^dracut_args .*\-\-mount" /etc/kdump.conf)") = nfs* ]]
}
is_raw_dump_target()
@@ -45,7 +46,8 @@ is_fs_dump_target()
is_user_configured_dump_target()
{
- return $(is_ssh_dump_target || is_nfs_dump_target || is_raw_dump_target || is_fs_dump_target)
+ return $(is_mount_in_dracut_args || is_ssh_dump_target || is_nfs_dump_target || \
+ is_raw_dump_target || is_fs_dump_target)
}
strip_comments()
@@ -406,3 +408,23 @@ is_wdt_mod_omitted() {
return $ret
}
+
+# If "dracut_args" contains "--mount" information, use it
+# directly without any check(users are expected to ensure
+# its correctness).
+is_mount_in_dracut_args()
+{
+ return $(grep ^dracut_args /etc/kdump.conf | grep -q "\-\-mount")
+}
+
+# If $1 contains dracut_args "--mount", return <filesystem type>
+get_dracut_args_fstype()
+{
+ echo $1 | grep "\-\-mount" | sed "s/.*--mount .\(.*\)/\1/" | cut -d' ' -f3
+}
+
+# If $1 contains dracut_args "--mount", return <device>
+get_dracut_args_target()
+{
+ echo $1 | grep "\-\-mount" | sed "s/.*--mount .\(.*\)/\1/" | cut -d' ' -f1
+}
diff --git a/kdumpctl b/kdumpctl
index 0436bfc..4d83e69 100755
--- a/kdumpctl
+++ b/kdumpctl
@@ -236,12 +236,18 @@ check_config()
{
local nr
- nr=$(awk 'BEGIN{cnt=0} /^raw|^ssh[[:blank:]]|^nfs|^ext[234]|^xfs|^btrfs|^minix/{cnt++} END{print cnt}' $KDUMP_CONFIG_FILE)
+ nr=$(awk 'BEGIN{cnt=0} /^raw|^ssh[[:blank:]]|^nfs|^ext[234]|^xfs|^btrfs|^minix|^dracut_args .*\-\-mount/{cnt++} END{print cnt}' $KDUMP_CONFIG_FILE)
[ $nr -gt 1 ] && {
echo "More than one dump targets specified."
return 1
}
+ nr=$(grep "^dracut_args .*\-\-mount" $KDUMP_CONFIG_FILE | grep -o "\-\-mount" | wc -l)
+ [ $nr -gt 1 ] && {
+ echo "More than one mount targets specified in \"dracut_args\"."
+ return 1
+ }
+
while read config_opt config_val; do
# remove inline comments after the end of a directive.
config_val=$(strip_comments $config_val)
@@ -365,6 +371,11 @@ check_dump_fs_modified()
local _new_dev _new_mntpoint _new_fstype
local _target _path _dracut_args
+ # No need to check in case of mount target specified via "dracut_args".
+ if is_mount_in_dracut_args; then
+ return 0
+ fi
+
# No need to check in case of raw target.
# Currently we do not check also if ssh/nfs target is specified
if is_ssh_dump_target || is_nfs_dump_target || is_raw_dump_target; then
--
1.8.3.1
7 years, 1 month
[PATCH v2 0/4] Support special mount via "dracut_args"
by Xunlei Pang
v1 -> v2:
For Patch 1, rename "does_dracut_args_contain_mount" to "is_mount_in_dracut_args".
No functional change.
There were some complaints about nfs kdump that users must mount
nfs beforehand, which may cause some overhead to nfs server.
For example, there're thounsands of diskless clients deployed with
nfs dumping, each time the client is boot up, it will trigger
kdump rebuilding so will mount nfs, thus resulting in thousands
of nfs request concurrently imposed on the same nfs server.
In order to solve this, we introduce a new way of specifying mount
information via the already-existent "dracut_args" directive, this
is not only for nfs mount, also including other supported filesystems
like ext[2-4]/xfs, etc.
This feature relys on the latest fedora dracut version:
http://koji.fedoraproject.org/koji/taskinfo?taskID=15307615
The last patch is a fix found when designing this feature for nfs, so
attached in passing.
Xunlei Pang (4):
Support special mount information via "dracut_args"
kexec-kdump-howto: Add doc about the special mount information via
"dracut_args"
spec: Update to add dependency on dracut version >= 044-117
mkdumprd: Do not add "nfs" dracut module explicitly
dracut-kdump.sh | 4 ++++
dracut-module-setup.sh | 5 +++++
kdump-lib.sh | 27 +++++++++++++++++++++++++--
kdumpctl | 13 ++++++++++++-
kexec-kdump-howto.txt | 18 ++++++++++++++++++
kexec-tools.spec | 5 +++--
mkdumprd | 8 --------
7 files changed, 67 insertions(+), 13 deletions(-)
--
1.8.3.1
7 years, 1 month
[PATCH 0/4] Support special mount via "dracut_args"
by Xunlei Pang
There were some complaints about nfs kdump that users must mount
nfs beforehand, which may cause some overhead to nfs server.
For example, there're thounsands of diskless clients deployed with
nfs dumping, each time the client is boot up, it will trigger
kdump rebuilding so will mount nfs, thus resulting in thousands
of nfs request concurrently imposed on the same nfs server.
In order to solve this, we introduce a new way of specifying mount
information via the already-existent "dracut_args" directive, this
is not only for nfs mount, also including other supported filesystems
like ext[2-4]/xfs, etc.
This feature relys on the latest fedora dracut version:
http://koji.fedoraproject.org/koji/taskinfo?taskID=15307615
The last patch is a fix found when designing this feature for nfs, so
attached bypassing.
Xunlei Pang (4):
Support special mount information via "dracut_args"
kexec-kdump-howto: Add doc about the special mount information via
"dracut_args"
spec: Update to add dependency on dracut version >= 044-117
mkdumprd: Do not add "nfs" dracut module explicitly
dracut-kdump.sh | 4 ++++
dracut-module-setup.sh | 5 +++++
kdump-lib.sh | 27 +++++++++++++++++++++++++--
kdumpctl | 13 ++++++++++++-
kexec-kdump-howto.txt | 18 ++++++++++++++++++
kexec-tools.spec | 5 +++--
mkdumprd | 8 --------
7 files changed, 67 insertions(+), 13 deletions(-)
--
1.8.3.1
7 years, 1 month