On very large modern machines, there are a huge amount of memory deplored, the initialization of the memory is very slow, thus some new technology is introduced in Linux kernel to delay memory(pagetable) initialization after the system is boot up using runtime memory hotplug.
Unfortunately, the frequent memory onlining will trigger too many memory online kernel/udev events, thereby causing some trouble to kdump.service which uses udev (98-kexec.rules) to listen to the memory/cpu hotplug events and further trigger kdump.service restart(because kdump must update memory layout for memory bank changes, or note elfcorehdr info for cpu changes). The most annoying one is the cpus are busying doing "kdump.service restart" for a long time.
For example, The current rules in 98-kexec.rules for memory online is: SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service"
There is no way to control the number of kernel events delivered to udev and futher to systemd udev threads to invoke PROGRAM.
For systemd service there are several parameters to control the number of service restart: "StartLimitInterval" and "StartLimitBurst". But the service will become unvailable(with some failed status) if the number of service restart is over the limit. They don't help our situation(kdump.service has StartLimitInterval=0, allowing unlimited number of service restart).
We here provide a doable solution to mitigate this kdump service restart issue(will hopefull eliminate the side effect of hogging the cpu for some perceivable period): From above-mentioned example, we know udev calls "systemctl try-restart kdump.service" for every event, the whole process of restart is actually very time-consuming, as kdump checks all the possible conditions to decide if the initramfs rebuild is needed or not. However for cpu/memory hotplug scenarios, we don't need any initramfs check, actually we can simply reload the kernel and all set.
This patch series, provide a new function reload_dump() to be used by kdump.service reload, and further used by 98-kexec udev PROGRAM.
Here is some comparable time data to show the effect of this patch series: 1)/bin/systemctl try-restart kdump.service: 15:07:56 localhost systemd: Stopping Crash recovery kernel arming... 15:07:56 localhost kdumpctl: kexec: unloaded kdump kernel 15:07:56 localhost kdumpctl: Stopping kdump: [OK] 15:07:56 localhost systemd: Starting Crash recovery kernel arming... 15:08:02 localhost kdumpctl: kexec: loaded kdump kernel 15:08:02 localhost kdumpctl: Starting kdump: [OK] 15:08:02 localhost systemd: Started Crash recovery kernel arming. 2)/bin/systemctl reload-or-try-restart kdump.service: 15:09:24 localhost kdumpctl: kexec: loaded kdump kernel 15:09:24 localhost systemd: Reloaded Crash recovery kernel arming.
Another test fact is that: 1)/bin/systemctl try-restart kdump.service: I modified the udev/rules.d/98-kexec.rules for cpu subsytem to use: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service"
And executed the follow shell commands for seconds which triggers a huge amount of events: echo 0 > /sys/devices/system/cpu/cpu3/online; echo 1 > /sys/devices/system/cpu/cpu3/online;
Then observed the cpu utilization using top, the total kdump restart processes lasted more than half an hour(maybe over an hour IIRC).
2)/bin/systemctl reload-or-try-restart kdump.service: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service"
Executed the same shell commands for seconds to trigger events, and observed the cpu utilization using top, the total kdump reload processes lasted for a couple of minutes.
Xunlei Pang (3): kdumpctl: introduce prepare_dump() to refactor some code kdumpctl: provide reload_kdump to be used for memory/cpu hotplug kdump.service: add systemctl "ExecReload" for kdump
kdump.service | 1 + kdumpctl | 61 ++++++++++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 49 insertions(+), 13 deletions(-)
These code actually prepares the kernel/initrd/cmdline arguments, move them to a new function named prepare_dump(), it will also be used by the new reload_dump() in the following patch.
Signed-off-by: Xunlei Pang xlpang@redhat.com --- kdumpctl | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 8d0ab81..69b2f3b 100755 --- a/kdumpctl +++ b/kdumpctl @@ -333,6 +333,25 @@ setup_target_initrd() fi }
+prepare_dump() +{ + check_boot_dir + + if [ -z "$KDUMP_KERNELVER" ]; then + kdump_kver=`uname -r` + else + kdump_kver=$KDUMP_KERNELVER + fi + + kdump_kernel="${KDUMP_BOOTDIR}/${KDUMP_IMG}-${kdump_kver}${KDUMP_IMG_EXT}" + + KDUMP_COMMANDLINE=$(prepare_cmdline) + + setup_target_initrd + + return $? +} + check_files_modified() { local modified_files="" @@ -509,16 +528,7 @@ check_rebuild() local ret system_modified="0" local initramfs_has_fadump
- check_boot_dir - - if [ -z "$KDUMP_KERNELVER" ]; then - kdump_kver=`uname -r` - else - kdump_kver=$KDUMP_KERNELVER - fi - - kdump_kernel="${KDUMP_BOOTDIR}/${KDUMP_IMG}-${kdump_kver}${KDUMP_IMG_EXT}" - setup_target_initrd + prepare_dump if [ $? -ne 0 ]; then return 1 fi @@ -634,8 +644,6 @@ load_kdump() fi fi
- KDUMP_COMMANDLINE=`prepare_cmdline` - # For secureboot enabled machines, use new kexec file based syscall. # Old syscall will always fail as it does not have capability to # to kernel signature verification.
On 30/08/2016:04:21:18 PM, Xunlei Pang wrote:
These code actually prepares the kernel/initrd/cmdline arguments, move them to a new function named prepare_dump(), it will also be used by the new reload_dump() in the following patch.
Although, there is no side effect of preparing command line in prepare_dump(), however since that step changes the existing functionality a bit, so we should mention the same in commit log.
Otherwise, this patch seems fine to me.
Reviewed-by: Pratyush Anand panand@redhat.com
Signed-off-by: Xunlei Pang xlpang@redhat.com
kdumpctl | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 8d0ab81..69b2f3b 100755 --- a/kdumpctl +++ b/kdumpctl @@ -333,6 +333,25 @@ setup_target_initrd() fi }
+prepare_dump() +{
- check_boot_dir
- if [ -z "$KDUMP_KERNELVER" ]; then
kdump_kver=`uname -r`- else
kdump_kver=$KDUMP_KERNELVER- fi
- kdump_kernel="${KDUMP_BOOTDIR}/${KDUMP_IMG}-${kdump_kver}${KDUMP_IMG_EXT}"
- KDUMP_COMMANDLINE=$(prepare_cmdline)
- setup_target_initrd
- return $?
+}
check_files_modified() { local modified_files="" @@ -509,16 +528,7 @@ check_rebuild() local ret system_modified="0" local initramfs_has_fadump
- check_boot_dir
- if [ -z "$KDUMP_KERNELVER" ]; then
kdump_kver=`uname -r`- else
kdump_kver=$KDUMP_KERNELVER- fi
- kdump_kernel="${KDUMP_BOOTDIR}/${KDUMP_IMG}-${kdump_kver}${KDUMP_IMG_EXT}"
- setup_target_initrd
- prepare_dump if [ $? -ne 0 ]; then return 1 fi
@@ -634,8 +644,6 @@ load_kdump() fi fi
- KDUMP_COMMANDLINE=`prepare_cmdline`
- # For secureboot enabled machines, use new kexec file based syscall. # Old syscall will always fail as it does not have capability to # to kernel signature verification.
-- 1.8.3.1 _______________________________________________ kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/kexec@lists.fedoraproject.org
On 2016/09/06 at 14:31, Pratyush Anand wrote:
On 30/08/2016:04:21:18 PM, Xunlei Pang wrote:
These code actually prepares the kernel/initrd/cmdline arguments, move them to a new function named prepare_dump(), it will also be used by the new reload_dump() in the following patch.
Although, there is no side effect of preparing command line in prepare_dump(), however since that step changes the existing functionality a bit, so we should mention the same in commit log.
Will update, thanks!
Regards, Xunlei
Otherwise, this patch seems fine to me.
Reviewed-by: Pratyush Anand panand@redhat.com
Signed-off-by: Xunlei Pang xlpang@redhat.com
kdumpctl | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 8d0ab81..69b2f3b 100755 --- a/kdumpctl +++ b/kdumpctl @@ -333,6 +333,25 @@ setup_target_initrd() fi }
+prepare_dump() +{
- check_boot_dir
- if [ -z "$KDUMP_KERNELVER" ]; then
kdump_kver=`uname -r`- else
kdump_kver=$KDUMP_KERNELVER- fi
- kdump_kernel="${KDUMP_BOOTDIR}/${KDUMP_IMG}-${kdump_kver}${KDUMP_IMG_EXT}"
- KDUMP_COMMANDLINE=$(prepare_cmdline)
- setup_target_initrd
- return $?
+}
check_files_modified() { local modified_files="" @@ -509,16 +528,7 @@ check_rebuild() local ret system_modified="0" local initramfs_has_fadump
- check_boot_dir
- if [ -z "$KDUMP_KERNELVER" ]; then
kdump_kver=`uname -r`- else
kdump_kver=$KDUMP_KERNELVER- fi
- kdump_kernel="${KDUMP_BOOTDIR}/${KDUMP_IMG}-${kdump_kver}${KDUMP_IMG_EXT}"
- setup_target_initrd
- prepare_dump if [ $? -ne 0 ]; then return 1 fi
@@ -634,8 +644,6 @@ load_kdump() fi fi
- KDUMP_COMMANDLINE=`prepare_cmdline`
- # For secureboot enabled machines, use new kexec file based syscall. # Old syscall will always fail as it does not have capability to # to kernel signature verification.
-- 1.8.3.1 _______________________________________________ kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/kexec@lists.fedoraproject.org
This is useful for memory/cpu hotplug udev events. With this function, we can add "ExecReload=/usr/bin/kdumpctl reload" in "kdump.service", and "/bin/systemctl reload kdump.service" will trigger the kdump kernel reload.
We will specify it in "/usr/lib/udev/rules.d/98-kexec.rules" as the "PROGRAM" executed as follows: PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" instead of PROGRAM="/bin/systemctl try-restart kdump.service"
Signed-off-by: Xunlei Pang xlpang@redhat.com --- kdumpctl | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/kdumpctl b/kdumpctl index 69b2f3b..811d7f2 100755 --- a/kdumpctl +++ b/kdumpctl @@ -615,7 +615,7 @@ need_64bit_headers() print (strtonum("0x" r[2]) > strtonum("0xffffffff")); }'` }
-# Load the kdump kerel specified in /etc/sysconfig/kdump +# Load the kdump kernel specified in /etc/sysconfig/kdump # If none is specified, try to load a kdump kernel with the same version # as the currently running kernel. load_kdump() @@ -664,6 +664,30 @@ load_kdump() fi }
+# Reload the kdump kernel specified in /etc/sysconfig/kdump. +# If none is specified, try to load a kdump kernel with the same version +# as the currently running kernel. +# +# This is useful for memory/cpu hotplug udev events. With this function, +# we can add "ExecReload=/usr/bin/kdumpctl reload" in "kdump.service", +# and "/bin/systemctl reload kdump.service" will trigger the kernel reload. + +# We use it as "PROGRAM" executed in "/usr/lib/udev/rules.d/98-kexec.rules": +# PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" +reload_dump() +{ + # Since systemctl reload kdump.service is called to enter it, we must have + # an active kdump.service, thus we assume all prerequisites are going well + # and call kexec -p (for kdump) to reload the kernel without any validation. + + prepare_dump + if [ $? -ne 0 ]; then + return + fi + + start_dump +} + check_ssh_config() { while read config_opt config_val; do @@ -1146,6 +1170,9 @@ main () stop start ;; + reload) + reload_dump + ;; condrestart) ;; propagate)
On 30/08/2016:04:21:19 PM, Xunlei Pang wrote:
This is useful for memory/cpu hotplug udev events. With this function, we can add "ExecReload=/usr/bin/kdumpctl reload" in "kdump.service", and "/bin/systemctl reload kdump.service" will trigger the kdump kernel reload.
We will specify it in "/usr/lib/udev/rules.d/98-kexec.rules" as the "PROGRAM" executed as follows: PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" instead of PROGRAM="/bin/systemctl try-restart kdump.service"
Signed-off-by: Xunlei Pang xlpang@redhat.com
kdumpctl | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/kdumpctl b/kdumpctl index 69b2f3b..811d7f2 100755 --- a/kdumpctl +++ b/kdumpctl @@ -615,7 +615,7 @@ need_64bit_headers() print (strtonum("0x" r[2]) > strtonum("0xffffffff")); }'` }
-# Load the kdump kerel specified in /etc/sysconfig/kdump +# Load the kdump kernel specified in /etc/sysconfig/kdump # If none is specified, try to load a kdump kernel with the same version # as the currently running kernel. load_kdump() @@ -664,6 +664,30 @@ load_kdump() fi }
+# Reload the kdump kernel specified in /etc/sysconfig/kdump. +# If none is specified, try to load a kdump kernel with the same version +# as the currently running kernel. +# +# This is useful for memory/cpu hotplug udev events. With this function, +# we can add "ExecReload=/usr/bin/kdumpctl reload" in "kdump.service", +# and "/bin/systemctl reload kdump.service" will trigger the kernel reload.
+# We use it as "PROGRAM" executed in "/usr/lib/udev/rules.d/98-kexec.rules": +# PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" +reload_dump() +{
- # Since systemctl reload kdump.service is called to enter it, we must have
- # an active kdump.service, thus we assume all prerequisites are going well
- # and call kexec -p (for kdump) to reload the kernel without any validation.
Probably, calling check_current_status() here would be good. We should go for prepare_dump only when "Kdump already running", otherwise a restart should be preferred instead of reload.
~Pratyush
- prepare_dump
- if [ $? -ne 0 ]; then
return- fi
- start_dump
+}
check_ssh_config() { while read config_opt config_val; do @@ -1146,6 +1170,9 @@ main () stop start ;;
reload)reload_dump;;-- 1.8.3.1 _______________________________________________ kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/kexec@lists.fedoraproject.org
On 2016/09/06 at 14:34, Pratyush Anand wrote:
On 30/08/2016:04:21:19 PM, Xunlei Pang wrote:
This is useful for memory/cpu hotplug udev events. With this function, we can add "ExecReload=/usr/bin/kdumpctl reload" in "kdump.service", and "/bin/systemctl reload kdump.service" will trigger the kdump kernel reload.
We will specify it in "/usr/lib/udev/rules.d/98-kexec.rules" as the "PROGRAM" executed as follows: PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" instead of PROGRAM="/bin/systemctl try-restart kdump.service"
Signed-off-by: Xunlei Pang xlpang@redhat.com
kdumpctl | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/kdumpctl b/kdumpctl index 69b2f3b..811d7f2 100755 --- a/kdumpctl +++ b/kdumpctl @@ -615,7 +615,7 @@ need_64bit_headers() print (strtonum("0x" r[2]) > strtonum("0xffffffff")); }'` }
-# Load the kdump kerel specified in /etc/sysconfig/kdump +# Load the kdump kernel specified in /etc/sysconfig/kdump # If none is specified, try to load a kdump kernel with the same version # as the currently running kernel. load_kdump() @@ -664,6 +664,30 @@ load_kdump() fi }
+# Reload the kdump kernel specified in /etc/sysconfig/kdump. +# If none is specified, try to load a kdump kernel with the same version +# as the currently running kernel. +# +# This is useful for memory/cpu hotplug udev events. With this function, +# we can add "ExecReload=/usr/bin/kdumpctl reload" in "kdump.service", +# and "/bin/systemctl reload kdump.service" will trigger the kernel reload.
+# We use it as "PROGRAM" executed in "/usr/lib/udev/rules.d/98-kexec.rules": +# PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" +reload_dump() +{
- # Since systemctl reload kdump.service is called to enter it, we must have
- # an active kdump.service, thus we assume all prerequisites are going well
- # and call kexec -p (for kdump) to reload the kernel without any validation.
Probably, calling check_current_status() here would be good. We should go for prepare_dump only when "Kdump already running", otherwise a restart should be preferred instead of reload.
Yes, adding it is a guarantee. The reason why I didn't add it is that this function is used for memory/cpu hotplug udev events, sometimes the number of memory events is huge, adding it may consumes some time.
The purpose of this function is for systemd service reload, if it is called when the service is not running(like the comment I wrote above the code), it will not take any effect, i.e. systemd will abort and can't reach reload_dump().
Regards, Xunlei
~Pratyush
- prepare_dump
- if [ $? -ne 0 ]; then
return- fi
- start_dump
+}
check_ssh_config() { while read config_opt config_val; do @@ -1146,6 +1170,9 @@ main () stop start ;;
reload)reload_dump;;-- 1.8.3.1 _______________________________________________ kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/kexec@lists.fedoraproject.org
On 2016/09/06 at 19:29, Xunlei Pang wrote:
On 2016/09/06 at 14:34, Pratyush Anand wrote:
On 30/08/2016:04:21:19 PM, Xunlei Pang wrote:
This is useful for memory/cpu hotplug udev events. With this function, we can add "ExecReload=/usr/bin/kdumpctl reload" in "kdump.service", and "/bin/systemctl reload kdump.service" will trigger the kdump kernel reload.
We will specify it in "/usr/lib/udev/rules.d/98-kexec.rules" as the "PROGRAM" executed as follows: PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" instead of PROGRAM="/bin/systemctl try-restart kdump.service"
Signed-off-by: Xunlei Pang xlpang@redhat.com
kdumpctl | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/kdumpctl b/kdumpctl index 69b2f3b..811d7f2 100755 --- a/kdumpctl +++ b/kdumpctl @@ -615,7 +615,7 @@ need_64bit_headers() print (strtonum("0x" r[2]) > strtonum("0xffffffff")); }'` }
-# Load the kdump kerel specified in /etc/sysconfig/kdump +# Load the kdump kernel specified in /etc/sysconfig/kdump # If none is specified, try to load a kdump kernel with the same version # as the currently running kernel. load_kdump() @@ -664,6 +664,30 @@ load_kdump() fi }
+# Reload the kdump kernel specified in /etc/sysconfig/kdump. +# If none is specified, try to load a kdump kernel with the same version +# as the currently running kernel. +# +# This is useful for memory/cpu hotplug udev events. With this function, +# we can add "ExecReload=/usr/bin/kdumpctl reload" in "kdump.service", +# and "/bin/systemctl reload kdump.service" will trigger the kernel reload.
+# We use it as "PROGRAM" executed in "/usr/lib/udev/rules.d/98-kexec.rules": +# PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" +reload_dump() +{
- # Since systemctl reload kdump.service is called to enter it, we must have
- # an active kdump.service, thus we assume all prerequisites are going well
- # and call kexec -p (for kdump) to reload the kernel without any validation.
Probably, calling check_current_status() here would be good. We should go for prepare_dump only when "Kdump already running", otherwise a restart should be preferred instead of reload.
Yes, adding it is a guarantee. The reason why I didn't add it is that this function is used for memory/cpu hotplug udev events, sometimes the number of memory events is huge, adding it may consumes some time.
The purpose of this function is for systemd service reload, if it is called when the service is not running(like the comment I wrote above the code), it will not take any effect, i.e. systemd will abort and can't reach reload_dump().
Seems check_current_status() is simple enough, I will add it to the formal version if the customer's feedback is good.
Thanks, Xunlei
Regards, Xunlei
~Pratyush
- prepare_dump
- if [ $? -ne 0 ]; then
return- fi
- start_dump
+}
check_ssh_config() { while read config_opt config_val; do @@ -1146,6 +1170,9 @@ main () stop start ;;
reload)reload_dump;;-- 1.8.3.1 _______________________________________________ kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/kexec@lists.fedoraproject.org
Add "ExecReload=/usr/bin/kdumpctl reload", after this we can use something like systemctl "reload" or "reload-or-try-restart" in "udev/rules.d/98-kexec.rules".
Here is some time data to show how time-consuming the orignal one is: 1)/bin/systemctl try-restart kdump.service: 15:07:56 localhost systemd: Stopping Crash recovery kernel arming... 15:07:56 localhost kdumpctl: kexec: unloaded kdump kernel 15:07:56 localhost kdumpctl: Stopping kdump: [OK] 15:07:56 localhost systemd: Starting Crash recovery kernel arming... 15:08:02 localhost kdumpctl: kexec: loaded kdump kernel 15:08:02 localhost kdumpctl: Starting kdump: [OK] 15:08:02 localhost systemd: Started Crash recovery kernel arming.
Here is some time data to show how effecient with these patches applied: 2)/bin/systemctl reload-or-try-restart kdump.service: 15:09:24 localhost kdumpctl: kexec: loaded kdump kernel 15:09:24 localhost systemd: Reloaded Crash recovery kernel arming.
Signed-off-by: Xunlei Pang xlpang@redhat.com --- kdump.service | 1 + 1 file changed, 1 insertion(+)
diff --git a/kdump.service b/kdump.service index 5144597..f888dd6 100644 --- a/kdump.service +++ b/kdump.service @@ -7,6 +7,7 @@ DefaultDependencies=no Type=oneshot ExecStart=/usr/bin/kdumpctl start ExecStop=/usr/bin/kdumpctl stop +ExecReload=/usr/bin/kdumpctl reload RemainAfterExit=yes StartLimitInterval=0
On 08/30/16 at 04:21pm, Xunlei Pang wrote:
On very large modern machines, there are a huge amount of memory deplored, the initialization of the memory is very slow, thus some new technology is introduced in Linux kernel to delay memory(pagetable) initialization after the system is boot up using runtime memory hotplug.
Unfortunately, the frequent memory onlining will trigger too many memory online kernel/udev events, thereby causing some trouble to kdump.service which uses udev (98-kexec.rules) to listen to the memory/cpu hotplug events and further trigger kdump.service restart(because kdump must update memory layout for memory bank changes, or note elfcorehdr info for cpu changes). The most annoying one is the cpus are busying doing "kdump.service restart" for a long time.
For example, The current rules in 98-kexec.rules for memory online is: SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service"
There is no way to control the number of kernel events delivered to udev and futher to systemd udev threads to invoke PROGRAM.
For systemd service there are several parameters to control the number of service restart: "StartLimitInterval" and "StartLimitBurst". But the service will become unvailable(with some failed status) if the number of service restart is over the limit. They don't help our situation(kdump.service has StartLimitInterval=0, allowing unlimited number of service restart).
We here provide a doable solution to mitigate this kdump service restart issue(will hopefull eliminate the side effect of hogging the cpu for some perceivable period): From above-mentioned example, we know udev calls "systemctl try-restart kdump.service" for every event, the whole process of restart is actually very time-consuming, as kdump checks all the possible conditions to decide if the initramfs rebuild is needed or not. However for cpu/memory hotplug scenarios, we don't need any initramfs check, actually we can simply reload the kernel and all set.
This patch series, provide a new function reload_dump() to be used by kdump.service reload, and further used by 98-kexec udev PROGRAM.
Here is some comparable time data to show the effect of this patch series: 1)/bin/systemctl try-restart kdump.service: 15:07:56 localhost systemd: Stopping Crash recovery kernel arming... 15:07:56 localhost kdumpctl: kexec: unloaded kdump kernel 15:07:56 localhost kdumpctl: Stopping kdump: [OK] 15:07:56 localhost systemd: Starting Crash recovery kernel arming... 15:08:02 localhost kdumpctl: kexec: loaded kdump kernel 15:08:02 localhost kdumpctl: Starting kdump: [OK] 15:08:02 localhost systemd: Started Crash recovery kernel arming. 2)/bin/systemctl reload-or-try-restart kdump.service: 15:09:24 localhost kdumpctl: kexec: loaded kdump kernel 15:09:24 localhost systemd: Reloaded Crash recovery kernel arming.
Another test fact is that: 1)/bin/systemctl try-restart kdump.service: I modified the udev/rules.d/98-kexec.rules for cpu subsytem to use: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service"
And executed the follow shell commands for seconds which triggers a huge amount of events: echo 0 > /sys/devices/system/cpu/cpu3/online; echo 1 > /sys/devices/system/cpu/cpu3/online;
Then observed the cpu utilization using top, the total kdump restart processes lasted more than half an hour(maybe over an hour IIRC).
2)/bin/systemctl reload-or-try-restart kdump.service: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service"
Xunlei, in systemctl I only see try-reload-or-restart, is above a typo?
BTW, what is the behavior of below
kdump service disabled -> memory online
reload fails due to some reason..
Executed the same shell commands for seconds to trigger events, and observed the cpu utilization using top, the total kdump reload processes lasted for a couple of minutes.
Xunlei Pang (3): kdumpctl: introduce prepare_dump() to refactor some code kdumpctl: provide reload_kdump to be used for memory/cpu hotplug kdump.service: add systemctl "ExecReload" for kdump
kdump.service | 1 + kdumpctl | 61 ++++++++++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 49 insertions(+), 13 deletions(-)
-- 1.8.3.1 _______________________________________________ kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/kexec@lists.fedoraproject.org
Thanks Dave
On 2016/09/12 at 10:34, Dave Young wrote:
On 08/30/16 at 04:21pm, Xunlei Pang wrote:
On very large modern machines, there are a huge amount of memory deplored, the initialization of the memory is very slow, thus some new technology is introduced in Linux kernel to delay memory(pagetable) initialization after the system is boot up using runtime memory hotplug.
Unfortunately, the frequent memory onlining will trigger too many memory online kernel/udev events, thereby causing some trouble to kdump.service which uses udev (98-kexec.rules) to listen to the memory/cpu hotplug events and further trigger kdump.service restart(because kdump must update memory layout for memory bank changes, or note elfcorehdr info for cpu changes). The most annoying one is the cpus are busying doing "kdump.service restart" for a long time.
For example, The current rules in 98-kexec.rules for memory online is: SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service"
There is no way to control the number of kernel events delivered to udev and futher to systemd udev threads to invoke PROGRAM.
For systemd service there are several parameters to control the number of service restart: "StartLimitInterval" and "StartLimitBurst". But the service will become unvailable(with some failed status) if the number of service restart is over the limit. They don't help our situation(kdump.service has StartLimitInterval=0, allowing unlimited number of service restart).
We here provide a doable solution to mitigate this kdump service restart issue(will hopefull eliminate the side effect of hogging the cpu for some perceivable period): From above-mentioned example, we know udev calls "systemctl try-restart kdump.service" for every event, the whole process of restart is actually very time-consuming, as kdump checks all the possible conditions to decide if the initramfs rebuild is needed or not. However for cpu/memory hotplug scenarios, we don't need any initramfs check, actually we can simply reload the kernel and all set.
This patch series, provide a new function reload_dump() to be used by kdump.service reload, and further used by 98-kexec udev PROGRAM.
Here is some comparable time data to show the effect of this patch series: 1)/bin/systemctl try-restart kdump.service: 15:07:56 localhost systemd: Stopping Crash recovery kernel arming... 15:07:56 localhost kdumpctl: kexec: unloaded kdump kernel 15:07:56 localhost kdumpctl: Stopping kdump: [OK] 15:07:56 localhost systemd: Starting Crash recovery kernel arming... 15:08:02 localhost kdumpctl: kexec: loaded kdump kernel 15:08:02 localhost kdumpctl: Starting kdump: [OK] 15:08:02 localhost systemd: Started Crash recovery kernel arming. 2)/bin/systemctl reload-or-try-restart kdump.service: 15:09:24 localhost kdumpctl: kexec: loaded kdump kernel 15:09:24 localhost systemd: Reloaded Crash recovery kernel arming.
Another test fact is that: 1)/bin/systemctl try-restart kdump.service: I modified the udev/rules.d/98-kexec.rules for cpu subsytem to use: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service"
And executed the follow shell commands for seconds which triggers a huge amount of events: echo 0 > /sys/devices/system/cpu/cpu3/online; echo 1 > /sys/devices/system/cpu/cpu3/online;
Then observed the cpu utilization using top, the total kdump restart processes lasted more than half an hour(maybe over an hour IIRC).
2)/bin/systemctl reload-or-try-restart kdump.service: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service"
Xunlei, in systemctl I only see try-reload-or-restart, is above a typo?
That's interesting, the fedora and rhel7 uses different names, on rhel7, it's "reload-or-try-restart".
BTW, what is the behavior of below
kdump service disabled -> memory online
reload fails due to some reason..
If the service is not running, this command triggers nothing, just like the original "try-restart".
Regards, Xunlei
Executed the same shell commands for seconds to trigger events, and observed the cpu utilization using top, the total kdump reload processes lasted for a couple of minutes.
Xunlei Pang (3): kdumpctl: introduce prepare_dump() to refactor some code kdumpctl: provide reload_kdump to be used for memory/cpu hotplug kdump.service: add systemctl "ExecReload" for kdump
kdump.service | 1 + kdumpctl | 61 ++++++++++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 49 insertions(+), 13 deletions(-)
-- 1.8.3.1 _______________________________________________ kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/kexec@lists.fedoraproject.org
Thanks Dave
On 2016/09/12 at 10:55, Xunlei Pang wrote:
On 2016/09/12 at 10:34, Dave Young wrote:
On 08/30/16 at 04:21pm, Xunlei Pang wrote:
On very large modern machines, there are a huge amount of memory deplored, the initialization of the memory is very slow, thus some new technology is introduced in Linux kernel to delay memory(pagetable) initialization after the system is boot up using runtime memory hotplug.
Unfortunately, the frequent memory onlining will trigger too many memory online kernel/udev events, thereby causing some trouble to kdump.service which uses udev (98-kexec.rules) to listen to the memory/cpu hotplug events and further trigger kdump.service restart(because kdump must update memory layout for memory bank changes, or note elfcorehdr info for cpu changes). The most annoying one is the cpus are busying doing "kdump.service restart" for a long time.
For example, The current rules in 98-kexec.rules for memory online is: SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service"
There is no way to control the number of kernel events delivered to udev and futher to systemd udev threads to invoke PROGRAM.
For systemd service there are several parameters to control the number of service restart: "StartLimitInterval" and "StartLimitBurst". But the service will become unvailable(with some failed status) if the number of service restart is over the limit. They don't help our situation(kdump.service has StartLimitInterval=0, allowing unlimited number of service restart).
We here provide a doable solution to mitigate this kdump service restart issue(will hopefull eliminate the side effect of hogging the cpu for some perceivable period): From above-mentioned example, we know udev calls "systemctl try-restart kdump.service" for every event, the whole process of restart is actually very time-consuming, as kdump checks all the possible conditions to decide if the initramfs rebuild is needed or not. However for cpu/memory hotplug scenarios, we don't need any initramfs check, actually we can simply reload the kernel and all set.
This patch series, provide a new function reload_dump() to be used by kdump.service reload, and further used by 98-kexec udev PROGRAM.
Here is some comparable time data to show the effect of this patch series: 1)/bin/systemctl try-restart kdump.service: 15:07:56 localhost systemd: Stopping Crash recovery kernel arming... 15:07:56 localhost kdumpctl: kexec: unloaded kdump kernel 15:07:56 localhost kdumpctl: Stopping kdump: [OK] 15:07:56 localhost systemd: Starting Crash recovery kernel arming... 15:08:02 localhost kdumpctl: kexec: loaded kdump kernel 15:08:02 localhost kdumpctl: Starting kdump: [OK] 15:08:02 localhost systemd: Started Crash recovery kernel arming. 2)/bin/systemctl reload-or-try-restart kdump.service: 15:09:24 localhost kdumpctl: kexec: loaded kdump kernel 15:09:24 localhost systemd: Reloaded Crash recovery kernel arming.
Another test fact is that: 1)/bin/systemctl try-restart kdump.service: I modified the udev/rules.d/98-kexec.rules for cpu subsytem to use: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service" SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service"
And executed the follow shell commands for seconds which triggers a huge amount of events: echo 0 > /sys/devices/system/cpu/cpu3/online; echo 1 > /sys/devices/system/cpu/cpu3/online;
Then observed the cpu utilization using top, the total kdump restart processes lasted more than half an hour(maybe over an hour IIRC).
2)/bin/systemctl reload-or-try-restart kdump.service: SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service" SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service"
Xunlei, in systemctl I only see try-reload-or-restart, is above a typo?
That's interesting, the fedora and rhel7 uses different names, on rhel7, it's "reload-or-try-restart".
Anyway "try-reload-or-restart" sounds more reasonable, I guess it was renamed to be this one on Fedora.
Regards, Xunlei
BTW, what is the behavior of below
kdump service disabled -> memory online
reload fails due to some reason..
If the service is not running, this command triggers nothing, just like the original "try-restart".
Regards, Xunlei
Executed the same shell commands for seconds to trigger events, and observed the cpu utilization using top, the total kdump reload processes lasted for a couple of minutes.
Xunlei Pang (3): kdumpctl: introduce prepare_dump() to refactor some code kdumpctl: provide reload_kdump to be used for memory/cpu hotplug kdump.service: add systemctl "ExecReload" for kdump
kdump.service | 1 + kdumpctl | 61 ++++++++++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 49 insertions(+), 13 deletions(-)
-- 1.8.3.1 _______________________________________________ kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/kexec@lists.fedoraproject.org
Thanks Dave
-
Dave Young -
Pratyush Anand -
Xunlei Pang -
Xunlei Pang