On very large modern machines, there are a huge amount of memory deplored,
the initialization of the memory is very slow, thus some new technology is
introduced in Linux kernel to delay memory(pagetable) initialization after
the system is boot up using runtime memory hotplug.
Unfortunately, the frequent memory onlining will trigger too many memory online
kernel/udev events, thereby causing some trouble to kdump.service which uses udev
(98-kexec.rules) to listen to the memory/cpu hotplug events and further trigger
kdump.service restart(because kdump must update memory layout for memory bank
changes, or note elfcorehdr info for cpu changes). The most annoying one is the
cpus are busying doing "kdump.service restart" for a long time.
For example,
The current rules in 98-kexec.rules for memory online is:
SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service"
There is no way to control the number of kernel events delivered to udev and
futher to systemd udev threads to invoke PROGRAM.
For systemd service there are several parameters to control the number of service
restart: "StartLimitInterval" and "StartLimitBurst". But the service will become
unvailable(with some failed status) if the number of service restart is over the
limit. They don't help our situation(kdump.service has StartLimitInterval=0, allowing
unlimited number of service restart).
We here provide a doable solution to mitigate this kdump service restart issue(will
hopefull eliminate the side effect of hogging the cpu for some perceivable period):
From above-mentioned example, we know udev calls "systemctl try-restart kdump.service"
for every event, the whole process of restart is actually very time-consuming, as kdump
checks all the possible conditions to decide if the initramfs rebuild is needed or not.
However for cpu/memory hotplug scenarios, we don't need any initramfs check, actually
we can simply reload the kernel and all set.
This patch series, provide a new function reload_dump() to be used by kdump.service reload,
and further used by 98-kexec udev PROGRAM.
Here is some comparable time data to show the effect of this patch series:
1)/bin/systemctl try-restart kdump.service:
15:07:56 localhost systemd: Stopping Crash recovery kernel arming...
15:07:56 localhost kdumpctl: kexec: unloaded kdump kernel
15:07:56 localhost kdumpctl: Stopping kdump: [OK]
15:07:56 localhost systemd: Starting Crash recovery kernel arming...
15:08:02 localhost kdumpctl: kexec: loaded kdump kernel
15:08:02 localhost kdumpctl: Starting kdump: [OK]
15:08:02 localhost systemd: Started Crash recovery kernel arming.
2)/bin/systemctl reload-or-try-restart kdump.service:
15:09:24 localhost kdumpctl: kexec: loaded kdump kernel
15:09:24 localhost systemd: Reloaded Crash recovery kernel arming.
Another test fact is that:
1)/bin/systemctl try-restart kdump.service:
I modified the udev/rules.d/98-kexec.rules for cpu subsytem to use:
SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump.service"
SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump.service"
And executed the follow shell commands for seconds which triggers a huge amount of events:
echo 0 > /sys/devices/system/cpu/cpu3/online;
echo 1 > /sys/devices/system/cpu/cpu3/online;
Then observed the cpu utilization using top, the total kdump restart processes
lasted more than half an hour(maybe over an hour IIRC).
2)/bin/systemctl reload-or-try-restart kdump.service:
SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service"
SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl reload-or-try-restart kdump.service"
Executed the same shell commands for seconds to trigger events, and observed the cpu
utilization using top, the total kdump reload processes lasted for a couple of minutes.
Xunlei Pang (3):
kdumpctl: introduce prepare_dump() to refactor some code
kdumpctl: provide reload_kdump to be used for memory/cpu hotplug
kdump.service: add systemctl "ExecReload" for kdump
kdump.service | 1 +
kdumpctl | 61 ++++++++++++++++++++++++++++++++++++++++++++++-------------
2 files changed, 49 insertions(+), 13 deletions(-)
--
1.8.3.1