On 2016/09/12 at 10:34, Dave Young wrote:
On 08/30/16 at 04:21pm, Xunlei Pang wrote:
> On very large modern machines, there are a huge amount of memory deplored,
> the initialization of the memory is very slow, thus some new technology is
> introduced in Linux kernel to delay memory(pagetable) initialization after
> the system is boot up using runtime memory hotplug.
>
> Unfortunately, the frequent memory onlining will trigger too many memory online
> kernel/udev events, thereby causing some trouble to kdump.service which uses udev
> (98-kexec.rules) to listen to the memory/cpu hotplug events and further trigger
> kdump.service restart(because kdump must update memory layout for memory bank
> changes, or note elfcorehdr info for cpu changes). The most annoying one is the
> cpus are busying doing "kdump.service restart" for a long time.
>
> For example,
> The current rules in 98-kexec.rules for memory online is:
> SUBSYSTEM=="memory", ACTION=="online",
PROGRAM="/bin/systemctl try-restart kdump.service"
>
> There is no way to control the number of kernel events delivered to udev and
> futher to systemd udev threads to invoke PROGRAM.
>
> For systemd service there are several parameters to control the number of service
> restart: "StartLimitInterval" and "StartLimitBurst". But the
service will become
> unvailable(with some failed status) if the number of service restart is over the
> limit. They don't help our situation(kdump.service has StartLimitInterval=0,
allowing
> unlimited number of service restart).
>
>
> We here provide a doable solution to mitigate this kdump service restart issue(will
> hopefull eliminate the side effect of hogging the cpu for some perceivable period):
> From above-mentioned example, we know udev calls "systemctl try-restart
kdump.service"
> for every event, the whole process of restart is actually very time-consuming, as
kdump
> checks all the possible conditions to decide if the initramfs rebuild is needed or
not.
> However for cpu/memory hotplug scenarios, we don't need any initramfs check,
actually
> we can simply reload the kernel and all set.
>
> This patch series, provide a new function reload_dump() to be used by kdump.service
reload,
> and further used by 98-kexec udev PROGRAM.
>
> Here is some comparable time data to show the effect of this patch series:
> 1)/bin/systemctl try-restart kdump.service:
> 15:07:56 localhost systemd: Stopping Crash recovery kernel arming...
> 15:07:56 localhost kdumpctl: kexec: unloaded kdump kernel
> 15:07:56 localhost kdumpctl: Stopping kdump: [OK]
> 15:07:56 localhost systemd: Starting Crash recovery kernel arming...
> 15:08:02 localhost kdumpctl: kexec: loaded kdump kernel
> 15:08:02 localhost kdumpctl: Starting kdump: [OK]
> 15:08:02 localhost systemd: Started Crash recovery kernel arming.
> 2)/bin/systemctl reload-or-try-restart kdump.service:
> 15:09:24 localhost kdumpctl: kexec: loaded kdump kernel
> 15:09:24 localhost systemd: Reloaded Crash recovery kernel arming.
>
> Another test fact is that:
> 1)/bin/systemctl try-restart kdump.service:
> I modified the udev/rules.d/98-kexec.rules for cpu subsytem to use:
> SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl
try-restart kdump.service"
> SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl
try-restart kdump.service"
>
> And executed the follow shell commands for seconds which triggers a huge amount of
events:
> echo 0 > /sys/devices/system/cpu/cpu3/online;
> echo 1 > /sys/devices/system/cpu/cpu3/online;
>
> Then observed the cpu utilization using top, the total kdump restart processes
> lasted more than half an hour(maybe over an hour IIRC).
>
> 2)/bin/systemctl reload-or-try-restart kdump.service:
> SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl
reload-or-try-restart kdump.service"
> SUBSYSTEM=="cpu", ACTION=="offline", PROGRAM="/bin/systemctl
reload-or-try-restart kdump.service"
Xunlei, in systemctl I only see try-reload-or-restart, is above a typo?
That's interesting, the fedora and rhel7 uses different names, on rhel7, it's
"reload-or-try-restart".
BTW, what is the behavior of below
kdump service disabled -> memory online
reload fails due to some reason..
If the service is not running, this command triggers nothing, just like the original
"try-restart".
Regards,
Xunlei
> Executed the same shell commands for seconds to trigger events, and observed the cpu
> utilization using top, the total kdump reload processes lasted for a couple of
minutes.
>
>
> Xunlei Pang (3):
> kdumpctl: introduce prepare_dump() to refactor some code
> kdumpctl: provide reload_kdump to be used for memory/cpu hotplug
> kdump.service: add systemctl "ExecReload" for kdump
>
> kdump.service | 1 +
> kdumpctl | 61 ++++++++++++++++++++++++++++++++++++++++++++++-------------
> 2 files changed, 49 insertions(+), 13 deletions(-)
>
> --
> 1.8.3.1
> _______________________________________________
> kexec mailing list
> kexec(a)lists.fedoraproject.org
>
https://lists.fedoraproject.org/admin/lists/kexec@lists.fedoraproject.org
Thanks
Dave