On 2016/09/12 at 10:55, Xunlei Pang wrote:
On 2016/09/12 at 10:34, Dave Young wrote:
> On 08/30/16 at 04:21pm, Xunlei Pang wrote:
>> On very large modern machines, there are a huge amount of memory deplored,
>> the initialization of the memory is very slow, thus some new technology is
>> introduced in Linux kernel to delay memory(pagetable) initialization after
>> the system is boot up using runtime memory hotplug.
>>
>> Unfortunately, the frequent memory onlining will trigger too many memory online
>> kernel/udev events, thereby causing some trouble to kdump.service which uses
udev
>> (98-kexec.rules) to listen to the memory/cpu hotplug events and further trigger
>> kdump.service restart(because kdump must update memory layout for memory bank
>> changes, or note elfcorehdr info for cpu changes). The most annoying one is the
>> cpus are busying doing "kdump.service restart" for a long time.
>>
>> For example,
>> The current rules in 98-kexec.rules for memory online is:
>> SUBSYSTEM=="memory", ACTION=="online",
PROGRAM="/bin/systemctl try-restart kdump.service"
>>
>> There is no way to control the number of kernel events delivered to udev and
>> futher to systemd udev threads to invoke PROGRAM.
>>
>> For systemd service there are several parameters to control the number of service
>> restart: "StartLimitInterval" and "StartLimitBurst". But the
service will become
>> unvailable(with some failed status) if the number of service restart is over the
>> limit. They don't help our situation(kdump.service has StartLimitInterval=0,
allowing
>> unlimited number of service restart).
>>
>>
>> We here provide a doable solution to mitigate this kdump service restart
issue(will
>> hopefull eliminate the side effect of hogging the cpu for some perceivable
period):
>> From above-mentioned example, we know udev calls "systemctl try-restart
kdump.service"
>> for every event, the whole process of restart is actually very time-consuming, as
kdump
>> checks all the possible conditions to decide if the initramfs rebuild is needed
or not.
>> However for cpu/memory hotplug scenarios, we don't need any initramfs check,
actually
>> we can simply reload the kernel and all set.
>>
>> This patch series, provide a new function reload_dump() to be used by
kdump.service reload,
>> and further used by 98-kexec udev PROGRAM.
>>
>> Here is some comparable time data to show the effect of this patch series:
>> 1)/bin/systemctl try-restart kdump.service:
>> 15:07:56 localhost systemd: Stopping Crash recovery kernel arming...
>> 15:07:56 localhost kdumpctl: kexec: unloaded kdump kernel
>> 15:07:56 localhost kdumpctl: Stopping kdump: [OK]
>> 15:07:56 localhost systemd: Starting Crash recovery kernel arming...
>> 15:08:02 localhost kdumpctl: kexec: loaded kdump kernel
>> 15:08:02 localhost kdumpctl: Starting kdump: [OK]
>> 15:08:02 localhost systemd: Started Crash recovery kernel arming.
>> 2)/bin/systemctl reload-or-try-restart kdump.service:
>> 15:09:24 localhost kdumpctl: kexec: loaded kdump kernel
>> 15:09:24 localhost systemd: Reloaded Crash recovery kernel arming.
>>
>> Another test fact is that:
>> 1)/bin/systemctl try-restart kdump.service:
>> I modified the udev/rules.d/98-kexec.rules for cpu subsytem to use:
>> SUBSYSTEM=="cpu", ACTION=="online",
PROGRAM="/bin/systemctl try-restart kdump.service"
>> SUBSYSTEM=="cpu", ACTION=="offline",
PROGRAM="/bin/systemctl try-restart kdump.service"
>>
>> And executed the follow shell commands for seconds which triggers a huge amount
of events:
>> echo 0 > /sys/devices/system/cpu/cpu3/online;
>> echo 1 > /sys/devices/system/cpu/cpu3/online;
>>
>> Then observed the cpu utilization using top, the total kdump restart processes
>> lasted more than half an hour(maybe over an hour IIRC).
>>
>> 2)/bin/systemctl reload-or-try-restart kdump.service:
>> SUBSYSTEM=="cpu", ACTION=="online",
PROGRAM="/bin/systemctl reload-or-try-restart kdump.service"
>> SUBSYSTEM=="cpu", ACTION=="offline",
PROGRAM="/bin/systemctl reload-or-try-restart kdump.service"
> Xunlei, in systemctl I only see try-reload-or-restart, is above a typo?
That's interesting, the fedora and rhel7 uses different names, on rhel7, it's
"reload-or-try-restart".
Anyway "try-reload-or-restart" sounds more reasonable, I guess it was renamed to
be this one on Fedora.
Regards,
Xunlei
> BTW, what is the behavior of below
>
> kdump service disabled -> memory online
>
> reload fails due to some reason..
If the service is not running, this command triggers nothing, just like the original
"try-restart".
Regards,
Xunlei
>> Executed the same shell commands for seconds to trigger events, and observed the
cpu
>> utilization using top, the total kdump reload processes lasted for a couple of
minutes.
>>
>>
>> Xunlei Pang (3):
>> kdumpctl: introduce prepare_dump() to refactor some code
>> kdumpctl: provide reload_kdump to be used for memory/cpu hotplug
>> kdump.service: add systemctl "ExecReload" for kdump
>>
>> kdump.service | 1 +
>> kdumpctl | 61 ++++++++++++++++++++++++++++++++++++++++++++++-------------
>> 2 files changed, 49 insertions(+), 13 deletions(-)
>>
>> --
>> 1.8.3.1
>> _______________________________________________
>> kexec mailing list
>> kexec(a)lists.fedoraproject.org
>>
https://lists.fedoraproject.org/admin/lists/kexec@lists.fedoraproject.org
> Thanks
> Dave