Hi Dave,
kdump-udev-throttler uses its own lock: /var/lock/kdump-udev-throttle,
if another process
is owning this lock, it means another service restart request is
pending and the service will
be restarted later (in less than 2 seconds). It's safe to exit here to
avoid duplicated service
restart request. This lock won't conflict from kdumpctl's lock.
But I just noticed this does have a risk of missing some udev event,
the systemctl call later may block,
and make the process holding two locks at the same time. I didn't meet
the problem because
if in past 2 seconds there are no udev events comming up, the system
would most likely be settled
already. I'll update the patch, thanks for the catch!
On Fri, Oct 19, 2018 at 5:38 PM Dave Young <dyoung(a)redhat.com> wrote:
Hi Kairui,
On 10/19/18 at 04:24pm, Kairui Song wrote:
> When doing CPU or memory hotplug, multiple udev events will be
> triggered together. This will bring unnecessary workload for the
> host and make udev take much longer to settle.
>
> Previous commit f2883275 added a workaround to solve the problem of
> systemd failing kdump servive due to burst of restart requests, but the
> unnecessary workload is not reduced. And currently, this is causing
> timeout failure on Hyper-V environment.
>
> This patch fixes the problem by merging concurrent kdump restart
> requests. Tested with Hyper-V VM, no new issues found.
>
> Signed-off-by: Kairui Song <kasong(a)redhat.com>
> ---
> Update from V1:
> Fix packaging problem, make kdump-udev-throttler executable and include
> it in main package.
>
> 98-kexec.rules | 8 ++++----
> kdump-udev-throttler | 25 +++++++++++++++++++++++++
> kexec-tools.spec | 3 +++
> 3 files changed, 32 insertions(+), 4 deletions(-)
> create mode 100755 kdump-udev-throttler
>
> diff --git a/98-kexec.rules b/98-kexec.rules
> index e32ee13..eee1df1 100644
> --- a/98-kexec.rules
> +++ b/98-kexec.rules
> @@ -1,4 +1,4 @@
> -SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl
try-restart kdump.service"
> -SUBSYSTEM=="cpu", ACTION=="remove",
PROGRAM="/bin/systemctl try-restart kdump.service"
> -SUBSYSTEM=="memory", ACTION=="online",
PROGRAM="/bin/systemctl try-restart kdump.service"
> -SUBSYSTEM=="memory", ACTION=="offline",
PROGRAM="/bin/systemctl try-restart kdump.service"
> +SUBSYSTEM=="cpu", ACTION=="add",
PROGRAM="kdump-udev-throttler"
> +SUBSYSTEM=="cpu", ACTION=="remove",
PROGRAM="kdump-udev-throttler"
> +SUBSYSTEM=="memory", ACTION=="online",
PROGRAM="kdump-udev-throttler"
> +SUBSYSTEM=="memory", ACTION=="offline",
PROGRAM="kdump-udev-throttler"
> diff --git a/kdump-udev-throttler b/kdump-udev-throttler
> new file mode 100755
> index 0000000..272d226
> --- /dev/null
> +++ b/kdump-udev-throttler
> @@ -0,0 +1,25 @@
> +#!/bin/bash
> +
> +throttle_lock="/var/lock/kdump-udev-throttle"
> +interval=2
> +
> +exec 9>$throttle_lock
> +flock -n 9
> +
> +if [ $? -ne 0 ]; then
> + echo "Another app is currently holding the throttle lock, extend
throttle timing and exiting"
> + exit 0
> +fi
Kairui, here we need restart the kdump service because of something
changed in the system. If we exit here then the kdump initrd will be
not updated for the change.
> +
> +timeout_time() {
> + echo $(($(stat -c "%Y" $throttle_lock) + $interval))
> +}
> +
> +# Wait until no one try to acquire the throttle_lock in past $interval seconds
> +while [[ $(timeout_time) -gt $(date +%s) ]]; do
> + sleep 1
> +done
> +
> +systemctl try-restart kdump
> +
> +exit 0
> diff --git a/kexec-tools.spec b/kexec-tools.spec
> index 6330534..91d322f 100644
> --- a/kexec-tools.spec
> +++ b/kexec-tools.spec
> @@ -29,6 +29,7 @@ Source24: kdump.sysconfig.ppc64le
> Source25: kdumpctl.8
> Source26: live-image-kdump-howto.txt
> Source27: early-kdump-howto.txt
> +Source28: kdump-udev-throttler
>
> #######################################
> # These are sources for mkdumpramfs
> @@ -171,6 +172,7 @@ install -m 755 %{SOURCE23}
$RPM_BUILD_ROOT%{_prefix}/lib/kdump/kdump-lib-initram
> # For s390x the ELF header is created in the kdump kernel and therefore kexec
> # udev rules are not required
> install -m 644 %{SOURCE14} $RPM_BUILD_ROOT%{_udevrulesdir}/98-kexec.rules
> +install -m 755 %{SOURCE28} $RPM_BUILD_ROOT%{_udevrulesdir}/../kdump-udev-throttler
> %endif
> install -m 644 %{SOURCE15} $RPM_BUILD_ROOT%{_mandir}/man5/kdump.conf.5
> install -m 644 %{SOURCE16} $RPM_BUILD_ROOT%{_unitdir}/kdump.service
> @@ -298,6 +300,7 @@ done
> %config(noreplace,missingok) %{_sysconfdir}/kdump.conf
> %ifnarch s390x
> %config %{_udevrulesdir}
> +%{_udevrulesdir}/../kdump-udev-throttler
> %endif
> %{dracutlibdir}/modules.d/*
> %dir %{_localstatedir}/crash
> --
> 2.17.1
>
Thanks
Dave
--
Best Regards,
Kairui Song