在 2018年12月20日 15:31, Xiaowu Wu 写道:
----- Original Message -----
> From: "Dave Young" <dyoung(a)redhat.com>
> To: "Lianbo Jiang" <lijiang(a)redhat.com>
> Cc: kexec(a)lists.fedoraproject.org, bhe(a)redhat.com, xiawu(a)redhat.com
> Sent: Thursday, December 20, 2018 2:06:17 PM
> Subject: Re: [PATCH] Avoid falling into infinite loop restart when using a
problematic system
>
> + Emma
> On 12/20/18 at 01:57pm, Dave Young wrote:
>> On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
>>> By default, early kdump reboots the system after capturing the vmcore.
>>> If the problematic system is continuously crashing due to some issue
>>> during early boot stage, the system may fall into infinite loop restart
>>> like this:
>>>
>>> boot -----> crash -----> early kdump (dump vmcore)
>>> ^ |
>>> '.........(reboot).............'
>>>
>>> But now, the system crash at early stage is only captured by early kdump,
>>> and the rest is captured by normal kdump. That to say, when normal kdump
>>> service starts, it will load it again and override early kdump. It is
>>> helpful to control the logic of early kdump and normal kdump separately
>>> in final action(it is called by kdump-capture.service). For example,
>>> early kdump always passes the 'rd.earlykdump' to the second kernel
when
>>> early kdump is enabled, but normal kdump doesn't pass the
'rd.earlykdump'
>>> to the second kernel at any time. So they can be distinguished in the
>>> second kernel.
So the new logic is to decide to run final poweroff or reboot logic by checking whether
it's running the kexec loaded during early boot or by kdump.service? right?
By checking if this 'rd.earlykdump' presents in the 2nd kernel boot cmdline.
But what if, early kdump is enabled and kexec load succeeds during at early boot.
However, kdump.service is not started successfully by kdump.service for some reaseon
during system startup. Then when the sysetm panic, it will poweroff system after saving a
vmcore becuase it believes system panic in early boot, which is unexpected..
I am not sure how pratical it is, but it may happen..
Thanks for your comment.
This is a good case. Let's consider two scenarios:
1. If kdump.service is disabled, it makes no sense distinguish early kdump and normal
kdump. Because it has only one kdump.
2. If kdump.service is enabled, but it fails to start, that should be a bug. We should try
to fix it.
Actually, the purpose of distinguishing early kdump and normal kdump is to perform the
different action in kdump capture
service. Anyway, early kdump and normal kdump can get the correct vmcore, what's the
difference?
>>
>> Hmm, I'm confused about the param passing above.
>>
>> Early or non early just means about the service loading phase, in
>> initramfs or not, I notice dracut/systemd will print some message about
>> they are running in initramfs, so probably you can check how to get it
>> with same way, if this is not initramfs then just unload before the
>> check in kdump loading.
>>
>> The picture like below:
>>
>> Kernel boot ->
>>
>> initramfs ---
>> early kdump load
>> ---- Mark A ----
>> initramfs switch root
>>
>> system startup (real root fs)
>> service a
>> service b ... (eg. networking etc.)
>> kdump service start
>> -----Mark B -----
>> load kdump kernel again
>>
>>
>> The problem will happen between Mark A and Mark B, during this period,
>> there could be repeated crash -> earlykdump_load, there might be some
>> random crash as well since during the real root fs service startup,
>> for example after network is ready if some network workload cause a
>> panic, it maybe not 100% reproducible, so it seems we still need to
>> make the poweroff configurable. eg.
>>
>> default is poweroff, but one can choose if he can.
>>
>>>
>>> Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
>>> ---
>>> dracut-early-kdump.sh | 2 +-
>>> dracut-kdump-error-handler.sh | 7 ++++++-
>>> dracut-kdump.sh | 6 +++++-
>>> kdump-lib-initramfs.sh | 16 +++++++++++++---
>>> kdump.sysconfig | 2 +-
>>> kdump.sysconfig.aarch64 | 2 +-
>>> kdump.sysconfig.i386 | 2 +-
>>> kdump.sysconfig.ppc64 | 2 +-
>>> kdump.sysconfig.ppc64le | 2 +-
>>> kdump.sysconfig.s390x | 2 +-
>>> kdump.sysconfig.x86_64 | 2 +-
>>> kdumpctl | 15 +++++++++++++--
>>> 12 files changed, 45 insertions(+), 15 deletions(-)
>>>
>>> diff --git a/dracut-early-kdump.sh b/dracut-early-kdump.sh
>>> index 34a9909..a799dbf 100755
>>> --- a/dracut-early-kdump.sh
>>> +++ b/dracut-early-kdump.sh
>>> @@ -58,7 +58,7 @@ early_kdump_load()
>>> fi
>>>
>>> $KEXEC ${EARLY_KEXEC_ARGS} $standard_kexec_args \
>>> - --command-line="$EARLY_KDUMP_CMDLINE" \
>>> + --command-line="$EARLY_KDUMP_CMDLINE rd.earlykdump" \
>>> --initrd=$EARLY_KDUMP_INITRD $EARLY_KDUMP_KERNEL
>>> if [ $? == 0 ]; then
>>> echo "kexec: loaded early-kdump kernel"
>>> diff --git a/dracut-kdump-error-handler.sh
>>> b/dracut-kdump-error-handler.sh
>>> index 2f0f1d1..4f0e58c 100755
>>> --- a/dracut-kdump-error-handler.sh
>>> +++ b/dracut-kdump-error-handler.sh
>>> @@ -1,5 +1,6 @@
>>> #!/bin/sh
>>>
>>> +. /lib/dracut-lib.sh
>>> . /lib/kdump-lib-initramfs.sh
>>>
>>> set -o pipefail
>>> @@ -7,4 +8,8 @@ export PATH=$PATH:$KDUMP_SCRIPT_DIR
>>>
>>> get_kdump_confs
>>> do_default_action
>>> -do_final_action
>>> +if getargbool 0 rd.earlykdump; then
>>> + do_earlykdump_final_action
>>> +else
>>> + do_final_action
>>> +fi
>>> diff --git a/dracut-kdump.sh b/dracut-kdump.sh
>>> index b75c2a5..119b006 100755
>>> --- a/dracut-kdump.sh
>>> +++ b/dracut-kdump.sh
>>> @@ -201,4 +201,8 @@ if [ $DUMP_RETVAL -ne 0 ]; then
>>> exit 1
>>> fi
>>>
>>> -do_final_action
>>> +if getargbool 0 rd.earlykdump; then
>>> + do_earlykdump_final_action
>>> +else
>>> + do_final_action
>>> +fi
>>> diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh
>>> index 7ba99b6..f95cbcf 100755
>>> --- a/kdump-lib-initramfs.sh
>>> +++ b/kdump-lib-initramfs.sh
>>> @@ -6,7 +6,7 @@ KDUMP_PATH="/var/crash"
>>> CORE_COLLECTOR=""
>>> DEFAULT_CORE_COLLECTOR="makedumpfile -l --message-level 1 -d 31"
>>> DMESG_COLLECTOR="/sbin/vmcore-dmesg"
>>> -DEFAULT_ACTION="systemctl reboot -f"
>>> +DEFAULT_ACTION=""
>>> DATEDIR=`date +%Y-%m-%d-%T`
>>> HOST_IP='127.0.0.1'
>>> DUMP_INSTRUCTION=""
>>> @@ -14,6 +14,7 @@ SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa"
>>> KDUMP_SCRIPT_DIR="/kdumpscripts"
>>> DD_BLKSIZE=512
>>> FINAL_ACTION="systemctl reboot -f"
>>> +EARLYKDUMP_FINAL_ACTION="systemctl poweroff"
>>> KDUMP_CONF="/etc/kdump.conf"
>>> KDUMP_PRE=""
>>> KDUMP_POST=""
>>> @@ -155,11 +156,20 @@ kdump_emergency_shell()
>>>
>>> do_default_action()
>>> {
>>> - echo "Kdump: Executing default action $DEFAULT_ACTION"
>>> - eval $DEFAULT_ACTION
>>> + if [ $DEFAULT_ACTION == "" ]; then
>>> + echo "Kdump: default action string is null."
$DEFAULT_ACTION should be double quoted. Now it reports an error in Kdump Error Handler:
Starting Kdump Error Handler...
/lib/kdump-lib-initramfs.sh: line 159: [: ==: unary operator expected
And this "Kdump: default action string is null." message is bit confusing.
In kdump.conf, it clearly states "The default option is "reboot"".
User may wonder this 'string is null' indicates an error or not.
I guess it would be better saying like,
"Kdump: no default action is specified. Executing default option"
^^ this would probably be replaced by
$FINAL_ACTION or $EARLYKDUMP_FINAL_ACTION which will be executed anyway.
Great. They will be improved in v2.
Thanks.
>>>> + else
>>>> + echo "Kdump: Executing default action
$DEFAULT_ACTION"
>>>> + eval $DEFAULT_ACTION
>>>> + fi
>>>> }
>>>>
>>>> do_final_action()
>>>> {
>>>> eval $FINAL_ACTION
>>>> }
>>>> +
>>>> +do_earlykdump_final_action()
>>>> +{
>>>> + eval $EARLYKDUMP_FINAL_ACTION
>>>> +}
>>>> diff --git a/kdump.sysconfig b/kdump.sysconfig
>>>> index ffe1df8..b011c1c 100644
>>>> --- a/kdump.sysconfig
>>>> +++ b/kdump.sysconfig
>>>> @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE=""
>>>> # This variable lets us remove arguments from the current kdump
>>>> commandline
>>>> # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline
>>>> # NOTE: some arguments such as crashkernel will always be removed
>>>> -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug
quiet"
>>>> +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet
>>>> rd.earlykdump"
>>>>
>>>> # This variable lets us append arguments to the current kdump
>>>> commandline
>>>> # after processed by KDUMP_COMMANDLINE_REMOVE
>>>> diff --git a/kdump.sysconfig.aarch64 b/kdump.sysconfig.aarch64
>>>> index 0a6b14c..b8b8865 100644
>>>> --- a/kdump.sysconfig.aarch64
>>>> +++ b/kdump.sysconfig.aarch64
>>>> @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE=""
>>>> # This variable lets us remove arguments from the current kdump
>>>> commandline
>>>> # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline
>>>> # NOTE: some arguments such as crashkernel will always be removed
>>>> -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug
quiet"
>>>> +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet
>>>> rd.earlykdump"
>>>>
>>>> # This variable lets us append arguments to the current kdump
>>>> commandline
>>>> # after processed by KDUMP_COMMANDLINE_REMOVE
>>>> diff --git a/kdump.sysconfig.i386 b/kdump.sysconfig.i386
>>>> index 18c407e..b9a2835 100644
>>>> --- a/kdump.sysconfig.i386
>>>> +++ b/kdump.sysconfig.i386
>>>> @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE=""
>>>> # This variable lets us remove arguments from the current kdump
>>>> commandline
>>>> # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline
>>>> # NOTE: some arguments such as crashkernel will always be removed
>>>> -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug
quiet"
>>>> +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet
>>>> rd.earlykdump"
>>>>
>>>> # This variable lets us append arguments to the current kdump
>>>> commandline
>>>> # after processed by KDUMP_COMMANDLINE_REMOVE
>>>> diff --git a/kdump.sysconfig.ppc64 b/kdump.sysconfig.ppc64
>>>> index 55a01cc..e1ea9c6 100644
>>>> --- a/kdump.sysconfig.ppc64
>>>> +++ b/kdump.sysconfig.ppc64
>>>> @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE=""
>>>> # This variable lets us remove arguments from the current kdump
>>>> commandline
>>>> # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline
>>>> # NOTE: some arguments such as crashkernel will always be removed
>>>> -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug
quiet"
>>>> +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet
>>>> rd.earlykdump"
>>>>
>>>> # This variable lets us append arguments to the current kdump
>>>> commandline
>>>> # after processed by KDUMP_COMMANDLINE_REMOVE
>>>> diff --git a/kdump.sysconfig.ppc64le b/kdump.sysconfig.ppc64le
>>>> index 55a01cc..e1ea9c6 100644
>>>> --- a/kdump.sysconfig.ppc64le
>>>> +++ b/kdump.sysconfig.ppc64le
>>>> @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE=""
>>>> # This variable lets us remove arguments from the current kdump
>>>> commandline
>>>> # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline
>>>> # NOTE: some arguments such as crashkernel will always be removed
>>>> -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug
quiet"
>>>> +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet
>>>> rd.earlykdump"
>>>>
>>>> # This variable lets us append arguments to the current kdump
>>>> commandline
>>>> # after processed by KDUMP_COMMANDLINE_REMOVE
>>>> diff --git a/kdump.sysconfig.s390x b/kdump.sysconfig.s390x
>>>> index b3aec3c..4c5fc00 100644
>>>> --- a/kdump.sysconfig.s390x
>>>> +++ b/kdump.sysconfig.s390x
>>>> @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE=""
>>>> # This variable lets us remove arguments from the current kdump
>>>> commandline
>>>> # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline
>>>> # NOTE: some arguments such as crashkernel will always be removed
>>>> -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug
quiet"
>>>> +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet
>>>> rd.earlykdump"
>>>>
>>>> # This variable lets us append arguments to the current kdump
>>>> commandline
>>>> # after processed by KDUMP_COMMANDLINE_REMOVE
>>>> diff --git a/kdump.sysconfig.x86_64 b/kdump.sysconfig.x86_64
>>>> index f269d02..3cccbfc 100644
>>>> --- a/kdump.sysconfig.x86_64
>>>> +++ b/kdump.sysconfig.x86_64
>>>> @@ -17,7 +17,7 @@ KDUMP_COMMANDLINE=""
>>>> # This variable lets us remove arguments from the current kdump
>>>> commandline
>>>> # as taken from either KDUMP_COMMANDLINE above, or from /proc/cmdline
>>>> # NOTE: some arguments such as crashkernel will always be removed
>>>> -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug
quiet"
>>>> +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet
>>>> rd.earlykdump"
>>>>
>>>> # This variable lets us append arguments to the current kdump
>>>> commandline
>>>> # after processed by KDUMP_COMMANDLINE_REMOVE
>>>> diff --git a/kdumpctl b/kdumpctl
>>>> index fe6af22..a8fd53d 100755
>>>> --- a/kdumpctl
>>>> +++ b/kdumpctl
>>>> @@ -26,7 +26,7 @@ image_time=0
>>>> standard_kexec_args="-p"
>>>>
>>>> # Some default values in case /etc/sysconfig/kdump doesn't include
>>>> -KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug"
>>>> +KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug
rd.earlykdump"
>>>>
>>>> if [ -f /etc/sysconfig/kdump ]; then
>>>> . /etc/sysconfig/kdump
>>>> @@ -942,6 +942,11 @@ check_default_config()
>>>> fi
>>>> }
>>>>
>>>> +check_rd_earlykdump()
>>>> +{
>>>> + egrep "rd.earlykdump" /proc/cmdline
>>>> +}
>>>> +
>>>> start()
>>>> {
>>>> check_dump_feasibility
>>>> @@ -969,7 +974,13 @@ start()
>>>> check_current_status
>>>> if [ $? == 0 ]; then
>>>> echo "Kdump already running: [WARNING]"
>>>> - return 0
>>>> + check_rd_earlykdump
>>>> + #if earlykdump loaded, it will stop and start.
>>>> + if [ $? -eq 0 ]; then
>>>> + stop
>>>
>>> kdumpctl start can run not only by system startup services, one can also
>>> run it manually or in udev rule.
>>>
>>> The checking of kernel cmdline seems not enough.
>>>
>>>> + else
>>>> + return 0
>>>> + fi
>>>> fi
>>>>
>>>> if check_ssh_config; then
>>>> --
>>>> 2.17.1
>>>>
>>>
>>> Thanks
>>> Dave
>>