On 12/21/18 12:29 PM, Kairui Song wrote:
Hi, Dave, Lianbo
My concern is that crash loop may generate tons of dump cores, and the
dump target may get filled up by dump cores,
that may have larger potential risk. Else I think it's good to leave
it as it is.
On Fri, Dec 21, 2018 at 2:05 PM lijiang <lijiang(a)redhat.com> wrote:
>
> 在 2018年12月21日 10:49, Dave Young 写道:
>> + more people
>> On 12/20/18 at 04:49pm, lijiang wrote:
>>> 在 2018年12月20日 13:57, Dave Young 写道:
>>>> On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
>>>>> By default, early kdump reboots the system after capturing the
vmcore.
>>>>> If the problematic system is continuously crashing due to some issue
>>>>> during early boot stage, the system may fall into infinite loop
restart
>>>>> like this:
>>>>>
>>>>> boot -----> crash -----> early kdump (dump vmcore)
>>>>> ^ |
>>>>> '.........(reboot).............'
>>>>>
>>>>> But now, the system crash at early stage is only captured by early
kdump,
>>>>> and the rest is captured by normal kdump. That to say, when normal
kdump
>>>>> service starts, it will load it again and override early kdump. It
is
>>>>> helpful to control the logic of early kdump and normal kdump
separately
>>>>> in final action(it is called by kdump-capture.service). For example,
>>>>> early kdump always passes the 'rd.earlykdump' to the second
kernel when
>>>>> early kdump is enabled, but normal kdump doesn't pass the
'rd.earlykdump'
>>>>> to the second kernel at any time. So they can be distinguished in
the
>>>>> second kernel.
>>>>
>>>> Hmm, I'm confused about the param passing above.
>>>>
>>>
>>> I copy some messages from another email, please refer to this one:
>>> [--->
>>> The rd.earlykdump is added to kernel command line in grub.cfg. However, early
kdump
>>> and normal kdump can get the same parameters from /proc/cmdline in the first
kernel.
>>>
>>> Early kdump passes the rd.earlykdump to the second kernel, but normal kdump
doesn't
>>> need it, normal kdump needs to remove the rd.earlykdump.
>>>
>>> So which can distinguish early kdump and normal kdump in the second kernel.
It helps
>>> to control the logic of kdump capture service. For example: default
action/final action.
>>> ]
>>
>> The description is confusing, "ealy kdump passes ... to the second
>> kernel", for example about this, the real thing is one person adds the
>> param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd
>> kernel.
>>
>
> Yes. Good point. Thanks for your explanation.
>
>> Anyway this is patch log issue.
>>
>>>
>>>> Early or non early just means about the service loading phase, in
>>>
>>> Yes. This patch used the same method what you said. When normal kdump service
starts,
>>> it will reload. At the same time, early kdump will be overwritten by normal
kdump.
>>
>> Probably "early kdump load" is better than "early kdump" in
words.
>>
>>>
>>>> initramfs or not, I notice dracut/systemd will print some message about
>>>> they are running in initramfs, so probably you can check how to get it
>>>> with same way, if this is not initramfs then just unload before the
>>>> check in kdump loading.
>>>>
>>>> The picture like below:
>>>>
>>>> Kernel boot ->
>>>>
>>>> initramfs ---
>>>> early kdump load
>>>> ---- Mark A ----
>>>> initramfs switch root
>>>>
>>>> system startup (real root fs)
>>>> service a
>>>> service b ... (eg. networking etc.)
>>>> kdump service start
>>>> -----Mark B -----
>>>> load kdump kernel again
>>>>
>>>>
>>>> The problem will happen between Mark A and Mark B, during this period,
>>>> there could be repeated crash -> earlykdump_load, there might be
some
>>>> random crash as well since during the real root fs service startup,
>>>> for example after network is ready if some network workload cause a
>>>> panic, it maybe not 100% reproducible, so it seems we still need to
>>>> make the poweroff configurable. eg.
>>>>
>>>> default is poweroff, but one can choose if he can.
>>>>
>>>
>>> Yes, default is poweroff for early kdump. Unless kdump capture service
>>> happens error or enters the emergency service, one can choose the default
>>> action.(configure default=xxx in kdump.conf)
>>
>> For default action instead of final action if you hardcode it, then even if
>> one set default as reboot it still poweroff.
>>
>
> If really need, that can be improved.
>
>> [snip]
>>
>>>>>
>>>>> +check_rd_earlykdump()
>>>>> +{
>>>>> + egrep "rd.earlykdump" /proc/cmdline
>>>>> +}
>>>>> +
>>>>> start()
>>>>> {
>>>>> check_dump_feasibility
>>>>> @@ -969,7 +974,13 @@ start()
>>>>> check_current_status
>>>>> if [ $? == 0 ]; then
>>>>> echo "Kdump already running: [WARNING]"
>>>>> - return 0
>>>>> + check_rd_earlykdump
>>>>> + #if earlykdump loaded, it will stop and start.
>>>>> + if [ $? -eq 0 ]; then
>>>>> + stop
>>>>
>>>> kdumpctl start can run not only by system startup services, one can also
>>>> run it manually or in udev rule.
>>>>
>>>> The checking of kernel cmdline seems not enough.
>>>>
>>>
>>> Here it means that if kdump has beend loaded, check whether early kdump did
it.
>>> If yes, let normal kdump load it again, otherwise no need to do anything.
>>
>> As we discussed you do not get my points here :)
>>
>> check_rd_earlykdump will be always true once kernel bootup, so there is
>> no way to get the first time of normal kdump load and the later one.
>>
>> The early boot time panic to address for this patch is the 100%
>> reproducible panic, for this kinds of panic admin should be able to see
>> it when he boot the machine. So rethinking about this the best way may
>> be just a wontfix.
>>
>> Let's consider below use cases:
>>
>> * First install:
>>
>> install os ->
>> ---A
>> reboot ->
>> ...
>> ---B
>> kdump service start
>> -> create kdump initrd
>> ...
>> boot finished
>> recreate default initrd and enable early kdump
>> goto A
>>
>> Panic happened between A and B, if it is predictable, eg. 100%
>> reproducible, then admin should already see it, then he/she can control
>> and stop the repeating crash/kdump loop
>> if the panic is not 100% reproducible then use reboot as final action is just
fine
>>
>> * Other use cases eg. updating kernel or some other components:
>> It is similar with the intall os use case because if one update kernel
>> or critical components it is likely they need regenerate kdump initrd,
>> and then repack it into early kdump default initrd, in this case admin
>> should be able to see the panic loop and handle it.
>> Also if the panic is not 100% reproducible then we are just fine.
>>
>> If we choose to split early and late kdump load, there could be other
>> side effects, and make the logic even complicated.
>>
>> So... the better way may be just leave it as is, and maybe add some
>> documentation.
>>
>
> It is a good way to document the risks that may exist.
>
> Just like the public transportation, we all know that it has the risk, but we still
choose it.
>
> Thanks.
>
>> Thoughts?
>>
>> Thanks
>> Dave
>>
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Condition:
[0x1] System is up and running with the kernel version X.
[0x2] Admin performed the kernel Y upgrade.
[0x3] Running kernel X crashed.
[0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel
X.
[0x5] System rebooted with the newly installed kernel Y.
[0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed
(assume that the panic is 100% reproducible).
[0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y.
[0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this
situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the
kernel Y]
[0x2] The system will reboot in the loop?
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of
the kernel X and Y from the resume mode.
[0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting
kernel Y.
[0x3] Admin might get confused while differentiating between the kernel crash dump of the
kernel X and Kernel Y.
Regards,
Buland