On 12/21/18 4:14 PM, Dave Young wrote:
On 12/21/18 at 12:59pm, Buland Singh wrote:
> On 12/21/18 12:29 PM, Kairui Song wrote:
>> Hi, Dave, Lianbo
>>
>> My concern is that crash loop may generate tons of dump cores, and the
>> dump target may get filled up by dump cores,
>> that may have larger potential risk. Else I think it's good to leave
>> it as it is.
>>
>> On Fri, Dec 21, 2018 at 2:05 PM lijiang <lijiang(a)redhat.com> wrote:
>>>
>>> 在 2018年12月21日 10:49, Dave Young 写道:
>>>> + more people
>>>> On 12/20/18 at 04:49pm, lijiang wrote:
>>>>> 在 2018年12月20日 13:57, Dave Young 写道:
>>>>>> On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
>>>>>>> By default, early kdump reboots the system after capturing
the vmcore.
>>>>>>> If the problematic system is continuously crashing due to
some issue
>>>>>>> during early boot stage, the system may fall into infinite
loop restart
>>>>>>> like this:
>>>>>>>
>>>>>>> boot -----> crash -----> early kdump (dump
vmcore)
>>>>>>> ^ |
>>>>>>> '.........(reboot).............'
>>>>>>>
>>>>>>> But now, the system crash at early stage is only captured by
early kdump,
>>>>>>> and the rest is captured by normal kdump. That to say, when
normal kdump
>>>>>>> service starts, it will load it again and override early
kdump. It is
>>>>>>> helpful to control the logic of early kdump and normal kdump
separately
>>>>>>> in final action(it is called by kdump-capture.service). For
example,
>>>>>>> early kdump always passes the 'rd.earlykdump' to the
second kernel when
>>>>>>> early kdump is enabled, but normal kdump doesn't pass the
'rd.earlykdump'
>>>>>>> to the second kernel at any time. So they can be
distinguished in the
>>>>>>> second kernel.
>>>>>>
>>>>>> Hmm, I'm confused about the param passing above.
>>>>>>
>>>>>
>>>>> I copy some messages from another email, please refer to this one:
>>>>> [--->
>>>>> The rd.earlykdump is added to kernel command line in grub.cfg.
However, early kdump
>>>>> and normal kdump can get the same parameters from /proc/cmdline in
the first kernel.
>>>>>
>>>>> Early kdump passes the rd.earlykdump to the second kernel, but normal
kdump doesn't
>>>>> need it, normal kdump needs to remove the rd.earlykdump.
>>>>>
>>>>> So which can distinguish early kdump and normal kdump in the second
kernel. It helps
>>>>> to control the logic of kdump capture service. For example: default
action/final action.
>>>>> ]
>>>>
>>>> The description is confusing, "ealy kdump passes ... to the second
>>>> kernel", for example about this, the real thing is one person adds
the
>>>> param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd
>>>> kernel.
>>>>
>>>
>>> Yes. Good point. Thanks for your explanation.
>>>
>>>> Anyway this is patch log issue.
>>>>
>>>>>
>>>>>> Early or non early just means about the service loading phase,
in
>>>>>
>>>>> Yes. This patch used the same method what you said. When normal kdump
service starts,
>>>>> it will reload. At the same time, early kdump will be overwritten by
normal kdump.
>>>>
>>>> Probably "early kdump load" is better than "early
kdump" in words.
>>>>
>>>>>
>>>>>> initramfs or not, I notice dracut/systemd will print some message
about
>>>>>> they are running in initramfs, so probably you can check how to
get it
>>>>>> with same way, if this is not initramfs then just unload before
the
>>>>>> check in kdump loading.
>>>>>>
>>>>>> The picture like below:
>>>>>>
>>>>>> Kernel boot ->
>>>>>>
>>>>>> initramfs ---
>>>>>> early kdump load
>>>>>> ---- Mark A ----
>>>>>> initramfs switch root
>>>>>>
>>>>>> system startup (real root fs)
>>>>>> service a
>>>>>> service b ... (eg. networking etc.)
>>>>>> kdump service start
>>>>>> -----Mark B -----
>>>>>> load kdump kernel again
>>>>>>
>>>>>>
>>>>>> The problem will happen between Mark A and Mark B, during this
period,
>>>>>> there could be repeated crash -> earlykdump_load, there might
be some
>>>>>> random crash as well since during the real root fs service
startup,
>>>>>> for example after network is ready if some network workload cause
a
>>>>>> panic, it maybe not 100% reproducible, so it seems we still need
to
>>>>>> make the poweroff configurable. eg.
>>>>>>
>>>>>> default is poweroff, but one can choose if he can.
>>>>>>
>>>>>
>>>>> Yes, default is poweroff for early kdump. Unless kdump capture
service
>>>>> happens error or enters the emergency service, one can choose the
default
>>>>> action.(configure default=xxx in kdump.conf)
>>>>
>>>> For default action instead of final action if you hardcode it, then even
if
>>>> one set default as reboot it still poweroff.
>>>>
>>>
>>> If really need, that can be improved.
>>>
>>>> [snip]
>>>>
>>>>>>>
>>>>>>> +check_rd_earlykdump()
>>>>>>> +{
>>>>>>> + egrep "rd.earlykdump" /proc/cmdline
>>>>>>> +}
>>>>>>> +
>>>>>>> start()
>>>>>>> {
>>>>>>> check_dump_feasibility
>>>>>>> @@ -969,7 +974,13 @@ start()
>>>>>>> check_current_status
>>>>>>> if [ $? == 0 ]; then
>>>>>>> echo "Kdump already running:
[WARNING]"
>>>>>>> - return 0
>>>>>>> + check_rd_earlykdump
>>>>>>> + #if earlykdump loaded, it will stop and start.
>>>>>>> + if [ $? -eq 0 ]; then
>>>>>>> + stop
>>>>>>
>>>>>> kdumpctl start can run not only by system startup services, one
can also
>>>>>> run it manually or in udev rule.
>>>>>>
>>>>>> The checking of kernel cmdline seems not enough.
>>>>>>
>>>>>
>>>>> Here it means that if kdump has beend loaded, check whether early
kdump did it.
>>>>> If yes, let normal kdump load it again, otherwise no need to do
anything.
>>>>
>>>> As we discussed you do not get my points here :)
>>>>
>>>> check_rd_earlykdump will be always true once kernel bootup, so there is
>>>> no way to get the first time of normal kdump load and the later one.
>>>>
>>>> The early boot time panic to address for this patch is the 100%
>>>> reproducible panic, for this kinds of panic admin should be able to see
>>>> it when he boot the machine. So rethinking about this the best way may
>>>> be just a wontfix.
>>>>
>>>> Let's consider below use cases:
>>>>
>>>> * First install:
>>>>
>>>> install os ->
>>>> ---A
>>>> reboot ->
>>>> ...
>>>> ---B
>>>> kdump service start
>>>> -> create kdump initrd
>>>> ...
>>>> boot finished
>>>> recreate default initrd and enable early kdump
>>>> goto A
>>>>
>>>> Panic happened between A and B, if it is predictable, eg. 100%
>>>> reproducible, then admin should already see it, then he/she can control
>>>> and stop the repeating crash/kdump loop
>>>> if the panic is not 100% reproducible then use reboot as final action is
just fine
>>>>
>>>> * Other use cases eg. updating kernel or some other components:
>>>> It is similar with the intall os use case because if one update kernel
>>>> or critical components it is likely they need regenerate kdump initrd,
>>>> and then repack it into early kdump default initrd, in this case admin
>>>> should be able to see the panic loop and handle it.
>>>> Also if the panic is not 100% reproducible then we are just fine.
>>>>
>>>> If we choose to split early and late kdump load, there could be other
>>>> side effects, and make the logic even complicated.
>>>>
>>>> So... the better way may be just leave it as is, and maybe add some
>>>> documentation.
>>>>
>>>
>>> It is a good way to document the risks that may exist.
>>>
>>> Just like the public transportation, we all know that it has the risk, but we
still choose it.
>>>
>>> Thanks.
>>>
>>>> Thoughts?
>>>>
>>>> Thanks
>>>> Dave
>>>>
>>
>
> Hello Dave et al.
>
> Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
>
> Condition:
>
> [0x1] System is up and running with the kernel version X.
> [0x2] Admin performed the kernel Y upgrade.
> [0x3] Running kernel X crashed.
> [0x4] Normal kdump rebooted the system and captured the kernel crash dump of the
kernel X.
> [0x5] System rebooted with the newly installed kernel Y.
> [0x6] Let's assume that due to some unknown reason the booting kernel Y also
crashed (assume that the panic is 100% reproducible).
> [0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y.
> [0x8] Early kdump rebooted the system and stuck in the loop.
>
> Assumption: 1
>
> What if the problematic system is in data center and admin is not aware of this
situation?
>
> [0x1] The dump target will be filled with the multiple copies of the kernel crash
dump?
>
> [ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the
kernel Y]
>
> [0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump
load, it is also possible after one replaced a kernel the new kernel panics
during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Agree, but not sure if all the admins will follow this rule :)
> Assumption: 2
>
> What if the dump target in on the local disk?
>
> [0x1] Admin needs to power off the system manually to retrieve the kernel crash dump
of the kernel X and Y from the resume mode.
> [0x2] Admin needs to remove the multiple copies of the kernel crash dump of the
booting kernel Y.
> [0x3] Admin might get confused while differentiating between the kernel crash dump of
the kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than
happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good
solution.
But if we can assume predictable panic can be avoid then the situation
will be better.
One suggestion, can we have a separate default behavior for normal kdump and early kdump?
Eg:
In /etc/kdump.conf file.
default reboot (Default action to be taken by normal kdump and early kdump if dumping
fails)
ndefault reboot (Default action to be taken once the dumping is successful by normal
kdump)
edefault poweroff (Default action to be taken once the dumping is successful by early
kdump)
Note: Create 'ndefault' & 'edefault' options identical to
'default' option so that an admin can alter the behavior as per the requirement.
[0x1] If kernel crash dump is captured by normal kdump then 'reboot' the system
(by default) or take action as per 'ndefault' value.
[0x2] If kernel crash dump is captured by early kdump then 'poweroff' the system
9by default) or take action as per 'edefault' value.
--
Buland