On 12/21/18 at 12:59pm, Buland Singh wrote:
On 12/21/18 12:29 PM, Kairui Song wrote:
> Hi, Dave, Lianbo
>
> My concern is that crash loop may generate tons of dump cores, and the
> dump target may get filled up by dump cores,
> that may have larger potential risk. Else I think it's good to leave
> it as it is.
>
> On Fri, Dec 21, 2018 at 2:05 PM lijiang <lijiang(a)redhat.com> wrote:
> >
> > 在 2018年12月21日 10:49, Dave Young 写道:
> > > + more people
> > > On 12/20/18 at 04:49pm, lijiang wrote:
> > > > 在 2018年12月20日 13:57, Dave Young 写道:
> > > > > On 12/20/18 at 01:06pm, Lianbo Jiang wrote:
> > > > > > By default, early kdump reboots the system after capturing
the vmcore.
> > > > > > If the problematic system is continuously crashing due to
some issue
> > > > > > during early boot stage, the system may fall into infinite
loop restart
> > > > > > like this:
> > > > > >
> > > > > > boot -----> crash -----> early kdump (dump
vmcore)
> > > > > > ^ |
> > > > > > '.........(reboot).............'
> > > > > >
> > > > > > But now, the system crash at early stage is only captured
by early kdump,
> > > > > > and the rest is captured by normal kdump. That to say, when
normal kdump
> > > > > > service starts, it will load it again and override early
kdump. It is
> > > > > > helpful to control the logic of early kdump and normal
kdump separately
> > > > > > in final action(it is called by kdump-capture.service). For
example,
> > > > > > early kdump always passes the 'rd.earlykdump' to
the second kernel when
> > > > > > early kdump is enabled, but normal kdump doesn't pass
the 'rd.earlykdump'
> > > > > > to the second kernel at any time. So they can be
distinguished in the
> > > > > > second kernel.
> > > > >
> > > > > Hmm, I'm confused about the param passing above.
> > > > >
> > > >
> > > > I copy some messages from another email, please refer to this one:
> > > > [--->
> > > > The rd.earlykdump is added to kernel command line in grub.cfg.
However, early kdump
> > > > and normal kdump can get the same parameters from /proc/cmdline in
the first kernel.
> > > >
> > > > Early kdump passes the rd.earlykdump to the second kernel, but normal
kdump doesn't
> > > > need it, normal kdump needs to remove the rd.earlykdump.
> > > >
> > > > So which can distinguish early kdump and normal kdump in the second
kernel. It helps
> > > > to control the logic of kdump capture service. For example: default
action/final action.
> > > > ]
> > >
> > > The description is confusing, "ealy kdump passes ... to the second
> > > kernel", for example about this, the real thing is one person adds
the
> > > param in 1st kernel cmdline, kexec-tools takes/inherits and pass to 2nd
> > > kernel.
> > >
> >
> > Yes. Good point. Thanks for your explanation.
> >
> > > Anyway this is patch log issue.
> > >
> > > >
> > > > > Early or non early just means about the service loading phase,
in
> > > >
> > > > Yes. This patch used the same method what you said. When normal kdump
service starts,
> > > > it will reload. At the same time, early kdump will be overwritten by
normal kdump.
> > >
> > > Probably "early kdump load" is better than "early
kdump" in words.
> > >
> > > >
> > > > > initramfs or not, I notice dracut/systemd will print some
message about
> > > > > they are running in initramfs, so probably you can check how to
get it
> > > > > with same way, if this is not initramfs then just unload before
the
> > > > > check in kdump loading.
> > > > >
> > > > > The picture like below:
> > > > >
> > > > > Kernel boot ->
> > > > >
> > > > > initramfs ---
> > > > > early kdump load
> > > > > ---- Mark A ----
> > > > > initramfs switch root
> > > > >
> > > > > system startup (real root fs)
> > > > > service a
> > > > > service b ... (eg. networking etc.)
> > > > > kdump service start
> > > > > -----Mark B -----
> > > > > load kdump kernel again
> > > > >
> > > > >
> > > > > The problem will happen between Mark A and Mark B, during this
period,
> > > > > there could be repeated crash -> earlykdump_load, there
might be some
> > > > > random crash as well since during the real root fs service
startup,
> > > > > for example after network is ready if some network workload
cause a
> > > > > panic, it maybe not 100% reproducible, so it seems we still
need to
> > > > > make the poweroff configurable. eg.
> > > > >
> > > > > default is poweroff, but one can choose if he can.
> > > > >
> > > >
> > > > Yes, default is poweroff for early kdump. Unless kdump capture
service
> > > > happens error or enters the emergency service, one can choose the
default
> > > > action.(configure default=xxx in kdump.conf)
> > >
> > > For default action instead of final action if you hardcode it, then even
if
> > > one set default as reboot it still poweroff.
> > >
> >
> > If really need, that can be improved.
> >
> > > [snip]
> > >
> > > > > >
> > > > > > +check_rd_earlykdump()
> > > > > > +{
> > > > > > + egrep "rd.earlykdump" /proc/cmdline
> > > > > > +}
> > > > > > +
> > > > > > start()
> > > > > > {
> > > > > > check_dump_feasibility
> > > > > > @@ -969,7 +974,13 @@ start()
> > > > > > check_current_status
> > > > > > if [ $? == 0 ]; then
> > > > > > echo "Kdump already running:
[WARNING]"
> > > > > > - return 0
> > > > > > + check_rd_earlykdump
> > > > > > + #if earlykdump loaded, it will stop and start.
> > > > > > + if [ $? -eq 0 ]; then
> > > > > > + stop
> > > > >
> > > > > kdumpctl start can run not only by system startup services, one
can also
> > > > > run it manually or in udev rule.
> > > > >
> > > > > The checking of kernel cmdline seems not enough.
> > > > >
> > > >
> > > > Here it means that if kdump has beend loaded, check whether early
kdump did it.
> > > > If yes, let normal kdump load it again, otherwise no need to do
anything.
> > >
> > > As we discussed you do not get my points here :)
> > >
> > > check_rd_earlykdump will be always true once kernel bootup, so there is
> > > no way to get the first time of normal kdump load and the later one.
> > >
> > > The early boot time panic to address for this patch is the 100%
> > > reproducible panic, for this kinds of panic admin should be able to see
> > > it when he boot the machine. So rethinking about this the best way may
> > > be just a wontfix.
> > >
> > > Let's consider below use cases:
> > >
> > > * First install:
> > >
> > > install os ->
> > > ---A
> > > reboot ->
> > > ...
> > > ---B
> > > kdump service start
> > > -> create kdump initrd
> > > ...
> > > boot finished
> > > recreate default initrd and enable early kdump
> > > goto A
> > >
> > > Panic happened between A and B, if it is predictable, eg. 100%
> > > reproducible, then admin should already see it, then he/she can control
> > > and stop the repeating crash/kdump loop
> > > if the panic is not 100% reproducible then use reboot as final action is
just fine
> > >
> > > * Other use cases eg. updating kernel or some other components:
> > > It is similar with the intall os use case because if one update kernel
> > > or critical components it is likely they need regenerate kdump initrd,
> > > and then repack it into early kdump default initrd, in this case admin
> > > should be able to see the panic loop and handle it.
> > > Also if the panic is not 100% reproducible then we are just fine.
> > >
> > > If we choose to split early and late kdump load, there could be other
> > > side effects, and make the logic even complicated.
> > >
> > > So... the better way may be just leave it as is, and maybe add some
> > > documentation.
> > >
> >
> > It is a good way to document the risks that may exist.
> >
> > Just like the public transportation, we all know that it has the risk, but we
still choose it.
> >
> > Thanks.
> >
> > > Thoughts?
> > >
> > > Thanks
> > > Dave
> > >
>
Hello Dave et al.
Kindly check the below condition, assumption and share your thoughts.
Buland, thank you for the reply.
Condition:
[0x1] System is up and running with the kernel version X.
[0x2] Admin performed the kernel Y upgrade.
[0x3] Running kernel X crashed.
[0x4] Normal kdump rebooted the system and captured the kernel crash dump of the kernel
X.
[0x5] System rebooted with the newly installed kernel Y.
[0x6] Let's assume that due to some unknown reason the booting kernel Y also crashed
(assume that the panic is 100% reproducible).
[0x7] Early kdump started dumping the kernel crash dump of the booting kernel Y.
[0x8] Early kdump rebooted the system and stuck in the loop.
Assumption: 1
What if the problematic system is in data center and admin is not aware of this
situation?
[0x1] The dump target will be filled with the multiple copies of the kernel crash dump?
[ 1 kernel crash dump of the kernel X and 'n' kernel crash dump of the
kernel Y]
[0x2] The system will reboot in the loop?
It is hard to define, for original kdump service without early kdump
load, it is also possible after one replaced a kernel the new kernel panics
during boot phase just after kdump get loaded.
Admin at least should test a reboot after updating a kernel?
Assumption: 2
What if the dump target in on the local disk?
[0x1] Admin needs to power off the system manually to retrieve the kernel crash dump of
the kernel X and Y from the resume mode.
[0x2] Admin needs to remove the multiple copies of the kernel crash dump of the booting
kernel Y.
[0x3] Admin might get confused while differentiating between the kernel crash dump of the
kernel X and Kernel Y.
As for the worst case we all admit this is a problem, we are more than
happy to make admin be easier and fix it :)
As I said we are exporting about this problem see if we have a good
solution.
But if we can assume predictable panic can be avoid then the situation
will be better.
Regards,
Buland
Thanks
Dave