在 2020年10月26日 16:04, Kairui Song 写道:
Hi Lianbo, thanks for the work, the log infrastruct will be improved
by a log with this patch series.
On Tue, Oct 20, 2020 at 10:45 AM Lianbo Jiang <lijiang(a)redhat.com> wrote:
>
> Sometimes, debugging the kdump service failure becomes very challenging
> because there is no complete debugging information, which requires
> modification of the options or the scripts like kdumpctl, mkdumprd, etc
> to collect the information for troubleshooting.
>
> That means users have to wait for the next failure so that they can
> capture the additional information, which could waste valuable time.
>
> This patch series will improve kdump debugging messages and have a
> chance to save them to journald or a file. It includes the following
> patches:
> [1] [PATCH 1/5] introduce the kdump logger from the dracut
> [2] [PATCH 2/5] enable the logger for kdump
> [3] [PATCH 3/5] kdump.sysconfig: add the kdump logger configurations
> [4] [PATCH 4/5] kdumpctl: add the '-d' option to enable the kexec loading
> debugging messages
> [5] [PATCH 5/5] Improve debugging in the kdump kernel
> [6] [PATCH 6/6] Doc: add a documentation for the usage of logger
>
> In addition, this patch series won't cause performance degradation,
> basically this effect can be ignored. I simply checked the result of
> executing the loading command as follow:
>
> [1] before
> # time kdumpctl start
> Detected change(s) in the following file(s):
> /etc/kdump.conf
> Rebuilding /boot/initramfs-4.18.0-232.el8.x86_64kdump.img
> Tips: If early kdump is enabled, also require rebuilding the system initramfs to make
the changes take effect for early kdump.
> kexec: loaded kdump kernel
> Starting kdump: [OK]
>
> real 0m9.145s
> user 0m20.733s
> sys 0m1.765s
>
> [2] after
> # time kdumpctl start
> kdump: Detected change(s) in the following file(s): /etc/kdump.conf
> kdump: Rebuilding /boot/initramfs-4.18.0-232.el8.x86_64kdump.img
> kdump: Tips: If early kdump is enabled, also require rebuilding the system initramfs
to make the changes take effect for early kdump.
> kdump: kexec: loaded kdump kernel
> kdump: Starting kdump: [OK]
>
> real 0m9.177s
> user 0m20.784s
> sys 0m1.901s
>
> Changes since v5:
> [1] Only added a documentation for the usage of logger, there's
> no code change.
>
> Note:
> Currently, kexec-tools disable storing logs to journald in order
> to save memory, please refer to the following changes:
> +++ b/dracut-module-setup.sh
> @@ -788,6 +788,8 @@ kdump_install_random_seed() {
> }
> ...
> kdump_install_systemd_conf() {
> if ! is_fadump_capable && [ "$failure_action" !=
"shell" ]; then
> mkdir -p ${initdir}/etc/systemd/journald.conf.d
> echo "[Journal]" >
${initdir}/etc/systemd/journald.conf.d/kdump.conf
> echo "Storage=none" >>
${initdir}/etc/systemd/journald.conf.d/kdump.conf
> ^^^^
> If we want to use the journald in kdump kernel, we have to change it to
> "Storage=auto" or "volatile".
For the log storage in kdump kernel, I'm thinking maybe we have
Storage=none and ForwardToKmsg=yes, so kmsg will be used as the sole
log storage in kdump kernel, which looks a good balance of keeping the
logs and reduce memory usage. How do you think? If you want to keep
We can get more additional logs from journald, but some logs are not in
the kmsg. For example:
kmsg(/var/crash/127.0.0.1-2020-10-26-22:38:26/kexec-dmesg.log):
--------------------------------------------------------------
...
[ 16.854646] kdump[387] saving to /sysroot/var/crash/127.0.0.1-2020-10-26-22:38:26/
[ 16.904723] kdump[387] saving vmcore-dmesg.txt to
/sysroot/var/crash/127.0.0.1-2020-10-26-22:38:26/
[ 16.986037] kdump[387] saving vmcore-dmesg.txt complete
[ 16.991214] kdump[387] saving vmcore
"/var/crash/127.0.0.1-2020-10-26-22:38:26/kexec-dmesg.log" 772L, 54046B
772,1 Bot
journald(/var/crash/127.0.0.1-2020-10-26-22:44:01/kexec-dmesg.log):
------------------------------------------------------------------
...
Oct 26 22:44:01
dell-pec5125-03.khw.lab.eng.bos.redhat.com systemd[1]: Starting Kdump
Vmcore Save Service...
Oct 26 22:44:02
dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[387]: kdump: saving to
/sysroot/var/crash/127.0.0.1-2020-10-26-22:44:01/
Oct 26 22:44:02
dell-pec5125-03.khw.lab.eng.bos.redhat.com unknown: kdump[387] saving to
/sysroot/var/crash/127.0.0.1-2020-10-26-22:44:01/
Oct 26 22:44:02
dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[387]: kdump: saving
vmcore-dmesg.txt to /sysroot/var/crash/127.0.0.1-2020-10-26-22:44:01/
Oct 26 22:44:02
dell-pec5125-03.khw.lab.eng.bos.redhat.com unknown: kdump[387] saving
vmcore-dmesg.txt to /sysroot/var/crash/127.0.0.1-2020-10-26-22:44:01/
Oct 26 22:44:02
dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[387]: kdump: saving
vmcore-dmesg.txt complete
Oct 26 22:44:02
dell-pec5125-03.khw.lab.eng.bos.redhat.com unknown: kdump[387] saving
vmcore-dmesg.txt complete
Oct 26 22:44:02
dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[387]: kdump: saving
vmcore
Oct 26 22:44:02
dell-pec5125-03.khw.lab.eng.bos.redhat.com unknown: kdump[387] saving
vmcore
Oct 26 22:44:04
dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[445]: ^MChecking for
memory holes : [ 0.0 %] / ^MChecking for memory
holes : [100.0 %] | ^MExcluding unnecessary pages
: [100.0 %] \ ^MCopying data
: [ 1.4 %] - eta: 2s^MCopying data
: [ 52.9 %] / eta: 0s^MCopying data :
[100.0 %] | eta: 0s^MCopying data : [100.0
%] \ eta: 0s
Oct 26 22:44:04
dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[445]: The kernel
version is not supported.
Oct 26 22:44:04
dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[445]: The makedumpfile
operation may be incomplete.
Oct 26 22:44:04
dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[445]: The dumpfile is
saved to /sysroot/var/crash/127.0.0.1-2020-10-26-22:44:01//vmcore-incomplete.
Oct 26 22:44:04
dell-pec5125-03.khw.lab.eng.bos.redhat.com kdump.sh[445]: makedumpfile
Completed.
"/var/crash/127.0.0.1-2020-10-26-22:44:01/kexec-dmesg.log" 858L, 106493B
more log, just increase log_bufsize, simple and easy.
If we manually increase the size of log_buf, which indicates that we
have to wait for the next failure to capture the additional information.
In addition, increasing the size of log_buf still increases memory usage,
and the log_buf is easy to be overwritten if the size is not enough.
Maybe we can have a is_kdump_kernel helper in kdump-libs.sh, to
judge
if it's kdump kernel.
Also, how about have a standalong log level like kdump_dumploglvl,
I would not
recommend having a standalong log level for kdump. Currently, we
can use the following configurations to achieve the same purpose.
kdump_sysloglvl=0
kdump_stdloglvl=4
kdump_kmsgloglvl=4
Thanks.
Lianbo
aside from "kdump_stdloglvl kdump_sysloglvl
kdump_kmsgloglvl".
kdump_dumploglvl could be used as the only loglevel in kdump kernel,
and all log content just goes to logger (journalctl, it will forward
log to console, and store log in kmsg), as I think ideally, there is
only one log output/storage in kdump kernel anyway (log is always
printed to console, and stored in kmsg). Then to save the log in kdump
kernel, just use dmsg.
I've also added some comment in other individual patches.
>
> Lianbo Jiang (6):
> introduce the kdump logger from the dracut
> enable the logger for kdump
> kdump.sysconfig: add the kdump logger configurations
> kdumpctl: add the '-d' option to enable the kexec loading debugging
> messages
> Improve debugging in the kdump kernel
> Doc: add a documentation for the usage of logger
>
> dracut-early-kdump-module-setup.sh | 3 +
> dracut-early-kdump.sh | 24 +-
> dracut-kdump.sh | 68 +++---
> dracut-module-setup.sh | 4 +
> kdump-lib-initramfs.sh | 67 ++++--
> kdump-lib.sh | 23 +-
> kdump-logger.sh | 344 +++++++++++++++++++++++++++++
> kdump.sysconfig | 16 ++
> kdump.sysconfig.aarch64 | 16 ++
> kdump.sysconfig.i386 | 16 ++
> kdump.sysconfig.ppc64 | 16 ++
> kdump.sysconfig.ppc64le | 16 ++
> kdump.sysconfig.s390x | 16 ++
> kdump.sysconfig.x86_64 | 16 ++
> kdumpctl | 241 +++++++++++---------
> kexec-kdump-howto.txt | 35 +++
> kexec-tools.spec | 2 +
> mkdumprd | 12 +-
> 18 files changed, 766 insertions(+), 169 deletions(-)
> create mode 100755 kdump-logger.sh
>
> --
> 2.17.1
>
--
Best Regards,
Kairui Song