In RHEL, kernel supports a `crashkernel=auto` param and when it's used, kernel will use its built-in default value as the crashkernel value. But upstream doesn't like this idea. [1]
`crashkernel=auto` was introduced to provide user a default value, and make it possible to update the crashkernel value automatically as kernel package updates if user is using the default `auto` value.
After more discussion, `crashkernel=auto` will be dropped. To track the default crashkernel value, kernel package will include a crashkernel.default file with it[2], and kexec-tools can make use of that file, update kernel's boot cmdline accordingly, and help user to reset the crashkernel value to default value.
[1]: https://lore.kernel.org/linux-mm/20210507010432.IN24PudKT%25akpm@linux-found... [2]: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171
Update from V2: - Update the doc as suggested by Baoquan - Fix some comment and code issues in 92-crashkernel.install
Update from V1: - Add support for ostree - Add a crashkernel-howto.txt - Change crashkernel.conf to crashkernel.default
Kairui Song (4): Revert "kdump-lib.sh: Remove is_atomic" kdumpctl: Add kdumpctl reset-crashkernel Add a new hook: 92-crashkernel.install Add a crashkernel-howto.conf doc
92-crashkernel.install | 143 +++++++++++++++++++++++++++++ crashkernel-howto.conf | 204 +++++++++++++++++++++++++++++++++++++++++ kdump-lib.sh | 5 + kdumpctl | 41 ++++++++- kdumpctl.8 | 9 ++ kexec-tools.spec | 7 ++ 6 files changed, 407 insertions(+), 2 deletions(-) create mode 100755 92-crashkernel.install create mode 100644 crashkernel-howto.conf
Now we need this helper again, for `reset-crashkernel`
This reverts commit ff46cfb19e9bf944a114164e33203b1a532d35f6. --- kdump-lib.sh | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/kdump-lib.sh b/kdump-lib.sh index 8e618f8..df0aa02 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -341,6 +341,11 @@ kdump_get_persistent_dev() { echo $(get_persistent_dev "$dev") }
+is_atomic() +{ + grep -q "ostree" /proc/cmdline +} + is_ipv6_address() { echo $1 | grep -q ":"
In newer kernel, crashkernel.default will contain the default crashkernel value of a kernel build. So introduce a new sub command to help user reset kernel crashkernel size to the default value.
Signed-off-by: Kairui Song kasong@redhat.com --- kdumpctl | 41 +++++++++++++++++++++++++++++++++++++++-- kdumpctl.8 | 9 +++++++++ 2 files changed, 48 insertions(+), 2 deletions(-)
diff --git a/kdumpctl b/kdumpctl index e4334c5..c5b210c 100755 --- a/kdumpctl +++ b/kdumpctl @@ -1330,6 +1330,40 @@ do_estimate() { fi }
+reset_crashkernel() { + local kernel=$1 entry crashkernel_default + local grub_etc_default="/etc/default/grub" + + [[ -z "$kernel" ]] && kernel=$(uname -r) + crashkernel_default=$(cat "/usr/lib/modules/$kernel/crashkernel.default" 2>/dev/null) + + if [[ -z "$crashkernel_default" ]]; then + derror "$kernel doesn't have a crashkernel.default" + exit 1 + fi + + if is_atomic; then + if rpm-ostree kargs | grep -q "crashkernel="; then + rpm-ostree --replace="crashkernel=$crashkernel_default" + else + rpm-ostree --append="crashkernel=$crashkernel_default" + fi + else + entry=$(grubby --info ALL | grep "^kernel=.*$kernel") + entry=${entry#kernel=} + entry=${entry#"} + entry=${entry%"} + + if [[ -f "$grub_etc_default" ]]; then + sed -i -e "s/^(GRUB_CMDLINE_LINUX=.*)crashkernel=[^\ "]*([\ "].*)$/\1$crashkernel_default\2/" "$grub_etc_default" + fi + + [[ -f /etc/zipl.conf ]] && zipl_arg="--zipl" + grubby --args "$crashkernel_default" --update-kernel "$entry" $zipl_arg + [[ $zipl_arg ]] && zipl > /dev/null + fi + } + if [ ! -f "$KDUMP_CONFIG_FILE" ]; then derror "Error: No kdump config file found!" exit 1 @@ -1388,8 +1422,11 @@ main () estimate) do_estimate ;; + reset-crashkernel) + reset_crashkernel "$2" + ;; *) - dinfo $"Usage: $0 {estimate|start|stop|status|restart|reload|rebuild|propagate|showmem}" + dinfo $"Usage: $0 {estimate|start|stop|status|restart|reload|rebuild|reset-crashkernel|propagate|showmem}" exit 1 esac } @@ -1399,6 +1436,6 @@ single_instance_lock
# To avoid fd 9 leaking, we invoke a subshell, close fd 9 and call main. # So that fd isn't leaking when main is invoking a subshell. -(exec 9<&-; main $1) +(exec 9<&-; main "$@")
exit $? diff --git a/kdumpctl.8 b/kdumpctl.8 index a32a972..74be062 100644 --- a/kdumpctl.8 +++ b/kdumpctl.8 @@ -49,6 +49,15 @@ Prints the size of reserved memory for crash kernel in megabytes. Estimate a suitable crashkernel value for current machine. This is a best-effort estimate. It will print a recommanded crashkernel value based on current kdump setup, and list some details of memory usage. +.TP +.I reset-crashkernel [KERNEL] +Reset crashkernel value to default value. kdumpctl will try to read +from /usr/lib/modules/<KERNEL>/crashkernel.default and reset specified +kernel's crashkernel cmdline value. If no kernel is +specified, will reset current running kernel's crashkernel value. +If /usr/lib/modules/<KERNEL>/crashkernel.default doesn't exist, will +simply exit return 1. +
.SH "SEE ALSO" .BR kdump.conf (5),
To track and manage kernel's crashkernel usage by kernel version, each kernel package will include a crashkernel.default containing the default `crashkernel=` value of that kernel. So we can use a hook to update the kernel cmdline of new installed kernel accordingly.
Put it after all other grub boot loader setup hooks, so it can simply call grubby to modify the kernel cmdline.
Signed-off-by: Kairui Song kasong@redhat.com --- 92-crashkernel.install | 143 +++++++++++++++++++++++++++++++++++++++++ kexec-tools.spec | 4 ++ 2 files changed, 147 insertions(+) create mode 100755 92-crashkernel.install
diff --git a/92-crashkernel.install b/92-crashkernel.install new file mode 100755 index 0000000..d12f32e --- /dev/null +++ b/92-crashkernel.install @@ -0,0 +1,143 @@ +#!/usr/bin/bash + +COMMAND="$1" +KERNEL_VERSION="$2" +KDUMP_INITRD_DIR_ABS="$3" +KERNEL_IMAGE="$4" + +grub_etc_default="/etc/default/grub" + +ver_lt() { + [[ "$(echo -e "$1\n$2" | sort -V)" == $1$'\n'* ]] && [[ $1 != "$2" ]] +} + +# Read crashkernel= value in /etc/default/grub +get_grub_etc_ck() { + [[ -e $grub_etc_default ]] && \ + sed -n -e "s/^GRUB_CMDLINE_LINUX=.*(crashkernel=[^\ "]*)[\ "].*$/\1/p" $grub_etc_default +} + +# Read crashkernel.default value of specified kernel +get_ck_default() { + ck_file="/usr/lib/modules/$1/crashkernel.default" + [[ -f "$ck_file" ]] && cat "$ck_file" +} + +# Iterate installed kernels, find the kernel with the highest version that has a +# valid crashkernel.default file, exclude current installing/removing kernel +# +# $1: a string representing a crashkernel= cmdline. If given, will also check the +# content of crashkernel.default, only crashkernel.default with the same value will match +get_highest_ck_default_kver() { + for kernel in $(find /usr/lib/modules -maxdepth 1 -mindepth 1 -printf "%f\n" | sort --version-sort -r); do + [[ $kernel == "$KERNEL_VERSION" ]] && continue + [[ -s "/usr/lib/modules/$kernel/crashkernel.default" ]] || continue + + echo "$kernel" + return 0 + done + + return 1 +} + +set_grub_ck() { + sed -i -e "s/^(GRUB_CMDLINE_LINUX=.*)crashkernel=[^\ "]*([\ "].*)$/\1$1\2/" "$grub_etc_default" +} + +# Set specified kernel's crashkernel cmdline value +set_kernel_ck() { + kernel=$1 + ck_cmdline=$2 + + entry=$(grubby --info ALL | grep "^kernel=.*$kernel") + entry=${entry#kernel=} + entry=${entry#"} + entry=${entry%"} + + if [[ -z "$entry" ]]; then + echo "$0: failed to find boot entry for kernel $kernel" + return 1 + fi + + [[ -f /etc/zipl.conf ]] && zipl_arg="--zipl" + grubby --args "$ck_cmdline" --update-kernel "$entry" $zipl_arg + [[ $zipl_arg ]] && zipl > /dev/null +} + +case "$COMMAND" in +add) + # - If current boot kernel is using default crashkernel value, update + # installing kernel's crashkernel value to its default value, + # - If intalling a higher version kernel, and /etc/default/grub's + # crashkernel value is using default value, update it to installing + # installing kernel's default value. + inst_ck_default=$(get_ck_default "$KERNEL_VERSION") + # If installing kernel doesn't have crashkernel.default, just exit. + [[ -z "$inst_ck_default" ]] && exit 0 + + boot_kernel=$(uname -r) + boot_ck_cmdline=$(sed -n -e "s/^.*(crashkernel=\S*).*$/\1/p" /proc/cmdline) + highest_ck_default_kver=$(get_highest_ck_default_kver) + highest_ck_default=$(get_ck_default "$highest_ck_default_kver") + + # Try update /etc/default/grub if present, else grub2-mkconfig could + # override crashkernel value. + grub_etc_ck=$(get_grub_etc_ck) + if [[ -n "$grub_etc_ck" ]]; then + if [[ -z "$highest_ck_default_kver" ]]; then + # None of installed kernel have a crashkernel.default, + # check for 'crashkernel=auto' in case of legacy kernel + [[ "$grub_etc_ck" == "crashkernel=auto" ]] && \ + set_grub_ck "$inst_ck_default" + else + # There is a valid crashkernel.default, check if installing kernel + # have a higher version and grub config is using default value + ver_lt "$highest_ck_default_kver" "$KERNEL_VERSION" && \ + [[ "$grub_etc_ck" == "$highest_ck_default" ]] && \ + [[ "$grub_etc_ck" != "$inst_ck_default" ]] && \ + set_grub_ck "$inst_ck_default" + fi + fi + + # Exit if crashkernel is not used in current cmdline + [[ -z $boot_ck_cmdline ]] && exit 0 + + # Get current boot kernel's default value + boot_ck_default=$(get_ck_default "$boot_kernel") + if [[ $boot_ck_cmdline == "crashkernel=auto" ]]; then + # Legacy RHEL kernel defaults to "auto" + boot_ck_default="$boot_ck_cmdline" + fi + + # If boot kernel doesn't have a crashkernel.default, check + # if it's using any installed kernel's crashkernel.default + if [[ -z $boot_ck_default ]]; then + [[ $(get_highest_ck_default_kver "$boot_ck_cmdline") ]] && boot_ck_default="$boot_ck_cmdline" + fi + + # If boot kernel is using a default crashkernel, update + # installing kernel's crashkernel to new default value + if [[ "$boot_ck_cmdline" != "$inst_ck_default" ]] && [[ "$boot_ck_cmdline" == "$boot_ck_default" ]]; then + set_kernel_ck "$KERNEL_VERSION" "$inst_ck_default" + fi + ;; + +remove) + # If grub default value is upgraded when this kernel was installed, try downgrade it + grub_etc_ck=$(get_grub_etc_ck) + [[ $grub_etc_ck ]] || exit 0 + + removing_ck_conf=$(get_ck_default "$KERNEL_VERSION") + [[ $removing_ck_conf ]] || exit 0 + + highest_ck_default_kver=$(get_highest_ck_default_kver) || exit 0 + highest_ck_default=$(get_ck_default "$highest_ck_default_kver") + [[ $highest_ck_default ]] || exit 0 + + if ver_lt "$highest_ck_default_kver" "$KERNEL_VERSION"; then + if [[ $grub_etc_ck == "$removing_ck_conf" ]] && [[ $grub_etc_ck != "$highest_ck_default" ]]; then + set_grub_ck "$highest_ck_default" + fi + fi + ;; +esac diff --git a/kexec-tools.spec b/kexec-tools.spec index 8fed959..ec327da 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -40,6 +40,7 @@ Source29: kdump.sysconfig.aarch64 Source30: 60-kdump.install Source31: kdump-logger.sh Source32: mkfadumprd +Source33: 92-crashkernel.install
####################################### # These are sources for mkdumpramfs @@ -66,6 +67,7 @@ Requires: dracut >= 050 Requires: dracut-network >= 050 Requires: dracut-squash >= 050 Requires: ethtool +Requires: grubby BuildRequires: make BuildRequires: zlib-devel elfutils-devel glib2-devel bzip2-devel ncurses-devel bison flex lzo-devel snappy-devel BuildRequires: pkgconfig intltool gettext @@ -210,6 +212,7 @@ install -m 644 %{SOURCE15} $RPM_BUILD_ROOT%{_mandir}/man5/kdump.conf.5 install -m 644 %{SOURCE16} $RPM_BUILD_ROOT%{_unitdir}/kdump.service install -m 755 -D %{SOURCE22} $RPM_BUILD_ROOT%{_prefix}/lib/systemd/system-generators/kdump-dep-generator.sh install -m 755 -D %{SOURCE30} $RPM_BUILD_ROOT%{_prefix}/lib/kernel/install.d/60-kdump.install +install -m 755 -D %{SOURCE32} $RPM_BUILD_ROOT%{_prefix}/lib/kernel/install.d/92-crashkernel.install
%ifarch %{ix86} x86_64 ppc64 s390x ppc64le aarch64 install -m 755 makedumpfile-%{mkdf_ver}/makedumpfile $RPM_BUILD_ROOT/usr/sbin/makedumpfile @@ -360,6 +363,7 @@ done %{_unitdir}/kdump.service %{_prefix}/lib/systemd/system-generators/kdump-dep-generator.sh %{_prefix}/lib/kernel/install.d/60-kdump.install +%{_prefix}/lib/kernel/install.d/92-crashkernel.install %doc News %license COPYING %doc TODO
Signed-off-by: Kairui Song kasong@redhat.com --- crashkernel-howto.conf | 204 +++++++++++++++++++++++++++++++++++++++++ kexec-tools.spec | 3 + 2 files changed, 207 insertions(+) create mode 100644 crashkernel-howto.conf
diff --git a/crashkernel-howto.conf b/crashkernel-howto.conf new file mode 100644 index 0000000..4abd090 --- /dev/null +++ b/crashkernel-howto.conf @@ -0,0 +1,204 @@ +Introduction +============ + +This document describes features the kexec-tools package provides for setting +and estimating the crashkernel value. + +Kdump lives in a pre-reserved chunk of memory, and the size of the reserved +memory is specified by the `crashkernel=` kernel parameter. It's hard to +estimate an accurate `crashkernel=` value, so it's always recommended to test +kdump after you updated the `crashkernel=` value or changed the dump target. + + +Default crashkernel value +========================= + +Latest kernel packages includes a `crashkernel.default` file installed in kernel +modules folder, available as: + + /usr/lib/modules/<kernel>/crashkernel.default + +The content of the file will be taken as the default value of 'crashkernel=', or +take this file as a reference for setting crashkernel value manually. + + +New installed system +==================== + +Anaconda is the OS installer which sets all the kernel boot cmdline on a new +installed system. If kdump is set enabled during Anaconda installation, Anaconda +will use the `crashkernel.default` file as the default `crashkernel=` value on +the new installed system. + +Users can also override the value during Anaconda installation manually. + + +Auto update of crashkernel boot parameter +========================================= + +Following context in this section assumes all kernel packages have a +`crashkernel.default` file bundled, which is true for the latest official kernel +packages. For kexec-tools behavior with a kernel that doesn't have a +`crashkernel.default` file, please refer to the “Custom Kernel” section of this +doc. + +When `crashkernel=` is using the default value, kexec-tools will need to update +the `crashkernel=` value of new installed kernels, since the default value may +change in new kernel packages. + +kexec-tools does so by adding a kernel installation hook, which gets triggered +every time a new kernel is installed, so kexec-tools can do necessary checks and +updates. + + +Supported Bootloaders +--------------------- + +This auto update only works with GRUB2 and ZIPL, as kexec-tools heavily depends +on `grubby`. If other boot loaders are used, the user will have to update the +`crashkernel=` value manually. + + +Updating kernel package +----------------------- + +When a new version of package kernel is released in the official repository, the +package will always come with a `crashkernel.default` file bundled. Kexec-tools +will act with following rules: + +If current boot kernel is using the default `crashkernel=` boot param value from +its `crashkernel.default` file, then kexec-tools will update new installed +kernel’s `crashkernel=` boot param using the value from the new installed +kernel’s `crashkernel.default` file. This ensures `crashkernel=` is always using +the latest default value. + +If current boot kernel's `crashkernel=` value is set to a non-default value, the +new installed kernel simply inherits this value. + +On systems using GRUB2 as the bootloader, each kernel has its own boot entry, +making it possible to set different `crashkernel=` boot param values for +different kernels. So kexec-tools won’t touch any already installed kernel's +boot param, only new installed kernel's `crashkernel=` boot param value will be +updated. + +But some utilities like `grub2-mkconfig` and `grubby` can override all boot +entry's boot params with the boot params value from the GRUB config file +`/etc/defaults/grub`, so kexec-tools will also update the GRUB config file in +case old `crashkernel=` value overrides new installed kernel’s boot param. + + +Downgrading kernel package +-------------------------- + +When upgrading a kernel package, kexec-tools may update the `crashkernel=` value +in GRUB2 config file to the new value. So when downgrading the kernel package, +kexec-tools will also try to revert that update by setting GRUB2 config file’s +`crashkernel=` value back to the default value in the older kernel package. This +will only occur when the GRUB2 config file is using the default `crashkernel=` +value. + + +Custom kernel +============= + +To make auto crashkernel update more robust, kexec-tools will try to keep +tracking the default 'crashkernel=` value with kernels that don’t have a +`crashkernel.default` file, such kernels are referred to as “custom kernel” in +this doc. This is only a best-effort support to make it easier debugging and +testing the system. + +When installing a custom kernel that doesn’t have a `crashkernel.default` file, +the `crashkernel=` value will be simply inherited from the current boot kernel. + +When installing a new official kernel package and current boot kernel is a +custom kernel, since the boot kernel doesn’t have a `crashkernel.default` file, +kexec-tools will iterate installed kernels and check if the boot kernel +inherited the default value from any other existing kernels’ +`crashkernel.default` file. If a matching `crashkernel.default` file is found, +kexec-tools will update the new installed kernel `crashkernel=` boot param using +the value from the new installed kernel’s `crashkernel.default` file, ensures +the auto crashkernel value update won’t break over one or two custom kernel +installations. + +It is possible that the auto crashkernel value update will fail when custom +kernels are used. One example is a custom kernel inheriting the default +`crashkernel=` value from an older official kernel package, but later that +kernel package is uninstalled. So when booted with the custom kernel, +kexec-tools can't determine if the boot kernel is inheriting a default +`crashkernel=` value from any official build. In such a case, please refer to +the "Reset crashkernel to default value" section of this doc. + + +Reset crashkernel to default value +================================== + +kexec-tools only perform the auto update of crashkernel value when it can +confirm the boot kernel's crashkernel value is using its corresponding default +value or inherited from any installed kernel. + +kexec-tools may fail to determine if the boot kernel is using default +crashkernel value in some use cases: +- kexec-tools package is absent during a kernel package upgrade, and the new + kernel package’s `crashkernel.default` value has changed. +- Custom kernel is used and the kernel it inherits `crashkernel=` value from is + uninstalled. + +So it's recommended to reset the crashkernel value if users have uninstalled +kexec-tools or using a custom kernel. + +Reset using kdumpctl +-------------------- + +To make it easier to reset the `crashkernel=` kernel cmdline to this default +value properly, `kdumpctl` also provides a sub-command: + + `kdumpctl reset-crashkernel [<kernel version>]` + +This command will read from the `crashkernel.default` file and reset +bootloader's kernel cmdline to the default value. It will also update bootloader +config if the bootloader has a standalone config file. User will have to reboot +the machine after this command to make it take effect. + +Reset manually +-------------- + +To reset the crashkernel value manually, it's recommended to use utils like +`grubby`. A one liner script for resetting `crashkernel=` value of all installed +kernels to current boot kernel's crashkernel.default` is: + + grubby --update-kernel ALL --args "$(cat /usr/lib/modules/$(uname -r)/crashkernel.default)" + +Estimate crashkernel +==================== + +The best way to estimate a usable crashkernel value is by testing kdump +manually. And you can set crashkernel to a large value, then adjust the +crashkernel value to an acceptable value gradually. + +`kdumpctl` also provides a sub-command for doing rough estimating without +triggering kdump: + + `kdumpctl estimate` + +The output will be like this: + +``` + Encrypted kdump target requires extra memory, assuming using the keyslot with minimun memory requirement + + Reserved crashkernel: 256M + Recommended crashkernel: 655M + + Kernel image size: 47M + Kernel modules size: 12M + Initramfs size: 19M + Runtime reservation: 64M + LUKS required size: 512M + Large modules: + xfs: 1892352 + nouveau: 2318336 + WARNING: Current crashkernel size is lower than recommended size 655M. +``` + +It will generate a summary report about the estimated memory consumption +of each component of kdump. The value may not be accurate enough, but +would be a good start for finding a suitable crashkernel value. diff --git a/kexec-tools.spec b/kexec-tools.spec index ec327da..deedcf1 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -41,6 +41,7 @@ Source30: 60-kdump.install Source31: kdump-logger.sh Source32: mkfadumprd Source33: 92-crashkernel.install +Source34: crashkernel-howto.txt
####################################### # These are sources for mkdumpramfs @@ -151,6 +152,7 @@ cp %{SOURCE11} . cp %{SOURCE21} . cp %{SOURCE26} . cp %{SOURCE27} . +cp %{SOURCE34} .
make %ifarch %{ix86} x86_64 ppc64 s390x ppc64le aarch64 @@ -372,6 +374,7 @@ done %doc fadump-howto.txt %doc kdump-in-cluster-environment.txt %doc live-image-kdump-howto.txt +%doc crashkernel-howto.txt %ifarch %{ix86} x86_64 ppc64 s390x ppc64le aarch64 %{_libdir}/eppic_makedumpfile.so /usr/share/makedumpfile/
On 07/07/21 at 02:20pm, Kairui Song wrote:
In RHEL, kernel supports a `crashkernel=auto` param and when it's used, kernel will use its built-in default value as the crashkernel value. But upstream doesn't like this idea. [1]
`crashkernel=auto` was introduced to provide user a default value, and make it possible to update the crashkernel value automatically as kernel package updates if user is using the default `auto` value.
After more discussion, `crashkernel=auto` will be dropped. To track the default crashkernel value, kernel package will include a crashkernel.default file with it[2], and kexec-tools can make use of that file, update kernel's boot cmdline accordingly, and help user to reset the crashkernel value to default value.
The patchset looks good to me, ack. Thanks for the effort.
Acked-by: Baoquan He bhe@redhat.com
Thanks Baoquan
Update from V2:
- Update the doc as suggested by Baoquan
- Fix some comment and code issues in 92-crashkernel.install
Update from V1:
- Add support for ostree
- Add a crashkernel-howto.txt
- Change crashkernel.conf to crashkernel.default
Kairui Song (4): Revert "kdump-lib.sh: Remove is_atomic" kdumpctl: Add kdumpctl reset-crashkernel Add a new hook: 92-crashkernel.install Add a crashkernel-howto.conf doc
92-crashkernel.install | 143 +++++++++++++++++++++++++++++ crashkernel-howto.conf | 204 +++++++++++++++++++++++++++++++++++++++++ kdump-lib.sh | 5 + kdumpctl | 41 ++++++++- kdumpctl.8 | 9 ++ kexec-tools.spec | 7 ++ 6 files changed, 407 insertions(+), 2 deletions(-) create mode 100755 92-crashkernel.install create mode 100644 crashkernel-howto.conf
-- 2.31.1
-
Baoquan He -
Kairui Song