6k aarch64 has significant memory consumption more than 4k variant, and requires an separated crashkernel formula.
This series generates the formula dynamiclly based on the system environment.
v4 -> v5: fix complain by shellcheck on [2/4] rebase to latest rawhide (commit 29fe563 kdump.conf: redirect unknown architecture warning to stderr)
Pingfan Liu (4): kdump-lib: Introduce a help function _crashkernel_add() kdump-lib: Introduce parse_kver_from_path() to get kernel version from its path name kdump-lib: add support for 64K aarch64 kdump-lib: Evaluate the memory consumption by smmu and mlx5 separately
kdump-lib.sh | 138 ++++++++++++++++++++++++++++++++++++++++- kdumpctl | 6 +- spec/kdump-lib_spec.sh | 20 ++++++ 3 files changed, 161 insertions(+), 3 deletions(-)
This help function can manipulate the crashkernel cmdline by adding an number for each item. Also a basic test case for _crashkernel_add() is provided in this patch.
Credit to Philipp, who contributes the original code.
Signed-off-by: Pingfan Liu piliu@redhat.com --- kdump-lib.sh | 58 ++++++++++++++++++++++++++++++++++++++++++ spec/kdump-lib_spec.sh | 20 +++++++++++++++ 2 files changed, 78 insertions(+)
diff --git a/kdump-lib.sh b/kdump-lib.sh index c6bf378..992e8d9 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -786,6 +786,64 @@ get_recommend_size() echo "0M" }
+# $1 crashkernel="" +# $2 delta in unit of MB +_crashkernel_add() +{ + local _ck _add _entry _ret + local _range _size _offset + + _ck="$1" + _add="$2" + _ret="" + + if [[ "$_ck" == *@* ]]; then + _offset="@${_ck##*@}" + _ck=${_ck%@*} + elif [[ "$_ck" == *,high ]] || [[ "$_ck" == *,low ]]; then + _offset=",${_ck##*,}" + _ck=${_ck%,*} + else + _offset='' + fi + + while read -d , -r _entry; do + [[ -n "$_entry" ]] || continue + if [[ "$_entry" == *:* ]]; then + _range=${_entry%:*} + _size=${_entry#*:} + else + _range="" + _size=${_entry} + fi + + case "${_size: -1}" in + K) + _size=${_size::-1} + _size="$((_size + (_add * 1024)))K" + ;; + M) + _size=${_size::-1} + _size="$((_size + _add))M" + ;; + G) + _size=${_size::-1} + _size="$((_size * 1024 + _add))M" + ;; + *) + _size="$((_size + (_add * 1024 * 1024)))" + ;; + esac + + [[ -n "$_range" ]] && _ret+="$_range:" + _ret+="$_size," + done <<< "$_ck," + + _ret=${_ret%,} + [[ -n "$_offset" ]] && _ret+=$_offset + echo "$_ret" +} + # get default crashkernel # $1 dump mode, if not specified, dump_mode will be judged by is_fadump_capable kdump_get_arch_recommend_crashkernel() diff --git a/spec/kdump-lib_spec.sh b/spec/kdump-lib_spec.sh index 3d10006..81f0f86 100644 --- a/spec/kdump-lib_spec.sh +++ b/spec/kdump-lib_spec.sh @@ -48,6 +48,26 @@ Describe 'kdump-lib' End End
+ Describe "_crashkernel_add()" + Context "when the input parameter is '1G-4G:256M,4G-64G:320M,64G-:576M'" + delta=100 + Parameters + "1G-4G:256M,4G-64G:320M,64G-:576M" "1G-4G:356M,4G-64G:420M,64G-:676M" + "1G-4G:256M,4G-64G:320M,64G-:576M@4G" "1G-4G:356M,4G-64G:420M,64G-:676M@4G" + "1G-4G:1G,4G-64G:2G,64G-:3G@4G" "1G-4G:1124M,4G-64G:2148M,64G-:3172M@4G" + "1G-4G:10000K,4G-64G:20000K,64G-:40000K@4G" "1G-4G:112400K,4G-64G:122400K,64G-:142400K@4G" + "300M,high" "400M,high" + "300M,low" "400M,low" + "500M@1G" "600M@1G" + End + It "should add delta to the values after ':'" + + When call _crashkernel_add "$1" "$delta" + The output should equal "$2" + End + End + End + Describe 'prepare_cmdline()' get_bootcpu_apicid() { echo 1
kdump_get_arch_recommend_crashkernel() expects the kernel version info, while _update_kernel() provides the absolute path, which contains the kernel version info.
This patch introduce a dedicated function parse_kver_from_path() to extract the kernel info from the path
Credit to Philipp, who contributes the original code.
Signed-off-by: Pingfan Liu piliu@redhat.com --- kdump-lib.sh | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
diff --git a/kdump-lib.sh b/kdump-lib.sh index 992e8d9..4f31b74 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -526,6 +526,49 @@ prepare_kdump_kernel() echo "$kdump_kernel" }
+_is_valid_kver() +{ + [[ -f /usr/lib/modules/$1/modules.dep ]] +} + +# This function is introduced since 64k variant may be installed on 4k or vice versa +# $1 the kernel path name. +parse_kver_from_path() +{ + local _img _kver + + [[ -z "$1" ]] && return + + _img=$1 + BLS_ENTRY_TOKEN=$(</etc/machine-id) + + # Fedora standard installation, i.e. $BOOT/vmlinuz-<version> + _kver=${_img##*/vmlinuz-} + _kver=${_kver%"$KDUMP_IMG_EXT"} + if _is_valid_kver "$_kver"; then + echo "$_kver" + return + fi + + # BLS recommended image names, i.e. $BOOT/<token>/<version>/linux + _kver=${_img##*/"$BLS_ENTRY_TOKEN"/} + _kver=${_kver%%/*} + if _is_valid_kver "$_kver"; then + echo "$_kver" + return + fi + + # Fedora UKI installation, i.e. $BOOT/efi/EFI/Linux/<token>-<version>.efi + _kver=${_img##*/"$BLS_ENTRY_TOKEN"-} + _kver=${_kver%.efi} + if _is_valid_kver "$_kver"; then + echo "$_kver" + return + fi + + ddebug "Could not parse version from $_img" +} + _get_kdump_kernel_version() { local _version _version_nondebug
From: Pingfan Liu <piliu at redhat.com>
On aarch64, both 4K and 64K kernel can be installed, while they demand different size reserved memory for kdump kernel.
'get_conf PAGE_SIZE' can not work if installing a 64K kernel when running a 4K kernel. Hence resorting to the kernel release naming rules. At present, the 64K kernel has the keyword '64k' in its suffix.
The base line for 64K is decided based on 4K. The diff 100M is picked up since on a high end machine without smmu enabled, the diff of MemFree is 82M.
As for the smmu case, a huge difference in the memory consumption lies between 64k and 4k driver. And it should be calculated separatedly.
Signed-off-by: Pingfan Liu piliu@redhat.com --- kdump-lib.sh | 21 ++++++++++++++++++++- kdumpctl | 6 ++++-- 2 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/kdump-lib.sh b/kdump-lib.sh index 4f31b74..68431a0 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -889,6 +889,7 @@ _crashkernel_add()
# get default crashkernel # $1 dump mode, if not specified, dump_mode will be judged by is_fadump_capable +# $2 kernel-release, if not specified, got by _get_kdump_kernel_version kdump_get_arch_recommend_crashkernel() { local _arch _ck_cmdline _dump_mode @@ -908,8 +909,26 @@ kdump_get_arch_recommend_crashkernel() if [[ $_arch == "x86_64" ]] || [[ $_arch == "s390x" ]]; then _ck_cmdline="1G-4G:192M,4G-64G:256M,64G-:512M" elif [[ $_arch == "aarch64" ]]; then - # For 4KB page size, the formula is based on x86 plus extra = 64M + local _running_kernel + local _delta=0 + + # Base line for 4K variant kernel. The formula is based on x86 plus extra = 64M _ck_cmdline="1G-4G:256M,4G-64G:320M,64G-:576M" + if [[ -z "$2" ]]; then + _running_kernel=$(_get_kdump_kernel_version) + else + _running_kernel=$2 + fi + + # the naming convention of 64k variant suffixes with +64k, e.g. "vmlinuz-5.14.0-312.el9.aarch64+64k" + if echo "$_running_kernel" | grep -q 64k; then + # Without smmu, the diff of MemFree between 4K and 64K measured on a high end aarch64 machine is 82M. + # Picking up 100M to cover this diff. And finally, we have "1G-4G:356M;4G-64G:420M;64G-:676M" + ((_delta += 100)) + else + ((_delta += 0)) + fi + _ck_cmdline=$(_crashkernel_add "$_ck_cmdline" "$_delta") elif [[ $_arch == "ppc64le" ]]; then if [[ $_dump_mode == "fadump" ]]; then _ck_cmdline="4G-16G:768M,16G-64G:1G,64G-128G:2G,128G-1T:4G,1T-2T:6G,2T-4T:12G,4T-8T:20G,8T-16T:36G,16T-32T:64G,32T-64T:128G,64T-:180G" diff --git a/kdumpctl b/kdumpctl index 5043a7b..827b595 100755 --- a/kdumpctl +++ b/kdumpctl @@ -1678,12 +1678,14 @@ _is_bootloader_installed()
_update_crashkernel() { - local _kernel _dump_mode _old_default_crashkernel _new_default_crashkernel _fadump_val _msg + local _kernel _kver _dump_mode _old_default_crashkernel _new_default_crashkernel _fadump_val _msg
_kernel=$1 _dump_mode=$(get_dump_mode_by_kernel "$_kernel") _old_default_crashkernel=$(get_grub_kernel_boot_parameter "$_kernel" crashkernel) - _new_default_crashkernel=$(kdump_get_arch_recommend_crashkernel "$_dump_mode") + _kver=$(parse_kver_from_path "$_kernel") + # The second argument is for the case of aarch64, where installing a 64k variant on a 4k kernel, or vice versa + _new_default_crashkernel=$(kdump_get_arch_recommend_crashkernel "$_dump_mode" "$_kver") if [[ $_old_default_crashkernel != "$_new_default_crashkernel" ]]; then _fadump_val=$(get_grub_kernel_boot_parameter "$_kernel" fadump) if _update_kernel_cmdline "$_kernel" "$_new_default_crashkernel" "$_dump_mode" "$_fadump_val"; then
On 4k and 64k kernels, the typical consumption values for SMMU are 36MB and 384MB, respectively. Hence for 64k kernel, the consumption by smmu should be taken into account carefully.
To do it by adding the extra 384MB value if installing a 64k kernel. The upper limit value 384MB is calculated according to the formula in the kernel smmu driver.
As for mlx5 network cards, it is measured by a pratical test, 200M for 64k variant, 150M for 4k variant
Signed-off-by: Pingfan Liu piliu@redhat.com --- kdump-lib.sh | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/kdump-lib.sh b/kdump-lib.sh index 68431a0..9e818d3 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -829,6 +829,16 @@ get_recommend_size() echo "0M" }
+has_mlx5() +{ + [[ -d /sys/bus/pci/drivers/mlx5_core ]] +} + +has_aarch64_smmu() +{ + ls /sys/devices/platform/arm-smmu-* 1> /dev/null 2>&1 +} + # $1 crashkernel="" # $2 delta in unit of MB _crashkernel_add() @@ -925,8 +935,14 @@ kdump_get_arch_recommend_crashkernel() # Without smmu, the diff of MemFree between 4K and 64K measured on a high end aarch64 machine is 82M. # Picking up 100M to cover this diff. And finally, we have "1G-4G:356M;4G-64G:420M;64G-:676M" ((_delta += 100)) + # On a 64K system, the extra 384MB is calculated by: cmdq_num * 16 bytes + evtq_num * 32B + priq_num * 16B + # While on a 4K system, it is negligible + has_aarch64_smmu && ((_delta += 384)) + #64k kernel, mlx5 consumes extra 188M memory, and choose 200M + has_mlx5 && ((_delta += 200)) else - ((_delta += 0)) + #4k kernel, mlx5 consumes extra 124M memory, and choose 150M + has_mlx5 && ((_delta += 150)) fi _ck_cmdline=$(_crashkernel_add "$_ck_cmdline" "$_delta") elif [[ $_arch == "ppc64le" ]]; then
On Tue, Jun 13, 2023 at 05:43:19PM +0800, Pingfan Liu wrote:
6k aarch64 has significant memory consumption more than 4k variant, and requires an separated crashkernel formula.
This series generates the formula dynamiclly based on the system environment.
v4 -> v5: fix complain by shellcheck on [2/4] rebase to latest rawhide (commit 29fe563 kdump.conf: redirect unknown architecture warning to stderr)
Pingfan Liu (4): kdump-lib: Introduce a help function _crashkernel_add() kdump-lib: Introduce parse_kver_from_path() to get kernel version from its path name kdump-lib: add support for 64K aarch64 kdump-lib: Evaluate the memory consumption by smmu and mlx5 separately
kdump-lib.sh | 138 ++++++++++++++++++++++++++++++++++++++++- kdumpctl | 6 +- spec/kdump-lib_spec.sh | 20 ++++++ 3 files changed, 161 insertions(+), 3 deletions(-)
This patch set has been merged, thanks to Pingfan and Philipp!
-- 2.31.1
-
Coiby Xu -
Pingfan Liu