By default kernel have vm.zone_reclaim_mode = 0 and large page allocation might fail as kernel is very conservative on memory reclaiming. If the page allocation failure is not handled carefully it could lead to more serious problems.
This issue can be reproduced by change with following steps:
- Fill up page cache use: # dd if=/dev/urandom of=/test bs=1M count=1300
- Now the memory is filled with write cache: # free -m total used free shared buff/cache available Mem: 1790 184 132 2 1473 1348 Swap: 2119 7 2112
- Insert a module which simply calls "kmalloc(SZ_1M, GFP_KERNEL)" for 512 times: (Notice: vmalloc don't have such problem) # insmod debug_module.ko
- Got following allocation failure: insmod: page allocation failure: order:8, mode:0x40cc0(GFP_KERNEL|__GFP_COMP), nodemask=(null),cpuset=/,mems_allowed=0
- Clean up and repeat again with vm.zone_reclaim_mode = 3, OOM is not observed.
In kdump kernel there is usually only one online CPU and limited memory, so we set vm.zone_reclaim_mode = 3 to let kernel reclaim memory more aggresively to avoid such issue.
Signed-off-by: Kairui Song kasong@redhat.com --- dracut-module-setup.sh | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 37aaa80..e6358a8 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -537,16 +537,18 @@ kdump_install_conf() { rm -f ${initdir}/tmp/$$-kdump.conf }
-# Default sysctl parameters should suffice for kdump kernel. -# Remove custom configurations sysctl.conf & sysctl.d/* -remove_sysctl_conf() { - +# Remove user custom configurations sysctl.conf & sysctl.d/* +# and apply some optimization for kdump +overwrite_sysctl_conf() { # As custom configurations like vm.min_free_kbytes can lead # to OOM issues in kdump kernel, avoid them rm -f "${initdir}/etc/sysctl.conf" rm -rf "${initdir}/etc/sysctl.d" rm -rf "${initdir}/run/sysctl.d" rm -rf "${initdir}/usr/lib/sysctl.d" + + mkdir -p "${initdir}/etc/sysctl.d" + echo "vm.zone_reclaim_mode = 3" > "${initdir}/etc/sysctl.d/99-zone-reclaim.conf" }
kdump_iscsi_get_rec_val() { @@ -812,7 +814,7 @@ kdump_install_random_seed() {
install() { kdump_install_conf - remove_sysctl_conf + overwrite_sysctl_conf
if is_ssh_dump_target; then kdump_install_random_seed
On 11/12/2019 01:35 PM, Kairui Song wrote:
By default kernel have vm.zone_reclaim_mode = 0 and large page allocation might fail as kernel is very conservative on memory reclaiming. If the page allocation failure is not handled carefully it could lead to more serious problems.
This issue can be reproduced by change with following steps:
Fill up page cache use: # dd if=/dev/urandom of=/test bs=1M count=1300
Now the memory is filled with write cache: # free -m total used free shared buff/cache available Mem: 1790 184 132 2 1473 1348 Swap: 2119 7 2112
Insert a module which simply calls "kmalloc(SZ_1M, GFP_KERNEL)" for 512 times: (Notice: vmalloc don't have such problem) # insmod debug_module.ko
Got following allocation failure: insmod: page allocation failure: order:8, mode:0x40cc0(GFP_KERNEL|__GFP_COMP), nodemask=(null),cpuset=/,mems_allowed=0
Clean up and repeat again with vm.zone_reclaim_mode = 3, OOM is not observed.
In kdump kernel there is usually only one online CPU and limited memory, so we set vm.zone_reclaim_mode = 3 to let kernel reclaim memory more aggresively to avoid such issue.
Signed-off-by: Kairui Song kasong@redhat.com
dracut-module-setup.sh | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 37aaa80..e6358a8 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -537,16 +537,18 @@ kdump_install_conf() { rm -f ${initdir}/tmp/$$-kdump.conf }
-# Default sysctl parameters should suffice for kdump kernel. -# Remove custom configurations sysctl.conf & sysctl.d/* -remove_sysctl_conf() {
+# Remove user custom configurations sysctl.conf & sysctl.d/* +# and apply some optimization for kdump +overwrite_sysctl_conf() { # As custom configurations like vm.min_free_kbytes can lead # to OOM issues in kdump kernel, avoid them rm -f "${initdir}/etc/sysctl.conf" rm -rf "${initdir}/etc/sysctl.d" rm -rf "${initdir}/run/sysctl.d" rm -rf "${initdir}/usr/lib/sysctl.d"
- mkdir -p "${initdir}/etc/sysctl.d"
- echo "vm.zone_reclaim_mode = 3" > "${initdir}/etc/sysctl.d/99-zone-reclaim.conf"
}
kdump_iscsi_get_rec_val() { @@ -812,7 +814,7 @@ kdump_install_random_seed() {
install() { kdump_install_conf
- remove_sysctl_conf
overwrite_sysctl_conf
if is_ssh_dump_target; then kdump_install_random_seed
Acked-by: Pingfan Liu piliu@redhat.com