For some systems with a large number of CPUs and peripherals the default
"nr_cpus{maxcpus}=1" hard-coded in "/etc/sysconfig/kdump" won't
work.
We need a algorithm/formula to choose the proper value in these cases.
Known affected factors:
*)reserved crash memory size for kdump kernel.
*)the number of cpus in first kernel.
*)architecture
-different memory overhead adding one extra cpu.
-different page size.
-different reserved crash memory policy.
*)large number of io devices
-probably requires large crash memory, so firstly it has
something to do with the reserved crash memory size.
-then we met an issue for x86_64: kdump runs out of vectors
with few cpus boot up, when requesting tons of irqs.
e.g. some x86_64 large machine : 24TB RAM, nearly 1024 cpus, with the default
kdump configuration, kdump kernel hung at boot up stage. Manually changed to
"crashkernel=512M,high crashkernel=256M,low nr_cpus=4", then it works properly.
Some investigation and tests has been done as follows.
"extra memory consumption for one extra cpu"
a) x86_64
tested on
amd, 2-smt 12-cpu, memory: 10GB, crashkernel=161MB, f25
and 4-core kvm.fedora, f24
result: less than 2MB with one extra cpu.
b) s390x
tested on
ibm-z10-47.rhts.eng.bos.redhat.com 1994MB crashkernel=161MB
result: less than 2MB with one extra logical cpu.
c) ppc64("maxcpus=X")
tested on a 4-smt 7-core 28-cpu machine.
result: around 5MB with one extra logical cpu.
From the data described above, we can see clearly that more cpus means
more memory consumption, e.g. 1024-cpu system will consume around 2GB
more memory, even more for ppc64 systems, this is obviously unacceptable
for kdump, we should have an upper bound for the number of kdump cpus.
We assume that an extra cpu every 256MB more memory is safe to be with
the flexible of 2MB variant, i.e. we allow one extra cpu for x86/s390
(not test arm yet) every 256MB crash memory size(we call it "stepsize")
in case there is insufficient memory, and allow one extra cpu for ppc64
every 512MB considering its more memory consumption with more cpu added.
We also limit the max cpu for kdump to 16, as the upper bound.
Formula:
An extra cpu every stepsize, round down to power of two. Specifically,
we will get the following results(kind of conservative in order to
achieve good reliability):
For x86_64/s390x/arm64:
"crash memory" "nr_cpus/maxcpus"
(0M, 256M]: 1
(256M, 1G]: 2
(1G, 2G]: 4
(2G, 4G]: 8
(4G, UMAX]: 16
For ppc64:
"crash memory" "nr_cpus/maxcpus"
(0M, 512M]: 1
(512M, 2G]: 2
(2G, 4G]: 4
(4G, 8G]: 8
(8G, UMAX]: 16
After this, we make a special treatment for the x86_64 vector issue.
Finally, ensure it doesn't exceed the number of online cpus on 1st kernel.
Xunlei Pang (4):
kdumpctl: Protect "/etc/sysconfig/kdump" operations with lock
kdump-lib: Introduce get_reserved_memory_mb()
kdump.conf: Introduce "calc_kdump_cpus=<0 | 1>" option
kdumpctl: Implement calculate_kdump_cpus()
kdump-lib.sh | 12 ++++++
kdump.conf | 7 +++
kdumpctl | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
3 files changed, 150 insertions(+), 7 deletions(-)
--
1.8.3.1