Hi, Xunlei
On 12/19/16 at 11:24pm, Xunlei Pang wrote:
For some systems with a large number of CPUs and peripherals the
default
"nr_cpus{maxcpus}=1" hard-coded in "/etc/sysconfig/kdump" won't
work.
We need a algorithm/formula to choose the proper value in these cases.
Known affected factors:
*)reserved crash memory size for kdump kernel.
*)the number of cpus in first kernel.
*)architecture
-different memory overhead adding one extra cpu.
-different page size.
-different reserved crash memory policy.
*)large number of io devices
-probably requires large crash memory, so firstly it has
something to do with the reserved crash memory size.
-then we met an issue for x86_64: kdump runs out of vectors
with few cpus boot up, when requesting tons of irqs.
e.g. some x86_64 large machine : 24TB RAM, nearly 1024 cpus, with the default
kdump configuration, kdump kernel hung at boot up stage. Manually changed to
"crashkernel=512M,high crashkernel=256M,low nr_cpus=4", then it works
properly.
Some investigation and tests has been done as follows.
"extra memory consumption for one extra cpu"
a) x86_64
tested on
amd, 2-smt 12-cpu, memory: 10GB, crashkernel=161MB, f25
and 4-core kvm.fedora, f24
result: less than 2MB with one extra cpu.
b) s390x
tested on
ibm-z10-47.rhts.eng.bos.redhat.com 1994MB crashkernel=161MB
result: less than 2MB with one extra logical cpu.
c) ppc64("maxcpus=X")
tested on a 4-smt 7-core 28-cpu machine.
result: around 5MB with one extra logical cpu.
From the data described above, we can see clearly that more cpus means
more memory consumption, e.g. 1024-cpu system will consume around 2GB
more memory, even more for ppc64 systems, this is obviously unacceptable
for kdump, we should have an upper bound for the number of kdump cpus.
We assume that an extra cpu every 256MB more memory is safe to be with
the flexible of 2MB variant, i.e. we allow one extra cpu for x86/s390
(not test arm yet) every 256MB crash memory size(we call it "stepsize")
in case there is insufficient memory, and allow one extra cpu for ppc64
every 512MB considering its more memory consumption with more cpu added.
We also limit the max cpu for kdump to 16, as the upper bound.
Formula:
An extra cpu every stepsize, round down to power of two. Specifically,
we will get the following results(kind of conservative in order to
achieve good reliability):
For x86_64/s390x/arm64:
"crash memory" "nr_cpus/maxcpus"
(0M, 256M]: 1
(256M, 1G]: 2
(1G, 2G]: 4
(2G, 4G]: 8
(4G, UMAX]: 16
For ppc64:
"crash memory" "nr_cpus/maxcpus"
(0M, 512M]: 1
(512M, 2G]: 2
(2G, 4G]: 4
(4G, 8G]: 8
(8G, UMAX]: 16
After this, we make a special treatment for the x86_64 vector issue.
Finally, ensure it doesn't exceed the number of online cpus on 1st kernel.
Sorry that I did not find time to review it in bugzilla. It seems be far
more than to cover the irq vector issues only.
There are a few worries and comments below in my mind, we can discuss it
in kdump meeting:
* We should leave user to specify the cpu numbers instead of we do that
in our code unless we have to.
The x86 irq vector thing is a have-to because kdump kernel can not
boot with nr_cpus=1 in case large number of io devices. In this case
we can set nr_cpus as the minimum cpu number which can serve the large
io devices irq vectors.
* Dynamiclly modify sysconfig file sounds odd, it is just like what we
previously did in RHEL kernel to change the cmdline buffer. It should be
not touched. If we have to change a value, we can consider to drop
"nr_cpus=1" in x86 sysconfig file and handle the default in kdumpctl
script. If user specify nr_cpus in sysconfig then in kdumpctl we can
just use the user provided value.
* Since we just auto add minimum bootable cpu numbers, we can ignore
the extra memory requirement in crashkernel=auto, from the HP setup
nr_cpus=4 is enough, the extra memory is neglective.
Xunlei Pang (4):
kdumpctl: Protect "/etc/sysconfig/kdump" operations with lock
kdump-lib: Introduce get_reserved_memory_mb()
kdump.conf: Introduce "calc_kdump_cpus=<0 | 1>" option
kdumpctl: Implement calculate_kdump_cpus()
kdump-lib.sh | 12 ++++++
kdump.conf | 7 +++
kdumpctl | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
3 files changed, 150 insertions(+), 7 deletions(-)
--
1.8.3.1
Thanks
Dave