Hi Xunlei:
On 12/20/16 at 12:21pm, Xunlei Pang wrote:
> On 12/20/2016 at 09:37 AM, Dave Young wrote:
>> Hi, Xunlei
>> On 12/19/16 at 11:24pm, Xunlei Pang wrote:
>>> For some systems with a large number of CPUs and peripherals the default
>>> "nr_cpus{maxcpus}=1" hard-coded in "/etc/sysconfig/kdump"
won't work.
>>> We need a algorithm/formula to choose the proper value in these cases.
>>>
>>> Known affected factors:
>>> *)reserved crash memory size for kdump kernel.
>>> *)the number of cpus in first kernel.
>>> *)architecture
>>> -different memory overhead adding one extra cpu.
>>> -different page size.
>>> -different reserved crash memory policy.
>>> *)large number of io devices
>>> -probably requires large crash memory, so firstly it has
>>> something to do with the reserved crash memory size.
>>> -then we met an issue for x86_64: kdump runs out of vectors
>>> with few cpus boot up, when requesting tons of irqs.
>>>
>>> e.g. some x86_64 large machine : 24TB RAM, nearly 1024 cpus, with the
default
>>> kdump configuration, kdump kernel hung at boot up stage. Manually changed to
>>> "crashkernel=512M,high crashkernel=256M,low nr_cpus=4", then it
works properly.
>>>
>>> Some investigation and tests has been done as follows.
>>> "extra memory consumption for one extra cpu"
>>> a) x86_64
>>> tested on
>>> amd, 2-smt 12-cpu, memory: 10GB, crashkernel=161MB, f25
>>> and 4-core kvm.fedora, f24
>>>
>>> result: less than 2MB with one extra cpu.
>>>
>>> b) s390x
>>> tested on
ibm-z10-47.rhts.eng.bos.redhat.com 1994MB crashkernel=161MB
>>>
>>> result: less than 2MB with one extra logical cpu.
>>>
>>> c) ppc64("maxcpus=X")
>>> tested on a 4-smt 7-core 28-cpu machine.
>>>
>>> result: around 5MB with one extra logical cpu.
>>>
>>> From the data described above, we can see clearly that more cpus means
>>> more memory consumption, e.g. 1024-cpu system will consume around 2GB
>>> more memory, even more for ppc64 systems, this is obviously unacceptable
>>> for kdump, we should have an upper bound for the number of kdump cpus.
>>>
>>> We assume that an extra cpu every 256MB more memory is safe to be with
>>> the flexible of 2MB variant, i.e. we allow one extra cpu for x86/s390
>>> (not test arm yet) every 256MB crash memory size(we call it
"stepsize")
>>> in case there is insufficient memory, and allow one extra cpu for ppc64
>>> every 512MB considering its more memory consumption with more cpu added.
>>>
>>> We also limit the max cpu for kdump to 16, as the upper bound.
>>>
>>> Formula:
>>> An extra cpu every stepsize, round down to power of two. Specifically,
>>> we will get the following results(kind of conservative in order to
>>> achieve good reliability):
>>> For x86_64/s390x/arm64:
>>> "crash memory" "nr_cpus/maxcpus"
>>> (0M, 256M]: 1
>>> (256M, 1G]: 2
>>> (1G, 2G]: 4
>>> (2G, 4G]: 8
>>> (4G, UMAX]: 16
>>>
>>> For ppc64:
>>> "crash memory" "nr_cpus/maxcpus"
>>> (0M, 512M]: 1
>>> (512M, 2G]: 2
>>> (2G, 4G]: 4
>>> (4G, 8G]: 8
>>> (8G, UMAX]: 16
>>>
>>> After this, we make a special treatment for the x86_64 vector issue.
>>>
>>> Finally, ensure it doesn't exceed the number of online cpus on 1st
kernel.
>> Sorry that I did not find time to review it in bugzilla. It seems be far
>> more than to cover the irq vector issues only.
>>
>> There are a few worries and comments below in my mind, we can discuss it
>> in kdump meeting:
>>
>> * We should leave user to specify the cpu numbers instead of we do that
>> in our code unless we have to.
>> The x86 irq vector thing is a have-to because kdump kernel can not
>> boot with nr_cpus=1 in case large number of io devices. In this case
>> we can set nr_cpus as the minimum cpu number which can serve the large
>> io devices irq vectors.
> Yes, the first step is to make sure our default configuration works well,
> then to increase the performance as possible as we can.
>
>> * Dynamiclly modify sysconfig file sounds odd, it is just like what we
>> previously did in RHEL kernel to change the cmdline buffer. It should be
>> not touched. If we have to change a value, we can consider to drop
>> "nr_cpus=1" in x86 sysconfig file and handle the default in kdumpctl
>> script. If user specify nr_cpus in sysconfig then in kdumpctl we can
>> just use the user provided value.
> There is a new kdump.conf option introduced in PATCH3 to avoid conflicts with
user's manual intervention.
We should be careful to add new kdump.conf option also, once we add it
it is hard to remove it because of compatibility issue..
>> * Since we just auto add minimum bootable cpu numbers, we can ignore
>> the extra memory requirement in crashkernel=auto, from the HP setup
>> nr_cpus=4 is enough, the extra memory is neglective.
> My original purpose of making these patches is not only to add minimum bootable cpus,
but also want to
> add more to increase performance under the condition that it can work reliably for
all scenarios.
Understood, do the first step first should be better.
There are many thing need to check for adding automatic handling, it is
hard to estimate risks to all kinds of arches and different setups..
There could be kinds of corner cases, for the time being leaving it to
user to specify correct numbers for performance is better.