Currently, kdump may experience failure on some aws aarch64 platform. The final scenario is:
[ 79.145089] printk: console [ttyS0] disabled Then the system has no response any more. And after reboot, there is no vmcore generated under /var/crash/. More detail [1].
In a short word, it is caused by the irqpoll policy and some unknown acpi issue. The serial device is hot-removed as a pci device.
More detailed, the irqpoll policy demands to iterate over all interrupt handler, if the interrupt line is shared, then the handler is dispatched. And acpi handler acpi_irq() is on a shared interrupt line, so it is called. But for some unknown reason, the acpi hardware regs hold wrong state, and the acpi driver decides that a hot-removed event happens on a pci slot, which finally removes the pci serial device.
To tackle this issue by removing the irqpoll parameter on aws aarch64 platform, until the real root cause in acpi is found and resolved.
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2080468#c0
Signed-off-by: Pingfan Liu piliu@redhat.com --- kdumpctl | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
diff --git a/kdumpctl b/kdumpctl index 6188d47..3eb57e3 100755 --- a/kdumpctl +++ b/kdumpctl @@ -641,6 +641,18 @@ function remove_kdump_kernel_key() keyctl unlink "$KDUMP_KEY_ID" %:.ima }
+function is_aws_aarch64() +{ + local _bios_model; + + _bios_model=$(lscpu | grep "BIOS Model name") + if [[ "${_bios_model}" =~ "AWS Graviton" ]]; then + return 0 + fi + + return 1 +} + # Load the kdump kernel specified in /etc/sysconfig/kdump # If none is specified, try to load a kdump kernel with the same version # as the currently running kernel. @@ -650,6 +662,9 @@ load_kdump()
KEXEC_ARGS=$(prepare_kexec_args "${KEXEC_ARGS}") KDUMP_COMMANDLINE=$(prepare_cmdline "${KDUMP_COMMANDLINE}" "${KDUMP_COMMANDLINE_REMOVE}" "${KDUMP_COMMANDLINE_APPEND}") + if is_aws_aarch64; then + KDUMP_COMMANDLINE=$(echo ${KDUMP_COMMANDLINE} | sed 's/irqpoll//' ) + fi
# For secureboot enabled machines, use new kexec file based syscall. # Old syscall will always fail as it does not have capability to
On 6/30/22 00:04, Pingfan Liu wrote:
Currently, kdump may experience failure on some aws aarch64 platform. The final scenario is:
[ 79.145089] printk: console [ttyS0] disabledThen the system has no response any more. And after reboot, there is no vmcore generated under /var/crash/. More detail [1].
In a short word, it is caused by the irqpoll policy and some unknown acpi issue. The serial device is hot-removed as a pci device.
More detailed, the irqpoll policy demands to iterate over all interrupt handler, if the interrupt line is shared, then the handler is dispatched. And acpi handler acpi_irq() is on a shared interrupt line, so it is called. But for some unknown reason, the acpi hardware regs hold wrong state, and the acpi driver decides that a hot-removed event happens on a pci slot, which finally removes the pci serial device.
To tackle this issue by removing the irqpoll parameter on aws aarch64 platform, until the real root cause in acpi is found and resolved.
Signed-off-by: Pingfan Liu piliu@redhat.com
kdumpctl | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
diff --git a/kdumpctl b/kdumpctl index 6188d47..3eb57e3 100755 --- a/kdumpctl +++ b/kdumpctl @@ -641,6 +641,18 @@ function remove_kdump_kernel_key() keyctl unlink "$KDUMP_KEY_ID" %:.ima }
+function is_aws_aarch64() +{
- local _bios_model;
- _bios_model=$(lscpu | grep "BIOS Model name")
I'm not sure lscpu is reliable here. I just checked two instance types (two I had running were a1.metal and c6g.xlarge) and this didn't work.
- if [[ "${_bios_model}" =~ "AWS Graviton" ]]; then
return 0- fi
- return 1
+}
# Load the kdump kernel specified in /etc/sysconfig/kdump # If none is specified, try to load a kdump kernel with the same version # as the currently running kernel. @@ -650,6 +662,9 @@ load_kdump()
KEXEC_ARGS=$(prepare_kexec_args "${KEXEC_ARGS}") KDUMP_COMMANDLINE=$(prepare_cmdline "${KDUMP_COMMANDLINE}" "${KDUMP_COMMANDLINE_REMOVE}" "${KDUMP_COMMANDLINE_APPEND}")
if is_aws_aarch64; then
KDUMP_COMMANDLINE=$(echo ${KDUMP_COMMANDLINE} | sed 's/irqpoll//' )fi
# For secureboot enabled machines, use new kexec file based syscall. # Old syscall will always fail as it does not have capability to
On 7/8/22 12:54, Dusty Mabe wrote:
On 6/30/22 00:04, Pingfan Liu wrote:
Currently, kdump may experience failure on some aws aarch64 platform. The final scenario is:
[ 79.145089] printk: console [ttyS0] disabledThen the system has no response any more. And after reboot, there is no vmcore generated under /var/crash/. More detail [1].
In a short word, it is caused by the irqpoll policy and some unknown acpi issue. The serial device is hot-removed as a pci device.
More detailed, the irqpoll policy demands to iterate over all interrupt handler, if the interrupt line is shared, then the handler is dispatched. And acpi handler acpi_irq() is on a shared interrupt line, so it is called. But for some unknown reason, the acpi hardware regs hold wrong state, and the acpi driver decides that a hot-removed event happens on a pci slot, which finally removes the pci serial device.
To tackle this issue by removing the irqpoll parameter on aws aarch64 platform, until the real root cause in acpi is found and resolved.
Signed-off-by: Pingfan Liu piliu@redhat.com
kdumpctl | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
diff --git a/kdumpctl b/kdumpctl index 6188d47..3eb57e3 100755 --- a/kdumpctl +++ b/kdumpctl @@ -641,6 +641,18 @@ function remove_kdump_kernel_key() keyctl unlink "$KDUMP_KEY_ID" %:.ima }
+function is_aws_aarch64() +{
- local _bios_model;
- _bios_model=$(lscpu | grep "BIOS Model name")
I'm not sure lscpu is reliable here. I just checked two instance types (two I had running were a1.metal and c6g.xlarge) and this didn't work.
Actually, it was because I was running `lscpu` as non-root. As root it shows up:
``` [bound] -bash-5.1$ sudo lscpu | grep "BIOS Model name" BIOS Model name: AWS Graviton2 AWS Graviton2 CPU @ 2.5GHz ```
So this LGTM.
On Sat, Jul 9, 2022 at 1:07 AM Dusty Mabe dustymabe@redhat.com wrote:
On 7/8/22 12:54, Dusty Mabe wrote:
On 6/30/22 00:04, Pingfan Liu wrote:
Currently, kdump may experience failure on some aws aarch64 platform. The final scenario is:
[ 79.145089] printk: console [ttyS0] disabledThen the system has no response any more. And after reboot, there is no vmcore generated under /var/crash/. More detail [1].
In a short word, it is caused by the irqpoll policy and some unknown acpi issue. The serial device is hot-removed as a pci device.
More detailed, the irqpoll policy demands to iterate over all interrupt handler, if the interrupt line is shared, then the handler is dispatched. And acpi handler acpi_irq() is on a shared interrupt line, so it is called. But for some unknown reason, the acpi hardware regs hold wrong state, and the acpi driver decides that a hot-removed event happens on a pci slot, which finally removes the pci serial device.
To tackle this issue by removing the irqpoll parameter on aws aarch64 platform, until the real root cause in acpi is found and resolved.
Signed-off-by: Pingfan Liu piliu@redhat.com
kdumpctl | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
diff --git a/kdumpctl b/kdumpctl index 6188d47..3eb57e3 100755 --- a/kdumpctl +++ b/kdumpctl @@ -641,6 +641,18 @@ function remove_kdump_kernel_key() keyctl unlink "$KDUMP_KEY_ID" %:.ima }
+function is_aws_aarch64() +{
- local _bios_model;
- _bios_model=$(lscpu | grep "BIOS Model name")
I'm not sure lscpu is reliable here. I just checked two instance types (two I had running were a1.metal and c6g.xlarge) and this didn't work.
Actually, it was because I was running `lscpu` as non-root. As root it shows up:
Yes, as kdump should run as a root.
Thanks,
Pingfan
Patch merged, thanks!
On Thu, Jun 30, 2022 at 12:04:08PM +0800, Pingfan Liu wrote:
Currently, kdump may experience failure on some aws aarch64 platform. The final scenario is:
[ 79.145089] printk: console [ttyS0] disabled Then the system has no response any more. And after reboot, there is no vmcore generated under /var/crash/. More detail [1].
In a short word, it is caused by the irqpoll policy and some unknown acpi issue. The serial device is hot-removed as a pci device.
More detailed, the irqpoll policy demands to iterate over all interrupt handler, if the interrupt line is shared, then the handler is dispatched. And acpi handler acpi_irq() is on a shared interrupt line, so it is called. But for some unknown reason, the acpi hardware regs hold wrong state, and the acpi driver decides that a hot-removed event happens on a pci slot, which finally removes the pci serial device.
To tackle this issue by removing the irqpoll parameter on aws aarch64 platform, until the real root cause in acpi is found and resolved.
Signed-off-by: Pingfan Liu piliu@redhat.com
kdumpctl | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
diff --git a/kdumpctl b/kdumpctl index 6188d47..3eb57e3 100755 --- a/kdumpctl +++ b/kdumpctl @@ -641,6 +641,18 @@ function remove_kdump_kernel_key() keyctl unlink "$KDUMP_KEY_ID" %:.ima }
+function is_aws_aarch64() +{
- local _bios_model;
- _bios_model=$(lscpu | grep "BIOS Model name")
- if [[ "${_bios_model}" =~ "AWS Graviton" ]]; then
return 0- fi
- return 1
+}
# Load the kdump kernel specified in /etc/sysconfig/kdump # If none is specified, try to load a kdump kernel with the same version # as the currently running kernel. @@ -650,6 +662,9 @@ load_kdump()
KEXEC_ARGS=$(prepare_kexec_args "${KEXEC_ARGS}") KDUMP_COMMANDLINE=$(prepare_cmdline "${KDUMP_COMMANDLINE}" "${KDUMP_COMMANDLINE_REMOVE}" "${KDUMP_COMMANDLINE_APPEND}")
if is_aws_aarch64; then
KDUMP_COMMANDLINE=$(echo ${KDUMP_COMMANDLINE} | sed 's/irqpoll//' )fi
# For secureboot enabled machines, use new kexec file based syscall. # Old syscall will always fail as it does not have capability to
-- 2.31.1
-
Coiby Xu -
Dusty Mabe -
Pingfan Liu