The root cause is likely an infinite loop or corrupted data somewhere, but not too many vmcore dmessages. 
 
The fix of limiting vmcore-dmesg.txt size is mainly to defend kdump servers against the existing problem, as well as other future problems. It doesn’t prevent us from further investigating the root cause of the problem that is hard to reproduce either. 

Thanks,
Jun

Sent from Yahoo Mail for iPhone

On Thursday, August 8, 2019, 10:45 PM, Dave Young <dyoung@redhat.com> wrote:

On 08/06/19 at 07:22pm, Lianbo Jiang wrote:
> From: Jun Wang <junw99@yahoo.com>
>
> With some corrupted vmcore files, the vmcore-dmesg.txt file may grow
> forever till the kdump disk becomes full, and also probably causes
> the disk error messages as follow:
> ...
> sd 0:0:0:0: [sda] tag#6 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
> sd 0:0:0:0: [sda] tag#6 CDB: Read(10) 28 00 08 06 4c 98 00 00 08 00
> blk_update_request: I/O error, dev sda, sector 134630552
> sd 0:0:0:0: [sda] tag#7 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
> sd 0:0:0:0: [sda] tag#7 CDB: Read(10) 28 00 08 06 4c 98 00 00 08 00
> blk_update_request: I/O error, dev sda, sector 134630552
> ...

Rethink about the problem, if the log_buf_len is not corrupted, it will
be fine, in kernel code log_buf_len is a u32 int. so it will be not possible to fill disk forever
unless vmcore-dmesg  is buggy.

Also about the implementation, it looks not very elegant.  Can you try
to add some limitation in vmcore-dmesg.c?

Actually in upstream kernel there is a macro LOG_BUF_LEN_MAX which is
defined as 2G,  so limit the file output to within 2G should be fine.


>
> Lets limit the size of vmcore-dmesg.txt to avoid such problems.
>
> Signed-off-by: Jun Wang <junw99@yahoo.com>
> Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
> ---
> Changes since v1:
> [1] Add dump_fs path to limit the size of vmcore-dmesg.txt
> [2] Add the option 'iflag=fullblock' for the dd command.
>
>  dracut-kdump.sh        | 4 ++--
>  kdump-lib-initramfs.sh | 4 ++--
>  2 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/dracut-kdump.sh b/dracut-kdump.sh
> index 2ae1c7c5d70d..ddd96efb5184 100755
> --- a/dracut-kdump.sh
> +++ b/dracut-kdump.sh
> @@ -102,8 +102,8 @@ save_vmcore_dmesg_ssh() {
>      local _opts="$3"
>      local _location=$4

> -    echo "kdump: saving vmcore-dmesg.txt"
> -    $_dmesg_collector /proc/vmcore | ssh $_opts $_location "dd of=$_path/vmcore-dmesg-incomplete.txt"
> +    echo "kdump: saving vmcore-dmesg.txt, up to 100MB"
> +    $_dmesg_collector /proc/vmcore | ssh $_opts $_location "dd of=$_path/vmcore-dmesg-incomplete.txt bs=512 count=204800 iflag=fullblock"
>      _exitcode=$?

>      if [ $_exitcode -eq 0 ]; then
> diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh
> index 608dc6efc07e..44f9ae4dfb8f 100755
> --- a/kdump-lib-initramfs.sh
> +++ b/kdump-lib-initramfs.sh
> @@ -137,8 +137,8 @@ save_vmcore_dmesg_fs() {
>      local _dmesg_collector=$1
>      local _path=$2

> -    echo "kdump: saving vmcore-dmesg.txt"
> -    $_dmesg_collector /proc/vmcore > ${_path}/vmcore-dmesg-incomplete.txt
> +    echo "kdump: saving vmcore-dmesg.txt, up to 100MB"
> +    $_dmesg_collector /proc/vmcore | dd of=$_path/vmcore-dmesg-incomplete.txt bs=512 count=204800 iflag=fullblock
>      _exitcode=$?
>      if [ $_exitcode -eq 0 ]; then
>          mv ${_path}/vmcore-dmesg-incomplete.txt ${_path}/vmcore-dmesg.txt
> --
> 2.17.1

>

Thanks
Dave