[PATCHv3] kdumpctl: wait a while for network ready if dump target is ssh
by Pingfan Liu
If dump target is ipv6 address, a host should have ipv6 address ready
before starting kdump service. Otherwise, kdump service fails to start due
to the failure "ssh dump_server_ip mkdir -p $SAVE_PATH".
And user can see message like:
"Could not create root@2620:52:0:10da:46a8:42ff:fe23:3272/var/crash"
I observe a long period (about 30s) on some machine before they got ipv6
address dynamiclly, which is never seen on ipv4 host.
Hence kdump service has a dependency on ipv6 address. But there is no good
way to resolve it. One way is asking user to run the cmd "nmcli connection
modify eth0 ipv6.may-fail false". But this will block systemd until ipv6
address is ready. Despite doing so, kdump can try its best (wait 1 minutes
after it starts up) before failure.
How to implement the wait is arguable. It will involve too many technique
details if explicitly waiting on ipv6 address, instead, just lean on 'ssh'
return value to see the availability of network.
Signed-off-by: Pingfan Liu <piliu(a)redhat.com>
---
kdumpctl | 36 ++++++++++++++++++++++++++++++++----
1 file changed, 32 insertions(+), 4 deletions(-)
diff --git a/kdumpctl b/kdumpctl
index a1a6ee2..86511b8 100755
--- a/kdumpctl
+++ b/kdumpctl
@@ -730,12 +730,40 @@ check_ssh_config()
return 0
}
+# ipv6 host address may takes a long time to be ready.
+# Instead of checking against ipv6 address, we just check the network reachable
+# by the return val of 'ssh'
+check_and_wait_network_ready()
+{
+ local start_time=$(date +%s)
+ local cur
+ local diff
+
+
+ while true; do
+ ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH
+ # ssh exits with the exit status of the remote command or with 255 if an error occurred
+ if [ $? -eq 0 ]; then
+ return 0
+ elif [ $? -ne 255 ]; then
+ return 1
+ fi
+ cur=$(date +%s)
+ diff=$( $cur - $start_time )
+ # 60s time out
+ if [ $diff -gt 60 ]; then
+ break;
+ fi
+ sleep 1
+ done
+
+ return 1
+}
+
check_ssh_target()
{
- local _ret
- ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH
- _ret=$?
- if [ $_ret -ne 0 ]; then
+ check_and_wait_network_ready
+ if [ $? -ne 0 ]; then
echo "Could not create $DUMP_TARGET:$SAVE_PATH, you probably need to run \"kdumpctl propagate\"" >&2
return 1
fi
--
2.7.5
4 years, 3 months
[PATCH] kdumpctl: wait a while for ipv6 addr ready if dump target is ipv6
by Pingfan Liu
If dump target is ipv6 address, a host should have ipv6 address ready
before starting kdump service. Otherwise, kdump service fails to start due
to the failure "ssh dump_server_ip mkdir -p $SAVE_PATH".
And user can see message like:
"Could not create root@2620:52:0:10da:46a8:42ff:fe23:3272/var/crash"
A host can get ipv6 address by three ways: Neighbor Discovery Protocol/
DHCPv6/ static.
I observe a long period (about 30s) on some machine before they got ipv6
address dynamiclly, which is never seen on ipv4 host.
Hence kdump service has a dependency on ipv6 address. But there is no good
way to resolve it. One way is asking user to run the cmd "nmcli connection
modify eth0 ipv6.may-fail false". But this will block systemd until ipv6
address is ready. Despite doing so, kdump can try its best (wait 1 minutes
after it starts up) before failure.
Signed-off-by: Pingfan Liu <piliu(a)redhat.com>
---
kdumpctl | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/kdumpctl b/kdumpctl
index a1a6ee2..02302a1 100755
--- a/kdumpctl
+++ b/kdumpctl
@@ -730,9 +730,38 @@ check_ssh_config()
return 0
}
+# Alternative way, using "nmcli connection modify eth0 ipv6.may-fail false"
+# to force systemd to wait until ipv6 address ready.
+# But it is better to let kdump wait a minute on ipv6 address
+check_and_wait_ipv6_ready()
+{
+ local ipaddr=$1
+ local ipv6_up="false"
+ local left=60
+
+ ipaddr=$(echo $ipaddr | awk -F "@" '{ print $2 }')
+ if is_ipv6_address $ipaddr; then
+ while [ $left -gt 0 ]; do
+ ip -f inet6 addr | grep global
+ if [ ${PIPESTATUS[1]} -eq 0 ]; then
+ ipv6_up="true"
+ break;
+ fi
+ left=$(expr $left - 1)
+ sleep 1
+ done
+
+ if [ "$ipv6_up" == "false" ]; then
+ echo "ipv6 address is not ready within more than 1 minute"
+ fi
+ fi
+}
+
check_ssh_target()
{
local _ret
+
+ check_and_wait_ipv6_ready $DUMP_TARGET
ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH
_ret=$?
if [ $_ret -ne 0 ]; then
--
2.7.5
4 years, 3 months
[PATCH] Add the 'UserKnownHostsFile' option to authenticate the kdump server
by Lianbo Jiang
When using the ssh kdump, the ssh client will use the '-o StrictHostKeyChecking=yes'
option to connect the kdump server. To ensure the dump file is going to be sent to
a true kdump server as opposed to any fake one, the ssh client needs to authenticate
the server as well. A known_hosts file with server hostkey can be provisioned on the
client side in advance to facilitate the authentication(which is added by ssh-client
module in mkdumprd).
Add a configuration in '/etc/kdump.conf' to specify the location of the known_hosts
file to be used in the 'ssh -o UserKnownHostsFile' option.
Suggested-by: Jun Wang <junw99(a)yahoo.com>
Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
---
dracut-kdump.sh | 4 ++--
kdump-lib-initramfs.sh | 6 ++++++
kdump.conf | 4 ++++
kdumpctl | 12 ++++++++++--
mkdumprd | 11 +++++++++--
5 files changed, 31 insertions(+), 6 deletions(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh
index ce56459ed088..0eafe6458530 100755
--- a/dracut-kdump.sh
+++ b/dracut-kdump.sh
@@ -71,7 +71,7 @@ dump_raw()
dump_ssh()
{
- local _opt="-i $1 -o BatchMode=yes -o StrictHostKeyChecking=yes"
+ local _opt="-i $1 -o BatchMode=yes -o StrictHostKeyChecking=yes -i $3 -o UserKnownHostsFile"
local _dir="$KDUMP_PATH/$HOST_IP-$DATEDIR"
local _host=$2
@@ -156,7 +156,7 @@ read_kdump_conf()
add_dump_code "dump_raw $config_val"
;;
ssh)
- add_dump_code "dump_ssh $SSH_KEY_LOCATION $config_val"
+ add_dump_code "dump_ssh $SSH_KEY_LOCATION $config_val $SSH_KNOWN_HOSTS"
;;
esac
done <<< "$(read_strip_comments $KDUMP_CONF)"
diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh
index 608dc6efc07e..7d595d5b7d06 100755
--- a/kdump-lib-initramfs.sh
+++ b/kdump-lib-initramfs.sh
@@ -11,6 +11,7 @@ DATEDIR=`date +%Y-%m-%d-%T`
HOST_IP='127.0.0.1'
DUMP_INSTRUCTION=""
SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa"
+SSH_KNOWN_HOSTS="/root/.ssh/known_hosts"
KDUMP_SCRIPT_DIR="/kdumpscripts"
DD_BLKSIZE=512
FINAL_ACTION="systemctl reboot -f"
@@ -38,6 +39,11 @@ get_kdump_confs()
SSH_KEY_LOCATION=$config_val
fi
;;
+ known_hosts)
+ if [ -f "$config_val" ];then
+ SSH_KNOWN_HOSTS="$config_val"
+ fi
+ ;;
kdump_pre)
KDUMP_PRE="$config_val"
;;
diff --git a/kdump.conf b/kdump.conf
index 1f0fc2ddc40b..2b10d57ac561 100644
--- a/kdump.conf
+++ b/kdump.conf
@@ -152,6 +152,9 @@
# to send fence_kdump notifications to.
# (this option is mandatory to enable fence_kdump).
#
+# known_hosts <path>
+# - The "path" represents the path of know_hosts, the default value
+# is /root/.ssh/known_hosts.
#raw /dev/vg/lv_kdump
#ext4 /dev/vg/lv_kdump
@@ -173,3 +176,4 @@ core_collector makedumpfile -l --message-level 1 -d 31
#dracut_args --omit-drivers "cfg80211 snd" --add-drivers "ext2 ext3"
#fence_kdump_args -p 7410 -f auto -c 0 -i 10
#fence_kdump_nodes node1 node2
+#known_hosts /root/.ssh/known_hosts
diff --git a/kdumpctl b/kdumpctl
index a1a6ee24b768..7ba7e8cf2685 100755
--- a/kdumpctl
+++ b/kdumpctl
@@ -9,6 +9,7 @@ MKDUMPRD="/sbin/mkdumprd -f"
DRACUT_MODULES_FILE="/usr/lib/dracut/modules.txt"
SAVE_PATH=/var/crash
SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa"
+SSH_KNOWN_HOSTS="/root/.ssh/known_hosts"
INITRD_CHECKSUM_LOCATION="/boot/.fadump_initrd_checksum"
DUMP_TARGET=""
DEFAULT_INITRD=""
@@ -243,7 +244,7 @@ check_config()
case "$config_opt" in
\#* | "")
;;
- raw|ext2|ext3|ext4|minix|btrfs|xfs|nfs|ssh|sshkey|path|core_collector|kdump_post|kdump_pre|extra_bins|extra_modules|failure_action|default|final_action|force_rebuild|force_no_rebuild|dracut_args|fence_kdump_args|fence_kdump_nodes)
+ raw|ext2|ext3|ext4|minix|btrfs|xfs|nfs|ssh|sshkey|known_hosts|path|core_collector|kdump_post|kdump_pre|extra_bins|extra_modules|failure_action|default|final_action|force_rebuild|force_no_rebuild|dracut_args|fence_kdump_args|fence_kdump_nodes)
# remove inline comments after the end of a directive.
[ -z "$config_val" ] && {
echo "Invalid kdump config value for option $config_opt."
@@ -711,6 +712,13 @@ check_ssh_config()
echo "WARNING: '$config_val' doesn't exist, using default value '$SSH_KEY_LOCATION'"
fi
;;
+ known_hosts)
+ if [ -f "$config_val" ];then
+ SSH_KNOWN_HOSTS=$(/usr/bin/readlink -m $config_val)
+ else
+ echo "WARNING: '$config_val' doesn't exist, using default value '$SSH_KNOWN_HOSTS'"
+ fi
+ ;;
path)
SAVE_PATH=$config_val
;;
@@ -733,7 +741,7 @@ check_ssh_config()
check_ssh_target()
{
local _ret
- ssh -q -i $SSH_KEY_LOCATION -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH
+ ssh -q -i $SSH_KEY_LOCATION -i $SSH_KNOWN_HOSTS -o BatchMode=yes $DUMP_TARGET mkdir -p $SAVE_PATH
_ret=$?
if [ $_ret -ne 0 ]; then
echo "Could not create $DUMP_TARGET:$SAVE_PATH, you probably need to run \"kdumpctl propagate\"" >&2
diff --git a/mkdumprd b/mkdumprd
index cf3533fe2be9..c4e2f8ba4c31 100644
--- a/mkdumprd
+++ b/mkdumprd
@@ -13,6 +13,7 @@ export IN_KDUMP=1
conf_file="/etc/kdump.conf"
SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa"
+SSH_KNOWN_HOSTS="/root/.ssh/known_hosts"
SAVE_PATH=$(awk '/^path/ {print $2}' $conf_file)
[ -z "$SAVE_PATH" ] && SAVE_PATH=$DEFAULT_PATH
# strip the duplicated "/"
@@ -144,7 +145,7 @@ is_readonly_mount() {
#called from while loop and shouldn't read from stdin, so we're using "ssh -n"
get_ssh_size() {
local _opt _out _size
- _opt="-i $SSH_KEY_LOCATION -o BatchMode=yes -o StrictHostKeyChecking=yes"
+ _opt="-i $SSH_KEY_LOCATION -o BatchMode=yes -o StrictHostKeyChecking=yes -i $SSH_KNOWN_HOSTS -o UserKnownHostsFile"
_out=$(ssh -q -n $_opt $1 "df -P $SAVE_PATH")
[ $? -ne 0 ] && {
perror_exit "checking remote ssh server available size failed."
@@ -162,7 +163,7 @@ get_ssh_size() {
mkdir_save_path_ssh()
{
local _opt _dir
- _opt="-i $SSH_KEY_LOCATION -o BatchMode=yes -o StrictHostKeyChecking=yes"
+ _opt="-i $SSH_KEY_LOCATION -o BatchMode=yes -o StrictHostKeyChecking=yes -i $SSH_KNOWN_HOSTS -o UserKnownHostsFile"
ssh -qn $_opt $1 mkdir -p $SAVE_PATH 2>&1 > /dev/null
_ret=$?
if [ $_ret -ne 0 ]; then
@@ -385,6 +386,12 @@ if [ -f "$keyfile" ]; then
SSH_KEY_LOCATION=$(/usr/bin/readlink -m $keyfile)
fi
+# if specified, get the known_hosts
+known_hosts=$(awk '/^known_hosts/ {print $2}' $conf_file)
+if [ -f "$known_hosts" ];then
+ SSH_KNOWN_HOSTS=$(/usr/bin/readlink -m $known_hosts)
+fi
+
if [ "$(uname -m)" = "s390x" ]; then
add_dracut_module "znet"
fi
--
2.17.1
4 years, 4 months
[PATCH] Limit the size of vmcore-dmesg.txt to no more than 100MB
by Lianbo Jiang
From: Jun Wang <junw99(a)yahoo.com>
With some corrupted vmcore files, the vmcore-dmesg.txt file may grow
forever till the kdump disk becomes full, and also probably causes
the disk error messages as follow:
...
sd 0:0:0:0: [sda] tag#6 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] tag#6 CDB: Read(10) 28 00 08 06 4c 98 00 00 08 00
blk_update_request: I/O error, dev sda, sector 134630552
sd 0:0:0:0: [sda] tag#7 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 0:0:0:0: [sda] tag#7 CDB: Read(10) 28 00 08 06 4c 98 00 00 08 00
blk_update_request: I/O error, dev sda, sector 134630552
...
Lets limit the size of vmcore-dmesg.txt to avoid such problems.
Signed-off-by: Jun Wang <junw99(a)yahoo.com>
Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
---
dracut-kdump.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh
index 2ae1c7c5d70d..ce56459ed088 100755
--- a/dracut-kdump.sh
+++ b/dracut-kdump.sh
@@ -102,8 +102,8 @@ save_vmcore_dmesg_ssh() {
local _opts="$3"
local _location=$4
- echo "kdump: saving vmcore-dmesg.txt"
- $_dmesg_collector /proc/vmcore | ssh $_opts $_location "dd of=$_path/vmcore-dmesg-incomplete.txt"
+ echo "kdump: saving vmcore-dmesg.txt, up to 100MB"
+ $_dmesg_collector /proc/vmcore | ssh $_opts $_location "dd of=$_path/vmcore-dmesg-incomplete.txt bs=512 count=204800"
_exitcode=$?
if [ $_exitcode -eq 0 ]; then
--
2.17.1
4 years, 4 months
[PATCH 0/2] Fix failure of makedumpfile on Linux 5.3-rc1
by Kazuhito Hagio
From: Kazuhito Hagio <k-hagio(a)ab.jp.nec.com>
# resend this set bacause the previous one is being held..
These are both backports from upstream, and tested OK with kernel
5.3.0-0.rc1.git3.1.fc31.x86_64.
Patch 2/2 fixes a failure of makedumpfile on Linux 5.3-rc1 that
prints error like the following:
readmem: Can't convert a virtual address(fffffc97d1000000) to physical address.
readmem: type_addr: 0, addr:fffffc97d1000000, size:32768
__exclude_unnecessary_pages: Can't read the buffer of struct page.
create_2nd_bitmap: Can't exclude unnecessary pages.
Patch 1/2 is not necessary for fixing the issue, but without the patch,
if a similar issue occurs in the future, makedumpfile can take a long
time to finish generating a broken vmcore meaninglessly, and 2nd kernel
cannot reboot soon.
Kazuhito Hagio (2):
makedumpfile: Do not proceed when get_num_dumpable_cyclic() fails
makedumpfile: Increase SECTION_MAP_LAST_BIT to 4
...d-when-get_num_dumpable_cyclic-fails.patch | 44 +++++++++++++++++++
...e-Increase-SECTION_MAP_LAST_BIT-to-4.patch | 38 ++++++++++++++++
kexec-tools.spec | 4 ++
3 files changed, 86 insertions(+)
create mode 100644 kexec-tools-2.0.20-makedumpfile-Do-not-proceed-when-get_num_dumpable_cyclic-fails.patch
create mode 100644 kexec-tools-2.0.20-makedumpfile-Increase-SECTION_MAP_LAST_BIT-to-4.patch
--
2.20.1
4 years, 4 months
[PATCH] Don't forward and drop journalctl logs for fadump
by Kairui Song
fadump will alter the normal boot initramfs and we don't want a normal
boot to foward and drop the journalctl logs.
Signed-off-by: Kairui Song <kasong(a)redhat.com>
---
dracut-module-setup.sh | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index 9ef60210..3fa696da 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -839,9 +839,12 @@ install() {
fi
# Forward logs to console directly, this avoids unneccessary memory
- # consumption and make console output more useful
- mkdir -p ${initdir}/etc/systemd/journald.conf.d
- echo "[Journal]" > ${initdir}/etc/systemd/journald.conf.d/kdump.conf
- echo "Storage=none" >> ${initdir}/etc/systemd/journald.conf.d/kdump.conf
- echo "ForwardToConsole=yes" >> ${initdir}/etc/systemd/journald.conf.d/kdump.conf
+ # consumption and make console output more useful.
+ # Only do so for non fadump image.
+ if ! is_fadump_capable; then
+ mkdir -p ${initdir}/etc/systemd/journald.conf.d
+ echo "[Journal]" > ${initdir}/etc/systemd/journald.conf.d/kdump.conf
+ echo "Storage=none" >> ${initdir}/etc/systemd/journald.conf.d/kdump.conf
+ echo "ForwardToConsole=yes" >> ${initdir}/etc/systemd/journald.conf.d/kdump.conf
+ fi
}
--
2.21.0
4 years, 4 months
[PATCH 0/2] Fix failure of makedumpfile on Linux 5.3-rc1
by Kazuhito Hagio
These are both backports from upstream, and tested OK with kernel
5.3.0-0.rc1.git3.1.fc31.x86_64.
Patch 2/2 fixes a failure of makedumpfile on Linux 5.3-rc1 that
prints error like the following:
readmem: Can't convert a virtual address(fffffc97d1000000) to physical address.
readmem: type_addr: 0, addr:fffffc97d1000000, size:32768
__exclude_unnecessary_pages: Can't read the buffer of struct page.
create_2nd_bitmap: Can't exclude unnecessary pages.
Patch 1/2 is not necessary for fixing the issue, but without the patch,
if a similar issue occurs in the future, makedumpfile can take a long
time to finish generating a broken vmcore meaninglessly, and 2nd kernel
cannot reboot soon.
Kazuhito Hagio (2):
makedumpfile: Do not proceed when get_num_dumpable_cyclic() fails
makedumpfile: Increase SECTION_MAP_LAST_BIT to 4
...d-when-get_num_dumpable_cyclic-fails.patch | 44 +++++++++++++++++++
...e-Increase-SECTION_MAP_LAST_BIT-to-4.patch | 38 ++++++++++++++++
kexec-tools.spec | 4 ++
3 files changed, 86 insertions(+)
create mode 100644 kexec-tools-2.0.20-makedumpfile-Do-not-proceed-when-get_num_dumpable_cyclic-fails.patch
create mode 100644 kexec-tools-2.0.20-makedumpfile-Increase-SECTION_MAP_LAST_BIT-to-4.patch
--
2.20.1
4 years, 4 months
[PATCHv2] dracut-module-setup.sh: skip alias of localhost in get_pcs_fence_kdump_nodes()
by Pingfan Liu
The current code only exclude the hostname, while localhost can have alias in
/etc/hosts. All of the alias should be excluded from the fence dump node to
avoid deadlock issue.
Signed-off-by: Pingfan Liu <piliu(a)redhat.com>
---
dracut-module-setup.sh | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index 27b9f02..ca3b105 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -689,6 +689,21 @@ kdump_check_iscsi_targets () {
}
}
+is_localhost() {
+ local hostnames=$(hostname -A)
+ local shortnames=$(hostname -A -s)
+ local nodename=$1
+
+ hostnames="$hostnames $shortnames"
+
+ for name in ${hostnames}; do
+ if [ "$name" == "$nodename" ]; then
+ return 0
+ fi
+ done
+ return 1
+}
+
# retrieves fence_kdump nodes from Pacemaker cluster configuration
get_pcs_fence_kdump_nodes() {
local nodes
@@ -703,7 +718,7 @@ get_pcs_fence_kdump_nodes() {
eval $node
nodename=$uname
# Skip its own node name
- if [ "$nodename" = `hostname` -o "$nodename" = `hostname -s` ]; then
+ if is_localhost $nodename; then
continue
fi
nodes="$nodes $nodename"
--
2.20.1
4 years, 4 months