This is a patchset to add fence kdump support.
In cluster environment, fence kdump is used to notify all the other nodes that current is crashed and stop from being fenced off.
The patchset has the following features:
1. rebuild kdump initrd regarding timestamp of fence kdump config or cluster configuration. 2. setup a required working environment for fence kdump in 2nd kernel. 3. fence_kdump_send notify other nodes to stop the crashed one being fenced off before dumping process.
WANG Chao (5): kdump-lib: add common variables and function for fence kdump kdumpctl: rebuild kdump initramfs if cluster or fence_kdump config is changed. kdump.sh: send fence kdump message to other nodes in the cluster module-setup.sh: setup fence kdump environment module-setup: remove duplicated ip= line
arthur (1): doc: Add kdump-in-cluster-environment.txt
dracut-kdump.sh | 15 +++++++++++ dracut-module-setup.sh | 54 ++++++++++++++++++++++++++++++++++++-- kdump-in-cluster-environment.txt | 56 ++++++++++++++++++++++++++++++++++++++++ kdump-lib.sh | 17 +++++++++++- kdumpctl | 26 +++++++++++++++++++ kexec-tools.spec | 3 +++ 6 files changed, 168 insertions(+), 3 deletions(-) create mode 100644 kdump-in-cluster-environment.txt
From: arthur zzou@redhat.com
Since kdump already support dump in cluster environment, this patch add a howto file to RPM package to describe how to configure kdump in cluster environment.
Signed-off-by: arthur zzou@redhat.com --- kdump-in-cluster-environment.txt | 56 ++++++++++++++++++++++++++++++++++++++++ kexec-tools.spec | 3 +++ 2 files changed, 59 insertions(+) create mode 100644 kdump-in-cluster-environment.txt
diff --git a/kdump-in-cluster-environment.txt b/kdump-in-cluster-environment.txt new file mode 100644 index 0000000..1e6a43a --- /dev/null +++ b/kdump-in-cluster-environment.txt @@ -0,0 +1,56 @@ +Kdump-in-cluster-environment HOWTO + +Introduction + +Kdump is a kexec based crash dumping mechansim for Linux. This docuement +illustrate how to configure kdump in cluster environment to allow the kdump +crash recovery service complete without being preempted by traditional power +fencing methods. + +Overview + +Kexec/Kdump + +Details about Kexec/Kdump are available in Kexec-Kdump-howto file and will not +be described here. + +fence_kdump + +fence_kdump is an I/O fencing agent to be used with the kdump crash recovery +service. When the fence_kdump agent is invoked, it will listen for a message +from the failed node that acknowledges that the failed node it executing the +kdump crash kernel. Note that fence_kdump is not a replacement for traditional +fencing methods. The fence_kdump agent can only detect that a node has entered +the kdump crash recovery service. This allows the kdump crash recovery service +complete without being preempted by traditional power fencing methods. + +How to configure cluster environment: + +If we want to use kdump in cluster environment, fence-agents-kdump should be +installed in every nodes in the cluster. You can achieve this via the following +command: + + # yum install -y fence-agents-kdump + +Next is to add kdump_fence to the cluster. Assuming that the cluster consists +of three nodes, they are node1, node2 and node3, and use Pacemaker to perform +resource management and pcs as cli configuration tool. + +With pcs it is easy to add a stonith resource to the cluster. For example, add +a stonith resource named mykdumpfence with fence type of fence_kdump via the +following commands: + + # pcs stonith create mykdumpfence fence_kdump \ + pcmk_host_check=static-list pcmk_host_list="node1 node2 node3" + # pcs stonith update mykdumpfence pcmk_monitor_action=metadata --force + # pcs stonith update mykdumpfence pcmk_status_action=metadata --force + # pcs stonith update mykdumpfence pcmk_reboot_action=off --force + +Then enable stonith + # pcs property set stonith-enabled=true + +How to configure kdump: + +Actually there is nothing special in configuration between normal kdump and +cluster environment kdump. So please refer to Kexec-Kdump-howto file for more +information. diff --git a/kexec-tools.spec b/kexec-tools.spec index 0b948c2..219bef1 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -25,6 +25,7 @@ Source17: rhcrashkernel-param Source18: kdump.sysconfig.s390x Source19: eppic_030413.tar.gz Source20: kdump-lib.sh +Source21: kdump-in-cluster-environment.txt
####################################### # These are sources for mkdumpramfs @@ -162,6 +163,7 @@ export CFLAGS="-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2" rm -f kexec-tools.spec.in # setup the docs cp %{SOURCE10} . +cp %{SOURCE21} .
make %ifarch %{ix86} x86_64 ia64 ppc64 s390x @@ -346,6 +348,7 @@ done %doc COPYING %doc TODO %doc kexec-kdump-howto.txt +%doc kdump-in-cluster-environment.txt
%ifarch %{ix86} x86_64 ia64 ppc64 s390x %files eppic
Add following common variables and function:
$FENCE_KDUMP_CONIFG: configuration file /etc/sysconfig/fence_kdump $FENCE_KDUMP_NODES: configuration file /etc/fence_kdump_nodes $FENCE_KDUMP_SEND: executable /usr/libexec/fence_kdump_send is_fence_kdump(): used to determine if the system is in a cluster and configured with fence_kdump.
Signed-off-by: WANG Chao chaowang@redhat.com --- kdump-lib.sh | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/kdump-lib.sh b/kdump-lib.sh index e73ac09..aac0c5f 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -1,8 +1,12 @@ #!/bin/sh # -# Kdump common functions +# Kdump common variables and functions #
+FENCE_KDUMP_CONFIG="/etc/sysconfig/fence_kdump" +FENCE_KDUMP_SEND="/usr/libexec/fence_kdump_send" +FENCE_KDUMP_NODES="/etc/fence_kdump_nodes" + is_ssh_dump_target() { grep -q "^ssh[[:blank:]].*@" /etc/kdump.conf @@ -22,3 +26,14 @@ strip_comments() { echo $@ | sed -e 's/(.*)#.*/\1/' } + +# Check if fence kdump is configured in cluster +is_fence_kdump() +{ + # no pcs or fence_kdump_send executables installed? + type -P pcs > /dev/null || return 1 + [ -x $FENCE_KDUMP_SEND ] || return 1 + + # fence kdump not configured? + (pcs cluster cib | grep -q 'type="fence_kdump"') &> /dev/null || return 1 +}
If the system is configured fence kdump, we need to update kdump initramfs if cluster or fence_kdump config is newer.
In RHEL7, cluster config is no longer keeping locally but stored remotely. Fortunately we can use a pcs tool to retrieve the xml based config and parse the last changed time from that.
/etc/sysconfig/fence_kdump is used to configure runtime arguments to fence_kdump_send. So We have to pass the arguments to 2nd kernel.
When cluster config or /etc/sysconfig/fence_kdump is newer than local kdump initramfs, we must rebuild initramfs to adapt changes in cluster.
For example:
Detected change(s) the following file(s):
cluster-cib /etc/sysconfig/fence_kdump Rebuilding /boot/initramfs-xxxkdump.img [..]
Signed-off-by: WANG Chao chaowang@redhat.com --- kdumpctl | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
diff --git a/kdumpctl b/kdumpctl index 46ae633..abcdffd 100755 --- a/kdumpctl +++ b/kdumpctl @@ -132,6 +132,25 @@ function check_config() return 0 }
+# check_fence_kdump <image timestamp> +# return 0 if fence_kdump is configured and kdump initrd needs to be rebuilt +function check_fence_kdump() +{ + local image_time=$1 + local cib_time + + is_fence_kdump || return 1 + + cib_time=`pcs cluster cib | xmllint --xpath 'string(/cib/@cib-last-written)' - | \ + xargs -0 date +%s --date` + + if [ -z $cib_time -o $cib_time -le $image_time ]; then + return 1 + fi + + return 0 +} + function check_rebuild() { local extra_modules modified_files="" @@ -167,6 +186,9 @@ function check_rebuild() image_time=0 fi
+ #also rebuild when cluster conf is changed and fence kdump is enabled. + check_fence_kdump $image_time && modified_files="cluster-cib" + EXTRA_BINS=`grep ^kdump_post $KDUMP_CONFIG_FILE | cut -d\ -f2` CHECK_FILES=`grep ^kdump_pre $KDUMP_CONFIG_FILE | cut -d\ -f2` EXTRA_BINS="$EXTRA_BINS $CHECK_FILES" @@ -174,6 +196,10 @@ function check_rebuild() EXTRA_BINS="$EXTRA_BINS $CHECK_FILES" files="$KDUMP_CONFIG_FILE $kdump_kernel $EXTRA_BINS"
+ if [ -f $FENCE_KDUMP_CONFIG ]; then + files="$files $FENCE_KDUMP_CONFIG" + fi + check_exist "$files" && check_executable "$EXTRA_BINS" [ $? -ne 0 ] && return 1
In 2nd kernel, to prevent the crashed system from being fenced off, fence kdump message must be send to other nodes in the cluster periodically before dumping process.
We preserve every node's name in /etc/fence_kdump_nodes in the initrd, so we parse this file and send notify them.
Signed-off-by: WANG Chao chaowang@redhat.com --- dracut-kdump.sh | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index 4d8616f..bdef83b 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -287,6 +287,21 @@ read_kdump_conf() done < $conf_file }
+fence_kdump_notify() +{ + if [ -f $FENCE_KDUMP_NODES ]; then + if [ -f $FENCE_KDUMP_CONFIG ]; then + . $FENCE_KDUMP_CONFIG + fi + + while read nodes; + do + $FENCE_KDUMP_SEND $FENCE_KDUMP_OPTS $nodes & + done + fi +} + +fence_kdump_notify read_kdump_conf
if [ -z "$CORE_COLLECTOR" ];then
This patch is used to setup fence kdump environment when building kdump initrd: 1. Check if it's cluster and fence_kdump is configured. 2. Get all the nodes in the cluster and pass them to 2nd kernel via /etc/fence_kdump_nodes 3. Setup network interface which will be used by fence kdump notifier in 2nd kernel. 4. Install fence kdump notifier (/usr/libexec/fence_kdump_send) to initrd.
Signed-off-by: WANG Chao chaowang@redhat.com --- dracut-module-setup.sh | 43 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-)
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index c013430..e889aa1 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -20,6 +20,10 @@ depends() { _dep="$_dep drm" fi
+ if is_fence_kdump; then + _dep="$_dep network" + fi + echo $_dep return 0 } @@ -234,9 +238,14 @@ kdump_install_net() { fi
kdump_setup_netdev "${_netdev}" + #save netdev used for kdump as cmdline - echo "kdumpnic=${_netdev}" > ${initdir}/etc/cmdline.d/60kdumpnic.conf - echo "bootdev=${_netdev}" > ${initdir}/etc/cmdline.d/70bootdev.conf + #fence kdump would override bootdev and kdumpnic, we should avoid that. + if [ ! -f ${initdir}${initdir}/etc/cmdline.d/60kdumpnic.conf ] && + [ ! -f ${initdir}/etc/cmdline.d/70bootdev.conf ]; then + echo "kdumpnic=${_netdev}" > ${initdir}/etc/cmdline.d/60kdumpnic.conf + echo "bootdev=${_netdev}" > ${initdir}/etc/cmdline.d/70bootdev.conf + fi }
#install kdump.conf and what user specifies in kdump.conf @@ -263,6 +272,7 @@ kdump_install_conf() { esac done < /etc/kdump.conf
+ kdump_check_fence_kdump > ${initdir}/$FENCE_KDUMP_NODES inst "/tmp/$$-kdump.conf" "/etc/kdump.conf" rm -f /tmp/$$-kdump.conf } @@ -393,6 +403,35 @@ kdump_check_iscsi_targets () { }
+# setup fence_kdump in cluster +# setup proper network and install needed files +# also output '[node list]' for 2nd kernel /etc/fence_kdump_nodes +kdump_check_fence_kdump () { + is_fence_kdump || return 1 + + # get cluster nodes from cluster cib, get interface and ip address + nodelist=`pcs cluster cib | xmllint --xpath "/cib/status/node_state/@uname" -` + + # nodelist is formed as 'uname="node1" uname="node2" ... uname="nodeX"' + # we need to convert each to node1, node2 ... nodeX in each iteration + for node in ${nodelist}; do + # convert $node from 'uname="nodeX"' to 'nodeX' + eval $node + nodename=$uname + # Skip its own node name + if [ "$nodename" = `hostname` ]; then + continue + fi + echo -n "$nodename " + + kdump_install_net $nodename + done + echo + + dracut_install $FENCE_KDUMP_SEND + dracut_install -o $FENCE_KDUMP_CONFIG +} + install() { kdump_install_conf >"$initdir/lib/dracut/no-emergency-shell"
In the remote dump case, and if fence kdump is configured, it's almost 100% sure that the same network interface will be setup more than once. One time for network dump, the other times for fence kdump. The result is we will have two or more duplicated ip= configuration in 40ip.conf.
These are exactly duplicates, however dracut will refuse to continue and raise a fatal error if there are duplicated configuration for the same interface. We should simply remove the duplicates to avoid this awkward situation.
Signed-off-by: WANG Chao chaowang@redhat.com --- dracut-module-setup.sh | 11 +++++++++++ 1 file changed, 11 insertions(+)
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index e889aa1..07cfd12 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -179,6 +179,13 @@ kdump_setup_znet() { echo rd.znet=${NETTYPE},${SUBCHANNELS}${_options} > ${initdir}/etc/cmdline.d/30znet.conf }
+# Remove duplicate ip configurations in 40ip.conf +kdump_remove_dupliate_ip_opts() { + mv ${initdir}/etc/cmdline.d/40ip.conf ${initdir}/etc/cmdline.d/40ip.conf.tmp + sort ${initdir}/etc/cmdline.d/40ip.conf.tmp | uniq > ${initdir}/etc/cmdline.d/40ip.conf + rm -f ${initdir}/etc/cmdline.d/40ip.conf.tmp +} + # Setup dracut to bringup a given network interface kdump_setup_netdev() { local _netdev=$1 @@ -210,6 +217,10 @@ kdump_setup_netdev() { echo " ifname=$_netdev:$(kdump_get_mac_addr $_netdev)" >> ${initdir}/etc/cmdline.d/40ip.conf fi
+ # dracut doesn't allow duplicated configuration for same NIC, even they're exactly the same. + # so we have to filter out the duplicates. + kdump_remove_dupliate_ip_opts + kdump_setup_dns "$_netdev" }
-
WANG Chao