cluster: RHEL6 - dlm_controld: adjust fence time comparison

David Teigland teigland at fedoraproject.org
Fri Jan 10 22:15:45 UTC 2014


Gitweb:        http://git.fedorahosted.org/git/?p=cluster.git;a=commitdiff;h=2d06dd478c27bf864ba1a5ac0cbb1ba6c3ed947f
Commit:        2d06dd478c27bf864ba1a5ac0cbb1ba6c3ed947f
Parent:        cca7cf733d03a58d94eb4ab3bee7dcc2e39b7ea1
Author:        David Teigland <teigland at redhat.com>
AuthorDate:    Fri Jan 10 16:01:35 2014 -0600
Committer:     David Teigland <teigland at redhat.com>
CommitterDate: Fri Jan 10 16:01:35 2014 -0600

dlm_controld: adjust fence time comparison

An unusual combination of events can cause the fence
time comparison to not work properly, leaving
dlm_controld recovery stuck.

If fencing in fenced completes very quickly, and the
cpg callback into dlm_controld is delayed, the effect
is that the fence_time returned from fenced is later
than the fail_time recorded in the cpg callback.
dlm_controld requires that the fencing time is after
the fail time.

This is solved by saving the add_time when fail_time
is recorded as need_fence_after.  The fencing check
is then changed to also succeed if fence_time is
later than need_fence_after.  A simple comparison
with add_time does not work as shown in commit
4039bf4817a96b6aab20de948389f43b89ce4a8e.

bz 843160

Signed-off-by: David Teigland <teigland at redhat.com>
---
 group/dlm_controld/cpg.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/group/dlm_controld/cpg.c b/group/dlm_controld/cpg.c
index 6a4023b..795efc4 100644
--- a/group/dlm_controld/cpg.c
+++ b/group/dlm_controld/cpg.c
@@ -47,6 +47,7 @@ struct node {
 	uint64_t add_time;
 	uint64_t fail_time;
 	uint64_t fence_time;	/* for debug */
+	uint64_t need_fence_after;
 	uint64_t cluster_add_time;
 	uint64_t cluster_remove_time;
 	uint32_t fence_queries;	/* for debug */
@@ -502,6 +503,7 @@ static void node_history_fail(struct lockspace *ls, int nodeid,
 		node->fence_time = 0;
 		node->fence_queries = 0;
 		node->fail_time = time(NULL);
+		node->need_fence_after = node->add_time;
 	}
 
 	/* fenced will take care of making sure the quorum value
@@ -546,12 +548,20 @@ static int check_fencing_done(struct lockspace *ls)
 		   we've seen fenced_time within the same second as
 		   fail_time: with external fencing, e.g. fence_node */
 
-		if (last_fenced_time >= node->fail_time) {
+		/* the comparison with need_fence_after is to deal with
+		   the odd case where fencing completes very quickly in
+		   fenced and there is a delay of the delivery of the cpg
+		   callback (and setting fail_time) in dlm_controld,
+		   placing the fail_time after the fence_time. */
+
+		if ((last_fenced_time >= node->fail_time) ||
+		    (last_fenced_time > node->need_fence_after)) {
 			log_group(ls, "check_fencing %d done "
-				  "add %llu fail %llu last %llu",
+				  "add %llu fail %llu need %llu last %llu",
 				  node->nodeid,
 				  (unsigned long long)node->add_time,
 				  (unsigned long long)node->fail_time,
+				  (unsigned long long)node->need_fence_after,
 				  (unsigned long long)last_fenced_time);
 			node->check_fencing = 0;
 			node->add_time = 0;
@@ -560,10 +570,11 @@ static int check_fencing_done(struct lockspace *ls)
 			if (!node->fence_queries ||
 			    node->fence_time != last_fenced_time) {
 				log_group(ls, "check_fencing %d wait "
-					  "add %llu fail %llu last %llu",
+					  "add %llu fail %llu need %llu last %llu",
 					  node->nodeid,
 					 (unsigned long long)node->add_time,
 					 (unsigned long long)node->fail_time,
+				  	 (unsigned long long)node->need_fence_after,
 					 (unsigned long long)last_fenced_time);
 				node->fence_queries++;
 				node->fence_time = last_fenced_time;


More information about the cluster-commits mailing list