Re: Sanlock lockspace storage options
by David Weber
> Hi all/David,
Hi,
> Apologies if this is not the correct list - didn't see a user specific one.
>
> I'm a bit confused regarding what storage options I have regarding the
> lockspace directory with my particular setup. I have a 2 node RHCS
> cluster with shared iscsi-backed GFS2 storage for the VM disk images -
> fairly standard stuff. I am wanting to add Sanlock into the mix, but do
> not want to introduce a dependency on NFS for the Lockspace directory.
> It is also my understanding from reading various mailing list posts that
> it is also not considered good practice to have the Lockspace directory
> on the GFS2 storage due to blocked I/O on the GFS2 volume while fencing
> is taking place.
>
> I am not sure what my other options are for where to host the Lockspace
> directory at. The man page seems to show creating and initializing
> storage pools from a SAN, but it's not clear exactly what is being
> configured and how configuration, and performance, would differ with it
> vs using an NFS directory.
>
> Any help and direction towards what my options are for a 2 node setup
> with this is greatly appreciated!
We do something similar in a 2 node DRBD/OCFS2 setup in which the vm
images and the sanlock lockspace reside in the OCFS2 volume. We did a
thorough testing and didn't run into any major problems with fencing.
Problematic are of course all split-brain situations. The setup has now
ran since over a year. AFAIK OCFS2 handles fencing in a different
way than GFS2 so I don't know if this really helps you.
You could take a look at virtlockd which was included into libvirt
1.0.1. It is perhaps sufficient to your needs and doesn't need a
lockspace.
See: http://www.redhat.com/archives/libvir-list/2011-July/msg00337.html
Cheers
David
>
>
> Thanks!
_______________________________________________
> sanlock-devel mailing list
sanlock-devel(a)lists.fedorahosted.org
https://
> lists.fedorahosted.org/mailman/listinfo/sanlock-devel
11 years
Sanlock lockspace storage options
by Russell Jones
Hi all/David,
Apologies if this is not the correct list - didn't see a user specific one.
I'm a bit confused regarding what storage options I have regarding the
lockspace directory with my particular setup. I have a 2 node RHCS
cluster with shared iscsi-backed GFS2 storage for the VM disk images -
fairly standard stuff. I am wanting to add Sanlock into the mix, but do
not want to introduce a dependency on NFS for the Lockspace directory.
It is also my understanding from reading various mailing list posts that
it is also not considered good practice to have the Lockspace directory
on the GFS2 storage due to blocked I/O on the GFS2 volume while fencing
is taking place.
I am not sure what my other options are for where to host the Lockspace
directory at. The man page seems to show creating and initializing
storage pools from a SAN, but it's not clear exactly what is being
configured and how configuration, and performance, would differ with it
vs using an NFS directory.
Any help and direction towards what my options are for a 2 node setup
with this is greatly appreciated!
Thanks!
11 years
3 commits - python/example.py python/sanlock.c
by David Teigland
python/example.py | 7 ++++---
python/sanlock.c | 27 +++++++++++++++++++++------
2 files changed, 25 insertions(+), 9 deletions(-)
New commits:
commit 986362473b2dc493113dbfa75a88d7669a9c7500
Author: Federico Simoncelli <fsimonce(a)redhat.com>
Date: Wed Apr 10 12:09:45 2013 -0400
python: return lease version in read_resource
Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com>
diff --git a/python/sanlock.c b/python/sanlock.c
index 888e3b6..7d39deb 100644
--- a/python/sanlock.c
+++ b/python/sanlock.c
@@ -461,6 +461,14 @@ py_read_resource(PyObject *self __unused, PyObject *args, PyObject *keywds)
if (rv != 0)
goto exit_fail;
+ /* fill the dictionary information: version */
+ if ((rs_entry = PyInt_FromLong(rs->lver)) == NULL)
+ goto exit_fail;
+ rv = PyDict_SetItemString(rs_info, "version", rs_entry);
+ Py_DECREF(rs_entry);
+ if (rv != 0)
+ goto exit_fail;
+
/* success */
free(rs);
return rs_info;
commit 0ce413a53c768b147e0d0b97717e5e7f94479bdf
Author: Federico Simoncelli <fsimonce(a)redhat.com>
Date: Wed Apr 10 12:09:44 2013 -0400
python: add the versioning support to acquire
Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com>
diff --git a/python/example.py b/python/example.py
index 28b5180..354579e 100644
--- a/python/example.py
+++ b/python/example.py
@@ -29,7 +29,8 @@ def main():
try:
print "Acquiring '%s' on '%s'" % (RESOURCE_NAME, LOCKSPACE_NAME)
- sanlock.acquire(LOCKSPACE_NAME, RESOURCE_NAME, SNLK_DISKS, slkfd=fd)
+ sanlock.acquire(LOCKSPACE_NAME, RESOURCE_NAME, SNLK_DISKS, slkfd=fd,
+ version=0)
print "Releasing '%s' on '%s'" % (RESOURCE_NAME, LOCKSPACE_NAME)
sanlock.release(LOCKSPACE_NAME, RESOURCE_NAME, SNLK_DISKS, slkfd=fd)
finally:
diff --git a/python/sanlock.c b/python/sanlock.c
index 750157f..888e3b6 100644
--- a/python/sanlock.c
+++ b/python/sanlock.c
@@ -779,28 +779,29 @@ exit_fail:
/* acquire */
PyDoc_STRVAR(pydoc_acquire, "\
-acquire(lockspace, resource, disks [, slkfd=fd, pid=owner, shared=False])\n\
+acquire(lockspace, resource, disks \
+[, slkfd=fd, pid=owner, shared=False, version=0])\n\
Acquire a resource lease for the current process (using the slkfd argument\n\
to specify the sanlock file descriptor) or for an other process (using the\n\
pid argument). If shared is True the resource will be acquired in the shared\n\
-mode.\n\
+mode. The version is the version of the lease that must be acquired or fail.\n\
The disks must be in the format: [(path, offset), ... ]\n");
static PyObject *
py_acquire(PyObject *self __unused, PyObject *args, PyObject *keywds)
{
- int rv, sanlockfd = -1, pid = -1, shared = 0;
+ int rv, sanlockfd = -1, pid = -1, shared = 0, version = 0;
const char *lockspace, *resource;
struct sanlk_resource *res;
PyObject *disks;
static char *kwlist[] = {"lockspace", "resource", "disks", "slkfd",
- "pid", "shared", NULL};
+ "pid", "shared", "version", NULL};
/* parse python tuple */
- if (!PyArg_ParseTupleAndKeywords(args, keywds, "ssO!|iii", kwlist,
+ if (!PyArg_ParseTupleAndKeywords(args, keywds, "ssO!|iiii", kwlist,
&lockspace, &resource, &PyList_Type, &disks, &sanlockfd, &pid,
- &shared)) {
+ &shared, &version)) {
return NULL;
}
@@ -824,6 +825,12 @@ py_acquire(PyObject *self __unused, PyObject *args, PyObject *keywds)
res->flags |= SANLK_RES_SHARED;
}
+ /* prepare the resource version */
+ if (version) {
+ res->flags |= SANLK_RES_LVER;
+ res->lver = version;
+ }
+
/* acquire sanlock resource (gil disabled) */
Py_BEGIN_ALLOW_THREADS
rv = sanlock_acquire(sanlockfd, pid, 0, 1, &res, 0);
commit 3e1795789a87625bbfbf02e1e83d8bf2db85a51e
Author: Federico Simoncelli <fsimonce(a)redhat.com>
Date: Wed Apr 10 12:09:43 2013 -0400
python: use the write commands in example.py
The init_lockspace/resource commands have been obsoleted, the new ones
are write_lockspace/resource.
Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com>
diff --git a/python/example.py b/python/example.py
index ef1c957..28b5180 100644
--- a/python/example.py
+++ b/python/example.py
@@ -19,10 +19,10 @@ def main():
fd = sanlock.register()
print "Initializing '%s'" % (LOCKSPACE_NAME,)
- sanlock.init_lockspace(LOCKSPACE_NAME, disk)
+ sanlock.write_lockspace(LOCKSPACE_NAME, disk)
print "Initializing '%s' on '%s'" % (RESOURCE_NAME, LOCKSPACE_NAME)
- sanlock.init_resource(LOCKSPACE_NAME, RESOURCE_NAME, SNLK_DISKS)
+ sanlock.write_resource(LOCKSPACE_NAME, RESOURCE_NAME, SNLK_DISKS)
print "Acquiring the id '%i' on '%s'" % (HOST_ID, LOCKSPACE_NAME)
sanlock.add_lockspace(LOCKSPACE_NAME, HOST_ID, disk)
11 years
[PATCH 1/6] sanlock: rename the request actions
by Federico Simoncelli
For consistency the request actions are renamed to:
SANLK_REQ_SIGKILL: sigkill the process holding the resource lease.
SANLK_REQ_RELEASE: gracefully release the lease sending SIGUSR1 to
the process holding the resource lease.
Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com>
---
src/resource.c | 4 ++--
src/sanlock.8 | 4 ++--
src/sanlock_resource.h | 14 ++++++++++----
3 files changed, 14 insertions(+), 8 deletions(-)
diff --git a/src/resource.c b/src/resource.c
index 2a60a16..6c02107 100644
--- a/src/resource.c
+++ b/src/resource.c
@@ -856,11 +856,11 @@ static void do_request(struct token *tt, int pid, uint32_t force_mode)
memset(&hm, 0, sizeof(hm));
- if (force_mode == SANLK_REQ_KILL_PID) {
+ if (force_mode == SANLK_REQ_SIGKILL) {
hm.type = HELPER_MSG_KILLPID;
hm.pid = pid;
hm.sig = (flags & R_RESTRICT_SIGKILL) ? SIGTERM : SIGKILL;
- } else if (force_mode == SANLK_REQ_SIGUSR1) {
+ } else if (force_mode == SANLK_REQ_RELEASE) {
hm.type = HELPER_MSG_KILLPID;
hm.pid = pid;
hm.sig = SIGUSR1;
diff --git a/src/sanlock.8 b/src/sanlock.8
index a23e54a..ea188f6 100644
--- a/src/sanlock.8
+++ b/src/sanlock.8
@@ -515,12 +515,12 @@ set in A's bitmap for request_finish_seconds.)
\fIforce_mode\fP determines the action the resource lease owner should
take:
-\fB1\fP (KILL_PID): kill the process holding the resource lease. When the
+\fB1\fP (SIGKILL): kill the process holding the resource lease. When the
process has exited, the resource lease will be released, and can then be
acquired by anyone. The kill signal is SIGKILL (or SIGTERM if SIGKILL
is restricted.)
-\fB2\fP (SIGUSR1): send SIGUSR1 to the process holding the resource lease.
+\fB2\fP (RELEASE): send SIGUSR1 to the process holding the resource lease.
\fB3\fP (KILLPATH): run the program configured by sanlock_killpath
against the process holding the resource lease.
diff --git a/src/sanlock_resource.h b/src/sanlock_resource.h
index f68fc3e..4969b65 100644
--- a/src/sanlock_resource.h
+++ b/src/sanlock_resource.h
@@ -33,15 +33,21 @@
/*
* request force_mode
- * SANLK_REQ_KILL_PID: send SIGKILL to pid holding the resource, or
- * SIGTERM if SIGKILL is restricted
- * SANLK_REQ_SIGUSR1: send SIGUSR1 to pid holding the resource
+ * SANLK_REQ_SIGKILL: send SIGKILL to pid holding the resource, or
+ * SIGTERM if SIGKILL is restricted
+ * SANLK_REQ_RELEASE: send SIGUSR1 to pid holding the resource
* SANLK_REQ_KILLPATH: run configured killpath against the pid
*/
+#define SANLK_REQ_SIGKILL 0x00000001
+#define SANLK_REQ_RELEASE 0x00000002
+#define SANLK_REQ_KILLPATH 0x00000003
+
+/*
+ * kept for backward compatibility
+ */
#define SANLK_REQ_KILL_PID 0x00000001
#define SANLK_REQ_SIGUSR1 0x00000002
-#define SANLK_REQ_KILLPATH 0x00000003
int sanlock_register(void);
--
1.7.1
11 years
src/cmd.c src/resource.c src/resource.h src/sanlock.8 src/sanlock_internal.h src/sanlock_resource.h
by David Teigland
src/cmd.c | 7 ++++++-
src/resource.c | 35 +++++++++++++++++++++--------------
src/resource.h | 3 ++-
src/sanlock.8 | 3 +++
src/sanlock_internal.h | 2 ++
src/sanlock_resource.h | 2 ++
6 files changed, 36 insertions(+), 16 deletions(-)
New commits:
commit 0ea5e61b9e73459a75ccabfc248359405b2a6b3f
Author: David Teigland <teigland(a)redhat.com>
Date: Tue Apr 9 14:41:25 2013 -0500
sanlock: add request killpath
sanlock_request with force_mode 3 "REQ_KILLPATH"
will cause killpath to be run against the pid.
Signed-off-by: David Teigland <teigland(a)redhat.com>
diff --git a/src/cmd.c b/src/cmd.c
index 6f9310f..19416be 100644
--- a/src/cmd.c
+++ b/src/cmd.c
@@ -126,6 +126,8 @@ static void cmd_acquire(struct task *task, struct cmd_args *ca)
struct sanlk_resource res;
struct sanlk_options opt;
struct space_info spi;
+ char killpath[SANLK_HELPER_PATH_LEN];
+ char killargs[SANLK_HELPER_ARGS_LEN];
char *opt_str;
int token_len, disks_len;
int fd, rv, i, j, empty_slots, lvl;
@@ -165,6 +167,9 @@ static void cmd_acquire(struct task *task, struct cmd_args *ca)
if (!cl->tokens[i])
empty_slots++;
}
+
+ memcpy(killpath, cl->killpath, SANLK_HELPER_PATH_LEN);
+ memcpy(killargs, cl->killargs, SANLK_HELPER_ARGS_LEN);
pthread_mutex_unlock(&cl->mutex);
if (empty_slots < new_tokens_count) {
@@ -320,7 +325,7 @@ static void cmd_acquire(struct task *task, struct cmd_args *ca)
for (i = 0; i < new_tokens_count; i++) {
token = new_tokens[i];
- rv = acquire_token(task, token);
+ rv = acquire_token(task, token, killpath, killargs);
if (rv < 0) {
switch (rv) {
case -EEXIST:
diff --git a/src/resource.c b/src/resource.c
index 1aa7d6e..2a60a16 100644
--- a/src/resource.c
+++ b/src/resource.c
@@ -569,7 +569,8 @@ static struct resource *new_resource(struct token *token)
return r;
}
-int acquire_token(struct task *task, struct token *token)
+int acquire_token(struct task *task, struct token *token,
+ char *killpath, char *killargs)
{
struct leader_record leader;
struct resource *r;
@@ -626,6 +627,8 @@ int acquire_token(struct task *task, struct token *token)
return -ENOMEM;
}
+ memcpy(r->killpath, killpath, SANLK_HELPER_PATH_LEN);
+ memcpy(r->killargs, killargs, SANLK_HELPER_ARGS_LEN);
list_add(&token->list, &r->tokens);
list_add(&r->list, &resources_add);
token->resource = r;
@@ -820,6 +823,8 @@ static int examine_token(struct task *task, struct token *token,
static void do_request(struct token *tt, int pid, uint32_t force_mode)
{
+ char killpath[SANLK_HELPER_PATH_LEN];
+ char killargs[SANLK_HELPER_ARGS_LEN];
struct helper_msg hm;
struct resource *r;
uint32_t flags;
@@ -830,6 +835,8 @@ static void do_request(struct token *tt, int pid, uint32_t force_mode)
if (r && r->pid == pid) {
found = 1;
flags = r->flags;
+ memcpy(killpath, r->killpath, SANLK_HELPER_PATH_LEN);
+ memcpy(killargs, r->killargs, SANLK_HELPER_ARGS_LEN);
}
pthread_mutex_unlock(&resource_mutex);
@@ -857,24 +864,24 @@ static void do_request(struct token *tt, int pid, uint32_t force_mode)
hm.type = HELPER_MSG_KILLPID;
hm.pid = pid;
hm.sig = SIGUSR1;
+ } else if (force_mode == SANLK_REQ_KILLPATH) {
+ if (!killpath[0]) {
+ log_error("do_request %d force_mode %d no killpath",
+ pid, force_mode);
+ return;
+ }
+ /* there's no thread locking for the clients array so
+ we can't go searching for killpath/killargs for this
+ pid from here, so we copy the info into struct
+ resource so we can use it here. */
+ hm.type = HELPER_MSG_RUNPATH;
+ memcpy(hm.path, killpath, SANLK_HELPER_PATH_LEN);
+ memcpy(hm.args, killargs, SANLK_HELPER_ARGS_LEN);
} else {
log_error("do_request %d unknown force_mode %d",
pid, force_mode);
return;
}
-#if 0
- /* TODO: this is difficult because we can't dig into the clients
- array to get the killpath/killargs; the clients array is not
- locked and can only be accessed by the main thread. */
-
- else if (force_mode == SANLK_REQ_KILLPATH) {
- hm.type = HELPER_MSG_RUNPATH;
- memcpy(hm.path, cl->killpath, SANLK_HELPER_PATH_LEN);
- memcpy(hm.args, cl->killargs, SANLK_HELPER_ARGS_LEN);
- if (cl->flags & CL_KILLPATH_PID)
- hm.pid = pid;
- }
-#endif
retry:
rv = write(helper_kill_fd, &hm, sizeof(hm));
diff --git a/src/resource.h b/src/resource.h
index a75f1a1..b3e99a2 100644
--- a/src/resource.h
+++ b/src/resource.h
@@ -15,7 +15,8 @@ int lockspace_is_used(struct sanlk_lockspace *ls);
void check_mode_block(struct token *token, int q, char *dblock);
-int acquire_token(struct task *task, struct token *token);
+int acquire_token(struct task *task, struct token *token,
+ char *killpath, char *killargs);
int release_token(struct task *task, struct token *token);
void release_token_async(struct token *token);
diff --git a/src/sanlock.8 b/src/sanlock.8
index dca5db8..a23e54a 100644
--- a/src/sanlock.8
+++ b/src/sanlock.8
@@ -522,6 +522,9 @@ is restricted.)
\fB2\fP (SIGUSR1): send SIGUSR1 to the process holding the resource lease.
+\fB3\fP (KILLPATH): run the program configured by sanlock_killpath
+against the process holding the resource lease.
+
.SS Graceful recovery
When a lockspace host_id cannot be renewed for a specific period of time,
diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h
index 6deef0e..34ce260 100644
--- a/src/sanlock_internal.h
+++ b/src/sanlock_internal.h
@@ -109,6 +109,8 @@ struct resource {
int pid; /* copied from token when ex */
uint32_t flags;
uint32_t release_token_id; /* copy to temp token (tt) for log messages */
+ char killpath[SANLK_HELPER_PATH_LEN]; /* copied from client */
+ char killargs[SANLK_HELPER_ARGS_LEN]; /* copied from client */
struct leader_record leader; /* copy of last leader_record we wrote */
struct sanlk_resource r;
};
diff --git a/src/sanlock_resource.h b/src/sanlock_resource.h
index cf2a798..f68fc3e 100644
--- a/src/sanlock_resource.h
+++ b/src/sanlock_resource.h
@@ -36,10 +36,12 @@
* SANLK_REQ_KILL_PID: send SIGKILL to pid holding the resource, or
* SIGTERM if SIGKILL is restricted
* SANLK_REQ_SIGUSR1: send SIGUSR1 to pid holding the resource
+ * SANLK_REQ_KILLPATH: run configured killpath against the pid
*/
#define SANLK_REQ_KILL_PID 0x00000001
#define SANLK_REQ_SIGUSR1 0x00000002
+#define SANLK_REQ_KILLPATH 0x00000003
int sanlock_register(void);
11 years