Changes to 'refs/tags/1.2.0'
by Chris Feist
Changes since the dawn of time:
Daniel P. Berrange (15):
Fix const-ness of many APIs.
Fix warnings in watchdog module
Fix function prototypes for no-arg methods
Remove use of 'index' as a variable name
Make many functions static
Fix missing include in logging file
Annotate many unused parameters to avoid warnings
Remove redundant redeclaration of 'to' variable
Fix args to execv()
Remove redundant arg to 'run_command'
Rename optarg to optionarg to avoid clashing with getopt.h
Disable the read_request method since it is unused
Add many more compiler warning flags & safety checks
Hard code a sector size of 512 if the lease volume is a regular file
Ensure libsanlock.so is built with debug/warning flags
David Teigland (185):
sync_manager: initial commit
sync_manager: misc updates
sync_manager: misc updates
sync_manager: misc updates
sync_manager: misc updates
sync_manager: add more logging
sync_manager: misc updates
sync_manager: misc updates
sync_manager: num_hosts/MAX_HOSTS stuff
daemon: reworking notions of resource_id/token_name
sync_manager: resource lockfiles
sync_manager: lease arg processing
sync_manager: Began multiple lease support
sync_manager: use first command line arg as action
sync_manager: leader record changes and verify
sync_manager: clean up released leases
sync_manager: move functions around
sync_manager: add more tool actions
sync_manager: naming changes
sync_manager: separate token index and id
sync_manager: fix index usage and other misc
sync_manager: use pthread cond in acquire
sync_manager: write all log entries
sync_manager: simplify polling
sync_manager: fix waitpid use
sync_manager: acquire can fail early
sync_manager: write log entries at exit
sync_manager: add test program
sync_manager: move secondary pid check
sync_manager: fix disk paxos contention
devcount: fix verify checks
sync_manager: add GPL license file
sync_manager: fix leader block offsets
sync_manager: increase COMMAND_MAX
sync_manager: renewal should verify
sync_manager: use sector size from libblkid
sync_manager: use a real checksum function
sync_manager: add libblkid to spec file
sync_manager: print status info
sync_manager: one watchdog file per lease
sync_manager: lease_threads handle watchdog files
sync_manager: fix/add some text/comments
sync_manager: refactor read/write
sync_manager: move disk io functions
sync_manager: remove token arg
sync_manager: rename paxos_disk sync_disk
sync_manager: add aio read/write
sync_manager: make io_timeout_seconds a diskio arg
sync_manager: forgot to add new files
sync_manager: use log thread
sync_manager: client connections
sync_manager: connection processing
sync_manager: send/recv pid
sync_manager: add write_sectors
sync_manager: restructuring
sync_manager: write_sectors code factoring
sync_manager: daemonize
sync_manager: releasing leases
sync_manager: async releasing
sync_manager: release fixes
sync_manager: add direct and indirect acquire/release
sync_manager: reacquire resources
sync_manager: move code
sync_manager: same pid may reacquire resource
sync_manager: lease migration
sync_manager: handle client errors
sync_manager: improve error handling
sync_manager: host_id leases
sync_manager: remove empty files
sync_manager: print initialization info
sync_manager: rename files
sync_manager: clean up header org
sync_manager: delta_lease implementation
sync_manager: accept offset units
sync_manager: fix up init output
sync_manager: put back watchdog calls
sync_manager: fix start_host_id error paths
sync_manager: add log_error's for watchdog file errors
sync_manager: actual timeouts
sync_manager: change timeouts on cmd line
sanlock: create new external api
sanlock: build libsanlock
sanlock: use MAX_LEASES everywhere
sanlock: add libvirt plugin
sanlock plugin: couple minor fixes
sanlock: clean up /var file names
sanlock plugin: fix symbol needed by libvirt
sanlock: add some debug output
sanlock plugin: fix uuid copy
sanlock plugin: fix names
sanlock: add "owner_name"
sanlock: fix renewal checks
sanlock: clean up host_id types
sanlock: set_host_id command
sanlock: fix killing pids
sanlock: add status command
sanlock: set version to 1.0
sanlock: delta_lease cleanup
sanlock: changing num_hosts
sanlock: add dump command
sanlock: renewal timings
sanlock: add direct option
sanlock: check for watchdog file
sanlock: recovery fixes
lock_driver_sanlock: fix compile problems
sanlock: improve command options
sanlock: tidying help text
sanlock: move binary to /usr/sbin
sanlock: add init script
sanlock: fix sigterm shutdown
sanlock: init stop
sanlock: add wdtest command
sanlock.spec: new url
lock_driver_sanlock: remove close
sanlock: introduce lockspaces
lock_driver_sanlock: remove files
sanlock: better logging functions
sanlock: misc log message
sanlock.spec: sbin not libexec
sanlock init: remove watchdog reference
wdmd: watchdog multiplexing daemon
sanlock: add code to use wdmd
sanlock/wdmd: use wdmd in sanlock
sanlock/wdmd: add copyright header to source files
sanlock: rename sanlock source dir
sanlock: move tests dir
move COPYING file
wdmd: use signalfd for signal handling
Fix Makefile comments
wdmd: fix daemon debug option
wdmd: add init script
sanlock.spec: updates
sanlock.spec: src dir
sanlock: build with uninstalled libwdmd
sanlock: version 1.1
sanlock: high priority options
wdmd: high priority options
sanlock: return migration state
sanlock: migration.txt describes libvirt/sanlock steps
libsanlock: include admin functions
sanlock: fix host_id expiration check
sanlock: migration working
devcount: migrate test
sanlock: setowner improvements
sanlock: migrate to target fix
sanlock: fix wdmd stop order
sanlock: various fixes
sanlock: remove wdtest
sanlock: remove migration
sanlock: clean up command return data
sanlock: add resource string conversion functions
sanlock: rework internal structs
devcount: add relock test
sanlock: fix release and inquire
sanlock: add_lockspace EEXIST
sanlock: rework client handling
sanlock: clean up warnings
sanlock: debug message changes
sanlock: add lockspace checks
wdmd: enable test scripts
sanlock: add str_to_lockspace to lib
WIP devcount migrate
devcount: new migrate test
sanlock: read_id and live_id commands
sanlock: check lockspace name and host_id
sanlock: remove remaining cluster_mode
sanlock: add libsanlock_direct
devcountn: start multiple devcount tests
devcount: small changes
sanlock: new return values
sanlock: misc changes and fix
sanlock: log error of full bad block
sanlock: interval between renewal checks
sanlock: renewal changes
sanlock: fix log_dump
sanlock: fix find_client_pid
sanlock: fix host_id reads from paxos_acquire
sanlock: init with one write
devcount: improve output
devcount: new pause/resume
devcount: add expire test
sanlock: correct paxos usage
sanlock: direct read_leader
sanlock: paxos delays
sanlock: use thread pool
Fabio M. Di Nitto (5):
build: sanlock should link with libsanlock
build: install shared lib header files and fix DESTDIR usage
build: drop rpm target
spec file: do first cut for total spec file
build: fix linking with libsanlock and install target
Federico Simoncelli (17):
rpm: sync specfile with fedora
rpm: add sanlock_admin.h header
rpm: add the lib package, install the initscripts
python: remove unused python code
python: add python binding
python: release the gil during sanlock operations
python: wrap sanlock extension with a module
rpm: add python binding package
python: pass a lockspace copy to str_to_lockspace
makefile: fix install typo
rpm: add sanlock_direct header
python: add sanlock init functions and exception
direct: close disks after initialization
python: register process only once
daemon: configurable socket permissions
rpm: add sanlock user and group
python: exceptions must contain the errno
Saggi Mizrahi (21):
Added the begining of the testing and debugging tools
Better handling of max hosts
sync_manager: Updated tests to work with new lease struct
sync_manager: fixed skipping first arg in command
sync_manager: acquire and release actions
sync_manager: minor fixes
sync_manager: renamed stuff
sync_manager: made acquire synchronous again
sync_manager: added set_host_id action
sync_manager: use kill(0) for secondary pid check
sync_manager: make rpm and install
sync_manager: spec file update
sync_manager: Allow longer resource names
sync_manager: allow repeated set_host_id
sync_manager: Added escaping for the leases arg
sync_manager: Created the python bindings for sync_manager
sync_manager: listener socket permissions
sync_manager: Updated python binding and tests
sync_manager: Made 'token' a const in log_level
sync_manager: refactor messaging system
sync_manager: use getsockopt PEERCRED
12 years, 11 months
python/sanlockmod.c
by David Teigland
python/sanlockmod.c | 51 ++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 38 insertions(+), 13 deletions(-)
New commits:
commit 16bf11f3f33148c05a89db8bb5e263fb5e70b817
Author: Federico Simoncelli <fsimonce(a)redhat.com>
Date: Mon May 9 16:41:00 2011 +0100
python: exceptions must contain the errno
diff --git a/python/sanlockmod.c b/python/sanlockmod.c
index 9d214f5..a042030 100644
--- a/python/sanlockmod.c
+++ b/python/sanlockmod.c
@@ -12,28 +12,48 @@
#include <sanlock_admin.h>
#include <sanlock_direct.h>
+#define SKERRNO(x) (-x)
+
int __sanlockmod_fd = -1;
PyObject *py_module;
/* SANLock exception */
static PyObject *sanlockmod_exception;
+static void
+__set_exception(int en, char *msg)
+{
+ PyObject *exc_tuple;
+
+ exc_tuple = Py_BuildValue("(is)", en, msg);
+
+ if (exc_tuple == NULL) {
+ PyErr_NoMemory();
+ } else {
+ PyErr_SetObject(sanlockmod_exception, exc_tuple);
+ Py_DECREF(exc_tuple);
+ }
+}
+
static int
__parse_lockspace(char *lockspace, struct sanlk_lockspace *ret_ls)
{
+ int rv;
char *lockspace_arg;
/* sanlock_str_to_lockspace is destructive */
lockspace_arg = strdup(lockspace);
if (lockspace_arg == NULL) {
- PyErr_SetString(sanlockmod_exception, "SANLock extension memory error");
+ PyErr_NoMemory();
return -1;
}
/* convert lockspace string to structure */
- if (sanlock_str_to_lockspace(lockspace_arg, ret_ls) != 0) {
- PyErr_SetString(sanlockmod_exception, "Invalid SANLock lockspace");
+ rv = sanlock_str_to_lockspace(lockspace_arg, ret_ls);
+
+ if (rv != 0) {
+ __set_exception(SKERRNO(rv), "Invalid SANLock lockspace");
goto exit_fail;
}
@@ -48,9 +68,13 @@ exit_fail:
static int
__parse_resource(char *resource, struct sanlk_resource **ret_res)
{
+ int rv;
+
/* convert resource string to structure */
- if (sanlock_str_to_res(resource, ret_res) != 0) {
- PyErr_SetString(sanlockmod_exception, "Invalid SANLock resource");
+ rv = sanlock_str_to_res(resource, ret_res);
+
+ if (rv != 0) {
+ __set_exception(SKERRNO(rv), "Invalid SANLock resource");
return -1;
}
@@ -65,7 +89,8 @@ py_register(PyObject *self, PyObject *args)
}
if (__sanlockmod_fd < 0) {
- PyErr_SetString(sanlockmod_exception, "SANLock registration failed");
+ __set_exception(SKERRNO(__sanlockmod_fd),
+ "SANLock registration failed");
return NULL;
}
@@ -95,7 +120,7 @@ static PyObject *py_init_lockspace(PyObject *self, PyObject *args)
Py_END_ALLOW_THREADS
if (rv != 0) {
- PyErr_SetString(sanlockmod_exception, "SANLock lockspace init failure");
+ __set_exception(SKERRNO(rv), "SANLock lockspace init failure");
return NULL;
}
@@ -125,7 +150,7 @@ static PyObject *py_init_resource(PyObject *self, PyObject *args)
Py_END_ALLOW_THREADS
if (rv != 0) {
- PyErr_SetString(sanlockmod_exception, "SANLock resource init failure");
+ __set_exception(SKERRNO(rv), "SANLock resource init failure");
return NULL;
}
@@ -155,7 +180,7 @@ py_add_lockspace(PyObject *self, PyObject *args)
Py_END_ALLOW_THREADS
if (rv != 0) {
- PyErr_SetString(sanlockmod_exception, "SANLock lockspace add failure");
+ __set_exception(SKERRNO(rv), "SANLock lockspace add failure");
return NULL;
}
@@ -185,7 +210,7 @@ py_rem_lockspace(PyObject *self, PyObject *args)
Py_END_ALLOW_THREADS
if (rv != 0) {
- PyErr_SetString(sanlockmod_exception, "SANLock lockspace remove failure");
+ __set_exception(SKERRNO(rv), "SANLock lockspace remove failure");
return NULL;
}
@@ -215,7 +240,7 @@ py_acquire(PyObject *self, PyObject *args)
Py_END_ALLOW_THREADS
if (rv != 0) {
- PyErr_SetString(sanlockmod_exception, "SANLock resource not acquired");
+ __set_exception(SKERRNO(rv), "SANLock resource not acquired");
goto exit_fail;
}
@@ -244,13 +269,13 @@ py_release(PyObject *self, PyObject *args)
return NULL;
}
- /* release sanlock resource (gil disabled)*/
+ /* release sanlock resource (gil disabled) */
Py_BEGIN_ALLOW_THREADS
rv = sanlock_release(__sanlockmod_fd, -1, 0, 1, &res);
Py_END_ALLOW_THREADS
if (rv != 0) {
- PyErr_SetString(sanlockmod_exception, "SANLock resource not released");
+ __set_exception(SKERRNO(rv), "SANLock resource not released");
goto exit_fail;
}
12 years, 11 months
src/main.c src/sanlock_internal.h
by David Teigland
src/main.c | 233 ++++++++++++++++++++++++++++++++++---------------
src/sanlock_internal.h | 3
2 files changed, 167 insertions(+), 69 deletions(-)
New commits:
commit b18e2dc9bd7e2e5e6e141073835cb119332fdcc8
Author: David Teigland <teigland(a)redhat.com>
Date: Mon May 9 13:28:38 2011 -0500
sanlock: use thread pool
instead of creating a new transient thread to handle each cmd.
The worker thread max also naturally prevents too much disk i/o
due to trying to acquire too many leases at once. Modeled after
www.redhat.com/archives/libvir-list/2010-November/msg00754.html
diff --git a/src/main.c b/src/main.c
index c9287d0..1c3a8e9 100644
--- a/src/main.c
+++ b/src/main.c
@@ -82,6 +82,7 @@ static int token_id_counter = 1;
static int space_id_counter = 1;
struct cmd_args {
+ struct list_head list; /* thread_pool data */
int ci_in;
int ci_target;
int cl_fd;
@@ -89,6 +90,19 @@ struct cmd_args {
struct sm_header header;
};
+struct thread_pool {
+ int num_workers;
+ int max_workers;
+ int free_workers;
+ int quit;
+ struct list_head work_data;
+ pthread_mutex_t mutex;
+ pthread_cond_t cond;
+ pthread_cond_t quit_wait;
+};
+
+static struct thread_pool pool;
+
extern struct list_head spaces;
extern struct list_head spaces_remove;
extern pthread_mutex_t spaces_mutex;
@@ -698,9 +712,8 @@ static int check_new_tokens_space(struct client *cl,
return 0;
}
-static void *cmd_acquire_thread(void *args_in)
+static void cmd_acquire(struct cmd_args *ca)
{
- struct cmd_args *ca = args_in;
struct sm_header h;
struct client *cl;
struct token *token = NULL;
@@ -1061,13 +1074,10 @@ static void *cmd_acquire_thread(void *args_in)
client_recv_all(ca->ci_in, &ca->header, pos);
client_resume(ca->ci_in);
- free(ca);
- return NULL;
}
-static void *cmd_release_thread(void *args_in)
+static void cmd_release(struct cmd_args *ca)
{
- struct cmd_args *ca = args_in;
struct sm_header h;
struct client *cl;
struct token *token;
@@ -1175,13 +1185,10 @@ static void *cmd_release_thread(void *args_in)
send(fd, &h, sizeof(h), MSG_NOSIGNAL);
client_resume(ca->ci_in);
- free(ca);
- return NULL;
}
-static void *cmd_inquire_thread(void *args_in)
+static void cmd_inquire(struct cmd_args *ca)
{
- struct cmd_args *ca = args_in;
struct sm_header h;
struct token *token;
struct client *cl;
@@ -1298,13 +1305,10 @@ static void *cmd_inquire_thread(void *args_in)
}
client_resume(ca->ci_in);
- free(ca);
- return NULL;
}
-static void *cmd_add_lockspace_thread(void *args_in)
+static void cmd_add_lockspace(struct cmd_args *ca)
{
- struct cmd_args *ca = args_in;
struct sm_header h;
struct space *sp;
struct sanlk_lockspace lockspace;
@@ -1363,13 +1367,10 @@ static void *cmd_add_lockspace_thread(void *args_in)
send(fd, &h, sizeof(h), MSG_NOSIGNAL);
client_resume(ca->ci_in);
- free(ca);
- return NULL;
}
-static void *cmd_rem_lockspace_thread(void *args_in)
+static void cmd_rem_lockspace(struct cmd_args *ca)
{
- struct cmd_args *ca = args_in;
struct sm_header h;
struct sanlk_lockspace lockspace;
int fd, rv, result;
@@ -1419,10 +1420,130 @@ static void *cmd_rem_lockspace_thread(void *args_in)
send(fd, &h, sizeof(h), MSG_NOSIGNAL);
client_resume(ca->ci_in);
- free(ca);
+}
+
+static void call_cmd(struct cmd_args *ca)
+{
+ switch (ca->header.cmd) {
+ case SM_CMD_ACQUIRE:
+ cmd_acquire(ca);
+ break;
+ case SM_CMD_RELEASE:
+ cmd_release(ca);
+ break;
+ case SM_CMD_INQUIRE:
+ cmd_inquire(ca);
+ break;
+ case SM_CMD_ADD_LOCKSPACE:
+ strcpy(client[ca->ci_in].owner_name, "add_lockspace");
+ cmd_add_lockspace(ca);
+ break;
+ case SM_CMD_REM_LOCKSPACE:
+ strcpy(client[ca->ci_in].owner_name, "rem_lockspace");
+ cmd_rem_lockspace(ca);
+ break;
+ };
+}
+
+static void *thread_pool_worker(void *data GNUC_UNUSED)
+{
+ struct cmd_args *ca;
+
+ pthread_mutex_lock(&pool.mutex);
+
+ while (1) {
+ while (!pool.quit && list_empty(&pool.work_data)) {
+ pool.free_workers++;
+ pthread_cond_wait(&pool.cond, &pool.mutex);
+ pool.free_workers--;
+ }
+
+ while (!list_empty(&pool.work_data)) {
+ ca = list_first_entry(&pool.work_data, struct cmd_args, list);
+ list_del(&ca->list);
+ pthread_mutex_unlock(&pool.mutex);
+
+ call_cmd(ca);
+ free(ca);
+
+ pthread_mutex_lock(&pool.mutex);
+ }
+
+ if (pool.quit)
+ break;
+ }
+
+ pool.num_workers--;
+ if (!pool.num_workers)
+ pthread_cond_signal(&pool.quit_wait);
+ pthread_mutex_unlock(&pool.mutex);
return NULL;
}
+static int thread_pool_add_work(struct cmd_args *ca)
+{
+ pthread_t th;
+ int rv;
+
+ pthread_mutex_lock(&pool.mutex);
+ if (pool.quit) {
+ pthread_mutex_unlock(&pool.mutex);
+ return -1;
+ }
+
+ list_add_tail(&ca->list, &pool.work_data);
+
+ if (!pool.free_workers && pool.num_workers < pool.max_workers) {
+ rv = pthread_create(&th, NULL, thread_pool_worker, &pool);
+ if (rv < 0) {
+ list_del(&ca->list);
+ pthread_mutex_unlock(&pool.mutex);
+ return rv;
+ }
+ pool.num_workers++;
+ }
+
+ pthread_cond_signal(&pool.cond);
+ pthread_mutex_unlock(&pool.mutex);
+ return 0;
+}
+
+static void thread_pool_free(void)
+{
+ pthread_mutex_lock(&pool.mutex);
+ pool.quit = 1;
+ if (pool.num_workers > 0) {
+ pthread_cond_broadcast(&pool.cond);
+ pthread_cond_wait(&pool.quit_wait, &pool.mutex);
+ }
+ pthread_mutex_unlock(&pool.mutex);
+}
+
+static int thread_pool_create(int min_workers, int max_workers)
+{
+ pthread_t th;
+ int i, rv;
+
+ memset(&pool, 0, sizeof(pool));
+ INIT_LIST_HEAD(&pool.work_data);
+ pthread_mutex_init(&pool.mutex, NULL);
+ pthread_cond_init(&pool.cond, NULL);
+ pthread_cond_init(&pool.quit_wait, NULL);
+ pool.max_workers = max_workers;
+
+ for (i = 0; i < min_workers; i++) {
+ rv = pthread_create(&th, NULL, thread_pool_worker, &pool);
+ if (rv < 0)
+ break;
+ pool.num_workers++;
+ }
+
+ if (rv < 0)
+ thread_pool_free();
+
+ return rv;
+}
+
static int print_daemon_state(char *str)
{
memset(str, 0, SANLK_STATE_MAXSTR);
@@ -1637,8 +1758,6 @@ static void cmd_log_dump(int fd, struct sm_header *h_recv)
static void process_cmd_thread_lockspace(int ci_in, struct sm_header *h_recv)
{
- pthread_t cmd_thread;
- pthread_attr_t attr;
struct cmd_args *ca;
struct sm_header h;
int rv;
@@ -1651,26 +1770,16 @@ static void process_cmd_thread_lockspace(int ci_in, struct sm_header *h_recv)
ca->ci_in = ci_in;
memcpy(&ca->header, h_recv, sizeof(struct sm_header));
- pthread_attr_init(&attr);
- pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
-
- switch (h_recv->cmd) {
- case SM_CMD_ADD_LOCKSPACE:
+ if (h_recv->cmd == SM_CMD_ADD_LOCKSPACE)
strcpy(client[ci_in].owner_name, "add_lockspace");
- rv = pthread_create(&cmd_thread, &attr, cmd_add_lockspace_thread, ca);
- break;
- case SM_CMD_REM_LOCKSPACE:
+ else if (h_recv->cmd == SM_CMD_REM_LOCKSPACE)
strcpy(client[ci_in].owner_name, "rem_lockspace");
- rv = pthread_create(&cmd_thread, &attr, cmd_rem_lockspace_thread, ca);
- break;
- };
+ else
+ strcpy(client[ci_in].owner_name, "cmd_lockspace");
- pthread_attr_destroy(&attr);
- if (rv < 0) {
- log_error("create cmd thread failed");
+ rv = thread_pool_add_work(ca);
+ if (rv < 0)
goto fail_free;
- }
-
return;
fail_free:
@@ -1686,8 +1795,6 @@ static void process_cmd_thread_lockspace(int ci_in, struct sm_header *h_recv)
static void process_cmd_thread_resource(int ci_in, struct sm_header *h_recv)
{
- pthread_t cmd_thread;
- pthread_attr_t attr;
struct cmd_args *ca;
struct sm_header h;
struct client *cl;
@@ -1787,25 +1894,7 @@ static void process_cmd_thread_resource(int ci_in, struct sm_header *h_recv)
ca->cl_fd = cl->fd;
memcpy(&ca->header, h_recv, sizeof(struct sm_header));
- /* TODO: use a thread pool? */
-
- pthread_attr_init(&attr);
- pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
-
- switch (h_recv->cmd) {
- case SM_CMD_ACQUIRE:
- rv = pthread_create(&cmd_thread, &attr, cmd_acquire_thread, ca);
- break;
- case SM_CMD_RELEASE:
- rv = pthread_create(&cmd_thread, &attr, cmd_release_thread, ca);
- break;
- case SM_CMD_INQUIRE:
- rv = pthread_create(&cmd_thread, &attr, cmd_inquire_thread, ca);
- break;
- };
-
- pthread_attr_destroy(&attr);
-
+ rv = thread_pool_add_work(ca);
if (rv < 0) {
/* we don't have to worry about client_pid_dead having
been called while mutex was unlocked with cmd_active set,
@@ -2037,16 +2126,9 @@ static int do_daemon(void)
if (rv < 0)
return rv;
- /*
- * after creating dirs and setting up logging the daemon can
- * use log_error/log_debug
- */
-
rv = make_dirs();
- if (rv < 0) {
- log_tool("cannot create logging dirs\n");
+ if (rv < 0)
return rv;
- }
setup_logging();
@@ -2056,17 +2138,21 @@ static int do_daemon(void)
if (fd < 0)
goto out;
- rv = setup_watchdog();
+ rv = thread_pool_create(DEFAULT_MIN_WORKER_THREADS, com.max_worker_threads);
if (rv < 0)
goto out_lockfile;
+ rv = setup_watchdog();
+ if (rv < 0)
+ goto out_threads;
+
rv = setup_listener();
if (rv < 0)
- goto out_lockfile;
+ goto out_threads;
setup_token_manager();
if (rv < 0)
- goto out_lockfile;
+ goto out_threads;
setup_spaces();
@@ -2076,6 +2162,8 @@ static int do_daemon(void)
close_watchdog();
+ out_threads:
+ thread_pool_free();
out_lockfile:
unlink_lockfile(fd, SANLK_RUN_DIR, SANLK_LOCKFILE_NAME);
out:
@@ -2283,6 +2371,7 @@ static void print_usage(void)
printf(" -Q <num> quiet error messages for common lock contention\n");
printf(" -L <level> write logging at level and up to logfile (-1 none)\n");
printf(" -S <level> write logging at level and up to syslog (-1 none)\n");
+ printf(" -t <num> max worker threads (default %d)\n", DEFAULT_MAX_WORKER_THREADS);
printf(" -w <num> use watchdog through wdmd (1 yes, 0 no, default %d)\n", DEFAULT_USE_WATCHDOG);
printf(" -a <num> use async io (1 yes, 0 no, default %d)\n", DEFAULT_USE_AIO);
printf(" -h <num> use high priority features (1 yes, 0 no, default %d)\n", DEFAULT_HIGH_PRIORITY);
@@ -2516,6 +2605,11 @@ static int read_command_line(int argc, char *argv[])
case 'a':
to.use_aio = atoi(optionarg);
break;
+ case 't':
+ com.max_worker_threads = atoi(optionarg);
+ if (com.max_worker_threads < DEFAULT_MIN_WORKER_THREADS)
+ com.max_worker_threads = DEFAULT_MIN_WORKER_THREADS;
+ break;
case 'w':
com.use_watchdog = atoi(optionarg);
break;
@@ -2858,6 +2952,7 @@ int main(int argc, char *argv[])
com.max_hosts = DEFAULT_MAX_HOSTS;
com.use_watchdog = DEFAULT_USE_WATCHDOG;
com.high_priority = DEFAULT_HIGH_PRIORITY;
+ com.max_worker_threads = DEFAULT_MAX_WORKER_THREADS;
com.uid = DEFAULT_SOCKET_UID;
com.gid = DEFAULT_SOCKET_GID;
com.pid = -1;
diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h
index 5ad996a..754153e 100644
--- a/src/sanlock_internal.h
+++ b/src/sanlock_internal.h
@@ -263,6 +263,8 @@ EXTERN struct timeout to;
#define DEFAULT_SOCKET_UID 0
#define DEFAULT_SOCKET_GID 0
#define DEFAULT_SOCKET_MODE (S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP)
+#define DEFAULT_MIN_WORKER_THREADS 2
+#define DEFAULT_MAX_WORKER_THREADS 8
struct command_line {
int type; /* COM_ */
@@ -272,6 +274,7 @@ struct command_line {
int quiet_fail;
int use_watchdog;
int high_priority;
+ int max_worker_threads;
int uid; /* -U */
int gid; /* -G */
int pid; /* -p */
12 years, 11 months
2 commits - src/delta_lease.c src/direct.c src/direct.h src/main.c src/paxos_lease.c src/sanlock_internal.h
by David Teigland
src/delta_lease.c | 4
src/direct.c | 241 +++++++++++++++++++++++++++----------------------
src/direct.h | 11 +-
src/main.c | 41 +++++++-
src/paxos_lease.c | 118 +++++++++++++++++------
src/sanlock_internal.h | 1
6 files changed, 267 insertions(+), 149 deletions(-)
New commits:
commit 7a63dd77ab564e2f34c54d2b53b44fae259c226a
Author: David Teigland <teigland(a)redhat.com>
Date: Thu May 5 16:39:58 2011 -0500
sanlock: paxos delays
Change two delays in the paxos code:
- sleep 1 instead of 2 between host_id liveness changes;
I was using 1 during the most successful test run, but
should go back and see if there's actually any difference
between using 1 and 2.
- sleep random time between 0 and 1 sec between ballot retries;
this should hopefully resolve the situation I saw where all
hosts were in fast ballot retry loops, always seeing larger
mbal values from other hosts (even if it would have resolved
itself eventually, it's not nice to hammer the system in a
loop like this for so long if we can avoid it.)
Also more comment, and log message changes.
diff --git a/src/paxos_lease.c b/src/paxos_lease.c
index 116823d..c93b4b3 100644
--- a/src/paxos_lease.c
+++ b/src/paxos_lease.c
@@ -183,6 +183,53 @@ static int read_request(struct timeout *ti,
}
#endif
+/*
+ * It's possible that we pick a bk_max from another host which has our own
+ * inp values in it, and we can end up commiting our own inp values, copied
+ * from another host's dblock:
+ *
+ * host2 leader free
+ * host2 phase1 mbal 14002
+ * host2 writes dblock[1] mbal 14002
+ * host2 reads no higher mbal
+ * host2 choose own inp 2,1
+ * host2 phase2 mbal 14002 bal 14002 inp 2,1
+ * host2 writes dblock[1] bal 14002 inp 2,1
+ * host1 leader free
+ * host1 phase1 mbal 20001
+ * host1 writes dblock[0] mbal 20001
+ * host1 reads no higher mbal
+ * host1 choose dblock[1] bal 14002 inp 2,1
+ * host1 phase2 mbal 20001 bal 20001 inp 2,1
+ * host1 writes dblock[0] bal 20001 inp 2,1
+ * host2 reads dblock[0] mbal 20001 > 14002
+ * abort2, retry
+ * host2 leader free
+ * host2 phase1 mbal 16002
+ * host2 writes dblock[1] mbal 16002
+ * host2 reads dblock[0] mbal 20001 > 16002
+ * abort1 retry
+ * host2 leader free
+ * host2 phase1 mbal 18002
+ * host2 writes dblock[1] mbal 18002
+ * host2 reads dblock[0] mbal 20001 > 18002
+ * abort1 retry
+ * host2 leader free
+ * host2 phase1 mbal 20002
+ * host2 writes dblock[1] mbal 20002
+ * host2 reads no higher mbal
+ * host2 choose dblock[0] bal 20001 inp 2,1
+ * host1 reads dblock[1] mbal 20002 > 20001
+ * abort2 retry
+ * host2 phase2 mbal 20002 bal 20002 inp 2,1
+ * host2 writes dblock[1] bal 20002 inp 2,1
+ * host2 reads no higher mbal
+ * host2 commit inp 2,1
+ * host2 success
+ * host1 leader owner 2,1
+ * host1 fail
+ */
+
static int run_ballot(struct timeout *ti, struct token *token, int num_hosts,
uint64_t next_lver, uint64_t our_mbal,
struct paxos_dblock *dblock_out)
@@ -245,7 +292,7 @@ static int run_ballot(struct timeout *ti, struct token *token, int num_hosts,
if (bk[q].lver > dblock.lver) {
/* I don't think this should happen */
- log_errot(token, "ballot %llu larger lver[%d] %llu",
+ log_errot(token, "ballot %llu larger1 lver[%d] %llu",
(unsigned long long)next_lver, q,
(unsigned long long)bk[q].lver);
return SANLK_DBLOCK_LVER;
@@ -254,7 +301,7 @@ static int run_ballot(struct timeout *ti, struct token *token, int num_hosts,
/* see "It aborts the ballot" in comment above */
if (bk[q].mbal > dblock.mbal) {
- log_errot(token, "ballot %llu mbal %llu larger mbal[%d] %llu",
+ log_errot(token, "ballot %llu abort1 mbal %llu mbal[%d] %llu",
(unsigned long long)next_lver,
(unsigned long long)our_mbal, q,
(unsigned long long)bk[q].mbal);
@@ -317,7 +364,7 @@ static int run_ballot(struct timeout *ti, struct token *token, int num_hosts,
if (bk_max.inp) {
/* not a problem, but interesting to see, so use log_error */
- log_errot(token, "ballot %llu bk_max[%d] lver %llu mbal %llu bal %llu inp %llu %llu %llu",
+ log_errot(token, "ballot %llu choose bk_max[%d] lver %llu mbal %llu bal %llu inp %llu %llu %llu",
(unsigned long long)next_lver, q_max,
(unsigned long long)bk_max.lver,
(unsigned long long)bk_max.mbal,
@@ -380,7 +427,7 @@ static int run_ballot(struct timeout *ti, struct token *token, int num_hosts,
/* see "It aborts the ballot" in comment above */
if (bk[q].mbal > dblock.mbal) {
- log_errot(token, "ballot %llu mbal %llu larger2 mbal[%d] %llu",
+ log_errot(token, "ballot %llu abort2 mbal %llu mbal[%d] %llu",
(unsigned long long)next_lver,
(unsigned long long)our_mbal, q,
(unsigned long long)bk[q].mbal);
@@ -589,7 +636,7 @@ int paxos_lease_leader_read(struct timeout *ti,
}
if (!majority_disks(token, num_reads)) {
- log_errot(token, "%s leader_read error %d", caller, rv);
+ log_errot(token, "%s leader read error %d", caller, rv);
error = SANLK_LEADER_READ;
goto fail;
}
@@ -611,12 +658,12 @@ int paxos_lease_leader_read(struct timeout *ti,
}
if (!found) {
- log_errot(token, "%s leader_read inconsistent", caller);
+ log_errot(token, "%s leader inconsistent", caller);
error = SANLK_LEADER_DIFF;
goto fail;
}
- log_token(token, "%s leader_read %llu owner %llu %llu %llu", caller,
+ log_token(token, "%s leader %llu owner %llu %llu %llu", caller,
(unsigned long long)leader.lver,
(unsigned long long)leader.owner_id,
(unsigned long long)leader.owner_generation,
@@ -632,6 +679,13 @@ int paxos_lease_leader_read(struct timeout *ti,
return error;
}
+/* return a random int between a and b inclusive */
+
+static int get_rand(int a, int b)
+{
+ return a + (int) (((float)(b - a + 1)) * random() / (RAND_MAX+1.0));
+}
+
static int write_new_leader(struct timeout *ti, struct token *token,
struct leader_record *nl, const char *caller)
{
@@ -648,7 +702,11 @@ static int write_new_leader(struct timeout *ti, struct token *token,
}
if (!majority_disks(token, num_writes)) {
- log_errot(token, "%s write_new_leader no majority writes", caller);
+ log_errot(token, "%s write_new_leader error %d owner %llu %llu %llu",
+ caller, rv,
+ (unsigned long long)nl->owner_id,
+ (unsigned long long)nl->owner_generation,
+ (unsigned long long)nl->timestamp);
error = SANLK_LEADER_WRITE;
}
@@ -683,7 +741,7 @@ int paxos_lease_acquire(struct timeout *ti,
uint64_t next_lver;
uint64_t our_mbal = 0;
uint64_t last_timestamp = 0;
- int error, rv, d, num_reads, disk_open = 0;
+ int error, rv, d, us, num_reads, disk_open = 0;
log_token(token, "paxos_acquire begin acquire_lver %llu flags %x",
(unsigned long long)acquire_lver, flags);
@@ -705,7 +763,8 @@ int paxos_lease_acquire(struct timeout *ti,
}
if (cur_leader.timestamp == LEASE_FREE) {
- log_token(token, "paxos_acquire lease free");
+ log_token(token, "paxos_acquire leader %llu free",
+ (unsigned long long)cur_leader.lver);
goto run;
}
@@ -840,7 +899,8 @@ int paxos_lease_acquire(struct timeout *ti,
last_timestamp = host_id_leader.timestamp;
- sleep(2);
+ /* TODO: test with sleep(2) here */
+ sleep(1);
error = paxos_lease_leader_read(ti, token, &tmp_leader, "paxos_acquire");
if (error < 0)
@@ -908,8 +968,12 @@ int paxos_lease_acquire(struct timeout *ti,
tmp_leader.owner_generation == token->host_generation) {
/* not a problem, but interesting to see, so use log_error */
- log_errot(token, "paxos_acquire %llu our id commited by %llu",
+ log_errot(token, "paxos_acquire %llu owner our inp "
+ "%llu %llu %llu commited by %llu",
(unsigned long long)next_lver,
+ (unsigned long long)tmp_leader.owner_id,
+ (unsigned long long)tmp_leader.owner_generation,
+ (unsigned long long)tmp_leader.timestamp,
(unsigned long long)tmp_leader.write_id);
memcpy(leader_ret, &tmp_leader, sizeof(struct leader_record));
@@ -930,8 +994,11 @@ int paxos_lease_acquire(struct timeout *ti,
&dblock);
if (error == SANLK_DBLOCK_MBAL) {
- log_token(token, "paxos_acquire %llu retry ballot",
- (unsigned long long)next_lver);
+ us = get_rand(0, 1000000);
+ /* not a problem, but interesting to see, so use log_error */
+ log_errot(token, "paxos_acquire %llu retry delay %d us",
+ (unsigned long long)next_lver, us);
+ usleep(us);
our_mbal += cur_leader.max_hosts;
goto retry_ballot;
}
@@ -965,7 +1032,7 @@ int paxos_lease_acquire(struct timeout *ti,
if (new_leader.owner_id != token->host_id) {
/* not a problem, but interesting to see, so use log_error */
- log_errot(token, "paxos_acquire %llu commit other owner %llu %llu %llu",
+ log_errot(token, "ballot %llu commit other owner %llu %llu %llu",
(unsigned long long)new_leader.lver,
(unsigned long long)new_leader.owner_id,
(unsigned long long)new_leader.owner_generation,
@@ -975,7 +1042,7 @@ int paxos_lease_acquire(struct timeout *ti,
goto out;
}
- log_token(token, "paxos_acquire %llu owner %llu %llu %llu done",
+ log_token(token, "ballot %llu commit self owner %llu %llu %llu",
(unsigned long long)next_lver,
(unsigned long long)new_leader.owner_id,
(unsigned long long)new_leader.owner_generation,
@@ -992,24 +1059,6 @@ int paxos_lease_acquire(struct timeout *ti,
}
#if 0
-int paxos_lease_leader_write(struct timeout *ti,
- struct token *token,
- struct leader_record *leader_new)
-{
- int error;
-
- log_token(token, "paxos_lease_leader_write begin");
-
- leader_new->checksum = leader_checksum(leader_new);
-
- error = write_new_leader(ti, token, leader_new);
-
- log_token(token, "paxos_lease_leader_write done %d", error);
- return error;
-}
-#endif
-
-#if 0
int paxos_lease_renew(struct timeout *ti,
struct token *token,
struct leader_record *leader_last,
commit c7c1ce615bd633f11520105b4352097c319a2998
Author: David Teigland <teigland(a)redhat.com>
Date: Thu May 5 12:23:51 2011 -0500
sanlock: direct read_leader
new command to directly read and print a leader_record
diff --git a/src/delta_lease.c b/src/delta_lease.c
index 036d7dc..9b2a6d1 100644
--- a/src/delta_lease.c
+++ b/src/delta_lease.c
@@ -171,11 +171,9 @@ int delta_lease_leader_read(struct timeout *ti,
return SANLK_LEADER_READ;
error = verify_leader(disk, space_name, host_id, &leader, caller);
- if (error < 0)
- return error;
memcpy(leader_ret, &leader, sizeof(struct leader_record));
- return SANLK_OK;
+ return error;
}
int delta_lease_acquire(struct timeout *ti,
diff --git a/src/direct.c b/src/direct.c
index 1a8312b..ff1597e 100644
--- a/src/direct.c
+++ b/src/direct.c
@@ -31,6 +31,54 @@
#include "delta_lease.h"
#include "sanlock_direct.h"
+/*
+ * cli: sanlock direct init
+ * cli: sanlock direct read_leader
+ * cli: sanlock direct acquire
+ * cli: sanlock direct release
+ * lib: sanlock_direct_init()
+ *
+ * direct.c:
+ * direct_init()
+ * direct_read_leader()
+ * direct_acquire()
+ * direct_release()
+ * do_paxos_action()
+ * paxos_lease.c:
+ * paxos_lease_init()
+ * paxos_lease_leader_read()
+ * paxos_lease_acquire()
+ * paxos_lease_release()
+ *
+ * cli: sanlock direct init
+ * cli: sanlock direct read_leader
+ * cli: sanlock direct acquire_id
+ * cli: sanlock direct release_id
+ * cli: sanlock direct renew_id
+ * cli: sanlock direct read_id
+ * cli: sanlock direct live_id
+ * lib: sanlock_direct_read_id()
+ * lib: sanlock_direct_live_id()
+ * lib: sanlock_direct_init()
+ *
+ * direct.c:
+ * direct_init()
+ * direct_read_leader()
+ * direct_acquire_id()
+ * direct_release_id()
+ * direct_renew_id()
+ * direct_read_id()
+ * direct_live_id()
+ * do_delta_action()
+ * delta_lease.c:
+ * delta_lease_init()
+ * delta_lease_leader_read()
+ * delta_lease_acquire()
+ * delta_lease_release()
+ * delta_lease_renew()
+ */
+
+
/* TODO: include from sanlock_internal */
static struct timeout to_default = {
DEFAULT_USE_AIO,
@@ -40,16 +88,15 @@ static struct timeout to_default = {
DEFAULT_HOST_ID_RENEWAL_FAIL_SECONDS,
DEFAULT_HOST_ID_RENEWAL_WARN_SECONDS };
-static int do_paxos_action(int action,
- struct timeout *ti,
+static int do_paxos_action(int action, struct timeout *ti,
struct sanlk_resource *res,
- int max_hosts,
- int num_hosts,
+ int max_hosts, int num_hosts,
uint64_t local_host_id,
- uint64_t local_host_generation)
+ uint64_t local_host_generation,
+ struct leader_record *leader_ret)
{
struct token *token;
- struct leader_record leader_read, leader_ret;
+ struct leader_record leader;
int disks_len, token_len;
int num_opened;
int j, rv = 0;
@@ -76,49 +123,42 @@ static int do_paxos_action(int action,
}
num_opened = open_disks(token->disks, token->r.num_disks);
- if (!majority_disks(token, num_opened)) {
- log_tool("cannot open majority of disks");
- return -1;
- }
+ if (!majority_disks(token, num_opened))
+ return -ENODEV;
switch (action) {
case ACT_INIT:
rv = paxos_lease_init(ti, token, num_hosts, max_hosts);
- if (rv < 0) {
- log_tool("cannot initialize disks");
- goto exit_fail;
- }
break;
case ACT_ACQUIRE:
token->host_id = local_host_id;
token->host_generation = local_host_generation;
- rv = paxos_lease_acquire(ti, token, 0, &leader_ret, 0, num_hosts);
- if (rv < 0) {
- log_tool("cannot acquire lease on %s", token->r.name);
- goto exit_fail;
- }
+ rv = paxos_lease_acquire(ti, token, 0, leader_ret, 0, num_hosts);
break;
case ACT_RELEASE:
- rv = paxos_lease_leader_read(ti, token, &leader_read, "direct_release");
- if (rv < 0) {
- log_tool("cannot read lease on %s", token->r.name);
- goto exit_fail;
- }
- rv = paxos_lease_release(ti, token, &leader_read, &leader_ret);
- if (rv < 0) {
- log_tool("cannot release lease on %s", token->r.name);
- goto exit_fail;
- }
+ rv = paxos_lease_leader_read(ti, token, &leader, "direct_release");
+ if (rv < 0)
+ break;
+ rv = paxos_lease_release(ti, token, &leader, leader_ret);
+ break;
+
+ case ACT_READ_LEADER:
+ rv = paxos_lease_leader_read(ti, token, &leader, "direct_read_leader");
break;
}
-exit_fail:
close_disks(token->disks, token->r.num_disks);
free(token);
+ if (rv == SANLK_OK)
+ rv = 0;
+
+ if (leader_ret)
+ memcpy(leader_ret, &leader, sizeof(struct leader_record));
+
return rv;
}
@@ -131,21 +171,29 @@ int direct_acquire(struct timeout *ti,
struct sanlk_resource *res,
int num_hosts,
uint64_t local_host_id,
- uint64_t local_host_generation)
+ uint64_t local_host_generation,
+ struct leader_record *leader_ret)
{
- return do_paxos_action(ACT_ACQUIRE, ti, res, -1, num_hosts,
- local_host_id, local_host_generation);
+ return do_paxos_action(ACT_ACQUIRE, ti, res,
+ -1, num_hosts,
+ local_host_id, local_host_generation,
+ leader_ret);
}
int direct_release(struct timeout *ti,
- struct sanlk_resource *res)
+ struct sanlk_resource *res,
+ struct leader_record *leader_ret)
{
- return do_paxos_action(ACT_RELEASE, ti, res, -1, -1, 0, 0);
+ return do_paxos_action(ACT_RELEASE, ti, res,
+ -1, -1,
+ 0, 0,
+ leader_ret);
}
static int do_delta_action(int action,
struct timeout *ti,
struct sanlk_lockspace *ls,
+ int max_hosts,
struct leader_record *leader_ret)
{
struct leader_record leader;
@@ -153,32 +201,24 @@ static int do_delta_action(int action,
struct space space;
int rv;
- if (!ls->name[0])
- return -1;
-
- if (!ls->host_id_disk.path[0]) {
- log_tool("invalid disk path");
- return -1;
- }
-
- if (!ls->host_id) {
- log_tool("zero host_id");
- return -1;
- }
-
/* for log_space in delta functions */
memset(&space, 0, sizeof(space));
+ if (!ls->host_id_disk.path[0])
+ return -ENODEV;
+
memset(&sd, 0, sizeof(struct sync_disk));
memcpy(&sd, &ls->host_id_disk, sizeof(struct sanlk_disk));
rv = open_disks(&sd, 1);
- if (rv != 1) {
- log_tool("open_disk failed %d %s", rv, sd.path);
- return -1;
- }
+ if (rv != 1)
+ return -ENODEV;
switch (action) {
+ case ACT_INIT:
+ rv = delta_lease_init(ti, &sd, ls->name, max_hosts);
+ break;
+
case ACT_ACQUIRE_ID:
rv = delta_lease_acquire(ti, &space, &sd,
ls->name,
@@ -207,6 +247,7 @@ static int do_delta_action(int action,
&leader, &leader);
break;
case ACT_READ_ID:
+ case ACT_READ_LEADER:
rv = delta_lease_leader_read(ti, &sd,
ls->name,
ls->host_id,
@@ -215,6 +256,8 @@ static int do_delta_action(int action,
break;
}
+ close_disks(&sd, 1);
+
if (rv == SANLK_OK)
rv = 0;
@@ -233,17 +276,17 @@ static int do_delta_action(int action,
int direct_acquire_id(struct timeout *ti, struct sanlk_lockspace *ls)
{
- return do_delta_action(ACT_ACQUIRE_ID, ti, ls, NULL);
+ return do_delta_action(ACT_ACQUIRE_ID, ti, ls, -1, NULL);
}
int direct_release_id(struct timeout *ti, struct sanlk_lockspace *ls)
{
- return do_delta_action(ACT_RELEASE_ID, ti, ls, NULL);
+ return do_delta_action(ACT_RELEASE_ID, ti, ls, -1, NULL);
}
int direct_renew_id(struct timeout *ti, struct sanlk_lockspace *ls)
{
- return do_delta_action(ACT_RENEW_ID, ti, ls, NULL);
+ return do_delta_action(ACT_RENEW_ID, ti, ls, -1, NULL);
}
int direct_read_id(struct timeout *ti,
@@ -257,7 +300,7 @@ int direct_read_id(struct timeout *ti,
memset(&leader, 0, sizeof(struct leader_record));
- rv = do_delta_action(ACT_READ_ID, ti, ls, &leader);
+ rv = do_delta_action(ACT_READ_ID, ti, ls, -1, &leader);
*timestamp = leader.timestamp;
*owner_id = leader.owner_id;
@@ -290,7 +333,7 @@ int direct_live_id(struct timeout *ti,
time_t start;
int rv;
- rv = do_delta_action(ACT_READ_ID, ti, ls, &leader_begin);
+ rv = do_delta_action(ACT_READ_ID, ti, ls, -1, &leader_begin);
if (rv < 0)
return rv;
@@ -299,7 +342,7 @@ int direct_live_id(struct timeout *ti,
while (1) {
sleep(1);
- rv = do_delta_action(ACT_READ_ID, ti, ls, &leader);
+ rv = do_delta_action(ACT_READ_ID, ti, ls, -1, &leader);
if (rv < 0)
return rv;
@@ -355,61 +398,29 @@ int direct_init(struct timeout *ti,
struct sanlk_resource *res,
int max_hosts, int num_hosts)
{
- struct sync_disk sd;
- int num_opened;
- int rv;
+ int rv = -1;
if (ls && ls->host_id_disk.path[0]) {
- memset(&sd, 0, sizeof(struct sync_disk));
- memcpy(&sd, &ls->host_id_disk, sizeof(struct sanlk_disk));
-
- num_opened = open_disks(&sd, 1);
- if (num_opened != 1) {
- log_tool("cannot open disk %s", sd.path);
- return -1;
- }
+ rv = do_delta_action(ACT_INIT, ti, ls, max_hosts, NULL);
- rv = delta_lease_init(ti, &sd, ls->name, max_hosts);
- close_disks(&sd, 1);
+ } else if (res) {
+ if (!num_hosts)
+ return -EINVAL;
- if (rv < 0) {
- log_tool("lockspace init failed %d", rv);
- return -1;
- }
- }
+ if (num_hosts > max_hosts)
+ return SANLK_LEADER_NUMHOSTS;
- if (res) {
- if (!num_hosts) {
- log_tool("num_hosts option required for paxos lease init");
- return -1;
- }
+ if (!res->num_disks)
+ return -ENODEV;
- if (!max_hosts)
- max_hosts = DEFAULT_MAX_HOSTS;
+ if (!res->disks[0].path[0])
+ return -ENODEV;
- if (num_hosts > max_hosts) {
- log_tool("num_hosts cannot be greater than max_hosts");
- return -1;
- }
-
- if (!res->num_disks) {
- log_tool("num_disks zero");
- return -1;
- }
-
- if (!res->disks[0].path[0]) {
- log_tool("invalid disk path");
- return -1;
- }
-
- rv = do_paxos_action(ACT_INIT, ti, res, max_hosts, num_hosts, 0, 0);
- if (rv < 0) {
- log_tool("resource init failed %d", rv);
- return rv;
- }
+ rv = do_paxos_action(ACT_INIT, ti, res,
+ max_hosts, num_hosts, 0, 0, NULL);
}
- return 0;
+ return rv;
}
int sanlock_direct_init(struct sanlk_lockspace *ls,
@@ -425,6 +436,22 @@ int sanlock_direct_init(struct sanlk_lockspace *ls,
return direct_init(&ti, ls, res, max_hosts, num_hosts);
}
+int direct_read_leader(struct timeout *ti,
+ struct sanlk_lockspace *ls,
+ struct sanlk_resource *res,
+ struct leader_record *leader_ret)
+{
+ int rv = -1;
+
+ if (ls && ls->host_id_disk.path[0])
+ rv = do_delta_action(ACT_READ_LEADER, ti, ls, -1, leader_ret);
+
+ else if (res)
+ rv = do_paxos_action(ACT_READ_LEADER, ti, res,
+ -1, -1, 0, 0, leader_ret);
+ return rv;
+}
+
int direct_dump(struct timeout *ti, char *dump_path)
{
char *data;
@@ -448,10 +475,8 @@ int direct_dump(struct timeout *ti, char *dump_path)
strncpy(sd.path, dump_path, SANLK_PATH_LEN);
num_opened = open_disks(&sd, 1);
- if (num_opened != 1) {
- log_tool("cannot open disk %s", sd.path);
- return -1;
- }
+ if (num_opened != 1)
+ return -ENODEV;
data = malloc(sd.sector_size);
if (!data)
diff --git a/src/direct.h b/src/direct.h
index b73affc..e500638 100644
--- a/src/direct.h
+++ b/src/direct.h
@@ -13,10 +13,12 @@ int direct_acquire(struct timeout *ti,
struct sanlk_resource *res,
int num_hosts,
uint64_t local_host_id,
- uint64_t local_host_generation);
+ uint64_t local_host_generation,
+ struct leader_record *leader_ret);
int direct_release(struct timeout *ti,
- struct sanlk_resource *res);
+ struct sanlk_resource *res,
+ struct leader_record *leader_ret);
int direct_acquire_id(struct timeout *ti, struct sanlk_lockspace *ls);
int direct_release_id(struct timeout *ti, struct sanlk_lockspace *ls);
@@ -40,6 +42,11 @@ int direct_init(struct timeout *ti,
struct sanlk_resource *res,
int max_hosts, int num_hosts);
+int direct_read_leader(struct timeout *ti,
+ struct sanlk_lockspace *ls,
+ struct sanlk_resource *res,
+ struct leader_record *leader_ret);
+
int direct_dump(struct timeout *ti, char *dump_path);
#endif
diff --git a/src/main.c b/src/main.c
index 40cd20c..c9287d0 100644
--- a/src/main.c
+++ b/src/main.c
@@ -2268,6 +2268,7 @@ static void print_usage(void)
printf("direct actions: read/write storage directly to:\n");
printf(" init initialize disk areas for host_id and resource leases\n");
printf(" dump print initialized leases\n");
+ printf(" read_leader print values in leader_record\n");
printf(" acquire acquire leases\n");
printf(" release release leases\n");
printf(" acquire_id acquire a host_id lease\n");
@@ -2334,6 +2335,9 @@ static void print_usage(void)
printf(" -D debug: print extra info for debugging\n");
printf(" -a <num> use async io (1 yes, 0 no)\n");
printf("\n");
+ printf("direct read_leader [-s LOCKSPACE] [-r RESOURCE]\n");
+ printf(" -a <num> use async io (1 yes, 0 no)\n");
+ printf("\n");
printf("direct acquire|release -i <num> -g <num> -r RESOURCE\n");
printf(" -a <num> use async io (1 yes, 0 no)\n");
printf(" -n <num_hosts> change num_hosts in leases when acquired\n");
@@ -2440,6 +2444,8 @@ static int read_command_line(int argc, char *argv[])
com.action = ACT_INIT;
else if (!strcmp(act, "dump"))
com.action = ACT_DUMP;
+ else if (!strcmp(act, "read_leader"))
+ com.action = ACT_READ_LEADER;
else if (!strcmp(act, "acquire"))
com.action = ACT_ACQUIRE;
else if (!strcmp(act, "release"))
@@ -2734,6 +2740,7 @@ static int do_client(void)
static int do_direct(void)
{
+ struct leader_record leader;
uint64_t timestamp, owner_id, owner_generation;
int live;
int rv;
@@ -2750,14 +2757,44 @@ static int do_direct(void)
log_tool("dump done %d", rv);
break;
+ case ACT_READ_LEADER:
+ rv = direct_read_leader(&to, &com.lockspace, com.res_args[0], &leader);
+ log_tool("read_leader done %d", rv);
+ log_tool("magic 0x%x", leader.magic);
+ log_tool("version 0x%x", leader.version);
+ log_tool("sector_size %u", leader.sector_size);
+ log_tool("num_hosts %llu",
+ (unsigned long long)leader.num_hosts);
+ log_tool("max_hosts %llu",
+ (unsigned long long)leader.max_hosts);
+ log_tool("owner_id %llu",
+ (unsigned long long)leader.owner_id);
+ log_tool("owner_generation %llu",
+ (unsigned long long)leader.owner_generation);
+ log_tool("lver %llu",
+ (unsigned long long)leader.lver);
+ log_tool("space_name %.48s", leader.space_name);
+ log_tool("resource_name %.48s", leader.resource_name);
+ log_tool("timestamp %llu",
+ (unsigned long long)leader.timestamp);
+ log_tool("checksum %u", leader.checksum);
+ log_tool("write_id %llu",
+ (unsigned long long)leader.write_id);
+ log_tool("write_generation %llu",
+ (unsigned long long)leader.write_generation);
+ log_tool("write_timestamp %llu",
+ (unsigned long long)leader.write_timestamp);
+ break;
+
case ACT_ACQUIRE:
rv = direct_acquire(&to, com.res_args[0], com.num_hosts,
- com.local_host_id, com.local_host_generation);
+ com.local_host_id, com.local_host_generation,
+ &leader);
log_tool("acquire done %d", rv);
break;
case ACT_RELEASE:
- rv = direct_release(&to, com.res_args[0]);
+ rv = direct_release(&to, com.res_args[0], &leader);
log_tool("release done %d", rv);
break;
diff --git a/src/paxos_lease.c b/src/paxos_lease.c
index 1e84dc7..116823d 100644
--- a/src/paxos_lease.c
+++ b/src/paxos_lease.c
@@ -626,6 +626,7 @@ int paxos_lease_leader_read(struct timeout *ti,
return SANLK_OK;
fail:
+ memcpy(leader_ret, &leader, sizeof(struct leader_record));
free(leaders);
free(leader_reps);
return error;
diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h
index f8c4e2f..5ad996a 100644
--- a/src/sanlock_internal.h
+++ b/src/sanlock_internal.h
@@ -310,6 +310,7 @@ enum {
ACT_LIVE_ID,
ACT_INIT,
ACT_DUMP,
+ ACT_READ_LEADER,
};
#endif
12 years, 11 months
src/delta_lease.c src/leader.h src/main.c src/paxos_lease.c src/sanlock_rv.h
by David Teigland
src/delta_lease.c | 18 -
src/leader.h | 5
src/main.c | 2
src/paxos_lease.c | 571 +++++++++++++++++++++++++++++++++---------------------
src/sanlock_rv.h | 67 +++---
5 files changed, 402 insertions(+), 261 deletions(-)
New commits:
commit 5a56aa105d308615661e249ff9e486ddda7065ff
Author: David Teigland <teigland(a)redhat.com>
Date: Tue May 3 16:28:14 2011 -0500
sanlock: correct paxos usage
Rework how the disk paxos algorithm is applied to the
ownership and updating of the leader_record. How this
was done and worked previously had never been thoroughly
understood and was wrong in a number of general ways.
Some main concepts driving these changes:
. ballots are run to commit each new lver
. a node that successfully completes the ballot needs to
commit the result from that ballot, which means
writing the lver from that ballot and the dblock values
to the leader_record
. nodes need to monitor the leader_record to learn if
the next lver has been commited, and if so, return that
result
. the hosts participating in a ballot are the ones with
the latest matching lver in their dblocks
. each node's mbal should always increase
This includes a new incompatible disk format change,
due to the addition of some extra info fields to the leader
and dblock structs.
Error codes are also changed to describe the new conditions.
diff --git a/src/delta_lease.c b/src/delta_lease.c
index 63fb2a0..036d7dc 100644
--- a/src/delta_lease.c
+++ b/src/delta_lease.c
@@ -83,7 +83,7 @@ static int verify_leader(struct sync_disk *disk,
log_error("verify_leader %llu wrong magic %x %s",
(unsigned long long)host_id,
lr->magic, disk->path);
- result = SANLK_BAD_MAGIC;
+ result = SANLK_LEADER_MAGIC;
goto fail;
}
@@ -91,7 +91,7 @@ static int verify_leader(struct sync_disk *disk,
log_error("verify_leader %llu wrong version %x %s",
(unsigned long long)host_id,
lr->version, disk->path);
- result = SANLK_BAD_VERSION;
+ result = SANLK_LEADER_VERSION;
goto fail;
}
@@ -99,7 +99,7 @@ static int verify_leader(struct sync_disk *disk,
log_error("verify_leader %llu wrong sector size %d %d %s",
(unsigned long long)host_id,
lr->sector_size, disk->sector_size, disk->path);
- result = SANLK_BAD_SECTORSIZE;
+ result = SANLK_LEADER_SECTORSIZE;
goto fail;
}
@@ -107,7 +107,7 @@ static int verify_leader(struct sync_disk *disk,
log_error("verify_leader %llu wrong space name %.48s %.48s %s",
(unsigned long long)host_id,
lr->space_name, space_name, disk->path);
- result = SANLK_BAD_LOCKSPACE;
+ result = SANLK_LEADER_LOCKSPACE;
goto fail;
}
@@ -119,7 +119,7 @@ static int verify_leader(struct sync_disk *disk,
log_error("verify_leader %llu wrong resource name %.48s %.48s %s",
(unsigned long long)host_id,
lr->resource_name, resource_name, disk->path);
- result = SANLK_BAD_RESOURCEID;
+ result = SANLK_LEADER_RESOURCE;
goto fail;
}
@@ -129,7 +129,7 @@ static int verify_leader(struct sync_disk *disk,
log_error("verify_leader %llu wrong checksum %x %x %s",
(unsigned long long)host_id,
lr->checksum, sum, disk->path);
- result = SANLK_BAD_CHECKSUM;
+ result = SANLK_LEADER_CHECKSUM;
goto fail;
}
@@ -168,7 +168,7 @@ int delta_lease_leader_read(struct timeout *ti,
sizeof(struct leader_record),
ti->io_timeout_seconds, ti->use_aio, "delta_leader");
if (rv < 0)
- return SANLK_READ_LEADERS;
+ return SANLK_LEADER_READ;
error = verify_leader(disk, space_name, host_id, &leader, caller);
if (error < 0)
@@ -293,7 +293,7 @@ int delta_lease_renew(struct timeout *ti,
return error;
if (leader.owner_id != our_host_id)
- return SANLK_BAD_LEADER;
+ return SANLK_RENEW_OWNER;
new_ts = time(NULL);
@@ -331,7 +331,7 @@ int delta_lease_renew(struct timeout *ti,
(unsigned long long)host_id);
log_leader_error(0, space_name, host_id, disk, &leader, "delta_renew_write");
log_leader_error(0, space_name, host_id, disk, &leader_read, "delta_renew_reread");
- return SANLK_BAD_LEADER;
+ return SANLK_RENEW_DIFF;
}
memcpy(leader_ret, &leader, sizeof(struct leader_record));
diff --git a/src/leader.h b/src/leader.h
index 87b3d31..e100bb6 100644
--- a/src/leader.h
+++ b/src/leader.h
@@ -16,7 +16,7 @@
#define NAME_ID_SIZE 48
#define PAXOS_DISK_MAGIC 0x06152010
-#define PAXOS_DISK_VERSION_MAJOR 0x00040000
+#define PAXOS_DISK_VERSION_MAJOR 0x00050000
#define PAXOS_DISK_VERSION_MINOR 0x00000001
#define DELTA_DISK_MAGIC 0x12212010
@@ -58,6 +58,9 @@ struct leader_record {
uint64_t unused1;
uint32_t checksum;
uint32_t unused2;
+ uint64_t write_id; /* for extra info, debug */
+ uint64_t write_generation; /* for extra info, debug */
+ uint64_t write_timestamp; /* for extra info, debug */
};
#endif
diff --git a/src/main.c b/src/main.c
index d30dd00..40cd20c 100644
--- a/src/main.c
+++ b/src/main.c
@@ -920,7 +920,7 @@ static void *cmd_acquire_thread(void *args_in)
rv = acquire_token(token, acquire_lver, new_num_hosts);
if (rv < 0) {
- if (rv == SANLK_LIVE_LEADER && com.quiet_fail) {
+ if (rv == SANLK_ACQUIRE_IDLIVE && com.quiet_fail) {
log_token(token, "cmd_acquire %d,%d,%d paxos_lease %d",
cl_ci, cl_fd, cl_pid, rv);
} else {
diff --git a/src/paxos_lease.c b/src/paxos_lease.c
index c846d47..1e84dc7 100644
--- a/src/paxos_lease.c
+++ b/src/paxos_lease.c
@@ -29,23 +29,18 @@
#include "delta_lease.h"
#include "paxos_lease.h"
-/*
- * largely copied from vdsm.git/sync_manager/
- */
-
-#define NO_VAL 0
-
struct request_record {
uint64_t lver;
uint8_t force_mode;
};
-/* ref: ballot_ticket_record */
struct paxos_dblock {
- uint64_t mbal; /* aka curr_bal */
- uint64_t bal; /* aka inp_bal */
- uint64_t inp; /* aka inp_val */
- uint64_t lver; /* leader version */
+ uint64_t mbal;
+ uint64_t bal;
+ uint64_t inp; /* host_id */
+ uint64_t inp2; /* host_id generation */
+ uint64_t inp3; /* host_id's timestamp */
+ uint64_t lver;
};
int majority_disks(struct token *token, int num)
@@ -132,7 +127,7 @@ static int read_dblocks(struct timeout *ti,
data = malloc(data_len);
if (!data) {
log_error("read_dblocks malloc %d %s", data_len, disk->path);
- rv = -1;
+ rv = -ENOMEM;
goto out;
}
@@ -188,12 +183,9 @@ static int read_request(struct timeout *ti,
}
#endif
-/* host_id and inp are both generally our_host_id */
-
-static int run_disk_paxos(struct timeout *ti,
- struct token *token, uint64_t host_id, uint64_t inp,
- int num_hosts, uint64_t lver,
- struct paxos_dblock *dblock_out)
+static int run_ballot(struct timeout *ti, struct token *token, int num_hosts,
+ uint64_t next_lver, uint64_t our_mbal,
+ struct paxos_dblock *dblock_out)
{
struct paxos_dblock bk[num_hosts];
struct paxos_dblock bk_max;
@@ -201,48 +193,7 @@ static int run_disk_paxos(struct timeout *ti,
int num_disks = token->r.num_disks;
int num_writes, num_reads;
int d, q, rv;
-
- if (!host_id) {
- log_errot(token, "invalid host_id");
- return SANLK_INVAL;
- }
-
- if (!inp) {
- log_errot(token, "invalid inp");
- return SANLK_INVAL;
- }
-
- /* read one of our own dblock's to get initial dblock values */
-
- memset(&dblock, 0, sizeof(struct paxos_dblock));
-
- for (d = 0; d < num_disks; d++) {
- rv = read_dblock(ti, &token->disks[d], host_id, &dblock);
- if (rv < 0)
- continue;
- /* need only one dblock to get initial values */
- break;
- }
-
- if (rv < 0) {
- log_errot(token, "no initial dblock found");
- return SANLK_OWN_DBLOCK;
- }
-
- log_token(token, "initial dblock %u mbal %llu bal %llu inp %llu lver %llu", d,
- (unsigned long long)dblock.mbal,
- (unsigned long long)dblock.bal,
- (unsigned long long)dblock.inp,
- (unsigned long long)dblock.lver);
-
- if (lver > dblock.lver) {
- dblock.mbal = host_id;
- dblock.bal = 0; /* or NO_VAL? lamport paper has 0 */
- dblock.inp = NO_VAL;
- dblock.lver = lver;
- } else {
- dblock.mbal += num_hosts;
- }
+ int q_max = -1;
/*
* phase 1
@@ -255,22 +206,29 @@ static int run_disk_paxos(struct timeout *ti,
* component is greater than dblock[p].mbal."
*/
+ log_token(token, "ballot %llu phase1 mbal %llu",
+ (unsigned long long)next_lver,
+ (unsigned long long)our_mbal);
+
+ memset(&dblock, 0, sizeof(struct paxos_dblock));
+ dblock.mbal = our_mbal;
+ dblock.lver = next_lver;
+
memset(&bk_max, 0, sizeof(struct paxos_dblock));
- bk_max.bal = NO_VAL;
- bk_max.inp = NO_VAL;
num_writes = 0;
for (d = 0; d < num_disks; d++) {
- rv = write_dblock(ti, &token->disks[d], host_id, &dblock);
+ rv = write_dblock(ti, &token->disks[d], token->host_id, &dblock);
if (rv < 0)
continue;
num_writes++;
}
if (!majority_disks(token, num_writes)) {
- log_errot(token, "cannot write dblock to majority of disks");
- return SANLK_WRITE1_DBLOCKS;
+ log_errot(token, "ballot %llu dblock write error %d",
+ (unsigned long long)next_lver, rv);
+ return SANLK_DBLOCK_WRITE;
}
num_reads = 0;
@@ -286,38 +244,49 @@ static int run_disk_paxos(struct timeout *ti,
continue;
if (bk[q].lver > dblock.lver) {
- log_errot(token, "bk %d %d lver %llu dblock lver %llu",
- d, q,
- (unsigned long long)bk[q].lver,
- (unsigned long long)dblock.lver);
- return SANLK_READ1_LVER;
+ /* I don't think this should happen */
+ log_errot(token, "ballot %llu larger lver[%d] %llu",
+ (unsigned long long)next_lver, q,
+ (unsigned long long)bk[q].lver);
+ return SANLK_DBLOCK_LVER;
}
/* see "It aborts the ballot" in comment above */
if (bk[q].mbal > dblock.mbal) {
- log_errot(token, "bk %d %d mbal %llu dblock mbal %llu",
- d, q,
- (unsigned long long)bk[q].mbal,
- (unsigned long long)dblock.mbal);
- return SANLK_READ1_MBAL;
+ log_errot(token, "ballot %llu mbal %llu larger mbal[%d] %llu",
+ (unsigned long long)next_lver,
+ (unsigned long long)our_mbal, q,
+ (unsigned long long)bk[q].mbal);
+ return SANLK_DBLOCK_MBAL;
}
/* see choosing inp for phase 2 in comment below */
- if (bk[q].inp == NO_VAL)
+ if (!bk[q].inp)
continue;
- if (bk_max.bal == NO_VAL || bk[q].bal > bk_max.bal)
+ if (!bk[q].bal) {
+ log_errot(token, "ballot %llu zero bal inp[%d] %llu",
+ (unsigned long long)next_lver, q,
+ (unsigned long long)bk[q].inp);
+ continue;
+ }
+
+ if (bk[q].bal > bk_max.bal) {
bk_max = bk[q];
+ q_max = q;
+ }
}
}
if (!majority_disks(token, num_reads)) {
- log_errot(token, "cannot read dblocks on majority of disks");
- return SANLK_READ1_DBLOCKS;
+ log_errot(token, "ballot %llu dblock read error %d",
+ (unsigned long long)next_lver, rv);
+ return SANLK_DBLOCK_READ;
}
+
/*
* "When it completes phase 1, p chooses a new value of dblock[p].inp,
* sets dblock[p].bal to dblock[p].mbal (its current ballot number),
@@ -333,31 +302,59 @@ static int run_disk_paxos(struct timeout *ti,
* nonInitBlks having the largest value of bk.bal."
*/
- log_token(token, "bk_max inp %llu bal %llu",
- (unsigned long long)bk_max.inp,
- (unsigned long long)bk_max.bal);
-
- dblock.inp = (bk_max.inp == NO_VAL) ? inp : bk_max.inp;
+ if (bk_max.inp) {
+ /* lver and mbal are already set */
+ dblock.inp = bk_max.inp;
+ dblock.inp2 = bk_max.inp2;
+ dblock.inp3 = bk_max.inp3;
+ } else {
+ /* lver and mbal are already set */
+ dblock.inp = token->host_id;
+ dblock.inp2 = token->host_generation;
+ dblock.inp3 = time(NULL);
+ }
dblock.bal = dblock.mbal;
+ if (bk_max.inp) {
+ /* not a problem, but interesting to see, so use log_error */
+ log_errot(token, "ballot %llu bk_max[%d] lver %llu mbal %llu bal %llu inp %llu %llu %llu",
+ (unsigned long long)next_lver, q_max,
+ (unsigned long long)bk_max.lver,
+ (unsigned long long)bk_max.mbal,
+ (unsigned long long)bk_max.bal,
+ (unsigned long long)bk_max.inp,
+ (unsigned long long)bk_max.inp2,
+ (unsigned long long)bk_max.inp3);
+ }
+
+
/*
* phase 2
*
* Same description as phase 1, same sequence of writes/reads.
*/
+ log_token(token, "ballot %llu phase2 bal %llu inp %llu %llu %llu q_max %d",
+ (unsigned long long)dblock.lver,
+ (unsigned long long)dblock.bal,
+ (unsigned long long)dblock.inp,
+ (unsigned long long)dblock.inp2,
+ (unsigned long long)dblock.inp3,
+ q_max);
+
num_writes = 0;
for (d = 0; d < num_disks; d++) {
- rv = write_dblock(ti, &token->disks[d], host_id, &dblock);
+ rv = write_dblock(ti, &token->disks[d], token->host_id, &dblock);
if (rv < 0)
continue;
num_writes++;
}
if (!majority_disks(token, num_writes)) {
- log_errot(token, "cannot write dblock to majority of disks 2");
- return SANLK_WRITE2_DBLOCKS;
+ log_errot(token, "ballot %llu our dblock write2 error %d",
+ (unsigned long long)next_lver, rv);
+ return SANLK_DBLOCK_WRITE;
}
num_reads = 0;
@@ -373,28 +370,29 @@ static int run_disk_paxos(struct timeout *ti,
continue;
if (bk[q].lver > dblock.lver) {
- log_errot(token, "bk %d %d lver %llu dblock lver %llu",
- d, q,
- (unsigned long long)bk[q].lver,
- (unsigned long long)dblock.lver);
- return SANLK_READ2_LVER;
+ /* I don't think this should happen */
+ log_errot(token, "ballot %llu larger2 lver[%d] %llu",
+ (unsigned long long)next_lver, q,
+ (unsigned long long)bk[q].lver);
+ return SANLK_DBLOCK_LVER;
}
/* see "It aborts the ballot" in comment above */
if (bk[q].mbal > dblock.mbal) {
- log_errot(token, "bk %d %d mbal %llu dblock mbal %llu",
- d, q,
- (unsigned long long)bk[q].mbal,
- (unsigned long long)dblock.mbal);
- return SANLK_READ2_MBAL;
+ log_errot(token, "ballot %llu mbal %llu larger2 mbal[%d] %llu",
+ (unsigned long long)next_lver,
+ (unsigned long long)our_mbal, q,
+ (unsigned long long)bk[q].mbal);
+ return SANLK_DBLOCK_MBAL;
}
}
}
if (!majority_disks(token, num_reads)) {
- log_errot(token, "cannot read dblocks from majority of disks 2");
- return SANLK_READ2_DBLOCKS;
+ log_errot(token, "ballot %llu dblock read2 error %d",
+ (unsigned long long)next_lver, rv);
+ return SANLK_DBLOCK_READ;
}
/* "When it completes phase 2, p has committed dblock[p].inp." */
@@ -440,6 +438,11 @@ static void log_leader_error(int result,
lr->resource_name,
(unsigned long long)lr->timestamp,
lr->checksum);
+
+ log_errot(token, "leader5 wi %llu wg %llu wt %llu",
+ (unsigned long long)lr->write_id,
+ (unsigned long long)lr->write_generation,
+ (unsigned long long)lr->write_timestamp);
}
static int verify_leader(struct token *token, struct sync_disk *disk,
@@ -453,35 +456,35 @@ static int verify_leader(struct token *token, struct sync_disk *disk,
if (lr->magic != PAXOS_DISK_MAGIC) {
log_errot(token, "verify_leader wrong magic %x %s",
lr->magic, disk->path);
- result = SANLK_BAD_MAGIC;
+ result = SANLK_LEADER_MAGIC;
goto fail;
}
if ((lr->version & 0xFFFF0000) != PAXOS_DISK_VERSION_MAJOR) {
log_errot(token, "verify_leader wrong version %x %s",
lr->version, disk->path);
- result = SANLK_BAD_VERSION;
+ result = SANLK_LEADER_VERSION;
goto fail;
}
if (lr->sector_size != disk->sector_size) {
log_errot(token, "verify_leader wrong sector size %d %d %s",
lr->sector_size, disk->sector_size, disk->path);
- result = SANLK_BAD_SECTORSIZE;
+ result = SANLK_LEADER_SECTORSIZE;
goto fail;
}
if (strncmp(lr->space_name, token->r.lockspace_name, NAME_ID_SIZE)) {
log_errot(token, "verify_leader wrong space name %.48s %.48s %s",
lr->space_name, token->r.lockspace_name, disk->path);
- result = SANLK_BAD_LOCKSPACE;
+ result = SANLK_LEADER_LOCKSPACE;
goto fail;
}
if (strncmp(lr->resource_name, token->r.name, NAME_ID_SIZE)) {
log_errot(token, "verify_leader wrong resource name %.48s %.48s %s",
lr->resource_name, token->r.name, disk->path);
- result = SANLK_BAD_RESOURCEID;
+ result = SANLK_LEADER_RESOURCE;
goto fail;
}
@@ -489,7 +492,7 @@ static int verify_leader(struct token *token, struct sync_disk *disk,
log_errot(token, "verify_leader num_hosts too small %llu %llu %s",
(unsigned long long)lr->num_hosts,
(unsigned long long)token->host_id, disk->path);
- result = SANLK_BAD_NUMHOSTS;
+ result = SANLK_LEADER_NUMHOSTS;
goto fail;
}
@@ -498,7 +501,7 @@ static int verify_leader(struct token *token, struct sync_disk *disk,
if (lr->checksum != sum) {
log_errot(token, "verify_leader wrong checksum %x %x %s",
lr->checksum, sum, disk->path);
- result = SANLK_BAD_CHECKSUM;
+ result = SANLK_LEADER_CHECKSUM;
goto fail;
}
@@ -529,7 +532,7 @@ int paxos_lease_leader_read(struct timeout *ti,
struct token *token, struct leader_record *leader_ret,
const char *caller)
{
- struct leader_record prev_leader;
+ struct leader_record leader;
struct leader_record *leaders;
int *leader_reps;
int leaders_len, leader_reps_len;
@@ -543,21 +546,20 @@ int paxos_lease_leader_read(struct timeout *ti,
leaders = malloc(leaders_len);
if (!leaders)
- return SANLK_NOMEM;
+ return -ENOMEM;
leader_reps = malloc(leader_reps_len);
if (!leader_reps) {
free(leaders);
- return SANLK_NOMEM;
+ return -ENOMEM;
}
/*
* find a leader block that's consistent on the majority of disks,
* so we can use as the basis for the new leader
- * ref: validate_multiple_disk_leader
*/
- memset(&prev_leader, 0, sizeof(struct leader_record));
+ memset(&leader, 0, sizeof(struct leader_record));
memset(leaders, 0, leaders_len);
memset(leader_reps, 0, leader_reps_len);
@@ -587,8 +589,8 @@ int paxos_lease_leader_read(struct timeout *ti,
}
if (!majority_disks(token, num_reads)) {
- log_errot(token, "paxos_leader_read no majority reads");
- error = SANLK_READ_LEADERS;
+ log_errot(token, "%s leader_read error %d", caller, rv);
+ error = SANLK_LEADER_READ;
goto fail;
}
@@ -601,29 +603,26 @@ int paxos_lease_leader_read(struct timeout *ti,
continue;
/* leader on d is the same on a majority of disks,
- prev_leader becomes the prototype for new_leader */
+ leader becomes the prototype for new_leader */
- memcpy(&prev_leader, &leaders[d], sizeof(struct leader_record));
+ memcpy(&leader, &leaders[d], sizeof(struct leader_record));
found = 1;
break;
}
if (!found) {
- log_errot(token, "paxos_leader_read no majority reps");
- error = SANLK_DIFF_LEADERS;
+ log_errot(token, "%s leader_read inconsistent", caller);
+ error = SANLK_LEADER_DIFF;
goto fail;
}
- log_token(token, "%s leader_read owner %llu lver %llu hosts %llu "
- "time %llu res %s",
- caller ? caller : "unknown",
- (unsigned long long)prev_leader.owner_id,
- (unsigned long long)prev_leader.lver,
- (unsigned long long)prev_leader.num_hosts,
- (unsigned long long)prev_leader.timestamp,
- prev_leader.resource_name);
-
- memcpy(leader_ret, &prev_leader, sizeof(struct leader_record));
+ log_token(token, "%s leader_read %llu owner %llu %llu %llu", caller,
+ (unsigned long long)leader.lver,
+ (unsigned long long)leader.owner_id,
+ (unsigned long long)leader.owner_generation,
+ (unsigned long long)leader.timestamp);
+
+ memcpy(leader_ret, &leader, sizeof(struct leader_record));
return SANLK_OK;
fail:
@@ -632,8 +631,8 @@ int paxos_lease_leader_read(struct timeout *ti,
return error;
}
-static int write_new_leader(struct timeout *ti,
- struct token *token, struct leader_record *nl)
+static int write_new_leader(struct timeout *ti, struct token *token,
+ struct leader_record *nl, const char *caller)
{
int num_disks = token->r.num_disks;
int num_writes = 0;
@@ -648,16 +647,23 @@ static int write_new_leader(struct timeout *ti,
}
if (!majority_disks(token, num_writes)) {
- log_errot(token, "write_new_leader no majority writes");
- error = SANLK_WRITE_LEADERS;
+ log_errot(token, "%s write_new_leader no majority writes", caller);
+ error = SANLK_LEADER_WRITE;
}
return error;
}
/*
- * acquire a lease
- * ref: obtain()
+ * If we hang or crash after completing a ballot successfully, but before
+ * commiting the leader_record, then the next host that runs a ballot (with the
+ * same lver since we did not commit the new lver to the leader_record) will
+ * commit the same inp values that we were about to commit. If the inp values
+ * they commit indicate we (who crashed or hung) are the new owner, then the
+ * other hosts will begin monitoring the liveness of our host_id. Once enough
+ * time has passed, they assume we're dead, and go on with new versions. The
+ * "enough time" ensures that if we hung before writing the leader, that we
+ * won't wake up and finally write what will then be an old invalid leader.
*/
int paxos_lease_acquire(struct timeout *ti,
@@ -666,32 +672,44 @@ int paxos_lease_acquire(struct timeout *ti,
uint64_t acquire_lver,
int new_num_hosts)
{
- struct leader_record prev_leader;
+ struct leader_record cur_leader;
+ struct leader_record tmp_leader;
struct leader_record new_leader;
struct leader_record host_id_leader;
struct sync_disk host_id_disk;
struct paxos_dblock dblock;
time_t start;
+ uint64_t next_lver;
+ uint64_t our_mbal = 0;
uint64_t last_timestamp = 0;
- int error, rv, disk_open = 0;
+ int error, rv, d, num_reads, disk_open = 0;
- log_token(token, "paxos_acquire begin lver %llu flags %x",
+ log_token(token, "paxos_acquire begin acquire_lver %llu flags %x",
(unsigned long long)acquire_lver, flags);
+ restart:
- error = paxos_lease_leader_read(ti, token, &prev_leader, "paxos_acquire");
+ error = paxos_lease_leader_read(ti, token, &cur_leader, "paxos_acquire");
if (error < 0)
goto out;
if (flags & PAXOS_ACQUIRE_FORCE)
goto run;
- if (prev_leader.timestamp == LEASE_FREE) {
+ if (acquire_lver && cur_leader.lver != acquire_lver) {
+ log_errot(token, "paxos_acquire acquire_lver %llu cur_leader %llu",
+ (unsigned long long)acquire_lver,
+ (unsigned long long)cur_leader.lver);
+ error = SANLK_ACQUIRE_LVER;
+ goto out;
+ }
+
+ if (cur_leader.timestamp == LEASE_FREE) {
log_token(token, "paxos_acquire lease free");
goto run;
}
- if (prev_leader.owner_id == token->host_id &&
- prev_leader.owner_generation == token->host_generation) {
+ if (cur_leader.owner_id == token->host_id &&
+ cur_leader.owner_generation == token->host_generation) {
log_token(token, "paxos_acquire already owner id %llu gen %llu",
(unsigned long long)token->host_id,
(unsigned long long)token->host_generation);
@@ -704,37 +722,40 @@ int paxos_lease_acquire(struct timeout *ti,
* its watchdog has triggered and we can go for the paxos lease.
*/
- log_token(token, "paxos_acquire check owner_id %llu",
- (unsigned long long)prev_leader.owner_id);
+ log_token(token, "paxos_acquire check owner_id %llu gen %llu",
+ (unsigned long long)cur_leader.owner_id,
+ (unsigned long long)cur_leader.owner_generation);
- memset(&host_id_disk, 0, sizeof(host_id_disk));
+ if (!disk_open) {
+ memset(&host_id_disk, 0, sizeof(host_id_disk));
- rv = host_id_disk_info(prev_leader.space_name, &host_id_disk);
- if (rv < 0) {
- log_errot(token, "paxos_acquire no lockspace info %.48s",
- prev_leader.space_name);
- error = SANLK_BAD_SPACE_NAME;
- goto out;
- }
+ rv = host_id_disk_info(cur_leader.space_name, &host_id_disk);
+ if (rv < 0) {
+ log_errot(token, "paxos_acquire no lockspace info %.48s",
+ cur_leader.space_name);
+ error = SANLK_ACQUIRE_LOCKSPACE;
+ goto out;
+ }
- disk_open = open_disks_fd(&host_id_disk, 1);
- if (disk_open != 1) {
- log_errot(token, "paxos_acquire cannot open host_id_disk");
- error = SANLK_BAD_SPACE_DISK;
- goto out;
+ disk_open = open_disks_fd(&host_id_disk, 1);
+ if (disk_open != 1) {
+ log_errot(token, "paxos_acquire cannot open host_id_disk");
+ error = SANLK_ACQUIRE_IDDISK;
+ goto out;
+ }
}
start = time(NULL);
while (1) {
error = delta_lease_leader_read(ti, &host_id_disk,
- prev_leader.space_name,
- prev_leader.owner_id,
+ cur_leader.space_name,
+ cur_leader.owner_id,
&host_id_leader,
"paxos_acquire");
if (error < 0) {
log_errot(token, "paxos_acquire host_id %llu read %d",
- (unsigned long long)prev_leader.owner_id,
+ (unsigned long long)cur_leader.owner_id,
error);
goto out;
}
@@ -746,7 +767,7 @@ int paxos_lease_acquire(struct timeout *ti,
if (host_id_leader.timestamp == LEASE_FREE) {
log_token(token, "paxos_acquire host_id %llu free",
- (unsigned long long)prev_leader.owner_id);
+ (unsigned long long)cur_leader.owner_id);
goto run;
}
@@ -754,9 +775,9 @@ int paxos_lease_acquire(struct timeout *ti,
owned this paxos lease; acquiring a host_id also cannot be
done in less than host_id_timeout_sec */
- if (host_id_leader.owner_id != prev_leader.owner_id) {
+ if (host_id_leader.owner_id != cur_leader.owner_id) {
log_token(token, "paxos_acquire host_id %llu owner %llu",
- (unsigned long long)prev_leader.owner_id,
+ (unsigned long long)cur_leader.owner_id,
(unsigned long long)host_id_leader.owner_id);
goto run;
}
@@ -765,12 +786,12 @@ int paxos_lease_acquire(struct timeout *ti,
owned the lease in a previous generation without freeing it,
and no longer owns it */
- if (host_id_leader.owner_generation > prev_leader.owner_generation) {
+ if (host_id_leader.owner_generation > cur_leader.owner_generation) {
log_token(token, "paxos_acquire host_id %llu "
"generation now %llu old %llu",
- (unsigned long long)prev_leader.owner_id,
+ (unsigned long long)cur_leader.owner_id,
(unsigned long long)host_id_leader.owner_generation,
- (unsigned long long)prev_leader.owner_generation);
+ (unsigned long long)cur_leader.owner_generation);
goto run;
}
@@ -789,14 +810,14 @@ int paxos_lease_acquire(struct timeout *ti,
if (time(NULL) - start > ti->host_id_timeout_seconds) {
log_token(token, "paxos_acquire host_id %llu expired %llu",
- (unsigned long long)prev_leader.owner_id,
+ (unsigned long long)cur_leader.owner_id,
(unsigned long long)host_id_leader.timestamp);
goto run;
}
#if 0
if (time(NULL) - host_id_leader.timestamp > ti->host_id_timeout_seconds) {
log_token(token, "paxos_acquire host_id %llu expired %llu",
- (unsigned long long)prev_leader.owner_id,
+ (unsigned long long)cur_leader.owner_id,
(unsigned long long)host_id_leader.timestamp);
goto run;
}
@@ -807,88 +828,162 @@ int paxos_lease_acquire(struct timeout *ti,
if (last_timestamp && (host_id_leader.timestamp != last_timestamp)) {
if (flags & PAXOS_ACQUIRE_QUIET_FAIL) {
log_token(token, "paxos_acquire host_id %llu alive",
- (unsigned long long)prev_leader.owner_id);
+ (unsigned long long)cur_leader.owner_id);
} else {
log_errot(token, "paxos_acquire host_id %llu alive",
- (unsigned long long)prev_leader.owner_id);
+ (unsigned long long)cur_leader.owner_id);
}
- error = SANLK_LIVE_LEADER;
+ error = SANLK_ACQUIRE_IDLIVE;
goto out;
}
last_timestamp = host_id_leader.timestamp;
- sleep(1);
+ sleep(2);
+
+ error = paxos_lease_leader_read(ti, token, &tmp_leader, "paxos_acquire");
+ if (error < 0)
+ goto out;
+
+ if (memcmp(&cur_leader, &tmp_leader, sizeof(struct leader_record))) {
+ log_token(token, "paxos_acquire restart leader changed");
+ goto restart;
+ }
}
run:
- if (acquire_lver && prev_leader.lver != acquire_lver) {
- log_errot(token, "paxos_acquire acquire_lver %llu prev_leader %llu",
- (unsigned long long)acquire_lver,
- (unsigned long long)prev_leader.lver);
- error = SANLK_REACQUIRE_LVER;
+ /*
+ * Use the disk paxos algorithm to attempt to commit a new leader.
+ *
+ * If we complete a ballot successfully, we can commit a leader record
+ * with next_lver. If we find a higher mbal during a ballot, we increase
+ * our own mbal and try the ballot again.
+ *
+ * next_lver is derived from cur_leader with a zero or timed out owner.
+ * We need to monitor the leader record to see if another host commits
+ * a new leader_record with next_lver.
+ */
+
+ next_lver = cur_leader.lver + 1;
+
+ num_reads = 0;
+
+ for (d = 0; d < token->r.num_disks; d++) {
+ rv = read_dblock(ti, &token->disks[d], token->host_id, &dblock);
+ if (rv < 0)
+ continue;
+ num_reads++;
+
+ if (dblock.mbal > our_mbal)
+ our_mbal = dblock.mbal;
+ }
+
+ if (!num_reads) {
+ log_errot(token, "paxos_acquire cannot read our dblock %d", rv);
+ error = SANLK_DBLOCK_READ;
goto out;
}
- /* TODO: test: while we were waiting in host_id_timeout_seconds loop
- * above, another host has finished that loop, come through here
- * and become the new leader (so if we were to read the leader record
- * again right here it would be different from our prev_leader).
- * what if the other host not only acquired the leader but also
- * freed it when we get here? */
+ /* TODO: may not need to increase mbal if dblock.inp and inp2 match
+ current host_id and generation? */
- /*
- * run disk paxos to reach consensus on a new leader
- */
+ if (!our_mbal)
+ our_mbal = token->host_id;
+ else
+ our_mbal += cur_leader.max_hosts;
- memcpy(&new_leader, &prev_leader, sizeof(struct leader_record));
- new_leader.lver += 1; /* req.lver */
+ retry_ballot:
- error = run_disk_paxos(ti, token, token->host_id, token->host_id,
- new_leader.num_hosts, new_leader.lver, &dblock);
- if (error < 0) {
- log_errot(token, "paxos_acquire paxos error %d", error);
+ error = paxos_lease_leader_read(ti, token, &tmp_leader, "paxos_acquire");
+ if (error < 0)
+ goto out;
+
+ if (tmp_leader.lver == next_lver) {
+ /*
+ * another host has commited a leader_record for next_lver,
+ * check which inp (owner_id) they commited (possibly us).
+ */
+
+ if (tmp_leader.owner_id == token->host_id &&
+ tmp_leader.owner_generation == token->host_generation) {
+ /* not a problem, but interesting to see, so use log_error */
+
+ log_errot(token, "paxos_acquire %llu our id commited by %llu",
+ (unsigned long long)next_lver,
+ (unsigned long long)tmp_leader.write_id);
+
+ memcpy(leader_ret, &tmp_leader, sizeof(struct leader_record));
+ error = SANLK_OK;
+ } else {
+ /* not a problem, but interesting to see, so use log_error */
+
+ log_errot(token, "paxos_acquire %llu owner is %llu",
+ (unsigned long long)next_lver,
+ (unsigned long long)tmp_leader.owner_id);
+
+ error = SANLK_ACQUIRE_OWNED;
+ }
goto out;
}
- log_token(token, "paxos_acquire paxos result dblock mbal %llu bal %llu inp %llu lver %llu",
- (unsigned long long)dblock.mbal,
- (unsigned long long)dblock.bal,
- (unsigned long long)dblock.inp,
- (unsigned long long)dblock.lver);
+ error = run_ballot(ti, token, cur_leader.num_hosts, next_lver, our_mbal,
+ &dblock);
- /* the inp value we commited wasn't us */
+ if (error == SANLK_DBLOCK_MBAL) {
+ log_token(token, "paxos_acquire %llu retry ballot",
+ (unsigned long long)next_lver);
+ our_mbal += cur_leader.max_hosts;
+ goto retry_ballot;
+ }
- if (dblock.inp != token->host_id) {
- log_errot(token, "paxos_acquire paxos contention our_host_id %llu "
- "mbal %llu bal %llu inp %llu lver %llu",
- (unsigned long long)token->host_id,
- (unsigned long long)dblock.mbal,
- (unsigned long long)dblock.bal,
- (unsigned long long)dblock.inp,
- (unsigned long long)dblock.lver);
- error = SANLK_OTHER_INP;
+ if (error < 0) {
+ log_errot(token, "paxos_acquire %llu ballot error %d",
+ (unsigned long long)next_lver, error);
goto out;
}
- /* dblock has the disk paxos result: consensus inp and lver */
+ /* ballot success, commit next_lver with dblock values */
- new_leader.owner_id = token->host_id;
- new_leader.owner_generation = token->host_generation;
+ memcpy(&new_leader, &cur_leader, sizeof(struct leader_record));
new_leader.lver = dblock.lver;
- new_leader.timestamp = time(NULL);
+ new_leader.owner_id = dblock.inp;
+ new_leader.owner_generation = dblock.inp2;
+ new_leader.timestamp = dblock.inp3;
+
+ new_leader.write_id = token->host_id;
+ new_leader.write_generation = token->host_generation;
+ new_leader.write_timestamp = time(NULL);
+
if (new_num_hosts)
new_leader.num_hosts = new_num_hosts;
new_leader.checksum = leader_checksum(&new_leader);
- error = write_new_leader(ti, token, &new_leader);
+ error = write_new_leader(ti, token, &new_leader, "paxos_acquire");
if (error < 0)
goto out;
+ if (new_leader.owner_id != token->host_id) {
+ /* not a problem, but interesting to see, so use log_error */
+
+ log_errot(token, "paxos_acquire %llu commit other owner %llu %llu %llu",
+ (unsigned long long)new_leader.lver,
+ (unsigned long long)new_leader.owner_id,
+ (unsigned long long)new_leader.owner_generation,
+ (unsigned long long)new_leader.timestamp);
+
+ error = SANLK_ACQUIRE_OTHER;
+ goto out;
+ }
+
+ log_token(token, "paxos_acquire %llu owner %llu %llu %llu done",
+ (unsigned long long)next_lver,
+ (unsigned long long)new_leader.owner_id,
+ (unsigned long long)new_leader.owner_generation,
+ (unsigned long long)new_leader.timestamp);
+
memcpy(leader_ret, &new_leader, sizeof(struct leader_record));
+ error = SANLK_OK;
out:
- log_token(token, "paxos_acquire done %d", error);
-
if (disk_open)
close_disks(&host_id_disk, 1);
@@ -964,21 +1059,57 @@ int paxos_lease_release(struct timeout *ti,
goto out;
}
- if (memcmp(&leader, leader_last, sizeof(struct leader_record))) {
- log_errot(token, "release error leader changed");
- return SANLK_BAD_LEADER;
+ if (leader.lver != leader_last->lver) {
+ log_errot(token, "paxos_release %llu other lver %llu",
+ (unsigned long long)leader_last->lver,
+ (unsigned long long)leader.lver);
+ return SANLK_RELEASE_LVER;
}
- if (leader.owner_id != token->host_id) {
- log_errot(token, "release error other owner_id %llu",
- (unsigned long long)leader.owner_id);
- return SANLK_OTHER_OWNER;
+ if (leader.owner_id != token->host_id ||
+ leader.owner_generation != token->host_generation) {
+ log_errot(token, "paxos_release %llu other owner %llu %llu %llu",
+ (unsigned long long)leader_last->lver,
+ (unsigned long long)leader.owner_id,
+ (unsigned long long)leader.owner_generation,
+ (unsigned long long)leader.timestamp);
+ return SANLK_RELEASE_OWNER;
+ }
+
+ if (memcmp(&leader, leader_last, sizeof(struct leader_record))) {
+ /*
+ * This will happen when two hosts finish the same ballot
+ * successfully, the second commiting the same inp values
+ * that the first did, as it should. But the second will
+ * write it's own write_id/gen/timestap, which will differ
+ * from what the first host wrote. So when the first host
+ * rereads here in the release, it will find different
+ * write_id/gen/timestamp from what it wrote. This is
+ * perfectly fine (use log_error since it's interesting
+ * to see when this happens.)
+ */
+ log_errot(token, "paxos_release %llu leader different "
+ "write %llu %llu %llu vs %llu %llu %llu",
+ (unsigned long long)leader_last->lver,
+ (unsigned long long)leader_last->write_id,
+ (unsigned long long)leader_last->write_generation,
+ (unsigned long long)leader_last->write_timestamp,
+ (unsigned long long)leader.write_id,
+ (unsigned long long)leader.write_generation,
+ (unsigned long long)leader.write_timestamp);
+ /*
+ log_leader_error(0, token, &token->disks[0], leader_last, "paxos_release");
+ log_leader_error(0, token, &token->disks[0], &leader, "paxos_release");
+ */
}
leader.timestamp = LEASE_FREE;
+ leader.write_id = token->host_id;
+ leader.write_generation = token->host_generation;
+ leader.write_timestamp = time(NULL);
leader.checksum = leader_checksum(&leader);
- error = write_new_leader(ti, token, &leader);
+ error = write_new_leader(ti, token, &leader, "paxos_release");
if (error < 0)
goto out;
diff --git a/src/sanlock_rv.h b/src/sanlock_rv.h
index 614e08c..85db9c3 100644
--- a/src/sanlock_rv.h
+++ b/src/sanlock_rv.h
@@ -12,35 +12,42 @@
#define SANLK_OK 1
#define SANLK_NONE 0 /* unused */
#define SANLK_ERROR -201
-#define SANLK_INVAL -202
-#define SANLK_NOMEM -203
-#define SANLK_LIVE_LEADER -204
-#define SANLK_DIFF_LEADERS -205
-#define SANLK_READ_LEADERS -206
-#define SANLK_OWN_DBLOCK -207
-#define SANLK_WRITE1_DBLOCKS -208
-#define SANLK_WRITE2_DBLOCKS -209
-#define SANLK_WRITE_REQUESTS -210
-#define SANLK_WRITE_LEADERS -211
-#define SANLK_READ1_MBAL -212
-#define SANLK_READ1_LVER -213
-#define SANLK_READ2_MBAL -214
-#define SANLK_READ2_LVER -215
-#define SANLK_READ1_DBLOCKS -216
-#define SANLK_READ2_DBLOCKS -217
-#define SANLK_BAD_MAGIC -218
-#define SANLK_BAD_VERSION -219
-#define SANLK_BAD_CLUSTERMODE -220
-#define SANLK_BAD_RESOURCEID -221
-#define SANLK_BAD_NUMHOSTS -222
-#define SANLK_BAD_CHECKSUM -223
-#define SANLK_BAD_LEADER -224
-#define SANLK_OTHER_INP -225
-#define SANLK_BAD_SECTORSIZE -226
-#define SANLK_REACQUIRE_LVER -227
-#define SANLK_BAD_LOCKSPACE -228
-#define SANLK_OTHER_OWNER -229
-#define SANLK_BAD_SPACE_NAME -230
-#define SANLK_BAD_SPACE_DISK -231
+
+/* run_ballot */
+
+#define SANLK_DBLOCK_READ -210
+#define SANLK_DBLOCK_WRITE -211
+#define SANLK_DBLOCK_LVER -212
+#define SANLK_DBLOCK_MBAL -213
+
+/* verify_leader, leader_read, leader_write (paxos or delta) */
+
+#define SANLK_LEADER_READ -220
+#define SANLK_LEADER_WRITE -221
+#define SANLK_LEADER_DIFF -222
+#define SANLK_LEADER_MAGIC -223
+#define SANLK_LEADER_VERSION -224
+#define SANLK_LEADER_SECTORSIZE -225
+#define SANLK_LEADER_LOCKSPACE -226
+#define SANLK_LEADER_RESOURCE -227
+#define SANLK_LEADER_NUMHOSTS -228
+#define SANLK_LEADER_CHECKSUM -229
+
+/* paxos_lease_acquire, paxos_lease_release */
+
+#define SANLK_ACQUIRE_LVER -240
+#define SANLK_ACQUIRE_LOCKSPACE -241
+#define SANLK_ACQUIRE_IDDISK -242
+#define SANLK_ACQUIRE_IDLIVE -243
+#define SANLK_ACQUIRE_OWNED -244
+#define SANLK_ACQUIRE_OTHER -245
+
+#define SANLK_RELEASE_LVER -250
+#define SANLK_RELEASE_OWNER -251
+
+/* delta_lease_renew */
+
+#define SANLK_RENEW_OWNER -260
+#define SANLK_RENEW_DIFF -261
#endif
12 years, 11 months