A host can send a predefined message code to another host.

The destination host is identified by host_id and generation number. (These are typically 64 bit values, but the host message system uses only the lower 32 bits of these values because of space limitations in the leader record, so host messages should not be used with host_ids or generation numbers larger than 32 bits.)

The message code is combined with a sequence number used to identify the message. If the caller does not provide a sequence number (uses 0), sanlock will generate a sequence number using a local counter, and return this number to the caller.

The host messages are sent from one host to another via a lockspace that both hosts are using. The lockspace used to transmit the message may or may not have any other relation to the message itself. The message data is included in the leader record for each host's delta lease.

The message code and sequence number are placed in the sending host's delta lease, and remain there until replaced with another. The caller must take care to not overwrite one message with another before the first has been written.

A message is written by the next scheduled delta lease renewal.

When the receiving host next renews its own delta lease, it reads the delta leases of all other hosts, and sees itself addressed for the message in the sending host's lease. It then processes the message from the sending host. It will only process the message if the sequence number is larger than the previously seen sequence number from the same host.

When the destination host reads a message from the source host, it saves the source host_id and the sequence number of the message. This host_id and seq number are included in the destination host's next delta lease renewal. This can act as an acknowlegement that that destination has seen the message. The sending host can monitor the destination host's delta lease to see if the message has been received.

The sanlock_read_lockspace_message() function reads the delta lease of any host and returns the last host message that the given host has sent, and the last host_id/seq that the given host last received.

Host messages sent around the same time, from the same host, using the same lockspace may clobber each other and cause earlier messages to not be written/sent.

Host messages sent around the same time from different hosts to the same destination host may cause earlier acknowledgement records from the destination to clobber each other and not be written/sent.

The host_message msg code is a bitfield, so multiple messages can be combined.

msg code SEQ_DATA means that the sequence number is treated as opaque caller-specific data. These messages are not acked by the receiving host.

msg code WD_RESET means that the host receiving the message should use its watchdog device to reset itself as soon as possible. The sanlock daemon uses the watchdog to do this. The WD_RESET message has no effect on any lockspaces or resources that may exist. Existing lockspaces and resources continue to operate as usual until the reset. (A watchdog reset due to "standard" lockspace failure could in fact occur before the watchdog reset caused by the host message.)

Because host messages may not be received if the destination host fails, or loses storage access, there are no guaranteed times associated with the delivery, processing or effect of a host message. Guaranteed times for another host being dead should continue to be based on either acquiring a resource, or sanlock_get_hosts().

Signed-off-by: David Teigland teigland@redhat.com --- src/Makefile | 1 + src/client.c | 142 ++++++++++++++++++++++++++++++++++++++++++++++++ src/cmd.c | 49 ++++++++++++++++- src/delta_lease.c | 20 ++++++- src/delta_lease.h | 4 +- src/direct.c | 1 + src/leader.h | 2 +- src/lockspace.c | 143 +++++++++++++++++++++++++++++++++++++++++++++++-- src/lockspace.h | 3 ++ src/main.c | 68 +++++++++++++++++++---- src/paxos_lease.c | 2 + src/resource.c | 61 +++++++++++++++++++++ src/resource.h | 3 ++ src/sanlock.8 | 36 +++++++++++-- src/sanlock.h | 14 +++++ src/sanlock_admin.h | 41 ++++++++++++++ src/sanlock_internal.h | 20 +++++-- src/sanlock_sock.h | 2 + src/watchdog.c | 65 ++++++++++++++++++++++ src/watchdog.h | 1 + 20 files changed, 653 insertions(+), 25 deletions(-)

diff --git a/src/Makefile b/src/Makefile index dc5668aed57d..5bb9e560521c 100644 --- a/src/Makefile +++ b/src/Makefile @@ -76,6 +76,7 @@ CFLAGS += -D_GNU_SOURCE \ -Winline \ -Wredundant-decls \ -Wno-sign-compare \ + -Wno-strict-overflow \ -fexceptions \ -fasynchronous-unwind-tables \ -fdiagnostics-show-option \ diff --git a/src/client.c b/src/client.c index 9e523aea68be..01d56b22706d 100644 --- a/src/client.c +++ b/src/client.c @@ -408,6 +408,80 @@ int sanlock_read_lockspace(struct sanlk_lockspace *ls, uint32_t flags, uint32_t return rv; }

+int sanlock_read_lockspace_message(struct sanlk_lockspace *ls, uint32_t flags, + uint32_t *io_timeout, + struct sanlk_host_message *hm) +{ + struct sm_header h; + int rv, fd; + + if (!ls || !ls->host_id_disk.path[0]) + return -EINVAL; + + rv = connect_socket(&fd); + if (rv < 0) + return rv; + + rv = send_header(fd, SM_CMD_READ_LOCKSPACE_MESSAGE, flags, + sizeof(struct sanlk_lockspace), + 0, 0); + if (rv < 0) + goto out; + + rv = send(fd, ls, sizeof(struct sanlk_lockspace), 0); + if (rv < 0) { + rv = -errno; + goto out; + } + + /* receive result, io_timeout and ls struct */ + + memset(&h, 0, sizeof(struct sm_header)); + + rv = recv(fd, &h, sizeof(h), MSG_WAITALL); + if (rv < 0) { + rv = -errno; + goto out; + } + + if (rv != sizeof(h)) { + rv = -1; + goto out; + } + + rv = (int)h.data; + if (rv < 0) + goto out; + + rv = recv(fd, ls, sizeof(struct sanlk_lockspace), MSG_WAITALL); + if (rv < 0) { + rv = -errno; + goto out; + } + + if (rv != sizeof(struct sanlk_lockspace)) { + rv = -1; + goto out; + } + + rv = recv(fd, hm, sizeof(struct sanlk_host_message), MSG_WAITALL); + if (rv < 0) { + rv = -errno; + goto out; + } + + if (rv != sizeof(struct sanlk_host_message)) { + rv = -1; + goto out; + } + + *io_timeout = h.data2; + rv = (int)h.data; + out: + close(fd); + return rv; +} + int sanlock_read_resource(struct sanlk_resource *res, uint32_t flags) { struct sm_header h; @@ -702,6 +776,74 @@ int sanlock_test_resource_owners(struct sanlk_resource *res GNUC_UNUSED, return 0; }

+int sanlock_set_message(const char *ls_name, uint32_t flags, + int hm_size, struct sanlk_host_message *hm) +{ + struct sanlk_lockspace ls; + struct sanlk_host_message msg; + struct sm_header h; + int rv, fd; + + if (!ls_name || !hm || !hm_size) + return -EINVAL; + + memset(&ls, 0, sizeof(struct sanlk_lockspace)); + strncpy(ls.name, ls_name, SANLK_NAME_LEN); + + rv = connect_socket(&fd); + if (rv < 0) + return rv; + + memset(&msg, 0, sizeof(msg)); + + if (hm_size < sizeof(msg)) + memcpy(&msg, hm, hm_size); + else if (hm_size > sizeof(msg)) + memcpy(&msg, hm, sizeof(msg)); + else + memcpy(&msg, hm, sizeof(msg)); + + rv = send_header(fd, SM_CMD_SET_MESSAGE, flags, + sizeof(struct sanlk_lockspace) + sizeof(struct sanlk_host_message), + 0, 0); + if (rv < 0) + goto out; + + rv = send(fd, &ls, sizeof(struct sanlk_lockspace), 0); + if (rv < 0) { + rv = -errno; + goto out; + } + + rv = send(fd, &msg, sizeof(struct sanlk_host_message), 0); + if (rv < 0) { + rv = -errno; + goto out; + } + + memset(&h, 0, sizeof(struct sm_header)); + + rv = recv(fd, &h, sizeof(h), MSG_WAITALL); + if (rv < 0) { + rv = -errno; + goto out; + } + + if (rv != sizeof(h)) { + rv = -1; + goto out; + } + + rv = (int)h.data; + if (rv < 0) + goto out; + + hm->seq = h.data2; + out: + close(fd); + return rv; +} + /* old api */ int sanlock_init(struct sanlk_lockspace *ls, struct sanlk_resource *res, diff --git a/src/cmd.c b/src/cmd.c index f19e8fb91c01..727d782e4764 100644 --- a/src/cmd.c +++ b/src/cmd.c @@ -1159,6 +1159,40 @@ static void cmd_get_lvb(struct task *task GNUC_UNUSED, struct cmd_args *ca) client_resume(ca->ci_in); }

+static void cmd_set_message(struct task *task GNUC_UNUSED, struct cmd_args *ca) +{ + struct sanlk_lockspace lockspace; + struct sanlk_host_message hm; + struct sm_header h; + int rv, fd, result; + + fd = client[ca->ci_in].fd; + + rv = recv(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); + if (rv != sizeof(struct sanlk_lockspace)) { + result = -ENOTCONN; + goto reply; + } + + rv = recv(fd, &hm, sizeof(struct sanlk_host_message), MSG_WAITALL); + if (rv != sizeof(struct sanlk_host_message)) { + result = -ENOTCONN; + goto reply; + } + + result = set_lockspace_message(&lockspace, &hm); + reply: + memcpy(&h, &ca->header, sizeof(struct sm_header)); + h.version = SM_PROTO; + h.data = result; + h.data2 = hm.seq; + h.length = sizeof(h); + + send(fd, &h, sizeof(h), MSG_NOSIGNAL); + + client_resume(ca->ci_in); +} + static void cmd_add_lockspace(struct cmd_args *ca) { struct sanlk_lockspace lockspace; @@ -1389,11 +1423,16 @@ static void cmd_read_lockspace(struct task *task, struct cmd_args *ca) { struct sm_header h; struct sanlk_lockspace lockspace; + struct sanlk_host_message hm; + struct delta_extra extra; struct sync_disk sd; uint64_t host_id; int io_timeout = 0; int fd, rv, result;

+ memset(&hm, 0, sizeof(hm)); + memset(&extra , 0, sizeof(extra)); + fd = client[ca->ci_in].fd;

rv = recv(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); @@ -1432,11 +1471,13 @@ static void cmd_read_lockspace(struct task *task, struct cmd_args *ca)

/* sets ls->name and io_timeout */ result = delta_read_lockspace(task, &sd, host_id, &lockspace, - DEFAULT_IO_TIMEOUT, &io_timeout); + DEFAULT_IO_TIMEOUT, &io_timeout, &extra); if (result == SANLK_OK) result = 0;

close_disks(&sd, 1); + + host_message_from_extra(&hm, &extra); reply: log_debug("cmd_read_lockspace %d,%d done %d", ca->ci_in, fd, result);

@@ -1447,6 +1488,8 @@ static void cmd_read_lockspace(struct task *task, struct cmd_args *ca) h.length = sizeof(h) + sizeof(lockspace); send(fd, &h, sizeof(h), MSG_NOSIGNAL); send(fd, &lockspace, sizeof(lockspace), MSG_NOSIGNAL); + if (ca->header.cmd == SM_CMD_READ_LOCKSPACE_MESSAGE) + send(fd, &hm, sizeof(hm), MSG_NOSIGNAL); client_resume(ca->ci_in); }

@@ -1887,6 +1930,7 @@ void call_cmd_thread(struct task *task, struct cmd_args *ca) cmd_write_resource(task, ca); break; case SM_CMD_READ_LOCKSPACE: + case SM_CMD_READ_LOCKSPACE_MESSAGE: cmd_read_lockspace(task, ca); break; case SM_CMD_READ_RESOURCE: @@ -1908,6 +1952,9 @@ void call_cmd_thread(struct task *task, struct cmd_args *ca) case SM_CMD_GET_LVB: cmd_get_lvb(task, ca); break; + case SM_CMD_SET_MESSAGE: + cmd_set_message(task, ca); + break; }; }

diff --git a/src/delta_lease.c b/src/delta_lease.c index 91d0a83bc811..9fd2235b66c7 100644 --- a/src/delta_lease.c +++ b/src/delta_lease.c @@ -156,7 +156,8 @@ int delta_read_lockspace(struct task *task, uint64_t host_id, struct sanlk_lockspace *ls, int io_timeout, - int *io_timeout_ret) + int *io_timeout_ret, + struct delta_extra *extra) { struct leader_record leader; char *space_name; @@ -182,6 +183,12 @@ int delta_read_lockspace(struct task *task, memcpy(ls->name, leader.space_name, SANLK_NAME_LEN); ls->host_id = host_id; *io_timeout_ret = leader.io_timeout; + + if (extra) { + extra->field1 = leader.write_id; + extra->field2 = leader.write_generation; + extra->field3 = leader.write_timestamp; + } }

return error; @@ -352,6 +359,10 @@ int delta_lease_acquire(struct task *task, snprintf(leader.resource_name, NAME_ID_SIZE, "%s", our_host_name); leader.checksum = leader_checksum(&leader);

+ leader.write_id = 0; + leader.write_generation = 0; + leader.write_timestamp = 0; + log_space(sp, "delta_acquire write %llu %llu %llu %.48s", (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, @@ -410,6 +421,7 @@ int delta_lease_renew(struct task *task, struct sync_disk *disk, char *space_name, char *bitmap, + struct delta_extra *extra, int prev_result, int *read_result, struct leader_record *leader_last, @@ -567,6 +579,12 @@ int delta_lease_renew(struct task *task, leader.timestamp = new_ts; leader.checksum = leader_checksum(&leader);

+ if (extra) { + leader.write_id = extra->field1; + leader.write_generation = extra->field2; + leader.write_timestamp = extra->field3; + } + p_wbuf = &wbuf; rv = posix_memalign((void *)p_wbuf, getpagesize(), sector_size); if (rv) { diff --git a/src/delta_lease.h b/src/delta_lease.h index f015d1eecb23..f663a4005890 100644 --- a/src/delta_lease.h +++ b/src/delta_lease.h @@ -30,6 +30,7 @@ int delta_lease_renew(struct task *task, struct sync_disk *disk, char *space_name, char *bitmap, + struct delta_extra *extra, int prev_result, int *read_result, struct leader_record *leader_last, @@ -53,6 +54,7 @@ int delta_read_lockspace(struct task *task, uint64_t host_id, struct sanlk_lockspace *ls, int io_timeout, - int *io_timeout_ret); + int *io_timeout_ret, + struct delta_extra *extra);

#endif diff --git a/src/direct.c b/src/direct.c index ed6cca458364..3989da542877 100644 --- a/src/direct.c +++ b/src/direct.c @@ -240,6 +240,7 @@ static int do_delta_action(int action, rv = delta_lease_renew(task, &space, &sd, ls->name, bitmap, + NULL, -1, &read_result, &leader, diff --git a/src/leader.h b/src/leader.h index 1321f72fbf0d..16a52f32af32 100644 --- a/src/leader.h +++ b/src/leader.h @@ -21,7 +21,7 @@

#define DELTA_DISK_MAGIC 0x12212010 #define DELTA_DISK_VERSION_MAJOR 0x00030000 -#define DELTA_DISK_VERSION_MINOR 0x00000002 +#define DELTA_DISK_VERSION_MINOR 0x00000003

/* for all disk structures: uint64 aligned on 8 byte boundaries, diff --git a/src/lockspace.c b/src/lockspace.c index e96e1e1e7983..c57497fa4a03 100644 --- a/src/lockspace.c +++ b/src/lockspace.c @@ -222,8 +222,11 @@ int host_info(char *space_name, uint64_t host_id, struct host_status *hs_out) return 0; }

-static void create_bitmap(struct space *sp, char *bitmap) +static void create_bitmap_and_extra(struct space *sp, char *bitmap, + struct delta_extra *extra) { + struct sanlk_host_message *hm; + uint64_t extra1 = 0, extra2 = 0, extra3 = 0; uint64_t now; int i; char c; @@ -247,14 +250,50 @@ static void create_bitmap(struct space *sp, char *bitmap) log_space(sp, "bitmap set host_id %d byte %x", i+1, c); } } + + if (sp->host_message_send.host_id) { + hm = &sp->host_message_send; + extra1 = hm->host_id & 0xFFFFFFFF; + extra1 = (extra1 << 32) | (hm->generation & 0xFFFFFFFF); + extra2 = hm->msg; + extra2 = (extra2 << 32) | hm->seq; + } + + /* + * Including the last received message causes us to skip replies + * when multiple hosts send us a message within a single interval. + * To ensure acks to all, we would need to keep a list of received + * messages, and take the next one off the list here. + */ + + extra3 = sp->host_message_recv_from & 0xFFFFFFFF; + extra3 = (extra3 << 32) | sp->host_message_recv_seq; + + extra->field1 = extra1; + extra->field2 = extra2; + extra->field3 = extra3; + pthread_mutex_unlock(&sp->mutex); }

+void host_message_from_extra(struct sanlk_host_message *hm, + struct delta_extra *extra) +{ + hm->host_id = extra->field1 >> 32; + hm->generation = extra->field1 & 0xFFFFFFFF; + hm->msg = extra->field2 >> 32; + hm->seq = extra->field2 & 0xFFFFFFFF; + hm->ack_host = extra->field3 >> 32; + hm->ack_seq = extra->field3 & 0xFFFFFFFF; +} + void check_other_leases(struct space *sp, char *buf) { struct leader_record *leader; struct sync_disk *disk; struct host_status *hs; + struct delta_extra extra; + struct sanlk_host_message hm; char *bitmap; uint64_t now; int i, new; @@ -280,15 +319,66 @@ void check_other_leases(struct space *sp, char *buf) continue; }

+ pthread_mutex_lock(&sp->mutex); hs->owner_id = leader->owner_id; hs->owner_generation = leader->owner_generation; hs->timestamp = leader->timestamp; hs->io_timeout = leader->io_timeout; hs->last_live = now; + pthread_mutex_unlock(&sp->mutex);

if (i+1 == sp->host_id) continue;

+ /* + * check if we are target of message + */ + + extra.field1 = leader->write_id; + extra.field2 = leader->write_generation; + extra.field3 = leader->write_timestamp; + + if (!extra.field1) + goto check_bitmap; + + host_message_from_extra(&hm, &extra); + + if (hm.host_id != sp->host_id) + goto check_bitmap; + + if (hm.generation != sp->host_generation) + goto check_bitmap; + + if (!hm.msg && !hm.seq) + goto check_bitmap; + + if (!(hm.msg & SANLK_HM_SEQ_DATA) && + (hm.seq <= hs->host_message_recv_seq)) + goto check_bitmap; + + if (!(hm.msg & SANLK_HM_SEQ_DATA)) { + hs->host_message_recv_seq = hm.seq; + + pthread_mutex_lock(&sp->mutex); + sp->host_message_recv_from = i+1; + sp->host_message_recv_seq = hm.seq; + pthread_mutex_unlock(&sp->mutex); + } + + log_level(sp->space_id, 0, NULL, LOG_WARNING, + "host message from %llu %llu msg 0x%08x seq 0x%08x", + (unsigned long long)hs->owner_id, + (unsigned long long)hs->owner_generation, + hm.msg, hm.seq); + + set_message_examine(sp->space_name, &hm, + hs->owner_id, hs->owner_generation); + + + /* + * check if we are notified in bitmap + */ + check_bitmap: bitmap = (char *)leader + HOSTID_BITMAP_OFFSET;

if (!test_id_bit(sp->host_id, bitmap)) @@ -385,6 +475,7 @@ static int corrupt_result(int result) static void *lockspace_thread(void *arg_in) { char bitmap[HOSTID_BITMAP_SIZE]; + struct delta_extra extra; struct task task; struct space *sp; struct leader_record leader; @@ -502,12 +593,14 @@ static void *lockspace_thread(void *arg_in) */

memset(bitmap, 0, sizeof(bitmap)); - create_bitmap(sp, bitmap); + memset(&extra, 0, sizeof(extra)); + create_bitmap_and_extra(sp, bitmap, &extra);

delta_begin = monotime();

delta_result = delta_lease_renew(&task, sp, &sp->host_id_disk, - sp->space_name, bitmap, + sp->space_name, + bitmap, &extra, delta_result, &read_result, &leader, &leader); delta_length = monotime() - delta_begin; @@ -1109,6 +1202,50 @@ int get_hosts(struct sanlk_lockspace *ls, char *buf, int *len, int *count, int m return rv; }

+int set_lockspace_message(struct sanlk_lockspace *ls, struct sanlk_host_message *hm) +{ + struct space *sp; + struct host_status *hs; + + if (!hm->host_id) + return -EINVAL; + + if (!ls->name[0]) + return -EINVAL; + + pthread_mutex_lock(&spaces_mutex); + sp = _search_space(ls->name, NULL, 0, &spaces, NULL, NULL, NULL); + if (!sp) { + pthread_mutex_unlock(&spaces_mutex); + return -ENOENT; + } + pthread_mutex_unlock(&spaces_mutex); + + pthread_mutex_lock(&sp->mutex); + + /* supply the current generation if caller did not specify the generation */ + if (!hm->generation) { + hs = &(sp->host_status[hm->host_id-1]); + hm->generation = hs->owner_generation; + } + + if (!hm->msg && !hm->seq) { + /* clear message to this node */ + } else if (!(hm->msg & SANLK_HM_SEQ_DATA) && !hm->seq) + hm->seq = ++sp->host_message_send_seq; + + memcpy(&sp->host_message_send, hm, sizeof(struct sanlk_host_message)); + + pthread_mutex_unlock(&sp->mutex); + + log_erros(sp, "host message send %llu %llu msg 0x%08x seq 0x%08x", + (unsigned long long)hm->host_id, + (unsigned long long)hm->generation, + hm->msg, hm->seq); + + return 0; +} + /* * we call stop_host_id() when all pids are gone and we're in a safe state, so * it's safe to unlink the watchdog right away here. We want to sp the unlink diff --git a/src/lockspace.h b/src/lockspace.h index 3ffe17ffc048..0189d00fd524 100644 --- a/src/lockspace.h +++ b/src/lockspace.h @@ -27,5 +27,8 @@ int rem_lockspace_wait(struct sanlk_lockspace *ls, unsigned int space_id); void free_lockspaces(int wait); int get_lockspaces(char *buf, int *len, int *count, int maxlen); int get_hosts(struct sanlk_lockspace *ls, char *buf, int *len, int *count, int maxlen); +void host_message_from_extra(struct sanlk_host_message *hm, struct delta_extra *extra); +int set_lockspace_message(struct sanlk_lockspace *ls, struct sanlk_host_message *hm); +

#endif diff --git a/src/main.c b/src/main.c index 5f3c6edd6d56..a693652a85fc 100644 --- a/src/main.c +++ b/src/main.c @@ -1190,10 +1190,12 @@ static void process_connection(int ci) case SM_CMD_WRITE_LOCKSPACE: case SM_CMD_WRITE_RESOURCE: case SM_CMD_READ_LOCKSPACE: + case SM_CMD_READ_LOCKSPACE_MESSAGE: case SM_CMD_READ_RESOURCE: case SM_CMD_READ_RESOURCE_OWNERS: case SM_CMD_SET_LVB: case SM_CMD_GET_LVB: + case SM_CMD_SET_MESSAGE: rv = client_suspend(ci); if (rv < 0) return; @@ -1806,13 +1808,15 @@ static void print_usage(void) printf(" -w 0|1 use watchdog through wdmd (%d)\n", DEFAULT_USE_WATCHDOG); printf(" -h 0|1 use high priority (RR) scheduling (%d)\n", DEFAULT_HIGH_PRIORITY); printf(" -l <num> use mlockall (0 none, 1 current, 2 current and future) (%d)\n", DEFAULT_MLOCK_LEVEL); - printf(" -a 0|1 use async io (%d)\n", DEFAULT_USE_AIO); + /* non-aio is untested and may not work */ + /* printf(" -a 0|1 use async io (%d)\n", DEFAULT_USE_AIO); */ printf(" -o 0|1 io timeout in seconds (%d)\n", DEFAULT_IO_TIMEOUT); printf("\n"); printf("sanlock client <action> [options]\n"); printf("sanlock client status [-D] [-o p|s]\n"); printf("sanlock client gets [-h 0|1]\n"); printf("sanlock client host_status -s LOCKSPACE [-D]\n"); + printf("sanlock client set_message -s LOCKSPACE -i <host_id> [-g gen] -m <msg> [-n <seq>]\n"); printf("sanlock client log_dump\n"); printf("sanlock client shutdown [-f 0|1]\n"); printf("sanlock client init -s LOCKSPACE | -r RESOURCE\n"); @@ -1955,6 +1959,8 @@ static int read_command_line(int argc, char *argv[]) com.action = ACT_CLIENT_READ; else if (!strcmp(act, "version")) com.action = ACT_VERSION; + else if (!strcmp(act, "set_message")) + com.action = ACT_SET_MESSAGE; else { log_tool("client action "%s" is unknown", act); exit(EXIT_FAILURE); @@ -2068,9 +2074,11 @@ static int read_command_line(int argc, char *argv[]) break; case 'n': com.num_hosts = atoi(optionarg); + com.hm_seq = strtoul(optionarg, NULL, 0); break; case 'm': com.max_hosts = atoi(optionarg); + com.hm_msg = strtoul(optionarg, NULL, 0); break; case 'p': com.pid = atoi(optionarg); @@ -2079,7 +2087,7 @@ static int read_command_line(int argc, char *argv[]) strncpy(com.our_host_name, optionarg, NAME_ID_SIZE); break; case 'i': - com.local_host_id = atoll(optionarg); + com.host_id = atoll(optionarg); break; case 'g': if (com.type == COM_DAEMON) { @@ -2087,7 +2095,7 @@ static int read_command_line(int argc, char *argv[]) if (sec <= 60 && sec >= 0) kill_grace_seconds = sec; } else { - com.local_host_generation = atoll(optionarg); + com.host_generation = atoll(optionarg); } break; case 'f': @@ -2267,13 +2275,18 @@ static int do_client_gets(void)

static int do_client_read(void) { + struct sanlk_host_message hm; struct sanlk_host *hss = NULL, *hs; char *res_str = NULL; uint32_t io_timeout = 0; int rv, i, hss_count = 0;

if (com.lockspace.host_id_disk.path[0]) { - rv = sanlock_read_lockspace(&com.lockspace, 0, &io_timeout); + if (!com.get_hosts) { + rv = sanlock_read_lockspace(&com.lockspace, 0, &io_timeout); + } else { + rv = sanlock_read_lockspace_message(&com.lockspace, 0, &io_timeout, &hm); + } } else { if (!com.get_hosts) { rv = sanlock_read_resource(com.res_args[0], 0); @@ -2295,6 +2308,15 @@ static int do_client_read(void) com.lockspace.host_id_disk.path, (unsigned long long)com.lockspace.host_id_disk.offset); log_tool("io_timeout %u", io_timeout); + + if (com.get_hosts) { + log_tool("message host_id %llu gen %llu msg 0x%08x seq 0x%08x", + (unsigned long long)hm.host_id, + (unsigned long long)hm.generation, + hm.msg, hm.seq); + log_tool("ack host_id %llu seq 0x%08x", + (unsigned long long)hm.ack_host, hm.ack_seq); + } goto out; }

@@ -2365,6 +2387,7 @@ static void do_client_version(void)

static int do_client(void) { + struct sanlk_host_message hm; struct sanlk_resource **res_args = NULL; struct sanlk_resource *res; char *res_state = NULL; @@ -2551,6 +2574,19 @@ static int do_client(void) do_client_version(); break;

+ case ACT_SET_MESSAGE: + log_tool("set_message %llu %llu msg %u seq %u", + (unsigned long long)com.host_id, + (unsigned long long)com.host_generation, + com.hm_msg, com.hm_seq); + hm.host_id = com.host_id; + hm.generation = com.host_generation; + hm.msg = com.hm_msg; + hm.seq = com.hm_seq; + rv = sanlock_set_message(com.lockspace.name, 0, sizeof(struct sanlk_host_message), &hm); + log_tool("set_message seq %u done %d", hm.seq, rv); + break; + default: log_tool("action not implemented"); rv = -1; @@ -2611,18 +2647,28 @@ static int do_direct(void) (unsigned long long)leader.timestamp); log_tool("checksum 0x%0x", leader.checksum); log_tool("io_timeout %u", leader.io_timeout); - log_tool("write_id %llu", - (unsigned long long)leader.write_id); - log_tool("write_generation %llu", - (unsigned long long)leader.write_generation); - log_tool("write_timestamp %llu", - (unsigned long long)leader.write_timestamp); + + if (leader.magic == DELTA_DISK_MAGIC) { + log_tool("extra1 0x%016llx", + (unsigned long long)leader.write_id); + log_tool("extra2 0x%016llx", + (unsigned long long)leader.write_generation); + log_tool("extra3 0x%016llx", + (unsigned long long)leader.write_timestamp); + } else { + log_tool("write_id %llu", + (unsigned long long)leader.write_id); + log_tool("write_generation %llu", + (unsigned long long)leader.write_generation); + log_tool("write_timestamp %llu", + (unsigned long long)leader.write_timestamp); + } break;

case ACT_ACQUIRE: rv = direct_acquire(&main_task, com.io_timeout_arg, com.res_args[0], com.num_hosts, - com.local_host_id, com.local_host_generation, + com.host_id, com.host_generation, &leader); log_tool("acquire done %d", rv); break; diff --git a/src/paxos_lease.c b/src/paxos_lease.c index 33f9e4fe6c59..09b172ed55fe 100644 --- a/src/paxos_lease.c +++ b/src/paxos_lease.c @@ -1336,6 +1336,8 @@ int paxos_lease_acquire(struct task *task, int error, rv, us; int other_io_timeout, other_host_dead_seconds;

+ memset(&dblock, 0, sizeof(dblock)); /* shut up compiler */ + log_token(token, "paxos_acquire begin %x %llu %d", flags, (unsigned long long)acquire_lver, new_num_hosts);

diff --git a/src/resource.c b/src/resource.c index 98a07114bf5c..3c499b5afeda 100644 --- a/src/resource.c +++ b/src/resource.c @@ -33,6 +33,7 @@ #include "timeouts.h" #include "mode_block.h" #include "helper.h" +#include "watchdog.h"

/* from cmd.c */ void send_state_resource(int fd, struct resource *r, const char *list_name, int pid, uint32_t token_id); @@ -49,6 +50,7 @@ static struct list_head resources_add; static struct list_head resources_rem; static pthread_mutex_t resource_mutex; static pthread_cond_t resource_cond; +static struct list_head host_messages;

static void free_resource(struct resource *r) @@ -1693,6 +1695,60 @@ int set_resource_examine(char *space_name, char *res_name) return count; }

+struct recv_message { + struct list_head list; + uint64_t from_host_id; + uint64_t from_generation; + struct sanlk_host_message hm; +}; + +void set_message_examine(char *space_name GNUC_UNUSED, + struct sanlk_host_message *hm, + uint64_t from_host_id, uint64_t from_generation) +{ + struct recv_message *rm; + + rm = malloc(sizeof(struct recv_message)); + if (!rm) { + log_error("set_message_examine no mem"); + return; + } + + memset(rm, 0, sizeof(struct recv_message)); + memcpy(&rm->hm, hm, sizeof(struct sanlk_host_message)); + rm->from_host_id = from_host_id; + rm->from_generation = from_generation; + + pthread_mutex_lock(&resource_mutex); + list_add_tail(&rm->list, &host_messages); + resource_thread_work = 1; + pthread_cond_signal(&resource_cond); + pthread_mutex_unlock(&resource_mutex); +} + +static void resource_thread_messages(void) +{ + struct recv_message *rm, *safe; + + list_for_each_entry_safe(rm, safe, &host_messages, list) { + list_del(&rm->list); + + log_debug("examine message from %llu %llu msg 0x%08x seq 0x%08x", + (unsigned long long)rm->from_host_id, + (unsigned long long)rm->from_generation, + rm->hm.msg, rm->hm.seq); + + if (rm->hm.msg & SANLK_HM_WD_RESET) { + log_error("Host reset request from host_id %llu gen %llu", + (unsigned long long)rm->from_host_id, + (unsigned long long)rm->from_generation); + watchdog_reset_host(); + } + + free(rm); + } +} + /* * resource_thread * - on-disk lease release for pid's that exit without doing release @@ -1896,6 +1952,10 @@ static void *resource_thread(void *arg GNUC_UNUSED) pthread_cond_wait(&resource_cond, &resource_mutex); }

+ /* process any host messages that have been received */ + resource_thread_messages(); + + /* FIXME: it's not nice how we copy a bunch of stuff * from token to r so that we can later copy it back from * r into a temp token. The whole duplication of stuff @@ -1994,6 +2054,7 @@ int setup_token_manager(void) INIT_LIST_HEAD(&resources_add); INIT_LIST_HEAD(&resources_rem); INIT_LIST_HEAD(&resources_held); + INIT_LIST_HEAD(&host_messages);

rv = pthread_create(&resource_pt, NULL, resource_thread, NULL); if (rv) diff --git a/src/resource.h b/src/resource.h index e5c7d4184646..ce97884b1c90 100644 --- a/src/resource.h +++ b/src/resource.h @@ -27,6 +27,9 @@ int request_token(struct task *task, struct token *token, uint32_t force_mode,

int set_resource_examine(char *space_name, char *res_name);

+void set_message_examine(char *space_name, struct sanlk_host_message *hm, + uint64_t from_host_id, uint64_t from_generation); + int res_set_lvb(struct sanlk_resource *res, char *lvb, int lvblen); int res_get_lvb(struct sanlk_resource *res, char **lvb_out, int *lvblen);

diff --git a/src/sanlock.8 b/src/sanlock.8 index 0d2b63ca60c3..3197cb37b4a9 100644 --- a/src/sanlock.8 +++ b/src/sanlock.8 @@ -201,6 +201,8 @@ COMMAND can be one of three primary top level choices .br .BR "sanlock direct" " access storage directly (no coordination with daemon)"

+.SS Daemon Command + .BR "sanlock daemon" " [options]"

.BR -D " " @@ -239,8 +241,11 @@ use high priority (RR) scheduling .BR -l " num" use mlockall (0 none, 1 current, 2 current and future)

-.BR -a " 0|1" -use async i/o +./" non-aio is untested and may not work +./" .BR -a " 0|1" +./" use async i/o + +.SS Client Command

.B "sanlock client" .I action @@ -372,12 +377,35 @@ out the action specified by the requested force_mode. Examine requests for all resource leases currently held in the named lockspace. Only lockspace_name is used from the LOCKSPACE argument.

+.BR "sanlock client set_message -s" " LOCKSPACE " \ +\fB-i\fP " " \fIhost_id\fP " " \ +\fB-g\fP " " \fIgeneration\fP " " +.br + \fB-m\fP \fImsg\fP \fB-n\fP \fIseq\fP + +Tell the sanlock daemon to include a message code in the lockspace's +next delta lease renewal, addressed to the specified host_id. +Both hosts must be members of LOCKSPACE (only the name is used, +the host_id in LOCKSPACE is unused). +.br +If generation is specified, the message will only apply if the target host +has the given generation number. If generation is not specified, sanlock +will use the current generation of the destination host. +.br +If seq is specified, the given sequence number is used for the message. +If seq is not specified, sanlock generates the message sequence number. +.br +msg 1: the host should use its watchdog to reset itself. + +.SS Direct Command + .B "sanlock direct" .I action [options]

-.BR -a " 0|1" -use async i/o +./" non-aio is untested and may not work +./" .BR -a " 0|1" +./" use async i/o

.BI -o " sec" io timeout in seconds diff --git a/src/sanlock.h b/src/sanlock.h index 58779b128e3f..b95d010ac97b 100644 --- a/src/sanlock.h +++ b/src/sanlock.h @@ -101,6 +101,20 @@ struct sanlk_host { uint32_t flags; };

+/* hm msg is a bitfield, and messages can be combined. */ + +#define SANLK_HM_SEQ_DATA 0x00000001 +#define SANLK_HM_WD_RESET 0x00000002 + +struct sanlk_host_message { + uint64_t host_id; + uint64_t generation; + uint32_t msg; + uint32_t seq; + uint64_t ack_host; + uint32_t ack_seq; +}; + size_t sanlock_path_export(char *dst, const char *src, size_t dstlen); size_t sanlock_path_import(char *dst, const char *src, size_t dstlen);

diff --git a/src/sanlock_admin.h b/src/sanlock_admin.h index f76b2dccd116..ee00e478027b 100644 --- a/src/sanlock_admin.h +++ b/src/sanlock_admin.h @@ -278,4 +278,45 @@ int sanlock_test_resource_owners(struct sanlk_resource *res, uint32_t flags,

int sanlock_version(uint32_t flags, uint32_t *version, uint32_t *proto);

+/* + * Set host_id/generation/msg/seq in our own delta lease when it's next renewed. + * + * msg values + * + * SEQ_DATA + * - Treat the seq number as opaque data from the caller, + * not as a sequence number. + * - The receiving node will not ack this message. + * - The receiving node will receive this msg/seq in each renewal + * until it is cleared by the caller. + * + * WD_RESET + * - The host receiving the message should use its watchdog + * device to reset itself as soon as possible. + * - The sanlock daemon will do this itself. + * + * If seq is zero, the sanlock daemon generates one based on a local + * sequential counter, and the value is returned in hm->seq. + * + * A host receiving a message with a specific sequence number will + * not receive another message from the same host until the sequence + * number increases. + */ + +int sanlock_set_message(const char *ls_name, uint32_t flags, + int len, struct sanlk_host_message *hm); + +/* + * Same as sanlock_read_lockspace(), but all hm fields are filled in + * using the extra fields in the delta lease. + * + * The hm struct shows the last host message sent from ls->host_id, + * and the last host/seq acked by ls->host_id. + */ + +int sanlock_read_lockspace_message(struct sanlk_lockspace *ls, + uint32_t flags, uint32_t *io_timeout, + struct sanlk_host_message *hm); + + #endif diff --git a/src/sanlock_internal.h b/src/sanlock_internal.h index c3fa574686aa..414d3f841eb0 100644 --- a/src/sanlock_internal.h +++ b/src/sanlock_internal.h @@ -148,16 +148,21 @@ struct host_status { uint64_t timestamp; /* remote monotime */ uint64_t set_bit_time; uint16_t io_timeout; + uint32_t host_message_recv_seq; };

struct space { struct list_head list; char space_name[NAME_ID_SIZE]; uint32_t space_id; /* used to refer to this space instance in log messages */ + uint32_t io_timeout; uint64_t host_id; uint64_t host_generation; struct sync_disk host_id_disk; - uint32_t io_timeout; + struct sanlk_host_message host_message_send; + uint32_t host_message_send_seq; + uint32_t host_message_recv_seq; + uint64_t host_message_recv_from; int align_size; int renew_fail; int space_dead; @@ -182,6 +187,12 @@ struct space_info { int killing_pids; };

+struct delta_extra { + uint64_t field1; + uint64_t field2; + uint64_t field3; +}; + #define HOSTID_AIO_CB_SIZE 4 #define WORKER_AIO_CB_SIZE 2 #define DIRECT_AIO_CB_SIZE 1 @@ -278,13 +289,15 @@ struct command_line { int gid; /* -G */ int pid; /* -p */ char sort_arg; - uint64_t local_host_id; /* -i */ - uint64_t local_host_generation; /* -g */ + uint64_t host_id; /* -i */ + uint64_t host_generation; /* -g */ int num_hosts; /* -n */ int max_hosts; /* -m */ int res_count; int sh_retries; uint32_t force_mode; + uint32_t hm_msg; /* -m */ + uint32_t hm_seq; /* -n */ char our_host_name[SANLK_NAME_LEN+1]; char *dump_path; struct sanlk_lockspace lockspace; /* -s LOCKSPACE */ @@ -326,6 +339,7 @@ enum { ACT_EXAMINE, ACT_GETS, ACT_VERSION, + ACT_SET_MESSAGE, };

EXTERN int external_shutdown; diff --git a/src/sanlock_sock.h b/src/sanlock_sock.h index c5c1ec9649f1..b61023720130 100644 --- a/src/sanlock_sock.h +++ b/src/sanlock_sock.h @@ -47,6 +47,8 @@ enum { SM_CMD_GET_LVB = 26, SM_CMD_CONVERT = 27, SM_CMD_VERSION = 28, + SM_CMD_SET_MESSAGE = 29, + SM_CMD_READ_LOCKSPACE_MESSAGE = 30, };

struct sm_header { diff --git a/src/watchdog.c b/src/watchdog.c index a8798802d75f..df569dbff57d 100644 --- a/src/watchdog.c +++ b/src/watchdog.c @@ -28,6 +28,8 @@ #include "log.h" #include "watchdog.h"

+static int watchdog_reset_con; + /* * Purpose of watchdog: to forcibly reset the host in the case where a * supervised pid is running but sanlock daemon does not renew its lease @@ -152,3 +154,66 @@ void close_watchdog_file(struct space *sp) close(sp->wd_fd); }

+/* + * Use the watchdog to reset the host as soon as possible. + * Intentionally set the expire time on the connection to + * the current time so that the watchdog will expire and + * reset as soon as possible. + */ + +void watchdog_reset_host(void) +{ + char name[WDMD_NAME_SIZE]; + uint64_t now; + int con, rv; + + if (!com.use_watchdog) + return; + + if (watchdog_reset_con) { + log_debug("watchdog_reset_host already active"); + return; + } + + con = wdmd_connect(); + if (con < 0) { + log_error("watchdog_reset_host connect failed %d", con); + return; + } + + memset(name, 0, sizeof(name)); + + snprintf(name, WDMD_NAME_SIZE - 1, "sanlock_reset_host"); + + rv = wdmd_register(con, name); + if (rv < 0) { + log_error("watchdog_reset_host register failed %d", rv); + goto fail_close; + } + + /* the refcount tells wdmd that it should not cleanly exit */ + + rv = wdmd_refcount_set(con); + if (rv < 0) { + log_error("watchdog_reset_host refcount_set failed %d", rv); + goto fail_close; + } + + now = monotime(); + + rv = wdmd_test_live(con, now, now); + if (rv < 0) { + log_error("watchdog_reset_host test_live failed %d", rv); + goto fail_clear; + } + + log_error("Resetting host with watchdog"); + watchdog_reset_con = con; + return; + + fail_clear: + wdmd_refcount_clear(con); + fail_close: + close(con); +} + diff --git a/src/watchdog.h b/src/watchdog.h index 90e82eb0a44e..794ec30ff9ed 100644 --- a/src/watchdog.h +++ b/src/watchdog.h @@ -15,5 +15,6 @@ int create_watchdog_file(struct space *sp, uint64_t timestamp, int id_renewal_fail_seconds); void unlink_watchdog_file(struct space *sp); void close_watchdog_file(struct space *sp); +void watchdog_reset_host(void);

#endif