Changes to 'refs/tags/sanlock-3.2.2'
by David Teigland
Changes since the dawn of time:
Antoni S. Puimedon (1):
systemd: prevent sigkill from being sent
Daniel P. Berrange (15):
Fix const-ness of many APIs.
Fix warnings in watchdog module
Fix function prototypes for no-arg methods
Remove use of 'index' as a variable name
Make many functions static
Fix missing include in logging file
Annotate many unused parameters to avoid warnings
Remove redundant redeclaration of 'to' variable
Fix args to execv()
Remove redundant arg to 'run_command'
Rename optarg to optionarg to avoid clashing with getopt.h
Disable the read_request method since it is unused
Add many more compiler warning flags & safety checks
Hard code a sector size of 512 if the lease volume is a regular file
Ensure libsanlock.so is built with debug/warning flags
David Teigland (467):
sync_manager: initial commit
sync_manager: misc updates
sync_manager: misc updates
sync_manager: misc updates
sync_manager: misc updates
sync_manager: add more logging
sync_manager: misc updates
sync_manager: misc updates
sync_manager: num_hosts/MAX_HOSTS stuff
daemon: reworking notions of resource_id/token_name
sync_manager: resource lockfiles
sync_manager: lease arg processing
sync_manager: Began multiple lease support
sync_manager: use first command line arg as action
sync_manager: leader record changes and verify
sync_manager: clean up released leases
sync_manager: move functions around
sync_manager: add more tool actions
sync_manager: naming changes
sync_manager: separate token index and id
sync_manager: fix index usage and other misc
sync_manager: use pthread cond in acquire
sync_manager: write all log entries
sync_manager: simplify polling
sync_manager: fix waitpid use
sync_manager: acquire can fail early
sync_manager: write log entries at exit
sync_manager: add test program
sync_manager: move secondary pid check
sync_manager: fix disk paxos contention
devcount: fix verify checks
sync_manager: add GPL license file
sync_manager: fix leader block offsets
sync_manager: increase COMMAND_MAX
sync_manager: renewal should verify
sync_manager: use sector size from libblkid
sync_manager: use a real checksum function
sync_manager: add libblkid to spec file
sync_manager: print status info
sync_manager: one watchdog file per lease
sync_manager: lease_threads handle watchdog files
sync_manager: fix/add some text/comments
sync_manager: refactor read/write
sync_manager: move disk io functions
sync_manager: remove token arg
sync_manager: rename paxos_disk sync_disk
sync_manager: add aio read/write
sync_manager: make io_timeout_seconds a diskio arg
sync_manager: forgot to add new files
sync_manager: use log thread
sync_manager: client connections
sync_manager: connection processing
sync_manager: send/recv pid
sync_manager: add write_sectors
sync_manager: restructuring
sync_manager: write_sectors code factoring
sync_manager: daemonize
sync_manager: releasing leases
sync_manager: async releasing
sync_manager: release fixes
sync_manager: add direct and indirect acquire/release
sync_manager: reacquire resources
sync_manager: move code
sync_manager: same pid may reacquire resource
sync_manager: lease migration
sync_manager: handle client errors
sync_manager: improve error handling
sync_manager: host_id leases
sync_manager: remove empty files
sync_manager: print initialization info
sync_manager: rename files
sync_manager: clean up header org
sync_manager: delta_lease implementation
sync_manager: accept offset units
sync_manager: fix up init output
sync_manager: put back watchdog calls
sync_manager: fix start_host_id error paths
sync_manager: add log_error's for watchdog file errors
sync_manager: actual timeouts
sync_manager: change timeouts on cmd line
sanlock: create new external api
sanlock: build libsanlock
sanlock: use MAX_LEASES everywhere
sanlock: add libvirt plugin
sanlock plugin: couple minor fixes
sanlock: clean up /var file names
sanlock plugin: fix symbol needed by libvirt
sanlock: add some debug output
sanlock plugin: fix uuid copy
sanlock plugin: fix names
sanlock: add "owner_name"
sanlock: fix renewal checks
sanlock: clean up host_id types
sanlock: set_host_id command
sanlock: fix killing pids
sanlock: add status command
sanlock: set version to 1.0
sanlock: delta_lease cleanup
sanlock: changing num_hosts
sanlock: add dump command
sanlock: renewal timings
sanlock: add direct option
sanlock: check for watchdog file
sanlock: recovery fixes
lock_driver_sanlock: fix compile problems
sanlock: improve command options
sanlock: tidying help text
sanlock: move binary to /usr/sbin
sanlock: add init script
sanlock: fix sigterm shutdown
sanlock: init stop
sanlock: add wdtest command
sanlock.spec: new url
lock_driver_sanlock: remove close
sanlock: introduce lockspaces
lock_driver_sanlock: remove files
sanlock: better logging functions
sanlock: misc log message
sanlock.spec: sbin not libexec
sanlock init: remove watchdog reference
wdmd: watchdog multiplexing daemon
sanlock: add code to use wdmd
sanlock/wdmd: use wdmd in sanlock
sanlock/wdmd: add copyright header to source files
sanlock: rename sanlock source dir
sanlock: move tests dir
move COPYING file
wdmd: use signalfd for signal handling
Fix Makefile comments
wdmd: fix daemon debug option
wdmd: add init script
sanlock.spec: updates
sanlock.spec: src dir
sanlock: build with uninstalled libwdmd
sanlock: version 1.1
sanlock: high priority options
wdmd: high priority options
sanlock: return migration state
sanlock: migration.txt describes libvirt/sanlock steps
libsanlock: include admin functions
sanlock: fix host_id expiration check
sanlock: migration working
devcount: migrate test
sanlock: setowner improvements
sanlock: migrate to target fix
sanlock: fix wdmd stop order
sanlock: various fixes
sanlock: remove wdtest
sanlock: remove migration
sanlock: clean up command return data
sanlock: add resource string conversion functions
sanlock: rework internal structs
devcount: add relock test
sanlock: fix release and inquire
sanlock: add_lockspace EEXIST
sanlock: rework client handling
sanlock: clean up warnings
sanlock: debug message changes
sanlock: add lockspace checks
wdmd: enable test scripts
sanlock: add str_to_lockspace to lib
WIP devcount migrate
devcount: new migrate test
sanlock: read_id and live_id commands
sanlock: check lockspace name and host_id
sanlock: remove remaining cluster_mode
sanlock: add libsanlock_direct
devcountn: start multiple devcount tests
devcount: small changes
sanlock: new return values
sanlock: misc changes and fix
sanlock: log error of full bad block
sanlock: interval between renewal checks
sanlock: renewal changes
sanlock: fix log_dump
sanlock: fix find_client_pid
sanlock: fix host_id reads from paxos_acquire
sanlock: init with one write
devcount: improve output
devcount: new pause/resume
devcount: add expire test
sanlock: correct paxos usage
sanlock: direct read_leader
sanlock: paxos delays
sanlock: use thread pool
sanlock: client status output format changes
sanlock: fix inquire of dead pid
sanlock: use native linux aio
sanlock: i/o changes
sanlock: aio changes
sanlock: reduce paxos acquire read ops
sanlock: quiet error case
sanlock: don't free aio buf until event completes
sanlock: io timeout related changes
sanlock: read dblocks in single aligned io
sanlock: add sanlock_restrict api
sanlock: add sanlock_direct_sector_size api
sanlock: add checksum to dblocks
sanlock: fix init restart
sanlock: don't release tokens in dead lockspace
sanlock: fix adding lockspace
sanlock: official 1MB/8MB alignment
devcount: use aio in init
libsanlock: link with LDFLAGS
sanlock: increase version to 1.3
sanlock/wdmd: shut up warnings
sanlock: fix libwdmd linking
remove spec file
sanlock: use a completed read after renewal timeout
sanlock: use unique host name in delta leases
sanlock: remove sector_size api
sanlock: abort delta wait on shutdown
sanlock: fix add_lockspace failure
sanlk_load: add new test
sanlock: fix recv and inquire
sanlock: initial pid_dead check in acquire
sanlock: release 1.4
sanlock: generate a uuid for host id
sanlock: return -EINPROGRESS from add_lockspace
sanlk_load: periodically kill and replace a pid
sanlock: zero num_hosts uses DEFAULT_MAX_HOSTS
tests: misc changes
sanlock: break paxos_acquire wait loop
sanlock: increase log line to 512 bytes
sanlock: change a log_error to log_debug
sanlock: fail host_id when corrupted
sanlock: release 1.5
sanlock: release 1.6
sanlock: handle colon escaping in path strings
wdmd: add option for high priority
wdmd: use accept4 with SOCK_NONBLOCK
wdmd: tidy sun_addr snprintf
wdmd: pid and sock file changes
wdmd: add man page
wdmd: disable test scripts
sanlock: use accept4 with SOCK_NONBLOCK
sanlock: tidy sun_addr snprintf
sanlock: add explicit -luuid
sanlock: pid and sock file changes
sanlock: add man page
sanlock/wdmd: improve mkdir of run dir
wdmd: new build flags
sanlock: new build flags
sanlock/wdmd: use monotonic time
sanlock: build with pie
sanlock/wdmd: nonblocking listening/accept
sanlock: add missing monotime files
sanlock: update man page
sanlock: man page update
sanlock: update man page and help text
sanlock: print connections limit
release: sanlock 1.7
makefile: install mode for man pages
sanlock: read align_size in renewal
sanlock: check other host_id leases
sanlock: minor fixes and cleanups
sanlock: add request api/cmd
sanlock: crc code tidying
sanlock/wdmd: add license header to files
sanlock: create libsanlock_client
sanlock: move client code
remove COPYING file
sanlock: remove internal header from sanlock_sock
libsanlock_client: use LGPLv2+
libwdmd: use LGPLv2+
sanlock_rv.h: switch to LGPLv2+
README.license: document licenses
sanlock: client align and init
sanlock: write request record
sanlock: request struct magic and version numbers
sanlock: set bitmap for requests
sanlock: examine resource requests
sanlock: update man page
sanlock: remove old comment
sanlock: renaming functions
sanlock: optimize paxos wait
sanlock: use flags in struct resource
sanlock: restructure lockspace checks
sanlock: remove BLOCK_WD force mode
libsanlock: fix function stubs
sanlock: new status and host_status
sanlock: improve status output
sanlock: status output sorting
sanlock: SIGTERM and SIGKILL for REQ_KILL_PID
sanlock: add flag RESTRICT_SIGKILL
sanlock: setup fewer aio events
sanlock: move cmd processing
sanlock: suppress log messages
sanlock: rename source files
sanlock: improve killing pids
sanlock: fix log_dump
version 1.8
sanlock: quick host_id reacquire
sanlock: add force option to shutdown command
simpler copyright line
default to GPLv2+ for our original files
sanlock: improve daemon permission errors
add systemd files
fixing up init stuff
release 1.9
sanlock: setmode
Revert "sanlock: setmode"
sanlock: fix error exit
sanlock: fix debug line
init scripts: fix path to restorecon
sanlock: shared mode for leases
sanlock: fix missing close_disks
sanlock: fix problem in paxos
sanlock: add paxos sanity check
sanlock: fix leaking fd
sanlock: change to paxos algorithm
sanlock: change ondisk version
sanlock: remove log noise
sanlock: ignore SIGTERM if lockspaces exist
sanlock: quiet more error messages
sanlk_load: munging variable names
sanlk_load: add shared locks
sanlock: kill client connection on recv error
sanlk_load: ignore release errors
sanlock: set sector_size in shared tokens
sanlock man page: mention SH
release 2.0
sanlock: clean up warnings
sanlock: fix inquire state string
sanlock: retry transient sh failures
sanlock: status for all shared tokens
sanlock: add a logrotate file
release 2.1
python: fix path strncpy
sanlock: remove physical sector size requirement
sanlock: add sample sysconfig file
sanlock: ASYNC flag for lockspace add and rem
sanlock: UNUSED flag for lockspace rem
release 2.2
sanlock.log: empty file to install from rpm
Revert "sanlock.log: empty file to install from rpm"
sanlock: remove limits.conf
sysconfig: remove user option
init: root user
release 2.3
latest spec file
sanlock.spec: fix exclusive arch
sanlock/wdmd: use /var/log/subsys/file
daemon: skip setup_groups when no -U or -G
sanlock.spec: require useradd and groupadd
sanlock.spec updated
daemon: don't put struct space on stack
daemon: fix add_lockspace that has been removed
daemon: fix inquire lver
daemon: graceful pid handling
Revert "init: make explicit to run sanlock as root"
remove spec file
release 2.4
daemon: include resource.h
daemon: fix lockfile ownership
wdmd: use lockfile mode 644
sanlock: use lockfile mode 644
wdmd: use shm_open to prevent restart
sanlock: remove umask 0
sanlock/wdmd: remove global connection
daemon: extend grace time
wdmd: close device when test fails
wdmd: preemptive close before test fails
wdmd: pet after reopen and use 1 sec interval after failure
sanlock: base kill sig on last renewal
clientn: add tests
tests: add test-recovery script
sanlock: fix paxos acquire host_id check
daemon: use helper for examine request kill
sanlock: show host_status for all lockspaces
sanlock: fix status of lockspaces in add and rem
sanlock: adjustable io timeouts
sanlock: add direct next_free command
sanlock: request force_mode 2 is SIGUSR1
sanlock: print escape chars in path from status
release 2.5
sanlock: configurable mlockall level
sanlock: change log level for startup and io timeouts
sanlock: update man page
sanlock: fix aborted command clearing
sanlock: client add_lockspace with timeout
sanlock: print command output to stdout
wdmd: new script handling
fence_sanlock: new code
fence_sanlock: various fixes and changes
fence_sanlock: variable alignment
fence_sanlockd: use init script
fence_sanlock: add man pages
fence_sanlock: use SIGHUP for shutdown
fence_sanlock: handle two node fence duel
sanlock/wdmd: get version from common file
wdmd: show option values in start message
release 2.6
fix systemd service files
wdmd: use mode 775 for run dir
sanlock: clean up daemon exit error path
sanlock: read and write apis for lockspace and resource
sanlock: new get_lockspaces api
sanlock: get_lockspaces count only
sanlock: make get_lockspaces allocate space for data
wdmd: dynamically select working watchdog device
fence_sanlock: fix man page name of path arg
sanlock: shutdown or rem_lockspace should SIGKILL
sanlock: fix usage output typos
release 2.7
sanlock: add request killpath
sanlock/fence_sanlock: request modes FORCE and GRACEFUL
sanlock: update request types in man page
sanlock: add get_hosts api
sanlock: remove direct read_id and live_id
sanlock: add read_resource_owners
sanlock: add test_resource_owners
sanlock: add flag SANLK_REQUEST_NEXT_LVER
sanlock: improve check of max_hosts and num_hosts
sanlock: add a comment
test: add another test program
sanlock: check for null arg in sanlock_request
release 2.8
sanlock: add lvb feature
sanlock: add SANLK_REL_RENAME
release 3.0.0
sanlock: free hss on error
sanlock: reinit hss in loop
sanlock: fix read_lockspace name comparison
sanlock: initialize owner_id
sanlock: initialize send_len and send_buf
release 3.0.1
sanlock: fix debug status for add/rem lockspace
sanlock: add_lockspace error for conflicting lockspace definitions
sanlock: add some acquire error descriptions
sanlock: use ADD and REM suffix in status output
sanlock: transient timeout handling in acquire and release
sanlock: add convert
sanlock: allow more than 8 resources per client
tests: sanlk_load
sanlock: fix spaces in path names
wdmd: add probe option
release 3.1.0
sanlock: version interface
sanlock: shutdown reply
sanlock: fix get_hosts off by one
sanlock: lockspace host events
sanlock: set_config api
reset: daemon and program for resetting hosts
sanlock: disk endian handling
sanlock: persistent and orphan resource leases
fence_sanlock: use persistent and orphan features
reset: remove DEBUG
init: add service file for sanlk-resetd
init: misc changes
sanlock: log error for invalid set_event args
sanlock: close event fds when lockspace is removed
sanlock: fix missing mutex unlock in previous commit
sanlock: host event struct missing host_id in get_event
reset: add ability to clear all
sanlk-reset: no args should print help
tests: update sanlk_events
reset: add man pages
release 3.2.0
reset: install man pages
sanlock: print better names in direct read -s
sanlock: log bad delta leases and delta lease owners
sanlock: direct write_leader
release 3.2.1
sanlock, reset: coverity fixes
reset: native timeout
release 3.2.2
David Weber (3):
Fix order of linking
Install another symlink to shared library
Replace restrict field name with restricted
Fabio M. Di Nitto (15):
build: sanlock should link with libsanlock
build: install shared lib header files and fix DESTDIR usage
build: drop rpm target
spec file: do first cut for total spec file
build: fix linking with libsanlock and install target
fence_sanlock: add install bits
fence_sanlockd: add missing \n
fence_sanlock: cleanup shell usage
fence_sanlock: don't restart fence_sanlockd if it's already running
fence_sanlock: add basic metadata to integrate with ccs_config_validate
fence_sanlock: improve action_on error checking and report
fence_sanlock: unify code and silence some operations
fence_sanlockd init: cleanup, remove hardcoded paths and use system service
fence_sanlock: fix build to be selfcontained
systemd: add fence_sanlockd service file
Federico Simoncelli (76):
rpm: sync specfile with fedora
rpm: add sanlock_admin.h header
rpm: add the lib package, install the initscripts
python: remove unused python code
python: add python binding
python: release the gil during sanlock operations
python: wrap sanlock extension with a module
rpm: add python binding package
python: pass a lockspace copy to str_to_lockspace
makefile: fix install typo
rpm: add sanlock_direct header
python: add sanlock init functions and exception
direct: close disks after initialization
python: register process only once
daemon: configurable socket permissions
rpm: add sanlock user and group
python: exceptions must contain the errno
rpm: add missing libaio-devel dependency
rpm: add daemon options in the init file
python: add missing aio library
python: add get_alignment function
libs: include libsanlock_direct into libsanlock
python: align num_hosts and max_hosts defaults
python: expose sanlock file descriptor
python: improve error reporting
python: parse lockspaces and resources natively
python: add usage example
python: initial support for sanlock errors
python: document the sanlock module
python: module cleanup
build: fix documentation install path
client: return appropriate errno on failure
python: expose errno in the exception
python: acquire leases for other processes
sanlock: implement the inq_lockspace command
python: add the inq_lockspace command binding
misc: run sanlock daemon as sanlock user
wdmd: use getopt to parse the command line
wdmd: make socket group ownership configurable
python: add shared resource parameter to acquire
python: release leases for other processes
python: ASYNC flag for lockspace add and rem
python: UNUSED flag for lockspace rem
log: add full time and date to the log messages
sanlock: set the supplementary groups at startup
log: add the thread id in the log messages
sanlock: fix an rv check typo in setup_groups
init: add a stop timeout for the sanlock service
build: add a generic rpm spec file
init: make explicit to run sanlock as root
wdmd: load the softdog module when needed
sanlock: WAIT flag for sanlock_inq_lockspace
daemon: drop root privileges and use the helper
init: use checkpid when stopping the services
sanlock: expose sanlock path import and export utils
sanlock: configuring the core dump output
python: fix the tuple parsing for py_add_lockspace
sanlock: use signal handler for all terminating signals
systemd: add wdmd dependency for sanlock
wdmd: trap also SIGINT for a clean exit
wdmd: make the watchdog device configurable
python: fix typo for the offset parameter
python: add write lockspace and resource
python: add read lockspace and resource
wdmd: fix miscellaneous build warnings
misc: add wdmd to gitignore
python: fix multiple issues reported by cpychecker
python: add get_lockspaces
python: use the write commands in example.py
python: add the versioning support to acquire
python: return lease version in read_resource
python: rename SANLK_LSF flags and constants macro
python: improve example with signals and exceptions
python: add the support for the resource request
python: add the get_hosts support
python: add support to read resource owners
Nir Soffer (9):
sanlk-resetd: Allow only privileged user to modify configuration
reset: Use find_ls to remove lockspace by name
python: add support for lockspace events
reset: Require lockspace name for local deamon operations
reset: Add the clear command to the usage
reset: Improve error message when command is missing
reset: Calling with too many lockspaces is a usage error
reset: Normalize exit codes
reset: Reduce logging noise to stderr and syslog
Saggi Mizrahi (25):
Added the begining of the testing and debugging tools
Better handling of max hosts
sync_manager: Updated tests to work with new lease struct
sync_manager: fixed skipping first arg in command
sync_manager: acquire and release actions
sync_manager: minor fixes
sync_manager: renamed stuff
sync_manager: made acquire synchronous again
sync_manager: added set_host_id action
sync_manager: use kill(0) for secondary pid check
sync_manager: make rpm and install
sync_manager: spec file update
sync_manager: Allow longer resource names
sync_manager: allow repeated set_host_id
sync_manager: Added escaping for the leases arg
sync_manager: Created the python bindings for sync_manager
sync_manager: listener socket permissions
sync_manager: Updated python binding and tests
sync_manager: Made 'token' a const in log_level
sync_manager: refactor messaging system
sync_manager: use getsockopt PEERCRED
Fix various build warnings
Tidy up some copy pasted code
Set constness in some places
Add a gitignore file
Sandro Bonazzola (1):
setup: aligned with rpm package
9 years, 2 months
VERSION
by David Teigland
VERSION | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
New commits:
commit 5ad5d6a3dfa88b40d7824bb27e8e50181a1f6071
Author: David Teigland <teigland(a)redhat.com>
Date: Mon Sep 29 15:10:35 2014 -0500
release 3.2.2
Signed-off-by: David Teigland <teigland(a)redhat.com>
diff --git a/VERSION b/VERSION
index e4604e3..be94e6f 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-3.2.1
+3.2.2
9 years, 2 months
reset/sanlk-reset.8 reset/sanlk_reset.c
by David Teigland
reset/sanlk-reset.8 | 6 +
reset/sanlk_reset.c | 200 ++++++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 201 insertions(+), 5 deletions(-)
New commits:
commit cf4a94c3481fdc4ff8d0a4fbe12ffaafc64ee68b
Author: David Teigland <teigland(a)redhat.com>
Date: Mon Sep 15 13:32:30 2014 -0500
reset: native timeout
Allow reset to complete in a shorter time based on a
native timeout calculation. This uses the fact that
sanlk-resetd sets up an expired watchdog connection
itself, through wdmd, before its RESETTING event.
The completion timing calculation can begin at that
point, rather than at the final renewal time, which
the sanlock host status uses.
Signed-off-by: David Teigland <teigland(a)redhat.com>
diff --git a/reset/sanlk-reset.8 b/reset/sanlk-reset.8
index 468ea0a..8aede4d 100644
--- a/reset/sanlk-reset.8
+++ b/reset/sanlk-reset.8
@@ -56,9 +56,13 @@ Generation of host. (Use only with single lockspace name.)
Enable/Disable (1/0) use of /proc/sysrq\-trigger to reboot.
.TP
-.B \-\-resource-mode, \-R 0|1
+.B \-\-resource\-mode, \-R 0|1
Resource leases are used (1) or not used (0) to protect storage.
+.TP
+.B \-\-native\-timeout, \-t " num
+Disable native timeout by setting to 0.
+
.SS Update the local sanlk\-resetd
\&
diff --git a/reset/sanlk_reset.c b/reset/sanlk_reset.c
index 51189f8..9e952ab 100644
--- a/reset/sanlk_reset.c
+++ b/reset/sanlk_reset.c
@@ -41,6 +41,91 @@ static socklen_t update_addrlen;
#define EXIT_USAGE 2
#define MAX_LS 64
+/*
+ * native timeout: calculate directly when a host's watchdog
+ * should have fired, based on sanlock/wdmd/watchdog timings.
+ * This can complete much quicker than waiting for the sanlock
+ * host status states. The host status states are based on a
+ * lockspace failing to renew a lease, and determining the
+ * latest possible watchdog firing based on that. The reset
+ * case is based on lockspace storage remaining accessable,
+ * and a host acknowledging through a RESETTING event that
+ * its watchdog has been set to expire. Until the watchdog
+ * fires, the host will continue renewing its lease. We can
+ * derive a faster completion based on the RESETTING event
+ * rather than waiting for the sanlock host status, which
+ * would still be correct, but longer, because it's based on
+ * the obvious lack of renewals following the reset. So,
+ * with native timeout, the completion timing begins from the
+ * RESETTING acknowledgment, but with the sanlock host status,
+ * the completion timing begins from the final renewal before
+ * the reset.
+ *
+ * Native timeout calculation:
+ *
+ * This timeout calculation begins when we see the RESETTING
+ * event from the host. It can take multiple lease renewal
+ * intervals for the RESET and RESETTING events to be transmitted
+ * between hosts. The time required for these events precedes the
+ * timeout considered in the following. The total time required
+ * for the program would be the sum of the time required for
+ * the RESET/RESETTING events, and the native timeout.
+ *
+ * When we first see the host is resetting, make a record
+ * of the local time (ls_resetting_begin_local) and the remote
+ * timestamp from the delta lease renewal (ls_resetting_begin_timestamp).
+ *
+ * We'll continue watching the resetting host for another
+ * local 90 seconds from the local time saved here. After
+ * that 90 seconds, we'll check the last timestamp seen from
+ * the host. If the last timestamp is more than 70 seconds after
+ * than the first timestamp we saved upon seeing resetting,
+ * then the host's watchdog device failed to fire when required
+ * (watchdog_failed_to_fire).
+ *
+ * The resetting host should be reset (its watchdog should fire)
+ * at most 70 seconds after it sets RESETTING. So, it should not
+ * be possible for a renewal timestamp to be more than 70 seconds
+ * later than the timestamp when it set the RESETTING event.
+ *
+ * T0: host sets RESETTING
+ * T0: host sets up expired wdmd connection
+ * T10: wdmd test interval wakes and sees new expired connection
+ * T10: wdmd closes /dev/watchdog uncleanly, which issues final ping
+ * T70: 60 seconds later, the watchdog device fires
+ *
+ * The host continues renewing its delta lease up until its watchdog
+ * fires at T70, so there may be a renewal right at T70, 70 seconds
+ * after it set RESETTING.
+ *
+ * We continue watching for a renewal for 20 seconds after this
+ * latest possible renewal, to give the host time for another
+ * renewal if its watchdog failed to fire. If we do not see another
+ * renewal for 20 seconds (the max standard renewal interval), then
+ * this is a confirmation that the watchdog fired as expected by T70.
+ *
+ * If the host's watchdog fails to fire, and storage access is maintained,
+ * then the resetting host will continue to renew its lease. This program
+ * will then see renewal timestamps later than T70, and will fail.
+ *
+ * If the host's watchdog fails to reset it, and the host also loses its
+ * storage access, then this program will incorrectly conclude that the
+ * host has been reset when it is not.
+ *
+ * The times (90, 70) are based on the following sanlock/wdmd defaults:
+ * . 10 second io_timeout
+ * . 60 second watchdog_fire_timeout
+ * . 20 second id_renewal_seconds
+ * . 10 second wdmd test interval
+ *
+ * If the resetting host has a different io_timeout, then disable
+ * the native timeout check and depend on the host status check.
+ */
+
+#define NATIVE_TIMEOUT_SECONDS 90
+#define NATIVE_RENEWAL_SECONDS 70
+#define NATIVE_VERIFY_SECONDS (NATIVE_TIMEOUT_SECONDS - NATIVE_RENEWAL_SECONDS)
+
static char *prog_name;
static uint64_t begin;
static struct pollfd *pollfd;
@@ -50,10 +135,15 @@ static int resource_mode;
static int debug_mode;
static int target_host_id;
static uint64_t target_generation;
+static int native_timeout = NATIVE_TIMEOUT_SECONDS;
+static int native_renewal = NATIVE_RENEWAL_SECONDS;
+static int watchdog_failed_to_fire;
static int ls_count;
static char *ls_names[MAX_LS];
static int ls_hostids[MAX_LS];
static int ls_fd[MAX_LS];
+static uint64_t ls_resetting_begin_timestamp[MAX_LS];
+static uint64_t ls_resetting_begin_local[MAX_LS];
static uint64_t ls_timestamp[MAX_LS];
static uint32_t ls_host_flags[MAX_LS];
static int ls_is_resetting[MAX_LS];
@@ -212,6 +302,9 @@ static int reset_fail(void)
int cmd_wait = 0;
int i;
+ if (watchdog_failed_to_fire)
+ return 1;
+
for (i = 0; i < MAX_LS; i++) {
if (!ls_names[i])
continue;
@@ -293,9 +386,11 @@ static int reset_fail(void)
static int reset_done(void)
{
struct sanlk_host *hs;
+ uint64_t now;
uint64_t host_id;
uint32_t state;
int hs_count;
+ int ls_is_done;
int is_done = 0;
int i, rv;
@@ -334,10 +429,78 @@ static int reset_done(void)
(unsigned long long)hs->timestamp,
ls_names[i], ls_hostids[i]);
+ if (hs->timestamp && (hs->io_timeout != 10) && native_timeout) {
+ log_error("disable native_timeout due to zero io_timeout in %s:%d",
+ ls_names[i], ls_hostids[i]);
+ native_timeout = 0;
+ }
+
free(hs);
}
/*
+ * The native timeout check.
+ */
+
+ if (!native_timeout)
+ goto check_host_status;
+
+ for (i = 0; i < MAX_LS; i++) {
+ if (!ls_names[i])
+ continue;
+
+ if (!ls_is_resetting[i])
+ continue;
+
+ now = monotime();
+
+ if (!ls_resetting_begin_local[i]) {
+ ls_resetting_begin_timestamp[i] = ls_timestamp[i];
+ ls_resetting_begin_local[i] = now;
+
+ log_debug("resetting begin local %llu timestamp %llu in ls %s:%d",
+ (unsigned long long)ls_resetting_begin_local[i],
+ (unsigned long long)ls_resetting_begin_timestamp[i],
+ ls_names[i], ls_hostids[i]);
+ }
+
+ if (now - ls_resetting_begin_local[i] > native_timeout) {
+ if (ls_timestamp[i] - ls_resetting_begin_timestamp[i] > native_renewal) {
+ /*
+ * This should never happen.
+ */
+ log_error("watchdog failed to fire in ls %s:%d", ls_names[i], ls_hostids[i]);
+ log_error("resetting_begin_local %llu now %llu "
+ "resetting_begin_timestamp %llu timestamp %llu "
+ "native_timeout %d native_renewal %d "
+ "ls %s:%d",
+ (unsigned long long)ls_resetting_begin_local[i],
+ (unsigned long long)now,
+ (unsigned long long)ls_resetting_begin_timestamp[i],
+ (unsigned long long)ls_timestamp[i],
+ native_timeout, native_renewal,
+ ls_names[i], ls_hostids[i]);
+
+ watchdog_failed_to_fire = 1;
+ } else {
+ log_info("reset done by native_timeout in ls %s:%d", ls_names[i], ls_hostids[i]);
+ is_done = 1;
+ }
+ } else {
+ log_debug("native timeout seconds remaining %d in ls %s:%d",
+ native_timeout - (int)(now - ls_resetting_begin_local[i]),
+ ls_names[i], ls_hostids[i]);
+ }
+ }
+
+ if (watchdog_failed_to_fire)
+ return 0;
+
+ check_host_status:
+
+ /*
+ * The host status check.
+ *
* The lockspace behavior is different when resource leases
* are not used to protect storage, so the conditions to check
* depend on the --with-resources option.
@@ -365,6 +528,8 @@ static int reset_done(void)
if (!ls_names[i])
continue;
+ ls_is_done = 0;
+
state = ls_host_flags[i] & SANLK_HOST_MASK;
if (state == SANLK_HOST_DEAD && !ls_is_dead[i]) {
@@ -377,14 +542,18 @@ static int reset_done(void)
log_info("host free in ls %s:%d", ls_names[i], ls_hostids[i]);
}
- if (resource_mode && ls_is_dead[i])
+ if (resource_mode && ls_is_dead[i]) {
+ ls_is_done = 1;
is_done = 1;
+ }
- if (!resource_mode && ls_is_dead[i] && ls_is_resetting[i])
+ if (!resource_mode && ls_is_dead[i] && ls_is_resetting[i]) {
+ ls_is_done = 1;
is_done = 1;
+ }
- if (is_done)
- log_info("reset done in ls %s:%d", ls_names[i], ls_hostids[i]);
+ if (ls_is_done)
+ log_info("reset done by host_status in ls %s:%d", ls_names[i], ls_hostids[i]);
}
return is_done;
@@ -483,16 +652,30 @@ static void usage(void)
printf("\n");
printf("Reset another host through a lockspace it is watching:\n");
printf("%s reset lockspace_name:host_id ...\n", prog_name);
+ printf("\n");
printf(" --host-id | -i <num>\n");
printf(" Host id to reset.\n");
+ printf("\n");
printf(" --generation | -g <num>\n");
printf(" Generation of host id (default 0 for current generation).\n");
+ printf("\n");
printf(" --watchdog | -w 0|1\n");
printf(" Disable (0) use of wdmd/watchdog for testing.\n");
+ printf("\n");
printf(" --sysrq-reboot | -b 0|1\n");
printf(" Enable/Disable (1/0) use of /proc/sysrq-trigger to reboot (default 0).\n");
+ printf("\n");
printf(" --resource-mode | -R 0|1\n");
printf(" Resource leases are used (1) or not used (0) to protect storage.\n");
+ printf("\n");
+ printf(" --native-timeout | -t <num>\n");
+ printf(" Disable native timeout by setting to 0.\n");
+#if 0
+ printf(" Reset completion is calculated natively and is faster than\n");
+ printf(" waiting for the sanlock host status. Set to 0 to disable.\n");
+ printf(" (default %d, using a lower value can produce invalid result.)\n", NATIVE_TIMEOUT_SECONDS);
+#endif
+ printf("\n");
printf(" The event will be set in each lockspace_name (max %d).\n", MAX_LS);
printf(" The -i and -g options can only be used with a single lockspace_name arg.\n");
printf("\n");
@@ -525,6 +708,7 @@ int main(int argc, char *argv[])
{"watchdog", required_argument, 0, 'w' },
{"sysrq-reboot", required_argument, 0, 'b' },
{"resource-mode", required_argument, 0, 'R' },
+ {"native-timeout", required_argument, 0, 't' },
{"debug-mode", no_argument, 0, 'D' },
{0, 0, 0, 0 }
};
@@ -562,6 +746,14 @@ int main(int argc, char *argv[])
case 'R':
resource_mode = atoi(optarg);
break;
+ case 't':
+ if (!atoi(optarg))
+ native_timeout = 0;
+#if 0
+ if (native_timeout > NATIVE_VERIFY_SECONDS)
+ native_renewal = native_timeout - NATIVE_VERIFY_SECONDS;
+#endif
+ break;
case 'D':
debug_mode = 1;
break;
9 years, 2 months
Changes to 'reset'
by David Teigland
New branch 'reset' available with the following commits:
commit 298796a4821e3107da42d4509639cc588aab448c
Author: David Teigland <teigland(a)redhat.com>
Date: Mon Sep 15 13:32:30 2014 -0500
reset: native timeout
Allow reset to complete in a shorter time based on a
native timeout calculation. This uses the fact that
sanlk-resetd sets up an expired watchdog connection
itself, through wdmd, before its RESETTING event.
The completion timing calculation can begin at that
point, rather than at the final renewal time, which
the sanlock host status uses.
Signed-off-by: David Teigland <teigland(a)redhat.com>
9 years, 2 months
reset/sanlk_reset.c
by David Teigland
reset/sanlk_reset.c | 105 ++++++++++++++++++++++++++++++----------------------
1 file changed, 62 insertions(+), 43 deletions(-)
New commits:
commit 7c4d08493390c303c3bd9142ead71784a86fa378
Author: Nir Soffer <nsoffer(a)redhat.com>
Date: Fri Sep 12 14:14:24 2014 -0500
reset: Reduce logging noise to stderr and syslog
When reseting another host, we used to create lot of noise by logging
the result of any lockspace check every 2 seconds. This info is useless
for the user of this command, but may be valuable for debugging. So now
we log everything when using --debug-mode, and only state changes
otherwise.
Some log messages were too verbose and technical. Now log_debug messages
are verbose and technical, but log_info messages are trying to be user
friendly.
Previously we logged few messages per lockspace to syslog for each reset
operation. This is unwanted noise on a machine performing a reset,
specially when using 10's of lockspaces. Now only warning and errors
are logged to syslog. We assume that a program running this tool will
log its own logs if needed for each reset operation.
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
diff --git a/reset/sanlk_reset.c b/reset/sanlk_reset.c
index 63e6616..51189f8 100644
--- a/reset/sanlk_reset.c
+++ b/reset/sanlk_reset.c
@@ -47,6 +47,7 @@ static struct pollfd *pollfd;
static int use_watchdog = 1;
static int use_sysrq_reboot = 0;
static int resource_mode;
+static int debug_mode;
static int target_host_id;
static uint64_t target_generation;
static int ls_count;
@@ -60,27 +61,31 @@ static int ls_is_dead[MAX_LS];
static int ls_is_free[MAX_LS];
static int ls_renewals[MAX_LS];
-#define log_debug(fmt, args...) \
+
+#define errlog(fmt, args...) \
do { \
fprintf(stderr, "%llu " fmt "\n", (unsigned long long)time(NULL), ##args); \
} while (0)
-#define log_error(fmt, args...) \
+#define log_debug(fmt, args...) \
do { \
- log_debug(fmt, ##args); \
- syslog(LOG_ERR, fmt, ##args); \
+ if (debug_mode) \
+ errlog(fmt, ##args); \
} while (0)
+#define log_info(fmt, args...) \
+ errlog(fmt, ##args)
+
#define log_warn(fmt, args...) \
do { \
- log_debug(fmt, ##args); \
+ errlog(fmt, ##args); \
syslog(LOG_WARNING, fmt, ##args); \
} while (0)
-#define log_notice(fmt, args...) \
+#define log_error(fmt, args...) \
do { \
- log_debug(fmt, ##args); \
- syslog(LOG_NOTICE, fmt, ##args); \
+ errlog(fmt, ##args); \
+ syslog(LOG_ERR, fmt, ##args); \
} while (0)
static uint64_t monotime(void)
@@ -362,11 +367,15 @@ static int reset_done(void)
state = ls_host_flags[i] & SANLK_HOST_MASK;
- if (state == SANLK_HOST_DEAD)
+ if (state == SANLK_HOST_DEAD && !ls_is_dead[i]) {
ls_is_dead[i] = 1;
+ log_info("host dead in ls %s:%d", ls_names[i], ls_hostids[i]);
+ }
- if (state == SANLK_HOST_FREE)
+ if (state == SANLK_HOST_FREE && !ls_is_free[i]) {
ls_is_free[i] = 1;
+ log_info("host free in ls %s:%d", ls_names[i], ls_hostids[i]);
+ }
if (resource_mode && ls_is_dead[i])
is_done = 1;
@@ -374,11 +383,8 @@ static int reset_done(void)
if (!resource_mode && ls_is_dead[i] && ls_is_resetting[i])
is_done = 1;
- if (ls_is_dead[i])
- log_notice("host dead in ls %s:%d", ls_names[i], ls_hostids[i]);
-
if (is_done)
- log_notice("reset done in ls %s:%d", ls_names[i], ls_hostids[i]);
+ log_info("reset done in ls %s:%d", ls_names[i], ls_hostids[i]);
}
return is_done;
@@ -388,6 +394,8 @@ static void get_events(int i)
{
struct sanlk_host_event from_he;
uint64_t from_host, from_gen;
+ int resetting = 0;
+ int rebooting = 0;
int rv;
while (1) {
@@ -401,26 +409,26 @@ static void get_events(int i)
break;
}
- if ((from_host == ls_hostids[i]) &&
- ((from_he.event & EVENT_RESETTING) || (from_he.event & EVENT_REBOOTING))) {
- log_notice("notice of %s%s(%llx %llx) from host %llu %llu ls %s",
- (from_he.event & EVENT_RESETTING) ? "resetting " : "",
- (from_he.event & EVENT_REBOOTING) ? "rebooting " : "",
- (unsigned long long)from_he.event,
- (unsigned long long)from_he.data,
- (unsigned long long)from_host,
- (unsigned long long)from_gen,
- ls_names[i]);
-
- if (from_he.event & EVENT_RESETTING)
+ log_debug("got event %llx %llx from host %llu %llu in ls %s:%d",
+ (unsigned long long)from_he.event,
+ (unsigned long long)from_he.data,
+ (unsigned long long)from_host,
+ (unsigned long long)from_gen,
+ ls_names[i],
+ ls_hostids[i]);
+
+ resetting = from_he.event & EVENT_RESETTING;
+ rebooting = from_he.event & EVENT_REBOOTING;
+
+ if ((from_host == ls_hostids[i]) && (resetting || rebooting)) {
+ log_info("host %s%sin ls %s:%d",
+ resetting ? "resetting " : "",
+ rebooting ? "rebooting " : "",
+ ls_names[i],
+ ls_hostids[i]);
+
+ if (resetting)
ls_is_resetting[i] = 1;
- } else {
- log_notice("event ignored %llx %llx from host %llu %llu ls %s",
- (unsigned long long)from_he.event,
- (unsigned long long)from_he.data,
- (unsigned long long)from_host,
- (unsigned long long)from_gen,
- ls_names[i]);
}
}
}
@@ -461,6 +469,8 @@ static void usage(void)
printf(" Show this help information.\n");
printf(" --version | -V\n");
printf(" Show version.\n");
+ printf(" --debug-mode | -D\n");
+ printf(" Log debugging information.\n");
printf("\n");
printf("Update the local sanlk-resetd to watch lockspaces for reset events:\n");
printf("%s reg lockspace_name ...\n", prog_name);
@@ -515,6 +525,7 @@ int main(int argc, char *argv[])
{"watchdog", required_argument, 0, 'w' },
{"sysrq-reboot", required_argument, 0, 'b' },
{"resource-mode", required_argument, 0, 'R' },
+ {"debug-mode", no_argument, 0, 'D' },
{0, 0, 0, 0 }
};
@@ -522,7 +533,7 @@ int main(int argc, char *argv[])
int c;
int option_index = 0;
- c = getopt_long(argc, argv, "hVi:g:w:b:R:",
+ c = getopt_long(argc, argv, "hVi:g:w:b:R:D",
long_options, &option_index);
if (c == -1)
break;
@@ -551,6 +562,9 @@ int main(int argc, char *argv[])
case 'R':
resource_mode = atoi(optarg);
break;
+ case 'D':
+ debug_mode = 1;
+ break;
case '?':
default:
usage();
@@ -679,14 +693,19 @@ int main(int argc, char *argv[])
log_error("set_event error %d ls %s", rv, ls_names[i]);
unregister_ls(i);
} else {
- log_notice("asked host %llu %llu to %s%s(%llx %llx) through ls %s",
- (unsigned long long)he.host_id,
- (unsigned long long)he.generation,
- (he.event & EVENT_RESET) ? "reset " : "",
- (he.event & EVENT_REBOOT) ? "reboot " : "",
- (unsigned long long)he.event,
- (unsigned long long)he.data,
- ls_names[i]);
+ log_debug("set event %llx %llx for host %llu %llu in ls %s:%d",
+ (unsigned long long)he.event,
+ (unsigned long long)he.data,
+ (unsigned long long)he.host_id,
+ (unsigned long long)he.generation,
+ ls_names[i],
+ ls_hostids[i]);
+
+ log_info("asked host to %s%sin ls %s:%d",
+ (he.event & EVENT_RESET) ? "reset " : "",
+ (he.event & EVENT_REBOOT) ? "reboot " : "",
+ ls_names[i],
+ ls_hostids[i]);
}
}
@@ -734,7 +753,7 @@ int main(int argc, char *argv[])
}
if (done) {
- log_notice("reset done in %u seconds", (uint32_t)(monotime() - begin));
+ log_info("reset done in %u seconds", (uint32_t)(monotime() - begin));
exit(EXIT_SUCCESS);
} else {
log_error("reset failed in %u seconds", (uint32_t)(monotime() - begin));
9 years, 2 months
reset/sanlk_reset.c reset/sanlk_resetd.c src/main.c src/resource.c
by David Teigland
reset/sanlk_reset.c | 4 ++++
reset/sanlk_resetd.c | 2 ++
src/main.c | 1 +
src/resource.c | 2 +-
4 files changed, 8 insertions(+), 1 deletion(-)
New commits:
commit 8b4c8866054666b4c04675a5fc9969705aeef36c
Author: David Teigland <teigland(a)redhat.com>
Date: Mon Sep 8 14:59:05 2014 -0500
sanlock, reset: coverity fixes
Minor things from coverity.
Signed-off-by: David Teigland <teigland(a)redhat.com>
diff --git a/reset/sanlk_reset.c b/reset/sanlk_reset.c
index 9b121e8..63e6616 100644
--- a/reset/sanlk_reset.c
+++ b/reset/sanlk_reset.c
@@ -311,6 +311,8 @@ static int reset_done(void)
if ((rv < 0) || (hs == NULL) || (hs_count != 1) || (hs->host_id != host_id)) {
log_error("sanlock_get_hosts error %d ls %s:%d",
rv, ls_names[i], ls_hostids[i]);
+ if (hs)
+ free(hs);
continue;
}
@@ -441,12 +443,14 @@ static int update_local_daemon(char *cmd)
rv = sendto(s, buf, UPDATE_SIZE, 0, (struct sockaddr *)&update_addr, update_addrlen);
if (rv < 0) {
printf("Failed to update local sanlk-resetd: %s\n", strerror(errno));
+ close(s);
return EXIT_FAILURE;
} else {
printf("Updated %s %s\n", cmd, ls_names[i]);
}
}
+ close(s);
return EXIT_SUCCESS;
}
diff --git a/reset/sanlk_resetd.c b/reset/sanlk_resetd.c
index d3ea158..3e4c8f5 100644
--- a/reset/sanlk_resetd.c
+++ b/reset/sanlk_resetd.c
@@ -451,6 +451,8 @@ static int setup_update(void)
return rv;
s = setup_resetd_socket();
+ if (s < 0)
+ return s;
unlink(update_addr.sun_path);
rv = bind(s, (struct sockaddr *) &update_addr, update_addrlen);
diff --git a/src/main.c b/src/main.c
index 5d3b270..cfd7f42 100644
--- a/src/main.c
+++ b/src/main.c
@@ -2776,6 +2776,7 @@ static int read_file_leader(struct leader_record *leader, int is_ls)
memset(line, 0, sizeof(line));
}
+ fclose(file);
new_checksum = leader_checksum(&lr);
diff --git a/src/resource.c b/src/resource.c
index a14de10..9834a84 100644
--- a/src/resource.c
+++ b/src/resource.c
@@ -2157,12 +2157,12 @@ int release_orphan(struct sanlk_resource *res)
count++;
}
}
- pthread_mutex_unlock(&resource_mutex);
if (count) {
resource_thread_work = 1;
pthread_cond_signal(&resource_cond);
}
+ pthread_mutex_unlock(&resource_mutex);
return count;
}
9 years, 2 months