> I increased the sanlock io_timeout to 30 seconds (default = 10),
because
> the sanlock dir is on a GFS2 volume and can be blocked for some time while
> fencing and journal recovery takes place.
This is one of the reasons why you should not put sanlock leases on gfs2.
They should be put directly on a shared block device.
I'm a little bit confused about the io_timeout option for
sanlock. I
increased the io_timeout to 30 seconds, but it seems like the overall
initialization becomes slower now.
libvirtd is the client through the sanlock plugin.
sanlock runs as "sanlock daemon -R 1 -o 30"
Restarting sanlock + libvirtd takes about 60 seconds before libvirtd
acquires the lease (or at least, before libvirtd starts responding).
After a reboot, for some reason, this delay increases up to 360 seconds...
I have no idea why this would take longer...
The libvirtd guys don't seem to know why this happens... so I hope to find
an answer on this list... I didn't find any other timeouts that are
configurable. From the source code, it looks like most of the timeouts are
based on the io_timeout.
Yes, all the timeouts are derived from the io_timeout and are dictated by
the recovery requirements and the algorithm the host_id leases are based
on: "Light-Weight Leases for Storage-Centric Coordination" by Gregory
Chockler and Dahlia Malkhi.
Here are the actual equations copied from sanlock_internal.h.
"delta" refers to host_id leases that take a long time to acquire at startup
"free" corresponds to starting up after a clean shutdown
"held" corresponds to starting up after an unclean shutdown
You should find that with 30 sec io timeout these come out to 1 min / 4 min
which you see when starting after a clean / unclean shutdown.
* io_timeout_seconds: defined by us
*
* id_renewal_seconds: defined by us
*
* id_renewal_fail_seconds: defined by us
*
* watchdog_fire_timeout: /dev/watchdog will fire without being petted this long
* = 60 constant
*
* host_dead_seconds: the length of time from the last successful host_id
* renewal until that host is killed by its watchdog.
* = id_renewal_fail_seconds + watchdog_fire_timeout
*
* delta_large_delay: from the algorithm
* = id_renewal_seconds + (6 * io_timeout_seconds)
*
* delta_short_delay: from the algorithm
* = 2 * io_timeout_seconds
*
* delta_acquire_held_max: max time it can take to successfully
* acquire a non-free delta lease
* = io_timeout_seconds (read) +
* max(delta_large_delay, host_dead_seconds) +
* io_timeout_seconds (read) +
* io_timeout_seconds (write) +
* delta_short_delay +
* io_timeout_seconds (read)
*
* delta_acquire_held_min: min time it can take to successfully
* acquire a non-free delta lease
* = max(delta_large_delay, host_dead_seconds)
*
* delta_acquire_free_max: max time it can take to successfully
* acquire a free delta lease.
* = io_timeout_seconds (read) +
* io_timeout_seconds (write) +
* delta_short_delay +
* io_timeout_seconds (read)
*
* delta_acquire_free_min: min time it can take to successfully
* acquire a free delta lease.
* = delta_short_delay
*
* delta_renew_max: max time it can take to successfully
* renew a delta lease.
* = io_timeout_seconds (read) +
* io_timeout_seconds (write)
*
* delta_renew_min: min time it can take to successfully
* renew a delta lease.
* = 0