[PATCH] sanlock: host_message

David Teigland teigland at redhat.com
Wed Jan 29 17:16:09 UTC 2014


On Wed, Jan 29, 2014 at 07:49:29AM -0500, Nir Soffer wrote:
> > The host messages are sent from one host to another via
> > a lockspace that both hosts are using.  If no lockspace
> > name is specified, the sanlock daemon will search for a
> > common lockspace to use.  (N.B. hosts do not necessarily
> > use the same host_id in all lockspaces, so not specifying
> > the lockspace could result in targeting the wrong host.)
> 
> I think that making the lockspace a required parameter makes
> more sense and will avoid fatal errors.

You should specify the lockspace if you know it, then this won't matter.

> > The lockspace used to transmit the message may or may not
> > have any other relation to the message itself.
> > 
> > A host can send one message to a one other host at a time.
> 
> Can we increase this number, simplifying (unlikely) case where
> more then one host need to be fenced?

This was the one big question I had about the design.  If it's necessary
to address more than one host simultaneously I can do that, but I'll need
to go back and come up with a more complex design.  The existing design is
simple (and completely compatible with the existing format) because it
uses three unused fields in the delta lease area.  So, perhaps think a
little more about how important this would be and let me know.

> > The message is placed in the sending host's delta lease,
> > and remains there for two renewals.  When the receiving
> > host renews its own delta lease, it checks the delta leases
> > of all other hosts, and sees itself addressed in the sending
> > host's lease.  It then processes the message from the
> > sending host.
> 
> Why did you choose the keep the message for two renewals?

Because the targeted host would generally observe it in that time.

> We would like to have this value configurable, to make
> it easier to solve issues in the field.

How configurable would you need:
1. a daemon config option (set it when the daemon starts)
2. an duration-based api option (set it when you call the function
   in terms of seconds to remain active.)
3. an on/off api option (one function call to set the message,
   and a second function call to turn it off.)

> I think we have to handle the (unlikely) case, where a host
> lost its lease without seeing the WD_RESET message, then 
> acquire the lease again (not sure if this is possible in vdsm
> currently).

This case would require you to send another message because a host_message
is addressed to host_id:generation.  The generation number of a host_id is
increased when it joins the lockspace again.  (Before we derive too much
from this example, I think we'd want to outline exactly what the case is
and what we want to happen.)

> The fencing host may assume wrongly that the host
> was fenced in this case.

Whether the message is received or not, sanlock should never make an
incorrect conclusion about the safe/unsafe state of a failed host.

It is not based on the host_message being received or not, but based only
on the existing link between delta lease renewals and watchdog renewals on
the targeted host.

(Remember that a failed host may actually be really dead and incapable of
doing anything, e.g. powered off.  sanlock can't distinguish this from
other failure cases, but needs to work regardless.)

If a program outside of sanlock, like vdsm, wants to depend on sanlock's
determination about the state of another host, you have two ways to do it:

1. wait until sanlock grants you a lease that was held by the other host
2. wait until sanlock_get_hosts() returns FREE or DEAD for the other host

The WD_RESET host_message is not a factor there.  The WD_RESET is a way to
force a partially failed host to be fully failed so that one of the two
options above can be effective more quickly.  i.e. it quickly reduces some
varieties of partial failures into full failures from which our programs
can go forward, safely forgetting about the failed host.


> What if we leave the WD_RESET message until the fencing host
> send a WD_UNRESET message?

- The other host may be dead and incapable of doing anything.

- It's only purpose would be to let the initiating host know that
  it could clear its message.  This provides very minimal benefit
  over other forms of clearing the message.

- Bidirectional send message / acknowledge message would be more
  complicated to implement than we want to do here.
  (I've been thinking about acknowledgments for another types
  of messages that I'd like to use elsewhere, and I've given up
  and gone with other ideas.)

> This way I can send a WD_RESET message, wait for some renwals,
> ensuring that the host either lost it's lease, or will *not* 
> get a new lease, until I decide to allow the host to get one.

I don't understand what you're trying to do there.

As I mentioned above, sending WD_RESET host_message should be completely
orthogonal to the logic around leases.  They are not related.

> We may have a case where we cannot access a host, we fence it,
> ensuring that it cannot access the storage, but the host never
> see the fence request, and keeping it in "fenced" mode is 
> required until we can reboot the bost using power management 
> or manually.

I don't understand this.  Maybe it's helpful to think about the
differences among:

- power fencing
  . toggle the power of victim host on a switch
  . assume that worked
  . let programs use locks/resources that the victim had been using

- storage fencing
  . cut off storage access of victim host by turning off a switch port,
    (or removing its SCSI persistent reservation)
  . assume that worked
  . let programs use locks/resources that the victim had been using

- sanlock/wdmd/watchdog lease protection
  . wait for a fixed timeout from the victim host's last storage renewal
  . assume that the victim's watchdog has reset due to no lease renewal
  . let programs use locks/resources that the victim had been using

- sanlock/wdmd/watchdog lease protection + WD_RESET host_message
  . send victim the WD_RESET host_message, which would cause the victim
    to force it's own watchdog to expire in a minute
  . assume nothing about the receipt or effect of WD_RESET
  . wait for a fixed timeout from the victim host's last storage renewal
  . assume that the victim's watchdog has reset due to no lease renewal
  . let programs use locks/resources that the victim had been using

Notice that:
- the end goal/result is the same in all cases
- there are assumptions made in all cases
- the first two are usually much quicker than the second two because
  the feedback from the third party (switch) is the basis of success
- the results of WD_RESET are not a factor in the end result
- if a host is partially operating and can receive sanlock messages
  from storage, then WD_RESET can speed up the progression
- if a host is partially operating and has lost all storage access, 
  the WD_RESET does nothing, but the end result is the same


> > Because host messages may not be received if the
> > destination host fails, or looses storage access,
> > there are no guaranteed times associated with the
> > delivery, processing or effect of a host message.
> > Guaranteed times for another host being dead should
> > continue to be based on either acquiring a resource,
> > or sanlock_get_hosts().
> 
> What would be the best way to detect that the host was fenced?

See the two options I listed above (acquire lease or use get_hosts state)

> > TODO: will be adding another msg_num to cause the
> > destination to use /proc/sysrq-trigger to reboot itself.
> > (After setting up the watchdog to reset the machine in
> > case the sysrq mechanism fails.)  The sysrq reboot is
> > immediate, whereas the watchdog takes a minute to reset.
> 
> Maybe use WD_FENCE, and let sanlock use the best available
> method for fencing? 

We want the target host to reset itself as dependably and as quickly as
possible.  Using the watchdog is dependable, and using sysrq-trigger is
quick, so the idea is to combine them to get the best of both.

> Can we customize the actions taken by sanlock when receiving
> the WD_RESET message? For example, running a script after
> the message was received?

That kind of thing is already supported using the sanlock_request()
interface, which is based on a resource lease held by the target host.
There is room to extend this with new options if needed.

> What would be the best way to detect if a host supports
> the new fencing feature - check its sanlock version?

Yes, that will probably work.

Dave


More information about the sanlock-devel mailing list