[PATCH] sanlock: host_message

David Teigland teigland at redhat.com
Wed Mar 5 22:32:42 UTC 2014


I visited with Nir and one of the big problems with my initial
host_message design was the lack of acknowledgements, which I'd
been strongly resisting.

I've come up with a new design that could be a workable way of doing host
messages with acknowledgements.  I don't like it very much, but will give
it a try.

There are three 64 bit fields in the delta lease leader record that we can
use as follows:

field 1:
uint32_t send_to_host_id;         /* message destination */
uint32_t send_to_host_generation; /* message destination */

field 2:
uint32_t send_msg;                /* the caller-specified message */
uint32_t send_seq;                /* internal sequence number */

field 3:
uint32_t recv_from_host_id;       /* acknowledgement: message source */
uint32_t recv_seq;                /* acknowledgement: send_seq */

host_id and host_generation are 64 bit values everywhere else in sanlock,
and this shortens them to 32 bits to fit them into the available space.
Realistically, they should always fit in 32 bits, but it's ugly.

This also removes the 32 bit "data" field that could previously be sent
along with the 32 bit msg number.  160 bits of overhead for a 32 bit
message is a little sad.


Sending a message
-----------------

int sanlock_host_message(const char *ls_name, uint32_t flags,
                         int hm_size, struct sanlk_host_message *hm
                         uint32_t *send_seq);

struct sanlk_host_message {
        uint64_t host_id;
        uint64_t generation;
        uint32_t send_msg;
}

The sending host sets the following in its delta_lease:

field 1:
send_to_host_id = hm.host_id & 0xFFFFFFFF;
send_to_host_gen = hm.generation & 0xFFFFFFFF;

field 2:
send_msg = hm.send_msg;
send_seq = local_msg_seq++;

send_seq is returned to the caller for matching an acknowledgement.


Receiving a message
-------------------

The receiving host sees its own host_id/generation in the sending host's
lease, processes send_msg, and saves host_id/seq in a list of messages to
be acknowledged.

At the next delta lease renewal, it takes the next host_id/seq from its
list and sets:

field 3:
recv_from_host_id = the host_id that send the message;
recv_seq = the seq number that accompanied the message;


Receiving acknowledgement
-------------------------

sanlock will not keep any state about the host messages it has sent or try
to match acknowledgements.  But, sanlock does keep track of other host's
delta lease state, and that could include recv_from_host_id/recv_seq.  We
can add an api for the caller to query the recv_from_host_id/recv_seq for
a given host_id.

In the caller, sanlock_host_message() returned the send_seq value that was
used for the message.  After this, the caller would query sanlock for the
recv_seq until it matched send_seq (or until it wants to give up.)


Problems with the acknowledgement scheme:

1. It will not work with a fast reset option using /proc/sysrq-trigger
   because there will not be enough time for the acknowledgement to be
   written before the host is reset.  (With another independent message
   area, we could write an acknowledgement immediately, but borrowing the
   lockspace lease means we do not have this option.)

2. If multiple hosts send messages to a single destination at once, the
   destination host will need to acknowledge them one at a time in
   consecutive renewals.  It takes longer to get an ack, each ack would
   be visible for one renewal and could be missed.



More information about the sanlock-devel mailing list