[RFC] oplock with a guarantee of progress (description of the primitive) - take 2

Jeff Darcy jdarcy at redhat.com
Thu Jun 2 21:07:45 UTC 2011


On 06/02/2011 04:39 PM, Edward Shishkin wrote:
>> I don't think there's any reasonable way, without wire-protocol changes,
>> to pass a serial number along with a write.
> 
> 
> It's a pity: I kept a hope that there are "legal" means of
> inter-translator communication..

There are, in the form of "xattr" operations, but those are separate
network requests.  If we want to pass additional information *as part of
the read or write request*, that requires a protocol change.

>> The server needs to be able
>> to associate a write request which consumes a lease with an earlier
>> request that established the lease based only on the write request's
>> "normal" characteristics such as fd/start/end.
> 
> 
> Our initial goal is to serialize writes to the same atoms, i.e.
> writes with the same (fd, start, end). So making associations
> based on those parameters would be a logical mistake. Since per-atom 
> serialization is not easy to implement, I suggest to consider
> per-file serialization for now.

The start/end can be tweaked to the same atom boundaries we'll use for
the final write, but as a starting point file granularity is probably
OK.  We're not optimizing for fine-grain concurrent access to the same
file; we just need to make sure we don't corrupt data when there's
occasional contention.

> Thus, any rmw-request should have a unique id on the server. Yes,
> we can obligate clients to generate such ids, so that server won't
> need to pass it to the client. I think we need a simple generator,
> which increments the last used value. No collisions are possible.

Since the ID is only used for conflict detection, which is only done on
the server, I don't think the client even needs to see it.  As soon as
the fgetxattr reaches the head of the queue, the server can return the
head/tail bits and the client can proceed without an ID (which it has no
way to pass back anyway).

> So you suggest to pass a generated id to the server as an xattr value
> for some "magic" xattr name via ->fgetxattr(). Seems to be interesting
> idea.  Oplock will need to terminate such calls to not involve IO
> operations though. I'll try to develop such approach.

Yes, the magic fgetxattr starts out doing three things on the server.

(1) Generates a unique ID based on inode and (tweaked) start/end.

(2) Adds an entry with that ID.

(3) Checks the queue.

When the entry reaches the head, which will be immediately unless
there's contention, it does two more things.

(4) Reads the unaligned head/tail bits.

(5) Returns those bits to the client.

This is all part of one network round trip, with a possible delay in the
middle when there's contention, leaving only the final write.  Without
keys on the server, I don't think we can do any better than that.


More information about the cloudfs-devel mailing list