On Tue, 12 Jul 2011 20:02:39 +0200
Edward Shishkin <edward(a)redhat.com> wrote:
In the case of whole file leasing we discussed here:
http://fedorahosted.org/pipermail/cloudfs-devel/2011-June/000075.html
we assume that any two not-fully-aligned writes are conflicting
(a fully aligned ones conflicts with nothing).
That is incorrect. A fully aligned write can indeed conflict with a
read-modify-write, in the sense that if they are not processed
sequentially then data corruption can result.
And what is in your
case?
Any two writes that will actually modify the same byte range (even if
it's just padding to the next cipher-block boundary) will conflict. As
you can see in my reply to the message you reference above, file
granularity is probably enough to start with, but it's bad protocol
design to preclude finer granularity.
>> I think client should pass a 128-bit generation counter.
And
>> server should append (or prepends) a unique client id.
>
> There is no way for the client to pass either of these back as part
> of the write, so that the write can be associated with the earlier
> lease operation.
Why do we need to pass it back?
Client knows his id that he has generated.
He can store it in the local area for example.
Yes, the client knows it, but the server needs some way to associate a
lease-acquire operation with a subsequent write. It can only do this
using the write's basic parameters such as offset and length, because
there is no way to pass any client-generated ID (no matter how it's
generated) along with the write.
>> oplock->writev_cbk(), and oplock->truncate_cbk can
drop
>> the lease. We must know the atom size on the server side
>> though (to recognize RMWs). But IMHO this is not a
>> problem (we can store this in xattrs (**) for example..)
>
> Knowing the atom size is not enough; we need to know the actual end
> of the user's write
IMHO we can obligate the oplock xlator to align offsets and to
round-up counts. If oplock has received fully aligned data,
then set a flag to the local area to not drop a lease by
oplock->writev_cbk(), oplock->(f)truncate_cbk(), etc..
...and if it receives unaligned data, what then? The logic to deal
with the case where the head and tail bits are all in the same
"atom" (so that we only need to fetch and return that atom once) is
non-trivial. Should that be replicated on the server? That's a
protocol rife with potential for mismatches between what the client
needs/expects and what the server actually sends. That's why the
protocol has this logic entirely on the client, with the server (which
has enough other things to do) merely returning what it's told to.
>> I don't see any reasons why reads must be aware of this
>> protocol. What problems are resolved by making them aware
>> of this? We do know what are conflicting writes, but what
>> are conflicting read and write?
>
> Atomicity (of the read-modify-write sequence) requires that
> inconsistent intermediate states not be observable by other
> requests. Between the time a writev actually hits the disk on the
> server and the time we finish doing EOF-related fixups we're in
> such an inconsistent state. If we're extending the file and the
> read is near the new EOF, this could result in a read receiving
> more data than is actually supposed to be in the file.
Supposed by whom?
Could you please point to any (POSIX?) documentation, which
regulates such relationships between read(2) and write(2)?
It doesn't take a POSIX expert to know that if nobody ever even tried
to write past point X (or extend past X via f/truncate) then no read
should return data past X. Does that really require a specific
citation?
If one process reads a file, and second process writes to
the same file, then you can NOT predict the "winner" without
additional serialization means. Such means are NOT provided
by local file systems in the kernel. Why should Cloudfs
bother with them?
This isn't about serializing whole requests. It's about intermediate
states that exist within one request, and which would represent POSIX
violations if they were observable by others.
> If we're extending the file and the read
> is near the *old* EOF, which requires moving the old end fragment
> from an xattr to its "proper" location after the old EOF, then the
> read could get garbage.
Here we can see that the need in reads-and-oplock
interation is artificial (i.e. stipulated by the
EOF-in-xattr design solution).
I guess you could look at it that way. Without solving the problems
inherent in its alternative, though, it seems rather obstructionist.
>> This makes things queasy. Every such removal can cause
rebalance of
>> local-fs storage. Could you please remind why to not have files
>> with rounded-up sizes on server? After all we can keep their
>> actual size in xattrs and to not bother with its removal...
>
> With that approach, we'd have to intercept every lookup, stat, etc.
> to ensure that we return the correct st_size.
Yes. And what is wrong in such interception?
Performance.
> Similarly, every read would
> have to check the xattr value to ensure that it doesn't return data
> past EOF even though the underlying local FS will.
This insurance is bogus. I have already explained above:
there is no inconsistency points between reads and writes.
Your "explanation" was itself erroneous. Fetching an xattr on every
read so that we can ensure it doesn't cross the "true" EOF even though
the lower levels returned more data would be unacceptable.
I suggest to implement an approach with a whole file leasing
for the beginning (which has been discussed here:
http://fedorahosted.org/pipermail/cloudfs-devel/2011-June/000075.html
) and store aligned files on local fs with the following respective
size translations. This requires to have a persistent atom size
and crypto algorithm id in the private part of inode on the server.
We can keep/load them in/from xattr. (Imho not the same as
EOF-in-xattr horror).
How long will it take you to exchange one set of horrors for another?
Is there any reason to suppose it will be done and debugged to the same
level as this code before the Fedora 16 deadline? I see none.
One more reason to adhere this approach is supporting HMACs
for authentication: where are we going to store them? In xattrs?
IMHO it is not serious. They should be placed in file's body
right after respective atoms.
A moment ago you were complaining about fragmentation and performance,
now you want to intersperse data with metadata at a 16-byte
granularity? In any case, authentication is not part of the current
deliverable. Since it is useful for purposes other than crypto,
Gluster has expressed willingness to extend the wire protocol for this
purpose in time for the next deliverable, and there will probably be a
separate translator to handle that. It need not - should not - affect
how we do encryption.