On 09/13/2011 08:49 PM, Etsuji Nakai wrote:
Hi,
Here's a simple question.
- Forward the write to all N replicas 2. If all N replicas
indicate success, we're done 3. If any replica fails, add information about the failed request (e.g. file, offset, length) to journals on the replicas where it succeeded 4. As part of the startup process, defer completion of startup until brought up to date by replaying peers' journals
What would happen if the primary (forwarder) node fails in the middle of writing its own local file (i.e. before starting the replication)? The file on the primary node is now corrupted, and should be roll-backed to be synced with other replicas. How would it be achieved?
That's a very good question, Etsuji. In fact, Kaleb raised pretty much the same issue offline after I'd sent the previous email. I had already written some stuff into the document addressing that, but since it has come up twice already I'll address it here as well.
First, I would not say that the file on the server where the write succeeded (there's no real primary in this architecture) is corrupt. It's just more up to date that on a server where the same write failed. The key here is that it can update its journal *locally* as part of the write itself. This would involve extra disk I/O but not extra network traffic. (It occurs to me BTW that the same optimization could possibly be applied to AFR as it is now.) If the journal is fully updated before the write itself is attempted, then that server has sufficient information to forward the write to other nodes even if the client that sent it dies - and in fact even if there's a total power outage taking out the servers as well. Journal entries can then be retired *asynchronously* when all servers have been updated. This involves network traffic, but doesn't affect latency. In some orderings this could lead to a journal entry being replayed on a node where it had already succeeded, but - assuming that other mechanisms are in place to deal with ordering (see next message) - that's OK.