I had always wanted to take 6.824. I had heard that if you wanted to do backend work, this course was required.

At the end of the year, I found out that it had been renamed. 6.5840.

Before taking this course, or reading DDIA, my understanding of distributed systems was roughly this: if one machine is not enough, add a few more. Let each of them do part of the work, then exchange results through RPC.

After finishing the course, I finally understood that having more machines is only the surface. The hardest part of a distributed system is that you cannot easily tell whether something has happened at all.

A local function either returns or throws an error. A remote call that times out can mean many different things: the request never arrived; the server has not processed it yet; the server processed it, but the reply was lost; or the reply is simply stuck somewhere in the network. Meanwhile, different machines may be living in different terms, holding logs of different lengths, with different answers to the question, “Who is the Leader right now?”

So what this course really taught me was not how to write a few RPCs. It taught me how to maintain a history that can still be trusted when information is incomplete, messages arrive out of order, and machines can disappear at any moment.

Failure Is Not a Clear-Cut Event

In a single-process program, when one module crashes, the whole process usually knows. In a distributed system, only part of the system is often broken: one Worker loses contact while the others keep running; one replica is cut off by a network partition but does not know it; a client receives no reply and cannot tell whether the server executed its request.

This is called a partial failure. What makes it so unpleasant is that, from the outside, “failed” and “slow” look almost the same.

Usually, all the system can do is set a timeout and then act. When a MapReduce Coordinator cannot hear from a Worker, it gives the task to someone else. When a KV client does not receive a reply to Put, it tries again. But a timeout only means that a local timer expired. It is not proof that the remote side failed. The old Worker may return later. The first Put may already have succeeded.

So every retry has to answer one question first: if the original operation actually completed, what happens when I do it again?

MapReduce tasks must be safe to execute more than once. The KV Server uses versions and conditional writes to prevent the same change from taking effect twice. More generally, a system needs to give an operation a stable identity, or make the operation idempotent, so that running it again creates no additional effect.

I used to think of retry as the last few lines inside a try-catch block. Now I would rather think of it as part of the protocol itself.

As long as communication can fail, the normal path must include duplicates, late arrivals, and uncertain outcomes.

Consistency Does Not Mean “The Data Will Eventually Match”

It is easy for a distributed system to make its replicas look roughly the same after enough time has passed.

The hard part is giving the outside world behavior it can understand while the system continues to accept reads and writes.

The KV service in 6.5840 requires linearizability. There is a very plain way to describe it: from the client's point of view, every operation appears to happen, one by one, in some order, on a single machine.

Not the very common, so-called eventual consistency.

If a write has returned successfully, a read that starts afterward must see it. Two concurrent writes may happen in either order, but the final result must be explainable by some single-threaded ordering.

Internally, the system may have replicas, retries, elections, and logs. The client should not see the cracks torn open by those mechanisms.

What matters about this definition is that it draws a boundary around the histories the system must never produce. An old Leader that has lost contact cannot keep acknowledging writes. A stale replica cannot casually answer the latest reads. A shard in the middle of migration cannot be served by both the old and new groups at once. Otherwise, the results observed by clients can no longer be arranged into one coherent timeline.

The Lab asks us to implement a distributed lock on top of the KV service. It is a good test. The lock itself does not take much code, but if conditional writes or retry semantics are even slightly ambiguous, two clients may both believe they acquired it.

At that point, consistency stops being a term from a paper. It becomes a promise that directly determines whether the program above it can work correctly.

Raft Replicates History, Not Data

When I first started writing Raft, I thought of it as a synchronization mechanism: data changes on the Leader, so send those changes to the Followers.

Later, I realized that what Raft really replicates is an ordered sequence of operations. If every replica starts from the same initial state and executes the same commands in the same order, they arrive at the same state. This is a replicated state machine.

The question changed from “How do I synchronize a few maps?” to “How do I keep a history moving forward, and have a majority recognize it?”

Raft uses terms to distinguish one generation of Leaders from the next, elections to decide who may propose new log entries, and majorities to confirm which entries have become shared history.

The point of a majority is not merely that it has more than half the votes. It is that any two majorities must intersect. A new Leader cannot win majority support while completely bypassing every node that participated in an earlier commit.

The easiest thing to overlook here is time. A node may still be a Candidate when an RPC leaves, but already be in a higher term when the reply returns. An old log-replication reply may arrive after one from a newer request. If the system looks only at the reply, without checking the term, role, and log position again, a message from the past can rewrite the present.

So what Raft really trains is a habit for handling asynchronous results: every message must be judged inside the context that produced it. A result can have been correct once, and still be stale now.

Consensus, however, does not directly solve the entire service. Raft can decide where a command belongs in the log, but it does not know whether the client received the reply, or what happens if a business operation runs twice. The KV Server still has to handle request identities, waiting, and retries. Consensus provides a reliable order of operations. The service above it still has to give that order the right business meaning.

Recovery Is Not Restarting. It Is Continuing the Past

An in-memory Raft can survive a shaky network, but not a process restart. If a node forgets whom it voted for, which term it is in, or which log entries it stored, it may contradict a history that has already been decided.

Persistence, then, is not an archival feature added at the end. Some state has to reach stable storage before it is allowed to affect the outside world. When a machine comes back, it does not start over. It rejoins the protocol carrying the promises it made in the past.

The log cannot grow forever, either. The longer the system runs, the more history it has to replay after a restart, and the longer a lagging node takes to catch up. Snapshots solve this problem: the state machine compresses a prefix of executed log entries into its current state, and Raft keeps only the useful tail that follows it.

A snapshot is not an ordinary backup. It must correspond exactly to the log prefix it replaces, and it must be able to stand in for that history during recovery. Once snapshots exist, a logical log index is no longer the same thing as an array index in memory. A Follower that is too far behind can no longer receive the old log one entry at a time. It has to install a snapshot, then continue from the new position.

Almost every database-related implementation runs into snapshots eventually.

I use a similar mechanism in my own project, Drifting. Yjs handles multi-device synchronization and conflict resolution, with incremental updates flowing across devices. But a new device will always begin by “getting a snapshot.”

The Hard Part of Scaling Is Transferring Ownership

One group of Raft replicas can tolerate failures, but every request still has to pass through the same log. Eventually, throughput reaches a limit. Sharding assigns different keys to different shard groups, allowing multiple replicated state machines to work in parallel.

Static sharding is simple. The hard part is reconfiguration while the system is running: new groups join, old groups leave, and hot data needs to move. What is being moved is not merely a set of key-value pairs. It is the ownership of the right to answer for those keys.

If both the old and new groups serve the shard during migration, the same key splits into two diverging histories. If the old group has stopped but the new one is not ready, the data becomes unavailable for too long. A client may also have cached an old configuration and keep sending requests to a server that no longer owns the shard.

Handling this requires versioned configurations, migration stages that can be recognized and safely repeated, and durable records of unfinished intent. If a controller crashes halfway through, the next controller can determine how far the migration progressed instead of guessing from an ambiguous intermediate state. When multiple controllers run concurrently, they need the same kinds of version and ownership constraints to avoid overwriting one another.

At this point, I realized that the control plane is itself a distributed system. A controller that “manages the other nodes” is not naturally more reliable. It can crash, lose its network, or repeat an operation like anything else, and it has to live under the same rules of failure.

Redis is known for high-throughput writes. Its data plane uses asynchronous replication, but its high-availability and cluster control plane—Sentinel, for example—uses epochs, quorums, and leader election. Group Replication in MySQL InnoDB Cluster uses a similar consensus mechanism to provide fault tolerance.

The Questions That Stayed with Me

Looking back, MapReduce, KV, Raft, replicated state machines, and sharded KV are only the forms the course uses to carry these ideas. What keeps returning are these questions:

• When no reply arrives, what might already have happened “out there”?
• When an operation is retried, how does the system recognize that it is still the same operation?
• How can multiple replicas present one continuous, singular history to the outside world?
• After a machine restarts, which promises must still hold?
• If state or ownership is transferred only halfway, how can someone else continue the work?

Now, when I am given a distributed protocol, the first things I look at are no longer how it handles concurrency or how much throughput it has. I look at how it expresses “I don't know,” how it rejects stale messages, and how it moves forward from any intermediate state.

That is probably the biggest change 6.5840 made in me.

A distributed system is not a single-machine program copied a few times. It is a system that maintains one reality the outside world can trust, even without a global view, and without a reliable way to tell whether something has failed.

MIT's collaboration policy does not allow Lab code to be made public. But compared with the implementation details, the ideas that run through the entire course are what benefited me the most.