Post

Replication Strategies

Replicate data across multiple nodes to improve read scalability, availability, and disaster recovery -- choose your topology (single-leader, multi-leader, leaderless) and sync mode based on your durability/latency trade-offs.

Posted Updated
By
20 min read
Replication Strategies

Replication is a trade between durability, latency and availability. Every topology answers two questions: who may accept a write, and how many copies must confirm it before the client hears “OK”. Get those two answers right for your workload and the rest is tuning.

Key Properties

Property Single-Leader Multi-Leader Leaderless (Quorum) Consensus (Raft / Paxos)
Who accepts writes One primary Any leader Any node (coordinator) Elected leader per group / shard
Write conflicts None Yes — must be resolved Yes — concurrent writes to a key (LWW, vector clocks) None
Replication lag ms to seconds (async replicas) ms to seconds between leaders Depends on W/R; stragglers fixed by repair Majority is always current; minority may lag
Failover 30 s – 2 min (detect + promote) None needed for writes None needed Automatic election, typically seconds
Consistency Strong on primary, eventual on replicas Eventual Tunable; W+R>N gives overlap, not linearizability Linearizable
Examples PostgreSQL, MySQL, MongoDB CouchDB, EDB PGD / pgEdge, DynamoDB global tables (MREC) Cassandra, ScyllaDB, Riak etcd, CockroachDB, Spanner, TiDB, YugabyteDB
Typical use OLTP, read scaling Geo-local writes, offline-first Very high write throughput, always-on Metadata, distributed SQL, strong geo-consistency

Replication Fundamentals

How Replication Works

All replication systems follow the same basic pattern:

  1. Write goes to leader(s) — Client sends write to primary/leader node(s).
  2. Leader applies write to local state — Update is written to leader’s database.
  3. Leader sends change to replicas — Write operation (or full state change) is sent to replica nodes.
  4. Replicas apply change — Each replica updates its local copy.
  5. Client is acknowledged — After replicas confirm (or immediately in async mode), client gets ACK.
sequenceDiagram
    participant C as Client
    participant L as Leader
    participant F as Follower
    C->>L: write
    L->>L: apply + append to replication log
    L-->>F: ship log entry
    F->>F: replay entry
    F-->>L: ack (only awaited in sync mode)
    L-->>C: ack

The key variation is when the client is acknowledged:

  • Synchronous: After replicas apply the write (durability guaranteed, but higher latency)
  • Asynchronous: Immediately after leader applies (low latency, but data loss possible if leader crashes)
  • Semi-synchronous: After leader + one replica apply (compromise)

Single-Leader (Primary-Follower) Replication

Architecture

One primary (leader) node accepts all writes. Multiple replica (follower) nodes asynchronously replicate the primary’s changes by consuming a replication log (PostgreSQL: WAL, MySQL: binlog, MongoDB: oplog).

flowchart LR
    W["Client writes"] --> P["Primary"]
    P -- "WAL / binlog shipped" --> R1["Replica 1"]
    P -- "WAL / binlog shipped" --> R2["Replica 2"]
    R1 --> RD["Client reads<br/>(eventual consistency is OK)"]
    R2 --> RD

All writes must go to the primary; reads can be spread across replicas.

How Replication Works in Detail

PostgreSQL Streaming Replication

PostgreSQL ships the Write-Ahead Log (WAL) to replicas. Every transaction is written to the WAL on disk before the database heap pages are modified. Replicas connect to the primary and consume the WAL stream:

  1. Primary receives INSERT INTO users VALUES (1, 'alice').
  2. Primary writes this to the WAL file (000000010000000000000001).
  3. Primary modifies heap pages in memory/disk.
  4. Replica reads from WAL stream: LSN=0/16001234 (Log Sequence Number)
  5. Replica replays the WAL: applies the INSERT to its own heap.
  6. Replica updates its own LSN to confirm it’s replayed up to that point.

Synchronous vs Asynchronous:

  • Asynchronous (default): Primary acknowledges immediately after writing WAL locally. Replica replication happens in the background. If primary crashes before replica replicates, data is lost.
  • Synchronous: Primary waits for replica to acknowledge it has written the WAL before responding to client. Guarantees replica has durable copy before client succeeds. Adds one network round trip per commit: about 1 ms within a datacenter, 50–150 ms across continents.

Real production numbers: A medium-sized PostgreSQL primary (100GB database, 10,000 QPS) generates ~500MB of WAL per hour. Streaming to a replica in the same datacenter incurs ~1ms latency; shipping to another continent adds 50–200ms depending on network quality.

MySQL Binlog Replication

MySQL records all data-modifying statements (or row changes) in the binlog. Replicas connect to the primary and download binlog events:

1
2
3
4
5
6

Primary binlog:  pos 154  INSERT INTO orders VALUES (...)
                 pos 289  UPDATE orders SET status='paid' ...

Replica:  IO thread asks for events from pos 154 -> writes them to its relay log
          SQL thread applies INSERT, then UPDATE
          records "executed up to pos 289" (or the GTID set, in GTID mode)

Reads (SELECT) never appear in the binlog — only changes do.

Gotcha — statement-based vs row-based:

  • Statement-based (SBR): Replicate the actual SQL statement. Fast for large bulk operations. Problem: non-deterministic functions (NOW(), RAND()) may produce different results on replica.
  • Row-based (RBR): Replicate the actual row changes (INSERT these specific columns). Larger binlog (every row change becomes an event). Deterministic — replica always produces identical results. Most production systems use RBR.

Real production numbers: A high-write MySQL primary (50,000 INSERTs/sec, row-based replication) generates ~2GB of binlog per hour. Replicas lag behind by 10–100ms depending on network congestion.

Read-Your-Writes Consistency Problem

Single-leader replication has a consistency gap: replicas lag behind the primary.

Scenario:

1
2
3
4

T1: Client writes to primary: UPDATE profile SET status='online'
T2: (1ms later) Primary returns OK
T3: Client immediately reads from replica: SELECT status FROM profile
T4: Replica hasn't replicated yet, so client sees status='offline' (stale!)

Solutions:

  1. Read-after-write consistency: For user’s own data, always read from primary. 
    1
2
3
4
5

    if (user_id == current_user_id) {
    read from primary  // Guarantee we see our own writes
} else {
    read from replica  // Eventual consistency OK for others' data
}
  2. Timestamp-based read routing: Client provides write timestamp; replica waits until it has replicated that timestamp before serving reads. 
    1
2

    primary_version = client.write(data)  // returns version 42
replica.read_with_version(42)         // wait for replica to reach version 42
  3. Session-based consistency: Use sticky sessions; route all requests from same client to same replica, which guarantees it has replicated all previous writes.

Failover Challenges

When the primary crashes, the system must detect failure and promote a replica to primary.

Failure detection latency: How fast does the system realize the primary is down?

  • Active health checking (ping) can detect failure in 100–500ms.
  • In production, you usually see 30–60 seconds of silence before manual intervention or timeout kicks in.

Failover process:

flowchart LR
    A["Detect primary failure<br/>30-120 s"] --> B["Choose replica<br/>highest replicated LSN"]
    B --> C["Promote to primary<br/>a few seconds"]
    C --> D["Redirect clients<br/>seconds to minutes"]
    D --> E["Re-point other replicas<br/>to the new primary"]

Total downtime: 30 seconds (detection) + 10 seconds (promotion) + 30 seconds (client redirect) = ~70–90 seconds in a well-automated system.

Data loss scenario: If the primary crashes after writing to local disk but before replicating to any replica (asynchronous mode), committed data is lost. The newly promoted replica won’t have that write. In synchronous mode, you guarantee at least one replica has the data, so failover has zero data loss (but higher write latency).

Multi-Leader (Multi-Master) Replication

Architecture

Multiple nodes are active leaders. Each leader accepts writes; leaders replicate their changes to each other.

flowchart LR
    A["Leader A<br/>San Francisco"] <-->|"async replication"| B["Leader B<br/>London"]
    A --> A1["Replica A1 (US)"]
    B --> B1["Replica B1 (EU)"]

Both leaders can serve write traffic simultaneously. This is critical for geo-distributed systems where sending all writes across the Atlantic to a single primary adds 50–100ms latency.

Write Conflict Resolution

With multiple leaders, the same data can be written on different leaders simultaneously:

Scenario:

sequenceDiagram
    participant SF as User (SF)
    participant A as Leader A
    participant B as Leader B
    participant LN as User (London)
    SF->>A: status = 'online'
    LN->>B: status = 'offline'
    A-->>B: replicate 'online'
    B-->>A: replicate 'offline'
    Note over A,B: Both leaders now hold conflicting values

Resolution strategies:

  1. Last Write Wins (LWW): Each write has a timestamp. Later timestamp wins. Problem: data loss (one write discarded). Acceptable for non-critical data (user status, cache).

  2. CRDT (Conflict-free Replicated Data Type): Data structure that is designed so concurrent writes always merge correctly.
    • Counter: Operations are commutative; +1 on A and +1 on B both result in +2 regardless of merge order.
    • Grow-only set: Only supports add, not remove. Conflict-free by definition.
    • Register with metadata: Each write includes a unique replica ID; merge by comparing metadata (e.g., replica ‘A’ beats replica ‘B’ lexicographically).
  3. Application logic / Manual resolution: Detect conflicts and ask application/user to resolve.

Use Cases

  • Offline-capable apps (CouchDB on mobile): Changes are replicated when connection is available; conflicts are resolved via CRDTs or application logic.
  • Geo-distributed systems (multi-datacenter masters): Each datacenter has a leader for local write latency. Conflicts rare if data is partitioned by geography (different users in different regions).
  • Collaborative editing (Google Docs): Uses OT (Operational Transformation) or CRDTs to merge concurrent edits.

Real Systems

  • CouchDB: Every replica accepts writes; conflicting revisions are kept in a revision tree, one is picked deterministically as the winner, and the losers stay available for the application to merge.
  • PostgreSQL active-active (EDB PGD, pgEdge): Bidirectional logical replication with configurable conflict resolution (often LWW per row).
  • DynamoDB global tables (default MREC mode): Every region accepts writes; replication is async (typically under a second) and conflicts resolve last-writer-wins.
  • MySQL Group Replication (multi-primary mode): Certification-based; conflicting transactions are rolled back rather than merged.

Leaderless (Quorum-based) Replication

Architecture

Every node is equal. Writes go to multiple nodes in parallel, read from multiple nodes in parallel. No master/follower distinction.

flowchart LR
    C["Client"] -- "write to all N, wait for W acks" --> N1["Node 1"]
    C --> N2["Node 2"]
    C --> N3["Node 3"]
    N1 -- "read from R nodes,<br/>return highest version" --> C
    N3 --> C

Quorum Mathematics

The quorum intersection property ensures that reads and writes overlap:

If W + R > N, then every read overlaps with every write.

flowchart TB
    subgraph W["Write set (W=2)"]
        N1["Node 1"]
        N2["Node 2"]
    end
    subgraph R["Read set (R=2)"]
        N2b["Node 2"]
        N3["Node 3"]
    end
    N2 -. "same node: the overlap<br/>guarantees the read sees the write" .- N2b

With N=3, W=2, R=2: 2+2=4 > 3, so any write set and any read set must share at least one node.

Consistency table:

N W R Consistency Example Use
3 1 3 Overlapping (R=N) Fast writes, slow reads; any node down blocks reads
3 2 2 Overlapping (W+R>N) Cassandra QUORUM; the usual balance
3 3 1 Overlapping (W=N) Fast reads; any node down blocks writes
3 1 1 Weak (W+R≤N) Write one; read any (may be stale)

“Overlapping” means a read sees the latest acknowledged write under normal operation. It is not linearizability: sloppy quorums with hinted handoff, concurrent writes resolved by LWW, and partially failed writes can all still return surprising results (Kleppmann, DDIA, ch. 5).

Read Repair & Anti-Entropy

Nodes may temporarily have stale data. Recovery happens via:

  1. Read repair: During a read, if one node has stale data and another has newer data, the stale node is updated immediately.
  2. Anti-entropy (background repair): Periodically compare all replicas and fix mismatches (Merkle tree hashing).

Use Cases

  • High write throughput: Write to W nodes in parallel (typically W=1 or W=2); asynchronously propagate to remaining nodes.
  • High availability: No single point of failure (no primary); any node can accept writes/reads.
  • Acceptable eventual consistency: Can tolerate reads returning slightly stale data if nodes are recovering.

Synchronous vs Asynchronous Replication

sequenceDiagram
    participant C as Client
    participant P as Primary
    participant R1 as Replica 1
    participant R2 as Replica 2
    Note over C,R2: Synchronous: wait for every replica
    C->>P: write
    P->>R1: log entry
    P->>R2: log entry
    R1-->>P: ack
    R2-->>P: ack
    P-->>C: ack
    Note over C,R2: Semi-synchronous: wait for one replica
    C->>P: write
    P->>R1: log entry
    R1-->>P: ack
    P-->>C: ack
    P--)R2: log entry (background)
    Note over C,R2: Asynchronous: wait for none
    C->>P: write
    P-->>C: ack
    P--)R1: log entry (background)
    P--)R2: log entry (background)

Synchronous Replication

Definition: Primary waits for replicas to acknowledge before responding to client.

Durability: ✅ Extremely high — client ACK guarantees replicas have written to disk.

Latency: 🔶 Higher — every commit waits for the replica round trip (~1 ms in a datacenter, 50–150 ms across continents).

Availability: ❌ If even one replica is slow/down, all writes are blocked.

Real-world impact: A synchronous write from a primary in San Francisco to a replica in New York pays a ~70 ms round trip on every commit. Throughput does not collapse — databases commit many transactions concurrently and group commits — but each client’s commit latency rises by that round trip, and a single connection that commits serially is capped at ~14 commits/sec. The bigger risk is availability: if the only synchronous replica stalls, writes stall with it.

Asynchronous Replication

Definition: Primary acknowledges immediately after writing to its own disk; replication happens in background.

Durability: ❌ Weak — if primary crashes before replicating, data is lost.

Latency: ✅ Very low — no waiting for replicas (~1–5ms to write locally).

Availability: ✅ Replicas can be down/slow without affecting writes.

Real-world impact: Same scenario (SF primary, NY replica). Each write takes ~1ms (local disk write only). You can sustain 10,000+ writes/sec. Asynchronously, replicas catch up in the background (typically within 100ms).

Semi-Synchronous Replication

Definition: Primary waits for one replica to acknowledge, then immediately responds to client. Other replicas replicate asynchronously in background.

Trade-off: Compromise between sync and async.

  • Durability is strong: at least one replica has the write.
  • Latency is reasonable: only waiting for one replica (~10–20ms).
  • Availability is reasonable: one replica can be down without blocking writes.

Real production defaults: PostgreSQL and MySQL are asynchronous by default; synchronous (PostgreSQL synchronous_standby_names) and semi-synchronous (MySQL plugin) are opt-in. MongoDB’s default write concern has been majority since 5.0, so a write is acknowledged once most replica-set members have it.

How Real Systems Use Replication

PostgreSQL Streaming Replication

PostgreSQL uses synchronous or asynchronous WAL streaming based on synchronous_commit setting. A typical production setup: Primary in us-east-1, synchronous replica in us-east-1 (same AZ, ~1ms latency), and asynchronous replica in us-west-2 (cold standby for disaster recovery, no ACK impact).

Configuration:

1

synchronous_commit = 'remote_apply'  -- Wait for replica to apply WAL

Behavior: Client write → Primary writes WAL → Primary sends WAL to sync replica → Sync replica writes and applies WAL → Sync replica ACK → Primary ACK to client. Total latency: ~5–15ms (1ms local write + 1ms network + 1ms replica write). Async replica eventually catches up (100ms lag typical). If primary crashes, promotion takes ~30 seconds (manual or automated tools like Patroni). If the sync replica is overloaded, all writes slow down. Many teams therefore use synchronous_commit = on (replica has flushed WAL) rather than remote_apply, and list two candidate standbys (ANY 1 (s1, s2)) so one slow node does not block writes.

MySQL Binlog Replication

MySQL replicates binlog events. A typical production setup: Primary in us-east, replicas in us-east (for reads), and us-west (for geo-locality or warm standby). Asynchronous by default (client ACK after primary binlog write, replica lags 10–100ms).

The relay log on replica buffers the binlog from primary; a separate SQL thread applies relay log changes to the replica’s database. If this SQL thread falls behind the IO thread, replica lag increases (can reach minutes if applying complex transactions).

Production numbers: A primary handling 50,000 QPS generates ~2GB binlog/hour. Replicas with sufficient disk I/O stay within 100ms lag. If a replica is slow (I/O bound), lag can hit seconds; monitoring tools alert if lag > 10 seconds.

Cassandra Leaderless Quorum Replication

Cassandra is fully leaderless and eventually consistent. Every node is equivalent. When you write a key:

1
2
3
4
5
6

coordinator = any node; replicas = 3 nodes chosen by partition-key hash
send write to all replicas in parallel
ack the client once W replicas have acked
for each replica that missed the write:
    coordinator stores a hint (hinted handoff)
    retry until the replica is back, then replay

Consistency tuning (per query, with RF=3):

  • ONE / ONE — fastest, may read stale data; fine for caches and events
  • QUORUM / QUORUM (or LOCAL_QUORUM across datacenters) — overlapping read and write sets; the usual default
  • ALL — any one replica down blocks the operation; rarely worth it

Conflict resolution: Cassandra resolves per-cell by timestamp: last write wins, and the other write is silently discarded. Clock skew therefore matters. Counters are a special type with their own merge logic. For compare-and-set semantics, Cassandra offers lightweight transactions (Paxos) at a significant latency cost.

DynamoDB Global Replication

DynamoDB replicates across multiple AWS regions for disaster recovery and geo-locality. You create a global table that replicates across regions:

Global tables now come in two modes, chosen at creation and not changeable afterwards:

Mode How writes replicate Reads Conflicts
MREC (multi-Region eventual, default) Async to other regions, typically < 1 s Strongly consistent only within the region Last writer wins (service timestamps)
MRSC (multi-Region strong, GA June 2025) Synchronously to at least one other region before success Strongly consistent in any region Prevented — no concurrent conflicting writes

Within a single region, reads are eventually consistent by default; a strongly consistent read costs twice the read units. MRSC trades higher write latency for the ability to fail over between regions with no data loss.

MongoDB Replica Sets

MongoDB uses single-leader replication with automatic failover. A replica set is 3+ nodes: 1 primary + 2+ secondaries. When the primary crashes, secondaries elect a new one using a Raft-based protocol.

1
2
3
4
5
6
7
8
9

Client writes to primary
Primary writes to oplog (operation log)
Secondaries read from primary oplog, apply operations locally

Read preference options:
  - primary: only read from primary (strong consistency)
  - primary_preferred: read from primary, fallback to secondary (fast)
  - secondary: read from secondary only (fast reads, stale data)
  - secondary_preferred: read from secondary, fallback to primary (balanced)

Replication lag: A busy primary might generate oplog entries faster than secondaries can apply. Lag is measured in bytes (how far behind in the oplog the secondary is). MongoDB can handle 10–100ms lag easily; if lag exceeds 1 second, alerts fire (secondary I/O is overloaded).

Failover: Secondaries declare the primary dead after electionTimeoutMillis (10 s by default) and elect the most up-to-date eligible member; writes typically resume within ~12–30 seconds. Data loss depends on write concern: writes acknowledged with w: "majority" (the default since 5.0) survive; writes acknowledged with w: 1 that had not reached a majority are rolled back when the old primary rejoins.

Consensus-Based Replication (Raft / Paxos)

Modern distributed databases mostly replicate with a consensus protocol rather than classic primary/replica shipping. Data is split into ranges or shards; each range is a small replication group (usually 3 or 5 replicas) with an elected leader. A write commits when a majority has it in its log.

sequenceDiagram
    participant C as Client
    participant L as Leader (range 7)
    participant F1 as Follower A
    participant F2 as Follower B
    C->>L: write
    L->>F1: append entry (term 5, index 42)
    L->>F2: append entry (term 5, index 42)
    F1-->>L: ack
    Note over L: majority (2 of 3) has it: commit
    L-->>C: ok
    F2-->>L: ack (late, still fine)
  • No conflicts and no data loss on failover: a new leader must hold every committed entry, by construction.
  • Automatic failover in seconds: followers elect a new leader after a missed heartbeat timeout.
  • Cost: every write waits for a majority round trip, and a minority partition cannot accept writes. Geo-spreading replicas across regions puts cross-region latency on every commit.
  • Where you meet it: etcd (Kubernetes’ brain), CockroachDB, TiDB, YugabyteDB (Raft); Google Spanner (Paxos + TrueTime); Kafka KRaft for metadata; MongoDB elections.

Choosing a Topology

flowchart TD
    A{"Must every read see<br/>the latest write?"} -->|Yes| B{"Writes from multiple<br/>regions at low latency?"}
    A -->|No, stale reads OK| C{"Write volume beyond<br/>one primary?"}
    B -->|No| SL["Single-leader with<br/>sync / semi-sync replica"]
    B -->|Yes| CS["Consensus-based<br/>distributed SQL"]
    C -->|No| SLA["Single-leader with<br/>async read replicas"]
    C -->|Yes| D{"Offline clients or<br/>region-local writes?"}
    D -->|Yes| ML["Multi-leader<br/>with explicit conflict handling"]
    D -->|No| LL["Leaderless quorum<br/>Cassandra / ScyllaDB"]

When to Use / Avoid

Use Single-Leader When

  • ✅ Consistency is critical (banking, e-commerce)
  • ✅ Workload is mostly reads (read scaling via replicas)
  • ✅ Writes are bursty or low-volume
  • ✅ All data fits on a single machine
  • ✅ Replication lag of 100ms–1sec is acceptable

Use Multi-Leader When

  • ✅ Geo-distributed (multiple datacenters with local write latency)
  • ✅ Offline-capable (mobile apps, collaborative tools)
  • ✅ Conflicts are rare (users in different regions)
  • ✅ Application can handle eventual consistency

Use Leaderless When

  • ✅ Write throughput is extremely high (no bottleneck at primary)
  • ✅ Availability is paramount (no failover needed)
  • ✅ Acceptable eventual consistency (cache, events, analytics)
  • ✅ Partitioned data (can shard by key)

Replication Lag — Real-World Scenarios

Scenario 1: Read-Your-Writes Consistency Problem

sequenceDiagram
    participant A as Alice's browser
    participant P as Primary (NY)
    participant R as Replica (SF)
    A->>P: status = 'online'
    P-->>A: ack
    P--)R: replicate (still in flight)
    A->>R: read profile (5 ms later)
    R-->>A: status = 'offline' (stale)

Solution: Sticky routing. After writing, route alice’s reads to the primary (NY) for 100ms, then switch to replica. Adds latency but guarantees consistency.

Scenario 2: Multi-Replica Read Inconsistency

sequenceDiagram
    participant C as Client
    participant A as Replica A (caught up)
    participant B as Replica B (lagging)
    C->>A: read
    A-->>C: v1 (fresh)
    C->>B: read again
    B-->>C: v0 (stale)
    Note over C: Data appears to go back in time: v1, then v0

Solution: Monotonic reads. Client remembers max version it has seen. On next read, doesn’t read from replicas with older version; waits for them to catch up or reads from primary.

Scenario 3: Write-Write Conflict (Multi-Leader)

This is the conflict shown in the multi-leader section: two leaders accept different values for the same field, and with last-writer-wins one write is silently lost.

Solution: Use CRDTs or application logic to resolve conflicts explicitly (don’t silently lose data). Or partition data so Alice’s writes go to leader-SF and Bob’s go to leader-London.

References

Software Architecture, Databases
distributed-systems
This post is licensed under CC BY 4.0 by the author.