Replication & Handover

Read scaling with coherent followers, plus zero-downtime leader handover for datacenter moves. This page covers the model; for a step-by-step setup walkthrough see the replication guide.

Concepts

DocumentForge supports two replication modes. They solve different problems and can be used together.

Logical replication

On a write, the leader broadcasts an operation record to every connected follower. The follower runs the operation through its own engine, so indexes, location maps, and cached state all stay consistent with the data file.

           Leader                           Follower
       ┌──────────┐                      ┌──────────┐
  write│  Insert  │ ──(seq 42, Insert)──▶│  Insert  │
       │  ↓       │ ──(seq 43, Insert)──▶│  ↓       │
       │  engine  │ ──(seq 44, CreIdx)──▶│  engine  │
       │  ↓       │ ───(heartbeat)──────▶│          │
       │  disk    │                      │  disk    │
       └──────────┘                      └──────────┘

Each op carries a monotonic seq. Followers persist the last-applied seq to a sidecar file. On reconnect, they tell the leader their position and get replayed everything after.

Physical replication

Page-level byte streaming. The follower ends up with a byte-identical data file. Use this when you want a hot backup or plan to manually promote the follower as a new leader without any engine warmup.

Physical replication doesn’t solve the “can the follower serve reads” problem — its in-memory structures (indexes, location map) were built from the original data and aren’t updated by page writes. Close and reopen the follower to rebuild them.

Catchup on reconnect

When a follower disconnects and reconnects:

1. Follower loads its last-applied seq from disk (the .followerseq sidecar file).
2. Follower sends a handshake to the leader: “I’m at seq N, please bring me up to date”.
3. Leader replays all ops with seq greater than N from its in-memory ring buffer.
4. Leader adds the follower to its live broadcast list.

Planned handover

When you need to move the leader role — typically for maintenance or a datacenter move — use the planned-handover flow. It guarantees zero data loss because the new leader catches up fully before the old one releases the role.

// 1. Old leader enters read-only and waits for new leader to catch up
ulong finalSeq = oldLeader.BeginPlannedHandover(
    followerLastSeqProbe: () => newLeader.FollowerLastSeq,
    timeout: TimeSpan.FromMinutes(5));
 
// 2. Old leader is now read-only. New leader is caught up.
//    Promote new leader.
newLeader.PromoteToLeader(port: 5500);
 
// 3. Direct clients to new leader's address.
//    Old leader can be shut down or repurposed as a follower.

Between steps 1 and 2, writes to oldLeader throw DocumentForgeException("Database is in read-only mode..."). After step 2, newLeader accepts writes.

Auto-failover

Planned handover is the safe path — zero data loss, no split-brain risk. For crashes (leader process died, host went dark, network partition), you can opt in to automatic promotion. A follower watches the leader’s heartbeat stream and, if nothing arrives within a silence threshold, promotes itself to leader.

using var follower = DocumentForgeDb.OpenOrCreate("replica.dfdb");
follower.StartLogicalReplicationFollower("leader-host", 5500);
 
// Enable auto-promotion if the leader goes silent for 10 seconds.
follower.EnableAutoFailover(
    newLeaderPort: 5500,
    silenceTimeout: TimeSpan.FromSeconds(10),
    onPromoted: port => Console.WriteLine($"Promoted to leader on :{port}"));

The watcher has a 3-second grace period after startup (so normal connect-time silence doesn’t trigger promotion). Once promoted, the follower stops consuming from the old leader, opens its own replication server on newLeaderPort, and begins accepting writes.

Useful properties while running:

follower.WasAutoFailoverPromoted   // true once it has taken over
follower.DisableAutoFailover()     // stop watching (safe to call any time)

Replicated reference collections

In a sharded cluster, small reference tables can be marked as replicated so every shard holds a full copy and JOINs stay local — no cross-shard lookup. This complements logical replication’s leader/follower model; see sharding for the per-collection strategy.

Caveats and current limits

• Auto-failover has a small data-loss window. Ops the leader committed but hadn’t yet broadcast when it died are lost. Planned handover avoids this — use it for maintenance.
• No built-in split-brain protection. Auto-failover has no witness/quorum — enable it on one follower only, or front it with your own fencing mechanism. Planned handover prevents split-brain by design (the old leader is read-only before the new one is promoted).
• No multi-master. Writes go to the leader only. You cannot write to two nodes simultaneously.
• Ring-buffered catchup. The leader keeps the last 10,000 ops for reconnecting followers. A follower that’s been offline longer than that needs a fresh snapshot (manual file copy) to catch up.
• Client-side failover. DocumentForge doesn’t include a client-side load balancer or discovery service. Your application needs to know which host is the current leader — typically via a config flip after promotion, DNS, or a small service-discovery layer in front.