Replication, the ISR and the High Watermark

After this lesson your program is the leader of one partition: it watches its followers fetch, decides who is in the in-sync set, moves the high watermark, and says when the partition may accept writes.

Followers are consumers

Every partition has a replica list, for example 5, 6, 7. The first is the preferred leader. Clients only talk to the leader; the others are followers, and a follower does nothing but send the leader ordinary Fetch requests ("give me records from offset N") and append what it receives. That is the whole replication protocol: there is no push, no separate channel.

Two offsets describe progress:

• LEO, the log end offset: the offset the next record will get. The leader has its own LEO. For each follower it also remembers the follower's LEO, which is the fetch offset in the follower's latest request (a follower asking for offset 7 has records 0 to 6).
• HW, the high watermark: the offset below which every replica in the ISR has the data. Consumers only see records below the HW, and it never moves backwards.

The in-sync set

The ISR (in-sync replicas) is the leader plus every follower that is keeping up. "Keeping up" is not "has the same data now", because a busy leader is always a little ahead. The rule: a follower is in sync if it has caught up to the leader's log within the last replica.lag.time.max.ms. The leader tracks, per follower:

• lastCaughtUpTime
• lastFetchTime and the leader's LEO at that fetch (lastFetchLeaderLEO)

When a follower fetches at time t with offset o, and the leader's LEO is L:

if o >= L:               lastCaughtUpTime = max(lastCaughtUpTime, t)
else if o >= lastFetchLeaderLEO:
                         lastCaughtUpTime = max(lastCaughtUpTime, lastFetchTime)
follower LEO = o;  lastFetchLeaderLEO = L;  lastFetchTime = t

The second branch is what keeps a follower in sync under constant load: if it now asks for everything the leader had at its previous fetch, it was caught up at that previous fetch, even though new records have arrived since.

Shrinking. A periodic check removes a follower from the ISR when now - lastCaughtUpTime > replica.lag.time.max.ms (strictly greater). The leader is never removed.

Expanding. After a fetch, a follower that is not in the ISR joins it as soon as its LEO is at or above the current HW.

The high watermark is the smallest LEO among the ISR members (the leader's LEO included), and it only ever increases. Removing a slow follower can therefore make it jump up: the slow one was holding it back.

Durability knobs

acks=all means the leader answers a produce request only once every replica in the ISR has the records, that is, once the HW has passed them. min.insync.replicas is the smallest ISR size for which the leader accepts such writes at all. With min.insync.replicas=2 and a shrunken ISR of one, produce requests fail with NOT_ENOUGH_REPLICAS rather than silently running with a single copy. That is the trade: availability for durability.

Worked trace

Different data from your exercise. LAG_MAX 500, replicas 5 6 7 (5 is the leader).

t=0    APPEND 4                leader LEO = 4
t=100  FETCH 6 offset 4        4 >= 4: caught up, lastCaughtUp(6)=100.  LEO(6)=4
t=120  FETCH 7 offset 2        2 < 4 but 2 >= 0 (its previous fetch saw LEO 0): lastCaughtUp(7)=0.  LEO(7)=2
t=130  STATUS                  everyone within 500ms. HW = min(4,4,2) = 2
t=400  FETCH 6 offset 4        caught up again, lastCaughtUp(6)=400
t=550  STATUS                  550 - 0 = 550 > 500: broker 7 leaves the ISR.
                               ISR = 5,6. HW = min(4,4) = 4      (it jumped from 2)
t=600  FETCH 7 offset 4        7 is not in the ISR and 4 >= HW(4): it rejoins
t=601  STATUS                  ISR = 5,6,7

Common mistakes

• Requiring equality with the leader's current LEO to stay in sync. Under load, that kicks every follower out constantly.
• Using >= when shrinking. A follower exactly replica.lag.time.max.ms behind is still in sync.
• Computing the HW over all replicas instead of only the ISR, or letting it decrease.
• Re-adding a follower too early. It must be at or above the HW, or the HW would be vouching for data it does not have.

Your exercise

The starter parses the commands. Implement the leader's bookkeeping. The most likely mistake is the second branch of the caught-up rule.