Pattern · seen in 1 breakdown across 1 company

Checkpoint-Bounded Scans

Definition

When background machinery repeatedly scans a table by a monotonic column — 'everything with timestamp ≤ now' — maintain a persistent checkpoint of the last position processed and bound every scan from below as well as above. The unbounded form forces the storage engine to consider every pending row on every pass (in MVCC systems like MySQL, walking a history list that grows with backlog); the bounded form touches only the increment since the last pass, keeping scan cost proportional to progress rather than to backlog depth.

The pattern is what makes a buffer's performance independent of how full it is — the property any queue, scheduler, or expiry sweeper needs if its whole purpose is absorbing backlogs. The same shape appears as watermarks in stream processors and incremental cursors in reconciliation jobs.

When it applies

01Queues, schedulers, and TTL/expiry sweepers whose background passes scan by time or sequence over a table that is deep precisely when the system is under stress
02MVCC storage (MySQL/InnoDB, Postgres) where broad range predicates pay for row-version history proportional to pending work
03Any repeated incremental job that can record how far it got and resume from there

Tradeoffs

The checkpoint is state that must be persisted and advanced correctly — a stalled checkpoint silently unbounds the scan again
Items that become ready behind the checkpoint (clock skew, late updates to the scanned column) are skipped unless the design accounts for them
Per-shard or per-partition checkpoints multiply the bookkeeping the pattern was meant to simplify

The same move, 1 ways

Every row is a production system that bet on this pattern — the note says how, in that system's own terms.

Meta
Engineering at Meta
2021
This is the post's most transferable idea. Background operations that scan by timestamp carry a persistent lower bound, the last checkpoint processed, so the where clause is bounded on both sides. MySQL's history-list walk then stays short no matter how deep the backlog. It is what makes the queue's performance independent of how full it is. Read the breakdown →

Problems this pattern answers

The walls where its breakdowns live — each opens the cross-company comparison.