fix(recovery): a torn generation-log tail is a terminal verdict, never a wait
Some checks failed
CI / Node 22 (push) Successful in 12m22s
CI / Node 24 (push) Successful in 12m21s
CI / Integration + conformance (Node 22) (push) Failing after 16m58s
CI / Bun (latest) (push) Successful in 12m23s

Two halves of one defect, found by a seeded-SIGKILL crash lane.

THE FALSE POSITIVE. stampEntityTree() recorded generationStore.generation()
— the ALLOCATED counter, a number a write in flight has claimed and may
never commit — while the JSDoc beside it already said the source is the
committed generation. Every crash inside a write window therefore produced
a spurious verdict at the next open: either 'sourceGeneration N is ahead of
the log head N-1' (the allocated generation died with the process) or
'rollup invariant nounCount: stamped X, observed Y' (the recovery fold
folded facts the stamp's counts predate). Both told the operator to run
repairIndex() — a whole-store recount — for a store that was coherent.
Measured before this commit: 4 of 11 SIGKILL cycles on a healthy store
raised one of the two. The stamp and the open now both read
committedGeneration(), which is what every other open-time watermark in the
class already reasons about.

THE TERMINAL VERDICT. A stamp still ahead of committed truth after the
recovery fold witnesses a generation that is not in the log — the stamp's
fsync outlived the tail's, and there is nothing to arrive. That is its own
verdict state now ('torn'), never folded in with 'incoherent': the two have
opposite cures. A writer open demotes it — the unusable stamped surface is
re-derived at the committed generation from the live counters, O(1),
straight-line, no loop and no await on external progress, narrated with
both count sets, the stamp's path and its committedAt. A read-only open
cannot re-stamp, so it says so and names the cure instead of guessing, and
still serves. Neither branch waits, and neither locks an owner out of a
canonical tree the stamp only describes.

Pins: the verifier returns the torn verdict with both generations; a
fabricated head-behind-source store narrates precisely, demotes inside a
bounded open, serves its rows, and is quiet at the next open (the demotion
converges); a read-only open narrates the same verdict and leaves the bytes
untouched.
This commit is contained in:
David Snelling 2026-08-31 09:07:18 -07:00
parent b8475cc86a
commit 298cb6daca
3 changed files with 241 additions and 16 deletions

View file

@ -12,9 +12,11 @@
* the verified surface is a small set of rollup invariants (entity/
* relationship counts) plus `sourceGeneration`.
*
* `sourceGeneration` is the generation of the source-of-truth log this
* projection reflects open-time coherence becomes a COMPARISON (stamp vs
* log head), not a walk:
* `sourceGeneration` is the COMMITTED generation of the source-of-truth log
* this projection reflects never the allocated counter, which names a
* generation that may never commit (see {@link StampVerdict.torn}) so
* open-time coherence becomes a COMPARISON (stamp vs committed head), not a
* walk:
*
* - equal + invariants hold coherent, serve.
* - behind the projection missed the tail (crash between commit and stamp);
@ -24,6 +26,9 @@
* - invariants FAIL at equal generation genuine incoherence: loud, and the
* repair ritual (`repairIndex()`, whose recount rebuilds the rollups from a
* canonical walk) heals it.
* - AHEAD a torn generation-log tail: the stamp's fsync outlived the log
* tail's. TERMINAL, never a wait the generation the stamp names does not
* exist to arrive.
*
* Stamps are JSON on purpose every incident gets debugged by reading a
* stamp in a terminal.
@ -70,6 +75,12 @@ export type StampVerdict =
| { state: 'coherent' }
| { state: 'absent' } // legacy store — first stamp writes at the next flush
| { state: 'behind'; stampSource: number; head: number }
/**
* TORN GENERATION-LOG TAIL: the stamp witnesses a source generation the
* store's committed watermark can no longer show. TERMINAL there is no
* generation to wait for, so the open demotes (or refuses) and never spins.
*/
| { state: 'torn'; stampSource: number; head: number }
| { state: 'incoherent'; failures: string[] }
| { state: 'unverifiable'; reason: string } // a FAULT reading the stamp — never conflated with absence
@ -118,12 +129,15 @@ export function verifyFamilyStamp(
): StampVerdict {
if (stamp === null) return { state: 'absent' }
if (stamp.sourceGeneration > head) {
// A stamp AHEAD of the log claims state that never committed — the
// projection was stamped against truth that a crash rolled back.
return {
state: 'incoherent',
failures: [`sourceGeneration ${stamp.sourceGeneration} is ahead of the log head ${head}`]
}
// A stamp AHEAD of committed truth witnesses a generation the store can no
// longer show: the stamp's fsync survived a crash that the log tail did
// not. This is the TORN GENERATION-LOG TAIL — its own class, never folded
// in with `incoherent` (a count that drifted at a generation both sides
// agree on), because the two have opposite cures: incoherence is recounted,
// a tear is DEMOTED. It is also terminal by construction — there is no
// generation the open can wait for, because the one the stamp names is
// gone.
return { state: 'torn', stampSource: stamp.sourceGeneration, head }
}
if (stamp.sourceGeneration < head) {
return { state: 'behind', stampSource: stamp.sourceGeneration, head }