fix(recovery): the fold streams and narrates; the checkpoint chain arms at the flip
All checks were successful
CI / Node 22 (push) Successful in 12m13s
CI / Node 24 (push) Successful in 12m9s
CI / Integration + conformance (Node 22) (push) Successful in 18m16s
CI / Bun (latest) (push) Successful in 12m20s

A production brain's first process boot after a live authority flip looked
hung and was restarted three times mid-recovery — three defects with one
scene. (1) THE FOLD MATERIALIZED THE LOG: peekFactsAbove(0) decoded every
fact into one array (GBs of after-images on a ~7k-fact log, a GC storm, a
starved write lane). The fold now STREAMS one segment-batch at a time —
memory is one segment at any log size — with structural ordering asserted
loudly. (2) THE FOLD WAS SILENT UNTIL DONE: minutes of boot work with zero
narration is what invited the restarts. It now announces itself BEFORE the
work ('do not restart, the fold is finite') and prints progress every
thousand facts. (3) THE CHAIN COULD ONLY ARM AT A CRASH: a live mid-session
flip left the fold checkpoint unfounded, so the brain's first unclean boot
paid a whole-log fold. Adoption now founds the checkpoint AT THE FLIP — one
paged full canonical barrier (bounded memory), then the stamp — so bounded
recovery holds from minute zero for every store that flips, at any size.

Pinned: a non-fresh flip stamps immediately; the first post-flip unclean
boot folds bounded (an unflushed at-ack fact above the checkpoint is
restored; a barrier-covered row below it is outside the fold). Kill matrix
and both adoption suites green alongside.
This commit is contained in:
David Snelling 2026-08-18 12:53:50 -07:00
parent 8fb6cb7e54
commit ed7d1db97e
4 changed files with 215 additions and 33 deletions

View file

@ -770,6 +770,43 @@ export class FactLog {
* segments directly; the torn tail's invalid suffix is ignored exactly
* like open() would).
*/
/**
* STREAMING twin of {@link FactLog.peekFactsAbove} for the recovery fold:
* yields facts above the bound one SEGMENT at a time, ascending, without
* ever materializing the whole log (a production first-boot fold OOM-class
* allocation storm came from exactly that GBs of decoded after-images in
* one array while the process looked hung). Memory is one segment's worth.
* Works manifest-direct (safe before {@link FactLog.open}). Ordering is
* structural (segments rotate in order; appends are ordered within one) and
* ASSERTED a violation aborts loudly, never a silent misordered replay.
*/
async *streamFactsAbove(committedGeneration: number): AsyncGenerator<CommitFact[], void> {
const stored = (await this.storage.readRawObject(FACTS_MANIFEST_PATH)) as FactsManifest | null
if (!stored || typeof stored !== 'object' || !Array.isArray(stored.segments)) return
if (stored.formatVersion !== FACTS_FORMAT_VERSION) return
const files = [...stored.segments.map((s) => s.file)]
if (stored.tailSegment) files.push(stored.tailSegment)
let lastGen = committedGeneration
for (const file of files) {
const bytes = await this.storage.readRawBytes(`${FACTS_PREFIX}/${file}`)
if (bytes === null) continue
const { facts } = parseSegment(file, bytes)
const batch: CommitFact[] = []
for (const f of facts) {
if (f.generation <= committedGeneration) continue
if (f.generation <= lastGen) {
throw new Error(
`fact log: streamFactsAbove found non-ascending generations ` +
`(${f.generation} after ${lastGen} in ${file}) — refusing to replay out of order`
)
}
lastGen = f.generation
batch.push(f)
}
if (batch.length > 0) yield batch
}
}
async peekFactsAbove(committedGeneration: number): Promise<CommitFact[]> {
const stored = (await this.storage.readRawObject(FACTS_MANIFEST_PATH)) as FactsManifest | null
if (!stored || typeof stored !== 'object' || !Array.isArray(stored.segments)) return []