fix(recovery): the fold streams and narrates; the checkpoint chain arms at the flip
All checks were successful
CI / Node 22 (push) Successful in 12m13s
CI / Node 24 (push) Successful in 12m9s
CI / Integration + conformance (Node 22) (push) Successful in 18m16s
CI / Bun (latest) (push) Successful in 12m20s

A production brain's first process boot after a live authority flip looked
hung and was restarted three times mid-recovery — three defects with one
scene. (1) THE FOLD MATERIALIZED THE LOG: peekFactsAbove(0) decoded every
fact into one array (GBs of after-images on a ~7k-fact log, a GC storm, a
starved write lane). The fold now STREAMS one segment-batch at a time —
memory is one segment at any log size — with structural ordering asserted
loudly. (2) THE FOLD WAS SILENT UNTIL DONE: minutes of boot work with zero
narration is what invited the restarts. It now announces itself BEFORE the
work ('do not restart, the fold is finite') and prints progress every
thousand facts. (3) THE CHAIN COULD ONLY ARM AT A CRASH: a live mid-session
flip left the fold checkpoint unfounded, so the brain's first unclean boot
paid a whole-log fold. Adoption now founds the checkpoint AT THE FLIP — one
paged full canonical barrier (bounded memory), then the stamp — so bounded
recovery holds from minute zero for every store that flips, at any size.

Pinned: a non-fresh flip stamps immediately; the first post-flip unclean
boot folds bounded (an unflushed at-ack fact above the checkpoint is
restored; a barrier-covered row below it is outside the fold). Kill matrix
and both adoption suites green alongside.
This commit is contained in:
David Snelling 2026-08-18 12:53:50 -07:00
parent 8fb6cb7e54
commit ed7d1db97e
4 changed files with 215 additions and 33 deletions

View file

@ -8392,6 +8392,57 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Fold-checkpoint chain, phase 2: the flip is recorded — open the stamp
// gate so the next flush/close barrier writes the first checkpoint.
this.generationStore.completeFoldCheckpointBootstrap()
// ARM-AT-FLIP for the NON-FRESH brain (the chain refused the fresh-brain
// arm because committed > 0): run one paged FULL canonical barrier now —
// every live row's canonical bytes fsynced, bounded memory — then stamp
// the first checkpoint. Without this, the chain could only arm at the
// brain's first crash, and that crash paid a WHOLE-LOG fold: a production
// brain hit exactly that on its first post-flip boot (a full-log
// materializing fold, restarted three times mid-flight). Adoption already
// pays O(N) oracle work; one more O(N) barrier founds bounded recovery
// from minute zero.
if (!this.generationStore.foldCheckpointChainArmed()) {
const PAGE = 500
let synced = 0
prodLog.info(
`[Brainy] adoptLogAuthority: founding the fold checkpoint — syncing every ` +
`row's canonical bytes (paged; progress every 2000 rows)`
)
let offset = 0
let cursor: string | undefined
for (;;) {
const page = await this.storage.getNouns({
pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset }
})
const ids = page.items.map((i) => (i as { id: string }).id)
if (ids.length > 0) {
await this.storage.syncEntityCanonical?.(ids, [])
synced += ids.length
if (synced % 2000 < PAGE && synced >= 2000) {
prodLog.info(`[Brainy] adoptLogAuthority: checkpoint founding — ${synced} rows synced`)
}
}
if (page.hasMore && page.nextCursor) { cursor = page.nextCursor; offset += ids.length; continue }
if (page.hasMore && !page.nextCursor) { offset += PAGE; continue }
break
}
let vOffset = 0
let vCursor: string | undefined
for (;;) {
const page = await this.storage.getVerbs({
pagination: vCursor ? { limit: PAGE, cursor: vCursor } : { limit: PAGE, offset: vOffset }
})
const ids = page.items.map((i) => (i as { id: string }).id)
if (ids.length > 0) {
await this.storage.syncEntityCanonical?.([], ids)
synced += ids.length
}
if (page.hasMore && page.nextCursor) { vCursor = page.nextCursor; vOffset += ids.length; continue }
if (page.hasMore && !page.nextCursor) { vOffset += PAGE; continue }
break
}
await this.generationStore.stampFoldCheckpointAfterFullBarrier()
}
return report
}