fix(recovery): the fold streams and narrates; the checkpoint chain arms at the flip
A production brain's first process boot after a live authority flip looked
hung and was restarted three times mid-recovery — three defects with one
scene. (1) THE FOLD MATERIALIZED THE LOG: peekFactsAbove(0) decoded every
fact into one array (GBs of after-images on a ~7k-fact log, a GC storm, a
starved write lane). The fold now STREAMS one segment-batch at a time —
memory is one segment at any log size — with structural ordering asserted
loudly. (2) THE FOLD WAS SILENT UNTIL DONE: minutes of boot work with zero
narration is what invited the restarts. It now announces itself BEFORE the
work ('do not restart, the fold is finite') and prints progress every
thousand facts. (3) THE CHAIN COULD ONLY ARM AT A CRASH: a live mid-session
flip left the fold checkpoint unfounded, so the brain's first unclean boot
paid a whole-log fold. Adoption now founds the checkpoint AT THE FLIP — one
paged full canonical barrier (bounded memory), then the stamp — so bounded
recovery holds from minute zero for every store that flips, at any size.
Pinned: a non-fresh flip stamps immediately; the first post-flip unclean
boot folds bounded (an unflushed at-ack fact above the checkpoint is
restored; a barrier-covered row below it is outside the fold). Kill matrix
and both adoption suites green alongside.
This commit is contained in:
parent
8fb6cb7e54
commit
ed7d1db97e
4 changed files with 215 additions and 33 deletions
|
|
@ -8392,6 +8392,57 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// Fold-checkpoint chain, phase 2: the flip is recorded — open the stamp
|
||||
// gate so the next flush/close barrier writes the first checkpoint.
|
||||
this.generationStore.completeFoldCheckpointBootstrap()
|
||||
// ARM-AT-FLIP for the NON-FRESH brain (the chain refused the fresh-brain
|
||||
// arm because committed > 0): run one paged FULL canonical barrier now —
|
||||
// every live row's canonical bytes fsynced, bounded memory — then stamp
|
||||
// the first checkpoint. Without this, the chain could only arm at the
|
||||
// brain's first crash, and that crash paid a WHOLE-LOG fold: a production
|
||||
// brain hit exactly that on its first post-flip boot (a full-log
|
||||
// materializing fold, restarted three times mid-flight). Adoption already
|
||||
// pays O(N) oracle work; one more O(N) barrier founds bounded recovery
|
||||
// from minute zero.
|
||||
if (!this.generationStore.foldCheckpointChainArmed()) {
|
||||
const PAGE = 500
|
||||
let synced = 0
|
||||
prodLog.info(
|
||||
`[Brainy] adoptLogAuthority: founding the fold checkpoint — syncing every ` +
|
||||
`row's canonical bytes (paged; progress every 2000 rows)`
|
||||
)
|
||||
let offset = 0
|
||||
let cursor: string | undefined
|
||||
for (;;) {
|
||||
const page = await this.storage.getNouns({
|
||||
pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset }
|
||||
})
|
||||
const ids = page.items.map((i) => (i as { id: string }).id)
|
||||
if (ids.length > 0) {
|
||||
await this.storage.syncEntityCanonical?.(ids, [])
|
||||
synced += ids.length
|
||||
if (synced % 2000 < PAGE && synced >= 2000) {
|
||||
prodLog.info(`[Brainy] adoptLogAuthority: checkpoint founding — ${synced} rows synced`)
|
||||
}
|
||||
}
|
||||
if (page.hasMore && page.nextCursor) { cursor = page.nextCursor; offset += ids.length; continue }
|
||||
if (page.hasMore && !page.nextCursor) { offset += PAGE; continue }
|
||||
break
|
||||
}
|
||||
let vOffset = 0
|
||||
let vCursor: string | undefined
|
||||
for (;;) {
|
||||
const page = await this.storage.getVerbs({
|
||||
pagination: vCursor ? { limit: PAGE, cursor: vCursor } : { limit: PAGE, offset: vOffset }
|
||||
})
|
||||
const ids = page.items.map((i) => (i as { id: string }).id)
|
||||
if (ids.length > 0) {
|
||||
await this.storage.syncEntityCanonical?.([], ids)
|
||||
synced += ids.length
|
||||
}
|
||||
if (page.hasMore && page.nextCursor) { vCursor = page.nextCursor; vOffset += ids.length; continue }
|
||||
if (page.hasMore && !page.nextCursor) { vOffset += PAGE; continue }
|
||||
break
|
||||
}
|
||||
await this.generationStore.stampFoldCheckpointAfterFullBarrier()
|
||||
}
|
||||
return report
|
||||
}
|
||||
|
||||
|
|
|
|||
Reference in a new issue