fix(recovery): the fold streams and narrates; the checkpoint chain arms at the flip
A production brain's first process boot after a live authority flip looked
hung and was restarted three times mid-recovery — three defects with one
scene. (1) THE FOLD MATERIALIZED THE LOG: peekFactsAbove(0) decoded every
fact into one array (GBs of after-images on a ~7k-fact log, a GC storm, a
starved write lane). The fold now STREAMS one segment-batch at a time —
memory is one segment at any log size — with structural ordering asserted
loudly. (2) THE FOLD WAS SILENT UNTIL DONE: minutes of boot work with zero
narration is what invited the restarts. It now announces itself BEFORE the
work ('do not restart, the fold is finite') and prints progress every
thousand facts. (3) THE CHAIN COULD ONLY ARM AT A CRASH: a live mid-session
flip left the fold checkpoint unfounded, so the brain's first unclean boot
paid a whole-log fold. Adoption now founds the checkpoint AT THE FLIP — one
paged full canonical barrier (bounded memory), then the stamp — so bounded
recovery holds from minute zero for every store that flips, at any size.
Pinned: a non-fresh flip stamps immediately; the first post-flip unclean
boot folds bounded (an unflushed at-ack fact above the checkpoint is
restored; a barrier-covered row below it is outside the fold). Kill matrix
and both adoption suites green alongside.
This commit is contained in:
parent
8fb6cb7e54
commit
ed7d1db97e
4 changed files with 215 additions and 33 deletions
|
|
@ -770,6 +770,43 @@ export class FactLog {
|
|||
* segments directly; the torn tail's invalid suffix is ignored exactly
|
||||
* like open() would).
|
||||
*/
|
||||
/**
|
||||
* STREAMING twin of {@link FactLog.peekFactsAbove} for the recovery fold:
|
||||
* yields facts above the bound one SEGMENT at a time, ascending, without
|
||||
* ever materializing the whole log (a production first-boot fold OOM-class
|
||||
* allocation storm came from exactly that — GBs of decoded after-images in
|
||||
* one array while the process looked hung). Memory is one segment's worth.
|
||||
* Works manifest-direct (safe before {@link FactLog.open}). Ordering is
|
||||
* structural (segments rotate in order; appends are ordered within one) and
|
||||
* ASSERTED — a violation aborts loudly, never a silent misordered replay.
|
||||
*/
|
||||
async *streamFactsAbove(committedGeneration: number): AsyncGenerator<CommitFact[], void> {
|
||||
const stored = (await this.storage.readRawObject(FACTS_MANIFEST_PATH)) as FactsManifest | null
|
||||
if (!stored || typeof stored !== 'object' || !Array.isArray(stored.segments)) return
|
||||
if (stored.formatVersion !== FACTS_FORMAT_VERSION) return
|
||||
const files = [...stored.segments.map((s) => s.file)]
|
||||
if (stored.tailSegment) files.push(stored.tailSegment)
|
||||
let lastGen = committedGeneration
|
||||
for (const file of files) {
|
||||
const bytes = await this.storage.readRawBytes(`${FACTS_PREFIX}/${file}`)
|
||||
if (bytes === null) continue
|
||||
const { facts } = parseSegment(file, bytes)
|
||||
const batch: CommitFact[] = []
|
||||
for (const f of facts) {
|
||||
if (f.generation <= committedGeneration) continue
|
||||
if (f.generation <= lastGen) {
|
||||
throw new Error(
|
||||
`fact log: streamFactsAbove found non-ascending generations ` +
|
||||
`(${f.generation} after ${lastGen} in ${file}) — refusing to replay out of order`
|
||||
)
|
||||
}
|
||||
lastGen = f.generation
|
||||
batch.push(f)
|
||||
}
|
||||
if (batch.length > 0) yield batch
|
||||
}
|
||||
}
|
||||
|
||||
async peekFactsAbove(committedGeneration: number): Promise<CommitFact[]> {
|
||||
const stored = (await this.storage.readRawObject(FACTS_MANIFEST_PATH)) as FactsManifest | null
|
||||
if (!stored || typeof stored !== 'object' || !Array.isArray(stored.segments)) return []
|
||||
|
|
|
|||
Reference in a new issue