fix(recovery): walks are healers — the typed/tolerant boundary redrawn where block-layer fault injection proved it belonged
The quiet-loss cure regressed recovery: the new typed torn-record error was correct at identity-read time but threw inside init-time recovery walks, killing opens that previously survived. The boundary, redrawn: - IDENTITY READS (get-by-id of a specific record, CAS blob point-get): typed TornRecordError, unchanged — a caller who asked for THAT record can act on the answer. - SET-SHAPED READS AND WALKS (enumeration, pagination, batch hydration — the paths recovery rebuilds and finds page over): HEAL PAST the torn victim. The adapter's loud floor (error log + counted gauge) fires at the encounter; the walk serves the remaining rows. One crash casualty can no longer kill every query on its shard — or the open itself. - WRITES OVER TORN RECORDS ARE THE CURE: the save path's read-merge, the commit path's before-image capture, and the operations' rollback captures all treat a torn prior as the create sentinel, narrated — the incoming bytes replace the unreadable ones, and history for the id honestly restarts at that generation. Corruption can never block its own heal. - THE NaN SOURCE: torn mapper state (nextId/entries carrying garbage) discards with narration and re-derives via the existing rebuild path; the mint gains a source guard healing a non-integer counter from the live map. The reopen and first-write RangeError shapes are dead at the source, both authority branches. Pinned with the exact fault-injection scenarios: a torn entity record (including the VFS root) no longer kills the open — walks heal past it, the keeper rows serve, and the identity read of the victim itself is typed-or-healed; a torn mapper reopens and mints sanely on the first post-recovery write. Gates: tsc 0 · unit 2065/2065 · integration 828 · conformance 31/31.
This commit is contained in:
parent
214c98b4d5
commit
0e3facf4a8
6 changed files with 350 additions and 39 deletions
|
|
@ -918,6 +918,37 @@ export class GenerationStore {
|
|||
else this.pins.set(gen, count - 1)
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Torn-tolerant raw read for BEFORE-IMAGE contexts: a write landing on a
|
||||
* TORN record (power-loss survivor) is a HEAL — the new after-image
|
||||
* replaces the unreadable bytes. The before-image is unknowable, so it
|
||||
* reads as the CREATE SENTINEL ({metadata:null, vector:null}) with
|
||||
* narration: history for this id restarts at this generation (an asOf
|
||||
* below it resolves absent for the id — the honest statement of what the
|
||||
* crash destroyed). The adapter's loud floor (error + gauge) fired at
|
||||
* throw time; real storage faults still propagate.
|
||||
*/
|
||||
private async readRawForBeforeImage(
|
||||
kind: 'noun' | 'verb',
|
||||
id: string
|
||||
): Promise<{ metadata: unknown | null; vector: unknown | null }> {
|
||||
try {
|
||||
return kind === 'noun'
|
||||
? await this.storage.readNounRaw(id)
|
||||
: await this.storage.readVerbRaw(id)
|
||||
} catch (err) {
|
||||
if ((err as { code?: string }).code === 'TORN_RECORD') {
|
||||
prodLog.warn(
|
||||
`[GenerationStore] before-image of ${kind} ${id} is TORN — the incoming ` +
|
||||
`write HEALS the record; its history restarts at this generation`
|
||||
)
|
||||
return { metadata: null, vector: null }
|
||||
}
|
||||
throw err
|
||||
}
|
||||
}
|
||||
|
||||
/** @returns Total number of live pins across all generations. */
|
||||
activePinCount(): number {
|
||||
let total = 0
|
||||
|
|
@ -1083,11 +1114,11 @@ export class GenerationStore {
|
|||
// conflicting batch aborts with zero staging I/O. The maps hold the
|
||||
// byte-identical records the staged files are written from.
|
||||
for (const id of nouns) {
|
||||
const prev = await this.storage.readNounRaw(id)
|
||||
const prev = await this.readRawForBeforeImage('noun', id)
|
||||
nounBefore.set(id, { kind: 'noun', metadata: prev.metadata, vector: prev.vector })
|
||||
}
|
||||
for (const id of verbs) {
|
||||
const prev = await this.storage.readVerbRaw(id)
|
||||
const prev = await this.readRawForBeforeImage('verb', id)
|
||||
verbBefore.set(id, { kind: 'verb', metadata: prev.metadata, vector: prev.vector })
|
||||
}
|
||||
|
||||
|
|
@ -1415,12 +1446,12 @@ export class GenerationStore {
|
|||
// {metadata:null, vector:null} = the create sentinel.
|
||||
const nounBefore = new Map<string, GenerationRecord>()
|
||||
for (const id of nouns) {
|
||||
const prev = await this.storage.readNounRaw(id)
|
||||
const prev = await this.readRawForBeforeImage('noun', id)
|
||||
nounBefore.set(id, { kind: 'noun', metadata: prev.metadata, vector: prev.vector })
|
||||
}
|
||||
const verbBefore = new Map<string, GenerationRecord>()
|
||||
for (const id of verbs) {
|
||||
const prev = await this.storage.readVerbRaw(id)
|
||||
const prev = await this.readRawForBeforeImage('verb', id)
|
||||
verbBefore.set(id, { kind: 'verb', metadata: prev.metadata, vector: prev.vector })
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue