fix(recovery): walks are healers — the typed/tolerant boundary redrawn where block-layer fault injection proved it belonged
The quiet-loss cure regressed recovery: the new typed torn-record error was correct at identity-read time but threw inside init-time recovery walks, killing opens that previously survived. The boundary, redrawn: - IDENTITY READS (get-by-id of a specific record, CAS blob point-get): typed TornRecordError, unchanged — a caller who asked for THAT record can act on the answer. - SET-SHAPED READS AND WALKS (enumeration, pagination, batch hydration — the paths recovery rebuilds and finds page over): HEAL PAST the torn victim. The adapter's loud floor (error log + counted gauge) fires at the encounter; the walk serves the remaining rows. One crash casualty can no longer kill every query on its shard — or the open itself. - WRITES OVER TORN RECORDS ARE THE CURE: the save path's read-merge, the commit path's before-image capture, and the operations' rollback captures all treat a torn prior as the create sentinel, narrated — the incoming bytes replace the unreadable ones, and history for the id honestly restarts at that generation. Corruption can never block its own heal. - THE NaN SOURCE: torn mapper state (nextId/entries carrying garbage) discards with narration and re-derives via the existing rebuild path; the mint gains a source guard healing a non-integer counter from the live map. The reopen and first-write RangeError shapes are dead at the source, both authority branches. Pinned with the exact fault-injection scenarios: a torn entity record (including the VFS root) no longer kills the open — walks heal past it, the keeper rows serve, and the identity read of the victim itself is typed-or-healed; a torn mapper reopens and mints sanely on the first post-recovery write. Gates: tsc 0 · unit 2065/2065 · integration 828 · conformance 31/31.
This commit is contained in:
parent
214c98b4d5
commit
0e3facf4a8
6 changed files with 350 additions and 39 deletions
|
|
@ -677,7 +677,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
} catch (error) {
|
||||
// A TORN blob object (exists but undecodable) must not read as
|
||||
// "blob absent" — that would misdiagnose disk corruption as a
|
||||
// missing blob. Propagate the typed error to the blob layer.
|
||||
// missing blob. This is an IDENTITY read (a caller asked for THIS
|
||||
// key): propagate the typed error to the blob layer.
|
||||
if (isTornRecordError(error)) throw error
|
||||
return undefined
|
||||
}
|
||||
|
|
@ -2183,7 +2184,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
} catch (error) {
|
||||
// A TORN record must surface typed — a paginated read that
|
||||
// silently skips a corrupt row hides data loss from the caller.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip nouns that fail to load
|
||||
return null
|
||||
}
|
||||
|
|
@ -2214,7 +2219,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
}
|
||||
} catch (error) {
|
||||
// A TORN record propagates (typed) — only shard-listing absence is skippable.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip shards that have no data
|
||||
}
|
||||
}
|
||||
|
|
@ -2325,7 +2334,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
return { id, metadata: await this.getNounMetadata(id) }
|
||||
} catch (error) {
|
||||
// A TORN record must surface typed, never as a skipped id.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
return null
|
||||
}
|
||||
})
|
||||
|
|
@ -2348,7 +2361,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
}
|
||||
} catch (error) {
|
||||
// A TORN record propagates (typed) — only shard-listing absence is skippable.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip shards with no data
|
||||
}
|
||||
}
|
||||
|
|
@ -2561,13 +2578,21 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
} catch (error) {
|
||||
// A TORN record must surface typed — a paginated read that
|
||||
// silently skips a corrupt row hides data loss from the caller.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip verbs that fail to load
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// A TORN record propagates (typed) — only shard-listing absence is skippable.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip shards that have no data
|
||||
}
|
||||
}
|
||||
|
|
@ -3352,7 +3377,14 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
const path = getNounMetadataPath(id)
|
||||
|
||||
// Determine if this is a new entity by checking if metadata already exists
|
||||
const existingMetadata = await this.readCanonicalObject(path)
|
||||
// Torn-tolerant: a WRITE landing on a torn record HEALS it — the read
|
||||
// here only classifies new-vs-update and captures the prior subtype;
|
||||
// a torn prior reads as "no previous" (fresh write) with the adapter's
|
||||
// loud floor already fired. Never let corruption block its own cure.
|
||||
const existingMetadata = await this.readCanonicalObject(path).catch((err) => {
|
||||
if ((err as { code?: string }).code === 'TORN_RECORD') return null
|
||||
throw err
|
||||
})
|
||||
const isNew = !existingMetadata
|
||||
|
||||
// Save the metadata (write-cache coherent canonical write)
|
||||
|
|
@ -3722,12 +3754,17 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
if (result.value.data !== null) {
|
||||
results.set(result.value.path, result.value.data)
|
||||
}
|
||||
} else if (isTornRecordError(result.reason)) {
|
||||
// A torn record inside a SET-SHAPED read (batch hydration behind
|
||||
// find/sort pages and recovery walks): the adapter narrated +
|
||||
// counted at throw time; the batch HEALS PAST the victim and
|
||||
// serves the remaining rows — one crash casualty must not kill
|
||||
// every query that pages over its shard (and init-time recovery
|
||||
// walks ride this exact path). Identity point-reads still throw.
|
||||
continue
|
||||
} else {
|
||||
// A rejected read is a torn record or a real storage fault — NOT an
|
||||
// absent object. Batch hydration backs entity reads (getNounBatch /
|
||||
// getVerbsBatch / find hydration); swallowing the rejection would
|
||||
// silently drop a row the caller cannot distinguish from "never
|
||||
// existed". Propagate the typed/real error loudly instead.
|
||||
// A REAL storage fault (EIO-class) is not a torn victim —
|
||||
// propagate loudly, never absorb.
|
||||
throw result.reason
|
||||
}
|
||||
}
|
||||
|
|
@ -3864,7 +3901,14 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
const path = getVerbMetadataPath(id)
|
||||
|
||||
// Determine if this is a new verb by checking if metadata already exists
|
||||
const existingMetadata = await this.readCanonicalObject(path)
|
||||
// Torn-tolerant: a WRITE landing on a torn record HEALS it — the read
|
||||
// here only classifies new-vs-update and captures the prior subtype;
|
||||
// a torn prior reads as "no previous" (fresh write) with the adapter's
|
||||
// loud floor already fired. Never let corruption block its own cure.
|
||||
const existingMetadata = await this.readCanonicalObject(path).catch((err) => {
|
||||
if ((err as { code?: string }).code === 'TORN_RECORD') return null
|
||||
throw err
|
||||
})
|
||||
const isNew = !existingMetadata
|
||||
|
||||
// Save the metadata (write-cache coherent canonical write)
|
||||
|
|
@ -4696,13 +4740,21 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
} catch (error) {
|
||||
// A TORN record must surface typed — an enumeration that silently
|
||||
// skips a corrupt row hides data loss from the caller.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip nouns that fail to load
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// A TORN record propagates (typed) — only shard-listing absence is skippable.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip shards that have no data
|
||||
}
|
||||
}
|
||||
|
|
@ -4890,14 +4942,22 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
} catch (error) {
|
||||
// A TORN record must surface typed — an enumeration that silently
|
||||
// skips a corrupt row hides data loss from the caller.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip verbs that fail to load
|
||||
prodLog.debug(`[BaseStorage] Failed to load verb from ${verbPath}:`, error)
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// A TORN record propagates (typed) — only shard-listing absence is skippable.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip shards that have no data
|
||||
}
|
||||
}
|
||||
|
|
@ -5015,7 +5075,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
} catch (error) {
|
||||
// A TORN record propagates (typed) — batch hydration must not
|
||||
// silently drop a corrupt row. Only shard-listing absence is skippable.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip shards that have no data
|
||||
}
|
||||
}
|
||||
|
|
@ -5103,13 +5167,21 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
} catch (error) {
|
||||
// A TORN record must surface typed — an enumeration that silently
|
||||
// skips a corrupt row hides data loss from the caller.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip verbs that fail to load
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// A TORN record propagates (typed) — only shard-listing absence is skippable.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip shards that have no data
|
||||
}
|
||||
}
|
||||
|
|
@ -5156,13 +5228,21 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
} catch (error) {
|
||||
// A TORN record must surface typed — an enumeration that silently
|
||||
// skips a corrupt row hides data loss from the caller.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip verbs that fail to load
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// A TORN record propagates (typed) — only shard-listing absence is skippable.
|
||||
if (isTornRecordError(error)) throw error
|
||||
// Torn record inside an ENUMERATION/RECOVERY walk: the adapter already
|
||||
// narrated + counted it (TornRecordError registers at creation); the
|
||||
// walk's job is to HEAL PAST it — skip the victim, serve the rest.
|
||||
// Identity point-reads (get-by-id) still throw typed upstream.
|
||||
if (isTornRecordError(error)) { /* skip torn victim; loud floor already fired */ }
|
||||
// Skip shards that have no data
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue