feat(log): log authority is the fleet default — adopt-at-open, oracle-gated; plus the power-cut throw-site cures and the loud torn-record contract
THE DEFAULT FLIP (ruled on proven evidence — at-ack survived 301/301 acked-writes-through-power-cut in block-layer fault injection; deferred tree authority demonstrably loses flush-covered acks): a brain with NO stored authority artifact now ADOPTS LOG AUTHORITY AT OPEN. The oracle gates the flip exactly as the guarded adoption path always did — curable divergences baseline-backfilled, the flip lands ONLY on a green verdict — and a brain that cannot verify STAYS tree-authoritative loudly, with the refusal recorded on the switch artifact so subsequent opens are cheap. config logAuthority: 'defer' is the explicit documented opt-out (no automatic adoption; declared flush-window loss; adoptLogAuthority() flips later). A stored artifact always wins. RELEASES.md carries the posture. Two standing .fails debt pins FLIP TO HOLDING under the default: the at-ack crash-survival gap and the ack-at-log durability target — both now permanent asserted truths, not aspirations. POWER-CUT THROW SITES (fault-injection findings, brainy-alone config): - A manifest-listed-but-unloadable column segment QUARANTINES at discovery (loud once, counted always, quarantinedSegments() exposed for the heal) and the field serves its remaining segments DEGRADED — never a raw throw killing every query on the field. Real storage faults still propagate untouched. - Torn generation artifacts (NaN/garbage in manifest or counter) DISCARD with narration at the store's open and recovery re-derives — plus a defensive finite-integer guard at the init consumer. Never a RangeError killing an open. THE LOUD TORN-RECORD CONTRACT: an existing-but-unparseable stored record now surfaces as a typed, counted TornRecordError on every entity-read surface (including fifteen previously-blind per-item batch catches); ENOENT stays clean-absent; artifact readers with designed absent-recovery keep null-tolerance behind the loud floor. Disk corruption can no longer read as silent data invisibility. Suite migration: the default's pins inverted deliberately, generation baselines made relative, quarantine-contract pins rewritten to the ruled behavior. Gates: tsc 0 · unit 2065/2065 (159 files) · integration 826 (93 files) · conformance 31/31 · kill-matrix 15/15 · torn-open guards 2/2.
This commit is contained in:
parent
67c606be69
commit
214c98b4d5
23 changed files with 833 additions and 154 deletions
|
|
@ -31,6 +31,7 @@ import { ColumnSegmentCursor, TailBufferCursor, type CursorEntry } from './Colum
|
|||
import { writeSegmentToBuffer, readSegmentFromBuffer } from './ColumnSegmentFormat.js'
|
||||
import { RoaringBitmap32 } from '../../utils/roaring/index.js'
|
||||
import { compareCodePoints } from '../../utils/collation.js'
|
||||
import { prodLog } from '../../utils/logger.js'
|
||||
|
||||
/**
|
||||
* Configuration for the ColumnStore.
|
||||
|
|
@ -612,6 +613,24 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
/**
|
||||
* Get all segment cursors for a field, loading from storage if needed.
|
||||
*/
|
||||
/**
|
||||
* Per-field quarantine ledger for torn segments (power-loss survivors:
|
||||
* manifest-listed but unloadable). A quarantined segment is skipped with
|
||||
* per-doubling narration and the field serves its REMAINING segments as a
|
||||
* DEGRADED-ANNOUNCED result — never a raw throw killing the query, never
|
||||
* a silent drop. Cleared when a heal/rebuild rewrites the field.
|
||||
*/
|
||||
private readonly segmentQuarantine = new Map<string, { error: string; hits: number }>()
|
||||
|
||||
/** Torn-segment quarantine entries for a field (observability + heal input). */
|
||||
quarantinedSegments(field: string): Array<{ segment: string; error: string; hits: number }> {
|
||||
const out: Array<{ segment: string; error: string; hits: number }> = []
|
||||
for (const [key, q] of this.segmentQuarantine) {
|
||||
if (key.startsWith(`${field}:`)) out.push({ segment: key.slice(field.length + 1), error: q.error, hits: q.hits })
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
private async getSegmentCursors(field: string): Promise<ColumnSegmentCursor[]> {
|
||||
const manifest = this.manifests.get(field)
|
||||
if (!manifest) return []
|
||||
|
|
@ -622,11 +641,38 @@ export class ColumnStore implements ColumnStoreProvider {
|
|||
let cursor = this.segmentCache.get(cacheKey)
|
||||
|
||||
if (!cursor) {
|
||||
// loadSegmentCursor either returns a cursor or THROWS — a corrupt /
|
||||
// missing manifest-listed segment raises ColumnSegmentLoadError and a
|
||||
// real storage fault propagates, so a listed segment is never silently
|
||||
// dropped from the result set.
|
||||
cursor = await this.loadSegmentCursor(field, seg)
|
||||
const quarantined = this.segmentQuarantine.get(cacheKey)
|
||||
if (quarantined) {
|
||||
// Already-quarantined torn segment: skip, count, narrate per doubling.
|
||||
quarantined.hits++
|
||||
if ((quarantined.hits & (quarantined.hits - 1)) === 0) {
|
||||
prodLog.warn(
|
||||
`[ColumnStore] field '${field}' serving DEGRADED: torn segment ${seg.id} ` +
|
||||
`quarantined (${quarantined.error}) — ${quarantined.hits} queries served ` +
|
||||
`without it; heal/rebuild the metadata index to restore`
|
||||
)
|
||||
}
|
||||
continue
|
||||
}
|
||||
try {
|
||||
cursor = await this.loadSegmentCursor(field, seg)
|
||||
} catch (err) {
|
||||
if (err instanceof ColumnSegmentLoadError) {
|
||||
// POWER-LOSS SURVIVOR: a manifest-listed segment whose bytes are
|
||||
// torn/absent. Quarantine at DISCOVERY and serve the remaining
|
||||
// segments degraded-announced — a raw throw here killed every
|
||||
// query on the field forever; a silent skip hid the loss. The
|
||||
// quarantine is the middle: loud once, counted always, healable.
|
||||
this.segmentQuarantine.set(cacheKey, { error: (err as Error).message, hits: 1 })
|
||||
prodLog.error(
|
||||
`[ColumnStore] torn segment QUARANTINED at discovery: field '${field}' ` +
|
||||
`segment ${seg.id} — ${(err as Error).message}. The field serves its ` +
|
||||
`remaining segments DEGRADED until a heal/rebuild rewrites it.`
|
||||
)
|
||||
continue
|
||||
}
|
||||
throw err // real storage faults propagate — never absorbed
|
||||
}
|
||||
this.segmentCache.set(cacheKey, cursor)
|
||||
}
|
||||
|
||||
|
|
|
|||
Reference in a new issue