fix(generations): a sealed segment may only declare the generations it holds

Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.

A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.

repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.

Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.

Three changes:

  - repackHistory folds each contiguous RUN as its own segment
    (`contiguousRuns`), so ranges describe exactly what the segments contain.
  - fold() REFUSES a non-contiguous batch, naming the gap and its width. The
    density law is now mechanical, so no future caller can reintroduce it. A
    refusal loses nothing: the generations stay live and readable.
  - Stores already carrying the damage heal instead of wedging. A segment
    whose declared span exceeds its frame count is SPARSE; `actualRanges()`
    reads the real generation list from its sidecar so open() never re-admits
    the holes, and readFrame reports such a hole as unpacked with a narration
    naming the segment, rather than throwing. A DENSE segment missing a frame
    is still loud damage — that one means the manifest and sidecar disagree.

Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.

(cherry picked from commit 9a888c37e9)
This commit is contained in:
David Snelling 2026-08-31 09:13:42 -07:00
parent c99308710a
commit a963a744cc
4 changed files with 389 additions and 13 deletions

View file

@ -96,6 +96,35 @@ export const FOLD_CHECKPOINT_PATH = '_system/fold-checkpoint.json'
/** Storage-root-relative prefix of the per-generation record directories. */
export const GENERATIONS_PREFIX = '_generations'
/**
* @description Split an ascending list of fold candidates into maximal
* CONTIGUOUS runs `[7,8,9,12,13]` becomes `[[7,8,9],[12,13]]`.
*
* A sealed segment declares one dense range `[firstGeneration,
* lastGeneration]`, and every reader treats that range as containment. So a
* batch with a hole in it must never become one segment: it would claim a
* generation it does not hold, and the first read of that hole reports the
* packed history as damaged. One run, one segment the ranges then describe
* exactly what the segments contain.
*
* @param gens - Fold candidates, strictly ascending by generation.
* @returns One array per contiguous run, in ascending order. Empty in, empty out.
*/
export function contiguousRuns(gens: FoldGeneration[]): FoldGeneration[][] {
const runs: FoldGeneration[][] = []
let run: FoldGeneration[] = []
for (const g of gens) {
const prev = run[run.length - 1]
if (prev !== undefined && g.generation !== prev.generation + 1) {
runs.push(run)
run = []
}
run.push(g)
}
if (run.length > 0) runs.push(run)
return runs
}
/**
* @description Phases of the {@link GenerationStore.commitTransaction} commit
* protocol at which a test-only fault injector can simulate a process crash.
@ -784,9 +813,15 @@ export class GenerationStore {
if (storageSupportsFactLog(this.storage)) {
this.segments = new GenerationSegmentStore(this.storage)
await this.segments.open()
const packedRanges = this.segments
.segments()
.map((s): [number, number] => [s.firstGeneration, Math.min(s.lastGeneration, this.committed)])
// ACTUAL ranges, not declared ones. A segment sealed by a pre-density-law
// writer can declare a span wider than the frames it holds; seeding
// committedRanges from the declared span re-admits those holes as
// committed generations, and every later maintenance pass then asks for a
// frame that was never written. `actualRanges()` reads the real
// generation list from the sidecar for exactly those segments (and does
// no I/O for the dense ones, which is all of them on a healthy store).
const packedRanges = (await this.segments.actualRanges())
.map((r): [number, number] => [r[0], Math.min(r[1], this.committed)])
.filter(([lo, hi]) => lo <= hi)
if (packedRanges.length > 0) {
// Merge packed (older) + live (newer) interval sets — both ascending;
@ -3121,13 +3156,26 @@ export class GenerationStore {
foldInput.push({ generation: gen, timestamp: delta.timestamp, delta, records })
}
if (foldInput.length === 0) continue
await segments.fold(foldInput)
segmentsCreated++
// Segment + manifest durable → the live copies retire.
for (const g of foldInput) {
await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${g.generation}`)
// SPLIT AT DISCONTINUITIES. `eligible` is NOT contiguous — three
// filters above punch holes in it: a generation missing from
// committedRanges never appears, one still in the pending buffer is
// skipped, and one whose tx.json will not read is skipped. A sealed
// segment declares a DENSE range, so folding across such a hole makes
// the segment claim a generation it does not hold; the next open
// merges that mis-declared range into committedRanges, and every
// subsequent auto-compaction pass then asks for the missing frame and
// fails with "packed history is damaged". Fold each contiguous RUN as
// its own segment instead — same bytes, honest ranges.
for (const run of contiguousRuns(foldInput)) {
if (deadline !== undefined && Date.now() >= deadline) break
await segments.fold(run)
segmentsCreated++
// Segment + manifest durable → the live copies retire.
for (const g of run) {
await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${g.generation}`)
}
folded += run.length
}
folded += foldInput.length
}
if (folded > 0) {
prodLog.info(