fix(generations): a sealed segment may only declare the generations it holds

Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.

A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.

repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.

Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.

Three changes:

  - repackHistory folds each contiguous RUN as its own segment
    (`contiguousRuns`), so ranges describe exactly what the segments contain.
  - fold() REFUSES a non-contiguous batch, naming the gap and its width. The
    density law is now mechanical, so no future caller can reintroduce it. A
    refusal loses nothing: the generations stay live and readable.
  - Stores already carrying the damage heal instead of wedging. A segment
    whose declared span exceeds its frame count is SPARSE; `actualRanges()`
    reads the real generation list from its sidecar so open() never re-admits
    the holes, and readFrame reports such a hole as unpacked with a narration
    naming the segment, rather than throwing. A DENSE segment missing a frame
    is still loud damage — that one means the manifest and sidecar disagree.

Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.

tests/integration/history-repacking.test.ts

@@ -16,6 +16,7 @@ import { describe, it, expect, afterEach } from 'vitest'
 import * as fs from 'node:fs'
 import * as path from 'node:path'
 import * as os from 'node:os'
+import * as zlib from 'node:zlib'
 import { Brainy } from '../../src/brainy.js'
 import { NounType } from '../../src/types/graphTypes.js'
 import { GenerationStore } from '../../src/db/generationStore.js'
@@ -57,6 +58,107 @@ describe('history repacking — the two-tier lifecycle', () => {
     }
   })
 
+  /**
+   * THE HOLE, END TO END — the shape a real store carries.
+   *
+   * A forensic fixture was measured with generation directories 1..2503
+   * present except for exactly one: 1416. Its fact-log segment already showed
+   * the tell — `seg-...1410.bfl` declaring firstGeneration 1410, lastGeneration
+   * 1940 (531 generations) while recording only 530 facts.
+   *
+   * Before the fix, repacking such a store folded ACROSS that hole: the batch
+   * skipped 1416 (no readable delta) and the sealed segment declared a range
+   * spanning it anyway. The next open merged that declared range back into
+   * committedRanges, re-admitting 1416 as committed history, and every
+   * subsequent auto-compaction pass then asked the packed tier for a frame
+   * that was never written — producing, on EVERY run, the non-fatal narration
+   *
+   *   Auto-compaction of generational history failed (non-fatal): generation
+   *   N is inside sealed segment seg-....bgs's declared range but has no frame
+   *   — packed history is damaged
+   *
+   * This pin removes a generation directory to make the same hole, then
+   * requires repack + reopen + compaction to complete cleanly.
+   */
+  it('a missing generation directory does not poison the packed tier', async () => {
+    const dir = tempDir()
+    // `retention: 'all'` throughout: close() otherwise auto-compacts the
+    // history away, and this pin needs the cold generations still on disk so
+    // there is something to punch a hole in. The live window stays at its
+    // production default for the build phase, so nothing folds yet.
+    const archival = async (): Promise<Brainy> => {
+      const b = new Brainy({
+        requireSubtype: false,
+        storage: { type: 'filesystem', path: dir },
+        embeddingFunction: stub,
+        retention: 'all'
+      })
+      await b.init()
+      return b
+    }
+    const brain = await archival()
+
+    const id = await brain.add({
+      data: 'holed-entity',
+      type: NounType.Document,
+      metadata: { v: 0 }
+    })
+    // One flush per update: single-op writes coalesce inside a flush window,
+    // so a history deep enough to have a middle needs the windows separated.
+    for (let v = 1; v <= 12; v++) {
+      await brain.update({ id, metadata: { v } })
+      await brain.flush()
+    }
+    await brain.close()
+
+    // Punch the hole: delete ONE generation directory in the middle of the
+    // cold range, exactly as the real store presents it.
+    const genRoot = path.join(dir, '_generations')
+    const numeric = fs
+      .readdirSync(genRoot, { withFileTypes: true })
+      .filter((e) => e.isDirectory() && /^\d+$/.test(e.name))
+      .map((e) => Number(e.name))
+      .sort((a, b) => a - b)
+    expect(numeric.length).toBeGreaterThan(6)
+    const victim = numeric[Math.floor(numeric.length / 2)]
+    fs.rmSync(path.join(genRoot, String(victim)), { recursive: true, force: true })
+
+    // Now shrink the live window and reopen. close() repacks automatically
+    // (brainy.ts phase 0b), so this is the production sequence exactly: a
+    // store with a hole in its history gets folded by ordinary housekeeping,
+    // with nobody asking for it.
+    ;(GenerationStore as any).REPACK_LIVE_WINDOW = 3
+    const reopened = await archival()
+    const result = await reopened.repackHistory()
+    expect(result.foldedGenerations).toBeGreaterThan(0)
+
+    const segDir = path.join(dir, SEGMENTS_PREFIX)
+    const manifestPath = ['manifest.json', 'manifest.json.gz']
+      .map((f) => path.join(segDir, f))
+      .find((p) => fs.existsSync(p))!
+    const raw = manifestPath.endsWith('.gz')
+      ? zlib.gunzipSync(fs.readFileSync(manifestPath)).toString('utf8')
+      : fs.readFileSync(manifestPath, 'utf8')
+    const manifest = JSON.parse(raw) as {
+      segments: Array<{ firstGeneration: number; lastGeneration: number; frames: number }>
+    }
+
+    // THE LAW: every sealed segment declares exactly as many generations as it
+    // holds frames, and none of them spans the victim.
+    for (const s of manifest.segments) {
+      expect(s.lastGeneration - s.firstGeneration + 1).toBe(s.frames)
+      expect(victim >= s.firstGeneration && victim <= s.lastGeneration).toBe(false)
+    }
+
+    await reopened.close()
+
+    // And the pass that used to fail on every run now completes: reopen (which
+    // re-seeds committedRanges from the packed tier) then compact history.
+    const third = await openBrain(dir)
+    await expect(third.compactHistory({ maxGenerations: 2 })).resolves.toBeDefined()
+    await third.close()
+  })
+
   it('repack preserves every historical read across cold reopen; folded dirs are gone', async () => {
     ;(GenerationStore as any).REPACK_LIVE_WINDOW = 3
     const dir = tempDir()

tests/unit/db/generation-segments.test.ts

@@ -147,4 +147,119 @@ describe('db/GenerationSegmentStore — the D1+D3 packed tier', () => {
     await expect(store.fold([gen(4), gen(4)])).rejects.toThrow(/strictly ascending/)
     await expect(store.fold([])).rejects.toThrow(/at least one generation/)
   })
+
+  // ==========================================================================
+  // THE DENSITY LAW
+  // ==========================================================================
+  //
+  // A sealed segment declares a CONTIGUOUS range and every reader treats that
+  // range as containment. Folding a sparse batch therefore makes the segment
+  // claim generations it does not hold — and because `open()` merges declared
+  // ranges back into committedRanges, the hole is re-admitted as committed
+  // history and every later maintenance pass fails asking for a frame that was
+  // never written. That is the "generation N is inside sealed segment
+  // seg-....bgs's declared range but has no frame — packed history is damaged"
+  // narration seen on every run of the affected stores.
+
+  it('fold REFUSES a batch with a hole — a dense range may not be declared over sparse input', async () => {
+    await expect(store.fold([gen(1), gen(2), gen(4)])).rejects.toThrow(
+      /not contiguous: 2 → 4 skips 1 generation/
+    )
+    // The refusal loses nothing: no segment was sealed, so the generations
+    // stay in the live tier and the next pass folds them correctly.
+    expect(store.segments()).toHaveLength(0)
+    expect(store.hasGeneration(1)).toBe(false)
+  })
+
+  it('a wider gap names how many generations it would have swallowed', async () => {
+    await expect(store.fold([gen(10), gen(20)])).rejects.toThrow(
+      /not contiguous: 10 → 20 skips 9 generation\(s\)/
+    )
+  })
+
+  it('two contiguous runs folded separately declare honest ranges', async () => {
+    // What the caller now does instead of folding across the gap.
+    const a = await store.fold([gen(1), gen(2), gen(3)])
+    const b = await store.fold([gen(7), gen(8)])
+    expect(a).toMatchObject({ firstGeneration: 1, lastGeneration: 3, frames: 3 })
+    expect(b).toMatchObject({ firstGeneration: 7, lastGeneration: 8, frames: 2 })
+    // The gap is honestly outside the packed tier.
+    for (const g of [4, 5, 6]) expect(store.hasGeneration(g)).toBe(false)
+    for (const g of [1, 2, 3, 7, 8]) expect(store.hasGeneration(g)).toBe(true)
+    expect(await store.actualRanges()).toEqual([
+      [1, 3],
+      [7, 8]
+    ])
+  })
+
+  it('actualRanges() is exact and I/O-free for dense segments', async () => {
+    await store.fold([gen(1), gen(2)])
+    await store.fold([gen(3), gen(4)])
+    // Adjacent dense segments each contribute their declared range.
+    expect(await store.actualRanges()).toEqual([
+      [1, 2],
+      [3, 4]
+    ])
+  })
+
+  // ---- pre-existing damage: a store sealed by the old writer ----------------
+
+  /**
+   * Seal a SPARSE segment the way the pre-fix writer did: write the bytes and
+   * sidecar for a contiguous run, then rewrite the manifest so the segment
+   * declares a wider range than the frames it holds. This reproduces on disk
+   * exactly what the affected stores carry, without needing the old code.
+   */
+  const sealSparseSegment = async (): Promise<void> => {
+    await store.fold([gen(1), gen(2), gen(3)])
+    const manifest = (await storage.readRawObject(`${SEGMENTS_PREFIX}/manifest.json`)) as any
+    // Declare 1..5 while holding frames for 1..3 — generations 4 and 5 become
+    // holes inside a sealed range.
+    manifest.segments[0].lastGeneration = 5
+    await storage.writeRawObject(`${SEGMENTS_PREFIX}/manifest.json`, manifest)
+  }
+
+  it('a pre-existing sparse segment reports its holes as UNPACKED, not as damage', async () => {
+    await sealSparseSegment()
+    const reopened = new GenerationSegmentStore(storage as any)
+    await reopened.open()
+
+    // The frames it really holds still serve, byte-faithfully.
+    expect((await reopened.readDelta(2))?.timestamp).toBe(1_700_000_000_002)
+    expect(await reopened.readRecords(3)).toHaveLength(2)
+
+    // The holes answer "not packed" instead of throwing. This is the fix for
+    // the wedge: the old reader threw here on EVERY maintenance pass.
+    expect(await reopened.readDelta(4)).toBeNull()
+    expect(await reopened.readRecords(5)).toBeNull()
+  })
+
+  it('actualRanges() excludes the holes so they are never re-admitted as committed', async () => {
+    await sealSparseSegment()
+    const reopened = new GenerationSegmentStore(storage as any)
+    await reopened.open()
+    // Declared 1..5; actually holds 1..3. The store seeds committedRanges from
+    // THIS, so generations 4 and 5 never become committed history again.
+    expect(await reopened.actualRanges()).toEqual([[1, 3]])
+  })
+
+  it('a DENSE segment missing a frame is still loud damage', async () => {
+    // The other side of the branch: when the manifest claims a complete span,
+    // a missing frame means the manifest and sidecar disagree — real damage,
+    // and it must not be quietly downgraded to "unpacked".
+    await store.fold([gen(1), gen(2), gen(3)])
+    const idxPath = `${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.idx`
+    const raw = (await storage.readRawBytes(idxPath))!
+    const { decode, encode } = await import('@msgpack/msgpack')
+    const idx = decode(raw) as any
+    // Drop generation 2's entry while the manifest still declares 3 frames.
+    idx.generations = idx.generations.filter(([g]: [number]) => g !== 2)
+    await storage.writeRawBytes(idxPath, encode(idx))
+
+    const reopened = new GenerationSegmentStore(storage as any)
+    await reopened.open()
+    await expect(reopened.readDelta(2)).rejects.toThrow(
+      /manifest and the sidecar disagree; packed history is damaged/
+    )
+  })
 })