fix(generations): a sealed segment may only declare the generations it holds
Some checks failed
CI / Node 22 (push) Successful in 12m24s
CI / Node 24 (push) Successful in 12m21s
CI / Bun (latest) (push) Successful in 12m28s
CI / Integration + conformance (Node 22) (push) Failing after 16m55s

Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.

A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.

repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.

Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.

Three changes:

  - repackHistory folds each contiguous RUN as its own segment
    (`contiguousRuns`), so ranges describe exactly what the segments contain.
  - fold() REFUSES a non-contiguous batch, naming the gap and its width. The
    density law is now mechanical, so no future caller can reintroduce it. A
    refusal loses nothing: the generations stay live and readable.
  - Stores already carrying the damage heal instead of wedging. A segment
    whose declared span exceeds its frame count is SPARSE; `actualRanges()`
    reads the real generation list from its sidecar so open() never re-admits
    the holes, and readFrame reports such a hole as unpacked with a narration
    naming the segment, rather than throwing. A DENSE segment missing a frame
    is still loud damage — that one means the manifest and sidecar disagree.

Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
This commit is contained in:
David Snelling 2026-08-31 09:13:42 -07:00
parent b8475cc86a
commit 9a888c37e9
4 changed files with 389 additions and 13 deletions

View file

@ -147,4 +147,119 @@ describe('db/GenerationSegmentStore — the D1+D3 packed tier', () => {
await expect(store.fold([gen(4), gen(4)])).rejects.toThrow(/strictly ascending/)
await expect(store.fold([])).rejects.toThrow(/at least one generation/)
})
// ==========================================================================
// THE DENSITY LAW
// ==========================================================================
//
// A sealed segment declares a CONTIGUOUS range and every reader treats that
// range as containment. Folding a sparse batch therefore makes the segment
// claim generations it does not hold — and because `open()` merges declared
// ranges back into committedRanges, the hole is re-admitted as committed
// history and every later maintenance pass fails asking for a frame that was
// never written. That is the "generation N is inside sealed segment
// seg-....bgs's declared range but has no frame — packed history is damaged"
// narration seen on every run of the affected stores.
it('fold REFUSES a batch with a hole — a dense range may not be declared over sparse input', async () => {
await expect(store.fold([gen(1), gen(2), gen(4)])).rejects.toThrow(
/not contiguous: 2 → 4 skips 1 generation/
)
// The refusal loses nothing: no segment was sealed, so the generations
// stay in the live tier and the next pass folds them correctly.
expect(store.segments()).toHaveLength(0)
expect(store.hasGeneration(1)).toBe(false)
})
it('a wider gap names how many generations it would have swallowed', async () => {
await expect(store.fold([gen(10), gen(20)])).rejects.toThrow(
/not contiguous: 10 → 20 skips 9 generation\(s\)/
)
})
it('two contiguous runs folded separately declare honest ranges', async () => {
// What the caller now does instead of folding across the gap.
const a = await store.fold([gen(1), gen(2), gen(3)])
const b = await store.fold([gen(7), gen(8)])
expect(a).toMatchObject({ firstGeneration: 1, lastGeneration: 3, frames: 3 })
expect(b).toMatchObject({ firstGeneration: 7, lastGeneration: 8, frames: 2 })
// The gap is honestly outside the packed tier.
for (const g of [4, 5, 6]) expect(store.hasGeneration(g)).toBe(false)
for (const g of [1, 2, 3, 7, 8]) expect(store.hasGeneration(g)).toBe(true)
expect(await store.actualRanges()).toEqual([
[1, 3],
[7, 8]
])
})
it('actualRanges() is exact and I/O-free for dense segments', async () => {
await store.fold([gen(1), gen(2)])
await store.fold([gen(3), gen(4)])
// Adjacent dense segments each contribute their declared range.
expect(await store.actualRanges()).toEqual([
[1, 2],
[3, 4]
])
})
// ---- pre-existing damage: a store sealed by the old writer ----------------
/**
* Seal a SPARSE segment the way the pre-fix writer did: write the bytes and
* sidecar for a contiguous run, then rewrite the manifest so the segment
* declares a wider range than the frames it holds. This reproduces on disk
* exactly what the affected stores carry, without needing the old code.
*/
const sealSparseSegment = async (): Promise<void> => {
await store.fold([gen(1), gen(2), gen(3)])
const manifest = (await storage.readRawObject(`${SEGMENTS_PREFIX}/manifest.json`)) as any
// Declare 1..5 while holding frames for 1..3 — generations 4 and 5 become
// holes inside a sealed range.
manifest.segments[0].lastGeneration = 5
await storage.writeRawObject(`${SEGMENTS_PREFIX}/manifest.json`, manifest)
}
it('a pre-existing sparse segment reports its holes as UNPACKED, not as damage', async () => {
await sealSparseSegment()
const reopened = new GenerationSegmentStore(storage as any)
await reopened.open()
// The frames it really holds still serve, byte-faithfully.
expect((await reopened.readDelta(2))?.timestamp).toBe(1_700_000_000_002)
expect(await reopened.readRecords(3)).toHaveLength(2)
// The holes answer "not packed" instead of throwing. This is the fix for
// the wedge: the old reader threw here on EVERY maintenance pass.
expect(await reopened.readDelta(4)).toBeNull()
expect(await reopened.readRecords(5)).toBeNull()
})
it('actualRanges() excludes the holes so they are never re-admitted as committed', async () => {
await sealSparseSegment()
const reopened = new GenerationSegmentStore(storage as any)
await reopened.open()
// Declared 1..5; actually holds 1..3. The store seeds committedRanges from
// THIS, so generations 4 and 5 never become committed history again.
expect(await reopened.actualRanges()).toEqual([[1, 3]])
})
it('a DENSE segment missing a frame is still loud damage', async () => {
// The other side of the branch: when the manifest claims a complete span,
// a missing frame means the manifest and sidecar disagree — real damage,
// and it must not be quietly downgraded to "unpacked".
await store.fold([gen(1), gen(2), gen(3)])
const idxPath = `${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.idx`
const raw = (await storage.readRawBytes(idxPath))!
const { decode, encode } = await import('@msgpack/msgpack')
const idx = decode(raw) as any
// Drop generation 2's entry while the manifest still declares 3 frames.
idx.generations = idx.generations.filter(([g]: [number]) => g !== 2)
await storage.writeRawBytes(idxPath, encode(idx))
const reopened = new GenerationSegmentStore(storage as any)
await reopened.open()
await expect(reopened.readDelta(2)).rejects.toThrow(
/manifest and the sidecar disagree; packed history is damaged/
)
})
})