fix(generations): a sealed segment may only declare the generations it holds
Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.
A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.
repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.
Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.
Three changes:
- repackHistory folds each contiguous RUN as its own segment
(`contiguousRuns`), so ranges describe exactly what the segments contain.
- fold() REFUSES a non-contiguous batch, naming the gap and its width. The
density law is now mechanical, so no future caller can reintroduce it. A
refusal loses nothing: the generations stay live and readable.
- Stores already carrying the damage heal instead of wedging. A segment
whose declared span exceeds its frame count is SPARSE; `actualRanges()`
reads the real generation list from its sidecar so open() never re-admits
the holes, and readFrame reports such a hole as unpacked with a narration
naming the segment, rather than throwing. A DENSE segment missing a frame
is still loud damage — that one means the manifest and sidecar disagree.
Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
This commit is contained in:
parent
b8475cc86a
commit
9a888c37e9
4 changed files with 389 additions and 13 deletions
|
|
@ -147,4 +147,119 @@ describe('db/GenerationSegmentStore — the D1+D3 packed tier', () => {
|
|||
await expect(store.fold([gen(4), gen(4)])).rejects.toThrow(/strictly ascending/)
|
||||
await expect(store.fold([])).rejects.toThrow(/at least one generation/)
|
||||
})
|
||||
|
||||
// ==========================================================================
|
||||
// THE DENSITY LAW
|
||||
// ==========================================================================
|
||||
//
|
||||
// A sealed segment declares a CONTIGUOUS range and every reader treats that
|
||||
// range as containment. Folding a sparse batch therefore makes the segment
|
||||
// claim generations it does not hold — and because `open()` merges declared
|
||||
// ranges back into committedRanges, the hole is re-admitted as committed
|
||||
// history and every later maintenance pass fails asking for a frame that was
|
||||
// never written. That is the "generation N is inside sealed segment
|
||||
// seg-....bgs's declared range but has no frame — packed history is damaged"
|
||||
// narration seen on every run of the affected stores.
|
||||
|
||||
it('fold REFUSES a batch with a hole — a dense range may not be declared over sparse input', async () => {
|
||||
await expect(store.fold([gen(1), gen(2), gen(4)])).rejects.toThrow(
|
||||
/not contiguous: 2 → 4 skips 1 generation/
|
||||
)
|
||||
// The refusal loses nothing: no segment was sealed, so the generations
|
||||
// stay in the live tier and the next pass folds them correctly.
|
||||
expect(store.segments()).toHaveLength(0)
|
||||
expect(store.hasGeneration(1)).toBe(false)
|
||||
})
|
||||
|
||||
it('a wider gap names how many generations it would have swallowed', async () => {
|
||||
await expect(store.fold([gen(10), gen(20)])).rejects.toThrow(
|
||||
/not contiguous: 10 → 20 skips 9 generation\(s\)/
|
||||
)
|
||||
})
|
||||
|
||||
it('two contiguous runs folded separately declare honest ranges', async () => {
|
||||
// What the caller now does instead of folding across the gap.
|
||||
const a = await store.fold([gen(1), gen(2), gen(3)])
|
||||
const b = await store.fold([gen(7), gen(8)])
|
||||
expect(a).toMatchObject({ firstGeneration: 1, lastGeneration: 3, frames: 3 })
|
||||
expect(b).toMatchObject({ firstGeneration: 7, lastGeneration: 8, frames: 2 })
|
||||
// The gap is honestly outside the packed tier.
|
||||
for (const g of [4, 5, 6]) expect(store.hasGeneration(g)).toBe(false)
|
||||
for (const g of [1, 2, 3, 7, 8]) expect(store.hasGeneration(g)).toBe(true)
|
||||
expect(await store.actualRanges()).toEqual([
|
||||
[1, 3],
|
||||
[7, 8]
|
||||
])
|
||||
})
|
||||
|
||||
it('actualRanges() is exact and I/O-free for dense segments', async () => {
|
||||
await store.fold([gen(1), gen(2)])
|
||||
await store.fold([gen(3), gen(4)])
|
||||
// Adjacent dense segments each contribute their declared range.
|
||||
expect(await store.actualRanges()).toEqual([
|
||||
[1, 2],
|
||||
[3, 4]
|
||||
])
|
||||
})
|
||||
|
||||
// ---- pre-existing damage: a store sealed by the old writer ----------------
|
||||
|
||||
/**
|
||||
* Seal a SPARSE segment the way the pre-fix writer did: write the bytes and
|
||||
* sidecar for a contiguous run, then rewrite the manifest so the segment
|
||||
* declares a wider range than the frames it holds. This reproduces on disk
|
||||
* exactly what the affected stores carry, without needing the old code.
|
||||
*/
|
||||
const sealSparseSegment = async (): Promise<void> => {
|
||||
await store.fold([gen(1), gen(2), gen(3)])
|
||||
const manifest = (await storage.readRawObject(`${SEGMENTS_PREFIX}/manifest.json`)) as any
|
||||
// Declare 1..5 while holding frames for 1..3 — generations 4 and 5 become
|
||||
// holes inside a sealed range.
|
||||
manifest.segments[0].lastGeneration = 5
|
||||
await storage.writeRawObject(`${SEGMENTS_PREFIX}/manifest.json`, manifest)
|
||||
}
|
||||
|
||||
it('a pre-existing sparse segment reports its holes as UNPACKED, not as damage', async () => {
|
||||
await sealSparseSegment()
|
||||
const reopened = new GenerationSegmentStore(storage as any)
|
||||
await reopened.open()
|
||||
|
||||
// The frames it really holds still serve, byte-faithfully.
|
||||
expect((await reopened.readDelta(2))?.timestamp).toBe(1_700_000_000_002)
|
||||
expect(await reopened.readRecords(3)).toHaveLength(2)
|
||||
|
||||
// The holes answer "not packed" instead of throwing. This is the fix for
|
||||
// the wedge: the old reader threw here on EVERY maintenance pass.
|
||||
expect(await reopened.readDelta(4)).toBeNull()
|
||||
expect(await reopened.readRecords(5)).toBeNull()
|
||||
})
|
||||
|
||||
it('actualRanges() excludes the holes so they are never re-admitted as committed', async () => {
|
||||
await sealSparseSegment()
|
||||
const reopened = new GenerationSegmentStore(storage as any)
|
||||
await reopened.open()
|
||||
// Declared 1..5; actually holds 1..3. The store seeds committedRanges from
|
||||
// THIS, so generations 4 and 5 never become committed history again.
|
||||
expect(await reopened.actualRanges()).toEqual([[1, 3]])
|
||||
})
|
||||
|
||||
it('a DENSE segment missing a frame is still loud damage', async () => {
|
||||
// The other side of the branch: when the manifest claims a complete span,
|
||||
// a missing frame means the manifest and sidecar disagree — real damage,
|
||||
// and it must not be quietly downgraded to "unpacked".
|
||||
await store.fold([gen(1), gen(2), gen(3)])
|
||||
const idxPath = `${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.idx`
|
||||
const raw = (await storage.readRawBytes(idxPath))!
|
||||
const { decode, encode } = await import('@msgpack/msgpack')
|
||||
const idx = decode(raw) as any
|
||||
// Drop generation 2's entry while the manifest still declares 3 frames.
|
||||
idx.generations = idx.generations.filter(([g]: [number]) => g !== 2)
|
||||
await storage.writeRawBytes(idxPath, encode(idx))
|
||||
|
||||
const reopened = new GenerationSegmentStore(storage as any)
|
||||
await reopened.open()
|
||||
await expect(reopened.readDelta(2)).rejects.toThrow(
|
||||
/manifest and the sidecar disagree; packed history is damaged/
|
||||
)
|
||||
})
|
||||
})
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue