fix(generations): a sealed segment may only declare the generations it holds
Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.
A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.
repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.
Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.
Three changes:
- repackHistory folds each contiguous RUN as its own segment
(`contiguousRuns`), so ranges describe exactly what the segments contain.
- fold() REFUSES a non-contiguous batch, naming the gap and its width. The
density law is now mechanical, so no future caller can reintroduce it. A
refusal loses nothing: the generations stay live and readable.
- Stores already carrying the damage heal instead of wedging. A segment
whose declared span exceeds its frame count is SPARSE; `actualRanges()`
reads the real generation list from its sidecar so open() never re-admits
the holes, and readFrame reports such a hole as unpacked with a narration
naming the segment, rather than throwing. A DENSE segment missing a frame
is still loud damage — that one means the manifest and sidecar disagree.
Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
This commit is contained in:
parent
b8475cc86a
commit
9a888c37e9
4 changed files with 389 additions and 13 deletions
|
|
@ -16,6 +16,7 @@ import { describe, it, expect, afterEach } from 'vitest'
|
|||
import * as fs from 'node:fs'
|
||||
import * as path from 'node:path'
|
||||
import * as os from 'node:os'
|
||||
import * as zlib from 'node:zlib'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { NounType } from '../../src/types/graphTypes.js'
|
||||
import { GenerationStore } from '../../src/db/generationStore.js'
|
||||
|
|
@ -57,6 +58,107 @@ describe('history repacking — the two-tier lifecycle', () => {
|
|||
}
|
||||
})
|
||||
|
||||
/**
|
||||
* THE HOLE, END TO END — the shape a real store carries.
|
||||
*
|
||||
* A forensic fixture was measured with generation directories 1..2503
|
||||
* present except for exactly one: 1416. Its fact-log segment already showed
|
||||
* the tell — `seg-...1410.bfl` declaring firstGeneration 1410, lastGeneration
|
||||
* 1940 (531 generations) while recording only 530 facts.
|
||||
*
|
||||
* Before the fix, repacking such a store folded ACROSS that hole: the batch
|
||||
* skipped 1416 (no readable delta) and the sealed segment declared a range
|
||||
* spanning it anyway. The next open merged that declared range back into
|
||||
* committedRanges, re-admitting 1416 as committed history, and every
|
||||
* subsequent auto-compaction pass then asked the packed tier for a frame
|
||||
* that was never written — producing, on EVERY run, the non-fatal narration
|
||||
*
|
||||
* Auto-compaction of generational history failed (non-fatal): generation
|
||||
* N is inside sealed segment seg-....bgs's declared range but has no frame
|
||||
* — packed history is damaged
|
||||
*
|
||||
* This pin removes a generation directory to make the same hole, then
|
||||
* requires repack + reopen + compaction to complete cleanly.
|
||||
*/
|
||||
it('a missing generation directory does not poison the packed tier', async () => {
|
||||
const dir = tempDir()
|
||||
// `retention: 'all'` throughout: close() otherwise auto-compacts the
|
||||
// history away, and this pin needs the cold generations still on disk so
|
||||
// there is something to punch a hole in. The live window stays at its
|
||||
// production default for the build phase, so nothing folds yet.
|
||||
const archival = async (): Promise<Brainy> => {
|
||||
const b = new Brainy({
|
||||
requireSubtype: false,
|
||||
storage: { type: 'filesystem', path: dir },
|
||||
embeddingFunction: stub,
|
||||
retention: 'all'
|
||||
})
|
||||
await b.init()
|
||||
return b
|
||||
}
|
||||
const brain = await archival()
|
||||
|
||||
const id = await brain.add({
|
||||
data: 'holed-entity',
|
||||
type: NounType.Document,
|
||||
metadata: { v: 0 }
|
||||
})
|
||||
// One flush per update: single-op writes coalesce inside a flush window,
|
||||
// so a history deep enough to have a middle needs the windows separated.
|
||||
for (let v = 1; v <= 12; v++) {
|
||||
await brain.update({ id, metadata: { v } })
|
||||
await brain.flush()
|
||||
}
|
||||
await brain.close()
|
||||
|
||||
// Punch the hole: delete ONE generation directory in the middle of the
|
||||
// cold range, exactly as the real store presents it.
|
||||
const genRoot = path.join(dir, '_generations')
|
||||
const numeric = fs
|
||||
.readdirSync(genRoot, { withFileTypes: true })
|
||||
.filter((e) => e.isDirectory() && /^\d+$/.test(e.name))
|
||||
.map((e) => Number(e.name))
|
||||
.sort((a, b) => a - b)
|
||||
expect(numeric.length).toBeGreaterThan(6)
|
||||
const victim = numeric[Math.floor(numeric.length / 2)]
|
||||
fs.rmSync(path.join(genRoot, String(victim)), { recursive: true, force: true })
|
||||
|
||||
// Now shrink the live window and reopen. close() repacks automatically
|
||||
// (brainy.ts phase 0b), so this is the production sequence exactly: a
|
||||
// store with a hole in its history gets folded by ordinary housekeeping,
|
||||
// with nobody asking for it.
|
||||
;(GenerationStore as any).REPACK_LIVE_WINDOW = 3
|
||||
const reopened = await archival()
|
||||
const result = await reopened.repackHistory()
|
||||
expect(result.foldedGenerations).toBeGreaterThan(0)
|
||||
|
||||
const segDir = path.join(dir, SEGMENTS_PREFIX)
|
||||
const manifestPath = ['manifest.json', 'manifest.json.gz']
|
||||
.map((f) => path.join(segDir, f))
|
||||
.find((p) => fs.existsSync(p))!
|
||||
const raw = manifestPath.endsWith('.gz')
|
||||
? zlib.gunzipSync(fs.readFileSync(manifestPath)).toString('utf8')
|
||||
: fs.readFileSync(manifestPath, 'utf8')
|
||||
const manifest = JSON.parse(raw) as {
|
||||
segments: Array<{ firstGeneration: number; lastGeneration: number; frames: number }>
|
||||
}
|
||||
|
||||
// THE LAW: every sealed segment declares exactly as many generations as it
|
||||
// holds frames, and none of them spans the victim.
|
||||
for (const s of manifest.segments) {
|
||||
expect(s.lastGeneration - s.firstGeneration + 1).toBe(s.frames)
|
||||
expect(victim >= s.firstGeneration && victim <= s.lastGeneration).toBe(false)
|
||||
}
|
||||
|
||||
await reopened.close()
|
||||
|
||||
// And the pass that used to fail on every run now completes: reopen (which
|
||||
// re-seeds committedRanges from the packed tier) then compact history.
|
||||
const third = await openBrain(dir)
|
||||
await expect(third.compactHistory({ maxGenerations: 2 })).resolves.toBeDefined()
|
||||
await third.close()
|
||||
})
|
||||
|
||||
it('repack preserves every historical read across cold reopen; folded dirs are gone', async () => {
|
||||
;(GenerationStore as any).REPACK_LIVE_WINDOW = 3
|
||||
const dir = tempDir()
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue