David Snelling
a963a744cc
Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.
A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.
repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.
Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.
Three changes:
- repackHistory folds each contiguous RUN as its own segment
(`contiguousRuns`), so ranges describe exactly what the segments contain.
- fold() REFUSES a non-contiguous batch, naming the gap and its width. The
density law is now mechanical, so no future caller can reintroduce it. A
refusal loses nothing: the generations stay live and readable.
- Stores already carrying the damage heal instead of wedging. A segment
whose declared span exceeds its frame count is SPARSE; `actualRanges()`
reads the real generation list from its sidecar so open() never re-admits
the holes, and readFrame reports such a hole as unpacked with a narration
naming the segment, rather than throwing. A DENSE segment missing a frame
is still loud damage — that one means the manifest and sidecar disagree.
Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
(cherry picked from commit
|
||
|---|---|---|
| .. | ||
| aggregation | ||
| brainy | ||
| db | ||
| graph | ||
| hnsw | ||
| import | ||
| indexes/columnStore | ||
| neural | ||
| performance | ||
| plugin | ||
| reprojection | ||
| storage | ||
| transaction | ||
| types | ||
| universal | ||
| utils | ||
| vfs | ||
| boundary-no-native.test.ts | ||
| brainy-core.unit.test.ts | ||
| brainy-get-optimization.test.ts | ||
| cold-open-rebuild-gate.test.ts | ||
| create-entities-default.test.ts | ||
| fieldTypeInference.test.ts | ||
| get-index-status-readiness.test.ts | ||
| metadata-cold-read-guard.test.ts | ||
| migration-lock.test.ts | ||
| plugin-activation-loudness.test.ts | ||
| plugin-autodetect.test.ts | ||
| plugin-version-coupling.test.ts | ||
| plugin.test.ts | ||
| process-exit-sweep.test.ts | ||
| shutdown-hooks-lifecycle.test.ts | ||
| test-suite-coverage-guard.test.ts | ||
| type-filtering.unit.test.ts | ||
| validate-invariants-delegation.test.ts | ||
| vector-cold-read-guard.test.ts | ||
| vfs-multi-instance-diagnostic.test.ts | ||
| vfs-readdir-recursive.test.ts | ||
| vfs-restart-fix.test.ts | ||
| where-operator-validation.test.ts | ||