Compare commits

..

11 commits

Author SHA1 Message Date
4f1e27c9a0 docs(releases): the 11.0.5 note — graph-first finds in production, bounded recovery
Some checks failed
CI / Node 22 (push) Successful in 12m14s
CI / Node 24 (push) Successful in 12m12s
CI / Integration + conformance (Node 22) (push) Failing after 17m0s
CI / Bun (latest) (push) Successful in 12m36s
2026-09-02 08:32:21 -07:00
297a3d7657 docs(releases): the 10.4.9 note — graph-first finds, honest verb arrays, bounded recovery
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
2026-09-02 08:30:01 -07:00
7ab670b525 docs(releases): the 11.0.4 note — millisecond closes, storm-free rebuilds
Some checks failed
CI / Node 22 (push) Successful in 12m21s
CI / Node 24 (push) Successful in 12m12s
CI / Integration + conformance (Node 22) (push) Failing after 17m1s
CI / Bun (latest) (push) Successful in 12m24s
2026-09-01 13:55:27 -07:00
f097cbf6f2 docs(releases): the 10.4.7 note — count ledgers can no longer race themselves
Some checks are pending
CI / Node 22 (push) Waiting to run
CI / Node 24 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
2026-09-01 13:49:26 -07:00
e64e2bc175 docs(releases): the release-notes door — owner-language notes for both engines, backfilled
Some checks failed
CI / Node 22 (push) Successful in 12m23s
CI / Node 24 (push) Successful in 12m20s
CI / Integration + conformance (Node 22) (push) Failing after 17m3s
CI / Bun (latest) (push) Successful in 12m20s
The fleet's releases wall reads one public URL per product. These files are
that door for Brainy and Open Brainy: newest first, honest history from the
changelog, one entry appended by every release from here on.
2026-09-01 12:04:29 -07:00
655aa13ea7 build(release): the docs-push step retires — this engine documents itself in its own repository
Some checks failed
CI / Node 22 (push) Successful in 12m22s
CI / Node 24 (push) Successful in 12m25s
CI / Integration + conformance (Node 22) (push) Failing after 16m55s
CI / Bun (latest) (push) Successful in 12m28s
The one-doc-set ruling (2026-08-31) gives soulcraft.com/docs to the paid
product alone; the site serves redirects for the slugs this rail used to
push. The push script stays in the tree as history; the rail stops calling
it.
2026-08-31 09:30:46 -07:00
39c71ecdac Merge remote-tracking branch 'origin/reclaim/packed-history-density' 2026-08-31 09:30:08 -07:00
0759c03a82 Merge remote-tracking branch 'origin/fix/torn-log-tail-terminal-verdict' 2026-08-31 09:30:08 -07:00
9a888c37e9 fix(generations): a sealed segment may only declare the generations it holds
Some checks failed
CI / Node 22 (push) Successful in 12m24s
CI / Node 24 (push) Successful in 12m21s
CI / Bun (latest) (push) Successful in 12m28s
CI / Integration + conformance (Node 22) (push) Failing after 16m55s
Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.

A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.

repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.

Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.

Three changes:

  - repackHistory folds each contiguous RUN as its own segment
    (`contiguousRuns`), so ranges describe exactly what the segments contain.
  - fold() REFUSES a non-contiguous batch, naming the gap and its width. The
    density law is now mechanical, so no future caller can reintroduce it. A
    refusal loses nothing: the generations stay live and readable.
  - Stores already carrying the damage heal instead of wedging. A segment
    whose declared span exceeds its frame count is SPARSE; `actualRanges()`
    reads the real generation list from its sidecar so open() never re-admits
    the holes, and readFrame reports such a hole as unpacked with a narration
    naming the segment, rather than throwing. A DENSE segment missing a frame
    is still loud damage — that one means the manifest and sidecar disagree.

Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
2026-08-31 09:13:42 -07:00
David Snelling
298cb6daca fix(recovery): a torn generation-log tail is a terminal verdict, never a wait
Some checks failed
CI / Node 22 (push) Successful in 12m22s
CI / Node 24 (push) Successful in 12m21s
CI / Integration + conformance (Node 22) (push) Failing after 16m58s
CI / Bun (latest) (push) Successful in 12m23s
Two halves of one defect, found by a seeded-SIGKILL crash lane.

THE FALSE POSITIVE. stampEntityTree() recorded generationStore.generation()
— the ALLOCATED counter, a number a write in flight has claimed and may
never commit — while the JSDoc beside it already said the source is the
committed generation. Every crash inside a write window therefore produced
a spurious verdict at the next open: either 'sourceGeneration N is ahead of
the log head N-1' (the allocated generation died with the process) or
'rollup invariant nounCount: stamped X, observed Y' (the recovery fold
folded facts the stamp's counts predate). Both told the operator to run
repairIndex() — a whole-store recount — for a store that was coherent.
Measured before this commit: 4 of 11 SIGKILL cycles on a healthy store
raised one of the two. The stamp and the open now both read
committedGeneration(), which is what every other open-time watermark in the
class already reasons about.

THE TERMINAL VERDICT. A stamp still ahead of committed truth after the
recovery fold witnesses a generation that is not in the log — the stamp's
fsync outlived the tail's, and there is nothing to arrive. That is its own
verdict state now ('torn'), never folded in with 'incoherent': the two have
opposite cures. A writer open demotes it — the unusable stamped surface is
re-derived at the committed generation from the live counters, O(1),
straight-line, no loop and no await on external progress, narrated with
both count sets, the stamp's path and its committedAt. A read-only open
cannot re-stamp, so it says so and names the cure instead of guessing, and
still serves. Neither branch waits, and neither locks an owner out of a
canonical tree the stamp only describes.

Pins: the verifier returns the torn verdict with both generations; a
fabricated head-behind-source store narrates precisely, demotes inside a
bounded open, serves its rows, and is quiet at the next open (the demotion
converges); a read-only open narrates the same verdict and leaves the bytes
untouched.
2026-08-31 09:07:18 -07:00
b8475cc86a fix(release): the release page posts to this repository — soulcraftlabs/open-brainy, never the engine's
Some checks failed
CI / Node 22 (push) Successful in 12m21s
CI / Node 24 (push) Successful in 12m12s
CI / Bun (latest) (push) Successful in 12m25s
CI / Integration + conformance (Node 22) (push) Failing after 17m2s
Step 11 POSTed to repos/soulcraft/brainy while printing the correct URL; dormant only because FORGEJO_RELEASE_TOKEN was unset. Found during the 10.4.4 cut verification.
2026-08-28 13:00:42 -07:00
10 changed files with 823 additions and 41 deletions

76
releases/brainy.json Normal file
View file

@ -0,0 +1,76 @@
{
"product": "brainy",
"entries": [
{
"version": "11.0.5",
"date": "2026-09-02",
"headline": "Graph-first finds in production, and opens that stop rescanning history",
"items": [
"find({ connected, where }) now walks the neighbours first and filters only those rows through a native door — correct at every page and O(neighbours), never the whole store.",
"related() with a list of verb types returns every requested kind (a fast path had silently kept only the first).",
"Deferred-embedding recovery resumes from a low-water mark instead of rescanning the whole generation log at every open — measured at two minutes on a large brain, now milliseconds."
],
"url": null,
"thumb": null
},
{
"version": "11.0.4",
"date": "2026-09-01",
"headline": "Closes in milliseconds, index rebuilds without the disk-sync storm",
"items": [
"close() no longer pays deferred compaction or waits out an in-flight rebuild — measured 8 ms against the 4-minute closes it replaces; deferred work resumes at the next open, in the background.",
"The metadata index's rebuild syncs to disk per shard instead of per row, and the durability point moved to the publish step — the same guarantee, a fraction of the disk traffic.",
"A new native filter door evaluates queries over exactly the candidate rows a graph walk found, never the whole store."
],
"url": null,
"thumb": null
},
{
"version": "11.0.3",
"date": "2026-09-01",
"headline": "The embedding upgrade ceremony runs on every brain",
"items": [
"A brain opened through the standard plugin now carries its embedding-model identity, so the full-precision upgrade ceremony can run on it.",
"A one-fix release; nothing else changed."
],
"url": null,
"thumb": null
},
{
"version": "11.0.2",
"date": "2026-08-31",
"headline": "One embedding quality everywhere, 34× faster imports",
"items": [
"Every runtime embeds with the same full-precision model — search quality no longer depends on where you run.",
"Bulk embedding measured 3.14.2× faster, and an online re-embed ceremony upgrades existing stores without downtime.",
"The engine's change feed is documented, with the SSE/WebSocket fan-out pattern for realtime surfaces."
],
"url": null,
"thumb": null
},
{
"version": "11.0.1",
"date": "2026-08-31",
"headline": "Deletes inside transactions are safe",
"items": [
"Deleting relations inside a transact() no longer corrupts index bookkeeping.",
"A store that deletes its last relation keeps serving instead of refusing."
],
"url": null,
"thumb": null
},
{
"version": "11.0.0",
"date": "2026-08-28",
"headline": "One install, one engine — Brainy",
"items": [
"The former two-package pair is one package: the native engine under the familiar API. One import is the whole install.",
"A missing native build refuses loudly with its cures named; nothing falls back silently.",
"Stores open in place — no migration."
],
"url": null,
"thumb": null
}
],
"history": "The version line continues from the 4.3.x native-engine releases; their record lives in the product repository's CHANGELOG.md."
}

110
releases/open-brainy.json Normal file
View file

@ -0,0 +1,110 @@
{
"product": "open-brainy",
"entries": [
{
"version": "10.4.9",
"date": "2026-09-02",
"headline": "Graph-first finds, honest verb arrays, and opens that stop rescanning history",
"items": [
"find({ connected, where }) now walks the neighbours first and filters only those rows — correct at every page, and O(neighbours) instead of O(store).",
"related() with a list of verb types (or sources, or targets) returns every requested kind — four fast paths silently kept only the first.",
"Deferred-embedding recovery resumes from a low-water mark instead of rescanning the whole generation log at every open — measured at two minutes on a large brain, now milliseconds."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.9",
"thumb": null
},
{
"version": "10.4.7",
"date": "2026-09-01",
"headline": "Count ledgers can no longer race themselves",
"items": [
"Concurrent count flushes coalesce into one writer with a trailing pass — parallel flushes can no longer corrupt a store's count ledger.",
"Atomic writes carry a per-process sequence, so two processes' temp files can never collide."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.7",
"thumb": null
},
{
"version": "10.4.6",
"date": "2026-08-31",
"headline": "Transactions cross the index seam safely",
"items": [
"Deleting relations inside a transact() no longer fails against the metadata index — operations take a JSON-safe view at the moment they execute.",
"Fixes a class of transaction failures on stores with integer-mapped relation endpoints."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.6",
"thumb": null
},
{
"version": "10.4.5",
"date": "2026-08-31",
"headline": "Recovery tells the truth, docs live at home",
"items": [
"A torn generation-log tail is a terminal verdict with a named cure — never an endless wait at open.",
"A sealed segment declares only the generations it actually holds.",
"The engine's documentation now publishes from its own repository."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.5",
"thumb": null
},
{
"version": "10.4.4",
"date": "2026-08-28",
"headline": "Faster opens, quieter idle",
"items": [
"Opening a store discovers generations from directory names instead of walking the log, and answers \"any entities?\" with one directory read.",
"The flush-request watch is event-driven; idle stores stop paying a polling heartbeat.",
"A slow open now names the exact step it is in, so operators see what is being paid and why."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.4",
"thumb": null
},
{
"version": "10.4.3",
"date": "2026-08-27",
"headline": "Open Brainy, under its own name",
"items": [
"The same engine as 10.4.2, now published as @soulcraftlabs/brainy — the MIT reference engine, on The Source.",
"No code changes; your imports change once and everything else stays put."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.3",
"thumb": null
},
{
"version": "10.4.2",
"date": "2026-08-27",
"headline": "Vectors that lie are refused, counts that drift are caught",
"items": [
"A zero-norm vector is not a vector: the index refuses them, rebuilds skip them, and a sanctioned unvector door removes them cleanly.",
"The canonical count ledger derives from identity records and marks legacy-derived ledgers suspect at load.",
"Plugin activation failures keep their original error as cause, so the real frame reaches your logs."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.2",
"thumb": null
},
{
"version": "10.4.1",
"date": "2026-08-26",
"headline": "Writes that change nothing cost nothing",
"items": [
"The read gate is per index family, and a write carrying unchanged data never re-embeds.",
"The vectored-row count joins the ledger, so vector coverage is a number you can read, not a guess."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.1",
"thumb": null
},
{
"version": "10.4.0",
"date": "2026-08-26",
"headline": "Repair routing, the vector ledger, and honest empties",
"items": [
"Repairs route to the index that owns the damage, and the open gate closes the vector leg until coverage is proven.",
"An empty string is real data, not a missing field.",
"The metadata crossing never carries raw integer relation endpoints — a whole class of serialization faults closed."
],
"url": "https://source.soulcraft.com/soulcraftlabs/open-brainy/releases/tag/v10.4.0",
"thumb": null
}
],
"history": "Earlier releases are recorded in CHANGELOG.md in this repository."
}

View file

@ -237,7 +237,7 @@ fi
# and RELEASES.md are the record; this just gives The Source's UI a release page).
echo -e "${BLUE}🔟 Creating release page on The Source...${NC}"
if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then
if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \
if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraftlabs/open-brainy/releases" \
-H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \
-d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then
echo -e "${GREEN}✅ Release page created on The Source${NC}\n"
@ -248,17 +248,12 @@ else
echo -e "${RED}⚠️ FORGEJO_RELEASE_TOKEN unset — no release page created; tag + CHANGELOG remain the record${NC}\n"
fi
# Step 12: Push public docs to the soulcraft.com docs ingest door
# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when
# DOCS_INGEST_SECRET is unset; fails loudly (without undoing the publish —
# that already happened) when a push errors, so the docs site never
# silently trails npm.
echo -e "${BLUE}1⃣2⃣ Pushing public docs to soulcraft.com/docs...${NC}"
if node scripts/push-docs.js; then
echo -e "${GREEN}✅ Docs push step done${NC}\n"
else
echo -e "${RED}❌ Docs push FAILED — soulcraft.com/docs trails npm until re-run or interim sync${NC}\n"
fi
# Step 12 RETIRED (2026-08-31, CORTEX-SITE-BRAINY-RENAME round 12, David-ruled):
# soulcraft.com/docs carries the paid product's documentation only. This
# engine's documentation home is THIS repository — README and docs/ — and the
# site serves 301s for the slugs this rail used to push. The push script stays
# in the tree for history; the rail no longer calls it.
echo -e "${BLUE}Docs step: this engine documents itself in its own repo (site push retired 2026-08-31)${NC}"
echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}"

View file

@ -12497,6 +12497,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* healed by `repairIndex()`, whose unconditional recount rebuilds the
* rollups from a canonical walk and re-stamps. Best-effort: a stamp-write
* fault warns loudly but never fails the flush that carried real data.
*
* THE SOURCE IS `committedGeneration()`, NEVER `generation()`. The latter is
* the ALLOCATED counter a number a write in flight has claimed and may
* never commit. Stamping it made the stamp's generation label a claim about
* counts it was not taken at, and every crash inside a write window then
* produced a spurious verdict at the next open: either `sourceGeneration N
* is ahead of the log head N-1` (the allocated generation died with the
* process) or `rollup invariant 'nounCount': stamped X, observed Y` (the
* recovery fold folded facts the stamp's counts predate). MEASURED on the
* crash-consistency lane before this line changed: 4 of 11 SIGKILL cycles on
* a coherent store raised one of those two verdicts, each of them naming
* `repairIndex()` a whole-store recount as the cure for nothing.
*/
private async stampEntityTree(): Promise<void> {
if (this.isReadOnly) return
@ -12507,7 +12519,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
])
await writeFamilyStamp(this.storage, ENTITY_TREE_STAMP_PATH, {
family: 'entity-tree',
sourceGeneration: this.generationStore.generation(),
sourceGeneration: this.generationStore.committedGeneration(),
members: { mode: 'rollup', invariants: { nounCount, verbCount } }
})
} catch (error) {
@ -12520,16 +12532,24 @@ export class Brainy<T = any> implements BrainyInterface<T> {
/**
* @description Open-time coherence check for the entity tree's family stamp:
* compare `sourceGeneration` against the log head and the stamped rollup
* invariants against the live counters. Verdicts:
* compare `sourceGeneration` against the store's COMMITTED generation and
* the stamped rollup invariants against the live counters. Verdicts:
* - `coherent` / `absent` (legacy store; first flush stamps) silent.
* - `behind` benign for the tree (it is written BY the commit; only the
* stamp is stale a crash landed between commit and flush). Refreshed at
* the next flush.
* - `torn` a TORN GENERATION-LOG TAIL, handled by
* {@link demoteTornEntityTreeStamp}: terminal, never a wait.
* - `incoherent` LOUD: the tree or its counters diverged from what was
* stamped `repairIndex()` recounts from canonical and re-stamps.
* Never blocks open; a fault reading the stamp is surfaced as unverifiable,
* never conflated with absence.
*
* THE COMPARISON IS AGAINST `committedGeneration()`, matching what
* {@link stampEntityTree} writes and what every other open-time watermark in
* this class already reasons about (the fact-scan capability, the metadata /
* graph / HNSW watermark verdicts). Comparing against the allocated counter
* was the one place that disagreed, and disagreeing was the whole defect.
*/
private async verifyEntityTreeStamp(): Promise<void> {
let stamp: FamilyStamp | null
@ -12546,11 +12566,16 @@ export class Brainy<T = any> implements BrainyInterface<T> {
this.storage.getNounCount(),
this.storage.getVerbCount()
])
const verdict = verifyFamilyStamp(stamp, this.generationStore.generation(), {
const verdict = verifyFamilyStamp(stamp, this.generationStore.committedGeneration(), {
nounCount,
verbCount
})
if (verdict.state === 'incoherent') {
if (verdict.state === 'torn') {
await this.demoteTornEntityTreeStamp(stamp as FamilyStamp, verdict.stampSource, verdict.head, {
nounCount,
verbCount
})
} else if (verdict.state === 'incoherent') {
prodLog.warn(
`[Brainy] entity-tree stamp INCOHERENT at open: ${verdict.failures.join('; ')}. ` +
`The canonical tree or its counters diverged from the stamped state — run ` +
@ -12564,6 +12589,92 @@ export class Brainy<T = any> implements BrainyInterface<T> {
}
}
/**
* @description THE TERMINAL VERDICT for a torn generation-log tail.
*
* A stamp whose `sourceGeneration` sits ABOVE the store's committed
* watermark witnesses a generation that is not in the log: the stamp's fsync
* outlived the tail's. By the time this runs, log-authority recovery has
* already folded every intact fact above the manifest and advanced the
* watermark to cover them so if the stamp is STILL ahead, the generation
* it names is not merely late, it is GONE. There is nothing to wait for.
*
* That is the whole point of this method. A field report of this class
* (single-process store, abrupt termination mid-fold) described a reopen
* that narrated the tear and then held 100% CPU with zero log growth for
* eight minutes before an operator wiped the directory. A recovery that
* cannot say what it is waiting for has no business spinning; the honest
* answer here is a verdict, taken now, at O(1) cost.
*
* WHAT THE VERDICT DOES the stamped surface is UNUSABLE, so it is
* discarded rather than believed: the stamped counts describe a generation
* that never became durable, and comparing them against live counters can
* only produce noise. The tree itself is not in question (it IS canonical
* every commit writes it, and the fold re-applied every after-image the log
* still holds), so the demotion is a re-derivation of this family's verified
* surface at the generation the store can actually show:
*
* - WRITER open re-stamp at `committedGeneration()` from the live
* counters exactly what the next flush would write, taken now so the
* tear cannot re-narrate on every subsequent open. Both count sets are
* logged so an operator can see whether anything really moved.
* - READER open a reader cannot re-stamp. Narrate the same terminal
* verdict with the named cure and carry on serving; a read-only inspector
* is never locked out of a store, and never left waiting either.
*
* BOUNDEDNESS: straight-line code. No loop, no retry, no await on any
* external progress signal the two counter reads and one stamp write are
* the entire cost, and none of them scales with the store.
*/
private async demoteTornEntityTreeStamp(
stamp: FamilyStamp,
stampSource: number,
head: number,
observed: { nounCount: number; verbCount: number }
): Promise<void> {
const stamped = stamp.members.mode === 'rollup' ? stamp.members.invariants : {}
const detail =
`[Brainy] TORN GENERATION-LOG TAIL at open: ${ENTITY_TREE_STAMP_PATH} witnesses source ` +
`generation ${stampSource} (stamped ${stamp.committedAt}), but the store's committed ` +
`generation is ${head} after crash recovery — the stamp's fsync outlived the log tail's, ` +
`and generation ${stampSource} is not in the log to arrive. Stamped rollups ` +
`${JSON.stringify(stamped)}; observed ${JSON.stringify(observed)}.`
if (this.isReadOnly) {
prodLog.warn(
`${detail} This open is READ-ONLY, so the stamp cannot be re-derived: the entity-tree ` +
`family stays UNVERIFIED for this session (reads are unaffected — the canonical tree ` +
`is the truth this stamp only describes). Cure: open the store with a writer, or run ` +
`brain.repairIndex() there, to recount from canonical and re-stamp.`
)
return
}
const startedAt = Date.now()
try {
await writeFamilyStamp(this.storage, ENTITY_TREE_STAMP_PATH, {
family: 'entity-tree',
sourceGeneration: head,
members: {
mode: 'rollup',
invariants: { nounCount: observed.nounCount, verbCount: observed.verbCount }
}
})
prodLog.warn(
`${detail} DEMOTED: the unusable stamp was re-derived at committed generation ${head} ` +
`from the live counters in ${Date.now() - startedAt}ms — terminal, not a wait. If the ` +
`observed counts above look wrong for your data, run brain.repairIndex() to recount ` +
`from canonical.`
)
} catch (error) {
prodLog.warn(
`${detail} The demotion's re-stamp FAILED (${(error as Error).message}) — the tear will ` +
`narrate again at the next open, which is the honest outcome; the store still serves ` +
`from canonical. Cure: run brain.repairIndex() to recount from canonical and re-stamp.`
)
}
}
/**
* Ask the writer process serving this data directory to flush its in-memory
* indexes to disk, so a read-only inspector can observe fresh state.

View file

@ -12,9 +12,11 @@
* the verified surface is a small set of rollup invariants (entity/
* relationship counts) plus `sourceGeneration`.
*
* `sourceGeneration` is the generation of the source-of-truth log this
* projection reflects open-time coherence becomes a COMPARISON (stamp vs
* log head), not a walk:
* `sourceGeneration` is the COMMITTED generation of the source-of-truth log
* this projection reflects never the allocated counter, which names a
* generation that may never commit (see {@link StampVerdict.torn}) so
* open-time coherence becomes a COMPARISON (stamp vs committed head), not a
* walk:
*
* - equal + invariants hold coherent, serve.
* - behind the projection missed the tail (crash between commit and stamp);
@ -24,6 +26,9 @@
* - invariants FAIL at equal generation genuine incoherence: loud, and the
* repair ritual (`repairIndex()`, whose recount rebuilds the rollups from a
* canonical walk) heals it.
* - AHEAD a torn generation-log tail: the stamp's fsync outlived the log
* tail's. TERMINAL, never a wait the generation the stamp names does not
* exist to arrive.
*
* Stamps are JSON on purpose every incident gets debugged by reading a
* stamp in a terminal.
@ -70,6 +75,12 @@ export type StampVerdict =
| { state: 'coherent' }
| { state: 'absent' } // legacy store — first stamp writes at the next flush
| { state: 'behind'; stampSource: number; head: number }
/**
* TORN GENERATION-LOG TAIL: the stamp witnesses a source generation the
* store's committed watermark can no longer show. TERMINAL there is no
* generation to wait for, so the open demotes (or refuses) and never spins.
*/
| { state: 'torn'; stampSource: number; head: number }
| { state: 'incoherent'; failures: string[] }
| { state: 'unverifiable'; reason: string } // a FAULT reading the stamp — never conflated with absence
@ -118,12 +129,15 @@ export function verifyFamilyStamp(
): StampVerdict {
if (stamp === null) return { state: 'absent' }
if (stamp.sourceGeneration > head) {
// A stamp AHEAD of the log claims state that never committed — the
// projection was stamped against truth that a crash rolled back.
return {
state: 'incoherent',
failures: [`sourceGeneration ${stamp.sourceGeneration} is ahead of the log head ${head}`]
}
// A stamp AHEAD of committed truth witnesses a generation the store can no
// longer show: the stamp's fsync survived a crash that the log tail did
// not. This is the TORN GENERATION-LOG TAIL — its own class, never folded
// in with `incoherent` (a count that drifted at a generation both sides
// agree on), because the two have opposite cures: incoherence is recounted,
// a tear is DEMOTED. It is also terminal by construction — there is no
// generation the open can wait for, because the one the stamp names is
// gone.
return { state: 'torn', stampSource: stamp.sourceGeneration, head }
}
if (stamp.sourceGeneration < head) {
return { state: 'behind', stampSource: stamp.sourceGeneration, head }

View file

@ -147,6 +147,60 @@ export class GenerationSegmentStore {
return this.coveringSegment(gen) !== null
}
/**
* @description True when `meta` declares more generations than it holds
* frames a segment sealed by a writer that folded across a hole. The
* manifest records `frames` at fold time, so this is an O(1) comparison
* against the declared span and needs no I/O.
*/
private isSparse(meta: SegmentMeta): boolean {
return meta.lastGeneration - meta.firstGeneration + 1 !== meta.frames
}
/**
* @description The generations this tier ACTUALLY holds, as coalesced
* ascending intervals not what the segments declare.
*
* Dense segments (every one a current writer produces) contribute their
* declared range with no I/O. A SPARSE segment one sealed before the
* density law was enforced, whose declared range spans generations it has
* no frame for has its real generation list read from its sidecar and
* contributed instead, with the discrepancy narrated once.
*
* This is what keeps a store that already carries the damage from wedging.
* `open()` seeds `committedRanges` from these intervals, so a hole is never
* re-admitted as a committed generation, and the auto-compaction pass that
* used to fail on every run with "packed history is damaged" simply never
* asks for the missing frame.
*
* @returns Ascending, non-overlapping `[first, last]` intervals.
*/
async actualRanges(): Promise<Array<[number, number]>> {
const out: Array<[number, number]> = []
for (const meta of this.manifest.segments) {
if (!this.isSparse(meta)) {
out.push([meta.firstGeneration, meta.lastGeneration])
continue
}
const missing = meta.lastGeneration - meta.firstGeneration + 1 - meta.frames
prodLog.warn(
`[GenerationSegments] sealed segment ${meta.file} declares generations ` +
`${meta.firstGeneration}..${meta.lastGeneration} but holds only ${meta.frames} ` +
`frame(s) — ${missing} generation(s) in that span were never folded into it. ` +
`Serving the frames it actually holds; the declared span is not treated as ` +
`committed history. (Written by a pre-density-law writer that folded across a ` +
`gap; the segment itself is intact and no record is lost.)`
)
const idx = await this.sidecarFor(meta)
for (const [gen] of idx.generations) {
const last = out[out.length - 1]
if (last !== undefined && gen === last[1] + 1) last[1] = gen
else out.push([gen, gen])
}
}
return out
}
/**
* Fold consecutive generations into ONE new sealed segment + sidecar and
* append it to the manifest atomically. Caller guarantees: `gens` is
@ -164,6 +218,38 @@ export class GenerationSegmentStore {
throw new Error('[GenerationSegments] fold() input must be strictly ascending')
}
}
// THE DENSITY LAW, MADE MECHANICAL.
//
// A sealed segment declares a CONTIGUOUS range [firstGeneration,
// lastGeneration] and every reader treats that range as containment:
// `coveringSegment` is an interval test, `hasGeneration` returns true for
// anything inside it, and `open()` seeds committedRanges from it. So a
// segment folded from a SPARSE input silently claims generations it does
// not hold, and the first read of one of those holes throws
// "inside sealed segment ... but has no frame — packed history is damaged".
//
// That is exactly how the damage was produced. `repackHistory` skipped
// generations mid-batch — ones absent from committedRanges, ones still in
// the pending buffer, ones whose tx.json would not read — and handed the
// survivors here, where the range was computed from the first and last of
// them. Worse, the mis-declared range was then merged back into
// committedRanges at the next open, which is what turned a quiet hole into
// a repeating auto-compaction failure on every subsequent run.
//
// Callers now split at discontinuities; this refusal is what keeps any
// future caller from reintroducing the class. A refusal here loses
// nothing — the generations stay in the live tier, readable, and the next
// pass folds them correctly.
for (let i = 1; i < gens.length; i++) {
if (gens[i].generation !== gens[i - 1].generation + 1) {
throw new Error(
`[GenerationSegments] fold() input is not contiguous: ${gens[i - 1].generation}` +
`${gens[i].generation} skips ${gens[i].generation - gens[i - 1].generation - 1} ` +
`generation(s). A sealed segment declares a dense range, so folding a sparse ` +
`batch would claim generations it does not hold. Split the batch at the gap.`
)
}
}
const last = this.manifest.segments[this.manifest.segments.length - 1]
if (last && gens[0].generation <= last.lastGeneration) {
throw new Error(
@ -364,12 +450,37 @@ export class GenerationSegmentStore {
return this.decodeFrame(payload)
}
}
// In the covering range but not present: the packed tier is dense by
// construction (fold packs every generation it is handed, including
// record-less ones) — absence inside a sealed range is damage.
// Inside the covering range but with no frame. Two very different causes,
// and conflating them is what made this class wedge every maintenance pass
// on the affected stores.
//
// (1) A SPARSE SEGMENT — the manifest's own `frames` count is smaller than
// the span it declares. That segment was sealed by a writer that
// folded across a hole (the class this file's density law now bars).
// The segment is INTACT and nothing is lost; it simply never held this
// generation. Answering "not packed" is the honest answer, and it lets
// the caller's two-tier read decide what a genuinely absent generation
// means, instead of every compaction pass dying on a repeating throw.
// `actualRanges()` keeps such holes out of committedRanges at open, so
// in a healed store nobody asks this question in the first place.
//
// (2) A DENSE SEGMENT missing a frame it says it has — the manifest and
// the sidecar disagree about a segment that claims to be complete.
// That IS damage, and it stays loud.
if (this.isSparse(meta)) {
prodLog.warn(
`[GenerationSegments] generation ${gen} falls inside sealed segment ${meta.file}'s ` +
`declared range ${meta.firstGeneration}..${meta.lastGeneration}, but that segment ` +
`holds ${meta.frames} frame(s) for a ${meta.lastGeneration - meta.firstGeneration + 1}` +
`-generation span — it was sealed across a gap and never held this generation. ` +
`Reporting it as unpacked rather than as damage; no record is lost.`
)
return null
}
throw new Error(
`[GenerationSegments] generation ${gen} is inside sealed segment ${meta.file}'s declared ` +
`range but has no frame — packed history is damaged`
`range but has no frame, and that segment declares a complete ${meta.frames}-frame ` +
`span — the manifest and the sidecar disagree; packed history is damaged`
)
}

View file

@ -96,6 +96,35 @@ export const FOLD_CHECKPOINT_PATH = '_system/fold-checkpoint.json'
/** Storage-root-relative prefix of the per-generation record directories. */
export const GENERATIONS_PREFIX = '_generations'
/**
* @description Split an ascending list of fold candidates into maximal
* CONTIGUOUS runs `[7,8,9,12,13]` becomes `[[7,8,9],[12,13]]`.
*
* A sealed segment declares one dense range `[firstGeneration,
* lastGeneration]`, and every reader treats that range as containment. So a
* batch with a hole in it must never become one segment: it would claim a
* generation it does not hold, and the first read of that hole reports the
* packed history as damaged. One run, one segment the ranges then describe
* exactly what the segments contain.
*
* @param gens - Fold candidates, strictly ascending by generation.
* @returns One array per contiguous run, in ascending order. Empty in, empty out.
*/
export function contiguousRuns(gens: FoldGeneration[]): FoldGeneration[][] {
const runs: FoldGeneration[][] = []
let run: FoldGeneration[] = []
for (const g of gens) {
const prev = run[run.length - 1]
if (prev !== undefined && g.generation !== prev.generation + 1) {
runs.push(run)
run = []
}
run.push(g)
}
if (run.length > 0) runs.push(run)
return runs
}
/**
* @description Phases of the {@link GenerationStore.commitTransaction} commit
* protocol at which a test-only fault injector can simulate a process crash.
@ -784,9 +813,15 @@ export class GenerationStore {
if (storageSupportsFactLog(this.storage)) {
this.segments = new GenerationSegmentStore(this.storage)
await this.segments.open()
const packedRanges = this.segments
.segments()
.map((s): [number, number] => [s.firstGeneration, Math.min(s.lastGeneration, this.committed)])
// ACTUAL ranges, not declared ones. A segment sealed by a pre-density-law
// writer can declare a span wider than the frames it holds; seeding
// committedRanges from the declared span re-admits those holes as
// committed generations, and every later maintenance pass then asks for a
// frame that was never written. `actualRanges()` reads the real
// generation list from the sidecar for exactly those segments (and does
// no I/O for the dense ones, which is all of them on a healthy store).
const packedRanges = (await this.segments.actualRanges())
.map((r): [number, number] => [r[0], Math.min(r[1], this.committed)])
.filter(([lo, hi]) => lo <= hi)
if (packedRanges.length > 0) {
// Merge packed (older) + live (newer) interval sets — both ascending;
@ -3121,13 +3156,26 @@ export class GenerationStore {
foldInput.push({ generation: gen, timestamp: delta.timestamp, delta, records })
}
if (foldInput.length === 0) continue
await segments.fold(foldInput)
// SPLIT AT DISCONTINUITIES. `eligible` is NOT contiguous — three
// filters above punch holes in it: a generation missing from
// committedRanges never appears, one still in the pending buffer is
// skipped, and one whose tx.json will not read is skipped. A sealed
// segment declares a DENSE range, so folding across such a hole makes
// the segment claim a generation it does not hold; the next open
// merges that mis-declared range into committedRanges, and every
// subsequent auto-compaction pass then asks for the missing frame and
// fails with "packed history is damaged". Fold each contiguous RUN as
// its own segment instead — same bytes, honest ranges.
for (const run of contiguousRuns(foldInput)) {
if (deadline !== undefined && Date.now() >= deadline) break
await segments.fold(run)
segmentsCreated++
// Segment + manifest durable → the live copies retire.
for (const g of foldInput) {
for (const g of run) {
await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${g.generation}`)
}
folded += foldInput.length
folded += run.length
}
}
if (folded > 0) {
prodLog.info(

View file

@ -57,7 +57,11 @@ describe('entity-tree family stamp', () => {
const invariants = (stamp.members as any).invariants
expect(invariants.nounCount).toBe(await brain.storage.getNounCount())
expect(invariants.verbCount).toBe(await brain.storage.getVerbCount())
expect(stamp.sourceGeneration).toBe(brain.generation())
// THE SOURCE IS COMMITTED TRUTH, never the allocated counter. Stamping the
// counter labelled the stamp with a generation a write in flight had merely
// claimed, so every crash inside a write window produced a spurious verdict
// at the next open (see the torn-tail pins below).
expect(stamp.sourceGeneration).toBe(brain.generationStore.committedGeneration())
expect(stamp.generation).toBeGreaterThanOrEqual(1)
})
@ -112,6 +116,96 @@ describe('entity-tree family stamp', () => {
expect(stillIncoherent).toEqual([])
})
/**
* Rewrite the on-disk stamp so its `sourceGeneration` sits ABOVE the store's
* committed watermark the durable shape a torn generation-log tail leaves
* behind (the stamp's fsync outlived the tail's). Fabricated rather than
* crash-produced so the pin is deterministic; the seeded-SIGKILL lane
* (`scripts/crash-consistency.mjs` in the engine repo) produces the same
* shape from a real abrupt termination.
*/
const fabricateTear = (ahead: number): FamilyStamp => {
const file = path.join(dir, `${ENTITY_TREE_STAMP_PATH}.gz`)
const zlib = require('node:zlib')
const raw = JSON.parse(zlib.gunzipSync(fs.readFileSync(file)).toString('utf-8')) as FamilyStamp
const torn: FamilyStamp = { ...raw, sourceGeneration: raw.sourceGeneration + ahead }
fs.writeFileSync(file, zlib.gzipSync(JSON.stringify(torn)))
return torn
}
it('a torn generation-log tail is a TERMINAL VERDICT at open: narrated, demoted, never a wait', async () => {
for (let i = 0; i < 3; i++)
await brain.add({ data: `torn${i}`, type: 'document', metadata: { i } })
await brain.close()
const torn = fabricateTear(5)
const warn = vi.spyOn(prodLog, 'warn')
const startedAt = Date.now()
brain = await open()
const openMs = Date.now() - startedAt
const tearLines = warn.mock.calls.filter((c) => String(c[0]).includes('TORN GENERATION-LOG TAIL'))
expect(tearLines.length).toBe(1)
const said = String(tearLines[0][0])
// Narrated PRECISELY: both generations, the file, and the named cure.
expect(said).toContain(`source generation ${torn.sourceGeneration}`)
expect(said).toContain(`committed generation ${brain.generationStore.committedGeneration()}`)
expect(said).toContain(ENTITY_TREE_STAMP_PATH)
expect(said).toContain('DEMOTED')
expect(said).toMatch(/repairIndex\(\)/)
// Terminal, not a wait: the demotion is O(1) straight-line work, so a tear
// cannot turn an open into the 8-minute spin this class was reported as.
expect(openMs).toBeLessThan(30_000)
// The store SERVES — a tear in a stamp never locks an owner out of the
// canonical tree the stamp merely describes.
expect((await brain.find({ type: 'document', limit: 100 })).length).toBe(3)
// The demotion CONVERGED: the stamp now names committed truth, and the
// next open is quiet. A verdict that re-narrates every open is a wait
// wearing a different hat.
const restamped = (await readFamilyStamp(brain.storage, ENTITY_TREE_STAMP_PATH)) as FamilyStamp
expect(restamped.sourceGeneration).toBe(brain.generationStore.committedGeneration())
await brain.close()
const warn2 = vi.spyOn(prodLog, 'warn')
brain = await open()
expect(warn2.mock.calls.filter((c) => String(c[0]).includes('TORN'))).toEqual([])
})
it('a READ-ONLY open on a torn tail refuses to guess: terminal verdict + named cure, no re-stamp', async () => {
await brain.add({ data: 'ro', type: 'document', metadata: {} })
await brain.close()
const torn = fabricateTear(3)
const warn = vi.spyOn(prodLog, 'warn')
const reader: any = await Brainy.openReadOnly({
requireSubtype: false,
storage: { type: 'filesystem', path: dir },
silent: true,
dimensions: 384
})
const tearLines = warn.mock.calls.filter((c) => String(c[0]).includes('TORN GENERATION-LOG TAIL'))
expect(tearLines.length).toBe(1)
const said = String(tearLines[0][0])
expect(said).toContain('READ-ONLY')
expect(said).toContain('UNVERIFIED')
expect(said).toMatch(/repairIndex\(\)/)
await reader.close()
// A reader never rewrites the store: read the bytes back off disk (not
// through a writer open, which would demote them) — the torn stamp is
// exactly as it was found.
const onDisk = JSON.parse(
require('node:zlib')
.gunzipSync(fs.readFileSync(path.join(dir, `${ENTITY_TREE_STAMP_PATH}.gz`)))
.toString('utf-8')
) as FamilyStamp
expect(onDisk.sourceGeneration).toBe(torn.sourceGeneration)
expect(onDisk.generation).toBe(torn.generation)
brain = await open()
})
it('the one verifier handles both member modes', () => {
const rollup: FamilyStamp = {
family: 'x',
@ -127,7 +221,13 @@ describe('entity-tree family stamp', () => {
stampSource: 5,
head: 9
})
expect(verifyFamilyStamp(rollup, 3, { nounCount: 10 }).state).toBe('incoherent') // ahead of head
// AHEAD is its own class — a torn generation-log tail, never folded in
// with `incoherent`: the two have opposite cures (recount vs demote).
expect(verifyFamilyStamp(rollup, 3, { nounCount: 10 })).toEqual({
state: 'torn',
stampSource: 5,
head: 3
})
expect(verifyFamilyStamp(null, 5, {})).toEqual({ state: 'absent' })
const enumerated: FamilyStamp = {

View file

@ -16,6 +16,7 @@ import { describe, it, expect, afterEach } from 'vitest'
import * as fs from 'node:fs'
import * as path from 'node:path'
import * as os from 'node:os'
import * as zlib from 'node:zlib'
import { Brainy } from '../../src/brainy.js'
import { NounType } from '../../src/types/graphTypes.js'
import { GenerationStore } from '../../src/db/generationStore.js'
@ -57,6 +58,107 @@ describe('history repacking — the two-tier lifecycle', () => {
}
})
/**
* THE HOLE, END TO END the shape a real store carries.
*
* A forensic fixture was measured with generation directories 1..2503
* present except for exactly one: 1416. Its fact-log segment already showed
* the tell `seg-...1410.bfl` declaring firstGeneration 1410, lastGeneration
* 1940 (531 generations) while recording only 530 facts.
*
* Before the fix, repacking such a store folded ACROSS that hole: the batch
* skipped 1416 (no readable delta) and the sealed segment declared a range
* spanning it anyway. The next open merged that declared range back into
* committedRanges, re-admitting 1416 as committed history, and every
* subsequent auto-compaction pass then asked the packed tier for a frame
* that was never written producing, on EVERY run, the non-fatal narration
*
* Auto-compaction of generational history failed (non-fatal): generation
* N is inside sealed segment seg-....bgs's declared range but has no frame
* packed history is damaged
*
* This pin removes a generation directory to make the same hole, then
* requires repack + reopen + compaction to complete cleanly.
*/
it('a missing generation directory does not poison the packed tier', async () => {
const dir = tempDir()
// `retention: 'all'` throughout: close() otherwise auto-compacts the
// history away, and this pin needs the cold generations still on disk so
// there is something to punch a hole in. The live window stays at its
// production default for the build phase, so nothing folds yet.
const archival = async (): Promise<Brainy> => {
const b = new Brainy({
requireSubtype: false,
storage: { type: 'filesystem', path: dir },
embeddingFunction: stub,
retention: 'all'
})
await b.init()
return b
}
const brain = await archival()
const id = await brain.add({
data: 'holed-entity',
type: NounType.Document,
metadata: { v: 0 }
})
// One flush per update: single-op writes coalesce inside a flush window,
// so a history deep enough to have a middle needs the windows separated.
for (let v = 1; v <= 12; v++) {
await brain.update({ id, metadata: { v } })
await brain.flush()
}
await brain.close()
// Punch the hole: delete ONE generation directory in the middle of the
// cold range, exactly as the real store presents it.
const genRoot = path.join(dir, '_generations')
const numeric = fs
.readdirSync(genRoot, { withFileTypes: true })
.filter((e) => e.isDirectory() && /^\d+$/.test(e.name))
.map((e) => Number(e.name))
.sort((a, b) => a - b)
expect(numeric.length).toBeGreaterThan(6)
const victim = numeric[Math.floor(numeric.length / 2)]
fs.rmSync(path.join(genRoot, String(victim)), { recursive: true, force: true })
// Now shrink the live window and reopen. close() repacks automatically
// (brainy.ts phase 0b), so this is the production sequence exactly: a
// store with a hole in its history gets folded by ordinary housekeeping,
// with nobody asking for it.
;(GenerationStore as any).REPACK_LIVE_WINDOW = 3
const reopened = await archival()
const result = await reopened.repackHistory()
expect(result.foldedGenerations).toBeGreaterThan(0)
const segDir = path.join(dir, SEGMENTS_PREFIX)
const manifestPath = ['manifest.json', 'manifest.json.gz']
.map((f) => path.join(segDir, f))
.find((p) => fs.existsSync(p))!
const raw = manifestPath.endsWith('.gz')
? zlib.gunzipSync(fs.readFileSync(manifestPath)).toString('utf8')
: fs.readFileSync(manifestPath, 'utf8')
const manifest = JSON.parse(raw) as {
segments: Array<{ firstGeneration: number; lastGeneration: number; frames: number }>
}
// THE LAW: every sealed segment declares exactly as many generations as it
// holds frames, and none of them spans the victim.
for (const s of manifest.segments) {
expect(s.lastGeneration - s.firstGeneration + 1).toBe(s.frames)
expect(victim >= s.firstGeneration && victim <= s.lastGeneration).toBe(false)
}
await reopened.close()
// And the pass that used to fail on every run now completes: reopen (which
// re-seeds committedRanges from the packed tier) then compact history.
const third = await openBrain(dir)
await expect(third.compactHistory({ maxGenerations: 2 })).resolves.toBeDefined()
await third.close()
})
it('repack preserves every historical read across cold reopen; folded dirs are gone', async () => {
;(GenerationStore as any).REPACK_LIVE_WINDOW = 3
const dir = tempDir()

View file

@ -147,4 +147,119 @@ describe('db/GenerationSegmentStore — the D1+D3 packed tier', () => {
await expect(store.fold([gen(4), gen(4)])).rejects.toThrow(/strictly ascending/)
await expect(store.fold([])).rejects.toThrow(/at least one generation/)
})
// ==========================================================================
// THE DENSITY LAW
// ==========================================================================
//
// A sealed segment declares a CONTIGUOUS range and every reader treats that
// range as containment. Folding a sparse batch therefore makes the segment
// claim generations it does not hold — and because `open()` merges declared
// ranges back into committedRanges, the hole is re-admitted as committed
// history and every later maintenance pass fails asking for a frame that was
// never written. That is the "generation N is inside sealed segment
// seg-....bgs's declared range but has no frame — packed history is damaged"
// narration seen on every run of the affected stores.
it('fold REFUSES a batch with a hole — a dense range may not be declared over sparse input', async () => {
await expect(store.fold([gen(1), gen(2), gen(4)])).rejects.toThrow(
/not contiguous: 2 → 4 skips 1 generation/
)
// The refusal loses nothing: no segment was sealed, so the generations
// stay in the live tier and the next pass folds them correctly.
expect(store.segments()).toHaveLength(0)
expect(store.hasGeneration(1)).toBe(false)
})
it('a wider gap names how many generations it would have swallowed', async () => {
await expect(store.fold([gen(10), gen(20)])).rejects.toThrow(
/not contiguous: 10 → 20 skips 9 generation\(s\)/
)
})
it('two contiguous runs folded separately declare honest ranges', async () => {
// What the caller now does instead of folding across the gap.
const a = await store.fold([gen(1), gen(2), gen(3)])
const b = await store.fold([gen(7), gen(8)])
expect(a).toMatchObject({ firstGeneration: 1, lastGeneration: 3, frames: 3 })
expect(b).toMatchObject({ firstGeneration: 7, lastGeneration: 8, frames: 2 })
// The gap is honestly outside the packed tier.
for (const g of [4, 5, 6]) expect(store.hasGeneration(g)).toBe(false)
for (const g of [1, 2, 3, 7, 8]) expect(store.hasGeneration(g)).toBe(true)
expect(await store.actualRanges()).toEqual([
[1, 3],
[7, 8]
])
})
it('actualRanges() is exact and I/O-free for dense segments', async () => {
await store.fold([gen(1), gen(2)])
await store.fold([gen(3), gen(4)])
// Adjacent dense segments each contribute their declared range.
expect(await store.actualRanges()).toEqual([
[1, 2],
[3, 4]
])
})
// ---- pre-existing damage: a store sealed by the old writer ----------------
/**
* Seal a SPARSE segment the way the pre-fix writer did: write the bytes and
* sidecar for a contiguous run, then rewrite the manifest so the segment
* declares a wider range than the frames it holds. This reproduces on disk
* exactly what the affected stores carry, without needing the old code.
*/
const sealSparseSegment = async (): Promise<void> => {
await store.fold([gen(1), gen(2), gen(3)])
const manifest = (await storage.readRawObject(`${SEGMENTS_PREFIX}/manifest.json`)) as any
// Declare 1..5 while holding frames for 1..3 — generations 4 and 5 become
// holes inside a sealed range.
manifest.segments[0].lastGeneration = 5
await storage.writeRawObject(`${SEGMENTS_PREFIX}/manifest.json`, manifest)
}
it('a pre-existing sparse segment reports its holes as UNPACKED, not as damage', async () => {
await sealSparseSegment()
const reopened = new GenerationSegmentStore(storage as any)
await reopened.open()
// The frames it really holds still serve, byte-faithfully.
expect((await reopened.readDelta(2))?.timestamp).toBe(1_700_000_000_002)
expect(await reopened.readRecords(3)).toHaveLength(2)
// The holes answer "not packed" instead of throwing. This is the fix for
// the wedge: the old reader threw here on EVERY maintenance pass.
expect(await reopened.readDelta(4)).toBeNull()
expect(await reopened.readRecords(5)).toBeNull()
})
it('actualRanges() excludes the holes so they are never re-admitted as committed', async () => {
await sealSparseSegment()
const reopened = new GenerationSegmentStore(storage as any)
await reopened.open()
// Declared 1..5; actually holds 1..3. The store seeds committedRanges from
// THIS, so generations 4 and 5 never become committed history again.
expect(await reopened.actualRanges()).toEqual([[1, 3]])
})
it('a DENSE segment missing a frame is still loud damage', async () => {
// The other side of the branch: when the manifest claims a complete span,
// a missing frame means the manifest and sidecar disagree — real damage,
// and it must not be quietly downgraded to "unpacked".
await store.fold([gen(1), gen(2), gen(3)])
const idxPath = `${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.idx`
const raw = (await storage.readRawBytes(idxPath))!
const { decode, encode } = await import('@msgpack/msgpack')
const idx = decode(raw) as any
// Drop generation 2's entry while the manifest still declares 3 frames.
idx.generations = idx.generations.filter(([g]: [number]) => g !== 2)
await storage.writeRawBytes(idxPath, encode(idx))
const reopened = new GenerationSegmentStore(storage as any)
await reopened.open()
await expect(reopened.readDelta(2)).rejects.toThrow(
/manifest and the sidecar disagree; packed history is damaged/
)
})
})