The packed tier goes live inside GenerationStore:
- Two-tier reads: getDelta / readBeforeImage / readGenerationRecords
fall through live-tier → sealed segments (live-tier-wins: a crash
mid-fold leaves a duplicate representation, never a gap). Cold-open
seeds committedRanges from the segment manifest via interval merge —
packed generations resolve without their directories existing.
- repackHistory({timeBudgetMs, batchGenerations}): folds cold
generations (older than the newest 1024) oldest-first into sealed
segments, deleting per-generation directories only after segment +
manifest are durable. Public API + automatic time-bounded pass at
close() (before compaction, so reclaim can drop whole segments);
re-representation only — the sole history transform under the
archival profile. Early stop = consistent prefix, next pass resumes.
- compact(): packed generations reclaim logically in the loop and
physically at whole-segment boundaries via dropSegmentsBelow (the
frozen partial-segments-wait rule).
- generationDigest(g) (D8): deterministic content digest through g —
sealed-segment checksum chain + live-tier delta hashes; O(segments +
live window); RangeError out of range, GenerationCompactedError
below the horizon (a gate can never silently pin reclaimed history).
Four end-to-end pins: asOf answers byte-identical across fold + cold
reopen with folded dirs physically gone; repack+reclaim composition;
digest reopen-stability/divergence/loud-horizon; budget no-op+resume.
First stage of the co-frozen D1+D3+repacking unit: the format core,
self-contained under _generations/segments/.
- seg-<firstGen,20pad>.bgs: append-once packs of consecutive
generations (magic BGS1; frame = u32 len + u32 crc32c + msgpack
[generation, timestamp, delta, records, flags]; flags reserves
compressed-payload evolution without a format break). Sealed
segments are immutable — fold refuses overlap with sealed ranges.
- seg-<firstGen>.idx: DERIVED sidecar (per-generation frame offsets +
per-id generation postings + checksums); lost/corrupt sidecars
rebuild from their segment loudly; a damaged segment (frame CRC
mismatch) fails loudly, never serves wrong bytes.
- manifest.json: the one discovery path — open() reads it and never
lists the packed backlog (the scan-wedge class's cure); refuses a
newer manifest version rather than serving partial history.
- D3 semantics: dropSegmentsBelow reclaims WHOLE segments at
boundaries only and bumps compactedBelow durably; archival-profile
enforcement stays with the caller per the co-freeze.
- D8 rider: digestThroughPacked(g) — deterministic crc32c chain over
sealed-segment checksums (+ frame-level prefix mid-segment),
O(segments), reopen-stable.
Also fixes a cross-adapter contract bug the suite caught: memory
storage's deleteObjectFromPath ignored the raw-bytes store, so
deleteRawObject on a raw-bytes path (fact-log or segment files)
silently no-op'd — deletes now match filesystem unlink semantics.
Six pins. Wiring into GenerationStore (two-tier reads, the repacker,
cold-open manifest path) lands with the rest of the unit before its
release; cortex's fact-record/stamp shapes reconcile the sidecar
keying when they post.
Stage-2 D1 contract item (co-frozen): a fact scan may be slow, never
silent. batches() now races its FIRST pull against
SCANFACTS_FIRST_BATCH_MS (10s, exported; test-overridable) — a wedged
or unreadably slow store produces a loud abort naming the contract
instead of a consumer hanging indistinguishably from progress (the
production shape: a heal against a generations-backlogged brain
wedged silently on the first segment read).
Only the first pull is raced: the bound is time-to-first-batch (proof
the producer is alive), not per-batch pacing, and it runs only while
a pull is pending — consumer think-time between pulls never counts
against the producer (pinned).
Three pins: wedged-store loud failure within the bound, healthy scan
untouched end-to-end, slow-consumer immunity.
flush() is durability work: it must cost what the current window's
deltas cost, never what the history backlog costs. Under adaptive
retention the byte budget derives from free memory, so bulk-load
pressure shrank the budget exactly at peak write volume and flush paid
actual reclaim inline — a production deployment measured single writes
blocked 25-191s behind reclaim-on-flush.
- flush() no longer calls autoCompactHistory(); close() is THE
auto-compaction site (already ran there; now alone).
- Every auto pass is time-bounded (CLOSE_COMPACTION_BUDGET_MS = 5s):
reclamation is oldest-first, so an early stop is a consistent prefix
and the next pass resumes. Explicit compactHistory() gains an
optional timeBudgetMs for caller-chosen maintenance windows.
- Documented trade stated where operators read: a long-lived writer
that never closes accumulates history until its next explicit
compactHistory() — predictable writes, explicit maintenance.
Pins: flush-never-reclaims + close-reclaims-durably (db-mvcc), bounded
pass stops-then-resumes as a consistent prefix (generationStore unit).
Under default adaptive retention, every flush() recomputed total history
bytes by walking EVERY committed generation's delta — O(all generations)
with disk re-reads past the 4096-entry delta-cache bound. On a production
brain with 70,000+ accumulated generations this turned every write into a
full-tail scan (60-100s writes, escalating with history growth), even
though the free-RAM budget never tripped and nothing was ever reclaimed
(SELF-GENERATIONS-GROWTH).
historyBytes() now maintains a running total: seeded by one walk on first
use, then updated incrementally at both commit paths (+bytes) and the
compaction reclaim loop (−bytes), dropped on reopenAfterRestore. The
adaptive retention check on every flush is O(1). Invariant regression-
pinned: running total ≡ fresh walk through transact commits, single-op
group commits, and compaction.
New brain.historyStats() (exported HistoryStats): read-only generation
count / bytes / generation+timestamp range / horizon / retention mode /
effective budget — the one-call per-brain fleet audit for retention
exposure.
- The transact apply budget scales with the batch — max(30s, opCount x 2s) —
or is exactly the caller's new TransactOptions.timeoutMs. A flat 30s cap
silently limited honest bulk work to ~15 operations on network-attached
storage (~2s/op measured in a production import) while looking generous for
small batches. Internal batch paths (removeMany chunks) get the same
scaling via the shared transactTimeoutBudget helper.
- TransactionTimeoutError now reports the operation it stopped at as i/N with
the operation's name, elapsed vs budgeted time, and states the batch rolled
back atomically and is retryable; context carries the fields
programmatically.
- The transact operational envelope is documented (optimistic-concurrency
guide): budget math, chunking with ifAbsent idempotency, when to reach for
addMany vs transact, and the precompute pattern (embedBatch + per-op
vector) that keeps inference out of the commit path.
- Embedding claims made honest per an end-to-end probe of the built dist:
batch and single embedding paths produce bit-identical vectors and a
vector-supplied add is fully searchable, but batch throughput on the
default WASM engine measures comparable to sequential (~160ms/text) — the
5-10x speedup claim in the addMany JSDoc is replaced with the measured
reality and the actual win (inference outside the budgeted write path).
- New public guide (guides/external-backups): the sparse-mmap reality
for external backup tooling — why a brain directory can show 100+ GB
apparent size on a small disk, which files are sparse, tar czSf /
rsync --sparse / cp --sparse=always, live-store caveats, and what
persist()/restore() already handle natively.
- New public concept (concepts/generation-fact-log): what facts are
(after-image commit records, body-less tombstones), the crash-safety
model (checksummed frames, reconcile-to-committed, absent = never
committed), the scanFacts()/factSegmentPaths() surfaces with the
telemetry shape, family stamps + open-time coherence, and the
storage capability seam for plugin authors.
- Cross-link from the snapshots guide; sparse notes added to the
public persist()/restore() JSDoc (the operator-facing sites).
- storage.committedGeneration(): the committed watermark exposed on the
fact-scan capability, so a provider compares its stamp's
sourceGeneration against the store's truth without parsing the
private manifest format (the capability source now carries both the
live fact log and the committed closure).
- Pinned contract tests: fsync-before-ack (transact passes today; the
single-op path is pinned via it.fails as the documented target —
group commit must become LATENCY batching, ack waiting on the shared
fsync, never durability skipping; the pin flips red when that lands
so there is no cliff to discover) and scan-stability-under-rotation
(a scan snapshot yields exactly its facts — no gaps, duplicates, or
bleed-in — while segments rotate beneath it; the reclaim-during-scan
variant lands with fact compaction, which does not exist yet).
- FactLog accepts a rotateBytes option so tests exercise real rotation.
Three additive surfaces for native index providers, which hold only
`storage` and must never construct their own fact-log reader (the log's
open path is writer-side — it reconciles by truncating/rewriting):
- Fact-scan capability on the storage adapter (the getBinaryBlobPath
pattern): the host brain wires a closure over its LIVE fact log at
init; providers call storage.scanFacts()/factLogHeadGeneration()/
factSegmentPaths() and fall back to the enumeration walk on null.
Restore/reopen swaps the underlying log transparently.
- The family-stamp trio (readFamilyStamp/writeFamilyStamp/
verifyFamilyStamp + types) exported from @soulcraft/brainy/internals,
so 'one verifier' is literally one function shared with the native
side, never two synchronized copies.
- Rollup invariants widened to number|string: content fingerprints
(e.g. a per-tree SHA-256) are valid invariant values; strict equality
either way, and a type mismatch reads as incoherence, never a pass.
The canonical entity tree now carries a FAMILY STAMP
(_system/family-stamps/entity-tree.json, JSON — forensics stay
terminal-readable): which committed generation the tree reflects
(sourceGeneration) plus the rollup invariants (entity/relationship
counts) that verify a millions-of-files projection whole where per-file
checks cannot scale. Written at flush/close boundaries; open-time
coherence is a COMPARISON, not a walk:
- coherent / absent (legacy store) → silent
- behind → benign for the tree (it is written BY the commit; only the
stamp is stale after a crash between commit and flush) — refreshes at
the next flush
- incoherent (counts diverged at equal generation, or a stamp AHEAD of
the log head) → loud, names the failing invariant; repairIndex()'s
unconditional recount heals and RE-STAMPS so repair leaves a coherent
stamp behind
- a fault reading the stamp is UNVERIFIABLE — never conflated with
absence
One verifier (verifyFamilyStamp, exported) reads both member modes:
enumerated (exact byte size per member, bounded families) and rollup
(invariants, unbounded families). New exports: readFamilyStamp,
verifyFamilyStamp, ENTITY_TREE_STAMP_PATH, FamilyStamp, StampMembers,
StampVerdict.
Every committed generation now also appends a FACT — an after-image
commit record (what each touched entity/relationship became, or a
body-less tombstone for a removal) — to an append-only, crc32c-framed
segment log under _generations/facts/. The before-image history and the
canonical tree remain authoritative; the fact log gives consumers ONE
sequential, self-verifying stream (index heals, incremental replays)
in place of a per-entity directory walk.
- Wire format: positional msgpack facts [generation, timestamp, ops,
meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone];
32-byte segment header (magic, formatVersion, firstGeneration,
zeroed+verified reserved); length+crc32c frame per fact; zero-padded
segment names so lexicographic order == generation order; JSON
manifest with an atomic rename flip, manifest-first rotation.
- Commit protocol: facts append+fsync BEFORE the commit point inside
the existing durability window, so a crash can only leave the log
AHEAD of committed truth — open() truncates back (torn tails detected
by CRC). Absent generation = never committed; a scan can never see an
uncommitted fact. transact() facts are durable-on-return; single-op
facts ride the group-commit flush exactly like buffered history. A
fact-append failure fails the write, loudly — a silent gap would be a
lie a later replay discovers.
- New public surface: brain.scanFacts() (sequential batches with heal
telemetry: head/segments/approx up front, per-batch generation range
+ bytes + segment id, loud abort on gaps, summary cross-check) and
brain.factSegmentPaths() (immutable sealed segments for zero-copy
consumers; the mutable tail excluded). Exported types CommitFact,
FactOp, FactScanBatch, FactScanHandle.
- Storage: optional binary raw-byte primitives (appendRawBytes,
readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter —
feature-detected; filesystem + memory adapters implement them; an
adapter without them hosts no fact log. Fact segments are byte-copied
(never hard-linked) into snapshots. The _generations/facts/ namespace
is registered as a protected family (rebuildable: false): no sweeper
or GC may delete under it.
- New crc32c (Castagnoli) utility with RFC known-answer tests.
The async group-commit flush that persists single-op generation history
swallowed persist failures as a bare warn: writes kept succeeding while their
before-images piled up in memory, never durable and unbounded, with no signal.
The generation store now accounts for every failed flush at one place
(flushPendingSingleOps), tolerates a transient blip (retry with capped
exponential backoff), and after PENDING_FLUSH_FAILURE_THRESHOLD consecutive
failures LATCHES a durability failure and refuses further single-op and transact
writes with a typed, exported PendingFlushDurabilityError — rather than promise a
durability it cannot deliver. Live canonical data is untouched; only the
immutable history is stuck. The latch self-heals: the moment a flush succeeds
(a retry, or an explicit flush()/close()) it lifts and writes resume.
Loud errors, never quiet losses.
When a transaction failed and its rollback ALSO failed to undo a canonical
write (retries exhausted), the old path logged, continued, threw
TransactionRollbackError, and set state='rolled_back' — while the record it
could not undo stayed durable on disk. The caller got an error implying the
write was undone; a read-back showed the record. A failed rollback had no
truthful response (the post-commit response lie).
Give a failed rollback a two-branch honest contract, decided by observation:
both commit paths already hold byte-identical before-images, so after a failed
rollback the store compares current canonical state to them and classifies each
touched record as reconciled, an additive orphan (present when it should be
gone), or a restorative loss (gone/wrong when it should have been restored).
- Adopt-forward: a single-op write whose only damage is a durably-present
orphan — the record the caller asked for — is committed forward (its
generation buffered) and returns success with a loud warning that the derived
index may be incomplete for that id until the next rebuild/repairIndex(). No
error, no double-write; the record is durable and get-able immediately.
- Fail loud + quarantine: a multi-op batch, or ANY restorative loss, throws the
new StoreInconsistentError naming every unreconciled record and its
disposition, and puts the brain into write-quarantine (reads keep working,
writes refused via assertWritable) until repairIndex() reconciles the derived
indexes against canonical and lifts it. The counter is not advanced, and
Transaction.rollback now reports state 'inconsistent' instead of the
'rolled_back' lie when any undo failed.
New: StoreInconsistentError + UnreconciledRecord exported from the root;
repairIndex() forces a rebuild and clears the quarantine. Regression
tests/integration/rollback-trapdoor.test.ts injects index-add-throws +
canonical-delete-undo-fails and pins adopt-forward (durable, get-able, not
quarantined), fail-loud (StoreInconsistentError + quarantine + reads work +
repairIndex lifts it), and the error's record naming.
A committed transact() reported success while its canonical entity writes were
still only in the OS page cache (tmp+rename, not fsync'd), even though the
generation counter and manifest were fsync'd. A hard kill in that window could
leave the durable counter ahead of the persisted entity bytes — phantom
progress for any generation-based consumer resuming from the counter. Reported
from a downstream migration's crash-lifecycle forensics.
Add an optional transaction durability barrier to GenerationStorage
(beginWriteBarrier / flushWriteBarrier). FileSystemStorage implements it:
writeObjectToPath records each successful canonical write and
deleteObjectFromPath records each delete's parent dir, between begin and flush;
flush fsyncs every recorded write (contents + rename dir entries, via
syncRawObjects) and the parent dir of every delete. commitTransaction opens the
barrier before running the planned operations and flushes it after, BEFORE
persisting the counter and manifest — so the batch's entire canonical footprint
is durable before the generation stamp advances. The barrier is optional: the
generation store calls it through optional chaining, so in-memory and
durable-per-call (cloud object-PUT) adapters treat it as a no-op.
The single-op group-commit path is unchanged (deferred durability is the
Model-B design that avoids a 3-5x per-write fsync regression), but its
durability contract is now documented explicitly on commitSingleOp: transact =
durable on return; single-op = durable at the next flush()/close(), with the
counter buffered alongside the data so a crash loses both together (never a
torn counter-ahead-of-state store).
Regression (tests/integration/transact-durability-barrier.test.ts): the entity
writes fsync in an earlier syncRawObjects batch than the manifest for single-op
and multi-op (add+relate) transactions; a precommit-rejected batch opens no
barrier and advances nothing; MemoryStorage exposes no barrier (optional-chain
no-op).
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.
Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:
- The commit seam counts one history reference per persisted before-image
record carrying a content hash (commitTransaction staging and the
group-commit flush), recorded BEFORE the record-set persists and carried
in the generation delta (blobHashes — always present on new deltas, so
compaction only falls back to reading records for pre-contract
generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
release; overwrite finally releases the superseded hash — cancelling the
dedup increment on same-content rewrites and closing the leak), and only
AFTER the canonical mutation commits, so a failed delete can never leave a
live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
generation's record-set it releases that set's references and physically
reclaims any hash at zero live AND zero history references. Pins are
exempt automatically. Crash ordering is over-count-only in every path
(record before persist, release after delete), so a crash can leak until
the scrub recounts but can never reclaim bytes a retained generation
needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
a one-time marker-gated backfill on open, failing into leak-safe mode
(reclamation disabled) rather than guessing.
On top of the protected history, the temporal API the generational model
always implied:
- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
materialized from the history (pinned view released so compaction is
never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
search and the data field previously served the FIRST version's text
forever (the stale-field defect a consumer's incident recovery depended
on by luck).
Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
N concurrent update({ ifRev }) calls carrying the same expected revision all
fulfilled (zero RevisionConflictErrors, last-writer-wins) — the check ran
before the commit mutex, so interleaved callers all passed it before any apply
landed. Sequential calls conflicted correctly, which hid the race. A production
cutover rehearsal caught it: 8 parallel ifRev-guarded ledger writes all
"succeeded" and 7 were silently lost. Advisory locks built on exactly-one-winner
semantics could hand the same lock to two workers.
Fix: conditional commit. commitSingleOp and commitTransaction accept an
optional precommit(beforeImages) precondition, invoked under the commit mutex
against the just-read authoritative before-images and before anything is staged
or applied — the per-record analogue of ifAtGeneration, which always ran there.
A throw aborts the commit atomically (generation reservation returned, zero
staging I/O in the transaction path). The store stays domain-ignorant; update()
and transact() supply the ifRev predicate:
- update(): the planning-time check remains as a fast-fail (avoids embedding
cost on an obviously stale expectation); the authoritative check re-verifies
the before-image's _rev and re-stamps the update's _rev from it, so the
counter is monotonic even for concurrent non-CAS updates (N plain updates
advance _rev by N). CAS against a concurrently-removed entity now throws
EntityNotFoundError instead of silently resurrecting it; plain updates keep
their last-writer-wins re-create semantics.
- transact(): per-op ifRev expectations registered during planning
(PlannedTransact.casUpdates) are re-verified in the batch's precommit; any
conflict rejects the whole batch before staging. Multiple updates to one
entity sequence through a running rev; ids the batch itself adds
(PlannedTransact.createdNouns — add resets the rev baseline even on
overwrite) keep their exact plan-time sequencing.
Regression suite (tests/integration/ifrev-concurrent-cas.test.ts): 8 parallel
same-rev updates → exactly 1 winner + 7 conflicts; same for transact batches;
_rev monotonicity; the documented read→CAS→retry ledger loop converging exactly
(8 workers, 8 decrements, 0 lost); sequential behavior unchanged; forward-ref
add+update in one batch unchanged.
The MVCC time-travel layer kept nounChains/verbChains as unbounded
Map<id, number[]> — one resident chain per id ever touched across retained
history (O(N) RAM, defeating billion-scale time travel). Replace with a hot-tail
window + bounded cold LRU + a mutex-free bulk resolver, keeping resolveAt exactly
correct.
The most-recent W generations stay resident as full chains (the common recent-pin
read is O(log), zero scan); deeper pins reconstruct one id's chain on demand into
an L-bounded LRU. Reconstruction is LOCK-LIGHT — it holds no commit mutex, so a
historical read never stalls writers; correctness holds because a live pin's
answer is always > minPinnedGeneration, which compaction can never reclaim, and a
concurrently-reclaimed gen below that is skipped. materializeAtGeneration routes
through a new mutex-free resolveManyAt (one forward pass, O(R + |ids|)) — without
it a deep-pin materialize both regresses to O(N*R) and deadlocks on snapshotWith's
mutex. The cold cache is invalidated on re-touch to stay coherent. Resident RAM is
O(W*d + L), independent of N. 11-case test (oracle-vs-bruteforce, held-Db across
eviction + concurrent compaction, no-deadlock, write-path I/O-free); unit 1735 +
integration 613 (db-mvcc/db-temporal/db-asof green).
committedGens was a number[] with one element per committed generation. Every
single-op write reserves a distinct generation, so an insert-built 1B corpus
held a ~1B-element array resident regardless of provider — the MVCC sibling of
the storage-cache gate.
Replace it with a sorted disjoint interval set (committedRanges). With no
compaction gaps the whole ledger collapses to a single [start,end] pair, so
resident size is O(number-of-gaps) not O(writes). reservedGens becomes a
generator; point-resolution binary-searches the ranges; compaction trims the
oldest prefix with a defensive prefix-invariant guard. Behaviour is identical
across asOf/diff/since/history/transactionLog — unit 1718 + integration 607 +
the focused MVCC suite green.
8.0 keeps the canonical operators (eq/ne/gt/gte/lt/lte) and their clean
long-form aliases (equals/notEquals/greaterThan/greaterThanOrEqual/lessThan/
lessThanOrEqual), and drops the four redundant deprecated spellings:
is → eq, isNot → ne, greaterEqual → gte, lessEqual → lte
Removed from every evaluator (metadataIndex criteria + range switches,
metadataFilter, the db whereMatcher egress path) and from the
BrainyFieldOperators type, the unsupported-operator error message, the docs
(QUERY_OPERATORS / api README / VFS projection + semantic guides), and the
whereMatcher alias tests. Also migrated Brainy's own internal use — the VFS
TemporalProjection queried with greaterEqual/lessEqual, which would have
silently broken — to gte/lte.
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- find() search-mode: collapsed the two overlapping options to one canonical
`searchMode: SearchMode` (the redundant `mode` alias was silently ignored on
the primary find() path while honored on the historical path — a footgun) and
removed the unwired `explain?` FindParams field (never read by find()).
- Documentation accuracy on the public type surface: entity ids documented as
UUID v7 (auto) / v5 (natural key), not v4; dropped the stale "Brainy 3.0"
banners and "backward compatibility" hedging on the fresh-8.0 Result type;
documented the via/type alias; removed the dead GraphConstraints.bidirectional;
fixed the no-op `EmbeddedGraphVerb = Omit<GraphVerb,'source'>` to omit the real
sourceId; corrected the GraphIndexProvider.isInitialized JSDoc; documented
removeMany's per-chunk generation granularity; JSDoc'd the public VFS surface.
- Honest perf comments in source: removed unmeasured billion-scale figures from
the LSMTree / graphAdjacencyIndex headers (the JS fallback is in-memory after
open; native is the scale path).
- Robustness: getVerbMetadata propagates read errors symmetrically with
getNounMetadata; the find() egress integrity-guard keeps a row when the JS
matcher doesn't implement an operator the provider already matched on; hardened
the boundary-no-native CI guard to catch side-effect imports + re-exports.
- Tests: de-theatricalized a relateMany test to assert the real contract; fixed
a stale Model-B header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A filtered semantic read at a historical generation (db.asOf(g).find({ query, where }))
no longer unconditionally rebuilds an ephemeral JS-HNSW over every at-g vector
(O(n@G), the OOM ceiling the 2026-06-24 scaling audit flagged). When the vector
index is a VersionedIndexProvider that advertises isGenerationVisible(g), Brainy:
1. resolves the at-g metadata∩graph universe from the record-overlay path (no
materialization — it's the metadata-only historical find),
2. routes the vector leg to the provider with { allowedIds, generation }, and
3. composes the at-g entities ranked by the provider's at-g vector distance.
Without a versioned provider (the JS index, or a native one that refuses the
generation) it falls through to the existing materialization — unchanged.
- Seam (A): VectorIndexProvider.search gains an optional `generation?: bigint`
("omitted = now"), the vector mirror of the graph provider's trailing-gen + the
#46 allowedIds field. JsHnswVectorIndex accepts and ignores it (no per-gen
segments → refuse/fall-back posture; Brainy never routes a historical read there).
- Gate: Db.find tries the native at-gen vector path before materialize(); host
exposes canServeVectorAtGeneration + vectorSearchAtGeneration.
- generation is Brainy's u64 commit counter — the same value handed to the graph
index on writes (graphWriteGeneration), so it maps 1:1 to the native side.
Decided lockstep with the cor team (handoff #35 thread: (A) + vector-only). The
gen-g candidate-vector supply for cor's exact-rerank (part-3) is a separate,
non-blocking seam still being confirmed; the honesty guard keeps this path inactive
(falls through to materialize) until cor's native at-gen rerank is live.
Tested: provider routing + at-gen universe correctness (excludes future-born,
applies the filter) + page window via a mock versioned provider; seam-ignore on the
JS index; 38 db-mvcc/db-temporal historical-read tests green (materialize fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
Historical reads (asOf/get) scanned the global committedGens list linearly,
making them O(database-age): a read of an unchanged entity at an old pin scaled
~12x for 10x history depth. Add per-id inverted history chains (nounChains/
verbChains) so resolveAt binary-searches the id's own generation chain instead —
O(log) and flat with depth. Chains build lazily under the commit mutex,
maintain incrementally on commit, and invalidate on compaction.
Also bound deltaCache (LRU cap 4096; getDelta re-reads evicted deltas) so a
long-lived high-write process's heap is O(cap) not O(generations) on the
disk-backed path, and binary-search commitTimestampAtOrBefore (O(log)).
Verified: 373 unit tests green; new tests/unit/db/generation-chain.test.ts
covers chain resolution, eviction re-read, and chain rebuild after compaction.
related({ from }) / related({ to }) each return one direction; related({ node })
unions both (every edge where the node is source OR target), deduped, in a single
O(degree) call — the indexed equivalent of merging two related() calls by hand.
The 'all edges on a noun' query the graph-viz path needs.
- RelatedParams.node: mutually exclusive with from/to (throws if combined);
composes with type/subtype/visibility filters; natural-key ids resolved.
- Implemented as a parallel out-edge (sourceId) + in-edge (targetId) fetch through
the O(degree) graph-index fast paths, merged + deduped by id (a self-loop appears
once), sorted for a stable page. Full incidence is the intended O(degree) cost;
deep offset paging of a very-high-degree node is best-effort (use the native
edgesForNode / a graph cursor for exhaustive paging).
- db.related() parity: node resolved + honored in relationMatchesParams (matches an
edge incident in either direction) for the historical/overlay paths.
Test: related-node-bidirectional.test.ts — both directions, equals from∪to, self-loop
dedupe, type-filter composition, default-hides-internal, node+from/to throws.
Three additive ergonomics from the API-simplification audit (no behavior change
to existing call sites):
- AddParams.upsert: create-or-update in one call. With a custom id, an existing
entity is MERGED via the update path (merges metadata, re-embeds changed data,
bumps _rev, PRESERVES createdAt) instead of the destructive full overwrite a
plain add() does. Mutually exclusive with ifAbsent (throws if both set);
ignored when no id is supplied. Wired into add(), addMany (per-item flag
propagation), and the transact add op (routes to planTxUpdate). Kills the
get()-then-add() round-trip for idempotent writes.
- FindParams.includeVectors: mirror of GetOptions.includeVectors — find() returns
stored vectors when set; default stays empty (the perf contract is preserved).
Honored on both the query and metadata-only where paths, and in db.find().
- removeMany adaptive chunking: replaced the hardcoded chunkSize=10 with
params.chunkSize ?? storageConfig.maxBatchSize, matching addMany/relateMany —
one storage-adaptive batch policy across all *Many methods (no thrash on
high-latency backends).
Tests: tests/unit/brainy/upsert.test.ts (insert/merge/createdAt-preserved/
re-embed/ifAbsent-conflict/no-id/addMany/transact), find-include-vectors.test.ts
(true/default/where-path), batch-operations.test.ts (removeMany >10 items).
- id-normalization (#18): `brain.newId()` (UUID v7) + v7 default ids; non-UUID string ids are
transparently normalized to a stable UUID v5 on creation AND every lookup — get/update/remove,
relate(from,to), related, find({connected}), getMany, removeMany, the transact op-handler, and
db.with() overlays — with the caller's original key preserved under `_originalId`. The engine only
ever sees a UUID; real UUIDs pass through untouched. universal/uuid.ts gains v5/v7/isUUID +
BRAINY_ID_NAMESPACE; new utils/idNormalization.ts (coerceNewEntityId / resolveEntityId).
- aggregation min/max delete-safety: `queryAggregate` no longer leaves a stale min/max after
deleting the current extreme — min/max now track a value multiset and recompute in-memory (no
entity scan; that scan was the prior delete-then-hang a consumer reported). Other metrics were
already exact across deletes. Regression test proves resolve-after-delete + correct new min/max.
- release.sh RC-safe: explicit version arg, prerelease detection (npm `--tag rc` + GitHub
`--prerelease`), full `test:ci` gate (unit + integration), current-branch push, npm-access
verification after publish.
- native-engine references point at `@soulcraft/cor` (8.0's partner) in user-facing strings/docs.
Unit 1461/0 · integration 599/0 · tsc clean.
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.
- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
chokepoint (reachability + Date semantics identical everywhere). asOf()'s
inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
existence at both endpoints + a key-order-insensitive value compare
(new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
value === asOf(version.generation).get(id); null = removal; kind auto-detected
(throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
(contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
horizon; history TRUNCATES to it.
Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
Re-adds the portable graph round-trip on 8.0 (deleted with DataAPI), now on the
immutable Db so it composes with the generational model.
- src/db/backup.ts: BackupData v1 engine (identical wire format to the 7.32.0 line).
exportGraph() reads through a Db at its pinned generation; importGraph() applies the
whole backup as ONE atomic transact() (a single generation).
- Db.export(selector?, options?) — read at this view's generation, so
brain.asOf(g).export() is a time-travel export and brain.now().with(ops).export() a
what-if export.
- brain.export(selector?, options?) — sugar for now().export().
- brain.import() is polymorphic: a BackupData document -> graph round-trip; a
file/buffer -> existing foreign-file ingestion (dispatched on the 'brainy-backup'
tag; zero migration for ingestion callers).
- Selectors (ids/collection/connected/vfsPath/predicate/whole) + edge policy
(induced/incident/none) + includeVectors/includeContent/includeSystem; system
entities excluded by visibility. onConflict merge/replace/skip, reembed auto/never,
remapIds. DbHost gains storage (VFS blob bytes).
- Types exported from the package root: BackupData/BackupEntity/BackupRelation/
ExportSelector/ExportOptions/ImportOptions/ImportResult + isBackupData.
Tests: tests/unit/db/db-backup.test.ts (10, green) — round-trip, selectors, edges,
vectors+re-embed, merge, asOf/with composition, validation. Full unit suite green.
Follow-ups: docs (8.0 guide + RELEASES + api/README), completeness adds
(since(prior).export() delta, exportStream/importStream, validate()), includeContent
filesystem test, ids-at-asOf get() refinement, @soulcraft/formats Zod schema mirror.
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):
1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
metadata (and the transact() ops that extend them) reject a literal
reserved key as a TypeScript error while keeping generic T ergonomics
(typed bags, untyped brains, index-signature shapes, and a documented
exemption for T-declared reserved keys). Pinned by @ts-expect-error
type tests run under vitest typecheck mode on every unit run.
2. Write time — the 7.x update() remap is ported to 8.0 and extended to
every write path: add/update/relate/updateRelation, their transact()
mirrors, and db.with() overlays. User-settable fields lift to their
dedicated param (top-level wins when both are supplied — closes the 7.x
trap where update({metadata:{confidence}}) silently no-oped), and
system-managed fields drop with a one-shot warning naming the right
path. A remapped subtype satisfies subtype-pairing enforcement exactly
like a top-level one.
3. Read time — every storage combine goes through one canonical hydration
helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
and entity/relation.metadata carry ONLY custom fields on live reads,
batch reads, paginated listings, getRelations by source/target, streamed
verbs, and historical asOf() materialization alike.
Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.
Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.
Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
immutable before-images otherwise) into a fresh MemoryStorage; a final
reconciliation pass under the commit mutex makes the copy exact even
when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
graph-adjacency indexes from the records; the vector index is built by
inserting every at-G vector (the at-G HNSW graph never existed on disk,
so there is nothing to restore). Host embedder and aggregate definitions
are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
(handle cached; freed by release(), with a FinalizationRegistry backstop
that also closes leaked readers). A native VersionedIndexProvider serves
the same reads from retained segments with no rebuild.
Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.
UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.
GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.
Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
The COW version-control surface (fork, branches, checkout, commit,
getHistory/streamHistory, asOfCommit, brain.versions) is gone, along with
its subsystems: src/versioning/, the COW object store (CommitLog,
CommitObject, RefManager, TreeObject), HistoricalStorageAdapter, and the
TypeAwareHNSWIndex path it kept alive. The Db API (now/transact/asOf/
with/persist/restore) is the one versioning model in 8.0.
Survivors and replacements:
- BlobStorage survives (the VFS stores file content through it), relocated
to src/storage/blobStorage.ts with binaryDataCodec.ts; its adapter
interface is now BlobStoreAdapter, slimmed to the consumed surface
(write/read/has/delete/getMetadata + MIME-aware compression policy).
- brain.migrate() backup branches are replaced by persist-before-migrate:
MigrateOptions.backupTo persists a hard-link snapshot of the current
generation before any transform runs; MigrationResult.backupPath reports
it, and brain.restore(path) brings it back wholesale.
- CLI: cow.ts (fork/branch/checkout/history/migrate) is replaced by
snapshot.ts — snapshot <path>, restore <path>, history (tx-log),
generation.
- New public read API: brain.transactionLog({limit}) exposes the reified
tx-log (generation/timestamp/meta, newest first) that backs the CLI
history command; TxLogEntry is exported.
Tests: superseded suites deleted; fork/commit blocks excised from shared
suites; BlobStorage tests relocated + reworked against the slimmed store;
migration tests now prove the backupTo snapshot/restore round trip; new
transactionLog coverage in db-mvcc.
One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.
Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
transact() commit and once per single-operation write (storage hook), so
brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
(the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
and forces an index rebuild; refcounted pins gate compactHistory(), which
records a horizon (asOf below it throws GenerationCompactedError).
Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
(GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
{confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
generation; index-accelerated queries at historical generations throw
NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
plugin.ts: feature-detected, balanced pin/release in lockstep with Db
lifecycle, post-commit applier + replay-gap model documented.
Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.
Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
