brainy/RELEASES.md

@soulcraft/brainy — Release Notes for Consumers

This file is the quick reference for downstream sessions tracking Brainy changes. Full auto-generated changelog: CHANGELOG.md · Releases: https://github.com/soulcraftlabs/brainy/releases

How to use: Brainy is the underlying data engine for downstream applications. Read this when:

• Upgrading @soulcraft/brainy in your application
• Debugging data, query, or storage behaviour
• A new Brainy feature is available that you want to adopt

---

v8.7.1 — 2026-07-17 (writer-lock acquisition is race-proof + machine-readable through init)

Two hardenings of the multi-process writer lock (the locks/_writer.lock lease that makes a second writer on the same brain directory fail loudly):

• Lock acquisition claims atomically. The acquire path used read-then-write, leaving a window where two processes racing an absent lock could both "succeed" — and the loser kept running unlocked, silently. The claim is now an atomic create-exclusive write (O_EXCL): exactly one racer wins; the loser re-evaluates and either fails loudly with the winner's details or performs a verified stale-takeover. Bounded retries; contention beyond them fails loudly rather than degrading into a lockless open.
• BRAINY_WRITER_LOCKED survives init(). The conflict error documents a machine-readable contract (err.code, err.lockInfo with the holder's pid/host/ heartbeat), but init's error wrapping silently stripped both, leaving consumers a message to regex against. The error now passes through unwrapped.

Measured while verifying (for operators sizing audits): brain.auditGraph() at a production-consumer scale of ~2,600 relationships / 800 entities costs ~0.1 s warm and ~0.5 s cold, with exact scar counting across reopen.

v8.7.0 — 2026-07-17 (bulk-transact ergonomics: scaled budgets + timeout telemetry)

The bulk-import ergonomics release, from a consumer's measured production incident (a serial import on network-attached storage at ~2 s/op met a flat 30 s transaction budget):

• The transact apply budget now scales with the batch — max(30 s, opCount × 2 s) — or is exactly what you pass as the new TransactOptions.timeoutMs. A flat 30 s cap silently limited honest bulk work to ~15 operations on slow disks while looking generous for small batches. Internal batch paths (e.g. removeMany chunks) get the same scaling.
• TransactionTimeoutError is a diagnosis, not just a failure: it now reports the operation it stopped at as i/N with the operation's name, elapsed vs budgeted time, and states the batch rolled back atomically and is retryable. Its context carries the same fields programmatically.
• The transact envelope is documented — batch sizing, budget math, chunking with ifAbsent idempotency, and the precompute pattern (embedBatch + per-op vector) that keeps model inference out of the commit path. Guide: docs/guides/optimistic-concurrency.md.
• Note: brain.embed() / brain.embedBatch() (the precompute APIs) already ship — public, with native-provider passthrough, verified end-to-end (batch and single paths produce bit-identical vectors; a vector-supplied add is fully searchable). Honest measurement: on the default WASM engine, batch throughput ≈ sequential (~160 ms/text) — the precompute win is keeping inference out of the budgeted commit path, not raw embedding speed.

v8.6.0 — 2026-07-17 (brain.auditGraph — the graph-truth verification instrument)

A minor release adding one new public API, from the fleet's graph-trust program: a read-only audit that proves whether relationship reads return stored truth on a given brain.

• brain.auditGraph(options?) walks every canonical relationship record, queries the same read path applications use (related() / VFS readdir) with all visibility tiers included, and classifies every discrepancy into its failure family: missingFromReads (records the read path omits — a stale adjacency index), danglingEndpoints (relationships whose endpoint entity no longer exists — the historical partial-delete scar class), and readOnlyVerbIds (read-path edges with no stored record — ghosts). Design-hidden internal/system edges are counted separately so intentional hiding is never misclassified as loss. Counts are exact; example lists cap at maxExamples with an explicit truncatedExamples flag; the result is narrated loudly on incoherence. Mutates nothing — safe on a live brain.
• The operational pairing: audit → if incoherent, repairIndex() → audit again. A coherent report after the repair is the verified all-clear. Run it after any engine upgrade, restore, or migration. Guide: docs/guides/inspection.md.
• Also: getNounIds pagination now refuses an undecodable resume cursor loudly (the same contract getNouns/getVerbs gained in 8.5.2 — the third and final walk brought under it).
• Types exported: GraphAuditReport, GraphAuditDiscrepancy.

v8.5.2 — 2026-07-17 (aggregation backfill: exception-safe, generation-verified, and loud)

Hardening patch from a migration incident (a byte-copied store on new hardware; the service entered a silent full-CPU loop at boot). Four changes, all in the aggregation engine's backfill/adoption path:

• Backfill walks are exception-safe and non-destructive. A rescan now builds into a staging map and swaps in atomically on completion; a mid-walk failure drops the staging map, keeps the previous live state serving, and surfaces the storage error to the failing query. Previously the walk wiped live state before a scan that could throw, never cleared the pending flag on failure, and re-ran a full walk on every subsequent query — a silent wipe/walk/throw loop at the caller's retry rate.
• Failed walks are latched. After a walk fails, retries within a 30-second cooldown rethrow the recorded error instantly instead of re-walking — a tight caller-side retry loop now costs one loud error per query, never a full store walk per query.
• Adoption is generation-verified. Persisted aggregation state is stamped with the store's committed generation at flush; reopen adoption requires the stamp to equal the current watermark. Stale state (unclean shutdown) or over-counting state (a fact-log truncation on a copied store pulled the watermark back) triggers exactly one loud rescan — never a silent adopt. Pre-8.5.2 state on generation-aware stores rescans once after upgrade, then is stamped.
• The path narrates. Adoption decisions, no-adoptable-state outcomes, walk start/finish (entity count + duration), and walk failures all log by default; a non-advancing storage pagination cursor aborts the walk loudly instead of looping forever.

Plus three guards from a full audit of every loop in the open/init path:

• Invalid pagination cursors fail loudly. A supplied-but-undecodable resume token to getNouns/getVerbs used to silently restart the walk at offset 0 — to a while(hasMore) caller that re-serves page 1 forever (an unbounded silent CPU loop). It now throws with a clear message instead.
• The graph cold-load verb walk has a stall guard. hasMore=true with a missing or non-advancing cursor aborts loudly instead of re-reading the same page forever.
• A derived index AHEAD of the store is named at open. Brainy already surfaced a provider generation behind the committed watermark; the ahead direction (the signature of a byte-copy of a live service, or a log truncation during crash recovery) now logs a loud warning explaining what happened and that brain.repairIndex() forces a heal — instead of passing unnamed into whatever the derived index does next.

v8.5.1 — 2026-07-17 (aggregation state survives restarts + the query-cap ratchet removed)

Patch release from a production incident (aggregate/count paths taking 40–90 s on an idle box while vector search stayed fast, and every find({ limit: 5000 }) suddenly failing against an "auto-configured query limit of 1000"). Three fixes, one cosmetic:

• Aggregation state is actually adopted on reopen. The boot pattern defineAggregate() → query raced the engine's async state load: the synchronous define always won, flagged a backfill, and the first query then wiped the just-loaded persisted state and re-walked the entire store — every restart, forever. Reopening with an unchanged definition now adopts the persisted state directly (zero scans); a backfill runs only on a real definition change, a missing/failed state load, or a write that landed before adoption (exactness wins). Apps that rely on persisted definitions without re-defining at boot also no longer race a spurious "Aggregate not defined".
• Backfills are single-flight and batched. Concurrent queries on a cold aggregate used to each wipe the others' partial state and start their own full store walk — under steady query arrival the store never converged (the 40–90 s loop). Now all concurrent queries share one walk, and one walk fills every aggregate pending backfill (M aggregates ≠ M scans).
• The query-cap "learning" ratchet is removed. maxLimit is a memory-protection bound, but a hidden tuner shrank it 20 % per recorded query while the lifetime-average query time exceeded 1 s — down to a floor of 1000, below the documented 10 000 auto floor, with no practical recovery, and the resulting error blamed "available free memory" (stale basis label). The cap now comes from its construction-time basis (or your explicit maxQueryLimit / reservedQueryMemory) alone and never changes at runtime; query timing is recorded for diagnostics only.
• Cosmetic: the engine's own persistence keys (__aggregation_*, brainy:entityIdMapper) no longer log [Storage] Unknown key format at boot — they were always routed correctly; they're now recognized before the warning fires.

Operationally: if a host was bitten, upgrading and restarting is the whole fix — no repair ritual needed. Setting maxQueryLimit explicitly remains the valve that bypasses auto-detection entirely.

v8.5.0 — 2026-07-15 (provider access to the fact log + the shared stamp verifier)

Small additive follow-up to 8.4.0, from the native accelerator's first consumption pass:

• Index providers can now reach the fact log through the storage adapter — new optional capability storage.scanFacts() / storage.factLogHeadGeneration() / storage.factSegmentPaths(), wired by the host brain at init as a closure over its live log. Providers hold only storage and must never construct their own fact-log reader (the log's open path is writer-side); null means "no fact log here — use the enumeration walk."
• The family-stamp verifier is shared via @soulcraft/brainy/internals (readFamilyStamp / writeFamilyStamp / verifyFamilyStamp + types) so first-party native providers run literally the same verification function, never a synchronized copy.
• Rollup stamp invariants accept strings (number | string) — content fingerprints such as a per-tree SHA-256 are valid invariant values; a type mismatch reads as incoherence, never a pass.
• storage.committedGeneration() — the committed watermark as a capability, so a provider compares its stamp's sourceGeneration against the store's truth without parsing the private manifest format.
• Two durability/stability contracts pinned in the suite: fsync-before-ack (holds for transact() today; the single-op path is pinned as the documented future target — group commit becomes latency batching, never durability skipping) and scan-stability-under-rotation (a scan snapshot yields exactly its facts — no gaps, duplicates, or bleed-in — while segments rotate beneath it).

No behavior change for applications; all additions.

v8.4.0 — 2026-07-15 (the generation fact log — a sequential, self-verifying commit stream)

A minor release, fully backward-compatible (all additions; no behavior change for existing APIs). This is infrastructure: it changes nothing about how you query today, and lays the substrate that makes index heals and incremental catch-up sequential-read problems instead of directory walks.

• Every committed write now also appends a "fact" — an after-image commit record. Alongside the existing before-image history, each committed generation (single-op and transact() alike) appends what each touched entity/relationship became — or a body-less tombstone for a removal — to an append-only, checksummed segment log under _generations/facts/. Crash-safe by construction: a torn tail is detected and ignored; on open the log is reconciled to committed truth, so an absent generation always means "never committed." transact() facts are durable when transact() returns; single-op facts ride the same group-commit flush as their history.

• New: brain.scanFacts() — stream committed facts in commit order, in batches, with heal-grade telemetry (total scope up front; per-batch generation range, byte size, and segment; a summary cross-check at the end; loud abort on any gap — never a silent skip). brain.factSegmentPaths() hands zero-copy consumers the immutable sealed segment files directly. New exported types: CommitFact, FactOp, FactScanBatch, FactScanHandle. Facts accumulate from the first write after upgrading — pre-existing history is not retroactively converted (enumeration remains the fallback for old data).

• New: the entity-tree family stamp. At every flush/close, brainy stamps which committed generation the canonical entity files reflect plus the rollup invariants (entity/relationship counts) that verify the tree whole. At open, coherence is a comparison — a genuine divergence is loud and names the failing invariant; repairIndex() recounts from canonical and re-stamps. New exports: readFamilyStamp, verifyFamilyStamp, ENTITY_TREE_STAMP_PATH, FamilyStamp types.

• Storage adapters gain optional binary raw-byte primitives (appendRawBytes, readRawBytes, writeRawBytes, rawByteSize) — feature-detected; the filesystem and memory adapters implement them; an adapter without them simply hosts no fact log. The fact-log namespace is registered as a protected family: no sweeper or GC can delete under it.

No API breaks. 24 new tests; the full commit-path regression suite is green.

v8.3.3 — 2026-07-15 (rename moves the containment edge — no ghost in the old directory)

One production-reported fix plus a repair path, completing the delete/move hygiene arc (8.3.1 fixed deletes, 8.3.2 fixed counters, this fixes moves).

• A cross-directory vfs.rename() now MOVES the containment edge instead of accumulating one per parent. The old parent's Contains edge was never removed on a move, leaving the entity a child of both directories: readdir(oldDir) kept listing it after the move, re-creating the old path showed the same name twice, and any tree-walking consumer (sync engines, file browsers) saw the file in two places. The old edge is now removed by edge id, resolved from the graph's own adjacency — a removal never requires reading the thing being removed. Bonus fix in the same seam: a move to the root now gets its containment edge (it was previously skipped, orphaning the file out of readdir('/')).

• repairIndex() now also reconciles VFS containment (new vfs.repairContainment()): every VFS entity's containment edges are checked against its canonical metadata.path — stale old-parent ghosts and duplicate edges are removed, a missing expected edge is restored, and user knowledge-graph edges are never touched (only vfs-contains edges are candidates). Loud per repair. Stores that performed cross-directory renames under ≤8.3.2 should run brain.repairIndex() once after upgrading — the same single ritual now heals orphan directories, counters, and containment edges.

• Also ships a permanent lens-consistency regression suite (combined type+subtype vs subtype-only vs canonical ground truth, id-for-id, warm and after a cold reopen), ported from the field investigation that closed the historical lens-drop report.

No API changes beyond the new optional vfs.repairContainment() (also invoked by repairIndex()).

v8.3.2 — 2026-07-14 (honest counters — the recount + removal-without-re-reading)

Completes 8.3.1's delete-hygiene story at the counter layer, from a production proof chain reported by a downstream deployment: persisted entity totals were permanently inflated — deletes whose count decrement was silently skipped — and because paginated totalCount serves Math.max(persistedTotal, scanned), the inflated number always won and no disk cleanup could ever lower it.

• A removal's count decrement no longer requires re-reading the record being removed. The decrement was sourced from re-reading the entity's metadata inside the delete; if that read returned null (a replace race, or a ghost left by a pre-8.3.1 partial delete) the decrement was silently skipped while the paired add had counted — minting drift on every write→delete→re-create cycle. The caller's pre-delete read now rides through the whole delete path (remove()/removeMany() → the delete operation → deleteNoun/deleteVerb → deleteNounMetadata/deleteVerbMetadata, both sides symmetric): a null internal read falls back to the known prior record instead of skipping. The StorageAdapter signatures gain an optional priorMetadata parameter (additive; existing adapters unaffected).

• repairIndex() is the sanctioned counter recount — unconditional, and it actually persists. rebuildTypeCounts() previously rebuilt only the type-statistics arrays and computed the total just to log it — the persisted scalar (counts.json) survived every "rebuild" untouched, so an already-inflated brain could never be corrected. One canonical walk now rebuilds every counter rollup — scalar totals, per-type maps, and type statistics — and persists them, and repairIndex() runs it unconditionally (not only when orphan directories are found: counters can be inflated over perfectly clean shelves). Brains with delete history should run brain.repairIndex() once after upgrading; the correction survives reopen.

No API breaks (optional-parameter additions only). Regression tests cover the drift cycle, the null-read decrement fallback, and the persisted recount across reopen.

v8.3.1 — 2026-07-14 (full-removal deletes + family-scoped migration gate)

Two production-reported fixes in the write/index spine, plus an operator repair path. No API changes; all behavior changes make previously-wrong states honest.

• Deleting an entity now removes it completely — no more "ghost" leftovers on disk. A canonical noun delete removed the metadata (content) leg but left the entity's vectors.json and its <id>/ directory behind. Consequences observed in a production deployment: deleted rows were indistinguishable on disk from damage scars, enumerated counts inflated monotonically with every delete (the leftovers were counted forever), and locator-style reads hit unreadable ghost rows. remove()/removeMany()/deleteNoun/deleteVerb now remove both legs and the entity container, with a full two-leg before-image rollback inside the transaction. The generation log still holds the delete's before-image, so asOf() time-travel reconstructs deleted entities exactly as before — this is live-HEAD hygiene, not a history change. This also fixes duplicate readdir entries for re-created VFS paths at the root: with no ghost state, a delete always unposts its index rows, so a delete→recreate cycle lists the path exactly once (regression-tested across repeated cycles).

• repairIndex() prunes ghost/scar directories left by earlier versions. Stores that deleted entities under ≤8.3.0 may hold orphaned entity directories (a vector-only leg, or an empty dir). brain.repairIndex() now sweeps them: it removes only containers with no metadata content leg (never a directory that still holds content), logs every removal, and recomputes type/subtype counts afterward so totals stop counting ghosts.

• Reads no longer hang behind an unrelated index migration (family-scoped gate). During a native provider's one-time background migration, every read — including plain get(), VFS readdir/readFile, and metadata-only find({ where }) — blocked on the whole-brain migration lock until timeout, even when the migrating index was irrelevant to the read. The gate is now scoped to the index families a read actually consults: canonical reads (get, batchGet, VFS content) never wait; a find waits only on the families its query shape needs (vector for semantic, metadata for where/type, graph for connected); graph traversals wait only on the graph family. Writes and unclassified operations keep the conservative whole-brain wait. A read that does need the migrating family still blocks (bounded by migrationWaitTimeoutMs) and surfaces the retryable MigrationInProgressError — never a partial result.

No breaking API change. Each fix ships with regression tests.

v8.3.0 — 2026-07-13 (faster index heals + the cross-layer integrity contract)

Three additive changes. The first is an immediate, standalone performance win; the other two are the brainy side of the write/index-spine integrity contract, inert until a native accelerator that implements the matching hooks is present — so this release changes nothing for a JS-only brain beyond the speedup.

• Canonical enumeration is up to ~16× faster — the dominant term in an index heal. The paginated entity walk (getNounsWithPagination) hydrated each entity's vector + metadata one-at-a-time; since every index rebuild enumerates canonical storage, that serial per-item latency dominated multi-minute heals. Hydration is now 16-way bounded-concurrency, with the pagination contract (order, cursor resume, filters, totalCount) byte-identical to before. New getNounIdsWithPagination() returns ids without hydrating anything (zero per-entity reads when unfiltered) for callers that own their own IO schedule.

• Cross-layer integrity — validateIndexConsistency() is no longer blind to native providers. It only ran the JS metadata index's own check, so a native provider whose manifest/segments/counts had diverged still read as "healthy". It now feature-detects and aggregates each provider's optional validateInvariants() self-report, names every failing invariant with its numbers, and exposes the per-provider reports. repairIndex() now reconciles native derived state from canonical too (rebuilding any provider whose failing invariant asks for it). New exported types ProviderInvariantReport / InvariantResult / InvariantHeal.

• Registered-blob families — declared index files are undeletable through the storage layer. A provider can declare a derived-index blob family (a set of members that are load-bearing together); once declared, deleteBinaryBlob / removeRawPrefix refuse to remove a member (new exported ProtectedArtifactError), so a stray in-process sweeper cannot delete a load-bearing index file. The declaration persists across reopen; checkDerivedFamiliesPresent() names any member missing on open. New optional StorageAdapter surface (registerDerivedFamily / unregisterDerivedFamily / listDerivedFamilies + DerivedFamilyDeclaration) and exported DerivedArtifactMissingError.

No breaking API change (all additions are optional/new). Each change ships with regression tests.

v8.2.8 — 2026-07-13 (honest index readiness — no more silently-empty queries on a cold index)

Closes the last of the three spine anti-patterns: the "dishonest readiness proxy," where size() > 0 was treated as "this index actually serves queries." On a cold open (fresh boot, restart, crash recovery) a native index can load its count before its serving structure — so for a brief window it has data but cannot answer, and a query returned a silent empty result indistinguishable from "no such data." Every fix here asks the index whether it can actually serve, and if not, self-heals or fails loudly instead of returning [].

• Semantic search no longer returns a silent [] on a cold vector index. A pure semantic find({ query }) has no filter, so nothing previously guarded the vector index. A new one-shot guard verifies the vector index serves a known persisted vector on the first semantic/proximity search — preferring the provider's honest isReady() signal, else a known-vector self-match probe. It rebuilds from canonical records if the serving structure did not load, and throws the new VectorIndexNotReadyError only if a rebuild still cannot serve — never a silent empty result.

• Relationship reads fall back to the canonical scan instead of an empty result on a cold graph index. getVerbsBySource/getVerbsByTarget (used by relationship queries and virtual-filesystem traversal) skipped the fast path only on isInitialized — which reads true once the manifest loaded even if the source→target adjacency did not. They now consult the honest readiness signal and, when the adjacency is not serving, take the correct-but-slower canonical shard scan. A one-shot self-heal probe covers providers that expose no readiness signal.

• getIndexStatus() tells the truth for readiness probes. It reported populated: size>0 only, so a Kubernetes readiness check could route traffic to a brain still warming up. It now folds in the honest per-index ready signal (making populated honest), plus the degraded states already surfaced by checkHealth()/validateIndexConsistency() (rebuildFailed/rebuildError and a degradedIds count) — so a probe never reports 200-ready over a known-degraded index.

This completes the write/index-spine hardening end to end. New export: VectorIndexNotReadyError; getIndexStatus() gains additive fields (rebuildFailed, rebuildError?, degradedIds, per-index ready?). No breaking API change. Each fix ships with a dedicated regression test.

v8.2.7 — 2026-07-13 (loadBinaryBlob fault-propagation — the lockstep completion of 8.2.6)

Completes the Pass-1 spine hardening. loadBinaryBlob (the raw-blob read a native accelerator mmaps for its index files) previously returned null on ANY read error, so a real IO fault (EIO/EACCES/EMFILE) on a present-but-unreadable index blob masqueraded as "the blob is absent" — driving a needless full rebuild or an empty read. It now distinguishes genuine absence (ENOENT → null, the documented contract) from a real fault (→ throw), so a transient disk fault surfaces loudly instead of silently degrading the index.

This one change was deliberately held out of 8.2.6 and ships now, in lockstep with the native accelerator release that hardened its two column-store read sites to handle the throw (a faulted segment read marks the field unavailable and throws a named error, instead of relying on null-on-error). A consumer on new brainy + an older accelerator was never at risk: 8.2.6 kept the prior swallow-on-fault behavior for this method until the accelerator was ready.

No API change. Regression: tests/unit/storage/blob-save-durability.test.ts gains the loadBinaryBlob leg (absent → null; present → bytes; real fault → throws).

v8.2.6 — 2026-07-13 (write/index-spine hardening — loud errors, never quiet losses)

Durability + integrity hardening across the write and index paths. Every fix converts a place that could silently lose data, serve a partial result, or acknowledge a write that did not land into a loud, observable failure. No breaking API changes; one new exported error.

• A durable blob write that stored nothing is no longer acknowledged. The atomic blob write (saveBinaryBlob, used for vector/graph index segments and the native index files) uses a unique per-writer temp file, so a rename ENOENT can only mean our temp vanished before the rename — the bytes never landed. It previously returned success on that path; it now retries once with a fresh temp and, if the temp vanishes again, throws instead of acknowledging a write that persisted nothing. A downstream deployment that mmaps these files no longer finds a "successfully written" blob missing on the next open.

• Single-op history durability is enforced, not assumed. The asynchronous group-commit flush that persists single-op generation history previously swallowed a persist failure as a warning while writes kept succeeding and their history piled up, undurable, in memory. It now tolerates a transient blip (retry with capped backoff) and, after repeated failures, refuses further writes with the new PendingFlushDurabilityError rather than promise a durability it cannot deliver. Live canonical data is untouched; the latch self-heals the moment a flush succeeds. Callers needing hard per-write durability should keep using transact() (or flush() after a single-op).

• A degraded derived index is surfaced on reads, not served silently. When a non-fatal index rebuild fails at open, or a write commits via adopt-forward recovery with an incomplete derived index, find() and get() now emit one loud warning per degraded window (reads still return — canonical is the source of truth), the state folds into checkHealth() / validateIndexConsistency(), and repairIndex() reconciles and clears it.

• clear() removes the full derived footprint. It previously left raw index blobs, the native id-mapper, and column-index manifests on disk, so a cleared brain could re-read stale native state on reopen. It now wipes them together as a set.

• Counts stay honest across deletes. A delete decremented the per-type breakdown but not the scalar total, so the total inflated permanently (and won pagination). Delete now decrements both; the invariant total === Σ per-type holds across any interleaving of add / visibility-flip / delete and across a reopen.

• Index-maintenance and aggregation failures are loud. A partial LSM/segment load no longer publishes the manifest's full count as if healthy; an HNSW flush that can't persist a node throws (HnswFlushError) instead of returning a lying count and dropping the node; a corrupt or missing manifest-listed column segment throws (ColumnSegmentLoadError) instead of silently dropping its entities from every query; and aggregation materialization / state-load failures now warn instead of vanishing into an empty catch.

New export: PendingFlushDurabilityError (with .cause and .failedAttempts). No other public API change. Each fix ships with a dedicated regression test.

v8.2.5 — 2026-07-12 (honest response when a transaction rollback can't complete)

Data-integrity fix. When a transaction failed and its rollback then also failed to undo a canonical write (retries exhausted), the old behavior logged, continued, and threw TransactionRollbackError — while the record it couldn't undo stayed durable on disk, and the transaction even reported its state as 'rolled_back'. The caller got an error implying the write was undone; a read-back showed the record. A failed rollback had no truthful response.

Rollback now tells the truth, with a two-branch contract:

• Adopt-forward (safe case). A single-op write (add/update/…) whose only damage is a durably-present record — the write the caller asked for — is adopted: its generation is committed, the write returns success, and a loud warning records that the derived index may be incomplete for that id until the next rebuild/repairIndex(). No error, no double-write; the record is immediately durable and retrievable by get().
• Fail loud + quarantine (unsafe case). A multi-operation batch, or any case where a record was lost (a remove/update whose restore-undo failed), throws the new StoreInconsistentError naming every unreconciled record and its disposition (orphan vs loss), and puts the brain into write-quarantine: reads keep working, but writes are refused until repairIndex() reconciles the derived indexes against canonical storage and lifts the quarantine. The generation counter is not advanced, and a transaction whose rollback failed is now 'inconsistent', never the 'rolled_back' lie.

The decision is made by observation, not guesswork: both commit paths already hold byte-identical before-images, so after a failed rollback the store compares current canonical state to them to classify exactly which records are orphaned or lost.

New export: StoreInconsistentError (with .records and .cause) and the UnreconciledRecord type. No other API change; repairIndex() gains the quarantine-lift behavior. Regression (tests/integration/rollback-trapdoor.test.ts) injects the exact failure (index add throws → canonical delete-undo fails) and pins adopt-forward (durable, get-able, not quarantined), fail-loud (StoreInconsistentError + quarantine + reads work + repairIndex() lifts it), and the error's record naming.

v8.2.4 — 2026-07-12 (restore can no longer destroy the store it's recovering)

Recovery-safety fix. restore() removed the entire live brain directory and THEN copied the snapshot in — so any copy failure left the store destroyed with only a partial copy. The sharpest edge: the copy (fs.cp) materialized the holes of sparse mmap blob files, so a snapshot that fits on disk could balloon and ENOSPC mid-copy, and the recovery tool would have just destroyed the brain it was asked to recover. Reported from a downstream incident's recovery forensics.

Restore is now non-destructive and crash-resumable:

• The snapshot is copied into a staging area (_restore_staging/) before any live data is touched. The copy is sparse-aware — all-zero regions are left as holes, so a store of mostly-hole blobs restores at its true allocated size instead of its apparent size.
• A copy failure (including ENOSPC) removes only the half-written staging area and throws; the live store is left exactly as it was.
• Only after the copy succeeds and a completion marker is fsync'd does an atomic per-entry swap move the staged data into place — same-filesystem renames that cannot fail for disk space.
• A crash mid-swap is finished forward on the next open: startup resumes a committed-but- incomplete swap, or discards an uncommitted staging area (live data still authoritative).

No API change — restore(path, { confirm: true }) is unchanged. persist() was already safe (hard-link snapshot). Regression (tests/integration/restore-nondestructive.test.ts): a forced copy failure leaves live data fully intact, a normal restore round-trips, an interrupted-but- committed restore completes on reopen, an uncommitted staging area is discarded, and the sparse copy is byte-identical with allocation far below apparent size.

v8.2.3 — 2026-07-12 (a committed transaction is durable on return)

Durability fix. A transact() reported "committed" while its canonical entity writes were still only in the OS page cache (written via tmp+rename, not yet fsync'd), even though the generation counter and manifest WERE fsync'd. A hard kill (power loss, SIGKILL) in that window could leave the durable generation counter ahead of the persisted entity bytes — so a consumer resuming from the counter would see "phantom progress": a generation that claims writes the disk never kept. Reported from a downstream migration's crash-lifecycle forensics.

commitTransaction now runs a durability barrier: it records every canonical write and delete the batch's operations make, then fsyncs that entire footprint (file contents, the rename directory entries, and the parent directories of any deletes) before advancing the generation counter and manifest. A committed transaction is therefore durable the moment transact() returns — the counter can never outrun the entity bytes.

Durability contract, now explicit. transact() is durable-on-return (above). A single-op write (add/update/remove/relate/…) is Model-B group-commit: its live bytes are written but become durable at the next flush() or close(), not the instant the call resolves — the counter is buffered alongside the data, so a crash loses both together (never a torn counter-ahead-of-state store). Need per-write durability? Use transact() (even for one op), or flush() after the write. This deliberately trades single-op fsync latency (a 3-5x write regression) for throughput.

No API change; no accelerator involvement (the barrier is in the filesystem storage adapter). In-memory and durable-per-call (cloud object-PUT) adapters treat the barrier as a no-op. Regression: tests/integration/transact-durability-barrier.test.ts proves entity writes fsync in an earlier batch than the manifest, for single-op and multi-op (add+relate) transactions, and that a precommit-rejected batch opens no barrier and advances nothing.

v8.2.2 — 2026-07-11 (P0: a timed-out transaction now rolls back — no torn state)

Data-integrity fix. A transaction that exceeded its time budget mid-flight (e.g. a bulk transact() on slower hardware crossing the 30s ceiling) threw a timeout error WITHOUT rolling back the operations it had already applied. The budget check sat outside the per-operation rollback path, so only per-operation failures rolled back — a timeout stranded the partial writes in canonical storage while the generation was never stamped, leaving torn, generation-less state. This was caught by a downstream migration that crossed the ceiling on a large batch.

Transaction.execute() now has a single rollback point: any error that escapes the operation loop — an operation failure OR a mid-flight timeout — rolls back every applied operation in reverse order, then surfaces the original error (a rollback failure supersedes it, loudly, as before). This restores the invariant the generation-store commit path already depended on: a throw from execute means the applied operations were undone byte-identically, and the aborted transaction leaves generation() unchanged and storage byte-identical to its pre-transaction state.

No API change. Regression pins the reported requirement — after a mid-flight timeout, storage is byte-identical and the transaction ends in the rolled_back terminal state (tests/unit/transaction/timeout-rollback.test.ts), plus the operation-failure and rollback-failure paths through the same single rollback point.

Note on the 30s ceiling itself (configurable/scaled timeout, batched embedding precompute, timeout telemetry) — that ergonomics work is tracked separately; this release fixes only the correctness bug (a timeout must never leave partial state), independent of where the ceiling sits.

v8.2.1 — 2026-07-11 (transact forward references work on the native accelerator)

Parity fix. transact() has always promised atomic forward references — add an entity and relate to it in one batch — but the transaction planner resolved relationship endpoint ids at plan time, before the batch's adds had applied. Asking the id mapper about an entity that doesn't exist yet made the accelerator's (correctly strict) native mapper throw in EntityIdMapper.getOrAssign, so transact([{op:'add', id:X}, {op:'relate', to:X}]) failed on native deployments while the permissive JS mapper masked the bug — and silently leaked an id assignment whenever a batch was later rejected by a commit precondition.

Endpoint resolution is now lazy — evaluated when the graph operation executes inside the commit, after the batch's adds have applied (the same lazy pattern the operation's generation stamp already used). Fixed across all transact-planned graph operations: relate (including the bidirectional reverse edge), remove's relationship cascade, and unrelate. Single-operation writes are unchanged. Also fixed as a byproduct: a rejected batch no longer pollutes the id mapper.

Regression suite covers the exact reported shape, both-endpoints-in-batch, bidirectional, add+relate+remove in one batch, and the mapper-cleanliness proof on rejected batches (tests/integration/transact-forward-ref-graph.test.ts). No API changes; no accelerator version pairing required.

v8.2.0 — 2026-07-10 (temporal VFS — file content joins the immutability model)

Time travel now covers Virtual Filesystem content. Previously the temporal model had a hole exactly where files were concerned: every entity write was an immutable generation, but the content bytes lived under an eager reference-count GC — deleting a file could physically destroy bytes that in-window history still referenced, while overwriting never released the old content at all (an unbounded silent leak that only accidentally preserved history). A production consumer's recovery from a bad deploy succeeded only because a stale field happened to hold the good value — luck, not a guarantee. Both directions are now fixed by making blob reclamation a history decision instead of a liveness decision:

• Content blobs are retention-protected. Each blob tracks live references AND history references (one per persisted generation record that carries its hash). Deleting/overwriting a file drops only the live reference; bytes are physically reclaimed in exactly one place — history compaction — once no live reference and no retained generation references the hash. Pinned views are exempt automatically (compaction already respects pins). Crash ordering is over-count-only (never under-count), so a crash can leak until the built-in scrub recounts, but can never reclaim bytes history still needs. Existing stores get a one-time, marker-gated backfill on open; cross-file content dedup is handled exactly.
• vfs.readFile(path, { asOf }) — the file's exact bytes as of a generation or Date, guaranteed present within the retention window.
• vfs.history(path) — the file's versions ({ generation, timestamp, hash, size, mimeType? }, ascending). Restore = read the old bytes asOf and write them back (a new write; history is never rewritten).
• Overwrites now refresh the file entity's data/embedding text — previously semantic search and the data field served the FIRST version's text forever.

Lifecycle note: deleting a file no longer frees its bytes immediately — old content lives until compaction reclaims its generations, under the same retention budget as all Model-B history (retention: 'all' keeps every version forever). Guide: the new "Time travel for files" section in docs/guides/snapshots-and-time-travel.md. Native accelerator: unaffected (canonical write path only) — no version pairing required.

v8.1.0 — 2026-07-10 (brain.onChange — the in-process change feed)

New public API: subscribe to every committed mutation with brain.onChange(cb) → unsubscribe. One event per affected record, for every canonical write regardless of origin — direct calls, batch methods, transact(), imports, and Virtual Filesystem writes all funnel through the same commit point the feed is emitted from. This is the authoritative in-process signal for live UIs, cache invalidation, and realtime sync layers (downstream SDKs can forward it over their own transports to make local and remote brains uniform).

Event shape (BrainyChangeEvent, exported): kind (entity/relation/store), op (add/update/remove/relate/unrelate/updateRelation/clear/restore), the post-commit entity or relation view (type + full custom metadata), the committed generation, and timestamp. Notable properties:

• Post-commit only — an aborted write (a losing ifRev CAS, a rejected transaction) never emits. If you got the event, the write is durable.
• Deletes fully described — remove/unrelate carry the record's last committed state (from the commit's own history record), not just an id; batch deletes included.
• Batches emit per item; a transact() batch's events share its single generation. Entity deletes also emit unrelate for each cascaded relationship.
• Commit-ordered, asynchronous, isolated — events arrive in commit order, dispatched in a microtask so a slow listener never delays a write and a throwing listener never affects the write or other listeners. Zero overhead with no subscribers.
• kind: 'store' events for clear()/restore() mean "refetch everything".
• Fire-and-forget by design; each event's generation composes with asOf()/transactionLog() for catch-up after a gap.

Guide: docs/guides/reacting-to-changes.md. No behavior changes for existing code.

v8.0.17 — 2026-07-08 (count recovery scans the real layout · ~1,100 lines of dead 7.x machinery removed)

A cleanup of the vestigial 7.x "hnsw sharding" machinery that turned up two real fixes.

1 — Count recovery now scans the layout the database actually uses. When counts.json is lost or corrupted (container restarts, partial copies), the recovery scan rebuilt the entity/ relationship counters by counting files in entities/*/hnsw/ — directories the 8.0 write path never populates — so an established store recovered to zero counts on real data: wrong getNounCount()/stats, a "New installation"-style boot log, and a mis-sized rebuild-strategy decision at open. The scan now counts the canonical entities/<kind>/<shard>/<id>/ tree. Regression-tested: delete counts.json from a populated store → reopen → exact counts.

2 — A latent init-time deadlock removed. The recovery scan's type-distribution sampler called a guarded accessor (getNounMetadata → ensureInitialized) from inside init(), which re-enters init() and hangs the open. It was unreachable before only because the old scan never found anything to sample; the fix reads the sampled metadata files directly.

3 — The dead machinery itself is gone (−1,138 lines). Twenty-three methods with zero callers: the 7.x hnsw-layout entity/edge CRUD, the sharding-depth prober + depth-migration engine, and an orphaned streaming paginator. Verified by reachability analysis before deletion; no public API touched; the full suite is green. New stores no longer carry the always-empty legacy directories' scan cost, and the boot log keeps the truthful new-vs-established wording from v8.0.13.

v8.0.16 — 2026-07-08 (concurrency fast-follow: atomic ifAbsent/upsert + exact blob reference counts)

Closes the two remaining check-then-act races found in the sweep that followed v8.0.15's CAS fix (disclosed in that release's notes). Same bug class — a check performed before the serialization point that guards the apply — same cure.

1 — add({ ifAbsent }) and add({ upsert }) are now atomic. The absence check ran before the commit mutex, so N concurrent same-id creates could all pass it and all write — the second silently overwriting the first, violating ifAbsent's "returns the existing id without writing" contract and upsert's "merge, never clobber" contract. The insert leg now carries a must-be-absent precondition (the same conditional-commit primitive as ifRev), verified under the commit mutex: exactly one concurrent create wins; every other caller takes its documented resolution — ifAbsent returns the existing id with zero writes, upsert merges into the now-existing entity (with a bounded retry if a concurrent delete intervenes). Verified: 8 concurrent ifAbsent creates advance the store by exactly ONE generation; 8 concurrent upserts on an absent id produce one create + seven merges (_rev lands at exactly 8). Note: transact()'s batch-level ifAbsent/upsert keep planning-time semantics (the batch converges to a valid entity; the window is documented, not silent).

2 — Blob reference counts are exact under concurrency. The content-addressed blob store's write() (dedup: exists → add a reference, absent → create) and delete() (decrement → remove at zero) were unserialized read-modify-writes over the blob's metadata. Concurrent writes of identical content could lose references — making a later delete remove bytes another file still referenced (data loss), or leak unreferenced blobs. All reference-count-bearing mutations are now serialized per content hash (distinct content never contends): N concurrent identical writes yield exactly N references, and a blob is physically removed only when the true last reference drops. Verified with concurrent write/delete storms.

Both fixes are in-process complete by construction — storage enforces single-writer-per-directory, so the process is the whole concurrency domain. No API changes.

v8.0.15 — 2026-07-08 (ifRev CAS is now atomic — exactly one winner under concurrency)

Correctness fix for optimistic concurrency, reported from a production cutover rehearsal. N concurrent update({ ifRev }) calls carrying the same expected revision ALL succeeded — zero RevisionConflictErrors, last-writer-wins, the other N−1 writes silently lost. (Sequential calls conflicted correctly.) The revision check ran before the commit mutex, so interleaved callers all passed it before any apply landed — breaking the exactly-one-winner semantics the optimistic-concurrency guide promises, which advisory locks and per-entity ledgers/counters build on.

The fix makes the check-and-apply a conditional commit: the generation store's commit paths accept a precondition that runs under the commit mutex against the just-read authoritative before-images — the per-record analogue of ifAtGeneration, which always ran there. update() and transact() per-op ifRev both re-verify at that point (the earlier check remains as a cheap fast-fail). A conflict aborts atomically: nothing staged, nothing applied, the same RevisionConflictError as before. Two adjacent behaviors also became honest:

• _rev is now monotonic under concurrency. The winner's stamp derives from the authoritative before-image, so N concurrent plain updates advance _rev by N (previously they could all stamp the same stale value).
• CAS against a concurrently-deleted entity now throws EntityNotFoundError instead of silently resurrecting it (plain updates keep their last-writer-wins re-create semantics).

Verified with a concurrency regression suite: 8 parallel same-rev updates → exactly 1 winner + 7 conflicts; the documented read→CAS→retry ledger loop converges exactly (8 workers, 8 decrements, 0 lost) — tests/integration/ifrev-concurrent-cas.test.ts. If you serialized writes in your own adapter as a workaround, it can come out after this upgrade.

v8.0.14 — 2026-07-07 (7→8 migration preserves branch-scoped non-entity state instead of deleting it)

Defense-in-depth for the one-time 7→8 layout migration. The migration rescues the head branch's entities (branches/<head>/entities/* → entities/*) and then drained the whole branches/<head>/ directory. If a 7.x engine had written durable state under the head branch outside entities/ (a branch-scoped index, blob area, or field registry), that drain would have silently deleted it — the same failure class as the VFS content blobs a 7.x store kept in _cow/. The migration now drains the branch only when nothing but the moved entities remains; if any non-entity object survives, it preserves the branch (no delete) and logs a loud warning naming the leftover keys, so the state is recoverable rather than lost. No effect on a normal migration (a clean branch still drains); purely a guard against silent data loss.

Cosmetic boot-log fix. Every persisted 8.0 store logged 📁 New installation: using depth 1 sharding on every open — even established brains holding thousands of entities — which is alarming to read during a restart or incident. Cause: 8.0 stores nouns in the canonical entities/nouns/<shard>/<id>/vectors.json layout, but the legacy sharding probe inspected the entities/nouns/hnsw/ directory, which the 8.0 write path never populates, so it always concluded "new". The new-vs-existing decision now consults the layout the database actually reads and writes (plus the known noun count), so an established store logs 📁 Using depth 1 sharding (N entities) and only a genuinely empty store reports a new installation. No behavior change beyond the log line — it never triggered a rebuild or migration; entities were always read correctly.

v8.0.12 — 2026-07-07 (7→8 VFS-content recovery · zero-rebuild cold open · strict query operators)

Three consumer-facing fixes.

1 — A 7→8 upgrade recovers Virtual Filesystem content automatically (data-integrity). 7.x stored VFS file content as blobs in the branch system's copy-on-write area (_cow/). 8.0 removed that system and stores content blobs in the content-addressed store (_cas/), but the one-time layout migration only moved entities — it never adopted the _cow/ content blobs. On a store that used the VFS (for example, a CMS with published pages), that left every VFS-backed read throwing Blob metadata not found and the pages 500ing. 8.0.12 adds an on-open recovery pass that adopts every orphaned _cow/ blob into _cas/ — copying both the bytes and the metadata, idempotently and non-destructively (the _cow/ originals are never deleted). It heals both a fresh 7→8 upgrade and a store already upgraded by an earlier 8.0.x that stranded them, with no operator action: just open the store under 8.0.12. An explicit force path is exposed as brain.vfs.adoptOrphanedBlobs() (returns { cowBlobs, adopted, alreadyPresent, incomplete }). The automatic pre-upgrade backup is now retained if any blob can't be fully adopted (incomplete > 0) instead of being removed on entity-migration "success" alone. Affects any 7.x store that used the VFS; native-8.0 and non-VFS stores no-op on a cheap existence check. Full guide: docs/guides/upgrading-7-to-8.md.

2 — A cold open no longer re-derives durable indexes it can just load. Opening a persisted store rebuilt the vector, graph, and metadata indexes from the canonical records even when the durable index state was present and loadable — an O(N) cost paid on every boot (measured at ~48 s on an ~11k-entity store). The rebuild gate now consults an honest per-provider durability signal (init() / isReady()) instead of an in-memory size heuristic, so a provider that has loaded (or can cheaply demand-load) its persisted index is not rebuilt. The built-in JS indexes keep today's behavior (a rebuild is their load path); the live query-time guards that self-heal a genuinely lost index are unchanged. The zero-rebuild boot activates when the accelerator exposes the durability signal (@soulcraft/cor@3.0.5); on earlier accelerator versions 8.0.12 falls back to the previous behavior safely — no regression, just no speedup yet.

3 — Unknown query operators now throw instead of silently returning nothing. find({ where }) had two gaps: the in-memory matcher (used for egress re-validation and historical reads) was missing several documented operators (in, greaterThanOrEqual, lessThanOrEqual), so it silently disagreed with the index path; and an unknown operator key (a typo, or notIn written where not: { in } was meant) was treated as a nested-object field and silently matched nothing. 8.0.12 aligns the matcher to the full documented operator set and validates the where filter up front, throwing a typed BrainyError('INVALID_QUERY') naming the bad operator. Dotted paths remain the supported form for nested fields. If you relied on an unknown key silently returning [], it now throws — the fix is to use the documented operator or dot-notation.

v8.0.11 — 2026-07-02 (no script shape can hang on brainy's internals)

Completes v8.0.10's exit fix for every operation class. Two further mechanisms found and fixed: the beforeExit auto-flush hook looped forever on any script that never reaches close() (Node re-emits beforeExit after each event-loop drain and the async flush schedules new work — it now self-deregisters before its single flush, which still lands your buffered data before exit), and every background-maintenance interval (graph auto-flush, LSM compaction, metadata write-buffer, VFS/path-cache maintenance, statistics debounce) is now unref'd at creation. Verified against the published package: an add + relate script exits cleanly both with close() (~0.5 s) and with no teardown at all — with the data confirmed durable on reopen. A per-operation-class sweep test now asserts no ref'd timer survives close(), so this bug class stays closed.

v8.0.10 — 2026-07-02 (a bare script exits cleanly after close())

A minimal init → add → close script used to hang forever after close() returned — brainy held four process keep-alives it never released: the global SIGTERM/SIGINT shutdown hooks (never removed; now deregistered when the last live instance closes), an internal cache-fairness interval (unclearable by construction; now unref'd), and the VFS/PathResolver maintenance intervals (close() never shut the VFS down; now wired). Verified against the published package: the same script now exits ~1 ms after close() resolves. No API change; servers and long-running processes are unaffected.

v8.0.9 — 2026-07-02 (guarded plugin auto-detection — the "install it and it's on" contract)

With the default config (plugins unset), brainy now auto-detects the first-party accelerator: installing @soulcraft/cor is the opt-in. The detection is guarded — everything except "not installed" fails loud:

• Not installed → plain brainy, silently (the free path — zero noise, zero cost).
• Installed and healthy → it loads and announces itself ([brainy] Plugin activated).
• Installed but broken (unresolvable, invalid shape, failed activation, version mismatch) → init() throws. An installed accelerator never silently vanishes behind the JS engines.
• plugins: [] / false = explicit opt-out; plugins: ['@soulcraft/cor'] pins the exact list (unchanged semantics).

Also new: if a plugin activates but registers zero native providers (e.g. a licensing gate declining to engage), brainy warns loudly instead of leaving you to discover every query is running on the JS engines.

This supersedes v8.0.8's "plugins are explicit opt-in" wording, which documented the pre-GA loader accurately but contradicted the published product contract ("add the package and it activates"). 8.0.9 makes the contract true — with the loud-failure guarantees intact.

v8.0.8 — 2026-07-02 (docs-only patch on the GA)

Corrects the README's scale-up section: the native provider is not auto-detected — plugins are explicit opt-in (new Brainy({ plugins: ['@soulcraft/cor'] }); brainy never auto-imports a package you didn't list, and a listed plugin that fails to load throws rather than silently falling back to the JS engines). Also fixes the stale plugins config comment in the public types and one phrase in the plugin-author guide. No code change.

v8.0.7 — 2026-07-02 (GA, npm tag latest)

Why 8.0.7: npm permanently retires unpublished version numbers, and 8.0.0–8.0.6 were consumed by a January development cycle (published and immediately unpublished). 8.0.7 is the first stable release of the 8.x line; there are no earlier stable 8.0.x releases.

8.0.7 is the first stable major on the u64-id core. It ships in lockstep with the optional native provider's 3.0 (the billion-scale path). Everything below works standalone on the open-core JS engine — a native provider is feature-detected and only changes the scale ceiling, never the API.

Upgrading from 7.x: nothing to script. The first time 8.0 opens a 7.x brain it upgrades itself — an automatic, observable, coordinated migration lock rebuilds every derived index from your canonical records while Brainy blocks and queues reads and writes, so no operation ever touches a half-built index and no write is ever lost. Budget seconds-to-minutes per brain at large sizes; small brains rebuild inline on open. A pre-upgrade hard-link backup is taken automatically (removed on success, kept on failure for rollback). Observe it via getIndexStatus().migration; a caller that hits the window gets a typed, retryable MigrationInProgressError (catch → HTTP 503 + Retry-After). Bounded by migrationWaitTimeoutMs (default 30 s — it bounds your wait, not the rebuild). Opt out of the backup with migrationBackup: false.

Headline changes

• The cold-open silent-[] class is gone. On a cold reopen, filtered finds (find({ where })) and graph reads (find({ connected }) / neighbors() / related()) never silently return [] for data that is actually present. With a native provider the durable indexes cold-serve every filter and edge with zero rebuild; the open-core engine adds belt-and-suspenders guards that self-heal from canonical data or throw a loud, typed error — never a silent empty result. Two new exported errors: MetadataIndexNotReadyError, GraphIndexNotReadyError.

• Faster vector search (open-core). The JS distance functions are allocation-free loops instead of reduce: ~6× cosine, ~1.4× euclidean (MEASURED — tests/benchmarks/distance-microbench.mjs, 384-dim, median of 41). Numerically identical, so recall is unchanged. Exact metadata-filter pushdown and scale-aware search width keep find() recall-correct as data grows.

• Per-write immutable history. Every write is generation-stamped; now() / asOf() / transact() read a consistent point in time, and a retention knob bounds history growth. Single writes participate in the same immutable timeline as transactions.

• Billion-scale resident memory. Nothing is O(N)-resident on the write or time-travel path — per-id caches removed, the committed-generation ledger is an interval set, per-id history chains are bounded (hot-window + LRU). Resident memory is independent of entity count.

• Deterministic, round-trip-free writes. Supply your own ids, forward-reference not-yet-written entities inside a transact(), ifAbsent upsert, and brain.newId() (uuidv7) — write graphs without a write→wait→get round-trip.

Breaking changes (summary — the full migration guide follows below)

• Runtime floor: Node ≥ 22 / Bun ≥ 1.1 (Bun recommended). Compiler target ES2023; the DOM lib is dropped — 8.0 is a Node/Bun/Deno engine with no browser path.
• neural() removed and Db.search removed — use find() on the brain.
• Storage config is one path key. Removed aliases now throw with a message pointing at path.
• get() omits the vector by default — pass { includeVectors: true } when you need it.
• Export/import type is PortableGraph (was BackupData; wire tag brainy-backup → brainy-portable-graph). Re-export any snapshots you version outside Brainy.
• Reserved visibility field (public / internal / system) on nouns and verbs; internal and system are excluded from normal reads by default.
• 4 deprecated query operators removed (is/isNot/greaterEqual/lessEqual) and reserved keys in a metadata bag now throw by default — details in the rc notes below.

Post-rc.9 hardening folded into GA (no on-disk or provider-contract change vs 8.0.0-rc.9): the metadata cold-read guard, the auto pre-upgrade backup (with an _id_mapper/* mmap byte-copy correctness fix), and a CI fix so a fresh clone builds green.

RC history (rc.1 → rc.9, npm tag rc)

rc.9 changes (2026-07-01):

• Whole-brain auto-upgrade is now a coordinated, observable LOCK — supersedes rc.8's no-freeze approach. When a large brain's derived-index format changes (7.x→8.0), the native provider rebuilds the indexes from the canonical records in place while Brainy blocks and queues reads and writes — so no operation ever touches a half-built index. This is a clean blocking upgrade to a known-good state (vs rc.8's online background swap): unknown/halfway states are more dangerous than a bounded wait. Small brains still rebuild inline on open. New surface, all additive: a typed, exported, retryable MigrationInProgressError (catch it → HTTP 503 + Retry-After); getIndexStatus() gains migrating + migration progress (never gated — the readiness-probe signal); a migrationWaitTimeoutMs config (default 30 s — it bounds the caller's wait, NOT the rebuild, which is unbounded); health() / checkHealth() report the upgrade without blocking. No effect without a native provider.
• Faster vector search (open-core). The JS distance functions are rewritten from reduce to allocation-free loops — ~6× cosine, ~1.4× euclidean (MEASURED, tests/benchmarks/distance-microbench.mjs, 384-dim, median of 41). Numerically identical → recall unchanged.
• Runtime floor + Bun. Engines are now Node ≥22 / Bun ≥1.1 (fixes a Node-24 EBADENGINE); compiler target ES2023 with DOM dropped from the type lib (8.0 is Node/Bun/Deno-only, no browser path). Bun is recommended as a runtime (bun add / bun run); the single-binary bun build --compile is not a supported target (native addon can't embed + a Bun 1.3.10 codegen regression). .d.ts stays TS-5.x-parseable.

rc.8 additions (2026-06-30) — additive, no breaking change:

• No-freeze (online) whole-brain auto-upgrade. When a derived-index format changes, a large brain upgrades without blocking — the native provider rebuilds the new indexes in the background and serves correct reads from the canonical records meanwhile, then atomically swaps; no minutes-long freeze on the first open/query. (Small brains still auto-rebuild inline on open.) New surface: an optional provider isMigrating() deference signal, a public brain.stampBrainFormat() the provider calls once its swap verifies, and a @soulcraft/brainy/brain-format export so the provider shares the indexEpoch constant. No effect without a native provider.

rc.7 additions (2026-06-30) — additive, no breaking change:

• 8.0 cold-graph self-heal. find({ connected }) / neighbors() / related() never silently return [] on a cold open where the graph adjacency loaded its membership but not its source→target edges — it self-heals (rebuild from storage) or throws a loud GraphIndexNotReadyError, never empty-for-persisted-data. (The 8.0 equivalent of the 7.33.4 fix 7.x consumers already have; gates on the native provider's honest isReady() edge-readiness signal.)
• Billion-scale RAM — nothing O(N)-resident on the write/time-travel path. Eliminated the per-id storage caches (per-type counts now sourced from the record), made the committed-generation ledger an interval set, and bounded the per-id time-travel history chains (hot-window + LRU, with lock-light reconstruction so historical reads never stall writers). Resident memory is now independent of entity count.
• Whole-brain version handshake. A _system/brain-format.json marker + brain.formatInfo()
  • a shared, lockstep indexEpoch: a future derived-index format change auto-rebuilds the indexes from the canonical records on open (non-destructively) — the foundation for whole-brain auto-upgrade with the native provider. No effect on an unchanged brain.

rc.6 additions (2026-06-29) — additive, no breaking change:

• Open-core perf: HNSW delete is now O(in-degree) (was O(N²) for bulk delete) via a reverse-adjacency index; the negation/absence operators (ne/exists:false/missing:true) are served as a roaring-bitmap difference instead of materializing the whole corpus.
• Native provider contract (cor lockstep, optional/feature-detected — no effect without a native provider): a cold-open probeConsistency() self-heal hook, and a getIdsForFilter page bound so the native index can early-stop the unsorted find({ type, where, limit }) path.
• Test hygiene: re-homed previously-unrun test suites into CI + a guard so a test file can never silently fall outside every config again. No source/API change from these.

Affected products: every consumer — this is a major release with removed surfaces, hard renames (no aliases, no deprecation period), and one flipped default. This entry is the migration guide: the find/replace pairs, the sed snippet, and the upgrade checklist below are the complete story — there is no separate migration doc. 8.0.7 is GA on latest — install with npm i @soulcraft/brainy@latest.

rc.3–rc.5 additions (2026-06-24):

• Showcase-quality GA hardening (rc.5). A full readiness audit closed a cold-init version-coupling bug (a matched native provider could be rejected on first open), turned silent storage-read failures into named BrainyErrors (no more empty-result-on-error), stopped the JS graph LSM from orphaning compacted SSTables, corrected the public docs to the real API, and documented the two headline methods. Plus a dead/deprecated-code sweep.
• BREAKING — 4 deprecated query operators removed. is→eq, isNot→ne, greaterEqual→gte, lessEqual→lte. The canonical operators and their clean long-form aliases (equals/notEquals/greaterThan/greaterThanOrEqual/lessThan/lessThanOrEqual) are unchanged — only the four redundant spellings are gone. Find/replace if you used them.
• find() search mode is now the single searchMode: SearchMode option (the redundant, silently-ignored mode alias and the unwired explain flag were removed from FindParams).
• The phantom index-integrity guard (find() re-validates every result against its predicate) shipped in rc.3; rc.4 added the accel.isInitialized gate + #35 at-gen candidate vectors.

rc.1 additions (this entry predates them — full writeup lands at GA): entity-id normalization — brain.newId() (UUID v7) + v7 default ids, and non-UUID string ids are transparently normalized to a stable UUID v5 (original key preserved under _originalId); brain.neural() clustering and Db.search() removed (use find({ vector }) / find() + aggregation GROUP BY); storage config collapsed to one path key (the pre-8.0 aliases now throw); and queryAggregate no longer hangs/staled min-max after a delete. Reserved fields in a metadata bag now throw by default — passing a Brainy-reserved key (confidence, weight, subtype, visibility, service, createdBy, noun/verb, data, createdAt, updatedAt, _rev) inside metadata on add()/update()/relate()/updateRelation() (untyped/JS callers — TypeScript already blocks it) is rejected with an Error naming the correct param, instead of the old silent remap-or-drop. Pass reserved values as their dedicated top-level params. Opt back into the legacy behavior with new Brainy({ reservedFieldPolicy: 'warn' | 'remap' }).

rc.2 additions (2026-06-21):

• Graph performance + the brain.graph namespace. New brain.graph.subgraph(seeds, opts) (bounded multi-hop neighborhood → { nodes, edges, truncated }) and brain.graph.export(opts) (stream the whole graph in one O(N+E) pass — the right primitive for visualizing all data instead of paging per node). related({ node }) returns every edge incident to an entity in both directions in one O(degree) call. Under the hood: related({ from/to }) now stays O(degree) under the default visibility filter (was a full scan), and the verb and noun pagination walks are cursor-based, so a full edge/node walk is O(N), not O(N²) (a consumer measured a 19k-edge whole-graph read drop from ~27s toward a single scan). These transparently use a native graph engine when present (the @soulcraft/cor 3.0 acceleration layer) and fall back to pure-TS adjacency otherwise.
• Additive ergonomics: add({ upsert: true }) (create-or-update in one call — merges into an existing id instead of overwriting), find({ includeVectors: true }), and removeMany's batch size is now storage-adaptive (was a hardcoded 10).
• Correctness fixes (apply to all consumers, not just graph users): related() results now carry visibility; whole-graph/getNouns streaming no longer leaks system/internal entities; and small-page cursor walks over nouns no longer loop. No API change — just correct behavior.

rc.3 additions (2026-06-23):

• Graph analytics on the brain.graph namespace. Three intent-level reads that answer whole-graph questions in one call:
  • brain.graph.rank(opts?) → { id, score }[] descending — "which entities matter most" (importance / centrality). topK to cap.
  • brain.graph.communities(opts?) → { groups: string[][], count } — "which things group together". Weakly-connected components by default; { directed: true } returns strongly-connected components.
  • brain.graph.path(from, to, opts?) → { nodes, relationships, cost } | null — the best route between two entities. Fewest hops by default; { by: 'weight' } minimizes summed edge weight (the 0–1 connection strength); direction, type, and maxDepth filters apply. These are intent contracts, not algorithm contracts — the question is the promise, the algorithm is the engine's choice. They transparently use the native @soulcraft/cor 3.0 graph engine when present, and fall back to pure-TS kernels (PageRank, connected components / Tarjan SCC, BFS / Dijkstra) otherwise. Both paths return identical shapes and respect the default visibility filter (opt in with includeInternal / includeSystem).
• Filtered vector search keeps its recall (allowedIds pushdown). A find({ query, where }) that combines semantic search with a metadata filter now restricts the vector walk to the matching candidates inside the search (walk-all, collect-allowed) instead of filtering the top-k afterward — so a query whose nearest vectors are all filtered out still returns the best matches that DO pass the filter, rather than coming back empty. No API change. With the native @soulcraft/cor 3.0 stack the matched universe is forwarded as an opaque roaring buffer with zero id materialization in TypeScript; the pure-JS path restricts the beam walk with a string set.
• Historical (asOf) semantic search can skip the rebuild. A filtered semantic query at a past generation — db.asOf(g).find({ query, where }) — no longer always rebuilds an ephemeral in-memory vector index over every at-g vector (O(n@G)). When a native versioned vector engine is registered and can serve the pinned generation, Brainy resolves the at-g filter universe from its record layer (no rebuild) and routes the vector leg to the engine with the matched ids + the generation; without one it falls back to the materialization (unchanged). The VectorIndexProvider.search options gained an optional generation?: bigint ("omitted = now") to carry this — additive, the built-in index ignores it. No public API change.
• find({ where, orderBy }) bounds the sort to the page. A broad filter + orderBy returning one page no longer materializes every matching sorted id (it produced the full sorted match set, then sliced) — the page bound (offset + limit) is threaded into the column store's top-K sort, so returning 20 rows from a million matches stays a bounded-K heap. No API change; ordering and pagination are unchanged.
• brain.graph.subgraph() accepts a query (query→expand). The seed selector now takes not just entity id(s) but a find() result or a FindParams query — brain.graph.subgraph({ where: { team: 'platform' } }, { depth: 1 }) runs the query and expands the neighborhood of every match in one call. With the native engine a metadata-only query's matched universe is handed to the traversal as an opaque set with no id materialization in TypeScript (the query→expand fusion); the pure-JS path materializes the matched ids and expands from them. Existing id-seeded calls are unchanged.

Headline: Database as a Value

8.0 replaces Brainy's two overlapping version-control subsystems (copy-on-write branching and per-entity versioning, ~5,100 LOC combined) with one mechanism: generational MVCC over immutable, generation-stamped records, exposed through a Datomic-style immutable database value — the Db.

const db = brain.now()                  // pin the current state — O(1), no I/O

await brain.transact([
  { op: 'update', id: invoiceId, metadata: { status: 'paid' } }
], { meta: { author: 'billing-service', reason: 'PO-7741' } })

await db.get(invoiceId)                 // still 'pending' — pinned, forever
await brain.get(invoiceId)              // 'paid' — live
await db.release()                      // unpin when done

What you get:

• brain.now() — pins the current generation as an immutable Db view. True snapshot isolation: the view reads exactly its pinned state no matter what commits afterwards, including deletes. Readers never block writers and writers never block readers.
• brain.transact(ops, { meta, ifAtGeneration }) — atomic multi-write batches (add / update / remove / relate / unrelate) committed as exactly one generation. Either every operation applies or none do. ifAtGeneration is whole-store compare-and-swap (GenerationConflictError on conflict) — the big sibling of the per-entity ifRev CAS from 7.31. meta is reified Datomic-style into an append-only transaction log, readable via brain.transactionLog().
• brain.asOf(generation | Date | snapshotPath, { exclusive? }) — time travel with the full query surface: get(), find() in every mode, semantic search, graph traversal, cursors, aggregation, all at the pinned past state. exclusive: true pins the generation immediately before the target (strict-before).
• db.with(ops) — speculative writes applied in memory on top of a view. Nothing touches disk, the generation counter, or the indexes. What-if analysis, then transact() the same ops for real.
• db.persist(path) — instant self-contained snapshots. On filesystem storage they are built from hard links (no entity data is copied; later writes to the source can never alter the snapshot because rewrites swap inodes). brain.restore(path, { confirm: true }) replaces the whole store from one; Brainy.load(path) opens one read-only with the full query surface.
• Range verbs over history — answer "what happened BETWEEN two points":
  • db.since(Db | generation | Date) — the entity and relationship ids committed transactions touched after an exclusive lower bound (now accepts a generation or Date, not just a prior Db).
  • brain.diff(a, b) — the touched ids CLASSIFIED as { added, removed, modified } (split by nouns/verbs) by resolving each at both endpoints; a touched-but-reverted id lands in none of the buckets. Endpoints are a generation, Date, or Db, in either order.
  • brain.history(id, { from?, to? }) — every distinct version of ONE entity/relationship over a range, oldest first (value: null marks a removal); each version equals asOf(version.generation).get(id).
  • brain.transactionLog({ from?, to?, limit? }) — the commit log over an inclusive generation/Date window (contrast since's exclusive lower bound); limit applies last.
• Every write is versioned (Model-B). History granularity is per-write: EVERY write — transact() AND a single-operation add/update/remove/ relate — is its own immutable generation. A now() pin always freezes against later writes, and asOf/since/diff/history/transactionLog reflect single-ops exactly like transacts. transact() groups several operations into ONE atomic generation. Single-op history durability is async group-commit (the live write is acknowledged immediately; its before-image is batched to disk in one fsync) — a hard crash can lose only the last un-flushed window's history, never live data, and a crash mid-flush is recovered by drop-without-restore. A freshly-initialized brain is generation() === 0 with an empty transactionLog() (init-time infrastructure is the un-versioned baseline); the first user write is gen 1.
• new Brainy({ retention }) — the retention knob governs auto-compaction on flush()/close(): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config; a coordinator can drive it via brain.setRetentionBudget(bytes)) · 'all' → unbounded · { maxGenerations?, maxAge?, maxBytes? } → explicit caps (reclaim oldest-unpinned while ANY cap is exceeded). Live pins are ALWAYS exempt.
• brain.compactHistory({ maxGenerations?, maxAge?, maxBytes? }) — manual reclaim on the same caps. Compaction never breaks a pinned read. diff/since throw GenerationCompactedError below the horizon; history truncates to it.

The precise guarantees are documented in:

• docs/concepts/consistency-model.md — the guarantees, each proven by a dedicated test in tests/integration/db-mvcc.test.ts
• docs/guides/snapshots-and-time-travel.md — the recipes: backup, restore, time-travel debugging, what-if, audit trails
• docs/ADR-001-generational-mvcc.md — the design record: persisted layout, commit protocol, crash recovery, proof table

Portable export & import (PortableGraph v1)

A portable, versioned graph format that serializes part or all of a brain to a single JSON document and restores it — the cross-environment, cross-version (7.x↔8.0), partial-or-whole companion to the native persist() snapshot.

// Export is a method on the immutable Db, so it composes with now()/asOf()/with()
const graph = await brain.export({ collection: id }, { includeVectors: true })
await otherBrain.import(graph, { onConflict: 'merge' })  // dedup-by-id

;(await brain.asOf(gen)).export(sel)        // time-travel export
brain.now().with(ops).export(sel)           // what-if export

• brain.export(selector?, options?) / db.export(...) → a versioned PortableGraph document. Selectors reuse find()'s grammar: { ids }, { collection } (alias memberOf, transitive Contains), { connected: { from, depth } }, { vfsPath }, predicate ({ type, subtype, where, service }), or the whole brain (omit) — and they compose. Options: includeVectors, includeContent (VFS file bytes), includeSystem, edges: 'induced' | 'incident' | 'none'.
• brain.import(graph, options?) is polymorphic: a PortableGraph document is restored as one atomic transaction (onConflict: 'merge' | 'replace' | 'skip', reembed: 'auto' | 'never', remapIds for cloning); a file/buffer routes to the existing CSV/PDF/Excel/JSON ingestion. No migration for ingestion callers.
• validatePortableGraph(data) — dry-run structural/version/endpoint check before import.
• Format (format:'brainy-portable-graph', formatVersion: 1) is identical on 7.x and 8.0; standard fields (subtype/visibility/data/…) are top-level, metadata is custom-only. Current-state (no generation history — that lives in persist()). Types exported from the package root.
• Naming: the type is PortableGraph (with PortableGraphEntity / PortableGraphRelation) — it is the portable interchange form of a graph, not a backup (that role is persist()/load()). Renamed from the short-lived BackupData/'brainy-backup' (introduced in 7.32.0, never adopted) with no compatibility shim.
• Guide: docs/guides/export-and-import.md.

Aggregation: distinctCount over any value type

distinctCount now counts distinct values of any type (strings, booleans, numbers) — previously it silently returned 0 for non-numeric fields, missing its primary use (distinct categories / users / tags). percentile (with p; median = p: 0.5), stddev, and variance are exact and delete-safe alongside the write-time sum/count/avg/min/max.

Removed surfaces and their replacements

Removed in 8.0	Replacement
brain.fork(name)	Speculation: db.with(ops) (in-memory). Long-lived writable copy: brain.restore() a persisted snapshot into a fresh data directory.
brain.checkout(branch)	Open the snapshot you want — Brainy.load(path) read-only, or restore into its own directory. No in-place switching: every handle always sees one unambiguous store.
brain.listBranches() / brain.getCurrentBranch() / brain.deleteBranch()	A "branch" is now a name → snapshot-path mapping owned by your application.
brain.commit({ message })	brain.transact(ops, { meta }) — every batch is an atomic, logged, time-travelable commit; audit fields live in the database, not in commit messages.
brain.getHistory() / brain.streamHistory()	brain.transactionLog({ limit }) + db.since(olderDb).
brain.versions.* (save / list / compare / restore / prune)	A pinned Db or persisted snapshot captures every entity at that moment; asOf() reads any entity's past state; restore() for whole-store rollback.
brain.data() (DataAPI: clear / import / export / getStats)	brain.clear(), brain.import(), db.persist(path) (the full-fidelity export format), brain.restore(path, { confirm: true }), brain.stats().
Cloud + browser storage adapters (s3 / gcs / r2 / opfs storage types, Azure Blob, plus the storage.branch option)	filesystem and memory only. Cloud backup is now an operator concern: db.persist() produces a plain directory — sync it with gsutil / aws s3 cp / rclone / azcopy. Passing a removed storage type throws at construction with this exact guidance.
Browser support	Node.js 22 LTS or Bun ≥ 1.0 (engines enforces node: 22.x); Deno works through its Node compatibility layer.
brain.migrateToDiskAnn() / brain.migrateToHnsw()	Gone — there is one canonical 'vector' provider slot. A registered native vector provider selects its own operating mode; there is nothing to call.
Distributed-clustering subsystem — config.distributed, the DistributedRole enum, the 13 BRAINY_* cluster env vars (BRAINY_DISTRIBUTED, BRAINY_ROLE, BRAINY_HTTP_PORT, BRAINY_WS_PORT, BRAINY_DNS, BRAINY_SERVICE, BRAINY_NAMESPACE, BRAINY_CONSENSUS, BRAINY_COORDINATOR, BRAINY_NODES, BRAINY_REPLICAS, BRAINY_SHARDS, BRAINY_TRANSPORT), the storage setDistributedComponents hook, and the @soulcraft/brainy/config preset/augmentation registry	Removed. Brainy 8.0 is a single-process library — there is no coordinator, peer discovery, or consensus to operate. Scale via: the optional native provider (@soulcraft/cor — on-disk DiskANN to 10B+ vectors on one machine); per-tenant pools (one instance + storage dir per tenant); and horizontal read scaling (many reader processes against one shared store, single writer). The mode: 'reader' | 'writer' multi-process roles are unchanged.
CLI fork / branch / checkout / migrate	CLI snapshot <path> / restore <path> / history / generation.
BrainyZeroConfig type	Removed. 8.0 zero-config is automatic and internal — new Brainy() auto-detects storage, derives HNSW quality from config.vector.recall, picks persistMode from the adapter, and sizes caches to the detected container-memory limit. There is no config-generation type to import.
brain.isFullyInitialized() / brain.awaitBackgroundInit()	await brain.ready. The built-in filesystem/memory adapters finish initialization synchronously inside init(), so a single readiness promise is the whole story — these methods were no-ops once cloud adapters were removed.

Opening a 7.x store auto-migrates it. 8.0 stores entities at the root; 7.x stored them branch-scoped under branches/<branch>/. On first open, 8.0 collapses the HEAD branch (config.storage.branch, default main) to the 8.0 layout in place and rebuilds all derived state — no action required (autoMigrate defaults to true; set it to false to make 8.0 refuse a legacy layout with an explicit error instead). Two caveats:

• Non-HEAD branches are not imported. 8.0 has no COW branches; only the head branch's data is migrated. If you care about other branches, export each one while still on 7.x (fork → export, or copy its store).
• Back up first for rollback. The migration mutates the directory in place and 8.0 does not keep the old layout — copy the data directory before the first 8.0 open if you need to roll back.

Renames — find/replace pairs

Hard renames, no aliases. Every pair below is verified against the 8.0 source.

Brainy 7.x	Brainy 8.0
brain.delete(id)	brain.remove(id) — matches the transact op vocabulary (add / update / remove / relate / unrelate)
brain.deleteMany(params)	brain.removeMany(params)
DeleteManyParams (type)	RemoveManyParams
brain.getRelations(paramsOrId)	brain.related(paramsOrId) — the same name a pinned Db exposes (db.related())
GetRelationsParams (type)	RelatedParams
CLI delete <id>	CLI remove <id> (JSON output { id, removed: true }, matching unrelate)
HnswProvider (from @soulcraft/brainy/plugin)	VectorIndexProvider
HNSWIndex (exported class)	JsHnswVectorIndex
Provider keys 'hnsw' and 'diskann'	'vector' (the only key Brainy consults for the vector index)
config.hnsw = { quantization, vectorStorage }	config.vector = { recall?, persistMode? } (shape change — see below)
config.vector.quantization (and 7.x config.hnsw.quantization)	Removed. The JS vector path is full-precision (exact float32 distances) only — quantization at scale is the native provider's job (e.g. DiskANN PQ).
hnswPersistMode: 'immediate' | 'deferred' (top level)	config.vector.persistMode (auto-selected from the storage adapter when omitted: 'immediate' on filesystem, 'deferred' on memory)
config.storage.type: 's3' | 'gcs' | 'r2' | 'opfs'	'filesystem' (or 'memory' / 'auto')
config.storage.branch	Removed (no COW branches)
brain.stats().indexHealth.hnsw	brain.stats().indexHealth.vector
Storage adapter contract saveHNSWData() / getHNSWData()	saveVectorIndexData() / getVectorIndexData()
Cache category 'hnsw' (CacheProvider union)	'vectors'
config.history = { retainGenerations, retainMs, autoCompact }	config.retention — 'all' | 'adaptive' | { maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset → adaptive (was keep-100/7-days). The fields are now caps, not floors.
compactHistory({ retainGenerations, retainMs })	compactHistory({ maxGenerations, maxAge, maxBytes }) — retainGenerations→maxGenerations, retainMs→maxAge (now upper-bound CAPS), plus new maxBytes.
CompactHistoryOptions.retainGenerations / .retainMs	.maxGenerations / .maxAge (+ .maxBytes)

Mechanical renames as one command (run from your repo root, review the diff):

find . -name '*.ts' -not -path '*/node_modules/*' -print0 | xargs -0 sed -i \
  -e 's/\bHnswProvider\b/VectorIndexProvider/g' \
  -e 's/\bHNSWIndex\b/JsHnswVectorIndex/g' \
  -e 's/\bsaveHNSWData\b/saveVectorIndexData/g' \
  -e 's/\bgetHNSWData\b/getVectorIndexData/g' \
  -e 's/indexHealth\.hnsw\b/indexHealth.vector/g' \
  -e "s/registerProvider('hnsw'/registerProvider('vector'/g" \
  -e "s/registerProvider('diskann'/registerProvider('vector'/g" \
  -e "s/getProvider('hnsw'/getProvider('vector'/g" \
  -e "s/getProvider('diskann'/getProvider('vector'/g" \
  -e 's/\.getRelations(/.related(/g' \
  -e 's/\bGetRelationsParams\b/RelatedParams/g' \
  -e 's/\.deleteMany(/.removeMany(/g' \
  -e 's/\bDeleteManyParams\b/RemoveManyParams/g'

delete() → remove() is deliberately not in the snippet: .delete( is too common (Map/Set/storage adapters) for a blind sed. Find your brain.delete(...) call sites and rename them by hand.

The config shape changes need a human (they are not 1:1 textual):

// 7.x
new Brainy({
  hnswPersistMode: 'deferred',
  hnsw: { quantization: { enabled: true, bits: 8, rerankMultiplier: 3 } }
})

// 8.0 — config.vector is { recall?, persistMode? }
new Brainy({
  vector: {
    recall: 'balanced',        // 'fast' | 'balanced' | 'accurate'
    persistMode: 'deferred'    // optional; auto-selected from storage when omitted
  }
})

config.vector.recall replaces the algorithm-internal HNSW knobs: 'balanced' (the default) maps to exactly the 7.x default parameters, so an upgrade without explicit knobs changes nothing about search behaviour. config.vector.quantization and config.hnsw.vectorStorage are removed: the JS vector path now computes exact float32 distances throughout (no rerank/approximate branch), which is what made its quantization a memory increase — it stored both the full and the quantized vectors in RAM. Quantization at scale belongs to the native provider's own PQ. The on-disk vector index file names are unchanged (e.g. hnsw-system.json) — existing filesystem stores need no data migration for this rename; only the API surface moved.

Behavior changes

• subtype is required by default. 7.30's opt-in strict mode is now the default: every add() / addMany() / update() / relate() / relateMany() / updateRelation() rejects writes whose type carries no non-empty subtype (src/brainy.ts:9759 — requireSubtype ?? true). Escape hatches: requireSubtype: false (last-resort opt-out for legacy data) or requireSubtype: { except: [NounType.Thing, ...] } (per-type allowlist). Per-type rules registered via brain.requireSubtype(type, opts) still compose. brain.audit() reports entries missing a subtype and the new brain.fillSubtypes(rules) migration helper backfills them — one rule per NounType/VerbType (literal default or per-entry function), applied only to entries still missing a subtype, returning { scanned, filled, skipped, errors, byType }. Idempotent: re-running fills nothing, so a crashed run is resumed by running it again.
• Verb ids are Brainy-generated UUIDs — by contract. relate() now throws a teaching error if a caller passes an id field (src/utils/paramValidation.ts:526). In 7.x a supplied id was silently ignored (a generated UUID was used anyway); 8.0 says so out loud, because graph-index providers key verb-int interning on the raw UUID bytes.
• update({ vector }) is honored on its own. 7.x only applied a supplied pre-computed vector when data was also passed (it was silently dropped otherwise). 8.0 applies an explicit params.vector with dimension validation (src/brainy.ts:1702-1713; proven by tests/unit/brainy/update.test.ts:256).
• brain.clear() re-resolves all indexes exactly as init() does — including plugin-provided vector/metadata/entityIdMapper factories and VFS root re-creation. In 7.x, clearing a plugin-accelerated brain could leave the metadata index rebuilt without its native id-mapper wiring.
• find({ connected: { …, subtype } }) with depth > 1 now works. 7.30 threw NOT_SUPPORTED for multi-hop subtype-filtered traversal; 8.0 implements the BFS in the JS graph index and routes to a native provider when one is registered.
• Every write advances the generation clock; only transact() writes history. Single-operation writes (add / update / remove / relate outside transact()) bump brain.generation() so watermarks and CAS stay sound, but they do not stage historical records — they remain visible through earlier pins and are not reported by db.since(). Writes you want to travel back through go through transact(). This is the documented contract, stated rather than papered over.
• Reserved fields have one canonical location — enforced at every layer. The Brainy-owned field names (RESERVED_ENTITY_FIELDS: noun/subtype/createdAt/updatedAt/confidence/weight/service/ data/createdBy/_rev; verb mirror RESERVED_RELATION_FIELDS with verb for the type key) are now (1) a compile error inside any metadata param — add/update/relate/updateRelation and the matching transact() ops; (2) normalized at write time for untyped callers — user-settable fields remap to their dedicated top-level param (top-level wins; update({metadata:{confidence}}) no longer silently no-ops, closing a 7.x trap), system-managed fields drop with a one-shot warning naming the right path; (3) split at read time through one canonical helper, so entity.metadata / relation.metadata contain ONLY custom fields on every read path — get, find, related (several paginated/by-source/by-target paths previously echoed the full stored record, including the verb type key, inside metadata), batch reads, and historical asOf() reads. related() results now also surface confidence/updatedAt top-level, and updateRelation() no longer erases a relationship's service/createdBy. See "Reserved fields" in docs/concepts/consistency-model.md.
• Named errors for not-found contract failures. update(), relate(), updateRelation(), similar({ to: id }), transact() planning, and speculative db.with() planning now throw EntityNotFoundError / RelationNotFoundError (both exported from the package root, both carrying the missing id as a field) instead of a generic Error. Message texts are unchanged, so existing /not found/ matching keeps working — instanceof is now the supported way to branch.
• Unchanged from 7.31: per-entity _rev, update({ ifRev }) (RevisionConflictError), and add({ ifAbsent }) work exactly as before, and ifRev is also accepted on transact() update operations (a conflict rejects the whole batch).

For provider and plugin authors

The provider contracts in @soulcraft/brainy/plugin (src/plugin.ts) changed shape:

• GraphIndexProvider speaks BigInt at the boundary. Reads take entity ints and return entity/verb ints as bigint[]; the coordinator owns all UUID ↔ int conversion. addVerb(verb, sourceInt, targetInt) receives the resolved endpoint ints (also mirrored on the new optional GraphVerb.sourceInt / GraphVerb.targetInt fields) and returns the interned verb int. The batch reverse resolver verbIntsToIds(verbInts: bigint[]): Promise<(string | null)[]> is required — the provider owns durable verb-int interning; Brainy keeps only a bounded in-memory warm cache. Implementations may stay u32 internally (Number(bigint) narrowing is lossless under the EntityIdSpaceExceeded guard) but must speak the bigint contract.
• VersionedIndexProvider is a new optional 4-method capability — generation(), isGenerationVisible(g), pin(g), release(g), BigInt generations — feature-detected on every registered index provider. Providers are post-commit appliers: the storage-record commit is the source of truth; on open, a provider behind the committed watermark replays the gap or requests a rebuild. Explicit pins override any time-based snapshot retention the provider has. Speculative db.with() overlays never reach providers. A provider implementing this serves historical reads with no rebuild — the open-core materializer is the correctness baseline, the provider is the accelerator.
• The vector index registers under 'vector' and implements VectorIndexProvider. The 'hnsw' and 'diskann' keys are retired and never looked up. CacheProvider's category union uses 'vectors' in place of 'hnsw'.

Scale and cost — how the mechanisms behave

Mechanism descriptions, not benchmark numbers (none are published for 8.0 yet):

• brain.now() pins in O(1) and adds zero read overhead until history actually moves — while nothing has committed past the pin, every read delegates to the live fast paths.
• A transact() commit pays O(ids touched) extra writes (before-images + delta + manifest) — never O(store size). Single-operation writes pay an in-memory counter bump with coalesced persistence.
• db.persist() on filesystem storage is a hard-link farm: snapshot creation copies no entity data and the snapshot shares disk space with the source. Cross-device targets fall back to per-file byte copies; persisting an in-memory brain serializes a real, durable, loadable store.
• Historical index-accelerated queries (semantic search, traversal, cursors, aggregation) on the open-core path pay a one-time O(n at the pinned generation) in-memory index materialization per Db, cached until release(). Record-path reads (get, metadata find, filter related) at any reachable generation are served directly from the immutable record layer. A native VersionedIndexProvider serves the same reads rebuild-free.

The full cost model is in docs/concepts/consistency-model.md.

Upgrade checklist (from 7.x)

1. While still on 7.x: back up your data directory (a plain file copy is fine). If you used COW branches, materialize each branch you need — 8.0 does not read branch state. If you ran a cloud storage adapter, export your data with 7.x tooling ((await brain.data()).export()) — 8.0 opens filesystem and memory stores only.
2. Runtime: Node.js 22 LTS or Bun ≥ 1.0.
3. Config: change removed storage types to 'filesystem'; delete storage.branch; move config.hnsw / hnswPersistMode to config.vector per the shape example above.
4. Renames: run the sed snippet, then fix the manual shape changes.
5. Removed APIs: replace fork / checkout / branch methods / commit / getHistory / streamHistory / versions / data() per the table above.
6. Subtypes: if your data predates subtype discipline, start with requireSubtype: false, run brain.audit(), backfill with brain.fillSubtypes(rules) (one rule per type — a literal default or a function deriving the subtype from each entry), re-run audit() until total === 0, then remove the opt-out so the 8.0 default enforcement protects you going forward.
7. CLI scripts: fork / branch / checkout / migrate → snapshot <path> / restore <path> / history / generation.
8. Plugin authors: apply the contract renames and the BigInt graph contract; optionally implement VersionedIndexProvider.
9. Verify, then take your first snapshot: run your test suite, then const db = brain.now(); await db.persist('/backups/post-8.0-upgrade'); await db.release().

What did not change: the core query surface (find / search / get / add / update / relate and friends), the aggregation engine, the VFS, neural extraction, and the on-disk entity/relationship layout — 8.0 creates its MVCC bookkeeping (_system/generation.json, _system/manifest.json, _system/tx-log.jsonl, _generations/) alongside the existing files on first use.

Native-provider (Cor) compatibility

Brainy 8.0 pairs with @soulcraft/cor 3.0 — the renamed successor to the @soulcraft/cortex 2.x native provider — shipping in lockstep. The old @soulcraft/cortex 2.x cannot accelerate Brainy 8.0: 8.0 never consults the 'hnsw' / 'diskann' provider keys, and the graph/column provider contracts are BigInt at the boundary. Upgrade both packages together (@soulcraft/brainy@8

• @soulcraft/cor@3).

---

v7.31.2 — 2026-06-09

Affected products: none in behaviour; affects anyone reading Brainy's TypeScript type definitions or coreTypes for the hnsw.quantization.bits option. Drop-in from 7.31.1.

Fix: stale comment claimed SQ4 vector quantization required a paid native provider

Brainy ships a pure-JS SQ4 distance implementation (distanceSQ4Js in src/utils/vectorQuantization.ts) that's been part of the open-core path the whole time. Two type-definition comments incorrectly stated SQ4 required a native (paid) provider:

// coreTypes.ts:472 + types/brainy.types.ts:1123 — before
bits?: 8 | 4   // default: 8 (SQ8). SQ4 requires cortex native.

The comment was simply wrong — open-core users can use SQ4 today, and the native SIMD acceleration is a drop-in via setSQ4DistanceImplementation, not a hard requirement. Corrected to:

bits?: 8 | 4   // default: 8 (SQ8). SQ4 has a pure-JS implementation;
               // cortex's distance:sq4 SIMD provider accelerates it.

Both locations updated. Pure documentation; no behaviour change. Caught during an upstream open-core-boundary audit — thanks to whoever read the types carefully enough to spot the misleading comment.

Cortex compatibility

Zero changes required. The fix is documentation only; runtime behaviour was already correct (vectorQuantization.ts:412 ships distanceSQ4Js).

---

v7.31.1 — 2026-06-09

Affected products: any consumer running @soulcraft/brainy against FileSystemStorage that triggers concurrent flush / compaction (e.g. explicit brain.flush() calls overlapping with periodic background compaction, or a backup job + a flush job racing). Production-impacting when present: brain.flush() throws ENOENT on rename, downstream jobs that rely on flush (GCS / S3 / Azure backups, snapshot exports) fail continuously. Drop-in from 7.31.0.

Fixes a same-key rename race in saveBinaryBlob

FileSystemStorage.saveBinaryBlob used a bare ${filePath}.tmp suffix for its atomic-write temp file. Two concurrent same-key calls computed the same temp path; both writeFiled, the first rename succeeded, the second rename fired against a missing temp and threw ENOENT. The throw propagated up through brain.flush() and broke any downstream job that called it.

[job-queue] gcs-backup: failed — ENOENT: no such file or directory,
  rename '/data/brainy-data/.../_column_index/owner/DELETED.bin.tmp'
       -> '/data/brainy-data/.../_column_index/owner/DELETED.bin'

Patch: unique per-writer temp suffix (matches the pattern at six other atomic-write sites in the same file) + ENOENT swallow on rename (defensive: if the temp is gone, the work has already landed; saveBinaryBlob is idempotent for a given key) + temp cleanup on any other rename failure (no orphan .tmp.* files).

// before (7.31.0)
const tmpPath = filePath + '.tmp'
await fs.promises.writeFile(tmpPath, data)
await fs.promises.rename(tmpPath, filePath)

// after (7.31.1)
const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}`
await fs.promises.writeFile(tmpPath, data)
try {
  await fs.promises.rename(tmpPath, filePath)
} catch (err) {
  if ((err as NodeJS.ErrnoException).code === 'ENOENT') return
  await fs.promises.unlink(tmpPath).catch(() => {})
  throw err
}

Scope

The fix is one site — src/storage/adapters/fileSystemStorage.ts:992-999. Audit results:

• FileSystemStorage — six sibling atomic-write sites already used unique suffixes; only saveBinaryBlob was the outlier. All six were rechecked. Clean.
• OPFSStorage — writes via WritableStream (no tmp+rename). Not affected.
• GCSStorage / R2Storage / AzureBlobStorage / S3CompatibleStorage — use object-store PUT (atomic at the API). Not affected.
• MemoryStorage — in-memory. Not affected.
• HistoricalStorageAdapter — read-only. Not affected.
• COW / versioning / snapshot / HNSW / aggregation — all delegate to storage adapters via saveBinaryBlob / writeObjectToPath. They get the fix automatically by virtue of using the patched primitive.

Beneficiary surfaces (broader than the reported bug)

Column-store compaction (_column_index/<field>/DELETED.bin, segment files) is what the production report named, but saveBinaryBlob is also used by HNSW connection persistence (_hnsw_conn/<nodeId> — src/hnsw/hnswIndex.ts:252). If two flush paths ever raced on HNSW persistence for the same node, they would have hit the same ENOENT. No production reports of HNSW-side failures, but the fix removes the latent race.

Tests

• New tests/integration/savebinaryblob-concurrent-rename.test.ts (4 tests):
  • 20 concurrent saveBinaryBlob(sameKey, ...) calls all resolve without throwing
  • No orphan .tmp.* siblings remain in the blob directory after concurrent writes
  • Production-shape: two concurrent compactor passes over 10 column-index fields (owner, path, permissions, vfsType, modified, createdAt, accessed, updatedAt, mimeType, size)
  • Single-writer path still produces correct bytes (no regression)
• The first three tests reproduce the production ENOENT: rename '...DELETED.bin.tmp' -> '...DELETED.bin' error verbatim on the pre-7.31.1 code path. Verified by stashing the fix and watching the tests fail with the exact production error.
• Existing suites unchanged: 1468/1468 unit. All integration suites pass.

Cortex compatibility

Zero changes required. The bug and the fix are pure JS in the filesystem storage adapter; the Cortex mmap-filesystem adapter wraps FileSystemStorage and inherits the patch automatically.

Forward-compat

The bare-.tmp pattern is gone repo-wide. 8.0's Db.persist() will route through the same patched primitive; no additional work needed when the immutable Db API ships.

---

v7.31.0 — 2026-06-09

Affected products: consumers that need multi-writer coordination (job schedulers, idempotent state machines, distributed locks, optimistic UI updates) or idempotent bootstrap of well-known singletons. Additive; drop-in from 7.30.2.

Per-entity _rev + update({ ifRev }) + add({ ifAbsent })

CouchDB / PouchDB / ETag-style optimistic concurrency. Every entity now carries a monotonic _rev: number that Brainy auto-bumps on every successful update(). Pass it back as ifRev to make read-modify-write race-safe without an external lock service. ifAbsent adds by-ID idempotent insert for singletons / config rows / deterministic-ID bootstraps.

What's new:

• entity._rev: number — initialized to 1 on add(), bumped by 1 on every successful update() that reaches storage. Surfaced on get() (both fast metadata- only and full-vector paths), find(), search(), and on the Result flatten layer for backward compatibility. Pre-7.31.0 entities without _rev are read as 1.
• update({ id, ..., ifRev: number }) — optimistic-concurrency check. When provided, throws RevisionConflictError if the persisted _rev no longer matches. Carries { id, expected, actual } for principled recovery. Omitting ifRev keeps the prior unconditional-update behavior.
• add({ id, ifAbsent: true }) — by-ID idempotent insert. Returns the existing id without writing if the entity already exists. No throw, no overwrite. Ignored when id is omitted (a fresh UUID can never collide).
• addMany({ items, ifAbsent: true }) — applies the flag to every item; per-item ifAbsent overrides the batch flag.

The lock recipe (single-process or distributed)

import { Brainy, RevisionConflictError } from '@soulcraft/brainy'

const LOCK_ID = '...uuid...'

await brain.add({
  id: LOCK_ID,
  type: NounType.Document,
  data: { owner: null, expiresAt: 0 },
  ifAbsent: true
})

async function tryAcquireLock(workerId: string, ttlMs: number) {
  const lock = await brain.get(LOCK_ID)
  if (!lock) throw new Error('lock document missing')
  const state = lock.data as { owner: string | null; expiresAt: number }
  if (state.owner && state.expiresAt > Date.now()) return false

  try {
    await brain.update({
      id: LOCK_ID,
      data: { owner: workerId, expiresAt: Date.now() + ttlMs },
      ifRev: lock._rev
    })
    return true
  } catch (err) {
    if (err instanceof RevisionConflictError) return false
    throw err
  }
}

Why this is scoped to _rev + ifAbsent

A public brain.transaction(fn) wrapper was on the table but was cut. The internal TransactionManager exposes raw Operation classes that take a StorageAdapter constructor argument; a high-level facade in 7.31.0 would have meant either (a) delegating to brain.add() / update() / relate() — each of which commits its own internal transaction before the closure returns, so multi-write rollback wouldn't actually work (a footgun), or (b) threading tx? through every internal write path — ~2 days of refactor with regression surface, and the API shape changes again in 8.0 anyway.

The SDK-scheduler use case that drove the thread (read-then-CAS lock pattern) is fully solved by _rev + ifRev. Multi-write atomicity is the 8.0 brain.transact() use case, where the Datomic-style immutable Db makes it atomic by construction. We chose not to ship a worse version in the interim.

How _rev interacts with branches and the snapshot API

System	What it tracks	When it advances
_rev (NEW)	Per-entity write counter	Auto, on every successful update()
brain.versions.save()	Named snapshots per entity	Explicit — you call save()
brain.fork() / branches	Whole-brain copy-on-write	Explicit — you call fork(name)
VFS file versioning	Per-VFS-file snapshots	Same as brain.versions.save()

_rev is independent. Each branch has its own copy of every entity, so each branch has its own _rev per entity (same as every other field under COW). Snapshots taken via brain.versions.save() capture the entity at that moment including its _rev at that time; the snapshot's own version: number is the snapshot index, distinct from _rev.

Docs

• New docs/guides/optimistic-concurrency.md (public, indexed at soulcraft.com/docs after portal deploy) — full reference: the lock pattern, read-modify-write retry, idempotent bootstrap, how _rev interacts with branches / snapshots / VFS, what's coming in 8.0.
• docs/api/README.md add() and update() entries get the new params + tips pointing at the guide.

Tests

• New tests/integration/rev-and-ifabsent.test.ts (18 tests): rev initialization, rev bump on update, ifRev pass / fail / omit / legacy-entity-fallback, ifAbsent with custom id / no id / batch propagation / per-item override, plus a full SDK-scheduler-style two-concurrent-CAS-updaters scenario.
• Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-enforcement 30/30, strict-mode-self-test 13/13, find-limits 9/9. Unit 1468/1468.

Cortex compatibility

Zero changes required. _rev is a metadata column already supported by NativeColumnStore; the auto-bump runs in Brainy JS before any storage call. Cortex never sees the per-entity counter — it's a single integer field updated alongside every other metadata field on the existing write paths.

8.0 forward-compat

_rev, ifRev, RevisionConflictError, and ifAbsent all survive the 8.0 Db redesign unchanged. 8.0 layers brain.transact(tx, { ifAtGeneration }) for whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record patterns, generation-based for "did the world move under me."

---

v7.30.2 — 2026-06-08

Affected products: any consumer with find({ limit }) call sites that pass values ≥ ~9000 — a common safety-cap pattern in production code (limit: 10_000 against type-filtered queries that typically return 10–500 entities). Additive; drop-in from 7.30.1.

Why

Brainy 7.30 introduced a memory-derived cap on find({ limit }) to prevent OOM from runaway queries. The cap was sound in intent but ~4× too conservative in calibration: the formula assumed 100 KB per result while typical entity footprint is 7-10 KB (384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB free-memory box this capped limit at 9000, breaking production dashboards where cascading 500s from queries with limit: 10_000 silently degraded the Promise.all loading the canvas.

7.30.2 recalibrates the formula and changes the enforcement from synchronous-throw to two-tier (warn-then-throw) so existing pre-7.30 code doesn't break on upgrade.

Recalibrated formula — 25 KB per result (was 100 KB)

The auto-configured cap now uses a realistic per-result size:

Memory source	7.30.0–7.30.1 cap	7.30.2 cap
4 GB Cloud Run container	10 000	40 000
2 GB container	5 000	20 000
900 MB free system memory	9 000	~36 000
reservedQueryMemory: 1 GB	10 000	40 000

Hard ceiling stays at 100 000. Existing BrainyConfig.maxQueryLimit / reservedQueryMemory overrides continue to work unchanged.

Two-tier enforcement — warn, then throw

Below cap (limit ≤ maxLimit): silent pass. No signal, no friction. Unchanged.

Soft tier (maxLimit < limit ≤ 2 × maxLimit) NEW: one-time warning per call site (dedup keyed on stack location + limit value), query proceeds. Pre-7.30.2 code that relied on the cap silently allowing safety-cap limits keeps working — the warning teaches the recipe so consumers can fix it intentionally.

Hard tier (limit > 2 × maxLimit) NEW: throw with the same message format the warning uses. Real OOM territory; the cap stops being a recommendation and becomes a guardrail.

The 2× soft margin is chosen to absorb existing safety-cap patterns (limit: 10_000 against a 9 K-cap box) without disabling OOM protection. Real OOM territory on a JS in-memory brain is hundreds of thousands of results, not 10× the safety cap.

Improved error / warning message (mirrors the 7.30.1 enforcement-error format)

Before (7.30 → 7.30.1):

limit exceeds auto-configured maximum of 9000 (based on available memory)

After (7.30.2):

find({ limit: 10000 }) exceeds the auto-configured query limit of 9000 (basis:
available free memory). Choose one:
  • Increase the cap: new Brainy({ maxQueryLimit: 10000 })
  • Reserve more memory: new Brainy({ reservedQueryMemory: 256000000 })
  • Paginate: split the query with { limit, offset } pages
  at OrderService.loadDashboard (/app/src/orders/dashboard.ts:142:18)
Docs: https://soulcraft.com/docs/guides/find-limits

Caller location comes from the same findCallerLocation() helper introduced in 7.30.1 (now extracted to src/utils/callerLocation.ts so both subtype enforcement and limit enforcement share it).

New docs

New docs/guides/subtypes-and-facets.md-style guide at docs/guides/find-limits.md (public, indexed) — explains the cap, the four memory sources the auto-config considers, the three escape valves (maxQueryLimit, reservedQueryMemory, pagination), and when to use which. Explicit "pagination is the future-proof pattern" callout — the cap can get tighter in 8.0 (Datomic-style Db.find() may make per-call limits stricter to keep snapshot semantics cheap), but pagination keeps working unchanged.

docs/api/README.md find() entry gets a one-paragraph limit tip + pointer to the new guide.

Tests

• New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass; soft-tier warns once per call site (dedup verified by exercising same vs. different source lines); soft-tier message format (names all three escape valves + docs link); soft-tier message includes caller location; hard-tier throws; hard-tier message format same as soft-tier; consumer maxQueryLimit override raises the cap and shifts both tiers accordingly; the pre-7.30.2 regression scenario explicitly covered — limit: 10_000 against a maxQueryLimit: 9000 cap now warns and passes instead of throwing.
• Existing unit tests in tests/unit/utils/memoryLimits.test.ts updated to reflect the recalibrated formula (5 tests: hardcoded values for container / reserved / free-memory paths bumped 4×).
• Existing paramValidation.test.ts "should auto-limit based on system memory" test extended to cover the new three-tier semantics (below-cap pass / soft-tier silent / hard-tier throw).
• All prior suites still green: subtype-and-facets 26/26, verb-subtype-and- enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.

Cortex compatibility

No Cortex changes required for 7.30.2. Every change is JS-side: formula recalibration runs in ValidationConfig.constructor(), two-tier enforcement runs in validateFindParams(), both fire before any storage/index/Cortex call. Cortex 2.x and the pending Cortex 3.0 work both stay compatible without code changes.

The new docs/guides/find-limits.md calls out that 8.0's Datomic-style Db.find() may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point whose rollout staging now also covers query limits.

What consumers should do

• If you've been carrying a limit: 9000 workaround for the 7.30.0–7.30.1 cap: drop it on bump to 7.30.2 — existing limit: 10_000 patterns now warn (teaching the recipe) but no longer throw. The warning is one-time-per-call-site, so production logs don't spam.
• All other consumers: no action required if no enforcement-error was being hit. If you see new [Brainy] warnings in logs after upgrade, follow the recipe in the warning or read docs/guides/find-limits.md.
• SDK wrappers: wrapper-level pools that auto-construct Brainy instances should consider surfacing maxQueryLimit / reservedQueryMemory as consumer-facing config; Brainy now documents the knob explicitly.

---

v7.30.1 — 2026-06-08

Affected products: anyone running a brain where a platform layer (SDK, framework wrapper) has registered brain.requireSubtype() rules on common NounTypes, AND anyone preparing for the upcoming Brainy 8.0 default-on strict mode. Additive; drop-in from 7.30.0. No behavior change for consumers not using strict-mode enforcement.

Why

Production incident 2026-06-08: a consumer using SDK 3.20.0 (which registers requireSubtype() rules on NounType.{Event, Collection, Message, Contract, Media, Document}) saw their booking flow start returning 500s because brain.add({ type: NounType.Event, ... }) calls in their codebase lacked subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write paths that also omit subtype — VFS move/copy/symlink edges, aggregation materializer, neural extraction, importers, integrations (Sheets/OData), MCP client, CLI. Any consumer running requireSubtype() rules on those NounTypes was one step away from breaking Brainy's own infrastructure paths, not just their own code. 7.30.1 closes both gaps before 8.0 ships and makes strict mode the default.

New — brain.audit() diagnostic

Find entities and relationships missing a subtype value, grouped by type. The companion to migrateField() (and to 8.0's fillSubtypes()): answers "what would break if I enabled strict subtype enforcement?".

const report = await brain.audit()
// {
//   entitiesWithoutSubtype: { event: 24, document: 3 },
//   relationshipsWithoutSubtype: { relatedTo: 1402 },
//   total: 1429,
//   scanned: 8400,
//   recommendation: 'Found 1429 entries without subtype. Migrate via `brain.migrateField()`
//                    (7.x) — or wait for `brain.fillSubtypes()` (8.0) which closes the same
//                    gap with caller-supplied rules.'
// }

VFS infrastructure entities are excluded by default (they bypass enforcement via metadata.isVFSEntity markers). Pass { includeVFS: true } to surface them.

New — Improved enforcement error messages

The error fired when subtype enforcement rejects a write now includes:

1. The caller's source location — extracted from the JavaScript stack so you see your own call site, not a Brainy internal frame. Eliminates the "grep your repo for brain.add" step.
2. Specific guidance — points at the registered vocabulary when one exists; mentions brain-wide strict mode and the except escape valve when not; otherwise the brain.requireSubtype() registration recipe.
3. A documentation link — https://soulcraft.com/docs/guides/subtypes-and-facets#strict-mode for the canonical migration recipe.

Before:

add(): NounType.Event requires subtype but got undefined. Register vocabulary via brain.requireSubtype().

After:

add(): NounType.event requires subtype but got undefined.
  at OrderService.getOrCreateByToken (/app/src/orders/draft.ts:42:23)
  Pass one of: standard, recurring, milestone.
  Migration recipe: https://soulcraft.com/docs/guides/subtypes-and-facets#strict-mode

Internal subtype labels — Brainy's own infrastructure paths

Every internal Brainy write path now sets a stable, queryable subtype. Consumers don't need to do anything for these — they're documented here so you can query Brainy-managed data:

Code path	NounType / VerbType	Subtype label
VFS root directory /	Collection	'vfs-root'
VFS subdirectories	Collection	'vfs-directory'
VFS files	mime-driven (e.g. Document/Code/Image)	'vfs-file'
VFS symlinks	File	'vfs-symlink' (NEW — distinct from 'vfs-file')
VFS Contains edges (create + move/copy/symlink/batch)	Contains	'vfs-contains'
Aggregation materialized output	Measurement	'materialized-aggregate'
Import-source provenance entity	Document	'import-source'
Importer-extracted entities	extractor-driven type	'imported'
Importer placeholder targets	Thing	'import-placeholder'
Neural extraction	extractor-driven type	'extracted'
GoogleSheets API entity writes	request-driven	'imported-from-sheets'
OData API entity writes	request-driven	'imported-from-odata'
MCP message storage	Message	'mcp-message'
brainy add CLI default	user-supplied type	'cli-add'
brainy relate CLI default	user-supplied verb	'cli-relate'

Query examples:

// Every VFS-managed file in your brain
await brain.find({ subtype: 'vfs-file' })

// Document breakdown — distinguishes import-source from extracted/imported/user content
brain.counts.bySubtype(NounType.Document)
// → { 'import-source': 12, 'imported': 847, 'extracted': 34, 'vfs-file': 102, ... }

Caller-supplied defaultSubtype on importers and extraction

Importer + extraction paths now accept a caller-supplied defaultSubtype config so consumers can tag a whole batch with their own provenance label instead of the Brainy default:

// SmartImportOrchestrator / ImportCoordinator / NeuralImport all accept this
await brain.importer.import(file, {
  defaultSubtype: 'customer-upload-2026q2',  // your batch label
  // ...
})

Precedence: extractor-set subtype (highest) → caller's defaultSubtype → Brainy default ('imported' for importers, 'extracted' for extractors).

CLI --subtype flag

brainy add and brainy relate gain a --subtype <value> (-s) flag for use with strict-mode brains:

brainy add "Avery Brooks — runs the AI lab" --type person --subtype employee
brainy relate alice manages bob --subtype direct

When the flag isn't supplied, the CLI uses 'cli-add' / 'cli-relate' as defaults so ad-hoc CLI usage still works against strict-mode brains.

Cortex compatibility

No Cortex changes required for 7.30.1. Every change is JS-side: internal subtype labels are arbitrary strings stored transparently by Cortex; brain.audit() runs purely on JS via existing storage.getNouns() / getVerbs() pagination; the CLI is JS-only; error messages fire in Brainy before any native call.

For Cortex 3.0 (forward-looking, not blocking):

• Native audit() proxy. For billion-scale brains, audit() walks every entity (O(N)). A native implementation reading from a "null-subtype" bitmap in the column store would be O(buckets).
• Strict-mode parity test. Cortex should mirror Brainy's new tests/integration/strict-mode-self-test.test.ts against their native paths to catch any latent bug where native writes bypass JS validation.
• Reserved-label awareness (optional). Brainy's internal labels ('vfs-*', 'materialized-aggregate', 'imported', 'extracted', 'mcp-message', 'cli-*') become a documented part of the 8.0 contract; useful for telemetry that surfaces "X% of entities are Brainy-managed infrastructure".

Docs

Updated docs/guides/subtypes-and-facets.md with a new "Strict mode in practice" section covering the SDK_CORE_VOCABULARY pattern, a 4-step migration recipe, the Brainy-internal label reference table, and an 8.0 forward-look. docs/api/README.md documents brain.audit() and adds a strict-mode tip to the add() / relate() reference entries.

Tests

• New tests/integration/strict-mode-self-test.test.ts (13 tests): creates a brain under a realistic consumer vocabulary shape + brain-wide strict mode, then exercises every internal Brainy path (VFS root/mkdir/writeFile/cp/mv/ln, aggregation engine, audit diagnostic, error-message UX). Zero rejections expected.
• Existing 7.30.0 + 7.29.0 integration suites unchanged: 26/26 + 30/30.
• Unit suite unchanged: 1468/1468.

---

v7.30.0 — 2026-06-05

Affected products: consumers modeling typed relationships with sub-classification (direct vs dotted-line management; spouse / sibling / colleague; collaborator vs competitor; etc.), and anyone wanting to enforce the pairing of type + subtype on every write. Additive; drop-in from 7.29.x. No deprecations.

Symmetric — subtype on relationships (parity with 7.29.0 nouns)

subtype?: string is now a first-class standard field on every relationship, mirroring the noun-side work shipped in 7.29.0. Verbs and nouns are now first-class peers — every API available on the noun side has a verb-side mirror.

const ceoId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Avery' })
const vpId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Jordan' })

await brain.relate({
  from: ceoId,
  to: vpId,
  type: VerbType.ReportsTo,
  subtype: 'direct'                    // sub-classification on the edge
})

Read & filter:

// Fast-path filter — column-store hit, not metadata fallback
const direct = await brain.getRelations({
  from: ceoId,
  type: VerbType.ReportsTo,
  subtype: 'direct'
})

// Set membership
const all = await brain.getRelations({
  from: ceoId,
  type: VerbType.ReportsTo,
  subtype: ['direct', 'dotted-line']
})

// Traversal filter (depth-1 in JS; multi-hop lands on Cortex native)
const reports = await brain.find({
  connected: { from: ceoId, via: VerbType.ReportsTo, subtype: 'direct', depth: 1 }
})

New — updateRelation() closes a long-standing gap

Verbs previously had no update method — the only way to change a relationship was delete-then-recreate, which lost the relation id. 7.30 ships brain.updateRelation():

await brain.updateRelation({ id: relId, subtype: 'dotted-line' })
await brain.updateRelation({ id: relId, weight: 0.5, confidence: 0.9 })

// Change verb type — re-indexes in graph adjacency, id preserved
await brain.updateRelation({ id: relId, type: VerbType.WorksWith })

New — O(1) verb subtype counts via the persisted rollup

_system/verb-subtype-statistics.json mirrors the noun-side rollup shipped in 7.29.0. Per-VerbType-per-subtype counts are maintained incrementally and persisted; reads are O(1):

brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
// → { direct: 12, 'dotted-line': 3 }

brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct')   // O(1) point
// → 12

brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
// → [['direct', 12], ['dotted-line', 3]]

brain.relationshipSubtypesOf(VerbType.ReportsTo)
// → ['direct', 'dotted-line']

New — brain.requireSubtype(type, options) per-type enforcement

Unified API for noun OR verb types — register specific types as requiring a subtype, optionally with a fixed vocabulary:

brain.requireSubtype(NounType.Person, {
  values: ['employee', 'customer', 'vendor'],
  required: true
})

brain.requireSubtype(VerbType.ReportsTo, {
  values: ['direct', 'dotted-line'],
  required: true
})

// Now this throws — Person requires subtype:
await brain.add({ type: NounType.Person, data: 'no subtype' })

// And this throws — 'matrix' isn't in the registered vocabulary:
await brain.relate({ from: a, to: b, type: VerbType.ReportsTo, subtype: 'matrix' })

New — Brain-wide strict mode

new Brainy({ requireSubtype: true }) enforces subtype on every public write across the whole brain. Composes with per-type rules; per-type rules win when both apply.

// Every write must include subtype
const brain = new Brainy({ requireSubtype: true })

// Exempt specific types (e.g. catch-all Thing)
const brain2 = new Brainy({
  requireSubtype: { except: [NounType.Thing, NounType.Custom] }
})

When strict mode is on:

• Every add() / addMany() / update() / relate() / relateMany() / updateRelation() rejects writes missing a subtype on a non-exempt type.
• addMany() and relateMany() validate every item BEFORE any storage write — atomic-fail semantics, no partial writes.
• Brainy's own infrastructure writes (VFS root, directories, files) bypass via the metadata.isVFSEntity: true marker so existing consumers' VFS usage continues to work.

The brain-wide flag becomes the default in 8.0.0; the type-level subtype field becomes required at the type system level. The full 8.0 contract upgrade is coordinated through the internal platform handoff (CTX-SUBTYPE-8.0-CONTRACT).

migrateField() extended to verbs

The migration helper shipped in 7.29.0 now walks verbs too via the new entityKind option:

// Migrate verb-side metadata.kind → top-level subtype
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  entityKind: 'verb'
})

// Or walk nouns and verbs in one pass
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  entityKind: 'both'
})

Default is entityKind: 'noun' (backward-compatible).

Symmetry — noun + verb capability matrix

7.30 closes every gap between nouns and verbs. Every capability available on the noun side has a verb-side mirror:

Capability	Nouns	Verbs
subtype top-level field	✓ (7.29)	✓ (7.30 new)
Standard-field set	STANDARD_ENTITY_FIELDS	STANDARD_VERB_FIELDS (new)
Field resolver helper	resolveEntityField	resolveVerbField (new)
Statistics rollup	_system/subtype-statistics.json	_system/verb-subtype-statistics.json (new)
Counts breakdown	counts.bySubtype / topSubtypes / subtypesOf	counts.byRelationshipSubtype / topRelationshipSubtypes / relationshipSubtypesOf (new)
Fast-path filter	find({type, subtype})	getRelations({type, subtype}) + find({connected, subtype}) (new)
Update method	update()	updateRelation() (new — closed pre-7.30 gap)
Migration helper	migrateField()	migrateField({entityKind: 'verb'|'both'}) (new)
Enforcement	requireSubtype(NounType, ...)	requireSubtype(VerbType, ...) — one unified API
Brain-wide strict mode	new Brainy({ requireSubtype }) covers both	same

Cortex compatibility

Verb subtype works under Cortex out of the box via auto-field-indexing. Cortex parity items for the verb side (native query planner recognition, verbSubtypeCountsByType native rollup, per-edge subtype on native graph adjacency for multi-hop traversal filtering) ship in the next Cortex release. Not a Brainy blocker.

Docs

Full guide: docs/guides/subtypes-and-facets.md (extended with Layer V + Enforcement sections). The verb subtype + enforcement APIs are documented in docs/api/README.md.

---

v7.29.0 — 2026-06-04

Affected products: anyone modeling entities with per-product sub-classification — every consumer that's been reaching for metadata.kind, a custom metadata.subtype field, a data.kind shape inside the payload, or similar ad-hoc conventions. Additive; drop-in from 7.28.x. No deprecations.

New — subtype promoted to a top-level standard field

subtype?: string is now a first-class standard field on every entity, alongside type / confidence / weight. Use it as the platform-standard primitive for sub-classifying entities within a NounType (Person → 'employee' / 'customer'; Document → 'invoice' / 'contract'; Event → 'milestone' / 'meeting'):

await brain.add({
  data: 'Avery Brooks — runs the AI lab',
  type: NounType.Person,
  subtype: 'employee',                  // top-level write param
  metadata: { department: 'ai-lab' }
})

// Fast-path filter — column-store hit, not metadata fallback:
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })

// Set membership:
const internal = await brain.find({
  type: NounType.Person,
  subtype: ['employee', 'contractor']
})

Flat string, no hierarchy — your vocabulary, your choice. Brainy stores and counts, never validates.

New — O(1) subtype counts via the persisted rollup

A new _system/subtype-statistics.json rollup is maintained incrementally as entities are added, updated, and deleted — mirroring the existing nounCountsByType machinery with the same self-heal behavior on poison detection. Counts are O(1) at billion scale:

brain.counts.bySubtype(NounType.Person)
// → { employee: 12, customer: 847, vendor: 34 }

brain.counts.bySubtype(NounType.Person, 'employee')   // O(1) point count
// → 12

brain.counts.topSubtypes(NounType.Person, 3)
// → [['customer', 847], ['employee', 12], ['vendor', 34]]

brain.subtypesOf(NounType.Person)
// → ['customer', 'employee', 'vendor']

New — brain.trackField() for other metadata facets

For facets that aren't the primary sub-classification (status, source, role, paradigm), trackField registers a field for cardinality + per-NounType breakdown stats without promoting each one to a top-level slot. Piggybacks on the existing aggregation engine, so backfill-on-define applies — registering on a populated brain scans existing entities on the first query:

brain.trackField('status', { perType: true })

await brain.counts.byField('status')
// → { todo: 12, doing: 3, done: 47 }

await brain.counts.byField('status', { type: NounType.Task })
// → { todo: 8, doing: 2, done: 30 }

// Opt-in vocabulary validation:
brain.trackField('priority', { values: ['low', 'medium', 'high'] })
// brain.add({ ..., metadata: { priority: 'urgent' } }) → throws

New — generic brain.migrateField() for one-shot rewrites

Streams every entity and copies a value from one field path to another. Supports top-level standard fields, metadata.*, and data.* paths; idempotent; with optional readBoth: true deprecation window that preserves the source field alongside the new one:

// Phase 1: dual-populate (legacy readers still work)
await brain.migrateField({
  from: 'metadata.kind',
  to: 'subtype',
  readBoth: true
})

// ...readers migrate to subtype at their own pace...

// Phase 2: clear the source field
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
// → { scanned: 1500, migrated: 1500, skipped: 0, errors: [] }

Supports batchSize and onProgress for large brains.

Aggregation composition

subtype is a standard-field group-by dimension out of the box — no where: wrapper, no metadata-fallback overhead:

await brain.find({
  aggregate: {
    groupBy: ['type', 'subtype'],
    metrics: { count: { op: 'COUNT' } }
  }
})
// [
//   { groupKey: { type: 'person',   subtype: 'customer' }, count: 847 },
//   { groupKey: { type: 'person',   subtype: 'employee' }, count: 12 },
//   { groupKey: { type: 'document', subtype: 'invoice'  }, count: 2103 },
//   ...
// ]

Cortex compatibility

Subtype works under Cortex out of the box via auto-field-indexing. Cortex will land its own native-side query-planner recognition + subtypeCountsByType native rollup in the next Cortex release for full parity with type's fast path. Not a Brainy blocker — subtype reads/writes function correctly with current Cortex.

Docs

Full guide: docs/guides/subtypes-and-facets.md. Subtype is documented as a core primitive throughout README.md, docs/DATA_MODEL.md, docs/architecture/finite-type-system.md, docs/api/README.md, and docs/QUERY_OPERATORS.md.

---

v7.24.0 — 2026-05-26

Affected products: aggregation/reporting users + anyone calling extractEntities on multi-entity text. Additive; drop-in from 7.23.x.

New — array-unnest groupBy (tag frequency / faceted counts)

A groupBy dimension can now be { field, unnest: true }: the field holds an array and the entity contributes once per distinct element. Enables tag-frequency / label-count style aggregates:

brain.defineAggregate({
  name: 'tag_frequency',
  source: { type: NounType.Document },
  groupBy: [{ field: 'tags', unnest: true }],
  metrics: { count: { op: 'count' } }
})
// queryAggregate('tag_frequency', { orderBy: 'count', order: 'desc' })

Duplicate elements on one entity count once; an entity with an empty/missing array joins no group.

Perf — entity extraction batch-embeds candidates

extractEntities / extractConcepts now embed all unique candidate spans in a single embedBatch call instead of one embed() per candidate (N sequential model calls before). No behavior change — and with vectors reliably available, the embedding signal more consistently reinforces correct types (e.g. clearer Organization confidence). Falls back to per-candidate embedding if a batch call fails.

---

v7.23.0 — 2026-05-26

Affected products: anyone using aggregation (stats/dashboards), graph traversal, or entity extraction. Adds two report APIs and fixes three correctness gaps surfaced by BR-ADV-FEATURES-BUN. Drop-in upgrade from 7.22.x.

New — brain.queryAggregate(name, params)

A first-class report API returning the clean AggregateResult[] shape ({ groupKey, metrics, count }[]) directly, instead of the search-Result wrapper that find({ aggregate }) returns. Accepts where / having / orderBy / order / limit / offset.

New — HAVING (filter groups by metric value)

find({ aggregate, having: { revenue: { greaterThan: 1000 } } }) (and the same on queryAggregate) filters groups by their computed metrics — the analytics equivalent of SQL HAVING, complementing where (which filters group keys). Evaluated per group: O(groups), independent of entity count.

Fix — aggregates now backfill when defined over existing data (R1)

Defining an aggregate on a store that already holds matching entities returned [] because write-time hooks only saw future writes. It now backfills from existing entities on first query (one-time scan via getNouns, then incremental) — storage-agnostic, so it works under durable backends (Cortex) where a brain reopens pre-populated. Also: groupBy:['noun'] now resolves to the entity type instead of a single null group. find({ aggregate }) rows now expose groupKey/metrics/count at the top level (previously only under .metadata).

Fix — multi-hop find({ connected: { depth, via } }) (traversal)

depth and via are now honored at every hop (previously only the immediate neighbour was returned, and verb filtering applied to hop 1 only). The BFS is bounded by limit.

Fix — entity extraction type accuracy (R3)

extractEntities no longer lets a type indicator in one candidate's surrounding text bleed onto neighbours (e.g. "Corp" in "Sarah Chen founded Acme Corp" no longer types "Sarah Chen" as Organization). Each candidate is typed by its own span; context may only reinforce the same type.

---

v7.22.1 — 2026-05-26

Affected products: Anyone using extractEntities() / extractConcepts(), native aggregation (find({ aggregate })), or multi-hop graph traversal (find({ connected: { depth } })). Three advanced-API correctness fixes — all reproduce on Node, so they were not Bun-specific despite the original report (BR-ADV-FEATURES-BUN).

Entity/concept extraction no longer returns []

extractEntities() / extractConcepts() returned empty for entity-rich text. The SmartExtractor ensemble scored agreeing signals with a weighted sum against an absolute 0.60 gate, so a confident low-weight signal (e.g. a name pattern at 0.82) lost selection to a mediocre high-weight signal that then failed the gate — dropping the whole result. It now selects and gates on a normalized weighted average. Also: the confidence option now actually controls the threshold (was a dead hardcoded 0.60), and the embedding-signal timeout was raised 100ms → 2000ms so the neural signal isn't silently dropped on slower runtimes.

Known limitation: type accuracy on dense multi-entity sentences is still imperfect — a strong indicator in one entity's surrounding text can bleed onto neighbours. Tracked separately.

find({ aggregate }) exposes groupKey / metrics / count

Aggregation always computed correct values, but rows nested them under .metadata, so callers expecting the documented AggregateResult shape saw empty-looking rows. Those three fields are now also present at the top level of each result row.

Multi-hop find({ connected: { depth } }) honours depth

Previously returned only the immediate (1-hop) neighbour at any depth because the graph-search path ignored depth / via. It now performs the full depth-aware traversal.

---

v7.22.0 — 2026-05-15

Affected products: All. Fixes a silent-data-loss class affecting find() and brain.stats(), plus Cortex-compatibility regression from 7.21.0. Recommended upgrade for everyone on 7.20.x or 7.21.x.

Two correctness fixes

1. find({ where }) no longer silently returns [] (BR-FIND-WHERE-ZERO)

The 7.20.0 column-store refactor deleted the legacy sparse-index write path but left getStats() and getIds() reading from sparse indices. New workspaces had no sparse-index files, so:

• brain.stats().entityCount was 0 regardless of actual entity count
• find({ where: { ... } }) returned [] regardless of actual matches
• rebuildIndexesIfNeeded() saw 0 entries → fired the [Brainy] CRITICAL ... Second rebuild result: 0 entries log → application saw no indexed data

7.22.0 makes the ColumnStore the single source of truth post-7.20.0:

• MetadataIndex.getStats() reads from ColumnStore.getIndexedFields() and idMapper.size. Entity count is now accurate across writer → close → reader-open cycles.
• MetadataIndex.getIdsFromChunks() throws BrainyError(FIELD_NOT_INDEXED) when neither column store nor legacy sparse index has the field.
• MetadataIndex.getIdsForFilter() catches per-clause, logs [brainy] find() where-clause referenced unindexed field(s): ..., returns []. Production find() is now consistent with brain.explain()'s path: 'none' diagnostic.

Separately, BaseStorage.getNounType() was hardcoded to return 'thing' since a type-cache removal in commit 42ae5be. Every noun save attributed the entity to 'thing' regardless of its actual type, poisoning _system/type-statistics.json and brain.stats().entitiesByType. 7.22.0 restores type-correct attribution:

• New nounTypeByIdCache: Map<string, NounType> on BaseStorage, populated in saveNounMetadata_internal and consumed in saveNoun_internal.
• New flushCounts() override that persists nounCountsByType / verbCountsByType alongside the per-entity counter. Readers opening the same directory after a writer flush now see correct per-type counts.

Self-heal at init. If loadTypeStatistics() detects the poisoned-state signature (all nouns attributed to 'thing' with multiple non-thing entities on disk), it auto-runs rebuildTypeCounts() once and rewrites the file. A [BaseStorage] Detected poisoned type-statistics.json signature ... warning is logged. Operators don't need to take any action — the first 7.22 open of a directory poisoned by 7.20.0/7.21.0 fixes it.

brainy inspect repair can also be used to manually trigger rebuildTypeCounts(). It's now public on BaseStorage.

2. Cortex 2.2.x no longer crashes 7.21.0 boot (BR-DEFENSIVE-INTERFACE)

7.21.0 added new storage-adapter methods (supportsMultiProcessLocking, acquireWriterLock, etc.) and called them unconditionally. Cortex 2.2.0's mmap storage adapter bundles a pre-7.21 BaseStorage and doesn't define them, so a consumer's 7.21.0 upgrade attempt threw TypeError: this.storage.supportsMultiProcessLocking is not a function at boot.

7.22.0 introduces hasStorageMethod(name) on Brainy and guards every new- method call site. Older adapters degrade with a one-line warning at init:

[brainy] Storage adapter `CortexMmapStorage` predates the 7.21
multi-process methods. Writer locking and the flush-request RPC are
disabled for this directory. Upgrade the plugin (e.g.
`@soulcraft/cortex` ≥2.3.0) for full enforcement.

Dead-code cleanup

Removed MetadataIndexManager.dirtyChunks, dirtySparseIndices, and the flushDirtyMetadata() no-op machinery. These accumulators stopped being populated when the sparse-index write path was deleted in 7.20.0; the remaining 6 callers wasted async hops on an empty Map iteration.

Upgrade notes

• Behaviour change: find({ where: { unindexedField: ... } }) previously returned [] silently. It now logs a one-line warning and still returns []. The diagnostic message names the field — use brain.explain({ where: { ... } }) for full diagnosis. No code change needed for callers that already handle empty results.
• Behaviour change (good): brain.stats() and brain.health() now report accurate per-type counts. If you previously relied on the buggy entitiesByType.thing over-count, update consumers.
• No API breaks. Pure additive on the public surface.

Cortex consumers

Cortex 2.2.x continues to work against 7.22.0 thanks to the defensive guards, but for full multi-process protection on Cortex-backed stores and to make sure Cortex's native MetadataIndex picks up the find-where-zero fix, Cortex 2.3.0 is recommended. Tracked as CX-MULTIPROC-METHOD in the internal platform handoff.

---

v7.21.0 — 2026-05-15

Affected products: All consumers using filesystem storage (production deploys, local development).

Multi-process safety + first-class read-only inspector

Filesystem storage now enforces single-writer, many-reader. Previously, opening a second writer process against a live data directory silently produced wrong query results — no error, no warning, just empty or stale data. This release replaces that footgun with a hard refusal at init time and a dedicated read-only inspection mode.

New: Brainy.openReadOnly()

const reader = await Brainy.openReadOnly({
  storage: { type: 'filesystem', rootDirectory: '/data/brain' }
})
const bookings = await reader.find({ where: { entityType: 'booking' } })

• Does NOT acquire the writer lock — coexists with a live writer.
• Every mutation method throws Cannot mutate a read-only Brainy instance.
• flush() and close() are safe (no-op + clean shutdown).

New: writer lock at init

• new Brainy({ ... }) (default mode: 'writer') acquires <rootDir>/locks/_writer.lock containing { pid, hostname, startedAt, lastHeartbeat, version }.
• A second writer on the same directory throws with the PID, hostname, heartbeat, and a pointer to openReadOnly().
• Stale-lock detection: same hostname + dead PID OR heartbeat > 60s ago → overwritten with a warning.
• Heartbeat: writer rewrites lastHeartbeat every 10s (unref'd timer).
• Override: pass { force: true } to bypass when you've verified the existing lock is stale.

New: brain.requestFlush({ timeoutMs })

Cross-process RPC for inspectors to force a fresh snapshot:

• In-process: just calls flush().
• Out-of-process: writes a request file, polls for ack. Times out gracefully.
• Watcher runs in every writer instance (filesystem only).

New: brain.stats()

Operator-facing summary: entityCount, entitiesByType, relationCount, relationsByType, fieldRegistry, indexHealth, storage.backend, writerLock, version. Designed for /api/health endpoints and incident triage.

New: brain.explain(findParams)

Shows which index path serves each where clause: column-store | sparse-chunked | none. This is the answer to "why is find() returning empty?" — path: "none" means the field has no index entries and the query will silently return []. Includes notes on likely causes (writer hasn't flushed; typo; field genuinely absent).

New: brain.health()

Invariant-check battery: HNSW vs metadata count parity, field registry sanity, _seeded entity sweep, writer heartbeat freshness. Returns { overall: 'pass' | 'warn' | 'fail', checks: [...] }.

New: brainy inspect CLI (13 subcommands)

All read-only by default (internally uses openReadOnly()):

brainy inspect stats   <path>
brainy inspect find    <path> --type Event --where '{"status":"paid"}' --limit 20
brainy inspect get     <path> <id>
brainy inspect relations <path> <id> --direction both
brainy inspect explain <path> --where '{"entityType":"booking"}'
brainy inspect health  <path>
brainy inspect sample  <path> --type Event --n 20
brainy inspect fields  <path>
brainy inspect dump    <path> --type Event > backup.jsonl
brainy inspect watch   <path> --type Event
brainy inspect backup  <path> /backups/brain.tar
brainy inspect repair  <path>           # writer mode — stop live writer first
brainy inspect diff    <pathA> <pathB>

Default behaviour: ask the writer to flush via the RPC first (skip with --no-fresh).

Brainy + Cortex

Cortex segments (*.cidx) are immutable mmap files; MANIFEST.json uses atomic rename. The single writer lock at <rootDir>/locks/_writer.lock covers both Brainy and Cortex. Read-only inspectors mmap Cortex segments with zero coordination.

What's NOT enforced yet

• Cloud storage backends (S3, GCS, R2, Azure) — no multi-process locking. A best-effort warning logs in writer mode against non-filesystem backends.
• The find({ where })-returns-0 root-cause bug — tracked separately. The new brain.explain() and brain.health() surface it loudly (path: 'none', index-parity warn) instead of letting it be silent.

Upgrade notes

• Default behaviour change: opening a Brainy directory in writer mode while another writer is live now THROWS where it previously silently produced wrong results. If you have scripts that intentionally co-opened a writer directory, switch them to Brainy.openReadOnly() or pass { force: true }.
• Cloud backends unchanged — no lock acquisition for S3/GCS/R2/Azure.
• All existing APIs unchanged. Pure addition.

Docs

• README "Single-Writer Model" section.
• docs/concepts/multi-process.md — full model + Cortex compatibility.
• docs/guides/inspection.md — operator recipes.
• JSDoc on every new API.

---

v7.19.10 — 2026-02-24

Affected products: All Bun/ESM consumers

ESM crypto fix in SSTable

Replaced require('crypto') with import { createHash } from 'node:crypto' in the SSTable implementation. Fixes a crash in Bun and strict ESM environments where CommonJS require is unavailable.

No API changes — upgrade and redeploy.

---

v7.19.2 — 2026-02-18

Affected products: All

Metadata index cleanup on delete

Fixed: metadata indexes were not cleaned up after delete() / deleteMany(). Stale index entries could cause phantom results in metadata-filtered queries after deletion.

No API changes. If you were seeing ghost results in filtered queries, this fixes it.

---

v7.18.0 — 2026-02-16

Affected products: Analytics, reporting, and session-summary consumers

Aggregation engine

New brain.aggregate() API — incremental SUM, COUNT, AVG, MIN, MAX with GROUP BY and time window support. Computes over entity collections without loading all records into memory.

const result = await brain.aggregate({
  collection: 'bookings',
  metrics: [
    { field: 'revenue', fn: 'SUM' },
    { field: 'id', fn: 'COUNT' },
  ],
  groupBy: 'staffId',
  timeWindow: { field: 'createdAt', from: startOfMonth, to: now },
})

SDK exposure: sdk.brainy.aggregate() — available once SDK is updated to pass through.

---

v7.17.0 — 2026-02-09

Affected products: All (schema evolution, data migrations)

Migration system

New brain.migrate() API with error handling, validation, and enterprise hardening. Run schema migrations reliably across Brainy data directories.

await brain.migrate({
  version: 3,
  up: async (brain) => {
    // transform entities, rename fields, etc.
  },
})

---

v7.16.0 — 2026-02-09

Affected products: All

Data/metadata separation enforced + numeric range queries

• Entity data and metadata fields are now strictly separated at the storage layer
• Numeric range queries now supported in metadata filters: { age: { $gte: 18, $lt: 65 } }
• Fixes edge cases where mixed data/metadata storage caused inconsistent query results

Breaking for anyone storing numeric values in metadata and relying on range queries: verify your filter syntax matches the new $gte/$lte/$gt/$lt operators.

---

v7.15.5 — 2026-02-02

Affected products: Anyone using @soulcraft/cortex plugin

Plugin opt-in clarified

Cortex and other plugins are opt-in. Pass explicitly:

new Brainy({ plugins: ['@soulcraft/cortex'] })

Without plugins, no external plugins are loaded regardless of what's installed.

---

v7.15.2 — 2026-02-01

Affected products: All (data safety)

Graph LSM flush on close

Fixed: graph LSM-trees were not flushed on brain.close(), risking data loss across restarts. Graph edges written in the final seconds before shutdown are now guaranteed to be persisted.

No API changes — upgrade immediately if running Brainy in a long-lived server process.