2026-02-28 10:07:34 -08:00
# @soulcraft/brainy — Release Notes for Consumers
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
This file is the **quick reference for downstream sessions** tracking Brainy changes.
2026-02-28 10:07:34 -08:00
Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/soulcraftlabs/brainy/releases
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
**How to use:** Brainy is the underlying data engine for downstream applications. Read this when:
- Upgrading `@soulcraft/brainy` in your application
2026-02-28 10:07:34 -08:00
- Debugging data, query, or storage behaviour
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
- A new Brainy feature is available that you want to adopt
---
2026-06-20 14:55:16 -07:00
## v8.0.0 — release candidate (`8.0.0-rc.1` published 2026-06-20 on npm tag `rc`)
2026-06-11 09:31:07 -07:00
**Affected products:** every consumer — this is a major release with removed
surfaces, hard renames (no aliases, no deprecation period), and one flipped
default. This entry **is** the migration guide: the find/replace pairs, the
sed snippet, and the upgrade checklist below are the complete story — there is
2026-06-20 14:55:16 -07:00
no separate migration doc. ** `8.0.0-rc.1` is live** — install with
`npm i @soulcraft/brainy@rc` to try it; the `latest` tag stays on 7.x until GA.
> **rc.1 additions (this entry predates them — full writeup lands at GA):** entity-id
> normalization — `brain.newId()` (UUID v7) + v7 default ids, and non-UUID string ids are
> transparently normalized to a stable UUID v5 (original key preserved under `_originalId`);
> `brain.neural()` clustering and `Db.search()` removed (use `find({ vector })` /
> `find()` + aggregation `GROUP BY`); storage config collapsed to one `path` key (the
> pre-8.0 aliases now throw); and `queryAggregate` no longer hangs/staled min-max after a delete.
2026-06-11 09:31:07 -07:00
### Headline: Database as a Value
8.0 replaces Brainy's two overlapping version-control subsystems (copy-on-write
branching and per-entity versioning, ~5,100 LOC combined) with **one
mechanism**: generational MVCC over immutable, generation-stamped records,
exposed through a Datomic-style immutable database value — the ** `Db` **.
```ts
const db = brain.now() // pin the current state — O(1), no I/O
await brain.transact([
{ op: 'update', id: invoiceId, metadata: { status: 'paid' } }
], { meta: { author: 'billing-service', reason: 'PO-7741' } })
await db.get(invoiceId) // still 'pending' — pinned, forever
await brain.get(invoiceId) // 'paid' — live
await db.release() // unpin when done
```
What you get:
- **`brain.now()` ** — pins the current generation as an immutable `Db` view.
True snapshot isolation: the view reads exactly its pinned state no matter
what commits afterwards, including deletes. Readers never block writers and
writers never block readers.
- **`brain.transact(ops, { meta, ifAtGeneration })` ** — atomic multi-write
batches (`add` / `update` / `remove` / `relate` / `unrelate` ) committed as
exactly one generation. Either every operation applies or none do.
`ifAtGeneration` is whole-store compare-and-swap (`GenerationConflictError`
on conflict) — the big sibling of the per-entity `ifRev` CAS from 7.31.
`meta` is reified Datomic-style into an append-only transaction log,
readable via `brain.transactionLog()` .
feat(8.0): temporal range verbs — diff, history, since(gen|Date), asOf{exclusive}, transactionLog window
asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.
- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
chokepoint (reachability + Date semantics identical everywhere). asOf()'s
inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
existence at both endpoints + a key-order-insensitive value compare
(new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
value === asOf(version.generation).get(id); null = removal; kind auto-detected
(throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
(contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
horizon; history TRUNCATES to it.
Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
2026-06-19 13:21:02 -07:00
- **`brain.asOf(generation | Date | snapshotPath, { exclusive? })` ** — time
travel with the **full query surface** : `get()` , `find()` in every mode,
semantic search, graph traversal, cursors, aggregation, all at the pinned past
state. `exclusive: true` pins the generation immediately before the target
(strict-before).
2026-06-11 09:31:07 -07:00
- **`db.with(ops)` ** — speculative writes applied in memory on top of a view.
Nothing touches disk, the generation counter, or the indexes. What-if
analysis, then `transact()` the same ops for real.
- **`db.persist(path)` ** — instant self-contained snapshots. On filesystem
storage they are built from hard links (no entity data is copied; later
writes to the source can never alter the snapshot because rewrites swap
inodes). `brain.restore(path, { confirm: true })` replaces the whole store
from one; `Brainy.load(path)` opens one read-only with the full query
surface.
feat(8.0): temporal range verbs — diff, history, since(gen|Date), asOf{exclusive}, transactionLog window
asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.
- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
chokepoint (reachability + Date semantics identical everywhere). asOf()'s
inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
existence at both endpoints + a key-order-insensitive value compare
(new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
value === asOf(version.generation).get(id); null = removal; kind auto-detected
(throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
(contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
horizon; history TRUNCATES to it.
Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
2026-06-19 13:21:02 -07:00
- **Range verbs over history** — answer "what happened BETWEEN two points":
- **`db.since(Db | generation | Date)` ** — the entity and relationship ids
committed transactions touched after an **exclusive** lower bound (now
accepts a generation or `Date` , not just a prior `Db` ).
- **`brain.diff(a, b)` ** — the touched ids CLASSIFIED as `{ added, removed,
modified }` (split by nouns/verbs) by resolving each at both endpoints; a
touched-but-reverted id lands in none of the buckets. Endpoints are a
generation, `Date` , or `Db` , in either order.
- **`brain.history(id, { from?, to? })` ** — every distinct version of ONE
entity/relationship over a range, oldest first (`value: null` marks a
removal); each version equals `asOf(version.generation).get(id)` .
- **`brain.transactionLog({ from?, to?, limit? })` ** — the commit log over an
**inclusive** generation/`Date` window (contrast `since` 's exclusive lower
bound); `limit` applies last.
2026-06-11 09:31:07 -07:00
- **`brain.compactHistory({ retainGenerations, retainMs })` ** — explicit
feat(8.0): temporal range verbs — diff, history, since(gen|Date), asOf{exclusive}, transactionLog window
asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.
- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
chokepoint (reachability + Date semantics identical everywhere). asOf()'s
inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
existence at both endpoints + a key-order-insensitive value compare
(new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
value === asOf(version.generation).get(id); null = removal; kind auto-detected
(throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
(contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
horizon; history TRUNCATES to it.
Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
2026-06-19 13:21:02 -07:00
retention. Compaction never breaks a pinned read. `diff` /`since` throw
`GenerationCompactedError` below the horizon; `history` truncates to it.
2026-06-11 09:31:07 -07:00
The precise guarantees (and the honest limits — history granularity is
`transact()` commits; single-operation writes advance the clock but produce no
historical records) are documented in:
- [docs/concepts/consistency-model.md ](docs/concepts/consistency-model.md ) —
the guarantees, each proven by a dedicated test in
`tests/integration/db-mvcc.test.ts`
- [docs/guides/snapshots-and-time-travel.md ](docs/guides/snapshots-and-time-travel.md )
— the recipes: backup, restore, time-travel debugging, what-if, audit trails
- [docs/ADR-001-generational-mvcc.md ](docs/ADR-001-generational-mvcc.md ) — the
design record: persisted layout, commit protocol, crash recovery, proof table
refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:09:19 -07:00
### Portable export & import (PortableGraph v1)
2026-06-17 12:05:00 -07:00
A portable, versioned graph format that serializes part or all of a brain to a
single JSON document and restores it — the cross-environment, cross-version
(7.x↔8.0), partial-or-whole companion to the native `persist()` snapshot.
```ts
// Export is a method on the immutable Db, so it composes with now()/asOf()/with()
refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:09:19 -07:00
const graph = await brain.export({ collection: id }, { includeVectors: true })
await otherBrain.import(graph, { onConflict: 'merge' }) // dedup-by-id
2026-06-17 12:05:00 -07:00
;(await brain.asOf(gen)).export(sel) // time-travel export
brain.now().with(ops).export(sel) // what-if export
```
- **`brain.export(selector?, options?)` / `db.export(...)` ** → a versioned
refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:09:19 -07:00
`PortableGraph` document. Selectors reuse `find()` 's grammar: `{ ids }` ,
2026-06-17 12:05:00 -07:00
`{ collection }` (alias `memberOf` , transitive `Contains` ),
`{ connected: { from, depth } }` , `{ vfsPath }` , predicate
(`{ type, subtype, where, service }` ), or the whole brain (omit) — and they
compose. Options: `includeVectors` , `includeContent` (VFS file bytes),
`includeSystem` , `edges: 'induced' | 'incident' | 'none'` .
refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:09:19 -07:00
- **`brain.import(graph, options?)` ** is polymorphic: a `PortableGraph` document is
2026-06-17 12:05:00 -07:00
restored as **one atomic transaction** (`onConflict: 'merge' | 'replace' | 'skip'` ,
`reembed: 'auto' | 'never'` , `remapIds` for cloning); a file/buffer routes to the
existing CSV/PDF/Excel/JSON ingestion. No migration for ingestion callers.
refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:09:19 -07:00
- **`validatePortableGraph(data)` ** — dry-run structural/version/endpoint check before import.
- **Format** (`format:'brainy-portable-graph'` , `formatVersion: 1` ) is identical on
7.x and 8.0; standard fields (`subtype` /`visibility` /`data` /…) are top-level,
2026-06-17 12:05:00 -07:00
`metadata` is custom-only. Current-state (no generation history — that lives in
`persist()` ). Types exported from the package root.
refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:09:19 -07:00
- **Naming:** the type is `PortableGraph` (with `PortableGraphEntity` /
`PortableGraphRelation` ) — it is the portable interchange form of a graph, not a
backup (that role is `persist()` /`load()` ). Renamed from the short-lived
`BackupData` /`'brainy-backup'` (introduced in 7.32.0, never adopted) with no
compatibility shim.
2026-06-17 12:05:00 -07:00
- Guide: [docs/guides/export-and-import.md ](docs/guides/export-and-import.md ).
### Aggregation: distinctCount over any value type
`distinctCount` now counts distinct values of **any** type (strings, booleans,
numbers) — previously it silently returned `0` for non-numeric fields, missing its
primary use (distinct categories / users / tags). `percentile` (with `p` ; median =
`p: 0.5` ), `stddev` , and `variance` are exact and delete-safe alongside the
write-time `sum` /`count` /`avg` /`min` /`max` .
2026-06-11 09:31:07 -07:00
### Removed surfaces and their replacements
| Removed in 8.0 | Replacement |
|---|---|
| `brain.fork(name)` | Speculation: `db.with(ops)` (in-memory). Long-lived writable copy: `brain.restore()` a persisted snapshot into a fresh data directory. |
| `brain.checkout(branch)` | Open the snapshot you want — `Brainy.load(path)` read-only, or restore into its own directory. No in-place switching: every handle always sees one unambiguous store. |
| `brain.listBranches()` / `brain.getCurrentBranch()` / `brain.deleteBranch()` | A "branch" is now a name → snapshot-path mapping owned by your application. |
| `brain.commit({ message })` | `brain.transact(ops, { meta })` — every batch is an atomic, logged, time-travelable commit; audit fields live in the database, not in commit messages. |
| `brain.getHistory()` / `brain.streamHistory()` | `brain.transactionLog({ limit })` + `db.since(olderDb)` . |
| `brain.versions.*` (`save` / `list` / `compare` / `restore` / `prune` ) | A pinned `Db` or persisted snapshot captures *every* entity at that moment; `asOf()` reads any entity's past state; `restore()` for whole-store rollback. |
| `brain.data()` (DataAPI: `clear` / `import` / `export` / `getStats` ) | `brain.clear()` , `brain.import()` , `db.persist(path)` (the full-fidelity export format), `brain.restore(path, { confirm: true })` , `brain.stats()` . |
| Cloud + browser storage adapters (`s3` / `gcs` / `r2` / `opfs` storage types, Azure Blob, plus the `storage.branch` option) | `filesystem` and `memory` only. Cloud backup is now an operator concern: `db.persist()` produces a plain directory — sync it with `gsutil` / `aws s3 cp` / `rclone` / `azcopy` . Passing a removed storage type throws at construction with this exact guidance. |
| Browser support | Node.js 22 LTS or Bun ≥ 1.0 (`engines` enforces `node: 22.x` ); Deno works through its Node compatibility layer. |
| `brain.migrateToDiskAnn()` / `brain.migrateToHnsw()` | Gone — there is one canonical `'vector'` provider slot. A registered native vector provider selects its own operating mode; there is nothing to call. |
refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider
The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).
Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
readWriteSeparation, queryPlanner, healthMonitor, configManager,
hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
(slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
src/config/extensibleConfig.ts (config/augmentation registry built on removed
cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
enableDistributedSearch (dead config flag); the metadata partition field;
the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
storage-architecture; reframed scale prose to the 8.0 model.
Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.
RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
2026-06-15 10:37:39 -07:00
| Distributed-clustering subsystem — `config.distributed` , the `DistributedRole` enum, the 13 `BRAINY_*` cluster env vars (`BRAINY_DISTRIBUTED` , `BRAINY_ROLE` , `BRAINY_HTTP_PORT` , `BRAINY_WS_PORT` , `BRAINY_DNS` , `BRAINY_SERVICE` , `BRAINY_NAMESPACE` , `BRAINY_CONSENSUS` , `BRAINY_COORDINATOR` , `BRAINY_NODES` , `BRAINY_REPLICAS` , `BRAINY_SHARDS` , `BRAINY_TRANSPORT` ), the storage `setDistributedComponents` hook, and the `@soulcraft/brainy/config` preset/augmentation registry | Removed. Brainy 8.0 is a single-process library — there is no coordinator, peer discovery, or consensus to operate. Scale via: the optional native provider (`@soulcraft/cortex` — on-disk DiskANN to 10B+ vectors on one machine); per-tenant pools (one instance + storage dir per tenant); and horizontal read scaling (many reader processes against one shared store, single writer). The `mode: 'reader' \| 'writer'` multi-process roles are unchanged. |
2026-06-11 09:31:07 -07:00
| CLI `fork` / `branch` / `checkout` / `migrate` | CLI `snapshot <path>` / `restore <path>` / `history` / `generation` . |
2026-06-15 11:12:26 -07:00
| `BrainyZeroConfig` type | Removed. 8.0 zero-config is automatic and internal — `new Brainy()` auto-detects storage, derives HNSW quality from `config.vector.recall` , picks `persistMode` from the adapter, and sizes caches to the detected container-memory limit. There is no config-generation type to import. |
| `brain.isFullyInitialized()` / `brain.awaitBackgroundInit()` | `await brain.ready` . The built-in filesystem/memory adapters finish initialization synchronously inside `init()` , so a single readiness promise is the whole story — these methods were no-ops once cloud adapters were removed. |
2026-06-11 09:31:07 -07:00
feat(8.0): 7.x→8.0 layout migration — fix silent total data loss on first open
7.x stored every object branch-scoped under branches/<branch>/<basePath>; 8.0
stores the IDENTICAL entity structure at the storage ROOT plus the generational
_system/_generations layer. GenerationStore.open tolerates a missing manifest
(opens at gen 0), so a naive 8.0 open of a 7.x directory reported ZERO entities
and SILENTLY LOST ALL DATA. The fix is blocked in the normal init order — cor 3.0's
storage-factory legacy guard fires inside setupStorage(), five steps before the
old migration check — so a new phase runs BEFORE setupStorage().
legacyLayoutMigrationPhase() (init, between loadPlugins and setupStorage):
- Fast-path: returns immediately when there's no top-level branches/ dir (every
native 8.0 brain + fresh dir pays only one fs.existsSync on the init hot path).
- Runs on a temporary BUILT-IN FileSystemStorage (never the plugin adapter, whose
guard would throw). Memory/cloud/pre-built-adapter stores are a strict no-op.
- Detect via _system/migration-layout.json marker + branches/<head>/entities/.
- autoMigrate:false on a legacy layout THROWS explicit guidance (no silent loss).
- Acquire the writer lock, then COLLAPSE branches/<head>/entities/* → entities/*
via the .gz-transparent raw primitives (read→write→delete, idempotent/resume-safe;
NOT fs.rename — logical paths + memory-adapter incompatibility).
- Rebuild persisted counts (rebuildCounts → totalNounCount/counts.json;
rebuildTypeCounts/rebuildSubtypeCounts → per-type stats) — the three indexes are
rebuilt by the normal init's rebuildIndexesIfNeeded afterward.
- Finalize: stamp the flat-v8 marker (re-open no-op), drain the head branch
(non-head branches = 7.x version history 8.0's MVCC does not import; left as-is).
Tests (tests/integration/migration-7x-to-8x.test.ts): the data-loss LOCK (a
built-in FileSystemStorage on a reshaped 7.x dir reports 0 nouns), a byte-equal
ROUND TRIP (find/related/counts equal the pre-reshape reference), IDEMPOTENCY,
the autoMigrate:false GUARD throw, and the native-flat NO-OP. RELEASES upgrade
checklist rewritten (the old note documented the opposite, lossy behavior).
1477 unit + migration 5 + db-mvcc 25 green; no init-path regression.
Follow-ups (hardening, not correctness for the common case): backupTo config
plumbing (currently a loud warning), a crash-mid-collapse resume test, and a
boundary-safe fake-plugin ordering test.
2026-06-19 13:44:03 -07:00
**Opening a 7.x store auto-migrates it.** 8.0 stores entities at the root; 7.x
stored them branch-scoped under `branches/<branch>/` . On first open, 8.0 collapses
the **HEAD branch** (`config.storage.branch` , default `main` ) to the 8.0 layout in
place and rebuilds all derived state — no action required (`autoMigrate` defaults
to `true` ; set it to `false` to make 8.0 refuse a legacy layout with an explicit
error instead). Two caveats:
- **Non-HEAD branches are not imported.** 8.0 has no COW branches; only the head
branch's data is migrated. If you care about other branches, **export each one
while still on 7.x** (`fork → export` , or copy its store).
- **Back up first for rollback.** The migration mutates the directory in place and
8.0 does not keep the old layout — copy the data directory before the first 8.0
open if you need to roll back.
2026-06-11 09:31:07 -07:00
### Renames — find/replace pairs
Hard renames, no aliases. Every pair below is verified against the 8.0 source.
| Brainy 7.x | Brainy 8.0 |
|---|---|
2026-06-11 14:51:00 -07:00
| `brain.delete(id)` | `brain.remove(id)` — matches the transact op vocabulary (`add` / `update` / `remove` / `relate` / `unrelate` ) |
| `brain.deleteMany(params)` | `brain.removeMany(params)` |
| `DeleteManyParams` (type) | `RemoveManyParams` |
| `brain.getRelations(paramsOrId)` | `brain.related(paramsOrId)` — the same name a pinned `Db` exposes (`db.related()` ) |
| `GetRelationsParams` (type) | `RelatedParams` |
| CLI `delete <id>` | CLI `remove <id>` (JSON output `{ id, removed: true }` , matching `unrelate` ) |
2026-06-11 09:31:07 -07:00
| `HnswProvider` (from `@soulcraft/brainy/plugin` ) | `VectorIndexProvider` |
| `HNSWIndex` (exported class) | `JsHnswVectorIndex` |
| Provider keys `'hnsw'` and `'diskann'` | `'vector'` (the only key Brainy consults for the vector index) |
2026-06-15 10:08:51 -07:00
| `config.hnsw = { quantization, vectorStorage }` | `config.vector = { recall?, persistMode? }` (shape change — see below) |
| `config.vector.quantization` (and 7.x `config.hnsw.quantization` ) | **Removed.** The JS vector path is full-precision (exact float32 distances) only — quantization at scale is the native provider's job (e.g. DiskANN PQ). |
| `hnswPersistMode: 'immediate' \| 'deferred'` (top level) | `config.vector.persistMode` (auto-selected from the storage adapter when omitted: `'immediate'` on filesystem, `'deferred'` on memory) |
2026-06-11 09:31:07 -07:00
| `config.storage.type: 's3' \| 'gcs' \| 'r2' \| 'opfs'` | `'filesystem'` (or `'memory'` / `'auto'` ) |
| `config.storage.branch` | Removed (no COW branches) |
| `brain.stats().indexHealth.hnsw` | `brain.stats().indexHealth.vector` |
| Storage adapter contract `saveHNSWData()` / `getHNSWData()` | `saveVectorIndexData()` / `getVectorIndexData()` |
| Cache category `'hnsw'` (`CacheProvider` union) | `'vectors'` |
Mechanical renames as one command (run from your repo root, review the diff):
```bash
find . -name '*.ts' -not -path '*/node_modules/*' -print0 | xargs -0 sed -i \
-e 's/\bHnswProvider\b/VectorIndexProvider/g' \
-e 's/\bHNSWIndex\b/JsHnswVectorIndex/g' \
-e 's/\bsaveHNSWData\b/saveVectorIndexData/g' \
-e 's/\bgetHNSWData\b/getVectorIndexData/g' \
-e 's/indexHealth\.hnsw\b/indexHealth.vector/g' \
-e "s/registerProvider('hnsw'/registerProvider('vector'/g" \
-e "s/registerProvider('diskann'/registerProvider('vector'/g" \
-e "s/getProvider('hnsw'/getProvider('vector'/g" \
2026-06-11 14:51:00 -07:00
-e "s/getProvider('diskann'/getProvider('vector'/g" \
-e 's/\.getRelations(/.related(/g' \
-e 's/\bGetRelationsParams\b/RelatedParams/g' \
-e 's/\.deleteMany(/.removeMany(/g' \
-e 's/\bDeleteManyParams\b/RemoveManyParams/g'
2026-06-11 09:31:07 -07:00
```
2026-06-11 14:51:00 -07:00
`delete()` → `remove()` is deliberately **not** in the snippet: `.delete(` is
too common (`Map` /`Set` /storage adapters) for a blind sed. Find your
`brain.delete(...)` call sites and rename them by hand.
2026-06-11 09:31:07 -07:00
The config shape changes need a human (they are not 1:1 textual):
```ts
// 7.x
new Brainy({
hnswPersistMode: 'deferred',
hnsw: { quantization: { enabled: true, bits: 8, rerankMultiplier: 3 } }
})
2026-06-15 10:08:51 -07:00
// 8.0 — config.vector is { recall?, persistMode? }
2026-06-11 09:31:07 -07:00
new Brainy({
vector: {
recall: 'balanced', // 'fast' | 'balanced' | 'accurate'
2026-06-15 10:08:51 -07:00
persistMode: 'deferred' // optional; auto-selected from storage when omitted
2026-06-11 09:31:07 -07:00
}
})
```
`config.vector.recall` replaces the algorithm-internal HNSW knobs: `'balanced'`
(the default) maps to exactly the 7.x default parameters, so an upgrade without
2026-06-15 10:08:51 -07:00
explicit knobs changes nothing about search behaviour. `config.vector.quantization`
and `config.hnsw.vectorStorage` are removed: the JS vector path now computes exact
float32 distances throughout (no rerank/approximate branch), which is what made
its quantization a memory *increase* — it stored both the full and the quantized
vectors in RAM. Quantization at scale belongs to the native provider's own PQ.
The on-disk vector index file names are **unchanged** (e.g. `hnsw-system.json` ) —
existing filesystem stores need no data migration for this rename; only the API
surface moved.
2026-06-11 09:31:07 -07:00
### Behavior changes
- **`subtype` is required by default.** 7.30's opt-in strict mode is now the
default: every `add()` / `addMany()` / `update()` / `relate()` /
`relateMany()` / `updateRelation()` rejects writes whose type carries no
non-empty `subtype` (`src/brainy.ts:9759` — `requireSubtype ?? true` ).
Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy
data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type
allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)`
2026-06-11 10:42:34 -07:00
still compose. `brain.audit()` reports entries missing a subtype and the new
`brain.fillSubtypes(rules)` migration helper backfills them — one rule per
NounType/VerbType (literal default or per-entry function), applied only to
entries still missing a subtype, returning
`{ scanned, filled, skipped, errors, byType }` . Idempotent: re-running fills
nothing, so a crashed run is resumed by running it again.
2026-06-11 09:31:07 -07:00
- **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws
a teaching error if a caller passes an `id` field
(`src/utils/paramValidation.ts:526` ). In 7.x a supplied id was silently
ignored (a generated UUID was used anyway); 8.0 says so out loud, because
graph-index providers key verb-int interning on the raw UUID bytes.
- **`update({ vector })` is honored on its own.** 7.x only applied a supplied
pre-computed vector when `data` was also passed (it was silently dropped
otherwise). 8.0 applies an explicit `params.vector` with dimension
validation (`src/brainy.ts:1702-1713` ; proven by
`tests/unit/brainy/update.test.ts:256` ).
- **`brain.clear()` re-resolves all indexes exactly as `init()` does** —
including plugin-provided vector/metadata/entityIdMapper factories and VFS
root re-creation. In 7.x, clearing a plugin-accelerated brain could leave
the metadata index rebuilt without its native id-mapper wiring.
- **`find({ connected: { …, subtype } })` with `depth > 1` now works.** 7.30
threw `NOT_SUPPORTED` for multi-hop subtype-filtered traversal; 8.0
implements the BFS in the JS graph index and routes to a native provider
when one is registered.
- **Every write advances the generation clock; only `transact()` writes
2026-06-11 14:51:00 -07:00
history.** Single-operation writes (`add` / `update` / `remove` / `relate`
2026-06-11 09:31:07 -07:00
outside `transact()` ) bump `brain.generation()` so watermarks and CAS stay
sound, but they do not stage historical records — they remain visible
through earlier pins and are not reported by `db.since()` . Writes you want
to travel back through go through `transact()` . This is the documented
contract, stated rather than papered over.
feat(8.0): reserved-field contract — one canonical location, typed prevention, unified read/write
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):
1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
metadata (and the transact() ops that extend them) reject a literal
reserved key as a TypeScript error while keeping generic T ergonomics
(typed bags, untyped brains, index-signature shapes, and a documented
exemption for T-declared reserved keys). Pinned by @ts-expect-error
type tests run under vitest typecheck mode on every unit run.
2. Write time — the 7.x update() remap is ported to 8.0 and extended to
every write path: add/update/relate/updateRelation, their transact()
mirrors, and db.with() overlays. User-settable fields lift to their
dedicated param (top-level wins when both are supplied — closes the 7.x
trap where update({metadata:{confidence}}) silently no-oped), and
system-managed fields drop with a one-shot warning naming the right
path. A remapped subtype satisfies subtype-pairing enforcement exactly
like a top-level one.
3. Read time — every storage combine goes through one canonical hydration
helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
and entity/relation.metadata carry ONLY custom fields on live reads,
batch reads, paginated listings, getRelations by source/target, streamed
verbs, and historical asOf() materialization alike.
Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.
Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
2026-06-11 13:12:50 -07:00
- **Reserved fields have one canonical location — enforced at every layer.**
The Brainy-owned field names (`RESERVED_ENTITY_FIELDS` :
`noun` /`subtype` /`createdAt` /`updatedAt` /`confidence` /`weight` /`service` /
`data` /`createdBy` /`_rev` ; verb mirror `RESERVED_RELATION_FIELDS` with
`verb` for the type key) are now (1) a **compile error** inside any
`metadata` param — `add` /`update` /`relate` /`updateRelation` and the
matching `transact()` ops; (2) **normalized at write time** for untyped
callers — user-settable fields remap to their dedicated top-level param
(top-level wins; `update({metadata:{confidence}})` no longer silently
no-ops, closing a 7.x trap), system-managed fields drop with a one-shot
warning naming the right path; (3) **split at read time** through one
canonical helper, so `entity.metadata` / `relation.metadata` contain ONLY
2026-06-11 14:51:00 -07:00
custom fields on every read path — `get` , `find` , `related` (several
feat(8.0): reserved-field contract — one canonical location, typed prevention, unified read/write
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):
1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
metadata (and the transact() ops that extend them) reject a literal
reserved key as a TypeScript error while keeping generic T ergonomics
(typed bags, untyped brains, index-signature shapes, and a documented
exemption for T-declared reserved keys). Pinned by @ts-expect-error
type tests run under vitest typecheck mode on every unit run.
2. Write time — the 7.x update() remap is ported to 8.0 and extended to
every write path: add/update/relate/updateRelation, their transact()
mirrors, and db.with() overlays. User-settable fields lift to their
dedicated param (top-level wins when both are supplied — closes the 7.x
trap where update({metadata:{confidence}}) silently no-oped), and
system-managed fields drop with a one-shot warning naming the right
path. A remapped subtype satisfies subtype-pairing enforcement exactly
like a top-level one.
3. Read time — every storage combine goes through one canonical hydration
helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
and entity/relation.metadata carry ONLY custom fields on live reads,
batch reads, paginated listings, getRelations by source/target, streamed
verbs, and historical asOf() materialization alike.
Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.
Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
2026-06-11 13:12:50 -07:00
paginated/by-source/by-target paths previously echoed the full stored
record, including the `verb` type key, inside `metadata` ), batch reads,
2026-06-11 14:51:00 -07:00
and historical `asOf()` reads. `related()` results now also surface
feat(8.0): reserved-field contract — one canonical location, typed prevention, unified read/write
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):
1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
metadata (and the transact() ops that extend them) reject a literal
reserved key as a TypeScript error while keeping generic T ergonomics
(typed bags, untyped brains, index-signature shapes, and a documented
exemption for T-declared reserved keys). Pinned by @ts-expect-error
type tests run under vitest typecheck mode on every unit run.
2. Write time — the 7.x update() remap is ported to 8.0 and extended to
every write path: add/update/relate/updateRelation, their transact()
mirrors, and db.with() overlays. User-settable fields lift to their
dedicated param (top-level wins when both are supplied — closes the 7.x
trap where update({metadata:{confidence}}) silently no-oped), and
system-managed fields drop with a one-shot warning naming the right
path. A remapped subtype satisfies subtype-pairing enforcement exactly
like a top-level one.
3. Read time — every storage combine goes through one canonical hydration
helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
and entity/relation.metadata carry ONLY custom fields on live reads,
batch reads, paginated listings, getRelations by source/target, streamed
verbs, and historical asOf() materialization alike.
Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.
Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
2026-06-11 13:12:50 -07:00
`confidence` /`updatedAt` top-level, and `updateRelation()` no longer
erases a relationship's `service` /`createdBy` . See "Reserved fields" in
`docs/concepts/consistency-model.md` .
2026-06-11 14:51:00 -07:00
- **Named errors for not-found contract failures.** `update()` , `relate()` ,
`updateRelation()` , `similar({ to: id })` , `transact()` planning, and
speculative `db.with()` planning now throw `EntityNotFoundError` /
`RelationNotFoundError` (both exported from the package root, both carrying
the missing `id` as a field) instead of a generic `Error` . Message texts are
unchanged, so existing `/not found/` matching keeps working — `instanceof`
is now the supported way to branch.
2026-06-11 09:31:07 -07:00
- **Unchanged from 7.31:** per-entity `_rev` , `update({ ifRev })`
(`RevisionConflictError` ), and `add({ ifAbsent })` work exactly as before,
and `ifRev` is also accepted on `transact()` update operations (a conflict
rejects the whole batch).
### For provider and plugin authors
The provider contracts in `@soulcraft/brainy/plugin` (`src/plugin.ts` ) changed
shape:
- **`GraphIndexProvider` speaks BigInt at the boundary.** Reads take entity
ints and return entity/verb ints as `bigint[]` ; the coordinator owns all
UUID ↔ int conversion. `addVerb(verb, sourceInt, targetInt)` receives the
resolved endpoint ints (also mirrored on the new optional
`GraphVerb.sourceInt` / `GraphVerb.targetInt` fields) and returns the
interned verb int. The batch reverse resolver
`verbIntsToIds(verbInts: bigint[]): Promise<(string | null)[]>` is
**required** — the provider owns durable verb-int interning; Brainy keeps
only a bounded in-memory warm cache. Implementations may stay u32 internally
(`Number(bigint)` narrowing is lossless under the `EntityIdSpaceExceeded`
guard) but must speak the bigint contract.
- **`VersionedIndexProvider` ** is a new optional 4-method capability —
`generation()` , `isGenerationVisible(g)` , `pin(g)` , `release(g)` , BigInt
generations — feature-detected on every registered index provider. Providers
are *post-commit appliers* : the storage-record commit is the source of
truth; on open, a provider behind the committed watermark replays the gap or
requests a rebuild. Explicit pins override any time-based snapshot retention
the provider has. Speculative `db.with()` overlays never reach providers. A
provider implementing this serves historical reads with **no rebuild** —
the open-core materializer is the correctness baseline, the provider is the
accelerator.
- **The vector index registers under `'vector'` ** and implements
`VectorIndexProvider` . The `'hnsw'` and `'diskann'` keys are retired and
never looked up. `CacheProvider` 's category union uses `'vectors'` in place
of `'hnsw'` .
### Scale and cost — how the mechanisms behave
Mechanism descriptions, not benchmark numbers (none are published for 8.0 yet):
- `brain.now()` pins in O(1) and adds **zero read overhead until history
actually moves** — while nothing has committed past the pin, every read
delegates to the live fast paths.
- A `transact()` commit pays O(ids touched) extra writes (before-images +
delta + manifest) — never O(store size). Single-operation writes pay an
in-memory counter bump with coalesced persistence.
- `db.persist()` on filesystem storage is a hard-link farm: snapshot creation
copies no entity data and the snapshot shares disk space with the source.
Cross-device targets fall back to per-file byte copies; persisting an
in-memory brain serializes a real, durable, loadable store.
- Historical **index-accelerated** queries (semantic search, traversal,
cursors, aggregation) on the open-core path pay a one-time O(n at the
pinned generation) in-memory index materialization per `Db` , cached until
`release()` . Record-path reads (`get` , metadata `find` , filter `related` )
at any reachable generation are served directly from the immutable record
layer. A native `VersionedIndexProvider` serves the same reads rebuild-free.
The full cost model is in
[docs/concepts/consistency-model.md ](docs/concepts/consistency-model.md ).
### Upgrade checklist (from 7.x)
1. **While still on 7.x:** back up your data directory (a plain file copy is
fine). If you used COW branches, materialize each branch you need — 8.0
does not read branch state. If you ran a cloud storage adapter, export your
data with 7.x tooling (`(await brain.data()).export()` ) — 8.0 opens
`filesystem` and `memory` stores only.
2. **Runtime:** Node.js 22 LTS or Bun ≥ 1.0.
3. **Config:** change removed storage types to `'filesystem'` ; delete
`storage.branch` ; move `config.hnsw` / `hnswPersistMode` to
`config.vector` per the shape example above.
4. **Renames:** run the sed snippet, then fix the manual shape changes.
5. **Removed APIs:** replace `fork` / `checkout` / branch methods / `commit` /
`getHistory` / `streamHistory` / `versions` / `data()` per the table above.
6. **Subtypes:** if your data predates subtype discipline, start with
`requireSubtype: false` , run `brain.audit()` , backfill with
2026-06-11 10:42:34 -07:00
`brain.fillSubtypes(rules)` (one rule per type — a literal default or a
function deriving the subtype from each entry), re-run `audit()` until
`total === 0` , then remove the opt-out so the 8.0 default enforcement
protects you going forward.
2026-06-11 09:31:07 -07:00
7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate` →
`snapshot <path>` / `restore <path>` / `history` / `generation` .
8. **Plugin authors:** apply the contract renames and the BigInt graph
contract; optionally implement `VersionedIndexProvider` .
9. **Verify, then take your first snapshot:** run your test suite, then
`const db = brain.now(); await db.persist('/backups/post-8.0-upgrade');
await db.release()`.
What did **not** change: the core query surface (`find` / `search` / `get` /
`add` / `update` / `relate` and friends), the aggregation engine, the VFS,
neural extraction, and the on-disk entity/relationship layout — 8.0 creates
its MVCC bookkeeping (`_system/generation.json` , `_system/manifest.json` ,
`_system/tx-log.jsonl` , `_generations/` ) alongside the existing files on
first use.
### Cortex compatibility
Brainy 8.0 pairs with `@soulcraft/cortex` 3.0 (shipping in lockstep). Cortex
2.x cannot accelerate Brainy 8.0: 8.0 never consults the `'hnsw'` /
`'diskann'` provider keys, and the graph/column provider contracts are BigInt
at the boundary. Upgrade both packages together.
---
2026-06-09 12:53:34 -07:00
## v7.31.2 — 2026-06-09
**Affected products:** none in behaviour; affects anyone reading Brainy's TypeScript
type definitions or coreTypes for the `hnsw.quantization.bits` option. Drop-in from 7.31.1.
### Fix: stale comment claimed SQ4 vector quantization required a paid native provider
Brainy ships a pure-JS SQ4 distance implementation (`distanceSQ4Js` in
`src/utils/vectorQuantization.ts` ) that's been part of the open-core path the whole
time. Two type-definition comments incorrectly stated SQ4 required a native (paid)
provider:
```ts
// coreTypes.ts:472 + types/brainy.types.ts:1123 — before
bits?: 8 | 4 // default: 8 (SQ8). SQ4 requires cortex native.
```
The comment was simply wrong — open-core users can use SQ4 today, and the native SIMD
acceleration is a drop-in via `setSQ4DistanceImplementation` , not a hard requirement.
Corrected to:
```ts
bits?: 8 | 4 // default: 8 (SQ8). SQ4 has a pure-JS implementation;
// cortex's distance:sq4 SIMD provider accelerates it.
```
Both locations updated. Pure documentation; no behaviour change. Caught during an
upstream open-core-boundary audit — thanks to whoever read the types carefully enough
to spot the misleading comment.
### Cortex compatibility
Zero changes required. The fix is documentation only; runtime behaviour was already
correct (`vectorQuantization.ts:412` ships `distanceSQ4Js` ).
---
fix: saveBinaryBlob unique tmp suffix + ENOENT swallow on rename
FileSystemStorage.saveBinaryBlob used a bare ${filePath}.tmp suffix for its
atomic-write temp file. Two concurrent same-key calls computed the SAME temp
path; both writeFile'd, the first rename succeeded, the second rename fired
against a missing temp and threw ENOENT. The throw propagated up through
brain.flush() and broke any downstream job that called it.
Reproduced in production (Brainy 7.30 + Cortex 2.7 + mmap-filesystem):
[job-queue] gcs-backup: failed - ENOENT: no such file or directory,
rename '/data/brainy-data/.../_column_index/owner/DELETED.bin.tmp'
-> '/data/brainy-data/.../_column_index/owner/DELETED.bin'
The race was two cron jobs (gcs-backup hourly + flush-brain every 15min) both
calling flush(), triggering column-store compaction over the same fields,
overlapping at the rename. Off-site backups had been failing continuously for
~48 hours.
PATCH
src/storage/adapters/fileSystemStorage.ts:992-999 — saveBinaryBlob now uses a
unique per-writer temp suffix matching the pattern at every other atomic-write
site in the same file (lines 336, 551, 744, 781, 1529, 2908):
const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}`
Adds defensive ENOENT swallow on rename: if the temp is gone, the work has
already landed (saveBinaryBlob is idempotent for a given key — all callers
persist the same logical bytes per key). Cleans up the temp on any other
rename failure to avoid orphan .tmp.* files.
SCOPE AUDIT
One bug site. Audit results:
- FileSystemStorage: six sibling atomic-write sites already used unique
suffixes (lines 336/551/744/781/1529/2908); only saveBinaryBlob was the
outlier. All six rechecked. Clean.
- OPFSStorage: WritableStream (no tmp+rename). Not affected.
- GCSStorage / R2Storage / AzureBlobStorage / S3CompatibleStorage: object-store
PUT (atomic at the API). Not affected.
- MemoryStorage: in-memory. Not affected.
- HistoricalStorageAdapter: read-only. Not affected.
- COW / versioning / snapshot / HNSW / aggregation: all delegate to storage
adapters via saveBinaryBlob / writeObjectToPath. They get the fix
automatically by using the patched primitive.
The bare-`.tmp` pattern is now gone repo-wide.
BENEFICIARIES
Beyond the reported column-store-compaction race:
- HNSW connection persistence (src/hnsw/hnswIndex.ts:252 → saveBinaryBlob)
was structurally susceptible to the same race. No production reports of
HNSW failures (probably because HNSW writes are more naturally serialized
by the index lock), but the fix removes the latent issue.
- Any future caller of saveBinaryBlob inherits the safer semantics.
TESTS
New tests/integration/savebinaryblob-concurrent-rename.test.ts (4 tests):
- 20 concurrent saveBinaryBlob(sameKey, ...) all resolve without throwing
- No orphan .tmp.* siblings remain in the blob directory afterwards
- Production-shape: two concurrent compactor passes over 10 column-index
fields (owner, path, permissions, vfsType, modified, createdAt, accessed,
updatedAt, mimeType, size). All 20 calls succeed; each field ends with
valid bytes.
- Single-writer path still produces correct bytes (no regression on common
case).
Verified the first three tests reproduce the production ENOENT error
verbatim on the pre-7.31.1 code path (stashed the fix, watched them fail
with the exact production error).
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- New integration suite: 4/4
- npm run build: clean
CORTEX COMPATIBILITY
Zero changes. The Cortex mmap-filesystem adapter wraps FileSystemStorage
and inherits the patch automatically.
FORWARD-COMPAT
8.0's Db.persist() will route through the same patched primitive; no
additional work needed when the immutable Db API ships.
2026-06-09 10:36:02 -07:00
## v7.31.1 — 2026-06-09
**Affected products:** any consumer running `@soulcraft/brainy` against `FileSystemStorage`
that triggers concurrent flush / compaction (e.g. explicit `brain.flush()` calls overlapping
with periodic background compaction, or a backup job + a flush job racing). Production-impacting
when present: `brain.flush()` throws `ENOENT` on rename, downstream jobs that rely on flush
(GCS / S3 / Azure backups, snapshot exports) fail continuously. Drop-in from 7.31.0.
### Fixes a same-key rename race in `saveBinaryBlob`
`FileSystemStorage.saveBinaryBlob` used a bare `${filePath}.tmp` suffix for its atomic-write
temp file. Two concurrent same-key calls computed the **same** temp path; both `writeFile` d,
the first `rename` succeeded, the second `rename` fired against a missing temp and threw
`ENOENT` . The throw propagated up through `brain.flush()` and broke any downstream job that
called it.
```
[job-queue] gcs-backup: failed — ENOENT: no such file or directory,
rename '/data/brainy-data/.../_column_index/owner/DELETED.bin.tmp'
-> '/data/brainy-data/.../_column_index/owner/DELETED.bin'
```
**Patch:** unique per-writer temp suffix (matches the pattern at six other atomic-write
sites in the same file) + ENOENT swallow on rename (defensive: if the temp is gone, the
work has already landed; `saveBinaryBlob` is idempotent for a given key) + temp cleanup
on any other rename failure (no orphan `.tmp.*` files).
```ts
// before (7.31.0)
const tmpPath = filePath + '.tmp'
await fs.promises.writeFile(tmpPath, data)
await fs.promises.rename(tmpPath, filePath)
// after (7.31.1)
const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}`
await fs.promises.writeFile(tmpPath, data)
try {
await fs.promises.rename(tmpPath, filePath)
} catch (err) {
if ((err as NodeJS.ErrnoException).code === 'ENOENT') return
await fs.promises.unlink(tmpPath).catch(() => {})
throw err
}
```
### Scope
The fix is one site — `src/storage/adapters/fileSystemStorage.ts:992-999` . Audit results:
- **`FileSystemStorage` ** — six sibling atomic-write sites already used unique suffixes;
only `saveBinaryBlob` was the outlier. All six were rechecked. Clean.
- **`OPFSStorage` ** — writes via WritableStream (no tmp+rename). Not affected.
- **`GCSStorage` / `R2Storage` / `AzureBlobStorage` / `S3CompatibleStorage` ** — use
object-store `PUT` (atomic at the API). Not affected.
- **`MemoryStorage` ** — in-memory. Not affected.
- **`HistoricalStorageAdapter` ** — read-only. Not affected.
- **COW / versioning / snapshot / HNSW / aggregation** — all delegate to storage adapters
via `saveBinaryBlob` / `writeObjectToPath` . They get the fix automatically by virtue of
using the patched primitive.
### Beneficiary surfaces (broader than the reported bug)
Column-store compaction (`_column_index/<field>/DELETED.bin` , segment files) is what the
production report named, but `saveBinaryBlob` is also used by HNSW connection persistence
(`_hnsw_conn/<nodeId>` — `src/hnsw/hnswIndex.ts:252` ). If two flush paths ever raced on
HNSW persistence for the same node, they would have hit the same ENOENT. No production
reports of HNSW-side failures, but the fix removes the latent race.
### Tests
- **New `tests/integration/savebinaryblob-concurrent-rename.test.ts` ** (4 tests):
- 20 concurrent `saveBinaryBlob(sameKey, ...)` calls all resolve without throwing
- No orphan `.tmp.*` siblings remain in the blob directory after concurrent writes
- Production-shape: two concurrent compactor passes over 10 column-index fields
(`owner` , `path` , `permissions` , `vfsType` , `modified` , `createdAt` , `accessed` ,
`updatedAt` , `mimeType` , `size` )
- Single-writer path still produces correct bytes (no regression)
- The first three tests reproduce the production `ENOENT: rename '...DELETED.bin.tmp' ->
'...DELETED.bin'` error verbatim on the pre-7.31.1 code path. Verified by stashing the
fix and watching the tests fail with the exact production error.
- Existing suites unchanged: 1468/1468 unit. All integration suites pass.
### Cortex compatibility
Zero changes required. The bug and the fix are pure JS in the filesystem storage adapter;
the Cortex mmap-filesystem adapter wraps `FileSystemStorage` and inherits the patch
automatically.
### Forward-compat
The bare-`.tmp` pattern is gone repo-wide. 8.0's `Db.persist()` will route through the
same patched primitive; no additional work needed when the immutable Db API ships.
---
feat: per-entity _rev + update({ ifRev }) CAS + add({ ifAbsent })
Optimistic concurrency for multi-writer coordination — the read-then-CAS
lock pattern + idempotent-bootstrap singleton inserts. Lands the surface the
SDK scheduler asked for in BRAINY-EXPOSE-TRANSACTIONS, scoped to what ships
cleanly without painting the 8.0 transact() API into a corner.
A. PER-ENTITY _rev FIELD
- src/coreTypes.ts — HNSWNounWithMetadata gains optional `_rev?: number`;
STANDARD_ENTITY_FIELDS includes it so resolveEntityField routes correctly.
- src/types/brainy.types.ts — Entity<T> gains optional `_rev?: number`;
Result<T> mirrors it on the convenience flatten layer.
- src/brainy.ts add() — initializes `_rev: 1` in storageMetadata. New entities
always start at 1.
- src/brainy.ts update() — reads currentRev off the persisted metadata
(falls back to 1 for pre-7.31.0 entities with no _rev), writes
`_rev: currentRev + 1` into the updated metadata. Every successful update()
bumps by exactly 1.
- src/brainy.ts convertNounToEntity + convertMetadataToEntity — both pull
_rev out of the storage metadata and surface it at the top level of Entity.
Pre-7.31.0 entities (no _rev in storage) read as `_rev: 1` so consumers
see a consistent value.
- src/storage/baseStorage.ts — six destructure sites updated to pull _rev
out of the metadata bag so it doesn't leak into customMetadata. Noun-side
returns surface _rev; verb-side destructures correctly but doesn't expose
it on HNSWVerbWithMetadata (verb CAS is future work).
- src/brainy.ts createResult() — flattens entity._rev onto the Result for
backward compat with the existing convenience-field layer.
B. update({ ifRev }) OPTIMISTIC CONCURRENCY
- src/types/brainy.types.ts — UpdateParams<T> gains optional `ifRev?: number`.
- src/transaction/RevisionConflictError.ts (NEW) — carries `{ id, expected,
actual }`. Message names the recipe: refetch with brain.get() and retry
with the latest _rev. Subclass of Error.
- src/brainy.ts update() — when params.ifRev is provided, compares against
currentRev (the rev we just read) and throws RevisionConflictError on
mismatch before any storage write. Omitting ifRev keeps the prior
unconditional-update behavior; existing consumers see no change.
- src/transaction/index.ts — exports RevisionConflictError.
- src/index.ts — public export of RevisionConflictError.
C. add({ ifAbsent }) BY-ID IDEMPOTENT INSERT
- src/types/brainy.types.ts — AddParams<T> gains optional `ifAbsent?: boolean`;
AddManyParams<T> mirrors it as a batch-level flag.
- src/brainy.ts add() — when params.id AND params.ifAbsent, pre-reads
storage.getNounMetadata(id); if present, returns the existing id without
writing. No throw, no overwrite. Ignored when id is omitted (a fresh UUID
can never collide).
- src/brainy.ts addMany() — propagates the batch-level ifAbsent to each
item's add() call. Per-item ifAbsent takes precedence so callers can
override individual rows.
WHAT'S NOT SHIPPED (and why)
A public brain.transaction(fn) wrapper was on the table but was cut. The
internal TransactionManager exposes raw Operation classes (SaveNounMetadata,
etc.) that take StorageAdapter as a constructor argument. A clean high-level
facade in 7.31.0 would have meant either:
(a) Delegate to brain.add() / update() / relate(). Each of those opens its
own internal transaction and commits before the closure returns. Looks
atomic, isn't — a footgun.
(b) Thread an optional `tx?` through every internal write site in
brainy.ts (8 transactionManager.executeTransaction sites). ~2 days of
real refactor with regression surface, and the API shape changes again
in 8.0 anyway.
The SDK scheduler's actual ask (BRAINY-EXPOSE-TRANSACTIONS) is the
read-then-CAS lock pattern. _rev + ifRev solves it completely. Multi-write
atomicity is the 8.0 brain.transact() use case where the Datomic-style
immutable Db makes it atomic by construction — that's the right home.
TESTS
- New tests/integration/rev-and-ifabsent.test.ts (18 tests):
- _rev initialization (1 on add) and surface on get fast/full paths + find
- _rev auto-bump on update across multiple writes
- update({ ifRev }) pass / fail / message format / omitted / legacy-no-rev
- add({ ifAbsent }) writes when absent / no-op when present / id-required
- addMany({ ifAbsent }) propagation + per-item override
- SDK-scheduler scenario: two concurrent CAS updates, one wins one throws
- Unit suite unchanged: 1468/1468.
- All integration subtype + verb + strict + find-limits + new rev suites
pass: 96/96.
DOCS
- New docs/guides/optimistic-concurrency.md (public: true) — full reference:
the lock pattern, read-modify-write retry, idempotent bootstrap, how _rev
interacts with brain.versions, branches/fork, and VFS, what's coming in 8.0.
- docs/api/README.md — add() and update() entries get the new params + tips
pointing at the new guide.
- RELEASES.md v7.31.0 entry.
CORTEX COMPATIBILITY
Zero changes required. _rev is a metadata column already supported by
NativeColumnStore. The auto-bump runs in Brainy JS before any storage call;
Cortex never sees the per-entity counter.
8.0 FORWARD-COMPAT
_rev, ifRev, RevisionConflictError, and ifAbsent survive the 8.0 Db redesign
unchanged. 8.0 layers brain.transact(tx, { ifAtGeneration }) for whole-tx CAS
on top of the same per-entity mechanism — per-entity for single-record
patterns (job locks, idempotent state machines), generation-based for
"did the world move under me." Locked in .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md § C-6.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All integration suites including new rev-and-ifabsent: 96/96
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-09 10:04:24 -07:00
## v7.31.0 — 2026-06-09
**Affected products:** consumers that need multi-writer coordination (job
schedulers, idempotent state machines, distributed locks, optimistic UI updates)
or idempotent bootstrap of well-known singletons. Additive; drop-in from 7.30.2.
### Per-entity `_rev` + `update({ ifRev })` + `add({ ifAbsent })`
CouchDB / PouchDB / ETag-style optimistic concurrency. Every entity now carries a
monotonic `_rev: number` that Brainy auto-bumps on every successful `update()` .
Pass it back as `ifRev` to make read-modify-write race-safe without an external
lock service. `ifAbsent` adds by-ID idempotent insert for singletons / config rows
/ deterministic-ID bootstraps.
**What's new:**
- **`entity._rev: number` ** — initialized to `1` on `add()` , bumped by `1` on every
successful `update()` that reaches storage. Surfaced on `get()` (both fast metadata-
only and full-vector paths), `find()` , `search()` , and on the `Result` flatten layer
for backward compatibility. Pre-7.31.0 entities without `_rev` are read as `1` .
- **`update({ id, ..., ifRev: number })` ** — optimistic-concurrency check. When
provided, throws `RevisionConflictError` if the persisted `_rev` no longer matches.
Carries `{ id, expected, actual }` for principled recovery. Omitting `ifRev` keeps
the prior unconditional-update behavior.
- **`add({ id, ifAbsent: true })` ** — by-ID idempotent insert. Returns the existing
`id` without writing if the entity already exists. No throw, no overwrite. Ignored
when `id` is omitted (a fresh UUID can never collide).
- **`addMany({ items, ifAbsent: true })` ** — applies the flag to every item; per-item
`ifAbsent` overrides the batch flag.
### The lock recipe (single-process or distributed)
```ts
import { Brainy, RevisionConflictError } from '@soulcraft/brainy '
const LOCK_ID = '...uuid...'
await brain.add({
id: LOCK_ID,
type: NounType.Document,
data: { owner: null, expiresAt: 0 },
ifAbsent: true
})
async function tryAcquireLock(workerId: string, ttlMs: number) {
const lock = await brain.get(LOCK_ID)
if (!lock) throw new Error('lock document missing')
const state = lock.data as { owner: string | null; expiresAt: number }
if (state.owner & & state.expiresAt > Date.now()) return false
try {
await brain.update({
id: LOCK_ID,
data: { owner: workerId, expiresAt: Date.now() + ttlMs },
ifRev: lock._rev
})
return true
} catch (err) {
if (err instanceof RevisionConflictError) return false
throw err
}
}
```
### Why this is scoped to `_rev` + `ifAbsent`
A public `brain.transaction(fn)` wrapper was on the table but was cut. The internal
`TransactionManager` exposes raw `Operation` classes that take a `StorageAdapter`
constructor argument; a high-level facade in 7.31.0 would have meant either
(a) delegating to `brain.add()` / `update()` / `relate()` — each of which commits
its own internal transaction before the closure returns, so multi-write rollback
wouldn't actually work (a footgun), or (b) threading `tx?` through every internal
write path — ~2 days of refactor with regression surface, and the API shape changes
again in 8.0 anyway.
The SDK-scheduler use case that drove the thread (read-then-CAS lock pattern) is
fully solved by `_rev` + `ifRev` . Multi-write atomicity is the 8.0 `brain.transact()`
use case, where the Datomic-style immutable Db makes it atomic by construction. We
chose not to ship a worse version in the interim.
### How `_rev` interacts with branches and the snapshot API
| System | What it tracks | When it advances |
|---|---|---|
| `_rev` (NEW) | Per-entity write counter | Auto, on every successful `update()` |
| `brain.versions.save()` | Named snapshots per entity | Explicit — you call `save()` |
| `brain.fork()` / branches | Whole-brain copy-on-write | Explicit — you call `fork(name)` |
| VFS file versioning | Per-VFS-file snapshots | Same as `brain.versions.save()` |
`_rev` is independent. Each branch has its own copy of every entity, so each
branch has its own `_rev` per entity (same as every other field under COW).
Snapshots taken via `brain.versions.save()` capture the entity at that moment
including its `_rev` at that time; the snapshot's own `version: number` is the
snapshot index, distinct from `_rev` .
### Docs
- **New `docs/guides/optimistic-concurrency.md` ** (public, indexed at
soulcraft.com/docs after portal deploy) — full reference: the lock pattern,
read-modify-write retry, idempotent bootstrap, how `_rev` interacts with branches /
snapshots / VFS, what's coming in 8.0.
- `docs/api/README.md` `add()` and `update()` entries get the new params + tips
pointing at the guide.
### Tests
- **New `tests/integration/rev-and-ifabsent.test.ts` ** (18 tests): rev initialization,
rev bump on update, ifRev pass / fail / omit / legacy-entity-fallback, ifAbsent
with custom id / no id / batch propagation / per-item override, plus a full
SDK-scheduler-style two-concurrent-CAS-updaters scenario.
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-enforcement
30/30, strict-mode-self-test 13/13, find-limits 9/9. Unit 1468/1468.
### Cortex compatibility
Zero changes required. `_rev` is a metadata column already supported by
`NativeColumnStore` ; the auto-bump runs in Brainy JS before any storage call.
Cortex never sees the per-entity counter — it's a single integer field updated
alongside every other metadata field on the existing write paths.
### 8.0 forward-compat
`_rev` , `ifRev` , `RevisionConflictError` , and `ifAbsent` all survive the 8.0 Db
redesign unchanged. 8.0 layers `brain.transact(tx, { ifAtGeneration })` for
whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record
2026-06-11 10:42:34 -07:00
patterns, generation-based for "did the world move under me."
feat: per-entity _rev + update({ ifRev }) CAS + add({ ifAbsent })
Optimistic concurrency for multi-writer coordination — the read-then-CAS
lock pattern + idempotent-bootstrap singleton inserts. Lands the surface the
SDK scheduler asked for in BRAINY-EXPOSE-TRANSACTIONS, scoped to what ships
cleanly without painting the 8.0 transact() API into a corner.
A. PER-ENTITY _rev FIELD
- src/coreTypes.ts — HNSWNounWithMetadata gains optional `_rev?: number`;
STANDARD_ENTITY_FIELDS includes it so resolveEntityField routes correctly.
- src/types/brainy.types.ts — Entity<T> gains optional `_rev?: number`;
Result<T> mirrors it on the convenience flatten layer.
- src/brainy.ts add() — initializes `_rev: 1` in storageMetadata. New entities
always start at 1.
- src/brainy.ts update() — reads currentRev off the persisted metadata
(falls back to 1 for pre-7.31.0 entities with no _rev), writes
`_rev: currentRev + 1` into the updated metadata. Every successful update()
bumps by exactly 1.
- src/brainy.ts convertNounToEntity + convertMetadataToEntity — both pull
_rev out of the storage metadata and surface it at the top level of Entity.
Pre-7.31.0 entities (no _rev in storage) read as `_rev: 1` so consumers
see a consistent value.
- src/storage/baseStorage.ts — six destructure sites updated to pull _rev
out of the metadata bag so it doesn't leak into customMetadata. Noun-side
returns surface _rev; verb-side destructures correctly but doesn't expose
it on HNSWVerbWithMetadata (verb CAS is future work).
- src/brainy.ts createResult() — flattens entity._rev onto the Result for
backward compat with the existing convenience-field layer.
B. update({ ifRev }) OPTIMISTIC CONCURRENCY
- src/types/brainy.types.ts — UpdateParams<T> gains optional `ifRev?: number`.
- src/transaction/RevisionConflictError.ts (NEW) — carries `{ id, expected,
actual }`. Message names the recipe: refetch with brain.get() and retry
with the latest _rev. Subclass of Error.
- src/brainy.ts update() — when params.ifRev is provided, compares against
currentRev (the rev we just read) and throws RevisionConflictError on
mismatch before any storage write. Omitting ifRev keeps the prior
unconditional-update behavior; existing consumers see no change.
- src/transaction/index.ts — exports RevisionConflictError.
- src/index.ts — public export of RevisionConflictError.
C. add({ ifAbsent }) BY-ID IDEMPOTENT INSERT
- src/types/brainy.types.ts — AddParams<T> gains optional `ifAbsent?: boolean`;
AddManyParams<T> mirrors it as a batch-level flag.
- src/brainy.ts add() — when params.id AND params.ifAbsent, pre-reads
storage.getNounMetadata(id); if present, returns the existing id without
writing. No throw, no overwrite. Ignored when id is omitted (a fresh UUID
can never collide).
- src/brainy.ts addMany() — propagates the batch-level ifAbsent to each
item's add() call. Per-item ifAbsent takes precedence so callers can
override individual rows.
WHAT'S NOT SHIPPED (and why)
A public brain.transaction(fn) wrapper was on the table but was cut. The
internal TransactionManager exposes raw Operation classes (SaveNounMetadata,
etc.) that take StorageAdapter as a constructor argument. A clean high-level
facade in 7.31.0 would have meant either:
(a) Delegate to brain.add() / update() / relate(). Each of those opens its
own internal transaction and commits before the closure returns. Looks
atomic, isn't — a footgun.
(b) Thread an optional `tx?` through every internal write site in
brainy.ts (8 transactionManager.executeTransaction sites). ~2 days of
real refactor with regression surface, and the API shape changes again
in 8.0 anyway.
The SDK scheduler's actual ask (BRAINY-EXPOSE-TRANSACTIONS) is the
read-then-CAS lock pattern. _rev + ifRev solves it completely. Multi-write
atomicity is the 8.0 brain.transact() use case where the Datomic-style
immutable Db makes it atomic by construction — that's the right home.
TESTS
- New tests/integration/rev-and-ifabsent.test.ts (18 tests):
- _rev initialization (1 on add) and surface on get fast/full paths + find
- _rev auto-bump on update across multiple writes
- update({ ifRev }) pass / fail / message format / omitted / legacy-no-rev
- add({ ifAbsent }) writes when absent / no-op when present / id-required
- addMany({ ifAbsent }) propagation + per-item override
- SDK-scheduler scenario: two concurrent CAS updates, one wins one throws
- Unit suite unchanged: 1468/1468.
- All integration subtype + verb + strict + find-limits + new rev suites
pass: 96/96.
DOCS
- New docs/guides/optimistic-concurrency.md (public: true) — full reference:
the lock pattern, read-modify-write retry, idempotent bootstrap, how _rev
interacts with brain.versions, branches/fork, and VFS, what's coming in 8.0.
- docs/api/README.md — add() and update() entries get the new params + tips
pointing at the new guide.
- RELEASES.md v7.31.0 entry.
CORTEX COMPATIBILITY
Zero changes required. _rev is a metadata column already supported by
NativeColumnStore. The auto-bump runs in Brainy JS before any storage call;
Cortex never sees the per-entity counter.
8.0 FORWARD-COMPAT
_rev, ifRev, RevisionConflictError, and ifAbsent survive the 8.0 Db redesign
unchanged. 8.0 layers brain.transact(tx, { ifAtGeneration }) for whole-tx CAS
on top of the same per-entity mechanism — per-entity for single-record
patterns (job locks, idempotent state machines), generation-based for
"did the world move under me." Locked in .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md § C-6.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All integration suites including new rev-and-ifabsent: 96/96
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-09 10:04:24 -07:00
---
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
## v7.30.2 — 2026-06-08
**Affected products:** any consumer with `find({ limit })` call sites that pass values
≥ ~9000 — a common safety-cap pattern in production code (`limit: 10_000` against
type-filtered queries that typically return 10–
from 7.30.1.
### Why
Brainy 7.30 introduced a memory-derived cap on `find({ limit })` to prevent OOM
from runaway queries. The cap was sound in intent but ** ~4×
calibration**: the formula assumed 100 KB per result while typical entity footprint
is 7-10 KB (384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box this capped `limit` at 9000, breaking production dashboards where
cascading 500s from queries with `limit: 10_000` silently degraded the `Promise.all`
loading the canvas.
7.30.2 recalibrates the formula and changes the enforcement from synchronous-throw
to two-tier (warn-then-throw) so existing pre-7.30 code doesn't break on upgrade.
### Recalibrated formula — `25 KB` per result (was `100 KB`)
The auto-configured cap now uses a realistic per-result size:
| Memory source | 7.30.0–
|---|---|---|
| 4 GB Cloud Run container | 10 000 | 40 000 |
| 2 GB container | 5 000 | 20 000 |
| 900 MB free system memory | 9 000 | ~36 000 |
| `reservedQueryMemory: 1 GB` | 10 000 | 40 000 |
Hard ceiling stays at 100 000. Existing `BrainyConfig.maxQueryLimit` /
`reservedQueryMemory` overrides continue to work unchanged.
### Two-tier enforcement — warn, then throw
**Below cap (`limit ≤ maxLimit` ):** silent pass. No signal, no friction. Unchanged.
**Soft tier (`maxLimit < limit ≤ 2 × )** *NEW* : one-time warning per call
site (dedup keyed on stack location + limit value), query proceeds. Pre-7.30.2 code
that relied on the cap silently allowing safety-cap limits keeps working — the
warning teaches the recipe so consumers can fix it intentionally.
**Hard tier (`limit > 2 × )** *NEW* : throw with the same message format the
warning uses. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
The 2× `limit: 10_000`
against a 9 K-cap box) without disabling OOM protection. Real OOM territory on a JS
in-memory brain is hundreds of thousands of results, not 10×
### Improved error / warning message (mirrors the 7.30.1 enforcement-error format)
Before (7.30 → 7.30.1):
```
limit exceeds auto-configured maximum of 9000 (based on available memory)
```
After (7.30.2):
```
find({ limit: 10000 }) exceeds the auto-configured query limit of 9000 (basis:
available free memory). Choose one:
• Increase the cap: new Brainy({ maxQueryLimit: 10000 })
• Reserve more memory: new Brainy({ reservedQueryMemory: 256000000 })
• Paginate: split the query with { limit, offset } pages
at OrderService.loadDashboard (/app/src/orders/dashboard.ts:142:18)
Docs: https://soulcraft.com/docs/guides/find-limits
```
Caller location comes from the same `findCallerLocation()` helper introduced in
7.30.1 (now extracted to `src/utils/callerLocation.ts` so both subtype enforcement
and limit enforcement share it).
### New docs
New `docs/guides/subtypes-and-facets.md` -style guide at
`docs/guides/find-limits.md` (public, indexed) — explains the cap, the four memory
sources the auto-config considers, the three escape valves (`maxQueryLimit` ,
`reservedQueryMemory` , pagination), and when to use which. Explicit "pagination is
the future-proof pattern" callout — the cap can get tighter in 8.0 (Datomic-style
`Db.find()` may make per-call limits stricter to keep snapshot semantics cheap),
but pagination keeps working unchanged.
`docs/api/README.md` `find()` entry gets a one-paragraph `limit` tip + pointer
to the new guide.
### Tests
- **New `tests/integration/find-limits.test.ts` ** (9 tests): below-cap silent
pass; soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines); soft-tier message format (names all three escape
valves + docs link); soft-tier message includes caller location; hard-tier
throws; hard-tier message format same as soft-tier; consumer `maxQueryLimit`
override raises the cap and shifts both tiers accordingly; **the pre-7.30.2
regression scenario** explicitly covered — `limit: 10_000` against a
`maxQueryLimit: 9000` cap now warns and passes instead of throwing.
- Existing unit tests in `tests/unit/utils/memoryLimits.test.ts` updated to
reflect the recalibrated formula (5 tests: hardcoded values for
container / reserved / free-memory paths bumped 4×
- Existing `paramValidation.test.ts` "should auto-limit based on system memory"
test extended to cover the new three-tier semantics (below-cap pass /
soft-tier silent / hard-tier throw).
- All prior suites still green: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
### Cortex compatibility
**No Cortex changes required for 7.30.2.** Every change is JS-side: formula
recalibration runs in `ValidationConfig.constructor()` , two-tier enforcement runs
in `validateFindParams()` , both fire before any storage/index/Cortex call. Cortex
2.x and the pending Cortex 3.0 work both stay compatible without code changes.
The new `docs/guides/find-limits.md` calls out that 8.0's Datomic-style `Db.find()`
may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point
2026-06-11 10:42:34 -07:00
whose rollout staging now also covers query limits.
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
### What consumers should do
- **If you've been carrying a `limit: 9000` workaround for the 7.30.0–
cap:** drop it on bump to 7.30.2 — existing `limit: 10_000` patterns now
warn (teaching the recipe) but no longer throw. The warning is
one-time-per-call-site, so production logs don't spam.
- **All other consumers:** no action required if no enforcement-error was being
hit. If you see new `[Brainy]` warnings in logs after upgrade, follow the
recipe in the warning or read `docs/guides/find-limits.md` .
- **SDK wrappers:** wrapper-level pools that auto-construct `Brainy` instances
should consider surfacing `maxQueryLimit` / `reservedQueryMemory` as
consumer-facing config; Brainy now documents the knob explicitly.
2026-02-28 10:07:34 -08:00
---
fix: internal subtype consistency + brain.audit() diagnostic + improved enforcement errors
Brainy 7.30 shipped opt-in subtype enforcement; SDK 3.20.0 then registered
SDK_CORE_VOCABULARY on every consumer's brain (Event, Collection, Message,
Contract, Media, Document NounTypes). On 2026-06-08 Venue's /book flow went 500
because their brain.add({ type: NounType.Event, ... }) call sites lacked
subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write
paths that also omit subtype — any consumer running the same vocabulary would
have hit Brainy's infrastructure paths next. 7.30.1 closes both gaps before
8.0 makes strict mode the default.
Additive across the board. Zero behavior change for consumers not using strict
mode. Every change is JS-side — Cortex needs no work for 7.30.1.
NEW — brain.audit() diagnostic
- Read-only method walking storage.getNouns() / getVerbs() pagination
- Returns { entitiesWithoutSubtype: { type: count }, relationshipsWithoutSubtype,
total, scanned, recommendation }
- VFS infrastructure entities excluded by default (they bypass enforcement via
isVFSEntity marker); pass { includeVFS: true } to surface them
- The companion to migrateField (7.x) and fillSubtypes (8.0): tells consumers
exactly what would break under strict enforcement, deterministically
NEW — Improved enforcement error messages
- Caller's source location extracted from Error().stack so users see their own
call site, not a Brainy internal frame
- Specific guidance branches: registered vocabulary → "Pass one of: a, b, c";
brain-wide strict mode → mentions the except clause; otherwise → registration
recipe via brain.requireSubtype()
- Documentation link to the canonical migration recipe
- Same shape for noun and verb enforcement
NEW — CLI --subtype flag
- brainy add and brainy relate gain -s/--subtype <value>
- Defaults to 'cli-add' / 'cli-relate' so the CLI works against strict-mode
brains without the user needing to know the vocabulary in advance
INTERNAL — every Brainy write path now sets subtype
- VFS Contains edges (5 sites at lines 503/905/1694/1772/1886) → 'vfs-contains'
- VFS symlink entity → 'vfs-symlink' (NEW — distinct from 'vfs-file')
- VFS copy-file → preserves source subtype, falls back to 'vfs-file'
- VFS symlink also adopts the isVFSEntity infrastructure marker so it bypasses
enforcement in strict mode
- Aggregation materializer (Measurement entities) → 'materialized-aggregate'
- ImportCoordinator (3 sites): document → 'import-source'; entities →
options.defaultSubtype ?? 'imported'; placeholder → 'import-placeholder'
- SmartImportOrchestrator (4 entity sites + 2 batch relate sites): same
precedence (extractor → options.defaultSubtype → 'imported')
- EntityDeduplicator → candidate.subtype ?? 'imported'
- UniversalImportAPI → extractor → 'extracted' for both entities and relations
- NeuralImport → adds defaultSubtype to NeuralImportOptions; precedence same
- GoogleSheetsIntegration → request body 'subtype' ?? 'imported-from-sheets'
- ODataIntegration → request body 'Subtype' ?? 'imported-from-odata'
- MCP client message storage → 'mcp-message' (also fixes pre-existing missing
data field and missing type by aliasing from the prior text field)
Side-effect fix: storage.getNouns() paginated now surfaces subtype to top-level
- Single-noun getNoun() already did this in 7.30; the paginated path was missed
- Without this fix brain.audit() saw missing subtype on entities that actually
had one (caught by the strict-mode self-test before release)
NEW — tests/integration/strict-mode-self-test.test.ts (13 tests)
- Creates a brain under the exact SDK_CORE_VOCABULARY shape Venue hit + brain-
wide strict mode
- Exercises every internal Brainy path: VFS root + mkdir + writeFile + cp + mv
+ ln + symlink; aggregation engine; audit diagnostic with includeVFS toggle
- Validates error message UX: caller location, vocabulary guidance, brain-wide
strict mode guidance, off-vocabulary value reporting
Docs
- New "Strict mode in practice" section in docs/guides/subtypes-and-facets.md
covering the SDK_CORE_VOCABULARY pattern, 4-step migration recipe
(audit → migrateField → hand-fix → re-audit), the Brainy-internal label
reference table, and an 8.0 forward-look on fillSubtypes()
- docs/api/README.md: new audit() entry, strict-mode tips on add() and relate()
- RELEASES.md: full 7.30.1 entry
Cortex parity (forward-looking, not blocking 7.30.1)
- 6th open question added to .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md: native
fast path for audit() and fillSubtypes() via column-store null-subtype
bitmap for billion-scale brains
- Cortex should add a parity test mirroring strict-mode-self-test.test.ts
against their native paths to catch any latent bug where native writes
bypass JS validation
- Brainy-internal subtype labels become a documented part of the 8.0 contract
(useful for Cortex telemetry surfacing Brainy-managed infrastructure %)
Verification
- npx tsc --noEmit: clean
- npm test: 1468/1468 unit
- 7.29 noun integration suite: 26/26 (no regression)
- 7.30 verb subtype + enforcement integration suite: 30/30 (no regression)
- New strict-mode-self-test integration suite: 13/13
- npm run build: clean
- Closed-source product reference audit: clean
Addresses VE-SUBTYPE-MIGRATION (Venue's reported request) and ships internal
labels Venue did NOT ask for but that would have broken them next under their
own vocabulary registration.
2026-06-08 11:31:47 -07:00
## v7.30.1 — 2026-06-08
**Affected products:** anyone running a brain where a platform layer (SDK, framework wrapper)
has registered `brain.requireSubtype()` rules on common NounTypes, AND anyone preparing for
the upcoming Brainy 8.0 default-on strict mode. Additive; drop-in from 7.30.0. No behavior
change for consumers not using strict-mode enforcement.
### Why
Production incident 2026-06-08: a consumer using SDK 3.20.0 (which registers
`requireSubtype()` rules on `NounType.{Event, Collection, Message, Contract, Media, Document}` )
saw their booking flow start returning 500s because `brain.add({ type: NounType.Event, ... })`
calls in their codebase lacked `subtype` . An audit of Brainy's OWN source revealed 14 HIGH-risk
internal write paths that also omit subtype — VFS move/copy/symlink edges, aggregation
materializer, neural extraction, importers, integrations (Sheets/OData), MCP client, CLI.
Any consumer running `requireSubtype()` rules on those NounTypes was one step away from breaking
Brainy's own infrastructure paths, not just their own code. 7.30.1 closes both gaps before
8.0 ships and makes strict mode the default.
### New — `brain.audit()` diagnostic
Find entities and relationships missing a `subtype` value, grouped by type. The companion to
`migrateField()` (and to 8.0's `fillSubtypes()` ): answers "what would break if I enabled
strict subtype enforcement?".
```typescript
const report = await brain.audit()
// {
// entitiesWithoutSubtype: { event: 24, document: 3 },
// relationshipsWithoutSubtype: { relatedTo: 1402 },
// total: 1429,
// scanned: 8400,
// recommendation: 'Found 1429 entries without subtype. Migrate via `brain.migrateField()`
// (7.x) — or wait for `brain.fillSubtypes()` (8.0) which closes the same
// gap with caller-supplied rules.'
// }
```
VFS infrastructure entities are excluded by default (they bypass enforcement via
`metadata.isVFSEntity` markers). Pass `{ includeVFS: true }` to surface them.
### New — Improved enforcement error messages
The error fired when subtype enforcement rejects a write now includes:
1. **The caller's source location** — extracted from the JavaScript stack so you see your own
call site, not a Brainy internal frame. Eliminates the "grep your repo for `brain.add` " step.
2. **Specific guidance** — points at the registered vocabulary when one exists; mentions
brain-wide strict mode and the `except` escape valve when not; otherwise the
`brain.requireSubtype()` registration recipe.
3. **A documentation link** — `https://soulcraft.com/docs/guides/subtypes-and-facets#strict-mode`
for the canonical migration recipe.
Before:
```
add(): NounType.Event requires subtype but got undefined. Register vocabulary via brain.requireSubtype().
```
After:
```
add(): NounType.event requires subtype but got undefined.
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
at OrderService.getOrCreateByToken (/app/src/orders/draft.ts:42:23)
Pass one of: standard, recurring, milestone.
fix: internal subtype consistency + brain.audit() diagnostic + improved enforcement errors
Brainy 7.30 shipped opt-in subtype enforcement; SDK 3.20.0 then registered
SDK_CORE_VOCABULARY on every consumer's brain (Event, Collection, Message,
Contract, Media, Document NounTypes). On 2026-06-08 Venue's /book flow went 500
because their brain.add({ type: NounType.Event, ... }) call sites lacked
subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write
paths that also omit subtype — any consumer running the same vocabulary would
have hit Brainy's infrastructure paths next. 7.30.1 closes both gaps before
8.0 makes strict mode the default.
Additive across the board. Zero behavior change for consumers not using strict
mode. Every change is JS-side — Cortex needs no work for 7.30.1.
NEW — brain.audit() diagnostic
- Read-only method walking storage.getNouns() / getVerbs() pagination
- Returns { entitiesWithoutSubtype: { type: count }, relationshipsWithoutSubtype,
total, scanned, recommendation }
- VFS infrastructure entities excluded by default (they bypass enforcement via
isVFSEntity marker); pass { includeVFS: true } to surface them
- The companion to migrateField (7.x) and fillSubtypes (8.0): tells consumers
exactly what would break under strict enforcement, deterministically
NEW — Improved enforcement error messages
- Caller's source location extracted from Error().stack so users see their own
call site, not a Brainy internal frame
- Specific guidance branches: registered vocabulary → "Pass one of: a, b, c";
brain-wide strict mode → mentions the except clause; otherwise → registration
recipe via brain.requireSubtype()
- Documentation link to the canonical migration recipe
- Same shape for noun and verb enforcement
NEW — CLI --subtype flag
- brainy add and brainy relate gain -s/--subtype <value>
- Defaults to 'cli-add' / 'cli-relate' so the CLI works against strict-mode
brains without the user needing to know the vocabulary in advance
INTERNAL — every Brainy write path now sets subtype
- VFS Contains edges (5 sites at lines 503/905/1694/1772/1886) → 'vfs-contains'
- VFS symlink entity → 'vfs-symlink' (NEW — distinct from 'vfs-file')
- VFS copy-file → preserves source subtype, falls back to 'vfs-file'
- VFS symlink also adopts the isVFSEntity infrastructure marker so it bypasses
enforcement in strict mode
- Aggregation materializer (Measurement entities) → 'materialized-aggregate'
- ImportCoordinator (3 sites): document → 'import-source'; entities →
options.defaultSubtype ?? 'imported'; placeholder → 'import-placeholder'
- SmartImportOrchestrator (4 entity sites + 2 batch relate sites): same
precedence (extractor → options.defaultSubtype → 'imported')
- EntityDeduplicator → candidate.subtype ?? 'imported'
- UniversalImportAPI → extractor → 'extracted' for both entities and relations
- NeuralImport → adds defaultSubtype to NeuralImportOptions; precedence same
- GoogleSheetsIntegration → request body 'subtype' ?? 'imported-from-sheets'
- ODataIntegration → request body 'Subtype' ?? 'imported-from-odata'
- MCP client message storage → 'mcp-message' (also fixes pre-existing missing
data field and missing type by aliasing from the prior text field)
Side-effect fix: storage.getNouns() paginated now surfaces subtype to top-level
- Single-noun getNoun() already did this in 7.30; the paginated path was missed
- Without this fix brain.audit() saw missing subtype on entities that actually
had one (caught by the strict-mode self-test before release)
NEW — tests/integration/strict-mode-self-test.test.ts (13 tests)
- Creates a brain under the exact SDK_CORE_VOCABULARY shape Venue hit + brain-
wide strict mode
- Exercises every internal Brainy path: VFS root + mkdir + writeFile + cp + mv
+ ln + symlink; aggregation engine; audit diagnostic with includeVFS toggle
- Validates error message UX: caller location, vocabulary guidance, brain-wide
strict mode guidance, off-vocabulary value reporting
Docs
- New "Strict mode in practice" section in docs/guides/subtypes-and-facets.md
covering the SDK_CORE_VOCABULARY pattern, 4-step migration recipe
(audit → migrateField → hand-fix → re-audit), the Brainy-internal label
reference table, and an 8.0 forward-look on fillSubtypes()
- docs/api/README.md: new audit() entry, strict-mode tips on add() and relate()
- RELEASES.md: full 7.30.1 entry
Cortex parity (forward-looking, not blocking 7.30.1)
- 6th open question added to .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md: native
fast path for audit() and fillSubtypes() via column-store null-subtype
bitmap for billion-scale brains
- Cortex should add a parity test mirroring strict-mode-self-test.test.ts
against their native paths to catch any latent bug where native writes
bypass JS validation
- Brainy-internal subtype labels become a documented part of the 8.0 contract
(useful for Cortex telemetry surfacing Brainy-managed infrastructure %)
Verification
- npx tsc --noEmit: clean
- npm test: 1468/1468 unit
- 7.29 noun integration suite: 26/26 (no regression)
- 7.30 verb subtype + enforcement integration suite: 30/30 (no regression)
- New strict-mode-self-test integration suite: 13/13
- npm run build: clean
- Closed-source product reference audit: clean
Addresses VE-SUBTYPE-MIGRATION (Venue's reported request) and ships internal
labels Venue did NOT ask for but that would have broken them next under their
own vocabulary registration.
2026-06-08 11:31:47 -07:00
Migration recipe: https://soulcraft.com/docs/guides/subtypes-and-facets#strict -mode
```
### Internal subtype labels — Brainy's own infrastructure paths
Every internal Brainy write path now sets a stable, queryable `subtype` . Consumers don't need
to do anything for these — they're documented here so you can query Brainy-managed data:
| Code path | NounType / VerbType | Subtype label |
|---|---|---|
| VFS root directory `/` | `Collection` | `'vfs-root'` |
| VFS subdirectories | `Collection` | `'vfs-directory'` |
| VFS files | mime-driven (e.g. `Document` /`Code` /`Image` ) | `'vfs-file'` |
| VFS symlinks | `File` | `'vfs-symlink'` (NEW — distinct from `'vfs-file'` ) |
| VFS Contains edges (create + move/copy/symlink/batch) | `Contains` | `'vfs-contains'` |
| Aggregation materialized output | `Measurement` | `'materialized-aggregate'` |
| Import-source provenance entity | `Document` | `'import-source'` |
| Importer-extracted entities | extractor-driven type | `'imported'` |
| Importer placeholder targets | `Thing` | `'import-placeholder'` |
| Neural extraction | extractor-driven type | `'extracted'` |
| GoogleSheets API entity writes | request-driven | `'imported-from-sheets'` |
| OData API entity writes | request-driven | `'imported-from-odata'` |
| MCP message storage | `Message` | `'mcp-message'` |
| `brainy add` CLI default | user-supplied type | `'cli-add'` |
| `brainy relate` CLI default | user-supplied verb | `'cli-relate'` |
Query examples:
```typescript
// Every VFS-managed file in your brain
await brain.find({ subtype: 'vfs-file' })
// Document breakdown — distinguishes import-source from extracted/imported/user content
brain.counts.bySubtype(NounType.Document)
// → { 'import-source': 12, 'imported': 847, 'extracted': 34, 'vfs-file': 102, ... }
```
### Caller-supplied `defaultSubtype` on importers and extraction
Importer + extraction paths now accept a caller-supplied `defaultSubtype` config so consumers
can tag a whole batch with their own provenance label instead of the Brainy default:
```typescript
// SmartImportOrchestrator / ImportCoordinator / NeuralImport all accept this
await brain.importer.import(file, {
defaultSubtype: 'customer-upload-2026q2', // your batch label
// ...
})
```
Precedence: extractor-set subtype (highest) → caller's `defaultSubtype` → Brainy default
(`'imported'` for importers, `'extracted'` for extractors).
### CLI `--subtype` flag
`brainy add` and `brainy relate` gain a `--subtype <value>` (`-s` ) flag for use with
strict-mode brains:
```bash
brainy add "Avery Brooks — runs the AI lab" --type person --subtype employee
brainy relate alice manages bob --subtype direct
```
When the flag isn't supplied, the CLI uses `'cli-add'` / `'cli-relate'` as defaults so
ad-hoc CLI usage still works against strict-mode brains.
### Cortex compatibility
**No Cortex changes required for 7.30.1.** Every change is JS-side: internal subtype labels are
arbitrary strings stored transparently by Cortex; `brain.audit()` runs purely on JS via existing
`storage.getNouns()` / `getVerbs()` pagination; the CLI is JS-only; error messages fire in
Brainy before any native call.
**For Cortex 3.0 (forward-looking, not blocking):**
- **Native `audit()` proxy.** For billion-scale brains, `audit()` walks every entity (O(N)).
A native implementation reading from a "null-subtype" bitmap in the column store would be
2026-06-11 10:42:34 -07:00
O(buckets).
fix: internal subtype consistency + brain.audit() diagnostic + improved enforcement errors
Brainy 7.30 shipped opt-in subtype enforcement; SDK 3.20.0 then registered
SDK_CORE_VOCABULARY on every consumer's brain (Event, Collection, Message,
Contract, Media, Document NounTypes). On 2026-06-08 Venue's /book flow went 500
because their brain.add({ type: NounType.Event, ... }) call sites lacked
subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write
paths that also omit subtype — any consumer running the same vocabulary would
have hit Brainy's infrastructure paths next. 7.30.1 closes both gaps before
8.0 makes strict mode the default.
Additive across the board. Zero behavior change for consumers not using strict
mode. Every change is JS-side — Cortex needs no work for 7.30.1.
NEW — brain.audit() diagnostic
- Read-only method walking storage.getNouns() / getVerbs() pagination
- Returns { entitiesWithoutSubtype: { type: count }, relationshipsWithoutSubtype,
total, scanned, recommendation }
- VFS infrastructure entities excluded by default (they bypass enforcement via
isVFSEntity marker); pass { includeVFS: true } to surface them
- The companion to migrateField (7.x) and fillSubtypes (8.0): tells consumers
exactly what would break under strict enforcement, deterministically
NEW — Improved enforcement error messages
- Caller's source location extracted from Error().stack so users see their own
call site, not a Brainy internal frame
- Specific guidance branches: registered vocabulary → "Pass one of: a, b, c";
brain-wide strict mode → mentions the except clause; otherwise → registration
recipe via brain.requireSubtype()
- Documentation link to the canonical migration recipe
- Same shape for noun and verb enforcement
NEW — CLI --subtype flag
- brainy add and brainy relate gain -s/--subtype <value>
- Defaults to 'cli-add' / 'cli-relate' so the CLI works against strict-mode
brains without the user needing to know the vocabulary in advance
INTERNAL — every Brainy write path now sets subtype
- VFS Contains edges (5 sites at lines 503/905/1694/1772/1886) → 'vfs-contains'
- VFS symlink entity → 'vfs-symlink' (NEW — distinct from 'vfs-file')
- VFS copy-file → preserves source subtype, falls back to 'vfs-file'
- VFS symlink also adopts the isVFSEntity infrastructure marker so it bypasses
enforcement in strict mode
- Aggregation materializer (Measurement entities) → 'materialized-aggregate'
- ImportCoordinator (3 sites): document → 'import-source'; entities →
options.defaultSubtype ?? 'imported'; placeholder → 'import-placeholder'
- SmartImportOrchestrator (4 entity sites + 2 batch relate sites): same
precedence (extractor → options.defaultSubtype → 'imported')
- EntityDeduplicator → candidate.subtype ?? 'imported'
- UniversalImportAPI → extractor → 'extracted' for both entities and relations
- NeuralImport → adds defaultSubtype to NeuralImportOptions; precedence same
- GoogleSheetsIntegration → request body 'subtype' ?? 'imported-from-sheets'
- ODataIntegration → request body 'Subtype' ?? 'imported-from-odata'
- MCP client message storage → 'mcp-message' (also fixes pre-existing missing
data field and missing type by aliasing from the prior text field)
Side-effect fix: storage.getNouns() paginated now surfaces subtype to top-level
- Single-noun getNoun() already did this in 7.30; the paginated path was missed
- Without this fix brain.audit() saw missing subtype on entities that actually
had one (caught by the strict-mode self-test before release)
NEW — tests/integration/strict-mode-self-test.test.ts (13 tests)
- Creates a brain under the exact SDK_CORE_VOCABULARY shape Venue hit + brain-
wide strict mode
- Exercises every internal Brainy path: VFS root + mkdir + writeFile + cp + mv
+ ln + symlink; aggregation engine; audit diagnostic with includeVFS toggle
- Validates error message UX: caller location, vocabulary guidance, brain-wide
strict mode guidance, off-vocabulary value reporting
Docs
- New "Strict mode in practice" section in docs/guides/subtypes-and-facets.md
covering the SDK_CORE_VOCABULARY pattern, 4-step migration recipe
(audit → migrateField → hand-fix → re-audit), the Brainy-internal label
reference table, and an 8.0 forward-look on fillSubtypes()
- docs/api/README.md: new audit() entry, strict-mode tips on add() and relate()
- RELEASES.md: full 7.30.1 entry
Cortex parity (forward-looking, not blocking 7.30.1)
- 6th open question added to .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md: native
fast path for audit() and fillSubtypes() via column-store null-subtype
bitmap for billion-scale brains
- Cortex should add a parity test mirroring strict-mode-self-test.test.ts
against their native paths to catch any latent bug where native writes
bypass JS validation
- Brainy-internal subtype labels become a documented part of the 8.0 contract
(useful for Cortex telemetry surfacing Brainy-managed infrastructure %)
Verification
- npx tsc --noEmit: clean
- npm test: 1468/1468 unit
- 7.29 noun integration suite: 26/26 (no regression)
- 7.30 verb subtype + enforcement integration suite: 30/30 (no regression)
- New strict-mode-self-test integration suite: 13/13
- npm run build: clean
- Closed-source product reference audit: clean
Addresses VE-SUBTYPE-MIGRATION (Venue's reported request) and ships internal
labels Venue did NOT ask for but that would have broken them next under their
own vocabulary registration.
2026-06-08 11:31:47 -07:00
- **Strict-mode parity test.** Cortex should mirror Brainy's new
`tests/integration/strict-mode-self-test.test.ts` against their native paths to catch any
latent bug where native writes bypass JS validation.
- **Reserved-label awareness (optional).** Brainy's internal labels (`'vfs-*'` ,
`'materialized-aggregate'` , `'imported'` , `'extracted'` , `'mcp-message'` , `'cli-*'` ) become
a documented part of the 8.0 contract; useful for telemetry that surfaces "X% of entities are
Brainy-managed infrastructure".
### Docs
Updated `docs/guides/subtypes-and-facets.md` with a new "Strict mode in practice" section
covering the SDK_CORE_VOCABULARY pattern, a 4-step migration recipe, the Brainy-internal label
reference table, and an 8.0 forward-look. `docs/api/README.md` documents `brain.audit()` and
adds a strict-mode tip to the `add()` / `relate()` reference entries.
### Tests
- **New `tests/integration/strict-mode-self-test.test.ts` ** (13 tests): creates a brain under
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
a realistic consumer vocabulary shape + brain-wide strict mode, then exercises every
fix: internal subtype consistency + brain.audit() diagnostic + improved enforcement errors
Brainy 7.30 shipped opt-in subtype enforcement; SDK 3.20.0 then registered
SDK_CORE_VOCABULARY on every consumer's brain (Event, Collection, Message,
Contract, Media, Document NounTypes). On 2026-06-08 Venue's /book flow went 500
because their brain.add({ type: NounType.Event, ... }) call sites lacked
subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write
paths that also omit subtype — any consumer running the same vocabulary would
have hit Brainy's infrastructure paths next. 7.30.1 closes both gaps before
8.0 makes strict mode the default.
Additive across the board. Zero behavior change for consumers not using strict
mode. Every change is JS-side — Cortex needs no work for 7.30.1.
NEW — brain.audit() diagnostic
- Read-only method walking storage.getNouns() / getVerbs() pagination
- Returns { entitiesWithoutSubtype: { type: count }, relationshipsWithoutSubtype,
total, scanned, recommendation }
- VFS infrastructure entities excluded by default (they bypass enforcement via
isVFSEntity marker); pass { includeVFS: true } to surface them
- The companion to migrateField (7.x) and fillSubtypes (8.0): tells consumers
exactly what would break under strict enforcement, deterministically
NEW — Improved enforcement error messages
- Caller's source location extracted from Error().stack so users see their own
call site, not a Brainy internal frame
- Specific guidance branches: registered vocabulary → "Pass one of: a, b, c";
brain-wide strict mode → mentions the except clause; otherwise → registration
recipe via brain.requireSubtype()
- Documentation link to the canonical migration recipe
- Same shape for noun and verb enforcement
NEW — CLI --subtype flag
- brainy add and brainy relate gain -s/--subtype <value>
- Defaults to 'cli-add' / 'cli-relate' so the CLI works against strict-mode
brains without the user needing to know the vocabulary in advance
INTERNAL — every Brainy write path now sets subtype
- VFS Contains edges (5 sites at lines 503/905/1694/1772/1886) → 'vfs-contains'
- VFS symlink entity → 'vfs-symlink' (NEW — distinct from 'vfs-file')
- VFS copy-file → preserves source subtype, falls back to 'vfs-file'
- VFS symlink also adopts the isVFSEntity infrastructure marker so it bypasses
enforcement in strict mode
- Aggregation materializer (Measurement entities) → 'materialized-aggregate'
- ImportCoordinator (3 sites): document → 'import-source'; entities →
options.defaultSubtype ?? 'imported'; placeholder → 'import-placeholder'
- SmartImportOrchestrator (4 entity sites + 2 batch relate sites): same
precedence (extractor → options.defaultSubtype → 'imported')
- EntityDeduplicator → candidate.subtype ?? 'imported'
- UniversalImportAPI → extractor → 'extracted' for both entities and relations
- NeuralImport → adds defaultSubtype to NeuralImportOptions; precedence same
- GoogleSheetsIntegration → request body 'subtype' ?? 'imported-from-sheets'
- ODataIntegration → request body 'Subtype' ?? 'imported-from-odata'
- MCP client message storage → 'mcp-message' (also fixes pre-existing missing
data field and missing type by aliasing from the prior text field)
Side-effect fix: storage.getNouns() paginated now surfaces subtype to top-level
- Single-noun getNoun() already did this in 7.30; the paginated path was missed
- Without this fix brain.audit() saw missing subtype on entities that actually
had one (caught by the strict-mode self-test before release)
NEW — tests/integration/strict-mode-self-test.test.ts (13 tests)
- Creates a brain under the exact SDK_CORE_VOCABULARY shape Venue hit + brain-
wide strict mode
- Exercises every internal Brainy path: VFS root + mkdir + writeFile + cp + mv
+ ln + symlink; aggregation engine; audit diagnostic with includeVFS toggle
- Validates error message UX: caller location, vocabulary guidance, brain-wide
strict mode guidance, off-vocabulary value reporting
Docs
- New "Strict mode in practice" section in docs/guides/subtypes-and-facets.md
covering the SDK_CORE_VOCABULARY pattern, 4-step migration recipe
(audit → migrateField → hand-fix → re-audit), the Brainy-internal label
reference table, and an 8.0 forward-look on fillSubtypes()
- docs/api/README.md: new audit() entry, strict-mode tips on add() and relate()
- RELEASES.md: full 7.30.1 entry
Cortex parity (forward-looking, not blocking 7.30.1)
- 6th open question added to .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md: native
fast path for audit() and fillSubtypes() via column-store null-subtype
bitmap for billion-scale brains
- Cortex should add a parity test mirroring strict-mode-self-test.test.ts
against their native paths to catch any latent bug where native writes
bypass JS validation
- Brainy-internal subtype labels become a documented part of the 8.0 contract
(useful for Cortex telemetry surfacing Brainy-managed infrastructure %)
Verification
- npx tsc --noEmit: clean
- npm test: 1468/1468 unit
- 7.29 noun integration suite: 26/26 (no regression)
- 7.30 verb subtype + enforcement integration suite: 30/30 (no regression)
- New strict-mode-self-test integration suite: 13/13
- npm run build: clean
- Closed-source product reference audit: clean
Addresses VE-SUBTYPE-MIGRATION (Venue's reported request) and ships internal
labels Venue did NOT ask for but that would have broken them next under their
own vocabulary registration.
2026-06-08 11:31:47 -07:00
internal Brainy path (VFS root/mkdir/writeFile/cp/mv/ln, aggregation engine, audit
diagnostic, error-message UX). Zero rejections expected.
- Existing 7.30.0 + 7.29.0 integration suites unchanged: 26/26 + 30/30.
- Unit suite unchanged: 1468/1468.
---
feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.
Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
with subtype from metadata
Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update
Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
_system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata
Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes
Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite
Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
Registers per-type rules with optional values whitelist; composes with the
brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
brain's own VFS writes don't get rejected when strict mode is on
VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
and distinguish Brainy's VFS Collections from user-created Collections
Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
Enforcement section. New full reference at the bottom split into Layer 1
(nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
three verb-side counts methods, requireSubtype(), and the brain-wide
constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix
Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
covering V1 round-trips, V1 set membership, updateRelation in place,
updateRelation preservation, V2 counts breakdown + point + topN + distinct,
V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
entity kinds, V4 readBoth preservation, V5 per-type required rejection,
V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.
Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean
Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
documents the contract upgrade Cortex 3.0 implements against: required-by-
default subtype, SubtypeRegistry typing hook, native simplification,
multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
## v7.30.0 — 2026-06-05
**Affected products:** consumers modeling typed relationships with sub-classification
(direct vs dotted-line management; spouse / sibling / colleague; collaborator vs competitor;
etc.), and anyone wanting to enforce the pairing of `type` + `subtype` on every write.
Additive; drop-in from 7.29.x. No deprecations.
### Symmetric — `subtype` on relationships (parity with 7.29.0 nouns)
`subtype?: string` is now a first-class standard field on every relationship, mirroring the
noun-side work shipped in 7.29.0. Verbs and nouns are now first-class peers — every API
available on the noun side has a verb-side mirror.
```typescript
const ceoId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Avery' })
const vpId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Jordan' })
await brain.relate({
from: ceoId,
to: vpId,
type: VerbType.ReportsTo,
subtype: 'direct' // sub-classification on the edge
})
```
**Read & filter:**
```typescript
// Fast-path filter — column-store hit, not metadata fallback
const direct = await brain.getRelations({
from: ceoId,
type: VerbType.ReportsTo,
subtype: 'direct'
})
// Set membership
const all = await brain.getRelations({
from: ceoId,
type: VerbType.ReportsTo,
subtype: ['direct', 'dotted-line']
})
// Traversal filter (depth-1 in JS; multi-hop lands on Cortex native)
const reports = await brain.find({
connected: { from: ceoId, via: VerbType.ReportsTo, subtype: 'direct', depth: 1 }
})
```
### New — `updateRelation()` closes a long-standing gap
Verbs previously had no update method — the only way to change a relationship was
delete-then-recreate, which lost the relation id. 7.30 ships `brain.updateRelation()` :
```typescript
await brain.updateRelation({ id: relId, subtype: 'dotted-line' })
await brain.updateRelation({ id: relId, weight: 0.5, confidence: 0.9 })
// Change verb type — re-indexes in graph adjacency, id preserved
await brain.updateRelation({ id: relId, type: VerbType.WorksWith })
```
### New — O(1) verb subtype counts via the persisted rollup
`_system/verb-subtype-statistics.json` mirrors the noun-side rollup shipped in 7.29.0.
Per-VerbType-per-subtype counts are maintained incrementally and persisted; reads are O(1):
```typescript
brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
// → { direct: 12, 'dotted-line': 3 }
brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct') // O(1) point
// → 12
brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3)
// → [['direct', 12], ['dotted-line', 3]]
brain.relationshipSubtypesOf(VerbType.ReportsTo)
// → ['direct', 'dotted-line']
```
### New — `brain.requireSubtype(type, options)` per-type enforcement
Unified API for noun OR verb types — register specific types as requiring a subtype,
optionally with a fixed vocabulary:
```typescript
brain.requireSubtype(NounType.Person, {
values: ['employee', 'customer', 'vendor'],
required: true
})
brain.requireSubtype(VerbType.ReportsTo, {
values: ['direct', 'dotted-line'],
required: true
})
// Now this throws — Person requires subtype:
await brain.add({ type: NounType.Person, data: 'no subtype' })
// And this throws — 'matrix' isn't in the registered vocabulary:
await brain.relate({ from: a, to: b, type: VerbType.ReportsTo, subtype: 'matrix' })
```
### New — Brain-wide strict mode
`new Brainy({ requireSubtype: true })` enforces subtype on every public write across the
whole brain. Composes with per-type rules; per-type rules win when both apply.
```typescript
// Every write must include subtype
const brain = new Brainy({ requireSubtype: true })
// Exempt specific types (e.g. catch-all Thing)
const brain2 = new Brainy({
requireSubtype: { except: [NounType.Thing, NounType.Custom] }
})
```
When strict mode is on:
- Every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()`
rejects writes missing a subtype on a non-exempt type.
- `addMany()` and `relateMany()` validate every item BEFORE any storage write —
atomic-fail semantics, no partial writes.
- Brainy's own infrastructure writes (VFS root, directories, files) bypass via the
`metadata.isVFSEntity: true` marker so existing consumers' VFS usage continues to work.
The brain-wide flag becomes the default in 8.0.0; the type-level `subtype` field becomes
required at the type system level. The full 8.0 contract upgrade is coordinated through the
internal platform handoff (`CTX-SUBTYPE-8.0-CONTRACT` ).
### `migrateField()` extended to verbs
The migration helper shipped in 7.29.0 now walks verbs too via the new `entityKind` option:
```typescript
// Migrate verb-side metadata.kind → top-level subtype
await brain.migrateField({
from: 'metadata.kind',
to: 'subtype',
entityKind: 'verb'
})
// Or walk nouns and verbs in one pass
await brain.migrateField({
from: 'metadata.kind',
to: 'subtype',
entityKind: 'both'
})
```
Default is `entityKind: 'noun'` (backward-compatible).
### Symmetry — noun + verb capability matrix
7.30 closes every gap between nouns and verbs. Every capability available on the noun side
has a verb-side mirror:
| Capability | Nouns | Verbs |
|---|---|---|
| `subtype` top-level field | ✓ (7.29) | ✓ (7.30 new) |
| Standard-field set | `STANDARD_ENTITY_FIELDS` | `STANDARD_VERB_FIELDS` (new) |
| Field resolver helper | `resolveEntityField` | `resolveVerbField` (new) |
| Statistics rollup | `_system/subtype-statistics.json` | `_system/verb-subtype-statistics.json` (new) |
| Counts breakdown | `counts.bySubtype` / `topSubtypes` / `subtypesOf` | `counts.byRelationshipSubtype` / `topRelationshipSubtypes` / `relationshipSubtypesOf` (new) |
| Fast-path filter | `find({type, subtype})` | `getRelations({type, subtype})` + `find({connected, subtype})` (new) |
| Update method | `update()` | `updateRelation()` (new — closed pre-7.30 gap) |
| Migration helper | `migrateField()` | `migrateField({entityKind: 'verb'\|'both'})` (new) |
| Enforcement | `requireSubtype(NounType, ...)` | `requireSubtype(VerbType, ...)` — one unified API |
| Brain-wide strict mode | `new Brainy({ requireSubtype })` covers both | same |
### Cortex compatibility
Verb subtype works under Cortex out of the box via auto-field-indexing. Cortex parity items
for the verb side (native query planner recognition, `verbSubtypeCountsByType` native rollup,
per-edge subtype on native graph adjacency for multi-hop traversal filtering) ship in the
next Cortex release. Not a Brainy blocker.
### Docs
Full guide: `docs/guides/subtypes-and-facets.md` (extended with Layer V + Enforcement
sections). The verb subtype + enforcement APIs are documented in `docs/api/README.md` .
---
feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
2026-06-04 17:24:36 -07:00
## v7.29.0 — 2026-06-04
**Affected products:** anyone modeling entities with per-product sub-classification — every
consumer that's been reaching for `metadata.kind` , a custom `metadata.subtype` field, a
`data.kind` shape inside the payload, or similar ad-hoc conventions. Additive; drop-in from
7.28.x. No deprecations.
### New — `subtype` promoted to a top-level standard field
`subtype?: string` is now a first-class standard field on every entity, alongside `type` /
`confidence` / `weight` . Use it as the platform-standard primitive for sub-classifying entities
within a NounType (`Person` → `'employee'` / `'customer'` ; `Document` → `'invoice'` / `'contract'` ;
`Event` → `'milestone'` / `'meeting'` ):
```typescript
await brain.add({
data: 'Avery Brooks — runs the AI lab',
type: NounType.Person,
subtype: 'employee', // top-level write param
metadata: { department: 'ai-lab' }
})
// Fast-path filter — column-store hit, not metadata fallback:
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
// Set membership:
const internal = await brain.find({
type: NounType.Person,
subtype: ['employee', 'contractor']
})
```
Flat string, no hierarchy — your vocabulary, your choice. Brainy stores and counts, never validates.
### New — O(1) subtype counts via the persisted rollup
A new `_system/subtype-statistics.json` rollup is maintained incrementally as entities are added,
updated, and deleted — mirroring the existing `nounCountsByType` machinery with the same self-heal
behavior on poison detection. Counts are O(1) at billion scale:
```typescript
brain.counts.bySubtype(NounType.Person)
// → { employee: 12, customer: 847, vendor: 34 }
brain.counts.bySubtype(NounType.Person, 'employee') // O(1) point count
// → 12
brain.counts.topSubtypes(NounType.Person, 3)
// → [['customer', 847], ['employee', 12], ['vendor', 34]]
brain.subtypesOf(NounType.Person)
// → ['customer', 'employee', 'vendor']
```
### New — `brain.trackField()` for other metadata facets
For facets that aren't the *primary* sub-classification (`status` , `source` , `role` , `paradigm` ),
`trackField` registers a field for cardinality + per-NounType breakdown stats without promoting
each one to a top-level slot. Piggybacks on the existing aggregation engine, so backfill-on-define
applies — registering on a populated brain scans existing entities on the first query:
```typescript
brain.trackField('status', { perType: true })
await brain.counts.byField('status')
// → { todo: 12, doing: 3, done: 47 }
await brain.counts.byField('status', { type: NounType.Task })
// → { todo: 8, doing: 2, done: 30 }
// Opt-in vocabulary validation:
brain.trackField('priority', { values: ['low', 'medium', 'high'] })
// brain.add({ ..., metadata: { priority: 'urgent' } }) → throws
```
### New — generic `brain.migrateField()` for one-shot rewrites
Streams every entity and copies a value from one field path to another. Supports top-level
standard fields, `metadata.*` , and `data.*` paths; idempotent; with optional `readBoth: true`
deprecation window that preserves the source field alongside the new one:
```typescript
// Phase 1: dual-populate (legacy readers still work)
await brain.migrateField({
from: 'metadata.kind',
to: 'subtype',
readBoth: true
})
// ...readers migrate to subtype at their own pace...
// Phase 2: clear the source field
await brain.migrateField({ from: 'metadata.kind', to: 'subtype' })
// → { scanned: 1500, migrated: 1500, skipped: 0, errors: [] }
```
Supports `batchSize` and `onProgress` for large brains.
### Aggregation composition
`subtype` is a standard-field group-by dimension out of the box — no `where:` wrapper, no
metadata-fallback overhead:
```typescript
await brain.find({
aggregate: {
groupBy: ['type', 'subtype'],
metrics: { count: { op: 'COUNT' } }
}
})
// [
// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 },
// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 },
// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 },
// ...
// ]
```
### Cortex compatibility
Subtype works under Cortex out of the box via auto-field-indexing. Cortex will land its own
native-side query-planner recognition + `subtypeCountsByType` native rollup in the next Cortex
release for full parity with `type` 's fast path. Not a Brainy blocker — subtype reads/writes
function correctly with current Cortex.
### Docs
Full guide: `docs/guides/subtypes-and-facets.md` . Subtype is documented as a core primitive
throughout `README.md` , `docs/DATA_MODEL.md` , `docs/architecture/finite-type-system.md` ,
`docs/api/README.md` , and `docs/QUERY_OPERATORS.md` .
---
2026-05-26 14:20:40 -07:00
## v7.24.0 — 2026-05-26
**Affected products:** aggregation/reporting users + anyone calling `extractEntities` on
multi-entity text. Additive; drop-in from 7.23.x.
### New — array-unnest `groupBy` (tag frequency / faceted counts)
A `groupBy` dimension can now be `{ field, unnest: true }` : the field holds an array and the
entity contributes once per **distinct** element. Enables tag-frequency / label-count style
aggregates:
```typescript
brain.defineAggregate({
name: 'tag_frequency',
source: { type: NounType.Document },
groupBy: [{ field: 'tags', unnest: true }],
metrics: { count: { op: 'count' } }
})
// queryAggregate('tag_frequency', { orderBy: 'count', order: 'desc' })
```
Duplicate elements on one entity count once; an entity with an empty/missing array joins no group.
### Perf — entity extraction batch-embeds candidates
`extractEntities` / `extractConcepts` now embed all unique candidate spans in a single
`embedBatch` call instead of one `embed()` per candidate (N sequential model calls before). No
behavior change — and with vectors reliably available, the embedding signal more consistently
reinforces correct types (e.g. clearer Organization confidence). Falls back to per-candidate
embedding if a batch call fails.
---
feat: queryAggregate() + HAVING, plus aggregate backfill, traversal depth/via, extraction typing (BR-ADV-FEATURES-BUN)
New report APIs:
- brain.queryAggregate(name, params) returns the clean AggregateResult[] shape
({ groupKey, metrics, count }) directly, instead of the search-Result wrapper that
find({ aggregate }) returns.
- HAVING: find({ aggregate, having }) and queryAggregate filter groups by computed
metric values (e.g. revenue > 1000), complementing where (which filters group keys).
Evaluated per group: O(groups), independent of entity count.
Fixes (all reproducible on Node; surfaced under Cortex+Bun by Memory):
- Aggregate backfill-on-define: defining an aggregate over a store that already holds
matching entities returned []. It now backfills from existing entities on first query
(storage-agnostic via getNouns), so it works under durable backends that reopen
pre-populated. groupBy:['noun'] resolves to the entity type; find({ aggregate }) rows
expose groupKey/metrics/count at the top level.
- Multi-hop find({ connected: { depth, via } }) honors depth and via at every hop
(was 1-hop only, with verb filtering applied to hop 1 only); BFS bounded by limit.
- extractEntities no longer bleeds a neighbour's type indicator across candidates; each
candidate is typed by its own span. Also fixes the "Dr." title pattern.
Adds real-embedding regression tests; full unit suite green.
2026-05-26 13:55:43 -07:00
## v7.23.0 — 2026-05-26
**Affected products:** anyone using aggregation (stats/dashboards), graph traversal, or entity
extraction. Adds two report APIs and fixes three correctness gaps surfaced by BR-ADV-FEATURES-BUN.
Drop-in upgrade from 7.22.x.
### New — `brain.queryAggregate(name, params)`
A first-class report API returning the clean `AggregateResult[]` shape
(`{ groupKey, metrics, count }[]` ) directly, instead of the search-`Result` wrapper that
`find({ aggregate })` returns. Accepts `where` / `having` / `orderBy` / `order` / `limit` / `offset` .
### New — HAVING (filter groups by metric value)
`find({ aggregate, having: { revenue: { greaterThan: 1000 } } })` (and the same on
`queryAggregate` ) filters groups by their computed metrics — the analytics equivalent of SQL
`HAVING` , complementing `where` (which filters group keys). Evaluated per group: **O(groups),
independent of entity count.**
### Fix — aggregates now backfill when defined over existing data (R1)
Defining an aggregate on a store that already holds matching entities returned `[]` because
write-time hooks only saw *future* writes. It now backfills from existing entities on first query
(one-time scan via `getNouns` , then incremental) — storage-agnostic, so it works under durable
backends (Cortex) where a brain reopens pre-populated. Also: `groupBy:['noun']` now resolves to
the entity type instead of a single null group. `find({ aggregate })` rows now expose
`groupKey` /`metrics` /`count` at the top level (previously only under `.metadata` ).
### Fix — multi-hop `find({ connected: { depth, via } })` (traversal)
`depth` and `via` are now honored at **every hop** (previously only the immediate neighbour was
returned, and verb filtering applied to hop 1 only). The BFS is bounded by `limit` .
### Fix — entity extraction type accuracy (R3)
`extractEntities` no longer lets a type indicator in one candidate's surrounding text bleed onto
neighbours (e.g. "Corp" in "Sarah Chen founded Acme Corp" no longer types "Sarah Chen" as
Organization). Each candidate is typed by its own span; context may only reinforce the same type.
---
2026-05-26 11:32:46 -07:00
## v7.22.1 — 2026-05-26
**Affected products:** Anyone using `extractEntities()` / `extractConcepts()` , native
aggregation (`find({ aggregate })` ), or multi-hop graph traversal
(`find({ connected: { depth } })` ). Three advanced-API correctness fixes — all reproduce on
Node, so they were **not** Bun-specific despite the original report (BR-ADV-FEATURES-BUN).
### Entity/concept extraction no longer returns `[]`
`extractEntities()` / `extractConcepts()` returned empty for entity-rich text. The SmartExtractor
ensemble scored agreeing signals with a weighted **sum** against an absolute 0.60 gate, so a
confident low-weight signal (e.g. a name pattern at 0.82) lost selection to a mediocre
high-weight signal that then failed the gate — dropping the whole result. It now selects and
gates on a normalized weighted **average** . Also: the `confidence` option now actually controls
the threshold (was a dead hardcoded 0.60), and the embedding-signal timeout was raised
100ms → 2000ms so the neural signal isn't silently dropped on slower runtimes.
*Known limitation:* type accuracy on dense multi-entity sentences is still imperfect — a strong
indicator in one entity's surrounding text can bleed onto neighbours. Tracked separately.
### `find({ aggregate })` exposes `groupKey` / `metrics` / `count`
Aggregation always computed correct values, but rows nested them under `.metadata` , so callers
expecting the documented `AggregateResult` shape saw empty-looking rows. Those three fields are
now also present at the top level of each result row.
### Multi-hop `find({ connected: { depth } })` honours depth
Previously returned only the immediate (1-hop) neighbour at any depth because the graph-search
path ignored `depth` / `via` . It now performs the full depth-aware traversal.
---
2026-05-15 12:31:28 -07:00
## v7.22.0 — 2026-05-15
**Affected products:** All. Fixes a silent-data-loss class affecting `find()` and
`brain.stats()` , plus Cortex-compatibility regression from 7.21.0. Recommended
upgrade for everyone on 7.20.x or 7.21.x.
### Two correctness fixes
#### 1. `find({ where })` no longer silently returns `[]` (BR-FIND-WHERE-ZERO)
The 7.20.0 column-store refactor deleted the legacy sparse-index write path
but left `getStats()` and `getIds()` reading from sparse indices. New
workspaces had no sparse-index files, so:
- `brain.stats().entityCount` was `0` regardless of actual entity count
- `find({ where: { ... } })` returned `[]` regardless of actual matches
- `rebuildIndexesIfNeeded()` saw `0` entries → fired the `[Brainy] CRITICAL ...
Second rebuild result: 0 entries` log → application saw no indexed data
7.22.0 makes the ColumnStore the single source of truth post-7.20.0:
- `MetadataIndex.getStats()` reads from `ColumnStore.getIndexedFields()` and
`idMapper.size` . Entity count is now accurate across writer → close →
reader-open cycles.
- `MetadataIndex.getIdsFromChunks()` throws `BrainyError(FIELD_NOT_INDEXED)`
when neither column store nor legacy sparse index has the field.
- `MetadataIndex.getIdsForFilter()` catches per-clause, logs
`[brainy] find() where-clause referenced unindexed field(s): ...` , returns
`[]` . Production `find()` is now consistent with `brain.explain()` 's
`path: 'none'` diagnostic.
Separately, `BaseStorage.getNounType()` was hardcoded to return `'thing'`
since a type-cache removal in commit `42ae5be` . Every noun save attributed
the entity to `'thing'` regardless of its actual type, poisoning
`_system/type-statistics.json` and `brain.stats().entitiesByType` . 7.22.0
restores type-correct attribution:
- New `nounTypeByIdCache: Map<string, NounType>` on `BaseStorage` , populated
in `saveNounMetadata_internal` and consumed in `saveNoun_internal` .
- New `flushCounts()` override that persists `nounCountsByType` /
`verbCountsByType` alongside the per-entity counter. Readers opening the
same directory after a writer flush now see correct per-type counts.
**Self-heal at init.** If `loadTypeStatistics()` detects the poisoned-state
signature (all nouns attributed to `'thing'` with multiple non-`thing`
entities on disk), it auto-runs `rebuildTypeCounts()` once and rewrites the
file. A `[BaseStorage] Detected poisoned type-statistics.json signature ...`
warning is logged. Operators don't need to take any action — the first 7.22
open of a directory poisoned by 7.20.0/7.21.0 fixes it.
**`brainy inspect repair` ** can also be used to manually trigger
`rebuildTypeCounts()` . It's now public on `BaseStorage` .
#### 2. Cortex 2.2.x no longer crashes 7.21.0 boot (BR-DEFENSIVE-INTERFACE)
7.21.0 added new storage-adapter methods (`supportsMultiProcessLocking` ,
`acquireWriterLock` , etc.) and called them unconditionally. Cortex 2.2.0's
mmap storage adapter bundles a pre-7.21 `BaseStorage` and doesn't define
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
them, so a consumer's 7.21.0 upgrade attempt threw
2026-05-15 12:31:28 -07:00
`TypeError: this.storage.supportsMultiProcessLocking is not a function`
at boot.
7.22.0 introduces `hasStorageMethod(name)` on Brainy and guards every new-
method call site. Older adapters degrade with a one-line warning at init:
```
[brainy] Storage adapter `CortexMmapStorage` predates the 7.21
multi-process methods. Writer locking and the flush-request RPC are
disabled for this directory. Upgrade the plugin (e.g.
`@soulcraft/cortex` ≥2.3.0) for full enforcement.
```
### Dead-code cleanup
Removed `MetadataIndexManager.dirtyChunks` , `dirtySparseIndices` , and the
`flushDirtyMetadata()` no-op machinery. These accumulators stopped being
populated when the sparse-index write path was deleted in 7.20.0; the
remaining 6 callers wasted async hops on an empty Map iteration.
### Upgrade notes
- **Behaviour change:** `find({ where: { unindexedField: ... } })` previously
returned `[]` silently. It now logs a one-line warning and still returns
`[]` . The diagnostic message names the field — use
`brain.explain({ where: { ... } })` for full diagnosis. No code change
needed for callers that already handle empty results.
- **Behaviour change (good):** `brain.stats()` and `brain.health()` now
report accurate per-type counts. If you previously relied on the buggy
`entitiesByType.thing` over-count, update consumers.
- **No API breaks.** Pure additive on the public surface.
### Cortex consumers
Cortex 2.2.x continues to work against 7.22.0 thanks to the defensive
guards, but for full multi-process protection on Cortex-backed stores and
to make sure Cortex's native MetadataIndex picks up the find-where-zero
fix, Cortex 2.3.0 is recommended. Tracked as `CX-MULTIPROC-METHOD` in the
internal platform handoff.
---
feat: multi-process safety + read-only inspector mode
Filesystem storage now enforces single-writer, many-reader semantics.
A second writer on the same data directory throws at init time with the
holder's PID, hostname, and heartbeat — replacing the previous silent
stale-reads failure mode.
- New: `Brainy.openReadOnly()` — coexists with a live writer, every
mutation throws clearly.
- New: writer lock at `<rootDir>/locks/_writer.lock` with 10s heartbeat
and stale-detection (PID liveness + heartbeat freshness).
- New: cross-process flush-request RPC (filesystem-based, no signals)
so inspectors can force fresh state on demand.
- New: `brain.stats()`, `brain.explain(findParams)`, `brain.health()`
for operator-facing introspection.
- New: `brainy inspect` CLI with 13 subcommands (stats, find, get,
relations, explain, health, sample, fields, dump, watch, backup,
repair, diff), all read-only by default.
- Same-PID re-opens allowed with a warning (preserves test "simulate
restart" patterns).
- Storage instances passed directly via `storage: new MemoryStorage()`
are now honoured instead of silently falling through to the
filesystem auto-detect path.
Brainy + Cortex compose under this model — the lock covers both because
they share `rootDir`, Cortex segments are immutable mmap files, and
MANIFEST updates use atomic-rename.
2026-05-15 11:25:05 -07:00
## v7.21.0 — 2026-05-15
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
**Affected products:** All consumers using filesystem storage (production deploys,
feat: multi-process safety + read-only inspector mode
Filesystem storage now enforces single-writer, many-reader semantics.
A second writer on the same data directory throws at init time with the
holder's PID, hostname, and heartbeat — replacing the previous silent
stale-reads failure mode.
- New: `Brainy.openReadOnly()` — coexists with a live writer, every
mutation throws clearly.
- New: writer lock at `<rootDir>/locks/_writer.lock` with 10s heartbeat
and stale-detection (PID liveness + heartbeat freshness).
- New: cross-process flush-request RPC (filesystem-based, no signals)
so inspectors can force fresh state on demand.
- New: `brain.stats()`, `brain.explain(findParams)`, `brain.health()`
for operator-facing introspection.
- New: `brainy inspect` CLI with 13 subcommands (stats, find, get,
relations, explain, health, sample, fields, dump, watch, backup,
repair, diff), all read-only by default.
- Same-PID re-opens allowed with a warning (preserves test "simulate
restart" patterns).
- Storage instances passed directly via `storage: new MemoryStorage()`
are now honoured instead of silently falling through to the
filesystem auto-detect path.
Brainy + Cortex compose under this model — the lock covers both because
they share `rootDir`, Cortex segments are immutable mmap files, and
MANIFEST updates use atomic-rename.
2026-05-15 11:25:05 -07:00
local development).
### Multi-process safety + first-class read-only inspector
Filesystem storage now enforces **single-writer, many-reader** . Previously, opening
a second writer process against a live data directory silently produced wrong query
results — no error, no warning, just empty or stale data. This release replaces
that footgun with a hard refusal at init time and a dedicated read-only inspection
mode.
#### New: `Brainy.openReadOnly()`
```ts
const reader = await Brainy.openReadOnly({
storage: { type: 'filesystem', rootDirectory: '/data/brain' }
})
const bookings = await reader.find({ where: { entityType: 'booking' } })
```
- Does NOT acquire the writer lock — coexists with a live writer.
- Every mutation method throws `Cannot mutate a read-only Brainy instance` .
- `flush()` and `close()` are safe (no-op + clean shutdown).
#### New: writer lock at init
- `new Brainy({ ... })` (default `mode: 'writer'` ) acquires
`<rootDir>/locks/_writer.lock` containing `{ pid, hostname, startedAt, lastHeartbeat, version }` .
- A second writer on the same directory **throws** with the PID, hostname,
heartbeat, and a pointer to `openReadOnly()` .
- Stale-lock detection: same hostname + dead PID OR heartbeat > 60s ago → overwritten with a warning.
- Heartbeat: writer rewrites `lastHeartbeat` every 10s (unref'd timer).
- Override: pass `{ force: true }` to bypass when you've verified the existing lock is stale.
#### New: `brain.requestFlush({ timeoutMs })`
Cross-process RPC for inspectors to force a fresh snapshot:
- In-process: just calls `flush()` .
- Out-of-process: writes a request file, polls for ack. Times out gracefully.
- Watcher runs in every writer instance (filesystem only).
#### New: `brain.stats()`
Operator-facing summary: `entityCount` , `entitiesByType` , `relationCount` ,
`relationsByType` , `fieldRegistry` , `indexHealth` , `storage.backend` , `writerLock` , `version` .
Designed for `/api/health` endpoints and incident triage.
#### New: `brain.explain(findParams)`
Shows which index path serves each `where` clause: `column-store` | `sparse-chunked` | `none` .
**This is the answer to "why is `find()` returning empty?"** — `path: "none"` means
the field has no index entries and the query will silently return `[]` . Includes
notes on likely causes (writer hasn't flushed; typo; field genuinely absent).
#### New: `brain.health()`
Invariant-check battery: HNSW vs metadata count parity, field registry sanity,
`_seeded` entity sweep, writer heartbeat freshness. Returns `{ overall: 'pass' | 'warn' | 'fail', checks: [...] }` .
#### New: `brainy inspect` CLI (13 subcommands)
All read-only by default (internally uses `openReadOnly()` ):
```
brainy inspect stats < path >
brainy inspect find < path > --type Event --where '{"status":"paid"}' --limit 20
brainy inspect get < path > < id >
brainy inspect relations < path > < id > --direction both
brainy inspect explain < path > --where '{"entityType":"booking"}'
brainy inspect health < path >
brainy inspect sample < path > --type Event --n 20
brainy inspect fields < path >
brainy inspect dump < path > --type Event > backup.jsonl
brainy inspect watch < path > --type Event
brainy inspect backup < path > /backups/brain.tar
brainy inspect repair < path > # writer mode — stop live writer first
brainy inspect diff < pathA > < pathB >
```
Default behaviour: ask the writer to flush via the RPC first (skip with `--no-fresh` ).
### Brainy + Cortex
Cortex segments (`*.cidx` ) are immutable mmap files; `MANIFEST.json` uses atomic
rename. The single writer lock at `<rootDir>/locks/_writer.lock` covers both Brainy
and Cortex. Read-only inspectors mmap Cortex segments with zero coordination.
### What's NOT enforced yet
- **Cloud storage backends** (S3, GCS, R2, Azure) — no multi-process locking.
A best-effort warning logs in writer mode against non-filesystem backends.
- **The `find({ where })` -returns-0 root-cause bug** — tracked separately. The
new `brain.explain()` and `brain.health()` surface it loudly
(`path: 'none'` , `index-parity warn` ) instead of letting it be silent.
### Upgrade notes
- **Default behaviour change:** opening a Brainy directory in writer mode while
another writer is live now THROWS where it previously silently produced wrong
results. If you have scripts that intentionally co-opened a writer directory,
switch them to `Brainy.openReadOnly()` or pass `{ force: true }` .
- **Cloud backends unchanged** — no lock acquisition for S3/GCS/R2/Azure.
- All existing APIs unchanged. Pure addition.
### Docs
- README "Single-Writer Model" section.
- `docs/concepts/multi-process.md` — full model + Cortex compatibility.
- `docs/guides/inspection.md` — operator recipes.
- JSDoc on every new API.
---
2026-02-28 10:07:34 -08:00
## v7.19.10 — 2026-02-24
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
**Affected products:** All Bun/ESM consumers
2026-02-28 10:07:34 -08:00
### ESM crypto fix in SSTable
Replaced `require('crypto')` with `import { createHash } from 'node:crypto'` in the
SSTable implementation. Fixes a crash in Bun and strict ESM environments where
CommonJS `require` is unavailable.
No API changes — upgrade and redeploy.
---
## v7.19.2 — 2026-02-18
**Affected products:** All
### Metadata index cleanup on delete
Fixed: metadata indexes were not cleaned up after `delete()` / `deleteMany()` . Stale
index entries could cause phantom results in metadata-filtered queries after deletion.
No API changes. If you were seeing ghost results in filtered queries, this fixes it.
---
## v7.18.0 — 2026-02-16
fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
2026-06-08 12:34:05 -07:00
**Affected products:** Analytics, reporting, and session-summary consumers
2026-02-28 10:07:34 -08:00
### Aggregation engine
New `brain.aggregate()` API — incremental SUM, COUNT, AVG, MIN, MAX with GROUP BY
and time window support. Computes over entity collections without loading all records
into memory.
```typescript
const result = await brain.aggregate({
collection: 'bookings',
metrics: [
{ field: 'revenue', fn: 'SUM' },
{ field: 'id', fn: 'COUNT' },
],
groupBy: 'staffId',
timeWindow: { field: 'createdAt', from: startOfMonth, to: now },
})
```
SDK exposure: `sdk.brainy.aggregate()` — available once SDK is updated to pass through.
---
## v7.17.0 — 2026-02-09
**Affected products:** All (schema evolution, data migrations)
### Migration system
New `brain.migrate()` API with error handling, validation, and enterprise hardening.
Run schema migrations reliably across Brainy data directories.
```typescript
await brain.migrate({
version: 3,
up: async (brain) => {
// transform entities, rename fields, etc.
},
})
```
---
## v7.16.0 — 2026-02-09
**Affected products:** All
### Data/metadata separation enforced + numeric range queries
- Entity `data` and `metadata` fields are now strictly separated at the storage layer
- Numeric range queries now supported in metadata filters: `{ age: { $gte: 18, $lt: 65 } }`
- Fixes edge cases where mixed data/metadata storage caused inconsistent query results
**Breaking for anyone storing numeric values in metadata and relying on range queries:**
verify your filter syntax matches the new `$gte/$lte/$gt/$lt` operators.
---
## v7.15.5 — 2026-02-02
**Affected products:** Anyone using `@soulcraft/cortex` plugin
### Plugin opt-in clarified
Cortex and other plugins are opt-in. Pass explicitly:
```typescript
new Brainy({ plugins: ['@soulcraft/cortex '] })
```
Without `plugins` , no external plugins are loaded regardless of what's installed.
---
## v7.15.2 — 2026-02-01
**Affected products:** All (data safety)
### Graph LSM flush on close
Fixed: graph LSM-trees were not flushed on `brain.close()` , risking data loss across
restarts. Graph edges written in the final seconds before shutdown are now guaranteed
to be persisted.
No API changes — upgrade immediately if running Brainy in a long-lived server process.
