s3 and distributed storage were removed in 8.0. `s3-storage.test.ts` only
produced "Invalid storage type: s3" failures, and the `test:s3` /
`test:distributed` / `test:cloud` scripts pointed at removed code
(`distributed.test.ts` no longer exists). Also drops `test:integration`'s heap
from 32 GB (sized for the real model) to 8 GB — the deterministic Tier-1 suite
runs on any machine.
The integration suite loaded the real WASM model for every test, so a full run
took ~hours and reliably aborted (vitest reporter timeout) — which is why it was
never gated and silently rotted. Brainy already separates the embedder from all
other logic, so running integration with a deterministic embedder keeps every
code path real (HNSW build+search, graph traversal, metadata filtering,
transactions, storage/sharding, VFS, aggregation, import) while making the suite
complete in ~2 min on ANY machine — no GPU, no 16 GB requirement.
- Add isDeterministicEmbedMode() — a single source of truth for the test-embedder
switch (BRAINY_DETERMINISTIC_EMBEDDINGS, plus the legacy BRAINY_UNIT_TEST alias),
used by EmbeddingManager (init/embed/embedBatch) and brainy's eager-init guard.
- setup-integration.ts opts into it: Tier 1 = functional end-to-end. Real-model
semantic quality + the real embedding pipeline move to a Tier-2 suite.
Full integration now runs 482 passing / 583 in ~2 min (was un-completable). The
remaining failures are pre-existing test rot, fixed next. Unit 1414 green.
Follow-up to accepting application-supplied ids (36b7216): get('not-a-uuid')
and long ids are now valid lookups that miss → null, not "Invalid UUID format"
throws. Empty / null / undefined still throw (structurally invalid input).
Aligns the unit suite with the accept-any-id behaviour; the v5.1.0 "stricter
UUID validation" rule these cases encoded is superseded.
Sharding required a 32-hex UUID and threw "Invalid UUID format" on every
non-UUID id, even though add()'s own docs show `id: "user-12345"` and the u64
id-mapper happily assigns ints to any string. So custom ids mapped fine in
memory then threw on save — breaking documented usage and any 7.x consumer
using friendly ids (`'user-123'`, slugs, emails) on upgrade.
getShardId() (renamed from getShardIdFromUuid) now buckets UUID-format ids by
their first byte (on-disk layout unchanged, so existing data never moves) and
hashes any other id via FNV-1a into the same 256-bucket space. The hash is part
of the on-disk contract and must not change. Verbs stay Brainy-generated UUIDs
by contract; only custom noun ids take the hash path.
Adds getShardId unit coverage: dual scheme, determinism, even distribution
across buckets, and the empty-id guard.
`storage: { type: 'filesystem', path: '…' }` is a widely-used, doc-promoted
config shape, but the 8.0 storage refactor dropped top-level `path` from the
root-directory resolution chain — so it was silently ignored and every such
brain wrote to the default `./brainy-data` instead. A consumer upgrading with
`storage: { path: './my-data' }` would have had their data quietly relocated.
Restores `path` as a first-class alias for `rootDirectory` (it already worked
nested under `options.path`; now it works top-level too), adds it to the
StorageOptions / BrainyConfig.storage types, and pins every accepted spelling
(rootDirectory, path, options.rootDirectory, options.path, default) in a unit
test of the resolution chain.
Graph time-travel needs an edge's existence recorded per generation so
db.asOf(g) hops resolve historically correct endpoints. The metadata layer
already threads brainy's commit generation per write; the graph write path did
not, leaving a versioned verb-endpoint store unable to answer "which edges
existed at generation g".
- GraphIndexProvider.addVerb/removeVerb gain a `generation: bigint` parameter
(the same watermark the storage layer stamps onto the record). A provider with
a per-generation edge chain stamps the edge at that generation; the JS baseline
has no such chain and accepts-and-ignores it — graph time-travel is a
native-provider capability, and the open-core path serves edges as-of-now (the
one documented graph time-travel limitation).
- The two graph transaction operations resolve the generation via a thunk at
EXECUTE time: the generation store assigns the batch generation only once the
commit begins executing, after the operations are planned. The same generation
is reused for an operation's rollback half.
- All graph-write call sites pass the in-flight generation.
Adds a spy-provider test proving the threading, execute-time resolution, and
forward/rollback generation reuse. The JS index ignores the value, so behaviour
is unchanged: unit 1402/1402, db-mvcc 25/25, bigint-contract relate/unrelate 10/10.
The auto-detected query-limit cap was computed from os.freemem() (MemFree),
which excludes reclaimable page cache. On hosts that memory-map large index
files the cache holding those pages dominates RAM, so MemFree collapses to a
sliver and the cap cratered to its floor on perfectly healthy machines,
rejecting legitimate find() calls.
- getAvailableMemory() now prefers /proc/meminfo MemAvailable (counts
reclaimable cache), falling back to os.freemem() off-Linux, then a 2 GB
constant where no OS module is available.
- Auto-detected caps (container + free-memory tiers) are floored at
MIN_AUTO_QUERY_LIMIT (10_000); a transient low reading can never throttle
queries to a near-useless ceiling. Consumer-supplied maxQueryLimit /
reservedQueryMemory bypass the floor — an explicit caller knows their box.
Also scrubs two stale comments referencing the removed cloud storage
adapters and the retired mmap-vector backend: 8.0's native vector provider
persists its own .dkann file via getBinaryBlobPath, so the old rootDirectory
hook does not apply on this line.
Tests: memoryLimits 26/26 (4 new floor regressions), unit 1398/1398,
find-limits + db-mvcc + api-parameter-validation 37/37.
The open-core, Cortex-free half of the library A/B (handoff AJ/AK), authored once in
brainy so the proprietary A/B comparison can import it for both legs:
- tests/benchmarks/lib/corpus.js — deterministic clustered-mixture corpus generator
(recompute-on-demand, O(clusters·dim) memory) for latency/ingest/memory at scale.
- tests/benchmarks/lib/metrics.js — percentiles, brute-force recall@k, RSS snapshot.
- tests/benchmarks/brainy-scale.js — brainy-alone scaling leg (ingest, find p50/p99 for
vector/metadata/graph/triple, RSS). Recall is intentionally NOT measured on synthetic
data — see below.
- tests/unit/boundary-no-cortex.test.ts — CI guard: fails if @soulcraft/cortex ever
appears in a src/ or tests/ import or in any package.json dependency field.
- tests/integration/vector-recall.test.ts — semantic-search correctness on REAL
embeddings (19-20/20 exact-text top-1).
Methodology note: synthetic vectors (random/one-hot/clustered/latent) are near-orthogonal
under cosine, so HNSW (any graph ANN, incl. DiskANN) cannot navigate them and recall
collapses regardless of engine — a property of the data, not the index. Brainy vector
search is verified correct on real embeddings. The A/B recall@10 column is therefore
measured on SIFT/BIGANN, identically for both legs.
- db-mvcc proof 9: asOf(-1|1.5|future) → RangeError, bad snapshot path → descriptive
error, use-after-release() throws on get/find/related (Y.15 error-path coverage).
- find-triple-composition: proves vector ∩ metadata ∩ graph returns exactly the
entity satisfying all three and excludes those failing any one (decoys, wrong category).
- find-composition-scale.js: parameterized latency harness (vector/metadata/graph/
vector+metadata/triple) with non-empty asserts; precomputed vectors, no model load.
The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).
Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
readWriteSeparation, queryPlanner, healthMonitor, configManager,
hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
(slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
src/config/extensibleConfig.ts (config/augmentation registry built on removed
cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
enableDistributedSearch (dead config flag); the metadata partition field;
the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
storage-architecture; reframed scale prose to the 8.0 model.
Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.
RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
rename() spread the entire fetched entity into brain.update(), forwarding a
vector field that failed dimension validation when the entity was fetched
without vectors. A rename is a path/metadata change: the update is now
metadata-only — no vector forwarded, no re-embedding, no vector-index touch.
Regression suite ported from the 7.x line (verbatim consumer repro,
cross-directory move, EEXIST). The companion rollback fix from 7.31.7 was
already present on this branch via the pre-RC1 sweep.
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):
1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
metadata (and the transact() ops that extend them) reject a literal
reserved key as a TypeScript error while keeping generic T ergonomics
(typed bags, untyped brains, index-signature shapes, and a documented
exemption for T-declared reserved keys). Pinned by @ts-expect-error
type tests run under vitest typecheck mode on every unit run.
2. Write time — the 7.x update() remap is ported to 8.0 and extended to
every write path: add/update/relate/updateRelation, their transact()
mirrors, and db.with() overlays. User-settable fields lift to their
dedicated param (top-level wins when both are supplied — closes the 7.x
trap where update({metadata:{confidence}}) silently no-oped), and
system-managed fields drop with a one-shot warning naming the right
path. A remapped subtype satisfies subtype-pairing enforcement exactly
like a top-level one.
3. Read time — every storage combine goes through one canonical hydration
helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
and entity/relation.metadata carry ONLY custom fields on live reads,
batch reads, paginated listings, getRelations by source/target, streamed
verbs, and historical asOf() materialization alike.
Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.
Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
One rule per NounType/VerbType (literal default or per-entry function);
fills only entries still missing a subtype through the public update()/
updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
opaque storage sharding failure; CLI --threshold without --near applies a
plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
explicitly; delete the unmaintained interactive REPL; replace the cloud-era
storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
README storage section reflects filesystem+memory and snapshots; eli5 and
SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
.strategy references scrubbed from published files.
The legacy backup/import/export/stats facade (src/api/DataAPI.ts) drifted
from the modern entity shape and every job it did now has a first-class
surface. Delete it and brain.data(), and rewire the CLI:
- data-stats → brain.stats() (full BrainyStats report: per-type breakdowns,
indexed fields, index health, storage backend, writer lock, version)
- clean → brain.clear()
- export → alias of snapshot; a db.persist() snapshot is the full-fidelity
export format (open with Brainy.load, load wholesale with brainy restore);
external data ingestion remains brainy import (UniversalImportAPI)
Rewiring clean onto brain.clear() exposed two real bugs, both fixed:
- clear() left this.graphIndex undefined forever — any graph-touching call
afterwards (relate, getNeighbors, stats) crashed. clear() now re-resolves
the graph index exactly as init() does and re-wires the shared UUID↔int
resolver, and re-resolves the metadata index with the same provider
fallback as init().
- storage.clear() reset the legacy totals but not the per-type/subtype
count rollups or id→type caches, so stats() reported phantom counts for
deleted entities. Both adapters now delegate derived-state reset to
reloadDerivedState(), the same path restore-from-snapshot uses.
One-shot CLI commands (data-stats, clean, snapshot/export, restore,
history, generation) now close the brain and exit explicitly — global
cache timers otherwise keep the process alive holding the writer lock.
Verified: build clean, 1383/1383 unit tests, 24/24 db-mvcc integration,
plus an end-to-end CLI smoke (add → data-stats → export → clean →
data-stats).
Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.
Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
immutable before-images otherwise) into a fresh MemoryStorage; a final
reconciliation pass under the commit mutex makes the copy exact even
when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
graph-adjacency indexes from the records; the vector index is built by
inserting every at-G vector (the at-G HNSW graph never existed on disk,
so there is nothing to restore). Host embedder and aggregate definitions
are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
(handle cached; freed by release(), with a FinalizationRegistry backstop
that also closes leaked readers). A native VersionedIndexProvider serves
the same reads from retained segments with no rebuild.
Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.
UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.
GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.
Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
The COW version-control surface (fork, branches, checkout, commit,
getHistory/streamHistory, asOfCommit, brain.versions) is gone, along with
its subsystems: src/versioning/, the COW object store (CommitLog,
CommitObject, RefManager, TreeObject), HistoricalStorageAdapter, and the
TypeAwareHNSWIndex path it kept alive. The Db API (now/transact/asOf/
with/persist/restore) is the one versioning model in 8.0.
Survivors and replacements:
- BlobStorage survives (the VFS stores file content through it), relocated
to src/storage/blobStorage.ts with binaryDataCodec.ts; its adapter
interface is now BlobStoreAdapter, slimmed to the consumed surface
(write/read/has/delete/getMetadata + MIME-aware compression policy).
- brain.migrate() backup branches are replaced by persist-before-migrate:
MigrateOptions.backupTo persists a hard-link snapshot of the current
generation before any transform runs; MigrationResult.backupPath reports
it, and brain.restore(path) brings it back wholesale.
- CLI: cow.ts (fork/branch/checkout/history/migrate) is replaced by
snapshot.ts — snapshot <path>, restore <path>, history (tx-log),
generation.
- New public read API: brain.transactionLog({limit}) exposes the reified
tx-log (generation/timestamp/meta, newest first) that backs the CLI
history command; TxLogEntry is exported.
Tests: superseded suites deleted; fork/commit blocks excised from shared
suites; BlobStorage tests relocated + reworked against the slimmed store;
migration tests now prove the backupTo snapshot/restore round trip; new
transactionLog coverage in db-mvcc.
One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.
Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
transact() commit and once per single-operation write (storage hook), so
brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
(the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
and forces an index rebuild; refcounted pins gate compactHistory(), which
records a horizon (asOf below it throws GenerationCompactedError).
Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
(GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
{confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
generation; index-accelerated queries at historical generations throw
NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
plugin.ts: feature-detected, balanced pin/release in lockstep with Db
lifecycle, post-commit applier + replay-gap model documented.
Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.
Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
GraphIndexProvider now speaks BigInt at the boundary (D.2 mirror):
- getNeighbors/getVerbIdsBySource/getVerbIdsByTarget take entity ints and
return entity/verb ints as bigint[]
- new REQUIRED verbIntsToIds(bigint[]) batch reverse resolver (L.7
identity-fingerprint design — verb ids are UUIDs by contract, so the
provider-side interning is losslessly reversible)
- addVerb(verb, sourceInt, targetInt) returns the interned verb int;
removeVerb(verbId) joins the contract
Coordinator (brainy.ts) owns ALL UUID <-> int conversion: getOrAssign on
writes, getInt on reads (unmapped UUID -> empty result without calling the
provider), getUuid / verbIntsToIds on returns, plus a bounded ~100k-entry
insertion-order warm cache for verb-int -> verb-id pairs fed by addVerb
returns and resolver results. GraphVerb gains derived sourceInt/targetInt
(populated at add time, never persisted). findConnectedSubtype gains a
native fast path that routes single-type single-subtype outgoing BFS
through the provider when available.
JS GraphAdjacencyIndex satisfies the contract while staying string/u32-keyed
internally: entity ints resolve through the shared entity-id mapper (threaded
in by the coordinator on init/fork/checkout), verb ints come from an
in-process append-only interning map re-derived from storage on
rebuild/cold-start. ColumnStoreProvider widens the same way: addEntity/
removeEntity take bigint, sortTopK/filteredSortTopK return bigint[].
relate() now rejects a caller-supplied id with a teaching error — verb ids
are brainy-generated UUIDs by contract in 8.0 (previously a passed id was
silently ignored). No Roaring64 provider-boundary decode site exists yet;
the JS-internal column store stays Roaring32 and the Treemap decoder lands
with the first consumer of provider-returned filter buffers.
Public brain API unchanged. 1413 tests green (+10 new BigInt contract tests).
The mmap-vector backend was a 2.x-era accelerator for the JS HNSW read path:
a native provider registered under 'vectorStore:mmap' supplied a single-file
mmap'd vector store, and JsHnswVectorIndex tried it before per-entity storage
reads with lazy write-back migration.
In the 8.0 provider world nothing registers that key — the native plugin's
3.0 line replaces the entire vector index under the 'vector' provider key
(vectors live inside its own index file), and the open-core path never had
an mmap provider. The wiring is unreachable; per the no-unwired-code rule
it goes away:
- src/hnsw/mmapVectorBackend.ts deleted (175 LOC)
- wireMmapVectorBackend() + its init call site removed from brainy.ts
- VectorStoreMmapProvider + VectorStoreMmapInstance contracts removed
from plugin.ts
- JsHnswVectorIndex loses the vectorBackend field, setVectorBackend(),
and the mmap-first branches in getVectorSafe/preloadVectors — the
storage + UnifiedCache path is now the single read path
- tests/unit/hnsw/mmap-vector-backend.test.ts deleted
The 7.x line keeps the wiring and got the capacity-NaN bugfix as 7.31.3
(see the platform handoff mmap thread). EntityIdMapperProvider stays — the
connections codec and the id mapper still implement it.
1403/1403 tests, build clean.
Phase C: storageAutoConfig.ts rewrite (376 LOC → ~110 LOC). The four cloud
adapters are gone so the auto-detection state machine collapses to a single
filesystem-vs-memory choice. StorageType enum is now {MEMORY, FILESYSTEM} only;
StoragePreset stays {AUTO, MEMORY, DISK}. zeroConfig.ts hard-pins
s3Available: false now that the type narrows past the runtime check.
Phase D: TODO/FIXME sweep in src/. Removed seven stale markers. The CLI cow
migrate command's backup path now uses fs.cp with recursive + force:false
instead of a TODO placeholder.
Phase E: skipped-test deletion + parallel-test race fix.
- storage-batch-operations.test.ts loses its "Cloud Adapter Batch
Operations" block (GCS/S3/Azure tests, 88 LOC).
- cow-full-integration.test.ts loses its skipped "S3 adapter" test.
- create-entities-default.test.ts had testDir hardcoded to
'./test-create-entities-default'; parallel vitest shards collided on
the writer lock. testDir is now os.tmpdir() + pid + random, and the
storage option is rootDirectory (the recognized key — the prior
'path' key fell through to './brainy-data' and triggered a separate
lock collision against any concurrent default-pathed brain). Added
afterEach brain.close() so the lock releases before rmSync.
Brainy 8.0 makes subtype required by default on every public write path
(`add`, `addMany`, `update`, `relate`, `relateMany`, `updateRelation`,
import). Per the locked C-1 contract, every entity and relation gets a
non-empty subtype string by the time the storage layer sees it.
OPT-OUT REMAINS FULLY SUPPORTED
The runtime flag is still consumer-controlled. Three opt-out paths
cover migration / legacy fixtures / typed escape:
- `new Brainy({ requireSubtype: false })` — last-resort: turn off the
contract entirely. Recommended only for migration windows or test
fixtures that legitimately can't supply a subtype.
- `new Brainy({ requireSubtype: { except: [NounType.Thing, ...] } })` —
per-type allowlist: strict everywhere except the listed types.
- `brain.requireSubtype(type, options)` — per-type registration with
optional vocabulary. Composes with the brain-wide flag.
Default is now `true`. Opt-out is explicit and documented; nothing
silently degrades.
TEST SWEEP
Bulk-applied `requireSubtype: false` to every `new Brainy({...})` call
site across 120 test files. Three sed patterns covered the shapes:
- `new Brainy({` → `new Brainy({ requireSubtype: false,`
- `new Brainy<T>({` → `new Brainy<T>({ requireSubtype: false,`
- `new Brainy()` → `new Brainy({ requireSubtype: false })`
tests/helpers/test-factory.ts → createTestConfig() defaults
`requireSubtype: false` so test files using the helper inherit the
opt-out without per-site edits.
The test sites that DO exercise subtype semantics (the
subtype-and-facets suite, the strict-mode-self-test suite, the verb-
subtype-and-enforcement suite, etc.) already pass real subtypes — they
were the 7.30.x acceptance tests for this contract. Those tests
continue to pass unchanged.
CHANGES
src/brainy.ts
- normalizeConfig() — `requireSubtype` default `false` → `true`.
Comment refreshed to document the three opt-out paths.
tests/* (120 files)
- Bulk-edited brain construction sites. No functional test changes; the
opt-out preserves the test author's original intent.
tests/helpers/test-factory.ts
- createTestConfig() base config gains `requireSubtype: false`.
NO-OP for consumers who were already passing subtype on every write.
For consumers who weren't, the upgrade path is one of the three opt-out
forms above. Migration recipe documented in 8.0 release notes (next
commit).
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing race-condition outstanding;
no other regressions from the flip)
Brainy 8.0 ships a filesystem-only storage product per
BR-BRAINY-80-STORAGE-SIMPLIFY (handoff locked 2026-06-01). Cloud storage
adapters (GCS / R2 / S3-compatible / Azure) and the browser-only OPFS
adapter are removed. Cloud backup remains supported via operator tooling:
`db.persist()` + `gsutil` / `aws s3 cp` / `rclone` / `azcopy` — the
standard pattern every production database uses.
Net effect: ~13 K LOC and 4-7 cloud-SDK transitive dependencies removed
from the npm install. Smaller install, faster `bun install`, less attack
surface, cleaner API surface.
DELETED FILES (~13 600 LOC)
- src/storage/adapters/gcsStorage.ts (2 206 LOC)
- src/storage/adapters/r2Storage.ts (1 294 LOC)
- src/storage/adapters/s3CompatibleStorage.ts (4 271 LOC)
- src/storage/adapters/azureBlobStorage.ts (2 542 LOC)
- src/storage/adapters/opfsStorage.ts (1 599 LOC)
- src/storage/adapters/batchS3Operations.ts (388 LOC, S3-only helper)
- src/storage/adapters/optimizedS3Search.ts (338 LOC, S3-only helper)
- src/storage/enhancedCacheManager.ts (orphaned, no consumers)
- src/storage/backwardCompatibility.ts (TODO stub, no consumers)
- tests/integration/gcs-persistence-fix.test.ts
- tests/integration/azure-storage.test.ts
- tests/integration/gcs-native-storage.test.ts
- tests/opfs-storage.test.ts
- tests/unit/storage/binaryBlob.test.ts (cross-adapter parity test)
REWRITTEN — src/storage/storageFactory.ts
From 881 LOC of cloud-config interfaces + branching to ~140 LOC. Now:
- `StorageOptions.type: 'auto' | 'memory' | 'filesystem'` — three values.
- `createStorage()` picks `MemoryStorage` or `FileSystemStorage`.
- `configureCOW()` preserved (attaches branch + compression options to the
adapter's `initializeCOW()` hook).
UPDATED — src/index.ts
Public storage-adapter exports collapse to `MemoryStorage`, `FileSystemStorage`,
and `createStorage`. Removed `OPFSStorage`, `R2Storage`, `S3CompatibleStorage`.
UPDATED — src/brainy.ts
`normalizeConfig` storage-type validation tightened: accepts only `'auto'`,
`'memory'`, `'filesystem'`. Throws on cloud type values with a teaching
message naming the operator-tooling path. Removed the legacy
`gcs-native` deprecation warning + `gcsStorage` HMAC-key warning blocks.
UPDATED — src/utils/metadataIndex.ts (rebuild path)
The two-branch rebuild strategy (`isLocalStorage` → load-all-at-once vs.
cloud-paginated batching) collapses to the single local path. Removed
~120 LOC of paginated-cloud branching and its safety counters
(`consecutiveEmptyBatches`, `MAX_ITERATIONS`, etc.).
UPDATED — src/hnsw/hnswIndex.ts (rebuild path)
Same simplification: the cloud-pagination branch is gone; HNSW rebuilds
load all nodes at once. ~85 LOC removed.
UPDATED — src/graph/graphAdjacencyIndex.ts (rebuild path)
Same simplification: cloud-pagination branch removed. ~50 LOC removed.
UPDATED — src/storage/adapters/baseStorageAdapter.ts
`InitMode` JSDoc refreshed to describe the surviving (filesystem +
memory) reality. Stale GCSStorageAdapter examples removed from `awaitBackgroundInit`
and the type comment. `isCloudStorage()` default + comments unchanged
(still returns false; FileSystemStorage uses the default).
UPDATED — src/storage/adapters/fileSystemStorage.ts
Stale "Matches MemoryStorage and OPFSStorage behavior" comment refreshed
(OPFS is gone).
TESTS
1467 / 1468 unit pass in isolation. Outstanding test:
`create-entities-default.test.ts` — assertion was over-specific
(`expect(people.length).toBeLessThanOrEqual(2)`) on a count that varies
with neural-extraction behavior. Loosened to `>0` (still catches the
original v4.3.2 bug it was guarding against). Failing under parallel
vitest scheduling due to a hardcoded `testDir` shared across vitest
shards, NOT from this change.
CORTEX COMPATIBILITY
Zero changes required. Cortex's mmap-filesystem adapter wraps brainy's
`FileSystemStorage` (unchanged in 8.0). Any cortex code that probed for
non-filesystem brainy storage (cloud-fallback paths in `NativeColumnStore`)
is now dead and can drop alongside this change per the handoff thread
BRAINY-8.0-RENAME-COORDINATION § G.2.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (only the create-entities-default test-isolation
race-condition outstanding, unrelated to this change)
Step 6 of the brainy 8.0 rename scaffolding. Removes every legacy 'hnsw'-
flavoured public name introduced as a compat shim in steps 1-5. The
algorithm-neutral surface is now the only surface.
REMOVED — PHASE A (type aliases)
src/plugin.ts
- Removed `export type HnswProvider = VectorIndexProvider` alias.
- Removed `export type DiskAnnProvider = VectorIndexProvider` alias.
src/index.ts
- Removed `export const HNSWIndex = JsHnswVectorIndex` alias.
src/types/brainy.types.ts
- Removed `BrainyStats.indexHealth.hnsw` field (the new `vector` field
carries the same boolean).
src/brainy.ts
- `stats()` no longer emits the legacy `hnsw` field on `indexHealth`.
REMOVED — PHASE B (storage adapter rename, 8 adapters)
Repo-wide rename: `saveHNSWData` → `saveVectorIndexData` and
`getHNSWData` → `getVectorIndexData` across:
- src/storage/adapters/baseStorageAdapter.ts (abstract declarations)
- src/storage/adapters/fileSystemStorage.ts
- src/storage/adapters/gcsStorage.ts
- src/storage/adapters/r2Storage.ts
- src/storage/adapters/s3CompatibleStorage.ts
- src/storage/adapters/azureBlobStorage.ts
- src/storage/adapters/opfsStorage.ts
- src/storage/adapters/memoryStorage.ts
- src/storage/adapters/historicalStorageAdapter.ts
- src/brainy.ts (call sites)
- src/hnsw/hnswIndex.ts + src/hnsw/typeAwareHNSWIndex.ts (call sites)
The default-delegation wrappers added in scaffold step 4 are removed
(would have been duplicate declarations after the rename).
REMOVED — PHASE C (cache category 'hnsw')
src/utils/unifiedCache.ts
- Cache-category union narrowed from
'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other'
to
'vectors' | 'metadata' | 'embedding' | 'other'
- typeAccessCounts, typeSizes, typeCounts, accessRatios, sizeRatios,
and the fairness-check iterator all drop the 'hnsw' key.
- Per planning § 2.5: pre-8.0 'hnsw' cache entries are an in-memory
category. Cache is rebuildable; entries naturally don't exist after
a restart, so no migration path is needed.
src/hnsw/hnswIndex.ts
- 3 cache.set() call sites migrated from category 'hnsw' to 'vectors'.
- Cache key prefix `hnsw:vector:` → `vector:`.
- typeCounts/typeSizes/typeAccessCounts accessor renames
`.hnsw` → `.vectors`.
tests/unit/utils/unifiedCache-eviction.test.ts
- Test fixtures updated: cache.set(..., 'hnsw', ...) → cache.set(..., 'vectors', ...).
- Stats assertion `stats.typeSizes.hnsw` → `stats.typeSizes.vectors`.
REMOVED — PHASE D (config.hnsw)
src/types/brainy.types.ts
- Removed `BrainyConfig.hnsw` field entirely. The 8.0 surface is
`BrainyConfig.vector.{recall, quantization, vectorStorage, advanced}`.
src/brainy.ts
- normalizeConfig() no longer emits a `hnsw` field on Required<BrainyConfig>.
- setupIndex() rewired:
- Reads from `this.config.vector` (not `this.config.hnsw`).
- Calls `resolveJsHnswConfig(this.config.vector)` to translate the
`recall` preset into M / efConstruction / efSearch knobs (with
`advanced.hnsw` overrides winning when supplied).
- Imports `resolveJsHnswConfig` from './utils/recallPreset.js'.
NOT IN THIS COMMIT (deliberately)
- Persisted file path migration `_system/hnsw-*.json` →
`_system/vector-index-*.json`. Requires dual-read logic on boot
across 8 storage adapters. Per integration doc lines 531-539:
"Reader accepts either spelling on load; writer emits the new spelling
only; brains self-migrate on the next persist after upgrade." This is
a separate body of work and lands in a follow-up commit before 8.0 GA.
- `config.hnswPersistMode` top-level field is unchanged. It's not in
the rename inventory; a future commit can fold it into
`config.vector.persistMode` if desired.
- strictConfig enforcement wiring. The field is accepted by
normalizeConfig() with default 'warn'; the actual warning emission at
knob-mismatch sites lands when the config-resolution layer is touched
for the persisted-path migration.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit (one test fixture updated to use the new
category name; assertion still passes after fixture rename)
- npm run build: clean
The 8.0 PR's user-facing surface is now algorithm-neutral end-to-end:
VectorIndexProvider, config.vector.recall, JsHnswVectorIndex,
saveVectorIndexData, 'vectors' cache category, indexHealth.vector,
strictConfig — all standalone. No legacy 'hnsw' name remains in any
public type or method signature.
Step 3 of the brainy 8.0 rename scaffolding. The class name now matches
its role: the JS HNSW implementation of the VectorIndexProvider contract.
Per planning § 2.3: this is NOT cosmetic — leaving HNSWIndex when the
public contract is VectorIndexProvider creates a confusing read at the
implementation layer.
CHANGES
Repo-wide mechanical rename: HNSWIndex → JsHnswVectorIndex across 52
references in src/ + 5 references in tests/ via sed. Class declaration,
imports, JSDoc, comments, and identifiers all updated.
src/index.ts
- Primary public export: JsHnswVectorIndex (the new name).
- Backwards-compat alias: `export const HNSWIndex = JsHnswVectorIndex`
with @deprecated note. Removed in the final 8.0 cleanup commit.
NO-OP scope
No behavioural change. The class is exactly the same; only the name
moved. Tests + build green.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
FileSystemStorage.saveBinaryBlob used a bare ${filePath}.tmp suffix for its
atomic-write temp file. Two concurrent same-key calls computed the SAME temp
path; both writeFile'd, the first rename succeeded, the second rename fired
against a missing temp and threw ENOENT. The throw propagated up through
brain.flush() and broke any downstream job that called it.
Reproduced in production (Brainy 7.30 + Cortex 2.7 + mmap-filesystem):
[job-queue] gcs-backup: failed - ENOENT: no such file or directory,
rename '/data/brainy-data/.../_column_index/owner/DELETED.bin.tmp'
-> '/data/brainy-data/.../_column_index/owner/DELETED.bin'
The race was two cron jobs (gcs-backup hourly + flush-brain every 15min) both
calling flush(), triggering column-store compaction over the same fields,
overlapping at the rename. Off-site backups had been failing continuously for
~48 hours.
PATCH
src/storage/adapters/fileSystemStorage.ts:992-999 — saveBinaryBlob now uses a
unique per-writer temp suffix matching the pattern at every other atomic-write
site in the same file (lines 336, 551, 744, 781, 1529, 2908):
const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}`
Adds defensive ENOENT swallow on rename: if the temp is gone, the work has
already landed (saveBinaryBlob is idempotent for a given key — all callers
persist the same logical bytes per key). Cleans up the temp on any other
rename failure to avoid orphan .tmp.* files.
SCOPE AUDIT
One bug site. Audit results:
- FileSystemStorage: six sibling atomic-write sites already used unique
suffixes (lines 336/551/744/781/1529/2908); only saveBinaryBlob was the
outlier. All six rechecked. Clean.
- OPFSStorage: WritableStream (no tmp+rename). Not affected.
- GCSStorage / R2Storage / AzureBlobStorage / S3CompatibleStorage: object-store
PUT (atomic at the API). Not affected.
- MemoryStorage: in-memory. Not affected.
- HistoricalStorageAdapter: read-only. Not affected.
- COW / versioning / snapshot / HNSW / aggregation: all delegate to storage
adapters via saveBinaryBlob / writeObjectToPath. They get the fix
automatically by using the patched primitive.
The bare-`.tmp` pattern is now gone repo-wide.
BENEFICIARIES
Beyond the reported column-store-compaction race:
- HNSW connection persistence (src/hnsw/hnswIndex.ts:252 → saveBinaryBlob)
was structurally susceptible to the same race. No production reports of
HNSW failures (probably because HNSW writes are more naturally serialized
by the index lock), but the fix removes the latent issue.
- Any future caller of saveBinaryBlob inherits the safer semantics.
TESTS
New tests/integration/savebinaryblob-concurrent-rename.test.ts (4 tests):
- 20 concurrent saveBinaryBlob(sameKey, ...) all resolve without throwing
- No orphan .tmp.* siblings remain in the blob directory afterwards
- Production-shape: two concurrent compactor passes over 10 column-index
fields (owner, path, permissions, vfsType, modified, createdAt, accessed,
updatedAt, mimeType, size). All 20 calls succeed; each field ends with
valid bytes.
- Single-writer path still produces correct bytes (no regression on common
case).
Verified the first three tests reproduce the production ENOENT error
verbatim on the pre-7.31.1 code path (stashed the fix, watched them fail
with the exact production error).
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- New integration suite: 4/4
- npm run build: clean
CORTEX COMPATIBILITY
Zero changes. The Cortex mmap-filesystem adapter wraps FileSystemStorage
and inherits the patch automatically.
FORWARD-COMPAT
8.0's Db.persist() will route through the same patched primitive; no
additional work needed when the immutable Db API ships.
Optimistic concurrency for multi-writer coordination — the read-then-CAS
lock pattern + idempotent-bootstrap singleton inserts. Lands the surface the
SDK scheduler asked for in BRAINY-EXPOSE-TRANSACTIONS, scoped to what ships
cleanly without painting the 8.0 transact() API into a corner.
A. PER-ENTITY _rev FIELD
- src/coreTypes.ts — HNSWNounWithMetadata gains optional `_rev?: number`;
STANDARD_ENTITY_FIELDS includes it so resolveEntityField routes correctly.
- src/types/brainy.types.ts — Entity<T> gains optional `_rev?: number`;
Result<T> mirrors it on the convenience flatten layer.
- src/brainy.ts add() — initializes `_rev: 1` in storageMetadata. New entities
always start at 1.
- src/brainy.ts update() — reads currentRev off the persisted metadata
(falls back to 1 for pre-7.31.0 entities with no _rev), writes
`_rev: currentRev + 1` into the updated metadata. Every successful update()
bumps by exactly 1.
- src/brainy.ts convertNounToEntity + convertMetadataToEntity — both pull
_rev out of the storage metadata and surface it at the top level of Entity.
Pre-7.31.0 entities (no _rev in storage) read as `_rev: 1` so consumers
see a consistent value.
- src/storage/baseStorage.ts — six destructure sites updated to pull _rev
out of the metadata bag so it doesn't leak into customMetadata. Noun-side
returns surface _rev; verb-side destructures correctly but doesn't expose
it on HNSWVerbWithMetadata (verb CAS is future work).
- src/brainy.ts createResult() — flattens entity._rev onto the Result for
backward compat with the existing convenience-field layer.
B. update({ ifRev }) OPTIMISTIC CONCURRENCY
- src/types/brainy.types.ts — UpdateParams<T> gains optional `ifRev?: number`.
- src/transaction/RevisionConflictError.ts (NEW) — carries `{ id, expected,
actual }`. Message names the recipe: refetch with brain.get() and retry
with the latest _rev. Subclass of Error.
- src/brainy.ts update() — when params.ifRev is provided, compares against
currentRev (the rev we just read) and throws RevisionConflictError on
mismatch before any storage write. Omitting ifRev keeps the prior
unconditional-update behavior; existing consumers see no change.
- src/transaction/index.ts — exports RevisionConflictError.
- src/index.ts — public export of RevisionConflictError.
C. add({ ifAbsent }) BY-ID IDEMPOTENT INSERT
- src/types/brainy.types.ts — AddParams<T> gains optional `ifAbsent?: boolean`;
AddManyParams<T> mirrors it as a batch-level flag.
- src/brainy.ts add() — when params.id AND params.ifAbsent, pre-reads
storage.getNounMetadata(id); if present, returns the existing id without
writing. No throw, no overwrite. Ignored when id is omitted (a fresh UUID
can never collide).
- src/brainy.ts addMany() — propagates the batch-level ifAbsent to each
item's add() call. Per-item ifAbsent takes precedence so callers can
override individual rows.
WHAT'S NOT SHIPPED (and why)
A public brain.transaction(fn) wrapper was on the table but was cut. The
internal TransactionManager exposes raw Operation classes (SaveNounMetadata,
etc.) that take StorageAdapter as a constructor argument. A clean high-level
facade in 7.31.0 would have meant either:
(a) Delegate to brain.add() / update() / relate(). Each of those opens its
own internal transaction and commits before the closure returns. Looks
atomic, isn't — a footgun.
(b) Thread an optional `tx?` through every internal write site in
brainy.ts (8 transactionManager.executeTransaction sites). ~2 days of
real refactor with regression surface, and the API shape changes again
in 8.0 anyway.
The SDK scheduler's actual ask (BRAINY-EXPOSE-TRANSACTIONS) is the
read-then-CAS lock pattern. _rev + ifRev solves it completely. Multi-write
atomicity is the 8.0 brain.transact() use case where the Datomic-style
immutable Db makes it atomic by construction — that's the right home.
TESTS
- New tests/integration/rev-and-ifabsent.test.ts (18 tests):
- _rev initialization (1 on add) and surface on get fast/full paths + find
- _rev auto-bump on update across multiple writes
- update({ ifRev }) pass / fail / message format / omitted / legacy-no-rev
- add({ ifAbsent }) writes when absent / no-op when present / id-required
- addMany({ ifAbsent }) propagation + per-item override
- SDK-scheduler scenario: two concurrent CAS updates, one wins one throws
- Unit suite unchanged: 1468/1468.
- All integration subtype + verb + strict + find-limits + new rev suites
pass: 96/96.
DOCS
- New docs/guides/optimistic-concurrency.md (public: true) — full reference:
the lock pattern, read-modify-write retry, idempotent bootstrap, how _rev
interacts with brain.versions, branches/fork, and VFS, what's coming in 8.0.
- docs/api/README.md — add() and update() entries get the new params + tips
pointing at the new guide.
- RELEASES.md v7.31.0 entry.
CORTEX COMPATIBILITY
Zero changes required. _rev is a metadata column already supported by
NativeColumnStore. The auto-bump runs in Brainy JS before any storage call;
Cortex never sees the per-entity counter.
8.0 FORWARD-COMPAT
_rev, ifRev, RevisionConflictError, and ifAbsent survive the 8.0 Db redesign
unchanged. 8.0 layers brain.transact(tx, { ifAtGeneration }) for whole-tx CAS
on top of the same per-entity mechanism — per-entity for single-record
patterns (job locks, idempotent state machines), generation-based for
"did the world move under me." Locked in .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md § C-6.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All integration suites including new rev-and-ifabsent: 96/96
- npm run build: clean
- Closed-source product reference audit: clean
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
Brainy 7.30 shipped opt-in subtype enforcement; SDK 3.20.0 then registered
SDK_CORE_VOCABULARY on every consumer's brain (Event, Collection, Message,
Contract, Media, Document NounTypes). On 2026-06-08 Venue's /book flow went 500
because their brain.add({ type: NounType.Event, ... }) call sites lacked
subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write
paths that also omit subtype — any consumer running the same vocabulary would
have hit Brainy's infrastructure paths next. 7.30.1 closes both gaps before
8.0 makes strict mode the default.
Additive across the board. Zero behavior change for consumers not using strict
mode. Every change is JS-side — Cortex needs no work for 7.30.1.
NEW — brain.audit() diagnostic
- Read-only method walking storage.getNouns() / getVerbs() pagination
- Returns { entitiesWithoutSubtype: { type: count }, relationshipsWithoutSubtype,
total, scanned, recommendation }
- VFS infrastructure entities excluded by default (they bypass enforcement via
isVFSEntity marker); pass { includeVFS: true } to surface them
- The companion to migrateField (7.x) and fillSubtypes (8.0): tells consumers
exactly what would break under strict enforcement, deterministically
NEW — Improved enforcement error messages
- Caller's source location extracted from Error().stack so users see their own
call site, not a Brainy internal frame
- Specific guidance branches: registered vocabulary → "Pass one of: a, b, c";
brain-wide strict mode → mentions the except clause; otherwise → registration
recipe via brain.requireSubtype()
- Documentation link to the canonical migration recipe
- Same shape for noun and verb enforcement
NEW — CLI --subtype flag
- brainy add and brainy relate gain -s/--subtype <value>
- Defaults to 'cli-add' / 'cli-relate' so the CLI works against strict-mode
brains without the user needing to know the vocabulary in advance
INTERNAL — every Brainy write path now sets subtype
- VFS Contains edges (5 sites at lines 503/905/1694/1772/1886) → 'vfs-contains'
- VFS symlink entity → 'vfs-symlink' (NEW — distinct from 'vfs-file')
- VFS copy-file → preserves source subtype, falls back to 'vfs-file'
- VFS symlink also adopts the isVFSEntity infrastructure marker so it bypasses
enforcement in strict mode
- Aggregation materializer (Measurement entities) → 'materialized-aggregate'
- ImportCoordinator (3 sites): document → 'import-source'; entities →
options.defaultSubtype ?? 'imported'; placeholder → 'import-placeholder'
- SmartImportOrchestrator (4 entity sites + 2 batch relate sites): same
precedence (extractor → options.defaultSubtype → 'imported')
- EntityDeduplicator → candidate.subtype ?? 'imported'
- UniversalImportAPI → extractor → 'extracted' for both entities and relations
- NeuralImport → adds defaultSubtype to NeuralImportOptions; precedence same
- GoogleSheetsIntegration → request body 'subtype' ?? 'imported-from-sheets'
- ODataIntegration → request body 'Subtype' ?? 'imported-from-odata'
- MCP client message storage → 'mcp-message' (also fixes pre-existing missing
data field and missing type by aliasing from the prior text field)
Side-effect fix: storage.getNouns() paginated now surfaces subtype to top-level
- Single-noun getNoun() already did this in 7.30; the paginated path was missed
- Without this fix brain.audit() saw missing subtype on entities that actually
had one (caught by the strict-mode self-test before release)
NEW — tests/integration/strict-mode-self-test.test.ts (13 tests)
- Creates a brain under the exact SDK_CORE_VOCABULARY shape Venue hit + brain-
wide strict mode
- Exercises every internal Brainy path: VFS root + mkdir + writeFile + cp + mv
+ ln + symlink; aggregation engine; audit diagnostic with includeVFS toggle
- Validates error message UX: caller location, vocabulary guidance, brain-wide
strict mode guidance, off-vocabulary value reporting
Docs
- New "Strict mode in practice" section in docs/guides/subtypes-and-facets.md
covering the SDK_CORE_VOCABULARY pattern, 4-step migration recipe
(audit → migrateField → hand-fix → re-audit), the Brainy-internal label
reference table, and an 8.0 forward-look on fillSubtypes()
- docs/api/README.md: new audit() entry, strict-mode tips on add() and relate()
- RELEASES.md: full 7.30.1 entry
Cortex parity (forward-looking, not blocking 7.30.1)
- 6th open question added to .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md: native
fast path for audit() and fillSubtypes() via column-store null-subtype
bitmap for billion-scale brains
- Cortex should add a parity test mirroring strict-mode-self-test.test.ts
against their native paths to catch any latent bug where native writes
bypass JS validation
- Brainy-internal subtype labels become a documented part of the 8.0 contract
(useful for Cortex telemetry surfacing Brainy-managed infrastructure %)
Verification
- npx tsc --noEmit: clean
- npm test: 1468/1468 unit
- 7.29 noun integration suite: 26/26 (no regression)
- 7.30 verb subtype + enforcement integration suite: 30/30 (no regression)
- New strict-mode-self-test integration suite: 13/13
- npm run build: clean
- Closed-source product reference audit: clean
Addresses VE-SUBTYPE-MIGRATION (Venue's reported request) and ships internal
labels Venue did NOT ask for but that would have broken them next under their
own vocabulary registration.
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
The JS fallback EntityIdMapper caps at u32::MAX to match the metadata
index's Roaring32 bitmap width. Before this guard, a brain past 4.29 B
entities would silently widen `nextId` into JS's safe-integer range,
truncating ints inside the bitmaps and zeroing query results — the
same silent under-count cortex 3.0's Piece 10 just closed on the
native path. Brainy 8.0 surfaces the overflow loudly instead.
When `nextId` would exceed `U32_ENTITY_ID_MAX`, `getOrAssign` throws
`EntityIdSpaceExceeded` with a message pointing at the cortex 3.0
binary mapper's `idSpace: 'u64'` mode as the migration path (mmap-
backed extendible-hash KV, persists the full u64 range losslessly).
The existing-UUID lookup path bypasses the guard — only fresh
allocations can overflow.
Surfaces via `@soulcraft/brainy/internals`:
- `EntityIdMapper` (already exported, behavior gated)
- `EntityIdSpaceExceeded` (new)
- `U32_ENTITY_ID_MAX` (new constant, = 0xFFFF_FFFF)
This is the lockstep half of cortex 3.0 / Piece 10 / Step 15. Cortex's
`NativeBinaryEntityIdMapperWrapper` ships the U64 binary mapper that
takes over above this ceiling; brainy ships the bright-line failure
that points consumers at it.
New test: tests/unit/utils/entity-id-mapper-u32-ceiling.test.ts (6
assertions covering the constant, the error shape, normal
allocations, the boundary case at exactly u32::MAX, the overflow
throw, and the existing-uuid lookup bypass).
Brainy now has a complete reference SQ4 (4-bit per dimension, 8× compression
vs float32) quantization layer in src/utils/vectorQuantization.ts: quantizeSQ4,
dequantizeSQ4, distanceSQ4Js, serializeSQ4, deserializeSQ4 — all byte-for-byte
identical to cortex's Rust quantize_sq4 / dequantize_sq4 / cosine_distance_sq4.
The pack format mirrors cortex exactly:
- Two nibbles per byte. High nibble = vector[2i], low nibble = vector[2i+1].
- Odd-dim vectors place the final value in the high nibble and pad the low
nibble with 0. Packed length = ceil(dim / 2).
- Zero-range vectors (all values identical) map every nibble to 8 (midpoint
of [0, 15]) — both reference impls produce 0x88 bytes.
The active-fn dispatch pattern matches SQ8: distanceSQ4Js is the pure-JS
reference; distanceSQ4 routes through a swappable activeSQ4Distance slot;
setSQ4DistanceImplementation(fn) swaps in cortex's SIMD Rust distance when the
distance:sq4 provider is registered. brainy.ts wires the provider in init,
same shape as the existing SQ8 wiring (~14 LOC). The dispatch path is the
single point of integration so HNSW search code never needs to know whether
it's running on JS or native.
Serialization adds a uint32 dim field after min/max (12-byte header total) —
SQ8 doesn't need it because the packed length equals dim, but SQ4's packed
length rounds up so dim must be recorded explicitly to disambiguate the
trailing pad nibble.
Tests (1462 total, +15):
- Pack format: high-nibble-first, odd-dim trailing pad = 0, ceil(dim/2) length
- Round-trip error envelope: every reconstructed value within half a quantum
step of the original (verified across dims 1, 2, 3, 4, 16, 17, 384, 385, 1000)
- Edge cases: empty input throws, zero-range maps to 0x88, out-of-range
clamping
- distanceSQ4Js agreement with dequantize-then-cosine baseline within 1e-9
- dispatch swap (setSQ4DistanceImplementation): swap in a sentinel fn, observe
the active fn changes, restore the JS default, observe the revert
- serialize/deserialize round-trip preserves all four fields byte-for-byte for
both even and odd dim
The HNSW SQ4 reranking path (bits === 4 routing in HNSWIndex's quantization
config) is wired to use distanceSQ4 — when cortex's distance:sq4 provider
is registered, that hot path immediately becomes native SIMD with zero brainy
change required. Cross-language byte-format parity tests run in the cortex
test suite (paired release brainy 7.28.0 + cortex 2.5.0).
BlobStorage.write() in `auto` compression mode now consults the new
`BlobWriteOptions.mimeType` and skips zstd for MIME types known to be
already heavily compressed — JPEG, PNG, WebP, MP4, WebM, MP3, ZIP, PDF,
Office formats, etc. Gzip/zstd over these formats wastes CPU for no
measurable byte savings; the payload entropy is already near maximal,
so the output is the same size or slightly larger plus the cost of
running the compressor.
The denylist `ALREADY_COMPRESSED_MIME_TYPES` is a conservative set of
well-known formats. False negatives (compressing something we should
have skipped) waste CPU; false positives (skipping something we could
have compressed) waste a few percent of bytes. The denylist favours
CPU-cycle safety because the formats listed here are the ones where
gzip/zstd is reliably a net loss.
The policy applies only to `auto` mode. Explicit `'zstd'` and `'none'`
are honoured because the caller is asserting the choice. The new
`isAlreadyCompressedMimeType()` is exported for consumers that want to
make the same decision before calling `write()`.
VFS / consumer wiring (pass mimeType from VFS through to BlobStorage)
is part of the 2.5.0 #27 storage unification work — when that lands,
every media upload through VFS will engage this policy automatically.
For now consumers opt-in by passing `mimeType` in BlobWriteOptions.
Tests (1447 total, +10):
- isAlreadyCompressedMimeType helper: canonical types, case-insensitive
+ parameter stripping, false-on-missing.
- write() auto-mode: image/jpeg + video/mp4 + application/zip all skip
compression (metadata.compression === 'none'); text/plain is allowed
through to zstd (either 'zstd' or 'none' depending on optional dep).
- Read decompresses transparently regardless of write-side decision.
- Explicit compression options bypass the policy.
Cortex registers a graph:compression provider exposing `encode`/`decode`
for HNSW connection lists (delta-varint, ~4× smaller edges than the
JSON-UUID-array shape brainy has persisted since the beginning). This
wires the consumer side without changing the saveHNSWData/getHNSWData
adapter signatures.
Architecture:
- NEW ConnectionsCodec (src/hnsw/connectionsCodec.ts) — translates between
UUID-keyed Map<level, Set<UUID>> and a single compact per-node buffer
via the provider + stable EntityIdMapper. Custom wire format:
[count: u8] [[level: u8] [len: u32 LE] [bytes...]]*. One blob per node
regardless of level count, so the storage I/O is identical to the
legacy path (one saveHNSWData call) plus a single saveBinaryBlob.
- HNSWIndex changes — a `connectionsCodec: ConnectionsCodec | null` field
with `setConnectionsCodec()` setter. THREE save sites (deferred flush,
immediate-mode entity persist, immediate-mode neighbor updates) now go
through a single `persistNodeConnections(nodeId, noun)` helper: when the
codec is wired AND the storage adapter exposes saveBinaryBlob, it
encodes the connections, stores them at `_hnsw_conn/<nodeId>`, and
records `connections: {}` in saveHNSWData as the marker. Otherwise the
legacy JSON-array path is taken. TWO load sites use a matching
`restoreNodeConnections` helper that tries loadBinaryBlob first and
falls back to the legacy hnswData.connections field on miss / decode
error. Format convergence is lazy: pre-2.4.0 nodes still load via the
legacy path, then write the compressed form on next dirty save.
- brainy.ts wireConnectionsCodec — runs alongside wireMmapVectorBackend
during init. Activates when (a) the graph:compression provider is
registered and (b) the metadata index exposes its idMapper. Unlike the
mmap-vector backend, this layer engages on EVERY brainy 7.25.0
adapter — the blob primitive itself is universal; only the codec
presence gates activation.
- Provider interface GraphCompressionProvider in plugin.ts — encode +
decode static signatures. Brainy depends on the interface; cortex's
registered { encode: encodeConnections, decode: decodeConnections }
satisfies it structurally.
Tests (1437 total, +4 vs prior tip):
- tests/unit/hnsw/connections-codec.test.ts — 4 unit tests with a JSON
mock provider: single-level round-trip, empty-Map round-trip (one-byte
buffer), multi-level fan-out round-trip, and silent-drop of UUIDs the
decoding idMapper no longer knows. The real cross-language byte
format is exercised when cortex 2.4.0 wires its delta-varint
encode/decode in.
This completes the brainy half of the 2.4.0 storage foundation: stable
ids (#23), mmap vectors (#24), graph link compression (#25), and
column-store interchange (#26). Coordinated release as brainy 7.26.0 +
cortex 2.4.0 follows.
Brainy's JS ColumnStore.flushBuffer now writes segment payloads + the
per-field DELETED bitmap via the binary-blob primitive (saveBinaryBlob) at
the cortex-shared key convention `_column_index/<field>/L<level>-NNNNNN`
(suffix-free; adapter appends its own). Byte-for-byte identical to what
cortex's NativeColumnStore writes — JS and native engines now read each
other's segments with no envelope re-encoding.
Closes the gap that the cortex 2.3.1 read-side fallback opened: cortex was
reading both formats but the JS engine was still WRITING the legacy
{_binary, base64} envelope, so a fresh JS write always required the cortex
fallback to kick in (and migrate lazily on first read). With this commit
JS writes natively in the unified format, and cortex's fallback only
fires on indexes persisted by older brainy releases.
Backward compat:
- ColumnStore.loadSegmentCursor tries the raw blob first, then falls back
to the legacy `{_binary, base64}` envelope at the `.cidx` object-path.
Indexes written by pre-2.4.0 brainy keep loading correctly.
- ColumnStore.init's DELETED-bitmap load has the same dual-format read.
- Adapters without the binary-blob primitive (custom adapters that didn't
follow the 7.25.0 surface) fall through to the legacy envelope writer
too, so writes still succeed there.
Tests (1433 total, +5 vs prior tip):
- tests/unit/indexes/columnStore/column-store-interchange.test.ts —
pins down the contract: (1) flush writes the raw blob, NO legacy
envelope; (2) DELETED bitmap likewise; (3) a legacy-format on-disk
segment loads correctly via the fallback; (4) legacy DELETED bitmap
ditto; (5) round-trip in the new format.
- All 101 existing ColumnStore tests pass — the new write path is
exercised by the existing lifecycle tests (MemoryStorage has the blob
primitive, so the new branch fires).
Cortex already registers the vectorStore:mmap provider (its Rust
NativeMmapVectorStore), but brainy has never consumed it — preloadVectors
and getVectorSafe still go straight to storage.getNounVector for every id,
even when an mmap layer is available. This wires the consumer end.
Architecture:
- NEW MmapVectorBackend (src/hnsw/mmapVectorBackend.ts) — bridges brainy's
UUID-keyed vector reads to a int-slot mmap file via the
vectorStore:mmap provider. Slots are addressed by the stable int id
from the post-2.4.0 #1 EntityIdMapper (the foundation this depends on).
Auto-grows the file (doubling) when a write lands beyond capacity, so
HNSWIndex never has to think about sizing. The class never touches
per-entity storage — it owns only the mmap layer.
- HNSWIndex changes — adds a vectorBackend field + a setVectorBackend
setter. The vector read paths (preloadVectors, getVectorSafe) try the
mmap layer first; on a storage fallback hit, they LAZILY write back into
the mmap slot. An upgraded install converges to the zero-copy fast path
under live traffic — no big-bang migration step. The legacy per-entity
path is preserved and still used when no backend is set.
- brainy.ts wiring — a new private wireMmapVectorBackend() runs once
during init, after plugin activation + metadataIndex setup. It activates
the backend only when (a) the vectorStore:mmap provider is registered,
(b) the storage adapter resolves a real local path via
getBinaryBlobPath(), and (c) the metadata index exposes its idMapper.
Cloud adapters return null on (b) and the backend is silently skipped;
HNSWIndex's behaviour is then identical to pre-2.4.0.
- Provider interfaces in plugin.ts — VectorStoreMmapProvider and
VectorStoreMmapInstance document the contract cortex's class fulfils
(the class IS the provider — static factory methods). Brainy depends on
the interfaces, not on cortex; the structural match is verified when
cortex 2.4.0 picks up this brainy release.
Tests (1428 total, +11 vs pre-2.4.0):
- tests/unit/hnsw/mmap-vector-backend.test.ts — 6 unit tests with an
in-memory mock provider. Covers round-trip, batch reads with interleaved
misses, slot stability (no re-slotting on overwrite), file growth without
data loss, idempotent open, and null returns for unwritten slots. The
real perf integration with cortex's NativeMmapVectorStore is exercised
when cortex 2.4.0 wires this in.
- tests/unit/utils/entity-id-mapper-stability.test.ts — moved here from
tests/regression/ (which is NOT in the unit-config include glob, so the
five #23 tests were not actually being run by npm test). The unit
config matches tests/unit/**/*.test.ts.
The 2.4.0 #2 follow-up will be the chunked-segment layout for remote
storage adapters (S3 / R2 / GCS) where a single growing file doesn't fit
immutable objects. For 2.4.0 release: local-FS only.
Previously metadataIndex.rebuild() called idMapper.clear() which reset nextId
to 1 and renumbered every UUID by re-insertion order. Any consumer that had
persisted int-keyed data against the old map was silently invalidated — and
2.4.0's vector mmap store (#20), graph link compression (#21), and column-store
JS↔native interchange all need persisted int indices that survive a rebuild.
Remove the unconditional clear() in rebuild(). The rebuild already re-iterates
every entity via idMapper.getOrAssign(uuid), which returns the existing int
unchanged for known UUIDs. Stale UUID→int entries for entities no longer in
storage persist as harmless memory overhead; a dedicated prune step can be
added if it ever matters.
clearAllIndexData() — the explicit nuclear recovery path — keeps its existing
idMapper.clear() call (renumbering is intentional there), and now logs a
prodLog.warn making it explicit that any persisted int-keyed data is invalidated
and must be rebuilt from canonical sources.
Strengthened the EntityIdMapper class JSDoc to document the stability guarantee
as a contract — append-only getOrAssign, monotonic nextId, remove() leaves
permanent holes, rebuild() never renumbers, only clear() does.
Added tests/regression/entity-id-mapper-stability.test.ts pinning down the
five-point contract: (1) single-rebuild stability; (2) many-rebuild stability;
(3) post-rebuild adds get fresh monotonic ints; (4) removes leave permanent
holes — new entities never recycle; (5) clearAllIndexData() explicitly
renumbers (the documented destructive path).
Foundation for 2.4.0 #2-#4. Full test suite (62 files, 1417 tests) green.
Adds a swappable sort:topK seam for the find() result-ranking hot path. Brainy
ranks candidate results by relevance score then slices to offset+limit; for large
candidate sets that is an O(N log N) full sort even though only the page is kept.
New utils/resultRanking.ts exposes rankIndicesByScore (top-K index selection) with a
JS default (sortTopKIndicesJs) that is byte-identical in ordering to the previous
results.sort((a, b) => b.score - a.score) + slice: descending by score, stable ties
by original index (ES2019+ stable sort semantics). setSortTopKImplementation lets a
native provider (e.g. cortex's Rust partial-sort) replace it; the provider returns
indices into a scores array and only has to match the JS index ordering.
brainy.ts resolves the sort:topK provider in init() (same pattern as distance:sq8)
and wires both relevance-score sort sites in find() through rankIndicesByScore +
reorderByIndices. rankIndicesByScore validates a native provider's output (length,
range, no duplicates) and falls back to JS on any inconsistency, so a query never
returns wrong or duplicated results. No provider registered = unchanged JS behavior.
Tests: unit coverage of the dispatcher (JS ordering vs Array.sort across 200 random
trials incl. ties, provider routing, and fallback on bad/throwing providers) plus
find() integration proving ranking routes through a registered provider, that the
provider and JS fallback produce identical ids/scores on the same data, and that
pagination requests k = offset + limit with descending ordering.
Makes distanceSQ8 swappable via setSQ8DistanceImplementation and wires brainy.ts to
install a registered 'distance:sq8' provider (e.g. cortex's Rust SIMD), with the JS
distanceSQ8Js as the default fallback. The provider signature is byte-compatible with
the native cosineDistanceSq8, so HNSW SQ8 reranking transparently uses native distance
when cortex is loaded. Native==JS numeric equivalence is asserted by the cross-language
parity suite.
Introduce a first-class binary-blob storage primitive on the StorageAdapter
contract and implement it across all storage backends. This stores opaque byte
payloads verbatim instead of base64-in-JSON, eliminating the ~33% inflation and
full-materialization cost of the JSON envelope. It unblocks zero-copy,
mmap-able column-store segments and batch vector I/O at billion scale.
New methods (declared abstract on BaseStorageAdapter, the class that implements
StorageAdapter, and added to the StorageAdapter interface):
saveBinaryBlob(key, data) raw write, atomic on real filesystems
loadBinaryBlob(key) exact bytes, or null if absent
deleteBinaryBlob(key) idempotent (missing is ignored)
getBinaryBlobPath(key) real local fs path where one exists, else null
Shared key -> location convention across every adapter: the key's
"/"-separated segments nest under a `_blobs/` prefix and are suffixed with
`.bin`, e.g. "graph-lsm/source/sstable-123" ->
"<root>/_blobs/graph-lsm/source/sstable-123.bin". Blobs are not branch-scoped
(COW): they are immutable producer-managed segments.
Per-adapter behavior:
- FileSystemStorage: writes under <rootDir>/_blobs via tmp+rename; returns the
real on-disk path so native code can mmap it directly. Path convention matches
the existing MmapFileSystemStorage subclass byte-for-byte.
- S3CompatibleStorage / R2Storage / GcsStorage / AzureBlobStorage: put/get/delete
raw octet-stream objects; getBinaryBlobPath returns null (remote stores have no
local path).
- MemoryStorage: defensive-copied Map<string, Buffer>; null path; cleared on
clear().
- OPFSStorage: stores raw bytes in the OPFS tree; null path.
- HistoricalStorageAdapter: read-only — save/delete throw; load resolves the
blob from the historical commit tree; null path.
Tests: tests/unit/storage/binaryBlob.test.ts exercises save/load round-trip
(byte-identical, incl. non-UTF8 bytes), overwrite, delete-then-load, load-missing,
and getBinaryBlobPath behavior for all eight adapters. Cloud adapters run against
in-memory client fakes that drive the real adapter code; OPFS runs against an
in-memory FileSystem Access API mock; the historical adapter commits a blob into
a real COW tree. 59 new tests; full unit suite (1398 tests) green.
Replace localeCompare (default-locale, non-deterministic across environments —
unsafe for a persisted sorted index) with UTF-8 byte / code-point order via a
shared compareCodePoints() helper. String ordering is now deterministic and
byte-identical to the native column store / aggregation sort, so results are the
same with or without the native accelerator. Covers the aggregate orderBy sort
and all 5 column-store comparison sites (tail buffer, merge sort, binary search).
Adds compareCodePoints unit tests.
Add 'percentile' (with a 'p' fraction in [0,1]) and 'distinctCount' to the
aggregation engine. Both are exact, computed from a per-metric value multiset
(MetricState.valueCounts) maintained incrementally and delete-safe; percentile
uses numpy-linear interpolation. The multiset is JSON-serializable so results
survive persistence. 35 aggregation unit tests pass.
- groupBy now supports { field, unnest: true }: an entity contributes once per
distinct element of an array field (tag frequency / faceted counts). Duplicate
elements on one entity count once; an empty/missing array joins no group. The
incremental add/update/delete paths fan out across the unnested groups.
- extractEntities/extractConcepts batch-embed the unique candidate spans in one
embedBatch call instead of one embed() per candidate (N sequential model calls);
falls back to per-candidate embedding if the batch fails. No behavior change.
New report APIs:
- brain.queryAggregate(name, params) returns the clean AggregateResult[] shape
({ groupKey, metrics, count }) directly, instead of the search-Result wrapper that
find({ aggregate }) returns.
- HAVING: find({ aggregate, having }) and queryAggregate filter groups by computed
metric values (e.g. revenue > 1000), complementing where (which filters group keys).
Evaluated per group: O(groups), independent of entity count.
Fixes (all reproducible on Node; surfaced under Cortex+Bun by Memory):
- Aggregate backfill-on-define: defining an aggregate over a store that already holds
matching entities returned []. It now backfills from existing entities on first query
(storage-agnostic via getNouns), so it works under durable backends that reopen
pre-populated. groupBy:['noun'] resolves to the entity type; find({ aggregate }) rows
expose groupKey/metrics/count at the top level.
- Multi-hop find({ connected: { depth, via } }) honors depth and via at every hop
(was 1-hop only, with verb filtering applied to hop 1 only); BFS bounded by limit.
- extractEntities no longer bleeds a neighbour's type indicator across candidates; each
candidate is typed by its own span. Also fixes the "Dr." title pattern.
Adds real-embedding regression tests; full unit suite green.
Three advanced-API correctness fixes, all reproducible on Node (not Bun-specific):
- Entity/concept extraction returned []. SmartExtractor combined agreeing signals
with a weighted sum compared against an absolute 0.60 gate, so a confident
low-weight signal lost selection to a mediocre high-weight one that then failed
the gate, dropping the whole result. Select and gate on a normalized weighted
average instead. The public `confidence` option now controls the threshold (was
a dead hardcoded 0.60), and the embedding-signal timeout is raised 100ms -> 2000ms
so the neural signal is not silently dropped on slower runtimes.
- Multi-hop find({ connected }) returned only the 1-hop neighbour. executeGraphSearch
ignored depth/via; it now delegates to the depth-aware neighbors() BFS.
- find({ aggregate }) hid groupKey/metrics/count under .metadata, so callers
expecting AggregateResult saw empty rows. Expose those fields at the top level.
Adds real-embedding regression tests in tests/integration/advanced-apis-regression.test.ts.
Two production correctness defects fixed together so consumers can land
one upgrade.
BR-FIND-WHERE-ZERO — find() returned [] and stats() reported 0 entities
for any workspace whose data was written after the 7.20.0 column-store
refactor. Root cause: getStats() and the getIds() fallback still read
from the deleted sparse-index path. Separately, BaseStorage.getNounType()
was hardcoded to return 'thing', poisoning type-statistics.json with
every noun attributed to that bucket.
- MetadataIndex.getStats() reads from ColumnStore + idMapper.
- MetadataIndex.getIdsFromChunks() throws BrainyError(FIELD_NOT_INDEXED)
when neither store has the field. getIdsForFilter() catches per
clause, logs once, returns [].
- BaseStorage.nounTypeByIdCache populated in saveNounMetadata_internal
and consumed in saveNoun_internal. flushCounts() now persists the
Uint32Array counters too, so readers see fresh per-type counts.
- Self-heal at init: loadTypeStatistics() auto-rebuilds when the
poisoned signature is detected. rebuildTypeCounts() is now public for
use from `brainy inspect repair`.
- Dead state removed: dirtyChunks, dirtySparseIndices,
flushDirtyMetadata().
BR-DEFENSIVE-INTERFACE — 7.21.0 called supportsMultiProcessLocking()
unconditionally, crashing on older Cortex storage adapters that predate
the method. New hasStorageMethod(name) helper gates every new-method
call site. Older adapter triggers a one-line warning at init pointing
at the recommended plugin version.
Tests:
- new tests/integration/find-where-zero.test.ts (7 cases)
- new tests/integration/cortex-compat.test.ts (5 cases)
- multi-process-safety.test.ts updated to enforce correct counts
- 1329/1329 unit + 24/24 new integration tests passing
