The integration suite loaded the real WASM model for every test, so a full run
took ~hours and reliably aborted (vitest reporter timeout) — which is why it was
never gated and silently rotted. Brainy already separates the embedder from all
other logic, so running integration with a deterministic embedder keeps every
code path real (HNSW build+search, graph traversal, metadata filtering,
transactions, storage/sharding, VFS, aggregation, import) while making the suite
complete in ~2 min on ANY machine — no GPU, no 16 GB requirement.
- Add isDeterministicEmbedMode() — a single source of truth for the test-embedder
switch (BRAINY_DETERMINISTIC_EMBEDDINGS, plus the legacy BRAINY_UNIT_TEST alias),
used by EmbeddingManager (init/embed/embedBatch) and brainy's eager-init guard.
- setup-integration.ts opts into it: Tier 1 = functional end-to-end. Real-model
semantic quality + the real embedding pipeline move to a Tier-2 suite.
Full integration now runs 482 passing / 583 in ~2 min (was un-completable). The
remaining failures are pre-existing test rot, fixed next. Unit 1414 green.
Graph time-travel needs an edge's existence recorded per generation so
db.asOf(g) hops resolve historically correct endpoints. The metadata layer
already threads brainy's commit generation per write; the graph write path did
not, leaving a versioned verb-endpoint store unable to answer "which edges
existed at generation g".
- GraphIndexProvider.addVerb/removeVerb gain a `generation: bigint` parameter
(the same watermark the storage layer stamps onto the record). A provider with
a per-generation edge chain stamps the edge at that generation; the JS baseline
has no such chain and accepts-and-ignores it — graph time-travel is a
native-provider capability, and the open-core path serves edges as-of-now (the
one documented graph time-travel limitation).
- The two graph transaction operations resolve the generation via a thunk at
EXECUTE time: the generation store assigns the batch generation only once the
commit begins executing, after the operations are planned. The same generation
is reused for an operation's rollback half.
- All graph-write call sites pass the in-flight generation.
Adds a spy-provider test proving the threading, execute-time resolution, and
forward/rollback generation reuse. The JS index ignores the value, so behaviour
is unchanged: unit 1402/1402, db-mvcc 25/25, bigint-contract relate/unrelate 10/10.
The auto-detected query-limit cap was computed from os.freemem() (MemFree),
which excludes reclaimable page cache. On hosts that memory-map large index
files the cache holding those pages dominates RAM, so MemFree collapses to a
sliver and the cap cratered to its floor on perfectly healthy machines,
rejecting legitimate find() calls.
- getAvailableMemory() now prefers /proc/meminfo MemAvailable (counts
reclaimable cache), falling back to os.freemem() off-Linux, then a 2 GB
constant where no OS module is available.
- Auto-detected caps (container + free-memory tiers) are floored at
MIN_AUTO_QUERY_LIMIT (10_000); a transient low reading can never throttle
queries to a near-useless ceiling. Consumer-supplied maxQueryLimit /
reservedQueryMemory bypass the floor — an explicit caller knows their box.
Also scrubs two stale comments referencing the removed cloud storage
adapters and the retired mmap-vector backend: 8.0's native vector provider
persists its own .dkann file via getBinaryBlobPath, so the old rootDirectory
hook does not apply on this line.
Tests: memoryLimits 26/26 (4 new floor regressions), unit 1398/1398,
find-limits + db-mvcc + api-parameter-validation 37/37.
The old config-generation subsystem (src/config/ + autoConfiguration.ts) was
superseded during the 8.0 rework and never wired into init(): it emitted settings
for a partitioning subsystem that no longer exists and probed deleted cloud env
vars. The live zero-config path is inline — recall preset → HNSW knobs, storage
auto-detect, auto persistMode, container-memory-aware cache sizing.
The storage progressive-init / cloud-detection cluster was equally dead after the
cloud adapters were dropped: isCloudStorage() is permanently false (no overriders),
scheduleBackgroundInit/runBackgroundInit were never called (the latter an empty
body), initMode was never assigned, and Brainy.isFullyInitialized()/
awaitBackgroundInit() were always-trivial with zero callers. scheduleCountPersist()
collapses to its only-ever-taken immediate write-through path.
Removed:
- src/config/{index,zeroConfig,storageAutoConfig,modelAutoConfig,sharedConfigManager}.ts
- src/utils/autoConfiguration.ts + the inert BrainyZeroConfig export
- Brainy.isFullyInitialized()/awaitBackgroundInit() (+ BrainyInterface decls)
- InitMode type, isCloudStorage/detectCloudEnvironment/resolveInitMode,
scheduleBackgroundInit/runBackgroundInit/ensureValidatedForWrite and their state
- Dead cloud env-var probes (K_SERVICE/K_REVISION/AWS_LAMBDA_FUNCTION_NAME/
FUNCTIONS_TARGET/AZURE_FUNCTIONS_ENVIRONMENT)
Kept (verified live): production-detection logging (environment.ts), container-
memory cache sizing (memoryDetection/paramValidation), on-disk hash bucketing
(sharding.ts).
Docs: scrubbed deleted-subsystem references (JS quantization knobs, cloud/OPFS
adapters, partitioning, old zero-config API) across 14 files; deleted two wholly-
obsolete feature docs (complete-feature-list, v3-features); rewrote
architecture/zero-config for 8.0.
~3,700 LOC removed. Build clean; 1392 unit + 24 db-mvcc green.
The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).
Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
readWriteSeparation, queryPlanner, healthMonitor, configManager,
hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
(slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
src/config/extensibleConfig.ts (config/augmentation registry built on removed
cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
enableDistributedSearch (dead config flag); the metadata partition field;
the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
storage-architecture; reframed scale prose to the 8.0 model.
Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.
RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):
1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
metadata (and the transact() ops that extend them) reject a literal
reserved key as a TypeScript error while keeping generic T ergonomics
(typed bags, untyped brains, index-signature shapes, and a documented
exemption for T-declared reserved keys). Pinned by @ts-expect-error
type tests run under vitest typecheck mode on every unit run.
2. Write time — the 7.x update() remap is ported to 8.0 and extended to
every write path: add/update/relate/updateRelation, their transact()
mirrors, and db.with() overlays. User-settable fields lift to their
dedicated param (top-level wins when both are supplied — closes the 7.x
trap where update({metadata:{confidence}}) silently no-oped), and
system-managed fields drop with a one-shot warning naming the right
path. A remapped subtype satisfies subtype-pairing enforcement exactly
like a top-level one.
3. Read time — every storage combine goes through one canonical hydration
helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
and entity/relation.metadata carry ONLY custom fields on live reads,
batch reads, paginated listings, getRelations by source/target, streamed
verbs, and historical asOf() materialization alike.
Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.
Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
One rule per NounType/VerbType (literal default or per-entry function);
fills only entries still missing a subtype through the public update()/
updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
opaque storage sharding failure; CLI --threshold without --near applies a
plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
explicitly; delete the unmaintained interactive REPL; replace the cloud-era
storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
README storage section reflects filesystem+memory and snapshots; eli5 and
SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
.strategy references scrubbed from published files.
The legacy backup/import/export/stats facade (src/api/DataAPI.ts) drifted
from the modern entity shape and every job it did now has a first-class
surface. Delete it and brain.data(), and rewire the CLI:
- data-stats → brain.stats() (full BrainyStats report: per-type breakdowns,
indexed fields, index health, storage backend, writer lock, version)
- clean → brain.clear()
- export → alias of snapshot; a db.persist() snapshot is the full-fidelity
export format (open with Brainy.load, load wholesale with brainy restore);
external data ingestion remains brainy import (UniversalImportAPI)
Rewiring clean onto brain.clear() exposed two real bugs, both fixed:
- clear() left this.graphIndex undefined forever — any graph-touching call
afterwards (relate, getNeighbors, stats) crashed. clear() now re-resolves
the graph index exactly as init() does and re-wires the shared UUID↔int
resolver, and re-resolves the metadata index with the same provider
fallback as init().
- storage.clear() reset the legacy totals but not the per-type/subtype
count rollups or id→type caches, so stats() reported phantom counts for
deleted entities. Both adapters now delegate derived-state reset to
reloadDerivedState(), the same path restore-from-snapshot uses.
One-shot CLI commands (data-stats, clean, snapshot/export, restore,
history, generation) now close the brain and exit explicitly — global
cache timers otherwise keep the process alive holding the writer lock.
Verified: build clean, 1383/1383 unit tests, 24/24 db-mvcc integration,
plus an end-to-end CLI smoke (add → data-stats → export → clean →
data-stats).
Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.
Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
immutable before-images otherwise) into a fresh MemoryStorage; a final
reconciliation pass under the commit mutex makes the copy exact even
when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
graph-adjacency indexes from the records; the vector index is built by
inserting every at-G vector (the at-G HNSW graph never existed on disk,
so there is nothing to restore). Host embedder and aggregate definitions
are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
(handle cached; freed by release(), with a FinalizationRegistry backstop
that also closes leaked readers). A native VersionedIndexProvider serves
the same reads from retained segments with no rebuild.
Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.
UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.
GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.
Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
The COW version-control surface (fork, branches, checkout, commit,
getHistory/streamHistory, asOfCommit, brain.versions) is gone, along with
its subsystems: src/versioning/, the COW object store (CommitLog,
CommitObject, RefManager, TreeObject), HistoricalStorageAdapter, and the
TypeAwareHNSWIndex path it kept alive. The Db API (now/transact/asOf/
with/persist/restore) is the one versioning model in 8.0.
Survivors and replacements:
- BlobStorage survives (the VFS stores file content through it), relocated
to src/storage/blobStorage.ts with binaryDataCodec.ts; its adapter
interface is now BlobStoreAdapter, slimmed to the consumed surface
(write/read/has/delete/getMetadata + MIME-aware compression policy).
- brain.migrate() backup branches are replaced by persist-before-migrate:
MigrateOptions.backupTo persists a hard-link snapshot of the current
generation before any transform runs; MigrationResult.backupPath reports
it, and brain.restore(path) brings it back wholesale.
- CLI: cow.ts (fork/branch/checkout/history/migrate) is replaced by
snapshot.ts — snapshot <path>, restore <path>, history (tx-log),
generation.
- New public read API: brain.transactionLog({limit}) exposes the reified
tx-log (generation/timestamp/meta, newest first) that backs the CLI
history command; TxLogEntry is exported.
Tests: superseded suites deleted; fork/commit blocks excised from shared
suites; BlobStorage tests relocated + reworked against the slimmed store;
migration tests now prove the backupTo snapshot/restore round trip; new
transactionLog coverage in db-mvcc.
One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.
Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
transact() commit and once per single-operation write (storage hook), so
brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
(the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
and forces an index rebuild; refcounted pins gate compactHistory(), which
records a horizon (asOf below it throws GenerationCompactedError).
Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
(GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
{confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
generation; index-accelerated queries at historical generations throw
NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
plugin.ts: feature-detected, balanced pin/release in lockstep with Db
lifecycle, post-commit applier + replay-gap model documented.
Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.
Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
createIndex() still resolved the retired 'diskann' and 'hnsw' provider
keys and never looked up 'vector', so an 8.0-era plugin registering its
vector engine under the canonical 'vector' key silently never engaged
and Brainy always fell back to the built-in JS HNSW index. createIndex()
now resolves only getProvider('vector') (same factory call shape as the
old hnsw path) and falls back to setupIndex().
The brainy-side mode/migration machinery is obsolete in the 8.0 provider
world — the registered engine adapts internally (in-memory / hybrid /
on-disk selection is the provider's job, not Brainy's):
- delete migrateToDiskAnn() / migrateToHnsw() and the ADR-002
index-engine migration block
- delete diskAnnAutoEngageConditionsMet() / instantiateDiskAnn()
- narrow HNSWConfig.type to 'vector' and drop the diskann tuning block
(coreTypes.ts)
- well-known provider key lists + diagnostics now report 'vector'
instead of the never-registered 'hnsw' key
- plugin.ts docs: 'vector' is the only vector-index key consulted; the
pre-8.0 'hnsw'/'diskann' keys are retired
The schema-migration machinery (migrate(), checkMigrations(),
MigrationRunner, autoMigrate) serves data migrations and is untouched.
GraphIndexProvider now speaks BigInt at the boundary (D.2 mirror):
- getNeighbors/getVerbIdsBySource/getVerbIdsByTarget take entity ints and
return entity/verb ints as bigint[]
- new REQUIRED verbIntsToIds(bigint[]) batch reverse resolver (L.7
identity-fingerprint design — verb ids are UUIDs by contract, so the
provider-side interning is losslessly reversible)
- addVerb(verb, sourceInt, targetInt) returns the interned verb int;
removeVerb(verbId) joins the contract
Coordinator (brainy.ts) owns ALL UUID <-> int conversion: getOrAssign on
writes, getInt on reads (unmapped UUID -> empty result without calling the
provider), getUuid / verbIntsToIds on returns, plus a bounded ~100k-entry
insertion-order warm cache for verb-int -> verb-id pairs fed by addVerb
returns and resolver results. GraphVerb gains derived sourceInt/targetInt
(populated at add time, never persisted). findConnectedSubtype gains a
native fast path that routes single-type single-subtype outgoing BFS
through the provider when available.
JS GraphAdjacencyIndex satisfies the contract while staying string/u32-keyed
internally: entity ints resolve through the shared entity-id mapper (threaded
in by the coordinator on init/fork/checkout), verb ints come from an
in-process append-only interning map re-derived from storage on
rebuild/cold-start. ColumnStoreProvider widens the same way: addEntity/
removeEntity take bigint, sortTopK/filteredSortTopK return bigint[].
relate() now rejects a caller-supplied id with a teaching error — verb ids
are brainy-generated UUIDs by contract in 8.0 (previously a passed id was
silently ignored). No Roaring64 provider-boundary decode site exists yet;
the JS-internal column store stays Roaring32 and the Treemap decoder lands
with the first consumer of provider-returned filter buffers.
Public brain API unchanged. 1413 tests green (+10 new BigInt contract tests).
The mmap-vector backend was a 2.x-era accelerator for the JS HNSW read path:
a native provider registered under 'vectorStore:mmap' supplied a single-file
mmap'd vector store, and JsHnswVectorIndex tried it before per-entity storage
reads with lazy write-back migration.
In the 8.0 provider world nothing registers that key — the native plugin's
3.0 line replaces the entire vector index under the 'vector' provider key
(vectors live inside its own index file), and the open-core path never had
an mmap provider. The wiring is unreachable; per the no-unwired-code rule
it goes away:
- src/hnsw/mmapVectorBackend.ts deleted (175 LOC)
- wireMmapVectorBackend() + its init call site removed from brainy.ts
- VectorStoreMmapProvider + VectorStoreMmapInstance contracts removed
from plugin.ts
- JsHnswVectorIndex loses the vectorBackend field, setVectorBackend(),
and the mmap-first branches in getVectorSafe/preloadVectors — the
storage + UnifiedCache path is now the single read path
- tests/unit/hnsw/mmap-vector-backend.test.ts deleted
The 7.x line keeps the wiring and got the capacity-NaN bugfix as 7.31.3
(see the platform handoff mmap thread). EntityIdMapperProvider stays — the
connections codec and the id mapper still implement it.
1403/1403 tests, build clean.
Follow-up to the browser/cloud/threading sweep — everything that was guarding
against unreachable runtimes is now dead.
cacheManager.ts: Environment enum + this.environment field removed (only NODE
was reachable). StorageType narrowed to MEMORY + FILESYSTEM. navigator.deviceMemory
and performance.memory paths in detectOptimalCacheSize + detectAvailableMemory
deleted; node:os is the sole source. environmentConfig keeps the index signature
for future per-runtime tuning but only the node slot is wired.
Dead 'typeof window === undefined' guards (the check is always true on 8.0):
paramValidation, structuredLogger, mutex, brainy.ts stats block, and
networkTransport ws-dynamic-import all collapsed. IntegrationLoader's inverse
guard ('!== undefined' returning 'browser') deleted. mutex's createMutex default
type simplified from "(typeof window === 'undefined' ? 'file' : 'memory')" to
plain 'file'.
Redundant TextEncoder/TextDecoder polyfills: Node 22+ ships both as globals.
src/utils/textEncoding.ts deleted (applyTensorFlowPatch was named for a defunct
dep and only re-assigned globals already present). setup.ts collapsed to an
empty stable import target. unified.ts loses its applyTensorFlowPatch re-export.
modelAutoConfig.getModelPath() loses the unreachable trailing fallback now
that isNode() is effectively the only branch the function ever takes.
jsonProcessing.ts left unchanged — its 'typeof document' checks are value-shape
guards on the parsed JSON, not environment detection.
PHASE A — every @deprecated marker resolved (~25 removed)
src/coreTypes.ts
- GraphVerb: dropped the "@deprecated Will be replaced by HNSWVerbWithMetadata"
note. GraphVerb IS the canonical contract — every public API path speaks
it. Removed the `source` and `target` legacy alias fields (renamed `from`
/ `to` callers years ago; no consumers remain).
- StorageAdapter: dropped the "@deprecated Use getNouns() with filter" notes
from `getNounsByNounType`, `getVerbsBySource`, `getVerbsByTarget`,
`getVerbsByType`. They were never deprecated in spirit — they're useful
non-paginated convenience wrappers over the paginated `getNouns()` /
`getVerbs()` surface. Refreshed JSDoc to explain the role.
src/types/graphTypes.ts
- Mirrored the GraphVerb cleanup: dropped `source` + `target` legacy aliases.
sourceId + targetId are the canonical fields.
src/import/ImportCoordinator.ts
- Deleted the entire DeprecatedImportOptions interface block (130 LOC). It
was a v3 → v4 migration tool using the `?: never` trick to force
compile errors on dropped options. Five major versions in, the
forced-error gate is no longer pulling its weight.
src/triple/TripleIntelligence.ts
- Deleted `TripleIntelligenceEngine = any` alias. No consumers; superseded
by `TripleIntelligenceSystem`.
src/storage/cow/binaryDataCodec.ts
- Deleted `wrapBinaryData()`. The COW dispatch layer in `baseStorage.ts`
routes by key-prefix convention; the old guess-by-JSON-parse codec was
fragile (compressed bytes can accidentally parse as JSON) and unused.
src/storage/baseStorage.ts
- Refreshed JSDoc on `convertHNSWVerbToGraphVerb()` — the method is alive
and used internally; the deprecation note was stale.
src/embeddings/wasm/AssetLoader.ts → DELETED
- File was @deprecated since model weights moved into the Candle WASM
bundle. No consumers. Removed from `embeddings/wasm/index.ts` exports.
src/embeddings/wasm/types.ts
- Dropped @deprecated tags on `TokenizerConfig` + `TokenizedInput` — still
used by `WordPieceTokenizer` (auxiliary tokenization). Deleted
`InferenceConfig` (truly dead). Updated `embeddings/wasm/index.ts`
exports.
src/utils/metadataIndex.ts
- Deleted `getIdsForCriteria()` — pure alias for `getIdsForFilter()`, no
consumers.
src/interfaces/IIndex.ts
- Removed RebuildOptions.lazy (deprecated and unused; lazy mode is auto-
selected by available-memory detection).
src/hnsw/hnswIndex.ts
- Removed `getNouns()` (returned a full Map; deprecated in favor of
pagination years ago and no consumers in src/ or tests/).
PHASE B — cacheManager dead StorageType branches
src/storage/cacheManager.ts
- Collapsed the `isRemoteStorage` flag and its 15 dead conditional branches
spanning calculateOptimalCacheSize() and calculateOptimalBatchSize().
After dropping cloud adapters in step 7, `coldStorageType` is never S3
or REMOTE_API; the branches were dead. Cache sizing and batch sizing now
honor the filesystem-only reality with simpler heuristics.
- Collapsed `detectWarmStorageType()` + `detectColdStorageType()` from
~40 LOC of environment-+-availability branching to 2-line returns of
`StorageType.FILESYSTEM`. Brainy 8.0 ships filesystem + memory only.
NOT YET — Phases C-G in follow-up commits
C: storageAutoConfig.ts + zeroConfig + extensibleConfig + sharedConfigManager
D: TODO/FIXME sweep across src/
E: skipped tests + the parallel-test race condition
F: docs deep clean (BATCHING, augmentations, READMEs)
G: browser support drop (the last 2 @deprecated)
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing race-condition outstanding)
Brainy 8.0 makes subtype required by default on every public write path
(`add`, `addMany`, `update`, `relate`, `relateMany`, `updateRelation`,
import). Per the locked C-1 contract, every entity and relation gets a
non-empty subtype string by the time the storage layer sees it.
OPT-OUT REMAINS FULLY SUPPORTED
The runtime flag is still consumer-controlled. Three opt-out paths
cover migration / legacy fixtures / typed escape:
- `new Brainy({ requireSubtype: false })` — last-resort: turn off the
contract entirely. Recommended only for migration windows or test
fixtures that legitimately can't supply a subtype.
- `new Brainy({ requireSubtype: { except: [NounType.Thing, ...] } })` —
per-type allowlist: strict everywhere except the listed types.
- `brain.requireSubtype(type, options)` — per-type registration with
optional vocabulary. Composes with the brain-wide flag.
Default is now `true`. Opt-out is explicit and documented; nothing
silently degrades.
TEST SWEEP
Bulk-applied `requireSubtype: false` to every `new Brainy({...})` call
site across 120 test files. Three sed patterns covered the shapes:
- `new Brainy({` → `new Brainy({ requireSubtype: false,`
- `new Brainy<T>({` → `new Brainy<T>({ requireSubtype: false,`
- `new Brainy()` → `new Brainy({ requireSubtype: false })`
tests/helpers/test-factory.ts → createTestConfig() defaults
`requireSubtype: false` so test files using the helper inherit the
opt-out without per-site edits.
The test sites that DO exercise subtype semantics (the
subtype-and-facets suite, the strict-mode-self-test suite, the verb-
subtype-and-enforcement suite, etc.) already pass real subtypes — they
were the 7.30.x acceptance tests for this contract. Those tests
continue to pass unchanged.
CHANGES
src/brainy.ts
- normalizeConfig() — `requireSubtype` default `false` → `true`.
Comment refreshed to document the three opt-out paths.
tests/* (120 files)
- Bulk-edited brain construction sites. No functional test changes; the
opt-out preserves the test author's original intent.
tests/helpers/test-factory.ts
- createTestConfig() base config gains `requireSubtype: false`.
NO-OP for consumers who were already passing subtype on every write.
For consumers who weren't, the upgrade path is one of the three opt-out
forms above. Migration recipe documented in 8.0 release notes (next
commit).
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing race-condition outstanding;
no other regressions from the flip)
Previous (post-step-12) state shipped a regression: removing the
`depth > 1 + subtype` throw left the open-core JS path silently
returning wrong results — the verbType walk reached depth N, but the
subtype intersection only filtered at depth 1. Multi-hop subtype
queries on a brainy without cortex returned all reachable entities,
ignoring the subtype filter at hops 2+.
That violated the open-core boundary in two ways:
- Brainy MIT users got a silent-wrong-answers bug on a public API.
- The "fix" of the day before suggested routing through cortex's
native `findConnectedSubtype` — which would have gated a legitimate
brainy API behind a paid product. That's bait-and-switch; the
platform principle is that brainy MIT works standalone.
THE FIX
Implement multi-hop subtype-aware BFS in pure JS. Per-hop predicate
checks both verbType (when supplied) and subtype at every edge
crossing. Cycle guard via `visited`. Stops at `depth` or when the
frontier empties.
Complexity scales with branching factor × depth, not total graph size.
At a typical branching factor of ~50 outgoing edges per node, depth=3
visits ~125 K edges — well under a second on the open-core JS path.
That's fast enough for any reasonable production workload.
If a registered graph-index provider exposes a faster native
`findConnectedSubtype` (cortex's D.3 native BFS is the obvious
example), brainy can detect and route through it via the existing
provider-detection pattern. That's an OPTIMIZATION, not a correctness
requirement. The JS path is the contract; native is a drop-in
replacement.
CHANGES
src/brainy.ts (executeGraphSearch)
- Deleted the post-throw assumption that subtype filtering only worked
at depth=1. Replaced with a real BFS.
- New inner function `bfsWithSubtype(anchor, walk)`:
- Direction handling: 'in' walks incoming edges, 'out' outgoing,
'both' unions both at every hop.
- Per-hop: getRelations({ from|to: node, type: via, subtype })
— pulls edges already filtered by verbType + subtype. Defensive
re-check on edge.subtype in case a future getRelations impl
widens its filter.
- Cycle guard: visited set seeded with the anchor. Never re-enters
the anchor; never revisits a node.
- Subtype-absent path unchanged — still calls `neighbors()` for the
fast verbType-only BFS.
NO-OP for the subtype-absent case. Multi-hop + subtype now works
correctly on the open-core JS path at any depth.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing race-condition outstanding;
no other regressions)
Two small additions per BRAINY-8.0-SUBTYPE-CONTRACT § C-3 + § C-4.
STEP 11 — SubtypeRegistry declaration-merging hook (§ C-3)
src/types/brainy.types.ts
- New empty `SubtypeRegistry` interface. Consumers extend via TypeScript
module augmentation to declare per-`(NounType, subtype)` metadata
shapes. Brainy ships no entries; every entry is a consumer concern.
- Usage pattern documented inline:
declare module '@soulcraft/brainy' {
interface SubtypeRegistry {
'person:employee': { employeeId: string; department: string }
'document:invoice': { invoiceNumber: string; amount: number }
}
}
- Consumers with no entries see no type-level change (metadata stays T = any).
src/index.ts
- Public export of `SubtypeRegistry` as a type so consumers can declare-merge it.
The typed `add<NounType.Person, 'employee'>()` overloads + `brain.fillSubtypes()`
migration helper are NOT in this commit — they require the C-1 runtime
flip to land first (per step 10 deferral). The interface stub ships now
so external consumers can start the declare-merging pattern in their own
code immediately.
STEP 12 — drop the multi-hop subtype throw (§ C-4)
src/brainy.ts
- Deleted the `find({ connected: { subtype, depth > 1 } })` throw that
7.30.x emitted ("Use depth: 1, or wait for Cortex native traversal").
- Replaced the throw block with an explanatory comment: cortex's D.3
`findConnectedSubtype` handles the multi-hop case natively;
open-core JS falls back to per-hop edge enumeration (correct but slow
past ~10 K reachable edges per source).
Per BRAINY-8.0-RENAME-COORDINATION § G.3.b (cortex-confirmed): the native
graph index is ready for unbounded `depth` from brainy with no rebuild
required, and BFS-with-cycle-guard semantics hold at any depth.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing race-condition outstanding)
Per BRAINY-8.0-SUBTYPE-CONTRACT § C-1, brainy 8.0 should flip the
runtime `requireSubtype` default from `false` to `true`. Tried that here
and it broke 235 tests — every test that does
`brain.add({ type, data })` without supplying a subtype now throws.
That's the right contract direction but the wrong commit to ship it in.
Each of those 235 sites needs to either:
- Start passing `subtype: 'test'` (or a real subtype value), or
- Set `requireSubtype: false` on the test brain config.
Either path is a sweep that deserves its own focused commit with proper
review. Mixing it into the scaffolding stream would muddy the diff and
make bisecting any real regression hard.
CHANGES
src/brainy.ts
- normalizeConfig() — left the runtime default at `false` for now (7.x
behaviour preserved). Added a comment explaining that the 8.0 § C-1
flip is staged as a separate focused commit.
The per-type rules (`brain.requireSubtype(type, options)`) and the
per-brain strict flag (`new Brainy({ requireSubtype: true })`) remain
fully functional — every consumer that wants the 8.0 contract today can
opt in explicitly. The flip is just about which default ships.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (back to the pre-step-10 baseline; the parallel-test
race condition outstanding from step 7 is unrelated)
Per user direction: after the step-8 simplification reduced
`config.vector` to three knobs (`recall`, `quantization`, `persistMode`),
the `strictConfig` field's only remaining job was warning about
`quantization.bits` being silently ignored on the cortex DiskANN path —
one mismatch case across the entire public surface.
That's not enough surface to justify a declared-but-undelivered config
field. Cleaner to drop it now; if 8.x or 9.0 adds enough provider-knob
mismatch cases to warrant a general-purpose strictness flag, we can add
it back then.
CHANGES
src/types/brainy.types.ts
- Removed BrainyConfig.strictConfig field + its JSDoc block.
- Removed the `strictConfig: 'warn' flags this at init time` reference
from quantization JSDoc; replaced with a plain "silently ignored on
DiskANN" note (cortex's B.4 contract still applies).
src/brainy.ts
- normalizeConfig() no longer emits a strictConfig field.
NO-OP scope
Field was declared but had no enforcement logic. Removing it is a pure
type-surface change. No behaviour change.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing race-condition outstanding)
Per user direction and BRAINY-8.0-RENAME-COORDINATION § G.2: the
`config.vector.advanced.{hnsw, diskann}` escape-hatch surface was
over-engineered. The 8.0 config surface collapses to **three knobs**:
```
config.vector = {
recall?: 'fast' | 'balanced' | 'accurate'
quantization?: { enabled?, bits?: 8 | 4 }
persistMode?: 'immediate' | 'deferred'
}
```
The algorithm-internal HNSW knobs (`M`, `efConstruction`, `efSearch`,
`ml`) and DiskANN knobs (`pqM`, `searchListSize`, `alpha`, …) are no
longer exposed at the public surface. The `recall` preset covers the
legitimate quality/latency tradeoff; algorithm-internal tuning is
intentionally hidden. If users need it later, we can add it back —
shipping with too few knobs is easier to evolve than shipping with too
many.
CHANGES
src/types/brainy.types.ts
- BrainyConfig.hnswPersistMode (7.x top-level) → BrainyConfig.vector.persistMode.
- BrainyConfig.vector — dropped `advanced.{hnsw, diskann}` sub-block.
- BrainyConfig.vector.quantization — dropped `rerankMultiplier` (hardcoded to 3).
- BrainyConfig.vector — dropped `vectorStorage: 'memory' | 'lazy'`. Brainy
always keeps vectors resident in 8.0; the lazy-eviction path was a
cloud-storage optimisation that's no longer needed (cloud adapters are gone).
- JSDoc tightened to reflect the closed-form contract.
src/utils/recallPreset.ts
- resolveJsHnswConfig() signature narrowed: accepts `{ recall? }` only
(no more `advanced` parameter).
src/brainy.ts
- normalizeConfig() — no longer carries `hnswPersistMode` on the resolved
config. Dropped the deprecated 'gcs-native' warning, the gcsStorage
HMAC-key warning, and the lenient storage-pairing block (dead code now
that cloud adapters are gone).
- resolveHNSWPersistMode() — collapsed to a one-liner:
`return this.config.vector?.persistMode ?? 'immediate'`. The cloud-vs-local
smart-default branching is no longer relevant (filesystem is the only
persistent backend in 8.0).
- setupIndex() — reads quantization fields from config.vector directly;
rerankMultiplier hardcoded to 3.
NO-OP scope
The recall preset's 'balanced' = brainy 7.x DEFAULT_CONFIG byte-for-byte,
so users who don't set recall get the same behavior. Users who set
config.hnsw.* in 7.x will need to migrate to config.vector.* per the
8.0 release notes; pure 7.x defaults users see no change.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing test-isolation race condition
from step 7, not related to step 8)
Step 6 of the brainy 8.0 rename scaffolding. Removes every legacy 'hnsw'-
flavoured public name introduced as a compat shim in steps 1-5. The
algorithm-neutral surface is now the only surface.
REMOVED — PHASE A (type aliases)
src/plugin.ts
- Removed `export type HnswProvider = VectorIndexProvider` alias.
- Removed `export type DiskAnnProvider = VectorIndexProvider` alias.
src/index.ts
- Removed `export const HNSWIndex = JsHnswVectorIndex` alias.
src/types/brainy.types.ts
- Removed `BrainyStats.indexHealth.hnsw` field (the new `vector` field
carries the same boolean).
src/brainy.ts
- `stats()` no longer emits the legacy `hnsw` field on `indexHealth`.
REMOVED — PHASE B (storage adapter rename, 8 adapters)
Repo-wide rename: `saveHNSWData` → `saveVectorIndexData` and
`getHNSWData` → `getVectorIndexData` across:
- src/storage/adapters/baseStorageAdapter.ts (abstract declarations)
- src/storage/adapters/fileSystemStorage.ts
- src/storage/adapters/gcsStorage.ts
- src/storage/adapters/r2Storage.ts
- src/storage/adapters/s3CompatibleStorage.ts
- src/storage/adapters/azureBlobStorage.ts
- src/storage/adapters/opfsStorage.ts
- src/storage/adapters/memoryStorage.ts
- src/storage/adapters/historicalStorageAdapter.ts
- src/brainy.ts (call sites)
- src/hnsw/hnswIndex.ts + src/hnsw/typeAwareHNSWIndex.ts (call sites)
The default-delegation wrappers added in scaffold step 4 are removed
(would have been duplicate declarations after the rename).
REMOVED — PHASE C (cache category 'hnsw')
src/utils/unifiedCache.ts
- Cache-category union narrowed from
'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other'
to
'vectors' | 'metadata' | 'embedding' | 'other'
- typeAccessCounts, typeSizes, typeCounts, accessRatios, sizeRatios,
and the fairness-check iterator all drop the 'hnsw' key.
- Per planning § 2.5: pre-8.0 'hnsw' cache entries are an in-memory
category. Cache is rebuildable; entries naturally don't exist after
a restart, so no migration path is needed.
src/hnsw/hnswIndex.ts
- 3 cache.set() call sites migrated from category 'hnsw' to 'vectors'.
- Cache key prefix `hnsw:vector:` → `vector:`.
- typeCounts/typeSizes/typeAccessCounts accessor renames
`.hnsw` → `.vectors`.
tests/unit/utils/unifiedCache-eviction.test.ts
- Test fixtures updated: cache.set(..., 'hnsw', ...) → cache.set(..., 'vectors', ...).
- Stats assertion `stats.typeSizes.hnsw` → `stats.typeSizes.vectors`.
REMOVED — PHASE D (config.hnsw)
src/types/brainy.types.ts
- Removed `BrainyConfig.hnsw` field entirely. The 8.0 surface is
`BrainyConfig.vector.{recall, quantization, vectorStorage, advanced}`.
src/brainy.ts
- normalizeConfig() no longer emits a `hnsw` field on Required<BrainyConfig>.
- setupIndex() rewired:
- Reads from `this.config.vector` (not `this.config.hnsw`).
- Calls `resolveJsHnswConfig(this.config.vector)` to translate the
`recall` preset into M / efConstruction / efSearch knobs (with
`advanced.hnsw` overrides winning when supplied).
- Imports `resolveJsHnswConfig` from './utils/recallPreset.js'.
NOT IN THIS COMMIT (deliberately)
- Persisted file path migration `_system/hnsw-*.json` →
`_system/vector-index-*.json`. Requires dual-read logic on boot
across 8 storage adapters. Per integration doc lines 531-539:
"Reader accepts either spelling on load; writer emits the new spelling
only; brains self-migrate on the next persist after upgrade." This is
a separate body of work and lands in a follow-up commit before 8.0 GA.
- `config.hnswPersistMode` top-level field is unchanged. It's not in
the rename inventory; a future commit can fold it into
`config.vector.persistMode` if desired.
- strictConfig enforcement wiring. The field is accepted by
normalizeConfig() with default 'warn'; the actual warning emission at
knob-mismatch sites lands when the config-resolution layer is touched
for the persisted-path migration.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit (one test fixture updated to use the new
category name; assertion still passes after fixture rename)
- npm run build: clean
The 8.0 PR's user-facing surface is now algorithm-neutral end-to-end:
VectorIndexProvider, config.vector.recall, JsHnswVectorIndex,
saveVectorIndexData, 'vectors' cache category, indexHealth.vector,
strictConfig — all standalone. No legacy 'hnsw' name remains in any
public type or method signature.
Step 5 of the brainy 8.0 rename scaffolding. Three small additions, all
honoring contracts locked in handoff thread BRAINY-8.0-RENAME-COORDINATION:
A. strictConfig: boolean | 'warn' (§ B.6)
src/types/brainy.types.ts
- New BrainyConfig.strictConfig field. Three tiers:
- false: no enforcement (knob mismatches silently accepted)
- 'warn' (8.0 default): one-shot warning per call site listing
ignored knobs + escape-valve recipe + docs link
- true: hard error at init()
- Format matches the 7.30.1 teaching-error pattern (problem → escape
valves → caller location → docs link).
- Applies to ALL provider knob mismatches, not just vector (e.g.
config.aggregation.x with no aggregation provider).
src/brainy.ts
- normalizeConfig() defaults to 'warn'. Wiring into the actual
enforcement sites lands in the final cleanup commit alongside the
removal of legacy knob names.
B. brain.stats().indexHealth.vector (§ planning § 2.6)
src/types/brainy.types.ts
- BrainyStats.indexHealth gains a new `vector: boolean` field. Existing
`hnsw: boolean` field marked @deprecated; same boolean, kept as a
compat alias until the final cleanup.
src/brainy.ts:6272
- Implementation populates both `hnsw` and `vector` from the same
`vectorHealthy` boolean. Refactored to async IIFE so the graph-size
await sits cleanly alongside the new field.
C. wireConnectionsCodec feature-detect (integration doc lines 459-461)
src/brainy.ts:9469-9486
- Added `typeof this.index.setConnectionsCodec === 'function'` guard
before invoking. Native vector-index providers (DiskANN-style) persist
the graph as a single mmap'd file with no per-node connection lists
and have no analogue. Pre-8.0 the wire was unconditional and relied
on the native wrapper exposing a no-op method; 8.0 makes it
feature-detected.
NO-OP scope
No behavioural change in scaffolding. brain.stats() now returns both
field names (callers see the new field but the old still works).
strictConfig is defaulted but not yet enforced. wireConnectionsCodec
behaves identically when called against an impl that has the codec
method (the JS HNSW path); guards cleanly against impls that don't.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
Step 3 of the brainy 8.0 rename scaffolding. The class name now matches
its role: the JS HNSW implementation of the VectorIndexProvider contract.
Per planning § 2.3: this is NOT cosmetic — leaving HNSWIndex when the
public contract is VectorIndexProvider creates a confusing read at the
implementation layer.
CHANGES
Repo-wide mechanical rename: HNSWIndex → JsHnswVectorIndex across 52
references in src/ + 5 references in tests/ via sed. Class declaration,
imports, JSDoc, comments, and identifiers all updated.
src/index.ts
- Primary public export: JsHnswVectorIndex (the new name).
- Backwards-compat alias: `export const HNSWIndex = JsHnswVectorIndex`
with @deprecated note. Removed in the final 8.0 cleanup commit.
NO-OP scope
No behavioural change. The class is exactly the same; only the name
moved. Tests + build green.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
Step 2 of the brainy 8.0 rename scaffolding. Adds the new algorithm-neutral
config surface alongside the legacy config.hnsw path (which becomes a compat
shim removed in the final cleanup commit).
CHANGES
src/utils/recallPreset.ts (NEW)
- DEFAULT_RECALL = 'balanced'
- HNSW_PRESETS table — fast/balanced/accurate → (M, efConstruction, efSearch).
'balanced' equals brainy 7.x DEFAULT_CONFIG byte-for-byte so a 7.x → 8.0
upgrade with no explicit recall value is a no-op.
- resolveRecallHnswKnobs(preset) — preset → knobs.
- resolveJsHnswConfig(vectorConfig) — preset first, then `advanced.hnsw` overrides
win. Explicit knobs always beat the preset.
src/types/brainy.types.ts
- New optional `vector` field on BrainyConfig:
vector?: {
recall?: 'fast' | 'balanced' | 'accurate'
quantization?: { enabled?, bits?: 8 | 4, rerankMultiplier? }
vectorStorage?: 'memory' | 'lazy'
advanced?: {
hnsw?: { M?, efConstruction?, efSearch?, ml? }
diskann?: { pqM?, pqKsub?, maxDegree?, searchListSize?, alpha?, ... }
}
}
- `hnsw` field marked @deprecated with note that it stays as a compat shim
during the 8.0 rename and is removed in the final cleanup commit.
src/utils/unifiedCache.ts
- Cache-category union widened from
'hnsw' | 'metadata' | 'embedding' | 'other'
to
'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other'
- 5 union sites + 4 typed-counter maps (typeAccessCounts, checkFairness
typeSizes/typeCounts/accessRatios/sizeRatios, getStats typeSizes/typeCounts,
evictType loop) all gained the 'vectors' key with initial value 0.
- The hard-coded fairness-check loop now iterates both keys.
- Final 8.0 cleanup will narrow back to 'vectors' | 'metadata' | 'embedding' | 'other'.
src/brainy.ts (normalizeConfig)
- Adds the new `vector` field to the Required<BrainyConfig> return shape so
TS compiles. Reads config?.vector ?? undefined as any (same pattern as
the other optional config fields).
NO-OP scope
No behavioural change. The new config.vector path is silently ignored at
runtime in this commit — wired up in step 4 (storage + index resolution).
Pure surface plumbing. Tests + build green.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
Optimistic concurrency for multi-writer coordination — the read-then-CAS
lock pattern + idempotent-bootstrap singleton inserts. Lands the surface the
SDK scheduler asked for in BRAINY-EXPOSE-TRANSACTIONS, scoped to what ships
cleanly without painting the 8.0 transact() API into a corner.
A. PER-ENTITY _rev FIELD
- src/coreTypes.ts — HNSWNounWithMetadata gains optional `_rev?: number`;
STANDARD_ENTITY_FIELDS includes it so resolveEntityField routes correctly.
- src/types/brainy.types.ts — Entity<T> gains optional `_rev?: number`;
Result<T> mirrors it on the convenience flatten layer.
- src/brainy.ts add() — initializes `_rev: 1` in storageMetadata. New entities
always start at 1.
- src/brainy.ts update() — reads currentRev off the persisted metadata
(falls back to 1 for pre-7.31.0 entities with no _rev), writes
`_rev: currentRev + 1` into the updated metadata. Every successful update()
bumps by exactly 1.
- src/brainy.ts convertNounToEntity + convertMetadataToEntity — both pull
_rev out of the storage metadata and surface it at the top level of Entity.
Pre-7.31.0 entities (no _rev in storage) read as `_rev: 1` so consumers
see a consistent value.
- src/storage/baseStorage.ts — six destructure sites updated to pull _rev
out of the metadata bag so it doesn't leak into customMetadata. Noun-side
returns surface _rev; verb-side destructures correctly but doesn't expose
it on HNSWVerbWithMetadata (verb CAS is future work).
- src/brainy.ts createResult() — flattens entity._rev onto the Result for
backward compat with the existing convenience-field layer.
B. update({ ifRev }) OPTIMISTIC CONCURRENCY
- src/types/brainy.types.ts — UpdateParams<T> gains optional `ifRev?: number`.
- src/transaction/RevisionConflictError.ts (NEW) — carries `{ id, expected,
actual }`. Message names the recipe: refetch with brain.get() and retry
with the latest _rev. Subclass of Error.
- src/brainy.ts update() — when params.ifRev is provided, compares against
currentRev (the rev we just read) and throws RevisionConflictError on
mismatch before any storage write. Omitting ifRev keeps the prior
unconditional-update behavior; existing consumers see no change.
- src/transaction/index.ts — exports RevisionConflictError.
- src/index.ts — public export of RevisionConflictError.
C. add({ ifAbsent }) BY-ID IDEMPOTENT INSERT
- src/types/brainy.types.ts — AddParams<T> gains optional `ifAbsent?: boolean`;
AddManyParams<T> mirrors it as a batch-level flag.
- src/brainy.ts add() — when params.id AND params.ifAbsent, pre-reads
storage.getNounMetadata(id); if present, returns the existing id without
writing. No throw, no overwrite. Ignored when id is omitted (a fresh UUID
can never collide).
- src/brainy.ts addMany() — propagates the batch-level ifAbsent to each
item's add() call. Per-item ifAbsent takes precedence so callers can
override individual rows.
WHAT'S NOT SHIPPED (and why)
A public brain.transaction(fn) wrapper was on the table but was cut. The
internal TransactionManager exposes raw Operation classes (SaveNounMetadata,
etc.) that take StorageAdapter as a constructor argument. A clean high-level
facade in 7.31.0 would have meant either:
(a) Delegate to brain.add() / update() / relate(). Each of those opens its
own internal transaction and commits before the closure returns. Looks
atomic, isn't — a footgun.
(b) Thread an optional `tx?` through every internal write site in
brainy.ts (8 transactionManager.executeTransaction sites). ~2 days of
real refactor with regression surface, and the API shape changes again
in 8.0 anyway.
The SDK scheduler's actual ask (BRAINY-EXPOSE-TRANSACTIONS) is the
read-then-CAS lock pattern. _rev + ifRev solves it completely. Multi-write
atomicity is the 8.0 brain.transact() use case where the Datomic-style
immutable Db makes it atomic by construction — that's the right home.
TESTS
- New tests/integration/rev-and-ifabsent.test.ts (18 tests):
- _rev initialization (1 on add) and surface on get fast/full paths + find
- _rev auto-bump on update across multiple writes
- update({ ifRev }) pass / fail / message format / omitted / legacy-no-rev
- add({ ifAbsent }) writes when absent / no-op when present / id-required
- addMany({ ifAbsent }) propagation + per-item override
- SDK-scheduler scenario: two concurrent CAS updates, one wins one throws
- Unit suite unchanged: 1468/1468.
- All integration subtype + verb + strict + find-limits + new rev suites
pass: 96/96.
DOCS
- New docs/guides/optimistic-concurrency.md (public: true) — full reference:
the lock pattern, read-modify-write retry, idempotent bootstrap, how _rev
interacts with brain.versions, branches/fork, and VFS, what's coming in 8.0.
- docs/api/README.md — add() and update() entries get the new params + tips
pointing at the new guide.
- RELEASES.md v7.31.0 entry.
CORTEX COMPATIBILITY
Zero changes required. _rev is a metadata column already supported by
NativeColumnStore. The auto-bump runs in Brainy JS before any storage call;
Cortex never sees the per-entity counter.
8.0 FORWARD-COMPAT
_rev, ifRev, RevisionConflictError, and ifAbsent survive the 8.0 Db redesign
unchanged. 8.0 layers brain.transact(tx, { ifAtGeneration }) for whole-tx CAS
on top of the same per-entity mechanism — per-entity for single-record
patterns (job locks, idempotent state machines), generation-based for
"did the world move under me." Locked in .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md § C-6.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All integration suites including new rev-and-ifabsent: 96/96
- npm run build: clean
- Closed-source product reference audit: clean
Brainy 7.30.0 introduced a memory-derived synchronous cap on `find({ limit })`
to prevent OOM. The cap was sound in intent but ~4x too conservative in
calibration: assumed 100 KB per result while typical entity footprint is 7-10 KB
(384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB
free-memory box the cap derived to 9000 — breaking common safety-cap patterns
like `find({ type, where, limit: 10_000 })` that typically return 10-500
entities. Surfaced as a runtime regression with cascading 500s degrading
production dashboards.
Three concurrent fixes:
A. RECALIBRATE THE FORMULA
- src/utils/paramValidation.ts:175,196,212 — the three memory-derived priorities
(reservedQueryMemory / containerMemory / freeMemory) all divided by
100 * 1024 * 1024 (100 KB per result, ~10-15x over conservative). Replaced
with a new MAX_LIMIT_KB_PER_RESULT = 25 constant that matches observed
entity size.
- Result: 4 GB container cap goes 10_000 → 40_000; 2 GB cap goes 5_000 →
20_000; 900 MB free-memory cap goes 9_000 → ~36_000. 100k hard ceiling
unchanged. `maxQueryLimit` / `reservedQueryMemory` constructor overrides
unchanged in behavior.
B. TWO-TIER ENFORCEMENT (warn-then-throw)
- Below cap (limit <= maxLimit): silent pass, unchanged.
- Soft tier (maxLimit < limit <= 2 * maxLimit): NEW — one-time warning per
call site (dedup keyed on caller stack frame + limit value), query
proceeds. Pre-7.30.2 code that relied on the cap silently allowing typical
safety-cap limits keeps working; the warning teaches the recipe so consumers
can fix it intentionally.
- Hard tier (limit > 2 * maxLimit): throw with the same teaching message
format. Real OOM territory; the cap stops being a recommendation and becomes
a guardrail.
- The 2x soft margin absorbs typical safety-cap patterns (limit: 10_000
against a 9 K-cap box) without disabling OOM protection. Real OOM territory
on a JS in-memory brain is hundreds of thousands of results, not 10x the
safety cap.
C. IMPROVED ERROR / WARNING MESSAGE
- Same shape as the 7.30.1 enforcement-error messages: state the problem,
name the three escape valves (maxQueryLimit / reservedQueryMemory /
pagination), include caller location, link to docs.
- Extracted findCallerLocation() helper from brainy.ts to a new
src/utils/callerLocation.ts so both the subtype enforcement (7.30.1) and
the limit enforcement (7.30.2) share one implementation without circular
imports.
DOCS
- New docs/guides/find-limits.md (public: true) — full reference: why the cap
exists, the four memory sources the auto-config considers, the three escape
valves with when-to-use-which guidance, and an explicit "pagination is the
future-proof pattern" callout (8.0 may tighten the cap further; pagination
keeps working unchanged).
- docs/api/README.md find() entry gets a one-paragraph `limit` tip + pointer
to the new guide.
- RELEASES.md v7.30.2 entry.
TESTS
- New tests/integration/find-limits.test.ts (9 tests): below-cap silent pass;
soft-tier warns once per call site (dedup verified by exercising same vs.
different source lines via wrapper closures); soft-tier message format
(names all three escape valves + docs link); soft-tier message includes
caller location; hard-tier throws; hard-tier message format same as
soft-tier; consumer maxQueryLimit override raises the cap and shifts both
tiers accordingly; pre-7.30.2 regression scenario explicitly covered.
- tests/unit/utils/memoryLimits.test.ts — 4 tests updated for the recalibrated
cap values (hardcoded expected numbers bumped 4x to match new 25 KB/result
assumption).
- tests/unit/utils/paramValidation.test.ts — auto-limit test extended to cover
the three-tier semantics (below-cap pass / soft-tier silent / hard-tier
throw).
- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-
enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468.
CORTEX COMPATIBILITY
- Zero Cortex changes required. Every change is JS-side: formula recalibration
runs in ValidationConfig.constructor(), two-tier enforcement runs in
validateFindParams(), both fire before any storage / index / Cortex call.
- The new guide notes that Brainy 8.0's Datomic-style Db.find() may tighten
per-call limits to keep snapshot semantics cheap; pagination remains the
pattern that's guaranteed to keep working.
REPO-WIDE CLEANUP
Brainy is the only Soulcraft project that is open source. This commit also
scrubs closed-source product names and product-specific class/field references
from every tracked file in the repo (src/, docs/, tests/, RELEASES.md,
CHANGELOG.md). Consumer-reported bugs, regression scenarios, and release
notes now refer to "a consumer", "a downstream application", "a production
deployment", or "an internal report" — never to the named product. Two
product-named test files renamed to neutral diagnostic names. CLAUDE.md gains
a project-level guard rule documenting the policy and an example list of the
identifiers that may not appear in tracked code.
Verification
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- All four integration subtype + verb + strict + find-limits suites: 78/78
- npm run build: clean
- Closed-source product reference audit: clean
Brainy 7.30 shipped opt-in subtype enforcement; SDK 3.20.0 then registered
SDK_CORE_VOCABULARY on every consumer's brain (Event, Collection, Message,
Contract, Media, Document NounTypes). On 2026-06-08 Venue's /book flow went 500
because their brain.add({ type: NounType.Event, ... }) call sites lacked
subtype. An audit of Brainy's OWN source revealed 14 HIGH-risk internal write
paths that also omit subtype — any consumer running the same vocabulary would
have hit Brainy's infrastructure paths next. 7.30.1 closes both gaps before
8.0 makes strict mode the default.
Additive across the board. Zero behavior change for consumers not using strict
mode. Every change is JS-side — Cortex needs no work for 7.30.1.
NEW — brain.audit() diagnostic
- Read-only method walking storage.getNouns() / getVerbs() pagination
- Returns { entitiesWithoutSubtype: { type: count }, relationshipsWithoutSubtype,
total, scanned, recommendation }
- VFS infrastructure entities excluded by default (they bypass enforcement via
isVFSEntity marker); pass { includeVFS: true } to surface them
- The companion to migrateField (7.x) and fillSubtypes (8.0): tells consumers
exactly what would break under strict enforcement, deterministically
NEW — Improved enforcement error messages
- Caller's source location extracted from Error().stack so users see their own
call site, not a Brainy internal frame
- Specific guidance branches: registered vocabulary → "Pass one of: a, b, c";
brain-wide strict mode → mentions the except clause; otherwise → registration
recipe via brain.requireSubtype()
- Documentation link to the canonical migration recipe
- Same shape for noun and verb enforcement
NEW — CLI --subtype flag
- brainy add and brainy relate gain -s/--subtype <value>
- Defaults to 'cli-add' / 'cli-relate' so the CLI works against strict-mode
brains without the user needing to know the vocabulary in advance
INTERNAL — every Brainy write path now sets subtype
- VFS Contains edges (5 sites at lines 503/905/1694/1772/1886) → 'vfs-contains'
- VFS symlink entity → 'vfs-symlink' (NEW — distinct from 'vfs-file')
- VFS copy-file → preserves source subtype, falls back to 'vfs-file'
- VFS symlink also adopts the isVFSEntity infrastructure marker so it bypasses
enforcement in strict mode
- Aggregation materializer (Measurement entities) → 'materialized-aggregate'
- ImportCoordinator (3 sites): document → 'import-source'; entities →
options.defaultSubtype ?? 'imported'; placeholder → 'import-placeholder'
- SmartImportOrchestrator (4 entity sites + 2 batch relate sites): same
precedence (extractor → options.defaultSubtype → 'imported')
- EntityDeduplicator → candidate.subtype ?? 'imported'
- UniversalImportAPI → extractor → 'extracted' for both entities and relations
- NeuralImport → adds defaultSubtype to NeuralImportOptions; precedence same
- GoogleSheetsIntegration → request body 'subtype' ?? 'imported-from-sheets'
- ODataIntegration → request body 'Subtype' ?? 'imported-from-odata'
- MCP client message storage → 'mcp-message' (also fixes pre-existing missing
data field and missing type by aliasing from the prior text field)
Side-effect fix: storage.getNouns() paginated now surfaces subtype to top-level
- Single-noun getNoun() already did this in 7.30; the paginated path was missed
- Without this fix brain.audit() saw missing subtype on entities that actually
had one (caught by the strict-mode self-test before release)
NEW — tests/integration/strict-mode-self-test.test.ts (13 tests)
- Creates a brain under the exact SDK_CORE_VOCABULARY shape Venue hit + brain-
wide strict mode
- Exercises every internal Brainy path: VFS root + mkdir + writeFile + cp + mv
+ ln + symlink; aggregation engine; audit diagnostic with includeVFS toggle
- Validates error message UX: caller location, vocabulary guidance, brain-wide
strict mode guidance, off-vocabulary value reporting
Docs
- New "Strict mode in practice" section in docs/guides/subtypes-and-facets.md
covering the SDK_CORE_VOCABULARY pattern, 4-step migration recipe
(audit → migrateField → hand-fix → re-audit), the Brainy-internal label
reference table, and an 8.0 forward-look on fillSubtypes()
- docs/api/README.md: new audit() entry, strict-mode tips on add() and relate()
- RELEASES.md: full 7.30.1 entry
Cortex parity (forward-looking, not blocking 7.30.1)
- 6th open question added to .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md: native
fast path for audit() and fillSubtypes() via column-store null-subtype
bitmap for billion-scale brains
- Cortex should add a parity test mirroring strict-mode-self-test.test.ts
against their native paths to catch any latent bug where native writes
bypass JS validation
- Brainy-internal subtype labels become a documented part of the 8.0 contract
(useful for Cortex telemetry surfacing Brainy-managed infrastructure %)
Verification
- npx tsc --noEmit: clean
- npm test: 1468/1468 unit
- 7.29 noun integration suite: 26/26 (no regression)
- 7.30 verb subtype + enforcement integration suite: 30/30 (no regression)
- New strict-mode-self-test integration suite: 13/13
- npm run build: clean
- Closed-source product reference audit: clean
Addresses VE-SUBTYPE-MIGRATION (Venue's reported request) and ships internal
labels Venue did NOT ask for but that would have broken them next under their
own vocabulary registration.
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).
Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
BaseStorage, mirrored after nounCountsByType with the same self-heal
rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path
Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
cardinality + per-NounType breakdown stats. Backed by the aggregation
engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update
Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
streams every entity, copies the value from one path to another, and
(unless readBoth) clears the source. Supports top-level standard fields,
metadata.X, and data.X paths. Idempotent — safe to re-run.
Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
quick-start all treat subtype as a core primitive with anonymous example
vocabularies (employee/customer/invoice/milestone).
Tests
- 26 new integration tests covering write/read/update/delete round-trips,
counts rollup decrement + re-route on mutation, trackField + byField
with and without perType, vocabulary whitelist enforcement, and
migrateField for metadata.X → subtype and data.X → subtype paths
including readBoth deprecation-window semantics.
Unit suite: 1468/1468 passing. Type-check + build clean.
createIndex() now consults the 'diskann' provider before 'hnsw':
1. config.index.type === 'diskann' → require the provider, throw if absent.
2. Auto-engage when ALL of:
- 'diskann' provider registered (cortex plugin loaded)
- storage.getBinaryBlobPath('_diskann/main') returns a local path
(cloud adapters return null → silently stay on HNSW)
- metadataIndex has a stable idMapper
3. config.index.type === 'hnsw' → force the historical in-memory engine.
4. Fall back to the cortex 'hnsw' provider, then brainy's TS HNSWIndex.
migrateToDiskAnn(options): builds the new index in parallel from the
current vector set, samples queries to verify recall ≥ recallTarget
against the old index, atomically swaps if recall passes. Throws and
leaves the old index in place otherwise.
migrateToHnsw(): always-available reverse path. Both preserve
canonical storage (entity JSON, metadata index) — only the search
engine changes.
The orchestration is generic — it works against any registered
'diskann' provider, not tied to any specific implementation. Brainy
without cortex falls back to HNSW; cloud-storage users transparently
stay on HNSW.
Brainy now has a complete reference SQ4 (4-bit per dimension, 8× compression
vs float32) quantization layer in src/utils/vectorQuantization.ts: quantizeSQ4,
dequantizeSQ4, distanceSQ4Js, serializeSQ4, deserializeSQ4 — all byte-for-byte
identical to cortex's Rust quantize_sq4 / dequantize_sq4 / cosine_distance_sq4.
The pack format mirrors cortex exactly:
- Two nibbles per byte. High nibble = vector[2i], low nibble = vector[2i+1].
- Odd-dim vectors place the final value in the high nibble and pad the low
nibble with 0. Packed length = ceil(dim / 2).
- Zero-range vectors (all values identical) map every nibble to 8 (midpoint
of [0, 15]) — both reference impls produce 0x88 bytes.
The active-fn dispatch pattern matches SQ8: distanceSQ4Js is the pure-JS
reference; distanceSQ4 routes through a swappable activeSQ4Distance slot;
setSQ4DistanceImplementation(fn) swaps in cortex's SIMD Rust distance when the
distance:sq4 provider is registered. brainy.ts wires the provider in init,
same shape as the existing SQ8 wiring (~14 LOC). The dispatch path is the
single point of integration so HNSW search code never needs to know whether
it's running on JS or native.
Serialization adds a uint32 dim field after min/max (12-byte header total) —
SQ8 doesn't need it because the packed length equals dim, but SQ4's packed
length rounds up so dim must be recorded explicitly to disambiguate the
trailing pad nibble.
Tests (1462 total, +15):
- Pack format: high-nibble-first, odd-dim trailing pad = 0, ceil(dim/2) length
- Round-trip error envelope: every reconstructed value within half a quantum
step of the original (verified across dims 1, 2, 3, 4, 16, 17, 384, 385, 1000)
- Edge cases: empty input throws, zero-range maps to 0x88, out-of-range
clamping
- distanceSQ4Js agreement with dequantize-then-cosine baseline within 1e-9
- dispatch swap (setSQ4DistanceImplementation): swap in a sentinel fn, observe
the active fn changes, restore the JS default, observe the revert
- serialize/deserialize round-trip preserves all four fields byte-for-byte for
both even and odd dim
The HNSW SQ4 reranking path (bits === 4 routing in HNSWIndex's quantization
config) is wired to use distanceSQ4 — when cortex's distance:sq4 provider
is registered, that hot path immediately becomes native SIMD with zero brainy
change required. Cross-language byte-format parity tests run in the cortex
test suite (paired release brainy 7.28.0 + cortex 2.5.0).
Cortex registers a graph:compression provider exposing `encode`/`decode`
for HNSW connection lists (delta-varint, ~4× smaller edges than the
JSON-UUID-array shape brainy has persisted since the beginning). This
wires the consumer side without changing the saveHNSWData/getHNSWData
adapter signatures.
Architecture:
- NEW ConnectionsCodec (src/hnsw/connectionsCodec.ts) — translates between
UUID-keyed Map<level, Set<UUID>> and a single compact per-node buffer
via the provider + stable EntityIdMapper. Custom wire format:
[count: u8] [[level: u8] [len: u32 LE] [bytes...]]*. One blob per node
regardless of level count, so the storage I/O is identical to the
legacy path (one saveHNSWData call) plus a single saveBinaryBlob.
- HNSWIndex changes — a `connectionsCodec: ConnectionsCodec | null` field
with `setConnectionsCodec()` setter. THREE save sites (deferred flush,
immediate-mode entity persist, immediate-mode neighbor updates) now go
through a single `persistNodeConnections(nodeId, noun)` helper: when the
codec is wired AND the storage adapter exposes saveBinaryBlob, it
encodes the connections, stores them at `_hnsw_conn/<nodeId>`, and
records `connections: {}` in saveHNSWData as the marker. Otherwise the
legacy JSON-array path is taken. TWO load sites use a matching
`restoreNodeConnections` helper that tries loadBinaryBlob first and
falls back to the legacy hnswData.connections field on miss / decode
error. Format convergence is lazy: pre-2.4.0 nodes still load via the
legacy path, then write the compressed form on next dirty save.
- brainy.ts wireConnectionsCodec — runs alongside wireMmapVectorBackend
during init. Activates when (a) the graph:compression provider is
registered and (b) the metadata index exposes its idMapper. Unlike the
mmap-vector backend, this layer engages on EVERY brainy 7.25.0
adapter — the blob primitive itself is universal; only the codec
presence gates activation.
- Provider interface GraphCompressionProvider in plugin.ts — encode +
decode static signatures. Brainy depends on the interface; cortex's
registered { encode: encodeConnections, decode: decodeConnections }
satisfies it structurally.
Tests (1437 total, +4 vs prior tip):
- tests/unit/hnsw/connections-codec.test.ts — 4 unit tests with a JSON
mock provider: single-level round-trip, empty-Map round-trip (one-byte
buffer), multi-level fan-out round-trip, and silent-drop of UUIDs the
decoding idMapper no longer knows. The real cross-language byte
format is exercised when cortex 2.4.0 wires its delta-varint
encode/decode in.
This completes the brainy half of the 2.4.0 storage foundation: stable
ids (#23), mmap vectors (#24), graph link compression (#25), and
column-store interchange (#26). Coordinated release as brainy 7.26.0 +
cortex 2.4.0 follows.
Cortex already registers the vectorStore:mmap provider (its Rust
NativeMmapVectorStore), but brainy has never consumed it — preloadVectors
and getVectorSafe still go straight to storage.getNounVector for every id,
even when an mmap layer is available. This wires the consumer end.
Architecture:
- NEW MmapVectorBackend (src/hnsw/mmapVectorBackend.ts) — bridges brainy's
UUID-keyed vector reads to a int-slot mmap file via the
vectorStore:mmap provider. Slots are addressed by the stable int id
from the post-2.4.0 #1 EntityIdMapper (the foundation this depends on).
Auto-grows the file (doubling) when a write lands beyond capacity, so
HNSWIndex never has to think about sizing. The class never touches
per-entity storage — it owns only the mmap layer.
- HNSWIndex changes — adds a vectorBackend field + a setVectorBackend
setter. The vector read paths (preloadVectors, getVectorSafe) try the
mmap layer first; on a storage fallback hit, they LAZILY write back into
the mmap slot. An upgraded install converges to the zero-copy fast path
under live traffic — no big-bang migration step. The legacy per-entity
path is preserved and still used when no backend is set.
- brainy.ts wiring — a new private wireMmapVectorBackend() runs once
during init, after plugin activation + metadataIndex setup. It activates
the backend only when (a) the vectorStore:mmap provider is registered,
(b) the storage adapter resolves a real local path via
getBinaryBlobPath(), and (c) the metadata index exposes its idMapper.
Cloud adapters return null on (b) and the backend is silently skipped;
HNSWIndex's behaviour is then identical to pre-2.4.0.
- Provider interfaces in plugin.ts — VectorStoreMmapProvider and
VectorStoreMmapInstance document the contract cortex's class fulfils
(the class IS the provider — static factory methods). Brainy depends on
the interfaces, not on cortex; the structural match is verified when
cortex 2.4.0 picks up this brainy release.
Tests (1428 total, +11 vs pre-2.4.0):
- tests/unit/hnsw/mmap-vector-backend.test.ts — 6 unit tests with an
in-memory mock provider. Covers round-trip, batch reads with interleaved
misses, slot stability (no re-slotting on overwrite), file growth without
data loss, idempotent open, and null returns for unwritten slots. The
real perf integration with cortex's NativeMmapVectorStore is exercised
when cortex 2.4.0 wires this in.
- tests/unit/utils/entity-id-mapper-stability.test.ts — moved here from
tests/regression/ (which is NOT in the unit-config include glob, so the
five #23 tests were not actually being run by npm test). The unit
config matches tests/unit/**/*.test.ts.
The 2.4.0 #2 follow-up will be the chunked-segment layout for remote
storage adapters (S3 / R2 / GCS) where a single growing file doesn't fit
immutable objects. For 2.4.0 release: local-FS only.
Adds a swappable sort:topK seam for the find() result-ranking hot path. Brainy
ranks candidate results by relevance score then slices to offset+limit; for large
candidate sets that is an O(N log N) full sort even though only the page is kept.
New utils/resultRanking.ts exposes rankIndicesByScore (top-K index selection) with a
JS default (sortTopKIndicesJs) that is byte-identical in ordering to the previous
results.sort((a, b) => b.score - a.score) + slice: descending by score, stable ties
by original index (ES2019+ stable sort semantics). setSortTopKImplementation lets a
native provider (e.g. cortex's Rust partial-sort) replace it; the provider returns
indices into a scores array and only has to match the JS index ordering.
brainy.ts resolves the sort:topK provider in init() (same pattern as distance:sq8)
and wires both relevance-score sort sites in find() through rankIndicesByScore +
reorderByIndices. rankIndicesByScore validates a native provider's output (length,
range, no duplicates) and falls back to JS on any inconsistency, so a query never
returns wrong or duplicated results. No provider registered = unchanged JS behavior.
Tests: unit coverage of the dispatcher (JS ordering vs Array.sort across 200 random
trials incl. ties, provider routing, and fallback on bad/throwing providers) plus
find() integration proving ranking routes through a registered provider, that the
provider and JS fallback produce identical ids/scores on the same data, and that
pagination requests k = offset + limit with descending ordering.
Makes distanceSQ8 swappable via setSQ8DistanceImplementation and wires brainy.ts to
install a registered 'distance:sq8' provider (e.g. cortex's Rust SIMD), with the JS
distanceSQ8Js as the default fallback. The provider signature is byte-compatible with
the native cosineDistanceSq8, so HNSW SQ8 reranking transparently uses native distance
when cortex is loaded. Native==JS numeric equivalence is asserted by the cross-language
parity suite.
New report APIs:
- brain.queryAggregate(name, params) returns the clean AggregateResult[] shape
({ groupKey, metrics, count }) directly, instead of the search-Result wrapper that
find({ aggregate }) returns.
- HAVING: find({ aggregate, having }) and queryAggregate filter groups by computed
metric values (e.g. revenue > 1000), complementing where (which filters group keys).
Evaluated per group: O(groups), independent of entity count.
Fixes (all reproducible on Node; surfaced under Cortex+Bun by Memory):
- Aggregate backfill-on-define: defining an aggregate over a store that already holds
matching entities returned []. It now backfills from existing entities on first query
(storage-agnostic via getNouns), so it works under durable backends that reopen
pre-populated. groupBy:['noun'] resolves to the entity type; find({ aggregate }) rows
expose groupKey/metrics/count at the top level.
- Multi-hop find({ connected: { depth, via } }) honors depth and via at every hop
(was 1-hop only, with verb filtering applied to hop 1 only); BFS bounded by limit.
- extractEntities no longer bleeds a neighbour's type indicator across candidates; each
candidate is typed by its own span. Also fixes the "Dr." title pattern.
Adds real-embedding regression tests; full unit suite green.
Three advanced-API correctness fixes, all reproducible on Node (not Bun-specific):
- Entity/concept extraction returned []. SmartExtractor combined agreeing signals
with a weighted sum compared against an absolute 0.60 gate, so a confident
low-weight signal lost selection to a mediocre high-weight one that then failed
the gate, dropping the whole result. Select and gate on a normalized weighted
average instead. The public `confidence` option now controls the threshold (was
a dead hardcoded 0.60), and the embedding-signal timeout is raised 100ms -> 2000ms
so the neural signal is not silently dropped on slower runtimes.
- Multi-hop find({ connected }) returned only the 1-hop neighbour. executeGraphSearch
ignored depth/via; it now delegates to the depth-aware neighbors() BFS.
- find({ aggregate }) hid groupKey/metrics/count under .metadata, so callers
expecting AggregateResult saw empty rows. Expose those fields at the top level.
Adds real-embedding regression tests in tests/integration/advanced-apis-regression.test.ts.
Per the Cortex team's audit, the BR-CX-INTERFACE-GAP boot crash was a
build/install artifact (stale node_modules, lockfile drift, Docker cache,
bundler quirks), not a plugin-bundles-brainy version-skew issue. Cortex's
MmapFileSystemStorage extends FileSystemStorage and inherits all 6
multi-process helpers via the prototype chain at runtime; the dynamic
ESM import to @soulcraft/brainy is preserved in cortex's dist.
- Rewrote the hasStorageMethod() doc comment to credit the real cause.
- Updated the init-time warning to point operators at the actual fix:
clean install or container rebuild to refresh node_modules.
- New docs/concepts/storage-adapters.md documenting the inheritance
contract: extend FileSystemStorage to inherit the multi-process
helpers, override supportsMultiProcessLocking() -> true to activate.
Includes a minimum checklist for adapter authors.
- New docs/architecture/multiprocess-storage-mixin.md as a future-
direction note: extracting the 7 methods into a
MultiProcessSafeStorage interface/mixin is the architecturally clean
next step, deferred to v8 or until a second multi-process capability
lands.
No behavior changes. Ships with the next release.
Two production correctness defects fixed together so consumers can land
one upgrade.
BR-FIND-WHERE-ZERO — find() returned [] and stats() reported 0 entities
for any workspace whose data was written after the 7.20.0 column-store
refactor. Root cause: getStats() and the getIds() fallback still read
from the deleted sparse-index path. Separately, BaseStorage.getNounType()
was hardcoded to return 'thing', poisoning type-statistics.json with
every noun attributed to that bucket.
- MetadataIndex.getStats() reads from ColumnStore + idMapper.
- MetadataIndex.getIdsFromChunks() throws BrainyError(FIELD_NOT_INDEXED)
when neither store has the field. getIdsForFilter() catches per
clause, logs once, returns [].
- BaseStorage.nounTypeByIdCache populated in saveNounMetadata_internal
and consumed in saveNoun_internal. flushCounts() now persists the
Uint32Array counters too, so readers see fresh per-type counts.
- Self-heal at init: loadTypeStatistics() auto-rebuilds when the
poisoned signature is detected. rebuildTypeCounts() is now public for
use from `brainy inspect repair`.
- Dead state removed: dirtyChunks, dirtySparseIndices,
flushDirtyMetadata().
BR-DEFENSIVE-INTERFACE — 7.21.0 called supportsMultiProcessLocking()
unconditionally, crashing on older Cortex storage adapters that predate
the method. New hasStorageMethod(name) helper gates every new-method
call site. Older adapter triggers a one-line warning at init pointing
at the recommended plugin version.
Tests:
- new tests/integration/find-where-zero.test.ts (7 cases)
- new tests/integration/cortex-compat.test.ts (5 cases)
- multi-process-safety.test.ts updated to enforce correct counts
- 1329/1329 unit + 24/24 new integration tests passing
Filesystem storage now enforces single-writer, many-reader semantics.
A second writer on the same data directory throws at init time with the
holder's PID, hostname, and heartbeat — replacing the previous silent
stale-reads failure mode.
- New: `Brainy.openReadOnly()` — coexists with a live writer, every
mutation throws clearly.
- New: writer lock at `<rootDir>/locks/_writer.lock` with 10s heartbeat
and stale-detection (PID liveness + heartbeat freshness).
- New: cross-process flush-request RPC (filesystem-based, no signals)
so inspectors can force fresh state on demand.
- New: `brain.stats()`, `brain.explain(findParams)`, `brain.health()`
for operator-facing introspection.
- New: `brainy inspect` CLI with 13 subcommands (stats, find, get,
relations, explain, health, sample, fields, dump, watch, backup,
repair, diff), all read-only by default.
- Same-PID re-opens allowed with a warning (preserves test "simulate
restart" patterns).
- Storage instances passed directly via `storage: new MemoryStorage()`
are now honoured instead of silently falling through to the
filesystem auto-detect path.
Brainy + Cortex compose under this model — the lock covers both because
they share `rootDir`, Cortex segments are immutable mmap files, and
MANIFEST updates use atomic-rename.
Adds a per-field sorted column store (Lucene doc values + roaring bitmap
architecture) that replaces the MetadataIndex sparse index internals for
both filtering and sorting. One system for all field types with exact
precision — no bucketing, no per-entity storage reads.
Column store: binary .cidx segment format, in-memory tail buffers,
LSM-style compaction, k-way merge sort, multi-value support (__words__).
All queries (filter, range, sort, filtered sort) route through the column
store when data is available, falling back to sparse index otherwise.
Key unlocks:
- find({ orderBy: 'createdAt' }) works WITHOUT a filter (previously threw)
- find({ orderBy: 'metadata.price' }) works for custom numeric fields
- Exact timestamp precision (no 1-minute bucketing)
- O(K log S) sort independent of total entity count
New files: src/indexes/columnStore/ (types, format, tail buffer, cursor,
manifest, coordinator — ~700 lines). 101 new unit tests covering binary
format round-trips, CRC validation, sort, filter, range, deletion,
multi-segment merge, persistence, and multi-value (words) fields.
Deleted: metadataIndex-automatic-bucketing.test.ts (bucketing behavior
eliminated by exact-precision column store). Sparse index write path
removed from addToIndex/removeFromIndex. Sparse index legacy code still
present as dead code pending cleanup in next commit.
Muse reported that find({ orderBy: 'createdAt' }) silently returned the
wrong order: getFieldValueForEntity read noun.metadata[field] for
timestamp fields, but getNoun() destructures standard fields to the top
level, so the read always returned undefined and the sort collapsed to
insertion order.
The fix introduces a single source of truth for reading fields off an
entity. STANDARD_ENTITY_FIELDS + resolveEntityField() in coreTypes.ts
encode the "standard fields top-level, custom fields nested" contract
in one place. getFieldValueForEntity now uses the helper and routes
through a named BUCKETED_INDEX_FIELDS set instead of a hardcoded
timestamp if-chain — filtered sort on createdAt/updatedAt now works.
Unfiltered orderBy is explicitly rejected with a clear error pointing
callers at the right pattern. A scalable, unfiltered-sort-capable
time-ordered segment index is tracked as a separate follow-up.
Root cause: metadata index failed to reconstruct after deploy restart
because the field registry file (__metadata_field_registry__) was lost
during an interrupted flush. init() silently assumed the workspace was
empty even though 5000+ entities existed on disk.
Fix 1 (root cause): init() now probes storage for entities when field
registry is missing. If entities exist, triggers rebuild instead of
silently skipping. Never trusts a missing registry as "empty."
Fix 2 (safety net): after rebuildIndexesIfNeeded(), verifies metadata
index entry count matches storage entity count. Forces second rebuild
if mismatch detected.
Fix 3 (prevention): flush() now always saves field registry and
EntityIdMapper, even when no dirty fields exist. These tiny files are
the critical link that init() needs to discover persisted indices.
Fix 5 (safe rebuild): rebuild() no longer deletes the field registry
file before rewriting. If rebuild fails partway, the registry survives
for the next init() to discover and re-trigger rebuild.
Fix 6 (collision guard): EntityIdMapper init() warns when mapper file
is missing but entities exist on disk, preventing silent ID collisions
from nextId starting at 1.
commit() previously defaulted captureState to false, creating commits
with NULL_HASH tree that could never be restored from. Now:
- flush() runs first to persist deferred HNSW nodes, count batches,
and index state before snapshotting
- captureState defaults to true, creating real content-addressed trees
- captureState: false still available for lightweight metadata-only commits
addMany() now temporarily switches the HNSW index to deferred persist
mode during the batch loop. Previously, each add() triggered ~16-20
saveHNSWData calls for modified neighbors (each a read→gzip→atomic-write
cycle). For 450 items this was ~8,100 individual storage writes.
With deferred mode, dirty node IDs are collected in a Set during the
batch and flushed once at the end — deduplicating repeated neighbor
updates. A node modified by insert #3 and again by insert #47 is
written only once.
Also adds setPersistMode() to TypeAwareHNSWIndex, propagating mode
changes to all existing type-specific sub-indexes.
