Previously metadataIndex.rebuild() called idMapper.clear() which reset nextId
to 1 and renumbered every UUID by re-insertion order. Any consumer that had
persisted int-keyed data against the old map was silently invalidated — and
2.4.0's vector mmap store (#20), graph link compression (#21), and column-store
JS↔native interchange all need persisted int indices that survive a rebuild.
Remove the unconditional clear() in rebuild(). The rebuild already re-iterates
every entity via idMapper.getOrAssign(uuid), which returns the existing int
unchanged for known UUIDs. Stale UUID→int entries for entities no longer in
storage persist as harmless memory overhead; a dedicated prune step can be
added if it ever matters.
clearAllIndexData() — the explicit nuclear recovery path — keeps its existing
idMapper.clear() call (renumbering is intentional there), and now logs a
prodLog.warn making it explicit that any persisted int-keyed data is invalidated
and must be rebuilt from canonical sources.
Strengthened the EntityIdMapper class JSDoc to document the stability guarantee
as a contract — append-only getOrAssign, monotonic nextId, remove() leaves
permanent holes, rebuild() never renumbers, only clear() does.
Added tests/regression/entity-id-mapper-stability.test.ts pinning down the
five-point contract: (1) single-rebuild stability; (2) many-rebuild stability;
(3) post-rebuild adds get fresh monotonic ints; (4) removes leave permanent
holes — new entities never recycle; (5) clearAllIndexData() explicitly
renumbers (the documented destructive path).
Foundation for 2.4.0 #2-#4. Full test suite (62 files, 1417 tests) green.
The native SQ8 distance hook renamed the original distanceSQ8 to distanceSQ8Js
and added a dispatcher in its place, but left the old function's doc block
orphaned above distanceSQ8Js (two consecutive comments, the first dangling).
Merge them into one accurate block on distanceSQ8Js with dash-notation params.
Native accelerators (cortex) register providers for metadataIndex, graphIndex,
hnsw, entityIdMapper, cache, columnStore, and aggregation. Until now the exact
method/property surface brainy calls on each was implicit — a provider could
drop a member brainy depends on and only fail at runtime when that path ran.
This defines and exports the provider contracts from the stable
@soulcraft/brainy/plugin entrypoint:
- MetadataIndexProvider, GraphIndexProvider, HnswProvider,
EntityIdMapperProvider, CacheProvider — each typed as exactly the surface
brainy calls (optional/feature-detected members like HNSW setPersistMode and
enableCOW are intentionally excluded).
- Re-exports ColumnStoreProvider and AggregationProvider (and the aggregate
types) from the same entrypoint so a plugin author can import the whole
provider surface from one place.
Brainy's own baseline classes now `implements` these contracts
(MetadataIndexManager, GraphAdjacencyIndex, HNSWIndex, EntityIdMapper,
UnifiedCache), so the interfaces can never silently diverge from what brainy
ships — and any provider that declares `implements` gets a compile error the
moment brainy starts requiring a new member.
The embeddings/embedBatch providers stay typed by the existing EmbeddingFunction
(no duplicate interface added). Type-only changes; no runtime behavior change.
Adds a swappable sort:topK seam for the find() result-ranking hot path. Brainy
ranks candidate results by relevance score then slices to offset+limit; for large
candidate sets that is an O(N log N) full sort even though only the page is kept.
New utils/resultRanking.ts exposes rankIndicesByScore (top-K index selection) with a
JS default (sortTopKIndicesJs) that is byte-identical in ordering to the previous
results.sort((a, b) => b.score - a.score) + slice: descending by score, stable ties
by original index (ES2019+ stable sort semantics). setSortTopKImplementation lets a
native provider (e.g. cortex's Rust partial-sort) replace it; the provider returns
indices into a scores array and only has to match the JS index ordering.
brainy.ts resolves the sort:topK provider in init() (same pattern as distance:sq8)
and wires both relevance-score sort sites in find() through rankIndicesByScore +
reorderByIndices. rankIndicesByScore validates a native provider's output (length,
range, no duplicates) and falls back to JS on any inconsistency, so a query never
returns wrong or duplicated results. No provider registered = unchanged JS behavior.
Tests: unit coverage of the dispatcher (JS ordering vs Array.sort across 200 random
trials incl. ties, provider routing, and fallback on bad/throwing providers) plus
find() integration proving ranking routes through a registered provider, that the
provider and JS fallback produce identical ids/scores on the same data, and that
pagination requests k = offset + limit with descending ordering.
Makes distanceSQ8 swappable via setSQ8DistanceImplementation and wires brainy.ts to
install a registered 'distance:sq8' provider (e.g. cortex's Rust SIMD), with the JS
distanceSQ8Js as the default fallback. The provider signature is byte-compatible with
the native cosineDistanceSq8, so HNSW SQ8 reranking transparently uses native distance
when cortex is loaded. Native==JS numeric equivalence is asserted by the cross-language
parity suite.
Replaces localeCompare / raw relational operators with compareCodePoints (UTF-8
byte order) in the remaining persisted or native-facing comparison sites:
- SSTable sourceId sort + zone-map range check + binary search (graph LSM)
- COW TreeObject + RefManager name sorts (reproducible content hashes and ref
listings across environments)
- getSortedIdsForFilter (numeric-aware; code-point for strings) so the JS sort
fallback matches the native column store exactly
Deterministic across OS/Node/ICU and byte-identical to the native engine. No
migration: COW serialize/deserialize preserve stored order, so existing trees
keep their hashes; only new writes adopt code-point order.
Introduce a first-class binary-blob storage primitive on the StorageAdapter
contract and implement it across all storage backends. This stores opaque byte
payloads verbatim instead of base64-in-JSON, eliminating the ~33% inflation and
full-materialization cost of the JSON envelope. It unblocks zero-copy,
mmap-able column-store segments and batch vector I/O at billion scale.
New methods (declared abstract on BaseStorageAdapter, the class that implements
StorageAdapter, and added to the StorageAdapter interface):
saveBinaryBlob(key, data) raw write, atomic on real filesystems
loadBinaryBlob(key) exact bytes, or null if absent
deleteBinaryBlob(key) idempotent (missing is ignored)
getBinaryBlobPath(key) real local fs path where one exists, else null
Shared key -> location convention across every adapter: the key's
"/"-separated segments nest under a `_blobs/` prefix and are suffixed with
`.bin`, e.g. "graph-lsm/source/sstable-123" ->
"<root>/_blobs/graph-lsm/source/sstable-123.bin". Blobs are not branch-scoped
(COW): they are immutable producer-managed segments.
Per-adapter behavior:
- FileSystemStorage: writes under <rootDir>/_blobs via tmp+rename; returns the
real on-disk path so native code can mmap it directly. Path convention matches
the existing MmapFileSystemStorage subclass byte-for-byte.
- S3CompatibleStorage / R2Storage / GcsStorage / AzureBlobStorage: put/get/delete
raw octet-stream objects; getBinaryBlobPath returns null (remote stores have no
local path).
- MemoryStorage: defensive-copied Map<string, Buffer>; null path; cleared on
clear().
- OPFSStorage: stores raw bytes in the OPFS tree; null path.
- HistoricalStorageAdapter: read-only — save/delete throw; load resolves the
blob from the historical commit tree; null path.
Tests: tests/unit/storage/binaryBlob.test.ts exercises save/load round-trip
(byte-identical, incl. non-UTF8 bytes), overwrite, delete-then-load, load-missing,
and getBinaryBlobPath behavior for all eight adapters. Cloud adapters run against
in-memory client fakes that drive the real adapter code; OPFS runs against an
in-memory FileSystem Access API mock; the historical adapter commits a blob into
a real COW tree. 59 new tests; full unit suite (1398 tests) green.
Replace localeCompare (default-locale, non-deterministic across environments —
unsafe for a persisted sorted index) with UTF-8 byte / code-point order via a
shared compareCodePoints() helper. String ordering is now deterministic and
byte-identical to the native column store / aggregation sort, so results are the
same with or without the native accelerator. Covers the aggregate orderBy sort
and all 5 column-store comparison sites (tail buffer, merge sort, binary search).
Adds compareCodePoints unit tests.
Add 'percentile' (with a 'p' fraction in [0,1]) and 'distinctCount' to the
aggregation engine. Both are exact, computed from a per-metric value multiset
(MetricState.valueCounts) maintained incrementally and delete-safe; percentile
uses numpy-linear interpolation. The multiset is JSON-serializable so results
survive persistence. 35 aggregation unit tests pass.
- groupBy now supports { field, unnest: true }: an entity contributes once per
distinct element of an array field (tag frequency / faceted counts). Duplicate
elements on one entity count once; an empty/missing array joins no group. The
incremental add/update/delete paths fan out across the unnested groups.
- extractEntities/extractConcepts batch-embed the unique candidate spans in one
embedBatch call instead of one embed() per candidate (N sequential model calls);
falls back to per-candidate embedding if the batch fails. No behavior change.
New report APIs:
- brain.queryAggregate(name, params) returns the clean AggregateResult[] shape
({ groupKey, metrics, count }) directly, instead of the search-Result wrapper that
find({ aggregate }) returns.
- HAVING: find({ aggregate, having }) and queryAggregate filter groups by computed
metric values (e.g. revenue > 1000), complementing where (which filters group keys).
Evaluated per group: O(groups), independent of entity count.
Fixes (all reproducible on Node; surfaced under Cortex+Bun by Memory):
- Aggregate backfill-on-define: defining an aggregate over a store that already holds
matching entities returned []. It now backfills from existing entities on first query
(storage-agnostic via getNouns), so it works under durable backends that reopen
pre-populated. groupBy:['noun'] resolves to the entity type; find({ aggregate }) rows
expose groupKey/metrics/count at the top level.
- Multi-hop find({ connected: { depth, via } }) honors depth and via at every hop
(was 1-hop only, with verb filtering applied to hop 1 only); BFS bounded by limit.
- extractEntities no longer bleeds a neighbour's type indicator across candidates; each
candidate is typed by its own span. Also fixes the "Dr." title pattern.
Adds real-embedding regression tests; full unit suite green.
A lightweight tag is skipped by 'git push --follow-tags', so v$VERSION stayed
local-only and 'gh release create' failed at the end of the release. Use an
annotated tag (-a) so it pushes with the release commit.
Three advanced-API correctness fixes, all reproducible on Node (not Bun-specific):
- Entity/concept extraction returned []. SmartExtractor combined agreeing signals
with a weighted sum compared against an absolute 0.60 gate, so a confident
low-weight signal lost selection to a mediocre high-weight one that then failed
the gate, dropping the whole result. Select and gate on a normalized weighted
average instead. The public `confidence` option now controls the threshold (was
a dead hardcoded 0.60), and the embedding-signal timeout is raised 100ms -> 2000ms
so the neural signal is not silently dropped on slower runtimes.
- Multi-hop find({ connected }) returned only the 1-hop neighbour. executeGraphSearch
ignored depth/via; it now delegates to the depth-aware neighbors() BFS.
- find({ aggregate }) hid groupKey/metrics/count under .metadata, so callers
expecting AggregateResult saw empty rows. Expose those fields at the top level.
Adds real-embedding regression tests in tests/integration/advanced-apis-regression.test.ts.
Per the Cortex team's audit, the BR-CX-INTERFACE-GAP boot crash was a
build/install artifact (stale node_modules, lockfile drift, Docker cache,
bundler quirks), not a plugin-bundles-brainy version-skew issue. Cortex's
MmapFileSystemStorage extends FileSystemStorage and inherits all 6
multi-process helpers via the prototype chain at runtime; the dynamic
ESM import to @soulcraft/brainy is preserved in cortex's dist.
- Rewrote the hasStorageMethod() doc comment to credit the real cause.
- Updated the init-time warning to point operators at the actual fix:
clean install or container rebuild to refresh node_modules.
- New docs/concepts/storage-adapters.md documenting the inheritance
contract: extend FileSystemStorage to inherit the multi-process
helpers, override supportsMultiProcessLocking() -> true to activate.
Includes a minimum checklist for adapter authors.
- New docs/architecture/multiprocess-storage-mixin.md as a future-
direction note: extracting the 7 methods into a
MultiProcessSafeStorage interface/mixin is the architecturally clean
next step, deferred to v8 or until a second multi-process capability
lands.
No behavior changes. Ships with the next release.
Two production correctness defects fixed together so consumers can land
one upgrade.
BR-FIND-WHERE-ZERO — find() returned [] and stats() reported 0 entities
for any workspace whose data was written after the 7.20.0 column-store
refactor. Root cause: getStats() and the getIds() fallback still read
from the deleted sparse-index path. Separately, BaseStorage.getNounType()
was hardcoded to return 'thing', poisoning type-statistics.json with
every noun attributed to that bucket.
- MetadataIndex.getStats() reads from ColumnStore + idMapper.
- MetadataIndex.getIdsFromChunks() throws BrainyError(FIELD_NOT_INDEXED)
when neither store has the field. getIdsForFilter() catches per
clause, logs once, returns [].
- BaseStorage.nounTypeByIdCache populated in saveNounMetadata_internal
and consumed in saveNoun_internal. flushCounts() now persists the
Uint32Array counters too, so readers see fresh per-type counts.
- Self-heal at init: loadTypeStatistics() auto-rebuilds when the
poisoned signature is detected. rebuildTypeCounts() is now public for
use from `brainy inspect repair`.
- Dead state removed: dirtyChunks, dirtySparseIndices,
flushDirtyMetadata().
BR-DEFENSIVE-INTERFACE — 7.21.0 called supportsMultiProcessLocking()
unconditionally, crashing on older Cortex storage adapters that predate
the method. New hasStorageMethod(name) helper gates every new-method
call site. Older adapter triggers a one-line warning at init pointing
at the recommended plugin version.
Tests:
- new tests/integration/find-where-zero.test.ts (7 cases)
- new tests/integration/cortex-compat.test.ts (5 cases)
- multi-process-safety.test.ts updated to enforce correct counts
- 1329/1329 unit + 24/24 new integration tests passing
Filesystem storage now enforces single-writer, many-reader semantics.
A second writer on the same data directory throws at init time with the
holder's PID, hostname, and heartbeat — replacing the previous silent
stale-reads failure mode.
- New: `Brainy.openReadOnly()` — coexists with a live writer, every
mutation throws clearly.
- New: writer lock at `<rootDir>/locks/_writer.lock` with 10s heartbeat
and stale-detection (PID liveness + heartbeat freshness).
- New: cross-process flush-request RPC (filesystem-based, no signals)
so inspectors can force fresh state on demand.
- New: `brain.stats()`, `brain.explain(findParams)`, `brain.health()`
for operator-facing introspection.
- New: `brainy inspect` CLI with 13 subcommands (stats, find, get,
relations, explain, health, sample, fields, dump, watch, backup,
repair, diff), all read-only by default.
- Same-PID re-opens allowed with a warning (preserves test "simulate
restart" patterns).
- Storage instances passed directly via `storage: new MemoryStorage()`
are now honoured instead of silently falling through to the
filesystem auto-detect path.
Brainy + Cortex compose under this model — the lock covers both because
they share `rootDir`, Cortex segments are immutable mmap files, and
MANIFEST updates use atomic-rename.
Column store handles all writes and primary queries. Sparse index write
methods are unreachable:
Deleted: addToChunkedIndex (~103 lines), removeFromChunkedIndex (~37 lines),
splitChunkDeferred (~82 lines), getValueChunkFilename + makeSafeFilename
(~20 lines). Total: ~240 lines of dead write-path code.
Sparse index read methods kept as migration fallback for existing workspaces.
Will be deleted once lazy migration (buildFromStorage) ships.
Adds a per-field sorted column store (Lucene doc values + roaring bitmap
architecture) that replaces the MetadataIndex sparse index internals for
both filtering and sorting. One system for all field types with exact
precision — no bucketing, no per-entity storage reads.
Column store: binary .cidx segment format, in-memory tail buffers,
LSM-style compaction, k-way merge sort, multi-value support (__words__).
All queries (filter, range, sort, filtered sort) route through the column
store when data is available, falling back to sparse index otherwise.
Key unlocks:
- find({ orderBy: 'createdAt' }) works WITHOUT a filter (previously threw)
- find({ orderBy: 'metadata.price' }) works for custom numeric fields
- Exact timestamp precision (no 1-minute bucketing)
- O(K log S) sort independent of total entity count
New files: src/indexes/columnStore/ (types, format, tail buffer, cursor,
manifest, coordinator — ~700 lines). 101 new unit tests covering binary
format round-trips, CRC validation, sort, filter, range, deletion,
multi-segment merge, persistence, and multi-value (words) fields.
Deleted: metadataIndex-automatic-bucketing.test.ts (bucketing behavior
eliminated by exact-precision column store). Sparse index write path
removed from addToIndex/removeFromIndex. Sparse index legacy code still
present as dead code pending cleanup in next commit.
Preventive cleanup of sites that previously used dual-lookup patterns
(metadata[field] ?? entity[field]) or hardcoded timestamp if-chains
to read fields off HNSWNounWithMetadata. These worked today but were
fragile — the same failure mode that caused the orderBy sort bug
would have recurred the next time someone reordered or dropped a
lookup.
- AggregationIndex.computeGroupKey + getNumericField now route all
dimension and metric field lookups through resolveEntityField.
- improvedNeuralAPI._getItemsByTimeWindow + _clusterItemsByField
use resolveEntityField as the primary path, with item.data as a
legacy producer fallback. The type/nounType legacy branch is
preserved as-is since it handles storage format compatibility
rather than shape contract drift.
No behavior change for correct inputs. All aggregation and orderby
regression tests pass unchanged.
Cortex and other first-party plugins need a shared contract for reading
fields off HNSWNounWithMetadata. Exporting from /internals keeps a single
source of truth and prevents the shape contract from drifting between
packages.
Muse reported that find({ orderBy: 'createdAt' }) silently returned the
wrong order: getFieldValueForEntity read noun.metadata[field] for
timestamp fields, but getNoun() destructures standard fields to the top
level, so the read always returned undefined and the sort collapsed to
insertion order.
The fix introduces a single source of truth for reading fields off an
entity. STANDARD_ENTITY_FIELDS + resolveEntityField() in coreTypes.ts
encode the "standard fields top-level, custom fields nested" contract
in one place. getFieldValueForEntity now uses the helper and routes
through a named BUCKETED_INDEX_FIELDS set instead of a hardcoded
timestamp if-chain — filtered sort on createdAt/updatedAt now works.
Unfiltered orderBy is explicitly rejected with a clear error pointing
callers at the right pattern. A scalable, unfiltered-sort-capable
time-ordered segment index is tracked as a separate follow-up.
Root cause: metadata index failed to reconstruct after deploy restart
because the field registry file (__metadata_field_registry__) was lost
during an interrupted flush. init() silently assumed the workspace was
empty even though 5000+ entities existed on disk.
Fix 1 (root cause): init() now probes storage for entities when field
registry is missing. If entities exist, triggers rebuild instead of
silently skipping. Never trusts a missing registry as "empty."
Fix 2 (safety net): after rebuildIndexesIfNeeded(), verifies metadata
index entry count matches storage entity count. Forces second rebuild
if mismatch detected.
Fix 3 (prevention): flush() now always saves field registry and
EntityIdMapper, even when no dirty fields exist. These tiny files are
the critical link that init() needs to discover persisted indices.
Fix 5 (safe rebuild): rebuild() no longer deletes the field registry
file before rewriting. If rebuild fails partway, the registry survives
for the next init() to discover and re-trigger rebuild.
Fix 6 (collision guard): EntityIdMapper init() warns when mapper file
is missing but entities exist on disk, preventing silent ID collisions
from nextId starting at 1.
commit() previously defaulted captureState to false, creating commits
with NULL_HASH tree that could never be restored from. Now:
- flush() runs first to persist deferred HNSW nodes, count batches,
and index state before snapshotting
- captureState defaults to true, creating real content-addressed trees
- captureState: false still available for lightweight metadata-only commits
UnifiedCache.setMaxSize() allows runtime resizing of the cache maximum.
When the new limit is smaller than current usage, lowest-value items are
evicted until the cache fits. Used by Cortex's ResourceManager to
rebalance cache vs instance memory as brainy instances are created and
evicted in multi-tenant deployments.
Also adds getMaxSize() for observability.
EmbeddingManager constructor logged 'Using Q8 precision (WASM)' before
plugins had a chance to register a native embedder. Deferred logging to
init() so the startup message reflects the actual embedding engine.
addMany() now temporarily switches the HNSW index to deferred persist
mode during the batch loop. Previously, each add() triggered ~16-20
saveHNSWData calls for modified neighbors (each a read→gzip→atomic-write
cycle). For 450 items this was ~8,100 individual storage writes.
With deferred mode, dirty node IDs are collected in a Set during the
batch and flushed once at the end — deduplicating repeated neighbor
updates. A node modified by insert #3 and again by insert #47 is
written only once.
Also adds setPersistMode() to TypeAwareHNSWIndex, propagating mode
changes to all existing type-specific sub-indexes.
The storage config type only exposed { type, options, branch } but plugin
storage factories (e.g. Cortex mmap) read rootDirectory from the top level
of the config object. This caused undefined storage paths when Cortex was
active. The underlying StorageOptions interface already declares
rootDirectory — this aligns BrainyConfig.storage to match.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SSTable.calculateChecksum() used require('crypto') which breaks in ESM
environments and Bun. Replace with top-level import { createHash } from
'node:crypto' — SSTable is Node-only (LSM-tree filesystem storage) so
a direct node:crypto import is appropriate.
Code blocks with Unicode box-drawing characters render poorly in web
doc viewers — font alignment breaks and the styled container creates
a visual box-in-a-box. Replaced with a bullet list (before) and a
bold statement (after) that render cleanly at any size and theme.
Replace flat 15-row table with a Before/After ASCII diagram plus a
capability grid (Search/Graph/Filter/VFS/Branch/Import) with competitors
grouped by category. Add new What Can You Build? section with generic
use cases and Built with Brainy showcase. Tighten header copy.
