Step 3 of the brainy 8.0 rename scaffolding. The class name now matches
its role: the JS HNSW implementation of the VectorIndexProvider contract.
Per planning § 2.3: this is NOT cosmetic — leaving HNSWIndex when the
public contract is VectorIndexProvider creates a confusing read at the
implementation layer.
CHANGES
Repo-wide mechanical rename: HNSWIndex → JsHnswVectorIndex across 52
references in src/ + 5 references in tests/ via sed. Class declaration,
imports, JSDoc, comments, and identifiers all updated.
src/index.ts
- Primary public export: JsHnswVectorIndex (the new name).
- Backwards-compat alias: `export const HNSWIndex = JsHnswVectorIndex`
with @deprecated note. Removed in the final 8.0 cleanup commit.
NO-OP scope
No behavioural change. The class is exactly the same; only the name
moved. Tests + build green.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
Cortex registers a graph:compression provider exposing `encode`/`decode`
for HNSW connection lists (delta-varint, ~4× smaller edges than the
JSON-UUID-array shape brainy has persisted since the beginning). This
wires the consumer side without changing the saveHNSWData/getHNSWData
adapter signatures.
Architecture:
- NEW ConnectionsCodec (src/hnsw/connectionsCodec.ts) — translates between
UUID-keyed Map<level, Set<UUID>> and a single compact per-node buffer
via the provider + stable EntityIdMapper. Custom wire format:
[count: u8] [[level: u8] [len: u32 LE] [bytes...]]*. One blob per node
regardless of level count, so the storage I/O is identical to the
legacy path (one saveHNSWData call) plus a single saveBinaryBlob.
- HNSWIndex changes — a `connectionsCodec: ConnectionsCodec | null` field
with `setConnectionsCodec()` setter. THREE save sites (deferred flush,
immediate-mode entity persist, immediate-mode neighbor updates) now go
through a single `persistNodeConnections(nodeId, noun)` helper: when the
codec is wired AND the storage adapter exposes saveBinaryBlob, it
encodes the connections, stores them at `_hnsw_conn/<nodeId>`, and
records `connections: {}` in saveHNSWData as the marker. Otherwise the
legacy JSON-array path is taken. TWO load sites use a matching
`restoreNodeConnections` helper that tries loadBinaryBlob first and
falls back to the legacy hnswData.connections field on miss / decode
error. Format convergence is lazy: pre-2.4.0 nodes still load via the
legacy path, then write the compressed form on next dirty save.
- brainy.ts wireConnectionsCodec — runs alongside wireMmapVectorBackend
during init. Activates when (a) the graph:compression provider is
registered and (b) the metadata index exposes its idMapper. Unlike the
mmap-vector backend, this layer engages on EVERY brainy 7.25.0
adapter — the blob primitive itself is universal; only the codec
presence gates activation.
- Provider interface GraphCompressionProvider in plugin.ts — encode +
decode static signatures. Brainy depends on the interface; cortex's
registered { encode: encodeConnections, decode: decodeConnections }
satisfies it structurally.
Tests (1437 total, +4 vs prior tip):
- tests/unit/hnsw/connections-codec.test.ts — 4 unit tests with a JSON
mock provider: single-level round-trip, empty-Map round-trip (one-byte
buffer), multi-level fan-out round-trip, and silent-drop of UUIDs the
decoding idMapper no longer knows. The real cross-language byte
format is exercised when cortex 2.4.0 wires its delta-varint
encode/decode in.
This completes the brainy half of the 2.4.0 storage foundation: stable
ids (#23), mmap vectors (#24), graph link compression (#25), and
column-store interchange (#26). Coordinated release as brainy 7.26.0 +
cortex 2.4.0 follows.
Cortex already registers the vectorStore:mmap provider (its Rust
NativeMmapVectorStore), but brainy has never consumed it — preloadVectors
and getVectorSafe still go straight to storage.getNounVector for every id,
even when an mmap layer is available. This wires the consumer end.
Architecture:
- NEW MmapVectorBackend (src/hnsw/mmapVectorBackend.ts) — bridges brainy's
UUID-keyed vector reads to a int-slot mmap file via the
vectorStore:mmap provider. Slots are addressed by the stable int id
from the post-2.4.0 #1 EntityIdMapper (the foundation this depends on).
Auto-grows the file (doubling) when a write lands beyond capacity, so
HNSWIndex never has to think about sizing. The class never touches
per-entity storage — it owns only the mmap layer.
- HNSWIndex changes — adds a vectorBackend field + a setVectorBackend
setter. The vector read paths (preloadVectors, getVectorSafe) try the
mmap layer first; on a storage fallback hit, they LAZILY write back into
the mmap slot. An upgraded install converges to the zero-copy fast path
under live traffic — no big-bang migration step. The legacy per-entity
path is preserved and still used when no backend is set.
- brainy.ts wiring — a new private wireMmapVectorBackend() runs once
during init, after plugin activation + metadataIndex setup. It activates
the backend only when (a) the vectorStore:mmap provider is registered,
(b) the storage adapter resolves a real local path via
getBinaryBlobPath(), and (c) the metadata index exposes its idMapper.
Cloud adapters return null on (b) and the backend is silently skipped;
HNSWIndex's behaviour is then identical to pre-2.4.0.
- Provider interfaces in plugin.ts — VectorStoreMmapProvider and
VectorStoreMmapInstance document the contract cortex's class fulfils
(the class IS the provider — static factory methods). Brainy depends on
the interfaces, not on cortex; the structural match is verified when
cortex 2.4.0 picks up this brainy release.
Tests (1428 total, +11 vs pre-2.4.0):
- tests/unit/hnsw/mmap-vector-backend.test.ts — 6 unit tests with an
in-memory mock provider. Covers round-trip, batch reads with interleaved
misses, slot stability (no re-slotting on overwrite), file growth without
data loss, idempotent open, and null returns for unwritten slots. The
real perf integration with cortex's NativeMmapVectorStore is exercised
when cortex 2.4.0 wires this in.
- tests/unit/utils/entity-id-mapper-stability.test.ts — moved here from
tests/regression/ (which is NOT in the unit-config include glob, so the
five #23 tests were not actually being run by npm test). The unit
config matches tests/unit/**/*.test.ts.
The 2.4.0 #2 follow-up will be the chunked-segment layout for remote
storage adapters (S3 / R2 / GCS) where a single growing file doesn't fit
immutable objects. For 2.4.0 release: local-FS only.
Makes distanceSQ8 swappable via setSQ8DistanceImplementation and wires brainy.ts to
install a registered 'distance:sq8' provider (e.g. cortex's Rust SIMD), with the JS
distanceSQ8Js as the default fallback. The provider signature is byte-compatible with
the native cosineDistanceSq8, so HNSW SQ8 reranking transparently uses native distance
when cortex is loaded. Native==JS numeric equivalence is asserted by the cross-language
parity suite.
- Store data opaquely in add() and update() instead of spreading object
properties into top-level metadata. data is for semantic search (HNSW),
metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
Shutdown/close/flush now properly flushes all 4 components in parallel:
metadataIndex, graphIndex, HNSW dirty nodes, and storage counts. Previously
only counts were flushed, causing native provider data loss on restart.
Also:
- Wire roaring, msgpack, entityIdMapper provider consumption from plugins
- Fix allOf filter O(n²) intersection → O(n) Set-based
- Fix ne/exists negation filter to use Set-based exclusion
- Add setMsgpackImplementation() swap in SSTable for native msgpack
- Add setRoaringImplementation() swap for native CRoaring bitmaps
- Add getAllIntIds() to EntityIdMapper for bitmap operations
- Remove TypeAwareHNSWIndex from default index creation path
- Export memory detection utilities from internals
- Clean up 26 permanently-skipped dead tests
