Cortex already registers the vectorStore:mmap provider (its Rust
NativeMmapVectorStore), but brainy has never consumed it — preloadVectors
and getVectorSafe still go straight to storage.getNounVector for every id,
even when an mmap layer is available. This wires the consumer end.
Architecture:
- NEW MmapVectorBackend (src/hnsw/mmapVectorBackend.ts) — bridges brainy's
UUID-keyed vector reads to a int-slot mmap file via the
vectorStore:mmap provider. Slots are addressed by the stable int id
from the post-2.4.0 #1 EntityIdMapper (the foundation this depends on).
Auto-grows the file (doubling) when a write lands beyond capacity, so
HNSWIndex never has to think about sizing. The class never touches
per-entity storage — it owns only the mmap layer.
- HNSWIndex changes — adds a vectorBackend field + a setVectorBackend
setter. The vector read paths (preloadVectors, getVectorSafe) try the
mmap layer first; on a storage fallback hit, they LAZILY write back into
the mmap slot. An upgraded install converges to the zero-copy fast path
under live traffic — no big-bang migration step. The legacy per-entity
path is preserved and still used when no backend is set.
- brainy.ts wiring — a new private wireMmapVectorBackend() runs once
during init, after plugin activation + metadataIndex setup. It activates
the backend only when (a) the vectorStore:mmap provider is registered,
(b) the storage adapter resolves a real local path via
getBinaryBlobPath(), and (c) the metadata index exposes its idMapper.
Cloud adapters return null on (b) and the backend is silently skipped;
HNSWIndex's behaviour is then identical to pre-2.4.0.
- Provider interfaces in plugin.ts — VectorStoreMmapProvider and
VectorStoreMmapInstance document the contract cortex's class fulfils
(the class IS the provider — static factory methods). Brainy depends on
the interfaces, not on cortex; the structural match is verified when
cortex 2.4.0 picks up this brainy release.
Tests (1428 total, +11 vs pre-2.4.0):
- tests/unit/hnsw/mmap-vector-backend.test.ts — 6 unit tests with an
in-memory mock provider. Covers round-trip, batch reads with interleaved
misses, slot stability (no re-slotting on overwrite), file growth without
data loss, idempotent open, and null returns for unwritten slots. The
real perf integration with cortex's NativeMmapVectorStore is exercised
when cortex 2.4.0 wires this in.
- tests/unit/utils/entity-id-mapper-stability.test.ts — moved here from
tests/regression/ (which is NOT in the unit-config include glob, so the
five #23 tests were not actually being run by npm test). The unit
config matches tests/unit/**/*.test.ts.
The 2.4.0 #2 follow-up will be the chunked-segment layout for remote
storage adapters (S3 / R2 / GCS) where a single growing file doesn't fit
immutable objects. For 2.4.0 release: local-FS only.
Previously metadataIndex.rebuild() called idMapper.clear() which reset nextId
to 1 and renumbered every UUID by re-insertion order. Any consumer that had
persisted int-keyed data against the old map was silently invalidated — and
2.4.0's vector mmap store (#20), graph link compression (#21), and column-store
JS↔native interchange all need persisted int indices that survive a rebuild.
Remove the unconditional clear() in rebuild(). The rebuild already re-iterates
every entity via idMapper.getOrAssign(uuid), which returns the existing int
unchanged for known UUIDs. Stale UUID→int entries for entities no longer in
storage persist as harmless memory overhead; a dedicated prune step can be
added if it ever matters.
clearAllIndexData() — the explicit nuclear recovery path — keeps its existing
idMapper.clear() call (renumbering is intentional there), and now logs a
prodLog.warn making it explicit that any persisted int-keyed data is invalidated
and must be rebuilt from canonical sources.
Strengthened the EntityIdMapper class JSDoc to document the stability guarantee
as a contract — append-only getOrAssign, monotonic nextId, remove() leaves
permanent holes, rebuild() never renumbers, only clear() does.
Added tests/regression/entity-id-mapper-stability.test.ts pinning down the
five-point contract: (1) single-rebuild stability; (2) many-rebuild stability;
(3) post-rebuild adds get fresh monotonic ints; (4) removes leave permanent
holes — new entities never recycle; (5) clearAllIndexData() explicitly
renumbers (the documented destructive path).
Foundation for 2.4.0 #2-#4. Full test suite (62 files, 1417 tests) green.
