feat: mmap-vector backend wiring — HNSWIndex consumes vectorStore:mmap (2.4.0 #2)

Cortex already registers the vectorStore:mmap provider (its Rust
NativeMmapVectorStore), but brainy has never consumed it — preloadVectors
and getVectorSafe still go straight to storage.getNounVector for every id,
even when an mmap layer is available. This wires the consumer end.

Architecture:

- NEW MmapVectorBackend (src/hnsw/mmapVectorBackend.ts) — bridges brainy's
  UUID-keyed vector reads to a int-slot mmap file via the
  vectorStore:mmap provider. Slots are addressed by the stable int id
  from the post-2.4.0 #1 EntityIdMapper (the foundation this depends on).
  Auto-grows the file (doubling) when a write lands beyond capacity, so
  HNSWIndex never has to think about sizing. The class never touches
  per-entity storage — it owns only the mmap layer.

- HNSWIndex changes — adds a vectorBackend field + a setVectorBackend
  setter. The vector read paths (preloadVectors, getVectorSafe) try the
  mmap layer first; on a storage fallback hit, they LAZILY write back into
  the mmap slot. An upgraded install converges to the zero-copy fast path
  under live traffic — no big-bang migration step. The legacy per-entity
  path is preserved and still used when no backend is set.

- brainy.ts wiring — a new private wireMmapVectorBackend() runs once
  during init, after plugin activation + metadataIndex setup. It activates
  the backend only when (a) the vectorStore:mmap provider is registered,
  (b) the storage adapter resolves a real local path via
  getBinaryBlobPath(), and (c) the metadata index exposes its idMapper.
  Cloud adapters return null on (b) and the backend is silently skipped;
  HNSWIndex's behaviour is then identical to pre-2.4.0.

- Provider interfaces in plugin.ts — VectorStoreMmapProvider and
  VectorStoreMmapInstance document the contract cortex's class fulfils
  (the class IS the provider — static factory methods). Brainy depends on
  the interfaces, not on cortex; the structural match is verified when
  cortex 2.4.0 picks up this brainy release.

Tests (1428 total, +11 vs pre-2.4.0):

- tests/unit/hnsw/mmap-vector-backend.test.ts — 6 unit tests with an
  in-memory mock provider. Covers round-trip, batch reads with interleaved
  misses, slot stability (no re-slotting on overwrite), file growth without
  data loss, idempotent open, and null returns for unwritten slots. The
  real perf integration with cortex's NativeMmapVectorStore is exercised
  when cortex 2.4.0 wires this in.

- tests/unit/utils/entity-id-mapper-stability.test.ts — moved here from
  tests/regression/ (which is NOT in the unit-config include glob, so the
  five #23 tests were not actually being run by npm test). The unit
  config matches tests/unit/**/*.test.ts.

The 2.4.0 #2 follow-up will be the chunked-segment layout for remote
storage adapters (S3 / R2 / GCS) where a single growing file doesn't fit
immutable objects. For 2.4.0 release: local-FS only.
This commit is contained in:
David Snelling 2026-05-28 10:10:05 -07:00
parent b2408cb5a6
commit d4cb26c604
6 changed files with 640 additions and 4 deletions

View file

@ -37,7 +37,8 @@ import { setGlobalCache } from './utils/unifiedCache.js'
import type { UnifiedCache } from './utils/unifiedCache.js'
import { rankIndicesByScore, reorderByIndices } from './utils/resultRanking.js'
import { PluginRegistry } from './plugin.js'
import type { BrainyPlugin, BrainyPluginContext } from './plugin.js'
import type { BrainyPlugin, BrainyPluginContext, VectorStoreMmapProvider } from './plugin.js'
import { MmapVectorBackend } from './hnsw/mmapVectorBackend.js'
import { TransactionManager } from './transaction/TransactionManager.js'
import {
ValidationConfig,
@ -559,6 +560,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
this.graphIndex = graphIndex
}
// Wire the mmap-vector backend (2.4.0 #2). When the vectorStore:mmap
// provider is registered AND the storage adapter resolves a real local
// path via getBinaryBlobPath(), open the mmap file there and inject the
// backend into HNSWIndex. Cloud adapters return null and skip silently;
// HNSWIndex's behaviour is then identical to pre-2.4.0 (per-entity reads
// from canonical storage). Failures are non-fatal — the index still
// works, just without the zero-copy fast path.
await this.wireMmapVectorBackend()
// Rebuild indexes if needed for existing data
await this.rebuildIndexesIfNeeded()
@ -7716,6 +7726,72 @@ export class Brainy<T = any> implements BrainyInterface<T> {
await this.lazyRebuildPromise
}
/**
* @description Open the mmap-vector backend and inject it into HNSWIndex.
* Called once during init after plugin activation, metadataIndex init, and
* graphIndex setup so the idMapper is available and providers are wired.
*
* Activation conditions (ALL must hold; otherwise this is a no-op and the
* index falls back to per-entity storage reads, the pre-2.4.0 behaviour):
* 1. The `vectorStore:mmap` provider is registered (cortex registers this).
* 2. The storage adapter resolves a real local path via `getBinaryBlobPath()`
* cloud adapters return null and the mmap layer is skipped silently.
* 3. The metadata index exposes its idMapper (a stable UUIDint map). The
* mmap layout is keyed by these ints, so 2.4.0 #1 is the prerequisite.
*
* Vector dimensionality: read from `this.dimensions` if set; otherwise the
* default for the standard embedding model (384). The dim is fixed at
* file-creation time and cannot change later, so a wrong default would only
* matter on the very first init of a brand-new instance. After the first
* `add()` the in-memory `this.dimensions` is set, and any subsequent init
* (or a re-open) sees the correct value.
*
* Failures here are non-fatal: a debug-level log records the reason, and
* HNSWIndex continues without the mmap fast path.
*/
private async wireMmapVectorBackend(): Promise<void> {
const provider = this.pluginRegistry.getProvider<VectorStoreMmapProvider>('vectorStore:mmap')
if (!provider) return
const storageWithBlob = this.storage as unknown as {
getBinaryBlobPath?: (key: string) => string | null
}
const vectorPath = storageWithBlob.getBinaryBlobPath?.('_vectors/main') ?? null
if (!vectorPath) return
const idMapper = this.metadataIndex.getIdMapper?.()
if (!idMapper) return
const dim = this.dimensions ?? 384
const initialCapacity = Math.max(idMapper.size * 2, 1024)
try {
const backend = await MmapVectorBackend.open(
provider,
vectorPath,
dim,
initialCapacity,
idMapper
)
this.index.setVectorBackend(backend)
if (!this.config.silent) {
console.log(
`[brainy] vector mmap backend wired (path=${vectorPath}, dim=${dim}, capacity=${initialCapacity})`
)
}
} catch (error) {
// Common cases: dim mismatch from a prior init at a different dimension,
// permission errors, disk full, file corruption. Index keeps working via
// the per-entity storage path; flag the cause for diagnosis.
if (!this.config.silent) {
console.warn(
`[brainy] mmap-vector backend not wired (${(error as Error).message}); ` +
`falling back to per-entity vector reads`
)
}
}
}
/**
* Rebuild indexes from persisted data if needed (LAZY LOADING)
*