feat: mmap-vector backend wiring — HNSWIndex consumes vectorStore:mmap (2.4.0 #2)

Cortex already registers the vectorStore:mmap provider (its Rust
NativeMmapVectorStore), but brainy has never consumed it — preloadVectors
and getVectorSafe still go straight to storage.getNounVector for every id,
even when an mmap layer is available. This wires the consumer end.

Architecture:

- NEW MmapVectorBackend (src/hnsw/mmapVectorBackend.ts) — bridges brainy's
  UUID-keyed vector reads to a int-slot mmap file via the
  vectorStore:mmap provider. Slots are addressed by the stable int id
  from the post-2.4.0 #1 EntityIdMapper (the foundation this depends on).
  Auto-grows the file (doubling) when a write lands beyond capacity, so
  HNSWIndex never has to think about sizing. The class never touches
  per-entity storage — it owns only the mmap layer.

- HNSWIndex changes — adds a vectorBackend field + a setVectorBackend
  setter. The vector read paths (preloadVectors, getVectorSafe) try the
  mmap layer first; on a storage fallback hit, they LAZILY write back into
  the mmap slot. An upgraded install converges to the zero-copy fast path
  under live traffic — no big-bang migration step. The legacy per-entity
  path is preserved and still used when no backend is set.

- brainy.ts wiring — a new private wireMmapVectorBackend() runs once
  during init, after plugin activation + metadataIndex setup. It activates
  the backend only when (a) the vectorStore:mmap provider is registered,
  (b) the storage adapter resolves a real local path via
  getBinaryBlobPath(), and (c) the metadata index exposes its idMapper.
  Cloud adapters return null on (b) and the backend is silently skipped;
  HNSWIndex's behaviour is then identical to pre-2.4.0.

- Provider interfaces in plugin.ts — VectorStoreMmapProvider and
  VectorStoreMmapInstance document the contract cortex's class fulfils
  (the class IS the provider — static factory methods). Brainy depends on
  the interfaces, not on cortex; the structural match is verified when
  cortex 2.4.0 picks up this brainy release.

Tests (1428 total, +11 vs pre-2.4.0):

- tests/unit/hnsw/mmap-vector-backend.test.ts — 6 unit tests with an
  in-memory mock provider. Covers round-trip, batch reads with interleaved
  misses, slot stability (no re-slotting on overwrite), file growth without
  data loss, idempotent open, and null returns for unwritten slots. The
  real perf integration with cortex's NativeMmapVectorStore is exercised
  when cortex 2.4.0 wires this in.

- tests/unit/utils/entity-id-mapper-stability.test.ts — moved here from
  tests/regression/ (which is NOT in the unit-config include glob, so the
  five #23 tests were not actually being run by npm test). The unit
  config matches tests/unit/**/*.test.ts.

The 2.4.0 #2 follow-up will be the chunked-segment layout for remote
storage adapters (S3 / R2 / GCS) where a single growing file doesn't fit
immutable objects. For 2.4.0 release: local-FS only.

tests/unit/hnsw/mmap-vector-backend.test.ts

@@ -0,0 +1,242 @@
+/**
+ * @module hnsw/mmapVectorBackend.test
+ * @description Unit tests for the `MmapVectorBackend` bridge — the brainy-side
+ * wrapper that translates UUID-keyed vector reads/writes into stable int slot
+ * ops against an `vectorStore:mmap` provider.
+ *
+ * Mocks the provider so the tests run without cortex installed (cortex is a
+ * downstream consumer of brainy, not a dev dep). The real integration with
+ * cortex's `NativeMmapVectorStore` is exercised when cortex 2.4.0 picks up
+ * this brainy release and re-runs its cross-language parity suite.
+ *
+ * Coverage:
+ * 1. Open-then-write-then-read round-trips for a single vector.
+ * 2. Batch reads return an array aligned with the input UUIDs, with `null`
+ *    entries for misses interleaved among hits — order preserved.
+ * 3. Slot assignment is stable across multiple writes for the same UUID
+ *    (no re-slot, no overwrite of an adjacent slot).
+ * 4. Writes beyond the initial capacity grow the file (doubling) without
+ *    losing the vectors already written.
+ * 5. `readByUuid` returns `null` for both unknown UUIDs and UUIDs in the map
+ *    whose slot has not yet been written.
+ * 6. Open is idempotent — opening an already-existing file reuses it.
+ */
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest'
+import { tmpdir } from 'node:os'
+import { mkdtemp, rm } from 'node:fs/promises'
+import { join } from 'node:path'
+import { MmapVectorBackend } from '../../../src/hnsw/mmapVectorBackend.js'
+import { EntityIdMapper } from '../../../src/utils/entityIdMapper.js'
+import type {
+  VectorStoreMmapInstance,
+  VectorStoreMmapProvider
+} from '../../../src/plugin.js'
+
+/**
+ * Minimal storage stub for the EntityIdMapper. The mapper only touches storage
+ * in init/flush; for these tests the mapper starts empty and is never flushed.
+ */
+const stubStorage = {
+  getMetadata: async () => undefined,
+  saveMetadata: async () => {},
+  getNouns: async () => ({ totalCount: 0, items: [] })
+} as any
+
+/**
+ * Pure in-memory mmap store. Mirrors cortex's NativeMmapVectorStore surface
+ * just closely enough to exercise the backend's contract — no real mmap, no
+ * file I/O, no f32 round-trip narrowing (the precision check is out of scope
+ * here; that's covered by cortex's parity suite).
+ */
+class MockMmapStore implements VectorStoreMmapInstance {
+  private readonly vectors: Array<number[] | undefined> = []
+  private highestWritten = -1
+  constructor(
+    public readonly dim: number,
+    private _capacity: number
+  ) {}
+  get count(): number {
+    return this.highestWritten + 1
+  }
+  get capacity(): number {
+    return this._capacity
+  }
+  writeVector(index: number, vector: number[]): void {
+    if (index >= this._capacity) {
+      throw new Error(`Slot ${index} >= capacity ${this._capacity}`)
+    }
+    if (vector.length !== this.dim) {
+      throw new Error(`Dim mismatch: expected ${this.dim}, got ${vector.length}`)
+    }
+    this.vectors[index] = [...vector]
+    if (index > this.highestWritten) this.highestWritten = index
+  }
+  writeVectorsBatch(startIndex: number, vectorsFlat: number[]): number {
+    if (vectorsFlat.length % this.dim !== 0) {
+      throw new Error('vectorsFlat length not a multiple of dim')
+    }
+    const n = vectorsFlat.length / this.dim
+    for (let i = 0; i < n; i++) {
+      this.writeVector(startIndex + i, vectorsFlat.slice(i * this.dim, (i + 1) * this.dim))
+    }
+    return n
+  }
+  readVector(index: number): number[] {
+    const v = this.vectors[index]
+    if (!v) throw new Error(`Slot ${index} not written`)
+    return [...v]
+  }
+  readVectorsBatch(indices: number[]): number[] {
+    const flat: number[] = []
+    for (const i of indices) {
+      const v = this.vectors[i]
+      if (!v) throw new Error(`Slot ${i} not written`)
+      for (let k = 0; k < this.dim; k++) flat.push(v[k])
+    }
+    return flat
+  }
+  prefetch(_indices: number[]): void {
+    /* no-op in mock */
+  }
+  resize(newCapacity: number): void {
+    if (newCapacity < this._capacity) throw new Error('Cannot shrink')
+    this._capacity = newCapacity
+  }
+  flush(): void {
+    /* no-op in mock */
+  }
+}
+
+/**
+ * Mock provider — keeps one MockMmapStore per path. open() throws if the path
+ * doesn't exist yet (matches cortex semantics: open() is for existing files
+ * only); create() throws if it does (cortex doesn't, but the brainy backend
+ * does open-first-then-create, so the throw path is exercised).
+ */
+class MockMmapProvider implements VectorStoreMmapProvider {
+  private files = new Map<string, MockMmapStore>()
+  create(path: string, dim: number, capacity: number): VectorStoreMmapInstance {
+    if (this.files.has(path)) throw new Error(`File exists at ${path}`)
+    const store = new MockMmapStore(dim, capacity)
+    this.files.set(path, store)
+    return store
+  }
+  open(path: string): VectorStoreMmapInstance {
+    const store = this.files.get(path)
+    if (!store) throw new Error(`No file at ${path}`)
+    return store
+  }
+  openReadOnly(path: string): VectorStoreMmapInstance {
+    return this.open(path)
+  }
+  /** Test helper — peek at the underlying store. */
+  _peek(path: string): MockMmapStore | undefined {
+    return this.files.get(path)
+  }
+}
+
+describe('MmapVectorBackend (2.4.0 #2 — wraps vectorStore:mmap provider)', () => {
+  let dir: string
+  let path: string
+  let idMapper: EntityIdMapper
+  let provider: MockMmapProvider
+
+  beforeEach(async () => {
+    dir = await mkdtemp(join(tmpdir(), 'brainy-mmap-vec-'))
+    path = join(dir, 'vectors.bin')
+    idMapper = new EntityIdMapper({ storage: stubStorage })
+    await idMapper.init()
+    provider = new MockMmapProvider()
+  })
+
+  afterEach(async () => {
+    await rm(dir, { recursive: true, force: true }).catch(() => {})
+  })
+
+  it('open creates a new file when none exists, then round-trips a vector by UUID', async () => {
+    const backend = await MmapVectorBackend.open(provider, path, 4, 16, idMapper)
+    expect(backend.dim).toBe(4)
+
+    backend.writeByUuid('alpha', [1, 2, 3, 4])
+    expect(backend.readByUuid('alpha')).toEqual([1, 2, 3, 4])
+  })
+
+  it('reads return null for unknown UUIDs and for UUIDs in the map but not yet written', async () => {
+    const backend = await MmapVectorBackend.open(provider, path, 2, 8, idMapper)
+
+    // Unknown — never seen by the mapper.
+    expect(backend.readByUuid('ghost')).toBeNull()
+
+    // Known to the mapper but no slot ever written. We assign through the
+    // mapper directly so the backend itself has not touched the slot.
+    idMapper.getOrAssign('reserved')
+    expect(backend.readByUuid('reserved')).toBeNull()
+  })
+
+  it('batch read returns an array aligned to the input UUIDs (nulls preserved in place)', async () => {
+    const backend = await MmapVectorBackend.open(provider, path, 3, 16, idMapper)
+    backend.writeByUuid('a', [1, 1, 1])
+    backend.writeByUuid('b', [2, 2, 2])
+    backend.writeByUuid('c', [3, 3, 3])
+
+    // Interleave hits + a never-seen UUID + a duplicate hit. Order preserved.
+    const result = backend.readBatchByUuid(['b', 'missing', 'a', 'c', 'missing-too', 'b'])
+    expect(result).toEqual([
+      [2, 2, 2],
+      null,
+      [1, 1, 1],
+      [3, 3, 3],
+      null,
+      [2, 2, 2]
+    ])
+  })
+
+  it('writes for the same UUID land in the same slot (stable id, no re-slotting)', async () => {
+    const backend = await MmapVectorBackend.open(provider, path, 2, 8, idMapper)
+    backend.writeByUuid('persistent', [1, 1])
+    const slotAfterFirstWrite = idMapper.getInt('persistent')
+
+    backend.writeByUuid('persistent', [9, 9]) // overwrite the same slot
+    expect(idMapper.getInt('persistent')).toBe(slotAfterFirstWrite)
+    expect(backend.readByUuid('persistent')).toEqual([9, 9])
+
+    // Another UUID gets a different slot, and the first vector is undisturbed.
+    backend.writeByUuid('other', [5, 5])
+    expect(idMapper.getInt('other')).not.toBe(slotAfterFirstWrite)
+    expect(backend.readByUuid('persistent')).toEqual([9, 9])
+    expect(backend.readByUuid('other')).toEqual([5, 5])
+  })
+
+  it('grows the file (doubling) when a write lands beyond capacity, without losing prior data', async () => {
+    // Start at the smallest sane initial capacity (clamped to 16 by the backend).
+    const backend = await MmapVectorBackend.open(provider, path, 2, 1, idMapper)
+    const store = provider._peek(path)!
+    expect(store.capacity).toBe(16) // backend floor
+
+    // Write 20 vectors so capacity must double at least once (16 → 32).
+    const uuids: string[] = []
+    for (let i = 0; i < 20; i++) {
+      const uuid = `u-${i}`
+      uuids.push(uuid)
+      backend.writeByUuid(uuid, [i, i * 2])
+    }
+    expect(store.capacity).toBeGreaterThanOrEqual(32)
+
+    // Every vector survived the growth — no slot got overwritten or lost.
+    for (let i = 0; i < uuids.length; i++) {
+      expect(backend.readByUuid(uuids[i])).toEqual([i, i * 2])
+    }
+  })
+
+  it('opens an existing file idempotently when the second open hits the same path', async () => {
+    // First open creates.
+    const backend1 = await MmapVectorBackend.open(provider, path, 2, 8, idMapper)
+    backend1.writeByUuid('persisted', [7, 7])
+
+    // Second open against the same path. The provider's create() throws on
+    // collision; the backend's open-first behaviour means we reuse the file.
+    const backend2 = await MmapVectorBackend.open(provider, path, 2, 8, idMapper)
+    expect(backend2.readByUuid('persisted')).toEqual([7, 7])
+  })
+})

tests/unit/utils/entity-id-mapper-stability.test.ts

@@ -0,0 +1,161 @@
+/**
+ * Regression test: EntityIdMapper stability across rebuild.
+ *
+ * The foundation 2.4.0 (vector mmap store, graph link compression, column-store
+ * JS↔native interchange) all key off UUID→int mappings that **must not change**
+ * across a metadata-index rebuild. Previously `metadataIndex.rebuild()` called
+ * `idMapper.clear()` which reset `nextId` to 1 and renumbered every UUID by
+ * re-insertion order, silently invalidating any consumer that had persisted
+ * int-keyed data against the old map.
+ *
+ * This test pins down the stability contract:
+ *
+ * 1. UUID→int mappings persist across a single rebuild.
+ * 2. Mappings persist across many consecutive rebuilds.
+ * 3. New entities added after rebuild get fresh ints greater than any prior
+ *    assignment — no collisions with existing UUIDs' ints.
+ * 4. Removed entities leave a permanent hole — new entities don't recycle the
+ *    gap, even across a rebuild.
+ * 5. `clearAllIndexData()` is the explicit, intentional nuclear path — it DOES
+ *    renumber. This is the only documented way to invalidate the int space, and
+ *    a warning is logged so consumers know persisted int-keyed data is now stale.
+ */
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest'
+import { Brainy } from '../../../src/brainy.js'
+
+const DIM = 384
+const makeVec = (seed = 1) =>
+  new Float32Array(DIM).map((_, i) => ((i + seed) % DIM) / DIM)
+
+describe('EntityIdMapper stability (foundation for 2.4.0)', () => {
+  let brain: Brainy
+
+  beforeEach(async () => {
+    brain = new Brainy({ storage: { type: 'memory' }, silent: true })
+    await brain.init()
+  })
+
+  afterEach(async () => {
+    await brain.close()
+  })
+
+  async function addEntity(name: string, seed: number): Promise<string> {
+    return brain.add({
+      data: name,
+      vector: makeVec(seed),
+      type: 'thing' as any,
+      metadata: { name }
+    })
+  }
+
+  function getInt(uuid: string): number | undefined {
+    return (brain as any).metadataIndex.idMapper.getInt(uuid)
+  }
+
+  async function rebuild(): Promise<void> {
+    await (brain as any).metadataIndex.rebuild()
+  }
+
+  it('UUID→int mappings persist across a single metadata-index rebuild', async () => {
+    const ids = [
+      await addEntity('a', 1),
+      await addEntity('b', 2),
+      await addEntity('c', 3),
+      await addEntity('d', 4),
+      await addEntity('e', 5)
+    ]
+    const before = ids.map(id => getInt(id))
+    expect(before.every(i => typeof i === 'number' && (i as number) > 0)).toBe(true)
+
+    await rebuild()
+
+    const after = ids.map(id => getInt(id))
+    expect(after).toEqual(before)
+  })
+
+  it('mappings stay byte-for-byte stable across many consecutive rebuilds', async () => {
+    const ids = [
+      await addEntity('a', 1),
+      await addEntity('b', 2),
+      await addEntity('c', 3)
+    ]
+    const before = ids.map(id => getInt(id))
+
+    for (let i = 0; i < 5; i++) {
+      await rebuild()
+      const after = ids.map(id => getInt(id))
+      expect(after).toEqual(before)
+    }
+  })
+
+  it('entities added after rebuild get fresh monotonic ints (no collision with existing)', async () => {
+    const priorIds = [
+      await addEntity('a', 1),
+      await addEntity('b', 2),
+      await addEntity('c', 3)
+    ]
+    const priorInts = priorIds.map(id => getInt(id) as number)
+    const maxPrior = Math.max(...priorInts)
+
+    await rebuild()
+
+    const newId = await addEntity('d', 4)
+    const newInt = getInt(newId) as number
+    expect(newInt).toBeGreaterThan(maxPrior)
+    // Prior entities' ints didn't drift.
+    expect(priorIds.map(id => getInt(id))).toEqual(priorInts)
+  })
+
+  it('removed entities leave a permanent hole — new entities never recycle the gap', async () => {
+    const ids = [
+      await addEntity('a', 1),
+      await addEntity('b', 2),
+      await addEntity('c', 3),
+      await addEntity('d', 4),
+      await addEntity('e', 5)
+    ]
+    const beforeInts = ids.map(id => getInt(id) as number)
+    const deletedId = ids[2]
+    const deletedInt = beforeInts[2]
+    const maxBefore = Math.max(...beforeInts)
+
+    await brain.delete(deletedId)
+    expect(getInt(deletedId)).toBeUndefined()
+
+    const newId = await addEntity('f', 6)
+    const newInt = getInt(newId) as number
+    expect(newInt).not.toBe(deletedInt)
+    expect(newInt).toBeGreaterThan(maxBefore)
+
+    // Surviving ids keep their ints across the deletion + the add.
+    const survivors = ids.filter((_, i) => i !== 2)
+    const survivorIntsBefore = beforeInts.filter((_, i) => i !== 2)
+    expect(survivors.map(id => getInt(id))).toEqual(survivorIntsBefore)
+
+    // Survivors' ints also survive a rebuild after the delete.
+    await rebuild()
+    expect(survivors.map(id => getInt(id))).toEqual(survivorIntsBefore)
+    // The deleted id is still gone after rebuild (no resurrection).
+    expect(getInt(deletedId)).toBeUndefined()
+  })
+
+  it('clearAllIndexData() is the explicit nuclear path that DOES renumber', async () => {
+    const id1 = await addEntity('a', 1)
+    const id2 = await addEntity('b', 2)
+    const priorInts = [getInt(id1) as number, getInt(id2) as number]
+    expect(priorInts.every(i => i >= 1)).toBe(true)
+
+    // Nuclear recovery: explicit destructive op. The warning logged here is
+    // the only documented way to invalidate the canonical int space.
+    await (brain as any).metadataIndex.clearAllIndexData()
+
+    // Both UUIDs are gone from the mapper.
+    expect(getInt(id1)).toBeUndefined()
+    expect(getInt(id2)).toBeUndefined()
+
+    // The int counter restarted from 1: the next add() gets int 1.
+    const idAfter = await addEntity('c', 3)
+    expect(getInt(idAfter)).toBe(1)
+  })
+})