soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

chore(8.0): delete vectorStore:mmap wiring — dead in the 8.0 provider world

Browse source 
The mmap-vector backend was a 2.x-era accelerator for the JS HNSW read path:
a native provider registered under 'vectorStore:mmap' supplied a single-file
mmap'd vector store, and JsHnswVectorIndex tried it before per-entity storage
reads with lazy write-back migration.

In the 8.0 provider world nothing registers that key — the native plugin's
3.0 line replaces the entire vector index under the 'vector' provider key
(vectors live inside its own index file), and the open-core path never had
an mmap provider. The wiring is unreachable; per the no-unwired-code rule
it goes away:

- src/hnsw/mmapVectorBackend.ts deleted (175 LOC)
- wireMmapVectorBackend() + its init call site removed from brainy.ts
- VectorStoreMmapProvider + VectorStoreMmapInstance contracts removed
  from plugin.ts
- JsHnswVectorIndex loses the vectorBackend field, setVectorBackend(),
  and the mmap-first branches in getVectorSafe/preloadVectors — the
  storage + UnifiedCache path is now the single read path
- tests/unit/hnsw/mmap-vector-backend.test.ts deleted

The 7.x line keeps the wiring and got the capacity-NaN bugfix as 7.31.3
(see the platform handoff mmap thread). EntityIdMapperProvider stays — the
connections codec and the id mapper still implement it.

1403/1403 tests, build clean.
This commit is contained in:
David Snelling 2026-06-10 09:40:38 -07:00
parent 42159f2bd7
commit 62f6472fa0
5 changed files with 2 additions and 640 deletions
Download patch file Download diff file Expand all files Collapse all files

79
src/brainy.ts
View file

@ -40,10 +40,8 @@ import { PluginRegistry } from './plugin.js'
import type {
BrainyPlugin,
BrainyPluginContext,
GraphCompressionProvider,
VectorStoreMmapProvider
GraphCompressionProvider
} from './plugin.js'
import { MmapVectorBackend } from './hnsw/mmapVectorBackend.js'
import { ConnectionsCodec } from './hnsw/connectionsCodec.js'
import { TransactionManager } from './transaction/TransactionManager.js'
import { RevisionConflictError } from './transaction/RevisionConflictError.js'
@ -604,15 +602,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
this.graphIndex = graphIndex
}
// Wire the mmap-vector backend (2.4.0 #2). When the vectorStore:mmap
// provider is registered AND the storage adapter resolves a real local
// path via getBinaryBlobPath(), open the mmap file there and inject the
// backend into JsHnswVectorIndex. Cloud adapters return null and skip silently;
// JsHnswVectorIndex's behaviour is then identical to pre-2.4.0 (per-entity reads
// from canonical storage). Failures are non-fatal — the index still
// works, just without the zero-copy fast path.
await this.wireMmapVectorBackend()
// Wire the connections codec (2.4.0 #3). When the graph:compression
// provider is registered AND the metadata index exposes a stable
// idMapper, inject a codec that encodes HNSW connections as
@ -9379,72 +9368,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
await this.lazyRebuildPromise
}
/**
* @description Open the mmap-vector backend and inject it into JsHnswVectorIndex.
* Called once during init after plugin activation, metadataIndex init, and
* graphIndex setup — so the idMapper is available and providers are wired.
*
* Activation conditions (ALL must hold; otherwise this is a no-op and the
* index falls back to per-entity storage reads, the pre-2.4.0 behaviour):
* 1. The `vectorStore:mmap` provider is registered (cortex registers this).
* 2. The storage adapter resolves a real local path via `getBinaryBlobPath()`
* — cloud adapters return null and the mmap layer is skipped silently.
* 3. The metadata index exposes its idMapper (a stable UUID↔int map). The
* mmap layout is keyed by these ints, so 2.4.0 #1 is the prerequisite.
*
* Vector dimensionality: read from `this.dimensions` if set; otherwise the
* default for the standard embedding model (384). The dim is fixed at
* file-creation time and cannot change later, so a wrong default would only
* matter on the very first init of a brand-new instance. After the first
* `add()` the in-memory `this.dimensions` is set, and any subsequent init
* (or a re-open) sees the correct value.
*
* Failures here are non-fatal: a debug-level log records the reason, and
* JsHnswVectorIndex continues without the mmap fast path.
*/
private async wireMmapVectorBackend(): Promise<void> {
const provider = this.pluginRegistry.getProvider<VectorStoreMmapProvider>('vectorStore:mmap')
if (!provider) return
const storageWithBlob = this.storage as unknown as {
getBinaryBlobPath?: (key: string) => string | null
}
const vectorPath = storageWithBlob.getBinaryBlobPath?.('_vectors/main') ?? null
if (!vectorPath) return
const idMapper = this.metadataIndex.getIdMapper?.()
if (!idMapper) return
const dim = this.dimensions ?? 384
const initialCapacity = Math.max(idMapper.size * 2, 1024)
try {
const backend = await MmapVectorBackend.open(
provider,
vectorPath,
dim,
initialCapacity,
idMapper
)
this.index.setVectorBackend(backend)
if (!this.config.silent) {
console.log(
`[brainy] vector mmap backend wired (path=${vectorPath}, dim=${dim}, capacity=${initialCapacity})`
)
}
} catch (error) {
// Common cases: dim mismatch from a prior init at a different dimension,
// permission errors, disk full, file corruption. Index keeps working via
// the per-entity storage path; flag the cause for diagnosis.
if (!this.config.silent) {
console.warn(
`[brainy] mmap-vector backend not wired (${(error as Error).message}); ` +
`falling back to per-entity vector reads`
)
}
}
}
/**
* @description Wire the HNSW connections codec (2.4.0 #3). Activates when
* BOTH (a) the `graph:compression` provider is registered (cortex registers

98
src/hnsw/hnswIndex.ts
View file

@ -17,7 +17,6 @@ import { prodLog } from '../utils/logger.js'
import { quantizeSQ8, distanceSQ8 } from '../utils/vectorQuantization.js'
import type { SQ8QuantizedVector } from '../utils/vectorQuantization.js'
import type { VectorIndexProvider } from '../plugin.js'
import { MmapVectorBackend } from './mmapVectorBackend.js'
import { ConnectionsCodec, compressedConnectionsKey } from './connectionsCodec.js'
// Default HNSW parameters
@ -50,13 +49,6 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
private unifiedCache: UnifiedCache // Shared cache with Graph and Metadata indexes
// Always-adaptive caching - no "mode" concept, system adapts automatically
// Optional mmap-vector backend (2.4.0 #2). When set, the read paths
// (preloadVectors, getVectorSafe) try the mmap layer before storage, and
// on a storage hit they write back into the mmap slot — a lazy migration
// that converges upgraded installs to the fast path under live traffic.
// Null on cloud storage adapters (no local path) and pre-injection init.
private vectorBackend: MmapVectorBackend | null = null
// Optional connections codec (2.4.0 #3). When set, node-persist writes the
// node's per-level connection sets as a single delta-varint-compressed
// binary blob (typically ~4× smaller than the legacy JSON-UUID-array
@ -119,22 +111,6 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
this.useParallelization = useParallelization
}
/**
* @description Inject (or detach) the mmap-vector backend. When set, the
* vector read paths (`preloadVectors`, `getVectorSafe`) try the mmap layer
* before storage and lazily write back on storage hits — converging an
* upgraded install to the zero-copy fast path without a migration step.
* Setting to null reverts to the existing per-entity read path.
*
* Lifecycle: brainy.ts wires this after plugin activation when (a) the
* `vectorStore:mmap` provider is registered AND (b) the storage adapter
* resolves a real local path via `getBinaryBlobPath()`. Cloud adapters
* leave it null; JsHnswVectorIndex's behaviour is then identical to pre-2.4.0.
*/
public setVectorBackend(backend: MmapVectorBackend | null): void {
this.vectorBackend = backend
}
/**
* @description Inject (or detach) the connections codec. When set, node
* persistence writes a delta-varint-compressed binary blob alongside an
@ -1180,23 +1156,6 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
const cacheKey = `hnsw:vector:${noun.id}`
const vector = await this.unifiedCache.get(cacheKey, async () => {
// Try the mmap-vector backend first when wired — zero-copy slot read,
// no per-entity object hydration.
if (this.vectorBackend) {
const fromMmap = this.vectorBackend.readByUuid(noun.id)
if (fromMmap) {
this.unifiedCache.set(
cacheKey,
fromMmap,
'vectors',
fromMmap.length * 4,
50
)
return fromMmap
}
}
// Storage fallback — the canonical source of truth.
if (!this.storage) {
throw new Error('Storage not available for vector loading')
}
@ -1206,18 +1165,6 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
throw new Error(`Vector not found for noun ${noun.id}`)
}
// Lazy migration: write back into the mmap slot so the next read is
// mmap-fast. Idempotent + resumable; a write failure is non-fatal —
// the field still serves this read, and the next storage hit re-tries
// the migration. (See mmapVectorBackend.ts for the contract.)
if (this.vectorBackend) {
try {
this.vectorBackend.writeByUuid(noun.id, loaded)
} catch (error) {
prodLog.debug(`MmapVectorBackend write-back failed for ${noun.id}`, error)
}
}
// Add to UnifiedCache with cost-aware eviction
// This competes fairly with Graph and Metadata indexes
this.unifiedCache.set(
@ -1275,50 +1222,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
private async preloadVectors(nodeIds: string[]): Promise<void> {
if (nodeIds.length === 0) return
// mmap-vector backend fast path: batched O(1) slot reads + lazy
// write-back migration on miss. Cache populated from both layers so
// subsequent in-session reads are O(1) memory.
if (this.vectorBackend) {
// Filter out ids already cached so we never re-read.
const uncachedIds: string[] = []
for (const id of nodeIds) {
if (this.unifiedCache.getSync(`hnsw:vector:${id}`) === undefined) {
uncachedIds.push(id)
}
}
if (uncachedIds.length === 0) return
const fromMmap = this.vectorBackend.readBatchByUuid(uncachedIds)
const misses: string[] = []
for (let i = 0; i < uncachedIds.length; i++) {
const id = uncachedIds[i]
const v = fromMmap[i]
if (v) {
this.unifiedCache.set(`vector:${id}`, v, 'vectors', v.length * 4, 50)
} else {
misses.push(id)
}
}
if (misses.length === 0) return
// Storage fallback for the misses, with concurrent lazy write-back.
await Promise.all(misses.map(async (id) => {
if (!this.storage) return
const vector = await this.storage.getNounVector(id)
if (!vector) return
if (this.vectorBackend) {
try {
this.vectorBackend.writeByUuid(id, vector)
} catch (error) {
prodLog.debug(`MmapVectorBackend write-back failed for ${id}`, error)
}
}
this.unifiedCache.set(`vector:${id}`, vector, 'vectors', vector.length * 4, 50)
}))
return
}
// Legacy path: per-entity storage load via UnifiedCache request coalescing.
// Per-entity storage load via UnifiedCache request coalescing.
const promises = nodeIds.map(async (id) => {
const cacheKey = `hnsw:vector:${id}`
return this.unifiedCache.get(cacheKey, async () => {

174
src/hnsw/mmapVectorBackend.ts
View file

@ -1,174 +0,0 @@
/**
* @module hnsw/mmapVectorBackend
* @description Disk-resident vector cache backing JsHnswVectorIndex, wrapping the
* `vectorStore:mmap` provider (cortex's NativeMmapVectorStore in production).
*
* **Why this exists.** Brainy's per-entity vector storage hits two ceilings as
* datasets grow: (1) every `JsHnswVectorIndex.preloadVectors` does a sequential
* `storage.getNounVector` per id (no batch), and (2) cloud and FS storage both
* serialize each vector through JSON / msgpack, blocking zero-copy reads. An
* mmap-backed single-file vector store, addressed by the stable int from
* `EntityIdMapper`, gives O(1) batch reads + OS-page-cache prefetch and is the
* foundation 2.4.0 #2 needs.
*
* **Lazy migration on miss.** First read of an unmigrated UUID falls back to
* `storage.getNounVector` and writes the bytes into the mmap slot before
* returning — idempotent, resumable, and lets an upgraded install converge to
* the fast path under live traffic without a big-bang migration step. The
* per-entity store remains the canonical source of truth; nothing in the per-
* entity layer is rewritten or deleted here.
*
* **Local-FS only in 2.4.0.** Activated when the storage adapter exposes a
* real local path via `getBinaryBlobPath()`. Cloud adapters (S3 / R2 / GCS)
* return null and the backend is not constructed, so JsHnswVectorIndex falls back to
* the existing per-entity read path. Chunked vector segments for remote
* storage are the planned 2.4.0 #2 follow-up.
*/
import { mkdir } from 'node:fs/promises'
import { dirname } from 'node:path'
import type { Vector } from '../coreTypes.js'
import type {
EntityIdMapperProvider,
VectorStoreMmapInstance,
VectorStoreMmapProvider
} from '../plugin.js'
/**
* @description Bridge between brainy's UUID-keyed vector API and cortex's
* int-slot mmap vector store. Translates UUIDs to stable int slots via the
* `EntityIdMapper`, grows the file on demand, and handles read misses by
* surfacing `null` (JsHnswVectorIndex's caller does the lazy storage fallback +
* write-back). The class never touches per-entity storage directly — it owns
* only the mmap layer.
*/
export class MmapVectorBackend {
private readonly store: VectorStoreMmapInstance
private readonly idMapper: EntityIdMapperProvider
private constructor(store: VectorStoreMmapInstance, idMapper: EntityIdMapperProvider) {
this.store = store
this.idMapper = idMapper
}
/**
* @description Open or create the mmap vector file at `path`. If the file
* does not exist, it is created with the given dim + initial capacity. If
* the directory containing `path` does not exist, it is created recursively
* (the cortex provider does not auto-create parent directories).
* @param provider - The `vectorStore:mmap` provider (cortex
* `NativeMmapVectorStore` in production).
* @param path - Absolute path to the vector file. Derived by the caller
* from `storage.getBinaryBlobPath()` on local-FS adapters; cloud adapters
* return null and this method is not called.
* @param dim - Vector dimensionality. Must match the embedding model.
* @param initialCapacity - Slot capacity to allocate when creating a new
* file. The file grows automatically on writes beyond capacity.
* @param idMapper - Stable UUID↔int mapper (post-2.4.0 #1 semantics — never
* renumbers across rebuild). The mmap slots are addressed by these ints,
* so id stability is the load-bearing invariant of the layout.
* @returns A ready-to-use backend.
*/
static async open(
provider: VectorStoreMmapProvider,
path: string,
dim: number,
initialCapacity: number,
idMapper: EntityIdMapperProvider
): Promise<MmapVectorBackend> {
await mkdir(dirname(path), { recursive: true })
let store: VectorStoreMmapInstance
try {
store = provider.open(path)
} catch {
store = provider.create(path, dim, Math.max(initialCapacity, 16))
}
return new MmapVectorBackend(store, idMapper)
}
/** Vector dimensionality of the underlying mmap file. */
get dim(): number {
return this.store.dim
}
/**
* @description Read a vector by UUID. Returns `null` for any of:
* (1) UUID not yet assigned an int (entity never written through this
* backend or storage), (2) int beyond the highest written slot, or
* (3) the underlying mmap read throwing (corrupt / out-of-bounds).
* The null path is the caller's signal to fall back to per-entity storage
* and trigger a write-back migration.
*/
readByUuid(uuid: string): Vector | null {
const intId = this.idMapper.getInt(uuid)
if (intId === undefined || intId >= this.store.count) return null
try {
return this.store.readVector(intId) as unknown as Vector
} catch {
return null
}
}
/**
* @description Batch read vectors by UUID. Returns an array aligned with
* `uuids` where each entry is the vector or `null` for a miss. Order-
* preserving. Internally collapses to a single batched mmap read of only the
* indices that resolve to a written slot, then redistributes the flat output
* back to the caller's order — so missing slots cost only a couple of map
* lookups, not a full N-element read.
*/
readBatchByUuid(uuids: string[]): (Vector | null)[] {
const indices: number[] = []
const slotByUuid = new Map<string, number>()
for (const uuid of uuids) {
const intId = this.idMapper.getInt(uuid)
if (intId !== undefined && intId < this.store.count) {
slotByUuid.set(uuid, indices.length)
indices.push(intId)
}
}
if (indices.length === 0) return uuids.map(() => null)
const flat = this.store.readVectorsBatch(indices)
const dim = this.store.dim
return uuids.map(uuid => {
const slot = slotByUuid.get(uuid)
if (slot === undefined) return null
return flat.slice(slot * dim, (slot + 1) * dim) as unknown as Vector
})
}
/**
* @description Write a vector at the slot for `uuid`. Assigns a new int if
* the UUID is unseen (stable, append-only). Grows the underlying file (slot
* capacity doubled until the target index fits) before writing past
* capacity, so the caller need not worry about file sizing.
*/
writeByUuid(uuid: string, vector: Vector): void {
const intId = this.idMapper.getOrAssign(uuid)
if (intId >= this.store.capacity) {
let newCap = Math.max(this.store.capacity, 16)
while (intId >= newCap) newCap *= 2
this.store.resize(newCap)
}
this.store.writeVector(intId, Array.from(vector as ArrayLike<number>))
}
/**
* @description madvise(WILLNEED) hint for the slots backing `uuids`. Cheap;
* silently drops UUIDs not yet in the mmap. Used by HNSW search to overlap
* disk I/O with graph traversal.
*/
prefetchByUuid(uuids: string[]): void {
const indices: number[] = []
for (const uuid of uuids) {
const intId = this.idMapper.getInt(uuid)
if (intId !== undefined && intId < this.store.count) indices.push(intId)
}
if (indices.length > 0) this.store.prefetch(indices)
}
/** msync — flush dirty pages to disk. */
flush(): void {
this.store.flush()
}
}

49
src/plugin.ts
View file

@ -285,55 +285,6 @@ export interface GraphCompressionProvider {
decode(data: Buffer): number[]
}
/**
* The `'vectorStore:mmap'` provider — a static factory class for an mmap-backed
* vector file. Brainy's JsHnswVectorIndex uses the static `.create()` / `.open()`
* factories to open a single-file store at a derived path on disk-resident
* storage adapters; non-FS adapters with no local-path support skip the mmap
* layer and fall back to per-entity vector reads.
*
* Cortex registers `NativeMmapVectorStore` as this provider — a Rust mmap'd
* f32 array with a header, batch read/write APIs, and madvise prefetch. Slot
* indices are the stable ints from `EntityIdMapper`; lazy migration on read
* miss writes back from per-entity storage into the slot, converging an
* upgraded install to the fast path without a big-bang step.
*/
export interface VectorStoreMmapProvider {
/** Create a new vector file with pre-allocated slot capacity. */
create(path: string, dim: number, capacity: number): VectorStoreMmapInstance
/** Open an existing vector file for read + write. */
open(path: string): VectorStoreMmapInstance
/** Open an existing vector file for read-only access. */
openReadOnly(path: string): VectorStoreMmapInstance
}
/**
* An open mmap-vector-store instance — the surface brainy's JsHnswVectorIndex consumes
* for batch vector reads + lazy migration from per-entity storage on miss.
*/
export interface VectorStoreMmapInstance {
/** Number of vectors written (highest written index + 1). */
readonly count: number
/** Maximum slot capacity before resize is needed. */
readonly capacity: number
/** Vector dimensionality. */
readonly dim: number
/** Write a single vector at `index`. Vector is f64 in JS, stored as f32. */
writeVector(index: number, vector: number[]): void
/** Write a flat batch starting at `startIndex`. Returns vectors written. */
writeVectorsBatch(startIndex: number, vectorsFlat: number[]): number
/** Read a single vector at `index`. Returns f64[]. */
readVector(index: number): number[]
/** Read multiple vectors by index. Returns flat f64[] of length n × dim. */
readVectorsBatch(indices: number[]): number[]
/** Prefetch via madvise(WILLNEED); no-op on unsupported platforms. */
prefetch(indices: number[]): void
/** Grow the vector file to a larger slot capacity. Re-maps the file. */
resize(newCapacity: number): void
/** msync — flush pending writes to disk. */
flush(): void
}
/**
* Storage adapter factory — plugins register these to provide
* new storage backends that users reference by name.