feat: mmap-vector backend wiring — HNSWIndex consumes vectorStore:mmap (2.4.0 #2)
Cortex already registers the vectorStore:mmap provider (its Rust NativeMmapVectorStore), but brainy has never consumed it — preloadVectors and getVectorSafe still go straight to storage.getNounVector for every id, even when an mmap layer is available. This wires the consumer end. Architecture: - NEW MmapVectorBackend (src/hnsw/mmapVectorBackend.ts) — bridges brainy's UUID-keyed vector reads to a int-slot mmap file via the vectorStore:mmap provider. Slots are addressed by the stable int id from the post-2.4.0 #1 EntityIdMapper (the foundation this depends on). Auto-grows the file (doubling) when a write lands beyond capacity, so HNSWIndex never has to think about sizing. The class never touches per-entity storage — it owns only the mmap layer. - HNSWIndex changes — adds a vectorBackend field + a setVectorBackend setter. The vector read paths (preloadVectors, getVectorSafe) try the mmap layer first; on a storage fallback hit, they LAZILY write back into the mmap slot. An upgraded install converges to the zero-copy fast path under live traffic — no big-bang migration step. The legacy per-entity path is preserved and still used when no backend is set. - brainy.ts wiring — a new private wireMmapVectorBackend() runs once during init, after plugin activation + metadataIndex setup. It activates the backend only when (a) the vectorStore:mmap provider is registered, (b) the storage adapter resolves a real local path via getBinaryBlobPath(), and (c) the metadata index exposes its idMapper. Cloud adapters return null on (b) and the backend is silently skipped; HNSWIndex's behaviour is then identical to pre-2.4.0. - Provider interfaces in plugin.ts — VectorStoreMmapProvider and VectorStoreMmapInstance document the contract cortex's class fulfils (the class IS the provider — static factory methods). Brainy depends on the interfaces, not on cortex; the structural match is verified when cortex 2.4.0 picks up this brainy release. Tests (1428 total, +11 vs pre-2.4.0): - tests/unit/hnsw/mmap-vector-backend.test.ts — 6 unit tests with an in-memory mock provider. Covers round-trip, batch reads with interleaved misses, slot stability (no re-slotting on overwrite), file growth without data loss, idempotent open, and null returns for unwritten slots. The real perf integration with cortex's NativeMmapVectorStore is exercised when cortex 2.4.0 wires this in. - tests/unit/utils/entity-id-mapper-stability.test.ts — moved here from tests/regression/ (which is NOT in the unit-config include glob, so the five #23 tests were not actually being run by npm test). The unit config matches tests/unit/**/*.test.ts. The 2.4.0 #2 follow-up will be the chunked-segment layout for remote storage adapters (S3 / R2 / GCS) where a single growing file doesn't fit immutable objects. For 2.4.0 release: local-FS only.
This commit is contained in:
parent
b2408cb5a6
commit
d4cb26c604
6 changed files with 640 additions and 4 deletions
|
|
@ -253,6 +253,55 @@ export interface CacheProvider {
|
|||
// uses at the `getProvider('embeddings')` call site. No separate interface is
|
||||
// added here to avoid a duplicate, unwired contract.
|
||||
|
||||
/**
|
||||
* The `'vectorStore:mmap'` provider — a static factory class for an mmap-backed
|
||||
* vector file. Brainy's HNSWIndex uses the static `.create()` / `.open()`
|
||||
* factories to open a single-file store at a derived path on disk-resident
|
||||
* storage adapters; non-FS adapters with no local-path support skip the mmap
|
||||
* layer and fall back to per-entity vector reads.
|
||||
*
|
||||
* Cortex registers `NativeMmapVectorStore` as this provider — a Rust mmap'd
|
||||
* f32 array with a header, batch read/write APIs, and madvise prefetch. Slot
|
||||
* indices are the stable ints from `EntityIdMapper`; lazy migration on read
|
||||
* miss writes back from per-entity storage into the slot, converging an
|
||||
* upgraded install to the fast path without a big-bang step.
|
||||
*/
|
||||
export interface VectorStoreMmapProvider {
|
||||
/** Create a new vector file with pre-allocated slot capacity. */
|
||||
create(path: string, dim: number, capacity: number): VectorStoreMmapInstance
|
||||
/** Open an existing vector file for read + write. */
|
||||
open(path: string): VectorStoreMmapInstance
|
||||
/** Open an existing vector file for read-only access. */
|
||||
openReadOnly(path: string): VectorStoreMmapInstance
|
||||
}
|
||||
|
||||
/**
|
||||
* An open mmap-vector-store instance — the surface brainy's HNSWIndex consumes
|
||||
* for batch vector reads + lazy migration from per-entity storage on miss.
|
||||
*/
|
||||
export interface VectorStoreMmapInstance {
|
||||
/** Number of vectors written (highest written index + 1). */
|
||||
readonly count: number
|
||||
/** Maximum slot capacity before resize is needed. */
|
||||
readonly capacity: number
|
||||
/** Vector dimensionality. */
|
||||
readonly dim: number
|
||||
/** Write a single vector at `index`. Vector is f64 in JS, stored as f32. */
|
||||
writeVector(index: number, vector: number[]): void
|
||||
/** Write a flat batch starting at `startIndex`. Returns vectors written. */
|
||||
writeVectorsBatch(startIndex: number, vectorsFlat: number[]): number
|
||||
/** Read a single vector at `index`. Returns f64[]. */
|
||||
readVector(index: number): number[]
|
||||
/** Read multiple vectors by index. Returns flat f64[] of length n × dim. */
|
||||
readVectorsBatch(indices: number[]): number[]
|
||||
/** Prefetch via madvise(WILLNEED); no-op on unsupported platforms. */
|
||||
prefetch(indices: number[]): void
|
||||
/** Grow the vector file to a larger slot capacity. Re-maps the file. */
|
||||
resize(newCapacity: number): void
|
||||
/** msync — flush pending writes to disk. */
|
||||
flush(): void
|
||||
}
|
||||
|
||||
/**
|
||||
* Storage adapter factory — plugins register these to provide
|
||||
* new storage backends that users reference by name.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue