feat(plugin): DiskAnnProvider contract + HNSWConfig.type/diskann knobs
Adds the plugin-side surface for the new billion-scale index option:
- plugin.ts: DiskAnnProvider type alias (mirrors HnswProvider's shape
so cortex's DiskANN wrapper is a drop-in for the 'hnsw' factory).
- coreTypes.ts: HNSWConfig.type ('hnsw' | 'diskann') for explicit
engine selection, plus HNSWConfig.diskann for the build-time tuning
knobs (pqM, pqKsub, maxDegree, searchListSize, alpha, mmap
adjacency).
No algorithm code here — the implementation lives in the cortex
plugin. Brainy without cortex continues to use HNSW exactly as before.
This commit is contained in:
parent
4927d49e87
commit
f885f813fe
2 changed files with 57 additions and 0 deletions
|
|
@ -216,6 +216,31 @@ export interface HnswProvider {
|
|||
getPersistMode(): 'immediate' | 'deferred'
|
||||
}
|
||||
|
||||
/**
|
||||
* The `'diskann'` provider — a billion-scale alternative to HNSW backed
|
||||
* by cortex's pure-Rust Vamana + Product Quantization implementation
|
||||
* (see ADR-002 in the cortex repo).
|
||||
*
|
||||
* Structurally a drop-in for [`HnswProvider`]: brainy calls the same
|
||||
* `addItem` / `search` / `rebuild` surface and never needs to know
|
||||
* which index is underneath. The differences are operational:
|
||||
*
|
||||
* - **Memory footprint**: ~16 GB RAM at 1 B vectors (PQ codes only).
|
||||
* HNSW would need ~1.5 TB to keep the full vectors resident.
|
||||
* - **Build model**: build-once, query-many. Dynamic insertions buffer
|
||||
* to an in-memory delta brute-forced alongside the main index;
|
||||
* `rebuild()` folds them in.
|
||||
* - **Storage**: requires a local filesystem path (NVMe SSD for the
|
||||
* published latency numbers). Cloud-storage adapters continue using
|
||||
* HNSW.
|
||||
*
|
||||
* Engagement is automatic when the cortex provider is registered, the
|
||||
* storage adapter exposes `getBinaryBlobPath`, and the metadata index
|
||||
* has a stable `idMapper`. Users force the legacy index via
|
||||
* `config.index.type = 'hnsw'`.
|
||||
*/
|
||||
export type DiskAnnProvider = HnswProvider
|
||||
|
||||
/**
|
||||
* The `'entityIdMapper'` provider — a drop-in for `EntityIdMapper`. Injected
|
||||
* into the TypeScript `MetadataIndexManager` when a native metadata index is
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue