refactor(8.0): simplify config.vector to 3 knobs + fold persistMode (scaffold step 8)

Per user direction and BRAINY-8.0-RENAME-COORDINATION § G.2: the
`config.vector.advanced.{hnsw, diskann}` escape-hatch surface was
over-engineered. The 8.0 config surface collapses to **three knobs**:

```
config.vector = {
  recall?: 'fast' | 'balanced' | 'accurate'
  quantization?: { enabled?, bits?: 8 | 4 }
  persistMode?: 'immediate' | 'deferred'
}
```

The algorithm-internal HNSW knobs (`M`, `efConstruction`, `efSearch`,
`ml`) and DiskANN knobs (`pqM`, `searchListSize`, `alpha`, …) are no
longer exposed at the public surface. The `recall` preset covers the
legitimate quality/latency tradeoff; algorithm-internal tuning is
intentionally hidden. If users need it later, we can add it back —
shipping with too few knobs is easier to evolve than shipping with too
many.

CHANGES

src/types/brainy.types.ts
- BrainyConfig.hnswPersistMode (7.x top-level) → BrainyConfig.vector.persistMode.
- BrainyConfig.vector — dropped `advanced.{hnsw, diskann}` sub-block.
- BrainyConfig.vector.quantization — dropped `rerankMultiplier` (hardcoded to 3).
- BrainyConfig.vector — dropped `vectorStorage: 'memory' | 'lazy'`. Brainy
  always keeps vectors resident in 8.0; the lazy-eviction path was a
  cloud-storage optimisation that's no longer needed (cloud adapters are gone).
- JSDoc tightened to reflect the closed-form contract.

src/utils/recallPreset.ts
- resolveJsHnswConfig() signature narrowed: accepts `{ recall? }` only
  (no more `advanced` parameter).

src/brainy.ts
- normalizeConfig() — no longer carries `hnswPersistMode` on the resolved
  config. Dropped the deprecated 'gcs-native' warning, the gcsStorage
  HMAC-key warning, and the lenient storage-pairing block (dead code now
  that cloud adapters are gone).
- resolveHNSWPersistMode() — collapsed to a one-liner:
  `return this.config.vector?.persistMode ?? 'immediate'`. The cloud-vs-local
  smart-default branching is no longer relevant (filesystem is the only
  persistent backend in 8.0).
- setupIndex() — reads quantization fields from config.vector directly;
  rerankMultiplier hardcoded to 3.

NO-OP scope

The recall preset's 'balanced' = brainy 7.x DEFAULT_CONFIG byte-for-byte,
so users who don't set recall get the same behavior. Users who set
config.hnsw.* in 7.x will need to migrate to config.vector.* per the
8.0 release notes; pure 7.x defaults users see no change.

VERIFICATION

- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing test-isolation race condition
  from step 7, not related to step 8)
This commit is contained in:
David Snelling 2026-06-09 14:20:57 -07:00
parent 0e6263a1bd
commit 8e767408d9
3 changed files with 68 additions and 105 deletions

View file

@ -9068,16 +9068,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
} }
/** /**
* Setup index single unified HNSW graph. * Setup index single unified vector graph.
* *
* Smart defaults for HNSW persistence mode: * Brainy 8.0 ships filesystem-only storage (no cloud adapters), so persistence
* - Cloud storage (GCS/S3/R2/Azure): 'deferred' for 30-50x faster adds * mode defaults to `'immediate'` everywhere. Operators who want bulk-ingest
* - Local storage (FileSystem/Memory/OPFS): 'immediate' (already fast) * speed can set `config.vector.persistMode = 'deferred'`.
*/ */
private setupIndex(): JsHnswVectorIndex { private setupIndex(): JsHnswVectorIndex {
// 8.0 config surface: config.vector.{quantization, vectorStorage, recall, advanced}. // 8.0 config surface: config.vector.{recall, quantization, persistMode}.
// The recall preset translates to HNSW knobs (M / efConstruction / efSearch) via // The recall preset translates to HNSW knobs (M / efConstruction / efSearch)
// resolveJsHnswConfig; explicit advanced.hnsw overrides win. // via resolveJsHnswConfig. No algorithm-internal knobs are exposed at the
// public surface.
const recallKnobs = resolveJsHnswConfig(this.config.vector) const recallKnobs = resolveJsHnswConfig(this.config.vector)
const vectorCfg = this.config.vector const vectorCfg = this.config.vector
const indexConfig = { const indexConfig = {
@ -9087,9 +9088,8 @@ export class Brainy<T = any> implements BrainyInterface<T> {
quantization: vectorCfg?.quantization ? { quantization: vectorCfg?.quantization ? {
enabled: vectorCfg.quantization.enabled ?? false, enabled: vectorCfg.quantization.enabled ?? false,
bits: vectorCfg.quantization.bits ?? 8, bits: vectorCfg.quantization.bits ?? 8,
rerankMultiplier: vectorCfg.quantization.rerankMultiplier ?? 3 rerankMultiplier: 3
} : undefined, } : undefined
vectorStorage: vectorCfg?.vectorStorage
} }
const persistMode = this.resolveHNSWPersistMode() const persistMode = this.resolveHNSWPersistMode()
@ -9211,49 +9211,28 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* Resolve HNSW persistence mode. * Resolve HNSW persistence mode.
* Extracted so both setupIndex() and the HNSW plugin factory path can use it. * Extracted so both setupIndex() and the HNSW plugin factory path can use it.
* *
* User config > smart default: * Brainy 8.0 ships filesystem-only; the cloud-storage smart-default
* - Cloud storage (GCS/S3/R2/Azure): 'deferred' for 30-50x faster adds * (deferred for cloud, immediate for local) collapses to "always
* - Local storage (FileSystem/Memory/OPFS): 'immediate' (already fast) * immediate unless the user overrides via config.vector.persistMode".
*/ */
private resolveHNSWPersistMode(): 'immediate' | 'deferred' { private resolveHNSWPersistMode(): 'immediate' | 'deferred' {
let persistMode: 'immediate' | 'deferred' = this.config.hnswPersistMode || 'immediate' return this.config.vector?.persistMode ?? 'immediate'
if (!this.config.hnswPersistMode) {
const storageType = this.config.storage?.type || this.getStorageType()
if (['gcs', 's3', 'r2', 'azure'].includes(storageType)) {
persistMode = 'deferred'
}
}
return persistMode
} }
/** /**
* Normalize and validate configuration * Normalize and validate configuration
*/ */
private normalizeConfig(config?: BrainyConfig): Required<BrainyConfig> { private normalizeConfig(config?: BrainyConfig): Required<BrainyConfig> {
// Validate storage configuration // Validate storage configuration. Brainy 8.0 ships two adapters only —
if (config?.storage?.type && !['auto', 'memory', 'filesystem', 'opfs', 'remote', 's3', 'r2', 'gcs', 'gcs-native', 'azure'].includes(config.storage.type)) { // FileSystemStorage and MemoryStorage — per BR-BRAINY-80-STORAGE-SIMPLIFY.
throw new Error(`Invalid storage type: ${config.storage.type}. Must be one of: auto, memory, filesystem, opfs, remote, s3, r2, gcs, gcs-native, azure`) // Cloud backup remains supported via operator tooling (db.persist() +
} // gsutil / aws s3 cp / rclone / azcopy).
if (config?.storage?.type && !['auto', 'memory', 'filesystem'].includes(config.storage.type)) {
// Warn about deprecated gcs-native throw new Error(
if (config?.storage?.type === ('gcs-native' as any)) { `Invalid storage type: ${config.storage.type}. Brainy 8.0 ships 'auto', 'memory', and 'filesystem' only. ` +
console.warn('⚠️ DEPRECATED: type "gcs-native" is deprecated. Use type "gcs" instead.') `Cloud storage adapters (GCS / S3 / R2 / Azure) and OPFS were removed in 8.0; ` +
console.warn(' This will continue to work but may be removed in a future version.') `back up locally with db.persist() and sync the on-disk artefact with your tool of choice.`
} )
// Validate storage type/config pairing (now more lenient)
if (config?.storage) {
const storage = config.storage as any
// Warn about legacy gcsStorage config with HMAC keys
if (storage.gcsStorage && storage.gcsStorage.accessKeyId && storage.gcsStorage.secretAccessKey) {
console.warn('⚠️ GCS with HMAC keys (gcsStorage) is legacy. Consider migrating to native GCS (gcsNativeStorage) with ADC.')
}
// No longer throw errors for mismatches - storageFactory now handles this intelligently
// Both 'gcs' and 'gcs-native' can now use either gcsStorage or gcsNativeStorage
} }
// Validate numeric configurations // Validate numeric configurations
@ -9292,10 +9271,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Memory management options // Memory management options
maxQueryLimit: config?.maxQueryLimit ?? undefined as any, maxQueryLimit: config?.maxQueryLimit ?? undefined as any,
reservedQueryMemory: config?.reservedQueryMemory ?? undefined as any, reservedQueryMemory: config?.reservedQueryMemory ?? undefined as any,
// HNSW persistence mode - undefined = smart default in setupIndex
hnswPersistMode: config?.hnswPersistMode ?? undefined as any,
// Vector index configuration (8.0) — algorithm-neutral surface with // Vector index configuration (8.0) — algorithm-neutral surface with
// `recall` preset. See BRAINY-8.0-RENAME-COORDINATION § A.2. // `recall` preset + `persistMode` (folded in from 7.x's hnswPersistMode).
// See BRAINY-8.0-RENAME-COORDINATION § A.2 + § G.2.
vector: config?.vector ?? undefined as any, vector: config?.vector ?? undefined as any,
// Embedding initialization - false = lazy init on first embed() // Embedding initialization - false = lazy init on first embed()
eagerEmbeddings: config?.eagerEmbeddings ?? false, eagerEmbeddings: config?.eagerEmbeddings ?? false,

View file

@ -1110,72 +1110,58 @@ export interface BrainyConfig {
batchWrites?: boolean // Enable write batching for better performance batchWrites?: boolean // Enable write batching for better performance
maxConcurrentOperations?: number // Limit concurrent file operations maxConcurrentOperations?: number // Limit concurrent file operations
// HNSW persistence mode
// Controls when HNSW graph connections are persisted to storage
// - 'immediate': Persist on every add (slow but durable, default for filesystem)
// - 'deferred': Persist only on flush/close (fast, default for cloud storage)
// Cloud storage (GCS/S3/R2/Azure) should use 'deferred' for 30-50× faster adds
hnswPersistMode?: 'immediate' | 'deferred'
/** /**
* Vector index configuration (Brainy 8.0). * Vector index configuration (Brainy 8.0).
* *
* Algorithm-neutral surface. The `recall` preset means the same thing * Three knobs. No escape hatch. The algorithm-internal HNSW knobs
* whether Brainy's open-core JS HNSW path is in play or a native * (`M`, `efConstruction`, `efSearch`, `ml`, ) and DiskANN knobs
* acceleration provider has taken over the `'vector'` provider key: * (`pqM`, `searchListSize`, `alpha`, ) are deliberately not exposed
* - `'fast'` minimum-latency search, accepts lower recall * the `recall` preset covers the legitimate quality/latency tradeoff
* - `'balanced'` Brainy 7.x defaults, the right pick for almost everyone * range, and the open-core defaults match Brainy 7.x's defaults
* - `'accurate'` maximum recall, accepts higher latency * byte-for-byte so a `'balanced'`-default upgrade is a no-op.
*
* Power users may override individual knobs via `advanced.hnsw` (the JS
* path) or `advanced.diskann` (a native DiskANN-style provider). The
* preset is the floor; explicit overrides win.
* *
* **Closed-form contract** locked in handoff thread * **Closed-form contract** locked in handoff thread
* BRAINY-8.0-RENAME-COORDINATION § A.2 (cortex-confirmed 2026-06-09). * BRAINY-8.0-RENAME-COORDINATION § A.2 + § G.2 (cortex-confirmed 2026-06-09).
*/ */
vector?: { vector?: {
/** Recall preset. Defaults to `'balanced'` when omitted. */ /**
* Recall preset.
* - `'fast'` minimum-latency search, accepts lower recall
* - `'balanced'` (default) Brainy 7.x defaults, the right pick for almost everyone
* - `'accurate'` maximum recall, accepts higher latency
*
* Means the same thing whether the open-core JS HNSW path is in play
* or a native vector provider has taken over the `'vector'` provider
* key Brainy translates to HNSW knobs; native providers translate
* to their own (e.g. DiskANN's `defaultLSearch` / `defaultPaddingFactor`).
*/
recall?: 'fast' | 'balanced' | 'accurate' recall?: 'fast' | 'balanced' | 'accurate'
/** /**
* Vector quantization. Independent of `recall` it trades RAM for a * Vector quantization. Independent of `recall` it trades RAM for a
* small recall hit at any preset. `bits: 8` is SQ8 (4× memory reduction, * small recall hit at any preset. `bits: 8` is SQ8 (4× memory reduction,
* ~0.4% recall loss). `bits: 4` is SQ4 (8× memory reduction, ~1-3% recall * ~0.4% recall loss). `bits: 4` is SQ4 (8× memory reduction, ~1-3% recall
* loss). Both ship in the open-core path; cortex's `distance:sq8` / * loss). Both ship in the open-core path; cortex's `distance:sq8` /
* `distance:sq4` SIMD providers accelerate them when available. * `distance:sq4` SIMD providers accelerate them when available.
*
* Silently ignored on a native DiskANN-style provider path (it uses its
* own PQ math); `strictConfig: 'warn'` flags this at init time.
*/ */
quantization?: { quantization?: {
enabled?: boolean enabled?: boolean
bits?: 8 | 4 bits?: 8 | 4
rerankMultiplier?: number
} }
/** Vector storage mode. `'memory'` keeps vectors resident; `'lazy'` evicts after insert. */
vectorStorage?: 'memory' | 'lazy'
/** /**
* Power-user overrides. The recall preset works for 99% of users use * Vector persistence mode. `'immediate'` writes per-noun graph state on
* these only if you've measured. `hnsw` overrides apply on the JS HNSW * every `add()`; durable but slower. `'deferred'` writes only on
* path; `diskann` overrides apply when a DiskANN-style native provider * `flush()` / `close()`; faster bulk ingest. Defaults to `'immediate'`
* is in play. Both blocks are honored on their respective paths and * on filesystem storage (the only persistent backend in 8.0).
* silently ignored on the other (a one-shot warning fires when *
* `strictConfig !== false`). * Brainy 7.x called this `hnswPersistMode` at the top level. The 8.0
* surface folds it under `config.vector` alongside `recall`.
*/ */
advanced?: { persistMode?: 'immediate' | 'deferred'
hnsw?: {
M?: number
efConstruction?: number
efSearch?: number
ml?: number
}
diskann?: {
pqM?: number
pqKsub?: number
maxDegree?: number
searchListSize?: number
alpha?: number
useMmapAdjacency?: boolean
mmapAdjacencyPath?: string
}
}
} }
// Memory management options // Memory management options

View file

@ -32,9 +32,10 @@ export type RecallPreset = 'fast' | 'balanced' | 'accurate'
export const DEFAULT_RECALL: RecallPreset = 'balanced' export const DEFAULT_RECALL: RecallPreset = 'balanced'
/** /**
* The set of HNSW knobs that the `recall` preset controls. Power users may * The set of HNSW knobs that the `recall` preset controls. Brainy 8.0 does
* override any of these via `config.vector.advanced.hnsw` the preset is the * not expose these directly the preset is the only quality/latency knob
* floor, explicit overrides win. * surfaced on the public config. Algorithm-internal tuning is intentionally
* hidden behind the preset.
*/ */
export interface RecallHnswKnobs { export interface RecallHnswKnobs {
M: number M: number
@ -66,8 +67,7 @@ export function resolveRecallHnswKnobs(preset: RecallPreset | undefined): Recall
/** /**
* Resolve a Brainy 8.0 `VectorIndexConfig` into a concrete `HNSWConfig` for * Resolve a Brainy 8.0 `VectorIndexConfig` into a concrete `HNSWConfig` for
* the open-core JS HNSW path. Applies the recall preset first, then layers * the open-core JS HNSW path. Applies the recall preset.
* the optional `advanced.hnsw` overrides so explicit knobs always win.
* *
* Native acceleration providers do NOT use this function they read * Native acceleration providers do NOT use this function they read
* `config.vector.recall` directly and apply their own preset table. This * `config.vector.recall` directly and apply their own preset table. This
@ -75,17 +75,16 @@ export function resolveRecallHnswKnobs(preset: RecallPreset | undefined): Recall
* *
* @param vectorConfig - The user-supplied `config.vector` block. May be omitted. * @param vectorConfig - The user-supplied `config.vector` block. May be omitted.
* @returns A complete `HNSWConfig` with `M`, `efConstruction`, `efSearch`, `ml` * @returns A complete `HNSWConfig` with `M`, `efConstruction`, `efSearch`, `ml`
* populated from preset + overrides. * populated from the preset.
*/ */
export function resolveJsHnswConfig( export function resolveJsHnswConfig(
vectorConfig?: { recall?: RecallPreset; advanced?: { hnsw?: Partial<HNSWConfig> } } vectorConfig?: { recall?: RecallPreset }
): Pick<HNSWConfig, 'M' | 'efConstruction' | 'efSearch' | 'ml'> { ): Pick<HNSWConfig, 'M' | 'efConstruction' | 'efSearch' | 'ml'> {
const preset = resolveRecallHnswKnobs(vectorConfig?.recall) const preset = resolveRecallHnswKnobs(vectorConfig?.recall)
const overrides = vectorConfig?.advanced?.hnsw ?? {}
return { return {
M: overrides.M ?? preset.M, M: preset.M,
efConstruction: overrides.efConstruction ?? preset.efConstruction, efConstruction: preset.efConstruction,
efSearch: overrides.efSearch ?? preset.efSearch, efSearch: preset.efSearch,
ml: overrides.ml ?? 16, ml: 16,
} }
} }