soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

refactor(8.0): simplify config.vector to 3 knobs + fold persistMode (scaffold step 8)

Browse source 
Per user direction and BRAINY-8.0-RENAME-COORDINATION § G.2: the
`config.vector.advanced.{hnsw, diskann}` escape-hatch surface was
over-engineered. The 8.0 config surface collapses to **three knobs**:

```
config.vector = {
  recall?: 'fast' | 'balanced' | 'accurate'
  quantization?: { enabled?, bits?: 8 | 4 }
  persistMode?: 'immediate' | 'deferred'
}
```

The algorithm-internal HNSW knobs (`M`, `efConstruction`, `efSearch`,
`ml`) and DiskANN knobs (`pqM`, `searchListSize`, `alpha`, …) are no
longer exposed at the public surface. The `recall` preset covers the
legitimate quality/latency tradeoff; algorithm-internal tuning is
intentionally hidden. If users need it later, we can add it back —
shipping with too few knobs is easier to evolve than shipping with too
many.

CHANGES

src/types/brainy.types.ts
- BrainyConfig.hnswPersistMode (7.x top-level) → BrainyConfig.vector.persistMode.
- BrainyConfig.vector — dropped `advanced.{hnsw, diskann}` sub-block.
- BrainyConfig.vector.quantization — dropped `rerankMultiplier` (hardcoded to 3).
- BrainyConfig.vector — dropped `vectorStorage: 'memory' | 'lazy'`. Brainy
  always keeps vectors resident in 8.0; the lazy-eviction path was a
  cloud-storage optimisation that's no longer needed (cloud adapters are gone).
- JSDoc tightened to reflect the closed-form contract.

src/utils/recallPreset.ts
- resolveJsHnswConfig() signature narrowed: accepts `{ recall? }` only
  (no more `advanced` parameter).

src/brainy.ts
- normalizeConfig() — no longer carries `hnswPersistMode` on the resolved
  config. Dropped the deprecated 'gcs-native' warning, the gcsStorage
  HMAC-key warning, and the lenient storage-pairing block (dead code now
  that cloud adapters are gone).
- resolveHNSWPersistMode() — collapsed to a one-liner:
  `return this.config.vector?.persistMode ?? 'immediate'`. The cloud-vs-local
  smart-default branching is no longer relevant (filesystem is the only
  persistent backend in 8.0).
- setupIndex() — reads quantization fields from config.vector directly;
  rerankMultiplier hardcoded to 3.

NO-OP scope

The recall preset's 'balanced' = brainy 7.x DEFAULT_CONFIG byte-for-byte,
so users who don't set recall get the same behavior. Users who set
config.hnsw.* in 7.x will need to migrate to config.vector.* per the
8.0 release notes; pure 7.x defaults users see no change.

VERIFICATION

- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing test-isolation race condition
  from step 7, not related to step 8)
This commit is contained in:
David Snelling 2026-06-09 14:20:57 -07:00
parent 0e6263a1bd
commit 8e767408d9
3 changed files with 68 additions and 105 deletions
Download patch file Download diff file Expand all files Collapse all files

74
src/brainy.ts
View file

@ -9068,16 +9068,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
}
/**
* Setup index — single unified HNSW graph.
* Setup index — single unified vector graph.
*
* Smart defaults for HNSW persistence mode:
* - Cloud storage (GCS/S3/R2/Azure): 'deferred' for 30-50x faster adds
* - Local storage (FileSystem/Memory/OPFS): 'immediate' (already fast)
* Brainy 8.0 ships filesystem-only storage (no cloud adapters), so persistence
* mode defaults to `'immediate'` everywhere. Operators who want bulk-ingest
* speed can set `config.vector.persistMode = 'deferred'`.
*/
private setupIndex(): JsHnswVectorIndex {
// 8.0 config surface: config.vector.{quantization, vectorStorage, recall, advanced}.
// The recall preset translates to HNSW knobs (M / efConstruction / efSearch) via
// resolveJsHnswConfig; explicit advanced.hnsw overrides win.
// 8.0 config surface: config.vector.{recall, quantization, persistMode}.
// The recall preset translates to HNSW knobs (M / efConstruction / efSearch)
// via resolveJsHnswConfig. No algorithm-internal knobs are exposed at the
// public surface.
const recallKnobs = resolveJsHnswConfig(this.config.vector)
const vectorCfg = this.config.vector
const indexConfig = {
@ -9087,9 +9088,8 @@ export class Brainy<T = any> implements BrainyInterface<T> {
quantization: vectorCfg?.quantization ? {
enabled: vectorCfg.quantization.enabled ?? false,
bits: vectorCfg.quantization.bits ?? 8,
rerankMultiplier: vectorCfg.quantization.rerankMultiplier ?? 3
} : undefined,
vectorStorage: vectorCfg?.vectorStorage
rerankMultiplier: 3
} : undefined
}
const persistMode = this.resolveHNSWPersistMode()
@ -9211,49 +9211,28 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* Resolve HNSW persistence mode.
* Extracted so both setupIndex() and the HNSW plugin factory path can use it.
*
* User config > smart default:
* - Cloud storage (GCS/S3/R2/Azure): 'deferred' for 30-50x faster adds
* - Local storage (FileSystem/Memory/OPFS): 'immediate' (already fast)
* Brainy 8.0 ships filesystem-only; the cloud-storage smart-default
* (deferred for cloud, immediate for local) collapses to "always
* immediate unless the user overrides via config.vector.persistMode".
*/
private resolveHNSWPersistMode(): 'immediate' | 'deferred' {
let persistMode: 'immediate' | 'deferred' = this.config.hnswPersistMode || 'immediate'
if (!this.config.hnswPersistMode) {
const storageType = this.config.storage?.type || this.getStorageType()
if (['gcs', 's3', 'r2', 'azure'].includes(storageType)) {
persistMode = 'deferred'
}
}
return persistMode
return this.config.vector?.persistMode ?? 'immediate'
}
/**
* Normalize and validate configuration
*/
private normalizeConfig(config?: BrainyConfig): Required<BrainyConfig> {
// Validate storage configuration
if (config?.storage?.type && !['auto', 'memory', 'filesystem', 'opfs', 'remote', 's3', 'r2', 'gcs', 'gcs-native', 'azure'].includes(config.storage.type)) {
throw new Error(`Invalid storage type: ${config.storage.type}. Must be one of: auto, memory, filesystem, opfs, remote, s3, r2, gcs, gcs-native, azure`)
}
// Warn about deprecated gcs-native
if (config?.storage?.type === ('gcs-native' as any)) {
console.warn('⚠️ DEPRECATED: type "gcs-native" is deprecated. Use type "gcs" instead.')
console.warn(' This will continue to work but may be removed in a future version.')
}
// Validate storage type/config pairing (now more lenient)
if (config?.storage) {
const storage = config.storage as any
// Warn about legacy gcsStorage config with HMAC keys
if (storage.gcsStorage && storage.gcsStorage.accessKeyId && storage.gcsStorage.secretAccessKey) {
console.warn('⚠️ GCS with HMAC keys (gcsStorage) is legacy. Consider migrating to native GCS (gcsNativeStorage) with ADC.')
}
// No longer throw errors for mismatches - storageFactory now handles this intelligently
// Both 'gcs' and 'gcs-native' can now use either gcsStorage or gcsNativeStorage
// Validate storage configuration. Brainy 8.0 ships two adapters only —
// FileSystemStorage and MemoryStorage — per BR-BRAINY-80-STORAGE-SIMPLIFY.
// Cloud backup remains supported via operator tooling (db.persist() +
// gsutil / aws s3 cp / rclone / azcopy).
if (config?.storage?.type && !['auto', 'memory', 'filesystem'].includes(config.storage.type)) {
throw new Error(
`Invalid storage type: ${config.storage.type}. Brainy 8.0 ships 'auto', 'memory', and 'filesystem' only. ` +
`Cloud storage adapters (GCS / S3 / R2 / Azure) and OPFS were removed in 8.0; ` +
`back up locally with db.persist() and sync the on-disk artefact with your tool of choice.`
)
}
// Validate numeric configurations
@ -9292,10 +9271,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Memory management options
maxQueryLimit: config?.maxQueryLimit ?? undefined as any,
reservedQueryMemory: config?.reservedQueryMemory ?? undefined as any,
// HNSW persistence mode - undefined = smart default in setupIndex
hnswPersistMode: config?.hnswPersistMode ?? undefined as any,
// Vector index configuration (8.0) — algorithm-neutral surface with
// `recall` preset. See BRAINY-8.0-RENAME-COORDINATION § A.2.
// `recall` preset + `persistMode` (folded in from 7.x's hnswPersistMode).
// See BRAINY-8.0-RENAME-COORDINATION § A.2 + § G.2.
vector: config?.vector ?? undefined as any,
// Embedding initialization - false = lazy init on first embed()
eagerEmbeddings: config?.eagerEmbeddings ?? false,

76
src/types/brainy.types.ts
View file

@ -1110,72 +1110,58 @@ export interface BrainyConfig {
batchWrites?: boolean // Enable write batching for better performance
maxConcurrentOperations?: number // Limit concurrent file operations
// HNSW persistence mode
// Controls when HNSW graph connections are persisted to storage
// - 'immediate': Persist on every add (slow but durable, default for filesystem)
// - 'deferred': Persist only on flush/close (fast, default for cloud storage)
// Cloud storage (GCS/S3/R2/Azure) should use 'deferred' for 30-50× faster adds
hnswPersistMode?: 'immediate' | 'deferred'
/**
* Vector index configuration (Brainy 8.0).
*
* Algorithm-neutral surface. The `recall` preset means the same thing
* whether Brainy's open-core JS HNSW path is in play or a native
* acceleration provider has taken over the `'vector'` provider key:
* - `'fast'` — minimum-latency search, accepts lower recall
* - `'balanced'` — Brainy 7.x defaults, the right pick for almost everyone
* - `'accurate'` — maximum recall, accepts higher latency
*
* Power users may override individual knobs via `advanced.hnsw` (the JS
* path) or `advanced.diskann` (a native DiskANN-style provider). The
* preset is the floor; explicit overrides win.
* Three knobs. No escape hatch. The algorithm-internal HNSW knobs
* (`M`, `efConstruction`, `efSearch`, `ml`, …) and DiskANN knobs
* (`pqM`, `searchListSize`, `alpha`, …) are deliberately not exposed —
* the `recall` preset covers the legitimate quality/latency tradeoff
* range, and the open-core defaults match Brainy 7.x's defaults
* byte-for-byte so a `'balanced'`-default upgrade is a no-op.
*
* **Closed-form contract** locked in handoff thread
* BRAINY-8.0-RENAME-COORDINATION § A.2 (cortex-confirmed 2026-06-09).
* BRAINY-8.0-RENAME-COORDINATION § A.2 + § G.2 (cortex-confirmed 2026-06-09).
*/
vector?: {
/** Recall preset. Defaults to `'balanced'` when omitted. */
/**
* Recall preset.
* - `'fast'` — minimum-latency search, accepts lower recall
* - `'balanced'` (default) — Brainy 7.x defaults, the right pick for almost everyone
* - `'accurate'` — maximum recall, accepts higher latency
*
* Means the same thing whether the open-core JS HNSW path is in play
* or a native vector provider has taken over the `'vector'` provider
* key — Brainy translates to HNSW knobs; native providers translate
* to their own (e.g. DiskANN's `defaultLSearch` / `defaultPaddingFactor`).
*/
recall?: 'fast' | 'balanced' | 'accurate'
/**
* Vector quantization. Independent of `recall` — it trades RAM for a
* small recall hit at any preset. `bits: 8` is SQ8 (4× memory reduction,
* ~0.4% recall loss). `bits: 4` is SQ4 (8× memory reduction, ~1-3% recall
* loss). Both ship in the open-core path; cortex's `distance:sq8` /
* `distance:sq4` SIMD providers accelerate them when available.
*
* Silently ignored on a native DiskANN-style provider path (it uses its
* own PQ math); `strictConfig: 'warn'` flags this at init time.
*/
quantization?: {
enabled?: boolean
bits?: 8 | 4
rerankMultiplier?: number
}
/** Vector storage mode. `'memory'` keeps vectors resident; `'lazy'` evicts after insert. */
vectorStorage?: 'memory' | 'lazy'
/**
* Power-user overrides. The recall preset works for 99% of users — use
* these only if you've measured. `hnsw` overrides apply on the JS HNSW
* path; `diskann` overrides apply when a DiskANN-style native provider
* is in play. Both blocks are honored on their respective paths and
* silently ignored on the other (a one-shot warning fires when
* `strictConfig !== false`).
* Vector persistence mode. `'immediate'` writes per-noun graph state on
* every `add()`; durable but slower. `'deferred'` writes only on
* `flush()` / `close()`; faster bulk ingest. Defaults to `'immediate'`
* on filesystem storage (the only persistent backend in 8.0).
*
* Brainy 7.x called this `hnswPersistMode` at the top level. The 8.0
* surface folds it under `config.vector` alongside `recall`.
*/
advanced?: {
hnsw?: {
M?: number
efConstruction?: number
efSearch?: number
ml?: number
}
diskann?: {
pqM?: number
pqKsub?: number
maxDegree?: number
searchListSize?: number
alpha?: number
useMmapAdjacency?: boolean
mmapAdjacencyPath?: string
}
}
persistMode?: 'immediate' | 'deferred'
}
// Memory management options

23
src/utils/recallPreset.ts
View file

@ -32,9 +32,10 @@ export type RecallPreset = 'fast' | 'balanced' | 'accurate'
export const DEFAULT_RECALL: RecallPreset = 'balanced'
/**
* The set of HNSW knobs that the `recall` preset controls. Power users may
* override any of these via `config.vector.advanced.hnsw` — the preset is the
* floor, explicit overrides win.
* The set of HNSW knobs that the `recall` preset controls. Brainy 8.0 does
* not expose these directly — the preset is the only quality/latency knob
* surfaced on the public config. Algorithm-internal tuning is intentionally
* hidden behind the preset.
*/
export interface RecallHnswKnobs {
M: number
@ -66,8 +67,7 @@ export function resolveRecallHnswKnobs(preset: RecallPreset | undefined): Recall
/**
* Resolve a Brainy 8.0 `VectorIndexConfig` into a concrete `HNSWConfig` for
* the open-core JS HNSW path. Applies the recall preset first, then layers
* the optional `advanced.hnsw` overrides so explicit knobs always win.
* the open-core JS HNSW path. Applies the recall preset.
*
* Native acceleration providers do NOT use this function — they read
* `config.vector.recall` directly and apply their own preset table. This
@ -75,17 +75,16 @@ export function resolveRecallHnswKnobs(preset: RecallPreset | undefined): Recall
*
* @param vectorConfig - The user-supplied `config.vector` block. May be omitted.
* @returns A complete `HNSWConfig` with `M`, `efConstruction`, `efSearch`, `ml`
* populated from preset + overrides.
* populated from the preset.
*/
export function resolveJsHnswConfig(
vectorConfig?: { recall?: RecallPreset; advanced?: { hnsw?: Partial<HNSWConfig> } }
vectorConfig?: { recall?: RecallPreset }
): Pick<HNSWConfig, 'M' | 'efConstruction' | 'efSearch' | 'ml'> {
const preset = resolveRecallHnswKnobs(vectorConfig?.recall)
const overrides = vectorConfig?.advanced?.hnsw ?? {}
return {
M: overrides.M ?? preset.M,
efConstruction: overrides.efConstruction ?? preset.efConstruction,
efSearch: overrides.efSearch ?? preset.efSearch,
ml: overrides.ml ?? 16,
M: preset.M,
efConstruction: preset.efConstruction,
efSearch: preset.efSearch,
ml: 16,
}
}