refactor(8.0): final cleanup — drop HnswProvider alias + config.hnsw + 'hnsw' surface (scaffold step 6)
Step 6 of the brainy 8.0 rename scaffolding. Removes every legacy 'hnsw'-
flavoured public name introduced as a compat shim in steps 1-5. The
algorithm-neutral surface is now the only surface.
REMOVED — PHASE A (type aliases)
src/plugin.ts
- Removed `export type HnswProvider = VectorIndexProvider` alias.
- Removed `export type DiskAnnProvider = VectorIndexProvider` alias.
src/index.ts
- Removed `export const HNSWIndex = JsHnswVectorIndex` alias.
src/types/brainy.types.ts
- Removed `BrainyStats.indexHealth.hnsw` field (the new `vector` field
carries the same boolean).
src/brainy.ts
- `stats()` no longer emits the legacy `hnsw` field on `indexHealth`.
REMOVED — PHASE B (storage adapter rename, 8 adapters)
Repo-wide rename: `saveHNSWData` → `saveVectorIndexData` and
`getHNSWData` → `getVectorIndexData` across:
- src/storage/adapters/baseStorageAdapter.ts (abstract declarations)
- src/storage/adapters/fileSystemStorage.ts
- src/storage/adapters/gcsStorage.ts
- src/storage/adapters/r2Storage.ts
- src/storage/adapters/s3CompatibleStorage.ts
- src/storage/adapters/azureBlobStorage.ts
- src/storage/adapters/opfsStorage.ts
- src/storage/adapters/memoryStorage.ts
- src/storage/adapters/historicalStorageAdapter.ts
- src/brainy.ts (call sites)
- src/hnsw/hnswIndex.ts + src/hnsw/typeAwareHNSWIndex.ts (call sites)
The default-delegation wrappers added in scaffold step 4 are removed
(would have been duplicate declarations after the rename).
REMOVED — PHASE C (cache category 'hnsw')
src/utils/unifiedCache.ts
- Cache-category union narrowed from
'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other'
to
'vectors' | 'metadata' | 'embedding' | 'other'
- typeAccessCounts, typeSizes, typeCounts, accessRatios, sizeRatios,
and the fairness-check iterator all drop the 'hnsw' key.
- Per planning § 2.5: pre-8.0 'hnsw' cache entries are an in-memory
category. Cache is rebuildable; entries naturally don't exist after
a restart, so no migration path is needed.
src/hnsw/hnswIndex.ts
- 3 cache.set() call sites migrated from category 'hnsw' to 'vectors'.
- Cache key prefix `hnsw:vector:` → `vector:`.
- typeCounts/typeSizes/typeAccessCounts accessor renames
`.hnsw` → `.vectors`.
tests/unit/utils/unifiedCache-eviction.test.ts
- Test fixtures updated: cache.set(..., 'hnsw', ...) → cache.set(..., 'vectors', ...).
- Stats assertion `stats.typeSizes.hnsw` → `stats.typeSizes.vectors`.
REMOVED — PHASE D (config.hnsw)
src/types/brainy.types.ts
- Removed `BrainyConfig.hnsw` field entirely. The 8.0 surface is
`BrainyConfig.vector.{recall, quantization, vectorStorage, advanced}`.
src/brainy.ts
- normalizeConfig() no longer emits a `hnsw` field on Required<BrainyConfig>.
- setupIndex() rewired:
- Reads from `this.config.vector` (not `this.config.hnsw`).
- Calls `resolveJsHnswConfig(this.config.vector)` to translate the
`recall` preset into M / efConstruction / efSearch knobs (with
`advanced.hnsw` overrides winning when supplied).
- Imports `resolveJsHnswConfig` from './utils/recallPreset.js'.
NOT IN THIS COMMIT (deliberately)
- Persisted file path migration `_system/hnsw-*.json` →
`_system/vector-index-*.json`. Requires dual-read logic on boot
across 8 storage adapters. Per integration doc lines 531-539:
"Reader accepts either spelling on load; writer emits the new spelling
only; brains self-migrate on the next persist after upgrade." This is
a separate body of work and lands in a follow-up commit before 8.0 GA.
- `config.hnswPersistMode` top-level field is unchanged. It's not in
the rename inventory; a future commit can fold it into
`config.vector.persistMode` if desired.
- strictConfig enforcement wiring. The field is accepted by
normalizeConfig() with default 'warn'; the actual warning emission at
knob-mismatch sites lands when the config-resolution layer is touched
for the persisted-path migration.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit (one test fixture updated to use the new
category name; assertion still passes after fixture rename)
- npm run build: clean
The 8.0 PR's user-facing surface is now algorithm-neutral end-to-end:
VectorIndexProvider, config.vector.recall, JsHnswVectorIndex,
saveVectorIndexData, 'vectors' cache category, indexHealth.vector,
strictConfig — all standalone. No legacy 'hnsw' name remains in any
public type or method signature.
This commit is contained in:
parent
3e1ef957bc
commit
b20666e020
17 changed files with 76 additions and 145 deletions
|
|
@ -110,6 +110,7 @@ import type { MigrationPreview, MigrationResult, MigrateOptions } from './migrat
|
|||
import { AggregationIndex } from './aggregation/AggregationIndex.js'
|
||||
import { AggregateMaterializer } from './aggregation/materializer.js'
|
||||
import type { AggregateDefinition, AggregateQueryParams, AggregateResult } from './types/brainy.types.js'
|
||||
import { resolveJsHnswConfig } from './utils/recallPreset.js'
|
||||
|
||||
/**
|
||||
* Stopwords for semantic highlighting
|
||||
|
|
@ -3615,7 +3616,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
|
||||
// OPTIMIZATION: Defer HNSW persistence during batch insert.
|
||||
// Without this, each add() triggers ~16-20 neighbor saveHNSWData calls
|
||||
// Without this, each add() triggers ~16-20 neighbor saveVectorIndexData calls
|
||||
// (each a read→gzip→atomic-write cycle). For 450 items that's ~8,100
|
||||
// individual storage writes. Deferred mode collects dirty node IDs and
|
||||
// flushes once at the end — deduplicating repeated neighbor updates.
|
||||
|
|
@ -6270,13 +6271,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
relationsByType,
|
||||
fieldRegistry,
|
||||
indexHealth: await (async () => {
|
||||
// 8.0 rename: `vector` is the canonical name; `hnsw` is a compat
|
||||
// alias that's removed in the final 8.0 cleanup commit.
|
||||
const vectorHealthy = this.index.size() > 0 || entityCount === 0
|
||||
const graphSize = await this.graphIndex.size()
|
||||
return {
|
||||
hnsw: vectorHealthy,
|
||||
vector: vectorHealthy,
|
||||
vector: this.index.size() > 0 || entityCount === 0,
|
||||
metadata: metadataStats.totalEntries > 0 || entityCount === 0,
|
||||
graph: graphSize > 0 || relationCount === 0
|
||||
}
|
||||
|
|
@ -9078,16 +9075,21 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* - Local storage (FileSystem/Memory/OPFS): 'immediate' (already fast)
|
||||
*/
|
||||
private setupIndex(): JsHnswVectorIndex {
|
||||
// 8.0 config surface: config.vector.{quantization, vectorStorage, recall, advanced}.
|
||||
// The recall preset translates to HNSW knobs (M / efConstruction / efSearch) via
|
||||
// resolveJsHnswConfig; explicit advanced.hnsw overrides win.
|
||||
const recallKnobs = resolveJsHnswConfig(this.config.vector)
|
||||
const vectorCfg = this.config.vector
|
||||
const indexConfig = {
|
||||
...this.config.index,
|
||||
...recallKnobs,
|
||||
distanceFunction: this.distance,
|
||||
// Wire HNSW optimization config (v7.11.0)
|
||||
quantization: this.config.hnsw?.quantization ? {
|
||||
enabled: this.config.hnsw.quantization.enabled ?? false,
|
||||
bits: this.config.hnsw.quantization.bits ?? 8,
|
||||
rerankMultiplier: this.config.hnsw.quantization.rerankMultiplier ?? 3
|
||||
quantization: vectorCfg?.quantization ? {
|
||||
enabled: vectorCfg.quantization.enabled ?? false,
|
||||
bits: vectorCfg.quantization.bits ?? 8,
|
||||
rerankMultiplier: vectorCfg.quantization.rerankMultiplier ?? 3
|
||||
} : undefined,
|
||||
vectorStorage: this.config.hnsw?.vectorStorage
|
||||
vectorStorage: vectorCfg?.vectorStorage
|
||||
}
|
||||
|
||||
const persistMode = this.resolveHNSWPersistMode()
|
||||
|
|
@ -9292,8 +9294,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
reservedQueryMemory: config?.reservedQueryMemory ?? undefined as any,
|
||||
// HNSW persistence mode - undefined = smart default in setupIndex
|
||||
hnswPersistMode: config?.hnswPersistMode ?? undefined as any,
|
||||
// HNSW optimization options (v7.11.0) — deprecated in 8.0, see `vector` below.
|
||||
hnsw: config?.hnsw ?? undefined as any,
|
||||
// Vector index configuration (8.0) — algorithm-neutral surface with
|
||||
// `recall` preset. See BRAINY-8.0-RENAME-COORDINATION § A.2.
|
||||
vector: config?.vector ?? undefined as any,
|
||||
|
|
@ -9464,7 +9464,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* storage adapter directly, so the codec is engaged on cloud adapters too.
|
||||
* Format convergence is lazy: pre-2.4.0 nodes load via the legacy path,
|
||||
* then the next dirty save writes the compressed form (and the legacy
|
||||
* field of saveHNSWData becomes empty), so all reads converge over time
|
||||
* field of saveVectorIndexData becomes empty), so all reads converge over time
|
||||
* without an explicit migration step.
|
||||
*/
|
||||
private wireConnectionsCodec(): void {
|
||||
|
|
|
|||
|
|
@ -61,7 +61,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
// Optional connections codec (2.4.0 #3). When set, node-persist writes the
|
||||
// node's per-level connection sets as a single delta-varint-compressed
|
||||
// binary blob (typically ~4× smaller than the legacy JSON-UUID-array
|
||||
// shape), plus an empty `connections: {}` in saveHNSWData as the marker.
|
||||
// shape), plus an empty `connections: {}` in saveVectorIndexData as the marker.
|
||||
// The read path tries to load the blob first; missing blob → legacy
|
||||
// connections field. Format convergence is lazy: pre-2.4.0 nodes still
|
||||
// load via the legacy path, then write the compressed form on next dirty
|
||||
|
|
@ -139,7 +139,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
/**
|
||||
* @description Inject (or detach) the connections codec. When set, node
|
||||
* persistence writes a delta-varint-compressed binary blob alongside an
|
||||
* empty `connections: {}` marker in saveHNSWData; the load path reads the
|
||||
* empty `connections: {}` marker in saveVectorIndexData; the load path reads the
|
||||
* blob and decodes. Null reverts to the legacy JSON-array path. Wiring is
|
||||
* done by brainy.ts when the `graph:compression` provider is registered
|
||||
* AND the storage adapter exposes the binary-blob primitive AND the
|
||||
|
|
@ -227,7 +227,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
* @description Persist one node's connections. When the connections codec is
|
||||
* wired AND the storage adapter exposes `saveBinaryBlob`, the per-level
|
||||
* connection sets are encoded into a single compact buffer and stored as a
|
||||
* binary blob; `saveHNSWData` then records the node's level and an empty
|
||||
* binary blob; `saveVectorIndexData` then records the node's level and an empty
|
||||
* `connections: {}` as the marker. Otherwise the legacy JSON-array path is
|
||||
* taken — the format every brainy release before 2.4.0 wrote. The two are
|
||||
* intentionally NOT dual-written: one or the other, never both, so format
|
||||
|
|
@ -250,7 +250,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
if (canCompress) {
|
||||
const encoded = this.connectionsCodec!.encode(noun.connections)
|
||||
await storageWithBlob.saveBinaryBlob!(compressedConnectionsKey(nodeId), encoded)
|
||||
await this.storage.saveHNSWData(nodeId, {
|
||||
await this.storage.saveVectorIndexData(nodeId, {
|
||||
level: noun.level,
|
||||
connections: {}
|
||||
})
|
||||
|
|
@ -261,7 +261,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
for (const [level, nounIds] of noun.connections.entries()) {
|
||||
connectionsObj[level.toString()] = Array.from(nounIds)
|
||||
}
|
||||
await this.storage.saveHNSWData(nodeId, {
|
||||
await this.storage.saveVectorIndexData(nodeId, {
|
||||
level: noun.level,
|
||||
connections: connectionsObj
|
||||
})
|
||||
|
|
@ -1197,7 +1197,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
this.unifiedCache.set(
|
||||
cacheKey,
|
||||
fromMmap,
|
||||
'hnsw',
|
||||
'vectors',
|
||||
fromMmap.length * 4,
|
||||
50
|
||||
)
|
||||
|
|
@ -1232,7 +1232,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
this.unifiedCache.set(
|
||||
cacheKey,
|
||||
loaded,
|
||||
'hnsw', // Type for fairness monitoring
|
||||
'vectors', // Type for fairness monitoring
|
||||
loaded.length * 4, // Size in bytes (float32)
|
||||
50 // Rebuild cost in ms (moderate priority)
|
||||
)
|
||||
|
|
@ -1303,7 +1303,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
const id = uncachedIds[i]
|
||||
const v = fromMmap[i]
|
||||
if (v) {
|
||||
this.unifiedCache.set(`hnsw:vector:${id}`, v, 'hnsw', v.length * 4, 50)
|
||||
this.unifiedCache.set(`vector:${id}`, v, 'vectors', v.length * 4, 50)
|
||||
} else {
|
||||
misses.push(id)
|
||||
}
|
||||
|
|
@ -1322,7 +1322,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
prodLog.debug(`MmapVectorBackend write-back failed for ${id}`, error)
|
||||
}
|
||||
}
|
||||
this.unifiedCache.set(`hnsw:vector:${id}`, vector, 'hnsw', vector.length * 4, 50)
|
||||
this.unifiedCache.set(`vector:${id}`, vector, 'vectors', vector.length * 4, 50)
|
||||
}))
|
||||
return
|
||||
}
|
||||
|
|
@ -1335,7 +1335,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
|
||||
const vector = await this.storage.getNounVector(id)
|
||||
if (vector) {
|
||||
this.unifiedCache.set(cacheKey, vector, 'hnsw', vector.length * 4, 50)
|
||||
this.unifiedCache.set(cacheKey, vector, 'vectors', vector.length * 4, 50)
|
||||
}
|
||||
return vector
|
||||
})
|
||||
|
|
@ -1500,7 +1500,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
for (const nounData of result.items) {
|
||||
try {
|
||||
// Load HNSW graph data for this entity
|
||||
const hnswData = await (this.storage as any).getHNSWData(nounData.id)
|
||||
const hnswData = await (this.storage as any).getVectorIndexData(nounData.id)
|
||||
|
||||
if (!hnswData) {
|
||||
// No HNSW data - skip (might be entity added before persistence)
|
||||
|
|
@ -1584,7 +1584,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
for (const nounData of result.items) {
|
||||
try {
|
||||
// Load HNSW graph data for this entity
|
||||
const hnswData = await (this.storage as any).getHNSWData(nounData.id)
|
||||
const hnswData = await (this.storage as any).getVectorIndexData(nounData.id)
|
||||
|
||||
if (!hnswData) {
|
||||
// No HNSW data - skip (might be entity added before persistence)
|
||||
|
|
@ -1816,12 +1816,12 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
const estimatedVectorMemoryMB = (entityCount * bytesPerVector) / (1024 * 1024)
|
||||
const availableCacheMB = (cacheStats.maxSize * 0.8) / (1024 * 1024) // 80% threshold
|
||||
|
||||
// Calculate HNSW-specific cache stats
|
||||
const vectorsInCache = cacheStats.typeCounts.hnsw || 0
|
||||
const hnswMemoryBytes = cacheStats.typeSizes.hnsw || 0
|
||||
// Calculate vector-index-specific cache stats
|
||||
const vectorsInCache = cacheStats.typeCounts.vectors || 0
|
||||
const hnswMemoryBytes = cacheStats.typeSizes.vectors || 0
|
||||
|
||||
// Calculate fairness metrics
|
||||
const hnswAccessCount = cacheStats.typeAccessCounts.hnsw || 0
|
||||
const hnswAccessCount = cacheStats.typeAccessCounts.vectors || 0
|
||||
const totalAccessCount = cacheStats.totalAccessCount
|
||||
const hnswAccessPercent = totalAccessCount > 0 ? (hnswAccessCount / totalAccessCount) * 100 : 0
|
||||
|
||||
|
|
|
|||
|
|
@ -551,7 +551,7 @@ export class TypeAwareHNSWIndex {
|
|||
const index = this.getIndexForType(nounType as NounType)
|
||||
|
||||
// Load HNSW graph data
|
||||
const hnswData = await (this.storage as any).getHNSWData(nounData.id)
|
||||
const hnswData = await (this.storage as any).getVectorIndexData(nounData.id)
|
||||
if (!hnswData) {
|
||||
continue // No HNSW data
|
||||
}
|
||||
|
|
|
|||
10
src/index.ts
10
src/index.ts
|
|
@ -301,16 +301,6 @@ import { JsHnswVectorIndex } from './hnsw/hnswIndex.js'
|
|||
|
||||
export { JsHnswVectorIndex }
|
||||
|
||||
/**
|
||||
* @deprecated Renamed to {@link JsHnswVectorIndex} in Brainy 8.0. The
|
||||
* implementation is unchanged — the new name disambiguates the JS HNSW
|
||||
* path from any native vector-index provider (DiskANN, etc.) that may be
|
||||
* loaded via the `'vector'` provider key. This alias is provided as a
|
||||
* compatibility shim for code mid-migration and is removed in the 8.0
|
||||
* final cleanup commit.
|
||||
*/
|
||||
export const HNSWIndex = JsHnswVectorIndex
|
||||
|
||||
export type {
|
||||
Vector,
|
||||
VectorDocument,
|
||||
|
|
|
|||
|
|
@ -221,23 +221,6 @@ export interface VectorIndexProvider {
|
|||
getPersistMode(): 'immediate' | 'deferred'
|
||||
}
|
||||
|
||||
/**
|
||||
* @deprecated Renamed to {@link VectorIndexProvider} in Brainy 8.0. The
|
||||
* interface shape is byte-for-byte identical; only the name changed because
|
||||
* "vector index" describes the role, while "HNSW" was the name of one
|
||||
* specific implementation. This alias is provided as a compatibility shim
|
||||
* for code mid-migration; new code should import `VectorIndexProvider`.
|
||||
*/
|
||||
export type HnswProvider = VectorIndexProvider
|
||||
|
||||
/**
|
||||
* @deprecated The `'diskann'` provider key has been folded into `'vector'` in
|
||||
* Brainy 8.0. The vector index contract is implementation-neutral: any provider
|
||||
* registered under `'vector'` may be HNSW, DiskANN, or a future algorithm. The
|
||||
* type alias is preserved so existing imports compile while consumers migrate
|
||||
* to `VectorIndexProvider`.
|
||||
*/
|
||||
export type DiskAnnProvider = VectorIndexProvider
|
||||
|
||||
/**
|
||||
* The `'entityIdMapper'` provider — a drop-in for `EntityIdMapper`. Injected
|
||||
|
|
@ -265,10 +248,10 @@ export interface EntityIdMapperProvider {
|
|||
*/
|
||||
export interface CacheProvider {
|
||||
getSync(key: string): any | undefined
|
||||
set(key: string, data: any, type: 'hnsw' | 'metadata' | 'embedding' | 'other', size: number, rebuildCost?: number): void
|
||||
set(key: string, data: any, type: 'vectors' | 'metadata' | 'embedding' | 'other', size: number, rebuildCost?: number): void
|
||||
delete(key: string): boolean
|
||||
deleteByPrefix(prefix: string): number
|
||||
clear(type?: 'hnsw' | 'metadata' | 'embedding' | 'other'): void
|
||||
clear(type?: 'vectors' | 'metadata' | 'embedding' | 'other'): void
|
||||
}
|
||||
|
||||
// The `'embeddings'` / `'embedBatch'` providers are function-shaped and already
|
||||
|
|
@ -287,7 +270,7 @@ export interface CacheProvider {
|
|||
* is fetched + decoded back into UUID sets — `setConnectionsCodec()` on
|
||||
* `JsHnswVectorIndex` is the injection point. Read path is dual-format: when no blob
|
||||
* exists for a node, the connections fall back to the legacy JSON-array path
|
||||
* embedded in `saveHNSWData`, so pre-2.4.0 indexes keep loading unchanged
|
||||
* embedded in `saveVectorIndexData`, so pre-2.4.0 indexes keep loading unchanged
|
||||
* and convergence to the compressed form happens lazily on next save.
|
||||
*
|
||||
* Activated only when the storage adapter exposes the binary-blob primitive
|
||||
|
|
|
|||
|
|
@ -1768,7 +1768,7 @@ export class AzureBlobStorage extends BaseStorage {
|
|||
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
|
||||
* CRITICAL: Uses mutex locking to prevent read-modify-write races
|
||||
*/
|
||||
public async saveHNSWData(nounId: string, hnswData: {
|
||||
public async saveVectorIndexData(nounId: string, hnswData: {
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
}): Promise<void> {
|
||||
|
|
@ -1829,7 +1829,7 @@ export class AzureBlobStorage extends BaseStorage {
|
|||
* Get HNSW graph data for a noun
|
||||
* Uses BaseStorage's getNoun (type-first paths)
|
||||
*/
|
||||
public async getHNSWData(nounId: string): Promise<{
|
||||
public async getVectorIndexData(nounId: string): Promise<{
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
} | null> {
|
||||
|
|
|
|||
|
|
@ -95,49 +95,24 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
|
||||
abstract getVerbMetadata(id: string): Promise<any | null>
|
||||
|
||||
// Vector Index Persistence (Brainy 8.0 — algorithm-neutral surface)
|
||||
// The legacy `saveHNSWData` / `getHNSWData` names persist as default-method
|
||||
// aliases so existing implementations continue to work mid-migration; the
|
||||
// 8.0 final cleanup commit removes the old names.
|
||||
// Vector Index Persistence (Brainy 8.0 — algorithm-neutral surface).
|
||||
// Concrete adapters persist the per-entity HNSW graph node (level + connections)
|
||||
// under these methods. Cortex's DiskANN-style native vector index doesn't use
|
||||
// them (it persists its own single mmap'd `.dkann` file under
|
||||
// `_system/vector-index/<shard>.dkann` per BRAINY-8.0-RENAME-COORDINATION § B.2).
|
||||
|
||||
abstract getNounVector(id: string): Promise<number[] | null>
|
||||
|
||||
abstract saveHNSWData(nounId: string, hnswData: {
|
||||
abstract saveVectorIndexData(nounId: string, data: {
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
}): Promise<void>
|
||||
|
||||
abstract getHNSWData(nounId: string): Promise<{
|
||||
abstract getVectorIndexData(nounId: string): Promise<{
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
} | null>
|
||||
|
||||
/**
|
||||
* Persist vector-index graph state for an entity (Brainy 8.0 name).
|
||||
* Default implementation delegates to {@link saveHNSWData}; concrete
|
||||
* adapters may override directly. The final 8.0 cleanup removes the
|
||||
* legacy `saveHNSWData` name.
|
||||
*/
|
||||
async saveVectorIndexData(nounId: string, data: {
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
}): Promise<void> {
|
||||
return this.saveHNSWData(nounId, data)
|
||||
}
|
||||
|
||||
/**
|
||||
* Read vector-index graph state for an entity (Brainy 8.0 name).
|
||||
* Default implementation delegates to {@link getHNSWData}; concrete
|
||||
* adapters may override directly. The final 8.0 cleanup removes the
|
||||
* legacy `getHNSWData` name.
|
||||
*/
|
||||
async getVectorIndexData(nounId: string): Promise<{
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
} | null> {
|
||||
return this.getHNSWData(nounId)
|
||||
}
|
||||
|
||||
abstract saveHNSWSystem(systemData: {
|
||||
entryPointId: string | null
|
||||
maxLevel: number
|
||||
|
|
|
|||
|
|
@ -2810,7 +2810,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
|
||||
* CRITICAL: Preserves mutex locking to prevent read-modify-write races
|
||||
*/
|
||||
public async saveHNSWData(nounId: string, hnswData: {
|
||||
public async saveVectorIndexData(nounId: string, hnswData: {
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
}): Promise<void> {
|
||||
|
|
@ -2871,7 +2871,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
* Get HNSW graph data for a noun
|
||||
* Uses BaseStorage's getNoun (type-first paths)
|
||||
*/
|
||||
public async getHNSWData(nounId: string): Promise<{
|
||||
public async getVectorIndexData(nounId: string): Promise<{
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
} | null> {
|
||||
|
|
|
|||
|
|
@ -1736,7 +1736,7 @@ export class GcsStorage extends BaseStorage {
|
|||
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
|
||||
* CRITICAL: Uses mutex locking to prevent read-modify-write races
|
||||
*/
|
||||
public async saveHNSWData(nounId: string, hnswData: {
|
||||
public async saveVectorIndexData(nounId: string, hnswData: {
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
}): Promise<void> {
|
||||
|
|
@ -1797,7 +1797,7 @@ export class GcsStorage extends BaseStorage {
|
|||
* Get HNSW graph data for a noun
|
||||
* Uses BaseStorage's getNoun (type-first paths)
|
||||
*/
|
||||
public async getHNSWData(nounId: string): Promise<{
|
||||
public async getVectorIndexData(nounId: string): Promise<{
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
} | null> {
|
||||
|
|
|
|||
|
|
@ -462,7 +462,7 @@ export class HistoricalStorageAdapter extends BaseStorage {
|
|||
/**
|
||||
* WRITE BLOCKED: Historical storage is read-only
|
||||
*/
|
||||
public async saveHNSWData(nounId: string, hnswData: {
|
||||
public async saveVectorIndexData(nounId: string, hnswData: {
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
}): Promise<void> {
|
||||
|
|
@ -492,7 +492,7 @@ export class HistoricalStorageAdapter extends BaseStorage {
|
|||
/**
|
||||
* Get HNSW data from historical state
|
||||
*/
|
||||
public async getHNSWData(nounId: string): Promise<{
|
||||
public async getVectorIndexData(nounId: string): Promise<{
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
} | null> {
|
||||
|
|
|
|||
|
|
@ -415,7 +415,7 @@ export class MemoryStorage extends BaseStorage {
|
|||
* Even in-memory operations can race due to async/await interleaving
|
||||
* Prevents data corruption when multiple entities connect to same neighbor simultaneously
|
||||
*/
|
||||
public async saveHNSWData(nounId: string, hnswData: {
|
||||
public async saveVectorIndexData(nounId: string, hnswData: {
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
}): Promise<void> {
|
||||
|
|
@ -458,7 +458,7 @@ export class MemoryStorage extends BaseStorage {
|
|||
/**
|
||||
* Get HNSW graph data for a noun
|
||||
*/
|
||||
public async getHNSWData(nounId: string): Promise<{
|
||||
public async getVectorIndexData(nounId: string): Promise<{
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
} | null> {
|
||||
|
|
|
|||
|
|
@ -1453,7 +1453,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
|
||||
* CRITICAL: Preserves mutex locking to prevent read-modify-write races
|
||||
*/
|
||||
public async saveHNSWData(nounId: string, hnswData: {
|
||||
public async saveVectorIndexData(nounId: string, hnswData: {
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
}): Promise<void> {
|
||||
|
|
@ -1503,7 +1503,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
* Get HNSW graph data for a noun
|
||||
* Uses BaseStorage's getNoun (type-first paths)
|
||||
*/
|
||||
public async getHNSWData(nounId: string): Promise<{
|
||||
public async getVectorIndexData(nounId: string): Promise<{
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
} | null> {
|
||||
|
|
|
|||
|
|
@ -1090,7 +1090,7 @@ export class R2Storage extends BaseStorage {
|
|||
return noun ? noun.vector : null
|
||||
}
|
||||
|
||||
public async saveHNSWData(nounId: string, hnswData: {
|
||||
public async saveVectorIndexData(nounId: string, hnswData: {
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
}): Promise<void> {
|
||||
|
|
@ -1130,7 +1130,7 @@ export class R2Storage extends BaseStorage {
|
|||
}
|
||||
}
|
||||
|
||||
public async getHNSWData(nounId: string): Promise<{
|
||||
public async getVectorIndexData(nounId: string): Promise<{
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
} | null> {
|
||||
|
|
|
|||
|
|
@ -3723,7 +3723,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
|
||||
* CRITICAL: Uses mutex locking to prevent read-modify-write races
|
||||
*/
|
||||
public async saveHNSWData(nounId: string, hnswData: {
|
||||
public async saveVectorIndexData(nounId: string, hnswData: {
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
}): Promise<void> {
|
||||
|
|
@ -3784,7 +3784,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
* Get HNSW graph data for a noun
|
||||
* Uses BaseStorage's getNoun (type-first paths)
|
||||
*/
|
||||
public async getHNSWData(nounId: string): Promise<{
|
||||
public async getVectorIndexData(nounId: string): Promise<{
|
||||
level: number
|
||||
connections: Record<string, string[]>
|
||||
} | null> {
|
||||
|
|
|
|||
|
|
@ -1031,9 +1031,7 @@ export interface BrainyStats {
|
|||
fieldRegistry: string[]
|
||||
/** Per-index health flags. `true` = index has entries OR no entities exist yet. */
|
||||
indexHealth: {
|
||||
/** @deprecated 8.0 — renamed to `vector`. Same boolean; kept as a compat alias until the final 8.0 cleanup. */
|
||||
hnsw: boolean
|
||||
/** Vector index health (8.0 name — open-core JS HNSW path or a native acceleration provider). */
|
||||
/** Vector index health (8.0 — open-core JS HNSW path or a native acceleration provider). */
|
||||
vector: boolean
|
||||
metadata: boolean
|
||||
graph: boolean
|
||||
|
|
@ -1119,19 +1117,6 @@ export interface BrainyConfig {
|
|||
// Cloud storage (GCS/S3/R2/Azure) should use 'deferred' for 30-50× faster adds
|
||||
hnswPersistMode?: 'immediate' | 'deferred'
|
||||
|
||||
// HNSW optimization options (v7.11.0)
|
||||
// @deprecated 8.0 — use `vector` instead. This field is kept as a compat
|
||||
// shim during the 8.0 rename scaffolding; values are merged into the new
|
||||
// `vector` block with explicit `vector.*` values winning on conflict.
|
||||
hnsw?: {
|
||||
quantization?: {
|
||||
enabled?: boolean // default: false — current behavior exactly
|
||||
bits?: 8 | 4 // default: 8 (SQ8). SQ4 has a pure-JS implementation; cortex's distance:sq4 SIMD provider accelerates it.
|
||||
rerankMultiplier?: number // default: 3 — over-retrieve 3x, rerank with float32
|
||||
}
|
||||
vectorStorage?: 'memory' | 'lazy' // default: 'memory' — 'lazy' evicts vectors after insert
|
||||
}
|
||||
|
||||
/**
|
||||
* Vector index configuration (Brainy 8.0).
|
||||
*
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ import type { CacheProvider } from '../plugin.js'
|
|||
|
||||
export interface CacheItem {
|
||||
key: string
|
||||
type: 'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other'
|
||||
type: 'vectors' | 'metadata' | 'embedding' | 'other'
|
||||
data: any
|
||||
size: number
|
||||
rebuildCost: number // milliseconds to rebuild
|
||||
|
|
@ -69,7 +69,7 @@ export class UnifiedCache implements CacheProvider {
|
|||
private cache = new Map<string, CacheItem>()
|
||||
private access = new Map<string, number>() // Access counts
|
||||
private loadingPromises = new Map<string, Promise<any>>()
|
||||
private typeAccessCounts = { hnsw: 0, vectors: 0, metadata: 0, embedding: 0, other: 0 }
|
||||
private typeAccessCounts = { vectors: 0, metadata: 0, embedding: 0, other: 0 }
|
||||
private totalAccessCount = 0
|
||||
private currentSize = 0
|
||||
private maxSize: number
|
||||
|
|
@ -209,7 +209,7 @@ export class UnifiedCache implements CacheProvider {
|
|||
set(
|
||||
key: string,
|
||||
data: any,
|
||||
type: 'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other',
|
||||
type: 'vectors' | 'metadata' | 'embedding' | 'other',
|
||||
size: number,
|
||||
rebuildCost: number = 1
|
||||
): void {
|
||||
|
|
@ -323,8 +323,8 @@ export class UnifiedCache implements CacheProvider {
|
|||
|
||||
private checkFairness(): void {
|
||||
// Calculate type ratios in cache
|
||||
const typeSizes = { hnsw: 0, vectors: 0, metadata: 0, embedding: 0, other: 0 }
|
||||
const typeCounts = { hnsw: 0, vectors: 0, metadata: 0, embedding: 0, other: 0 }
|
||||
const typeSizes = { vectors: 0, metadata: 0, embedding: 0, other: 0 }
|
||||
const typeCounts = { vectors: 0, metadata: 0, embedding: 0, other: 0 }
|
||||
|
||||
for (const item of this.cache.values()) {
|
||||
typeSizes[item.type] += item.size
|
||||
|
|
@ -334,7 +334,6 @@ export class UnifiedCache implements CacheProvider {
|
|||
// Calculate access ratios
|
||||
const totalAccess = this.totalAccessCount || 1
|
||||
const accessRatios = {
|
||||
hnsw: this.typeAccessCounts.hnsw / totalAccess,
|
||||
vectors: this.typeAccessCounts.vectors / totalAccess,
|
||||
metadata: this.typeAccessCounts.metadata / totalAccess,
|
||||
embedding: this.typeAccessCounts.embedding / totalAccess,
|
||||
|
|
@ -344,7 +343,6 @@ export class UnifiedCache implements CacheProvider {
|
|||
// Calculate size ratios
|
||||
const totalSize = this.currentSize || 1
|
||||
const sizeRatios = {
|
||||
hnsw: typeSizes.hnsw / totalSize,
|
||||
vectors: typeSizes.vectors / totalSize,
|
||||
metadata: typeSizes.metadata / totalSize,
|
||||
embedding: typeSizes.embedding / totalSize,
|
||||
|
|
@ -353,7 +351,7 @@ export class UnifiedCache implements CacheProvider {
|
|||
|
||||
// Check for starvation (more aggressive - 70% cache with <15% accesses)
|
||||
// Previous: 90% cache, <10% access (too lenient, caused thrashing)
|
||||
for (const type of ['hnsw', 'vectors', 'metadata', 'embedding', 'other'] as const) {
|
||||
for (const type of ['vectors', 'metadata', 'embedding', 'other'] as const) {
|
||||
if (sizeRatios[type] > 0.7 && accessRatios[type] < 0.15) {
|
||||
prodLog.warn(`Type ${type} is hogging cache (${(sizeRatios[type] * 100).toFixed(1)}% size, ${(accessRatios[type] * 100).toFixed(1)}% access)`)
|
||||
this.evictType(type)
|
||||
|
|
@ -366,7 +364,7 @@ export class UnifiedCache implements CacheProvider {
|
|||
* Called immediately when adding items to prevent imbalance formation
|
||||
* Uses same thresholds as periodic check but runs on-demand
|
||||
*/
|
||||
private checkProactiveFairness(addedType: 'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other'): void {
|
||||
private checkProactiveFairness(addedType: 'vectors' | 'metadata' | 'embedding' | 'other'): void {
|
||||
// Quick check: only evaluate the type being added
|
||||
let typeSize = 0
|
||||
for (const item of this.cache.values()) {
|
||||
|
|
@ -388,7 +386,7 @@ export class UnifiedCache implements CacheProvider {
|
|||
/**
|
||||
* Force evict items of a specific type
|
||||
*/
|
||||
private evictType(type: 'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other'): void {
|
||||
private evictType(type: 'vectors' | 'metadata' | 'embedding' | 'other'): void {
|
||||
const candidates: Array<[string, number, CacheItem]> = []
|
||||
|
||||
for (const [key, item] of this.cache) {
|
||||
|
|
@ -468,7 +466,7 @@ export class UnifiedCache implements CacheProvider {
|
|||
/**
|
||||
* Clear cache or specific type
|
||||
*/
|
||||
clear(type?: 'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other'): void {
|
||||
clear(type?: 'vectors' | 'metadata' | 'embedding' | 'other'): void {
|
||||
if (!type) {
|
||||
this.cache.clear()
|
||||
this.currentSize = 0
|
||||
|
|
@ -535,8 +533,8 @@ export class UnifiedCache implements CacheProvider {
|
|||
* Get cache statistics with memory information
|
||||
*/
|
||||
getStats() {
|
||||
const typeSizes = { hnsw: 0, vectors: 0, metadata: 0, embedding: 0, other: 0 }
|
||||
const typeCounts = { hnsw: 0, vectors: 0, metadata: 0, embedding: 0, other: 0 }
|
||||
const typeSizes = { vectors: 0, metadata: 0, embedding: 0, other: 0 }
|
||||
const typeCounts = { vectors: 0, metadata: 0, embedding: 0, other: 0 }
|
||||
|
||||
for (const item of this.cache.values()) {
|
||||
typeSizes[item.type] += item.size
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ describe('UnifiedCache Eviction Scoring', () => {
|
|||
cache.set('meta1', { data: 'metadata' }, 'metadata', 100, 1)
|
||||
|
||||
// Add HNSW vector (high rebuild cost, high access count)
|
||||
cache.set('hnsw1', { data: 'vector' }, 'hnsw', 100, 50)
|
||||
cache.set('hnsw1', { data: 'vector' }, 'vectors', 100, 50)
|
||||
|
||||
// Simulate frequent access to HNSW
|
||||
for (let i = 0; i < 10; i++) {
|
||||
|
|
@ -41,7 +41,7 @@ describe('UnifiedCache Eviction Scoring', () => {
|
|||
|
||||
// Metadata should likely be evicted (low value)
|
||||
// Note: Since we're filling most of cache, metadata is candidate for eviction
|
||||
expect(stats.typeSizes.hnsw).toBeGreaterThan(0)
|
||||
expect(stats.typeSizes.vectors).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('should prioritize frequently accessed items regardless of type', () => {
|
||||
|
|
@ -52,7 +52,7 @@ describe('UnifiedCache Eviction Scoring', () => {
|
|||
}
|
||||
|
||||
// Add item with low access count, high rebuild cost
|
||||
cache.set('cold1', { data: 'cold' }, 'hnsw', 100, 50)
|
||||
cache.set('cold1', { data: 'cold' }, 'vectors', 100, 50)
|
||||
// Only access once (implicit from set)
|
||||
|
||||
// Fill cache
|
||||
|
|
@ -112,7 +112,7 @@ describe('UnifiedCache Eviction Scoring', () => {
|
|||
}
|
||||
|
||||
// Add one HNSW with high value
|
||||
cache.set('hnsw1', { data: 'vector' }, 'hnsw', 100, 50)
|
||||
cache.set('hnsw1', { data: 'vector' }, 'vectors', 100, 50)
|
||||
for (let i = 0; i < 50; i++) {
|
||||
cache.getSync('hnsw1')
|
||||
}
|
||||
|
|
@ -196,8 +196,8 @@ describe('UnifiedCache Eviction Scoring', () => {
|
|||
}
|
||||
|
||||
// Add HNSW vectors (expensive to rebuild)
|
||||
cache.set('hnsw1', { vector: [1, 2, 3] }, 'hnsw', 100, 50)
|
||||
cache.set('hnsw2', { vector: [4, 5, 6] }, 'hnsw', 100, 50)
|
||||
cache.set('hnsw1', { vector: [1, 2, 3] }, 'vectors', 100, 50)
|
||||
cache.set('hnsw2', { vector: [4, 5, 6] }, 'vectors', 100, 50)
|
||||
|
||||
// High access to HNSW
|
||||
for (let i = 0; i < 100; i++) {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue