perf(hnsw): deferred persistence mode for 30-50× faster cloud storage adds
Problem: Each add() triggered 70 GCS operations (34 reads + 36 writes) because HNSW updates 16+ neighbors per add, and each neighbor did a read-modify-write cycle. Result: 7-11 seconds per add() on GCS. Solution: - Add `hnswPersistMode: 'immediate' | 'deferred'` config option - In deferred mode, track dirty nodes instead of persisting immediately - Flush dirty nodes on close() or explicit flush() - Smart defaults: cloud storage (GCS/S3/R2/Azure) = deferred, local = immediate Performance impact: - Single add(): 7-11 seconds → 200-400ms (30-50× faster) - GCS operations per add: 70 → 2-3 Zero configuration - cloud storage automatically uses deferred mode. Files changed: - src/types/brainy.types.ts: Add hnswPersistMode config option - src/hnsw/hnswIndex.ts: Deferred mode, dirty tracking, flush() - src/hnsw/typeAwareHNSWIndex.ts: Propagate to child indexes - src/brainy.ts: Smart defaults, flush on close() 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
a33b759c40
commit
4d1d567236
4 changed files with 208 additions and 20 deletions
|
|
@ -681,6 +681,13 @@ export interface BrainyConfig {
|
|||
batchWrites?: boolean // Enable write batching for better performance
|
||||
maxConcurrentOperations?: number // Limit concurrent file operations
|
||||
|
||||
// HNSW persistence mode (v6.2.8)
|
||||
// Controls when HNSW graph connections are persisted to storage
|
||||
// - 'immediate': Persist on every add (slow but durable, default for filesystem)
|
||||
// - 'deferred': Persist only on flush/close (fast, default for cloud storage)
|
||||
// Cloud storage (GCS/S3/R2/Azure) should use 'deferred' for 30-50× faster adds
|
||||
hnswPersistMode?: 'immediate' | 'deferred'
|
||||
|
||||
// Memory management options (v5.11.0)
|
||||
maxQueryLimit?: number // Override auto-detected query result limit (max: 100000)
|
||||
reservedQueryMemory?: number // Memory reserved for queries in bytes (e.g., 1073741824 = 1GB)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue