perf: defer HNSW persistence during addMany() batch operations

addMany() now temporarily switches the HNSW index to deferred persist
mode during the batch loop. Previously, each add() triggered ~16-20
saveHNSWData calls for modified neighbors (each a read→gzip→atomic-write
cycle). For 450 items this was ~8,100 individual storage writes.

With deferred mode, dirty node IDs are collected in a Set during the
batch and flushed once at the end — deduplicating repeated neighbor
updates. A node modified by insert #3 and again by insert #47 is
written only once.

Also adds setPersistMode() to TypeAwareHNSWIndex, propagating mode
changes to all existing type-specific sub-indexes.
This commit is contained in:
David Snelling 2026-03-22 14:53:11 -07:00
parent c054c41943
commit 973b6aaa39
2 changed files with 80 additions and 42 deletions

View file

@ -163,6 +163,19 @@ export class TypeAwareHNSWIndex {
return this.persistMode
}
/**
* Switch persist mode at runtime. Propagates to all existing type-specific indexes.
* Used by addMany() to defer persistence during batch operations.
*/
public setPersistMode(mode: 'immediate' | 'deferred'): void {
this.persistMode = mode
for (const index of this.indexes.values()) {
if (typeof (index as any).setPersistMode === 'function') {
(index as any).setPersistMode(mode)
}
}
}
/**
* Get or create HNSW index for a specific type (lazy initialization)
*