fix: populate totalNodes/totalEdges in ALL storage adapters for HNSW rebuild

Critical fix for v3.37.1 GCS persistence bug where HNSW rebuild reported
"Preloading 0 vectors" and hung indefinitely during container restarts.

Root Cause:
HNSW rebuild (hnswIndex.ts:835) calls storage.getStatistics() and expects
stats.totalNodes to determine entity count for adaptive caching. When
totalNodes was undefined, entityCount evaluated to 0, causing HNSW to
report "0 vectors" and hang waiting for entities that it couldn't find.

Fixes Applied:
- GcsStorage: Populate totalNodes/totalEdges from this.totalNounCount/totalVerbCount
- MemoryStorage: Populate totalNodes/totalEdges from in-memory counts
- OPFSStorage: Populate totalNodes/totalEdges in ALL return paths (cache, current day, yesterday, legacy)
- S3CompatibleStorage: Populate totalNodes/totalEdges in ALL return paths (cache, fresh stats, error fallback)

Impact:
-  GCS container restarts now work correctly
-  HNSW rebuild correctly detects entity count
-  Adaptive caching strategy works as designed
-  All storage adapters now consistent

Files exist in GCS, counts load correctly, but HNSW couldn't see them
because getStatisticsData() didn't populate the totalNodes field that
HNSW depends on.

Closes: BRAINY_V3_37_1_INDEX_REBUILD_BUG.md
Related: v3.37.0 (metadata reconstruction), v3.37.1 (getNoun_internal fix)
This commit is contained in:
David Snelling 2025-10-10 17:48:40 -07:00
parent 9fe790e0a7
commit a21a84545a
4 changed files with 58 additions and 10 deletions

View file

@ -691,6 +691,10 @@ export class MemoryStorage extends BaseStorage {
verbCount: {...this.statistics.verbCount},
metadataCount: {...this.statistics.metadataCount},
hnswIndexSize: this.statistics.hnswIndexSize,
// CRITICAL FIX: Populate totalNodes and totalEdges from in-memory counts
// HNSW rebuild depends on these fields to determine entity count
totalNodes: this.totalNounCount,
totalEdges: this.totalVerbCount,
lastUpdated: this.statistics.lastUpdated,
// Include serviceActivity if present
...(this.statistics.serviceActivity && {
@ -703,11 +707,11 @@ export class MemoryStorage extends BaseStorage {
services: this.statistics.services.map(s => ({...s}))
}),
// Include distributedConfig if present
...(this.statistics.distributedConfig && {
distributedConfig: JSON.parse(JSON.stringify(this.statistics.distributedConfig))
...(this.statistics.distributedConfig && {
distributedConfig: JSON.parse(JSON.stringify(this.statistics.distributedConfig))
})
}
// Since this is in-memory, there's no need for fallback mechanisms
// to check multiple storage locations
}