fix(storage): resolve persistence restart bug across all storage adapters
Critical bug fix that restores data persistence after application restart for all storage adapters (GCS, S3, OPFS, FileSystem). **Root Cause:** Storage adapters were loading entity counts from _system/counts.json but not returning totalCount in pagination methods, causing rebuildIndexesIfNeeded() to see undefined and incorrectly assume no data existed. **Changes:** - GCS Storage: Return totalCount in getNodesWithPagination() and getVerbsWithPagination() - S3 Storage: Call initializeCounts() in init() and return totalCount in pagination methods - OPFS Storage: Call initializeCounts() in init() to load pre-calculated counts - BaseStorage: Add defensive validation to prevent future adapters from missing totalCount **Impact:** - Production deployments now work correctly (Cloud Run, AWS, Kubernetes) - Data persists correctly across container restarts - Serverless scale-to-zero deployments now functional - All storage adapters use O(1) pre-calculated counts (scales to millions of entities) **Testing:** - Build passes with no TypeScript errors - All existing tests pass - Defensive validation ensures future storage adapters won't have this bug Fixes critical production issue affecting all persistent storage deployments.
This commit is contained in:
parent
c502b56bb4
commit
01e89fffca
4 changed files with 40 additions and 17 deletions
|
|
@ -346,7 +346,10 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
|
||||
// Auto-cleanup legacy /index folder on initialization
|
||||
await this.cleanupLegacyIndexFolder()
|
||||
|
||||
|
||||
// Initialize counts from storage
|
||||
await this.initializeCounts()
|
||||
|
||||
this.isInitialized = true
|
||||
this.logger.info(`Initialized ${this.serviceType} storage with bucket ${this.bucketName}`)
|
||||
} catch (error) {
|
||||
|
|
@ -1843,6 +1846,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
|
||||
return {
|
||||
items: filteredGraphVerbs,
|
||||
totalCount: this.totalVerbCount, // Use pre-calculated count from init()
|
||||
hasMore: result.hasMore,
|
||||
nextCursor: result.nextCursor
|
||||
}
|
||||
|
|
@ -3367,21 +3371,9 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
}
|
||||
}
|
||||
|
||||
// Calculate total count efficiently
|
||||
// For the first page (no cursor), we can estimate total count
|
||||
let totalCount: number | undefined
|
||||
if (!cursor) {
|
||||
try {
|
||||
totalCount = await this.estimateTotalNounCount()
|
||||
} catch (error) {
|
||||
this.logger.warn('Failed to estimate total noun count:', error)
|
||||
// totalCount remains undefined
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
items: filteredNodes,
|
||||
totalCount,
|
||||
totalCount: this.totalNounCount, // Use pre-calculated count from init()
|
||||
hasMore: result.hasMore,
|
||||
nextCursor: result.nextCursor
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue