fix(storage): resolve persistence restart bug across all storage adapters
Critical bug fix that restores data persistence after application restart for all storage adapters (GCS, S3, OPFS, FileSystem). **Root Cause:** Storage adapters were loading entity counts from _system/counts.json but not returning totalCount in pagination methods, causing rebuildIndexesIfNeeded() to see undefined and incorrectly assume no data existed. **Changes:** - GCS Storage: Return totalCount in getNodesWithPagination() and getVerbsWithPagination() - S3 Storage: Call initializeCounts() in init() and return totalCount in pagination methods - OPFS Storage: Call initializeCounts() in init() to load pre-calculated counts - BaseStorage: Add defensive validation to prevent future adapters from missing totalCount **Impact:** - Production deployments now work correctly (Cloud Run, AWS, Kubernetes) - Data persists correctly across container restarts - Serverless scale-to-zero deployments now functional - All storage adapters use O(1) pre-calculated counts (scales to millions of entities) **Testing:** - Build passes with no TypeScript errors - All existing tests pass - Defensive validation ensures future storage adapters won't have this bug Fixes critical production issue affecting all persistent storage deployments.
This commit is contained in:
parent
c502b56bb4
commit
01e89fffca
4 changed files with 40 additions and 17 deletions
|
|
@ -958,7 +958,7 @@ export class GcsStorage extends BaseStorage {
|
|||
useCache?: boolean
|
||||
}): Promise<{
|
||||
nodes: HNSWNode[]
|
||||
totalCount?: number
|
||||
totalCount: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
}> {
|
||||
|
|
@ -1029,6 +1029,7 @@ export class GcsStorage extends BaseStorage {
|
|||
|
||||
return {
|
||||
nodes,
|
||||
totalCount: this.totalNounCount,
|
||||
hasMore: !!nextCursor,
|
||||
nextCursor
|
||||
}
|
||||
|
|
@ -1038,6 +1039,7 @@ export class GcsStorage extends BaseStorage {
|
|||
if (response?.nextPageToken) {
|
||||
return {
|
||||
nodes,
|
||||
totalCount: this.totalNounCount,
|
||||
hasMore: true,
|
||||
nextCursor: `${shardIndex}:${response.nextPageToken}`
|
||||
}
|
||||
|
|
@ -1049,6 +1051,7 @@ export class GcsStorage extends BaseStorage {
|
|||
// No more shards or nodes
|
||||
return {
|
||||
nodes,
|
||||
totalCount: this.totalNounCount,
|
||||
hasMore: false,
|
||||
nextCursor: undefined
|
||||
}
|
||||
|
|
@ -1220,6 +1223,7 @@ export class GcsStorage extends BaseStorage {
|
|||
|
||||
return {
|
||||
items: filteredVerbs,
|
||||
totalCount: this.totalVerbCount,
|
||||
hasMore: !!response?.nextPageToken,
|
||||
nextCursor: response?.nextPageToken
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue