fix: resolve critical count persistence bugs affecting container restarts
Fixes two critical production bugs in GCS storage adapter:
1. brain.find({ where: {...} }) returned empty array after restart
- Root cause: Counts only persisted every 10 operations
- If <10 entities added before restart, counts were lost
- After restart: totalNounCount = 0, causing empty results
2. brain.init() returned 0 entities after container restart
- Same root cause as bug #1
- Counts file never written for small datasets
- getStats() returned 0 despite data in GCS bucket
Changes:
- baseStorageAdapter.ts: Persist counts on EVERY operation (not every 10)
- incrementEntityCountSafe(): Now persists immediately
- decrementEntityCountSafe(): Now persists immediately
- incrementVerbCount(): Now persists immediately
- decrementVerbCount(): Now persists immediately
- gcsStorage.ts: Better error handling for count initialization
- initializeCounts(): Fail loudly on network/permission errors
- initializeCountsFromScan(): Throw on scan failures instead of silent fail
- Added recovery logic with bucket scan fallback
Impact: Critical for serverless/containerized deployments (Cloud Run, Fargate, Lambda)
where containers restart frequently. The basic write→restart→read scenario now works.
This commit is contained in:
parent
2ec7536333
commit
39525aff26
2 changed files with 28 additions and 23 deletions
|
|
@ -923,10 +923,10 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
const mutex = getGlobalMutex()
|
||||
await mutex.runExclusive(`count-entity-${type}`, async () => {
|
||||
this.incrementEntityCount(type)
|
||||
// Persist counts periodically
|
||||
if (this.totalNounCount % 10 === 0) {
|
||||
await this.persistCounts()
|
||||
}
|
||||
// CRITICAL FIX: Persist counts on EVERY change for cloud storage adapters
|
||||
// This ensures counts survive container restarts (GCS, S3, etc.)
|
||||
// For memory/file storage, this is fast; for cloud storage, it's essential
|
||||
await this.persistCounts()
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -958,9 +958,8 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
const mutex = getGlobalMutex()
|
||||
await mutex.runExclusive(`count-entity-${type}`, async () => {
|
||||
this.decrementEntityCount(type)
|
||||
if (this.totalNounCount % 10 === 0) {
|
||||
await this.persistCounts()
|
||||
}
|
||||
// CRITICAL FIX: Persist counts on EVERY change for cloud storage adapters
|
||||
await this.persistCounts()
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -979,10 +978,8 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
timestamp: Date.now()
|
||||
})
|
||||
|
||||
// Persist counts immediately for consistency
|
||||
if (this.totalVerbCount % 10 === 0) {
|
||||
await this.persistCounts()
|
||||
}
|
||||
// CRITICAL FIX: Persist counts on EVERY change for cloud storage adapters
|
||||
await this.persistCounts()
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -1008,10 +1005,8 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
timestamp: Date.now()
|
||||
})
|
||||
|
||||
// Persist counts immediately for consistency
|
||||
if (this.totalVerbCount % 10 === 0) {
|
||||
await this.persistCounts()
|
||||
}
|
||||
// CRITICAL FIX: Persist counts on EVERY change for cloud storage adapters
|
||||
await this.persistCounts()
|
||||
})
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue