fix: resolve critical count persistence bugs affecting container restarts

Fixes two critical production bugs in GCS storage adapter:

1. brain.find({ where: {...} }) returned empty array after restart
   - Root cause: Counts only persisted every 10 operations
   - If <10 entities added before restart, counts were lost
   - After restart: totalNounCount = 0, causing empty results

2. brain.init() returned 0 entities after container restart
   - Same root cause as bug #1
   - Counts file never written for small datasets
   - getStats() returned 0 despite data in GCS bucket

Changes:
- baseStorageAdapter.ts: Persist counts on EVERY operation (not every 10)
  - incrementEntityCountSafe(): Now persists immediately
  - decrementEntityCountSafe(): Now persists immediately
  - incrementVerbCount(): Now persists immediately
  - decrementVerbCount(): Now persists immediately

- gcsStorage.ts: Better error handling for count initialization
  - initializeCounts(): Fail loudly on network/permission errors
  - initializeCountsFromScan(): Throw on scan failures instead of silent fail
  - Added recovery logic with bucket scan fallback

Impact: Critical for serverless/containerized deployments (Cloud Run, Fargate, Lambda)
where containers restart frequently. The basic write→restart→read scenario now works.
This commit is contained in:
David Snelling 2025-10-09 17:12:06 -07:00
parent 2ec7536333
commit 39525aff26
2 changed files with 28 additions and 23 deletions

View file

@ -923,10 +923,10 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex()
await mutex.runExclusive(`count-entity-${type}`, async () => {
this.incrementEntityCount(type)
// Persist counts periodically
if (this.totalNounCount % 10 === 0) {
await this.persistCounts()
}
// CRITICAL FIX: Persist counts on EVERY change for cloud storage adapters
// This ensures counts survive container restarts (GCS, S3, etc.)
// For memory/file storage, this is fast; for cloud storage, it's essential
await this.persistCounts()
})
}
@ -958,9 +958,8 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex()
await mutex.runExclusive(`count-entity-${type}`, async () => {
this.decrementEntityCount(type)
if (this.totalNounCount % 10 === 0) {
await this.persistCounts()
}
// CRITICAL FIX: Persist counts on EVERY change for cloud storage adapters
await this.persistCounts()
})
}
@ -979,10 +978,8 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
timestamp: Date.now()
})
// Persist counts immediately for consistency
if (this.totalVerbCount % 10 === 0) {
await this.persistCounts()
}
// CRITICAL FIX: Persist counts on EVERY change for cloud storage adapters
await this.persistCounts()
})
}
@ -1008,10 +1005,8 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
timestamp: Date.now()
})
// Persist counts immediately for consistency
if (this.totalVerbCount % 10 === 0) {
await this.persistCounts()
}
// CRITICAL FIX: Persist counts on EVERY change for cloud storage adapters
await this.persistCounts()
})
}