fix: prevent circuit breaker activation and data loss during bulk imports

Storage-aware batching system prevents rate limiting issues on cloud storage (GCS, S3, R2, Azure). Replaces entity-by-entity creation with addMany()/relateMany() batch operations in ImportCoordinator. Separate read/write circuit breakers prevent read lockouts during write throttling. Each storage adapter auto-configures optimal batch sizes and delays. Fixes silent data loss and 30+ second lockouts on 1000+ row imports.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
David Snelling 2025-10-30 08:54:04 -07:00
parent 3c5f622d64
commit 14231554e1
12 changed files with 551 additions and 112 deletions

View file

@ -14,7 +14,7 @@ import {
StatisticsData,
NounType
} from '../../coreTypes.js'
import { BaseStorage, STATISTICS_KEY } from '../baseStorage.js'
import { BaseStorage, StorageBatchConfig, STATISTICS_KEY } from '../baseStorage.js'
import { PaginatedResult } from '../../types/paginationTypes.js'
// No type aliases needed - using the original types directly
@ -47,6 +47,31 @@ export class MemoryStorage extends BaseStorage {
super()
}
/**
* Get Memory-optimized batch configuration
*
* Memory storage has no rate limits and can handle very high throughput:
* - Large batch sizes (1000 items)
* - No delays needed (0ms)
* - High concurrency (1000 operations)
* - Parallel processing maximizes throughput
*
* @returns Memory-optimized batch configuration
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
maxBatchSize: 1000,
batchDelayMs: 0,
maxConcurrent: 1000,
supportsParallelWrites: true, // Memory loves parallel operations
rateLimit: {
operationsPerSecond: 100000, // Virtually unlimited
burstCapacity: 100000
}
}
}
/**
* Initialize the storage adapter
* Nothing to initialize for in-memory storage