fix: prevent circuit breaker activation and data loss during bulk imports

Storage-aware batching system prevents rate limiting issues on cloud storage (GCS, S3, R2, Azure). Replaces entity-by-entity creation with addMany()/relateMany() batch operations in ImportCoordinator. Separate read/write circuit breakers prevent read lockouts during write throttling. Each storage adapter auto-configures optimal batch sizes and delays. Fixes silent data loss and 30+ second lockouts on 1000+ row imports.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
David Snelling 2025-10-30 08:54:04 -07:00
parent 3c5f622d64
commit 14231554e1
12 changed files with 551 additions and 112 deletions

View file

@ -25,6 +25,7 @@ import {
} from '../../coreTypes.js'
import {
BaseStorage,
StorageBatchConfig,
NOUNS_DIR,
VERBS_DIR,
METADATA_DIR,
@ -169,6 +170,35 @@ export class R2Storage extends BaseStorage {
}
}
/**
* Get R2-optimized batch configuration
*
* Cloudflare R2 has S3-compatible characteristics with some advantages:
* - Zero egress fees (can cache more aggressively)
* - Global edge network
* - Similar throughput to S3
*
* R2 benefits from the same configuration as S3:
* - Larger batch sizes (100 items)
* - Parallel processing
* - Short delays (50ms)
*
* @returns R2-optimized batch configuration
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
maxBatchSize: 100,
batchDelayMs: 50,
maxConcurrent: 100,
supportsParallelWrites: true, // R2 handles parallel writes like S3
rateLimit: {
operationsPerSecond: 3500, // Similar to S3 throughput
burstCapacity: 1000
}
}
}
/**
* Initialize the storage adapter
*/