fix: prevent circuit breaker activation and data loss during bulk imports

Storage-aware batching system prevents rate limiting issues on cloud storage (GCS, S3, R2, Azure). Replaces entity-by-entity creation with addMany()/relateMany() batch operations in ImportCoordinator. Separate read/write circuit breakers prevent read lockouts during write throttling. Each storage adapter auto-configures optimal batch sizes and delays. Fixes silent data loss and 30+ second lockouts on 1000+ row imports.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
David Snelling 2025-10-30 08:54:04 -07:00
parent 3c5f622d64
commit 14231554e1
12 changed files with 551 additions and 112 deletions

View file

@ -22,6 +22,7 @@ import {
} from '../../coreTypes.js'
import {
BaseStorage,
StorageBatchConfig,
NOUNS_DIR,
VERBS_DIR,
METADATA_DIR,
@ -175,6 +176,33 @@ export class GcsStorage extends BaseStorage {
}
}
/**
* Get GCS-optimized batch configuration
*
* GCS has strict rate limits (~5000 writes/second per bucket) and benefits from:
* - Moderate batch sizes (50 items)
* - Sequential processing (not parallel)
* - Delays between batches (100ms)
*
* Note: Each entity write involves 2 operations (vector + metadata),
* so 800 ops/sec = ~400 entities/sec = ~2500 actual GCS writes/sec
*
* @returns GCS-optimized batch configuration
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
maxBatchSize: 50,
batchDelayMs: 100,
maxConcurrent: 50,
supportsParallelWrites: false, // Sequential is safer for GCS rate limits
rateLimit: {
operationsPerSecond: 800, // Conservative estimate for entity operations
burstCapacity: 200
}
}
}
/**
* Initialize the storage adapter
*/