fix: prevent circuit breaker activation and data loss during bulk imports
Storage-aware batching system prevents rate limiting issues on cloud storage (GCS, S3, R2, Azure). Replaces entity-by-entity creation with addMany()/relateMany() batch operations in ImportCoordinator. Separate read/write circuit breakers prevent read lockouts during write throttling. Each storage adapter auto-configures optimal batch sizes and delays. Fixes silent data loss and 30+ second lockouts on 1000+ row imports. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
3c5f622d64
commit
14231554e1
12 changed files with 551 additions and 112 deletions
|
|
@ -22,6 +22,7 @@ import {
|
|||
} from '../../coreTypes.js'
|
||||
import {
|
||||
BaseStorage,
|
||||
StorageBatchConfig,
|
||||
NOUNS_DIR,
|
||||
VERBS_DIR,
|
||||
METADATA_DIR,
|
||||
|
|
@ -175,6 +176,33 @@ export class GcsStorage extends BaseStorage {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get GCS-optimized batch configuration
|
||||
*
|
||||
* GCS has strict rate limits (~5000 writes/second per bucket) and benefits from:
|
||||
* - Moderate batch sizes (50 items)
|
||||
* - Sequential processing (not parallel)
|
||||
* - Delays between batches (100ms)
|
||||
*
|
||||
* Note: Each entity write involves 2 operations (vector + metadata),
|
||||
* so 800 ops/sec = ~400 entities/sec = ~2500 actual GCS writes/sec
|
||||
*
|
||||
* @returns GCS-optimized batch configuration
|
||||
* @since v4.11.0
|
||||
*/
|
||||
public getBatchConfig(): StorageBatchConfig {
|
||||
return {
|
||||
maxBatchSize: 50,
|
||||
batchDelayMs: 100,
|
||||
maxConcurrent: 50,
|
||||
supportsParallelWrites: false, // Sequential is safer for GCS rate limits
|
||||
rateLimit: {
|
||||
operationsPerSecond: 800, // Conservative estimate for entity operations
|
||||
burstCapacity: 200
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Initialize the storage adapter
|
||||
*/
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue