fix: prevent circuit breaker activation and data loss during bulk imports
Storage-aware batching system prevents rate limiting issues on cloud storage (GCS, S3, R2, Azure). Replaces entity-by-entity creation with addMany()/relateMany() batch operations in ImportCoordinator. Separate read/write circuit breakers prevent read lockouts during write throttling. Each storage adapter auto-configures optimal batch sizes and delays. Fixes silent data loss and 30+ second lockouts on 1000+ row imports. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
3c5f622d64
commit
14231554e1
12 changed files with 551 additions and 112 deletions
|
|
@ -32,6 +32,36 @@ interface StorageKeyInfo {
|
|||
fullPath: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Storage adapter batch configuration profile
|
||||
* Each storage adapter declares its optimal batch behavior for rate limiting
|
||||
* and performance optimization
|
||||
*
|
||||
* @since v4.11.0
|
||||
*/
|
||||
export interface StorageBatchConfig {
|
||||
/** Maximum items per batch */
|
||||
maxBatchSize: number
|
||||
|
||||
/** Delay between batches in milliseconds (for rate limiting) */
|
||||
batchDelayMs: number
|
||||
|
||||
/** Maximum concurrent operations this storage can handle */
|
||||
maxConcurrent: number
|
||||
|
||||
/** Whether storage can handle parallel writes efficiently */
|
||||
supportsParallelWrites: boolean
|
||||
|
||||
/** Rate limit characteristics of this storage adapter */
|
||||
rateLimit: {
|
||||
/** Approximate operations per second this storage can handle */
|
||||
operationsPerSecond: number
|
||||
|
||||
/** Maximum burst capacity before throttling occurs */
|
||||
burstCapacity: number
|
||||
}
|
||||
}
|
||||
|
||||
// Clean directory structure (v4.7.2+)
|
||||
// All storage adapters use this consistent structure
|
||||
export const NOUNS_METADATA_DIR = 'entities/nouns/metadata'
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue