fix: prevent circuit breaker activation and data loss during bulk imports
Storage-aware batching system prevents rate limiting issues on cloud storage (GCS, S3, R2, Azure). Replaces entity-by-entity creation with addMany()/relateMany() batch operations in ImportCoordinator. Separate read/write circuit breakers prevent read lockouts during write throttling. Each storage adapter auto-configures optimal batch sizes and delays. Fixes silent data loss and 30+ second lockouts on 1000+ row imports. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
3c5f622d64
commit
14231554e1
12 changed files with 551 additions and 112 deletions
|
|
@ -14,6 +14,7 @@ import {
|
|||
NounMetadata,
|
||||
VerbMetadata
|
||||
} from '../../coreTypes.js'
|
||||
import { StorageBatchConfig } from '../baseStorage.js'
|
||||
import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js'
|
||||
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
|
||||
|
||||
|
|
@ -90,6 +91,33 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
details?: Record<string, any>
|
||||
}>
|
||||
|
||||
/**
|
||||
* Get optimal batch configuration for this storage adapter
|
||||
* Override in subclasses to provide storage-specific optimization
|
||||
*
|
||||
* This method allows each storage adapter to declare its optimal batch behavior
|
||||
* for rate limiting and performance. The configuration is used by addMany(),
|
||||
* relateMany(), and import operations to automatically adapt to storage capabilities.
|
||||
*
|
||||
* @returns Batch configuration optimized for this storage type
|
||||
* @since v4.11.0
|
||||
*/
|
||||
public getBatchConfig(): StorageBatchConfig {
|
||||
// Conservative defaults that work safely across all storage types
|
||||
// Cloud storage adapters should override with higher throughput values
|
||||
// Local storage adapters should override with no delays
|
||||
return {
|
||||
maxBatchSize: 50,
|
||||
batchDelayMs: 100,
|
||||
maxConcurrent: 50,
|
||||
supportsParallelWrites: false,
|
||||
rateLimit: {
|
||||
operationsPerSecond: 100,
|
||||
burstCapacity: 200
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// NOTE: getAllNouns and getAllVerbs have been removed to prevent expensive full scans.
|
||||
// Use getNouns() and getVerbs() with pagination instead.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue