David Snelling
1b265e8c42
CRITICAL ISSUE FIXED: Service initialization was failing due to socket exhaustion when reading metadata for 1400+ items during index rebuild Changes: - Add batch metadata reading to prevent 2000+ concurrent requests - Implement strict concurrency control (3 max concurrent requests) - Add proper yielding between batches to prevent event loop blocking - Reduce batch sizes during initialization (50 → 25 items per batch) - Add getMetadataBatch() and getVerbMetadataBatch() methods to S3 storage - Update StorageAdapter interface with batch methods - Add production environment auto-detection for smart logging - Auto-cleanup legacy /index folder during initialization Socket usage: Reduced from 1400+ concurrent to 3 max concurrent Expected production result: - Service initialization will complete successfully - firehoseServiceInitialized: true - Data collection will begin normally - No more socket exhaustion errors (100 socket limit exceeded) Fixes: #socket-exhaustion Breaking: None - backward compatible with fallback modes |
||
|---|---|---|
| .. | ||
| baseStorageAdapter.ts | ||
| batchS3Operations.ts | ||
| fileSystemStorage.ts | ||
| memoryStorage.ts | ||
| opfsStorage.ts | ||
| optimizedS3Search.ts | ||
| s3CompatibleStorage.ts | ||