CRITICAL ISSUE FIXED: Service initialization was failing due to socket exhaustion
when reading metadata for 1400+ items during index rebuild
Changes:
- Add batch metadata reading to prevent 2000+ concurrent requests
- Implement strict concurrency control (3 max concurrent requests)
- Add proper yielding between batches to prevent event loop blocking
- Reduce batch sizes during initialization (50 → 25 items per batch)
- Add getMetadataBatch() and getVerbMetadataBatch() methods to S3 storage
- Update StorageAdapter interface with batch methods
- Add production environment auto-detection for smart logging
- Auto-cleanup legacy /index folder during initialization
Socket usage: Reduced from 1400+ concurrent to 3 max concurrent
Expected production result:
- Service initialization will complete successfully
- firehoseServiceInitialized: true
- Data collection will begin normally
- No more socket exhaustion errors (100 socket limit exceeded)
Fixes: #socket-exhaustion
Breaking: None - backward compatible with fallback modes
David Snelling
d8ae76ba99