David Snelling
d8ae76ba99
CRITICAL ISSUE FIXED: Service initialization was failing due to socket exhaustion when reading metadata for 1400+ items during index rebuild Changes: - Add batch metadata reading to prevent 2000+ concurrent requests - Implement strict concurrency control (3 max concurrent requests) - Add proper yielding between batches to prevent event loop blocking - Reduce batch sizes during initialization (50 → 25 items per batch) - Add getMetadataBatch() and getVerbMetadataBatch() methods to S3 storage - Update StorageAdapter interface with batch methods - Add production environment auto-detection for smart logging - Auto-cleanup legacy /index folder during initialization Socket usage: Reduced from 1400+ concurrent to 3 max concurrent Expected production result: - Service initialization will complete successfully - firehoseServiceInitialized: true - Data collection will begin normally - No more socket exhaustion errors (100 socket limit exceeded) Fixes: #socket-exhaustion Breaking: None - backward compatible with fallback modes |
||
|---|---|---|
| .. | ||
| augmentations | ||
| distributed | ||
| errors | ||
| examples | ||
| hnsw | ||
| mcp | ||
| storage | ||
| types | ||
| utils | ||
| augmentationFactory.ts | ||
| augmentationPipeline.ts | ||
| augmentationRegistry.ts | ||
| augmentationRegistryLoader.ts | ||
| brainyData.ts | ||
| browserFramework.ts | ||
| coreTypes.ts | ||
| demo.ts | ||
| index.ts | ||
| pipeline.ts | ||
| sequentialPipeline.ts | ||
| setup.ts | ||
| unified.ts | ||
| worker.js | ||
| worker.ts | ||