David Snelling
c8a239aa8d
BREAKING: Fixes critical event loop blocking during metadata operations ## Event Loop Blocking Fixes - Add setImmediate() yields in metadata rebuild every 10 items - Add event loop yields in addToIndex() every 5 fields - Add event loop yields in flush() operations with smaller batches - Reduce flush threshold from 50 to 10 for more frequent non-blocking flushes - Add progress logging for rebuild operations with event loop yields ## Performance Testing Results - Single item addition: 308ms, max event loop delay 20ms ✅ NO BLOCKING - 200 items in batches of 10: max event loop delay 172ms ✅ NO BLOCKING - 200 items in batches of 50: max event loop delay 700ms+ ❌ BLOCKS ## bluesky-package Solution **Root Cause:** Large concurrent batches (50+ operations) overwhelm metadata indexing **Solution:** Process data in smaller batches (≤10 concurrent operations) with yields **Recommended Pattern for bluesky-package:** ```javascript // Instead of Promise.all([...hundreds of operations]) const BATCH_SIZE = 10 for (let i = 0; i < items.length; i += BATCH_SIZE) { const batch = items.slice(i, i + BATCH_SIZE) await Promise.all(batch.map(item => brainy.add(item))) await new Promise(resolve => setImmediate(resolve)) // Yield } ``` ## Statistics Caching (from previous commits) - 5-minute statistics caching eliminates expensive S3 lookups - getStorageStatus() uses estimation vs full bucket scans - 90%+ reduction in S3 API calls reducing socket pressure Combined solution addresses both root causes: 1. Event loop blocking fixed with proper yields and batch sizing 2. Socket exhaustion fixed with statistics caching and reduced API calls |
||
|---|---|---|
| .. | ||
| adaptiveBackpressure.ts | ||
| adaptiveSocketManager.ts | ||
| autoConfiguration.ts | ||
| cacheAutoConfig.ts | ||
| crypto.ts | ||
| distance.ts | ||
| embedding.ts | ||
| environment.ts | ||
| fieldNameTracking.ts | ||
| index.ts | ||
| jsonProcessing.ts | ||
| logger.ts | ||
| metadataFilter.ts | ||
| metadataIndex.ts | ||
| metadataIndexCache.ts | ||
| operationUtils.ts | ||
| performanceMonitor.ts | ||
| requestCoalescer.ts | ||
| searchCache.ts | ||
| statistics.ts | ||
| statisticsCollector.ts | ||
| textEncoding.ts | ||
| typeUtils.ts | ||
| version.ts | ||
| workerUtils.ts | ||
| writeBuffer.ts | ||