David Snelling
6161ce3f5e
- Issue: v4.2.1 field registry only helps on 2nd+ runs - first run still slow (8-9 min for 1,157 entities) - Root Cause: Batch size of 25 was designed for cloud storage socket exhaustion, too conservative for local storage - Solution: Adaptive batch sizing based on storage adapter type - FileSystemStorage/MemoryStorage/OPFSStorage: 500 items/batch (fast local I/O, no socket limits) - GCS/S3/R2 (cloud storage): 25 items/batch (prevent socket exhaustion) - Performance Impact: - FileSystem first-run rebuild: 8-9 min → 30-60 seconds (10-15x faster) - 1,157 entities: 46 batches @ 25 → 3 batches @ 500 (15x fewer I/O operations) - Cloud storage: No change (still 25/batch for safety) - Detection: Auto-detects storage type via constructor.name - Zero Config: Completely automatic, no configuration needed - Combined with v4.2.1: First run fast, subsequent runs instant (2-3 sec) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| adaptiveBackpressure.ts | ||
| adaptiveSocketManager.ts | ||
| autoConfiguration.ts | ||
| BoundedRegistry.ts | ||
| brainyTypes.ts | ||
| crypto.ts | ||
| deletedItemsIndex.ts | ||
| distance.ts | ||
| embedding.ts | ||
| enhancedLogger.ts | ||
| ensureDeleted.ts | ||
| entityIdMapper.ts | ||
| environment.ts | ||
| fieldNameTracking.ts | ||
| fieldTypeInference.ts | ||
| index.ts | ||
| intelligentTypeMapper.ts | ||
| jsonProcessing.ts | ||
| logger.ts | ||
| memoryDetection.ts | ||
| metadataFilter.ts | ||
| metadataIndex.ts | ||
| metadataIndexCache.ts | ||
| metadataIndexChunking.ts | ||
| metadataNamespace.ts | ||
| mutex.ts | ||
| nodeVersionCheck.ts | ||
| operationUtils.ts | ||
| paramValidation.ts | ||
| performanceMonitor.ts | ||
| periodicCleanup.ts | ||
| rateLimiter.ts | ||
| rebuildCounts.ts | ||
| requestCoalescer.ts | ||
| requestDeduplicator.ts | ||
| searchCache.ts | ||
| statisticsCollector.ts | ||
| structuredLogger.ts | ||
| textEncoding.ts | ||
| typeUtils.ts | ||
| typeValidation.ts | ||
| unifiedCache.ts | ||
| version.ts | ||
| workerUtils.ts | ||
| writeBuffer.ts | ||