David Snelling
e600865d96
Critical fix for metadata indexing that was creating 60+ chunk files per entity.
Root cause: Vector embeddings (384-dimensional arrays) were being indexed in
metadata, causing each dimension to create a separate chunk file with numeric
field names ("0", "1", "2", etc.).
Changes:
- Modified extractIndexableFields() to exclude vector/embedding fields
- Added NEVER_INDEX set: ['vector', 'embedding', 'embeddings', 'connections']
- Added safety check to skip arrays > 10 elements
- Preserves small array indexing (tags, categories, roles)
Impact:
- Reduces metadata files from 69,429 → ~1,200 (58x reduction)
- Fixes server initialization hangs
- Fixes metadata batch loading stalling at batch 23
- Fixes VFS getDescendants() hanging with large datasets
- Fixes Graph View UI not loading
Test Results:
- 7/7 integration tests passing
- Verified: 6 chunk files for 10 entities (was 7,210 before fix)
- 611/622 unit tests passing
Files Modified:
- src/utils/metadataIndex.ts - Core fix
- src/coreTypes.ts - HNSWVerb type enforcement with VerbType enum
- src/storage/adapters/* - Include core relational fields in HNSWVerb
- src/storage/adapters/baseStorageAdapter.ts - Type enforcement (HNSWNoun, GraphVerb)
- tests/integration/metadata-vector-exclusion.test.ts - Comprehensive test coverage
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| adaptiveBackpressure.ts | ||
| adaptiveSocketManager.ts | ||
| autoConfiguration.ts | ||
| BoundedRegistry.ts | ||
| brainyTypes.ts | ||
| crypto.ts | ||
| deletedItemsIndex.ts | ||
| distance.ts | ||
| embedding.ts | ||
| enhancedLogger.ts | ||
| ensureDeleted.ts | ||
| entityIdMapper.ts | ||
| environment.ts | ||
| fieldNameTracking.ts | ||
| fieldTypeInference.ts | ||
| index.ts | ||
| intelligentTypeMapper.ts | ||
| jsonProcessing.ts | ||
| logger.ts | ||
| memoryDetection.ts | ||
| metadataFilter.ts | ||
| metadataIndex.ts | ||
| metadataIndexCache.ts | ||
| metadataIndexChunking.ts | ||
| metadataNamespace.ts | ||
| mutex.ts | ||
| nodeVersionCheck.ts | ||
| operationUtils.ts | ||
| paramValidation.ts | ||
| performanceMonitor.ts | ||
| periodicCleanup.ts | ||
| rateLimiter.ts | ||
| requestCoalescer.ts | ||
| requestDeduplicator.ts | ||
| searchCache.ts | ||
| statisticsCollector.ts | ||
| structuredLogger.ts | ||
| textEncoding.ts | ||
| typeUtils.ts | ||
| typeValidation.ts | ||
| unifiedCache.ts | ||
| version.ts | ||
| workerUtils.ts | ||
| writeBuffer.ts | ||