soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions
brainy/src
History
David Snelling e600865d96 fix: metadata explosion bug - 69K files reduced to ~1K 
Critical fix for metadata indexing that was creating 60+ chunk files per entity.

Root cause: Vector embeddings (384-dimensional arrays) were being indexed in
metadata, causing each dimension to create a separate chunk file with numeric
field names ("0", "1", "2", etc.).

Changes:
- Modified extractIndexableFields() to exclude vector/embedding fields
- Added NEVER_INDEX set: ['vector', 'embedding', 'embeddings', 'connections']
- Added safety check to skip arrays > 10 elements
- Preserves small array indexing (tags, categories, roles)

Impact:
- Reduces metadata files from 69,429 → ~1,200 (58x reduction)
- Fixes server initialization hangs
- Fixes metadata batch loading stalling at batch 23
- Fixes VFS getDescendants() hanging with large datasets
- Fixes Graph View UI not loading

Test Results:
- 7/7 integration tests passing
- Verified: 6 chunk files for 10 entities (was 7,210 before fix)
- 611/622 unit tests passing

Files Modified:
- src/utils/metadataIndex.ts - Core fix
- src/coreTypes.ts - HNSWVerb type enforcement with VerbType enum
- src/storage/adapters/* - Include core relational fields in HNSWVerb
- src/storage/adapters/baseStorageAdapter.ts - Type enforcement (HNSWNoun, GraphVerb)
- tests/integration/metadata-vector-exclusion.test.ts - Comprehensive test coverage

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-16 16:10:31 -07:00
..
api feat: add real-time progress callbacks for relationship building phase 2025-10-16 12:08:46 -07:00
augmentations feat: Phase 3 - Unified Semantic Type Inference (Nouns + Verbs) 2025-10-16 10:59:26 -07:00
cli refactor: streamline core API surface 2025-10-04 08:52:06 -07:00
config fix: enable GCS native storage with Application Default Credentials 2025-10-09 10:39:54 -07:00
cortex feat: add node: protocol to all Node.js built-in imports for bundler compatibility 2025-09-17 14:20:21 -07:00
critical feat: add browser environment compatibility support 2025-09-17 15:48:02 -07:00
data feat: Phase 3 - Unified Semantic Type Inference (Nouns + Verbs) 2025-10-16 10:59:26 -07:00
distributed feat: add node: protocol to all Node.js built-in imports for bundler compatibility 2025-09-17 14:20:21 -07:00
embeddings feat: complete VFS with Knowledge Layer integration 2025-09-25 10:47:44 -07:00
errors feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
graph feat: billion-scale graph storage with LSM-tree 2025-10-14 16:36:26 -07:00
hnsw fix: 6000x speedup for TypeAwareHNSWIndex rebuild - enables billion-scale operations 2025-10-15 17:48:26 -07:00
import feat: add real-time progress callbacks for relationship building phase 2025-10-16 12:08:46 -07:00
importers feat: add real-time progress callbacks for relationship building phase 2025-10-16 12:08:46 -07:00
interfaces feat: implement always-adaptive caching with getCacheStats monitoring 2025-10-10 14:09:30 -07:00
mcp refactor: streamline core API surface 2025-10-04 08:52:06 -07:00
neural feat: production-ready value-based temporal field detection 2025-10-16 13:58:57 -07:00
patterns 🧠 Brainy 2.0.0 - Zero-Configuration AI Database with Triple Intelligence™ 2025-08-26 12:32:21 -07:00
query feat: Phase 3 - Unified Semantic Type Inference (Nouns + Verbs) 2025-10-16 10:59:26 -07:00
scripts feat: add node: protocol to all Node.js built-in imports for bundler compatibility 2025-09-17 14:20:21 -07:00
shared refactor: remove deprecated BrainyData class completely 2025-09-30 16:04:00 -07:00
storage fix: metadata explosion bug - 69K files reduced to ~1K 2025-10-16 16:10:31 -07:00
streaming feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
triple feat: Phase 3 - Unified Semantic Type Inference (Nouns + Verbs) 2025-10-16 10:59:26 -07:00
types feat(types): Phase 0 - Type System Foundation for billion-scale optimization 2025-10-15 12:26:25 -07:00
universal fix: prevent fs adapter from throwing on import in browser 2025-09-17 16:22:02 -07:00
utils fix: metadata explosion bug - 69K files reduced to ~1K 2025-10-16 16:10:31 -07:00
vfs feat: add resolvePathToId() method and fix test issues 2025-10-07 11:51:17 -07:00
augmentationManager.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
augmentationPipeline.ts feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
brainy.ts feat: add real-time progress callbacks for relationship building phase 2025-10-16 12:08:46 -07:00
coreTypes.ts fix: metadata explosion bug - 69K files reduced to ~1K 2025-10-16 16:10:31 -07:00
cortex.ts 🧠 Brainy 2.0.0 - Zero-Configuration AI Database with Triple Intelligence™ 2025-08-26 12:32:21 -07:00
index.ts feat: Phase 3 - Unified Semantic Type Inference (Nouns + Verbs) 2025-10-16 10:59:26 -07:00
pipeline.ts 🧠 Brainy 2.0.0 - Zero-Configuration AI Database with Triple Intelligence™ 2025-08-26 12:32:21 -07:00
setup.ts chore: enforce consistent coding style and semicolon removal 2025-09-29 09:50:59 -07:00
unified.ts chore: enforce consistent coding style and semicolon removal 2025-09-29 09:50:59 -07:00
worker.ts 🧠 Brainy 2.0.0 - Zero-Configuration AI Database with Triple Intelligence™ 2025-08-26 12:32:21 -07:00