soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions
brainy/docs/architecture
History
David Snelling 2f6ab9559a feat: optimize metadata indexing with roaring bitmaps for 90% memory reduction 
Replace JavaScript Sets with hardware-accelerated RoaringBitmap32 for metadata indexes.

Key improvements:
- 1.4x average speedup, up to 3.3x on 10K entities
- 90% memory reduction (40 bytes/UUID → 4 bytes/int)
- Hardware-accelerated multi-field intersection via SIMD (AVX2/SSE4.2)
- EntityIdMapper for bidirectional UUID ↔ integer mapping
- Portable serialization format

Benchmark results (1,000 queries):
- 10K entities: 3.74ms → 1.14ms (3.3x faster, 90% memory savings)
- 100K entities: 2.60ms → 1.78ms (1.5x faster, 88% memory savings)

Implementation:
- Add EntityIdMapper class for UUID/int mapping with persistence
- Modify ChunkData to use Map<string, RoaringBitmap32>
- Add getIdsForMultipleFields() for fast bitmap intersection
- Include comprehensive tests (25 tests passing)
- Add performance benchmark comparing Set vs Roaring

Technical details:
- roaring@2.4.0 dependency
- Maintains backward compatibility
- All queries still return UUID strings
- Automatic persistence via storage adapter
2025-10-13 16:39:06 -07:00
..
API_SURFACE_DESIGN.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
augmentation-system-audit.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
augmentations-actual.md feat: remove legacy ImportManager, standardize getStats() API 2025-10-09 11:40:31 -07:00
augmentations.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
CLUSTERING_ALGORITHMS_ANALYSIS.md docs: add deprecation warnings for addNoun and addVerb methods 2025-09-17 10:48:41 -07:00
data-storage-architecture.md feat: implement always-adaptive caching with getCacheStats monitoring 2025-10-10 14:09:30 -07:00
distributed-storage.md docs: add comprehensive scaling and storage architecture documentation 2025-09-08 14:49:25 -07:00
index-architecture.md feat: optimize metadata indexing with roaring bitmaps for 90% memory reduction 2025-10-13 16:39:06 -07:00
METADATA_ARCHITECTURE.md feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
noun-verb-taxonomy.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
overview.md docs: add comprehensive index architecture documentation 2025-10-13 13:42:04 -07:00
PERFORMANCE_ANALYSIS.md feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
storage-architecture.md feat: implement always-adaptive caching with getCacheStats monitoring 2025-10-10 14:09:30 -07:00
triple-intelligence.md 🧠 Brainy 2.0.0 - Zero-Configuration AI Database with Triple Intelligence™ 2025-08-26 12:32:21 -07:00
zero-config.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00