David Snelling
2f6ab9559a
Replace JavaScript Sets with hardware-accelerated RoaringBitmap32 for metadata indexes. Key improvements: - 1.4x average speedup, up to 3.3x on 10K entities - 90% memory reduction (40 bytes/UUID → 4 bytes/int) - Hardware-accelerated multi-field intersection via SIMD (AVX2/SSE4.2) - EntityIdMapper for bidirectional UUID ↔ integer mapping - Portable serialization format Benchmark results (1,000 queries): - 10K entities: 3.74ms → 1.14ms (3.3x faster, 90% memory savings) - 100K entities: 2.60ms → 1.78ms (1.5x faster, 88% memory savings) Implementation: - Add EntityIdMapper class for UUID/int mapping with persistence - Modify ChunkData to use Map<string, RoaringBitmap32> - Add getIdsForMultipleFields() for fast bitmap intersection - Include comprehensive tests (25 tests passing) - Add performance benchmark comparing Set vs Roaring Technical details: - roaring@2.4.0 dependency - Maintains backward compatibility - All queries still return UUID strings - Automatic persistence via storage adapter |
||
|---|---|---|
| .. | ||
| API_SURFACE_DESIGN.md | ||
| augmentation-system-audit.md | ||
| augmentations-actual.md | ||
| augmentations.md | ||
| CLUSTERING_ALGORITHMS_ANALYSIS.md | ||
| data-storage-architecture.md | ||
| distributed-storage.md | ||
| index-architecture.md | ||
| METADATA_ARCHITECTURE.md | ||
| noun-verb-taxonomy.md | ||
| overview.md | ||
| PERFORMANCE_ANALYSIS.md | ||
| storage-architecture.md | ||
| triple-intelligence.md | ||
| zero-config.md | ||