feat: optimize metadata indexing with roaring bitmaps for 90% memory reduction
Replace JavaScript Sets with hardware-accelerated RoaringBitmap32 for metadata indexes.
Key improvements:
- 1.4x average speedup, up to 3.3x on 10K entities
- 90% memory reduction (40 bytes/UUID → 4 bytes/int)
- Hardware-accelerated multi-field intersection via SIMD (AVX2/SSE4.2)
- EntityIdMapper for bidirectional UUID ↔ integer mapping
- Portable serialization format
Benchmark results (1,000 queries):
- 10K entities: 3.74ms → 1.14ms (3.3x faster, 90% memory savings)
- 100K entities: 2.60ms → 1.78ms (1.5x faster, 88% memory savings)
Implementation:
- Add EntityIdMapper class for UUID/int mapping with persistence
- Modify ChunkData to use Map<string, RoaringBitmap32>
- Add getIdsForMultipleFields() for fast bitmap intersection
- Include comprehensive tests (25 tests passing)
- Add performance benchmark comparing Set vs Roaring
Technical details:
- roaring@2.4.0 dependency
- Maintains backward compatibility
- All queries still return UUID strings
- Automatic persistence via storage adapter
David Snelling
2f6ab9559a