David Snelling
2f6ab9559a
Replace JavaScript Sets with hardware-accelerated RoaringBitmap32 for metadata indexes. Key improvements: - 1.4x average speedup, up to 3.3x on 10K entities - 90% memory reduction (40 bytes/UUID → 4 bytes/int) - Hardware-accelerated multi-field intersection via SIMD (AVX2/SSE4.2) - EntityIdMapper for bidirectional UUID ↔ integer mapping - Portable serialization format Benchmark results (1,000 queries): - 10K entities: 3.74ms → 1.14ms (3.3x faster, 90% memory savings) - 100K entities: 2.60ms → 1.78ms (1.5x faster, 88% memory savings) Implementation: - Add EntityIdMapper class for UUID/int mapping with persistence - Modify ChunkData to use Map<string, RoaringBitmap32> - Add getIdsForMultipleFields() for fast bitmap intersection - Include comprehensive tests (25 tests passing) - Add performance benchmark comparing Set vs Roaring Technical details: - roaring@2.4.0 dependency - Maintains backward compatibility - All queries still return UUID strings - Automatic persistence via storage adapter |
||
|---|---|---|
| .. | ||
| api | ||
| augmentations/intelligentImport | ||
| benchmarks | ||
| comprehensive | ||
| configs | ||
| fixtures/import | ||
| helpers | ||
| integration | ||
| performance | ||
| scripts | ||
| unit | ||
| vfs | ||
| brainy-3.test.ts | ||
| critical-error-handling.test.ts | ||
| critical-neural-validation.test.ts | ||
| critical-performance-benchmark.test.ts | ||
| distributed-demo.test.ts | ||
| model-loading.test.ts | ||
| opfs-storage.test.ts | ||
| package-size-breakdown.test.ts | ||
| package-size-limit.test.ts | ||
| setup-integration.ts | ||
| setup-unit.ts | ||
| setup.ts | ||
| streaming-pipeline.test.ts | ||
| type-utils.test.ts | ||