feat: optimize metadata indexing with roaring bitmaps for 90% memory reduction
Replace JavaScript Sets with hardware-accelerated RoaringBitmap32 for metadata indexes. Key improvements: - 1.4x average speedup, up to 3.3x on 10K entities - 90% memory reduction (40 bytes/UUID → 4 bytes/int) - Hardware-accelerated multi-field intersection via SIMD (AVX2/SSE4.2) - EntityIdMapper for bidirectional UUID ↔ integer mapping - Portable serialization format Benchmark results (1,000 queries): - 10K entities: 3.74ms → 1.14ms (3.3x faster, 90% memory savings) - 100K entities: 2.60ms → 1.78ms (1.5x faster, 88% memory savings) Implementation: - Add EntityIdMapper class for UUID/int mapping with persistence - Modify ChunkData to use Map<string, RoaringBitmap32> - Add getIdsForMultipleFields() for fast bitmap intersection - Include comprehensive tests (25 tests passing) - Add performance benchmark comparing Set vs Roaring Technical details: - roaring@2.4.0 dependency - Maintains backward compatibility - All queries still return UUID strings - Automatic persistence via storage adapter
This commit is contained in:
parent
84b657ac47
commit
2f6ab9559a
9 changed files with 2647 additions and 110 deletions
|
|
@ -179,10 +179,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
|
||||
// Initialize core metadata index
|
||||
this.metadataIndex = new MetadataIndexManager(this.storage)
|
||||
|
||||
await this.metadataIndex.init()
|
||||
|
||||
// Initialize core graph index
|
||||
this.graphIndex = new GraphAdjacencyIndex(this.storage)
|
||||
|
||||
|
||||
// Rebuild indexes if needed for existing data
|
||||
await this.rebuildIndexesIfNeeded()
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue