soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

335 releases 524 tags

RSS feed
  • v3.43.0 6c9157a274

    v3.43.0: Roaring Bitmap Optimization Stable

    dpsifr released this 2025-10-14 01:46:14 +02:00 | 792 commits to main since this release

    🚀 Performance: Roaring Bitmap Optimization

    This release introduces roaring bitmap optimization for metadata indexing, delivering significant performance and memory improvements.

    Key Improvements

    • 90% memory reduction in metadata indexes (40 bytes/UUID → 4 bytes/int)
    • 1.4x average speedup, up to 3.3x faster on 10K entities
    • Hardware-accelerated operations via SIMD instructions (AVX2/SSE4.2)
    • Portable serialization compatible across platforms

    Benchmark Results (1,000 queries)

    Dataset Size Operation Set Time Roaring Time Speedup Memory Savings
    10K entities 3-field intersection 3.74ms 1.14ms 3.3x faster 90%
    100K entities 3-field intersection 2.60ms 1.78ms 1.5x faster 88%

    Implementation Details

    • EntityIdMapper for bidirectional UUID ↔ integer mapping
    • RoaringBitmap32 replaces Set<string> in chunked sparse indexes
    • getIdsForMultipleFields() method for fast multi-field intersection queries
    • Comprehensive test suite (25 tests) and performance benchmarks included
    • All external APIs still return UUID strings (backward compatible)

    Technical Architecture

    • Dependency: roaring@2.4.0 (hardware-accelerated bitmap library)
    • Persistence: EntityIdMapper mappings automatically saved to storage
    • Compatibility: Works with all storage adapters (memory, filesystem, GCS, S3)
    • Zero breaking changes: All existing code continues to work

    See full implementation details in docs/architecture/index-architecture.md

    Full Changelog: https://github.com/soulcraftlabs/brainy/compare/v3.42.0...v3.43.0

    Downloads