soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions
250 commits 8 branches 522 tags 189 MiB
Commit graph

3 commits

Author SHA1 Message Date
David Snelling
2f6ab9559a feat: optimize metadata indexing with roaring bitmaps for 90% memory reduction 
Replace JavaScript Sets with hardware-accelerated RoaringBitmap32 for metadata indexes.

Key improvements:
- 1.4x average speedup, up to 3.3x on 10K entities
- 90% memory reduction (40 bytes/UUID → 4 bytes/int)
- Hardware-accelerated multi-field intersection via SIMD (AVX2/SSE4.2)
- EntityIdMapper for bidirectional UUID ↔ integer mapping
- Portable serialization format

Benchmark results (1,000 queries):
- 10K entities: 3.74ms → 1.14ms (3.3x faster, 90% memory savings)
- 100K entities: 2.60ms → 1.78ms (1.5x faster, 88% memory savings)

Implementation:
- Add EntityIdMapper class for UUID/int mapping with persistence
- Modify ChunkData to use Map<string, RoaringBitmap32>
- Add getIdsForMultipleFields() for fast bitmap intersection
- Include comprehensive tests (25 tests passing)
- Add performance benchmark comparing Set vs Roaring

Technical details:
- roaring@2.4.0 dependency
- Maintains backward compatibility
- All queries still return UUID strings
- Automatic persistence via storage adapter
 2025-10-13 16:39:06 -07:00
David Snelling
b7dfc52e94 feat: replace flat file indexing with adaptive chunked sparse indexing 
Major refactor of metadata indexing system for production scalability:

Performance improvements:
- 630x file reduction: 560,000 flat files → 89 chunk files
- O(1) exact match queries with bloom filters (1% false positive rate)
- O(log n) range queries with zone maps (ClickHouse-inspired)
- Adaptive chunking: ~50 values per chunk optimizes I/O

Technical changes:
- NEW: src/utils/metadataIndexChunking.ts
  - BloomFilter: Probabilistic membership testing (FNV-1a + DJB2)
  - SparseIndex: Directory of chunks with metadata
  - ChunkManager: Handles chunk CRUD operations
  - AdaptiveChunkingStrategy: Field-specific optimization
  - ZoneMap: Min/max tracking for range query optimization

- REFACTORED: src/utils/metadataIndex.ts
  - Removed indexCache (flat file entry cache)
  - Removed dirtyEntries (flat file dirty tracking)
  - Removed sortedIndices (sorted index for range queries)
  - Removed 13 obsolete methods (sorted index operations, flat file I/O)
  - Simplified flush() to only flush field indexes
  - All fields now use chunked sparse indexing exclusively

- UPDATED: docs/architecture/index-architecture.md
  - Documented new chunked sparse index architecture
  - Added bloom filter and zone map explanations
  - Updated query algorithm examples
  - Added v3.42.0 version history

Benefits:
- Single code path (no more dual flat file + chunks)
- Immediate chunk flushing (no dirty tracking needed)
- Better I/O patterns (chunk-based instead of per-value files)
- Production-ready for billions of entities
- Zero breaking changes to public API

All tests passing. Ready for production.
 2025-10-13 15:31:03 -07:00
David Snelling
75b4b02610 docs: add comprehensive index architecture documentation 
Add detailed documentation explaining Brainy's 4-index architecture:
- MetadataIndex: inverted indexing with temporal bucketing (v3.41.0)
- HNSWIndex: hierarchical vector similarity search
- GraphAdjacencyIndex: O(1) bidirectional relationship traversal
- DeletedItemsIndex: soft-delete tracking

Includes explanation of UnifiedCache for coordinated memory management
across all indexes, integration patterns, performance characteristics,
and best practices for developers.

Updated overview.md to reference the new index architecture documentation.
 2025-10-13 13:42:04 -07:00