soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions
253 commits 10 branches 523 tags 190 MiB
Commit graph

3 commits

Author SHA1 Message Date
David Snelling
aeffd32fe0 fix: correct import paths for TypeScript compilation 
- Add .js extension to entityIdMapper baseStorage import
- Change roaring imports from subpath to main package export
- Use import { RoaringBitmap32 } from 'roaring' for ESM compatibility
 2025-10-13 16:40:50 -07:00
David Snelling
2f6ab9559a feat: optimize metadata indexing with roaring bitmaps for 90% memory reduction 
Replace JavaScript Sets with hardware-accelerated RoaringBitmap32 for metadata indexes.

Key improvements:
- 1.4x average speedup, up to 3.3x on 10K entities
- 90% memory reduction (40 bytes/UUID → 4 bytes/int)
- Hardware-accelerated multi-field intersection via SIMD (AVX2/SSE4.2)
- EntityIdMapper for bidirectional UUID ↔ integer mapping
- Portable serialization format

Benchmark results (1,000 queries):
- 10K entities: 3.74ms → 1.14ms (3.3x faster, 90% memory savings)
- 100K entities: 2.60ms → 1.78ms (1.5x faster, 88% memory savings)

Implementation:
- Add EntityIdMapper class for UUID/int mapping with persistence
- Modify ChunkData to use Map<string, RoaringBitmap32>
- Add getIdsForMultipleFields() for fast bitmap intersection
- Include comprehensive tests (25 tests passing)
- Add performance benchmark comparing Set vs Roaring

Technical details:
- roaring@2.4.0 dependency
- Maintains backward compatibility
- All queries still return UUID strings
- Automatic persistence via storage adapter
 2025-10-13 16:39:06 -07:00
David Snelling
b7dfc52e94 feat: replace flat file indexing with adaptive chunked sparse indexing 
Major refactor of metadata indexing system for production scalability:

Performance improvements:
- 630x file reduction: 560,000 flat files → 89 chunk files
- O(1) exact match queries with bloom filters (1% false positive rate)
- O(log n) range queries with zone maps (ClickHouse-inspired)
- Adaptive chunking: ~50 values per chunk optimizes I/O

Technical changes:
- NEW: src/utils/metadataIndexChunking.ts
  - BloomFilter: Probabilistic membership testing (FNV-1a + DJB2)
  - SparseIndex: Directory of chunks with metadata
  - ChunkManager: Handles chunk CRUD operations
  - AdaptiveChunkingStrategy: Field-specific optimization
  - ZoneMap: Min/max tracking for range query optimization

- REFACTORED: src/utils/metadataIndex.ts
  - Removed indexCache (flat file entry cache)
  - Removed dirtyEntries (flat file dirty tracking)
  - Removed sortedIndices (sorted index for range queries)
  - Removed 13 obsolete methods (sorted index operations, flat file I/O)
  - Simplified flush() to only flush field indexes
  - All fields now use chunked sparse indexing exclusively

- UPDATED: docs/architecture/index-architecture.md
  - Documented new chunked sparse index architecture
  - Added bloom filter and zone map explanations
  - Updated query algorithm examples
  - Added v3.42.0 version history

Benefits:
- Single code path (no more dual flat file + chunks)
- Immediate chunk flushing (no dirty tracking needed)
- Better I/O patterns (chunk-based instead of per-value files)
- Production-ready for billions of entities
- Zero breaking changes to public API

All tests passing. Ready for production.
 2025-10-13 15:31:03 -07:00