feat: replace flat file indexing with adaptive chunked sparse indexing

Browse source

Major refactor of metadata indexing system for production scalability:

Performance improvements:
- 630x file reduction: 560,000 flat files → 89 chunk files
- O(1) exact match queries with bloom filters (1% false positive rate)
- O(log n) range queries with zone maps (ClickHouse-inspired)
- Adaptive chunking: ~50 values per chunk optimizes I/O

Technical changes:
- NEW: src/utils/metadataIndexChunking.ts
  - BloomFilter: Probabilistic membership testing (FNV-1a + DJB2)
  - SparseIndex: Directory of chunks with metadata
  - ChunkManager: Handles chunk CRUD operations
  - AdaptiveChunkingStrategy: Field-specific optimization
  - ZoneMap: Min/max tracking for range query optimization

- REFACTORED: src/utils/metadataIndex.ts
  - Removed indexCache (flat file entry cache)
  - Removed dirtyEntries (flat file dirty tracking)
  - Removed sortedIndices (sorted index for range queries)
  - Removed 13 obsolete methods (sorted index operations, flat file I/O)
  - Simplified flush() to only flush field indexes
  - All fields now use chunked sparse indexing exclusively

- UPDATED: docs/architecture/index-architecture.md
  - Documented new chunked sparse index architecture
  - Added bloom filter and zone map explanations
  - Updated query algorithm examples
  - Added v3.42.0 version history

Benefits:
- Single code path (no more dual flat file + chunks)
- Immediate chunk flushing (no dirty tracking needed)
- Better I/O patterns (chunk-based instead of per-value files)
- Production-ready for billions of entities
- Zero breaking changes to public API

All tests passing. Ready for production.

...

This commit is contained in:

David Snelling 2025-10-13 15:31:03 -07:00

parent af376dcdfd

commit b7dfc52e94

3 changed files with 1437 additions and 723 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

1126

src/utils/metadataIndex.ts

View file

File diff suppressed because it is too large Load diff