Commit graph

2 commits

Author SHA1 Message Date
773c5171c3 fix: flush all native providers on shutdown to prevent data loss
Shutdown/close/flush now properly flushes all 4 components in parallel:
metadataIndex, graphIndex, HNSW dirty nodes, and storage counts. Previously
only counts were flushed, causing native provider data loss on restart.

Also:
- Wire roaring, msgpack, entityIdMapper provider consumption from plugins
- Fix allOf filter O(n²) intersection → O(n) Set-based
- Fix ne/exists negation filter to use Set-based exclusion
- Add setMsgpackImplementation() swap in SSTable for native msgpack
- Add setRoaringImplementation() swap for native CRoaring bitmaps
- Add getAllIntIds() to EntityIdMapper for bitmap operations
- Remove TypeAwareHNSWIndex from default index creation path
- Export memory detection utilities from internals
- Clean up 26 permanently-skipped dead tests
2026-02-01 16:23:49 -08:00
0f3a88429d feat: add SQ8 vector quantization, lazy loading, and two-phase rerank to HNSW
- SQ8 scalar quantization (8-bit) for 4x vector storage reduction
- Lazy vector loading: evict float32 vectors after graph construction,
  load on-demand from storage via UnifiedCache
- Two-phase search: over-retrieve with SQ8 approximate distances,
  rerank top candidates with exact float32 distances
- Configuration surface: hnsw.quantization and hnsw.vectorStorage in BrainyConfig
- All features disabled by default (zero behavior change for existing users)
- 27 new tests covering quantization accuracy, lazy loading, reranking
- Remove GitHub Actions CI (build locally, cortex CI handles native builds)
2026-01-31 12:41:53 -08:00