David Snelling
46583f2d9b
Adds a per-field sorted column store (Lucene doc values + roaring bitmap
architecture) that replaces the MetadataIndex sparse index internals for
both filtering and sorting. One system for all field types with exact
precision — no bucketing, no per-entity storage reads.
Column store: binary .cidx segment format, in-memory tail buffers,
LSM-style compaction, k-way merge sort, multi-value support (__words__).
All queries (filter, range, sort, filtered sort) route through the column
store when data is available, falling back to sparse index otherwise.
Key unlocks:
- find({ orderBy: 'createdAt' }) works WITHOUT a filter (previously threw)
- find({ orderBy: 'metadata.price' }) works for custom numeric fields
- Exact timestamp precision (no 1-minute bucketing)
- O(K log S) sort independent of total entity count
New files: src/indexes/columnStore/ (types, format, tail buffer, cursor,
manifest, coordinator — ~700 lines). 101 new unit tests covering binary
format round-trips, CRC validation, sort, filter, range, deletion,
multi-segment merge, persistence, and multi-value (words) fields.
Deleted: metadataIndex-automatic-bucketing.test.ts (bucketing behavior
eliminated by exact-precision column store). Sparse index write path
removed from addToIndex/removeFromIndex. Sparse index legacy code still
present as dead code pending cleanup in next commit.
|
||
|---|---|---|
| .. | ||
| aggregation | ||
| brainy | ||
| graph | ||
| hnsw | ||
| import | ||
| indexes/columnStore | ||
| neural | ||
| performance | ||
| storage | ||
| types | ||
| utils | ||
| versioning | ||
| vfs | ||
| brainy-core.unit.test.ts | ||
| brainy-get-optimization.test.ts | ||
| create-entities-default.test.ts | ||
| fieldTypeInference.test.ts | ||
| plugin.test.ts | ||
| type-filtering.unit.test.ts | ||
| vfs-restart-fix.test.ts | ||
| workshop-vfs-diagnostic.test.ts | ||