David Snelling
46583f2d9b
Adds a per-field sorted column store (Lucene doc values + roaring bitmap
architecture) that replaces the MetadataIndex sparse index internals for
both filtering and sorting. One system for all field types with exact
precision — no bucketing, no per-entity storage reads.
Column store: binary .cidx segment format, in-memory tail buffers,
LSM-style compaction, k-way merge sort, multi-value support (__words__).
All queries (filter, range, sort, filtered sort) route through the column
store when data is available, falling back to sparse index otherwise.
Key unlocks:
- find({ orderBy: 'createdAt' }) works WITHOUT a filter (previously threw)
- find({ orderBy: 'metadata.price' }) works for custom numeric fields
- Exact timestamp precision (no 1-minute bucketing)
- O(K log S) sort independent of total entity count
New files: src/indexes/columnStore/ (types, format, tail buffer, cursor,
manifest, coordinator — ~700 lines). 101 new unit tests covering binary
format round-trips, CRC validation, sort, filter, range, deletion,
multi-segment merge, persistence, and multi-value (words) fields.
Deleted: metadataIndex-automatic-bucketing.test.ts (bucketing behavior
eliminated by exact-precision column store). Sparse index write path
removed from addToIndex/removeFromIndex. Sparse index legacy code still
present as dead code pending cleanup in next commit.
|
||
|---|---|---|
| .. | ||
| aggregation.test.ts | ||
| all-apis-comprehensive.test.ts | ||
| api-parameter-validation.test.ts | ||
| azure-storage.test.ts | ||
| batchImportWithRelations.test.ts | ||
| brainy-complete.integration.test.ts | ||
| brainy-core.integration.test.ts | ||
| brainy-phase1c-integration.test.ts | ||
| bun-compile-test.ts | ||
| clear-persistence.test.ts | ||
| clear-vfs.test.ts | ||
| commit-ref-bug.test.ts | ||
| commit-state-capture.test.ts | ||
| count-synchronization.test.ts | ||
| cow-commit-storage.test.ts | ||
| cow-full-integration.test.ts | ||
| empty-tree-bug.test.ts | ||
| entity-confidence-weight.test.ts | ||
| find-unified-integration.test.ts | ||
| fork-persistence.test.ts | ||
| gcs-native-storage.test.ts | ||
| gcs-persistence-fix.test.ts | ||
| get-relations-fix.test.ts | ||
| history-ref-resolution-bug.test.ts | ||
| hnsw-rebuild.test.ts | ||
| hybrid-search-cow.test.ts | ||
| initial-commit-null-parent.test.ts | ||
| memory-enhancements-v5.11.0.test.ts | ||
| metadata-only-comprehensive.test.ts | ||
| metadata-vector-exclusion.test.ts | ||
| migration.test.ts | ||
| orderby-sort-bug.test.ts | ||
| phase3TypeFirstQuery.integration.test.ts | ||
| readAfterWrite.test.ts | ||
| relationship-intelligence.test.ts | ||
| remaining-apis.test.ts | ||
| s3-storage.test.ts | ||
| smart-import.test.ts | ||
| storage-batch-operations.test.ts | ||
| typeAwareHNSW.integration.test.ts | ||
| update-asymmetry.test.ts | ||
| versioning.test.ts | ||
| vfs-and-graph-entities.test.ts | ||
| vfs-api-wiring.test.ts | ||
| vfs-debug.test.ts | ||
| vfs-historical-reads.test.ts | ||
| vfs-import-verification.test.ts | ||
| vfs-knowledge-separation.test.ts | ||
| vfs-performance-v5.11.1.test.ts | ||
| vfs-versioning.test.ts | ||
| wasm-embeddings.test.ts | ||