feat: unified column store for filtering + sorting at billion scale
Adds a per-field sorted column store (Lucene doc values + roaring bitmap
architecture) that replaces the MetadataIndex sparse index internals for
both filtering and sorting. One system for all field types with exact
precision — no bucketing, no per-entity storage reads.
Column store: binary .cidx segment format, in-memory tail buffers,
LSM-style compaction, k-way merge sort, multi-value support (__words__).
All queries (filter, range, sort, filtered sort) route through the column
store when data is available, falling back to sparse index otherwise.
Key unlocks:
- find({ orderBy: 'createdAt' }) works WITHOUT a filter (previously threw)
- find({ orderBy: 'metadata.price' }) works for custom numeric fields
- Exact timestamp precision (no 1-minute bucketing)
- O(K log S) sort independent of total entity count
New files: src/indexes/columnStore/ (types, format, tail buffer, cursor,
manifest, coordinator — ~700 lines). 101 new unit tests covering binary
format round-trips, CRC validation, sort, filter, range, deletion,
multi-segment merge, persistence, and multi-value (words) fields.
Deleted: metadataIndex-automatic-bucketing.test.ts (bucketing behavior
eliminated by exact-precision column store). Sparse index write path
removed from addToIndex/removeFromIndex. Sparse index legacy code still
present as dead code pending cleanup in next commit.
This commit is contained in:
parent
634ddfb2bb
commit
46583f2d9b
16 changed files with 3846 additions and 521 deletions
|
|
@ -125,16 +125,25 @@ describe('find({ orderBy }) sort bug regression', () => {
|
|||
})
|
||||
|
||||
/**
|
||||
* Unfiltered sort is explicitly rejected with a clear error pointing
|
||||
* callers at the right pattern. This guards against silent N-scans
|
||||
* on production workloads.
|
||||
* Unfiltered sort now works via the unified column store.
|
||||
* This was the Track 2 motivating use case — previously threw an error.
|
||||
*/
|
||||
it('orderBy without any filter throws a helpful error', async () => {
|
||||
await brain.add({ data: 'anything', type: NounType.Concept })
|
||||
it('orderBy without filter works via column store', async () => {
|
||||
const id1 = await brain.add({ data: 'first', type: NounType.Concept })
|
||||
await new Promise((r) => setTimeout(r, 20))
|
||||
await brain.add({ data: 'second', type: NounType.Concept })
|
||||
await new Promise((r) => setTimeout(r, 20))
|
||||
const id3 = await brain.add({ data: 'third', type: NounType.Concept })
|
||||
|
||||
await expect(
|
||||
brain.find({ orderBy: 'createdAt', order: 'desc', limit: 1 })
|
||||
).rejects.toThrow(/requires a filter/)
|
||||
const results = await brain.find({
|
||||
orderBy: 'createdAt',
|
||||
order: 'desc',
|
||||
limit: 2
|
||||
})
|
||||
|
||||
expect(results.length).toBeGreaterThanOrEqual(2)
|
||||
// Newest should be first (desc order)
|
||||
expect(results[0].id).toBe(id3)
|
||||
})
|
||||
})
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue