feat: unified column store for filtering + sorting at billion scale

Adds a per-field sorted column store (Lucene doc values + roaring bitmap
architecture) that replaces the MetadataIndex sparse index internals for
both filtering and sorting. One system for all field types with exact
precision — no bucketing, no per-entity storage reads.

Column store: binary .cidx segment format, in-memory tail buffers,
LSM-style compaction, k-way merge sort, multi-value support (__words__).
All queries (filter, range, sort, filtered sort) route through the column
store when data is available, falling back to sparse index otherwise.

Key unlocks:
- find({ orderBy: 'createdAt' }) works WITHOUT a filter (previously threw)
- find({ orderBy: 'metadata.price' }) works for custom numeric fields
- Exact timestamp precision (no 1-minute bucketing)
- O(K log S) sort independent of total entity count

New files: src/indexes/columnStore/ (types, format, tail buffer, cursor,
manifest, coordinator — ~700 lines). 101 new unit tests covering binary
format round-trips, CRC validation, sort, filter, range, deletion,
multi-segment merge, persistence, and multi-value (words) fields.

Deleted: metadataIndex-automatic-bucketing.test.ts (bucketing behavior
eliminated by exact-precision column store). Sparse index write path
removed from addToIndex/removeFromIndex. Sparse index legacy code still
present as dead code pending cleanup in next commit.
This commit is contained in:
David Snelling 2026-04-10 11:22:19 -07:00
parent 634ddfb2bb
commit 46583f2d9b
16 changed files with 3846 additions and 521 deletions

View file

@ -125,16 +125,25 @@ describe('find({ orderBy }) sort bug regression', () => {
})
/**
* Unfiltered sort is explicitly rejected with a clear error pointing
* callers at the right pattern. This guards against silent N-scans
* on production workloads.
* Unfiltered sort now works via the unified column store.
* This was the Track 2 motivating use case previously threw an error.
*/
it('orderBy without any filter throws a helpful error', async () => {
await brain.add({ data: 'anything', type: NounType.Concept })
it('orderBy without filter works via column store', async () => {
const id1 = await brain.add({ data: 'first', type: NounType.Concept })
await new Promise((r) => setTimeout(r, 20))
await brain.add({ data: 'second', type: NounType.Concept })
await new Promise((r) => setTimeout(r, 20))
const id3 = await brain.add({ data: 'third', type: NounType.Concept })
await expect(
brain.find({ orderBy: 'createdAt', order: 'desc', limit: 1 })
).rejects.toThrow(/requires a filter/)
const results = await brain.find({
orderBy: 'createdAt',
order: 'desc',
limit: 2
})
expect(results.length).toBeGreaterThanOrEqual(2)
// Newest should be first (desc order)
expect(results[0].id).toBe(id3)
})
})