feat: unified column store for filtering + sorting at billion scale

Adds a per-field sorted column store (Lucene doc values + roaring bitmap
architecture) that replaces the MetadataIndex sparse index internals for
both filtering and sorting. One system for all field types with exact
precision — no bucketing, no per-entity storage reads.

Column store: binary .cidx segment format, in-memory tail buffers,
LSM-style compaction, k-way merge sort, multi-value support (__words__).
All queries (filter, range, sort, filtered sort) route through the column
store when data is available, falling back to sparse index otherwise.

Key unlocks:
- find({ orderBy: 'createdAt' }) works WITHOUT a filter (previously threw)
- find({ orderBy: 'metadata.price' }) works for custom numeric fields
- Exact timestamp precision (no 1-minute bucketing)
- O(K log S) sort independent of total entity count

New files: src/indexes/columnStore/ (types, format, tail buffer, cursor,
manifest, coordinator — ~700 lines). 101 new unit tests covering binary
format round-trips, CRC validation, sort, filter, range, deletion,
multi-segment merge, persistence, and multi-value (words) fields.

Deleted: metadataIndex-automatic-bucketing.test.ts (bucketing behavior
eliminated by exact-precision column store). Sparse index write path
removed from addToIndex/removeFromIndex. Sparse index legacy code still
present as dead code pending cleanup in next commit.
This commit is contained in:
David Snelling 2026-04-10 11:22:19 -07:00
parent 634ddfb2bb
commit 46583f2d9b
16 changed files with 3846 additions and 521 deletions

View file

@ -2166,19 +2166,31 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const limit = params.limit || 20
const offset = params.offset || 0
// orderBy without any filter is not supported — it would require
// O(N) work across every entity in storage. Consumers must supply
// a filter (`type`, `where`, or `excludeVFS: true`) so the sort
// runs over a bounded, roaring-bitmap-filtered set. The long-term
// fix is a dedicated time-ordered segment index (Track 2).
// Unfiltered sort: column store handles this at O(K log S) scale.
// No per-entity storage reads, no bucketing precision loss.
if (params.orderBy) {
throw new Error(
`find({ orderBy: '${params.orderBy}' }) requires a filter. ` +
`Add 'type', 'where', or 'excludeVFS: true' so the sort runs ` +
`over a bounded set. Unfiltered sort over all entities is not ` +
`scalable with the current index and is tracked as a dedicated ` +
`time-ordered segment index (Track 2).`
const k = limit + offset
const sortedIntIds = await this.metadataIndex.columnStore.sortTopK(
params.orderBy,
params.order || 'asc',
k
)
// Convert int IDs to UUIDs and paginate
const idMapper = this.metadataIndex.getIdMapper()
const allUuids = sortedIntIds
.map(intId => idMapper.getUuid(intId))
.filter((uuid): uuid is string => uuid !== undefined)
const pageIds = allUuids.slice(offset, offset + limit)
const entitiesMap = await this.batchGet(pageIds)
for (const id of pageIds) {
const entity = entitiesMap.get(id)
if (entity) {
results.push(this.createResult(id, 1.0, entity))
}
}
return results
}
// ExcludeVFS helper - exclude VFS infrastructure entities