feat: unified column store for filtering + sorting at billion scale
Adds a per-field sorted column store (Lucene doc values + roaring bitmap
architecture) that replaces the MetadataIndex sparse index internals for
both filtering and sorting. One system for all field types with exact
precision — no bucketing, no per-entity storage reads.
Column store: binary .cidx segment format, in-memory tail buffers,
LSM-style compaction, k-way merge sort, multi-value support (__words__).
All queries (filter, range, sort, filtered sort) route through the column
store when data is available, falling back to sparse index otherwise.
Key unlocks:
- find({ orderBy: 'createdAt' }) works WITHOUT a filter (previously threw)
- find({ orderBy: 'metadata.price' }) works for custom numeric fields
- Exact timestamp precision (no 1-minute bucketing)
- O(K log S) sort independent of total entity count
New files: src/indexes/columnStore/ (types, format, tail buffer, cursor,
manifest, coordinator — ~700 lines). 101 new unit tests covering binary
format round-trips, CRC validation, sort, filter, range, deletion,
multi-segment merge, persistence, and multi-value (words) fields.
Deleted: metadataIndex-automatic-bucketing.test.ts (bucketing behavior
eliminated by exact-precision column store). Sparse index write path
removed from addToIndex/removeFromIndex. Sparse index legacy code still
present as dead code pending cleanup in next commit.
This commit is contained in:
parent
634ddfb2bb
commit
46583f2d9b
16 changed files with 3846 additions and 521 deletions
|
|
@ -2166,19 +2166,31 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
const limit = params.limit || 20
|
||||
const offset = params.offset || 0
|
||||
|
||||
// orderBy without any filter is not supported — it would require
|
||||
// O(N) work across every entity in storage. Consumers must supply
|
||||
// a filter (`type`, `where`, or `excludeVFS: true`) so the sort
|
||||
// runs over a bounded, roaring-bitmap-filtered set. The long-term
|
||||
// fix is a dedicated time-ordered segment index (Track 2).
|
||||
// Unfiltered sort: column store handles this at O(K log S) scale.
|
||||
// No per-entity storage reads, no bucketing precision loss.
|
||||
if (params.orderBy) {
|
||||
throw new Error(
|
||||
`find({ orderBy: '${params.orderBy}' }) requires a filter. ` +
|
||||
`Add 'type', 'where', or 'excludeVFS: true' so the sort runs ` +
|
||||
`over a bounded set. Unfiltered sort over all entities is not ` +
|
||||
`scalable with the current index and is tracked as a dedicated ` +
|
||||
`time-ordered segment index (Track 2).`
|
||||
const k = limit + offset
|
||||
const sortedIntIds = await this.metadataIndex.columnStore.sortTopK(
|
||||
params.orderBy,
|
||||
params.order || 'asc',
|
||||
k
|
||||
)
|
||||
|
||||
// Convert int IDs to UUIDs and paginate
|
||||
const idMapper = this.metadataIndex.getIdMapper()
|
||||
const allUuids = sortedIntIds
|
||||
.map(intId => idMapper.getUuid(intId))
|
||||
.filter((uuid): uuid is string => uuid !== undefined)
|
||||
const pageIds = allUuids.slice(offset, offset + limit)
|
||||
|
||||
const entitiesMap = await this.batchGet(pageIds)
|
||||
for (const id of pageIds) {
|
||||
const entity = entitiesMap.get(id)
|
||||
if (entity) {
|
||||
results.push(this.createResult(id, 1.0, entity))
|
||||
}
|
||||
}
|
||||
return results
|
||||
}
|
||||
|
||||
// ExcludeVFS helper - exclude VFS infrastructure entities
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue