perf(8.0): bound find({ where, orderBy }) sort to the page (CTX-BR-FIND-ORDERBY)

A filtered + ordered find() materialized the FULL sorted match set before slicing
the page — passing k = filteredIds.length to the column store's filteredSortTopK.
A broad filter + orderBy returning 20 rows at billion scale produced hundreds of
millions of sorted ids to discard all but the page (O(matches) heap).

Thread the page bound (offset + limit + the hidden-tier over-fetch) into
getSortedIdsForFilter → filteredSortTopK / sortTopK / the sparse-path slice, so
the sort produces only the page. Ordering and pagination are unchanged.

From cor's 2026-06-24 co-release scaling audit (item 1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
David Snelling 2026-06-23 15:01:49 -07:00
parent 82dde92077
commit 450084b6ce
5 changed files with 93 additions and 10 deletions

View file

@ -2121,13 +2121,19 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* @param filter - Metadata filter criteria (uses roaring bitmaps)
* @param orderBy - Field name to sort by (e.g., 'createdAt', 'title')
* @param order - Sort direction: 'asc' (default) or 'desc'
* @param topK - Optional page bound: produce only the top `K` sorted ids
* (`offset + limit`) instead of the full sorted match set. A broad filter +
* orderBy that returns one page no longer materializes hundreds of millions of
* sorted ids at billion scale the column store's top-K heap produces only the
* page. Omit for the full sorted set.
* @returns Promise<string[]> - Entity IDs sorted by specified field
*
*/
async getSortedIdsForFilter(
filter: any,
orderBy: string,
order: 'asc' | 'desc' = 'asc'
order: 'asc' | 'desc' = 'asc',
topK?: number
): Promise<string[]> {
// Column store path: O(K log S) sort via k-way merge across segments.
// No per-entity storage reads, no precision loss from bucketing.
@ -2146,13 +2152,17 @@ export class MetadataIndexManager implements MetadataIndexProvider {
const intId = this.idMapper.getInt(id)
if (intId !== undefined) filterBitmap.add(intId)
}
// Page-bounded: produce only the top `topK` (offset+limit), not every
// match, so a broad filter + orderBy returning one page stays O(matches
// log K) heap, not a full sort materialization.
const k = topK !== undefined ? Math.min(topK, filteredIds.length) : filteredIds.length
sortedIntIds = await this.columnStore.filteredSortTopK(
filterBitmap, orderBy, order, filteredIds.length
filterBitmap, orderBy, order, k
)
} else {
// Unfiltered sort — column store handles the full entity set efficiently
sortedIntIds = await this.columnStore.sortTopK(
orderBy, order, this.idMapper.size
orderBy, order, topK !== undefined ? Math.min(topK, this.idMapper.size) : this.idMapper.size
)
}
@ -2195,7 +2205,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
return order === 'asc' ? comparison : -comparison
})
return idValuePairs.map(p => p.id)
const sorted = idValuePairs.map(p => p.id)
return topK !== undefined ? sorted.slice(0, topK) : sorted
}
/**