perf(8.0): bound find({ where, orderBy }) sort to the page (CTX-BR-FIND-ORDERBY)
A filtered + ordered find() materialized the FULL sorted match set before slicing the page — passing k = filteredIds.length to the column store's filteredSortTopK. A broad filter + orderBy returning 20 rows at billion scale produced hundreds of millions of sorted ids to discard all but the page (O(matches) heap). Thread the page bound (offset + limit + the hidden-tier over-fetch) into getSortedIdsForFilter → filteredSortTopK / sortTopK / the sparse-path slice, so the sort produces only the page. Ordering and pagination are unchanged. From cor's 2026-06-24 co-release scaling audit (item 1). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
82dde92077
commit
450084b6ce
5 changed files with 93 additions and 10 deletions
|
|
@ -2121,13 +2121,19 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
* @param filter - Metadata filter criteria (uses roaring bitmaps)
|
||||
* @param orderBy - Field name to sort by (e.g., 'createdAt', 'title')
|
||||
* @param order - Sort direction: 'asc' (default) or 'desc'
|
||||
* @param topK - Optional page bound: produce only the top `K` sorted ids
|
||||
* (`offset + limit`) instead of the full sorted match set. A broad filter +
|
||||
* orderBy that returns one page no longer materializes hundreds of millions of
|
||||
* sorted ids at billion scale — the column store's top-K heap produces only the
|
||||
* page. Omit for the full sorted set.
|
||||
* @returns Promise<string[]> - Entity IDs sorted by specified field
|
||||
*
|
||||
*/
|
||||
async getSortedIdsForFilter(
|
||||
filter: any,
|
||||
orderBy: string,
|
||||
order: 'asc' | 'desc' = 'asc'
|
||||
order: 'asc' | 'desc' = 'asc',
|
||||
topK?: number
|
||||
): Promise<string[]> {
|
||||
// Column store path: O(K log S) sort via k-way merge across segments.
|
||||
// No per-entity storage reads, no precision loss from bucketing.
|
||||
|
|
@ -2146,13 +2152,17 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
const intId = this.idMapper.getInt(id)
|
||||
if (intId !== undefined) filterBitmap.add(intId)
|
||||
}
|
||||
// Page-bounded: produce only the top `topK` (offset+limit), not every
|
||||
// match, so a broad filter + orderBy returning one page stays O(matches
|
||||
// log K) heap, not a full sort materialization.
|
||||
const k = topK !== undefined ? Math.min(topK, filteredIds.length) : filteredIds.length
|
||||
sortedIntIds = await this.columnStore.filteredSortTopK(
|
||||
filterBitmap, orderBy, order, filteredIds.length
|
||||
filterBitmap, orderBy, order, k
|
||||
)
|
||||
} else {
|
||||
// Unfiltered sort — column store handles the full entity set efficiently
|
||||
sortedIntIds = await this.columnStore.sortTopK(
|
||||
orderBy, order, this.idMapper.size
|
||||
orderBy, order, topK !== undefined ? Math.min(topK, this.idMapper.size) : this.idMapper.size
|
||||
)
|
||||
}
|
||||
|
||||
|
|
@ -2195,7 +2205,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
return order === 'asc' ? comparison : -comparison
|
||||
})
|
||||
|
||||
return idValuePairs.map(p => p.id)
|
||||
const sorted = idValuePairs.map(p => p.id)
|
||||
return topK !== undefined ? sorted.slice(0, topK) : sorted
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue