soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

perf(8.0): bound find({ where, orderBy }) sort to the page (CTX-BR-FIND-ORDERBY)

Browse source 
A filtered + ordered find() materialized the FULL sorted match set before slicing
the page — passing k = filteredIds.length to the column store's filteredSortTopK.
A broad filter + orderBy returning 20 rows at billion scale produced hundreds of
millions of sorted ids to discard all but the page (O(matches) heap).

Thread the page bound (offset + limit + the hidden-tier over-fetch) into
getSortedIdsForFilter → filteredSortTopK / sortTopK / the sparse-path slice, so
the sort produces only the page. Ordering and pagination are unchanged.

From cor's 2026-06-24 co-release scaling audit (item 1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
David Snelling 2026-06-23 15:01:49 -07:00
parent 82dde92077
commit 450084b6ce
5 changed files with 93 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

17
src/brainy.ts
View file

@ -4763,11 +4763,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Apply sorting if requested, otherwise just filter
let filteredIds: string[]
if (params.orderBy) {
// Get sorted IDs using production-scale sorted filtering
// Get sorted IDs using production-scale sorted filtering. Bound the sort to
// the page (offset+limit) plus the hidden over-fetch — so a broad filter +
// orderBy returning 20 rows doesn't materialize every matching sorted id.
const sortTopK = (params.offset || 0) + (params.limit || 10) + hiddenIds.size
filteredIds = await this.metadataIndex.getSortedIdsForFilter(
filter,
params.orderBy,
params.order || 'asc'
params.order || 'asc',
sortTopK
)
} else {
// Just filter without sorting
@ -5049,15 +5053,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
if (!params.query && !params.connected) {
// Apply sorting if requested for metadata-only queries
if (params.orderBy) {
// Page-bounded sort (offset+limit + hidden over-fetch) so we don't
// materialize every matching sorted id to return one page.
const limit = params.limit || 10
const offset = params.offset || 0
const sortedIds = await this.metadataIndex.getSortedIdsForFilter(
preResolvedFilter,
params.orderBy,
params.order || 'asc'
params.order || 'asc',
offset + limit + hiddenIds.size
)
// Paginate sorted IDs BEFORE loading entities (production-scale!)
const limit = params.limit || 10
const offset = params.offset || 0
const pageIds = sortedIds.slice(offset, offset + limit)
// Batch-load entities for paginated results (10x faster on GCS)