fix: getNouns cursor pagination re-scanned the first page forever (permanent CPU loop)

The shard-scan pagination adapter is offset-based and ignored the cursor, so
getNouns({ pagination: { cursor } }) re-returned the first page on every cursor
call. Harmless until 7.32.1 made totalCount the true dataset total — after which
hasMore correctly stays true until a caller has paged through everything. A
caller that paginates by cursor (cursor = page.nextCursor) — notably aggregate
backfill over an already-populated store — then looped forever, re-walking the
entire entity shard tree each iteration and pegging 1-2 CPU cores permanently on
the JS main thread, with zero queries or traffic. (Pre-7.32.1 the same loop
ended after one page, silently backfilling only the first 500 entities — an
incomplete aggregate.)

getNouns() now treats the cursor as an opaque, advancing offset token, so cursor
pagination advances and terminates exactly like offset pagination — and an
aggregate backfill streams the whole corpus exactly once (no longer truncated,
no longer looping). Hardened the backfill loop to pure offset pagination as
defense in depth.

Regression (reproduces the infinite loop, fails fast on any re-scan):
tests/unit/storage/getNouns-cursor-pagination.test.ts
This commit is contained in:
David Snelling 2026-06-22 18:00:01 -07:00
parent 526aaad18f
commit 6721c52ad7
4 changed files with 209 additions and 21 deletions

View file

@ -10089,22 +10089,23 @@ export class Brainy<T = any> implements BrainyInterface<T> {
index.beginBackfill(name)
// Pure OFFSET pagination: advance by the number of items actually returned
// and stop as soon as the store reports no more (or returns an empty page).
// Driving this off `offset` alone — never re-feeding `nextCursor` — makes
// the loop provably terminating regardless of how the adapter implements
// cursors (an earlier cursor-fed variant re-fetched page 0 forever on a
// store larger than one page).
const PAGE = 500
let offset = 0
let cursor: string | undefined
for (;;) {
const page = await this.storage.getNouns({
pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset }
pagination: { limit: PAGE, offset }
})
for (const noun of page.items) {
index.backfillEntity(name, noun as unknown as Record<string, unknown>)
}
offset += page.items.length
if (!page.hasMore || page.items.length === 0) break
if (page.nextCursor) {
cursor = page.nextCursor
} else {
offset += page.items.length
}
}
index.finishBackfill(name)