perf(8.0): cursor pagination for the verb walk — full edge pagination O(N²) → O(N)

getVerbsWithPagination was offset-only: it re-scanned from shard 0 on every page
and early-terminated at offset+limit, so a full edge walk was O(N²) (a consumer
measured ~19k edges → 27s paging at 900/page). It now accepts an opaque resume
token (cv1:<shard>:<verbId>) that resumes the shard walk immediately AFTER the
last returned verb — O(N) total at any page size, no scale cliff.

- Stable within-shard order (sort by verb id); ids come from the path so verbs
  skipped by the cursor are never read.
- Cursor supersedes offset when present; offset path preserved for back-compat,
  and offset callers now get a usable nextCursor so they can switch to cursor
  paging. Foreign/malformed tokens decode to null → offset fallback (no throw).
- The relationships stream generator now uses cursor (was offset → O(N²)).
- Parity with the noun side (getNounsWithPagination already cursored).

Test: tests/unit/storage/verb-cursor-pagination.test.ts — a full cursored walk
visits every edge exactly once (no dup, no skip), matches the offset walk's set,
and a foreign cursor falls back gracefully.
This commit is contained in:
David Snelling 2026-06-20 16:55:02 -07:00
parent a914313e3a
commit 682e786cc3
3 changed files with 221 additions and 41 deletions

View file

@ -8279,15 +8279,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
}
}.bind(this),
// Stream relationships efficiently
// Stream relationships efficiently. Cursor-based when the adapter supports it
// (resumes after the last verb — O(N) for a full walk) and falls back to offset
// otherwise. Offset paging here re-scanned from the start every page (O(N²)).
relationships: async function* (this: Brainy<T>, filter?: { type?: string, sourceId?: string, targetId?: string }) {
let offset = 0
let cursor: string | undefined
const batchSize = 100
let hasMore = true
while (hasMore) {
for (;;) {
const result = await this.storage.getVerbs({
pagination: { offset, limit: batchSize },
pagination: cursor ? { limit: batchSize, cursor } : { offset, limit: batchSize },
filter
})
@ -8295,8 +8297,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
yield verb
}
hasMore = result.hasMore
offset += batchSize
if (!result.hasMore || result.items.length === 0) break
if (result.nextCursor) {
cursor = result.nextCursor
} else {
offset += result.items.length
}
}
}.bind(this),