perf(8.0): cursor pagination for the verb walk — full edge pagination O(N²) → O(N)
getVerbsWithPagination was offset-only: it re-scanned from shard 0 on every page and early-terminated at offset+limit, so a full edge walk was O(N²) (a consumer measured ~19k edges → 27s paging at 900/page). It now accepts an opaque resume token (cv1:<shard>:<verbId>) that resumes the shard walk immediately AFTER the last returned verb — O(N) total at any page size, no scale cliff. - Stable within-shard order (sort by verb id); ids come from the path so verbs skipped by the cursor are never read. - Cursor supersedes offset when present; offset path preserved for back-compat, and offset callers now get a usable nextCursor so they can switch to cursor paging. Foreign/malformed tokens decode to null → offset fallback (no throw). - The relationships stream generator now uses cursor (was offset → O(N²)). - Parity with the noun side (getNounsWithPagination already cursored). Test: tests/unit/storage/verb-cursor-pagination.test.ts — a full cursored walk visits every edge exactly once (no dup, no skip), matches the offset walk's set, and a foreign cursor falls back gracefully.
This commit is contained in:
parent
a914313e3a
commit
682e786cc3
3 changed files with 221 additions and 41 deletions
|
|
@ -8279,15 +8279,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}.bind(this),
|
||||
|
||||
// Stream relationships efficiently
|
||||
// Stream relationships efficiently. Cursor-based when the adapter supports it
|
||||
// (resumes after the last verb — O(N) for a full walk) and falls back to offset
|
||||
// otherwise. Offset paging here re-scanned from the start every page (O(N²)).
|
||||
relationships: async function* (this: Brainy<T>, filter?: { type?: string, sourceId?: string, targetId?: string }) {
|
||||
let offset = 0
|
||||
let cursor: string | undefined
|
||||
const batchSize = 100
|
||||
let hasMore = true
|
||||
|
||||
while (hasMore) {
|
||||
for (;;) {
|
||||
const result = await this.storage.getVerbs({
|
||||
pagination: { offset, limit: batchSize },
|
||||
pagination: cursor ? { limit: batchSize, cursor } : { offset, limit: batchSize },
|
||||
filter
|
||||
})
|
||||
|
||||
|
|
@ -8295,8 +8297,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
yield verb
|
||||
}
|
||||
|
||||
hasMore = result.hasMore
|
||||
offset += batchSize
|
||||
if (!result.hasMore || result.items.length === 0) break
|
||||
if (result.nextCursor) {
|
||||
cursor = result.nextCursor
|
||||
} else {
|
||||
offset += result.items.length
|
||||
}
|
||||
}
|
||||
}.bind(this),
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue