fix: exception-safe aggregation backfill + generation-verified adoption + loud open-path guards

- Aggregation backfill walks build into a STAGING map and swap in atomically
  on completion. A mid-walk failure drops the staging map — the previous live
  state keeps serving, the aggregate stays flagged pending, and the storage
  error surfaces to the failing query. Previously the walk wiped live state
  before a scan that could throw, never cleared the pending flag on failure,
  and re-ran a full walk on every subsequent query: a silent wipe/walk/throw
  loop at the caller's retry rate.
- Failed walks are latched: retries within a 30s cooldown rethrow the recorded
  error instantly instead of re-walking, so a tight caller-side retry loop
  costs one loud error per query, never a full store walk per query.
- Persisted aggregation state is stamped with the store's committed generation
  at flush; reopen adoption requires stamp equality. Stale state (unclean
  shutdown) or over-counting state (a log truncation on a copied store pulled
  the watermark back) triggers exactly one loud rescan, never a silent adopt.
- The backfill/adoption path narrates: adoption decisions, walk start/finish
  with counts and duration, and failures all log by default.
- getNouns/getVerbs refuse a supplied-but-undecodable pagination cursor with a
  loud error instead of silently restarting the walk at offset 0 (which
  re-served page 1 forever to any while(hasMore) caller).
- The graph cold-load verb walk aborts loudly on a missing or non-advancing
  cursor with hasMore=true.
- A versioned index provider whose generation is AHEAD of the committed
  watermark (torn copy / crash-recovery truncation) is now named loudly at
  open, alongside the existing behind-direction message.
This commit is contained in:
David Snelling 2026-07-17 16:00:11 -07:00
parent 01a7f3dd01
commit a77b064bd7
7 changed files with 335 additions and 36 deletions

View file

@ -10,6 +10,44 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
---
## v8.5.2 — 2026-07-17 (aggregation backfill: exception-safe, generation-verified, and loud)
Hardening patch from a migration incident (a byte-copied store on new hardware; the service
entered a silent full-CPU loop at boot). Four changes, all in the aggregation engine's
backfill/adoption path:
- **Backfill walks are exception-safe and non-destructive.** A rescan now builds into a
staging map and swaps in atomically on completion; a mid-walk failure drops the staging map,
keeps the previous live state serving, and surfaces the storage error to the failing query.
Previously the walk wiped live state *before* a scan that could throw, never cleared the
pending flag on failure, and re-ran a full walk on every subsequent query — a silent
wipe/walk/throw loop at the caller's retry rate.
- **Failed walks are latched.** After a walk fails, retries within a 30-second cooldown rethrow
the recorded error instantly instead of re-walking — a tight caller-side retry loop now costs
one loud error per query, never a full store walk per query.
- **Adoption is generation-verified.** Persisted aggregation state is stamped with the store's
committed generation at flush; reopen adoption requires the stamp to equal the current
watermark. Stale state (unclean shutdown) or over-counting state (a fact-log truncation on a
copied store pulled the watermark back) triggers exactly one loud rescan — never a silent
adopt. Pre-8.5.2 state on generation-aware stores rescans once after upgrade, then is stamped.
- **The path narrates.** Adoption decisions, no-adoptable-state outcomes, walk start/finish
(entity count + duration), and walk failures all log by default; a non-advancing storage
pagination cursor aborts the walk loudly instead of looping forever.
Plus three guards from a full audit of every loop in the open/init path:
- **Invalid pagination cursors fail loudly.** A supplied-but-undecodable resume token to
`getNouns`/`getVerbs` used to silently restart the walk at offset 0 — to a `while(hasMore)`
caller that re-serves page 1 forever (an unbounded silent CPU loop). It now throws with a
clear message instead.
- **The graph cold-load verb walk has a stall guard.** `hasMore=true` with a missing or
non-advancing cursor aborts loudly instead of re-reading the same page forever.
- **A derived index AHEAD of the store is named at open.** Brainy already surfaced a provider
generation *behind* the committed watermark; the *ahead* direction (the signature of a
byte-copy of a live service, or a log truncation during crash recovery) now logs a loud
warning explaining what happened and that `brain.repairIndex()` forces a heal — instead of
passing unnamed into whatever the derived index does next.
## v8.5.1 — 2026-07-17 (aggregation state survives restarts + the query-cap ratchet removed)
Patch release from a production incident (aggregate/count paths taking 4090 s on an idle box