fix: exception-safe aggregation backfill + generation-verified adoption + loud open-path guards
- Aggregation backfill walks build into a STAGING map and swap in atomically on completion. A mid-walk failure drops the staging map — the previous live state keeps serving, the aggregate stays flagged pending, and the storage error surfaces to the failing query. Previously the walk wiped live state before a scan that could throw, never cleared the pending flag on failure, and re-ran a full walk on every subsequent query: a silent wipe/walk/throw loop at the caller's retry rate. - Failed walks are latched: retries within a 30s cooldown rethrow the recorded error instantly instead of re-walking, so a tight caller-side retry loop costs one loud error per query, never a full store walk per query. - Persisted aggregation state is stamped with the store's committed generation at flush; reopen adoption requires stamp equality. Stale state (unclean shutdown) or over-counting state (a log truncation on a copied store pulled the watermark back) triggers exactly one loud rescan, never a silent adopt. - The backfill/adoption path narrates: adoption decisions, walk start/finish with counts and duration, and failures all log by default. - getNouns/getVerbs refuse a supplied-but-undecodable pagination cursor with a loud error instead of silently restarting the walk at offset 0 (which re-served page 1 forever to any while(hasMore) caller). - The graph cold-load verb walk aborts loudly on a missing or non-advancing cursor with hasMore=true. - A versioned index provider whose generation is AHEAD of the committed watermark (torn copy / crash-recovery truncation) is now named loudly at open, alongside the existing behind-direction message.
This commit is contained in:
parent
01a7f3dd01
commit
a77b064bd7
7 changed files with 335 additions and 36 deletions
38
RELEASES.md
38
RELEASES.md
|
|
@ -10,6 +10,44 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
|
|||
|
||||
---
|
||||
|
||||
## v8.5.2 — 2026-07-17 (aggregation backfill: exception-safe, generation-verified, and loud)
|
||||
|
||||
Hardening patch from a migration incident (a byte-copied store on new hardware; the service
|
||||
entered a silent full-CPU loop at boot). Four changes, all in the aggregation engine's
|
||||
backfill/adoption path:
|
||||
|
||||
- **Backfill walks are exception-safe and non-destructive.** A rescan now builds into a
|
||||
staging map and swaps in atomically on completion; a mid-walk failure drops the staging map,
|
||||
keeps the previous live state serving, and surfaces the storage error to the failing query.
|
||||
Previously the walk wiped live state *before* a scan that could throw, never cleared the
|
||||
pending flag on failure, and re-ran a full walk on every subsequent query — a silent
|
||||
wipe/walk/throw loop at the caller's retry rate.
|
||||
- **Failed walks are latched.** After a walk fails, retries within a 30-second cooldown rethrow
|
||||
the recorded error instantly instead of re-walking — a tight caller-side retry loop now costs
|
||||
one loud error per query, never a full store walk per query.
|
||||
- **Adoption is generation-verified.** Persisted aggregation state is stamped with the store's
|
||||
committed generation at flush; reopen adoption requires the stamp to equal the current
|
||||
watermark. Stale state (unclean shutdown) or over-counting state (a fact-log truncation on a
|
||||
copied store pulled the watermark back) triggers exactly one loud rescan — never a silent
|
||||
adopt. Pre-8.5.2 state on generation-aware stores rescans once after upgrade, then is stamped.
|
||||
- **The path narrates.** Adoption decisions, no-adoptable-state outcomes, walk start/finish
|
||||
(entity count + duration), and walk failures all log by default; a non-advancing storage
|
||||
pagination cursor aborts the walk loudly instead of looping forever.
|
||||
|
||||
Plus three guards from a full audit of every loop in the open/init path:
|
||||
|
||||
- **Invalid pagination cursors fail loudly.** A supplied-but-undecodable resume token to
|
||||
`getNouns`/`getVerbs` used to silently restart the walk at offset 0 — to a `while(hasMore)`
|
||||
caller that re-serves page 1 forever (an unbounded silent CPU loop). It now throws with a
|
||||
clear message instead.
|
||||
- **The graph cold-load verb walk has a stall guard.** `hasMore=true` with a missing or
|
||||
non-advancing cursor aborts loudly instead of re-reading the same page forever.
|
||||
- **A derived index AHEAD of the store is named at open.** Brainy already surfaced a provider
|
||||
generation *behind* the committed watermark; the *ahead* direction (the signature of a
|
||||
byte-copy of a live service, or a log truncation during crash recovery) now logs a loud
|
||||
warning explaining what happened and that `brain.repairIndex()` forces a heal — instead of
|
||||
passing unnamed into whatever the derived index does next.
|
||||
|
||||
## v8.5.1 — 2026-07-17 (aggregation state survives restarts + the query-cap ratchet removed)
|
||||
|
||||
Patch release from a production incident (aggregate/count paths taking 40–90 s on an idle box
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue