fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal

- AggregationIndex: defineAggregate before init no longer forces a backfill.
  init reconciles instead of clobbering: the app definition wins, persisted
  state is adopted on hash match, a write landing pre-adoption forces an exact
  rescan, and a successful state load clears the backfill flag. New ready()
  settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
  store walk and every pending aggregate fills from that same walk; the old
  behavior let each concurrent query wipe the others' partial state and start
  its own full walk, so a store under steady aggregate traffic never converged.
  queryAggregate also waits for persisted definitions before deciding an
  aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
  ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
  at 1000 - below the documented 10000 auto floor, reported under a stale
  basis label) is removed; the cap comes from its construction-time basis or
  explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
  are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
  semantics, RELEASES.md 8.5.1 entry.
This commit is contained in:
David Snelling 2026-07-17 09:20:09 -07:00
parent 593bb8b0f9
commit da55be7520
10 changed files with 584 additions and 44 deletions

View file

@ -10,6 +10,39 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
---
## v8.5.1 — 2026-07-17 (aggregation state survives restarts + the query-cap ratchet removed)
Patch release from a production incident (aggregate/count paths taking 4090 s on an idle box
while vector search stayed fast, and every `find({ limit: 5000 })` suddenly failing against an
"auto-configured query limit of 1000"). Three fixes, one cosmetic:
- **Aggregation state is actually adopted on reopen.** The boot pattern `defineAggregate()`
query raced the engine's async state load: the synchronous define always won, flagged a
backfill, and the first query then wiped the just-loaded persisted state and re-walked the
entire store — every restart, forever. Reopening with an unchanged definition now adopts the
persisted state directly (zero scans); a backfill runs only on a real definition change, a
missing/failed state load, or a write that landed before adoption (exactness wins). Apps that
rely on persisted definitions without re-defining at boot also no longer race a spurious
"Aggregate not defined".
- **Backfills are single-flight and batched.** Concurrent queries on a cold aggregate used to
each wipe the others' partial state and start their own full store walk — under steady query
arrival the store never converged (the 4090 s loop). Now all concurrent queries share one
walk, and one walk fills every aggregate pending backfill (M aggregates ≠ M scans).
- **The query-cap "learning" ratchet is removed.** `maxLimit` is a memory-protection bound, but
a hidden tuner shrank it 20 % per recorded query while the lifetime-average query time
exceeded 1 s — down to a floor of 1000, below the documented 10 000 auto floor, with no
practical recovery, and the resulting error blamed "available free memory" (stale basis
label). The cap now comes from its construction-time basis (or your explicit
`maxQueryLimit` / `reservedQueryMemory`) alone and never changes at runtime; query timing is
recorded for diagnostics only.
- **Cosmetic:** the engine's own persistence keys (`__aggregation_*`, `brainy:entityIdMapper`)
no longer log `[Storage] Unknown key format` at boot — they were always routed correctly;
they're now recognized before the warning fires.
Operationally: if a host was bitten, upgrading and restarting is the whole fix — no repair
ritual needed. Setting `maxQueryLimit` explicitly remains the valve that bypasses auto-detection
entirely.
## v8.5.0 — 2026-07-15 (provider access to the fact log + the shared stamp verifier)
Small additive follow-up to 8.4.0, from the native accelerator's first consumption pass: