fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal
- AggregationIndex: defineAggregate before init no longer forces a backfill. init reconciles instead of clobbering: the app definition wins, persisted state is adopted on hash match, a write landing pre-adoption forces an exact rescan, and a successful state load clears the backfill flag. New ready() settles every adoption decision before query paths consult backfill state. - brainy: backfills are single-flight and batched. Concurrent queries share one store walk and every pending aggregate fills from that same walk; the old behavior let each concurrent query wipe the others' partial state and start its own full walk, so a store under steady aggregate traffic never converged. queryAggregate also waits for persisted definitions before deciding an aggregate does not exist. - paramValidation: recordQuery is telemetry-only. The duration-based cap ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored at 1000 - below the documented 10000 auto floor, reported under a stale basis label) is removed; the cap comes from its construction-time basis or explicit overrides alone. - storage: __aggregation_* and singleton system keys (brainy:entityIdMapper) are recognized before the unknown-key warning fires; routing is unchanged. - docs: find-limits cap-immutability note, aggregation reopen/adoption semantics, RELEASES.md 8.5.1 entry.
This commit is contained in:
parent
593bb8b0f9
commit
da55be7520
10 changed files with 584 additions and 44 deletions
33
RELEASES.md
33
RELEASES.md
|
|
@ -10,6 +10,39 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
|
|||
|
||||
---
|
||||
|
||||
## v8.5.1 — 2026-07-17 (aggregation state survives restarts + the query-cap ratchet removed)
|
||||
|
||||
Patch release from a production incident (aggregate/count paths taking 40–90 s on an idle box
|
||||
while vector search stayed fast, and every `find({ limit: 5000 })` suddenly failing against an
|
||||
"auto-configured query limit of 1000"). Three fixes, one cosmetic:
|
||||
|
||||
- **Aggregation state is actually adopted on reopen.** The boot pattern `defineAggregate()` →
|
||||
query raced the engine's async state load: the synchronous define always won, flagged a
|
||||
backfill, and the first query then wiped the just-loaded persisted state and re-walked the
|
||||
entire store — every restart, forever. Reopening with an unchanged definition now adopts the
|
||||
persisted state directly (zero scans); a backfill runs only on a real definition change, a
|
||||
missing/failed state load, or a write that landed before adoption (exactness wins). Apps that
|
||||
rely on persisted definitions without re-defining at boot also no longer race a spurious
|
||||
"Aggregate not defined".
|
||||
- **Backfills are single-flight and batched.** Concurrent queries on a cold aggregate used to
|
||||
each wipe the others' partial state and start their own full store walk — under steady query
|
||||
arrival the store never converged (the 40–90 s loop). Now all concurrent queries share one
|
||||
walk, and one walk fills every aggregate pending backfill (M aggregates ≠ M scans).
|
||||
- **The query-cap "learning" ratchet is removed.** `maxLimit` is a memory-protection bound, but
|
||||
a hidden tuner shrank it 20 % per recorded query while the lifetime-average query time
|
||||
exceeded 1 s — down to a floor of 1000, below the documented 10 000 auto floor, with no
|
||||
practical recovery, and the resulting error blamed "available free memory" (stale basis
|
||||
label). The cap now comes from its construction-time basis (or your explicit
|
||||
`maxQueryLimit` / `reservedQueryMemory`) alone and never changes at runtime; query timing is
|
||||
recorded for diagnostics only.
|
||||
- **Cosmetic:** the engine's own persistence keys (`__aggregation_*`, `brainy:entityIdMapper`)
|
||||
no longer log `[Storage] Unknown key format` at boot — they were always routed correctly;
|
||||
they're now recognized before the warning fires.
|
||||
|
||||
Operationally: if a host was bitten, upgrading and restarting is the whole fix — no repair
|
||||
ritual needed. Setting `maxQueryLimit` explicitly remains the valve that bypasses auto-detection
|
||||
entirely.
|
||||
|
||||
## v8.5.0 — 2026-07-15 (provider access to the fact log + the shared stamp verifier)
|
||||
|
||||
Small additive follow-up to 8.4.0, from the native accelerator's first consumption pass:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue