fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal
- AggregationIndex: defineAggregate before init no longer forces a backfill. init reconciles instead of clobbering: the app definition wins, persisted state is adopted on hash match, a write landing pre-adoption forces an exact rescan, and a successful state load clears the backfill flag. New ready() settles every adoption decision before query paths consult backfill state. - brainy: backfills are single-flight and batched. Concurrent queries share one store walk and every pending aggregate fills from that same walk; the old behavior let each concurrent query wipe the others' partial state and start its own full walk, so a store under steady aggregate traffic never converged. queryAggregate also waits for persisted definitions before deciding an aggregate does not exist. - paramValidation: recordQuery is telemetry-only. The duration-based cap ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored at 1000 - below the documented 10000 auto floor, reported under a stale basis label) is removed; the cap comes from its construction-time basis or explicit overrides alone. - storage: __aggregation_* and singleton system keys (brainy:entityIdMapper) are recognized before the unknown-key warning fires; routing is unchanged. - docs: find-limits cap-immutability note, aggregation reopen/adoption semantics, RELEASES.md 8.5.1 entry.
This commit is contained in:
parent
593bb8b0f9
commit
da55be7520
10 changed files with 584 additions and 44 deletions
|
|
@ -11,7 +11,13 @@ No batch jobs. No scheduled recalculations. Aggregates stay current with every w
|
|||
**Defining over existing data:** if you define an aggregate on a store that already holds
|
||||
matching entities, Brainy backfills it from those entities on the first query (a one-time scan,
|
||||
then purely incremental). So `defineAggregate()` behaves the same whether you define it before
|
||||
or after the data exists — including when a persisted brain reopens already populated.
|
||||
or after the data exists.
|
||||
|
||||
**Reopening a persisted brain:** aggregate state persists across restarts. Re-defining the
|
||||
same aggregate at boot (the normal declarative pattern) adopts the persisted state directly —
|
||||
no rescan. A backfill scan runs only when the definition actually changed, when no persisted
|
||||
state exists, or when the state failed to load; and however many aggregates need backfilling,
|
||||
they share a single scan.
|
||||
|
||||
## Quick Start
|
||||
|
||||
|
|
|
|||
|
|
@ -40,6 +40,10 @@ Brainy picks `maxLimit` from the first of these that's available:
|
|||
|
||||
Worked example: a 4 GB Cloud Run container picks priority 3 → `floor(4 GB × 0.25 / 25 KB) = floor(40 960) = 40 000` results. A 900 MB free-memory box on priority 4 gets `floor(900 MB / 25 KB) = ~36 000`.
|
||||
|
||||
The cap is fixed at construction and never changes at runtime. Query timing is recorded
|
||||
for diagnostics only — a burst of slow queries cannot silently shrink the cap, and the
|
||||
auto-detected tiers (3 and 4) never go below a floor of 10 000.
|
||||
|
||||
> **Calibration note.** Pre-7.30.2 used 100 KB per result instead of 25 KB, which produced caps that were 4× too tight for typical workloads (an 8 KB / result reality). 7.30.2 recalibrated to match observed entity sizes; existing `limit: 10_000` safety patterns now pass silently on any reasonably-sized box.
|
||||
|
||||
## What happens when you exceed the cap
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue