fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal

- AggregationIndex: defineAggregate before init no longer forces a backfill.
  init reconciles instead of clobbering: the app definition wins, persisted
  state is adopted on hash match, a write landing pre-adoption forces an exact
  rescan, and a successful state load clears the backfill flag. New ready()
  settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
  store walk and every pending aggregate fills from that same walk; the old
  behavior let each concurrent query wipe the others' partial state and start
  its own full walk, so a store under steady aggregate traffic never converged.
  queryAggregate also waits for persisted definitions before deciding an
  aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
  ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
  at 1000 - below the documented 10000 auto floor, reported under a stale
  basis label) is removed; the cap comes from its construction-time basis or
  explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
  are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
  semantics, RELEASES.md 8.5.1 entry.
This commit is contained in:
David Snelling 2026-07-17 09:20:09 -07:00
parent 593bb8b0f9
commit da55be7520
10 changed files with 584 additions and 44 deletions

View file

@ -11,7 +11,13 @@ No batch jobs. No scheduled recalculations. Aggregates stay current with every w
**Defining over existing data:** if you define an aggregate on a store that already holds
matching entities, Brainy backfills it from those entities on the first query (a one-time scan,
then purely incremental). So `defineAggregate()` behaves the same whether you define it before
or after the data exists — including when a persisted brain reopens already populated.
or after the data exists.
**Reopening a persisted brain:** aggregate state persists across restarts. Re-defining the
same aggregate at boot (the normal declarative pattern) adopts the persisted state directly —
no rescan. A backfill scan runs only when the definition actually changed, when no persisted
state exists, or when the state failed to load; and however many aggregates need backfilling,
they share a single scan.
## Quick Start

View file

@ -40,6 +40,10 @@ Brainy picks `maxLimit` from the first of these that's available:
Worked example: a 4 GB Cloud Run container picks priority 3 → `floor(4 GB × 0.25 / 25 KB) = floor(40 960) = 40 000` results. A 900 MB free-memory box on priority 4 gets `floor(900 MB / 25 KB) = ~36 000`.
The cap is fixed at construction and never changes at runtime. Query timing is recorded
for diagnostics only — a burst of slow queries cannot silently shrink the cap, and the
auto-detected tiers (3 and 4) never go below a floor of 10 000.
> **Calibration note.** Pre-7.30.2 used 100 KB per result instead of 25 KB, which produced caps that were 4× too tight for typical workloads (an 8 KB / result reality). 7.30.2 recalibrated to match observed entity sizes; existing `limit: 10_000` safety patterns now pass silently on any reasonably-sized box.
## What happens when you exceed the cap