SELF-ENGINE-LIFECYCLE-SPRINT + BRAINY-PROD-LATENCY-TRIAD, the four asks:
(a) brain.flush() persists aggregation state stamped at the committed
generation. The stamp used to advance only at close(), so a long-lived
writer that flushes but never closes — the primary production shape —
left every write window behind the stamp, and ANY unclean exit forced a
whole-store backfill walk (per-entity work, measured >60s and
door-starving on a 9k-row production brain) on the first stats call.
(b) BEHIND-stamp adoption becomes adopt + INCREMENTAL CATCH-UP: the exact
missing window (stamp, committed] resolves its affected-id set from the
fact log and reconciles each entity with time-travel before/after reads
(asOf at both window bounds) through the same delta algebra the live
hooks use — cost bounded by writes since the last flush, never store
size, and exact under interleaving because reconciliation targets the
FIXED window end while later writes chain through hooks. Oversized
windows (>5000 affected) and unreadable windows demote to the announced
rescan — never a silent partial serve.
(c) The native provider's parallel rebuildAggregate — on the contract since
8.x but never invoked anywhere — is now the backfill walk's preferred
door: one call per aggregate with source-matched entities, replacing
the per-entity FFI stream.
(d) A delete whose before-image is unavailable can no longer SKIP the
aggregation hook silently (counts drifted upward forever): both delete
paths (remove() and transact) flag an exact rescan, loudly.
Pins: integration (flush stamp; unclean-exit reopen → exact counts through
an add + group-move + delete window with the walk spy proving ZERO
whole-store walks) + unit (provider rebuild invoked once with filtered
entities; flagAllForRescan; reconcile delta algebra). Gates: unit 1913/1913
· integration 760 · conformance 27/27.
- AggregationIndex: defineAggregate before init no longer forces a backfill.
init reconciles instead of clobbering: the app definition wins, persisted
state is adopted on hash match, a write landing pre-adoption forces an exact
rescan, and a successful state load clears the backfill flag. New ready()
settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
store walk and every pending aggregate fills from that same walk; the old
behavior let each concurrent query wipe the others' partial state and start
its own full walk, so a store under steady aggregate traffic never converged.
queryAggregate also waits for persisted definitions before deciding an
aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
at 1000 - below the documented 10000 auto floor, reported under a stale
basis label) is removed; the cap comes from its construction-time basis or
explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
semantics, RELEASES.md 8.5.1 entry.
The debounced materialization caught its failure with an empty `catch (() => {})`,
silently leaving the materialized Measurement entity stale; the aggregation-index
init() state-load failure was swallowed the same way, leaving aggregates reading
empty with no signal. Both are non-fatal (values rebuild via backfill-on-query),
but a silent stale/empty read violates loud-errors-never-quiet-losses. Both now
emit a loud warning naming the affected group / cause.
Add 'percentile' (with a 'p' fraction in [0,1]) and 'distinctCount' to the
aggregation engine. Both are exact, computed from a per-metric value multiset
(MetricState.valueCounts) maintained incrementally and delete-safe; percentile
uses numpy-linear interpolation. The multiset is JSON-serializable so results
survive persistence. 35 aggregation unit tests pass.
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
