fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal

- AggregationIndex: defineAggregate before init no longer forces a backfill.
  init reconciles instead of clobbering: the app definition wins, persisted
  state is adopted on hash match, a write landing pre-adoption forces an exact
  rescan, and a successful state load clears the backfill flag. New ready()
  settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
  store walk and every pending aggregate fills from that same walk; the old
  behavior let each concurrent query wipe the others' partial state and start
  its own full walk, so a store under steady aggregate traffic never converged.
  queryAggregate also waits for persisted definitions before deciding an
  aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
  ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
  at 1000 - below the documented 10000 auto floor, reported under a stale
  basis label) is removed; the cap comes from its construction-time basis or
  explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
  are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
  semantics, RELEASES.md 8.5.1 entry.
This commit is contained in:
David Snelling 2026-07-17 09:20:09 -07:00
parent 593bb8b0f9
commit da55be7520
10 changed files with 584 additions and 44 deletions

View file

@ -209,8 +209,10 @@ export interface ValidationConfigOptions {
}
/**
* Auto-configured limits based on system resources
* These adapt to available memory and observed performance
* Auto-configured limits based on system resources.
* Derived from memory (explicit overrides > reserved memory > container limit >
* free memory). Query timing is recorded for diagnostics only it never
* changes the cap (see `recordQuery`).
*/
export class ValidationConfig {
private static instance: ValidationConfig | null = null
@ -323,24 +325,23 @@ export class ValidationConfig {
}
/**
* Learn from actual usage to adjust limits
* Record query timing for diagnostics. Telemetry ONLY never mutates the cap.
*
* `maxLimit` is a MEMORY-protection bound; query duration says nothing about
* memory-per-result, so duration must never drive it. An earlier version
* "learned" here: while the lifetime-average query time exceeded 1s it shrank
* `maxLimit` by 20% per recorded query down to a floor of 1000 below the
* documented `MIN_AUTO_QUERY_LIMIT` (10 000) and with no recovery once slow
* samples poisoned the cumulative average. On a production host a burst of
* slow aggregate queries silently strangled every consumer's `find()` to
* 1000 while the error message blamed "available free memory" exactly the
* silent throttling this module's own contract forbids. The cap now comes
* from its construction-time basis (or explicit overrides) alone.
*/
recordQuery(duration: number, resultCount: number) {
void resultCount
this.queryCount++
this.avgQueryTime = (this.avgQueryTime * (this.queryCount - 1) + duration) / this.queryCount
// Only auto-adjust if not using explicit overrides
if (this.limitBasis !== 'override') {
// If queries are consistently fast with large results, increase limits
if (this.avgQueryTime < 100 && resultCount > this.maxLimit * 0.8) {
this.maxLimit = Math.min(this.maxLimit * 1.5, 100000)
}
// If queries are slow, reduce limits
if (this.avgQueryTime > 1000) {
this.maxLimit = Math.max(this.maxLimit * 0.8, 1000)
}
}
}
}