fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal

- AggregationIndex: defineAggregate before init no longer forces a backfill.
  init reconciles instead of clobbering: the app definition wins, persisted
  state is adopted on hash match, a write landing pre-adoption forces an exact
  rescan, and a successful state load clears the backfill flag. New ready()
  settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
  store walk and every pending aggregate fills from that same walk; the old
  behavior let each concurrent query wipe the others' partial state and start
  its own full walk, so a store under steady aggregate traffic never converged.
  queryAggregate also waits for persisted definitions before deciding an
  aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
  ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
  at 1000 - below the documented 10000 auto floor, reported under a stale
  basis label) is removed; the cap comes from its construction-time basis or
  explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
  are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
  semantics, RELEASES.md 8.5.1 entry.
This commit is contained in:
David Snelling 2026-07-17 09:20:09 -07:00
parent 593bb8b0f9
commit da55be7520
10 changed files with 584 additions and 44 deletions

View file

@ -270,27 +270,24 @@ describe('Zero-Config Parameter Validation', () => {
expect(config.availableMemory).toBeGreaterThan(0)
})
it('should adapt limits based on query performance', () => {
const initialConfig = getValidationConfig()
const initialLimit = initialConfig.maxLimit
// Simulate fast queries with large results
it('never mutates the cap from query timing (telemetry only)', () => {
const initialLimit = getValidationConfig().maxLimit
// Fast queries with large results: no silent growth.
for (let i = 0; i < 10; i++) {
recordQueryPerformance(50, initialLimit * 0.9)
}
const updatedConfig = getValidationConfig()
// Limit might increase if performance is good
expect(updatedConfig.maxLimit).toBeGreaterThanOrEqual(initialLimit)
// Simulate slow queries
for (let i = 0; i < 10; i++) {
recordQueryPerformance(2000, 100)
expect(getValidationConfig().maxLimit).toBe(initialLimit)
// A burst of catastrophically slow queries must not strangle the cap.
// The removed "learning" ratchet shrank it 20% per recorded query down
// to a floor of 1000 — below the documented MIN_AUTO_QUERY_LIMIT — and
// the error message blamed "available free memory" (a production
// incident: every find({ limit: 5000 }) failed on an idle 23GB-free box).
for (let i = 0; i < 50; i++) {
recordQueryPerformance(90_000, 100)
}
const finalConfig = getValidationConfig()
// Limit should decrease if performance is poor
expect(finalConfig.maxLimit).toBeLessThanOrEqual(updatedConfig.maxLimit)
expect(getValidationConfig().maxLimit).toBe(initialLimit)
})
})
})