fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal
- AggregationIndex: defineAggregate before init no longer forces a backfill. init reconciles instead of clobbering: the app definition wins, persisted state is adopted on hash match, a write landing pre-adoption forces an exact rescan, and a successful state load clears the backfill flag. New ready() settles every adoption decision before query paths consult backfill state. - brainy: backfills are single-flight and batched. Concurrent queries share one store walk and every pending aggregate fills from that same walk; the old behavior let each concurrent query wipe the others' partial state and start its own full walk, so a store under steady aggregate traffic never converged. queryAggregate also waits for persisted definitions before deciding an aggregate does not exist. - paramValidation: recordQuery is telemetry-only. The duration-based cap ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored at 1000 - below the documented 10000 auto floor, reported under a stale basis label) is removed; the cap comes from its construction-time basis or explicit overrides alone. - storage: __aggregation_* and singleton system keys (brainy:entityIdMapper) are recognized before the unknown-key warning fires; routing is unchanged. - docs: find-limits cap-immutability note, aggregation reopen/adoption semantics, RELEASES.md 8.5.1 entry.
This commit is contained in:
parent
593bb8b0f9
commit
da55be7520
10 changed files with 584 additions and 44 deletions
|
|
@ -209,8 +209,10 @@ export interface ValidationConfigOptions {
|
|||
}
|
||||
|
||||
/**
|
||||
* Auto-configured limits based on system resources
|
||||
* These adapt to available memory and observed performance
|
||||
* Auto-configured limits based on system resources.
|
||||
* Derived from memory (explicit overrides > reserved memory > container limit >
|
||||
* free memory). Query timing is recorded for diagnostics only — it never
|
||||
* changes the cap (see `recordQuery`).
|
||||
*/
|
||||
export class ValidationConfig {
|
||||
private static instance: ValidationConfig | null = null
|
||||
|
|
@ -323,24 +325,23 @@ export class ValidationConfig {
|
|||
}
|
||||
|
||||
/**
|
||||
* Learn from actual usage to adjust limits
|
||||
* Record query timing for diagnostics. Telemetry ONLY — never mutates the cap.
|
||||
*
|
||||
* `maxLimit` is a MEMORY-protection bound; query duration says nothing about
|
||||
* memory-per-result, so duration must never drive it. An earlier version
|
||||
* "learned" here: while the lifetime-average query time exceeded 1s it shrank
|
||||
* `maxLimit` by 20% per recorded query down to a floor of 1000 — below the
|
||||
* documented `MIN_AUTO_QUERY_LIMIT` (10 000) and with no recovery once slow
|
||||
* samples poisoned the cumulative average. On a production host a burst of
|
||||
* slow aggregate queries silently strangled every consumer's `find()` to
|
||||
* 1000 while the error message blamed "available free memory" — exactly the
|
||||
* silent throttling this module's own contract forbids. The cap now comes
|
||||
* from its construction-time basis (or explicit overrides) alone.
|
||||
*/
|
||||
recordQuery(duration: number, resultCount: number) {
|
||||
void resultCount
|
||||
this.queryCount++
|
||||
this.avgQueryTime = (this.avgQueryTime * (this.queryCount - 1) + duration) / this.queryCount
|
||||
|
||||
// Only auto-adjust if not using explicit overrides
|
||||
if (this.limitBasis !== 'override') {
|
||||
// If queries are consistently fast with large results, increase limits
|
||||
if (this.avgQueryTime < 100 && resultCount > this.maxLimit * 0.8) {
|
||||
this.maxLimit = Math.min(this.maxLimit * 1.5, 100000)
|
||||
}
|
||||
|
||||
// If queries are slow, reduce limits
|
||||
if (this.avgQueryTime > 1000) {
|
||||
this.maxLimit = Math.max(this.maxLimit * 0.8, 1000)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue