fix(persistence): the idle flush trigger debounces under load — deferred to the floor, never dropped, never a flush-per-gap amplifier
Some checks failed
CI / Node 22 (push) Successful in 12m14s
CI / Node 24 (push) Successful in 12m7s
CI / Bun (latest) (push) Has been cancelled

An internal report from cross-engine write-path instrumentation: with
individual writes slower than the idle window (a contended disk), every
inter-write gap looked idle and fired a background full flush — 15 extra
flushes during 100 contended adds, amplifying the very pressure that
slowed the writes. The law now: an idle fire landing within the spacing
floor of the last flush DEFERS to the floor boundary instead of flushing;
the floor is min(interval, 10× the CONFIGURED idle window) — scaled to
caller intent (a tiny idle window keeps fast idle-driven durability;
default 2s/30s config gets a 20s floor), derived from the configured
idle, never from a deferred re-arm delay (which would compound into
runaway deferral). Deferred is never dropped: a lone write on a
then-quiet store still persists at the floor without any further write
arriving.

Pins: the contended-shape pin (six slow-spaced writes fire ≤2 idle
flushes, not one per gap; then still persist) + the original quiet-store
idle pin unchanged. Unit 2055/2055.
This commit is contained in:
David Snelling 2026-08-10 12:15:02 -07:00
parent d1651f986c
commit a50726e6a8
2 changed files with 58 additions and 2 deletions

View file

@ -2342,10 +2342,42 @@ export class Brainy<T = any> implements BrainyInterface<T> {
}
if (this._persistIdleTimer) clearTimeout(this._persistIdleTimer)
this.armIdleFlushTimer(idleMs, intervalMs)
}
/**
* @description Arm the idle-flush timer DEBOUNCED UNDER LOAD. The idle
* trigger exists to make a QUIET system durable fast; it must never add
* flush pressure to a BUSY one. When individual writes are slower than
* the idle window (a contended disk), every inter-write gap looks like
* "idle" and would fire a full flush per write a measured 15-flush
* amplifier during 100 contended adds on a production-shaped box. The
* law: an idle fire landing within `intervalMs` of the last flush DEFERS
* (re-arms for the remaining interval) rather than flushing deferred,
* never dropped, so a lone write on a then-quiet system still persists at
* the interval boundary without any further write arriving; a genuinely
* quiet system (last flush long past) flushes on idle exactly as before.
*/
private armIdleFlushTimer(idleMs: number, intervalMs: number, delayMs = idleMs): void {
// The idle-fire spacing floor: 10× the CONFIGURED idle window, capped by
// the interval — always derived from idleMs, never from a deferred
// re-arm delay (recomputing from the delay compounds into runaway
// deferral). Scales with intent — a caller configuring a tiny idle
// window gets fast idle-driven durability (small floor); default config
// (2s idle / 30s interval) gets a 20s floor, capping the contended-disk
// shape at ~1 idle flush per 20s instead of one per inter-write gap.
const floorMs = Math.min(intervalMs, idleMs * 10)
const timer = setTimeout(() => {
this._persistIdleTimer = null
if (this._persistDirtyWrites > 0) this.kickBackgroundFlush('idle')
}, idleMs)
if (this._persistDirtyWrites === 0) return
const sinceFlush = Date.now() - this._persistLastFlushAt
if (sinceFlush >= floorMs) {
this.kickBackgroundFlush('idle')
} else {
// Deferred, never dropped: land exactly at the floor boundary.
this.armIdleFlushTimer(idleMs, intervalMs, Math.max(idleMs, floorMs - sinceFlush))
}
}, delayMs)
// Never hold the process open for a cadence timer.
;(timer as { unref?: () => void }).unref?.()
this._persistIdleTimer = timer