perf(idle): the flush-request watch is event-driven; the heartbeat is observability
Three idle-burn items from the steady-state audit, and one correction. THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request directory every 500 ms, per brain, for the life of every writer — armed on every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second on a completely idle service, plus a stale-request GC on every one of them. It now uses fs.watch, so the arrival itself wakes it and a request is seen SOONER than the poll saw it. Two concessions ride along, both stated in the code: a 30s safety sweep (fs.watch drops events on some network and fuse filesystems, and the GC needs a tick of its own — two orders of magnitude fewer reads than the poll made), and a fall back to the original 500 ms poll, narrated, on a filesystem that cannot watch at all, because an inspector whose request is never seen waits forever. THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is decided by pid liveness and the fence compares pid + hostname, so no decision anywhere reads the timestamp — and at 10s it was a lock-file write every ten seconds per brain forever, for a value nothing computes with. An operator still sees a heartbeat inside the minute. THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation counter. That counter bumps on every ledger mutation and rebuild boundary, so a provider bumping it on routine work re-emitted the same unchanged line on every read, while one that never bumped could suppress a line whose reasons had genuinely changed. The generation is still reported; it no longer decides whether the line is worth saying. CORRECTION, and it is against my own earlier claim: the idle-flush commit read a reported idle-CPU observation (many stores, no writes, a flush every ~35s, over a core burned) as caused by the flush path. That does not follow — this engine's cadence is write-driven (every trigger runs through noteWriteForPersistence, which only a committed write calls), so something was CALLING flush() on those brains and the caller is still unidentified. The clean-flush gate makes such a call free; it does not account for it. The code comments and the idle lane now say exactly that. Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer makes at most one request-directory read in 8 seconds (the old poll made ~16), and a dropped request is still acked well inside the safety sweep.
This commit is contained in:
parent
417ddb5143
commit
fb1da1c56d
4 changed files with 245 additions and 36 deletions
|
|
@ -749,12 +749,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* Whether a write has been committed since the last flush that ran. THE
|
||||
* ENGINE DOES NO PERIODIC WORK WITHOUT A CAUSE: a brain nobody has written
|
||||
* to has nothing to make durable, and a flush over it must cost nothing and
|
||||
* say nothing. Measured on a production process holding 21 brains: with no
|
||||
* writes for ten minutes it still printed "All indexes flushed to disk in
|
||||
* 216–601ms" per brain every ~35s and idled at 1.26 cores, because a flush
|
||||
* called every provider, stamped the watermarks, persisted the generation
|
||||
* counter and re-stamped the entity tree whether or not anything had
|
||||
* changed.
|
||||
* say nothing. Before this, a flush called every provider, stamped the
|
||||
* watermarks, persisted the generation counter and re-stamped the entity
|
||||
* tree whether or not anything had changed — roughly 28 writes for a store
|
||||
* that had not moved.
|
||||
*
|
||||
* WHAT THIS DOES NOT EXPLAIN, stated so nobody reads it as solved: a
|
||||
* production process holding 21 brains printed "All indexes flushed to disk
|
||||
* in 216-601ms" per brain every ~35s and idled at 1.26 cores with no writes
|
||||
* for ten minutes. This engine's cadence is WRITE-DRIVEN — every trigger
|
||||
* runs through noteWriteForPersistence, which only a committed write calls —
|
||||
* so something was calling flush() on those brains, and this gate makes such
|
||||
* a call free rather than accounting for it. The caller is still unidentified.
|
||||
*/
|
||||
private _dirtySinceLastFlush = false
|
||||
private _persistIdleTimer: ReturnType<typeof setTimeout> | null = null
|
||||
|
|
@ -851,7 +857,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// Read-gate narration dedup: a degraded-but-serving or not-ready health
|
||||
// report narrates via prodLog.warn ONCE per (provider, report.generation) —
|
||||
// never once per read. Keyed on the provider instance itself.
|
||||
private _lastNarratedHealthGeneration = new Map<unknown, number>()
|
||||
/**
|
||||
* The last health narration emitted per provider, keyed by its CONTENT.
|
||||
*
|
||||
* This used to dedupe on the provider's `generation` counter, which bumps on
|
||||
* every ledger mutation and every rebuild boundary — so a provider that
|
||||
* bumps its generation on routine work re-emitted the same unchanged health
|
||||
* line on every read that consulted it, and a provider that never bumped
|
||||
* could suppress a line whose reasons had genuinely changed. The dedupe key
|
||||
* is now what the line SAYS: an unchanged verdict is silent however the
|
||||
* generation moves, and a changed verdict is always heard.
|
||||
*/
|
||||
private _lastNarratedHealth = new Map<unknown, string>()
|
||||
|
||||
constructor(config?: BrainyConfig) {
|
||||
// The reserved-field write policy died with the field-addressing law:
|
||||
|
|
@ -12366,11 +12383,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// committed since the last flush, so every step below would re-persist
|
||||
// state identical to what is already on disk — provider flushes, the
|
||||
// watermark stamps, the generation counter, the entity-tree stamp — and
|
||||
// print two lines announcing it. On a process holding 21 brains that
|
||||
// no-op cost 1.26 cores at idle. The witness is set by every committed
|
||||
// print two lines announcing it. The witness is set by every committed
|
||||
// write (see noteWriteForPersistence) and cleared here; a write landing
|
||||
// DURING this flush sets it again, so it is never lost — the next flush
|
||||
// does that write's work.
|
||||
// does that write's work. This makes an unexplained flush FREE; it does
|
||||
// not explain one (see _dirtySinceLastFlush).
|
||||
if (!this._dirtySinceLastFlush) {
|
||||
return
|
||||
}
|
||||
|
|
@ -17243,12 +17260,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
|
||||
if (assessment.reasons.length > 0 && assessment.report != null) {
|
||||
const generation = assessment.report.generation
|
||||
if (this._lastNarratedHealthGeneration.get(provider) !== generation) {
|
||||
this._lastNarratedHealthGeneration.set(provider, generation)
|
||||
prodLog.warn(
|
||||
`[Brainy] ${assessment.report.provider} health (generation ${generation}): ` +
|
||||
assessment.reasons.join('; ')
|
||||
)
|
||||
// Dedupe by CONTENT, not by the provider's generation counter — see
|
||||
// _lastNarratedHealth. The generation is still REPORTED (an operator
|
||||
// wants to know which generation produced the verdict); it just no
|
||||
// longer decides whether the line is worth saying.
|
||||
const line =
|
||||
`[Brainy] ${assessment.report.provider} health (generation ${generation}): ` +
|
||||
assessment.reasons.join('; ')
|
||||
const key = `${assessment.report.provider}\u0000${assessment.reasons.join('; ')}`
|
||||
if (this._lastNarratedHealth.get(provider) !== key) {
|
||||
this._lastNarratedHealth.set(provider, key)
|
||||
prodLog.warn(line)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue