perf(idle): the flush-request watch is event-driven; the heartbeat is observability

Three idle-burn items from the steady-state audit, and one correction.

THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request
directory every 500 ms, per brain, for the life of every writer — armed on
every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second
on a completely idle service, plus a stale-request GC on every one of them. It now
uses fs.watch, so the arrival itself wakes it and a request is seen SOONER
than the poll saw it. Two concessions ride along, both stated in the code: a
30s safety sweep (fs.watch drops events on some network and fuse filesystems,
and the GC needs a tick of its own — two orders of magnitude fewer reads than
the poll made), and a fall back to the original 500 ms poll, narrated, on a
filesystem that cannot watch at all, because an inspector whose request is
never seen waits forever.

THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is
decided by pid liveness and the fence compares pid + hostname, so no decision
anywhere reads the timestamp — and at 10s it was a lock-file write every ten
seconds per brain forever, for a value nothing computes with. An operator
still sees a heartbeat inside the minute.

THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation
counter. That counter bumps on every ledger mutation and rebuild boundary, so
a provider bumping it on routine work re-emitted the same unchanged line on
every read, while one that never bumped could suppress a line whose reasons
had genuinely changed. The generation is still reported; it no longer decides
whether the line is worth saying.

CORRECTION, and it is against my own earlier claim: the idle-flush commit read
a reported idle-CPU observation (many stores, no writes, a flush every ~35s,
over a core burned) as caused by
the flush path. That does not follow — this engine's cadence is write-driven
(every trigger runs through noteWriteForPersistence, which only a committed
write calls), so something was CALLING flush() on those brains and the caller
is still unidentified. The clean-flush gate makes such a call free; it does not
account for it. The code comments and the idle lane now say exactly that.

Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer
makes at most one request-directory read in 8 seconds (the old poll made ~16),
and a dropped request is still acked well inside the safety sweep.
This commit is contained in:
David Snelling 2026-08-28 11:25:47 -07:00
parent 417ddb5143
commit fb1da1c56d
4 changed files with 245 additions and 36 deletions

View file

@ -749,12 +749,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* Whether a write has been committed since the last flush that ran. THE
* ENGINE DOES NO PERIODIC WORK WITHOUT A CAUSE: a brain nobody has written
* to has nothing to make durable, and a flush over it must cost nothing and
* say nothing. Measured on a production process holding 21 brains: with no
* writes for ten minutes it still printed "All indexes flushed to disk in
* 216601ms" per brain every ~35s and idled at 1.26 cores, because a flush
* called every provider, stamped the watermarks, persisted the generation
* counter and re-stamped the entity tree whether or not anything had
* changed.
* say nothing. Before this, a flush called every provider, stamped the
* watermarks, persisted the generation counter and re-stamped the entity
* tree whether or not anything had changed roughly 28 writes for a store
* that had not moved.
*
* WHAT THIS DOES NOT EXPLAIN, stated so nobody reads it as solved: a
* production process holding 21 brains printed "All indexes flushed to disk
* in 216-601ms" per brain every ~35s and idled at 1.26 cores with no writes
* for ten minutes. This engine's cadence is WRITE-DRIVEN every trigger
* runs through noteWriteForPersistence, which only a committed write calls
* so something was calling flush() on those brains, and this gate makes such
* a call free rather than accounting for it. The caller is still unidentified.
*/
private _dirtySinceLastFlush = false
private _persistIdleTimer: ReturnType<typeof setTimeout> | null = null
@ -851,7 +857,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Read-gate narration dedup: a degraded-but-serving or not-ready health
// report narrates via prodLog.warn ONCE per (provider, report.generation) —
// never once per read. Keyed on the provider instance itself.
private _lastNarratedHealthGeneration = new Map<unknown, number>()
/**
* The last health narration emitted per provider, keyed by its CONTENT.
*
* This used to dedupe on the provider's `generation` counter, which bumps on
* every ledger mutation and every rebuild boundary so a provider that
* bumps its generation on routine work re-emitted the same unchanged health
* line on every read that consulted it, and a provider that never bumped
* could suppress a line whose reasons had genuinely changed. The dedupe key
* is now what the line SAYS: an unchanged verdict is silent however the
* generation moves, and a changed verdict is always heard.
*/
private _lastNarratedHealth = new Map<unknown, string>()
constructor(config?: BrainyConfig) {
// The reserved-field write policy died with the field-addressing law:
@ -12366,11 +12383,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// committed since the last flush, so every step below would re-persist
// state identical to what is already on disk — provider flushes, the
// watermark stamps, the generation counter, the entity-tree stamp — and
// print two lines announcing it. On a process holding 21 brains that
// no-op cost 1.26 cores at idle. The witness is set by every committed
// print two lines announcing it. The witness is set by every committed
// write (see noteWriteForPersistence) and cleared here; a write landing
// DURING this flush sets it again, so it is never lost — the next flush
// does that write's work.
// does that write's work. This makes an unexplained flush FREE; it does
// not explain one (see _dirtySinceLastFlush).
if (!this._dirtySinceLastFlush) {
return
}
@ -17243,12 +17260,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
if (assessment.reasons.length > 0 && assessment.report != null) {
const generation = assessment.report.generation
if (this._lastNarratedHealthGeneration.get(provider) !== generation) {
this._lastNarratedHealthGeneration.set(provider, generation)
prodLog.warn(
`[Brainy] ${assessment.report.provider} health (generation ${generation}): ` +
assessment.reasons.join('; ')
)
// Dedupe by CONTENT, not by the provider's generation counter — see
// _lastNarratedHealth. The generation is still REPORTED (an operator
// wants to know which generation produced the verdict); it just no
// longer decides whether the line is worth saying.
const line =
`[Brainy] ${assessment.report.provider} health (generation ${generation}): ` +
assessment.reasons.join('; ')
const key = `${assessment.report.provider}\u0000${assessment.reasons.join('; ')}`
if (this._lastNarratedHealth.get(provider) !== key) {
this._lastNarratedHealth.set(provider, key)
prodLog.warn(line)
}
}