perf(idle): the flush-request watch is event-driven; the heartbeat is observability

Three idle-burn items from the steady-state audit, and one correction.

THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request
directory every 500 ms, per brain, for the life of every writer — armed on
every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second
on a completely idle service, plus a stale-request GC on every one of them. It now
uses fs.watch, so the arrival itself wakes it and a request is seen SOONER
than the poll saw it. Two concessions ride along, both stated in the code: a
30s safety sweep (fs.watch drops events on some network and fuse filesystems,
and the GC needs a tick of its own — two orders of magnitude fewer reads than
the poll made), and a fall back to the original 500 ms poll, narrated, on a
filesystem that cannot watch at all, because an inspector whose request is
never seen waits forever.

THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is
decided by pid liveness and the fence compares pid + hostname, so no decision
anywhere reads the timestamp — and at 10s it was a lock-file write every ten
seconds per brain forever, for a value nothing computes with. An operator
still sees a heartbeat inside the minute.

THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation
counter. That counter bumps on every ledger mutation and rebuild boundary, so
a provider bumping it on routine work re-emitted the same unchanged line on
every read, while one that never bumped could suppress a line whose reasons
had genuinely changed. The generation is still reported; it no longer decides
whether the line is worth saying.

CORRECTION, and it is against my own earlier claim: the idle-flush commit read
a reported idle-CPU observation (many stores, no writes, a flush every ~35s,
over a core burned) as caused by
the flush path. That does not follow — this engine's cadence is write-driven
(every trigger runs through noteWriteForPersistence, which only a committed
write calls), so something was CALLING flush() on those brains and the caller
is still unidentified. The clean-flush gate makes such a call free; it does not
account for it. The code comments and the idle lane now say exactly that.

Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer
makes at most one request-directory read in 8 seconds (the old poll made ~16),
and a dropped request is still acked well inside the safety sweep.
This commit is contained in:
David Snelling 2026-08-28 11:25:47 -07:00
parent 417ddb5143
commit fb1da1c56d
4 changed files with 245 additions and 36 deletions

View file

@ -2,12 +2,17 @@
* @module tests/integration/idle-costs-nothing
* @description AN IDLE BRAIN DOES NO WORK.
*
* Measured on a production process holding 21 brains: with no writes for ten
* minutes it printed "All indexes flushed to disk in 216601ms" per brain
* every ~35 seconds and idled at 1.26 cores. Every one of those flushes
* re-persisted state identical to what was already on disk the provider
* flushes, the watermark stamps, the generation counter, the entity-tree
* stamp because `flush()` never asked whether anything had changed.
* A flush used to re-persist state identical to what was already on disk
* the provider flushes, the watermark stamps, the generation counter, the
* entity-tree stamp, roughly 28 writes because `flush()` never asked whether
* anything had changed.
*
* The field observation that started this: a production process holding 21
* brains printed "All indexes flushed to disk in 216601ms" per brain every
* ~35 seconds and idled at 1.26 cores, with no writes for ten minutes. This
* engine's cadence is WRITE-DRIVEN, so that observation is NOT explained by
* the cadence and is not claimed to be fixed here what is fixed is that such
* a call now costs nothing. Who was calling flush() remains open.
*
* The laws pinned here:
* (a) the persistence cadence arms only on a write a brain nobody writes