perf(idle): the flush-request watch is event-driven; the heartbeat is observability
Three idle-burn items from the steady-state audit, and one correction. THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request directory every 500 ms, per brain, for the life of every writer — armed on every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second on a completely idle service, plus a stale-request GC on every one of them. It now uses fs.watch, so the arrival itself wakes it and a request is seen SOONER than the poll saw it. Two concessions ride along, both stated in the code: a 30s safety sweep (fs.watch drops events on some network and fuse filesystems, and the GC needs a tick of its own — two orders of magnitude fewer reads than the poll made), and a fall back to the original 500 ms poll, narrated, on a filesystem that cannot watch at all, because an inspector whose request is never seen waits forever. THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is decided by pid liveness and the fence compares pid + hostname, so no decision anywhere reads the timestamp — and at 10s it was a lock-file write every ten seconds per brain forever, for a value nothing computes with. An operator still sees a heartbeat inside the minute. THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation counter. That counter bumps on every ledger mutation and rebuild boundary, so a provider bumping it on routine work re-emitted the same unchanged line on every read, while one that never bumped could suppress a line whose reasons had genuinely changed. The generation is still reported; it no longer decides whether the line is worth saying. CORRECTION, and it is against my own earlier claim: the idle-flush commit read a reported idle-CPU observation (many stores, no writes, a flush every ~35s, over a core burned) as caused by the flush path. That does not follow — this engine's cadence is write-driven (every trigger runs through noteWriteForPersistence, which only a committed write calls), so something was CALLING flush() on those brains and the caller is still unidentified. The clean-flush gate makes such a call free; it does not account for it. The code comments and the idle lane now say exactly that. Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer makes at most one request-directory read in 8 seconds (the old poll made ~16), and a dropped request is still acked well inside the safety sweep.
This commit is contained in:
parent
417ddb5143
commit
fb1da1c56d
4 changed files with 245 additions and 36 deletions
|
|
@ -2,12 +2,17 @@
|
|||
* @module tests/integration/idle-costs-nothing
|
||||
* @description AN IDLE BRAIN DOES NO WORK.
|
||||
*
|
||||
* Measured on a production process holding 21 brains: with no writes for ten
|
||||
* minutes it printed "All indexes flushed to disk in 216–601ms" per brain
|
||||
* every ~35 seconds and idled at 1.26 cores. Every one of those flushes
|
||||
* re-persisted state identical to what was already on disk — the provider
|
||||
* flushes, the watermark stamps, the generation counter, the entity-tree
|
||||
* stamp — because `flush()` never asked whether anything had changed.
|
||||
* A flush used to re-persist state identical to what was already on disk —
|
||||
* the provider flushes, the watermark stamps, the generation counter, the
|
||||
* entity-tree stamp, roughly 28 writes — because `flush()` never asked whether
|
||||
* anything had changed.
|
||||
*
|
||||
* The field observation that started this: a production process holding 21
|
||||
* brains printed "All indexes flushed to disk in 216–601ms" per brain every
|
||||
* ~35 seconds and idled at 1.26 cores, with no writes for ten minutes. This
|
||||
* engine's cadence is WRITE-DRIVEN, so that observation is NOT explained by
|
||||
* the cadence and is not claimed to be fixed here — what is fixed is that such
|
||||
* a call now costs nothing. Who was calling flush() remains open.
|
||||
*
|
||||
* The laws pinned here:
|
||||
* (a) the persistence cadence arms only on a write — a brain nobody writes
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue