perf(idle): the flush-request watch is event-driven; the heartbeat is observability

Three idle-burn items from the steady-state audit, and one correction.

THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request
directory every 500 ms, per brain, for the life of every writer — armed on
every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second
on a completely idle service, plus a stale-request GC on every one of them. It now
uses fs.watch, so the arrival itself wakes it and a request is seen SOONER
than the poll saw it. Two concessions ride along, both stated in the code: a
30s safety sweep (fs.watch drops events on some network and fuse filesystems,
and the GC needs a tick of its own — two orders of magnitude fewer reads than
the poll made), and a fall back to the original 500 ms poll, narrated, on a
filesystem that cannot watch at all, because an inspector whose request is
never seen waits forever.

THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is
decided by pid liveness and the fence compares pid + hostname, so no decision
anywhere reads the timestamp — and at 10s it was a lock-file write every ten
seconds per brain forever, for a value nothing computes with. An operator
still sees a heartbeat inside the minute.

THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation
counter. That counter bumps on every ledger mutation and rebuild boundary, so
a provider bumping it on routine work re-emitted the same unchanged line on
every read, while one that never bumped could suppress a line whose reasons
had genuinely changed. The generation is still reported; it no longer decides
whether the line is worth saying.

CORRECTION, and it is against my own earlier claim: the idle-flush commit read
a reported idle-CPU observation (many stores, no writes, a flush every ~35s,
over a core burned) as caused by
the flush path. That does not follow — this engine's cadence is write-driven
(every trigger runs through noteWriteForPersistence, which only a committed
write calls), so something was CALLING flush() on those brains and the caller
is still unidentified. The clean-flush gate makes such a call free; it does not
account for it. The code comments and the idle lane now say exactly that.

Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer
makes at most one request-directory read in 8 seconds (the old poll made ~16),
and a dropped request is still acked well inside the safety sweep.

tests/integration/flush-watcher-event-driven.test.ts

@@ -0,0 +1,94 @@
+/**
+ * @module tests/integration/flush-watcher-event-driven
+ * @description THE FLUSH-REQUEST WATCH IS EVENT-DRIVEN.
+ *
+ * It used to `readdir` the request directory every 500 ms, per brain, for the
+ * life of every writer — armed on every non-reader brain whether or not any
+ * inspector process existed. MEASURED on a production process holding 21
+ * brains: 42 directory reads per second on a completely idle service, plus a
+ * stale-request GC pass on every one of them.
+ *
+ * The law: a request that has not been made is not a cause. The arrival itself
+ * wakes the watcher, so the request is seen SOONER than the poll saw it, and a
+ * slow safety sweep covers filesystems that drop watch events and the GC.
+ */
+
+import { describe, it, expect, afterEach, vi } from 'vitest'
+import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'node:fs'
+import * as nodeFs from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { Brainy } from '../../src/brainy.js'
+import { NounType } from '../../src/types/graphTypes.js'
+
+describe('the flush-request watcher', () => {
+  const dirs: string[] = []
+  const brains: Brainy[] = []
+
+  afterEach(async () => {
+    for (const b of brains.splice(0)) {
+      try { await b.close() } catch { /* already closed */ }
+    }
+    for (const d of dirs.splice(0)) {
+      try { rmSync(d, { recursive: true, force: true }) } catch { /* ignore */ }
+    }
+    vi.restoreAllMocks()
+  })
+
+  async function openWriter(): Promise<{ brain: Brainy; dir: string }> {
+    const dir = mkdtempSync(join(tmpdir(), 'brainy-flush-watch-'))
+    dirs.push(dir)
+    const brain = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } })
+    brains.push(brain)
+    await brain.init()
+    await brain.add({ data: 'a row', type: NounType.Concept })
+    await brain.flush()
+    return { brain, dir }
+  }
+
+  it('does not poll the request directory on an idle writer', async () => {
+    const { dir } = await openWriter()
+    const reqDir = join(dir, 'locks', '_flush_requests')
+
+    // Count real reads of the request directory over a window far longer than
+    // the old 500ms poll (which would have made ~16 of them).
+    const realReaddir = nodeFs.promises.readdir
+    let requestDirReads = 0
+    const spy = vi
+      .spyOn(nodeFs.promises, 'readdir')
+      .mockImplementation((async (p: unknown, ...rest: unknown[]) => {
+        if (String(p) === reqDir) requestDirReads++
+        return (realReaddir as unknown as (...a: unknown[]) => Promise<unknown>)(p, ...rest)
+      }) as typeof nodeFs.promises.readdir)
+
+    await new Promise((r) => setTimeout(r, 8_000))
+    spy.mockRestore()
+
+    // The old poll: 500ms → ~16 reads. The safety sweep is 30s → 0 in this window.
+    expect(requestDirReads).toBeLessThanOrEqual(1)
+  }, 120_000)
+
+  it('answers a request that arrives, without waiting for the sweep', async () => {
+    const { brain, dir } = await openWriter()
+    const reqDir = join(dir, 'locks', '_flush_requests')
+    const ackDir = join(dir, 'locks', '_flush_responses')
+    mkdirSync(reqDir, { recursive: true })
+
+    // Drop a request exactly as an out-of-process inspector does.
+    const id = 'test-request-0001'
+    writeFileSync(join(reqDir, `${id}.req`), JSON.stringify({ at: Date.now() }))
+
+    // The ack must land far sooner than the 30s safety sweep.
+    const deadline = Date.now() + 10_000
+    let acked = false
+    while (Date.now() < deadline) {
+      try {
+        const entries = await nodeFs.promises.readdir(ackDir)
+        if (entries.some((e) => e.startsWith(id))) { acked = true; break }
+      } catch { /* dir not created yet */ }
+      await new Promise((r) => setTimeout(r, 100))
+    }
+    expect(acked, 'the watcher must answer an arriving request').toBe(true)
+    void brain
+  }, 120_000)
+})

tests/integration/idle-costs-nothing.test.ts

@@ -2,12 +2,17 @@
  * @module tests/integration/idle-costs-nothing
  * @description AN IDLE BRAIN DOES NO WORK.
  *
- * Measured on a production process holding 21 brains: with no writes for ten
- * minutes it printed "All indexes flushed to disk in 216–601ms" per brain
- * every ~35 seconds and idled at 1.26 cores. Every one of those flushes
- * re-persisted state identical to what was already on disk — the provider
- * flushes, the watermark stamps, the generation counter, the entity-tree
- * stamp — because `flush()` never asked whether anything had changed.
+ * A flush used to re-persist state identical to what was already on disk —
+ * the provider flushes, the watermark stamps, the generation counter, the
+ * entity-tree stamp, roughly 28 writes — because `flush()` never asked whether
+ * anything had changed.
+ *
+ * The field observation that started this: a production process holding 21
+ * brains printed "All indexes flushed to disk in 216–601ms" per brain every
+ * ~35 seconds and idled at 1.26 cores, with no writes for ten minutes. This
+ * engine's cadence is WRITE-DRIVEN, so that observation is NOT explained by
+ * the cadence and is not claimed to be fixed here — what is fixed is that such
+ * a call now costs nothing. Who was calling flush() remains open.
  *
  * The laws pinned here:
  *   (a) the persistence cadence arms only on a write — a brain nobody writes