open-brainy/src
David Snelling fb1da1c56d perf(idle): the flush-request watch is event-driven; the heartbeat is observability
Three idle-burn items from the steady-state audit, and one correction.

THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request
directory every 500 ms, per brain, for the life of every writer — armed on
every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second
on a completely idle service, plus a stale-request GC on every one of them. It now
uses fs.watch, so the arrival itself wakes it and a request is seen SOONER
than the poll saw it. Two concessions ride along, both stated in the code: a
30s safety sweep (fs.watch drops events on some network and fuse filesystems,
and the GC needs a tick of its own — two orders of magnitude fewer reads than
the poll made), and a fall back to the original 500 ms poll, narrated, on a
filesystem that cannot watch at all, because an inspector whose request is
never seen waits forever.

THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is
decided by pid liveness and the fence compares pid + hostname, so no decision
anywhere reads the timestamp — and at 10s it was a lock-file write every ten
seconds per brain forever, for a value nothing computes with. An operator
still sees a heartbeat inside the minute.

THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation
counter. That counter bumps on every ledger mutation and rebuild boundary, so
a provider bumping it on routine work re-emitted the same unchanged line on
every read, while one that never bumped could suppress a line whose reasons
had genuinely changed. The generation is still reported; it no longer decides
whether the line is worth saying.

CORRECTION, and it is against my own earlier claim: the idle-flush commit read
a reported idle-CPU observation (many stores, no writes, a flush every ~35s,
over a core burned) as caused by
the flush path. That does not follow — this engine's cadence is write-driven
(every trigger runs through noteWriteForPersistence, which only a committed
write calls), so something was CALLING flush() on those brains and the caller
is still unidentified. The clean-flush gate makes such a call free; it does not
account for it. The code comments and the idle lane now say exactly that.

Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer
makes at most one request-directory read in 8 seconds (the old poll made ~16),
and a dropped request is still acked well inside the safety sweep.
2026-08-28 11:25:47 -07:00
..
aggregation feat(engine): the wiring wave — stamps ride every flush, provider generations, waitForIndexed, adopt-backfill, match-all serves 2026-08-10 10:55:11 -07:00
cli refactor(8.0): remove dead, unreachable, and unwired modules 2026-06-24 15:18:40 -07:00
db perf(generations): discover generations by directory name, not by walking the log 2026-08-28 11:09:05 -07:00
embeddings chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
errors chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
events feat: brain.onChange — the in-process change feed for every committed mutation 2026-07-10 11:24:31 -07:00
graph perf(flush): an idle brain does no work — no periodic flush without a write 2026-08-28 10:44:38 -07:00
hnsw Merge branch 'next/zero-norm-unvector-door' 2026-08-27 13:54:06 -07:00
import feat(namespace): NO SPECIAL NAMES + storage fidelity — the ruled completion of the field-addressing law 2026-08-03 16:59:32 -07:00
importers refactor(8.0): remove dead/deprecated code (legacy sweep) 2026-06-29 10:09:39 -07:00
indexes/columnStore feat(log): log authority is the fleet default — adopt-at-open, oracle-gated; plus the power-cut throw-site cures and the loud torn-record contract 2026-08-11 08:37:38 -07:00
integrations chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
mcp chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
migration feat(namespace): NO SPECIAL NAMES + storage fidelity — the ruled completion of the field-addressing law 2026-08-03 16:59:32 -07:00
neural chore: keep the generated neural stamps at main's values 2026-08-28 10:58:00 -07:00
patterns chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase 2026-06-11 14:51:00 -07:00
reprojection feat(reprojection): the one doors-open machinery — budget-capped, yielding, foreground-preempted, atomic-swap; poison records quarantine typed 2026-08-10 11:39:27 -07:00
storage perf(idle): the flush-request watch is event-driven; the heartbeat is observability 2026-08-28 11:25:47 -07:00
streaming chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase 2026-06-11 14:51:00 -07:00
transaction fix(hnsw): skip unvectored rows on rebuild; refuse empty vectors in the index 2026-08-27 13:29:11 -07:00
triple chore(8.0): Phase A + B — purge all @deprecated APIs + cacheManager dead branches 2026-06-09 15:33:56 -07:00
types feat(repair): repairIndex narrates every phase and its receipt carries the walls 2026-08-28 10:31:42 -07:00
universal refactor(8.0): remove dead, unreachable, and unwired modules 2026-06-24 15:18:40 -07:00
utils feat(contract): declare contract 1, serve three operators, refuse four by name 2026-08-28 10:57:43 -07:00
vfs perf(vfs): the old-root sweep runs once per store, not once per open 2026-08-28 11:01:43 -07:00
brainy.ts perf(idle): the flush-request watch is event-driven; the heartbeat is observability 2026-08-28 11:25:47 -07:00
coreTypes.ts fix(vfs): the VFS root never persists a zero-norm vector 2026-08-27 09:28:44 -07:00
index.ts feat(contract): declare contract 1, serve three operators, refuse four by name 2026-08-28 10:57:43 -07:00
internals.ts feat: provider access to the fact log + shared stamp verifier via internals 2026-07-15 12:36:27 -07:00
plugin.ts chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
setup.ts chore(8.0): collapse dead defensive guards + redundant polyfills 2026-06-09 16:46:16 -07:00