feat(persistence): the engine owns its flush cadence — callers never call flush() in hot paths again
Some checks failed
CI / Node 22 (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled

A4 of the service-class pair (SELF-ENGINE-LIFECYCLE-SPRINT, David-directed:
'why do we need manual flushes at all?'). The production disease: 829
caller-scheduled per-write flushes convoying into 45-66s write walls —
cadence hand-rolled a layer above the only layer that can see dirty-node
counts and IO pressure.

- BrainyConfig.persistence: policy 'auto' (DEFAULT) | 'manual', with
  flushEveryWrites (512) / flushIntervalMs (30s) / flushOnIdleMs (2s)
  triggers. Auto = the engine kicks ONE single-flight BACKGROUND flush at
  a threshold or when the store goes quiet; write acks NEVER await it (a
  hung flush cannot block a write — pinned); a failed background flush is
  LOUD and re-arms the trigger. 'manual' restores caller-owned cadence.
- Triggers wired at both write chokepoints (single-op post-commit +
  transact post-commit); idle timer unref'd; close() tears the timer down
  and drains the flight before its own final flush.
- RECOVERY SEMANTICS documented on the config: canonical records are
  durable per-write regardless of policy — a crash between background
  flushes loses derived state only, which converges at next open (epoch
  machinery + the new incremental aggregation catch-up), bounded by the
  un-flushed window. Never data loss.

Pins: write-count trigger fires one background flush with zero caller
calls · idle trigger · manual never self-flushes · THE ACK LAW (writes
acknowledge under a never-resolving flush). Gates: unit 1917/1917 ·
integration 760 · conformance 27/27 — green WITH auto as the default.
This commit is contained in:
David Snelling 2026-08-05 16:00:39 -07:00
parent 1dc861d299
commit 3236a01bef
3 changed files with 214 additions and 1 deletions

View file

@ -2028,6 +2028,39 @@ export interface BrainyConfig {
*/
force?: boolean
/**
* THE ENGINE OWNS ITS FLUSH CADENCE (the persistence policy
* SELF-ENGINE-LIFECYCLE-SPRINT, David-directed: "why do we need manual
* flushes at all?"). Under `'auto'` (the DEFAULT) the engine schedules
* single-flight background flushes itself triggered by write count,
* elapsed time, and idle so callers NEVER call `flush()` in a hot path
* (a production consumer's 829 per-write flushes convoyed into 4566s
* write walls; the cadence belongs to the layer that can see dirty-node
* counts and IO pressure). `flush()` remains public as an awaitable
* durability BARRIER for the rare "must be on disk before I proceed"
* moment calling it is never wrong, just no longer necessary.
*
* RECOVERY SEMANTICS (the documented promise): canonical records are
* durable per-write, independent of this policy a crash between
* background flushes loses NO data. What a flush persists is DERIVED
* state (index postings, deferred HNSW nodes, counters, aggregation
* stamps); after a crash, derived state converges at the next open from
* canonical records (epoch machinery + incremental aggregation catch-up),
* paying a bounded catch-up cost proportional to the un-flushed window
* never data loss.
*
* `'manual'` restores the pre-9.1 behavior: the engine never flushes on
* its own (except at `close()`); the caller owns the cadence.
*/
persistence?: {
policy?: 'auto' | 'manual'
/** Background flush after this many committed writes (default 512). */
flushEveryWrites?: number
/** Background flush when this much time has passed since the last flush, checked at write time (default 30_000). */
flushIntervalMs?: number
/** Background flush after the store goes quiet for this long with dirty state (default 2_000). */
flushOnIdleMs?: number
}
}
// ============= Neural API Types =============