fix(shutdown): beforeExit never closes a live brain — a drained event loop is not a shutdown
Some checks failed
CI / Node 22 (push) Successful in 12m30s
CI / Node 24 (push) Successful in 12m23s
CI / Bun (latest) (push) Successful in 12m36s
CI / Integration + conformance (Node 22) (push) Failing after 17m33s

10.4.11 gave shutdown one owner and one path — close() — and wired all three
process listeners to it. That is right for SIGTERM and SIGINT. It is wrong for
'beforeExit', which Node emits whenever the event loop has no REF'd work left:
not when the process is ending, and with no signal involved. A healthy script
reaches that state routinely, because this engine unref's its idle and cadence
timers ("an idle brain costs nothing"), so a script awaiting anything those
timers drive is, for that instant, a process with no ref'd work and an open
brain.

MEASURED on the 11.1 rehearsal lane against a copy of a real store: after the
heal phase the log printed "Shutdown signal received - flushing pending
data..." and "Flushed successfully (1 instance)" with no signal ever sent, and
the script's very next add() threw "Brainy instance is not initialized: it was
closed via close(). Create a new instance." The engine had closed a live brain
out from under a running script.

The beforeExit listener now runs its own pass, which closes nothing,
deregisters nothing, releases no writer lock, and never force-exits: it runs
flush() — the engine's own non-closing durability door — on each live brain and
leaves every one of them open and usable. flush() persists derived state only
(count ledger, projections, generation counter, aggregation, entity-tree
stamp); the clean-shutdown marker is generationStore.close()'s word about
itself, reached only from close(). Running it concurrently with live writes is
the engine's ordinary steady state — noteWriteForPersistence() kicks the same
call off an unref'd timer on every busy brain — and it is single-flight, so
there is no new race. A throw is reported per instance and the pass continues:
canonical data is durable at ack via the fact log, so a failed derived-state
flush costs the next open a rebuild, never the caller their brain.

The listener is no longer self-deregistered. It does not need to be: a flush on
a clean brain schedules no I/O, so the emit after it does no event-loop work
and the process exits on its own. A one-shot listener spent on a spurious
mid-script drain would leave the genuine end-of-script drain with nothing. The
drained-loop notice is printed once per registration cycle, because a
console.log to a pipe is itself event-loop work.

exitIfSoleShutdownOwner() stays on the signal path alone, and its contract now
says so: beforeExit suppresses no default behaviour, so exiting from it would
end a live script at code 0 mid-work.

THE NAMED TRADE: a script that opens a brain and never closes it now exits with
its writer lock still on disk and no clean-shutdown marker, so its next open
overwrites a stale lock and folds the log. That is the honest cost of never
closing, and the narration names the cure. Closing a live brain to avoid it was
the worse half of the trade.

Pins: tests/integration/beforeexit-never-closes.test.ts — a script that drains
the loop with a brain open keeps a working brain (add + find succeed, the lock
is still held, the process still exits 0), the pass flushed and wrote neither
of close()'s markers, and repeated drains are idempotent. Both cases fail on
10.4.11's handler with the exact production shape ("add() after the drain
failed", "pass 1 closed the brain"). Re-run green: shutdown-single-owner,
writer-lock-clean-close, idle-costs-nothing, shutdown-hooks-lifecycle.

docs/concepts/multi-process.md no longer claims beforeExit releases the lock.
This commit is contained in:
David Snelling 2026-09-02 14:18:19 -07:00
parent 27759a1be9
commit 6baa4d7f6c
3 changed files with 450 additions and 17 deletions

View file

@ -531,6 +531,19 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private static sigintListener?: () => void
private static beforeExitListener?: () => void
/** True while the `beforeExit` pass is running its flushes. Node re-emits
* 'beforeExit' after every loop drain and that pass schedules async work, so
* a second emit can arrive on top of the first; it returns instead of
* stacking a parallel pass. NOT a one-shot: every genuine drain still gets a
* flush. See {@link registerShutdownHooks}. */
private static beforeExitFlushInFlight = false
/** Whether the drained-event-loop notice has been printed for this
* registration cycle. Printed ONCE `console.log` to a pipe is itself
* event-loop work, so narrating on every emit would keep the loop turning
* and narrate forever. Reset by {@link deregisterShutdownHooksIfIdle}. */
private static beforeExitNarrated = false
/** Poll cadence (ms) for the migration LOCK when a provider exposes no
* event-driven `whenMigrationComplete()` signal. See {@link awaitMigrationLock}. */
private static readonly MIGRATION_POLL_INTERVAL_MS = 250
@ -2130,9 +2143,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* Critical for Cloud Run, Fargate, Lambda, and other containerized deployments.
*
* Handles:
* - SIGTERM: Graceful termination (Cloud Run, Fargate, Lambda)
* - SIGINT: Ctrl+C (development/local testing)
* - beforeExit: Node.js cleanup hook (fallback)
* - SIGTERM: Graceful termination (Cloud Run, Fargate, Lambda) CLOSES.
* - SIGINT: Ctrl+C (development/local testing) CLOSES.
* - beforeExit: the event loop drained FLUSHES, and closes NOTHING. A
* drained loop is not a shutdown; see {@link flushOnDrainedEventLoop}'s
* contract below.
*
* NOTE: Registers globally (once for all instances) to avoid MaxListenersExceededWarning
*/
@ -2229,6 +2244,106 @@ export class Brainy<T = any> implements BrainyInterface<T> {
}
}
/**
* THE DRAINED-EVENT-LOOP PATH. A DRAINED LOOP IS NOT A SHUTDOWN.
*
* Node emits `'beforeExit'` whenever the event loop has no REF'd work
* left NOT when the process is ending, and with no signal involved. A
* perfectly healthy script reaches that state routinely: this engine
* unref's its idle and cadence timers ("an idle brain costs nothing"), so
* a script awaiting anything those timers drive is, for that instant,
* a process with no ref'd work and an open brain.
*
* MEASURED on the 11.1 rehearsal lane against a copy of a real store: the
* `beforeExit` listener was wired to the SIGNAL path, so after the heal
* phase the log printed `Shutdown signal received - flushing pending
* data...` and `Flushed successfully (1 instance)` with NO signal ever
* sent, and the script's very next `add()` threw `Brainy instance is not
* initialized: it was closed via close(). Create a new instance.` The
* engine had closed a live brain out from under a running script.
*
* SO, THE LAW: this path NEVER closes, deregisters, tears down or
* force-exits anything, and never releases a writer lock. It runs
* `flush()` the engine's own non-closing durability door on each live
* brain, and leaves every one of them open and usable.
*
* WHY flush() AND NOT NOTHING. Each claim checked against the code it
* names:
* 1. IT CANNOT CLOSE ANYTHING. `flush()` `_flushSteps()` persists
* DERIVED state only: the count ledger, the metadata/graph/vector
* projections, the generation counter, aggregation state, the
* entity-tree stamp. It closes no component, deactivates no plugin,
* touches neither `initialized` nor `closed`, and never calls
* `releaseWriterLock()` the clean-shutdown marker is written by
* `generationStore.close()` alone, reached only from `close()`.
* 2. IT CANNOT RACE A LATER WRITE INTO CORRUPTION. A background flush
* concurrent with live writes is the engine's ORDINARY steady state:
* `noteWriteForPersistence()` kicks exactly this call off an unref'd
* timer on every busy brain. `flush()` is single-flight with one queued
* follow-up, and a write landing mid-flush re-sets the dirty witness,
* so its work is never lost it belongs to the next flush.
* 3. IT CANNOT SPIN. `flush()` on a clean brain returns without touching a
* provider or scheduling I/O, so the second emit does no event-loop
* work and the process exits. That is also why the listener is NOT
* self-deregistered any more: a one-shot listener spent on a spurious
* mid-script drain leaves the genuine end-of-script drain with nothing.
* 4. A FAILED FLUSH IS SURVIVABLE AND LOUD. The write path is durable at
* ack via the fact log; derived state is rebuildable. A throw is
* reported per instance and the loop continues exactly how
* `kickBackgroundFlush()` already treats the same failure.
*
* The one thing lost against a closing handler is the clean-shutdown
* marker for a script that opens a brain and never closes it: its next
* open folds the log. That is the correct trade a missing marker costs
* a recovery fold, closing a live brain costs the caller its brain and
* the narration below names the cure.
*/
const flushOnDrainedEventLoop = async () => {
// A second emit can land on top of the first (this pass schedules async
// work, the loop turns, the loop drains again). One pass at a time.
if (Brainy.beforeExitFlushInFlight) return
// Step aside for anyone whose close is running or done — the same
// ownership rule the signal path follows.
const live = [...Brainy.instances].filter(
(instance) => instance.initialized && !instance.closed && instance._closeInFlight === null
)
if (live.length === 0) return
// ONCE per registration cycle: a `console.log` to a pipe is itself
// event-loop work, so narrating on every emit would keep the loop
// turning and narrate forever.
if (!Brainy.beforeExitNarrated) {
Brainy.beforeExitNarrated = true
console.log(
`[Brainy] event loop drained with ${live.length} brain${live.length > 1 ? 's' : ''} ` +
`open — persisting derived state; NOTHING was closed. A drained loop is not a ` +
`shutdown: call close() (or send SIGTERM) when you mean one.`
)
}
Brainy.beforeExitFlushInFlight = true
try {
for (const instance of live) {
try {
await instance.flush()
} catch (error) {
// Per-instance isolation, and never fatal: canonical data is
// durable at ack, so a failed derived-state flush costs the next
// open a rebuild — it must not cost this one its brain.
console.error(
'[Brainy] flush on a drained event loop failed for one open brain ' +
'(the brain stays open and usable; derived-state persistence retries at the ' +
'next flush, and canonical data is unaffected):',
error
)
}
}
} finally {
Brainy.beforeExitFlushInFlight = false
}
}
// Graceful shutdown signals (registered once globally). The listeners are
// kept as statics so the last live instance's close() can deregister them
// — the signal handles they hold are ref'd and would otherwise keep the
@ -2254,6 +2369,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* last brain deregisters Brainy's own listeners — so a host application's
* single remaining listener would look like `<= 1` and get force-exited
* out of its own graceful shutdown, precisely the failure above.
*
* SIGNALS ONLY NEVER `beforeExit`. The reasoning above is entirely about
* a signal Brainy has suppressed Node's default terminate behaviour for.
* `beforeExit` suppresses nothing: Node exits by itself once the loop is
* genuinely done, and the script that is still running when it fires is
* not shutting down at all. Calling this from that path would end a live
* script at exit code 0 mid-work. It is called from the two signal
* listeners below and from nowhere else.
*/
const exitIfSoleShutdownOwner = (ownersWhenSignalled: number): void => {
if (ownersWhenSignalled <= 1) {
@ -2270,18 +2393,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
await closeOnShutdown()
exitIfSoleShutdownOwner(owners)
}
Brainy.beforeExitListener = async () => {
// Self-deregister FIRST: Node re-emits 'beforeExit' after every event-
// loop drain, and this flush schedules new async work — with the
// listener still attached, a script that never calls close() would spin
// flush → drain → flush forever and never exit. One flush, then the
// next drain finds no listener and the process exits.
if (Brainy.beforeExitListener) {
process.off('beforeExit', Brainy.beforeExitListener)
Brainy.beforeExitListener = undefined
}
await closeOnShutdown()
}
Brainy.beforeExitListener = flushOnDrainedEventLoop
process.on('SIGTERM', Brainy.sigtermListener)
process.on('SIGINT', Brainy.sigintListener)
process.on('beforeExit', Brainy.beforeExitListener)
@ -2303,6 +2415,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
Brainy.sigtermListener = undefined
Brainy.sigintListener = undefined
Brainy.beforeExitListener = undefined
// A later re-init is a fresh cycle: it may narrate its own drained-loop
// notice, and no pass of the previous cycle can still be running (the last
// close() drained the flush chain).
Brainy.beforeExitNarrated = false
Brainy.beforeExitFlushInFlight = false
Brainy.shutdownHooksRegisteredGlobally = false
}