fix(storage): a clean close is recorded, and the writer lock is always given up

A production restart made this necessary: a service stopped with exit code 0,
having awaited close() on every pooled brain, and its next boot announced
"Overwriting stale writer lock ... appears dead" for every store it owned.
Nothing had crashed. "The recorded pid is gone" is equally true of an orderly
restart and of a crash, so the verdict could not tell an operator which one
they had — and when the OS recycles a pid it fails the other way, refusing to
open a store whose writer died days ago.

Three changes, all at the law:

- close() is two parts, and the second is unconditional. The durable steps
  (flush, markers, component close, plugin deactivate, buffer drain) move to
  closeDurableSteps(); the terminal releases — the flush-request watcher, the
  WRITER LOCK, the VFS timers, the terminal `closed` flag — always run. The
  original failure is narrated with what it costs the next open, then rethrown.
- releaseWriterLock() writes a CLEAN-CLOSE RECORD (`locks/_writer.close`)
  naming the lock generation it released; the next claim consumes it, so a
  record can never vouch for a later crash. An open reads the record instead
  of guessing: recorded → nothing to recover; absent → say so, and name the
  crash recovery this open will now run.
- The signal path stops failing in a batch. It was one try around a loop over
  every open brain, so the first instance whose flush rejected stranded every
  remaining brain's lock and markers — at exit code 0. Now: per-instance
  isolation, the generation store's close (the clean-shutdown marker, without
  which the next open folds the whole log) is part of shutdown, the lock is
  given up in a finally, and the handler no longer calls process.exit() when
  the host application has its own signal handler — that race truncated the
  host's own close() mid-flight.

Pins: tests/integration/writer-lock-clean-close.test.ts — completed close
leaves no lock and a consumed-once record with a silent reopen; a failing
durable step still releases and still rethrows; SIGKILL leaves the lock with
no record and the reopen names the crash; a host SIGTERM handler runs to
completion.

Branch plan (10 lines):
 1. writer lock: clean-close record + always-release close  [this commit]
 2. open narration: an always-on channel; production clamps prodLog to ERROR,
    which is why a three-minute open printed nothing
 3. open narration: per-phase lines as each phase ENDS, with progress cadence
 4. measure both real-store fixtures on the box, before/after
 5. move the generation-log fold out of the foreground where the serving law
    allows; durable resumable progress marker
 6. same for the VFS bootstrap
 7. counts: a legacy container-rule ledger must not keep serving wrong
    denominators; counts.json written atomically
 8. counts pin with scar directories; two copies of one archive agree
 9. docs/canonical-layout-ratification.md — 12 facts confirmed/corrected
10. report: MEASURED before/after, findings, and whether this is 10.4.4

src/brainy.ts

@@ -1856,76 +1856,112 @@ export class Brainy<T = any> implements BrainyInterface<T> {
    * NOTE: Registers globally (once for all instances) to avoid MaxListenersExceededWarning
    */
   private registerShutdownHooks(): void {
+    /**
+     * The signal-path shutdown. THREE LAWS, each written by a production
+     * shutdown that looked clean and wasn't:
+     *
+     * 1. PER-INSTANCE ISOLATION. This used to be one `try` around a loop over
+     *    every open brain: the first instance whose flush rejected aborted the
+     *    loop, so every remaining brain kept its writer lock and its unwritten
+     *    markers — and the process still exited 0. A pool of brains failed in
+     *    a batch, not one at a time.
+     * 2. THE MARKER IS PART OF SHUTDOWN. Flushing the indexes without closing
+     *    the generation store leaves the clean-shutdown marker unwritten, so
+     *    the NEXT open reads the store as crashed and folds the whole
+     *    generation log — measured in tens of seconds on a real store, paid on
+     *    every restart, after a shutdown the operator saw exit 0.
+     * 3. THE LOCK IS ALWAYS GIVEN UP. In a `finally`, per instance: a process
+     *    on its way out holds nothing.
+     */
     const flushOnShutdown = async () => {
       console.log('Shutdown signal received - flushing pending data...')
-      try {
+      let flushedCount = 0
-        let flushedCount = 0
+      let failedCount = 0
-        for (const instance of Brainy.instances) {
+      // Snapshot: close() splices Brainy.instances while we iterate.
-          if (instance.initialized) {
+      for (const instance of [...Brainy.instances]) {
-            // Flush all buffered data, then close to release resources (timers, handles)
+        if (!instance.initialized) continue
-            await Promise.all([
+        try {
-              (async () => {
+          // Flush all buffered data (parallel across components, this brain only).
-                if (instance.storage && typeof instance.storage.flushCounts === 'function') {
+          await Promise.all([
-                  await instance.storage.flushCounts()
+            (async () => {
-                }
+              if (instance.storage && typeof instance.storage.flushCounts === 'function') {
-              })(),
+                await instance.storage.flushCounts()
-              (async () => {
+              }
-                if (instance.metadataIndex && typeof instance.metadataIndex.flush === 'function') {
+            })(),
-                  await instance.metadataIndex.flush()
+            (async () => {
-                }
+              if (instance.metadataIndex && typeof instance.metadataIndex.flush === 'function') {
-              })(),
+                await instance.metadataIndex.flush()
-              (async () => {
+              }
-                if (instance.graphIndex && typeof instance.graphIndex.flush === 'function') {
+            })(),
-                  await instance.graphIndex.flush()
+            (async () => {
-                }
+              if (instance.graphIndex && typeof instance.graphIndex.flush === 'function') {
-              })(),
+                await instance.graphIndex.flush()
-              (async () => {
+              }
-                if (instance.index && typeof instance.index.flush === 'function') {
+            })(),
-                  await instance.index.flush()
+            (async () => {
-                }
+              if (instance.index && typeof instance.index.flush === 'function') {
-              })()
+                await instance.index.flush()
-            ])
+              }
-            // Close components to stop timers that would prevent clean process exit
+            })()
-            await Promise.all([
+          ])
-              (async () => {
+
-                if (instance.graphIndex && typeof instance.graphIndex.close === 'function') {
+          // Close the generation store: persists the counter, advances the
-                  await instance.graphIndex.close()
+          // fold checkpoint, and stamps the clean-shutdown marker LAST — the
-                }
+          // one step that decides whether the next open adopts or folds. Law 2.
-              })(),
+          if (instance.generationStore && !instance.isReadOnly) {
-              (async () => {
+            await instance.generationStore.close()
-                const index = instance.index as JsHnswVectorIndex & VectorIndexOptionalHooks
+          }
-                if (index && typeof index.close === 'function') {
+
-                  await index.close()
+          // Close components to stop timers that would prevent clean process exit
-                }
+          await Promise.all([
-              })(),
+            (async () => {
-              (async () => {
+              if (instance.graphIndex && typeof instance.graphIndex.close === 'function') {
-                const metadataIndex = instance.metadataIndex as MetadataIndexManager & MetadataIndexOptionalHooks
+                await instance.graphIndex.close()
-                if (metadataIndex && typeof metadataIndex.close === 'function') {
+              }
-                  await metadataIndex.close()
+            })(),
-                }
+            (async () => {
-              })(),
+              const index = instance.index as JsHnswVectorIndex & VectorIndexOptionalHooks
-              // Release the writer lock so a successor process can take over.
+              if (index && typeof index.close === 'function') {
-              // No-op for readers and for backends without locking.
+                await index.close()
-              (async () => {
+              }
-                if (instance.storage && typeof instance.storage.releaseWriterLock === 'function') {
+            })(),
-                  await instance.storage.releaseWriterLock()
+            (async () => {
-                }
+              const metadataIndex = instance.metadataIndex as MetadataIndexManager & MetadataIndexOptionalHooks
-              })(),
+              if (metadataIndex && typeof metadataIndex.close === 'function') {
-              // Stop the flush-request watcher to release its interval timer.
+                await metadataIndex.close()
-              (async () => {
+              }
-                if (instance.storage && typeof instance.storage.stopFlushRequestWatcher === 'function') {
+            })()
-                  instance.storage.stopFlushRequestWatcher()
+          ])
-                }
+          flushedCount++
-              })(),
+        } catch (error) {
-            ])
+          failedCount++
-            flushedCount++
+          console.error('Failed to flush one Brainy instance on shutdown:', error)
+        } finally {
+          // Law 3 — the lock and the watcher go regardless.
+          try {
+            if (instance.storage && typeof instance.storage.stopFlushRequestWatcher === 'function') {
+              instance.storage.stopFlushRequestWatcher()
+            }
+          } catch (error) {
+            console.error('Failed to stop the flush-request watcher on shutdown:', error)
+          }
+          try {
+            if (instance.storage && typeof instance.storage.releaseWriterLock === 'function') {
+              await instance.storage.releaseWriterLock()
+            }
+          } catch (error) {
+            console.error('Failed to release the writer lock on shutdown:', error)
           }
         }
-        if (flushedCount > 0) {
+      }
-          console.log(`Flushed successfully (${flushedCount} instance${flushedCount > 1 ? 's' : ''})`)
+      if (flushedCount > 0) {
-        }
+        console.log(`Flushed successfully (${flushedCount} instance${flushedCount > 1 ? 's' : ''})`)
-      } catch (error) {
+      }
-        console.error('Failed to flush on shutdown:', error)
+      if (failedCount > 0) {
+        console.error(
+          `${failedCount} Brainy instance${failedCount > 1 ? 's' : ''} did not complete shutdown — ` +
+            `their writer locks were released, but their next open will run crash recovery.`
+        )
       }
     }
 
@@ -1933,13 +1969,32 @@ export class Brainy<T = any> implements BrainyInterface<T> {
     // kept as statics so the last live instance's close() can deregister them
     // — the signal handles they hold are ref'd and would otherwise keep the
     // process alive forever after every brain is closed.
+    /**
+     * Exit the process ONLY when Brainy is the sole handler for this signal.
+     *
+     * Registering a signal listener suppresses Node's default terminate
+     * behaviour, so a library that attaches one must either exit or be sure
+     * someone else will. Brainy attaching one AND exiting was the wrong half
+     * of that choice for every host application with its own graceful
+     * shutdown: both handlers run concurrently, and whichever finishes first
+     * wins — a library flush finishing before an application's close()
+     * terminated that close mid-flight, at exit code 0, with locks and
+     * markers unwritten. When the host has its own handler (listener count
+     * above our own), the host owns the exit; Brainy only makes its data
+     * durable and steps aside.
+     */
+    const exitIfSoleShutdownOwner = (signal: 'SIGTERM' | 'SIGINT'): void => {
+      if (process.listenerCount(signal) <= 1) {
+        process.exit(0)
+      }
+    }
     Brainy.sigtermListener = async () => {
       await flushOnShutdown()
-      process.exit(0)
+      exitIfSoleShutdownOwner('SIGTERM')
     }
     Brainy.sigintListener = async () => {
       await flushOnShutdown()
-      process.exit(0)
+      exitIfSoleShutdownOwner('SIGINT')
     }
     Brainy.beforeExitListener = async () => {
       // Self-deregister FIRST: Node re-emits 'beforeExit' after every event-
@@ -18877,12 +18932,105 @@ export class Brainy<T = any> implements BrainyInterface<T> {
   }
 
   /**
-   * Close and cleanup
+   * @description Close and clean up: flush every buffered component, stamp
+   * the durability markers, release resources, then give up the writer lock.
    *
-   * Now flushes HNSW dirty nodes before closing
+   * TWO PARTS, AND THE SECOND IS UNCONDITIONAL. Everything that persists data
-   * This ensures deferred persistence mode data is saved
+   * runs in {@link closeDurableSteps}; the terminal releases — the flush-request
+   * watcher, the WRITER LOCK, the VFS timers, and the terminal `closed` flag —
+   * run whether those steps succeeded or not, in a `finally`. A close that
+   * threw halfway used to strand the writer lock on disk with this process's
+   * (soon dead) pid in it, so the next boot of every affected store announced
+   * `Overwriting stale writer lock … appears dead` after an orderly exit and
+   * an operator had to decide whether their database had crashed. A closed
+   * brain holds no lock — there is no failure for which the opposite is the
+   * safer answer.
+   *
+   * The original failure is never swallowed: it is narrated with what it costs
+   * the next open, then rethrown to the caller.
+   * @returns Nothing.
+   * @throws The first failure from the durable close steps, after the
+   *   terminal releases have run.
    */
   async close(): Promise<void> {
+    let closeFailure: unknown = null
+    try {
+      await this.closeDurableSteps()
+    } catch (error) {
+      closeFailure = error
+    }
+
+    // ---- TERMINAL RELEASES: always, even after a failure above ----
+
+    // Stop the cross-process flush-request watcher (no-op if never started).
+    try {
+      if (this.storage && typeof this.storage.stopFlushRequestWatcher === 'function') {
+        this.storage.stopFlushRequestWatcher()
+      }
+    } catch (error) {
+      console.warn('[Brainy] close: stopping the flush-request watcher failed:', error)
+    }
+
+    // Release the writer lock. Runs after the metadata buffer drain in
+    // closeDurableSteps() — otherwise a pending write could land after a
+    // successor writer claimed the lock — and runs even if that drain threw:
+    // holding a lock from a process that is about to exit locks the store's
+    // next boot out of a clean verdict.
+    try {
+      if (this.storage && typeof this.storage.releaseWriterLock === 'function') {
+        await this.storage.releaseWriterLock()
+      }
+    } catch (error) {
+      console.warn('[Brainy] close: releasing the writer lock failed:', error)
+    }
+
+    // Shut down the VFS: stops its background maintenance interval and the
+    // PathResolver's — both are ref'd timers that would keep the process
+    // alive after the last brain closes (consumer-reported hang).
+    try {
+      if (this._vfs) {
+        await this._vfs.close()
+      }
+    } catch (error) {
+      console.warn('[Brainy] close: VFS shutdown failed:', error)
+    }
+
+    this.initialized = false
+    // close() is terminal: block lazy re-initialization on any subsequent
+    // operation (ensureInitialized() throws once this is set). Set even when
+    // the durable steps failed — a half-closed brain must not keep serving.
+    this.closed = true
+
+    // Drop this instance from the global registry, and when it was the last
+    // one, deregister the global shutdown hooks — their ref'd signal handles
+    // would otherwise keep the process alive after every brain is closed.
+    const instanceIndex = Brainy.instances.indexOf(this)
+    if (instanceIndex !== -1) {
+      Brainy.instances.splice(instanceIndex, 1)
+    }
+    Brainy.deregisterShutdownHooksIfIdle()
+
+    if (closeFailure !== null) {
+      console.error(
+        `[Brainy] close FAILED partway: ` +
+          `${closeFailure instanceof Error ? closeFailure.message : String(closeFailure)}\n` +
+          `  This brain is closed and holds no writer lock, but the clean-shutdown ` +
+          `marker may not have been written — the next open will run crash recovery ` +
+          `(a generation-log fold) and report its wall.`
+      )
+      throw closeFailure
+    }
+  }
+
+  /**
+   * @description The durable half of {@link close}: flush every component,
+   * persist the generation counter and its markers, close the components,
+   * deactivate plugins, drain the metadata write buffer. Separated from
+   * `close()` so the terminal releases there can run in a `finally` — see that
+   * method's contract.
+   * @returns Nothing.
+   */
+  private async closeDurableSteps(): Promise<void> {
     // Persistence cadence teardown: no background flush may fire after close
     // begins (close() runs its own final flush).
     if (this._persistIdleTimer) {
@@ -19033,38 +19181,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
       }
     }
 
-    // Stop the cross-process flush-request watcher (no-op if never started).
-    if (this.storage && typeof this.storage.stopFlushRequestWatcher === 'function') {
-      this.storage.stopFlushRequestWatcher()
-    }
-
-    // Release the writer lock (no-op for readers and for backends that don't
-    // hold a lock). Must run after the metadata buffer drain — otherwise a
-    // pending write could land after a successor writer claimed the lock.
-    if (this.storage && typeof this.storage.releaseWriterLock === 'function') {
-      await this.storage.releaseWriterLock()
-    }
-
-    // Shut down the VFS: stops its background maintenance interval and the
-    // PathResolver's — both are ref'd timers that would keep the process
-    // alive after the last brain closes (consumer-reported hang).
-    if (this._vfs) {
-      await this._vfs.close()
-    }
-
-    this.initialized = false
-    // close() is terminal: block lazy re-initialization on any subsequent
-    // operation (ensureInitialized() throws once this is set).
-    this.closed = true
-
-    // Drop this instance from the global registry, and when it was the last
-    // one, deregister the global shutdown hooks — their ref'd signal handles
-    // would otherwise keep the process alive after every brain is closed.
-    const instanceIndex = Brainy.instances.indexOf(this)
-    if (instanceIndex !== -1) {
-      Brainy.instances.splice(instanceIndex, 1)
-    }
-    Brainy.deregisterShutdownHooksIfIdle()
   }
 }
 

src/storage/adapters/fileSystemStorage.ts

@@ -14,7 +14,8 @@ import {
   StorageBatchConfig,
   SYSTEM_DIR,
   STATISTICS_KEY,
-  WriterLockInfo
+  WriterLockInfo,
+  WriterCloseRecord
 } from '../baseStorage.js'
 import { getBrainyVersion } from '../../utils/index.js'
 import { isAbsentError } from '../../utils/errorClassification.js'
@@ -99,6 +100,13 @@ export class FileSystemStorage extends BaseStorage {
   // timer rewrites the lock every 10s so stale-lock detection can tell a dead
   // writer from a slow one. The constant name matches the file path used.
   private static readonly WRITER_LOCK_FILE = '_writer.lock'
+  /**
+   * The clean-close record at `locks/_writer.close` (see
+   * {@link WriterCloseRecord}). Written when the lock is released, consumed by
+   * the next claim, so an open can distinguish "the previous writer left" from
+   * "the previous writer died" without inferring either from a pid.
+   */
+  private static readonly WRITER_CLOSE_FILE = '_writer.close'
   private static readonly WRITER_HEARTBEAT_MS = 10_000
   private static readonly WRITER_STALE_THRESHOLD_MS = 60_000
   private writerLockHeartbeat?: NodeJS.Timeout
@@ -1902,11 +1910,24 @@ export class FileSystemStorage extends BaseStorage {
             rootDir: this.rootDir
           }
           await this.writeFileAtomic(lockFile, JSON.stringify(info, null, 2))
+          await this.clearWriterCloseRecord()
           this.installWriterLock(info)
           return info
         }
 
-        const stale = !options?.force && (await this.isWriterLockStale(existing))
+        // THE CLEAN-CLOSE RECORD IS CONSULTED FIRST (see WriterCloseRecord).
+        // A lock file whose release was RECORDED is bookkeeping left behind by
+        // an orderly shutdown, not evidence of a crash — take it over calmly
+        // and say so. Only when no record vouches for this lock do we fall
+        // back to inferring liveness from the pid, and then we say THAT
+        // honestly too: an unrecorded lock means the writer did not complete
+        // its close, so the store was not closed cleanly and this open pays
+        // recovery.
+        const closeRecord = await this.readWriterCloseRecord()
+        const releasedCleanly =
+          closeRecord !== null && this.closeRecordVouchesFor(closeRecord, existing)
+        const stale =
+          releasedCleanly || (!options?.force && (await this.isWriterLockStale(existing)))
         if (!options?.force && !stale) {
           // Consumer-facing error contract: callers detect this case via
           // err.code and read the holder's details from err.lockInfo.
@@ -1917,8 +1938,16 @@ export class FileSystemStorage extends BaseStorage {
           options?.force
             ? `[brainy] Force-overwriting writer lock for ${this.rootDir} ` +
               `(was held by PID ${existing.pid} on ${existing.hostname}).`
-            : `[brainy] Overwriting stale writer lock for ${this.rootDir} ` +
+            : releasedCleanly
-              `(PID ${existing.pid} on ${existing.hostname} appears dead).`
+              ? `[brainy] Clearing the leftover writer lock for ${this.rootDir} — ` +
+                `PID ${existing.pid} on ${existing.hostname} RELEASED it cleanly at ` +
+                `${closeRecord!.closedAt} but could not remove the file. ` +
+                `Nothing to recover.`
+              : `[brainy] Overwriting stale writer lock for ${this.rootDir} ` +
+                `(PID ${existing.pid} on ${existing.hostname} is gone and left NO ` +
+                `clean-close record — that writer did not finish closing, so this ` +
+                `store was not closed cleanly; open will run crash recovery and ` +
+                `report its wall).`
         )
         // Takeover: verify the file still holds the lock we judged (a live
         // successor may have claimed meanwhile), then remove it and fall
@@ -1972,6 +2001,12 @@ export class FileSystemStorage extends BaseStorage {
         await fs.promises.unlink(claimTmp).catch(() => {})
       }
 
+      // CONSUME the previous writer's clean-close record. It described the
+      // lock generation that just ended; leaving it in place would let it
+      // vouch for OUR lock if this process later dies without closing —
+      // turning a real crash into a "closed cleanly" verdict. One unlink.
+      await this.clearWriterCloseRecord()
+
       this.installWriterLock(info)
       return info
     }
@@ -2095,13 +2130,27 @@ export class FileSystemStorage extends BaseStorage {
       return
     }
     const lockFile = path.join(this.lockDir, FileSystemStorage.WRITER_LOCK_FILE)
+    const released = this.writerLockInfo
     try {
       // Only delete if we still own it — avoid clobbering a successor that
       // claimed the lock via force-override.
       const current = await this.readWriterLock()
-      if (current && current.pid === this.writerLockInfo.pid && current.hostname === this.writerLockInfo.hostname) {
+      const ours =
+        current === null ||
+        (current.pid === released.pid && current.hostname === released.hostname)
+      if (current && ours) {
         await fs.promises.unlink(lockFile)
       }
+      // THE CLEAN-CLOSE RECORD (see WriterCloseRecord). Written whenever this
+      // instance gives up a lock nobody else has taken — the unlink above
+      // having succeeded OR the file already being gone. The next open reads
+      // it instead of guessing from pid liveness: a recorded release is an
+      // orderly shutdown, an absent record is a writer that never finished
+      // closing. Not written when a successor holds the lock: our release is
+      // then a no-op and a record would slander their live lock.
+      if (ours) {
+        await this.writeWriterCloseRecord(released)
+      }
     } catch (err: any) {
       if (err.code !== 'ENOENT') {
         console.warn('[brainy] Failed to release writer lock file:', err)
@@ -2111,6 +2160,94 @@ export class FileSystemStorage extends BaseStorage {
     }
   }
 
+  /**
+   * @description Read the clean-close record at `locks/_writer.close`, or
+   * `null` when it is absent or unparseable. A torn record is treated as
+   * absent — the conservative direction, since an unreadable record can
+   * vouch for nothing.
+   * @returns The record, or null.
+   */
+  public async readWriterCloseRecord(): Promise<WriterCloseRecord | null> {
+    await this.ensureInitialized()
+    const recordFile = path.join(this.lockDir, FileSystemStorage.WRITER_CLOSE_FILE)
+    try {
+      const raw = await fs.promises.readFile(recordFile, 'utf-8')
+      const parsed = JSON.parse(raw) as WriterCloseRecord
+      if (
+        typeof parsed?.pid !== 'number' ||
+        typeof parsed?.hostname !== 'string' ||
+        typeof parsed?.startedAt !== 'string' ||
+        typeof parsed?.closedAt !== 'string'
+      ) {
+        return null
+      }
+      return parsed
+    } catch (err: any) {
+      if (err.code === 'ENOENT') return null
+      return null
+    }
+  }
+
+  /**
+   * @description Whether a clean-close record describes the very lock
+   * generation `lock` represents. The match is pid + hostname + `startedAt`:
+   * `startedAt` is the lock generation's identity, so a record can never
+   * vouch for a LATER lock taken by the same pid on the same host (the
+   * same-process re-open path mints a fresh `startedAt`).
+   * @param record - The clean-close record read from disk.
+   * @param lock - The lock file's contents.
+   */
+  private closeRecordVouchesFor(record: WriterCloseRecord, lock: WriterLockInfo): boolean {
+    return (
+      record.pid === lock.pid &&
+      record.hostname === lock.hostname &&
+      record.startedAt === lock.startedAt
+    )
+  }
+
+  /**
+   * @description Write the clean-close record for a lock this instance just
+   * released. Atomic (temp + rename) so a concurrent opener never reads half
+   * a record. A failure here costs the next open nothing but the honest
+   * fallback (pid liveness), so it warns rather than failing the close.
+   * @param released - The lock info this instance held.
+   */
+  private async writeWriterCloseRecord(released: WriterLockInfo): Promise<void> {
+    const record: WriterCloseRecord = {
+      pid: released.pid,
+      hostname: released.hostname,
+      startedAt: released.startedAt,
+      closedAt: new Date().toISOString(),
+      version: released.version
+    }
+    const recordFile = path.join(this.lockDir, FileSystemStorage.WRITER_CLOSE_FILE)
+    try {
+      await this.writeFileAtomic(recordFile, JSON.stringify(record, null, 2))
+    } catch (err) {
+      console.warn(
+        `[brainy] Failed to write the writer clean-close record for ${this.rootDir} — ` +
+          `the next open will fall back to pid liveness and may report this orderly ` +
+          `shutdown as a crash:`,
+        err
+      )
+    }
+  }
+
+  /**
+   * @description Remove the clean-close record. Called by every successful
+   * lock claim so a record never outlives the lock generation it describes.
+   */
+  private async clearWriterCloseRecord(): Promise<void> {
+    const recordFile = path.join(this.lockDir, FileSystemStorage.WRITER_CLOSE_FILE)
+    try {
+      await fs.promises.unlink(recordFile)
+    } catch (err: any) {
+      if (err.code !== 'ENOENT') {
+        console.warn('[brainy] Failed to clear the writer clean-close record:', err)
+      }
+    }
+  }
+
   public override async readWriterLock(): Promise<WriterLockInfo | null> {
     await this.ensureInitialized()
     const lockFile = path.join(this.lockDir, FileSystemStorage.WRITER_LOCK_FILE)

src/storage/baseStorage.ts

@@ -125,6 +125,36 @@ export interface WriterLockInfo {
   rootDir?: string           // Convenience for log lines / error messages
 }
 
+/**
+ * THE CLEAN-CLOSE RECORD. Written by `releaseWriterLock()` at the instant it
+ * gives up the writer lock, naming the lock identity it released. The next
+ * `acquireWriterLock()` reads it and can then say — from a RECORD, not from a
+ * guess — whether the previous writer left on purpose.
+ *
+ * Why a record and not PID liveness: "the recorded PID is no longer alive" is
+ * true of every orderly restart AND of every crash, so the two were reported
+ * identically ("appears dead") and neither could be trusted. Worse, the same
+ * inference fails the other way when the operating system RECYCLES the pid —
+ * a live unrelated process makes a long-dead writer's lock look held, and the
+ * store refuses to open naming a pid that was never Brainy. A record settles
+ * both: matched → the previous writer closed cleanly, nothing to recover;
+ * absent → say so, and name what recovery the open will now run.
+ *
+ * Lifecycle: written at release, consumed (deleted) by the next successful
+ * lock claim — a record must never outlive the lock generation it describes,
+ * or it would vouch for a later crash.
+ */
+export interface WriterCloseRecord {
+  pid: number
+  hostname: string
+  /** `startedAt` of the lock this close released — the identity match key. */
+  startedAt: string
+  /** ISO timestamp at which the lock was released. */
+  closedAt: string
+  /** Brainy version that performed the close. */
+  version: string
+}
+
 /**
  * FNV-1a hash returning a 2-char hex bucket (00-ff).
  * Distributes system keys across 256 sub-prefixes to avoid

tests/integration/writer-lock-clean-close.test.ts

@@ -0,0 +1,250 @@
+/**
+ * @module tests/integration/writer-lock-clean-close
+ * @description THE CLEAN-CLOSE CONTRACT for the writer lock.
+ *
+ * A production restart made this lane necessary: a service stopped with exit
+ * code 0, having awaited `close()` on every pooled brain, and its next boot
+ * announced `[brainy] Overwriting stale writer lock … appears dead` for every
+ * store it owned. "The pid is gone" is equally true of an orderly restart and
+ * of a crash, so the message could not tell an operator which one they had.
+ *
+ * The contract pinned here:
+ *   1. A completed close leaves NO lock file and DOES leave a clean-close
+ *      record; the next open says nothing about staleness.
+ *   2. The next lock claim CONSUMES that record — it may never outlive the
+ *      lock generation it describes, or a later crash would read as clean.
+ *   3. A close whose durable steps FAIL still releases the lock (and still
+ *      rethrows the failure).
+ *   4. A killed process (SIGKILL, no close at all) leaves the lock behind with
+ *      NO record, and the next open says exactly that — crash, recovery ahead.
+ *   5. A host application with its own SIGTERM handler is never force-exited
+ *      out from under its own shutdown by Brainy's handler.
+ */
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest'
+import { mkdtempSync, rmSync, existsSync, readFileSync, writeFileSync } from 'node:fs'
+import { spawn } from 'node:child_process'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { Brainy } from '../../src/brainy.js'
+import { NounType } from '../../src/types/graphTypes.js'
+
+const REPO_ROOT = process.cwd()
+const TSX = join(REPO_ROOT, 'node_modules', '.bin', 'tsx')
+
+function makeTempDir(): string {
+  return mkdtempSync(join(tmpdir(), 'brainy-clean-close-'))
+}
+
+/**
+ * Write a child script to disk and start it under tsx. A file (not `tsx -e`)
+ * because the eval form compiles to CommonJS, which has no top-level await.
+ * The script imports Brainy by ABSOLUTE path, so its own dependency
+ * resolution still happens from inside the repository.
+ */
+function startChild(dir: string, body: string): ReturnType<typeof spawn> {
+  const scriptPath = join(dir, 'child-process.mts')
+  writeFileSync(scriptPath, body)
+  // `detached` puts the child in its own process GROUP: tsx runs the script in
+  // a grandchild process, and only a group-wide signal reaches the process
+  // that actually holds the writer lock.
+  return spawn(TSX, [scriptPath], {
+    cwd: REPO_ROOT,
+    stdio: ['ignore', 'pipe', 'pipe'],
+    detached: true
+  })
+}
+
+/** Capture every console.warn/error line emitted while `fn` runs. */
+async function captureConsole<T>(fn: () => Promise<T>): Promise<{ result: T; lines: string[] }> {
+  const lines: string[] = []
+  const origWarn = console.warn
+  const origError = console.error
+  const sink = (...args: unknown[]) => {
+    lines.push(args.map((a) => String(a)).join(' '))
+  }
+  console.warn = sink as typeof console.warn
+  console.error = sink as typeof console.error
+  try {
+    const result = await fn()
+    return { result, lines }
+  } finally {
+    console.warn = origWarn
+    console.error = origError
+  }
+}
+
+/**
+ * Run a child process that opens `dir`, writes one row, prints `READY`, and
+ * then waits forever. Resolves with the child once READY is seen.
+ */
+function spawnHoldingChild(dir: string): Promise<{
+  child: ReturnType<typeof spawn>
+  output: () => string
+}> {
+  const script = `
+    import { Brainy } from ${JSON.stringify(join(REPO_ROOT, 'src', 'brainy.ts'))}
+    const brain = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: ${JSON.stringify(dir)} } })
+    await brain.init()
+    await brain.add({ data: 'row from the child', type: 'concept' })
+    await brain.flush()
+    console.log('READY')
+    setInterval(() => {}, 1000)
+  `
+  const child = startChild(dir, script)
+  let out = ''
+  child.stdout.on('data', (d) => { out += String(d) })
+  child.stderr.on('data', (d) => { out += String(d) })
+  return new Promise((resolvePromise, rejectPromise) => {
+    const timer = setTimeout(() => rejectPromise(new Error(`child never became READY:\n${out}`)), 120_000)
+    child.stdout.on('data', () => {
+      if (out.includes('READY')) {
+        clearTimeout(timer)
+        resolvePromise({ child, output: () => out })
+      }
+    })
+    child.on('exit', (code) => {
+      clearTimeout(timer)
+      if (!out.includes('READY')) rejectPromise(new Error(`child exited ${code} before READY:\n${out}`))
+    })
+  })
+}
+
+describe('writer lock — the clean-close contract', () => {
+  let dir: string
+  let brain: Brainy | null = null
+
+  beforeEach(() => { dir = makeTempDir() })
+
+  afterEach(async () => {
+    if (brain) {
+      try { await brain.close() } catch { /* may already be closed */ }
+      brain = null
+    }
+    try { rmSync(dir, { recursive: true, force: true }) } catch { /* ignore */ }
+  })
+
+  const lockPath = () => join(dir, 'locks', '_writer.lock')
+  const recordPath = () => join(dir, 'locks', '_writer.close')
+
+  it('a completed close leaves no lock, leaves a record, and the reopen is silent about staleness', async () => {
+    brain = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } })
+    await brain.init()
+    expect(existsSync(lockPath())).toBe(true)
+
+    await brain.add({ data: 'seed entity', type: NounType.Concept })
+    await brain.flush()
+    await brain.close()
+    brain = null
+
+    // 1. The lock is gone and the release is RECORDED.
+    expect(existsSync(lockPath())).toBe(false)
+    expect(existsSync(recordPath())).toBe(true)
+    const record = JSON.parse(readFileSync(recordPath(), 'utf-8'))
+    expect(record.pid).toBe(process.pid)
+    expect(typeof record.closedAt).toBe('string')
+    expect(typeof record.startedAt).toBe('string')
+
+    // 2. The reopen says nothing about a stale lock.
+    const { result: reopened, lines } = await captureConsole(async () => {
+      const next = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } })
+      await next.init()
+      return next
+    })
+    brain = reopened
+    expect(lines.filter((l) => /stale writer lock|appears dead/i.test(l))).toEqual([])
+
+    // 3. The claim CONSUMED the record — it must not outlive its lock generation.
+    expect(existsSync(recordPath())).toBe(false)
+    expect(existsSync(lockPath())).toBe(true)
+  }, 120_000)
+
+  it('releases the writer lock even when a durable close step fails — and still rethrows', async () => {
+    brain = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } })
+    await brain.init()
+    await brain.add({ data: 'seed entity', type: NounType.Concept })
+    await brain.flush()
+    expect(existsSync(lockPath())).toBe(true)
+
+    // Inject a failure into a durable close step (the counts flush).
+    const storage = (brain as unknown as { storage: { flushCounts: () => Promise<void> } }).storage
+    const boom = new Error('injected: counts flush failed during close')
+    storage.flushCounts = async () => { throw boom }
+
+    await expect(brain.close()).rejects.toThrow(/injected: counts flush failed/)
+    brain = null
+
+    // The lock is released regardless: a process on its way out holds nothing.
+    expect(existsSync(lockPath())).toBe(false)
+
+    // And the next writer opens without a stale-lock verdict.
+    const { lines } = await captureConsole(async () => {
+      const next = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } })
+      await next.init()
+      await next.close()
+    })
+    expect(lines.filter((l) => /appears dead/i.test(l))).toEqual([])
+  }, 120_000)
+
+  it('a SIGKILLed writer leaves the lock with no record, and the next open names the crash', async () => {
+    const { child } = await spawnHoldingChild(dir)
+    expect(existsSync(lockPath())).toBe(true)
+    expect(existsSync(recordPath())).toBe(false)
+
+    // Group-wide: the lock holder is tsx's grandchild, not the spawned pid.
+    process.kill(-(child.pid as number), 'SIGKILL')
+    await new Promise<void>((r) => child.on('exit', () => r()))
+    // The grandchild's death is asynchronous with the wrapper's exit event.
+    await new Promise<void>((r) => setTimeout(r, 500))
+
+    // The lock survives the kill — a dead process releases nothing.
+    expect(existsSync(lockPath())).toBe(true)
+    expect(existsSync(recordPath())).toBe(false)
+
+    const { lines } = await captureConsole(async () => {
+      const next = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } })
+      await next.init()
+      await next.close()
+    })
+    const verdict = lines.filter((l) => /Overwriting stale writer lock/i.test(l))
+    expect(verdict.length).toBe(1)
+    // The verdict must name the ABSENT record and the recovery it implies —
+    // not merely that a pid is gone.
+    expect(verdict[0]).toMatch(/NO\s+clean-close record/i)
+    expect(verdict[0]).toMatch(/crash recovery/i)
+  }, 180_000)
+
+  it("does not force-exit a host application that owns its own SIGTERM handler", async () => {
+    const script = `
+      import { Brainy } from ${JSON.stringify(join(REPO_ROOT, 'src', 'brainy.ts'))}
+      const brain = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: ${JSON.stringify(dir)} } })
+      await brain.init()
+      await brain.add({ data: 'row from the host app', type: 'concept' })
+      await brain.flush()
+      // The host application's OWN graceful shutdown, registered after Brainy's.
+      process.on('SIGTERM', async () => {
+        await new Promise((r) => setTimeout(r, 1500))
+        console.log('APP-CLOSE-DONE')
+        process.exit(0)
+      })
+      console.log('READY')
+      setInterval(() => {}, 1000)
+    `
+    const child = startChild(dir, script)
+    let out = ''
+    child.stdout.on('data', (d) => { out += String(d) })
+    child.stderr.on('data', (d) => { out += String(d) })
+    await new Promise<void>((r, reject) => {
+      const timer = setTimeout(() => reject(new Error(`child never became READY:\n${out}`)), 120_000)
+      child.stdout.on('data', () => { if (out.includes('READY')) { clearTimeout(timer); r() } })
+      child.on('exit', () => { clearTimeout(timer); if (!out.includes('READY')) reject(new Error(`child died:\n${out}`)) })
+    })
+
+    process.kill(-(child.pid as number), 'SIGTERM')
+    const code = await new Promise<number | null>((r) => child.on('exit', (c) => r(c)))
+    expect(code).toBe(0)
+    // The host's own shutdown ran to completion — Brainy's handler did not
+    // exit the process out from under it.
+    expect(out).toContain('APP-CLOSE-DONE')
+  }, 180_000)
+})