fix(storage): a clean close is recorded, and the writer lock is always given up

A production restart made this necessary: a service stopped with exit code 0,
having awaited close() on every pooled brain, and its next boot announced
"Overwriting stale writer lock ... appears dead" for every store it owned.
Nothing had crashed. "The recorded pid is gone" is equally true of an orderly
restart and of a crash, so the verdict could not tell an operator which one
they had — and when the OS recycles a pid it fails the other way, refusing to
open a store whose writer died days ago.

Three changes, all at the law:

- close() is two parts, and the second is unconditional. The durable steps
  (flush, markers, component close, plugin deactivate, buffer drain) move to
  closeDurableSteps(); the terminal releases — the flush-request watcher, the
  WRITER LOCK, the VFS timers, the terminal `closed` flag — always run. The
  original failure is narrated with what it costs the next open, then rethrown.
- releaseWriterLock() writes a CLEAN-CLOSE RECORD (`locks/_writer.close`)
  naming the lock generation it released; the next claim consumes it, so a
  record can never vouch for a later crash. An open reads the record instead
  of guessing: recorded → nothing to recover; absent → say so, and name the
  crash recovery this open will now run.
- The signal path stops failing in a batch. It was one try around a loop over
  every open brain, so the first instance whose flush rejected stranded every
  remaining brain's lock and markers — at exit code 0. Now: per-instance
  isolation, the generation store's close (the clean-shutdown marker, without
  which the next open folds the whole log) is part of shutdown, the lock is
  given up in a finally, and the handler no longer calls process.exit() when
  the host application has its own signal handler — that race truncated the
  host's own close() mid-flight.

Pins: tests/integration/writer-lock-clean-close.test.ts — completed close
leaves no lock and a consumed-once record with a silent reopen; a failing
durable step still releases and still rethrows; SIGKILL leaves the lock with
no record and the reopen names the crash; a host SIGTERM handler runs to
completion.

Branch plan (10 lines):
 1. writer lock: clean-close record + always-release close  [this commit]
 2. open narration: an always-on channel; production clamps prodLog to ERROR,
    which is why a three-minute open printed nothing
 3. open narration: per-phase lines as each phase ENDS, with progress cadence
 4. measure both real-store fixtures on the box, before/after
 5. move the generation-log fold out of the foreground where the serving law
    allows; durable resumable progress marker
 6. same for the VFS bootstrap
 7. counts: a legacy container-rule ledger must not keep serving wrong
    denominators; counts.json written atomically
 8. counts pin with scar directories; two copies of one archive agree
 9. docs/canonical-layout-ratification.md — 12 facts confirmed/corrected
10. report: MEASURED before/after, findings, and whether this is 10.4.4

src/brainy.ts

@@ -1856,76 +1856,112 @@ export class Brainy<T = any> implements BrainyInterface<T> {
    * NOTE: Registers globally (once for all instances) to avoid MaxListenersExceededWarning
    */
   private registerShutdownHooks(): void {
+    /**
+     * The signal-path shutdown. THREE LAWS, each written by a production
+     * shutdown that looked clean and wasn't:
+     *
+     * 1. PER-INSTANCE ISOLATION. This used to be one `try` around a loop over
+     *    every open brain: the first instance whose flush rejected aborted the
+     *    loop, so every remaining brain kept its writer lock and its unwritten
+     *    markers — and the process still exited 0. A pool of brains failed in
+     *    a batch, not one at a time.
+     * 2. THE MARKER IS PART OF SHUTDOWN. Flushing the indexes without closing
+     *    the generation store leaves the clean-shutdown marker unwritten, so
+     *    the NEXT open reads the store as crashed and folds the whole
+     *    generation log — measured in tens of seconds on a real store, paid on
+     *    every restart, after a shutdown the operator saw exit 0.
+     * 3. THE LOCK IS ALWAYS GIVEN UP. In a `finally`, per instance: a process
+     *    on its way out holds nothing.
+     */
     const flushOnShutdown = async () => {
       console.log('Shutdown signal received - flushing pending data...')
-      try {
-        let flushedCount = 0
-        for (const instance of Brainy.instances) {
-          if (instance.initialized) {
-            // Flush all buffered data, then close to release resources (timers, handles)
-            await Promise.all([
-              (async () => {
-                if (instance.storage && typeof instance.storage.flushCounts === 'function') {
-                  await instance.storage.flushCounts()
-                }
-              })(),
-              (async () => {
-                if (instance.metadataIndex && typeof instance.metadataIndex.flush === 'function') {
-                  await instance.metadataIndex.flush()
-                }
-              })(),
-              (async () => {
-                if (instance.graphIndex && typeof instance.graphIndex.flush === 'function') {
-                  await instance.graphIndex.flush()
-                }
-              })(),
-              (async () => {
-                if (instance.index && typeof instance.index.flush === 'function') {
-                  await instance.index.flush()
-                }
-              })()
-            ])
-            // Close components to stop timers that would prevent clean process exit
-            await Promise.all([
-              (async () => {
-                if (instance.graphIndex && typeof instance.graphIndex.close === 'function') {
-                  await instance.graphIndex.close()
-                }
-              })(),
-              (async () => {
-                const index = instance.index as JsHnswVectorIndex & VectorIndexOptionalHooks
-                if (index && typeof index.close === 'function') {
-                  await index.close()
-                }
-              })(),
-              (async () => {
-                const metadataIndex = instance.metadataIndex as MetadataIndexManager & MetadataIndexOptionalHooks
-                if (metadataIndex && typeof metadataIndex.close === 'function') {
-                  await metadataIndex.close()
-                }
-              })(),
-              // Release the writer lock so a successor process can take over.
-              // No-op for readers and for backends without locking.
-              (async () => {
-                if (instance.storage && typeof instance.storage.releaseWriterLock === 'function') {
-                  await instance.storage.releaseWriterLock()
-                }
-              })(),
-              // Stop the flush-request watcher to release its interval timer.
-              (async () => {
-                if (instance.storage && typeof instance.storage.stopFlushRequestWatcher === 'function') {
-                  instance.storage.stopFlushRequestWatcher()
-                }
-              })(),
-            ])
-            flushedCount++
+      let flushedCount = 0
+      let failedCount = 0
+      // Snapshot: close() splices Brainy.instances while we iterate.
+      for (const instance of [...Brainy.instances]) {
+        if (!instance.initialized) continue
+        try {
+          // Flush all buffered data (parallel across components, this brain only).
+          await Promise.all([
+            (async () => {
+              if (instance.storage && typeof instance.storage.flushCounts === 'function') {
+                await instance.storage.flushCounts()
+              }
+            })(),
+            (async () => {
+              if (instance.metadataIndex && typeof instance.metadataIndex.flush === 'function') {
+                await instance.metadataIndex.flush()
+              }
+            })(),
+            (async () => {
+              if (instance.graphIndex && typeof instance.graphIndex.flush === 'function') {
+                await instance.graphIndex.flush()
+              }
+            })(),
+            (async () => {
+              if (instance.index && typeof instance.index.flush === 'function') {
+                await instance.index.flush()
+              }
+            })()
+          ])
+
+          // Close the generation store: persists the counter, advances the
+          // fold checkpoint, and stamps the clean-shutdown marker LAST — the
+          // one step that decides whether the next open adopts or folds. Law 2.
+          if (instance.generationStore && !instance.isReadOnly) {
+            await instance.generationStore.close()
+          }
+
+          // Close components to stop timers that would prevent clean process exit
+          await Promise.all([
+            (async () => {
+              if (instance.graphIndex && typeof instance.graphIndex.close === 'function') {
+                await instance.graphIndex.close()
+              }
+            })(),
+            (async () => {
+              const index = instance.index as JsHnswVectorIndex & VectorIndexOptionalHooks
+              if (index && typeof index.close === 'function') {
+                await index.close()
+              }
+            })(),
+            (async () => {
+              const metadataIndex = instance.metadataIndex as MetadataIndexManager & MetadataIndexOptionalHooks
+              if (metadataIndex && typeof metadataIndex.close === 'function') {
+                await metadataIndex.close()
+              }
+            })()
+          ])
+          flushedCount++
+        } catch (error) {
+          failedCount++
+          console.error('Failed to flush one Brainy instance on shutdown:', error)
+        } finally {
+          // Law 3 — the lock and the watcher go regardless.
+          try {
+            if (instance.storage && typeof instance.storage.stopFlushRequestWatcher === 'function') {
+              instance.storage.stopFlushRequestWatcher()
+            }
+          } catch (error) {
+            console.error('Failed to stop the flush-request watcher on shutdown:', error)
+          }
+          try {
+            if (instance.storage && typeof instance.storage.releaseWriterLock === 'function') {
+              await instance.storage.releaseWriterLock()
+            }
+          } catch (error) {
+            console.error('Failed to release the writer lock on shutdown:', error)
           }
         }
-        if (flushedCount > 0) {
-          console.log(`Flushed successfully (${flushedCount} instance${flushedCount > 1 ? 's' : ''})`)
-        }
-      } catch (error) {
-        console.error('Failed to flush on shutdown:', error)
+      }
+      if (flushedCount > 0) {
+        console.log(`Flushed successfully (${flushedCount} instance${flushedCount > 1 ? 's' : ''})`)
+      }
+      if (failedCount > 0) {
+        console.error(
+          `${failedCount} Brainy instance${failedCount > 1 ? 's' : ''} did not complete shutdown — ` +
+            `their writer locks were released, but their next open will run crash recovery.`
+        )
       }
     }
 
@@ -1933,13 +1969,32 @@ export class Brainy<T = any> implements BrainyInterface<T> {
     // kept as statics so the last live instance's close() can deregister them
     // — the signal handles they hold are ref'd and would otherwise keep the
     // process alive forever after every brain is closed.
+    /**
+     * Exit the process ONLY when Brainy is the sole handler for this signal.
+     *
+     * Registering a signal listener suppresses Node's default terminate
+     * behaviour, so a library that attaches one must either exit or be sure
+     * someone else will. Brainy attaching one AND exiting was the wrong half
+     * of that choice for every host application with its own graceful
+     * shutdown: both handlers run concurrently, and whichever finishes first
+     * wins — a library flush finishing before an application's close()
+     * terminated that close mid-flight, at exit code 0, with locks and
+     * markers unwritten. When the host has its own handler (listener count
+     * above our own), the host owns the exit; Brainy only makes its data
+     * durable and steps aside.
+     */
+    const exitIfSoleShutdownOwner = (signal: 'SIGTERM' | 'SIGINT'): void => {
+      if (process.listenerCount(signal) <= 1) {
+        process.exit(0)
+      }
+    }
     Brainy.sigtermListener = async () => {
       await flushOnShutdown()
-      process.exit(0)
+      exitIfSoleShutdownOwner('SIGTERM')
     }
     Brainy.sigintListener = async () => {
       await flushOnShutdown()
-      process.exit(0)
+      exitIfSoleShutdownOwner('SIGINT')
     }
     Brainy.beforeExitListener = async () => {
       // Self-deregister FIRST: Node re-emits 'beforeExit' after every event-
@@ -18877,12 +18932,105 @@ export class Brainy<T = any> implements BrainyInterface<T> {
   }
 
   /**
-   * Close and cleanup
+   * @description Close and clean up: flush every buffered component, stamp
+   * the durability markers, release resources, then give up the writer lock.
    *
-   * Now flushes HNSW dirty nodes before closing
-   * This ensures deferred persistence mode data is saved
+   * TWO PARTS, AND THE SECOND IS UNCONDITIONAL. Everything that persists data
+   * runs in {@link closeDurableSteps}; the terminal releases — the flush-request
+   * watcher, the WRITER LOCK, the VFS timers, and the terminal `closed` flag —
+   * run whether those steps succeeded or not, in a `finally`. A close that
+   * threw halfway used to strand the writer lock on disk with this process's
+   * (soon dead) pid in it, so the next boot of every affected store announced
+   * `Overwriting stale writer lock … appears dead` after an orderly exit and
+   * an operator had to decide whether their database had crashed. A closed
+   * brain holds no lock — there is no failure for which the opposite is the
+   * safer answer.
+   *
+   * The original failure is never swallowed: it is narrated with what it costs
+   * the next open, then rethrown to the caller.
+   * @returns Nothing.
+   * @throws The first failure from the durable close steps, after the
+   *   terminal releases have run.
    */
   async close(): Promise<void> {
+    let closeFailure: unknown = null
+    try {
+      await this.closeDurableSteps()
+    } catch (error) {
+      closeFailure = error
+    }
+
+    // ---- TERMINAL RELEASES: always, even after a failure above ----
+
+    // Stop the cross-process flush-request watcher (no-op if never started).
+    try {
+      if (this.storage && typeof this.storage.stopFlushRequestWatcher === 'function') {
+        this.storage.stopFlushRequestWatcher()
+      }
+    } catch (error) {
+      console.warn('[Brainy] close: stopping the flush-request watcher failed:', error)
+    }
+
+    // Release the writer lock. Runs after the metadata buffer drain in
+    // closeDurableSteps() — otherwise a pending write could land after a
+    // successor writer claimed the lock — and runs even if that drain threw:
+    // holding a lock from a process that is about to exit locks the store's
+    // next boot out of a clean verdict.
+    try {
+      if (this.storage && typeof this.storage.releaseWriterLock === 'function') {
+        await this.storage.releaseWriterLock()
+      }
+    } catch (error) {
+      console.warn('[Brainy] close: releasing the writer lock failed:', error)
+    }
+
+    // Shut down the VFS: stops its background maintenance interval and the
+    // PathResolver's — both are ref'd timers that would keep the process
+    // alive after the last brain closes (consumer-reported hang).
+    try {
+      if (this._vfs) {
+        await this._vfs.close()
+      }
+    } catch (error) {
+      console.warn('[Brainy] close: VFS shutdown failed:', error)
+    }
+
+    this.initialized = false
+    // close() is terminal: block lazy re-initialization on any subsequent
+    // operation (ensureInitialized() throws once this is set). Set even when
+    // the durable steps failed — a half-closed brain must not keep serving.
+    this.closed = true
+
+    // Drop this instance from the global registry, and when it was the last
+    // one, deregister the global shutdown hooks — their ref'd signal handles
+    // would otherwise keep the process alive after every brain is closed.
+    const instanceIndex = Brainy.instances.indexOf(this)
+    if (instanceIndex !== -1) {
+      Brainy.instances.splice(instanceIndex, 1)
+    }
+    Brainy.deregisterShutdownHooksIfIdle()
+
+    if (closeFailure !== null) {
+      console.error(
+        `[Brainy] close FAILED partway: ` +
+          `${closeFailure instanceof Error ? closeFailure.message : String(closeFailure)}\n` +
+          `  This brain is closed and holds no writer lock, but the clean-shutdown ` +
+          `marker may not have been written — the next open will run crash recovery ` +
+          `(a generation-log fold) and report its wall.`
+      )
+      throw closeFailure
+    }
+  }
+
+  /**
+   * @description The durable half of {@link close}: flush every component,
+   * persist the generation counter and its markers, close the components,
+   * deactivate plugins, drain the metadata write buffer. Separated from
+   * `close()` so the terminal releases there can run in a `finally` — see that
+   * method's contract.
+   * @returns Nothing.
+   */
+  private async closeDurableSteps(): Promise<void> {
     // Persistence cadence teardown: no background flush may fire after close
     // begins (close() runs its own final flush).
     if (this._persistIdleTimer) {
@@ -19033,38 +19181,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
       }
     }
 
-    // Stop the cross-process flush-request watcher (no-op if never started).
-    if (this.storage && typeof this.storage.stopFlushRequestWatcher === 'function') {
-      this.storage.stopFlushRequestWatcher()
-    }
-
-    // Release the writer lock (no-op for readers and for backends that don't
-    // hold a lock). Must run after the metadata buffer drain — otherwise a
-    // pending write could land after a successor writer claimed the lock.
-    if (this.storage && typeof this.storage.releaseWriterLock === 'function') {
-      await this.storage.releaseWriterLock()
-    }
-
-    // Shut down the VFS: stops its background maintenance interval and the
-    // PathResolver's — both are ref'd timers that would keep the process
-    // alive after the last brain closes (consumer-reported hang).
-    if (this._vfs) {
-      await this._vfs.close()
-    }
-
-    this.initialized = false
-    // close() is terminal: block lazy re-initialization on any subsequent
-    // operation (ensureInitialized() throws once this is set).
-    this.closed = true
-
-    // Drop this instance from the global registry, and when it was the last
-    // one, deregister the global shutdown hooks — their ref'd signal handles
-    // would otherwise keep the process alive after every brain is closed.
-    const instanceIndex = Brainy.instances.indexOf(this)
-    if (instanceIndex !== -1) {
-      Brainy.instances.splice(instanceIndex, 1)
-    }
-    Brainy.deregisterShutdownHooksIfIdle()
   }
 }