fix(log): acked writes survive power loss; rejected writes never silently commit — the kill-matrix goes 11/11 with zero .fails debt

Two release-blocking findings from the durability kill-matrix, both fixed
in the owning layer:

1. LOG-AUTHORITY REPLAY AT OPEN: durable-at-ack fsynced the fact before
   the ack, but open() truncated every fact above the manifest — after a
   power loss that takes the un-fsynced tmp+rename canonical bytes, the
   acked write's ONLY durable copy was discarded. Now: under 'log'
   authority, open() REPLAYS intact facts above the manifest into
   canonical (FactLog.peekFactsAbove — CRC-gated, order-sorted) and
   advances the manifest to cover them; tree-authority brains keep the
   truncate contract they were promised. Pinned end to end: the power-loss
   row constructs the exact disk state (fsynced log, vanished canonical
   rename) and the acked write lives.

2. NO SILENT COMMIT: commitSingleOp buffered the generation BEFORE the
   fact append; an append failure (ENOSPC) rejected the caller but the
   next flush durably committed the generation with NO fact — a permanent
   silent log gap. Now the failure path un-buffers and returns the counter
   reservation: nothing commits, the log stays gap-free, and the canonical
   execute-residue orphan is the documented crash-equivalent.

Plus: the kill-matrix itself (11 rows — every commit-path fault point ×
reopen-as-crash recovery contract, at-ack variants, disk-full row; five
new zero-cost faultPoint sites), the log-authority pin suite (oracle
green/red/state-differs, flip refusal, switch survives reopen, 9/9), and
the group-commit covering pins (5/5).

Gates: unit 2002/2002 (152 files) · integration 785 · conformance 27/27.

src/db/factLog.ts

@@ -342,6 +342,34 @@ export class FactLog {
    * crash between fact-append and the commit point). After open, the log is
    * exactly the committed prefix.
    */
+  /**
+   * Read (without truncating) every intact fact ABOVE a generation — the
+   * log-authority recovery surface: after a crash, facts beyond the
+   * manifest watermark that survived with valid CRCs are ACKED writes in
+   * durable-at-ack mode, and the owner REPLAYS them instead of letting
+   * open() truncate them. Must be called BEFORE open() (it reads the raw
+   * segments directly; the torn tail's invalid suffix is ignored exactly
+   * like open() would).
+   */
+  async peekFactsAbove(committedGeneration: number): Promise<CommitFact[]> {
+    const stored = (await this.storage.readRawObject(FACTS_MANIFEST_PATH)) as FactsManifest | null
+    if (!stored || typeof stored !== 'object' || !Array.isArray(stored.segments)) return []
+    if (stored.formatVersion !== FACTS_FORMAT_VERSION) return []
+    const out: CommitFact[] = []
+    const files = [...stored.segments.map((s) => s.file)]
+    if (stored.tailSegment) files.push(stored.tailSegment)
+    for (const file of files) {
+      const bytes = await this.storage.readRawBytes(`${FACTS_PREFIX}/${file}`)
+      if (bytes === null) continue
+      const { facts } = parseSegment(file, bytes)
+      for (const f of facts) {
+        if (f.generation > committedGeneration) out.push(f)
+      }
+    }
+    out.sort((a, b) => a.generation - b.generation)
+    return out
+  }
+
   async open(committedGeneration: number): Promise<void> {
     const stored = (await this.storage.readRawObject(FACTS_MANIFEST_PATH)) as FactsManifest | null
     if (stored && typeof stored === 'object' && Array.isArray(stored.segments)) {

src/db/generationStore.ts

@@ -45,6 +45,7 @@ import type {
   GenerationStorage,
   TxLogEntry
 } from './types.js'
+import { readLogAuthority } from './logAuthority.js'
 import { FactLog, storageSupportsFactLog, type CommitFact, type FactOp } from './factLog.js'
 import { GenerationSegmentStore, type FoldGeneration } from './generationSegments.js'
 import { crc32c } from '../utils/crc32c.js'
@@ -88,12 +89,43 @@ export const GENERATIONS_PREFIX = '_generations'
  *   IS committed); the tx-log append has NOT happened yet. A crash here must
  *   keep the transaction (the tx-log is advisory metadata, not the source of
  *   commit truth).
+ * - `'transact-after-fact-sync'` — the batch's fact is appended AND fsynced,
+ *   but neither the counter nor the manifest advanced. A crash here must cost
+ *   the whole batch: recovery restores the before-images and open() truncates
+ *   the synced fact back to the manifest watermark.
+ *
+ * Single-op (Model-B group-commit) phases — `commitSingleOp`:
+ *
+ * - `'singleop-after-execute'` — the live canonical write has applied (tmp+
+ *   rename, not individually fsynced); no history, fact, or generation record
+ *   exists yet. A crash here must cost only the never-returned ack — the
+ *   baseline stays intact and the log stays at the committed watermark.
+ * - `'singleop-after-fact-append'` — the fact is appended (and, in at-ack
+ *   mode, fsynced); the manifest never saw the generation. A crash here must
+ *   cost the buffered history + the fact (open() truncates it back), never
+ *   the baseline.
+ *
+ * Pending-tier flush phases — `flushPendingSingleOps`:
+ *
+ * - `'flush-after-staging'` — the window's record-set dirs are written but not
+ *   fsynced and the manifest never advanced. A crash here must cost only the
+ *   window's HISTORY (drop-without-restore) — the acked live writes stay.
+ * - `'flush-before-manifest'` — staging is fsynced and the facts are fsynced,
+ *   but the manifest never advanced. A crash here must cost only the window's
+ *   history and its facts (truncated at open) — the acked live writes stay.
+ * - `'before-manifest-rename'` is ALSO fired by the flush path just before its
+ *   commit point (see `flushPendingSingleOpsUnlocked`).
  */
 export type CommitFaultPhase =
   | 'after-staging'
   | 'after-execute'
   | 'before-manifest-rename'
   | 'after-manifest-rename'
+  | 'transact-after-fact-sync'
+  | 'singleop-after-execute'
+  | 'singleop-after-fact-append'
+  | 'flush-after-staging'
+  | 'flush-before-manifest'
 
 /**
  * @description Identifies which ids a transaction touches, split by kind.
@@ -461,6 +493,54 @@ export class GenerationStore {
     // hosts no fact log (readers fall back to canonical enumeration).
     if (storageSupportsFactLog(this.storage)) {
       this.factLog = new FactLog(this.storage)
+      // LOG-AUTHORITY REPLAY (durable-at-ack's recovery half): when this
+      // brain's stored authority is the log, an intact fact ABOVE the
+      // manifest is an ACKED write whose canonical bytes may not have
+      // survived the crash — its fsynced fact is the ONLY durable copy.
+      // Truncating it would lose an acked write; instead REPLAY it into
+      // canonical and advance the manifest to cover it. Tree-authority
+      // brains keep the truncate contract (their acks never promised the
+      // fact was durable). Derived indexes reconcile through the normal
+      // drift machinery at open — same as group-commit recovery.
+      const authority = await readLogAuthority(this.storage)
+      if (authority.authority === 'log') {
+        const orphans = await this.factLog.peekFactsAbove(this.committed)
+        if (orphans.length > 0) {
+          for (const fact of orphans) {
+            for (const op of fact.ops) {
+              const image =
+                op.record === null
+                  ? { metadata: null, vector: null }
+                  : { metadata: op.record.metadata, vector: op.record.vector }
+              if (op.kind === 'verb') await this.storage.writeVerbRaw(op.id, image)
+              else await this.storage.writeNounRaw(op.id, image)
+            }
+            this.committed = fact.generation
+            this.appendCommittedGen(fact.generation)
+            this.setDelta(fact.generation, {
+              nouns: new Set(fact.ops.filter((o) => o.kind === 'noun').map((o) => o.id)),
+              verbs: new Set(fact.ops.filter((o) => o.kind === 'verb').map((o) => o.id)),
+              timestamp: fact.timestamp,
+              bytes: 0
+            })
+          }
+          if (this.counter < this.committed) this.counter = this.committed
+          await this.persistCounterUnlocked()
+          const manifest: GenerationManifest = {
+            version: 1,
+            generation: this.committed,
+            committedAt: new Date().toISOString(),
+            horizon: this.horizonGen
+          }
+          await this.storage.writeRawObject(MANIFEST_PATH, manifest)
+          await this.storage.syncRawObjects([MANIFEST_PATH])
+          prodLog.warn(
+            `[GenerationStore] log-authority recovery REPLAYED ${orphans.length} acked ` +
+              `fact(s) beyond the manifest into canonical (now committed at ${this.committed}) — ` +
+              `an acked write is never lost`
+          )
+        }
+      }
       await this.factLog.open(this.committed)
     } else {
       this.factLog = null
@@ -977,6 +1057,9 @@ export class GenerationStore {
           await this.factLog.append(fact)
           await this.factLog.sync()
         }
+        // A crash here must cost the whole batch: the synced fact is truncated
+        // back at open() and the before-images are restored byte-identically.
+        faultPoint('transact-after-fact-sync')
 
         // -- 5. Counter + manifest rename (COMMIT POINT) ----------------------
         await this.persistCounterUnlocked()
@@ -1278,6 +1361,12 @@ export class GenerationStore {
         throw err
       }
       this.inTransact = false
+      // Test-only crash simulation (direct call — a throw propagates with no
+      // cleanup, exactly like a process death; recovery-on-open restores the
+      // contract). A crash here must cost only the never-returned ack: the
+      // live canonical write applied, but no history, fact, or generation
+      // record exists for it yet.
+      if (this.commitFaultInjector) this.commitFaultInjector('singleop-after-execute')
 
       // Buffer the pending generation + make it instantly visible to reads.
       this.pendingBuffer.set(gen, { nouns: nounBefore, verbs: verbBefore, timestamp })
@@ -1297,13 +1386,35 @@ export class GenerationStore {
       //   the log's group-commit (many concurrent writers share ONE sync) —
       //   an acked write's fact survives power loss, by contract.
       if (this.factLog) {
-        await this.factLog.append(
-          await this.buildCommitFact({ generation: gen, timestamp, nouns, verbs })
-        )
-        if (this.logDurability === 'at-ack') {
-          await this.factLog.ensureSynced()
+        try {
+          await this.factLog.append(
+            await this.buildCommitFact({ generation: gen, timestamp, nouns, verbs })
+          )
+          if (this.logDurability === 'at-ack') {
+            await this.factLog.ensureSynced()
+          }
+        } catch (err) {
+          // A rejected write must NOT commit: the generation was buffered
+          // before the append, so un-buffer it and return the counter
+          // reservation — otherwise the next flush would durably commit a
+          // generation with NO fact, a silent log gap a later replay would
+          // turn into loss. Canonical bytes from execute() remain as an
+          // uncommitted orphan — identical to a crash at this point; never
+          // a torn committed state.
+          this.pendingBuffer.delete(gen)
+          const idx = this.pendingGens.lastIndexOf(gen)
+          if (idx !== -1) this.pendingGens.splice(idx, 1)
+          this.invalidateChains()
+          if (this.counter === gen) this.counter = gen - 1
+          throw err
         }
       }
+      // Test-only crash simulation. A crash here must cost the buffered
+      // history + the appended fact in 'deferred' mode (open() truncates it
+      // back to the manifest watermark) — while under 'log' authority the
+      // intact fact is REPLAYED at open, never the baseline or the applied
+      // live write.
+      if (this.commitFaultInjector) this.commitFaultInjector('singleop-after-fact-append')
       this.schedulePendingFlush()
       return { generation: gen, timestamp }
     })
@@ -1422,6 +1533,11 @@ export class GenerationStore {
         logEntries.push({ generation: gen, timestamp: buf.timestamp })
       }
 
+      // Test-only crash simulation. A crash here must cost only the window's
+      // HISTORY: un-fsynced record-set dirs may sit above the manifest, and
+      // recovery drops them WITHOUT restore — the acked live writes stay.
+      if (this.commitFaultInjector) this.commitFaultInjector('flush-after-staging')
+
       // ONE fsync for the whole window — the durability-batching win.
       await this.storage.syncRawObjects(stagedPaths)
 
@@ -1431,6 +1547,12 @@ export class GenerationStore {
       // generation without its durable fact.
       await this.factLog?.sync()
 
+      // Test-only crash simulation. A crash here must cost only the window's
+      // history and its (already fsynced) facts — open() truncates the facts
+      // back to the manifest watermark and drops the staged group-commit dirs
+      // without restore; the acked live writes stay.
+      if (this.commitFaultInjector) this.commitFaultInjector('flush-before-manifest')
+
       // Test-only crash simulation: a throwing injector here leaves the staged
       // group-commit generation dirs on disk with NO manifest advance — the
       // exact "crashed mid-flush" state recovery must DROP-WITHOUT-RESTORE