fix(aggregation): the lifecycle cluster — flush stamps, behind-stamp catches up incrementally, the native rebuild finally gets invoked, deletes are never silently skipped

SELF-ENGINE-LIFECYCLE-SPRINT + BRAINY-PROD-LATENCY-TRIAD, the four asks:

(a) brain.flush() persists aggregation state stamped at the committed
    generation. The stamp used to advance only at close(), so a long-lived
    writer that flushes but never closes — the primary production shape —
    left every write window behind the stamp, and ANY unclean exit forced a
    whole-store backfill walk (per-entity work, measured >60s and
    door-starving on a 9k-row production brain) on the first stats call.

(b) BEHIND-stamp adoption becomes adopt + INCREMENTAL CATCH-UP: the exact
    missing window (stamp, committed] resolves its affected-id set from the
    fact log and reconciles each entity with time-travel before/after reads
    (asOf at both window bounds) through the same delta algebra the live
    hooks use — cost bounded by writes since the last flush, never store
    size, and exact under interleaving because reconciliation targets the
    FIXED window end while later writes chain through hooks. Oversized
    windows (>5000 affected) and unreadable windows demote to the announced
    rescan — never a silent partial serve.

(c) The native provider's parallel rebuildAggregate — on the contract since
    8.x but never invoked anywhere — is now the backfill walk's preferred
    door: one call per aggregate with source-matched entities, replacing
    the per-entity FFI stream.

(d) A delete whose before-image is unavailable can no longer SKIP the
    aggregation hook silently (counts drifted upward forever): both delete
    paths (remove() and transact) flag an exact rescan, loudly.

Pins: integration (flush stamp; unclean-exit reopen → exact counts through
an add + group-move + delete window with the walk spy proving ZERO
whole-store walks) + unit (provider rebuild invoked once with filtered
entities; flagAllForRescan; reconcile delta algebra). Gates: unit 1913/1913
· integration 760 · conformance 27/27.

src/aggregation/AggregationIndex.ts

@@ -371,6 +371,15 @@ export class AggregationIndex {
    */
   private pendingAdopt = new Set<string>()
 
+  /**
+   * Aggregates adopted with a BEHIND stamp: name → the exact generation
+   * window `(from, to]` whose writes the adopted state has not seen. The
+   * owner (Brainy) drains this via {@link getPendingCatchUps} +
+   * {@link reconcileEntity} + {@link finishCatchUp} BEFORE serving queries —
+   * cost bounded by the window's affected entities, never store size.
+   */
+  private pendingCatchUp = new Map<string, { from: number; to: number }>()
+
   /**
    * In-flight rescan targets. While a name has a staging map, ALL
    * contributions (the walk's and concurrent write hooks') land there instead
@@ -437,25 +446,47 @@ export class AggregationIndex {
   }
 
   /**
-   * May this persisted state be ADOPTED? When the store exposes its committed
-   * watermark, the state's `sourceGeneration` must EQUAL it: behind means
-   * later writes are missing from the state (unclean shutdown); ahead means
-   * it counts writes that no longer exist (e.g. a fact-log truncation on a
-   * copied store pulled the watermark back). Either way: one exact rescan,
-   * said out loud — never a silent adopt. Stores without the capability (and
-   * pre-stamp state on them) fall back to hash-only adoption.
+   * The adoption verdict for persisted state, against the store's committed
+   * watermark (SELF-ENGINE-LIFECYCLE-SPRINT ask (b) — behind-stamp is no
+   * longer a whole-store rescan):
+   *
+   * - `'adopt'` — stamp equals the watermark (clean), or the store has no
+   *   watermark capability (hash-only adoption, the pre-stamp behavior).
+   * - `'catchup'` — stamp is BEHIND the watermark (an unclean exit after
+   *   later writes, or a long-lived writer whose last flush predates recent
+   *   writes). The state is exact AS OF its stamp, so it is adopted and the
+   *   missing window `(stamp, committed]` is reconciled INCREMENTALLY per
+   *   affected entity via time-travel reads — bounded by writes since the
+   *   last flush, never by store size. The owner drains
+   *   {@link getPendingCatchUps} before serving queries.
+   * - `'rescan'` — no stamp (pre-stamp state on a stamped store) or stamp
+   *   AHEAD of the watermark (e.g. a fact-log truncation on a copied store
+   *   pulled the watermark back): the state over-counts unverifiably; one
+   *   exact rescan, said out loud.
    */
-  private stateGenerationAdoptable(name: string, stateData: unknown): boolean {
+  private stateAdoptionVerdict(
+    name: string,
+    stateData: unknown
+  ): 'adopt' | 'catchup' | 'rescan' {
     const committed = this.storage.committedGeneration?.() ?? null
-    if (committed === null) return true
+    if (committed === null) return 'adopt'
     const raw = (stateData as Record<string, unknown>).sourceGeneration
     const stamped = typeof raw === 'number' ? raw : null
-    if (stamped === committed) return true
+    if (stamped === committed) return 'adopt'
+    if (stamped !== null && stamped < committed) {
+      this.pendingCatchUp.set(name, { from: stamped, to: committed })
+      prodLog.info(
+        `[Aggregation] '${name}': persisted state is at generation ${stamped}, store is at ` +
+          `${committed} — adopting and reconciling the ${committed - stamped}-generation window ` +
+          `incrementally (no store rescan)`
+      )
+      return 'catchup'
+    }
     prodLog.warn(
       `[Aggregation] '${name}': persisted state is at generation ${stamped ?? 'unstamped'} ` +
         `but the store's committed generation is ${committed} — rescanning instead of adopting`
     )
-    return false
+    return 'rescan'
   }
 
   private async loadPersisted(): Promise<void> {
@@ -476,20 +507,21 @@ export class AggregationIndex {
           const appHash = this.definitionHashes.get(def.name) || ''
           if (appHash === savedHash && this.pendingAdopt.has(def.name)) {
             const stateData = await this.storage.getMetadata(`${STATE_KEY_PREFIX}${def.name}__`)
-            if (
-              stateData &&
-              stateData.groups &&
-              this.stateGenerationAdoptable(def.name, stateData)
-            ) {
+            const verdict =
+              stateData && stateData.groups
+                ? this.stateAdoptionVerdict(def.name, stateData)
+                : 'rescan'
+            if (verdict !== 'rescan') {
               const groupMap = new Map<string, AggregateGroupState>()
-              for (const group of stateData.groups as AggregateGroupState[]) {
+              for (const group of stateData!.groups as AggregateGroupState[]) {
                 groupMap.set(serializeGroupKey(group.groupKey), group)
               }
               this.states.set(def.name, groupMap)
               this.pendingAdopt.delete(def.name)
               this.needsBackfill.delete(def.name)
               prodLog.info(
-                `[Aggregation] '${def.name}': adopted persisted state (${groupMap.size} groups) — no rescan`
+                `[Aggregation] '${def.name}': adopted persisted state (${groupMap.size} groups) — ` +
+                  (verdict === 'catchup' ? 'incremental catch-up pending' : 'no rescan')
               )
             }
             // No/invalid persisted state: stays in pendingAdopt and resolves
@@ -504,22 +536,23 @@ export class AggregationIndex {
         const currentHash = hashDefinition(def)
 
         const stateData = await this.storage.getMetadata(`${STATE_KEY_PREFIX}${def.name}__`)
-        if (
-          stateData &&
-          stateData.groups &&
-          savedHash === currentHash &&
-          this.stateGenerationAdoptable(def.name, stateData)
-        ) {
-          // Definition unchanged — load state
+        const restoreVerdict =
+          stateData && stateData.groups && savedHash === currentHash
+            ? this.stateAdoptionVerdict(def.name, stateData)
+            : 'rescan'
+        if (restoreVerdict !== 'rescan') {
+          // Definition unchanged — load state (exact as of its stamp; a
+          // 'catchup' verdict reconciles the missing window incrementally).
           const groupMap = new Map<string, AggregateGroupState>()
-          for (const group of stateData.groups as AggregateGroupState[]) {
+          for (const group of stateData!.groups as AggregateGroupState[]) {
             const serialized = serializeGroupKey(group.groupKey)
             groupMap.set(serialized, group)
           }
           this.states.set(def.name, groupMap)
           this.needsBackfill.delete(def.name)
           prodLog.info(
-            `[Aggregation] '${def.name}': restored definition + adopted persisted state (${groupMap.size} groups)`
+            `[Aggregation] '${def.name}': restored definition + adopted persisted state (${groupMap.size} groups)` +
+              (restoreVerdict === 'catchup' ? ' — incremental catch-up pending' : '')
           )
         } else {
           // Definition changed or no saved state — start fresh and backfill from
@@ -747,6 +780,119 @@ export class AggregationIndex {
     this.dirty.add(name)
   }
 
+  // ============= Incremental Catch-Up (behind-stamp adoption) =============
+
+  /** The aggregates adopted behind the watermark, with their exact missing windows. */
+  getPendingCatchUps(): Array<{ name: string; from: number; to: number }> {
+    return Array.from(this.pendingCatchUp, ([name, w]) => ({ name, ...w }))
+  }
+
+  /**
+   * Reconcile ONE entity's contribution across a catch-up window using the
+   * same exact delta algebra the write-time hooks use: remove the
+   * contribution the adopted state counted (the entity AS OF the stamp),
+   * add the contribution it should count (AS OF the window's end). `null`
+   * on either side means the entity did not exist then. Composes exactly
+   * with live hooks because every application is a precise old/new pair —
+   * order between catch-up and post-window writes cannot drift the totals.
+   */
+  reconcileEntity(
+    name: string,
+    id: string,
+    before: Record<string, unknown> | null,
+    after: Record<string, unknown> | null
+  ): void {
+    const def = this.definitions.get(name)
+    if (!def) return
+    if (before && after) {
+      if (isAggregateEntity(after)) return
+      const oldMatches = matchesSource(before, def.source)
+      const newMatches = matchesSource(after, def.source)
+      if (this.nativeProvider && (oldMatches || newMatches)) {
+        this.applyNativeResults(
+          name,
+          this.nativeProvider.incrementalUpdate(name, def, after, 'update', before)
+        )
+        return
+      }
+      if (oldMatches) this.removeContribution(name, def, before)
+      if (newMatches) this.addContribution(name, def, after)
+      return
+    }
+    if (after) {
+      if (isAggregateEntity(after) || !matchesSource(after, def.source)) return
+      if (this.nativeProvider) {
+        this.applyNativeResults(name, this.nativeProvider.incrementalUpdate(name, def, after, 'add'))
+      } else {
+        this.addContribution(name, def, after)
+      }
+      return
+    }
+    if (before) {
+      if (isAggregateEntity(before) || !matchesSource(before, def.source)) return
+      if (this.nativeProvider) {
+        this.applyNativeResults(name, this.nativeProvider.incrementalUpdate(name, def, before, 'delete'))
+      } else {
+        this.removeContribution(name, def, before)
+      }
+    }
+  }
+
+  /** Whether the native provider offers the parallel whole-rebuild path. */
+  hasProviderRebuild(): boolean {
+    return typeof this.nativeProvider?.rebuildAggregate === 'function'
+  }
+
+  /** The catch-up window for `name` is fully reconciled; state is current. */
+  finishCatchUp(name: string): void {
+    this.pendingCatchUp.delete(name)
+    this.dirty.add(name)
+  }
+
+  /**
+   * A catch-up could not complete (window unreadable, affected set over the
+   * bound, …): demote to an exact rescan, loudly — never serve un-reconciled.
+   */
+  demoteCatchUpToBackfill(name: string, reason: string): void {
+    this.pendingCatchUp.delete(name)
+    this.needsBackfill.add(name)
+    prodLog.warn(`[Aggregation] '${name}': catch-up demoted to full rescan — ${reason}`)
+  }
+
+  /**
+   * Rebuild an aggregate through the native provider's parallel path
+   * (SELF-ENGINE-LIFECYCLE-SPRINT ask (c) — `rebuildAggregate` existed on
+   * the provider contract but was never invoked; the JS walk fed
+   * per-entity FFI calls instead). Returns false when no provider rebuild
+   * exists — the caller streams the JS walk as before.
+   */
+  rebuildWithProvider(name: string, entities: Array<Record<string, unknown>>): boolean {
+    const def = this.definitions.get(name)
+    if (!def || !this.nativeProvider?.rebuildAggregate) return false
+    const rebuilt = this.nativeProvider.rebuildAggregate(
+      def,
+      entities.filter(e => !isAggregateEntity(e) && matchesSource(e, def.source))
+    )
+    this.states.set(name, rebuilt)
+    this.backfillStaging.delete(name)
+    this.needsBackfill.delete(name)
+    this.dirty.add(name)
+    return true
+  }
+
+  /**
+   * A write-path hook could not see the entity it needed (e.g. a delete
+   * whose before-image was unavailable): flag EVERY defined aggregate for
+   * an exact rescan, loudly — the counts must never silently drift
+   * (SELF-ENGINE-LIFECYCLE-SPRINT ask (d): the gated hook used to SKIP).
+   */
+  flagAllForRescan(reason: string): void {
+    for (const name of this.definitions.keys()) this.needsBackfill.add(name)
+    prodLog.warn(
+      `[Aggregation] all ${this.definitions.size} aggregate(s) flagged for rescan — ${reason}`
+    )
+  }
+
   // ============= Write-Time Hooks =============
 
   /**