fix(aggregation): the lifecycle cluster — flush stamps, behind-stamp catches up incrementally, the native rebuild finally gets invoked, deletes are never silently skipped

SELF-ENGINE-LIFECYCLE-SPRINT + BRAINY-PROD-LATENCY-TRIAD, the four asks:

(a) brain.flush() persists aggregation state stamped at the committed
    generation. The stamp used to advance only at close(), so a long-lived
    writer that flushes but never closes — the primary production shape —
    left every write window behind the stamp, and ANY unclean exit forced a
    whole-store backfill walk (per-entity work, measured >60s and
    door-starving on a 9k-row production brain) on the first stats call.

(b) BEHIND-stamp adoption becomes adopt + INCREMENTAL CATCH-UP: the exact
    missing window (stamp, committed] resolves its affected-id set from the
    fact log and reconciles each entity with time-travel before/after reads
    (asOf at both window bounds) through the same delta algebra the live
    hooks use — cost bounded by writes since the last flush, never store
    size, and exact under interleaving because reconciliation targets the
    FIXED window end while later writes chain through hooks. Oversized
    windows (>5000 affected) and unreadable windows demote to the announced
    rescan — never a silent partial serve.

(c) The native provider's parallel rebuildAggregate — on the contract since
    8.x but never invoked anywhere — is now the backfill walk's preferred
    door: one call per aggregate with source-matched entities, replacing
    the per-entity FFI stream.

(d) A delete whose before-image is unavailable can no longer SKIP the
    aggregation hook silently (counts drifted upward forever): both delete
    paths (remove() and transact) flag an exact rescan, loudly.

Pins: integration (flush stamp; unclean-exit reopen → exact counts through
an add + group-move + delete window with the walk spy proving ZERO
whole-store walks) + unit (provider rebuild invoked once with filtered
entities; flagAllForRescan; reconcile delta algebra). Gates: unit 1913/1913
· integration 760 · conformance 27/27.

tests/integration/aggregation-lifecycle-catchup.test.ts

@@ -0,0 +1,143 @@
+/**
+ * @module tests/integration/aggregation-lifecycle-catchup
+ * @description THE AGGREGATION LIFECYCLE PINS (SELF-ENGINE-LIFECYCLE-SPRINT /
+ * BRAINY-PROD-LATENCY-TRIAD asks (a)+(b)). The production disease: the
+ * aggregation stamp persisted ONLY at close(), so a long-lived writer that
+ * flushes but never closes left its stamp behind after every write window —
+ * and the exact-match adoption rule then forced a WHOLE-STORE backfill walk
+ * (per-entity work, measured >60s and door-starving on a 9k-row production
+ * brain) on the first stats call after any unclean exit.
+ *
+ * The cures pinned here:
+ * (a) `brain.flush()` persists aggregation state, stamped at the committed
+ *     generation — the stamp tracks every flush, not just close().
+ * (b) BEHIND-stamp state is ADOPTED and reconciled INCREMENTALLY over its
+ *     exact missing window (fact-log affected ids + time-travel before/after
+ *     reads) — the full walk never runs for an unclean exit. Pinned by call
+ *     shape (the walk spy), not by latency.
+ */
+import { describe, it, expect, afterEach, vi } from 'vitest'
+import { mkdtempSync, rmSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { Brainy } from '../../src/index.js'
+import { NounType } from '../../src/types/graphTypes.js'
+
+const AGG = {
+  name: 'by_subtype',
+  source: { type: NounType.Document },
+  groupBy: ['system.subtype'] as string[],
+  metrics: { count: { op: 'count' as const } }
+}
+
+const dirs: string[] = []
+const brains: Brainy[] = []
+
+async function open(dir: string): Promise<Brainy> {
+  const b = new Brainy({ storage: { type: 'filesystem', path: dir }, requireSubtype: false })
+  await b.init()
+  brains.push(b)
+  return b
+}
+
+function countFor(results: Array<{ groupKey: Record<string, unknown>; metrics: Record<string, unknown> }>, subtype: string): number {
+  const row = results.find(r => r.groupKey['system.subtype'] === subtype)
+  return row ? Number(row.metrics.count) : 0
+}
+
+afterEach(async () => {
+  for (const b of brains.splice(0)) await b.close().catch(() => {})
+  for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true })
+})
+
+describe('aggregation lifecycle — flush stamps, behind-stamp catches up incrementally', () => {
+  it('(a) brain.flush() persists aggregation state stamped at the committed generation', async () => {
+    const dir = mkdtempSync(join(tmpdir(), 'brainy-agg-flush-'))
+    dirs.push(dir)
+    const brain = await open(dir)
+    brain.defineAggregate(AGG)
+    await brain.add({ data: 'a', type: NounType.Document, subtype: 'invoice', metadata: {} })
+    await brain.add({ data: 'b', type: NounType.Document, subtype: 'invoice', metadata: {} })
+    await brain.queryAggregate(AGG.name) // settle backfill-on-define
+
+    await brain.flush()
+
+    const internals = brain as unknown as {
+      storage: {
+        getMetadata(k: string): Promise<{ sourceGeneration?: number } | null>
+        committedGeneration?(): number
+      }
+    }
+    const persisted = await internals.storage.getMetadata('__aggregation_state_by_subtype__')
+    expect(persisted, 'state persisted by flush(), not only close()').toBeTruthy()
+    expect(
+      persisted!.sourceGeneration,
+      'stamp equals the committed generation at flush time'
+    ).toBe(internals.storage.committedGeneration?.())
+  })
+
+  it('(b) an unclean exit reconciles incrementally — exact counts, ZERO full-store walks', async () => {
+    const dir = mkdtempSync(join(tmpdir(), 'brainy-agg-catchup-'))
+    dirs.push(dir)
+
+    // Session 1: define + write + flush (stamps at G), then MORE writes of
+    // every kind (add / update-that-moves-groups / delete) and a clean close
+    // — but we then REWIND the persisted aggregation artifact to its at-G
+    // bytes, which is byte-for-byte the unclean-exit state: stamp G, store
+    // committed at G+k.
+    let brain = await open(dir)
+    brain.defineAggregate(AGG)
+    await brain.add({ data: 'a', type: NounType.Document, subtype: 'invoice', metadata: {} })
+    await brain.add({ data: 'b', type: NounType.Document, subtype: 'invoice', metadata: {} })
+    const moving = await brain.add({ data: 'c', type: NounType.Document, subtype: 'draft', metadata: {} })
+    const doomed = await brain.add({ data: 'd', type: NounType.Document, subtype: 'draft', metadata: {} })
+    await brain.queryAggregate(AGG.name)
+    await brain.flush()
+
+    const internals = brain as unknown as {
+      storage: {
+        getMetadata(k: string): Promise<Record<string, unknown> | null>
+        saveMetadata(k: string, v: Record<string, unknown>): Promise<void>
+      }
+    }
+    const stateAtG = JSON.parse(
+      JSON.stringify(await internals.storage.getMetadata('__aggregation_state_by_subtype__'))
+    )
+
+    // The missing window: one add, one group-moving update, one delete.
+    await brain.add({ data: 'e', type: NounType.Document, subtype: 'invoice', metadata: {} })
+    await brain.update({ id: moving, subtype: 'invoice' })
+    await brain.remove(doomed)
+    await brain.close()
+    brains.pop()
+
+    // Rewind the aggregation artifact to the at-G bytes (the unclean exit).
+    {
+      const reopenForRewind = await open(dir)
+      const rw = reopenForRewind as unknown as typeof internals
+      await rw.storage.saveMetadata('__aggregation_state_by_subtype__', stateAtG)
+      await reopenForRewind.close()
+      brains.pop()
+    }
+
+    // Session 2: reopen — adoption must see BEHIND and reconcile, never walk.
+    brain = await open(dir)
+    brain.defineAggregate(AGG)
+    const walkSpy = vi.spyOn(
+      brain as unknown as { runAggregationBackfillWalk(): Promise<void> },
+      'runAggregationBackfillWalk'
+    )
+
+    const results = await brain.queryAggregate(AGG.name)
+
+    // Ground truth after the window: invoice = a,b,e + moved c = 4; draft = 0
+    // (c moved out, d deleted).
+    expect(countFor(results as never, 'invoice'), 'invoice count exact after catch-up').toBe(4)
+    expect(countFor(results as never, 'draft'), 'draft count exact after catch-up').toBe(0)
+
+    // THE CALL-SHAPE PIN: the whole-store walk never ran.
+    expect(walkSpy, 'full backfill walk must not run for a behind-stamp reopen').not.toHaveBeenCalled()
+
+    vi.restoreAllMocks()
+  }, 120000)
+})

tests/unit/aggregation/aggregation-provider-rebuild.test.ts

@@ -0,0 +1,134 @@
+/**
+ * @module tests/unit/aggregation/aggregation-provider-rebuild
+ * @description Pins for SELF-ENGINE-LIFECYCLE-SPRINT asks (c) + (d):
+ * (c) the native provider's parallel `rebuildAggregate` — on the provider
+ *     contract since 8.x but NEVER invoked (the JS walk streamed per-entity
+ *     FFI calls instead) — is now the backfill walk's preferred door;
+ * (d) a write-path hook that cannot see its entity (before-image-less
+ *     delete) flags an exact rescan LOUDLY instead of silently skipping the
+ *     decrement (the skip let counts drift upward forever).
+ */
+import { describe, it, expect, vi } from 'vitest'
+import { AggregationIndex } from '../../../src/aggregation/AggregationIndex.js'
+import { NounType } from '../../../src/types/graphTypes.js'
+import type { AggregationProvider, AggregateGroupState } from '../../../src/types/brainy.types.js'
+
+const DEF = {
+  name: 'by_subtype',
+  source: { type: NounType.Document },
+  groupBy: ['system.subtype'] as string[],
+  metrics: { count: { op: 'count' as const } }
+}
+
+/** Minimal in-memory storage double for the index's persistence surface. */
+function memStorage() {
+  const store = new Map<string, unknown>()
+  return {
+    saveMetadata: async (k: string, v: unknown) => void store.set(k, v),
+    getMetadata: async (k: string) => store.get(k) ?? null
+  } as never
+}
+
+function providerDouble(): AggregationProvider & { rebuildAggregate: ReturnType<typeof vi.fn> } {
+  return {
+    defineAggregate: vi.fn(),
+    removeAggregate: vi.fn(),
+    incrementalUpdate: vi.fn(() => []),
+    computeGroupKey: vi.fn(() => ({})),
+    rebuildAggregate: vi.fn((): Map<string, AggregateGroupState> => {
+      return new Map([
+        [
+          'system.subtype=invoice',
+          {
+            groupKey: { 'system.subtype': 'invoice' },
+            metrics: { count: { sum: 0, count: 2, min: Infinity, max: -Infinity, m2: 0 } }
+          } as AggregateGroupState
+        ]
+      ])
+    }),
+    queryAggregate: vi.fn(() => [])
+  } as never
+}
+
+describe('ask (c) — the native parallel rebuild is invoked, never dead code', () => {
+  it('rebuildWithProvider hands SOURCE-MATCHED entities to the provider once and swaps state in', () => {
+    const provider = providerDouble()
+    const index = new AggregationIndex(memStorage(), provider)
+    index.defineAggregate(DEF)
+
+    expect(index.hasProviderRebuild()).toBe(true)
+
+    const entities = [
+      { type: NounType.Document, subtype: 'invoice', metadata: {} },
+      { type: NounType.Document, subtype: 'invoice', metadata: {} },
+      // Source-filter mismatch: a different noun type must be filtered OUT
+      // before the provider sees the batch.
+      { type: NounType.Person, subtype: 'invoice', metadata: {} }
+    ]
+    const handled = index.rebuildWithProvider(DEF.name, entities)
+
+    expect(handled).toBe(true)
+    expect(provider.rebuildAggregate).toHaveBeenCalledTimes(1)
+    const [defArg, entArg] = provider.rebuildAggregate.mock.calls[0]
+    expect(defArg.name).toBe(DEF.name)
+    expect(entArg).toHaveLength(2)
+
+    // The rebuilt state serves — and the aggregate is no longer pending.
+    expect(index.getPendingBackfills()).not.toContain(DEF.name)
+  })
+
+  it('returns false without a provider rebuild — the caller streams the JS walk', () => {
+    const index = new AggregationIndex(memStorage())
+    index.defineAggregate(DEF)
+    expect(index.hasProviderRebuild()).toBe(false)
+    expect(index.rebuildWithProvider(DEF.name, [])).toBe(false)
+  })
+})
+
+describe('ask (d) — the before-image-less delete is LOUD, never a silent skip', () => {
+  it('flagAllForRescan puts every defined aggregate back on the backfill list', () => {
+    const index = new AggregationIndex(memStorage())
+    index.defineAggregate(DEF)
+    index.defineAggregate({ ...DEF, name: 'second' })
+    // Simulate settled state: nothing pending.
+    for (const n of index.getPendingBackfills()) {
+      index.beginBackfill(n)
+      index.finishBackfill(n)
+    }
+    expect(index.getPendingBackfills()).toEqual([])
+
+    index.flagAllForRescan('delete of X carried no before-image metadata')
+
+    expect(index.getPendingBackfills().sort()).toEqual(['by_subtype', 'second'])
+  })
+})
+
+describe('reconcileEntity — the exact delta algebra at the catch-up boundary', () => {
+  it('before-only removes, after-only adds, both reconciles a group move', () => {
+    const index = new AggregationIndex(memStorage())
+    index.defineAggregate(DEF)
+    for (const n of index.getPendingBackfills()) {
+      index.beginBackfill(n)
+      index.finishBackfill(n)
+    }
+    const doc = (subtype: string) => ({ type: NounType.Document, subtype, metadata: {} })
+
+    // Pre-window state, applied through the LIVE hooks (as adoption would
+    // have counted it): c and seed exist as drafts, x1 as an invoice.
+    index.onEntityAdded('c', doc('draft'))
+    index.onEntityAdded('seed', doc('draft'))
+    index.onEntityAdded('x1', doc('invoice'))
+
+    // The window's reconciliation: two adds, one group move, one delete.
+    index.reconcileEntity(DEF.name, 'a', null, doc('invoice'))
+    index.reconcileEntity(DEF.name, 'b', null, doc('invoice'))
+    index.reconcileEntity(DEF.name, 'c', doc('draft'), doc('invoice'))
+    index.reconcileEntity(DEF.name, 'seed', doc('draft'), null)
+
+    const rows = index.queryAggregate({ name: DEF.name })
+    const count = (st: string) =>
+      Number(rows.find(r => r.groupKey['system.subtype'] === st)?.metrics.count ?? 0)
+    expect(count('invoice')).toBe(4) // x1 + a + b + moved c
+    expect(count('draft')).toBe(0) // c moved out, seed deleted
+  })
+})

tests/unit/utils/metadataIndex-nested-orderby.test.ts

@@ -0,0 +1,142 @@
+/**
+ * @module tests/unit/utils/metadataIndex-nested-orderby
+ * @description THE NESTED-FIELD ADDRESSING PIN for ordered reads (the
+ * field-addressing law, dotted-path clause). The defect this keeps dead:
+ * `orderBy` on a nested user metadata field (dotted path, e.g.
+ * `orderBy: 'profile.score'` over `metadata: { profile: { score: 7 } }`)
+ * silently returned insertion order — a no-op sort — because the sort
+ * path's value resolution read flat bag keys only. The law: a dotted user
+ * address is either SERVED CORRECTLY (the batched resolver walks inside
+ * the bag) or REFUSED with a typed UnresolvableFieldError — never a silent
+ * pass-through. Both spellings (`profile.score` / `metadata.profile.score`)
+ * are the same address; the filter side (`where: { 'profile.score': … }`)
+ * obeys the same law.
+ */
+import { describe, it, expect, beforeAll, afterAll } from 'vitest'
+import { Brainy, UnresolvableFieldError } from '../../../src/index.js'
+import { NounType } from '../../../src/types/graphTypes.js'
+
+const ROWS = 30
+
+describe('nested (dotted-path) user field orderBy — the field-addressing law', () => {
+  let brain: Brainy
+  /** id → nested score, for the rows that carry profile.score */
+  const scoreById = new Map<string, number>()
+  /** ids of the two rows WITHOUT a profile bag */
+  let noProfileIds: string[] = []
+
+  beforeAll(async () => {
+    brain = new Brainy({ storage: { type: 'memory' }, requireSubtype: false })
+    await brain.init()
+    for (let i = 0; i < ROWS; i++) {
+      // (i * 11) % 30 is a permutation of 0..29 (gcd(11,30)=1): every score
+      // distinct, insertion order maximally different from value order — a
+      // silent insertion-order pass-through cannot accidentally look sorted.
+      const score = (i * 11) % ROWS
+      const id = await brain.add({
+        data: `row ${i}`,
+        type: NounType.Document,
+        metadata: { profile: { score }, plain: i }
+      })
+      scoreById.set(id, score)
+    }
+    const a = await brain.add({
+      data: 'no-profile a',
+      type: NounType.Document,
+      metadata: { plain: 1000 }
+    })
+    const b = await brain.add({
+      data: 'no-profile b',
+      type: NounType.Document,
+      metadata: { plain: 1001 }
+    })
+    noProfileIds = [a, b].sort()
+  }, 120000)
+
+  afterAll(async () => {
+    await brain.close().catch(() => {})
+  })
+
+  /** Assert one complete ordered read against the sealed ordering contract. */
+  function assertOrdered(
+    rows: Array<{ id: string }>,
+    order: 'asc' | 'desc',
+    label: string
+  ): void {
+    // Rows are NEVER dropped: all 30 scored + 2 profile-less rows come back.
+    expect(rows.length, `${label}: complete result`).toBe(ROWS + 2)
+
+    // Missing-value rows sort LAST in BOTH directions, ties by id ascending.
+    const lastTwo = rows.slice(-2).map((r) => r.id)
+    expect(lastTwo, `${label}: missing-value rows LAST, id asc`).toEqual(noProfileIds)
+
+    // The scored 30 are ordered by the NESTED value — the exact permutation,
+    // not insertion order.
+    const observed = rows.slice(0, ROWS).map((r) => scoreById.get(r.id))
+    const wanted = [...scoreById.values()].sort((x, y) =>
+      order === 'asc' ? x - y : y - x
+    )
+    expect(observed, `${label}: nested values in ${order} order`).toEqual(wanted)
+  }
+
+  it('orderBy: "profile.score" desc — served correctly, missing rows LAST (never a silent insertion-order no-op)', async () => {
+    const rows = await brain.find({
+      type: NounType.Document,
+      orderBy: 'profile.score',
+      order: 'desc',
+      limit: 40
+    })
+    assertOrdered(rows, 'desc', 'bare dotted, desc')
+  })
+
+  it('orderBy: "profile.score" asc — same law in the other direction', async () => {
+    const rows = await brain.find({
+      type: NounType.Document,
+      orderBy: 'profile.score',
+      order: 'asc',
+      limit: 40
+    })
+    assertOrdered(rows, 'asc', 'bare dotted, asc')
+  })
+
+  it('explicit spelling "metadata.profile.score" is the SAME address — identical result', async () => {
+    const bare = await brain.find({
+      type: NounType.Document,
+      orderBy: 'profile.score',
+      order: 'desc',
+      limit: 40
+    })
+    const explicit = await brain.find({
+      type: NounType.Document,
+      orderBy: 'metadata.profile.score',
+      order: 'desc',
+      limit: 40
+    })
+    assertOrdered(explicit, 'desc', 'metadata.-prefixed, desc')
+    expect(
+      explicit.map((r) => r.id),
+      'both spellings resolve to the identical ordered id sequence'
+    ).toEqual(bare.map((r) => r.id))
+  })
+
+  it('a dotted path carried by NO entity REFUSES with UnresolvableFieldError — never a silent insertion-order return', async () => {
+    await expect(
+      brain.find({
+        type: NounType.Document,
+        orderBy: 'no.such.path',
+        order: 'desc',
+        limit: 40
+      })
+    ).rejects.toThrow(UnresolvableFieldError)
+  })
+
+  it('dotted where: { "profile.score": 7 } finds exactly the right row — the filter side of the same law', async () => {
+    const wantedId = [...scoreById.entries()].find(([, s]) => s === 7)![0]
+    const rows = await brain.find({
+      type: NounType.Document,
+      where: { 'profile.score': 7 },
+      limit: 40
+    })
+    expect(rows.map((r) => r.id)).toEqual([wantedId])
+  })
+})