fix(aggregation): the lifecycle cluster — flush stamps, behind-stamp catches up incrementally, the native rebuild finally gets invoked, deletes are never silently skipped

SELF-ENGINE-LIFECYCLE-SPRINT + BRAINY-PROD-LATENCY-TRIAD, the four asks:

(a) brain.flush() persists aggregation state stamped at the committed
    generation. The stamp used to advance only at close(), so a long-lived
    writer that flushes but never closes — the primary production shape —
    left every write window behind the stamp, and ANY unclean exit forced a
    whole-store backfill walk (per-entity work, measured >60s and
    door-starving on a 9k-row production brain) on the first stats call.

(b) BEHIND-stamp adoption becomes adopt + INCREMENTAL CATCH-UP: the exact
    missing window (stamp, committed] resolves its affected-id set from the
    fact log and reconciles each entity with time-travel before/after reads
    (asOf at both window bounds) through the same delta algebra the live
    hooks use — cost bounded by writes since the last flush, never store
    size, and exact under interleaving because reconciliation targets the
    FIXED window end while later writes chain through hooks. Oversized
    windows (>5000 affected) and unreadable windows demote to the announced
    rescan — never a silent partial serve.

(c) The native provider's parallel rebuildAggregate — on the contract since
    8.x but never invoked anywhere — is now the backfill walk's preferred
    door: one call per aggregate with source-matched entities, replacing
    the per-entity FFI stream.

(d) A delete whose before-image is unavailable can no longer SKIP the
    aggregation hook silently (counts drifted upward forever): both delete
    paths (remove() and transact) flag an exact rescan, loudly.

Pins: integration (flush stamp; unclean-exit reopen → exact counts through
an add + group-move + delete window with the walk spy proving ZERO
whole-store walks) + unit (provider rebuild invoked once with filtered
entities; flagAllForRescan; reconcile delta algebra). Gates: unit 1913/1913
· integration 760 · conformance 27/27.

tests/unit/aggregation/aggregation-provider-rebuild.test.ts

@@ -0,0 +1,134 @@
+/**
+ * @module tests/unit/aggregation/aggregation-provider-rebuild
+ * @description Pins for SELF-ENGINE-LIFECYCLE-SPRINT asks (c) + (d):
+ * (c) the native provider's parallel `rebuildAggregate` — on the provider
+ *     contract since 8.x but NEVER invoked (the JS walk streamed per-entity
+ *     FFI calls instead) — is now the backfill walk's preferred door;
+ * (d) a write-path hook that cannot see its entity (before-image-less
+ *     delete) flags an exact rescan LOUDLY instead of silently skipping the
+ *     decrement (the skip let counts drift upward forever).
+ */
+import { describe, it, expect, vi } from 'vitest'
+import { AggregationIndex } from '../../../src/aggregation/AggregationIndex.js'
+import { NounType } from '../../../src/types/graphTypes.js'
+import type { AggregationProvider, AggregateGroupState } from '../../../src/types/brainy.types.js'
+
+const DEF = {
+  name: 'by_subtype',
+  source: { type: NounType.Document },
+  groupBy: ['system.subtype'] as string[],
+  metrics: { count: { op: 'count' as const } }
+}
+
+/** Minimal in-memory storage double for the index's persistence surface. */
+function memStorage() {
+  const store = new Map<string, unknown>()
+  return {
+    saveMetadata: async (k: string, v: unknown) => void store.set(k, v),
+    getMetadata: async (k: string) => store.get(k) ?? null
+  } as never
+}
+
+function providerDouble(): AggregationProvider & { rebuildAggregate: ReturnType<typeof vi.fn> } {
+  return {
+    defineAggregate: vi.fn(),
+    removeAggregate: vi.fn(),
+    incrementalUpdate: vi.fn(() => []),
+    computeGroupKey: vi.fn(() => ({})),
+    rebuildAggregate: vi.fn((): Map<string, AggregateGroupState> => {
+      return new Map([
+        [
+          'system.subtype=invoice',
+          {
+            groupKey: { 'system.subtype': 'invoice' },
+            metrics: { count: { sum: 0, count: 2, min: Infinity, max: -Infinity, m2: 0 } }
+          } as AggregateGroupState
+        ]
+      ])
+    }),
+    queryAggregate: vi.fn(() => [])
+  } as never
+}
+
+describe('ask (c) — the native parallel rebuild is invoked, never dead code', () => {
+  it('rebuildWithProvider hands SOURCE-MATCHED entities to the provider once and swaps state in', () => {
+    const provider = providerDouble()
+    const index = new AggregationIndex(memStorage(), provider)
+    index.defineAggregate(DEF)
+
+    expect(index.hasProviderRebuild()).toBe(true)
+
+    const entities = [
+      { type: NounType.Document, subtype: 'invoice', metadata: {} },
+      { type: NounType.Document, subtype: 'invoice', metadata: {} },
+      // Source-filter mismatch: a different noun type must be filtered OUT
+      // before the provider sees the batch.
+      { type: NounType.Person, subtype: 'invoice', metadata: {} }
+    ]
+    const handled = index.rebuildWithProvider(DEF.name, entities)
+
+    expect(handled).toBe(true)
+    expect(provider.rebuildAggregate).toHaveBeenCalledTimes(1)
+    const [defArg, entArg] = provider.rebuildAggregate.mock.calls[0]
+    expect(defArg.name).toBe(DEF.name)
+    expect(entArg).toHaveLength(2)
+
+    // The rebuilt state serves — and the aggregate is no longer pending.
+    expect(index.getPendingBackfills()).not.toContain(DEF.name)
+  })
+
+  it('returns false without a provider rebuild — the caller streams the JS walk', () => {
+    const index = new AggregationIndex(memStorage())
+    index.defineAggregate(DEF)
+    expect(index.hasProviderRebuild()).toBe(false)
+    expect(index.rebuildWithProvider(DEF.name, [])).toBe(false)
+  })
+})
+
+describe('ask (d) — the before-image-less delete is LOUD, never a silent skip', () => {
+  it('flagAllForRescan puts every defined aggregate back on the backfill list', () => {
+    const index = new AggregationIndex(memStorage())
+    index.defineAggregate(DEF)
+    index.defineAggregate({ ...DEF, name: 'second' })
+    // Simulate settled state: nothing pending.
+    for (const n of index.getPendingBackfills()) {
+      index.beginBackfill(n)
+      index.finishBackfill(n)
+    }
+    expect(index.getPendingBackfills()).toEqual([])
+
+    index.flagAllForRescan('delete of X carried no before-image metadata')
+
+    expect(index.getPendingBackfills().sort()).toEqual(['by_subtype', 'second'])
+  })
+})
+
+describe('reconcileEntity — the exact delta algebra at the catch-up boundary', () => {
+  it('before-only removes, after-only adds, both reconciles a group move', () => {
+    const index = new AggregationIndex(memStorage())
+    index.defineAggregate(DEF)
+    for (const n of index.getPendingBackfills()) {
+      index.beginBackfill(n)
+      index.finishBackfill(n)
+    }
+    const doc = (subtype: string) => ({ type: NounType.Document, subtype, metadata: {} })
+
+    // Pre-window state, applied through the LIVE hooks (as adoption would
+    // have counted it): c and seed exist as drafts, x1 as an invoice.
+    index.onEntityAdded('c', doc('draft'))
+    index.onEntityAdded('seed', doc('draft'))
+    index.onEntityAdded('x1', doc('invoice'))
+
+    // The window's reconciliation: two adds, one group move, one delete.
+    index.reconcileEntity(DEF.name, 'a', null, doc('invoice'))
+    index.reconcileEntity(DEF.name, 'b', null, doc('invoice'))
+    index.reconcileEntity(DEF.name, 'c', doc('draft'), doc('invoice'))
+    index.reconcileEntity(DEF.name, 'seed', doc('draft'), null)
+
+    const rows = index.queryAggregate({ name: DEF.name })
+    const count = (st: string) =>
+      Number(rows.find(r => r.groupKey['system.subtype'] === st)?.metrics.count ?? 0)
+    expect(count('invoice')).toBe(4) // x1 + a + b + moved c
+    expect(count('draft')).toBe(0) // c moved out, seed deleted
+  })
+})

tests/unit/utils/metadataIndex-nested-orderby.test.ts

@@ -0,0 +1,142 @@
+/**
+ * @module tests/unit/utils/metadataIndex-nested-orderby
+ * @description THE NESTED-FIELD ADDRESSING PIN for ordered reads (the
+ * field-addressing law, dotted-path clause). The defect this keeps dead:
+ * `orderBy` on a nested user metadata field (dotted path, e.g.
+ * `orderBy: 'profile.score'` over `metadata: { profile: { score: 7 } }`)
+ * silently returned insertion order — a no-op sort — because the sort
+ * path's value resolution read flat bag keys only. The law: a dotted user
+ * address is either SERVED CORRECTLY (the batched resolver walks inside
+ * the bag) or REFUSED with a typed UnresolvableFieldError — never a silent
+ * pass-through. Both spellings (`profile.score` / `metadata.profile.score`)
+ * are the same address; the filter side (`where: { 'profile.score': … }`)
+ * obeys the same law.
+ */
+import { describe, it, expect, beforeAll, afterAll } from 'vitest'
+import { Brainy, UnresolvableFieldError } from '../../../src/index.js'
+import { NounType } from '../../../src/types/graphTypes.js'
+
+const ROWS = 30
+
+describe('nested (dotted-path) user field orderBy — the field-addressing law', () => {
+  let brain: Brainy
+  /** id → nested score, for the rows that carry profile.score */
+  const scoreById = new Map<string, number>()
+  /** ids of the two rows WITHOUT a profile bag */
+  let noProfileIds: string[] = []
+
+  beforeAll(async () => {
+    brain = new Brainy({ storage: { type: 'memory' }, requireSubtype: false })
+    await brain.init()
+    for (let i = 0; i < ROWS; i++) {
+      // (i * 11) % 30 is a permutation of 0..29 (gcd(11,30)=1): every score
+      // distinct, insertion order maximally different from value order — a
+      // silent insertion-order pass-through cannot accidentally look sorted.
+      const score = (i * 11) % ROWS
+      const id = await brain.add({
+        data: `row ${i}`,
+        type: NounType.Document,
+        metadata: { profile: { score }, plain: i }
+      })
+      scoreById.set(id, score)
+    }
+    const a = await brain.add({
+      data: 'no-profile a',
+      type: NounType.Document,
+      metadata: { plain: 1000 }
+    })
+    const b = await brain.add({
+      data: 'no-profile b',
+      type: NounType.Document,
+      metadata: { plain: 1001 }
+    })
+    noProfileIds = [a, b].sort()
+  }, 120000)
+
+  afterAll(async () => {
+    await brain.close().catch(() => {})
+  })
+
+  /** Assert one complete ordered read against the sealed ordering contract. */
+  function assertOrdered(
+    rows: Array<{ id: string }>,
+    order: 'asc' | 'desc',
+    label: string
+  ): void {
+    // Rows are NEVER dropped: all 30 scored + 2 profile-less rows come back.
+    expect(rows.length, `${label}: complete result`).toBe(ROWS + 2)
+
+    // Missing-value rows sort LAST in BOTH directions, ties by id ascending.
+    const lastTwo = rows.slice(-2).map((r) => r.id)
+    expect(lastTwo, `${label}: missing-value rows LAST, id asc`).toEqual(noProfileIds)
+
+    // The scored 30 are ordered by the NESTED value — the exact permutation,
+    // not insertion order.
+    const observed = rows.slice(0, ROWS).map((r) => scoreById.get(r.id))
+    const wanted = [...scoreById.values()].sort((x, y) =>
+      order === 'asc' ? x - y : y - x
+    )
+    expect(observed, `${label}: nested values in ${order} order`).toEqual(wanted)
+  }
+
+  it('orderBy: "profile.score" desc — served correctly, missing rows LAST (never a silent insertion-order no-op)', async () => {
+    const rows = await brain.find({
+      type: NounType.Document,
+      orderBy: 'profile.score',
+      order: 'desc',
+      limit: 40
+    })
+    assertOrdered(rows, 'desc', 'bare dotted, desc')
+  })
+
+  it('orderBy: "profile.score" asc — same law in the other direction', async () => {
+    const rows = await brain.find({
+      type: NounType.Document,
+      orderBy: 'profile.score',
+      order: 'asc',
+      limit: 40
+    })
+    assertOrdered(rows, 'asc', 'bare dotted, asc')
+  })
+
+  it('explicit spelling "metadata.profile.score" is the SAME address — identical result', async () => {
+    const bare = await brain.find({
+      type: NounType.Document,
+      orderBy: 'profile.score',
+      order: 'desc',
+      limit: 40
+    })
+    const explicit = await brain.find({
+      type: NounType.Document,
+      orderBy: 'metadata.profile.score',
+      order: 'desc',
+      limit: 40
+    })
+    assertOrdered(explicit, 'desc', 'metadata.-prefixed, desc')
+    expect(
+      explicit.map((r) => r.id),
+      'both spellings resolve to the identical ordered id sequence'
+    ).toEqual(bare.map((r) => r.id))
+  })
+
+  it('a dotted path carried by NO entity REFUSES with UnresolvableFieldError — never a silent insertion-order return', async () => {
+    await expect(
+      brain.find({
+        type: NounType.Document,
+        orderBy: 'no.such.path',
+        order: 'desc',
+        limit: 40
+      })
+    ).rejects.toThrow(UnresolvableFieldError)
+  })
+
+  it('dotted where: { "profile.score": 7 } finds exactly the right row — the filter side of the same law', async () => {
+    const wantedId = [...scoreById.entries()].find(([, s]) => s === 7)![0]
+    const rows = await brain.find({
+      type: NounType.Document,
+      where: { 'profile.score': 7 },
+      limit: 40
+    })
+    expect(rows.map((r) => r.id)).toEqual([wantedId])
+  })
+})