soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: exception-safe aggregation backfill + generation-verified adoption + loud open-path guards

Browse source 
- Aggregation backfill walks build into a STAGING map and swap in atomically
  on completion. A mid-walk failure drops the staging map — the previous live
  state keeps serving, the aggregate stays flagged pending, and the storage
  error surfaces to the failing query. Previously the walk wiped live state
  before a scan that could throw, never cleared the pending flag on failure,
  and re-ran a full walk on every subsequent query: a silent wipe/walk/throw
  loop at the caller's retry rate.
- Failed walks are latched: retries within a 30s cooldown rethrow the recorded
  error instantly instead of re-walking, so a tight caller-side retry loop
  costs one loud error per query, never a full store walk per query.
- Persisted aggregation state is stamped with the store's committed generation
  at flush; reopen adoption requires stamp equality. Stale state (unclean
  shutdown) or over-counting state (a log truncation on a copied store pulled
  the watermark back) triggers exactly one loud rescan, never a silent adopt.
- The backfill/adoption path narrates: adoption decisions, walk start/finish
  with counts and duration, and failures all log by default.
- getNouns/getVerbs refuse a supplied-but-undecodable pagination cursor with a
  loud error instead of silently restarting the walk at offset 0 (which
  re-served page 1 forever to any while(hasMore) caller).
- The graph cold-load verb walk aborts loudly on a missing or non-advancing
  cursor with hasMore=true.
- A versioned index provider whose generation is AHEAD of the committed
  watermark (torn copy / crash-recovery truncation) is now named loudly at
  open, alongside the existing behind-direction message.
This commit is contained in:
David Snelling 2026-07-17 16:00:11 -07:00
parent 01a7f3dd01
commit a77b064bd7
7 changed files with 335 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

101
src/aggregation/AggregationIndex.ts
View file

@ -29,6 +29,7 @@ import { matchesMetadataFilter } from '../utils/metadataFilter.js'
import { compareCodePoints } from '../utils/collation.js'
import { bucketTimestamp } from './timeWindows.js'
import { NounType } from '../types/graphTypes.js'
import { prodLog } from '../utils/logger.js'
/** Persistence key for aggregate definitions */
const DEFINITIONS_KEY = '__aggregation_definitions__'
@ -343,6 +344,14 @@ export class AggregationIndex {
*/
private pendingAdopt = new Set<string>()
/**
* In-flight rescan targets. While a name has a staging map, ALL
* contributions (the walk's and concurrent write hooks') land there instead
* of the live map; the live map keeps serving until {@link finishBackfill}
* swaps the staging map in atomically.
*/
private backfillStaging = new Map<string, Map<string, AggregateGroupState>>()
constructor(storage: StorageAdapter, nativeProvider?: AggregationProvider) {
this.storage = storage
this.nativeProvider = nativeProvider
@ -391,10 +400,37 @@ export class AggregationIndex {
* adopt (or init never ran / failed) — it must backfill.
*/
private resolvePendingAdoptToBackfill(): void {
if (this.pendingAdopt.size > 0) {
prodLog.info(
`[Aggregation] no adoptable persisted state for: ${Array.from(this.pendingAdopt).join(', ')} — flagged for backfill`
)
}
for (const name of this.pendingAdopt) this.needsBackfill.add(name)
this.pendingAdopt.clear()
}
/**
* May this persisted state be ADOPTED? When the store exposes its committed
* watermark, the state's `sourceGeneration` must EQUAL it: behind means
* later writes are missing from the state (unclean shutdown); ahead means
* it counts writes that no longer exist (e.g. a fact-log truncation on a
* copied store pulled the watermark back). Either way: one exact rescan,
* said out loud — never a silent adopt. Stores without the capability (and
* pre-stamp state on them) fall back to hash-only adoption.
*/
private stateGenerationAdoptable(name: string, stateData: unknown): boolean {
const committed = this.storage.committedGeneration?.() ?? null
if (committed === null) return true
const raw = (stateData as Record<string, unknown>).sourceGeneration
const stamped = typeof raw === 'number' ? raw : null
if (stamped === committed) return true
prodLog.warn(
`[Aggregation] '${name}': persisted state is at generation ${stamped ?? 'unstamped'} ` +
`but the store's committed generation is ${committed} — rescanning instead of adopting`
)
return false
}
private async loadPersisted(): Promise<void> {
// Load persisted definitions
const savedDefs = await this.storage.getMetadata(DEFINITIONS_KEY)
@ -413,7 +449,11 @@ export class AggregationIndex {
const appHash = this.definitionHashes.get(def.name) || ''
if (appHash === savedHash && this.pendingAdopt.has(def.name)) {
const stateData = await this.storage.getMetadata(`${STATE_KEY_PREFIX}${def.name}__`)
if (stateData && stateData.groups) {
if (
stateData &&
stateData.groups &&
this.stateGenerationAdoptable(def.name, stateData)
) {
const groupMap = new Map<string, AggregateGroupState>()
for (const group of stateData.groups as AggregateGroupState[]) {
groupMap.set(serializeGroupKey(group.groupKey), group)
@ -421,6 +461,9 @@ export class AggregationIndex {
this.states.set(def.name, groupMap)
this.pendingAdopt.delete(def.name)
this.needsBackfill.delete(def.name)
prodLog.info(
`[Aggregation] '${def.name}': adopted persisted state (${groupMap.size} groups) — no rescan`
)
}
// No/invalid persisted state: stays in pendingAdopt and resolves
// to backfill when init settles.
@ -434,7 +477,12 @@ export class AggregationIndex {
const currentHash = hashDefinition(def)
const stateData = await this.storage.getMetadata(`${STATE_KEY_PREFIX}${def.name}__`)
if (stateData && stateData.groups && savedHash === currentHash) {
if (
stateData &&
stateData.groups &&
savedHash === currentHash &&
this.stateGenerationAdoptable(def.name, stateData)
) {
// Definition unchanged — load state
const groupMap = new Map<string, AggregateGroupState>()
for (const group of stateData.groups as AggregateGroupState[]) {
@ -443,6 +491,9 @@ export class AggregationIndex {
}
this.states.set(def.name, groupMap)
this.needsBackfill.delete(def.name)
prodLog.info(
`[Aggregation] '${def.name}': restored definition + adopted persisted state (${groupMap.size} groups)`
)
} else {
// Definition changed or no saved state — start fresh and backfill from
// existing entities (the owner drains needsBackfill on first query).
@ -483,14 +534,22 @@ export class AggregationIndex {
}))
await this.storage.saveMetadata(DEFINITIONS_KEY, { definitions: defsToSave })
// Persist dirty states
// Persist dirty states, stamped with the committed generation they
// reflect. The stamp is what makes reopen-adoption verifiable: state at a
// different generation than the store's committed watermark is stale (an
// unclean shutdown after later writes) or over-counts (a fact-log
// truncation on a copied store pulled the watermark BACK below the
// stamp) — either way the answer is one exact rescan, never a silent
// adopt. Read the generation after collecting groups so any racing
// commit resolves toward rescan, not wrong-adopt.
for (const name of this.dirty) {
const stateMap = this.states.get(name)
if (stateMap) {
const groups = Array.from(stateMap.values())
const sourceGeneration = this.storage.committedGeneration?.() ?? null
await this.storage.saveMetadata(
`${STATE_KEY_PREFIX}${name}__`,
{ groups }
sourceGeneration === null ? { groups } : { groups, sourceGeneration }
)
}
}
@ -609,9 +668,17 @@ export class AggregationIndex {
return Array.from(this.needsBackfill)
}
/** Clear an aggregate's state so a full rescan cannot double-count. */
/**
* Begin a rescan into a STAGING map. The live state is not touched — it
* keeps serving (possibly stale, but flagged pending) until the rescan
* completes and swaps in atomically. A mid-walk failure drops the staging
* map via {@link abortBackfill} and loses nothing: wiping live state before
* a scan that could throw was the destructive-before-durable defect.
* Contributions (walk + concurrent write hooks) land in staging while it
* exists, so the swapped-in result reflects writes that raced the walk.
*/
beginBackfill(name: string): void {
this.states.set(name, new Map())
this.backfillStaging.set(name, new Map())
// Reset native provider state for this aggregate too, if present.
const def = this.definitions.get(name)
if (def && this.nativeProvider?.removeAggregate && this.nativeProvider?.defineAggregate) {
@ -620,6 +687,15 @@ export class AggregationIndex {
}
}
/**
* Abandon an in-flight rescan after a failure: drop the staging map, keep
* the live state serving, leave the aggregate flagged as pending so a later
* attempt rescans. The failure itself must be surfaced loudly by the owner.
*/
abortBackfill(name: string): void {
this.backfillStaging.delete(name)
}
/** Feed one already-stored entity into a single aggregate during backfill. */
backfillEntity(name: string, entity: Record<string, unknown>): void {
if (isAggregateEntity(entity)) return
@ -633,8 +709,13 @@ export class AggregationIndex {
}
}
/** Mark an aggregate's backfill complete; rebuilt state persists on next flush(). */
/** Swap the rebuilt staging state in atomically; persists on next flush(). */
finishBackfill(name: string): void {
const staged = this.backfillStaging.get(name)
if (staged) {
this.states.set(name, staged)
this.backfillStaging.delete(name)
}
this.needsBackfill.delete(name)
this.dirty.add(name)
}
@ -873,7 +954,7 @@ export class AggregationIndex {
def: AggregateDefinition,
entity: Record<string, unknown>
): void {
const stateMap = this.states.get(aggName)!
const stateMap = (this.backfillStaging.get(aggName) ?? this.states.get(aggName))!
// Fan out: an unnest dimension makes one entity contribute to several groups.
for (const groupKey of computeGroupKeys(entity, def.groupBy)) {
@ -931,7 +1012,7 @@ export class AggregationIndex {
def: AggregateDefinition,
entity: Record<string, unknown>
): void {
const stateMap = this.states.get(aggName)!
const stateMap = (this.backfillStaging.get(aggName) ?? this.states.get(aggName))!
// Fan out: reverse the entity's contribution from every group it joined.
for (const groupKey of computeGroupKeys(entity, def.groupBy)) {
@ -987,7 +1068,7 @@ export class AggregationIndex {
* Apply results from native provider back into the state maps.
*/
private applyNativeResults(aggName: string, results: AggregateGroupState[]): void {
const stateMap = this.states.get(aggName)!
const stateMap = (this.backfillStaging.get(aggName) ?? this.states.get(aggName))!
for (const group of results) {
const serialized = serializeGroupKey(group.groupKey)
stateMap.set(serialized, group)

104
src/brainy.ts
View file

@ -384,6 +384,14 @@ class InsertPreconditionExistsSignal extends Error {
*/
export type IndexFamily = 'vector' | 'metadata' | 'graph'
/**
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
* Within the window, queries rethrow the recorded failure instantly (loud,
* cheap); after it, one new attempt is allowed. Bounds the damage of a
* caller-side tight retry loop against a deterministically-failing store.
*/
const AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS = 30_000
/**
* The main Brainy class - Clean, Beautiful, Powerful
* REAL IMPLEMENTATION - No stubs, no mocks
@ -599,6 +607,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private _pendingMigrationRunner?: MigrationRunner // Deferred migration runner for large datasets
private _aggregationIndex?: AggregationIndex // Incremental aggregation engine
private _aggregationBackfillFlight: Promise<void> | null = null // Single-flight backfill walk
// A failed walk latches its error: retries within the cooldown rethrow it
// instantly instead of re-walking, so a tight caller-side retry loop costs
// one loud error per query, never a full store walk per query.
private _aggregationBackfillFailure: { at: number; error: Error } | null = null
private _materializer?: AggregateMaterializer // Debounced materialization of aggregate results
/**
* Fields registered via `brain.trackField()` — drives optional value validation on
@ -1182,6 +1194,22 @@ export class Brainy<T = any> implements BrainyInterface<T> {
`(storage committed: ${committed}) — provider replays the gap per ` +
`the post-commit applier contract`
)
} else if (providerGen > committed) {
// The AHEAD direction is incoherence, not a replay gap: the provider's
// persisted index claims writes the store no longer has — the signature
// of a torn copy or a log truncation that pulled the committed
// watermark back (crash recovery, byte-copy of a live store). A replay
// can never converge on it and index answers may reference vanished
// writes. Name it loudly at open so it is never diagnosed from a
// silent journal; the provider's own coherence check / heal walk (or
// brain.repairIndex()) is the cure.
prodLog.warn(
`[Brainy] Versioned index provider is AHEAD of the store: provider ` +
`generation ${providerGen} vs committed ${committed}. This store was ` +
`likely copied from a live service or truncated during crash recovery. ` +
`Derived-index answers may reference rolled-back writes until the ` +
`provider heals from canonical (brain.repairIndex() forces it).`
)
}
}
@ -15828,6 +15856,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// the in-flight walk snapshotted its batch before `name` became pending —
// the next iteration starts a fresh walk that includes it.
while (index.getPendingBackfills().includes(name)) {
// Failure latch: a deterministically-failing walk must not be re-run at
// the caller's retry rate — that is a silent CPU loop wearing a retry
// loop's clothes. Within the cooldown, rethrow the recorded failure
// immediately; after it, one fresh attempt is allowed.
const failure = this._aggregationBackfillFailure
if (failure && Date.now() - failure.at < AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS) {
throw new Error(
`Aggregation backfill for '${name}' is in failure cooldown (retry in ` +
`${Math.ceil((AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS - (Date.now() - failure.at)) / 1000)}s). ` +
`Last failure: ${failure.error.message}`
)
}
if (!this._aggregationBackfillFlight) {
this._aggregationBackfillFlight = this.runAggregationBackfillWalk()
.finally(() => {
@ -15850,30 +15890,60 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const names = index.getPendingBackfills()
if (names.length === 0) return
prodLog.info(`[Aggregation] backfill walk starting for: ${names.join(', ')}`)
const startedAt = Date.now()
for (const n of names) index.beginBackfill(n)
const PAGE = 500
let offset = 0
let cursor: string | undefined
for (;;) {
const page = await this.storage.getNouns({
pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset }
})
for (const noun of page.items) {
const record = noun as unknown as Record<string, unknown>
for (const n of names) {
index.backfillEntity(n, record)
let scanned = 0
try {
const PAGE = 500
let offset = 0
let cursor: string | undefined
for (;;) {
const page = await this.storage.getNouns({
pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset }
})
for (const noun of page.items) {
const record = noun as unknown as Record<string, unknown>
for (const n of names) {
index.backfillEntity(n, record)
}
}
scanned += page.items.length
if (!page.hasMore || page.items.length === 0) break
if (page.nextCursor) {
if (page.nextCursor === cursor) {
// A non-advancing cursor with hasMore=true would loop this walk at
// CPU speed forever, silently. That is a storage pagination defect —
// fail the waiting queries loudly instead of spinning.
throw new Error(
`Aggregation backfill aborted: storage pagination returned a non-advancing cursor ` +
`after ${scanned} entities with hasMore=true — the storage adapter's getNouns cursor is broken.`
)
}
cursor = page.nextCursor
} else {
offset += page.items.length
}
}
if (!page.hasMore || page.items.length === 0) break
if (page.nextCursor) {
cursor = page.nextCursor
} else {
offset += page.items.length
}
} catch (err) {
// Non-destructive failure: drop the staging maps (live state keeps
// serving), keep the aggregates flagged pending, latch the error so
// retries within the cooldown fail fast, and say all of it out loud.
for (const n of names) index.abortBackfill(n)
this._aggregationBackfillFailure = { at: Date.now(), error: err as Error }
prodLog.warn(
`[Aggregation] backfill walk FAILED after ${scanned} entities: ${(err as Error).message} — ` +
`prior aggregate state preserved; retries suppressed for ${AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS / 1000}s`
)
throw err
}
for (const n of names) index.finishBackfill(n)
this._aggregationBackfillFailure = null
prodLog.info(
`[Aggregation] backfill walk finished: ${scanned} entities → ${names.length} aggregate(s) in ${Date.now() - startedAt}ms`
)
}
/**

10
src/graph/graphAdjacencyIndex.ts
View file

@ -282,6 +282,16 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
}
hasMore = result.hasMore
if (hasMore && (!result.nextCursor || result.nextCursor === cursor)) {
// A stalled cursor with hasMore=true would re-read the same page
// forever — a silent full-CPU loop at cold open. Abort loudly; a
// graph read failing beats a process that spins without a log line.
throw new Error(
`GraphAdjacencyIndex: verb walk stalled after ${count} verbs — storage returned ` +
`hasMore=true with ${result.nextCursor ? 'a non-advancing' : 'no'} cursor. ` +
`Aborting the cold-load; run brain.repairIndex() if this persists.`
)
}
cursor = result.nextCursor
}

28
src/storage/baseStorage.ts
View file

@ -2068,11 +2068,20 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// Cursor (8.0): resume token carrying the (shard, nounId) of the last returned
// noun — the noun mirror of getVerbsWithPagination. When present it supersedes
// `offset` and resumes the shard walk immediately AFTER that position, so a full
// walk is O(N) instead of the O(N²) of offset paging. Malformed/foreign tokens
// decode to null → offset fallback. (Previously the cursor was ignored, which
// was latent — the only multi-page consumer used a single big page — until small
// chunk sizes needed page 2 and an offset-0-on-every-call walk never terminated.)
// walk is O(N) instead of the O(N²) of offset paging. (Previously the cursor was
// ignored, which was latent — the only multi-page consumer used a single big
// page — until small chunk sizes needed page 2 and an offset-0-on-every-call
// walk never terminated.)
const cursor = this.decodeNounWalkCursor(options.cursor)
if (options.cursor && cursor === null) {
// A supplied-but-undecodable resume token must FAIL, not silently restart
// at offset 0 — to a while(hasMore) caller the silent fallback re-serves
// page 1 forever: an unbounded CPU loop wearing pagination's clothes.
throw BrainyError.storage(
`getNouns: invalid pagination cursor '${options.cursor}' — cannot resume this walk. ` +
`Restart it without a cursor.`
)
}
const collected: Array<{ noun: HNSWNounWithMetadata; shard: number }> = []
// Peek one past the window so `hasMore` is decidable. Cursor mode collects one
@ -2366,8 +2375,17 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// (shard, verbId) of the last returned verb. When present it SUPERSEDES `offset`
// and resumes the shard walk immediately AFTER that position, so a full walk is
// O(N) total instead of the O(N²) of offset paging (which re-scans from shard 0
// every page). Malformed / foreign tokens decode to null → offset fallback.
// every page).
const cursor = this.decodeVerbWalkCursor(options.cursor)
if (options.cursor && cursor === null) {
// A supplied-but-undecodable resume token must FAIL, not silently restart
// at offset 0 — to a while(hasMore) caller the silent fallback re-serves
// page 1 forever: an unbounded CPU loop wearing pagination's clothes.
throw BrainyError.storage(
`getVerbs: invalid pagination cursor '${options.cursor}' — cannot resume this walk. ` +
`Restart it without a cursor.`
)
}
// Each collected entry remembers its shard so nextCursor can point at the exact
// (shard, id) resume position.