soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: exception-safe aggregation backfill + generation-verified adoption + loud open-path guards

Browse source 
- Aggregation backfill walks build into a STAGING map and swap in atomically
  on completion. A mid-walk failure drops the staging map — the previous live
  state keeps serving, the aggregate stays flagged pending, and the storage
  error surfaces to the failing query. Previously the walk wiped live state
  before a scan that could throw, never cleared the pending flag on failure,
  and re-ran a full walk on every subsequent query: a silent wipe/walk/throw
  loop at the caller's retry rate.
- Failed walks are latched: retries within a 30s cooldown rethrow the recorded
  error instantly instead of re-walking, so a tight caller-side retry loop
  costs one loud error per query, never a full store walk per query.
- Persisted aggregation state is stamped with the store's committed generation
  at flush; reopen adoption requires stamp equality. Stale state (unclean
  shutdown) or over-counting state (a log truncation on a copied store pulled
  the watermark back) triggers exactly one loud rescan, never a silent adopt.
- The backfill/adoption path narrates: adoption decisions, walk start/finish
  with counts and duration, and failures all log by default.
- getNouns/getVerbs refuse a supplied-but-undecodable pagination cursor with a
  loud error instead of silently restarting the walk at offset 0 (which
  re-served page 1 forever to any while(hasMore) caller).
- The graph cold-load verb walk aborts loudly on a missing or non-advancing
  cursor with hasMore=true.
- A versioned index provider whose generation is AHEAD of the committed
  watermark (torn copy / crash-recovery truncation) is now named loudly at
  open, alongside the existing behind-direction message.
This commit is contained in:
David Snelling 2026-07-17 16:00:11 -07:00
parent 01a7f3dd01
commit a77b064bd7
7 changed files with 335 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

104
src/brainy.ts
View file

@ -384,6 +384,14 @@ class InsertPreconditionExistsSignal extends Error {
*/
export type IndexFamily = 'vector' | 'metadata' | 'graph'
/**
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
* Within the window, queries rethrow the recorded failure instantly (loud,
* cheap); after it, one new attempt is allowed. Bounds the damage of a
* caller-side tight retry loop against a deterministically-failing store.
*/
const AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS = 30_000
/**
* The main Brainy class - Clean, Beautiful, Powerful
* REAL IMPLEMENTATION - No stubs, no mocks
@ -599,6 +607,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private _pendingMigrationRunner?: MigrationRunner // Deferred migration runner for large datasets
private _aggregationIndex?: AggregationIndex // Incremental aggregation engine
private _aggregationBackfillFlight: Promise<void> | null = null // Single-flight backfill walk
// A failed walk latches its error: retries within the cooldown rethrow it
// instantly instead of re-walking, so a tight caller-side retry loop costs
// one loud error per query, never a full store walk per query.
private _aggregationBackfillFailure: { at: number; error: Error } | null = null
private _materializer?: AggregateMaterializer // Debounced materialization of aggregate results
/**
* Fields registered via `brain.trackField()` — drives optional value validation on
@ -1182,6 +1194,22 @@ export class Brainy<T = any> implements BrainyInterface<T> {
`(storage committed: ${committed}) — provider replays the gap per ` +
`the post-commit applier contract`
)
} else if (providerGen > committed) {
// The AHEAD direction is incoherence, not a replay gap: the provider's
// persisted index claims writes the store no longer has — the signature
// of a torn copy or a log truncation that pulled the committed
// watermark back (crash recovery, byte-copy of a live store). A replay
// can never converge on it and index answers may reference vanished
// writes. Name it loudly at open so it is never diagnosed from a
// silent journal; the provider's own coherence check / heal walk (or
// brain.repairIndex()) is the cure.
prodLog.warn(
`[Brainy] Versioned index provider is AHEAD of the store: provider ` +
`generation ${providerGen} vs committed ${committed}. This store was ` +
`likely copied from a live service or truncated during crash recovery. ` +
`Derived-index answers may reference rolled-back writes until the ` +
`provider heals from canonical (brain.repairIndex() forces it).`
)
}
}
@ -15828,6 +15856,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// the in-flight walk snapshotted its batch before `name` became pending —
// the next iteration starts a fresh walk that includes it.
while (index.getPendingBackfills().includes(name)) {
// Failure latch: a deterministically-failing walk must not be re-run at
// the caller's retry rate — that is a silent CPU loop wearing a retry
// loop's clothes. Within the cooldown, rethrow the recorded failure
// immediately; after it, one fresh attempt is allowed.
const failure = this._aggregationBackfillFailure
if (failure && Date.now() - failure.at < AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS) {
throw new Error(
`Aggregation backfill for '${name}' is in failure cooldown (retry in ` +
`${Math.ceil((AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS - (Date.now() - failure.at)) / 1000)}s). ` +
`Last failure: ${failure.error.message}`
)
}
if (!this._aggregationBackfillFlight) {
this._aggregationBackfillFlight = this.runAggregationBackfillWalk()
.finally(() => {
@ -15850,30 +15890,60 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const names = index.getPendingBackfills()
if (names.length === 0) return
prodLog.info(`[Aggregation] backfill walk starting for: ${names.join(', ')}`)
const startedAt = Date.now()
for (const n of names) index.beginBackfill(n)
const PAGE = 500
let offset = 0
let cursor: string | undefined
for (;;) {
const page = await this.storage.getNouns({
pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset }
})
for (const noun of page.items) {
const record = noun as unknown as Record<string, unknown>
for (const n of names) {
index.backfillEntity(n, record)
let scanned = 0
try {
const PAGE = 500
let offset = 0
let cursor: string | undefined
for (;;) {
const page = await this.storage.getNouns({
pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset }
})
for (const noun of page.items) {
const record = noun as unknown as Record<string, unknown>
for (const n of names) {
index.backfillEntity(n, record)
}
}
scanned += page.items.length
if (!page.hasMore || page.items.length === 0) break
if (page.nextCursor) {
if (page.nextCursor === cursor) {
// A non-advancing cursor with hasMore=true would loop this walk at
// CPU speed forever, silently. That is a storage pagination defect —
// fail the waiting queries loudly instead of spinning.
throw new Error(
`Aggregation backfill aborted: storage pagination returned a non-advancing cursor ` +
`after ${scanned} entities with hasMore=true — the storage adapter's getNouns cursor is broken.`
)
}
cursor = page.nextCursor
} else {
offset += page.items.length
}
}
if (!page.hasMore || page.items.length === 0) break
if (page.nextCursor) {
cursor = page.nextCursor
} else {
offset += page.items.length
}
} catch (err) {
// Non-destructive failure: drop the staging maps (live state keeps
// serving), keep the aggregates flagged pending, latch the error so
// retries within the cooldown fail fast, and say all of it out loud.
for (const n of names) index.abortBackfill(n)
this._aggregationBackfillFailure = { at: Date.now(), error: err as Error }
prodLog.warn(
`[Aggregation] backfill walk FAILED after ${scanned} entities: ${(err as Error).message} — ` +
`prior aggregate state preserved; retries suppressed for ${AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS / 1000}s`
)
throw err
}
for (const n of names) index.finishBackfill(n)
this._aggregationBackfillFailure = null
prodLog.info(
`[Aggregation] backfill walk finished: ${scanned} entities → ${names.length} aggregate(s) in ${Date.now() - startedAt}ms`
)
}
/**