soulcraftlabs/open-brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 343 Packages Wiki Activity Actions

feat(recovery): the catchup verdict is consumed; verb rows go live; the metadata rebuild goes online

Browse source 
Three cures on the JS metadata index, one seam:

- THE CATCHUP WIRING. The index computed its three-way watermark verdict at
  open and nothing consumed it — after a crash + adopt reopen, find() served
  the pre-crash index while canonical reads and counts recovered (caught by
  the lifecycle lane's first run). The open path now consumes the verdict:
  'adopt' is a no-op, 'catchup' folds the fact window (stamped, committed]
  through the index legs — nouns and verbs, remove-then-add, one mechanism
  for add and update — and 'rescan' runs the explicit rebuild, each narrated.
  The lane's Ch4–6 release-blocking marker comes off: the contract holds.
  Bonus root-cause: close() never stamped the projection watermarks (only
  flush() did), so any close without a prior flush verdicted a needless
  'rescan' on reopen — both doors now stamp.

- THE LIVE VERB PATH. Verb rows entered the metadata index only via rebuild
  walks, so every rebuilt store minted phantom/stale verb postings from its
  first live relate(). relate()/unrelate()/updateRelation() and remove()'s
  cascade now post/retract the verb's row in the same commit as the graph
  leg — transact() planners mirror identically — using the exact record
  shape the rebuild walk uses, so live and rebuilt populations agree.

- THE ONLINE REBUILD. rebuild() was clear-then-walk — every metadata read
  empty for the duration. rebuildMetadataIndexOnline builds a fresh manager
  beside the serving one (shared identity, in-memory build, dual-write via
  a shadow seam with zero call-site changes), atomically swaps the
  reference, and persists exactly once post-swap. A find() polled ~200x
  during a 2k-noun rebuild never dropped below its baseline.
  repairIndex({ rebuild: ['metadata'] }) uses it automatically.
This commit is contained in:
David Snelling 2026-08-25 10:01:56 -07:00
parent f8f64780b1
commit 18f172e098
6 changed files with 1275 additions and 117 deletions
Download patch file Download diff file Expand all files Collapse all files

382
src/utils/metadataIndex.ts
View file

@ -20,6 +20,7 @@ import {
type WatermarkVerdict,
type WatermarkVerdictResult
} from './projectionWatermark.js'
import type { FactScanHandle } from '../db/factLog.js'
import {
NounType,
VerbType,
@ -77,6 +78,31 @@ export interface MetadataIndexStats {
indexSize: number // in bytes
}
/**
* @description What {@link MetadataIndexManager.applyWatermarkCatchup} did,
* for the caller's narration.
* - `'noop'` — the verdict was `null`/`'adopt'`: the artifact already
* reflects committed truth. Zero index writes.
* - `'rescan'` — the verdict was `'rescan'`, OR a `'catchup'` verdict was
* demoted (no window, or no fact log to scan) — either way a full
* {@link MetadataIndexManager.rebuild} already ran; `reason` names why.
* - `'caught-up'` — the `(from, to]` window folded successfully; the
* artifact is stamped and flushed at `to`.
*/
export interface CatchupApplyResult {
action: 'noop' | 'rescan' | 'caught-up'
/** Present on `'rescan'` — why the fold could not proceed as a catchup. */
reason?: string
/** Present on `'caught-up'` — the fact-log window that was folded. */
window?: { from: number; to: number }
/** Present on `'caught-up'` — noun ops applied (add/update/delete). */
nounsApplied?: number
/** Present on `'caught-up'` — verb ops applied (add/update/delete). */
verbsApplied?: number
/** Present on `'caught-up'` — distinct committed generations folded. */
factsApplied?: number
}
export interface MetadataIndexConfig {
maxIndexSize?: number // Max number of entries per field value (default: 10000)
rebuildThreshold?: number // Rebuild if index is this % stale (default: 0.1)
@ -147,6 +173,52 @@ export class MetadataIndexManager implements MetadataIndexProvider {
private stampedWatermark: number | null = null
/** The three-way verdict computed at init; null until init runs. */
private loadVerdict: WatermarkVerdictResult | null = null
/**
* Set only when {@link loadVerdict}.verdict is `'rescan'`: whether a
* persisted artifact existed at load (even an unstamped/unverifiable
* one) — distinguishes genuine first boot (nothing here yet, routine)
* from an artifact whose watermark is unverifiable (the loud case). The
* verdict value alone doesn't carry this distinction; see {@link
* watermarkArtifactPresent}.
*/
private rescanArtifactPresent = false
/**
* @description THE BUILD-BESIDE SEAM (B3 Deliverable 3): when set (via
* {@link beginShadow}), every live `addToIndex`/`removeFromIndex` call on
* THIS instance also applies to the shadow instance — so a caller building
* a fresh replacement manager beside this one (walking canonical into it)
* never misses a write that lands during the build. This is the ONE seam
* that makes build-beside possible without touching every call site: every
* existing `AddToMetadataIndexOperation`/`RemoveFromMetadataIndexOperation`
* (and the JS manager's own `rebuild()`/catchup fold) keep calling the SAME
* serving instance exactly as before; only THIS instance knows it is also
* mirroring to a shadow. Null = no build in flight (the overwhelmingly
* common case; the check costs one property read per write).
*/
private shadow: MetadataIndexManager | null = null
/**
* @description Start mirroring every `addToIndex`/`removeFromIndex` call on
* this instance to `shadow` too — see {@link shadow}'s JSDoc. The caller
* owns sequencing: writes mirrored WHILE a canonical walk is populating
* `shadow` may be clobbered by the walk's own (possibly stale) reads for
* the same id; the caller closes that window with a bounded fact-log fold
* AFTER the walk (the same mechanism {@link applyWatermarkCatchup} uses)
* before treating `shadow` as authoritative.
* @param shadow - The manager to mirror writes to.
*/
beginShadow(shadow: MetadataIndexManager): void {
this.shadow = shadow
}
/**
* @description Stop mirroring writes to a shadow (see {@link beginShadow}).
* Idempotent; a no-op when no shadow is attached.
*/
endShadow(): void {
this.shadow = null
}
// Cardinality and field statistics tracking
private fieldStats = new Map<string, FieldStats>()
@ -1604,6 +1676,15 @@ export class MetadataIndexManager implements MetadataIndexProvider {
for (const { field } of fields) {
this.metadataCache.invalidatePattern(`field_values_${field}`)
}
// THE BUILD-BESIDE SEAM — see `shadow`'s JSDoc. Mirrors this write to a
// shadow manager under construction, if one is attached. `skipFlush:
// true` always: the shadow's own persistence is the build orchestrator's
// job (it flushes once, after the swap — never mid-build, to avoid
// colliding with this instance's own persisted keys).
if (this.shadow) {
await this.shadow.addToIndex(id, entityOrMetadata, true, false, generation)
}
}
/**
@ -1676,6 +1757,11 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// the real commit watermark (the JS mapper ignores it).
this.idMapper.remove(id, generation)
await this.idMapper.flush()
// THE BUILD-BESIDE SEAM — see `shadow`'s JSDoc.
if (this.shadow) {
await this.shadow.removeFromIndex(id, metadata, generation)
}
}
/**
@ -2759,8 +2845,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* `'adopt'` (stamped == committed, zero work), `'catchup'` (stamped <
* committed; the gap from {@link watermarkGap} awaits an incremental
* fold), `'rescan'` (unstamped or stamped above committed — never
* trusted). Null until init() has run. Computed and exposed only; no
* load behavior changes ride on it yet.
* trusted). Null until init() has run. The coordinator (`Brainy.open()`)
* consumes this via {@link applyWatermarkCatchup} right after init.
*/
watermarkVerdict(): WatermarkVerdict | null {
return this.loadVerdict?.verdict ?? null
@ -2774,6 +2860,223 @@ export class MetadataIndexManager implements MetadataIndexProvider {
return this.loadVerdict?.gap ?? null
}
/**
* @description Meaningful only when {@link watermarkVerdict} is
* `'rescan'`: `true` when a persisted artifact existed at load (even an
* unstamped/unverifiable one — real prior state, worth narrating loudly);
* `false` for a genuine first boot (nothing persisted yet — a caller
* should narrate this at a routine log level, not as an alarm, even
* though the verdict value is the same `'rescan'` either way).
*/
watermarkArtifactPresent(): boolean {
return this.rescanArtifactPresent
}
/**
* @description Consume the three-way watermark verdict {@link
* watermarkVerdict} computed at init — the cure for a crash-recovered
* store whose canonical reads/counts recover every acked write but whose
* metadata projection (flushed only periodically, not per-commit) keeps
* serving the pre-crash state. Call once, right after `init()`, before
* anything reads from this projection.
*
* - `null`/`'adopt'` → the artifact already reflects the store's
* committed generation. Zero index writes.
* - `'catchup'` → the caller-supplied `scan` (expected already opened
* over `(watermarkGap().from, watermarkGap().to]`) is folded in, ONE
* op at a time, through the SAME two legs {@link rebuild} uses (ADR-007
* A4 — one mechanism, never a second hand-rolled add/update shape): a
* tombstone (`op.record === null`) retracts id-keyed (this projection
* keeps no per-record delta log, so the pre-crash metadata for that id
* — if any — is what a value-precise removal would need, and it isn't
* available; the same tradeoff `remove()`'s null-metadata closure
* already accepts elsewhere); an after-image retracts-then-reposts, so
* an update never leaves stale postings under the old field values. A
* fact outside the window is skipped defensively (belt: the scan is
* already opened to the window; suspenders: this loop never trusts an
* over-run). On success the artifact is stamped at `to` and flushed —
* the same STAMP-AFTER-DATA door {@link flush} always writes through.
* - `'rescan'` (or a `'catchup'` verdict with no window, or no `scan` to
* fold — the store hosts no fact log) → the persisted artifact is
* unverifiable; this method runs the existing {@link rebuild} itself
* rather than leave the caller to notice and trigger it separately.
*
* @param scan - An open fact scan covering the catchup window (see
* {@link Brainy.scanFacts}), or `null` when none is available/needed.
* Ignored when the verdict is not `'catchup'`.
* @returns What happened — see {@link CatchupApplyResult}.
*/
async applyWatermarkCatchup(scan: FactScanHandle | null): Promise<CatchupApplyResult> {
const verdict = this.watermarkVerdict()
if (verdict === null || verdict === 'adopt') return { action: 'noop' }
if (verdict === 'rescan') {
await this.rebuild()
return {
action: 'rescan',
reason: 'persisted artifact is unverifiable (unstamped, or stamped ABOVE the ' +
"store's committed generation) — never adopting unverifiable state"
}
}
// verdict === 'catchup'
const window = this.watermarkGap()
if (window === null) {
await this.rebuild()
return { action: 'rescan', reason: "'catchup' verdict exposed no window — cannot bound a fold" }
}
if (scan === null) {
await this.rebuild()
return {
action: 'rescan',
reason: `no fact log available to fold the (${window.from}, ${window.to}] catchup window`
}
}
const { nounsApplied, verbsApplied, factsApplied } = await this.foldFactWindow(scan, window.from, window.to)
this.stampWatermark(window.to)
await this.flush()
return { action: 'caught-up', window, nounsApplied, verbsApplied, factsApplied }
}
/**
* @description Fold an open fact scan's `(fromGeneration, toGeneration]`
* window into this projection, ONE op at a time, through the SAME two legs
* {@link rebuild} uses (ADR-007 A4 — one mechanism, never a second
* hand-rolled add/update shape): a tombstone retracts id-keyed; an
* after-image retracts-then-reposts. THE CORE LOOP shared by {@link
* applyWatermarkCatchup} (which stamps + flushes after) and {@link
* buildBeside} (which does neither — persistence is the caller's job,
* exactly once, after a swap). Never stamps, never flushes, never touches
* storage beyond what `addToIndex`/`removeFromIndex` do internally
* (skipFlush is always forced true).
* @param scan - An open fact scan.
* @param fromGeneration - Window lower bound (exclusive).
* @param toGeneration - Window upper bound (inclusive).
* @returns Counts for the caller's narration.
*/
private async foldFactWindow(
scan: FactScanHandle,
fromGeneration: number,
toGeneration: number
): Promise<{ nounsApplied: number; verbsApplied: number; factsApplied: number }> {
let nounsApplied = 0
let verbsApplied = 0
let factsApplied = 0
for await (const batch of scan.batches()) {
for (const fact of batch.facts) {
// Defensive containment: the scan is already opened to the window,
// but a fact outside it is never applied regardless.
if (fact.generation <= fromGeneration || fact.generation > toGeneration) continue
const generation = BigInt(fact.generation)
for (const op of fact.ops) {
if (op.record === null) {
// TOMBSTONE — the id-keyed removal path (no per-record delta
// log to recover the old field values from).
await this.removeFromIndex(op.id, undefined, generation)
} else {
// AFTER-IMAGE — retract any stale posting for this id, then
// repost the new shape. Covers both a fresh add (nothing to
// retract; a no-op-ish remove) and an update, through the same
// two calls.
await this.removeFromIndex(op.id, undefined, generation)
await this.indexStoredRecord(op.id, op.record.metadata, {
skipFlush: true,
deferWrites: false,
generation
})
}
if (op.kind === 'noun') nounsApplied++
else verbsApplied++
}
factsApplied++
}
}
return { nounsApplied, verbsApplied, factsApplied }
}
/**
* @description B3 Deliverable 3 — the shadow-build lifecycle's init: the
* MINIMUM setup {@link buildBeside} needs, deliberately NOT the general
* {@link init} sequence. Two reasons general `init()` is unsafe for a
* build-beside shadow:
* 1. `init()` unconditionally re-initializes the id mapper from storage
* (`idMapper.init()`) — safe for a FRESH mapper, but this instance is
* constructed with the CURRENTLY-SERVING manager's SHARED, already-live
* mapper (identity is shared, never a second mapper — this train's own
* law). Re-running its init() would DISCARD every not-yet-flushed
* UUID↔int assignment sitting in memory, breaking the live manager's
* own serving mid-build.
* 2. `init()` loads the field registry and, on a registry that's
* missing/empty while canonical has entities (exactly the shape a
* rebuild is often invoked to FIX), triggers `rebuild()` itself —
* WITHOUT `inMemoryOnly`, which would touch the shared storage keys
* the live manager depends on.
* What this DOES run: the WASM roaring-bitmap library init (idempotent;
* needed before any column-store write) and the column store's OWN
* segment-manifest discovery (read-only against shared storage; needed so
* THIS instance's eventual post-swap flush continues segment numbering
* correctly instead of colliding with the retiring manager's segments).
*/
private async initForShadowBuild(): Promise<void> {
await roaringLibraryInitialize()
try {
await this.columnStore.init(this.storage, this.idMapper)
} catch (err) {
prodLog.warn('[MetadataIndex] shadow build: column store storage discovery failed:', err)
}
}
/**
* @description B3 Deliverable 3 — THE ONLINE REBUILD's manager-side half:
* populate THIS instance (expected fresh/empty, constructed with the SAME
* storage + idMapper as the manager it will replace — see {@link
* initForShadowBuild}) from canonical storage without ever touching the
* shared storage keys the currently-serving manager depends on — no chunk
* deletion, no flush, anywhere in this call. The caller (the brain's
* rebuild-beside orchestrator) is responsible for:
* 1. Attaching this instance as a {@link beginShadow} target on the OLD
* manager BEFORE calling this, so live writes during the walk mirror
* here too (best-effort — the walk below may still clobber a mirrored
* write with a stale read for the same id; the fold after the walk is
* what makes the final state authoritative, not the mirror).
* 2. Swapping its own reference to this instance once this resolves.
* 3. Calling {@link stampWatermark} + {@link flush} EXACTLY ONCE, after
* the swap — this instance never persists itself.
* @param committedGenerationAtStart - The store's committed generation
* captured by the caller BEFORE this call — the fold's lower bound.
* @returns The generation this instance's canonical data reflects once the
* walk + fold settle — the fold's upper bound (writes committed after
* this point but before the swap only reach this instance via the live
* {@link beginShadow} mirror, so the caller re-reads the store's
* committed generation right before stamping, rather than trusting this
* return value as final).
* @throws If canonical advanced during the walk but no fact log is
* available to fold the gap — never a silently incomplete shadow.
*/
async buildBeside(committedGenerationAtStart: number): Promise<number> {
await this.initForShadowBuild()
await this.rebuild({ inMemoryOnly: true })
const committedAfterWalk = this.storage.committedGeneration?.() ?? committedGenerationAtStart
if (committedAfterWalk > committedGenerationAtStart) {
const scan = this.storage.scanFacts?.({
fromGeneration: committedGenerationAtStart + 1,
toGeneration: committedAfterWalk
}) ?? null
if (scan === null) {
throw new Error(
`MetadataIndexManager.buildBeside: canonical advanced from generation ` +
`${committedGenerationAtStart} to ${committedAfterWalk} during the walk, but this ` +
`store hosts no fact log to fold the gap — refusing a silently incomplete shadow`
)
}
await this.foldFactWindow(scan, committedGenerationAtStart, committedAfterWalk)
}
return committedAfterWalk
}
/**
* @description Write the pending watermark stamp as a sidecar record —
* always called AFTER the data it certifies is durable. A stamp-write
@ -2826,6 +3129,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
if (result.verdict === 'rescan') {
const artifactPresent = this.fieldIndexes.size > 0 || stamped !== null
this.rescanArtifactPresent = artifactPresent
if (artifactPresent) {
prodLog.warn(
`[MetadataIndex] watermark verdict: RESCAN — persisted index is ` +
@ -3484,13 +3788,48 @@ export class MetadataIndexManager implements MetadataIndexProvider {
}
}
/**
* @description Index one raw stored noun/verb record — THE ONE add leg
* shared by {@link rebuild}'s canonical walk and {@link
* applyWatermarkCatchup}'s fact-log fold (ADR-007 A4: one mechanism,
* never a second hand-rolled shape). No conversion step is needed here:
* a raw stored record (`storage.getNounMetadata`/`getVerbMetadata`, or a
* fact's after-image `record.metadata`) is byte-identical — both read the
* exact same canonical path — and already the v2 nested-bag
* ("entity-record") shape {@link extractIndexableFields} expects.
* @param id - Entity/relationship id.
* @param storedMetadata - The raw stored metadata record.
* @param opts.skipFlush - Forwarded to {@link addToIndex}.
* @param opts.deferWrites - Forwarded to {@link addToIndex}.
* @param opts.generation - Forwarded to {@link addToIndex}.
*/
private async indexStoredRecord(
id: string,
storedMetadata: unknown,
opts: { skipFlush: boolean; deferWrites: boolean; generation?: bigint }
): Promise<void> {
await this.addToIndex(id, storedMetadata, opts.skipFlush, opts.deferWrites, opts.generation)
}
/**
* Rebuild entire index from scratch using pagination
* Non-blocking version that yields control back to event loop
* Sparse indices now lazy-loaded via UnifiedCache (no need to clear Map)
*
* @param options.inMemoryOnly - B3 Deliverable 3 (build-beside): when
* `true`, this call never touches the shared storage keys another,
* currently-serving `MetadataIndexManager` over the SAME storage may
* depend on — it skips deleting persisted legacy chunk files AND skips
* the final `flush()` (which would otherwise write field indexes AND
* flush the column store's tail buffers to shared segment keys,
* colliding with a live manager's own writes). The caller ({@link
* buildBeside}) owns persistence entirely — exactly once, after this
* instance becomes the sole owner via an atomic swap. Default `false`
* (every other caller keeps today's clear-then-persist behavior).
*/
async rebuild(): Promise<void> {
async rebuild(options?: { inMemoryOnly?: boolean }): Promise<void> {
if (this.isRebuilding) return
const inMemoryOnly = options?.inMemoryOnly ?? false
this.isRebuilding = true
try {
@ -3519,15 +3858,22 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// here — it's always saved at the end of rebuild via flush(). This ensures
// that if rebuild fails partway, the next init() can still discover fields
// and trigger another rebuild attempt.
prodLog.info('Clearing existing metadata index chunks from storage...')
const existingFields = await this.getPersistedFieldList()
//
// SKIPPED for inMemoryOnly: these are the SHARED storage keys a live
// manager over the same storage may still be reading (see this
// method's JSDoc) — deleting them before the swap is a live-read
// hazard, not a cleanup.
if (!inMemoryOnly) {
prodLog.info('Clearing existing metadata index chunks from storage...')
const existingFields = await this.getPersistedFieldList()
if (existingFields.length > 0) {
for (const field of existingFields) {
await this.deleteFieldChunks(field)
if (existingFields.length > 0) {
for (const field of existingFields) {
await this.deleteFieldChunks(field)
}
prodLog.info(`Cleared ${existingFields.length} field indexes from storage`)
}
prodLog.info(`Cleared ${existingFields.length} field indexes from storage`)
}
// EntityIdMapper is intentionally NOT cleared here. Rebuild re-iterates
@ -3582,7 +3928,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
for (const noun of result.items) {
const metadata = metadataBatch.get(noun.id)
if (metadata) {
await this.addToIndex(noun.id, metadata, true, true)
await this.indexStoredRecord(noun.id, metadata, { skipFlush: true, deferWrites: true })
}
}
@ -3627,7 +3973,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
for (const verb of result.items) {
const metadata = verbMetadataBatch.get(verb.id)
if (metadata) {
await this.addToIndex(verb.id, metadata, true, true)
await this.indexStoredRecord(verb.id, metadata, { skipFlush: true, deferWrites: true })
}
}
@ -3637,8 +3983,16 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// Flush to storage. The column store's flush() handles tail-buffer-to-
// segment promotion + manifest persistence.
prodLog.debug('💾 Flushing metadata index to storage...')
await this.flush()
//
// SKIPPED for inMemoryOnly — see this method's JSDoc: flush() writes
// the shared field-index keys AND flushes the column store's tail
// buffers to shared segment keys, which would race a live manager's
// own flushes over the SAME storage. The caller flushes exactly once,
// after the swap.
if (!inMemoryOnly) {
prodLog.debug('💾 Flushing metadata index to storage...')
await this.flush()
}
prodLog.info(`✅ Metadata index rebuild completed! Processed ${totalNounsProcessed} nouns and ${totalVerbsProcessed} verbs`)