feat(recovery): the catchup verdict is consumed; verb rows go live; the metadata rebuild goes online
Three cures on the JS metadata index, one seam:
- THE CATCHUP WIRING. The index computed its three-way watermark verdict at
open and nothing consumed it — after a crash + adopt reopen, find() served
the pre-crash index while canonical reads and counts recovered (caught by
the lifecycle lane's first run). The open path now consumes the verdict:
'adopt' is a no-op, 'catchup' folds the fact window (stamped, committed]
through the index legs — nouns and verbs, remove-then-add, one mechanism
for add and update — and 'rescan' runs the explicit rebuild, each narrated.
The lane's Ch4–6 release-blocking marker comes off: the contract holds.
Bonus root-cause: close() never stamped the projection watermarks (only
flush() did), so any close without a prior flush verdicted a needless
'rescan' on reopen — both doors now stamp.
- THE LIVE VERB PATH. Verb rows entered the metadata index only via rebuild
walks, so every rebuilt store minted phantom/stale verb postings from its
first live relate(). relate()/unrelate()/updateRelation() and remove()'s
cascade now post/retract the verb's row in the same commit as the graph
leg — transact() planners mirror identically — using the exact record
shape the rebuild walk uses, so live and rebuilt populations agree.
- THE ONLINE REBUILD. rebuild() was clear-then-walk — every metadata read
empty for the duration. rebuildMetadataIndexOnline builds a fresh manager
beside the serving one (shared identity, in-memory build, dual-write via
a shadow seam with zero call-site changes), atomically swaps the
reference, and persists exactly once post-swap. A find() polled ~200x
during a 2k-noun rebuild never dropped below its baseline.
repairIndex({ rebuild: ['metadata'] }) uses it automatically.
This commit is contained in:
parent
f8f64780b1
commit
18f172e098
6 changed files with 1275 additions and 117 deletions
476
src/brainy.ts
476
src/brainy.ts
|
|
@ -1385,6 +1385,20 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
])
|
||||
}
|
||||
|
||||
// METADATA WATERMARK CATCHUP: the JS metadata index computed its
|
||||
// three-way watermark verdict inside metadataIndex.init() above,
|
||||
// against the generation store's now-FINAL committed generation (the
|
||||
// crash-recovery fold above — the durable-at-ack replay of acked
|
||||
// writes whose canonical bytes hadn't reached disk — has already run,
|
||||
// and any rolled-back-transaction rebuild just above already brought
|
||||
// every index current, so the verdict is consumed here whether or not
|
||||
// that rebuild ran). Consumed BEFORE the rebuild gate below and BEFORE
|
||||
// this open serves any read — the cure for the class of bug where
|
||||
// canonical get()/counts recover a crash-window write but find()
|
||||
// keeps serving the metadata index's pre-crash state (the index
|
||||
// flushes only periodically, not per-commit).
|
||||
await this.consumeMetadataWatermarkVerdict(generationOpenResult.rolledBackGenerations > 0)
|
||||
|
||||
// 8.0 versioned-provider replay-gap check: a provider whose persisted
|
||||
// index generation is behind the storage layer's committed generation
|
||||
// replays the gap itself (post-commit applier contract) — surface the
|
||||
|
|
@ -3672,6 +3686,85 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
if (deferringEmbed) this.kickEmbedWorker()
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Build the metadata-index retraction operation for one id
|
||||
* (noun or verb) — the null-metadata-safe closure shared by every removal
|
||||
* leg that reaches the metadata index with a possibly-missed pre-read:
|
||||
* `remove()`'s own noun leg, its verb-cascade retractions, `unrelate()`,
|
||||
* and their `transact()`/`planTx*` mirrors (both callers add the returned
|
||||
* operation to their own batch — `tx.addOperation()` for a single-op
|
||||
* transaction, `plan.operations.push()` for a planned `transact()` batch).
|
||||
* THE NULL-METADATA SKIP IS CLOSED (a posting-leak class):
|
||||
* - metadata present → the ordinary, provider-agnostic
|
||||
* `RemoveFromMetadataIndexOperation` (exact per-field retraction).
|
||||
* - metadata absent (a torn pre-read, or the row was already gone) →
|
||||
* a provider exposing `removeEntityById` (the id-keyed contract) gets
|
||||
* exact per-entity retraction via its reverse record; the JS index
|
||||
* gets `removeFromIndex(id)` — safe id-keyed cleanup (deleted bitmap +
|
||||
* id mapper; field statistics reconcile at the next rebuild/repairIndex),
|
||||
* narrated; a native provider WITHOUT the contract is never called
|
||||
* metadata-omitted (that path walks its value space) — the skip is
|
||||
* tracked in the degraded set instead, narrated, so `repairIndex()`
|
||||
* reconciles it (and this method returns `null` — no operation to add).
|
||||
* Silence is the only thing outlawed.
|
||||
* @param id - The noun/verb id being retracted.
|
||||
* @param metadata - The pre-read metadata/entity structure, or falsy when
|
||||
* the read missed.
|
||||
* @param context - Narration prefix identifying the caller/id, e.g.
|
||||
* `remove(${id})` or `remove(${entityId}) cascade unrelate ${verbId}`.
|
||||
* @returns The operation to add to the caller's batch, or `null` when
|
||||
* nothing could be done (already narrated + tracked as degraded).
|
||||
*/
|
||||
private metadataIndexRetractionOp(
|
||||
id: string,
|
||||
metadata: unknown,
|
||||
context: string
|
||||
): Operation | null {
|
||||
if (metadata) {
|
||||
return new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata, this.indexWriteGeneration)
|
||||
}
|
||||
const prov = this.metadataIndex as unknown as {
|
||||
removeEntityById?: (id: string) => Promise<void>
|
||||
removeFromIndex?: (id: string, metadata?: unknown, generation?: bigint) => Promise<void>
|
||||
}
|
||||
if (typeof prov.removeEntityById === 'function') {
|
||||
const g = this.indexWriteGeneration
|
||||
return {
|
||||
name: 'RemoveEntityByIdTombstone',
|
||||
execute: async () => {
|
||||
await prov.removeEntityById!(id)
|
||||
return async () => {
|
||||
// Undo of an id-keyed tombstone on an absent row: nothing to
|
||||
// restore (the row had no readable metadata to re-post).
|
||||
void g
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if (this.metadataIndex instanceof MetadataIndexManager) {
|
||||
const gv = this.indexWriteGeneration
|
||||
prodLog.warn(
|
||||
`[Brainy] ${context}: no metadata at delete — id-keyed index cleanup ran ` +
|
||||
`(deleted bitmap + id mapper); field statistics reconcile at the next rebuild/repairIndex.`
|
||||
)
|
||||
return {
|
||||
name: 'IdKeyedIndexCleanup',
|
||||
execute: async () => {
|
||||
await prov.removeFromIndex!(id, undefined, typeof gv === 'function' ? gv() : gv)
|
||||
return async () => {}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
this._indexDegradedIds.add(id)
|
||||
prodLog.warn(
|
||||
`[Brainy] ${context}: no metadata at delete and this provider has no id-keyed ` +
|
||||
`removal — its postings for this id are NOT tombstoned yet (tracked as degraded; ` +
|
||||
`repairIndex() reconciles). Never calling a metadata-omitted native removal: that ` +
|
||||
`path walks the store's value space.`
|
||||
)
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove an entity and all its relationships
|
||||
*
|
||||
|
|
@ -3736,61 +3829,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
)
|
||||
}
|
||||
|
||||
// Operation 2: Remove from metadata index. THE NULL-METADATA SKIP IS
|
||||
// CLOSED (a posting-leak class, confirmed at this site): when the
|
||||
// pre-read missed, the leg no longer silently skips —
|
||||
// - a provider exposing removeEntityById (the id-keyed contract)
|
||||
// gets it: exact per-entity retraction via its reverse record;
|
||||
// - the JS index gets removeFromIndex(id) — safe id-keyed cleanup
|
||||
// (deleted bitmap + id mapper; field stats reconcile at rebuild);
|
||||
// - a NATIVE provider WITHOUT the contract is never called
|
||||
// metadata-omitted (that path walks its value space) — the skip
|
||||
// happens, but NARRATED and tracked in the degraded set so
|
||||
// repairIndex reconciles it. Silence is the only thing outlawed.
|
||||
if (metadata) {
|
||||
tx.addOperation(
|
||||
new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata, this.indexWriteGeneration)
|
||||
)
|
||||
} else {
|
||||
const prov = this.metadataIndex as unknown as {
|
||||
removeEntityById?: (id: string) => Promise<void>
|
||||
removeFromIndex?: (id: string, metadata?: unknown, generation?: bigint) => Promise<void>
|
||||
}
|
||||
if (typeof prov.removeEntityById === 'function') {
|
||||
const g = this.indexWriteGeneration
|
||||
tx.addOperation({
|
||||
name: 'RemoveEntityByIdTombstone',
|
||||
execute: async () => {
|
||||
await prov.removeEntityById!(id)
|
||||
return async () => {
|
||||
// Undo of an id-keyed tombstone on an absent row: nothing
|
||||
// to restore (the row had no readable metadata to re-post).
|
||||
void g
|
||||
}
|
||||
}
|
||||
})
|
||||
} else if (this.metadataIndex instanceof MetadataIndexManager) {
|
||||
const gv = this.indexWriteGeneration
|
||||
tx.addOperation({
|
||||
name: 'IdKeyedIndexCleanup',
|
||||
execute: async () => {
|
||||
await prov.removeFromIndex!(id, undefined, typeof gv === 'function' ? gv() : gv)
|
||||
return async () => {}
|
||||
}
|
||||
})
|
||||
prodLog.warn(
|
||||
`[Brainy] remove(${id}): no metadata at delete — id-keyed index cleanup ran ` +
|
||||
`(deleted bitmap + id mapper); field statistics reconcile at the next rebuild/repairIndex.`
|
||||
)
|
||||
} else {
|
||||
this._indexDegradedIds.add(id)
|
||||
prodLog.warn(
|
||||
`[Brainy] remove(${id}): no metadata at delete and this provider has no id-keyed ` +
|
||||
`removal — its postings for this id are NOT tombstoned yet (tracked as degraded; ` +
|
||||
`repairIndex() reconciles). Never calling a metadata-omitted native removal: that ` +
|
||||
`path walks the store's value space.`
|
||||
)
|
||||
}
|
||||
// Operation 2: Remove from metadata index (null-metadata-safe — see
|
||||
// metadataIndexRetractionOp's JSDoc for the full closure).
|
||||
{
|
||||
const retractionOp = this.metadataIndexRetractionOp(id, metadata, `remove(${id})`)
|
||||
if (retractionOp) tx.addOperation(retractionOp)
|
||||
}
|
||||
|
||||
// Operation 3: Delete noun (full removal). The pre-read metadata rides
|
||||
|
|
@ -3808,6 +3851,21 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
tx.addOperation(
|
||||
new RemoveFromGraphIndexOperation(this.graphIndex, verb, { sourceInt, targetInt }, this.graphWriteGeneration)
|
||||
)
|
||||
// Retract the cascaded relation's metadata-index row too — the
|
||||
// live mirror of what a rebuild would derive for this (now-gone)
|
||||
// edge (mirrors the noun leg above). The whole hydrated verb
|
||||
// (system fields top-level + the custom bag under `metadata`,
|
||||
// same shape `extractIndexableFields` reads for any entity-record
|
||||
// frame) is the before-image — every entry in `allVerbs` was
|
||||
// already successfully hydrated by the reads above, so this is
|
||||
// never metadata-omitted in practice, but the closure stays
|
||||
// defensive rather than assuming.
|
||||
{
|
||||
const cascadeRetractionOp = this.metadataIndexRetractionOp(
|
||||
verb.id, verb, `remove(${id}) cascade unrelate ${verb.id}`
|
||||
)
|
||||
if (cascadeRetractionOp) tx.addOperation(cascadeRetractionOp)
|
||||
}
|
||||
// Delete verb metadata
|
||||
tx.addOperation(
|
||||
new DeleteVerbMetadataOperation(this.storage, verb.id)
|
||||
|
|
@ -4641,6 +4699,16 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
)
|
||||
)
|
||||
|
||||
// Operation 3b: Add the verb's metadata-index row, in the SAME
|
||||
// commit as the graph leg — the live mirror of what rebuild()'s
|
||||
// verb walk already derives (ADR-007 A4: one mechanism, never a
|
||||
// second hand-rolled shape). `verbMetadata` is the exact raw stored
|
||||
// record `SaveVerbMetadataOperation` above just persisted — the same
|
||||
// shape `storage.getVerbMetadata()`/rebuild() read back.
|
||||
tx.addOperation(
|
||||
new AddToMetadataIndexOperation(this.metadataIndex, id, verbMetadata, this.indexWriteGeneration)
|
||||
)
|
||||
|
||||
// Create bidirectional if requested
|
||||
if (params.bidirectional && reverseId) {
|
||||
const reverseVerb: GraphVerb = {
|
||||
|
|
@ -4678,6 +4746,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
(verbInt) => this.cacheVerbInt(verbInt, reverseId)
|
||||
)
|
||||
)
|
||||
|
||||
// Operation 6b: Add the reverse edge's metadata-index row (same
|
||||
// stored shape as the primary edge — SaveVerbMetadataOperation
|
||||
// above persists the same `verbMetadata` object for both).
|
||||
tx.addOperation(
|
||||
new AddToMetadataIndexOperation(this.metadataIndex, reverseId, verbMetadata, this.indexWriteGeneration)
|
||||
)
|
||||
}
|
||||
},
|
||||
undefined,
|
||||
|
|
@ -4760,6 +4835,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
)
|
||||
}
|
||||
|
||||
// Operation 1b: Retract the verb's metadata-index row — the live
|
||||
// mirror of remove()'s cascade leg (null-metadata-safe; see
|
||||
// metadataIndexRetractionOp's JSDoc). Nothing to retract when the
|
||||
// pre-read found no verb (already gone / never existed).
|
||||
if (verb) {
|
||||
const retractionOp = this.metadataIndexRetractionOp(id, verb, `unrelate(${id})`)
|
||||
if (retractionOp) tx.addOperation(retractionOp)
|
||||
}
|
||||
|
||||
// Operation 2: Delete verb metadata (which also deletes vector)
|
||||
tx.addOperation(
|
||||
new DeleteVerbMetadataOperation(this.storage, id)
|
||||
|
|
@ -4903,6 +4987,23 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
new UpdateVerbMetadataOperation(this.storage, params.id, updatedMetadata)
|
||||
)
|
||||
|
||||
// Re-post the verb's metadata-index row — remove the old shape, add
|
||||
// the new one, same commit (the plain pair; there is no update-op
|
||||
// capability for the metadata leg yet — see the GRAPH leg's
|
||||
// typeChanged branch just below for the capability this ISN'T:
|
||||
// that's the graph adjacency's own remove+add, keyed on the verb
|
||||
// TYPE changing; the metadata row updates on EVERY updateRelation()
|
||||
// call, since metadata/subtype/weight/etc. can all change without a
|
||||
// type change). `existing` is the pre-update hydrated verb (already
|
||||
// read above); `updatedMetadata` is the raw stored record just
|
||||
// persisted — the same shape relate()/rebuild() use to add.
|
||||
tx.addOperation(
|
||||
new RemoveFromMetadataIndexOperation(this.metadataIndex, params.id, existing, this.indexWriteGeneration)
|
||||
)
|
||||
tx.addOperation(
|
||||
new AddToMetadataIndexOperation(this.metadataIndex, params.id, updatedMetadata, this.indexWriteGeneration)
|
||||
)
|
||||
|
||||
// If the verb type changed, re-index in graph adjacency so traversal-by-type
|
||||
// stays consistent. The id is preserved across the swap.
|
||||
if (typeChanged && reindexInts) {
|
||||
|
|
@ -10603,6 +10704,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
new RemoveFromGraphIndexOperation(this.graphIndex, verb, () => this.resolveVerbEndpointInts(verb), this.graphWriteGeneration),
|
||||
new DeleteVerbMetadataOperation(this.storage, verb.id)
|
||||
)
|
||||
// Retract the cascaded relation's metadata-index row too — the
|
||||
// transact() mirror of remove()'s single-op cascade leg
|
||||
// (null-metadata-safe; see metadataIndexRetractionOp's JSDoc).
|
||||
{
|
||||
const cascadeRetractionOp = this.metadataIndexRetractionOp(
|
||||
verb.id, verb, `transact remove(${id}) cascade unrelate ${verb.id}`
|
||||
)
|
||||
if (cascadeRetractionOp) plan.operations.push(cascadeRetractionOp)
|
||||
}
|
||||
plan.touchedVerbs.push(verb.id)
|
||||
state.verbs.delete(verb.id)
|
||||
state.removedVerbs.add(verb.id)
|
||||
|
|
@ -10769,7 +10879,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// id mapper to assign an int for an entity that did not exist yet.
|
||||
new AddToGraphIndexOperation(this.graphIndex, verb, () => this.resolveVerbEndpointInts(verb), this.graphWriteGeneration, (verbInt) =>
|
||||
this.cacheVerbInt(verbInt, id)
|
||||
)
|
||||
),
|
||||
// The transact() mirror of relate()'s metadata-index leg — same
|
||||
// commit as the graph leg, same raw stored shape.
|
||||
new AddToMetadataIndexOperation(this.metadataIndex, id, verbMetadata, this.indexWriteGeneration)
|
||||
)
|
||||
plan.touchedVerbs.push(id)
|
||||
state.verbs.set(id, verb)
|
||||
|
|
@ -10811,7 +10924,8 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
new SaveVerbMetadataOperation(this.storage, reverseId, verbMetadata),
|
||||
new AddToGraphIndexOperation(this.graphIndex, reverseVerb, () => this.resolveVerbEndpointInts(reverseVerb), this.graphWriteGeneration, (verbInt) =>
|
||||
this.cacheVerbInt(verbInt, reverseId)
|
||||
)
|
||||
),
|
||||
new AddToMetadataIndexOperation(this.metadataIndex, reverseId, verbMetadata, this.indexWriteGeneration)
|
||||
)
|
||||
plan.touchedVerbs.push(reverseId)
|
||||
state.verbs.set(reverseId, reverseVerb)
|
||||
|
|
@ -10856,6 +10970,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// may have been created earlier in this same batch (forward refs).
|
||||
new RemoveFromGraphIndexOperation(this.graphIndex, verb, () => this.resolveVerbEndpointInts(verb), this.graphWriteGeneration)
|
||||
)
|
||||
// The transact() mirror of unrelate()'s metadata-index leg
|
||||
// (null-metadata-safe; see metadataIndexRetractionOp's JSDoc — a
|
||||
// present `verb` here is never metadata-omitted, but the closure
|
||||
// stays defensive rather than assuming).
|
||||
const retractionOp = this.metadataIndexRetractionOp(id, verb, `transact unrelate(${id})`)
|
||||
if (retractionOp) plan.operations.push(retractionOp)
|
||||
}
|
||||
plan.operations.push(new DeleteVerbMetadataOperation(this.storage, id))
|
||||
plan.touchedVerbs.push(id)
|
||||
|
|
@ -11564,6 +11684,31 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Stamp every projection's watermark with the store's
|
||||
* current committed generation — the door BOTH {@link flush} and {@link
|
||||
* close} open right before persisting, so EITHER path leaves a stamped,
|
||||
* `'adopt'`-verdicting artifact on disk (stamp-after-data still holds
|
||||
* inside each owner: this only hands the generation over — the owner's
|
||||
* OWN flush is what durably writes the stamp, LAST). Before this method
|
||||
* existed, `close()` had its own separate flush fan-out that never
|
||||
* stamped, so a `close()` without a preceding explicit `flush()` left
|
||||
* every projection unstamped — a real, closed store that legitimately
|
||||
* verdicts `'rescan'` on its very next open (not a bug in the verdict,
|
||||
* a gap in `close()`'s persistence completeness that this closes).
|
||||
* No `committedGeneration` capability, or a replacement provider that
|
||||
* doesn't carry the stamp method (a native pair swaps these managers) =
|
||||
* no stamp = the owner's verdict machinery treats the artifact as
|
||||
* legacy — never a flush/close crash either way.
|
||||
*/
|
||||
private stampProjectionWatermarks(): void {
|
||||
const wmGen = this.storage?.committedGeneration?.() ?? null
|
||||
if (wmGen === null) return
|
||||
;(this.metadataIndex as { stampWatermark?: (g: number) => void }).stampWatermark?.(wmGen)
|
||||
;(this.index as { stampWatermark?: (g: number) => void }).stampWatermark?.(wmGen)
|
||||
;(this.graphIndex as { stampWatermark?: (g: number) => void }).stampWatermark?.(wmGen)
|
||||
}
|
||||
|
||||
/**
|
||||
* Flush all indexes and caches to persistent storage
|
||||
* CRITICAL FIX: Ensures data survives server restarts
|
||||
|
|
@ -11603,22 +11748,8 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
await this.generationStore.flushPendingSingleOps()
|
||||
|
||||
// Flush all components in parallel for performance
|
||||
// Watermark stamps ride every flush fan-out: stamp each projection with
|
||||
// the committed generation BEFORE its flush persists (stamp-after-data
|
||||
// holds inside each owner — the stamp is its LAST write; here we only
|
||||
// hand the generation over). No committedGeneration capability = no
|
||||
// stamp = the owner's verdict machinery treats the artifact as legacy.
|
||||
{
|
||||
const wmGen = this.storage?.committedGeneration?.() ?? null
|
||||
if (wmGen !== null) {
|
||||
// ALL THREE optional-chained: a replacement provider (the native
|
||||
// pair swaps these managers) may not carry the stamp method — a
|
||||
// missing stamp is a verdict-side rescan, never a flush crash.
|
||||
;(this.metadataIndex as { stampWatermark?: (g: number) => void }).stampWatermark?.(wmGen)
|
||||
;(this.index as { stampWatermark?: (g: number) => void }).stampWatermark?.(wmGen)
|
||||
;(this.graphIndex as { stampWatermark?: (g: number) => void }).stampWatermark?.(wmGen)
|
||||
}
|
||||
}
|
||||
// Watermark stamps ride every flush fan-out — see stampProjectionWatermarks().
|
||||
this.stampProjectionWatermarks()
|
||||
await Promise.all([
|
||||
// 1. Flush storage adapter counts (entity/verb counts by type)
|
||||
(async () => {
|
||||
|
|
@ -16316,6 +16447,179 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Consume the JS metadata index's watermark verdict (see
|
||||
* {@link MetadataIndexManager.watermarkVerdict}) at open — the coordinator
|
||||
* half of the catchup wiring; {@link MetadataIndexManager.applyWatermarkCatchup}
|
||||
* is the mechanism half. Feature-detected to the JS manager only: a native
|
||||
* metadata-index provider consumes the same verdict door in its own train
|
||||
* (this method never touches the native-provider wrapper contract).
|
||||
*
|
||||
* Ordering: called from `performInit()` immediately after
|
||||
* `metadataIndex.init()` has computed the verdict against the generation
|
||||
* store's now-FINAL committed generation, and BEFORE `rebuildIndexesIfNeeded()`
|
||||
* (the open-time rebuild gate) or any read serves — so a caller can never
|
||||
* observe the pre-catchup state.
|
||||
*
|
||||
* @param alreadyRebuilt - `true` when crash recovery just rebuilt every
|
||||
* index from canonical (rolled-back uncommitted transactions) — the
|
||||
* verdict's prescribed action is redundant with what already ran (a
|
||||
* fresh canonical walk supersedes any catchup fold or rescan), so it is
|
||||
* skipped, narrated, rather than duplicating the work.
|
||||
*/
|
||||
private async consumeMetadataWatermarkVerdict(alreadyRebuilt: boolean): Promise<void> {
|
||||
if (!(this.metadataIndex instanceof MetadataIndexManager)) return
|
||||
const verdict = this.metadataIndex.watermarkVerdict()
|
||||
if (verdict === null || verdict === 'adopt') return
|
||||
|
||||
if (alreadyRebuilt) {
|
||||
prodLog.info(
|
||||
`[Brainy] metadata index watermark verdict '${verdict}' at open — skipped: crash ` +
|
||||
`recovery already rebuilt every index from canonical this open.`
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
const window = this.metadataIndex.watermarkGap()
|
||||
// A genuine first boot (no persisted artifact at all) verdicts 'rescan'
|
||||
// too — same as a real unverifiable artifact — but it is routine, not
|
||||
// alarming: narrate it at info level instead of warn (mirrors the
|
||||
// manager's own internal distinction in loadWatermarkVerdict()).
|
||||
const firstBoot = verdict === 'rescan' && !this.metadataIndex.watermarkArtifactPresent()
|
||||
const preNarrate = firstBoot ? prodLog.info.bind(prodLog) : prodLog.warn.bind(prodLog)
|
||||
preNarrate(
|
||||
verdict === 'catchup' && window
|
||||
? `[Brainy] metadata index watermark verdict: CATCHUP — folding generations ` +
|
||||
`(${window.from}, ${window.to}] from the fact log before this open serves reads.`
|
||||
: firstBoot
|
||||
? `[Brainy] metadata index watermark verdict: rescan (no persisted artifact — first ` +
|
||||
`boot; the rebuild below is a trivial no-op walk).`
|
||||
: `[Brainy] metadata index watermark verdict: RESCAN — the persisted artifact is ` +
|
||||
`unverifiable (unstamped, or ahead of the store's committed generation); ` +
|
||||
`forcing a full rebuild from canonical at open.`
|
||||
)
|
||||
|
||||
const scan = window
|
||||
? this.scanFacts({ fromGeneration: window.from + 1, toGeneration: window.to })
|
||||
: null
|
||||
const result = await this.metadataIndex.applyWatermarkCatchup(scan)
|
||||
|
||||
if (result.action === 'rescan') {
|
||||
const postNarrate = firstBoot ? prodLog.debug.bind(prodLog) : prodLog.warn.bind(prodLog)
|
||||
postNarrate(
|
||||
`[Brainy] metadata index catchup demoted to a full rebuild` +
|
||||
`${result.reason ? ` — ${result.reason}` : ''}.`
|
||||
)
|
||||
} else if (result.action === 'caught-up') {
|
||||
prodLog.warn(
|
||||
`[Brainy] metadata index catchup complete: ${result.factsApplied} fact(s) folded ` +
|
||||
`(${result.nounsApplied} noun op(s), ${result.verbsApplied} verb op(s)) — index now ` +
|
||||
`reflects generation ${result.window?.to}.`
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description B3 Deliverable 3 — THE ONLINE METADATA REBUILD.
|
||||
* `repairIndex()`'s ceremony door for the `'metadata'` family routes here
|
||||
* instead of calling `MetadataIndexManager.rebuild()` directly: build a
|
||||
* FRESH replacement manager BESIDE the live one (same storage, same
|
||||
* idMapper — identity is shared, never a second mapper), walk canonical
|
||||
* into it while every live write during the build ALSO mirrors there
|
||||
* (`MetadataIndexManager.beginShadow`), fold the generation window the
|
||||
* walk may have read stale, then atomically swap this brain's reference —
|
||||
* `this.metadataIndex` points at the OLD manager for the ENTIRE build, so
|
||||
* every read in progress (and every read that starts before the swap
|
||||
* line executes) keeps serving its full, unbuilt-adjacent population;
|
||||
* nothing ever observes a half-built index.
|
||||
*
|
||||
* PERSISTENCE CHOICE (named per the B3 brief): the JS manager's persisted
|
||||
* keys (field-index chunks, column-store segments, the watermark stamp,
|
||||
* the id-mapper record) are GLOBAL per storage — not namespaced per
|
||||
* manager instance — so two managers cannot safely persist independently
|
||||
* mid-build (a segment-number race, a stamp race, an id-mapper reload
|
||||
* that would discard the live manager's not-yet-flushed assignments —
|
||||
* see `MetadataIndexManager.initForShadowBuild`'s JSDoc for the id-mapper
|
||||
* hazard specifically). This build therefore PERSISTS ONLY AT SWAP: the
|
||||
* shadow builds entirely in memory (`rebuild({ inMemoryOnly: true })` +
|
||||
* a fact-log fold — neither touches storage) and flushes exactly once,
|
||||
* after the swap, as the sole owner of the shared keys.
|
||||
*
|
||||
* FALLBACK: a store with no fact log (or a non-JS/native metadata
|
||||
* provider — its own train owns its online-rebuild strategy) cannot
|
||||
* safely bound "what landed during the walk"; this method falls back to
|
||||
* the ORIGINAL blocking clear-then-walk `rebuild()`, narrated.
|
||||
*/
|
||||
private async rebuildMetadataIndexOnline(): Promise<void> {
|
||||
if (!(this.metadataIndex instanceof MetadataIndexManager)) {
|
||||
// A registered provider (e.g. a native accelerator) may replace
|
||||
// `this.metadataIndex` with a non-MetadataIndexManager object at
|
||||
// runtime even though the field's declared type is the JS class —
|
||||
// the cast mirrors the same reach-in used elsewhere in this file
|
||||
// (e.g. checkHealth()'s `metadataProvider` locals) for exactly this.
|
||||
const provider = this.metadataIndex as unknown as MetadataIndexProvider
|
||||
await provider.rebuild()
|
||||
return
|
||||
}
|
||||
|
||||
const committedAtStart = this.storage.committedGeneration?.() ?? null
|
||||
const factLogAvailable = committedAtStart !== null && this.scanFacts() !== null
|
||||
if (!factLogAvailable) {
|
||||
prodLog.warn(
|
||||
`[Brainy] repairIndex(): metadata rebuild — no fact log on this store, build-beside ` +
|
||||
`is unavailable; falling back to the blocking rebuild (reads may serve a ` +
|
||||
`partially-built index for its duration).`
|
||||
)
|
||||
await this.metadataIndex.rebuild()
|
||||
return
|
||||
}
|
||||
|
||||
prodLog.warn(
|
||||
`[Brainy] repairIndex(): metadata rebuild — building a fresh replacement index BESIDE ` +
|
||||
`the live one (reads keep serving the current index throughout); swapping in ` +
|
||||
`atomically once it is caught up.`
|
||||
)
|
||||
const startedAt = Date.now()
|
||||
const oldManager = this.metadataIndex
|
||||
const shadow = new MetadataIndexManager(this.storage, {}, {
|
||||
entityIdMapper: oldManager.getIdMapper()
|
||||
})
|
||||
|
||||
oldManager.beginShadow(shadow)
|
||||
let committedAtSwap: number
|
||||
try {
|
||||
await shadow.buildBeside(committedAtStart!)
|
||||
// Capture the true final generation right before the swap — a
|
||||
// synchronous read, no `await` between here and the reference
|
||||
// assignment below, so nothing can land ungoverned in the gap: the
|
||||
// shadow has been live-mirroring every write since beginShadow()
|
||||
// above, and this generation is the floor a FUTURE open's watermark
|
||||
// verdict will trust once stamped.
|
||||
committedAtSwap = this.storage.committedGeneration?.() ?? committedAtStart!
|
||||
} catch (err) {
|
||||
oldManager.endShadow()
|
||||
prodLog.error(
|
||||
`[Brainy] repairIndex(): online metadata rebuild FAILED during the walk/fold — the ` +
|
||||
`live index is UNCHANGED (never swapped); reads keep serving the current ` +
|
||||
`(pre-rebuild) metadata index. Error: ${(err as Error).message}`
|
||||
)
|
||||
throw err
|
||||
}
|
||||
|
||||
oldManager.endShadow()
|
||||
this.metadataIndex = shadow
|
||||
|
||||
// NOW persist — the shadow is the SOLE owner of the shared storage keys
|
||||
// (nothing references `oldManager` any more; it never flushes again).
|
||||
shadow.stampWatermark(committedAtSwap)
|
||||
await shadow.flush()
|
||||
|
||||
prodLog.warn(
|
||||
`[Brainy] repairIndex(): online metadata rebuild complete in ${Date.now() - startedAt}ms — ` +
|
||||
`swapped in a fresh index reflecting generation ${committedAtSwap}, zero read downtime.`
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Rebuild indexes from persisted data if needed — THE OPEN-TIME
|
||||
* BUILD. Called once per open (init calls it; `repairIndex()`'s
|
||||
|
|
@ -17189,7 +17493,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
`[Brainy] repairIndex(): explicit rebuild requested for '${familyName}' — ` +
|
||||
`rebuilding unconditionally (no invariant consulted).`
|
||||
)
|
||||
await p.rebuild()
|
||||
// The metadata family routes through the online build-beside
|
||||
// orchestrator (B3 D3) instead of the provider's own rebuild() —
|
||||
// zero read downtime when a fact log is available, narrated
|
||||
// fallback to the blocking rebuild() otherwise.
|
||||
if (familyName === 'metadata') {
|
||||
await this.rebuildMetadataIndexOnline()
|
||||
} else {
|
||||
await p.rebuild()
|
||||
}
|
||||
record(`provider:${familyName}`, {
|
||||
checked: true,
|
||||
healed: 1,
|
||||
|
|
@ -17230,7 +17542,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
`[Brainy] repairIndex(): provider '${report.provider}' has a failing invariant ` +
|
||||
`requiring a rebuild — reconciling its derived state from canonical.`
|
||||
)
|
||||
await p.rebuild()
|
||||
// See the explicit-rebuild branch above: 'metadata' routes through
|
||||
// the online build-beside orchestrator (B3 D3).
|
||||
if (familyName === 'metadata') {
|
||||
await this.rebuildMetadataIndexOnline()
|
||||
} else {
|
||||
await p.rebuild()
|
||||
}
|
||||
} else {
|
||||
record(`provider:${report.provider}`, {
|
||||
checked: true, healed: 0,
|
||||
|
|
@ -17821,6 +18139,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
await this.autoCompactHistory()
|
||||
|
||||
// Watermark stamps ride this flush too — see stampProjectionWatermarks().
|
||||
// Read-only instances skip it (no writes, no committed-generation drift
|
||||
// to certify; ensureInitialized()'s guard below never runs for them
|
||||
// either, so this must not assume a writer's invariants).
|
||||
if (!this.isReadOnly) {
|
||||
this.stampProjectionWatermarks()
|
||||
}
|
||||
|
||||
// Phase 1: Flush ALL components in parallel to persist buffered data
|
||||
// This is critical when cor native providers buffer data in Rust memory
|
||||
await Promise.all([
|
||||
|
|
|
|||
Reference in a new issue