fix: warm() metadata surface routes through the active provider (warm hook added to the metadata contract); add maintenanceDebt() observability surface
A production deployment's warm report showed metadata: 'unavailable' under a native metadata provider. brain.warm()'s metadata leg only duck-typed the built-in JS manager's hydrateAll() method, which a native provider has no reason to implement. - MetadataIndexProvider (src/plugin.ts) gains an optional warm?(): Promise<void> hook, mirroring the existing vector and graph provider hooks. brain.warm() now checks the active provider's own warm() FIRST, falls back to the JS manager's hydrateAll() when absent, and reports 'unavailable' only when neither exists -- never init() as a stand-in, since a native provider's init() may be a cheap verify rather than a real warm. - Tests (tests/unit/brainy/warm.test.ts): a live provider instance shaped to have warm() reports 'warmed' and the hook called with no hydrateAll fallback; shaped to have neither hook reports 'unavailable' (pins the honest branch); the unmodified built-in JS manager still reports 'warmed' via hydrateAll(), unchanged. Additive scope agreed mid-flight with the native-provider team: a maintenance-debt observability seam so an operator sees a grind coming instead of discovering it as a CPU storm. - New optional maintenanceDebt?(): Promise<ProviderMaintenanceDebt> hook on all three provider contracts (vector, metadata, graph -- the same three warm?() lives on). ProviderMaintenanceDebt is fields-all-optional: a provider reports only what it truly measures (pendingBytes, pendingItems, lastPassCompletedAt, lastPassOutcome, converging), never an estimate dressed as fact. - New public brain.maintenanceDebt(): a pure passthrough -- for each surface it calls only the active provider's own hook and reports the payload verbatim, or 'unavailable' when absent. No thresholds, no polling, no JS-side estimation; the provider owns the numbers, the operator owns the policy. - ProviderMaintenanceDebt, MaintenanceDebtReport, and MaintenanceDebtOutcome are exported from the package root. - Tests (tests/unit/brainy/maintenance-debt.test.ts): hook present reports 'reported' with the exact payload passed through; hook absent reports 'unavailable' on every surface; mixed surfaces resolve independently of each other. RELEASES.md gains the 8.10.1 entry covering both fixes above and this feature, including the no-hot-retry contract from the prior commit.
This commit is contained in:
parent
003e2a74ea
commit
5b2cbf74e5
6 changed files with 471 additions and 6 deletions
112
src/brainy.ts
112
src/brainy.ts
|
|
@ -65,7 +65,8 @@ import type {
|
|||
OpaqueIdSet,
|
||||
AtGenerationVectors,
|
||||
VectorIndexProvider,
|
||||
GraphIndexProvider
|
||||
GraphIndexProvider,
|
||||
ProviderMaintenanceDebt
|
||||
} from './plugin.js'
|
||||
import type {
|
||||
BrainyPlugin,
|
||||
|
|
@ -424,6 +425,30 @@ export interface WarmReport {
|
|||
totalDurationMs: number
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Result of {@link Brainy.maintenanceDebt}: one outcome per
|
||||
* index surface, mirroring {@link WarmReport}'s shape.
|
||||
* - `'reported'` — the active provider for this surface implements
|
||||
* `maintenanceDebt?()` and its {@link ProviderMaintenanceDebt} payload is
|
||||
* attached verbatim under `debt`.
|
||||
* - `'unavailable'` — the active provider does not implement the hook, so
|
||||
* nothing is known; brainy never estimates or infers a payload on its
|
||||
* behalf.
|
||||
*/
|
||||
export type MaintenanceDebtOutcome = 'reported' | 'unavailable'
|
||||
|
||||
/**
|
||||
* @description Per-surface result of {@link Brainy.maintenanceDebt}. Brainy
|
||||
* performs no thresholding, polling, or estimation over this data — it is a
|
||||
* pure passthrough of each active provider's own self-report (the provider
|
||||
* owns the numbers; the operator owns the policy).
|
||||
*/
|
||||
export interface MaintenanceDebtReport {
|
||||
vector: { outcome: MaintenanceDebtOutcome; debt?: ProviderMaintenanceDebt }
|
||||
metadata: { outcome: MaintenanceDebtOutcome; debt?: ProviderMaintenanceDebt }
|
||||
graph: { outcome: MaintenanceDebtOutcome; debt?: ProviderMaintenanceDebt }
|
||||
}
|
||||
|
||||
/**
|
||||
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
|
||||
* Within the window, queries rethrow the recorded failure instantly (loud,
|
||||
|
|
@ -14313,9 +14338,16 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* *some* backing storage as a side effect but is reported honestly as
|
||||
* `'probed'`, never `'warmed'`. An empty index or unknown dimension has
|
||||
* nothing to probe (`'unavailable'`).
|
||||
* - **Metadata**: full hydration — every persisted field's sparse index is
|
||||
* loaded from storage (`MetadataIndexManager.hydrateAll()`), not just the
|
||||
* heuristic common-fields subset `init()` warms.
|
||||
* - **Metadata**: calls the provider's own `warm?()` when the active
|
||||
* `'metadataIndex'` provider implements it (`'warmed'`) — the seam a
|
||||
* native metadata provider lights up so it is not duck-typed against the
|
||||
* JS manager's method. Otherwise falls back to full hydration on the
|
||||
* built-in JS manager — every persisted field's sparse index is loaded
|
||||
* from storage (`MetadataIndexManager.hydrateAll()`), not just the
|
||||
* heuristic common-fields subset `init()` warms — and reports `'warmed'`.
|
||||
* Neither seam present → `'unavailable'` (honest: `init()` is never used
|
||||
* as a substitute here, since a native provider's `init()` may be a
|
||||
* cheap verify rather than a real warm).
|
||||
* - **Graph**: calls the provider's own `warm?()` when the active graph
|
||||
* provider implements it; otherwise re-runs its existing eager cold-load
|
||||
* `init()` seam (idempotent — the JS adjacency index's `init()` already
|
||||
|
|
@ -14371,12 +14403,23 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// --- Metadata --------------------------------------------------------
|
||||
const metadataStart = Date.now()
|
||||
let metadataOutcome: WarmOutcome
|
||||
const metadataProvider = this.metadataIndex as unknown as MetadataIndexProvider
|
||||
const metadataWithHydrate = this.metadataIndex as unknown as { hydrateAll?: () => Promise<void> }
|
||||
if (typeof metadataWithHydrate.hydrateAll === 'function') {
|
||||
if (typeof metadataProvider.warm === 'function') {
|
||||
// Active provider (e.g. a native metadata index) declares its own warm
|
||||
// seam — route through it FIRST so a native provider's warmth is
|
||||
// reported honestly instead of being duck-typed against the JS
|
||||
// manager's hydrateAll(), which a native provider does not implement.
|
||||
await metadataProvider.warm()
|
||||
metadataOutcome = 'warmed'
|
||||
} else if (typeof metadataWithHydrate.hydrateAll === 'function') {
|
||||
// Built-in JS manager path — full sparse-index hydration.
|
||||
await metadataWithHydrate.hydrateAll()
|
||||
metadataOutcome = 'warmed'
|
||||
} else {
|
||||
// No hydration seam on this metadata provider — nothing to run.
|
||||
// No hydration seam on this metadata provider — nothing to run. (No
|
||||
// init() fallback here: init() on a native provider may be a cheap
|
||||
// verify, and reporting that as warmth would lie.)
|
||||
metadataOutcome = 'unavailable'
|
||||
}
|
||||
const metadataDurationMs = Date.now() - metadataStart
|
||||
|
|
@ -14410,6 +14453,63 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read each index surface's self-reported outstanding maintenance work —
|
||||
* the observability seam so an operator sees a grind coming (rising
|
||||
* pending bytes/items, a stalled background pass) instead of discovering
|
||||
* it as a CPU storm or a transaction blowing its budget mid-flight (see
|
||||
* {@link TransactionTimeoutError}).
|
||||
*
|
||||
* PURE PASSTHROUGH: for each of vector/metadata/graph, this calls ONLY the
|
||||
* ACTIVE provider's own `maintenanceDebt?()` hook (the same per-surface
|
||||
* provider resolution {@link Brainy.warm} uses) and reports its
|
||||
* {@link ProviderMaintenanceDebt} payload verbatim. There is no JS-side
|
||||
* fallback computation, no threshold evaluation, and no polling — brainy
|
||||
* surfaces the truth the provider measured; the provider owns the numbers
|
||||
* and the operator owns the policy (what threshold matters, what action to
|
||||
* take). A surface whose active provider does not implement the hook
|
||||
* reports `'unavailable'` — never a guessed or zeroed payload.
|
||||
*
|
||||
* @returns A {@link MaintenanceDebtReport}: per-surface outcome + payload.
|
||||
* @example
|
||||
* ```typescript
|
||||
* const debt = await brain.maintenanceDebt()
|
||||
* if (debt.metadata.outcome === 'reported' && debt.metadata.debt?.pendingBytes) {
|
||||
* console.log('metadata pending bytes:', debt.metadata.debt.pendingBytes)
|
||||
* }
|
||||
* ```
|
||||
*/
|
||||
async maintenanceDebt(): Promise<MaintenanceDebtReport> {
|
||||
await this.ensureInitialized({ needs: ['vector', 'metadata', 'graph'] })
|
||||
|
||||
// --- Vector ---------------------------------------------------------
|
||||
const vectorProvider = this.index as VectorIndexProvider & {
|
||||
maintenanceDebt?: () => Promise<ProviderMaintenanceDebt>
|
||||
}
|
||||
const vector =
|
||||
typeof vectorProvider.maintenanceDebt === 'function'
|
||||
? { outcome: 'reported' as const, debt: await vectorProvider.maintenanceDebt() }
|
||||
: { outcome: 'unavailable' as const }
|
||||
|
||||
// --- Metadata --------------------------------------------------------
|
||||
const metadataProvider = this.metadataIndex as unknown as MetadataIndexProvider
|
||||
const metadata =
|
||||
typeof metadataProvider.maintenanceDebt === 'function'
|
||||
? { outcome: 'reported' as const, debt: await metadataProvider.maintenanceDebt() }
|
||||
: { outcome: 'unavailable' as const }
|
||||
|
||||
// --- Graph -------------------------------------------------------------
|
||||
const graphProvider = this.graphIndex as GraphIndexProvider & {
|
||||
maintenanceDebt?: () => Promise<ProviderMaintenanceDebt>
|
||||
}
|
||||
const graph =
|
||||
typeof graphProvider.maintenanceDebt === 'function'
|
||||
? { outcome: 'reported' as const, debt: await graphProvider.maintenanceDebt() }
|
||||
: { outcome: 'unavailable' as const }
|
||||
|
||||
return { vector, metadata, graph }
|
||||
}
|
||||
|
||||
/**
|
||||
* Explicitly warm up the embedding engine
|
||||
*
|
||||
|
|
|
|||
10
src/index.ts
10
src/index.ts
|
|
@ -31,6 +31,11 @@ export type { DiagnosticsResult } from './brainy.js'
|
|||
// brain.warm() — eager index/storage readiness report (per-surface honest
|
||||
// outcome + timing). See the WarmReport JSDoc in brainy.ts.
|
||||
export type { WarmReport, WarmOutcome } from './brainy.js'
|
||||
// brain.maintenanceDebt() — per-surface passthrough of each active
|
||||
// provider's self-reported background maintenance debt. See the
|
||||
// MaintenanceDebtReport JSDoc in brainy.ts and ProviderMaintenanceDebt in
|
||||
// plugin.ts for the measure-only-what-you-track contract.
|
||||
export type { MaintenanceDebtReport, MaintenanceDebtOutcome } from './brainy.js'
|
||||
export type {
|
||||
GraphAuditReport,
|
||||
GraphAuditDiscrepancy
|
||||
|
|
@ -227,6 +232,11 @@ export type { FamilyStamp, StampMembers, StampVerdict } from './db/familyStamp.j
|
|||
export { isVersionedIndexProvider } from './plugin.js'
|
||||
export type { VersionedIndexProvider } from './plugin.js'
|
||||
export type { ProviderInvariantReport, InvariantResult, InvariantHeal } from './plugin.js'
|
||||
// Optional provider self-report of outstanding background maintenance work
|
||||
// (compaction, deferred writes, etc.) — the payload type for
|
||||
// brain.maintenanceDebt(). See the measure-only-what-you-track contract on
|
||||
// ProviderMaintenanceDebt in plugin.ts.
|
||||
export type { ProviderMaintenanceDebt } from './plugin.js'
|
||||
// Optional native graph-acceleration engine (cor 3.0) — the published provider
|
||||
// contract + its columnar wire types. Brainy feature-detects an implementation
|
||||
// and falls back to its pure-TS adjacency when absent.
|
||||
|
|
|
|||
|
|
@ -171,6 +171,37 @@ export interface ProviderInvariantReport {
|
|||
durationMs: number
|
||||
}
|
||||
|
||||
/**
|
||||
* @description A provider's self-report of its own outstanding background
|
||||
* maintenance work (compaction, deferred writes, a build-new→verify→swap in
|
||||
* flight, etc.) — the observability seam so an operator sees a grind coming
|
||||
* (rising pending bytes/items, a stalled pass) instead of discovering it as a
|
||||
* CPU storm or a timeout under transaction budget pressure. Every field is
|
||||
* OPTIONAL and every field is a MEASUREMENT: a provider reports ONLY what it
|
||||
* actually tracks, never an estimate dressed up as a fact. Absence of the
|
||||
* {@link VectorIndexProvider.maintenanceDebt} /
|
||||
* {@link GraphIndexProvider.maintenanceDebt} /
|
||||
* {@link MetadataIndexProvider.maintenanceDebt} hook itself means the
|
||||
* provider does not track debt at all — brainy reports that surface
|
||||
* `'unavailable'` rather than inventing zeros. Brainy performs NO threshold
|
||||
* checks, NO polling, and NO JS-side estimation over this payload — it is a
|
||||
* pure passthrough via {@link Brainy.maintenanceDebt}; the provider owns the
|
||||
* numbers and the operator owns the policy (what threshold matters, what to
|
||||
* do about it).
|
||||
*/
|
||||
export interface ProviderMaintenanceDebt {
|
||||
/** Bytes of outstanding/unmerged work, if the provider measures it (e.g. unflushed writes, unmerged segments). */
|
||||
pendingBytes?: number
|
||||
/** Count of outstanding items (records, segments, nodes) awaiting the provider's background pass. */
|
||||
pendingItems?: number
|
||||
/** Epoch millis when the provider's last maintenance pass finished, if it tracks one. */
|
||||
lastPassCompletedAt?: number
|
||||
/** How the last pass ended, if the provider tracks pass outcomes. */
|
||||
lastPassOutcome?: 'completed' | 'partial' | 'failed'
|
||||
/** `true` if the provider's own measurements show debt trending down (making progress); `false` if flat or growing; omitted if the provider can't tell. */
|
||||
converging?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* The `'metadataIndex'` provider — a drop-in for `MetadataIndexManager`.
|
||||
* Brainy calls this surface via `this.metadataIndex.*` (see `brainy.ts`) and
|
||||
|
|
@ -181,6 +212,34 @@ export interface MetadataIndexProvider {
|
|||
flush(): Promise<void>
|
||||
rebuild(): Promise<void>
|
||||
|
||||
/**
|
||||
* @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
|
||||
* pretouch, full sparse-index hydration) so first queries run at
|
||||
* steady-state cost. Optional; absence means the provider demand-loads.
|
||||
* Mirrors {@link GraphIndexProvider.warm} / the vector provider's `warm?()`
|
||||
* (`src/plugin.ts` VectorIndexProvider). Distinct from `init()`: `init` is
|
||||
* required and runs once automatically during brain startup; `warm` is a
|
||||
* separate, explicit step a caller opts into via `brain.warm()` (or
|
||||
* `warmOnOpen`) to pre-pay demand-load cost `init` left lazy. Idempotent —
|
||||
* calling it more than once must be safe and cheap on a brain that is
|
||||
* already warm. A provider that already loads everything eagerly in
|
||||
* `init()` may implement `warm` as a no-op or omit it — `brain.warm()`
|
||||
* falls back to the built-in JS manager's `hydrateAll()` duck-type when
|
||||
* absent, and to an honest `'unavailable'` when neither exists.
|
||||
*/
|
||||
warm?(): Promise<void>
|
||||
|
||||
/**
|
||||
* @description OPTIONAL self-reported {@link ProviderMaintenanceDebt} —
|
||||
* the observability seam so an operator sees outstanding background
|
||||
* maintenance work (e.g. unmerged postings) BEFORE it grinds a transaction
|
||||
* into a budget-busting op. Absence means this provider does not track
|
||||
* debt; `brain.maintenanceDebt()` reports this surface `'unavailable'`
|
||||
* rather than guessing. See {@link ProviderMaintenanceDebt} for the
|
||||
* measure-only-what-you-track contract.
|
||||
*/
|
||||
maintenanceDebt?(): Promise<ProviderMaintenanceDebt>
|
||||
|
||||
/**
|
||||
* @description OPTIONAL honest durability signal (readiness contract,
|
||||
* mirrors `isReady?()` on the graph and vector providers). `true` ⇔ the
|
||||
|
|
@ -395,6 +454,18 @@ export interface GraphIndexProvider {
|
|||
*/
|
||||
warm?(): Promise<void>
|
||||
|
||||
/**
|
||||
* @description OPTIONAL self-reported {@link ProviderMaintenanceDebt} —
|
||||
* the observability seam so an operator sees outstanding background
|
||||
* maintenance work (e.g. a build-new→verify→swap in flight, unmerged
|
||||
* adjacency segments) BEFORE it grinds a transaction into a
|
||||
* budget-busting op. Absence means this provider does not track debt;
|
||||
* `brain.maintenanceDebt()` reports this surface `'unavailable'` rather
|
||||
* than guessing. See {@link ProviderMaintenanceDebt} for the
|
||||
* measure-only-what-you-track contract.
|
||||
*/
|
||||
maintenanceDebt?(): Promise<ProviderMaintenanceDebt>
|
||||
|
||||
/**
|
||||
* @description OPTIONAL. A native provider returns true from the moment its
|
||||
* `init()` detects a large epoch-drift until its background
|
||||
|
|
@ -1057,6 +1128,18 @@ export interface VectorIndexProvider {
|
|||
*/
|
||||
warm?(): Promise<void>
|
||||
|
||||
/**
|
||||
* @description OPTIONAL self-reported {@link ProviderMaintenanceDebt} —
|
||||
* the observability seam so an operator sees outstanding background
|
||||
* maintenance work (e.g. unflushed writes, a pending rebuild) BEFORE it
|
||||
* grinds a transaction into a budget-busting op. Absence means this
|
||||
* provider does not track debt; `brain.maintenanceDebt()` reports this
|
||||
* surface `'unavailable'` rather than guessing. See
|
||||
* {@link ProviderMaintenanceDebt} for the measure-only-what-you-track
|
||||
* contract.
|
||||
*/
|
||||
maintenanceDebt?(): Promise<ProviderMaintenanceDebt>
|
||||
|
||||
/**
|
||||
* @description OPTIONAL honest durability signal (readiness contract,
|
||||
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue