soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: honest index readiness — no silently-empty queries on a cold index

Browse source 
Closes the "dishonest readiness proxy" anti-pattern (Pattern A): size()>0 /
isInitialized were treated as "this index serves queries", but a cold native
index can load its COUNT before its SERVING structure, so a query returned a
silent [] indistinguishable from "no such data". A shared assessIndexReadiness()
now reads only the provider's honest isReady() signal (never size()), applied at
every site:

- Vector: a one-shot verifyVectorLive() guard on the semantic/proximity search
  path (a pure semantic find({query}) has no filter, so nothing guarded it). It
  prefers isReady(), else a known-vector self-match probe; self-heals via rebuild
  or throws the new VectorIndexNotReadyError instead of a silent [].
- Graph: getVerbsBySource/ByTarget skip the fast path when the provider reports
  not-ready (falling to the canonical shard scan), plus a one-shot probe that
  self-heals a no-isReady provider whose adjacency did not cold-load.
- getIndexStatus(): folds in per-index honest `ready` (making `populated`
  honest) + rebuildFailed/rebuildError/degradedIds, so a readiness probe never
  200s a brain that is still warming up or degraded.

Unblocked by the native providers now reporting serving-truth (graph via
SSTable-residency readiness, vector via durableBaseLoadFailed). New export:
VectorIndexNotReadyError. getIndexStatus gains additive fields. No breaking API.
13 new tests; existing readiness guards green.
This commit is contained in:
David Snelling 2026-07-13 13:08:52 -07:00
parent 36d4e80ba2
commit d0f69c731f
9 changed files with 654 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

215
src/brainy.ts
View file

@ -175,7 +175,8 @@ import {
} from './events/changeFeed.js'
import { isDeterministicEmbedMode } from './embeddings/deterministicEmbedMode.js'
import { GenerationConflictError, StoreInconsistentError } from './db/errors.js'
import { BrainyError, GraphIndexNotReadyError, MetadataIndexNotReadyError, MigrationInProgressError } from './errors/brainyError.js'
import { BrainyError, GraphIndexNotReadyError, MetadataIndexNotReadyError, MigrationInProgressError, VectorIndexNotReadyError } from './errors/brainyError.js'
import { assessIndexReadiness } from './utils/indexReadiness.js'
import { MemoryStorage } from './storage/adapters/memoryStorage.js'
import type {
CompactHistoryOptions,
@ -500,6 +501,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private _metadataVerified = false
/** Re-entrancy guard for {@link verifyMetadataLive}. */
private _metadataVerifying = false
/** Vector-index cold-read guard: verified-serving this session (one-shot). */
private _vectorVerified = false
/** Re-entrancy guard for {@link verifyVectorLive}. */
private _vectorVerifying = false
/**
* Coordinated migration LOCK (#18): dedup guards so the "upgrading, blocking"
* and "upgrade complete, resumed" lines each log once per migration window,
@ -3523,6 +3528,142 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return null
}
/**
* @description The vector-index counterpart of {@link verifyGraphAdjacencyLive}
* / {@link verifyMetadataLive}. On a cold open a native vector provider can
* report a non-zero `size()` (its persisted COUNT loaded) yet not have loaded
* its serving structure (the mmap/DiskANN graph) — so a pure semantic
* `find({ query })` silently returns `[]`. A pure semantic query has
* `hasFilterCriteria === false`, so the metadata guard never fires; this guard
* closes that gap. Run one-shot on the first vector/proximity search:
* - **Preferred (honest signal):** the provider exposes `isReady()`. `false`
* → rebuild from storage, re-check; if still `false`, throw
* {@link VectorIndexNotReadyError} rather than serving `[]`.
* - **Fallback (no `isReady()`):** a KNOWN persisted vector (sampled +
* hydrated) is searched against the index; if it does not self-match, the
* serving structure did not load — rebuild + re-probe, else throw.
* Inconclusive cases (empty store, no probeable vector, `size()===0` — where
* the JS baseline's cold load is `ensureIndexesLoaded`'s job) are treated as
* live: never a false rebuild. A migrating provider is skipped (it owns its
* locked rebuild).
* @returns `'live'` when the index serves, `'rebuilt'` when a rebuild restored it.
*/
private async verifyVectorLive(): Promise<'live' | 'rebuilt'> {
if (this._vectorVerified) return 'live'
// Migration LOCK (#18): a migrating provider owns its in-place rebuild.
if (this.providerIsMigrating(this.index)) return 'live'
// Re-entrancy: rebuild() can trigger reads that call back into this guard.
if (this._vectorVerifying) return 'live'
this._vectorVerifying = true
try {
// ── Strategy 1: honest isReady() signal (native provider) ──────────────
const readiness = assessIndexReadiness(this.index)
if (readiness !== 'unknown') {
if (readiness === 'ready') {
this._vectorVerified = true
return 'live'
}
// Not ready: the serving structure did not load on open. Rebuild.
if (!this.config.silent) {
console.warn(
`[Brainy] Vector index reports not-ready (isReady() === false) — the persisted ` +
`vector index did not load on open. Rebuilding from storage…`
)
}
await this.index.rebuild()
if (assessIndexReadiness(this.index) === 'ready') {
this._vectorVerified = true
return 'rebuilt'
}
throw new VectorIndexNotReadyError(
`Vector index reports not-ready even after a rebuild — semantic find({ query }) and ` +
`proximity search cannot be served reliably for this brain (a silent empty result ` +
`would misrepresent existing data).`
)
}
// ── Strategy 2: known-vector probe (providers without isReady()) ───────
const claimed = this.index.size()
if (!claimed || claimed <= 0) return 'live' // JS cold path is ensureIndexesLoaded's job
const probe = await this.pickVectorProbe()
if (!probe) {
// Empty store, or nothing with a probeable vector — inconclusive.
this._vectorVerified = true
return 'live'
}
const p = probe
const probeServes = async (): Promise<boolean> => {
// The failure mode we guard is the SILENT EMPTY result: a cold index that
// loaded its COUNT but not its serving structure returns `[]` for a
// known-present vector, while a warm index returns at least one hit. We
// check for a NON-EMPTY result, NOT an exact self-match — HNSW is
// approximate and `get()` may return a re-hydrated/normalized vector, so
// demanding the exact self as top-1 would false-positive on a perfectly
// healthy index (and wrongly rebuild → throw).
const hits = await this.index.search(p.vector, 1)
return hits.length > 0
}
void p.id // probe keyed on the vector; id retained for diagnostics only
if (await probeServes()) {
this._vectorVerified = true
return 'live' // serving structure is live — the common case
}
if (!this.config.silent) {
console.warn(
`[Brainy] Vector index reports ${claimed} vector(s) but a known persisted vector ` +
`returns no results — the serving structure did not load on open. Rebuilding…`
)
}
await this.index.rebuild()
if (await probeServes()) {
this._vectorVerified = true
return 'rebuilt'
}
throw new VectorIndexNotReadyError(
`Vector index reports ${claimed} vector(s) but a known persisted vector returns no ` +
`results even after a rebuild — semantic find({ query }) cannot be served reliably ` +
`for this brain (a silent empty result would misrepresent existing data).`
)
} catch (err) {
if (err instanceof VectorIndexNotReadyError) throw err
// A transient probe/rebuild failure must not break the query NOR mask as
// "no data". Allow a re-check on the next vector read and fall through.
this._vectorVerified = false
if (!this.config.silent) {
console.warn(`[Brainy] Vector consistency check skipped (transient): ${err}`)
}
return 'live'
} finally {
this._vectorVerifying = false
}
}
/**
* @description Sample a KNOWN persisted noun and hydrate its vector, to probe
* the vector index with. `get()` omits vectors by default, so this passes
* `{ includeVectors: true }`. Samples a few (a system-only / vectorless entity
* must not make every open inconclusive). Returns `null` when nothing has a
* probeable vector.
*/
private async pickVectorProbe(): Promise<{ id: string; vector: number[] } | null> {
const sample = await this.storage.getNouns({ pagination: { limit: 5, offset: 0 } })
for (const noun of sample.items ?? []) {
const id = (noun as { id?: string }).id
if (!id) continue
const full = await this.get(id, { includeVectors: true })
const vector = (full as { vector?: number[] } | null)?.vector
if (Array.isArray(vector) && vector.length > 0) {
return { id, vector }
}
}
return null
}
// -------------------------------------------------------------------------
/**
@ -10175,17 +10316,32 @@ export class Brainy<T = any> implements BrainyInterface<T> {
migrating: boolean
/** Structured progress while {@link migrating}; absent otherwise. */
migration?: MigrationProgress
/** `true` when a non-fatal index rebuild failed at init() (degraded — queries
* may be incomplete). Folds in the same `_indexRebuildFailed` signal that
* {@link validateIndexConsistency} / {@link checkHealth} already expose. */
rebuildFailed: boolean
/** The rebuild failure message when {@link rebuildFailed}; absent otherwise. */
rebuildError?: string
/** Count of records committed via adopt-forward failed-rollback recovery whose
* derived index may be incomplete until repairIndex() (the 8.2.6 degraded
* set). Non-zero = degraded — a readiness probe should not report 200-ready. */
degradedIds: number
hnswIndex: {
size: number
/** Honest "serving" — the provider's `isReady()` when exposed, else `size>0`. */
populated: boolean
/** The provider's honest `isReady()` signal; absent when unexposed. */
ready?: boolean
}
metadataIndex: {
entries: number
populated: boolean
ready?: boolean
}
graphIndex: {
relationships: number
populated: boolean
ready?: boolean
}
storage: {
totalEntities: number
@ -10200,6 +10356,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
lazyRebuildCompleted: this.lazyRebuildCompleted,
disableAutoRebuild: this.config.disableAutoRebuild || false,
migrating: false,
rebuildFailed: this._indexRebuildFailed != null,
...(this._indexRebuildFailed ? { rebuildError: this._indexRebuildFailed.message } : {}),
degradedIds: this._indexDegradedIds.size,
hnswIndex: { size: 0, populated: false },
metadataIndex: { entries: 0, populated: false },
graphIndex: { relationships: 0, populated: false },
@ -10211,6 +10370,20 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const hnswSize = this.index.size()
const graphSize = await this.graphIndex.size()
// Honest readiness: when a provider exposes isReady(), it is the truth of
// whether the index actually SERVES (a native index can report a non-zero
// size/count yet not have loaded its serving structure). `populated` reflects
// that honest signal when present, falling back to size>0 only for providers
// that do not expose isReady(). Mirrors validateIndexConsistency/checkHealth,
// which already fold in the same signals.
const readyOf = (p: unknown): boolean | undefined => {
const r = assessIndexReadiness(p)
return r === 'unknown' ? undefined : r === 'ready'
}
const hnswReady = readyOf(this.index)
const metadataReady = readyOf(this.metadataIndex)
const graphReady = readyOf(this.graphIndex)
// Check storage entity count
let storageEntityCount = 0
try {
@ -10224,18 +10397,29 @@ export class Brainy<T = any> implements BrainyInterface<T> {
initialized: this.initialized,
lazyRebuildCompleted: this.lazyRebuildCompleted,
disableAutoRebuild: this.config.disableAutoRebuild || false,
// A non-fatal index-rebuild failure recorded at init(), or adopt-forward
// degraded ids, are degraded states (queries may be incomplete) — surface
// them here alongside the same signals validateIndexConsistency()/
// checkHealth() already expose, so a readiness probe never reports 200-ready
// over a known-degraded index.
rebuildFailed: this._indexRebuildFailed != null,
...(this._indexRebuildFailed ? { rebuildError: this._indexRebuildFailed.message } : {}),
degradedIds: this._indexDegradedIds.size,
...this.migrationSnapshot(),
hnswIndex: {
size: hnswSize,
populated: hnswSize > 0
populated: hnswReady ?? hnswSize > 0,
...(hnswReady !== undefined ? { ready: hnswReady } : {})
},
metadataIndex: {
entries: metadataStats.totalEntries,
populated: metadataStats.totalEntries > 0
populated: metadataReady ?? metadataStats.totalEntries > 0,
...(metadataReady !== undefined ? { ready: metadataReady } : {})
},
graphIndex: {
relationships: graphSize,
populated: graphSize > 0
populated: graphReady ?? graphSize > 0,
...(graphReady !== undefined ? { ready: graphReady } : {})
},
storage: {
totalEntities: storageEntityCount
@ -12863,6 +13047,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
candidateIds?: string[],
allowedIds?: OpaqueIdSet
): Promise<Result<T>[]> {
// Vector cold-read guard: before trusting a semantic/vector result, verify the
// vector index actually SERVES a known persisted vector (one-shot per brain).
// A pure semantic find({ query }) has no filter, so verifyMetadataLive never
// fires — a cold native index that loaded its COUNT but not its serving
// structure would return a silent []. This self-heals (rebuild) or throws
// VectorIndexNotReadyError instead.
await this.verifyVectorLive()
const vector = params.vector || (await this.embed(params.query!))
const limit = params.limit || 10
@ -12903,6 +13095,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private async executeProximitySearch(params: FindParams<T>): Promise<Result<T>[]> {
if (!params.near) return []
// Vector cold-read guard (see executeVectorSearch): proximity search also
// hits this.index.search — verify it serves before trusting an empty result.
await this.verifyVectorLive()
// Teaching error: without an anchor id the constraint is meaningless, and
// letting it fall through produces an opaque storage-layer sharding error.
if (!params.near.id) {
@ -14157,8 +14353,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return
}
// If indexes already populated, mark as complete and skip
if (this.index.size() > 0) {
// If indexes already populated AND honestly serving, mark complete and skip.
// Honest gate: when the provider exposes isReady(), that REPLACES the size()>0
// proxy (a native index can report a non-zero size while its serving structure
// is not loaded — the silent-empty cold-load class). A not-ready provider falls
// through so the rebuild path can load it; verifyVectorLive() is the query-time
// backstop either way. Providers without isReady() keep the size() heuristic
// (the JS index's size()>0 genuinely means loaded).
const vectorReadiness = assessIndexReadiness(this.index)
if (vectorReadiness === 'ready' || (vectorReadiness === 'unknown' && this.index.size() > 0)) {
this.lazyRebuildCompleted = true
return
}

27
src/errors/brainyError.ts
View file

@ -14,6 +14,7 @@ export type BrainyErrorType =
| 'FIELD_NOT_INDEXED'
| 'GRAPH_INDEX_NOT_READY'
| 'METADATA_INDEX_NOT_READY'
| 'VECTOR_INDEX_NOT_READY'
| 'MIGRATION_IN_PROGRESS'
/**
@ -280,6 +281,32 @@ export class MetadataIndexNotReadyError extends BrainyError {
}
}
/**
* Thrown when the vector index reports vectors (`size() > 0` or, on a native
* provider, `isReady() === false`) but cannot return a KNOWN persisted vector
* even after a rebuild — i.e. the semantic serving structure did not load on a
* cold open and could not be restored. The vector-search counterpart of
* {@link GraphIndexNotReadyError} / {@link MetadataIndexNotReadyError}: it
* replaces the silent-empty failure mode (a cold `find({ query })` returning
* `[]` indistinguishable from "no similar data") with a loud, catchable error,
* so a consumer never renders "nothing found" over data that is simply
* not-yet-warm.
*
* Detected once per brain by a known-vector serving probe on the first
* semantic / proximity `find()`; brainy self-heals (rebuilds the index from the
* canonical records) first and only raises this if the rebuild still cannot
* serve the known vector.
*/
export class VectorIndexNotReadyError extends BrainyError {
constructor(message: string, originalError?: Error) {
super(message, 'VECTOR_INDEX_NOT_READY', false, originalError)
this.name = 'VectorIndexNotReadyError'
if (Error.captureStackTrace) {
Error.captureStackTrace(this, VectorIndexNotReadyError)
}
}
}
/**
* Thrown when a data-plane read or write is issued against a brain that is
* running its one-time, automatic 7.x → 8.0 on-disk upgrade — the coordinated

2
src/index.ts
View file

@ -150,7 +150,7 @@ export { EntityNotFoundError, RelationNotFoundError } from './errors/notFound.js
// Base error + typed migration-lock error — thrown by any data-plane call while a
// brain runs its one-time 7.x→8.0 upgrade; catch to answer HTTP 503 + Retry-After.
export { BrainyError, MigrationInProgressError, GraphIndexNotReadyError, MetadataIndexNotReadyError } from './errors/brainyError.js'
export { BrainyError, MigrationInProgressError, GraphIndexNotReadyError, MetadataIndexNotReadyError, VectorIndexNotReadyError } from './errors/brainyError.js'
export type { BrainyErrorType } from './errors/brainyError.js'
// ============= 8.0 Db API — generational MVCC =============

79
src/storage/baseStorage.ts
View file

@ -5,6 +5,7 @@
import { GraphAdjacencyIndex } from '../graph/graphAdjacencyIndex.js'
import type { GraphEntityIdResolver } from '../graph/graphAdjacencyIndex.js'
import { assessIndexReadiness } from '../utils/indexReadiness.js'
import {
GraphVerb,
@ -248,6 +249,8 @@ function isCountedVisibility(visibility: unknown): boolean {
*/
export abstract class BaseStorage extends BaseStorageAdapter {
protected isInitialized = false
/** One-shot guard so the graph fast-path cold-load probe runs once per adapter. */
private _graphFastPathProbed = false
protected graphIndex?: GraphAdjacencyIndex
protected graphIndexPromise?: Promise<GraphAdjacencyIndex>
/**
@ -2681,6 +2684,61 @@ export abstract class BaseStorage extends BaseStorageAdapter {
return this.graphIndex
}
/**
* @description One-shot cold-load self-heal for a graph provider that does NOT
* expose the honest `isReady()` signal. A native provider wired via
* {@link setGraphIndex} bypasses `_initializeGraphIndex`'s `size()===0`
* self-heal, so a cold-open that loaded the COUNT but not the source→target
* adjacency would let the fast path return a silent `[]`. This probe (run once,
* before the fast path) samples ONE known persisted edge: if the index claims
* relationships (`size() > 0`) yet that edge's source resolves to no verb ints,
* the adjacency did not load → rebuild once from storage. Providers that expose
* `isReady()` are covered by the honest gate in
* getVerbsBy{Source,Target}_internal and skip this probe; the JS index (which
* self-heals in `_initializeGraphIndex`) resolves its known edge and no-ops.
*/
private async ensureGraphFastPathProbed(): Promise<void> {
if (this._graphFastPathProbed) return
const index = this.graphIndex
const resolver = this.graphEntityIdResolver
if (!index || !index.isInitialized || !resolver) return
// isReady()-capable providers report serving honestly — the gate handles them.
if (assessIndexReadiness(index) !== 'unknown') {
this._graphFastPathProbed = true
return
}
if (index.size() <= 0) {
this._graphFastPathProbed = true
return // no edges claimed — nothing to verify
}
try {
const sample = await this.getVerbs({ pagination: { limit: 1 } })
const verb = sample.items?.[0]
if (!verb || !verb.sourceId) {
this._graphFastPathProbed = true
return // no edges in storage — stale count, harmless
}
const sourceInt = resolver.getInt(verb.sourceId)
if (sourceInt === undefined) {
this._graphFastPathProbed = true
return // foreign / unmapped sample — inconclusive, never a false rebuild
}
const verbInts = await index.getVerbIdsBySource(BigInt(sourceInt))
if (verbInts.length === 0) {
prodLog.warn(
`[BaseStorage] Graph fast path: index reports ${index.size()} relationship(s) but a ` +
`known persisted edge resolves to none — the persisted adjacency did not load. ` +
`Rebuilding once from storage.`
)
await index.rebuild()
}
this._graphFastPathProbed = true
} catch (error) {
// Transient probe/rebuild failure must not break the read; re-arm for next call.
prodLog.debug(`[BaseStorage] Graph fast-path probe skipped (transient): ${error}`)
}
}
/**
* Clear all data from storage
* This method should be implemented by each specific adapter
@ -4199,13 +4257,22 @@ export abstract class BaseStorage extends BaseStorageAdapter {
sourceId: string
): Promise<HNSWVerbWithMetadata[]> {
await this.ensureInitialized()
await this.ensureGraphFastPathProbed()
prodLog.debug(`[BaseStorage] getVerbsBySource_internal: sourceId=${sourceId}, graphIndex=${!!this.graphIndex}, isInitialized=${this.graphIndex?.isInitialized}`)
// Fast path - use GraphAdjacencyIndex if available (lazy-loaded).
// 8.0 BigInt boundary: convert the UUID to an entity int up front and
// resolve returned verb ints back to verb-id strings.
if (this.graphIndex && this.graphIndex.isInitialized && this.graphEntityIdResolver) {
// Honest gate: a provider that exposes isReady() and reports not-ready
// (count/manifest loaded but source→target edges NOT) is SKIPPED so we fall
// to the correct-but-slower canonical shard scan below instead of a silent [].
if (
this.graphIndex &&
this.graphIndex.isInitialized &&
this.graphEntityIdResolver &&
assessIndexReadiness(this.graphIndex) !== 'not-ready'
) {
try {
const sourceInt = this.graphEntityIdResolver.getInt(sourceId)
if (sourceInt === undefined) {
@ -4424,11 +4491,19 @@ export abstract class BaseStorage extends BaseStorageAdapter {
targetId: string
): Promise<HNSWVerbWithMetadata[]> {
await this.ensureInitialized()
await this.ensureGraphFastPathProbed()
// Fast path - use GraphAdjacencyIndex if available (lazy-loaded).
// 8.0 BigInt boundary: convert the UUID to an entity int up front and
// resolve returned verb ints back to verb-id strings.
if (this.graphIndex && this.graphIndex.isInitialized && this.graphEntityIdResolver) {
// Honest gate: a not-ready provider (count loaded but edges NOT) is SKIPPED so
// we fall to the correct-but-slower canonical shard scan instead of a silent [].
if (
this.graphIndex &&
this.graphIndex.isInitialized &&
this.graphEntityIdResolver &&
assessIndexReadiness(this.graphIndex) !== 'not-ready'
) {
try {
const targetInt = this.graphEntityIdResolver.getInt(targetId)
if (targetInt === undefined) {

38
src/utils/indexReadiness.ts Normal file
View file

@ -0,0 +1,38 @@
/**
* @module indexReadiness
* @description The single honest-readiness classifier shared by the vector,
* graph and metadata index sites. It exists to kill "Pattern A" — the dishonest
* readiness proxy where `size() > 0` / `isInitialized` is treated as "this index
* actually serves queries." A cold native index that loaded its COUNT but not its
* SERVING structure passes those proxies and silently returns `[]`.
*
* This classifier reads ONLY the provider's OPTIONAL, honest `isReady()` signal
* (see {@link import('../plugin.js').VectorIndexProvider.isReady},
* {@link import('../plugin.js').GraphIndexProvider.isReady},
* {@link import('../plugin.js').MetadataIndexProvider.isReady}). It NEVER inspects
* `size()` or `isInitialized`. When `isReady()` is absent, callers must fall back
* to a KNOWN-ITEM PROBE (a real search/lookup that must return a known-present
* datum) before trusting an empty result — never a `size()` proxy.
*/
/** A provider that MAY expose the honest cold-load readiness signal. */
export interface MaybeReadyProvider {
isReady?: () => boolean
}
/** Three-valued honest-readiness verdict. */
export type IndexReadiness = 'ready' | 'not-ready' | 'unknown'
/**
* @description Classify an index provider's honest readiness.
* @param provider - Any index provider (vector / graph / metadata) or `null`.
* @returns
* - `'ready'` when `isReady() === true` (serving structure loaded — trust it);
* - `'not-ready'` when `isReady() === false` (count/manifest loaded, NOT serving — rebuild);
* - `'unknown'` when the provider exposes no `isReady()` (caller must probe / keep the JS heuristic).
*/
export function assessIndexReadiness(provider: unknown): IndexReadiness {
const p = provider as MaybeReadyProvider | null | undefined
if (p == null || typeof p.isReady !== 'function') return 'unknown'
return p.isReady() ? 'ready' : 'not-ready'
}