fix: honest index readiness — no silently-empty queries on a cold index

Closes the "dishonest readiness proxy" anti-pattern (Pattern A): size()>0 /
isInitialized were treated as "this index serves queries", but a cold native
index can load its COUNT before its SERVING structure, so a query returned a
silent [] indistinguishable from "no such data". A shared assessIndexReadiness()
now reads only the provider's honest isReady() signal (never size()), applied at
every site:

- Vector: a one-shot verifyVectorLive() guard on the semantic/proximity search
  path (a pure semantic find({query}) has no filter, so nothing guarded it). It
  prefers isReady(), else a known-vector self-match probe; self-heals via rebuild
  or throws the new VectorIndexNotReadyError instead of a silent [].
- Graph: getVerbsBySource/ByTarget skip the fast path when the provider reports
  not-ready (falling to the canonical shard scan), plus a one-shot probe that
  self-heals a no-isReady provider whose adjacency did not cold-load.
- getIndexStatus(): folds in per-index honest `ready` (making `populated`
  honest) + rebuildFailed/rebuildError/degradedIds, so a readiness probe never
  200s a brain that is still warming up or degraded.

Unblocked by the native providers now reporting serving-truth (graph via
SSTable-residency readiness, vector via durableBaseLoadFailed). New export:
VectorIndexNotReadyError. getIndexStatus gains additive fields. No breaking API.
13 new tests; existing readiness guards green.
This commit is contained in:
David Snelling 2026-07-13 13:08:52 -07:00
parent 36d4e80ba2
commit d0f69c731f
9 changed files with 654 additions and 9 deletions

View file

@ -10,6 +10,39 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
---
## v8.2.8 — 2026-07-13 (honest index readiness — no more silently-empty queries on a cold index)
Closes the last of the three spine anti-patterns: the "dishonest readiness proxy," where `size() > 0`
was treated as "this index actually serves queries." On a cold open (fresh boot, restart, crash
recovery) a native index can load its **count** before its **serving structure** — so for a brief
window it has data but cannot answer, and a query returned a silent empty result indistinguishable
from "no such data." Every fix here asks the index whether it can *actually serve*, and if not,
self-heals or fails loudly instead of returning `[]`.
- **Semantic search no longer returns a silent `[]` on a cold vector index.** A pure semantic
`find({ query })` has no filter, so nothing previously guarded the vector index. A new one-shot
guard verifies the vector index serves a known persisted vector on the first semantic/proximity
search — preferring the provider's honest `isReady()` signal, else a known-vector self-match probe.
It rebuilds from canonical records if the serving structure did not load, and throws the new
**`VectorIndexNotReadyError`** only if a rebuild still cannot serve — never a silent empty result.
- **Relationship reads fall back to the canonical scan instead of an empty result on a cold graph
index.** `getVerbsBySource`/`getVerbsByTarget` (used by relationship queries and virtual-filesystem
traversal) skipped the fast path only on `isInitialized` — which reads true once the manifest loaded
even if the source→target adjacency did not. They now consult the honest readiness signal and, when
the adjacency is not serving, take the correct-but-slower canonical shard scan. A one-shot self-heal
probe covers providers that expose no readiness signal.
- **`getIndexStatus()` tells the truth for readiness probes.** It reported `populated: size>0` only, so
a Kubernetes readiness check could route traffic to a brain still warming up. It now folds in the
honest per-index `ready` signal (making `populated` honest), plus the degraded states already
surfaced by `checkHealth()`/`validateIndexConsistency()` (`rebuildFailed`/`rebuildError` and a
`degradedIds` count) — so a probe never reports 200-ready over a known-degraded index.
This completes the write/index-spine hardening end to end. New export: **`VectorIndexNotReadyError`**;
`getIndexStatus()` gains additive fields (`rebuildFailed`, `rebuildError?`, `degradedIds`, per-index
`ready?`). No breaking API change. Each fix ships with a dedicated regression test.
## v8.2.7 — 2026-07-13 (loadBinaryBlob fault-propagation — the lockstep completion of 8.2.6)
Completes the Pass-1 spine hardening. `loadBinaryBlob` (the raw-blob read a native accelerator mmaps

View file

@ -175,7 +175,8 @@ import {
} from './events/changeFeed.js'
import { isDeterministicEmbedMode } from './embeddings/deterministicEmbedMode.js'
import { GenerationConflictError, StoreInconsistentError } from './db/errors.js'
import { BrainyError, GraphIndexNotReadyError, MetadataIndexNotReadyError, MigrationInProgressError } from './errors/brainyError.js'
import { BrainyError, GraphIndexNotReadyError, MetadataIndexNotReadyError, MigrationInProgressError, VectorIndexNotReadyError } from './errors/brainyError.js'
import { assessIndexReadiness } from './utils/indexReadiness.js'
import { MemoryStorage } from './storage/adapters/memoryStorage.js'
import type {
CompactHistoryOptions,
@ -500,6 +501,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private _metadataVerified = false
/** Re-entrancy guard for {@link verifyMetadataLive}. */
private _metadataVerifying = false
/** Vector-index cold-read guard: verified-serving this session (one-shot). */
private _vectorVerified = false
/** Re-entrancy guard for {@link verifyVectorLive}. */
private _vectorVerifying = false
/**
* Coordinated migration LOCK (#18): dedup guards so the "upgrading, blocking"
* and "upgrade complete, resumed" lines each log once per migration window,
@ -3523,6 +3528,142 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return null
}
/**
* @description The vector-index counterpart of {@link verifyGraphAdjacencyLive}
* / {@link verifyMetadataLive}. On a cold open a native vector provider can
* report a non-zero `size()` (its persisted COUNT loaded) yet not have loaded
* its serving structure (the mmap/DiskANN graph) so a pure semantic
* `find({ query })` silently returns `[]`. A pure semantic query has
* `hasFilterCriteria === false`, so the metadata guard never fires; this guard
* closes that gap. Run one-shot on the first vector/proximity search:
* - **Preferred (honest signal):** the provider exposes `isReady()`. `false`
* rebuild from storage, re-check; if still `false`, throw
* {@link VectorIndexNotReadyError} rather than serving `[]`.
* - **Fallback (no `isReady()`):** a KNOWN persisted vector (sampled +
* hydrated) is searched against the index; if it does not self-match, the
* serving structure did not load rebuild + re-probe, else throw.
* Inconclusive cases (empty store, no probeable vector, `size()===0` where
* the JS baseline's cold load is `ensureIndexesLoaded`'s job) are treated as
* live: never a false rebuild. A migrating provider is skipped (it owns its
* locked rebuild).
* @returns `'live'` when the index serves, `'rebuilt'` when a rebuild restored it.
*/
private async verifyVectorLive(): Promise<'live' | 'rebuilt'> {
if (this._vectorVerified) return 'live'
// Migration LOCK (#18): a migrating provider owns its in-place rebuild.
if (this.providerIsMigrating(this.index)) return 'live'
// Re-entrancy: rebuild() can trigger reads that call back into this guard.
if (this._vectorVerifying) return 'live'
this._vectorVerifying = true
try {
// ── Strategy 1: honest isReady() signal (native provider) ──────────────
const readiness = assessIndexReadiness(this.index)
if (readiness !== 'unknown') {
if (readiness === 'ready') {
this._vectorVerified = true
return 'live'
}
// Not ready: the serving structure did not load on open. Rebuild.
if (!this.config.silent) {
console.warn(
`[Brainy] Vector index reports not-ready (isReady() === false) — the persisted ` +
`vector index did not load on open. Rebuilding from storage…`
)
}
await this.index.rebuild()
if (assessIndexReadiness(this.index) === 'ready') {
this._vectorVerified = true
return 'rebuilt'
}
throw new VectorIndexNotReadyError(
`Vector index reports not-ready even after a rebuild — semantic find({ query }) and ` +
`proximity search cannot be served reliably for this brain (a silent empty result ` +
`would misrepresent existing data).`
)
}
// ── Strategy 2: known-vector probe (providers without isReady()) ───────
const claimed = this.index.size()
if (!claimed || claimed <= 0) return 'live' // JS cold path is ensureIndexesLoaded's job
const probe = await this.pickVectorProbe()
if (!probe) {
// Empty store, or nothing with a probeable vector — inconclusive.
this._vectorVerified = true
return 'live'
}
const p = probe
const probeServes = async (): Promise<boolean> => {
// The failure mode we guard is the SILENT EMPTY result: a cold index that
// loaded its COUNT but not its serving structure returns `[]` for a
// known-present vector, while a warm index returns at least one hit. We
// check for a NON-EMPTY result, NOT an exact self-match — HNSW is
// approximate and `get()` may return a re-hydrated/normalized vector, so
// demanding the exact self as top-1 would false-positive on a perfectly
// healthy index (and wrongly rebuild → throw).
const hits = await this.index.search(p.vector, 1)
return hits.length > 0
}
void p.id // probe keyed on the vector; id retained for diagnostics only
if (await probeServes()) {
this._vectorVerified = true
return 'live' // serving structure is live — the common case
}
if (!this.config.silent) {
console.warn(
`[Brainy] Vector index reports ${claimed} vector(s) but a known persisted vector ` +
`returns no results — the serving structure did not load on open. Rebuilding…`
)
}
await this.index.rebuild()
if (await probeServes()) {
this._vectorVerified = true
return 'rebuilt'
}
throw new VectorIndexNotReadyError(
`Vector index reports ${claimed} vector(s) but a known persisted vector returns no ` +
`results even after a rebuild — semantic find({ query }) cannot be served reliably ` +
`for this brain (a silent empty result would misrepresent existing data).`
)
} catch (err) {
if (err instanceof VectorIndexNotReadyError) throw err
// A transient probe/rebuild failure must not break the query NOR mask as
// "no data". Allow a re-check on the next vector read and fall through.
this._vectorVerified = false
if (!this.config.silent) {
console.warn(`[Brainy] Vector consistency check skipped (transient): ${err}`)
}
return 'live'
} finally {
this._vectorVerifying = false
}
}
/**
* @description Sample a KNOWN persisted noun and hydrate its vector, to probe
* the vector index with. `get()` omits vectors by default, so this passes
* `{ includeVectors: true }`. Samples a few (a system-only / vectorless entity
* must not make every open inconclusive). Returns `null` when nothing has a
* probeable vector.
*/
private async pickVectorProbe(): Promise<{ id: string; vector: number[] } | null> {
const sample = await this.storage.getNouns({ pagination: { limit: 5, offset: 0 } })
for (const noun of sample.items ?? []) {
const id = (noun as { id?: string }).id
if (!id) continue
const full = await this.get(id, { includeVectors: true })
const vector = (full as { vector?: number[] } | null)?.vector
if (Array.isArray(vector) && vector.length > 0) {
return { id, vector }
}
}
return null
}
// -------------------------------------------------------------------------
/**
@ -10175,17 +10316,32 @@ export class Brainy<T = any> implements BrainyInterface<T> {
migrating: boolean
/** Structured progress while {@link migrating}; absent otherwise. */
migration?: MigrationProgress
/** `true` when a non-fatal index rebuild failed at init() (degraded queries
* may be incomplete). Folds in the same `_indexRebuildFailed` signal that
* {@link validateIndexConsistency} / {@link checkHealth} already expose. */
rebuildFailed: boolean
/** The rebuild failure message when {@link rebuildFailed}; absent otherwise. */
rebuildError?: string
/** Count of records committed via adopt-forward failed-rollback recovery whose
* derived index may be incomplete until repairIndex() (the 8.2.6 degraded
* set). Non-zero = degraded a readiness probe should not report 200-ready. */
degradedIds: number
hnswIndex: {
size: number
/** Honest "serving" — the provider's `isReady()` when exposed, else `size>0`. */
populated: boolean
/** The provider's honest `isReady()` signal; absent when unexposed. */
ready?: boolean
}
metadataIndex: {
entries: number
populated: boolean
ready?: boolean
}
graphIndex: {
relationships: number
populated: boolean
ready?: boolean
}
storage: {
totalEntities: number
@ -10200,6 +10356,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
lazyRebuildCompleted: this.lazyRebuildCompleted,
disableAutoRebuild: this.config.disableAutoRebuild || false,
migrating: false,
rebuildFailed: this._indexRebuildFailed != null,
...(this._indexRebuildFailed ? { rebuildError: this._indexRebuildFailed.message } : {}),
degradedIds: this._indexDegradedIds.size,
hnswIndex: { size: 0, populated: false },
metadataIndex: { entries: 0, populated: false },
graphIndex: { relationships: 0, populated: false },
@ -10211,6 +10370,20 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const hnswSize = this.index.size()
const graphSize = await this.graphIndex.size()
// Honest readiness: when a provider exposes isReady(), it is the truth of
// whether the index actually SERVES (a native index can report a non-zero
// size/count yet not have loaded its serving structure). `populated` reflects
// that honest signal when present, falling back to size>0 only for providers
// that do not expose isReady(). Mirrors validateIndexConsistency/checkHealth,
// which already fold in the same signals.
const readyOf = (p: unknown): boolean | undefined => {
const r = assessIndexReadiness(p)
return r === 'unknown' ? undefined : r === 'ready'
}
const hnswReady = readyOf(this.index)
const metadataReady = readyOf(this.metadataIndex)
const graphReady = readyOf(this.graphIndex)
// Check storage entity count
let storageEntityCount = 0
try {
@ -10224,18 +10397,29 @@ export class Brainy<T = any> implements BrainyInterface<T> {
initialized: this.initialized,
lazyRebuildCompleted: this.lazyRebuildCompleted,
disableAutoRebuild: this.config.disableAutoRebuild || false,
// A non-fatal index-rebuild failure recorded at init(), or adopt-forward
// degraded ids, are degraded states (queries may be incomplete) — surface
// them here alongside the same signals validateIndexConsistency()/
// checkHealth() already expose, so a readiness probe never reports 200-ready
// over a known-degraded index.
rebuildFailed: this._indexRebuildFailed != null,
...(this._indexRebuildFailed ? { rebuildError: this._indexRebuildFailed.message } : {}),
degradedIds: this._indexDegradedIds.size,
...this.migrationSnapshot(),
hnswIndex: {
size: hnswSize,
populated: hnswSize > 0
populated: hnswReady ?? hnswSize > 0,
...(hnswReady !== undefined ? { ready: hnswReady } : {})
},
metadataIndex: {
entries: metadataStats.totalEntries,
populated: metadataStats.totalEntries > 0
populated: metadataReady ?? metadataStats.totalEntries > 0,
...(metadataReady !== undefined ? { ready: metadataReady } : {})
},
graphIndex: {
relationships: graphSize,
populated: graphSize > 0
populated: graphReady ?? graphSize > 0,
...(graphReady !== undefined ? { ready: graphReady } : {})
},
storage: {
totalEntities: storageEntityCount
@ -12863,6 +13047,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
candidateIds?: string[],
allowedIds?: OpaqueIdSet
): Promise<Result<T>[]> {
// Vector cold-read guard: before trusting a semantic/vector result, verify the
// vector index actually SERVES a known persisted vector (one-shot per brain).
// A pure semantic find({ query }) has no filter, so verifyMetadataLive never
// fires — a cold native index that loaded its COUNT but not its serving
// structure would return a silent []. This self-heals (rebuild) or throws
// VectorIndexNotReadyError instead.
await this.verifyVectorLive()
const vector = params.vector || (await this.embed(params.query!))
const limit = params.limit || 10
@ -12903,6 +13095,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private async executeProximitySearch(params: FindParams<T>): Promise<Result<T>[]> {
if (!params.near) return []
// Vector cold-read guard (see executeVectorSearch): proximity search also
// hits this.index.search — verify it serves before trusting an empty result.
await this.verifyVectorLive()
// Teaching error: without an anchor id the constraint is meaningless, and
// letting it fall through produces an opaque storage-layer sharding error.
if (!params.near.id) {
@ -14157,8 +14353,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return
}
// If indexes already populated, mark as complete and skip
if (this.index.size() > 0) {
// If indexes already populated AND honestly serving, mark complete and skip.
// Honest gate: when the provider exposes isReady(), that REPLACES the size()>0
// proxy (a native index can report a non-zero size while its serving structure
// is not loaded — the silent-empty cold-load class). A not-ready provider falls
// through so the rebuild path can load it; verifyVectorLive() is the query-time
// backstop either way. Providers without isReady() keep the size() heuristic
// (the JS index's size()>0 genuinely means loaded).
const vectorReadiness = assessIndexReadiness(this.index)
if (vectorReadiness === 'ready' || (vectorReadiness === 'unknown' && this.index.size() > 0)) {
this.lazyRebuildCompleted = true
return
}

View file

@ -14,6 +14,7 @@ export type BrainyErrorType =
| 'FIELD_NOT_INDEXED'
| 'GRAPH_INDEX_NOT_READY'
| 'METADATA_INDEX_NOT_READY'
| 'VECTOR_INDEX_NOT_READY'
| 'MIGRATION_IN_PROGRESS'
/**
@ -280,6 +281,32 @@ export class MetadataIndexNotReadyError extends BrainyError {
}
}
/**
* Thrown when the vector index reports vectors (`size() > 0` or, on a native
* provider, `isReady() === false`) but cannot return a KNOWN persisted vector
* even after a rebuild i.e. the semantic serving structure did not load on a
* cold open and could not be restored. The vector-search counterpart of
* {@link GraphIndexNotReadyError} / {@link MetadataIndexNotReadyError}: it
* replaces the silent-empty failure mode (a cold `find({ query })` returning
* `[]` indistinguishable from "no similar data") with a loud, catchable error,
* so a consumer never renders "nothing found" over data that is simply
* not-yet-warm.
*
* Detected once per brain by a known-vector serving probe on the first
* semantic / proximity `find()`; brainy self-heals (rebuilds the index from the
* canonical records) first and only raises this if the rebuild still cannot
* serve the known vector.
*/
export class VectorIndexNotReadyError extends BrainyError {
constructor(message: string, originalError?: Error) {
super(message, 'VECTOR_INDEX_NOT_READY', false, originalError)
this.name = 'VectorIndexNotReadyError'
if (Error.captureStackTrace) {
Error.captureStackTrace(this, VectorIndexNotReadyError)
}
}
}
/**
* Thrown when a data-plane read or write is issued against a brain that is
* running its one-time, automatic 7.x 8.0 on-disk upgrade the coordinated

View file

@ -150,7 +150,7 @@ export { EntityNotFoundError, RelationNotFoundError } from './errors/notFound.js
// Base error + typed migration-lock error — thrown by any data-plane call while a
// brain runs its one-time 7.x→8.0 upgrade; catch to answer HTTP 503 + Retry-After.
export { BrainyError, MigrationInProgressError, GraphIndexNotReadyError, MetadataIndexNotReadyError } from './errors/brainyError.js'
export { BrainyError, MigrationInProgressError, GraphIndexNotReadyError, MetadataIndexNotReadyError, VectorIndexNotReadyError } from './errors/brainyError.js'
export type { BrainyErrorType } from './errors/brainyError.js'
// ============= 8.0 Db API — generational MVCC =============

View file

@ -5,6 +5,7 @@
import { GraphAdjacencyIndex } from '../graph/graphAdjacencyIndex.js'
import type { GraphEntityIdResolver } from '../graph/graphAdjacencyIndex.js'
import { assessIndexReadiness } from '../utils/indexReadiness.js'
import {
GraphVerb,
@ -248,6 +249,8 @@ function isCountedVisibility(visibility: unknown): boolean {
*/
export abstract class BaseStorage extends BaseStorageAdapter {
protected isInitialized = false
/** One-shot guard so the graph fast-path cold-load probe runs once per adapter. */
private _graphFastPathProbed = false
protected graphIndex?: GraphAdjacencyIndex
protected graphIndexPromise?: Promise<GraphAdjacencyIndex>
/**
@ -2681,6 +2684,61 @@ export abstract class BaseStorage extends BaseStorageAdapter {
return this.graphIndex
}
/**
* @description One-shot cold-load self-heal for a graph provider that does NOT
* expose the honest `isReady()` signal. A native provider wired via
* {@link setGraphIndex} bypasses `_initializeGraphIndex`'s `size()===0`
* self-heal, so a cold-open that loaded the COUNT but not the sourcetarget
* adjacency would let the fast path return a silent `[]`. This probe (run once,
* before the fast path) samples ONE known persisted edge: if the index claims
* relationships (`size() > 0`) yet that edge's source resolves to no verb ints,
* the adjacency did not load rebuild once from storage. Providers that expose
* `isReady()` are covered by the honest gate in
* getVerbsBy{Source,Target}_internal and skip this probe; the JS index (which
* self-heals in `_initializeGraphIndex`) resolves its known edge and no-ops.
*/
private async ensureGraphFastPathProbed(): Promise<void> {
if (this._graphFastPathProbed) return
const index = this.graphIndex
const resolver = this.graphEntityIdResolver
if (!index || !index.isInitialized || !resolver) return
// isReady()-capable providers report serving honestly — the gate handles them.
if (assessIndexReadiness(index) !== 'unknown') {
this._graphFastPathProbed = true
return
}
if (index.size() <= 0) {
this._graphFastPathProbed = true
return // no edges claimed — nothing to verify
}
try {
const sample = await this.getVerbs({ pagination: { limit: 1 } })
const verb = sample.items?.[0]
if (!verb || !verb.sourceId) {
this._graphFastPathProbed = true
return // no edges in storage — stale count, harmless
}
const sourceInt = resolver.getInt(verb.sourceId)
if (sourceInt === undefined) {
this._graphFastPathProbed = true
return // foreign / unmapped sample — inconclusive, never a false rebuild
}
const verbInts = await index.getVerbIdsBySource(BigInt(sourceInt))
if (verbInts.length === 0) {
prodLog.warn(
`[BaseStorage] Graph fast path: index reports ${index.size()} relationship(s) but a ` +
`known persisted edge resolves to none — the persisted adjacency did not load. ` +
`Rebuilding once from storage.`
)
await index.rebuild()
}
this._graphFastPathProbed = true
} catch (error) {
// Transient probe/rebuild failure must not break the read; re-arm for next call.
prodLog.debug(`[BaseStorage] Graph fast-path probe skipped (transient): ${error}`)
}
}
/**
* Clear all data from storage
* This method should be implemented by each specific adapter
@ -4199,13 +4257,22 @@ export abstract class BaseStorage extends BaseStorageAdapter {
sourceId: string
): Promise<HNSWVerbWithMetadata[]> {
await this.ensureInitialized()
await this.ensureGraphFastPathProbed()
prodLog.debug(`[BaseStorage] getVerbsBySource_internal: sourceId=${sourceId}, graphIndex=${!!this.graphIndex}, isInitialized=${this.graphIndex?.isInitialized}`)
// Fast path - use GraphAdjacencyIndex if available (lazy-loaded).
// 8.0 BigInt boundary: convert the UUID to an entity int up front and
// resolve returned verb ints back to verb-id strings.
if (this.graphIndex && this.graphIndex.isInitialized && this.graphEntityIdResolver) {
// Honest gate: a provider that exposes isReady() and reports not-ready
// (count/manifest loaded but source→target edges NOT) is SKIPPED so we fall
// to the correct-but-slower canonical shard scan below instead of a silent [].
if (
this.graphIndex &&
this.graphIndex.isInitialized &&
this.graphEntityIdResolver &&
assessIndexReadiness(this.graphIndex) !== 'not-ready'
) {
try {
const sourceInt = this.graphEntityIdResolver.getInt(sourceId)
if (sourceInt === undefined) {
@ -4424,11 +4491,19 @@ export abstract class BaseStorage extends BaseStorageAdapter {
targetId: string
): Promise<HNSWVerbWithMetadata[]> {
await this.ensureInitialized()
await this.ensureGraphFastPathProbed()
// Fast path - use GraphAdjacencyIndex if available (lazy-loaded).
// 8.0 BigInt boundary: convert the UUID to an entity int up front and
// resolve returned verb ints back to verb-id strings.
if (this.graphIndex && this.graphIndex.isInitialized && this.graphEntityIdResolver) {
// Honest gate: a not-ready provider (count loaded but edges NOT) is SKIPPED so
// we fall to the correct-but-slower canonical shard scan instead of a silent [].
if (
this.graphIndex &&
this.graphIndex.isInitialized &&
this.graphEntityIdResolver &&
assessIndexReadiness(this.graphIndex) !== 'not-ready'
) {
try {
const targetInt = this.graphEntityIdResolver.getInt(targetId)
if (targetInt === undefined) {

View file

@ -0,0 +1,38 @@
/**
* @module indexReadiness
* @description The single honest-readiness classifier shared by the vector,
* graph and metadata index sites. It exists to kill "Pattern A" the dishonest
* readiness proxy where `size() > 0` / `isInitialized` is treated as "this index
* actually serves queries." A cold native index that loaded its COUNT but not its
* SERVING structure passes those proxies and silently returns `[]`.
*
* This classifier reads ONLY the provider's OPTIONAL, honest `isReady()` signal
* (see {@link import('../plugin.js').VectorIndexProvider.isReady},
* {@link import('../plugin.js').GraphIndexProvider.isReady},
* {@link import('../plugin.js').MetadataIndexProvider.isReady}). It NEVER inspects
* `size()` or `isInitialized`. When `isReady()` is absent, callers must fall back
* to a KNOWN-ITEM PROBE (a real search/lookup that must return a known-present
* datum) before trusting an empty result never a `size()` proxy.
*/
/** A provider that MAY expose the honest cold-load readiness signal. */
export interface MaybeReadyProvider {
isReady?: () => boolean
}
/** Three-valued honest-readiness verdict. */
export type IndexReadiness = 'ready' | 'not-ready' | 'unknown'
/**
* @description Classify an index provider's honest readiness.
* @param provider - Any index provider (vector / graph / metadata) or `null`.
* @returns
* - `'ready'` when `isReady() === true` (serving structure loaded trust it);
* - `'not-ready'` when `isReady() === false` (count/manifest loaded, NOT serving rebuild);
* - `'unknown'` when the provider exposes no `isReady()` (caller must probe / keep the JS heuristic).
*/
export function assessIndexReadiness(provider: unknown): IndexReadiness {
const p = provider as MaybeReadyProvider | null | undefined
if (p == null || typeof p.isReady !== 'function') return 'unknown'
return p.isReady() ? 'ready' : 'not-ready'
}

View file

@ -0,0 +1,59 @@
/**
* @module tests/unit/get-index-status-readiness
* @description Pattern-A / Finding 9: getIndexStatus reported `populated: size>0`
* and only `migrating`, so a native index that loaded its count but not its
* serving structure reported populated:true a k8s readiness probe would 200 a
* brain that serves []. It now folds in the honest isReady() signal + the
* _indexRebuildFailed / _indexDegradedIds degraded states (mirroring
* validateIndexConsistency / checkHealth).
*/
import { describe, it, expect, beforeEach } from 'vitest'
import { Brainy, NounType } from '../../src/index.js'
describe('getIndexStatus honest readiness (Finding 9)', () => {
let brain: any
beforeEach(async () => {
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, dimensions: 384 })
await brain.init()
await brain.add({ data: 'x', type: NounType.Concept })
await brain.flush()
})
it('a not-ready provider makes populated honest (false) and exposes ready:false', async () => {
brain.index.isReady = () => false // count present, serving structure NOT loaded
const status = await brain.getIndexStatus()
expect(status.hnswIndex.populated).toBe(false)
expect(status.hnswIndex.ready).toBe(false)
delete brain.index.isReady
})
it('a ready provider reports populated:true + ready:true', async () => {
brain.index.isReady = () => true
const status = await brain.getIndexStatus()
expect(status.hnswIndex.populated).toBe(true)
expect(status.hnswIndex.ready).toBe(true)
delete brain.index.isReady
})
it('rebuildFailed / rebuildError surface the init-degraded state', async () => {
brain._indexRebuildFailed = new Error('boom')
const status = await brain.getIndexStatus()
expect(status.rebuildFailed).toBe(true)
expect(status.rebuildError).toBe('boom')
brain._indexRebuildFailed = null
})
it('degradedIds surfaces the adopt-forward degraded set', async () => {
brain._indexDegradedIds.add('00000000-0000-4000-8000-0000000000de')
const status = await brain.getIndexStatus()
expect(status.degradedIds).toBe(1)
brain._indexDegradedIds.clear()
})
it('a JS-baseline provider (no isReady) omits ready and falls back to size>0', async () => {
const status = await brain.getIndexStatus()
expect(status.hnswIndex.ready).toBeUndefined()
expect(status.hnswIndex.populated).toBe(brain.index.size() > 0)
})
})

View file

@ -0,0 +1,95 @@
/**
* @module tests/unit/graph/graph-fastpath-honest-readiness
* @description Pattern-A / Finding 2: getVerbsBySource/ByTarget gated the graph
* fast path on `graphIndex.isInitialized` (a proxy that reads true once the
* manifest/count loaded even if the sourcetarget adjacency did NOT the fast
* path then returned a silent []). The honest gate skips the fast path when the
* provider reports isReady()===false, falling to the correct canonical shard
* scan; and a one-shot probe self-heals a no-isReady provider whose adjacency
* did not cold-load.
*/
import { describe, it, expect, beforeEach } from 'vitest'
import { Brainy, NounType, VerbType } from '../../../src/index.js'
describe('graph fast-path honest readiness (Finding 2)', () => {
let brain: any
let storage: any
let a: string
let b: string
let c: string
beforeEach(async () => {
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, dimensions: 384 })
await brain.init()
a = await brain.add({ data: 'a', type: NounType.Concept })
b = await brain.add({ data: 'b', type: NounType.Concept })
c = await brain.add({ data: 'c', type: NounType.Concept })
await brain.relate({ from: a, to: b, type: VerbType.Contains })
await brain.relate({ from: a, to: c, type: VerbType.Contains })
await brain.flush()
storage = brain.storage
// Warm + wire the storage graph index (lazy) so we can stub it.
await storage.getVerbsBySource(a)
})
it('not-ready provider → shard scan returns the REAL edges, not a silent []', async () => {
const gi = storage.graphIndex
// Simulate a cold native provider: count/manifest loaded (isInitialized) but
// the source→target adjacency is NOT (isReady false; fast-path lookup empty).
gi.isReady = () => false
const origBySource = gi.getVerbIdsBySource.bind(gi)
gi.getVerbIdsBySource = async () => [] // cold adjacency — old fast path trusted this
storage._graphFastPathProbed = true // isolate the GATE (probe skips isReady-capable anyway)
try {
const verbs = await storage.getVerbsBySource(a)
// Honest gate skipped the cold fast path → canonical shard scan → real edges.
expect(verbs.length).toBe(2)
} finally {
delete gi.isReady
gi.getVerbIdsBySource = origBySource
}
})
it('ready provider → fast path is used and an empty result is trusted', async () => {
const gi = storage.graphIndex
gi.isReady = () => true
let fastPathCalls = 0
const origBySource = gi.getVerbIdsBySource.bind(gi)
gi.getVerbIdsBySource = async (...args: any[]) => { fastPathCalls++; return origBySource(...args) }
storage._graphFastPathProbed = true
try {
// `c` has no OUTGOING edges — a genuinely empty result via the fast path.
const verbs = await storage.getVerbsBySource(c)
expect(verbs).toEqual([])
expect(fastPathCalls).toBeGreaterThan(0) // fast path was taken, not a shard scan
} finally {
delete gi.isReady
gi.getVerbIdsBySource = origBySource
}
})
it('no-isReady provider whose adjacency did not cold-load → probe rebuilds once', async () => {
const gi = storage.graphIndex
if ('isReady' in gi) delete gi.isReady // force the "unknown" (no honest signal) path
// Force the probe to see a cold adjacency: a known edge resolves to no ints.
const origBySource = gi.getVerbIdsBySource.bind(gi)
let cold = true
gi.getVerbIdsBySource = async (...args: any[]) => (cold ? [] : origBySource(...args))
let rebuilds = 0
const origRebuild = gi.rebuild.bind(gi)
gi.rebuild = async (...args: any[]) => { rebuilds++; cold = false; return origRebuild(...args) }
storage._graphFastPathProbed = false // re-arm the one-shot probe
try {
await storage.getVerbsBySource(a)
expect(rebuilds).toBe(1) // probe detected cold adjacency + rebuilt once
expect(storage._graphFastPathProbed).toBe(true) // latched — no second rebuild
rebuilds = 0
await storage.getVerbsByTarget(b)
expect(rebuilds).toBe(0) // probe does not re-run
} finally {
gi.getVerbIdsBySource = origBySource
gi.rebuild = origRebuild
}
})
})

View file

@ -0,0 +1,115 @@
/**
* @module tests/unit/vector-cold-read-guard
* @description Pattern-A / Finding 1: a pure semantic find({ query }) has no
* filter, so verifyMetadataLive never fires nothing guarded the vector index.
* A cold native vector index that loaded its COUNT but not its serving structure
* returned a silent []. verifyVectorLive() closes that: honest isReady() first,
* else a known-vector self-match probe; self-heal (rebuild) or throw
* VectorIndexNotReadyError never a silent empty result.
*/
import { describe, it, expect, beforeEach } from 'vitest'
import { Brainy, NounType, VectorIndexNotReadyError } from '../../src/index.js'
const V = (): number[] => Array.from({ length: 384 }, (_, i) => Math.sin(i * 0.1) + 0.001)
describe('Vector cold-read guard (verifyVectorLive) — silent-[] on cold semantic find', () => {
let brain: any
beforeEach(async () => {
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, dimensions: 384 })
await brain.init()
await brain.add({ vector: V(), type: NounType.Concept, metadata: { status: 'active' } })
await brain.add({ vector: V(), type: NounType.Concept, metadata: { status: 'archived' } })
await brain.flush()
})
it('warm brain: semantic find is correct and the guard does not rebuild', async () => {
const vi = brain.index
let rebuilds = 0
const origRebuild = vi.rebuild.bind(vi)
vi.rebuild = async (...a: any[]) => { rebuilds++; return origRebuild(...a) }
await brain.find({ query: 'anything', searchMode: 'semantic', limit: 100 })
expect(rebuilds).toBe(0)
expect(brain._vectorVerified).toBe(true)
vi.rebuild = origRebuild
})
it('cold index: verifyVectorLive self-heals via rebuild — semantic find is correct, NOT silent []', async () => {
const vi = brain.index
const origSearch = vi.search.bind(vi)
const origRebuild = vi.rebuild.bind(vi)
let cold = true
brain._vectorVerified = false
// size()>0 (count present) but search returns nothing until a rebuild warms it.
vi.search = async (...a: any[]) => (cold ? [] : origSearch(...a))
vi.rebuild = async (...a: any[]) => { await origRebuild(...a); cold = false }
try {
const res = await brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
expect(res.length).toBeGreaterThan(0) // self-healed
} finally {
vi.search = origSearch; vi.rebuild = origRebuild
}
})
it('unrecoverably cold index: semantic find throws VectorIndexNotReadyError', async () => {
const vi = brain.index
const origSearch = vi.search.bind(vi)
const origRebuild = vi.rebuild.bind(vi)
brain._vectorVerified = false
vi.search = async () => [] // always cold; rebuild can't fix it
vi.rebuild = async () => {}
try {
await expect(
brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
).rejects.toBeInstanceOf(VectorIndexNotReadyError)
} finally {
vi.search = origSearch; vi.rebuild = origRebuild
}
})
it('native provider reporting isReady()===false rebuilds, then serves', async () => {
const vi = brain.index
const origRebuild = vi.rebuild.bind(vi)
let ready = false
brain._vectorVerified = false
vi.isReady = () => ready
vi.rebuild = async (...a: any[]) => { await origRebuild(...a); ready = true }
try {
const res = await brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
expect(ready).toBe(true) // rebuild ran because isReady() was false
expect(res).toBeDefined()
} finally {
delete vi.isReady; vi.rebuild = origRebuild
}
})
it('a text-only query does not trigger the vector guard', async () => {
brain._vectorVerified = false
await brain.find({ query: 'active', searchMode: 'text', limit: 5 })
expect(brain._vectorVerified).toBe(false) // executeVectorSearch never called
})
// Regression: the probe must check "returns ANY hit", not an exact self-match —
// HNSW is approximate and get() may re-hydrate the vector, so a healthy
// many-entity index would false-positive under an exact-self check, wrongly
// rebuild, and throw VectorIndexNotReadyError on working data.
it('a healthy many-entity brain with distinct vectors serves semantic find, never throws/rebuilds', async () => {
const many = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, dimensions: 384 })
await many.init()
for (let i = 0; i < 25; i++) {
const v = Array.from({ length: 384 }, (_, j) => Math.sin((i * 7 + j) * 0.13) + 0.001)
await many.add({ vector: v, type: NounType.Concept, metadata: { n: i } })
}
await many.flush()
let rebuilds = 0
const vi = (many as any).index
const origRebuild = vi.rebuild.bind(vi)
vi.rebuild = async (...a: any[]) => { rebuilds++; return origRebuild(...a) }
const res = await many.find({ query: 'x', searchMode: 'semantic', limit: 10 })
expect(res.length).toBeGreaterThan(0)
expect(rebuilds).toBe(0)
expect((many as any)._vectorVerified).toBe(true)
vi.rebuild = origRebuild
await many.close()
})
})