fix: honest index readiness — no silently-empty queries on a cold index
Closes the "dishonest readiness proxy" anti-pattern (Pattern A): size()>0 /
isInitialized were treated as "this index serves queries", but a cold native
index can load its COUNT before its SERVING structure, so a query returned a
silent [] indistinguishable from "no such data". A shared assessIndexReadiness()
now reads only the provider's honest isReady() signal (never size()), applied at
every site:
- Vector: a one-shot verifyVectorLive() guard on the semantic/proximity search
path (a pure semantic find({query}) has no filter, so nothing guarded it). It
prefers isReady(), else a known-vector self-match probe; self-heals via rebuild
or throws the new VectorIndexNotReadyError instead of a silent [].
- Graph: getVerbsBySource/ByTarget skip the fast path when the provider reports
not-ready (falling to the canonical shard scan), plus a one-shot probe that
self-heals a no-isReady provider whose adjacency did not cold-load.
- getIndexStatus(): folds in per-index honest `ready` (making `populated`
honest) + rebuildFailed/rebuildError/degradedIds, so a readiness probe never
200s a brain that is still warming up or degraded.
Unblocked by the native providers now reporting serving-truth (graph via
SSTable-residency readiness, vector via durableBaseLoadFailed). New export:
VectorIndexNotReadyError. getIndexStatus gains additive fields. No breaking API.
13 new tests; existing readiness guards green.
This commit is contained in:
parent
36d4e80ba2
commit
d0f69c731f
9 changed files with 654 additions and 9 deletions
33
RELEASES.md
33
RELEASES.md
|
|
@ -10,6 +10,39 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## v8.2.8 — 2026-07-13 (honest index readiness — no more silently-empty queries on a cold index)
|
||||||
|
|
||||||
|
Closes the last of the three spine anti-patterns: the "dishonest readiness proxy," where `size() > 0`
|
||||||
|
was treated as "this index actually serves queries." On a cold open (fresh boot, restart, crash
|
||||||
|
recovery) a native index can load its **count** before its **serving structure** — so for a brief
|
||||||
|
window it has data but cannot answer, and a query returned a silent empty result indistinguishable
|
||||||
|
from "no such data." Every fix here asks the index whether it can *actually serve*, and if not,
|
||||||
|
self-heals or fails loudly instead of returning `[]`.
|
||||||
|
|
||||||
|
- **Semantic search no longer returns a silent `[]` on a cold vector index.** A pure semantic
|
||||||
|
`find({ query })` has no filter, so nothing previously guarded the vector index. A new one-shot
|
||||||
|
guard verifies the vector index serves a known persisted vector on the first semantic/proximity
|
||||||
|
search — preferring the provider's honest `isReady()` signal, else a known-vector self-match probe.
|
||||||
|
It rebuilds from canonical records if the serving structure did not load, and throws the new
|
||||||
|
**`VectorIndexNotReadyError`** only if a rebuild still cannot serve — never a silent empty result.
|
||||||
|
|
||||||
|
- **Relationship reads fall back to the canonical scan instead of an empty result on a cold graph
|
||||||
|
index.** `getVerbsBySource`/`getVerbsByTarget` (used by relationship queries and virtual-filesystem
|
||||||
|
traversal) skipped the fast path only on `isInitialized` — which reads true once the manifest loaded
|
||||||
|
even if the source→target adjacency did not. They now consult the honest readiness signal and, when
|
||||||
|
the adjacency is not serving, take the correct-but-slower canonical shard scan. A one-shot self-heal
|
||||||
|
probe covers providers that expose no readiness signal.
|
||||||
|
|
||||||
|
- **`getIndexStatus()` tells the truth for readiness probes.** It reported `populated: size>0` only, so
|
||||||
|
a Kubernetes readiness check could route traffic to a brain still warming up. It now folds in the
|
||||||
|
honest per-index `ready` signal (making `populated` honest), plus the degraded states already
|
||||||
|
surfaced by `checkHealth()`/`validateIndexConsistency()` (`rebuildFailed`/`rebuildError` and a
|
||||||
|
`degradedIds` count) — so a probe never reports 200-ready over a known-degraded index.
|
||||||
|
|
||||||
|
This completes the write/index-spine hardening end to end. New export: **`VectorIndexNotReadyError`**;
|
||||||
|
`getIndexStatus()` gains additive fields (`rebuildFailed`, `rebuildError?`, `degradedIds`, per-index
|
||||||
|
`ready?`). No breaking API change. Each fix ships with a dedicated regression test.
|
||||||
|
|
||||||
## v8.2.7 — 2026-07-13 (loadBinaryBlob fault-propagation — the lockstep completion of 8.2.6)
|
## v8.2.7 — 2026-07-13 (loadBinaryBlob fault-propagation — the lockstep completion of 8.2.6)
|
||||||
|
|
||||||
Completes the Pass-1 spine hardening. `loadBinaryBlob` (the raw-blob read a native accelerator mmaps
|
Completes the Pass-1 spine hardening. `loadBinaryBlob` (the raw-blob read a native accelerator mmaps
|
||||||
|
|
|
||||||
215
src/brainy.ts
215
src/brainy.ts
|
|
@ -175,7 +175,8 @@ import {
|
||||||
} from './events/changeFeed.js'
|
} from './events/changeFeed.js'
|
||||||
import { isDeterministicEmbedMode } from './embeddings/deterministicEmbedMode.js'
|
import { isDeterministicEmbedMode } from './embeddings/deterministicEmbedMode.js'
|
||||||
import { GenerationConflictError, StoreInconsistentError } from './db/errors.js'
|
import { GenerationConflictError, StoreInconsistentError } from './db/errors.js'
|
||||||
import { BrainyError, GraphIndexNotReadyError, MetadataIndexNotReadyError, MigrationInProgressError } from './errors/brainyError.js'
|
import { BrainyError, GraphIndexNotReadyError, MetadataIndexNotReadyError, MigrationInProgressError, VectorIndexNotReadyError } from './errors/brainyError.js'
|
||||||
|
import { assessIndexReadiness } from './utils/indexReadiness.js'
|
||||||
import { MemoryStorage } from './storage/adapters/memoryStorage.js'
|
import { MemoryStorage } from './storage/adapters/memoryStorage.js'
|
||||||
import type {
|
import type {
|
||||||
CompactHistoryOptions,
|
CompactHistoryOptions,
|
||||||
|
|
@ -500,6 +501,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
private _metadataVerified = false
|
private _metadataVerified = false
|
||||||
/** Re-entrancy guard for {@link verifyMetadataLive}. */
|
/** Re-entrancy guard for {@link verifyMetadataLive}. */
|
||||||
private _metadataVerifying = false
|
private _metadataVerifying = false
|
||||||
|
/** Vector-index cold-read guard: verified-serving this session (one-shot). */
|
||||||
|
private _vectorVerified = false
|
||||||
|
/** Re-entrancy guard for {@link verifyVectorLive}. */
|
||||||
|
private _vectorVerifying = false
|
||||||
/**
|
/**
|
||||||
* Coordinated migration LOCK (#18): dedup guards so the "upgrading, blocking"
|
* Coordinated migration LOCK (#18): dedup guards so the "upgrading, blocking"
|
||||||
* and "upgrade complete, resumed" lines each log once per migration window,
|
* and "upgrade complete, resumed" lines each log once per migration window,
|
||||||
|
|
@ -3523,6 +3528,142 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description The vector-index counterpart of {@link verifyGraphAdjacencyLive}
|
||||||
|
* / {@link verifyMetadataLive}. On a cold open a native vector provider can
|
||||||
|
* report a non-zero `size()` (its persisted COUNT loaded) yet not have loaded
|
||||||
|
* its serving structure (the mmap/DiskANN graph) — so a pure semantic
|
||||||
|
* `find({ query })` silently returns `[]`. A pure semantic query has
|
||||||
|
* `hasFilterCriteria === false`, so the metadata guard never fires; this guard
|
||||||
|
* closes that gap. Run one-shot on the first vector/proximity search:
|
||||||
|
* - **Preferred (honest signal):** the provider exposes `isReady()`. `false`
|
||||||
|
* → rebuild from storage, re-check; if still `false`, throw
|
||||||
|
* {@link VectorIndexNotReadyError} rather than serving `[]`.
|
||||||
|
* - **Fallback (no `isReady()`):** a KNOWN persisted vector (sampled +
|
||||||
|
* hydrated) is searched against the index; if it does not self-match, the
|
||||||
|
* serving structure did not load — rebuild + re-probe, else throw.
|
||||||
|
* Inconclusive cases (empty store, no probeable vector, `size()===0` — where
|
||||||
|
* the JS baseline's cold load is `ensureIndexesLoaded`'s job) are treated as
|
||||||
|
* live: never a false rebuild. A migrating provider is skipped (it owns its
|
||||||
|
* locked rebuild).
|
||||||
|
* @returns `'live'` when the index serves, `'rebuilt'` when a rebuild restored it.
|
||||||
|
*/
|
||||||
|
private async verifyVectorLive(): Promise<'live' | 'rebuilt'> {
|
||||||
|
if (this._vectorVerified) return 'live'
|
||||||
|
// Migration LOCK (#18): a migrating provider owns its in-place rebuild.
|
||||||
|
if (this.providerIsMigrating(this.index)) return 'live'
|
||||||
|
// Re-entrancy: rebuild() can trigger reads that call back into this guard.
|
||||||
|
if (this._vectorVerifying) return 'live'
|
||||||
|
this._vectorVerifying = true
|
||||||
|
try {
|
||||||
|
// ── Strategy 1: honest isReady() signal (native provider) ──────────────
|
||||||
|
const readiness = assessIndexReadiness(this.index)
|
||||||
|
if (readiness !== 'unknown') {
|
||||||
|
if (readiness === 'ready') {
|
||||||
|
this._vectorVerified = true
|
||||||
|
return 'live'
|
||||||
|
}
|
||||||
|
// Not ready: the serving structure did not load on open. Rebuild.
|
||||||
|
if (!this.config.silent) {
|
||||||
|
console.warn(
|
||||||
|
`[Brainy] Vector index reports not-ready (isReady() === false) — the persisted ` +
|
||||||
|
`vector index did not load on open. Rebuilding from storage…`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
await this.index.rebuild()
|
||||||
|
if (assessIndexReadiness(this.index) === 'ready') {
|
||||||
|
this._vectorVerified = true
|
||||||
|
return 'rebuilt'
|
||||||
|
}
|
||||||
|
throw new VectorIndexNotReadyError(
|
||||||
|
`Vector index reports not-ready even after a rebuild — semantic find({ query }) and ` +
|
||||||
|
`proximity search cannot be served reliably for this brain (a silent empty result ` +
|
||||||
|
`would misrepresent existing data).`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Strategy 2: known-vector probe (providers without isReady()) ───────
|
||||||
|
const claimed = this.index.size()
|
||||||
|
if (!claimed || claimed <= 0) return 'live' // JS cold path is ensureIndexesLoaded's job
|
||||||
|
|
||||||
|
const probe = await this.pickVectorProbe()
|
||||||
|
if (!probe) {
|
||||||
|
// Empty store, or nothing with a probeable vector — inconclusive.
|
||||||
|
this._vectorVerified = true
|
||||||
|
return 'live'
|
||||||
|
}
|
||||||
|
const p = probe
|
||||||
|
|
||||||
|
const probeServes = async (): Promise<boolean> => {
|
||||||
|
// The failure mode we guard is the SILENT EMPTY result: a cold index that
|
||||||
|
// loaded its COUNT but not its serving structure returns `[]` for a
|
||||||
|
// known-present vector, while a warm index returns at least one hit. We
|
||||||
|
// check for a NON-EMPTY result, NOT an exact self-match — HNSW is
|
||||||
|
// approximate and `get()` may return a re-hydrated/normalized vector, so
|
||||||
|
// demanding the exact self as top-1 would false-positive on a perfectly
|
||||||
|
// healthy index (and wrongly rebuild → throw).
|
||||||
|
const hits = await this.index.search(p.vector, 1)
|
||||||
|
return hits.length > 0
|
||||||
|
}
|
||||||
|
void p.id // probe keyed on the vector; id retained for diagnostics only
|
||||||
|
|
||||||
|
if (await probeServes()) {
|
||||||
|
this._vectorVerified = true
|
||||||
|
return 'live' // serving structure is live — the common case
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!this.config.silent) {
|
||||||
|
console.warn(
|
||||||
|
`[Brainy] Vector index reports ${claimed} vector(s) but a known persisted vector ` +
|
||||||
|
`returns no results — the serving structure did not load on open. Rebuilding…`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
await this.index.rebuild()
|
||||||
|
|
||||||
|
if (await probeServes()) {
|
||||||
|
this._vectorVerified = true
|
||||||
|
return 'rebuilt'
|
||||||
|
}
|
||||||
|
throw new VectorIndexNotReadyError(
|
||||||
|
`Vector index reports ${claimed} vector(s) but a known persisted vector returns no ` +
|
||||||
|
`results even after a rebuild — semantic find({ query }) cannot be served reliably ` +
|
||||||
|
`for this brain (a silent empty result would misrepresent existing data).`
|
||||||
|
)
|
||||||
|
} catch (err) {
|
||||||
|
if (err instanceof VectorIndexNotReadyError) throw err
|
||||||
|
// A transient probe/rebuild failure must not break the query NOR mask as
|
||||||
|
// "no data". Allow a re-check on the next vector read and fall through.
|
||||||
|
this._vectorVerified = false
|
||||||
|
if (!this.config.silent) {
|
||||||
|
console.warn(`[Brainy] Vector consistency check skipped (transient): ${err}`)
|
||||||
|
}
|
||||||
|
return 'live'
|
||||||
|
} finally {
|
||||||
|
this._vectorVerifying = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Sample a KNOWN persisted noun and hydrate its vector, to probe
|
||||||
|
* the vector index with. `get()` omits vectors by default, so this passes
|
||||||
|
* `{ includeVectors: true }`. Samples a few (a system-only / vectorless entity
|
||||||
|
* must not make every open inconclusive). Returns `null` when nothing has a
|
||||||
|
* probeable vector.
|
||||||
|
*/
|
||||||
|
private async pickVectorProbe(): Promise<{ id: string; vector: number[] } | null> {
|
||||||
|
const sample = await this.storage.getNouns({ pagination: { limit: 5, offset: 0 } })
|
||||||
|
for (const noun of sample.items ?? []) {
|
||||||
|
const id = (noun as { id?: string }).id
|
||||||
|
if (!id) continue
|
||||||
|
const full = await this.get(id, { includeVectors: true })
|
||||||
|
const vector = (full as { vector?: number[] } | null)?.vector
|
||||||
|
if (Array.isArray(vector) && vector.length > 0) {
|
||||||
|
return { id, vector }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
|
||||||
// -------------------------------------------------------------------------
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -10175,17 +10316,32 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
migrating: boolean
|
migrating: boolean
|
||||||
/** Structured progress while {@link migrating}; absent otherwise. */
|
/** Structured progress while {@link migrating}; absent otherwise. */
|
||||||
migration?: MigrationProgress
|
migration?: MigrationProgress
|
||||||
|
/** `true` when a non-fatal index rebuild failed at init() (degraded — queries
|
||||||
|
* may be incomplete). Folds in the same `_indexRebuildFailed` signal that
|
||||||
|
* {@link validateIndexConsistency} / {@link checkHealth} already expose. */
|
||||||
|
rebuildFailed: boolean
|
||||||
|
/** The rebuild failure message when {@link rebuildFailed}; absent otherwise. */
|
||||||
|
rebuildError?: string
|
||||||
|
/** Count of records committed via adopt-forward failed-rollback recovery whose
|
||||||
|
* derived index may be incomplete until repairIndex() (the 8.2.6 degraded
|
||||||
|
* set). Non-zero = degraded — a readiness probe should not report 200-ready. */
|
||||||
|
degradedIds: number
|
||||||
hnswIndex: {
|
hnswIndex: {
|
||||||
size: number
|
size: number
|
||||||
|
/** Honest "serving" — the provider's `isReady()` when exposed, else `size>0`. */
|
||||||
populated: boolean
|
populated: boolean
|
||||||
|
/** The provider's honest `isReady()` signal; absent when unexposed. */
|
||||||
|
ready?: boolean
|
||||||
}
|
}
|
||||||
metadataIndex: {
|
metadataIndex: {
|
||||||
entries: number
|
entries: number
|
||||||
populated: boolean
|
populated: boolean
|
||||||
|
ready?: boolean
|
||||||
}
|
}
|
||||||
graphIndex: {
|
graphIndex: {
|
||||||
relationships: number
|
relationships: number
|
||||||
populated: boolean
|
populated: boolean
|
||||||
|
ready?: boolean
|
||||||
}
|
}
|
||||||
storage: {
|
storage: {
|
||||||
totalEntities: number
|
totalEntities: number
|
||||||
|
|
@ -10200,6 +10356,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
lazyRebuildCompleted: this.lazyRebuildCompleted,
|
lazyRebuildCompleted: this.lazyRebuildCompleted,
|
||||||
disableAutoRebuild: this.config.disableAutoRebuild || false,
|
disableAutoRebuild: this.config.disableAutoRebuild || false,
|
||||||
migrating: false,
|
migrating: false,
|
||||||
|
rebuildFailed: this._indexRebuildFailed != null,
|
||||||
|
...(this._indexRebuildFailed ? { rebuildError: this._indexRebuildFailed.message } : {}),
|
||||||
|
degradedIds: this._indexDegradedIds.size,
|
||||||
hnswIndex: { size: 0, populated: false },
|
hnswIndex: { size: 0, populated: false },
|
||||||
metadataIndex: { entries: 0, populated: false },
|
metadataIndex: { entries: 0, populated: false },
|
||||||
graphIndex: { relationships: 0, populated: false },
|
graphIndex: { relationships: 0, populated: false },
|
||||||
|
|
@ -10211,6 +10370,20 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
const hnswSize = this.index.size()
|
const hnswSize = this.index.size()
|
||||||
const graphSize = await this.graphIndex.size()
|
const graphSize = await this.graphIndex.size()
|
||||||
|
|
||||||
|
// Honest readiness: when a provider exposes isReady(), it is the truth of
|
||||||
|
// whether the index actually SERVES (a native index can report a non-zero
|
||||||
|
// size/count yet not have loaded its serving structure). `populated` reflects
|
||||||
|
// that honest signal when present, falling back to size>0 only for providers
|
||||||
|
// that do not expose isReady(). Mirrors validateIndexConsistency/checkHealth,
|
||||||
|
// which already fold in the same signals.
|
||||||
|
const readyOf = (p: unknown): boolean | undefined => {
|
||||||
|
const r = assessIndexReadiness(p)
|
||||||
|
return r === 'unknown' ? undefined : r === 'ready'
|
||||||
|
}
|
||||||
|
const hnswReady = readyOf(this.index)
|
||||||
|
const metadataReady = readyOf(this.metadataIndex)
|
||||||
|
const graphReady = readyOf(this.graphIndex)
|
||||||
|
|
||||||
// Check storage entity count
|
// Check storage entity count
|
||||||
let storageEntityCount = 0
|
let storageEntityCount = 0
|
||||||
try {
|
try {
|
||||||
|
|
@ -10224,18 +10397,29 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
initialized: this.initialized,
|
initialized: this.initialized,
|
||||||
lazyRebuildCompleted: this.lazyRebuildCompleted,
|
lazyRebuildCompleted: this.lazyRebuildCompleted,
|
||||||
disableAutoRebuild: this.config.disableAutoRebuild || false,
|
disableAutoRebuild: this.config.disableAutoRebuild || false,
|
||||||
|
// A non-fatal index-rebuild failure recorded at init(), or adopt-forward
|
||||||
|
// degraded ids, are degraded states (queries may be incomplete) — surface
|
||||||
|
// them here alongside the same signals validateIndexConsistency()/
|
||||||
|
// checkHealth() already expose, so a readiness probe never reports 200-ready
|
||||||
|
// over a known-degraded index.
|
||||||
|
rebuildFailed: this._indexRebuildFailed != null,
|
||||||
|
...(this._indexRebuildFailed ? { rebuildError: this._indexRebuildFailed.message } : {}),
|
||||||
|
degradedIds: this._indexDegradedIds.size,
|
||||||
...this.migrationSnapshot(),
|
...this.migrationSnapshot(),
|
||||||
hnswIndex: {
|
hnswIndex: {
|
||||||
size: hnswSize,
|
size: hnswSize,
|
||||||
populated: hnswSize > 0
|
populated: hnswReady ?? hnswSize > 0,
|
||||||
|
...(hnswReady !== undefined ? { ready: hnswReady } : {})
|
||||||
},
|
},
|
||||||
metadataIndex: {
|
metadataIndex: {
|
||||||
entries: metadataStats.totalEntries,
|
entries: metadataStats.totalEntries,
|
||||||
populated: metadataStats.totalEntries > 0
|
populated: metadataReady ?? metadataStats.totalEntries > 0,
|
||||||
|
...(metadataReady !== undefined ? { ready: metadataReady } : {})
|
||||||
},
|
},
|
||||||
graphIndex: {
|
graphIndex: {
|
||||||
relationships: graphSize,
|
relationships: graphSize,
|
||||||
populated: graphSize > 0
|
populated: graphReady ?? graphSize > 0,
|
||||||
|
...(graphReady !== undefined ? { ready: graphReady } : {})
|
||||||
},
|
},
|
||||||
storage: {
|
storage: {
|
||||||
totalEntities: storageEntityCount
|
totalEntities: storageEntityCount
|
||||||
|
|
@ -12863,6 +13047,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
candidateIds?: string[],
|
candidateIds?: string[],
|
||||||
allowedIds?: OpaqueIdSet
|
allowedIds?: OpaqueIdSet
|
||||||
): Promise<Result<T>[]> {
|
): Promise<Result<T>[]> {
|
||||||
|
// Vector cold-read guard: before trusting a semantic/vector result, verify the
|
||||||
|
// vector index actually SERVES a known persisted vector (one-shot per brain).
|
||||||
|
// A pure semantic find({ query }) has no filter, so verifyMetadataLive never
|
||||||
|
// fires — a cold native index that loaded its COUNT but not its serving
|
||||||
|
// structure would return a silent []. This self-heals (rebuild) or throws
|
||||||
|
// VectorIndexNotReadyError instead.
|
||||||
|
await this.verifyVectorLive()
|
||||||
|
|
||||||
const vector = params.vector || (await this.embed(params.query!))
|
const vector = params.vector || (await this.embed(params.query!))
|
||||||
const limit = params.limit || 10
|
const limit = params.limit || 10
|
||||||
|
|
||||||
|
|
@ -12903,6 +13095,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
private async executeProximitySearch(params: FindParams<T>): Promise<Result<T>[]> {
|
private async executeProximitySearch(params: FindParams<T>): Promise<Result<T>[]> {
|
||||||
if (!params.near) return []
|
if (!params.near) return []
|
||||||
|
|
||||||
|
// Vector cold-read guard (see executeVectorSearch): proximity search also
|
||||||
|
// hits this.index.search — verify it serves before trusting an empty result.
|
||||||
|
await this.verifyVectorLive()
|
||||||
|
|
||||||
// Teaching error: without an anchor id the constraint is meaningless, and
|
// Teaching error: without an anchor id the constraint is meaningless, and
|
||||||
// letting it fall through produces an opaque storage-layer sharding error.
|
// letting it fall through produces an opaque storage-layer sharding error.
|
||||||
if (!params.near.id) {
|
if (!params.near.id) {
|
||||||
|
|
@ -14157,8 +14353,15 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// If indexes already populated, mark as complete and skip
|
// If indexes already populated AND honestly serving, mark complete and skip.
|
||||||
if (this.index.size() > 0) {
|
// Honest gate: when the provider exposes isReady(), that REPLACES the size()>0
|
||||||
|
// proxy (a native index can report a non-zero size while its serving structure
|
||||||
|
// is not loaded — the silent-empty cold-load class). A not-ready provider falls
|
||||||
|
// through so the rebuild path can load it; verifyVectorLive() is the query-time
|
||||||
|
// backstop either way. Providers without isReady() keep the size() heuristic
|
||||||
|
// (the JS index's size()>0 genuinely means loaded).
|
||||||
|
const vectorReadiness = assessIndexReadiness(this.index)
|
||||||
|
if (vectorReadiness === 'ready' || (vectorReadiness === 'unknown' && this.index.size() > 0)) {
|
||||||
this.lazyRebuildCompleted = true
|
this.lazyRebuildCompleted = true
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -14,6 +14,7 @@ export type BrainyErrorType =
|
||||||
| 'FIELD_NOT_INDEXED'
|
| 'FIELD_NOT_INDEXED'
|
||||||
| 'GRAPH_INDEX_NOT_READY'
|
| 'GRAPH_INDEX_NOT_READY'
|
||||||
| 'METADATA_INDEX_NOT_READY'
|
| 'METADATA_INDEX_NOT_READY'
|
||||||
|
| 'VECTOR_INDEX_NOT_READY'
|
||||||
| 'MIGRATION_IN_PROGRESS'
|
| 'MIGRATION_IN_PROGRESS'
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -280,6 +281,32 @@ export class MetadataIndexNotReadyError extends BrainyError {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Thrown when the vector index reports vectors (`size() > 0` or, on a native
|
||||||
|
* provider, `isReady() === false`) but cannot return a KNOWN persisted vector
|
||||||
|
* even after a rebuild — i.e. the semantic serving structure did not load on a
|
||||||
|
* cold open and could not be restored. The vector-search counterpart of
|
||||||
|
* {@link GraphIndexNotReadyError} / {@link MetadataIndexNotReadyError}: it
|
||||||
|
* replaces the silent-empty failure mode (a cold `find({ query })` returning
|
||||||
|
* `[]` indistinguishable from "no similar data") with a loud, catchable error,
|
||||||
|
* so a consumer never renders "nothing found" over data that is simply
|
||||||
|
* not-yet-warm.
|
||||||
|
*
|
||||||
|
* Detected once per brain by a known-vector serving probe on the first
|
||||||
|
* semantic / proximity `find()`; brainy self-heals (rebuilds the index from the
|
||||||
|
* canonical records) first and only raises this if the rebuild still cannot
|
||||||
|
* serve the known vector.
|
||||||
|
*/
|
||||||
|
export class VectorIndexNotReadyError extends BrainyError {
|
||||||
|
constructor(message: string, originalError?: Error) {
|
||||||
|
super(message, 'VECTOR_INDEX_NOT_READY', false, originalError)
|
||||||
|
this.name = 'VectorIndexNotReadyError'
|
||||||
|
if (Error.captureStackTrace) {
|
||||||
|
Error.captureStackTrace(this, VectorIndexNotReadyError)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Thrown when a data-plane read or write is issued against a brain that is
|
* Thrown when a data-plane read or write is issued against a brain that is
|
||||||
* running its one-time, automatic 7.x → 8.0 on-disk upgrade — the coordinated
|
* running its one-time, automatic 7.x → 8.0 on-disk upgrade — the coordinated
|
||||||
|
|
|
||||||
|
|
@ -150,7 +150,7 @@ export { EntityNotFoundError, RelationNotFoundError } from './errors/notFound.js
|
||||||
|
|
||||||
// Base error + typed migration-lock error — thrown by any data-plane call while a
|
// Base error + typed migration-lock error — thrown by any data-plane call while a
|
||||||
// brain runs its one-time 7.x→8.0 upgrade; catch to answer HTTP 503 + Retry-After.
|
// brain runs its one-time 7.x→8.0 upgrade; catch to answer HTTP 503 + Retry-After.
|
||||||
export { BrainyError, MigrationInProgressError, GraphIndexNotReadyError, MetadataIndexNotReadyError } from './errors/brainyError.js'
|
export { BrainyError, MigrationInProgressError, GraphIndexNotReadyError, MetadataIndexNotReadyError, VectorIndexNotReadyError } from './errors/brainyError.js'
|
||||||
export type { BrainyErrorType } from './errors/brainyError.js'
|
export type { BrainyErrorType } from './errors/brainyError.js'
|
||||||
|
|
||||||
// ============= 8.0 Db API — generational MVCC =============
|
// ============= 8.0 Db API — generational MVCC =============
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,7 @@
|
||||||
|
|
||||||
import { GraphAdjacencyIndex } from '../graph/graphAdjacencyIndex.js'
|
import { GraphAdjacencyIndex } from '../graph/graphAdjacencyIndex.js'
|
||||||
import type { GraphEntityIdResolver } from '../graph/graphAdjacencyIndex.js'
|
import type { GraphEntityIdResolver } from '../graph/graphAdjacencyIndex.js'
|
||||||
|
import { assessIndexReadiness } from '../utils/indexReadiness.js'
|
||||||
|
|
||||||
import {
|
import {
|
||||||
GraphVerb,
|
GraphVerb,
|
||||||
|
|
@ -248,6 +249,8 @@ function isCountedVisibility(visibility: unknown): boolean {
|
||||||
*/
|
*/
|
||||||
export abstract class BaseStorage extends BaseStorageAdapter {
|
export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
protected isInitialized = false
|
protected isInitialized = false
|
||||||
|
/** One-shot guard so the graph fast-path cold-load probe runs once per adapter. */
|
||||||
|
private _graphFastPathProbed = false
|
||||||
protected graphIndex?: GraphAdjacencyIndex
|
protected graphIndex?: GraphAdjacencyIndex
|
||||||
protected graphIndexPromise?: Promise<GraphAdjacencyIndex>
|
protected graphIndexPromise?: Promise<GraphAdjacencyIndex>
|
||||||
/**
|
/**
|
||||||
|
|
@ -2681,6 +2684,61 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
|
|
||||||
return this.graphIndex
|
return this.graphIndex
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description One-shot cold-load self-heal for a graph provider that does NOT
|
||||||
|
* expose the honest `isReady()` signal. A native provider wired via
|
||||||
|
* {@link setGraphIndex} bypasses `_initializeGraphIndex`'s `size()===0`
|
||||||
|
* self-heal, so a cold-open that loaded the COUNT but not the source→target
|
||||||
|
* adjacency would let the fast path return a silent `[]`. This probe (run once,
|
||||||
|
* before the fast path) samples ONE known persisted edge: if the index claims
|
||||||
|
* relationships (`size() > 0`) yet that edge's source resolves to no verb ints,
|
||||||
|
* the adjacency did not load → rebuild once from storage. Providers that expose
|
||||||
|
* `isReady()` are covered by the honest gate in
|
||||||
|
* getVerbsBy{Source,Target}_internal and skip this probe; the JS index (which
|
||||||
|
* self-heals in `_initializeGraphIndex`) resolves its known edge and no-ops.
|
||||||
|
*/
|
||||||
|
private async ensureGraphFastPathProbed(): Promise<void> {
|
||||||
|
if (this._graphFastPathProbed) return
|
||||||
|
const index = this.graphIndex
|
||||||
|
const resolver = this.graphEntityIdResolver
|
||||||
|
if (!index || !index.isInitialized || !resolver) return
|
||||||
|
// isReady()-capable providers report serving honestly — the gate handles them.
|
||||||
|
if (assessIndexReadiness(index) !== 'unknown') {
|
||||||
|
this._graphFastPathProbed = true
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if (index.size() <= 0) {
|
||||||
|
this._graphFastPathProbed = true
|
||||||
|
return // no edges claimed — nothing to verify
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const sample = await this.getVerbs({ pagination: { limit: 1 } })
|
||||||
|
const verb = sample.items?.[0]
|
||||||
|
if (!verb || !verb.sourceId) {
|
||||||
|
this._graphFastPathProbed = true
|
||||||
|
return // no edges in storage — stale count, harmless
|
||||||
|
}
|
||||||
|
const sourceInt = resolver.getInt(verb.sourceId)
|
||||||
|
if (sourceInt === undefined) {
|
||||||
|
this._graphFastPathProbed = true
|
||||||
|
return // foreign / unmapped sample — inconclusive, never a false rebuild
|
||||||
|
}
|
||||||
|
const verbInts = await index.getVerbIdsBySource(BigInt(sourceInt))
|
||||||
|
if (verbInts.length === 0) {
|
||||||
|
prodLog.warn(
|
||||||
|
`[BaseStorage] Graph fast path: index reports ${index.size()} relationship(s) but a ` +
|
||||||
|
`known persisted edge resolves to none — the persisted adjacency did not load. ` +
|
||||||
|
`Rebuilding once from storage.`
|
||||||
|
)
|
||||||
|
await index.rebuild()
|
||||||
|
}
|
||||||
|
this._graphFastPathProbed = true
|
||||||
|
} catch (error) {
|
||||||
|
// Transient probe/rebuild failure must not break the read; re-arm for next call.
|
||||||
|
prodLog.debug(`[BaseStorage] Graph fast-path probe skipped (transient): ${error}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
/**
|
/**
|
||||||
* Clear all data from storage
|
* Clear all data from storage
|
||||||
* This method should be implemented by each specific adapter
|
* This method should be implemented by each specific adapter
|
||||||
|
|
@ -4199,13 +4257,22 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
sourceId: string
|
sourceId: string
|
||||||
): Promise<HNSWVerbWithMetadata[]> {
|
): Promise<HNSWVerbWithMetadata[]> {
|
||||||
await this.ensureInitialized()
|
await this.ensureInitialized()
|
||||||
|
await this.ensureGraphFastPathProbed()
|
||||||
|
|
||||||
prodLog.debug(`[BaseStorage] getVerbsBySource_internal: sourceId=${sourceId}, graphIndex=${!!this.graphIndex}, isInitialized=${this.graphIndex?.isInitialized}`)
|
prodLog.debug(`[BaseStorage] getVerbsBySource_internal: sourceId=${sourceId}, graphIndex=${!!this.graphIndex}, isInitialized=${this.graphIndex?.isInitialized}`)
|
||||||
|
|
||||||
// Fast path - use GraphAdjacencyIndex if available (lazy-loaded).
|
// Fast path - use GraphAdjacencyIndex if available (lazy-loaded).
|
||||||
// 8.0 BigInt boundary: convert the UUID to an entity int up front and
|
// 8.0 BigInt boundary: convert the UUID to an entity int up front and
|
||||||
// resolve returned verb ints back to verb-id strings.
|
// resolve returned verb ints back to verb-id strings.
|
||||||
if (this.graphIndex && this.graphIndex.isInitialized && this.graphEntityIdResolver) {
|
// Honest gate: a provider that exposes isReady() and reports not-ready
|
||||||
|
// (count/manifest loaded but source→target edges NOT) is SKIPPED so we fall
|
||||||
|
// to the correct-but-slower canonical shard scan below instead of a silent [].
|
||||||
|
if (
|
||||||
|
this.graphIndex &&
|
||||||
|
this.graphIndex.isInitialized &&
|
||||||
|
this.graphEntityIdResolver &&
|
||||||
|
assessIndexReadiness(this.graphIndex) !== 'not-ready'
|
||||||
|
) {
|
||||||
try {
|
try {
|
||||||
const sourceInt = this.graphEntityIdResolver.getInt(sourceId)
|
const sourceInt = this.graphEntityIdResolver.getInt(sourceId)
|
||||||
if (sourceInt === undefined) {
|
if (sourceInt === undefined) {
|
||||||
|
|
@ -4424,11 +4491,19 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
||||||
targetId: string
|
targetId: string
|
||||||
): Promise<HNSWVerbWithMetadata[]> {
|
): Promise<HNSWVerbWithMetadata[]> {
|
||||||
await this.ensureInitialized()
|
await this.ensureInitialized()
|
||||||
|
await this.ensureGraphFastPathProbed()
|
||||||
|
|
||||||
// Fast path - use GraphAdjacencyIndex if available (lazy-loaded).
|
// Fast path - use GraphAdjacencyIndex if available (lazy-loaded).
|
||||||
// 8.0 BigInt boundary: convert the UUID to an entity int up front and
|
// 8.0 BigInt boundary: convert the UUID to an entity int up front and
|
||||||
// resolve returned verb ints back to verb-id strings.
|
// resolve returned verb ints back to verb-id strings.
|
||||||
if (this.graphIndex && this.graphIndex.isInitialized && this.graphEntityIdResolver) {
|
// Honest gate: a not-ready provider (count loaded but edges NOT) is SKIPPED so
|
||||||
|
// we fall to the correct-but-slower canonical shard scan instead of a silent [].
|
||||||
|
if (
|
||||||
|
this.graphIndex &&
|
||||||
|
this.graphIndex.isInitialized &&
|
||||||
|
this.graphEntityIdResolver &&
|
||||||
|
assessIndexReadiness(this.graphIndex) !== 'not-ready'
|
||||||
|
) {
|
||||||
try {
|
try {
|
||||||
const targetInt = this.graphEntityIdResolver.getInt(targetId)
|
const targetInt = this.graphEntityIdResolver.getInt(targetId)
|
||||||
if (targetInt === undefined) {
|
if (targetInt === undefined) {
|
||||||
|
|
|
||||||
38
src/utils/indexReadiness.ts
Normal file
38
src/utils/indexReadiness.ts
Normal file
|
|
@ -0,0 +1,38 @@
|
||||||
|
/**
|
||||||
|
* @module indexReadiness
|
||||||
|
* @description The single honest-readiness classifier shared by the vector,
|
||||||
|
* graph and metadata index sites. It exists to kill "Pattern A" — the dishonest
|
||||||
|
* readiness proxy where `size() > 0` / `isInitialized` is treated as "this index
|
||||||
|
* actually serves queries." A cold native index that loaded its COUNT but not its
|
||||||
|
* SERVING structure passes those proxies and silently returns `[]`.
|
||||||
|
*
|
||||||
|
* This classifier reads ONLY the provider's OPTIONAL, honest `isReady()` signal
|
||||||
|
* (see {@link import('../plugin.js').VectorIndexProvider.isReady},
|
||||||
|
* {@link import('../plugin.js').GraphIndexProvider.isReady},
|
||||||
|
* {@link import('../plugin.js').MetadataIndexProvider.isReady}). It NEVER inspects
|
||||||
|
* `size()` or `isInitialized`. When `isReady()` is absent, callers must fall back
|
||||||
|
* to a KNOWN-ITEM PROBE (a real search/lookup that must return a known-present
|
||||||
|
* datum) before trusting an empty result — never a `size()` proxy.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** A provider that MAY expose the honest cold-load readiness signal. */
|
||||||
|
export interface MaybeReadyProvider {
|
||||||
|
isReady?: () => boolean
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Three-valued honest-readiness verdict. */
|
||||||
|
export type IndexReadiness = 'ready' | 'not-ready' | 'unknown'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Classify an index provider's honest readiness.
|
||||||
|
* @param provider - Any index provider (vector / graph / metadata) or `null`.
|
||||||
|
* @returns
|
||||||
|
* - `'ready'` when `isReady() === true` (serving structure loaded — trust it);
|
||||||
|
* - `'not-ready'` when `isReady() === false` (count/manifest loaded, NOT serving — rebuild);
|
||||||
|
* - `'unknown'` when the provider exposes no `isReady()` (caller must probe / keep the JS heuristic).
|
||||||
|
*/
|
||||||
|
export function assessIndexReadiness(provider: unknown): IndexReadiness {
|
||||||
|
const p = provider as MaybeReadyProvider | null | undefined
|
||||||
|
if (p == null || typeof p.isReady !== 'function') return 'unknown'
|
||||||
|
return p.isReady() ? 'ready' : 'not-ready'
|
||||||
|
}
|
||||||
59
tests/unit/get-index-status-readiness.test.ts
Normal file
59
tests/unit/get-index-status-readiness.test.ts
Normal file
|
|
@ -0,0 +1,59 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/get-index-status-readiness
|
||||||
|
* @description Pattern-A / Finding 9: getIndexStatus reported `populated: size>0`
|
||||||
|
* and only `migrating`, so a native index that loaded its count but not its
|
||||||
|
* serving structure reported populated:true — a k8s readiness probe would 200 a
|
||||||
|
* brain that serves []. It now folds in the honest isReady() signal + the
|
||||||
|
* _indexRebuildFailed / _indexDegradedIds degraded states (mirroring
|
||||||
|
* validateIndexConsistency / checkHealth).
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, beforeEach } from 'vitest'
|
||||||
|
import { Brainy, NounType } from '../../src/index.js'
|
||||||
|
|
||||||
|
describe('getIndexStatus honest readiness (Finding 9)', () => {
|
||||||
|
let brain: any
|
||||||
|
beforeEach(async () => {
|
||||||
|
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
||||||
|
brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, dimensions: 384 })
|
||||||
|
await brain.init()
|
||||||
|
await brain.add({ data: 'x', type: NounType.Concept })
|
||||||
|
await brain.flush()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a not-ready provider makes populated honest (false) and exposes ready:false', async () => {
|
||||||
|
brain.index.isReady = () => false // count present, serving structure NOT loaded
|
||||||
|
const status = await brain.getIndexStatus()
|
||||||
|
expect(status.hnswIndex.populated).toBe(false)
|
||||||
|
expect(status.hnswIndex.ready).toBe(false)
|
||||||
|
delete brain.index.isReady
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a ready provider reports populated:true + ready:true', async () => {
|
||||||
|
brain.index.isReady = () => true
|
||||||
|
const status = await brain.getIndexStatus()
|
||||||
|
expect(status.hnswIndex.populated).toBe(true)
|
||||||
|
expect(status.hnswIndex.ready).toBe(true)
|
||||||
|
delete brain.index.isReady
|
||||||
|
})
|
||||||
|
|
||||||
|
it('rebuildFailed / rebuildError surface the init-degraded state', async () => {
|
||||||
|
brain._indexRebuildFailed = new Error('boom')
|
||||||
|
const status = await brain.getIndexStatus()
|
||||||
|
expect(status.rebuildFailed).toBe(true)
|
||||||
|
expect(status.rebuildError).toBe('boom')
|
||||||
|
brain._indexRebuildFailed = null
|
||||||
|
})
|
||||||
|
|
||||||
|
it('degradedIds surfaces the adopt-forward degraded set', async () => {
|
||||||
|
brain._indexDegradedIds.add('00000000-0000-4000-8000-0000000000de')
|
||||||
|
const status = await brain.getIndexStatus()
|
||||||
|
expect(status.degradedIds).toBe(1)
|
||||||
|
brain._indexDegradedIds.clear()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a JS-baseline provider (no isReady) omits ready and falls back to size>0', async () => {
|
||||||
|
const status = await brain.getIndexStatus()
|
||||||
|
expect(status.hnswIndex.ready).toBeUndefined()
|
||||||
|
expect(status.hnswIndex.populated).toBe(brain.index.size() > 0)
|
||||||
|
})
|
||||||
|
})
|
||||||
95
tests/unit/graph/graph-fastpath-honest-readiness.test.ts
Normal file
95
tests/unit/graph/graph-fastpath-honest-readiness.test.ts
Normal file
|
|
@ -0,0 +1,95 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/graph/graph-fastpath-honest-readiness
|
||||||
|
* @description Pattern-A / Finding 2: getVerbsBySource/ByTarget gated the graph
|
||||||
|
* fast path on `graphIndex.isInitialized` (a proxy that reads true once the
|
||||||
|
* manifest/count loaded even if the source→target adjacency did NOT → the fast
|
||||||
|
* path then returned a silent []). The honest gate skips the fast path when the
|
||||||
|
* provider reports isReady()===false, falling to the correct canonical shard
|
||||||
|
* scan; and a one-shot probe self-heals a no-isReady provider whose adjacency
|
||||||
|
* did not cold-load.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, beforeEach } from 'vitest'
|
||||||
|
import { Brainy, NounType, VerbType } from '../../../src/index.js'
|
||||||
|
|
||||||
|
describe('graph fast-path honest readiness (Finding 2)', () => {
|
||||||
|
let brain: any
|
||||||
|
let storage: any
|
||||||
|
let a: string
|
||||||
|
let b: string
|
||||||
|
let c: string
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
||||||
|
brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, dimensions: 384 })
|
||||||
|
await brain.init()
|
||||||
|
a = await brain.add({ data: 'a', type: NounType.Concept })
|
||||||
|
b = await brain.add({ data: 'b', type: NounType.Concept })
|
||||||
|
c = await brain.add({ data: 'c', type: NounType.Concept })
|
||||||
|
await brain.relate({ from: a, to: b, type: VerbType.Contains })
|
||||||
|
await brain.relate({ from: a, to: c, type: VerbType.Contains })
|
||||||
|
await brain.flush()
|
||||||
|
storage = brain.storage
|
||||||
|
// Warm + wire the storage graph index (lazy) so we can stub it.
|
||||||
|
await storage.getVerbsBySource(a)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('not-ready provider → shard scan returns the REAL edges, not a silent []', async () => {
|
||||||
|
const gi = storage.graphIndex
|
||||||
|
// Simulate a cold native provider: count/manifest loaded (isInitialized) but
|
||||||
|
// the source→target adjacency is NOT (isReady false; fast-path lookup empty).
|
||||||
|
gi.isReady = () => false
|
||||||
|
const origBySource = gi.getVerbIdsBySource.bind(gi)
|
||||||
|
gi.getVerbIdsBySource = async () => [] // cold adjacency — old fast path trusted this
|
||||||
|
storage._graphFastPathProbed = true // isolate the GATE (probe skips isReady-capable anyway)
|
||||||
|
try {
|
||||||
|
const verbs = await storage.getVerbsBySource(a)
|
||||||
|
// Honest gate skipped the cold fast path → canonical shard scan → real edges.
|
||||||
|
expect(verbs.length).toBe(2)
|
||||||
|
} finally {
|
||||||
|
delete gi.isReady
|
||||||
|
gi.getVerbIdsBySource = origBySource
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('ready provider → fast path is used and an empty result is trusted', async () => {
|
||||||
|
const gi = storage.graphIndex
|
||||||
|
gi.isReady = () => true
|
||||||
|
let fastPathCalls = 0
|
||||||
|
const origBySource = gi.getVerbIdsBySource.bind(gi)
|
||||||
|
gi.getVerbIdsBySource = async (...args: any[]) => { fastPathCalls++; return origBySource(...args) }
|
||||||
|
storage._graphFastPathProbed = true
|
||||||
|
try {
|
||||||
|
// `c` has no OUTGOING edges — a genuinely empty result via the fast path.
|
||||||
|
const verbs = await storage.getVerbsBySource(c)
|
||||||
|
expect(verbs).toEqual([])
|
||||||
|
expect(fastPathCalls).toBeGreaterThan(0) // fast path was taken, not a shard scan
|
||||||
|
} finally {
|
||||||
|
delete gi.isReady
|
||||||
|
gi.getVerbIdsBySource = origBySource
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('no-isReady provider whose adjacency did not cold-load → probe rebuilds once', async () => {
|
||||||
|
const gi = storage.graphIndex
|
||||||
|
if ('isReady' in gi) delete gi.isReady // force the "unknown" (no honest signal) path
|
||||||
|
// Force the probe to see a cold adjacency: a known edge resolves to no ints.
|
||||||
|
const origBySource = gi.getVerbIdsBySource.bind(gi)
|
||||||
|
let cold = true
|
||||||
|
gi.getVerbIdsBySource = async (...args: any[]) => (cold ? [] : origBySource(...args))
|
||||||
|
let rebuilds = 0
|
||||||
|
const origRebuild = gi.rebuild.bind(gi)
|
||||||
|
gi.rebuild = async (...args: any[]) => { rebuilds++; cold = false; return origRebuild(...args) }
|
||||||
|
storage._graphFastPathProbed = false // re-arm the one-shot probe
|
||||||
|
try {
|
||||||
|
await storage.getVerbsBySource(a)
|
||||||
|
expect(rebuilds).toBe(1) // probe detected cold adjacency + rebuilt once
|
||||||
|
expect(storage._graphFastPathProbed).toBe(true) // latched — no second rebuild
|
||||||
|
rebuilds = 0
|
||||||
|
await storage.getVerbsByTarget(b)
|
||||||
|
expect(rebuilds).toBe(0) // probe does not re-run
|
||||||
|
} finally {
|
||||||
|
gi.getVerbIdsBySource = origBySource
|
||||||
|
gi.rebuild = origRebuild
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
115
tests/unit/vector-cold-read-guard.test.ts
Normal file
115
tests/unit/vector-cold-read-guard.test.ts
Normal file
|
|
@ -0,0 +1,115 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/vector-cold-read-guard
|
||||||
|
* @description Pattern-A / Finding 1: a pure semantic find({ query }) has no
|
||||||
|
* filter, so verifyMetadataLive never fires — nothing guarded the vector index.
|
||||||
|
* A cold native vector index that loaded its COUNT but not its serving structure
|
||||||
|
* returned a silent []. verifyVectorLive() closes that: honest isReady() first,
|
||||||
|
* else a known-vector self-match probe; self-heal (rebuild) or throw
|
||||||
|
* VectorIndexNotReadyError — never a silent empty result.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, beforeEach } from 'vitest'
|
||||||
|
import { Brainy, NounType, VectorIndexNotReadyError } from '../../src/index.js'
|
||||||
|
|
||||||
|
const V = (): number[] => Array.from({ length: 384 }, (_, i) => Math.sin(i * 0.1) + 0.001)
|
||||||
|
|
||||||
|
describe('Vector cold-read guard (verifyVectorLive) — silent-[] on cold semantic find', () => {
|
||||||
|
let brain: any
|
||||||
|
beforeEach(async () => {
|
||||||
|
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
||||||
|
brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, dimensions: 384 })
|
||||||
|
await brain.init()
|
||||||
|
await brain.add({ vector: V(), type: NounType.Concept, metadata: { status: 'active' } })
|
||||||
|
await brain.add({ vector: V(), type: NounType.Concept, metadata: { status: 'archived' } })
|
||||||
|
await brain.flush()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('warm brain: semantic find is correct and the guard does not rebuild', async () => {
|
||||||
|
const vi = brain.index
|
||||||
|
let rebuilds = 0
|
||||||
|
const origRebuild = vi.rebuild.bind(vi)
|
||||||
|
vi.rebuild = async (...a: any[]) => { rebuilds++; return origRebuild(...a) }
|
||||||
|
await brain.find({ query: 'anything', searchMode: 'semantic', limit: 100 })
|
||||||
|
expect(rebuilds).toBe(0)
|
||||||
|
expect(brain._vectorVerified).toBe(true)
|
||||||
|
vi.rebuild = origRebuild
|
||||||
|
})
|
||||||
|
|
||||||
|
it('cold index: verifyVectorLive self-heals via rebuild — semantic find is correct, NOT silent []', async () => {
|
||||||
|
const vi = brain.index
|
||||||
|
const origSearch = vi.search.bind(vi)
|
||||||
|
const origRebuild = vi.rebuild.bind(vi)
|
||||||
|
let cold = true
|
||||||
|
brain._vectorVerified = false
|
||||||
|
// size()>0 (count present) but search returns nothing until a rebuild warms it.
|
||||||
|
vi.search = async (...a: any[]) => (cold ? [] : origSearch(...a))
|
||||||
|
vi.rebuild = async (...a: any[]) => { await origRebuild(...a); cold = false }
|
||||||
|
try {
|
||||||
|
const res = await brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
|
||||||
|
expect(res.length).toBeGreaterThan(0) // self-healed
|
||||||
|
} finally {
|
||||||
|
vi.search = origSearch; vi.rebuild = origRebuild
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('unrecoverably cold index: semantic find throws VectorIndexNotReadyError', async () => {
|
||||||
|
const vi = brain.index
|
||||||
|
const origSearch = vi.search.bind(vi)
|
||||||
|
const origRebuild = vi.rebuild.bind(vi)
|
||||||
|
brain._vectorVerified = false
|
||||||
|
vi.search = async () => [] // always cold; rebuild can't fix it
|
||||||
|
vi.rebuild = async () => {}
|
||||||
|
try {
|
||||||
|
await expect(
|
||||||
|
brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
|
||||||
|
).rejects.toBeInstanceOf(VectorIndexNotReadyError)
|
||||||
|
} finally {
|
||||||
|
vi.search = origSearch; vi.rebuild = origRebuild
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('native provider reporting isReady()===false rebuilds, then serves', async () => {
|
||||||
|
const vi = brain.index
|
||||||
|
const origRebuild = vi.rebuild.bind(vi)
|
||||||
|
let ready = false
|
||||||
|
brain._vectorVerified = false
|
||||||
|
vi.isReady = () => ready
|
||||||
|
vi.rebuild = async (...a: any[]) => { await origRebuild(...a); ready = true }
|
||||||
|
try {
|
||||||
|
const res = await brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
|
||||||
|
expect(ready).toBe(true) // rebuild ran because isReady() was false
|
||||||
|
expect(res).toBeDefined()
|
||||||
|
} finally {
|
||||||
|
delete vi.isReady; vi.rebuild = origRebuild
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a text-only query does not trigger the vector guard', async () => {
|
||||||
|
brain._vectorVerified = false
|
||||||
|
await brain.find({ query: 'active', searchMode: 'text', limit: 5 })
|
||||||
|
expect(brain._vectorVerified).toBe(false) // executeVectorSearch never called
|
||||||
|
})
|
||||||
|
|
||||||
|
// Regression: the probe must check "returns ANY hit", not an exact self-match —
|
||||||
|
// HNSW is approximate and get() may re-hydrate the vector, so a healthy
|
||||||
|
// many-entity index would false-positive under an exact-self check, wrongly
|
||||||
|
// rebuild, and throw VectorIndexNotReadyError on working data.
|
||||||
|
it('a healthy many-entity brain with distinct vectors serves semantic find, never throws/rebuilds', async () => {
|
||||||
|
const many = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, dimensions: 384 })
|
||||||
|
await many.init()
|
||||||
|
for (let i = 0; i < 25; i++) {
|
||||||
|
const v = Array.from({ length: 384 }, (_, j) => Math.sin((i * 7 + j) * 0.13) + 0.001)
|
||||||
|
await many.add({ vector: v, type: NounType.Concept, metadata: { n: i } })
|
||||||
|
}
|
||||||
|
await many.flush()
|
||||||
|
let rebuilds = 0
|
||||||
|
const vi = (many as any).index
|
||||||
|
const origRebuild = vi.rebuild.bind(vi)
|
||||||
|
vi.rebuild = async (...a: any[]) => { rebuilds++; return origRebuild(...a) }
|
||||||
|
const res = await many.find({ query: 'x', searchMode: 'semantic', limit: 10 })
|
||||||
|
expect(res.length).toBeGreaterThan(0)
|
||||||
|
expect(rebuilds).toBe(0)
|
||||||
|
expect((many as any)._vectorVerified).toBe(true)
|
||||||
|
vi.rebuild = origRebuild
|
||||||
|
await many.close()
|
||||||
|
})
|
||||||
|
})
|
||||||
Loading…
Add table
Add a link
Reference in a new issue