fix: graph adjacency cold-load consistency guard — no more silent [] on connected
A native graph provider can load its relationship COUNT (manifest) on a cold open
but fail to load the source→target adjacency itself (observed on mmap-filesystem:
the SSTable-segment load is swallowed). The result is find({ connected }),
neighbors(), and related() silently returning [] despite persisted edges — and
staying empty after warm-up. Because it is [] not an error, callers cannot tell
real data is missing.
Brainy now runs a one-time consistency check on the first graph read: if the index
reports relationships exist but a known persisted edge's source resolves to no
neighbors, force a rebuild from storage (which re-scans + repopulates the adjacency);
if even that cannot read the edge, throw the new GraphIndexNotReadyError instead of
serving [] as truth. O(1) probe (one verb + one neighbor lookup), cached per brain,
and a no-op once the adjacency is live — so it self-heals the cold-open case and is
loud when it genuinely can't.
Wired into neighbors() + getRelations() (the graph-read primitives behind
find({ connected })). The underlying cold-open load is a native-provider concern
(addressed upstream); this makes the failure self-healing + loud rather than silent.
Tests: tests/unit/brainy/graph-adjacency-cold-load.test.ts (live → no-op; broken →
rebuild; unfixable → throws; size 0 / no edges → no-op; transient failure → re-checks
without breaking the query). Full unit gate green (1531/1531).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
811c7da89e
commit
1694f68419
4 changed files with 240 additions and 0 deletions
26
RELEASES.md
26
RELEASES.md
|
|
@ -10,6 +10,32 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## v7.33.2 — 2026-06-24
|
||||||
|
|
||||||
|
**Affected products:** any consumer using graph traversals — `find({ connected: { from, via } })`,
|
||||||
|
`neighbors()`, `related()` — with a **native graph index** on **`mmap-filesystem`** (or any adapter
|
||||||
|
whose cold-open adjacency load can fail), where a process restart could leave graph queries returning
|
||||||
|
`[]` despite persisted edges. Drop-in; no API or data changes.
|
||||||
|
|
||||||
|
### Fix — graph adjacency no longer silently returns `[]` on a cold open
|
||||||
|
|
||||||
|
On a fresh brain open, a native graph provider can load its relationship **count** (the manifest) but
|
||||||
|
fail to load the source→target **adjacency** itself — so `find({ connected })` / `neighbors()` /
|
||||||
|
`related()` returned an empty array **indistinguishable from "no edges"**, and stayed empty after
|
||||||
|
warm-up. Because it was `[]` and not an error, callers could not tell real data was missing.
|
||||||
|
|
||||||
|
Brainy now runs a one-time consistency check on the first graph read: if the index reports that
|
||||||
|
relationships exist but a known persisted edge's source resolves to **no** neighbors, it **rebuilds
|
||||||
|
the adjacency from storage**. If even a rebuild cannot read the edge, it throws a loud, catchable
|
||||||
|
`GraphIndexNotReadyError` instead of serving `[]` as truth. The probe is O(1) (one edge + one neighbor
|
||||||
|
lookup), cached per brain, and a **no-op once the adjacency is live** — so consumers can return to
|
||||||
|
clean graph queries instead of in-memory-filter workarounds.
|
||||||
|
|
||||||
|
> The underlying cold-open load is a native-provider concern (being addressed upstream); this Brainy
|
||||||
|
> change makes the failure **self-healing and loud** rather than silent.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## v7.33.1 — 2026-06-22
|
## v7.33.1 — 2026-06-22
|
||||||
|
|
||||||
**Affected products:** any consumer on `filesystem` / `mmap-filesystem` storage with **more than one
|
**Affected products:** any consumer on `filesystem` / `mmap-filesystem` storage with **more than one
|
||||||
|
|
|
||||||
|
|
@ -48,6 +48,7 @@ import { MmapVectorBackend } from './hnsw/mmapVectorBackend.js'
|
||||||
import { ConnectionsCodec } from './hnsw/connectionsCodec.js'
|
import { ConnectionsCodec } from './hnsw/connectionsCodec.js'
|
||||||
import { TransactionManager } from './transaction/TransactionManager.js'
|
import { TransactionManager } from './transaction/TransactionManager.js'
|
||||||
import { RevisionConflictError } from './transaction/RevisionConflictError.js'
|
import { RevisionConflictError } from './transaction/RevisionConflictError.js'
|
||||||
|
import { GraphIndexNotReadyError } from './errors/brainyError.js'
|
||||||
import {
|
import {
|
||||||
ValidationConfig,
|
ValidationConfig,
|
||||||
validateAddParams,
|
validateAddParams,
|
||||||
|
|
@ -191,6 +192,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
private _versions?: VersioningAPI
|
private _versions?: VersioningAPI
|
||||||
private _vfs?: VirtualFileSystem
|
private _vfs?: VirtualFileSystem
|
||||||
private _vfsInitialized = false // Track VFS init completion separately
|
private _vfsInitialized = false // Track VFS init completion separately
|
||||||
|
private _graphAdjacencyVerified = false // 7.33.2: one-time graph-adjacency cold-load consistency check
|
||||||
private _hub?: IntegrationHub // Integration Hub for external tools
|
private _hub?: IntegrationHub // Integration Hub for external tools
|
||||||
private _pendingMigrationRunner?: MigrationRunner // Deferred migration runner for large datasets
|
private _pendingMigrationRunner?: MigrationRunner // Deferred migration runner for large datasets
|
||||||
private _aggregationIndex?: AggregationIndex // Incremental aggregation engine
|
private _aggregationIndex?: AggregationIndex // Incremental aggregation engine
|
||||||
|
|
@ -2449,6 +2451,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
paramsOrId?: string | GetRelationsParams
|
paramsOrId?: string | GetRelationsParams
|
||||||
): Promise<Relation<T>[]> {
|
): Promise<Relation<T>[]> {
|
||||||
await this.ensureInitialized()
|
await this.ensureInitialized()
|
||||||
|
await this.ensureGraphAdjacencyConsistent()
|
||||||
|
|
||||||
// Handle string ID shorthand: getRelations(id) -> getRelations({ from: id })
|
// Handle string ID shorthand: getRelations(id) -> getRelations({ from: id })
|
||||||
const params = typeof paramsOrId === 'string'
|
const params = typeof paramsOrId === 'string'
|
||||||
|
|
@ -8122,6 +8125,61 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
* verbType: VerbType.RELATES_TO
|
* verbType: VerbType.RELATES_TO
|
||||||
* })
|
* })
|
||||||
*/
|
*/
|
||||||
|
/**
|
||||||
|
* @description 7.33.2 — guard against a graph index that loaded its relationship
|
||||||
|
* COUNT (manifest) on a cold open but NOT its source→target adjacency, so graph
|
||||||
|
* traversals (`find({ connected })`, `neighbors`, `related`) would silently return
|
||||||
|
* `[]` despite persisted edges. Observed with a native graph provider whose
|
||||||
|
* cold-open SSTable load is swallowed on some storage adapters. Runs ONCE per brain
|
||||||
|
* (cached): if the index claims edges exist but a known persisted edge's source
|
||||||
|
* resolves to no neighbors, force a rebuild from storage; if even that can't read
|
||||||
|
* the edge, throw {@link GraphIndexNotReadyError} rather than serving `[]` as truth.
|
||||||
|
* O(1) (one verb read + one neighbor probe) and a no-op once the adjacency is live.
|
||||||
|
*/
|
||||||
|
private async ensureGraphAdjacencyConsistent(): Promise<void> {
|
||||||
|
if (this._graphAdjacencyVerified) return
|
||||||
|
// Set up-front so a re-entrant call (rebuild → reads) cannot loop.
|
||||||
|
this._graphAdjacencyVerified = true
|
||||||
|
try {
|
||||||
|
const claimed = await this.graphIndex.size()
|
||||||
|
if (!claimed || claimed <= 0) return // no edges claimed — nothing to verify
|
||||||
|
|
||||||
|
const sample = await this.storage.getVerbs({ pagination: { limit: 1 } })
|
||||||
|
const verb = sample.items?.[0]
|
||||||
|
if (!verb || !verb.sourceId) return // no edges in storage — stale count, harmless
|
||||||
|
|
||||||
|
const probe = await this.graphIndex.getNeighbors(verb.sourceId, { limit: 1 })
|
||||||
|
if (probe.length > 0) return // adjacency is live — the common case
|
||||||
|
|
||||||
|
// INCONSISTENT: the index reports edges but a persisted edge's source has none →
|
||||||
|
// the adjacency did not load on open. Rebuild it from storage.
|
||||||
|
if (!this.config.silent) {
|
||||||
|
console.warn(
|
||||||
|
`[Brainy] Graph adjacency reports ${claimed} relationship(s) but a persisted edge ` +
|
||||||
|
`resolves to none — the persisted adjacency did not load on open. Rebuilding from storage…`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
await this.graphIndex.rebuild()
|
||||||
|
|
||||||
|
const reprobe = await this.graphIndex.getNeighbors(verb.sourceId, { limit: 1 })
|
||||||
|
if (reprobe.length === 0) {
|
||||||
|
throw new GraphIndexNotReadyError(
|
||||||
|
`Graph adjacency index reports ${claimed} relationship(s) but returns no edges even ` +
|
||||||
|
`after a rebuild — the persisted adjacency could not be loaded. find({ connected }), ` +
|
||||||
|
`neighbors() and related() cannot be served reliably for this brain.`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
if (err instanceof GraphIndexNotReadyError) throw err
|
||||||
|
// A transient probe/rebuild failure must not break the actual query NOR be
|
||||||
|
// masked as "no data". Allow a re-check on the next graph read and fall through.
|
||||||
|
this._graphAdjacencyVerified = false
|
||||||
|
if (!this.config.silent) {
|
||||||
|
console.warn(`[Brainy] Graph adjacency consistency check skipped (transient): ${err}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async neighbors(
|
async neighbors(
|
||||||
entityId: string,
|
entityId: string,
|
||||||
options?: {
|
options?: {
|
||||||
|
|
@ -8132,6 +8190,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
}
|
}
|
||||||
): Promise<string[]> {
|
): Promise<string[]> {
|
||||||
await this.ensureInitialized()
|
await this.ensureInitialized()
|
||||||
|
await this.ensureGraphAdjacencyConsistent()
|
||||||
|
|
||||||
const direction = options?.direction || 'both'
|
const direction = options?.direction || 'both'
|
||||||
const limit = options?.limit
|
const limit = options?.limit
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,7 @@ export type BrainyErrorType =
|
||||||
| 'RETRY_EXHAUSTED'
|
| 'RETRY_EXHAUSTED'
|
||||||
| 'VALIDATION'
|
| 'VALIDATION'
|
||||||
| 'FIELD_NOT_INDEXED'
|
| 'FIELD_NOT_INDEXED'
|
||||||
|
| 'GRAPH_INDEX_NOT_READY'
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Custom error class for Brainy operations
|
* Custom error class for Brainy operations
|
||||||
|
|
@ -223,3 +224,27 @@ export class BrainyError extends Error {
|
||||||
return BrainyError.storage(error.message, error)
|
return BrainyError.storage(error.message, error)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Thrown when the graph adjacency index reports that relationships exist (its
|
||||||
|
* persisted manifest/count loaded) but the source→target adjacency itself did
|
||||||
|
* NOT load — so graph traversals (`find({ connected })`, `neighbors()`,
|
||||||
|
* `related()`) would otherwise return an EMPTY array indistinguishable from
|
||||||
|
* "no edges". Brainy detects this on the first graph read (one verb + one
|
||||||
|
* neighbor probe), attempts a rebuild from storage, and raises this LOUD,
|
||||||
|
* catchable error only if even that cannot read a known persisted edge —
|
||||||
|
* replacing silent data-invisibility with a clear failure.
|
||||||
|
*
|
||||||
|
* Observed with a native graph provider whose cold-open adjacency load is
|
||||||
|
* swallowed on certain storage adapters; the fix is upstream in the provider,
|
||||||
|
* but Brainy refuses to serve `[]` as if it were truth.
|
||||||
|
*/
|
||||||
|
export class GraphIndexNotReadyError extends BrainyError {
|
||||||
|
constructor(message: string, originalError?: Error) {
|
||||||
|
super(message, 'GRAPH_INDEX_NOT_READY', false, originalError)
|
||||||
|
this.name = 'GraphIndexNotReadyError'
|
||||||
|
if (Error.captureStackTrace) {
|
||||||
|
Error.captureStackTrace(this, GraphIndexNotReadyError)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
130
tests/unit/brainy/graph-adjacency-cold-load.test.ts
Normal file
130
tests/unit/brainy/graph-adjacency-cold-load.test.ts
Normal file
|
|
@ -0,0 +1,130 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/brainy/graph-adjacency-cold-load
|
||||||
|
* @description 7.33.2 — the graph-adjacency cold-load consistency guard. A native
|
||||||
|
* graph provider can load its relationship COUNT (manifest) on a cold open but fail
|
||||||
|
* to load the source→target adjacency (observed with cor 2.7.5's NativeGraphAdjacencyIndex
|
||||||
|
* on mmap-filesystem: the SSTable-segment load is swallowed). The result is
|
||||||
|
* `find({ connected })` / `neighbors()` / `related()` silently returning `[]` despite
|
||||||
|
* persisted edges. Brainy now probes one known persisted edge on the first graph read;
|
||||||
|
* if the index claims edges but the source resolves to none, it forces a rebuild from
|
||||||
|
* storage, and only if even that fails does it throw a loud, catchable error instead of
|
||||||
|
* serving `[]` as truth. These exercise the guard against a stubbed graph index.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, it, expect } from 'vitest'
|
||||||
|
import { Brainy } from '../../../src/index.js'
|
||||||
|
import { GraphIndexNotReadyError } from '../../../src/errors/brainyError.js'
|
||||||
|
import { createTestConfig } from '../../helpers/test-factory.js'
|
||||||
|
|
||||||
|
/** Replace the brain's graph index with a stub that models a (possibly broken) cold load. */
|
||||||
|
function stubGraphIndex(
|
||||||
|
brain: any,
|
||||||
|
opts: { size: number; neighborsBeforeRebuild: string[]; neighborsAfterRebuild?: string[] }
|
||||||
|
) {
|
||||||
|
let rebuilt = false
|
||||||
|
const calls = { rebuild: 0, getNeighbors: 0 }
|
||||||
|
brain.graphIndex = {
|
||||||
|
size: () => opts.size,
|
||||||
|
getNeighbors: async () => {
|
||||||
|
calls.getNeighbors++
|
||||||
|
return rebuilt ? opts.neighborsAfterRebuild ?? [] : opts.neighborsBeforeRebuild
|
||||||
|
},
|
||||||
|
rebuild: async () => {
|
||||||
|
calls.rebuild++
|
||||||
|
rebuilt = true
|
||||||
|
},
|
||||||
|
flush: async () => {},
|
||||||
|
close: async () => {}
|
||||||
|
}
|
||||||
|
return calls
|
||||||
|
}
|
||||||
|
|
||||||
|
async function freshBrain(): Promise<any> {
|
||||||
|
const brain: any = new Brainy(createTestConfig())
|
||||||
|
await brain.init()
|
||||||
|
brain._graphAdjacencyVerified = false
|
||||||
|
return brain
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('7.33.2 graph-adjacency cold-load consistency guard', () => {
|
||||||
|
it('is a no-op when the adjacency is live (a persisted edge resolves to neighbors)', async () => {
|
||||||
|
const brain = await freshBrain()
|
||||||
|
const calls = stubGraphIndex(brain, { size: 12, neighborsBeforeRebuild: ['n1'] })
|
||||||
|
brain.storage.getVerbs = async () => ({ items: [{ id: 'v1', sourceId: 's1', targetId: 't1' }], hasMore: false })
|
||||||
|
|
||||||
|
await brain.ensureGraphAdjacencyConsistent()
|
||||||
|
expect(calls.rebuild).toBe(0)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('force-rebuilds when the index reports edges but the source probes empty', async () => {
|
||||||
|
const brain = await freshBrain()
|
||||||
|
const calls = stubGraphIndex(brain, { size: 12, neighborsBeforeRebuild: [], neighborsAfterRebuild: ['n1'] })
|
||||||
|
brain.storage.getVerbs = async () => ({ items: [{ id: 'v1', sourceId: 's1', targetId: 't1' }], hasMore: false })
|
||||||
|
|
||||||
|
await brain.ensureGraphAdjacencyConsistent()
|
||||||
|
expect(calls.rebuild).toBe(1) // detected the empty adjacency + rebuilt it
|
||||||
|
|
||||||
|
// Cached: a second call does not re-probe or re-rebuild.
|
||||||
|
await brain.ensureGraphAdjacencyConsistent()
|
||||||
|
expect(calls.rebuild).toBe(1)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('throws GraphIndexNotReadyError when even a rebuild cannot load the adjacency', async () => {
|
||||||
|
const brain = await freshBrain()
|
||||||
|
stubGraphIndex(brain, { size: 12, neighborsBeforeRebuild: [], neighborsAfterRebuild: [] }) // never loads
|
||||||
|
brain.storage.getVerbs = async () => ({ items: [{ id: 'v1', sourceId: 's1', targetId: 't1' }], hasMore: false })
|
||||||
|
|
||||||
|
await expect(brain.ensureGraphAdjacencyConsistent()).rejects.toBeInstanceOf(GraphIndexNotReadyError)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('no-op when no edges are claimed (size 0) — never even probes storage', async () => {
|
||||||
|
const brain = await freshBrain()
|
||||||
|
const calls = stubGraphIndex(brain, { size: 0, neighborsBeforeRebuild: [] })
|
||||||
|
let probedStorage = false
|
||||||
|
brain.storage.getVerbs = async () => {
|
||||||
|
probedStorage = true
|
||||||
|
return { items: [], hasMore: false }
|
||||||
|
}
|
||||||
|
|
||||||
|
await brain.ensureGraphAdjacencyConsistent()
|
||||||
|
expect(calls.rebuild).toBe(0)
|
||||||
|
expect(probedStorage).toBe(false)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('no-op when storage has no edges (stale count) — does not rebuild or throw', async () => {
|
||||||
|
const brain = await freshBrain()
|
||||||
|
const calls = stubGraphIndex(brain, { size: 12, neighborsBeforeRebuild: [] })
|
||||||
|
brain.storage.getVerbs = async () => ({ items: [], hasMore: false })
|
||||||
|
|
||||||
|
await brain.ensureGraphAdjacencyConsistent()
|
||||||
|
expect(calls.rebuild).toBe(0)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('does not loop or break the query on a transient probe failure (re-checks next time)', async () => {
|
||||||
|
const brain = await freshBrain()
|
||||||
|
const calls = stubGraphIndex(brain, { size: 12, neighborsBeforeRebuild: [], neighborsAfterRebuild: ['n1'] })
|
||||||
|
let firstCall = true
|
||||||
|
brain.storage.getVerbs = async () => {
|
||||||
|
if (firstCall) {
|
||||||
|
firstCall = false
|
||||||
|
throw new Error('transient storage hiccup')
|
||||||
|
}
|
||||||
|
return { items: [{ id: 'v1', sourceId: 's1', targetId: 't1' }], hasMore: false }
|
||||||
|
}
|
||||||
|
|
||||||
|
// First call: probe throws transiently → swallowed (no GraphIndexNotReadyError), flag reset.
|
||||||
|
await brain.ensureGraphAdjacencyConsistent()
|
||||||
|
expect(calls.rebuild).toBe(0)
|
||||||
|
expect(brain._graphAdjacencyVerified).toBe(false) // re-check allowed
|
||||||
|
|
||||||
|
// Second call: storage works → detects the empty adjacency + rebuilds.
|
||||||
|
await brain.ensureGraphAdjacencyConsistent()
|
||||||
|
expect(calls.rebuild).toBe(1)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
})
|
||||||
Loading…
Add table
Add a link
Reference in a new issue