soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat(8.0): full query surface at historical generations via ephemeral index materialization

Browse source 
Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.

Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
  immutable before-images otherwise) into a fresh MemoryStorage; a final
  reconciliation pass under the commit mutex makes the copy exact even
  when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
  graph-adjacency indexes from the records; the vector index is built by
  inserting every at-G vector (the at-G HNSW graph never existed on disk,
  so there is nothing to restore). Host embedder and aggregate definitions
  are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
  (handle cached; freed by release(), with a FinalizationRegistry backstop
  that also closes leaked readers). A native VersionedIndexProvider serves
  the same reads from retained segments with no rebuild.

Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.

UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.

GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.

Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
This commit is contained in:
David Snelling 2026-06-11 08:12:11 -07:00
parent 8f93add705
commit e5feae4104
11 changed files with 869 additions and 340 deletions
Download patch file Download diff file Expand all files Collapse all files

215
tests/integration/db-mvcc.test.ts
View file

@ -26,10 +26,16 @@
* 9. Versioned provider wiring — a provider implementing the 4-method
* `VersionedIndexProvider` contract receives balanced pin/release (and a
* visibility consult per pin) as Db values are created and released.
* 10. Historical full query surface — vector search, graph traversal, and
* aggregation at a past pinned generation return at-generation results
* via ephemeral index materialization (for `now()` and `asOf()` pins
* alike, built once per Db and cached); `release()` frees the
* materialization by closing the ephemeral reader; speculative overlays
* keep the documented `SpeculativeOverlayError` boundary.
*
* All entities carry explicit vectors so the suite never invokes the
* embedding engine (semantic search is exercised only for its documented
* historical-generation error).
* embedding engine (vector queries use find({ vector }); semantic string
* search on overlays is exercised only for its documented error).
*/
import { describe, it, expect, afterEach, vi } from 'vitest'
@ -37,11 +43,11 @@ import * as fs from 'node:fs'
import * as os from 'node:os'
import * as path from 'node:path'
import { Brainy } from '../../src/brainy.js'
import { Db } from '../../src/db/db.js'
import { Db, type HistoricalQueryHandle } from '../../src/db/db.js'
import {
GenerationCompactedError,
GenerationConflictError,
NotYetSupportedAtHistoricalGenerationError
SpeculativeOverlayError
} from '../../src/db/errors.js'
import type { GenerationStore } from '../../src/db/generationStore.js'
import { GraphAdjacencyIndex } from '../../src/graph/graphAdjacencyIndex.js'
@ -472,9 +478,7 @@ describe('8.0 Db API — generational MVCC', () => {
])
const snapDir = path.join(makeTempDir(), 'stale-snapshot')
await expect(db.persist(snapDir)).rejects.toThrow(
NotYetSupportedAtHistoricalGenerationError
)
await expect(db.persist(snapDir)).rejects.toThrow(GenerationConflictError)
await db.release()
})
@ -870,32 +874,193 @@ describe('8.0 Db API — generational MVCC', () => {
await after.release()
})
it('index-accelerated queries at a historical generation throw the documented error — never silently-wrong results', async () => {
it('historical vector search is served at the pinned generation via index materialization', async () => {
const brain = await openMemoryBrain()
// Two one-hot vectors: orthogonal under cosine, so nearest-neighbor
// results are fully deterministic.
const axisA = Array.from({ length: 384 }, (_, i) => (i === 0 ? 1 : 0))
const axisB = Array.from({ length: 384 }, (_, i) => (i === 1 ? 1 : 0))
const axisC = Array.from({ length: 384 }, (_, i) => (i === 2 ? 1 : 0))
await brain.transact([
{ op: 'add', id: uid('hist-e'), type: NounType.Document, data: 'x', vector: vec(95), metadata: { v: 1 } }
{ op: 'add', id: uid('mat-a'), type: NounType.Document, data: 'a', vector: axisA, metadata: { who: 'a' } },
{ op: 'add', id: uid('mat-b'), type: NounType.Document, data: 'b', vector: axisB, metadata: { who: 'b' } },
{ op: 'add', id: uid('mat-del'), type: NounType.Document, data: 'del', vector: axisC, metadata: { who: 'del' } }
])
const db = brain.now()
await brain.transact([{ op: 'update', id: uid('hist-e'), metadata: { v: 2 } }])
// Storage-record reads remain fully supported…
expect(((await db.get(uid('hist-e')))?.metadata as { v: number }).v).toBe(1)
expect((await db.find({ type: NounType.Document, where: { v: 1 } })).length).toBe(1)
// Mutate distinctively after the pin: swap a/b's vectors, delete 'del'.
await brain.transact([
{ op: 'update', id: uid('mat-a'), vector: axisB },
{ op: 'update', id: uid('mat-b'), vector: axisA },
{ op: 'remove', id: uid('mat-del') }
])
// …index-accelerated dimensions throw the documented, named error.
await expect(db.search('anything')).rejects.toThrow(
NotYetSupportedAtHistoricalGenerationError
)
await expect(db.find({ vector: vec(95) })).rejects.toThrow(
NotYetSupportedAtHistoricalGenerationError
)
await expect(db.find({ connected: { from: uid('hist-e') } })).rejects.toThrow(
NotYetSupportedAtHistoricalGenerationError
)
// Live index reflects the swap…
const liveNearA = await brain.find({ vector: axisA, limit: 1 })
expect(liveNearA[0].id).toBe(uid('mat-b'))
// Released views refuse all reads.
// …the pinned view finds the OLD vector placement, including the
// since-deleted entity, via the at-generation materialization.
const pinnedNearA = await db.find({ vector: axisA, limit: 1 })
expect(pinnedNearA[0].id).toBe(uid('mat-a'))
const pinnedNearC = await db.find({ vector: axisC, limit: 1 })
expect(pinnedNearC[0].id).toBe(uid('mat-del'))
// Record-path reads on the same Db stay consistent with the materialization.
expect(((await db.get(uid('mat-del')))?.metadata as { who: string }).who).toBe('del')
// Released views refuse all reads (and free the materialization).
await db.release()
await expect(db.find({ vector: axisA, limit: 1 })).rejects.toThrow('released Db')
})
it('historical graph traversal is served at the pinned generation via index materialization', async () => {
const brain = await openMemoryBrain()
await brain.transact([
{ op: 'add', id: uid('trav-a'), type: NounType.Person, data: 'a', vector: vec(110), metadata: {} },
{ op: 'add', id: uid('trav-b'), type: NounType.Document, data: 'b', vector: vec(111), metadata: {} },
{ op: 'add', id: uid('trav-c'), type: NounType.Document, data: 'c', vector: vec(112), metadata: {} },
{ op: 'relate', from: uid('trav-a'), to: uid('trav-b'), type: VerbType.References }
])
const db = brain.now()
// Rewire the graph after the pin: a→b becomes a→c.
const oldEdge = (await brain.getRelations({ from: uid('trav-a') }))[0]
await brain.transact([
{ op: 'unrelate', id: oldEdge.id },
{ op: 'relate', from: uid('trav-a'), to: uid('trav-c'), type: VerbType.References }
])
// Live traversal sees the new wiring…
const liveConnected = await brain.find({ connected: { from: uid('trav-a') }, limit: 10 })
expect(liveConnected.map((row) => row.id)).toContain(uid('trav-c'))
expect(liveConnected.map((row) => row.id)).not.toContain(uid('trav-b'))
// …the pinned view traverses the OLD graph.
const pinnedConnected = await db.find({ connected: { from: uid('trav-a') }, limit: 10 })
expect(pinnedConnected.map((row) => row.id)).toContain(uid('trav-b'))
expect(pinnedConnected.map((row) => row.id)).not.toContain(uid('trav-c'))
// The record-path related() agrees with the materialized traversal.
const pinnedEdges = await db.related({ from: uid('trav-a') })
expect(pinnedEdges.length).toBe(1)
expect(pinnedEdges[0].to).toBe(uid('trav-b'))
await db.release()
})
it('historical aggregation computes at-generation values via index materialization', async () => {
const brain = await openMemoryBrain()
brain.defineAggregate({
name: 'order_totals',
source: { type: NounType.Document },
groupBy: ['category'],
metrics: {
total: { op: 'sum', field: 'amount' },
orders: { op: 'count' }
}
})
await brain.transact([
{ op: 'add', id: uid('agg-1'), type: NounType.Document, data: 'o1', vector: vec(120), metadata: { category: 'invoice', amount: 10 } },
{ op: 'add', id: uid('agg-2'), type: NounType.Document, data: 'o2', vector: vec(121), metadata: { category: 'invoice', amount: 20 } }
])
const db = brain.now()
// Mutate after the pin: add an order and inflate an existing one.
await brain.transact([
{ op: 'add', id: uid('agg-3'), type: NounType.Document, data: 'o3', vector: vec(122), metadata: { category: 'invoice', amount: 70 } },
{ op: 'update', id: uid('agg-1'), metadata: { amount: 100 } }
])
// Live aggregate reflects the mutations…
const liveRows = await brain.find({ aggregate: 'order_totals' })
const liveInvoice = liveRows.find((row) => (row.metadata as { category: string }).category === 'invoice')
expect((liveInvoice?.metadata as { total: number }).total).toBe(190)
expect((liveInvoice?.metadata as { orders: number }).orders).toBe(3)
// …the pinned view computes the aggregate over the at-G record set.
const pinnedRows = await db.find({ aggregate: 'order_totals' })
const pinnedInvoice = pinnedRows.find((row) => (row.metadata as { category: string }).category === 'invoice')
expect((pinnedInvoice?.metadata as { total: number }).total).toBe(30)
expect((pinnedInvoice?.metadata as { orders: number }).orders).toBe(2)
await db.release()
})
it('asOf() pins serve the materialized surface; the build is cached per Db; release() closes the ephemeral reader', async () => {
const brain = await openMemoryBrain()
// Two one-hot vectors — orthogonal, fully deterministic neighbors.
const axisA = Array.from({ length: 384 }, (_, i) => (i === 0 ? 1 : 0))
const axisB = Array.from({ length: 384 }, (_, i) => (i === 1 ? 1 : 0))
const tx = await brain.transact([
{ op: 'add', id: uid('asof-mat-a'), type: NounType.Document, data: 'a', vector: axisA, metadata: {} },
{ op: 'add', id: uid('asof-mat-b'), type: NounType.Document, data: 'b', vector: axisB, metadata: {} }
])
const pinnedGeneration = tx.generation
await tx.release()
// Swap the vectors after the pin point, then open the past via asOf().
await brain.transact([
{ op: 'update', id: uid('asof-mat-a'), vector: axisB },
{ op: 'update', id: uid('asof-mat-b'), vector: axisA }
])
const db = await brain.asOf(pinnedGeneration)
// The asOf() view serves index-accelerated queries at its pinned
// generation — same materialized surface as a now() pin.
expect((await brain.find({ vector: axisA, limit: 1 }))[0].id).toBe(uid('asof-mat-b'))
expect((await db.find({ vector: axisA, limit: 1 }))[0].id).toBe(uid('asof-mat-a'))
// The O(n at G) build runs ONCE per Db: the second index query reuses
// the cached handle (typed access to the private cache, mirroring
// generationStoreOf above).
const internals = db as unknown as { materializedHandle?: Promise<HistoricalQueryHandle> }
expect(internals.materializedHandle).toBeDefined()
const handle = await internals.materializedHandle!
expect((await db.find({ vector: axisB, limit: 1 }))[0].id).toBe(uid('asof-mat-b'))
expect(await internals.materializedHandle).toBe(handle)
// release() frees the materialization by CLOSING the ephemeral reader —
// the reader itself refuses reads afterwards, proving its indexes and
// timers were torn down (not merely hidden behind the released-Db guard).
await db.release()
await db.release() // idempotent
await expect(db.find({ vector: axisA, limit: 1 })).rejects.toThrow('released Db')
await expect(handle.find({ vector: axisA, limit: 1 })).rejects.toThrow('not initialized')
})
it('speculative overlays keep the one honest boundary: index queries and persist throw SpeculativeOverlayError', async () => {
const brain = await openMemoryBrain()
await brain.transact([
{ op: 'add', id: uid('spec-e'), type: NounType.Document, data: 'x', vector: vec(95), metadata: { v: 1 } }
])
const db = brain.now()
const spec = await db.with([{ op: 'update', id: uid('spec-e'), metadata: { v: 2 } }])
// Metadata reads on the overlay are fully supported…
expect(((await spec.get(uid('spec-e')))?.metadata as { v: number }).v).toBe(2)
expect((await spec.find({ type: NounType.Document, where: { v: 2 } })).length).toBe(1)
// …index-accelerated dimensions and persist throw the named error.
await expect(spec.search('anything')).rejects.toThrow(SpeculativeOverlayError)
await expect(spec.find({ vector: vec(95) })).rejects.toThrow(SpeculativeOverlayError)
await expect(spec.find({ connected: { from: uid('spec-e') } })).rejects.toThrow(
SpeculativeOverlayError
)
await expect(spec.persist(path.join(makeTempDir(), 'overlay-snap'))).rejects.toThrow(
SpeculativeOverlayError
)
// The non-speculative base view is unaffected by the overlay boundary.
expect(((await db.get(uid('spec-e')))?.metadata as { v: number }).v).toBe(1)
await spec.release()
await db.release()
await expect(db.get(uid('hist-e'))).rejects.toThrow('released Db')
})
it('restore() requires confirmation and replaces state from a snapshot (generation floor preserved)', async () => {

25
tests/unit/brainy/update.test.ts
View file

@ -253,31 +253,44 @@ describe('Brainy.update()', () => {
})).rejects.toThrow('invalid NounType')
})
it('should not update vector directly via update method', async () => {
it('applies an explicit pre-computed vector (UpdateParams.vector contract)', async () => {
// Arrange
const id = await brain.add(createAddParams({
data: 'Test',
type: 'thing'
}))
// v5.11.1: Need includeVectors to get actual vector
const original = await brain.get(id, { includeVectors: true })
const originalVector = original!.vector
// Create a properly dimensioned but different vector
const differentVector = originalVector.map(v => v * 2)
// Act - Update with vector param (not supported)
// Act — update with an explicit vector and no new data. The
// UpdateParams contract ("New pre-computed vector") applies it
// directly, with no re-embedding (mirrored by transact update ops).
await brain.update({
id,
vector: differentVector
})
// Assert - Vector should not change (update ignores vector param)
// v5.11.1: Need includeVectors to check vector
// Assert — the stored vector is the supplied one.
const updated = await brain.get(id, { includeVectors: true })
expect(updated).not.toBeNull()
expect(updated!.vector).toEqual(originalVector)
expect(updated!.vector).toEqual(differentVector)
})
it('rejects an explicit vector with mismatched dimensions', async () => {
const id = await brain.add(createAddParams({
data: 'Test',
type: 'thing'
}))
// Param validation rejects wrong dimensionality before the update runs
// (update() also re-checks against the store's actual dimensionality).
await expect(
brain.update({ id, vector: [0.1, 0.2, 0.3] })
).rejects.toThrow(/dimensions?/)
})
it('should reject empty update parameters', async () => {