feat(8.0): full query surface at historical generations via ephemeral index materialization
Historical Db values (now()/asOf() pins that history has moved past) now serve the COMPLETE query surface - vector/hybrid search, graph traversal, cursor pagination, and aggregation - by materializing ephemeral in-memory indexes over the exact at-generation record set. The historical-query throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted. Materializer (Brainy.materializeAtGeneration): - Copies the at-G record set (live bytes for ids untouched since the pin, immutable before-images otherwise) into a fresh MemoryStorage; a final reconciliation pass under the commit mutex makes the copy exact even when transactions commit mid-build. - Opens a read-only Brainy over the copy: init rebuilds the metadata and graph-adjacency indexes from the records; the vector index is built by inserting every at-G vector (the at-G HNSW graph never existed on disk, so there is nothing to restore). Host embedder and aggregate definitions are shared - no second model load, aggregates backfill at-G values. - Cost is the documented contract: O(n at G) time and memory, ONCE per Db (handle cached; freed by release(), with a FinalizationRegistry backstop that also closes leaked readers). A native VersionedIndexProvider serves the same reads from retained segments with no rebuild. Db routing (src/db/db.ts): metadata-level find()/related() keep the free record path; index-only dimensions (query/vector/near/connected/cursor/ aggregate/includeRelations/non-metadata modes) route to the cached materialization; unsupported where-operators on the record path re-route there too instead of erroring. Speculative with() overlays keep the one honest boundary - SpeculativeOverlayError (overlay entities carry no embeddings, so index reads over them would be silently incomplete); metadata find()/get()/filter related() work on overlays. UpdateParams.vector contract now honored: an explicit pre-computed vector applies directly (with dimension validation) in update() and transact update ops, re-indexing HNSW - previously it was silently ignored unless data also changed. GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees filtered through the live-verb tombstone set (entity->entity edge trees deleted - they carried no verb ids, so removeVerb could never tombstone them and traversal served stale neighbors forever). Neighbor reads batch- load live verbs via the unified cache; addVerb seeds the cache. Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector search finds old vector placement including since-deleted entities; historical graph traversal walks the old wiring after a rewire; historical aggregation computes at-G group values; asOf() pins get the same surface; the materialization builds once per Db and release() closes the ephemeral reader (it refuses reads afterwards); overlays throw the documented error. ADR-001 updated to the no-throws historical model.
This commit is contained in:
parent
8f93add705
commit
e5feae4104
11 changed files with 869 additions and 340 deletions
|
|
@ -26,10 +26,16 @@
|
|||
* 9. Versioned provider wiring — a provider implementing the 4-method
|
||||
* `VersionedIndexProvider` contract receives balanced pin/release (and a
|
||||
* visibility consult per pin) as Db values are created and released.
|
||||
* 10. Historical full query surface — vector search, graph traversal, and
|
||||
* aggregation at a past pinned generation return at-generation results
|
||||
* via ephemeral index materialization (for `now()` and `asOf()` pins
|
||||
* alike, built once per Db and cached); `release()` frees the
|
||||
* materialization by closing the ephemeral reader; speculative overlays
|
||||
* keep the documented `SpeculativeOverlayError` boundary.
|
||||
*
|
||||
* All entities carry explicit vectors so the suite never invokes the
|
||||
* embedding engine (semantic search is exercised only for its documented
|
||||
* historical-generation error).
|
||||
* embedding engine (vector queries use find({ vector }); semantic string
|
||||
* search on overlays is exercised only for its documented error).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, afterEach, vi } from 'vitest'
|
||||
|
|
@ -37,11 +43,11 @@ import * as fs from 'node:fs'
|
|||
import * as os from 'node:os'
|
||||
import * as path from 'node:path'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { Db } from '../../src/db/db.js'
|
||||
import { Db, type HistoricalQueryHandle } from '../../src/db/db.js'
|
||||
import {
|
||||
GenerationCompactedError,
|
||||
GenerationConflictError,
|
||||
NotYetSupportedAtHistoricalGenerationError
|
||||
SpeculativeOverlayError
|
||||
} from '../../src/db/errors.js'
|
||||
import type { GenerationStore } from '../../src/db/generationStore.js'
|
||||
import { GraphAdjacencyIndex } from '../../src/graph/graphAdjacencyIndex.js'
|
||||
|
|
@ -472,9 +478,7 @@ describe('8.0 Db API — generational MVCC', () => {
|
|||
])
|
||||
|
||||
const snapDir = path.join(makeTempDir(), 'stale-snapshot')
|
||||
await expect(db.persist(snapDir)).rejects.toThrow(
|
||||
NotYetSupportedAtHistoricalGenerationError
|
||||
)
|
||||
await expect(db.persist(snapDir)).rejects.toThrow(GenerationConflictError)
|
||||
await db.release()
|
||||
})
|
||||
|
||||
|
|
@ -870,32 +874,193 @@ describe('8.0 Db API — generational MVCC', () => {
|
|||
await after.release()
|
||||
})
|
||||
|
||||
it('index-accelerated queries at a historical generation throw the documented error — never silently-wrong results', async () => {
|
||||
it('historical vector search is served at the pinned generation via index materialization', async () => {
|
||||
const brain = await openMemoryBrain()
|
||||
|
||||
// Two one-hot vectors: orthogonal under cosine, so nearest-neighbor
|
||||
// results are fully deterministic.
|
||||
const axisA = Array.from({ length: 384 }, (_, i) => (i === 0 ? 1 : 0))
|
||||
const axisB = Array.from({ length: 384 }, (_, i) => (i === 1 ? 1 : 0))
|
||||
const axisC = Array.from({ length: 384 }, (_, i) => (i === 2 ? 1 : 0))
|
||||
|
||||
await brain.transact([
|
||||
{ op: 'add', id: uid('hist-e'), type: NounType.Document, data: 'x', vector: vec(95), metadata: { v: 1 } }
|
||||
{ op: 'add', id: uid('mat-a'), type: NounType.Document, data: 'a', vector: axisA, metadata: { who: 'a' } },
|
||||
{ op: 'add', id: uid('mat-b'), type: NounType.Document, data: 'b', vector: axisB, metadata: { who: 'b' } },
|
||||
{ op: 'add', id: uid('mat-del'), type: NounType.Document, data: 'del', vector: axisC, metadata: { who: 'del' } }
|
||||
])
|
||||
const db = brain.now()
|
||||
await brain.transact([{ op: 'update', id: uid('hist-e'), metadata: { v: 2 } }])
|
||||
|
||||
// Storage-record reads remain fully supported…
|
||||
expect(((await db.get(uid('hist-e')))?.metadata as { v: number }).v).toBe(1)
|
||||
expect((await db.find({ type: NounType.Document, where: { v: 1 } })).length).toBe(1)
|
||||
// Mutate distinctively after the pin: swap a/b's vectors, delete 'del'.
|
||||
await brain.transact([
|
||||
{ op: 'update', id: uid('mat-a'), vector: axisB },
|
||||
{ op: 'update', id: uid('mat-b'), vector: axisA },
|
||||
{ op: 'remove', id: uid('mat-del') }
|
||||
])
|
||||
|
||||
// …index-accelerated dimensions throw the documented, named error.
|
||||
await expect(db.search('anything')).rejects.toThrow(
|
||||
NotYetSupportedAtHistoricalGenerationError
|
||||
)
|
||||
await expect(db.find({ vector: vec(95) })).rejects.toThrow(
|
||||
NotYetSupportedAtHistoricalGenerationError
|
||||
)
|
||||
await expect(db.find({ connected: { from: uid('hist-e') } })).rejects.toThrow(
|
||||
NotYetSupportedAtHistoricalGenerationError
|
||||
)
|
||||
// Live index reflects the swap…
|
||||
const liveNearA = await brain.find({ vector: axisA, limit: 1 })
|
||||
expect(liveNearA[0].id).toBe(uid('mat-b'))
|
||||
|
||||
// Released views refuse all reads.
|
||||
// …the pinned view finds the OLD vector placement, including the
|
||||
// since-deleted entity, via the at-generation materialization.
|
||||
const pinnedNearA = await db.find({ vector: axisA, limit: 1 })
|
||||
expect(pinnedNearA[0].id).toBe(uid('mat-a'))
|
||||
const pinnedNearC = await db.find({ vector: axisC, limit: 1 })
|
||||
expect(pinnedNearC[0].id).toBe(uid('mat-del'))
|
||||
|
||||
// Record-path reads on the same Db stay consistent with the materialization.
|
||||
expect(((await db.get(uid('mat-del')))?.metadata as { who: string }).who).toBe('del')
|
||||
|
||||
// Released views refuse all reads (and free the materialization).
|
||||
await db.release()
|
||||
await expect(db.find({ vector: axisA, limit: 1 })).rejects.toThrow('released Db')
|
||||
})
|
||||
|
||||
it('historical graph traversal is served at the pinned generation via index materialization', async () => {
|
||||
const brain = await openMemoryBrain()
|
||||
await brain.transact([
|
||||
{ op: 'add', id: uid('trav-a'), type: NounType.Person, data: 'a', vector: vec(110), metadata: {} },
|
||||
{ op: 'add', id: uid('trav-b'), type: NounType.Document, data: 'b', vector: vec(111), metadata: {} },
|
||||
{ op: 'add', id: uid('trav-c'), type: NounType.Document, data: 'c', vector: vec(112), metadata: {} },
|
||||
{ op: 'relate', from: uid('trav-a'), to: uid('trav-b'), type: VerbType.References }
|
||||
])
|
||||
const db = brain.now()
|
||||
|
||||
// Rewire the graph after the pin: a→b becomes a→c.
|
||||
const oldEdge = (await brain.getRelations({ from: uid('trav-a') }))[0]
|
||||
await brain.transact([
|
||||
{ op: 'unrelate', id: oldEdge.id },
|
||||
{ op: 'relate', from: uid('trav-a'), to: uid('trav-c'), type: VerbType.References }
|
||||
])
|
||||
|
||||
// Live traversal sees the new wiring…
|
||||
const liveConnected = await brain.find({ connected: { from: uid('trav-a') }, limit: 10 })
|
||||
expect(liveConnected.map((row) => row.id)).toContain(uid('trav-c'))
|
||||
expect(liveConnected.map((row) => row.id)).not.toContain(uid('trav-b'))
|
||||
|
||||
// …the pinned view traverses the OLD graph.
|
||||
const pinnedConnected = await db.find({ connected: { from: uid('trav-a') }, limit: 10 })
|
||||
expect(pinnedConnected.map((row) => row.id)).toContain(uid('trav-b'))
|
||||
expect(pinnedConnected.map((row) => row.id)).not.toContain(uid('trav-c'))
|
||||
|
||||
// The record-path related() agrees with the materialized traversal.
|
||||
const pinnedEdges = await db.related({ from: uid('trav-a') })
|
||||
expect(pinnedEdges.length).toBe(1)
|
||||
expect(pinnedEdges[0].to).toBe(uid('trav-b'))
|
||||
|
||||
await db.release()
|
||||
})
|
||||
|
||||
it('historical aggregation computes at-generation values via index materialization', async () => {
|
||||
const brain = await openMemoryBrain()
|
||||
brain.defineAggregate({
|
||||
name: 'order_totals',
|
||||
source: { type: NounType.Document },
|
||||
groupBy: ['category'],
|
||||
metrics: {
|
||||
total: { op: 'sum', field: 'amount' },
|
||||
orders: { op: 'count' }
|
||||
}
|
||||
})
|
||||
|
||||
await brain.transact([
|
||||
{ op: 'add', id: uid('agg-1'), type: NounType.Document, data: 'o1', vector: vec(120), metadata: { category: 'invoice', amount: 10 } },
|
||||
{ op: 'add', id: uid('agg-2'), type: NounType.Document, data: 'o2', vector: vec(121), metadata: { category: 'invoice', amount: 20 } }
|
||||
])
|
||||
const db = brain.now()
|
||||
|
||||
// Mutate after the pin: add an order and inflate an existing one.
|
||||
await brain.transact([
|
||||
{ op: 'add', id: uid('agg-3'), type: NounType.Document, data: 'o3', vector: vec(122), metadata: { category: 'invoice', amount: 70 } },
|
||||
{ op: 'update', id: uid('agg-1'), metadata: { amount: 100 } }
|
||||
])
|
||||
|
||||
// Live aggregate reflects the mutations…
|
||||
const liveRows = await brain.find({ aggregate: 'order_totals' })
|
||||
const liveInvoice = liveRows.find((row) => (row.metadata as { category: string }).category === 'invoice')
|
||||
expect((liveInvoice?.metadata as { total: number }).total).toBe(190)
|
||||
expect((liveInvoice?.metadata as { orders: number }).orders).toBe(3)
|
||||
|
||||
// …the pinned view computes the aggregate over the at-G record set.
|
||||
const pinnedRows = await db.find({ aggregate: 'order_totals' })
|
||||
const pinnedInvoice = pinnedRows.find((row) => (row.metadata as { category: string }).category === 'invoice')
|
||||
expect((pinnedInvoice?.metadata as { total: number }).total).toBe(30)
|
||||
expect((pinnedInvoice?.metadata as { orders: number }).orders).toBe(2)
|
||||
|
||||
await db.release()
|
||||
})
|
||||
|
||||
it('asOf() pins serve the materialized surface; the build is cached per Db; release() closes the ephemeral reader', async () => {
|
||||
const brain = await openMemoryBrain()
|
||||
|
||||
// Two one-hot vectors — orthogonal, fully deterministic neighbors.
|
||||
const axisA = Array.from({ length: 384 }, (_, i) => (i === 0 ? 1 : 0))
|
||||
const axisB = Array.from({ length: 384 }, (_, i) => (i === 1 ? 1 : 0))
|
||||
|
||||
const tx = await brain.transact([
|
||||
{ op: 'add', id: uid('asof-mat-a'), type: NounType.Document, data: 'a', vector: axisA, metadata: {} },
|
||||
{ op: 'add', id: uid('asof-mat-b'), type: NounType.Document, data: 'b', vector: axisB, metadata: {} }
|
||||
])
|
||||
const pinnedGeneration = tx.generation
|
||||
await tx.release()
|
||||
|
||||
// Swap the vectors after the pin point, then open the past via asOf().
|
||||
await brain.transact([
|
||||
{ op: 'update', id: uid('asof-mat-a'), vector: axisB },
|
||||
{ op: 'update', id: uid('asof-mat-b'), vector: axisA }
|
||||
])
|
||||
const db = await brain.asOf(pinnedGeneration)
|
||||
|
||||
// The asOf() view serves index-accelerated queries at its pinned
|
||||
// generation — same materialized surface as a now() pin.
|
||||
expect((await brain.find({ vector: axisA, limit: 1 }))[0].id).toBe(uid('asof-mat-b'))
|
||||
expect((await db.find({ vector: axisA, limit: 1 }))[0].id).toBe(uid('asof-mat-a'))
|
||||
|
||||
// The O(n at G) build runs ONCE per Db: the second index query reuses
|
||||
// the cached handle (typed access to the private cache, mirroring
|
||||
// generationStoreOf above).
|
||||
const internals = db as unknown as { materializedHandle?: Promise<HistoricalQueryHandle> }
|
||||
expect(internals.materializedHandle).toBeDefined()
|
||||
const handle = await internals.materializedHandle!
|
||||
expect((await db.find({ vector: axisB, limit: 1 }))[0].id).toBe(uid('asof-mat-b'))
|
||||
expect(await internals.materializedHandle).toBe(handle)
|
||||
|
||||
// release() frees the materialization by CLOSING the ephemeral reader —
|
||||
// the reader itself refuses reads afterwards, proving its indexes and
|
||||
// timers were torn down (not merely hidden behind the released-Db guard).
|
||||
await db.release()
|
||||
await db.release() // idempotent
|
||||
await expect(db.find({ vector: axisA, limit: 1 })).rejects.toThrow('released Db')
|
||||
await expect(handle.find({ vector: axisA, limit: 1 })).rejects.toThrow('not initialized')
|
||||
})
|
||||
|
||||
it('speculative overlays keep the one honest boundary: index queries and persist throw SpeculativeOverlayError', async () => {
|
||||
const brain = await openMemoryBrain()
|
||||
await brain.transact([
|
||||
{ op: 'add', id: uid('spec-e'), type: NounType.Document, data: 'x', vector: vec(95), metadata: { v: 1 } }
|
||||
])
|
||||
const db = brain.now()
|
||||
const spec = await db.with([{ op: 'update', id: uid('spec-e'), metadata: { v: 2 } }])
|
||||
|
||||
// Metadata reads on the overlay are fully supported…
|
||||
expect(((await spec.get(uid('spec-e')))?.metadata as { v: number }).v).toBe(2)
|
||||
expect((await spec.find({ type: NounType.Document, where: { v: 2 } })).length).toBe(1)
|
||||
|
||||
// …index-accelerated dimensions and persist throw the named error.
|
||||
await expect(spec.search('anything')).rejects.toThrow(SpeculativeOverlayError)
|
||||
await expect(spec.find({ vector: vec(95) })).rejects.toThrow(SpeculativeOverlayError)
|
||||
await expect(spec.find({ connected: { from: uid('spec-e') } })).rejects.toThrow(
|
||||
SpeculativeOverlayError
|
||||
)
|
||||
await expect(spec.persist(path.join(makeTempDir(), 'overlay-snap'))).rejects.toThrow(
|
||||
SpeculativeOverlayError
|
||||
)
|
||||
|
||||
// The non-speculative base view is unaffected by the overlay boundary.
|
||||
expect(((await db.get(uid('spec-e')))?.metadata as { v: number }).v).toBe(1)
|
||||
|
||||
await spec.release()
|
||||
await db.release()
|
||||
await expect(db.get(uid('hist-e'))).rejects.toThrow('released Db')
|
||||
})
|
||||
|
||||
it('restore() requires confirmation and replaces state from a snapshot (generation floor preserved)', async () => {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue