soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions 1

feat(8.0): full query surface at historical generations via ephemeral index materialization

Browse source 
Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.

Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
  immutable before-images otherwise) into a fresh MemoryStorage; a final
  reconciliation pass under the commit mutex makes the copy exact even
  when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
  graph-adjacency indexes from the records; the vector index is built by
  inserting every at-G vector (the at-G HNSW graph never existed on disk,
  so there is nothing to restore). Host embedder and aggregate definitions
  are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
  (handle cached; freed by release(), with a FinalizationRegistry backstop
  that also closes leaked readers). A native VersionedIndexProvider serves
  the same reads from retained segments with no rebuild.

Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.

UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.

GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.

Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
This commit is contained in:
David Snelling 2026-06-11 08:12:11 -07:00
parent 8f93add705
commit e5feae4104
11 changed files with 869 additions and 340 deletions
Download patch file Download diff file Expand all files Collapse all files

123
src/db/errors.ts
View file

@ -5,18 +5,22 @@
* Three failure modes have dedicated classes so callers can branch on them
* with `instanceof` instead of string matching:
*
* - {@link GenerationConflictError} — a `transact()` compare-and-swap
* (`ifAtGeneration`) observed a different current generation than the
* caller expected. The standard retry pattern is: re-read via
* `brain.now()`, re-derive the transaction, and re-submit.
* - {@link NotYetSupportedAtHistoricalGenerationError} — an index-accelerated
* query (vector / hybrid / graph-traversal search) was issued against a
* `Db` pinned at a generation the live indexes no longer represent.
* `get()` and metadata-filter `find()` remain fully supported at pinned
* generations; index-accelerated queries at historical generations require
* an index rebuild from the pinned record set, which ships as a follow-up
* (`asOf` rebuild). The error message names the unsupported capability and
* points at the supported alternatives.
* - {@link GenerationConflictError} — the store's current generation differs
* from what the caller's operation requires: a `transact()`
* compare-and-swap (`ifAtGeneration`) observed a different generation than
* expected, or `db.persist()` was called on a view that history has moved
* past (snapshots capture current bytes, so persisting requires the view
* to still be the latest generation). The standard retry pattern is:
* re-read via `brain.now()`, re-derive the operation, and re-submit.
* - {@link SpeculativeOverlayError} — an index-accelerated query (vector /
* hybrid / graph-traversal search, cursors, aggregation) or `persist()`
* was issued against a speculative `db.with()` overlay. Overlays are pure
* in-memory values whose entities carry no embeddings (`with()` never
* invokes the embedder), so a "full" index query over one would silently
* miss the overlay's own entities — an honest error beats silently-wrong
* results. Commit the operations with `brain.transact()` to get the full
* query surface. Historical (non-overlay) views do NOT throw this: they
* serve the full query surface via at-generation index materialization.
* - {@link GenerationCompactedError} — `asOf()` asked for a generation whose
* immutable records were reclaimed by `compactHistory()`.
*
@ -24,14 +28,19 @@
*/
/**
* @description Thrown by `brain.transact(ops, { ifAtGeneration })` when the
* store's current generation does not equal the caller-supplied expectation.
* This is the whole-store compare-and-swap counterpart to the per-entity
* `ifRev` / `RevisionConflictError` pair: it guarantees that *nothing* was
* committed between the caller's read (`brain.now()`) and this transaction.
* @description Thrown when an operation requires the store to be at a
* specific generation and it is not:
*
* The transaction is rejected before any record is staged — the store is
* untouched and the generation counter is unchanged.
* - `brain.transact(ops, { ifAtGeneration })` — the whole-store
* compare-and-swap counterpart to the per-entity `ifRev` /
* `RevisionConflictError` pair: it guarantees that *nothing* was committed
* between the caller's read (`brain.now()`) and this transaction. The
* transaction is rejected before any record is staged — the store is
* untouched and the generation counter is unchanged.
* - `db.persist(path)` — snapshots capture current bytes, so persisting
* requires the view to still be the store's **latest** generation. A view
* that history has moved past throws this error: pin with `brain.now()`
* and persist before further writes.
*
* @example
* const db = brain.now()
@ -45,20 +54,21 @@
* }
*/
export class GenerationConflictError extends Error {
/** The generation the caller expected the store to be at. */
/** The generation the operation required the store to be at. */
public readonly expected: number
/** The generation the store was actually at when the transaction arrived. */
/** The generation the store was actually at. */
public readonly actual: number
/**
* @param expected - The generation supplied via `ifAtGeneration`.
* @param actual - The store's current generation at submission time.
* @param expected - The required generation (`ifAtGeneration`, or the
* pinned generation of the view being persisted).
* @param actual - The store's current generation.
*/
constructor(expected: number, actual: number) {
super(
`Generation conflict: transact() was submitted with ifAtGeneration: ${expected}, ` +
`but the store is at generation ${actual}. Another write committed in between. ` +
`Re-read with brain.now(), rebuild the transaction, and retry.`
`Generation conflict: the operation requires the store at generation ${expected}, ` +
`but it is at generation ${actual}. Another write committed in between. ` +
`Re-read with brain.now(), rebuild the operation, and retry.`
)
this.name = 'GenerationConflictError'
this.expected = expected
@ -67,45 +77,50 @@ export class GenerationConflictError extends Error {
}
/**
* @description Thrown when an operation on a pinned `Db` requires
* index-accelerated machinery (vector search, hybrid search, graph
* traversal, cursor pagination, aggregation) that the live indexes cannot
* answer for a *historical* generation. Storage-record reads (`get()`) and
* metadata-filter `find()` are always supported at pinned generations.
* @description Thrown when an operation on a speculative `db.with()` overlay
* requires index machinery (vector / hybrid / graph-traversal search, cursor
* pagination, aggregation) or durability (`persist()`).
*
* Rebuilding indexes from a pinned record set (the standard LSM answer for
* historical index queries) is the documented `asOf`-rebuild follow-up; this
* error is the honest boundary until it ships.
* Why this single boundary exists: overlays are pure in-memory values —
* `with()` never touches disk, the generation counter, or the embedder, so
* overlay entities carry **no embeddings**. A "full" vector or hybrid query
* over an overlay would silently exclude the overlay's own entities, and a
* persisted overlay would be a store whose entities cannot be semantically
* searched. An honest error beats silently-wrong results. Commit the
* operations with `brain.transact()` to get the full query surface, or query
* the overlay's base view (historical views serve the complete surface via
* at-generation index materialization).
*
* `get()`, metadata-filter `find()`, and filter-based `related()` remain
* fully supported on overlays.
*
* @example
* const db = brain.now()
* await brain.transact([{ op: 'update', id, metadata: { v: 2 } }])
* await db.get(id) // ✅ pinned read — supported
* await db.find({ where: { v: 1 } }) // ✅ metadata find — supported
* await db.search('semantic query') // ❌ throws this error
* const spec = await brain.now().with([{ op: 'add', type, data: 'draft' }])
* await spec.find({ where: { draft: true } }) // ✅ metadata find — supported
* await spec.search('semantic query') // ❌ throws this error
* await brain.transact([{ op: 'add', type, data: 'draft' }]) // → full surface
*/
export class NotYetSupportedAtHistoricalGenerationError extends Error {
/** The pinned generation the operation was attempted against. */
public readonly generation: number
/** The capability that is not supported at historical generations. */
export class SpeculativeOverlayError extends Error {
/** The capability that is not supported on speculative overlays. */
public readonly capability: string
/** The overlay's pinned base generation. */
public readonly generation: number
/**
* @param capability - Human-readable name of the unsupported capability
* (e.g. `'vector search'`, `'graph traversal (connected)'`).
* @param generation - The Db's pinned generation.
* @param currentGeneration - The store's current generation, for context.
* (e.g. `'vector search'`, `'persist'`).
* @param generation - The overlay's pinned base generation.
*/
constructor(capability: string, generation: number, currentGeneration: number) {
constructor(capability: string, generation: number) {
super(
`${capability} is not yet supported at a historical generation ` +
`(Db pinned at generation ${generation}; store is at ${currentGeneration}). ` +
`Supported at pinned generations: get() and metadata-filter find(). ` +
`For index-accelerated queries over historical state, persist() the generation ` +
`you need and open it with Brainy.load(path) — the loaded snapshot rebuilds ` +
`its indexes and supports the full query surface.`
`${capability} is not supported on a speculative with() overlay ` +
`(pinned at base generation ${generation}). Overlays are pure in-memory ` +
`values whose entities carry no embeddings, so index-accelerated reads ` +
`over them would be silently incomplete. Supported on overlays: get(), ` +
`metadata-filter find(), and filter-based related(). Commit the operations ` +
`with brain.transact() for the full query surface, or query the base view.`
)
this.name = 'NotYetSupportedAtHistoricalGenerationError'
this.name = 'SpeculativeOverlayError'
this.capability = capability
this.generation = generation
}