feat(8.0): full query surface at historical generations via ephemeral index materialization

Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.

Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
  immutable before-images otherwise) into a fresh MemoryStorage; a final
  reconciliation pass under the commit mutex makes the copy exact even
  when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
  graph-adjacency indexes from the records; the vector index is built by
  inserting every at-G vector (the at-G HNSW graph never existed on disk,
  so there is nothing to restore). Host embedder and aggregate definitions
  are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
  (handle cached; freed by release(), with a FinalizationRegistry backstop
  that also closes leaked readers). A native VersionedIndexProvider serves
  the same reads from retained segments with no rebuild.

Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.

UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.

GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.

Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
This commit is contained in:
David Snelling 2026-06-11 08:12:11 -07:00
parent 8f93add705
commit e5feae4104
11 changed files with 869 additions and 340 deletions

View file

@ -123,17 +123,31 @@ While nothing has committed past G, *every* read on the `Db` delegates to
the live fast paths untouched — `now()` adds no read overhead until history
actually moves.
**Honesty boundary.** `get()`, metadata-level `find()`, and `related()` are
fully correct at any reachable pinned generation. Index-accelerated
dimensions (semantic/vector search, graph traversal, cursors, aggregation)
are served by live indexes that represent only the current generation, so
at a historical generation they throw
`NotYetSupportedAtHistoricalGenerationError` — never silently-wrong
results. The escape hatch is `persist()` + `Brainy.load()`, which rebuilds
indexes from the snapshot and supports the full query surface. Rebuilding
indexes from a pinned record set in-place (the standard LSM answer) is the
documented follow-up; versioned native providers already expose
`isGenerationVisible()` for provider-accelerated historical reads.
**Two read paths, one result set.** `get()`, metadata-level `find()`, and
filter-based `related()` resolve directly through the record layer at any
reachable pinned generation — no extra cost beyond scanning the deltas of
later commits. Index-accelerated dimensions (semantic/vector search, graph
traversal, cursors, aggregation) are served by **at-generation index
materialization**: the first such query on a historical `Db` copies the
exact at-G record set (live bytes for ids untouched since the pin,
before-images for the rest; a final reconciliation pass runs under the
commit mutex so transactions racing the copy cannot skew it) into an
ephemeral in-memory store and opens a read-only engine over it — the same
vector/metadata/graph index classes the live brain uses, sharing the host's
embedder and aggregate definitions. The handle is cached on the `Db` and
freed by `release()`.
**Cost, stated plainly:** materialization is O(n at G) time and memory,
once per `Db`. That is the open-core price of historical index queries. A
native `VersionedIndexProvider` (`isGenerationVisible()` + pins over
retained LSM segments) serves the same reads with no rebuild at all — the
materializer is the correctness baseline, the provider is the accelerator.
**The one remaining boundary.** Speculative `with()` overlays throw
`SpeculativeOverlayError` for index-accelerated queries and `persist()`:
overlay entities carry no embeddings (`with()` never invokes the embedder),
so a "full" index query over an overlay would silently exclude the
overlay's own entities. Commit with `transact()` to get the full surface.
**History granularity.** Generation *records* are written per `transact()`
batch only. Single-operation writes advance the counter (so watermarks and
@ -230,7 +244,8 @@ only counter values that nothing durable ever referenced.
| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
| Index-accelerated query at a historical pin | `NotYetSupportedAtHistoricalGenerationError` — explicit, never silently-wrong results. |
| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). |
| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. |
| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |
## Lineage
@ -257,6 +272,8 @@ boring where it counts:
- Every commit pays O(ids touched) extra writes (before-images + delta +
manifest). Single-operation writes pay only an in-memory counter bump
with coalesced persistence.
- Historical index-accelerated queries are an explicit error until the
asOf-rebuild ships; correctness-critical historical reads (`get`,
metadata `find`, `related`) work today and forever at any reachable pin.
- The full query surface works at every reachable pinned generation.
Record-path reads (`get`, metadata `find`, filter `related`) are
effectively free; index-accelerated historical queries pay a one-time
O(n at G) materialization per `Db` on the open-core path (freed on
`release()`), and run rebuild-free on a native `VersionedIndexProvider`.