feat(8.0): full query surface at historical generations via ephemeral index materialization

Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.

Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
  immutable before-images otherwise) into a fresh MemoryStorage; a final
  reconciliation pass under the commit mutex makes the copy exact even
  when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
  graph-adjacency indexes from the records; the vector index is built by
  inserting every at-G vector (the at-G HNSW graph never existed on disk,
  so there is nothing to restore). Host embedder and aggregate definitions
  are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
  (handle cached; freed by release(), with a FinalizationRegistry backstop
  that also closes leaked readers). A native VersionedIndexProvider serves
  the same reads from retained segments with no rebuild.

Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.

UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.

GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.

Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
This commit is contained in:
David Snelling 2026-06-11 08:12:11 -07:00
parent 8f93add705
commit e5feae4104
11 changed files with 869 additions and 340 deletions

View file

@ -123,17 +123,31 @@ While nothing has committed past G, *every* read on the `Db` delegates to
the live fast paths untouched — `now()` adds no read overhead until history
actually moves.
**Honesty boundary.** `get()`, metadata-level `find()`, and `related()` are
fully correct at any reachable pinned generation. Index-accelerated
dimensions (semantic/vector search, graph traversal, cursors, aggregation)
are served by live indexes that represent only the current generation, so
at a historical generation they throw
`NotYetSupportedAtHistoricalGenerationError` — never silently-wrong
results. The escape hatch is `persist()` + `Brainy.load()`, which rebuilds
indexes from the snapshot and supports the full query surface. Rebuilding
indexes from a pinned record set in-place (the standard LSM answer) is the
documented follow-up; versioned native providers already expose
`isGenerationVisible()` for provider-accelerated historical reads.
**Two read paths, one result set.** `get()`, metadata-level `find()`, and
filter-based `related()` resolve directly through the record layer at any
reachable pinned generation — no extra cost beyond scanning the deltas of
later commits. Index-accelerated dimensions (semantic/vector search, graph
traversal, cursors, aggregation) are served by **at-generation index
materialization**: the first such query on a historical `Db` copies the
exact at-G record set (live bytes for ids untouched since the pin,
before-images for the rest; a final reconciliation pass runs under the
commit mutex so transactions racing the copy cannot skew it) into an
ephemeral in-memory store and opens a read-only engine over it — the same
vector/metadata/graph index classes the live brain uses, sharing the host's
embedder and aggregate definitions. The handle is cached on the `Db` and
freed by `release()`.
**Cost, stated plainly:** materialization is O(n at G) time and memory,
once per `Db`. That is the open-core price of historical index queries. A
native `VersionedIndexProvider` (`isGenerationVisible()` + pins over
retained LSM segments) serves the same reads with no rebuild at all — the
materializer is the correctness baseline, the provider is the accelerator.
**The one remaining boundary.** Speculative `with()` overlays throw
`SpeculativeOverlayError` for index-accelerated queries and `persist()`:
overlay entities carry no embeddings (`with()` never invokes the embedder),
so a "full" index query over an overlay would silently exclude the
overlay's own entities. Commit with `transact()` to get the full surface.
**History granularity.** Generation *records* are written per `transact()`
batch only. Single-operation writes advance the counter (so watermarks and
@ -230,7 +244,8 @@ only counter values that nothing durable ever referenced.
| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
| Index-accelerated query at a historical pin | `NotYetSupportedAtHistoricalGenerationError` — explicit, never silently-wrong results. |
| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). |
| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. |
| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |
## Lineage
@ -257,6 +272,8 @@ boring where it counts:
- Every commit pays O(ids touched) extra writes (before-images + delta +
manifest). Single-operation writes pay only an in-memory counter bump
with coalesced persistence.
- Historical index-accelerated queries are an explicit error until the
asOf-rebuild ships; correctness-critical historical reads (`get`,
metadata `find`, `related`) work today and forever at any reachable pin.
- The full query surface works at every reachable pinned generation.
Record-path reads (`get`, metadata `find`, filter `related`) are
effectively free; index-accelerated historical queries pay a one-time
O(n at G) materialization per `Db` on the open-core path (freed on
`release()`), and run rebuild-free on a native `VersionedIndexProvider`.

View file

@ -12,7 +12,7 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
|-------|---------|----------------|------------|---------------|------------------|
| **TypeAwareVectorIndex** | Type-aware vector similarity search | 42 type-specific hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
| **MetadataIndexManager** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps + roaring bitmaps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | ✅ Line 2318 |
| **GraphAdjacencyIndex** | Relationship traversal | 4 LSM-trees + bidirectional adjacency maps | O(1) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
| **GraphAdjacencyIndex** | Relationship traversal | 2 verb-id LSM-trees + tombstone-filtered adjacency derivation | O(degree) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
### Sub-Indexes (Level 2)
@ -847,7 +847,7 @@ Where:
|-------|-------------------|-------|
| **MetadataIndexManager** | ~100 bytes | Depends on field count and cardinality (RoaringBitmap32 compression) |
| **TypeAwareVectorIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional references + metadata in 4 LSM-trees |
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional verb-id references in 2 LSM-trees |
**Total overhead**: ~1.6 KB per entity + ~50 bytes per relationship
@ -932,7 +932,7 @@ All have rebuild() methods and are covered by lazy loading:
Automatically managed by parent rebuild():
- **42 type-specific vector indexes** (one per NounType)
- **6 metadata components** (ChunkManager, EntityIdMapper, FieldTypeInference, Field Sparse Indexes, Sorted Indexes)
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
- **2 LSM-trees** (lsmTreeVerbsBySource, lsmTreeVerbsByTarget — the verb set is the single adjacency source of truth; neighbor reads derive from live verbs so removals are honored)
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
### Lazy Loading