feat(8.0): full query surface at historical generations via ephemeral index materialization

Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.

Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
  immutable before-images otherwise) into a fresh MemoryStorage; a final
  reconciliation pass under the commit mutex makes the copy exact even
  when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
  graph-adjacency indexes from the records; the vector index is built by
  inserting every at-G vector (the at-G HNSW graph never existed on disk,
  so there is nothing to restore). Host embedder and aggregate definitions
  are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
  (handle cached; freed by release(), with a FinalizationRegistry backstop
  that also closes leaked readers). A native VersionedIndexProvider serves
  the same reads from retained segments with no rebuild.

Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.

UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.

GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.

Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
This commit is contained in:
David Snelling 2026-06-11 08:12:11 -07:00
parent 8f93add705
commit e5feae4104
11 changed files with 869 additions and 340 deletions

View file

@ -12,7 +12,7 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
|-------|---------|----------------|------------|---------------|------------------|
| **TypeAwareVectorIndex** | Type-aware vector similarity search | 42 type-specific hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
| **MetadataIndexManager** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps + roaring bitmaps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | ✅ Line 2318 |
| **GraphAdjacencyIndex** | Relationship traversal | 4 LSM-trees + bidirectional adjacency maps | O(1) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
| **GraphAdjacencyIndex** | Relationship traversal | 2 verb-id LSM-trees + tombstone-filtered adjacency derivation | O(degree) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
### Sub-Indexes (Level 2)
@ -847,7 +847,7 @@ Where:
|-------|-------------------|-------|
| **MetadataIndexManager** | ~100 bytes | Depends on field count and cardinality (RoaringBitmap32 compression) |
| **TypeAwareVectorIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional references + metadata in 4 LSM-trees |
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional verb-id references in 2 LSM-trees |
**Total overhead**: ~1.6 KB per entity + ~50 bytes per relationship
@ -932,7 +932,7 @@ All have rebuild() methods and are covered by lazy loading:
Automatically managed by parent rebuild():
- **42 type-specific vector indexes** (one per NounType)
- **6 metadata components** (ChunkManager, EntityIdMapper, FieldTypeInference, Field Sparse Indexes, Sorted Indexes)
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
- **2 LSM-trees** (lsmTreeVerbsBySource, lsmTreeVerbsByTarget — the verb set is the single adjacency source of truth; neighbor reads derive from live verbs so removals are honored)
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
### Lazy Loading