feat(8.0): full query surface at historical generations via ephemeral index materialization
Historical Db values (now()/asOf() pins that history has moved past) now serve the COMPLETE query surface - vector/hybrid search, graph traversal, cursor pagination, and aggregation - by materializing ephemeral in-memory indexes over the exact at-generation record set. The historical-query throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted. Materializer (Brainy.materializeAtGeneration): - Copies the at-G record set (live bytes for ids untouched since the pin, immutable before-images otherwise) into a fresh MemoryStorage; a final reconciliation pass under the commit mutex makes the copy exact even when transactions commit mid-build. - Opens a read-only Brainy over the copy: init rebuilds the metadata and graph-adjacency indexes from the records; the vector index is built by inserting every at-G vector (the at-G HNSW graph never existed on disk, so there is nothing to restore). Host embedder and aggregate definitions are shared - no second model load, aggregates backfill at-G values. - Cost is the documented contract: O(n at G) time and memory, ONCE per Db (handle cached; freed by release(), with a FinalizationRegistry backstop that also closes leaked readers). A native VersionedIndexProvider serves the same reads from retained segments with no rebuild. Db routing (src/db/db.ts): metadata-level find()/related() keep the free record path; index-only dimensions (query/vector/near/connected/cursor/ aggregate/includeRelations/non-metadata modes) route to the cached materialization; unsupported where-operators on the record path re-route there too instead of erroring. Speculative with() overlays keep the one honest boundary - SpeculativeOverlayError (overlay entities carry no embeddings, so index reads over them would be silently incomplete); metadata find()/get()/filter related() work on overlays. UpdateParams.vector contract now honored: an explicit pre-computed vector applies directly (with dimension validation) in update() and transact update ops, re-indexing HNSW - previously it was silently ignored unless data also changed. GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees filtered through the live-verb tombstone set (entity->entity edge trees deleted - they carried no verb ids, so removeVerb could never tombstone them and traversal served stale neighbors forever). Neighbor reads batch- load live verbs via the unified cache; addVerb seeds the cache. Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector search finds old vector placement including since-deleted entities; historical graph traversal walks the old wiring after a rewire; historical aggregation computes at-G group values; asOf() pins get the same surface; the materialization builds once per Db and release() closes the ephemeral reader (it refuses reads afterwards); overlays throw the documented error. ADR-001 updated to the no-throws historical model.
This commit is contained in:
parent
8f93add705
commit
e5feae4104
11 changed files with 869 additions and 340 deletions
|
|
@ -123,17 +123,31 @@ While nothing has committed past G, *every* read on the `Db` delegates to
|
|||
the live fast paths untouched — `now()` adds no read overhead until history
|
||||
actually moves.
|
||||
|
||||
**Honesty boundary.** `get()`, metadata-level `find()`, and `related()` are
|
||||
fully correct at any reachable pinned generation. Index-accelerated
|
||||
dimensions (semantic/vector search, graph traversal, cursors, aggregation)
|
||||
are served by live indexes that represent only the current generation, so
|
||||
at a historical generation they throw
|
||||
`NotYetSupportedAtHistoricalGenerationError` — never silently-wrong
|
||||
results. The escape hatch is `persist()` + `Brainy.load()`, which rebuilds
|
||||
indexes from the snapshot and supports the full query surface. Rebuilding
|
||||
indexes from a pinned record set in-place (the standard LSM answer) is the
|
||||
documented follow-up; versioned native providers already expose
|
||||
`isGenerationVisible()` for provider-accelerated historical reads.
|
||||
**Two read paths, one result set.** `get()`, metadata-level `find()`, and
|
||||
filter-based `related()` resolve directly through the record layer at any
|
||||
reachable pinned generation — no extra cost beyond scanning the deltas of
|
||||
later commits. Index-accelerated dimensions (semantic/vector search, graph
|
||||
traversal, cursors, aggregation) are served by **at-generation index
|
||||
materialization**: the first such query on a historical `Db` copies the
|
||||
exact at-G record set (live bytes for ids untouched since the pin,
|
||||
before-images for the rest; a final reconciliation pass runs under the
|
||||
commit mutex so transactions racing the copy cannot skew it) into an
|
||||
ephemeral in-memory store and opens a read-only engine over it — the same
|
||||
vector/metadata/graph index classes the live brain uses, sharing the host's
|
||||
embedder and aggregate definitions. The handle is cached on the `Db` and
|
||||
freed by `release()`.
|
||||
|
||||
**Cost, stated plainly:** materialization is O(n at G) time and memory,
|
||||
once per `Db`. That is the open-core price of historical index queries. A
|
||||
native `VersionedIndexProvider` (`isGenerationVisible()` + pins over
|
||||
retained LSM segments) serves the same reads with no rebuild at all — the
|
||||
materializer is the correctness baseline, the provider is the accelerator.
|
||||
|
||||
**The one remaining boundary.** Speculative `with()` overlays throw
|
||||
`SpeculativeOverlayError` for index-accelerated queries and `persist()`:
|
||||
overlay entities carry no embeddings (`with()` never invokes the embedder),
|
||||
so a "full" index query over an overlay would silently exclude the
|
||||
overlay's own entities. Commit with `transact()` to get the full surface.
|
||||
|
||||
**History granularity.** Generation *records* are written per `transact()`
|
||||
batch only. Single-operation writes advance the counter (so watermarks and
|
||||
|
|
@ -230,7 +244,8 @@ only counter values that nothing durable ever referenced.
|
|||
| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
|
||||
| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
|
||||
| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
|
||||
| Index-accelerated query at a historical pin | `NotYetSupportedAtHistoricalGenerationError` — explicit, never silently-wrong results. |
|
||||
| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). |
|
||||
| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. |
|
||||
| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |
|
||||
|
||||
## Lineage
|
||||
|
|
@ -257,6 +272,8 @@ boring where it counts:
|
|||
- Every commit pays O(ids touched) extra writes (before-images + delta +
|
||||
manifest). Single-operation writes pay only an in-memory counter bump
|
||||
with coalesced persistence.
|
||||
- Historical index-accelerated queries are an explicit error until the
|
||||
asOf-rebuild ships; correctness-critical historical reads (`get`,
|
||||
metadata `find`, `related`) work today and forever at any reachable pin.
|
||||
- The full query surface works at every reachable pinned generation.
|
||||
Record-path reads (`get`, metadata `find`, filter `related`) are
|
||||
effectively free; index-accelerated historical queries pay a one-time
|
||||
O(n at G) materialization per `Db` on the open-core path (freed on
|
||||
`release()`), and run rebuild-free on a native `VersionedIndexProvider`.
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
|
|||
|-------|---------|----------------|------------|---------------|------------------|
|
||||
| **TypeAwareVectorIndex** | Type-aware vector similarity search | 42 type-specific hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 |
|
||||
| **MetadataIndexManager** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps + roaring bitmaps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | ✅ Line 2318 |
|
||||
| **GraphAdjacencyIndex** | Relationship traversal | 4 LSM-trees + bidirectional adjacency maps | O(1) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
|
||||
| **GraphAdjacencyIndex** | Relationship traversal | 2 verb-id LSM-trees + tombstone-filtered adjacency derivation | O(degree) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 |
|
||||
|
||||
### Sub-Indexes (Level 2)
|
||||
|
||||
|
|
@ -847,7 +847,7 @@ Where:
|
|||
|-------|-------------------|-------|
|
||||
| **MetadataIndexManager** | ~100 bytes | Depends on field count and cardinality (RoaringBitmap32 compression) |
|
||||
| **TypeAwareVectorIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes |
|
||||
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional references + metadata in 4 LSM-trees |
|
||||
| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional verb-id references in 2 LSM-trees |
|
||||
|
||||
**Total overhead**: ~1.6 KB per entity + ~50 bytes per relationship
|
||||
|
||||
|
|
@ -932,7 +932,7 @@ All have rebuild() methods and are covered by lazy loading:
|
|||
Automatically managed by parent rebuild():
|
||||
- **42 type-specific vector indexes** (one per NounType)
|
||||
- **6 metadata components** (ChunkManager, EntityIdMapper, FieldTypeInference, Field Sparse Indexes, Sorted Indexes)
|
||||
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
|
||||
- **2 LSM-trees** (lsmTreeVerbsBySource, lsmTreeVerbsByTarget — the verb set is the single adjacency source of truth; neighbor reads derive from live verbs so removals are honored)
|
||||
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
|
||||
|
||||
### Lazy Loading
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue