feat(8.0): full query surface at historical generations via ephemeral index materialization
Historical Db values (now()/asOf() pins that history has moved past) now serve the COMPLETE query surface - vector/hybrid search, graph traversal, cursor pagination, and aggregation - by materializing ephemeral in-memory indexes over the exact at-generation record set. The historical-query throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted. Materializer (Brainy.materializeAtGeneration): - Copies the at-G record set (live bytes for ids untouched since the pin, immutable before-images otherwise) into a fresh MemoryStorage; a final reconciliation pass under the commit mutex makes the copy exact even when transactions commit mid-build. - Opens a read-only Brainy over the copy: init rebuilds the metadata and graph-adjacency indexes from the records; the vector index is built by inserting every at-G vector (the at-G HNSW graph never existed on disk, so there is nothing to restore). Host embedder and aggregate definitions are shared - no second model load, aggregates backfill at-G values. - Cost is the documented contract: O(n at G) time and memory, ONCE per Db (handle cached; freed by release(), with a FinalizationRegistry backstop that also closes leaked readers). A native VersionedIndexProvider serves the same reads from retained segments with no rebuild. Db routing (src/db/db.ts): metadata-level find()/related() keep the free record path; index-only dimensions (query/vector/near/connected/cursor/ aggregate/includeRelations/non-metadata modes) route to the cached materialization; unsupported where-operators on the record path re-route there too instead of erroring. Speculative with() overlays keep the one honest boundary - SpeculativeOverlayError (overlay entities carry no embeddings, so index reads over them would be silently incomplete); metadata find()/get()/filter related() work on overlays. UpdateParams.vector contract now honored: an explicit pre-computed vector applies directly (with dimension validation) in update() and transact update ops, re-indexing HNSW - previously it was silently ignored unless data also changed. GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees filtered through the live-verb tombstone set (entity->entity edge trees deleted - they carried no verb ids, so removeVerb could never tombstone them and traversal served stale neighbors forever). Neighbor reads batch- load live verbs via the unified cache; addVerb seeds the cache. Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector search finds old vector placement including since-deleted entities; historical graph traversal walks the old wiring after a rewire; historical aggregation computes at-G group values; asOf() pins get the same surface; the materialization builds once per Db and release() closes the ephemeral reader (it refuses reads afterwards); overlays throw the documented error. ADR-001 updated to the no-throws historical model.
This commit is contained in:
parent
8f93add705
commit
e5feae4104
11 changed files with 869 additions and 340 deletions
|
|
@ -2,8 +2,13 @@
|
|||
* @module graph/graphAdjacencyIndex
|
||||
* @description GraphAdjacencyIndex — billion-scale graph traversal engine.
|
||||
*
|
||||
* LSM-tree storage reduces memory from 500GB to 1.3GB for 1 billion
|
||||
* relationships while maintaining sub-5ms neighbor lookups.
|
||||
* Adjacency lives in two verb-id LSM trees (sourceId → verbIds,
|
||||
* targetId → verbIds) filtered through an in-memory live-verb tombstone set,
|
||||
* with full verb objects loaded on demand through the unified cache. The
|
||||
* verb set is the single source of truth: neighbor reads derive from live
|
||||
* verbs, so removals are visible to every read path immediately. ID-only
|
||||
* in-memory tracking keeps the resident footprint to the verb-id set (~8
|
||||
* bytes per relationship) regardless of relationship payload size.
|
||||
*
|
||||
* **8.0 u64 boundary:** this class implements the BigInt
|
||||
* {@link GraphIndexProvider} contract — entity ints in, entity/verb ints out.
|
||||
|
|
@ -58,16 +63,17 @@ export interface GraphIndexStats {
|
|||
/**
|
||||
* GraphAdjacencyIndex - Billion-scale adjacency list with LSM-tree storage
|
||||
*
|
||||
* Core innovation: LSM-tree for disk-based storage with bloom filter optimization
|
||||
* Memory efficient: 385x less memory (1.3GB vs 500GB for 1B relationships)
|
||||
* Performance: Sub-5ms neighbor lookups with bloom filter optimization
|
||||
* Disk-resident verb-id adjacency (LSM trees with bloom filter optimization)
|
||||
* plus an in-memory live-verb tombstone set; neighbor reads derive from live
|
||||
* verbs via cache-assisted batch loads — O(node degree) per lookup, correct
|
||||
* under verb removal.
|
||||
*/
|
||||
export class GraphAdjacencyIndex implements GraphIndexProvider {
|
||||
// LSM-tree storage for outgoing and incoming edges
|
||||
private lsmTreeSource: LSMTree // sourceId -> targetIds (outgoing edges)
|
||||
private lsmTreeTarget: LSMTree // targetId -> sourceIds (incoming edges)
|
||||
|
||||
// LSM-tree storage for verb ID lookups (billion-scale optimization)
|
||||
// LSM-tree storage for verb ID lookups — the single adjacency source of
|
||||
// truth. Neighbor reads derive from these (live-verb filtered via
|
||||
// verbIdSet) so removeVerb tombstones are honored by EVERY read path;
|
||||
// a separate entity→entity edge tree cannot be tombstone-filtered (it
|
||||
// carries no verb ids) and previously served stale neighbors forever.
|
||||
private lsmTreeVerbsBySource: LSMTree // sourceId -> verbIds
|
||||
private lsmTreeVerbsByTarget: LSMTree // targetId -> verbIds
|
||||
|
||||
|
|
@ -126,19 +132,6 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
flushInterval: config.flushInterval ?? 30000
|
||||
}
|
||||
|
||||
// Create LSM-trees for source and target indexes
|
||||
this.lsmTreeSource = new LSMTree(storage, {
|
||||
memTableThreshold: 100000,
|
||||
storagePrefix: 'graph-lsm-source',
|
||||
enableCompaction: true
|
||||
})
|
||||
|
||||
this.lsmTreeTarget = new LSMTree(storage, {
|
||||
memTableThreshold: 100000,
|
||||
storagePrefix: 'graph-lsm-target',
|
||||
enableCompaction: true
|
||||
})
|
||||
|
||||
// Create LSM-trees for verb ID lookups (billion-scale optimization)
|
||||
this.lsmTreeVerbsBySource = new LSMTree(storage, {
|
||||
memTableThreshold: 100000,
|
||||
|
|
@ -155,7 +148,7 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
// Use SAME UnifiedCache as MetadataIndexManager for coordinated memory management
|
||||
this.unifiedCache = getGlobalCache()
|
||||
|
||||
prodLog.info('GraphAdjacencyIndex initialized with LSM-tree storage (4 LSM-trees total)')
|
||||
prodLog.info('GraphAdjacencyIndex initialized with LSM-tree storage (2 LSM-trees total)')
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -210,8 +203,6 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
return
|
||||
}
|
||||
|
||||
await this.lsmTreeSource.init()
|
||||
await this.lsmTreeTarget.init()
|
||||
await this.lsmTreeVerbsBySource.init()
|
||||
await this.lsmTreeVerbsByTarget.init()
|
||||
|
||||
|
|
@ -279,11 +270,13 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
}
|
||||
|
||||
/**
|
||||
* @description Core API — neighbor lookup with LSM-tree storage (BigInt
|
||||
* boundary). O(log n) with bloom filter optimization (90% of queries skip
|
||||
* disk I/O); pagination support for high-degree nodes. The entity int is
|
||||
* resolved to a UUID via the shared mapper (unknown int → empty result) and
|
||||
* neighbor UUIDs convert back via `BigInt(getOrAssign(uuid))`.
|
||||
* @description Core API — neighbor lookup (BigInt boundary). Neighbors
|
||||
* derive from the node's live verbs (tombstone-filtered verb-id LSM trees
|
||||
* + unified-cache batch loads), so `removeVerb()` is honored immediately;
|
||||
* cost is O(node degree) with cache-assisted verb resolution. Pagination
|
||||
* support for high-degree nodes. The entity int is resolved to a UUID via
|
||||
* the shared mapper (unknown int → empty result) and neighbor UUIDs
|
||||
* convert back via `BigInt(getOrAssign(uuid))`.
|
||||
*
|
||||
* @param id - Entity int to get neighbors for (from the shared idMapper).
|
||||
* @param options - Optional direction ('both' default) + limit/offset.
|
||||
|
|
@ -322,6 +315,13 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
* String-keyed neighbor lookup — the internal implementation behind
|
||||
* {@link getNeighbors}. Kept UUID-based because the LSM trees are keyed by
|
||||
* UUID; only the public contract speaks BigInt.
|
||||
*
|
||||
* Neighbors are derived from the node's **live** verbs (the verb-id LSM
|
||||
* trees filtered through the `verbIdSet` tombstones, then batch-loaded via
|
||||
* the unified cache) rather than from a separate entity→entity edge tree.
|
||||
* That keeps `removeVerb()` visible to traversal: an entity-level edge
|
||||
* entry carries no verb id, so it could never be tombstone-filtered, and
|
||||
* a removed relationship would keep its endpoints "connected" forever.
|
||||
*/
|
||||
private async getNeighborUuids(
|
||||
id: string,
|
||||
|
|
@ -335,18 +335,15 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
const direction = options?.direction || 'both'
|
||||
const neighbors = new Set<string>()
|
||||
|
||||
// Query LSM-trees with bloom filter optimization
|
||||
if (direction !== 'in') {
|
||||
const outgoing = await this.lsmTreeSource.get(id)
|
||||
if (outgoing) {
|
||||
outgoing.forEach(neighborId => neighbors.add(neighborId))
|
||||
for (const verb of (await this.liveVerbsForNode(this.lsmTreeVerbsBySource, id)).values()) {
|
||||
neighbors.add(verb.targetId)
|
||||
}
|
||||
}
|
||||
|
||||
if (direction !== 'out') {
|
||||
const incoming = await this.lsmTreeTarget.get(id)
|
||||
if (incoming) {
|
||||
incoming.forEach(neighborId => neighbors.add(neighborId))
|
||||
for (const verb of (await this.liveVerbsForNode(this.lsmTreeVerbsByTarget, id)).values()) {
|
||||
neighbors.add(verb.sourceId)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -370,6 +367,23 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* @description The live (non-tombstoned) verb objects adjacent to `nodeId`
|
||||
* in one verb-id LSM tree: read the node's verb-id list, drop ids deleted
|
||||
* by `removeVerb()` (the `verbIdSet` tombstone filter — same rule as
|
||||
* {@link verbIdsToPaginatedInts}), and batch-load the survivors through the
|
||||
* unified cache. Shared by {@link getNeighborUuids} for both directions.
|
||||
* @param tree - `lsmTreeVerbsBySource` (out-edges) or `lsmTreeVerbsByTarget` (in-edges).
|
||||
* @param nodeId - The node's UUID (LSM trees are UUID-keyed).
|
||||
* @returns The node's live verbs in that direction, keyed by verb id.
|
||||
*/
|
||||
private async liveVerbsForNode(tree: LSMTree, nodeId: string): Promise<Map<string, GraphVerb>> {
|
||||
const verbIds = (await tree.get(nodeId)) || []
|
||||
const liveIds = [...new Set(verbIds)].filter(verbId => this.verbIdSet.has(verbId))
|
||||
if (liveIds.length === 0) return new Map()
|
||||
return this.getVerbsBatchCached(liveIds)
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Verb ints for all edges originating at `sourceInt` (BigInt
|
||||
* boundary). O(log n) LSM-tree lookup with bloom filter optimization;
|
||||
|
|
@ -569,8 +583,8 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
* Get total relationship count - O(1) operation
|
||||
*/
|
||||
size(): number {
|
||||
// Use LSM-tree size for accurate count
|
||||
return this.lsmTreeSource.size()
|
||||
// Use LSM-tree size for accurate count (one entry per indexed verb)
|
||||
return this.lsmTreeVerbsBySource.size()
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -604,13 +618,9 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
uniqueTargetNodes: number
|
||||
totalNodes: number
|
||||
} {
|
||||
const totalRelationships = this.lsmTreeSource.size()
|
||||
const totalRelationships = this.lsmTreeVerbsBySource.size()
|
||||
const relationshipsByType = Object.fromEntries(this.relationshipCountsByType)
|
||||
|
||||
// Get stats from LSM-trees
|
||||
const sourceStats = this.lsmTreeSource.getStats()
|
||||
const targetStats = this.lsmTreeTarget.getStats()
|
||||
|
||||
// Note: Exact unique node counts would require full LSM-tree scan
|
||||
// Using verbIdSet (ID-only tracking) for memory efficiency
|
||||
const uniqueSourceNodes = this.verbIdSet.size
|
||||
|
|
@ -654,11 +664,14 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
this.verbIdSet.add(verb.id)
|
||||
const verbInt = this.internVerbId(verb.id)
|
||||
|
||||
// Add to LSM-trees (outgoing and incoming edges)
|
||||
await this.lsmTreeSource.add(verb.sourceId, verb.targetId)
|
||||
await this.lsmTreeTarget.add(verb.targetId, verb.sourceId)
|
||||
// Seed the unified cache with the authoritative verb object: neighbor
|
||||
// reads ({@link liveVerbsForNode}) resolve verbs through the cache with a
|
||||
// storage fallback, so an indexed verb is immediately traversable — and
|
||||
// freshly written verbs are the likeliest next reads.
|
||||
this.unifiedCache.set(`graph:verb:${verb.id}`, verb, 'other', 128, 50)
|
||||
|
||||
// Add to verbId tracking LSM-trees (billion-scale optimization for getVerbsBySource/Target)
|
||||
// Add to the verb-id adjacency LSM-trees (the single adjacency source of
|
||||
// truth — neighbor and verb-id reads both derive from these).
|
||||
await this.lsmTreeVerbsBySource.add(verb.sourceId, verb.id)
|
||||
await this.lsmTreeVerbsByTarget.add(verb.targetId, verb.id)
|
||||
|
||||
|
|
@ -682,9 +695,13 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
|
||||
/**
|
||||
* @description Remove a relationship from the index by its id string.
|
||||
* LSM-tree edges persist (tombstone deletion via verbIdSet filtering); the
|
||||
* verb's interned int is intentionally retained so previously returned verb
|
||||
* ints stay resolvable via {@link verbIntsToIds}.
|
||||
* Deletion is tombstone-based: the verb id leaves `verbIdSet`, and every
|
||||
* read path (verb-id reads via {@link verbIdsToPaginatedInts}, neighbor
|
||||
* reads via {@link liveVerbsForNode}) filters the append-only LSM trees
|
||||
* through that set — so the verb disappears from traversal immediately
|
||||
* while the trees stay immutable. The verb's interned int is intentionally
|
||||
* retained so previously returned verb ints stay resolvable via
|
||||
* {@link verbIntsToIds}.
|
||||
* @param verbId - The verb's UUID string.
|
||||
* @returns Resolves once the verb no longer appears in reads.
|
||||
*/
|
||||
|
|
@ -709,10 +726,6 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
this.relationshipCountsByType.delete(verbType)
|
||||
}
|
||||
|
||||
// Note: LSM-tree edges persist
|
||||
// Full tombstone deletion can be implemented via compaction
|
||||
// For now, removed verbs won't appear in queries (verbIndex check)
|
||||
|
||||
const elapsed = performance.now() - startTime
|
||||
|
||||
// Performance assertion
|
||||
|
|
@ -794,7 +807,7 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
prodLog.info(`GraphAdjacencyIndex: Rebuild complete in ${rebuildTime}ms`)
|
||||
prodLog.info(` - Total relationships: ${totalVerbs}`)
|
||||
prodLog.info(` - Memory usage: ${(memoryUsage / 1024 / 1024).toFixed(1)}MB`)
|
||||
prodLog.info(` - LSM-tree stats:`, this.lsmTreeSource.getStats())
|
||||
prodLog.info(` - LSM-tree stats:`, this.lsmTreeVerbsBySource.getStats())
|
||||
|
||||
} finally {
|
||||
this.isRebuilding = false
|
||||
|
|
@ -808,8 +821,8 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
let bytes = 0
|
||||
|
||||
// LSM-tree memory (MemTable + bloom filters + zone maps)
|
||||
const sourceStats = this.lsmTreeSource.getStats()
|
||||
const targetStats = this.lsmTreeTarget.getStats()
|
||||
const sourceStats = this.lsmTreeVerbsBySource.getStats()
|
||||
const targetStats = this.lsmTreeVerbsByTarget.getStats()
|
||||
|
||||
bytes += sourceStats.memTableMemory
|
||||
bytes += targetStats.memTableMemory
|
||||
|
|
@ -829,8 +842,8 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
* Get comprehensive statistics
|
||||
*/
|
||||
getStats(): GraphIndexStats {
|
||||
const sourceStats = this.lsmTreeSource.getStats()
|
||||
const targetStats = this.lsmTreeTarget.getStats()
|
||||
const sourceStats = this.lsmTreeVerbsBySource.getStats()
|
||||
const targetStats = this.lsmTreeVerbsByTarget.getStats()
|
||||
|
||||
return {
|
||||
totalRelationships: this.size(),
|
||||
|
|
@ -862,14 +875,8 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
|
||||
const startTime = Date.now()
|
||||
|
||||
// Flush all 4 LSM-trees in parallel (MemTables → SSTables on disk)
|
||||
// Flush both LSM-trees in parallel (MemTables → SSTables on disk)
|
||||
await Promise.all([
|
||||
this.lsmTreeSource.flush().then(() => {
|
||||
prodLog.debug(`GraphAdjacencyIndex: Flushed source tree`)
|
||||
}),
|
||||
this.lsmTreeTarget.flush().then(() => {
|
||||
prodLog.debug(`GraphAdjacencyIndex: Flushed target tree`)
|
||||
}),
|
||||
this.lsmTreeVerbsBySource.flush().then(() => {
|
||||
prodLog.debug(`GraphAdjacencyIndex: Flushed verbs-by-source tree`)
|
||||
}),
|
||||
|
|
@ -892,11 +899,9 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
this.flushTimer = undefined
|
||||
}
|
||||
|
||||
// Close all 4 LSM-trees (will flush MemTables to SSTables)
|
||||
// Close both LSM-trees (will flush MemTables to SSTables)
|
||||
if (this.initialized) {
|
||||
await Promise.all([
|
||||
this.lsmTreeSource.close(),
|
||||
this.lsmTreeTarget.close(),
|
||||
this.lsmTreeVerbsBySource.close(),
|
||||
this.lsmTreeVerbsByTarget.close(),
|
||||
])
|
||||
|
|
@ -915,8 +920,8 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
|
||||
return (
|
||||
!this.isRebuilding &&
|
||||
this.lsmTreeSource.isHealthy() &&
|
||||
this.lsmTreeTarget.isHealthy()
|
||||
this.lsmTreeVerbsBySource.isHealthy() &&
|
||||
this.lsmTreeVerbsByTarget.isHealthy()
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue