soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat(8.0): u64 BigInt graph provider contract — punch list a-d,g,h

Browse source 
GraphIndexProvider now speaks BigInt at the boundary (D.2 mirror):
- getNeighbors/getVerbIdsBySource/getVerbIdsByTarget take entity ints and
  return entity/verb ints as bigint[]
- new REQUIRED verbIntsToIds(bigint[]) batch reverse resolver (L.7
  identity-fingerprint design — verb ids are UUIDs by contract, so the
  provider-side interning is losslessly reversible)
- addVerb(verb, sourceInt, targetInt) returns the interned verb int;
  removeVerb(verbId) joins the contract

Coordinator (brainy.ts) owns ALL UUID <-> int conversion: getOrAssign on
writes, getInt on reads (unmapped UUID -> empty result without calling the
provider), getUuid / verbIntsToIds on returns, plus a bounded ~100k-entry
insertion-order warm cache for verb-int -> verb-id pairs fed by addVerb
returns and resolver results. GraphVerb gains derived sourceInt/targetInt
(populated at add time, never persisted). findConnectedSubtype gains a
native fast path that routes single-type single-subtype outgoing BFS
through the provider when available.

JS GraphAdjacencyIndex satisfies the contract while staying string/u32-keyed
internally: entity ints resolve through the shared entity-id mapper (threaded
in by the coordinator on init/fork/checkout), verb ints come from an
in-process append-only interning map re-derived from storage on
rebuild/cold-start. ColumnStoreProvider widens the same way: addEntity/
removeEntity take bigint, sortTopK/filteredSortTopK return bigint[].

relate() now rejects a caller-supplied id with a teaching error — verb ids
are brainy-generated UUIDs by contract in 8.0 (previously a passed id was
silently ignored). No Roaring64 provider-boundary decode site exists yet;
the JS-internal column store stays Roaring32 and the Treemap decoder lands
with the first consumer of provider-returned filter buffers.

Public brain API unchanged. 1413 tests green (+10 new BigInt contract tests).
This commit is contained in:
David Snelling 2026-06-10 10:45:45 -07:00
parent 62f6472fa0
commit 2427bb7960
17 changed files with 1164 additions and 328 deletions
Download patch file Download diff file Expand all files Collapse all files

308
src/brainy.ts
View file

@ -164,6 +164,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
private distance: DistanceFunction
private config: Required<BrainyConfig>
/**
* Bounded warm cache for verb-int → verb-id resolution (8.0 u64 contract).
* Fed by `addVerb` returns and `verbIntsToIds` results; evicted in insertion
* order once {@link Brainy.VERB_INT_WARM_CACHE_MAX} is exceeded. Pure
* optimization — the graph-index provider owns the durable interning, so a
* miss just means one extra `verbIntsToIds` round trip.
*/
private readonly verbIntWarmCache = new Map<bigint, string>()
/** Cap for {@link Brainy.verbIntWarmCache} (~100k entries ≈ a few MB). */
private static readonly VERB_INT_WARM_CACHE_MAX = 100_000
// Distributed components (optional)
private coordinator?: DistributedCoordinator
private shardManager?: ShardManager
@ -602,6 +613,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
this.graphIndex = graphIndex
}
// 8.0 u64 contract: thread the shared UUID ↔ int resolver into the JS
// graph index and the storage layer's verb read paths.
this.wireGraphIdResolver()
// Wire the connections codec (2.4.0 #3). When the graph:compression
// provider is registered AND the metadata index exposes a stable
// idMapper, inject a codec that encodes HNSW connections as
@ -1741,9 +1756,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Operations 4+: Delete all related verbs atomically
for (const verb of allVerbs) {
// Remove from graph index
// Remove from graph index (endpoint ints resolved up front so a
// rollback can re-add through the BigInt addVerb contract)
const { sourceInt, targetInt } = this.resolveVerbEndpointInts(verb)
tx.addOperation(
new RemoveFromGraphIndexOperation(this.graphIndex, verb as any)
new RemoveFromGraphIndexOperation(this.graphIndex, verb, sourceInt, targetInt)
)
// Delete verb metadata
tx.addOperation(
@ -1768,6 +1785,126 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// ============= RELATIONSHIP OPERATIONS =============
// --- 8.0 u64 boundary helpers -------------------------------------------
// UUID ↔ int conversion happens ONCE here at the coordinator boundary:
// `getOrAssign` on writes, `getInt` on reads (undefined → the entity was
// never mapped, i.e. it has no relations — return empty without calling the
// provider). Provider returns convert back via `getUuid` (entities) and
// `verbIntsToIds` + the warm cache (verbs). `Number(bigint)` narrowing is
// lossless under the shipped EntityIdSpaceExceeded u32 guard.
/**
* Resolve a UUID to its entity int for a READ — `undefined` means the
* entity was never mapped and therefore has no relations.
*/
private graphEntityInt(uuid: string): bigint | undefined {
const intId = this.metadataIndex.getIdMapper().getInt(uuid)
return intId === undefined ? undefined : BigInt(intId)
}
/**
* Resolve a verb's endpoint UUIDs to entity ints for a WRITE
* (`getOrAssign`), mirroring them onto `verb.sourceInt`/`verb.targetInt`
* (derived state — never persisted to storage JSON) before the verb is
* handed to the graph-index provider. Accepts any verb shape carrying
* endpoint UUIDs (`GraphVerb`, `HNSWVerbWithMetadata`, …).
*/
private resolveVerbEndpointInts(
verb: Pick<GraphVerb, 'sourceId' | 'targetId'> & { sourceInt?: bigint; targetInt?: bigint }
): { sourceInt: bigint; targetInt: bigint } {
const idMapper = this.metadataIndex.getIdMapper()
const sourceInt = BigInt(idMapper.getOrAssign(verb.sourceId))
const targetInt = BigInt(idMapper.getOrAssign(verb.targetId))
verb.sourceInt = sourceInt
verb.targetInt = targetInt
return { sourceInt, targetInt }
}
/**
* Convert provider-returned entity ints back to UUIDs, dropping ints the
* mapper no longer knows (deleted entities).
*/
private entityIntsToUuids(entityInts: bigint[]): string[] {
const idMapper = this.metadataIndex.getIdMapper()
const uuids: string[] = []
for (const entityInt of entityInts) {
const uuid = idMapper.getUuid(Number(entityInt))
if (uuid !== undefined) uuids.push(uuid)
}
return uuids
}
/**
* Record one verb-int → verb-id pair in the bounded warm cache, evicting
* the oldest entries (insertion order) past the cap.
*/
private cacheVerbInt(verbInt: bigint, verbId: string): void {
if (!this.verbIntWarmCache.has(verbInt) &&
this.verbIntWarmCache.size >= Brainy.VERB_INT_WARM_CACHE_MAX) {
// Evict oldest insertions until under cap (single eviction in the
// common case; loop guards against future cap reductions).
for (const oldest of this.verbIntWarmCache.keys()) {
this.verbIntWarmCache.delete(oldest)
if (this.verbIntWarmCache.size < Brainy.VERB_INT_WARM_CACHE_MAX) break
}
}
this.verbIntWarmCache.set(verbInt, verbId)
}
/**
* Resolve provider-returned verb ints to verb-id strings: warm cache first,
* then one batched `verbIntsToIds` call for the misses (which also refills
* the cache). Unknown ints are dropped.
*/
private async resolveVerbIntsToIds(verbInts: bigint[]): Promise<string[]> {
if (verbInts.length === 0) return []
const resolved = new Array<string | undefined>(verbInts.length)
const missIndices: number[] = []
const missInts: bigint[] = []
for (let i = 0; i < verbInts.length; i++) {
const cached = this.verbIntWarmCache.get(verbInts[i])
if (cached !== undefined) {
resolved[i] = cached
} else {
missIndices.push(i)
missInts.push(verbInts[i])
}
}
if (missInts.length > 0) {
const ids = await this.graphIndex.verbIntsToIds(missInts)
for (let j = 0; j < missInts.length; j++) {
const id = ids[j]
if (id !== null) {
resolved[missIndices[j]] = id
this.cacheVerbInt(missInts[j], id)
}
}
}
return resolved.filter((id): id is string => id !== undefined)
}
/**
* UUID-level neighbor lookup over the BigInt provider contract: resolves
* the anchor via `getInt` (unmapped → empty), then maps returned entity
* ints back to UUIDs. Shared by the traversal paths and the
* TripleIntelligenceSystem adapter.
*/
private async getNeighborUuids(
uuid: string,
options?: { direction?: 'in' | 'out' | 'both'; limit?: number; offset?: number }
): Promise<string[]> {
const entityInt = this.graphEntityInt(uuid)
if (entityInt === undefined) return []
const neighborInts = await this.graphIndex.getNeighbors(entityInt, options)
return this.entityIntsToUuids(neighborInts)
}
// -------------------------------------------------------------------------
/**
* Create a relationship (verb) between two entities
*
@ -1790,7 +1927,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* @param params.service - Multi-tenancy service name
* @param params.confidence - Relationship certainty 0-1
* @param params.evidence - Why this relationship exists
* @returns Promise that resolves to the relationship ID
* @returns Promise that resolves to the relationship ID. **Contract (8.0):**
* relationship ids are always UUIDs (36-char 8-4-4-4-12 hex). Brainy
* generates every verb id itself (`relate()` takes no custom id), so the
* UUID shape is a guarantee, not a coincidence — graph-index providers may
* key their verb-int interning on the raw UUID bytes for lossless
* reverse lookup.
*
* @example
* // Basic relationship creation
@ -1918,7 +2060,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// This prevents infinite loops where same relationship is created repeatedly
// Bug #1 showed incrementing verb counts (7→8→9...) indicating duplicates
// OPTIMIZATION: Use GraphAdjacencyIndex for O(log n) lookup instead of O(n) storage scan
const verbIds = await this.graphIndex.getVerbIdsBySource(params.from)
// 8.0 BigInt boundary: unmapped source → no existing relations to check.
const dupSourceInt = this.graphEntityInt(params.from)
const verbIds = dupSourceInt === undefined
? []
: await this.resolveVerbIntsToIds(await this.graphIndex.getVerbIdsBySource(dupSourceInt))
// Batch-load verbs for 5x faster duplicate checking on GCS
// GCS: 5 verbs = 1×50ms vs 5×50ms = 250ms (5x faster)
@ -1969,6 +2115,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
createdAt: Date.now()
}
// 8.0 BigInt boundary: resolve endpoint ints ONCE (getOrAssign — both
// entities were existence-checked above) and mirror them onto the verb.
const { sourceInt, targetInt } = this.resolveVerbEndpointInts(verb)
// Execute atomically with transaction system
await this.transactionManager.executeTransaction(async (tx) => {
// Operation 1: Save verb vector data
@ -1990,7 +2140,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Operation 3: Add to graph index for O(1) lookups
tx.addOperation(
new AddToGraphIndexOperation(this.graphIndex, verb)
new AddToGraphIndexOperation(
this.graphIndex, verb, sourceInt, targetInt,
(verbInt) => this.cacheVerbInt(verbInt, id)
)
)
// Create bidirectional if requested
@ -2000,7 +2153,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
...verb,
id: reverseId,
sourceId: params.to,
targetId: params.from
targetId: params.from,
// Endpoints swap, so the derived ints swap with them.
sourceInt: targetInt,
targetInt: sourceInt
}
// Operation 4: Save reverse verb vector data
@ -2022,7 +2178,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Operation 6: Add reverse relationship to graph index
tx.addOperation(
new AddToGraphIndexOperation(this.graphIndex, reverseVerb)
new AddToGraphIndexOperation(
this.graphIndex, reverseVerb, targetInt, sourceInt,
(verbInt) => this.cacheVerbInt(verbInt, reverseId)
)
)
}
})
@ -2053,12 +2212,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Get verb data before deletion for rollback
const verb = await this.storage.getVerb(id)
// 8.0 BigInt boundary: resolve endpoint ints before the transaction so
// a rollback can re-add through the BigInt addVerb contract.
const endpointInts = verb ? this.resolveVerbEndpointInts(verb) : undefined
// Execute atomically with transaction system
await this.transactionManager.executeTransaction(async (tx) => {
// Operation 1: Remove from graph index
if (verb) {
if (verb && endpointInts) {
tx.addOperation(
new RemoveFromGraphIndexOperation(this.graphIndex, verb as any)
new RemoveFromGraphIndexOperation(
this.graphIndex, verb, endpointInts.sourceInt, endpointInts.targetInt
)
)
}
@ -2158,6 +2323,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
createdAt: existingAny.createdAt
}
// 8.0 BigInt boundary: endpoints are unchanged across a type swap, so one
// resolution serves both the remove (rollback re-add) and the re-add.
const reindexInts = typeChanged ? this.resolveVerbEndpointInts(verbForIndex) : undefined
await this.transactionManager.executeTransaction(async (tx) => {
tx.addOperation(
new UpdateVerbMetadataOperation(this.storage, params.id, updatedMetadata)
@ -2165,12 +2334,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// If the verb type changed, re-index in graph adjacency so traversal-by-type
// stays consistent. The id is preserved across the swap.
if (typeChanged) {
if (typeChanged && reindexInts) {
tx.addOperation(
new RemoveFromGraphIndexOperation(this.graphIndex, existing as any)
new RemoveFromGraphIndexOperation(
this.graphIndex, existing, reindexInts.sourceInt, reindexInts.targetInt
)
)
tx.addOperation(
new AddToGraphIndexOperation(this.graphIndex, verbForIndex)
new AddToGraphIndexOperation(
this.graphIndex, verbForIndex, reindexInts.sourceInt, reindexInts.targetInt,
(verbInt) => this.cacheVerbInt(verbInt, params.id)
)
)
}
})
@ -3027,10 +3201,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
k
)
// Convert int IDs to UUIDs and paginate
// Convert int IDs (BigInt at the provider boundary) to UUIDs and
// paginate. Number() narrowing is lossless under the u32 guard.
const idMapper = this.metadataIndex.getIdMapper()
const allUuids = sortedIntIds
.map(intId => idMapper.getUuid(intId))
.map(intId => idMapper.getUuid(Number(intId)))
.filter((uuid): uuid is string => uuid !== undefined)
const pageIds = allUuids.slice(offset, offset + limit)
@ -3759,8 +3934,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
)
for (const verb of allVerbs) {
const { sourceInt, targetInt } = this.resolveVerbEndpointInts(verb)
tx.addOperation(
new RemoveFromGraphIndexOperation(this.graphIndex, verb as any)
new RemoveFromGraphIndexOperation(this.graphIndex, verb, sourceInt, targetInt)
)
tx.addOperation(
new DeleteVerbMetadataOperation(this.storage, verb.id)
@ -4218,6 +4394,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
}
// getGraphIndex() will rebuild automatically if data exists (via _initializeGraphIndex)
// 8.0 u64 contract: the clone has its own metadata index (and idMapper),
// so thread the clone's resolver into its graph index + storage.
clone.wireGraphIdResolver()
// Mark as initialized
clone.initialized = true
clone.dimensions = this.dimensions
@ -4318,6 +4498,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
this.graphIndex = await (this.storage as any).getGraphIndex()
}
// 8.0 u64 contract: the branch switch recreated the metadata index (and
// idMapper) plus the graph index — re-thread the shared resolver.
this.wireGraphIdResolver()
// Reset lazy loading state when switching branches
// Indexes contain data from previous branch, must rebuild for new branch
this.lazyRebuildCompleted = false
@ -5697,11 +5881,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
*/
getTripleIntelligence(): TripleIntelligenceSystem {
if (!this._tripleIntelligence) {
// Use core components directly - no lazy loading needed
// Use core components directly - no lazy loading needed.
// TripleIntelligenceSystem speaks UUIDs; adapt the BigInt graph-index
// contract through the coordinator's UUID-level helper so the int
// conversion stays at this boundary.
this._tripleIntelligence = new TripleIntelligenceSystem(
this.metadataIndex,
this.index,
this.graphIndex,
{
getNeighbors: (id: string, direction?: 'in' | 'out' | 'both') =>
this.getNeighborUuids(id, { direction }),
size: () => this.graphIndex.size()
},
async (text: string) => this.embedder(text),
this.storage
)
@ -7917,13 +8108,20 @@ export class Brainy<T = any> implements BrainyInterface<T> {
verbTypes?: Set<VerbType>,
limit?: number
): Promise<string[]> {
const neighbors = await this.graphIndex.getNeighbors(nodeId, { direction: graphDirection, limit })
// 8.0 BigInt boundary: unmapped node → no relations.
const nodeInt = this.graphEntityInt(nodeId)
if (nodeInt === undefined) return []
const neighbors = this.entityIntsToUuids(
await this.graphIndex.getNeighbors(nodeInt, { direction: graphDirection, limit })
)
if (!verbTypes || verbTypes.size === 0) return neighbors
// Gather candidate edges in the relevant direction(s).
const verbIds: string[] = []
if (graphDirection !== 'in') verbIds.push(...await this.graphIndex.getVerbIdsBySource(nodeId))
if (graphDirection !== 'out') verbIds.push(...await this.graphIndex.getVerbIdsByTarget(nodeId))
const verbInts: bigint[] = []
if (graphDirection !== 'in') verbInts.push(...await this.graphIndex.getVerbIdsBySource(nodeInt))
if (graphDirection !== 'out') verbInts.push(...await this.graphIndex.getVerbIdsByTarget(nodeInt))
const verbIds = await this.resolveVerbIntsToIds(verbInts)
const verbs = await this.graphIndex.getVerbsBatchCached(verbIds)
const neighborSet = new Set(neighbors)
@ -8397,14 +8595,52 @@ export class Brainy<T = any> implements BrainyInterface<T> {
if (subtypeFilter !== undefined) {
// Multi-hop BFS with per-hop (verbType, subtype) predicate. Works on the
// open-core JS path at any depth. If a graph-index provider exposes a
// open-core JS path at any depth. When a graph-index provider exposes a
// faster `findConnectedSubtype` (native BFS with the same semantics),
// we'll route through it transparently — same pattern as every other
// provider hook.
// `nativeSubtypeBfs` routes through it transparently — same pattern as
// every other provider hook.
const subtypeArr = Array.isArray(subtypeFilter) ? subtypeFilter : [subtypeFilter]
const subtypeSet = new Set(subtypeArr)
// Native fast path (D.3): single verb type + single subtype + outgoing
// walk match the native BFS semantics exactly (out-edges only, source
// excluded, visited-set cycle guard). Entity ints in, entity ints out —
// UUID conversion stays at this boundary. Returns null when the query
// shape (or provider) can't take the native route.
const nativeSubtypeBfs = async (
anchor: string,
walk: 'in' | 'out' | 'both'
): Promise<Set<string> | null> => {
if (walk !== 'out') return null
if (via === undefined || Array.isArray(via)) return null
if (subtypeArr.length !== 1) return null
const provider = this.graphIndex as Partial<{
findConnectedSubtype(
sourceInt: bigint,
verbTypeIndex: number,
subtype: string | null,
depth: number,
limit?: number | null
): Promise<bigint[]>
}>
if (typeof provider.findConnectedSubtype !== 'function') return null
const anchorInt = this.graphEntityInt(anchor)
if (anchorInt === undefined) return new Set() // unmapped → no relations
const verbTypeIndex = TypeUtils.getVerbIndex(via as VerbType)
// No limit: match the JS BFS exactly — overall result limiting happens
// downstream against existingResults.
const reachedInts = await provider.findConnectedSubtype(
anchorInt, verbTypeIndex, subtypeArr[0], effectiveDepth, null
)
return new Set(this.entityIntsToUuids(reachedInts))
}
const bfsWithSubtype = async (anchor: string, walk: 'in' | 'out' | 'both'): Promise<Set<string>> => {
const nativeResult = await nativeSubtypeBfs(anchor, walk)
if (nativeResult !== null) return nativeResult
const visited = new Set<string>([anchor])
const reached = new Set<string>()
let frontier = new Set<string>([anchor])
@ -8730,13 +8966,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
if (constraints.to) {
// Check if this entity connects TO the target (O(1) lookup)
const outgoingNeighbors = await this.graphIndex.getNeighbors(result.id, 'out')
const outgoingNeighbors = await this.getNeighborUuids(result.id, { direction: 'out' })
hasConnection = outgoingNeighbors.includes(constraints.to)
}
if (constraints.from && !hasConnection) {
// Check if this entity connects FROM the source (O(1) lookup)
const incomingNeighbors = await this.graphIndex.getNeighbors(result.id, 'in')
const incomingNeighbors = await this.getNeighborUuids(result.id, { direction: 'in' })
hasConnection = incomingNeighbors.includes(constraints.from)
}
@ -9368,6 +9604,24 @@ export class Brainy<T = any> implements BrainyInterface<T> {
await this.lazyRebuildPromise
}
/**
* @description Wire the shared UUID ↔ int resolver (the metadata index's
* idMapper) into the JS graph index and the storage layer (8.0 u64
* contract). Called after the graph index is resolved on init, fork, and
* checkout so every consumer of the BigInt boundary shares one int
* universe. Native graph-index providers carry their own mapper and don't
* expose `setEntityIdMapper` — the storage-level wiring still applies so
* its verb read paths can convert UUIDs before provider calls.
*/
private wireGraphIdResolver(): void {
const resolver = this.metadataIndex.getIdMapper()
this.storage.setGraphEntityIdResolver(resolver)
const jsGraphIndex = this.graphIndex as Partial<Pick<GraphAdjacencyIndex, 'setEntityIdMapper'>>
if (typeof jsGraphIndex.setEntityIdMapper === 'function') {
jsGraphIndex.setEntityIdMapper(resolver)
}
}
/**
* @description Wire the HNSW connections codec (2.4.0 #3). Activates when
* BOTH (a) the `graph:compression` provider is registered (cortex registers