soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat(8.0): u64 BigInt graph provider contract — punch list a-d,g,h

Browse source 
GraphIndexProvider now speaks BigInt at the boundary (D.2 mirror):
- getNeighbors/getVerbIdsBySource/getVerbIdsByTarget take entity ints and
  return entity/verb ints as bigint[]
- new REQUIRED verbIntsToIds(bigint[]) batch reverse resolver (L.7
  identity-fingerprint design — verb ids are UUIDs by contract, so the
  provider-side interning is losslessly reversible)
- addVerb(verb, sourceInt, targetInt) returns the interned verb int;
  removeVerb(verbId) joins the contract

Coordinator (brainy.ts) owns ALL UUID <-> int conversion: getOrAssign on
writes, getInt on reads (unmapped UUID -> empty result without calling the
provider), getUuid / verbIntsToIds on returns, plus a bounded ~100k-entry
insertion-order warm cache for verb-int -> verb-id pairs fed by addVerb
returns and resolver results. GraphVerb gains derived sourceInt/targetInt
(populated at add time, never persisted). findConnectedSubtype gains a
native fast path that routes single-type single-subtype outgoing BFS
through the provider when available.

JS GraphAdjacencyIndex satisfies the contract while staying string/u32-keyed
internally: entity ints resolve through the shared entity-id mapper (threaded
in by the coordinator on init/fork/checkout), verb ints come from an
in-process append-only interning map re-derived from storage on
rebuild/cold-start. ColumnStoreProvider widens the same way: addEntity/
removeEntity take bigint, sortTopK/filteredSortTopK return bigint[].

relate() now rejects a caller-supplied id with a teaching error — verb ids
are brainy-generated UUIDs by contract in 8.0 (previously a passed id was
silently ignored). No Roaring64 provider-boundary decode site exists yet;
the JS-internal column store stays Roaring32 and the Treemap decoder lands
with the first consumer of provider-returned filter buffers.

Public brain API unchanged. 1413 tests green (+10 new BigInt contract tests).
This commit is contained in:
David Snelling 2026-06-10 10:45:45 -07:00
parent 62f6472fa0
commit 2427bb7960
17 changed files with 1164 additions and 328 deletions
Download patch file Download diff file Expand all files Collapse all files

46
src/indexes/columnStore/ColumnStore.ts
View file

@ -58,12 +58,16 @@ interface HeapEntry {
/**
* Unified column store coordinator.
*
* Entity ints cross the {@link ColumnStoreProvider} boundary as `bigint`
* (8.0 u64 contract); internally this JS baseline stays u32 — `Number(bigint)`
* narrowing is lossless under the `EntityIdSpaceExceeded` u32 guard.
*
* @example
* const store = new ColumnStore()
* await store.init(storage, idMapper)
* store.addEntity(42, { createdAt: Date.now(), status: 'active' })
* store.addEntity(42n, { createdAt: Date.now(), status: 'active' })
* await store.flush()
* const newest = await store.sortTopK('createdAt', 'desc', 10)
* const newest = await store.sortTopK('createdAt', 'desc', 10) // bigint[]
*/
export class ColumnStore implements ColumnStoreProvider {
private storage!: StorageAdapter
@ -170,8 +174,13 @@ export class ColumnStore implements ColumnStoreProvider {
* one entry per element is added.
*
* Auto-flushes when any tail buffer exceeds its threshold.
*
* @param entityIntId - The entity's interned integer ID (u64-safe BigInt;
* narrowed to u32 internally under the `EntityIdSpaceExceeded` guard)
* @param fields - Map of field name → value (or array of values for multi-value)
*/
addEntity(entityIntId: number, fields: Record<string, unknown>): void {
addEntity(entityIntId: bigint, fields: Record<string, unknown>): void {
const intId = Number(entityIntId)
for (const [field, rawValue] of Object.entries(fields)) {
if (rawValue === undefined || rawValue === null) continue
@ -179,11 +188,11 @@ export class ColumnStore implements ColumnStoreProvider {
if (Array.isArray(rawValue)) {
for (const v of rawValue) {
if (v !== undefined && v !== null) {
this.pushToBuffer(field, v, entityIntId, true)
this.pushToBuffer(field, v, intId, true)
}
}
} else {
this.pushToBuffer(field, rawValue, entityIntId, false)
this.pushToBuffer(field, rawValue, intId, false)
}
}
}
@ -195,11 +204,16 @@ export class ColumnStore implements ColumnStoreProvider {
* during ALL queries) and removes any pending entries from tail buffers.
* The global bitmap is persisted in the manifest on flush and cleared
* during compaction.
*
* @param entityIntId - The entity's interned integer ID (u64-safe BigInt;
* narrowed to u32 internally under the `EntityIdSpaceExceeded` guard)
*/
removeEntity(entityIntId: number): void {
removeEntity(entityIntId: bigint): void {
const intId = Number(entityIntId)
// Remove from tail buffers (pending writes)
for (const [, buffer] of this.tailBuffers) {
buffer.remove(entityIntId)
buffer.remove(intId)
}
// Add to global deleted bitmap for every known field
@ -209,7 +223,7 @@ export class ColumnStore implements ColumnStoreProvider {
deleted = new RoaringBitmap32()
this.deletedEntities.set(field, deleted)
}
deleted.add(entityIntId)
deleted.add(intId)
}
}
@ -273,26 +287,28 @@ export class ColumnStore implements ColumnStoreProvider {
}
/**
* Sort top-K: return K entity int IDs in sorted order.
* Sort top-K: return K entity int IDs in sorted order (u64-safe BigInt).
*
* Uses k-way merge across all segment cursors + tail buffer via a min/max heap.
* Complexity: O(K log S) where S = number of cursors.
*/
async sortTopK(field: string, order: 'asc' | 'desc', k: number): Promise<number[]> {
return this.mergeSort(field, order, k, null)
async sortTopK(field: string, order: 'asc' | 'desc', k: number): Promise<bigint[]> {
const intIds = await this.mergeSort(field, order, k, null)
return intIds.map((id) => BigInt(id))
}
/**
* Filtered sort top-K: return K entity int IDs in sorted order,
* considering only entities present in the filter bitmap.
* Filtered sort top-K: return K entity int IDs in sorted order (u64-safe
* BigInt), considering only entities present in the filter bitmap.
*/
async filteredSortTopK(
filterBitmap: RoaringBitmap32,
field: string,
order: 'asc' | 'desc',
k: number
): Promise<number[]> {
return this.mergeSort(field, order, k, filterBitmap)
): Promise<bigint[]> {
const intIds = await this.mergeSort(field, order, k, filterBitmap)
return intIds.map((id) => BigInt(id))
}
/**

24
src/indexes/columnStore/types.ts
View file

@ -170,13 +170,19 @@ export const FLAG_MULTI_VALUE = 0x01
* Brainy ships a TypeScript baseline that implements this interface.
* Cortex registers a Rust-accelerated implementation at higher priority.
* The MetadataIndex coordinator calls whichever is registered.
*
* **8.0 u64 contract — BigInt entity ints at the boundary.** Entity ints flow
* in and out as `bigint` so a native u64-keyed column store loses nothing at
* the wrapper. Brainy's JS baseline stays u32 internally (`Number(bigint)`
* narrowing is lossless under the shipped `EntityIdSpaceExceeded` u32 guard)
* and its filter/range bitmaps stay Roaring32 — only this boundary widens.
*/
export interface ColumnStoreProvider {
/**
* Initialize the column store: discover fields, load manifests.
*
* @param storage - The storage adapter for reading/writing segment files
* @param idMapper - Shared EntityIdMapper for UUID ↔ u32 conversion
* @param idMapper - Shared EntityIdMapper for UUID ↔ int conversion
*/
init(storage: StorageAdapter, idMapper: EntityIdMapper): Promise<void>
@ -184,17 +190,17 @@ export interface ColumnStoreProvider {
* Index an entity's field values. For multi-value fields (arrays),
* one entry per element is added to the column.
*
* @param entityIntId - The entity's u32 integer ID (from EntityIdMapper)
* @param entityIntId - The entity's interned integer ID (u64-safe BigInt)
* @param fields - Map of field name → value (or array of values for multi-value)
*/
addEntity(entityIntId: number, fields: Record<string, unknown>): void
addEntity(entityIntId: bigint, fields: Record<string, unknown>): void
/**
* Remove an entity from all indexed columns by adding tombstones.
*
* @param entityIntId - The entity's u32 integer ID
* @param entityIntId - The entity's interned integer ID (u64-safe BigInt)
*/
removeEntity(entityIntId: number): void
removeEntity(entityIntId: bigint): void
/**
* Point filter: find all entities where field equals value.
@ -222,9 +228,9 @@ export interface ColumnStoreProvider {
* @param field - Field to sort by
* @param order - 'asc' or 'desc'
* @param k - Maximum number of results
* @returns Sorted array of entity int IDs
* @returns Sorted array of entity int IDs (u64-safe BigInt)
*/
sortTopK(field: string, order: 'asc' | 'desc', k: number): Promise<number[]>
sortTopK(field: string, order: 'asc' | 'desc', k: number): Promise<bigint[]>
/**
* Filtered sort top-K: same as sortTopK but only considers entities
@ -234,14 +240,14 @@ export interface ColumnStoreProvider {
* @param field - Field to sort by
* @param order - 'asc' or 'desc'
* @param k - Maximum number of results
* @returns Sorted array of entity int IDs (subset of filterBitmap)
* @returns Sorted array of entity int IDs (u64-safe BigInt, subset of filterBitmap)
*/
filteredSortTopK(
filterBitmap: RoaringBitmap32,
field: string,
order: 'asc' | 'desc',
k: number
): Promise<number[]>
): Promise<bigint[]>
/**
* Get all distinct values for a field across all segments.