soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

perf(8.0): bound per-id generation history chains (O(W+L) resident, was O(N))

Browse source 
The MVCC time-travel layer kept nounChains/verbChains as unbounded
Map<id, number[]> — one resident chain per id ever touched across retained
history (O(N) RAM, defeating billion-scale time travel). Replace with a hot-tail
window + bounded cold LRU + a mutex-free bulk resolver, keeping resolveAt exactly
correct.

The most-recent W generations stay resident as full chains (the common recent-pin
read is O(log), zero scan); deeper pins reconstruct one id's chain on demand into
an L-bounded LRU. Reconstruction is LOCK-LIGHT — it holds no commit mutex, so a
historical read never stalls writers; correctness holds because a live pin's
answer is always > minPinnedGeneration, which compaction can never reclaim, and a
concurrently-reclaimed gen below that is skipped. materializeAtGeneration routes
through a new mutex-free resolveManyAt (one forward pass, O(R + |ids|)) — without
it a deep-pin materialize both regresses to O(N*R) and deadlocks on snapshotWith's
mutex. The cold cache is invalidated on re-touch to stay coherent. Resident RAM is
O(W*d + L), independent of N. 11-case test (oracle-vs-bruteforce, held-Db across
eviction + concurrent compaction, no-deadlock, write-path I/O-free); unit 1735 +
integration 613 (db-mvcc/db-temporal/db-asof green).
This commit is contained in:
David Snelling 2026-06-30 10:30:17 -07:00
parent 8f4787bb10
commit a859d6ecf8
4 changed files with 956 additions and 72 deletions
Download patch file Download diff file Expand all files Collapse all files

74
src/brainy.ts
View file

@ -7475,23 +7475,42 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const snapshotStorage = new MemoryStorage()
await snapshotStorage.init()
/** Copy one id's at-`generation` state into the snapshot (or ensure absence). */
const copyAt = async (kind: 'noun' | 'verb', id: string): Promise<void> => {
const resolved = await this.generationStore.resolveAt(kind, id, generation)
// Copy one id's at-`generation` state into the snapshot (or ensure absence)
// as a PURE LOOKUP against a precomputed `firstAfter` map (id → the first
// generation after the pin that touched it). An id absent from the map was
// untouched since the pin → its live storage state IS its at-`generation`
// state; an id present resolves from that generation's immutable
// before-image. Both `resolveManyAt` (builds the map) and
// `readGenerationRecord` (reads the before-image) are MUTEX-FREE, so this is
// safe to call inside the reconciliation pass below (which holds the commit
// mutex via `snapshotWith`) — a per-id `resolveAt` there would re-enter the
// mutex and DEADLOCK, and would regress the O(R) bulk copy to O(N·R).
const copyAt = async (
kind: 'noun' | 'verb',
id: string,
firstAfter: Map<string, number>
): Promise<void> => {
let record: { metadata: any; vector: any | null } | null = null
if (resolved.source === 'record') {
if (resolved.metadata !== null) {
record = { metadata: resolved.metadata, vector: resolved.vector }
}
} else if (resolved.source === 'current') {
const candidate = firstAfter.get(id)
if (candidate === undefined) {
// Untouched since the pin → live storage IS the at-`generation` state.
const raw =
kind === 'noun' ? await this.storage.readNounRaw(id) : await this.storage.readVerbRaw(id)
if (raw.metadata !== null) {
record = raw
if (raw.metadata !== null) record = raw
} else {
const before = await this.generationStore.readGenerationRecord(kind, candidate, id)
if (before === null) {
throw new Error(
`Generation record missing: _generations/${candidate}/prev/${id}.json ` +
`(store corrupted or records removed outside compactHistory())`
)
}
// A non-null metadata before-image is the at-`generation` live state; a
// null-metadata before-image (the create sentinel, or a metadata-less
// stored state) is absent at this generation.
if (before.metadata !== null) record = { metadata: before.metadata, vector: before.vector }
}
// `record === null` — absent at this generation (or a metadata-less
// stored state, which is not a live entity): writing nulls ensures the
// `record === null` — absent at this generation: writing nulls ensures the
// id is absent in the snapshot, which also un-copies ids created by
// transactions that commit mid-build.
if (kind === 'noun') {
@ -7516,25 +7535,42 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const id = entityIdFromCanonicalPath(path)
if (id) verbIds.add(id)
}
for (const id of nounIds) await copyAt('noun', id)
for (const id of verbIds) await copyAt('verb', id)
// ONE ascending pass per kind resolves the whole universe's first-after
// generations (O(R) getDelta reads, NOT O(N·R)), then each copy is a lookup.
const nounFirstAfter = await this.generationStore.resolveManyAt('noun', nounIds, generation)
const verbFirstAfter = await this.generationStore.resolveManyAt('verb', verbIds, generation)
for (const id of nounIds) await copyAt('noun', id, nounFirstAfter)
for (const id of verbIds) await copyAt('verb', id, verbFirstAfter)
// Reconciliation pass under the commit mutex: nothing can commit during
// this section, so afterwards the snapshot is exactly the at-`generation`
// record set even when transactions committed while the bulk copy ran.
// Reconciled ids join the enumeration sets so the vector-index build
// below sees them too.
// below sees them too. `resolveManyAt`/`readGenerationRecord` are mutex-free,
// so this section never re-enters the commit mutex (no deadlock).
await this.generationStore.snapshotWith(async () => {
const committedNow = this.generationStore.committedGeneration()
if (committedNow === watermark) return
const delta = await this.generationStore.changedBetween(watermark, committedNow)
for (const id of delta.nouns) {
const reconNouns = new Set<string>(delta.nouns)
const reconVerbs = new Set<string>(delta.verbs)
const reconNounFirstAfter = await this.generationStore.resolveManyAt(
'noun',
reconNouns,
generation
)
const reconVerbFirstAfter = await this.generationStore.resolveManyAt(
'verb',
reconVerbs,
generation
)
for (const id of reconNouns) {
nounIds.add(id)
await copyAt('noun', id)
await copyAt('noun', id, reconNounFirstAfter)
}
for (const id of delta.verbs) {
for (const id of reconVerbs) {
verbIds.add(id)
await copyAt('verb', id)
await copyAt('verb', id, reconVerbFirstAfter)
}
watermark = committedNow
})