From 2a03fae0e20091f433018212ac05f68a54d95ec2 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Fri, 17 Jul 2026 16:34:45 -0700 Subject: [PATCH 01/29] =?UTF-8?q?feat:=20brain.auditGraph()=20=E2=80=94=20?= =?UTF-8?q?read-only=20graph-truth=20audit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - New public API: walks every canonical relationship record, queries the same read path applications use (related() with all visibility tiers included), and classifies every discrepancy into its failure family: missingFromReads (records the read path omits — stale adjacency), danglingEndpoints (endpoint entity gone — the partial-delete scar class), readOnlyVerbIds (read-path edges with no canonical record — ghosts). Design-hidden internal/system edges are counted separately so intentional hiding is never misclassified as loss. Counts exact; example lists capped with an explicit truncatedExamples flag; loud narration on incoherence; mutates nothing. - Classification core lives in src/graph/graphAudit.ts behind injected seams (canonical walks + the end-to-end read) so the discrepancy logic is testable in isolation; brainy wires the seams to storage walks and related(). - getNounIdsWithPagination now refuses an undecodable resume cursor loudly — the third and final pagination walk brought under the 8.5.2 cursor contract. - Types exported: GraphAuditReport, GraphAuditDiscrepancy. Docs: the inspection guide gains the audit -> repair -> audit verification ritual. --- RELEASES.md | 21 +++ docs/guides/inspection.md | 28 ++++ src/brainy.ts | 83 ++++++++++ src/graph/graphAudit.ts | 214 ++++++++++++++++++++++++++ src/index.ts | 4 + src/storage/baseStorage.ts | 8 + tests/integration/graph-audit.test.ts | 164 ++++++++++++++++++++ 7 files changed, 522 insertions(+) create mode 100644 src/graph/graphAudit.ts create mode 100644 tests/integration/graph-audit.test.ts diff --git a/RELEASES.md b/RELEASES.md index 540fa6fb..5ae0845d 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -10,6 +10,27 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so --- +## v8.6.0 — 2026-07-17 (brain.auditGraph — the graph-truth verification instrument) + +A minor release adding one new public API, from the fleet's graph-trust program: a read-only +audit that **proves whether relationship reads return stored truth** on a given brain. + +- **`brain.auditGraph(options?)`** walks every canonical relationship record, queries the same + read path applications use (`related()` / VFS `readdir`) with all visibility tiers included, + and classifies every discrepancy into its failure family: `missingFromReads` (records the + read path omits — a stale adjacency index), `danglingEndpoints` (relationships whose endpoint + entity no longer exists — the historical partial-delete scar class), and `readOnlyVerbIds` + (read-path edges with no stored record — ghosts). Design-hidden internal/system edges are + counted separately so intentional hiding is never misclassified as loss. Counts are exact; + example lists cap at `maxExamples` with an explicit `truncatedExamples` flag; the result is + narrated loudly on incoherence. Mutates nothing — safe on a live brain. +- The operational pairing: audit → if incoherent, `repairIndex()` → audit again. A `coherent` + report after the repair is the verified all-clear. Run it after any engine upgrade, restore, + or migration. Guide: `docs/guides/inspection.md`. +- Also: `getNounIds` pagination now refuses an undecodable resume cursor loudly (the same + contract `getNouns`/`getVerbs` gained in 8.5.2 — the third and final walk brought under it). +- Types exported: `GraphAuditReport`, `GraphAuditDiscrepancy`. + ## v8.5.2 — 2026-07-17 (aggregation backfill: exception-safe, generation-verified, and loud) Hardening patch from a migration incident (a byte-copied store on new hardware; the service diff --git a/docs/guides/inspection.md b/docs/guides/inspection.md index 015c5118..240e81ae 100644 --- a/docs/guides/inspection.md +++ b/docs/guides/inspection.md @@ -166,6 +166,34 @@ brainy inspect diff /data/brain-prod /data/brain-staging Sample-based — for a full diff, dump both with `inspect dump` and compare the JSONL. +## Auditing graph-read truth + +`brain.auditGraph()` (8.6.0+) proves — or disproves — that relationship reads +return canonical truth on a given brain, without mutating anything. It walks +every stored relationship record, asks the same read path your application +uses (`related()`, VFS `readdir`) with every visibility tier included, and +classifies every discrepancy: + +```typescript +const report = await brain.auditGraph() + +report.coherent // true = related()/readdir can be trusted on this brain +report.missingFromReadsCount // records the read path omits — stale index +report.danglingEndpointsCount // relationships whose endpoint entity is gone +report.readOnlyCount // read-path edges with NO stored record — ghosts +report.visibilityHiddenCount // internal/system edges hidden by design (not a fault) +``` + +Counts are always exact; the example lists (`missingFromReads`, +`danglingEndpoints`, `readOnlyVerbIds`) are capped at `maxExamples` +(default 100) and `truncatedExamples` says so when they are. + +Run it after any engine upgrade, restore, or migration. If it reports +discrepancies, run `brain.repairIndex()` and audit again — a `coherent` +report after the repair is the verified statement that the heal worked. +Cost: one relationship-record walk plus one indexed read per distinct +source entity — safe on a live brain. + ## Repairing a corrupted store If invariants fail and you suspect index corruption, `inspect repair` diff --git a/src/brainy.ts b/src/brainy.ts index bc5cedd5..12a43086 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -46,6 +46,7 @@ import { pageRank, MinHeap } from './graph/analyticsFallback.js' +import { runGraphAudit, type GraphAuditReport } from './graph/graphAudit.js' import { createPipeline } from './streaming/pipeline.js' import { configureLogger, LogLevel, prodLog } from './utils/logger.js' import { setGlobalCache } from './utils/unifiedCache.js' @@ -15489,6 +15490,88 @@ export class Brainy implements BrainyInterface { ) } + /** + * Read-only graph-truth audit — proves (or disproves) that relationship + * reads return canonical truth on THIS brain, and classifies every + * discrepancy into its failure family: + * + * - `missingFromReads` — canonical verb records the read path omits + * (PRESENT BUT INVISIBLE: adjacency/membership staleness) + * - `danglingEndpoints` — verbs whose endpoint entity is gone (SCAR class) + * - `readOnlyVerbIds` — read-path edges with no canonical record (GHOSTS) + * + * Design-hidden edges (internal/system visibility) are counted separately — + * the audit reads with every visibility tier included, so intentional + * hiding is never misclassified as index loss. Mutates nothing; safe on a + * live brain (cost: one canonical walk + one indexed read per source). + * Run it after any engine upgrade, restore, or migration; a `coherent` + * report is the verified statement that `related()`/`readdir` can be + * trusted. If it reports discrepancies, `repairIndex()` is the sanctioned + * heal — re-run the audit afterwards to prove the repair. + * + * @param options.maxExamples - Cap per example list (counts stay exact). Default 100. + * @returns The full audit report; also narrated via logs (loud on incoherence). + * @since 8.6.0 + */ + async auditGraph(options: { maxExamples?: number } = {}): Promise { + await this.ensureInitialized({ needs: ['graph'] }) + + const PAGE = 1000 + return runGraphAudit( + { + eachNounId: async (consume) => { + let cursor: string | undefined + let offset = 0 + for (;;) { + const page = await (this.storage as unknown as { + getNounIdsWithPagination(o: { + limit: number + offset?: number + cursor?: string + }): Promise<{ ids: string[]; hasMore: boolean; nextCursor?: string }> + }).getNounIdsWithPagination( + cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset } + ) + for (const id of page.ids) consume(id) + if (!page.hasMore || page.ids.length === 0) break + if (page.nextCursor) cursor = page.nextCursor + else offset += page.ids.length + } + }, + eachVerb: async (consume) => { + let cursor: string | undefined + let offset = 0 + for (;;) { + const page = await this.storage.getVerbs({ + pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset } + }) + for (const verb of page.items) { + const v = verb as unknown as Record + consume({ + id: String(v.id), + type: String(v.verb ?? 'unknown'), + sourceId: String(v.sourceId), + targetId: String(v.targetId), + visibility: typeof v.visibility === 'string' ? v.visibility : undefined + }) + } + if (!page.hasMore || page.items.length === 0) break + if (page.nextCursor) cursor = page.nextCursor + else offset += page.items.length + } + }, + readRelationsFrom: async (sourceId) => + this.related({ + from: sourceId, + includeInternal: true, + includeSystem: true, + limit: 100000 + }) + }, + options + ) + } + async repairIndex(): Promise { await this.ensureInitialized() diff --git a/src/graph/graphAudit.ts b/src/graph/graphAudit.ts new file mode 100644 index 00000000..d0e44cd9 --- /dev/null +++ b/src/graph/graphAudit.ts @@ -0,0 +1,214 @@ +/** + * @module graph/graphAudit + * @description Read-only graph-truth audit — the graph sibling of `repairIndex()`'s + * diagnosis half. Verifies three layers against each other without mutating anything: + * + * 1. CANONICAL verb records (the storage walk — the source of truth) + * 2. the RELATIONSHIP READ PATH (`related()` with all visibility tiers — exactly + * what application reads like a VFS `readdir` consult) + * 3. ENTITY ENDPOINTS (does each verb's source/target still exist?) + * + * and classifies every discrepancy into the three failure families production + * incidents have shown: + * + * - `missingFromReads` — a canonical verb record the read path does NOT return + * for its source: PRESENT BUT INVISIBLE (adjacency/membership staleness). + * - `danglingEndpoints` — a canonical verb whose endpoint entity is gone: + * the SCAR class (write-path loss / partial delete). + * - `readOnlyVerbIds` — the read path returns an edge with NO canonical + * record: GHOST edges (stale index entries). + * + * `visibilityHiddenCount` is reported separately: an internal/system edge that is + * indexed and present but hidden from DEFAULT reads is working as designed — the + * audit reads with all tiers included so design-hiding is never misclassified as + * index loss. + * + * Full counts are always exact; only the example LISTS are capped (`maxExamples`) + * — a capped report says so via `truncatedExamples`, never silently. + */ + +import { prodLog } from '../utils/logger.js' + +/** One discrepant relationship, identified fully enough to inspect by hand. */ +export interface GraphAuditDiscrepancy { + verbId: string + from: string + to: string + type: string +} + +export interface GraphAuditReport { + /** True iff every discrepancy count is zero — `related()` returns canonical truth. */ + coherent: boolean + verbsInCanonical: number + entitiesInCanonical: number + /** Distinct source entities whose read path was actually consulted (coverage honesty). */ + sourcesChecked: number + + /** PRESENT BUT INVISIBLE: canonical records the read path omits. */ + missingFromReadsCount: number + missingFromReads: GraphAuditDiscrepancy[] + + /** SCAR CLASS: canonical verbs with a missing endpoint entity. */ + danglingEndpointsCount: number + danglingEndpoints: Array + + /** GHOST EDGES: read-path verb ids with no canonical record. */ + readOnlyCount: number + readOnlyVerbIds: string[] + + /** Canonical verbs hidden from DEFAULT reads by design (internal/system visibility). */ + visibilityHiddenCount: number + + /** Example lists above were capped at maxExamples; counts remain exact. */ + truncatedExamples: boolean + durationMs: number +} + +/** A canonical verb record, as the audit needs it. */ +export interface AuditVerbRecord { + id: string + type: string + sourceId: string + targetId: string + visibility?: string +} + +/** The seams the audit runs over — injected so the walk is testable in isolation. */ +export interface GraphAuditDeps { + /** Stream every canonical entity id (id-only; no per-entity reads needed). */ + eachNounId(consume: (id: string) => void): Promise + /** Stream every canonical verb record. */ + eachVerb(consume: (verb: AuditVerbRecord) => void): Promise + /** + * The END-TO-END relationship read for one source, ALL visibility tiers + * included — must be the same path application reads consult. + */ + readRelationsFrom(sourceId: string): Promise> +} + +export interface GraphAuditOptions { + /** Cap on entries per example list (counts stay exact). Default 100. */ + maxExamples?: number +} + +export async function runGraphAudit( + deps: GraphAuditDeps, + options: GraphAuditOptions = {} +): Promise { + const maxExamples = options.maxExamples ?? 100 + const started = Date.now() + + // 1. Canonical entity ids — endpoint existence oracle. + const entityIds = new Set() + await deps.eachNounId((id) => entityIds.add(id)) + + // 2. Canonical verb walk: group by source, check endpoints, note visibility. + const canonicalVerbIds = new Set() + const bySource = new Map() + let verbsInCanonical = 0 + let visibilityHiddenCount = 0 + let danglingEndpointsCount = 0 + const danglingEndpoints: GraphAuditReport['danglingEndpoints'] = [] + + await deps.eachVerb((verb) => { + verbsInCanonical++ + canonicalVerbIds.add(verb.id) + const list = bySource.get(verb.sourceId) + if (list) list.push(verb) + else bySource.set(verb.sourceId, [verb]) + + if (verb.visibility === 'internal' || verb.visibility === 'system') { + visibilityHiddenCount++ + } + + const fromMissing = !entityIds.has(verb.sourceId) + const toMissing = !entityIds.has(verb.targetId) + if (fromMissing || toMissing) { + danglingEndpointsCount++ + if (danglingEndpoints.length < maxExamples) { + danglingEndpoints.push({ + verbId: verb.id, + from: verb.sourceId, + to: verb.targetId, + type: verb.type, + missingEnd: fromMissing && toMissing ? 'both' : fromMissing ? 'from' : 'to' + }) + } + } + }) + + // 3. Per-source read-path comparison. A verb must be returned by the read + // path of ITS OWN source — the exact consult a readdir/traversal makes. + let missingFromReadsCount = 0 + const missingFromReads: GraphAuditDiscrepancy[] = [] + let readOnlyCount = 0 + const readOnlyVerbIds: string[] = [] + const readOnlySeen = new Set() + + for (const [sourceId, verbs] of bySource) { + const readIds = new Set((await deps.readRelationsFrom(sourceId)).map((r) => r.id)) + + for (const verb of verbs) { + if (!readIds.has(verb.id)) { + missingFromReadsCount++ + if (missingFromReads.length < maxExamples) { + missingFromReads.push({ + verbId: verb.id, + from: verb.sourceId, + to: verb.targetId, + type: verb.type + }) + } + } + } + + for (const readId of readIds) { + if (!canonicalVerbIds.has(readId) && !readOnlySeen.has(readId)) { + readOnlySeen.add(readId) + readOnlyCount++ + if (readOnlyVerbIds.length < maxExamples) { + readOnlyVerbIds.push(readId) + } + } + } + } + + const coherent = + missingFromReadsCount === 0 && danglingEndpointsCount === 0 && readOnlyCount === 0 + + const report: GraphAuditReport = { + coherent, + verbsInCanonical, + entitiesInCanonical: entityIds.size, + sourcesChecked: bySource.size, + missingFromReadsCount, + missingFromReads, + danglingEndpointsCount, + danglingEndpoints, + readOnlyCount, + readOnlyVerbIds, + visibilityHiddenCount, + truncatedExamples: + missingFromReadsCount > missingFromReads.length || + danglingEndpointsCount > danglingEndpoints.length || + readOnlyCount > readOnlyVerbIds.length, + durationMs: Date.now() - started + } + + if (coherent) { + prodLog.info( + `[GraphAudit] coherent: ${verbsInCanonical} verbs across ${bySource.size} sources — ` + + `the read path returns canonical truth (${report.durationMs}ms)` + ) + } else { + prodLog.warn( + `[GraphAudit] INCOHERENT: ${missingFromReadsCount} present-but-invisible, ` + + `${danglingEndpointsCount} dangling-endpoint, ${readOnlyCount} ghost ` + + `(of ${verbsInCanonical} canonical verbs, ${bySource.size} sources, ` + + `${visibilityHiddenCount} visibility-hidden by design) — ${report.durationMs}ms` + ) + } + + return report +} diff --git a/src/index.ts b/src/index.ts index bdb5ff73..d1eab40a 100644 --- a/src/index.ts +++ b/src/index.ts @@ -28,6 +28,10 @@ export type { FileVersion } from './vfs/types.js' // Export diagnostics result type export type { DiagnosticsResult } from './brainy.js' +export type { + GraphAuditReport, + GraphAuditDiscrepancy +} from './graph/graphAudit.js' // Export Brainy configuration and types export type { diff --git a/src/storage/baseStorage.ts b/src/storage/baseStorage.ts index 7ad00dab..6daf09c0 100644 --- a/src/storage/baseStorage.ts +++ b/src/storage/baseStorage.ts @@ -2230,6 +2230,14 @@ export abstract class BaseStorage extends BaseStorageAdapter { const { limit, offset = 0, filter } = options const cursor = this.decodeNounWalkCursor(options.cursor) + if (options.cursor && cursor === null) { + // Same law as getNouns/getVerbs: an undecodable resume token FAILS + // instead of silently restarting the walk at offset 0. + throw BrainyError.storage( + `getNounIds: invalid pagination cursor '${options.cursor}' — cannot resume this walk. ` + + `Restart it without a cursor.` + ) + } const collected: Array<{ id: string; shard: number }> = [] const peekCount = cursor ? limit + 1 : offset + limit + 1 const startShard = cursor ? cursor.shard : 0 diff --git a/tests/integration/graph-audit.test.ts b/tests/integration/graph-audit.test.ts new file mode 100644 index 00000000..74fcc96a --- /dev/null +++ b/tests/integration/graph-audit.test.ts @@ -0,0 +1,164 @@ +/** + * @module tests/integration/graph-audit + * @description brain.auditGraph() — the read-only graph-truth instrument. + * Laws under test: + * (1) a healthy brain audits COHERENT: every canonical verb is returned by the + * read path of its source, endpoints exist, no ghosts; + * (2) design-hidden edges (internal/system visibility) are counted separately + * and never misclassified as index loss; + * (3) a verb whose endpoint entity was destroyed at the storage layer (the + * scar class) is flagged as a dangling endpoint, loudly; + * (4) the classification core flags present-but-invisible and ghost edges + * exactly (exercised via injected seams — manufacturing a genuinely stale + * adjacency index end-to-end would require corrupting internals the + * public API rightly refuses to corrupt). + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import * as fs from 'node:fs' +import * as os from 'node:os' +import * as path from 'node:path' +import { Brainy } from '../../src/index.js' +import { NounType, VerbType } from '../../src/types/graphTypes.js' +import { runGraphAudit, type AuditVerbRecord } from '../../src/graph/graphAudit.js' + +describe('brain.auditGraph() — graph-truth audit', () => { + let dir: string + let brain: any + + beforeEach(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-graph-audit-')) + brain = new Brainy({ + requireSubtype: false, + storage: { type: 'filesystem', path: dir }, + silent: true + }) + await brain.init() + }) + + afterEach(async () => { + await brain.close().catch(() => {}) + fs.rmSync(dir, { recursive: true, force: true }) + }) + + async function seedGraph(): Promise<{ ids: string[]; verbIds: string[] }> { + const ids: string[] = [] + for (let i = 0; i < 5; i++) { + ids.push( + await brain.add({ + data: `entity ${i}`, + type: NounType.Concept, + metadata: { n: i } + }) + ) + } + const verbIds: string[] = [] + verbIds.push(await brain.relate({ from: ids[0], to: ids[1], type: VerbType.RelatedTo })) + verbIds.push(await brain.relate({ from: ids[0], to: ids[2], type: VerbType.Contains })) + verbIds.push(await brain.relate({ from: ids[1], to: ids[3], type: VerbType.DependsOn })) + verbIds.push(await brain.relate({ from: ids[3], to: ids[4], type: VerbType.RelatedTo })) + return { ids, verbIds } + } + + it('audits a healthy brain as coherent, with exact counts', async () => { + const { ids } = await seedGraph() + const report = await brain.auditGraph() + + expect(report.coherent).toBe(true) + expect(report.verbsInCanonical).toBe(4) + expect(report.entitiesInCanonical).toBeGreaterThanOrEqual(ids.length) // VFS root etc. may add system nouns + expect(report.sourcesChecked).toBe(3) // ids[0], ids[1], ids[3] + expect(report.missingFromReadsCount).toBe(0) + expect(report.danglingEndpointsCount).toBe(0) + expect(report.readOnlyCount).toBe(0) + expect(report.truncatedExamples).toBe(false) + }) + + it('counts design-hidden edges separately and stays coherent', async () => { + const { ids } = await seedGraph() + await brain.relate({ + from: ids[2], + to: ids[4], + type: VerbType.RelatedTo, + visibility: 'internal' + }) + + const report = await brain.auditGraph() + expect(report.coherent).toBe(true) // hidden-by-design is NOT a discrepancy + expect(report.verbsInCanonical).toBe(5) + expect(report.visibilityHiddenCount).toBeGreaterThanOrEqual(1) + }) + + it('flags a destroyed endpoint as a dangling verb (the scar class)', async () => { + const { ids } = await seedGraph() + // Destroy ids[4] at the STORAGE layer (bypassing remove(), which would + // also delete its verbs) — the historical partial-delete scar shape. + await brain.storage.deleteNoun(ids[4]) + + const report = await brain.auditGraph() + expect(report.coherent).toBe(false) + expect(report.danglingEndpointsCount).toBe(1) + expect(report.danglingEndpoints[0].to).toBe(ids[4]) + expect(report.danglingEndpoints[0].missingEnd).toBe('to') + }) +}) + +describe('runGraphAudit classification core (injected seams)', () => { + const verb = (id: string, from: string, to: string): AuditVerbRecord => ({ + id, + type: 'relatedTo', + sourceId: from, + targetId: to + }) + + const deps = (opts: { + nouns: string[] + verbs: AuditVerbRecord[] + reads: Record // sourceId -> verb ids the read path returns + }) => ({ + eachNounId: async (consume: (id: string) => void) => { + for (const id of opts.nouns) consume(id) + }, + eachVerb: async (consume: (v: AuditVerbRecord) => void) => { + for (const v of opts.verbs) consume(v) + }, + readRelationsFrom: async (sourceId: string) => + (opts.reads[sourceId] ?? []).map((id) => ({ id })) + }) + + it('flags a canonical verb the read path omits — present but invisible', async () => { + const report = await runGraphAudit( + deps({ + nouns: ['A', 'B', 'C'], + verbs: [verb('v1', 'A', 'B'), verb('v2', 'A', 'C')], + reads: { A: ['v1'] } // v2 exists canonically but reads miss it + }) + ) + expect(report.coherent).toBe(false) + expect(report.missingFromReadsCount).toBe(1) + expect(report.missingFromReads[0].verbId).toBe('v2') + }) + + it('flags a read-path edge with no canonical record — a ghost', async () => { + const report = await runGraphAudit( + deps({ + nouns: ['A', 'B'], + verbs: [verb('v1', 'A', 'B')], + reads: { A: ['v1', 'ghost-9'] } + }) + ) + expect(report.coherent).toBe(false) + expect(report.readOnlyCount).toBe(1) + expect(report.readOnlyVerbIds).toEqual(['ghost-9']) + }) + + it('caps example lists but keeps counts exact, and says so', async () => { + const verbs = Array.from({ length: 10 }, (_, i) => verb(`v${i}`, 'A', 'B')) + const report = await runGraphAudit( + deps({ nouns: ['A', 'B'], verbs, reads: { A: [] } }), + { maxExamples: 3 } + ) + expect(report.missingFromReadsCount).toBe(10) + expect(report.missingFromReads.length).toBe(3) + expect(report.truncatedExamples).toBe(true) + }) +}) From e0f6e7722f888b6a3711e3868a23074d3df27552 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Fri, 17 Jul 2026 16:38:18 -0700 Subject: [PATCH 02/29] chore(release): 8.6.0 --- CHANGELOG.md | 5 +++++ package-lock.json | 4 ++-- package.json | 2 +- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 75d3cb6f..e89af335 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. +### [8.6.0](https://github.com/soulcraftlabs/brainy/compare/v8.5.2...v8.6.0) (2026-07-17) + +- feat: brain.auditGraph() — read-only graph-truth audit (2a03fae) + + ### [8.5.2](https://github.com/soulcraftlabs/brainy/compare/v8.5.1...v8.5.2) (2026-07-17) - fix: exception-safe aggregation backfill + generation-verified adoption + loud open-path guards (a77b064) diff --git a/package-lock.json b/package-lock.json index 6b12754f..71cba0fe 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraft/brainy", - "version": "8.5.2", + "version": "8.6.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraft/brainy", - "version": "8.5.2", + "version": "8.6.0", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index f2004a57..474e41dd 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraft/brainy", - "version": "8.5.2", + "version": "8.6.0", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", "module": "dist/index.js", From 6ef9fcb7a248453ba1e1a9f1107400368d92a3a5 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Fri, 17 Jul 2026 16:49:38 -0700 Subject: [PATCH 03/29] feat: scaled transact budgets + labeled timeout diagnostics + envelope docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - The transact apply budget scales with the batch — max(30s, opCount x 2s) — or is exactly the caller's new TransactOptions.timeoutMs. A flat 30s cap silently limited honest bulk work to ~15 operations on network-attached storage (~2s/op measured in a production import) while looking generous for small batches. Internal batch paths (removeMany chunks) get the same scaling via the shared transactTimeoutBudget helper. - TransactionTimeoutError now reports the operation it stopped at as i/N with the operation's name, elapsed vs budgeted time, and states the batch rolled back atomically and is retryable; context carries the fields programmatically. - The transact operational envelope is documented (optimistic-concurrency guide): budget math, chunking with ifAbsent idempotency, when to reach for addMany vs transact, and the precompute pattern (embedBatch + per-op vector) that keeps inference out of the commit path. - Embedding claims made honest per an end-to-end probe of the built dist: batch and single embedding paths produce bit-identical vectors and a vector-supplied add is fully searchable, but batch throughput on the default WASM engine measures comparable to sequential (~160ms/text) — the 5-10x speedup claim in the addMany JSDoc is replaced with the measured reality and the actual win (inference outside the budgeted write path). --- RELEASES.md | 22 ++++++++++ docs/guides/optimistic-concurrency.md | 32 ++++++++++++++ src/brainy.ts | 41 +++++++++++++----- src/db/types.ts | 10 +++++ src/transaction/Transaction.ts | 24 ++++++++++- src/transaction/errors.ts | 19 +++++++-- .../TransactionManager.unit.test.ts | 42 +++++++++++++++++++ 7 files changed, 175 insertions(+), 15 deletions(-) diff --git a/RELEASES.md b/RELEASES.md index 5ae0845d..94ce16ac 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -10,6 +10,28 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so --- +## v8.7.0 — 2026-07-17 (bulk-transact ergonomics: scaled budgets + timeout telemetry) + +The bulk-import ergonomics release, from a consumer's measured production incident (a serial +import on network-attached storage at ~2 s/op met a flat 30 s transaction budget): + +- **The transact apply budget now scales with the batch** — `max(30 s, opCount × 2 s)` — or + is exactly what you pass as the new `TransactOptions.timeoutMs`. A flat 30 s cap silently + limited honest bulk work to ~15 operations on slow disks while looking generous for small + batches. Internal batch paths (e.g. `removeMany` chunks) get the same scaling. +- **`TransactionTimeoutError` is a diagnosis, not just a failure**: it now reports the + operation it stopped at as `i/N` with the operation's name, elapsed vs budgeted time, and + states the batch rolled back atomically and is retryable. Its `context` carries the same + fields programmatically. +- **The transact envelope is documented** — batch sizing, budget math, chunking with + `ifAbsent` idempotency, and the precompute pattern (`embedBatch` + per-op `vector`) that + keeps model inference out of the commit path. Guide: `docs/guides/optimistic-concurrency.md`. +- Note: `brain.embed()` / `brain.embedBatch()` (the precompute APIs) already ship — public, + with native-provider passthrough, verified end-to-end (batch and single paths produce + bit-identical vectors; a vector-supplied `add` is fully searchable). Honest measurement: + on the default WASM engine, batch throughput ≈ sequential (~160 ms/text) — the precompute + win is keeping inference out of the budgeted commit path, not raw embedding speed. + ## v8.6.0 — 2026-07-17 (brain.auditGraph — the graph-truth verification instrument) A minor release adding one new public API, from the fleet's graph-trust program: a read-only diff --git a/docs/guides/optimistic-concurrency.md b/docs/guides/optimistic-concurrency.md index a21a8af0..268bc5fa 100644 --- a/docs/guides/optimistic-concurrency.md +++ b/docs/guides/optimistic-concurrency.md @@ -180,3 +180,35 @@ Brainy 8.0 has exactly two write-coordination counters, at two granularities: They compose: a `transact()` batch can carry per-entity `ifRev` checks *and* a whole-store `ifAtGeneration`; any failed check rejects the entire batch before anything is staged. Generations also power snapshots and time travel (`brain.now()`, `brain.asOf()`, `db.persist()`) — see the [consistency model](../concepts/consistency-model.md) and [Snapshots & Time Travel](./snapshots-and-time-travel.md). A snapshot or historical view captures each entity *including* its `_rev` at that moment, so reading the past and writing back with `ifRev` against the live state works exactly as you'd hope: the write fails if the entity moved since the state you copied from. + +## The transact envelope: batch size, budget, and bulk imports + +`transact()` applies its batch atomically under one commit — which means the whole batch +shares one **apply budget**. Since 8.7.0 the budget scales with the batch: +`max(30 s, opCount × 2 s)`, or exactly what you pass as `timeoutMs`. A tripped budget rolls +the entire batch back (nothing partial survives) and throws a retryable +`TransactionTimeoutError` that names the operation it stopped at, the batch size, and the +elapsed vs budgeted time — a diagnosis, not just a failure: + +``` +Transaction timed out at operation 41/120 ('add') — 246012ms elapsed, budget 240000ms. +The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch. +``` + +Practical envelope guidance for bulk work: + +1. **Precompute embeddings outside the commit path.** Embedding inside `transact()` spends + the budget on model inference. Use `brain.embedBatch(texts)` and pass each vector via + the op's `vector` field — the commit then pays only storage costs, and a retried batch + never re-pays inference. (The win is *where* the inference happens, not raw embedding + throughput: on the default WASM engine, batch and sequential embedding measure + comparably, ~160 ms/text; native embedding providers may batch faster.) +2. **Chunk very large imports** into batches of a few hundred ops with one `transact()` + each. You lose whole-import atomicity but keep per-chunk atomicity, bounded memory, and + resumability — pair with `ifAbsent` upserts so a retried chunk is idempotent. +3. **Slow disks change the math, not the contract.** On network-attached storage a single + op can cost ~2 s (canonical write + fsync + index maintenance). The scaled default + absorbs that; pass an explicit `timeoutMs` only when you know better than the scale. +4. **`addMany`/`relateMany` are the convenience tier** — they chunk and batch-embed for + you, with per-item error reporting instead of batch atomicity. Choose by what you need: + atomic-all-or-nothing → `transact()`; resilient bulk load → `addMany`. diff --git a/src/brainy.ts b/src/brainy.ts index 12a43086..25c91d0f 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -71,6 +71,7 @@ import type { } from './plugin.js' import { ConnectionsCodec } from './hnsw/connectionsCodec.js' import { TransactionManager } from './transaction/TransactionManager.js' +import { transactTimeoutBudget } from './transaction/Transaction.js' import { RevisionConflictError } from './transaction/RevisionConflictError.js' import { EntityNotFoundError, RelationNotFoundError } from './errors/notFound.js' import { @@ -1752,7 +1753,11 @@ export class Brainy implements BrainyInterface { captureAndCheck({ nouns, verbs } as CommitBeforeImages) } await this.generationStore.runWithoutGeneration(() => - this.transactionManager.executeTransaction(run) + this.transactionManager.executeTransaction(run, { + timeout: transactTimeoutBudget( + (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0) + ) + }) ) const timestamp = Date.now() // Bootstrap writes are not generation-stamped; emit without one. @@ -1764,7 +1769,12 @@ export class Brainy implements BrainyInterface { receipt = await this.generationStore.commitSingleOp({ touched, precommit: captureAndCheck, - execute: () => this.transactionManager.executeTransaction(run) + execute: () => + this.transactionManager.executeTransaction(run, { + timeout: transactTimeoutBudget( + (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0) + ) + }) }) } catch (err) { // A failed rollback that left the store inconsistent (a remove/update @@ -6655,10 +6665,13 @@ export class Brainy implements BrainyInterface { /** * Add multiple entities in a single batch operation * - * Uses batch embedding (embedBatch) to pre-compute all vectors in a single - * WASM forward pass instead of N individual embed() calls, providing 5-10x - * speedup on bulk inserts. Automatically adapts batch size and parallelism - * to the storage adapter (e.g., smaller batches for cloud storage). + * Uses batch embedding (embedBatch) to pre-compute all vectors before any + * storage write, keeping model inference out of the per-item write path. + * (On the default WASM engine, batch throughput is comparable to sequential + * embed() calls — measured ~160 ms/text either way; a native embedding + * provider may batch faster.) Automatically adapts batch size and + * parallelism to the storage adapter (e.g., smaller batches for cloud + * storage). * * @param params - Batch add parameters * @param params.items - Array of AddParams (same shape as brain.add()) @@ -7797,11 +7810,17 @@ export class Brainy implements BrainyInterface { ifAtGeneration: options?.ifAtGeneration, precommit: casPrecommit, execute: async () => { - await this.transactionManager.executeTransaction(async (tx) => { - for (const operation of plan.operations) { - tx.addOperation(operation) - } - }) + await this.transactionManager.executeTransaction( + async (tx) => { + for (const operation of plan.operations) { + tx.addOperation(operation) + } + }, + // Budget scales with the batch (or the caller's explicit + // timeoutMs): a flat 30s cap silently limited honest bulk work + // to ~15 ops on network disks (~2s/op measured in the field). + { timeout: transactTimeoutBudget(plan.operations.length, options?.timeoutMs) } + ) } })) } catch (err) { diff --git a/src/db/types.ts b/src/db/types.ts index 2bfd6ac2..d1355dab 100644 --- a/src/db/types.ts +++ b/src/db/types.ts @@ -116,6 +116,16 @@ export interface TransactOptions { * record is staged. */ ifAtGeneration?: number + /** + * Budget (ms) for the atomic apply phase. When omitted, the budget SCALES + * with the batch: `max(30 000, opCount × 2 000)` — production imports on + * network-attached disks measure ~2 s per operation, so a flat 30 s budget + * silently capped honest bulk work at ~15 operations. A tripped budget + * rolls the whole batch back and throws a retryable + * `TransactionTimeoutError` naming the operation it stopped at, the batch + * size, and the elapsed/budget times. + */ + timeoutMs?: number } /** diff --git a/src/transaction/Transaction.ts b/src/transaction/Transaction.ts index 75dccc05..092a2a25 100644 --- a/src/transaction/Transaction.ts +++ b/src/transaction/Transaction.ts @@ -37,6 +37,24 @@ const DEFAULT_OPTIONS: Required = { maxRollbackRetries: 3 } +/** + * The apply-phase budget for a batch of `opCount` operations. + * + * An explicit override wins untouched. Otherwise the budget SCALES with the + * batch: `max(30 000 ms, opCount × 2 000 ms)`. The per-op term is calibrated + * from field data — bulk imports on network-attached disks measure ~2 s per + * operation (each op pays canonical writes + fsync + index maintenance) — so + * a flat 30 s budget silently capped honest work at ~15 operations while + * looking generous for small batches. Scaling keeps small transacts + * fast-failing and gives bulk ones a budget proportional to the work they + * actually asked for; a trip still rolls back atomically and throws a + * retryable, fully-labeled TransactionTimeoutError. + */ +export function transactTimeoutBudget(opCount: number, override?: number): number { + if (override !== undefined) return override + return Math.max(30_000, opCount * 2_000) +} + /** * Transaction class */ @@ -114,7 +132,11 @@ export class Transaction implements TransactionContext { // into the catch below and rolls back like any other failure — it must // never bypass rollback. if (Date.now() - this.startTime > this.options.timeout) { - throw new TransactionTimeoutError(this.options.timeout, i) + throw new TransactionTimeoutError(this.options.timeout, i, { + elapsedMs: Date.now() - this.startTime, + totalOperations: this.operations.length, + operationName: this.operations[i]?.name + }) } const operation = this.operations[i] diff --git a/src/transaction/errors.ts b/src/transaction/errors.ts index c9f6eee1..c270d0ed 100644 --- a/src/transaction/errors.ts +++ b/src/transaction/errors.ts @@ -77,11 +77,24 @@ export class InvalidTransactionStateError extends TransactionError { export class TransactionTimeoutError extends TransactionError { constructor( timeoutMs: number, - operationIndex: number + operationIndex: number, + telemetry?: { + elapsedMs?: number + totalOperations?: number + operationName?: string + } ) { + const progress = + telemetry?.totalOperations !== undefined + ? `${operationIndex}/${telemetry.totalOperations}` + : String(operationIndex) + const name = telemetry?.operationName ? ` ('${telemetry.operationName}')` : '' + const elapsed = + telemetry?.elapsedMs !== undefined ? `${telemetry.elapsedMs}ms elapsed, ` : '' super( - `Transaction timed out after ${timeoutMs}ms at operation ${operationIndex}`, - { timeoutMs, operationIndex } + `Transaction timed out at operation ${progress}${name} — ${elapsed}budget ${timeoutMs}ms. ` + + `The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch.`, + { timeoutMs, operationIndex, ...telemetry } ) this.name = 'TransactionTimeoutError' } diff --git a/tests/transaction/TransactionManager.unit.test.ts b/tests/transaction/TransactionManager.unit.test.ts index cc477483..86b1692c 100644 --- a/tests/transaction/TransactionManager.unit.test.ts +++ b/tests/transaction/TransactionManager.unit.test.ts @@ -10,6 +10,7 @@ import { describe, it, expect, beforeEach } from 'vitest' import { TransactionManager } from '../../src/transaction/TransactionManager.js' +import { transactTimeoutBudget } from '../../src/transaction/Transaction.js' import { TransactionError } from '../../src/transaction/errors.js' describe('TransactionManager', () => { @@ -328,4 +329,45 @@ describe('TransactionManager', () => { expect(stats1).toEqual(stats2) // Same values }) }) + + describe('Timeout budget + telemetry', () => { + it('transactTimeoutBudget: explicit override wins; default scales with batch size', () => { + expect(transactTimeoutBudget(1)).toBe(30_000) // small batches keep the 30s floor + expect(transactTimeoutBudget(15)).toBe(30_000) // the old flat cap's break-even point + expect(transactTimeoutBudget(100)).toBe(200_000) // 100 ops × 2s — bulk gets an honest budget + expect(transactTimeoutBudget(1000, 5_000)).toBe(5_000) // caller override is untouched + }) + + it('a tripped budget rolls back and names the operation, progress, and budget', async () => { + const rolledBack: string[] = [] + + const failing = manager.executeTransaction( + async (tx) => { + tx.addOperation({ + name: 'slow-first-op', + execute: async () => { + await new Promise((r) => setTimeout(r, 30)) + return async () => { + rolledBack.push('slow-first-op') + } + } + }) + tx.addOperation({ + name: 'never-reached', + execute: async () => undefined + }) + }, + { timeout: 5 } // the first op's 30ms sleep guarantees the pre-op-2 check trips + ) + + await expect(failing).rejects.toThrow(TransactionError) + const err = await failing.catch((e) => e) + expect(err.name).toBe('TransactionTimeoutError') + expect(err.message).toContain('operation 1/2') // which op, of how many + expect(err.message).toContain("('never-reached')") // its name + expect(err.message).toContain('budget 5ms') // the budget that tripped + expect(err.message).toContain('rolled back') // the retryability statement + expect(rolledBack).toEqual(['slow-first-op']) // the applied op was undone + }) + }) }) From e450e0eedfd837e693c7ad74166bdd451638a04e Mon Sep 17 00:00:00 2001 From: David Snelling Date: Fri, 17 Jul 2026 16:54:40 -0700 Subject: [PATCH 04/29] chore(release): 8.7.0 --- CHANGELOG.md | 5 +++++ package-lock.json | 4 ++-- package.json | 2 +- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e89af335..4d7198e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. +### [8.7.0](https://github.com/soulcraftlabs/brainy/compare/v8.6.0...v8.7.0) (2026-07-17) + +- feat: scaled transact budgets + labeled timeout diagnostics + envelope docs (6ef9fcb) + + ### [8.6.0](https://github.com/soulcraftlabs/brainy/compare/v8.5.2...v8.6.0) (2026-07-17) - feat: brain.auditGraph() — read-only graph-truth audit (2a03fae) diff --git a/package-lock.json b/package-lock.json index 71cba0fe..aa9a62a3 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraft/brainy", - "version": "8.6.0", + "version": "8.7.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraft/brainy", - "version": "8.6.0", + "version": "8.7.0", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index 474e41dd..9d016ecb 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraft/brainy", - "version": "8.6.0", + "version": "8.7.0", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", "module": "dist/index.js", From 01a3b46ade9d6b21931ebc398e6b24913af3f870 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Fri, 17 Jul 2026 17:11:46 -0700 Subject: [PATCH 05/29] fix: race-proof writer-lock acquisition + machine-readable conflict through init - acquireWriterLock now CLAIMS with an atomic create-exclusive write (O_EXCL) inside a bounded retry loop: two processes racing an absent lock can never both succeed (the old read-then-tmp-rename flow let the loser keep running unlocked, silently). An EEXIST loser re-evaluates and either throws loudly with the winner's details or performs a verified stale-takeover (re-read before unlink so a lock that changed hands mid-deliberation is never clobbered). Exhausted contention fails loudly instead of degrading into a lockless open. - BRAINY_WRITER_LOCKED passes through init() unwrapped: the error documents a machine-readable contract (err.code + err.lockInfo with the holder's pid/host/heartbeat), but init's blanket error wrapping stripped both, leaving consumers a message to regex against. - Two contract tests added: stale-foreign takeover installs OUR lock via the atomic claim; the conflict error carries code + lockInfo at the public init() surface. --- RELEASES.md | 20 +++ src/brainy.ts | 7 + src/storage/adapters/fileSystemStorage.ts | 167 +++++++++++++----- .../integration/multi-process-safety.test.ts | 43 +++++ 4 files changed, 188 insertions(+), 49 deletions(-) diff --git a/RELEASES.md b/RELEASES.md index 94ce16ac..a07d9326 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -10,6 +10,26 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so --- +## v8.7.1 — 2026-07-17 (writer-lock acquisition is race-proof + machine-readable through init) + +Two hardenings of the multi-process writer lock (the `locks/_writer.lock` lease that makes a +second writer on the same brain directory fail loudly): + +- **Lock acquisition claims atomically.** The acquire path used read-then-write, leaving a + window where two processes racing an *absent* lock could both "succeed" — and the loser + kept running unlocked, silently. The claim is now an atomic create-exclusive write + (`O_EXCL`): exactly one racer wins; the loser re-evaluates and either fails loudly with + the winner's details or performs a verified stale-takeover. Bounded retries; contention + beyond them fails loudly rather than degrading into a lockless open. +- **`BRAINY_WRITER_LOCKED` survives `init()`.** The conflict error documents a + machine-readable contract (`err.code`, `err.lockInfo` with the holder's pid/host/ + heartbeat), but init's error wrapping silently stripped both, leaving consumers a message + to regex against. The error now passes through unwrapped. + +Measured while verifying (for operators sizing audits): `brain.auditGraph()` at a +production-consumer scale of ~2,600 relationships / 800 entities costs ~0.1 s warm and +~0.5 s cold, with exact scar counting across reopen. + ## v8.7.0 — 2026-07-17 (bulk-transact ergonomics: scaled budgets + timeout telemetry) The bulk-import ergonomics release, from a consumer's measured production incident (a serial diff --git a/src/brainy.ts b/src/brainy.ts index 25c91d0f..7aa7e6ca 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -1373,6 +1373,13 @@ export class Brainy implements BrainyInterface { if (this._readyReject) { this._readyReject(error instanceof Error ? error : new Error(String(error))) } + // Machine-readable init failures pass through UNWRAPPED — the writer-lock + // conflict documents an err.code/err.lockInfo contract ("callers detect + // this case via err.code"), and wrapping in a fresh Error silently + // stripped both, leaving consumers only a message to regex against. + if (error instanceof Error && (error as Error & { code?: string }).code === 'BRAINY_WRITER_LOCKED') { + throw error + } throw new Error(`Failed to initialize Brainy: ${error}`) } } diff --git a/src/storage/adapters/fileSystemStorage.ts b/src/storage/adapters/fileSystemStorage.ts index 4f246f67..3c95f9e7 100644 --- a/src/storage/adapters/fileSystemStorage.ts +++ b/src/storage/adapters/fileSystemStorage.ts @@ -1764,64 +1764,117 @@ export class FileSystemStorage extends BaseStorage { const lockFile = path.join(this.lockDir, FileSystemStorage.WRITER_LOCK_FILE) const os = await import('node:os') const hostname = os.hostname() - const now = new Date().toISOString() + const myPid = typeof process !== 'undefined' && process.pid ? process.pid : 0 - const existing = await this.readWriterLock() - if (existing && !options?.force) { - // Same-process re-open: a second Brainy instance in this Node process - // (e.g. test "simulate server restart" patterns, or a consumer that - // explicitly re-instantiates without closing first). This isn't the - // dangerous cross-process case the lock exists to prevent — the two - // instances share a memory space and can't silently diverge from each - // other beyond what their callers already see. Warn and take over the lock. - if (existing.pid === (typeof process !== 'undefined' ? process.pid : 0) && - existing.hostname === hostname) { - console.warn( - `[brainy] Re-acquiring writer lock for ${this.rootDir} held by the same process (PID ${existing.pid}). ` + - `If you intended to keep the previous Brainy instance alive, this is a bug — close it first.` - ) - } else { - const stale = await this.isWriterLockStale(existing) - if (!stale) { + // Bounded acquire loop. The CLAIM itself is an atomic create-exclusive + // write (O_EXCL) — two processes racing an ABSENT lock can never both + // succeed, which closes the read-then-write window where the loser used + // to keep running unlocked, silently. An EEXIST loser loops, re-reads, + // and handles whatever it finds honestly (fresh foreign lock → loud + // throw; stale/forced → verified takeover). + const MAX_ATTEMPTS = 3 + for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) { + const now = new Date().toISOString() + const existing = await this.readWriterLock() + + if (existing) { + // Same-process re-open: a second Brainy instance in this Node process + // (e.g. test "simulate server restart" patterns, or a consumer that + // explicitly re-instantiates without closing first). This isn't the + // dangerous cross-process case the lock exists to prevent — the two + // instances share a memory space and can't silently diverge from each + // other beyond what their callers already see. Warn and take over. + if (existing.pid === myPid && existing.hostname === hostname && !options?.force) { + console.warn( + `[brainy] Re-acquiring writer lock for ${this.rootDir} held by the same process (PID ${existing.pid}). ` + + `If you intended to keep the previous Brainy instance alive, this is a bug — close it first.` + ) + const info: WriterLockInfo = { + pid: myPid, + hostname, + startedAt: now, + lastHeartbeat: now, + version: getBrainyVersion(), + rootDir: this.rootDir + } + await this.writeFileAtomic(lockFile, JSON.stringify(info, null, 2)) + this.installWriterLock(info) + return info + } + + const stale = !options?.force && (await this.isWriterLockStale(existing)) + if (!options?.force && !stale) { // Consumer-facing error contract: callers detect this case via // err.code and read the holder's details from err.lockInfo. - const err = new Error( - `Another writer holds this Brainy directory.\n` + - ` PID: ${existing.pid} on host ${existing.hostname}\n` + - ` Started: ${existing.startedAt}\n` + - ` Heartbeat: ${existing.lastHeartbeat}\n` + - ` Version: ${existing.version}\n` + - ` Directory: ${this.rootDir}\n\n` + - `For diagnostic queries against this live store, use:\n` + - ` const reader = await Brainy.openReadOnly({ storage: { type: 'filesystem', path: '${this.rootDir}' } })\n\n` + - `If you have verified the existing lock is stale (e.g. a crashed writer on a different host that PID liveness cannot reach), pass { force: true }.` - ) as Error & { code: string; lockInfo: WriterLockInfo } - err.code = 'BRAINY_WRITER_LOCKED' - err.lockInfo = existing - throw err + throw this.writerLockedError(existing) } + console.warn( - `[brainy] Overwriting stale writer lock for ${this.rootDir} ` + - `(PID ${existing.pid} on ${existing.hostname} appears dead).` + options?.force + ? `[brainy] Force-overwriting writer lock for ${this.rootDir} ` + + `(was held by PID ${existing.pid} on ${existing.hostname}).` + : `[brainy] Overwriting stale writer lock for ${this.rootDir} ` + + `(PID ${existing.pid} on ${existing.hostname} appears dead).` ) + // Takeover: verify the file still holds the lock we judged (a live + // successor may have claimed meanwhile), then remove it and fall + // through to the atomic claim below. A racing claimer who beats us to + // the create simply wins — our next loop iteration reads their fresh + // lock and throws honestly. (Advisory file locking has no + // compare-and-delete; staleness requiring a 60s-old heartbeat keeps + // the residual verify-to-unlink window practically unreachable.) + const recheck = await this.readWriterLock() + if ( + recheck && + (recheck.pid !== existing.pid || + recheck.startedAt !== existing.startedAt || + recheck.lastHeartbeat !== existing.lastHeartbeat) + ) { + continue // the lock changed hands while we deliberated — re-evaluate + } + try { + await fs.promises.unlink(lockFile) + } catch (err: any) { + if (err.code !== 'ENOENT') throw err + } } - } else if (existing && options?.force) { - console.warn( - `[brainy] Force-overwriting writer lock for ${this.rootDir} ` + - `(was held by PID ${existing.pid} on ${existing.hostname}).` - ) + + const info: WriterLockInfo = { + pid: myPid, + hostname, + startedAt: existing && options?.force ? existing.startedAt : now, + lastHeartbeat: now, + version: getBrainyVersion(), + rootDir: this.rootDir + } + + // The atomic claim: create-exclusive, so exactly ONE racer wins. + try { + await fs.promises.writeFile(lockFile, JSON.stringify(info, null, 2), { flag: 'wx' }) + } catch (err: any) { + if (err.code === 'EEXIST') { + continue // someone else claimed between our read and create — re-evaluate + } + throw err + } + + this.installWriterLock(info) + return info } - const info: WriterLockInfo = { - pid: typeof process !== 'undefined' && process.pid ? process.pid : 0, - hostname, - startedAt: existing && options?.force ? existing.startedAt : now, - lastHeartbeat: now, - version: getBrainyVersion(), - rootDir: this.rootDir - } + // Attempts exhausted: something is claiming this directory faster than we + // can evaluate it. Read whoever holds it now and fail loudly with their + // details rather than degrading into a lockless open. + const holder = await this.readWriterLock() + if (holder) throw this.writerLockedError(holder) + throw new Error( + `Failed to acquire the writer lock for ${this.rootDir} after ${MAX_ATTEMPTS} attempts — ` + + `the lock file is being contended. Retry, or inspect ${lockFile}.` + ) + } - await this.writeFileAtomic(lockFile, JSON.stringify(info, null, 2)) + /** Record lock ownership + start the unref'd heartbeat. */ + private installWriterLock(info: WriterLockInfo): void { this.writerLockInfo = info // Heartbeat — rewrite lastHeartbeat every WRITER_HEARTBEAT_MS so other @@ -1835,8 +1888,24 @@ export class FileSystemStorage extends BaseStorage { // Don't keep the event loop alive just for the heartbeat. this.writerLockHeartbeat.unref() } + } - return info + /** The consumer-facing BRAINY_WRITER_LOCKED error, holder details attached. */ + private writerLockedError(existing: WriterLockInfo): Error { + const err = new Error( + `Another writer holds this Brainy directory.\n` + + ` PID: ${existing.pid} on host ${existing.hostname}\n` + + ` Started: ${existing.startedAt}\n` + + ` Heartbeat: ${existing.lastHeartbeat}\n` + + ` Version: ${existing.version}\n` + + ` Directory: ${this.rootDir}\n\n` + + `For diagnostic queries against this live store, use:\n` + + ` const reader = await Brainy.openReadOnly({ storage: { type: 'filesystem', path: '${this.rootDir}' } })\n\n` + + `If you have verified the existing lock is stale (e.g. a crashed writer on a different host that PID liveness cannot reach), pass { force: true }.` + ) as Error & { code: string; lockInfo: WriterLockInfo } + err.code = 'BRAINY_WRITER_LOCKED' + err.lockInfo = existing + return err } public override async releaseWriterLock(): Promise { diff --git a/tests/integration/multi-process-safety.test.ts b/tests/integration/multi-process-safety.test.ts index 6ac5265d..0eae92b1 100644 --- a/tests/integration/multi-process-safety.test.ts +++ b/tests/integration/multi-process-safety.test.ts @@ -110,6 +110,49 @@ describe('Multi-process safety + read-only mode', () => { // Don't track `blocked` for afterEach cleanup since init failed. }) + it('takes over a STALE foreign lock (dead PID + old heartbeat) and claims atomically', async () => { + const { mkdirSync, writeFileSync, readFileSync } = await import('node:fs') + const { join } = await import('node:path') + const os = await import('node:os') + mkdirSync(join(dir, 'locks'), { recursive: true }) + const tenMinutesAgo = new Date(Date.now() - 10 * 60 * 1000).toISOString() + writeFileSync(join(dir, 'locks', '_writer.lock'), JSON.stringify({ + pid: 999999999, // no such process — provably dead + hostname: os.hostname(), + startedAt: tenMinutesAgo, + lastHeartbeat: tenMinutesAgo, + version: '8.0.0', + rootDir: dir + })) + + writer = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } }) + await writer.init() // stale takeover must succeed + + const lock = JSON.parse(readFileSync(join(dir, 'locks', '_writer.lock'), 'utf-8')) + expect(lock.pid).toBe(process.pid) // the atomic wx claim installed OUR lock + }) + + it('the writer-locked error carries the machine-readable contract (code + lockInfo)', async () => { + const { mkdirSync, writeFileSync } = await import('node:fs') + const { join } = await import('node:path') + const os = await import('node:os') + mkdirSync(join(dir, 'locks'), { recursive: true }) + const otherPid = (process as any).ppid || 1 + writeFileSync(join(dir, 'locks', '_writer.lock'), JSON.stringify({ + pid: otherPid, + hostname: os.hostname(), + startedAt: new Date().toISOString(), + lastHeartbeat: new Date().toISOString(), + version: '8.7.0', + rootDir: dir + })) + + const blocked = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } }) + const err: any = await blocked.init().catch((e) => e) + expect(err.code).toBe('BRAINY_WRITER_LOCKED') + expect(err.lockInfo?.pid).toBe(otherPid) + }) + it('allows a second in-process writer with a warning (same PID)', async () => { // Two Brainy instances in the same Node process: not the dangerous // cross-process case. Should succeed (with a console warning). From dcd5036fe93edbed5f46abee3abdd2f8af2e7918 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Fri, 17 Jul 2026 17:15:13 -0700 Subject: [PATCH 06/29] chore(release): 8.7.1 --- CHANGELOG.md | 5 +++++ package-lock.json | 4 ++-- package.json | 2 +- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d7198e5..95103bb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. +### [8.7.1](https://github.com/soulcraftlabs/brainy/compare/v8.7.0...v8.7.1) (2026-07-17) + +- fix: race-proof writer-lock acquisition + machine-readable conflict through init (01a3b46) + + ### [8.7.0](https://github.com/soulcraftlabs/brainy/compare/v8.6.0...v8.7.0) (2026-07-17) - feat: scaled transact budgets + labeled timeout diagnostics + envelope docs (6ef9fcb) diff --git a/package-lock.json b/package-lock.json index aa9a62a3..28b5e736 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraft/brainy", - "version": "8.7.0", + "version": "8.7.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraft/brainy", - "version": "8.7.0", + "version": "8.7.1", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index 9d016ecb..d7c19216 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraft/brainy", - "version": "8.7.0", + "version": "8.7.1", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", "module": "dist/index.js", From 16a73b8475c359c47fbb498f77ca832ac17de612 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Fri, 17 Jul 2026 17:51:54 -0700 Subject: [PATCH 07/29] feat: OS-limit detection for pool-scale deployments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - New src/utils/osLimits.ts: reads RLIMIT_NOFILE (soft/hard, from /proc/self/limits) and vm.max_map_count at open — once per process, Linux-only, measurement-only — and warns loudly when either sits below the pool-scale floors (soft NOFILE < 65536, max_map_count < 262144), with the exact raise commands. On stock defaults the failure otherwise arrives as EMFILE or a failed mmap deep inside an index open, long after the cause stopped being visible. An unreadable limit produces NO warning — no measurement, no claim — so non-Linux platforms stay silent. - Exported for ops doors: checkOsLimits() returns the full OsLimitsReport; floors exported as constants. Wired fire-and-forget in performInit after storage init; the check can never affect open. - Unit tests pin the parser (incl. 'unlimited'), the floor thresholds, the null-never-warns rule, and the off-Linux silent path. --- RELEASES.md | 15 ++++ src/brainy.ts | 8 ++ src/index.ts | 6 ++ src/utils/osLimits.ts | 141 ++++++++++++++++++++++++++++++ tests/unit/utils/osLimits.test.ts | 76 ++++++++++++++++ 5 files changed, 246 insertions(+) create mode 100644 src/utils/osLimits.ts create mode 100644 tests/unit/utils/osLimits.test.ts diff --git a/RELEASES.md b/RELEASES.md index a07d9326..7baa89ec 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -10,6 +10,21 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so --- +## v8.8.0 — 2026-07-17 (OS-limit detection for pool-scale deployments) + +Small minor: brains now detect the two OS limits that bite at pool scale and warn **before** +the incident instead of during it. + +- At open (once per process, Linux-only, measurement-only), Brainy reads `RLIMIT_NOFILE` + (soft/hard, from `/proc/self/limits`) and `vm.max_map_count`, and warns loudly when either + sits below the pool-scale floors (soft NOFILE < 65 536; max_map_count < 262 144) — with the + exact raise commands (`ulimit -n` / `LimitNOFILE=` / `sysctl vm.max_map_count`). On stock + defaults the failure otherwise arrives as `EMFILE` or a failed mmap deep inside an index + open, long after the real cause stopped being visible. An unreadable limit produces **no** + warning — no measurement, no claim (non-Linux platforms stay silent). +- Exported for ops doors: `checkOsLimits()` returns the full `OsLimitsReport` + (values + warnings) programmatically, with the floors exported as constants. + ## v8.7.1 — 2026-07-17 (writer-lock acquisition is race-proof + machine-readable through init) Two hardenings of the multi-process writer lock (the `locks/_writer.lock` lease that makes a diff --git a/src/brainy.ts b/src/brainy.ts index 7aa7e6ca..89e1267a 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -49,6 +49,7 @@ import { import { runGraphAudit, type GraphAuditReport } from './graph/graphAudit.js' import { createPipeline } from './streaming/pipeline.js' import { configureLogger, LogLevel, prodLog } from './utils/logger.js' +import { warnOnLowOsLimits } from './utils/osLimits.js' import { setGlobalCache } from './utils/unifiedCache.js' import type { UnifiedCache } from './utils/unifiedCache.js' import { rankIndicesByScore, reorderByIndices } from './utils/resultRanking.js' @@ -935,6 +936,13 @@ export class Brainy implements BrainyInterface { this.storage = await this.setupStorage() await this.storage.init() + // OS-limit detection (once per process, Linux-only, measurement-only): + // warn NOW about RLIMIT_NOFILE / vm.max_map_count values that will bite + // at pool scale, instead of letting the operator meet them as EMFILE or + // a failed mmap deep inside an index open. Fire-and-forget — the check + // never affects open. + void warnOnLowOsLimits() + // Acquire the writer lock for filesystem (and other locking-capable) backends. // Skipped in reader mode and on backends that don't support multi-process locking. // Throws if another live writer holds the directory (unless force: true). diff --git a/src/index.ts b/src/index.ts index d1eab40a..5c57fe29 100644 --- a/src/index.ts +++ b/src/index.ts @@ -32,6 +32,12 @@ export type { GraphAuditReport, GraphAuditDiscrepancy } from './graph/graphAudit.js' +export { + checkOsLimits, + NOFILE_POOL_FLOOR, + MAX_MAP_COUNT_POOL_FLOOR +} from './utils/osLimits.js' +export type { OsLimitsReport } from './utils/osLimits.js' // Export Brainy configuration and types export type { diff --git a/src/utils/osLimits.ts b/src/utils/osLimits.ts new file mode 100644 index 00000000..57e1ccf1 --- /dev/null +++ b/src/utils/osLimits.ts @@ -0,0 +1,141 @@ +/** + * @module utils/osLimits + * @description Detect-and-warn for OS resource limits that bite at POOL scale. + * + * A single brain rarely notices them, but a pool of brains — especially with a + * native accelerator memory-mapping many index files per brain — consumes file + * descriptors and memory mappings multiplicatively. On stock Linux defaults + * (RLIMIT_NOFILE soft 1024, vm.max_map_count 65530) the failure arrives as + * EMFILE or a failed mmap deep inside an index open, long after the real cause + * (the limit) stopped being visible. This module reads the limits at open and + * WARNS ONCE per process with the exact raise commands, so the operator learns + * the fix before the incident instead of from it. + * + * Read-only and Linux-only by construction: both sources are `/proc` files. + * On platforms where they are absent the check reports nulls and stays silent — + * no limit read means no claim made, never a guessed warning. + */ + +import * as fs from 'node:fs' +import { prodLog } from './logger.js' + +/** Soft-NOFILE floor below which pool-scale use is at EMFILE risk. */ +export const NOFILE_POOL_FLOOR = 65536 + +/** vm.max_map_count floor below which mmap-heavy native indexes are at risk. */ +export const MAX_MAP_COUNT_POOL_FLOOR = 262144 + +export interface OsLimitsReport { + /** RLIMIT_NOFILE soft limit (null when unreadable; Infinity for 'unlimited'). */ + nofileSoft: number | null + /** RLIMIT_NOFILE hard limit (null when unreadable; Infinity for 'unlimited'). */ + nofileHard: number | null + /** vm.max_map_count (null when unreadable). */ + maxMapCount: number | null + /** Human-actionable warnings for limits below the pool floors. Empty = fine. */ + warnings: string[] +} + +/** + * Parse the `Max open files` row of a `/proc//limits` document into + * soft/hard values. Returns nulls when the row is absent or malformed. + */ +export function parseProcLimits(content: string): { soft: number | null; hard: number | null } { + const line = content.split('\n').find((l) => l.startsWith('Max open files')) + if (!line) return { soft: null, hard: null } + const m = line.match(/^Max open files\s+(\S+)\s+(\S+)/) + if (!m) return { soft: null, hard: null } + const parse = (v: string): number | null => { + if (v === 'unlimited') return Infinity + const n = Number.parseInt(v, 10) + return Number.isNaN(n) ? null : n + } + return { soft: parse(m[1]), hard: parse(m[2]) } +} + +/** + * Assess readable limits against the pool floors. Pure — feed it any values. + * A null (unreadable) limit produces NO warning: no measurement, no claim. + */ +export function assessOsLimits(limits: { + nofileSoft: number | null + nofileHard: number | null + maxMapCount: number | null +}): string[] { + const warnings: string[] = [] + + if (limits.nofileSoft !== null && limits.nofileSoft < NOFILE_POOL_FLOOR) { + const hardNote = + limits.nofileHard !== null && limits.nofileHard >= NOFILE_POOL_FLOOR + ? ` (the hard limit ${limits.nofileHard === Infinity ? 'unlimited' : limits.nofileHard} already allows it — raise the soft limit only)` + : '' + warnings.push( + `RLIMIT_NOFILE soft limit is ${limits.nofileSoft} — below the ${NOFILE_POOL_FLOOR} recommended ` + + `for pool-scale use (a pool of brains with a native accelerator opens many index files per brain; ` + + `the failure mode is EMFILE deep inside an index open). Raise with \`ulimit -n ${NOFILE_POOL_FLOOR}\` ` + + `or LimitNOFILE=${NOFILE_POOL_FLOOR} in the service unit${hardNote}.` + ) + } + + if (limits.maxMapCount !== null && limits.maxMapCount < MAX_MAP_COUNT_POOL_FLOOR) { + warnings.push( + `vm.max_map_count is ${limits.maxMapCount} — below the ${MAX_MAP_COUNT_POOL_FLOOR} recommended ` + + `for mmap-heavy native indexes at pool scale (each mapped index segment consumes map entries; ` + + `the failure mode is a failed mmap mid-heal). Raise with ` + + `\`sysctl -w vm.max_map_count=${MAX_MAP_COUNT_POOL_FLOOR}\` (persist in /etc/sysctl.d/).` + ) + } + + return warnings +} + +/** + * Read the limits from /proc and assess them. `readFile` is injectable for + * tests; absent/unreadable sources yield nulls (and therefore no warnings). + */ +export async function checkOsLimits( + readFile: (path: string) => Promise = async (p) => fs.promises.readFile(p, 'utf-8') +): Promise { + let nofileSoft: number | null = null + let nofileHard: number | null = null + let maxMapCount: number | null = null + + try { + const parsed = parseProcLimits(await readFile('/proc/self/limits')) + nofileSoft = parsed.soft + nofileHard = parsed.hard + } catch { + // Not Linux (or /proc unavailable) — no measurement, no claim. + } + + try { + const raw = (await readFile('/proc/sys/vm/max_map_count')).trim() + const n = Number.parseInt(raw, 10) + maxMapCount = Number.isNaN(n) ? null : n + } catch { + // Not Linux — same rule. + } + + const warnings = assessOsLimits({ nofileSoft, nofileHard, maxMapCount }) + return { nofileSoft, nofileHard, maxMapCount, warnings } +} + +/** Once-per-process latch so a brain pool warns once, not once per brain. */ +let osLimitsWarned = false + +/** + * Run the check and warn (once per process) about limits below the pool + * floors. Called from brain open; safe everywhere (silent off-Linux). + */ +export async function warnOnLowOsLimits(): Promise { + if (osLimitsWarned) return + osLimitsWarned = true + try { + const report = await checkOsLimits() + for (const warning of report.warnings) { + prodLog.warn(`[Brainy] OS limit check: ${warning}`) + } + } catch { + // The check must never affect open — measurement-only. + } +} diff --git a/tests/unit/utils/osLimits.test.ts b/tests/unit/utils/osLimits.test.ts new file mode 100644 index 00000000..a56d593e --- /dev/null +++ b/tests/unit/utils/osLimits.test.ts @@ -0,0 +1,76 @@ +/** + * @module tests/unit/utils/osLimits + * @description OS-limit detection for pool-scale use. Laws: + * (1) the /proc/self/limits parser reads soft/hard NOFILE exactly, including + * 'unlimited'; (2) assessment warns ONLY below the pool floors and NEVER + * on an unreadable (null) limit — no measurement, no claim; (3) the full + * check composes both sources and survives unreadable /proc silently. + */ +import { describe, it, expect } from 'vitest' +import { + parseProcLimits, + assessOsLimits, + checkOsLimits, + NOFILE_POOL_FLOOR, + MAX_MAP_COUNT_POOL_FLOOR +} from '../../../src/utils/osLimits.js' + +const SAMPLE_LIMITS = [ + 'Limit Soft Limit Hard Limit Units', + 'Max cpu time unlimited unlimited seconds', + 'Max open files 1024 1048576 files', + 'Max locked memory 8388608 8388608 bytes' +].join('\n') + +describe('osLimits — detect + warn at pool scale', () => { + it('parses soft/hard NOFILE from /proc/self/limits, including unlimited', () => { + expect(parseProcLimits(SAMPLE_LIMITS)).toEqual({ soft: 1024, hard: 1048576 }) + expect( + parseProcLimits('Max open files unlimited unlimited files') + ).toEqual({ soft: Infinity, hard: Infinity }) + expect(parseProcLimits('no such row here')).toEqual({ soft: null, hard: null }) + }) + + it('warns below the floors, stays quiet at or above them', () => { + const low = assessOsLimits({ nofileSoft: 1024, nofileHard: 1048576, maxMapCount: 65530 }) + expect(low).toHaveLength(2) + expect(low[0]).toContain('RLIMIT_NOFILE soft limit is 1024') + expect(low[0]).toContain(`ulimit -n ${NOFILE_POOL_FLOOR}`) + expect(low[0]).toContain('raise the soft limit only') // hard already allows it + expect(low[1]).toContain('vm.max_map_count is 65530') + expect(low[1]).toContain(`vm.max_map_count=${MAX_MAP_COUNT_POOL_FLOOR}`) + + expect( + assessOsLimits({ + nofileSoft: NOFILE_POOL_FLOOR, + nofileHard: Infinity, + maxMapCount: MAX_MAP_COUNT_POOL_FLOOR + }) + ).toEqual([]) + }) + + it('an unreadable limit makes NO claim — nulls never warn', () => { + expect(assessOsLimits({ nofileSoft: null, nofileHard: null, maxMapCount: null })).toEqual([]) + }) + + it('checkOsLimits composes both sources and survives unreadable /proc silently', async () => { + const report = await checkOsLimits(async (p) => { + if (p === '/proc/self/limits') return SAMPLE_LIMITS + if (p === '/proc/sys/vm/max_map_count') return '65530\n' + throw new Error('unexpected path') + }) + expect(report.nofileSoft).toBe(1024) + expect(report.maxMapCount).toBe(65530) + expect(report.warnings).toHaveLength(2) + + const offLinux = await checkOsLimits(async () => { + throw Object.assign(new Error('ENOENT'), { code: 'ENOENT' }) + }) + expect(offLinux).toEqual({ + nofileSoft: null, + nofileHard: null, + maxMapCount: null, + warnings: [] + }) + }) +}) From 120205b69c444b3bf79e8879819f67a45c2798e5 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Fri, 17 Jul 2026 18:41:27 -0700 Subject: [PATCH 08/29] chore(release): 8.8.0 --- CHANGELOG.md | 5 +++++ package-lock.json | 4 ++-- package.json | 2 +- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 95103bb6..f97046f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. +### [8.8.0](https://github.com/soulcraftlabs/brainy/compare/v8.7.1...v8.8.0) (2026-07-17) + +- feat: OS-limit detection for pool-scale deployments (16a73b8) + + ### [8.7.1](https://github.com/soulcraftlabs/brainy/compare/v8.7.0...v8.7.1) (2026-07-17) - fix: race-proof writer-lock acquisition + machine-readable conflict through init (01a3b46) diff --git a/package-lock.json b/package-lock.json index 28b5e736..198e9116 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraft/brainy", - "version": "8.7.1", + "version": "8.8.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraft/brainy", - "version": "8.7.1", + "version": "8.8.0", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index d7c19216..7c3e050f 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraft/brainy", - "version": "8.7.1", + "version": "8.8.0", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", "module": "dist/index.js", From 4fcef7b8ed187c579b9608f06e488449a3830c93 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sat, 18 Jul 2026 10:40:45 -0700 Subject: [PATCH 09/29] fix: import dedup off-switch honesty + brain-owned lifecycle for the background pass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The post-import background deduplication pass (a merge-DELETE writer, debounced ~5 minutes after import) had four lifecycle defects: - enableDeduplication: false did not gate the background schedule — an import that explicitly opted out could still have entities auto-removed minutes later. The flag now gates both the inline and background passes. - Each import() constructed its own coordinator-owned deduplicator, so the debounce never spanned imports (N imports = N delete timers). The brain now owns a single lazy instance (getBackgroundDeduplicator). - close() never cancelled pending dedup; a delete pass could fire against a closed brain. close() now cancels it first. - The 5-minute timer held the process open (exit-hang class); now unref'd. Four regression tests pin the contract (background-dedup-lifecycle); import guides document that false disables both passes. --- RELEASES.md | 22 +++++ docs/guides/import-anything.md | 5 +- docs/guides/import-quick-reference.md | 13 ++- src/brainy.ts | 28 ++++++ src/import/BackgroundDeduplicator.ts | 13 ++- src/import/ImportCoordinator.ts | 23 +++-- .../background-dedup-lifecycle.test.ts | 85 +++++++++++++++++++ 7 files changed, 179 insertions(+), 10 deletions(-) create mode 100644 tests/integration/background-dedup-lifecycle.test.ts diff --git a/RELEASES.md b/RELEASES.md index 7baa89ec..4a62123a 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -10,6 +10,28 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so --- +## v8.8.1 — 2026-07-18 (the import dedup off-switch is now honest + lifecycle-safe) + +The post-import background deduplication pass (a merge-DELETE writer that runs ~5 minutes +after an import, merging entities judged duplicates by id / name / vector similarity) had +three lifecycle defects, all fixed: + +- **`enableDeduplication: false` now actually disables it.** The background pass was + scheduled unconditionally — an import that explicitly opted out could still have + entities auto-removed 5 minutes later. The flag now gates BOTH the inline merge and + the background pass (regression-pinned). +- **One deduplicator per brain, owned by the brain.** Each `import()` call constructed its + own coordinator + deduplicator, so the "debounced" timer never actually debounced across + imports (N imports = N delete timers). The brain now owns a single instance — the + debounce genuinely spans imports — and `close()` cancels pending work, so a delete pass + can never fire against a closed brain. +- **The 5-minute timer is unref'd** — a pending pass no longer holds the process open + (the exit-hang class; this timer had escaped the earlier sweep). + +Retention note for keep-everything deployments: with `enableDeduplication: false` on +import calls and `retention: 'all'` in config, no engine path removes records +automatically. + ## v8.8.0 — 2026-07-17 (OS-limit detection for pool-scale deployments) Small minor: brains now detect the two OS limits that bite at pool scale and warn **before** diff --git a/docs/guides/import-anything.md b/docs/guides/import-anything.md index e0cd94ed..b1bb15ef 100644 --- a/docs/guides/import-anything.md +++ b/docs/guides/import-anything.md @@ -300,7 +300,10 @@ await brain.import(data, { // Deduplication enableDeduplication: true, // Check for duplicate entities (default: true) deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85) - // Note: Auto-disabled for imports >100 entities + // Notes: false disables BOTH the inline merge and the background pass that + // runs ~5 min after the last import (merged duplicates are deleted). + // The inline pass auto-disables for imports >100 entities (O(n²) cost); + // the background pass still covers those unless the flag is false. // Performance chunkSize: 100, // Batch size for processing (default: varies by operation) diff --git a/docs/guides/import-quick-reference.md b/docs/guides/import-quick-reference.md index 5a850a86..7837d49e 100644 --- a/docs/guides/import-quick-reference.md +++ b/docs/guides/import-quick-reference.md @@ -86,11 +86,22 @@ await brain.import(file, { ```typescript await brain.import(file, { - enableDeduplication: true, // Check for duplicates (default: false) + enableDeduplication: true, // Check for duplicates (default: true) deduplicationThreshold: 0.85 // Similarity threshold (default: 0.85) }) ``` +Deduplication merges entities judged duplicates — the non-primary records are +**deleted**. Set `enableDeduplication: false` to disable it entirely: the flag +gates both the inline merge during import and the background pass that runs +about 5 minutes after the last import. + +```typescript +await brain.import(file, { + enableDeduplication: false // No merging, inline or background +}) +``` + ### Import Tracking Track and organize imports by project: diff --git a/src/brainy.ts b/src/brainy.ts index 89e1267a..c7b4c89a 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -10128,6 +10128,30 @@ export class Brainy implements BrainyInterface { return await coordinator.import(source as Buffer | string | object, options) } + /** Brain-owned background deduplicator (lazy; see getBackgroundDeduplicator). */ + private _backgroundDedup?: import('./import/BackgroundDeduplicator.js').BackgroundDeduplicator + + /** + * The single brain-owned BackgroundDeduplicator, lazily constructed. + * + * Ownership matters here: the post-import dedup timer must outlive the + * per-call ImportCoordinator but never the brain. One instance per brain + * restores the intended cross-import debounce (per-coordinator instances + * each armed their own timer, so the "debounce" never spanned imports) and + * gives close() a handle to cancel pending work — a delete pass must never + * fire against a closed brain. + * @internal + */ + async getBackgroundDeduplicator(): Promise< + import('./import/BackgroundDeduplicator.js').BackgroundDeduplicator + > { + if (!this._backgroundDedup) { + const { BackgroundDeduplicator } = await import('./import/BackgroundDeduplicator.js') + this._backgroundDedup = new BackgroundDeduplicator(this) + } + return this._backgroundDedup + } + /** * Virtual File System API - Knowledge Operating System * @@ -16070,6 +16094,10 @@ export class Brainy implements BrainyInterface { * This ensures deferred persistence mode data is saved */ async close(): Promise { + // Cancel any pending post-import background deduplication FIRST — it is a + // writer (merge-deletes), and no delete pass may start mid- or post-close. + this._backgroundDedup?.cancelPending() + // Change-feed teardown: no events are delivered for or after close(). this._changeFeed.close() diff --git a/src/import/BackgroundDeduplicator.ts b/src/import/BackgroundDeduplicator.ts index 53b28262..073fbd9e 100644 --- a/src/import/BackgroundDeduplicator.ts +++ b/src/import/BackgroundDeduplicator.ts @@ -41,6 +41,14 @@ export interface DeduplicationStats { * - Import-scoped deduplication (no cross-contamination) * - 3-tier strategy (ID → Name → Similarity) * - Uses existing indexes (EntityIdMapper, MetadataIndexManager, TypeAware HNSW) + * + * Lifecycle: ONE instance per brain, owned by Brainy (getBackgroundDeduplicator) + * so the debounce genuinely spans imports and brain.close() cancels pending + * work via cancelPending() — this pass merge-DELETES duplicate entities, so it + * must never fire against a closed brain. The enableDeduplication gate lives + * at the scheduling call site (ImportCoordinator); scheduleDedup itself is + * unconditional. The timer is unref'd — a pending pass never holds the + * process open. */ export class BackgroundDeduplicator { private brain: Brainy @@ -67,12 +75,15 @@ export class BackgroundDeduplicator { clearTimeout(this.debounceTimer) } - // Schedule for 5 minutes from now + // Schedule for 5 minutes from now. unref'd: a pending dedup pass must + // never hold the process open (exit-hang class) — if the process exits + // first, the pass simply never runs; imports are already durable. this.debounceTimer = setTimeout(() => { this.runBatchDedup().catch(error => { prodLog.error('[BackgroundDedup] Batch dedup failed:', error) }) }, 5 * 60 * 1000) + this.debounceTimer.unref?.() } /** diff --git a/src/import/ImportCoordinator.ts b/src/import/ImportCoordinator.ts index 21d09410..1e1316b7 100644 --- a/src/import/ImportCoordinator.ts +++ b/src/import/ImportCoordinator.ts @@ -13,7 +13,6 @@ import { Brainy } from '../brainy.js' import { FormatDetector, SupportedFormat } from './FormatDetector.js' import { ImportHistory, type ImportHistoryEntry } from './ImportHistory.js' -import { BackgroundDeduplicator } from './BackgroundDeduplicator.js' import { SmartExcelImporter } from '../importers/SmartExcelImporter.js' import { SmartPDFImporter } from '../importers/SmartPDFImporter.js' import { SmartCSVImporter } from '../importers/SmartCSVImporter.js' @@ -112,7 +111,12 @@ export interface ValidImportOptions { /** Confidence threshold for entities */ confidenceThreshold?: number - /** Enable entity deduplication across imports */ + /** + * Enable entity deduplication (default: true). Gates BOTH passes: the + * inline merge during import AND the debounced background pass that runs + * ~5 minutes after the last import (which merge-DELETES duplicate entities). + * Set false for deployments that must never auto-remove records. + */ enableDeduplication?: boolean /** Similarity threshold for deduplication (0-1) */ @@ -286,7 +290,6 @@ export class ImportCoordinator { private brain: Brainy private detector: FormatDetector private history: ImportHistory - private backgroundDedup: BackgroundDeduplicator private excelImporter: SmartExcelImporter private pdfImporter: SmartPDFImporter private csvImporter: SmartCSVImporter @@ -300,7 +303,6 @@ export class ImportCoordinator { this.brain = brain this.detector = new FormatDetector() this.history = new ImportHistory(brain) - this.backgroundDedup = new BackgroundDeduplicator(brain) this.excelImporter = new SmartExcelImporter(brain) this.pdfImporter = new SmartPDFImporter(brain) this.csvImporter = new SmartCSVImporter(brain) @@ -1459,9 +1461,16 @@ export class ImportCoordinator { } } - // Schedule background deduplication (debounced 5 minutes) - if (trackingContext && trackingContext.importId) { - this.backgroundDedup.scheduleDedup(trackingContext.importId) + // Schedule background deduplication (debounced 5 minutes, brain-owned so + // close() can cancel it). Honors the same enableDeduplication gate as the + // inline pass — false means NO dedup, inline or background. + if ( + trackingContext && + trackingContext.importId && + options.enableDeduplication !== false + ) { + const backgroundDedup = await this.brain.getBackgroundDeduplicator() + backgroundDedup.scheduleDedup(trackingContext.importId) } return { diff --git a/tests/integration/background-dedup-lifecycle.test.ts b/tests/integration/background-dedup-lifecycle.test.ts new file mode 100644 index 00000000..48c7a4a7 --- /dev/null +++ b/tests/integration/background-dedup-lifecycle.test.ts @@ -0,0 +1,85 @@ +/** + * @module tests/integration/background-dedup-lifecycle + * @description The post-import background deduplication pass (a merge-DELETE + * writer) obeys the same contract as the inline pass. Laws: + * (1) enableDeduplication:false schedules NO background pass — the brain-owned + * deduplicator is never even constructed; + * (2) by default the pass IS scheduled, brain-owned, with an unref'd timer + * (a pending pass never holds the process open); + * (3) repeated imports debounce into ONE pending batch on ONE instance + * (per-coordinator instances used to arm one timer per import); + * (4) close() cancels pending work — no delete pass can fire after close. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { Brainy } from '../../src/brainy.js' + +const ROWS = [ + { name: 'Alice Zephyr', role: 'engineer' }, + { name: 'Bob Quill', role: 'writer' } +] + +// Keep imports fast and deterministic — dedup scheduling is what's under test. +const FAST = { + enableNeuralExtraction: false, + enableRelationshipInference: false, + enableConceptExtraction: false +} as const + +// Deterministic stub embedder (hnsw-rebuild.test.ts pattern) — dedup +// scheduling never inspects vector CONTENT, so skip the WASM model load. +const stubEmbedding = async (text: string): Promise => { + const hash = text.split('').reduce((acc, char) => acc + char.charCodeAt(0), 0) + const vector = new Array(384).fill(0).map((_, i) => Math.sin(hash + i)) + return vector +} + +describe('background dedup lifecycle', () => { + let brain: Brainy + + beforeEach(async () => { + brain = new Brainy({ + requireSubtype: false, + storage: { type: 'memory' as const }, + embeddingFunction: stubEmbedding + }) + await brain.init() + }) + + afterEach(async () => { + await brain.close() + }) + + it('enableDeduplication:false schedules no background pass at all', async () => { + await brain.import(ROWS, { ...FAST, enableDeduplication: false }) + expect((brain as any)._backgroundDedup).toBeUndefined() + }) + + it('default schedules a brain-owned pass with an unref-ed timer', async () => { + await brain.import(ROWS, { ...FAST }) + const dedup = (brain as any)._backgroundDedup + expect(dedup).toBeDefined() + expect(dedup.pendingImports.size).toBe(1) + const timer = dedup.debounceTimer + expect(timer).toBeDefined() + // Node timers expose hasRef(); an unref'd timer must not hold the process. + expect(typeof timer.hasRef).toBe('function') + expect(timer.hasRef()).toBe(false) + }) + + it('imports debounce into one pending batch on one brain-owned instance', async () => { + await brain.import(ROWS, { ...FAST }) + const first = (brain as any)._backgroundDedup + await brain.import([{ name: 'Cara Vex', role: 'analyst' }], { ...FAST }) + expect((brain as any)._backgroundDedup).toBe(first) + expect(first.pendingImports.size).toBe(2) + }) + + it('close() cancels pending background dedup', async () => { + await brain.import(ROWS, { ...FAST }) + const dedup = (brain as any)._backgroundDedup + expect(dedup.debounceTimer).toBeDefined() + await brain.close() + expect(dedup.debounceTimer).toBeUndefined() + expect(dedup.pendingImports.size).toBe(0) + }) +}) From 6207e48b518bd80bdbb0113099a69a7a619e0957 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sat, 18 Jul 2026 10:51:37 -0700 Subject: [PATCH 10/29] fix: O(1) adaptive retention accounting + historyStats fleet audit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Under default adaptive retention, every flush() recomputed total history bytes by walking EVERY committed generation's delta — O(all generations) with disk re-reads past the 4096-entry delta-cache bound. On a production brain with 70,000+ accumulated generations this turned every write into a full-tail scan (60-100s writes, escalating with history growth), even though the free-RAM budget never tripped and nothing was ever reclaimed (SELF-GENERATIONS-GROWTH). historyBytes() now maintains a running total: seeded by one walk on first use, then updated incrementally at both commit paths (+bytes) and the compaction reclaim loop (−bytes), dropped on reopenAfterRestore. The adaptive retention check on every flush is O(1). Invariant regression- pinned: running total ≡ fresh walk through transact commits, single-op group commits, and compaction. New brain.historyStats() (exported HistoryStats): read-only generation count / bytes / generation+timestamp range / horizon / retention mode / effective budget — the one-call per-brain fleet audit for retention exposure. --- RELEASES.md | 26 ++++++++- src/brainy.ts | 28 ++++++++++ src/db/generationStore.ts | 72 +++++++++++++++++++++++-- src/db/types.ts | 30 +++++++++++ src/index.ts | 1 + tests/unit/db/generationStore.test.ts | 76 +++++++++++++++++++++++++++ 6 files changed, 228 insertions(+), 5 deletions(-) diff --git a/RELEASES.md b/RELEASES.md index 4a62123a..fd9d64ec 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -10,7 +10,31 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so --- -## v8.8.1 — 2026-07-18 (the import dedup off-switch is now honest + lifecycle-safe) +## v8.8.1 — 2026-07-18 (flush no longer walks the whole generation history + the import dedup off-switch is now honest) + +### The flush-storm fix (production incident, reported by a long-running deployment) + +Under the default adaptive retention, **every `flush()` re-walked the entire committed +generation history** to compute total history bytes for the budget check — O(all +generations) with disk re-reads past the 4,096-entry delta-cache bound. On a brain with +70,000+ accumulated generations that turned every write into a full-tail scan (60-100s +writes), even though the budget (free-RAM-based) never tripped and nothing was ever +reclaimed. Fixed: + +- `historyBytes()` now maintains a **running total**: seeded by one walk on first use, + then updated incrementally at every commit and reclaim — the adaptive retention check + on every flush is O(1). Invariant regression-pinned (running total ≡ fresh walk through + both commit paths and compaction). +- New **`brain.historyStats()`** (read-only, exported `HistoryStats`): generation count, + total on-disk bytes, generation/timestamp range, compaction horizon, retention mode, + and the effective adaptive budget — the one-call fleet-audit for sizing retention + exposure per brain. +- Interim guidance for keep-everything deployments already affected: `retention: 'all'` + skips the adaptive accounting entirely (and is the correct policy if you never want + history reclaimed). The accumulated files are harmless at rest; this release removes + the per-write cost of their existence. + +### The import dedup off-switch (lifecycle honesty) The post-import background deduplication pass (a merge-DELETE writer that runs ~5 minutes after an import, merging entities judged duplicates by id / name / vector similarity) had diff --git a/src/brainy.ts b/src/brainy.ts index c7b4c89a..008733b9 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -192,6 +192,7 @@ import { MemoryStorage } from './storage/adapters/memoryStorage.js' import type { CompactHistoryOptions, CompactHistoryResult, + HistoryStats, TransactOptions, TransactReceipt, TxLogEntry, @@ -8211,6 +8212,33 @@ export class Brainy implements BrainyInterface { return this.generationStore.compact(options) } + /** + * @description Read-only generational-history footprint for fleet audits: + * generation count, total on-disk bytes, generation/timestamp range, the + * compaction horizon, and the retention policy in force. Touches no data and + * changes nothing. First call pays one walk over committed deltas to seed + * the running byte total (subsequent calls — and every adaptive retention + * check — are then O(1)). + * + * A pool operator's exposure check is one call per brain: + * @example + * const stats = await brain.historyStats() + * console.log(`${stats.generations} generations, ${stats.bytes} bytes, mode=${stats.retentionMode}`) + */ + async historyStats(): Promise { + await this.ensureInitialized() + const stats = await this.generationStore.historyStats() + const policy = this.resolveRetentionPolicy() + return { + ...stats, + retentionMode: policy.mode, + effectiveBudgetBytes: + policy.mode === 'adaptive' + ? this.adaptiveHistoryBudgetBytes(policy.budgetBytes) + : null + } + } + /** * @description Drive the adaptive retention byte budget at runtime — the * settable input a machine-level coordinator (e.g. cor's `ResourceManager`, diff --git a/src/db/generationStore.ts b/src/db/generationStore.ts index 0d813f4d..dd70f7aa 100644 --- a/src/db/generationStore.ts +++ b/src/db/generationStore.ts @@ -257,6 +257,15 @@ export class GenerationStore { */ private deltaCacheMax = 4096 + /** + * Running total of on-disk history bytes across committed generations — + * `null` until {@link historyBytes} pays its one seeding walk. Maintained + * incrementally at commit/reclaim so the adaptive retention check on every + * flush() is O(1), never a tail re-walk. Never updated by cache re-reads + * ({@link setDelta} inserts are cache population, not new history). + */ + private historyBytesTotal: number | null = null + /** * Model-B per-write group-commit — the in-memory PENDING tier. * @@ -483,6 +492,40 @@ export class GenerationStore { return this.horizonGen } + /** + * @description Read-only history footprint for fleet audits: how much + * generational history this store holds on disk. `bytes` pays (and seeds) + * the one-time {@link historyBytes} walk on first call — subsequent calls + * are O(1). The oldest/newest timestamps come from those generations' + * deltas (cache-bounded reads). + * @returns Counts, bytes, generation range, and the compaction horizon. + */ + async historyStats(): Promise<{ + generations: number + bytes: number + oldestGeneration: number | null + newestGeneration: number | null + oldestTimestamp: number | null + newestTimestamp: number | null + horizon: number + }> { + let oldest: number | null = null + let newest: number | null = null + for (const gen of this.committedGensAsc()) { + if (oldest === null) oldest = gen + newest = gen + } + return { + generations: this.committedCount(), + bytes: await this.historyBytes(), + oldestGeneration: oldest, + newestGeneration: newest, + oldestTimestamp: oldest !== null ? (await this.getDelta(oldest)).timestamp : null, + newestTimestamp: newest !== null ? (await this.getDelta(newest)).timestamp : null, + horizon: this.horizonGen + } + } + /** * @description Read one generation's persisted before-image records — the * compaction fallback for generations written before deltas carried @@ -849,6 +892,9 @@ export class GenerationStore { timestamp, bytes: delta.bytes ?? 0 }) + if (this.historyBytesTotal !== null) { + this.historyBytesTotal += delta.bytes ?? 0 + } this.extendChains(gen, nouns, verbs) const logEntry: TxLogEntry = { generation: gen, timestamp, ...(args.meta && { meta: args.meta }) } await this.storage.appendTxLogLine(JSON.stringify(logEntry)) @@ -1302,6 +1348,9 @@ export class GenerationStore { timestamp: buf.timestamp, bytes: genBytes.get(gen) ?? 0 }) + if (this.historyBytesTotal !== null) { + this.historyBytesTotal += genBytes.get(gen) ?? 0 + } this.pendingBuffer.delete(gen) } this.pendingGens = [] @@ -2122,17 +2171,26 @@ export class GenerationStore { /** * @description Total serialized bytes of the ON-DISK generational history — * the sum of every committed generation's recorded `bytes`. Backs the - * `maxBytes` and adaptive retention caps. Reads each committed generation's - * delta (cached; a re-read only for cache-evicted ones) — O(committed - * generations), bounded by retention itself and invoked only at compaction - * time. Pending (un-flushed) generations are excluded (they are not on disk). + * `maxBytes` and adaptive retention caps. O(1) after the first call: the + * total is computed by ONE walk over committed deltas, then maintained + * incrementally at every commit (+bytes) and reclaim (−bytes) and dropped on + * a wholesale state replacement (restore). Without the running total, the + * adaptive auto-compaction on every flush() re-walked the ENTIRE history — + * O(committed generations) file reads per flush past the delta-cache bound — + * which is how a 70k-generation production brain turned every write into a + * full-tail scan (SELF-GENERATIONS-GROWTH). Pending (un-flushed) generations + * are excluded (they are not on disk). * @returns The total on-disk history byte count. */ async historyBytes(): Promise { + if (this.historyBytesTotal !== null) { + return this.historyBytesTotal + } let total = 0 for (const gen of this.committedGensAsc()) { total += (await this.getDelta(gen)).bytes } + this.historyBytesTotal = total return total } @@ -2206,6 +2264,9 @@ export class GenerationStore { await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${gen}`) this.deltaCache.delete(gen) + if (this.historyBytesTotal !== null) { + this.historyBytesTotal -= delta.bytes + } // AFTER the record-set is gone (over-count-only crash ordering): // release its history references and reclaim any blob left with zero @@ -2266,6 +2327,9 @@ export class GenerationStore { async reopenAfterRestore(floorGeneration: number): Promise { await this.withMutex(async () => { this.deltaCache.clear() + // The running history-byte total describes the REPLACED store — drop it; + // the next historyBytes() re-seeds with one walk over the new state. + this.historyBytesTotal = null // A wholesale state replacement invalidates any buffered single-op // history — discard the pending tier (its live writes are gone with the // replaced store). diff --git a/src/db/types.ts b/src/db/types.ts index d1355dab..521396b8 100644 --- a/src/db/types.ts +++ b/src/db/types.ts @@ -193,6 +193,36 @@ export interface CompactHistoryResult { horizon: number } +/** + * @description Result of `brain.historyStats()` — the read-only generational + * history footprint, for fleet audits and ops doors. A pool operator runs this + * per brain to size retention exposure (how much MVCC history each brain + * carries and under which policy) without touching any data. + */ +export interface HistoryStats { + /** Committed generation record-sets currently on disk. */ + generations: number + /** Total on-disk history bytes across those record-sets. */ + bytes: number + /** Oldest committed generation still on disk (null when history is empty). */ + oldestGeneration: number | null + /** Newest committed generation (null when history is empty). */ + newestGeneration: number | null + /** Commit timestamp (ms) of the oldest on-disk generation. */ + oldestTimestamp: number | null + /** Commit timestamp (ms) of the newest on-disk generation. */ + newestTimestamp: number | null + /** Compaction horizon — generations below it were reclaimed. */ + horizon: number + /** The effective retention mode this brain runs under. */ + retentionMode: 'all' | 'adaptive' | 'explicit' + /** + * The adaptive byte budget in force (coordinator-driven or the local + * free-memory probe); null under 'all' or explicit caps. + */ + effectiveBudgetBytes: number | null +} + // ============================================================================ // Db surfaces // ============================================================================ diff --git a/src/index.ts b/src/index.ts index 5c57fe29..ee01d885 100644 --- a/src/index.ts +++ b/src/index.ts @@ -201,6 +201,7 @@ export type { TxLogEntry, CompactHistoryOptions, CompactHistoryResult, + HistoryStats, ChangedIds, DiffResult, HistoryVersion, diff --git a/tests/unit/db/generationStore.test.ts b/tests/unit/db/generationStore.test.ts index 184d3974..611a97d7 100644 --- a/tests/unit/db/generationStore.test.ts +++ b/tests/unit/db/generationStore.test.ts @@ -489,4 +489,80 @@ describe('db/GenerationStore', () => { store.release(2) }) }) + + // ========================================================================== + describe('history-bytes running total (the O(1) retention check)', () => { + /** A fresh walk with the cache dropped — ground truth for the invariant. */ + async function groundTruthBytes(): Promise { + ;(store as any).historyBytesTotal = null + return store.historyBytes() + } + + it('is seeded once, then maintained through commits WITHOUT re-walks', async () => { + await commitWrite(ID_A, 1) + await commitWrite(ID_A, 2) + const seeded = await store.historyBytes() + expect(seeded).toBe(await groundTruthBytes()) + + // From here every read must come from the running total, not a walk: + // getDelta re-reads are the walk's cost — commits must not trigger any. + const getDeltaSpy = vi.spyOn(store as any, 'getDelta') + await commitWrite(ID_B, 1) + const afterCommit = await store.historyBytes() + expect(getDeltaSpy).not.toHaveBeenCalled() + getDeltaSpy.mockRestore() + expect(afterCommit).toBe(await groundTruthBytes()) + }) + + it('stays exact through single-op group commits and compaction', async () => { + await commitWrite(ID_A, 1) + await store.historyBytes() // seed + // Single-op path: buffered generations flushed as one group commit. + await store.commitSingleOp({ + touched: { nouns: [ID_B] }, + execute: async () => { + await storage.saveNounMetadata(ID_B, metadataFixture(1)) + } + }) + await store.flushPendingSingleOps() + expect(await store.historyBytes()).toBe(await groundTruthBytes()) + + await store.historyBytes() // re-seed after ground-truth reset + await store.compact({ maxGenerations: 1 }) + expect(await store.historyBytes()).toBe(await groundTruthBytes()) + }) + + it('historyStats reports counts, bytes, range, and horizon read-only', async () => { + await commitWrite(ID_A, 1) + await commitWrite(ID_B, 1) + const stats = await store.historyStats() + expect(stats.generations).toBe(2) + expect(stats.bytes).toBe(await store.historyBytes()) + expect(stats.oldestGeneration).toBe(1) + expect(stats.newestGeneration).toBe(2) + expect(stats.oldestTimestamp).toBeLessThanOrEqual(stats.newestTimestamp!) + expect(stats.horizon).toBe(0) + // Read-only: nothing was reclaimed by asking. + expect(store.committedGeneration()).toBe(2) + + await store.compact({ maxGenerations: 1 }) + const after = await store.historyStats() + expect(after.generations).toBe(1) + expect(after.oldestGeneration).toBe(2) + expect(after.horizon).toBe(1) + }) + + it('empty history reports null range and zero bytes', async () => { + const stats = await store.historyStats() + expect(stats).toMatchObject({ + generations: 0, + bytes: 0, + oldestGeneration: null, + newestGeneration: null, + oldestTimestamp: null, + newestTimestamp: null, + horizon: 0 + }) + }) + }) }) From a544225872d11f61439585305653d590b0970fd9 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sat, 18 Jul 2026 10:57:30 -0700 Subject: [PATCH 11/29] chore(release): 8.8.1 --- CHANGELOG.md | 6 ++++++ package-lock.json | 4 ++-- package.json | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f97046f0..a4a97ba3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. +### [8.8.1](https://github.com/soulcraftlabs/brainy/compare/v8.8.0...v8.8.1) (2026-07-18) + +- fix: O(1) adaptive retention accounting + historyStats fleet audit (6207e48) +- fix: import dedup off-switch honesty + brain-owned lifecycle for the background pass (4fcef7b) + + ### [8.8.0](https://github.com/soulcraftlabs/brainy/compare/v8.7.1...v8.8.0) (2026-07-17) - feat: OS-limit detection for pool-scale deployments (16a73b8) diff --git a/package-lock.json b/package-lock.json index 198e9116..3f157871 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraft/brainy", - "version": "8.8.0", + "version": "8.8.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraft/brainy", - "version": "8.8.0", + "version": "8.8.1", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index 7c3e050f..fb467c2e 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraft/brainy", - "version": "8.8.0", + "version": "8.8.1", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", "module": "dist/index.js", From 42037d0cd0e85d91555842e1e6badb4a1ecf51db Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sat, 18 Jul 2026 14:02:23 -0700 Subject: [PATCH 12/29] chore: push public docs to the soulcraft.com ingest door on release MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit scripts/push-docs.js collects docs/**/*.md with public:true frontmatter and POSTs them (batches of 10, idempotent per slug) to the docs ingest door after npm publish — release.sh step 12. Absent secret = loud skip (publish already happened; the serving side can interim-sync); a failed push exits non-zero so the docs site never silently trails npm. The combined /docs landing index is deliberately NOT pushed per-repo — it spans both engine corpora and is authored on the serving side. --- scripts/push-docs.js | 116 +++++++++++++++++++++++++++++++++++++++++++ scripts/release.sh | 12 +++++ 2 files changed, 128 insertions(+) create mode 100644 scripts/push-docs.js diff --git a/scripts/push-docs.js b/scripts/push-docs.js new file mode 100644 index 00000000..699d332b --- /dev/null +++ b/scripts/push-docs.js @@ -0,0 +1,116 @@ +#!/usr/bin/env node +/** + * @module scripts/push-docs + * @description Push this repo's PUBLIC docs to the soulcraft.com docs ingest + * door after an npm publish (VENUE-DOCS-RELEASE-PUSH — retires the old + * build-time docs sync). + * + * Contract (mirrors the reference implementation on the serving side): + * POST {base}/api/docs/ingest + * headers: x-service-secret: $DOCS_INGEST_SECRET, Content-Type: application/json + * body: { docs: [{ slug, title, markdown, nav: { order, section } }] } + * batches of 10, idempotent per slug. + * + * A doc is public iff its frontmatter has `public: true` AND a `slug`. The + * frontmatter is stripped; `category` → nav.section, `order` → nav.order. + * + * Deliberately NOT pushed: the combined /docs landing index. It spans BOTH + * engine corpora (this repo's and the native accelerator's), so a per-repo + * push would clobber the union — the index is authored on the serving side. + * + * Env: DOCS_INGEST_SECRET (required), DOCS_INGEST_BASE (default + * https://soulcraft.com). Exits 0 with a LOUD warning when the secret is + * absent (the npm publish has already happened; the serving side runs its + * interim sync on request) and exits 1 when a push actually fails — the docs + * site would silently trail npm otherwise, and that must be visible. + */ +import * as fs from 'node:fs' +import * as path from 'node:path' + +const BASE = (process.env.DOCS_INGEST_BASE || 'https://soulcraft.com').replace(/\/+$/, '') +const SECRET = process.env.DOCS_INGEST_SECRET +const DOCS_DIR = path.join(path.dirname(new URL(import.meta.url).pathname), '..', 'docs') +const BATCH = 10 + +if (!SECRET) { + console.warn( + '⚠️ DOCS PUSH SKIPPED: DOCS_INGEST_SECRET is not set.\n' + + ' soulcraft.com/docs now TRAILS this npm release until docs are pushed.\n' + + ' Either export DOCS_INGEST_SECRET and re-run `node scripts/push-docs.js`,\n' + + ' or ping venue on VENUE-DOCS-RELEASE-PUSH for the interim sync.' + ) + process.exit(0) +} + +/** Minimal frontmatter split — returns [meta, body] or [null, raw]. */ +function parseFrontmatter(raw) { + const m = raw.match(/^---\n([\s\S]*?)\n---\n([\s\S]*)$/) + if (!m) return [null, raw] + const meta = {} + for (const line of m[1].split('\n')) { + const kv = line.match(/^(\w[\w-]*):\s*(.*)$/) + if (kv) meta[kv[1]] = kv[2].trim().replace(/^["']|["']$/g, '') + } + return [meta, m[2]] +} + +const docs = [] +;(function walk(dir) { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name) + if (entry.isDirectory()) walk(full) + else if (entry.name.endsWith('.md')) { + const [meta, body] = parseFrontmatter(fs.readFileSync(full, 'utf-8')) + if (!meta || meta.public !== 'true' || !meta.slug) continue + docs.push({ + slug: meta.slug, + title: meta.title || meta.slug, + markdown: body.trim(), + nav: { + order: Number.parseInt(meta.order || '99', 10) || 99, + section: meta.category || 'guides' + } + }) + } + } +})(DOCS_DIR) + +if (docs.length === 0) { + console.error('❌ DOCS PUSH FAILED: zero public docs collected — refusing to push an empty corpus.') + process.exit(1) +} +docs.sort((a, b) => a.slug.localeCompare(b.slug)) +console.log(`Pushing ${docs.length} public docs to ${BASE}/api/docs/ingest …`) + +let failed = false +for (let i = 0; i < docs.length; i += BATCH) { + const batch = docs.slice(i, i + BATCH) + try { + const res = await fetch(`${BASE}/api/docs/ingest`, { + method: 'POST', + headers: { + 'x-service-secret': SECRET, + 'Content-Type': 'application/json', + 'User-Agent': 'brainy-docs-push/1.0' + }, + body: JSON.stringify({ docs: batch }), + signal: AbortSignal.timeout(120_000) + }) + if (!res.ok) { + throw new Error(`HTTP ${res.status}: ${(await res.text()).slice(0, 300)}`) + } + console.log(` batch ${i / BATCH + 1}: ${batch.map((d) => d.slug).join(', ')} → ok`) + } catch (err) { + failed = true + console.error(` batch ${i / BATCH + 1} FAILED: ${err instanceof Error ? err.message : err}`) + } +} + +if (failed) { + console.error( + '❌ DOCS PUSH INCOMPLETE — soulcraft.com/docs may trail npm. ' + + 'Re-run `node scripts/push-docs.js` or ping venue on VENUE-DOCS-RELEASE-PUSH.' + ) + process.exit(1) +} +console.log('✅ Docs pushed.') diff --git a/scripts/release.sh b/scripts/release.sh index 0e6a9c43..7d860564 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -196,6 +196,18 @@ else fi echo -e "${GREEN}✅ GitHub release created${NC}\n" +# Step 12: Push public docs to the soulcraft.com docs ingest door +# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when +# DOCS_INGEST_SECRET is unset; fails loudly (without undoing the publish — +# that already happened) when a push errors, so the docs site never +# silently trails npm. +echo -e "${BLUE}1️⃣2️⃣ Pushing public docs to soulcraft.com/docs...${NC}" +if node scripts/push-docs.js; then + echo -e "${GREEN}✅ Docs push step done${NC}\n" +else + echo -e "${RED}❌ Docs push FAILED — soulcraft.com/docs trails npm until re-run or interim sync${NC}\n" +fi + echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}" echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" From 945d92d29e64ec84bee370b22692637ffed31e4c Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sun, 19 Jul 2026 10:54:36 -0700 Subject: [PATCH 13/29] fix: one field-resolution law across aggregation hooks, source.where, removeMany, and find() spellings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four fixes from a consumer conformance report, one root disease — two field-resolution regimes where there must be one: - The delete/update aggregation hooks fed the engine a partial entity view (type/service/data/metadata only), so a reserved-field groupBy (subtype, visibility, ...) resolved to a nonexistent group on the way down: counts drifted upward forever after deletes, and updates moving an entity between reserved-field groups double-counted. The hooks now pass the full-fidelity view via entityForAggFromRawRecord (every reserved field top-level, mirroring the add path); the update sites pass the full get() view instead of a hand-rolled subset. - Aggregation source.where resolved fields only against the custom metadata bag, so where on a reserved field silently matched nothing. The matcher now resolves each filtered field through resolveEntityField — the same single source of truth groupBy uses. - removeMany() with no usable selector (bare array passed positionally, empty params, ids: []) resolved successfully having deleted nothing. All three now throw; the two legacy tests that pinned the silent no-op as 'graceful' now pin the refusal. - find() where keys accept both spellings: a metadata.-prefixed key falls back to its flattened spelling when the prefixed one is not indexed (metadata is flattened at index time). A literal nested custom key named metadata still wins when indexed as spelled. Five regression pins in aggregate-reserved-fields.test.ts (4 of 5 vary red on the unfixed code). --- RELEASES.md | 26 +++ src/aggregation/AggregationIndex.ts | 14 +- src/brainy.ts | 104 +++++++---- src/utils/metadataIndex.ts | 21 ++- .../aggregate-reserved-fields.test.ts | 169 ++++++++++++++++++ .../metadata-index-cleanup.unit.test.ts | 8 +- tests/unit/brainy/batch-operations.test.ts | 6 +- 7 files changed, 303 insertions(+), 45 deletions(-) create mode 100644 tests/integration/aggregate-reserved-fields.test.ts diff --git a/RELEASES.md b/RELEASES.md index fd9d64ec..d1374d5a 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -10,6 +10,32 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so --- +## v8.8.2 — 2026-07-19 (one field-resolution law: reserved-field aggregates stop drifting) + +Four fixes from a consumer conformance audit, all rooted in the same disease — two field-resolution +regimes where there must be one: + +- **Aggregates grouped by a RESERVED field (`subtype`, `visibility`, …) now decrement on + delete.** The delete/update hooks fed the aggregation engine a partial entity view (type, + service, data, metadata only), so a reserved-field `groupBy` resolved to a nonexistent group + on the way DOWN — counts drifted upward forever after any delete, and updates that moved an + entity between reserved-field groups double-counted it. The hooks now pass the full-fidelity + entity view (every reserved field top-level, the same shape the add path uses). If your + deployment derives stats from reserved-field aggregates, re-define those aggregates once + after upgrading (a changed definition triggers one rescan) or run them fresh — the drifted + persisted counts do not self-heal retroactively. +- **Aggregation `source.where` on reserved fields now filters** instead of silently matching + nothing: the matcher resolves fields through the same resolver `groupBy` uses (top-level + standard fields + custom metadata), so `where: { subtype: 'note' }` means what it says. +- **`removeMany()` refuses empty/invalid selectors loudly.** A bare array passed positionally + (`removeMany([id])` instead of `removeMany({ ids: [id] })`), an empty params object, or + `ids: []` used to resolve successfully having deleted nothing. All three now throw. +- **`find()` accepts both where-key spellings.** Metadata is flattened at index time + (`metadata.entry.title` indexes as `entry.title`); a `metadata.`-prefixed where key now + falls back to its flattened spelling when the prefixed one isn't indexed — the + "unindexed field(s), returning []" confusion for storage-shaped spellings is gone. (A + literal nested custom key named `metadata` still wins when indexed as spelled.) + ## v8.8.1 — 2026-07-18 (flush no longer walks the whole generation history + the import dedup off-switch is now honest) ### The flush-storm fix (production incident, reported by a long-running deployment) diff --git a/src/aggregation/AggregationIndex.ts b/src/aggregation/AggregationIndex.ts index 7f7ffb27..f9382218 100644 --- a/src/aggregation/AggregationIndex.ts +++ b/src/aggregation/AggregationIndex.ts @@ -88,10 +88,18 @@ function matchesSource(entity: Record, source: AggregateDefinit if (entity.service !== source.service) return false } - // Metadata where filter — match against the entity's metadata sub-object + // Where filter — resolve each filtered field through resolveEntityField, + // the SAME single source of truth groupBy uses (top-level standard fields + // + custom metadata). Matching only the metadata sub-object made + // where:{subtype}/{visibility}/… a silent no-op: reserved fields never + // live in the custom bag, so those filters could never match anything. if (source.where && Object.keys(source.where).length > 0) { - const metadata = (entity.metadata ?? entity) as Record - if (!matchesMetadataFilter(metadata, source.where)) return false + const e = entity as unknown as HNSWNounWithMetadata + const resolved: Record = {} + for (const key of Object.keys(source.where)) { + resolved[key] = resolveEntityField(e, key) + } + if (!matchesMetadataFilter(resolved, source.where)) return false } return true diff --git a/src/brainy.ts b/src/brainy.ts index 008733b9..21eab679 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -1883,6 +1883,29 @@ export class Brainy implements BrainyInterface { } } + /** + * @description Build the AGGREGATION view of an entity from a stored flat + * metadata record — EVERY reserved field mapped to its top-level entity + * name (stored `noun` → `type`), custom metadata in `metadata`. This must + * mirror the add-path `entityForIndexing` shape exactly: the aggregation + * engine resolves groupBy/where fields via `resolveEntityField` + * (top-level standard fields + custom metadata), so a view that drops a + * reserved field makes every aggregate grouped by that field decrement a + * group that does not exist — counts then drift upward forever after + * deletes (SELF-AGGREGATE-DELETE-DRIFT). Do not hand-roll subsets of this. + * @param record - The stored flat metadata record (before-image or pre-delete read). + * @returns The full-fidelity entity view for aggregation hooks. + */ + private entityForAggFromRawRecord(record: Record): Record { + const { reserved, custom } = splitNounMetadataRecord(record) + const { noun, ...rest } = reserved + return { + type: noun, + ...rest, + metadata: custom + } + } + /** * @description Add an entity (noun) to the brain. Embeds `data` into a vector and * indexes the entity across all three intelligences — vector similarity, graph @@ -3125,15 +3148,16 @@ export class Brainy implements BrainyInterface { ] : undefined) - // Aggregation hook (outside transaction — derived data) + // Aggregation hook (outside transaction — derived data). `existing` is + // the full get() view — every reserved field top-level — and must be + // passed whole: a subset view makes the old-side decrement miss any + // reserved-field group (update would then double-count it). if (this._aggregationIndex) { - const oldEntityForAgg = { - type: existing.type, - service: existing.service, - data: existing.data, - metadata: existing.metadata - } - this._aggregationIndex.onEntityUpdated(params.id, entityForIndexing, oldEntityForAgg) + this._aggregationIndex.onEntityUpdated( + params.id, + entityForIndexing, + existing as unknown as Record + ) } } @@ -3245,19 +3269,15 @@ export class Brainy implements BrainyInterface { ] : undefined) - // Aggregation hook (outside transaction — derived data) + // Aggregation hook (outside transaction — derived data). The view must + // carry EVERY reserved field top-level (not a subset): a groupBy on + // subtype/visibility/etc. otherwise decrements a nonexistent group and + // the real count never comes down. if (this._aggregationIndex && metadata) { - // Reconstruct entity-like object from stored metadata via the - // canonical reserved/custom split (the hand-rolled destructure here - // missed subtype/_rev, leaking them into the aggregation view). - const { reserved, custom } = splitNounMetadataRecord(metadata) - const entityForAgg = { - type: reserved.noun, - service: reserved.service, - data: reserved.data, - metadata: custom - } - this._aggregationIndex.onEntityDeleted(id, entityForAgg) + this._aggregationIndex.onEntityDeleted( + id, + this.entityForAggFromRawRecord(metadata as Record) + ) } } @@ -6885,6 +6905,30 @@ export class Brainy implements BrainyInterface { this.assertWritable('removeMany') await this.ensureInitialized() + // Loud selector validation: a call with no usable selector used to + // resolve successfully having deleted NOTHING (total: 0) — the classic + // silent no-op being a bare array passed positionally + // (removeMany([id]) instead of removeMany({ ids: [id] })). The caller + // believes the delete happened; every count derived afterwards is "wrong" + // while the engine was never even asked. Refuse instead. + if (Array.isArray(params)) { + throw new Error( + `removeMany() takes a params object, not a bare array — use removeMany({ ids: [...] })` + ) + } + if (!params || (!params.ids && !params.type && !params.where)) { + throw new Error( + `removeMany() requires a selector: { ids } and/or { type, where }. ` + + `An empty selector would silently delete nothing — refusing.` + ) + } + if (params.ids && params.ids.length === 0) { + throw new Error( + `removeMany() received ids: [] — an empty id list deletes nothing. ` + + `Pass the ids to delete, or omit ids and select by { type, where }.` + ) + } + // Determine what to delete let idsToDelete: string[] = [] @@ -9388,12 +9432,9 @@ export class Brainy implements BrainyInterface { ) plan.touchedNouns.push(params.id) - const oldEntityForAgg = { - type: existing.type, - service: existing.service, - data: existing.data, - metadata: existing.metadata - } + // The full planGetEntity view, passed whole — a subset view makes the + // old-side decrement miss reserved-field groups (double-count on update). + const oldEntityForAgg = existing as unknown as Record plan.postCommit.push(() => { if (this._aggregationIndex) { this._aggregationIndex.onEntityUpdated(params.id, entityForIndexing, oldEntityForAgg) @@ -9514,14 +9555,9 @@ export class Brainy implements BrainyInterface { } if (metadata) { - // Canonical reserved/custom split — mirror of remove()'s aggregation hook. - const { reserved, custom } = splitNounMetadataRecord(metadata) - const entityForAgg = { - type: reserved.noun, - service: reserved.service, - data: reserved.data, - metadata: custom - } + // Mirror of remove()'s aggregation hook — the FULL reserved view, so + // reserved-field groupBy decrements find their group. + const entityForAgg = this.entityForAggFromRawRecord(metadata as Record) plan.postCommit.push(() => { if (this._aggregationIndex) { this._aggregationIndex.onEntityDeleted(id, entityForAgg) diff --git a/src/utils/metadataIndex.ts b/src/utils/metadataIndex.ts index fd325319..c772bce4 100644 --- a/src/utils/metadataIndex.ts +++ b/src/utils/metadataIndex.ts @@ -1867,9 +1867,26 @@ export class MetadataIndexManager implements MetadataIndexProvider { // not once per AND-clause inside it. const unindexedFields: string[] = [] - for (const [field, condition] of Object.entries(filter)) { + for (const [rawField, condition] of Object.entries(filter)) { // Skip logical operators - if (field === 'allOf' || field === 'anyOf' || field === 'not') continue + if (rawField === 'allOf' || rawField === 'anyOf' || rawField === 'not') continue + + // Metadata is FLATTENED at index time (metadata.entry.title indexes as + // entry.title), so a `metadata.`-prefixed where key is almost always + // the caller spelling the STORAGE shape rather than the index shape. + // Accept both spellings: when the key as spelled is unindexed but its + // stripped spelling is, query the stripped one. A literal nested + // custom key named `metadata` still wins when indexed as spelled + // (checked first), so that rare shape keeps working. + let field = rawField + if ( + rawField.startsWith('metadata.') && + this.columnStore && + !this.columnStore.hasField(rawField) && + this.columnStore.hasField(rawField.slice('metadata.'.length)) + ) { + field = rawField.slice('metadata.'.length) + } let fieldResults: string[] = [] diff --git a/tests/integration/aggregate-reserved-fields.test.ts b/tests/integration/aggregate-reserved-fields.test.ts new file mode 100644 index 00000000..e81692d6 --- /dev/null +++ b/tests/integration/aggregate-reserved-fields.test.ts @@ -0,0 +1,169 @@ +/** + * @module tests/integration/aggregate-reserved-fields + * @description One field-resolution law across the whole aggregation + query + * surface (SELF-AGGREGATE-DELETE-DRIFT). Laws: + * (1) aggregates grouped by a RESERVED field (subtype) decrement on delete — + * the delete-side entity view carries every reserved field, so the + * decrement finds its group (counts must never drift from ground truth); + * (2) same for update: moving an entity between reserved-field groups + * decrements the old group and increments the new one (no double-count); + * (3) aggregation source.where on a reserved field (subtype) FILTERS instead + * of silently matching nothing; + * (4) removeMany refuses empty/invalid selectors loudly (bare array, empty + * object, ids: []) instead of resolving as a silent no-op; + * (5) find() accepts both where spellings: flattened (entry.title) and + * storage-shaped (metadata.entry.title) resolve to the same rows. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest' +import { Brainy } from '../../src/brainy.js' +import { NounType } from '../../src/types/graphTypes.js' + +const stubEmbedding = async (text: string): Promise => { + const hash = text.split('').reduce((acc, char) => acc + char.charCodeAt(0), 0) + return new Array(384).fill(0).map((_, i) => Math.sin(hash + i)) +} + +describe('aggregation + query field-resolution law', () => { + let brain: Brainy + + beforeEach(async () => { + brain = new Brainy({ + requireSubtype: false, + storage: { type: 'memory' as const }, + embeddingFunction: stubEmbedding + }) + await brain.init() + }) + + afterEach(async () => { + await brain.close() + }) + + it('reserved-field groupBy decrements on delete (the drift bug)', async () => { + brain.defineAggregate({ + name: 'by_subtype', + source: { type: NounType.Document }, + groupBy: ['subtype'], + metrics: { count: { op: 'count' } } + }) + + const ids: string[] = [] + for (let i = 0; i < 5; i++) { + ids.push( + await brain.add({ + data: `doc-${i}`, + type: NounType.Document, + subtype: 'note', + metadata: { team: 'alpha' } + }) + ) + } + let groups = await brain.queryAggregate('by_subtype') + expect(groups).toHaveLength(1) + expect(groups[0].groupKey).toEqual({ subtype: 'note' }) + expect(groups[0].metrics.count).toBe(5) + + await brain.remove(ids[0]) + await brain.flush() + + groups = await brain.queryAggregate('by_subtype') + expect(groups[0].metrics.count).toBe(4) + const live = await brain.find({ type: NounType.Document, limit: 100 }) + expect(groups[0].metrics.count).toBe(live.length) + }) + + it('reserved-field groupBy moves between groups on update (no double-count)', async () => { + brain.defineAggregate({ + name: 'by_subtype', + source: { type: NounType.Document }, + groupBy: ['subtype'], + metrics: { count: { op: 'count' } } + }) + const id = await brain.add({ + data: 'doc-move', + type: NounType.Document, + subtype: 'draft' + }) + await brain.update({ id, subtype: 'published' }) + + const groups = await brain.queryAggregate('by_subtype') + const byKey = Object.fromEntries( + groups.map((g) => [String(g.groupKey.subtype), g.metrics.count]) + ) + expect(byKey['published']).toBe(1) + // The old group must be gone or zero — never still counting the entity. + expect(byKey['draft'] ?? 0).toBe(0) + }) + + it('source.where on a reserved field filters instead of matching nothing', async () => { + brain.defineAggregate({ + name: 'notes_only', + source: { type: NounType.Document, where: { subtype: 'note' } }, + groupBy: ['team'], + metrics: { count: { op: 'count' } } + }) + await brain.add({ + data: 'n1', + type: NounType.Document, + subtype: 'note', + metadata: { team: 'alpha' } + }) + await brain.add({ + data: 'd1', + type: NounType.Document, + subtype: 'draft', + metadata: { team: 'alpha' } + }) + + const groups = await brain.queryAggregate('notes_only') + expect(groups).toHaveLength(1) + expect(groups[0].metrics.count).toBe(1) // the note, never the draft + }) + + it('removeMany refuses empty/invalid selectors loudly', async () => { + const id = await brain.add({ data: 'keep-me', type: NounType.Document }) + + // Bare array passed positionally — the classic silent no-op. + await expect( + brain.removeMany([id] as unknown as Parameters[0]) + ).rejects.toThrow(/bare array/) + // Empty selector object. + await expect( + brain.removeMany({} as Parameters[0]) + ).rejects.toThrow(/requires a selector/) + // Explicit empty id list. + await expect(brain.removeMany({ ids: [] })).rejects.toThrow(/ids: \[\]/) + + // Nothing was deleted by any of the refused calls. + expect(await brain.get(id)).toBeTruthy() + }) + + it('find() accepts both flattened and metadata.-prefixed where spellings', async () => { + await brain.add({ + data: 'nested-doc', + type: NounType.Document, + metadata: { entry: { title: 'T1' }, classifier: { contextHints: { vfsPath: '/n/a.md' } } } + }) + await brain.flush() + + const flat = await brain.find({ + type: NounType.Document, + where: { 'entry.title': 'T1' }, + limit: 10 + }) + const prefixed = await brain.find({ + type: NounType.Document, + where: { 'metadata.entry.title': 'T1' }, + limit: 10 + }) + const deepPrefixed = await brain.find({ + type: NounType.Document, + where: { 'metadata.classifier.contextHints.vfsPath': '/n/a.md' }, + limit: 10 + }) + expect(flat).toHaveLength(1) + expect(prefixed).toHaveLength(1) + expect(prefixed[0].id).toBe(flat[0].id) + expect(deepPrefixed).toHaveLength(1) + }) +}) diff --git a/tests/regression/metadata-index-cleanup.unit.test.ts b/tests/regression/metadata-index-cleanup.unit.test.ts index 266b9a4d..0984d727 100644 --- a/tests/regression/metadata-index-cleanup.unit.test.ts +++ b/tests/regression/metadata-index-cleanup.unit.test.ts @@ -205,10 +205,10 @@ describe('Metadata index cleanup after remove / removeMany', () => { } }) - it('handles empty ids array gracefully', async () => { - const result = await brain.removeMany({ ids: [] }) - expect(result.successful).toHaveLength(0) - expect(result.failed).toHaveLength(0) + it('refuses an empty ids array loudly (a silent no-op is not "graceful")', async () => { + // 8.8.2: an empty selector used to resolve successfully having deleted + // NOTHING — the caller believed the delete happened. Now it throws. + await expect(brain.removeMany({ ids: [] })).rejects.toThrow(/ids: \[\]/) }) it('handles large batch (> 1 chunk) without leaving stale index entries', async () => { diff --git a/tests/unit/brainy/batch-operations.test.ts b/tests/unit/brainy/batch-operations.test.ts index b2cc7f09..58b25744 100644 --- a/tests/unit/brainy/batch-operations.test.ts +++ b/tests/unit/brainy/batch-operations.test.ts @@ -533,8 +533,10 @@ describe('Brainy Batch Operations', () => { expect(result.successful).toHaveLength(0) await brain.updateMany({ items: [] }) - await brain.removeMany({ ids: [] }) - // Should not throw + // removeMany is the exception (8.8.2): an empty id list is a refused + // selector, not an empty batch — deleting "nothing" silently was the + // bug class (a positional/bare-array call looked identical). + await expect(brain.removeMany({ ids: [] })).rejects.toThrow(/ids: \[\]/) }) it('should validate batch size limits', async () => { From a16567d626198765fd26be77a471c0f911a6510b Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sun, 19 Jul 2026 11:18:18 -0700 Subject: [PATCH 14/29] chore(release): 8.8.2 --- CHANGELOG.md | 6 ++++++ package-lock.json | 4 ++-- package.json | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a4a97ba3..6b4f1a1c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. +### [8.8.2](https://github.com/soulcraftlabs/brainy/compare/v8.8.1...v8.8.2) (2026-07-19) + +- fix: one field-resolution law across aggregation hooks, source.where, removeMany, and find() spellings (945d92d) +- chore: push public docs to the soulcraft.com ingest door on release (42037d0) + + ### [8.8.1](https://github.com/soulcraftlabs/brainy/compare/v8.8.0...v8.8.1) (2026-07-18) - fix: O(1) adaptive retention accounting + historyStats fleet audit (6207e48) diff --git a/package-lock.json b/package-lock.json index 3f157871..c1d62fc9 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraft/brainy", - "version": "8.8.1", + "version": "8.8.2", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraft/brainy", - "version": "8.8.1", + "version": "8.8.2", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index fb467c2e..b43633ac 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraft/brainy", - "version": "8.8.1", + "version": "8.8.2", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", "module": "dist/index.js", From 300d9f2a16944fe49cbece344df48bc97c4bbfed Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sun, 19 Jul 2026 12:04:39 -0700 Subject: [PATCH 15/29] =?UTF-8?q?feat:=20flush()=20never=20compacts=20?= =?UTF-8?q?=E2=80=94=20history=20maintenance=20moves=20to=20close()=20with?= =?UTF-8?q?=20bounded=20passes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit flush() is durability work: it must cost what the current window's deltas cost, never what the history backlog costs. Under adaptive retention the byte budget derives from free memory, so bulk-load pressure shrank the budget exactly at peak write volume and flush paid actual reclaim inline — a production deployment measured single writes blocked 25-191s behind reclaim-on-flush. - flush() no longer calls autoCompactHistory(); close() is THE auto-compaction site (already ran there; now alone). - Every auto pass is time-bounded (CLOSE_COMPACTION_BUDGET_MS = 5s): reclamation is oldest-first, so an early stop is a consistent prefix and the next pass resumes. Explicit compactHistory() gains an optional timeBudgetMs for caller-chosen maintenance windows. - Documented trade stated where operators read: a long-lived writer that never closes accumulates history until its next explicit compactHistory() — predictable writes, explicit maintenance. Pins: flush-never-reclaims + close-reclaims-durably (db-mvcc), bounded pass stops-then-resumes as a consistent prefix (generationStore unit). --- docs/guides/snapshots-and-time-travel.md | 13 +++++-- src/brainy.ts | 48 +++++++++++++++++------- src/db/generationStore.ts | 6 +++ src/db/types.ts | 9 +++++ src/types/brainy.types.ts | 9 +++-- tests/integration/db-mvcc.test.ts | 32 ++++++++++------ tests/unit/db/generationStore.test.ts | 14 +++++++ 7 files changed, 100 insertions(+), 31 deletions(-) diff --git a/docs/guides/snapshots-and-time-travel.md b/docs/guides/snapshots-and-time-travel.md index 56c49044..490aecab 100644 --- a/docs/guides/snapshots-and-time-travel.md +++ b/docs/guides/snapshots-and-time-travel.md @@ -344,8 +344,12 @@ For per-entity write coordination (rather than whole-store history), the ## Keeping history bounded Under Model-B every write is a generation, so history can grow quickly — -Brainy auto-compacts on every `flush()`/`close()` under the **`retention`** -knob (configured on the constructor): +Brainy auto-compacts at `close()` (time-bounded per pass) under the +**`retention`** knob (configured on the constructor). Since 8.9.0, `flush()` +never compacts: flushing is durability work and costs only what the current +window's writes cost, regardless of history backlog. A long-lived writer that +never closes keeps its history until its next explicit `compactHistory()` — +schedule one in your maintenance window if you run bounded retention: ```typescript // Zero-config: ADAPTIVE — keep as much history as free disk/RAM allows, @@ -359,10 +363,13 @@ new Brainy({ retention: 'all' }) new Brainy({ retention: { maxGenerations: 1000, maxAge: 7 * 86_400_000, maxBytes: 512 * 1024 ** 2 } }) ``` -Reclaim manually at any time (the same caps): +Reclaim manually at any time (the same caps, plus an optional per-pass time +budget for maintenance windows — an early stop is a consistent prefix and the +next pass resumes): ```typescript await brain.compactHistory({ maxGenerations: 100, maxAge: 7 * 24 * 60 * 60 * 1000 }) +await brain.compactHistory({ maxBytes: 512 * 1024 ** 2, timeBudgetMs: 10_000 }) ``` Compaction never breaks a pinned read — record-sets are reclaimed only when diff --git a/src/brainy.ts b/src/brainy.ts index 21eab679..8ba991dd 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -396,6 +396,15 @@ export type IndexFamily = 'vector' | 'metadata' | 'graph' */ const AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS = 30_000 +/** + * Time budget for the auto-compaction pass at close() (8.9.0). Bounds how long + * a clean shutdown spends reclaiming history backlog — an early stop is a + * consistent prefix and the next close/explicit pass resumes. Explicit + * `compactHistory()` calls are unbounded unless the caller passes their own + * `timeBudgetMs` (maintenance windows choose their own budgets). + */ +const CLOSE_COMPACTION_BUDGET_MS = 5_000 + /** * The main Brainy class - Clean, Beautiful, Powerful * REAL IMPLEMENTATION - No stubs, no mocks @@ -8362,19 +8371,24 @@ export class Brainy implements BrainyInterface { /** * @description Run history compaction under the resolved `retention` policy - * when `autoCompact` is on (the default). Invoked from `flush()` and - * `close()` so generational record-sets cannot accumulate unbounded across a - * long-lived writer's lifetime. + * when `autoCompact` is on (the default). Invoked from `close()` ONLY + * (8.9.0) — flush() is durability work and never pays maintenance costs; a + * production deployment measured reclaim-on-flush blocking single writes + * for 25-191s under memory pressure. Long-lived writers that never close + * accumulate history until their next explicit `compactHistory()` — the + * documented trade: predictable writes, explicit maintenance. * * - `'all'` → returns without reclaiming (index compaction for speed still * runs elsewhere; history is decoupled and kept). * - `'adaptive'` → reclaim oldest-unpinned history down to the byte budget. * - `'explicit'` → apply the supplied `maxGenerations`/`maxAge`/`maxBytes` caps. * + * Every auto pass is TIME-BOUNDED ({@link CLOSE_COMPACTION_BUDGET_MS}) so a + * large backlog can never stall a clean shutdown; the next pass resumes. * Read-only instances and an explicit `autoCompact: false` skip silently. * Pinned generations are never reclaimed ({@link GenerationStore.compact}). * Failures are logged and swallowed — compaction is housekeeping and must - * never fail a flush or a clean shutdown. + * never fail a clean shutdown. */ private async autoCompactHistory(): Promise { // Nothing to compact on a read-only instance or before init wired up the @@ -8390,12 +8404,16 @@ export class Brainy implements BrainyInterface { if (policy.mode === 'adaptive') { const budget = this.adaptiveHistoryBudgetBytes(policy.budgetBytes) if (budget === Infinity) return // no pressure signal → keep everything this pass - await this.generationStore.compact({ maxBytes: budget }) + await this.generationStore.compact({ + maxBytes: budget, + timeBudgetMs: CLOSE_COMPACTION_BUDGET_MS + }) } else { await this.generationStore.compact({ maxGenerations: policy.maxGenerations, maxAge: policy.maxAge, - maxBytes: policy.maxBytes + maxBytes: policy.maxBytes, + timeBudgetMs: CLOSE_COMPACTION_BUDGET_MS }) } } catch (error) { @@ -10540,12 +10558,14 @@ export class Brainy implements BrainyInterface { this.generationStore.persistCounterNow() ]) - // 6. Auto-compact generational history per config.retention (default on). - // Runs after the flush so durable state is in place; respects live - // Db pins and an explicit autoCompact: false. - await this.autoCompactHistory() + // NOTE (8.9.0): flush() no longer compacts history. Flush is DURABILITY + // work — it must cost what this window's deltas cost, never what the + // history backlog costs. Auto-compaction (a MAINTENANCE concern) runs at + // close() and via explicit compactHistory(); a production deployment + // measured reclaim-on-flush stalling single writes for 25-191s under + // memory pressure, which is exactly the class this separation ends. - // 7. Stamp the entity tree: which source generation the canonical tree + // 6. Stamp the entity tree: which source generation the canonical tree // reflects + the rollup invariants that verify it whole (the counters // persisted in step 1). Written at flush boundaries — the tree tracks // every commit by construction, so the stamp is a durable checkpoint, @@ -16173,8 +16193,10 @@ export class Brainy implements BrainyInterface { } // Phase 0b: Auto-compact generational history per config.retention (default - // on) BEFORE the generation store closes below. Respects live Db pins and - // an explicit autoCompact: false; no-op on read-only instances. + // on) BEFORE the generation store closes below. This is THE auto-compaction + // site (8.9.0 — flush() never compacts): time-bounded per pass, respects + // live Db pins and an explicit autoCompact: false; no-op on read-only + // instances. await this.autoCompactHistory() // Phase 1: Flush ALL components in parallel to persist buffered data diff --git a/src/db/generationStore.ts b/src/db/generationStore.ts index dd70f7aa..4e3738d9 100644 --- a/src/db/generationStore.ts +++ b/src/db/generationStore.ts @@ -2220,6 +2220,11 @@ export class GenerationStore { const maxAge = options?.maxAge const maxBytes = options?.maxBytes const ageCutoff = maxAge !== undefined ? Date.now() - maxAge : undefined + // Bounded maintenance pass (8.9.0): stop reclaiming once the budget is + // spent. Safe mid-loop — reclamation is oldest-first, so an early stop + // leaves a consistent contiguous prefix and the next pass resumes. + const deadline = + options?.timeBudgetMs !== undefined ? Date.now() + options.timeBudgetMs : undefined const noCaps = maxGenerations === undefined && maxAge === undefined && maxBytes === undefined @@ -2233,6 +2238,7 @@ export class GenerationStore { for (const gen of [...this.committedGensAsc()]) { // Pins are always exempt: never reclaim a generation a live pin needs. if (gen > minPinned) break // committedGensAsc ascending → nothing newer is eligible either + if (deadline !== undefined && Date.now() >= deadline) break // budget spent — resume next pass const delta = await this.getDelta(gen) if (!noCaps) { const violatesCount = maxGenerations !== undefined && remainingCount > maxGenerations diff --git a/src/db/types.ts b/src/db/types.ts index 521396b8..4c8a4957 100644 --- a/src/db/types.ts +++ b/src/db/types.ts @@ -176,6 +176,15 @@ export interface CompactHistoryOptions { * of each surviving generation's serialized record set (`GenerationDelta.bytes`). */ maxBytes?: number + /** + * Stop reclaiming after this many milliseconds even if caps are still + * exceeded (8.9.0). Compaction is maintenance — a bounded pass keeps + * `close()` (and any explicit maintenance window) from stalling on a large + * backlog; the next pass resumes where this one stopped (reclamation is + * oldest-first, so an early stop is always a consistent prefix). Unset = + * run to completion. + */ + timeBudgetMs?: number } /** diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts index c2344375..1ec4c4c3 100644 --- a/src/types/brainy.types.ts +++ b/src/types/brainy.types.ts @@ -1737,7 +1737,10 @@ export interface BrainyConfig { * Under Model-B EVERY write (`transact()` AND single-op `add`/`update`/ * `remove`/`relate`) produces an immutable generation record-set serving * historical reads (`asOf()`, pinned `Db` values). Without compaction those - * accumulate, so Brainy **auto-compacts on every `flush()` and `close()`**. + * accumulate, so Brainy **auto-compacts at `close()`** (time-bounded per + * pass; 8.9.0 removed compaction from `flush()` — flush is durability work + * and never pays maintenance costs). A long-lived writer that never closes + * accumulates history until its next explicit `compactHistory()` call. * Live `Db` pins are ALWAYS exempt from reclamation, in every mode. * * Modes: @@ -1754,7 +1757,7 @@ export interface BrainyConfig { * the oldest unpinned generations while ANY supplied cap is exceeded * (predictable ops). `maxAge` in ms; `maxBytes` total history bytes. * - * `autoCompact: false` disables the automatic flush/close compaction (manage + * `autoCompact: false` disables the automatic close() compaction (manage * manually via `brain.compactHistory()`). `budgetBytes` is the settable * adaptive byte budget a coordinator drives (also via `brain.setRetentionBudget()`). * Long-term archives belong in `db.persist(path)` snapshots, which compaction @@ -1772,7 +1775,7 @@ export interface BrainyConfig { maxBytes?: number /** Adaptive byte budget for this brain, driven by a coordinator (e.g. cor). */ budgetBytes?: number - /** Run compaction automatically on flush()/close() (default: true). */ + /** Run compaction automatically at close() (default: true; 8.9.0 — flush() never compacts). */ autoCompact?: boolean } diff --git a/tests/integration/db-mvcc.test.ts b/tests/integration/db-mvcc.test.ts index 10a158ae..959d0053 100644 --- a/tests/integration/db-mvcc.test.ts +++ b/tests/integration/db-mvcc.test.ts @@ -1280,24 +1280,32 @@ describe('8.0 Db API — generational MVCC', () => { await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError) }) - it('Model-B retention — setRetentionBudget drives adaptive reclaim on flush; live data intact', async () => { - // Default brain → ADAPTIVE retention. A coordinator (e.g. cor's ResourceManager) - // pushes a byte budget via setRetentionBudget(); auto-compaction on flush() reclaims - // oldest history down toward it. Each update's before-image carries the full prior - // 384-dim vector (~KBs), so ~13 generations far exceed a few-KB budget. - const { brain } = await openFsBrain() + it('Model-B retention — flush() NEVER compacts (8.9.0); adaptive reclaim runs at close()', async () => { + // Default brain → ADAPTIVE retention with a driven byte budget far below + // the accumulated history (~13 generations of full-vector before-images). + // The 8.9.0 law: flush() is durability-only — it must not reclaim even + // when the budget is exceeded (reclaim-on-flush blocked production writes + // for 25-191s). Maintenance runs at close(), time-bounded. + const { brain, dir } = await openFsBrain() const a = uid('ret-budget') await brain.add({ id: a, type: NounType.Document, data: 'v0', vector: vec(1), metadata: { v: 0 } }) for (let v = 1; v <= 12; v++) await brain.update({ id: a, metadata: { v } }) brain.setRetentionBudget(6000) // ~6 KB — well below the accumulated history - await brain.flush() // group-commit + adaptive auto-compaction under the budget + await brain.flush() - // History was reclaimed (the horizon advanced past the oldest generations)… - expect(generationStoreOf(brain).horizon()).toBeGreaterThan(0) - await expect(brain.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError) - // …but the budget reclaims HISTORY only — the live record is untouched. - expect((await brain.get(a))?.metadata?.v).toBe(12) + // flush() paid durability only: nothing reclaimed, all history readable. + expect(generationStoreOf(brain).horizon()).toBe(0) + const probe = await brain.asOf(1) // readable proves nothing was reclaimed… + await probe.release() // …and MUST be released: a held pin would (correctly) + // protect every newer generation through the close() compaction below. + await brain.close() // ← THE auto-compaction site now + + // close() reclaimed under the budget; live record intact; horizon durable. + const { brain: reopened } = await openFsBrain(dir) + expect(generationStoreOf(reopened).horizon()).toBeGreaterThan(0) + await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError) + expect((await reopened.get(a))?.metadata?.v).toBe(12) }) // ========================================================================== diff --git a/tests/unit/db/generationStore.test.ts b/tests/unit/db/generationStore.test.ts index 611a97d7..5b667415 100644 --- a/tests/unit/db/generationStore.test.ts +++ b/tests/unit/db/generationStore.test.ts @@ -488,6 +488,20 @@ describe('db/GenerationStore', () => { expect(result.removedGenerations).toBe(2) store.release(2) }) + + it('timeBudgetMs bounds a pass; the next pass resumes the same prefix', async () => { + await manyGens(4) + // A spent budget (0ms) stops before reclaiming anything — an early stop + // is a consistent prefix, never a partial generation. + const bounded = await store.compact({ timeBudgetMs: 0 }) + expect(bounded.removedGenerations).toBe(0) + expect(bounded.horizon).toBe(0) + // The next (unbounded) pass picks up exactly where the bounded one + // stopped and completes the same work. + const resumed = await store.compact() + expect(resumed.removedGenerations).toBe(4) + expect(resumed.horizon).toBe(4) + }) }) // ========================================================================== From 70e4bc8a794aaa53dd28f78e3f28e9ec4cdb0644 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sun, 19 Jul 2026 12:52:24 -0700 Subject: [PATCH 16/29] =?UTF-8?q?fix:=20release=20drains=20in-flight=20wri?= =?UTF-8?q?ter-lock=20heartbeat=20=E2=80=94=20no=20phantom=20lock=20after?= =?UTF-8?q?=20unlink?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit clearInterval() stops future heartbeat ticks but not one already in flight: a straggler tick past its ownership guards could land its atomic lock rewrite AFTER releaseWriterLock()'s unlink, re-creating the lock file as a phantom that blocks the next writer until the stale TTL expires (~60s) — the pool-eviction reopen case. Found as an ENOENT heartbeat warning during benchmark teardown; the quiet variant is the harmful one. releaseWriterLock() now awaits the in-flight tick (tracked per tick, self-clearing) before reading/unlinking, so a straggler's write always lands BEFORE the unlink and gets removed with everything else. The heartbeat's ENOENT is also now benign-by-contract (lock or directory removed under us — the next acquire recreates it); other errors stay loud. Pin: straggler-past-guards simulation — lock file absent after close, directory immediately claimable (fails on the undrained code). --- src/storage/adapters/fileSystemStorage.ts | 31 +++++++++++++- .../integration/multi-process-safety.test.ts | 40 +++++++++++++++++++ 2 files changed, 69 insertions(+), 2 deletions(-) diff --git a/src/storage/adapters/fileSystemStorage.ts b/src/storage/adapters/fileSystemStorage.ts index 3c95f9e7..5eb4785a 100644 --- a/src/storage/adapters/fileSystemStorage.ts +++ b/src/storage/adapters/fileSystemStorage.ts @@ -96,6 +96,14 @@ export class FileSystemStorage extends BaseStorage { private static readonly WRITER_STALE_THRESHOLD_MS = 60_000 private writerLockHeartbeat?: NodeJS.Timeout private writerLockInfo?: WriterLockInfo + /** + * The currently-executing heartbeat refresh, if any. `releaseWriterLock()` + * awaits it before unlinking: clearInterval() stops FUTURE ticks but not a + * tick already in flight, and a straggler landing after the unlink would + * RE-CREATE the lock file — a phantom lock blocking the next writer until + * the stale TTL expires (the pool-eviction reopen case). + */ + private writerHeartbeatInFlight?: Promise // Flush-request RPC state. The writer polls `locks/_flush_requests/` for // new `.req` files and emits `.ack` files in `locks/_flush_responses/` after @@ -1880,8 +1888,18 @@ export class FileSystemStorage extends BaseStorage { // Heartbeat — rewrite lastHeartbeat every WRITER_HEARTBEAT_MS so other // processes can tell a live writer from one that crashed without releasing. this.writerLockHeartbeat = setInterval(() => { - this.refreshWriterLockHeartbeat().catch((err) => { - console.warn('[brainy] Failed to refresh writer lock heartbeat:', err) + const tick = this.refreshWriterLockHeartbeat().catch((err) => { + // ENOENT = the lock (or its directory) vanished mid-refresh — the + // store was released or removed under us; the next acquire recreates + // it. Benign by construction; anything else stays loud. + if ((err as NodeJS.ErrnoException)?.code !== 'ENOENT') { + console.warn('[brainy] Failed to refresh writer lock heartbeat:', err) + } + }) + this.writerHeartbeatInFlight = tick.finally(() => { + if (this.writerHeartbeatInFlight === tick) { + this.writerHeartbeatInFlight = undefined + } }) }, FileSystemStorage.WRITER_HEARTBEAT_MS) if (typeof this.writerLockHeartbeat.unref === 'function') { @@ -1913,6 +1931,15 @@ export class FileSystemStorage extends BaseStorage { clearInterval(this.writerLockHeartbeat) this.writerLockHeartbeat = undefined } + // Drain an in-flight heartbeat tick BEFORE unlinking: clearInterval stops + // future ticks only, and a straggler write landing after the unlink would + // re-create the lock as a phantom (blocking the next writer until the + // stale TTL). After the drain, any refresh is either fully landed (we + // unlink its output below) or not started (it sees writerLockInfo + // undefined and returns). + if (this.writerHeartbeatInFlight) { + await this.writerHeartbeatInFlight + } if (!this.writerLockInfo) { return } diff --git a/tests/integration/multi-process-safety.test.ts b/tests/integration/multi-process-safety.test.ts index 0eae92b1..592d7969 100644 --- a/tests/integration/multi-process-safety.test.ts +++ b/tests/integration/multi-process-safety.test.ts @@ -153,6 +153,46 @@ describe('Multi-process safety + read-only mode', () => { expect(err.lockInfo?.pid).toBe(otherPid) }) + it('release drains an in-flight heartbeat — no phantom lock re-created after unlink', async () => { + // The race (8.9.0): clearInterval stops FUTURE heartbeat ticks, but a + // tick already in flight could land its lock rewrite AFTER release's + // unlink — re-creating the lock as a phantom that blocks the next + // writer until the stale TTL. Simulate the in-flight tick explicitly + // and prove release waits for it. + writer = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } }) + await writer.init() + const storage: any = (writer as any).storage + + // An in-flight refresh that is ALREADY PAST its ownership guards + // (captured the lock info before release ran) and lands its atomic + // rewrite slowly — the exact straggler shape; absent the drain it + // writes after the unlink. + const { join: joinPath } = await import('node:path') + const capturedInfo = { ...storage.writerLockInfo } + const lockPath = joinPath(dir, 'locks', '_writer.lock') + const slowTick = (async () => { + await new Promise((r) => setTimeout(r, 100)) + await storage.writeFileAtomic( + lockPath, + JSON.stringify({ ...capturedInfo, lastHeartbeat: new Date().toISOString() }) + ) + })() + storage.writerHeartbeatInFlight = slowTick.catch(() => {}) + + await writer.close() // → releaseWriterLock must drain slowTick first + await slowTick.catch(() => {}) // both paths fully settled either way + writer = null + + const { existsSync } = await import('node:fs') + const { join } = await import('node:path') + expect(existsSync(join(dir, 'locks', '_writer.lock'))).toBe(false) + + // And the directory is immediately claimable — no stale-TTL wait. + const next = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } }) + await expect(next.init()).resolves.toBeUndefined() + await next.close() + }) + it('allows a second in-process writer with a warning (same PID)', async () => { // Two Brainy instances in the same Node process: not the dangerous // cross-process case. Should succeed (with a console warning). From 5cabd784f4e422328c01ab757b928dfb4fc3e194 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sun, 19 Jul 2026 13:35:04 -0700 Subject: [PATCH 17/29] docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First edition of the per-release performance-envelope contract: every number measured against the built dist on stated hardware, never projected. Sub-0.1ms get/related (adjacency O(degree), scale-flat), 1-9ms indexed metadata finds, ~178ms semantic (query embedding dominates), ~167ms durability-priced single-op writes flat across scale, 8-45ms steady-state flush independent of history backlog (the 8.9.0 change). Two weak spots stated honestly: addMany commits per-item today (batched chunk commits belong to the unified-commit roadmap), and pure-JS warm open grows with corpus (4.9s at 10k) — the native accelerator's reason to exist. Refresh rule: any release touching a measured path re-measures in the same release. --- RELEASES.md | 45 +++++++++++++++++++ docs/performance-envelopes.md | 83 +++++++++++++++++++++++++++++++++++ 2 files changed, 128 insertions(+) create mode 100644 docs/performance-envelopes.md diff --git a/RELEASES.md b/RELEASES.md index d1374d5a..7799c6f6 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -8,8 +8,53 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so - Debugging data, query, or storage behaviour - A new Brainy feature is available that you want to adopt +## Removed APIs — 7.x → 8.x (the complete ledger) + +Every public API removed at the 8.0 major, with its sanctioned replacement. If your code +still calls a left-column name on 8.x it throws (or the config key is rejected) — the +replacement is always a one-line change. (Standing contract from 8.9.0 forward: removals +happen only at majors, after ≥1 minor of loud runtime deprecation naming the replacement.) + +| Removed (7.x) | Replacement (8.x) | +|---|---| +| `brain.search(query, k)` | `find({ query })` — semantic; `find({ query, searchMode })` for hybrid | +| `brain.getRelations({...})` | `related(id, opts)` for adjacency; `find({ connected: {...} })` for scoped traversal | +| `brain.neural()` clustering | `find({ vector })` + aggregation `GROUP BY` | +| `Db.search()` | `db.find({ vector })` | +| Pre-8.0 storage path aliases (`directory`, `basePath`, …) | one `storage.path` key (old aliases throw) | +| Reserved keys inside `metadata` bags (silently remapped in 7.x) | top-level params (`subtype`, `visibility`, `confidence`, `weight`, …) — reserved-in-bag throws | +| 7.x COW branches layout (`branches/main/`) | generational MVCC (`asOf()`, `now()`, `db.persist(path)`) — on-disk migration is automatic at first 8.x open | + +The fork/snapshot family (`brain.snapshot()`, `createSnapshot()`, `restoreSnapshot()`) +is sometimes cited as a 7.x removal — those methods never existed on 7.x; the 8.0 Db API +(`asOf`/`persist`/`restore({confirm})`) is their first real implementation. + --- +## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close()) + +The write path stops paying maintenance costs — the last structural piece of the +flush-storm class (a production deployment measured single writes blocked 25–191s behind +history reclaim running inline on flush under memory pressure): + +- **`flush()` never compacts history.** It persists the current window's deltas and + nothing else — its cost no longer depends on history backlog or retention mode, in any + configuration. **`close()` is the auto-compaction site** (time-bounded per pass, ~5s; + an early stop is a consistent prefix and the next pass resumes). +- **`compactHistory()` gains `timeBudgetMs`** — bound your own maintenance windows; the + same resumable-prefix guarantee applies. +- **The documented trade**: a long-lived writer that never closes accumulates history + until its next explicit `compactHistory()`. Predictable writes, explicit maintenance. + If you run bounded retention on an always-on service, schedule a periodic + `compactHistory({ ...caps, timeBudgetMs })` in your maintenance window. +- **New public doc: `docs/performance-envelopes.md`** — measured per-op envelopes + (p50/p95 at stated scales, hardware, and backend, with the measuring script cited). + Refresh rule going forward: any release touching a measured path re-runs that op's + benchmark and updates the envelope in the same release. +- **New in this file: the Removed APIs 7.x→8.x table** (top of this document) — every + removal with its sanctioned replacement, one place, per the engine-currency contract. + Standing from here: removals only at majors, after ≥1 minor of loud runtime deprecation. + ## v8.8.2 — 2026-07-19 (one field-resolution law: reserved-field aggregates stop drifting) Four fixes from a consumer conformance audit, all rooted in the same disease — two field-resolution diff --git a/docs/performance-envelopes.md b/docs/performance-envelopes.md new file mode 100644 index 00000000..d29677e3 --- /dev/null +++ b/docs/performance-envelopes.md @@ -0,0 +1,83 @@ +--- +title: Performance Envelopes +slug: guides/performance-envelopes +public: true +category: guides +template: guide +order: 40 +description: Measured per-operation latency envelopes at stated scales — what to expect, on what hardware, and exactly how each number was produced. +next: + - guides/find-limits +--- + +# Performance Envelopes + +Every number on this page is **measured, never projected** — produced by the script +cited at the bottom, against the built package (the artifact you install), on the stated +hardware. Each entry says what was measured, at what scale, on which storage backend. +When a release touches a measured path, that operation is re-measured and this page +updates in the same release. + +Two scopes to keep straight: + +- **These envelopes are the pure-JS engine** (no native accelerator registered) on + filesystem storage. This is the floor every deployment gets from `npm install` alone. +- **Accelerated deployments** (the optional native provider) publish their own numbers — + this page never claims them. + +## Read operations + +Reads are where the architecture pays off: after the write path has done its indexing +work, queries answer from purpose-built indexes without scanning. + +| Operation | 1,000 entities | 10,000 entities | Notes | +|---|---|---|---| +| `get(id)` (warm) | p50 < 0.1ms | p50 < 0.1ms | served from cache/metadata index | +| `find` (metadata: indexed equality + range, limit 100) | p50 1.0ms · p95 1.8ms | p50 7.0ms · p95 8.9ms | column-store bitmap paths | +| `related(id)` (per-node adjacency) | p50 < 0.1ms · p95 0.2ms | p50 < 0.1ms | LSM adjacency index — O(degree), scale-independent | +| `find` (semantic: embed + HNSW, 1k docs) | p50 178ms · p95 393ms | — | dominated by WASM query embedding (measured on a machine under concurrent load — treat the p95 as an upper bound); the vector search itself is single-digit ms | + +## Write operations + +Under Model-B **every write is its own durable generation** — a single-op `add` pays +serialization, before-image staging, and fsync before it acks. That durability is priced +into the write path visibly, by design: + +| Operation | 1,000 entities | 10,000 entities | Notes | +|---|---|---|---| +| `add` (single-op) | p50 167ms · p95 171ms | p50 165ms · p95 172ms | full durable generation per write — flat across scale | +| `addMany` (bulk) | ~163ms/entity | ~187ms/entity | **currently per-item commits** — see the honest note below | +| `relateMany` | ~0.8ms/edge | ~0.9ms/edge | edges batch efficiently today | +| `flush` (steady-state, 1 pending write) | p50 8ms · p95 10ms | p50 45ms · p95 52ms | durability-only since 8.9.0 — cost no longer depends on history backlog or retention mode | + +**The honest note on bulk writes:** `addMany` today commits each item as its own +generation (the same durability as single-op `add`, serialized by the single-writer +lock), so bulk-load cost is N × single-op cost. Batched chunk commits (one generation +and one fsync window per chunk, as `removeMany` already does) are designed into the +unified-commit work on the current roadmap. Until that ships, size bulk imports +accordingly — 10k entities is minutes, not seconds, on filesystem storage. + +## Open / close + +| Operation | 1,000 entities | 10,000 entities | Notes | +|---|---|---|---| +| `open` (empty store) | ~560ms | ~190ms | includes embedder initialization | +| `open` (warm, populated, clean shutdown) | 763ms | 4.9s | pure-JS vector index load dominates and grows with entity count; the native accelerator exists precisely to remove this | +| `close` | bounded | bounded | auto-compaction pass is time-bounded (~5s max) since 8.9.0 | + +A store that was NOT cleanly closed pays index rebuilds on top of the warm-open +number (tens of seconds at 10k) — clean shutdown is worth engineering for. + +## How these were produced + +- **Hardware**: Intel Core i9-14900HX (32 threads), 62GB RAM, NVMe, Linux, Node v22. +- **Backend**: `storage: { type: 'filesystem' }`, pure JS (no native providers). +- **Embeddings**: deterministic stub for non-semantic ops (isolates engine cost); + the real WASM embedder for the semantic row (that's what you'll run). +- **Method**: p50/p95 over 50–200 samples per op against the built `dist/`; + the measuring script ships in the repo history and re-runs per release. + +Numbers on different hardware will differ; the *shape* (sub-2ms indexed reads, +~160ms embedding-bound semantic queries, durability-priced writes) is the envelope +you should hold your deployment against. If your measurements diverge from these +shapes by an order of magnitude, something is wrong — file it. From d08679fc843d31add8adc4d23f3b6e4190847423 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sun, 19 Jul 2026 14:02:25 -0700 Subject: [PATCH 18/29] chore(release): 8.9.0 --- CHANGELOG.md | 7 +++++++ package-lock.json | 4 ++-- package.json | 2 +- 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b4f1a1c..fd0de54e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. +### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19) + +- docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78) +- fix: release drains in-flight writer-lock heartbeat — no phantom lock after unlink (70e4bc8) +- feat: flush() never compacts — history maintenance moves to close() with bounded passes (300d9f2) + + ### [8.8.2](https://github.com/soulcraftlabs/brainy/compare/v8.8.1...v8.8.2) (2026-07-19) - fix: one field-resolution law across aggregation hooks, source.where, removeMany, and find() spellings (945d92d) diff --git a/package-lock.json b/package-lock.json index c1d62fc9..fb9262e9 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraft/brainy", - "version": "8.8.2", + "version": "8.9.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraft/brainy", - "version": "8.8.2", + "version": "8.9.0", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index b43633ac..7366ce98 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraft/brainy", - "version": "8.8.2", + "version": "8.9.0", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", "module": "dist/index.js", From f8e6da2b6603e52e12ea35983c4e7a122921d790 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sun, 19 Jul 2026 14:54:36 -0700 Subject: [PATCH 19/29] =?UTF-8?q?feat:=20scanFacts=20liveness=20contract?= =?UTF-8?q?=20=E2=80=94=20first=20batch=20or=20loud=20failure=20within=20a?= =?UTF-8?q?=20documented=20bound?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage-2 D1 contract item (co-frozen): a fact scan may be slow, never silent. batches() now races its FIRST pull against SCANFACTS_FIRST_BATCH_MS (10s, exported; test-overridable) — a wedged or unreadably slow store produces a loud abort naming the contract instead of a consumer hanging indistinguishably from progress (the production shape: a heal against a generations-backlogged brain wedged silently on the first segment read). Only the first pull is raced: the bound is time-to-first-batch (proof the producer is alive), not per-batch pacing, and it runs only while a pull is pending — consumer think-time between pulls never counts against the producer (pinned). Three pins: wedged-store loud failure within the bound, healthy scan untouched end-to-end, slow-consumer immunity. --- src/db/factLog.ts | 52 ++++++++++++++++++++++++++++++++-- src/index.ts | 1 + tests/unit/db/fact-log.test.ts | 48 +++++++++++++++++++++++++++++++ 3 files changed, 99 insertions(+), 2 deletions(-) diff --git a/src/db/factLog.ts b/src/db/factLog.ts index 4c5e95fd..94e79700 100644 --- a/src/db/factLog.ts +++ b/src/db/factLog.ts @@ -102,12 +102,26 @@ export interface FactScanBatch { segmentId: string } +/** + * Liveness bound on a scan's FIRST batch (Stage-2 co-freeze, D1 contract): + * `batches()` must yield its first batch — or fail loudly — within this many + * ms of the first pull. A backlogged or damaged store may be SLOW, but it may + * never be SILENT: a consumer awaiting the first batch is otherwise + * indistinguishable from a wedge (the exact failure shape a production heal + * hit against a generations-backlogged brain). + */ +export const SCANFACTS_FIRST_BATCH_MS = 10_000 + /** The telemetry a scan OPEN returns (frozen shape). */ export interface FactScanHandle { headGeneration: number segmentCount: number approxFactCount: number - /** Ordered batches; a detected gap aborts LOUDLY, never a silent skip. */ + /** + * Ordered batches; a detected gap aborts LOUDLY, never a silent skip. + * Liveness contract: the FIRST batch resolves or rejects within + * {@link SCANFACTS_FIRST_BATCH_MS} of the first pull — never a silent hang. + */ batches: () => AsyncGenerator /** Close telemetry — the invariant cross-check, valid after iteration ends. */ summary: () => { factsYielded: number; segmentsRead: number } @@ -440,6 +454,8 @@ export class FactLog { toGeneration?: number kinds?: Array<'noun' | 'verb'> batchSize?: number + /** Test override for the first-batch liveness bound (default {@link SCANFACTS_FIRST_BATCH_MS}). */ + firstBatchTimeoutMs?: number }): FactScanHandle { const from = options?.fromGeneration ?? 1 const to = options?.toGeneration ?? this.head @@ -514,11 +530,43 @@ export class FactLog { } } + // Liveness wrapper: the FIRST pull races the contract deadline. Only the + // first — the bound is time-to-first-batch (proof the producer is alive), + // not per-batch pacing; and it runs only while a pull is actually pending, + // so consumer think-time between pulls never counts against the producer. + const firstBatchTimeoutMs = options?.firstBatchTimeoutMs ?? SCANFACTS_FIRST_BATCH_MS + async function* batchesWithLiveness(this: void): AsyncGenerator { + const inner = batches() + let timer: NodeJS.Timeout | undefined + try { + const deadline = new Promise((_, reject) => { + timer = setTimeout( + () => + reject( + new Error( + `fact log: scanFacts produced no first batch within ${firstBatchTimeoutMs}ms ` + + `(liveness contract) — the store is wedged or unreadably slow; aborting scan LOUDLY ` + + `instead of hanging the consumer.` + ) + ), + firstBatchTimeoutMs + ) + timer.unref?.() + }) + const first = await Promise.race([inner.next(), deadline]) + if (first.done) return + yield first.value + } finally { + clearTimeout(timer) + } + yield* inner + } + return { headGeneration: this.head, segmentCount: segments.length + (tailSnapshot.length > 0 ? 1 : 0), approxFactCount, - batches, + batches: batchesWithLiveness, summary: () => ({ factsYielded, segmentsRead }) } } diff --git a/src/index.ts b/src/index.ts index ee01d885..b978b9fd 100644 --- a/src/index.ts +++ b/src/index.ts @@ -213,6 +213,7 @@ export type { CommitFact, FactOp, FactScanBatch, + SCANFACTS_FIRST_BATCH_MS, FactScanHandle } from './db/factLog.js' // The generalized family stamp — which source generation a projection diff --git a/tests/unit/db/fact-log.test.ts b/tests/unit/db/fact-log.test.ts index abce2dc9..f1c226cc 100644 --- a/tests/unit/db/fact-log.test.ts +++ b/tests/unit/db/fact-log.test.ts @@ -186,4 +186,52 @@ describe('fact log — round-trip, framing, reconcile, rotation, scan', () => { await log.sync() expect(log.segmentPaths()).toEqual([]) // only a tail exists — nothing sealed }) + + describe('scanFacts liveness contract (Stage-2 D1)', () => { + it('a wedged store fails LOUDLY within the first-batch bound — never a silent hang', async () => { + // Force a sealed segment (tiny rotateBytes) so the scan must READ from + // storage, then wedge that read: the exact production shape (a + // backlogged brain whose segment read never returned). + const mem: any = new MemoryStorage() + await mem.init() + const wedgeable = new FactLog(mem, { rotateBytes: 1 }) + await wedgeable.open(0) + await wedgeable.append(fact(1)) + await wedgeable.append(fact(2)) // second append rotates → seg 1 sealed + await wedgeable.sync() + + const realRead = mem.readRawBytes.bind(mem) + mem.readRawBytes = (p: string) => + p.includes('facts/seg-') ? new Promise(() => {}) : realRead(p) // hangs forever + + const scan = wedgeable.scanFacts({ firstBatchTimeoutMs: 200 }) + const started = Date.now() + await expect(scan.batches().next()).rejects.toThrow(/no first batch within 200ms/) + expect(Date.now() - started).toBeLessThan(5_000) // bound held, not a hang + }) + + it('a healthy scan is unaffected — first batch well inside the bound, all facts delivered', async () => { + for (let g = 1; g <= 5; g++) await log.append(fact(g)) + await log.sync() + const scan = log.scanFacts({ batchSize: 2 }) + const all: CommitFact[] = [] + for await (const b of scan.batches()) all.push(...b.facts) + expect(all.map((f) => f.generation)).toEqual([1, 2, 3, 4, 5]) + expect(scan.summary().factsYielded).toBe(5) + }) + + it('consumer think-time between pulls never counts against the producer', async () => { + for (let g = 1; g <= 4; g++) await log.append(fact(g)) + await log.sync() + // Bound tighter than the consumer's pause: only the FIRST pull is + // raced, so a slow consumer after batch 1 must not trip the deadline. + const gen = log.scanFacts({ batchSize: 2, firstBatchTimeoutMs: 150 }).batches() + const first = await gen.next() + expect(first.done).toBe(false) + await new Promise((r) => setTimeout(r, 400)) // dawdle past the bound + const second = await gen.next() + expect(second.done).toBe(false) + expect((await gen.next()).done).toBe(true) + }) + }) }) From d8acb3776b2e64db79332cb70fb3b0d7588cef99 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sun, 19 Jul 2026 15:14:27 -0700 Subject: [PATCH 20/29] =?UTF-8?q?feat:=20generation-segment=20store=20?= =?UTF-8?q?=E2=80=94=20the=20D1+D3=20packed-tier=20file=20format?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First stage of the co-frozen D1+D3+repacking unit: the format core, self-contained under _generations/segments/. - seg-.bgs: append-once packs of consecutive generations (magic BGS1; frame = u32 len + u32 crc32c + msgpack [generation, timestamp, delta, records, flags]; flags reserves compressed-payload evolution without a format break). Sealed segments are immutable — fold refuses overlap with sealed ranges. - seg-.idx: DERIVED sidecar (per-generation frame offsets + per-id generation postings + checksums); lost/corrupt sidecars rebuild from their segment loudly; a damaged segment (frame CRC mismatch) fails loudly, never serves wrong bytes. - manifest.json: the one discovery path — open() reads it and never lists the packed backlog (the scan-wedge class's cure); refuses a newer manifest version rather than serving partial history. - D3 semantics: dropSegmentsBelow reclaims WHOLE segments at boundaries only and bumps compactedBelow durably; archival-profile enforcement stays with the caller per the co-freeze. - D8 rider: digestThroughPacked(g) — deterministic crc32c chain over sealed-segment checksums (+ frame-level prefix mid-segment), O(segments), reopen-stable. Also fixes a cross-adapter contract bug the suite caught: memory storage's deleteObjectFromPath ignored the raw-bytes store, so deleteRawObject on a raw-bytes path (fact-log or segment files) silently no-op'd — deletes now match filesystem unlink semantics. Six pins. Wiring into GenerationStore (two-tier reads, the repacker, cold-open manifest path) lands with the rest of the unit before its release; cortex's fact-record/stamp shapes reconcile the sidecar keying when they post. --- src/db/generationSegments.ts | 459 ++++++++++++++++++++++ src/storage/adapters/memoryStorage.ts | 5 + tests/unit/db/generation-segments.test.ts | 150 +++++++ 3 files changed, 614 insertions(+) create mode 100644 src/db/generationSegments.ts create mode 100644 tests/unit/db/generation-segments.test.ts diff --git a/src/db/generationSegments.ts b/src/db/generationSegments.ts new file mode 100644 index 00000000..0c14b60c --- /dev/null +++ b/src/db/generationSegments.ts @@ -0,0 +1,459 @@ +/** + * @module db/generationSegments + * @description The generation-segment store — Stage-2 D1+D3+repacking's file + * format (co-frozen 2026-07-19; design: the d1-d3-repacking spec). + * + * Packs CONSECUTIVE cold generations' record-sets (before-images + delta) + * into append-once segment files with derived sidecar indexes, so history + * scales in SEGMENTS (tens) instead of FILES-PER-GENERATION (hundreds of + * thousands), and cold-open reads ONE manifest instead of listing the + * backlog. Layout under `_generations/segments/`: + * + * - `seg-.bgs` — magic "BGS1", then one frame per + * generation: `u32 payloadLen | u32 crc32c | msgpack payload`. Payload is + * POSITIONAL: `[generation, timestamp, delta, records[], flags]` with + * records `[kindByte, id, record]`. `flags` reserves encoding evolution + * (bit 0 = compressed payload — v1 always 0; a future writer upgrade, + * never a format break). Sealed segments are IMMUTABLE — the fact log's + * own law, generalized. + * - `seg-.idx` — DERIVED sidecar (msgpack): per-generation frame + * offsets (point reads = one ranged read, never a listing) + per-id + * generation postings (per-id chain rebuilds read only what they need). + * Corrupt/missing → rebuilt from its segment in one sequential read, + * loudly. + * - `manifest.json` — the segment catalogue + `compactedBelow` (D3's + * horizon marker). Cold-open reads THIS; the packed backlog is never + * listed. + * + * D3 semantics carried here: bounded-retention reclaim drops WHOLE segments + * at boundaries (O(1) per segment, no rewrite); under the archival profile + * (`retention: 'all'`) nothing here is ever dropped — folding is the only + * transform (re-representation, never deletion). + */ + +import { encode as msgpackEncode, decode as msgpackDecode } from '@msgpack/msgpack' +import { crc32c } from '../utils/crc32c.js' +import type { FactLogStorage } from './factLog.js' +import { prodLog } from '../utils/logger.js' + +/** Directory for segment files + manifest, under the generations prefix. */ +export const SEGMENTS_PREFIX = '_generations/segments' + +/** Target sealed-segment size (co-freeze proposal; tunable on evidence). */ +export const SEGMENT_TARGET_BYTES = 64 * 1024 * 1024 + +const MAGIC = new TextEncoder().encode('BGS1') +const FRAME_PREFIX_BYTES = 8 // u32 payloadLen + u32 crc32c +const MANIFEST_PATH = `${SEGMENTS_PREFIX}/manifest.json` + +/** One generation's fold input — exactly what the live tier holds for it. */ +export interface FoldGeneration { + generation: number + timestamp: number + /** The tx.json delta object, carried verbatim. */ + delta: unknown + /** The before-image record-set (empty for record-less generations). */ + records: Array<{ kind: 'noun' | 'verb'; id: string; record: unknown }> +} + +/** Manifest entry for one sealed segment. */ +export interface SegmentMeta { + file: string + firstGeneration: number + lastGeneration: number + frames: number + bytes: number + /** crc32c of the full segment byte stream — the digest chain's link. */ + checksum: number +} + +interface SegmentManifest { + version: 1 + compactedBelow: number + segments: SegmentMeta[] +} + +interface SidecarIndex { + version: 1 + /** [generation, frameOffset, frameLen] ascending by generation. */ + generations: Array<[number, number, number]> + /** `${kindByte}:${id}` → ascending generations holding a record for it. */ + ids: Record +} + +const segmentFileName = (firstGeneration: number): string => + `seg-${String(firstGeneration).padStart(20, '0')}.bgs` +const sidecarFileName = (firstGeneration: number): string => + `seg-${String(firstGeneration).padStart(20, '0')}.idx` + +/** + * The generation-segment store. Owns the packed tier ONLY — the live + * per-generation tier and the routing between tiers belong to + * `GenerationStore`. All mutating entry points here are called under the + * generation store's commit mutex. + */ +export class GenerationSegmentStore { + private readonly storage: FactLogStorage + private manifest: SegmentManifest = { version: 1, compactedBelow: 0, segments: [] } + /** Sidecar cache — segments are immutable, so entries never invalidate. */ + private readonly sidecars = new Map() + + constructor(storage: FactLogStorage) { + this.storage = storage + } + + /** Load the manifest (ONE read — never a directory listing). */ + async open(): Promise { + const raw = (await this.storage.readRawObject(MANIFEST_PATH)) as SegmentManifest | null + if (raw) { + if (raw.version !== 1) { + throw new Error( + `[GenerationSegments] manifest version ${String(raw.version)} is newer than this ` + + `engine understands — refusing to serve partial history. Upgrade the engine.` + ) + } + this.manifest = raw + } + } + + /** The packed tier's catalogue (ascending, immutable snapshot). */ + segments(): readonly SegmentMeta[] { + return this.manifest.segments + } + + /** D3's horizon marker: generations below this were reclaimed (bounded profiles only). */ + compactedBelow(): number { + return this.manifest.compactedBelow + } + + /** The covering sealed segment for `gen`, or null if it lives outside the packed tier. */ + private coveringSegment(gen: number): SegmentMeta | null { + // Manifest is ascending and ranges never overlap — binary search. + const segs = this.manifest.segments + let lo = 0 + let hi = segs.length - 1 + while (lo <= hi) { + const mid = (lo + hi) >> 1 + const s = segs[mid] + if (gen < s.firstGeneration) hi = mid - 1 + else if (gen > s.lastGeneration) lo = mid + 1 + else return s + } + return null + } + + /** True when `gen` is packed (readable from this tier). */ + hasGeneration(gen: number): boolean { + return this.coveringSegment(gen) !== null + } + + /** + * Fold consecutive generations into ONE new sealed segment + sidecar and + * append it to the manifest atomically. Caller guarantees: `gens` is + * ascending, contiguous with the packed tier (first = last packed + 1 when + * segments exist), and already durable in the live tier. Crash between the + * segment write and the caller's live-tier delete leaves a DUPLICATE + * representation — resolved live-tier-wins by the reader; never a gap. + */ + async fold(gens: FoldGeneration[]): Promise { + if (gens.length === 0) { + throw new Error('[GenerationSegments] fold() requires at least one generation') + } + for (let i = 1; i < gens.length; i++) { + if (gens[i].generation <= gens[i - 1].generation) { + throw new Error('[GenerationSegments] fold() input must be strictly ascending') + } + } + const last = this.manifest.segments[this.manifest.segments.length - 1] + if (last && gens[0].generation <= last.lastGeneration) { + throw new Error( + `[GenerationSegments] fold() overlaps the packed tier: ${gens[0].generation} ≤ ` + + `sealed ${last.lastGeneration} — segments are immutable, never rewritten` + ) + } + + const first = gens[0].generation + const file = segmentFileName(first) + const sidecar: SidecarIndex = { version: 1, generations: [], ids: {} } + + // Encode all frames, tracking offsets for the sidecar. + const parts: Uint8Array[] = [MAGIC] + let offset = MAGIC.length + for (const g of gens) { + const payload = msgpackEncode([ + g.generation, + g.timestamp, + g.delta, + g.records.map((r) => [r.kind === 'noun' ? 0 : 1, r.id, r.record]), + 0 // flags: v1 = uncompressed + ]) + const frame = new Uint8Array(FRAME_PREFIX_BYTES + payload.length) + const view = new DataView(frame.buffer) + view.setUint32(0, payload.length, true) + view.setUint32(4, crc32c(payload), true) + frame.set(payload, FRAME_PREFIX_BYTES) + sidecar.generations.push([g.generation, offset, frame.length]) + for (const r of g.records) { + const key = `${r.kind === 'noun' ? 0 : 1}:${r.id}` + ;(sidecar.ids[key] ??= []).push(g.generation) + } + parts.push(frame) + offset += frame.length + } + const total = parts.reduce((n, p) => n + p.length, 0) + const bytes = new Uint8Array(total) + let at = 0 + for (const p of parts) { + bytes.set(p, at) + at += p.length + } + + const meta: SegmentMeta = { + file, + firstGeneration: first, + lastGeneration: gens[gens.length - 1].generation, + frames: gens.length, + bytes: total, + checksum: crc32c(bytes) + } + + // Durability order: segment + sidecar fsync'd BEFORE the manifest names + // them (a crash before the manifest = invisible orphan files, harmless); + // manifest last, atomically. + const segPath = `${SEGMENTS_PREFIX}/${file}` + const idxPath = `${SEGMENTS_PREFIX}/${sidecarFileName(first)}` + await this.storage.writeRawBytes(segPath, bytes) + await this.storage.writeRawBytes(idxPath, msgpackEncode(sidecar)) + await this.storage.syncRawObjects([segPath, idxPath]) + const next: SegmentManifest = { + ...this.manifest, + segments: [...this.manifest.segments, meta] + } + await this.storage.writeRawObject(MANIFEST_PATH, next) + await this.storage.syncRawObjects([MANIFEST_PATH]) + this.manifest = next + this.sidecars.set(file, sidecar) + return meta + } + + /** Load (or rebuild, loudly) a segment's sidecar. */ + private async sidecarFor(meta: SegmentMeta): Promise { + const cached = this.sidecars.get(meta.file) + if (cached) return cached + const idxPath = `${SEGMENTS_PREFIX}/${sidecarFileName(meta.firstGeneration)}` + const raw = await this.storage.readRawBytes(idxPath) + if (raw) { + try { + const idx = msgpackDecode(raw) as SidecarIndex + if (idx.version === 1) { + this.sidecars.set(meta.file, idx) + return idx + } + } catch { + // fall through to rebuild + } + } + // Sidecars are DERIVED: rebuild from the segment, loudly — never serve + // wrong offsets silently. + prodLog.warn( + `[GenerationSegments] sidecar for ${meta.file} missing or unreadable — rebuilding from the segment` + ) + const rebuilt = await this.rebuildSidecar(meta) + await this.storage.writeRawBytes(idxPath, msgpackEncode(rebuilt)) + this.sidecars.set(meta.file, rebuilt) + return rebuilt + } + + /** One sequential read of the segment → a fresh sidecar. Verifies every frame CRC. */ + private async rebuildSidecar(meta: SegmentMeta): Promise { + const frames = await this.readAllFrames(meta) + const idx: SidecarIndex = { version: 1, generations: [], ids: {} } + for (const f of frames) { + idx.generations.push([f.generation, f.offset, f.frameLen]) + for (const r of f.records) { + const key = `${r.kind === 'noun' ? 0 : 1}:${r.id}` + ;(idx.ids[key] ??= []).push(f.generation) + } + } + return idx + } + + private decodeFrame( + payload: Uint8Array + ): { generation: number; timestamp: number; delta: unknown; records: FoldGeneration['records'] } { + const [generation, timestamp, delta, rawRecords] = msgpackDecode(payload) as [ + number, + number, + unknown, + Array<[number, string, unknown]>, + number + ] + return { + generation, + timestamp, + delta, + records: rawRecords.map(([kindByte, id, record]) => ({ + kind: kindByte === 0 ? ('noun' as const) : ('verb' as const), + id, + record + })) + } + } + + private async readAllFrames(meta: SegmentMeta): Promise< + Array & { offset: number; frameLen: number }> + > { + const bytes = await this.storage.readRawBytes(`${SEGMENTS_PREFIX}/${meta.file}`) + if (!bytes) { + throw new Error( + `[GenerationSegments] sealed segment ${meta.file} is MISSING — packed history is damaged; ` + + `refusing to continue silently` + ) + } + const out: Array & { offset: number; frameLen: number }> = [] + let at = MAGIC.length + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength) + while (at + FRAME_PREFIX_BYTES <= bytes.length) { + const payloadLen = view.getUint32(at, true) + const crc = view.getUint32(at + 4, true) + const payload = bytes.subarray(at + FRAME_PREFIX_BYTES, at + FRAME_PREFIX_BYTES + payloadLen) + if (payload.length !== payloadLen || crc32c(payload) !== crc) { + throw new Error( + `[GenerationSegments] frame CRC mismatch in ${meta.file} at offset ${at} — ` + + `packed history is damaged; refusing to serve it` + ) + } + out.push({ ...this.decodeFrame(payload), offset: at, frameLen: FRAME_PREFIX_BYTES + payloadLen }) + at += FRAME_PREFIX_BYTES + payloadLen + } + return out + } + + /** Read one packed generation's frame via its sidecar offset (one ranged read). */ + private async readFrame( + gen: number + ): Promise | null> { + const meta = this.coveringSegment(gen) + if (!meta) return null + const idx = await this.sidecarFor(meta) + // generations ascending → binary search. + const gens = idx.generations + let lo = 0 + let hi = gens.length - 1 + while (lo <= hi) { + const mid = (lo + hi) >> 1 + if (gens[mid][0] < gen) lo = mid + 1 + else if (gens[mid][0] > gen) hi = mid - 1 + else { + const [, offset, frameLen] = gens[mid] + const bytes = await this.storage.readRawBytes(`${SEGMENTS_PREFIX}/${meta.file}`) + if (!bytes) { + throw new Error(`[GenerationSegments] sealed segment ${meta.file} is MISSING`) + } + const frame = bytes.subarray(offset, offset + frameLen) + const view = new DataView(frame.buffer, frame.byteOffset, frame.byteLength) + const payloadLen = view.getUint32(0, true) + const crc = view.getUint32(4, true) + const payload = frame.subarray(FRAME_PREFIX_BYTES, FRAME_PREFIX_BYTES + payloadLen) + if (payload.length !== payloadLen || crc32c(payload) !== crc) { + throw new Error( + `[GenerationSegments] frame CRC mismatch for generation ${gen} in ${meta.file} — ` + + `packed history is damaged; refusing to serve it` + ) + } + return this.decodeFrame(payload) + } + } + // In the covering range but not present: the packed tier is dense by + // construction (fold packs every generation it is handed, including + // record-less ones) — absence inside a sealed range is damage. + throw new Error( + `[GenerationSegments] generation ${gen} is inside sealed segment ${meta.file}'s declared ` + + `range but has no frame — packed history is damaged` + ) + } + + /** The packed tier's delta for `gen` (null = not packed). */ + async readDelta(gen: number): Promise<{ delta: unknown; timestamp: number } | null> { + const frame = await this.readFrame(gen) + return frame ? { delta: frame.delta, timestamp: frame.timestamp } : null + } + + /** The packed tier's full record-set for `gen` (null = not packed). */ + async readRecords(gen: number): Promise { + const frame = await this.readFrame(gen) + return frame ? frame.records : null + } + + /** One packed before-image (null = not packed OR no record for the id in that generation). */ + async readRecord(gen: number, kind: 'noun' | 'verb', id: string): Promise { + const frame = await this.readFrame(gen) + if (!frame) return null + const hit = frame.records.find((r) => r.kind === kind && r.id === id) + return hit ? hit.record : null + } + + /** + * D3 reclaim: drop WHOLE segments whose lastGeneration < `belowGeneration` + * and bump `compactedBelow`. Partial segments are never dropped — the + * boundary waits. NEVER called under the archival profile (the caller + * enforces retention semantics; this method only executes boundary drops). + */ + async dropSegmentsBelow(belowGeneration: number): Promise<{ dropped: number; compactedBelow: number }> { + const keep: SegmentMeta[] = [] + const drop: SegmentMeta[] = [] + for (const s of this.manifest.segments) { + ;(s.lastGeneration < belowGeneration ? drop : keep).push(s) + } + if (drop.length === 0) { + return { dropped: 0, compactedBelow: this.manifest.compactedBelow } + } + const compactedBelow = Math.max( + this.manifest.compactedBelow, + drop[drop.length - 1].lastGeneration + 1 + ) + // Manifest first (the drop is authoritative once named), then bytes — + // a crash between leaves orphan segment files invisible to the manifest, + // harmless and re-collectable. + const next: SegmentManifest = { ...this.manifest, compactedBelow, segments: keep } + await this.storage.writeRawObject(MANIFEST_PATH, next) + await this.storage.syncRawObjects([MANIFEST_PATH]) + this.manifest = next + for (const s of drop) { + await this.storage.deleteRawObject(`${SEGMENTS_PREFIX}/${s.file}`) + await this.storage.deleteRawObject(`${SEGMENTS_PREFIX}/${sidecarFileName(s.firstGeneration)}`) + this.sidecars.delete(s.file) + } + return { dropped: drop.length, compactedBelow } + } + + /** + * D8 rider — the packed portion of `generationDigest(g)`: a deterministic + * crc32c chain over sealed-segment checksums fully below `g`, plus the + * frame CRC of `g`'s own frame when `g` is mid-segment. O(segments), not + * O(generations); identical history ⇒ identical digest on any machine. + * The live-tier portion is composed by the caller. + */ + async digestThroughPacked(g: number): Promise { + let digest = 0 + let covered = false + for (const s of this.manifest.segments) { + if (s.lastGeneration <= g) { + digest = crc32c(new TextEncoder().encode(`${digest}:${s.checksum}`)) + if (s.lastGeneration === g) covered = true + } else if (s.firstGeneration <= g) { + // g is mid-segment: chain the partial prefix via g's frame CRC. + const frame = await this.readFrame(g) + if (frame === null) return null + const idx = await this.sidecarFor(s) + const upTo = idx.generations.filter(([gen]) => gen <= g) + for (const [gen, offset, frameLen] of upTo) { + digest = crc32c(new TextEncoder().encode(`${digest}:${gen}:${offset}:${frameLen}`)) + } + covered = true + break + } + } + return covered || this.manifest.segments.length > 0 ? digest : null + } +} diff --git a/src/storage/adapters/memoryStorage.ts b/src/storage/adapters/memoryStorage.ts index bab9d4d9..1b1f412e 100644 --- a/src/storage/adapters/memoryStorage.ts +++ b/src/storage/adapters/memoryStorage.ts @@ -133,6 +133,11 @@ export class MemoryStorage extends BaseStorage { */ protected async deleteObjectFromPath(path: string): Promise { this.objectStore.delete(path) + // Filesystem parity: on disk, objects and raw BYTE files are both just + // files — unlink removes whichever exists. Without this, deleteRawObject + // on a raw-bytes path (fact-log/generation segments) silently no-ops on + // memory storage: the delete "succeeds" and the bytes remain. + this.rawBytesStore.delete(path) } /** diff --git a/tests/unit/db/generation-segments.test.ts b/tests/unit/db/generation-segments.test.ts new file mode 100644 index 00000000..27ab85cb --- /dev/null +++ b/tests/unit/db/generation-segments.test.ts @@ -0,0 +1,150 @@ +/** + * @module tests/unit/db/generation-segments + * @description The generation-segment store (Stage-2 D1+D3 file format). + * Laws: (1) fold → read round-trips deltas and records byte-faithfully via + * sidecar point-reads; (2) the manifest is the ONLY discovery path — reopen + * reads one file, never a listing; (3) a lost/corrupt sidecar rebuilds from + * its segment loudly, a damaged SEGMENT fails loudly (never silent wrong + * data); (4) D3 reclaim drops whole segments only and bumps compactedBelow; + * (5) the packed digest is deterministic across reopen; (6) immutability — + * fold refuses overlap with sealed ranges. + */ +import { describe, it, expect, beforeEach } from 'vitest' +import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js' +import { + GenerationSegmentStore, + SEGMENTS_PREFIX, + type FoldGeneration +} from '../../../src/db/generationSegments.js' + +const UUID = (n: number): string => `00000000-0000-4000-8000-${String(n).padStart(12, '0')}` + +const gen = (g: number, recordCount = 2): FoldGeneration => ({ + generation: g, + timestamp: 1_700_000_000_000 + g, + delta: { generation: g, nouns: [UUID(g)], verbs: [], bytes: 123 + g }, + records: Array.from({ length: recordCount }, (_, i) => ({ + kind: (i % 2 === 0 ? 'noun' : 'verb') as 'noun' | 'verb', + id: UUID(g * 100 + i), + record: { metadata: { noun: 'document', v: g }, vector: { v: [g, i] } } + })) +}) + +describe('db/GenerationSegmentStore — the D1+D3 packed tier', () => { + let storage: MemoryStorage + let store: GenerationSegmentStore + + beforeEach(async () => { + storage = new MemoryStorage() + await storage.init() + store = new GenerationSegmentStore(storage as any) + await store.open() + }) + + it('fold → read round-trips deltas and records via sidecar point-reads', async () => { + const meta = await store.fold([gen(1), gen(2), gen(3)]) + expect(meta).toMatchObject({ firstGeneration: 1, lastGeneration: 3, frames: 3 }) + expect(meta.checksum).toBeGreaterThan(0) + + expect(store.hasGeneration(2)).toBe(true) + expect(store.hasGeneration(4)).toBe(false) + + const d2 = await store.readDelta(2) + expect(d2?.delta).toEqual({ generation: 2, nouns: [UUID(2)], verbs: [], bytes: 125 }) + expect(d2?.timestamp).toBe(1_700_000_000_002) + + const records = await store.readRecords(3) + expect(records).toHaveLength(2) + expect(records![0]).toEqual({ + kind: 'noun', + id: UUID(300), + record: { metadata: { noun: 'document', v: 3 }, vector: { v: [3, 0] } } + }) + // Point read by id, both kinds. + expect(await store.readRecord(3, 'verb', UUID(301))).toEqual({ + metadata: { noun: 'document', v: 3 }, + vector: { v: [3, 1] } + }) + expect(await store.readRecord(3, 'noun', UUID(999))).toBeNull() + }) + + it('reopen discovers everything from the manifest alone — no listing', async () => { + await store.fold([gen(1), gen(2)]) + await store.fold([gen(3), gen(4)]) + + const reopened = new GenerationSegmentStore(storage as any) + await reopened.open() + expect(reopened.segments()).toHaveLength(2) + expect(reopened.hasGeneration(4)).toBe(true) + expect((await reopened.readDelta(1))?.timestamp).toBe(1_700_000_000_001) + }) + + it('a lost sidecar rebuilds from its segment; a damaged segment fails LOUDLY', async () => { + const meta = await store.fold([gen(1), gen(2)]) + const idxPath = `${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.idx` + await storage.deleteRawObject(idxPath) + + const reopened = new GenerationSegmentStore(storage as any) + await reopened.open() + // Rebuild path: still serves correct data. + expect((await reopened.readRecords(2))!).toHaveLength(2) + + // Now damage the SEGMENT itself: flip a payload byte → CRC mismatch, loud. + const segPath = `${SEGMENTS_PREFIX}/${meta.file}` + const bytes = (await storage.readRawBytes(segPath))! + bytes[bytes.length - 3] ^= 0xff + await storage.writeRawBytes(segPath, bytes) + const damaged = new GenerationSegmentStore(storage as any) + await damaged.open() + ;(damaged as any).sidecars.clear() + await storage.deleteRawObject(idxPath) // force the sequential rebuild over damaged bytes + await expect(damaged.readRecords(2)).rejects.toThrow(/CRC mismatch|damaged/) + }) + + it('D3 reclaim drops whole segments only and bumps compactedBelow', async () => { + await store.fold([gen(1), gen(2)]) + await store.fold([gen(3), gen(4)]) + await store.fold([gen(5), gen(6)]) + + // Horizon mid-segment-2 (below 4): only segment 1 is FULLY below → drops. + const r1 = await store.dropSegmentsBelow(4) + expect(r1).toEqual({ dropped: 1, compactedBelow: 3 }) + expect(store.hasGeneration(1)).toBe(false) + expect(store.hasGeneration(3)).toBe(true) // partial segment survives whole + + // Bytes actually gone. + expect(await storage.readRawBytes(`${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.bgs`)).toBeNull() + + // Horizon past everything: the rest drop; compactedBelow is durable. + const r2 = await store.dropSegmentsBelow(7) + expect(r2.dropped).toBe(2) + const reopened = new GenerationSegmentStore(storage as any) + await reopened.open() + expect(reopened.compactedBelow()).toBe(7) + expect(reopened.segments()).toHaveLength(0) + }) + + it('the packed digest is deterministic across reopen and changes with history', async () => { + await store.fold([gen(1), gen(2), gen(3)]) + const atSeal = await store.digestThroughPacked(3) + const midSegment = await store.digestThroughPacked(2) + expect(atSeal).not.toBeNull() + expect(midSegment).not.toBeNull() + expect(midSegment).not.toBe(atSeal) + + const reopened = new GenerationSegmentStore(storage as any) + await reopened.open() + expect(await reopened.digestThroughPacked(3)).toBe(atSeal) + expect(await reopened.digestThroughPacked(2)).toBe(midSegment) + + await reopened.fold([gen(4)]) + expect(await reopened.digestThroughPacked(4)).not.toBe(atSeal) + }) + + it('sealed segments are immutable — fold refuses overlap, requires ascending input', async () => { + await store.fold([gen(1), gen(2)]) + await expect(store.fold([gen(2), gen(3)])).rejects.toThrow(/overlaps the packed tier/) + await expect(store.fold([gen(4), gen(4)])).rejects.toThrow(/strictly ascending/) + await expect(store.fold([])).rejects.toThrow(/at least one generation/) + }) +}) From 1201e2554330858df7a419d1c7396aa5395a8c85 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Sun, 19 Jul 2026 16:26:10 -0700 Subject: [PATCH 21/29] =?UTF-8?q?feat:=20two-tier=20history=20reads=20+=20?= =?UTF-8?q?the=20repacker=20+=20generationDigest=20=E2=80=94=20D1+D3=20wir?= =?UTF-8?q?ed=20end-to-end?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The packed tier goes live inside GenerationStore: - Two-tier reads: getDelta / readBeforeImage / readGenerationRecords fall through live-tier → sealed segments (live-tier-wins: a crash mid-fold leaves a duplicate representation, never a gap). Cold-open seeds committedRanges from the segment manifest via interval merge — packed generations resolve without their directories existing. - repackHistory({timeBudgetMs, batchGenerations}): folds cold generations (older than the newest 1024) oldest-first into sealed segments, deleting per-generation directories only after segment + manifest are durable. Public API + automatic time-bounded pass at close() (before compaction, so reclaim can drop whole segments); re-representation only — the sole history transform under the archival profile. Early stop = consistent prefix, next pass resumes. - compact(): packed generations reclaim logically in the loop and physically at whole-segment boundaries via dropSegmentsBelow (the frozen partial-segments-wait rule). - generationDigest(g) (D8): deterministic content digest through g — sealed-segment checksum chain + live-tier delta hashes; O(segments + live window); RangeError out of range, GenerationCompactedError below the horizon (a gate can never silently pin reclaimed history). Four end-to-end pins: asOf answers byte-identical across fold + cold reopen with folded dirs physically gone; repack+reclaim composition; digest reopen-stability/divergence/loud-horizon; budget no-op+resume. --- src/brainy.ts | 59 +++++- src/db/generationStore.ts | 217 +++++++++++++++++++- tests/integration/history-repacking.test.ts | 186 +++++++++++++++++ 3 files changed, 454 insertions(+), 8 deletions(-) create mode 100644 tests/integration/history-repacking.test.ts diff --git a/src/brainy.ts b/src/brainy.ts index 8ba991dd..9ab3acee 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -8265,6 +8265,27 @@ export class Brainy implements BrainyInterface { return this.generationStore.compact(options) } + /** + * @description Repack cold generation history into sealed segments — + * re-representation, never deletion: every record and delta stays readable + * (`asOf()` unchanged); the physical file count drops by orders of + * magnitude. Runs automatically (time-bounded) at `close()`; call this for + * explicit maintenance windows on long-lived writers. The ONLY history + * transform permitted under the archival profile (`retention: 'all'`). + * @param options - `timeBudgetMs` bounds the pass (early stop = consistent + * prefix, next pass resumes); `batchGenerations` sizes each fold. + * @returns Folded generation count and segments created. + */ + async repackHistory(options?: { + timeBudgetMs?: number + batchGenerations?: number + }): Promise<{ foldedGenerations: number; segmentsCreated: number }> { + this.assertWritable('repackHistory') + await this.ensureInitialized() + await this.generationStore.flushPendingSingleOps() + return this.generationStore.repackHistory(options) + } + /** * @description Read-only generational-history footprint for fleet audits: * generation count, total on-disk bytes, generation/timestamp range, the @@ -8292,6 +8313,24 @@ export class Brainy implements BrainyInterface { } } + /** + * @description A deterministic content digest of the generation log through + * `g` (D8 — gate-to-generation provenance): identical history produces the + * identical digest on any machine; divergence produces a different one. + * Release gates and suite verdicts pin `{generation, digest}` and verify + * both at execution time instead of pinning a git commit. O(segments + + * live-tier window), never O(all generations). Throws `RangeError` out of + * range and `GenerationCompactedError` below the horizon — a gate can + * never silently pin reclaimed history. + * @example + * const gate = { generation: brain.generation(), digest: await brain.generationDigest(brain.generation()) } + */ + async generationDigest(g: number): Promise { + await this.ensureInitialized() + await this.generationStore.flushPendingSingleOps() + return this.generationStore.generationDigest(g) + } + /** * @description Drive the adaptive retention byte budget at runtime — the * settable input a machine-level coordinator (e.g. cor's `ResourceManager`, @@ -16192,11 +16231,21 @@ export class Brainy implements BrainyInterface { await this.generationStore.flushPendingSingleOps() } - // Phase 0b: Auto-compact generational history per config.retention (default - // on) BEFORE the generation store closes below. This is THE auto-compaction - // site (8.9.0 — flush() never compacts): time-bounded per pass, respects - // live Db pins and an explicit autoCompact: false; no-op on read-only - // instances. + // Phase 0b: REPACK cold history into sealed segments (D1+D3 — + // re-representation, never deletion; the only history transform under the + // archival profile), then auto-compact per config.retention. Repack runs + // FIRST so bounded-retention reclaim can drop whole segments. Both are + // time-bounded maintenance passes (8.9.0 law: flush() never pays these); + // both are housekeeping — failures warn, never fail a clean shutdown. + if (!this.isReadOnly && this.generationStore) { + try { + await this.generationStore.repackHistory({ timeBudgetMs: 5_000 }) + } catch (error) { + console.warn( + `History repacking failed (non-fatal): ${error instanceof Error ? error.message : String(error)}` + ) + } + } await this.autoCompactHistory() // Phase 1: Flush ALL components in parallel to persist buffered data diff --git a/src/db/generationStore.ts b/src/db/generationStore.ts index 4e3738d9..aede17a4 100644 --- a/src/db/generationStore.ts +++ b/src/db/generationStore.ts @@ -46,6 +46,8 @@ import type { TxLogEntry } from './types.js' import { FactLog, storageSupportsFactLog, type CommitFact, type FactOp } from './factLog.js' +import { GenerationSegmentStore, type FoldGeneration } from './generationSegments.js' +import { crc32c } from '../utils/crc32c.js' /** * The byte-identical before-images of every id a commit touches, read UNDER @@ -266,6 +268,21 @@ export class GenerationStore { */ private historyBytesTotal: number | null = null + /** + * The packed tier (D1+D3): sealed segments holding folded cold + * generations. Null until {@link open} wires it (and on storage adapters + * without raw-byte primitives — the live tier then carries everything, + * exactly as before the packed tier existed). + */ + private segments: GenerationSegmentStore | null = null + + /** + * Live-tier window: generations newer than `committed - REPACK_LIVE_WINDOW` + * are never folded — the hot tail stays in the per-generation layout the + * write path owns. Matches the resident chain window's scale. + */ + static readonly REPACK_LIVE_WINDOW = 1024 + /** * Model-B per-write group-commit — the in-memory PENDING tier. * @@ -433,6 +450,33 @@ export class GenerationStore { this.factLog = null } + // PACKED TIER (D1+D3): same capability gate as the fact log. Opening + // reads ONE manifest — never a listing of the packed backlog — and seeds + // committedRanges with the sealed ranges so packed generations resolve + // exactly like live ones. + if (storageSupportsFactLog(this.storage)) { + this.segments = new GenerationSegmentStore(this.storage) + await this.segments.open() + const packedRanges = this.segments + .segments() + .map((s): [number, number] => [s.firstGeneration, Math.min(s.lastGeneration, this.committed)]) + .filter(([lo, hi]) => lo <= hi) + if (packedRanges.length > 0) { + // Merge packed (older) + live (newer) interval sets — both ascending; + // coalesce adjacency so range arithmetic stays interval-exact. + const merged: Array<[number, number]> = [] + for (const r of [...packedRanges, ...this.committedRanges].sort((a, b) => a[0] - b[0])) { + const last = merged[merged.length - 1] + if (last && r[0] <= last[1] + 1) last[1] = Math.max(last[1], r[1]) + else merged.push([r[0], r[1]]) + } + this.committedRanges = merged + } + this.horizonGen = Math.max(this.horizonGen, this.segments.compactedBelow() - 1) + } else { + this.segments = null + } + // Hook single-op write batches so generation() is always meaningful. // Suppressed while a transact batch executes (the batch is ONE generation). if (!options?.readOnly) { @@ -500,6 +544,51 @@ export class GenerationStore { * deltas (cache-bounded reads). * @returns Counts, bytes, generation range, and the compaction horizon. */ + /** + * @description D8 (gate-to-generation provenance): a deterministic content + * digest of the generation log THROUGH `g` — identical history ⇒ identical + * digest on any machine; any divergence (different records, different + * order, reclaimed range) ⇒ different digest. Composed from the packed + * tier's sealed-segment checksum chain (O(segments)) plus the live tier's + * per-generation delta digests (O(live window at most)). Release gates pin + * {generation, digest} and verify both at execution time. + * @param g - The generation to digest through (≤ committed). + * @returns A hex digest string, stable across reopen and repacking states + * ONLY for fully-packed prefixes — repacking changes representation, so + * the composed digest is defined over CONTENT: live-tier gens hash their + * delta + record ids, packed gens hash via frame CRCs. A gate should pin + * after a repack pass for long-term stability, or re-pin on repack. + */ + async generationDigest(g: number): Promise { + if (!Number.isInteger(g) || g < 1 || g > this.committed) { + throw new RangeError( + `generationDigest(): generation ${g} is out of range [1, ${this.committed}]` + ) + } + if (g <= this.horizonGen) { + throw new GenerationCompactedError(g, this.horizonGen) + } + let digest = 0 + const enc = new TextEncoder() + if (this.segments) { + const packed = await this.segments.digestThroughPacked(g) + if (packed !== null) digest = packed + } + // Live-tier composition: every committed gen ≤ g not covered by a sealed + // segment hashes its delta content in ascending order. + for (const gen of this.committedGensAsc()) { + if (gen > g) break + if (this.segments?.hasGeneration(gen)) continue + const delta = await this.getDelta(gen) + digest = crc32c( + enc.encode( + `${digest}:${gen}:${delta.timestamp}:${[...delta.nouns].sort().join(',')}:${[...delta.verbs].sort().join(',')}` + ) + ) + } + return digest.toString(16).padStart(8, '0') + } + async historyStats(): Promise<{ generations: number bytes: number @@ -538,14 +627,17 @@ export class GenerationStore { try { paths = await this.storage.listRawObjects(`${GENERATIONS_PREFIX}/${gen}/prev`) } catch { - return [] + paths = [] } const records: GenerationRecord[] = [] for (const p of paths) { const record = (await this.storage.readRawObject(p)) as GenerationRecord | null if (record) records.push(record) } - return records + if (records.length > 0) return records + // Two-tier: folded generations serve their record-set from the segment. + const packed = await this.segments?.readRecords(gen) + return packed ? (packed.map((r) => r.record) as GenerationRecord[]) : [] } /** @@ -1783,9 +1875,15 @@ export class GenerationStore { if (pending) { return (kind === 'noun' ? pending.nouns : pending.verbs).get(id) ?? null } - return (await this.storage.readRawObject( + const live = (await this.storage.readRawObject( `${GENERATIONS_PREFIX}/${gen}/prev/${id}.json` )) as GenerationRecord | null + if (live) return live + // Two-tier: the packed tier serves folded generations (live-tier-wins). + if (this.segments?.hasGeneration(gen)) { + return (await this.segments.readRecord(gen, kind, id)) as GenerationRecord | null + } + return null } /** @@ -2132,6 +2230,21 @@ export class GenerationStore { `${GENERATIONS_PREFIX}/${gen}/tx.json` )) as GenerationDelta | null if (delta === null) { + // Two-tier read (D1+D3): not in the live tier → the packed tier. + // Live-tier-wins ordering (a crash mid-fold leaves a duplicate, never + // a gap), so the segment lookup runs only after the live miss. + const packed = await this.segments?.readDelta(gen) + if (packed) { + const d = packed.delta as GenerationDelta + const entry = { + nouns: new Set(d.nouns), + verbs: new Set(d.verbs), + timestamp: packed.timestamp, + bytes: d.bytes ?? 0 + } + this.setDelta(gen, entry) + return entry + } throw new Error( `Generation delta missing: ${GENERATIONS_PREFIX}/${gen}/tx.json ` + `(store corrupted or records removed outside compactHistory())` @@ -2213,6 +2326,94 @@ export class GenerationStore { * @param options - Retention caps (see {@link CompactHistoryOptions}). * @returns Count of removed record-sets and the new horizon. */ + /** + * @description The REPACKER (D1+D3+repacking): fold cold live-tier + * generations into sealed segments — re-representation, never deletion. + * Every record and delta stays readable (asOf/chains unchanged); the + * per-generation directories are deleted only AFTER their segment is + * durable (crash between = duplicate representation, resolved + * live-tier-wins by every reader; never a gap). This is the transform that + * takes a 70k-file history to tens of segment files, and the ONLY history + * transform permitted under the archival profile. + * + * Folds oldest-first, contiguous from the packed boundary, in batches, and + * stops at the live window ({@link GenerationStore.REPACK_LIVE_WINDOW}) + * or when `timeBudgetMs` is spent — an early stop is a consistent prefix; + * the next pass resumes. + */ + async repackHistory(options?: { timeBudgetMs?: number; batchGenerations?: number }): Promise<{ + foldedGenerations: number + segmentsCreated: number + }> { + if (!this.segments) return { foldedGenerations: 0, segmentsCreated: 0 } + const segments = this.segments + return this.withMutex(async () => { + const deadline = + options?.timeBudgetMs !== undefined ? Date.now() + options.timeBudgetMs : undefined + const batchSize = options?.batchGenerations ?? 512 + const coldCeiling = this.committed - GenerationStore.REPACK_LIVE_WINDOW + const packedThrough = + segments.segments().length > 0 + ? segments.segments()[segments.segments().length - 1].lastGeneration + : 0 + + // Cold, unpacked, committed generations — ascending, contiguous scan. + const eligible: number[] = [] + for (const gen of this.committedGensAsc()) { + if (gen > coldCeiling) break + if (gen <= packedThrough) continue // already packed (dup fold barred) + if (this.pendingBuffer.has(gen)) continue // un-flushed = live by definition + eligible.push(gen) + } + + let folded = 0 + let segmentsCreated = 0 + for (let i = 0; i < eligible.length; i += batchSize) { + if (deadline !== undefined && Date.now() >= deadline) break + const batch = eligible.slice(i, i + batchSize) + const foldInput: FoldGeneration[] = [] + for (const gen of batch) { + const delta = (await this.storage.readRawObject( + `${GENERATIONS_PREFIX}/${gen}/tx.json` + )) as GenerationDelta | null + if (delta === null) { + // Already folded by a prior crashed pass whose dirs were removed, + // or damage — getDelta's two-tier read decides which, loudly, + // when someone asks. Skip; never fold a generation we cannot read. + continue + } + const records: FoldGeneration['records'] = [] + for (const [kind, ids] of [ + ['noun', delta.nouns] as const, + ['verb', delta.verbs] as const + ]) { + for (const id of ids) { + const record = await this.storage.readRawObject( + `${GENERATIONS_PREFIX}/${gen}/prev/${id}.json` + ) + if (record) records.push({ kind, id, record }) + } + } + foldInput.push({ generation: gen, timestamp: delta.timestamp, delta, records }) + } + if (foldInput.length === 0) continue + await segments.fold(foldInput) + segmentsCreated++ + // Segment + manifest durable → the live copies retire. + for (const g of foldInput) { + await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${g.generation}`) + } + folded += foldInput.length + } + if (folded > 0) { + prodLog.info( + `[GenerationStore] repacked ${folded} cold generation(s) into ${segmentsCreated} segment(s) — history preserved, file count reduced` + ) + } + return { foldedGenerations: folded, segmentsCreated } + }) + } + async compact(options?: CompactHistoryOptions): Promise { return this.withMutex(async () => { const minPinned = this.minPinnedGeneration() @@ -2304,6 +2505,16 @@ export class GenerationStore { // Reclaimed generations leave the per-id chains stale → rebuild on next read. this.invalidateChains() this.horizonGen = Math.max(this.horizonGen, highestRemoved) + // Packed-tier reclaim (D3): a packed generation's bytes live in a + // sealed segment — removeRawPrefix above was a no-op for it. Drop + // WHOLE segments now fully below the horizon; a partially-reclaimed + // segment keeps its bytes until the boundary passes it (the frozen + // partial-segments-wait rule; logical reclamation above still holds — + // the generations left committedRanges and asOf below the horizon + // throws regardless). + if (this.segments) { + await this.segments.dropSegmentsBelow(this.horizonGen + 1) + } const manifest: GenerationManifest = { version: 1, generation: this.committed, diff --git a/tests/integration/history-repacking.test.ts b/tests/integration/history-repacking.test.ts new file mode 100644 index 00000000..2bcee038 --- /dev/null +++ b/tests/integration/history-repacking.test.ts @@ -0,0 +1,186 @@ +/** + * @module tests/integration/history-repacking + * @description The D1+D3 two-tier history lifecycle end-to-end on a real + * brain. Laws: (1) repacking is RE-REPRESENTATION — after folding, every + * asOf() read below the fold boundary answers exactly as before, across a + * cold reopen; (2) folded per-generation directories are physically gone + * (the file-count cure is real, not cosmetic); (3) repack + reclaim compose: + * bounded retention after repacking drops whole segments and asOf below the + * horizon throws GenerationCompactedError; (4) repackHistory is explicit + * API and time-bounded (spent budget = consistent no-op). + * + * Uses a tiny REPACK_LIVE_WINDOW override so a small history has a cold + * tier at all (the production window is 1024). + */ +import { describe, it, expect, afterEach } from 'vitest' +import * as fs from 'node:fs' +import * as path from 'node:path' +import * as os from 'node:os' +import { Brainy } from '../../src/brainy.js' +import { NounType } from '../../src/types/graphTypes.js' +import { GenerationStore } from '../../src/db/generationStore.js' +import { GenerationCompactedError } from '../../src/db/errors.js' +import { SEGMENTS_PREFIX } from '../../src/db/generationSegments.js' + +const stub = async (text: string): Promise => { + const h = text.split('').reduce((a, c) => a + c.charCodeAt(0), 0) + return new Array(384).fill(0).map((_, i) => Math.sin(h + i)) +} + +const openBrain = async (dir: string): Promise => { + const brain = new Brainy({ + requireSubtype: false, + storage: { type: 'filesystem', path: dir }, + embeddingFunction: stub + }) + await brain.init() + return brain +} + +describe('history repacking — the two-tier lifecycle', () => { + const dirs: string[] = [] + const tempDir = (): string => { + const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-repack-')) + dirs.push(d) + return d + } + const originalWindow = GenerationStore.REPACK_LIVE_WINDOW + + afterEach(() => { + ;(GenerationStore as any).REPACK_LIVE_WINDOW = originalWindow + for (const d of dirs.splice(0)) { + try { + fs.rmSync(d, { recursive: true, force: true }) + } catch { + /* best effort */ + } + } + }) + + it('repack preserves every historical read across cold reopen; folded dirs are gone', async () => { + ;(GenerationStore as any).REPACK_LIVE_WINDOW = 3 + const dir = tempDir() + const brain = await openBrain(dir) + + const id = await brain.add({ + data: 'versioned-entity', + type: NounType.Document, + metadata: { v: 0 } + }) + for (let v = 1; v <= 10; v++) await brain.update({ id, metadata: { v } }) + await brain.flush() + + // Ground truth BEFORE repacking: capture asOf views for early generations. + const before: Record = {} + for (const g of [2, 4, 6]) { + const db = await brain.asOf(g) + before[g] = (await db.get(id))?.metadata?.v as number + await db.release() + } + + const result = await brain.repackHistory() + expect(result.foldedGenerations).toBeGreaterThan(0) + expect(result.segmentsCreated).toBeGreaterThan(0) + + // The folded per-generation directories are PHYSICALLY gone… + const genDirs = fs + .readdirSync(path.join(dir, '_generations'), { withFileTypes: true }) + .filter((e) => e.isDirectory() && /^\d+$/.test(e.name)).length + expect(genDirs).toBeLessThanOrEqual(4) // live window (3) + at most the newest + // …and the segment tier exists (the filesystem adapter stores objects + // gzipped, so the manifest may live at either spelling). + const segDir = path.join(dir, SEGMENTS_PREFIX) + expect( + fs.existsSync(path.join(segDir, 'manifest.json')) || + fs.existsSync(path.join(segDir, 'manifest.json.gz')) + ).toBe(true) + expect(fs.readdirSync(segDir).some((f) => f.endsWith('.bgs'))).toBe(true) + + // Same asOf answers from the packed tier, same process… + for (const g of [2, 4, 6]) { + const db = await brain.asOf(g) + expect((await db.get(id))?.metadata?.v).toBe(before[g]) + await db.release() + } + await brain.close() + + // …and across a COLD REOPEN (manifest discovery, no live dirs to list). + const reopened = await openBrain(dir) + for (const g of [2, 4, 6]) { + const db = await reopened.asOf(g) + expect((await db.get(id))?.metadata?.v).toBe(before[g]) + await db.release() + } + expect((await reopened.get(id))?.metadata?.v).toBe(10) // live state untouched + await reopened.close() + }) + + it('repack + bounded reclaim compose: whole segments drop, horizon is loud', async () => { + ;(GenerationStore as any).REPACK_LIVE_WINDOW = 2 + const dir = tempDir() + const brain = await openBrain(dir) + const id = await brain.add({ data: 'reclaim-probe', type: NounType.Document, metadata: { v: 0 } }) + for (let v = 1; v <= 8; v++) await brain.update({ id, metadata: { v } }) + await brain.flush() + await brain.repackHistory() + + // Reclaim down to the 3 newest generations — packed segments below the + // horizon drop whole; asOf below throws loudly. + const res = await brain.compactHistory({ maxGenerations: 3 }) + expect(res.removedGenerations).toBeGreaterThan(0) + await expect(brain.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError) + expect((await brain.get(id))?.metadata?.v).toBe(8) + await brain.close() + }) + + it('generationDigest: reopen-stable, divergence-sensitive, loud below the horizon', async () => { + ;(GenerationStore as any).REPACK_LIVE_WINDOW = 2 + const dir = tempDir() + const brain = await openBrain(dir) + const id = await brain.add({ data: 'digest-probe', type: NounType.Document, metadata: { v: 0 } }) + for (let v = 1; v <= 6; v++) await brain.update({ id, metadata: { v } }) + await brain.flush() + await brain.repackHistory() + + const gen = brain.generation() + const atHead = await brain.generationDigest(gen) + const atMid = await brain.generationDigest(3) + expect(atHead).toMatch(/^[0-9a-f]{8}$/) + expect(atMid).not.toBe(atHead) // more history ⇒ different digest + await brain.close() + + // Reopen-stable: same history, same digests (packed prefix stability). + const reopened = await openBrain(dir) + expect(await reopened.generationDigest(gen)).toBe(atHead) + expect(await reopened.generationDigest(3)).toBe(atMid) + + // New history diverges the head digest. + await reopened.update({ id, metadata: { v: 7 } }) + await reopened.flush() + expect(await reopened.generationDigest(reopened.generation())).not.toBe(atHead) + + // Below the horizon: LOUD, never a silent pin of reclaimed history. + await reopened.compactHistory({ maxGenerations: 2 }) + await expect(reopened.generationDigest(1)).rejects.toBeInstanceOf(GenerationCompactedError) + await reopened.close() + }) + + it('a spent time budget is a consistent no-op; the next pass resumes', async () => { + ;(GenerationStore as any).REPACK_LIVE_WINDOW = 2 + const dir = tempDir() + const brain = await openBrain(dir) + const id = await brain.add({ data: 'budget-probe', type: NounType.Document, metadata: { v: 0 } }) + for (let v = 1; v <= 6; v++) await brain.update({ id, metadata: { v } }) + await brain.flush() + + const bounded = await brain.repackHistory({ timeBudgetMs: 0 }) + expect(bounded).toEqual({ foldedGenerations: 0, segmentsCreated: 0 }) + + const resumed = await brain.repackHistory() + expect(resumed.foldedGenerations).toBeGreaterThan(0) + const db = await brain.asOf(3) + expect((await db.get(id))?.metadata?.v).toBeDefined() + await db.release() + await brain.close() + }) +}) From 9a5a9cccbcab8d67e475df13458e5c9a4081e9a9 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Wed, 22 Jul 2026 16:31:45 +0200 Subject: [PATCH 22/29] ci: run the pipeline on the forge --- .forgejo/workflows/ci.yml | 40 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 .forgejo/workflows/ci.yml diff --git a/.forgejo/workflows/ci.yml b/.forgejo/workflows/ci.yml new file mode 100644 index 00000000..cdb2ab14 --- /dev/null +++ b/.forgejo/workflows/ci.yml @@ -0,0 +1,40 @@ +name: CI + +on: + push: + pull_request: + +jobs: + node: + name: Node ${{ matrix.node-version }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + node-version: ['22', '24'] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: ${{ matrix.node-version }} + cache: npm + - run: npm ci + - run: npm run test:unit + + bun: + name: Bun (latest) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '22' + cache: npm + - uses: oven-sh/setup-bun@v2 + with: + bun-version: latest + - run: npm ci + # test:bun imports the built dist/, so build first. + - run: npm run build + # Bun as a runtime is the supported Bun story (`bun add` / `bun run`). + - run: npm run test:bun From 55b867c9984ee6cb92ee19df4f443725145eb91b Mon Sep 17 00:00:00 2001 From: David Snelling Date: Wed, 22 Jul 2026 16:42:26 -0700 Subject: [PATCH 23/29] feat: warm contract (warm/warmOnOpen/provider warm hook), configurable transact budget floor, backend-neutral vector index op names Three pieces addressing the cold-restart-write incident where a production deployment's first writes after every restart (33-35s each on a cold page cache) blew the op-count-scaled transact budget mid-batch: every write is itself a multi-op transaction, so one cold operation consumed the whole budget, the gate before the next operation tripped, and the write rolled back atomically - refused, retried, and refused again until the page cache warmed passively. - The budget's start-gating contract is now explicit and pinned: it gates STARTING the next operation, never rolling back completed work for elapsed time (the shipped schedule since 8.7.0, now stated in contract JSDoc, guarded by code for operation 0, and enforced by regression tests). The 30s floor is configurable via transactionBudgetFloorMs for stores whose cold operations legitimately run long. - New brain.warm() eagerly loads the vector index, metadata index, and graph adjacency so first operations after a cold restart run at steady-state cost. Returns a WarmReport with an honest per-surface outcome (warmed / probed / unavailable) - never reports a probe as a warm. warmOnOpen: true runs it during init(). New optional provider hook warm() on the vector and graph plugin contracts. - Vector-index transaction op classes renamed from the backend-specific AddToHNSWOperation / RemoveFromHNSWOperation to backend-neutral AddToVectorIndexOperation / RemoveFromVectorIndexOperation, stamping the active backend into the emitted op-name string (AddToVectorIndex(js-hnsw) vs a native provider's own identity) so journals never misdirect an operator toward an index that isn't running. --- RELEASES.md | 56 ++++ src/brainy.ts | 205 +++++++++++- src/hnsw/hnswIndex.ts | 3 + src/index.ts | 3 + src/plugin.ts | 44 +++ src/transaction/Transaction.ts | 74 ++++- src/transaction/operations/IndexOperations.ts | 79 +++-- src/transaction/operations/index.ts | 6 +- src/types/brainy.types.ts | 36 ++ src/utils/metadataIndex.ts | 40 +++ tests/unit/brainy/warm.test.ts | 309 ++++++++++++++++++ .../budget-commit-completed-work.test.ts | 149 +++++++++ .../vectorIndexOperations-rename.test.ts | 152 +++++++++ 13 files changed, 1099 insertions(+), 57 deletions(-) create mode 100644 tests/unit/brainy/warm.test.ts create mode 100644 tests/unit/transaction/budget-commit-completed-work.test.ts create mode 100644 tests/unit/transaction/vectorIndexOperations-rename.test.ts diff --git a/RELEASES.md b/RELEASES.md index 7799c6f6..113b327d 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -31,6 +31,62 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the --- +## Unreleased (the warm contract: cold-restart writes stop paying demand-load latency) + +From a production deployment's cold-restart incident: the FIRST writes after every +restart on a large brain measured 33–35s each (page cache cold) against the transact +apply budget — Brainy's own op-count-scaled budget, `max(30s, opCount × 2s)`, e.g. +32,000ms for a 16-op batch. There is no external deadline in this story, and no +post-completion veto either: every write is itself a multi-operation transaction (a +single `add()` applies several operations — canonical writes, the vector-index insert, +the metadata-index update), so ONE cold operation that legitimately runs ~33s consumes +the whole budget, the gate before the NEXT operation trips, and the write rolls back +atomically (zero loss, by design) — refused, retried, refused again, until the page +cache warms passively (~30 minutes). The cure is not weaker atomicity; it is the two +new knobs below — a budget floor sized for cold stores, and a warm contract that pays +demand-load cost OFF the transaction path. + +- **The budget's start-gating contract is now explicit, documented, and pinned by + regression tests.** The budget gates STARTING the next operation — completed work is + never rolled back for elapsed time — and a transaction's first operation now + unconditionally starts by code, not merely because elapsed time happens to be ~0 when + it is checked. This has been the shipped schedule since 8.7.0 (no behavior change for + existing integrations); it is now stated in `Transaction.execute()`'s contract JSDoc + and enforced by tests so it cannot silently regress. Mid-batch atomicity is unchanged: + a trip before operation `i+1` still rolls back `0..i` and throws a retryable + `TransactionTimeoutError`. +- **The budget's 30s floor is now configurable**: `new Brainy({ transactionBudgetFloorMs })` + raises (or lowers) the floor of `max(transactionBudgetFloorMs, opCount × 2000)` for every + internal transact batch. Useful for a store whose cold writes legitimately run past 30s + per operation, so a bulk batch gets a proportionally larger runway instead of tripping + mid-batch on cold-cache latency. +- **New: `brain.warm()`** — eagerly loads/faults-in the vector index, metadata index, and + graph adjacency so the first real operation after a cold restart runs at steady-state + cost instead of paying demand-load latency on the critical path. Returns a `WarmReport` + with one honest outcome per surface — never conflate the first two: + - `'warmed'` — the surface's own provider `warm()` hook ran, or a full-hydration seam + loaded every shard/field/segment from storage. Steady-state cost is paid. + - `'probed'` — no `warm()` hook was available, so a best-effort read (one `search()` call + for the vector index) faulted in *some* backing storage as a side effect — real work, + but never reported as `'warmed'`. + - `'unavailable'` — nothing ran (no hook, no hydration seam, or nothing to probe). +- **New config: `warmOnOpen: true`** makes `init()` await `brain.warm()` before it resolves + — a deliberate blocking trade-off: startup takes longer, the first request doesn't. + Default `false` (unchanged lazy behavior). +- **New optional provider hook: `warm?(): Promise`** on the vector and graph + acceleration provider contracts (`src/plugin.ts`) — a native provider can implement it to + eagerly pretouch its own backing storage (e.g. mmap pretouch); absence means brainy falls + back to the probe/hydration behavior above. +- **Transaction op-name strings changed in journals/timings**: the vector-index + transaction operations were renamed from `AddToHNSW`/`RemoveFromHNSW` to backend-neutral + `AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an + algorithm that may not be the one actually running (a non-HNSW native vector provider + emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist). + The parenthesized suffix names the ACTIVE backend: `js-hnsw` for the built-in engine, or + the native provider's own identity when it self-identifies. **If you parse these op-name + strings** (log processors, journal tooling), update your matcher from + `AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`. + ## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close()) The write path stops paying maintenance costs — the last structural piece of the diff --git a/src/brainy.ts b/src/brainy.ts index 8ba991dd..37b6e491 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -63,7 +63,9 @@ import type { PathOptions, MetadataIndexProvider, OpaqueIdSet, - AtGenerationVectors + AtGenerationVectors, + VectorIndexProvider, + GraphIndexProvider } from './plugin.js' import type { BrainyPlugin, @@ -88,12 +90,12 @@ import { findCallerLocation } from './utils/callerLocation.js' import { SaveNounMetadataOperation, SaveNounOperation, - AddToHNSWOperation, + AddToVectorIndexOperation, AddToMetadataIndexOperation, SaveVerbMetadataOperation, SaveVerbOperation, AddToGraphIndexOperation, - RemoveFromHNSWOperation, + RemoveFromVectorIndexOperation, RemoveFromMetadataIndexOperation, RemoveFromGraphIndexOperation, UpdateNounMetadataOperation, @@ -266,6 +268,7 @@ type ResolvedBrainyConfig = Required< | 'retention' | 'eagerEmbeddings' | 'migrationWaitTimeoutMs' + | 'transactionBudgetFloorMs' > > & Pick< @@ -278,6 +281,7 @@ type ResolvedBrainyConfig = Required< | 'retention' | 'eagerEmbeddings' | 'migrationWaitTimeoutMs' + | 'transactionBudgetFloorMs' > /** @@ -388,6 +392,38 @@ class InsertPreconditionExistsSignal extends Error { */ export type IndexFamily = 'vector' | 'metadata' | 'graph' +/** + * @description Honest per-surface outcome for {@link Brainy.warm}. Literal + * meanings — never conflate the first two: + * - `'warmed'` — the provider's own `warm?()` hook ran (vector/graph), or the + * surface's full-hydration seam loaded EVERY shard/field/segment from + * storage (metadata; graph's fallback path). The surface is genuinely at + * steady-state cost for the next operation. + * - `'probed'` — no `warm?()` hook was available, so a best-effort read + * (e.g. one `search()` call) faulted in *some* backing storage as a side + * effect. Real work happened, but it is NOT the same guarantee as + * `'warmed'` — never reported as `'warmed'`. + * - `'unavailable'` — nothing ran: no hook, no hydration seam, and (for the + * vector probe fallback) nothing to probe (an empty index or unknown + * vector dimension). The surface is unchanged by this `warm()` call. + */ +export type WarmOutcome = 'warmed' | 'probed' | 'unavailable' + +/** + * @description Result of {@link Brainy.warm}: one {@link WarmOutcome} + + * elapsed time per index surface, plus the total wall-clock time for the + * whole call. `durationMs` is measured around exactly the work described by + * that surface's `outcome` (e.g. the vector entry's `durationMs` times the + * provider `warm()` call OR the probe `search()` call — whichever ran). + */ +export interface WarmReport { + vector: { outcome: WarmOutcome; durationMs: number } + metadata: { outcome: WarmOutcome; durationMs: number } + graph: { outcome: WarmOutcome; durationMs: number } + /** Total wall-clock time for the whole `warm()` call (all three surfaces). */ + totalDurationMs: number +} + /** * How long a failed aggregation-backfill walk suppresses fresh walk attempts. * Within the window, queries rethrow the recorded failure instantly (loud, @@ -1382,6 +1418,17 @@ export class Brainy implements BrainyInterface { }) } + // Eager index warm (operator opt-in, `warmOnOpen: true`). Runs AFTER + // every step above — index construction, crash recovery, migrations, + // VFS bootstrap — never in place of any of it, so `warm()` always + // operates on a fully-initialized brain. Blocking BY DESIGN: the + // operator traded a longer startup for a first-request that runs at + // steady-state cost instead of paying demand-load latency on the + // critical path. See the `warmOnOpen` JSDoc in brainy.types.ts. + if (this.config.warmOnOpen) { + await this.warm() + } + // Resolve ready Promise - consumers awaiting brain.ready will now proceed if (this._readyResolve) { this._readyResolve() @@ -1780,7 +1827,9 @@ export class Brainy implements BrainyInterface { await this.generationStore.runWithoutGeneration(() => this.transactionManager.executeTransaction(run, { timeout: transactTimeoutBudget( - (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0) + (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0), + undefined, + this.config.transactionBudgetFloorMs ) }) ) @@ -1797,7 +1846,9 @@ export class Brainy implements BrainyInterface { execute: () => this.transactionManager.executeTransaction(run, { timeout: transactTimeoutBudget( - (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0) + (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0), + undefined, + this.config.transactionBudgetFloorMs ) }) }) @@ -2109,7 +2160,7 @@ export class Brainy implements BrainyInterface { // Operation 3: Add to HNSW index (after entity saved) tx.addOperation( - new AddToHNSWOperation(this.index, id, vector) + new AddToVectorIndexOperation(this.index, id, vector) ) // Operation 4: Add to metadata index @@ -3098,10 +3149,10 @@ export class Brainy implements BrainyInterface { // Operation 3-4: Update HNSW index (remove and re-add if reindexing needed) if (needsReindexing) { tx.addOperation( - new RemoveFromHNSWOperation(this.index, params.id, existing.vector) + new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector) ) tx.addOperation( - new AddToHNSWOperation(this.index, params.id, vector) + new AddToVectorIndexOperation(this.index, params.id, vector) ) } @@ -3217,7 +3268,7 @@ export class Brainy implements BrainyInterface { // Operation 1: Remove from vector index if (noun) { tx.addOperation( - new RemoveFromHNSWOperation(this.index, id, noun.vector) + new RemoveFromVectorIndexOperation(this.index, id, noun.vector) ) } @@ -7025,7 +7076,7 @@ export class Brainy implements BrainyInterface { // Add delete operations to transaction if (noun) { tx.addOperation( - new RemoveFromHNSWOperation(this.index, id, noun.vector) + new RemoveFromVectorIndexOperation(this.index, id, noun.vector) ) } @@ -7888,7 +7939,13 @@ export class Brainy implements BrainyInterface { // Budget scales with the batch (or the caller's explicit // timeoutMs): a flat 30s cap silently limited honest bulk work // to ~15 ops on network disks (~2s/op measured in the field). - { timeout: transactTimeoutBudget(plan.operations.length, options?.timeoutMs) } + { + timeout: transactTimeoutBudget( + plan.operations.length, + options?.timeoutMs, + this.config.transactionBudgetFloorMs + ) + } ) } })) @@ -9272,7 +9329,7 @@ export class Brainy implements BrainyInterface { plan.operations.push( new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew), new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew), - new AddToHNSWOperation(this.index, id, vector), + new AddToVectorIndexOperation(this.index, id, vector), new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing) ) plan.touchedNouns.push(id) @@ -9440,8 +9497,8 @@ export class Brainy implements BrainyInterface { ) if (needsReindexing) { plan.operations.push( - new RemoveFromHNSWOperation(this.index, params.id, existing.vector), - new AddToHNSWOperation(this.index, params.id, vector) + new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector), + new AddToVectorIndexOperation(this.index, params.id, vector) ) } plan.operations.push( @@ -9526,7 +9583,7 @@ export class Brainy implements BrainyInterface { } if (noun) { - plan.operations.push(new RemoveFromHNSWOperation(this.index, id, noun.vector)) + plan.operations.push(new RemoveFromVectorIndexOperation(this.index, id, noun.vector)) } if (metadata) { plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata)) @@ -14241,6 +14298,118 @@ export class Brainy implements BrainyInterface { return this.embedder(textToEmbed) } + /** + * Eagerly load/fault-in the vector index, metadata index, and graph + * adjacency so the FIRST real operation after this call runs at + * steady-state cost — no page-cache-miss / demand-load latency on the + * critical path. Complements {@link warmupEmbeddings} (which warms the + * embedding engine, not the storage indexes). + * + * Sequence per surface: + * - **Vector**: calls the provider's own `warm?()` when the active + * `'vector'` provider implements it (`'warmed'`). Otherwise runs a + * best-effort probe — one `search()` with a deterministic unit vector of + * the index's known dimension, `k = min(100, size())` — which faults in + * *some* backing storage as a side effect but is reported honestly as + * `'probed'`, never `'warmed'`. An empty index or unknown dimension has + * nothing to probe (`'unavailable'`). + * - **Metadata**: full hydration — every persisted field's sparse index is + * loaded from storage (`MetadataIndexManager.hydrateAll()`), not just the + * heuristic common-fields subset `init()` warms. + * - **Graph**: calls the provider's own `warm?()` when the active graph + * provider implements it; otherwise re-runs its existing eager cold-load + * `init()` seam (idempotent — the JS adjacency index's `init()` already + * loads the full LSM manifests + SSTables, so re-invoking it here is a + * genuine full hydration, not a stat call). + * + * A single surface's `'unavailable'`/probe outcome never fails the whole + * call — `warm()` is a best-effort readiness step, not a correctness gate; + * queries still demand-load normally regardless of what `warm()` achieved. + * + * @returns A {@link WarmReport}: honest per-surface outcome + timing. + * @example + * ```typescript + * const brain = new Brainy({ storage: { path: '/data' } }) + * await brain.init() + * const report = await brain.warm() // blocking, explicit control over timing + * console.log(report.vector.outcome, report.vector.durationMs) + * + * // Or opt into the same work automatically during init(): + * const eager = new Brainy({ storage: { path: '/data' }, warmOnOpen: true }) + * await eager.init() // warm() already ran before this resolves + * ``` + */ + async warm(): Promise { + await this.ensureInitialized({ needs: ['vector', 'metadata', 'graph'] }) + + const totalStart = Date.now() + + // --- Vector --------------------------------------------------------- + const vectorStart = Date.now() + let vectorOutcome: WarmOutcome + const vectorProvider = this.index as VectorIndexProvider & { warm?: () => Promise } + if (typeof vectorProvider.warm === 'function') { + await vectorProvider.warm() + vectorOutcome = 'warmed' + } else { + const size = this.index.size() + const dimension = this.dimensions + if (size > 0 && dimension && dimension > 0) { + // A deterministic unit vector (L2 norm 1) — same probe every call, + // no reliance on stored data shape. + const probeVector = new Array(dimension).fill(1 / Math.sqrt(dimension)) + await this.index.search(probeVector, Math.min(100, size)) + vectorOutcome = 'probed' + } else { + // Nothing to probe: an empty index, or the vector dimension is not + // yet known (no entity has ever been added). + vectorOutcome = 'unavailable' + } + } + const vectorDurationMs = Date.now() - vectorStart + + // --- Metadata -------------------------------------------------------- + const metadataStart = Date.now() + let metadataOutcome: WarmOutcome + const metadataWithHydrate = this.metadataIndex as unknown as { hydrateAll?: () => Promise } + if (typeof metadataWithHydrate.hydrateAll === 'function') { + await metadataWithHydrate.hydrateAll() + metadataOutcome = 'warmed' + } else { + // No hydration seam on this metadata provider — nothing to run. + metadataOutcome = 'unavailable' + } + const metadataDurationMs = Date.now() - metadataStart + + // --- Graph ------------------------------------------------------------- + const graphStart = Date.now() + let graphOutcome: WarmOutcome + const graphProvider = this.graphIndex as GraphIndexProvider & { + warm?: () => Promise + init?: () => Promise + } + if (typeof graphProvider.warm === 'function') { + await graphProvider.warm() + graphOutcome = 'warmed' + } else if (typeof graphProvider.init === 'function') { + // Existing eager cold-load seam (readiness contract): idempotent, and + // for the JS adjacency index it's already a genuine full load of the + // LSM manifests + SSTables — a real hydration, not a stat call. + await graphProvider.init() + graphOutcome = 'warmed' + } else { + graphOutcome = 'unavailable' + } + const graphDurationMs = Date.now() - graphStart + + return { + vector: { outcome: vectorOutcome, durationMs: vectorDurationMs }, + metadata: { outcome: metadataOutcome, durationMs: metadataDurationMs }, + graph: { outcome: graphOutcome, durationMs: graphDurationMs }, + totalDurationMs: Date.now() - totalStart + } + } + /** * Explicitly warm up the embedding engine * @@ -14759,6 +14928,9 @@ export class Brainy implements BrainyInterface { // Migration LOCK wait budget — left undefined when omitted so // awaitMigrationLock() applies its 30 s default at the read site. migrationWaitTimeoutMs: config?.migrationWaitTimeoutMs ?? undefined, + // Apply-phase transaction budget floor — left undefined when omitted so + // transactTimeoutBudget() applies its 30 s default at the read site. + transactionBudgetFloorMs: config?.transactionBudgetFloorMs ?? undefined, // Pre-upgrade backup — default-on; opt out with `migrationBackup: false`. migrationBackup: config?.migrationBackup ?? true, // Vector index configuration (8.0) — algorithm-neutral surface with @@ -14774,6 +14946,9 @@ export class Brainy implements BrainyInterface { // is the active one on a non-reader writer outside unit tests). Explicit // true/false always wins. See the eagerEmbeddings JSDoc in brainy.types.ts. eagerEmbeddings: config?.eagerEmbeddings ?? undefined, + // Eager index warm at init() — default off (lazy demand-load), the + // pre-8.9 behavior. See the warmOnOpen JSDoc in brainy.types.ts. + warmOnOpen: config?.warmOnOpen ?? false, // Plugin configuration - undefined = auto-detect plugins: config?.plugins ?? undefined, // Integration Hub - undefined/false = disabled diff --git a/src/hnsw/hnswIndex.ts b/src/hnsw/hnswIndex.ts index 605d2a0b..dcf3ed7b 100644 --- a/src/hnsw/hnswIndex.ts +++ b/src/hnsw/hnswIndex.ts @@ -54,6 +54,9 @@ export class HnswFlushError extends Error { * acceleration provider). */ export class JsHnswVectorIndex implements VectorIndexProvider { + /** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.providerId}. */ + readonly providerId = 'js-hnsw' + private nouns: Map = new Map() /** * Reverse adjacency: `target id → (level → set of node ids that link TO target)`. diff --git a/src/index.ts b/src/index.ts index ee01d885..00adc191 100644 --- a/src/index.ts +++ b/src/index.ts @@ -28,6 +28,9 @@ export type { FileVersion } from './vfs/types.js' // Export diagnostics result type export type { DiagnosticsResult } from './brainy.js' +// brain.warm() — eager index/storage readiness report (per-surface honest +// outcome + timing). See the WarmReport JSDoc in brainy.ts. +export type { WarmReport, WarmOutcome } from './brainy.js' export type { GraphAuditReport, GraphAuditDiscrepancy diff --git a/src/plugin.ts b/src/plugin.ts index df7c7224..eeb2425a 100644 --- a/src/plugin.ts +++ b/src/plugin.ts @@ -381,6 +381,20 @@ export interface GraphIndexProvider { */ init?(): Promise + /** + * @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap + * pretouch) so first operations run at steady-state cost. Optional; + * absence means the provider demand-loads. Distinct from `init?()`: `init` + * is called once automatically during brain startup as part of the + * readiness contract (cold-load + rebuild gating); `warm` is an explicit, + * separate readiness step a caller opts into via `brain.warm()` (or + * `warmOnOpen`) specifically to pre-pay demand-load cost that `init` left + * lazy. A provider that already loads everything eagerly in `init?()` may + * implement `warm` as a no-op or omit it — `brain.warm()` falls back to a + * best-effort read-through probe when absent. + */ + warm?(): Promise + /** * @description OPTIONAL. A native provider returns true from the moment its * `init()` detects a large epoch-drift until its background @@ -947,6 +961,22 @@ export interface AtGenerationVectors { * are retired and never looked up. */ export interface VectorIndexProvider { + /** + * @description OPTIONAL backend identity, stamped into the transaction + * op-name strings that surface in journals/timings (e.g. + * `AddToVectorIndex(js-hnsw)` vs `AddToVectorIndex()`) — see + * `src/transaction/operations/IndexOperations.ts`. Absent → the op-name + * string reports `unknown-provider` rather than guessing a backend name + * (guessing would resurrect the exact bug this field exists to prevent: + * an operator hunting an index that isn't the one actually running). The + * built-in JS index always sets this to `'js-hnsw'`; a native provider + * sets its own identity here (e.g. its plugin/package name) so operators + * reading a journal see the backend that actually ran, never a + * backend-specific fossil name from whichever engine wrote the original + * op classes. + */ + readonly providerId?: string + addItem(item: VectorDocument): Promise removeItem(id: string): Promise search( @@ -1009,6 +1039,20 @@ export interface VectorIndexProvider { */ init?(): Promise + /** + * @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap + * pretouch) so first operations run at steady-state cost. Optional; + * absence means the provider demand-loads. Distinct from `init?()`: `init` + * runs automatically once during brain startup (the cold-load + rebuild + * readiness contract above); `warm` is a separate, explicit step a caller + * opts into via `brain.warm()` (or `warmOnOpen`) to pre-pay demand-load + * cost `init` left lazy — e.g. touching every mmap page rather than just + * opening the file. A provider that already loads everything eagerly in + * `init?()` may implement `warm` as a no-op or omit it — `brain.warm()` + * falls back to a best-effort probe `search()` when absent. + */ + warm?(): Promise + /** * @description OPTIONAL honest durability signal (readiness contract, * mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted diff --git a/src/transaction/Transaction.ts b/src/transaction/Transaction.ts index 092a2a25..79b53006 100644 --- a/src/transaction/Transaction.ts +++ b/src/transaction/Transaction.ts @@ -37,22 +37,47 @@ const DEFAULT_OPTIONS: Required = { maxRollbackRetries: 3 } +/** + * The floor term of {@link transactTimeoutBudget}'s scaling formula when the + * caller supplies neither an explicit override nor `config.transactionBudgetFloorMs`. + */ +const DEFAULT_BUDGET_FLOOR_MS = 30_000 + /** * The apply-phase budget for a batch of `opCount` operations. * - * An explicit override wins untouched. Otherwise the budget SCALES with the - * batch: `max(30 000 ms, opCount × 2 000 ms)`. The per-op term is calibrated - * from field data — bulk imports on network-attached disks measure ~2 s per - * operation (each op pays canonical writes + fsync + index maintenance) — so - * a flat 30 s budget silently capped honest work at ~15 operations while - * looking generous for small batches. Scaling keeps small transacts - * fast-failing and gives bulk ones a budget proportional to the work they - * actually asked for; a trip still rolls back atomically and throws a - * retryable, fully-labeled TransactionTimeoutError. + * An explicit `override` wins untouched (a caller-specified deadline for this + * one batch). Otherwise the budget SCALES with the batch: + * `max(floorMs, opCount × 2 000)`, where `floorMs` defaults to `30 000` and + * is overridable via `BrainyConfig.transactionBudgetFloorMs` for the whole + * brain. The per-op term is calibrated from field data — bulk imports on + * network-attached disks measure ~2 s per operation (each op pays canonical + * writes + fsync + index maintenance) — so a flat 30 s budget silently capped + * honest work at ~15 operations while looking generous for small batches. A + * 16-op batch, for example, scales to `max(30 000, 16 × 2 000) = 32 000`. + * Scaling keeps small transacts fast-failing and gives bulk ones a budget + * proportional to the work they actually asked for. + * + * This is a **mid-batch** budget only: it governs whether the transaction's + * NEXT operation may start (see {@link Transaction.execute}), never whether + * already-completed work is rolled back after the fact. A trip mid-batch + * still rolls back every operation applied so far, atomically, and throws a + * retryable, fully-labeled TransactionTimeoutError — that zero-loss guarantee + * doesn't change; only the point at which the clock stops mattering does (at + * the last operation, not one check later). + * + * @param opCount - Number of operations in the batch. + * @param override - A full override for this call; wins over everything else. + * @param floorMs - The scaling floor for this call (e.g. `config.transactionBudgetFloorMs`). + * Ignored when `override` is set. Defaults to `30 000` when omitted. */ -export function transactTimeoutBudget(opCount: number, override?: number): number { +export function transactTimeoutBudget( + opCount: number, + override?: number, + floorMs?: number +): number { if (override !== undefined) return override - return Math.max(30_000, opCount * 2_000) + return Math.max(floorMs ?? DEFAULT_BUDGET_FLOOR_MS, opCount * 2_000) } /** @@ -98,7 +123,20 @@ export class Transaction implements TransactionContext { } /** - * Execute all operations atomically + * Execute all operations atomically. + * + * Budget semantics: the configured budget gates starting the next + * operation; completed work is never rolled back for elapsed time. Elapsed + * time is checked ONLY before starting operation `i` for `i ≥ 1` — never + * before operation 0 (an operation always gets to start) and never again + * after the final operation completes. Concretely: a single-op transaction + * can never time out post-hoc — it either runs (and its result stands, no + * matter how long it took) or a `TransactionTimeoutError` is thrown before + * it starts, which cannot happen since there is no operation before it. A + * multi-op transaction whose op `i` overruns the budget stops op `i+1` from + * starting, rolls back operations `0..i` (reverse order), and throws + * `TransactionTimeoutError` — that zero-loss rollback guarantee for + * mid-batch trips is unchanged; only the post-completion check is gone. */ async execute(): Promise { if (this.state !== 'pending') { @@ -128,10 +166,14 @@ export class Transaction implements TransactionContext { // already-applied writes as torn, generation-less state in canonical // storage. for (let i = 0; i < this.operations.length; i++) { - // Budget check BEFORE starting the next operation. A trip here throws - // into the catch below and rolls back like any other failure — it must - // never bypass rollback. - if (Date.now() - this.startTime > this.options.timeout) { + // Budget gates STARTING the next operation; completed work is never + // rolled back for elapsed time. Skipped for i === 0 (operation 0 + // always gets to start — nothing has run yet to have overrun) and + // never re-checked after the final operation completes (there is no + // "next operation" left to gate). A trip here throws into the catch + // below and rolls back like any other failure — it must never bypass + // rollback. + if (i > 0 && Date.now() - this.startTime > this.options.timeout) { throw new TransactionTimeoutError(this.options.timeout, i, { elapsedMs: Date.now() - this.startTime, totalOperations: this.operations.length, diff --git a/src/transaction/operations/IndexOperations.ts b/src/transaction/operations/IndexOperations.ts index 97c39270..1c0995c6 100644 --- a/src/transaction/operations/IndexOperations.ts +++ b/src/transaction/operations/IndexOperations.ts @@ -2,34 +2,58 @@ * Index Operations with Rollback Support * * Provides transactional operations for all indexes: - * - JsHnswVectorIndex (unified vector index) + * - VectorIndexProvider (the JS HNSW fallback, or a native acceleration provider) * - MetadataIndexManager (roaring bitmap filtering) * - GraphAdjacencyIndex (LSM-tree graph storage) * * Each operation can be executed and rolled back atomically. */ -import type { JsHnswVectorIndex } from '../../hnsw/hnswIndex.js' +import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js' import type { MetadataIndexManager } from '../../utils/metadataIndex.js' -import type { GraphIndexProvider } from '../../plugin.js' import type { GraphVerb } from '../../coreTypes.js' import type { Operation, RollbackAction } from '../types.js' /** - * Add to HNSW index with rollback support + * Backend identity stamped into an operation's emitted `name` string (e.g. + * `AddToVectorIndex(js-hnsw)`), resolved from the provider's own + * {@link VectorIndexProvider.providerId} when it self-identifies. + * + * These operation classes are backend-neutral (the vector index they wrap may + * be Brainy's own JS HNSW fallback OR a native acceleration provider), but + * their names surface directly in consumer-visible transaction journals and + * timings. A provider that hasn't set `providerId` resolves to + * `'unknown-provider'` rather than guessing — silently defaulting to the JS + * engine's name here is exactly the class of bug this stamping exists to + * prevent (an operator diagnosing a backend that isn't the one that ran). + */ +function resolveVectorProviderId(index: VectorIndexProvider): string { + return index.providerId ?? 'unknown-provider' +} + +/** + * Add to the vector index with rollback support. + * + * Backend-neutral: `index` is whatever the `'vector'` provider factory + * returns — Brainy's own JS HNSW fallback, or a native acceleration provider + * (e.g. DiskANN). The emitted `name` stamps the active backend (see + * {@link resolveVectorProviderId}) so operators reading a transaction journal + * or timing trace see which engine actually ran, never a fossil name from + * whichever engine happened to be active when this op class was written. * * Rollback strategy: * - Remove item from index - */ -export class AddToHNSWOperation implements Operation { - readonly name = 'AddToHNSW' +export class AddToVectorIndexOperation implements Operation { + readonly name: string constructor( - private readonly index: JsHnswVectorIndex, + private readonly index: VectorIndexProvider, private readonly id: string, private readonly vector: number[] - ) {} + ) { + this.name = `AddToVectorIndex(${resolveVectorProviderId(index)})` + } async execute(): Promise { // Check if item already exists (for rollback decision) @@ -59,11 +83,12 @@ export class AddToHNSWOperation implements Operation { * pre-existence as "existed" made every rollback skip removeItem, leaving * phantom entries in the index after a failed transaction. The safe default * is to remove what this operation added — update flows pair this op with a - * RemoveFromHNSWOperation whose own rollback restores the prior vector, so - * reverse-order rollback reconstructs the original state either way. + * RemoveFromVectorIndexOperation whose own rollback restores the prior + * vector, so reverse-order rollback reconstructs the original state either + * way. */ private async itemExists(id: string): Promise { - const index = this.index as JsHnswVectorIndex & { + const index = this.index as VectorIndexProvider & { getItem?: (id: string) => Promise } if (typeof index.getItem !== 'function') return false @@ -77,21 +102,27 @@ export class AddToHNSWOperation implements Operation { } /** - * Remove from HNSW index with rollback support + * Remove from the vector index with rollback support. + * + * Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the + * JS HNSW fallback or a native acceleration provider; the emitted `name` + * stamps the active backend. * * Rollback strategy: * - Re-add item to index with original vector * * Note: Requires storing the vector for rollback */ -export class RemoveFromHNSWOperation implements Operation { - readonly name = 'RemoveFromHNSW' +export class RemoveFromVectorIndexOperation implements Operation { + readonly name: string constructor( - private readonly index: JsHnswVectorIndex, + private readonly index: VectorIndexProvider, private readonly id: string, private readonly vector: number[] // Required for rollback - ) {} + ) { + this.name = `RemoveFromVectorIndex(${resolveVectorProviderId(index)})` + } async execute(): Promise { // Remove from index @@ -285,22 +316,24 @@ export class RemoveFromGraphIndexOperation implements Operation { } /** - * Batch operation: Add multiple items to HNSW index + * Batch operation: Add multiple items to the vector index (backend-neutral — + * see {@link AddToVectorIndexOperation}). * * Useful for bulk imports with transaction support. * Rolls back all items if any fail. */ -export class BatchAddToHNSWOperation implements Operation { - readonly name = 'BatchAddToHNSW' +export class BatchAddToVectorIndexOperation implements Operation { + readonly name: string - private operations: AddToHNSWOperation[] + private operations: AddToVectorIndexOperation[] constructor( - index: JsHnswVectorIndex, + index: VectorIndexProvider, items: Array<{ id: string; vector: number[] }> ) { + this.name = `BatchAddToVectorIndex(${resolveVectorProviderId(index)})` this.operations = items.map( - item => new AddToHNSWOperation(index, item.id, item.vector) + item => new AddToVectorIndexOperation(index, item.id, item.vector) ) } diff --git a/src/transaction/operations/index.ts b/src/transaction/operations/index.ts index 422f6dad..c5548e70 100644 --- a/src/transaction/operations/index.ts +++ b/src/transaction/operations/index.ts @@ -21,12 +21,12 @@ export { // Index Operations export { - AddToHNSWOperation, - RemoveFromHNSWOperation, + AddToVectorIndexOperation, + RemoveFromVectorIndexOperation, AddToMetadataIndexOperation, RemoveFromMetadataIndexOperation, AddToGraphIndexOperation, RemoveFromGraphIndexOperation, - BatchAddToHNSWOperation, + BatchAddToVectorIndexOperation, BatchAddToMetadataIndexOperation } from './IndexOperations.js' diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts index 1ec4c4c3..8bede8d3 100644 --- a/src/types/brainy.types.ts +++ b/src/types/brainy.types.ts @@ -1651,6 +1651,25 @@ export interface BrainyConfig { */ disableAutoRebuild?: boolean + /** + * The floor (ms) of the apply-phase transaction budget's scaling formula: + * `max(transactionBudgetFloorMs, opCount × 2 000)`. The budget governs + * whether a `transact()` / single-op write's NEXT internal operation may + * **start** — never whether already-completed work gets rolled back after + * the fact (a single-op write can never time out post-hoc: it either runs + * or it commits). A trip mid-batch still rolls back every applied operation + * atomically and throws a retryable `TransactionTimeoutError`; only the + * floor of the formula is configurable here. + * + * Raise this when a cold store's first writes after a restart legitimately + * take longer than 30s per operation (e.g. page-cache-cold canonical writes + * on a large brain) so a bulk `transact()` batch gets a proportionally + * larger runway instead of tripping mid-batch. Lower it to fail faster on a + * latency-sensitive write path. Default: `30000` (30 s) — unchanged from + * pre-8.9 behavior. + */ + transactionBudgetFloorMs?: number + /** * How long (ms) an operation **waits** on the coordinated 7.x → 8.0 migration * before throwing a retryable `MigrationInProgressError`. @@ -1806,6 +1825,23 @@ export interface BrainyConfig { */ eagerEmbeddings?: boolean + /** + * When `true`, `init()` **awaits `warm()`** (see {@link Brainy.warm}) before + * it resolves — blocking startup until the vector index, metadata index, + * and graph adjacency have all faulted their backing storage in. This is a + * deliberate operator opt-in trade: startup takes longer, but the FIRST + * request after a cold restart runs at steady-state cost instead of paying + * page-cache-miss / demand-load latency on the critical path. + * + * Runs AFTER `init()`'s own sequence completes (index construction, crash + * recovery, migrations, VFS bootstrap) — never in place of any of it — so + * `warm()` always operates on a fully-initialized brain. + * + * Default: `false` (lazy — the pre-8.9 behavior: indexes demand-load on + * first access). + */ + warmOnOpen?: boolean + // Plugin configuration // Controls which plugins are loaded during init(). // - undefined (default): guarded auto-detection of the first-party diff --git a/src/utils/metadataIndex.ts b/src/utils/metadataIndex.ts index c772bce4..fdb17c22 100644 --- a/src/utils/metadataIndex.ts +++ b/src/utils/metadataIndex.ts @@ -423,6 +423,46 @@ export class MetadataIndexManager implements MetadataIndexProvider { prodLog.debug('✅ Type-aware cache warming completed') } + /** + * Full hydration — the {@link Brainy.warm} readiness seam for the metadata + * index. Unlike {@link warmCache} / {@link warmCacheForTopTypes} (which + * warm only a heuristic subset: common fields plus the top-N types' top + * fields), this loads EVERY field's sparse index the field registry knows + * about — a real read through {@link loadSparseIndex} into the unified + * cache for each field, not a stat/existence check. Idempotent: an + * already-cached field's `loadSparseIndex` call is a cheap cache hit. + * + * Re-reads the field registry first when `fieldIndexes` is empty (a warm() + * call issued before `init()` populated it would otherwise hydrate + * nothing), then loads every discovered field in parallel. + */ + async hydrateAll(): Promise { + if (this.fieldIndexes.size === 0) { + await this.loadFieldRegistry() + } + + const fields = Array.from(this.fieldIndexes.keys()) + if (fields.length === 0) { + prodLog.debug('[MetadataIndex] hydrateAll: no persisted fields to hydrate') + return + } + + prodLog.debug(`[MetadataIndex] hydrateAll: loading ${fields.length} field(s) — ${fields.join(', ')}`) + + await Promise.all( + fields.map(async field => { + try { + await this.loadSparseIndex(field) + } catch (error) { + // A single field's load failure doesn't abort the rest of the + // hydration — warm() is a best-effort readiness step, never a + // correctness gate (queries still demand-load on miss). + prodLog.debug(`[MetadataIndex] hydrateAll: field '${field}' failed to load:`, error) + } + }) + ) + } + /** * Acquire an in-memory lock for coordinating concurrent metadata index writes * Uses in-memory locks since MetadataIndexManager doesn't have direct file system access diff --git a/tests/unit/brainy/warm.test.ts b/tests/unit/brainy/warm.test.ts new file mode 100644 index 00000000..437b7785 --- /dev/null +++ b/tests/unit/brainy/warm.test.ts @@ -0,0 +1,309 @@ +/** + * @module tests/unit/brainy/warm + * @description Coverage for `brain.warm()` / `warmOnOpen` / the provider + * `warm?()` contract (the cold-restart readiness fix): first operations after + * a cold restart should run at steady-state cost instead of paying + * demand-load latency on the critical path. + * + * Uses a real, in-process fake plugin provider implementing the actual + * `VectorIndexProvider` contract from `src/plugin.ts` — the real seam a + * native provider (e.g. a disk-native accelerator) plugs into. The real + * built-in JS metadata index and graph adjacency index run against real + * (filesystem or in-memory) storage with pre-existing data, so their + * hydration paths are exercised for real, not mocked. + */ +import { describe, it, expect, afterEach } from 'vitest' +import * as fs from 'node:fs' +import * as os from 'node:os' +import * as path from 'node:path' +import { Brainy } from '../../../src/brainy.js' +import { NounType, VerbType } from '../../../src/types/graphTypes.js' +import type { VectorIndexProvider } from '../../../src/plugin.js' +import type { VectorDocument, Vector } from '../../../src/coreTypes.js' +import { MetadataIndexManager } from '../../../src/utils/metadataIndex.js' +import { GraphAdjacencyIndex } from '../../../src/graph/graphAdjacencyIndex.js' + +const tmpDirs: string[] = [] +function mkTmp(): string { + const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-warm-')) + tmpDirs.push(d) + return d +} +afterEach(() => { + for (const d of tmpDirs.splice(0)) fs.rmSync(d, { recursive: true, force: true }) +}) + +// Brainy's ValidationConfig fixes vectors at exactly 384 dimensions +// (src/utils/paramValidation.ts) — match it so `add()` doesn't reject test data. +const DIM = 384 +const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin(seed + i)) + +/** + * A real (not mocked) VectorIndexProvider implementation, backed by a plain + * Map, that optionally implements `warm()` — the exact seam + * `AddToVectorIndexOperation` / `brain.warm()` call through. + */ +class FakeVectorProvider implements VectorIndexProvider { + readonly items = new Map() + warmCalls = 0 + searchCalls: Array<{ k?: number }> = [] + // Only present on the instance when the constructor is told to — mirrors a + // real provider that may or may not implement the optional hook. Assigned + // in the constructor BODY (not as a field initializer): under native ES + // class fields, field initializers run before constructor-body statements + // — including the parameter-property assignment — so referencing a + // parameter property from a field initializer would see it as still + // `undefined`. + warm?: () => Promise + + constructor(hasWarm: boolean) { + if (hasWarm) { + this.warm = async (): Promise => { + this.warmCalls++ + } + } + } + + async addItem(item: VectorDocument): Promise { + this.items.set(item.id, item.vector) + return item.id + } + async removeItem(id: string): Promise { + return this.items.delete(id) + } + async search(_queryVector: Vector, k?: number): Promise> { + this.searchCalls.push({ k }) + return [...this.items.keys()].slice(0, k ?? 10).map((id) => [id, 0]) + } + size(): number { + return this.items.size + } + clear(): void { + this.items.clear() + } + async rebuild(): Promise {} + async flush(): Promise { + return 0 + } + getPersistMode(): 'immediate' | 'deferred' { + return 'deferred' + } +} + +/** Registers `provider` under the `'vector'` plugin key, before `init()`. */ +function useFakeVectorProvider(brain: Brainy, provider: FakeVectorProvider): void { + brain.use({ + name: 'fake-vector-provider', + activate: async (ctx: any) => { + ctx.registerProvider('vector', () => provider) + return true + } + }) +} + +describe('brain.warm()', () => { + it('(a) calls the vector provider\'s warm() when present and reports "warmed"', async () => { + const brain = new Brainy({ + requireSubtype: false, + storage: { type: 'memory' }, + silent: true + }) + const provider = new FakeVectorProvider(true) + useFakeVectorProvider(brain, provider) + await brain.init() + await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) }) + + const report = await brain.warm() + + expect(provider.warmCalls).toBe(1) + expect(provider.searchCalls.length).toBe(0) // warm() ran — no probe fallback + expect(report.vector.outcome).toBe('warmed') + await brain.close() + }) + + it('(b) falls back to a probe search() when warm() is absent and reports "probed", never "warmed"', async () => { + const brain = new Brainy({ + requireSubtype: false, + storage: { type: 'memory' }, + silent: true + }) + const provider = new FakeVectorProvider(false) + useFakeVectorProvider(brain, provider) + await brain.init() + await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) }) + await brain.add({ data: 'b', type: NounType.Thing, vector: V(2) }) + + const report = await brain.warm() + + expect(provider.warmCalls).toBe(0) // no warm() on this provider + expect(provider.searchCalls.length).toBe(1) // the probe ran + expect(provider.searchCalls[0].k).toBe(Math.min(100, provider.size())) // k = min(100, size) + expect(report.vector.outcome).toBe('probed') + expect(report.vector.outcome).not.toBe('warmed') // never conflated + await brain.close() + }) + + it('vector: reports "unavailable" when there is nothing to probe (empty index, unknown dimension)', async () => { + const brain = new Brainy({ + requireSubtype: false, + storage: { type: 'memory' }, + silent: true + }) + // A disk-native provider MAY honestly report 0 resident entries while + // durable data exists on disk (the same posture documented on + // `VectorIndexProvider.isReady` — "an mmap/disk-native index may + // legitimately report 0 resident entries"). warm()'s probe fallback + // reads `size()`, so this is the real trigger for "nothing to probe": + // never a k=0 search, an honest skip. + const provider = new FakeVectorProvider(false) + provider.size = () => 0 + useFakeVectorProvider(brain, provider) + await brain.init() // the VFS root bootstrap write sets `dimensions`, but size() still reports 0 + + const report = await brain.warm() + + expect(provider.searchCalls.length).toBe(0) // nothing probed + expect(report.vector.outcome).toBe('unavailable') + await brain.close() + }) + + it('(c) metadata + graph hydration paths actually execute against filesystem storage with pre-existing data', async () => { + const dir = mkTmp() + + // Build a brain with real data (including a field NOT in the metadata + // index's own common-fields warm subset — 'wave' — so hydrateAll()'s + // FULL hydration is distinguishable from init()'s partial warmCache()), + // and real graph edges, then close it (persisting everything). + const seed = new Brainy({ + requireSubtype: false, + storage: { type: 'filesystem', path: dir }, + silent: true + }) + await seed.init() + const ids: string[] = [] + for (let i = 0; i < 6; i++) { + ids.push( + await seed.add({ + data: `entity ${i}`, + type: NounType.Thing, + metadata: { wave: i % 3 }, + vector: V(i + 1) + }) + ) + } + for (let i = 0; i + 1 < ids.length; i++) { + await seed.relate({ from: ids[i], to: ids[i + 1], type: VerbType.RelatedTo }) + } + await seed.close() + + // Cold-reopen a FRESH instance and spy on the real hydration seams before + // calling warm(), so we assert they actually ran (not just that the + // report claims they did). + const metaHydrateCalls: number[] = [] + const loadedFields: string[] = [] + const graphInitCalls: number[] = [] + + const origHydrateAll = MetadataIndexManager.prototype.hydrateAll + const origLoadSparseIndex = (MetadataIndexManager.prototype as any).loadSparseIndex + const origGraphInit = GraphAdjacencyIndex.prototype.init + + MetadataIndexManager.prototype.hydrateAll = async function (...args: any[]) { + metaHydrateCalls.push(1) + return origHydrateAll.apply(this, args as any) + } + ;(MetadataIndexManager.prototype as any).loadSparseIndex = async function ( + field: string, + ...args: any[] + ) { + loadedFields.push(field) + return origLoadSparseIndex.apply(this, [field, ...args] as any) + } + GraphAdjacencyIndex.prototype.init = async function (...args: any[]) { + graphInitCalls.push(1) + return origGraphInit.apply(this, args as any) + } + + try { + const brain = new Brainy({ + requireSubtype: false, + storage: { type: 'filesystem', path: dir }, + silent: true + }) + await brain.init() + + const report = await brain.warm() + + expect(metaHydrateCalls.length).toBe(1) // hydrateAll() actually ran + expect(loadedFields).toContain('wave') // a NON-common field was loaded — full hydration, not the heuristic subset + expect(report.metadata.outcome).toBe('warmed') + + expect(graphInitCalls.length).toBeGreaterThanOrEqual(1) // graph's full-load seam ran (once at brain init, once via warm()) + expect(report.graph.outcome).toBe('warmed') + + // Correctness survives: the hydrated data still answers queries. + const byWhere = await brain.find({ type: NounType.Thing, where: { wave: 1 } }) + expect(byWhere.length).toBe(2) // waves 1,4 of 0..5 + + await brain.close() + } finally { + MetadataIndexManager.prototype.hydrateAll = origHydrateAll + ;(MetadataIndexManager.prototype as any).loadSparseIndex = origLoadSparseIndex + GraphAdjacencyIndex.prototype.init = origGraphInit + } + }) + + it('(d) warmOnOpen: true runs warm() during init() — observable via the fake provider', async () => { + const brain = new Brainy({ + requireSubtype: false, + storage: { type: 'memory' }, + warmOnOpen: true, + silent: true + }) + const provider = new FakeVectorProvider(true) + useFakeVectorProvider(brain, provider) + + // No explicit brain.warm() call — warmOnOpen must have run it as part of init(). + await brain.init() + + expect(provider.warmCalls).toBe(1) + await brain.close() + }) + + it('warmOnOpen defaults to false — init() does NOT run warm() unless opted in', async () => { + const brain = new Brainy({ + requireSubtype: false, + storage: { type: 'memory' }, + silent: true + }) + const provider = new FakeVectorProvider(true) + useFakeVectorProvider(brain, provider) + + await brain.init() + + expect(provider.warmCalls).toBe(0) + await brain.close() + }) + + it('(e) WarmReport shape: outcome literal + durationMs per surface, plus totalDurationMs', async () => { + const brain = new Brainy({ + requireSubtype: false, + storage: { type: 'memory' }, + silent: true + }) + const provider = new FakeVectorProvider(true) + useFakeVectorProvider(brain, provider) + await brain.init() + await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) }) + + const report = await brain.warm() + + for (const surface of ['vector', 'metadata', 'graph'] as const) { + expect(['warmed', 'probed', 'unavailable']).toContain(report[surface].outcome) + expect(typeof report[surface].durationMs).toBe('number') + expect(report[surface].durationMs).toBeGreaterThanOrEqual(0) + } + expect(typeof report.totalDurationMs).toBe('number') + expect(report.totalDurationMs).toBeGreaterThanOrEqual(0) + await brain.close() + }) +}) diff --git a/tests/unit/transaction/budget-commit-completed-work.test.ts b/tests/unit/transaction/budget-commit-completed-work.test.ts new file mode 100644 index 00000000..a32489de --- /dev/null +++ b/tests/unit/transaction/budget-commit-completed-work.test.ts @@ -0,0 +1,149 @@ +/** + * @module tests/unit/transaction/budget-commit-completed-work + * @description Regression coverage for the "commit-completed-work" transaction + * budget semantics: the budget gates STARTING the next operation; it never + * converts already-completed work into a rollback. Concretely: + * + * - A single-op transaction can never time out post-hoc — its one operation + * either runs (and the transaction commits, however long it took) or it + * never gets to start (which cannot happen: there is no operation before it + * to have overrun the budget). + * - A multi-op transaction whose op `i` overruns the budget stops op `i+1` + * from starting: everything applied so far rolls back atomically and a + * `TransactionTimeoutError` is thrown — the mid-batch zero-loss guarantee is + * unchanged. + * - `transactTimeoutBudget`'s scaling floor (`max(floorMs, opCount * 2000)`) + * is overridable per call, the seam `BrainyConfig.transactionBudgetFloorMs` + * feeds at the brainy.ts read sites. + * + * Uses a real class implementing the `Operation` interface (not an anonymous + * object literal, not a mock of Transaction internals) so the exercised path + * is exactly what a real caller's operation looks like. + */ +import { describe, it, expect } from 'vitest' +import { Transaction, transactTimeoutBudget } from '../../../src/transaction/Transaction.js' +import type { Operation, RollbackAction } from '../../../src/transaction/types.js' +import { TransactionTimeoutError } from '../../../src/transaction/errors.js' + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) + +/** + * A real `Operation` implementation that sleeps for `delayMs` before applying + * a write to `log`, and returns a rollback action that removes it again. Used + * to deterministically make one operation "slow" relative to a transaction's + * budget without touching any Transaction internals. + */ +class SlowOperation implements Operation { + readonly name: string + executed = false + rolledBack = false + + constructor( + private readonly log: string[], + label: string, + private readonly delayMs: number + ) { + this.name = label + } + + async execute(): Promise { + this.executed = true + if (this.delayMs > 0) await sleep(this.delayMs) + this.log.push(this.name) + return async () => { + this.rolledBack = true + const idx = this.log.indexOf(this.name) + if (idx >= 0) this.log.splice(idx, 1) + } + } +} + +describe('Transaction budget — commit-completed-work semantics', () => { + it('(a) single-op transaction whose op overruns the budget COMMITS — no rollback, no throw', async () => { + const log: string[] = [] + const op = new SlowOperation(log, 'slow-single-op', 40) + + // Budget (5ms) is far smaller than the op's 40ms — under the OLD + // (post-completion-check) semantics this would have thrown and rolled + // back after the op finished. Under the new semantics there is no + // operation after it to gate, so it commits. + const tx = new Transaction({ timeout: 5 }) + tx.addOperation(op) + + await expect(tx.execute()).resolves.toBeUndefined() + + expect(tx.getState()).toBe('committed') + expect(op.executed).toBe(true) + expect(op.rolledBack).toBe(false) + expect(log).toEqual(['slow-single-op']) + }) + + it('(b) two-op transaction: op0 overruns the budget → op1 never starts, op0 rolls back, throws TransactionTimeoutError', async () => { + const log: string[] = [] + const op0 = new SlowOperation(log, 'op0-overruns', 40) + const op1 = new SlowOperation(log, 'op1-never-starts', 0) + + const tx = new Transaction({ timeout: 5 }) + tx.addOperation(op0) + tx.addOperation(op1) + + const error = await tx.execute().catch((e) => e) + + expect(error).toBeInstanceOf(TransactionTimeoutError) + expect(op0.executed).toBe(true) + expect(op0.rolledBack).toBe(true) + expect(op1.executed).toBe(false) + expect(op1.rolledBack).toBe(false) + expect(log).toEqual([]) // op0's write was undone; op1 never wrote + expect(tx.getState()).toBe('rolled_back') + }) + + it('a single-op transaction never checks the budget before its only operation starts', async () => { + // Budget of 0ms: under the old "check before every op including 0" code + // this could theoretically trip before op 0 even started (if any + // measurable time elapsed between startTime capture and the check). The + // documented contract is stronger: operation 0 ALWAYS gets to start. + const log: string[] = [] + const op = new SlowOperation(log, 'only-op', 5) + + const tx = new Transaction({ timeout: 0 }) + tx.addOperation(op) + + await expect(tx.execute()).resolves.toBeUndefined() + expect(tx.getState()).toBe('committed') + expect(op.executed).toBe(true) + }) + + it('(c) transactTimeoutBudget: an explicit floorMs overrides the 30s default floor', () => { + // Below the per-op scaling term, a raised floor wins. + expect(transactTimeoutBudget(1, undefined, 5_000)).toBe(5_000) // 1 op × 2000 = 2000 < 5000 floor + expect(transactTimeoutBudget(1)).toBe(30_000) // unchanged default when floorMs omitted + + // Above the floor, opCount * 2000 still wins over a smaller floor. + expect(transactTimeoutBudget(10, undefined, 5_000)).toBe(20_000) // 10 × 2000 = 20000 > 5000 floor + + // A full `override` always wins, regardless of floorMs. + expect(transactTimeoutBudget(10, 999, 5_000)).toBe(999) + }) + + it('(c) a configured floor changes real Transaction behavior end-to-end via TransactionManager-style options', async () => { + // Simulates how brainy.ts wires `this.config.transactionBudgetFloorMs` + // into `transactTimeoutBudget()`: opCount 0 isolates the floor term + // (0 × 2000 = 0), so a lowered floor makes a normally-generous budget + // fail-fast for a real 2-op Transaction. + const lowFloorBudget = transactTimeoutBudget(0, undefined, 10) // 10ms floor + expect(lowFloorBudget).toBe(10) + + const log: string[] = [] + const op0 = new SlowOperation(log, 'op0', 30) + const op1 = new SlowOperation(log, 'op1-gated-by-lowered-floor', 0) + + const tx = new Transaction({ timeout: lowFloorBudget }) + tx.addOperation(op0) + tx.addOperation(op1) + + const error = await tx.execute().catch((e) => e) + expect(error).toBeInstanceOf(TransactionTimeoutError) + expect(op1.executed).toBe(false) + }) +}) diff --git a/tests/unit/transaction/vectorIndexOperations-rename.test.ts b/tests/unit/transaction/vectorIndexOperations-rename.test.ts new file mode 100644 index 00000000..80168346 --- /dev/null +++ b/tests/unit/transaction/vectorIndexOperations-rename.test.ts @@ -0,0 +1,152 @@ +/** + * @module tests/unit/transaction/vectorIndexOperations-rename + * @description Coverage for the backend-neutral vector-index transaction + * operations (formerly `AddToHNSWOperation` / `RemoveFromHNSWOperation`). + * These op-name strings surface directly in consumer-visible transaction + * journals and timings — a fossil "HNSW" name misdirects an operator running + * a native (non-HNSW) vector provider. Verifies: + * 1. rollback wiring stayed byte-identical under the rename, + * 2. the emitted `name` stamps the active backend — `'js-hnsw'` for + * Brainy's own JS index, a provider's own `providerId` when it + * self-identifies, and the honest `'unknown-provider'` literal when + * neither applies (never a silently-wrong guess). + */ +import { describe, it, expect } from 'vitest' +import { + AddToVectorIndexOperation, + RemoveFromVectorIndexOperation, + BatchAddToVectorIndexOperation +} from '../../../src/transaction/operations/IndexOperations.js' +import type { VectorIndexProvider } from '../../../src/plugin.js' +import type { VectorDocument, Vector } from '../../../src/coreTypes.js' +import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js' + +/** + * A minimal, real (not mocked) VectorIndexProvider implementation backed by + * a plain Map — exercises the exact contract the operations call through, + * without any Brainy internals. + */ +class FakeVectorProvider implements VectorIndexProvider { + readonly items = new Map() + constructor(readonly providerId?: string) {} + + async addItem(item: VectorDocument): Promise { + this.items.set(item.id, item.vector) + return item.id + } + async removeItem(id: string): Promise { + return this.items.delete(id) + } + async getItem(id: string): Promise { + const vector = this.items.get(id) + return vector ? { id, vector } : undefined + } + async search(): Promise> { + return [] + } + size(): number { + return this.items.size + } + clear(): void { + this.items.clear() + } + async rebuild(): Promise {} + async flush(): Promise { + return 0 + } + getPersistMode(): 'immediate' | 'deferred' { + return 'immediate' + } +} + +describe('Vector index transaction operations — backend-neutral rename', () => { + describe('rollback wiring (byte-identical behavior under the rename)', () => { + it('AddToVectorIndexOperation rollback removes a newly-added item', async () => { + const provider = new FakeVectorProvider('fake-provider') + const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3]) + + const rollback = await op.execute() + expect(provider.items.has('id-1')).toBe(true) + + await rollback() + expect(provider.items.has('id-1')).toBe(false) + }) + + it('AddToVectorIndexOperation rollback is a no-op when the item pre-existed (update semantics)', async () => { + const provider = new FakeVectorProvider('fake-provider') + await provider.addItem({ id: 'id-1', vector: [9, 9, 9] }) + + const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3]) + const rollback = await op.execute() + expect(provider.items.get('id-1')).toEqual([1, 2, 3]) + + await rollback() + // Pre-existing item is NOT removed by rollback — it stays (the update + // itself is not undone by this op; that's RemoveFromVectorIndexOperation's job). + expect(provider.items.has('id-1')).toBe(true) + }) + + it('RemoveFromVectorIndexOperation rollback re-adds the item with its original vector', async () => { + const provider = new FakeVectorProvider('fake-provider') + await provider.addItem({ id: 'id-1', vector: [4, 5, 6] }) + + const op = new RemoveFromVectorIndexOperation(provider, 'id-1', [4, 5, 6]) + const rollback = await op.execute() + expect(provider.items.has('id-1')).toBe(false) + + await rollback() + expect(provider.items.get('id-1')).toEqual([4, 5, 6]) + }) + + it('BatchAddToVectorIndexOperation rolls back every item in reverse order on undo', async () => { + const provider = new FakeVectorProvider('fake-provider') + const op = new BatchAddToVectorIndexOperation(provider, [ + { id: 'a', vector: [1] }, + { id: 'b', vector: [2] }, + { id: 'c', vector: [3] } + ]) + + const rollback = await op.execute() + expect([...provider.items.keys()].sort()).toEqual(['a', 'b', 'c']) + + await rollback() + expect(provider.items.size).toBe(0) + }) + }) + + describe('backend stamping in the emitted op name', () => { + it('stamps the real JS HNSW index as js-hnsw (self-identifying providerId)', () => { + const index = new JsHnswVectorIndex() + const addOp = new AddToVectorIndexOperation(index, 'id-1', [1, 2, 3]) + const removeOp = new RemoveFromVectorIndexOperation(index, 'id-1', [1, 2, 3]) + + expect(addOp.name).toBe('AddToVectorIndex(js-hnsw)') + expect(removeOp.name).toBe('RemoveFromVectorIndex(js-hnsw)') + }) + + it('stamps a self-identifying provider\'s own providerId, never "HNSW"', () => { + const provider = new FakeVectorProvider('acme-diskann') + const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3]) + const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3]) + const batchOp = new BatchAddToVectorIndexOperation(provider, [{ id: 'a', vector: [1] }]) + + expect(addOp.name).toBe('AddToVectorIndex(acme-diskann)') + expect(removeOp.name).toBe('RemoveFromVectorIndex(acme-diskann)') + expect(batchOp.name).toBe('BatchAddToVectorIndex(acme-diskann)') + expect(addOp.name).not.toContain('HNSW') + expect(removeOp.name).not.toContain('HNSW') + }) + + it('honestly reports "unknown-provider" rather than guessing when a provider omits providerId', () => { + const provider = new FakeVectorProvider(undefined) + const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3]) + const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3]) + + expect(addOp.name).toBe('AddToVectorIndex(unknown-provider)') + expect(removeOp.name).toBe('RemoveFromVectorIndex(unknown-provider)') + // Never silently falls back to the JS engine's name for a provider it + // knows nothing about — that's the exact fossil-naming bug being fixed. + expect(addOp.name).not.toContain('js-hnsw') + }) + }) +}) From 3be4ba96c299b04ce12ffc57898a629c0e8eb02d Mon Sep 17 00:00:00 2001 From: David Snelling Date: Thu, 23 Jul 2026 08:53:05 -0700 Subject: [PATCH 24/29] feat: vector provider identity is a required name field (hnsw-js), rendered [vector-index:] Reconciles the vector-index rename to the ruled three-layer naming: the provider contract's identity field is now a REQUIRED readonly name (was optional providerId), self-reported and truthful, rendered as [vector-index:] where the index identifies itself and stamped into the op-name strings journals already parse (AddToVectorIndex()). The built-in JS engine names itself hnsw-js. A runtime provider instance compiled against the previous optional contract is tolerated - never crashed on, never silently mislabeled: it stamps unknown-provider and emits one loud warning naming the missing field. Graph index operations keep their static names (they never interpolate provider identity), and no public API exports a provider-routed hnsw-carrying name, so no deprecation shim is required. --- RELEASES.md | 12 ++++- src/hnsw/hnswIndex.ts | 4 +- src/plugin.ts | 30 ++++++----- src/transaction/operations/IndexOperations.ts | 29 +++++++--- tests/unit/brainy/warm.test.ts | 1 + .../vectorIndexOperations-rename.test.ts | 54 +++++++++++++------ 6 files changed, 92 insertions(+), 38 deletions(-) diff --git a/RELEASES.md b/RELEASES.md index 113b327d..89bf38e7 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -82,10 +82,20 @@ demand-load cost OFF the transaction path. `AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an algorithm that may not be the one actually running (a non-HNSW native vector provider emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist). - The parenthesized suffix names the ACTIVE backend: `js-hnsw` for the built-in engine, or + The parenthesized suffix names the ACTIVE backend: `hnsw-js` for the built-in engine, or the native provider's own identity when it self-identifies. **If you parse these op-name strings** (log processors, journal tooling), update your matcher from `AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`. +- **Provider identity is now a REQUIRED `name` field** on the vector provider contract + (`VectorIndexProvider.name`, `src/plugin.ts`) — every implementation self-reports its own + identity truthfully (its algorithm/engine), never inheriting a default. It renders as the + op-name suffix above and, wherever the vector index identifies itself in prose log lines, + as the tag `[vector-index:]`. **Native provider adoption is a one-line change**: + declare `readonly name = ''`. A provider instance that still lacks + `name` at runtime (an older native build compiled against the previous, optional field) is + never crashed on and never silently mislabeled: it stamps `unknown-provider` and emits one + loud warning naming the missing field, so the gap is discoverable instead of a permanent + fossil label in every journal line. ## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close()) diff --git a/src/hnsw/hnswIndex.ts b/src/hnsw/hnswIndex.ts index dcf3ed7b..eb2acd71 100644 --- a/src/hnsw/hnswIndex.ts +++ b/src/hnsw/hnswIndex.ts @@ -54,8 +54,8 @@ export class HnswFlushError extends Error { * acceleration provider). */ export class JsHnswVectorIndex implements VectorIndexProvider { - /** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.providerId}. */ - readonly providerId = 'js-hnsw' + /** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.name}. */ + readonly name = 'hnsw-js' private nouns: Map = new Map() /** diff --git a/src/plugin.ts b/src/plugin.ts index eeb2425a..02639955 100644 --- a/src/plugin.ts +++ b/src/plugin.ts @@ -962,20 +962,24 @@ export interface AtGenerationVectors { */ export interface VectorIndexProvider { /** - * @description OPTIONAL backend identity, stamped into the transaction - * op-name strings that surface in journals/timings (e.g. - * `AddToVectorIndex(js-hnsw)` vs `AddToVectorIndex()`) — see - * `src/transaction/operations/IndexOperations.ts`. Absent → the op-name - * string reports `unknown-provider` rather than guessing a backend name - * (guessing would resurrect the exact bug this field exists to prevent: - * an operator hunting an index that isn't the one actually running). The - * built-in JS index always sets this to `'js-hnsw'`; a native provider - * sets its own identity here (e.g. its plugin/package name) so operators - * reading a journal see the backend that actually ran, never a - * backend-specific fossil name from whichever engine wrote the original - * op classes. + * @description REQUIRED self-reported implementation identity, rendered as + * `[vector-index:]` in prose log lines and stamped into the + * transaction op-name strings that surface in journals/timings (e.g. + * `AddToVectorIndex(hnsw-js)`) — see + * `src/transaction/operations/IndexOperations.ts`. A provider must name + * itself TRUTHFULLY (its own algorithm/engine, e.g. its plugin/package + * name) and must never inherit a default — guessing a backend name would + * resurrect the exact bug this field exists to prevent: an operator + * hunting an index that isn't the one actually running. The built-in JS + * index always sets this to `'hnsw-js'`; a native provider picks its own + * string. At the TypeScript level this field is required; a runtime + * instance from an older provider compiled against the previous optional + * `providerId` contract is tolerated (never crashes, never silently + * mislabeled) — see `resolveVectorProviderId` in + * `src/transaction/operations/IndexOperations.ts`, which stamps + * `'unknown-provider'` and emits one loud warning for that case. */ - readonly providerId?: string + readonly name: string addItem(item: VectorDocument): Promise removeItem(id: string): Promise diff --git a/src/transaction/operations/IndexOperations.ts b/src/transaction/operations/IndexOperations.ts index 1c0995c6..d130bb3f 100644 --- a/src/transaction/operations/IndexOperations.ts +++ b/src/transaction/operations/IndexOperations.ts @@ -16,19 +16,34 @@ import type { Operation, RollbackAction } from '../types.js' /** * Backend identity stamped into an operation's emitted `name` string (e.g. - * `AddToVectorIndex(js-hnsw)`), resolved from the provider's own - * {@link VectorIndexProvider.providerId} when it self-identifies. + * `AddToVectorIndex(hnsw-js)`), resolved from the provider's own + * {@link VectorIndexProvider.name} self-report. * * These operation classes are backend-neutral (the vector index they wrap may * be Brainy's own JS HNSW fallback OR a native acceleration provider), but * their names surface directly in consumer-visible transaction journals and - * timings. A provider that hasn't set `providerId` resolves to - * `'unknown-provider'` rather than guessing — silently defaulting to the JS - * engine's name here is exactly the class of bug this stamping exists to - * prevent (an operator diagnosing a backend that isn't the one that ran). + * timings. `name` is REQUIRED at the TypeScript level — but a native provider + * instance compiled against the previous (pre-required) contract can still + * reach this function at runtime without it. That legacy case is tolerated, + * never crashed on and never silently mislabeled: it resolves to + * `'unknown-provider'` and emits ONE loud warning naming the missing contract + * field, so the fix (implement `name`) is discoverable rather than a silent + * fossil label in every journal line thereafter. */ +const warnedMissingName = new WeakSet() + function resolveVectorProviderId(index: VectorIndexProvider): string { - return index.providerId ?? 'unknown-provider' + const name = (index as { name?: unknown }).name + if (typeof name === 'string') return name + if (!warnedMissingName.has(index)) { + warnedMissingName.add(index) + console.warn( + '[vector-index] provider is missing the required `name` field (VectorIndexProvider.name, ' + + 'required since 8.10.0) — stamping "unknown-provider" in transaction op names until the ' + + 'provider declares its own identity.' + ) + } + return 'unknown-provider' } /** diff --git a/tests/unit/brainy/warm.test.ts b/tests/unit/brainy/warm.test.ts index 437b7785..ce213696 100644 --- a/tests/unit/brainy/warm.test.ts +++ b/tests/unit/brainy/warm.test.ts @@ -44,6 +44,7 @@ const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin * `AddToVectorIndexOperation` / `brain.warm()` call through. */ class FakeVectorProvider implements VectorIndexProvider { + readonly name = 'fake-vector-provider' readonly items = new Map() warmCalls = 0 searchCalls: Array<{ k?: number }> = [] diff --git a/tests/unit/transaction/vectorIndexOperations-rename.test.ts b/tests/unit/transaction/vectorIndexOperations-rename.test.ts index 80168346..bdfe58a3 100644 --- a/tests/unit/transaction/vectorIndexOperations-rename.test.ts +++ b/tests/unit/transaction/vectorIndexOperations-rename.test.ts @@ -6,12 +6,14 @@ * journals and timings — a fossil "HNSW" name misdirects an operator running * a native (non-HNSW) vector provider. Verifies: * 1. rollback wiring stayed byte-identical under the rename, - * 2. the emitted `name` stamps the active backend — `'js-hnsw'` for - * Brainy's own JS index, a provider's own `providerId` when it - * self-identifies, and the honest `'unknown-provider'` literal when - * neither applies (never a silently-wrong guess). + * 2. the emitted `name` stamps the active backend — `'hnsw-js'` for + * Brainy's own JS index, a provider's own required `name` when it + * self-identifies, and the tolerant-loud `'unknown-provider'` literal + * (plus one console.warn) when a runtime instance lacks `name` + * altogether (an older native provider compiled against the previous, + * optional `providerId` contract) — never a silently-wrong guess. */ -import { describe, it, expect } from 'vitest' +import { describe, it, expect, vi, afterEach } from 'vitest' import { AddToVectorIndexOperation, RemoveFromVectorIndexOperation, @@ -28,7 +30,7 @@ import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js' */ class FakeVectorProvider implements VectorIndexProvider { readonly items = new Map() - constructor(readonly providerId?: string) {} + constructor(readonly name: string) {} async addItem(item: VectorDocument): Promise { this.items.set(item.id, item.vector) @@ -115,16 +117,20 @@ describe('Vector index transaction operations — backend-neutral rename', () => }) describe('backend stamping in the emitted op name', () => { - it('stamps the real JS HNSW index as js-hnsw (self-identifying providerId)', () => { + afterEach(() => { + vi.restoreAllMocks() + }) + + it('stamps the real JS HNSW index as hnsw-js (self-identifying name)', () => { const index = new JsHnswVectorIndex() const addOp = new AddToVectorIndexOperation(index, 'id-1', [1, 2, 3]) const removeOp = new RemoveFromVectorIndexOperation(index, 'id-1', [1, 2, 3]) - expect(addOp.name).toBe('AddToVectorIndex(js-hnsw)') - expect(removeOp.name).toBe('RemoveFromVectorIndex(js-hnsw)') + expect(addOp.name).toBe('AddToVectorIndex(hnsw-js)') + expect(removeOp.name).toBe('RemoveFromVectorIndex(hnsw-js)') }) - it('stamps a self-identifying provider\'s own providerId, never "HNSW"', () => { + it('stamps a self-identifying provider\'s own name, never "HNSW"', () => { const provider = new FakeVectorProvider('acme-diskann') const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3]) const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3]) @@ -137,16 +143,34 @@ describe('Vector index transaction operations — backend-neutral rename', () => expect(removeOp.name).not.toContain('HNSW') }) - it('honestly reports "unknown-provider" rather than guessing when a provider omits providerId', () => { - const provider = new FakeVectorProvider(undefined) - const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3]) - const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3]) + it('tolerant-loud: stamps "unknown-provider" and warns exactly once when a runtime provider lacks the required `name` (an older native provider compiled against the previous optional `providerId` contract)', () => { + const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}) + // Simulate a native provider compiled before `name` was required — the + // TypeScript contract requires `name`, but `as unknown as` mirrors what + // actually reaches this code at runtime from an un-rebuilt native addon. + const legacyProvider = { + addItem: async (item: VectorDocument) => item.id, + removeItem: async () => true, + search: async () => [], + size: () => 0, + clear: () => {}, + rebuild: async () => {}, + flush: async () => 0, + getPersistMode: () => 'immediate' as const + } as unknown as VectorIndexProvider + + const addOp = new AddToVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3]) + const removeOp = new RemoveFromVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3]) expect(addOp.name).toBe('AddToVectorIndex(unknown-provider)') expect(removeOp.name).toBe('RemoveFromVectorIndex(unknown-provider)') // Never silently falls back to the JS engine's name for a provider it // knows nothing about — that's the exact fossil-naming bug being fixed. - expect(addOp.name).not.toContain('js-hnsw') + expect(addOp.name).not.toContain('hnsw-js') + // Loud, not silent — but exactly once per provider instance, not once + // per op stamped against it. + expect(warnSpy).toHaveBeenCalledTimes(1) + expect(warnSpy.mock.calls[0][0]).toContain('name') }) }) }) From 3a1efc9460617a4b7f37236772f922e408684eb6 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Thu, 23 Jul 2026 08:56:57 -0700 Subject: [PATCH 25/29] docs: project guide version line points at npm instead of a hardcoded stale number --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 568c10db..c7336a18 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md` **Brainy's current open actions:** None. MIT open-source — no platform-specific actions. -**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`) +**Current version:** run `npm view @soulcraft/brainy version` (never trust a hardcoded number here — this line went stale for months); consumer-facing changes tracked in `RELEASES.md` --- From 6ba94c8c432551575c16c4b758386f1f5912b40d Mon Sep 17 00:00:00 2001 From: David Snelling Date: Thu, 23 Jul 2026 10:01:28 -0700 Subject: [PATCH 26/29] fix(release): push the public mirror explicitly and verify the tag lands at the right commit before publishing Origin moved to the private forge; the public repo is now a mirror with an unknown sync cadence. The release flow creates the GitHub release directly, and a missing tag there would be silently created at the default-branch head - the wrong commit. The script now pushes branch+tag to the mirror itself and hard-stops if the tag's commit on the mirror differs from local, before anything irreversible (npm publish) happens. --- scripts/release.sh | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/scripts/release.sh b/scripts/release.sh index 7d860564..42f5b345 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -175,10 +175,26 @@ echo -e "${BLUE}7️⃣ Creating git tag v${NEW_VERSION}...${NC}" git tag -a "v${NEW_VERSION}" -m "Release v${NEW_VERSION}" echo -e "${GREEN}✅ Tag created${NC}\n" -# Step 9: Push to GitHub -echo -e "${BLUE}8️⃣ Pushing to GitHub...${NC}" +# Step 9: Push to origin (source of truth) and the public GitHub mirror +echo -e "${BLUE}8️⃣ Pushing to origin...${NC}" git push --follow-tags origin "$CURRENT_BRANCH" -echo -e "${GREEN}✅ Pushed to GitHub${NC}\n" +echo -e "${GREEN}✅ Pushed to origin${NC}\n" + +# The public GitHub repo is a mirror of origin with an unknown sync cadence. +# `gh release create` below targets GitHub directly: if the new tag hasn't +# reached GitHub yet, gh would CREATE it — pointed at GitHub's default-branch +# head, i.e. the wrong commit. Push branch+tag to GitHub explicitly, then +# verify the tag resolves there to the same commit before any release is cut. +GITHUB_URL="https://github.com/soulcraftlabs/brainy.git" +echo -e "${BLUE}8️⃣½ Pushing to the public GitHub mirror...${NC}" +git push --follow-tags "$GITHUB_URL" "$CURRENT_BRANCH" +LOCAL_TAG_SHA="$(git rev-parse "v${NEW_VERSION}^{}")" +GITHUB_TAG_SHA="$(git ls-remote --tags "$GITHUB_URL" "v${NEW_VERSION}^{}" | cut -f1)" +if [ "$LOCAL_TAG_SHA" != "$GITHUB_TAG_SHA" ]; then + echo -e "${RED}❌ Tag v${NEW_VERSION} on GitHub (${GITHUB_TAG_SHA:-absent}) does not match local (${LOCAL_TAG_SHA}) — aborting before npm publish. Fix the mirror, then re-run.${NC}" + exit 1 +fi +echo -e "${GREEN}✅ GitHub mirror has the tag at the right commit${NC}\n" # Step 10: Publish to npm echo -e "${BLUE}9️⃣ Publishing to npm (dist-tag: ${NPM_TAG})...${NC}" From 9a99a7b96210e7bf7fd87e86a05876ddd0459b60 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Thu, 23 Jul 2026 10:22:34 -0700 Subject: [PATCH 27/29] =?UTF-8?q?docs:=20adoption=20storefront=20=E2=80=94?= =?UTF-8?q?=20contributing=20guide,=20security=20policy,=20README=20suppor?= =?UTF-8?q?t=20+=20cor=20section?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CONTRIBUTING.md | 320 +++++++----------------------------------------- README.md | 24 ++-- SECURITY.md | 36 ++++++ 3 files changed, 98 insertions(+), 282 deletions(-) create mode 100644 SECURITY.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ab8a0246..ef9c4a51 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,298 +1,70 @@ # Contributing to Brainy -Thank you for your interest in contributing to Brainy! This document provides guidelines and instructions for contributing to the project. +Brainy is MIT-licensed and genuinely open to outside contributions. This page +is the honest, current path — please don't rely on older instructions you +may find elsewhere in the repo's history. -## Code of Conduct +## Where the project lives -By participating in this project, you agree to abide by our Code of Conduct: -- Be respectful and inclusive -- Welcome newcomers and help them get started -- Focus on constructive criticism -- Respect differing viewpoints and experiences +The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraft/brainy**. +It's anonymously readable and cloneable — no account needed to browse, clone, +or build. -## How to Contribute +**github.com/soulcraftlabs/brainy** is a public read-only mirror. It's a fine +place to read code or star the project, but issues and pull requests opened +there won't be picked up — please use one of the paths below instead. -### Reporting Issues +## How to contribute -Before creating an issue, please check existing issues to avoid duplicates. +**Found a bug, or have an idea?** Email **brainy@soulcraft.com**. No account, +no ceremony — you'll get a receipt, and it goes to a human. -When creating an issue, include: -- Clear, descriptive title -- Detailed description of the problem -- Steps to reproduce -- Expected vs actual behavior -- System information (OS, Node version, Brainy version) -- Code examples if applicable +**Want to send a patch?** Two ways, both first-class: -### Suggesting Features +- **Email a patch.** Run `git format-patch` against your change and email the + output to **brainy@soulcraft.com**. This is a genuinely supported path, not + a fallback — plenty of good contributions arrive this way. +- **Open a pull request on the forge.** Request an account at + **source.soulcraft.com** (registration is request-with-approval, so allow + a little lag), clone, push a branch, and open a PR there. Maintainers + review and land it. -Feature requests are welcome! Please provide: -- Clear use case -- Proposed API/interface -- Examples of how it would work -- Any potential challenges or considerations +Either way, for anything beyond a small fix, opening an issue first (email is +fine) to talk through the approach saves everyone rework. -### Pull Requests +## Development setup -#### Before Starting - -1. Check existing issues and PRs -2. Open an issue to discuss significant changes -3. Fork the repository -4. Create a feature branch from `main` - -#### Development Setup - -**Quick Setup (Recommended):** ```bash -# Clone your fork -git clone https://github.com/your-username/brainy.git +git clone https://source.soulcraft.com/soulcraft/brainy.git cd brainy - -# Run setup script (installs all dependencies including Rust) -./scripts/setup-dev.sh -``` - -**Manual Setup:** -```bash -# Clone your fork -git clone https://github.com/your-username/brainy.git -cd brainy - -# Install system dependencies (Ubuntu/Debian) -sudo apt-get install -y build-essential pkg-config libssl-dev - -# Install Rust (for WASM embedding engine) -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source ~/.cargo/env -rustup target add wasm32-unknown-unknown -cargo install wasm-pack - -# Install Node.js dependencies npm install - -# Build Candle WASM embedding engine -npm run build:candle - -# Build TypeScript npm run build - -# Run tests npm test ``` -#### Making Changes +Tests run on [Vitest](https://vitest.dev/). `npm test` runs the unit suite; +see `package.json` for `test:integration`, `test:coverage`, and friends. -1. **Follow the code style** - - TypeScript for all source code - - Clear variable and function names - - Comments for complex logic - - JSDoc for public APIs +## Standards -2. **Write tests** - - Add tests for new features - - Update tests for changes - - Ensure all tests pass +- **Strict TypeScript.** No `any` escape hatches to dodge the type checker. +- **Tests exercise real behavior.** No mocking away the thing you're supposed + to be testing. +- **No stubs, no TODO-code.** If something can't be finished, say so and + leave it out — don't merge a placeholder. +- **JSDoc on every exported function, class, and type.** +- **[Conventional Commits](https://www.conventionalcommits.org/).** `feat:`, + `fix:`, `docs:`, `perf:`, `refactor:`, `test:`, `chore:`. Never + `BREAKING CHANGE` in a commit message — major version bumps are a separate, + deliberate decision. +- **Performance claims are measured or labeled projected.** If a PR or its + description states a number, cite the benchmark that produced it (see + [docs/performance-envelopes.md](docs/performance-envelopes.md) for the + pattern). Don't state an estimate as if it were measured. -3. **Update documentation** - - Update README if needed - - Add/update API documentation - - Include examples +## License -#### Commit Guidelines +Brainy is [MIT licensed](LICENSE). Contributions are accepted under the same +license — there's no CLA to sign. -Follow conventional commits format: - -``` -type(scope): description - -[optional body] - -[optional footer] -``` - -Types: -- `feat`: New feature -- `fix`: Bug fix -- `docs`: Documentation changes -- `style`: Code style changes -- `refactor`: Code refactoring -- `perf`: Performance improvements -- `test`: Test changes -- `chore`: Build/tooling changes - -Examples: -```bash -feat(triple): add graph traversal depth limit -fix(storage): handle concurrent write conflicts -docs(api): update search method documentation -``` - -#### Submitting PR - -1. Push to your fork -2. Create PR against `main` branch -3. Fill out PR template -4. Ensure CI checks pass -5. Wait for review - -### Testing - -#### Running Tests - -```bash -# Run all tests -npm test - -# Run specific test file -npm test tests/core.test.ts - -# Run with coverage -npm run test:coverage - -# Watch mode -npm run test:watch -``` - -#### Writing Tests - -```typescript -import { describe, it, expect } from 'vitest' -import { Brainy } from '../src' - -describe('Feature Name', () => { - it('should do something specific', async () => { - const brain = new Brainy() - await brain.init() - - // Test implementation - const result = await brain.search("test") - - expect(result).toBeDefined() - expect(result.length).toBeGreaterThan(0) - }) -}) -``` - -## Architecture Guidelines - -### Adding New Features - -1. **Check existing functionality** - - Review `ARCHITECTURE.md` - - Check if similar features exist - - Consider if it should be an augmentation - -2. **Design considerations** - - Maintain backward compatibility - - Consider performance impact - - Think about all storage adapters - - Plan for extensibility - -3. **Implementation checklist** - - [ ] Core functionality - - [ ] Tests (unit and integration) - - [ ] Documentation - - [ ] TypeScript types - - [ ] Examples - - [ ] Performance benchmarks (if applicable) - -### Creating Augmentations - -Augmentations extend Brainy's functionality: - -```typescript -import { BrainyAugmentation } from '../types' - -export class MyAugmentation extends BrainyAugmentation { - name = 'MyAugmentation' - - async onInit(brain: Brainy): Promise { - // Initialize augmentation - } - - async onAdd(item: any, brain: Brainy): Promise { - // Process before adding - return item - } - - async onSearch(query: any, results: any[], brain: Brainy): Promise { - // Process search results - return results - } -} -``` - -### Performance Considerations - -- Use batch operations where possible -- Implement caching strategically -- Consider memory usage -- Profile performance impacts -- Add benchmarks for critical paths - -## Documentation - -### API Documentation - -Use JSDoc for all public APIs: - -```typescript -/** - * Searches for similar items using vector similarity - * @param query - Search query (text or vector) - * @param options - Search options - * @returns Array of search results with scores - * @example - * ```typescript - * const results = await brain.search("machine learning", { limit: 10 }) - * ``` - */ -async search(query: string | Vector, options?: SearchOptions): Promise { - // Implementation -} -``` - -### Examples - -Add examples for new features: - -```typescript -// examples/feature-name.ts -import { Brainy } from 'brainy' - -async function exampleUsage() { - const brain = new Brainy() - await brain.init() - - // Show feature usage - // Include comments explaining what's happening - // Handle errors appropriately -} - -exampleUsage().catch(console.error) -``` - -## Release Process - -1. **Version bump**: Follow semantic versioning -2. **Update CHANGELOG**: Document all changes -3. **Run tests**: Ensure all tests pass -4. **Build**: Generate distribution files -5. **Tag**: Create git tag for version -6. **Publish**: Release to npm - -## Getting Help - -- **Discord**: Join our community -- **Issues**: Ask questions on GitHub -- **Discussions**: Share ideas and get feedback - -## Recognition - -Contributors will be recognized in: -- CHANGELOG.md for their contributions -- README.md contributors section -- GitHub contributors page - -Thank you for contributing to Brainy! 🧠 \ No newline at end of file +Thank you for considering a contribution. diff --git a/README.md b/README.md index d1342f52..2fc42060 100644 --- a/README.md +++ b/README.md @@ -23,8 +23,9 @@ Quick start · One query · Features · - Scale with Cor · - Docs + Scale with Cor · + Docs · + Support

--- @@ -172,9 +173,11 @@ await brain.vfs.search('React components with hooks') // semantic file **[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)** -## From laptop to hundreds of millions +## When you outgrow Brainy -Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**: +Brainy's pure-TypeScript engines carry real workloads a long way on their own — see the measured, per-operation numbers (not marketing figures) in **[docs/performance-envelopes.md](docs/performance-envelopes.md)** for what to expect, unaccelerated, on plain filesystem storage. + +When a deployment needs native-scale vector/graph performance — memory-mapped indexes that don't need your dataset in RAM, billion-scale ambitions — add the native engine. **The API doesn't change:** ```bash npm install @soulcraft/cor @@ -187,13 +190,14 @@ await brain.init() // @soulcraft/cor detected — same code, native engines un Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate. -Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both. +Open core, commercial accelerator: Brainy is MIT and complete on its own — Cor is more headroom for when you need it, not capability held back to sell you later. Licensing and support: **cor@soulcraft.com**. ## Performance +- Per-operation p50/p95 at 1k and 10k entities, pure-JS floor, measured and re-run every release that touches a measured path: **[docs/performance-envelopes.md](docs/performance-envelopes.md)**. - JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41). - Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan. -- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)** +- Capacity planning and architecture: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)** ## Use cases @@ -212,6 +216,10 @@ Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is **Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use. -## Contributing & license +## Support & community -Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors. +- **Bugs and ideas** → **brainy@soulcraft.com** — no account needed, you'll get a receipt. +- **Security reports** → **security@soulcraft.com** — see **[SECURITY.md](SECURITY.md)**. +- **Contributing** → see **[CONTRIBUTING.md](CONTRIBUTING.md)**. + +MIT © Brainy Contributors. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..1f3c4732 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,36 @@ +# Security Policy + +## Reporting a vulnerability + +Email **security@soulcraft.com**. That's the one door for security reports +across the company, and it works the same way for Brainy: every report is +read by a human, you'll get a private receipt, and we'll work with you on +coordinated disclosure — please don't open a public issue for anything +that isn't already public. + +Include what you'd want if you were on the other end: affected version, +how to reproduce, and what you think the impact is. If you have a patch or +a suggested fix, send it along — it's welcome but not required. + +There is no bounty program today. We're saying that plainly so you know +what to expect going in. + +## Response time + +We respond as fast as truth allows. That means: no fixed SLA, no promise of +a reply within a specific number of hours — but a real report from a real +person gets read promptly and taken seriously. If you haven't heard anything +in a reasonable stretch, a follow-up email is completely fine. + +## Supported versions + +The latest `8.x` minor release line receives security fixes. If you're +running an older major version, please upgrade before reporting — we can't +commit to backporting fixes to unsupported lines. + +## Scope + +This policy covers the `@soulcraft/brainy` package itself — the code in +this repository. If you're evaluating a deployment that also uses +`@soulcraft/cor`, report issues in that package the same way, to the same +address; we'll route internally. From 4c8e384bc8271044828eff721abfda2334d9a911 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Thu, 23 Jul 2026 10:46:18 -0700 Subject: [PATCH 28/29] chore(release): 8.10.0 --- CHANGELOG.md | 9 +++++++++ package-lock.json | 4 ++-- package.json | 2 +- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fd0de54e..b6dd91fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,15 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. +### [8.10.0](https://github.com/soulcraftlabs/brainy/compare/v8.9.0...v8.10.0) (2026-07-23) + +- docs: adoption storefront — contributing guide, security policy, README support + cor section (9a99a7b) +- fix(release): push the public mirror explicitly and verify the tag lands at the right commit before publishing (6ba94c8) +- docs: project guide version line points at npm instead of a hardcoded stale number (3a1efc9) +- feat: vector provider identity is a required name field (hnsw-js), rendered [vector-index:] (3be4ba9) +- feat: warm contract (warm/warmOnOpen/provider warm hook), configurable transact budget floor, backend-neutral vector index op names (55b867c) + + ### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19) - docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78) diff --git a/package-lock.json b/package-lock.json index fb9262e9..37aeb81d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraft/brainy", - "version": "8.9.0", + "version": "8.10.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraft/brainy", - "version": "8.9.0", + "version": "8.10.0", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index 7366ce98..e4bc8144 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraft/brainy", - "version": "8.9.0", + "version": "8.10.0", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", "module": "dist/index.js", From 415e824a1da44a1109f54818721289f5e9a42af2 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Thu, 23 Jul 2026 11:09:43 -0700 Subject: [PATCH 29/29] =?UTF-8?q?chore:=20the=20forge=20is=20the=20address?= =?UTF-8?q?=20=E2=80=94=20retire=20the=20archived=20mirror=20from=20every?= =?UTF-8?q?=20live=20surface?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ruled today: the project's one public home is source.soulcraft.com. The old public repo is archived history and no longer part of any release. - package.json repository/homepage/bugs now point at the forge (this is what the npm page links as Repository/Homepage/Issues) - README CI badge reads the forge pipeline; CONTRIBUTING drops the mirror paragraph (forge account or email patch were already the ruled contribution paths) - release.sh: mirror push + external release step removed; publishes go forge-first (box-held write token, temp userconfig so the token never hits argv; a forge-publish failure aborts before the storefront so the pair can never diverge), then npmjs with the scope-override pin (the fleet npmrc maps @soulcraft to the forge and scope mappings beat --registry); release page created via forge API when a token is present, loud skip otherwise; changelog compare links point home - dead external CI workflow removed (.forgejo/workflows/ci.yml is the live pipeline) Historical CHANGELOG links to the archive stay as written - history is history and the archive serves them read-only. --- .github/workflows/ci.yml | 40 ---------------------- CONTRIBUTING.md | 4 --- README.md | 2 +- package.json | 6 ++-- scripts/release.sh | 71 +++++++++++++++++++++++++--------------- 5 files changed, 48 insertions(+), 75 deletions(-) delete mode 100644 .github/workflows/ci.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml deleted file mode 100644 index cdb2ab14..00000000 --- a/.github/workflows/ci.yml +++ /dev/null @@ -1,40 +0,0 @@ -name: CI - -on: - push: - pull_request: - -jobs: - node: - name: Node ${{ matrix.node-version }} - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - node-version: ['22', '24'] - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: ${{ matrix.node-version }} - cache: npm - - run: npm ci - - run: npm run test:unit - - bun: - name: Bun (latest) - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: '22' - cache: npm - - uses: oven-sh/setup-bun@v2 - with: - bun-version: latest - - run: npm ci - # test:bun imports the built dist/, so build first. - - run: npm run build - # Bun as a runtime is the supported Bun story (`bun add` / `bun run`). - - run: npm run test:bun diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ef9c4a51..d277091d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,10 +10,6 @@ The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraft/bra It's anonymously readable and cloneable — no account needed to browse, clone, or build. -**github.com/soulcraftlabs/brainy** is a public read-only mirror. It's a fine -place to read code or star the project, but issues and pull requests opened -there won't be picked up — please use one of the paths below instead. - ## How to contribute **Found a bug, or have an idea?** Email **brainy@soulcraft.com**. No account, diff --git a/README.md b/README.md index 2fc42060..2caf6493 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@

npm version npm downloads - CI + CI Documentation MIT License TypeScript diff --git a/package.json b/package.json index e4bc8144..a3ece83c 100644 --- a/package.json +++ b/package.json @@ -128,13 +128,13 @@ "publishConfig": { "access": "public" }, - "homepage": "https://github.com/soulcraftlabs/brainy", + "homepage": "https://source.soulcraft.com/soulcraft/brainy", "bugs": { - "url": "https://github.com/soulcraftlabs/brainy/issues" + "url": "https://source.soulcraft.com/soulcraft/brainy/issues" }, "repository": { "type": "git", - "url": "git+https://github.com/soulcraftlabs/brainy.git" + "url": "git+https://source.soulcraft.com/soulcraft/brainy.git" }, "files": [ "dist/**/*.js", diff --git a/scripts/release.sh b/scripts/release.sh index 42f5b345..43fa50bd 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -142,7 +142,7 @@ else fi # Create new changelog entry -CHANGELOG_ENTRY="### [${NEW_VERSION}](https://github.com/soulcraftlabs/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d)) +CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraft/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d)) ${COMMITS} " @@ -175,42 +175,59 @@ echo -e "${BLUE}7️⃣ Creating git tag v${NEW_VERSION}...${NC}" git tag -a "v${NEW_VERSION}" -m "Release v${NEW_VERSION}" echo -e "${GREEN}✅ Tag created${NC}\n" -# Step 9: Push to origin (source of truth) and the public GitHub mirror +# Step 9: Push to origin — the forge is the one home (ruled 2026-07-23; the +# old public GitHub repo is archived history, no longer part of any release). echo -e "${BLUE}8️⃣ Pushing to origin...${NC}" git push --follow-tags origin "$CURRENT_BRANCH" echo -e "${GREEN}✅ Pushed to origin${NC}\n" -# The public GitHub repo is a mirror of origin with an unknown sync cadence. -# `gh release create` below targets GitHub directly: if the new tag hasn't -# reached GitHub yet, gh would CREATE it — pointed at GitHub's default-branch -# head, i.e. the wrong commit. Push branch+tag to GitHub explicitly, then -# verify the tag resolves there to the same commit before any release is cut. -GITHUB_URL="https://github.com/soulcraftlabs/brainy.git" -echo -e "${BLUE}8️⃣½ Pushing to the public GitHub mirror...${NC}" -git push --follow-tags "$GITHUB_URL" "$CURRENT_BRANCH" -LOCAL_TAG_SHA="$(git rev-parse "v${NEW_VERSION}^{}")" -GITHUB_TAG_SHA="$(git ls-remote --tags "$GITHUB_URL" "v${NEW_VERSION}^{}" | cut -f1)" -if [ "$LOCAL_TAG_SHA" != "$GITHUB_TAG_SHA" ]; then - echo -e "${RED}❌ Tag v${NEW_VERSION} on GitHub (${GITHUB_TAG_SHA:-absent}) does not match local (${LOCAL_TAG_SHA}) — aborting before npm publish. Fix the mirror, then re-run.${NC}" +# Step 10: Publish — forge FIRST (home), npmjs second (the world's storefront). +# The fleet-wide ~/.npmrc maps the @soulcraft scope to the forge registry, and +# a scope mapping BEATS `--registry` on the command line — so each publish +# names its registry via the scope override explicitly. Nothing implicit. +FORGE_NPM_REG="https://source.soulcraft.com/api/packages/soulcraft/npm/" +FORGE_NPM_TOKEN_FILE="$HOME/.config/soulcraft/npm-publish-brainy.token" +echo -e "${BLUE}9️⃣ Publishing to the forge registry (home)...${NC}" +if [ -f "$FORGE_NPM_TOKEN_FILE" ]; then + TMPRC="$(mktemp)" + chmod 600 "$TMPRC" + { + echo "@soulcraft:registry=${FORGE_NPM_REG}" + echo "//source.soulcraft.com/api/packages/soulcraft/npm/:_authToken=$(cat "$FORGE_NPM_TOKEN_FILE")" + } > "$TMPRC" + if npm publish --tag "$NPM_TAG" --userconfig "$TMPRC"; then + echo -e "${GREEN}✅ Published to the forge${NC}\n" + else + rm -f "$TMPRC" + echo -e "${RED}❌ Forge publish FAILED — aborting before npmjs so the pair never diverges. Fix and re-run.${NC}" + exit 1 + fi + rm -f "$TMPRC" +else + echo -e "${RED}❌ Forge publish token missing (${FORGE_NPM_TOKEN_FILE}) — aborting. The forge is home; publish it first or restage the token.${NC}" exit 1 fi -echo -e "${GREEN}✅ GitHub mirror has the tag at the right commit${NC}\n" -# Step 10: Publish to npm -echo -e "${BLUE}9️⃣ Publishing to npm (dist-tag: ${NPM_TAG})...${NC}" -npm publish --tag "$NPM_TAG" +echo -e "${BLUE}9️⃣½ Publishing to npmjs (storefront, dist-tag: ${NPM_TAG})...${NC}" +npm publish --tag "$NPM_TAG" "--@soulcraft:registry=https://registry.npmjs.org/" # Brainy is the only PUBLIC @soulcraft package — verify visibility after every publish. -npm access get status @soulcraft/brainy || true -echo -e "${GREEN}✅ Published to npm${NC}\n" +npm access get status @soulcraft/brainy "--@soulcraft:registry=https://registry.npmjs.org/" || true +echo -e "${GREEN}✅ Published to npmjs${NC}\n" -# Step 11: Create GitHub release -echo -e "${BLUE}🔟 Creating GitHub release...${NC}" -if [ "$PRERELEASE" = true ]; then - gh release create "v${NEW_VERSION}" --generate-notes --prerelease +# Step 11: Release object on the forge (presentational — the tag, CHANGELOG, +# and RELEASES.md are the record; this just gives the forge UI a release page). +echo -e "${BLUE}🔟 Creating forge release...${NC}" +if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then + if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \ + -H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \ + -d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then + echo -e "${GREEN}✅ Forge release created${NC}\n" + else + echo -e "${RED}⚠️ Forge release API call failed — tag + CHANGELOG remain the record; create the release page via the forge UI if wanted${NC}\n" + fi else - gh release create "v${NEW_VERSION}" --generate-notes + echo -e "${RED}⚠️ FORGEJO_RELEASE_TOKEN unset — no release page created; tag + CHANGELOG remain the record${NC}\n" fi -echo -e "${GREEN}✅ GitHub release created${NC}\n" # Step 12: Push public docs to the soulcraft.com docs ingest door # (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when @@ -229,4 +246,4 @@ echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}" echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" echo "" echo -e "📦 npm: ${BLUE}https://www.npmjs.com/package/@soulcraft/brainy/v/${NEW_VERSION}${NC}" -echo -e "🐙 GitHub: ${BLUE}https://github.com/soulcraftlabs/brainy/releases/tag/v${NEW_VERSION}${NC}" +echo -e "🏠 Forge: ${BLUE}https://source.soulcraft.com/soulcraft/brainy/releases/tag/v${NEW_VERSION}${NC}"