Open Brainy — the MIT reference engine (@soulcraftlabs/brainy). https://soulcraft.com
Find a file
David Snelling 15d4f65dcf perf(open): the pending-embed fold is bounded by a checkpoint of the SET, not an empty-only mark
The low-water mark shipped in 10.4.9 can only be written when the pending
set is EMPTY, because it carries no set — it means "everything at or below
G is consumed". A brain holding even one id that never lands (an embed that
keeps failing, a data-less row reaped in memory only and re-folded every
open) never drains, so it never writes a mark, so the bound never engaged on
exactly the brains whose fold is expensive: `recover-pending-embeds` re-read
the WHOLE fact log at every open, on the open's foreground.

_system/pending_embeds_checkpoint.json carries the set: { generation,
pending, writtenAt } = "as of durable generation G the pending set was
exactly this list". Open seeds the set from the list and scans from G + 1,
so the fold is O(facts since G) whether or not the set ever drains. Measured
on a 301-row brain with one stuck id: 302 facts read before, 0 after; at 601
rows, 602 before, 0 after — same pending set both ways.

THE DURABILITY LAW, by construction. A checkpoint at head H taken while the
facts up to H are still buffered would be read back after a crash that
truncated the tail: an `embed.landed` in a truncated fact would be gone from
the log while the checkpoint still recorded its id as landed, and its
landing vector went with the fact — a LOST VECTOR. So a capture is refused
unless `0 < head <= committed`, the manifest watermark below which
FactLog.open() never truncates and which the group-commit flush only
advances after fsyncing the log. The (generation, set) pair is taken in one
synchronous instant with no await between reading the generations and
snapshotting the set. The one remaining asymmetry runs the safe way: an id
enqueued in memory whose marker lands at G+1 is captured as pending at G —
one idempotent re-embed, never a loss.

Written at clean close (inside closeDurableSteps, after the generation
store's own close flushed the log and advanced the manifest), at
drain-to-empty, and on a cadence of max(64, ceil(|pending| / 64))
transitions while open — an interval that holds the mechanism's amortized
cost at <= 64 ids written per transition however large the backlog grows, so
the cure cannot reintroduce the defect class it fixes. No timer, no knob.
The debt stays armed across attempts the durability law refuses, so a write
burst does not skip a checkpoint, it defers it.

Degradation is loud and always toward a LONGER scan: a torn checkpoint
throws typed on read (the adapter's tmp+rename write means it can never
parse into a partial list) and a malformed one is refused whole, both
falling back to the low-water mark — still written, still read — and then to
generation 1. The fold narrates which bound applied and how many facts it
read, on every open, so a bound that stops engaging is visible instead of
silent.

The worker's orphan reap splits: a row that is GONE clears durably (its
tombstone is in the log, or its create never was), while a present-but
data-less row keeps clearing in memory only and is carried in the checkpoint
list, so the bounded fold and a full fold from generation 1 agree exactly.
The crash-recovery contract is unchanged: the fold stays on the open's
foreground, markers re-armed when open() returns.
2026-09-02 10:20:38 -07:00
.claude/skills chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
.forgejo/workflows ci(delta-gate): add a push fallback trigger alongside workflow_dispatch 2026-09-02 09:36:46 -07:00
assets/models/all-MiniLM-L6-v2 feat: migrate embeddings to Candle WASM + remove semantic type inference 2026-01-06 12:52:34 -08:00
bin chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
docs docs: the contract manifest stands alone; public docs describe this engine only 2026-08-28 12:08:19 -07:00
examples chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
integrations chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
scripts build(release): the docs-push step retires — this engine documents itself in its own repository 2026-08-31 10:47:08 -07:00
src perf(open): the pending-embed fold is bounded by a checkpoint of the SET, not an empty-only mark 2026-09-02 10:20:38 -07:00
tests perf(open): a sealed segment the manifest proves is below the bound is never read 2026-09-02 10:07:30 -07:00
.aiignore feat: add distributed scaling and enterprise features for v3 2025-09-08 14:26:09 -07:00
.dockerignore feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
.gitignore fix(ci): commit the prebuilt wasm pkg + build before test:bun (green CI on fresh clone) 2026-07-01 14:40:11 -07:00
.npmignore chore: Add .npmignore to exclude models from npm package 2025-08-26 13:37:44 -07:00
.nvmrc feat: update Node.js requirements to 22 LTS for ONNX compatibility 2025-08-28 16:05:14 -07:00
brainy.png 🧠 Brainy 2.0.0 - Zero-Configuration AI Database with Triple Intelligence™ 2025-08-26 12:32:21 -07:00
bun.lock chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
CHANGELOG.md chore(release): 10.4.9 2026-09-02 08:20:58 -07:00
CLAUDE.md chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
CONTRIBUTING.md docs: the contract manifest stands alone; public docs describe this engine only 2026-08-28 12:08:19 -07:00
docker-compose.yml feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
Dockerfile feat: Brainy 3.0 - Production-ready Triple Intelligence database 2025-09-11 16:23:32 -07:00
eslint.config.js chore: enforce consistent coding style and semicolon removal 2025-09-29 09:50:59 -07:00
LICENSE 🧠 Brainy 2.0.0 - Zero-Configuration AI Database with Triple Intelligence™ 2025-08-26 12:32:21 -07:00
package-lock.json chore(release): 10.4.9 2026-09-02 08:20:58 -07:00
package.json chore(release): 10.4.9 2026-09-02 08:20:58 -07:00
README.md docs: repository links point at soulcraftlabs/open-brainy — the soulcraft/brainy path becomes the native engine's repo tonight 2026-08-27 17:26:44 -07:00
RELEASES.md docs(releases): 10.4.4 consumer notes — correctness and observability, with the performance line stated exactly 2026-08-28 12:05:46 -07:00
SECURITY.md chore: rename to @soulcraftlabs/brainy for Open Brainy on The Source 2026-08-27 17:07:09 -07:00
tsconfig.cli.json feat: complete CLI with VFS, data management, and Triple Intelligence search 2025-09-29 16:57:14 -07:00
tsconfig.json chore(8.0): ES2023 target + drop DOM lib + downlevelIteration (config truth-up) 2026-07-01 10:58:04 -07:00
vitest.config.memory.ts 🧠 Brainy 2.0.0 - Zero-Configuration AI Database with Triple Intelligence™ 2025-08-26 12:32:21 -07:00
vitest.config.ts 🧠 Brainy 2.0.0 - Zero-Configuration AI Database with Triple Intelligence™ 2025-08-26 12:32:21 -07:00

Brainy

Brainy

Three database paradigms. One API. Zero configuration.
The in-process knowledge database for TypeScript — vector search, graph traversal,
and metadata filtering unified in a single query.

Package on The Source Repository CI Documentation MIT License TypeScript

Quick start · One query · Features · Scale with Cor · Docs · Support


Open Brainy is the MIT engine — the open API, client library, types, and protocol; an openly specified canonical on-disk format; and this TypeScript reference engine, scoped as a single-node engine for stores up to roughly one million rows. @soulcraft/brainy 10.4.2 was the last release under the old package name — the name passes to the native engine, Brainy, at 11.0.0: the same API over the same open format at production scale, and it requires a license.

Built because we were tired of stitching a vector store to a graph database to a document store — and spending weeks on plumbing before writing a line of business logic. Brainy indexes every fact three ways at once and lets one call query them together:

You write Brainy indexes it as You query it with
data: 'Ada wrote the first program' a 384-dim vector (local embedding — no API key) find({ query: 'computing pioneers' })
metadata: { field: 'CS', year: 1843 } structured fields (O(1) exact, O(log n) range) find({ where: { year: { lessThan: 1900 } } })
relate({ from: ada, to: babbage }) a typed, directed graph edge find({ connected: { to: babbage, depth: 2 } })

It runs inside your process — no server, no Docker, nothing to operate — and persists to plain files you can snapshot with a hard link.

New here?What is Brainy? — plain-language overview, no jargon

Quick start

bun add @soulcraftlabs/brainy        # Bun ≥ 1.1 — recommended
npm install @soulcraftlabs/brainy    # Node.js ≥ 22

Registry: add @soulcraftlabs:registry=https://source.soulcraft.com/api/packages/soulcraftlabs/npm/ to your .npmrc (anonymous read).

import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'

const brain = new Brainy()                       // in-memory; one line swaps to disk
await brain.init()

// Text auto-embeds locally; metadata auto-indexes
const react = await brain.add({
  data: 'React is a JavaScript library for building user interfaces',
  type: NounType.Concept,
  subtype: 'library',
  metadata: { category: 'frontend', year: 2013 }
})

const next = await brain.add({
  data: 'Next.js is a React framework with server-side rendering',
  type: NounType.Concept,
  subtype: 'framework',
  metadata: { category: 'frontend', year: 2016 }
})

await brain.relate({ from: next, to: react, type: VerbType.DependsOn, subtype: 'runtime' })

One query, three engines

const results = await brain.find({
  query: 'modern frontend frameworks',           // vector — what it means
  where: { year: { greaterThan: 2015 } },        // metadata — what it is
  connected: { to: react, depth: 2 }             // graph — what it touches
})

Every clause is optional; any combination composes. Under the hood Brainy plans the query across an HNSW vector index, a roaring-bitmap field index, and an adjacency graph index — and re-validates every result against your predicate before returning it, so a corrupt index can never hand you a wrong answer.

Feature tour

The database is a value

Pin it, rewind it, fork it. Snapshot isolation without a server.

const db = brain.now()                           // pin current state — O(1)

await brain.transact([                           // atomic all-or-nothing, CAS-guarded
  { op: 'update', id: order, metadata: { status: 'paid' } },
  { op: 'relate', from: invoice, to: order, type: VerbType.References, subtype: 'billing' }
], { ifAtGeneration: db.generation })

await db.get(order)                              // still 'pending' — pinned forever
await brain.get(order)                           // 'paid' — live

const lastWeek = await brain.asOf(Date.now() - 7 * 86_400_000)   // full query surface, past state
const whatIf = await db.with([{ op: 'remove', id: order }])      // speculative — never touches disk
await brain.now().persist('/backups/today')                      // instant hard-link snapshot

Consistency model · Snapshots & time travel

Local embeddings — no API keys

Strings embed on-device with a bundled MiniLM model (WASM). Semantic search works offline, in CI, and on air-gapped machines, at zero cost per call. Hybrid keyword + semantic ranking is the default:

await brain.find({ query: 'David Smith' })                            // auto: text + semantic
await brain.find({ query: 'AI concepts', searchMode: 'semantic' })    // semantic only

A typed graph, not a bag of edges

42 entity types × 127 relationship types form a shared vocabulary for any domain — healthcare (Patient → diagnoses → Condition), finance (Account → transfers → Transaction), yours. Your own taxonomy layers on with subtype, enforced at write time:

await brain.add({ data: 'Avery Brooks', type: NounType.Person, subtype: 'employee' })

brain.counts.bySubtype(NounType.Person)          // O(1) — { employee: 12, customer: 847 }
brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })

Type system · Subtypes & facets

Graph analytics built in

await brain.graph.rank()                          // which entities matter most (centrality)
await brain.graph.communities()                   // natural clusters
await brain.graph.path(a, b)                      // how two things connect
await brain.graph.subgraph([seed], { depth: 2 })  // bounded neighborhood → { nodes, edges }
await brain.graph.export()                        // whole graph, one O(N+E) streaming pass

Write-time aggregations

SUM / COUNT / AVG / MIN / MAX with GROUP BY and time windows, maintained incrementally on every write — reads are O(1) lookups, not scans. Aggregation guide

Import anything

await brain.import('customers.csv')
await brain.import('sales.xlsx')                              // every sheet
await brain.import('research-paper.pdf')                      // tables extracted
await brain.import('https://api.example.com/data.json')

Entities auto-classify on the way in; brain.extractEntities(text) exposes the same NER ensemble directly. Import guide

A filesystem that understands content

await brain.vfs.writeFile('/docs/readme.md', 'Project documentation')
await brain.vfs.search('React components with hooks')          // semantic file search

VFS quick start

Operations-grade by default

  • Single-writer, many-reader — an exclusive lock protects the data directory; Brainy.openReadOnly() and the brainy inspect CLI examine a live brain from another process, safely.
  • Self-upgrading data files — a 7.x brain opens under 8.x and migrates itself behind an observable lock (getIndexStatus().migration), with an automatic pre-upgrade backup. No migration scripts.
  • No silent wrong answers — cold-open guards self-heal or throw typed errors (MetadataIndexNotReadyError, GraphIndexNotReadyError); they never return [] for data that exists.

Multi-process model · Inspection guide

When you outgrow Brainy

Brainy's pure-TypeScript engines carry real workloads a long way on their own — see the measured, per-operation numbers (not marketing figures) in docs/performance-envelopes.md for what to expect, unaccelerated, on plain filesystem storage.

When a deployment needs native-scale vector/graph performance — memory-mapped indexes that don't need your dataset in RAM, billion-scale ambitions — add the native engine. The API doesn't change:

npm install @soulcraft/cor
const brain = new Brainy({ storage: { type: 'filesystem', path: './data' } })
await brain.init()   // @soulcraft/cor detected — same code, native engines underneath

Installing the package is the opt-in: if @soulcraft/cor is present, it loads and announces itself in the init log; if it's present but broken, init() throws — an installed accelerator never silently vanishes behind the JS engines. Opt out with plugins: [], or pin exactly what loads with plugins: ['@soulcraft/cor']. @soulcraft/cor (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured 0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors in Cor's release gate.

Open core, commercial accelerator: Brainy is MIT and complete on its own — Cor is more headroom for when you need it, not capability held back to sell you later. Licensing and support: cor@soulcraft.com.

Performance

  • Per-operation p50/p95 at 1k and 10k entities, pure-JS floor, measured and re-run every release that touches a measured path: docs/performance-envelopes.md.
  • JS distance kernels: ~6× faster cosine, ~1.4× euclidean than 7.x (measured: tests/benchmarks/distance-microbench.mjs, 384-dim, median of 41).
  • Whole-graph reads are single O(N + E) cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
  • Capacity planning and architecture: docs/PERFORMANCE.md · docs/SCALING.md

Use cases

AI agent memory — persistent semantic recall with relationship tracking · Knowledge bases — auto-linking and meaning-aware navigation · Semantic search over codebases, documents, media · Enterprise data — CRM, catalogs, institutional memory · Games & simulations — worlds and characters that remember.

Documentation

Start Core Going deeper
Brainy explained simply API reference Architecture overview
Installation Data model Consistency model
Natural-language queries Query operators Multi-process model
Find system Scaling

Requirements

Bun ≥ 1.1 (recommended) or Node.js ≥ 22. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.

Support & community

MIT © Brainy Contributors.