A production restart made this necessary: a service stopped with exit code 0,
having awaited close() on every pooled brain, and its next boot announced
"Overwriting stale writer lock ... appears dead" for every store it owned.
Nothing had crashed. "The recorded pid is gone" is equally true of an orderly
restart and of a crash, so the verdict could not tell an operator which one
they had — and when the OS recycles a pid it fails the other way, refusing to
open a store whose writer died days ago.
Three changes, all at the law:
- close() is two parts, and the second is unconditional. The durable steps
(flush, markers, component close, plugin deactivate, buffer drain) move to
closeDurableSteps(); the terminal releases — the flush-request watcher, the
WRITER LOCK, the VFS timers, the terminal `closed` flag — always run. The
original failure is narrated with what it costs the next open, then rethrown.
- releaseWriterLock() writes a CLEAN-CLOSE RECORD (`locks/_writer.close`)
naming the lock generation it released; the next claim consumes it, so a
record can never vouch for a later crash. An open reads the record instead
of guessing: recorded → nothing to recover; absent → say so, and name the
crash recovery this open will now run.
- The signal path stops failing in a batch. It was one try around a loop over
every open brain, so the first instance whose flush rejected stranded every
remaining brain's lock and markers — at exit code 0. Now: per-instance
isolation, the generation store's close (the clean-shutdown marker, without
which the next open folds the whole log) is part of shutdown, the lock is
given up in a finally, and the handler no longer calls process.exit() when
the host application has its own signal handler — that race truncated the
host's own close() mid-flight.
Pins: tests/integration/writer-lock-clean-close.test.ts — completed close
leaves no lock and a consumed-once record with a silent reopen; a failing
durable step still releases and still rethrows; SIGKILL leaves the lock with
no record and the reopen names the crash; a host SIGTERM handler runs to
completion.
Branch plan (10 lines):
1. writer lock: clean-close record + always-release close [this commit]
2. open narration: an always-on channel; production clamps prodLog to ERROR,
which is why a three-minute open printed nothing
3. open narration: per-phase lines as each phase ENDS, with progress cadence
4. measure both real-store fixtures on the box, before/after
5. move the generation-log fold out of the foreground where the serving law
allows; durable resumable progress marker
6. same for the VFS bootstrap
7. counts: a legacy container-rule ledger must not keep serving wrong
denominators; counts.json written atomically
8. counts pin with scar directories; two copies of one archive agree
9. docs/canonical-layout-ratification.md — 12 facts confirmed/corrected
10. report: MEASURED before/after, findings, and whether this is 10.4.4
|
||
|---|---|---|
| .claude/skills | ||
| .forgejo/workflows | ||
| assets/models/all-MiniLM-L6-v2 | ||
| bin | ||
| docs | ||
| examples | ||
| integrations | ||
| scripts | ||
| src | ||
| tests | ||
| .aiignore | ||
| .dockerignore | ||
| .gitignore | ||
| .npmignore | ||
| .nvmrc | ||
| brainy.png | ||
| bun.lock | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| CONTRIBUTING.md | ||
| docker-compose.yml | ||
| Dockerfile | ||
| eslint.config.js | ||
| LICENSE | ||
| package-lock.json | ||
| package.json | ||
| README.md | ||
| RELEASES.md | ||
| SECURITY.md | ||
| tsconfig.cli.json | ||
| tsconfig.json | ||
| vitest.config.memory.ts | ||
| vitest.config.ts | ||
Brainy
Three database paradigms. One API. Zero configuration.
The in-process knowledge database for TypeScript — vector search, graph traversal,
and metadata filtering unified in a single query.
Quick start · One query · Features · Scale with Cor · Docs · Support
Open Brainy is the MIT engine — the open API, client library, types, and protocol; an openly specified canonical on-disk format; and this TypeScript reference engine, scoped as a single-node engine for stores up to roughly one million rows. @soulcraft/brainy 10.4.2 was the last release under the old package name — the name passes to the native engine, Brainy, at 11.0.0: the same API over the same open format at production scale, and it requires a license.
Built because we were tired of stitching a vector store to a graph database to a document store — and spending weeks on plumbing before writing a line of business logic. Brainy indexes every fact three ways at once and lets one call query them together:
| You write | Brainy indexes it as | You query it with |
|---|---|---|
data: 'Ada wrote the first program' |
a 384-dim vector (local embedding — no API key) | find({ query: 'computing pioneers' }) |
metadata: { field: 'CS', year: 1843 } |
structured fields (O(1) exact, O(log n) range) | find({ where: { year: { lessThan: 1900 } } }) |
relate({ from: ada, to: babbage }) |
a typed, directed graph edge | find({ connected: { to: babbage, depth: 2 } }) |
It runs inside your process — no server, no Docker, nothing to operate — and persists to plain files you can snapshot with a hard link.
New here? → What is Brainy? — plain-language overview, no jargon
Quick start
bun add @soulcraftlabs/brainy # Bun ≥ 1.1 — recommended
npm install @soulcraftlabs/brainy # Node.js ≥ 22
Registry: add
@soulcraftlabs:registry=https://source.soulcraft.com/api/packages/soulcraftlabs/npm/to your.npmrc(anonymous read).
import { Brainy, NounType, VerbType } from '@soulcraftlabs/brainy'
const brain = new Brainy() // in-memory; one line swaps to disk
await brain.init()
// Text auto-embeds locally; metadata auto-indexes
const react = await brain.add({
data: 'React is a JavaScript library for building user interfaces',
type: NounType.Concept,
subtype: 'library',
metadata: { category: 'frontend', year: 2013 }
})
const next = await brain.add({
data: 'Next.js is a React framework with server-side rendering',
type: NounType.Concept,
subtype: 'framework',
metadata: { category: 'frontend', year: 2016 }
})
await brain.relate({ from: next, to: react, type: VerbType.DependsOn, subtype: 'runtime' })
One query, three engines
const results = await brain.find({
query: 'modern frontend frameworks', // vector — what it means
where: { year: { greaterThan: 2015 } }, // metadata — what it is
connected: { to: react, depth: 2 } // graph — what it touches
})
Every clause is optional; any combination composes. Under the hood Brainy plans the query across an HNSW vector index, a roaring-bitmap field index, and an adjacency graph index — and re-validates every result against your predicate before returning it, so a corrupt index can never hand you a wrong answer.
Feature tour
The database is a value
Pin it, rewind it, fork it. Snapshot isolation without a server.
const db = brain.now() // pin current state — O(1)
await brain.transact([ // atomic all-or-nothing, CAS-guarded
{ op: 'update', id: order, metadata: { status: 'paid' } },
{ op: 'relate', from: invoice, to: order, type: VerbType.References, subtype: 'billing' }
], { ifAtGeneration: db.generation })
await db.get(order) // still 'pending' — pinned forever
await brain.get(order) // 'paid' — live
const lastWeek = await brain.asOf(Date.now() - 7 * 86_400_000) // full query surface, past state
const whatIf = await db.with([{ op: 'remove', id: order }]) // speculative — never touches disk
await brain.now().persist('/backups/today') // instant hard-link snapshot
Consistency model · Snapshots & time travel
Local embeddings — no API keys
Strings embed on-device with a bundled MiniLM model (WASM). Semantic search works offline, in CI, and on air-gapped machines, at zero cost per call. Hybrid keyword + semantic ranking is the default:
await brain.find({ query: 'David Smith' }) // auto: text + semantic
await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // semantic only
A typed graph, not a bag of edges
42 entity types × 127 relationship types form a shared vocabulary for any domain — healthcare (Patient → diagnoses → Condition), finance (Account → transfers → Transaction), yours. Your own taxonomy layers on with subtype, enforced at write time:
await brain.add({ data: 'Avery Brooks', type: NounType.Person, subtype: 'employee' })
brain.counts.bySubtype(NounType.Person) // O(1) — { employee: 12, customer: 847 }
brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
Type system · Subtypes & facets
Graph analytics built in
await brain.graph.rank() // which entities matter most (centrality)
await brain.graph.communities() // natural clusters
await brain.graph.path(a, b) // how two things connect
await brain.graph.subgraph([seed], { depth: 2 }) // bounded neighborhood → { nodes, edges }
await brain.graph.export() // whole graph, one O(N+E) streaming pass
Write-time aggregations
SUM / COUNT / AVG / MIN / MAX with GROUP BY and time windows, maintained incrementally on every write — reads are O(1) lookups, not scans. Aggregation guide
Import anything
await brain.import('customers.csv')
await brain.import('sales.xlsx') // every sheet
await brain.import('research-paper.pdf') // tables extracted
await brain.import('https://api.example.com/data.json')
Entities auto-classify on the way in; brain.extractEntities(text) exposes the same NER ensemble directly. Import guide
A filesystem that understands content
await brain.vfs.writeFile('/docs/readme.md', 'Project documentation')
await brain.vfs.search('React components with hooks') // semantic file search
Operations-grade by default
- Single-writer, many-reader — an exclusive lock protects the data directory;
Brainy.openReadOnly()and thebrainy inspectCLI examine a live brain from another process, safely. - Self-upgrading data files — a 7.x brain opens under 8.x and migrates itself behind an observable lock (
getIndexStatus().migration), with an automatic pre-upgrade backup. No migration scripts. - No silent wrong answers — cold-open guards self-heal or throw typed errors (
MetadataIndexNotReadyError,GraphIndexNotReadyError); they never return[]for data that exists.
Multi-process model · Inspection guide
When you outgrow Brainy
Brainy's pure-TypeScript engines carry real workloads a long way on their own — see the measured, per-operation numbers (not marketing figures) in docs/performance-envelopes.md for what to expect, unaccelerated, on plain filesystem storage.
When a deployment needs native-scale vector/graph performance — memory-mapped indexes that don't need your dataset in RAM, billion-scale ambitions — add the native engine. The API doesn't change:
npm install @soulcraft/cor
const brain = new Brainy({ storage: { type: 'filesystem', path: './data' } })
await brain.init() // @soulcraft/cor detected — same code, native engines underneath
Installing the package is the opt-in: if @soulcraft/cor is present, it loads and announces itself in the init log; if it's present but broken, init() throws — an installed accelerator never silently vanishes behind the JS engines. Opt out with plugins: [], or pin exactly what loads with plugins: ['@soulcraft/cor']. @soulcraft/cor (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured 0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors in Cor's release gate.
Open core, commercial accelerator: Brainy is MIT and complete on its own — Cor is more headroom for when you need it, not capability held back to sell you later. Licensing and support: cor@soulcraft.com.
Performance
- Per-operation p50/p95 at 1k and 10k entities, pure-JS floor, measured and re-run every release that touches a measured path: docs/performance-envelopes.md.
- JS distance kernels: ~6× faster cosine, ~1.4× euclidean than 7.x (measured:
tests/benchmarks/distance-microbench.mjs, 384-dim, median of 41). - Whole-graph reads are single O(N + E) cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
- Capacity planning and architecture: docs/PERFORMANCE.md · docs/SCALING.md
Use cases
AI agent memory — persistent semantic recall with relationship tracking · Knowledge bases — auto-linking and meaning-aware navigation · Semantic search over codebases, documents, media · Enterprise data — CRM, catalogs, institutional memory · Games & simulations — worlds and characters that remember.
Documentation
Requirements
Bun ≥ 1.1 (recommended) or Node.js ≥ 22. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
Support & community
- Bugs and ideas → brainy@soulcraft.com — no account needed, you'll get a receipt.
- Security reports → security@soulcraft.com — see SECURITY.md.
- Contributing → see CONTRIBUTING.md.
MIT © Brainy Contributors.