+
-
+
+ Three database paradigms. One API. Zero configuration.
+ The in-process knowledge database for TypeScript — vector search, graph traversal,
+ and metadata filtering unified in a single query.
+
+ 
+ 
+ 
+ 
+ 
+ 
+
+ Quick start · + One query · + Features · + Scale with Cor · + Docs +
--- -## Install +Built because we were tired of stitching a vector store to a graph database to a document store — and spending weeks on plumbing before writing a line of business logic. Brainy indexes every fact **three ways at once** and lets one call query them together: + +| You write | Brainy indexes it as | You query it with | +|---|---|---| +| `data: 'Ada wrote the first program'` | a **384-dim vector** (local embedding — no API key) | `find({ query: 'computing pioneers' })` | +| `metadata: { field: 'CS', year: 1843 }` | **structured fields** (O(1) exact, O(log n) range) | `find({ where: { year: { lessThan: 1900 } } })` | +| `relate({ from: ada, to: babbage })` | a **typed, directed graph edge** | `find({ connected: { to: babbage, depth: 2 } })` | + +It runs **inside your process** — no server, no Docker, nothing to operate — and persists to plain files you can snapshot with a hard link. + +**New here?** → **[What is Brainy? — plain-language overview, no jargon](docs/eli5.md)** + +## Quick start ```bash -bun add @soulcraft/brainy # fastest — recommended -npm install @soulcraft/brainy # Node.js — fully supported +bun add @soulcraft/brainy # Bun ≥ 1.1 — recommended +npm install @soulcraft/brainy # Node.js ≥ 22 ``` -## Quick Start - ```javascript import { Brainy, NounType, VerbType } from '@soulcraft/brainy' -const brain = new Brainy() +const brain = new Brainy() // in-memory; one line swaps to disk await brain.init() -// Add knowledge — text auto-embeds, metadata auto-indexes -const reactId = await brain.add({ +// Text auto-embeds locally; metadata auto-indexes +const react = await brain.add({ data: 'React is a JavaScript library for building user interfaces', type: NounType.Concept, + subtype: 'library', metadata: { category: 'frontend', year: 2013 } }) -const nextId = await brain.add({ - data: 'Next.js framework for React with server-side rendering', +const next = await brain.add({ + data: 'Next.js is a React framework with server-side rendering', type: NounType.Concept, - metadata: { category: 'framework', year: 2016 } + subtype: 'framework', + metadata: { category: 'frontend', year: 2016 } }) -// Create a relationship -await brain.relate({ from: nextId, to: reactId, type: VerbType.BuiltOn }) - -// Query all three paradigms at once -const results = await brain.find({ - query: 'modern frontend frameworks', // Vector similarity - where: { year: { greaterThan: 2015 } }, // Metadata filtering - connected: { to: reactId, depth: 2 } // Graph traversal -}) +await brain.relate({ from: next, to: react, type: VerbType.DependsOn, subtype: 'runtime' }) ``` -**[Full API Reference](docs/api/README.md)** | **[soulcraft.com/docs](https://soulcraft.com/docs)** - ---- - -## Three Indexes, One Query - -Every piece of knowledge lives in three indexes simultaneously: - -- **`data`** → **Vector index** — Content for semantic search. Strings auto-embed into 384-dim vectors. Queried with `find({ query: '...' })`. -- **`metadata`** → **Metadata index** — Structured fields for filtering. O(1) lookups. Queried with `find({ where: { ... } })`. -- **`relate()`** → **Graph index** — Typed, directed relationships between entities. Traversed with `find({ connected: { ... } })`. - -```javascript -// Data → vector index (semantic search) -const articleId = await brain.add({ - data: 'A deep dive into transformer architectures', - type: NounType.Document, - metadata: { author: 'Dr. Chen', year: 2024, tags: ['AI'] } // → metadata index -}) - -// Relationships → graph index -await brain.relate({ from: authorId, to: articleId, type: VerbType.Authored }) - -// Query all three at once -brain.find({ - query: 'attention mechanisms', // Vector similarity - where: { year: { greaterThan: 2023 } }, // Metadata filter - connected: { from: authorId, depth: 1 } // Graph traversal -}) -``` - -**[Data Model Reference](docs/DATA_MODEL.md)** | **[Query Operators](docs/QUERY_OPERATORS.md)** - ---- - -## Features - -### Triple Intelligence - -Vector search + graph traversal + metadata filtering in every query. No stitching services together — one `find()` call combines all three. +## One query, three engines ```javascript const results = await brain.find({ - query: 'machine learning', - where: { department: 'engineering', level: 'senior' }, - connected: { from: teamLeadId, via: VerbType.WorksWith, depth: 2 } + query: 'modern frontend frameworks', // vector — what it means + where: { year: { greaterThan: 2015 } }, // metadata — what it is + connected: { to: react, depth: 2 } // graph — what it touches }) ``` -### Hybrid Search +Every clause is optional; any combination composes. Under the hood Brainy plans the query across an HNSW vector index, a roaring-bitmap field index, and an adjacency graph index — and re-validates every result against your predicate before returning it, so a corrupt index can never hand you a wrong answer. -Automatically combines keyword (text) and semantic (vector) search. No configuration needed. +## Feature tour + +### The database is a value + +Pin it, rewind it, fork it. Snapshot isolation without a server. ```javascript -await brain.find({ query: 'David Smith' }) // Auto: text + semantic -await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // Semantic only -await brain.find({ query: 'exact id', searchMode: 'text' }) // Text only +const db = brain.now() // pin current state — O(1) + +await brain.transact([ // atomic all-or-nothing, CAS-guarded + { op: 'update', id: order, metadata: { status: 'paid' } }, + { op: 'relate', from: invoice, to: order, type: VerbType.References, subtype: 'billing' } +], { ifAtGeneration: db.generation }) + +await db.get(order) // still 'pending' — pinned forever +await brain.get(order) // 'paid' — live + +const lastWeek = await brain.asOf(Date.now() - 7 * 86_400_000) // full query surface, past state +const whatIf = await db.with([{ op: 'remove', id: order }]) // speculative — never touches disk +await brain.now().persist('/backups/today') // instant hard-link snapshot ``` -### Query Operators +**[Consistency model](docs/concepts/consistency-model.md)** · **[Snapshots & time travel](docs/guides/snapshots-and-time-travel.md)** -Filter metadata with equality, comparison, array, existence, pattern, and logical operators: +### Local embeddings — no API keys + +Strings embed on-device with a bundled MiniLM model (WASM). Semantic search works offline, in CI, and on air-gapped machines, at zero cost per call. Hybrid keyword + semantic ranking is the default: ```javascript -await brain.find({ - where: { - status: 'active', // Exact match - score: { greaterThan: 90 }, // Comparison - tags: { contains: 'ai' }, // Array - anyOf: [{ role: 'admin' }, { role: 'owner' }] // Logical OR - } -}) +await brain.find({ query: 'David Smith' }) // auto: text + semantic +await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // semantic only ``` -**[Query Operators Reference](docs/QUERY_OPERATORS.md)** — all operators with indexed/in-memory matrix +### A typed graph, not a bag of edges -### Graph Relationships - -Typed, directed edges between entities. Traverse connections at any depth. +42 entity types × 127 relationship types form a shared vocabulary for any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), yours. Your own taxonomy layers on with `subtype`, enforced at write time: ```javascript -await brain.relate({ from: personId, to: projectId, type: VerbType.WorksOn }) +await brain.add({ data: 'Avery Brooks', type: NounType.Person, subtype: 'employee' }) -const results = await brain.find({ - connected: { from: personId, via: VerbType.WorksOn, depth: 3 } -}) +brain.counts.bySubtype(NounType.Person) // O(1) — { employee: 12, customer: 847 } +brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true }) ``` -### Database as a Value +**[Type system](docs/architecture/noun-verb-taxonomy.md)** · **[Subtypes & facets](docs/guides/subtypes-and-facets.md)** -The whole database, pinned as an immutable value. Snapshot isolation, time travel, atomic transactions, instant hard-link snapshots. +### Graph analytics built in ```javascript -const db = brain.now() // Pin current state — O(1) - -// Atomic multi-write transaction (all-or-nothing, with CAS) -await brain.transact([ - { op: 'update', id: orderId, metadata: { status: 'paid' } }, - { op: 'relate', from: invoiceId, to: orderId, type: VerbType.References, subtype: 'billing' } -], { meta: { author: 'billing-service' }, ifAtGeneration: db.generation }) - -await db.get(orderId) // Still 'pending' — pinned, forever -await brain.get(orderId) // 'paid' — live - -// Time travel: full query surface at any past state -const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000)) -const past = await yesterday.find({ query: 'unpaid orders' }) - -// What-if: speculative writes, nothing touches disk -const whatIf = await db.with([{ op: 'remove', id: orderId }]) - -// Instant backup: hard-link snapshot, opens read-only with Brainy.load() -await brain.now().persist('/backups/today') +await brain.graph.rank() // which entities matter most (centrality) +await brain.graph.communities() // natural clusters +await brain.graph.path(a, b) // how two things connect +await brain.graph.subgraph([seed], { depth: 2 }) // bounded neighborhood → { nodes, edges } +await brain.graph.export() // whole graph, one O(N+E) streaming pass ``` -**[Consistency Model](docs/concepts/consistency-model.md)** | **[Snapshots & Time Travel](docs/guides/snapshots-and-time-travel.md)** +### Write-time aggregations -### Virtual Filesystem +`SUM` / `COUNT` / `AVG` / `MIN` / `MAX` with `GROUP BY` and time windows, maintained incrementally on every write — reads are O(1) lookups, not scans. **[Aggregation guide](docs/guides/aggregation.md)** -File operations with semantic search built in. - -```javascript -const vfs = brain.vfs - -await vfs.writeFile('/docs/readme.md', 'Project documentation') -const content = await vfs.readFile('/docs/readme.md') -const tree = await vfs.getTreeStructure('/docs', { maxDepth: 3 }) - -// Semantic file search -const matches = await vfs.search('React components with hooks') -``` - -**[VFS Quick Start](docs/vfs/QUICK_START.md)** | **[Common Patterns](docs/vfs/COMMON_PATTERNS.md)** - -### Import Anything - -CSV, Excel, PDF, URLs — auto-detected format, auto-classified entities. +### Import anything ```javascript await brain.import('customers.csv') -await brain.import('sales-data.xlsx') // all sheets processed -await brain.import('research-paper.pdf') // tables extracted automatically +await brain.import('sales.xlsx') // every sheet +await brain.import('research-paper.pdf') // tables extracted await brain.import('https://api.example.com/data.json') -await brain.import('./handbook.md', { vfsPath: '/imports/handbook' }) // preserve in VFS ``` -**[Import Guide](docs/guides/import-anything.md)** +Entities auto-classify on the way in; `brain.extractEntities(text)` exposes the same NER ensemble directly. **[Import guide](docs/guides/import-anything.md)** -### Entity Extraction - -AI-powered named entity recognition with 4-signal ensemble scoring. +### A filesystem that understands content ```javascript -const entities = await brain.extractEntities('John Smith founded Acme Corp in New York') -// [ -// { text: 'John Smith', type: NounType.Person, confidence: 0.95 }, -// { text: 'Acme Corp', type: NounType.Organization, confidence: 0.92 }, -// { text: 'New York', type: NounType.Location, confidence: 0.88 } -// ] +await brain.vfs.writeFile('/docs/readme.md', 'Project documentation') +await brain.vfs.search('React components with hooks') // semantic file search ``` -**[Neural Extraction Guide](docs/neural-extraction.md)** +**[VFS quick start](docs/vfs/QUICK_START.md)** -### Plugin System +### Operations-grade by default -Optional native acceleration via `@soulcraft/cortex` — SIMD distance calculations, CRoaring bitmaps, Candle ML embeddings. +- **Single-writer, many-reader** — an exclusive lock protects the data directory; `Brainy.openReadOnly()` and the `brainy inspect` CLI examine a live brain from another process, safely. +- **Self-upgrading data files** — a 7.x brain opens under 8.x and migrates itself behind an observable lock (`getIndexStatus().migration`), with an automatic pre-upgrade backup. No migration scripts. +- **No silent wrong answers** — cold-open guards self-heal or throw typed errors (`MetadataIndexNotReadyError`, `GraphIndexNotReadyError`); they never return `[]` for data that exists. + +**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)** + +## From laptop to hundreds of millions + +Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**: + +```bash +npm install @soulcraft/cor +``` ```javascript -const brain = new Brainy({ plugins: ['@soulcraft/cortex'] }) -await brain.init() +const brain = new Brainy({ storage: { type: 'filesystem', path: './data' } }) +await brain.init() // Cor auto-detected — same code, native engines underneath ``` -Plugins are opt-in. Brainy never auto-imports packages unless listed in `plugins`. +[`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate. -**[Plugin Documentation](docs/PLUGINS.md)** +Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both. ---- +## Performance -## Type System +- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41). +- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan. +- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)** -42 noun types and 127 verb types form a universal knowledge protocol: +## Use cases -``` -42 Nouns × 127 Verbs = 5,334 base relationship combinations -``` - -Model any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), education (`Student → completes → Course`), or your own. - -### Subtypes — sub-classification within a NounType *or* VerbType - -Both noun types and verb types are intentionally coarse. Use the top-level `subtype` field to sub-classify entities AND relationships within a type — flat string, no hierarchy, your choice of vocabulary: - -```javascript -// Nouns: sub-classify entities -await brain.add({ - data: 'Avery Brooks — runs the AI lab', - type: NounType.Person, - subtype: 'employee' // 'customer', 'vendor', 'contractor', … -}) - -// Verbs: sub-classify relationships -await brain.relate({ - from: ceoId, - to: vpId, - type: VerbType.ReportsTo, - subtype: 'direct' // 'dotted-line', 'matrix', … -}) - -// Filter on the fast path — column-store hit, not metadata fallback: -const employees = await brain.find({ type: NounType.Person, subtype: 'employee' }) -const directReports = await brain.related({ from: ceoId, subtype: 'direct' }) - -// O(1) counts via the persisted rollups: -brain.counts.bySubtype(NounType.Person) -// → { employee: 12, customer: 847, vendor: 34 } - -brain.counts.byRelationshipSubtype(VerbType.ReportsTo) -// → { direct: 12, 'dotted-line': 3 } -``` - -**Enforce the pairing.** Register a vocabulary per type or turn on brain-wide strict mode to ensure every entity AND relationship has both `type` AND `subtype`: - -```javascript -// Per-type rule with a closed vocabulary -brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true }) - -// 8.0 default: every write requires a subtype. Exempt genuine catch-all types… -const brain = new Brainy({ requireSubtype: { except: [NounType.Thing] } }) - -// …or opt out while migrating pre-8.0 data, then audit and back-fill: -const legacy = new Brainy({ requireSubtype: false }) -await legacy.audit() // gaps, grouped by type -await legacy.fillSubtypes({ [NounType.Person]: 'unspecified' }) // close them -``` - -For other facets you want counted (`status`, `source`, `role`), register them with `brain.trackField(name)`. Renaming an existing convention to `subtype`? Use `brain.migrateField({from, to, entityKind: 'both'})` to walk nouns AND verbs in one pass. Full guide: **[Subtypes & Facets](docs/guides/subtypes-and-facets.md)**. - -**[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** | **[Stage 3 Canonical Reference](docs/STAGE3-CANONICAL-TAXONOMY.md)** - ---- - -## Storage: Memory and Filesystem - -The same API at every scale. Change one config line to go from prototype to production. - -### Development — Zero Config - -```javascript -const brain = new Brainy() -``` - -### Production — Filesystem (gzip compression on by default) - -```javascript -const brain = new Brainy({ - storage: { type: 'filesystem', path: './data' } -}) -``` - -### Backups and Portability — Snapshots - -```javascript -const db = brain.now() -await db.persist('/backups/2026-06-11') // instant hard-link snapshot -await db.release() - -const snapshot = await Brainy.load('/backups/2026-06-11') // read-only Db -const hits = await snapshot.find({ query: 'quarterly invoices' }) -await snapshot.release() -``` - -A snapshot directory is self-contained — copy it to another machine, open it with `Brainy.load()`, or restore it wholesale with `brain.restore(path, { confirm: true })`. - -Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)**. - -**[Capacity Planning](docs/operations/capacity-planning.md)** - ---- - -## Use Cases - -- **AI agents** — Persistent memory with semantic recall and relationship tracking -- **Knowledge bases** — Auto-linking, semantic search, relationship-aware navigation -- **Semantic search** — Find by meaning across codebases, documents, or media -- **Enterprise knowledge** — CRM, product catalogs, institutional memory -- **Interactive experiences** — Game worlds, NPCs, and characters that remember -- **Content platforms** — Similarity-based discovery, intelligent tagging - ---- +**AI agent memory** — persistent semantic recall with relationship tracking · **Knowledge bases** — auto-linking and meaning-aware navigation · **Semantic search** over codebases, documents, media · **Enterprise data** — CRM, catalogs, institutional memory · **Games & simulations** — worlds and characters that remember. ## Documentation -### Start Here - -- **[Brainy explained simply](docs/eli5.md)** — Plain-language overview, no jargon, no code - -### Core - -- **[API Reference](docs/api/README.md)** — Every method with parameters, returns, and examples -- **[Data Model](docs/DATA_MODEL.md)** — Entity structure, data vs metadata -- **[Query Operators](docs/QUERY_OPERATORS.md)** — All BFO operators with examples -- **[Find System](docs/FIND_SYSTEM.md)** — Natural language find() and hybrid search - -### Architecture - -- **[Architecture Overview](docs/architecture/overview.md)** — System design and components -- **[Triple Intelligence](docs/architecture/triple-intelligence.md)** — Vector + graph + metadata unified query -- **[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** — Universal type system -- **[Data Storage Architecture](docs/architecture/data-storage-architecture.md)** — Type-aware indexing and HNSW - -### Virtual Filesystem - -- **[VFS Quick Start](docs/vfs/QUICK_START.md)** — Build file explorers that never crash -- **[VFS Core](docs/vfs/VFS_CORE.md)** — Full VFS API reference -- **[Semantic VFS](docs/vfs/SEMANTIC_VFS.md)** — AI-powered file navigation - -### Guides - -- **[Import Anything](docs/guides/import-anything.md)** — CSV, Excel, PDF, URLs -- **[Framework Integration](docs/guides/framework-integration.md)** — React, Vue, Angular, Svelte -- **[Natural Language Queries](docs/guides/natural-language.md)** — Master the find() method - -### Operations - -- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling -- **[Performance](docs/PERFORMANCE.md)** — Benchmarks and architecture details - ---- +| Start | Core | Going deeper | +|---|---|---| +| [Brainy explained simply](docs/eli5.md) | [API reference](docs/api/README.md) | [Architecture overview](docs/architecture/overview.md) | +| [Installation](docs/guides/installation.md) | [Data model](docs/DATA_MODEL.md) | [Consistency model](docs/concepts/consistency-model.md) | +| [Natural-language queries](docs/guides/natural-language.md) | [Query operators](docs/QUERY_OPERATORS.md) | [Multi-process model](docs/concepts/multi-process.md) | +| | [Find system](docs/FIND_SYSTEM.md) | [Scaling](docs/SCALING.md) | ## Requirements -**Bun 1.1+** (recommended) or **Node.js 22 LTS** +**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use. -```bash -bun add @soulcraft/brainy # Bun — best performance -npm install @soulcraft/brainy # Node.js — fully supported -``` +## Contributing & license -> Brainy 8.0 is server-only. Browser support (OPFS storage, Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line remains available on npm if you need it. - -## Single-Writer Model - -Brainy is **single-writer, many-reader** on filesystem storage. One writer -holds an exclusive lock on the data directory; any number of readers can -inspect it concurrently. Opening a second writer throws with the PID of the -existing one. - -```typescript -// Live application — writer mode is the default -const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/brain' } }) -await brain.init() - -// Out-of-band diagnostics from a separate process — safe to run while the -// writer is live -const reader = await Brainy.openReadOnly({ - storage: { type: 'filesystem', path: '/data/brain' } -}) -await reader.requestFlush({ timeoutMs: 5000 }) -const stats = await reader.stats() -``` - -For incident debugging, use the `brainy inspect` CLI: - -```bash -brainy inspect stats /data/brain -brainy inspect find /data/brain --where '{"entityType":"booking"}' -brainy inspect explain /data/brain --where '{"entityType":"booking"}' -brainy inspect health /data/brain -``` - -See [the multi-process model](docs/concepts/multi-process.md) and the -[inspection guide](docs/guides/inspection.md) for the full story, including -stale-lock detection and the cross-process flush RPC. - -## Contributing - -We welcome contributions! See **[CONTRIBUTING.md](CONTRIBUTING.md)** for guidelines. - -## License - -MIT © Brainy Contributors +Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors. diff --git a/RELEASES.md b/RELEASES.md index 7e2adc8f..3153517d 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -10,9 +10,14 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so --- -## v8.0.0 — 2026-07-02 (GA, npm tag `latest`) +## v8.0.1 — 2026-07-02 (GA, npm tag `latest`) -**8.0.0 is the first stable major on the u64-id core.** It ships in lockstep with the optional +> **Why 8.0.1, not 8.0.0:** the `8.0.0` version number was consumed on npm by a January +> development-cycle publish that was immediately unpublished — npm permanently retires an +> unpublished version number. `8.0.1` is the first stable release of the 8.x line; there is no +> separate 8.0.0. + +**8.0.1 is the first stable major on the u64-id core.** It ships in lockstep with the optional native provider's `3.0` (the billion-scale path). Everything below works standalone on the open-core JS engine — a native provider is feature-detected and only changes the scale ceiling, never the API. @@ -134,7 +139,7 @@ rebuild). Opt out of the backup with `migrationBackup: false`. surfaces, hard renames (no aliases, no deprecation period), and one flipped default. This entry **is** the migration guide: the find/replace pairs, the sed snippet, and the upgrade checklist below are the complete story — there is -no separate migration doc. **`8.0.0` is GA on `latest`** — install with +no separate migration doc. **`8.0.1` is GA on `latest`** — install with `npm i @soulcraft/brainy@latest`. > **rc.3–rc.5 additions (2026-06-24):**