feat: enforce data/metadata separation, numeric range queries, improved docs

- Store data opaquely in add() and update() instead of spreading object
  properties into top-level metadata. data is for semantic search (HNSW),
  metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
  instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
  showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
This commit is contained in:
David Snelling 2026-02-09 12:06:59 -08:00
parent edb5ec4696
commit 0ddc05a5bb
30 changed files with 1356 additions and 7001 deletions

800
README.md
View file

@ -10,95 +10,19 @@
[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![TypeScript](https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg)](https://www.typescriptlang.org/)
## The Knowledge Operating System
**Three database paradigms. One API. Zero configuration.**
**Every piece of knowledge in your application — living, connected, and intelligent.**
Stop fighting with vector databases, graph databases, and document stores. Stop stitching together Pinecone + Neo4j + MongoDB. **Brainy does all three, in one elegant API, from prototype to planet-scale.**
```javascript
const brain = new Brainy()
await brain.init()
// That's it. You now have semantic search, graph relationships,
// and document filtering. Zero configuration. Just works.
```
**Built by developers who were tired of:**
- Spending weeks configuring embeddings, indexes, and schemas
- Choosing between vector similarity OR graph relationships OR metadata filtering
- Rewriting everything when you need to scale from 1,000 to 1,000,000,000 entities
**Brainy makes the impossible simple: All three paradigms. One API. Any scale.**
Built because we were tired of stitching together Pinecone + Neo4j + MongoDB and spending weeks on configuration before writing a single line of business logic. Brainy unifies vector search, graph traversal, and metadata filtering so you don't have to choose.
---
## 👉 Choose Your Path
**New to Brainy? Pick your starting point:**
### 🚀 Path 1: I want to build something NOW
**→ [Full Documentation](https://soulcraft.com/docs)** ⭐ **Most developers start here**
- Interactive API reference with examples
- Quick start guides and tutorials
- Copy-paste ready code snippets
- **This is your primary resource**
### 🧠 Path 2: I want to understand the big picture first
**→ Keep reading below** for demos, architecture, and use cases
### 📊 Path 3: I'm evaluating database options
**→ Jump to [Why Revolutionary](#why-brainy-is-revolutionary)** or **[Benchmarks](#benchmarks)**
---
## See It In Action
**30 seconds to understand why Brainy is different:**
```javascript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
const brain = new Brainy()
await brain.init()
// Add knowledge with context
const reactId = await brain.add({
data: "React is a JavaScript library for building user interfaces",
type: NounType.Concept,
metadata: { category: "frontend", year: 2013 }
})
const nextId = await brain.add({
data: "Next.js framework for React with server-side rendering",
type: NounType.Concept,
metadata: { category: "framework", year: 2016 }
})
// Create relationships
await brain.relate({ from: nextId, to: reactId, type: VerbType.BuiltOn })
// NOW THE MAGIC: Query with natural language
const results = await brain.find({
query: "modern frontend frameworks", // 🔍 Vector similarity
where: { year: { greaterThan: 2015 } }, // 📊 Document filtering
connected: { to: reactId, depth: 2 } // 🕸️ Graph traversal
})
// ALL THREE PARADIGMS. ONE QUERY. 10ms response time.
```
**This is impossible with traditional databases.** Brainy makes it trivial.
---
## Quick Start
## Install
```bash
npm install @soulcraft/brainy
```
### Your First Knowledge Graph (60 seconds)
## Quick Start
```javascript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
@ -106,685 +30,323 @@ import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
const brain = new Brainy()
await brain.init()
// Add knowledge
const jsId = await brain.add({
data: "JavaScript is a programming language",
type: NounType.Concept,
metadata: { category: "language", year: 1995 }
// Add knowledge — text auto-embeds, metadata auto-indexes
const reactId = await brain.add({
data: 'React is a JavaScript library for building user interfaces',
type: NounType.Concept,
metadata: { category: 'frontend', year: 2013 }
})
const nodeId = await brain.add({
data: "Node.js runtime environment",
type: NounType.Concept,
metadata: { category: "runtime", year: 2009 }
const nextId = await brain.add({
data: 'Next.js framework for React with server-side rendering',
type: NounType.Concept,
metadata: { category: 'framework', year: 2016 }
})
// Create relationships
await brain.relate({ from: nodeId, to: jsId, type: VerbType.Executes })
// Create a relationship
await brain.relate({ from: nextId, to: reactId, type: VerbType.BuiltOn })
// Query with Triple Intelligence
// Query all three paradigms at once
const results = await brain.find({
query: "JavaScript", // 🔍 Vector
where: { category: "language" }, // 📊 Document
connected: { from: nodeId, depth: 1 } // 🕸️ Graph
query: 'modern frontend frameworks', // Vector similarity
where: { year: { greaterThan: 2015 } }, // Metadata filtering
connected: { to: reactId, depth: 2 } // Graph traversal
})
```
**Done.** No configuration. No complexity. Production-ready from day one.
**→ Ready to dive deeper? [Complete API Documentation](docs/api/README.md)** has every method with examples.
**[Full API Reference](docs/api/README.md)** | **[soulcraft.com/docs](https://soulcraft.com/docs)**
---
## Entity Extraction (NEW in v5.7.6)
## Three Indexes, One Query
**Extract entities from text with AI-powered classification:**
Every piece of knowledge lives in three indexes simultaneously:
- **`data`** → **Vector index** — Content for semantic search. Strings auto-embed into 384-dim vectors. Queried with `find({ query: '...' })`.
- **`metadata`** → **Metadata index** — Structured fields for filtering. O(1) lookups. Queried with `find({ where: { ... } })`.
- **`relate()`** → **Graph index** — Typed, directed relationships between entities. Traversed with `find({ connected: { ... } })`.
```javascript
import { Brainy, NounType } from '@soulcraft/brainy'
const brain = new Brainy()
await brain.init()
// Extract all entities
const entities = await brain.extractEntities('John Smith founded Acme Corp in New York')
// Returns:
// [
// { text: 'John Smith', type: NounType.Person, confidence: 0.95 },
// { text: 'Acme Corp', type: NounType.Organization, confidence: 0.92 },
// { text: 'New York', type: NounType.Location, confidence: 0.88 }
// ]
// Extract with filters
const people = await brain.extractEntities(resume, {
types: [NounType.Person],
confidence: 0.8
// Data → vector index (semantic search)
const articleId = await brain.add({
data: 'A deep dive into transformer architectures',
type: NounType.Document,
metadata: { author: 'Dr. Chen', year: 2024, tags: ['AI'] } // → metadata index
})
// Advanced: Direct access to extractors
import { SmartExtractor } from '@soulcraft/brainy'
// Relationships → graph index
await brain.relate({ from: authorId, to: articleId, type: VerbType.Authored })
const extractor = new SmartExtractor(brain, { minConfidence: 0.7 })
const result = await extractor.extract('CEO', {
formatContext: { format: 'excel', columnHeader: 'Title' }
// Query all three at once
brain.find({
query: 'attention mechanisms', // Vector similarity
where: { year: { greaterThan: 2023 } }, // Metadata filter
connected: { from: authorId, depth: 1 } // Graph traversal
})
```
**Features:**
- 🎯 **4-Signal Ensemble** - ExactMatch (40%) + Embedding (35%) + Pattern (20%) + Context (5%)
- 📊 **Format Intelligence** - Adapts to Excel, CSV, PDF, YAML, DOCX, JSON, Markdown
- ⚡ **Fast** - ~15-20ms per extraction with LRU caching
- 🌍 **42 Types** - Person, Organization, Location, Document, and 38 more
**→ [Neural Extraction Guide](docs/neural-extraction.md)** | **[Import Preview Mode](docs/neural-extraction.md#import-preview-mode)**
**[Data Model Reference](docs/DATA_MODEL.md)** | **[Query Operators](docs/QUERY_OPERATORS.md)**
---
## From Prototype to Planet Scale
## Features
**The same API. Zero rewrites. Any scale.**
### Triple Intelligence
Vector search + graph traversal + metadata filtering in every query. No stitching services together — one `find()` call combines all three.
### 👤 Individual Developer → Weekend Prototype
```javascript
const brain = new Brainy() // Zero config, starts in memory
await brain.init()
```
**Perfect for:** Hackathons, side projects, prototyping, learning
### 👥 Small Team → Production MVP
```javascript
const brain = new Brainy({
storage: { type: 'filesystem', path: './data', compression: true }
const results = await brain.find({
query: 'machine learning',
where: { department: 'engineering', level: 'senior' },
connected: { from: teamLeadId, via: VerbType.WorksWith, depth: 2 }
})
```
**Scale:** Thousands to hundreds of thousands • **Performance:** <5ms queries
**→ [Production Service Architecture](docs/PRODUCTION_SERVICE_ARCHITECTURE.md)** — Singleton patterns, caching, and scaling for Bun/Node.js services
### 🏢 Growing Company → Multi-Million Scale
### Hybrid Search
Automatically combines keyword (text) and semantic (vector) search. No configuration needed.
```javascript
const brain = new Brainy({
storage: { type: 's3', s3Storage: { bucketName: 'my-kb', region: 'us-east-1' } },
hnsw: { typeAware: true }
await brain.find({ query: 'David Smith' }) // Auto: text + semantic
await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // Semantic only
await brain.find({ query: 'exact id', searchMode: 'text' }) // Text only
```
### Query Operators
Filter metadata with equality, comparison, array, existence, pattern, and logical operators:
```javascript
await brain.find({
where: {
status: 'active', // Exact match
score: { greaterThan: 90 }, // Comparison
tags: { contains: 'ai' }, // Array
anyOf: [{ role: 'admin' }, { role: 'owner' }] // Logical OR
}
})
```
**Scale:** Millions of entities • **Performance:** <10ms queries (measured at 1M scale)
### 🌍 Enterprise → Billion+ Scale
**[Query Operators Reference](docs/QUERY_OPERATORS.md)** — all operators with indexed/in-memory matrix
### Graph Relationships
Typed, directed edges between entities. Traverse connections at any depth.
```javascript
const brain = new Brainy({
storage: { type: 'gcs', gcsStorage: { bucketName: 'global-kb' } },
hnsw: { typeAware: true, M: 32, efConstruction: 400 }
await brain.relate({ from: personId, to: projectId, type: VerbType.WorksOn })
const results = await brain.find({
connected: { from: personId, via: VerbType.WorksOn, depth: 3 }
})
```
**Scale:** PROJECTED billion+ (extrapolated from 10M tests, not tested at 1B)
**Cost:** $138k/year → $6k/year with intelligent tiering (96% savings)
**→ [Capacity Planning Guide](docs/operations/capacity-planning.md)** | **[Cost Optimization](docs/operations/)**
### Git-Style Branching
### 🎯 The Point
**Start simple. Scale infinitely. Never rewrite.**
Most systems make you choose: Simple (SQLite) OR Scalable (Kubernetes + 7 databases).
**Brainy gives you both.** Starts simple as SQLite. Scales like Google.
---
## Why Brainy Is Revolutionary
### 🧠 **Triple Intelligence™** — The Impossible Made Possible
**The world's first to unify three database paradigms in ONE API:**
| What You Get | Like Having | But Unified |
|-------------|-------------|-------------|
| 🔍 **Vector Search** | Pinecone, Weaviate | Find by meaning |
| 🕸️ **Graph Relationships** | Neo4j, ArangoDB | Navigate connections |
| 📊 **Document Filtering** | MongoDB, Elasticsearch | Query metadata |
**Every other system makes you choose.** Brainy does all three together.
**Why this matters:** Your data isn't just vectors or just documents or just graphs. It's all three at once. A research paper is semantically similar to other papers (vector), written by an author (graph), and published in 2023 (document). **Brainy is the only system that understands this.**
### 🎯 **42 Noun Types × 127 Verb Types = Universal Protocol**
Model **any domain** with mathematical completeness:
```
42 Nouns × 127 Verbs × ∞ Metadata = 5,334+ base combinations
Stage 3 CANONICAL: 96-97% coverage of all human knowledge
```
**Real-world expressiveness:**
- Healthcare: `Patient → diagnoses → Condition`
- Finance: `Account → transfers → Transaction`
- Manufacturing: `Product → assembles → Component`
- Education: `Student → completes → Course`
- **YOUR domain** → Your types + relationships = Your knowledge graph
[→ See the Mathematical Proof](docs/architecture/noun-verb-taxonomy.md)
### ⚡ **Zero Configuration Philosophy**
**We hate configuration files. So we eliminated them.**
Fork your entire database in <100ms. Snowflake-style copy-on-write.
```javascript
const brain = new Brainy() // Auto-detects everything
await brain.init() // Optimizes for your environment
```
Brainy automatically:
- Detects optimal storage (memory/filesystem/cloud)
- Configures memory based on available RAM
- Optimizes for containers (Docker/K8s)
- Tunes indexes for your data patterns
- Manages embedding models and caching
**You write business logic. Brainy handles infrastructure.**
### 🚀 **Git-Style Version Control** — Database & Entity Level (v5.0.0+)
**Clone your entire database in <100ms. Track every entity change. Full Git-style workflow.**
```javascript
// Fork instantly - Snowflake-style copy-on-write
const experiment = await brain.fork('test-migration')
// Make changes safely in isolation
await experiment.add({ type: 'user', data: { name: 'Test User' } })
await experiment.updateAll({ /* migration logic */ })
// Commit your work
await experiment.commit({ message: 'Add test user', author: 'dev@example.com' })
// Switch to experimental branch to make it active
await experiment.add({ data: 'test data', type: NounType.Concept })
await experiment.commit({ message: 'Add test data', author: 'dev@co.com' })
await brain.checkout('test-migration')
// Time-travel: Query database at any past commit (read-only)
const commits = await brain.getHistory({ limit: 10 })
const snapshot = await brain.asOf(commits[5].id)
// Time-travel: query at any past commit
const snapshot = await brain.asOf(commitId)
const pastResults = await snapshot.find({ query: 'historical data' })
await snapshot.close()
// Entity versioning: Track changes to individual entities (v5.3.0+)
const userId = await brain.add({ type: 'user', data: { name: 'Alice' } })
await brain.versions.save(userId, { tag: 'v1.0', description: 'Initial profile' })
await brain.update(userId, { data: { name: 'Alice Smith', role: 'admin' } })
await brain.versions.save(userId, { tag: 'v2.0', description: 'Added role' })
// Compare versions or restore previous state
const diff = await brain.versions.compare(userId, 1, 2) // See what changed
await brain.versions.restore(userId, 1) // Restore v1.0
```
**Database-level version control (v5.0.0):**
- ✅ `fork()` - Instant clone in <100ms
- ✅ `merge()` - Merge with conflict resolution
- ✅ `commit()` - Snapshot state
- ✅ `asOf()` - Time-travel queries (query at any commit)
- ✅ `getHistory()` - View commit history
- ✅ `checkout()`, `listBranches()` - Full branch management
- ✅ CLI support for all features
**[Branching Documentation](docs/features/instant-fork.md)**
**Entity-level version control (v5.3.0):**
- ✅ `versions.save()` - Save entity snapshots with tags
- ✅ `versions.restore()` - Restore previous versions
- ✅ `versions.compare()` - Diff between versions
- ✅ `versions.list()` - View version history
- ✅ Automatic deduplication (content-addressable storage)
### Entity Versioning
**How it works:** Snowflake-style COW shares HNSW index structures, copying only modified nodes (10-20% memory overhead).
**Perfect for:** Safe migrations, A/B testing, feature branches, distributed development, time-travel debugging, audit trails, document versioning, compliance tracking
[→ See Full Documentation](docs/features/instant-fork.md)
---
## What Can You Build?
**If your app needs to remember, understand, or connect information — Brainy makes it trivial.**
### 🤖 **AI Agents with Perfect Memory**
Give your AI unlimited context that persists forever. Not just chat history — true understanding of relationships, evolution, and meaning over time.
**Examples:** Personal assistants, code assistants, conversational AI, research agents
### 📚 **Living Documentation & Knowledge Bases**
Documentation that understands itself. Auto-links related concepts, detects outdated information, finds connections across your entire knowledge base.
**Examples:** Internal wikis, research platforms, smart documentation, learning systems
### 🔍 **Semantic Search at Any Scale**
Find by meaning, not keywords. Search codebases, research papers, customer data, or media libraries with natural language.
**Examples:** Code search, research platforms, content discovery, recommendation engines
### 🏢 **Enterprise Knowledge Management**
Corporate memory that never forgets. Track every customer interaction, product evolution, and business relationship.
**Examples:** CRM systems, product catalogs, customer intelligence, institutional knowledge
### 🎮 **Rich Interactive Experiences**
NPCs that remember. Characters that persist across stories. Worlds that evolve based on real relationships.
**Examples:** Game worlds, interactive fiction, educational platforms, creative tools
### 🎨 **Content & Media Platforms**
Every asset knows its relationships. Intelligent tagging, similarity-based discovery, and relationship-aware management.
**Examples:** DAM systems, media libraries, writing assistants, content management
**The pattern:** Knowledge that needs to live, connect, and evolve. That's what Brainy was built for.
---
## Core Features
### 🧠 **Natural Language Queries**
Save, restore, and compare entity snapshots.
```javascript
// Ask naturally - Brainy understands
await brain.find("recent React components with tests")
await brain.find("JavaScript libraries similar to Vue")
const userId = await brain.add({ data: 'Alice', type: NounType.Person })
await brain.versions.save(userId, { tag: 'v1.0' })
// Or use structured Triple Intelligence queries
await brain.find({
query: "React",
where: { type: "library", year: { greaterThan: 2020 } },
connected: { to: "JavaScript", depth: 2 }
})
await brain.update(userId, { data: 'Alice Smith' })
await brain.versions.save(userId, { tag: 'v2.0' })
const diff = await brain.versions.compare(userId, 1, 2)
await brain.versions.restore(userId, 1)
```
**→ [See all query methods in API Reference](docs/api/README.md#search--query)**
### Virtual Filesystem
### 🔍 **Zero-Config Hybrid Search** (v7.7.0)
Automatically combines text (keyword) and semantic (vector) search for optimal results:
File operations with semantic search built in.
```javascript
// Just works - no configuration needed
const results = await brain.find({ query: 'David Smith' })
// Finds exact text matches AND semantically similar content
const vfs = brain.vfs
// Override when needed
await brain.find({ query: 'exact match', searchMode: 'text' }) // Text only
await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // Semantic only
await brain.find({ query: 'hybrid', hybridAlpha: 0.3 }) // Custom weighting
// Highlight structured content (v7.8.0) — plain text, JSON, HTML, Markdown
const highlights = await brain.highlight({
query: 'warrior',
text: entity.data // Auto-detects TipTap, Slate, Lexical, HTML, Markdown
})
// Each highlight has: text, score, position, matchType, contentCategory
```
**→ [Hybrid Search Documentation](docs/api/README.md#hybrid-search-v770)**
### 🌐 **Virtual Filesystem** — Intelligent File Management
Build file explorers and IDEs that never crash:
```javascript
const vfs = brain.vfs()
// Tree-aware operations prevent infinite recursion
const tree = await vfs.getTreeStructure('/projects', { maxDepth: 3 })
await vfs.writeFile('/docs/readme.md', 'Project documentation')
const content = await vfs.readFile('/docs/readme.md')
const tree = await vfs.getTreeStructure('/docs', { maxDepth: 3 })
// Semantic file search
const reactFiles = await vfs.search('React components with hooks')
const matches = await vfs.search('React components with hooks')
```
**[📖 VFS Quick Start](docs/vfs/QUICK_START.md)** | **[Common Patterns](docs/vfs/COMMON_PATTERNS.md)** | **[Neural Extraction →](docs/vfs/NEURAL_EXTRACTION.md)**
**[VFS Quick Start](docs/vfs/QUICK_START.md)** | **[Common Patterns](docs/vfs/COMMON_PATTERNS.md)**
### 🚀 **Import Anything** — CSV, Excel, PDF, URLs
### Import Anything
CSV, Excel, PDF, URLs — auto-detected format, auto-classified entities.
```javascript
await brain.import('customers.csv') // Auto-detects everything
await brain.import('customers.csv')
await brain.import('sales-data.xlsx', { excelSheets: ['Q1', 'Q2'] })
await brain.import('research-paper.pdf', { pdfExtractTables: true })
await brain.import('https://api.example.com/data.json')
```
**[📖 Complete Import Guide](docs/guides/import-anything.md)**
**[Import Guide](docs/guides/import-anything.md)**
### 🧠 **Neural API** — Advanced Semantic Analysis
### Entity Extraction
AI-powered named entity recognition with 4-signal ensemble scoring.
```javascript
// Clustering, similarity, outlier detection, visualization
const clusters = await brain.neural.clusters({ algorithm: 'kmeans' })
const similarity = await brain.neural.similar('item1', 'item2')
const outliers = await brain.neural.outliers(0.3)
const vizData = await brain.neural.visualize({ maxNodes: 100 })
const entities = await brain.extractEntities('John Smith founded Acme Corp in New York')
// [
// { text: 'John Smith', type: NounType.Person, confidence: 0.95 },
// { text: 'Acme Corp', type: NounType.Organization, confidence: 0.92 },
// { text: 'New York', type: NounType.Location, confidence: 0.88 }
// ]
```
**[Neural Extraction Guide](docs/neural-extraction.md)**
### Plugin System
Optional native acceleration via `@soulcraft/cortex` — SIMD distance calculations, CRoaring bitmaps, Candle ML embeddings.
```javascript
const brain = new Brainy({ plugins: ['@soulcraft/cortex'] })
await brain.init()
```
Plugins are opt-in. Brainy never auto-imports packages unless listed in `plugins`.
**[Plugin Documentation](docs/PLUGINS.md)**
---
## Framework Integration
## Type System
**Works with any modern framework.** React, Vue, Angular, Svelte, Solid.js — your choice.
42 noun types and 127 verb types form a universal knowledge protocol:
```javascript
// React
const [brain] = useState(() => new Brainy())
useEffect(() => { brain.init() }, [])
// Vue
async mounted() { this.brain = await new Brainy().init() }
// Angular
@Injectable() export class BrainyService { brain = new Brainy() }
```
42 Nouns × 127 Verbs = 5,334 base relationship combinations
```
**Supports:** All bundlers (Webpack, Vite, Rollup) • SSR/SSG • Edge runtimes • Browser/Node.js
Model any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), education (`Student → completes → Course`), or your own.
**[📖 Framework Integration Guide →](docs/guides/framework-integration.md)** | **[Next.js →](docs/guides/nextjs-integration.md)** | **[Vue →](docs/guides/vue-integration.md)**
**[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** | **[Stage 3 Canonical Reference](docs/STAGE3-CANONICAL-TAXONOMY.md)**
---
## Storage — From Memory to Planet-Scale
## Storage: Memory to Cloud
The same API at every scale. Change one config line to go from prototype to production.
### Development — Zero Config
### Development → Just Works
```javascript
const brain = new Brainy() // Memory storage, zero config
const brain = new Brainy()
```
### Production → Persistence with Compression
### Production — Filesystem with Compression
```javascript
const brain = new Brainy({
storage: { type: 'filesystem', path: './data', compression: true }
})
// 60-80% space savings with gzip
```
### Cloud → AWS, GCS, Azure, Cloudflare R2
### Cloud — S3, GCS, Azure, Cloudflare R2
```javascript
// AWS S3 / Cloudflare R2
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'my-knowledge-base',
region: 'us-east-1'
}
s3Storage: { bucketName: 'my-knowledge-base', region: 'us-east-1' }
}
})
// Enable Intelligent-Tiering: 96% cost savings
await brain.storage.enableIntelligentTiering('entities/', 'auto-tier')
```
**Cost optimization at scale:**
Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)**.
| Scale | Standard | With Intelligent Tiering | Annual Savings |
|-------|----------|--------------------------|----------------|
| 5TB | $1,380 | $59 | $1,321 (96%) |
| 50TB | $13,800 | $594 | $13,206 (96%) |
| 500TB | $138,000 | $5,940 | $132,060 (96%) |
**[📖 Cloud Storage Guide →](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** | **[AWS Cost Optimization →](docs/operations/cost-optimization-aws-s3.md)** | **[GCS →](docs/operations/cost-optimization-gcs.md)** | **[Azure →](docs/operations/cost-optimization-azure.md)**
**[Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** | **[Capacity Planning](docs/operations/capacity-planning.md)**
---
## Production Features
## Use Cases
### 🎯 Type-Aware HNSW Indexing
Efficient type-based organization for large-scale deployments:
- **Type-based queries:** Faster via directory structure (measured at 1K-1M scale)
- **Type count tracking:** 284 bytes (Uint32Array, measured)
- **Billion-scale projections:** NOT tested at 1B entities (extrapolated from 1M)
```javascript
const brain = new Brainy({ hnsw: { typeAware: true } })
```
**[📖 How Type-Aware Indexing Works →](docs/architecture/data-storage-architecture.md)**
### ⚡ Enterprise-Ready Operations (v4.0.0)
- **Batch operations** with retry logic (1000x faster deletes)
- **Gzip compression** (60-80% space savings)
- **OPFS quota monitoring** (browser storage)
- **Metadata/Vector separation** (billion-entity scalability)
- **Circuit breakers & backpressure** (enterprise reliability)
```javascript
// Batch operations
await brain.storage.batchDelete(keys, { maxRetries: 3 })
// Monitor storage
const status = await brain.storage.getStorageStatus()
```
### 📊 Adaptive Memory Management
Auto-scales 2GB → 128GB+ based on environment:
- Container-aware (Docker/K8s cgroups)
- Environment-optimized (dev/staging/production)
- Built-in cache monitoring with tuning recommendations
```javascript
const stats = brain.getCacheStats() // Performance insights
```
**[📖 Capacity Planning Guide →](docs/operations/capacity-planning.md)**
### Native Acceleration (Optional)
Install `@soulcraft/cortex` for Rust-powered native acceleration: SIMD distance calculations, native metadata/graph indexes, CRoaring bitmaps, and Candle ML embeddings.
```bash
npm install @soulcraft/cortex
```
```typescript
const brain = new Brainy({
plugins: ['@soulcraft/cortex']
})
await brain.init()
// [brainy] Plugin activated: @soulcraft/cortex
```
Plugins are opt-in — brainy never auto-imports packages unless you list them in `plugins`.
- **AI agents** — Persistent memory with semantic recall and relationship tracking
- **Knowledge bases** — Auto-linking, semantic search, relationship-aware navigation
- **Semantic search** — Find by meaning across codebases, documents, or media
- **Enterprise knowledge** — CRM, product catalogs, institutional memory
- **Interactive experiences** — Game worlds, NPCs, and characters that remember
- **Content platforms** — Similarity-based discovery, intelligent tagging
---
## Benchmarks
## Documentation
| Operation | Performance | Memory | Notes |
|-----------|-------------|--------|-------|
| Initialize | 450ms | 24MB | Measured |
| Add entity | 12ms | +0.1MB | Measured |
| Vector search (1K) | 3ms | - | Measured |
| Metadata filter (10K) | 0.8ms | - | Measured |
| Bulk import (1K) | 2.3s | +8MB | Measured |
| **10M entities** | **5.8ms** | **12GB** | Measured |
| **1B entities** | **~18ms** | **~50GB** | PROJECTED (extrapolated from 10M) |
### Core
---
- **[API Reference](docs/api/README.md)** — Every method with parameters, returns, and examples
- **[Data Model](docs/DATA_MODEL.md)** — Entity structure, data vs metadata
- **[Query Operators](docs/QUERY_OPERATORS.md)** — All BFO operators with examples
- **[Find System](docs/FIND_SYSTEM.md)** — Natural language find() and hybrid search
## 🧠 Deep Dive: How Brainy Actually Works
### Architecture
**Want to understand the magic under the hood?**
### 🔍 Triple Intelligence & find() API
Understand how vector search, graph relationships, and document filtering work together in one unified query:
**[📖 Triple Intelligence Architecture →](docs/architecture/triple-intelligence.md)**
**[📖 Natural Language Guide →](docs/guides/natural-language.md)**
**[📖 API Reference: find() →](docs/api/README.md)**
### 🗂️ Type-Aware Indexing & HNSW
Learn about our indexing architecture with measured performance optimizations:
**[📖 Data Storage Architecture →](docs/architecture/data-storage-architecture.md)**
**[📖 Architecture Overview →](docs/architecture/overview.md)**
### 📈 Scaling: Individual → Planet
Understand how the same code scales from prototype to billions of entities:
**[📖 Capacity Planning →](docs/operations/capacity-planning.md)**
**[📖 Cloud Deployment Guide →](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)**
### 🎯 The Universal Type System
Explore the mathematical foundation: 42 nouns × 127 verbs = Stage 3 CANONICAL taxonomy:
**[📖 Noun-Verb Taxonomy →](docs/architecture/noun-verb-taxonomy.md)**
---
## CLI Tools
```bash
npm install -g brainy
brainy add "JavaScript is awesome" --metadata '{"type":"opinion"}'
brainy find "awesome programming languages"
brainy search "programming"
```
47 commands available, including storage management, imports, and neural operations.
---
## 📖 Complete Documentation
**For most developers:** Start with the **[Complete API Reference](docs/api/README.md)** ⭐
This comprehensive guide includes:
- ✅ Every method with parameters, returns, and examples
- ✅ Quick start in 60 seconds
- ✅ Core CRUD → Advanced features (branching, versioning, time-travel)
- ✅ TypeScript types and patterns
- ✅ 1,870 lines of copy-paste ready code
---
### 🎯 Essential Reading (Start Here)
1. **[📖 Complete API Reference](docs/api/README.md)** ⭐ **START HERE**
- Your primary resource for building with Brainy
- Every method documented with working examples
2. **[Filter & Query Syntax Guide](docs/FIND_SYSTEM.md)**
- Complete reference for operators, compound filters, and optimization tips
3. **[Natural Language Queries](docs/guides/natural-language.md)**
- Master the `find()` method and Triple Intelligence queries
4. **[v4.0.0 Migration Guide](docs/MIGRATION-V3-TO-V4.md)**
- Upgrading from v3 (100% backward compatible)
### 🧠 Core Concepts & Architecture
- **[Triple Intelligence Architecture](docs/architecture/triple-intelligence.md)** — How vector + graph + document work together
- **[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** — The universal type system (42 nouns × 127 verbs)
- **[Transactions](docs/transactions.md)** — Atomic operations with automatic rollback
- **[Architecture Overview](docs/architecture/overview.md)** — System design and components
- **[Triple Intelligence](docs/architecture/triple-intelligence.md)** — Vector + graph + metadata unified query
- **[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** — Universal type system
- **[Data Storage Architecture](docs/architecture/data-storage-architecture.md)** — Type-aware indexing and HNSW
### ☁️ Production & Operations
- **[Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** — Deploy to AWS, GCS, Azure
- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling to billions
- **Cost Optimization:** **[AWS S3](docs/operations/cost-optimization-aws-s3.md)** | **[GCS](docs/operations/cost-optimization-gcs.md)** | **[Azure](docs/operations/cost-optimization-azure.md)** | **[Cloudflare R2](docs/operations/cost-optimization-cloudflare-r2.md)**
### 🌐 Framework Integration
- **[Framework Integration Guide](docs/guides/framework-integration.md)** — React, Vue, Angular, Svelte
- **[Next.js Integration](docs/guides/nextjs-integration.md)**
- **[Vue.js Integration](docs/guides/vue-integration.md)**
### 🌳 Virtual Filesystem (VFS)
### Virtual Filesystem
- **[VFS Quick Start](docs/vfs/QUICK_START.md)** — Build file explorers that never crash
- **[VFS Core Documentation](docs/vfs/VFS_CORE.md)**
- **[Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md)** — AI-powered file navigation
- **[Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md)**
- **[VFS Core](docs/vfs/VFS_CORE.md)** — Full VFS API reference
- **[Semantic VFS](docs/vfs/SEMANTIC_VFS.md)** — AI-powered file navigation
### 📦 Data Import & Processing
### Guides
- **[Import Anything Guide](docs/guides/import-anything.md)** — CSV, Excel, PDF, URLs with auto-detection
- **[Import Anything](docs/guides/import-anything.md)** — CSV, Excel, PDF, URLs
- **[Framework Integration](docs/guides/framework-integration.md)** — React, Vue, Angular, Svelte
- **[Natural Language Queries](docs/guides/natural-language.md)** — Master the find() method
---
### Operations
## What's New in v4.0.0
**Enterprise-scale cost optimization and performance improvements:**
- 🎯 **96% cloud storage cost savings** with intelligent tiering (AWS, GCS, Azure)
- ⚡ **1000x faster batch deletions** (533 entities/sec vs 0.5/sec)
- 📦 **60-80% compression** with gzip (FileSystem storage)
- 🔄 **Enhanced metadata/vector separation** for billion-scale deployments
**[📖 Full v4.0.0 Changelog →](CHANGELOG.md)** | **[Migration Guide →](docs/MIGRATION-V3-TO-V4.md)** (100% backward compatible)
- **[Cloud Deployment](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** — AWS, GCS, Azure
- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling
- **[Performance](docs/PERFORMANCE.md)** — Benchmarks and architecture details
- Cost Optimization: **[AWS S3](docs/operations/cost-optimization-aws-s3.md)** | **[GCS](docs/operations/cost-optimization-gcs.md)** | **[Azure](docs/operations/cost-optimization-azure.md)** | **[R2](docs/operations/cost-optimization-cloudflare-r2.md)**
---
## Requirements
> **Deprecation Notice:** Browser support (OPFS storage, Web Workers, WASM embeddings) is deprecated in v7.10.0 and will be removed in v8.0.0. Brainy v8+ will be server-only (Node.js >= 22, Bun >= 1.3.0). Migrate browser deployments to a server-side API.
**Bun 1.0+** (recommended) or **Node.js 22 LTS**
```bash
# Bun (recommended - best performance, single-binary deployment)
bun install @soulcraft/brainy
# Node.js (fully supported)
npm install @soulcraft/brainy
bun install @soulcraft/brainy # Bun — best performance
npm install @soulcraft/brainy # Node.js — fully supported
```
> **Why Bun?** Brainy's Candle WASM engine works seamlessly with `bun --compile` for standalone binary deployment. No external model files, no runtime downloads.
---
## Why Brainy Exists
**The Vision:** Traditional systems force you to choose between vector databases, graph databases, and document stores. You need all three, but combining them is complex and fragile.
**Brainy solved the impossible:** One API. All three paradigms. Any scale.
Like HTTP standardized web communication, **Brainy standardizes knowledge representation.** One protocol that any AI model understands. One system that scales from prototype to planet.
**[📖 Read the Mathematical Proof →](docs/architecture/noun-verb-taxonomy.md)**
---
## Enterprise & Support
**🏢 Brain Cloud** — Managed Brainy with team sync, persistent memory, and enterprise connectors.
Visit **[soulcraft.com](https://soulcraft.com)** for more information.
**💖 Support Development:**
- ⭐ Star us on **[GitHub](https://github.com/soulcraftlabs/brainy)**
- 💝 Sponsor via **[GitHub Sponsors](https://github.com/sponsors/soulcraftlabs)**
- 🐛 Report issues and contribute code
- 📣 Share with your team and community
**Brainy is 100% free and open source.** No paywalls, no premium tiers, no feature gates.
---
> **Deprecation Notice:** Browser support (OPFS, Web Workers, WASM embeddings) is deprecated in v7.10.0 and will be removed in v8.0.0. Brainy v8+ will be server-only.
## Contributing
We welcome contributions! See **[CONTRIBUTING.md](CONTRIBUTING.md)** for guidelines.
---
## License
MIT © Brainy Contributors
---
<p align="center">
<strong>Built with ❤️ by the Brainy community</strong><br>
<em>The Knowledge Operating System</em><br>
<em>From prototype to planet-scale • Zero configuration • Triple Intelligence™</em>
</p>