2025-08-26 12:32:21 -07:00
# Brainy
< p align = "center" >
2025-08-26 13:48:09 -07:00
< img src = "https://raw.githubusercontent.com/soulcraftlabs/brainy/main/brainy.png" alt = "Brainy Logo" width = "200" >
2025-08-26 12:32:21 -07:00
< / p >
2025-08-26 13:48:09 -07:00
[](https://www.npmjs.com/package/@soulcraft/brainy )
[](https://www.npmjs.com/package/@soulcraft/brainy )
2025-08-26 12:32:21 -07:00
[](LICENSE)
[](https://www.typescriptlang.org/)
2025-09-25 10:47:44 -07:00
**🧠 Brainy - The Knowledge Operating System**
2025-08-26 12:32:21 -07:00
2025-09-25 10:47:44 -07:00
**The world's first Knowledge Operating System** where every piece of knowledge - files, concepts, entities, ideas - exists as living information that understands itself, evolves over time, and connects to everything related. Built on revolutionary Triple Intelligence™ that unifies vector similarity, graph relationships, and document filtering in one magical API.
2025-08-27 09:27:12 -07:00
2025-09-25 10:47:44 -07:00
**Why Brainy Changes Everything**: Traditional systems trap knowledge in files or database rows. Brainy liberates it. Your characters exist across stories. Your concepts span projects. Your APIs remember their evolution. Every piece of knowledge - whether it's code, prose, or pure ideas - lives, breathes, and connects in a unified intelligence layer where everything understands its meaning, remembers its history, and relates to everything else.
2025-08-27 09:27:12 -07:00
2025-09-15 14:53:59 -07:00
**Framework-first design.** Built for modern web development with zero configuration and automatic framework compatibility. O(log n) performance, < 10ms search latency , production-ready .
2025-08-26 12:32:21 -07:00
2025-09-22 16:19:27 -07:00
## 🎉 Key Features
2025-08-26 12:32:21 -07:00
2025-10-01 15:12:54 -07:00
### 🚀 **NEW in 3.21.0: Enhanced Import & Neural Processing**
- **📊 Progress Tracking**: Unified progress reporting with automatic time estimation
- **⚡ Entity Caching**: 10-100x speedup on repeated entity extraction
- **🔗 Relationship Confidence**: Multi-factor confidence scoring (0-1 scale)
- **📝 Evidence Tracking**: Understand why relationships were detected
- **🎯 Production Ready**: Fully backward compatible, opt-in features
2025-09-15 11:06:16 -07:00
### 🧠 **Triple Intelligence™ Engine**
2025-09-11 16:23:32 -07:00
2025-09-15 11:06:16 -07:00
- **Vector Search**: HNSW-powered semantic similarity
- **Graph Relationships**: Navigate connected knowledge
- **Document Filtering**: MongoDB-style metadata queries
- **Unified API**: All three in a single query interface
2025-09-11 16:23:32 -07:00
2025-09-15 11:06:16 -07:00
### 🎯 **Clean API Design**
2025-09-11 16:23:32 -07:00
2025-09-15 11:06:16 -07:00
- **Modern Syntax**: `brain.add()` , `brain.find()` , `brain.relate()`
- **Type Safety**: Full TypeScript integration
- **Zero Config**: Works out of the box with memory storage
- **Consistent Parameters**: Clean, predictable API surface
2025-09-11 16:23:32 -07:00
2025-09-15 11:06:16 -07:00
### ⚡ **Performance & Reliability**
2025-09-11 16:23:32 -07:00
2025-09-15 11:06:16 -07:00
- **< 10ms Search ** : Fast semantic queries
- **384D Vectors**: Optimized embeddings (all-MiniLM-L6-v2)
2025-10-01 15:12:54 -07:00
- **Built-in Caching**: Intelligent result caching + new entity extraction cache
2025-09-15 11:06:16 -07:00
- **Production Ready**: Thoroughly tested core functionality
2025-08-26 12:32:21 -07:00
2025-08-29 15:39:07 -07:00
## ⚡ Quick Start - Zero Configuration
2025-08-26 12:32:21 -07:00
```bash
2025-08-26 13:48:09 -07:00
npm install @soulcraft/brainy
2025-08-26 12:32:21 -07:00
```
2025-08-29 15:39:07 -07:00
### 🎯 **True Zero Configuration**
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
```javascript
2025-09-22 16:19:27 -07:00
import { Brainy, NounType } from '@soulcraft/brainy '
2025-08-26 12:32:21 -07:00
2025-08-29 15:39:07 -07:00
// Just this - auto-detects everything!
2025-09-15 11:06:16 -07:00
const brain = new Brainy()
2025-08-26 12:32:21 -07:00
await brain.init()
2025-09-15 11:06:16 -07:00
// Add entities with automatic embedding
const jsId = await brain.add({
data: "JavaScript is a programming language",
2025-09-22 16:19:27 -07:00
nounType: NounType.Concept,
2025-09-15 11:06:16 -07:00
metadata: {
type: "language",
year: 1995,
paradigm: "multi-paradigm"
}
2025-08-26 12:32:21 -07:00
})
2025-09-15 11:06:16 -07:00
const nodeId = await brain.add({
data: "Node.js runtime environment",
2025-09-22 16:19:27 -07:00
nounType: NounType.Concept,
2025-09-15 11:06:16 -07:00
metadata: {
type: "runtime",
year: 2009,
platform: "server-side"
}
2025-08-27 09:27:12 -07:00
})
2025-08-26 12:32:21 -07:00
2025-09-15 11:06:16 -07:00
// Create relationships between entities
await brain.relate({
from: nodeId,
to: jsId,
type: "executes",
metadata: {
since: 2009,
performance: "high"
}
2025-08-27 09:27:12 -07:00
})
// Natural language search with graph relationships
2025-09-15 11:06:16 -07:00
const results = await brain.find({query: "programming languages used by server runtimes"})
2025-08-27 09:27:12 -07:00
// Triple Intelligence: vector + metadata + relationships
const filtered = await brain.find({
2025-09-15 11:06:16 -07:00
query: "JavaScript", // Vector similarity
where: {type: "language"}, // Metadata filtering
2025-09-11 16:23:32 -07:00
connected: {from: nodeId, depth: 1} // Graph relationships
2025-08-26 12:32:21 -07:00
})
```
2025-09-15 14:53:59 -07:00
## 🌐 Framework Integration
2025-09-22 16:19:27 -07:00
**Brainy is framework-first!** Works seamlessly with any modern JavaScript framework:
2025-09-15 14:53:59 -07:00
### ⚛️ **React & Next.js**
```javascript
import { Brainy } from '@soulcraft/brainy '
function SearchComponent() {
const [brain] = useState(() => new Brainy())
useEffect(() => {
brain.init()
}, [])
const handleSearch = async (query) => {
const results = await brain.find(query)
setResults(results)
}
}
```
### 🟢 **Vue.js & Nuxt.js**
```javascript
import { Brainy } from '@soulcraft/brainy '
export default {
async mounted() {
this.brain = new Brainy()
await this.brain.init()
},
methods: {
async search(query) {
return await this.brain.find(query)
}
}
}
```
### 🅰️ **Angular**
```typescript
import { Injectable } from '@angular/core '
import { Brainy } from '@soulcraft/brainy '
@Injectable ({ providedIn: 'root' })
export class BrainyService {
private brain = new Brainy()
async init() {
await this.brain.init()
}
async search(query: string) {
return await this.brain.find(query)
}
}
```
### 🔥 **Other Frameworks**
Brainy works with **any** framework that supports ES6 imports: Svelte, Solid.js, Qwik, Fresh, and more!
**Framework Compatibility:**
- ✅ All modern bundlers (Webpack, Vite, Rollup, Parcel)
- ✅ SSR/SSG (Next.js, Nuxt, SvelteKit, Astro)
- ✅ Edge runtimes (Vercel Edge, Cloudflare Workers)
- ✅ Browser and Node.js environments
2025-08-28 16:05:14 -07:00
## 📋 System Requirements
**Node.js Version:** 22 LTS or later (recommended)
- ✅ **Node.js 22 LTS** - Fully supported and recommended for production
- ✅ **Node.js 20 LTS** - Compatible (maintenance mode)
- ❌ **Node.js 24** - Not supported (known ONNX runtime compatibility issues)
2025-09-11 16:23:32 -07:00
> **Important:** Brainy uses ONNX runtime for AI embeddings. Node.js 24 has known compatibility issues that cause
> crashes during inference operations. We recommend Node.js 22 LTS for maximum stability.
2025-08-28 16:05:14 -07:00
If using nvm: `nvm use` (we provide a `.nvmrc` file)
2025-08-26 12:32:21 -07:00
## 🚀 Key Features
2025-08-27 09:27:12 -07:00
### World's First Triple Intelligence™ Engine
2025-09-11 16:23:32 -07:00
2025-09-25 10:47:44 -07:00
**The breakthrough that enables The Knowledge Operating System:**
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
- **Vector Search**: Semantic similarity with HNSW indexing
2025-08-27 09:27:12 -07:00
- **Graph Relationships**: Navigate connected knowledge like Neo4j
- **Document Filtering**: MongoDB-style queries with O(log n) performance
- **Unified in ONE API**: No separate queries, no complex joins
- **First to solve this**: Others do vector OR graph OR document—we do ALL
2025-09-25 10:47:44 -07:00
### The Knowledge Operating System with Infinite Expressiveness
2025-09-11 16:23:32 -07:00
2025-08-27 09:27:12 -07:00
**Enabled by Triple Intelligence, standardized for everyone:**
2025-09-11 16:23:32 -07:00
2025-09-22 16:19:27 -07:00
- **31 Noun Types × 40 Verb Types**: 1,240 base combinations
2025-08-27 09:27:12 -07:00
- **∞ Expressiveness**: Unlimited metadata = model ANY data
- **One Language**: All tools, augmentations, AI models speak the same types
- **Perfect Interoperability**: Move data between any Brainy instance
- **No Schema Lock-in**: Evolve without migrations
2025-08-26 12:32:21 -07:00
### Natural Language Understanding
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
```javascript
// Ask questions naturally
await brain.find("Show me recent React components with tests")
await brain.find("Popular JavaScript libraries similar to Vue")
await brain.find("Documentation about authentication from last month")
```
2025-09-26 13:32:44 -07:00
### 🧠🌐 **Virtual Filesystem - Intelligent File Management**
2025-09-25 10:47:44 -07:00
2025-09-26 13:32:44 -07:00
**Build file explorers, IDEs, and knowledge systems that never crash from infinite recursion.**
2025-09-25 10:47:44 -07:00
2025-09-26 13:32:44 -07:00
- **Tree-Aware Operations**: Safe directory listing prevents recursive loops
- **Semantic Search**: Find files by content, not just filename
- **Production Storage**: Filesystem and cloud storage for real applications
- **Zero-Config**: Works out of the box with intelligent defaults
2025-09-25 10:47:44 -07:00
```javascript
import { Brainy } from '@soulcraft/brainy '
2025-09-26 13:32:44 -07:00
// ✅ CORRECT: Use persistent storage for file systems
const brain = new Brainy({
storage: {
type: 'filesystem', // Persisted to disk
path: './brainy-data' // Your file storage
}
})
2025-09-25 10:47:44 -07:00
await brain.init()
2025-09-26 13:32:44 -07:00
2025-09-25 10:47:44 -07:00
const vfs = brain.vfs()
await vfs.init()
2025-09-26 13:32:44 -07:00
// ✅ Safe file operations
await vfs.writeFile('/projects/app/index.js', 'console.log("Hello")')
await vfs.mkdir('/docs')
await vfs.writeFile('/docs/README.md', '# My Project')
2025-09-25 10:47:44 -07:00
2025-09-26 13:32:44 -07:00
// ✅ NEVER crashes: Tree-aware directory listing
const children = await vfs.getDirectChildren('/projects')
// Returns only direct children, never the directory itself
2025-09-25 10:47:44 -07:00
2025-09-26 13:32:44 -07:00
// ✅ Build file explorers safely
const tree = await vfs.getTreeStructure('/projects', {
maxDepth: 3, // Prevent deep recursion
sort: 'name' // Organized results
2025-09-25 10:47:44 -07:00
})
2025-09-26 13:32:44 -07:00
// ✅ Semantic file search
const reactFiles = await vfs.search('React components with hooks')
const docs = await vfs.search('API documentation', {
path: '/docs' // Search within specific directory
})
2025-09-25 10:47:44 -07:00
2025-09-26 13:32:44 -07:00
// ✅ Connect related files
await vfs.addRelationship('/src/auth.js', '/tests/auth.test.js', 'tested-by')
2025-09-25 10:47:44 -07:00
2025-09-26 13:32:44 -07:00
// Perfect for: File explorers, IDEs, documentation systems, code analysis
2025-09-25 10:47:44 -07:00
```
2025-09-26 13:32:44 -07:00
**🚨 Prevents Common Mistakes:**
- ❌ No infinite recursion in file trees (like brain-cloud team experienced)
- ❌ No data loss from memory storage
- ❌ No performance issues with large directories
- ❌ No need for complex fallback patterns
**[📖 VFS Quick Start → ](docs/vfs/QUICK_START.md )** | ** [🎯 Common Patterns → ](docs/vfs/COMMON_PATTERNS.md )**
2025-09-25 10:47:44 -07:00
**Your knowledge isn't trapped anymore.** Characters live beyond stories. APIs exist beyond code files. Concepts connect across domains. This is knowledge that happens to support files, not a filesystem that happens to store knowledge.
2025-10-01 15:12:54 -07:00
### 🚀 **NEW: Enhanced Directory Import with Caching**
**Import large projects 10-100x faster with intelligent caching:**
```javascript
import { Brainy } from '@soulcraft/brainy '
import { ProgressTracker, formatProgress } from '@soulcraft/brainy/types '
import { detectRelationshipsWithConfidence } from '@soulcraft/brainy/neural '
const brain = new Brainy()
await brain.init()
// Progress tracking for long operations
const tracker = ProgressTracker.create(1000)
tracker.start()
for await (const progress of importer.importStream('./project', {
batchSize: 100,
generateEmbeddings: true
})) {
const p = tracker.update(progress.processed, progress.current)
console.log(formatProgress(p))
// [RUNNING] 45% (450/1000) - 23.5 items/s - 23s remaining
}
// Entity extraction with intelligent caching
const entities = await brain.neural.extractor.extract(text, {
types: ['person', 'organization', 'technology'],
confidence: 0.7,
cache: {
enabled: true,
ttl: 7 * 24 * 60 * 60 * 1000, // 7 days
invalidateOn: 'mtime' // Re-extract when file changes
}
})
// Relationship detection with confidence scores
const relationships = detectRelationshipsWithConfidence(entities, text, {
minConfidence: 0.7
})
// Create relationships with evidence tracking
await brain.relate({
from: sourceId,
to: targetId,
type: 'creates',
confidence: 0.85,
evidence: {
sourceText: 'John created the database',
method: 'pattern',
reasoning: 'Matches creation pattern; entities in same sentence'
}
})
// Monitor cache performance
const stats = brain.neural.extractor.getCacheStats()
console.log(`Cache hit rate: ${(stats.hitRate * 100).toFixed(1)}%` )
// Cache hit rate: 89.5%
```
**📚 [See Full Example → ](examples/directory-import-with-caching.ts )**
2025-08-29 15:39:07 -07:00
### 🎯 Zero Configuration Philosophy
2025-09-11 16:23:32 -07:00
2025-09-22 16:19:27 -07:00
Brainy automatically configures **everything** :
2025-08-29 15:39:07 -07:00
```javascript
2025-09-22 16:19:27 -07:00
import { Brainy } from '@soulcraft/brainy '
2025-08-29 15:39:07 -07:00
// 1. Pure zero-config - detects everything
2025-09-15 11:06:16 -07:00
const brain = new Brainy()
2025-08-29 15:39:07 -07:00
2025-09-15 11:06:16 -07:00
// 2. Custom configuration
const brain = new Brainy({
storage: { type: 'memory' },
embeddings: { model: 'all-MiniLM-L6-v2' },
cache: { enabled: true, maxSize: 1000 }
})
2025-08-29 15:39:07 -07:00
2025-09-15 11:06:16 -07:00
// 3. Production configuration
const customBrain = new Brainy({
2025-09-11 16:23:32 -07:00
mode: 'production',
model: 'q8', // Optimized model (99% accuracy, 75% smaller)
storage: 'cloud', // or 'memory', 'disk', 'auto'
features: ['core', 'search', 'cache']
2025-08-29 15:39:07 -07:00
})
```
**What's Auto-Detected:**
2025-09-11 16:23:32 -07:00
2025-08-29 15:39:07 -07:00
- **Storage**: S3/GCS/R2 → Filesystem → Memory (priority order)
2025-09-11 16:23:32 -07:00
- **Models**: Always Q8 for optimal balance
2025-08-29 15:39:07 -07:00
- **Features**: Minimal → Default → Full based on environment
- **Memory**: Optimal cache sizes and batching
- **Performance**: Threading, chunking, indexing strategies
2025-08-26 12:32:21 -07:00
### Production Performance
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
- **3ms average search** - Lightning fast queries
- **24MB memory footprint** - Efficient resource usage
- **Worker-based embeddings** - Non-blocking operations
- **Automatic caching** - Intelligent result caching
2025-08-29 15:39:07 -07:00
### 🎛️ Advanced Configuration (When Needed)
2025-08-29 11:09:40 -07:00
2025-08-29 15:39:07 -07:00
Most users **never need this** - zero-config handles everything. For advanced use cases:
2025-08-29 11:09:40 -07:00
```javascript
2025-09-11 16:23:32 -07:00
// Model is always Q8 for optimal performance
2025-09-15 11:06:16 -07:00
const brain = new Brainy() // Uses Q8 automatically
2025-08-29 15:39:07 -07:00
// Storage control (auto-detected by default)
2025-09-15 11:06:16 -07:00
const memoryBrain = new Brainy({storage: 'memory'}) // RAM only
const diskBrain = new Brainy({storage: 'disk'}) // Local filesystem
const cloudBrain = new Brainy({storage: 'cloud'}) // S3/GCS/R2
2025-08-29 11:09:40 -07:00
2025-08-29 15:39:07 -07:00
// Legacy full config (still supported)
2025-09-15 11:06:16 -07:00
const legacyBrain = new Brainy({
2025-09-11 16:23:32 -07:00
storage: {forceMemoryStorage: true}
2025-08-29 11:09:40 -07:00
})
```
2025-09-11 16:23:32 -07:00
**Model Details:**
2025-08-29 11:09:40 -07:00
2025-09-11 16:23:32 -07:00
- **Q8**: 33MB, 99% accuracy, 75% smaller than full precision
- Fast loading and optimal memory usage
- Perfect for all environments
2025-08-29 11:09:40 -07:00
**Air-gap deployment:**
2025-09-11 16:23:32 -07:00
2025-08-29 11:09:40 -07:00
```bash
2025-09-11 16:23:32 -07:00
npm run download-models # Download Q8 model
npm run download-models:q8 # Download Q8 model
2025-08-29 11:09:40 -07:00
```
2025-10-01 16:51:03 -07:00
## 🚀 Import Anything - Files, Data, URLs
Brainy's universal import intelligently handles **any data format** :
```javascript
// Import CSV with auto-detection
await brain.import('customers.csv')
// ✨ Auto-detects: encoding, delimiter, types, creates entities!
// Import Excel workbooks with multi-sheet support
await brain.import('sales-data.xlsx', {
excelSheets: ['Q1', 'Q2'] // or 'all' for all sheets
})
// ✨ Processes all sheets, preserves structure, infers types!
// Import PDF documents with table extraction
await brain.import('research-paper.pdf', {
pdfExtractTables: true
})
// ✨ Extracts text, detects tables, preserves metadata!
// Import JSON/YAML data
await brain.import([
{ name: 'Alice', role: 'Engineer' },
{ name: 'Bob', role: 'Designer' }
])
// ✨ Automatically creates Person entities with relationships!
// Import from URLs (auto-fetched)
await brain.import('https://api.example.com/data.json')
// ✨ Auto-detects URL, fetches, parses, processes!
```
**📖 [Complete Import Guide → ](docs/guides/import-anything.md )** | ** [Live Example → ](examples/import-excel-pdf-csv.ts )**
2025-08-26 12:32:21 -07:00
## 📚 Core API
### `search()` - Vector Similarity
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
```javascript
const results = await brain.search("machine learning", {
2025-09-11 16:23:32 -07:00
limit: 10, // Number of results
metadata: {type: "article"}, // Filter by metadata
includeContent: true // Include full content
2025-08-26 12:32:21 -07:00
})
```
### `find()` - Natural Language Queries
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
```javascript
// Simple natural language
const results = await brain.find("recent important documents")
// Structured query with Triple Intelligence
const results = await brain.find({
2025-09-11 16:23:32 -07:00
like: "JavaScript", // Vector similarity
where: { // Metadata filters
year: {greaterThan: 2020},
important: true
},
related: {to: "React"} // Graph relationships
2025-08-26 12:32:21 -07:00
})
```
### CRUD Operations
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
```javascript
2025-08-27 09:27:12 -07:00
// Create entities (nouns)
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
const id = await brain.add(data, { nounType: nounType, ...metadata })
2025-08-26 12:32:21 -07:00
2025-08-27 09:27:12 -07:00
// Create relationships (verbs)
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
const verbId = await brain.relate(sourceId, targetId, "relationType", {
2025-09-11 16:23:32 -07:00
strength: 0.9,
bidirectional: false
2025-08-27 09:27:12 -07:00
})
2025-08-26 12:32:21 -07:00
// Read
const item = await brain.getNoun(id)
2025-08-27 09:27:12 -07:00
const verb = await brain.getVerb(verbId)
2025-08-26 12:32:21 -07:00
// Update
await brain.updateNoun(id, newData, newMetadata)
2025-08-27 09:27:12 -07:00
await brain.updateVerb(verbId, newMetadata)
2025-08-26 12:32:21 -07:00
// Delete
await brain.deleteNoun(id)
2025-08-27 09:27:12 -07:00
await brain.deleteVerb(verbId)
2025-08-26 12:32:21 -07:00
// Bulk operations
await brain.import(arrayOfData)
2025-09-11 16:23:32 -07:00
const exported = await brain.export({format: 'json'})
2025-10-01 16:51:03 -07:00
// Import from CSV, Excel, PDF files (auto-detected)
await brain.import('customers.csv') // CSV with encoding detection
await brain.import('sales-report.xlsx') // Excel with multi-sheet support
await brain.import('research.pdf') // PDF with table extraction
2025-09-11 16:23:32 -07:00
```
## 🌐 Distributed System (NEW!)
### Zero-Config Distributed Setup
```javascript
// Single node (default)
2025-09-15 11:06:16 -07:00
const brain = new Brainy({
2025-09-11 16:23:32 -07:00
storage: {type: 's3', options: {bucket: 'my-data'}}
})
// Distributed cluster - just add one flag!
2025-09-15 11:06:16 -07:00
const brain = new Brainy({
2025-09-11 16:23:32 -07:00
storage: {type: 's3', options: {bucket: 'my-data'}},
distributed: true // That's it! Everything else is automatic
})
```
### How It Works
- **Storage-Based Discovery**: Nodes find each other via S3/GCS (no Consul/etcd!)
- **Automatic Sharding**: Data distributed by content hash
- **Smart Query Planning**: Queries routed to optimal shards
- **Live Rebalancing**: Handles node joins/leaves automatically
- **Zero Downtime**: Streaming shard migration
### Real-World Example: Social Media Firehose
```javascript
2025-09-22 16:19:27 -07:00
import { Brainy, NounType } from '@soulcraft/brainy '
2025-09-11 16:23:32 -07:00
// Ingestion nodes (optimized for writes)
2025-09-15 11:06:16 -07:00
const ingestionNode = new Brainy({
2025-09-11 16:23:32 -07:00
storage: {type: 's3', options: {bucket: 'social-data'}},
distributed: true,
writeOnly: true // Optimized for high-throughput writes
})
// Process Bluesky firehose
blueskyStream.on('post', async (post) => {
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
await ingestionNode.add(post, {
2025-09-22 16:19:27 -07:00
nounType: NounType.Message,
2025-09-11 16:23:32 -07:00
platform: 'bluesky',
author: post.author,
timestamp: post.createdAt
})
})
// Search nodes (optimized for queries)
2025-09-15 11:06:16 -07:00
const searchNode = new Brainy({
2025-09-11 16:23:32 -07:00
storage: {type: 's3', options: {bucket: 'social-data'}},
distributed: true,
readOnly: true // Optimized for fast queries
})
// Search across ALL data from ALL nodes
const trending = await searchNode.find('trending AI topics', {
where: {timestamp: {greaterThan: Date.now() - 3600000}},
limit: 100
})
2025-08-26 12:32:21 -07:00
```
2025-09-11 16:23:32 -07:00
### Benefits Over Traditional Systems
| Feature | Traditional (Pinecone, Weaviate) | Brainy Distributed |
|----------------|----------------------------------|-------------------------------|
| Setup | Complex (k8s, operators) | One flag: `distributed: true` |
| Coordination | External (etcd, Consul) | Built-in (via storage) |
| Minimum Nodes | 3-5 for HA | 1 (scale as needed) |
| Sharding | Random | Domain-aware |
| Query Planning | Basic | Triple Intelligence |
| Cost | High (always-on clusters) | Low (scale to zero) |
2025-08-26 12:32:21 -07:00
## 🎯 Use Cases
2025-08-27 09:27:12 -07:00
### Knowledge Management with Relationships
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
```javascript
2025-08-27 09:27:12 -07:00
// Store documentation with rich relationships
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
const apiGuide = await brain.add("REST API Guide", {
2025-09-22 16:19:27 -07:00
nounType: NounType.Document,
2025-09-11 16:23:32 -07:00
title: "API Guide",
category: "documentation",
version: "2.0"
2025-08-26 12:32:21 -07:00
})
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
const author = await brain.add("Jane Developer", {
2025-09-22 16:19:27 -07:00
nounType: NounType.Person,
2025-09-11 16:23:32 -07:00
role: "tech-lead"
2025-08-27 09:27:12 -07:00
})
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
const project = await brain.add("E-commerce Platform", {
2025-09-22 16:19:27 -07:00
nounType: NounType.Project,
2025-09-11 16:23:32 -07:00
status: "active"
2025-08-27 09:27:12 -07:00
})
// Create knowledge graph
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
await brain.relate(author, apiGuide, "authored", {
2025-09-11 16:23:32 -07:00
date: "2024-03-15"
2025-08-27 09:27:12 -07:00
})
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
await brain.relate(apiGuide, project, "documents", {
2025-09-11 16:23:32 -07:00
coverage: "complete"
2025-08-27 09:27:12 -07:00
})
// Query the knowledge graph naturally
const docs = await brain.find("documentation authored by tech leads for active projects")
2025-08-26 12:32:21 -07:00
```
### Semantic Search
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
```javascript
// Find similar content
const similar = await brain.search(existingContent, {
2025-09-11 16:23:32 -07:00
limit: 5,
threshold: 0.8
2025-08-26 12:32:21 -07:00
})
```
2025-08-27 09:27:12 -07:00
### AI Memory Layer with Context
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
```javascript
2025-08-27 09:27:12 -07:00
// Store conversation with relationships
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
const userId = await brain.add("User 123", {
2025-09-22 16:19:27 -07:00
nounType: NounType.User,
2025-09-11 16:23:32 -07:00
tier: "premium"
2025-08-27 09:27:12 -07:00
})
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
const messageId = await brain.add(userMessage, {
2025-09-22 16:19:27 -07:00
nounType: NounType.Message,
2025-09-11 16:23:32 -07:00
timestamp: Date.now(),
session: "abc"
2025-08-26 12:32:21 -07:00
})
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
const topicId = await brain.add("Product Support", {
2025-09-22 16:19:27 -07:00
nounType: NounType.Topic,
2025-09-11 16:23:32 -07:00
category: "support"
2025-08-27 09:27:12 -07:00
})
// Link conversation elements
docs: add deprecation warnings for addNoun and addVerb methods
- Add @deprecated JSDoc tags to TypeScript definitions
- Update all documentation examples to use modern add() and relate() API
- Preserve batch operations (addNouns, addVerbs) as they remain current
- Mark deprecated methods in both source and compiled definitions
Migration guide:
- addNoun(data, type, metadata) → add(data, { nounType: type, ...metadata })
- addVerb(source, target, type, metadata) → relate(source, target, type, metadata)
2025-09-17 10:48:41 -07:00
await brain.relate(userId, messageId, "sent")
await brain.relate(messageId, topicId, "about")
2025-08-27 09:27:12 -07:00
// Retrieve context with relationships
const context = await brain.find({
2025-09-11 16:23:32 -07:00
where: {type: "message"},
connected: {from: userId, type: "sent"},
like: "previous product issues"
2025-08-27 09:27:12 -07:00
})
2025-08-26 12:32:21 -07:00
```
## 💾 Storage Options
Brainy supports multiple storage backends:
```javascript
// Memory (default for testing)
2025-09-15 11:06:16 -07:00
const brain = new Brainy({
2025-09-11 16:23:32 -07:00
storage: {type: 'memory'}
2025-08-26 12:32:21 -07:00
})
// FileSystem (Node.js)
2025-09-15 11:06:16 -07:00
const brain = new Brainy({
2025-09-11 16:23:32 -07:00
storage: {
type: 'filesystem',
path: './data'
}
2025-08-26 12:32:21 -07:00
})
2025-09-15 14:53:59 -07:00
// Browser Storage (OPFS) - Works with frameworks
2025-09-15 11:06:16 -07:00
const brain = new Brainy({
2025-09-15 14:53:59 -07:00
storage: {type: 'opfs'} // Framework handles browser polyfills
2025-08-26 12:32:21 -07:00
})
// S3 Compatible (Production)
2025-09-15 11:06:16 -07:00
const brain = new Brainy({
2025-09-11 16:23:32 -07:00
storage: {
type: 's3',
bucket: 'my-bucket',
region: 'us-east-1'
}
2025-08-26 12:32:21 -07:00
})
```
## 🛠️ CLI
Brainy includes a powerful CLI for testing and management:
```bash
# Install globally
npm install -g brainy
# Add data
brainy add "JavaScript is awesome" --metadata '{"type":"opinion"}'
# Search
brainy search "programming"
# Natural language find
brainy find "awesome programming languages"
# Interactive mode
brainy chat
# Export data
brainy export --format json > backup.json
```
2025-08-28 13:59:59 -07:00
## 🧠 Neural API - Advanced AI Features
Brainy includes a powerful Neural API for advanced semantic analysis:
### Clustering & Analysis
2025-09-11 16:23:32 -07:00
2025-08-28 13:59:59 -07:00
```javascript
// Access via brain.neural
const neural = brain.neural
// Automatic semantic clustering
const clusters = await neural.clusters()
// Returns groups of semantically similar items
// Cluster with options
const clusters = await neural.clusters({
2025-09-11 16:23:32 -07:00
algorithm: 'kmeans', // or 'hierarchical', 'sample'
maxClusters: 5, // Maximum number of clusters
threshold: 0.8 // Similarity threshold
2025-08-28 13:59:59 -07:00
})
// Calculate similarity between any items
const similarity = await neural.similar('item1', 'item2')
// Returns 0-1 score
// Find nearest neighbors
const neighbors = await neural.neighbors('item-id', 10)
// Build semantic hierarchy
const hierarchy = await neural.hierarchy('item-id')
// Detect outliers
const outliers = await neural.outliers(0.3)
// Generate visualization data for D3/Cytoscape
const vizData = await neural.visualize({
2025-09-11 16:23:32 -07:00
maxNodes: 100,
dimensions: 3,
algorithm: 'force'
2025-08-28 13:59:59 -07:00
})
```
### Real-World Examples
2025-09-11 16:23:32 -07:00
2025-08-28 13:59:59 -07:00
```javascript
// Group customer feedback into themes
const feedbackClusters = await neural.clusters()
for (const cluster of feedbackClusters) {
2025-09-11 16:23:32 -07:00
console.log(`Theme: ${cluster.label}` )
console.log(`Items: ${cluster.members.length}` )
2025-08-28 13:59:59 -07:00
}
// Find related documents
2025-09-22 16:19:27 -07:00
const docId = await brain.add("Machine learning guide", { nounType: NounType.Document })
2025-08-28 13:59:59 -07:00
const similar = await neural.neighbors(docId, 5)
// Returns 5 most similar documents
// Detect anomalies in data
const anomalies = await neural.outliers(0.2)
console.log(`Found ${anomalies.length} outliers` )
```
2025-08-26 12:32:21 -07:00
## 🔌 Augmentations
Extend Brainy with powerful augmentations:
```bash
# List available augmentations
brainy augment list
# Install an augmentation
brainy augment install explorer
# Connect to Brain Cloud
brainy cloud setup
```
## 🏢 Enterprise Features - Included for Everyone
Brainy includes enterprise-grade capabilities at no extra cost. **No premium tiers, no paywalls.**
- **Scales to 10M+ items** with consistent 3ms search latency
- **Distributed architecture** with sharding and replication
- **Read/write separation** for horizontal scaling
- **Connection pooling** and request deduplication
- **Built-in monitoring** with metrics and health checks
- **Production ready** with circuit breakers and backpressure
2025-09-22 16:19:27 -07:00
📖 **More enterprise features coming soon** - Stay tuned!
2025-08-26 12:32:21 -07:00
## 📊 Benchmarks
2025-09-11 16:23:32 -07:00
| Operation | Performance | Memory |
|----------------------------------|-------------|----------|
| Initialize | 450ms | 24MB |
| Add Item | 12ms | +0.1MB |
| Vector Search (1k items) | 3ms | - |
| Metadata Filter (10k items) | 0.8ms | - |
| Natural Language Query | 15ms | - |
| Bulk Import (1000 items) | 2.3s | +8MB |
| **Production Scale (10M items)** | **5.8ms** | **12GB** |
2025-08-26 12:32:21 -07:00
2025-09-22 16:19:27 -07:00
## 🔄 Migration from Previous Versions
2025-08-26 12:32:21 -07:00
2025-09-22 16:19:27 -07:00
Key changes in the latest version:
2025-09-11 16:23:32 -07:00
2025-08-26 12:32:21 -07:00
- Search methods consolidated into `search()` and `find()`
- Result format now includes full objects with metadata
2025-09-22 16:19:27 -07:00
- Enhanced natural language capabilities
- Distributed architecture support
2025-08-26 12:32:21 -07:00
## 🤝 Contributing
We welcome contributions! See [CONTRIBUTING.md ](CONTRIBUTING.md ) for guidelines.
2025-09-25 10:47:44 -07:00
## 🧠 The Knowledge Operating System Explained
2025-08-27 09:27:12 -07:00
### How We Achieved The Impossible
**Triple Intelligence™** makes us the **world's first** to unify three database paradigms:
2025-09-11 16:23:32 -07:00
2025-08-27 09:27:12 -07:00
1. **Vector databases** (Pinecone, Weaviate) - semantic similarity
2025-09-11 16:23:32 -07:00
2. **Graph databases** (Neo4j, ArangoDB) - relationships
2025-08-27 09:27:12 -07:00
3. **Document databases** (MongoDB, Elasticsearch) - metadata filtering
**One API to rule them all.** Others make you choose. We unified them.
### The Math of Infinite Expressiveness
```
2025-09-22 16:19:27 -07:00
31 Nouns × 40 Verbs × ∞ Metadata × Triple Intelligence = Universal Protocol
2025-08-27 09:27:12 -07:00
```
2025-09-22 16:19:27 -07:00
- **1,240 base combinations** from standardized types
2025-08-27 09:27:12 -07:00
- **∞ domain specificity** via unlimited metadata
- **∞ relationship depth** via graph traversal
- **= Model ANYTHING**: From quantum physics to social networks
### Why This Changes Everything
**Like HTTP for the web, Brainy for knowledge:**
2025-09-11 16:23:32 -07:00
2025-08-27 09:27:12 -07:00
- All augmentations compose perfectly - same noun-verb language
- All AI models share knowledge - GPT, Claude, Llama all understand
- All tools integrate seamlessly - no translation layers
- All data flows freely - perfect portability
**The Vision**: One protocol. All knowledge. Every tool. Any AI.
**Proven across industries**: Healthcare, Finance, Manufacturing, Education, Legal, Retail, Government, and beyond.
[→ See the Mathematical Proof & Full Taxonomy ](docs/architecture/noun-verb-taxonomy.md )
2025-08-26 12:32:21 -07:00
## 📖 Documentation
2025-09-29 15:37:11 -07:00
### Infinite Agent Memory 💬
- [Conversation API Overview ](docs/conversation/README.md ) - **NEW!** Complete conversation management guide
- [MCP Integration for Claude Code ](docs/conversation/MCP_INTEGRATION.md ) - **NEW!** One-command setup
- [API Reference ](docs/conversation/API_REFERENCE.md ) - **NEW!** Full API documentation
2025-09-15 14:53:59 -07:00
### Framework Integration
2025-09-29 15:37:11 -07:00
- [Framework Integration Guide ](docs/guides/framework-integration.md ) - Complete framework setup guide
- [Next.js Integration ](docs/guides/nextjs-integration.md ) - React and Next.js examples
- [Vue.js Integration ](docs/guides/vue-integration.md ) - Vue and Nuxt examples
2025-09-15 14:53:59 -07:00
2025-09-29 13:51:47 -07:00
### Virtual Filesystem (Semantic VFS) 🧠📁
- [VFS Core Documentation ](docs/vfs/VFS_CORE.md ) - Complete filesystem architecture and API
- [Semantic VFS Guide ](docs/vfs/SEMANTIC_VFS.md ) - Multi-dimensional file access (6 semantic dimensions)
- [Neural Extraction API ](docs/vfs/NEURAL_EXTRACTION.md ) - AI-powered concept and entity extraction
- [Examples & Scenarios ](docs/vfs/VFS_EXAMPLES_SCENARIOS.md ) - Real-world use cases and code
2025-09-25 10:47:44 -07:00
- [VFS API Guide ](docs/vfs/VFS_API_GUIDE.md ) - Complete API reference
2025-09-15 14:53:59 -07:00
### Core Documentation
2025-08-26 12:32:21 -07:00
- [Getting Started Guide ](docs/guides/getting-started.md )
- [API Reference ](docs/api/README.md )
- [Architecture Overview ](docs/architecture/overview.md )
- [Natural Language Guide ](docs/guides/natural-language.md )
- [Triple Intelligence ](docs/architecture/triple-intelligence.md )
2025-08-27 09:27:12 -07:00
- [Noun-Verb Taxonomy ](docs/architecture/noun-verb-taxonomy.md )
2025-08-26 12:32:21 -07:00
## 🏢 Enterprise & Cloud
**Brain Cloud** - Managed Brainy with team sync, persistent memory, and enterprise connectors.
```bash
# Get started with free trial
brainy cloud setup
```
Visit [soulcraft.com ](https://soulcraft.com ) for more information.
## 📄 License
MIT © Brainy Contributors
---
< p align = "center" >
< strong > Built with ❤️ by the Brainy community< / strong > < br >
< em > Zero-Configuration AI Database with Triple Intelligence™< / em >
< / p >