[](LICENSE)
[](https://nodejs.org/)
[](https://www.typescriptlang.org/)
[](CONTRIBUTING.md)
[//]: # ([](https://github.com/sodal-project/cartographer))
**A powerful graph & vector data platform for AI applications across any environment**
## ✨ Overview
Brainy combines the power of vector search with graph relationships in a lightweight, cross-platform database. Whether
you're building AI applications, recommendation systems, or knowledge graphs, Brainy provides the tools you need to
store, connect, and retrieve your data intelligently.
What makes Brainy special? It intelligently adapts to your environment! Brainy automatically detects your platform,
adjusts its storage strategy, and optimizes performance based on your usage patterns. The more you use it, the smarter
it gets - learning from your data to provide increasingly relevant results and connections.
### 🚀 Key Features
- **🧠 Zero Configuration** - Auto-detects environment and optimizes automatically
- **⚡ Production-Scale Performance** - Handles millions of vectors with sub-second search
- **🎯 Intelligent Partitioning** - Semantic clustering with auto-tuning
- **📊 Adaptive Learning** - Gets smarter with usage, optimizes itself over time
- **🗄️ Smart Storage** - OPFS, FileSystem, S3 auto-selection based on environment
- **💾 Massive Memory Optimization** - 75% reduction with compression, intelligent caching
- **🚀 Distributed Search** - Parallel processing with load balancing
- **🔄 Real-Time Adaptation** - Automatically adjusts to your data patterns
- **Run Everywhere** - Works in browsers, Node.js, serverless functions, and containers
- **Vector Search** - Find semantically similar content using embeddings
- **Advanced JSON Document Search** - Search within specific fields of JSON documents with field prioritization and
service-based field standardization
- **Graph Relationships** - Connect data with meaningful relationships
- **Streaming Pipeline** - Process data in real-time as it flows through the system
- **Extensible Augmentations** - Customize and extend functionality with pluggable components
- **Built-in Conduits** - Sync and scale across instances with WebSocket and WebRTC
- **TensorFlow Integration** - Use TensorFlow.js for high-quality embeddings
- **Persistent Storage** - Data persists across sessions and scales to any size
- **TypeScript Support** - Fully typed API with generics
- **CLI Tools & Web Service** - Command-line interface and REST API web service for data management
- **Model Control Protocol (MCP)** - Allow external AI models to access Brainy data and use augmentation pipeline as
tools
## ⚡ Large-Scale Performance Optimizations
**New in v0.36.0**: Brainy now includes 6 core optimizations that transform it from a prototype into a production-ready system capable of handling millions of vectors:
### 🎯 Performance Benchmarks
| Dataset Size | Search Time | Memory Usage | API Calls Reduction |
|-------------|-------------|--------------|-------------------|
| **10k vectors** | ~50ms | Standard | N/A |
| **100k vectors** | ~200ms | 30% reduction | 50-70% fewer |
| **1M+ vectors** | ~500ms | 75% reduction | 50-90% fewer |
### 🧠 6 Core Optimization Systems
1. **🎛️ Auto-Configuration System** - Detects environment, resources, and data patterns
2. **🔀 Semantic Partitioning** - Intelligent clustering with auto-tuning (4-32 clusters)
3. **🚀 Distributed Search** - Parallel processing across partitions with load balancing
4. **🧠 Multi-Level Caching** - Hot/Warm/Cold caching with predictive prefetching
5. **📦 Batch S3 Operations** - Reduces cloud storage API calls by 50-90%
6. **💾 Advanced Compression** - Vector quantization and memory-mapping for large datasets
### 🎯 Automatic Environment Detection
| Environment | Auto-Configured | Performance Focus |
|-------------|-----------------|-------------------|
| **Browser** | OPFS + Web Workers | Memory efficiency, 512MB-1GB limits |
| **Node.js** | FileSystem + Worker Threads | High performance, 4GB-8GB+ usage |
| **Serverless** | S3 + Memory cache | Cold start optimization, latency focus |
### 📊 Intelligent Scaling Strategy
The system automatically adapts based on your dataset size:
- **< 25k vectors**: Single optimized index, no partitioning needed
- **25k - 100k**: Semantic clustering (4-8 clusters), balanced performance
- **100k - 1M**: Advanced partitioning (8-16 clusters), scale-optimized
- **1M+ vectors**: Maximum optimization (16-32 clusters), enterprise-grade
### 🧠 Adaptive Learning Features
- **Performance Monitoring**: Tracks latency, cache hits, memory usage
- **Dynamic Tuning**: Adjusts parameters every 50 searches based on performance
- **Pattern Recognition**: Learns from access patterns to improve predictions
- **Self-Optimization**: Automatically enables/disables features based on workload
> **📖 Full Documentation**: See the complete [Large-Scale Optimizations Guide](docs/large-scale-optimizations.md) for detailed configuration options and advanced usage.
## 🚀 Live Demo
**[Try the live demo](https://soulcraft-research.github.io/brainy/demo/index.html)** - Check out the interactive demo on
GitHub Pages that showcases Brainy's main features.
## 📊 What Can You Build?
- **Semantic Search Engines** - Find content based on meaning, not just keywords
- **Recommendation Systems** - Suggest similar items based on vector similarity
- **Knowledge Graphs** - Build connected data structures with relationships
- **AI Applications** - Store and retrieve embeddings for machine learning models
- **AI-Enhanced Applications** - Build applications that leverage vector embeddings for intelligent data processing
- **Data Organization Tools** - Automatically categorize and connect related information
- **Adaptive Experiences** - Create applications that learn and evolve with your users
- **Model-Integrated Systems** - Connect external AI models to Brainy data and tools using MCP
## 🔧 Installation
```bash
npm install @soulcraft/brainy
```
TensorFlow.js packages are included as bundled dependencies and will be automatically installed without any additional
configuration.
### Additional Packages
Brainy offers specialized packages for different use cases:
#### CLI Package
```bash
npm install -g @soulcraft/brainy-cli
```
Command-line interface for data management, bulk operations, and database administration.
#### Web Service Package
```bash
npm install @soulcraft/brainy-web-service
```
REST API web service wrapper that provides HTTP endpoints for search operations and database queries.
## 🚀 Quick Setup - Zero Configuration!
**New in v0.36.0**: Brainy now automatically detects your environment and optimizes itself! Choose your scenario:
### ✨ Instant Setup (Auto-Everything)
```typescript
import { createAutoBrainy } from '@soulcraft/brainy'
// That's it! Everything is auto-configured
const brainy = createAutoBrainy()
// Add data and search - all optimizations enabled automatically
await brainy.addVector({ id: '1', vector: [0.1, 0.2, 0.3], text: 'Hello world' })
const results = await brainy.search([0.1, 0.2, 0.3], 10)
```
### 📦 With S3 Storage (Still Auto-Configured)
```typescript
import { createAutoBrainy } from '@soulcraft/brainy'
// Auto-detects AWS credentials from environment variables
const brainy = createAutoBrainy({
bucketName: 'my-vector-storage'
// region: 'us-east-1' (default)
// AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from env
})
```
### 🎯 Scenario-Based Setup
```typescript
import { createQuickBrainy } from '@soulcraft/brainy'
// Choose your scale: 'small', 'medium', 'large', 'enterprise'
const brainy = await createQuickBrainy('large', {
bucketName: 'my-big-vector-db'
})
```
| Scenario | Dataset Size | Memory Usage | S3 Required | Best For |
|----------|-------------|--------------|-------------|----------|
| `small` | ≤10k vectors | ≤1GB | No | Development, testing |
| `medium` | ≤100k vectors | ≤4GB | Serverless only | Production apps |
| `large` | ≤1M vectors | ≤8GB | Yes | Large applications |
| `enterprise` | ≤10M vectors | ≤32GB | Yes | Enterprise systems |
### 🧠 What Auto-Configuration Does
- **🎯 Environment Detection**: Browser, Node.js, or Serverless
- **💾 Smart Memory Management**: Uses available RAM optimally
- **🗄️ Storage Selection**: OPFS, FileSystem, S3, or Memory
- **⚡ Performance Tuning**: Threading, caching, compression
- **📊 Adaptive Learning**: Improves performance over time
- **🔍 Semantic Partitioning**: Auto-clusters similar vectors
## 🏁 Traditional Setup (Manual Configuration)
If you prefer manual control:
```typescript
import { BrainyData, NounType, VerbType } from '@soulcraft/brainy'
// Create and initialize the database
const db = new BrainyData()
await db.init()
// Add data (automatically converted to vectors)
const catId = await db.add("Cats are independent pets", {
noun: NounType.Thing,
category: 'animal'
})
const dogId = await db.add("Dogs are loyal companions", {
noun: NounType.Thing,
category: 'animal'
})
// Search for similar items
const results = await db.searchText("feline pets", 2)
console.log(results)
// Add a relationship between items
await db.addVerb(catId, dogId, {
verb: VerbType.RelatedTo,
description: 'Both are common household pets'
})
```
### Import Options
```typescript
// Standard import - automatically adapts to any environment
import { BrainyData } from '@soulcraft/brainy'
// Minified version for production
import { BrainyData } from '@soulcraft/brainy/min'
```
> **Note**: The CLI functionality is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size
> of the main package. Install it globally with `npm install -g @soulcraft/brainy-cli` to use the command-line
> interface.
### Browser Usage
```html
```
Modern bundlers like Webpack, Rollup, and Vite will automatically use the unified build which adapts to any environment.
## 🧩 How It Works
Brainy combines **six advanced optimization systems** with core vector database technologies to create a production-ready, self-optimizing system:
### 🔧 Core Technologies
1. **Vector Embeddings** - Converts data (text, images, etc.) into numerical vectors using TensorFlow.js
2. **Optimized HNSW Algorithm** - Fast similarity search with semantic partitioning and distributed processing
3. **🧠 Auto-Configuration Engine** - Detects environment, resources, and data patterns to optimize automatically
4. **🎯 Intelligent Storage System** - Multi-level caching with predictive prefetching and batch operations
### ⚡ Advanced Optimization Layer
5. **Semantic Partitioning** - Auto-clusters similar vectors for faster search (4-32 clusters based on scale)
6. **Distributed Search** - Parallel processing across partitions with intelligent load balancing
7. **Multi-Level Caching** - Hot (RAM) → Warm (Fast Storage) → Cold (S3/Disk) with 70-90% hit rates
8. **Batch Operations** - Reduces S3 API calls by 50-90% through intelligent batching
9. **Adaptive Learning** - Continuously learns from usage patterns and optimizes performance
10. **Advanced Compression** - Vector quantization achieves 75% memory reduction for large datasets
### 🎯 Environment-Specific Optimizations
| Environment | Storage | Threading | Memory | Focus |
|-------------|---------|-----------|---------|-------|
| **Browser** | OPFS + Cache | Web Workers | 512MB-1GB | Responsiveness |
| **Node.js** | FileSystem + S3 | Worker Threads | 4GB-8GB+ | Throughput |
| **Serverless** | S3 + Memory | Limited | 1GB-2GB | Cold Start Speed |
### 🔄 Adaptive Intelligence Flow
```
Data Input → Auto-Detection → Environment Optimization → Semantic Partitioning →
Distributed Search → Multi-Level Caching → Performance Learning → Self-Tuning
```
The system **automatically adapts** to your environment, learns from your usage patterns, and **continuously optimizes itself** for better performance over time.
## 🚀 The Brainy Pipeline
Brainy's data processing pipeline transforms raw data into searchable, connected knowledge that gets smarter over time:
```
Raw Data → Embedding → Vector Storage → Graph Connections → Adaptive Learning → Query & Retrieval
```
Each time data flows through this pipeline, Brainy learns more about your usage patterns and environment, making future
operations faster and more relevant.
### Pipeline Stages
1. **Data Ingestion**
- Raw text or pre-computed vectors enter the pipeline
- Data is validated and prepared for processing
2. **Embedding Generation**
- Text is transformed into numerical vectors using embedding models
- Uses TensorFlow Universal Sentence Encoder for high-quality text embeddings
- Custom embedding functions can be plugged in for specialized domains
3. **Vector Indexing**
- Vectors are indexed using the HNSW algorithm
- Hierarchical structure enables fast similarity search
- Configurable parameters for precision vs. performance tradeoffs
4. **Graph Construction**
- Nouns (entities) become nodes in the knowledge graph
- Verbs (relationships) connect related entities
- Typed relationships add semantic meaning to connections
5. **Adaptive Learning**
- Analyzes usage patterns to optimize future operations
- Tunes performance parameters based on your environment
- Adjusts search strategies based on query history
- Becomes more efficient and relevant the more you use it
6. **Intelligent Storage**
- Data is saved using the optimal storage for your environment
- Automatic selection between OPFS, filesystem, S3, or memory
- Migrates between storage types as your application's needs evolve
- Scales from tiny datasets to massive data collections
- Configurable storage adapters for custom persistence needs
### Augmentation Types
Brainy uses a powerful augmentation system to extend functionality. Augmentations are processed in the following order:
1. **SENSE**
- Ingests and processes raw, unstructured data into nouns and verbs
- Handles text, images, audio streams, and other input formats
- Example: Converting raw text into structured entities
2. **MEMORY**
- Provides storage capabilities for data in different formats
- Manages persistence across sessions
- Example: Storing vectors in OPFS or filesystem
3. **COGNITION**
- Enables advanced reasoning, inference, and logical operations
- Analyzes relationships between entities
- Examples:
- Inferring new connections between existing data
- Deriving insights from graph relationships
4. **CONDUIT**
- Establishes channels for structured data exchange
- Connects with external systems and syncs between Brainy instances
- Two built-in iConduit augmentations for scaling out and syncing:
- **WebSocket iConduit** - Syncs data between browsers and servers
- **WebRTC iConduit** - Direct peer-to-peer syncing between browsers
- Examples:
- Integrating with third-party APIs
- Syncing Brainy instances between browsers using WebSockets
- Peer-to-peer syncing between browsers using WebRTC
5. **ACTIVATION**
- Initiates actions, responses, or data manipulations
- Triggers events based on data changes
- Example: Sending notifications when new data is processed
6. **PERCEPTION**
- Interprets, contextualizes, and visualizes identified nouns and verbs
- Creates meaningful representations of data
- Example: Generating visualizations of graph relationships
7. **DIALOG**
- Facilitates natural language understanding and generation
- Enables conversational interactions
- Example: Processing user queries and generating responses
8. **WEBSOCKET**
- Enables real-time communication via WebSockets
- Can be combined with other augmentation types
- Example: Streaming data processing in real-time
### Streaming Data Support
Brainy's pipeline is designed to handle streaming data efficiently:
1. **WebSocket Integration**
- Built-in support for WebSocket connections
- Process data as it arrives without blocking
- Example: `setupWebSocketPipeline(url, dataType, options)`
2. **Asynchronous Processing**
- Non-blocking architecture for real-time data handling
- Parallel processing of incoming streams
- Example: `createWebSocketHandler(connection, dataType, options)`
3. **Event-Based Architecture**
- Augmentations can listen to data feeds and streams
- Real-time updates propagate through the pipeline
- Example: `listenToFeed(feedUrl, callback)`
4. **Threaded Execution**
- Comprehensive multi-threading for high-performance operations
- Parallel processing for batch operations, vector calculations, and embedding generation
- Configurable execution modes (SEQUENTIAL, PARALLEL, THREADED)
- Automatic thread management based on environment capabilities
- Example: `executeTypedPipeline(augmentations, method, args, { mode: ExecutionMode.THREADED })`
### Running the Pipeline
The pipeline runs automatically when you:
```typescript
// Add data (runs embedding → indexing → storage)
const id = await db.add("Your text data here", { metadata })
// Search (runs embedding → similarity search)
const results = await db.searchText("Your query here", 5)
// Connect entities (runs graph construction → storage)
await db.addVerb(sourceId, targetId, { verb: VerbType.RelatedTo })
```
Using the CLI:
```bash
# Add data through the CLI pipeline
brainy add "Your text data here" '{"noun":"Thing"}'
# Search through the CLI pipeline
brainy search "Your query here" --limit 5
# Connect entities through the CLI
brainy addVerb