[](LICENSE)
[](https://nodejs.org/)
[](https://www.typescriptlang.org/)
[](CONTRIBUTING.md)
[](https://www.npmjs.com/package/@soulcraft/brainy)
[//]: # ([](https://github.com/sodal-project/cartographer))
**A powerful graph & vector data platform for AI applications across any environment**
## ✨ Overview
Brainy combines the power of vector search with graph relationships in a lightweight, cross-platform database. Whether
you're building AI applications, recommendation systems, or knowledge graphs, Brainy provides the tools you need to
store, connect, and retrieve your data intelligently.
What makes Brainy special? It intelligently adapts to your environment! Brainy automatically detects your platform,
adjusts its storage strategy, and optimizes performance based on your usage patterns. The more you use it, the smarter
it gets - learning from your data to provide increasingly relevant results and connections.
### 🚀 Key Features
- **Run Everywhere** - Works in browsers, Node.js, serverless functions, and containers
- **Vector Search** - Find semantically similar content using embeddings
- **Graph Relationships** - Connect data with meaningful relationships
- **Streaming Pipeline** - Process data in real-time as it flows through the system
- **Extensible Augmentations** - Customize and extend functionality with pluggable components
- **Built-in Conduits** - Sync and scale across instances with WebSocket and WebRTC
- **TensorFlow Integration** - Use TensorFlow.js for high-quality embeddings
- **Adaptive Intelligence** - Automatically optimizes for your environment and usage patterns
- **Persistent Storage** - Data persists across sessions and scales to any size
- **TypeScript Support** - Fully typed API with generics
- **CLI Tools & Web Service** - Command-line interface and REST API web service for data management
- **Model Control Protocol (MCP)** - Allow external AI models to access Brainy data and use augmentation pipeline as
tools
## 🚀 Live Demo
**[Try the live demo](https://soulcraft-research.github.io/brainy/demo/index.html)** - Check out the interactive demo on
GitHub Pages that showcases Brainy's main features.
## 📊 What Can You Build?
- **Semantic Search Engines** - Find content based on meaning, not just keywords
- **Recommendation Systems** - Suggest similar items based on vector similarity
- **Knowledge Graphs** - Build connected data structures with relationships
- **AI Applications** - Store and retrieve embeddings for machine learning models
- **AI-Enhanced Applications** - Build applications that leverage vector embeddings for intelligent data processing
- **Data Organization Tools** - Automatically categorize and connect related information
- **Adaptive Experiences** - Create applications that learn and evolve with your users
- **Model-Integrated Systems** - Connect external AI models to Brainy data and tools using MCP
## 🔧 Installation
```bash
npm install @soulcraft/brainy
```
TensorFlow.js packages are included as bundled dependencies and will be automatically installed without any additional configuration.
### Additional Packages
Brainy offers specialized packages for different use cases:
#### CLI Package
```bash
npm install -g @soulcraft/brainy-cli
```
Command-line interface for data management, bulk operations, and database administration.
#### Web Service Package
```bash
npm install @soulcraft/brainy-web-service
```
REST API web service wrapper that provides HTTP endpoints for search operations and database queries.
## 🏁 Quick Start
Brainy uses a unified build that automatically adapts to your environment (Node.js, browser, or serverless):
```typescript
import { BrainyData, NounType, VerbType } from '@soulcraft/brainy'
// Create and initialize the database
const db = new BrainyData()
await db.init()
// Add data (automatically converted to vectors)
const catId = await db.add("Cats are independent pets", {
noun: NounType.Thing,
category: 'animal'
})
const dogId = await db.add("Dogs are loyal companions", {
noun: NounType.Thing,
category: 'animal'
})
// Search for similar items
const results = await db.searchText("feline pets", 2)
console.log(results)
// Add a relationship between items
await db.addVerb(catId, dogId, {
verb: VerbType.RelatedTo,
description: 'Both are common household pets'
})
```
### Import Options
```typescript
// Standard import - automatically adapts to any environment
import { BrainyData } from '@soulcraft/brainy'
// Minified version for production
import { BrainyData } from '@soulcraft/brainy/min'
```
> **Note**: The CLI functionality is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size of the main package. Install it globally with `npm install -g @soulcraft/brainy-cli` to use the command-line interface.
### Browser Usage
```html
```
Modern bundlers like Webpack, Rollup, and Vite will automatically use the unified build which adapts to any environment.
## 🧩 How It Works
Brainy combines four key technologies to create its adaptive intelligence:
1. **Vector Embeddings** - Converts data (text, images, etc.) into numerical vectors that capture semantic meaning
2. **HNSW Algorithm** - Enables fast similarity search through a hierarchical graph structure
3. **Adaptive Environment Detection** - Automatically senses your platform and optimizes accordingly:
- Detects browser, Node.js, and serverless environments
- Adjusts performance parameters based on available resources
- Learns from query patterns to optimize future searches
- Tunes itself for your specific use cases
4. **Intelligent Storage Selection** - Uses the best available storage option for your environment:
- Browser: Origin Private File System (OPFS)
- Node.js: File system
- Server: S3-compatible storage (optional)
- Serverless: In-memory storage with optional cloud persistence
- Fallback: In-memory storage
- Automatically migrates between storage types as needed
## 🚀 The Brainy Pipeline
Brainy's data processing pipeline transforms raw data into searchable, connected knowledge that gets smarter over time:
```
Raw Data → Embedding → Vector Storage → Graph Connections → Adaptive Learning → Query & Retrieval
```
Each time data flows through this pipeline, Brainy learns more about your usage patterns and environment, making future
operations faster and more relevant.
### Pipeline Stages
1. **Data Ingestion**
- Raw text or pre-computed vectors enter the pipeline
- Data is validated and prepared for processing
2. **Embedding Generation**
- Text is transformed into numerical vectors using embedding models
- Uses TensorFlow Universal Sentence Encoder for high-quality text embeddings
- Custom embedding functions can be plugged in for specialized domains
3. **Vector Indexing**
- Vectors are indexed using the HNSW algorithm
- Hierarchical structure enables fast similarity search
- Configurable parameters for precision vs. performance tradeoffs
4. **Graph Construction**
- Nouns (entities) become nodes in the knowledge graph
- Verbs (relationships) connect related entities
- Typed relationships add semantic meaning to connections
5. **Adaptive Learning**
- Analyzes usage patterns to optimize future operations
- Tunes performance parameters based on your environment
- Adjusts search strategies based on query history
- Becomes more efficient and relevant the more you use it
6. **Intelligent Storage**
- Data is saved using the optimal storage for your environment
- Automatic selection between OPFS, filesystem, S3, or memory
- Migrates between storage types as your application's needs evolve
- Scales from tiny datasets to massive data collections
- Configurable storage adapters for custom persistence needs
### Augmentation Types
Brainy uses a powerful augmentation system to extend functionality. Augmentations are processed in the following order:
1. **SENSE**
- Ingests and processes raw, unstructured data into nouns and verbs
- Handles text, images, audio streams, and other input formats
- Example: Converting raw text into structured entities
2. **MEMORY**
- Provides storage capabilities for data in different formats
- Manages persistence across sessions
- Example: Storing vectors in OPFS or filesystem
3. **COGNITION**
- Enables advanced reasoning, inference, and logical operations
- Analyzes relationships between entities
- Examples:
- Inferring new connections between existing data
- Deriving insights from graph relationships
4. **CONDUIT**
- Establishes channels for structured data exchange
- Connects with external systems and syncs between Brainy instances
- Two built-in iConduit augmentations for scaling out and syncing:
- **WebSocket iConduit** - Syncs data between browsers and servers
- **WebRTC iConduit** - Direct peer-to-peer syncing between browsers
- Examples:
- Integrating with third-party APIs
- Syncing Brainy instances between browsers using WebSockets
- Peer-to-peer syncing between browsers using WebRTC
5. **ACTIVATION**
- Initiates actions, responses, or data manipulations
- Triggers events based on data changes
- Example: Sending notifications when new data is processed
6. **PERCEPTION**
- Interprets, contextualizes, and visualizes identified nouns and verbs
- Creates meaningful representations of data
- Example: Generating visualizations of graph relationships
7. **DIALOG**
- Facilitates natural language understanding and generation
- Enables conversational interactions
- Example: Processing user queries and generating responses
8. **WEBSOCKET**
- Enables real-time communication via WebSockets
- Can be combined with other augmentation types
- Example: Streaming data processing in real-time
### Streaming Data Support
Brainy's pipeline is designed to handle streaming data efficiently:
1. **WebSocket Integration**
- Built-in support for WebSocket connections
- Process data as it arrives without blocking
- Example: `setupWebSocketPipeline(url, dataType, options)`
2. **Asynchronous Processing**
- Non-blocking architecture for real-time data handling
- Parallel processing of incoming streams
- Example: `createWebSocketHandler(connection, dataType, options)`
3. **Event-Based Architecture**
- Augmentations can listen to data feeds and streams
- Real-time updates propagate through the pipeline
- Example: `listenToFeed(feedUrl, callback)`
4. **Threaded Execution**
- Comprehensive multi-threading for high-performance operations
- Parallel processing for batch operations, vector calculations, and embedding generation
- Configurable execution modes (SEQUENTIAL, PARALLEL, THREADED)
- Automatic thread management based on environment capabilities
- Example: `executeTypedPipeline(augmentations, method, args, { mode: ExecutionMode.THREADED })`
### Running the Pipeline
The pipeline runs automatically when you:
```typescript
// Add data (runs embedding → indexing → storage)
const id = await db.add("Your text data here", { metadata })
// Search (runs embedding → similarity search)
const results = await db.searchText("Your query here", 5)
// Connect entities (runs graph construction → storage)
await db.addVerb(sourceId, targetId, { verb: VerbType.RelatedTo })
```
Using the CLI:
```bash
# Add data through the CLI pipeline
brainy add "Your text data here" '{"noun":"Thing"}'
# Search through the CLI pipeline
brainy search "Your query here" --limit 5
# Connect entities through the CLI
brainy addVerb