From ab0ee4d4add153151178a84695dc17f9116d92df Mon Sep 17 00:00:00 2001 From: David Snelling Date: Mon, 4 Aug 2025 08:56:05 -0700 Subject: [PATCH] Reverting --- README.md | 1888 +++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 1776 insertions(+), 112 deletions(-) diff --git a/README.md b/README.md index 5361267f..f7736fec 100644 --- a/README.md +++ b/README.md @@ -7,43 +7,147 @@ [![TypeScript](https://img.shields.io/badge/TypeScript-5.4.5-blue.svg)](https://www.typescriptlang.org/) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md) -**A lightning-fast vector database that runs everywhere - from browsers to servers to edge functions** +[//]: # ([![Cartographer](https://img.shields.io/badge/Cartographer-Official%20Standard-brightgreen)](https://github.com/sodal-project/cartographer)) -[Live Demo](https://soulcraft-research.github.io/brainy/demo/index.html) | [Quick Start](#-quick-start) | [Documentation](docs/) | [Examples](examples/) +**A powerful graph & vector data platform for AI applications across any environment** -## ✨ Why Brainy? +## ✨ Overview -Brainy is a production-ready vector database that **automatically adapts** to your environment. Whether you're building AI applications, recommendation systems, or knowledge graphs, Brainy provides blazing-fast semantic search with zero configuration. +Brainy combines the power of vector search with graph relationships in a lightweight, cross-platform database. Whether +you're building AI applications, recommendation systems, or knowledge graphs, Brainy provides the tools you need to +store, connect, and retrieve your data intelligently. + +What makes Brainy special? It intelligently adapts to your environment! Brainy automatically detects your platform, +adjusts its storage strategy, and optimizes performance based on your usage patterns. The more you use it, the smarter +it gets - learning from your data to provide increasingly relevant results and connections. ### 🚀 Key Features -- **⚡ Lightning Fast** - Sub-second search across millions of vectors -- **🌍 Runs Everywhere** - Browser, Node.js, Deno, Bun, Edge Workers, Containers - **🧠 Zero Configuration** - Auto-detects environment and optimizes automatically -- **📈 Self-Optimizing** - Gets faster the more you use it through adaptive learning -- **🔄 Graph Relationships** - Connect data with semantic relationships, not just vectors -- **💾 75% Memory Reduction** - Advanced compression and intelligent caching -- **🎯 Production Ready** - Battle-tested with comprehensive optimizations +- **⚡ Production-Scale Performance** - Handles millions of vectors with sub-second search +- **🎯 Intelligent Partitioning** - Semantic clustering with auto-tuning +- **📊 Adaptive Learning** - Gets smarter with usage, optimizes itself over time +- **🗄️ Smart Storage** - OPFS, FileSystem, S3 auto-selection based on environment +- **💾 Massive Memory Optimization** - 75% reduction with compression, intelligent caching +- **🚀 Distributed Search** - Parallel processing with load balancing +- **🔄 Real-Time Adaptation** - Automatically adjusts to your data patterns +- **Run Everywhere** - Works in browsers, Node.js, serverless functions, and containers +- **Vector Search** - Find semantically similar content using embeddings +- **Advanced JSON Document Search** - Search within specific fields of JSON documents with field prioritization and + service-based field standardization +- **Graph Relationships** - Connect data with meaningful relationships +- **Streaming Pipeline** - Process data in real-time as it flows through the system +- **Extensible Augmentations** - Customize and extend functionality with pluggable components +- **Built-in Conduits** - Sync and scale across instances with WebSocket and WebRTC +- **TensorFlow Integration** - Use TensorFlow.js for high-quality embeddings +- **Persistent Storage** - Data persists across sessions and scales to any size +- **TypeScript Support** - Fully typed API with generics +- **CLI Tools & Web Service** - Command-line interface and REST API web service for data management +- **Model Control Protocol (MCP)** - Allow external AI models to access Brainy data and use augmentation pipeline as + tools -### 📊 Performance at Scale +## ⚡ Large-Scale Performance Optimizations -| Dataset Size | Search Time | Memory Usage | -|-------------|-------------|--------------| -| 10k vectors | ~50ms | Standard | -| 100k vectors | ~200ms | 30% reduction | -| 1M+ vectors | ~500ms | 75% reduction | +**New in v0.36.0**: Brainy now includes 6 core optimizations that transform it from a prototype into a production-ready +system capable of handling millions of vectors: -## 🎯 Quick Start +### 🎯 Performance Benchmarks -### Installation +| Dataset Size | Search Time | Memory Usage | API Calls Reduction | +|------------------|-------------|---------------|---------------------| +| **10k vectors** | ~50ms | Standard | N/A | +| **100k vectors** | ~200ms | 30% reduction | 50-70% fewer | +| **1M+ vectors** | ~500ms | 75% reduction | 50-90% fewer | + +### 🧠 6 Core Optimization Systems + +1. **🎛️ Auto-Configuration System** - Detects environment, resources, and data patterns +2. **🔀 Semantic Partitioning** - Intelligent clustering with auto-tuning (4-32 clusters) +3. **🚀 Distributed Search** - Parallel processing across partitions with load balancing +4. **🧠 Multi-Level Caching** - Hot/Warm/Cold caching with predictive prefetching +5. **📦 Batch S3 Operations** - Reduces cloud storage API calls by 50-90% +6. **💾 Advanced Compression** - Vector quantization and memory-mapping for large datasets + +### 🎯 Automatic Environment Detection + +| Environment | Auto-Configured | Performance Focus | +|----------------|-----------------------------|----------------------------------------| +| **Browser** | OPFS + Web Workers | Memory efficiency, 512MB-1GB limits | +| **Node.js** | FileSystem + Worker Threads | High performance, 4GB-8GB+ usage | +| **Serverless** | S3 + Memory cache | Cold start optimization, latency focus | + +### 📊 Intelligent Scaling Strategy + +The system automatically adapts based on your dataset size: + +- **< 25k vectors**: Single optimized index, no partitioning needed +- **25k - 100k**: Semantic clustering (4-8 clusters), balanced performance +- **100k - 1M**: Advanced partitioning (8-16 clusters), scale-optimized +- **1M+ vectors**: Maximum optimization (16-32 clusters), enterprise-grade + +### 🧠 Adaptive Learning Features + +- **Performance Monitoring**: Tracks latency, cache hits, memory usage +- **Dynamic Tuning**: Adjusts parameters every 50 searches based on performance +- **Pattern Recognition**: Learns from access patterns to improve predictions +- **Self-Optimization**: Automatically enables/disables features based on workload + +> **📖 Full Documentation**: See the +> complete [Large-Scale Optimizations Guide](docs/optimization-guides/large-scale-optimizations.md) for detailed +> configuration options and advanced usage. + +## 🚀 Live Demo + +**[Try the live demo](https://soulcraft-research.github.io/brainy/demo/index.html)** - Check out the interactive demo on +GitHub Pages that showcases Brainy's main features. + +## 📊 What Can You Build? + +- **Semantic Search Engines** - Find content based on meaning, not just keywords +- **Recommendation Systems** - Suggest similar items based on vector similarity +- **Knowledge Graphs** - Build connected data structures with relationships +- **AI Applications** - Store and retrieve embeddings for machine learning models +- **AI-Enhanced Applications** - Build applications that leverage vector embeddings for intelligent data processing +- **Data Organization Tools** - Automatically categorize and connect related information +- **Adaptive Experiences** - Create applications that learn and evolve with your users +- **Model-Integrated Systems** - Connect external AI models to Brainy data and tools using MCP + +## 🔧 Installation ```bash npm install @soulcraft/brainy ``` -### Zero Configuration Setup +TensorFlow.js packages are included as bundled dependencies and will be automatically installed without any additional +configuration. + +### Additional Packages + +Brainy offers specialized packages for different use cases: + +#### CLI Package + +```bash +npm install -g @soulcraft/brainy-cli +``` + +Command-line interface for data management, bulk operations, and database administration. + +#### Web Service Package + +```bash +npm install @soulcraft/brainy-web-service +``` + +REST API web service wrapper that provides HTTP endpoints for search operations and database queries. + +## 🚀 Quick Setup - Zero Configuration! + +**New in v0.36.0**: Brainy now automatically detects your environment and optimizes itself! Choose your scenario: + +### ✨ Instant Setup (Auto-Everything) ```typescript import { createAutoBrainy } from '@soulcraft/brainy' @@ -51,43 +155,63 @@ import { createAutoBrainy } from '@soulcraft/brainy' // That's it! Everything is auto-configured const brainy = createAutoBrainy() -// Add data (automatically converted to vectors) -await brainy.add("Cats are independent pets", { - category: 'animal', - type: 'domestic' -}) - -// Search semantically -const results = await brainy.searchText("feline companions", 5) -console.log(results) // Returns similar items with scores +// Add data and search - all optimizations enabled automatically +await brainy.addVector({ id: '1', vector: [0.1, 0.2, 0.3], text: 'Hello world' }) +const results = await brainy.search([0.1, 0.2, 0.3], 10) ``` -### Choose Your Scale +### 📦 With S3 Storage (Still Auto-Configured) + +```typescript +import { createAutoBrainy } from '@soulcraft/brainy' + +// Auto-detects AWS credentials from environment variables +const brainy = createAutoBrainy({ + bucketName: 'my-vector-storage' + // region: 'us-east-1' (default) + // AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from env +}) +``` + +### 🎯 Scenario-Based Setup ```typescript import { createQuickBrainy } from '@soulcraft/brainy' -// Pick your scale: 'small', 'medium', 'large', 'enterprise' +// Choose your scale: 'small', 'medium', 'large', 'enterprise' const brainy = await createQuickBrainy('large', { - bucketName: 'my-vector-storage' // Optional S3 storage + bucketName: 'my-big-vector-db' }) ``` -| Scale | Dataset Size | Best For | -|-------|-------------|----------| -| `small` | ≤10k vectors | Development, prototypes | -| `medium` | ≤100k vectors | Production apps | -| `large` | ≤1M vectors | Large applications | -| `enterprise` | ≤10M vectors | Enterprise systems | +| Scenario | Dataset Size | Memory Usage | S3 Required | Best For | +|--------------|---------------|--------------|-----------------|----------------------| +| `small` | ≤10k vectors | ≤1GB | No | Development, testing | +| `medium` | ≤100k vectors | ≤4GB | Serverless only | Production apps | +| `large` | ≤1M vectors | ≤8GB | Yes | Large applications | +| `enterprise` | ≤10M vectors | ≤32GB | Yes | Enterprise systems | -## 🔥 Core Concepts +### 🧠 What Auto-Configuration Does -### Vector + Graph = Knowledge +- **🎯 Environment Detection**: Browser, Node.js, or Serverless +- **💾 Smart Memory Management**: Uses available RAM optimally +- **🗄️ Storage Selection**: OPFS, FileSystem, S3, or Memory +- **⚡ Performance Tuning**: Threading, caching, compression +- **📊 Adaptive Learning**: Improves performance over time +- **🔍 Semantic Partitioning**: Auto-clusters similar vectors -Brainy combines vector similarity with graph relationships: +## 🏁 Traditional Setup (Manual Configuration) + +If you prefer manual control: ```typescript -// Add entities (automatically vectorized) +import { BrainyData, NounType, VerbType } from '@soulcraft/brainy' + +// Create and initialize the database +const db = new BrainyData() +await db.init() + +// Add data (automatically converted to vectors) const catId = await db.add("Cats are independent pets", { noun: NounType.Thing, category: 'animal' @@ -98,124 +222,1664 @@ const dogId = await db.add("Dogs are loyal companions", { category: 'animal' }) -// Create relationships +// Search for similar items +const results = await db.searchText("feline pets", 2) +console.log(results) + +// Add a relationship between items await db.addVerb(catId, dogId, { verb: VerbType.RelatedTo, - relationship: 'both are pets' + description: 'Both are common household pets' }) - -// Search returns both similar vectors AND related entities -const results = await db.searchText("household pets", 5) ``` -### Advanced Search Capabilities +### Import Options ```typescript -// Search within JSON document fields -const results = await db.search("Acme Corp", 10, { - searchField: "company" -}) +// Standard import - automatically adapts to any environment +import { BrainyData } from '@soulcraft/brainy' -// Search by entity type -const people = await db.searchByNounTypes([NounType.Person], 10) - -// Batch operations for performance -const ids = await db.addBatch([ - { vectorOrData: "First item", metadata: { category: 'A' }}, - { vectorOrData: "Second item", metadata: { category: 'B' }} -], { concurrency: 4, batchSize: 50 }) +// Minified version for production +import { BrainyData } from '@soulcraft/brainy/min' ``` -## 🏗️ Architecture +> **Note**: The CLI functionality is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size +> of the main package. Install it globally with `npm install -g @soulcraft/brainy-cli` to use the command-line +> interface. -Brainy uses cutting-edge technologies to deliver exceptional performance: +### Browser Usage -- **HNSW Algorithm** - Hierarchical Navigable Small World for O(log n) search -- **TensorFlow.js** - Hardware-accelerated embeddings using GPU/WebGL -- **Multi-tier Storage** - Hot (RAM) → Warm (Fast Storage) → Cold (S3/Disk) -- **Semantic Partitioning** - Auto-clusters similar vectors for faster retrieval -- **Adaptive Caching** - Learns your access patterns and optimizes accordingly +```html + + +``` + +Modern bundlers like Webpack, Rollup, and Vite will automatically use the unified build which adapts to any environment. + +## 🧩 How It Works + +Brainy combines **six advanced optimization systems** with core vector database technologies to create a +production-ready, self-optimizing system: + +### 🔧 Core Technologies + +1. **Vector Embeddings** - Converts data (text, images, etc.) into numerical vectors using TensorFlow.js +2. **Optimized HNSW Algorithm** - Fast similarity search with semantic partitioning and distributed processing +3. **🧠 Auto-Configuration Engine** - Detects environment, resources, and data patterns to optimize automatically +4. **🎯 Intelligent Storage System** - Multi-level caching with predictive prefetching and batch operations + +### ⚡ Advanced Optimization Layer + +5. **Semantic Partitioning** - Auto-clusters similar vectors for faster search (4-32 clusters based on scale) +6. **Distributed Search** - Parallel processing across partitions with intelligent load balancing +7. **Multi-Level Caching** - Hot (RAM) → Warm (Fast Storage) → Cold (S3/Disk) with 70-90% hit rates +8. **Batch Operations** - Reduces S3 API calls by 50-90% through intelligent batching +9. **Adaptive Learning** - Continuously learns from usage patterns and optimizes performance +10. **Advanced Compression** - Vector quantization achieves 75% memory reduction for large datasets + +### 🎯 Environment-Specific Optimizations + +| Environment | Storage | Threading | Memory | Focus | +|----------------|-----------------|----------------|-----------|------------------| +| **Browser** | OPFS + Cache | Web Workers | 512MB-1GB | Responsiveness | +| **Node.js** | FileSystem + S3 | Worker Threads | 4GB-8GB+ | Throughput | +| **Serverless** | S3 + Memory | Limited | 1GB-2GB | Cold Start Speed | + +### 🔄 Adaptive Intelligence Flow + +``` +Data Input → Auto-Detection → Environment Optimization → Semantic Partitioning → +Distributed Search → Multi-Level Caching → Performance Learning → Self-Tuning +``` + +The system **automatically adapts** to your environment, learns from your usage patterns, and **continuously optimizes +itself** for better performance over time. + +## 🚀 The Brainy Pipeline + +Brainy's data processing pipeline transforms raw data into searchable, connected knowledge that gets smarter over time: + +``` +Raw Data → Embedding → Vector Storage → Graph Connections → Adaptive Learning → Query & Retrieval +``` + +Each time data flows through this pipeline, Brainy learns more about your usage patterns and environment, making future +operations faster and more relevant. + +### Pipeline Stages + +1. **Data Ingestion** + - Raw text or pre-computed vectors enter the pipeline + - Data is validated and prepared for processing + +2. **Embedding Generation** + - Text is transformed into numerical vectors using embedding models + - Uses TensorFlow Universal Sentence Encoder for high-quality text embeddings + - Custom embedding functions can be plugged in for specialized domains + +3. **Vector Indexing** + - Vectors are indexed using the HNSW algorithm + - Hierarchical structure enables fast similarity search + - Configurable parameters for precision vs. performance tradeoffs + +4. **Graph Construction** + - Nouns (entities) become nodes in the knowledge graph + - Verbs (relationships) connect related entities + - Typed relationships add semantic meaning to connections + +5. **Adaptive Learning** + - Analyzes usage patterns to optimize future operations + - Tunes performance parameters based on your environment + - Adjusts search strategies based on query history + - Becomes more efficient and relevant the more you use it + +6. **Intelligent Storage** + - Data is saved using the optimal storage for your environment + - Automatic selection between OPFS, filesystem, S3, or memory + - Migrates between storage types as your application's needs evolve + - Scales from tiny datasets to massive data collections + - Configurable storage adapters for custom persistence needs + +### Augmentation Types + +Brainy uses a powerful augmentation system to extend functionality. Augmentations are processed in the following order: + +1. **SENSE** + - Ingests and processes raw, unstructured data into nouns and verbs + - Handles text, images, audio streams, and other input formats + - Example: Converting raw text into structured entities + +2. **MEMORY** + - Provides storage capabilities for data in different formats + - Manages persistence across sessions + - Example: Storing vectors in OPFS or filesystem + +3. **COGNITION** + - Enables advanced reasoning, inference, and logical operations + - Analyzes relationships between entities + - Examples: + - Inferring new connections between existing data + - Deriving insights from graph relationships + +4. **CONDUIT** + - Establishes channels for structured data exchange + - Connects with external systems and syncs between Brainy instances + - Two built-in iConduit augmentations for scaling out and syncing: + - **WebSocket iConduit** - Syncs data between browsers and servers + - **WebRTC iConduit** - Direct peer-to-peer syncing between browsers + - Examples: + - Integrating with third-party APIs + - Syncing Brainy instances between browsers using WebSockets + - Peer-to-peer syncing between browsers using WebRTC + +5. **ACTIVATION** + - Initiates actions, responses, or data manipulations + - Triggers events based on data changes + - Example: Sending notifications when new data is processed + +6. **PERCEPTION** + - Interprets, contextualizes, and visualizes identified nouns and verbs + - Creates meaningful representations of data + - Example: Generating visualizations of graph relationships + +7. **DIALOG** + - Facilitates natural language understanding and generation + - Enables conversational interactions + - Example: Processing user queries and generating responses + +8. **WEBSOCKET** + - Enables real-time communication via WebSockets + - Can be combined with other augmentation types + - Example: Streaming data processing in real-time + +### Streaming Data Support + +Brainy's pipeline is designed to handle streaming data efficiently: + +1. **WebSocket Integration** + - Built-in support for WebSocket connections + - Process data as it arrives without blocking + - Example: `setupWebSocketPipeline(url, dataType, options)` + +2. **Asynchronous Processing** + - Non-blocking architecture for real-time data handling + - Parallel processing of incoming streams + - Example: `createWebSocketHandler(connection, dataType, options)` + +3. **Event-Based Architecture** + - Augmentations can listen to data feeds and streams + - Real-time updates propagate through the pipeline + - Example: `listenToFeed(feedUrl, callback)` + +4. **Threaded Execution** + - Comprehensive multi-threading for high-performance operations + - Parallel processing for batch operations, vector calculations, and embedding generation + - Configurable execution modes (SEQUENTIAL, PARALLEL, THREADED) + - Automatic thread management based on environment capabilities + - Example: `executeTypedPipeline(augmentations, method, args, { mode: ExecutionMode.THREADED })` + +### Running the Pipeline + +The pipeline runs automatically when you: + +```typescript +// Add data (runs embedding → indexing → storage) +const id = await db.add("Your text data here", { metadata }) + +// Search (runs embedding → similarity search) +const results = await db.searchText("Your query here", 5) + +// Connect entities (runs graph construction → storage) +await db.addVerb(sourceId, targetId, { verb: VerbType.RelatedTo }) +``` + +Using the CLI: + +```bash +# Add data through the CLI pipeline +brainy add "Your text data here" '{"noun":"Thing"}' + +# Search through the CLI pipeline +brainy search "Your query here" --limit 5 + +# Connect entities through the CLI +brainy addVerb RelatedTo +``` + +### Extending the Pipeline + +Brainy's pipeline is designed for extensibility at every stage: + +1. **Custom Embedding** + ```typescript + // Create your own embedding function + const myEmbedder = async (text) => { + // Your custom embedding logic here + return [0.1, 0.2, 0.3, ...] // Return a vector + } + + // Use it in Brainy + const db = new BrainyData({ + embeddingFunction: myEmbedder + }) + ``` + +2. **Custom Distance Functions** + ```typescript + // Define your own distance function + const myDistance = (a, b) => { + // Your custom distance calculation + return Math.sqrt(a.reduce((sum, val, i) => sum + Math.pow(val - b[i], 2), 0)) + } + + // Use it in Brainy + const db = new BrainyData({ + distanceFunction: myDistance + +}) + + ``` + +3. **Custom Storage Adapters** + ```typescript + // Implement the StorageAdapter interface + class MyStorage implements StorageAdapter { + // Your storage implementation + } + + // Use it in Brainy + const db = new BrainyData({ + storageAdapter: new MyStorage() + }) + ``` + +4. **Augmentations System** + ```typescript + // Create custom augmentations to extend functionality + const myAugmentation = { + type: 'memory', + name: 'my-custom-storage', + // Implementation details + } + + // Register with Brainy + db.registerAugmentation(myAugmentation) + +``` + +## Data Model + +Brainy uses a graph-based data model with two primary concepts: + +### Nouns (Entities) + +The main entities in your data (nodes in the graph): + +- Each noun has a unique ID, vector representation, and metadata +- Nouns can be categorized by type (Person, Place, Thing, Event, Concept, etc.) +- Nouns are automatically vectorized for similarity search + +### Verbs (Relationships) + +Connections between nouns (edges in the graph): + +- Each verb connects a source noun to a target noun +- Verbs have types that define the relationship (RelatedTo, Controls, Contains, etc.) +- Verbs can have their own metadata to describe the relationship + +### Type Utilities + +Brainy provides utility functions to access lists of noun and verb types: + +```typescript +import { + NounType, + VerbType, + getNounTypes, + getVerbTypes, + getNounTypeMap, + getVerbTypeMap +} from '@soulcraft/brainy' + +// At development time: +// Access specific types directly from the NounType and VerbType objects +console.log(NounType.Person) // 'person' +console.log(VerbType.Contains) // 'contains' + +// At runtime: +// Get a list of all noun types +const nounTypes = getNounTypes() // ['person', 'organization', 'location', ...] + +// Get a list of all verb types +const verbTypes = getVerbTypes() // ['relatedTo', 'contains', 'partOf', ...] + +// Get a map of noun type keys to values +const nounTypeMap = getNounTypeMap() // { Person: 'person', Organization: 'organization', ... } + +// Get a map of verb type keys to values +const verbTypeMap = getVerbTypeMap() // { RelatedTo: 'relatedTo', Contains: 'contains', ... } +``` + +These utility functions make it easy to: + +- Get a complete list of available noun and verb types +- Validate user input against valid types +- Create dynamic UI components that display or select from available types +- Map between type keys and their string values + +## Command Line Interface + +Brainy includes a powerful CLI for managing your data. The CLI is available as a separate package +`@soulcraft/brainy-cli` to reduce the bundle size of the main package. + +### Installing and Using the CLI + +```bash +# Install the CLI globally +npm install -g @soulcraft/brainy-cli + +# Initialize a database +brainy init + +# Add some data +brainy add "Cats are independent pets" '{"noun":"Thing","category":"animal"}' +brainy add "Dogs are loyal companions" '{"noun":"Thing","category":"animal"}' + +# Search for similar items +brainy search "feline pets" 5 + +# Add relationships between items +brainy addVerb RelatedTo '{"description":"Both are pets"}' + +# Visualize the graph structure +brainy visualize +brainy visualize --root --depth 3 +``` + +### Using the CLI in Your Code + +The CLI functionality is available as a separate package `@soulcraft/brainy-cli`. If you need CLI functionality in your +application, install the CLI package: + +```bash +npm install @soulcraft/brainy-cli +``` + +Then you can use the CLI commands programmatically or through the command line interface. + +### Available Commands + +#### Basic Database Operations: + +- `init` - Initialize a new database +- `add [metadata]` - Add a new noun with text and optional metadata +- `search [limit]` - Search for nouns similar to the query +- `get ` - Get a noun by ID +- `delete ` - Delete a noun by ID +- `addVerb [metadata]` - Add a relationship +- `getVerbs ` - Get all relationships for a noun +- `status` - Show database status +- `clear` - Clear all data from the database +- `generate-random-graph` - Generate test data +- `visualize` - Visualize the graph structure +- `completion-setup` - Setup shell autocomplete + +#### Pipeline and Augmentation Commands: + +- `list-augmentations` - List all available augmentation types and registered augmentations +- `augmentation-info ` - Get detailed information about a specific augmentation type +- `test-pipeline [text]` - Test the sequential pipeline with sample data + - `-t, --data-type ` - Type of data to process (default: 'text') + - `-m, --mode ` - Execution mode: sequential, parallel, threaded (default: 'sequential') + - `-s, --stop-on-error` - Stop execution if an error occurs + - `-v, --verbose` - Show detailed output +- `stream-test` - Test streaming data through the pipeline (simulated) + - `-c, --count ` - Number of data items to stream (default: 5) + - `-i, --interval ` - Interval between data items in milliseconds (default: 1000) + - `-t, --data-type ` - Type of data to process (default: 'text') + - `-v, --verbose` - Show detailed output ## 📚 Documentation -- **[Getting Started Guide](docs/getting-started/)** - Installation and first steps -- **[User Guides](docs/user-guides/)** - Advanced usage and best practices -- **[API Reference](docs/api-reference/)** - Complete method documentation -- **[Optimization Guide](docs/optimization-guides/large-scale-optimizations.md)** - Scale to millions of vectors -- **[Examples](examples/)** - Real-world usage examples +### 🚀 [Getting Started](docs/getting-started/) -## 🛠️ Advanced Features +Quick setup guides and first steps with Brainy. -### Augmentation Pipeline +- **[Installation](docs/getting-started/installation.md)** - Installation and setup +- **[Quick Start](docs/getting-started/quick-start.md)** - Get running in 2 minutes +- **[First Steps](docs/getting-started/first-steps.md)** - Core concepts and features +- **[Environment Setup](docs/getting-started/environment-setup.md)** - Environment-specific configuration -Extend Brainy with custom data processing: +### 📖 [User Guides](docs/user-guides/) + +Comprehensive guides for using Brainy effectively. + +- **[Search and Metadata](docs/user-guides/SEARCH_AND_METADATA_GUIDE.md)** - Advanced search techniques +- **[Write-Only Mode](docs/user-guides/WRITEONLY_MODE_IMPLEMENTATION.md)** - High-throughput data loading +- **[JSON Document Search](docs/guides/json-document-search.md)** - Search within JSON fields +- **[Production Migration](docs/guides/production-migration-guide.md)** - Deployment best practices + +### ⚡ [Optimization Guides](docs/optimization-guides/) + +Transform Brainy from prototype to production-ready system. + +- **[Large-Scale Optimizations](docs/optimization-guides/large-scale-optimizations.md)** - Complete v0.36.0 optimization + system +- **[Auto-Configuration](docs/optimization-guides/auto-configuration.md)** - Intelligent environment detection +- **[Memory Optimization](docs/optimization-guides/memory-optimization.md)** - Advanced memory management +- **[Storage Optimization](docs/optimization-guides/storage-optimization.md)** - S3 and storage optimization + +### 🔧 [API Reference](docs/api-reference/) + +Complete API documentation and method references. + +- **[Core API](docs/api-reference/core-api.md)** - Main BrainyData class methods +- **[Vector Operations](docs/api-reference/vector-operations.md)** - Vector storage and search +- **[Configuration](docs/api-reference/configuration.md)** - System configuration +- **[Auto-Configuration API](docs/api-reference/auto-configuration-api.md)** - Intelligent configuration + +### 💡 [Examples](docs/examples/) + +Practical code examples and real-world applications. + +- **[Basic Usage](docs/examples/basic-usage.md)** - Simple examples to get started +- **[Advanced Patterns](docs/examples/advanced-patterns.md)** - Complex use cases +- **[Integrations](docs/examples/integrations.md)** - Third-party service integrations +- **[Performance Examples](docs/examples/performance.md)** - Optimization and scaling + +### 🔬 Technical Documentation + +- **[Testing Guide](docs/technical/TESTING.md)** - Testing strategies and best practices +- **[Statistics Guide](STATISTICS.md)** - Database statistics and monitoring +- **[Technical Guides](TECHNICAL_GUIDES.md)** - Advanced technical topics + +## API Reference + +### Database Management ```typescript -import { createSenseAugmentation, pipeline } from '@soulcraft/brainy' +// Initialize the database +await db.init() -// Create custom augmentation -const customProcessor = createSenseAugmentation({ - name: 'my-processor', - processRawData: async (data, type) => { - // Your custom logic - return { nouns: [], verbs: [] } +// Clear all data +await db.clear() + +// Get database status +const status = await db.status() + +// Backup all data from the database +const backupData = await db.backup() + +// Restore data into the database +const restoreResult = await db.restore(backupData, { clearExisting: true }) +``` + +### Database Statistics + +Brainy provides a way to get statistics about the current state of the database. For detailed information about the +statistics system, including implementation details, scalability improvements, and usage examples, see +our [Statistics Guide](STATISTICS.md). + +```typescript +import { BrainyData, getStatistics } from '@soulcraft/brainy' + +// Create and initialize the database +const db = new BrainyData() +await db.init() + +// Get statistics using the instance method +const stats = await db.getStatistics() +console.log(stats) +// Output: { nounCount: 0, verbCount: 0, metadataCount: 0, hnswIndexSize: 0, serviceBreakdown: {...} } +``` + +### Working with Nouns (Entities) + +```typescript +// Add a noun (automatically vectorized) +const id = await db.add(textOrVector, { + noun: NounType.Thing, + // other metadata... +}) + +// Add multiple nouns in parallel (with multithreading and batch embedding) +const ids = await db.addBatch([ + { + vectorOrData: "First item to add", + metadata: { noun: NounType.Thing, category: 'example' } + }, + { + vectorOrData: "Second item to add", + metadata: { noun: NounType.Thing, category: 'example' } + }, + // More items... +], { + forceEmbed: false, + concurrency: 4, // Control the level of parallelism (default: 4) + batchSize: 50 // Control the number of items to process in a single batch (default: 50) +}) + +// Retrieve a noun +const noun = await db.get(id) + +// Update noun metadata +await db.updateMetadata(id, { + noun: NounType.Thing, + // updated metadata... +}) + +// Delete a noun +await db.delete(id) + +// Search for similar nouns +const results = await db.search(vectorOrText, numResults) +const textResults = await db.searchText("query text", numResults) + +// Search by noun type +const thingNouns = await db.searchByNounTypes([NounType.Thing], numResults) + +// Search within specific fields of JSON documents +const fieldResults = await db.search("Acme Corporation", 10, { + searchField: "company" +}) + +// Search using standard field names across different services +const titleResults = await db.searchByStandardField("title", "climate change", 10) +const authorResults = await db.searchByStandardField("author", "johndoe", 10, { + services: ["github", "reddit"] +}) +``` + +### Field Standardization and Service Tracking + +Brainy automatically tracks field names from JSON documents and associates them with the service that inserted the data. +This enables powerful cross-service search capabilities: + +```typescript +// Get all available field names organized by service +const fieldNames = await db.getAvailableFieldNames() +// Example output: { "github": ["repository.name", "issue.title"], "reddit": ["title", "selftext"] } + +// Get standard field mappings +const standardMappings = await db.getStandardFieldMappings() +// Example output: { "title": { "github": ["repository.name"], "reddit": ["title"] } } +``` + +When adding data, specify the service name to ensure proper field tracking: + +```typescript +// Add data with service name +await db.add(jsonData, metadata, { service: "github" }) +``` + +### Working with Verbs (Relationships) + +```typescript +// Add a relationship between nouns +await db.addVerb(sourceId, targetId, { + verb: VerbType.RelatedTo, + // other metadata... +}) + +// Add a relationship with auto-creation of missing nouns +// This is useful when the target noun might not exist yet +await db.addVerb(sourceId, targetId, { + verb: VerbType.RelatedTo, + // Enable auto-creation of missing nouns + autoCreateMissingNouns: true, + // Optional metadata for auto-created nouns + missingNounMetadata: { + noun: NounType.Concept, + description: 'Auto-created noun' } }) -// Register and use -pipeline.register(customProcessor) +// Get all relationships +const verbs = await db.getAllVerbs() + +// Get relationships by source noun +const outgoingVerbs = await db.getVerbsBySource(sourceId) + +// Get relationships by target noun +const incomingVerbs = await db.getVerbsByTarget(targetId) + +// Get relationships by type +const containsVerbs = await db.getVerbsByType(VerbType.Contains) + +// Get a specific relationship +const verb = await db.getVerb(verbId) + +// Delete a relationship +await db.deleteVerb(verbId) ``` +## Advanced Configuration + +### Database Modes + +Brainy supports special operational modes that restrict certain operations: + +```typescript +import { BrainyData } from '@soulcraft/brainy' + +// Create and initialize the database +const db = new BrainyData() +await db.init() + +// Set the database to read-only mode (prevents write operations) +db.setReadOnly(true) + +// Check if the database is in read-only mode +const isReadOnly = db.isReadOnly() // Returns true + +// Set the database to write-only mode (prevents search operations) +db.setWriteOnly(true) + +// Check if the database is in write-only mode +const isWriteOnly = db.isWriteOnly() // Returns true + +// Reset to normal mode (allows both read and write operations) +db.setReadOnly(false) +db.setWriteOnly(false) +``` + +- **Read-Only Mode**: When enabled, prevents all write operations (add, update, delete). Useful for deployment scenarios + where you want to prevent modifications to the database. +- **Write-Only Mode**: When enabled, prevents all search operations. Useful for initial data loading or when you want to + optimize for write performance. + +### Embedding + +```typescript +import { + BrainyData, + createTensorFlowEmbeddingFunction, + createThreadedEmbeddingFunction +} from '@soulcraft/brainy' + +// Use the standard TensorFlow Universal Sentence Encoder embedding function +const db = new BrainyData({ + embeddingFunction: createTensorFlowEmbeddingFunction() +}) +await db.init() + +// Or use the threaded embedding function for better performance +const threadedDb = new BrainyData({ + embeddingFunction: createThreadedEmbeddingFunction() +}) +await threadedDb.init() + +// Directly embed text to vectors +const vector = await db.embed("Some text to convert to a vector") + +// Calculate similarity between two texts or vectors +const similarity = await db.calculateSimilarity( + "Cats are furry pets", + "Felines make good companions" +) +console.log(`Similarity score: ${similarity}`) // Higher value means more similar + +// Calculate similarity with custom options +const vectorA = await db.embed("First text") +const vectorB = await db.embed("Second text") +const customSimilarity = await db.calculateSimilarity( + vectorA, // Can use pre-computed vectors + vectorB, + { + forceEmbed: false, // Skip embedding if inputs are already vectors + distanceFunction: cosineDistance // Optional custom distance function + } +) +``` + +The threaded embedding function runs in a separate thread (Web Worker in browsers, Worker Thread in Node.js) to improve +performance, especially for embedding operations. It uses GPU acceleration when available (via WebGL in browsers) and +falls back to CPU processing for compatibility. Universal Sentence Encoder is always used for embeddings. The +implementation includes worker reuse and model caching for optimal performance. + +### Performance Tuning + +Brainy includes comprehensive performance optimizations that work across all environments (browser, CLI, Node.js, +container, server): + +#### GPU and CPU Optimization + +Brainy uses GPU and CPU optimization for compute-intensive operations: + +1. **GPU-Accelerated Embeddings**: Generate text embeddings using TensorFlow.js with WebGL backend when available +2. **Automatic Fallback**: Falls back to CPU backend when GPU is not available +3. **Optimized Distance Calculations**: Perform vector similarity calculations with optimized algorithms +4. **Cross-Environment Support**: Works consistently across browsers and Node.js environments +5. **Memory Management**: Properly disposes of tensors to prevent memory leaks + +#### Multithreading Support + +Brainy includes comprehensive multithreading support to improve performance across all environments: + +1. **Parallel Batch Processing**: Add multiple items concurrently with controlled parallelism +2. **Multithreaded Vector Search**: Perform distance calculations in parallel for faster search operations +3. **Threaded Embedding Generation**: Generate embeddings in separate threads to avoid blocking the main thread +4. **Worker Reuse**: Maintains a pool of workers to avoid the overhead of creating and terminating workers +5. **Model Caching**: Initializes the embedding model once per worker and reuses it for multiple operations +6. **Batch Embedding**: Processes multiple items in a single embedding operation for better performance +7. **Automatic Environment Detection**: Adapts to browser (Web Workers) and Node.js (Worker Threads) environments + +```typescript +import { BrainyData, euclideanDistance } from '@soulcraft/brainy' + +// Configure with custom options +const db = new BrainyData({ + // Use Euclidean distance instead of default cosine distance + distanceFunction: euclideanDistance, + + // HNSW index configuration for search performance + hnsw: { + M: 16, // Max connections per noun + efConstruction: 200, // Construction candidate list size + efSearch: 50, // Search candidate list size + }, + + // Performance optimization options + performance: { + useParallelization: true, // Enable multithreaded search operations + }, + + // Noun and Verb type validation + typeValidation: { + enforceNounTypes: true, // Validate noun types against NounType enum + enforceVerbTypes: true, // Validate verb types against VerbType enum + }, + + // Storage configuration + storage: { + requestPersistentStorage: true, + // Example configuration for cloud storage (replace with your own values): + // s3Storage: { + // bucketName: 'your-s3-bucket-name', + // region: 'your-aws-region' + // // Credentials should be provided via environment variables + // // AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY + // } + } +}) +``` + +### Optimized HNSW for Large Datasets + +Brainy includes an optimized HNSW index implementation for large datasets that may not fit entirely in memory, using a +hybrid approach: + +1. **Product Quantization** - Reduces vector dimensionality while preserving similarity relationships +2. **Disk-Based Storage** - Offloads vectors to disk when memory usage exceeds a threshold +3. **Memory-Efficient Indexing** - Optimizes memory usage for large-scale vector collections + +```typescript +import { BrainyData } from '@soulcraft/brainy' + +// Configure with optimized HNSW index for large datasets +const db = new BrainyData({ + hnswOptimized: { + // Standard HNSW parameters + M: 16, // Max connections per noun + efConstruction: 200, // Construction candidate list size + efSearch: 50, // Search candidate list size + + // Memory threshold in bytes - when exceeded, will use disk-based approach + memoryThreshold: 1024 * 1024 * 1024, // 1GB default threshold + + // Product quantization settings for dimensionality reduction + productQuantization: { + enabled: true, // Enable product quantization + numSubvectors: 16, // Number of subvectors to split the vector into + numCentroids: 256 // Number of centroids per subvector + }, + + // Whether to use disk-based storage for the index + useDiskBasedIndex: true // Enable disk-based storage + }, + + // Storage configuration (required for disk-based index) + storage: { + requestPersistentStorage: true + } +}) + +// The optimized index automatically adapts based on dataset size: +// 1. For small datasets: Uses standard in-memory approach +// 2. For medium datasets: Applies product quantization to reduce memory usage +// 3. For large datasets: Combines product quantization with disk-based storage + +// Check status to see memory usage and optimization details +const status = await db.status() +console.log(status.details.index) +``` + +## Distance Functions + +Brainy provides several distance functions for vector similarity calculations: + +- `cosineDistance` (default): Measures the cosine of the angle between vectors (1 - cosine similarity) +- `euclideanDistance`: Measures the straight-line distance between vectors +- `manhattanDistance`: Measures the sum of absolute differences between vector components +- `dotProductDistance`: Measures the negative dot product between vectors + +All distance functions are optimized for performance and automatically use the most efficient implementation based on +the dataset size and available resources. For large datasets and high-dimensional vectors, Brainy uses batch processing +and multithreading when available to improve performance. + +## Backup and Restore + +Brainy provides backup and restore capabilities that allow you to: + +- Back up your data +- Transfer data between Brainy instances +- Restore existing data into Brainy for vectorization and indexing +- Backup data for analysis or visualization in other tools + +### Backing Up Data + +```typescript +// Backup all data from the database +const backupData = await db.backup() + +// The backup data includes: +// - All nouns (entities) with their vectors and metadata +// - All verbs (relationships) between nouns +// - Noun types and verb types +// - HNSW index data for fast similarity search +// - Version information + +// Save the backup data to a file (Node.js environment) +import fs from 'fs' + +fs.writeFileSync('brainy-backup.json', JSON.stringify(backupData, null, 2)) +``` + +### Restoring Data + +Brainy's restore functionality can handle: + +1. Complete backups with vectors and index data +2. Sparse data without vectors (vectors will be created during restore) +3. Data without HNSW index (index will be reconstructed if needed) + +```typescript +// Restore data with all options +const restoreResult = await db.restore(backupData, { + clearExisting: true // Whether to clear existing data before restore +}) + +// Import sparse data (without vectors) +// Vectors will be automatically created using the embedding function +const sparseData = { + nouns: [ + { + id: '123', + // No vector field - will be created during import + metadata: { + noun: 'Thing', + text: 'This text will be used to generate a vector' + } + } + ], + verbs: [], + version: '1.0.0' +} + +const sparseImportResult = await db.importSparseData(sparseData) +``` + +### CLI Backup/Restore + +```bash +# Backup data to a file +brainy backup --output brainy-backup.json + +# Restore data from a file +brainy restore --input brainy-backup.json --clear-existing + +# Import sparse data (without vectors) +brainy import-sparse --input sparse-data.json +``` + +## Embedding + +Brainy uses the following embedding approach: + +- TensorFlow Universal Sentence Encoder (high-quality text embeddings) +- GPU acceleration when available (via WebGL in browsers) +- Batch embedding for processing multiple items efficiently +- Worker reuse and model caching for optimal performance +- Custom embedding functions can be plugged in for specialized domains + +## Extensions + +Brainy includes an augmentation system for extending functionality: + +- **Memory Augmentations**: Different storage backends +- **Sense Augmentations**: Process raw data +- **Cognition Augmentations**: Reasoning and inference +- **Dialog Augmentations**: Text processing and interaction +- **Perception Augmentations**: Data interpretation and visualization +- **Activation Augmentations**: Trigger actions + +### Simplified Augmentation System + +Brainy provides a simplified factory system for creating, importing, and executing augmentations with minimal +boilerplate: + +```typescript +import { + createMemoryAugmentation, + createConduitAugmentation, + createSenseAugmentation, + addWebSocketSupport, + executeStreamlined, + processStaticData, + processStreamingData, + createPipeline +} from '@soulcraft/brainy' + +// Create a memory augmentation with minimal code +const memoryAug = createMemoryAugmentation({ + name: 'simple-memory', + description: 'A simple in-memory storage augmentation', + autoRegister: true, + autoInitialize: true, + + // Implement only the methods you need + storeData: async (key, data) => { + // Your implementation here + return { + success: true, + data: true + } + }, + + retrieveData: async (key) => { + // Your implementation here + return { + success: true, + data: { example: 'data', key } + } + } +}) + +// Add WebSocket support to any augmentation +const wsAugmentation = addWebSocketSupport(memoryAug, { + connectWebSocket: async (url) => { + // Your implementation here + return { + connectionId: 'ws-1', + url, + status: 'connected' + } + } +}) + +// Process static data through a pipeline +const result = await processStaticData( + 'Input data', + [ + { + augmentation: senseAug, + method: 'processRawData', + transformArgs: (data) => [data, 'text'] + }, + { + augmentation: memoryAug, + method: 'storeData', + transformArgs: (data) => ['processed-data', data] + } + ] +) + +// Create a reusable pipeline +const pipeline = createPipeline([ + { + augmentation: senseAug, + method: 'processRawData', + transformArgs: (data) => [data, 'text'] + }, + { + augmentation: memoryAug, + method: 'storeData', + transformArgs: (data) => ['processed-data', data] + } +]) + +// Use the pipeline +const result = await pipeline('New input data') + +// Dynamically load augmentations at runtime +const loadedAugmentations = await loadAugmentationModule( + import('./my-augmentations.js'), + { + autoRegister: true, + autoInitialize: true + } +) +``` + +The simplified augmentation system provides: + +1. **Factory Functions** - Create augmentations with minimal boilerplate +2. **WebSocket Support** - Add WebSocket capabilities to any augmentation +3. **Streamlined Pipeline** - Process data through augmentations more efficiently +4. **Dynamic Loading** - Load augmentations at runtime when needed +5. **Static & Streaming Data** - Handle both static and streaming data with the same API + +#### WebSocket Augmentation Types + +Brainy exports several WebSocket augmentation types that can be used by augmentation creators to add WebSocket +capabilities to their augmentations: + +```typescript +import { + // Base WebSocket support interface + IWebSocketSupport, + + // Combined WebSocket augmentation types + IWebSocketSenseAugmentation, + IWebSocketConduitAugmentation, + IWebSocketCognitionAugmentation, + IWebSocketMemoryAugmentation, + IWebSocketPerceptionAugmentation, + IWebSocketDialogAugmentation, + IWebSocketActivationAugmentation, + + // Function to add WebSocket support to any augmentation + addWebSocketSupport +} from '@soulcraft/brainy' + +// Example: Creating a typed WebSocket-enabled sense augmentation +const mySenseAug = createSenseAugmentation({ + name: 'my-sense', + processRawData: async (data, dataType) => { + // Implementation + return { + success: true, + data: { nouns: [], verbs: [] } + } + } +}) as IWebSocketSenseAugmentation + +// Add WebSocket support +addWebSocketSupport(mySenseAug, { + connectWebSocket: async (url) => { + // WebSocket implementation + return { + connectionId: 'ws-1', + url, + status: 'connected' + } + }, + sendWebSocketMessage: async (connectionId, data) => { + // Send message implementation + }, + onWebSocketMessage: async (connectionId, callback) => { + // Register callback implementation + }, + offWebSocketMessage: async (connectionId, callback) => { + // Remove callback implementation + }, + closeWebSocket: async (connectionId, code, reason) => { + // Close connection implementation + } +}) + +// Now mySenseAug has both sense augmentation methods and WebSocket methods +await mySenseAug.processRawData('data', 'text') +await mySenseAug.connectWebSocket('wss://example.com') +``` + +These WebSocket augmentation types combine the base augmentation interfaces with the `IWebSocketSupport` interface, +providing type safety and autocompletion for augmentations with WebSocket capabilities. + ### Model Control Protocol (MCP) -Let AI models interact with your Brainy database: +Brainy includes a Model Control Protocol (MCP) implementation that allows external models to access Brainy data and use +the augmentation pipeline as tools: + +- **BrainyMCPAdapter**: Provides access to Brainy data through MCP +- **MCPAugmentationToolset**: Exposes the augmentation pipeline as tools +- **BrainyMCPService**: Integrates the adapter and toolset, providing WebSocket and REST server implementations + +Environment compatibility: + +- **BrainyMCPAdapter** and **MCPAugmentationToolset** can run in any environment (browser, Node.js, server) +- **BrainyMCPService** core functionality works in any environment + +For detailed documentation and usage examples, see the [MCP documentation](src/mcp/README.md). + +## Cross-Environment Compatibility + +Brainy is designed to run seamlessly in any environment, from browsers to Node.js to serverless functions and +containers. All Brainy data, functions, and augmentations are environment-agnostic, allowing you to use the same code +everywhere. + +### Environment Detection + +Brainy automatically detects the environment it's running in: ```typescript -import { BrainyMCPAdapter } from '@soulcraft/brainy' +import { environment } from '@soulcraft/brainy' -const adapter = new BrainyMCPAdapter(brainy) -// Now AI models can query and update your database +// Check which environment we're running in +console.log(`Running in ${ + environment.isBrowser ? 'browser' : + environment.isNode ? 'Node.js' : + 'serverless/unknown' +} environment`) ``` -## 🎮 Live Demo +### Adaptive Storage -**[Try it now!](https://soulcraft-research.github.io/brainy/demo/index.html)** +Storage adapters are automatically selected based on the environment: -See Brainy in action with our interactive demo showcasing vector search, graph relationships, and cross-environment compatibility. +- **Browser**: Uses Origin Private File System (OPFS) when available, falls back to in-memory storage +- **Node.js**: Uses file system storage by default, with options for S3-compatible cloud storage +- **Serverless**: Uses in-memory storage with options for cloud persistence +- **Container**: Automatically detects and uses the appropriate storage based on available capabilities -## 🤝 Contributing +### Dynamic Imports -We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) and [Code of Conduct](CODE_OF_CONDUCT.md). +Brainy uses dynamic imports to load environment-specific dependencies only when needed, keeping the bundle size small +and ensuring compatibility across environments. -### Development +### Browser Support -```bash -# Install dependencies -npm install +Works in all modern browsers: -# Run tests -npm test +- Chrome 86+ +- Edge 86+ +- Opera 72+ +- Chrome for Android 86+ -# Build project -npm run build +For browsers without OPFS support, falls back to in-memory storage. -# Run demo locally -npm run demo +## Related Projects + +- **[Cartographer](https://github.com/sodal-project/cartographer)** - A companion project that provides standardized + interfaces for interacting with Brainy + +## Demo + +The repository includes a comprehensive demo that showcases Brainy's main features: + +- `demo/index.html` - A single demo page with animations demonstrating Brainy's features. + - **[Try the live demo](https://soulcraft-research.github.io/brainy/demo/index.html)** - Check out the + interactive demo on + GitHub Pages + - Or run it locally with `npm run demo` (see [demo instructions](demo.md) for details) + - To deploy your own version to GitHub Pages, use the GitHub Actions workflow in + `.github/workflows/deploy-demo.yml`, + which automatically deploys when pushing to the main branch or can be manually triggered + - To use a custom domain (like www.soulcraft.com): + 1. A CNAME file is already included in the demo directory + 2. In your GitHub repository settings, go to Pages > Custom domain and enter your domain + 3. Configure your domain's DNS settings to point to GitHub Pages: + + - Add a CNAME record for www pointing to `.github.io` (e.g., `soulcraft-research.github.io`) + - Or for an apex domain (soulcraft.com), add A records pointing to GitHub Pages IP addresses + +The demo showcases: + +- How Brainy runs in different environments (browser, Node.js, server, cloud) +- How the noun-verb data model works +- How HNSW search works + +## Syncing Brainy Instances + +You can use the conduit augmentations to sync Brainy instances: + +- **WebSocket iConduit**: For syncing between browsers and servers, or between servers. WebSockets cannot be used for + direct browser-to-browser communication without a server in the middle. +- **WebRTC iConduit**: For direct peer-to-peer syncing between browsers. This is the recommended approach for + browser-to-browser communication. + +#### WebSocket Sync Example + +```typescript +import { + BrainyData, + pipeline, + createConduitAugmentation +} from '@soulcraft/brainy' + +// Create and initialize the database +const db = new BrainyData() +await db.init() + +// Create a WebSocket conduit augmentation +const wsConduit = await createConduitAugmentation('websocket', 'my-websocket-sync') + +// Register the augmentation with the pipeline +pipeline.register(wsConduit) + +// Connect to another Brainy instance (server or browser) +// Replace the example URL below with your actual WebSocket server URL +const connectionResult = await pipeline.executeConduitPipeline( + 'establishConnection', + ['wss://example-websocket-server.com/brainy-sync', { protocols: 'brainy-sync' }] +) + +if (connectionResult[0] && (await connectionResult[0]).success) { + const connection = (await connectionResult[0]).data + + // Read data from the remote instance + const readResult = await pipeline.executeConduitPipeline( + 'readData', + [{ connectionId: connection.connectionId, query: { type: 'getAllNouns' } }] + ) + + // Process and add the received data to the local instance + if (readResult[0] && (await readResult[0]).success) { + const remoteNouns = (await readResult[0]).data + for (const noun of remoteNouns) { + await db.add(noun.vector, noun.metadata) + } + } + + // Set up real-time sync by monitoring the stream + await wsConduit.monitorStream(connection.connectionId, async (data) => { + // Handle incoming data (e.g., new nouns, verbs, updates) + if (data.type === 'newNoun') { + await db.add(data.vector, data.metadata) + } else if (data.type === 'newVerb') { + await db.addVerb(data.sourceId, data.targetId, data.vector, data.options) + } + }) +} ``` -## 📄 License +#### WebRTC Peer-to-Peer Sync Example -[MIT](LICENSE) - Use it freely in your projects! +```typescript +import { + BrainyData, + pipeline, + createConduitAugmentation +} from '@soulcraft/brainy' -## 🙏 Acknowledgments +// Create and initialize the database +const db = new BrainyData() +await db.init() -Built with ❤️ using: -- [TensorFlow.js](https://www.tensorflow.org/js) for embeddings -- [Universal Sentence Encoder](https://tfhub.dev/google/universal-sentence-encoder/4) for text vectorization +// Create a WebRTC conduit augmentation +const webrtcConduit = await createConduitAugmentation('webrtc', 'my-webrtc-sync') + +// Register the augmentation with the pipeline +pipeline.register(webrtcConduit) + +// Connect to a peer using a signaling server +// Replace the example values below with your actual configuration +const connectionResult = await pipeline.executeConduitPipeline( + 'establishConnection', + [ + 'peer-id-to-connect-to', // Replace with actual peer ID + { + signalServerUrl: 'wss://example-signal-server.com', // Replace with your signal server + localPeerId: 'my-local-peer-id', // Replace with your local peer ID + iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] // Public STUN server + } + ] +) + +if (connectionResult[0] && (await connectionResult[0]).success) { + const connection = (await connectionResult[0]).data + + // Set up real-time sync by monitoring the stream + await webrtcConduit.monitorStream(connection.connectionId, async (data) => { + // Handle incoming data (e.g., new nouns, verbs, updates) + if (data.type === 'newNoun') { + await db.add(data.vector, data.metadata) + } else if (data.type === 'newVerb') { + await db.addVerb(data.sourceId, data.targetId, data.vector, data.options) + } + }) + + // When adding new data locally, also send to the peer + const nounId = await db.add("New data to sync", { noun: "Thing" }) + + // Send the new noun to the peer + await pipeline.executeConduitPipeline( + 'writeData', + [ + { + connectionId: connection.connectionId, + data: { + type: 'newNoun', + id: nounId, + vector: (await db.get(nounId)).vector, + metadata: (await db.get(nounId)).metadata + } + } + ] + ) +} +``` + +#### Browser-Server Search Example + +Brainy supports searching a server-hosted instance from a browser, storing results locally, and performing further +searches against the local instance: + +```typescript +import { BrainyData } from '@soulcraft/brainy' + +// Create and initialize the database with remote server configuration +// Replace the example URL below with your actual Brainy server URL +const db = new BrainyData({ + remoteServer: { + url: 'wss://example-brainy-server.com/ws', // Replace with your server URL + protocols: 'brainy-sync', + autoConnect: true // Connect automatically during initialization + } +}) +await db.init() + +// Or connect manually after initialization +if (!db.isConnectedToRemoteServer()) { + // Replace the example URL below with your actual Brainy server URL + await db.connectToRemoteServer('wss://example-brainy-server.com/ws', 'brainy-sync') +} + +// Search the remote server (results are stored locally) +const remoteResults = await db.searchText('machine learning', 5, { searchMode: 'remote' }) + +// Search the local database (includes previously stored results) +const localResults = await db.searchText('machine learning', 5, { searchMode: 'local' }) + +// Perform a combined search (local first, then remote if needed) +const combinedResults = await db.searchText('neural networks', 5, { searchMode: 'combined' }) + +// Add data to both local and remote instances +const id = await db.addToBoth('Deep learning is a subset of machine learning', { + noun: 'Concept', + category: 'AI', + tags: ['deep learning', 'neural networks'] +}) + +// Clean up when done (this also cleans up worker pools) +await db.shutDown() +``` --- -
-Ready to build something amazing?
-Get Started → -
\ No newline at end of file +## 📈 Scaling Strategy + +Brainy is designed to handle datasets of various sizes, from small collections to large-scale deployments. For +terabyte-scale data that can't fit entirely in memory, we provide several approaches: + +- **Disk-Based HNSW**: Modified implementations using intelligent caching and partial loading +- **Distributed HNSW**: Sharding and partitioning across multiple machines +- **Hybrid Solutions**: Combining quantization techniques with multi-tier architectures + +For detailed information on how to scale Brainy for large datasets, vector dimension standardization, threading +implementation, storage testing, and other technical topics, see our +comprehensive [Technical Guides](TECHNICAL_GUIDES.md). + +## Recent Changes and Performance Improvements + +### Enhanced Memory Management and Scalability + +Brainy has been significantly improved to handle larger datasets more efficiently: + +- **Pagination Support**: All data retrieval methods now support pagination to avoid loading entire datasets into memory + at once. The deprecated `getAllNouns()` and `getAllVerbs()` methods have been replaced with `getNouns()` and + `getVerbs()` methods that support pagination, filtering, and cursor-based navigation. + +- **Multi-level Caching**: A sophisticated three-level caching strategy has been implemented: + - **Level 1**: Hot cache (most accessed nodes) - RAM (automatically detecting and adjusting in each environment) + - **Level 2**: Warm cache (recent nodes) - OPFS, Filesystem or S3 depending on environment + - **Level 3**: Cold storage (all nodes) - OPFS, Filesystem or S3 depending on environment + +- **Adaptive Memory Usage**: The system automatically detects available memory and adjusts cache sizes accordingly: + - In Node.js: Uses 10% of free memory (minimum 1000 entries) + - In browsers: Scales based on device memory (500 entries per GB, minimum 1000) + +- **Intelligent Cache Eviction**: Implements a Least Recently Used (LRU) policy that evicts the oldest 20% of items when + the cache reaches the configured threshold. + +- **Prefetching Strategy**: Implements batch prefetching to improve performance while avoiding overwhelming system + resources. + +### S3-Compatible Storage Improvements + +- **Enhanced Cloud Storage**: Improved support for S3-compatible storage services including AWS S3, Cloudflare R2, and + others. + +- **Optimized Data Access**: Batch operations and error handling for efficient cloud storage access. + +- **Change Log Management**: Efficient synchronization through change logs to track updates. + +### Data Compatibility + +Yes, you can use existing data indexed from an old version. Brainy includes robust data migration capabilities: + +- **Vector Regeneration**: If vectors are missing in imported data, they will be automatically created using the + embedding function. + +- **HNSW Index Reconstruction**: The system can reconstruct the HNSW index from backup data, ensuring compatibility with + previous versions. + +- **Sparse Data Import**: Support for importing sparse data (without vectors) through the `importSparseData()` method. + +### System Requirements + +#### Default Mode + +- **Memory**: + - Minimum: 512MB RAM + - Recommended: 2GB+ RAM for medium datasets, 8GB+ for large datasets + +- **CPU**: + - Minimum: 2 cores + - Recommended: 4+ cores for better performance with parallel operations + +- **Storage**: + - Minimum: 1GB available storage + - Recommended: Storage space at least 3x the size of your dataset + +#### Read-Only Mode + +Read-only mode prevents all write operations (add, update, delete) and is optimized for search operations. + +- **Memory**: + - Minimum: 256MB RAM + - Recommended: 1GB+ RAM + +- **CPU**: + - Minimum: 1 core + - Recommended: 2+ cores + +- **Storage**: + - Minimum: Storage space equal to the size of your dataset + - Recommended: 2x the size of your dataset for caching + +- **New Feature**: Lazy loading support in read-only mode for improved performance with large datasets. + +#### Write-Only Mode + +Write-only mode prevents all search operations and is optimized for initial data loading or when you want to optimize +for write performance. + +- **Memory**: + - Minimum: 512MB RAM + - Recommended: 2GB+ RAM + +- **CPU**: + - Minimum: 2 cores + - Recommended: 4+ cores for faster data ingestion + +- **Storage**: + - Minimum: Storage space at least 2x the size of your dataset + - Recommended: 4x the size of your dataset for optimal performance + +### Performance Tuning Parameters + +Brainy offers comprehensive configuration options for performance tuning, with enhanced support for large datasets in S3 +or other remote storage. **All configuration is optional** - the system automatically detects the optimal settings based +on your environment, dataset size, and usage patterns. + +#### Intelligent Defaults + +Brainy uses intelligent defaults that automatically adapt to your environment: + +- **Environment Detection**: Automatically detects whether you're running in Node.js, browser, or worker environment +- **Memory-Aware Caching**: Adjusts cache sizes based on available system memory +- **Dataset Size Adaptation**: Tunes parameters based on the size of your dataset +- **Usage Pattern Optimization**: Adjusts to read-heavy vs. write-heavy workloads +- **Storage Type Awareness**: Optimizes for local vs. remote storage (S3, R2, etc.) +- **Operating Mode Specialization**: Special optimizations for read-only and write-only modes + +#### Cache Configuration (Optional) + +You can override any of these automatically tuned parameters if needed: + +- **Hot Cache Size**: Control the maximum number of items to keep in memory. + - For large datasets (>100K items), consider values between 5,000-50,000 depending on available memory. + - In read-only mode, larger values (10,000-100,000) can be used for better performance. + +- **Eviction Threshold**: Set the threshold at which cache eviction begins (default: 0.8 or 80% of max size). + - For write-heavy workloads, lower values (0.6-0.7) may improve performance. + - For read-heavy workloads, higher values (0.8-0.9) are recommended. + +- **Warm Cache TTL**: Set the time-to-live for items in the warm cache (default: 3600000 ms or 1 hour). + - For frequently changing data, shorter TTLs are recommended. + - For relatively static data, longer TTLs improve performance. + +- **Batch Size**: Control the number of items to process in a single batch for operations like prefetching. + - For S3 or remote storage with large datasets, larger values (50-200) significantly improve throughput. + - In read-only mode with remote storage, even larger values (100-300) can be used. + +#### Auto-Tuning (Enabled by Default) + +- **Auto-Tune**: Enable or disable automatic tuning of cache parameters based on usage patterns (default: true). +- **Auto-Tune Interval**: Set how frequently the system adjusts cache parameters (default: 60000 ms or 1 minute). + +#### Read-Only Mode Optimizations (Automatic) + +Read-only mode includes special optimizations for search performance that are automatically applied: + +- **Larger Cache Sizes**: Automatically uses more memory for caching (up to 40% of free memory for large datasets). +- **Aggressive Prefetching**: Loads more data in each batch to reduce the number of storage requests. +- **Prefetch Strategy**: Defaults to 'aggressive' prefetching strategy in read-only mode. + +#### Example Configuration for Large S3 Datasets + +```javascript +const brainy = new BrainyData({ + readOnly: true, + lazyLoadInReadOnlyMode: true, + storage: { + type: 's3', + s3Storage: { + bucketName: 'your-bucket', + accessKeyId: 'your-access-key', + secretAccessKey: 'your-secret-key', + region: 'your-region' + } + }, + cache: { + hotCacheMaxSize: 20000, + hotCacheEvictionThreshold: 0.85, + batchSize: 100, + readOnlyMode: { + hotCacheMaxSize: 50000, + batchSize: 200, + prefetchStrategy: 'aggressive' + } + } +}); +``` + +These configuration options make Brainy more efficient, scalable, and adaptable to different environments and usage +patterns, especially for large datasets in cloud storage. + +## Testing + +Brainy uses Vitest for testing. For detailed information about testing in Brainy, including test configuration, scripts, +reporting tools, and best practices, see our [Testing Guide](docs/technical/TESTING.md). + +Here are some common test commands: + +```bash +# Run all tests +npm test + +# Run tests with comprehensive reporting +npm run test:report + +# Run tests with coverage +npm run test:coverage +``` + +## Contributing + +For detailed contribution guidelines, please see [CONTRIBUTING.md](CONTRIBUTING.md). + +For developer documentation, including building, testing, and publishing instructions, please +see [DEVELOPERS.md](DEVELOPERS.md). + +We have a [Code of Conduct](CODE_OF_CONDUCT.md) that all contributors are expected to follow. + +### Commit Message Format + +For best results with automatic changelog generation, follow +the [Conventional Commits](https://www.conventionalcommits.org/) specification for your commit messages: + +``` +AI Template for automated commit messages: + +Use Conventional Commit format +Specify the changes in a structured format +Add information about the purpose of the commit +``` + +``` +(): + +[optional body] + +[optional footer(s)] +``` + +Where `` is one of: + +- `feat`: A new feature (maps to **Added** section) +- `fix`: A bug fix (maps to **Fixed** section) +- `chore`: Regular maintenance tasks (maps to **Changed** section) +- `docs`: Documentation changes (maps to **Documentation** section) +- `refactor`: Code changes that neither fix bugs nor add features (maps to **Changed** section) +- `perf`: Performance improvements (maps to **Changed** section) + +### Manual Release Process + +If you need more control over the release process, you can use the individual commands: + +```bash +# Update version and generate changelog +npm run _release:patch # or _release:minor, _release:major + +# Create GitHub release +npm run _github-release + +# Publish to NPM +npm publish +``` + +## License + +[MIT](LICENSE)