diff --git a/.github/git-commit-instructions.md b/.github/git-commit-instructions.md
deleted file mode 100644
index ba724198..00000000
--- a/.github/git-commit-instructions.md
+++ /dev/null
@@ -1,3 +0,0 @@
-Use Conventional Commit format
-Specify the changes in a structured format
-Add information about the purpose of the commit
diff --git a/.gitignore b/.gitignore
index 344ed849..e08106b2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -70,4 +70,3 @@ debug*.ts
/temp/
/temp-tests/
/brainy-models-package/node_modules/
-/CLAUDE.md
diff --git a/README.md b/README.md
index f7736fec..f4bced4b 100644
--- a/README.md
+++ b/README.md
@@ -7,1879 +7,750 @@
[](https://www.typescriptlang.org/)
[](CONTRIBUTING.md)
-[//]: # ([](https://github.com/sodal-project/cartographer))
-
**A powerful graph & vector data platform for AI applications across any environment**
-## โจ Overview
+## โจ What is Brainy?
-Brainy combines the power of vector search with graph relationships in a lightweight, cross-platform database. Whether
-you're building AI applications, recommendation systems, or knowledge graphs, Brainy provides the tools you need to
-store, connect, and retrieve your data intelligently.
+Imagine a database that thinks like you do - connecting ideas, finding patterns, and getting smarter over time. Brainy is the **AI-native database** that brings vector search and knowledge graphs together in one powerful, ridiculously easy-to-use package.
-What makes Brainy special? It intelligently adapts to your environment! Brainy automatically detects your platform,
-adjusts its storage strategy, and optimizes performance based on your usage patterns. The more you use it, the smarter
-it gets - learning from your data to provide increasingly relevant results and connections.
+### ๐ Why Developers Love Brainy
-### ๐ Key Features
+- **๐ง It Just Worksโข** - No config files, no tuning parameters, no DevOps headaches. Brainy auto-detects your environment and optimizes itself
+- **๐ True Write-Once, Run-Anywhere** - Same code runs in React, Angular, Vue, Node.js, Deno, Bun, serverless, edge workers, and even vanilla HTML
+- **โก Scary Fast** - Handles millions of vectors with sub-millisecond search. Built-in GPU acceleration when available
+- **๐ฏ Self-Learning** - Like having a database that goes to the gym. Gets faster and smarter the more you use it
+- **๐ฎ AI-First Design** - Built for the age of embeddings, RAG, and semantic search. Your LLMs will thank you
+- **๐ฎ Actually Fun to Use** - Clean API, great DX, and it does the heavy lifting so you can build cool stuff
-- **๐ง Zero Configuration** - Auto-detects environment and optimizes automatically
-- **โก Production-Scale Performance** - Handles millions of vectors with sub-second search
-- **๐ฏ Intelligent Partitioning** - Semantic clustering with auto-tuning
-- **๐ Adaptive Learning** - Gets smarter with usage, optimizes itself over time
-- **๐๏ธ Smart Storage** - OPFS, FileSystem, S3 auto-selection based on environment
-- **๐พ Massive Memory Optimization** - 75% reduction with compression, intelligent caching
-- **๐ Distributed Search** - Parallel processing with load balancing
-- **๐ Real-Time Adaptation** - Automatically adjusts to your data patterns
-- **Run Everywhere** - Works in browsers, Node.js, serverless functions, and containers
-- **Vector Search** - Find semantically similar content using embeddings
-- **Advanced JSON Document Search** - Search within specific fields of JSON documents with field prioritization and
- service-based field standardization
-- **Graph Relationships** - Connect data with meaningful relationships
-- **Streaming Pipeline** - Process data in real-time as it flows through the system
-- **Extensible Augmentations** - Customize and extend functionality with pluggable components
-- **Built-in Conduits** - Sync and scale across instances with WebSocket and WebRTC
-- **TensorFlow Integration** - Use TensorFlow.js for high-quality embeddings
-- **Persistent Storage** - Data persists across sessions and scales to any size
-- **TypeScript Support** - Fully typed API with generics
-- **CLI Tools & Web Service** - Command-line interface and REST API web service for data management
-- **Model Control Protocol (MCP)** - Allow external AI models to access Brainy data and use augmentation pipeline as
- tools
-
-## โก Large-Scale Performance Optimizations
-
-**New in v0.36.0**: Brainy now includes 6 core optimizations that transform it from a prototype into a production-ready
-system capable of handling millions of vectors:
-
-### ๐ฏ Performance Benchmarks
-
-| Dataset Size | Search Time | Memory Usage | API Calls Reduction |
-|------------------|-------------|---------------|---------------------|
-| **10k vectors** | ~50ms | Standard | N/A |
-| **100k vectors** | ~200ms | 30% reduction | 50-70% fewer |
-| **1M+ vectors** | ~500ms | 75% reduction | 50-90% fewer |
-
-### ๐ง 6 Core Optimization Systems
-
-1. **๐๏ธ Auto-Configuration System** - Detects environment, resources, and data patterns
-2. **๐ Semantic Partitioning** - Intelligent clustering with auto-tuning (4-32 clusters)
-3. **๐ Distributed Search** - Parallel processing across partitions with load balancing
-4. **๐ง Multi-Level Caching** - Hot/Warm/Cold caching with predictive prefetching
-5. **๐ฆ Batch S3 Operations** - Reduces cloud storage API calls by 50-90%
-6. **๐พ Advanced Compression** - Vector quantization and memory-mapping for large datasets
-
-### ๐ฏ Automatic Environment Detection
-
-| Environment | Auto-Configured | Performance Focus |
-|----------------|-----------------------------|----------------------------------------|
-| **Browser** | OPFS + Web Workers | Memory efficiency, 512MB-1GB limits |
-| **Node.js** | FileSystem + Worker Threads | High performance, 4GB-8GB+ usage |
-| **Serverless** | S3 + Memory cache | Cold start optimization, latency focus |
-
-### ๐ Intelligent Scaling Strategy
-
-The system automatically adapts based on your dataset size:
-
-- **< 25k vectors**: Single optimized index, no partitioning needed
-- **25k - 100k**: Semantic clustering (4-8 clusters), balanced performance
-- **100k - 1M**: Advanced partitioning (8-16 clusters), scale-optimized
-- **1M+ vectors**: Maximum optimization (16-32 clusters), enterprise-grade
-
-### ๐ง Adaptive Learning Features
-
-- **Performance Monitoring**: Tracks latency, cache hits, memory usage
-- **Dynamic Tuning**: Adjusts parameters every 50 searches based on performance
-- **Pattern Recognition**: Learns from access patterns to improve predictions
-- **Self-Optimization**: Automatically enables/disables features based on workload
-
-> **๐ Full Documentation**: See the
-> complete [Large-Scale Optimizations Guide](docs/optimization-guides/large-scale-optimizations.md) for detailed
-> configuration options and advanced usage.
-
-## ๐ Live Demo
-
-**[Try the live demo](https://soulcraft-research.github.io/brainy/demo/index.html)** - Check out the interactive demo on
-GitHub Pages that showcases Brainy's main features.
-
-## ๐ What Can You Build?
-
-- **Semantic Search Engines** - Find content based on meaning, not just keywords
-- **Recommendation Systems** - Suggest similar items based on vector similarity
-- **Knowledge Graphs** - Build connected data structures with relationships
-- **AI Applications** - Store and retrieve embeddings for machine learning models
-- **AI-Enhanced Applications** - Build applications that leverage vector embeddings for intelligent data processing
-- **Data Organization Tools** - Automatically categorize and connect related information
-- **Adaptive Experiences** - Create applications that learn and evolve with your users
-- **Model-Integrated Systems** - Connect external AI models to Brainy data and tools using MCP
-
-## ๐ง Installation
+## ๐ Quick Start (30 seconds!)
+### Node.js TLDR
```bash
-npm install @soulcraft/brainy
+# Install
+npm install brainy
+
+# Use it
```
+```javascript
+import { createAutoBrainy, NounType, VerbType } from 'brainy'
-TensorFlow.js packages are included as bundled dependencies and will be automatically installed without any additional
-configuration.
-
-### Additional Packages
-
-Brainy offers specialized packages for different use cases:
-
-#### CLI Package
-
-```bash
-npm install -g @soulcraft/brainy-cli
-```
-
-Command-line interface for data management, bulk operations, and database administration.
-
-#### Web Service Package
-
-```bash
-npm install @soulcraft/brainy-web-service
-```
-
-REST API web service wrapper that provides HTTP endpoints for search operations and database queries.
-
-## ๐ Quick Setup - Zero Configuration!
-
-**New in v0.36.0**: Brainy now automatically detects your environment and optimizes itself! Choose your scenario:
-
-### โจ Instant Setup (Auto-Everything)
-
-```typescript
-import { createAutoBrainy } from '@soulcraft/brainy'
-
-// That's it! Everything is auto-configured
const brainy = createAutoBrainy()
-// Add data and search - all optimizations enabled automatically
-await brainy.addVector({ id: '1', vector: [0.1, 0.2, 0.3], text: 'Hello world' })
-const results = await brainy.search([0.1, 0.2, 0.3], 10)
+// Add data with Nouns (entities)
+const catId = await brainy.add("Siamese cats are elegant and vocal", {
+ noun: NounType.Thing,
+ breed: "Siamese",
+ category: "animal"
+})
+
+const ownerId = await brainy.add("John loves his pets", {
+ noun: NounType.Person,
+ name: "John Smith"
+})
+
+// Connect with Verbs (relationships)
+await brainy.addVerb(ownerId, catId, {
+ verb: VerbType.Owns,
+ since: "2020-01-01"
+})
+
+// Search by meaning
+const results = await brainy.searchText("feline companions", 5)
+
+// Search JSON documents by specific fields
+const docs = await brainy.searchDocuments("Siamese", {
+ fields: ['breed', 'category'], // Search these fields
+ weights: { breed: 2.0 }, // Prioritize breed matches
+ limit: 10
+})
+
+// Find relationships
+const johnsPets = await brainy.getVerbsBySource(ownerId, VerbType.Owns)
```
-### ๐ฆ With S3 Storage (Still Auto-Configured)
+That's it! No config, no setup, it just worksโข
-```typescript
-import { createAutoBrainy } from '@soulcraft/brainy'
+## ๐ญ Key Features
-// Auto-detects AWS credentials from environment variables
+### Core Capabilities
+- **Vector Search** - Find semantically similar content using embeddings
+- **Graph Relationships** - Connect data with meaningful relationships
+- **JSON Document Search** - Search within specific fields with prioritization
+- **Real-Time Syncing** - WebSocket and WebRTC for distributed instances
+- **Streaming Pipeline** - Process data in real-time as it flows through
+- **Model Control Protocol** - Let AI models access your data
+
+### Smart Optimizations
+- **Auto-Configuration** - Detects environment and optimizes automatically
+- **Adaptive Learning** - Gets smarter with usage, optimizes itself over time
+- **Intelligent Partitioning** - Semantic clustering with auto-tuning
+- **Multi-Level Caching** - Hot/warm/cold caching with predictive prefetching
+- **Memory Optimization** - 75% reduction with compression for large datasets
+
+### Developer Experience
+- **TypeScript Support** - Fully typed API with generics
+- **Extensible Augmentations** - Customize and extend functionality
+- **REST API** - Web service wrapper for HTTP endpoints
+- **Auto-Complete** - IntelliSense for all APIs and types
+
+## ๐ฆ Installation
+
+### Main Package
+```bash
+npm install brainy
+```
+
+### Optional: Offline Models Package
+```bash
+npm install @soulcraft/brainy-models
+```
+
+The `@soulcraft/brainy-models` package provides **offline access** to the Universal Sentence Encoder model, eliminating network dependencies and ensuring consistent performance. Perfect for:
+- **Air-gapped environments** - No internet? No problem
+- **Consistent performance** - No network latency or throttling
+- **Privacy-focused apps** - Keep everything local
+- **High-reliability systems** - No external dependencies
+
+```javascript
+import { createAutoBrainy } from 'brainy'
+import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
+
+// Use the bundled model for offline operation
const brainy = createAutoBrainy({
- bucketName: 'my-vector-storage'
- // region: 'us-east-1' (default)
- // AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from env
+ embeddingModel: BundledUniversalSentenceEncoder
})
```
-### ๐ฏ Scenario-Based Setup
+## ๐จ Build Amazing Things
+
+**๐ค AI Chat Applications** - Build ChatGPT-like apps with long-term memory and context awareness
+**๐ Semantic Search Engines** - Search by meaning, not keywords. Find "that thing that's like a cat but bigger" โ returns "tiger"
+**๐ฏ Recommendation Engines** - "Users who liked this also liked..." but actually good
+**๐งฌ Knowledge Graphs** - Connect everything to everything. Wikipedia meets Neo4j meets magic
+**๐๏ธ Computer Vision Apps** - Store and search image embeddings. "Find all photos with dogs wearing hats"
+**๐ต Music Discovery** - Find songs that "feel" similar. Spotify's Discover Weekly in your app
+**๐ Smart Documentation** - Docs that answer questions. "How do I deploy to production?" โ relevant guides
+**๐ก๏ธ Fraud Detection** - Find patterns humans can't see. Anomaly detection on steroids
+**๐ Real-Time Collaboration** - Sync vector data across devices. Figma for AI data
+**๐ฅ Medical Diagnosis Tools** - Match symptoms to conditions using embedding similarity
+
+## ๐งฌ The Power of Nouns & Verbs
+
+Brainy uses a **graph-based data model** that mirrors how humans think - with **Nouns** (entities) connected by **Verbs** (relationships). This isn't just vectors in a void; it's structured, meaningful data.
+
+### ๐ Nouns (What Things Are)
+
+Nouns are your entities - the "things" in your data. Each noun has:
+- A unique ID
+- A vector representation (for similarity search)
+- A type (Person, Document, Concept, etc.)
+- Custom metadata
+
+**Available Noun Types:**
+
+| Category | Types | Use For |
+|----------|-------|---------|
+| **Core Entities** | `Person`, `Organization`, `Location`, `Thing`, `Concept`, `Event` | People, companies, places, objects, ideas, happenings |
+| **Digital Content** | `Document`, `Media`, `File`, `Message`, `Content` | PDFs, images, videos, emails, posts, generic content |
+| **Collections** | `Collection`, `Dataset` | Groups of items, structured data sets |
+| **Business** | `Product`, `Service`, `User`, `Task`, `Project` | E-commerce, SaaS, project management |
+| **Descriptive** | `Process`, `State`, `Role` | Workflows, conditions, responsibilities |
+
+### ๐ Verbs (How Things Connect)
+
+Verbs are your relationships - they give meaning to connections. Not just "these vectors are similar" but "this OWNS that" or "this CAUSES that".
+
+**Available Verb Types:**
+
+| Category | Types | Examples |
+|----------|-------|----------|
+| **Core** | `RelatedTo`, `Contains`, `PartOf`, `LocatedAt`, `References` | Generic relations, containment, location |
+| **Temporal** | `Precedes`, `Succeeds`, `Causes`, `DependsOn`, `Requires` | Time sequences, causality, dependencies |
+| **Creation** | `Creates`, `Transforms`, `Becomes`, `Modifies`, `Consumes` | Creation, change, consumption |
+| **Ownership** | `Owns`, `AttributedTo`, `CreatedBy`, `BelongsTo` | Ownership, authorship, belonging |
+| **Social** | `MemberOf`, `WorksWith`, `FriendOf`, `Follows`, `Likes`, `ReportsTo` | Social networks, organizations |
+| **Functional** | `Describes`, `Implements`, `Validates`, `Triggers`, `Serves` | Functions, implementations, services |
+
+### ๐ก Why This Matters
+
+```javascript
+// Traditional vector DB: Just similarity
+const similar = await vectorDB.search(embedding, 10)
+// Result: [vector1, vector2, ...] - What do these mean? ๐คท
+
+// Brainy: Similarity + Meaning + Relationships
+const catId = await brainy.add("Siamese cat", {
+ noun: NounType.Thing,
+ breed: "Siamese"
+})
+const ownerId = await brainy.add("John Smith", {
+ noun: NounType.Person
+})
+await brainy.addVerb(ownerId, catId, {
+ verb: VerbType.Owns,
+ since: "2020-01-01"
+})
+
+// Now you can search with context!
+const johnsPets = await brainy.getVerbsBySource(ownerId, VerbType.Owns)
+const catOwners = await brainy.getVerbsByTarget(catId, VerbType.Owns)
+```
+
+## ๐ค Why Choose Brainy?
+
+### vs. Traditional Databases
+โ **PostgreSQL with pgvector** - Requires complex setup, tuning, and DevOps expertise
+โ **Brainy** - Zero config, auto-optimizes, works everywhere from browser to cloud
+
+### vs. Vector Databases
+โ **Pinecone/Weaviate/Qdrant** - Cloud-only, expensive, vendor lock-in
+โ **Brainy** - Run locally, in browser, or cloud. Your choice, your data
+
+### vs. Graph Databases
+โ **Neo4j** - Great for graphs, no vector support
+โ **Brainy** - Vectors + graphs in one. Best of both worlds
+
+### vs. DIY Solutions
+โ **Building your own** - Months of work, optimization nightmares
+โ **Brainy** - Production-ready in 30 seconds
+
+## ๐ Getting Started in 30 Seconds
+
+### React
+
+```jsx
+import { createAutoBrainy } from 'brainy'
+import { useEffect, useState } from 'react'
+
+function SemanticSearch() {
+ const [brainy] = useState(() => createAutoBrainy())
+ const [results, setResults] = useState([])
+
+ const search = async (query) => {
+ const items = await brainy.searchText(query, 10)
+ setResults(items)
+ }
+
+ return (
+ search(e.target.value)}
+ placeholder="Search by meaning..." />
+ )
+}
+```
+
+### Angular
```typescript
-import { createQuickBrainy } from '@soulcraft/brainy'
+import { Component, OnInit } from '@angular/core'
+import { createAutoBrainy } from 'brainy'
-// Choose your scale: 'small', 'medium', 'large', 'enterprise'
-const brainy = await createQuickBrainy('large', {
- bucketName: 'my-big-vector-db'
+@Component({
+ selector: 'app-search',
+ template: `
+
+
+{/each}
+```
+
+### Next.js (App Router)
+
+```jsx
+// app/search/page.js
+import { createAutoBrainy } from 'brainy'
+
+export default function SearchPage() {
+ async function search(formData) {
+ 'use server'
+ const brainy = createAutoBrainy({ bucketName: 'vectors' })
+ const query = formData.get('query')
+ return await brainy.searchText(query, 10)
+ }
+
+ return (
+
+ )
+}
+```
+
+### Node.js / Bun / Deno
+
+```javascript
+import { createAutoBrainy } from 'brainy'
+
+const brainy = createAutoBrainy()
+
+// Add some data
+await brainy.add("TypeScript is a typed superset of JavaScript", {
+ category: 'programming'
+})
+
+// Search for similar content
+const results = await brainy.searchText("JavaScript with types", 5)
+console.log(results)
+```
+
+### Vanilla JavaScript
+
+```html
+
+
+
+
+
+
+
+
+
+
+```
+
+### Cloudflare Workers
+
+```javascript
+import { createAutoBrainy } from 'brainy'
+
+export default {
+ async fetch(request, env) {
+ const brainy = createAutoBrainy({
+ bucketName: env.R2_BUCKET
+ })
+
+ const url = new URL(request.url)
+ const query = url.searchParams.get('q')
+
+ const results = await brainy.searchText(query, 10)
+ return Response.json(results)
+ }
+}
+```
+
+### AWS Lambda
+
+```javascript
+import { createAutoBrainy } from 'brainy'
+
+export const handler = async (event) => {
+ const brainy = createAutoBrainy({
+ bucketName: process.env.S3_BUCKET
+ })
+
+ const results = await brainy.searchText(event.query, 10)
+
+ return {
+ statusCode: 200,
+ body: JSON.stringify(results)
+ }
+}
+```
+
+### Azure Functions
+
+```javascript
+import { createAutoBrainy } from 'brainy'
+
+module.exports = async function (context, req) {
+ const brainy = createAutoBrainy({
+ bucketName: process.env.AZURE_STORAGE_CONTAINER
+ })
+
+ const results = await brainy.searchText(req.query.q, 10)
+
+ context.res = {
+ body: results
+ }
+}
+```
+
+### Google Cloud Functions
+
+```javascript
+import { createAutoBrainy } from 'brainy'
+
+export const searchHandler = async (req, res) => {
+ const brainy = createAutoBrainy({
+ bucketName: process.env.GCS_BUCKET
+ })
+
+ const results = await brainy.searchText(req.query.q, 10)
+ res.json(results)
+}
+```
+
+### Google Cloud Run
+
+```dockerfile
+# Dockerfile
+FROM node:20-alpine
+USER node
+WORKDIR /app
+COPY package*.json ./
+RUN npm install brainy
+COPY . .
+CMD ["node", "server.js"]
+```
+
+```javascript
+// server.js
+import { createAutoBrainy } from 'brainy'
+import express from 'express'
+
+const app = express()
+const brainy = createAutoBrainy({
+ bucketName: process.env.GCS_BUCKET
+})
+
+app.get('/search', async (req, res) => {
+ const results = await brainy.searchText(req.query.q, 10)
+ res.json(results)
+})
+
+const port = process.env.PORT || 8080
+app.listen(port, () => console.log(`Brainy on Cloud Run: ${port}`))
+```
+
+```bash
+# Deploy to Cloud Run
+gcloud run deploy brainy-api \
+ --source . \
+ --platform managed \
+ --region us-central1 \
+ --allow-unauthenticated
+```
+
+### Vercel Edge Functions
+
+```javascript
+import { createAutoBrainy } from 'brainy'
+
+export const config = {
+ runtime: 'edge'
+}
+
+export default async function handler(request) {
+ const brainy = createAutoBrainy()
+ const { searchParams } = new URL(request.url)
+ const query = searchParams.get('q')
+
+ const results = await brainy.searchText(query, 10)
+ return Response.json(results)
+}
+```
+
+### Netlify Functions
+
+```javascript
+import { createAutoBrainy } from 'brainy'
+
+export async function handler(event, context) {
+ const brainy = createAutoBrainy()
+ const query = event.queryStringParameters.q
+
+ const results = await brainy.searchText(query, 10)
+
+ return {
+ statusCode: 200,
+ body: JSON.stringify(results)
+ }
+}
+```
+
+### Supabase Edge Functions
+
+```typescript
+import { createAutoBrainy } from 'brainy'
+import { serve } from 'https://deno.land/std@0.168.0/http/server.ts'
+
+serve(async (req) => {
+ const brainy = createAutoBrainy()
+ const url = new URL(req.url)
+ const query = url.searchParams.get('q')
+
+ const results = await brainy.searchText(query, 10)
+
+ return new Response(JSON.stringify(results), {
+ headers: { 'Content-Type': 'application/json' }
+ })
})
```
-| Scenario | Dataset Size | Memory Usage | S3 Required | Best For |
-|--------------|---------------|--------------|-----------------|----------------------|
-| `small` | โค10k vectors | โค1GB | No | Development, testing |
-| `medium` | โค100k vectors | โค4GB | Serverless only | Production apps |
-| `large` | โค1M vectors | โค8GB | Yes | Large applications |
-| `enterprise` | โค10M vectors | โค32GB | Yes | Enterprise systems |
+### Docker Container
-### ๐ง What Auto-Configuration Does
+```dockerfile
+FROM node:20-alpine
+USER node
+WORKDIR /app
+COPY package*.json ./
+RUN npm install brainy
+COPY . .
-- **๐ฏ Environment Detection**: Browser, Node.js, or Serverless
-- **๐พ Smart Memory Management**: Uses available RAM optimally
-- **๐๏ธ Storage Selection**: OPFS, FileSystem, S3, or Memory
-- **โก Performance Tuning**: Threading, caching, compression
-- **๐ Adaptive Learning**: Improves performance over time
-- **๐ Semantic Partitioning**: Auto-clusters similar vectors
+CMD ["node", "server.js"]
+```
-## ๐ Traditional Setup (Manual Configuration)
+```javascript
+// server.js
+import { createAutoBrainy } from 'brainy'
+import express from 'express'
-If you prefer manual control:
+const app = express()
+const brainy = createAutoBrainy()
-```typescript
-import { BrainyData, NounType, VerbType } from '@soulcraft/brainy'
+app.get('/search', async (req, res) => {
+ const results = await brainy.searchText(req.query.q, 10)
+ res.json(results)
+})
-// Create and initialize the database
+app.listen(3000, () => console.log('Brainy running on port 3000'))
+```
+
+### Kubernetes
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: brainy-api
+spec:
+ replicas: 3
+ template:
+ spec:
+ containers:
+ - name: brainy
+ image: your-registry/brainy-api:latest
+ env:
+ - name: S3_BUCKET
+ value: "your-vector-bucket"
+```
+
+### Railway.app
+
+```javascript
+// server.js
+import { createAutoBrainy } from 'brainy'
+
+const brainy = createAutoBrainy({
+ bucketName: process.env.RAILWAY_VOLUME_NAME
+})
+
+// Railway automatically handles the rest!
+```
+
+### Render.com
+
+```yaml
+# render.yaml
+services:
+ - type: web
+ name: brainy-api
+ env: node
+ buildCommand: npm install brainy
+ startCommand: node server.js
+ envVars:
+ - key: BRAINY_STORAGE
+ value: persistent-disk
+```
+
+## ๐ Quick Examples
+
+### Basic Usage
+
+```javascript
+import { BrainyData, NounType, VerbType } from 'brainy'
+
+// Initialize
const db = new BrainyData()
await db.init()
-// Add data (automatically converted to vectors)
+// Add data (automatically vectorized)
const catId = await db.add("Cats are independent pets", {
noun: NounType.Thing,
category: 'animal'
})
-const dogId = await db.add("Dogs are loyal companions", {
- noun: NounType.Thing,
- category: 'animal'
-})
-
// Search for similar items
-const results = await db.searchText("feline pets", 2)
-console.log(results)
+const results = await db.searchText("feline pets", 5)
-// Add a relationship between items
+// Add relationships
await db.addVerb(catId, dogId, {
verb: VerbType.RelatedTo,
- description: 'Both are common household pets'
+ description: 'Both are pets'
})
```
-### Import Options
+### AutoBrainy (Recommended)
-```typescript
-// Standard import - automatically adapts to any environment
-import { BrainyData } from '@soulcraft/brainy'
+```javascript
+import { createAutoBrainy } from 'brainy'
-// Minified version for production
-import { BrainyData } from '@soulcraft/brainy/min'
+// Everything auto-configured!
+const brainy = createAutoBrainy()
+
+// Just start using it
+await brainy.addVector({ id: '1', vector: [0.1, 0.2, 0.3], text: 'Hello' })
+const results = await brainy.search([0.1, 0.2, 0.3], 10)
```
-> **Note**: The CLI functionality is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size
-> of the main package. Install it globally with `npm install -g @soulcraft/brainy-cli` to use the command-line
-> interface.
+### Scenario-Based Setup
-### Browser Usage
+```javascript
+import { createQuickBrainy } from 'brainy'
-```html
-
-
+// Choose your scale: 'small', 'medium', 'large', 'enterprise'
+const brainy = await createQuickBrainy('large', {
+ bucketName: 'my-vector-db'
+})
```
-Modern bundlers like Webpack, Rollup, and Vite will automatically use the unified build which adapts to any environment.
+### With Offline Models
-## ๐งฉ How It Works
-
-Brainy combines **six advanced optimization systems** with core vector database technologies to create a
-production-ready, self-optimizing system:
-
-### ๐ง Core Technologies
-
-1. **Vector Embeddings** - Converts data (text, images, etc.) into numerical vectors using TensorFlow.js
-2. **Optimized HNSW Algorithm** - Fast similarity search with semantic partitioning and distributed processing
-3. **๐ง Auto-Configuration Engine** - Detects environment, resources, and data patterns to optimize automatically
-4. **๐ฏ Intelligent Storage System** - Multi-level caching with predictive prefetching and batch operations
-
-### โก Advanced Optimization Layer
-
-5. **Semantic Partitioning** - Auto-clusters similar vectors for faster search (4-32 clusters based on scale)
-6. **Distributed Search** - Parallel processing across partitions with intelligent load balancing
-7. **Multi-Level Caching** - Hot (RAM) โ Warm (Fast Storage) โ Cold (S3/Disk) with 70-90% hit rates
-8. **Batch Operations** - Reduces S3 API calls by 50-90% through intelligent batching
-9. **Adaptive Learning** - Continuously learns from usage patterns and optimizes performance
-10. **Advanced Compression** - Vector quantization achieves 75% memory reduction for large datasets
-
-### ๐ฏ Environment-Specific Optimizations
-
-| Environment | Storage | Threading | Memory | Focus |
-|----------------|-----------------|----------------|-----------|------------------|
-| **Browser** | OPFS + Cache | Web Workers | 512MB-1GB | Responsiveness |
-| **Node.js** | FileSystem + S3 | Worker Threads | 4GB-8GB+ | Throughput |
-| **Serverless** | S3 + Memory | Limited | 1GB-2GB | Cold Start Speed |
-
-### ๐ Adaptive Intelligence Flow
-
-```
-Data Input โ Auto-Detection โ Environment Optimization โ Semantic Partitioning โ
-Distributed Search โ Multi-Level Caching โ Performance Learning โ Self-Tuning
-```
-
-The system **automatically adapts** to your environment, learns from your usage patterns, and **continuously optimizes
-itself** for better performance over time.
-
-## ๐ The Brainy Pipeline
-
-Brainy's data processing pipeline transforms raw data into searchable, connected knowledge that gets smarter over time:
-
-```
-Raw Data โ Embedding โ Vector Storage โ Graph Connections โ Adaptive Learning โ Query & Retrieval
-```
-
-Each time data flows through this pipeline, Brainy learns more about your usage patterns and environment, making future
-operations faster and more relevant.
-
-### Pipeline Stages
-
-1. **Data Ingestion**
- - Raw text or pre-computed vectors enter the pipeline
- - Data is validated and prepared for processing
-
-2. **Embedding Generation**
- - Text is transformed into numerical vectors using embedding models
- - Uses TensorFlow Universal Sentence Encoder for high-quality text embeddings
- - Custom embedding functions can be plugged in for specialized domains
-
-3. **Vector Indexing**
- - Vectors are indexed using the HNSW algorithm
- - Hierarchical structure enables fast similarity search
- - Configurable parameters for precision vs. performance tradeoffs
-
-4. **Graph Construction**
- - Nouns (entities) become nodes in the knowledge graph
- - Verbs (relationships) connect related entities
- - Typed relationships add semantic meaning to connections
-
-5. **Adaptive Learning**
- - Analyzes usage patterns to optimize future operations
- - Tunes performance parameters based on your environment
- - Adjusts search strategies based on query history
- - Becomes more efficient and relevant the more you use it
-
-6. **Intelligent Storage**
- - Data is saved using the optimal storage for your environment
- - Automatic selection between OPFS, filesystem, S3, or memory
- - Migrates between storage types as your application's needs evolve
- - Scales from tiny datasets to massive data collections
- - Configurable storage adapters for custom persistence needs
-
-### Augmentation Types
-
-Brainy uses a powerful augmentation system to extend functionality. Augmentations are processed in the following order:
-
-1. **SENSE**
- - Ingests and processes raw, unstructured data into nouns and verbs
- - Handles text, images, audio streams, and other input formats
- - Example: Converting raw text into structured entities
-
-2. **MEMORY**
- - Provides storage capabilities for data in different formats
- - Manages persistence across sessions
- - Example: Storing vectors in OPFS or filesystem
-
-3. **COGNITION**
- - Enables advanced reasoning, inference, and logical operations
- - Analyzes relationships between entities
- - Examples:
- - Inferring new connections between existing data
- - Deriving insights from graph relationships
-
-4. **CONDUIT**
- - Establishes channels for structured data exchange
- - Connects with external systems and syncs between Brainy instances
- - Two built-in iConduit augmentations for scaling out and syncing:
- - **WebSocket iConduit** - Syncs data between browsers and servers
- - **WebRTC iConduit** - Direct peer-to-peer syncing between browsers
- - Examples:
- - Integrating with third-party APIs
- - Syncing Brainy instances between browsers using WebSockets
- - Peer-to-peer syncing between browsers using WebRTC
-
-5. **ACTIVATION**
- - Initiates actions, responses, or data manipulations
- - Triggers events based on data changes
- - Example: Sending notifications when new data is processed
-
-6. **PERCEPTION**
- - Interprets, contextualizes, and visualizes identified nouns and verbs
- - Creates meaningful representations of data
- - Example: Generating visualizations of graph relationships
-
-7. **DIALOG**
- - Facilitates natural language understanding and generation
- - Enables conversational interactions
- - Example: Processing user queries and generating responses
-
-8. **WEBSOCKET**
- - Enables real-time communication via WebSockets
- - Can be combined with other augmentation types
- - Example: Streaming data processing in real-time
-
-### Streaming Data Support
-
-Brainy's pipeline is designed to handle streaming data efficiently:
-
-1. **WebSocket Integration**
- - Built-in support for WebSocket connections
- - Process data as it arrives without blocking
- - Example: `setupWebSocketPipeline(url, dataType, options)`
-
-2. **Asynchronous Processing**
- - Non-blocking architecture for real-time data handling
- - Parallel processing of incoming streams
- - Example: `createWebSocketHandler(connection, dataType, options)`
-
-3. **Event-Based Architecture**
- - Augmentations can listen to data feeds and streams
- - Real-time updates propagate through the pipeline
- - Example: `listenToFeed(feedUrl, callback)`
-
-4. **Threaded Execution**
- - Comprehensive multi-threading for high-performance operations
- - Parallel processing for batch operations, vector calculations, and embedding generation
- - Configurable execution modes (SEQUENTIAL, PARALLEL, THREADED)
- - Automatic thread management based on environment capabilities
- - Example: `executeTypedPipeline(augmentations, method, args, { mode: ExecutionMode.THREADED })`
-
-### Running the Pipeline
-
-The pipeline runs automatically when you:
-
-```typescript
-// Add data (runs embedding โ indexing โ storage)
-const id = await db.add("Your text data here", { metadata })
-
-// Search (runs embedding โ similarity search)
-const results = await db.searchText("Your query here", 5)
-
-// Connect entities (runs graph construction โ storage)
-await db.addVerb(sourceId, targetId, { verb: VerbType.RelatedTo })
-```
-
-Using the CLI:
-
-```bash
-# Add data through the CLI pipeline
-brainy add "Your text data here" '{"noun":"Thing"}'
-
-# Search through the CLI pipeline
-brainy search "Your query here" --limit 5
-
-# Connect entities through the CLI
-brainy addVerb RelatedTo
-```
-
-### Extending the Pipeline
-
-Brainy's pipeline is designed for extensibility at every stage:
-
-1. **Custom Embedding**
- ```typescript
- // Create your own embedding function
- const myEmbedder = async (text) => {
- // Your custom embedding logic here
- return [0.1, 0.2, 0.3, ...] // Return a vector
- }
-
- // Use it in Brainy
- const db = new BrainyData({
- embeddingFunction: myEmbedder
- })
- ```
-
-2. **Custom Distance Functions**
- ```typescript
- // Define your own distance function
- const myDistance = (a, b) => {
- // Your custom distance calculation
- return Math.sqrt(a.reduce((sum, val, i) => sum + Math.pow(val - b[i], 2), 0))
- }
-
- // Use it in Brainy
- const db = new BrainyData({
- distanceFunction: myDistance
+```javascript
+import { createAutoBrainy } from 'brainy'
+import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
+// Use bundled model for offline operation
+const brainy = createAutoBrainy({
+ embeddingModel: BundledUniversalSentenceEncoder,
+ // Model loads from local files, no network needed!
})
- ```
-
-3. **Custom Storage Adapters**
- ```typescript
- // Implement the StorageAdapter interface
- class MyStorage implements StorageAdapter {
- // Your storage implementation
- }
-
- // Use it in Brainy
- const db = new BrainyData({
- storageAdapter: new MyStorage()
- })
- ```
-
-4. **Augmentations System**
- ```typescript
- // Create custom augmentations to extend functionality
- const myAugmentation = {
- type: 'memory',
- name: 'my-custom-storage',
- // Implementation details
- }
-
- // Register with Brainy
- db.registerAugmentation(myAugmentation)
-
+// Works exactly the same, but 100% offline
+await brainy.add("This works without internet!", {
+ noun: NounType.Content
+})
```
-## Data Model
+## ๐ Live Demo
-Brainy uses a graph-based data model with two primary concepts:
+**[Try the interactive demo](https://soulcraft-research.github.io/brainy/demo/index.html)** - See Brainy in action with animations and examples.
-### Nouns (Entities)
+## ๐ง Environment Support
-The main entities in your data (nodes in the graph):
-
-- Each noun has a unique ID, vector representation, and metadata
-- Nouns can be categorized by type (Person, Place, Thing, Event, Concept, etc.)
-- Nouns are automatically vectorized for similarity search
-
-### Verbs (Relationships)
-
-Connections between nouns (edges in the graph):
-
-- Each verb connects a source noun to a target noun
-- Verbs have types that define the relationship (RelatedTo, Controls, Contains, etc.)
-- Verbs can have their own metadata to describe the relationship
-
-### Type Utilities
-
-Brainy provides utility functions to access lists of noun and verb types:
-
-```typescript
-import {
- NounType,
- VerbType,
- getNounTypes,
- getVerbTypes,
- getNounTypeMap,
- getVerbTypeMap
-} from '@soulcraft/brainy'
-
-// At development time:
-// Access specific types directly from the NounType and VerbType objects
-console.log(NounType.Person) // 'person'
-console.log(VerbType.Contains) // 'contains'
-
-// At runtime:
-// Get a list of all noun types
-const nounTypes = getNounTypes() // ['person', 'organization', 'location', ...]
-
-// Get a list of all verb types
-const verbTypes = getVerbTypes() // ['relatedTo', 'contains', 'partOf', ...]
-
-// Get a map of noun type keys to values
-const nounTypeMap = getNounTypeMap() // { Person: 'person', Organization: 'organization', ... }
-
-// Get a map of verb type keys to values
-const verbTypeMap = getVerbTypeMap() // { RelatedTo: 'relatedTo', Contains: 'contains', ... }
-```
-
-These utility functions make it easy to:
-
-- Get a complete list of available noun and verb types
-- Validate user input against valid types
-- Create dynamic UI components that display or select from available types
-- Map between type keys and their string values
-
-## Command Line Interface
-
-Brainy includes a powerful CLI for managing your data. The CLI is available as a separate package
-`@soulcraft/brainy-cli` to reduce the bundle size of the main package.
-
-### Installing and Using the CLI
-
-```bash
-# Install the CLI globally
-npm install -g @soulcraft/brainy-cli
-
-# Initialize a database
-brainy init
-
-# Add some data
-brainy add "Cats are independent pets" '{"noun":"Thing","category":"animal"}'
-brainy add "Dogs are loyal companions" '{"noun":"Thing","category":"animal"}'
-
-# Search for similar items
-brainy search "feline pets" 5
-
-# Add relationships between items
-brainy addVerb RelatedTo '{"description":"Both are pets"}'
-
-# Visualize the graph structure
-brainy visualize
-brainy visualize --root --depth 3
-```
-
-### Using the CLI in Your Code
-
-The CLI functionality is available as a separate package `@soulcraft/brainy-cli`. If you need CLI functionality in your
-application, install the CLI package:
-
-```bash
-npm install @soulcraft/brainy-cli
-```
-
-Then you can use the CLI commands programmatically or through the command line interface.
-
-### Available Commands
-
-#### Basic Database Operations:
-
-- `init` - Initialize a new database
-- `add [metadata]` - Add a new noun with text and optional metadata
-- `search [limit]` - Search for nouns similar to the query
-- `get ` - Get a noun by ID
-- `delete ` - Delete a noun by ID
-- `addVerb [metadata]` - Add a relationship
-- `getVerbs ` - Get all relationships for a noun
-- `status` - Show database status
-- `clear` - Clear all data from the database
-- `generate-random-graph` - Generate test data
-- `visualize` - Visualize the graph structure
-- `completion-setup` - Setup shell autocomplete
-
-#### Pipeline and Augmentation Commands:
-
-- `list-augmentations` - List all available augmentation types and registered augmentations
-- `augmentation-info ` - Get detailed information about a specific augmentation type
-- `test-pipeline [text]` - Test the sequential pipeline with sample data
- - `-t, --data-type ` - Type of data to process (default: 'text')
- - `-m, --mode ` - Execution mode: sequential, parallel, threaded (default: 'sequential')
- - `-s, --stop-on-error` - Stop execution if an error occurs
- - `-v, --verbose` - Show detailed output
-- `stream-test` - Test streaming data through the pipeline (simulated)
- - `-c, --count ` - Number of data items to stream (default: 5)
- - `-i, --interval ` - Interval between data items in milliseconds (default: 1000)
- - `-t, --data-type ` - Type of data to process (default: 'text')
- - `-v, --verbose` - Show detailed output
+| Environment | Storage | Threading | Auto-Configured |
+|-------------|---------|-----------|-----------------|
+| Browser | OPFS | Web Workers | โ |
+| Node.js | FileSystem/S3 | Worker Threads | โ |
+| Serverless | Memory/S3 | Limited | โ |
+| Edge Functions | Memory/KV | Limited | โ |
## ๐ Documentation
-### ๐ [Getting Started](docs/getting-started/)
+### Getting Started
+- [**Quick Start Guide**](docs/getting-started/) - Get up and running in minutes
+- [**Installation**](docs/getting-started/installation.md) - Detailed setup instructions
+- [**Environment Setup**](docs/getting-started/environment-setup.md) - Platform-specific configuration
-Quick setup guides and first steps with Brainy.
+### User Guides
+- [**Search and Metadata**](docs/user-guides/) - Advanced search techniques
+- [**JSON Document Search**](docs/guides/json-document-search.md) - Field-based searching
+- [**Production Migration**](docs/guides/production-migration-guide.md) - Deployment best practices
-- **[Installation](docs/getting-started/installation.md)** - Installation and setup
-- **[Quick Start](docs/getting-started/quick-start.md)** - Get running in 2 minutes
-- **[First Steps](docs/getting-started/first-steps.md)** - Core concepts and features
-- **[Environment Setup](docs/getting-started/environment-setup.md)** - Environment-specific configuration
+### API Reference
+- [**Core API**](docs/api-reference/) - Complete method reference
+- [**Configuration Options**](docs/api-reference/configuration.md) - All configuration parameters
+- [**Auto-Configuration API**](docs/api-reference/auto-configuration-api.md) - Intelligent setup
-### ๐ [User Guides](docs/user-guides/)
+### Optimization & Scaling
+- [**Large-Scale Optimizations**](docs/optimization-guides/) - Handle millions of vectors
+- [**Memory Management**](docs/optimization-guides/memory-optimization.md) - Efficient resource usage
+- [**S3 Migration Guide**](docs/optimization-guides/s3-migration-guide.md) - Cloud storage setup
-Comprehensive guides for using Brainy effectively.
+### Examples & Patterns
+- [**Code Examples**](docs/examples/) - Real-world usage patterns
+- [**Integrations**](docs/examples/integrations.md) - Third-party services
+- [**Performance Patterns**](docs/examples/performance.md) - Optimization techniques
-- **[Search and Metadata](docs/user-guides/SEARCH_AND_METADATA_GUIDE.md)** - Advanced search techniques
-- **[Write-Only Mode](docs/user-guides/WRITEONLY_MODE_IMPLEMENTATION.md)** - High-throughput data loading
-- **[JSON Document Search](docs/guides/json-document-search.md)** - Search within JSON fields
-- **[Production Migration](docs/guides/production-migration-guide.md)** - Deployment best practices
+### Technical Documentation
+- [**Architecture Overview**](docs/technical/) - System design and internals
+- [**Testing Guide**](docs/technical/TESTING.md) - Testing strategies
+- [**Statistics & Monitoring**](docs/technical/STATISTICS.md) - Performance tracking
-### โก [Optimization Guides](docs/optimization-guides/)
+## ๐ค Contributing
-Transform Brainy from prototype to production-ready system.
+We welcome contributions! Please see:
+- [Contributing Guidelines](CONTRIBUTING.md)
+- [Developer Documentation](docs/development/DEVELOPERS.md)
+- [Code of Conduct](CODE_OF_CONDUCT.md)
-- **[Large-Scale Optimizations](docs/optimization-guides/large-scale-optimizations.md)** - Complete v0.36.0 optimization
- system
-- **[Auto-Configuration](docs/optimization-guides/auto-configuration.md)** - Intelligent environment detection
-- **[Memory Optimization](docs/optimization-guides/memory-optimization.md)** - Advanced memory management
-- **[Storage Optimization](docs/optimization-guides/storage-optimization.md)** - S3 and storage optimization
+## ๐ License
-### ๐ง [API Reference](docs/api-reference/)
+[MIT](LICENSE)
-Complete API documentation and method references.
+## ๐ Related Projects
-- **[Core API](docs/api-reference/core-api.md)** - Main BrainyData class methods
-- **[Vector Operations](docs/api-reference/vector-operations.md)** - Vector storage and search
-- **[Configuration](docs/api-reference/configuration.md)** - System configuration
-- **[Auto-Configuration API](docs/api-reference/auto-configuration-api.md)** - Intelligent configuration
-
-### ๐ก [Examples](docs/examples/)
-
-Practical code examples and real-world applications.
-
-- **[Basic Usage](docs/examples/basic-usage.md)** - Simple examples to get started
-- **[Advanced Patterns](docs/examples/advanced-patterns.md)** - Complex use cases
-- **[Integrations](docs/examples/integrations.md)** - Third-party service integrations
-- **[Performance Examples](docs/examples/performance.md)** - Optimization and scaling
-
-### ๐ฌ Technical Documentation
-
-- **[Testing Guide](docs/technical/TESTING.md)** - Testing strategies and best practices
-- **[Statistics Guide](STATISTICS.md)** - Database statistics and monitoring
-- **[Technical Guides](TECHNICAL_GUIDES.md)** - Advanced technical topics
-
-## API Reference
-
-### Database Management
-
-```typescript
-// Initialize the database
-await db.init()
-
-// Clear all data
-await db.clear()
-
-// Get database status
-const status = await db.status()
-
-// Backup all data from the database
-const backupData = await db.backup()
-
-// Restore data into the database
-const restoreResult = await db.restore(backupData, { clearExisting: true })
-```
-
-### Database Statistics
-
-Brainy provides a way to get statistics about the current state of the database. For detailed information about the
-statistics system, including implementation details, scalability improvements, and usage examples, see
-our [Statistics Guide](STATISTICS.md).
-
-```typescript
-import { BrainyData, getStatistics } from '@soulcraft/brainy'
-
-// Create and initialize the database
-const db = new BrainyData()
-await db.init()
-
-// Get statistics using the instance method
-const stats = await db.getStatistics()
-console.log(stats)
-// Output: { nounCount: 0, verbCount: 0, metadataCount: 0, hnswIndexSize: 0, serviceBreakdown: {...} }
-```
-
-### Working with Nouns (Entities)
-
-```typescript
-// Add a noun (automatically vectorized)
-const id = await db.add(textOrVector, {
- noun: NounType.Thing,
- // other metadata...
-})
-
-// Add multiple nouns in parallel (with multithreading and batch embedding)
-const ids = await db.addBatch([
- {
- vectorOrData: "First item to add",
- metadata: { noun: NounType.Thing, category: 'example' }
- },
- {
- vectorOrData: "Second item to add",
- metadata: { noun: NounType.Thing, category: 'example' }
- },
- // More items...
-], {
- forceEmbed: false,
- concurrency: 4, // Control the level of parallelism (default: 4)
- batchSize: 50 // Control the number of items to process in a single batch (default: 50)
-})
-
-// Retrieve a noun
-const noun = await db.get(id)
-
-// Update noun metadata
-await db.updateMetadata(id, {
- noun: NounType.Thing,
- // updated metadata...
-})
-
-// Delete a noun
-await db.delete(id)
-
-// Search for similar nouns
-const results = await db.search(vectorOrText, numResults)
-const textResults = await db.searchText("query text", numResults)
-
-// Search by noun type
-const thingNouns = await db.searchByNounTypes([NounType.Thing], numResults)
-
-// Search within specific fields of JSON documents
-const fieldResults = await db.search("Acme Corporation", 10, {
- searchField: "company"
-})
-
-// Search using standard field names across different services
-const titleResults = await db.searchByStandardField("title", "climate change", 10)
-const authorResults = await db.searchByStandardField("author", "johndoe", 10, {
- services: ["github", "reddit"]
-})
-```
-
-### Field Standardization and Service Tracking
-
-Brainy automatically tracks field names from JSON documents and associates them with the service that inserted the data.
-This enables powerful cross-service search capabilities:
-
-```typescript
-// Get all available field names organized by service
-const fieldNames = await db.getAvailableFieldNames()
-// Example output: { "github": ["repository.name", "issue.title"], "reddit": ["title", "selftext"] }
-
-// Get standard field mappings
-const standardMappings = await db.getStandardFieldMappings()
-// Example output: { "title": { "github": ["repository.name"], "reddit": ["title"] } }
-```
-
-When adding data, specify the service name to ensure proper field tracking:
-
-```typescript
-// Add data with service name
-await db.add(jsonData, metadata, { service: "github" })
-```
-
-### Working with Verbs (Relationships)
-
-```typescript
-// Add a relationship between nouns
-await db.addVerb(sourceId, targetId, {
- verb: VerbType.RelatedTo,
- // other metadata...
-})
-
-// Add a relationship with auto-creation of missing nouns
-// This is useful when the target noun might not exist yet
-await db.addVerb(sourceId, targetId, {
- verb: VerbType.RelatedTo,
- // Enable auto-creation of missing nouns
- autoCreateMissingNouns: true,
- // Optional metadata for auto-created nouns
- missingNounMetadata: {
- noun: NounType.Concept,
- description: 'Auto-created noun'
- }
-})
-
-// Get all relationships
-const verbs = await db.getAllVerbs()
-
-// Get relationships by source noun
-const outgoingVerbs = await db.getVerbsBySource(sourceId)
-
-// Get relationships by target noun
-const incomingVerbs = await db.getVerbsByTarget(targetId)
-
-// Get relationships by type
-const containsVerbs = await db.getVerbsByType(VerbType.Contains)
-
-// Get a specific relationship
-const verb = await db.getVerb(verbId)
-
-// Delete a relationship
-await db.deleteVerb(verbId)
-```
-
-## Advanced Configuration
-
-### Database Modes
-
-Brainy supports special operational modes that restrict certain operations:
-
-```typescript
-import { BrainyData } from '@soulcraft/brainy'
-
-// Create and initialize the database
-const db = new BrainyData()
-await db.init()
-
-// Set the database to read-only mode (prevents write operations)
-db.setReadOnly(true)
-
-// Check if the database is in read-only mode
-const isReadOnly = db.isReadOnly() // Returns true
-
-// Set the database to write-only mode (prevents search operations)
-db.setWriteOnly(true)
-
-// Check if the database is in write-only mode
-const isWriteOnly = db.isWriteOnly() // Returns true
-
-// Reset to normal mode (allows both read and write operations)
-db.setReadOnly(false)
-db.setWriteOnly(false)
-```
-
-- **Read-Only Mode**: When enabled, prevents all write operations (add, update, delete). Useful for deployment scenarios
- where you want to prevent modifications to the database.
-- **Write-Only Mode**: When enabled, prevents all search operations. Useful for initial data loading or when you want to
- optimize for write performance.
-
-### Embedding
-
-```typescript
-import {
- BrainyData,
- createTensorFlowEmbeddingFunction,
- createThreadedEmbeddingFunction
-} from '@soulcraft/brainy'
-
-// Use the standard TensorFlow Universal Sentence Encoder embedding function
-const db = new BrainyData({
- embeddingFunction: createTensorFlowEmbeddingFunction()
-})
-await db.init()
-
-// Or use the threaded embedding function for better performance
-const threadedDb = new BrainyData({
- embeddingFunction: createThreadedEmbeddingFunction()
-})
-await threadedDb.init()
-
-// Directly embed text to vectors
-const vector = await db.embed("Some text to convert to a vector")
-
-// Calculate similarity between two texts or vectors
-const similarity = await db.calculateSimilarity(
- "Cats are furry pets",
- "Felines make good companions"
-)
-console.log(`Similarity score: ${similarity}`) // Higher value means more similar
-
-// Calculate similarity with custom options
-const vectorA = await db.embed("First text")
-const vectorB = await db.embed("Second text")
-const customSimilarity = await db.calculateSimilarity(
- vectorA, // Can use pre-computed vectors
- vectorB,
- {
- forceEmbed: false, // Skip embedding if inputs are already vectors
- distanceFunction: cosineDistance // Optional custom distance function
- }
-)
-```
-
-The threaded embedding function runs in a separate thread (Web Worker in browsers, Worker Thread in Node.js) to improve
-performance, especially for embedding operations. It uses GPU acceleration when available (via WebGL in browsers) and
-falls back to CPU processing for compatibility. Universal Sentence Encoder is always used for embeddings. The
-implementation includes worker reuse and model caching for optimal performance.
-
-### Performance Tuning
-
-Brainy includes comprehensive performance optimizations that work across all environments (browser, CLI, Node.js,
-container, server):
-
-#### GPU and CPU Optimization
-
-Brainy uses GPU and CPU optimization for compute-intensive operations:
-
-1. **GPU-Accelerated Embeddings**: Generate text embeddings using TensorFlow.js with WebGL backend when available
-2. **Automatic Fallback**: Falls back to CPU backend when GPU is not available
-3. **Optimized Distance Calculations**: Perform vector similarity calculations with optimized algorithms
-4. **Cross-Environment Support**: Works consistently across browsers and Node.js environments
-5. **Memory Management**: Properly disposes of tensors to prevent memory leaks
-
-#### Multithreading Support
-
-Brainy includes comprehensive multithreading support to improve performance across all environments:
-
-1. **Parallel Batch Processing**: Add multiple items concurrently with controlled parallelism
-2. **Multithreaded Vector Search**: Perform distance calculations in parallel for faster search operations
-3. **Threaded Embedding Generation**: Generate embeddings in separate threads to avoid blocking the main thread
-4. **Worker Reuse**: Maintains a pool of workers to avoid the overhead of creating and terminating workers
-5. **Model Caching**: Initializes the embedding model once per worker and reuses it for multiple operations
-6. **Batch Embedding**: Processes multiple items in a single embedding operation for better performance
-7. **Automatic Environment Detection**: Adapts to browser (Web Workers) and Node.js (Worker Threads) environments
-
-```typescript
-import { BrainyData, euclideanDistance } from '@soulcraft/brainy'
-
-// Configure with custom options
-const db = new BrainyData({
- // Use Euclidean distance instead of default cosine distance
- distanceFunction: euclideanDistance,
-
- // HNSW index configuration for search performance
- hnsw: {
- M: 16, // Max connections per noun
- efConstruction: 200, // Construction candidate list size
- efSearch: 50, // Search candidate list size
- },
-
- // Performance optimization options
- performance: {
- useParallelization: true, // Enable multithreaded search operations
- },
-
- // Noun and Verb type validation
- typeValidation: {
- enforceNounTypes: true, // Validate noun types against NounType enum
- enforceVerbTypes: true, // Validate verb types against VerbType enum
- },
-
- // Storage configuration
- storage: {
- requestPersistentStorage: true,
- // Example configuration for cloud storage (replace with your own values):
- // s3Storage: {
- // bucketName: 'your-s3-bucket-name',
- // region: 'your-aws-region'
- // // Credentials should be provided via environment variables
- // // AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
- // }
- }
-})
-```
-
-### Optimized HNSW for Large Datasets
-
-Brainy includes an optimized HNSW index implementation for large datasets that may not fit entirely in memory, using a
-hybrid approach:
-
-1. **Product Quantization** - Reduces vector dimensionality while preserving similarity relationships
-2. **Disk-Based Storage** - Offloads vectors to disk when memory usage exceeds a threshold
-3. **Memory-Efficient Indexing** - Optimizes memory usage for large-scale vector collections
-
-```typescript
-import { BrainyData } from '@soulcraft/brainy'
-
-// Configure with optimized HNSW index for large datasets
-const db = new BrainyData({
- hnswOptimized: {
- // Standard HNSW parameters
- M: 16, // Max connections per noun
- efConstruction: 200, // Construction candidate list size
- efSearch: 50, // Search candidate list size
-
- // Memory threshold in bytes - when exceeded, will use disk-based approach
- memoryThreshold: 1024 * 1024 * 1024, // 1GB default threshold
-
- // Product quantization settings for dimensionality reduction
- productQuantization: {
- enabled: true, // Enable product quantization
- numSubvectors: 16, // Number of subvectors to split the vector into
- numCentroids: 256 // Number of centroids per subvector
- },
-
- // Whether to use disk-based storage for the index
- useDiskBasedIndex: true // Enable disk-based storage
- },
-
- // Storage configuration (required for disk-based index)
- storage: {
- requestPersistentStorage: true
- }
-})
-
-// The optimized index automatically adapts based on dataset size:
-// 1. For small datasets: Uses standard in-memory approach
-// 2. For medium datasets: Applies product quantization to reduce memory usage
-// 3. For large datasets: Combines product quantization with disk-based storage
-
-// Check status to see memory usage and optimization details
-const status = await db.status()
-console.log(status.details.index)
-```
-
-## Distance Functions
-
-Brainy provides several distance functions for vector similarity calculations:
-
-- `cosineDistance` (default): Measures the cosine of the angle between vectors (1 - cosine similarity)
-- `euclideanDistance`: Measures the straight-line distance between vectors
-- `manhattanDistance`: Measures the sum of absolute differences between vector components
-- `dotProductDistance`: Measures the negative dot product between vectors
-
-All distance functions are optimized for performance and automatically use the most efficient implementation based on
-the dataset size and available resources. For large datasets and high-dimensional vectors, Brainy uses batch processing
-and multithreading when available to improve performance.
-
-## Backup and Restore
-
-Brainy provides backup and restore capabilities that allow you to:
-
-- Back up your data
-- Transfer data between Brainy instances
-- Restore existing data into Brainy for vectorization and indexing
-- Backup data for analysis or visualization in other tools
-
-### Backing Up Data
-
-```typescript
-// Backup all data from the database
-const backupData = await db.backup()
-
-// The backup data includes:
-// - All nouns (entities) with their vectors and metadata
-// - All verbs (relationships) between nouns
-// - Noun types and verb types
-// - HNSW index data for fast similarity search
-// - Version information
-
-// Save the backup data to a file (Node.js environment)
-import fs from 'fs'
-
-fs.writeFileSync('brainy-backup.json', JSON.stringify(backupData, null, 2))
-```
-
-### Restoring Data
-
-Brainy's restore functionality can handle:
-
-1. Complete backups with vectors and index data
-2. Sparse data without vectors (vectors will be created during restore)
-3. Data without HNSW index (index will be reconstructed if needed)
-
-```typescript
-// Restore data with all options
-const restoreResult = await db.restore(backupData, {
- clearExisting: true // Whether to clear existing data before restore
-})
-
-// Import sparse data (without vectors)
-// Vectors will be automatically created using the embedding function
-const sparseData = {
- nouns: [
- {
- id: '123',
- // No vector field - will be created during import
- metadata: {
- noun: 'Thing',
- text: 'This text will be used to generate a vector'
- }
- }
- ],
- verbs: [],
- version: '1.0.0'
-}
-
-const sparseImportResult = await db.importSparseData(sparseData)
-```
-
-### CLI Backup/Restore
-
-```bash
-# Backup data to a file
-brainy backup --output brainy-backup.json
-
-# Restore data from a file
-brainy restore --input brainy-backup.json --clear-existing
-
-# Import sparse data (without vectors)
-brainy import-sparse --input sparse-data.json
-```
-
-## Embedding
-
-Brainy uses the following embedding approach:
-
-- TensorFlow Universal Sentence Encoder (high-quality text embeddings)
-- GPU acceleration when available (via WebGL in browsers)
-- Batch embedding for processing multiple items efficiently
-- Worker reuse and model caching for optimal performance
-- Custom embedding functions can be plugged in for specialized domains
-
-## Extensions
-
-Brainy includes an augmentation system for extending functionality:
-
-- **Memory Augmentations**: Different storage backends
-- **Sense Augmentations**: Process raw data
-- **Cognition Augmentations**: Reasoning and inference
-- **Dialog Augmentations**: Text processing and interaction
-- **Perception Augmentations**: Data interpretation and visualization
-- **Activation Augmentations**: Trigger actions
-
-### Simplified Augmentation System
-
-Brainy provides a simplified factory system for creating, importing, and executing augmentations with minimal
-boilerplate:
-
-```typescript
-import {
- createMemoryAugmentation,
- createConduitAugmentation,
- createSenseAugmentation,
- addWebSocketSupport,
- executeStreamlined,
- processStaticData,
- processStreamingData,
- createPipeline
-} from '@soulcraft/brainy'
-
-// Create a memory augmentation with minimal code
-const memoryAug = createMemoryAugmentation({
- name: 'simple-memory',
- description: 'A simple in-memory storage augmentation',
- autoRegister: true,
- autoInitialize: true,
-
- // Implement only the methods you need
- storeData: async (key, data) => {
- // Your implementation here
- return {
- success: true,
- data: true
- }
- },
-
- retrieveData: async (key) => {
- // Your implementation here
- return {
- success: true,
- data: { example: 'data', key }
- }
- }
-})
-
-// Add WebSocket support to any augmentation
-const wsAugmentation = addWebSocketSupport(memoryAug, {
- connectWebSocket: async (url) => {
- // Your implementation here
- return {
- connectionId: 'ws-1',
- url,
- status: 'connected'
- }
- }
-})
-
-// Process static data through a pipeline
-const result = await processStaticData(
- 'Input data',
- [
- {
- augmentation: senseAug,
- method: 'processRawData',
- transformArgs: (data) => [data, 'text']
- },
- {
- augmentation: memoryAug,
- method: 'storeData',
- transformArgs: (data) => ['processed-data', data]
- }
- ]
-)
-
-// Create a reusable pipeline
-const pipeline = createPipeline([
- {
- augmentation: senseAug,
- method: 'processRawData',
- transformArgs: (data) => [data, 'text']
- },
- {
- augmentation: memoryAug,
- method: 'storeData',
- transformArgs: (data) => ['processed-data', data]
- }
-])
-
-// Use the pipeline
-const result = await pipeline('New input data')
-
-// Dynamically load augmentations at runtime
-const loadedAugmentations = await loadAugmentationModule(
- import('./my-augmentations.js'),
- {
- autoRegister: true,
- autoInitialize: true
- }
-)
-```
-
-The simplified augmentation system provides:
-
-1. **Factory Functions** - Create augmentations with minimal boilerplate
-2. **WebSocket Support** - Add WebSocket capabilities to any augmentation
-3. **Streamlined Pipeline** - Process data through augmentations more efficiently
-4. **Dynamic Loading** - Load augmentations at runtime when needed
-5. **Static & Streaming Data** - Handle both static and streaming data with the same API
-
-#### WebSocket Augmentation Types
-
-Brainy exports several WebSocket augmentation types that can be used by augmentation creators to add WebSocket
-capabilities to their augmentations:
-
-```typescript
-import {
- // Base WebSocket support interface
- IWebSocketSupport,
-
- // Combined WebSocket augmentation types
- IWebSocketSenseAugmentation,
- IWebSocketConduitAugmentation,
- IWebSocketCognitionAugmentation,
- IWebSocketMemoryAugmentation,
- IWebSocketPerceptionAugmentation,
- IWebSocketDialogAugmentation,
- IWebSocketActivationAugmentation,
-
- // Function to add WebSocket support to any augmentation
- addWebSocketSupport
-} from '@soulcraft/brainy'
-
-// Example: Creating a typed WebSocket-enabled sense augmentation
-const mySenseAug = createSenseAugmentation({
- name: 'my-sense',
- processRawData: async (data, dataType) => {
- // Implementation
- return {
- success: true,
- data: { nouns: [], verbs: [] }
- }
- }
-}) as IWebSocketSenseAugmentation
-
-// Add WebSocket support
-addWebSocketSupport(mySenseAug, {
- connectWebSocket: async (url) => {
- // WebSocket implementation
- return {
- connectionId: 'ws-1',
- url,
- status: 'connected'
- }
- },
- sendWebSocketMessage: async (connectionId, data) => {
- // Send message implementation
- },
- onWebSocketMessage: async (connectionId, callback) => {
- // Register callback implementation
- },
- offWebSocketMessage: async (connectionId, callback) => {
- // Remove callback implementation
- },
- closeWebSocket: async (connectionId, code, reason) => {
- // Close connection implementation
- }
-})
-
-// Now mySenseAug has both sense augmentation methods and WebSocket methods
-await mySenseAug.processRawData('data', 'text')
-await mySenseAug.connectWebSocket('wss://example.com')
-```
-
-These WebSocket augmentation types combine the base augmentation interfaces with the `IWebSocketSupport` interface,
-providing type safety and autocompletion for augmentations with WebSocket capabilities.
-
-### Model Control Protocol (MCP)
-
-Brainy includes a Model Control Protocol (MCP) implementation that allows external models to access Brainy data and use
-the augmentation pipeline as tools:
-
-- **BrainyMCPAdapter**: Provides access to Brainy data through MCP
-- **MCPAugmentationToolset**: Exposes the augmentation pipeline as tools
-- **BrainyMCPService**: Integrates the adapter and toolset, providing WebSocket and REST server implementations
-
-Environment compatibility:
-
-- **BrainyMCPAdapter** and **MCPAugmentationToolset** can run in any environment (browser, Node.js, server)
-- **BrainyMCPService** core functionality works in any environment
-
-For detailed documentation and usage examples, see the [MCP documentation](src/mcp/README.md).
-
-## Cross-Environment Compatibility
-
-Brainy is designed to run seamlessly in any environment, from browsers to Node.js to serverless functions and
-containers. All Brainy data, functions, and augmentations are environment-agnostic, allowing you to use the same code
-everywhere.
-
-### Environment Detection
-
-Brainy automatically detects the environment it's running in:
-
-```typescript
-import { environment } from '@soulcraft/brainy'
-
-// Check which environment we're running in
-console.log(`Running in ${
- environment.isBrowser ? 'browser' :
- environment.isNode ? 'Node.js' :
- 'serverless/unknown'
-} environment`)
-```
-
-### Adaptive Storage
-
-Storage adapters are automatically selected based on the environment:
-
-- **Browser**: Uses Origin Private File System (OPFS) when available, falls back to in-memory storage
-- **Node.js**: Uses file system storage by default, with options for S3-compatible cloud storage
-- **Serverless**: Uses in-memory storage with options for cloud persistence
-- **Container**: Automatically detects and uses the appropriate storage based on available capabilities
-
-### Dynamic Imports
-
-Brainy uses dynamic imports to load environment-specific dependencies only when needed, keeping the bundle size small
-and ensuring compatibility across environments.
-
-### Browser Support
-
-Works in all modern browsers:
-
-- Chrome 86+
-- Edge 86+
-- Opera 72+
-- Chrome for Android 86+
-
-For browsers without OPFS support, falls back to in-memory storage.
-
-## Related Projects
-
-- **[Cartographer](https://github.com/sodal-project/cartographer)** - A companion project that provides standardized
- interfaces for interacting with Brainy
-
-## Demo
-
-The repository includes a comprehensive demo that showcases Brainy's main features:
-
-- `demo/index.html` - A single demo page with animations demonstrating Brainy's features.
- - **[Try the live demo](https://soulcraft-research.github.io/brainy/demo/index.html)** - Check out the
- interactive demo on
- GitHub Pages
- - Or run it locally with `npm run demo` (see [demo instructions](demo.md) for details)
- - To deploy your own version to GitHub Pages, use the GitHub Actions workflow in
- `.github/workflows/deploy-demo.yml`,
- which automatically deploys when pushing to the main branch or can be manually triggered
- - To use a custom domain (like www.soulcraft.com):
- 1. A CNAME file is already included in the demo directory
- 2. In your GitHub repository settings, go to Pages > Custom domain and enter your domain
- 3. Configure your domain's DNS settings to point to GitHub Pages:
-
- - Add a CNAME record for www pointing to `.github.io` (e.g., `soulcraft-research.github.io`)
- - Or for an apex domain (soulcraft.com), add A records pointing to GitHub Pages IP addresses
-
-The demo showcases:
-
-- How Brainy runs in different environments (browser, Node.js, server, cloud)
-- How the noun-verb data model works
-- How HNSW search works
-
-## Syncing Brainy Instances
-
-You can use the conduit augmentations to sync Brainy instances:
-
-- **WebSocket iConduit**: For syncing between browsers and servers, or between servers. WebSockets cannot be used for
- direct browser-to-browser communication without a server in the middle.
-- **WebRTC iConduit**: For direct peer-to-peer syncing between browsers. This is the recommended approach for
- browser-to-browser communication.
-
-#### WebSocket Sync Example
-
-```typescript
-import {
- BrainyData,
- pipeline,
- createConduitAugmentation
-} from '@soulcraft/brainy'
-
-// Create and initialize the database
-const db = new BrainyData()
-await db.init()
-
-// Create a WebSocket conduit augmentation
-const wsConduit = await createConduitAugmentation('websocket', 'my-websocket-sync')
-
-// Register the augmentation with the pipeline
-pipeline.register(wsConduit)
-
-// Connect to another Brainy instance (server or browser)
-// Replace the example URL below with your actual WebSocket server URL
-const connectionResult = await pipeline.executeConduitPipeline(
- 'establishConnection',
- ['wss://example-websocket-server.com/brainy-sync', { protocols: 'brainy-sync' }]
-)
-
-if (connectionResult[0] && (await connectionResult[0]).success) {
- const connection = (await connectionResult[0]).data
-
- // Read data from the remote instance
- const readResult = await pipeline.executeConduitPipeline(
- 'readData',
- [{ connectionId: connection.connectionId, query: { type: 'getAllNouns' } }]
- )
-
- // Process and add the received data to the local instance
- if (readResult[0] && (await readResult[0]).success) {
- const remoteNouns = (await readResult[0]).data
- for (const noun of remoteNouns) {
- await db.add(noun.vector, noun.metadata)
- }
- }
-
- // Set up real-time sync by monitoring the stream
- await wsConduit.monitorStream(connection.connectionId, async (data) => {
- // Handle incoming data (e.g., new nouns, verbs, updates)
- if (data.type === 'newNoun') {
- await db.add(data.vector, data.metadata)
- } else if (data.type === 'newVerb') {
- await db.addVerb(data.sourceId, data.targetId, data.vector, data.options)
- }
- })
-}
-```
-
-#### WebRTC Peer-to-Peer Sync Example
-
-```typescript
-import {
- BrainyData,
- pipeline,
- createConduitAugmentation
-} from '@soulcraft/brainy'
-
-// Create and initialize the database
-const db = new BrainyData()
-await db.init()
-
-// Create a WebRTC conduit augmentation
-const webrtcConduit = await createConduitAugmentation('webrtc', 'my-webrtc-sync')
-
-// Register the augmentation with the pipeline
-pipeline.register(webrtcConduit)
-
-// Connect to a peer using a signaling server
-// Replace the example values below with your actual configuration
-const connectionResult = await pipeline.executeConduitPipeline(
- 'establishConnection',
- [
- 'peer-id-to-connect-to', // Replace with actual peer ID
- {
- signalServerUrl: 'wss://example-signal-server.com', // Replace with your signal server
- localPeerId: 'my-local-peer-id', // Replace with your local peer ID
- iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] // Public STUN server
- }
- ]
-)
-
-if (connectionResult[0] && (await connectionResult[0]).success) {
- const connection = (await connectionResult[0]).data
-
- // Set up real-time sync by monitoring the stream
- await webrtcConduit.monitorStream(connection.connectionId, async (data) => {
- // Handle incoming data (e.g., new nouns, verbs, updates)
- if (data.type === 'newNoun') {
- await db.add(data.vector, data.metadata)
- } else if (data.type === 'newVerb') {
- await db.addVerb(data.sourceId, data.targetId, data.vector, data.options)
- }
- })
-
- // When adding new data locally, also send to the peer
- const nounId = await db.add("New data to sync", { noun: "Thing" })
-
- // Send the new noun to the peer
- await pipeline.executeConduitPipeline(
- 'writeData',
- [
- {
- connectionId: connection.connectionId,
- data: {
- type: 'newNoun',
- id: nounId,
- vector: (await db.get(nounId)).vector,
- metadata: (await db.get(nounId)).metadata
- }
- }
- ]
- )
-}
-```
-
-#### Browser-Server Search Example
-
-Brainy supports searching a server-hosted instance from a browser, storing results locally, and performing further
-searches against the local instance:
-
-```typescript
-import { BrainyData } from '@soulcraft/brainy'
-
-// Create and initialize the database with remote server configuration
-// Replace the example URL below with your actual Brainy server URL
-const db = new BrainyData({
- remoteServer: {
- url: 'wss://example-brainy-server.com/ws', // Replace with your server URL
- protocols: 'brainy-sync',
- autoConnect: true // Connect automatically during initialization
- }
-})
-await db.init()
-
-// Or connect manually after initialization
-if (!db.isConnectedToRemoteServer()) {
- // Replace the example URL below with your actual Brainy server URL
- await db.connectToRemoteServer('wss://example-brainy-server.com/ws', 'brainy-sync')
-}
-
-// Search the remote server (results are stored locally)
-const remoteResults = await db.searchText('machine learning', 5, { searchMode: 'remote' })
-
-// Search the local database (includes previously stored results)
-const localResults = await db.searchText('machine learning', 5, { searchMode: 'local' })
-
-// Perform a combined search (local first, then remote if needed)
-const combinedResults = await db.searchText('neural networks', 5, { searchMode: 'combined' })
-
-// Add data to both local and remote instances
-const id = await db.addToBoth('Deep learning is a subset of machine learning', {
- noun: 'Concept',
- category: 'AI',
- tags: ['deep learning', 'neural networks']
-})
-
-// Clean up when done (this also cleans up worker pools)
-await db.shutDown()
-```
+- [**Cartographer**](https://github.com/sodal-project/cartographer) - Standardized interfaces for Brainy
---
-## ๐ Scaling Strategy
-
-Brainy is designed to handle datasets of various sizes, from small collections to large-scale deployments. For
-terabyte-scale data that can't fit entirely in memory, we provide several approaches:
-
-- **Disk-Based HNSW**: Modified implementations using intelligent caching and partial loading
-- **Distributed HNSW**: Sharding and partitioning across multiple machines
-- **Hybrid Solutions**: Combining quantization techniques with multi-tier architectures
-
-For detailed information on how to scale Brainy for large datasets, vector dimension standardization, threading
-implementation, storage testing, and other technical topics, see our
-comprehensive [Technical Guides](TECHNICAL_GUIDES.md).
-
-## Recent Changes and Performance Improvements
-
-### Enhanced Memory Management and Scalability
-
-Brainy has been significantly improved to handle larger datasets more efficiently:
-
-- **Pagination Support**: All data retrieval methods now support pagination to avoid loading entire datasets into memory
- at once. The deprecated `getAllNouns()` and `getAllVerbs()` methods have been replaced with `getNouns()` and
- `getVerbs()` methods that support pagination, filtering, and cursor-based navigation.
-
-- **Multi-level Caching**: A sophisticated three-level caching strategy has been implemented:
- - **Level 1**: Hot cache (most accessed nodes) - RAM (automatically detecting and adjusting in each environment)
- - **Level 2**: Warm cache (recent nodes) - OPFS, Filesystem or S3 depending on environment
- - **Level 3**: Cold storage (all nodes) - OPFS, Filesystem or S3 depending on environment
-
-- **Adaptive Memory Usage**: The system automatically detects available memory and adjusts cache sizes accordingly:
- - In Node.js: Uses 10% of free memory (minimum 1000 entries)
- - In browsers: Scales based on device memory (500 entries per GB, minimum 1000)
-
-- **Intelligent Cache Eviction**: Implements a Least Recently Used (LRU) policy that evicts the oldest 20% of items when
- the cache reaches the configured threshold.
-
-- **Prefetching Strategy**: Implements batch prefetching to improve performance while avoiding overwhelming system
- resources.
-
-### S3-Compatible Storage Improvements
-
-- **Enhanced Cloud Storage**: Improved support for S3-compatible storage services including AWS S3, Cloudflare R2, and
- others.
-
-- **Optimized Data Access**: Batch operations and error handling for efficient cloud storage access.
-
-- **Change Log Management**: Efficient synchronization through change logs to track updates.
-
-### Data Compatibility
-
-Yes, you can use existing data indexed from an old version. Brainy includes robust data migration capabilities:
-
-- **Vector Regeneration**: If vectors are missing in imported data, they will be automatically created using the
- embedding function.
-
-- **HNSW Index Reconstruction**: The system can reconstruct the HNSW index from backup data, ensuring compatibility with
- previous versions.
-
-- **Sparse Data Import**: Support for importing sparse data (without vectors) through the `importSparseData()` method.
-
-### System Requirements
-
-#### Default Mode
-
-- **Memory**:
- - Minimum: 512MB RAM
- - Recommended: 2GB+ RAM for medium datasets, 8GB+ for large datasets
-
-- **CPU**:
- - Minimum: 2 cores
- - Recommended: 4+ cores for better performance with parallel operations
-
-- **Storage**:
- - Minimum: 1GB available storage
- - Recommended: Storage space at least 3x the size of your dataset
-
-#### Read-Only Mode
-
-Read-only mode prevents all write operations (add, update, delete) and is optimized for search operations.
-
-- **Memory**:
- - Minimum: 256MB RAM
- - Recommended: 1GB+ RAM
-
-- **CPU**:
- - Minimum: 1 core
- - Recommended: 2+ cores
-
-- **Storage**:
- - Minimum: Storage space equal to the size of your dataset
- - Recommended: 2x the size of your dataset for caching
-
-- **New Feature**: Lazy loading support in read-only mode for improved performance with large datasets.
-
-#### Write-Only Mode
-
-Write-only mode prevents all search operations and is optimized for initial data loading or when you want to optimize
-for write performance.
-
-- **Memory**:
- - Minimum: 512MB RAM
- - Recommended: 2GB+ RAM
-
-- **CPU**:
- - Minimum: 2 cores
- - Recommended: 4+ cores for faster data ingestion
-
-- **Storage**:
- - Minimum: Storage space at least 2x the size of your dataset
- - Recommended: 4x the size of your dataset for optimal performance
-
-### Performance Tuning Parameters
-
-Brainy offers comprehensive configuration options for performance tuning, with enhanced support for large datasets in S3
-or other remote storage. **All configuration is optional** - the system automatically detects the optimal settings based
-on your environment, dataset size, and usage patterns.
-
-#### Intelligent Defaults
-
-Brainy uses intelligent defaults that automatically adapt to your environment:
-
-- **Environment Detection**: Automatically detects whether you're running in Node.js, browser, or worker environment
-- **Memory-Aware Caching**: Adjusts cache sizes based on available system memory
-- **Dataset Size Adaptation**: Tunes parameters based on the size of your dataset
-- **Usage Pattern Optimization**: Adjusts to read-heavy vs. write-heavy workloads
-- **Storage Type Awareness**: Optimizes for local vs. remote storage (S3, R2, etc.)
-- **Operating Mode Specialization**: Special optimizations for read-only and write-only modes
-
-#### Cache Configuration (Optional)
-
-You can override any of these automatically tuned parameters if needed:
-
-- **Hot Cache Size**: Control the maximum number of items to keep in memory.
- - For large datasets (>100K items), consider values between 5,000-50,000 depending on available memory.
- - In read-only mode, larger values (10,000-100,000) can be used for better performance.
-
-- **Eviction Threshold**: Set the threshold at which cache eviction begins (default: 0.8 or 80% of max size).
- - For write-heavy workloads, lower values (0.6-0.7) may improve performance.
- - For read-heavy workloads, higher values (0.8-0.9) are recommended.
-
-- **Warm Cache TTL**: Set the time-to-live for items in the warm cache (default: 3600000 ms or 1 hour).
- - For frequently changing data, shorter TTLs are recommended.
- - For relatively static data, longer TTLs improve performance.
-
-- **Batch Size**: Control the number of items to process in a single batch for operations like prefetching.
- - For S3 or remote storage with large datasets, larger values (50-200) significantly improve throughput.
- - In read-only mode with remote storage, even larger values (100-300) can be used.
-
-#### Auto-Tuning (Enabled by Default)
-
-- **Auto-Tune**: Enable or disable automatic tuning of cache parameters based on usage patterns (default: true).
-- **Auto-Tune Interval**: Set how frequently the system adjusts cache parameters (default: 60000 ms or 1 minute).
-
-#### Read-Only Mode Optimizations (Automatic)
-
-Read-only mode includes special optimizations for search performance that are automatically applied:
-
-- **Larger Cache Sizes**: Automatically uses more memory for caching (up to 40% of free memory for large datasets).
-- **Aggressive Prefetching**: Loads more data in each batch to reduce the number of storage requests.
-- **Prefetch Strategy**: Defaults to 'aggressive' prefetching strategy in read-only mode.
-
-#### Example Configuration for Large S3 Datasets
-
-```javascript
-const brainy = new BrainyData({
- readOnly: true,
- lazyLoadInReadOnlyMode: true,
- storage: {
- type: 's3',
- s3Storage: {
- bucketName: 'your-bucket',
- accessKeyId: 'your-access-key',
- secretAccessKey: 'your-secret-key',
- region: 'your-region'
- }
- },
- cache: {
- hotCacheMaxSize: 20000,
- hotCacheEvictionThreshold: 0.85,
- batchSize: 100,
- readOnlyMode: {
- hotCacheMaxSize: 50000,
- batchSize: 200,
- prefetchStrategy: 'aggressive'
- }
- }
-});
-```
-
-These configuration options make Brainy more efficient, scalable, and adaptable to different environments and usage
-patterns, especially for large datasets in cloud storage.
-
-## Testing
-
-Brainy uses Vitest for testing. For detailed information about testing in Brainy, including test configuration, scripts,
-reporting tools, and best practices, see our [Testing Guide](docs/technical/TESTING.md).
-
-Here are some common test commands:
-
-```bash
-# Run all tests
-npm test
-
-# Run tests with comprehensive reporting
-npm run test:report
-
-# Run tests with coverage
-npm run test:coverage
-```
-
-## Contributing
-
-For detailed contribution guidelines, please see [CONTRIBUTING.md](CONTRIBUTING.md).
-
-For developer documentation, including building, testing, and publishing instructions, please
-see [DEVELOPERS.md](DEVELOPERS.md).
-
-We have a [Code of Conduct](CODE_OF_CONDUCT.md) that all contributors are expected to follow.
-
-### Commit Message Format
-
-For best results with automatic changelog generation, follow
-the [Conventional Commits](https://www.conventionalcommits.org/) specification for your commit messages:
-
-```
-AI Template for automated commit messages:
-
-Use Conventional Commit format
-Specify the changes in a structured format
-Add information about the purpose of the commit
-```
-
-```
-():
-
-[optional body]
-
-[optional footer(s)]
-```
-
-Where `` is one of:
-
-- `feat`: A new feature (maps to **Added** section)
-- `fix`: A bug fix (maps to **Fixed** section)
-- `chore`: Regular maintenance tasks (maps to **Changed** section)
-- `docs`: Documentation changes (maps to **Documentation** section)
-- `refactor`: Code changes that neither fix bugs nor add features (maps to **Changed** section)
-- `perf`: Performance improvements (maps to **Changed** section)
-
-### Manual Release Process
-
-If you need more control over the release process, you can use the individual commands:
-
-```bash
-# Update version and generate changelog
-npm run _release:patch # or _release:minor, _release:major
-
-# Create GitHub release
-npm run _github-release
-
-# Publish to NPM
-npm publish
-```
-
-## License
-
-[MIT](LICENSE)
+
+Ready to build something amazing? Get started with Brainy today!
+
diff --git a/docs/tools/claude-commit/README.md b/docs/tools/claude-commit/README.md
new file mode 100644
index 00000000..06f5ef05
--- /dev/null
+++ b/docs/tools/claude-commit/README.md
@@ -0,0 +1,128 @@
+# Claude Commit - AI-Powered Git Commits
+
+Automatically generate Conventional Commit messages using Claude AI by analyzing your git diff.
+
+## ๐ Quick Setup
+
+### One-Line Install (Simplest)
+
+On any computer, just run:
+
+```bash
+# Download and run the setup script
+curl -sSL https://raw.githubusercontent.com/soulcraft-research/brainy/main/docs/tools/claude-commit/setup.sh | bash
+```
+
+### Manual Install
+
+If you have the brainy repo cloned:
+
+```bash
+cd ~/Projects/brainy/docs/tools/claude-commit
+./setup.sh
+```
+
+### Using Dotfiles (For Multiple Machines)
+
+1. **First time:** Create your dotfiles repository
+ ```bash
+ # Copy the dotfiles folder from brainy to a new repo
+ cp -r ~/Projects/brainy/docs/tools/claude-commit/dotfiles ~/Projects/my-dotfiles
+ cd ~/Projects/my-dotfiles
+ git init
+ git add .
+ git commit -m "feat: add claude-commit dotfiles"
+ # Create a GitHub repo and push
+ ```
+
+2. **On any new computer:**
+ ```bash
+ # One-line install from your dotfiles
+ curl -sSL https://raw.githubusercontent.com/yourusername/my-dotfiles/main/install.sh | bash
+ ```
+
+## ๐ What Gets Installed
+
+- `~/.local/bin/claude-commit` - The main script
+- Git aliases: `git cc` and `git smart-commit`
+- PATH update (if needed)
+
+## ๐ Keeping Computers in Sync
+
+### Manual Sync
+
+On any computer, update to latest version:
+
+```bash
+# From brainy repo
+curl -sSL https://raw.githubusercontent.com/yourusername/brainy/main/dotfiles/bin/claude-commit > ~/.local/bin/claude-commit
+chmod +x ~/.local/bin/claude-commit
+```
+
+### Automatic Sync with Dotfiles
+
+If you set up a dotfiles repo:
+
+```bash
+cd ~/Projects/my-dotfiles
+git pull
+./install.sh
+```
+
+## ๐ป Per-Computer Setup Checklist
+
+When setting up a new computer:
+
+- [ ] Install Claude CLI (`claude` command must work)
+- [ ] Run the setup script
+- [ ] Restart terminal or `source ~/.bashrc`
+- [ ] Test with `git cc` in any repo
+
+## ๐งช Testing
+
+After setup, test in any git repository:
+
+```bash
+# Make a change to any file
+echo "test" >> README.md
+
+# Use claude to commit
+git cc
+```
+
+## ๐ง Troubleshooting
+
+### "command not found: claude"
+- Install Claude CLI from https://claude.ai/code
+
+### "command not found: git cc"
+- Run: `source ~/.bashrc` (or `~/.zshrc` for Zsh)
+- Verify: `ls -la ~/.local/bin/claude-commit`
+
+### Script not generating commits
+- Check Claude CLI works: `claude --version`
+- Ensure you have changes: `git status`
+
+## ๐ Files in This Setup
+
+```
+brainy/
+โโโ setup-claude-commit.sh # Standalone installer
+โโโ CLAUDE_COMMIT_SETUP.md # This file
+โโโ dotfiles/ # Portable dotfiles
+ โโโ README.md # Dotfiles documentation
+ โโโ install.sh # Master installer
+ โโโ setup-claude-commit.sh # Claude commit installer
+ โโโ bin/
+ โโโ claude-commit # The actual script
+```
+
+## ๐ฏ Summary
+
+**For your laptop**, you have three options:
+
+1. **Quickest:** Run the setup script from brainy repo
+2. **Portable:** Create a dotfiles repo and install from there
+3. **Manual:** Copy the script directly to `~/.local/bin/`
+
+All methods give you the same `git cc` command that works in any git repository!
\ No newline at end of file
diff --git a/docs/tools/claude-commit/dotfiles/README.md b/docs/tools/claude-commit/dotfiles/README.md
new file mode 100644
index 00000000..4dcbbdad
--- /dev/null
+++ b/docs/tools/claude-commit/dotfiles/README.md
@@ -0,0 +1,121 @@
+# Dotfiles - Claude Commit Setup
+
+This repository contains my development environment configuration, including the Claude AI-powered git commit message generator.
+
+## Quick Setup
+
+### Option 1: One-Line Install (Recommended)
+
+```bash
+curl -sSL https://raw.githubusercontent.com/yourusername/dotfiles/main/install.sh | bash
+```
+
+### Option 2: Manual Install
+
+```bash
+# Clone the repository
+git clone https://github.com/yourusername/dotfiles.git ~/dotfiles
+
+# Run the setup script
+cd ~/dotfiles
+./install.sh
+```
+
+### Option 3: Just Claude Commit
+
+If you only want the `git cc` command:
+
+```bash
+# Download and run the setup script
+curl -sSL https://raw.githubusercontent.com/yourusername/dotfiles/main/setup-claude-commit.sh | bash
+```
+
+## What's Included
+
+### Claude Commit (`git cc`)
+
+An AI-powered git commit message generator that:
+- Analyzes your git diff
+- Generates Conventional Commit formatted messages
+- Works in any git repository
+- No configuration needed
+
+**Usage:**
+```bash
+# Make your changes
+git cc # Claude generates the commit message
+```
+
+## Syncing Between Computers
+
+### First Time Setup (on new computer)
+
+1. Run the install script (see Quick Setup above)
+2. That's it! The `git cc` command is now available
+
+### Keeping in Sync
+
+To update to the latest version on any computer:
+
+```bash
+sync-claude-commit
+```
+
+Or manually:
+```bash
+curl -sSL https://raw.githubusercontent.com/yourusername/dotfiles/main/bin/claude-commit > ~/.local/bin/claude-commit
+chmod +x ~/.local/bin/claude-commit
+```
+
+## File Structure
+
+```
+dotfiles/
+โโโ README.md # This file
+โโโ install.sh # Main installation script
+โโโ setup-claude-commit.sh # Standalone claude-commit installer
+โโโ bin/
+ โโโ claude-commit # The actual claude-commit script
+```
+
+## Requirements
+
+- Git
+- Claude CLI (`claude` command)
+- Bash or Zsh
+
+## Conventional Commit Format
+
+The tool generates messages following this format:
+
+```
+():
+
+
+
+