diff --git a/dev/README.md b/dev/README.md deleted file mode 100644 index ff6dfb7c..00000000 --- a/dev/README.md +++ /dev/null @@ -1,53 +0,0 @@ -# Development Tools & Documentation - -This directory contains development tools, scripts, and documentation files that are not included in the published npm package. - -## Directory Structure - -``` -dev/ -├── docs/ # Development documentation files -│ ├── brainy_architecture_diagram.md -│ ├── PDF_GENERATION_GUIDE.md -│ ├── QUICK_PDF_SETUP.md -│ └── brainy_architecture_visual.md -├── scripts/ # Development scripts -│ └── generate-architecture-pdf.js -└── README.md # This file -``` - -## Scripts - -### generate-architecture-pdf.js - -Generates a professional PDF documentation of Brainy's architecture using: -- Material Design styling -- Custom SVG diagrams -- Comprehensive content from README.md -- Professional visual presentation - -**Usage:** -```bash -# From project root -node dev/scripts/generate-architecture-pdf.js - -# Or add to package.json scripts -npm run generate-docs -``` - -**Requirements:** -- puppeteer (for PDF generation) - -**Output:** -- `docs/Brainy_Architecture_Documentation.pdf` - -## Documentation Files - -- **brainy_architecture_diagram.md**: ASCII art diagrams of system architecture -- **PDF_GENERATION_GUIDE.md**: Detailed guide for PDF generation setup -- **QUICK_PDF_SETUP.md**: Quick setup instructions -- **brainy_architecture_visual.md**: Visual architecture documentation - -## NPM Package Exclusion - -This entire `dev/` directory is excluded from the published npm package via `.npmignore` to keep the package size minimal and focused on the core library functionality. \ No newline at end of file diff --git a/dev/docs/PDF_GENERATION_GUIDE.md b/dev/docs/PDF_GENERATION_GUIDE.md deleted file mode 100644 index 4e2c2b30..00000000 --- a/dev/docs/PDF_GENERATION_GUIDE.md +++ /dev/null @@ -1,242 +0,0 @@ -# Brainy Architecture PDF Generation Guide - -This guide shows you how to generate a professional PDF from the Brainy architecture documentation with beautiful diagrams. - -## Quick Start - -### Option 1: Using the npm script (Recommended) - -```bash -# Make sure you're in the brainy project directory -cd /path/to/brainy - -# Install dependencies if not already installed -npm install - -# Generate the PDF -npm run generate-pdf -``` - -### Option 2: Direct script execution - -```bash -# Make sure you're in the brainy project directory -cd /path/to/brainy - -# Install Puppeteer if not already installed -npm install puppeteer - -# Run the script directly -node dev/dev/scripts/generate-architecture-pdf.js -``` - -## Installation Requirements - -### Prerequisites -- Node.js 18+ -- npm or yarn - -### Dependencies -The script uses: -- **Puppeteer**: For PDF generation and browser automation -- **Mermaid**: For rendering diagrams (loaded via CDN) -- **Google Fonts**: For professional typography (loaded via CDN) - -### Install Dependencies - -```bash -# If you don't have puppeteer installed globally or in the project -npm install puppeteer - -# Or install as dev dependency -npm install --save-dev puppeteer -``` - -## Output - -The PDF will be generated at: -``` -docs/Brainy_Architecture_Documentation.pdf -``` - -## Features of the Generated PDF - -### Professional Styling -- **Modern Typography**: Uses Inter font family for clean, readable text -- **Code Font**: JetBrains Mono for code blocks and technical content -- **Color Scheme**: Professional blue theme with proper contrast -- **Layout**: A4 format with proper margins and spacing - -### Rich Diagrams -- **Mermaid Diagrams**: All diagrams are rendered as vector graphics -- **Interactive Elements**: Flowcharts, sequence diagrams, mindmaps, and more -- **Consistent Styling**: All diagrams follow the same color scheme -- **High Quality**: Vector-based rendering for crisp output - -### Document Structure -- **Table of Contents**: Linked navigation -- **Page Headers/Footers**: Professional branding and page numbers -- **Section Breaks**: Logical page breaks between major sections -- **Code Highlighting**: Syntax highlighting for JSON and code blocks - -## Customization - -### Modify Styling -Edit the `professionalCSS` variable in `dev/scripts/generate-architecture-pdf.js`: - -```javascript -const professionalCSS = ` - /* Your custom CSS here */ - h1 { - color: #your-color; - font-size: 24pt; - } - /* ... */ -` -``` - -### Change Output Location -Modify the `config` object: - -```javascript -const config = { - inputFile: path.join(__dirname, '../docs/brainy_architecture_visual.md'), - outputFile: path.join(__dirname, '../docs/YOUR_CUSTOM_NAME.pdf'), - // ... -} -``` - -### Adjust PDF Settings -Modify the `page.pdf()` options: - -```javascript -await page.pdf({ - path: config.outputFile, - format: 'A4', // or 'Letter', 'Legal', etc. - printBackground: true, - margin: { - top: '20mm', - right: '15mm', - bottom: '20mm', - left: '15mm' - }, - // ... other options -}) -``` - -## Troubleshooting - -### Common Issues - -#### 1. "Puppeteer not found" -```bash -npm install puppeteer -``` - -#### 2. "Chrome/Chromium not found" -```bash -# On Ubuntu/Debian -sudo apt-get install chromium-browser - -# On macOS -brew install chromium - -# Or let Puppeteer download Chromium -npm install puppeteer --unsafe-perm=true -``` - -#### 3. "Permission denied" -```bash -chmod +x dev/scripts/generate-architecture-pdf.js -``` - -#### 4. "Diagrams not rendering" -Check your internet connection - Mermaid is loaded from CDN. For offline use, you can download mermaid.min.js locally and update the path. - -### Advanced Configuration - -#### Use Local Mermaid -Download mermaid.min.js and update the config: - -```javascript -const config = { - // ... - mermaidCDN: './path/to/mermaid.min.js' -} -``` - -#### Custom Fonts -Add additional fonts to the CSS: - -```css -@import url('https://fonts.googleapis.com/css2?family=YourFont:wght@400;500;600&display=swap'); - -body { - font-family: 'YourFont', sans-serif; -} -``` - -## Adding to package.json - -Add this script to your `package.json`: - -```json -{ - "scripts": { - "generate-pdf": "node dev/dev/scripts/generate-architecture-pdf.js", - "docs:pdf": "npm run generate-pdf" - }, - "devDependencies": { - "puppeteer": "^22.5.0" - } -} -``` - -## Alternative PDF Generators - -If you prefer other tools, you can also use: - -### 1. Pandoc + LaTeX -```bash -# Install pandoc and latex -sudo apt-get install pandoc texlive-latex-recommended - -# Convert (note: won't render Mermaid diagrams) -pandoc docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf -``` - -### 2. mdpdf -```bash -npm install -g mdpdf -mdpdf docs/brainy_architecture_visual.md --output=docs/brainy_architecture.pdf -``` - -### 3. markdown-pdf -```bash -npm install -g markdown-pdf -markdown-pdf docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf -``` - -**Note**: The custom Puppeteer script provides the best results with proper Mermaid diagram rendering and professional styling. - -## Sample Output - -The generated PDF will include: - -1. **Cover Page** with title and subtitle -2. **Table of Contents** with page links -3. **System Overview** with environment detection diagram -4. **Core Architecture** with layered architecture diagram -5. **Data Model** with noun/verb type hierarchies -6. **Vector Search Engine** with HNSW visualization -7. **Storage Architecture** with multi-tier caching diagrams -8. **Augmentation Pipeline** with flow diagrams -9. **Performance Optimizations** with threading models -10. **Integration Patterns** with network topology -11. **Data Flow Examples** with sequence diagrams - -Total pages: ~25-30 pages with full diagrams and explanations. - ---- - -*For questions or issues with PDF generation, please check the troubleshooting section or create an issue in the repository.* \ No newline at end of file diff --git a/dev/docs/QUICK_PDF_SETUP.md b/dev/docs/QUICK_PDF_SETUP.md deleted file mode 100644 index fd888885..00000000 --- a/dev/docs/QUICK_PDF_SETUP.md +++ /dev/null @@ -1,80 +0,0 @@ -# Quick PDF Generation Setup - -## 🚀 Generate Professional Brainy Architecture PDF - -### One-Command Setup & Generation - -```bash -# Install Puppeteer and generate PDF in one go -npm install puppeteer && npm run generate-pdf -``` - -### Step-by-Step - -1. **Install Puppeteer** (if not already installed): - ```bash - npm install puppeteer - ``` - -2. **Generate the PDF**: - ```bash - npm run generate-pdf - ``` - -3. **Find your PDF**: - ``` - docs/Brainy_Architecture_Documentation.pdf - ``` - -## ✨ What You Get - -- **25-30 page professional PDF** with full diagrams -- **Vector graphics** for all Mermaid diagrams -- **Modern typography** with Inter font family -- **Consistent branding** throughout the document -- **Table of contents** with page links -- **Professional headers/footers** - -## 📊 Sample Sections Include - -- System Overview with environment detection -- Core Architecture layers -- Data Model (23 Noun Types, 38 Verb Types) -- Vector Search Engine with HNSW visualization -- Storage Architecture with multi-tier caching -- Augmentation Pipeline flows -- Performance optimizations -- Cross-platform integration patterns -- Real data flow examples - -## 🛠️ Troubleshooting - -### Issue: "Puppeteer not found" -```bash -npm install puppeteer -``` - -### Issue: "Chrome not found" -```bash -# Let Puppeteer download Chromium -npm install puppeteer --unsafe-perm=true -``` - -### Issue: "Permission denied" -```bash -chmod +x dev/scripts/generate-architecture-pdf.js -``` - -## 🎨 Customization - -Edit `dev/scripts/generate-architecture-pdf.js` to: -- Change colors and fonts -- Modify page layout -- Adjust diagram styling -- Add custom branding - ---- - -**Ready to generate?** Run `npm run generate-pdf` and get your professional architecture documentation! - -For detailed setup instructions, see `PDF_GENERATION_GUIDE.md`. \ No newline at end of file diff --git a/dev/docs/brainy_architecture_diagram.md b/dev/docs/brainy_architecture_diagram.md deleted file mode 100644 index 935e55b0..00000000 --- a/dev/docs/brainy_architecture_diagram.md +++ /dev/null @@ -1,313 +0,0 @@ -# Brainy Architecture Diagram - -## System Overview -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ BRAINY PLATFORM │ -│ Vector Graph Database with AI Pipeline │ -└─────────────────────────────────────────────────────────────────────────────────┘ - -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ ENVIRONMENT DETECTION │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ Browser │ Node.js │ Serverless │ Container │ Server │ -│ (OPFS) │ (File System) │ (In-Memory) │ (Adaptive) │ (S3/Cloud) │ -└─────────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -``` - -## Core Architecture - -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ BRAINY DATA API │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ add() │ search() │ addVerb() │ get() │ delete() │ backup() │ restore() │ etc. │ -└─────────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ AUGMENTATION PIPELINE │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ SENSE → MEMORY → COGNITION → CONDUIT → ACTIVATION → PERCEPTION → DIALOG → WS │ -└─────────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ DATA PROCESSING │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ Text/JSON → Embedding → Vector Storage │ -│ │ │ -│ ┌─────────────────────────┼─────────────────────────┐ │ -│ │ EMBEDDING │ VECTOR INDEX │ │ -│ │ │ │ │ -│ │ TensorFlow.js │ HNSW Algorithm │ │ -│ │ Universal Sentence │ - Hierarchical │ │ -│ │ Encoder (USE) │ - Fast Similarity │ │ -│ │ - GPU Acceleration │ - Configurable │ │ -│ │ - Batch Processing │ - Memory Efficient │ │ -│ │ - Worker Threads │ - Product Quantized │ │ -│ └─────────────────────────┼─────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -``` - -## Data Model & Graph Structure - -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ GRAPH DATA MODEL │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ NOUNS (Entities/Nodes) │ -│ ┌─────────────────────────────────────────────────────────────────────────┐ │ -│ │ Core Entity Types: │ Digital/Content Types: │ │ -│ │ • Person │ • Document │ │ -│ │ • Organization │ • Media │ │ -│ │ • Location │ • File │ │ -│ │ • Thing │ • Message │ │ -│ │ • Concept │ • Content │ │ -│ │ • Event │ │ │ -│ │ │ Collection Types: │ │ -│ │ Business/App Types: │ • Collection │ │ -│ │ • Product │ • Dataset │ │ -│ │ • Service │ │ │ -│ │ • User │ Descriptive Types: │ │ -│ │ • Task │ • Process, State, Role │ │ -│ │ • Project │ • Topic, Language, Currency, Measurement │ │ -│ └─────────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ VERBS (Relationships/Edges) │ -│ ┌─────────────────────────────────────────────────────────────────────────┐ │ -│ │ Core Relationships: │ Social/Organizational: │ │ -│ │ • RelatedTo │ • MemberOf, WorksWith │ │ -│ │ • Contains, PartOf │ • FriendOf, Follows, Likes │ │ -│ │ • LocatedAt, References │ • ReportsTo, Supervises, Mentors │ │ -│ │ │ • Communicates │ │ -│ │ Temporal/Causal: │ │ │ -│ │ • Precedes, Succeeds │ Descriptive/Functional: │ │ -│ │ • Causes, DependsOn │ • Describes, Defines, Categorizes │ │ -│ │ • Requires │ • Measures, Evaluates │ │ -│ │ │ • Uses, Implements, Extends │ │ -│ │ Creation/Transformation: │ │ │ -│ │ • Creates, Transforms │ Ownership/Attribution: │ │ -│ │ • Becomes, Modifies │ • Owns, AttributedTo │ │ -│ │ • Consumes │ • CreatedBy, BelongsTo │ │ -│ └─────────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ -``` - -## Vector Storage & Search Engine - -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ VECTOR SEARCH ENGINE │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ Query Text/Vector → Embedding → HNSW Search → Ranked Results │ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────────┐ │ -│ │ HNSW INDEX STRUCTURE │ │ -│ │ │ │ -│ │ Layer 2: ●────────●────────● (Sparse connections) │ │ -│ │ ╱│ │ │╲ │ │ -│ │ Layer 1: ●─●──●─●─●─●──●─●─●─● (Medium density) │ │ -│ │ ╱│││││││││││││││││││││╲ │ │ -│ │ Layer 0: ●●●●●●●●●●●●●●●●●●●●●●● (Dense connections) │ │ -│ │ │ │ -│ │ • Hierarchical navigation for fast search │ │ -│ │ • Configurable M (max connections), efConstruction, efSearch │ │ -│ │ • Memory-efficient with disk-based storage for large datasets │ │ -│ │ • Product quantization for dimensionality reduction │ │ -│ └─────────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ -``` - -## Storage Architecture - -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ ADAPTIVE STORAGE │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌─ Hot Cache (RAM) ──┐ │ -│ │ Most accessed │ │ -│ │ LRU eviction │ │ -│ │ Auto-tuned size │ │ -│ └─────────────────────┘ │ -│ │ │ -│ ┌─ Warm Cache (Storage) ─┐ │ -│ │ Recent nodes │ │ -│ │ OPFS/Filesystem/S3 │ │ -│ │ TTL-based │ │ -│ └─────────────────────────┘ │ -│ │ │ -│ ┌─ Cold Storage (Persistent) ─┐ │ -│ │ All nodes │ │ -│ │ OPFS/Filesystem/S3 │ │ -│ │ Batch operations │ │ -│ └─────────────────────────────┘ │ -│ │ -│ Environment-Specific Storage Adapters: │ -│ ┌─────────────┬─────────────┬─────────────┬─────────────┬─────────────┐ │ -│ │ Browser │ Node.js │ Serverless │ Container │ Server │ │ -│ │ OPFS │ FileSystem │ In-Memory │ Adaptive │ S3/Cloud │ │ -│ │ (Fallback: │ (Backup: │ (Optional: │ (Auto- │ (Multi- │ │ -│ │ IndexedDB) │ S3/Cloud) │ S3/Cloud) │ Detect) │ Provider) │ │ -│ └─────────────┴─────────────┴─────────────┴─────────────┴─────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ -``` - -## Augmentation Pipeline System - -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ AUGMENTATION PIPELINE FLOW │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ Raw Data → [SENSE] → [MEMORY] → [COGNITION] → [CONDUIT] → [ACTIVATION] → │ -│ │ │ │ │ │ │ -│ ▼ ▼ ▼ ▼ ▼ │ -│ Process Storage Reasoning Data Sync Actions │ -│ Input Persist Inference External Triggers │ -│ Convert Retrieve Logic Ops Systems Responses │ -│ │ -│ → [PERCEPTION] → [DIALOG] → [WEBSOCKET] → │ -│ │ │ │ │ -│ ▼ ▼ ▼ │ -│ Visualization NLP/Chat Real-time │ -│ Interpretation Response Streaming │ -│ Organization Context Communication │ -│ │ -│ Execution Modes: │ -│ • SEQUENTIAL: Step-by-step processing │ -│ • PARALLEL: Concurrent augmentation execution │ -│ • THREADED: Multi-threaded with worker pools │ -└─────────────────────────────────────────────────────────────────────────────────┘ -``` - -## Performance & Scaling Features - -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ PERFORMANCE OPTIMIZATIONS │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────────┐ │ -│ │ MULTITHREADING │ │ -│ │ │ │ -│ │ Main Thread ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ -│ │ ├──────→ │ Worker 1 │ │ Worker 2 │ │ Worker N │ │ │ -│ │ │ │Embedding │ │ Search │ │ Batch │ │ │ -│ │ │ │Generation│ │Operations│ │Processing│ │ │ -│ │ ←──────── └──────────┘ └──────────┘ └──────────┘ │ │ -│ │ │ │ -│ │ • Web Workers (Browser) / Worker Threads (Node.js) │ │ -│ │ • Model caching and reuse across workers │ │ -│ │ • Batch embedding for better performance │ │ -│ └─────────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────────┐ │ -│ │ GPU ACCELERATION │ │ -│ │ │ │ -│ │ TensorFlow.js → WebGL Backend → GPU │ │ -│ │ ↓ │ │ -│ │ Fallback: CPU Backend for compatibility │ │ -│ │ │ │ -│ │ • Vector similarity calculations │ │ -│ │ • Embedding generation │ │ -│ │ • Tensor operations │ │ -│ │ • Automatic memory management │ │ -│ └─────────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────────┐ │ -│ │ INTELLIGENT CACHING │ │ -│ │ │ │ -│ │ • Auto-tuning based on usage patterns │ │ -│ │ • Memory-aware cache sizing │ │ -│ │ • Prefetching strategies │ │ -│ │ • LRU eviction with batch processing │ │ -│ │ • Read-only mode optimizations │ │ -│ └─────────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ -``` - -## Cross-Platform Integration - -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ SYNCHRONIZATION & SCALING │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ Browser ←→ WebSocket ←→ Server ←→ S3/Cloud Storage │ -│ ↓ ↓ │ -│ Browser ←→ WebRTC ←→ Browser (Peer-to-Peer) │ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────────┐ │ -│ │ CONDUIT AUGMENTATIONS │ │ -│ │ │ │ -│ │ WebSocket iConduit: │ │ -│ │ • Browser ↔ Server sync │ │ -│ │ • Server ↔ Server sync │ │ -│ │ • Real-time data streaming │ │ -│ │ │ │ -│ │ WebRTC iConduit: │ │ -│ │ • Direct browser ↔ browser sync │ │ -│ │ • Peer-to-peer without server │ │ -│ │ • Decentralized data sharing │ │ -│ └─────────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────────────────────┐ │ -│ │ MODEL CONTROL PROTOCOL (MCP) │ │ -│ │ │ │ -│ │ External AI Models ←→ MCP Server ←→ Brainy Data & Tools │ │ -│ │ │ │ -│ │ • BrainyMCPAdapter: Data access for external models │ │ -│ │ • MCPAugmentationToolset: Pipeline tools for models │ │ -│ │ • BrainyMCPService: WebSocket & REST integration │ │ -│ └─────────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ -``` - -## Data Flow Example - -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ DATA FLOW EXAMPLE │ -├─────────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ 1. Input: "Cats are independent pets" │ -│ ↓ │ -│ 2. SENSE Augmentation: Process raw text │ -│ ↓ │ -│ 3. Embedding: TensorFlow USE → [0.123, -0.456, 0.789, ...] │ -│ ↓ │ -│ 4. MEMORY Augmentation: Store with metadata │ -│ ↓ │ -│ 5. HNSW Index: Add vector to hierarchical graph │ -│ ↓ │ -│ 6. Storage: Persist to OPFS/FileSystem/S3 │ -│ │ -│ Query: "feline pets" → Embedding → HNSW Search → Ranked Results │ -│ Result: [{text: "Cats are independent pets", similarity: 0.89, id: "123"}] │ -│ │ -│ Relationship Example: │ -│ addVerb(catId, dogId, VerbType.RelatedTo, {description: "Both are pets"}) │ -│ ↓ │ -│ Graph: [Cat] ──RelatedTo──→ [Dog] │ -└─────────────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -**Key Architecture Principles:** - -1. **Environment Agnostic**: Automatically adapts to browser, Node.js, serverless, container, or server environments -2. **Intelligent Storage**: Multi-tier caching with automatic storage selection (OPFS, filesystem, S3, memory) -3. **Vector + Graph**: Combines semantic vector search with graph relationships in a unified model -4. **Extensible Pipeline**: Modular augmentation system for custom processing and integration -5. **Performance Optimized**: GPU acceleration, multithreading, intelligent caching, and memory management -6. **Scalable Sync**: WebSocket and WebRTC conduits for real-time synchronization across instances -7. **AI Integration**: MCP protocol for external AI model integration and tool access \ No newline at end of file diff --git a/dev/scripts/generate-architecture-pdf.js b/dev/scripts/generate-architecture-pdf.js deleted file mode 100755 index ff13d8ce..00000000 --- a/dev/scripts/generate-architecture-pdf.js +++ /dev/null @@ -1,1421 +0,0 @@ -#!/usr/bin/env node - -/** - * Generate Professional PDF from Brainy Architecture Documentation - * - * This script converts the Markdown documentation with Mermaid diagrams - * into a beautiful, professional PDF using Puppeteer and modern CSS styling. - */ - -import puppeteer from 'puppeteer' -import fs from 'fs' -import path from 'path' -import { fileURLToPath } from 'url' - -const __filename = fileURLToPath(import.meta.url) -const __dirname = path.dirname(__filename) - -// Configuration -const config = { - readmeFile: path.join(__dirname, '../../README.md'), - architectureFile: path.join(__dirname, '../docs/brainy_architecture_diagram.md'), - outputFile: path.join(__dirname, '../../docs/Brainy_Architecture_Documentation.pdf') -} - -// Apple-inspired CSS styling with Brainy retro sci-fi aesthetic -const appleInspiredCSS = ` - -` - -// SVG Diagram Generation Functions -function generateArchitectureDiagram() { - return ` -
-

Brainy System Architecture

- - - - - - - Environment Detection & Adaptation - Browser - Node.js - Serverless - Container - Server - Cloud - - - - Brainy Data API - add() | search() | addVerb() | get() | delete() | backup() | restore() - - - - Augmentation Pipeline - - SENSE - - MEMORY - - COGNITION - - CONDUIT - - PERCEPTION - - DIALOG - - - - Vector Processing - - TensorFlow - Embedding - - HNSW - Index - - - - - Adaptive Storage - - Hot Cache - - Warm Cache - - Cold Storage - OPFS - FileSystem - S3/Cloud - Memory - - - - Cross-Platform Synchronization - - WebSocket Conduit - - WebRTC Conduit - - MCP Protocol - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-`; -} - -function generateDataFlowDiagram() { - return ` -
-

Data Processing Flow

- - - - - - - Raw Data - Text/JSON - Images/Media - - - - Embedding - TensorFlow.js - GPU Accel - - - - Vector Index - HNSW - Algorithm - - - - Graph Build - Noun-Verb - Relations - - - - Storage - Multi-tier - Adaptive - - - - Query Processing - Semantic Search - Graph Traversal - Result Ranking - - - - Performance - Multithreading - Caching - - - Scalability - Auto-tuning - Load Balancing - - - Synchronization - Real-time - Cross-platform - - - Intelligence - Adaptive Learning - Pattern Analysis - - - - - - - - - - - - - - - - - - - - - -
-`; -} - -function generateHNSWDiagram() { - return ` -
-

HNSW (Hierarchical Navigable Small World) Index Structure

- - - - - - Layer 2 - Layer 1 - Layer 0 - (Base Layer) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Search Path (Logarithmic Complexity) - - - - HNSW Benefits - • Fast similarity search (log complexity) - • Memory efficient indexing - • Scalable to millions of vectors - • Configurable precision/performance - -
-`; -} - -// Retro logo-inspired diagram for title page -function generateBrainyLogo() { - return ` -
- - - - - - - - - - - - - - - - - - - - - - - - - - - BRAINY - ARCHITECTURE - - - - - - - -
-`; -} - -// HTML template with Apple-inspired structure -const htmlTemplate = ` - - - - - - Brainy Architecture Documentation - ${appleInspiredCSS} - - - - {{CONTENT}} - - -` - -// Generate comprehensive content combining README and architecture documentation -function generateComprehensiveContent() { - const readmeContent = fs.readFileSync(config.readmeFile, 'utf8') - const architectureContent = fs.readFileSync(config.architectureFile, 'utf8') - - // Extract key sections from README - const overviewMatch = readmeContent.match(/## ✨ Overview([\s\S]*?)(?=\n## |$)/) - const featuresMatch = readmeContent.match(/### 🚀 Key Features([\s\S]*?)(?=\n## |\n### |$)/) - const howItWorksMatch = readmeContent.match(/## 🧩 How It Works([\s\S]*?)(?=\n## |$)/) - const pipelineMatch = readmeContent.match(/## 🚀 The Brainy Pipeline([\s\S]*?)(?=\n## |$)/) - const dataModelMatch = readmeContent.match(/## Data Model([\s\S]*?)(?=\n## |$)/) - const scalingMatch = readmeContent.match(/## 📈 Scaling Strategy([\s\S]*?)(?=\n## |$)/) - const performanceMatch = readmeContent.match(/### Performance Tuning([\s\S]*?)(?=\n## |\n### |$)/) - - const content = ` -${generateBrainyLogo()} - -# Brainy Architecture Documentation -
A Comprehensive Guide to the Advanced AI Graph Database Platform
- -
-

Table of Contents

- -1. Executive Summary -2. System Overview & Key Features -3. Core Architecture & Design Principles -4. Advanced Data Processing Pipeline -5. Vector Search & Graph Database Technology -6. Storage Architecture & Environment Adaptation -7. Performance & Scalability Features -8. Cross-Platform Integration & Synchronization -9. Technical Implementation Details -10. Architecture Diagrams & Visual Guide -
- -
-

1. Executive Summary

- -Brainy represents a revolutionary approach to AI-powered data management, combining the semantic understanding of vector databases with the relational power of graph structures. This platform automatically adapts to any computing environment while providing enterprise-grade performance and scalability. - -
-
-
🌐 Universal Deployment
-Runs seamlessly across browsers, servers, containers, and cloud platforms -
-
-
🧠 Intelligent Adaptation
-Automatically optimizes for your environment and usage patterns -
-
-
⚡ Advanced Performance
-Built-in TensorFlow.js with GPU acceleration and multithreading -
-
-
🔄 Real-time Sync
-WebSocket and WebRTC support for distributed systems -
-
- -${overviewMatch ? overviewMatch[1].trim() : ''} - -
-

Why Choose Brainy?

- -Brainy eliminates the complexity typically associated with vector databases and graph systems. It automatically handles environment detection, storage optimization, performance tuning, and scaling—allowing developers to focus on building innovative applications rather than managing infrastructure. - -
-Zero Configuration Required: Works out-of-the-box with intelligent defaults that adapt to your specific use case and environment. -
-
-
- -## 2. System Overview & Key Features - -${featuresMatch ? featuresMatch[1].trim() : ''} - -### Advanced Capabilities - -- **Multi-Environment Deployment**: Seamlessly runs in browsers, Node.js, serverless functions, containers, and dedicated servers -- **Intelligent Storage Selection**: Automatically chooses optimal storage (OPFS, filesystem, S3, or memory) based on environment -- **GPU-Accelerated Processing**: Leverages WebGL and TensorFlow.js for high-performance vector operations -- **Real-time Data Streaming**: Built-in WebSocket support for live data processing -- **Extensible Augmentation System**: Modular architecture for custom functionality -- **Enterprise-Ready Scaling**: Handles datasets from small collections to terabyte-scale deployments - -## 3. Core Architecture & Design Principles - -${howItWorksMatch ? howItWorksMatch[1].trim() : ''} - -### Architectural Principles - -Brainy is built on seven core principles that ensure scalability, performance, and ease of use: - -
-
-

🌐 Environment Agnostic

-Automatically adapts to any computing environment without configuration changes -
- -
-

🧠 Intelligent Storage

-Multi-tier caching with automatic storage selection and optimization -
- -
-

🔍 Vector + Graph

-Combines semantic search with graph relationships in a unified model -
- -
-

🔧 Extensible Pipeline

-Modular augmentation system for custom processing and integration -
- -
-

⚡ Performance Optimized

-GPU acceleration, multithreading, and intelligent caching -
- -
-

🔄 Scalable Sync

-WebSocket and WebRTC for real-time synchronization -
- -
-

🤖 AI Integration

-MCP protocol for external AI model integration -
-
- -## 4. Advanced Data Processing Pipeline - -${pipelineMatch ? pipelineMatch[1].trim() : ''} - -### Pipeline Innovation - -The Brainy pipeline represents a breakthrough in data processing architecture: - -- **Adaptive Learning**: The system learns from usage patterns and automatically optimizes performance -- **Multi-threaded Execution**: Parallel processing across Web Workers and Worker Threads -- **Streaming Data Support**: Real-time processing of incoming data streams -- **Intelligent Caching**: Multi-level caching that adapts to data access patterns -- **Error Resilience**: Robust error handling and recovery mechanisms - -## 5. Vector Search & Graph Database Technology - -### HNSW Algorithm Implementation - -Brainy uses an optimized Hierarchical Navigable Small World (HNSW) algorithm for fast similarity search: - -- **Hierarchical Structure**: Multi-layer graph for logarithmic search complexity -- **Memory Efficiency**: Product quantization for large datasets -- **Disk-Based Storage**: Hybrid approach for datasets exceeding memory limits -- **Configurable Parameters**: Tunable for precision vs. performance tradeoffs - -### Vector Embedding Technology - -- **TensorFlow Universal Sentence Encoder**: High-quality text embeddings -- **GPU Acceleration**: WebGL backend for fast computation -- **Batch Processing**: Efficient handling of multiple embeddings -- **Custom Embedding Support**: Pluggable embedding functions - -## 6. Storage Architecture & Environment Adaptation - -### Multi-Tier Storage Strategy - -Brainy implements a sophisticated three-tier storage architecture: - -1. **Hot Cache (RAM)**: Most frequently accessed data with LRU eviction -2. **Warm Cache (Local Storage)**: Recently accessed data with TTL management -3. **Cold Storage (Persistent)**: Complete dataset with batch optimization - -### Environment-Specific Optimizations - -- **Browser**: Origin Private File System (OPFS) with IndexedDB fallback -- **Node.js**: File system with optional S3 integration -- **Serverless**: In-memory with cloud persistence options -- **Container**: Automatic detection and adaptation -- **Server**: Full S3-compatible cloud storage support - -## 7. Performance & Scalability Features - -${performanceMatch ? performanceMatch[1].trim() : ''} - -### Scaling Capabilities - -${scalingMatch ? scalingMatch[1].trim() : ''} - -### Performance Optimizations - -- **Intelligent Defaults**: Automatic parameter tuning based on environment and dataset -- **Memory-Aware Caching**: Dynamic cache sizing based on available resources -- **Batch Operations**: Optimized bulk data processing -- **Lazy Loading**: On-demand data loading for large datasets -- **Query Optimization**: Smart query planning and execution - -## 8. Cross-Platform Integration & Synchronization - -### Real-Time Synchronization - -Brainy provides multiple synchronization mechanisms: - -- **WebSocket Conduits**: Browser-server and server-server synchronization -- **WebRTC Conduits**: Direct peer-to-peer browser communication -- **Change Logging**: Efficient delta synchronization -- **Conflict Resolution**: Automatic handling of concurrent updates - -### Model Control Protocol (MCP) - -Integration with external AI models through standardized protocols: - -- **Data Access**: Secure access to Brainy data for external models -- **Tool Integration**: Expose Brainy functionality as AI tools -- **Service Architecture**: WebSocket and REST API support - -## 9. Technical Implementation Details - -${dataModelMatch ? dataModelMatch[1].trim() : ''} - -### Advanced Features - -- **Type Safety**: Full TypeScript support with generics -- **Field Standardization**: Cross-service field mapping and search -- **Backup & Restore**: Complete data portability -- **Testing Suite**: Comprehensive test coverage with Vitest -- **CLI Tools**: Command-line interface for data management - -### Development & Operations - -- **Zero Configuration**: Works out-of-the-box with intelligent defaults -- **Monitoring**: Built-in statistics and performance metrics -- **Documentation**: Comprehensive guides and API reference -- **Community**: Open-source with active development - -## 10. Architecture Diagrams & Visual Guide - -${generateArchitectureDiagram()} - -${generateDataFlowDiagram()} - -${generateHNSWDiagram()} - -### Original Architecture Documentation - -${architectureContent} - -
-

Conclusion

- -Brainy represents the next generation of AI-powered data platforms, combining ease of use with enterprise-grade capabilities. Its intelligent adaptation, powerful features, and comprehensive architecture make it the ideal choice for modern applications requiring semantic search, graph relationships, and real-time data processing. - -Whether you're building a simple browser application or a complex distributed system, Brainy automatically adapts to provide optimal performance and functionality in any environment. - -*For more information, visit the [Brainy GitHub repository](https://github.com/soulcraft-research/brainy) or try the [live demo](https://soulcraft-research.github.io/brainy/demo/index.html).* -` - - return content -} - -// Function to convert markdown to HTML with enhanced formatting -function markdownToHTML(markdown) { - let html = markdown - - // Convert headers - html = html.replace(/^# (.*$)/gm, '

$1

') - html = html.replace(/^## (.*$)/gm, '

$1

') - html = html.replace(/^### (.*$)/gm, '

$1

') - html = html.replace(/^#### (.*$)/gm, '

$1

') - - // Convert bold and italic - html = html.replace(/\*\*(.*?)\*\*/g, '$1') - html = html.replace(/\*(.*?)\*/g, '$1') - - // Convert inline code - html = html.replace(/`([^`]+)`/g, '$1') - - // Convert links - html = html.replace(/\[([^\]]+)\]\(([^)]+)\)/g, '$1') - - // Convert horizontal rules - html = html.replace(/^---$/gm, '
') - - // Remove image syntax (since we can't embed images in PDF easily) - html = html.replace(/!\[([^\]]*)\]\([^)]+\)/g, '') - - // Convert paragraphs - html = html.replace(/\n\n/g, '

') - html = '

' + html + '

' - - // Clean up empty paragraphs - html = html.replace(/

<\/p>/g, '') - html = html.replace(/

(\s*)\s*<\/p>/g, '$1') - html = html.replace(/

(\s*)\s*<\/p>/g, '$1') - html = html.replace(/

(\s*)\s*<\/p>/g, '$1') - - // Convert lists - html = html.replace(/^\* (.*$)/gm, '

  • $1
    • ') - html = html.replace(/^- (.*$)/gm, '
  • $1
    • ') - html = html.replace(/^(\d+)\. (.*$)/gm, '
  • $2
    • ') - - // Wrap consecutive list items in ul/ol tags - html = html.replace(/(
  • .*?<\/li>)(\s*
  • .*?<\/li>)*/gs, (match) => { - return '' - }) - - // Handle code blocks - html = html.replace(/```(\w+)?\n([\s\S]*?)\n```/g, (match, lang, code) => { - return `
    ${code}
    ` - }) - - // Convert custom SVG diagram calls - html = html.replace(/\$\{generateBrainyLogo\(\)\}/g, generateBrainyLogo()) - html = html.replace(/\$\{generateArchitectureDiagram\(\)\}/g, generateArchitectureDiagram()) - html = html.replace(/\$\{generateDataFlowDiagram\(\)\}/g, generateDataFlowDiagram()) - html = html.replace(/\$\{generateHNSWDiagram\(\)\}/g, generateHNSWDiagram()) - - // Handle special divs - html = html.replace(/
    ([\s\S]*?)<\/div>/g, '
    $2
    ') - - // Add page breaks for major sections - html = html.replace(/

    /g, '

    ') - - // Enhance key features section - html = html.replace(/### Key Features([\s\S]*?)(?=

    |$)/g, (match) => { - return match.replace(/
  • /g, '
  • ') - }) - - return html -} - -// Main function to generate PDF -async function generatePDF() { - try { - console.log('🚀 Starting PDF generation...') - - // Check required files exist - if (!fs.existsSync(config.readmeFile)) { - throw new Error(`README file not found: ${config.readmeFile}`) - } - if (!fs.existsSync(config.architectureFile)) { - throw new Error(`Architecture file not found: ${config.architectureFile}`) - } - - console.log('📖 Generating comprehensive content...') - const comprehensiveContent = generateComprehensiveContent() - - console.log('🔄 Converting to HTML...') - const contentHTML = markdownToHTML(comprehensiveContent) - const fullHTML = htmlTemplate.replace('{{CONTENT}}', contentHTML) - - // Launch Puppeteer - console.log('🌐 Launching browser...') - const browser = await puppeteer.launch({ - headless: 'new', - args: ['--no-sandbox', '--disable-setuid-sandbox'] - }) - - const page = await browser.newPage() - - // Set content and wait for SVG diagrams to render - await page.setContent(fullHTML, { waitUntil: 'networkidle0' }) - console.log('🎨 Rendering SVG diagrams...') - - // Wait for all SVG diagrams to be present - await page.waitForFunction(() => { - const svgElements = document.querySelectorAll('.svg-diagram svg') - return svgElements.length >= 3 // We have 3 main diagrams - }, { timeout: 15000 }) - - // Generate PDF with professional settings - console.log('📄 Generating PDF...') - await page.pdf({ - path: config.outputFile, - format: 'A4', - printBackground: true, - margin: { - top: '20mm', - right: '15mm', - bottom: '20mm', - left: '15mm' - }, - displayHeaderFooter: true, - headerTemplate: ` -
    - Brainy Architecture Documentation -
    - `, - footerTemplate: ` -
    - Generated from Brainy v0.34.0 | Page of -
    - ` - }) - - await browser.close() - - console.log('✅ PDF generated successfully!') - console.log(`📁 Output file: ${config.outputFile}`) - - // Check file size - const stats = fs.statSync(config.outputFile) - const fileSizeInMB = (stats.size / (1024 * 1024)).toFixed(2) - console.log(`📊 File size: ${fileSizeInMB} MB`) - - } catch (error) { - console.error('❌ Error generating PDF:', error) - process.exit(1) - } -} - -// Check dependencies -async function checkDependencies() { - try { - const puppeteerVersion = (await import('puppeteer')).default - console.log('✅ Dependencies verified') - } catch (error) { - console.error('❌ Missing dependencies. Run: npm install puppeteer') - process.exit(1) - } -} - -// Main execution -async function main() { - console.log('🏗️ Brainy Architecture PDF Generator') - console.log('=====================================') - - await checkDependencies() - - // Ensure output directory exists - const outputDir = path.dirname(config.outputFile) - if (!fs.existsSync(outputDir)) { - fs.mkdirSync(outputDir, { recursive: true }) - } - - await generatePDF() - - console.log('=====================================') - console.log('🎉 PDF generation complete!') -} - -// Run if called directly -if (import.meta.url === `file://${process.argv[1]}`) { - main().catch(console.error) -} - -export { generatePDF, config } \ No newline at end of file diff --git a/docs/Brainy_Architecture_Documentation.pdf b/docs/Brainy_Architecture_Documentation.pdf deleted file mode 100644 index 7fc60dcf..00000000 Binary files a/docs/Brainy_Architecture_Documentation.pdf and /dev/null differ diff --git a/docs/brainy_architecture_visual.md b/docs/brainy_architecture_visual.md deleted file mode 100644 index 49933e75..00000000 --- a/docs/brainy_architecture_visual.md +++ /dev/null @@ -1,729 +0,0 @@ -# Brainy Architecture Documentation -## Vector Graph Database with AI Pipeline - ---- - -## Table of Contents - -1. [System Overview](#system-overview) -2. [Core Architecture](#core-architecture) -3. [Data Model & Graph Structure](#data-model--graph-structure) -4. [Vector Search Engine](#vector-search-engine) -5. [Storage Architecture](#storage-architecture) -6. [Augmentation Pipeline](#augmentation-pipeline) -7. [Performance Optimizations](#performance-optimizations) -8. [Cross-Platform Integration](#cross-platform-integration) -9. [Data Flow Example](#data-flow-example) - ---- - -## System Overview - -Brainy is a powerful, cross-platform vector graph database that intelligently adapts to any environment while providing both semantic vector search and graph relationship capabilities. - -```mermaid -graph TD - A[User Application] --> B[Brainy Platform] - B --> C[Environment Detection] - - C --> D[Browser
    OPFS Storage] - C --> E[Node.js
    File System] - C --> F[Serverless
    In-Memory] - C --> G[Container
    Adaptive] - C --> H[Server
    S3/Cloud] - - B --> I[Vector Search Engine] - B --> J[Graph Database] - B --> K[Augmentation Pipeline] - - style B fill:#e1f5fe - style I fill:#f3e5f5 - style J fill:#e8f5e8 - style K fill:#fff3e0 -``` - -### Key Features - -- **Universal Compatibility**: Runs everywhere - browsers, Node.js, serverless functions, containers -- **Intelligent Adaptation**: Automatically optimizes for environment and usage patterns -- **Dual Nature**: Vector similarity search + graph relationships in one system -- **Real-time Streaming**: Live data processing through extensible pipeline -- **AI Integration**: Built-in TensorFlow.js with GPU acceleration - ---- - -## Core Architecture - -```mermaid -graph TB - subgraph "Application Layer" - API[Brainy Data API
    add() | search() | addVerb() | get() | delete()] - end - - subgraph "Processing Layer" - PIPELINE[Augmentation Pipeline
    SENSE → MEMORY → COGNITION → CONDUIT → ACTIVATION → PERCEPTION → DIALOG → WS] - end - - subgraph "Engine Layer" - EMBED[Embedding Engine
    TensorFlow.js Universal Sentence Encoder] - VECTOR[Vector Index
    HNSW Algorithm] - GRAPH[Graph Engine
    Noun-Verb Model] - end - - subgraph "Storage Layer" - CACHE[Multi-tier Caching
    Hot → Warm → Cold] - STORAGE[Adaptive Storage
    OPFS | FileSystem | S3 | Memory] - end - - API --> PIPELINE - PIPELINE --> EMBED - PIPELINE --> VECTOR - PIPELINE --> GRAPH - EMBED --> CACHE - VECTOR --> CACHE - GRAPH --> STORAGE - CACHE --> STORAGE - - style API fill:#e3f2fd - style PIPELINE fill:#f1f8e9 - style EMBED fill:#fce4ec - style VECTOR fill:#fff8e1 - style GRAPH fill:#e8f5e8 - style CACHE fill:#f3e5f5 - style STORAGE fill:#efebe9 -``` - ---- - -## Data Model & Graph Structure - -### Noun Types (Entities/Nodes) - -```mermaid -mindmap - root((Brainy
    Noun Types)) - Core Entities - Person - Organization - Location - Thing - Concept - Event - Digital Content - Document - Media - File - Message - Content - Collections - Collection - Dataset - Business/App - Product - Service - User - Task - Project - Descriptive - Process - State - Role - Topic - Language - Currency - Measurement -``` - -### Verb Types (Relationships/Edges) - -```mermaid -mindmap - root((Brainy
    Verb Types)) - Core Relations - RelatedTo - Contains - PartOf - LocatedAt - References - Temporal/Causal - Precedes - Succeeds - Causes - DependsOn - Requires - Creation/Transform - Creates - Transforms - Becomes - Modifies - Consumes - Ownership/Attribution - Owns - AttributedTo - CreatedBy - BelongsTo - Social/Organizational - MemberOf - WorksWith - FriendOf - Follows - Likes - ReportsTo - Supervises - Mentors - Communicates - Descriptive/Functional - Describes - Defines - Categorizes - Measures - Evaluates - Uses - Implements - Extends -``` - -### Graph Example - -```mermaid -graph LR - A[Person: John Doe
    ID: person-123] -->|WorksWith| B[Organization: Acme Corp
    ID: org-456] - A -->|CreatedBy| C[Document: Report
    ID: doc-789] - A -->|LocatedAt| D[Location: New York
    ID: loc-101] - B -->|Contains| E[Project: AI Initiative
    ID: proj-202] - C -->|PartOf| E - E -->|Uses| F[Concept: Machine Learning
    ID: concept-303] - - style A fill:#ffcdd2 - style B fill:#c8e6c9 - style C fill:#bbdefb - style D fill:#fff9c4 - style E fill:#f8bbd9 - style F fill:#d1c4e9 -``` - ---- - -## Vector Search Engine - -### HNSW Index Structure - -```mermaid -graph TB - subgraph "HNSW Hierarchical Structure" - subgraph "Layer 2 (Sparse)" - L2A((●)) --- L2B((●)) - L2B --- L2C((●)) - end - - subgraph "Layer 1 (Medium Density)" - L1A((●)) --- L1B((●)) - L1B --- L1C((●)) - L1C --- L1D((●)) - L1D --- L1E((●)) - L1E --- L1F((●)) - L1F --- L1G((●)) - L1G --- L1H((●)) - end - - subgraph "Layer 0 (Dense Connections)" - L0A((●)) --- L0B((●)) - L0B --- L0C((●)) - L0C --- L0D((●)) - L0D --- L0E((●)) - L0E --- L0F((●)) - L0F --- L0G((●)) - L0G --- L0H((●)) - L0H --- L0I((●)) - L0I --- L0J((●)) - L0J --- L0K((●)) - L0K --- L0L((●)) - L0L --- L0M((●)) - L0M --- L0N((●)) - L0N --- L0O((●)) - L0O --- L0P((●)) - end - - L2A -.-> L1A - L2A -.-> L1D - L2B -.-> L1C - L2B -.-> L1F - L2C -.-> L1G - - L1A -.-> L0A - L1A -.-> L0B - L1B -.-> L0C - L1B -.-> L0D - L1C -.-> L0E - L1C -.-> L0F - L1D -.-> L0G - L1D -.-> L0H - L1E -.-> L0I - L1E -.-> L0J - L1F -.-> L0K - L1F -.-> L0L - L1G -.-> L0M - L1G -.-> L0N - L1H -.-> L0O - L1H -.-> L0P - end - - style L2A fill:#ff9999 - style L2B fill:#ff9999 - style L2C fill:#ff9999 - style L1A fill:#99ccff - style L1B fill:#99ccff - style L1C fill:#99ccff - style L1D fill:#99ccff - style L1E fill:#99ccff - style L1F fill:#99ccff - style L1G fill:#99ccff - style L1H fill:#99ccff - style L0A fill:#99ff99 - style L0B fill:#99ff99 - style L0C fill:#99ff99 - style L0D fill:#99ff99 - style L0E fill:#99ff99 - style L0F fill:#99ff99 - style L0G fill:#99ff99 - style L0H fill:#99ff99 - style L0I fill:#99ff99 - style L0J fill:#99ff99 - style L0K fill:#99ff99 - style L0L fill:#99ff99 - style L0M fill:#99ff99 - style L0N fill:#99ff99 - style L0O fill:#99ff99 - style L0P fill:#99ff99 -``` - -### Search Process Flow - -```mermaid -sequenceDiagram - participant User - participant API - participant Embedding - participant HNSW - participant Storage - - User->>API: searchText("feline pets", 5) - API->>Embedding: embed("feline pets") - Embedding->>Embedding: TensorFlow.js Universal Sentence Encoder - Embedding-->>API: [0.123, -0.456, 0.789, ...] - - API->>HNSW: search(vector, k=5) - HNSW->>HNSW: Navigate from top layer - HNSW->>HNSW: Descend to lower layers - HNSW->>HNSW: Find k nearest neighbors - HNSW-->>API: [id1, id2, id3, id4, id5] - - API->>Storage: get([id1, id2, id3, id4, id5]) - Storage-->>API: [noun1, noun2, noun3, noun4, noun5] - - API-->>User: [{text: "Cats are independent pets", similarity: 0.89}, ...] -``` - ---- - -## Storage Architecture - -### Multi-Tier Caching System - -```mermaid -graph TD - subgraph "Memory Hierarchy" - subgraph "Hot Cache (RAM)" - HC[Most Accessed Items
    LRU Eviction
    Auto-tuned Size
    Millisecond Access] - end - - subgraph "Warm Cache (Storage)" - WC[Recent Items
    TTL-based
    Sub-second Access
    OPFS/FS/S3] - end - - subgraph "Cold Storage (Persistent)" - CS[All Items
    Batch Operations
    Full Persistence
    OPFS/FS/S3] - end - end - - subgraph "Environment Adapters" - Browser[Browser
    OPFS → IndexedDB] - NodeJS[Node.js
    FileSystem → S3] - Serverless[Serverless
    Memory → S3] - Container[Container
    Auto-detect] - Server[Server
    S3/Multi-cloud] - end - - User[User Query] --> HC - HC -->|Cache Miss| WC - WC -->|Cache Miss| CS - - CS --> Browser - CS --> NodeJS - CS --> Serverless - CS --> Container - CS --> Server - - style HC fill:#ffcdd2 - style WC fill:#fff9c4 - style CS fill:#c8e6c9 - style Browser fill:#e1f5fe - style NodeJS fill:#e8f5e8 - style Serverless fill:#f3e5f5 - style Container fill:#fff3e0 - style Server fill:#efebe9 -``` - -### Storage Performance Characteristics - -```mermaid -xychart-beta - title "Storage Performance by Environment" - x-axis [Browser, Node.js, Serverless, Container, Server] - y-axis "Latency (ms)" 0 --> 1000 - line [50, 10, 200, 30, 100] -``` - ---- - -## Augmentation Pipeline - -### Pipeline Flow Architecture - -```mermaid -flowchart LR - subgraph "Data Processing Pipeline" - Input[Raw Data] --> SENSE[SENSE
    Process Input
    Convert & Validate] - SENSE --> MEMORY[MEMORY
    Storage Operations
    Persist & Retrieve] - MEMORY --> COGNITION[COGNITION
    Reasoning
    Inference & Logic] - COGNITION --> CONDUIT[CONDUIT
    Data Sync
    External Systems] - CONDUIT --> ACTIVATION[ACTIVATION
    Actions
    Triggers & Events] - ACTIVATION --> PERCEPTION[PERCEPTION
    Visualization
    Interpretation] - PERCEPTION --> DIALOG[DIALOG
    NLP & Chat
    Context & Response] - DIALOG --> WEBSOCKET[WEBSOCKET
    Real-time
    Streaming & Sync] - WEBSOCKET --> Output[Processed Output] - end - - subgraph "Execution Modes" - SEQ[Sequential
    Step-by-step] - PAR[Parallel
    Concurrent] - THR[Threaded
    Worker Pools] - end - - Input -.-> SEQ - Input -.-> PAR - Input -.-> THR - - style SENSE fill:#e8f5e8 - style MEMORY fill:#e3f2fd - style COGNITION fill:#fff3e0 - style CONDUIT fill:#f3e5f5 - style ACTIVATION fill:#ffebee - style PERCEPTION fill:#e0f2f1 - style DIALOG fill:#fce4ec - style WEBSOCKET fill:#e8eaf6 -``` - -### Augmentation Types Detail - -```mermaid -mindmap - root((Augmentation
    System)) - SENSE - Process Raw Data - Listen to Feeds - Data Validation - Format Conversion - MEMORY - Store Data - Retrieve Data - Update Data - Delete Data - List Keys - COGNITION - Reason - Infer - Execute Logic - Pattern Recognition - CONDUIT - Establish Connection - Read Data - Write Data - Monitor Stream - Sync Instances - ACTIVATION - Trigger Actions - Generate Output - Interact External - Event Handling - PERCEPTION - Interpret Data - Organize Info - Generate Visualization - Context Analysis - DIALOG - Process User Input - Generate Response - Manage Context - NLP Operations - WEBSOCKET - Connect WebSocket - Send Messages - Message Callbacks - Stream Monitoring -``` - ---- - -## Performance Optimizations - -### Multithreading Architecture - -```mermaid -graph TB - subgraph "Main Thread" - MT[Main Thread
    Coordination & API] - end - - subgraph "Worker Pool" - W1[Worker 1
    Embedding
    Generation] - W2[Worker 2
    Vector
    Search] - W3[Worker 3
    Batch
    Processing] - WN[Worker N
    Custom
    Operations] - end - - subgraph "GPU Acceleration" - GPU[TensorFlow.js
    WebGL Backend
    GPU Compute] - CPU[CPU Fallback
    Compatibility
    Mode] - end - - MT -->|Distribute Tasks| W1 - MT -->|Distribute Tasks| W2 - MT -->|Distribute Tasks| W3 - MT -->|Distribute Tasks| WN - - W1 --> GPU - W2 --> GPU - W3 --> GPU - WN --> GPU - - GPU -.->|Fallback| CPU - - W1 -->|Results| MT - W2 -->|Results| MT - W3 -->|Results| MT - WN -->|Results| MT - - style MT fill:#e3f2fd - style W1 fill:#e8f5e8 - style W2 fill:#e8f5e8 - style W3 fill:#e8f5e8 - style WN fill:#e8f5e8 - style GPU fill:#ffebee - style CPU fill:#fff3e0 -``` - -### Performance Metrics - -```mermaid -xychart-beta - title "Performance Improvements with Optimizations" - x-axis [Baseline, Caching, Multithreading, GPU, All Combined] - y-axis "Operations/Second" 0 --> 10000 - bar [1000, 3000, 5000, 7000, 9500] -``` - ---- - -## Cross-Platform Integration - -### Synchronization Network - -```mermaid -graph TB - subgraph "Browser Instances" - B1[Browser 1] - B2[Browser 2] - B3[Browser 3] - end - - subgraph "Server Infrastructure" - WS[WebSocket Server] - API[REST API Server] - S3[S3/Cloud Storage] - end - - subgraph "Peer-to-Peer" - STUN[STUN Server] - SIGNAL[Signaling Server] - end - - subgraph "External AI" - MCP[MCP Server] - AI[AI Models] - end - - B1 <-->|WebSocket| WS - B2 <-->|WebSocket| WS - B3 <-->|WebSocket| WS - - B1 <-.->|WebRTC| B2 - B2 <-.->|WebRTC| B3 - B1 <-.->|WebRTC| B3 - - WS <--> S3 - API <--> S3 - - B1 -.->|Signaling| SIGNAL - B2 -.->|Signaling| SIGNAL - B3 -.->|Signaling| SIGNAL - - SIGNAL -.-> STUN - - WS <--> MCP - MCP <--> AI - - style B1 fill:#e3f2fd - style B2 fill:#e3f2fd - style B3 fill:#e3f2fd - style WS fill:#e8f5e8 - style API fill:#e8f5e8 - style S3 fill:#fff3e0 - style MCP fill:#f3e5f5 - style AI fill:#ffebee -``` - -### Model Control Protocol (MCP) Integration - -```mermaid -sequenceDiagram - participant AI as External AI Model - participant MCP as MCP Server - participant Adapter as Brainy MCP Adapter - participant Brainy as Brainy Database - - AI->>MCP: Request data access - MCP->>Adapter: Forward request - Adapter->>Brainy: Query data - Brainy-->>Adapter: Return results - Adapter-->>MCP: Formatted response - MCP-->>AI: Data payload - - AI->>MCP: Execute augmentation - MCP->>Adapter: Pipeline request - Adapter->>Brainy: Run augmentation - Brainy-->>Adapter: Processing result - Adapter-->>MCP: Tool response - MCP-->>AI: Execution result -``` - ---- - -## Data Flow Example - -### Complete Processing Pipeline - -```mermaid -flowchart TD - subgraph "Input Processing" - I1[Input: "Cats are independent pets"] - I2[Metadata: {noun: "Thing", category: "animal"}] - end - - subgraph "Embedding Generation" - E1[TensorFlow.js Universal Sentence Encoder] - E2[Vector: [0.123, -0.456, 0.789, ...]] - end - - subgraph "Storage & Indexing" - S1[Store in Multi-tier Cache] - S2[Add to HNSW Index] - S3[Persist to Storage Layer] - end - - subgraph "Query Processing" - Q1[Query: "feline pets"] - Q2[Generate Query Vector] - Q3[HNSW Similarity Search] - Q4[Retrieve & Rank Results] - end - - subgraph "Graph Operations" - G1[Add Relationship] - G2[catId --RelatedTo--> dogId] - G3[Store Verb Metadata] - end - - I1 --> E1 - I2 --> E1 - E1 --> E2 - E2 --> S1 - S1 --> S2 - S2 --> S3 - - Q1 --> Q2 - Q2 --> Q3 - Q3 --> Q4 - - E2 -.-> G1 - G1 --> G2 - G2 --> G3 - - style I1 fill:#e8f5e8 - style E1 fill:#e3f2fd - style E2 fill:#f3e5f5 - style S1 fill:#fff3e0 - style Q1 fill:#e8f5e8 - style Q4 fill:#ffebee - style G2 fill:#e0f2f1 -``` - -### Result Example - -```json -{ - "results": [ - { - "id": "noun-123", - "text": "Cats are independent pets", - "similarity": 0.89, - "metadata": { - "noun": "Thing", - "category": "animal" - } - } - ], - "query": "feline pets", - "processingTime": "15ms", - "cacheHit": false -} -``` - ---- - -## Key Architecture Principles - -### 🌐 **Environment Agnostic** -Automatically adapts to browser, Node.js, serverless, container, or server environments without code changes. - -### 🧠 **Intelligent Storage** -Multi-tier caching with automatic storage selection optimizes for performance and persistence across platforms. - -### 🔍 **Vector + Graph Unified** -Combines semantic vector search with graph relationships in a single, coherent data model. - -### 🔧 **Extensible Pipeline** -Modular augmentation system allows custom processing, AI integration, and workflow automation. - -### ⚡ **Performance Optimized** -GPU acceleration, multithreading, intelligent caching, and memory management deliver enterprise-grade performance. - -### 🔄 **Scalable Synchronization** -WebSocket and WebRTC conduits enable real-time synchronization across instances and platforms. - -### 🤖 **AI Integration Ready** -Built-in MCP protocol support allows external AI models to access Brainy data and utilize augmentation tools. - ---- - -*Generated from Brainy v0.34.0 Architecture Documentation* \ No newline at end of file