diff --git a/.npmignore b/.npmignore index 41ed9e65..8012e36e 100644 --- a/.npmignore +++ b/.npmignore @@ -12,6 +12,7 @@ examples/ .idea/ cloud-wrapper/ scripts/ +dev/ # Configuration files .eslintrc diff --git a/dev/README.md b/dev/README.md new file mode 100644 index 00000000..ff6dfb7c --- /dev/null +++ b/dev/README.md @@ -0,0 +1,53 @@ +# Development Tools & Documentation + +This directory contains development tools, scripts, and documentation files that are not included in the published npm package. + +## Directory Structure + +``` +dev/ +├── docs/ # Development documentation files +│ ├── brainy_architecture_diagram.md +│ ├── PDF_GENERATION_GUIDE.md +│ ├── QUICK_PDF_SETUP.md +│ └── brainy_architecture_visual.md +├── scripts/ # Development scripts +│ └── generate-architecture-pdf.js +└── README.md # This file +``` + +## Scripts + +### generate-architecture-pdf.js + +Generates a professional PDF documentation of Brainy's architecture using: +- Material Design styling +- Custom SVG diagrams +- Comprehensive content from README.md +- Professional visual presentation + +**Usage:** +```bash +# From project root +node dev/scripts/generate-architecture-pdf.js + +# Or add to package.json scripts +npm run generate-docs +``` + +**Requirements:** +- puppeteer (for PDF generation) + +**Output:** +- `docs/Brainy_Architecture_Documentation.pdf` + +## Documentation Files + +- **brainy_architecture_diagram.md**: ASCII art diagrams of system architecture +- **PDF_GENERATION_GUIDE.md**: Detailed guide for PDF generation setup +- **QUICK_PDF_SETUP.md**: Quick setup instructions +- **brainy_architecture_visual.md**: Visual architecture documentation + +## NPM Package Exclusion + +This entire `dev/` directory is excluded from the published npm package via `.npmignore` to keep the package size minimal and focused on the core library functionality. \ No newline at end of file diff --git a/dev/docs/PDF_GENERATION_GUIDE.md b/dev/docs/PDF_GENERATION_GUIDE.md new file mode 100644 index 00000000..4e2c2b30 --- /dev/null +++ b/dev/docs/PDF_GENERATION_GUIDE.md @@ -0,0 +1,242 @@ +# Brainy Architecture PDF Generation Guide + +This guide shows you how to generate a professional PDF from the Brainy architecture documentation with beautiful diagrams. + +## Quick Start + +### Option 1: Using the npm script (Recommended) + +```bash +# Make sure you're in the brainy project directory +cd /path/to/brainy + +# Install dependencies if not already installed +npm install + +# Generate the PDF +npm run generate-pdf +``` + +### Option 2: Direct script execution + +```bash +# Make sure you're in the brainy project directory +cd /path/to/brainy + +# Install Puppeteer if not already installed +npm install puppeteer + +# Run the script directly +node dev/dev/scripts/generate-architecture-pdf.js +``` + +## Installation Requirements + +### Prerequisites +- Node.js 18+ +- npm or yarn + +### Dependencies +The script uses: +- **Puppeteer**: For PDF generation and browser automation +- **Mermaid**: For rendering diagrams (loaded via CDN) +- **Google Fonts**: For professional typography (loaded via CDN) + +### Install Dependencies + +```bash +# If you don't have puppeteer installed globally or in the project +npm install puppeteer + +# Or install as dev dependency +npm install --save-dev puppeteer +``` + +## Output + +The PDF will be generated at: +``` +docs/Brainy_Architecture_Documentation.pdf +``` + +## Features of the Generated PDF + +### Professional Styling +- **Modern Typography**: Uses Inter font family for clean, readable text +- **Code Font**: JetBrains Mono for code blocks and technical content +- **Color Scheme**: Professional blue theme with proper contrast +- **Layout**: A4 format with proper margins and spacing + +### Rich Diagrams +- **Mermaid Diagrams**: All diagrams are rendered as vector graphics +- **Interactive Elements**: Flowcharts, sequence diagrams, mindmaps, and more +- **Consistent Styling**: All diagrams follow the same color scheme +- **High Quality**: Vector-based rendering for crisp output + +### Document Structure +- **Table of Contents**: Linked navigation +- **Page Headers/Footers**: Professional branding and page numbers +- **Section Breaks**: Logical page breaks between major sections +- **Code Highlighting**: Syntax highlighting for JSON and code blocks + +## Customization + +### Modify Styling +Edit the `professionalCSS` variable in `dev/scripts/generate-architecture-pdf.js`: + +```javascript +const professionalCSS = ` + /* Your custom CSS here */ + h1 { + color: #your-color; + font-size: 24pt; + } + /* ... */ +` +``` + +### Change Output Location +Modify the `config` object: + +```javascript +const config = { + inputFile: path.join(__dirname, '../docs/brainy_architecture_visual.md'), + outputFile: path.join(__dirname, '../docs/YOUR_CUSTOM_NAME.pdf'), + // ... +} +``` + +### Adjust PDF Settings +Modify the `page.pdf()` options: + +```javascript +await page.pdf({ + path: config.outputFile, + format: 'A4', // or 'Letter', 'Legal', etc. + printBackground: true, + margin: { + top: '20mm', + right: '15mm', + bottom: '20mm', + left: '15mm' + }, + // ... other options +}) +``` + +## Troubleshooting + +### Common Issues + +#### 1. "Puppeteer not found" +```bash +npm install puppeteer +``` + +#### 2. "Chrome/Chromium not found" +```bash +# On Ubuntu/Debian +sudo apt-get install chromium-browser + +# On macOS +brew install chromium + +# Or let Puppeteer download Chromium +npm install puppeteer --unsafe-perm=true +``` + +#### 3. "Permission denied" +```bash +chmod +x dev/scripts/generate-architecture-pdf.js +``` + +#### 4. "Diagrams not rendering" +Check your internet connection - Mermaid is loaded from CDN. For offline use, you can download mermaid.min.js locally and update the path. + +### Advanced Configuration + +#### Use Local Mermaid +Download mermaid.min.js and update the config: + +```javascript +const config = { + // ... + mermaidCDN: './path/to/mermaid.min.js' +} +``` + +#### Custom Fonts +Add additional fonts to the CSS: + +```css +@import url('https://fonts.googleapis.com/css2?family=YourFont:wght@400;500;600&display=swap'); + +body { + font-family: 'YourFont', sans-serif; +} +``` + +## Adding to package.json + +Add this script to your `package.json`: + +```json +{ + "scripts": { + "generate-pdf": "node dev/dev/scripts/generate-architecture-pdf.js", + "docs:pdf": "npm run generate-pdf" + }, + "devDependencies": { + "puppeteer": "^22.5.0" + } +} +``` + +## Alternative PDF Generators + +If you prefer other tools, you can also use: + +### 1. Pandoc + LaTeX +```bash +# Install pandoc and latex +sudo apt-get install pandoc texlive-latex-recommended + +# Convert (note: won't render Mermaid diagrams) +pandoc docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf +``` + +### 2. mdpdf +```bash +npm install -g mdpdf +mdpdf docs/brainy_architecture_visual.md --output=docs/brainy_architecture.pdf +``` + +### 3. markdown-pdf +```bash +npm install -g markdown-pdf +markdown-pdf docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf +``` + +**Note**: The custom Puppeteer script provides the best results with proper Mermaid diagram rendering and professional styling. + +## Sample Output + +The generated PDF will include: + +1. **Cover Page** with title and subtitle +2. **Table of Contents** with page links +3. **System Overview** with environment detection diagram +4. **Core Architecture** with layered architecture diagram +5. **Data Model** with noun/verb type hierarchies +6. **Vector Search Engine** with HNSW visualization +7. **Storage Architecture** with multi-tier caching diagrams +8. **Augmentation Pipeline** with flow diagrams +9. **Performance Optimizations** with threading models +10. **Integration Patterns** with network topology +11. **Data Flow Examples** with sequence diagrams + +Total pages: ~25-30 pages with full diagrams and explanations. + +--- + +*For questions or issues with PDF generation, please check the troubleshooting section or create an issue in the repository.* \ No newline at end of file diff --git a/dev/docs/QUICK_PDF_SETUP.md b/dev/docs/QUICK_PDF_SETUP.md new file mode 100644 index 00000000..fd888885 --- /dev/null +++ b/dev/docs/QUICK_PDF_SETUP.md @@ -0,0 +1,80 @@ +# Quick PDF Generation Setup + +## 🚀 Generate Professional Brainy Architecture PDF + +### One-Command Setup & Generation + +```bash +# Install Puppeteer and generate PDF in one go +npm install puppeteer && npm run generate-pdf +``` + +### Step-by-Step + +1. **Install Puppeteer** (if not already installed): + ```bash + npm install puppeteer + ``` + +2. **Generate the PDF**: + ```bash + npm run generate-pdf + ``` + +3. **Find your PDF**: + ``` + docs/Brainy_Architecture_Documentation.pdf + ``` + +## ✨ What You Get + +- **25-30 page professional PDF** with full diagrams +- **Vector graphics** for all Mermaid diagrams +- **Modern typography** with Inter font family +- **Consistent branding** throughout the document +- **Table of contents** with page links +- **Professional headers/footers** + +## 📊 Sample Sections Include + +- System Overview with environment detection +- Core Architecture layers +- Data Model (23 Noun Types, 38 Verb Types) +- Vector Search Engine with HNSW visualization +- Storage Architecture with multi-tier caching +- Augmentation Pipeline flows +- Performance optimizations +- Cross-platform integration patterns +- Real data flow examples + +## 🛠️ Troubleshooting + +### Issue: "Puppeteer not found" +```bash +npm install puppeteer +``` + +### Issue: "Chrome not found" +```bash +# Let Puppeteer download Chromium +npm install puppeteer --unsafe-perm=true +``` + +### Issue: "Permission denied" +```bash +chmod +x dev/scripts/generate-architecture-pdf.js +``` + +## 🎨 Customization + +Edit `dev/scripts/generate-architecture-pdf.js` to: +- Change colors and fonts +- Modify page layout +- Adjust diagram styling +- Add custom branding + +--- + +**Ready to generate?** Run `npm run generate-pdf` and get your professional architecture documentation! + +For detailed setup instructions, see `PDF_GENERATION_GUIDE.md`. \ No newline at end of file diff --git a/dev/docs/brainy_architecture_diagram.md b/dev/docs/brainy_architecture_diagram.md new file mode 100644 index 00000000..935e55b0 --- /dev/null +++ b/dev/docs/brainy_architecture_diagram.md @@ -0,0 +1,313 @@ +# Brainy Architecture Diagram + +## System Overview +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ BRAINY PLATFORM │ +│ Vector Graph Database with AI Pipeline │ +└─────────────────────────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ ENVIRONMENT DETECTION │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ Browser │ Node.js │ Serverless │ Container │ Server │ +│ (OPFS) │ (File System) │ (In-Memory) │ (Adaptive) │ (S3/Cloud) │ +└─────────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +``` + +## Core Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ BRAINY DATA API │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ add() │ search() │ addVerb() │ get() │ delete() │ backup() │ restore() │ etc. │ +└─────────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ AUGMENTATION PIPELINE │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ SENSE → MEMORY → COGNITION → CONDUIT → ACTIVATION → PERCEPTION → DIALOG → WS │ +└─────────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ DATA PROCESSING │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ Text/JSON → Embedding → Vector Storage │ +│ │ │ +│ ┌─────────────────────────┼─────────────────────────┐ │ +│ │ EMBEDDING │ VECTOR INDEX │ │ +│ │ │ │ │ +│ │ TensorFlow.js │ HNSW Algorithm │ │ +│ │ Universal Sentence │ - Hierarchical │ │ +│ │ Encoder (USE) │ - Fast Similarity │ │ +│ │ - GPU Acceleration │ - Configurable │ │ +│ │ - Batch Processing │ - Memory Efficient │ │ +│ │ - Worker Threads │ - Product Quantized │ │ +│ └─────────────────────────┼─────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ +``` + +## Data Model & Graph Structure + +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ GRAPH DATA MODEL │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ NOUNS (Entities/Nodes) │ +│ ┌─────────────────────────────────────────────────────────────────────────┐ │ +│ │ Core Entity Types: │ Digital/Content Types: │ │ +│ │ • Person │ • Document │ │ +│ │ • Organization │ • Media │ │ +│ │ • Location │ • File │ │ +│ │ • Thing │ • Message │ │ +│ │ • Concept │ • Content │ │ +│ │ • Event │ │ │ +│ │ │ Collection Types: │ │ +│ │ Business/App Types: │ • Collection │ │ +│ │ • Product │ • Dataset │ │ +│ │ • Service │ │ │ +│ │ • User │ Descriptive Types: │ │ +│ │ • Task │ • Process, State, Role │ │ +│ │ • Project │ • Topic, Language, Currency, Measurement │ │ +│ └─────────────────────────────────────────────────────────────────────────┘ │ +│ │ +│ VERBS (Relationships/Edges) │ +│ ┌─────────────────────────────────────────────────────────────────────────┐ │ +│ │ Core Relationships: │ Social/Organizational: │ │ +│ │ • RelatedTo │ • MemberOf, WorksWith │ │ +│ │ • Contains, PartOf │ • FriendOf, Follows, Likes │ │ +│ │ • LocatedAt, References │ • ReportsTo, Supervises, Mentors │ │ +│ │ │ • Communicates │ │ +│ │ Temporal/Causal: │ │ │ +│ │ • Precedes, Succeeds │ Descriptive/Functional: │ │ +│ │ • Causes, DependsOn │ • Describes, Defines, Categorizes │ │ +│ │ • Requires │ • Measures, Evaluates │ │ +│ │ │ • Uses, Implements, Extends │ │ +│ │ Creation/Transformation: │ │ │ +│ │ • Creates, Transforms │ Ownership/Attribution: │ │ +│ │ • Becomes, Modifies │ • Owns, AttributedTo │ │ +│ │ • Consumes │ • CreatedBy, BelongsTo │ │ +│ └─────────────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Vector Storage & Search Engine + +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ VECTOR SEARCH ENGINE │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Query Text/Vector → Embedding → HNSW Search → Ranked Results │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────────┐ │ +│ │ HNSW INDEX STRUCTURE │ │ +│ │ │ │ +│ │ Layer 2: ●────────●────────● (Sparse connections) │ │ +│ │ ╱│ │ │╲ │ │ +│ │ Layer 1: ●─●──●─●─●─●──●─●─●─● (Medium density) │ │ +│ │ ╱│││││││││││││││││││││╲ │ │ +│ │ Layer 0: ●●●●●●●●●●●●●●●●●●●●●●● (Dense connections) │ │ +│ │ │ │ +│ │ • Hierarchical navigation for fast search │ │ +│ │ • Configurable M (max connections), efConstruction, efSearch │ │ +│ │ • Memory-efficient with disk-based storage for large datasets │ │ +│ │ • Product quantization for dimensionality reduction │ │ +│ └─────────────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Storage Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ ADAPTIVE STORAGE │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─ Hot Cache (RAM) ──┐ │ +│ │ Most accessed │ │ +│ │ LRU eviction │ │ +│ │ Auto-tuned size │ │ +│ └─────────────────────┘ │ +│ │ │ +│ ┌─ Warm Cache (Storage) ─┐ │ +│ │ Recent nodes │ │ +│ │ OPFS/Filesystem/S3 │ │ +│ │ TTL-based │ │ +│ └─────────────────────────┘ │ +│ │ │ +│ ┌─ Cold Storage (Persistent) ─┐ │ +│ │ All nodes │ │ +│ │ OPFS/Filesystem/S3 │ │ +│ │ Batch operations │ │ +│ └─────────────────────────────┘ │ +│ │ +│ Environment-Specific Storage Adapters: │ +│ ┌─────────────┬─────────────┬─────────────┬─────────────┬─────────────┐ │ +│ │ Browser │ Node.js │ Serverless │ Container │ Server │ │ +│ │ OPFS │ FileSystem │ In-Memory │ Adaptive │ S3/Cloud │ │ +│ │ (Fallback: │ (Backup: │ (Optional: │ (Auto- │ (Multi- │ │ +│ │ IndexedDB) │ S3/Cloud) │ S3/Cloud) │ Detect) │ Provider) │ │ +│ └─────────────┴─────────────┴─────────────┴─────────────┴─────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Augmentation Pipeline System + +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ AUGMENTATION PIPELINE FLOW │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Raw Data → [SENSE] → [MEMORY] → [COGNITION] → [CONDUIT] → [ACTIVATION] → │ +│ │ │ │ │ │ │ +│ ▼ ▼ ▼ ▼ ▼ │ +│ Process Storage Reasoning Data Sync Actions │ +│ Input Persist Inference External Triggers │ +│ Convert Retrieve Logic Ops Systems Responses │ +│ │ +│ → [PERCEPTION] → [DIALOG] → [WEBSOCKET] → │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ Visualization NLP/Chat Real-time │ +│ Interpretation Response Streaming │ +│ Organization Context Communication │ +│ │ +│ Execution Modes: │ +│ • SEQUENTIAL: Step-by-step processing │ +│ • PARALLEL: Concurrent augmentation execution │ +│ • THREADED: Multi-threaded with worker pools │ +└─────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Performance & Scaling Features + +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ PERFORMANCE OPTIMIZATIONS │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────────┐ │ +│ │ MULTITHREADING │ │ +│ │ │ │ +│ │ Main Thread ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ +│ │ ├──────→ │ Worker 1 │ │ Worker 2 │ │ Worker N │ │ │ +│ │ │ │Embedding │ │ Search │ │ Batch │ │ │ +│ │ │ │Generation│ │Operations│ │Processing│ │ │ +│ │ ←──────── └──────────┘ └──────────┘ └──────────┘ │ │ +│ │ │ │ +│ │ • Web Workers (Browser) / Worker Threads (Node.js) │ │ +│ │ • Model caching and reuse across workers │ │ +│ │ • Batch embedding for better performance │ │ +│ └─────────────────────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────────┐ │ +│ │ GPU ACCELERATION │ │ +│ │ │ │ +│ │ TensorFlow.js → WebGL Backend → GPU │ │ +│ │ ↓ │ │ +│ │ Fallback: CPU Backend for compatibility │ │ +│ │ │ │ +│ │ • Vector similarity calculations │ │ +│ │ • Embedding generation │ │ +│ │ • Tensor operations │ │ +│ │ • Automatic memory management │ │ +│ └─────────────────────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────────┐ │ +│ │ INTELLIGENT CACHING │ │ +│ │ │ │ +│ │ • Auto-tuning based on usage patterns │ │ +│ │ • Memory-aware cache sizing │ │ +│ │ • Prefetching strategies │ │ +│ │ • LRU eviction with batch processing │ │ +│ │ • Read-only mode optimizations │ │ +│ └─────────────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Cross-Platform Integration + +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ SYNCHRONIZATION & SCALING │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ Browser ←→ WebSocket ←→ Server ←→ S3/Cloud Storage │ +│ ↓ ↓ │ +│ Browser ←→ WebRTC ←→ Browser (Peer-to-Peer) │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────────┐ │ +│ │ CONDUIT AUGMENTATIONS │ │ +│ │ │ │ +│ │ WebSocket iConduit: │ │ +│ │ • Browser ↔ Server sync │ │ +│ │ • Server ↔ Server sync │ │ +│ │ • Real-time data streaming │ │ +│ │ │ │ +│ │ WebRTC iConduit: │ │ +│ │ • Direct browser ↔ browser sync │ │ +│ │ • Peer-to-peer without server │ │ +│ │ • Decentralized data sharing │ │ +│ └─────────────────────────────────────────────────────────────────────────┘ │ +│ │ +│ ┌─────────────────────────────────────────────────────────────────────────┐ │ +│ │ MODEL CONTROL PROTOCOL (MCP) │ │ +│ │ │ │ +│ │ External AI Models ←→ MCP Server ←→ Brainy Data & Tools │ │ +│ │ │ │ +│ │ • BrainyMCPAdapter: Data access for external models │ │ +│ │ • MCPAugmentationToolset: Pipeline tools for models │ │ +│ │ • BrainyMCPService: WebSocket & REST integration │ │ +│ └─────────────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Data Flow Example + +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ DATA FLOW EXAMPLE │ +├─────────────────────────────────────────────────────────────────────────────────┤ +│ │ +│ 1. Input: "Cats are independent pets" │ +│ ↓ │ +│ 2. SENSE Augmentation: Process raw text │ +│ ↓ │ +│ 3. Embedding: TensorFlow USE → [0.123, -0.456, 0.789, ...] │ +│ ↓ │ +│ 4. MEMORY Augmentation: Store with metadata │ +│ ↓ │ +│ 5. HNSW Index: Add vector to hierarchical graph │ +│ ↓ │ +│ 6. Storage: Persist to OPFS/FileSystem/S3 │ +│ │ +│ Query: "feline pets" → Embedding → HNSW Search → Ranked Results │ +│ Result: [{text: "Cats are independent pets", similarity: 0.89, id: "123"}] │ +│ │ +│ Relationship Example: │ +│ addVerb(catId, dogId, VerbType.RelatedTo, {description: "Both are pets"}) │ +│ ↓ │ +│ Graph: [Cat] ──RelatedTo──→ [Dog] │ +└─────────────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +**Key Architecture Principles:** + +1. **Environment Agnostic**: Automatically adapts to browser, Node.js, serverless, container, or server environments +2. **Intelligent Storage**: Multi-tier caching with automatic storage selection (OPFS, filesystem, S3, memory) +3. **Vector + Graph**: Combines semantic vector search with graph relationships in a unified model +4. **Extensible Pipeline**: Modular augmentation system for custom processing and integration +5. **Performance Optimized**: GPU acceleration, multithreading, intelligent caching, and memory management +6. **Scalable Sync**: WebSocket and WebRTC conduits for real-time synchronization across instances +7. **AI Integration**: MCP protocol for external AI model integration and tool access \ No newline at end of file diff --git a/dev/scripts/generate-architecture-pdf.js b/dev/scripts/generate-architecture-pdf.js new file mode 100755 index 00000000..ff13d8ce --- /dev/null +++ b/dev/scripts/generate-architecture-pdf.js @@ -0,0 +1,1421 @@ +#!/usr/bin/env node + +/** + * Generate Professional PDF from Brainy Architecture Documentation + * + * This script converts the Markdown documentation with Mermaid diagrams + * into a beautiful, professional PDF using Puppeteer and modern CSS styling. + */ + +import puppeteer from 'puppeteer' +import fs from 'fs' +import path from 'path' +import { fileURLToPath } from 'url' + +const __filename = fileURLToPath(import.meta.url) +const __dirname = path.dirname(__filename) + +// Configuration +const config = { + readmeFile: path.join(__dirname, '../../README.md'), + architectureFile: path.join(__dirname, '../docs/brainy_architecture_diagram.md'), + outputFile: path.join(__dirname, '../../docs/Brainy_Architecture_Documentation.pdf') +} + +// Apple-inspired CSS styling with Brainy retro sci-fi aesthetic +const appleInspiredCSS = ` + +` + +// SVG Diagram Generation Functions +function generateArchitectureDiagram() { + return ` +
$1')
+
+ // Convert links
+ html = html.replace(/\[([^\]]+)\]\(([^)]+)\)/g, '$1')
+
+ // Convert horizontal rules
+ html = html.replace(/^---$/gm, '') + html = '
' + html + '
' + + // Clean up empty paragraphs + html = html.replace(/<\/p>/g, '') + html = html.replace(/
(\s* (\s* (\s*
)\s*<\/p>/g, '$1')
+ html = html.replace(/' + match + '
'
+ })
+
+ // Handle code blocks
+ html = html.replace(/```(\w+)?\n([\s\S]*?)\n```/g, (match, lang, code) => {
+ return `
`
+ })
+
+ // Convert custom SVG diagram calls
+ html = html.replace(/\$\{generateBrainyLogo\(\)\}/g, generateBrainyLogo())
+ html = html.replace(/\$\{generateArchitectureDiagram\(\)\}/g, generateArchitectureDiagram())
+ html = html.replace(/\$\{generateDataFlowDiagram\(\)\}/g, generateDataFlowDiagram())
+ html = html.replace(/\$\{generateHNSWDiagram\(\)\}/g, generateHNSWDiagram())
+
+ // Handle special divs
+ html = html.replace(/${code}/g, '
')
+
+ // Enhance key features section
+ html = html.replace(/### Key Features([\s\S]*?)(?=
|$)/g, (match) => {
+ return match.replace(/
OPFS Storage]
+ C --> E[Node.js
File System]
+ C --> F[Serverless
In-Memory]
+ C --> G[Container
Adaptive]
+ C --> H[Server
S3/Cloud]
+
+ B --> I[Vector Search Engine]
+ B --> J[Graph Database]
+ B --> K[Augmentation Pipeline]
+
+ style B fill:#e1f5fe
+ style I fill:#f3e5f5
+ style J fill:#e8f5e8
+ style K fill:#fff3e0
+```
+
+### Key Features
+
+- **Universal Compatibility**: Runs everywhere - browsers, Node.js, serverless functions, containers
+- **Intelligent Adaptation**: Automatically optimizes for environment and usage patterns
+- **Dual Nature**: Vector similarity search + graph relationships in one system
+- **Real-time Streaming**: Live data processing through extensible pipeline
+- **AI Integration**: Built-in TensorFlow.js with GPU acceleration
+
+---
+
+## Core Architecture
+
+```mermaid
+graph TB
+ subgraph "Application Layer"
+ API[Brainy Data API
add() | search() | addVerb() | get() | delete()]
+ end
+
+ subgraph "Processing Layer"
+ PIPELINE[Augmentation Pipeline
SENSE → MEMORY → COGNITION → CONDUIT → ACTIVATION → PERCEPTION → DIALOG → WS]
+ end
+
+ subgraph "Engine Layer"
+ EMBED[Embedding Engine
TensorFlow.js Universal Sentence Encoder]
+ VECTOR[Vector Index
HNSW Algorithm]
+ GRAPH[Graph Engine
Noun-Verb Model]
+ end
+
+ subgraph "Storage Layer"
+ CACHE[Multi-tier Caching
Hot → Warm → Cold]
+ STORAGE[Adaptive Storage
OPFS | FileSystem | S3 | Memory]
+ end
+
+ API --> PIPELINE
+ PIPELINE --> EMBED
+ PIPELINE --> VECTOR
+ PIPELINE --> GRAPH
+ EMBED --> CACHE
+ VECTOR --> CACHE
+ GRAPH --> STORAGE
+ CACHE --> STORAGE
+
+ style API fill:#e3f2fd
+ style PIPELINE fill:#f1f8e9
+ style EMBED fill:#fce4ec
+ style VECTOR fill:#fff8e1
+ style GRAPH fill:#e8f5e8
+ style CACHE fill:#f3e5f5
+ style STORAGE fill:#efebe9
+```
+
+---
+
+## Data Model & Graph Structure
+
+### Noun Types (Entities/Nodes)
+
+```mermaid
+mindmap
+ root((Brainy
Noun Types))
+ Core Entities
+ Person
+ Organization
+ Location
+ Thing
+ Concept
+ Event
+ Digital Content
+ Document
+ Media
+ File
+ Message
+ Content
+ Collections
+ Collection
+ Dataset
+ Business/App
+ Product
+ Service
+ User
+ Task
+ Project
+ Descriptive
+ Process
+ State
+ Role
+ Topic
+ Language
+ Currency
+ Measurement
+```
+
+### Verb Types (Relationships/Edges)
+
+```mermaid
+mindmap
+ root((Brainy
Verb Types))
+ Core Relations
+ RelatedTo
+ Contains
+ PartOf
+ LocatedAt
+ References
+ Temporal/Causal
+ Precedes
+ Succeeds
+ Causes
+ DependsOn
+ Requires
+ Creation/Transform
+ Creates
+ Transforms
+ Becomes
+ Modifies
+ Consumes
+ Ownership/Attribution
+ Owns
+ AttributedTo
+ CreatedBy
+ BelongsTo
+ Social/Organizational
+ MemberOf
+ WorksWith
+ FriendOf
+ Follows
+ Likes
+ ReportsTo
+ Supervises
+ Mentors
+ Communicates
+ Descriptive/Functional
+ Describes
+ Defines
+ Categorizes
+ Measures
+ Evaluates
+ Uses
+ Implements
+ Extends
+```
+
+### Graph Example
+
+```mermaid
+graph LR
+ A[Person: John Doe
ID: person-123] -->|WorksWith| B[Organization: Acme Corp
ID: org-456]
+ A -->|CreatedBy| C[Document: Report
ID: doc-789]
+ A -->|LocatedAt| D[Location: New York
ID: loc-101]
+ B -->|Contains| E[Project: AI Initiative
ID: proj-202]
+ C -->|PartOf| E
+ E -->|Uses| F[Concept: Machine Learning
ID: concept-303]
+
+ style A fill:#ffcdd2
+ style B fill:#c8e6c9
+ style C fill:#bbdefb
+ style D fill:#fff9c4
+ style E fill:#f8bbd9
+ style F fill:#d1c4e9
+```
+
+---
+
+## Vector Search Engine
+
+### HNSW Index Structure
+
+```mermaid
+graph TB
+ subgraph "HNSW Hierarchical Structure"
+ subgraph "Layer 2 (Sparse)"
+ L2A((●)) --- L2B((●))
+ L2B --- L2C((●))
+ end
+
+ subgraph "Layer 1 (Medium Density)"
+ L1A((●)) --- L1B((●))
+ L1B --- L1C((●))
+ L1C --- L1D((●))
+ L1D --- L1E((●))
+ L1E --- L1F((●))
+ L1F --- L1G((●))
+ L1G --- L1H((●))
+ end
+
+ subgraph "Layer 0 (Dense Connections)"
+ L0A((●)) --- L0B((●))
+ L0B --- L0C((●))
+ L0C --- L0D((●))
+ L0D --- L0E((●))
+ L0E --- L0F((●))
+ L0F --- L0G((●))
+ L0G --- L0H((●))
+ L0H --- L0I((●))
+ L0I --- L0J((●))
+ L0J --- L0K((●))
+ L0K --- L0L((●))
+ L0L --- L0M((●))
+ L0M --- L0N((●))
+ L0N --- L0O((●))
+ L0O --- L0P((●))
+ end
+
+ L2A -.-> L1A
+ L2A -.-> L1D
+ L2B -.-> L1C
+ L2B -.-> L1F
+ L2C -.-> L1G
+
+ L1A -.-> L0A
+ L1A -.-> L0B
+ L1B -.-> L0C
+ L1B -.-> L0D
+ L1C -.-> L0E
+ L1C -.-> L0F
+ L1D -.-> L0G
+ L1D -.-> L0H
+ L1E -.-> L0I
+ L1E -.-> L0J
+ L1F -.-> L0K
+ L1F -.-> L0L
+ L1G -.-> L0M
+ L1G -.-> L0N
+ L1H -.-> L0O
+ L1H -.-> L0P
+ end
+
+ style L2A fill:#ff9999
+ style L2B fill:#ff9999
+ style L2C fill:#ff9999
+ style L1A fill:#99ccff
+ style L1B fill:#99ccff
+ style L1C fill:#99ccff
+ style L1D fill:#99ccff
+ style L1E fill:#99ccff
+ style L1F fill:#99ccff
+ style L1G fill:#99ccff
+ style L1H fill:#99ccff
+ style L0A fill:#99ff99
+ style L0B fill:#99ff99
+ style L0C fill:#99ff99
+ style L0D fill:#99ff99
+ style L0E fill:#99ff99
+ style L0F fill:#99ff99
+ style L0G fill:#99ff99
+ style L0H fill:#99ff99
+ style L0I fill:#99ff99
+ style L0J fill:#99ff99
+ style L0K fill:#99ff99
+ style L0L fill:#99ff99
+ style L0M fill:#99ff99
+ style L0N fill:#99ff99
+ style L0O fill:#99ff99
+ style L0P fill:#99ff99
+```
+
+### Search Process Flow
+
+```mermaid
+sequenceDiagram
+ participant User
+ participant API
+ participant Embedding
+ participant HNSW
+ participant Storage
+
+ User->>API: searchText("feline pets", 5)
+ API->>Embedding: embed("feline pets")
+ Embedding->>Embedding: TensorFlow.js Universal Sentence Encoder
+ Embedding-->>API: [0.123, -0.456, 0.789, ...]
+
+ API->>HNSW: search(vector, k=5)
+ HNSW->>HNSW: Navigate from top layer
+ HNSW->>HNSW: Descend to lower layers
+ HNSW->>HNSW: Find k nearest neighbors
+ HNSW-->>API: [id1, id2, id3, id4, id5]
+
+ API->>Storage: get([id1, id2, id3, id4, id5])
+ Storage-->>API: [noun1, noun2, noun3, noun4, noun5]
+
+ API-->>User: [{text: "Cats are independent pets", similarity: 0.89}, ...]
+```
+
+---
+
+## Storage Architecture
+
+### Multi-Tier Caching System
+
+```mermaid
+graph TD
+ subgraph "Memory Hierarchy"
+ subgraph "Hot Cache (RAM)"
+ HC[Most Accessed Items
LRU Eviction
Auto-tuned Size
Millisecond Access]
+ end
+
+ subgraph "Warm Cache (Storage)"
+ WC[Recent Items
TTL-based
Sub-second Access
OPFS/FS/S3]
+ end
+
+ subgraph "Cold Storage (Persistent)"
+ CS[All Items
Batch Operations
Full Persistence
OPFS/FS/S3]
+ end
+ end
+
+ subgraph "Environment Adapters"
+ Browser[Browser
OPFS → IndexedDB]
+ NodeJS[Node.js
FileSystem → S3]
+ Serverless[Serverless
Memory → S3]
+ Container[Container
Auto-detect]
+ Server[Server
S3/Multi-cloud]
+ end
+
+ User[User Query] --> HC
+ HC -->|Cache Miss| WC
+ WC -->|Cache Miss| CS
+
+ CS --> Browser
+ CS --> NodeJS
+ CS --> Serverless
+ CS --> Container
+ CS --> Server
+
+ style HC fill:#ffcdd2
+ style WC fill:#fff9c4
+ style CS fill:#c8e6c9
+ style Browser fill:#e1f5fe
+ style NodeJS fill:#e8f5e8
+ style Serverless fill:#f3e5f5
+ style Container fill:#fff3e0
+ style Server fill:#efebe9
+```
+
+### Storage Performance Characteristics
+
+```mermaid
+xychart-beta
+ title "Storage Performance by Environment"
+ x-axis [Browser, Node.js, Serverless, Container, Server]
+ y-axis "Latency (ms)" 0 --> 1000
+ line [50, 10, 200, 30, 100]
+```
+
+---
+
+## Augmentation Pipeline
+
+### Pipeline Flow Architecture
+
+```mermaid
+flowchart LR
+ subgraph "Data Processing Pipeline"
+ Input[Raw Data] --> SENSE[SENSE
Process Input
Convert & Validate]
+ SENSE --> MEMORY[MEMORY
Storage Operations
Persist & Retrieve]
+ MEMORY --> COGNITION[COGNITION
Reasoning
Inference & Logic]
+ COGNITION --> CONDUIT[CONDUIT
Data Sync
External Systems]
+ CONDUIT --> ACTIVATION[ACTIVATION
Actions
Triggers & Events]
+ ACTIVATION --> PERCEPTION[PERCEPTION
Visualization
Interpretation]
+ PERCEPTION --> DIALOG[DIALOG
NLP & Chat
Context & Response]
+ DIALOG --> WEBSOCKET[WEBSOCKET
Real-time
Streaming & Sync]
+ WEBSOCKET --> Output[Processed Output]
+ end
+
+ subgraph "Execution Modes"
+ SEQ[Sequential
Step-by-step]
+ PAR[Parallel
Concurrent]
+ THR[Threaded
Worker Pools]
+ end
+
+ Input -.-> SEQ
+ Input -.-> PAR
+ Input -.-> THR
+
+ style SENSE fill:#e8f5e8
+ style MEMORY fill:#e3f2fd
+ style COGNITION fill:#fff3e0
+ style CONDUIT fill:#f3e5f5
+ style ACTIVATION fill:#ffebee
+ style PERCEPTION fill:#e0f2f1
+ style DIALOG fill:#fce4ec
+ style WEBSOCKET fill:#e8eaf6
+```
+
+### Augmentation Types Detail
+
+```mermaid
+mindmap
+ root((Augmentation
System))
+ SENSE
+ Process Raw Data
+ Listen to Feeds
+ Data Validation
+ Format Conversion
+ MEMORY
+ Store Data
+ Retrieve Data
+ Update Data
+ Delete Data
+ List Keys
+ COGNITION
+ Reason
+ Infer
+ Execute Logic
+ Pattern Recognition
+ CONDUIT
+ Establish Connection
+ Read Data
+ Write Data
+ Monitor Stream
+ Sync Instances
+ ACTIVATION
+ Trigger Actions
+ Generate Output
+ Interact External
+ Event Handling
+ PERCEPTION
+ Interpret Data
+ Organize Info
+ Generate Visualization
+ Context Analysis
+ DIALOG
+ Process User Input
+ Generate Response
+ Manage Context
+ NLP Operations
+ WEBSOCKET
+ Connect WebSocket
+ Send Messages
+ Message Callbacks
+ Stream Monitoring
+```
+
+---
+
+## Performance Optimizations
+
+### Multithreading Architecture
+
+```mermaid
+graph TB
+ subgraph "Main Thread"
+ MT[Main Thread
Coordination & API]
+ end
+
+ subgraph "Worker Pool"
+ W1[Worker 1
Embedding
Generation]
+ W2[Worker 2
Vector
Search]
+ W3[Worker 3
Batch
Processing]
+ WN[Worker N
Custom
Operations]
+ end
+
+ subgraph "GPU Acceleration"
+ GPU[TensorFlow.js
WebGL Backend
GPU Compute]
+ CPU[CPU Fallback
Compatibility
Mode]
+ end
+
+ MT -->|Distribute Tasks| W1
+ MT -->|Distribute Tasks| W2
+ MT -->|Distribute Tasks| W3
+ MT -->|Distribute Tasks| WN
+
+ W1 --> GPU
+ W2 --> GPU
+ W3 --> GPU
+ WN --> GPU
+
+ GPU -.->|Fallback| CPU
+
+ W1 -->|Results| MT
+ W2 -->|Results| MT
+ W3 -->|Results| MT
+ WN -->|Results| MT
+
+ style MT fill:#e3f2fd
+ style W1 fill:#e8f5e8
+ style W2 fill:#e8f5e8
+ style W3 fill:#e8f5e8
+ style WN fill:#e8f5e8
+ style GPU fill:#ffebee
+ style CPU fill:#fff3e0
+```
+
+### Performance Metrics
+
+```mermaid
+xychart-beta
+ title "Performance Improvements with Optimizations"
+ x-axis [Baseline, Caching, Multithreading, GPU, All Combined]
+ y-axis "Operations/Second" 0 --> 10000
+ bar [1000, 3000, 5000, 7000, 9500]
+```
+
+---
+
+## Cross-Platform Integration
+
+### Synchronization Network
+
+```mermaid
+graph TB
+ subgraph "Browser Instances"
+ B1[Browser 1]
+ B2[Browser 2]
+ B3[Browser 3]
+ end
+
+ subgraph "Server Infrastructure"
+ WS[WebSocket Server]
+ API[REST API Server]
+ S3[S3/Cloud Storage]
+ end
+
+ subgraph "Peer-to-Peer"
+ STUN[STUN Server]
+ SIGNAL[Signaling Server]
+ end
+
+ subgraph "External AI"
+ MCP[MCP Server]
+ AI[AI Models]
+ end
+
+ B1 <-->|WebSocket| WS
+ B2 <-->|WebSocket| WS
+ B3 <-->|WebSocket| WS
+
+ B1 <-.->|WebRTC| B2
+ B2 <-.->|WebRTC| B3
+ B1 <-.->|WebRTC| B3
+
+ WS <--> S3
+ API <--> S3
+
+ B1 -.->|Signaling| SIGNAL
+ B2 -.->|Signaling| SIGNAL
+ B3 -.->|Signaling| SIGNAL
+
+ SIGNAL -.-> STUN
+
+ WS <--> MCP
+ MCP <--> AI
+
+ style B1 fill:#e3f2fd
+ style B2 fill:#e3f2fd
+ style B3 fill:#e3f2fd
+ style WS fill:#e8f5e8
+ style API fill:#e8f5e8
+ style S3 fill:#fff3e0
+ style MCP fill:#f3e5f5
+ style AI fill:#ffebee
+```
+
+### Model Control Protocol (MCP) Integration
+
+```mermaid
+sequenceDiagram
+ participant AI as External AI Model
+ participant MCP as MCP Server
+ participant Adapter as Brainy MCP Adapter
+ participant Brainy as Brainy Database
+
+ AI->>MCP: Request data access
+ MCP->>Adapter: Forward request
+ Adapter->>Brainy: Query data
+ Brainy-->>Adapter: Return results
+ Adapter-->>MCP: Formatted response
+ MCP-->>AI: Data payload
+
+ AI->>MCP: Execute augmentation
+ MCP->>Adapter: Pipeline request
+ Adapter->>Brainy: Run augmentation
+ Brainy-->>Adapter: Processing result
+ Adapter-->>MCP: Tool response
+ MCP-->>AI: Execution result
+```
+
+---
+
+## Data Flow Example
+
+### Complete Processing Pipeline
+
+```mermaid
+flowchart TD
+ subgraph "Input Processing"
+ I1[Input: "Cats are independent pets"]
+ I2[Metadata: {noun: "Thing", category: "animal"}]
+ end
+
+ subgraph "Embedding Generation"
+ E1[TensorFlow.js Universal Sentence Encoder]
+ E2[Vector: [0.123, -0.456, 0.789, ...]]
+ end
+
+ subgraph "Storage & Indexing"
+ S1[Store in Multi-tier Cache]
+ S2[Add to HNSW Index]
+ S3[Persist to Storage Layer]
+ end
+
+ subgraph "Query Processing"
+ Q1[Query: "feline pets"]
+ Q2[Generate Query Vector]
+ Q3[HNSW Similarity Search]
+ Q4[Retrieve & Rank Results]
+ end
+
+ subgraph "Graph Operations"
+ G1[Add Relationship]
+ G2[catId --RelatedTo--> dogId]
+ G3[Store Verb Metadata]
+ end
+
+ I1 --> E1
+ I2 --> E1
+ E1 --> E2
+ E2 --> S1
+ S1 --> S2
+ S2 --> S3
+
+ Q1 --> Q2
+ Q2 --> Q3
+ Q3 --> Q4
+
+ E2 -.-> G1
+ G1 --> G2
+ G2 --> G3
+
+ style I1 fill:#e8f5e8
+ style E1 fill:#e3f2fd
+ style E2 fill:#f3e5f5
+ style S1 fill:#fff3e0
+ style Q1 fill:#e8f5e8
+ style Q4 fill:#ffebee
+ style G2 fill:#e0f2f1
+```
+
+### Result Example
+
+```json
+{
+ "results": [
+ {
+ "id": "noun-123",
+ "text": "Cats are independent pets",
+ "similarity": 0.89,
+ "metadata": {
+ "noun": "Thing",
+ "category": "animal"
+ }
+ }
+ ],
+ "query": "feline pets",
+ "processingTime": "15ms",
+ "cacheHit": false
+}
+```
+
+---
+
+## Key Architecture Principles
+
+### 🌐 **Environment Agnostic**
+Automatically adapts to browser, Node.js, serverless, container, or server environments without code changes.
+
+### 🧠 **Intelligent Storage**
+Multi-tier caching with automatic storage selection optimizes for performance and persistence across platforms.
+
+### 🔍 **Vector + Graph Unified**
+Combines semantic vector search with graph relationships in a single, coherent data model.
+
+### 🔧 **Extensible Pipeline**
+Modular augmentation system allows custom processing, AI integration, and workflow automation.
+
+### ⚡ **Performance Optimized**
+GPU acceleration, multithreading, intelligent caching, and memory management deliver enterprise-grade performance.
+
+### 🔄 **Scalable Synchronization**
+WebSocket and WebRTC conduits enable real-time synchronization across instances and platforms.
+
+### 🤖 **AI Integration Ready**
+Built-in MCP protocol support allows external AI models to access Brainy data and utilize augmentation tools.
+
+---
+
+*Generated from Brainy v0.34.0 Architecture Documentation*
\ No newline at end of file
diff --git a/package-lock.json b/package-lock.json
index b24a88a6..5e5eda54 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -39,7 +39,7 @@
"happy-dom": "^18.0.1",
"jsdom": "^26.1.0",
"node-fetch": "^3.3.2",
- "puppeteer": "^22.5.0",
+ "puppeteer": "^22.15.0",
"rollup": "^4.13.0",
"rollup-plugin-terser": "^7.0.2",
"standard-version": "^9.5.0",
diff --git a/package.json b/package.json
index ef6a7ee9..ea988e6e 100644
--- a/package.json
+++ b/package.json
@@ -72,6 +72,7 @@
"test:specialized": "vitest run tests/specialized-scenarios.test.ts",
"test:performance": "vitest run tests/performance.test.ts",
"test:comprehensive": "npm run test:error-handling && npm run test:edge-cases && npm run test:storage && npm run test:environments && npm run test:specialized",
+ "_generate-pdf": "node dev/scripts/generate-architecture-pdf.js",
"_release": "standard-version",
"_release:patch": "standard-version --release-as patch",
"_release:minor": "standard-version --release-as minor",
@@ -151,7 +152,7 @@
"happy-dom": "^18.0.1",
"jsdom": "^26.1.0",
"node-fetch": "^3.3.2",
- "puppeteer": "^22.5.0",
+ "puppeteer": "^22.15.0",
"rollup": "^4.13.0",
"rollup-plugin-terser": "^7.0.2",
"standard-version": "^9.5.0",