**feat(docs): add comprehensive architecture documentation for Brainy**
- **Documentation Additions**:
- Created `brainy_architecture_diagram.md` to detail Brainy's architecture using diagrams and structured descriptions:
- Added overviews of the system, core architecture, and augmentation pipeline.
- Defined data models, graph structures, storage architecture, and performance optimizations.
- Explained vector search engine design, HNSW index structure, and usage flow examples.
- Developed `brainy_architecture_visual.md` to complement the architecture with visual aids in Mermaid.js:
- Provided detailed flowcharts, mind maps, and sequence diagrams for system components and data flow.
- **Purpose**:
- Provide in-depth technical insights into Brainy's architecture for developers and stakeholders.
- Enhance understanding of the system's core design principles with easy-to-follow diagrams and examples.
This commit is contained in:
parent
672be32bea
commit
5f9e807952
10 changed files with 2842 additions and 2 deletions
242
dev/docs/PDF_GENERATION_GUIDE.md
Normal file
242
dev/docs/PDF_GENERATION_GUIDE.md
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
# Brainy Architecture PDF Generation Guide
|
||||
|
||||
This guide shows you how to generate a professional PDF from the Brainy architecture documentation with beautiful diagrams.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Option 1: Using the npm script (Recommended)
|
||||
|
||||
```bash
|
||||
# Make sure you're in the brainy project directory
|
||||
cd /path/to/brainy
|
||||
|
||||
# Install dependencies if not already installed
|
||||
npm install
|
||||
|
||||
# Generate the PDF
|
||||
npm run generate-pdf
|
||||
```
|
||||
|
||||
### Option 2: Direct script execution
|
||||
|
||||
```bash
|
||||
# Make sure you're in the brainy project directory
|
||||
cd /path/to/brainy
|
||||
|
||||
# Install Puppeteer if not already installed
|
||||
npm install puppeteer
|
||||
|
||||
# Run the script directly
|
||||
node dev/dev/scripts/generate-architecture-pdf.js
|
||||
```
|
||||
|
||||
## Installation Requirements
|
||||
|
||||
### Prerequisites
|
||||
- Node.js 18+
|
||||
- npm or yarn
|
||||
|
||||
### Dependencies
|
||||
The script uses:
|
||||
- **Puppeteer**: For PDF generation and browser automation
|
||||
- **Mermaid**: For rendering diagrams (loaded via CDN)
|
||||
- **Google Fonts**: For professional typography (loaded via CDN)
|
||||
|
||||
### Install Dependencies
|
||||
|
||||
```bash
|
||||
# If you don't have puppeteer installed globally or in the project
|
||||
npm install puppeteer
|
||||
|
||||
# Or install as dev dependency
|
||||
npm install --save-dev puppeteer
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
The PDF will be generated at:
|
||||
```
|
||||
docs/Brainy_Architecture_Documentation.pdf
|
||||
```
|
||||
|
||||
## Features of the Generated PDF
|
||||
|
||||
### Professional Styling
|
||||
- **Modern Typography**: Uses Inter font family for clean, readable text
|
||||
- **Code Font**: JetBrains Mono for code blocks and technical content
|
||||
- **Color Scheme**: Professional blue theme with proper contrast
|
||||
- **Layout**: A4 format with proper margins and spacing
|
||||
|
||||
### Rich Diagrams
|
||||
- **Mermaid Diagrams**: All diagrams are rendered as vector graphics
|
||||
- **Interactive Elements**: Flowcharts, sequence diagrams, mindmaps, and more
|
||||
- **Consistent Styling**: All diagrams follow the same color scheme
|
||||
- **High Quality**: Vector-based rendering for crisp output
|
||||
|
||||
### Document Structure
|
||||
- **Table of Contents**: Linked navigation
|
||||
- **Page Headers/Footers**: Professional branding and page numbers
|
||||
- **Section Breaks**: Logical page breaks between major sections
|
||||
- **Code Highlighting**: Syntax highlighting for JSON and code blocks
|
||||
|
||||
## Customization
|
||||
|
||||
### Modify Styling
|
||||
Edit the `professionalCSS` variable in `dev/scripts/generate-architecture-pdf.js`:
|
||||
|
||||
```javascript
|
||||
const professionalCSS = `
|
||||
/* Your custom CSS here */
|
||||
h1 {
|
||||
color: #your-color;
|
||||
font-size: 24pt;
|
||||
}
|
||||
/* ... */
|
||||
`
|
||||
```
|
||||
|
||||
### Change Output Location
|
||||
Modify the `config` object:
|
||||
|
||||
```javascript
|
||||
const config = {
|
||||
inputFile: path.join(__dirname, '../docs/brainy_architecture_visual.md'),
|
||||
outputFile: path.join(__dirname, '../docs/YOUR_CUSTOM_NAME.pdf'),
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### Adjust PDF Settings
|
||||
Modify the `page.pdf()` options:
|
||||
|
||||
```javascript
|
||||
await page.pdf({
|
||||
path: config.outputFile,
|
||||
format: 'A4', // or 'Letter', 'Legal', etc.
|
||||
printBackground: true,
|
||||
margin: {
|
||||
top: '20mm',
|
||||
right: '15mm',
|
||||
bottom: '20mm',
|
||||
left: '15mm'
|
||||
},
|
||||
// ... other options
|
||||
})
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. "Puppeteer not found"
|
||||
```bash
|
||||
npm install puppeteer
|
||||
```
|
||||
|
||||
#### 2. "Chrome/Chromium not found"
|
||||
```bash
|
||||
# On Ubuntu/Debian
|
||||
sudo apt-get install chromium-browser
|
||||
|
||||
# On macOS
|
||||
brew install chromium
|
||||
|
||||
# Or let Puppeteer download Chromium
|
||||
npm install puppeteer --unsafe-perm=true
|
||||
```
|
||||
|
||||
#### 3. "Permission denied"
|
||||
```bash
|
||||
chmod +x dev/scripts/generate-architecture-pdf.js
|
||||
```
|
||||
|
||||
#### 4. "Diagrams not rendering"
|
||||
Check your internet connection - Mermaid is loaded from CDN. For offline use, you can download mermaid.min.js locally and update the path.
|
||||
|
||||
### Advanced Configuration
|
||||
|
||||
#### Use Local Mermaid
|
||||
Download mermaid.min.js and update the config:
|
||||
|
||||
```javascript
|
||||
const config = {
|
||||
// ...
|
||||
mermaidCDN: './path/to/mermaid.min.js'
|
||||
}
|
||||
```
|
||||
|
||||
#### Custom Fonts
|
||||
Add additional fonts to the CSS:
|
||||
|
||||
```css
|
||||
@import url('https://fonts.googleapis.com/css2?family=YourFont:wght@400;500;600&display=swap');
|
||||
|
||||
body {
|
||||
font-family: 'YourFont', sans-serif;
|
||||
}
|
||||
```
|
||||
|
||||
## Adding to package.json
|
||||
|
||||
Add this script to your `package.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"generate-pdf": "node dev/dev/scripts/generate-architecture-pdf.js",
|
||||
"docs:pdf": "npm run generate-pdf"
|
||||
},
|
||||
"devDependencies": {
|
||||
"puppeteer": "^22.5.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Alternative PDF Generators
|
||||
|
||||
If you prefer other tools, you can also use:
|
||||
|
||||
### 1. Pandoc + LaTeX
|
||||
```bash
|
||||
# Install pandoc and latex
|
||||
sudo apt-get install pandoc texlive-latex-recommended
|
||||
|
||||
# Convert (note: won't render Mermaid diagrams)
|
||||
pandoc docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf
|
||||
```
|
||||
|
||||
### 2. mdpdf
|
||||
```bash
|
||||
npm install -g mdpdf
|
||||
mdpdf docs/brainy_architecture_visual.md --output=docs/brainy_architecture.pdf
|
||||
```
|
||||
|
||||
### 3. markdown-pdf
|
||||
```bash
|
||||
npm install -g markdown-pdf
|
||||
markdown-pdf docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf
|
||||
```
|
||||
|
||||
**Note**: The custom Puppeteer script provides the best results with proper Mermaid diagram rendering and professional styling.
|
||||
|
||||
## Sample Output
|
||||
|
||||
The generated PDF will include:
|
||||
|
||||
1. **Cover Page** with title and subtitle
|
||||
2. **Table of Contents** with page links
|
||||
3. **System Overview** with environment detection diagram
|
||||
4. **Core Architecture** with layered architecture diagram
|
||||
5. **Data Model** with noun/verb type hierarchies
|
||||
6. **Vector Search Engine** with HNSW visualization
|
||||
7. **Storage Architecture** with multi-tier caching diagrams
|
||||
8. **Augmentation Pipeline** with flow diagrams
|
||||
9. **Performance Optimizations** with threading models
|
||||
10. **Integration Patterns** with network topology
|
||||
11. **Data Flow Examples** with sequence diagrams
|
||||
|
||||
Total pages: ~25-30 pages with full diagrams and explanations.
|
||||
|
||||
---
|
||||
|
||||
*For questions or issues with PDF generation, please check the troubleshooting section or create an issue in the repository.*
|
||||
80
dev/docs/QUICK_PDF_SETUP.md
Normal file
80
dev/docs/QUICK_PDF_SETUP.md
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
# Quick PDF Generation Setup
|
||||
|
||||
## 🚀 Generate Professional Brainy Architecture PDF
|
||||
|
||||
### One-Command Setup & Generation
|
||||
|
||||
```bash
|
||||
# Install Puppeteer and generate PDF in one go
|
||||
npm install puppeteer && npm run generate-pdf
|
||||
```
|
||||
|
||||
### Step-by-Step
|
||||
|
||||
1. **Install Puppeteer** (if not already installed):
|
||||
```bash
|
||||
npm install puppeteer
|
||||
```
|
||||
|
||||
2. **Generate the PDF**:
|
||||
```bash
|
||||
npm run generate-pdf
|
||||
```
|
||||
|
||||
3. **Find your PDF**:
|
||||
```
|
||||
docs/Brainy_Architecture_Documentation.pdf
|
||||
```
|
||||
|
||||
## ✨ What You Get
|
||||
|
||||
- **25-30 page professional PDF** with full diagrams
|
||||
- **Vector graphics** for all Mermaid diagrams
|
||||
- **Modern typography** with Inter font family
|
||||
- **Consistent branding** throughout the document
|
||||
- **Table of contents** with page links
|
||||
- **Professional headers/footers**
|
||||
|
||||
## 📊 Sample Sections Include
|
||||
|
||||
- System Overview with environment detection
|
||||
- Core Architecture layers
|
||||
- Data Model (23 Noun Types, 38 Verb Types)
|
||||
- Vector Search Engine with HNSW visualization
|
||||
- Storage Architecture with multi-tier caching
|
||||
- Augmentation Pipeline flows
|
||||
- Performance optimizations
|
||||
- Cross-platform integration patterns
|
||||
- Real data flow examples
|
||||
|
||||
## 🛠️ Troubleshooting
|
||||
|
||||
### Issue: "Puppeteer not found"
|
||||
```bash
|
||||
npm install puppeteer
|
||||
```
|
||||
|
||||
### Issue: "Chrome not found"
|
||||
```bash
|
||||
# Let Puppeteer download Chromium
|
||||
npm install puppeteer --unsafe-perm=true
|
||||
```
|
||||
|
||||
### Issue: "Permission denied"
|
||||
```bash
|
||||
chmod +x dev/scripts/generate-architecture-pdf.js
|
||||
```
|
||||
|
||||
## 🎨 Customization
|
||||
|
||||
Edit `dev/scripts/generate-architecture-pdf.js` to:
|
||||
- Change colors and fonts
|
||||
- Modify page layout
|
||||
- Adjust diagram styling
|
||||
- Add custom branding
|
||||
|
||||
---
|
||||
|
||||
**Ready to generate?** Run `npm run generate-pdf` and get your professional architecture documentation!
|
||||
|
||||
For detailed setup instructions, see `PDF_GENERATION_GUIDE.md`.
|
||||
313
dev/docs/brainy_architecture_diagram.md
Normal file
313
dev/docs/brainy_architecture_diagram.md
Normal file
|
|
@ -0,0 +1,313 @@
|
|||
# Brainy Architecture Diagram
|
||||
|
||||
## System Overview
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ BRAINY PLATFORM │
|
||||
│ Vector Graph Database with AI Pipeline │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ ENVIRONMENT DETECTION │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ Browser │ Node.js │ Serverless │ Container │ Server │
|
||||
│ (OPFS) │ (File System) │ (In-Memory) │ (Adaptive) │ (S3/Cloud) │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
```
|
||||
|
||||
## Core Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ BRAINY DATA API │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ add() │ search() │ addVerb() │ get() │ delete() │ backup() │ restore() │ etc. │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ AUGMENTATION PIPELINE │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ SENSE → MEMORY → COGNITION → CONDUIT → ACTIVATION → PERCEPTION → DIALOG → WS │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ DATA PROCESSING │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ Text/JSON → Embedding → Vector Storage │
|
||||
│ │ │
|
||||
│ ┌─────────────────────────┼─────────────────────────┐ │
|
||||
│ │ EMBEDDING │ VECTOR INDEX │ │
|
||||
│ │ │ │ │
|
||||
│ │ TensorFlow.js │ HNSW Algorithm │ │
|
||||
│ │ Universal Sentence │ - Hierarchical │ │
|
||||
│ │ Encoder (USE) │ - Fast Similarity │ │
|
||||
│ │ - GPU Acceleration │ - Configurable │ │
|
||||
│ │ - Batch Processing │ - Memory Efficient │ │
|
||||
│ │ - Worker Threads │ - Product Quantized │ │
|
||||
│ └─────────────────────────┼─────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
```
|
||||
|
||||
## Data Model & Graph Structure
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ GRAPH DATA MODEL │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ NOUNS (Entities/Nodes) │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Core Entity Types: │ Digital/Content Types: │ │
|
||||
│ │ • Person │ • Document │ │
|
||||
│ │ • Organization │ • Media │ │
|
||||
│ │ • Location │ • File │ │
|
||||
│ │ • Thing │ • Message │ │
|
||||
│ │ • Concept │ • Content │ │
|
||||
│ │ • Event │ │ │
|
||||
│ │ │ Collection Types: │ │
|
||||
│ │ Business/App Types: │ • Collection │ │
|
||||
│ │ • Product │ • Dataset │ │
|
||||
│ │ • Service │ │ │
|
||||
│ │ • User │ Descriptive Types: │ │
|
||||
│ │ • Task │ • Process, State, Role │ │
|
||||
│ │ • Project │ • Topic, Language, Currency, Measurement │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ VERBS (Relationships/Edges) │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Core Relationships: │ Social/Organizational: │ │
|
||||
│ │ • RelatedTo │ • MemberOf, WorksWith │ │
|
||||
│ │ • Contains, PartOf │ • FriendOf, Follows, Likes │ │
|
||||
│ │ • LocatedAt, References │ • ReportsTo, Supervises, Mentors │ │
|
||||
│ │ │ • Communicates │ │
|
||||
│ │ Temporal/Causal: │ │ │
|
||||
│ │ • Precedes, Succeeds │ Descriptive/Functional: │ │
|
||||
│ │ • Causes, DependsOn │ • Describes, Defines, Categorizes │ │
|
||||
│ │ • Requires │ • Measures, Evaluates │ │
|
||||
│ │ │ • Uses, Implements, Extends │ │
|
||||
│ │ Creation/Transformation: │ │ │
|
||||
│ │ • Creates, Transforms │ Ownership/Attribution: │ │
|
||||
│ │ • Becomes, Modifies │ • Owns, AttributedTo │ │
|
||||
│ │ • Consumes │ • CreatedBy, BelongsTo │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Vector Storage & Search Engine
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ VECTOR SEARCH ENGINE │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Query Text/Vector → Embedding → HNSW Search → Ranked Results │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ HNSW INDEX STRUCTURE │ │
|
||||
│ │ │ │
|
||||
│ │ Layer 2: ●────────●────────● (Sparse connections) │ │
|
||||
│ │ ╱│ │ │╲ │ │
|
||||
│ │ Layer 1: ●─●──●─●─●─●──●─●─●─● (Medium density) │ │
|
||||
│ │ ╱│││││││││││││││││││││╲ │ │
|
||||
│ │ Layer 0: ●●●●●●●●●●●●●●●●●●●●●●● (Dense connections) │ │
|
||||
│ │ │ │
|
||||
│ │ • Hierarchical navigation for fast search │ │
|
||||
│ │ • Configurable M (max connections), efConstruction, efSearch │ │
|
||||
│ │ • Memory-efficient with disk-based storage for large datasets │ │
|
||||
│ │ • Product quantization for dimensionality reduction │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ ADAPTIVE STORAGE │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─ Hot Cache (RAM) ──┐ │
|
||||
│ │ Most accessed │ │
|
||||
│ │ LRU eviction │ │
|
||||
│ │ Auto-tuned size │ │
|
||||
│ └─────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─ Warm Cache (Storage) ─┐ │
|
||||
│ │ Recent nodes │ │
|
||||
│ │ OPFS/Filesystem/S3 │ │
|
||||
│ │ TTL-based │ │
|
||||
│ └─────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─ Cold Storage (Persistent) ─┐ │
|
||||
│ │ All nodes │ │
|
||||
│ │ OPFS/Filesystem/S3 │ │
|
||||
│ │ Batch operations │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ │
|
||||
│ Environment-Specific Storage Adapters: │
|
||||
│ ┌─────────────┬─────────────┬─────────────┬─────────────┬─────────────┐ │
|
||||
│ │ Browser │ Node.js │ Serverless │ Container │ Server │ │
|
||||
│ │ OPFS │ FileSystem │ In-Memory │ Adaptive │ S3/Cloud │ │
|
||||
│ │ (Fallback: │ (Backup: │ (Optional: │ (Auto- │ (Multi- │ │
|
||||
│ │ IndexedDB) │ S3/Cloud) │ S3/Cloud) │ Detect) │ Provider) │ │
|
||||
│ └─────────────┴─────────────┴─────────────┴─────────────┴─────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Augmentation Pipeline System
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ AUGMENTATION PIPELINE FLOW │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Raw Data → [SENSE] → [MEMORY] → [COGNITION] → [CONDUIT] → [ACTIVATION] → │
|
||||
│ │ │ │ │ │ │
|
||||
│ ▼ ▼ ▼ ▼ ▼ │
|
||||
│ Process Storage Reasoning Data Sync Actions │
|
||||
│ Input Persist Inference External Triggers │
|
||||
│ Convert Retrieve Logic Ops Systems Responses │
|
||||
│ │
|
||||
│ → [PERCEPTION] → [DIALOG] → [WEBSOCKET] → │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ Visualization NLP/Chat Real-time │
|
||||
│ Interpretation Response Streaming │
|
||||
│ Organization Context Communication │
|
||||
│ │
|
||||
│ Execution Modes: │
|
||||
│ • SEQUENTIAL: Step-by-step processing │
|
||||
│ • PARALLEL: Concurrent augmentation execution │
|
||||
│ • THREADED: Multi-threaded with worker pools │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Performance & Scaling Features
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ PERFORMANCE OPTIMIZATIONS │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ MULTITHREADING │ │
|
||||
│ │ │ │
|
||||
│ │ Main Thread ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
|
||||
│ │ ├──────→ │ Worker 1 │ │ Worker 2 │ │ Worker N │ │ │
|
||||
│ │ │ │Embedding │ │ Search │ │ Batch │ │ │
|
||||
│ │ │ │Generation│ │Operations│ │Processing│ │ │
|
||||
│ │ ←──────── └──────────┘ └──────────┘ └──────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ • Web Workers (Browser) / Worker Threads (Node.js) │ │
|
||||
│ │ • Model caching and reuse across workers │ │
|
||||
│ │ • Batch embedding for better performance │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ GPU ACCELERATION │ │
|
||||
│ │ │ │
|
||||
│ │ TensorFlow.js → WebGL Backend → GPU │ │
|
||||
│ │ ↓ │ │
|
||||
│ │ Fallback: CPU Backend for compatibility │ │
|
||||
│ │ │ │
|
||||
│ │ • Vector similarity calculations │ │
|
||||
│ │ • Embedding generation │ │
|
||||
│ │ • Tensor operations │ │
|
||||
│ │ • Automatic memory management │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ INTELLIGENT CACHING │ │
|
||||
│ │ │ │
|
||||
│ │ • Auto-tuning based on usage patterns │ │
|
||||
│ │ • Memory-aware cache sizing │ │
|
||||
│ │ • Prefetching strategies │ │
|
||||
│ │ • LRU eviction with batch processing │ │
|
||||
│ │ • Read-only mode optimizations │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Cross-Platform Integration
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ SYNCHRONIZATION & SCALING │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Browser ←→ WebSocket ←→ Server ←→ S3/Cloud Storage │
|
||||
│ ↓ ↓ │
|
||||
│ Browser ←→ WebRTC ←→ Browser (Peer-to-Peer) │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ CONDUIT AUGMENTATIONS │ │
|
||||
│ │ │ │
|
||||
│ │ WebSocket iConduit: │ │
|
||||
│ │ • Browser ↔ Server sync │ │
|
||||
│ │ • Server ↔ Server sync │ │
|
||||
│ │ • Real-time data streaming │ │
|
||||
│ │ │ │
|
||||
│ │ WebRTC iConduit: │ │
|
||||
│ │ • Direct browser ↔ browser sync │ │
|
||||
│ │ • Peer-to-peer without server │ │
|
||||
│ │ • Decentralized data sharing │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ MODEL CONTROL PROTOCOL (MCP) │ │
|
||||
│ │ │ │
|
||||
│ │ External AI Models ←→ MCP Server ←→ Brainy Data & Tools │ │
|
||||
│ │ │ │
|
||||
│ │ • BrainyMCPAdapter: Data access for external models │ │
|
||||
│ │ • MCPAugmentationToolset: Pipeline tools for models │ │
|
||||
│ │ • BrainyMCPService: WebSocket & REST integration │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Data Flow Example
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ DATA FLOW EXAMPLE │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ 1. Input: "Cats are independent pets" │
|
||||
│ ↓ │
|
||||
│ 2. SENSE Augmentation: Process raw text │
|
||||
│ ↓ │
|
||||
│ 3. Embedding: TensorFlow USE → [0.123, -0.456, 0.789, ...] │
|
||||
│ ↓ │
|
||||
│ 4. MEMORY Augmentation: Store with metadata │
|
||||
│ ↓ │
|
||||
│ 5. HNSW Index: Add vector to hierarchical graph │
|
||||
│ ↓ │
|
||||
│ 6. Storage: Persist to OPFS/FileSystem/S3 │
|
||||
│ │
|
||||
│ Query: "feline pets" → Embedding → HNSW Search → Ranked Results │
|
||||
│ Result: [{text: "Cats are independent pets", similarity: 0.89, id: "123"}] │
|
||||
│ │
|
||||
│ Relationship Example: │
|
||||
│ addVerb(catId, dogId, VerbType.RelatedTo, {description: "Both are pets"}) │
|
||||
│ ↓ │
|
||||
│ Graph: [Cat] ──RelatedTo──→ [Dog] │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Key Architecture Principles:**
|
||||
|
||||
1. **Environment Agnostic**: Automatically adapts to browser, Node.js, serverless, container, or server environments
|
||||
2. **Intelligent Storage**: Multi-tier caching with automatic storage selection (OPFS, filesystem, S3, memory)
|
||||
3. **Vector + Graph**: Combines semantic vector search with graph relationships in a unified model
|
||||
4. **Extensible Pipeline**: Modular augmentation system for custom processing and integration
|
||||
5. **Performance Optimized**: GPU acceleration, multithreading, intelligent caching, and memory management
|
||||
6. **Scalable Sync**: WebSocket and WebRTC conduits for real-time synchronization across instances
|
||||
7. **AI Integration**: MCP protocol for external AI model integration and tool access
|
||||
Loading…
Add table
Add a link
Reference in a new issue