**chore(docs): remove deprecated architecture documentation**
- **Documentation Removal**:
- Deleted `brainy_architecture_diagram.md` and `brainy_architecture_visual.md`:
- Removed outdated architecture descriptions, diagrams, and structured content no longer in use.
- **Purpose**:
- Clean up legacy documentation to reduce confusion and ensure only the latest and most accurate resources are available for developers and stakeholders.
This commit is contained in:
parent
cc6c75befb
commit
b846342681
7 changed files with 0 additions and 2838 deletions
|
|
@ -1,53 +0,0 @@
|
|||
# Development Tools & Documentation
|
||||
|
||||
This directory contains development tools, scripts, and documentation files that are not included in the published npm package.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
dev/
|
||||
├── docs/ # Development documentation files
|
||||
│ ├── brainy_architecture_diagram.md
|
||||
│ ├── PDF_GENERATION_GUIDE.md
|
||||
│ ├── QUICK_PDF_SETUP.md
|
||||
│ └── brainy_architecture_visual.md
|
||||
├── scripts/ # Development scripts
|
||||
│ └── generate-architecture-pdf.js
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
## Scripts
|
||||
|
||||
### generate-architecture-pdf.js
|
||||
|
||||
Generates a professional PDF documentation of Brainy's architecture using:
|
||||
- Material Design styling
|
||||
- Custom SVG diagrams
|
||||
- Comprehensive content from README.md
|
||||
- Professional visual presentation
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# From project root
|
||||
node dev/scripts/generate-architecture-pdf.js
|
||||
|
||||
# Or add to package.json scripts
|
||||
npm run generate-docs
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
- puppeteer (for PDF generation)
|
||||
|
||||
**Output:**
|
||||
- `docs/Brainy_Architecture_Documentation.pdf`
|
||||
|
||||
## Documentation Files
|
||||
|
||||
- **brainy_architecture_diagram.md**: ASCII art diagrams of system architecture
|
||||
- **PDF_GENERATION_GUIDE.md**: Detailed guide for PDF generation setup
|
||||
- **QUICK_PDF_SETUP.md**: Quick setup instructions
|
||||
- **brainy_architecture_visual.md**: Visual architecture documentation
|
||||
|
||||
## NPM Package Exclusion
|
||||
|
||||
This entire `dev/` directory is excluded from the published npm package via `.npmignore` to keep the package size minimal and focused on the core library functionality.
|
||||
|
|
@ -1,242 +0,0 @@
|
|||
# Brainy Architecture PDF Generation Guide
|
||||
|
||||
This guide shows you how to generate a professional PDF from the Brainy architecture documentation with beautiful diagrams.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Option 1: Using the npm script (Recommended)
|
||||
|
||||
```bash
|
||||
# Make sure you're in the brainy project directory
|
||||
cd /path/to/brainy
|
||||
|
||||
# Install dependencies if not already installed
|
||||
npm install
|
||||
|
||||
# Generate the PDF
|
||||
npm run generate-pdf
|
||||
```
|
||||
|
||||
### Option 2: Direct script execution
|
||||
|
||||
```bash
|
||||
# Make sure you're in the brainy project directory
|
||||
cd /path/to/brainy
|
||||
|
||||
# Install Puppeteer if not already installed
|
||||
npm install puppeteer
|
||||
|
||||
# Run the script directly
|
||||
node dev/dev/scripts/generate-architecture-pdf.js
|
||||
```
|
||||
|
||||
## Installation Requirements
|
||||
|
||||
### Prerequisites
|
||||
- Node.js 18+
|
||||
- npm or yarn
|
||||
|
||||
### Dependencies
|
||||
The script uses:
|
||||
- **Puppeteer**: For PDF generation and browser automation
|
||||
- **Mermaid**: For rendering diagrams (loaded via CDN)
|
||||
- **Google Fonts**: For professional typography (loaded via CDN)
|
||||
|
||||
### Install Dependencies
|
||||
|
||||
```bash
|
||||
# If you don't have puppeteer installed globally or in the project
|
||||
npm install puppeteer
|
||||
|
||||
# Or install as dev dependency
|
||||
npm install --save-dev puppeteer
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
The PDF will be generated at:
|
||||
```
|
||||
docs/Brainy_Architecture_Documentation.pdf
|
||||
```
|
||||
|
||||
## Features of the Generated PDF
|
||||
|
||||
### Professional Styling
|
||||
- **Modern Typography**: Uses Inter font family for clean, readable text
|
||||
- **Code Font**: JetBrains Mono for code blocks and technical content
|
||||
- **Color Scheme**: Professional blue theme with proper contrast
|
||||
- **Layout**: A4 format with proper margins and spacing
|
||||
|
||||
### Rich Diagrams
|
||||
- **Mermaid Diagrams**: All diagrams are rendered as vector graphics
|
||||
- **Interactive Elements**: Flowcharts, sequence diagrams, mindmaps, and more
|
||||
- **Consistent Styling**: All diagrams follow the same color scheme
|
||||
- **High Quality**: Vector-based rendering for crisp output
|
||||
|
||||
### Document Structure
|
||||
- **Table of Contents**: Linked navigation
|
||||
- **Page Headers/Footers**: Professional branding and page numbers
|
||||
- **Section Breaks**: Logical page breaks between major sections
|
||||
- **Code Highlighting**: Syntax highlighting for JSON and code blocks
|
||||
|
||||
## Customization
|
||||
|
||||
### Modify Styling
|
||||
Edit the `professionalCSS` variable in `dev/scripts/generate-architecture-pdf.js`:
|
||||
|
||||
```javascript
|
||||
const professionalCSS = `
|
||||
/* Your custom CSS here */
|
||||
h1 {
|
||||
color: #your-color;
|
||||
font-size: 24pt;
|
||||
}
|
||||
/* ... */
|
||||
`
|
||||
```
|
||||
|
||||
### Change Output Location
|
||||
Modify the `config` object:
|
||||
|
||||
```javascript
|
||||
const config = {
|
||||
inputFile: path.join(__dirname, '../docs/brainy_architecture_visual.md'),
|
||||
outputFile: path.join(__dirname, '../docs/YOUR_CUSTOM_NAME.pdf'),
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### Adjust PDF Settings
|
||||
Modify the `page.pdf()` options:
|
||||
|
||||
```javascript
|
||||
await page.pdf({
|
||||
path: config.outputFile,
|
||||
format: 'A4', // or 'Letter', 'Legal', etc.
|
||||
printBackground: true,
|
||||
margin: {
|
||||
top: '20mm',
|
||||
right: '15mm',
|
||||
bottom: '20mm',
|
||||
left: '15mm'
|
||||
},
|
||||
// ... other options
|
||||
})
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. "Puppeteer not found"
|
||||
```bash
|
||||
npm install puppeteer
|
||||
```
|
||||
|
||||
#### 2. "Chrome/Chromium not found"
|
||||
```bash
|
||||
# On Ubuntu/Debian
|
||||
sudo apt-get install chromium-browser
|
||||
|
||||
# On macOS
|
||||
brew install chromium
|
||||
|
||||
# Or let Puppeteer download Chromium
|
||||
npm install puppeteer --unsafe-perm=true
|
||||
```
|
||||
|
||||
#### 3. "Permission denied"
|
||||
```bash
|
||||
chmod +x dev/scripts/generate-architecture-pdf.js
|
||||
```
|
||||
|
||||
#### 4. "Diagrams not rendering"
|
||||
Check your internet connection - Mermaid is loaded from CDN. For offline use, you can download mermaid.min.js locally and update the path.
|
||||
|
||||
### Advanced Configuration
|
||||
|
||||
#### Use Local Mermaid
|
||||
Download mermaid.min.js and update the config:
|
||||
|
||||
```javascript
|
||||
const config = {
|
||||
// ...
|
||||
mermaidCDN: './path/to/mermaid.min.js'
|
||||
}
|
||||
```
|
||||
|
||||
#### Custom Fonts
|
||||
Add additional fonts to the CSS:
|
||||
|
||||
```css
|
||||
@import url('https://fonts.googleapis.com/css2?family=YourFont:wght@400;500;600&display=swap');
|
||||
|
||||
body {
|
||||
font-family: 'YourFont', sans-serif;
|
||||
}
|
||||
```
|
||||
|
||||
## Adding to package.json
|
||||
|
||||
Add this script to your `package.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"generate-pdf": "node dev/dev/scripts/generate-architecture-pdf.js",
|
||||
"docs:pdf": "npm run generate-pdf"
|
||||
},
|
||||
"devDependencies": {
|
||||
"puppeteer": "^22.5.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Alternative PDF Generators
|
||||
|
||||
If you prefer other tools, you can also use:
|
||||
|
||||
### 1. Pandoc + LaTeX
|
||||
```bash
|
||||
# Install pandoc and latex
|
||||
sudo apt-get install pandoc texlive-latex-recommended
|
||||
|
||||
# Convert (note: won't render Mermaid diagrams)
|
||||
pandoc docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf
|
||||
```
|
||||
|
||||
### 2. mdpdf
|
||||
```bash
|
||||
npm install -g mdpdf
|
||||
mdpdf docs/brainy_architecture_visual.md --output=docs/brainy_architecture.pdf
|
||||
```
|
||||
|
||||
### 3. markdown-pdf
|
||||
```bash
|
||||
npm install -g markdown-pdf
|
||||
markdown-pdf docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf
|
||||
```
|
||||
|
||||
**Note**: The custom Puppeteer script provides the best results with proper Mermaid diagram rendering and professional styling.
|
||||
|
||||
## Sample Output
|
||||
|
||||
The generated PDF will include:
|
||||
|
||||
1. **Cover Page** with title and subtitle
|
||||
2. **Table of Contents** with page links
|
||||
3. **System Overview** with environment detection diagram
|
||||
4. **Core Architecture** with layered architecture diagram
|
||||
5. **Data Model** with noun/verb type hierarchies
|
||||
6. **Vector Search Engine** with HNSW visualization
|
||||
7. **Storage Architecture** with multi-tier caching diagrams
|
||||
8. **Augmentation Pipeline** with flow diagrams
|
||||
9. **Performance Optimizations** with threading models
|
||||
10. **Integration Patterns** with network topology
|
||||
11. **Data Flow Examples** with sequence diagrams
|
||||
|
||||
Total pages: ~25-30 pages with full diagrams and explanations.
|
||||
|
||||
---
|
||||
|
||||
*For questions or issues with PDF generation, please check the troubleshooting section or create an issue in the repository.*
|
||||
|
|
@ -1,80 +0,0 @@
|
|||
# Quick PDF Generation Setup
|
||||
|
||||
## 🚀 Generate Professional Brainy Architecture PDF
|
||||
|
||||
### One-Command Setup & Generation
|
||||
|
||||
```bash
|
||||
# Install Puppeteer and generate PDF in one go
|
||||
npm install puppeteer && npm run generate-pdf
|
||||
```
|
||||
|
||||
### Step-by-Step
|
||||
|
||||
1. **Install Puppeteer** (if not already installed):
|
||||
```bash
|
||||
npm install puppeteer
|
||||
```
|
||||
|
||||
2. **Generate the PDF**:
|
||||
```bash
|
||||
npm run generate-pdf
|
||||
```
|
||||
|
||||
3. **Find your PDF**:
|
||||
```
|
||||
docs/Brainy_Architecture_Documentation.pdf
|
||||
```
|
||||
|
||||
## ✨ What You Get
|
||||
|
||||
- **25-30 page professional PDF** with full diagrams
|
||||
- **Vector graphics** for all Mermaid diagrams
|
||||
- **Modern typography** with Inter font family
|
||||
- **Consistent branding** throughout the document
|
||||
- **Table of contents** with page links
|
||||
- **Professional headers/footers**
|
||||
|
||||
## 📊 Sample Sections Include
|
||||
|
||||
- System Overview with environment detection
|
||||
- Core Architecture layers
|
||||
- Data Model (23 Noun Types, 38 Verb Types)
|
||||
- Vector Search Engine with HNSW visualization
|
||||
- Storage Architecture with multi-tier caching
|
||||
- Augmentation Pipeline flows
|
||||
- Performance optimizations
|
||||
- Cross-platform integration patterns
|
||||
- Real data flow examples
|
||||
|
||||
## 🛠️ Troubleshooting
|
||||
|
||||
### Issue: "Puppeteer not found"
|
||||
```bash
|
||||
npm install puppeteer
|
||||
```
|
||||
|
||||
### Issue: "Chrome not found"
|
||||
```bash
|
||||
# Let Puppeteer download Chromium
|
||||
npm install puppeteer --unsafe-perm=true
|
||||
```
|
||||
|
||||
### Issue: "Permission denied"
|
||||
```bash
|
||||
chmod +x dev/scripts/generate-architecture-pdf.js
|
||||
```
|
||||
|
||||
## 🎨 Customization
|
||||
|
||||
Edit `dev/scripts/generate-architecture-pdf.js` to:
|
||||
- Change colors and fonts
|
||||
- Modify page layout
|
||||
- Adjust diagram styling
|
||||
- Add custom branding
|
||||
|
||||
---
|
||||
|
||||
**Ready to generate?** Run `npm run generate-pdf` and get your professional architecture documentation!
|
||||
|
||||
For detailed setup instructions, see `PDF_GENERATION_GUIDE.md`.
|
||||
|
|
@ -1,313 +0,0 @@
|
|||
# Brainy Architecture Diagram
|
||||
|
||||
## System Overview
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ BRAINY PLATFORM │
|
||||
│ Vector Graph Database with AI Pipeline │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ ENVIRONMENT DETECTION │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ Browser │ Node.js │ Serverless │ Container │ Server │
|
||||
│ (OPFS) │ (File System) │ (In-Memory) │ (Adaptive) │ (S3/Cloud) │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
```
|
||||
|
||||
## Core Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ BRAINY DATA API │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ add() │ search() │ addVerb() │ get() │ delete() │ backup() │ restore() │ etc. │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ AUGMENTATION PIPELINE │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ SENSE → MEMORY → COGNITION → CONDUIT → ACTIVATION → PERCEPTION → DIALOG → WS │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ DATA PROCESSING │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ Text/JSON → Embedding → Vector Storage │
|
||||
│ │ │
|
||||
│ ┌─────────────────────────┼─────────────────────────┐ │
|
||||
│ │ EMBEDDING │ VECTOR INDEX │ │
|
||||
│ │ │ │ │
|
||||
│ │ TensorFlow.js │ HNSW Algorithm │ │
|
||||
│ │ Universal Sentence │ - Hierarchical │ │
|
||||
│ │ Encoder (USE) │ - Fast Similarity │ │
|
||||
│ │ - GPU Acceleration │ - Configurable │ │
|
||||
│ │ - Batch Processing │ - Memory Efficient │ │
|
||||
│ │ - Worker Threads │ - Product Quantized │ │
|
||||
│ └─────────────────────────┼─────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
```
|
||||
|
||||
## Data Model & Graph Structure
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ GRAPH DATA MODEL │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ NOUNS (Entities/Nodes) │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Core Entity Types: │ Digital/Content Types: │ │
|
||||
│ │ • Person │ • Document │ │
|
||||
│ │ • Organization │ • Media │ │
|
||||
│ │ • Location │ • File │ │
|
||||
│ │ • Thing │ • Message │ │
|
||||
│ │ • Concept │ • Content │ │
|
||||
│ │ • Event │ │ │
|
||||
│ │ │ Collection Types: │ │
|
||||
│ │ Business/App Types: │ • Collection │ │
|
||||
│ │ • Product │ • Dataset │ │
|
||||
│ │ • Service │ │ │
|
||||
│ │ • User │ Descriptive Types: │ │
|
||||
│ │ • Task │ • Process, State, Role │ │
|
||||
│ │ • Project │ • Topic, Language, Currency, Measurement │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ VERBS (Relationships/Edges) │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Core Relationships: │ Social/Organizational: │ │
|
||||
│ │ • RelatedTo │ • MemberOf, WorksWith │ │
|
||||
│ │ • Contains, PartOf │ • FriendOf, Follows, Likes │ │
|
||||
│ │ • LocatedAt, References │ • ReportsTo, Supervises, Mentors │ │
|
||||
│ │ │ • Communicates │ │
|
||||
│ │ Temporal/Causal: │ │ │
|
||||
│ │ • Precedes, Succeeds │ Descriptive/Functional: │ │
|
||||
│ │ • Causes, DependsOn │ • Describes, Defines, Categorizes │ │
|
||||
│ │ • Requires │ • Measures, Evaluates │ │
|
||||
│ │ │ • Uses, Implements, Extends │ │
|
||||
│ │ Creation/Transformation: │ │ │
|
||||
│ │ • Creates, Transforms │ Ownership/Attribution: │ │
|
||||
│ │ • Becomes, Modifies │ • Owns, AttributedTo │ │
|
||||
│ │ • Consumes │ • CreatedBy, BelongsTo │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Vector Storage & Search Engine
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ VECTOR SEARCH ENGINE │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Query Text/Vector → Embedding → HNSW Search → Ranked Results │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ HNSW INDEX STRUCTURE │ │
|
||||
│ │ │ │
|
||||
│ │ Layer 2: ●────────●────────● (Sparse connections) │ │
|
||||
│ │ ╱│ │ │╲ │ │
|
||||
│ │ Layer 1: ●─●──●─●─●─●──●─●─●─● (Medium density) │ │
|
||||
│ │ ╱│││││││││││││││││││││╲ │ │
|
||||
│ │ Layer 0: ●●●●●●●●●●●●●●●●●●●●●●● (Dense connections) │ │
|
||||
│ │ │ │
|
||||
│ │ • Hierarchical navigation for fast search │ │
|
||||
│ │ • Configurable M (max connections), efConstruction, efSearch │ │
|
||||
│ │ • Memory-efficient with disk-based storage for large datasets │ │
|
||||
│ │ • Product quantization for dimensionality reduction │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ ADAPTIVE STORAGE │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─ Hot Cache (RAM) ──┐ │
|
||||
│ │ Most accessed │ │
|
||||
│ │ LRU eviction │ │
|
||||
│ │ Auto-tuned size │ │
|
||||
│ └─────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─ Warm Cache (Storage) ─┐ │
|
||||
│ │ Recent nodes │ │
|
||||
│ │ OPFS/Filesystem/S3 │ │
|
||||
│ │ TTL-based │ │
|
||||
│ └─────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─ Cold Storage (Persistent) ─┐ │
|
||||
│ │ All nodes │ │
|
||||
│ │ OPFS/Filesystem/S3 │ │
|
||||
│ │ Batch operations │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ │
|
||||
│ Environment-Specific Storage Adapters: │
|
||||
│ ┌─────────────┬─────────────┬─────────────┬─────────────┬─────────────┐ │
|
||||
│ │ Browser │ Node.js │ Serverless │ Container │ Server │ │
|
||||
│ │ OPFS │ FileSystem │ In-Memory │ Adaptive │ S3/Cloud │ │
|
||||
│ │ (Fallback: │ (Backup: │ (Optional: │ (Auto- │ (Multi- │ │
|
||||
│ │ IndexedDB) │ S3/Cloud) │ S3/Cloud) │ Detect) │ Provider) │ │
|
||||
│ └─────────────┴─────────────┴─────────────┴─────────────┴─────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Augmentation Pipeline System
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ AUGMENTATION PIPELINE FLOW │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Raw Data → [SENSE] → [MEMORY] → [COGNITION] → [CONDUIT] → [ACTIVATION] → │
|
||||
│ │ │ │ │ │ │
|
||||
│ ▼ ▼ ▼ ▼ ▼ │
|
||||
│ Process Storage Reasoning Data Sync Actions │
|
||||
│ Input Persist Inference External Triggers │
|
||||
│ Convert Retrieve Logic Ops Systems Responses │
|
||||
│ │
|
||||
│ → [PERCEPTION] → [DIALOG] → [WEBSOCKET] → │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ Visualization NLP/Chat Real-time │
|
||||
│ Interpretation Response Streaming │
|
||||
│ Organization Context Communication │
|
||||
│ │
|
||||
│ Execution Modes: │
|
||||
│ • SEQUENTIAL: Step-by-step processing │
|
||||
│ • PARALLEL: Concurrent augmentation execution │
|
||||
│ • THREADED: Multi-threaded with worker pools │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Performance & Scaling Features
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ PERFORMANCE OPTIMIZATIONS │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ MULTITHREADING │ │
|
||||
│ │ │ │
|
||||
│ │ Main Thread ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
|
||||
│ │ ├──────→ │ Worker 1 │ │ Worker 2 │ │ Worker N │ │ │
|
||||
│ │ │ │Embedding │ │ Search │ │ Batch │ │ │
|
||||
│ │ │ │Generation│ │Operations│ │Processing│ │ │
|
||||
│ │ ←──────── └──────────┘ └──────────┘ └──────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ • Web Workers (Browser) / Worker Threads (Node.js) │ │
|
||||
│ │ • Model caching and reuse across workers │ │
|
||||
│ │ • Batch embedding for better performance │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ GPU ACCELERATION │ │
|
||||
│ │ │ │
|
||||
│ │ TensorFlow.js → WebGL Backend → GPU │ │
|
||||
│ │ ↓ │ │
|
||||
│ │ Fallback: CPU Backend for compatibility │ │
|
||||
│ │ │ │
|
||||
│ │ • Vector similarity calculations │ │
|
||||
│ │ • Embedding generation │ │
|
||||
│ │ • Tensor operations │ │
|
||||
│ │ • Automatic memory management │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ INTELLIGENT CACHING │ │
|
||||
│ │ │ │
|
||||
│ │ • Auto-tuning based on usage patterns │ │
|
||||
│ │ • Memory-aware cache sizing │ │
|
||||
│ │ • Prefetching strategies │ │
|
||||
│ │ • LRU eviction with batch processing │ │
|
||||
│ │ • Read-only mode optimizations │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Cross-Platform Integration
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ SYNCHRONIZATION & SCALING │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Browser ←→ WebSocket ←→ Server ←→ S3/Cloud Storage │
|
||||
│ ↓ ↓ │
|
||||
│ Browser ←→ WebRTC ←→ Browser (Peer-to-Peer) │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ CONDUIT AUGMENTATIONS │ │
|
||||
│ │ │ │
|
||||
│ │ WebSocket iConduit: │ │
|
||||
│ │ • Browser ↔ Server sync │ │
|
||||
│ │ • Server ↔ Server sync │ │
|
||||
│ │ • Real-time data streaming │ │
|
||||
│ │ │ │
|
||||
│ │ WebRTC iConduit: │ │
|
||||
│ │ • Direct browser ↔ browser sync │ │
|
||||
│ │ • Peer-to-peer without server │ │
|
||||
│ │ • Decentralized data sharing │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ MODEL CONTROL PROTOCOL (MCP) │ │
|
||||
│ │ │ │
|
||||
│ │ External AI Models ←→ MCP Server ←→ Brainy Data & Tools │ │
|
||||
│ │ │ │
|
||||
│ │ • BrainyMCPAdapter: Data access for external models │ │
|
||||
│ │ • MCPAugmentationToolset: Pipeline tools for models │ │
|
||||
│ │ • BrainyMCPService: WebSocket & REST integration │ │
|
||||
│ └─────────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Data Flow Example
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ DATA FLOW EXAMPLE │
|
||||
├─────────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ 1. Input: "Cats are independent pets" │
|
||||
│ ↓ │
|
||||
│ 2. SENSE Augmentation: Process raw text │
|
||||
│ ↓ │
|
||||
│ 3. Embedding: TensorFlow USE → [0.123, -0.456, 0.789, ...] │
|
||||
│ ↓ │
|
||||
│ 4. MEMORY Augmentation: Store with metadata │
|
||||
│ ↓ │
|
||||
│ 5. HNSW Index: Add vector to hierarchical graph │
|
||||
│ ↓ │
|
||||
│ 6. Storage: Persist to OPFS/FileSystem/S3 │
|
||||
│ │
|
||||
│ Query: "feline pets" → Embedding → HNSW Search → Ranked Results │
|
||||
│ Result: [{text: "Cats are independent pets", similarity: 0.89, id: "123"}] │
|
||||
│ │
|
||||
│ Relationship Example: │
|
||||
│ addVerb(catId, dogId, VerbType.RelatedTo, {description: "Both are pets"}) │
|
||||
│ ↓ │
|
||||
│ Graph: [Cat] ──RelatedTo──→ [Dog] │
|
||||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Key Architecture Principles:**
|
||||
|
||||
1. **Environment Agnostic**: Automatically adapts to browser, Node.js, serverless, container, or server environments
|
||||
2. **Intelligent Storage**: Multi-tier caching with automatic storage selection (OPFS, filesystem, S3, memory)
|
||||
3. **Vector + Graph**: Combines semantic vector search with graph relationships in a unified model
|
||||
4. **Extensible Pipeline**: Modular augmentation system for custom processing and integration
|
||||
5. **Performance Optimized**: GPU acceleration, multithreading, intelligent caching, and memory management
|
||||
6. **Scalable Sync**: WebSocket and WebRTC conduits for real-time synchronization across instances
|
||||
7. **AI Integration**: MCP protocol for external AI model integration and tool access
|
||||
File diff suppressed because it is too large
Load diff
Binary file not shown.
|
|
@ -1,729 +0,0 @@
|
|||
# Brainy Architecture Documentation
|
||||
## Vector Graph Database with AI Pipeline
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [System Overview](#system-overview)
|
||||
2. [Core Architecture](#core-architecture)
|
||||
3. [Data Model & Graph Structure](#data-model--graph-structure)
|
||||
4. [Vector Search Engine](#vector-search-engine)
|
||||
5. [Storage Architecture](#storage-architecture)
|
||||
6. [Augmentation Pipeline](#augmentation-pipeline)
|
||||
7. [Performance Optimizations](#performance-optimizations)
|
||||
8. [Cross-Platform Integration](#cross-platform-integration)
|
||||
9. [Data Flow Example](#data-flow-example)
|
||||
|
||||
---
|
||||
|
||||
## System Overview
|
||||
|
||||
Brainy is a powerful, cross-platform vector graph database that intelligently adapts to any environment while providing both semantic vector search and graph relationship capabilities.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[User Application] --> B[Brainy Platform]
|
||||
B --> C[Environment Detection]
|
||||
|
||||
C --> D[Browser<br/>OPFS Storage]
|
||||
C --> E[Node.js<br/>File System]
|
||||
C --> F[Serverless<br/>In-Memory]
|
||||
C --> G[Container<br/>Adaptive]
|
||||
C --> H[Server<br/>S3/Cloud]
|
||||
|
||||
B --> I[Vector Search Engine]
|
||||
B --> J[Graph Database]
|
||||
B --> K[Augmentation Pipeline]
|
||||
|
||||
style B fill:#e1f5fe
|
||||
style I fill:#f3e5f5
|
||||
style J fill:#e8f5e8
|
||||
style K fill:#fff3e0
|
||||
```
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Universal Compatibility**: Runs everywhere - browsers, Node.js, serverless functions, containers
|
||||
- **Intelligent Adaptation**: Automatically optimizes for environment and usage patterns
|
||||
- **Dual Nature**: Vector similarity search + graph relationships in one system
|
||||
- **Real-time Streaming**: Live data processing through extensible pipeline
|
||||
- **AI Integration**: Built-in TensorFlow.js with GPU acceleration
|
||||
|
||||
---
|
||||
|
||||
## Core Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Application Layer"
|
||||
API[Brainy Data API<br/>add() | search() | addVerb() | get() | delete()]
|
||||
end
|
||||
|
||||
subgraph "Processing Layer"
|
||||
PIPELINE[Augmentation Pipeline<br/>SENSE → MEMORY → COGNITION → CONDUIT → ACTIVATION → PERCEPTION → DIALOG → WS]
|
||||
end
|
||||
|
||||
subgraph "Engine Layer"
|
||||
EMBED[Embedding Engine<br/>TensorFlow.js Universal Sentence Encoder]
|
||||
VECTOR[Vector Index<br/>HNSW Algorithm]
|
||||
GRAPH[Graph Engine<br/>Noun-Verb Model]
|
||||
end
|
||||
|
||||
subgraph "Storage Layer"
|
||||
CACHE[Multi-tier Caching<br/>Hot → Warm → Cold]
|
||||
STORAGE[Adaptive Storage<br/>OPFS | FileSystem | S3 | Memory]
|
||||
end
|
||||
|
||||
API --> PIPELINE
|
||||
PIPELINE --> EMBED
|
||||
PIPELINE --> VECTOR
|
||||
PIPELINE --> GRAPH
|
||||
EMBED --> CACHE
|
||||
VECTOR --> CACHE
|
||||
GRAPH --> STORAGE
|
||||
CACHE --> STORAGE
|
||||
|
||||
style API fill:#e3f2fd
|
||||
style PIPELINE fill:#f1f8e9
|
||||
style EMBED fill:#fce4ec
|
||||
style VECTOR fill:#fff8e1
|
||||
style GRAPH fill:#e8f5e8
|
||||
style CACHE fill:#f3e5f5
|
||||
style STORAGE fill:#efebe9
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Model & Graph Structure
|
||||
|
||||
### Noun Types (Entities/Nodes)
|
||||
|
||||
```mermaid
|
||||
mindmap
|
||||
root((Brainy<br/>Noun Types))
|
||||
Core Entities
|
||||
Person
|
||||
Organization
|
||||
Location
|
||||
Thing
|
||||
Concept
|
||||
Event
|
||||
Digital Content
|
||||
Document
|
||||
Media
|
||||
File
|
||||
Message
|
||||
Content
|
||||
Collections
|
||||
Collection
|
||||
Dataset
|
||||
Business/App
|
||||
Product
|
||||
Service
|
||||
User
|
||||
Task
|
||||
Project
|
||||
Descriptive
|
||||
Process
|
||||
State
|
||||
Role
|
||||
Topic
|
||||
Language
|
||||
Currency
|
||||
Measurement
|
||||
```
|
||||
|
||||
### Verb Types (Relationships/Edges)
|
||||
|
||||
```mermaid
|
||||
mindmap
|
||||
root((Brainy<br/>Verb Types))
|
||||
Core Relations
|
||||
RelatedTo
|
||||
Contains
|
||||
PartOf
|
||||
LocatedAt
|
||||
References
|
||||
Temporal/Causal
|
||||
Precedes
|
||||
Succeeds
|
||||
Causes
|
||||
DependsOn
|
||||
Requires
|
||||
Creation/Transform
|
||||
Creates
|
||||
Transforms
|
||||
Becomes
|
||||
Modifies
|
||||
Consumes
|
||||
Ownership/Attribution
|
||||
Owns
|
||||
AttributedTo
|
||||
CreatedBy
|
||||
BelongsTo
|
||||
Social/Organizational
|
||||
MemberOf
|
||||
WorksWith
|
||||
FriendOf
|
||||
Follows
|
||||
Likes
|
||||
ReportsTo
|
||||
Supervises
|
||||
Mentors
|
||||
Communicates
|
||||
Descriptive/Functional
|
||||
Describes
|
||||
Defines
|
||||
Categorizes
|
||||
Measures
|
||||
Evaluates
|
||||
Uses
|
||||
Implements
|
||||
Extends
|
||||
```
|
||||
|
||||
### Graph Example
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Person: John Doe<br/>ID: person-123] -->|WorksWith| B[Organization: Acme Corp<br/>ID: org-456]
|
||||
A -->|CreatedBy| C[Document: Report<br/>ID: doc-789]
|
||||
A -->|LocatedAt| D[Location: New York<br/>ID: loc-101]
|
||||
B -->|Contains| E[Project: AI Initiative<br/>ID: proj-202]
|
||||
C -->|PartOf| E
|
||||
E -->|Uses| F[Concept: Machine Learning<br/>ID: concept-303]
|
||||
|
||||
style A fill:#ffcdd2
|
||||
style B fill:#c8e6c9
|
||||
style C fill:#bbdefb
|
||||
style D fill:#fff9c4
|
||||
style E fill:#f8bbd9
|
||||
style F fill:#d1c4e9
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Vector Search Engine
|
||||
|
||||
### HNSW Index Structure
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "HNSW Hierarchical Structure"
|
||||
subgraph "Layer 2 (Sparse)"
|
||||
L2A((●)) --- L2B((●))
|
||||
L2B --- L2C((●))
|
||||
end
|
||||
|
||||
subgraph "Layer 1 (Medium Density)"
|
||||
L1A((●)) --- L1B((●))
|
||||
L1B --- L1C((●))
|
||||
L1C --- L1D((●))
|
||||
L1D --- L1E((●))
|
||||
L1E --- L1F((●))
|
||||
L1F --- L1G((●))
|
||||
L1G --- L1H((●))
|
||||
end
|
||||
|
||||
subgraph "Layer 0 (Dense Connections)"
|
||||
L0A((●)) --- L0B((●))
|
||||
L0B --- L0C((●))
|
||||
L0C --- L0D((●))
|
||||
L0D --- L0E((●))
|
||||
L0E --- L0F((●))
|
||||
L0F --- L0G((●))
|
||||
L0G --- L0H((●))
|
||||
L0H --- L0I((●))
|
||||
L0I --- L0J((●))
|
||||
L0J --- L0K((●))
|
||||
L0K --- L0L((●))
|
||||
L0L --- L0M((●))
|
||||
L0M --- L0N((●))
|
||||
L0N --- L0O((●))
|
||||
L0O --- L0P((●))
|
||||
end
|
||||
|
||||
L2A -.-> L1A
|
||||
L2A -.-> L1D
|
||||
L2B -.-> L1C
|
||||
L2B -.-> L1F
|
||||
L2C -.-> L1G
|
||||
|
||||
L1A -.-> L0A
|
||||
L1A -.-> L0B
|
||||
L1B -.-> L0C
|
||||
L1B -.-> L0D
|
||||
L1C -.-> L0E
|
||||
L1C -.-> L0F
|
||||
L1D -.-> L0G
|
||||
L1D -.-> L0H
|
||||
L1E -.-> L0I
|
||||
L1E -.-> L0J
|
||||
L1F -.-> L0K
|
||||
L1F -.-> L0L
|
||||
L1G -.-> L0M
|
||||
L1G -.-> L0N
|
||||
L1H -.-> L0O
|
||||
L1H -.-> L0P
|
||||
end
|
||||
|
||||
style L2A fill:#ff9999
|
||||
style L2B fill:#ff9999
|
||||
style L2C fill:#ff9999
|
||||
style L1A fill:#99ccff
|
||||
style L1B fill:#99ccff
|
||||
style L1C fill:#99ccff
|
||||
style L1D fill:#99ccff
|
||||
style L1E fill:#99ccff
|
||||
style L1F fill:#99ccff
|
||||
style L1G fill:#99ccff
|
||||
style L1H fill:#99ccff
|
||||
style L0A fill:#99ff99
|
||||
style L0B fill:#99ff99
|
||||
style L0C fill:#99ff99
|
||||
style L0D fill:#99ff99
|
||||
style L0E fill:#99ff99
|
||||
style L0F fill:#99ff99
|
||||
style L0G fill:#99ff99
|
||||
style L0H fill:#99ff99
|
||||
style L0I fill:#99ff99
|
||||
style L0J fill:#99ff99
|
||||
style L0K fill:#99ff99
|
||||
style L0L fill:#99ff99
|
||||
style L0M fill:#99ff99
|
||||
style L0N fill:#99ff99
|
||||
style L0O fill:#99ff99
|
||||
style L0P fill:#99ff99
|
||||
```
|
||||
|
||||
### Search Process Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant API
|
||||
participant Embedding
|
||||
participant HNSW
|
||||
participant Storage
|
||||
|
||||
User->>API: searchText("feline pets", 5)
|
||||
API->>Embedding: embed("feline pets")
|
||||
Embedding->>Embedding: TensorFlow.js Universal Sentence Encoder
|
||||
Embedding-->>API: [0.123, -0.456, 0.789, ...]
|
||||
|
||||
API->>HNSW: search(vector, k=5)
|
||||
HNSW->>HNSW: Navigate from top layer
|
||||
HNSW->>HNSW: Descend to lower layers
|
||||
HNSW->>HNSW: Find k nearest neighbors
|
||||
HNSW-->>API: [id1, id2, id3, id4, id5]
|
||||
|
||||
API->>Storage: get([id1, id2, id3, id4, id5])
|
||||
Storage-->>API: [noun1, noun2, noun3, noun4, noun5]
|
||||
|
||||
API-->>User: [{text: "Cats are independent pets", similarity: 0.89}, ...]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
### Multi-Tier Caching System
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Memory Hierarchy"
|
||||
subgraph "Hot Cache (RAM)"
|
||||
HC[Most Accessed Items<br/>LRU Eviction<br/>Auto-tuned Size<br/>Millisecond Access]
|
||||
end
|
||||
|
||||
subgraph "Warm Cache (Storage)"
|
||||
WC[Recent Items<br/>TTL-based<br/>Sub-second Access<br/>OPFS/FS/S3]
|
||||
end
|
||||
|
||||
subgraph "Cold Storage (Persistent)"
|
||||
CS[All Items<br/>Batch Operations<br/>Full Persistence<br/>OPFS/FS/S3]
|
||||
end
|
||||
end
|
||||
|
||||
subgraph "Environment Adapters"
|
||||
Browser[Browser<br/>OPFS → IndexedDB]
|
||||
NodeJS[Node.js<br/>FileSystem → S3]
|
||||
Serverless[Serverless<br/>Memory → S3]
|
||||
Container[Container<br/>Auto-detect]
|
||||
Server[Server<br/>S3/Multi-cloud]
|
||||
end
|
||||
|
||||
User[User Query] --> HC
|
||||
HC -->|Cache Miss| WC
|
||||
WC -->|Cache Miss| CS
|
||||
|
||||
CS --> Browser
|
||||
CS --> NodeJS
|
||||
CS --> Serverless
|
||||
CS --> Container
|
||||
CS --> Server
|
||||
|
||||
style HC fill:#ffcdd2
|
||||
style WC fill:#fff9c4
|
||||
style CS fill:#c8e6c9
|
||||
style Browser fill:#e1f5fe
|
||||
style NodeJS fill:#e8f5e8
|
||||
style Serverless fill:#f3e5f5
|
||||
style Container fill:#fff3e0
|
||||
style Server fill:#efebe9
|
||||
```
|
||||
|
||||
### Storage Performance Characteristics
|
||||
|
||||
```mermaid
|
||||
xychart-beta
|
||||
title "Storage Performance by Environment"
|
||||
x-axis [Browser, Node.js, Serverless, Container, Server]
|
||||
y-axis "Latency (ms)" 0 --> 1000
|
||||
line [50, 10, 200, 30, 100]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Augmentation Pipeline
|
||||
|
||||
### Pipeline Flow Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph "Data Processing Pipeline"
|
||||
Input[Raw Data] --> SENSE[SENSE<br/>Process Input<br/>Convert & Validate]
|
||||
SENSE --> MEMORY[MEMORY<br/>Storage Operations<br/>Persist & Retrieve]
|
||||
MEMORY --> COGNITION[COGNITION<br/>Reasoning<br/>Inference & Logic]
|
||||
COGNITION --> CONDUIT[CONDUIT<br/>Data Sync<br/>External Systems]
|
||||
CONDUIT --> ACTIVATION[ACTIVATION<br/>Actions<br/>Triggers & Events]
|
||||
ACTIVATION --> PERCEPTION[PERCEPTION<br/>Visualization<br/>Interpretation]
|
||||
PERCEPTION --> DIALOG[DIALOG<br/>NLP & Chat<br/>Context & Response]
|
||||
DIALOG --> WEBSOCKET[WEBSOCKET<br/>Real-time<br/>Streaming & Sync]
|
||||
WEBSOCKET --> Output[Processed Output]
|
||||
end
|
||||
|
||||
subgraph "Execution Modes"
|
||||
SEQ[Sequential<br/>Step-by-step]
|
||||
PAR[Parallel<br/>Concurrent]
|
||||
THR[Threaded<br/>Worker Pools]
|
||||
end
|
||||
|
||||
Input -.-> SEQ
|
||||
Input -.-> PAR
|
||||
Input -.-> THR
|
||||
|
||||
style SENSE fill:#e8f5e8
|
||||
style MEMORY fill:#e3f2fd
|
||||
style COGNITION fill:#fff3e0
|
||||
style CONDUIT fill:#f3e5f5
|
||||
style ACTIVATION fill:#ffebee
|
||||
style PERCEPTION fill:#e0f2f1
|
||||
style DIALOG fill:#fce4ec
|
||||
style WEBSOCKET fill:#e8eaf6
|
||||
```
|
||||
|
||||
### Augmentation Types Detail
|
||||
|
||||
```mermaid
|
||||
mindmap
|
||||
root((Augmentation<br/>System))
|
||||
SENSE
|
||||
Process Raw Data
|
||||
Listen to Feeds
|
||||
Data Validation
|
||||
Format Conversion
|
||||
MEMORY
|
||||
Store Data
|
||||
Retrieve Data
|
||||
Update Data
|
||||
Delete Data
|
||||
List Keys
|
||||
COGNITION
|
||||
Reason
|
||||
Infer
|
||||
Execute Logic
|
||||
Pattern Recognition
|
||||
CONDUIT
|
||||
Establish Connection
|
||||
Read Data
|
||||
Write Data
|
||||
Monitor Stream
|
||||
Sync Instances
|
||||
ACTIVATION
|
||||
Trigger Actions
|
||||
Generate Output
|
||||
Interact External
|
||||
Event Handling
|
||||
PERCEPTION
|
||||
Interpret Data
|
||||
Organize Info
|
||||
Generate Visualization
|
||||
Context Analysis
|
||||
DIALOG
|
||||
Process User Input
|
||||
Generate Response
|
||||
Manage Context
|
||||
NLP Operations
|
||||
WEBSOCKET
|
||||
Connect WebSocket
|
||||
Send Messages
|
||||
Message Callbacks
|
||||
Stream Monitoring
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Optimizations
|
||||
|
||||
### Multithreading Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Main Thread"
|
||||
MT[Main Thread<br/>Coordination & API]
|
||||
end
|
||||
|
||||
subgraph "Worker Pool"
|
||||
W1[Worker 1<br/>Embedding<br/>Generation]
|
||||
W2[Worker 2<br/>Vector<br/>Search]
|
||||
W3[Worker 3<br/>Batch<br/>Processing]
|
||||
WN[Worker N<br/>Custom<br/>Operations]
|
||||
end
|
||||
|
||||
subgraph "GPU Acceleration"
|
||||
GPU[TensorFlow.js<br/>WebGL Backend<br/>GPU Compute]
|
||||
CPU[CPU Fallback<br/>Compatibility<br/>Mode]
|
||||
end
|
||||
|
||||
MT -->|Distribute Tasks| W1
|
||||
MT -->|Distribute Tasks| W2
|
||||
MT -->|Distribute Tasks| W3
|
||||
MT -->|Distribute Tasks| WN
|
||||
|
||||
W1 --> GPU
|
||||
W2 --> GPU
|
||||
W3 --> GPU
|
||||
WN --> GPU
|
||||
|
||||
GPU -.->|Fallback| CPU
|
||||
|
||||
W1 -->|Results| MT
|
||||
W2 -->|Results| MT
|
||||
W3 -->|Results| MT
|
||||
WN -->|Results| MT
|
||||
|
||||
style MT fill:#e3f2fd
|
||||
style W1 fill:#e8f5e8
|
||||
style W2 fill:#e8f5e8
|
||||
style W3 fill:#e8f5e8
|
||||
style WN fill:#e8f5e8
|
||||
style GPU fill:#ffebee
|
||||
style CPU fill:#fff3e0
|
||||
```
|
||||
|
||||
### Performance Metrics
|
||||
|
||||
```mermaid
|
||||
xychart-beta
|
||||
title "Performance Improvements with Optimizations"
|
||||
x-axis [Baseline, Caching, Multithreading, GPU, All Combined]
|
||||
y-axis "Operations/Second" 0 --> 10000
|
||||
bar [1000, 3000, 5000, 7000, 9500]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cross-Platform Integration
|
||||
|
||||
### Synchronization Network
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Browser Instances"
|
||||
B1[Browser 1]
|
||||
B2[Browser 2]
|
||||
B3[Browser 3]
|
||||
end
|
||||
|
||||
subgraph "Server Infrastructure"
|
||||
WS[WebSocket Server]
|
||||
API[REST API Server]
|
||||
S3[S3/Cloud Storage]
|
||||
end
|
||||
|
||||
subgraph "Peer-to-Peer"
|
||||
STUN[STUN Server]
|
||||
SIGNAL[Signaling Server]
|
||||
end
|
||||
|
||||
subgraph "External AI"
|
||||
MCP[MCP Server]
|
||||
AI[AI Models]
|
||||
end
|
||||
|
||||
B1 <-->|WebSocket| WS
|
||||
B2 <-->|WebSocket| WS
|
||||
B3 <-->|WebSocket| WS
|
||||
|
||||
B1 <-.->|WebRTC| B2
|
||||
B2 <-.->|WebRTC| B3
|
||||
B1 <-.->|WebRTC| B3
|
||||
|
||||
WS <--> S3
|
||||
API <--> S3
|
||||
|
||||
B1 -.->|Signaling| SIGNAL
|
||||
B2 -.->|Signaling| SIGNAL
|
||||
B3 -.->|Signaling| SIGNAL
|
||||
|
||||
SIGNAL -.-> STUN
|
||||
|
||||
WS <--> MCP
|
||||
MCP <--> AI
|
||||
|
||||
style B1 fill:#e3f2fd
|
||||
style B2 fill:#e3f2fd
|
||||
style B3 fill:#e3f2fd
|
||||
style WS fill:#e8f5e8
|
||||
style API fill:#e8f5e8
|
||||
style S3 fill:#fff3e0
|
||||
style MCP fill:#f3e5f5
|
||||
style AI fill:#ffebee
|
||||
```
|
||||
|
||||
### Model Control Protocol (MCP) Integration
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant AI as External AI Model
|
||||
participant MCP as MCP Server
|
||||
participant Adapter as Brainy MCP Adapter
|
||||
participant Brainy as Brainy Database
|
||||
|
||||
AI->>MCP: Request data access
|
||||
MCP->>Adapter: Forward request
|
||||
Adapter->>Brainy: Query data
|
||||
Brainy-->>Adapter: Return results
|
||||
Adapter-->>MCP: Formatted response
|
||||
MCP-->>AI: Data payload
|
||||
|
||||
AI->>MCP: Execute augmentation
|
||||
MCP->>Adapter: Pipeline request
|
||||
Adapter->>Brainy: Run augmentation
|
||||
Brainy-->>Adapter: Processing result
|
||||
Adapter-->>MCP: Tool response
|
||||
MCP-->>AI: Execution result
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Flow Example
|
||||
|
||||
### Complete Processing Pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph "Input Processing"
|
||||
I1[Input: "Cats are independent pets"]
|
||||
I2[Metadata: {noun: "Thing", category: "animal"}]
|
||||
end
|
||||
|
||||
subgraph "Embedding Generation"
|
||||
E1[TensorFlow.js Universal Sentence Encoder]
|
||||
E2[Vector: [0.123, -0.456, 0.789, ...]]
|
||||
end
|
||||
|
||||
subgraph "Storage & Indexing"
|
||||
S1[Store in Multi-tier Cache]
|
||||
S2[Add to HNSW Index]
|
||||
S3[Persist to Storage Layer]
|
||||
end
|
||||
|
||||
subgraph "Query Processing"
|
||||
Q1[Query: "feline pets"]
|
||||
Q2[Generate Query Vector]
|
||||
Q3[HNSW Similarity Search]
|
||||
Q4[Retrieve & Rank Results]
|
||||
end
|
||||
|
||||
subgraph "Graph Operations"
|
||||
G1[Add Relationship]
|
||||
G2[catId --RelatedTo--> dogId]
|
||||
G3[Store Verb Metadata]
|
||||
end
|
||||
|
||||
I1 --> E1
|
||||
I2 --> E1
|
||||
E1 --> E2
|
||||
E2 --> S1
|
||||
S1 --> S2
|
||||
S2 --> S3
|
||||
|
||||
Q1 --> Q2
|
||||
Q2 --> Q3
|
||||
Q3 --> Q4
|
||||
|
||||
E2 -.-> G1
|
||||
G1 --> G2
|
||||
G2 --> G3
|
||||
|
||||
style I1 fill:#e8f5e8
|
||||
style E1 fill:#e3f2fd
|
||||
style E2 fill:#f3e5f5
|
||||
style S1 fill:#fff3e0
|
||||
style Q1 fill:#e8f5e8
|
||||
style Q4 fill:#ffebee
|
||||
style G2 fill:#e0f2f1
|
||||
```
|
||||
|
||||
### Result Example
|
||||
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"id": "noun-123",
|
||||
"text": "Cats are independent pets",
|
||||
"similarity": 0.89,
|
||||
"metadata": {
|
||||
"noun": "Thing",
|
||||
"category": "animal"
|
||||
}
|
||||
}
|
||||
],
|
||||
"query": "feline pets",
|
||||
"processingTime": "15ms",
|
||||
"cacheHit": false
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Architecture Principles
|
||||
|
||||
### 🌐 **Environment Agnostic**
|
||||
Automatically adapts to browser, Node.js, serverless, container, or server environments without code changes.
|
||||
|
||||
### 🧠 **Intelligent Storage**
|
||||
Multi-tier caching with automatic storage selection optimizes for performance and persistence across platforms.
|
||||
|
||||
### 🔍 **Vector + Graph Unified**
|
||||
Combines semantic vector search with graph relationships in a single, coherent data model.
|
||||
|
||||
### 🔧 **Extensible Pipeline**
|
||||
Modular augmentation system allows custom processing, AI integration, and workflow automation.
|
||||
|
||||
### ⚡ **Performance Optimized**
|
||||
GPU acceleration, multithreading, intelligent caching, and memory management deliver enterprise-grade performance.
|
||||
|
||||
### 🔄 **Scalable Synchronization**
|
||||
WebSocket and WebRTC conduits enable real-time synchronization across instances and platforms.
|
||||
|
||||
### 🤖 **AI Integration Ready**
|
||||
Built-in MCP protocol support allows external AI models to access Brainy data and utilize augmentation tools.
|
||||
|
||||
---
|
||||
|
||||
*Generated from Brainy v0.34.0 Architecture Documentation*
|
||||
Loading…
Add table
Add a link
Reference in a new issue