soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

**feat(docs): add comprehensive architecture documentation for Brainy**

Browse source 
- **Documentation Additions**:
  - Created `brainy_architecture_diagram.md` to detail Brainy's architecture using diagrams and structured descriptions:
    - Added overviews of the system, core architecture, and augmentation pipeline.
    - Defined data models, graph structures, storage architecture, and performance optimizations.
    - Explained vector search engine design, HNSW index structure, and usage flow examples.
  - Developed `brainy_architecture_visual.md` to complement the architecture with visual aids in Mermaid.js:
    - Provided detailed flowcharts, mind maps, and sequence diagrams for system components and data flow.

- **Purpose**:
  - Provide in-depth technical insights into Brainy's architecture for developers and stakeholders.
  - Enhance understanding of the system's core design principles with easy-to-follow diagrams and examples.
This commit is contained in:
David Snelling 2025-08-02 16:05:48 -07:00
parent 672be32bea
commit 5f9e807952
10 changed files with 2842 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

53
dev/README.md Normal file
View file

@ -0,0 +1,53 @@
# Development Tools & Documentation
This directory contains development tools, scripts, and documentation files that are not included in the published npm package.
## Directory Structure
```
dev/
├── docs/ # Development documentation files
│ ├── brainy_architecture_diagram.md
│ ├── PDF_GENERATION_GUIDE.md
│ ├── QUICK_PDF_SETUP.md
│ └── brainy_architecture_visual.md
├── scripts/ # Development scripts
│ └── generate-architecture-pdf.js
└── README.md # This file
```
## Scripts
### generate-architecture-pdf.js
Generates a professional PDF documentation of Brainy's architecture using:
- Material Design styling
- Custom SVG diagrams
- Comprehensive content from README.md
- Professional visual presentation
**Usage:**
```bash
# From project root
node dev/scripts/generate-architecture-pdf.js
# Or add to package.json scripts
npm run generate-docs
```
**Requirements:**
- puppeteer (for PDF generation)
**Output:**
- `docs/Brainy_Architecture_Documentation.pdf`
## Documentation Files
- **brainy_architecture_diagram.md**: ASCII art diagrams of system architecture
- **PDF_GENERATION_GUIDE.md**: Detailed guide for PDF generation setup
- **QUICK_PDF_SETUP.md**: Quick setup instructions
- **brainy_architecture_visual.md**: Visual architecture documentation
## NPM Package Exclusion
This entire `dev/` directory is excluded from the published npm package via `.npmignore` to keep the package size minimal and focused on the core library functionality.

242
dev/docs/PDF_GENERATION_GUIDE.md Normal file
View file

@ -0,0 +1,242 @@
# Brainy Architecture PDF Generation Guide
This guide shows you how to generate a professional PDF from the Brainy architecture documentation with beautiful diagrams.
## Quick Start
### Option 1: Using the npm script (Recommended)
```bash
# Make sure you're in the brainy project directory
cd /path/to/brainy
# Install dependencies if not already installed
npm install
# Generate the PDF
npm run generate-pdf
```
### Option 2: Direct script execution
```bash
# Make sure you're in the brainy project directory
cd /path/to/brainy
# Install Puppeteer if not already installed
npm install puppeteer
# Run the script directly
node dev/dev/scripts/generate-architecture-pdf.js
```
## Installation Requirements
### Prerequisites
- Node.js 18+
- npm or yarn
### Dependencies
The script uses:
- **Puppeteer**: For PDF generation and browser automation
- **Mermaid**: For rendering diagrams (loaded via CDN)
- **Google Fonts**: For professional typography (loaded via CDN)
### Install Dependencies
```bash
# If you don't have puppeteer installed globally or in the project
npm install puppeteer
# Or install as dev dependency
npm install --save-dev puppeteer
```
## Output
The PDF will be generated at:
```
docs/Brainy_Architecture_Documentation.pdf
```
## Features of the Generated PDF
### Professional Styling
- **Modern Typography**: Uses Inter font family for clean, readable text
- **Code Font**: JetBrains Mono for code blocks and technical content
- **Color Scheme**: Professional blue theme with proper contrast
- **Layout**: A4 format with proper margins and spacing
### Rich Diagrams
- **Mermaid Diagrams**: All diagrams are rendered as vector graphics
- **Interactive Elements**: Flowcharts, sequence diagrams, mindmaps, and more
- **Consistent Styling**: All diagrams follow the same color scheme
- **High Quality**: Vector-based rendering for crisp output
### Document Structure
- **Table of Contents**: Linked navigation
- **Page Headers/Footers**: Professional branding and page numbers
- **Section Breaks**: Logical page breaks between major sections
- **Code Highlighting**: Syntax highlighting for JSON and code blocks
## Customization
### Modify Styling
Edit the `professionalCSS` variable in `dev/scripts/generate-architecture-pdf.js`:
```javascript
const professionalCSS = `
/* Your custom CSS here */
h1 {
color: #your-color;
font-size: 24pt;
}
/* ... */
`
```
### Change Output Location
Modify the `config` object:
```javascript
const config = {
inputFile: path.join(__dirname, '../docs/brainy_architecture_visual.md'),
outputFile: path.join(__dirname, '../docs/YOUR_CUSTOM_NAME.pdf'),
// ...
}
```
### Adjust PDF Settings
Modify the `page.pdf()` options:
```javascript
await page.pdf({
path: config.outputFile,
format: 'A4', // or 'Letter', 'Legal', etc.
printBackground: true,
margin: {
top: '20mm',
right: '15mm',
bottom: '20mm',
left: '15mm'
},
// ... other options
})
```
## Troubleshooting
### Common Issues
#### 1. "Puppeteer not found"
```bash
npm install puppeteer
```
#### 2. "Chrome/Chromium not found"
```bash
# On Ubuntu/Debian
sudo apt-get install chromium-browser
# On macOS
brew install chromium
# Or let Puppeteer download Chromium
npm install puppeteer --unsafe-perm=true
```
#### 3. "Permission denied"
```bash
chmod +x dev/scripts/generate-architecture-pdf.js
```
#### 4. "Diagrams not rendering"
Check your internet connection - Mermaid is loaded from CDN. For offline use, you can download mermaid.min.js locally and update the path.
### Advanced Configuration
#### Use Local Mermaid
Download mermaid.min.js and update the config:
```javascript
const config = {
// ...
mermaidCDN: './path/to/mermaid.min.js'
}
```
#### Custom Fonts
Add additional fonts to the CSS:
```css
@import url('https://fonts.googleapis.com/css2?family=YourFont:wght@400;500;600&display=swap');
body {
font-family: 'YourFont', sans-serif;
}
```
## Adding to package.json
Add this script to your `package.json`:
```json
{
"scripts": {
"generate-pdf": "node dev/dev/scripts/generate-architecture-pdf.js",
"docs:pdf": "npm run generate-pdf"
},
"devDependencies": {
"puppeteer": "^22.5.0"
}
}
```
## Alternative PDF Generators
If you prefer other tools, you can also use:
### 1. Pandoc + LaTeX
```bash
# Install pandoc and latex
sudo apt-get install pandoc texlive-latex-recommended
# Convert (note: won't render Mermaid diagrams)
pandoc docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf
```
### 2. mdpdf
```bash
npm install -g mdpdf
mdpdf docs/brainy_architecture_visual.md --output=docs/brainy_architecture.pdf
```
### 3. markdown-pdf
```bash
npm install -g markdown-pdf
markdown-pdf docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf
```
**Note**: The custom Puppeteer script provides the best results with proper Mermaid diagram rendering and professional styling.
## Sample Output
The generated PDF will include:
1. **Cover Page** with title and subtitle
2. **Table of Contents** with page links
3. **System Overview** with environment detection diagram
4. **Core Architecture** with layered architecture diagram
5. **Data Model** with noun/verb type hierarchies
6. **Vector Search Engine** with HNSW visualization
7. **Storage Architecture** with multi-tier caching diagrams
8. **Augmentation Pipeline** with flow diagrams
9. **Performance Optimizations** with threading models
10. **Integration Patterns** with network topology
11. **Data Flow Examples** with sequence diagrams
Total pages: ~25-30 pages with full diagrams and explanations.
---
*For questions or issues with PDF generation, please check the troubleshooting section or create an issue in the repository.*

80
dev/docs/QUICK_PDF_SETUP.md Normal file
View file

@ -0,0 +1,80 @@
# Quick PDF Generation Setup
## 🚀 Generate Professional Brainy Architecture PDF
### One-Command Setup & Generation
```bash
# Install Puppeteer and generate PDF in one go
npm install puppeteer && npm run generate-pdf
```
### Step-by-Step
1. **Install Puppeteer** (if not already installed):
```bash
npm install puppeteer
```
2. **Generate the PDF**:
```bash
npm run generate-pdf
```
3. **Find your PDF**:
```
docs/Brainy_Architecture_Documentation.pdf
```
## ✨ What You Get
- **25-30 page professional PDF** with full diagrams
- **Vector graphics** for all Mermaid diagrams
- **Modern typography** with Inter font family
- **Consistent branding** throughout the document
- **Table of contents** with page links
- **Professional headers/footers**
## 📊 Sample Sections Include
- System Overview with environment detection
- Core Architecture layers
- Data Model (23 Noun Types, 38 Verb Types)
- Vector Search Engine with HNSW visualization
- Storage Architecture with multi-tier caching
- Augmentation Pipeline flows
- Performance optimizations
- Cross-platform integration patterns
- Real data flow examples
## 🛠️ Troubleshooting
### Issue: "Puppeteer not found"
```bash
npm install puppeteer
```
### Issue: "Chrome not found"
```bash
# Let Puppeteer download Chromium
npm install puppeteer --unsafe-perm=true
```
### Issue: "Permission denied"
```bash
chmod +x dev/scripts/generate-architecture-pdf.js
```
## 🎨 Customization
Edit `dev/scripts/generate-architecture-pdf.js` to:
- Change colors and fonts
- Modify page layout
- Adjust diagram styling
- Add custom branding
---
**Ready to generate?** Run `npm run generate-pdf` and get your professional architecture documentation!
For detailed setup instructions, see `PDF_GENERATION_GUIDE.md`.

313
dev/docs/brainy_architecture_diagram.md Normal file
View file

@ -0,0 +1,313 @@
# Brainy Architecture Diagram
## System Overview
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ BRAINY PLATFORM │
│ Vector Graph Database with AI Pipeline │
└─────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────┐
│ ENVIRONMENT DETECTION │
├─────────────────────────────────────────────────────────────────────────────────┤
│ Browser │ Node.js │ Serverless │ Container │ Server │
│ (OPFS) │ (File System) │ (In-Memory) │ (Adaptive) │ (S3/Cloud) │
└─────────────────────────────────────────────────────────────────────────────────┘
│
▼
```
## Core Architecture
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ BRAINY DATA API │
├─────────────────────────────────────────────────────────────────────────────────┤
│ add() │ search() │ addVerb() │ get() │ delete() │ backup() │ restore() │ etc. │
└─────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│ AUGMENTATION PIPELINE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ SENSE → MEMORY → COGNITION → CONDUIT → ACTIVATION → PERCEPTION → DIALOG → WS │
└─────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│ DATA PROCESSING │
├─────────────────────────────────────────────────────────────────────────────────┤
│ Text/JSON → Embedding → Vector Storage │
│ │ │
│ ┌─────────────────────────┼─────────────────────────┐ │
│ │ EMBEDDING │ VECTOR INDEX │ │
│ │ │ │ │
│ │ TensorFlow.js │ HNSW Algorithm │ │
│ │ Universal Sentence │ - Hierarchical │ │
│ │ Encoder (USE) │ - Fast Similarity │ │
│ │ - GPU Acceleration │ - Configurable │ │
│ │ - Batch Processing │ - Memory Efficient │ │
│ │ - Worker Threads │ - Product Quantized │ │
│ └─────────────────────────┼─────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
│
▼
```
## Data Model & Graph Structure
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ GRAPH DATA MODEL │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ NOUNS (Entities/Nodes) │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ Core Entity Types: │ Digital/Content Types: │ │
│ │ • Person │ • Document │ │
│ │ • Organization │ • Media │ │
│ │ • Location │ • File │ │
│ │ • Thing │ • Message │ │
│ │ • Concept │ • Content │ │
│ │ • Event │ │ │
│ │ │ Collection Types: │ │
│ │ Business/App Types: │ • Collection │ │
│ │ • Product │ • Dataset │ │
│ │ • Service │ │ │
│ │ • User │ Descriptive Types: │ │
│ │ • Task │ • Process, State, Role │ │
│ │ • Project │ • Topic, Language, Currency, Measurement │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ VERBS (Relationships/Edges) │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ Core Relationships: │ Social/Organizational: │ │
│ │ • RelatedTo │ • MemberOf, WorksWith │ │
│ │ • Contains, PartOf │ • FriendOf, Follows, Likes │ │
│ │ • LocatedAt, References │ • ReportsTo, Supervises, Mentors │ │
│ │ │ • Communicates │ │
│ │ Temporal/Causal: │ │ │
│ │ • Precedes, Succeeds │ Descriptive/Functional: │ │
│ │ • Causes, DependsOn │ • Describes, Defines, Categorizes │ │
│ │ • Requires │ • Measures, Evaluates │ │
│ │ │ • Uses, Implements, Extends │ │
│ │ Creation/Transformation: │ │ │
│ │ • Creates, Transforms │ Ownership/Attribution: │ │
│ │ • Becomes, Modifies │ • Owns, AttributedTo │ │
│ │ • Consumes │ • CreatedBy, BelongsTo │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Vector Storage & Search Engine
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ VECTOR SEARCH ENGINE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ Query Text/Vector → Embedding → HNSW Search → Ranked Results │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ HNSW INDEX STRUCTURE │ │
│ │ │ │
│ │ Layer 2: ●────────●────────● (Sparse connections) │ │
│ │ ╱│ │ │╲ │ │
│ │ Layer 1: ●─●──●─●─●─●──●─●─●─● (Medium density) │ │
│ │ ╱│││││││││││││││││││││╲ │ │
│ │ Layer 0: ●●●●●●●●●●●●●●●●●●●●●●● (Dense connections) │ │
│ │ │ │
│ │ • Hierarchical navigation for fast search │ │
│ │ • Configurable M (max connections), efConstruction, efSearch │ │
│ │ • Memory-efficient with disk-based storage for large datasets │ │
│ │ • Product quantization for dimensionality reduction │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Storage Architecture
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ ADAPTIVE STORAGE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─ Hot Cache (RAM) ──┐ │
│ │ Most accessed │ │
│ │ LRU eviction │ │
│ │ Auto-tuned size │ │
│ └─────────────────────┘ │
│ │ │
│ ┌─ Warm Cache (Storage) ─┐ │
│ │ Recent nodes │ │
│ │ OPFS/Filesystem/S3 │ │
│ │ TTL-based │ │
│ └─────────────────────────┘ │
│ │ │
│ ┌─ Cold Storage (Persistent) ─┐ │
│ │ All nodes │ │
│ │ OPFS/Filesystem/S3 │ │
│ │ Batch operations │ │
│ └─────────────────────────────┘ │
│ │
│ Environment-Specific Storage Adapters: │
│ ┌─────────────┬─────────────┬─────────────┬─────────────┬─────────────┐ │
│ │ Browser │ Node.js │ Serverless │ Container │ Server │ │
│ │ OPFS │ FileSystem │ In-Memory │ Adaptive │ S3/Cloud │ │
│ │ (Fallback: │ (Backup: │ (Optional: │ (Auto- │ (Multi- │ │
│ │ IndexedDB) │ S3/Cloud) │ S3/Cloud) │ Detect) │ Provider) │ │
│ └─────────────┴─────────────┴─────────────┴─────────────┴─────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Augmentation Pipeline System
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ AUGMENTATION PIPELINE FLOW │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ Raw Data → [SENSE] → [MEMORY] → [COGNITION] → [CONDUIT] → [ACTIVATION] → │
│ │ │ │ │ │ │
│ ▼ ▼ ▼ ▼ ▼ │
│ Process Storage Reasoning Data Sync Actions │
│ Input Persist Inference External Triggers │
│ Convert Retrieve Logic Ops Systems Responses │
│ │
│ → [PERCEPTION] → [DIALOG] → [WEBSOCKET] → │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ Visualization NLP/Chat Real-time │
│ Interpretation Response Streaming │
│ Organization Context Communication │
│ │
│ Execution Modes: │
│ • SEQUENTIAL: Step-by-step processing │
│ • PARALLEL: Concurrent augmentation execution │
│ • THREADED: Multi-threaded with worker pools │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Performance & Scaling Features
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ PERFORMANCE OPTIMIZATIONS │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ MULTITHREADING │ │
│ │ │ │
│ │ Main Thread ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ ├──────→ │ Worker 1 │ │ Worker 2 │ │ Worker N │ │ │
│ │ │ │Embedding │ │ Search │ │ Batch │ │ │
│ │ │ │Generation│ │Operations│ │Processing│ │ │
│ │ ←──────── └──────────┘ └──────────┘ └──────────┘ │ │
│ │ │ │
│ │ • Web Workers (Browser) / Worker Threads (Node.js) │ │
│ │ • Model caching and reuse across workers │ │
│ │ • Batch embedding for better performance │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ GPU ACCELERATION │ │
│ │ │ │
│ │ TensorFlow.js → WebGL Backend → GPU │ │
│ │ ↓ │ │
│ │ Fallback: CPU Backend for compatibility │ │
│ │ │ │
│ │ • Vector similarity calculations │ │
│ │ • Embedding generation │ │
│ │ • Tensor operations │ │
│ │ • Automatic memory management │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ INTELLIGENT CACHING │ │
│ │ │ │
│ │ • Auto-tuning based on usage patterns │ │
│ │ • Memory-aware cache sizing │ │
│ │ • Prefetching strategies │ │
│ │ • LRU eviction with batch processing │ │
│ │ • Read-only mode optimizations │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Cross-Platform Integration
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ SYNCHRONIZATION & SCALING │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ Browser ←→ WebSocket ←→ Server ←→ S3/Cloud Storage │
│ ↓ ↓ │
│ Browser ←→ WebRTC ←→ Browser (Peer-to-Peer) │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ CONDUIT AUGMENTATIONS │ │
│ │ │ │
│ │ WebSocket iConduit: │ │
│ │ • Browser ↔ Server sync │ │
│ │ • Server ↔ Server sync │ │
│ │ • Real-time data streaming │ │
│ │ │ │
│ │ WebRTC iConduit: │ │
│ │ • Direct browser ↔ browser sync │ │
│ │ • Peer-to-peer without server │ │
│ │ • Decentralized data sharing │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ MODEL CONTROL PROTOCOL (MCP) │ │
│ │ │ │
│ │ External AI Models ←→ MCP Server ←→ Brainy Data & Tools │ │
│ │ │ │
│ │ • BrainyMCPAdapter: Data access for external models │ │
│ │ • MCPAugmentationToolset: Pipeline tools for models │ │
│ │ • BrainyMCPService: WebSocket & REST integration │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Data Flow Example
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ DATA FLOW EXAMPLE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. Input: "Cats are independent pets" │
│ ↓ │
│ 2. SENSE Augmentation: Process raw text │
│ ↓ │
│ 3. Embedding: TensorFlow USE → [0.123, -0.456, 0.789, ...] │
│ ↓ │
│ 4. MEMORY Augmentation: Store with metadata │
│ ↓ │
│ 5. HNSW Index: Add vector to hierarchical graph │
│ ↓ │
│ 6. Storage: Persist to OPFS/FileSystem/S3 │
│ │
│ Query: "feline pets" → Embedding → HNSW Search → Ranked Results │
│ Result: [{text: "Cats are independent pets", similarity: 0.89, id: "123"}] │
│ │
│ Relationship Example: │
│ addVerb(catId, dogId, VerbType.RelatedTo, {description: "Both are pets"}) │
│ ↓ │
│ Graph: [Cat] ──RelatedTo──→ [Dog] │
└─────────────────────────────────────────────────────────────────────────────────┘
```
---
**Key Architecture Principles:**
1. **Environment Agnostic**: Automatically adapts to browser, Node.js, serverless, container, or server environments
2. **Intelligent Storage**: Multi-tier caching with automatic storage selection (OPFS, filesystem, S3, memory)
3. **Vector + Graph**: Combines semantic vector search with graph relationships in a unified model
4. **Extensible Pipeline**: Modular augmentation system for custom processing and integration
5. **Performance Optimized**: GPU acceleration, multithreading, intelligent caching, and memory management
6. **Scalable Sync**: WebSocket and WebRTC conduits for real-time synchronization across instances
7. **AI Integration**: MCP protocol for external AI model integration and tool access

1421
dev/scripts/generate-architecture-pdf.js Executable file
View file

File diff suppressed because it is too large Load diff