soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

**feat(docs): add comprehensive architecture documentation for Brainy**

Browse source 
- **Documentation Additions**:
  - Created `brainy_architecture_diagram.md` to detail Brainy's architecture using diagrams and structured descriptions:
    - Added overviews of the system, core architecture, and augmentation pipeline.
    - Defined data models, graph structures, storage architecture, and performance optimizations.
    - Explained vector search engine design, HNSW index structure, and usage flow examples.
  - Developed `brainy_architecture_visual.md` to complement the architecture with visual aids in Mermaid.js:
    - Provided detailed flowcharts, mind maps, and sequence diagrams for system components and data flow.

- **Purpose**:
  - Provide in-depth technical insights into Brainy's architecture for developers and stakeholders.
  - Enhance understanding of the system's core design principles with easy-to-follow diagrams and examples.
This commit is contained in:
David Snelling 2025-08-02 16:05:48 -07:00
parent 672be32bea
commit 5f9e807952
10 changed files with 2842 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.npmignore
View file

@ -12,6 +12,7 @@ examples/
.idea/
cloud-wrapper/
scripts/
dev/
# Configuration files
.eslintrc

53
dev/README.md Normal file
View file

@ -0,0 +1,53 @@
# Development Tools & Documentation
This directory contains development tools, scripts, and documentation files that are not included in the published npm package.
## Directory Structure
```
dev/
├── docs/ # Development documentation files
│ ├── brainy_architecture_diagram.md
│ ├── PDF_GENERATION_GUIDE.md
│ ├── QUICK_PDF_SETUP.md
│ └── brainy_architecture_visual.md
├── scripts/ # Development scripts
│ └── generate-architecture-pdf.js
└── README.md # This file
```
## Scripts
### generate-architecture-pdf.js
Generates a professional PDF documentation of Brainy's architecture using:
- Material Design styling
- Custom SVG diagrams
- Comprehensive content from README.md
- Professional visual presentation
**Usage:**
```bash
# From project root
node dev/scripts/generate-architecture-pdf.js
# Or add to package.json scripts
npm run generate-docs
```
**Requirements:**
- puppeteer (for PDF generation)
**Output:**
- `docs/Brainy_Architecture_Documentation.pdf`
## Documentation Files
- **brainy_architecture_diagram.md**: ASCII art diagrams of system architecture
- **PDF_GENERATION_GUIDE.md**: Detailed guide for PDF generation setup
- **QUICK_PDF_SETUP.md**: Quick setup instructions
- **brainy_architecture_visual.md**: Visual architecture documentation
## NPM Package Exclusion
This entire `dev/` directory is excluded from the published npm package via `.npmignore` to keep the package size minimal and focused on the core library functionality.

242
dev/docs/PDF_GENERATION_GUIDE.md Normal file
View file

@ -0,0 +1,242 @@
# Brainy Architecture PDF Generation Guide
This guide shows you how to generate a professional PDF from the Brainy architecture documentation with beautiful diagrams.
## Quick Start
### Option 1: Using the npm script (Recommended)
```bash
# Make sure you're in the brainy project directory
cd /path/to/brainy
# Install dependencies if not already installed
npm install
# Generate the PDF
npm run generate-pdf
```
### Option 2: Direct script execution
```bash
# Make sure you're in the brainy project directory
cd /path/to/brainy
# Install Puppeteer if not already installed
npm install puppeteer
# Run the script directly
node dev/dev/scripts/generate-architecture-pdf.js
```
## Installation Requirements
### Prerequisites
- Node.js 18+
- npm or yarn
### Dependencies
The script uses:
- **Puppeteer**: For PDF generation and browser automation
- **Mermaid**: For rendering diagrams (loaded via CDN)
- **Google Fonts**: For professional typography (loaded via CDN)
### Install Dependencies
```bash
# If you don't have puppeteer installed globally or in the project
npm install puppeteer
# Or install as dev dependency
npm install --save-dev puppeteer
```
## Output
The PDF will be generated at:
```
docs/Brainy_Architecture_Documentation.pdf
```
## Features of the Generated PDF
### Professional Styling
- **Modern Typography**: Uses Inter font family for clean, readable text
- **Code Font**: JetBrains Mono for code blocks and technical content
- **Color Scheme**: Professional blue theme with proper contrast
- **Layout**: A4 format with proper margins and spacing
### Rich Diagrams
- **Mermaid Diagrams**: All diagrams are rendered as vector graphics
- **Interactive Elements**: Flowcharts, sequence diagrams, mindmaps, and more
- **Consistent Styling**: All diagrams follow the same color scheme
- **High Quality**: Vector-based rendering for crisp output
### Document Structure
- **Table of Contents**: Linked navigation
- **Page Headers/Footers**: Professional branding and page numbers
- **Section Breaks**: Logical page breaks between major sections
- **Code Highlighting**: Syntax highlighting for JSON and code blocks
## Customization
### Modify Styling
Edit the `professionalCSS` variable in `dev/scripts/generate-architecture-pdf.js`:
```javascript
const professionalCSS = `
/* Your custom CSS here */
h1 {
color: #your-color;
font-size: 24pt;
}
/* ... */
`
```
### Change Output Location
Modify the `config` object:
```javascript
const config = {
inputFile: path.join(__dirname, '../docs/brainy_architecture_visual.md'),
outputFile: path.join(__dirname, '../docs/YOUR_CUSTOM_NAME.pdf'),
// ...
}
```
### Adjust PDF Settings
Modify the `page.pdf()` options:
```javascript
await page.pdf({
path: config.outputFile,
format: 'A4', // or 'Letter', 'Legal', etc.
printBackground: true,
margin: {
top: '20mm',
right: '15mm',
bottom: '20mm',
left: '15mm'
},
// ... other options
})
```
## Troubleshooting
### Common Issues
#### 1. "Puppeteer not found"
```bash
npm install puppeteer
```
#### 2. "Chrome/Chromium not found"
```bash
# On Ubuntu/Debian
sudo apt-get install chromium-browser
# On macOS
brew install chromium
# Or let Puppeteer download Chromium
npm install puppeteer --unsafe-perm=true
```
#### 3. "Permission denied"
```bash
chmod +x dev/scripts/generate-architecture-pdf.js
```
#### 4. "Diagrams not rendering"
Check your internet connection - Mermaid is loaded from CDN. For offline use, you can download mermaid.min.js locally and update the path.
### Advanced Configuration
#### Use Local Mermaid
Download mermaid.min.js and update the config:
```javascript
const config = {
// ...
mermaidCDN: './path/to/mermaid.min.js'
}
```
#### Custom Fonts
Add additional fonts to the CSS:
```css
@import url('https://fonts.googleapis.com/css2?family=YourFont:wght@400;500;600&display=swap');
body {
font-family: 'YourFont', sans-serif;
}
```
## Adding to package.json
Add this script to your `package.json`:
```json
{
"scripts": {
"generate-pdf": "node dev/dev/scripts/generate-architecture-pdf.js",
"docs:pdf": "npm run generate-pdf"
},
"devDependencies": {
"puppeteer": "^22.5.0"
}
}
```
## Alternative PDF Generators
If you prefer other tools, you can also use:
### 1. Pandoc + LaTeX
```bash
# Install pandoc and latex
sudo apt-get install pandoc texlive-latex-recommended
# Convert (note: won't render Mermaid diagrams)
pandoc docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf
```
### 2. mdpdf
```bash
npm install -g mdpdf
mdpdf docs/brainy_architecture_visual.md --output=docs/brainy_architecture.pdf
```
### 3. markdown-pdf
```bash
npm install -g markdown-pdf
markdown-pdf docs/brainy_architecture_visual.md -o docs/brainy_architecture.pdf
```
**Note**: The custom Puppeteer script provides the best results with proper Mermaid diagram rendering and professional styling.
## Sample Output
The generated PDF will include:
1. **Cover Page** with title and subtitle
2. **Table of Contents** with page links
3. **System Overview** with environment detection diagram
4. **Core Architecture** with layered architecture diagram
5. **Data Model** with noun/verb type hierarchies
6. **Vector Search Engine** with HNSW visualization
7. **Storage Architecture** with multi-tier caching diagrams
8. **Augmentation Pipeline** with flow diagrams
9. **Performance Optimizations** with threading models
10. **Integration Patterns** with network topology
11. **Data Flow Examples** with sequence diagrams
Total pages: ~25-30 pages with full diagrams and explanations.
---
*For questions or issues with PDF generation, please check the troubleshooting section or create an issue in the repository.*

80
dev/docs/QUICK_PDF_SETUP.md Normal file
View file

@ -0,0 +1,80 @@
# Quick PDF Generation Setup
## 🚀 Generate Professional Brainy Architecture PDF
### One-Command Setup & Generation
```bash
# Install Puppeteer and generate PDF in one go
npm install puppeteer && npm run generate-pdf
```
### Step-by-Step
1. **Install Puppeteer** (if not already installed):
```bash
npm install puppeteer
```
2. **Generate the PDF**:
```bash
npm run generate-pdf
```
3. **Find your PDF**:
```
docs/Brainy_Architecture_Documentation.pdf
```
## ✨ What You Get
- **25-30 page professional PDF** with full diagrams
- **Vector graphics** for all Mermaid diagrams
- **Modern typography** with Inter font family
- **Consistent branding** throughout the document
- **Table of contents** with page links
- **Professional headers/footers**
## 📊 Sample Sections Include
- System Overview with environment detection
- Core Architecture layers
- Data Model (23 Noun Types, 38 Verb Types)
- Vector Search Engine with HNSW visualization
- Storage Architecture with multi-tier caching
- Augmentation Pipeline flows
- Performance optimizations
- Cross-platform integration patterns
- Real data flow examples
## 🛠️ Troubleshooting
### Issue: "Puppeteer not found"
```bash
npm install puppeteer
```
### Issue: "Chrome not found"
```bash
# Let Puppeteer download Chromium
npm install puppeteer --unsafe-perm=true
```
### Issue: "Permission denied"
```bash
chmod +x dev/scripts/generate-architecture-pdf.js
```
## 🎨 Customization
Edit `dev/scripts/generate-architecture-pdf.js` to:
- Change colors and fonts
- Modify page layout
- Adjust diagram styling
- Add custom branding
---
**Ready to generate?** Run `npm run generate-pdf` and get your professional architecture documentation!
For detailed setup instructions, see `PDF_GENERATION_GUIDE.md`.

313
dev/docs/brainy_architecture_diagram.md Normal file
View file

@ -0,0 +1,313 @@
# Brainy Architecture Diagram
## System Overview
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ BRAINY PLATFORM │
│ Vector Graph Database with AI Pipeline │
└─────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────┐
│ ENVIRONMENT DETECTION │
├─────────────────────────────────────────────────────────────────────────────────┤
│ Browser │ Node.js │ Serverless │ Container │ Server │
│ (OPFS) │ (File System) │ (In-Memory) │ (Adaptive) │ (S3/Cloud) │
└─────────────────────────────────────────────────────────────────────────────────┘
│
▼
```
## Core Architecture
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ BRAINY DATA API │
├─────────────────────────────────────────────────────────────────────────────────┤
│ add() │ search() │ addVerb() │ get() │ delete() │ backup() │ restore() │ etc. │
└─────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│ AUGMENTATION PIPELINE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ SENSE → MEMORY → COGNITION → CONDUIT → ACTIVATION → PERCEPTION → DIALOG → WS │
└─────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│ DATA PROCESSING │
├─────────────────────────────────────────────────────────────────────────────────┤
│ Text/JSON → Embedding → Vector Storage │
│ │ │
│ ┌─────────────────────────┼─────────────────────────┐ │
│ │ EMBEDDING │ VECTOR INDEX │ │
│ │ │ │ │
│ │ TensorFlow.js │ HNSW Algorithm │ │
│ │ Universal Sentence │ - Hierarchical │ │
│ │ Encoder (USE) │ - Fast Similarity │ │
│ │ - GPU Acceleration │ - Configurable │ │
│ │ - Batch Processing │ - Memory Efficient │ │
│ │ - Worker Threads │ - Product Quantized │ │
│ └─────────────────────────┼─────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
│
▼
```
## Data Model & Graph Structure
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ GRAPH DATA MODEL │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ NOUNS (Entities/Nodes) │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ Core Entity Types: │ Digital/Content Types: │ │
│ │ • Person │ • Document │ │
│ │ • Organization │ • Media │ │
│ │ • Location │ • File │ │
│ │ • Thing │ • Message │ │
│ │ • Concept │ • Content │ │
│ │ • Event │ │ │
│ │ │ Collection Types: │ │
│ │ Business/App Types: │ • Collection │ │
│ │ • Product │ • Dataset │ │
│ │ • Service │ │ │
│ │ • User │ Descriptive Types: │ │
│ │ • Task │ • Process, State, Role │ │
│ │ • Project │ • Topic, Language, Currency, Measurement │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ VERBS (Relationships/Edges) │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ Core Relationships: │ Social/Organizational: │ │
│ │ • RelatedTo │ • MemberOf, WorksWith │ │
│ │ • Contains, PartOf │ • FriendOf, Follows, Likes │ │
│ │ • LocatedAt, References │ • ReportsTo, Supervises, Mentors │ │
│ │ │ • Communicates │ │
│ │ Temporal/Causal: │ │ │
│ │ • Precedes, Succeeds │ Descriptive/Functional: │ │
│ │ • Causes, DependsOn │ • Describes, Defines, Categorizes │ │
│ │ • Requires │ • Measures, Evaluates │ │
│ │ │ • Uses, Implements, Extends │ │
│ │ Creation/Transformation: │ │ │
│ │ • Creates, Transforms │ Ownership/Attribution: │ │
│ │ • Becomes, Modifies │ • Owns, AttributedTo │ │
│ │ • Consumes │ • CreatedBy, BelongsTo │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Vector Storage & Search Engine
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ VECTOR SEARCH ENGINE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ Query Text/Vector → Embedding → HNSW Search → Ranked Results │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ HNSW INDEX STRUCTURE │ │
│ │ │ │
│ │ Layer 2: ●────────●────────● (Sparse connections) │ │
│ │ ╱│ │ │╲ │ │
│ │ Layer 1: ●─●──●─●─●─●──●─●─●─● (Medium density) │ │
│ │ ╱│││││││││││││││││││││╲ │ │
│ │ Layer 0: ●●●●●●●●●●●●●●●●●●●●●●● (Dense connections) │ │
│ │ │ │
│ │ • Hierarchical navigation for fast search │ │
│ │ • Configurable M (max connections), efConstruction, efSearch │ │
│ │ • Memory-efficient with disk-based storage for large datasets │ │
│ │ • Product quantization for dimensionality reduction │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Storage Architecture
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ ADAPTIVE STORAGE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─ Hot Cache (RAM) ──┐ │
│ │ Most accessed │ │
│ │ LRU eviction │ │
│ │ Auto-tuned size │ │
│ └─────────────────────┘ │
│ │ │
│ ┌─ Warm Cache (Storage) ─┐ │
│ │ Recent nodes │ │
│ │ OPFS/Filesystem/S3 │ │
│ │ TTL-based │ │
│ └─────────────────────────┘ │
│ │ │
│ ┌─ Cold Storage (Persistent) ─┐ │
│ │ All nodes │ │
│ │ OPFS/Filesystem/S3 │ │
│ │ Batch operations │ │
│ └─────────────────────────────┘ │
│ │
│ Environment-Specific Storage Adapters: │
│ ┌─────────────┬─────────────┬─────────────┬─────────────┬─────────────┐ │
│ │ Browser │ Node.js │ Serverless │ Container │ Server │ │
│ │ OPFS │ FileSystem │ In-Memory │ Adaptive │ S3/Cloud │ │
│ │ (Fallback: │ (Backup: │ (Optional: │ (Auto- │ (Multi- │ │
│ │ IndexedDB) │ S3/Cloud) │ S3/Cloud) │ Detect) │ Provider) │ │
│ └─────────────┴─────────────┴─────────────┴─────────────┴─────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Augmentation Pipeline System
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ AUGMENTATION PIPELINE FLOW │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ Raw Data → [SENSE] → [MEMORY] → [COGNITION] → [CONDUIT] → [ACTIVATION] → │
│ │ │ │ │ │ │
│ ▼ ▼ ▼ ▼ ▼ │
│ Process Storage Reasoning Data Sync Actions │
│ Input Persist Inference External Triggers │
│ Convert Retrieve Logic Ops Systems Responses │
│ │
│ → [PERCEPTION] → [DIALOG] → [WEBSOCKET] → │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ Visualization NLP/Chat Real-time │
│ Interpretation Response Streaming │
│ Organization Context Communication │
│ │
│ Execution Modes: │
│ • SEQUENTIAL: Step-by-step processing │
│ • PARALLEL: Concurrent augmentation execution │
│ • THREADED: Multi-threaded with worker pools │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Performance & Scaling Features
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ PERFORMANCE OPTIMIZATIONS │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ MULTITHREADING │ │
│ │ │ │
│ │ Main Thread ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ ├──────→ │ Worker 1 │ │ Worker 2 │ │ Worker N │ │ │
│ │ │ │Embedding │ │ Search │ │ Batch │ │ │
│ │ │ │Generation│ │Operations│ │Processing│ │ │
│ │ ←──────── └──────────┘ └──────────┘ └──────────┘ │ │
│ │ │ │
│ │ • Web Workers (Browser) / Worker Threads (Node.js) │ │
│ │ • Model caching and reuse across workers │ │
│ │ • Batch embedding for better performance │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ GPU ACCELERATION │ │
│ │ │ │
│ │ TensorFlow.js → WebGL Backend → GPU │ │
│ │ ↓ │ │
│ │ Fallback: CPU Backend for compatibility │ │
│ │ │ │
│ │ • Vector similarity calculations │ │
│ │ • Embedding generation │ │
│ │ • Tensor operations │ │
│ │ • Automatic memory management │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ INTELLIGENT CACHING │ │
│ │ │ │
│ │ • Auto-tuning based on usage patterns │ │
│ │ • Memory-aware cache sizing │ │
│ │ • Prefetching strategies │ │
│ │ • LRU eviction with batch processing │ │
│ │ • Read-only mode optimizations │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Cross-Platform Integration
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ SYNCHRONIZATION & SCALING │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ Browser ←→ WebSocket ←→ Server ←→ S3/Cloud Storage │
│ ↓ ↓ │
│ Browser ←→ WebRTC ←→ Browser (Peer-to-Peer) │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ CONDUIT AUGMENTATIONS │ │
│ │ │ │
│ │ WebSocket iConduit: │ │
│ │ • Browser ↔ Server sync │ │
│ │ • Server ↔ Server sync │ │
│ │ • Real-time data streaming │ │
│ │ │ │
│ │ WebRTC iConduit: │ │
│ │ • Direct browser ↔ browser sync │ │
│ │ • Peer-to-peer without server │ │
│ │ • Decentralized data sharing │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ MODEL CONTROL PROTOCOL (MCP) │ │
│ │ │ │
│ │ External AI Models ←→ MCP Server ←→ Brainy Data & Tools │ │
│ │ │ │
│ │ • BrainyMCPAdapter: Data access for external models │ │
│ │ • MCPAugmentationToolset: Pipeline tools for models │ │
│ │ • BrainyMCPService: WebSocket & REST integration │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
```
## Data Flow Example
```
┌─────────────────────────────────────────────────────────────────────────────────┐
│ DATA FLOW EXAMPLE │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. Input: "Cats are independent pets" │
│ ↓ │
│ 2. SENSE Augmentation: Process raw text │
│ ↓ │
│ 3. Embedding: TensorFlow USE → [0.123, -0.456, 0.789, ...] │
│ ↓ │
│ 4. MEMORY Augmentation: Store with metadata │
│ ↓ │
│ 5. HNSW Index: Add vector to hierarchical graph │
│ ↓ │
│ 6. Storage: Persist to OPFS/FileSystem/S3 │
│ │
│ Query: "feline pets" → Embedding → HNSW Search → Ranked Results │
│ Result: [{text: "Cats are independent pets", similarity: 0.89, id: "123"}] │
│ │
│ Relationship Example: │
│ addVerb(catId, dogId, VerbType.RelatedTo, {description: "Both are pets"}) │
│ ↓ │
│ Graph: [Cat] ──RelatedTo──→ [Dog] │
└─────────────────────────────────────────────────────────────────────────────────┘
```
---
**Key Architecture Principles:**
1. **Environment Agnostic**: Automatically adapts to browser, Node.js, serverless, container, or server environments
2. **Intelligent Storage**: Multi-tier caching with automatic storage selection (OPFS, filesystem, S3, memory)
3. **Vector + Graph**: Combines semantic vector search with graph relationships in a unified model
4. **Extensible Pipeline**: Modular augmentation system for custom processing and integration
5. **Performance Optimized**: GPU acceleration, multithreading, intelligent caching, and memory management
6. **Scalable Sync**: WebSocket and WebRTC conduits for real-time synchronization across instances
7. **AI Integration**: MCP protocol for external AI model integration and tool access

1421
dev/scripts/generate-architecture-pdf.js Executable file
View file

File diff suppressed because it is too large Load diff

BIN
docs/Brainy_Architecture_Documentation.pdf Normal file
View file

Binary file not shown.

729
docs/brainy_architecture_visual.md Normal file
View file

@ -0,0 +1,729 @@
# Brainy Architecture Documentation
## Vector Graph Database with AI Pipeline
---
## Table of Contents
1. [System Overview](#system-overview)
2. [Core Architecture](#core-architecture)
3. [Data Model & Graph Structure](#data-model--graph-structure)
4. [Vector Search Engine](#vector-search-engine)
5. [Storage Architecture](#storage-architecture)
6. [Augmentation Pipeline](#augmentation-pipeline)
7. [Performance Optimizations](#performance-optimizations)
8. [Cross-Platform Integration](#cross-platform-integration)
9. [Data Flow Example](#data-flow-example)
---
## System Overview
Brainy is a powerful, cross-platform vector graph database that intelligently adapts to any environment while providing both semantic vector search and graph relationship capabilities.
```mermaid
graph TD
A[User Application] --> B[Brainy Platform]
B --> C[Environment Detection]
C --> D[Browser<br/>OPFS Storage]
C --> E[Node.js<br/>File System]
C --> F[Serverless<br/>In-Memory]
C --> G[Container<br/>Adaptive]
C --> H[Server<br/>S3/Cloud]
B --> I[Vector Search Engine]
B --> J[Graph Database]
B --> K[Augmentation Pipeline]
style B fill:#e1f5fe
style I fill:#f3e5f5
style J fill:#e8f5e8
style K fill:#fff3e0
```
### Key Features
- **Universal Compatibility**: Runs everywhere - browsers, Node.js, serverless functions, containers
- **Intelligent Adaptation**: Automatically optimizes for environment and usage patterns
- **Dual Nature**: Vector similarity search + graph relationships in one system
- **Real-time Streaming**: Live data processing through extensible pipeline
- **AI Integration**: Built-in TensorFlow.js with GPU acceleration
---
## Core Architecture
```mermaid
graph TB
subgraph "Application Layer"
API[Brainy Data API<br/>add() | search() | addVerb() | get() | delete()]
end
subgraph "Processing Layer"
PIPELINE[Augmentation Pipeline<br/>SENSE → MEMORY → COGNITION → CONDUIT → ACTIVATION → PERCEPTION → DIALOG → WS]
end
subgraph "Engine Layer"
EMBED[Embedding Engine<br/>TensorFlow.js Universal Sentence Encoder]
VECTOR[Vector Index<br/>HNSW Algorithm]
GRAPH[Graph Engine<br/>Noun-Verb Model]
end
subgraph "Storage Layer"
CACHE[Multi-tier Caching<br/>Hot → Warm → Cold]
STORAGE[Adaptive Storage<br/>OPFS | FileSystem | S3 | Memory]
end
API --> PIPELINE
PIPELINE --> EMBED
PIPELINE --> VECTOR
PIPELINE --> GRAPH
EMBED --> CACHE
VECTOR --> CACHE
GRAPH --> STORAGE
CACHE --> STORAGE
style API fill:#e3f2fd
style PIPELINE fill:#f1f8e9
style EMBED fill:#fce4ec
style VECTOR fill:#fff8e1
style GRAPH fill:#e8f5e8
style CACHE fill:#f3e5f5
style STORAGE fill:#efebe9
```
---
## Data Model & Graph Structure
### Noun Types (Entities/Nodes)
```mermaid
mindmap
root((Brainy<br/>Noun Types))
Core Entities
Person
Organization
Location
Thing
Concept
Event
Digital Content
Document
Media
File
Message
Content
Collections
Collection
Dataset
Business/App
Product
Service
User
Task
Project
Descriptive
Process
State
Role
Topic
Language
Currency
Measurement
```
### Verb Types (Relationships/Edges)
```mermaid
mindmap
root((Brainy<br/>Verb Types))
Core Relations
RelatedTo
Contains
PartOf
LocatedAt
References
Temporal/Causal
Precedes
Succeeds
Causes
DependsOn
Requires
Creation/Transform
Creates
Transforms
Becomes
Modifies
Consumes
Ownership/Attribution
Owns
AttributedTo
CreatedBy
BelongsTo
Social/Organizational
MemberOf
WorksWith
FriendOf
Follows
Likes
ReportsTo
Supervises
Mentors
Communicates
Descriptive/Functional
Describes
Defines
Categorizes
Measures
Evaluates
Uses
Implements
Extends
```
### Graph Example
```mermaid
graph LR
A[Person: John Doe<br/>ID: person-123] -->|WorksWith| B[Organization: Acme Corp<br/>ID: org-456]
A -->|CreatedBy| C[Document: Report<br/>ID: doc-789]
A -->|LocatedAt| D[Location: New York<br/>ID: loc-101]
B -->|Contains| E[Project: AI Initiative<br/>ID: proj-202]
C -->|PartOf| E
E -->|Uses| F[Concept: Machine Learning<br/>ID: concept-303]
style A fill:#ffcdd2
style B fill:#c8e6c9
style C fill:#bbdefb
style D fill:#fff9c4
style E fill:#f8bbd9
style F fill:#d1c4e9
```
---
## Vector Search Engine
### HNSW Index Structure
```mermaid
graph TB
subgraph "HNSW Hierarchical Structure"
subgraph "Layer 2 (Sparse)"
L2A((●)) --- L2B((●))
L2B --- L2C((●))
end
subgraph "Layer 1 (Medium Density)"
L1A((●)) --- L1B((●))
L1B --- L1C((●))
L1C --- L1D((●))
L1D --- L1E((●))
L1E --- L1F((●))
L1F --- L1G((●))
L1G --- L1H((●))
end
subgraph "Layer 0 (Dense Connections)"
L0A((●)) --- L0B((●))
L0B --- L0C((●))
L0C --- L0D((●))
L0D --- L0E((●))
L0E --- L0F((●))
L0F --- L0G((●))
L0G --- L0H((●))
L0H --- L0I((●))
L0I --- L0J((●))
L0J --- L0K((●))
L0K --- L0L((●))
L0L --- L0M((●))
L0M --- L0N((●))
L0N --- L0O((●))
L0O --- L0P((●))
end
L2A -.-> L1A
L2A -.-> L1D
L2B -.-> L1C
L2B -.-> L1F
L2C -.-> L1G
L1A -.-> L0A
L1A -.-> L0B
L1B -.-> L0C
L1B -.-> L0D
L1C -.-> L0E
L1C -.-> L0F
L1D -.-> L0G
L1D -.-> L0H
L1E -.-> L0I
L1E -.-> L0J
L1F -.-> L0K
L1F -.-> L0L
L1G -.-> L0M
L1G -.-> L0N
L1H -.-> L0O
L1H -.-> L0P
end
style L2A fill:#ff9999
style L2B fill:#ff9999
style L2C fill:#ff9999
style L1A fill:#99ccff
style L1B fill:#99ccff
style L1C fill:#99ccff
style L1D fill:#99ccff
style L1E fill:#99ccff
style L1F fill:#99ccff
style L1G fill:#99ccff
style L1H fill:#99ccff
style L0A fill:#99ff99
style L0B fill:#99ff99
style L0C fill:#99ff99
style L0D fill:#99ff99
style L0E fill:#99ff99
style L0F fill:#99ff99
style L0G fill:#99ff99
style L0H fill:#99ff99
style L0I fill:#99ff99
style L0J fill:#99ff99
style L0K fill:#99ff99
style L0L fill:#99ff99
style L0M fill:#99ff99
style L0N fill:#99ff99
style L0O fill:#99ff99
style L0P fill:#99ff99
```
### Search Process Flow
```mermaid
sequenceDiagram
participant User
participant API
participant Embedding
participant HNSW
participant Storage
User->>API: searchText("feline pets", 5)
API->>Embedding: embed("feline pets")
Embedding->>Embedding: TensorFlow.js Universal Sentence Encoder
Embedding-->>API: [0.123, -0.456, 0.789, ...]
API->>HNSW: search(vector, k=5)
HNSW->>HNSW: Navigate from top layer
HNSW->>HNSW: Descend to lower layers
HNSW->>HNSW: Find k nearest neighbors
HNSW-->>API: [id1, id2, id3, id4, id5]
API->>Storage: get([id1, id2, id3, id4, id5])
Storage-->>API: [noun1, noun2, noun3, noun4, noun5]
API-->>User: [{text: "Cats are independent pets", similarity: 0.89}, ...]
```
---
## Storage Architecture
### Multi-Tier Caching System
```mermaid
graph TD
subgraph "Memory Hierarchy"
subgraph "Hot Cache (RAM)"
HC[Most Accessed Items<br/>LRU Eviction<br/>Auto-tuned Size<br/>Millisecond Access]
end
subgraph "Warm Cache (Storage)"
WC[Recent Items<br/>TTL-based<br/>Sub-second Access<br/>OPFS/FS/S3]
end
subgraph "Cold Storage (Persistent)"
CS[All Items<br/>Batch Operations<br/>Full Persistence<br/>OPFS/FS/S3]
end
end
subgraph "Environment Adapters"
Browser[Browser<br/>OPFS → IndexedDB]
NodeJS[Node.js<br/>FileSystem → S3]
Serverless[Serverless<br/>Memory → S3]
Container[Container<br/>Auto-detect]
Server[Server<br/>S3/Multi-cloud]
end
User[User Query] --> HC
HC -->|Cache Miss| WC
WC -->|Cache Miss| CS
CS --> Browser
CS --> NodeJS
CS --> Serverless
CS --> Container
CS --> Server
style HC fill:#ffcdd2
style WC fill:#fff9c4
style CS fill:#c8e6c9
style Browser fill:#e1f5fe
style NodeJS fill:#e8f5e8
style Serverless fill:#f3e5f5
style Container fill:#fff3e0
style Server fill:#efebe9
```
### Storage Performance Characteristics
```mermaid
xychart-beta
title "Storage Performance by Environment"
x-axis [Browser, Node.js, Serverless, Container, Server]
y-axis "Latency (ms)" 0 --> 1000
line [50, 10, 200, 30, 100]
```
---
## Augmentation Pipeline
### Pipeline Flow Architecture
```mermaid
flowchart LR
subgraph "Data Processing Pipeline"
Input[Raw Data] --> SENSE[SENSE<br/>Process Input<br/>Convert & Validate]
SENSE --> MEMORY[MEMORY<br/>Storage Operations<br/>Persist & Retrieve]
MEMORY --> COGNITION[COGNITION<br/>Reasoning<br/>Inference & Logic]
COGNITION --> CONDUIT[CONDUIT<br/>Data Sync<br/>External Systems]
CONDUIT --> ACTIVATION[ACTIVATION<br/>Actions<br/>Triggers & Events]
ACTIVATION --> PERCEPTION[PERCEPTION<br/>Visualization<br/>Interpretation]
PERCEPTION --> DIALOG[DIALOG<br/>NLP & Chat<br/>Context & Response]
DIALOG --> WEBSOCKET[WEBSOCKET<br/>Real-time<br/>Streaming & Sync]
WEBSOCKET --> Output[Processed Output]
end
subgraph "Execution Modes"
SEQ[Sequential<br/>Step-by-step]
PAR[Parallel<br/>Concurrent]
THR[Threaded<br/>Worker Pools]
end
Input -.-> SEQ
Input -.-> PAR
Input -.-> THR
style SENSE fill:#e8f5e8
style MEMORY fill:#e3f2fd
style COGNITION fill:#fff3e0
style CONDUIT fill:#f3e5f5
style ACTIVATION fill:#ffebee
style PERCEPTION fill:#e0f2f1
style DIALOG fill:#fce4ec
style WEBSOCKET fill:#e8eaf6
```
### Augmentation Types Detail
```mermaid
mindmap
root((Augmentation<br/>System))
SENSE
Process Raw Data
Listen to Feeds
Data Validation
Format Conversion
MEMORY
Store Data
Retrieve Data
Update Data
Delete Data
List Keys
COGNITION
Reason
Infer
Execute Logic
Pattern Recognition
CONDUIT
Establish Connection
Read Data
Write Data
Monitor Stream
Sync Instances
ACTIVATION
Trigger Actions
Generate Output
Interact External
Event Handling
PERCEPTION
Interpret Data
Organize Info
Generate Visualization
Context Analysis
DIALOG
Process User Input
Generate Response
Manage Context
NLP Operations
WEBSOCKET
Connect WebSocket
Send Messages
Message Callbacks
Stream Monitoring
```
---
## Performance Optimizations
### Multithreading Architecture
```mermaid
graph TB
subgraph "Main Thread"
MT[Main Thread<br/>Coordination & API]
end
subgraph "Worker Pool"
W1[Worker 1<br/>Embedding<br/>Generation]
W2[Worker 2<br/>Vector<br/>Search]
W3[Worker 3<br/>Batch<br/>Processing]
WN[Worker N<br/>Custom<br/>Operations]
end
subgraph "GPU Acceleration"
GPU[TensorFlow.js<br/>WebGL Backend<br/>GPU Compute]
CPU[CPU Fallback<br/>Compatibility<br/>Mode]
end
MT -->|Distribute Tasks| W1
MT -->|Distribute Tasks| W2
MT -->|Distribute Tasks| W3
MT -->|Distribute Tasks| WN
W1 --> GPU
W2 --> GPU
W3 --> GPU
WN --> GPU
GPU -.->|Fallback| CPU
W1 -->|Results| MT
W2 -->|Results| MT
W3 -->|Results| MT
WN -->|Results| MT
style MT fill:#e3f2fd
style W1 fill:#e8f5e8
style W2 fill:#e8f5e8
style W3 fill:#e8f5e8
style WN fill:#e8f5e8
style GPU fill:#ffebee
style CPU fill:#fff3e0
```
### Performance Metrics
```mermaid
xychart-beta
title "Performance Improvements with Optimizations"
x-axis [Baseline, Caching, Multithreading, GPU, All Combined]
y-axis "Operations/Second" 0 --> 10000
bar [1000, 3000, 5000, 7000, 9500]
```
---
## Cross-Platform Integration
### Synchronization Network
```mermaid
graph TB
subgraph "Browser Instances"
B1[Browser 1]
B2[Browser 2]
B3[Browser 3]
end
subgraph "Server Infrastructure"
WS[WebSocket Server]
API[REST API Server]
S3[S3/Cloud Storage]
end
subgraph "Peer-to-Peer"
STUN[STUN Server]
SIGNAL[Signaling Server]
end
subgraph "External AI"
MCP[MCP Server]
AI[AI Models]
end
B1 <-->|WebSocket| WS
B2 <-->|WebSocket| WS
B3 <-->|WebSocket| WS
B1 <-.->|WebRTC| B2
B2 <-.->|WebRTC| B3
B1 <-.->|WebRTC| B3
WS <--> S3
API <--> S3
B1 -.->|Signaling| SIGNAL
B2 -.->|Signaling| SIGNAL
B3 -.->|Signaling| SIGNAL
SIGNAL -.-> STUN
WS <--> MCP
MCP <--> AI
style B1 fill:#e3f2fd
style B2 fill:#e3f2fd
style B3 fill:#e3f2fd
style WS fill:#e8f5e8
style API fill:#e8f5e8
style S3 fill:#fff3e0
style MCP fill:#f3e5f5
style AI fill:#ffebee
```
### Model Control Protocol (MCP) Integration
```mermaid
sequenceDiagram
participant AI as External AI Model
participant MCP as MCP Server
participant Adapter as Brainy MCP Adapter
participant Brainy as Brainy Database
AI->>MCP: Request data access
MCP->>Adapter: Forward request
Adapter->>Brainy: Query data
Brainy-->>Adapter: Return results
Adapter-->>MCP: Formatted response
MCP-->>AI: Data payload
AI->>MCP: Execute augmentation
MCP->>Adapter: Pipeline request
Adapter->>Brainy: Run augmentation
Brainy-->>Adapter: Processing result
Adapter-->>MCP: Tool response
MCP-->>AI: Execution result
```
---
## Data Flow Example
### Complete Processing Pipeline
```mermaid
flowchart TD
subgraph "Input Processing"
I1[Input: "Cats are independent pets"]
I2[Metadata: {noun: "Thing", category: "animal"}]
end
subgraph "Embedding Generation"
E1[TensorFlow.js Universal Sentence Encoder]
E2[Vector: [0.123, -0.456, 0.789, ...]]
end
subgraph "Storage & Indexing"
S1[Store in Multi-tier Cache]
S2[Add to HNSW Index]
S3[Persist to Storage Layer]
end
subgraph "Query Processing"
Q1[Query: "feline pets"]
Q2[Generate Query Vector]
Q3[HNSW Similarity Search]
Q4[Retrieve & Rank Results]
end
subgraph "Graph Operations"
G1[Add Relationship]
G2[catId --RelatedTo--> dogId]
G3[Store Verb Metadata]
end
I1 --> E1
I2 --> E1
E1 --> E2
E2 --> S1
S1 --> S2
S2 --> S3
Q1 --> Q2
Q2 --> Q3
Q3 --> Q4
E2 -.-> G1
G1 --> G2
G2 --> G3
style I1 fill:#e8f5e8
style E1 fill:#e3f2fd
style E2 fill:#f3e5f5
style S1 fill:#fff3e0
style Q1 fill:#e8f5e8
style Q4 fill:#ffebee
style G2 fill:#e0f2f1
```
### Result Example
```json
{
"results": [
{
"id": "noun-123",
"text": "Cats are independent pets",
"similarity": 0.89,
"metadata": {
"noun": "Thing",
"category": "animal"
}
}
],
"query": "feline pets",
"processingTime": "15ms",
"cacheHit": false
}
```
---
## Key Architecture Principles
### 🌐 **Environment Agnostic**
Automatically adapts to browser, Node.js, serverless, container, or server environments without code changes.
### 🧠 **Intelligent Storage**
Multi-tier caching with automatic storage selection optimizes for performance and persistence across platforms.
### 🔍 **Vector + Graph Unified**
Combines semantic vector search with graph relationships in a single, coherent data model.
### 🔧 **Extensible Pipeline**
Modular augmentation system allows custom processing, AI integration, and workflow automation.
### ⚡ **Performance Optimized**
GPU acceleration, multithreading, intelligent caching, and memory management deliver enterprise-grade performance.
### 🔄 **Scalable Synchronization**
WebSocket and WebRTC conduits enable real-time synchronization across instances and platforms.
### 🤖 **AI Integration Ready**
Built-in MCP protocol support allows external AI models to access Brainy data and utilize augmentation tools.
---
*Generated from Brainy v0.34.0 Architecture Documentation*

2
package-lock.json generated
View file

@ -39,7 +39,7 @@
"happy-dom": "^18.0.1",
"jsdom": "^26.1.0",
"node-fetch": "^3.3.2",
"puppeteer": "^22.5.0",
"puppeteer": "^22.15.0",
"rollup": "^4.13.0",
"rollup-plugin-terser": "^7.0.2",
"standard-version": "^9.5.0",

3
package.json
View file

@ -72,6 +72,7 @@
"test:specialized": "vitest run tests/specialized-scenarios.test.ts",
"test:performance": "vitest run tests/performance.test.ts",
"test:comprehensive": "npm run test:error-handling && npm run test:edge-cases && npm run test:storage && npm run test:environments && npm run test:specialized",
"_generate-pdf": "node dev/scripts/generate-architecture-pdf.js",
"_release": "standard-version",
"_release:patch": "standard-version --release-as patch",
"_release:minor": "standard-version --release-as minor",
@ -151,7 +152,7 @@
"happy-dom": "^18.0.1",
"jsdom": "^26.1.0",
"node-fetch": "^3.3.2",
"puppeteer": "^22.5.0",
"puppeteer": "^22.15.0",
"rollup": "^4.13.0",
"rollup-plugin-terser": "^7.0.2",
"standard-version": "^9.5.0",