brainy/docs/SCALING.md

🚀 Brainy Scaling Guide - Enterprise for Everyone

One Line Summary: Start with one node, scale to hundreds. Zero configuration required.

📖 Table of Contents

• Quick Start
• How It Works
• Storage Configurations
• Scaling Patterns
• Real World Examples

Quick Start

Single Node (Default)

import Brainy from '@soulcraft/brainy'
const brain = new Brainy()  // That's it!

Multi-Node (Auto-Discovery)

// Node 1
const brain = new Brainy()  // Starts as primary

// Node 2 (different server)
const brain = new Brainy()  // Auto-discovers Node 1, becomes replica!

That's literally all you need! Brainy handles everything else automatically.

How It Works

🎯 The Magic: Zero Configuration

Brainy uses intelligent defaults and auto-discovery to eliminate configuration:

1. First node starts → Becomes primary automatically
2. Second node starts → Discovers first node via UDP broadcast
3. Nodes negotiate → Elect leader, distribute shards
4. Data flows → Automatic replication and routing
5. Node fails → Automatic failover in <1 second

🔄 Automatic Node Discovery

// Three ways Brainy finds other nodes (auto-selected):

// 1. LOCAL NETWORK (Default)
// Uses UDP broadcast on port 7946
// Perfect for: On-premise, same VPC

// 2. CLOUD NATIVE (Auto-detected)
// Kubernetes: Uses k8s DNS service discovery
// AWS: Uses EC2 tags or Route53
// Azure: Uses Azure DNS

// 3. EXPLICIT (When needed)
const brain = new Brainy({
  peers: ['node1.example.com', 'node2.example.com']
})

📊 Data Distribution

When you add data, Brainy automatically:

brain.add({ name: "John" }, 'person')

// Behind the scenes:
// 1. Hash ID to determine shard (consistent hashing)
// 2. Find nodes responsible for this shard
// 3. Write to primary shard owner
// 4. Replicate to N backup nodes (default: 2)
// 5. Confirm write when majority acknowledge

Storage Configurations

🗂️ Storage Adapter Patterns

Brainy intelligently adapts to your storage setup:

Pattern 1: Separate Storage Per Node (Recommended)

// Node 1 - Own filesystem
const brain1 = new Brainy({
  storage: '/data/node1'  // or auto: './brainy-data'
})

// Node 2 - Own filesystem  
const brain2 = new Brainy({
  storage: '/data/node2'  // or auto: './brainy-data'
})

// ✅ BENEFITS:
// - No conflicts between nodes
// - Fast local reads
// - True horizontal scaling
// - Survives network partitions

Pattern 2: Separate S3 Buckets Per Node

// Node 1 - Own S3 bucket
const brain1 = new Brainy({
  storage: 's3://brainy-node-1'  // Auto-uses AWS credentials
})

// Node 2 - Own S3 bucket
const brain2 = new Brainy({
  storage: 's3://brainy-node-2'
})

// ✅ BENEFITS:
// - Infinite storage capacity
// - Geographic distribution
// - No local disk needed
// - Built-in durability

Pattern 3: Shared S3 Bucket (Coordinated)

// All nodes - Shared bucket with coordination
const brain = new Brainy({
  storage: 's3://shared-brainy-data',
  // Brainy automatically adds node-specific prefixes!
})

// What happens automatically:
// - Node 1 writes to: s3://shared-brainy-data/node-1/
// - Node 2 writes to: s3://shared-brainy-data/node-2/
// - Metadata in: s3://shared-brainy-data/_cluster/
// - Coordination via S3 conditional writes

// ✅ BENEFITS:
// - Single bucket to manage
// - Easy backup/restore
// - Cost effective
// - Automatic namespace isolation

Pattern 4: Mixed Storage (Hybrid)

// Hot data on local SSD, cold data in S3
const brain = new Brainy({
  storage: {
    hot: '/fast-ssd/brainy',     // Recent/frequent data
    cold: 's3://brainy-archive'  // Older data
  }
  // Brainy automatically promotes/demotes data!
})

🌍 Cloud Provider Auto-Detection

const brain = new Brainy({
  storage: 'cloud://brainy-data'  // Auto-detects provider!
})

// Automatically uses:
// - AWS: S3 + DynamoDB for metadata
// - Google Cloud: GCS + Firestore
// - Azure: Blob Storage + Cosmos DB
// - Cloudflare: R2 + D1
// - Vercel: Blob + KV

📝 Storage Coordination Rules

When multiple nodes share storage, Brainy automatically:

1. Namespace Isolation: Each node gets unique prefix
2. Lock-Free Writes: Uses atomic operations
3. Consistent Metadata: Coordinated via consensus
4. Conflict Resolution: Version vectors for conflicts
5. Garbage Collection: Automatic cleanup of old data

Scaling Patterns

📈 Progressive Scaling Journey

Stage 1: Prototype (1 node, memory)

const brain = new Brainy()  // Memory storage, single node
// Perfect for: Development, testing, <1000 items

Stage 2: Production (1 node, disk)

const brain = new Brainy({
  storage: './data'  // Persistent storage
})
// Perfect for: Small apps, <100K items

Stage 3: High Availability (2-3 nodes)

// Just start same code on multiple servers!
const brain = new Brainy({
  storage: './data'  // Each node's own storage
})
// Automatic: Leader election, replication, failover
// Perfect for: Critical apps, <1M items

Stage 4: Scale Out (N nodes)

// Same code, more servers!
const brain = new Brainy({
  storage: 's3://brainy-{{nodeId}}'  // Template auto-filled
})
// Automatic: Sharding, load balancing, geo-distribution
// Perfect for: Large apps, unlimited items

🎯 Common Scaling Scenarios

Scenario: Read-Heavy Application

// Brainy auto-detects read-heavy pattern and:
// 1. Increases cache size
// 2. Creates more read replicas
// 3. Routes reads to nearest node
// 4. Caches popular items on all nodes

const brain = new Brainy()  // No config needed!

Scenario: Multi-Tenant SaaS

// Brainy auto-detects tenant patterns and:
// 1. Shards by tenant ID
// 2. Isolates tenant data
// 3. Routes by tenant
// 4. Separate rate limits per tenant

const brain = new Brainy()  // Detects from your queries!

Scenario: Geographic Distribution

// Deploy nodes in different regions
// Brainy automatically:
// 1. Detects node locations (via latency)
// 2. Replicates data geographically
// 3. Routes to nearest node
// 4. Handles region failures

// US-East
const brain = new Brainy({ region: 'us-east' })  // Optional hint

// EU-West (auto-discovers US-East)
const brain = new Brainy({ region: 'eu-west' })

Real World Examples

Example 1: Blog Platform

// Day 1: Single server
const brain = new Brainy({
  storage: './blog-data'
})

// Month 6: Add redundancy (on second server)
const brain = new Brainy({
  storage: './blog-data'  // Different machine!
})
// Automatically syncs with first server

// Year 2: Global scale
// US Server
const brain = new Brainy({
  storage: 's3://blog-us/data'
})

// EU Server  
const brain = new Brainy({
  storage: 's3://blog-eu/data'
})

// Asia Server
const brain = new Brainy({
  storage: 's3://blog-asia/data'
})
// All automatically coordinate!

Example 2: E-Commerce Site

// Development
const brain = new Brainy()  // Memory storage

// Staging (Kubernetes)
const brain = new Brainy({
  storage: process.env.STORAGE_PATH  // Uses PVC
})
// Auto-discovers other pods via K8s DNS

// Production (AWS)
const brain = new Brainy({
  storage: 's3://shop-data',
  cache: 'elasticache://shop-cache'  // Optional
})
// Auto-scales with ECS/EKS

Example 3: Analytics Platform

// Ingestion nodes (write-optimized)
const brain = new Brainy({
  role: 'writer',  // Hint for optimization
  storage: '/fast-nvme/ingest'
})

// Query nodes (read-optimized)
const brain = new Brainy({
  role: 'reader',  // More cache, indexes
  storage: 's3://analytics-archive'
})

// Automatically coordinates between writers and readers!

🔧 Storage Adapter Specifics

Local Filesystem

{
  storage: './data'  // or absolute: '/var/lib/brainy'
  // Each node MUST have separate directory
  // Can be network mounted (NFS, EFS)
}

AWS S3

{
  storage: 's3://bucket-name/prefix'
  // Uses AWS SDK credentials (env, IAM role, etc)
  // Supports S3-compatible (MinIO, Ceph)
}

Cloudflare R2

{
  storage: 'r2://bucket-name'
  // Uses Wrangler or API tokens
  // Zero egress fees!
}

Google Cloud Storage

{
  storage: 'gs://bucket-name'
  // Uses Application Default Credentials
}

Azure Blob Storage

{
  storage: 'azure://container-name'
  // Uses DefaultAzureCredential
}

Mixed/Tiered

{
  storage: {
    hot: './local-cache',      // Fast SSD
    warm: 's3://regular-data',  // Standard storage
    cold: 's3://glacier-archive' // Cheap archive
  }
  // Automatic tiering based on access patterns
}

🎭 Advanced Patterns

Pattern: Blue-Green Deployment

// Blue cluster (current)
const brain = new Brainy({
  cluster: 'blue',
  storage: 's3://prod-blue'
})

// Green cluster (new version)
const brain = new Brainy({
  cluster: 'green',
  storage: 's3://prod-green',
  syncFrom: 'blue'  // Real-time sync during migration
})

Pattern: Federation

// Region 1 Cluster
const brain1 = new Brainy({
  federation: 'global',
  region: 'us-east',
  storage: 's3://us-east-data'
})

// Region 2 Cluster
const brain2 = new Brainy({
  federation: 'global',
  region: 'eu-west',
  storage: 's3://eu-west-data'
})
// Clusters coordinate for global queries!

Pattern: Edge Computing

// Edge nodes (in CDN POPs)
const brain = new Brainy({
  mode: 'edge',
  storage: 'memory',  // RAM only
  upstream: 'https://main-cluster.example.com'
})
// Caches frequently accessed data at edge

📊 Monitoring & Observability

Brainy automatically exposes metrics:

const metrics = brain.getMetrics()
// {
//   nodes: { total: 5, healthy: 5 },
//   shards: { total: 20, local: 4 },
//   replication: { factor: 2, lag: 45 },
//   operations: { reads: 10000, writes: 1000 },
//   storage: { used: '45GB', available: '955GB' }
// }

🚨 Troubleshooting

Issue: Nodes don't discover each other

// Solution 1: Check network allows UDP 7946
// Solution 2: Use explicit peers
const brain = new Brainy({
  peers: ['10.0.0.1:7946', '10.0.0.2:7946']
})

Issue: Storage conflicts

// Ensure each node has unique storage path
// ❌ WRONG: All nodes use './data'
// ✅ RIGHT: Node1: './data1', Node2: './data2'
// ✅ RIGHT: Use {{nodeId}} template

Issue: Slow performance

// Brainy auto-tunes, but you can hint:
const brain = new Brainy({
  profile: 'read-heavy'  // or 'write-heavy', 'balanced'
})

🎯 Best Practices

1. Let Brainy Auto-Configure: Don't over-configure
2. Separate Storage Per Node: Avoids conflicts
3. Use S3 for Large Scale: Infinite capacity
4. Start Simple: Single node → Scale when needed
5. Monitor Metrics: Watch for bottlenecks
6. Trust Auto-Scaling: It learns your patterns

🚀 Summary

• Zero Config: Just new Brainy() at any scale
• Auto-Discovery: Nodes find each other
• Smart Storage: Adapts to any backend
• Progressive Scaling: 1 → 100 nodes seamlessly
• Self-Tuning: Learns and optimizes
• No DevOps: It just works!

This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!

---

Questions? Issues? Visit github.com/soullabs/brainy