docs: add comprehensive scaling and storage architecture documentation
- Add user-friendly SCALING.md explaining Enterprise for Everyone - Document zero-config philosophy and auto-discovery - Explain storage adapter patterns and coordination strategies - Add real-world examples and best practices - Create technical deep-dive on distributed storage architecture - Document how different storage backends work together - Explain coordination strategies for shared vs isolated storage
This commit is contained in:
parent
8ead063df8
commit
f65455fb22
2 changed files with 1052 additions and 0 deletions
502
docs/SCALING.md
Normal file
502
docs/SCALING.md
Normal file
|
|
@ -0,0 +1,502 @@
|
|||
# 🚀 Brainy Scaling Guide - Enterprise for Everyone
|
||||
|
||||
> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required.
|
||||
|
||||
## 📖 Table of Contents
|
||||
- [Quick Start](#quick-start)
|
||||
- [How It Works](#how-it-works)
|
||||
- [Storage Configurations](#storage-configurations)
|
||||
- [Scaling Patterns](#scaling-patterns)
|
||||
- [Real World Examples](#real-world-examples)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Single Node (Default)
|
||||
```typescript
|
||||
import Brainy from '@soulcraft/brainy'
|
||||
const brain = new Brainy() // That's it!
|
||||
```
|
||||
|
||||
### Multi-Node (Auto-Discovery)
|
||||
```typescript
|
||||
// Node 1
|
||||
const brain = new Brainy() // Starts as primary
|
||||
|
||||
// Node 2 (different server)
|
||||
const brain = new Brainy() // Auto-discovers Node 1, becomes replica!
|
||||
```
|
||||
|
||||
**That's literally all you need!** Brainy handles everything else automatically.
|
||||
|
||||
## How It Works
|
||||
|
||||
### 🎯 The Magic: Zero Configuration
|
||||
|
||||
Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration:
|
||||
|
||||
1. **First node starts** → Becomes primary automatically
|
||||
2. **Second node starts** → Discovers first node via UDP broadcast
|
||||
3. **Nodes negotiate** → Elect leader, distribute shards
|
||||
4. **Data flows** → Automatic replication and routing
|
||||
5. **Node fails** → Automatic failover in <1 second
|
||||
|
||||
### 🔄 Automatic Node Discovery
|
||||
|
||||
```typescript
|
||||
// Three ways Brainy finds other nodes (auto-selected):
|
||||
|
||||
// 1. LOCAL NETWORK (Default)
|
||||
// Uses UDP broadcast on port 7946
|
||||
// Perfect for: On-premise, same VPC
|
||||
|
||||
// 2. CLOUD NATIVE (Auto-detected)
|
||||
// Kubernetes: Uses k8s DNS service discovery
|
||||
// AWS: Uses EC2 tags or Route53
|
||||
// Azure: Uses Azure DNS
|
||||
|
||||
// 3. EXPLICIT (When needed)
|
||||
const brain = new Brainy({
|
||||
peers: ['node1.example.com', 'node2.example.com']
|
||||
})
|
||||
```
|
||||
|
||||
### 📊 Data Distribution
|
||||
|
||||
When you add data, Brainy automatically:
|
||||
|
||||
```typescript
|
||||
brain.add({ name: "John" }, 'person')
|
||||
|
||||
// Behind the scenes:
|
||||
// 1. Hash ID to determine shard (consistent hashing)
|
||||
// 2. Find nodes responsible for this shard
|
||||
// 3. Write to primary shard owner
|
||||
// 4. Replicate to N backup nodes (default: 2)
|
||||
// 5. Confirm write when majority acknowledge
|
||||
```
|
||||
|
||||
## Storage Configurations
|
||||
|
||||
### 🗂️ Storage Adapter Patterns
|
||||
|
||||
Brainy intelligently adapts to your storage setup:
|
||||
|
||||
#### Pattern 1: Separate Storage Per Node (Recommended)
|
||||
|
||||
```typescript
|
||||
// Node 1 - Own filesystem
|
||||
const brain1 = new Brainy({
|
||||
storage: '/data/node1' // or auto: './brainy-data'
|
||||
})
|
||||
|
||||
// Node 2 - Own filesystem
|
||||
const brain2 = new Brainy({
|
||||
storage: '/data/node2' // or auto: './brainy-data'
|
||||
})
|
||||
|
||||
// ✅ BENEFITS:
|
||||
// - No conflicts between nodes
|
||||
// - Fast local reads
|
||||
// - True horizontal scaling
|
||||
// - Survives network partitions
|
||||
```
|
||||
|
||||
#### Pattern 2: Separate S3 Buckets Per Node
|
||||
|
||||
```typescript
|
||||
// Node 1 - Own S3 bucket
|
||||
const brain1 = new Brainy({
|
||||
storage: 's3://brainy-node-1' // Auto-uses AWS credentials
|
||||
})
|
||||
|
||||
// Node 2 - Own S3 bucket
|
||||
const brain2 = new Brainy({
|
||||
storage: 's3://brainy-node-2'
|
||||
})
|
||||
|
||||
// ✅ BENEFITS:
|
||||
// - Infinite storage capacity
|
||||
// - Geographic distribution
|
||||
// - No local disk needed
|
||||
// - Built-in durability
|
||||
```
|
||||
|
||||
#### Pattern 3: Shared S3 Bucket (Coordinated)
|
||||
|
||||
```typescript
|
||||
// All nodes - Shared bucket with coordination
|
||||
const brain = new Brainy({
|
||||
storage: 's3://shared-brainy-data',
|
||||
// Brainy automatically adds node-specific prefixes!
|
||||
})
|
||||
|
||||
// What happens automatically:
|
||||
// - Node 1 writes to: s3://shared-brainy-data/node-1/
|
||||
// - Node 2 writes to: s3://shared-brainy-data/node-2/
|
||||
// - Metadata in: s3://shared-brainy-data/_cluster/
|
||||
// - Coordination via S3 conditional writes
|
||||
|
||||
// ✅ BENEFITS:
|
||||
// - Single bucket to manage
|
||||
// - Easy backup/restore
|
||||
// - Cost effective
|
||||
// - Automatic namespace isolation
|
||||
```
|
||||
|
||||
#### Pattern 4: Mixed Storage (Hybrid)
|
||||
|
||||
```typescript
|
||||
// Hot data on local SSD, cold data in S3
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
hot: '/fast-ssd/brainy', // Recent/frequent data
|
||||
cold: 's3://brainy-archive' // Older data
|
||||
}
|
||||
// Brainy automatically promotes/demotes data!
|
||||
})
|
||||
```
|
||||
|
||||
### 🌍 Cloud Provider Auto-Detection
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: 'cloud://brainy-data' // Auto-detects provider!
|
||||
})
|
||||
|
||||
// Automatically uses:
|
||||
// - AWS: S3 + DynamoDB for metadata
|
||||
// - Google Cloud: GCS + Firestore
|
||||
// - Azure: Blob Storage + Cosmos DB
|
||||
// - Cloudflare: R2 + D1
|
||||
// - Vercel: Blob + KV
|
||||
```
|
||||
|
||||
### 📝 Storage Coordination Rules
|
||||
|
||||
When multiple nodes share storage, Brainy automatically:
|
||||
|
||||
1. **Namespace Isolation**: Each node gets unique prefix
|
||||
2. **Lock-Free Writes**: Uses atomic operations
|
||||
3. **Consistent Metadata**: Coordinated via consensus
|
||||
4. **Conflict Resolution**: Version vectors for conflicts
|
||||
5. **Garbage Collection**: Automatic cleanup of old data
|
||||
|
||||
## Scaling Patterns
|
||||
|
||||
### 📈 Progressive Scaling Journey
|
||||
|
||||
#### Stage 1: Prototype (1 node, memory)
|
||||
```typescript
|
||||
const brain = new Brainy() // Memory storage, single node
|
||||
// Perfect for: Development, testing, <1000 items
|
||||
```
|
||||
|
||||
#### Stage 2: Production (1 node, disk)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: './data' // Persistent storage
|
||||
})
|
||||
// Perfect for: Small apps, <100K items
|
||||
```
|
||||
|
||||
#### Stage 3: High Availability (2-3 nodes)
|
||||
```typescript
|
||||
// Just start same code on multiple servers!
|
||||
const brain = new Brainy({
|
||||
storage: './data' // Each node's own storage
|
||||
})
|
||||
// Automatic: Leader election, replication, failover
|
||||
// Perfect for: Critical apps, <1M items
|
||||
```
|
||||
|
||||
#### Stage 4: Scale Out (N nodes)
|
||||
```typescript
|
||||
// Same code, more servers!
|
||||
const brain = new Brainy({
|
||||
storage: 's3://brainy-{{nodeId}}' // Template auto-filled
|
||||
})
|
||||
// Automatic: Sharding, load balancing, geo-distribution
|
||||
// Perfect for: Large apps, unlimited items
|
||||
```
|
||||
|
||||
### 🎯 Common Scaling Scenarios
|
||||
|
||||
#### Scenario: Read-Heavy Application
|
||||
```typescript
|
||||
// Brainy auto-detects read-heavy pattern and:
|
||||
// 1. Increases cache size
|
||||
// 2. Creates more read replicas
|
||||
// 3. Routes reads to nearest node
|
||||
// 4. Caches popular items on all nodes
|
||||
|
||||
const brain = new Brainy() // No config needed!
|
||||
```
|
||||
|
||||
#### Scenario: Multi-Tenant SaaS
|
||||
```typescript
|
||||
// Brainy auto-detects tenant patterns and:
|
||||
// 1. Shards by tenant ID
|
||||
// 2. Isolates tenant data
|
||||
// 3. Routes by tenant
|
||||
// 4. Separate rate limits per tenant
|
||||
|
||||
const brain = new Brainy() // Detects from your queries!
|
||||
```
|
||||
|
||||
#### Scenario: Geographic Distribution
|
||||
```typescript
|
||||
// Deploy nodes in different regions
|
||||
// Brainy automatically:
|
||||
// 1. Detects node locations (via latency)
|
||||
// 2. Replicates data geographically
|
||||
// 3. Routes to nearest node
|
||||
// 4. Handles region failures
|
||||
|
||||
// US-East
|
||||
const brain = new Brainy({ region: 'us-east' }) // Optional hint
|
||||
|
||||
// EU-West (auto-discovers US-East)
|
||||
const brain = new Brainy({ region: 'eu-west' })
|
||||
```
|
||||
|
||||
## Real World Examples
|
||||
|
||||
### Example 1: Blog Platform
|
||||
|
||||
```typescript
|
||||
// Day 1: Single server
|
||||
const brain = new Brainy({
|
||||
storage: './blog-data'
|
||||
})
|
||||
|
||||
// Month 6: Add redundancy (on second server)
|
||||
const brain = new Brainy({
|
||||
storage: './blog-data' // Different machine!
|
||||
})
|
||||
// Automatically syncs with first server
|
||||
|
||||
// Year 2: Global scale
|
||||
// US Server
|
||||
const brain = new Brainy({
|
||||
storage: 's3://blog-us/data'
|
||||
})
|
||||
|
||||
// EU Server
|
||||
const brain = new Brainy({
|
||||
storage: 's3://blog-eu/data'
|
||||
})
|
||||
|
||||
// Asia Server
|
||||
const brain = new Brainy({
|
||||
storage: 's3://blog-asia/data'
|
||||
})
|
||||
// All automatically coordinate!
|
||||
```
|
||||
|
||||
### Example 2: E-Commerce Site
|
||||
|
||||
```typescript
|
||||
// Development
|
||||
const brain = new Brainy() // Memory storage
|
||||
|
||||
// Staging (Kubernetes)
|
||||
const brain = new Brainy({
|
||||
storage: process.env.STORAGE_PATH // Uses PVC
|
||||
})
|
||||
// Auto-discovers other pods via K8s DNS
|
||||
|
||||
// Production (AWS)
|
||||
const brain = new Brainy({
|
||||
storage: 's3://shop-data',
|
||||
cache: 'elasticache://shop-cache' // Optional
|
||||
})
|
||||
// Auto-scales with ECS/EKS
|
||||
```
|
||||
|
||||
### Example 3: Analytics Platform
|
||||
|
||||
```typescript
|
||||
// Ingestion nodes (write-optimized)
|
||||
const brain = new Brainy({
|
||||
role: 'writer', // Hint for optimization
|
||||
storage: '/fast-nvme/ingest'
|
||||
})
|
||||
|
||||
// Query nodes (read-optimized)
|
||||
const brain = new Brainy({
|
||||
role: 'reader', // More cache, indexes
|
||||
storage: 's3://analytics-archive'
|
||||
})
|
||||
|
||||
// Automatically coordinates between writers and readers!
|
||||
```
|
||||
|
||||
## 🔧 Storage Adapter Specifics
|
||||
|
||||
### Local Filesystem
|
||||
```typescript
|
||||
{
|
||||
storage: './data' // or absolute: '/var/lib/brainy'
|
||||
// Each node MUST have separate directory
|
||||
// Can be network mounted (NFS, EFS)
|
||||
}
|
||||
```
|
||||
|
||||
### AWS S3
|
||||
```typescript
|
||||
{
|
||||
storage: 's3://bucket-name/prefix'
|
||||
// Uses AWS SDK credentials (env, IAM role, etc)
|
||||
// Supports S3-compatible (MinIO, Ceph)
|
||||
}
|
||||
```
|
||||
|
||||
### Cloudflare R2
|
||||
```typescript
|
||||
{
|
||||
storage: 'r2://bucket-name'
|
||||
// Uses Wrangler or API tokens
|
||||
// Zero egress fees!
|
||||
}
|
||||
```
|
||||
|
||||
### Google Cloud Storage
|
||||
```typescript
|
||||
{
|
||||
storage: 'gs://bucket-name'
|
||||
// Uses Application Default Credentials
|
||||
}
|
||||
```
|
||||
|
||||
### Azure Blob Storage
|
||||
```typescript
|
||||
{
|
||||
storage: 'azure://container-name'
|
||||
// Uses DefaultAzureCredential
|
||||
}
|
||||
```
|
||||
|
||||
### Mixed/Tiered
|
||||
```typescript
|
||||
{
|
||||
storage: {
|
||||
hot: './local-cache', // Fast SSD
|
||||
warm: 's3://regular-data', // Standard storage
|
||||
cold: 's3://glacier-archive' // Cheap archive
|
||||
}
|
||||
// Automatic tiering based on access patterns
|
||||
}
|
||||
```
|
||||
|
||||
## 🎭 Advanced Patterns
|
||||
|
||||
### Pattern: Blue-Green Deployment
|
||||
```typescript
|
||||
// Blue cluster (current)
|
||||
const brain = new Brainy({
|
||||
cluster: 'blue',
|
||||
storage: 's3://prod-blue'
|
||||
})
|
||||
|
||||
// Green cluster (new version)
|
||||
const brain = new Brainy({
|
||||
cluster: 'green',
|
||||
storage: 's3://prod-green',
|
||||
syncFrom: 'blue' // Real-time sync during migration
|
||||
})
|
||||
```
|
||||
|
||||
### Pattern: Federation
|
||||
```typescript
|
||||
// Region 1 Cluster
|
||||
const brain1 = new Brainy({
|
||||
federation: 'global',
|
||||
region: 'us-east',
|
||||
storage: 's3://us-east-data'
|
||||
})
|
||||
|
||||
// Region 2 Cluster
|
||||
const brain2 = new Brainy({
|
||||
federation: 'global',
|
||||
region: 'eu-west',
|
||||
storage: 's3://eu-west-data'
|
||||
})
|
||||
// Clusters coordinate for global queries!
|
||||
```
|
||||
|
||||
### Pattern: Edge Computing
|
||||
```typescript
|
||||
// Edge nodes (in CDN POPs)
|
||||
const brain = new Brainy({
|
||||
mode: 'edge',
|
||||
storage: 'memory', // RAM only
|
||||
upstream: 'https://main-cluster.example.com'
|
||||
})
|
||||
// Caches frequently accessed data at edge
|
||||
```
|
||||
|
||||
## 📊 Monitoring & Observability
|
||||
|
||||
Brainy automatically exposes metrics:
|
||||
|
||||
```typescript
|
||||
const metrics = brain.getMetrics()
|
||||
// {
|
||||
// nodes: { total: 5, healthy: 5 },
|
||||
// shards: { total: 20, local: 4 },
|
||||
// replication: { factor: 2, lag: 45 },
|
||||
// operations: { reads: 10000, writes: 1000 },
|
||||
// storage: { used: '45GB', available: '955GB' }
|
||||
// }
|
||||
```
|
||||
|
||||
## 🚨 Troubleshooting
|
||||
|
||||
### Issue: Nodes don't discover each other
|
||||
```typescript
|
||||
// Solution 1: Check network allows UDP 7946
|
||||
// Solution 2: Use explicit peers
|
||||
const brain = new Brainy({
|
||||
peers: ['10.0.0.1:7946', '10.0.0.2:7946']
|
||||
})
|
||||
```
|
||||
|
||||
### Issue: Storage conflicts
|
||||
```typescript
|
||||
// Ensure each node has unique storage path
|
||||
// ❌ WRONG: All nodes use './data'
|
||||
// ✅ RIGHT: Node1: './data1', Node2: './data2'
|
||||
// ✅ RIGHT: Use {{nodeId}} template
|
||||
```
|
||||
|
||||
### Issue: Slow performance
|
||||
```typescript
|
||||
// Brainy auto-tunes, but you can hint:
|
||||
const brain = new Brainy({
|
||||
profile: 'read-heavy' // or 'write-heavy', 'balanced'
|
||||
})
|
||||
```
|
||||
|
||||
## 🎯 Best Practices
|
||||
|
||||
1. **Let Brainy Auto-Configure**: Don't over-configure
|
||||
2. **Separate Storage Per Node**: Avoids conflicts
|
||||
3. **Use S3 for Large Scale**: Infinite capacity
|
||||
4. **Start Simple**: Single node → Scale when needed
|
||||
5. **Monitor Metrics**: Watch for bottlenecks
|
||||
6. **Trust Auto-Scaling**: It learns your patterns
|
||||
|
||||
## 🚀 Summary
|
||||
|
||||
- **Zero Config**: Just `new Brainy()` at any scale
|
||||
- **Auto-Discovery**: Nodes find each other
|
||||
- **Smart Storage**: Adapts to any backend
|
||||
- **Progressive Scaling**: 1 → 100 nodes seamlessly
|
||||
- **Self-Tuning**: Learns and optimizes
|
||||
- **No DevOps**: It just works!
|
||||
|
||||
**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!**
|
||||
|
||||
---
|
||||
|
||||
*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)*
|
||||
Loading…
Add table
Add a link
Reference in a new issue