soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

docs: add comprehensive scaling and storage architecture documentation

Browse source 
- Add user-friendly SCALING.md explaining Enterprise for Everyone
- Document zero-config philosophy and auto-discovery
- Explain storage adapter patterns and coordination strategies
- Add real-world examples and best practices
- Create technical deep-dive on distributed storage architecture
- Document how different storage backends work together
- Explain coordination strategies for shared vs isolated storage
This commit is contained in:
David Snelling 2025-09-08 14:49:25 -07:00
parent 8ead063df8
commit f65455fb22
2 changed files with 1052 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

502
docs/SCALING.md Normal file
View file

@ -0,0 +1,502 @@
# 🚀 Brainy Scaling Guide - Enterprise for Everyone
> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required.
## 📖 Table of Contents
- [Quick Start](#quick-start)
- [How It Works](#how-it-works)
- [Storage Configurations](#storage-configurations)
- [Scaling Patterns](#scaling-patterns)
- [Real World Examples](#real-world-examples)
## Quick Start
### Single Node (Default)
```typescript
import Brainy from '@soulcraft/brainy'
const brain = new Brainy() // That's it!
```
### Multi-Node (Auto-Discovery)
```typescript
// Node 1
const brain = new Brainy() // Starts as primary
// Node 2 (different server)
const brain = new Brainy() // Auto-discovers Node 1, becomes replica!
```
**That's literally all you need!** Brainy handles everything else automatically.
## How It Works
### 🎯 The Magic: Zero Configuration
Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration:
1. **First node starts** → Becomes primary automatically
2. **Second node starts** → Discovers first node via UDP broadcast
3. **Nodes negotiate** → Elect leader, distribute shards
4. **Data flows** → Automatic replication and routing
5. **Node fails** → Automatic failover in <1 second
### 🔄 Automatic Node Discovery
```typescript
// Three ways Brainy finds other nodes (auto-selected):
// 1. LOCAL NETWORK (Default)
// Uses UDP broadcast on port 7946
// Perfect for: On-premise, same VPC
// 2. CLOUD NATIVE (Auto-detected)
// Kubernetes: Uses k8s DNS service discovery
// AWS: Uses EC2 tags or Route53
// Azure: Uses Azure DNS
// 3. EXPLICIT (When needed)
const brain = new Brainy({
peers: ['node1.example.com', 'node2.example.com']
})
```
### 📊 Data Distribution
When you add data, Brainy automatically:
```typescript
brain.add({ name: "John" }, 'person')
// Behind the scenes:
// 1. Hash ID to determine shard (consistent hashing)
// 2. Find nodes responsible for this shard
// 3. Write to primary shard owner
// 4. Replicate to N backup nodes (default: 2)
// 5. Confirm write when majority acknowledge
```
## Storage Configurations
### 🗂️ Storage Adapter Patterns
Brainy intelligently adapts to your storage setup:
#### Pattern 1: Separate Storage Per Node (Recommended)
```typescript
// Node 1 - Own filesystem
const brain1 = new Brainy({
storage: '/data/node1' // or auto: './brainy-data'
})
// Node 2 - Own filesystem
const brain2 = new Brainy({
storage: '/data/node2' // or auto: './brainy-data'
})
// ✅ BENEFITS:
// - No conflicts between nodes
// - Fast local reads
// - True horizontal scaling
// - Survives network partitions
```
#### Pattern 2: Separate S3 Buckets Per Node
```typescript
// Node 1 - Own S3 bucket
const brain1 = new Brainy({
storage: 's3://brainy-node-1' // Auto-uses AWS credentials
})
// Node 2 - Own S3 bucket
const brain2 = new Brainy({
storage: 's3://brainy-node-2'
})
// ✅ BENEFITS:
// - Infinite storage capacity
// - Geographic distribution
// - No local disk needed
// - Built-in durability
```
#### Pattern 3: Shared S3 Bucket (Coordinated)
```typescript
// All nodes - Shared bucket with coordination
const brain = new Brainy({
storage: 's3://shared-brainy-data',
// Brainy automatically adds node-specific prefixes!
})
// What happens automatically:
// - Node 1 writes to: s3://shared-brainy-data/node-1/
// - Node 2 writes to: s3://shared-brainy-data/node-2/
// - Metadata in: s3://shared-brainy-data/_cluster/
// - Coordination via S3 conditional writes
// ✅ BENEFITS:
// - Single bucket to manage
// - Easy backup/restore
// - Cost effective
// - Automatic namespace isolation
```
#### Pattern 4: Mixed Storage (Hybrid)
```typescript
// Hot data on local SSD, cold data in S3
const brain = new Brainy({
storage: {
hot: '/fast-ssd/brainy', // Recent/frequent data
cold: 's3://brainy-archive' // Older data
}
// Brainy automatically promotes/demotes data!
})
```
### 🌍 Cloud Provider Auto-Detection
```typescript
const brain = new Brainy({
storage: 'cloud://brainy-data' // Auto-detects provider!
})
// Automatically uses:
// - AWS: S3 + DynamoDB for metadata
// - Google Cloud: GCS + Firestore
// - Azure: Blob Storage + Cosmos DB
// - Cloudflare: R2 + D1
// - Vercel: Blob + KV
```
### 📝 Storage Coordination Rules
When multiple nodes share storage, Brainy automatically:
1. **Namespace Isolation**: Each node gets unique prefix
2. **Lock-Free Writes**: Uses atomic operations
3. **Consistent Metadata**: Coordinated via consensus
4. **Conflict Resolution**: Version vectors for conflicts
5. **Garbage Collection**: Automatic cleanup of old data
## Scaling Patterns
### 📈 Progressive Scaling Journey
#### Stage 1: Prototype (1 node, memory)
```typescript
const brain = new Brainy() // Memory storage, single node
// Perfect for: Development, testing, <1000 items
```
#### Stage 2: Production (1 node, disk)
```typescript
const brain = new Brainy({
storage: './data' // Persistent storage
})
// Perfect for: Small apps, <100K items
```
#### Stage 3: High Availability (2-3 nodes)
```typescript
// Just start same code on multiple servers!
const brain = new Brainy({
storage: './data' // Each node's own storage
})
// Automatic: Leader election, replication, failover
// Perfect for: Critical apps, <1M items
```
#### Stage 4: Scale Out (N nodes)
```typescript
// Same code, more servers!
const brain = new Brainy({
storage: 's3://brainy-{{nodeId}}' // Template auto-filled
})
// Automatic: Sharding, load balancing, geo-distribution
// Perfect for: Large apps, unlimited items
```
### 🎯 Common Scaling Scenarios
#### Scenario: Read-Heavy Application
```typescript
// Brainy auto-detects read-heavy pattern and:
// 1. Increases cache size
// 2. Creates more read replicas
// 3. Routes reads to nearest node
// 4. Caches popular items on all nodes
const brain = new Brainy() // No config needed!
```
#### Scenario: Multi-Tenant SaaS
```typescript
// Brainy auto-detects tenant patterns and:
// 1. Shards by tenant ID
// 2. Isolates tenant data
// 3. Routes by tenant
// 4. Separate rate limits per tenant
const brain = new Brainy() // Detects from your queries!
```
#### Scenario: Geographic Distribution
```typescript
// Deploy nodes in different regions
// Brainy automatically:
// 1. Detects node locations (via latency)
// 2. Replicates data geographically
// 3. Routes to nearest node
// 4. Handles region failures
// US-East
const brain = new Brainy({ region: 'us-east' }) // Optional hint
// EU-West (auto-discovers US-East)
const brain = new Brainy({ region: 'eu-west' })
```
## Real World Examples
### Example 1: Blog Platform
```typescript
// Day 1: Single server
const brain = new Brainy({
storage: './blog-data'
})
// Month 6: Add redundancy (on second server)
const brain = new Brainy({
storage: './blog-data' // Different machine!
})
// Automatically syncs with first server
// Year 2: Global scale
// US Server
const brain = new Brainy({
storage: 's3://blog-us/data'
})
// EU Server
const brain = new Brainy({
storage: 's3://blog-eu/data'
})
// Asia Server
const brain = new Brainy({
storage: 's3://blog-asia/data'
})
// All automatically coordinate!
```
### Example 2: E-Commerce Site
```typescript
// Development
const brain = new Brainy() // Memory storage
// Staging (Kubernetes)
const brain = new Brainy({
storage: process.env.STORAGE_PATH // Uses PVC
})
// Auto-discovers other pods via K8s DNS
// Production (AWS)
const brain = new Brainy({
storage: 's3://shop-data',
cache: 'elasticache://shop-cache' // Optional
})
// Auto-scales with ECS/EKS
```
### Example 3: Analytics Platform
```typescript
// Ingestion nodes (write-optimized)
const brain = new Brainy({
role: 'writer', // Hint for optimization
storage: '/fast-nvme/ingest'
})
// Query nodes (read-optimized)
const brain = new Brainy({
role: 'reader', // More cache, indexes
storage: 's3://analytics-archive'
})
// Automatically coordinates between writers and readers!
```
## 🔧 Storage Adapter Specifics
### Local Filesystem
```typescript
{
storage: './data' // or absolute: '/var/lib/brainy'
// Each node MUST have separate directory
// Can be network mounted (NFS, EFS)
}
```
### AWS S3
```typescript
{
storage: 's3://bucket-name/prefix'
// Uses AWS SDK credentials (env, IAM role, etc)
// Supports S3-compatible (MinIO, Ceph)
}
```
### Cloudflare R2
```typescript
{
storage: 'r2://bucket-name'
// Uses Wrangler or API tokens
// Zero egress fees!
}
```
### Google Cloud Storage
```typescript
{
storage: 'gs://bucket-name'
// Uses Application Default Credentials
}
```
### Azure Blob Storage
```typescript
{
storage: 'azure://container-name'
// Uses DefaultAzureCredential
}
```
### Mixed/Tiered
```typescript
{
storage: {
hot: './local-cache', // Fast SSD
warm: 's3://regular-data', // Standard storage
cold: 's3://glacier-archive' // Cheap archive
}
// Automatic tiering based on access patterns
}
```
## 🎭 Advanced Patterns
### Pattern: Blue-Green Deployment
```typescript
// Blue cluster (current)
const brain = new Brainy({
cluster: 'blue',
storage: 's3://prod-blue'
})
// Green cluster (new version)
const brain = new Brainy({
cluster: 'green',
storage: 's3://prod-green',
syncFrom: 'blue' // Real-time sync during migration
})
```
### Pattern: Federation
```typescript
// Region 1 Cluster
const brain1 = new Brainy({
federation: 'global',
region: 'us-east',
storage: 's3://us-east-data'
})
// Region 2 Cluster
const brain2 = new Brainy({
federation: 'global',
region: 'eu-west',
storage: 's3://eu-west-data'
})
// Clusters coordinate for global queries!
```
### Pattern: Edge Computing
```typescript
// Edge nodes (in CDN POPs)
const brain = new Brainy({
mode: 'edge',
storage: 'memory', // RAM only
upstream: 'https://main-cluster.example.com'
})
// Caches frequently accessed data at edge
```
## 📊 Monitoring & Observability
Brainy automatically exposes metrics:
```typescript
const metrics = brain.getMetrics()
// {
// nodes: { total: 5, healthy: 5 },
// shards: { total: 20, local: 4 },
// replication: { factor: 2, lag: 45 },
// operations: { reads: 10000, writes: 1000 },
// storage: { used: '45GB', available: '955GB' }
// }
```
## 🚨 Troubleshooting
### Issue: Nodes don't discover each other
```typescript
// Solution 1: Check network allows UDP 7946
// Solution 2: Use explicit peers
const brain = new Brainy({
peers: ['10.0.0.1:7946', '10.0.0.2:7946']
})
```
### Issue: Storage conflicts
```typescript
// Ensure each node has unique storage path
// ❌ WRONG: All nodes use './data'
// ✅ RIGHT: Node1: './data1', Node2: './data2'
// ✅ RIGHT: Use {{nodeId}} template
```
### Issue: Slow performance
```typescript
// Brainy auto-tunes, but you can hint:
const brain = new Brainy({
profile: 'read-heavy' // or 'write-heavy', 'balanced'
})
```
## 🎯 Best Practices
1. **Let Brainy Auto-Configure**: Don't over-configure
2. **Separate Storage Per Node**: Avoids conflicts
3. **Use S3 for Large Scale**: Infinite capacity
4. **Start Simple**: Single node → Scale when needed
5. **Monitor Metrics**: Watch for bottlenecks
6. **Trust Auto-Scaling**: It learns your patterns
## 🚀 Summary
- **Zero Config**: Just `new Brainy()` at any scale
- **Auto-Discovery**: Nodes find each other
- **Smart Storage**: Adapts to any backend
- **Progressive Scaling**: 1 → 100 nodes seamlessly
- **Self-Tuning**: Learns and optimizes
- **No DevOps**: It just works!
**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!**
---
*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)*

550
docs/architecture/distributed-storage.md Normal file
View file

@ -0,0 +1,550 @@
# 🏗️ Distributed Storage Architecture
> **Technical deep-dive**: How Brainy coordinates storage across multiple nodes and adapters
## Storage Adapter Layer
### Base Storage Interface
Every storage adapter implements this interface:
```typescript
interface StorageAdapter {
// Basic operations
get(key: string): Promise<any>
set(key: string, value: any): Promise<void>
delete(key: string): Promise<void>
// Batch operations
getBatch(keys: string[]): Promise<Map<string, any>>
setBatch(items: Map<string, any>): Promise<void>
// Atomic operations (for coordination)
compareAndSwap(key: string, oldVal: any, newVal: any): Promise<boolean>
increment(key: string, delta: number): Promise<number>
// Namespace support
withNamespace(namespace: string): StorageAdapter
}
```
### Storage Coordination Strategies
#### Strategy 1: Isolated Storage (Default)
Each node has completely separate storage:
```
Node-1 → Local FS: /data/node1/
└── shards/
├── shard-001/
├── shard-045/
└── shard-127/
Node-2 → Local FS: /data/node2/
└── shards/
├── shard-023/
├── shard-067/
└── shard-089/
```
**Coordination**: Via network messages only
- Shard ownership tracked in distributed consensus
- Data transfer via direct node-to-node communication
- No storage-level conflicts possible
#### Strategy 2: Shared Storage with Namespacing
Multiple nodes share storage but use namespaces:
```
S3 Bucket: brainy-cluster/
├── node-abc123/
│ ├── shards/
│ └── wal/
├── node-def456/
│ ├── shards/
│ └── wal/
└── _cluster/
├── topology.json
├── shard-map.json
└── elections/
```
**Coordination**: Via storage-level atomic operations
- Each node owns its namespace
- Cluster metadata in shared `_cluster/` namespace
- Atomic operations for leader election
- Conditional writes prevent conflicts
#### Strategy 3: Shared Storage with Fine-Grained Locking
Advanced mode for full shared storage:
```
S3 Bucket: brainy-shared/
├── shards/
│ ├── 001/
│ │ ├── data.bin
│ │ └── .lock (atomic)
│ ├── 002/
│ │ ├── data.bin
│ │ └── .lock
└── metadata/
├── index/
└── locks/
```
**Coordination**: Via distributed locking
- Shard-level locks using atomic operations
- Lock acquisition via compare-and-swap
- Automatic lock expiry (lease-based)
- Deadlock detection and recovery
## Storage Adapter Implementations
### 1. Filesystem Adapter
```typescript
class FilesystemAdapter implements StorageAdapter {
constructor(private basePath: string) {}
async get(key: string) {
const path = this.keyToPath(key)
return fs.readFile(path, 'json')
}
async compareAndSwap(key: string, oldVal: any, newVal: any) {
// Use file locking for atomicity
const lockfile = `${this.keyToPath(key)}.lock`
await flock(lockfile, 'ex') // Exclusive lock
try {
const current = await this.get(key)
if (deepEqual(current, oldVal)) {
await this.set(key, newVal)
return true
}
return false
} finally {
await funlock(lockfile)
}
}
withNamespace(ns: string) {
return new FilesystemAdapter(path.join(this.basePath, ns))
}
}
```
### 2. S3 Adapter
```typescript
class S3Adapter implements StorageAdapter {
constructor(
private bucket: string,
private prefix: string = ''
) {}
async get(key: string) {
const result = await s3.getObject({
Bucket: this.bucket,
Key: `${this.prefix}${key}`
})
return JSON.parse(result.Body)
}
async compareAndSwap(key: string, oldVal: any, newVal: any) {
// Use S3's conditional writes
const fullKey = `${this.prefix}${key}`
// Get current version
const head = await s3.headObject({
Bucket: this.bucket,
Key: fullKey
})
// Conditional put with ETag
try {
await s3.putObject({
Bucket: this.bucket,
Key: fullKey,
Body: JSON.stringify(newVal),
IfMatch: head.ETag // Only succeeds if unchanged
})
return true
} catch (err) {
if (err.code === 'PreconditionFailed') {
return false
}
throw err
}
}
withNamespace(ns: string) {
const newPrefix = `${this.prefix}${ns}/`
return new S3Adapter(this.bucket, newPrefix)
}
}
```
### 3. Cloudflare R2 Adapter
```typescript
class R2Adapter implements StorageAdapter {
// Similar to S3 but with R2-specific optimizations
async compareAndSwap(key: string, oldVal: any, newVal: any) {
// R2 supports conditional headers
const response = await fetch(`${this.endpoint}/${key}`, {
method: 'PUT',
body: JSON.stringify(newVal),
headers: {
'If-Match': await this.getETag(key)
}
})
return response.ok
}
// R2-specific: Use Workers for edge computing
async getWithCache(key: string) {
// Check Cloudflare edge cache first
const cached = await caches.default.match(key)
if (cached) return cached.json()
// Fallback to R2
const value = await this.get(key)
// Cache at edge
await caches.default.put(key, new Response(JSON.stringify(value)))
return value
}
}
```
## Distributed Coordination Patterns
### Pattern 1: Leader-Based Coordination
```typescript
class LeaderCoordinator {
async acquireShardOwnership(shardId: string) {
if (!this.isLeader()) {
// Only leader assigns shards
return this.requestFromLeader('acquireShard', shardId)
}
// Leader logic
const shardMap = await this.storage.get('_cluster/shard-map')
if (!shardMap[shardId].owner) {
shardMap[shardId].owner = this.nodeId
// Atomic update
const success = await this.storage.compareAndSwap(
'_cluster/shard-map',
shardMap,
{ ...shardMap, [shardId]: { owner: this.nodeId } }
)
if (success) {
this.broadcast('shardAssigned', { shardId, owner: this.nodeId })
}
}
}
}
```
### Pattern 2: Consensus-Based Coordination
```typescript
class ConsensusCoordinator {
async acquireShardOwnership(shardId: string) {
// Propose to all nodes
const proposal = {
type: 'ACQUIRE_SHARD',
shardId,
nodeId: this.nodeId,
term: this.currentTerm
}
// Raft consensus
const votes = await this.gatherVotes(proposal)
if (votes.length > this.nodes.length / 2) {
// Majority agreed
await this.commitProposal(proposal)
return true
}
return false
}
}
```
### Pattern 3: Storage-Native Coordination
```typescript
class StorageNativeCoordinator {
async acquireShardOwnership(shardId: string) {
// Use storage adapter's native coordination
const lockKey = `_locks/shard-${shardId}`
const lease = {
owner: this.nodeId,
expires: Date.now() + 30000 // 30 second lease
}
// Try to acquire lock atomically
const acquired = await this.storage.compareAndSwap(
lockKey,
null, // Must not exist
lease
)
if (acquired) {
// Start lease renewal
this.startLeaseRenewal(lockKey, lease)
return true
}
return false
}
private startLeaseRenewal(key: string, lease: any) {
setInterval(async () => {
const renewed = await this.storage.compareAndSwap(
key,
lease,
{ ...lease, expires: Date.now() + 30000 }
)
if (!renewed) {
// Lost lease
this.handleLeaseLoss(key)
}
}, 10000) // Renew every 10s
}
}
```
## Multi-Storage Patterns
### Hybrid Storage (Hot/Cold)
```typescript
class HybridStorageAdapter implements StorageAdapter {
constructor(
private hot: StorageAdapter, // Fast SSD
private cold: StorageAdapter // Cheap S3
) {}
async get(key: string) {
// Try hot storage first
const hotValue = await this.hot.get(key).catch(() => null)
if (hotValue) {
this.updateAccessTime(key)
return hotValue
}
// Fallback to cold storage
const coldValue = await this.cold.get(key)
// Promote to hot storage if frequently accessed
if (this.shouldPromote(key)) {
await this.hot.set(key, coldValue)
}
return coldValue
}
async set(key: string, value: any) {
// Write to hot storage
await this.hot.set(key, value)
// Async write to cold storage
setImmediate(() => {
this.cold.set(key, value).catch(console.error)
})
}
// Background process to demote cold data
async runTiering() {
const hotKeys = await this.hot.listKeys()
for (const key of hotKeys) {
const lastAccess = await this.getAccessTime(key)
if (Date.now() - lastAccess > 7 * 24 * 60 * 60 * 1000) {
// Not accessed in 7 days, demote to cold
await this.cold.set(key, await this.hot.get(key))
await this.hot.delete(key)
}
}
}
}
```
### Geo-Distributed Storage
```typescript
class GeoDistributedAdapter implements StorageAdapter {
constructor(
private regions: Map<string, StorageAdapter>
) {}
async get(key: string) {
// Determine closest region
const region = await this.getClosestRegion()
// Try local region first
const localValue = await this.regions.get(region)
.get(key)
.catch(() => null)
if (localValue) return localValue
// Fallback to other regions
for (const [name, adapter] of this.regions) {
if (name !== region) {
const value = await adapter.get(key).catch(() => null)
if (value) {
// Replicate to local region for next time
this.regions.get(region).set(key, value)
return value
}
}
}
throw new Error('Key not found in any region')
}
async set(key: string, value: any) {
// Write to local region immediately
const region = await this.getClosestRegion()
await this.regions.get(region).set(key, value)
// Async replication to other regions
for (const [name, adapter] of this.regions) {
if (name !== region) {
adapter.set(key, value).catch(console.error)
}
}
}
}
```
## Storage Optimization Strategies
### 1. Write Batching
```typescript
class BatchingAdapter implements StorageAdapter {
private writeBatch = new Map()
private batchTimer?: NodeJS.Timeout
async set(key: string, value: any) {
this.writeBatch.set(key, value)
if (!this.batchTimer) {
this.batchTimer = setTimeout(() => this.flush(), 100)
}
if (this.writeBatch.size >= 1000) {
await this.flush()
}
}
private async flush() {
if (this.writeBatch.size === 0) return
const batch = new Map(this.writeBatch)
this.writeBatch.clear()
await this.underlying.setBatch(batch)
if (this.batchTimer) {
clearTimeout(this.batchTimer)
this.batchTimer = undefined
}
}
}
```
### 2. Read Caching
```typescript
class CachingAdapter implements StorageAdapter {
private cache = new LRU({ max: 10000 })
async get(key: string) {
// Check cache
if (this.cache.has(key)) {
return this.cache.get(key)
}
// Read from storage
const value = await this.underlying.get(key)
// Cache for next time
this.cache.set(key, value)
return value
}
async set(key: string, value: any) {
// Invalidate cache
this.cache.delete(key)
// Write through
await this.underlying.set(key, value)
}
}
```
### 3. Compression
```typescript
class CompressingAdapter implements StorageAdapter {
async set(key: string, value: any) {
const json = JSON.stringify(value)
// Compress if beneficial
if (json.length > 1024) {
const compressed = await gzip(json)
await this.underlying.set(key, {
_compressed: true,
data: compressed.toString('base64')
})
} else {
await this.underlying.set(key, value)
}
}
async get(key: string) {
const stored = await this.underlying.get(key)
if (stored._compressed) {
const compressed = Buffer.from(stored.data, 'base64')
const json = await gunzip(compressed)
return JSON.parse(json)
}
return stored
}
}
```
## Summary
Brainy's storage layer is designed for:
1. **Flexibility**: Works with any storage backend
2. **Coordination**: Multiple strategies for different needs
3. **Performance**: Batching, caching, compression
4. **Scalability**: From single file to geo-distributed
5. **Simplicity**: Complexity hidden behind simple interface
The key insight: **Storage is just a plugin**. The intelligence is in the coordination layer above it!
---
*For user-facing documentation, see [SCALING.md](../SCALING.md)*