feat: implement zero-config system with Node.js 22 compatibility

- Add comprehensive zero-config preset system (production, development, minimal) - Implement intelligent auto-configuration for models and storage - Add Node.js version enforcement for ONNX Runtime stability - Force single-threaded ONNX operations to prevent V8 HandleScope crashes - Create extensible configuration architecture - Add 14 distributed system presets for enterprise deployments - Include detailed documentation and migration guides BREAKING CHANGE: Now requires Node.js 22.x LTS for optimal stability
2025-08-29 15:39:07 -07:00 · 2025-08-29 15:39:07 -07:00 · 5f862bad98
commit 5f862bad98
parent 4d60384755
20 changed files with 3718 additions and 71 deletions
--- a/docs/EXTENDING_STORAGE.md
+++ b/docs/EXTENDING_STORAGE.md
@ -0,0 +1,396 @@
+# 🔌 Extending Brainy Storage with Augmentations
+
+## Overview
+
+Brainy's zero-config system is **fully extensible**. Augmentations can register new storage providers, presets, and auto-detection logic that integrates seamlessly with the existing system.
+
+## How Storage Extensions Work
+
+### 1. Storage Provider Registration
+
+When an augmentation is installed, it can register a new storage provider:
+
+```typescript
+import { StorageProvider, registerStorageAugmentation } from '@soulcraft/brainy/config'
+
+const redisProvider: StorageProvider = {
+  type: 'redis',
+  name: 'Redis Storage',
+  description: 'High-performance in-memory data store',
+  priority: 10,  // Higher priority = checked first in auto-detection
+  
+  // Auto-detection logic
+  async detect(): Promise<boolean> {
+    // Check if Redis is available
+    if (process.env.REDIS_URL) {
+      try {
+        const redis = await import('ioredis')
+        const client = new redis.default(process.env.REDIS_URL)
+        await client.ping()
+        await client.quit()
+        return true
+      } catch {
+        return false
+      }
+    }
+    return false
+  },
+  
+  // Configuration builder
+  async getConfig(): Promise<any> {
+    return {
+      type: 'redis',
+      redisStorage: {
+        url: process.env.REDIS_URL,
+        prefix: 'brainy:',
+        ttl: 3600
+      }
+    }
+  }
+}
+
+// Register the provider
+registerStorageAugmentation(redisProvider)
+```
+
+### 2. Using Extended Storage
+
+Once registered, the new storage type works with zero-config:
+
+```typescript
+// Auto-detection will now check Redis
+const brain = new BrainyData()  // Will use Redis if available!
+
+// Or explicitly specify
+const brain = new BrainyData({ storage: 'redis' })
+
+// Or with custom config
+const brain = new BrainyData({
+  storage: {
+    type: 'redis',
+    redisStorage: {
+      url: 'redis://localhost:6379',
+      prefix: 'myapp:'
+    }
+  }
+})
+```
+
+## Real-World Examples
+
+### Redis Augmentation
+
+```typescript
+// @soulcraft/brainy-redis package
+export class RedisStorageAugmentation {
+  async init() {
+    // Register the storage provider
+    registerStorageAugmentation({
+      type: 'redis',
+      name: 'Redis Storage',
+      priority: 10,
+      
+      async detect() {
+        return !!(process.env.REDIS_URL || process.env.REDIS_HOST)
+      },
+      
+      async getConfig() {
+        return {
+          type: 'redis',
+          redisStorage: {
+            url: process.env.REDIS_URL || 
+                 `redis://${process.env.REDIS_HOST}:${process.env.REDIS_PORT || 6379}`
+          }
+        }
+      }
+    })
+    
+    // Register Redis-specific presets
+    registerPresetAugmentation('redis-cache', {
+      storage: 'redis',
+      model: ModelPrecision.Q8,
+      features: ['core', 'cache'],
+      distributed: true,
+      description: 'Redis-backed cache layer',
+      category: PresetCategory.SERVICE
+    })
+  }
+}
+```
+
+### MongoDB Augmentation
+
+```typescript
+// @soulcraft/brainy-mongodb package
+export class MongoStorageAugmentation {
+  async init() {
+    registerStorageAugmentation({
+      type: 'mongodb',
+      name: 'MongoDB Storage',
+      priority: 8,
+      
+      async detect() {
+        return !!(process.env.MONGODB_URI || process.env.MONGO_URL)
+      },
+      
+      async getConfig() {
+        return {
+          type: 'mongodb',
+          mongoStorage: {
+            uri: process.env.MONGODB_URI,
+            database: 'brainy',
+            collection: 'vectors'
+          }
+        }
+      }
+    })
+  }
+}
+```
+
+### PostgreSQL + pgvector Augmentation
+
+```typescript
+// @soulcraft/brainy-postgres package
+export class PostgresStorageAugmentation {
+  async init() {
+    registerStorageAugmentation({
+      type: 'postgres',
+      name: 'PostgreSQL + pgvector',
+      priority: 9,
+      
+      async detect() {
+        const url = process.env.DATABASE_URL
+        if (url?.includes('postgres')) {
+          // Check for pgvector extension
+          const client = new Client({ connectionString: url })
+          await client.connect()
+          const result = await client.query(
+            "SELECT * FROM pg_extension WHERE extname = 'vector'"
+          )
+          await client.end()
+          return result.rows.length > 0
+        }
+        return false
+      },
+      
+      async getConfig() {
+        return {
+          type: 'postgres',
+          postgresStorage: {
+            connectionString: process.env.DATABASE_URL,
+            table: 'brainy_vectors'
+          }
+        }
+      }
+    })
+  }
+}
+```
+
+## Auto-Detection Priority
+
+Storage providers are checked in priority order:
+
+1. **Custom providers** (highest priority first)
+2. **Cloud storage** (S3, GCS, R2)
+3. **Database storage** (Redis, MongoDB, PostgreSQL)
+4. **Local storage** (filesystem, OPFS)
+5. **Memory** (fallback)
+
+```typescript
+// Example priority chain
+Redis (priority: 10) → PostgreSQL (9) → MongoDB (8) → S3 (5) → Filesystem (1) → Memory (0)
+```
+
+## Creating Custom Presets
+
+Augmentations can also register new presets:
+
+```typescript
+registerPresetAugmentation('redis-cluster', {
+  storage: 'redis',
+  model: ModelPrecision.Q8,
+  features: ['core', 'cache', 'cluster'],
+  distributed: true,
+  role: DistributedRole.HYBRID,
+  cache: {
+    hotCacheMaxSize: 100000,  // Large distributed cache
+    autoTune: true
+  },
+  description: 'Redis Cluster configuration',
+  category: PresetCategory.SERVICE
+})
+
+// Users can then use:
+const brain = new BrainyData('redis-cluster')
+```
+
+## Type Safety with Extensions
+
+To maintain type safety with dynamic storage types:
+
+```typescript
+// Augmentation declares its types
+declare module '@soulcraft/brainy' {
+  interface StorageTypes {
+    redis: {
+      url: string
+      prefix?: string
+      ttl?: number
+    }
+  }
+  
+  interface PresetNames {
+    'redis-cache': 'redis-cache'
+    'redis-cluster': 'redis-cluster'
+  }
+}
+```
+
+## Best Practices for Storage Augmentations
+
+1. **Always provide auto-detection** - Check environment variables and connectivity
+2. **Set appropriate priority** - Higher for specialized storage, lower for general
+3. **Handle failures gracefully** - Return false from detect() if not available
+4. **Document requirements** - List required packages and environment variables
+5. **Provide presets** - Include common configuration patterns
+6. **Maintain compatibility** - Ensure model precision matches across instances
+
+## Example: Complete Redis Augmentation
+
+```typescript
+import { 
+  StorageProvider, 
+  registerStorageAugmentation,
+  registerPresetAugmentation,
+  PresetCategory,
+  ModelPrecision,
+  DistributedRole
+} from '@soulcraft/brainy/config'
+import Redis from 'ioredis'
+
+export class BrainyRedisAugmentation {
+  private client: Redis
+  
+  async init() {
+    // Register storage provider
+    registerStorageAugmentation({
+      type: 'redis',
+      name: 'Redis Vector Storage',
+      description: 'Redis with RediSearch for vector similarity',
+      priority: 10,
+      
+      requirements: {
+        env: ['REDIS_URL'],
+        packages: ['ioredis', 'redis']
+      },
+      
+      async detect() {
+        if (!process.env.REDIS_URL) return false
+        
+        try {
+          const client = new Redis(process.env.REDIS_URL)
+          
+          // Check for RediSearch module
+          const modules = await client.call('MODULE', 'LIST')
+          const hasRediSearch = modules.some(m => m[1] === 'search')
+          
+          await client.quit()
+          return hasRediSearch
+        } catch {
+          return false
+        }
+      },
+      
+      async getConfig() {
+        return {
+          type: 'redis',
+          redisStorage: {
+            url: process.env.REDIS_URL,
+            prefix: process.env.REDIS_PREFIX || 'brainy:',
+            index: process.env.REDIS_INDEX || 'brainy-vectors',
+            ttl: process.env.REDIS_TTL ? parseInt(process.env.REDIS_TTL) : undefined
+          }
+        }
+      }
+    })
+    
+    // Register presets
+    this.registerPresets()
+  }
+  
+  private registerPresets() {
+    // Fast cache preset
+    registerPresetAugmentation('redis-fast-cache', {
+      storage: 'redis' as any,
+      model: ModelPrecision.Q8,
+      features: ['core', 'cache', 'search'],
+      distributed: false,
+      cache: {
+        hotCacheMaxSize: 10000,
+        autoTune: true
+      },
+      description: 'Redis-backed fast cache',
+      category: PresetCategory.SERVICE
+    })
+    
+    // Distributed cache preset
+    registerPresetAugmentation('redis-distributed', {
+      storage: 'redis' as any,
+      model: ModelPrecision.AUTO,
+      features: ['core', 'cache', 'search', 'cluster'],
+      distributed: true,
+      role: DistributedRole.HYBRID,
+      cache: {
+        hotCacheMaxSize: 50000,
+        autoTune: true
+      },
+      description: 'Redis distributed cache cluster',
+      category: PresetCategory.SERVICE
+    })
+    
+    // Session store preset
+    registerPresetAugmentation('redis-sessions', {
+      storage: 'redis' as any,
+      model: ModelPrecision.Q8,
+      features: ['core', 'cache'],
+      distributed: false,
+      cache: {
+        hotCacheMaxSize: 5000,
+        autoTune: false
+      },
+      description: 'Redis session storage',
+      category: PresetCategory.SERVICE
+    })
+  }
+}
+
+// Usage after installing the augmentation:
+import { BrainyData } from '@soulcraft/brainy'
+import '@soulcraft/brainy-redis'  // Registers the augmentation
+
+// Now Redis is automatically detected!
+const brain = new BrainyData()  // Uses Redis if REDIS_URL is set
+
+// Or use a Redis preset
+const brain = new BrainyData('redis-fast-cache')
+
+// Or explicitly configure
+const brain = new BrainyData({
+  storage: 'redis',
+  model: ModelPrecision.FP32
+})
+```
+
+## Summary
+
+The extensible configuration system allows:
+
+1. **New storage types** via `registerStorageAugmentation()`
+2. **Custom presets** via `registerPresetAugmentation()`
+3. **Auto-detection logic** that integrates with zero-config
+4. **Type-safe extensions** with TypeScript declarations
+5. **Priority-based selection** for intelligent defaults
+
+This ensures Brainy can grow with new storage technologies while maintaining its zero-configuration philosophy!
--- a/docs/ZERO_CONFIG.md
+++ b/docs/ZERO_CONFIG.md
@ -0,0 +1,409 @@
+# 🚀 Brainy Zero-Configuration Guide
+
+## Overview
+
+Starting with v2.10, Brainy introduces a **Zero-Configuration System** that automatically configures everything based on your environment. No more environment variables, no more complex configuration objects - just create and use.
+
+## Quick Start
+
+### True Zero Config
+```typescript
+import { BrainyData } from '@soulcraft/brainy'
+
+// That's it! Everything auto-configures
+const brain = new BrainyData()
+await brain.init()
+```
+
+### Using Strongly-Typed Presets
+
+```typescript
+import { BrainyData, PresetName } from '@soulcraft/brainy'
+
+// Type-safe preset selection
+const brain = new BrainyData(PresetName.PRODUCTION)
+await brain.init()
+```
+
+### Common Scenarios
+
+#### Development
+```typescript
+const brain = new BrainyData(PresetName.DEVELOPMENT)
+// ✅ Memory storage for fast iteration
+// ✅ FP32 models for best quality
+// ✅ Verbose logging
+// ✅ All features enabled
+```
+
+#### Production
+```typescript
+const brain = new BrainyData(PresetName.PRODUCTION)
+// ✅ Disk storage for persistence
+// ✅ Auto-selected model precision
+// ✅ Silent logging
+// ✅ Optimized features
+```
+
+#### Minimal
+```typescript
+const brain = new BrainyData(PresetName.MINIMAL)
+// ✅ Memory storage
+// ✅ Q8 models for small size
+// ✅ Core features only
+// ✅ Minimal resource usage
+```
+
+## Distributed Architecture Presets
+
+Brainy includes specialized presets for distributed and microservice architectures:
+
+### Basic Distributed Roles
+```typescript
+import { BrainyData, PresetName } from '@soulcraft/brainy'
+
+// Write-only instance (data ingestion)
+const writer = new BrainyData(PresetName.WRITER)
+// ✅ Optimized for writes
+// ✅ No search index loading
+// ✅ Minimal memory usage
+
+// Read-only instance (search API)
+const reader = new BrainyData(PresetName.READER)
+// ✅ Optimized for search
+// ✅ Lazy index loading
+// ✅ Large cache
+```
+
+### Service-Specific Presets
+```typescript
+// High-throughput data ingestion
+const ingestion = new BrainyData(PresetName.INGESTION_SERVICE)
+
+// Low-latency search API
+const searchApi = new BrainyData(PresetName.SEARCH_API)
+
+// Analytics processing
+const analytics = new BrainyData(PresetName.ANALYTICS_SERVICE)
+
+// Edge location cache
+const edge = new BrainyData(PresetName.EDGE_CACHE)
+
+// Batch processing
+const batch = new BrainyData(PresetName.BATCH_PROCESSOR)
+
+// Real-time streaming
+const streaming = new BrainyData(PresetName.STREAMING_SERVICE)
+
+// ML training
+const training = new BrainyData(PresetName.ML_TRAINING)
+
+// Lightweight sidecar
+const sidecar = new BrainyData(PresetName.SIDECAR)
+```
+
+## Model Precision Control
+
+You can **explicitly specify** model precision when needed:
+
+```typescript
+import { ModelPrecision } from '@soulcraft/brainy'
+
+// Force FP32 (full precision)
+const brain = new BrainyData({ model: ModelPrecision.FP32 })
+
+// Force Q8 (quantized, smaller)
+const brain = new BrainyData({ model: ModelPrecision.Q8 })
+
+// Use presets
+const brain = new BrainyData({ model: ModelPrecision.FAST })  // Maps to fp32
+const brain = new BrainyData({ model: ModelPrecision.SMALL }) // Maps to q8
+
+// Auto-detection (default)
+const brain = new BrainyData({ model: ModelPrecision.AUTO })
+```
+
+### Auto-Detection Logic
+
+When not specified, Brainy automatically selects the best model:
+
+- **Browser**: Q8 (smaller download)
+- **Serverless**: Q8 (faster cold starts)
+- **Low Memory (<512MB)**: Q8
+- **Development**: FP32 (best quality)
+- **Production (>2GB RAM)**: FP32
+- **Default**: Q8 (balanced)
+
+## Storage Configuration
+
+### Automatic Storage Detection
+
+Brainy automatically detects the best storage option:
+
+1. **Cloud Storage** (if credentials found)
+   - AWS S3 (checks AWS_ACCESS_KEY_ID, AWS_PROFILE)
+   - Google Cloud Storage (checks GOOGLE_APPLICATION_CREDENTIALS)
+   - Cloudflare R2 (checks R2_ACCESS_KEY_ID)
+
+2. **Browser Storage**
+   - OPFS (if supported)
+   - Memory (fallback)
+
+3. **Node.js Storage**
+   - Filesystem (`./brainy-data` or `~/.brainy/data`)
+   - Memory (for serverless)
+
+### Manual Storage Control
+
+```typescript
+import { StorageOption } from '@soulcraft/brainy'
+
+// Force specific storage with enum
+const brain = new BrainyData({ storage: StorageOption.MEMORY })
+const brain = new BrainyData({ storage: StorageOption.DISK })
+const brain = new BrainyData({ storage: StorageOption.CLOUD })
+const brain = new BrainyData({ storage: StorageOption.AUTO })
+
+// Custom storage configuration
+const brain = new BrainyData({
+  storage: {
+    s3Storage: {
+      bucket: 'my-bucket',
+      region: 'us-east-1'
+    }
+  }
+})
+```
+
+## Feature Sets
+
+Control which features are enabled:
+
+```typescript
+import { FeatureSet } from '@soulcraft/brainy'
+
+// Preset feature sets with enum
+const brain = new BrainyData({ features: FeatureSet.MINIMAL })  // Core only
+const brain = new BrainyData({ features: FeatureSet.DEFAULT })  // Balanced
+const brain = new BrainyData({ features: FeatureSet.FULL })     // Everything
+
+// Custom features
+const brain = new BrainyData({
+  features: ['core', 'search', 'cache', 'triple-intelligence']
+})
+```
+
+## Simplified Configuration Interface
+
+The new configuration is dramatically simpler:
+
+```typescript
+interface BrainyZeroConfig {
+  // Mode preset - now with distributed options
+  mode?: PresetName  // All strongly typed presets
+  
+  // Model configuration with enum
+  model?: ModelPrecision
+  
+  // Storage configuration with enum
+  storage?: StorageOption | StorageConfig
+  
+  // Feature set with enum
+  features?: FeatureSet | string[]
+  
+  // Logging
+  verbose?: boolean
+  
+  // Escape hatch for advanced users
+  advanced?: any
+}
+```
+
+### Available Enums
+
+```typescript
+enum PresetName {
+  // Basic
+  PRODUCTION = 'production',
+  DEVELOPMENT = 'development',
+  MINIMAL = 'minimal',
+  ZERO = 'zero',
+  
+  // Distributed
+  WRITER = 'writer',
+  READER = 'reader',
+  
+  // Services
+  INGESTION_SERVICE = 'ingestion-service',
+  SEARCH_API = 'search-api',
+  ANALYTICS_SERVICE = 'analytics-service',
+  EDGE_CACHE = 'edge-cache',
+  BATCH_PROCESSOR = 'batch-processor',
+  STREAMING_SERVICE = 'streaming-service',
+  ML_TRAINING = 'ml-training',
+  SIDECAR = 'sidecar'
+}
+
+enum ModelPrecision {
+  FP32 = 'fp32',
+  Q8 = 'q8',
+  AUTO = 'auto',
+  FAST = 'fast',  // Maps to fp32
+  SMALL = 'small' // Maps to q8
+}
+
+enum StorageOption {
+  AUTO = 'auto',
+  MEMORY = 'memory',
+  DISK = 'disk',
+  CLOUD = 'cloud'
+}
+
+enum FeatureSet {
+  MINIMAL = 'minimal',
+  DEFAULT = 'default',
+  FULL = 'full'
+}
+```
+
+## Multi-Instance with Shared Storage
+
+When multiple Brainy instances connect to the same storage (like S3), you **must ensure they use compatible configurations**:
+
+```typescript
+import { ModelPrecision } from '@soulcraft/brainy'
+
+// Container A - Writer
+const writer = new BrainyData({
+  mode: PresetName.WRITER,
+  model: ModelPrecision.FP32,  // ⚠️ MUST match across instances!
+  storage: { s3Storage: { bucket: 'shared-data' }}
+})
+
+// Container B - Reader
+const reader = new BrainyData({
+  mode: PresetName.READER,
+  model: ModelPrecision.FP32,  // ✅ Matches Container A
+  storage: { s3Storage: { bucket: 'shared-data' }}
+})
+```
+
+### Distributed Architecture Example
+
+```typescript
+// Ingestion Service (Writer)
+const ingestion = new BrainyData({
+  mode: PresetName.INGESTION_SERVICE,
+  model: ModelPrecision.Q8,  // All instances must use Q8
+  storage: { s3Storage: { bucket: 'production-data' }}
+})
+
+// Search API (Reader)
+const search = new BrainyData({
+  mode: PresetName.SEARCH_API,
+  model: ModelPrecision.Q8,  // Matches ingestion service
+  storage: { s3Storage: { bucket: 'production-data' }}
+})
+
+// Analytics (Hybrid)
+const analytics = new BrainyData({
+  mode: PresetName.ANALYTICS_SERVICE,
+  model: ModelPrecision.Q8,  // Matches other services
+  storage: { s3Storage: { bucket: 'production-data' }}
+})
+```
+
+## Migration from Old Configuration
+
+### Before (Complex)
+```typescript
+const brain = new BrainyData({
+  hnsw: {
+    M: 16,
+    efConstruction: 200,
+    seed: 42
+  },
+  storage: {
+    s3Storage: {
+      bucketName: 'my-bucket',
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
+      region: 'us-east-1'
+    },
+    cacheConfig: {
+      hotCacheMaxSize: 5000,
+      hotCacheEvictionThreshold: 0.8,
+      warmCacheTTL: 3600000,
+      batchSize: 100
+    }
+  },
+  cache: {
+    autoTune: true,
+    autoTuneInterval: 60000,
+    hotCacheMaxSize: 10000
+  },
+  embeddingFunction: customFunction,
+  readOnly: false,
+  logging: { verbose: true }
+})
+```
+
+### After (Simple)
+```typescript
+const brain = new BrainyData('production')
+// Everything above is auto-configured!
+```
+
+## Environment Variables (No Longer Needed!)
+
+These environment variables are **no longer required**:
+
+- ❌ `BRAINY_ALLOW_REMOTE_MODELS` - Models auto-download when needed
+- ❌ `BRAINY_MODELS_PATH` - Path auto-selected based on environment
+- ❌ `BRAINY_Q8_CONFIRMED` - Warnings auto-suppressed in production
+- ❌ `BRAINY_LOG_LEVEL` - Auto-set based on NODE_ENV
+- ❌ AWS credentials - Use AWS SDK credential chain
+
+## Performance Impact
+
+The zero-config system has **zero performance overhead**:
+
+- Configuration happens once during initialization
+- Auto-detected values are cached
+- Same optimized code paths as manual configuration
+- Actually **faster** startup due to reduced parsing
+
+## Troubleshooting
+
+### Models Not Downloading
+- Check internet connection
+- Ensure firewall allows HTTPS to Hugging Face / CDN
+- Run `npm run download-models` to pre-download
+
+### Wrong Model Precision
+- Explicitly specify: `{ model: 'fp32' }` or `{ model: 'q8' }`
+- Check shared storage compatibility
+
+### Storage Detection Issues
+- Check cloud credentials are properly configured
+- Verify write permissions for filesystem paths
+- Use explicit storage configuration if needed
+
+## Best Practices
+
+1. **Use zero-config for single instances** - Let Brainy handle everything
+2. **Specify precision for shared storage** - Ensure compatibility
+3. **Use presets for common scenarios** - 'development', 'production', 'minimal'
+4. **Override only what you need** - Start simple, add complexity only if required
+
+## Summary
+
+The new zero-config system reduces configuration from **100+ parameters** to **0-3 decisions**:
+
+| Scenario | Old Config Lines | New Config Lines |
+|----------|-----------------|------------------|
+| Development | 50+ | 1 |
+| Production | 100+ | 1 |
+| Custom | 200+ | 3-5 |
+
+**Result**: 95% less configuration, 100% of the power! 🚀