soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

chore: remove development planning document

Browse source 
Remove ZERO_CONFIG_PLAN.md from codebase as it was an internal
development document not intended for distribution
This commit is contained in:
David Snelling 2025-08-29 15:41:06 -07:00
parent bd6c154c1f
commit 4d32dbd341
2 changed files with 1 additions and 494 deletions
Download patch file Download diff file Expand all files Collapse all files

493
ZERO_CONFIG_PLAN.md
View file

@ -1,493 +0,0 @@
# Brainy Zero-Config Initiative
## Strategic Plan for Configuration Simplification
Created: 2025-08-29
Version: 2.9.0 Released with `precision` parameter fix
---
## 🚨 Current State: Configuration Overload
### Problem Scale
- **60+ configuration options** across 10+ categories
- **500+ lines of TypeScript interfaces** just for config
- **Nested configs up to 5 levels deep**
- Users overwhelmed with choices that should be automatic
### Major Issues Discovered
1. **`embeddingOptions` doesn't exist!**
- Used in explorer but not in actual API
- Should be via `embeddingFunction` with `createEmbeddingFunction()`
2. **Storage Confusion**
- 7+ different storage configurations
- Complex nested options for S3/GCS/R2
- No auto-detection from environment
3. **Redundant Mode Flags**
- `readOnly`, `frozen`, `writeOnly`, `lazyLoadInReadOnlyMode`, `allowDirectReads`
- Too many overlapping concepts
4. **Manual Performance Tuning**
- Users set `batchSize`, `hotCacheMaxSize`, `efConstruction`
- Should be automatic based on data size and resources
---
## ✅ What Brainy Already Auto-Detects
### Environment & Resources
- **Environment Type**: Browser vs Node vs Serverless (✅ Working)
- **Memory Available**: Via `AutoConfiguration` class (✅ Exists)
- **CPU Cores**: Detected for thread management (✅ Working)
- **Threading**: Auto-detects worker thread availability (✅ Working)
- **Storage Type**: Auto-picks OPFS (browser) or filesystem (Node) (✅ Working)
### Model & Embedding Management
- **Default Embedding**: `defaultEmbeddingFunction` works automatically (✅)
- **Model Downloads**: Auto-downloads on first use (✅ Fixed in 2.7.x)
- **Model Manager**: Falls back through multiple sources (✅)
- **Universal Memory Manager**: Auto-selects embedding strategy (✅)
### Performance Optimization
- **HNSW Defaults**: M=16, efConstruction=200 (✅ Set)
- **Distance Function**: Defaults to cosine (✅)
- **Dimensions**: Fixed at 384 for all-MiniLM-L6-v2 (✅)
- **Cache Auto-tuning**: `CacheAutoConfigurator` adapts to usage (✅)
- **Batch Processing**: Auto-batches operations (✅)
- **Cleanup**: Periodic cleanup runs automatically (✅)
### Augmentations
- **Default Set**: Core augmentations load automatically (✅)
- **WAL**: Write-ahead logging for durability (✅)
- **Connection Pool**: Manages connections automatically (✅)
- **Entity Registry**: Fast ID lookups (✅)
- **Request Deduplication**: Prevents duplicate work (✅)
---
## 🎯 Zero-Config Vision
### True Zero Config (Works Today!)
```typescript
const brain = new BrainyData()
await brain.init()
// ✅ Everything works!
```
**Current Limitations:**
- Uses memory storage (not persistent)
- Uses FP32 models (not Q8)
- Loads all augmentations (might be heavy)
### Ideal Minimal Config (Proposed)
```typescript
// Option 1: Preset-based
const brain = new BrainyData('production') // or 'development', 'minimal'
// Option 2: Three key decisions
const brain = new BrainyData({
storage: 'disk', // or 'memory', 'cloud', auto-detected
model: 'small', // or 'fast' (q8 vs fp32)
features: 'default' // or 'minimal', 'full'
})
// Option 3: Even simpler
const brain = new BrainyData({ mode: 'production' })
```
---
## 📋 Implementation Plan
### Phase 1: Add Missing Auto-Detection (v2.10)
#### 1.1 Storage Auto-Detection
```typescript
function autoDetectStorage(): StorageConfig {
// Check environment variables
if (process.env.AWS_BUCKET) return { type: 's3', bucket: process.env.AWS_BUCKET }
if (process.env.GCS_BUCKET) return { type: 'gcs', bucket: process.env.GCS_BUCKET }
// Check environment type
if (isBrowser()) return { type: 'opfs' }
if (process.env.VERCEL) return { type: 'memory' } // Serverless
// Default to filesystem with smart path
return {
type: 'filesystem',
path: process.env.BRAINY_DATA_PATH || './brainy-data'
}
}
```
#### 1.2 Model Precision Auto-Selection
```typescript
function autoSelectModelPrecision(): 'fp32' | 'q8' {
const memoryMB = getAvailableMemory() / 1024 / 1024
// Use Q8 for constrained environments
if (isBrowser()) return 'q8'
if (memoryMB < 512) return 'q8'
if (process.env.BRAINY_MODEL === 'small') return 'q8'
return 'fp32' // Default to full precision
}
```
#### 1.3 Feature Level Auto-Configuration
```typescript
function autoConfigureFeatures(): FeatureLevel {
// Minimal for browsers and serverless
if (isBrowser() || isServerless()) return 'minimal'
// Full for development
if (process.env.NODE_ENV === 'development') return 'full'
// Default for production
return 'default'
}
```
### Phase 2: Simplify Config Interface (v3.0)
#### 2.1 New Simple Config
```typescript
export interface BrainyConfig {
// Just 3-5 top-level options
mode?: 'development' | 'production' | 'minimal'
storage?: 'auto' | 'memory' | 'disk' | 'cloud' | StorageConfig
model?: 'fast' | 'small' | 'auto' | ModelConfig
features?: 'minimal' | 'default' | 'full' | string[]
// Everything else hidden
advanced?: LegacyConfig
}
```
#### 2.2 Mode Presets
```typescript
const PRESETS = {
development: {
storage: 'memory',
model: 'fast',
features: 'full',
logging: { verbose: true }
},
production: {
storage: 'disk',
model: 'auto',
features: 'default',
logging: { verbose: false }
},
minimal: {
storage: 'memory',
model: 'small',
features: 'minimal',
logging: { verbose: false }
}
}
```
#### 2.3 Smart Defaults by Environment
```typescript
constructor(config?: string | BrainyConfig) {
// String shorthand
if (typeof config === 'string') {
this.config = PRESETS[config]
return
}
// Auto-detect everything if no config
if (!config) {
config = {
mode: process.env.NODE_ENV === 'production' ? 'production' : 'development'
}
}
// Expand simple config to full config
this.config = expandConfig(config)
}
```
### Phase 3: Intelligent Resource Adaptation (v3.1)
#### 3.1 Dynamic Performance Tuning
```typescript
class ResourceAdaptiveConfig {
async adapt() {
const resources = await detectResources()
// Auto-configure based on available resources
if (resources.memoryGB > 8) {
this.enableFeature('aggressive-caching')
this.setHNSW({ M: 32, efConstruction: 400 })
} else if (resources.memoryGB < 1) {
this.enableFeature('minimal-memory')
this.setHNSW({ M: 8, efConstruction: 100 })
}
// Adapt to data size
const dataSize = await this.estimateDataSize()
if (dataSize > 1_000_000) {
this.enableFeature('partitioned-index')
}
}
}
```
#### 3.2 Usage-Based Optimization
```typescript
class UsageOptimizer {
async optimizeFromUsage() {
const patterns = await this.analyzeUsagePatterns()
if (patterns.readWriteRatio > 10) {
// Read-heavy: optimize for search
this.config.cache.size = 'large'
this.config.index.eager = true
} else if (patterns.readWriteRatio < 0.1) {
// Write-heavy: optimize for ingestion
this.config.batch.size = 'large'
this.config.index.lazy = true
}
}
}
```
---
## 🔧 Required Changes
### Immediate (v2.10)
1. **Add `embeddingOptions` support** or document correct usage
2. **Fix storage auto-detection** from environment variables
3. **Add model precision auto-selection**
4. **Create preset system** for common configurations
### Short-term (v3.0)
1. **New simplified config interface** with backward compatibility
2. **Mode-based presets** (development/production/minimal)
3. **Deprecation warnings** for complex configs
4. **Migration guide** from old to new config
### Long-term (v3.1+)
1. **Full resource adaptation** based on runtime environment
2. **Usage-based optimization** that learns from patterns
3. **Cloud config service** for centralized configuration
4. **Zero-config by default** with optional overrides
---
## 📊 Success Metrics
### Developer Experience
- **Time to first query**: < 30 seconds (from npm install)
- **Lines of config**: 0-5 (vs current 50+)
- **Documentation needed**: 1 page (vs current 10+)
### Performance
- **Auto-config performance**: Within 10% of manual tuning
- **Resource usage**: Automatically optimized for environment
- **Startup time**: < 2 seconds with auto-config
### Adoption
- **Zero-config usage**: > 80% of new projects
- **Migration rate**: > 50% of existing projects in 6 months
- **Support tickets**: 50% reduction in config-related issues
---
## 🚀 Next Steps
1. **Release v2.9.0** ✅ DONE - Fixed precision parameter
2. **Create feature flag** for new config system
3. **Implement auto-detection** functions
4. **Test in different environments** (browser, serverless, Node)
5. **Document migration path** from complex to simple config
6. **Release v2.10** with auto-detection
7. **Gather feedback** and iterate
8. **Release v3.0** with new config interface
---
## 📝 Notes
### What Makes Config Complex
- **Too many choices** that could be automatic
- **Nested structures** that hide important options
- **Unclear relationships** between options
- **Manual tuning** of performance parameters
- **Environment-specific** configurations
### What Users Actually Need
1. **Where to store data** (99% just want persistence)
2. **How fast vs how small** (model tradeoff)
3. **Feature level** (minimal/default/full)
Everything else should be automatic!
### Key Insight
> "The best configuration is no configuration. The second best is one that fits in a tweet."
Most users just want:
```typescript
const brain = new BrainyData({ mode: 'production' })
```
And everything else should just work perfectly.
---
## 🔍 Deep Codebase Analysis Results
### Environment Detection Already Available
```typescript
// src/utils/environment.ts
- isBrowser() ✅
- isNode() ✅
- isWebWorker() ✅
- isProductionEnvironment() ✅ (checks NODE_ENV, K_SERVICE, GOOGLE_CLOUD_PROJECT)
- areWorkerThreadsAvailable() ✅
- checkWebGPUSupport() ✅
// Serverless detection (src/embeddings/universal-memory-manager.ts)
- VERCEL ✅
- NETLIFY ✅
- AWS_LAMBDA_FUNCTION_NAME ✅
- FUNCTIONS_WORKER_RUNTIME ✅
```
### Resource Detection Capabilities
```typescript
// src/utils/autoConfiguration.ts - EXISTS but NOT USED!
class AutoConfiguration {
detectEnvironment() ✅
detectResources() ✅ // CPU cores, memory
detectStorageCapabilities() ✅
generateRecommendedConfig() ✅
adaptToDataset() ✅
}
// src/utils/cacheAutoConfig.ts - ACTIVELY USED
class CacheAutoConfigurator {
autoDetectOptimalConfig() ✅
adaptConfiguration() ✅ // Runtime adaptation
detectEnvironment() ✅
}
```
### Environment Variables Already Checked
```typescript
// Model paths
BRAINY_MODELS_PATH ✅
BRAINY_ALLOW_REMOTE_MODELS ✅
BRAINY_Q8_CONFIRMED ✅
// Distributed mode
BRAINY_ROLE ✅ (writer/reader/hybrid)
SERVICE_ENDPOINT ✅
// Cloud storage
AWS_ACCESS_KEY_ID ✅ (but only in scaledHNSWSystem)
AWS_SECRET_ACCESS_KEY ✅
AWS_LAMBDA_FUNCTION_NAME ✅
// Environment
NODE_ENV ✅
K_SERVICE ✅ (Google Cloud Run)
GOOGLE_CLOUD_PROJECT ✅
VERCEL ✅
NETLIFY ✅
// Memory optimization
ORT_DISABLE_MEMORY_ARENA ✅
```
### Storage Auto-Detection MISSING
```typescript
// createStorage() in storageFactory.ts
// Currently requires explicit type specification
// NO auto-detection from environment!
// What we need to add:
function autoDetectStorageType(): StorageType {
// Check cloud providers
if (process.env.AWS_BUCKET || process.env.AWS_ACCESS_KEY_ID) return 's3'
if (process.env.GCS_BUCKET || process.env.GOOGLE_APPLICATION_CREDENTIALS) return 'gcs'
if (process.env.AZURE_STORAGE_ACCOUNT) return 'azure'
// Check environment
if (isBrowser()) return 'opfs'
if (process.env.VERCEL || process.env.NETLIFY) return 'memory'
// Default to filesystem
return 'filesystem'
}
```
### The BIG Discovery: AutoConfiguration EXISTS but UNUSED!
```typescript
// src/utils/autoConfiguration.ts
// COMPLETE auto-configuration system already built!
// But NEVER called from BrainyData constructor!
// It can detect:
- Environment (browser/nodejs/serverless)
- Available memory
- CPU cores
- Threading availability
- Storage capabilities
- S3 availability
- Recommended HNSW params
- Optimization flags
// But it's just sitting there unused! 😱
```
### What Needs to be Connected
1. **Wire up AutoConfiguration in constructor**
```typescript
constructor(config?: BrainyConfig) {
if (!config) {
// Use the existing AutoConfiguration!
const autoConfig = await AutoConfiguration.getInstance().detectAndConfigure()
config = this.convertAutoConfigToBrainyConfig(autoConfig)
}
}
```
2. **Add embedding options to config**
```typescript
// Currently embeddingOptions doesn't exist in BrainyDataConfig
// But explorer uses it!
interface BrainyDataConfig {
embeddingOptions?: {
precision?: 'fp32' | 'q8'
modelCachePath?: string
}
}
```
3. **Create preset system**
```typescript
const PRESETS = {
'zero': {}, // True zero config
'production': { storage: 'disk', model: 'q8', features: 'default' },
'development': { storage: 'memory', model: 'fp32', features: 'full' },
'serverless': { storage: 'memory', model: 'q8', features: 'minimal' }
}
```
### The Path Forward is Clear!
**90% of the auto-detection code already exists!** We just need to:
1. Connect AutoConfiguration to BrainyData constructor
2. Add embeddingOptions to config interface
3. Create preset system
4. Add storage auto-detection from env vars
5. Wire model precision to embedding function creation
This could literally be done in < 200 lines of code!

2
package.json
View file

@ -1,6 +1,6 @@
{
"name": "@soulcraft/brainy",
"version": "2.9.0",
"version": "2.10.0",
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. 31 nouns × 40 verbs for infinite expressiveness.",
"main": "dist/index.js",
"module": "dist/index.js",