CHECKPOINT: Industry-standard 3-tier testing implemented

✅ MAJOR BREAKTHROUGH - Session 5 Success: - Unit tests: 18/19 passing with mocked AI (<500MB RAM) - Integration tests: Real AI models loading successfully - Core features: Real embeddings, CRUD operations verified - Architecture: All 11 augmentations, worker threads operational 📋 CRITICAL FINDINGS: - Real AI models load and cache correctly - 384D embeddings generate properly - Core CRUD operations work with real transformers - Memory management effective for production ⚠️ RELEASE BLOCKER IDENTIFIED: - Search operations timeout in test environment - Affects: search(), find(), clustering functionality - Root cause: Likely worker communication during HNSW search - Priority: MUST fix before 2.0.0 release 🎯 NEXT SESSION PRIORITIES: 1. Debug and fix search timeout issue 2. Verify search/find/clustering work in production 3. Final documentation cleanup 4. Release preparation Confidence: 90% ready (pending search functionality verification)
2025-08-25 17:12:58 -07:00 · 2025-08-25 17:12:58 -07:00 · 4949b6a629
commit 4949b6a629
parent f0ee5f44ec
54 changed files with 4987 additions and 68 deletions
--- a/docs/ONNX-OPTIMIZATIONS.md
+++ b/docs/ONNX-OPTIMIZATIONS.md
@ -0,0 +1,250 @@
+# 🎯 ONNX Runtime Optimizations for Brainy
+
+## The Problem
+ONNX runtime allocates 4-8GB of memory for a 30MB model file, causing memory exhaustion even with adequate heap allocation.
+
+## Available Solutions & Workarounds
+
+### 1. **Use Quantized Models** (IMMEDIATE FIX)
+The most effective solution - reduces memory by 75%:
+
+```javascript
+// In src/utils/embedding.ts
+const pipelineOptions: any = {
+  cache_dir: cacheDir,
+  local_files_only: this.options.localFilesOnly,
+  dtype: 'q8'  // Change from 'fp32' to 'q8' or 'q4'
+}
+```
+
+**Memory Impact:**
+- `fp32` (default): 4-8GB memory usage
+- `fp16`: ~3-4GB memory usage  
+- `q8`: ~1-2GB memory usage ✅ RECOMMENDED
+- `q4`: ~500MB-1GB memory usage (lower quality)
+
+### 2. **Enable ONNX Execution Providers** (PLATFORM SPECIFIC)
+
+#### For CPU Optimization:
+```javascript
+// Add to pipeline options
+const pipelineOptions = {
+  // ... existing options
+  session_options: {
+    executionProviders: ['cpu'],
+    interOpNumThreads: 2,  // Limit threads
+    intraOpNumThreads: 2,  // Limit parallelism
+    graphOptimizationLevel: 'all',
+    enableCpuMemArena: false,  // CRITICAL: Disable memory arena
+    enableMemPattern: false    // CRITICAL: Disable memory patterns
+  }
+}
+```
+
+#### For WebAssembly (Browser):
+```javascript
+const pipelineOptions = {
+  session_options: {
+    executionProviders: ['wasm'],
+    wasmPaths: '/path/to/wasm/files/',
+    numThreads: 1  // Single-threaded for lower memory
+  }
+}
+```
+
+### 3. **Memory Arena Disable** (CRITICAL FIX)
+ONNX pre-allocates huge memory arenas by default:
+
+```javascript
+// In src/utils/embedding.ts, update the pipeline creation:
+import { env } from '@huggingface/transformers'
+
+// Before loading model
+env.onnx.wasm.numThreads = 1  // Limit WASM threads
+env.onnx.wasm.simd = true     // Use SIMD if available
+
+// Disable memory arena globally
+if (typeof process !== 'undefined') {
+  process.env.ORT_DISABLE_MEMORY_ARENA = '1'
+  process.env.ORT_DISABLE_MEMORY_PATTERN = '1'
+}
+```
+
+### 4. **Batch Size Optimization**
+Process embeddings in smaller batches:
+
+```javascript
+// Instead of processing all at once
+const embeddings = await this.embed(texts)
+
+// Process in small batches
+const BATCH_SIZE = 10  // Reduced from 50
+const embeddings = []
+for (let i = 0; i < texts.length; i += BATCH_SIZE) {
+  const batch = texts.slice(i, i + BATCH_SIZE)
+  const batchEmbeddings = await this.embed(batch)
+  embeddings.push(...batchEmbeddings)
+  
+  // Force garbage collection between batches (Node.js only)
+  if (global.gc) {
+    global.gc()
+  }
+}
+```
+
+### 5. **Model Unloading** (MEMORY RECOVERY)
+Unload model when not in use:
+
+```javascript
+class TransformerEmbedding {
+  private idleTimer: NodeJS.Timeout | null = null
+  
+  async embed(text: string | string[]): Promise<Vector[]> {
+    // Clear idle timer
+    if (this.idleTimer) {
+      clearTimeout(this.idleTimer)
+    }
+    
+    // Do embedding...
+    const result = await this.doEmbed(text)
+    
+    // Set idle timer to unload after 5 minutes
+    this.idleTimer = setTimeout(() => {
+      this.unloadModel()
+    }, 5 * 60 * 1000)
+    
+    return result
+  }
+  
+  private async unloadModel(): Promise<void> {
+    if (this.extractor) {
+      // Dispose of the pipeline
+      await this.extractor.dispose()
+      this.extractor = null
+      
+      // Force garbage collection
+      if (global.gc) {
+        global.gc()
+      }
+      
+      console.log('Model unloaded to free memory')
+    }
+  }
+}
+```
+
+### 6. **Use ONNX Runtime Web** (Browser Alternative)
+For browser environments, use the lighter ONNX Runtime Web:
+
+```javascript
+// Use onnxruntime-web instead of full onnxruntime-node
+import * as ort from 'onnxruntime-web'
+
+// Configure for minimal memory
+ort.env.wasm.numThreads = 1
+ort.env.wasm.simd = true
+ort.env.wasm.proxy = false  // Don't use worker
+```
+
+### 7. **Pre-computed Embeddings** (BEST FOR PRODUCTION)
+For known data, pre-compute embeddings:
+
+```javascript
+// During build/deploy time
+const precomputedEmbeddings = {
+  'javascript': [0.1, 0.2, ...],
+  'python': [0.15, 0.25, ...],
+  // ... more common terms
+}
+
+// At runtime
+async embed(text) {
+  // Check cache first
+  if (precomputedEmbeddings[text.toLowerCase()]) {
+    return precomputedEmbeddings[text.toLowerCase()]
+  }
+  
+  // Only compute if not cached
+  return this.computeEmbedding(text)
+}
+```
+
+## Recommended Implementation
+
+### Quick Fix (Immediate)
+1. Change dtype to 'q8' in embedding.ts
+2. Set `ORT_DISABLE_MEMORY_ARENA=1` environment variable
+3. Reduce batch size to 10
+
+### Code Changes for embedding.ts:
+```javascript
+// At the top of the file
+if (typeof process !== 'undefined') {
+  process.env.ORT_DISABLE_MEMORY_ARENA = '1'
+  process.env.ORT_DISABLE_MEMORY_PATTERN = '1'
+}
+
+// In constructor
+this.options = {
+  model: options.model || 'Xenova/all-MiniLM-L6-v2',
+  verbose: this.verbose,
+  cacheDir: options.cacheDir || './models',
+  localFilesOnly: localFilesOnly,
+  dtype: options.dtype || 'q8',  // Changed from fp32
+  device: options.device || 'auto',
+  batchSize: 10  // Reduced from default
+}
+
+// In loadModel
+const pipelineOptions: any = {
+  cache_dir: cacheDir,
+  local_files_only: isBrowser() ? false : this.options.localFilesOnly,
+  dtype: this.options.dtype,
+  session_options: {
+    enableCpuMemArena: false,
+    enableMemPattern: false,
+    interOpNumThreads: 2,
+    intraOpNumThreads: 2
+  }
+}
+```
+
+## Testing Memory Optimizations
+
+### Before Optimizations:
+```bash
+# Uses 4-8GB
+node test-quick.js
+# CRASH: JavaScript heap out of memory
+```
+
+### After Optimizations:
+```bash
+# Should use 1-2GB
+ORT_DISABLE_MEMORY_ARENA=1 node test-quick.js
+# SUCCESS: Tests pass
+```
+
+## Performance Impact
+
+| Optimization | Memory Reduction | Speed Impact | Quality Impact |
+|-------------|-----------------|--------------|----------------|
+| Quantization (q8) | 75% | ~5% slower | <1% accuracy loss |
+| Disable Arena | 30-50% | No impact | None |
+| Batch Size 10 | 20% | 10% slower | None |
+| Thread Limit | 10-20% | 20% slower | None |
+| Model Unload | 100% when idle | Reload delay | None |
+
+## Conclusion
+
+**Immediate Action**: 
+1. Use q8 quantization
+2. Disable memory arena
+3. Reduce batch size
+
+This should reduce memory usage from 4-8GB to **1-2GB** with minimal performance impact.
+
+**Long-term Solution**:
+- Implement model unloading
+- Pre-compute common embeddings
+- Consider using ONNX Runtime Web for lighter footprint