brainy/PERFORMANCE_OPTIMIZATION_TODO.md

🚀 Metadata Filtering Performance Optimization

Priority: HIGH | Complexity: Medium | Est. Time: 2-3 hours

🎯 The Issue

Current metadata filtering has a 300-400% search overhead due to a fixed 3x ef multiplier in HNSW search, regardless of filter selectivity.

Performance Analysis

• No filtering: 31.46ms average
• Simple filter: 150.82ms average (379% overhead)
• Complex filter: 153.05ms average (386% overhead)

Root Cause

// Current implementation (inefficient)
// File: /home/dpsifr/Projects/brainy/src/hnsw/hnswIndex.ts:377
const ef = filter ? Math.max(this.config.efSearch * 3, k * 3) : Math.max(this.config.efSearch, k)

The fixed 3x multiplier is used for ALL filtered searches, whether the filter matches 1% or 90% of items.

🛠️ Solution: Dynamic ef Multiplier

1. Calculate Filter Selectivity

// Add to searchByNounTypes method
const selectivity = candidateIds.length / totalItems

2. Dynamic Multiplier Strategy

function getEfMultiplier(selectivity: number): number {
  if (selectivity <= 0.01) return 1.1    // 1% match: minimal overhead
  if (selectivity <= 0.05) return 1.3    // 5% match: small overhead  
  if (selectivity <= 0.15) return 1.6    // 15% match: medium overhead
  if (selectivity <= 0.30) return 2.0    // 30% match: higher overhead
  return 1.0                             // 30%+ match: no overhead (post-filter)
}

3. Implementation Location

File: /home/dpsifr/Projects/brainy/src/brainyData.ts
Method: searchByNounTypes (around line 2165)

// Current code around line 2165:
if (hasMetadataFilter && this.metadataIndex) {
  const candidateIds = await this.metadataIndex.getIdsForCriteria(options.metadata)
  
  // ADD THIS: Calculate selectivity
  const totalItems = this.index.size()
  const selectivity = candidateIds.length / totalItems
  const efMultiplier = getEfMultiplier(selectivity)
  
  // Store efMultiplier for use in HNSW search
}

File: /home/dpsifr/Projects/brainy/src/hnsw/hnswIndex.ts
Method: search (around line 377)

// Replace this line:
const ef = filter ? Math.max(this.config.efSearch * 3, k * 3) : Math.max(this.config.efSearch, k)

// With dynamic calculation:
const ef = filter && filterSelectivity 
  ? Math.max(this.config.efSearch * filterSelectivity.multiplier, k)
  : Math.max(this.config.efSearch, k)

📈 Expected Performance Improvements

Filter Selectivity	Current Overhead	Expected Overhead	Improvement
1% (high selectivity)	379%	50%	85% faster
5% (medium selectivity)	379%	80%	70% faster
15% (low selectivity)	379%	150%	50% faster
30%+ (very low selectivity)	379%	10%	90% faster

🔧 Implementation Steps

Step 1: Add Selectivity Calculation

1. Modify searchByNounTypes to calculate selectivity
2. Pass selectivity info to HNSW search method
3. Create getEfMultiplier helper function

Step 2: Update HNSW Search

1. Modify HNSWIndex.search to accept selectivity parameter
2. Update HNSWIndexOptimized.search to forward selectivity
3. Replace fixed 3x multiplier with dynamic calculation

Step 3: Test Performance

1. Run performance benchmarks with different selectivity scenarios
2. Verify filtering accuracy is maintained
3. Test edge cases (empty results, 100% selectivity)

Step 4: Optional Enhancements

1. Index-First Strategy: For <5% selectivity, search only candidate vectors
2. Post-Filter Strategy: For >50% selectivity, search all then filter
3. Query Pattern Learning: Track and optimize based on common patterns

🧪 Test Cases

// High selectivity (1% match) - should be very fast
await brainy.search("developer", 10, {
  metadata: { rare_certification: "specific_cert" }
})

// Medium selectivity (10% match) - should be moderately fast  
await brainy.search("developer", 10, {
  metadata: { level: "senior" }
})

// Low selectivity (50% match) - should use post-filtering
await brainy.search("developer", 10, {
  metadata: { active: true }
})

📊 Monitoring

Add performance logging to track improvements:

const start = Date.now()
const selectivity = candidateIds.length / totalItems
const efMultiplier = getEfMultiplier(selectivity)

console.log(`Filter selectivity: ${(selectivity * 100).toFixed(1)}%, ef multiplier: ${efMultiplier}`)

// After search
console.log(`Search completed in ${Date.now() - start}ms`)

🚨 Known Issues to Fix

Issue 1: HNSWIndexOptimized Filtering Bug

Status: Identified but not fixed
Problem: HNSWIndexOptimized.search() method signature was missing filter parameter
Solution: Already added filter parameter, but needs verification

Issue 2: Method Binding

Problem: TypeScript might not be calling overridden methods correctly
Solution: Verify method resolution and binding

🎉 Success Criteria

• Filtered search performance improves by 50-90% based on selectivity
• Filtering accuracy remains 100% correct
• No breaking changes to existing API
• Performance monitoring shows expected improvements
• All existing tests continue to pass

🧠 Additional Optimization: LRU Cache for Metadata Indexes

Priority: MEDIUM | Complexity: Low | Est. Time: 1-2 hours

The Opportunity

Add LRU caching to metadata indexes similar to HNSW index caching:

// Reuse existing infrastructure
this.metadataCache = new LRUCache<string, any>({
  maxSize: config.maxCacheSize ?? 1000,
  ttl: config.cacheTTL ?? 300000  // 5 minutes
})

// Cache field indexes and value chunks
const cachedFieldIndex = this.metadataCache.get(`field_${field}`)

Expected Benefits

• 10-100x faster repeated filter discovery
• Zero latency for common filter UI operations
• 90% code reuse from existing HNSW cache system
• Automatic learning of usage patterns

Implementation

1. Extend existing LRU cache to metadata indexes
2. Cache field indexes (field_category.json)
3. Cache hot value chunks (category_electronics_chunk0.json)
4. Add cache invalidation on metadata updates
5. Reuse existing cache statistics and monitoring

📝 Files to Modify

1. /home/dpsifr/Projects/brainy/src/brainyData.ts (selectivity calculation)
2. /home/dpsifr/Projects/brainy/src/hnsw/hnswIndex.ts (dynamic ef multiplier)
3. /home/dpsifr/Projects/brainy/src/hnsw/optimizedHNSWIndex.ts (forward selectivity)

🔄 Rollback Plan

If performance optimization causes issues:

1. Revert to fixed 3x multiplier
2. Feature flag the optimization for gradual rollout
3. Add configuration option to disable dynamic multiplier

---

This optimization will make metadata filtering 50-90% faster while maintaining the same powerful querying capabilities! 🚀