brainy/PERFORMANCE_OPTIMIZATION_TODO.md

# 🚀 Metadata Filtering Performance Optimization

**Priority: HIGH** | **Complexity: Medium** | **Est. Time: 2-3 hours**

## 🎯 The Issue

Current metadata filtering has a **300-400% search overhead** due to a fixed 3x ef multiplier in HNSW search, regardless of filter selectivity.

### Performance Analysis
- **No filtering**: 31.46ms average
- **Simple filter**: 150.82ms average (**379% overhead**)  
- **Complex filter**: 153.05ms average (**386% overhead**)

### Root Cause
```typescript
// Current implementation (inefficient)
// File: src/hnsw/hnswIndex.ts:377
const ef = filter ? Math.max(this.config.efSearch * 3, k * 3) : Math.max(this.config.efSearch, k)
```

The **fixed 3x multiplier** is used for ALL filtered searches, whether the filter matches 1% or 90% of items.

## 🛠️ Solution: Dynamic ef Multiplier

### 1. Calculate Filter Selectivity
```typescript
// Add to searchByNounTypes method
const selectivity = candidateIds.length / totalItems
```

### 2. Dynamic Multiplier Strategy
```typescript
function getEfMultiplier(selectivity: number): number {
  if (selectivity <= 0.01) return 1.1    // 1% match: minimal overhead
  if (selectivity <= 0.05) return 1.3    // 5% match: small overhead  
  if (selectivity <= 0.15) return 1.6    // 15% match: medium overhead
  if (selectivity <= 0.30) return 2.0    // 30% match: higher overhead
  return 1.0                             // 30%+ match: no overhead (post-filter)
}
```

### 3. Implementation Location

**File**: `src/brainyData.ts`  
**Method**: `searchByNounTypes` (around line 2165)

```typescript
// Current code around line 2165:
if (hasMetadataFilter && this.metadataIndex) {
  const candidateIds = await this.metadataIndex.getIdsForCriteria(options.metadata)
  
  // ADD THIS: Calculate selectivity
  const totalItems = this.index.size()
  const selectivity = candidateIds.length / totalItems
  const efMultiplier = getEfMultiplier(selectivity)
  
  // Store efMultiplier for use in HNSW search
}
```

**File**: `src/hnsw/hnswIndex.ts`  
**Method**: `search` (around line 377)

```typescript
// Replace this line:
const ef = filter ? Math.max(this.config.efSearch * 3, k * 3) : Math.max(this.config.efSearch, k)

// With dynamic calculation:
const ef = filter && filterSelectivity 
  ? Math.max(this.config.efSearch * filterSelectivity.multiplier, k)
  : Math.max(this.config.efSearch, k)
```

## 📈 Expected Performance Improvements

| Filter Selectivity | Current Overhead | Expected Overhead | Improvement |
|-------------------|------------------|-------------------|-------------|
| 1% (high selectivity) | 379% | 50% | **85% faster** |
| 5% (medium selectivity) | 379% | 80% | **70% faster** |
| 15% (low selectivity) | 379% | 150% | **50% faster** |
| 30%+ (very low selectivity) | 379% | 10% | **90% faster** |

## 🔧 Implementation Steps

### Step 1: Add Selectivity Calculation
1. Modify `searchByNounTypes` to calculate selectivity
2. Pass selectivity info to HNSW search method
3. Create `getEfMultiplier` helper function

### Step 2: Update HNSW Search
1. Modify `HNSWIndex.search` to accept selectivity parameter
2. Update `HNSWIndexOptimized.search` to forward selectivity  
3. Replace fixed 3x multiplier with dynamic calculation

### Step 3: Test Performance
1. Run performance benchmarks with different selectivity scenarios
2. Verify filtering accuracy is maintained
3. Test edge cases (empty results, 100% selectivity)

### Step 4: Optional Enhancements
1. **Index-First Strategy**: For <5% selectivity, search only candidate vectors
2. **Post-Filter Strategy**: For >50% selectivity, search all then filter
3. **Query Pattern Learning**: Track and optimize based on common patterns

## 🧪 Test Cases

```typescript
// High selectivity (1% match) - should be very fast
await brainy.search("developer", 10, {
  metadata: { rare_certification: "specific_cert" }
})

// Medium selectivity (10% match) - should be moderately fast  
await brainy.search("developer", 10, {
  metadata: { level: "senior" }
})

// Low selectivity (50% match) - should use post-filtering
await brainy.search("developer", 10, {
  metadata: { active: true }
})
```

## 📊 Monitoring

Add performance logging to track improvements:

```typescript
const start = Date.now()
const selectivity = candidateIds.length / totalItems
const efMultiplier = getEfMultiplier(selectivity)

console.log(`Filter selectivity: ${(selectivity * 100).toFixed(1)}%, ef multiplier: ${efMultiplier}`)

// After search
console.log(`Search completed in ${Date.now() - start}ms`)
```

## 🚨 Known Issues to Fix

### Issue 1: HNSWIndexOptimized Filtering Bug
**Status**: Identified but not fixed  
**Problem**: `HNSWIndexOptimized.search()` method signature was missing filter parameter  
**Solution**: Already added filter parameter, but needs verification

### Issue 2: Method Binding
**Problem**: TypeScript might not be calling overridden methods correctly  
**Solution**: Verify method resolution and binding

## 🎉 Success Criteria

- [ ] Filtered search performance improves by 50-90% based on selectivity
- [ ] Filtering accuracy remains 100% correct
- [ ] No breaking changes to existing API
- [ ] Performance monitoring shows expected improvements
- [ ] All existing tests continue to pass

## 🧠 Additional Optimization: LRU Cache for Metadata Indexes

**Priority: MEDIUM** | **Complexity: Low** | **Est. Time: 1-2 hours**

### The Opportunity
Add LRU caching to metadata indexes similar to HNSW index caching:

```typescript
// Reuse existing infrastructure
this.metadataCache = new LRUCache<string, any>({
  maxSize: config.maxCacheSize ?? 1000,
  ttl: config.cacheTTL ?? 300000  // 5 minutes
})

// Cache field indexes and value chunks
const cachedFieldIndex = this.metadataCache.get(`field_${field}`)
```

### Expected Benefits
- **10-100x faster** repeated filter discovery
- **Zero latency** for common filter UI operations  
- **90% code reuse** from existing HNSW cache system
- **Automatic learning** of usage patterns

### Implementation
1. Extend existing LRU cache to metadata indexes
2. Cache field indexes (`field_category.json`)
3. Cache hot value chunks (`category_electronics_chunk0.json`)
4. Add cache invalidation on metadata updates
5. Reuse existing cache statistics and monitoring

## 📝 Files to Modify

1. `src/brainyData.ts` (selectivity calculation)
2. `src/hnsw/hnswIndex.ts` (dynamic ef multiplier)
3. `src/hnsw/optimizedHNSWIndex.ts` (forward selectivity)

## 🔄 Rollback Plan

If performance optimization causes issues:
1. Revert to fixed 3x multiplier
2. Feature flag the optimization for gradual rollout
3. Add configuration option to disable dynamic multiplier

---

**This optimization will make metadata filtering 50-90% faster while maintaining the same powerful querying capabilities!** 🚀