brainy/METADATA_OPTIMIZATION_PROPOSAL.md

# Metadata Filtering Performance Optimization Proposal

## Problem Statement

Current metadata filtering has a **379-386% performance overhead** due to a fixed 3x ef multiplier in HNSW search, regardless of filter selectivity. This makes filtered searches 3-4x slower than necessary.

## Proposed Solution: Smart Filtering Strategy

### 1. Dynamic ef Multiplier Based on Selectivity

**Implementation Location**: `/src/brainyData.ts` in search method around line 2164

**Current Code**:
```typescript
// Fixed 3x multiplier regardless of filter selectivity
const ef = filter ? Math.max(this.config.efSearch * 3, k * 3) : Math.max(this.config.efSearch, k)
```

**Proposed Enhancement**:
```typescript
interface FilterSelectivity {
  candidateCount: number
  totalCount: number  
  selectivity: number // 0.0 - 1.0
  strategy: 'index-first' | 'hnsw-filter' | 'post-filter'
}

async calculateFilterStrategy(filter: MetadataFilter): Promise<FilterSelectivity> {
  const totalCount = this.index.getSize()
  
  if (!this.metadataIndex || totalCount === 0) {
    return { candidateCount: totalCount, totalCount, selectivity: 1.0, strategy: 'post-filter' }
  }
  
  const candidateIds = await this.metadataIndex.getIdsForCriteria(filter)
  const selectivity = candidateIds.length / totalCount
  
  let strategy: FilterSelectivity['strategy']
  if (selectivity <= 0.05) {
    strategy = 'index-first' // < 5% match: search only candidates
  } else if (selectivity <= 0.3) {
    strategy = 'hnsw-filter' // 5-30% match: HNSW with smart ef
  } else {
    strategy = 'post-filter' // > 30% match: search all, filter after
  }
  
  return { candidateCount: candidateIds.length, totalCount, selectivity, strategy }
}

// Dynamic ef calculation
calculateSmartEf(baseEf: number, k: number, selectivity: number, strategy: string): number {
  switch (strategy) {
    case 'index-first':
      return Math.max(k * 1.2, 20) // Minimal ef for candidate-only search
      
    case 'hnsw-filter':
      // Dynamic multiplier: high selectivity = lower multiplier
      const multiplier = 1.2 + (selectivity * 1.8) // 1.2x - 3.0x range
      return Math.max(baseEf * multiplier, k * multiplier)
      
    case 'post-filter':
      return Math.max(baseEf, k) // No filtering overhead
      
    default:
      return Math.max(baseEf, k)
  }
}
```

### 2. Index-First Search Implementation

**New Method**: Add to `/src/hnsw/hnswIndex.ts`

```typescript
/**
 * Search only within a pre-filtered set of candidates
 * Optimized for high-selectivity metadata filters
 */
async searchCandidatesOnly(
  queryVector: Vector,
  candidateIds: string[],
  k: number
): Promise<SearchResult<HNSWNoun>[]> {
  if (candidateIds.length === 0) return []
  
  const candidates: { noun: HNSWNoun; distance: number }[] = []
  
  // Calculate distances only for candidate nouns
  for (const id of candidateIds) {
    const noun = this.nouns.get(id)
    if (noun) {
      const distance = this.distanceFunction(queryVector, noun.vector)
      candidates.push({ noun, distance })
    }
  }
  
  // Sort by distance and return top k
  candidates.sort((a, b) => a.distance - b.distance)
  
  return candidates.slice(0, k).map(({ noun, distance }) => ({
    id: noun.id,
    score: 1 / (1 + distance),
    vector: noun.vector,
    metadata: noun.metadata
  }))
}
```

### 3. Smart Search Strategy Selection

**Enhanced Search Flow** in `/src/brainyData.ts`:

```typescript
async search<T>(query: string, k: number, options?: SearchOptions<T>) {
  // ... existing code for embedding generation ...
  
  if (hasMetadataFilter && this.metadataIndex) {
    // Calculate optimal strategy
    const filterStrategy = await this.calculateFilterStrategy(options.metadata)
    
    console.log(`Filter strategy: ${filterStrategy.strategy} (${(filterStrategy.selectivity * 100).toFixed(1)}% selectivity)`)
    
    switch (filterStrategy.strategy) {
      case 'index-first':
        // Direct candidate search - fastest for high selectivity
        const candidateIds = await this.metadataIndex.getIdsForCriteria(options.metadata)
        return this.index.searchCandidatesOnly(queryVector, candidateIds, k)
        
      case 'hnsw-filter':
        // Smart HNSW search with optimized ef
        const smartEf = this.calculateSmartEf(
          this.config.hnsw?.efSearch || 50, 
          k, 
          filterStrategy.selectivity,
          filterStrategy.strategy
        )
        return this.index.search(queryVector, k, undefined, smartEf)
        
      case 'post-filter':
        // Search all, then filter (best for low selectivity)
        const allResults = await this.index.search(queryVector, k * 3) // Get more results
        return this.applyMetadataFilter(allResults, options.metadata).slice(0, k)
    }
  }
  
  // ... existing non-filtered search code ...
}
```

## Expected Performance Improvements

### High Selectivity Filters (< 5% match rate)
- **Current**: 150ms with 3x ef multiplier
- **Optimized**: ~15ms with direct candidate search  
- **Improvement**: **90% faster**

### Medium Selectivity Filters (5-30% match rate)  
- **Current**: 150ms with fixed 3x multiplier
- **Optimized**: ~45-75ms with dynamic multiplier
- **Improvement**: **50-70% faster**

### Low Selectivity Filters (> 30% match rate)
- **Current**: 150ms with unnecessary filtering overhead
- **Optimized**: ~32ms with post-filtering
- **Improvement**: **80% faster**

## Implementation Plan

### Phase 1: Core Infrastructure (1-2 days)
1. Add selectivity calculation methods to `BrainyData`
2. Implement dynamic ef calculation logic
3. Add configuration options for strategy thresholds

### Phase 2: Search Strategy Implementation (2-3 days)
1. Implement `searchCandidatesOnly` in `HNSWIndex`
2. Add smart strategy selection to search method
3. Implement post-filtering strategy

### Phase 3: Configuration & Tuning (1 day)
1. Add configuration options for selectivity thresholds
2. Add performance monitoring and logging
3. Update documentation with optimization guidance

### Phase 4: Testing & Validation (1-2 days)  
1. Update performance tests with new benchmarks
2. Add test cases for different selectivity scenarios
3. Validate backward compatibility

## Configuration Options

```typescript
interface MetadataIndexConfig {
  // Existing options...
  
  // New optimization options
  selectivityThresholds?: {
    indexFirst: number // Default: 0.05 (5%)
    hnswFilter: number // Default: 0.3 (30%)
  }
  
  dynamicEf?: {
    enabled: boolean // Default: true
    minMultiplier: number // Default: 1.2
    maxMultiplier: number // Default: 3.0
  }
  
  performanceLogging?: {
    enabled: boolean // Default: false
    logSelectivity: boolean // Default: false
  }
}
```

## Backward Compatibility

- All existing APIs remain unchanged
- Default behavior maintains current functionality if optimizations disabled
- Gradual rollout possible with feature flags
- Performance improvements are opt-in via configuration

## Testing Strategy

```typescript
// New performance tests to add
describe('Metadata Search Optimization', () => {
  it('should use index-first for high selectivity filters', async () => {
    // Test with filter matching <5% of items
    // Verify strategy selection and performance improvement
  })
  
  it('should use dynamic ef for medium selectivity filters', async () => {
    // Test with filter matching 5-30% of items  
    // Verify ef calculation and performance improvement
  })
  
  it('should use post-filtering for low selectivity filters', async () => {
    // Test with filter matching >30% of items
    // Verify strategy selection and performance improvement  
  })
})
```

## Risk Assessment

**Low Risk Changes**:
- Configuration additions
- New method implementations  
- Performance logging

**Medium Risk Changes**:
- Strategy selection logic
- Dynamic ef calculation

**Mitigation Strategies**:
- Feature flags for gradual rollout
- Extensive testing with different dataset sizes
- Fallback to original behavior on errors
- Performance monitoring to detect regressions

## Success Metrics

1. **Performance Improvement**: 50-90% reduction in filtered search time
2. **Selectivity Accuracy**: Strategy selection matches actual selectivity
3. **Resource Usage**: No significant increase in memory or CPU
4. **Compatibility**: All existing tests continue to pass
5. **User Experience**: Improved search response times in real applications

## Future Enhancements

1. **Query Pattern Analysis**: Learn from search patterns to optimize strategy selection
2. **Index Warmup**: Pre-calculate common filter combinations  
3. **Parallel Candidate Search**: Multi-threaded candidate evaluation
4. **Index Compression**: Reduce storage overhead for large datasets
5. **Adaptive Thresholds**: Machine learning-based threshold optimization

This optimization will significantly improve the metadata filtering system's performance while maintaining the robust architecture and backward compatibility.