brainy/METADATA_OPTIMIZATION_PROPOSAL.md

Metadata Filtering Performance Optimization Proposal

Problem Statement

Current metadata filtering has a 379-386% performance overhead due to a fixed 3x ef multiplier in HNSW search, regardless of filter selectivity. This makes filtered searches 3-4x slower than necessary.

Proposed Solution: Smart Filtering Strategy

1. Dynamic ef Multiplier Based on Selectivity

Implementation Location: /src/brainyData.ts in search method around line 2164

Current Code:

// Fixed 3x multiplier regardless of filter selectivity
const ef = filter ? Math.max(this.config.efSearch * 3, k * 3) : Math.max(this.config.efSearch, k)

Proposed Enhancement:

interface FilterSelectivity {
  candidateCount: number
  totalCount: number  
  selectivity: number // 0.0 - 1.0
  strategy: 'index-first' | 'hnsw-filter' | 'post-filter'
}

async calculateFilterStrategy(filter: MetadataFilter): Promise<FilterSelectivity> {
  const totalCount = this.index.getSize()
  
  if (!this.metadataIndex || totalCount === 0) {
    return { candidateCount: totalCount, totalCount, selectivity: 1.0, strategy: 'post-filter' }
  }
  
  const candidateIds = await this.metadataIndex.getIdsForCriteria(filter)
  const selectivity = candidateIds.length / totalCount
  
  let strategy: FilterSelectivity['strategy']
  if (selectivity <= 0.05) {
    strategy = 'index-first' // < 5% match: search only candidates
  } else if (selectivity <= 0.3) {
    strategy = 'hnsw-filter' // 5-30% match: HNSW with smart ef
  } else {
    strategy = 'post-filter' // > 30% match: search all, filter after
  }
  
  return { candidateCount: candidateIds.length, totalCount, selectivity, strategy }
}

// Dynamic ef calculation
calculateSmartEf(baseEf: number, k: number, selectivity: number, strategy: string): number {
  switch (strategy) {
    case 'index-first':
      return Math.max(k * 1.2, 20) // Minimal ef for candidate-only search
      
    case 'hnsw-filter':
      // Dynamic multiplier: high selectivity = lower multiplier
      const multiplier = 1.2 + (selectivity * 1.8) // 1.2x - 3.0x range
      return Math.max(baseEf * multiplier, k * multiplier)
      
    case 'post-filter':
      return Math.max(baseEf, k) // No filtering overhead
      
    default:
      return Math.max(baseEf, k)
  }
}

2. Index-First Search Implementation

New Method: Add to /src/hnsw/hnswIndex.ts

/**
 * Search only within a pre-filtered set of candidates
 * Optimized for high-selectivity metadata filters
 */
async searchCandidatesOnly(
  queryVector: Vector,
  candidateIds: string[],
  k: number
): Promise<SearchResult<HNSWNoun>[]> {
  if (candidateIds.length === 0) return []
  
  const candidates: { noun: HNSWNoun; distance: number }[] = []
  
  // Calculate distances only for candidate nouns
  for (const id of candidateIds) {
    const noun = this.nouns.get(id)
    if (noun) {
      const distance = this.distanceFunction(queryVector, noun.vector)
      candidates.push({ noun, distance })
    }
  }
  
  // Sort by distance and return top k
  candidates.sort((a, b) => a.distance - b.distance)
  
  return candidates.slice(0, k).map(({ noun, distance }) => ({
    id: noun.id,
    score: 1 / (1 + distance),
    vector: noun.vector,
    metadata: noun.metadata
  }))
}

3. Smart Search Strategy Selection

Enhanced Search Flow in /src/brainyData.ts:

async search<T>(query: string, k: number, options?: SearchOptions<T>) {
  // ... existing code for embedding generation ...
  
  if (hasMetadataFilter && this.metadataIndex) {
    // Calculate optimal strategy
    const filterStrategy = await this.calculateFilterStrategy(options.metadata)
    
    console.log(`Filter strategy: ${filterStrategy.strategy} (${(filterStrategy.selectivity * 100).toFixed(1)}% selectivity)`)
    
    switch (filterStrategy.strategy) {
      case 'index-first':
        // Direct candidate search - fastest for high selectivity
        const candidateIds = await this.metadataIndex.getIdsForCriteria(options.metadata)
        return this.index.searchCandidatesOnly(queryVector, candidateIds, k)
        
      case 'hnsw-filter':
        // Smart HNSW search with optimized ef
        const smartEf = this.calculateSmartEf(
          this.config.hnsw?.efSearch || 50, 
          k, 
          filterStrategy.selectivity,
          filterStrategy.strategy
        )
        return this.index.search(queryVector, k, undefined, smartEf)
        
      case 'post-filter':
        // Search all, then filter (best for low selectivity)
        const allResults = await this.index.search(queryVector, k * 3) // Get more results
        return this.applyMetadataFilter(allResults, options.metadata).slice(0, k)
    }
  }
  
  // ... existing non-filtered search code ...
}

Expected Performance Improvements

High Selectivity Filters (< 5% match rate)

• Current: 150ms with 3x ef multiplier
• Optimized: ~15ms with direct candidate search
• Improvement: 90% faster

Medium Selectivity Filters (5-30% match rate)

• Current: 150ms with fixed 3x multiplier
• Optimized: ~45-75ms with dynamic multiplier
• Improvement: 50-70% faster

Low Selectivity Filters (> 30% match rate)

• Current: 150ms with unnecessary filtering overhead
• Optimized: ~32ms with post-filtering
• Improvement: 80% faster

Implementation Plan

Phase 1: Core Infrastructure (1-2 days)

1. Add selectivity calculation methods to BrainyData
2. Implement dynamic ef calculation logic
3. Add configuration options for strategy thresholds

Phase 2: Search Strategy Implementation (2-3 days)

1. Implement searchCandidatesOnly in HNSWIndex
2. Add smart strategy selection to search method
3. Implement post-filtering strategy

Phase 3: Configuration & Tuning (1 day)

1. Add configuration options for selectivity thresholds
2. Add performance monitoring and logging
3. Update documentation with optimization guidance

Phase 4: Testing & Validation (1-2 days)

1. Update performance tests with new benchmarks
2. Add test cases for different selectivity scenarios
3. Validate backward compatibility

Configuration Options

interface MetadataIndexConfig {
  // Existing options...
  
  // New optimization options
  selectivityThresholds?: {
    indexFirst: number // Default: 0.05 (5%)
    hnswFilter: number // Default: 0.3 (30%)
  }
  
  dynamicEf?: {
    enabled: boolean // Default: true
    minMultiplier: number // Default: 1.2
    maxMultiplier: number // Default: 3.0
  }
  
  performanceLogging?: {
    enabled: boolean // Default: false
    logSelectivity: boolean // Default: false
  }
}

Backward Compatibility

• All existing APIs remain unchanged
• Default behavior maintains current functionality if optimizations disabled
• Gradual rollout possible with feature flags
• Performance improvements are opt-in via configuration

Testing Strategy

// New performance tests to add
describe('Metadata Search Optimization', () => {
  it('should use index-first for high selectivity filters', async () => {
    // Test with filter matching <5% of items
    // Verify strategy selection and performance improvement
  })
  
  it('should use dynamic ef for medium selectivity filters', async () => {
    // Test with filter matching 5-30% of items  
    // Verify ef calculation and performance improvement
  })
  
  it('should use post-filtering for low selectivity filters', async () => {
    // Test with filter matching >30% of items
    // Verify strategy selection and performance improvement  
  })
})

Risk Assessment

Low Risk Changes:

• Configuration additions
• New method implementations
• Performance logging

Medium Risk Changes:

• Strategy selection logic
• Dynamic ef calculation

Mitigation Strategies:

• Feature flags for gradual rollout
• Extensive testing with different dataset sizes
• Fallback to original behavior on errors
• Performance monitoring to detect regressions

Success Metrics

1. Performance Improvement: 50-90% reduction in filtered search time
2. Selectivity Accuracy: Strategy selection matches actual selectivity
3. Resource Usage: No significant increase in memory or CPU
4. Compatibility: All existing tests continue to pass
5. User Experience: Improved search response times in real applications

Future Enhancements

1. Query Pattern Analysis: Learn from search patterns to optimize strategy selection
2. Index Warmup: Pre-calculate common filter combinations
3. Parallel Candidate Search: Multi-threaded candidate evaluation
4. Index Compression: Reduce storage overhead for large datasets
5. Adaptive Thresholds: Machine learning-based threshold optimization

This optimization will significantly improve the metadata filtering system's performance while maintaining the robust architecture and backward compatibility.