brainy/production-migration-guide.md

Production Migration Guide for Dimension Mismatch Issue

Root Cause Analysis

The search functionality in Brainy stopped working due to a dimension mismatch between stored vectors and the expected dimensions in the current version of the codebase.

What Happened

1. Dimension Change: The system previously used vectors with 3 dimensions, but now expects 512-dimensional vectors from the Universal Sentence Encoder.
2. Code Changes: Recent updates (around July 16, 2025) introduced dimension validation during initialization, which skips vectors with mismatched dimensions.
3. Result: During initialization, vectors with 3 dimensions were skipped, resulting in an empty search index and no search results.

Impact Assessment

1. Data Integrity: The original data is still intact in storage, but it cannot be used for search operations due to the dimension mismatch.
2. Functionality: Search operations return no results, but other operations like adding new data still work correctly.
3. Scope: All existing data with 3-dimensional vectors is affected.

Production Migration Strategy

Option 1: Full Re-embedding (Recommended)

The most reliable solution is to re-embed all existing data using the current embedding function, as implemented in the fix-dimension-mismatch.js script.

Process:

1. Backup: Create a complete backup of the existing data.
2. Re-embed: Process each noun by extracting its text content and re-embedding it with the current embedding function.
3. Recreate Relationships: Recreate all verb relationships between the re-embedded nouns.
4. Verify: Test search functionality to ensure it works correctly.

Production Considerations:

1. Scheduling: Perform the migration during a maintenance window to minimize disruption.

2. Backup Strategy: Use a more robust backup mechanism for production:

   // Enhanced backup for production
   const backupDir = './brainy-data-backup-' + Date.now();
   const backupMetadata = {
     timestamp: Date.now(),
     reason: 'Dimension mismatch fix',
     originalDimensions: 3,
     newDimensions: 512,
     version: process.env.APP_VERSION || 'unknown'
   };
   
   // Save backup metadata
   await fs.promises.mkdir(backupDir, { recursive: true });
   await fs.promises.writeFile(
     path.join(backupDir, 'backup-metadata.json'), 
     JSON.stringify(backupMetadata, null, 2)
   );
   
   // Copy data
   await copyDirectory('./brainy-data', backupDir);
   

3. Batching: For large datasets, process nouns in batches to reduce memory usage:

   // Process nouns in batches
   const batchSize = 100;
   let processedCount = 0;
   
   for (let i = 0; i < files.length; i += batchSize) {
     const batch = files.slice(i, i + batchSize);
     console.log(`Processing batch ${Math.floor(i/batchSize) + 1} of ${Math.ceil(files.length/batchSize)}...`);
   
     // Process batch
     for (const file of batch) {
       // Process noun as in the original script
       // ...
     }
   
     processedCount += batch.length;
     console.log(`Progress: ${processedCount}/${files.length} nouns (${Math.round(processedCount/files.length*100)}%)`);
   }
   

4. Error Handling: Implement more robust error handling and recovery:

   // Enhanced error handling
   const failedNouns = [];
   const failedVerbs = [];
   
   // During noun processing
   try {
     // Process noun
   } catch (error) {
     console.error(`Error processing noun ${parsedNoun.id}:`, error);
     failedNouns.push({ id: parsedNoun.id, error: error.message });
     // Continue with next noun
   }
   
   // At the end of processing
   if (failedNouns.length > 0 || failedVerbs.length > 0) {
     const errorReport = {
       timestamp: Date.now(),
       failedNouns,
       failedVerbs
     };
   
     await fs.promises.writeFile(
       './migration-errors.json',
       JSON.stringify(errorReport, null, 2)
     );
   
     console.warn(`Migration completed with errors. See migration-errors.json for details.`);
   }
   

5. Monitoring: Add progress monitoring and reporting:

   // Progress monitoring
   const startTime = Date.now();
   const logProgress = (current, total, label) => {
     const percent = Math.round((current / total) * 100);
     const elapsed = (Date.now() - startTime) / 1000;
     const itemsPerSecond = current / elapsed;
     const estimatedTotal = total / itemsPerSecond;
     const remaining = estimatedTotal - elapsed;
   
     console.log(
       `${label}: ${current}/${total} (${percent}%) - ` +
       `Elapsed: ${formatTime(elapsed)}, ` +
       `Remaining: ${formatTime(remaining)}, ` +
       `Rate: ${itemsPerSecond.toFixed(2)} items/sec`
     );
   };
   
   // Format time helper
   const formatTime = (seconds) => {
     const hrs = Math.floor(seconds / 3600);
     const mins = Math.floor((seconds % 3600) / 60);
     const secs = Math.floor(seconds % 60);
     return `${hrs}h ${mins}m ${secs}s`;
   };
   

Option 2: Parallel Database Approach

For mission-critical systems where downtime must be minimized:

1. Create Parallel Database: Set up a new database instance with the correct dimensions.
2. Migrate in Background: Run the migration process against this new database while the original continues to serve requests.
3. Switch Over: Once migration is complete and verified, switch to the new database.

// Example of parallel database approach
async function parallelDatabaseMigration() {
  // Create new database with a different storage location
  const newDb = new BrainyData({
    storage: {
      forceFileSystemStorage: true,
      rootDirectory: './brainy-data-new'
    }
  });
  await newDb.init();
  
  // Migrate data to new database
  // (similar to fix-dimension-mismatch.js but using newDb)
  
  // Once complete, create a switch file
  await fs.promises.writeFile(
    './database-switch.json',
    JSON.stringify({
      readyToSwitch: true,
      newDatabasePath: './brainy-data-new',
      oldDatabasePath: './brainy-data',
      migrationCompleted: Date.now()
    })
  );
  
  console.log('Migration complete. Ready to switch to new database.');
}

Preventive Measures for Future Changes

1. Version Tracking for Vectors

Add version information to stored vectors to track the embedding model and dimensions:

// When adding a noun
const id = await db.add(text, {
  ...metadata,
  _vectorInfo: {
    dimensions: 512,
    model: 'UniversalSentenceEncoder',
    version: '1.0.0',
    createdAt: Date.now()
  }
});

2. Dimension Validation with Auto-Migration

Enhance the initialization process to automatically re-embed vectors with mismatched dimensions:

// In BrainyData.init()
for (const noun of nouns) {
  // Check if the vector dimensions match the expected dimensions
  if (noun.vector.length !== this._dimensions) {
    console.warn(
      `Found noun ${noun.id} with dimension mismatch: expected ${this._dimensions}, got ${noun.vector.length}`
    );
    
    // Get metadata to extract text
    const metadata = await this.storage.getMetadata(noun.id);
    if (metadata && (metadata.text || metadata.description)) {
      console.log(`Auto-migrating noun ${noun.id} to correct dimensions...`);
      
      // Extract text
      const text = metadata.text || metadata.description || `Noun ${noun.id}`;
      
      // Re-embed with current embedding function
      const newVector = await this.embeddingFunction(text);
      
      // Update the noun with new vector
      noun.vector = newVector;
      await this.storage.saveNoun(noun);
      
      console.log(`Successfully migrated noun ${noun.id} to ${this._dimensions} dimensions`);
    } else {
      console.warn(`Cannot auto-migrate noun ${noun.id}: no text found in metadata`);
      continue;
    }
  }
  
  // Add to index
  await this.index.addItem({
    id: noun.id,
    vector: noun.vector
  });
}

3. Regular Database Validation

Implement a regular validation process to check for dimension mismatches:

// database-validator.js
import { BrainyData } from './dist/brainyData.js';

async function validateDatabase() {
  const db = new BrainyData();
  await db.init();
  
  console.log('Validating database...');
  
  // Get all nouns
  const nouns = await db.getAllNouns();
  console.log(`Found ${nouns.length} nouns.`);
  
  // Check dimensions
  const dimensionMismatches = nouns.filter(noun => noun.vector.length !== 512);
  
  if (dimensionMismatches.length > 0) {
    console.warn(`Found ${dimensionMismatches.length} nouns with dimension mismatches.`);
    console.warn('Sample of mismatched nouns:');
    
    for (let i = 0; i < Math.min(5, dimensionMismatches.length); i++) {
      const noun = dimensionMismatches[i];
      console.warn(`Noun ${noun.id}: expected 512 dimensions, got ${noun.vector.length}`);
    }
    
    console.warn('Run fix-dimension-mismatch.js to fix these issues.');
  } else {
    console.log('All nouns have correct dimensions.');
  }
  
  // Check search functionality
  const searchResults = await db.searchText('test', 5);
  console.log(`Search test returned ${searchResults.length} results.`);
  
  console.log('Database validation complete.');
}

validateDatabase().catch(console.error);

4. Documentation and Change Management

1. Document Embedding Changes: Clearly document any changes to embedding functions or vector dimensions in release notes.
2. Migration Scripts: Include migration scripts with any release that changes vector dimensions.
3. Version Compatibility: Implement version compatibility checks in the codebase.

Conclusion

The dimension mismatch issue was caused by a change in the embedding function that increased vector dimensions from 3 to 512. The recommended solution is to re-embed all existing data using the current embedding function, which can be done using the provided fix-dimension-mismatch.js script with the enhancements suggested for production environments.

By implementing the preventive measures outlined above, you can avoid similar issues in the future and ensure smoother transitions when embedding functions or vector dimensions change.