soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions
brainy/docs/technical/VECTOR_DIMENSION_STANDARDIZATION.md
David Snelling 79df44351c **chore: remove outdated changelog and summary documents** 
- Deleted `CHANGES.md`, `CHANGES_SUMMARY.md`, `CONCURRENCY_ANALYSIS.md`, `CONCURRENCY_IMPLEMENTATION_SUMMARY.md`, and related developer documentation files.
- Removed redundant or legacy content no longer aligned with the current codebase and workflows.
- Updated repository to reflect streamlined documentation approach, reducing clutter and improving maintainability.

**Purpose**: Simplify and declutter repository by removing obsolete documentation files, ensuring it remains focused and relevant.
2025-07-30 11:51:39 -07:00

2.7 KiB
Raw Blame History

Vector Dimension Standardization

Overview

As of version 0.18.0, Brainy has standardized all vector dimensions to 512, which is the dimension used by the Universal Sentence Encoder. This change ensures consistency between data insertion, storage, and search operations, eliminating potential dimension mismatch issues.

Changes Made

  1. Fixed Dimension Value: Vector dimensions are now fixed at 512 throughout the codebase.
  2. Removed Configuration Option: The dimensions configuration option has been removed from BrainyDataConfig.
  3. Consistent Validation: All vectors are validated to ensure they have exactly 512 dimensions.

Rationale

Previously, vector dimensions were configurable, which could lead to mismatches between:

  • Vectors stored in the database
  • The expected dimensions in the HNSW index
  • Vectors generated by the embedding function

These mismatches could cause search functionality to break, as vectors with different dimensions would be skipped during initialization.

By standardizing all vectors to 512 dimensions (matching the Universal Sentence Encoder's output), we ensure that:

  • All vectors in the database have consistent dimensions
  • The HNSW index always works with vectors of the expected size
  • Search queries always match the dimension of stored vectors

Impact on Existing Code

Breaking Changes

  • The dimensions property in BrainyDataConfig has been removed
  • Attempting to add vectors with dimensions other than 512 will throw an error
  • Existing data with non-512 dimensions will be skipped during initialization

Migration

If you have existing data with dimensions other than 512, you can use the provided fix-dimension-mismatch.js script to re-embed your data with the correct dimensions:

node fix-dimension-mismatch.js

This script:

  1. Creates a backup of your existing data
  2. Re-embeds all nouns using the Universal Sentence Encoder (512 dimensions)
  3. Recreates all verb relationships
  4. Verifies that search functionality works correctly

Best Practices

  • Always use the built-in embedding function for text data, which will automatically produce 512-dimensional vectors
  • If you're creating vectors manually, ensure they have exactly 512 dimensions
  • When migrating from previous versions, run the fix-dimension-mismatch.js script to ensure all data has consistent dimensions

Technical Details

The Universal Sentence Encoder model produces 512-dimensional vectors by default. This is now the standard dimension for all vectors in Brainy, ensuring consistency across all operations.

For more information about the dimension mismatch issue and its resolution, see DIMENSION_MISMATCH_SUMMARY.md.