# Vector Dimension Standardization

## Overview

As of version 0.18.0, Brainy has standardized all vector dimensions to 512, which is the dimension used by the Universal Sentence Encoder. This change ensures consistency between data insertion, storage, and search operations, eliminating potential dimension mismatch issues.

## Changes Made

1. **Fixed Dimension Value**: Vector dimensions are now fixed at 512 throughout the codebase.
2. **Removed Configuration Option**: The `dimensions` configuration option has been removed from `BrainyDataConfig`.
3. **Consistent Validation**: All vectors are validated to ensure they have exactly 512 dimensions.

## Rationale

Previously, vector dimensions were configurable, which could lead to mismatches between:
- Vectors stored in the database
- The expected dimensions in the HNSW index
- Vectors generated by the embedding function

These mismatches could cause search functionality to break, as vectors with different dimensions would be skipped during initialization.

By standardizing all vectors to 512 dimensions (matching the Universal Sentence Encoder's output), we ensure that:
- All vectors in the database have consistent dimensions
- The HNSW index always works with vectors of the expected size
- Search queries always match the dimension of stored vectors

## Impact on Existing Code

### Breaking Changes

- The `dimensions` property in `BrainyDataConfig` has been removed
- Attempting to add vectors with dimensions other than 512 will throw an error
- Existing data with non-512 dimensions will be skipped during initialization

### Migration

If you have existing data with dimensions other than 512, you can use the provided `fix-dimension-mismatch.js` script to re-embed your data with the correct dimensions:

```bash
node fix-dimension-mismatch.js
```

This script:
1. Creates a backup of your existing data
2. Re-embeds all nouns using the Universal Sentence Encoder (512 dimensions)
3. Recreates all verb relationships
4. Verifies that search functionality works correctly

## Best Practices

- Always use the built-in embedding function for text data, which will automatically produce 512-dimensional vectors
- If you're creating vectors manually, ensure they have exactly 512 dimensions
- When migrating from previous versions, run the `fix-dimension-mismatch.js` script to ensure all data has consistent dimensions

## Technical Details

The Universal Sentence Encoder model produces 512-dimensional vectors by default. This is now the standard dimension for all vectors in Brainy, ensuring consistency across all operations.

This standardization resolves dimension mismatch issues that could previously cause search functionality to break.