David Snelling
1539ba74de
- Deleted the following obsolete files: - `CHANGES.md`, `changes-summary.md`, `CHANGES_SUMMARY.md`: Contained redundant or outdated change logs and implementation summaries. - `COMPATIBILITY.md`: Detailed compatibility behavior no longer relevant after environment detection updates. - `fix-documentation.md`: Addressed a resolved issue regarding `process.memoryUsage` errors in testing. - `DIMENSION_MISMATCH_SUMMARY.md`: Provided a legacy summary of resolved embedding dimension mismatch issues. - `demo.md`: Documented an outdated demo process for testing Brainy features. - `CONCURRENCY_IMPLEMENTATION_SUMMARY.md`: Summarized already-documented concurrency features. - `IMPLEMENTATION_SUMMARY.md`: Detailed an obsolete implementation of optional model bundling. - Purpose: - Streamline and declutter archive by removing redundant or outdated documentation. - Align repository with current feature set and documentation standards.
59 lines
2.7 KiB
Markdown
59 lines
2.7 KiB
Markdown
# Vector Dimension Standardization
|
|
|
|
## Overview
|
|
|
|
As of version 0.18.0, Brainy has standardized all vector dimensions to 512, which is the dimension used by the Universal Sentence Encoder. This change ensures consistency between data insertion, storage, and search operations, eliminating potential dimension mismatch issues.
|
|
|
|
## Changes Made
|
|
|
|
1. **Fixed Dimension Value**: Vector dimensions are now fixed at 512 throughout the codebase.
|
|
2. **Removed Configuration Option**: The `dimensions` configuration option has been removed from `BrainyDataConfig`.
|
|
3. **Consistent Validation**: All vectors are validated to ensure they have exactly 512 dimensions.
|
|
|
|
## Rationale
|
|
|
|
Previously, vector dimensions were configurable, which could lead to mismatches between:
|
|
- Vectors stored in the database
|
|
- The expected dimensions in the HNSW index
|
|
- Vectors generated by the embedding function
|
|
|
|
These mismatches could cause search functionality to break, as vectors with different dimensions would be skipped during initialization.
|
|
|
|
By standardizing all vectors to 512 dimensions (matching the Universal Sentence Encoder's output), we ensure that:
|
|
- All vectors in the database have consistent dimensions
|
|
- The HNSW index always works with vectors of the expected size
|
|
- Search queries always match the dimension of stored vectors
|
|
|
|
## Impact on Existing Code
|
|
|
|
### Breaking Changes
|
|
|
|
- The `dimensions` property in `BrainyDataConfig` has been removed
|
|
- Attempting to add vectors with dimensions other than 512 will throw an error
|
|
- Existing data with non-512 dimensions will be skipped during initialization
|
|
|
|
### Migration
|
|
|
|
If you have existing data with dimensions other than 512, you can use the provided `fix-dimension-mismatch.js` script to re-embed your data with the correct dimensions:
|
|
|
|
```bash
|
|
node fix-dimension-mismatch.js
|
|
```
|
|
|
|
This script:
|
|
1. Creates a backup of your existing data
|
|
2. Re-embeds all nouns using the Universal Sentence Encoder (512 dimensions)
|
|
3. Recreates all verb relationships
|
|
4. Verifies that search functionality works correctly
|
|
|
|
## Best Practices
|
|
|
|
- Always use the built-in embedding function for text data, which will automatically produce 512-dimensional vectors
|
|
- If you're creating vectors manually, ensure they have exactly 512 dimensions
|
|
- When migrating from previous versions, run the `fix-dimension-mismatch.js` script to ensure all data has consistent dimensions
|
|
|
|
## Technical Details
|
|
|
|
The Universal Sentence Encoder model produces 512-dimensional vectors by default. This is now the standard dimension for all vectors in Brainy, ensuring consistency across all operations.
|
|
|
|
This standardization resolves dimension mismatch issues that could previously cause search functionality to break.
|