soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

refactor: simplify build system and improve model loading flexibility

Browse source 
- Remove Rollup bundling in favor of direct TypeScript compilation
- Move from bundled models to dynamic model loading with configurable paths
- Add Docker deployment examples and documentation
- Implement robust model loader with fallback mechanisms
- Update storage adapters for better cross-environment compatibility
- Add comprehensive tests for model loading and package installation
- Simplify package.json scripts and remove complex build configurations
- Clean up deprecated demo files and old bundling scripts

BREAKING CHANGE: Models are no longer bundled with the package. They are now loaded dynamically from CDN or custom paths.
This commit is contained in:
David Snelling 2025-08-05 16:09:30 -07:00
parent 89413ebec2
commit 52a43d51d4
51 changed files with 4835 additions and 8007 deletions
Download patch file Download diff file Expand all files Collapse all files

115
docs/technical/TECHNICAL_GUIDES.md
View file

@ -19,11 +19,13 @@ This document consolidates technical guides and documentation for specific aspec
## Vector Dimension Standardization
Brainy uses a standardized approach to vector dimensions to ensure consistency across the system. This section explains how vector dimensions are handled and standardized.
Brainy uses a standardized approach to vector dimensions to ensure consistency across the system. This section explains
how vector dimensions are handled and standardized.
### Default Dimensions
The default dimension for vectors in Brainy is 512, which is the output dimension of the Universal Sentence Encoder (USE) model used for text embeddings.
The default dimension for vectors in Brainy is 512, which is the output dimension of the Universal Sentence Encoder (
USE) model used for text embeddings.
### Dimension Validation
@ -55,7 +57,9 @@ When changing the embedding model or dimension size, you need to:
### Standardization Changes
As of version 0.18.0, Brainy has standardized all vector dimensions to 512, which is the dimension used by the Universal Sentence Encoder. This change ensures consistency between data insertion, storage, and search operations, eliminating potential dimension mismatch issues.
As of version 0.18.0, Brainy has standardized all vector dimensions to 512, which is the dimension used by the Universal
Sentence Encoder. This change ensures consistency between data insertion, storage, and search operations, eliminating
potential dimension mismatch issues.
#### Changes Made
@ -66,13 +70,16 @@ As of version 0.18.0, Brainy has standardized all vector dimensions to 512, whic
#### Rationale
Previously, vector dimensions were configurable, which could lead to mismatches between:
- Vectors stored in the database
- The expected dimensions in the HNSW index
- Vectors generated by the embedding function
These mismatches could cause search functionality to break, as vectors with different dimensions would be skipped during initialization.
These mismatches could cause search functionality to break, as vectors with different dimensions would be skipped during
initialization.
By standardizing all vectors to 512 dimensions (matching the Universal Sentence Encoder's output), we ensure that:
- All vectors in the database have consistent dimensions
- The HNSW index always works with vectors of the expected size
- Search queries always match the dimension of stored vectors
@ -85,13 +92,15 @@ By standardizing all vectors to 512 dimensions (matching the Universal Sentence
#### Migration
If you have existing data with dimensions other than 512, you can use the provided `fix-dimension-mismatch.js` script to re-embed your data with the correct dimensions:
If you have existing data with dimensions other than 512, you can use the provided `fix-dimension-mismatch.js` script to
re-embed your data with the correct dimensions:
```bash
node fix-dimension-mismatch.js
```
This script:
1. Creates a backup of your existing data
2. Re-embeds all nouns using the Universal Sentence Encoder (512 dimensions)
3. Recreates all verb relationships
@ -101,20 +110,25 @@ This script:
- Always use the built-in embedding function for text data, which will automatically produce 512-dimensional vectors
- If you're creating vectors manually, ensure they have exactly 512 dimensions
- When migrating from previous versions, run the `fix-dimension-mismatch.js` script to ensure all data has consistent dimensions
- When migrating from previous versions, run the `fix-dimension-mismatch.js` script to ensure all data has consistent
dimensions
## Dimension Mismatch Issue
This section summarizes a dimension mismatch issue that occurred in Brainy and provides recommendations for handling similar issues.
This section summarizes a dimension mismatch issue that occurred in Brainy and provides recommendations for handling
similar issues.
### What Happened
The search functionality in Brainy stopped working because of a dimension mismatch between stored vectors and the expected dimensions in the current version of the codebase:
The search functionality in Brainy stopped working because of a dimension mismatch between stored vectors and the
expected dimensions in the current version of the codebase:
1. **Previous State**: The system was using vectors with 3 dimensions.
2. **Current State**: The system now expects 512-dimensional vectors from the Universal Sentence Encoder.
3. **Code Change**: Recent updates introduced dimension validation during initialization, which skips vectors with mismatched dimensions.
4. **Result**: During initialization, vectors with 3 dimensions were skipped, resulting in an empty search index and no search results.
3. **Code Change**: Recent updates introduced dimension validation during initialization, which skips vectors with
mismatched dimensions.
4. **Result**: During initialization, vectors with 3 dimensions were skipped, resulting in an empty search index and no
search results.
### Root Cause Analysis
@ -142,7 +156,8 @@ The root cause was identified by examining the codebase:
return new Array(512).fill(0)
```
This indicates that the system previously used 3-dimensional vectors, but after the update, it expects 512-dimensional vectors. The existing data was not migrated, causing the search functionality to break.
This indicates that the system previously used 3-dimensional vectors, but after the update, it expects 512-dimensional
vectors. The existing data was not migrated, causing the search functionality to break.
### Solution Implemented
@ -151,13 +166,14 @@ We created and tested a fix script (`fix-dimension-mismatch.js`) that:
1. Creates a backup of the existing data
2. Reads all noun files directly from the filesystem
3. For each noun:
- Extracts text from metadata
- Deletes the existing noun
- Re-adds the noun with the same ID but using the current embedding function
- Extracts text from metadata
- Deletes the existing noun
- Re-adds the noun with the same ID but using the current embedding function
4. Recreates all verb relationships between the re-embedded nouns
5. Verifies that search works by performing a test search
The script successfully fixed the issue by re-embedding all data with the correct dimensions, and search functionality was restored.
The script successfully fixed the issue by re-embedding all data with the correct dimensions, and search functionality
was restored.
### Production Recommendations
@ -191,7 +207,8 @@ To prevent similar issues in the future:
## Production Migration Guide
This section provides a comprehensive guide for migrating Brainy databases in production environments, particularly when dealing with dimension changes or other breaking changes.
This section provides a comprehensive guide for migrating Brainy databases in production environments, particularly when
dealing with dimension changes or other breaking changes.
### Preparation
@ -243,7 +260,8 @@ After completing the migration:
## Threading Implementation
Brainy includes comprehensive multithreading support to improve performance across all environments. This section explains how threading is implemented and used in the project.
Brainy includes comprehensive multithreading support to improve performance across all environments. This section
explains how threading is implemented and used in the project.
### Threading Architecture
@ -261,7 +279,8 @@ Brainy uses a unified threading approach that adapts to the environment it's run
2. **Browser**: Uses Web Workers API
3. **Fallback**: Executes on the main thread when neither Worker Threads nor Web Workers are available
This implementation ensures that compute-intensive operations (like embedding generation and vector calculations) can be performed efficiently without blocking the main thread, while maintaining compatibility across all environments.
This implementation ensures that compute-intensive operations (like embedding generation and vector calculations) can be
performed efficiently without blocking the main thread, while maintaining compatibility across all environments.
### Environment Detection
@ -325,6 +344,7 @@ export function executeInThread<T>(fnString: string, args: any): Promise<T> {
```
This function:
1. Checks if it's running in Node.js and uses Worker Threads if available
2. Checks if it's running in a browser and uses Web Workers if available
3. Falls back to executing on the main thread if neither is available
@ -343,6 +363,7 @@ function executeInNodeWorker<T>(fnString: string, args: any): Promise<T> {
```
Key optimizations:
- Worker pool to reuse workers and minimize overhead
- Dynamic imports with the `node:` protocol prefix
- Error handling and cleanup
@ -361,6 +382,7 @@ function executeInWebWorker<T>(fnString: string, args: any): Promise<T> {
```
Key features:
- Creates workers using Blob URLs
- Proper cleanup of resources (terminating workers and revoking URLs)
- Error handling
@ -411,7 +433,8 @@ const db = new BrainyData({
})
```
The threading implementation is used throughout Brainy, particularly for compute-intensive operations like embedding generation:
The threading implementation is used throughout Brainy, particularly for compute-intensive operations like embedding
generation:
```typescript
export function createThreadedEmbeddingFunction(
@ -441,33 +464,12 @@ The threading implementation includes:
### Testing
Two test scripts are provided to verify the threading implementation:
1. `demo/test-browser-worker.html`: Tests the threading implementation in a browser environment
2. `demo/test-fallback.html`: Tests the fallback mechanism when threading is not available
To run these tests:
1. Build the project: `npm run build`
2. Start a local server: `npx http-server`
3. Open the test pages in a browser:
- http://localhost:8080/demo/test-browser-worker.html
- http://localhost:8080/demo/test-fallback.html
### Compatibility
The threading implementation has been tested and works in:
- Node.js 24+ (using Worker Threads)
- Modern browsers (using Web Workers):
- Chrome
- Firefox
- Safari
- Edge
- Environments without threading support (using fallback mechanism)
// Demo references removed: The demo is now maintained in the separate @soulcraft/demos project.
## Storage Testing
This section provides guidance on testing storage adapters in Brainy, including file system storage, OPFS storage, and S3-compatible storage.
This section provides guidance on testing storage adapters in Brainy, including file system storage, OPFS storage, and
S3-compatible storage.
### Testing Approach
@ -529,7 +531,8 @@ npm test -- tests/filesystem-storage.test.ts
## Scaling Strategy
Brainy is designed to handle datasets of various sizes, from small collections to large-scale deployments. This section outlines strategies for scaling Brainy to handle terabyte-scale data.
Brainy is designed to handle datasets of various sizes, from small collections to large-scale deployments. This section
outlines strategies for scaling Brainy to handle terabyte-scale data.
### Scaling Challenges
@ -594,7 +597,8 @@ const nodeDb = new BrainyData({
Combining multiple techniques for optimal performance:
- **Product Quantization**: Compress vectors to reduce memory usage
- **Multi-Tier Storage**: Use fast storage for frequently accessed data and slower storage for less frequently accessed data
- **Multi-Tier Storage**: Use fast storage for frequently accessed data and slower storage for less frequently accessed
data
- **Hierarchical Clustering**: Group similar vectors together for more efficient search
Implementation:
@ -774,7 +778,8 @@ const nounsWithTag = allNouns.filter(noun =>
## Model Loading
This section explains how model loading works in Brainy, particularly for the Universal Sentence Encoder (USE) model used for text embeddings.
This section explains how model loading works in Brainy, particularly for the Universal Sentence Encoder (USE) model
used for text embeddings.
### Model Loading Process
@ -842,19 +847,19 @@ To improve performance, Brainy caches the loaded model:
Common issues with model loading and their solutions:
1. **Memory Issues**: If you encounter memory issues, try:
- Using the simplified model (`useSimplifiedModel: true`)
- Loading the model in a separate thread (`useThreading: true`)
- Reducing the batch size for embedding operations
- Using the simplified model (`useSimplifiedModel: true`)
- Loading the model in a separate thread (`useThreading: true`)
- Reducing the batch size for embedding operations
2. **Loading Failures**: If the model fails to load, try:
- Checking network connectivity (for TensorFlow Hub loading)
- Verifying the local model path (for local loading)
- Using the bundled model as a fallback
- Checking network connectivity (for TensorFlow Hub loading)
- Verifying the local model path (for local loading)
- Using the bundled model as a fallback
3. **Performance Issues**: If embedding is slow, try:
- Using threaded embedding (`useThreading: true`)
- Increasing the number of workers (`maxWorkers`)
- Using batch embedding for multiple items
- Using threaded embedding (`useThreading: true`)
- Increasing the number of workers (`maxWorkers`)
- Using batch embedding for multiple items
### Best Practices