soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

**feat(docs): add comprehensive documentation for model bundling and robust loading**

Browse source 
- Introduced new documentation files under `docs/`:
  - `model-bundling-analysis.md`: Provides detailed analysis of current, bundled, hybrid, and dynamic model loading approaches, including pros, cons, and recommendations.
  - `model-management.md`: Explains how Brainy manages Universal Sentence Encoder models, including setup, usage, and troubleshooting.
  - `optional-model-bundling.md`: Details the `@soulcraft/brainy-models` package for offline reliability with pre-bundled models.

- Added `src/utils/robustModelLoader.ts`:
  - Implements enhanced model loading with retry mechanisms, timeout handling, fallback URLs, and optional local model bundling.
  - Supports Node.js and browser environments with exponential backoff logic.

- Key Updates:
  - **Hybrid Loading Strategy**: Recommended for balancing reliability and flexibility via hybrid online/offline mechanisms.
  - **Enhanced Fallback Scenarios**: Robust loader improves network-dependent reliability for embedding workflows.
  - **Offline Reliability Support**: Optional model bundling eliminates dependency on external services, supporting air-gapped and edge environments.

**Purpose**: Introduce a hybrid model loading approach with robust options for
This commit is contained in:
David Snelling 2025-08-01 15:35:29 -07:00
parent 563b983fcc
commit 42571c5883
6 changed files with 1098 additions and 214 deletions
Download patch file Download diff file Expand all files Collapse all files

130
docs/guides/model-management.md Normal file
View file

@ -0,0 +1,130 @@
# Model Management Guide
This guide explains how to manage the TensorFlow Universal Sentence Encoder model used by Brainy for text embeddings.
## Overview
Brainy uses the Universal Sentence Encoder model from TensorFlow.js to convert text into vector embeddings. These
embeddings are essential for semantic search and other features.
The model is referenced through local configuration files that point to the TensorFlow Hub URL. This approach ensures
that:
1. The model is automatically downloaded when first needed
2. Subsequent uses leverage the cached model
3. The application works consistently across all environments (browser, Node.js, serverless, workers)
## Model Files
The model files are stored in the `models/sentence-encoder/` directory and include:
- `model.json` (~1KB) - The main model configuration file that references the TensorFlow Hub URL
- `group1-shard1of1.bin` (~2KB) - A sample embedding file (not the full model weights)
- `metadata.json` (~0.4KB) - Additional information about the model
**Important Note**: These are small reference files (total ~3KB), not the full model (~25MB). The full model is
downloaded automatically when first needed and then cached locally. This approach keeps the repository size small while
ensuring the model is available when needed.
These files should be checked into version control to ensure they're available in all environments.
## Setting Up Model Reference Files
The model reference files are generated using the `download-model.js` script:
```bash
node scripts/download-model.js
```
**Important**: This script does NOT download the full model (~25MB) locally. Instead, it creates small reference
files (~3KB total) that point to the TensorFlow Hub URL. The full model will be downloaded automatically when your
application first uses it, and then cached for future use.
### When to Run the Script
You should run this script in the following situations:
1. **Initial Setup**: When first setting up the project
2. **Model Updates**: When updating to a new version of the Universal Sentence Encoder
3. **Missing Files**: If the model reference files are missing or corrupted
The script will:
1. Load the model from TensorFlow Hub to verify it works (temporarily downloading it to memory)
2. Create a model.json file that references the TensorFlow Hub URL
3. Generate a sample embedding and save it as a small binary file
4. Create a metadata file with information about the model
### What to Expect
After running the script:
1. You'll see small files in the `models/sentence-encoder/` directory (total ~3KB)
2. When your application first uses the model, it will download the full model (~25MB) from TensorFlow Hub
3. The downloaded model will be cached locally for subsequent use
4. No further downloads will be needed unless the cache is cleared
## Using the Model
The model is automatically loaded by the `UniversalSentenceEncoder` class in `src/utils/embedding.ts`. The loading
process follows these steps:
1. Check if local model reference files exist
2. If they exist, load the model using the TensorFlow Hub URL referenced in the model.json file
3. The first time this happens, the full model will be downloaded and cached
4. Subsequent uses will use the cached model
## Environments
The model loading works across all environments:
- **Node.js**: Uses file system paths to load the model
- **Browser**: Uses relative URLs to load the model
- **Serverless/Workers**: Uses the embedded model files
## Best Practices
1. **Always check in model files**: The model files should be committed to version control
2. **Run tests after model updates**: Use `node test-model-loading.js` to verify the model works
3. **Update model during build**: If you prefer not to check in model files, run the download script as part of your
build process
## Troubleshooting
### Common Issues
#### "The model files are too small (only a few KB)"
This is expected behavior. The script creates small reference files (~3KB total), not the full model (~25MB). The full
model will be downloaded automatically when your application first uses it.
#### "Model not found or loading errors"
1. Verify the model reference files exist in `models/sentence-encoder/`
2. Run `node scripts/download-model.js` to generate fresh reference files
3. Check for errors in the console related to model loading
4. Ensure the model reference files are properly included in your deployment package
5. Make sure your application has internet access the first time it runs to download the full model
#### "Model download is slow"
The first time your application uses the model, it will download the full model (~25MB) from TensorFlow Hub. This may
take a few minutes depending on your internet connection. Subsequent uses will use the cached model and will be much
faster.
## Technical Details
The Universal Sentence Encoder model:
- Produces 512-dimensional embeddings
- Works with text in any language
- Is optimized for semantic similarity tasks
- Has a size of approximately 25MB when fully downloaded
- Is downloaded from TensorFlow Hub on first use and then cached
Our approach of referencing the model via TensorFlow Hub URL provides several benefits:
1. Small package size (only reference files are included, not the full model)
2. Automatic caching for improved performance after first use
3. Consistent behavior across all environments
4. Always uses the correct model weights

364
docs/guides/optional-model-bundling.md Normal file
View file

@ -0,0 +1,364 @@
# Optional Model Bundling Package
## Overview
The `@soulcraft/brainy-models` package provides pre-bundled TensorFlow models for maximum reliability with the Brainy
vector database. This optional package eliminates network dependencies and ensures consistent performance by including
the complete Universal Sentence Encoder model (~25MB) locally.
## When to Use
### Use the Optional Model Bundling Package When:
- ✅ **Maximum Reliability Required**: Production applications that cannot tolerate network failures
- ✅ **Offline Environments**: Air-gapped systems or environments without internet access
- ✅ **Strict SLA Requirements**: Applications with stringent uptime and performance requirements
- ✅ **Edge Computing**: IoT devices and edge deployments with limited connectivity
- ✅ **Development Stability**: Development environments with unreliable internet connections
### Use Standard Online Loading When:
- ✅ **Package Size Matters**: Applications where the additional ~25MB is significant
- ✅ **Prototyping**: Quick development and testing scenarios
- ✅ **Reliable Internet**: Environments with consistent, fast internet connectivity
- ✅ **Infrequent Usage**: Applications that rarely generate embeddings
## Installation
```bash
# Install the optional model bundling package
npm install @soulcraft/brainy-models
```
## Quick Start
### Basic Usage with Brainy
```typescript
import Brainy from '@soulcraft/brainy'
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
// Create and load the bundled encoder
const bundledEncoder = new BundledUniversalSentenceEncoder({
verbose: true,
preferCompressed: false
})
await bundledEncoder.load()
// Use with Brainy
const brainy = new Brainy({
// Configure Brainy to use the bundled encoder
customEmbedding: async (texts) => {
return await bundledEncoder.embedToArrays(texts)
}
})
// Now use Brainy as normal - it will use the bundled model
await brainy.addDocument('doc1', 'This is a sample document')
const results = await brainy.search('sample text', { limit: 5 })
console.log('Search results:', results)
// Clean up
bundledEncoder.dispose()
```
### Advanced Configuration
```typescript
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
// High-reliability configuration
const encoder = new BundledUniversalSentenceEncoder({
verbose: true,
preferCompressed: false // Use full model for maximum accuracy
})
// Memory-optimized configuration
const memoryOptimizedEncoder = new BundledUniversalSentenceEncoder({
verbose: true,
preferCompressed: true // Use compressed model to save memory
})
```
## Comparison: Online vs Bundled Models
| Feature | Online Loading | Bundled Models |
|----------------------|--------------------|----------------|
| **Reliability** | Network dependent | 100% offline |
| **First load time** | 30-60 seconds | < 1 second |
| **Subsequent loads** | Cached (~1 second) | < 1 second |
| **Package size** | ~3KB | ~25MB |
| **Network required** | Yes (first time) | No |
| **Offline support** | Limited | Complete |
| **Memory usage** | Standard | Configurable |
| **Startup time** | Variable | Consistent |
## Model Variants
The bundled package includes multiple optimized variants:
### Original (Float32)
- **Size**: ~25MB
- **Accuracy**: Maximum
- **Memory**: High
- **Speed**: Fast
- **Use case**: Production applications requiring highest accuracy
### Float16 Compressed
- **Size**: ~12-15MB
- **Accuracy**: Very High (minimal loss)
- **Memory**: Medium
- **Speed**: Fast
- **Use case**: Balanced performance and size
### Int8 Quantized
- **Size**: ~6-8MB
- **Accuracy**: High (some loss acceptable)
- **Memory**: Low
- **Speed**: Medium
- **Use case**: Memory-constrained environments
## Integration Patterns
### Pattern 1: Direct Replacement
Replace the standard embedding approach with bundled models:
```typescript
// Before (online loading)
import Brainy from '@soulcraft/brainy'
const brainyOnline = new Brainy()
// After (bundled models)
import Brainy from '@soulcraft/brainy'
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
const bundledEncoder = new BundledUniversalSentenceEncoder()
await bundledEncoder.load()
const brainyBundled = new Brainy({
customEmbedding: async (texts) => await bundledEncoder.embedToArrays(texts)
})
```
### Pattern 2: Fallback Strategy
Use bundled models as a fallback when online loading fails:
```typescript
import Brainy from '@soulcraft/brainy'
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
async function createReliableBrainy() {
try {
// Try online loading first
const brainy = new Brainy()
await brainy.initialize() // This might fail due to network issues
return brainy
} catch (error) {
console.log('Online loading failed, using bundled models:', error.message)
// Fallback to bundled models
const encoder = new BundledUniversalSentenceEncoder({ verbose: true })
await encoder.load()
return new Brainy({
customEmbedding: async (texts) => await encoder.embedToArrays(texts)
})
}
}
const brainy = await createReliableBrainy()
```
### Pattern 3: Environment-Based Selection
Choose the approach based on the environment:
```typescript
import Brainy from '@soulcraft/brainy'
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
async function createEnvironmentOptimizedBrainy() {
const isProduction = process.env.NODE_ENV === 'production'
const isOffline = !navigator.onLine // Browser only
const requiresReliability = process.env.REQUIRE_MAX_RELIABILITY === 'true'
if (isProduction || isOffline || requiresReliability) {
// Use bundled models for maximum reliability
const encoder = new BundledUniversalSentenceEncoder({
verbose: !isProduction,
preferCompressed: process.env.MEMORY_CONSTRAINED === 'true'
})
await encoder.load()
return new Brainy({
customEmbedding: async (texts) => await encoder.embedToArrays(texts)
})
} else {
// Use online loading for development
return new Brainy()
}
}
```
## Performance Optimization
### Memory Management
```typescript
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
// For memory-constrained environments
const encoder = new BundledUniversalSentenceEncoder({
preferCompressed: true // Uses int8 quantized model
})
// Always dispose when done
encoder.dispose()
```
### Batch Processing
```typescript
// Process texts in batches for optimal performance
async function processLargeDataset(texts: string[]) {
const encoder = new BundledUniversalSentenceEncoder()
await encoder.load()
const batchSize = 32
const results = []
for (let i = 0; i < texts.length; i += batchSize) {
const batch = texts.slice(i, i + batchSize)
const embeddings = await encoder.embedToArrays(batch)
results.push(...embeddings)
}
encoder.dispose()
return results
}
```
## Troubleshooting
### Common Issues
#### Package Size Concerns
**Issue**: The bundled package is large (~25MB)
**Solutions**:
- Use compressed models: `preferCompressed: true`
- Consider if your use case truly requires maximum reliability
- Use online loading for development, bundled for production
#### Memory Usage
**Issue**: High memory usage with bundled models
**Solutions**:
- Use int8 quantized models
- Dispose of encoder instances when not needed
- Process data in smaller batches
#### Model Loading Errors
**Issue**: "Bundled model not found" error
**Solutions**:
```bash
# Navigate to the package directory and download models
cd node_modules/@soulcraft/brainy-models
npm run download-models
```
### Performance Tuning
For optimal performance:
1. **Choose the right variant**:
- Production: Original float32 model
- Balanced: Float16 compressed model
- Memory-limited: Int8 quantized model
2. **Manage memory properly**:
- Always call `dispose()` when done
- Use appropriate batch sizes
- Monitor memory usage in production
3. **Optimize for your use case**:
- High-throughput: Use original model with larger batches
- Low-memory: Use int8 model with smaller batches
- Balanced: Use float16 model with medium batches
## Migration Guide
### From Online Loading to Bundled Models
1. **Install the package**:
```bash
npm install @soulcraft/brainy-models
```
2. **Update your code**:
```typescript
// Before
import Brainy from '@soulcraft/brainy'
const originalBrainy = new Brainy()
// After
import Brainy from '@soulcraft/brainy'
import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models'
const modelEncoder = new BundledUniversalSentenceEncoder()
await modelEncoder.load()
const reliableBrainy = new Brainy({
customEmbedding: async (texts) => await modelEncoder.embedToArrays(texts)
})
```
3. **Test thoroughly**:
- Verify embeddings are generated correctly
- Check memory usage
- Test offline functionality
4. **Deploy with confidence**:
- No network dependencies
- Consistent performance
- Maximum reliability
## Best Practices
1. **Choose the Right Approach**:
- Use bundled models for production and critical applications
- Use online loading for development and prototyping
2. **Memory Management**:
- Always dispose of encoder instances
- Use compressed models when appropriate
- Monitor memory usage in production
3. **Error Handling**:
- Implement proper error handling for model loading
- Consider fallback strategies
- Log errors appropriately
4. **Performance**:
- Use appropriate batch sizes
- Choose the right model variant for your use case
- Profile your application to optimize performance
## Support
For issues with the optional model bundling package:
- [GitHub Issues](https://github.com/soulcraft-research/brainy/issues)
- [Main Documentation](https://github.com/soulcraft-research/brainy)
- [Model Management Guide](./model-management.md)