brainy/TECHNICAL_GUIDES.md

Brainy Technical Guides

This document consolidates technical guides and documentation for specific aspects of the Brainy project.

Table of Contents

• Vector Dimension Standardization
• Dimension Mismatch Issue
• Production Migration Guide
• Threading Implementation
• Storage Testing
• Scaling Strategy
• Metadata Handling
• Model Loading

Vector Dimension Standardization

Brainy uses a standardized approach to vector dimensions to ensure consistency across the system. This section explains how vector dimensions are handled and standardized.

Default Dimensions

The default dimension for vectors in Brainy is 512, which is the output dimension of the Universal Sentence Encoder (USE) model used for text embeddings.

Dimension Validation

Brainy validates vector dimensions during database operations to ensure consistency:

1. During initialization, vectors with mismatched dimensions are skipped
2. When adding new vectors, their dimensions are validated against the expected dimension
3. When searching, query vectors are validated to ensure they match the expected dimension

Configuration

You can configure the expected dimension when creating a BrainyData instance:

const db = new BrainyData({
  dimensions: 768 // Use a different dimension (e.g., for BERT embeddings)
})

If not specified, the default dimension of 512 is used.

Handling Dimension Changes

When changing the embedding model or dimension size, you need to:

1. Back up your existing data
2. Re-embed all vectors with the new embedding function
3. Update the dimension configuration

Standardization Changes

As of version 0.18.0, Brainy has standardized all vector dimensions to 512, which is the dimension used by the Universal Sentence Encoder. This change ensures consistency between data insertion, storage, and search operations, eliminating potential dimension mismatch issues.

Changes Made

1. Fixed Dimension Value: Vector dimensions are now fixed at 512 throughout the codebase.
2. Removed Configuration Option: The dimensions configuration option has been removed from BrainyDataConfig.
3. Consistent Validation: All vectors are validated to ensure they have exactly 512 dimensions.

Rationale

Previously, vector dimensions were configurable, which could lead to mismatches between:

• Vectors stored in the database
• The expected dimensions in the HNSW index
• Vectors generated by the embedding function

These mismatches could cause search functionality to break, as vectors with different dimensions would be skipped during initialization.

By standardizing all vectors to 512 dimensions (matching the Universal Sentence Encoder's output), we ensure that:

• All vectors in the database have consistent dimensions
• The HNSW index always works with vectors of the expected size
• Search queries always match the dimension of stored vectors

Impact on Existing Code

• The dimensions property in BrainyDataConfig has been removed
• Attempting to add vectors with dimensions other than 512 will throw an error
• Existing data with non-512 dimensions will be skipped during initialization

Migration

If you have existing data with dimensions other than 512, you can use the provided fix-dimension-mismatch.js script to re-embed your data with the correct dimensions:

node fix-dimension-mismatch.js

This script:

1. Creates a backup of your existing data
2. Re-embeds all nouns using the Universal Sentence Encoder (512 dimensions)
3. Recreates all verb relationships
4. Verifies that search functionality works correctly

Best Practices

• Always use the built-in embedding function for text data, which will automatically produce 512-dimensional vectors
• If you're creating vectors manually, ensure they have exactly 512 dimensions
• When migrating from previous versions, run the fix-dimension-mismatch.js script to ensure all data has consistent dimensions

Dimension Mismatch Issue

This section summarizes a dimension mismatch issue that occurred in Brainy and provides recommendations for handling similar issues.

What Happened

The search functionality in Brainy stopped working because of a dimension mismatch between stored vectors and the expected dimensions in the current version of the codebase:

1. Previous State: The system was using vectors with 3 dimensions.
2. Current State: The system now expects 512-dimensional vectors from the Universal Sentence Encoder.
3. Code Change: Recent updates introduced dimension validation during initialization, which skips vectors with mismatched dimensions.
4. Result: During initialization, vectors with 3 dimensions were skipped, resulting in an empty search index and no search results.

Root Cause Analysis

The root cause was identified by examining the codebase:

1. In brainyData.ts, the init() method checks if vector dimensions match the expected dimensions:

   if (noun.vector.length !== this._dimensions) {
     console.warn(
       `Skipping noun ${noun.id} due to dimension mismatch: expected ${this._dimensions}, got ${noun.vector.length}`
     )
     // Skip this noun and continue with the next one
     return;
   }
   

2. The default dimension is set to 512 in the constructor:

   this._dimensions = config.dimensions || 512
   

3. The UniversalSentenceEncoder class in embedding.ts produces 512-dimensional vectors:

   // Return a zero vector of appropriate dimension (512 is the default for USE)
   return new Array(512).fill(0)
   

This indicates that the system previously used 3-dimensional vectors, but after the update, it expects 512-dimensional vectors. The existing data was not migrated, causing the search functionality to break.

Solution Implemented

We created and tested a fix script (fix-dimension-mismatch.js) that:

1. Creates a backup of the existing data
2. Reads all noun files directly from the filesystem
3. For each noun:
   • Extracts text from metadata
   • Deletes the existing noun
   • Re-adds the noun with the same ID but using the current embedding function
4. Recreates all verb relationships between the re-embedded nouns
5. Verifies that search works by performing a test search

The script successfully fixed the issue by re-embedding all data with the correct dimensions, and search functionality was restored.

Production Recommendations

For production environments, we recommend:

1. Use the Enhanced Migration Script

We've created a comprehensive production migration guide that includes:

• Enhanced backup strategies with metadata
• Batching for large datasets
• Robust error handling and recovery
• Progress monitoring and reporting
• A parallel database approach for mission-critical systems

2. Implement Preventive Measures

To prevent similar issues in the future:

• Version Tracking: Add version information to stored vectors
• Auto-Migration: Enhance initialization to automatically re-embed mismatched vectors
• Regular Validation: Implement a database validation process
• Documentation: Document embedding changes in release notes

3. Scheduling and Communication

• Schedule the migration during a maintenance window
• Communicate the change to all stakeholders
• Have a rollback plan in case of issues
• Monitor the system after the migration

Production Migration Guide

This section provides a comprehensive guide for migrating Brainy databases in production environments, particularly when dealing with dimension changes or other breaking changes.

Preparation

Before starting the migration:

1. Create a Backup: Always create a complete backup of your database before migration
2. Test the Migration: Test the migration process on a copy of your production data
3. Schedule Downtime: Plan for a maintenance window if the migration requires downtime
4. Communicate: Inform all stakeholders about the planned migration

Migration Strategies

Strategy 1: In-Place Migration

For smaller databases or when downtime is acceptable:

1. Stop all services that use the database
2. Run the migration script
3. Verify the migration was successful
4. Restart the services

Strategy 2: Parallel Database

For mission-critical systems or large databases:

1. Create a new database instance
2. Run the migration script to populate the new database
3. Test the new database thoroughly
4. Switch over to the new database with minimal downtime

Migration Script

The enhanced migration script includes:

1. Comprehensive Backup: Creates a complete backup with all metadata
2. Batched Processing: Processes data in batches to avoid memory issues
3. Progress Tracking: Shows progress and estimated time remaining
4. Error Handling: Robust error handling with automatic retries
5. Validation: Validates the migrated data to ensure correctness

Post-Migration Steps

After completing the migration:

1. Verify Data Integrity: Run validation queries to ensure data was migrated correctly
2. Monitor Performance: Watch for any performance issues after the migration
3. Update Documentation: Document the migration and any changes to the data structure
4. Retain Backups: Keep the pre-migration backups for a reasonable period

Threading Implementation

Brainy includes comprehensive multithreading support to improve performance across all environments. This section explains how threading is implemented and used in the project.

Threading Architecture

The threading architecture in Brainy is designed to work consistently across different environments:

1. Browser Environment: Uses Web Workers for parallel processing
2. Node.js Environment: Uses Worker Threads for parallel processing
3. Fallback: Gracefully falls back to sequential processing when threading is not available

Overview

Brainy uses a unified threading approach that adapts to the environment it's running in:

1. Node.js: Uses Worker Threads API (optimized for Node.js 24+)
2. Browser: Uses Web Workers API
3. Fallback: Executes on the main thread when neither Worker Threads nor Web Workers are available

This implementation ensures that compute-intensive operations (like embedding generation and vector calculations) can be performed efficiently without blocking the main thread, while maintaining compatibility across all environments.

Environment Detection

Brainy automatically detects the environment it's running in:

// From unified.ts
export const environment = {
  isBrowser: typeof window !== 'undefined',
  isNode: typeof process !== 'undefined' && process.versions && process.versions.node,
  isServerless: typeof window === 'undefined' && 
    (typeof process === 'undefined' || !process.versions || !process.versions.node)
}

Additional environment detection functions are available in src/utils/environment.ts:

// Check if threading is available
export function isThreadingAvailable(): boolean {
  return areWebWorkersAvailable() || areWorkerThreadsAvailable();
}

// Check if Web Workers are available (browser)
export function areWebWorkersAvailable(): boolean {
  return isBrowser() && typeof Worker !== 'undefined';
}

// Check if Worker Threads are available (Node.js)
export function areWorkerThreadsAvailable(): boolean {
  if (!isNode()) return false;
  try {
    require('worker_threads');
    return true;
  } catch (e) {
    return false;
  }
}

Thread Execution

The core of the threading implementation is the executeInThread function in src/utils/workerUtils.ts:

export function executeInThread<T>(fnString: string, args: any): Promise<T> {
  if (environment.isNode) {
    return executeInNodeWorker<T>(fnString, args)
  } else if (environment.isBrowser && typeof window !== 'undefined' && window.Worker) {
    return executeInWebWorker<T>(fnString, args)
  } else {
    // Fallback to main thread execution
    try {
      const fn = new Function('return ' + fnString)()
      return Promise.resolve(fn(args) as T)
    } catch (error) {
      return Promise.reject(error)
    }
  }
}

This function:

1. Checks if it's running in Node.js and uses Worker Threads if available
2. Checks if it's running in a browser and uses Web Workers if available
3. Falls back to executing on the main thread if neither is available

Node.js Implementation

For Node.js environments, Brainy uses the Worker Threads API with optimizations for Node.js 24:

function executeInNodeWorker<T>(fnString: string, args: any): Promise<T> {
  // Implementation using Node.js Worker Threads
  // Includes worker pool management for better performance
  // Uses dynamic imports with the 'node:' protocol prefix
  // ...
}

Key optimizations:

• Worker pool to reuse workers and minimize overhead
• Dynamic imports with the node: protocol prefix
• Error handling and cleanup

Browser Implementation

For browser environments, Brainy uses the Web Workers API:

function executeInWebWorker<T>(fnString: string, args: any): Promise<T> {
  // Implementation using browser Web Workers
  // Creates a blob URL for the worker code
  // Handles message passing and error handling
  // ...
}

Key features:

• Creates workers using Blob URLs
• Proper cleanup of resources (terminating workers and revoking URLs)
• Error handling

Fallback Mechanism

When neither Worker Threads nor Web Workers are available, Brainy falls back to executing on the main thread:

// Fallback to main thread execution
try {
  const fn = new Function('return ' + fnString)()
  return Promise.resolve(fn(args) as T)
} catch (error) {
  return Promise.reject(error)
}

This ensures that Brainy works in all environments, even if threading is not available.

Key Threading Features

1. Parallel Batch Processing: Add multiple items concurrently with controlled parallelism
2. Multithreaded Vector Search: Perform distance calculations in parallel for faster search operations
3. Threaded Embedding Generation: Generate embeddings in separate threads to avoid blocking the main thread
4. Worker Reuse: Maintains a pool of workers to avoid the overhead of creating and terminating workers
5. Model Caching: Initializes the embedding model once per worker and reuses it for multiple operations
6. Batch Embedding: Processes multiple items in a single embedding operation for better performance
7. Automatic Environment Detection: Adapts to browser (Web Workers) and Node.js (Worker Threads) environments

Usage

Threading is used automatically in several parts of Brainy:

1. Embedding Generation: When using createThreadedEmbeddingFunction()
2. Batch Operations: When using addBatch() with multiple items
3. Vector Search: When performing similarity searches with large datasets

You can control threading behavior through configuration:

const db = new BrainyData({
  performance: {
    useParallelization: true, // Enable multithreaded operations
    maxWorkers: 4,            // Maximum number of worker threads
    batchSize: 50             // Number of items to process in a single batch
  }
})

The threading implementation is used throughout Brainy, particularly for compute-intensive operations like embedding generation:

export function createThreadedEmbeddingFunction(
  model: EmbeddingModel
): EmbeddingFunction {
  const embeddingFunction = createEmbeddingFunction(model)

  return async (data: any): Promise<Vector> => {
    // Convert the embedding function to a string
    const fnString = embeddingFunction.toString()

    // Execute the embedding function in a thread
    return await executeInThread<Vector>(fnString, data)
  }
}

Implementation Details

The threading implementation includes:

1. Worker Pool: A reusable pool of workers that avoids the overhead of creating and destroying workers
2. Task Queue: A queue of tasks that are distributed to available workers
3. Message Passing: A standardized message format for communication between the main thread and workers
4. Error Handling: Robust error handling for worker failures
5. Resource Management: Proper cleanup of resources when workers are no longer needed

Testing

Two test scripts are provided to verify the threading implementation:

1. demo/test-browser-worker.html: Tests the threading implementation in a browser environment
2. demo/test-fallback.html: Tests the fallback mechanism when threading is not available

To run these tests:

1. Build the project: npm run build
2. Start a local server: npx http-server
3. Open the test pages in a browser:
   • http://localhost:8080/demo/test-browser-worker.html
   • http://localhost:8080/demo/test-fallback.html

Compatibility

The threading implementation has been tested and works in:

• Node.js 24+ (using Worker Threads)
• Modern browsers (using Web Workers):
  • Chrome
  • Firefox
  • Safari
  • Edge
• Environments without threading support (using fallback mechanism)

Storage Testing

This section provides guidance on testing storage adapters in Brainy, including file system storage, OPFS storage, and S3-compatible storage.

Testing Approach

When testing storage adapters, consider the following:

1. Isolation: Test each storage adapter in isolation
2. Edge Cases: Test edge cases like empty data, large data, and invalid data
3. Error Handling: Test error conditions and recovery
4. Performance: Test performance with different data sizes
5. Concurrency: Test concurrent access to the same data

Testing File System Storage

To test the file system storage adapter:

1. Create a temporary directory for testing
2. Initialize a BrainyData instance with the file system storage adapter
3. Perform CRUD operations on nouns and verbs
4. Verify that data is correctly stored and retrieved
5. Clean up the temporary directory after testing

Testing OPFS Storage

To test the Origin Private File System (OPFS) storage adapter:

1. Use a browser environment (real or simulated)
2. Initialize a BrainyData instance with the OPFS storage adapter
3. Perform CRUD operations on nouns and verbs
4. Verify that data is correctly stored and retrieved
5. Test persistence across page reloads

Testing S3-Compatible Storage

To test the S3-compatible storage adapter:

1. Use a mock S3 service or a real S3-compatible service
2. Configure the S3 storage adapter with appropriate credentials
3. Perform CRUD operations on nouns and verbs
4. Verify that data is correctly stored and retrieved
5. Test error conditions like network failures and permission issues

Automated Testing

Brainy includes automated tests for all storage adapters in the tests/ directory:

1. tests/filesystem-storage.test.ts: Tests for the file system storage adapter
2. tests/opfs-storage.test.ts: Tests for the OPFS storage adapter
3. tests/s3-storage.test.ts: Tests for the S3-compatible storage adapter

These tests can be run using the standard test commands:

# Run all storage tests
npm test -- tests/*-storage.test.ts

# Run specific storage tests
npm test -- tests/filesystem-storage.test.ts

Scaling Strategy

Brainy is designed to handle datasets of various sizes, from small collections to large-scale deployments. This section outlines strategies for scaling Brainy to handle terabyte-scale data.

Scaling Challenges

When scaling Brainy to handle very large datasets, several challenges need to be addressed:

1. Memory Constraints: Vector data can consume significant memory
2. Search Performance: Maintaining fast search with millions of vectors
3. Storage Efficiency: Optimizing how vectors are stored and retrieved
4. Concurrent Access: Handling multiple simultaneous operations

Scaling Approaches

1. Disk-Based HNSW

For datasets that can't fit entirely in memory:

• Memory-Mapped Files: Store vectors on disk but access them as if they were in memory
• Partial Loading: Load only the most frequently accessed vectors into memory
• Intelligent Caching: Cache vectors based on access patterns
• Optimized I/O: Minimize disk operations through batching and prefetching

Implementation:

const db = new BrainyData({
  hnswOptimized: {
    useDiskBasedIndex: true,
    memoryThreshold: 1024 * 1024 * 1024, // 1GB threshold
    cacheSize: 100000 // Number of vectors to keep in memory
  }
})

2. Distributed HNSW

For extremely large datasets that need to be distributed across multiple machines:

• Sharding: Partition the vector space into multiple shards
• Routing: Direct queries to the appropriate shard(s)
• Result Merging: Combine results from multiple shards
• Load Balancing: Distribute data evenly across shards

Implementation:

// On each node
const nodeDb = new BrainyData({
  sharding: {
    enabled: true,
    nodeId: 'node-1',
    totalNodes: 4,
    shardingFunction: (vector) => {
      // Determine which shard this vector belongs to
      return hashVector(vector) % 4
    }
  }
})

3. Hybrid Solutions

Combining multiple techniques for optimal performance:

• Product Quantization: Compress vectors to reduce memory usage
• Multi-Tier Storage: Use fast storage for frequently accessed data and slower storage for less frequently accessed data
• Hierarchical Clustering: Group similar vectors together for more efficient search

Implementation:

const db = new BrainyData({
  hnswOptimized: {
    productQuantization: {
      enabled: true,
      numSubvectors: 16,
      numCentroids: 256
    },
    multiTierStorage: {
      enabled: true,
      tiers: [
        { type: 'memory', capacity: '2GB' },
        { type: 'ssd', capacity: '100GB' },
        { type: 'hdd', capacity: '1TB' }
      ]
    }
  }
})

Performance Optimizations

Regardless of the scaling approach, these optimizations can improve performance:

1. Batch Operations: Process multiple items at once to reduce overhead
2. Parallel Processing: Use multiple threads for compute-intensive operations
3. Index Tuning: Adjust HNSW parameters based on dataset characteristics
4. Compression: Use vector compression techniques to reduce memory usage
5. Pruning: Periodically remove unused or low-quality vectors

Monitoring and Maintenance

To ensure optimal performance as your dataset grows:

1. Performance Metrics: Track search latency, memory usage, and throughput
2. Index Health: Monitor index quality and rebuild when necessary
3. Resource Utilization: Watch CPU, memory, and disk usage
4. Scaling Triggers: Set thresholds for when to scale up or out

Recommended Scaling Path

As your dataset grows, follow this progression:

1. Small Datasets (< 1M vectors): Standard in-memory HNSW
2. Medium Datasets (1M-10M vectors): Optimized in-memory HNSW with product quantization
3. Large Datasets (10M-100M vectors): Disk-based HNSW with intelligent caching
4. Very Large Datasets (> 100M vectors): Distributed HNSW with sharding

Metadata Handling

This section explains how metadata is handled in Brainy, including storage, retrieval, and best practices.

Metadata Structure

In Brainy, metadata is associated with both nouns (entities) and verbs (relationships):

• Noun Metadata: Describes properties of an entity
• Verb Metadata: Describes properties of a relationship between entities

Metadata is stored as JSON objects and can include any valid JSON data:

// Noun metadata example
const nounMetadata = {
  noun: NounType.Thing,
  category: 'animal',
  tags: ['pet', 'mammal'],
  attributes: {
    size: 'medium',
    lifespan: '10-15 years'
  },
  created: new Date().toISOString()
}

// Verb metadata example
const verbMetadata = {
  verb: VerbType.RelatedTo,
  strength: 0.85,
  bidirectional: true,
  created: new Date().toISOString()
}

Adding Metadata

Metadata is added when creating nouns and verbs:

// Adding a noun with metadata
const catId = await db.add("Cats are independent pets", {
  noun: NounType.Thing,
  category: 'animal',
  tags: ['pet', 'mammal'],
  attributes: {
    size: 'medium',
    lifespan: '10-15 years'
  }
})

// Adding a verb with metadata
await db.addVerb(catId, dogId, {
  verb: VerbType.RelatedTo,
  strength: 0.85,
  bidirectional: true
})

Retrieving Metadata

Metadata is included when retrieving nouns and verbs:

// Get a noun with its metadata
const noun = await db.get(catId)
console.log(noun.metadata)

// Get verbs with their metadata
const verbs = await db.getVerbsBySource(catId)
verbs.forEach(verb => console.log(verb.metadata))

Updating Metadata

Metadata can be updated using the updateMetadata method:

// Update noun metadata
await db.updateMetadata(catId, {
  ...existingMetadata,
  tags: [...existingMetadata.tags, 'domestic'],
  lastUpdated: new Date().toISOString()
})

// Update verb metadata
await db.updateVerbMetadata(verbId, {
  ...existingVerbMetadata,
  strength: 0.9,
  lastUpdated: new Date().toISOString()
})

Metadata Storage

Metadata is stored differently depending on the storage adapter:

• FileSystemStorage: Metadata is stored in separate JSON files
• OPFSStorage: Metadata is stored in separate files in the Origin Private File System
• S3CompatibleStorage: Metadata is stored as separate objects in S3
• MemoryStorage: Metadata is stored in memory as part of the noun or verb object

Metadata Indexing

Brainy does not currently index metadata for direct querying. To find nouns or verbs based on metadata, you need to:

1. Retrieve all relevant nouns or verbs
2. Filter them based on metadata properties

// Find all nouns with a specific tag
const allNouns = await db.getAllNouns()
const nounsWithTag = allNouns.filter(noun => 
  noun.metadata.tags && noun.metadata.tags.includes('domestic')
)

Best Practices

1. Consistent Structure: Use a consistent metadata structure across similar entities
2. Required Fields: Always include required fields like noun and verb types
3. Timestamps: Add creation and update timestamps for tracking changes
4. Avoid Large Objects: Keep metadata reasonably sized to avoid performance issues
5. Versioning: Consider adding a version field to track metadata schema changes

Model Loading

This section explains how model loading works in Brainy, particularly for the Universal Sentence Encoder (USE) model used for text embeddings.

Model Loading Process

Brainy uses TensorFlow.js to load and run the Universal Sentence Encoder model. The loading process follows these steps:

1. Check Cache: Check if the model is already loaded and cached
2. Load Model: If not cached, load the model from the appropriate source
3. Warm Up: Run a sample input through the model to initialize it
4. Cache Model: Store the loaded model for future use

Model Sources

The Universal Sentence Encoder model can be loaded from different sources:

1. Bundled Model: A simplified version of the model is bundled with Brainy
2. TensorFlow Hub: The full model can be loaded from TensorFlow Hub
3. Local Path: The model can be loaded from a local path

Loading Modes

Brainy supports different loading modes for the Universal Sentence Encoder:

1. Lazy Loading: The model is loaded only when needed (default)
2. Eager Loading: The model is loaded during initialization
3. Preloaded: A pre-loaded model is provided to Brainy

Configuration

You can configure model loading behavior when creating a BrainyData instance:

const db = new BrainyData({
  embedding: {
    modelLoadingMode: 'eager', // 'lazy', 'eager', or 'preloaded'
    modelPath: 'path/to/local/model', // Optional local model path
    useSimplifiedModel: true, // Use the bundled simplified model
    cacheModel: true // Cache the model for reuse
  }
})

Threaded Model Loading

For better performance, Brainy can load and run the model in a separate thread:

const db = new BrainyData({
  embedding: {
    useThreading: true, // Load and run the model in a separate thread
    maxWorkers: 4 // Maximum number of worker threads for model inference
  }
})

Model Caching

To improve performance, Brainy caches the loaded model:

1. In-Memory Caching: The model is cached in memory for reuse
2. Worker Caching: In threaded mode, each worker caches its own model instance
3. Cross-Request Caching: In server environments, the model is cached across requests

Troubleshooting

Common issues with model loading and their solutions:

1. Memory Issues: If you encounter memory issues, try:

   • Using the simplified model (useSimplifiedModel: true)
   • Loading the model in a separate thread (useThreading: true)
   • Reducing the batch size for embedding operations

2. Loading Failures: If the model fails to load, try:

   • Checking network connectivity (for TensorFlow Hub loading)
   • Verifying the local model path (for local loading)
   • Using the bundled model as a fallback

3. Performance Issues: If embedding is slow, try:

   • Using threaded embedding (useThreading: true)
   • Increasing the number of workers (maxWorkers)
   • Using batch embedding for multiple items

Best Practices

1. Eager Loading: For production environments, use eager loading to avoid delays during operation
2. Threaded Embedding: Use threaded embedding for better performance, especially for batch operations
3. Simplified Model: Use the simplified model for resource-constrained environments
4. Batch Processing: Process multiple items in a single embedding operation for better performance