refactor: simplify build system and improve model loading flexibility

- Remove Rollup bundling in favor of direct TypeScript compilation - Move from bundled models to dynamic model loading with configurable paths - Add Docker deployment examples and documentation - Implement robust model loader with fallback mechanisms - Update storage adapters for better cross-environment compatibility - Add comprehensive tests for model loading and package installation - Simplify package.json scripts and remove complex build configurations - Clean up deprecated demo files and old bundling scripts BREAKING CHANGE: Models are no longer bundled with the package. They are now loaded dynamically from CDN or custom paths.
2025-08-05 16:09:30 -07:00 · 2025-08-05 16:09:30 -07:00 · 52a43d51d4
commit 52a43d51d4
parent 89413ebec2
51 changed files with 4835 additions and 8007 deletions
--- a/examples/docker-deployment/Dockerfile
+++ b/examples/docker-deployment/Dockerfile
@ -0,0 +1,66 @@
+# Multi-stage Dockerfile for Brainy with embedded models
+FROM node:24-alpine AS builder
+
+# Set working directory
+WORKDIR /app
+
+# Copy package files
+COPY package*.json ./
+
+# Install dependencies including @soulcraft/brainy-models
+RUN npm ci --only=production
+
+# Copy application code
+COPY . .
+
+# Create models directory and copy models from node_modules
+# This ensures models are included even when node_modules is ignored
+RUN mkdir -p /app/models && \
+    if [ -d "./node_modules/@soulcraft/brainy-models" ]; then \
+        echo "Copying models from @soulcraft/brainy-models..."; \
+        cp -r ./node_modules/@soulcraft/brainy-models/models/* /app/models/ || \
+        cp -r ./node_modules/@soulcraft/brainy-models/* /app/models/ || \
+        echo "Models structure may vary, copying entire package"; \
+    else \
+        echo "⚠️ @soulcraft/brainy-models not found - will fall back to remote loading"; \
+    fi
+
+# Production stage
+FROM node:24-alpine AS production
+
+# Set working directory
+WORKDIR /app
+
+# Copy package files
+COPY package*.json ./
+
+# Install only production dependencies (without @soulcraft/brainy-models to save space)
+RUN npm ci --only=production --omit=optional && \
+    npm cache clean --force
+
+# Copy application code
+COPY . .
+
+# Copy models from builder stage
+COPY --from=builder /app/models /app/models
+
+# Set environment variable to use custom models path
+ENV BRAINY_MODELS_PATH=/app/models
+
+# Create non-root user
+RUN addgroup -g 1001 -S nodejs && \
+    adduser -S brainy -u 1001
+
+# Change ownership of the app directory
+RUN chown -R brainy:nodejs /app
+USER brainy
+
+# Expose port (adjust as needed)
+EXPOSE 3000
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
+    CMD node -e "console.log('Health check passed')" || exit 1
+
+# Start the application
+CMD ["node", "dist/your-app.js"]
--- a/examples/docker-deployment/README.md
+++ b/examples/docker-deployment/README.md
@ -0,0 +1,235 @@
+# Docker Deployment for Brainy
+
+This guide shows how to deploy Brainy applications with embedded models in Docker containers, avoiding the need to download models at runtime.
+
+## Problem
+
+When deploying Node.js applications to Docker containers, `node_modules` is often ignored in `.dockerignore` to reduce image size. However, this means the `@soulcraft/brainy-models` package (containing pre-trained models) won't be included in the container, forcing runtime model downloads from the internet.
+
+## Solution
+
+Brainy now supports loading models from custom directories using:
+
+1. **Environment variable**: `BRAINY_MODELS_PATH` or `MODELS_PATH`
+2. **Configuration option**: `customModelsPath` in `RobustModelLoader`
+
+## Deployment Strategies
+
+### Strategy 1: Embed Models in Docker Image (Recommended)
+
+This approach copies models from `node_modules` to a custom location during the Docker build process.
+
+```dockerfile
+# Multi-stage build to extract models
+FROM node:24-alpine AS builder
+WORKDIR /app
+COPY package*.json ./
+RUN npm ci --only=production
+COPY . .
+
+# Extract models from node_modules
+RUN mkdir -p /app/models && \
+    if [ -d "./node_modules/@soulcraft/brainy-models" ]; then \
+        cp -r ./node_modules/@soulcraft/brainy-models/models/* /app/models/; \
+    fi
+
+# Production stage
+FROM node:24-alpine AS production
+WORKDIR /app
+COPY package*.json ./
+RUN npm ci --only=production --omit=optional
+COPY . .
+COPY --from=builder /app/models /app/models
+
+# Set environment variable
+ENV BRAINY_MODELS_PATH=/app/models
+
+CMD ["node", "dist/your-app.js"]
+```
+
+### Strategy 2: Mount Models as Volume
+
+Mount pre-downloaded models as a Docker volume:
+
+```yaml
+services:
+  brainy-app:
+    image: your-brainy-app
+    environment:
+      - BRAINY_MODELS_PATH=/models
+    volumes:
+      - ./models:/models:ro
+```
+
+### Strategy 3: Init Container Pattern
+
+Use an init container to download models before starting your application:
+
+```yaml
+services:
+  model-downloader:
+    image: node:24-alpine
+    volumes:
+      - brainy-models:/models
+    command: |
+      sh -c "
+        npm install @soulcraft/brainy-models
+        cp -r node_modules/@soulcraft/brainy-models/models/* /models/
+      "
+
+  brainy-app:
+    image: your-brainy-app
+    environment:
+      - BRAINY_MODELS_PATH=/models
+    volumes:
+      - brainy-models:/models:ro
+    depends_on:
+      - model-downloader
+
+volumes:
+  brainy-models:
+```
+
+## Configuration Options
+
+### Environment Variables
+
+```bash
+# Primary environment variable
+BRAINY_MODELS_PATH=/app/models
+
+# Alternative variable name
+MODELS_PATH=/app/models
+```
+
+### Programmatic Configuration
+
+```javascript
+import { BrainyData, RobustModelLoader } from '@soulcraft/brainy'
+
+// Option 1: Configure via RobustModelLoader
+const loader = new RobustModelLoader({
+  customModelsPath: '/app/models'
+})
+
+// Option 2: Environment variable (automatic)
+// Just set BRAINY_MODELS_PATH and it will be used automatically
+```
+
+## Model Directory Structure
+
+Brainy will search for models in the following subdirectories (in order):
+
+```
+/app/models/
+├── universal-sentence-encoder/          # Direct path
+├── models/universal-sentence-encoder/   # @soulcraft/brainy-models structure
+├── tfhub/universal-sentence-encoder/    # TensorFlow Hub structure
+├── use/                                 # Short name
+└── model.json                          # Root directory
+```
+
+## Cloud Run Deployment
+
+For Google Cloud Run, use the embedded models strategy:
+
+```dockerfile
+FROM node:24-alpine
+WORKDIR /app
+COPY package*.json ./
+RUN npm ci --only=production
+COPY . .
+
+# Copy models during build
+RUN if [ -d "./node_modules/@soulcraft/brainy-models" ]; then \
+        mkdir -p /app/models && \
+        cp -r ./node_modules/@soulcraft/brainy-models/models/* /app/models/; \
+    fi
+
+ENV BRAINY_MODELS_PATH=/app/models
+ENV PORT=8080
+EXPOSE 8080
+
+CMD ["node", "dist/server.js"]
+```
+
+Deploy with:
+
+```bash
+gcloud run deploy brainy-app \
+  --source . \
+  --platform managed \
+  --region us-central1 \
+  --allow-unauthenticated \
+  --memory 2Gi \
+  --cpu 1
+```
+
+## AWS Lambda/ECS Deployment
+
+Similar approach works for AWS services:
+
+```dockerfile
+FROM public.ecr.aws/lambda/nodejs:24
+
+# Copy models during build
+COPY package*.json ./
+RUN npm ci --only=production
+COPY . .
+RUN mkdir -p /var/task/models && \
+    cp -r ./node_modules/@soulcraft/brainy-models/models/* /var/task/models/
+
+ENV BRAINY_MODELS_PATH=/var/task/models
+
+CMD ["index.handler"]
+```
+
+## Verification
+
+Your application logs should show:
+
+```
+✅ Found model at custom path: /app/models/universal-sentence-encoder/model.json
+✅ Successfully loaded model from custom directory
+   Using custom model path for Docker/production deployment
+```
+
+Instead of:
+
+```
+⚠️ Local model not found. Falling back to remote model loading.
+```
+
+## Benefits
+
+1. **Faster startup**: No model download time
+2. **Offline deployment**: Works without internet access
+3. **Predictable performance**: No network dependency
+4. **Smaller runtime image**: Can exclude dev dependencies
+5. **Security**: No external network calls required
+
+## Troubleshooting
+
+### Models not found
+
+1. Check the environment variable is set: `echo $BRAINY_MODELS_PATH`
+2. Verify models directory exists: `ls -la /app/models`
+3. Check model structure: `find /app/models -name "model.json"`
+
+### Memory issues
+
+Models require approximately 500MB of RAM. Ensure your container has sufficient memory:
+
+```yaml
+deploy:
+  resources:
+    limits:
+      memory: 2G
+```
+
+### Build failures
+
+If `@soulcraft/brainy-models` is not available during build:
+1. Ensure it's in `dependencies`, not `devDependencies`
+2. Use `npm ci --only=production` instead of `npm install`
+3. Check the package is properly published and accessible
--- a/examples/docker-deployment/docker-compose.yml
+++ b/examples/docker-deployment/docker-compose.yml
@ -0,0 +1,66 @@
+version: '3.8'
+
+services:
+  brainy-app:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    ports:
+      - "3000:3000"
+    environment:
+      # Set custom models path for Brainy
+      - BRAINY_MODELS_PATH=/app/models
+      # Optional: Set other Brainy configuration
+      - NODE_ENV=production
+    volumes:
+      # Optional: Mount models from host (alternative to embedding in image)
+      # - ./models:/app/models:ro
+      - ./logs:/app/logs
+    restart: unless-stopped
+    deploy:
+      resources:
+        limits:
+          cpus: '1.0'
+          memory: 2G
+        reservations:
+          cpus: '0.5'
+          memory: 1G
+
+  # Alternative: Mount models from a separate volume
+  brainy-app-with-volume:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    ports:
+      - "3001:3000"
+    environment:
+      - BRAINY_MODELS_PATH=/models
+      - NODE_ENV=production
+    volumes:
+      # Mount models from external volume
+      - brainy-models:/models:ro
+      - ./logs:/app/logs
+    restart: unless-stopped
+    depends_on:
+      - model-downloader
+
+  # Service to download and prepare models
+  model-downloader:
+    image: node:24-alpine
+    volumes:
+      - brainy-models:/models
+    command: >
+      sh -c "
+        if [ ! -f /models/universal-sentence-encoder/model.json ]; then
+          echo 'Downloading Universal Sentence Encoder models...';
+          mkdir -p /models/universal-sentence-encoder;
+          # Add your model download logic here
+          echo 'Models would be downloaded here in a real setup';
+        else
+          echo 'Models already exist';
+        fi
+      "
+
+volumes:
+  brainy-models:
+    driver: local
--- a/examples/docker-deployment/example-app.js
+++ b/examples/docker-deployment/example-app.js
@ -0,0 +1,86 @@
+#!/usr/bin/env node
+
+/**
+ * Example Brainy application for Docker deployment
+ * Demonstrates custom models path usage
+ */
+
+import { BrainyData } from '@soulcraft/brainy'
+
+async function main() {
+  console.log('🚀 Starting Brainy Docker Example App...')
+  console.log(`Environment: ${process.env.NODE_ENV || 'development'}`)
+  console.log(`Models Path: ${process.env.BRAINY_MODELS_PATH || 'default (node_modules)'}`)
+  
+  const db = new BrainyData({
+    forceMemoryStorage: true,
+    dimensions: 512,
+    // Don't skip embeddings - let it try to load models
+    skipEmbeddings: false
+  })
+
+  try {
+    console.log('\n📊 Initializing BrainyData...')
+    await db.init()
+    console.log('✅ BrainyData initialized successfully!')
+
+    // Add some test data
+    console.log('\n📝 Adding test data...')
+    const testItems = [
+      'Artificial intelligence is transforming the world',
+      'Machine learning enables computers to learn without being explicitly programmed',
+      'Docker containers provide a consistent environment for applications',
+      'Cloud deployment makes applications scalable and reliable'
+    ]
+
+    const ids = []
+    for (const text of testItems) {
+      const id = await db.add({ content: text })
+      ids.push(id)
+      console.log(`   Added: "${text}" (ID: ${id})`)
+    }
+
+    // Search for similar content
+    console.log('\n🔍 Searching for AI-related content...')
+    const searchResults = await db.search('artificial intelligence and machine learning', 2)
+    
+    console.log(`Found ${searchResults.length} results:`)
+    searchResults.forEach((result, index) => {
+      console.log(`   ${index + 1}. "${result.content}" (Score: ${result.similarity?.toFixed(3)})`)
+    })
+
+    // Get database statistics
+    console.log('\n📈 Database Statistics:')
+    const stats = await db.getStatistics()
+    console.log(`   Total items: ${stats.totalVectors}`)
+    console.log(`   Storage type: ${stats.storageType}`)
+    console.log(`   Index type: ${stats.indexType}`)
+
+  } catch (error) {
+    console.error('❌ Error:', error.message)
+    console.error('\n💡 Troubleshooting tips:')
+    console.error('   1. Ensure models are available in BRAINY_MODELS_PATH')
+    console.error('   2. Check that @soulcraft/brainy-models is installed')
+    console.error('   3. Verify network connectivity for remote model loading')
+    process.exit(1)
+  }
+
+  console.log('\n🎉 Example completed successfully!')
+}
+
+// Handle graceful shutdown
+process.on('SIGINT', () => {
+  console.log('\n👋 Shutting down gracefully...')
+  process.exit(0)
+})
+
+process.on('SIGTERM', () => {
+  console.log('\n👋 Received SIGTERM, shutting down...')
+  process.exit(0)
+})
+
+// Run the example
+main().catch(error => {
+  console.error('Fatal error:', error)
+  process.exit(1)
+})