brainy/docs/deployment/aws-deployment.md

AWS Deployment Guide for Brainy

Overview

Deploy Brainy on AWS with automatic scaling, high availability, and zero-config dynamic adaptation. Brainy automatically adapts to your AWS environment without manual configuration.

Quick Start (Zero-Config)

Option 1: AWS Lambda (Serverless)

# Install Brainy
npm install @soulcraft/brainy

# Create handler.js
cat > handler.js << 'EOF'
const { Brainy } = require('@soulcraft/brainy')

let brain

exports.handler = async (event) => {
  // Brainy auto-detects Lambda environment and configures accordingly
  if (!brain) {
    brain = new Brainy() // Zero config - auto-adapts to Lambda
    await brain.init()
  }
  
  const { method, ...params } = JSON.parse(event.body)
  
  switch(method) {
    case 'add':
      const id = await brain.add(params)
      return { statusCode: 200, body: JSON.stringify({ id }) }
    case 'find':
      const results = await brain.find(params)
      return { statusCode: 200, body: JSON.stringify({ results }) }
    default:
      return { statusCode: 400, body: 'Unknown method' }
  }
}
EOF

# Deploy with AWS SAM
sam init --runtime nodejs20.x --name brainy-app
sam deploy --guided

Option 2: ECS Fargate (Container)

# Build and push Docker image
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $ECR_URI
docker build -t brainy .
docker tag brainy:latest $ECR_URI/brainy:latest
docker push $ECR_URI/brainy:latest

# Deploy with minimal ECS task definition
cat > task-definition.json << 'EOF'
{
  "family": "brainy",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "containerDefinitions": [{
    "name": "brainy",
    "image": "$ECR_URI/brainy:latest",
    "environment": [
      {"name": "NODE_ENV", "value": "production"}
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/brainy",
        "awslogs-region": "us-east-1",
        "awslogs-stream-prefix": "ecs"
      }
    }
  }]
}
EOF

# Brainy auto-detects ECS environment and uses S3 for storage
aws ecs register-task-definition --cli-input-json file://task-definition.json
aws ecs create-service --cluster default --service-name brainy --task-definition brainy --desired-count 2

Option 3: EC2 Auto-Scaling

# User data script for EC2 instances
#!/bin/bash
curl -fsSL https://rpm.nodesource.com/setup_20.x | sudo bash -
sudo yum install -y nodejs git

# Clone and setup (or use your deployment method)
git clone https://github.com/yourorg/brainy-app.git /app
cd /app
npm install --production

# Create systemd service
cat > /etc/systemd/system/brainy.service << 'EOF'
[Unit]
Description=Brainy
After=network.target

[Service]
Type=simple
User=ec2-user
WorkingDirectory=/app
ExecStart=/usr/bin/node index.js
Restart=on-failure
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target
EOF

systemctl start brainy
systemctl enable brainy

Zero-Config Storage (Automatic)

Brainy automatically detects and uses the best available storage:

// No configuration needed - Brainy auto-detects:
const brain = new Brainy()

// Auto-detection priority:
// 1. S3 (if IAM role has permissions)
// 2. EFS (if mounted at /mnt/efs)
// 3. EBS volume (if available)
// 4. Instance storage (fallback)

Manual S3 Configuration (Optional)

const brain = new Brainy({
  storage: {
    type: 's3',
    options: {
      bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket
      region: process.env.AWS_REGION || 'auto', // 'auto' detects region
      // IAM role provides credentials automatically
    }
  }
})

Scaling Strategies

1. Horizontal Scaling (Recommended)

# Auto-scaling policy
Resources:
  AutoScalingTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    Properties:
      ServiceNamespace: ecs
      ResourceId: service/default/brainy
      ScalableDimension: ecs:service:DesiredCount
      MinCapacity: 2
      MaxCapacity: 100
      
  AutoScalingPolicy:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
      PolicyType: TargetTrackingScaling
      TargetTrackingScalingPolicyConfiguration:
        TargetValue: 70.0
        PredefinedMetricSpecification:
          PredefinedMetricType: ECSServiceAverageCPUUtilization

2. Vertical Scaling

Brainy automatically adapts to available memory:

• 256MB: Minimal mode, optimized caching
• 512MB: Standard mode, balanced performance
• 1GB+: Full mode, maximum performance

High Availability Setup

Multi-AZ Deployment

// Brainy automatically handles multi-AZ with S3
const brain = new Brainy({
  distributed: {
    enabled: true,  // Auto-enables with S3 storage
    coordinationMethod: 'auto' // Uses S3 for coordination
  }
})

Load Balancing

# Application Load Balancer with health checks
aws elbv2 create-load-balancer \
  --name brainy-alb \
  --subnets subnet-xxx subnet-yyy \
  --security-groups sg-xxx

aws elbv2 create-target-group \
  --name brainy-targets \
  --protocol HTTP \
  --port 3000 \
  --vpc-id vpc-xxx \
  --health-check-path /health \
  --health-check-interval-seconds 30

Monitoring & Observability

CloudWatch Integration

Brainy automatically sends metrics when running on AWS:

// Automatic CloudWatch metrics (no config needed)
// - Request count
// - Response time
// - Error rate
// - Storage usage
// - Memory usage

Custom Metrics

const brain = new Brainy({
  monitoring: {
    enabled: true,
    customMetrics: {
      namespace: 'Brainy/Production',
      dimensions: [
        { Name: 'Environment', Value: 'production' },
        { Name: 'Service', Value: 'api' }
      ]
    }
  }
})

Security Best Practices

1. IAM Role (Recommended)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::brainy-*/*",
        "arn:aws:s3:::brainy-*"
      ]
    }
  ]
}

2. VPC Configuration

# Private subnets with NAT Gateway
aws ec2 create-vpc --cidr-block 10.0.0.0/16
aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.1.0/24 --availability-zone us-east-1a
aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.2.0/24 --availability-zone us-east-1b

3. Encryption

// Automatic encryption with S3
const brain = new Brainy({
  storage: {
    type: 's3',
    options: {
      encryption: 'auto' // Uses S3 SSE-S3 by default
    }
  }
})

Cost Optimization

1. Spot Instances (70% savings)

aws ec2 request-spot-fleet --spot-fleet-request-config '{
  "IamFleetRole": "arn:aws:iam::xxx:role/fleet-role",
  "TargetCapacity": 2,
  "SpotPrice": "0.05",
  "LaunchSpecifications": [{
    "ImageId": "ami-xxx",
    "InstanceType": "t3.medium",
    "UserData": "BASE64_ENCODED_STARTUP_SCRIPT"
  }]
}'

2. S3 Intelligent-Tiering

// Brainy automatically uses S3 Intelligent-Tiering
const brain = new Brainy({
  storage: {
    type: 's3',
    options: {
      storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization
    }
  }
})

3. Lambda Reserved Concurrency

aws lambda put-function-concurrency \
  --function-name brainy-handler \
  --reserved-concurrent-executions 10

Deployment Automation

GitHub Actions CI/CD

name: Deploy to AWS
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      
      - name: Deploy to ECS
        run: |
          docker build -t brainy .
          docker tag brainy:latest $ECR_URI/brainy:latest
          docker push $ECR_URI/brainy:latest
          aws ecs update-service --cluster default --service brainy --force-new-deployment

Troubleshooting

Common Issues

1. Storage Auto-Detection Fails

   // Explicitly specify storage
   const brain = new Brainy({
     storage: { type: 's3', options: { bucket: 'my-bucket' } }
   })
   

2. Memory Issues

   // Optimize for low memory
   const brain = new Brainy({
     cache: { maxSize: 100 }, // Reduce cache size
     index: { M: 8 }          // Reduce HNSW connections
   })
   

3. Cold Starts (Lambda)

   v7.3.0+ Progressive Initialization (Zero-Config)

   Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME) and uses progressive initialization for <200ms cold starts:

   // Zero-config - Brainy auto-detects Lambda
   const brain = new Brainy({
     storage: {
       type: 's3',
       s3Storage: {
         bucketName: 'my-bucket',
         region: 'us-east-1',
         accessKeyId: process.env.AWS_ACCESS_KEY_ID,
         secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
       }
     }
   })
   await brain.init() // Returns in <200ms
   
   // First write validates bucket (lazy validation)
   await brain.add('noun', { name: 'test' }) // Validates here
   

   Manual Override (if needed)

   const brain = new Brainy({
     storage: {
       type: 's3',
       s3Storage: {
         bucketName: 'my-bucket',
         // Force specific mode
         initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
       }
     }
   })
   

   Mode	Cold Start	Best For
   auto (default)	<200ms in Lambda	Zero-config, auto-detects
   progressive	<200ms always	Force fast init everywhere
   strict	100-500ms+	Local dev, tests, debugging

   Warm-up (Alternative)

   exports.warmup = async () => {
     if (!brain) {
       brain = new Brainy({ warmup: true })
       await brain.init()
     }
   }
   

   Readiness Detection (v7.3.0+)

   Use the brain.ready Promise to ensure Brainy is initialized before handling requests:

   let brain
   
   exports.handler = async (event) => {
     if (!brain) {
       brain = new Brainy({ storage: { type: 's3', ... } })
       brain.init()  // Fire and forget
     }
   
     // Wait for initialization to complete
     await brain.ready
   
     // Now safe to use brain methods
     const results = await brain.find({ query: event.queryStringParameters.q })
     return { statusCode: 200, body: JSON.stringify(results) }
   }
   

   For health checks, use isFullyInitialized() to verify all background tasks are complete:

   exports.healthCheck = async () => {
     try {
       await brain.ready
       return {
         statusCode: 200,
         body: JSON.stringify({
           status: 'ready',
           fullyInitialized: brain.isFullyInitialized()
         })
       }
     } catch (error) {
       return {
         statusCode: 503,
         body: JSON.stringify({ status: 'initializing' })
       }
     }
   }
   

Production Checklist

• IAM roles configured with minimal permissions
• VPC with private subnets
• Auto-scaling configured
• CloudWatch alarms set up
• Backup strategy (S3 versioning enabled)
• SSL/TLS certificates configured
• Rate limiting enabled
• Health checks configured
• Monitoring dashboard created
• Cost alerts configured

Support

• Documentation: https://brainy.soulcraft.ai/docs
• Issues: https://github.com/soulcraft/brainy/issues
• Community: https://discord.gg/brainy