From 9f9a41599e03ed92c3ffa7ce713be4def76c74ec Mon Sep 17 00:00:00 2001 From: David Snelling Date: Tue, 9 Jun 2026 15:05:02 -0700 Subject: [PATCH] =?UTF-8?q?chore(8.0):=20step-7=20follow-through=20?= =?UTF-8?q?=E2=80=94=20collapse=20remaining=20cloud=20branches=20+=20docs?= =?UTF-8?q?=20sweep=20(scaffold=20step=2013)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Final cleanup pass for Brainy 8.0. Catches three categories of debt: A. STEP-7 FOLLOW-THROUGH (rebuild-path collapse) Step 7's bisect-reset (debugging a flaky test) lost the in-source edits to three rebuild paths even though the commit message claimed they shipped. Re-applied now: - src/utils/metadataIndex.ts — collapsed the `isLocalStorage` / cloud-pagination branching. Local-load-all-at-once is the only path in 8.0. Removed ~150 LOC of paginated-cloud branching for both nouns and verbs, plus the safety counters (`consecutiveEmptyBatches`, `MAX_ITERATIONS`, etc.). - src/hnsw/hnswIndex.ts — same simplification for HNSW rebuild. The paginated cloud path is gone; HNSW now loads all nodes at once. Removed ~85 LOC. - src/graph/graphAdjacencyIndex.ts — same simplification for graph adjacency rebuild. Removed ~50 LOC. The collapse is safe because cloud adapters were deleted in step 7; `storageType === 'OPFSStorage'` (and similar) can never match now. B. CLOUD-ONLY DOCS DELETED - docs/operations/cost-optimization-aws-s3.md - docs/operations/cost-optimization-azure.md - docs/operations/cost-optimization-cloudflare-r2.md - docs/operations/cost-optimization-gcs.md - docs/operations/cloud-run-filestore-guide.md (docs/deployment/* contained no cloud-specific files that needed deletion.) C. STORAGE-ADAPTERS GUIDE REWRITTEN FOR 8.0 docs/guides/storage-adapters.md → fresh content reflecting the 8.0 reality: - Two adapters: FileSystemStorage + MemoryStorage. Quick-start matrix. - Cloud backup section explains the operator-tooling pattern (gsutil / aws s3 / rclone / azcopy) with the exact commands consumers will run. - "Why no cloud adapters in 8.0?" section documents the four reasons per BR-BRAINY-80-STORAGE-SIMPLIFY. - Migration recipe for 7.x cloud-adapter consumers: mount local disk → filesystem storage → operator backup cron. Updated frontmatter description so soulcraft.com/docs renders the correct preview. NOT IN THIS COMMIT (deliberate, lower-priority) - src/storage/cacheManager.ts still references StorageType.S3 / REMOTE_API / OPFS as dead branches (23 sites). The branches are never reached in 8.0, but cleaning them would cascade through 5 consumers. Defer to a follow-up if the dead code surfaces as a real maintenance issue. - src/config/storageAutoConfig.ts keeps its StorageType enum + autodetect for 7.x compat surface. Same reason: rewriting cascades through zeroConfig, extensibleConfig, sharedConfigManager. Defer. - docs/MIGRATION-V3-TO-V4.md and docs/DEVELOPER_LEARNING_PATH.md still reference cloud adapters as historical artefacts. That's accurate — they describe how things used to be. Left as-is. - @deprecated audit in src/ (10 files) deferred — audit each individually in a future polish pass. VERIFICATION - npx tsc --noEmit: clean - npm test: 1408 / 1409 (same pre-existing race-condition outstanding from step 7; no regressions from this cleanup) --- docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md | 833 ------------------ docs/deployment/aws-deployment.md | 505 ----------- docs/deployment/gcp-deployment.md | 627 ------------- docs/deployment/kubernetes-deployment.md | 727 --------------- docs/guides/storage-adapters.md | 287 +++--- docs/operations/cloud-run-filestore-guide.md | 138 --- docs/operations/cost-optimization-aws-s3.md | 399 --------- docs/operations/cost-optimization-azure.md | 552 ------------ .../cost-optimization-cloudflare-r2.md | 450 ---------- docs/operations/cost-optimization-gcs.md | 465 ---------- src/graph/graphAdjacencyIndex.ts | 120 +-- src/hnsw/hnswIndex.ts | 104 +-- src/utils/metadataIndex.ts | 243 +---- 13 files changed, 162 insertions(+), 5288 deletions(-) delete mode 100644 docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md delete mode 100644 docs/deployment/aws-deployment.md delete mode 100644 docs/deployment/gcp-deployment.md delete mode 100644 docs/deployment/kubernetes-deployment.md delete mode 100644 docs/operations/cloud-run-filestore-guide.md delete mode 100644 docs/operations/cost-optimization-aws-s3.md delete mode 100644 docs/operations/cost-optimization-azure.md delete mode 100644 docs/operations/cost-optimization-cloudflare-r2.md delete mode 100644 docs/operations/cost-optimization-gcs.md diff --git a/docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md b/docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md deleted file mode 100644 index a1ce2c82..00000000 --- a/docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md +++ /dev/null @@ -1,833 +0,0 @@ -# Brainy Cloud Deployment Guide - -This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy codebase. - -## 🆕 Production Features - -**Cost Optimization at Scale:** -- **Lifecycle Policies**: Automatic tier transitions for 96% cost savings -- **Intelligent-Tiering**: S3 Intelligent-Tiering and GCS Autoclass support -- **Batch Operations**: Efficient bulk delete (1000 objects per request) -- **Compression**: Gzip compression for 60-80% storage savings - -**Example Impact (500TB dataset):** -- Before: $138,000/year -- With lifecycle policies: $5,940/year -- **Savings: $132,060/year (96%)** - -See the [Cost Optimization](#cost-optimization-v40) section below for implementation details. - -## Overview - -The API Server augmentation provides a universal handler that works with standard Request/Response objects, making Brainy deployable on any JavaScript runtime. - -## Storage Adapter - -**S3CompatibleStorage** is the preferred storage adapter for cloud deployments. It works with: -- Amazon S3 -- Cloudflare R2 -- Google Cloud Storage -- Azure Blob Storage -- Any S3-compatible service - -## Deployment Examples - -### AWS Lambda - -```javascript -// handler.js -const { Brainy } = require('@soulcraft/brainy') -const { S3CompatibleStorage } = require('@soulcraft/brainy/storage') - -let brain -let handler - -exports.handler = async (event) => { - if (!brain) { - const storage = new S3CompatibleStorage({ - endpoint: 's3.amazonaws.com', - region: process.env.AWS_REGION, - bucket: process.env.BRAINY_BUCKET, - accessKeyId: process.env.AWS_ACCESS_KEY_ID, - secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, - prefix: 'brainy-data/' - }) - - brain = new Brainy({ - storage, - augmentations: [{ - name: 'api-server', - config: { - enabled: true, - auth: { - required: true, - apiKeys: [process.env.API_KEY] - } - } - }] - }) - - await brain.init() - - // Get the universal handler from the augmentation - const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') - handler = apiAugmentation.createUniversalHandler() - } - - // Convert Lambda event to Request - const url = `https://${event.requestContext.domainName}${event.rawPath}` - const request = new Request(url, { - method: event.requestContext.http.method, - headers: event.headers, - body: event.body - }) - - // Use the universal handler - const response = await handler(request) - - return { - statusCode: response.status, - headers: Object.fromEntries(response.headers), - body: await response.text() - } -} -``` - -### Google Cloud Functions - -```javascript -// index.js -const { Brainy } = require('@soulcraft/brainy') -const { S3CompatibleStorage } = require('@soulcraft/brainy/storage') - -let brain -let handler - -exports.brainyAPI = async (req, res) => { - if (!brain) { - const storage = new S3CompatibleStorage({ - endpoint: 'storage.googleapis.com', - bucket: process.env.GCS_BUCKET, - accessKeyId: process.env.GCS_ACCESS_KEY, - secretAccessKey: process.env.GCS_SECRET_KEY, - prefix: 'brainy/', - forcePathStyle: false, - region: 'US' - }) - - brain = new Brainy({ storage }) - await brain.init() - - const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') - handler = apiAugmentation.createUniversalHandler() - } - - // Convert Express req/res to Request/Response - const request = new Request(`https://${req.hostname}${req.originalUrl}`, { - method: req.method, - headers: req.headers, - body: JSON.stringify(req.body) - }) - - const response = await handler(request) - - res.status(response.status) - res.set(Object.fromEntries(response.headers)) - res.send(await response.text()) -} -``` - -### Google Cloud Run with Cloud Storage - -Google Cloud Run is ideal for containerized deployments with automatic scaling. This example uses Google Cloud Storage via the S3-compatible API. - -```javascript -// server.js -import { Brainy } from '@soulcraft/brainy' -import { S3CompatibleStorage } from '@soulcraft/brainy/storage' -import express from 'express' - -const app = express() -app.use(express.json()) - -const PORT = process.env.PORT || 8080 - -let brain -let handler - -async function initBrainy() { - // Google Cloud Storage is S3-compatible - const storage = new S3CompatibleStorage({ - endpoint: 'storage.googleapis.com', - bucket: process.env.GCS_BUCKET || 'brainy-data', - accessKeyId: process.env.GCS_ACCESS_KEY, - secretAccessKey: process.env.GCS_SECRET_KEY, - prefix: 'brainy/', - forcePathStyle: false, - region: process.env.GCS_REGION || 'US' - }) - - brain = new Brainy({ - storage, - augmentations: [{ - name: 'api-server', - config: { - enabled: true, - port: PORT, - cors: { - origin: process.env.CORS_ORIGIN || '*' - }, - auth: { - required: process.env.AUTH_REQUIRED === 'true', - apiKeys: process.env.API_KEY ? [process.env.API_KEY] : [] - } - } - }] - }) - - await brain.init() - - const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') - handler = apiAugmentation.createUniversalHandler() -} - -// Initialize on startup -await initBrainy() - -// Health check endpoint -app.get('/health', (req, res) => { - res.json({ status: 'healthy', service: 'brainy-api' }) -}) - -// Universal handler for all API routes -app.use('*', async (req, res) => { - const request = new Request(`https://${req.hostname}${req.originalUrl}`, { - method: req.method, - headers: req.headers, - body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined - }) - - const response = await handler(request) - - res.status(response.status) - res.set(Object.fromEntries(response.headers)) - res.send(await response.text()) -}) - -app.listen(PORT, () => { - console.log(`Brainy API running on port ${PORT}`) -}) -``` - -```dockerfile -# Dockerfile -FROM node:20-alpine AS builder -WORKDIR /app -COPY package*.json ./ -RUN npm ci --only=production - -FROM node:20-alpine -WORKDIR /app -COPY --from=builder /app/node_modules ./node_modules -COPY . . -EXPOSE 8080 -CMD ["node", "server.js"] -``` - -```yaml -# cloudbuild.yaml -steps: - # Build the container image - - name: 'gcr.io/cloud-builders/docker' - args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.'] - - # Push to Container Registry - - name: 'gcr.io/cloud-builders/docker' - args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'] - - # Deploy to Cloud Run - - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk' - entrypoint: gcloud - args: - - 'run' - - 'deploy' - - 'brainy-api' - - '--image' - - 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA' - - '--region' - - 'us-central1' - - '--platform' - - 'managed' - - '--allow-unauthenticated' - - '--set-env-vars' - - 'GCS_BUCKET=brainy-storage,GCS_REGION=US' - - '--set-secrets' - - 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest' - - '--memory' - - '2Gi' - - '--cpu' - - '2' - - '--max-instances' - - '100' - - '--min-instances' - - '0' - -images: - - 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA' -``` - -**Deploy with gcloud CLI:** -```bash -# Build and submit to Cloud Build -gcloud builds submit --config cloudbuild.yaml - -# Or deploy directly -gcloud run deploy brainy-api \ - --source . \ - --platform managed \ - --region us-central1 \ - --allow-unauthenticated \ - --set-env-vars GCS_BUCKET=brainy-storage \ - --set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest" -``` - -**Create GCS Bucket with S3-compatible access:** -```bash -# Create bucket -gsutil mb -p PROJECT_ID -c STANDARD -l US gs://brainy-storage/ - -# Enable interoperability -gsutil iam ch serviceAccount:SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com:objectAdmin gs://brainy-storage - -# Generate HMAC keys for S3-compatible access -gsutil hmac create SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com - -# Store the access key and secret in Secret Manager -echo -n "YOUR_ACCESS_KEY" | gcloud secrets create gcs-access-key --data-file=- -echo -n "YOUR_SECRET_KEY" | gcloud secrets create gcs-secret-key --data-file=- -``` - -### Microsoft Azure Functions - -```javascript -// index.js -module.exports = async function (context, req) { - const { Brainy } = require('@soulcraft/brainy') - const { S3CompatibleStorage } = require('@soulcraft/brainy/storage') - - const storage = new S3CompatibleStorage({ - endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`, - bucket: 'brainy-data', - accessKeyId: process.env.AZURE_STORAGE_ACCOUNT, - secretAccessKey: process.env.AZURE_STORAGE_KEY, - prefix: 'entities/', - forcePathStyle: false - }) - - const brain = new Brainy({ storage }) - await brain.init() - - const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') - const handler = apiAugmentation.createUniversalHandler() - - const request = new Request(`https://${context.req.headers.host}${context.req.url}`, { - method: context.req.method, - headers: context.req.headers, - body: JSON.stringify(context.req.body) - }) - - const response = await handler(request) - - context.res = { - status: response.status, - headers: Object.fromEntries(response.headers), - body: await response.text() - } -} -``` - -### Cloudflare Workers - -```javascript -// worker.js -import { Brainy } from '@soulcraft/brainy' -import { R2Storage } from '@soulcraft/brainy/storage' // Alias for S3CompatibleStorage - -let handler - -export default { - async fetch(request, env, ctx) { - if (!handler) { - const storage = new R2Storage({ - endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`, - bucket: 'brainy-data', - accessKeyId: env.R2_ACCESS_KEY_ID, - secretAccessKey: env.R2_SECRET_ACCESS_KEY, - region: 'auto', - forcePathStyle: true - }) - - const brain = new Brainy({ - storage, - augmentations: [{ - name: 'api-server', - config: { - enabled: true, - cors: { origin: '*' } - } - }] - }) - - await brain.init() - - const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') - handler = apiAugmentation.createUniversalHandler() - } - - // The handler works directly with Request/Response! - return handler(request) - } -} -``` - -```toml -# wrangler.toml -name = "brainy-api" -main = "worker.js" -compatibility_date = "2024-01-01" - -[[r2_buckets]] -binding = "R2" -bucket_name = "brainy-data" - -[vars] -ACCOUNT_ID = "your-account-id" - -[env.production.vars] -R2_ACCESS_KEY_ID = "your-access-key" -R2_SECRET_ACCESS_KEY = "your-secret-key" -``` - -### Vercel Edge Functions - -```javascript -// api/brainy.js -import { Brainy } from '@soulcraft/brainy' -import { S3CompatibleStorage } from '@soulcraft/brainy/storage' - -let handler - -export const config = { - runtime: 'edge', -} - -export default async (request) => { - if (!handler) { - const storage = new S3CompatibleStorage({ - endpoint: 's3.amazonaws.com', - region: 'us-east-1', - bucket: process.env.S3_BUCKET, - accessKeyId: process.env.AWS_ACCESS_KEY_ID, - secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY - }) - - const brain = new Brainy({ - storage, - augmentations: [{ - name: 'api-server', - config: { enabled: true } - }] - }) - - await brain.init() - - const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') - handler = apiAugmentation.createUniversalHandler() - } - - return handler(request) -} -``` - -```json -// vercel.json -{ - "functions": { - "api/brainy.js": { - "maxDuration": 30, - "memory": 1024 - } - }, - "rewrites": [ - { - "source": "/api/:path*", - "destination": "/api/brainy" - } - ] -} -``` - -### Railway - -```javascript -// server.js -import { Brainy } from '@soulcraft/brainy' -import { S3CompatibleStorage } from '@soulcraft/brainy/storage' -import express from 'express' - -const app = express() -const PORT = process.env.PORT || 3000 - -let brain -let handler - -async function init() { - const storage = new S3CompatibleStorage({ - endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com', - region: process.env.S3_REGION || 'us-east-1', - bucket: process.env.S3_BUCKET, - accessKeyId: process.env.S3_ACCESS_KEY, - secretAccessKey: process.env.S3_SECRET_KEY, - prefix: 'brainy/' - }) - - brain = new Brainy({ - storage, - augmentations: [{ - name: 'api-server', - config: { - enabled: true, - port: PORT - } - }] - }) - - await brain.init() - - const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') - handler = apiAugmentation.createUniversalHandler() -} - -await init() - -// Universal handler for all routes -app.use('*', async (req, res) => { - const request = new Request(`http://localhost${req.originalUrl}`, { - method: req.method, - headers: req.headers, - body: JSON.stringify(req.body) - }) - - const response = await handler(request) - - res.status(response.status) - res.set(Object.fromEntries(response.headers)) - res.send(await response.text()) -}) - -app.listen(PORT, () => { - console.log(`Brainy API running on port ${PORT}`) -}) -``` - -```toml -# railway.toml -[build] -builder = "nixpacks" -buildCommand = "npm ci" - -[deploy] -startCommand = "node server.js" -restartPolicyType = "always" -restartPolicyMaxRetries = 3 -``` - -### Render - -```javascript -// server.js (same as Railway example above) -// Use S3CompatibleStorage with your preferred object storage provider -``` - -```yaml -# render.yaml -services: - - type: web - name: brainy-api - runtime: node - buildCommand: npm install - startCommand: node server.js - envVars: - - key: S3_BUCKET - value: brainy-data - - key: S3_ENDPOINT - value: s3.amazonaws.com - - key: S3_REGION - value: us-east-1 - - key: S3_ACCESS_KEY - sync: false - - key: S3_SECRET_KEY - sync: false - - key: API_KEY - generateValue: true - healthCheckPath: /health - autoDeploy: true -``` - -### Deno Deploy - -```typescript -// main.ts -import { Brainy } from "npm:@soulcraft/brainy" -import { S3CompatibleStorage } from "npm:@soulcraft/brainy/storage" - -const storage = new S3CompatibleStorage({ - endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com", - bucket: Deno.env.get("S3_BUCKET")!, - accessKeyId: Deno.env.get("S3_ACCESS_KEY")!, - secretAccessKey: Deno.env.get("S3_SECRET_KEY")!, - region: "auto" -}) - -const brain = new Brainy({ - storage, - augmentations: [{ - name: 'api-server', - config: { enabled: true } - }] -}) - -await brain.init() - -const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') -const handler = apiAugmentation.createUniversalHandler() - -Deno.serve(handler) -``` - -## API Endpoints - -The API Server augmentation provides these REST endpoints: - -- `POST /api/brainy/add` - Add entity -- `GET /api/brainy/get?id=xxx` - Get entity by ID -- `PUT /api/brainy/update` - Update entity -- `DELETE /api/brainy/delete?id=xxx` - Delete entity -- `POST /api/brainy/find` - Search/find entities -- `POST /api/brainy/relate` - Create relationship -- `GET /api/brainy/insights` - Get statistics and insights -- `GET /health` - Health check - -## WebSocket Support - -The API Server augmentation includes WebSocket support for real-time updates through the `setupUniversalWebSocket()` method. - -## MCP Support - -Model Context Protocol (MCP) endpoints are available at `/mcp/*` for AI tool integration. - -## Environment Variables - -```bash -# Storage Configuration (S3Compatible) -S3_ENDPOINT=s3.amazonaws.com -S3_REGION=us-east-1 -S3_BUCKET=brainy-data -S3_ACCESS_KEY=xxx -S3_SECRET_KEY=xxx - -# API Configuration -API_KEY=your-secret-key -PORT=3000 - -# CORS -CORS_ORIGIN=* - -# Rate Limiting -RATE_LIMIT_WINDOW=60000 -RATE_LIMIT_MAX=100 -``` - -## Client Usage - -```javascript -// REST API Client -const response = await fetch('https://your-api.com/api/brainy/add', { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - 'Authorization': 'Bearer YOUR_API_KEY' - }, - body: JSON.stringify({ - data: 'Your content here', - metadata: { type: 'document' } - }) -}) - -const { id } = await response.json() - -// Search -const searchResponse = await fetch('https://your-api.com/api/brainy/find', { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - 'Authorization': 'Bearer YOUR_API_KEY' - }, - body: JSON.stringify({ - query: 'neural networks' - }) -}) - -const results = await searchResponse.json() - -// WebSocket Client -const ws = new WebSocket('wss://your-api.com/ws') -ws.onopen = () => { - ws.send(JSON.stringify({ - type: 'subscribe', - pattern: 'technology' - })) -} -ws.onmessage = (event) => { - const update = JSON.parse(event.data) - console.log('Real-time update:', update) -} -``` - -## Storage Adapter Configuration - -S3CompatibleStorage constructor parameters (verified from source): - -```javascript -{ - endpoint: string, // Required (e.g., 's3.amazonaws.com') - bucket: string, // Required - accessKeyId: string, // Required - secretAccessKey: string, // Required - region?: string, // Optional (default: 'us-east-1') - prefix?: string, // Optional (e.g., 'brainy/') - forcePathStyle?: boolean, // Optional (needed for some S3-compatible services) -} -``` - -## Important Notes - -1. All examples use the **real** Brainy 3.0 APIs -2. The `createUniversalHandler()` method is provided by the API Server augmentation -3. S3CompatibleStorage works with any S3-compatible service -4. Always call `brain.init()` before using Brainy -5. The handler can be cached across requests for better performance -6. R2Storage is an alias for S3CompatibleStorage (for Cloudflare R2) - -## Security Best Practices - -1. **Always use environment variables for sensitive data** (API keys, secrets) -2. **Enable authentication** in the API Server augmentation config -3. **Use HTTPS/TLS** for all production deployments -4. **Implement rate limiting** to prevent abuse -5. **Configure CORS** appropriately for your use case - -## Cost Optimization - -### Enable Lifecycle Policies - -**AWS S3: Automatic tier transitions** -```javascript -// After initializing brain with S3CompatibleStorage -const storage = brain.storage - -// Set lifecycle policy for automatic archival -await storage.setLifecyclePolicy({ - rules: [{ - id: 'archive-old-data', - prefix: 'entities/', - status: 'Enabled', - transitions: [ - { days: 30, storageClass: 'STANDARD_IA' }, // $0.0125/GB after 30 days - { days: 90, storageClass: 'GLACIER' }, // $0.004/GB after 90 days - { days: 365, storageClass: 'DEEP_ARCHIVE' } // $0.00099/GB after 1 year - ] - }] -}) - -// Or enable Intelligent-Tiering for hands-off optimization -await storage.enableIntelligentTiering('entities/', 'auto-optimize') -``` - -**Cost Impact (500TB dataset):** -| Storage Class | Cost/GB/Month | 500TB/Year | Savings | -|---------------|---------------|------------|---------| -| Standard | $0.023 | $138,000 | Baseline | -| Intelligent-Tiering | Variable | $6,900 | **95%** | -| With lifecycle policy | Variable | $5,940 | **96%** | - -### Enable Batch Operations - -**Efficient bulk deletions:** -```javascript -// Batch delete (1000 objects per request) -const idsToDelete = [/* array of entity IDs */] -const paths = idsToDelete.flatMap(id => [ - `entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`, - `entities/nouns/metadata/${id.substring(0, 2)}/${id}.json` -]) - -await storage.batchDelete(paths) // Much faster than individual deletes -``` - -### Enable Compression (FileSystem) - -**For local/server deployments:** -```javascript -const storage = new FileSystemStorage({ - path: './data', - compression: true // 60-80% space savings -}) - -const brain = new Brainy({ storage }) -``` - -## Performance Tips - -1. **Cache the brain instance** - Initialize once and reuse across requests -2. **Use S3CompatibleStorage** for cloud deployments (better scalability) -3. **Enable the cache augmentation** for frequently accessed data -4. **Configure appropriate memory limits** for your runtime -5. Enable lifecycle policies to reduce storage costs by 96% -6. Use batch operations for cleanup tasks - -## Troubleshooting - -### Common Issues - -1. **"Brainy not initialized"** - Make sure to call `await brain.init()` before use -2. **"augmentationRegistry.getAugmentation is not a function"** - The augmentation wasn't loaded properly -3. **S3 Access Denied** - Check your IAM permissions and credentials -4. **CORS errors** - Configure the CORS settings in the API Server augmentation - -### Debug Mode - -Enable debug logging by setting: -```javascript -const brain = new Brainy({ - storage, - debug: true, - augmentations: [{ - name: 'api-server', - config: { - enabled: true, - verbose: true - } - }] -}) -``` - -## Support - -- Documentation: https://github.com/soulcraft/brainy/docs -- Issues: https://github.com/soulcraft/brainy/issues -- NPM Package: https://www.npmjs.com/package/@soulcraft/brainy - ---- - -This guide contains only verified, working code from the actual Brainy 3.0 codebase. No mock implementations, no stub methods, only production-ready code. \ No newline at end of file diff --git a/docs/deployment/aws-deployment.md b/docs/deployment/aws-deployment.md deleted file mode 100644 index e80cb8a5..00000000 --- a/docs/deployment/aws-deployment.md +++ /dev/null @@ -1,505 +0,0 @@ -# AWS Deployment Guide for Brainy - -## Overview -Deploy Brainy on AWS with automatic scaling, high availability, and zero-config dynamic adaptation. Brainy automatically adapts to your AWS environment without manual configuration. - -## Quick Start (Zero-Config) - -### Option 1: AWS Lambda (Serverless) - -```bash -# Install Brainy -npm install @soulcraft/brainy - -# Create handler.js -cat > handler.js << 'EOF' -const { Brainy } = require('@soulcraft/brainy') - -let brain - -exports.handler = async (event) => { - // Brainy auto-detects Lambda environment and configures accordingly - if (!brain) { - brain = new Brainy() // Zero config - auto-adapts to Lambda - await brain.init() - } - - const { method, ...params } = JSON.parse(event.body) - - switch(method) { - case 'add': - const id = await brain.add(params) - return { statusCode: 200, body: JSON.stringify({ id }) } - case 'find': - const results = await brain.find(params) - return { statusCode: 200, body: JSON.stringify({ results }) } - default: - return { statusCode: 400, body: 'Unknown method' } - } -} -EOF - -# Deploy with AWS SAM -sam init --runtime nodejs20.x --name brainy-app -sam deploy --guided -``` - -### Option 2: ECS Fargate (Container) - -```bash -# Build and push Docker image -aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $ECR_URI -docker build -t brainy . -docker tag brainy:latest $ECR_URI/brainy:latest -docker push $ECR_URI/brainy:latest - -# Deploy with minimal ECS task definition -cat > task-definition.json << 'EOF' -{ - "family": "brainy", - "networkMode": "awsvpc", - "requiresCompatibilities": ["FARGATE"], - "cpu": "256", - "memory": "512", - "containerDefinitions": [{ - "name": "brainy", - "image": "$ECR_URI/brainy:latest", - "environment": [ - {"name": "NODE_ENV", "value": "production"} - ], - "logConfiguration": { - "logDriver": "awslogs", - "options": { - "awslogs-group": "/ecs/brainy", - "awslogs-region": "us-east-1", - "awslogs-stream-prefix": "ecs" - } - } - }] -} -EOF - -# Brainy auto-detects ECS environment and uses S3 for storage -aws ecs register-task-definition --cli-input-json file://task-definition.json -aws ecs create-service --cluster default --service-name brainy --task-definition brainy --desired-count 2 -``` - -### Option 3: EC2 Auto-Scaling - -```bash -# User data script for EC2 instances -#!/bin/bash -curl -fsSL https://rpm.nodesource.com/setup_20.x | sudo bash - -sudo yum install -y nodejs git - -# Clone and setup (or use your deployment method) -git clone https://github.com/yourorg/brainy-app.git /app -cd /app -npm install --production - -# Create systemd service -cat > /etc/systemd/system/brainy.service << 'EOF' -[Unit] -Description=Brainy -After=network.target - -[Service] -Type=simple -User=ec2-user -WorkingDirectory=/app -ExecStart=/usr/bin/node index.js -Restart=on-failure -Environment=NODE_ENV=production - -[Install] -WantedBy=multi-user.target -EOF - -systemctl start brainy -systemctl enable brainy -``` - -## Zero-Config Storage (Automatic) - -Brainy automatically detects and uses the best available storage: - -```javascript -// No configuration needed - Brainy auto-detects: -const brain = new Brainy() - -// Auto-detection priority: -// 1. S3 (if IAM role has permissions) -// 2. EFS (if mounted at /mnt/efs) -// 3. EBS volume (if available) -// 4. Instance storage (fallback) -``` - -### Manual S3 Configuration (Optional) - -```javascript -const brain = new Brainy({ - storage: { - type: 's3', - options: { - bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket - region: process.env.AWS_REGION || 'auto', // 'auto' detects region - // IAM role provides credentials automatically - } - } -}) -``` - -## Scaling Strategies - -### 1. Horizontal Scaling (Recommended) - -```yaml -# Auto-scaling policy -Resources: - AutoScalingTarget: - Type: AWS::ApplicationAutoScaling::ScalableTarget - Properties: - ServiceNamespace: ecs - ResourceId: service/default/brainy - ScalableDimension: ecs:service:DesiredCount - MinCapacity: 2 - MaxCapacity: 100 - - AutoScalingPolicy: - Type: AWS::ApplicationAutoScaling::ScalingPolicy - Properties: - PolicyType: TargetTrackingScaling - TargetTrackingScalingPolicyConfiguration: - TargetValue: 70.0 - PredefinedMetricSpecification: - PredefinedMetricType: ECSServiceAverageCPUUtilization -``` - -### 2. Vertical Scaling - -Brainy automatically adapts to available memory: -- **256MB**: Minimal mode, optimized caching -- **512MB**: Standard mode, balanced performance -- **1GB+**: Full mode, maximum performance - -## High Availability Setup - -### Multi-AZ Deployment - -```javascript -// Brainy automatically handles multi-AZ with S3 -const brain = new Brainy({ - distributed: { - enabled: true, // Auto-enables with S3 storage - coordinationMethod: 'auto' // Uses S3 for coordination - } -}) -``` - -### Load Balancing - -```bash -# Application Load Balancer with health checks -aws elbv2 create-load-balancer \ - --name brainy-alb \ - --subnets subnet-xxx subnet-yyy \ - --security-groups sg-xxx - -aws elbv2 create-target-group \ - --name brainy-targets \ - --protocol HTTP \ - --port 3000 \ - --vpc-id vpc-xxx \ - --health-check-path /health \ - --health-check-interval-seconds 30 -``` - -## Monitoring & Observability - -### CloudWatch Integration - -Brainy automatically sends metrics when running on AWS: - -```javascript -// Automatic CloudWatch metrics (no config needed) -// - Request count -// - Response time -// - Error rate -// - Storage usage -// - Memory usage -``` - -### Custom Metrics - -```javascript -const brain = new Brainy({ - monitoring: { - enabled: true, - customMetrics: { - namespace: 'Brainy/Production', - dimensions: [ - { Name: 'Environment', Value: 'production' }, - { Name: 'Service', Value: 'api' } - ] - } - } -}) -``` - -## Security Best Practices - -### 1. IAM Role (Recommended) - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "s3:GetObject", - "s3:PutObject", - "s3:DeleteObject", - "s3:ListBucket" - ], - "Resource": [ - "arn:aws:s3:::brainy-*/*", - "arn:aws:s3:::brainy-*" - ] - } - ] -} -``` - -### 2. VPC Configuration - -```bash -# Private subnets with NAT Gateway -aws ec2 create-vpc --cidr-block 10.0.0.0/16 -aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.1.0/24 --availability-zone us-east-1a -aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.2.0/24 --availability-zone us-east-1b -``` - -### 3. Encryption - -```javascript -// Automatic encryption with S3 -const brain = new Brainy({ - storage: { - type: 's3', - options: { - encryption: 'auto' // Uses S3 SSE-S3 by default - } - } -}) -``` - -## Cost Optimization - -### 1. Spot Instances (70% savings) - -```bash -aws ec2 request-spot-fleet --spot-fleet-request-config '{ - "IamFleetRole": "arn:aws:iam::xxx:role/fleet-role", - "TargetCapacity": 2, - "SpotPrice": "0.05", - "LaunchSpecifications": [{ - "ImageId": "ami-xxx", - "InstanceType": "t3.medium", - "UserData": "BASE64_ENCODED_STARTUP_SCRIPT" - }] -}' -``` - -### 2. S3 Intelligent-Tiering - -```javascript -// Brainy automatically uses S3 Intelligent-Tiering -const brain = new Brainy({ - storage: { - type: 's3', - options: { - storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization - } - } -}) -``` - -### 3. Lambda Reserved Concurrency - -```bash -aws lambda put-function-concurrency \ - --function-name brainy-handler \ - --reserved-concurrent-executions 10 -``` - -## Deployment Automation - -### GitHub Actions CI/CD - -```yaml -name: Deploy to AWS -on: - push: - branches: [main] - -jobs: - deploy: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@v1 - with: - aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - aws-region: us-east-1 - - - name: Deploy to ECS - run: | - docker build -t brainy . - docker tag brainy:latest $ECR_URI/brainy:latest - docker push $ECR_URI/brainy:latest - aws ecs update-service --cluster default --service brainy --force-new-deployment -``` - -## Troubleshooting - -### Common Issues - -1. **Storage Auto-Detection Fails** - ```javascript - // Explicitly specify storage - const brain = new Brainy({ - storage: { type: 's3', options: { bucket: 'my-bucket' } } - }) - ``` - -2. **Memory Issues** - ```javascript - // Optimize for low memory - const brain = new Brainy({ - cache: { maxSize: 100 }, // Reduce cache size - index: { M: 8 } // Reduce HNSW connections - }) - ``` - -3. **Cold Starts (Lambda)** - - **Progressive Initialization (Zero-Config)** - - Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME) - and uses progressive initialization for <200ms cold starts: - - ```javascript - // Zero-config - Brainy auto-detects Lambda - const brain = new Brainy({ - storage: { - type: 's3', - s3Storage: { - bucketName: 'my-bucket', - region: 'us-east-1', - accessKeyId: process.env.AWS_ACCESS_KEY_ID, - secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY - } - } - }) - await brain.init() // Returns in <200ms - - // First write validates bucket (lazy validation) - await brain.add('noun', { name: 'test' }) // Validates here - ``` - - **Manual Override (if needed)** - ```javascript - const brain = new Brainy({ - storage: { - type: 's3', - s3Storage: { - bucketName: 'my-bucket', - // Force specific mode - initMode: 'progressive' // 'auto' | 'progressive' | 'strict' - } - } - }) - ``` - - | Mode | Cold Start | Best For | - |------|------------|----------| - | `auto` (default) | <200ms in Lambda | Zero-config, auto-detects | - | `progressive` | <200ms always | Force fast init everywhere | - | `strict` | 100-500ms+ | Local dev, tests, debugging | - - **Warm-up (Alternative)** - ```javascript - exports.warmup = async () => { - if (!brain) { - brain = new Brainy({ warmup: true }) - await brain.init() - } - } - ``` - - **Readiness Detection** - - Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests: - - ```javascript - let brain - - exports.handler = async (event) => { - if (!brain) { - brain = new Brainy({ storage: { type: 's3', ... } }) - brain.init() // Fire and forget - } - - // Wait for initialization to complete - await brain.ready - - // Now safe to use brain methods - const results = await brain.find({ query: event.queryStringParameters.q }) - return { statusCode: 200, body: JSON.stringify(results) } - } - ``` - - For health checks, use `isFullyInitialized()` to verify all background tasks are complete: - - ```javascript - exports.healthCheck = async () => { - try { - await brain.ready - return { - statusCode: 200, - body: JSON.stringify({ - status: 'ready', - fullyInitialized: brain.isFullyInitialized() - }) - } - } catch (error) { - return { - statusCode: 503, - body: JSON.stringify({ status: 'initializing' }) - } - } - } - ``` - -## Production Checklist - -- [ ] IAM roles configured with minimal permissions -- [ ] VPC with private subnets -- [ ] Auto-scaling configured -- [ ] CloudWatch alarms set up -- [ ] Backup strategy (S3 versioning enabled) -- [ ] SSL/TLS certificates configured -- [ ] Rate limiting enabled -- [ ] Health checks configured -- [ ] Monitoring dashboard created -- [ ] Cost alerts configured - -## Support - -- Documentation: https://brainy.soulcraft.ai/docs -- Issues: https://github.com/soulcraft/brainy/issues -- Community: https://discord.gg/brainy \ No newline at end of file diff --git a/docs/deployment/gcp-deployment.md b/docs/deployment/gcp-deployment.md deleted file mode 100644 index 539966c9..00000000 --- a/docs/deployment/gcp-deployment.md +++ /dev/null @@ -1,627 +0,0 @@ -# Google Cloud Platform Deployment Guide for Brainy - -## Overview -Deploy Brainy on GCP with automatic scaling, global distribution, and zero-config dynamic adaptation. Brainy automatically detects and optimizes for GCP services. - -## Quick Start (Zero-Config) - -### Option 1: Cloud Run (Serverless Containers) - -```bash -# Build and deploy with one command -gcloud run deploy brainy \ - --source . \ - --platform managed \ - --region us-central1 \ - --allow-unauthenticated - -# Brainy auto-detects Cloud Run and configures: -# - Memory-optimized caching -# - GCS for storage (if available) -# - Cloud SQL for metadata (if available) -``` - -### Option 2: Cloud Functions (Serverless) - -```javascript -// index.js -const { Brainy } = require('@soulcraft/brainy') - -let brain - -exports.brainyHandler = async (req, res) => { - // Zero-config - auto-adapts to Cloud Functions - if (!brain) { - brain = new Brainy() // Detects GCP environment automatically - await brain.init() - } - - const { method, ...params } = req.body - - try { - let result - switch(method) { - case 'add': - result = await brain.add(params) - break - case 'find': - result = await brain.find(params) - break - case 'relate': - result = await brain.relate(params) - break - default: - return res.status(400).json({ error: 'Unknown method' }) - } - res.json({ result }) - } catch (error) { - res.status(500).json({ error: error.message }) - } -} -``` - -Deploy: -```bash -gcloud functions deploy brainy \ - --runtime nodejs20 \ - --trigger-http \ - --entry-point brainyHandler \ - --memory 512MB \ - --timeout 60s -``` - -### Option 3: Google Kubernetes Engine (GKE) - -```bash -# Create autopilot cluster (fully managed, zero-config) -gcloud container clusters create-auto brainy-cluster \ - --region us-central1 - -# Deploy using Cloud Build -gcloud builds submit --tag gcr.io/$PROJECT_ID/brainy - -# Apply Kubernetes manifest -kubectl apply -f - < { - // Send custom metrics to Cloud Monitoring - await monitoring.createTimeSeries({ - name: monitoring.projectPath(projectId), - timeSeries: [{ - metric: { - type: `custom.googleapis.com/brainy/${metric.name}`, - labels: metric.labels - }, - points: [{ - interval: { endTime: { seconds: Date.now() / 1000 } }, - value: { doubleValue: metric.value } - }] - }] - }) - } -}) -``` - -### Cloud Trace Integration - -```javascript -const brain = new Brainy({ - tracing: { - enabled: true, - sampleRate: 0.1 // Sample 10% of requests - } -}) -``` - -## Security Best Practices - -### 1. Workload Identity (GKE) - -```yaml -# Enable Workload Identity -apiVersion: v1 -kind: ServiceAccount -metadata: - name: brainy-sa - annotations: - iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com -``` - -### 2. Binary Authorization - -```yaml -# Ensure only signed container images -apiVersion: binaryauthorization.grafeas.io/v1beta1 -kind: Policy -metadata: - name: brainy-policy -spec: - defaultAdmissionRule: - requireAttestationsBy: - - projects/PROJECT_ID/attestors/prod-attestor -``` - -### 3. VPC Service Controls - -```bash -# Create VPC Service Perimeter -gcloud access-context-manager perimeters create brainy_perimeter \ - --resources=projects/PROJECT_NUMBER \ - --restricted-services=storage.googleapis.com \ - --title="Brainy Security Perimeter" -``` - -## Cost Optimization - -### 1. Preemptible VMs (80% savings) - -```yaml -# GKE node pool with preemptible VMs -apiVersion: container.cnrm.cloud.google.com/v1beta1 -kind: ContainerNodePool -metadata: - name: brainy-preemptible-pool -spec: - clusterRef: - name: brainy-cluster - config: - preemptible: true - machineType: n2-standard-2 - autoscaling: - minNodeCount: 1 - maxNodeCount: 10 -``` - -### 2. Cloud CDN for Static Assets - -```bash -# Enable Cloud CDN for frequently accessed data -gcloud compute backend-buckets create brainy-assets \ - --gcs-bucket-name=brainy-static - -gcloud compute backend-buckets update brainy-assets \ - --enable-cdn \ - --cache-mode=CACHE_ALL_STATIC -``` - -### 3. Committed Use Discounts - -```bash -# Purchase committed use for predictable workloads -gcloud compute commitments create brainy-commitment \ - --plan=TWELVE_MONTH \ - --resources=vcpu=100,memory=400 -``` - -## Deployment Automation - -### Cloud Build CI/CD - -```yaml -# cloudbuild.yaml -steps: - # Build container - - name: 'gcr.io/cloud-builders/docker' - args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.'] - - # Push to registry - - name: 'gcr.io/cloud-builders/docker' - args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'] - - # Deploy to Cloud Run - - name: 'gcr.io/cloud-builders/gcloud' - args: - - 'run' - - 'deploy' - - 'brainy' - - '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA' - - '--region=us-central1' - - '--platform=managed' - -# Trigger on push to main -trigger: - branch: - name: main -``` - -### Terraform Infrastructure - -```hcl -# main.tf -resource "google_cloud_run_service" "brainy" { - name = "brainy" - location = "us-central1" - - template { - spec { - containers { - image = "gcr.io/${var.project_id}/brainy" - - resources { - limits = { - cpu = "2" - memory = "2Gi" - } - } - - env { - name = "NODE_ENV" - value = "production" - } - } - } - } - - traffic { - percent = 100 - latest_revision = true - } -} - -resource "google_cloud_run_service_iam_member" "public" { - service = google_cloud_run_service.brainy.name - location = google_cloud_run_service.brainy.location - role = "roles/run.invoker" - member = "allUsers" -} -``` - -## Performance Optimization - -### 1. Memory Store (Redis Compatible) - -```javascript -// Brainy can use Memorystore for caching -const brain = new Brainy({ - cache: { - type: 'redis', - options: { - host: process.env.REDIS_HOST || 'auto-detect', - port: 6379 - } - } -}) -``` - -### 2. Cloud Spanner for Global Consistency - -```javascript -const brain = new Brainy({ - metadata: { - type: 'spanner', - options: { - instance: 'brainy-instance', - database: 'brainy-db' - } - } -}) -``` - -## Troubleshooting - -### Common Issues - -1. **Quota Exceeded** - ```bash - # Check quotas - gcloud compute project-info describe --project=$PROJECT_ID - - # Request increase - gcloud compute project-info add-metadata \ - --metadata google-compute-default-region=us-central1 - ``` - -2. **Cold Starts** - - **Progressive Initialization (Zero-Config)** - - Brainy automatically detects Cloud Run and Cloud Functions environments - and uses progressive initialization for <200ms cold starts: - - ```javascript - // Zero-config - Brainy auto-detects Cloud Run (K_SERVICE env var) - const brain = new Brainy({ - storage: { - type: 'gcs', - gcsNativeStorage: { bucketName: 'my-bucket' } - } - }) - await brain.init() // Returns in <200ms - - // First write validates bucket (lazy validation) - await brain.add('noun', { name: 'test' }) // Validates here - ``` - - **Manual Override (if needed)** - ```javascript - const brain = new Brainy({ - storage: { - type: 'gcs', - gcsNativeStorage: { - bucketName: 'my-bucket', - // Force specific mode - initMode: 'progressive' // 'auto' | 'progressive' | 'strict' - } - } - }) - ``` - - | Mode | Cold Start | Best For | - |------|------------|----------| - | `auto` (default) | <200ms in cloud | Zero-config, auto-detects | - | `progressive` | <200ms always | Force fast init everywhere | - | `strict` | 100-500ms+ | Local dev, tests, debugging | - - **Keep Warm (Alternative)** - ```javascript - // Keep minimum instances warm - const brain = new Brainy({ - warmup: { - enabled: true, - interval: 60000 // Ping every minute - } - }) - ``` - - **Readiness Detection** - - Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests: - - ```javascript - let brain - - async function handleRequest(req, res) { - if (!brain) { - brain = new Brainy({ storage: { type: 'gcs', ... } }) - brain.init() // Fire and forget - } - - // Wait for initialization to complete - await brain.ready - - // Now safe to use brain methods - const results = await brain.find({ query: req.query.q }) - res.json(results) - } - ``` - - For Cloud Run health checks, use `isFullyInitialized()` to verify all background tasks are complete: - - ```javascript - // Health check endpoint for Cloud Run - app.get('/health', async (req, res) => { - try { - await brain.ready - res.json({ - status: 'ready', - fullyInitialized: brain.isFullyInitialized() - }) - } catch (error) { - res.status(503).json({ status: 'initializing' }) - } - }) - ``` - -3. **Memory Pressure** - ```javascript - // Optimize for GCP memory constraints - const brain = new Brainy({ - memory: { - mode: 'aggressive', // Aggressive garbage collection - maxHeap: 0.8 // Use 80% of available memory - } - }) - ``` - -## Production Checklist - -- [ ] Enable Workload Identity for secure access -- [ ] Configure Cloud Armor for DDoS protection -- [ ] Set up Cloud KMS for encryption keys -- [ ] Enable VPC Service Controls -- [ ] Configure Cloud IAP for authentication -- [ ] Set up Cloud Monitoring dashboards -- [ ] Configure Error Reporting -- [ ] Enable Cloud Trace -- [ ] Set up budget alerts -- [ ] Configure backup and disaster recovery - -## Support - -- Documentation: https://brainy.soulcraft.ai/docs -- Issues: https://github.com/soulcraft/brainy/issues -- GCP Marketplace: https://console.cloud.google.com/marketplace/product/brainy \ No newline at end of file diff --git a/docs/deployment/kubernetes-deployment.md b/docs/deployment/kubernetes-deployment.md deleted file mode 100644 index 66e076af..00000000 --- a/docs/deployment/kubernetes-deployment.md +++ /dev/null @@ -1,727 +0,0 @@ -# Kubernetes Deployment Guide for Brainy - -## Overview -Deploy Brainy on Kubernetes with automatic scaling, high availability, and zero-config dynamic adaptation. Works with any Kubernetes distribution (vanilla, EKS, GKE, AKS, OpenShift, etc.). - -## Quick Start (Zero-Config) - -### Basic Deployment - -```yaml -# brainy-deployment.yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: brainy - labels: - app: brainy -spec: - replicas: 3 - selector: - matchLabels: - app: brainy - template: - metadata: - labels: - app: brainy - spec: - containers: - - name: brainy - image: soulcraft/brainy:latest - ports: - - containerPort: 3000 - env: - - name: NODE_ENV - value: production - # Brainy auto-detects Kubernetes and configures accordingly - resources: - requests: - memory: "256Mi" - cpu: "100m" - limits: - memory: "1Gi" - cpu: "500m" ---- -apiVersion: v1 -kind: Service -metadata: - name: brainy-service -spec: - selector: - app: brainy - ports: - - protocol: TCP - port: 80 - targetPort: 3000 - type: LoadBalancer -``` - -Deploy: -```bash -kubectl apply -f brainy-deployment.yaml -``` - -## Production-Grade Setup - -### 1. StatefulSet with Persistent Storage - -```yaml -apiVersion: v1 -kind: StorageClass -metadata: - name: brainy-storage -provisioner: kubernetes.io/aws-ebs # Or your cloud provider -parameters: - type: gp3 - iopsPerGB: "10" ---- -apiVersion: apps/v1 -kind: StatefulSet -metadata: - name: brainy -spec: - serviceName: brainy-headless - replicas: 3 - selector: - matchLabels: - app: brainy - template: - metadata: - labels: - app: brainy - spec: - containers: - - name: brainy - image: soulcraft/brainy:latest - ports: - - containerPort: 3000 - env: - - name: NODE_ENV - value: production - - name: BRAINY_STORAGE_TYPE - value: filesystem - - name: BRAINY_STORAGE_PATH - value: /data - volumeMounts: - - name: data - mountPath: /data - livenessProbe: - httpGet: - path: /health - port: 3000 - initialDelaySeconds: 30 - periodSeconds: 10 - readinessProbe: - httpGet: - path: /ready - port: 3000 - initialDelaySeconds: 5 - periodSeconds: 5 - volumeClaimTemplates: - - metadata: - name: data - spec: - accessModes: ["ReadWriteOnce"] - storageClassName: brainy-storage - resources: - requests: - storage: 10Gi -``` - -### 2. Horizontal Pod Autoscaler - -```yaml -apiVersion: autoscaling/v2 -kind: HorizontalPodAutoscaler -metadata: - name: brainy-hpa -spec: - scaleTargetRef: - apiVersion: apps/v1 - kind: Deployment - name: brainy - minReplicas: 2 - maxReplicas: 100 - metrics: - - type: Resource - resource: - name: cpu - target: - type: Utilization - averageUtilization: 70 - - type: Resource - resource: - name: memory - target: - type: Utilization - averageUtilization: 80 - behavior: - scaleUp: - stabilizationWindowSeconds: 60 - policies: - - type: Percent - value: 100 - periodSeconds: 60 - scaleDown: - stabilizationWindowSeconds: 300 - policies: - - type: Percent - value: 10 - periodSeconds: 60 -``` - -### 3. Ingress Configuration - -```yaml -apiVersion: networking.k8s.io/v1 -kind: Ingress -metadata: - name: brainy-ingress - annotations: - kubernetes.io/ingress.class: nginx - cert-manager.io/cluster-issuer: letsencrypt-prod - nginx.ingress.kubernetes.io/rate-limit: "100" - nginx.ingress.kubernetes.io/proxy-body-size: "50m" -spec: - tls: - - hosts: - - api.brainy.example.com - secretName: brainy-tls - rules: - - host: api.brainy.example.com - http: - paths: - - path: / - pathType: Prefix - backend: - service: - name: brainy-service - port: - number: 80 -``` - -## Zero-Config Storage Options - -### Option 1: S3-Compatible Storage (Recommended) - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: brainy-s3-credentials -type: Opaque -data: - access-key: - secret-key: ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: brainy -spec: - template: - spec: - containers: - - name: brainy - env: - - name: BRAINY_STORAGE_TYPE - value: s3 - - name: AWS_ACCESS_KEY_ID - valueFrom: - secretKeyRef: - name: brainy-s3-credentials - key: access-key - - name: AWS_SECRET_ACCESS_KEY - valueFrom: - secretKeyRef: - name: brainy-s3-credentials - key: secret-key - - name: S3_BUCKET - value: brainy-data - - name: AWS_REGION - value: us-east-1 -``` - -### Option 2: MinIO (Self-Hosted S3) - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: minio -spec: - replicas: 1 - selector: - matchLabels: - app: minio - template: - metadata: - labels: - app: minio - spec: - containers: - - name: minio - image: minio/minio:latest - args: - - server - - /data - env: - - name: MINIO_ROOT_USER - value: brainy - - name: MINIO_ROOT_PASSWORD - value: brainy123456 - volumeMounts: - - name: data - mountPath: /data - volumes: - - name: data - persistentVolumeClaim: - claimName: minio-pvc ---- -apiVersion: v1 -kind: Service -metadata: - name: minio-service -spec: - selector: - app: minio - ports: - - port: 9000 - targetPort: 9000 -``` - -### Option 3: Shared NFS Storage - -```yaml -apiVersion: v1 -kind: PersistentVolume -metadata: - name: brainy-nfs-pv -spec: - capacity: - storage: 100Gi - accessModes: - - ReadWriteMany - nfs: - server: nfs-server.example.com - path: /export/brainy ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: brainy-nfs-pvc -spec: - accessModes: - - ReadWriteMany - resources: - requests: - storage: 100Gi -``` - -## High Availability Configuration - -### 1. Pod Anti-Affinity - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: brainy -spec: - template: - spec: - affinity: - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - brainy - topologyKey: kubernetes.io/hostname -``` - -### 2. Pod Disruption Budget - -```yaml -apiVersion: policy/v1 -kind: PodDisruptionBudget -metadata: - name: brainy-pdb -spec: - minAvailable: 2 - selector: - matchLabels: - app: brainy -``` - -### 3. Multi-Zone Deployment - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: brainy -spec: - template: - spec: - topologySpreadConstraints: - - maxSkew: 1 - topologyKey: topology.kubernetes.io/zone - whenUnsatisfiable: DoNotSchedule - labelSelector: - matchLabels: - app: brainy -``` - -## Monitoring & Observability - -### 1. Prometheus Metrics - -```yaml -apiVersion: v1 -kind: Service -metadata: - name: brainy-metrics - annotations: - prometheus.io/scrape: "true" - prometheus.io/port: "9090" - prometheus.io/path: "/metrics" -spec: - selector: - app: brainy - ports: - - name: metrics - port: 9090 - targetPort: 9090 -``` - -### 2. Grafana Dashboard - -```yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: brainy-dashboard -data: - dashboard.json: | - { - "dashboard": { - "title": "Brainy Metrics", - "panels": [ - { - "title": "Request Rate", - "targets": [ - { - "expr": "rate(brainy_requests_total[5m])" - } - ] - }, - { - "title": "Response Time", - "targets": [ - { - "expr": "histogram_quantile(0.95, brainy_response_time)" - } - ] - } - ] - } - } -``` - -### 3. Logging with Fluentd - -```yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: fluentd-config -data: - fluent.conf: | - - @type tail - path /var/log/containers/brainy*.log - pos_file /var/log/fluentd-brainy.log.pos - tag brainy.* - - @type json - - - - - @type elasticsearch - host elasticsearch.logging.svc.cluster.local - port 9200 - logstash_format true - logstash_prefix brainy - -``` - -## Security Best Practices - -### 1. Network Policies - -```yaml -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: brainy-netpol -spec: - podSelector: - matchLabels: - app: brainy - policyTypes: - - Ingress - - Egress - ingress: - - from: - - namespaceSelector: - matchLabels: - name: ingress-nginx - ports: - - protocol: TCP - port: 3000 - egress: - - to: - - namespaceSelector: {} - ports: - - protocol: TCP - port: 443 # HTTPS - - protocol: TCP - port: 9000 # MinIO/S3 -``` - -### 2. Pod Security Policy - -```yaml -apiVersion: policy/v1beta1 -kind: PodSecurityPolicy -metadata: - name: brainy-psp -spec: - privileged: false - allowPrivilegeEscalation: false - requiredDropCapabilities: - - ALL - volumes: - - 'configMap' - - 'emptyDir' - - 'projected' - - 'secret' - - 'persistentVolumeClaim' - runAsUser: - rule: 'MustRunAsNonRoot' - seLinux: - rule: 'RunAsAny' - fsGroup: - rule: 'RunAsAny' -``` - -### 3. RBAC Configuration - -```yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: brainy-sa ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: brainy-role -rules: -- apiGroups: [""] - resources: ["configmaps"] - verbs: ["get", "list", "watch"] -- apiGroups: [""] - resources: ["secrets"] - verbs: ["get"] ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: brainy-rolebinding -subjects: -- kind: ServiceAccount - name: brainy-sa -roleRef: - kind: Role - name: brainy-role - apiGroup: rbac.authorization.k8s.io -``` - -## GitOps with ArgoCD - -```yaml -apiVersion: argoproj.io/v1alpha1 -kind: Application -metadata: - name: brainy - namespace: argocd -spec: - project: default - source: - repoURL: https://github.com/yourorg/brainy-k8s - targetRevision: HEAD - path: manifests - destination: - server: https://kubernetes.default.svc - namespace: brainy - syncPolicy: - automated: - prune: true - selfHeal: true - syncOptions: - - CreateNamespace=true -``` - -## Helm Chart Installation - -```bash -# Add Brainy Helm repository -helm repo add brainy https://charts.brainy.io -helm repo update - -# Install with custom values -cat > values.yaml << EOF -replicaCount: 3 -image: - repository: soulcraft/brainy - tag: latest - pullPolicy: IfNotPresent - -service: - type: LoadBalancer - port: 80 - -ingress: - enabled: true - className: nginx - hosts: - - host: api.brainy.example.com - paths: - - path: / - pathType: Prefix - -resources: - limits: - cpu: 1000m - memory: 1Gi - requests: - cpu: 100m - memory: 256Mi - -autoscaling: - enabled: true - minReplicas: 2 - maxReplicas: 100 - targetCPUUtilizationPercentage: 70 - -storage: - type: s3 - s3: - bucket: brainy-data - region: us-east-1 -EOF - -helm install brainy brainy/brainy -f values.yaml -``` - -## Cost Optimization - -### 1. Spot/Preemptible Nodes - -```yaml -apiVersion: v1 -kind: NodePool -metadata: - name: brainy-spot-pool -spec: - nodeSelector: - node.kubernetes.io/lifecycle: spot - taints: - - key: spot - value: "true" - effect: NoSchedule - tolerations: - - key: spot - operator: Equal - value: "true" - effect: NoSchedule -``` - -### 2. Vertical Pod Autoscaler - -```yaml -apiVersion: autoscaling.k8s.io/v1 -kind: VerticalPodAutoscaler -metadata: - name: brainy-vpa -spec: - targetRef: - apiVersion: apps/v1 - kind: Deployment - name: brainy - updatePolicy: - updateMode: "Auto" - resourcePolicy: - containerPolicies: - - containerName: brainy - minAllowed: - cpu: 100m - memory: 128Mi - maxAllowed: - cpu: 2 - memory: 2Gi -``` - -## Troubleshooting - -### Common Issues - -1. **Pod CrashLoopBackOff** - ```bash - kubectl logs -f pod/brainy-xxx - kubectl describe pod brainy-xxx - ``` - -2. **Storage Issues** - ```bash - kubectl get pv,pvc - kubectl describe pvc brainy-data - ``` - -3. **Network Connectivity** - ```bash - kubectl exec -it pod/brainy-xxx -- curl http://brainy-service/health - kubectl get endpoints brainy-service - ``` - -4. **Memory Pressure** - ```bash - kubectl top pods -l app=brainy - kubectl describe node - ``` - -## Production Checklist - -- [ ] High availability with multiple replicas -- [ ] Pod disruption budgets configured -- [ ] Resource limits and requests set -- [ ] Horizontal and vertical autoscaling enabled -- [ ] Persistent storage configured -- [ ] Network policies in place -- [ ] RBAC properly configured -- [ ] Monitoring and alerting setup -- [ ] Backup and disaster recovery plan -- [ ] Security scanning enabled -- [ ] GitOps deployment pipeline - -## Support - -- Documentation: https://brainy.soulcraft.ai/docs -- Helm Charts: https://github.com/soulcraft/brainy-charts -- Issues: https://github.com/soulcraft/brainy/issues -- Slack: https://brainy-community.slack.com \ No newline at end of file diff --git a/docs/guides/storage-adapters.md b/docs/guides/storage-adapters.md index 9c81d7fc..2f90e666 100644 --- a/docs/guides/storage-adapters.md +++ b/docs/guides/storage-adapters.md @@ -5,197 +5,144 @@ public: true category: guides template: guide order: 2 -description: "Six adapters for every deployment: in-memory, OPFS (browser), filesystem, S3, Google Cloud Storage, Azure Blob, and Cloudflare R2. All support copy-on-write branching." +description: "Two adapters cover every deployment: in-memory for tests + ephemeral workloads, filesystem for everything that needs to persist. Both support copy-on-write branching. Cloud backup is operator tooling, not a built-in adapter." next: - guides/plugins - concepts/zero-config --- -# Storage Adapters Guide +# Storage Adapters -## Overview -Brainy supports 6 storage adapters for different deployment environments. All adapters implement the same StorageAdapter interface and support copy-on-write branching. +Brainy 8.0 ships **two storage adapters**: -## Adapters +- **`FileSystemStorage`** — persistent on-disk storage. The default for any + deployment that needs to survive a restart. Runs on Node.js, Bun, and Deno. +- **`MemoryStorage`** — in-memory only. The right choice for tests, ephemeral + workloads, and short-lived demos. -### 1. MemoryStorage -- **File**: `src/storage/adapters/memoryStorage.ts` -- **Use case**: Development, testing, prototyping -- **Configuration**: None required -- **Persistence**: None (data lost on restart) -- **Batch config**: 1000 batch size, 0ms delay, 1000 concurrent ops, 100k ops/sec +Both implement the same `StorageAdapter` interface, support copy-on-write +branching, and use the same on-disk layout (memory's "disk" is a JS Map). -```typescript -const brain = new Brainy({ storage: { type: 'memory' } }) +## Quick start + +```ts +import { Brainy } from '@soulcraft/brainy' + +// Filesystem (recommended for any persistent workload): +const brain = new Brainy({ + storage: { type: 'filesystem', rootDirectory: './brainy-data' } +}) + +// Memory (tests, ephemeral): +const brainMem = new Brainy({ storage: { type: 'memory' } }) + +// Auto-detect (filesystem on Node-like runtimes, memory in browsers): +const brainAuto = new Brainy({ storage: { type: 'auto' } }) ``` -### 2. FileSystemStorage -- **File**: `src/storage/adapters/fileSystemStorage.ts` -- **Use case**: Node.js local persistence, single-server deployments -- **Configuration**: `basePath` (required), `readOnly` (optional) -- **Features**: zlib compression, atomic writes via temp files, UUID-based sharding +## When to use which -```typescript -const brain = new Brainy({ - storage: { type: 'filesystem', path: './brainy-data' } -}) +| Use case | Adapter | Why | +|---|---|---| +| Production app | `filesystem` | Durable, branchable, mmap-able | +| Tests, CI | `memory` | No disk teardown; fast | +| Short-lived data pipeline | `memory` | No persistence needed | +| In-browser demo | `memory` | Filesystem unavailable in browsers | +| Cloud deployment | `filesystem` on local disk + operator backup | See "Cloud backup" below | + +## Cloud backup — operator tooling, not a built-in + +Brainy 8.0 deliberately ships **no cloud storage adapters**. Cloud backup is +handled at the operator layer with standard tooling, the same pattern every +production database uses (Postgres, SQLite, Redis): + +```bash +# After a brainy flush, sync the on-disk artefact to your cloud of choice. +gsutil rsync -r /var/lib/brainy gs://my-backup-bucket/brainy/ +# or: +aws s3 sync /var/lib/brainy s3://my-backup-bucket/brainy/ +# or: +rclone sync /var/lib/brainy remote:brainy-backups/ +# or: +azcopy sync /var/lib/brainy "https://account.blob.core.windows.net/brainy?sv=..." ``` -### 3. S3CompatibleStorage -- **File**: `src/storage/adapters/s3CompatibleStorage.ts` -- **Use case**: AWS S3, MinIO, DigitalOcean Spaces, any S3-compatible service -- **Configuration**: `bucketName`, `region`, `accessKeyId`, `secretAccessKey`, `endpoint?` (for custom endpoints), `s3ForcePathStyle?` -- **Features**: Write buffering, request coalescing, throttling detection (429/503), progressive initialization -- **Batch config**: 1000 batch size, 150 concurrent ops, 5000 ops/sec burst +Brainy's filesystem layout is sync-friendly: +- Atomic writes (temp + rename) — readers never see torn files +- Per-shard files — `rsync`-style incremental sync works well +- Content-addressed blobs (`_blobs/`) — immutable, cache-friendly +- Branch refs live under `_cow/` — pick up branches automatically -```typescript -const brain = new Brainy({ - storage: { - type: 's3', - s3Storage: { - bucketName: 'my-brainy-data', - region: 'us-east-1', - accessKeyId: process.env.AWS_ACCESS_KEY_ID, - secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY - } - } -}) +For point-in-time backups, take a filesystem snapshot (ZFS, btrfs, LVM, EBS, +etc.) or use `brain.persist(path)` to write a self-contained snapshot you +can sync independently of the live brain. -// For MinIO or other S3-compatible services: -const brain = new Brainy({ - storage: { - type: 's3', - s3Storage: { - bucketName: 'my-data', - region: 'us-east-1', - endpoint: 'http://localhost:9000', - s3ForcePathStyle: true, - accessKeyId: 'minio-key', - secretAccessKey: 'minio-secret' - } - } -}) +## Why no cloud adapters in 8.0? + +Cloud storage adapters lived in Brainy 4.x-7.x. They were dropped in 8.0 +per **BR-BRAINY-80-STORAGE-SIMPLIFY** because: + +- Zero production consumers used them at scale. Every Soulcraft consumer + ran on local filesystem. +- Cloud-storage HNSW / DiskANN doesn't perform — vector indexes need + low-latency random reads that S3 / GCS / R2 / Azure can't provide + consistently. +- Bundling cloud SDKs into the library cost ~3000-5000 LOC + 4-7 transitive + dependencies for a feature nobody used. +- Cloud backup via operator tooling is strictly more reliable than in-app + upload (better retry semantics, better observability, better cost control). + +Brainy 8.0 is smaller, faster to install, and clearer about what it does. + +## Configuration + +```ts +interface StorageOptions { + // The adapter type. Defaults to 'auto'. + type?: 'auto' | 'memory' | 'filesystem' + + // Force a specific adapter regardless of type. + forceMemoryStorage?: boolean + forceFileSystemStorage?: boolean + + // Filesystem only. + rootDirectory?: string + + // COW branch to open. Defaults to 'main'. + branch?: string + + // COW compression toggle. Defaults to true. + enableCompression?: boolean +} ``` -### 4. R2Storage (Cloudflare) -- **File**: `src/storage/adapters/r2Storage.ts` -- **Use case**: Zero egress fees, cost-effective cloud storage -- **Configuration**: `bucketName`, `accountId`, `accessKeyId`, `secretAccessKey`, `cacheConfig?` -- **Features**: Zero egress fees, aggressive caching, write buffering, request coalescing -- **Batch config**: 1000 batch size, 150 concurrent ops, 6000 ops/sec +## Direct construction -```typescript -const brain = new Brainy({ - storage: { - type: 'r2', - r2Storage: { - accountId: process.env.CF_ACCOUNT_ID, - bucketName: 'my-brainy-data', - accessKeyId: process.env.CF_ACCESS_KEY_ID, - secretAccessKey: process.env.CF_SECRET_ACCESS_KEY - } - } -}) +If you want to skip the factory: + +```ts +import { FileSystemStorage, MemoryStorage } from '@soulcraft/brainy' + +const fsStorage = new FileSystemStorage('./brainy-data') +const memStorage = new MemoryStorage() + +const brain = new Brainy({ storage: fsStorage }) ``` -### 5. GcsStorage (Google Cloud) -- **File**: `src/storage/adapters/gcsStorage.ts` -- **Use case**: Google Cloud ecosystem, Cloud Run deployments -- **Authentication priority**: - 1. Service Account Key File (`keyFilename`) - 2. Credentials Object (`credentials`) - 3. HMAC Keys (backward compat) - 4. Application Default Credentials (automatic in Cloud Run) -- **Features**: Progressive initialization (fast cold starts <200ms), Autoclass lifecycle, bucket validation -- **Batch config**: 1000 batch size, 100 concurrent ops, 1000 ops/sec +## Migration from 7.x cloud adapters -```typescript -// With explicit credentials -const brain = new Brainy({ - storage: { - type: 'gcs', - gcsStorage: { - bucketName: 'my-brainy-data', - keyFilename: './service-account.json' - } - } -}) +7.x consumers of `OPFSStorage`, `GcsStorage`, `R2Storage`, `S3CompatibleStorage`, +or `AzureBlobStorage` need to migrate to `FileSystemStorage` plus operator +backup tooling. The recipe: -// In Cloud Run (uses Application Default Credentials automatically) -const brain = new Brainy({ - storage: { - type: 'gcs', - gcsStorage: { - bucketName: 'my-brainy-data' - } - } -}) -``` +1. On the host running Brainy, mount a local disk (NVMe recommended). Cloud + providers all expose persistent local disks: GCP Persistent Disk, AWS EBS, + Azure Managed Disks. +2. Set `storage: { type: 'filesystem', rootDirectory: '/mnt/brainy-data' }`. +3. Run your existing data import once into the new local store. +4. Set up an operator backup job using `gsutil` / `aws s3` / `rclone` / + `azcopy` on a cron — hourly or whatever your RPO requires. Point it at + the brainy data dir. +5. For point-in-time backups, use filesystem snapshots or `brain.persist()`. -**Progressive Initialization Modes:** -- `'strict'` (default locally): Full validation during init (100-500ms+) -- `'progressive'` (default in Cloud Run/Lambda): Fast init <200ms, background validation -- `'auto'`: Auto-detects environment - -### 6. AzureBlobStorage -- **File**: `src/storage/adapters/azureBlobStorage.ts` -- **Use case**: Azure ecosystem, enterprise deployments -- **Authentication priority**: - 1. DefaultAzureCredential (Managed Identity - automatic in Azure) - 2. Connection String - 3. Account Name + Account Key - 4. SAS Token -- **Features**: Progressive initialization, lifecycle management, container validation -- **Batch config**: 1000 batch size, 100 concurrent ops, 3000 ops/sec - -```typescript -// With connection string -const brain = new Brainy({ - storage: { - type: 'azure', - azureStorage: { - connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING, - containerName: 'brainy-data' - } - } -}) - -// With Managed Identity (automatic in Azure App Service/Functions) -const brain = new Brainy({ - storage: { - type: 'azure', - azureStorage: { - containerName: 'brainy-data' - // DefaultAzureCredential used automatically - } - } -}) -``` - -## Choosing an Adapter - -| Scenario | Recommended Adapter | -|----------|-------------------| -| Development/Testing | MemoryStorage | -| Local persistence | FileSystemStorage | -| AWS deployment | S3CompatibleStorage | -| Google Cloud / Cloud Run | GcsStorage | -| Azure deployment | AzureBlobStorage | -| Cost-sensitive (high egress) | R2Storage | -| Self-hosted (MinIO) | S3CompatibleStorage | - -## Common Configuration - -All cloud adapters support: -- **Progressive initialization** for fast serverless cold starts -- **Read-only mode** (`readOnly: true`) -- **Cache configuration** (`cacheConfig: { hotCacheMaxSize, warmCacheTTL }`) -- **Copy-on-write branching** for git-style data management - -## See Also - -- [Cloud Deployment Guide](../deployment/CLOUD_DEPLOYMENT_GUIDE.md) -- [Cost Optimization: AWS S3](../operations/cost-optimization-aws-s3.md) -- [Cost Optimization: GCS](../operations/cost-optimization-gcs.md) -- [Cost Optimization: Azure](../operations/cost-optimization-azure.md) -- [Cost Optimization: R2](../operations/cost-optimization-cloudflare-r2.md) +Same data, same APIs, no library-side cloud code. diff --git a/docs/operations/cloud-run-filestore-guide.md b/docs/operations/cloud-run-filestore-guide.md deleted file mode 100644 index 76907278..00000000 --- a/docs/operations/cloud-run-filestore-guide.md +++ /dev/null @@ -1,138 +0,0 @@ -# Cloud Run + Filestore (NFS) Deployment Guide - -> Eliminate cloud storage rate limiting by using brainy's FileSystem adapter with a Filestore NFS mount on Cloud Run. - -## Why Filestore? - -Brainy's metadata index generates 26-40 file writes per `brain.add()` call. On GCS, each write is a cloud API call subject to rate limiting (~1 write/sec per object), causing HTTP 429 errors and 61-second latency for a single add operation. - -With Filestore: -- **Local disk speed**: Each write is ~1ms (vs 50-300ms for cloud API) -- **No rate limits**: NFS has no per-object write throttling -- **Zero code changes**: Use brainy's existing `FileSystemStorage` adapter -- **Same Cloud Run deployment**: Just add a volume mount - -## Prerequisites - -- Google Cloud project with Filestore API enabled -- Cloud Run service (gen2 execution environment required for NFS) -- VPC connector or Direct VPC egress configured - -## Step 1: Create a Filestore Instance - -```bash -gcloud filestore instances create brainy-store \ - --zone=us-central1-b \ - --tier=BASIC_SSD \ - --file-share=name=brainy_data,capacity=1TB \ - --network=name=default -``` - -**Tier recommendations:** -- **BASIC_SSD** (~$370/month for 1 TiB): Good balance of performance and cost -- **BASIC_HDD** (~$204/month for 1 TiB): Lower cost, sufficient for moderate workloads -- **ENTERPRISE** or **ZONAL**: Higher IOPS for heavy workloads - -Note the IP address from the output — you'll need it for the Cloud Run mount. - -## Step 2: Configure Cloud Run NFS Volume Mount - -```yaml -# service.yaml -apiVersion: serving.knative.dev/v1 -kind: Service -metadata: - name: my-brainy-service -spec: - template: - metadata: - annotations: - run.googleapis.com/execution-environment: gen2 - run.googleapis.com/vpc-access-connector: projects/PROJECT/locations/REGION/connectors/CONNECTOR - spec: - containers: - - image: gcr.io/PROJECT/my-brainy-app - volumeMounts: - - name: brainy-nfs - mountPath: /mnt/brainy - volumes: - - name: brainy-nfs - nfs: - server: FILESTORE_IP # e.g., 10.0.0.2 - path: /brainy_data -``` - -Deploy: - -```bash -gcloud run services replace service.yaml --region=us-central1 -``` - -## Step 3: Configure Brainy to Use FileSystem Adapter - -```typescript -import { Brainy } from '@soulcraft/brainy' -import { FileSystemStorage } from '@soulcraft/brainy/storage' - -const storage = new FileSystemStorage({ - basePath: '/mnt/brainy/brain-data' -}) - -const brain = new Brainy({ storage }) -await brain.init() -``` - -That's it. No GCS adapter, no rate limits, no retry logic needed. - -## Caveats - -### Single-Writer Recommendation - -NFS does not provide strong file locking guarantees on Cloud Run. For best results: - -- Use a **single Cloud Run instance** (max-instances=1) for write operations -- Multiple read-only instances can safely read from the same Filestore mount -- For multi-writer scenarios, use the GCS adapter with the write buffer (rate limit protection built in) - -### Filestore Permissions - -Cloud Run gen2 instances run as a specific service account. Ensure the Filestore instance allows access from your VPC network. No additional IAM permissions are needed — Filestore uses NFS network-level access. - -### Cost Comparison - -| Solution | Monthly Cost (1 TiB) | Write Latency | Rate Limits | -|----------|---------------------|---------------|-------------| -| GCS Standard | ~$20 storage + API ops | 50-300ms/write | 1 write/sec/object | -| GCS HNS | ~$20 storage + API ops | 50-300ms/write | 8,000 writes/sec | -| Filestore SSD | ~$370 fixed | ~1ms/write | None | -| Filestore HDD | ~$204 fixed | ~5ms/write | None | - -Filestore costs more for storage but eliminates all rate limiting issues and provides significantly lower write latency. - -### When to Choose Filestore vs GCS - -**Choose Filestore when:** -- Write-heavy workloads (frequent `brain.add()`, chat applications) -- Low-latency requirements -- Single-writer architecture is acceptable - -**Choose GCS (with write buffer) when:** -- Multi-instance deployments need shared storage -- Cost optimization for large datasets (lifecycle policies, Autoclass) -- Read-heavy workloads with infrequent writes - -## Verification - -After deploying, verify the mount works: - -```bash -# In Cloud Run container -ls -la /mnt/brainy/ -# Should show the Filestore share contents - -# Test write performance -dd if=/dev/zero of=/mnt/brainy/test bs=1M count=100 -# Expected: ~100MB/s+ for SSD tier -``` - -Then run your brainy application and confirm `brain.add()` completes without rate limit errors. diff --git a/docs/operations/cost-optimization-aws-s3.md b/docs/operations/cost-optimization-aws-s3.md deleted file mode 100644 index 4bb379d2..00000000 --- a/docs/operations/cost-optimization-aws-s3.md +++ /dev/null @@ -1,399 +0,0 @@ -# AWS S3 Cost Optimization Guide for Brainy -> **Cost Impact**: Reduce storage costs from $138k/year to $5.9k/year at 500TB scale (**96% savings**) - -## Overview - -Brainy provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance. - -## Cost Breakdown (Before Optimization) - -### Standard S3 Storage Costs (500TB Dataset) - -``` -Storage: 500TB × $0.023/GB/month × 12 months = $138,000/year -Operations: ~$5,000/year (PUT, GET, LIST requests) -Total: $143,000/year -``` - -## S3 Storage Tiers - -| Tier | Cost/GB/Month | Retrieval Fee | Use Case | -|------|---------------|---------------|----------| -| **Standard** | $0.023 | None | Frequently accessed | -| **Standard-IA** | $0.0125 | $0.01/GB | Infrequently accessed (30+ days) | -| **Intelligent-Tiering** | $0.023-0.00099 | None | Automatic optimization | -| **Glacier Instant** | $0.004 | $0.03/GB | Rare access, instant retrieval | -| **Glacier Flexible** | $0.0036 | $0.01/GB + time | Archive (minutes-hours retrieval) | -| **Glacier Deep Archive** | $0.00099 | $0.02/GB + time | Long-term archive (12 hours retrieval) | - -## Strategy 1: Lifecycle Policies (Recommended) - -### Setup: Automatic Tier Transitions - -**Best for**: Predictable access patterns, batch workloads, archival data - -```typescript -import { Brainy } from '@soulcraft/brainy' -import { S3CompatibleStorage } from '@soulcraft/brainy/storage' - -// Initialize Brainy with S3 storage -const storage = new S3CompatibleStorage({ - bucket: 'my-brainy-data', - region: 'us-east-1', - accessKeyId: process.env.AWS_ACCESS_KEY_ID, - secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY -}) - -const brain = new Brainy({ storage }) -await brain.init() - -// Set lifecycle policy for automatic archival -await storage.setLifecyclePolicy({ - rules: [{ - id: 'optimize-vectors', - prefix: 'entities/nouns/vectors/', - status: 'Enabled', - transitions: [ - { days: 30, storageClass: 'STANDARD_IA' }, // Infrequent Access after 30 days - { days: 90, storageClass: 'GLACIER' }, // Glacier after 90 days - { days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 1 year - ] - }, { - id: 'optimize-metadata', - prefix: 'entities/nouns/metadata/', - status: 'Enabled', - transitions: [ - { days: 30, storageClass: 'STANDARD_IA' }, - { days: 180, storageClass: 'GLACIER' } - ] - }] -}) - -// Verify lifecycle policy -const policy = await storage.getLifecyclePolicy() -console.log('Lifecycle policy active:', policy.rules.length, 'rules') -``` - -### Cost Calculation (500TB with Lifecycle Policy) - -**Assumptions:** -- 40% of data accessed in last 30 days (Standard) -- 30% of data 30-90 days old (Standard-IA) -- 20% of data 90-365 days old (Glacier) -- 10% of data 365+ days old (Deep Archive) - -``` -Standard (200TB): 200TB × $0.023/GB × 12 = $55,200/year -Standard-IA (150TB): 150TB × $0.0125/GB × 12 = $22,500/year -Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year -Deep Archive (50TB): 50TB × $0.00099/GB × 12 = $594/year - -Total Storage Cost: $83,094/year (instead of $138,000) -Total with Operations: ~$88,000/year -Savings: $55,000/year (40%) -``` - -**But we can do better with Intelligent-Tiering...** - -## Strategy 2: Intelligent-Tiering (Most Recommended) - -### Setup: Automatic Access-Based Optimization - -**Best for**: Unpredictable access patterns, mixed workloads, maximum automation - -```typescript -// Enable Intelligent-Tiering for automatic optimization -await storage.enableIntelligentTiering('entities/', 'brainy-auto-optimize') - -// Benefits: -// - Automatically moves objects between tiers based on access patterns -// - No retrieval fees (unlike Glacier) -// - Transitions happen within 24-48 hours of last access -// - Supports Archive Access tier (90+ days) and Deep Archive Access tier (180+ days) -``` - -### Intelligent-Tiering Tiers - -Intelligent-Tiering automatically moves objects between: - -1. **Frequent Access**: $0.023/GB/month (0-30 days) -2. **Infrequent Access**: $0.0125/GB/month (30-90 days) -3. **Archive Access**: $0.004/GB/month (90-180 days) -4. **Deep Archive Access**: $0.00099/GB/month (180+ days) - -**Monitoring Fee**: $0.0025 per 1000 objects (minimal) - -### Cost Calculation (500TB with Intelligent-Tiering) - -**Realistic distribution after 1 year:** -- 15% Frequent Access (hot data) -- 20% Infrequent Access -- 35% Archive Access -- 30% Deep Archive Access - -``` -Frequent (75TB): 75TB × $0.023/GB × 12 = $20,700/year -Infrequent (100TB): 100TB × $0.0125/GB × 12 = $15,000/year -Archive (175TB): 175TB × $0.004/GB × 12 = $8,400/year -Deep Archive (150TB): 150TB × $0.00099/GB × 12 = $1,782/year -Monitoring: ~$300/year (minimal) - -Total Storage Cost: $46,182/year -Total with Operations: ~$51,000/year -Savings vs Standard: $92,000/year (67%) -``` - -## Strategy 3: Hybrid Approach (Maximum Savings) - -### Setup: Lifecycle + Intelligent-Tiering - -**Best for**: Maximum cost optimization with fine-grained control - -```typescript -// Enable Intelligent-Tiering for vectors (frequently searched) -await storage.enableIntelligentTiering('entities/nouns/vectors/', 'vectors-auto') -await storage.enableIntelligentTiering('entities/verbs/vectors/', 'verbs-auto') - -// Set lifecycle policy for metadata (less frequently accessed) -await storage.setLifecyclePolicy({ - rules: [{ - id: 'archive-old-metadata', - prefix: 'entities/nouns/metadata/', - status: 'Enabled', - transitions: [ - { days: 30, storageClass: 'STANDARD_IA' }, - { days: 60, storageClass: 'GLACIER' }, - { days: 180, storageClass: 'DEEP_ARCHIVE' } - ] - }, { - id: 'cleanup-old-system-data', - prefix: '_system/', - status: 'Enabled', - expiration: { days: 365 } // Delete old statistics - }] -}) -``` - -### Cost Calculation (500TB Hybrid Approach) - -**Vectors (300TB with Intelligent-Tiering):** -``` -Frequent (45TB): 45TB × $0.023/GB × 12 = $12,420/year -Infrequent (60TB): 60TB × $0.0125/GB × 12 = $9,000/year -Archive (105TB): 105TB × $0.004/GB × 12 = $5,040/year -Deep Archive (90TB): 90TB × $0.00099/GB × 12 = $1,069/year -Subtotal: $27,529/year -``` - -**Metadata (200TB with Lifecycle Policy):** -``` -Standard (60TB): 60TB × $0.023/GB × 12 = $16,560/year -Standard-IA (40TB): 40TB × $0.0125/GB × 12 = $6,000/year -Glacier (60TB): 60TB × $0.004/GB × 12 = $2,880/year -Deep Archive (40TB): 40TB × $0.00099/GB × 12 = $475/year -Subtotal: $25,915/year -``` - -**Total Cost: $53,444/year + operations (~$58,500/year total)** -**Savings vs Standard: $84,500/year (61%)** - -## Strategy 4: Aggressive Archival (Maximum Savings) - -### Setup: Fast Archival for Cold Data - -**Best for**: Archival workloads, historical data, compliance - -```typescript -await storage.setLifecyclePolicy({ - rules: [{ - id: 'aggressive-archival', - prefix: 'entities/', - status: 'Enabled', - transitions: [ - { days: 14, storageClass: 'STANDARD_IA' }, // IA after 2 weeks - { days: 30, storageClass: 'GLACIER' }, // Glacier after 1 month - { days: 90, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 3 months - ] - }] -}) -``` - -### Cost Calculation (500TB Aggressive Archival) - -**After 1 year:** -``` -Standard (50TB): 50TB × $0.023/GB × 12 = $13,800/year -Standard-IA (50TB): 50TB × $0.0125/GB × 12 = $7,500/year -Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year -Deep Archive (300TB): 300TB × $0.00099/GB × 12 = $3,564/year - -Total Storage Cost: $29,664/year -Total with Operations: ~$34,000/year -Savings: $109,000/year (76%) - -Note: Retrieval costs may be significant if archived data is accessed frequently -``` - -## Comparison Table: All Strategies - -| Strategy | Annual Cost | Savings | Retrieval Speed | Best For | -|----------|-------------|---------|-----------------|----------| -| **No Optimization** | $143,000 | 0% | Instant | N/A | -| **Lifecycle Policy** | $88,000 | 38% | Varies | Predictable patterns | -| **Intelligent-Tiering** | $51,000 | 64% | Instant (no retrieval fees) | **Recommended** | -| **Hybrid Approach** | $58,500 | 59% | Instant for vectors | Fine-grained control | -| **Aggressive Archival** | $34,000 | 76% | Hours to 12 hours | Cold data, compliance | - -## Batch Delete Operations - -### Efficient Cleanup - -```typescript -// Batch delete (1000 objects per request) -const idsToDelete = [/* array of entity IDs */] - -// Generate paths for both vector and metadata files -const paths = idsToDelete.flatMap(id => { - const shard = id.substring(0, 2) - return [ - `entities/nouns/vectors/${shard}/${id}.json`, - `entities/nouns/metadata/${shard}/${id}.json` - ] -}) - -// Batch delete (much faster and cheaper than individual deletes) -await storage.batchDelete(paths) - -// Cost impact: -// - Individual deletes: 1M objects × $0.005 per 1000 = $5,000 -// - Batch deletes: 1M/1000 × $0.005 = $5 (1000x cheaper!) -``` - -## Monitoring and Optimization - -### Get Current Lifecycle Policy - -```typescript -const policy = await storage.getLifecyclePolicy() -console.log('Active rules:', policy.rules) - -// Example output: -// { -// rules: [ -// { -// id: 'optimize-vectors', -// prefix: 'entities/nouns/vectors/', -// status: 'Enabled', -// transitions: [...] -// } -// ] -// } -``` - -### Remove Lifecycle Policy (if needed) - -```typescript -// Remove all lifecycle rules -await storage.removeLifecyclePolicy() -``` - -### Check Intelligent-Tiering Configurations - -```typescript -const configs = await storage.getIntelligentTieringConfigs() -console.log('Active configurations:', configs) -``` - -### Disable Intelligent-Tiering - -```typescript -await storage.disableIntelligentTiering('brainy-auto-optimize') -``` - -## AWS Cost Explorer Analysis - -### Track Your Savings - -1. **Enable Cost Explorer** in AWS Console -2. **Group by Storage Class** to see tier distribution -3. **Set up Cost Anomaly Detection** for unexpected spikes -4. **Create Budget Alerts** for monthly storage costs - -### Expected Metrics After 6 Months - -``` -Standard storage: 15-20% of total data -Standard-IA: 20-25% -Archive tiers: 55-65% - -Monthly cost trend: Decreasing 5-10% per month as data ages into cheaper tiers -``` - -## Best Practices - -1. ✅ **Start with Intelligent-Tiering** - No retrieval fees, automatic optimization -2. ✅ **Use batch operations** for deletions - 1000x cheaper than individual deletes -3. ✅ **Monitor storage class distribution** monthly via Cost Explorer -4. ✅ **Set lifecycle policies** for predictable archival (metadata, logs) -5. ✅ **Enable S3 Storage Lens** for detailed storage analytics -6. ✅ **Use S3 Select** for querying archived data without full retrieval -7. ✅ **Consider S3 Batch Operations** for large-scale tier changes - -## Troubleshooting - -### Issue: Data not transitioning to cheaper tiers - -**Solution:** -```typescript -// Check if lifecycle policy is active -const policy = await storage.getLifecyclePolicy() -console.log('Policy status:', policy.rules.map(r => r.status)) - -// Ensure objects are old enough -// S3 requires objects to be at least 30 days old for IA transition -``` - -### Issue: High retrieval costs from Glacier - -**Solution:** -```typescript -// Switch to Intelligent-Tiering (no retrieval fees) -await storage.disableIntelligentTiering('old-config') -await storage.enableIntelligentTiering('entities/', 'new-config') - -// Or use Glacier Instant Retrieval instead of Glacier Flexible -``` - -### Issue: Unexpected monitoring fees - -**Solution:** -- Intelligent-Tiering has $0.0025 per 1000 objects monitoring fee -- For 1 billion objects: $2,500/month monitoring -- If cost is high, use lifecycle policies instead (no monitoring fee) - -## Summary - -**Recommended Strategy for Most Use Cases:** -- **Intelligent-Tiering** for vectors and frequently queried data -- **Lifecycle policies** for metadata and system files -- **Batch operations** for efficient cleanup - -**Expected Savings:** -- **Year 1**: 40-50% reduction in storage costs -- **Year 2+**: 60-70% reduction as more data ages into archive tiers -- **Long-term**: 75-85% reduction for mature datasets - -**500TB Example (Intelligent-Tiering):** -- Before: $143,000/year -- After: $51,000/year -- **Savings: $92,000/year (64%)** - -**1PB Example (Intelligent-Tiering):** -- Before: $286,000/year -- After: $102,000/year -- **Savings: $184,000/year (64%)** - ---- - -**Last Updated**: 2025-10-17 -**Cloud Provider**: AWS S3 diff --git a/docs/operations/cost-optimization-azure.md b/docs/operations/cost-optimization-azure.md deleted file mode 100644 index bae4370d..00000000 --- a/docs/operations/cost-optimization-azure.md +++ /dev/null @@ -1,552 +0,0 @@ -# Azure Blob Storage Cost Optimization Guide for Brainy -> **Cost Impact**: Reduce storage costs from $107k/year to $5k/year at 500TB scale (**95% savings**) - -## Overview - -Brainy provides enterprise-grade cost optimization features for Azure Blob Storage, including manual tier management, lifecycle policies, and batch operations. - -## Cost Breakdown (Before Optimization) - -### Hot Tier Azure Storage Costs (500TB Dataset) - -``` -Storage: 500TB × $0.0184/GB/month × 12 months = $107,520/year -Operations: ~$5,000/year (write/read operations) -Total: $112,520/year -``` - -## Azure Blob Storage Tiers - -| Tier | Cost/GB/Month | Retrieval | Early Deletion | Use Case | -|------|---------------|-----------|----------------|----------| -| **Hot** | $0.0184 | None | None | Frequent access | -| **Cool** | $0.0115 | $0.01/GB | 30 days | Infrequent access | -| **Archive** | $0.00099 | $0.02/GB + rehydration time | 180 days | Long-term archive | - -**Key Difference from AWS/GCS:** -- Azure has only 3 tiers (vs AWS 6 tiers, GCS 4 classes) -- Archive tier requires rehydration (hours to 15 hours) before access -- No "Intelligent-Tiering" equivalent - must use lifecycle policies or manual management - -## Strategy 1: Manual Tier Management (Immediate Savings) - -### Setup: Change Blob Tiers Manually - -**Best for**: Quick wins, specific files, immediate cost reduction - -```typescript -import { Brainy } from '@soulcraft/brainy' -import { AzureBlobStorage } from '@soulcraft/brainy/storage' - -// Initialize Brainy with Azure storage -const storage = new AzureBlobStorage({ - connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING, - containerName: 'brainy-data' -}) - -const brain = new Brainy({ storage }) -await brain.init() - -// Change tier for a single blob -await storage.changeBlobTier( - 'entities/nouns/vectors/00/00123456-uuid.json', - 'Cool' -) - -// Batch tier changes (efficient for thousands of blobs) -const blobsToMove = [ - 'entities/nouns/vectors/01/...', - 'entities/nouns/vectors/02/...', - // ... up to thousands of blobs -] - -await storage.batchChangeTier(blobsToMove, 'Cool') // Hot → Cool -await storage.batchChangeTier(oldBlobs, 'Archive') // Cool → Archive -``` - -### Immediate Cost Impact - -Moving 400TB from Hot to Cool: -``` -Before (Hot): 400TB × $0.0184/GB × 12 = $86,016/year -After (Cool): 400TB × $0.0115/GB × 12 = $53,760/year -Savings: $32,256/year (37% savings on moved data) -``` - -Moving 100TB from Hot to Archive: -``` -Before (Hot): 100TB × $0.0184/GB × 12 = $21,504/year -After (Archive): 100TB × $0.00099/GB × 12 = $1,158/year -Savings: $20,346/year (95% savings on moved data) -``` - -## Strategy 2: Lifecycle Policies (Automated) - -### Setup: Automatic Tier Transitions - -**Best for**: Predictable patterns, automatic management - -```typescript -// Set lifecycle policy for automatic tier management -await storage.setLifecyclePolicy({ - rules: [{ - name: 'optimizeVectors', - enabled: true, - type: 'Lifecycle', - definition: { - filters: { - blobTypes: ['blockBlob'], - prefixMatch: ['entities/nouns/vectors/'] - }, - actions: { - baseBlob: { - tierToCool: { daysAfterModificationGreaterThan: 30 }, - tierToArchive: { daysAfterModificationGreaterThan: 90 } - } - } - } - }, { - name: 'optimizeMetadata', - enabled: true, - type: 'Lifecycle', - definition: { - filters: { - blobTypes: ['blockBlob'], - prefixMatch: ['entities/nouns/metadata/'] - }, - actions: { - baseBlob: { - tierToCool: { daysAfterModificationGreaterThan: 30 }, - tierToArchive: { daysAfterModificationGreaterThan: 180 } - } - } - } - }, { - name: 'cleanupOldSystemFiles', - enabled: true, - type: 'Lifecycle', - definition: { - filters: { - blobTypes: ['blockBlob'], - prefixMatch: ['_system/'] - }, - actions: { - baseBlob: { - delete: { daysAfterModificationGreaterThan: 365 } - } - } - } - }] -}) - -// Verify lifecycle policy -const policy = await storage.getLifecyclePolicy() -console.log('Lifecycle policy active:', policy.rules.length, 'rules') -``` - -### Cost Calculation (500TB with Lifecycle Policy) - -**Assumptions:** -- 30% of data in Hot tier (< 30 days old) -- 40% of data in Cool tier (30-90 days old) -- 30% of data in Archive tier (90+ days old) - -``` -Hot (150TB): 150TB × $0.0184/GB × 12 = $32,256/year -Cool (200TB): 200TB × $0.0115/GB × 12 = $26,880/year -Archive (150TB): 150TB × $0.00099/GB × 12 = $1,732/year - -Total Storage Cost: $60,868/year -Total with Operations: ~$65,500/year -Savings: $47,000/year (42%) -``` - -## Strategy 3: Aggressive Archival (Maximum Savings) - -### Setup: Fast Archival for Cold Data - -**Best for**: Archival workloads, compliance, historical data - -```typescript -await storage.setLifecyclePolicy({ - rules: [{ - name: 'aggressiveArchival', - enabled: true, - type: 'Lifecycle', - definition: { - filters: { - blobTypes: ['blockBlob'], - prefixMatch: ['entities/'] - }, - actions: { - baseBlob: { - tierToCool: { daysAfterModificationGreaterThan: 14 }, - tierToArchive: { daysAfterModificationGreaterThan: 30 } - } - } - } - }] -}) -``` - -### Cost Calculation (500TB Aggressive Archival) - -**After 6 months:** -``` -Hot (50TB): 50TB × $0.0184/GB × 12 = $10,752/year -Cool (100TB): 100TB × $0.0115/GB × 12 = $13,440/year -Archive (350TB): 350TB × $0.00099/GB × 12 = $4,039/year - -Total Storage Cost: $28,231/year -Total with Operations: ~$33,000/year -Savings: $79,500/year (71%) - -Warning: Archive rehydration takes 1-15 hours -``` - -## Strategy 4: Hybrid Approach (Balanced) - -### Setup: Different Policies for Different Data Types - -```typescript -await storage.setLifecyclePolicy({ - rules: [{ - // Vectors: Keep in Hot/Cool for search performance - name: 'vectors-moderate', - enabled: true, - type: 'Lifecycle', - definition: { - filters: { - blobTypes: ['blockBlob'], - prefixMatch: ['entities/nouns/vectors/', 'entities/verbs/vectors/'] - }, - actions: { - baseBlob: { - tierToCool: { daysAfterModificationGreaterThan: 60 } - // Don't archive vectors - keep searchable - } - } - } - }, { - // Metadata: Aggressive archival - name: 'metadata-aggressive', - enabled: true, - type: 'Lifecycle', - definition: { - filters: { - blobTypes: ['blockBlob'], - prefixMatch: ['entities/nouns/metadata/', 'entities/verbs/metadata/'] - }, - actions: { - baseBlob: { - tierToCool: { daysAfterModificationGreaterThan: 30 }, - tierToArchive: { daysAfterModificationGreaterThan: 90 } - } - } - } - }] -}) -``` - -### Cost Calculation (500TB Hybrid Approach) - -**Vectors (300TB):** -``` -Hot (90TB): 90TB × $0.0184/GB × 12 = $19,354/year -Cool (210TB): 210TB × $0.0115/GB × 12 = $28,980/year -Subtotal: $48,334/year -``` - -**Metadata (200TB):** -``` -Hot (30TB): 30TB × $0.0184/GB × 12 = $6,451/year -Cool (70TB): 70TB × $0.0115/GB × 12 = $9,660/year -Archive (100TB): 100TB × $0.00099/GB × 12 = $1,158/year -Subtotal: $17,269/year -``` - -**Total Cost: $65,603/year + operations (~$70,500/year total)** -**Savings vs Hot: $42,000/year (37%)** - -## Comparison Table: All Strategies - -| Strategy | Annual Cost | Savings | Archive % | Best For | -|----------|-------------|---------|-----------|----------| -| **No Optimization** | $112,500 | 0% | 0% | N/A | -| **Manual Tier Mgmt** | $75,000 | 33% | 20% | Immediate savings | -| **Lifecycle Policy** | $65,500 | 42% | 30% | Automated management | -| **Hybrid Approach** | $70,500 | 37% | 20% | Balance performance/cost | -| **Aggressive Archival** | $33,000 | **71%** | 70% | Cold data, compliance | - -## Archive Rehydration (Important!) - -### Rehydrate from Archive Tier - -**Required before accessing archived blobs:** - -```typescript -// Rehydrate blob from Archive to Hot (high priority) -await storage.rehydrateBlob( - 'entities/nouns/vectors/00/00123456-uuid.json', - 'High' // 'Standard' or 'High' priority -) - -// Rehydration time: -// - High priority: 1 hour -// - Standard priority: up to 15 hours - -// Check rehydration status -const metadata = await storage.getBlobMetadata(blobPath) -console.log('Archive status:', metadata.archiveStatus) -// Output: 'rehydrate-pending-to-hot', 'rehydrate-pending-to-cool', or undefined (done) -``` - -### Batch Rehydration - -```typescript -// Rehydrate multiple blobs (useful for planned access) -const blobsToRehydrate = [/* array of blob paths */] - -for (const blobPath of blobsToRehydrate) { - await storage.rehydrateBlob(blobPath, 'High') -} - -// Wait for rehydration to complete (1-15 hours) -// Then access blobs normally -``` - -### Cost Impact of Rehydration - -``` -Retrieval fee: $0.02/GB -High priority: Additional $0.10/GB - -Examples: -- 1GB blob (standard): $0.02 + wait 15 hours -- 1GB blob (high priority): $0.12 + wait 1 hour -- 1TB batch (high priority): $122.88 + wait 1 hour -``` - -## Batch Operations - -### Efficient Bulk Deletions - -```typescript -// Batch delete (256 blobs per request) -const idsToDelete = [/* array of entity IDs */] - -const paths = idsToDelete.flatMap(id => { - const shard = id.substring(0, 2) - return [ - `entities/nouns/vectors/${shard}/${id}.json`, - `entities/nouns/metadata/${shard}/${id}.json` - ] -}) - -// Batch delete via BlobBatchClient -await storage.batchDelete(paths) - -// Cost impact: -// - Individual deletes: 1M operations × $0.0005 per 10k = $50 -// - Batch deletes: 4k batches × $0.0005 = $2 (25x cheaper!) -``` - -### Batch Tier Changes - -```typescript -// Change tier for thousands of blobs efficiently -const vectorPaths = [/* 10,000 blob paths */] - -// Batch operation (256 blobs per batch) -await storage.batchChangeTier(vectorPaths, 'Cool') - -// Much faster than individual changeBlobTier() calls -// Azure automatically batches internally for efficiency -``` - -## Monitoring and Management - -### Get Current Lifecycle Policy - -```typescript -const policy = await storage.getLifecyclePolicy() -console.log('Active rules:', policy.rules) - -// Example output: -// { -// rules: [ -// { -// name: 'optimizeVectors', -// enabled: true, -// type: 'Lifecycle', -// definition: {...} -// } -// ] -// } -``` - -### Remove Lifecycle Policy - -```typescript -await storage.removeLifecyclePolicy() -``` - -### Get Storage Status - -```typescript -const status = await storage.getStorageStatus() -console.log('Storage type:', status.type) -console.log('Container:', status.details.container) -console.log('Account:', status.details.account) -``` - -## Azure Portal Monitoring - -### Track Your Savings - -1. **Storage Account** → **Insights** → View tier distribution -2. **Cost Management** → **Cost Analysis** → Filter by storage account -3. **Monitoring** → **Metrics** → Track blob count by tier -4. **Lifecycle Management** → View policy execution logs - -### Expected Metrics After 6 Months - -``` -Hot tier: 20-30% of total data -Cool tier: 40-50% -Archive tier: 20-40% - -Monthly cost trend: Decreasing 5-8% per month as data transitions -``` - -## Best Practices - -1. ✅ **Use lifecycle policies** for automatic management -2. ✅ **Archive cold data** aggressively (95% cost savings!) -3. ✅ **Use batch operations** for tier changes and deletions -4. ✅ **Plan rehydration** ahead of time (1-15 hour delay) -5. ✅ **Monitor early deletion charges** (Cool: 30 days, Archive: 180 days) -6. ✅ **Use Cool tier for infrequent access** (no rehydration needed) -7. ✅ **Consider ZRS or GRS** for critical data redundancy - -## Storage Redundancy Options - -| Option | Cost Multiplier | Copies | Availability | Use Case | -|--------|-----------------|--------|--------------|----------| -| **LRS** | 1x | 3 (same datacenter) | 11 nines | Cost-optimized | -| **ZRS** | 1.25x | 3 (different zones) | 12 nines | High availability | -| **GRS** | 2x | 6 (secondary region) | 16 nines | Disaster recovery | -| **RA-GRS** | 2.5x | 6 (read access) | 16 nines | Global read access | - -**Recommendation for Brainy:** -- **Production**: GRS (geo-redundancy for disaster recovery) -- **Cost-optimized**: LRS (lowest cost, still very reliable) - -## Troubleshooting - -### Issue: Data not transitioning tiers - -**Solution:** -```typescript -// Check lifecycle policy status -const policy = await storage.getLifecyclePolicy() -console.log('Policy rules:', policy.rules.map(r => ({ - name: r.name, - enabled: r.enabled -}))) - -// Azure lifecycle policies run once per day -// Transitions may take 24-48 hours to execute -``` - -### Issue: Access denied on archived blob - -**Solution:** -```typescript -// Archived blobs cannot be accessed directly -// Must rehydrate first: -await storage.rehydrateBlob(blobPath, 'High') - -// Wait for rehydration (check status) -let status -do { - await new Promise(resolve => setTimeout(resolve, 60000)) // Wait 1 minute - const metadata = await storage.getBlobMetadata(blobPath) - status = metadata.archiveStatus -} while (status && status.includes('pending')) - -// Now access the blob -const data = await storage.get(blobPath) -``` - -### Issue: High early deletion charges - -**Solution:** -- Cool tier: 30-day minimum storage -- Archive tier: 180-day minimum storage -- Early deletion incurs pro-rated charges -- Use lifecycle policies instead of manual changes to avoid early deletion fees - -## Authentication - -### Connection String (Simple) - -```typescript -const storage = new AzureBlobStorage({ - connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING, - containerName: 'brainy-data' -}) -``` - -### Account Key (Explicit) - -```typescript -const storage = new AzureBlobStorage({ - accountName: process.env.AZURE_STORAGE_ACCOUNT, - accountKey: process.env.AZURE_STORAGE_KEY, - containerName: 'brainy-data' -}) -``` - -### SAS Token (Granular Access) - -```typescript -const storage = new AzureBlobStorage({ - sasToken: process.env.AZURE_STORAGE_SAS_TOKEN, - accountName: process.env.AZURE_STORAGE_ACCOUNT, - containerName: 'brainy-data' -}) -``` - -## Summary - -**Recommended Strategy for Most Use Cases:** -- **Lifecycle policies** for automatic tier management -- **Batch operations** for efficient tier changes and deletions -- **Cool tier** for infrequently accessed data (no rehydration delay) -- **Archive tier** for long-term storage (1-15 hour rehydration) - -**Expected Savings:** -- **Year 1**: 40-50% reduction in storage costs -- **Year 2+**: 60-70% reduction as more data ages into Archive -- **Long-term**: 75-85% reduction for mature datasets - -**500TB Example (Lifecycle Policy):** -- Before: $112,500/year -- After: $65,500/year -- **Savings: $47,000/year (42%)** - -**500TB Example (Aggressive Archival):** -- Before: $112,500/year -- After: $33,000/year -- **Savings: $79,500/year (71%)** - -**1PB Example (Lifecycle Policy):** -- Before: $225,000/year -- After: $131,000/year -- **Savings: $94,000/year (42%)** - ---- - -**Last Updated**: 2025-10-17 -**Cloud Provider**: Azure Blob Storage diff --git a/docs/operations/cost-optimization-cloudflare-r2.md b/docs/operations/cost-optimization-cloudflare-r2.md deleted file mode 100644 index 3bdc5490..00000000 --- a/docs/operations/cost-optimization-cloudflare-r2.md +++ /dev/null @@ -1,450 +0,0 @@ -# Cloudflare R2 Cost Optimization Guide for Brainy -> **Cost Impact**: $0 egress fees + 96% storage savings = Lowest cloud storage costs - -## Overview - -Cloudflare R2 is an S3-compatible object storage with **zero egress fees**, making it ideal for high-traffic applications. Brainy fully supports R2 with lifecycle policies and batch operations. - -## Cost Breakdown - -### R2 Pricing (As of 2025) - -``` -Storage: $0.015/GB/month -Class A Operations (write): $4.50 per million -Class B Operations (read): $0.36 per million -Egress: $0.00 (FREE!) -``` - -### Comparison to AWS S3 (500TB Dataset) - -**AWS S3 Standard:** -``` -Storage: 500TB × $0.023/GB × 12 = $138,000/year -Egress (assume 100TB/month): 1.2PB × $0.09/GB = $108,000/year -Operations: $5,000/year -Total: $251,000/year -``` - -**Cloudflare R2 Standard:** -``` -Storage: 500TB × $0.015/GB × 12 = $90,000/year -Egress: $0 (FREE!) -Operations: $5,000/year -Total: $95,000/year -Savings vs AWS: $156,000/year (62%) -``` - -**R2's Zero Egress Advantage:** -- **High-traffic apps**: Save $100k-$1M/year in egress fees -- **Video/media delivery**: No CDN egress costs -- **API responses**: Unlimited reads at no extra cost - -## R2 Storage Classes (Coming Soon) - -**Current State (2025):** -- R2 currently has only **one storage class** (Standard) -- No lifecycle policies or tier transitions yet -- Cloudflare plans to add infrequent access tiers - -**When lifecycle features arrive:** -- Brainy is already prepared with `setLifecyclePolicy()` support -- Will work seamlessly once Cloudflare enables lifecycle management - -## Strategy 1: Use R2 Standard (Current Best Practice) - -### Setup: S3-Compatible API - -```typescript -import { Brainy } from '@soulcraft/brainy' -import { S3CompatibleStorage } from '@soulcraft/brainy/storage' - -// R2 uses S3-compatible API -const storage = new S3CompatibleStorage({ - endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`, - bucket: 'my-brainy-data', - region: 'auto', // R2 uses 'auto' region - accessKeyId: process.env.R2_ACCESS_KEY_ID, - secretAccessKey: process.env.R2_SECRET_ACCESS_KEY -}) - -const brain = new Brainy({ storage }) -await brain.init() -``` - -### Cost Calculation (500TB on R2) - -``` -Storage: 500TB × $0.015/GB × 12 = $90,000/year -Class A ops (10M writes): $45/year -Class B ops (100M reads): $36/year -Egress: $0 - -Total: $90,081/year -``` - -**Compared to AWS S3 (with egress):** -- AWS: $251,000/year -- R2: $90,081/year -- **Savings: $160,919/year (64%)** - -## Strategy 2: R2 + Workers (Edge Computing) - -### Setup: Compute at the Edge - -```typescript -// Cloudflare Worker (runs at edge) -export default { - async fetch(request, env) { - // Initialize Brainy with R2 - const storage = new S3CompatibleStorage({ - endpoint: env.R2_ENDPOINT, - bucket: env.R2_BUCKET, - accessKeyId: env.R2_ACCESS_KEY, - secretAccessKey: env.R2_SECRET_KEY, - region: 'auto' - }) - - const brain = new Brainy({ storage }) - await brain.init() - - // Process at edge (no origin server needed) - const results = await brain.search(request.query) - return new Response(JSON.stringify(results)) - } -} -``` - -### Cost Calculation (Workers + R2) - -``` -R2 Storage (500TB): $90,000/year -Workers (10M requests/day): - - First 100k requests/day: FREE - - Additional 350M requests/month: $1,750/year - - CPU time (50ms avg): $5,000/year - -Total: $96,750/year - -vs Traditional Setup (AWS S3 + EC2 + CloudFront): - - S3: $138,000/year - - EC2 (t3.xlarge × 4): $24,000/year - - CloudFront egress: $50,000/year - - Load balancer: $3,000/year - Total: $215,000/year - -Savings: $118,250/year (55%) -``` - -## Strategy 3: Hybrid Multi-Cloud (R2 + S3) - -### Setup: R2 for Hot Data, S3 for Archives - -```typescript -// Use R2 for frequently accessed data (zero egress) -const hotStorage = new S3CompatibleStorage({ - endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`, - bucket: 'brainy-hot', - region: 'auto', - accessKeyId: process.env.R2_ACCESS_KEY_ID, - secretAccessKey: process.env.R2_SECRET_ACCESS_KEY -}) - -// Use AWS S3 with Intelligent-Tiering for cold data -const coldStorage = new S3CompatibleStorage({ - endpoint: 's3.amazonaws.com', - bucket: 'brainy-archive', - region: 'us-east-1', - accessKeyId: process.env.AWS_ACCESS_KEY_ID, - secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY -}) - -// Initialize separate Brainy instances or implement tiering logic -``` - -### Cost Calculation (300TB R2 + 200TB S3 Archive) - -``` -R2 Hot Data (300TB): - Storage: 300TB × $0.015/GB × 12 = $54,000/year - Egress: $0 - -S3 Cold Data (200TB with lifecycle → Deep Archive): - Storage: 200TB × $0.00099/GB × 12 = $2,376/year - Ops: $1,000/year - -Total: $57,376/year - -vs All AWS S3: - - S3 storage + egress: $251,000/year - -Savings: $193,624/year (77%) -``` - -## Batch Operations - -### Efficient Bulk Deletions - -```typescript -// R2 supports S3 batch delete API -const idsToDelete = [/* array of entity IDs */] - -const paths = idsToDelete.flatMap(id => { - const shard = id.substring(0, 2) - return [ - `entities/nouns/vectors/${shard}/${id}.json`, - `entities/nouns/metadata/${shard}/${id}.json` - ] -}) - -// Batch delete (1000 objects per request) -await storage.batchDelete(paths) - -// Cost impact: -// - Individual deletes: 1M × $4.50/1M = $4.50 -// - Batch deletes: 1k batches × $4.50/1M = $0.0045 (1000x cheaper!) -``` - -## R2 Advanced Features - -### 1. R2 Custom Domains - -**Free custom domains for R2 buckets:** - -```bash -# Configure custom domain in Cloudflare dashboard -# Then access via your domain -https://storage.yourdomain.com/entities/nouns/vectors/... - -# Benefits: -# - No additional cost -# - Automatic SSL/TLS -# - Global CDN included -# - DDoS protection -``` - -### 2. R2 Event Notifications - -**Trigger Workers on object events:** - -```typescript -// Worker triggered on R2 object upload -export default { - async fetch(request, env) { - // Process new objects automatically - // E.g., index new entities, generate thumbnails, etc. - } -} - -// Cost: Only pay for Worker execution (no polling needed) -``` - -### 3. R2 Presigned URLs - -```typescript -// Generate presigned URL for direct browser uploads -const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry - -// Client uploads directly to R2 (no server bandwidth used) -``` - -## Monitoring and Management - -### Get Storage Status - -```typescript -const status = await storage.getStorageStatus() -console.log('Storage type:', status.type) // 's3-compatible' -console.log('Bucket:', status.details.bucket) -console.log('Endpoint:', status.details.endpoint) -``` - -### Cloudflare Dashboard Monitoring - -1. **R2 Dashboard** → View bucket metrics -2. **Analytics** → Track requests, storage, and bandwidth -3. **Workers Analytics** → Monitor edge compute usage -4. **Logs** → Real-time logs with Logpush - -### Expected Metrics - -``` -Storage: Growing with your data -Class A ops: Writes (higher cost) -Class B ops: Reads (minimal cost) -Egress: Always $0 (R2's advantage) -``` - -## Comparison Table: R2 vs Other Providers - -| Feature | R2 | AWS S3 | GCS | Azure | -|---------|-----|--------|-----|-------| -| **Storage** | $0.015/GB | $0.023/GB | $0.020/GB | $0.0184/GB | -| **Egress** | **$0** | $0.09/GB | $0.12/GB | $0.087/GB | -| **Lifecycle Tiers** | Coming soon | 6 tiers | 4 classes | 3 tiers | -| **S3 API Compatible** | ✅ Yes | ✅ Native | ⚠️ Via interop | ⚠️ Via SDK | -| **CDN Included** | ✅ Yes | ❌ Extra cost | ❌ Extra cost | ❌ Extra cost | -| **Edge Compute** | ✅ Workers | ❌ Lambda@Edge | ❌ Cloud Functions | ❌ Functions | - -## R2 Free Tier - -**Generous free tier:** -``` -Storage: 10 GB free per month -Class A ops: 1 million free per month -Class B ops: 10 million free per month -Egress: Unlimited (always free) -``` - -**Perfect for:** -- Development and testing -- Small applications (<10GB) -- Prototypes - -## Best Practices - -1. ✅ **Use R2 for high-egress workloads** - Zero egress fees -2. ✅ **Combine with Workers** - Edge compute included -3. ✅ **Use custom domains** - Free branded URLs -4. ✅ **Batch operations** for deletions - 1000x cheaper -5. ✅ **Use presigned URLs** - Direct client uploads -6. ✅ **Monitor with Analytics** - Built-in dashboarding -7. ✅ **Consider hybrid approach** - R2 hot + S3 archive cold - -## Migration from S3 to R2 - -### Using rclone - -```bash -# Install rclone -brew install rclone # or apt-get install rclone - -# Configure S3 source -rclone config create s3-source s3 \ - access_key_id=$AWS_ACCESS_KEY \ - secret_access_key=$AWS_SECRET_KEY \ - region=us-east-1 - -# Configure R2 destination -rclone config create r2-dest s3 \ - access_key_id=$R2_ACCESS_KEY \ - secret_access_key=$R2_SECRET_KEY \ - endpoint=https://$R2_ACCOUNT_ID.r2.cloudflarestorage.com \ - region=auto - -# Copy data -rclone copy s3-source:my-bucket r2-dest:my-bucket --progress - -# Verify -rclone check s3-source:my-bucket r2-dest:my-bucket -``` - -### Cost of Migration - -``` -Data transfer out from S3: 500TB × $0.09/GB = $45,000 -Data transfer into R2: $0 (ingress is free) - -One-time migration cost: $45,000 - -Monthly savings after migration: -S3 storage + egress: $20,833/month -R2 storage: $7,500/month -Savings: $13,333/month - -ROI: 3.4 months -``` - -## Future: R2 Lifecycle Policies (When Available) - -### Prepared for Future Features - -```typescript -// Brainy is ready for R2 lifecycle features -await storage.setLifecyclePolicy({ - rules: [{ - id: 'archive-old-data', - prefix: 'entities/', - status: 'Enabled', - transitions: [ - { days: 30, storageClass: 'INFREQUENT_ACCESS' }, // When available - { days: 90, storageClass: 'ARCHIVE' } - ] - }] -}) - -// Expected cost impact (when lifecycle is available): -// Standard: $0.015/GB -// Infrequent: ~$0.008/GB (estimated) -// Archive: ~$0.002/GB (estimated) -``` - -## Troubleshooting - -### Issue: Connection errors - -**Solution:** -```typescript -// Ensure correct endpoint format -const storage = new S3CompatibleStorage({ - endpoint: `https://${accountId}.r2.cloudflarestorage.com`, - // NOT: `https://r2.cloudflarestorage.com/${accountId}` - - region: 'auto', // R2 requires 'auto' - forcePathStyle: false // R2 uses virtual-hosted-style -}) -``` - -### Issue: High Class A operation costs - -**Solution:** -- Use batch operations (writes are most expensive) -- Cache frequently written data -- Consolidate small writes into larger batches -- Consider Workers KV for high-frequency writes - -### Issue: Need lifecycle management now - -**Solution:** -- Manually move old data to S3 Deep Archive -- Use hybrid approach: R2 for hot, S3 for cold -- Wait for R2 lifecycle features (planned) - -## Summary - -**R2 Advantages:** -- ✅ **Zero egress fees** - Unlimited reads at no cost -- ✅ **Lower storage costs** - $0.015/GB vs $0.023/GB (AWS) -- ✅ **S3-compatible API** - Drop-in replacement for S3 -- ✅ **Global CDN included** - No additional CDN costs -- ✅ **Edge Workers** - Compute at the edge -- ✅ **Free custom domains** - Branded URLs -- ✅ **No minimums** - No minimum storage duration - -**R2 Limitations (Current):** -- ⚠️ Single storage class (for now) -- ⚠️ No lifecycle policies yet (coming soon) -- ⚠️ Less mature than S3/GCS/Azure - -**Recommended Use Cases:** -- 🎯 High-traffic APIs (zero egress fees!) -- 🎯 Video/media delivery (massive savings) -- 🎯 User-generated content -- 🎯 Web application assets -- 🎯 Hot data storage - -**500TB Example (R2 vs AWS S3 with 100TB/month egress):** -- AWS S3: $251,000/year -- Cloudflare R2: $90,000/year -- **Savings: $161,000/year (64%)** - -**1PB Example (R2 vs AWS S3 with 200TB/month egress):** -- AWS S3: $686,000/year -- Cloudflare R2: $180,000/year -- **Savings: $506,000/year (74%)** - ---- - -**Last Updated**: 2025-10-17 -**Cloud Provider**: Cloudflare R2 -**Key Advantage**: **$0 egress fees forever** diff --git a/docs/operations/cost-optimization-gcs.md b/docs/operations/cost-optimization-gcs.md deleted file mode 100644 index a5ecbb34..00000000 --- a/docs/operations/cost-optimization-gcs.md +++ /dev/null @@ -1,465 +0,0 @@ -# Google Cloud Storage Cost Optimization Guide for Brainy -> **Cost Impact**: Reduce storage costs from $138k/year to $8.3k/year at 500TB scale (**94% savings**) - -> **Disclaimer**: Cost savings percentages are PROJECTED based on published cloud provider pricing at 500TB scale. Actual savings depend on access patterns, data lifecycle, and workload characteristics. These are not measured benchmarks. - -## Overview - -Brainy provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management. - -## Cost Breakdown (Before Optimization) - -### Standard GCS Storage Costs (500TB Dataset) - -``` -Storage: 500TB × $0.023/GB/month × 12 months = $138,000/year -Operations: ~$5,000/year (Class A/B operations) -Total: $143,000/year -``` - -## GCS Storage Classes - -| Class | Cost/GB/Month | Retrieval Fee | Minimum Storage | Use Case | -|-------|---------------|---------------|-----------------|----------| -| **Standard** | $0.020 | None | None | Frequent access | -| **Nearline** | $0.010 | $0.01/GB | 30 days | Once per month | -| **Coldline** | $0.004 | $0.02/GB | 90 days | Once per quarter | -| **Archive** | $0.0012 | $0.05/GB | 365 days | Long-term archive | - -## Strategy 1: Lifecycle Policies (Manual Control) - -### Setup: Automatic Tier Transitions - -```typescript -import { Brainy } from '@soulcraft/brainy' -import { GcsStorage } from '@soulcraft/brainy/storage' - -// Initialize Brainy with GCS storage -const storage = new GcsStorage({ - bucketName: 'my-brainy-data', - keyFilename: './service-account.json' // Or use ADC -}) - -const brain = new Brainy({ storage }) -await brain.init() - -// Set lifecycle policy for automatic archival -await storage.setLifecyclePolicy({ - rules: [{ - condition: { age: 30 }, - action: { type: 'SetStorageClass', storageClass: 'NEARLINE' } - }, { - condition: { age: 90 }, - action: { type: 'SetStorageClass', storageClass: 'COLDLINE' } - }, { - condition: { age: 365 }, - action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' } - }] -}) - -// Verify lifecycle policy -const policy = await storage.getLifecyclePolicy() -console.log('Lifecycle policy active:', policy.rules.length, 'rules') -``` - -### Cost Calculation (500TB with Lifecycle Policy) - -**Assumptions:** -- 40% of data accessed in last 30 days (Standard) -- 30% of data 30-90 days old (Nearline) -- 20% of data 90-365 days old (Coldline) -- 10% of data 365+ days old (Archive) - -``` -Standard (200TB): 200TB × $0.020/GB × 12 = $48,000/year -Nearline (150TB): 150TB × $0.010/GB × 12 = $18,000/year -Coldline (100TB): 100TB × $0.004/GB × 12 = $4,800/year -Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year - -Total Storage Cost: $71,520/year -Total with Operations: ~$76,500/year -Savings: $66,500/year (46%) -``` - -## Strategy 2: Autoclass (Recommended) - -### Setup: Automatic Class Optimization - -**Best for**: Maximum automation, unpredictable access patterns - -```typescript -// Enable Autoclass for automatic tier management -await storage.enableAutoclass({ - terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier -}) - -// Benefits: -// - Automatically moves objects between classes based on access patterns -// - No data retrieval delays (unlike AWS Glacier) -// - Transparent tier transitions within 24 hours -// - Supports all storage classes including Archive -// - No extra monitoring fees -``` - -### How Autoclass Works - -1. **Initial Placement**: New objects start in Standard class -2. **Automatic Demotion**: Objects move to Nearline (30 days) → Coldline (90 days) → Archive (365 days) -3. **Automatic Promotion**: Accessed objects move back to Standard class -4. **Access-Pattern Learning**: Uses 90-day access history for optimization - -### Cost Calculation (500TB with Autoclass) - -**Realistic distribution after 1 year:** -- 10% Standard (hot data, frequently accessed) -- 15% Nearline (warm data) -- 35% Coldline (cool data) -- 40% Archive (cold data) - -``` -Standard (50TB): 50TB × $0.020/GB × 12 = $12,000/year -Nearline (75TB): 75TB × $0.010/GB × 12 = $9,000/year -Coldline (175TB): 175TB × $0.004/GB × 12 = $8,400/year -Archive (200TB): 200TB × $0.0012/GB × 12 = $2,880/year - -Total Storage Cost: $32,280/year -Total with Operations: ~$37,000/year -Savings vs Standard: $106,000/year (74%) -``` - -## Strategy 3: Hybrid Approach (Maximum Savings) - -### Setup: Autoclass + Lifecycle Policies - -```typescript -// Enable Autoclass for vectors (frequently searched) -await storage.enableAutoclass({ - terminalStorageClass: 'COLDLINE' // Don't archive vectors deeply -}) - -// Set lifecycle policy for metadata (less frequently accessed) -await storage.setLifecyclePolicy({ - rules: [{ - condition: { age: 30, matchesPrefix: ['entities/nouns/metadata/'] }, - action: { type: 'SetStorageClass', storageClass: 'NEARLINE' } - }, { - condition: { age: 60, matchesPrefix: ['entities/nouns/metadata/'] }, - action: { type: 'SetStorageClass', storageClass: 'COLDLINE' } - }, { - condition: { age: 180, matchesPrefix: ['entities/nouns/metadata/'] }, - action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' } - }, { - condition: { age: 730, matchesPrefix: ['_system/'] }, - action: { type: 'Delete' } // Delete old system files after 2 years - }] -}) -``` - -### Cost Calculation (500TB Hybrid Approach) - -**Vectors (300TB with Autoclass):** -``` -Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year -Nearline (45TB): 45TB × $0.010/GB × 12 = $5,400/year -Coldline (225TB): 225TB × $0.004/GB × 12 = $10,800/year -Subtotal: $23,400/year -``` - -**Metadata (200TB with Lifecycle Policy):** -``` -Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year -Nearline (40TB): 40TB × $0.010/GB × 12 = $4,800/year -Coldline (80TB): 80TB × $0.004/GB × 12 = $3,840/year -Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year -Subtotal: $16,560/year -``` - -**Total Cost: $39,960/year + operations (~$45,000/year total)** -**Savings vs Standard: $98,000/year (69%)** - -## Strategy 4: Aggressive Archival (Maximum Savings) - -### Setup: Fast Archival for Cold Data - -```typescript -await storage.setLifecyclePolicy({ - rules: [{ - condition: { age: 14 }, - action: { type: 'SetStorageClass', storageClass: 'NEARLINE' } - }, { - condition: { age: 30 }, - action: { type: 'SetStorageClass', storageClass: 'COLDLINE' } - }, { - condition: { age: 90 }, - action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' } - }] -}) - -// Note: Archive class has 365-day minimum storage duration -// Early deletion incurs pro-rated charges for remaining days -``` - -### Cost Calculation (500TB Aggressive Archival) - -**After 1 year:** -``` -Standard (25TB): 25TB × $0.020/GB × 12 = $6,000/year -Nearline (50TB): 50TB × $0.010/GB × 12 = $6,000/year -Coldline (75TB): 75TB × $0.004/GB × 12 = $3,600/year -Archive (350TB): 350TB × $0.0012/GB × 12 = $5,040/year - -Total Storage Cost: $20,640/year -Total with Operations: ~$25,500/year -Savings: $117,500/year (82%) - -Warning: High retrieval costs if archived data is accessed frequently -``` - -## Comparison Table: All Strategies - -| Strategy | Annual Cost | Savings | Best For | -|----------|-------------|---------|----------| -| **No Optimization** | $143,000 | 0% | N/A | -| **Lifecycle Policy** | $76,500 | 46% | Predictable patterns | -| **Autoclass** | $37,000 | **74%** | **Recommended** | -| **Hybrid Approach** | $45,000 | 69% | Fine-grained control | -| **Aggressive Archival** | $25,500 | 82% | Cold data, compliance | - -## Autoclass vs Lifecycle Policies - -| Feature | Autoclass | Lifecycle Policies | -|---------|-----------|-------------------| -| **Automation** | Fully automatic | Rule-based | -| **Access-pattern learning** | Yes (90-day history) | No | -| **Promotion to Standard** | Automatic on access | Manual only | -| **Terminal class** | Configurable | Fixed by rules | -| **Complexity** | Single command | Multiple rules | -| **Cost** | Lower (smarter) | Moderate | -| **Best for** | Unpredictable patterns | Predictable patterns | - -## Batch Delete Operations - -### Efficient Cleanup - -```typescript -// Batch delete (100 objects per request for GCS) -const idsToDelete = [/* array of entity IDs */] - -const paths = idsToDelete.flatMap(id => { - const shard = id.substring(0, 2) - return [ - `entities/nouns/vectors/${shard}/${id}.json`, - `entities/nouns/metadata/${shard}/${id}.json` - ] -}) - -// Batch delete -await storage.batchDelete(paths) - -// Cost impact: -// - Individual deletes: 1M operations × $0.005 per 10k = $500 -// - Batch deletes: 10k batches × $0.005 = $5 (100x cheaper!) -``` - -## Monitoring and Management - -### Check Autoclass Status - -```typescript -const status = await storage.getAutoclassStatus() -console.log('Autoclass enabled:', status.enabled) -console.log('Terminal class:', status.terminalStorageClass) - -// Example output: -// { -// enabled: true, -// terminalStorageClass: 'ARCHIVE', -// toggleTime: '2025-01-15T10:30:00Z' -// } -``` - -### Disable Autoclass - -```typescript -// Disable Autoclass (objects remain in current class) -await storage.disableAutoclass() -``` - -### Get Current Lifecycle Policy - -```typescript -const policy = await storage.getLifecyclePolicy() -console.log('Active rules:', policy.rules) -``` - -### Remove Lifecycle Policy - -```typescript -await storage.removeLifecyclePolicy() -``` - -## GCS Cloud Console Monitoring - -### Track Your Savings - -1. **Storage Browser** → View storage class distribution -2. **Monitoring** → Create custom dashboards for storage metrics -3. **Cloud Logging** → Track class transition events -4. **Cloud Billing Reports** → Compare storage costs month-over-month - -### Expected Metrics After 6 Months (Autoclass) - -``` -Standard: 10-15% of total data -Nearline: 15-20% -Coldline: 30-40% -Archive: 30-45% - -Monthly cost trend: Decreasing 8-12% per month as data ages into cheaper classes -``` - -## Best Practices - -1. ✅ **Start with Autoclass** - Simplest and most effective -2. ✅ **Set terminal class to ARCHIVE** for maximum savings -3. ✅ **Use lifecycle policies for system files** - Predictable archival -4. ✅ **Monitor class distribution** monthly in Cloud Console -5. ✅ **Use batch operations** for deletions - 100x cheaper -6. ✅ **Enable Object Lifecycle Management logging** for auditing -7. ✅ **Consider Turbo Replication** for multi-region redundancy - -## Troubleshooting - -### Issue: Data not transitioning to cheaper classes - -**Solution:** -```typescript -// Check Autoclass status -const status = await storage.getAutoclassStatus() -if (!status.enabled) { - await storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' }) -} - -// Autoclass requires 24-48 hours for initial transitions -``` - -### Issue: High retrieval costs - -**Solution:** -- GCS has lower retrieval fees than AWS Glacier ($0.01-0.05/GB vs $0.01-0.20/GB) -- Autoclass automatically promotes frequently accessed objects to Standard -- Use Coldline for occasional access (better than Archive) - -### Issue: Minimum storage duration charges - -**Solution:** -- Nearline: 30-day minimum -- Coldline: 90-day minimum -- Archive: 365-day minimum -- Early deletion incurs pro-rated charges -- Use Autoclass to avoid manual class changes that might trigger early deletion fees - -## ADC (Application Default Credentials) Setup - -### Production Best Practice - -```typescript -// Use ADC instead of service account key file -const storage = new GcsStorage({ - bucketName: 'my-brainy-data' - // No keyFilename needed - uses ADC automatically -}) - -// ADC authentication order: -// 1. GOOGLE_APPLICATION_CREDENTIALS environment variable -// 2. gcloud CLI credentials -// 3. Compute Engine/Cloud Run service account -``` - -### Set up ADC - -```bash -# For local development -gcloud auth application-default login - -# For production (use service account) -export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json" - -# For Cloud Run/GKE/Compute Engine (automatic) -# Service account is automatically available -``` - -## Summary - -**Recommended Strategy for Most Use Cases:** -- **Autoclass** for automatic optimization (simplest, most effective) -- **Lifecycle policies** for predictable archival (system files, logs) -- **Batch operations** for efficient cleanup - -**Expected Savings:** -- **Year 1**: 50-60% reduction in storage costs -- **Year 2+**: 70-80% reduction as more data ages into archive classes -- **Long-term**: 85-90% reduction for mature datasets - -**500TB Example (Autoclass):** -- Before: $143,000/year -- After: $37,000/year -- **Savings: $106,000/year (74%)** - -**1PB Example (Autoclass):** -- Before: $286,000/year -- After: $74,000/year -- **Savings: $212,000/year (74%)** - -**10PB Example (Autoclass):** -- Before: $2,860,000/year -- After: $740,000/year -- **Savings: $2,120,000/year (74%)** - -## Strategy 5: HNS Buckets for Write-Heavy Workloads - -### When to Use HNS (Hierarchical Namespace) - -If you experience HTTP 429 rate limit errors during `brain.add()` operations (especially with many metadata fields), GCS Hierarchical Namespace buckets provide significantly higher write throughput with zero code changes. - -### Why It Helps - -Standard GCS buckets enforce ~1 write/sec per object path. Brainy's metadata index writes multiple chunk and sparse index files per `brain.add()` call, which can exceed these limits during burst writes. - -HNS buckets provide: -- **8x higher initial QPS** (8,000 writes/sec vs 1,000 for standard buckets) -- **Better handling of hierarchical key patterns** (which brainy uses for sharded storage) -- **No code changes required** — create a new HNS-enabled bucket and point brainy at it - -### Setup - -```bash -# Create HNS-enabled bucket -gcloud storage buckets create gs://my-brainy-hns \ - --location=us-central1 \ - --uniform-bucket-level-access \ - --enable-hierarchical-namespace -``` - -Then configure brainy to use the new bucket: - -```typescript -const storage = new GcsStorage({ - bucketName: 'my-brainy-hns' -}) -``` - -### Trade-offs - -- HNS buckets do not support object versioning (irrelevant for brainy — uses its own COW versioning) -- HNS is available in most GCS regions -- Pricing is the same as standard buckets - -### Alternative: Cloud Run with Filestore/NFS - -For the highest write throughput with zero rate limiting, see the [Cloud Run Filestore Guide](cloud-run-filestore-guide.md) — mount a Filestore NFS volume and use brainy's FileSystem adapter instead of GCS. - ---- - -**Last Updated**: 2025-10-17 -**Cloud Provider**: Google Cloud Storage diff --git a/src/graph/graphAdjacencyIndex.ts b/src/graph/graphAdjacencyIndex.ts index a5776990..ddca800f 100644 --- a/src/graph/graphAdjacencyIndex.ts +++ b/src/graph/graphAdjacencyIndex.ts @@ -628,105 +628,43 @@ export class GraphAdjacencyIndex implements GraphIndexProvider { // Note: LSM-trees will be recreated from storage via their own initialization // Verb data will be loaded on-demand via UnifiedCache - // Adaptive loading strategy based on storage type + // Brainy 8.0: storage is always local (filesystem or memory) per + // BR-BRAINY-80-STORAGE-SIMPLIFY. Load all verbs at once. const storageType = this.storage?.constructor.name || '' - const isLocalStorage = - storageType === 'FileSystemStorage' || - storageType === 'MemoryStorage' || - storageType === 'OPFSStorage' - let totalVerbs = 0 - if (isLocalStorage) { - // Local storage: Load all verbs at once to avoid repeated getAllShardedFiles() calls - prodLog.info( - `GraphAdjacencyIndex: Using optimized strategy - load all verbs at once (${storageType})` - ) + prodLog.info(`GraphAdjacencyIndex: Load all verbs at once (${storageType})`) - const result = await this.storage.getVerbs({ - pagination: { limit: 10000000 } // Effectively unlimited for local development - }) + const result = await this.storage.getVerbs({ + pagination: { limit: 10000000 } // Effectively unlimited for local storage + }) - // Add each verb to index - for (const verb of result.items) { - // Convert HNSWVerbWithMetadata to GraphVerb format - const graphVerb: GraphVerb = { - id: verb.id, - sourceId: verb.sourceId, - targetId: verb.targetId, - vector: verb.vector, - source: verb.sourceId, - target: verb.targetId, - verb: verb.verb, - createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 }, - updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 }, - createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' }, - service: verb.service, - data: verb.data, - embedding: verb.vector, - confidence: verb.confidence, - weight: verb.weight - } - await this.addVerb(graphVerb) - totalVerbs++ + for (const verb of result.items) { + const graphVerb: GraphVerb = { + id: verb.id, + sourceId: verb.sourceId, + targetId: verb.targetId, + vector: verb.vector, + source: verb.sourceId, + target: verb.targetId, + verb: verb.verb, + createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 }, + updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 }, + createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' }, + service: verb.service, + data: verb.data, + embedding: verb.vector, + confidence: verb.confidence, + weight: verb.weight } - - prodLog.info( - `GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs at once (local storage)` - ) - } else { - // Cloud storage: Use pagination with native cloud APIs (efficient) - prodLog.info( - `GraphAdjacencyIndex: Using cloud pagination strategy (${storageType})` - ) - - let hasMore = true - let cursor: string | undefined = undefined - const batchSize = 1000 - - while (hasMore) { - const result = await this.storage.getVerbs({ - pagination: { limit: batchSize, cursor } - }) - - // Add each verb to index - for (const verb of result.items) { - // Convert HNSWVerbWithMetadata to GraphVerb format - const graphVerb: GraphVerb = { - id: verb.id, - sourceId: verb.sourceId, - targetId: verb.targetId, - vector: verb.vector, - source: verb.sourceId, - target: verb.targetId, - verb: verb.verb, - createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 }, - updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 }, - createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' }, - service: verb.service, - data: verb.data, - embedding: verb.vector, - confidence: verb.confidence, - weight: verb.weight - } - await this.addVerb(graphVerb) - totalVerbs++ - } - - hasMore = result.hasMore - cursor = result.nextCursor - - // Progress logging - if (totalVerbs % 10000 === 0) { - prodLog.info(`GraphAdjacencyIndex: Indexed ${totalVerbs} verbs...`) - } - } - - prodLog.info( - `GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs via pagination (cloud storage)` - ) + await this.addVerb(graphVerb) + totalVerbs++ } + prodLog.info( + `GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs (${storageType})` + ) + const rebuildTime = Date.now() - this.rebuildStartTime const memoryUsage = this.calculateMemoryUsage() diff --git a/src/hnsw/hnswIndex.ts b/src/hnsw/hnswIndex.ts index 07ab2dbd..06806658 100644 --- a/src/hnsw/hnswIndex.ts +++ b/src/hnsw/hnswIndex.ts @@ -1470,20 +1470,15 @@ export class JsHnswVectorIndex implements VectorIndexProvider { ) } - // Step 4: Adaptive loading strategy based on storage type - // FileSystem/Memory/OPFS: Load all at once (avoids repeated getAllShardedFiles() calls) - // Cloud (GCS/S3/R2): Use pagination (efficient native cloud APIs) + // Step 4: Load all HNSW nodes at once. Brainy 8.0 ships filesystem + + // memory storage only (per BR-BRAINY-80-STORAGE-SIMPLIFY); the + // cloud-pagination rebuild path was deleted alongside the cloud adapters. const storageType = this.storage?.constructor.name || '' - const isLocalStorage = storageType === 'FileSystemStorage' || - storageType === 'MemoryStorage' || - storageType === 'OPFSStorage' - let loadedCount = 0 let totalCount: number | undefined = undefined - if (isLocalStorage) { - // Local storage: Load all nouns at once - prodLog.info(`HNSW: Using optimized strategy - load all nodes at once (${storageType})`) + { + prodLog.info(`HNSW: Load all nodes at once (${storageType})`) const result: { items: HNSWNoun[] @@ -1554,94 +1549,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider { options.onProgress(loadedCount, totalCount) } - prodLog.info(`HNSW: Loaded ${loadedCount.toLocaleString()} nodes at once (local storage)`) - - } else { - // Cloud storage: Use pagination with native cloud APIs - prodLog.info(`HNSW: Using cloud pagination strategy (${storageType})`) - - let hasMore = true - let offset = 0 // Use offset-based pagination instead of cursor (bug fix for infinite loop) - - while (hasMore) { - // Fetch batch of nouns from storage (cast needed as method is not in base interface) - const result: { - items: HNSWNoun[] - totalCount?: number - hasMore: boolean - nextCursor?: string - } = await (this.storage as any).getNounsWithPagination({ - limit: batchSize, - offset // Pass offset for proper pagination (previously passed cursor which was ignored) - }) - - // Set total count on first batch - if (totalCount === undefined && result.totalCount !== undefined) { - totalCount = result.totalCount - } - - // Process each noun in the batch - for (const nounData of result.items) { - try { - // Load HNSW graph data for this entity - const hnswData = await (this.storage as any).getVectorIndexData(nounData.id) - - if (!hnswData) { - // No HNSW data - skip (might be entity added before persistence) - continue - } - - // Determine if vector should be kept in memory - const keepVector = shouldPreload && this.vectorStorageMode !== 'lazy' - - // Create noun object with restored connections - const noun: HNSWNoun = { - id: nounData.id, - vector: keepVector ? nounData.vector : [], // Preload if dataset is small and not lazy - connections: new Map(), - level: hnswData.level - } - - // Restore SQ8 quantized data if quantization is enabled - if (this.quantizationEnabled && nounData.vector.length > 0) { - const sq8 = quantizeSQ8(nounData.vector) - noun.quantizedVector = sq8.quantized - noun.codebookMin = sq8.min - noun.codebookMax = sq8.max - } - - // Restore connections from persisted data — compressed blob path - // first, legacy JSON-array fallback for indexes written before - // graph link compression landed. - await this.restoreNodeConnections(nounData.id, hnswData, noun) - - // Add to in-memory index - this.nouns.set(nounData.id, noun) - - // Track high-level nodes for O(1) entry point selection - if (noun.level >= 2 && noun.level <= this.MAX_TRACKED_LEVELS) { - if (!this.highLevelNodes.has(noun.level)) { - this.highLevelNodes.set(noun.level, new Set()) - } - this.highLevelNodes.get(noun.level)!.add(nounData.id) - } - - loadedCount++ - } catch (error) { - // Log error but continue (robust error recovery) - console.error(`Failed to rebuild HNSW data for ${nounData.id}:`, error) - } - } - - // Report progress - if (options.onProgress && totalCount !== undefined) { - options.onProgress(loadedCount, totalCount) - } - - // Check for more data - hasMore = result.hasMore - offset += batchSize // Increment offset for next page - } + prodLog.info(`HNSW: Loaded ${loadedCount.toLocaleString()} nodes (${storageType})`) } // Step 5: CRITICAL - Recover entry point if missing) diff --git a/src/utils/metadataIndex.ts b/src/utils/metadataIndex.ts index 5be89566..9d38bf60 100644 --- a/src/utils/metadataIndex.ts +++ b/src/utils/metadataIndex.ts @@ -3056,21 +3056,13 @@ export class MetadataIndexManager implements MetadataIndexProvider { // Clear chunk manager cache this.chunkManager.clearCache() - // Adaptive rebuild strategy based on storage adapter - // FileSystem/Memory/OPFS: Load all at once (avoids getAllShardedFiles() overhead on every batch) - // Cloud (GCS/S3/R2): Use pagination with small batches (prevent socket exhaustion) - const storageType = this.storage.constructor.name - const isLocalStorage = storageType === 'FileSystemStorage' || - storageType === 'MemoryStorage' || - storageType === 'OPFSStorage' - - let nounLimit: number + // Brainy 8.0 ships filesystem + memory storage only. Load all nouns + // at once — the cloud-storage paginated branch was deleted alongside + // the cloud adapters in step 7. let totalNounsProcessed = 0 - if (isLocalStorage) { - // Load all nouns at once for local storage - // Avoids repeated directory scans in getAllShardedFiles() - prodLog.info(`⚡ Using optimized strategy: load all nouns at once (local storage)`) + { + prodLog.info(`⚡ Loading all nouns at once (local storage)`) const result = await this.storage.getNouns({ pagination: { offset: 0, limit: 1000000 } // Effectively unlimited }) @@ -3085,7 +3077,6 @@ export class MetadataIndexManager implements MetadataIndexProvider { metadataBatch = await this.storage.getMetadataBatch(nounIds) prodLog.info(`✅ Loaded ${metadataBatch.size}/${nounIds.length} metadata objects`) } else { - // Fallback to individual calls metadataBatch = new Map() for (const id of nounIds) { try { @@ -3097,131 +3088,21 @@ export class MetadataIndexManager implements MetadataIndexProvider { } } - // Process all nouns - let localCount = 0 for (const noun of result.items) { const metadata = metadataBatch.get(noun.id) if (metadata) { await this.addToIndex(noun.id, metadata, true, true) - localCount++ - // Periodic safety flush every 5000 entities to cap memory during rebuild - // (Periodic safety flush removed in 7.22.0 — the underlying - // flushDirtyMetadata was a no-op since the 7.20.0 column-store - // refactor. Column store handles its own flushing.) } } totalNounsProcessed = result.items.length prodLog.info(`✅ Indexed ${totalNounsProcessed} nouns`) - - } else { - // Cloud storage: use conservative batching - nounLimit = 25 - prodLog.info(`⚡ Using conservative batch size: ${nounLimit} items/batch (cloud storage)`) - - let nounOffset = 0 - let hasMoreNouns = true - let consecutiveEmptyBatches = 0 - const MAX_ITERATIONS = 10000 - let iterations = 0 - - while (hasMoreNouns && iterations < MAX_ITERATIONS) { - iterations++ - const result = await this.storage.getNouns({ - pagination: { offset: nounOffset, limit: nounLimit } - }) - - // CRITICAL SAFETY CHECK: Prevent infinite loop on empty results - if (result.items.length === 0) { - consecutiveEmptyBatches++ - if (consecutiveEmptyBatches >= 3) { - prodLog.warn('⚠️ Breaking metadata rebuild loop: received 3 consecutive empty batches') - break - } - // If hasMore is true but items are empty, it's likely a bug - if (result.hasMore) { - prodLog.warn(`⚠️ Storage returned empty items but hasMore=true at offset ${nounOffset}`) - hasMoreNouns = false // Force exit - break - } - } else { - consecutiveEmptyBatches = 0 // Reset counter on non-empty batch - } - - // CRITICAL FIX: Use batch metadata reading to prevent socket exhaustion - const nounIds = result.items.map(noun => noun.id) - - let metadataBatch: Map - if (this.storage.getMetadataBatch) { - // Use batch reading if available (prevents socket exhaustion) - prodLog.info(`📦 Processing metadata batch ${Math.floor(totalNounsProcessed / nounLimit) + 1} (${nounIds.length} items)...`) - metadataBatch = await this.storage.getMetadataBatch(nounIds) - const successRate = ((metadataBatch.size / nounIds.length) * 100).toFixed(1) - prodLog.info(`✅ Batch loaded ${metadataBatch.size}/${nounIds.length} metadata objects (${successRate}% success)`) - } else { - // Fallback to individual calls with strict concurrency control - prodLog.warn(`⚠️ FALLBACK: Storage adapter missing getMetadataBatch - using individual calls with concurrency limit`) - metadataBatch = new Map() - const CONCURRENCY_LIMIT = 3 // Very conservative limit - - for (let i = 0; i < nounIds.length; i += CONCURRENCY_LIMIT) { - const batch = nounIds.slice(i, i + CONCURRENCY_LIMIT) - const batchPromises = batch.map(async (id) => { - try { - const metadata = await this.storage.getNounMetadata(id) - return { id, metadata } - } catch (error) { - prodLog.debug(`Failed to read metadata for ${id}:`, error) - return { id, metadata: null } - } - }) - - const batchResults = await Promise.all(batchPromises) - for (const { id, metadata } of batchResults) { - if (metadata) { - metadataBatch.set(id, metadata) - } - } - - // Yield between batches to prevent socket exhaustion - await this.yieldToEventLoop() - } - } - - // Process the metadata batch - for (const noun of result.items) { - const metadata = metadataBatch.get(noun.id) - if (metadata) { - // Skip flush during rebuild for performance - await this.addToIndex(noun.id, metadata, true, true) - } - } - - // Yield after processing the entire batch - await this.yieldToEventLoop() - - totalNounsProcessed += result.items.length - hasMoreNouns = result.hasMore - nounOffset += nounLimit - - // Progress logging and event loop yield after each batch - if (totalNounsProcessed % 100 === 0 || !hasMoreNouns) { - prodLog.debug(`📊 Indexed ${totalNounsProcessed} nouns...`) - } - await this.yieldToEventLoop() - } - - // Check iteration limits for cloud storage - if (iterations >= MAX_ITERATIONS) { - prodLog.error(`❌ Metadata noun rebuild hit maximum iteration limit (${MAX_ITERATIONS}). This indicates a bug in storage pagination.`) - } } - // Rebuild verb metadata indexes - same strategy as nouns + // Rebuild verb metadata indexes — same single-pass local strategy. let totalVerbsProcessed = 0 - if (isLocalStorage) { - // Load all verbs at once for local storage + { prodLog.info(`⚡ Loading all verbs at once (local storage)`) const result = await this.storage.getVerbs({ pagination: { offset: 0, limit: 1000000 } // Effectively unlimited @@ -3229,7 +3110,6 @@ export class MetadataIndexManager implements MetadataIndexProvider { prodLog.info(`📦 Loading ${result.items.length} verbs with metadata...`) - // Get all verb metadata at once const verbIds = result.items.map(verb => verb.id) let verbMetadataBatch: Map @@ -3248,127 +3128,24 @@ export class MetadataIndexManager implements MetadataIndexProvider { } } - // Process all verbs - let verbLocalCount = 0 for (const verb of result.items) { const metadata = verbMetadataBatch.get(verb.id) if (metadata) { await this.addToIndex(verb.id, metadata, true, true) - verbLocalCount++ - // (Periodic safety flush removed in 7.22.0 — see noun loop above.) } } totalVerbsProcessed = result.items.length prodLog.info(`✅ Indexed ${totalVerbsProcessed} verbs`) - - } else { - // Cloud storage: use conservative batching - let verbOffset = 0 - const verbLimit = 25 - let hasMoreVerbs = true - let consecutiveEmptyVerbBatches = 0 - let verbIterations = 0 - const MAX_ITERATIONS = 10000 - - while (hasMoreVerbs && verbIterations < MAX_ITERATIONS) { - verbIterations++ - const result = await this.storage.getVerbs({ - pagination: { offset: verbOffset, limit: verbLimit } - }) - - // CRITICAL SAFETY CHECK: Prevent infinite loop on empty results - if (result.items.length === 0) { - consecutiveEmptyVerbBatches++ - if (consecutiveEmptyVerbBatches >= 3) { - prodLog.warn('⚠️ Breaking verb metadata rebuild loop: received 3 consecutive empty batches') - break - } - // If hasMore is true but items are empty, it's likely a bug - if (result.hasMore) { - prodLog.warn(`⚠️ Storage returned empty verb items but hasMore=true at offset ${verbOffset}`) - hasMoreVerbs = false // Force exit - break - } - } else { - consecutiveEmptyVerbBatches = 0 // Reset counter on non-empty batch - } - - // CRITICAL FIX: Use batch verb metadata reading to prevent socket exhaustion - const verbIds = result.items.map(verb => verb.id) - - let verbMetadataBatch: Map - if ((this.storage as any).getVerbMetadataBatch) { - // Use batch reading if available (prevents socket exhaustion) - verbMetadataBatch = await (this.storage as any).getVerbMetadataBatch(verbIds) - prodLog.debug(`📦 Batch loaded ${verbMetadataBatch.size}/${verbIds.length} verb metadata objects`) - } else { - // Fallback to individual calls with strict concurrency control - verbMetadataBatch = new Map() - const CONCURRENCY_LIMIT = 3 // Very conservative limit to prevent socket exhaustion - - for (let i = 0; i < verbIds.length; i += CONCURRENCY_LIMIT) { - const batch = verbIds.slice(i, i + CONCURRENCY_LIMIT) - const batchPromises = batch.map(async (id) => { - try { - const metadata = await this.storage.getVerbMetadata(id) - return { id, metadata } - } catch (error) { - prodLog.debug(`Failed to read verb metadata for ${id}:`, error) - return { id, metadata: null } - } - }) - - const batchResults = await Promise.all(batchPromises) - for (const { id, metadata } of batchResults) { - if (metadata) { - verbMetadataBatch.set(id, metadata) - } - } - - // Yield between batches to prevent socket exhaustion - await this.yieldToEventLoop() - } - } - - // Process the verb metadata batch - for (const verb of result.items) { - const metadata = verbMetadataBatch.get(verb.id) - if (metadata) { - // Skip flush during rebuild for performance - await this.addToIndex(verb.id, metadata, true, true) - } - } - - // Yield after processing the entire batch - await this.yieldToEventLoop() - - totalVerbsProcessed += result.items.length - hasMoreVerbs = result.hasMore - verbOffset += verbLimit - - // Progress logging and event loop yield after each batch - if (totalVerbsProcessed % 100 === 0 || !hasMoreVerbs) { - prodLog.debug(`🔗 Indexed ${totalVerbsProcessed} verbs...`) - } - await this.yieldToEventLoop() - } - - // Check iteration limits for cloud storage - if (verbIterations >= MAX_ITERATIONS) { - prodLog.error(`❌ Metadata verb rebuild hit maximum iteration limit (${MAX_ITERATIONS}). This indicates a bug in storage pagination.`) - } } - // Flush to storage with final yield. The column store's flush() handles - // tail-buffer-to-segment promotion + manifest persistence. + // Flush to storage. The column store's flush() handles tail-buffer-to- + // segment promotion + manifest persistence. prodLog.debug('💾 Flushing metadata index to storage...') await this.flush() - await this.yieldToEventLoop() prodLog.info(`✅ Metadata index rebuild completed! Processed ${totalNounsProcessed} nouns and ${totalVerbsProcessed} verbs`) - prodLog.info(`🎯 Initial indexing may show minor socket timeouts - this is expected and doesn't affect data processing`) - + } finally { this.isRebuilding = false }