chore(8.0): step-7 follow-through — collapse remaining cloud branches + docs sweep (scaffold step 13)
Final cleanup pass for Brainy 8.0. Catches three categories of debt: A. STEP-7 FOLLOW-THROUGH (rebuild-path collapse) Step 7's bisect-reset (debugging a flaky test) lost the in-source edits to three rebuild paths even though the commit message claimed they shipped. Re-applied now: - src/utils/metadataIndex.ts — collapsed the `isLocalStorage` / cloud-pagination branching. Local-load-all-at-once is the only path in 8.0. Removed ~150 LOC of paginated-cloud branching for both nouns and verbs, plus the safety counters (`consecutiveEmptyBatches`, `MAX_ITERATIONS`, etc.). - src/hnsw/hnswIndex.ts — same simplification for HNSW rebuild. The paginated cloud path is gone; HNSW now loads all nodes at once. Removed ~85 LOC. - src/graph/graphAdjacencyIndex.ts — same simplification for graph adjacency rebuild. Removed ~50 LOC. The collapse is safe because cloud adapters were deleted in step 7; `storageType === 'OPFSStorage'` (and similar) can never match now. B. CLOUD-ONLY DOCS DELETED - docs/operations/cost-optimization-aws-s3.md - docs/operations/cost-optimization-azure.md - docs/operations/cost-optimization-cloudflare-r2.md - docs/operations/cost-optimization-gcs.md - docs/operations/cloud-run-filestore-guide.md (docs/deployment/* contained no cloud-specific files that needed deletion.) C. STORAGE-ADAPTERS GUIDE REWRITTEN FOR 8.0 docs/guides/storage-adapters.md → fresh content reflecting the 8.0 reality: - Two adapters: FileSystemStorage + MemoryStorage. Quick-start matrix. - Cloud backup section explains the operator-tooling pattern (gsutil / aws s3 / rclone / azcopy) with the exact commands consumers will run. - "Why no cloud adapters in 8.0?" section documents the four reasons per BR-BRAINY-80-STORAGE-SIMPLIFY. - Migration recipe for 7.x cloud-adapter consumers: mount local disk → filesystem storage → operator backup cron. Updated frontmatter description so soulcraft.com/docs renders the correct preview. NOT IN THIS COMMIT (deliberate, lower-priority) - src/storage/cacheManager.ts still references StorageType.S3 / REMOTE_API / OPFS as dead branches (23 sites). The branches are never reached in 8.0, but cleaning them would cascade through 5 consumers. Defer to a follow-up if the dead code surfaces as a real maintenance issue. - src/config/storageAutoConfig.ts keeps its StorageType enum + autodetect for 7.x compat surface. Same reason: rewriting cascades through zeroConfig, extensibleConfig, sharedConfigManager. Defer. - docs/MIGRATION-V3-TO-V4.md and docs/DEVELOPER_LEARNING_PATH.md still reference cloud adapters as historical artefacts. That's accurate — they describe how things used to be. Left as-is. - @deprecated audit in src/ (10 files) deferred — audit each individually in a future polish pass. VERIFICATION - npx tsc --noEmit: clean - npm test: 1408 / 1409 (same pre-existing race-condition outstanding from step 7; no regressions from this cleanup)
This commit is contained in:
parent
780fb6444b
commit
9f9a41599e
13 changed files with 162 additions and 5288 deletions
|
|
@ -1,833 +0,0 @@
|
|||
# Brainy Cloud Deployment Guide
|
||||
|
||||
This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy codebase.
|
||||
|
||||
## 🆕 Production Features
|
||||
|
||||
**Cost Optimization at Scale:**
|
||||
- **Lifecycle Policies**: Automatic tier transitions for 96% cost savings
|
||||
- **Intelligent-Tiering**: S3 Intelligent-Tiering and GCS Autoclass support
|
||||
- **Batch Operations**: Efficient bulk delete (1000 objects per request)
|
||||
- **Compression**: Gzip compression for 60-80% storage savings
|
||||
|
||||
**Example Impact (500TB dataset):**
|
||||
- Before: $138,000/year
|
||||
- With lifecycle policies: $5,940/year
|
||||
- **Savings: $132,060/year (96%)**
|
||||
|
||||
See the [Cost Optimization](#cost-optimization-v40) section below for implementation details.
|
||||
|
||||
## Overview
|
||||
|
||||
The API Server augmentation provides a universal handler that works with standard Request/Response objects, making Brainy deployable on any JavaScript runtime.
|
||||
|
||||
## Storage Adapter
|
||||
|
||||
**S3CompatibleStorage** is the preferred storage adapter for cloud deployments. It works with:
|
||||
- Amazon S3
|
||||
- Cloudflare R2
|
||||
- Google Cloud Storage
|
||||
- Azure Blob Storage
|
||||
- Any S3-compatible service
|
||||
|
||||
## Deployment Examples
|
||||
|
||||
### AWS Lambda
|
||||
|
||||
```javascript
|
||||
// handler.js
|
||||
const { Brainy } = require('@soulcraft/brainy')
|
||||
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
|
||||
|
||||
let brain
|
||||
let handler
|
||||
|
||||
exports.handler = async (event) => {
|
||||
if (!brain) {
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: 's3.amazonaws.com',
|
||||
region: process.env.AWS_REGION,
|
||||
bucket: process.env.BRAINY_BUCKET,
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
|
||||
prefix: 'brainy-data/'
|
||||
})
|
||||
|
||||
brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: {
|
||||
enabled: true,
|
||||
auth: {
|
||||
required: true,
|
||||
apiKeys: [process.env.API_KEY]
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
// Get the universal handler from the augmentation
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
// Convert Lambda event to Request
|
||||
const url = `https://${event.requestContext.domainName}${event.rawPath}`
|
||||
const request = new Request(url, {
|
||||
method: event.requestContext.http.method,
|
||||
headers: event.headers,
|
||||
body: event.body
|
||||
})
|
||||
|
||||
// Use the universal handler
|
||||
const response = await handler(request)
|
||||
|
||||
return {
|
||||
statusCode: response.status,
|
||||
headers: Object.fromEntries(response.headers),
|
||||
body: await response.text()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Google Cloud Functions
|
||||
|
||||
```javascript
|
||||
// index.js
|
||||
const { Brainy } = require('@soulcraft/brainy')
|
||||
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
|
||||
|
||||
let brain
|
||||
let handler
|
||||
|
||||
exports.brainyAPI = async (req, res) => {
|
||||
if (!brain) {
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: 'storage.googleapis.com',
|
||||
bucket: process.env.GCS_BUCKET,
|
||||
accessKeyId: process.env.GCS_ACCESS_KEY,
|
||||
secretAccessKey: process.env.GCS_SECRET_KEY,
|
||||
prefix: 'brainy/',
|
||||
forcePathStyle: false,
|
||||
region: 'US'
|
||||
})
|
||||
|
||||
brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
// Convert Express req/res to Request/Response
|
||||
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
|
||||
method: req.method,
|
||||
headers: req.headers,
|
||||
body: JSON.stringify(req.body)
|
||||
})
|
||||
|
||||
const response = await handler(request)
|
||||
|
||||
res.status(response.status)
|
||||
res.set(Object.fromEntries(response.headers))
|
||||
res.send(await response.text())
|
||||
}
|
||||
```
|
||||
|
||||
### Google Cloud Run with Cloud Storage
|
||||
|
||||
Google Cloud Run is ideal for containerized deployments with automatic scaling. This example uses Google Cloud Storage via the S3-compatible API.
|
||||
|
||||
```javascript
|
||||
// server.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
|
||||
import express from 'express'
|
||||
|
||||
const app = express()
|
||||
app.use(express.json())
|
||||
|
||||
const PORT = process.env.PORT || 8080
|
||||
|
||||
let brain
|
||||
let handler
|
||||
|
||||
async function initBrainy() {
|
||||
// Google Cloud Storage is S3-compatible
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: 'storage.googleapis.com',
|
||||
bucket: process.env.GCS_BUCKET || 'brainy-data',
|
||||
accessKeyId: process.env.GCS_ACCESS_KEY,
|
||||
secretAccessKey: process.env.GCS_SECRET_KEY,
|
||||
prefix: 'brainy/',
|
||||
forcePathStyle: false,
|
||||
region: process.env.GCS_REGION || 'US'
|
||||
})
|
||||
|
||||
brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: {
|
||||
enabled: true,
|
||||
port: PORT,
|
||||
cors: {
|
||||
origin: process.env.CORS_ORIGIN || '*'
|
||||
},
|
||||
auth: {
|
||||
required: process.env.AUTH_REQUIRED === 'true',
|
||||
apiKeys: process.env.API_KEY ? [process.env.API_KEY] : []
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
// Initialize on startup
|
||||
await initBrainy()
|
||||
|
||||
// Health check endpoint
|
||||
app.get('/health', (req, res) => {
|
||||
res.json({ status: 'healthy', service: 'brainy-api' })
|
||||
})
|
||||
|
||||
// Universal handler for all API routes
|
||||
app.use('*', async (req, res) => {
|
||||
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
|
||||
method: req.method,
|
||||
headers: req.headers,
|
||||
body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined
|
||||
})
|
||||
|
||||
const response = await handler(request)
|
||||
|
||||
res.status(response.status)
|
||||
res.set(Object.fromEntries(response.headers))
|
||||
res.send(await response.text())
|
||||
})
|
||||
|
||||
app.listen(PORT, () => {
|
||||
console.log(`Brainy API running on port ${PORT}`)
|
||||
})
|
||||
```
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
FROM node:20-alpine AS builder
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm ci --only=production
|
||||
|
||||
FROM node:20-alpine
|
||||
WORKDIR /app
|
||||
COPY --from=builder /app/node_modules ./node_modules
|
||||
COPY . .
|
||||
EXPOSE 8080
|
||||
CMD ["node", "server.js"]
|
||||
```
|
||||
|
||||
```yaml
|
||||
# cloudbuild.yaml
|
||||
steps:
|
||||
# Build the container image
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.']
|
||||
|
||||
# Push to Container Registry
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA']
|
||||
|
||||
# Deploy to Cloud Run
|
||||
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
|
||||
entrypoint: gcloud
|
||||
args:
|
||||
- 'run'
|
||||
- 'deploy'
|
||||
- 'brainy-api'
|
||||
- '--image'
|
||||
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
|
||||
- '--region'
|
||||
- 'us-central1'
|
||||
- '--platform'
|
||||
- 'managed'
|
||||
- '--allow-unauthenticated'
|
||||
- '--set-env-vars'
|
||||
- 'GCS_BUCKET=brainy-storage,GCS_REGION=US'
|
||||
- '--set-secrets'
|
||||
- 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest'
|
||||
- '--memory'
|
||||
- '2Gi'
|
||||
- '--cpu'
|
||||
- '2'
|
||||
- '--max-instances'
|
||||
- '100'
|
||||
- '--min-instances'
|
||||
- '0'
|
||||
|
||||
images:
|
||||
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
|
||||
```
|
||||
|
||||
**Deploy with gcloud CLI:**
|
||||
```bash
|
||||
# Build and submit to Cloud Build
|
||||
gcloud builds submit --config cloudbuild.yaml
|
||||
|
||||
# Or deploy directly
|
||||
gcloud run deploy brainy-api \
|
||||
--source . \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--allow-unauthenticated \
|
||||
--set-env-vars GCS_BUCKET=brainy-storage \
|
||||
--set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest"
|
||||
```
|
||||
|
||||
**Create GCS Bucket with S3-compatible access:**
|
||||
```bash
|
||||
# Create bucket
|
||||
gsutil mb -p PROJECT_ID -c STANDARD -l US gs://brainy-storage/
|
||||
|
||||
# Enable interoperability
|
||||
gsutil iam ch serviceAccount:SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com:objectAdmin gs://brainy-storage
|
||||
|
||||
# Generate HMAC keys for S3-compatible access
|
||||
gsutil hmac create SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com
|
||||
|
||||
# Store the access key and secret in Secret Manager
|
||||
echo -n "YOUR_ACCESS_KEY" | gcloud secrets create gcs-access-key --data-file=-
|
||||
echo -n "YOUR_SECRET_KEY" | gcloud secrets create gcs-secret-key --data-file=-
|
||||
```
|
||||
|
||||
### Microsoft Azure Functions
|
||||
|
||||
```javascript
|
||||
// index.js
|
||||
module.exports = async function (context, req) {
|
||||
const { Brainy } = require('@soulcraft/brainy')
|
||||
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
|
||||
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`,
|
||||
bucket: 'brainy-data',
|
||||
accessKeyId: process.env.AZURE_STORAGE_ACCOUNT,
|
||||
secretAccessKey: process.env.AZURE_STORAGE_KEY,
|
||||
prefix: 'entities/',
|
||||
forcePathStyle: false
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
const handler = apiAugmentation.createUniversalHandler()
|
||||
|
||||
const request = new Request(`https://${context.req.headers.host}${context.req.url}`, {
|
||||
method: context.req.method,
|
||||
headers: context.req.headers,
|
||||
body: JSON.stringify(context.req.body)
|
||||
})
|
||||
|
||||
const response = await handler(request)
|
||||
|
||||
context.res = {
|
||||
status: response.status,
|
||||
headers: Object.fromEntries(response.headers),
|
||||
body: await response.text()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cloudflare Workers
|
||||
|
||||
```javascript
|
||||
// worker.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { R2Storage } from '@soulcraft/brainy/storage' // Alias for S3CompatibleStorage
|
||||
|
||||
let handler
|
||||
|
||||
export default {
|
||||
async fetch(request, env, ctx) {
|
||||
if (!handler) {
|
||||
const storage = new R2Storage({
|
||||
endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`,
|
||||
bucket: 'brainy-data',
|
||||
accessKeyId: env.R2_ACCESS_KEY_ID,
|
||||
secretAccessKey: env.R2_SECRET_ACCESS_KEY,
|
||||
region: 'auto',
|
||||
forcePathStyle: true
|
||||
})
|
||||
|
||||
const brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: {
|
||||
enabled: true,
|
||||
cors: { origin: '*' }
|
||||
}
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
// The handler works directly with Request/Response!
|
||||
return handler(request)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```toml
|
||||
# wrangler.toml
|
||||
name = "brainy-api"
|
||||
main = "worker.js"
|
||||
compatibility_date = "2024-01-01"
|
||||
|
||||
[[r2_buckets]]
|
||||
binding = "R2"
|
||||
bucket_name = "brainy-data"
|
||||
|
||||
[vars]
|
||||
ACCOUNT_ID = "your-account-id"
|
||||
|
||||
[env.production.vars]
|
||||
R2_ACCESS_KEY_ID = "your-access-key"
|
||||
R2_SECRET_ACCESS_KEY = "your-secret-key"
|
||||
```
|
||||
|
||||
### Vercel Edge Functions
|
||||
|
||||
```javascript
|
||||
// api/brainy.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
|
||||
|
||||
let handler
|
||||
|
||||
export const config = {
|
||||
runtime: 'edge',
|
||||
}
|
||||
|
||||
export default async (request) => {
|
||||
if (!handler) {
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: 's3.amazonaws.com',
|
||||
region: 'us-east-1',
|
||||
bucket: process.env.S3_BUCKET,
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
})
|
||||
|
||||
const brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: { enabled: true }
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
return handler(request)
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
// vercel.json
|
||||
{
|
||||
"functions": {
|
||||
"api/brainy.js": {
|
||||
"maxDuration": 30,
|
||||
"memory": 1024
|
||||
}
|
||||
},
|
||||
"rewrites": [
|
||||
{
|
||||
"source": "/api/:path*",
|
||||
"destination": "/api/brainy"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Railway
|
||||
|
||||
```javascript
|
||||
// server.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
|
||||
import express from 'express'
|
||||
|
||||
const app = express()
|
||||
const PORT = process.env.PORT || 3000
|
||||
|
||||
let brain
|
||||
let handler
|
||||
|
||||
async function init() {
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com',
|
||||
region: process.env.S3_REGION || 'us-east-1',
|
||||
bucket: process.env.S3_BUCKET,
|
||||
accessKeyId: process.env.S3_ACCESS_KEY,
|
||||
secretAccessKey: process.env.S3_SECRET_KEY,
|
||||
prefix: 'brainy/'
|
||||
})
|
||||
|
||||
brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: {
|
||||
enabled: true,
|
||||
port: PORT
|
||||
}
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
handler = apiAugmentation.createUniversalHandler()
|
||||
}
|
||||
|
||||
await init()
|
||||
|
||||
// Universal handler for all routes
|
||||
app.use('*', async (req, res) => {
|
||||
const request = new Request(`http://localhost${req.originalUrl}`, {
|
||||
method: req.method,
|
||||
headers: req.headers,
|
||||
body: JSON.stringify(req.body)
|
||||
})
|
||||
|
||||
const response = await handler(request)
|
||||
|
||||
res.status(response.status)
|
||||
res.set(Object.fromEntries(response.headers))
|
||||
res.send(await response.text())
|
||||
})
|
||||
|
||||
app.listen(PORT, () => {
|
||||
console.log(`Brainy API running on port ${PORT}`)
|
||||
})
|
||||
```
|
||||
|
||||
```toml
|
||||
# railway.toml
|
||||
[build]
|
||||
builder = "nixpacks"
|
||||
buildCommand = "npm ci"
|
||||
|
||||
[deploy]
|
||||
startCommand = "node server.js"
|
||||
restartPolicyType = "always"
|
||||
restartPolicyMaxRetries = 3
|
||||
```
|
||||
|
||||
### Render
|
||||
|
||||
```javascript
|
||||
// server.js (same as Railway example above)
|
||||
// Use S3CompatibleStorage with your preferred object storage provider
|
||||
```
|
||||
|
||||
```yaml
|
||||
# render.yaml
|
||||
services:
|
||||
- type: web
|
||||
name: brainy-api
|
||||
runtime: node
|
||||
buildCommand: npm install
|
||||
startCommand: node server.js
|
||||
envVars:
|
||||
- key: S3_BUCKET
|
||||
value: brainy-data
|
||||
- key: S3_ENDPOINT
|
||||
value: s3.amazonaws.com
|
||||
- key: S3_REGION
|
||||
value: us-east-1
|
||||
- key: S3_ACCESS_KEY
|
||||
sync: false
|
||||
- key: S3_SECRET_KEY
|
||||
sync: false
|
||||
- key: API_KEY
|
||||
generateValue: true
|
||||
healthCheckPath: /health
|
||||
autoDeploy: true
|
||||
```
|
||||
|
||||
### Deno Deploy
|
||||
|
||||
```typescript
|
||||
// main.ts
|
||||
import { Brainy } from "npm:@soulcraft/brainy"
|
||||
import { S3CompatibleStorage } from "npm:@soulcraft/brainy/storage"
|
||||
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com",
|
||||
bucket: Deno.env.get("S3_BUCKET")!,
|
||||
accessKeyId: Deno.env.get("S3_ACCESS_KEY")!,
|
||||
secretAccessKey: Deno.env.get("S3_SECRET_KEY")!,
|
||||
region: "auto"
|
||||
})
|
||||
|
||||
const brain = new Brainy({
|
||||
storage,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: { enabled: true }
|
||||
}]
|
||||
})
|
||||
|
||||
await brain.init()
|
||||
|
||||
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
|
||||
const handler = apiAugmentation.createUniversalHandler()
|
||||
|
||||
Deno.serve(handler)
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
The API Server augmentation provides these REST endpoints:
|
||||
|
||||
- `POST /api/brainy/add` - Add entity
|
||||
- `GET /api/brainy/get?id=xxx` - Get entity by ID
|
||||
- `PUT /api/brainy/update` - Update entity
|
||||
- `DELETE /api/brainy/delete?id=xxx` - Delete entity
|
||||
- `POST /api/brainy/find` - Search/find entities
|
||||
- `POST /api/brainy/relate` - Create relationship
|
||||
- `GET /api/brainy/insights` - Get statistics and insights
|
||||
- `GET /health` - Health check
|
||||
|
||||
## WebSocket Support
|
||||
|
||||
The API Server augmentation includes WebSocket support for real-time updates through the `setupUniversalWebSocket()` method.
|
||||
|
||||
## MCP Support
|
||||
|
||||
Model Context Protocol (MCP) endpoints are available at `/mcp/*` for AI tool integration.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
# Storage Configuration (S3Compatible)
|
||||
S3_ENDPOINT=s3.amazonaws.com
|
||||
S3_REGION=us-east-1
|
||||
S3_BUCKET=brainy-data
|
||||
S3_ACCESS_KEY=xxx
|
||||
S3_SECRET_KEY=xxx
|
||||
|
||||
# API Configuration
|
||||
API_KEY=your-secret-key
|
||||
PORT=3000
|
||||
|
||||
# CORS
|
||||
CORS_ORIGIN=*
|
||||
|
||||
# Rate Limiting
|
||||
RATE_LIMIT_WINDOW=60000
|
||||
RATE_LIMIT_MAX=100
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```javascript
|
||||
// REST API Client
|
||||
const response = await fetch('https://your-api.com/api/brainy/add', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer YOUR_API_KEY'
|
||||
},
|
||||
body: JSON.stringify({
|
||||
data: 'Your content here',
|
||||
metadata: { type: 'document' }
|
||||
})
|
||||
})
|
||||
|
||||
const { id } = await response.json()
|
||||
|
||||
// Search
|
||||
const searchResponse = await fetch('https://your-api.com/api/brainy/find', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': 'Bearer YOUR_API_KEY'
|
||||
},
|
||||
body: JSON.stringify({
|
||||
query: 'neural networks'
|
||||
})
|
||||
})
|
||||
|
||||
const results = await searchResponse.json()
|
||||
|
||||
// WebSocket Client
|
||||
const ws = new WebSocket('wss://your-api.com/ws')
|
||||
ws.onopen = () => {
|
||||
ws.send(JSON.stringify({
|
||||
type: 'subscribe',
|
||||
pattern: 'technology'
|
||||
}))
|
||||
}
|
||||
ws.onmessage = (event) => {
|
||||
const update = JSON.parse(event.data)
|
||||
console.log('Real-time update:', update)
|
||||
}
|
||||
```
|
||||
|
||||
## Storage Adapter Configuration
|
||||
|
||||
S3CompatibleStorage constructor parameters (verified from source):
|
||||
|
||||
```javascript
|
||||
{
|
||||
endpoint: string, // Required (e.g., 's3.amazonaws.com')
|
||||
bucket: string, // Required
|
||||
accessKeyId: string, // Required
|
||||
secretAccessKey: string, // Required
|
||||
region?: string, // Optional (default: 'us-east-1')
|
||||
prefix?: string, // Optional (e.g., 'brainy/')
|
||||
forcePathStyle?: boolean, // Optional (needed for some S3-compatible services)
|
||||
}
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
|
||||
1. All examples use the **real** Brainy 3.0 APIs
|
||||
2. The `createUniversalHandler()` method is provided by the API Server augmentation
|
||||
3. S3CompatibleStorage works with any S3-compatible service
|
||||
4. Always call `brain.init()` before using Brainy
|
||||
5. The handler can be cached across requests for better performance
|
||||
6. R2Storage is an alias for S3CompatibleStorage (for Cloudflare R2)
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
1. **Always use environment variables for sensitive data** (API keys, secrets)
|
||||
2. **Enable authentication** in the API Server augmentation config
|
||||
3. **Use HTTPS/TLS** for all production deployments
|
||||
4. **Implement rate limiting** to prevent abuse
|
||||
5. **Configure CORS** appropriately for your use case
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
### Enable Lifecycle Policies
|
||||
|
||||
**AWS S3: Automatic tier transitions**
|
||||
```javascript
|
||||
// After initializing brain with S3CompatibleStorage
|
||||
const storage = brain.storage
|
||||
|
||||
// Set lifecycle policy for automatic archival
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'archive-old-data',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' }, // $0.0125/GB after 30 days
|
||||
{ days: 90, storageClass: 'GLACIER' }, // $0.004/GB after 90 days
|
||||
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // $0.00099/GB after 1 year
|
||||
]
|
||||
}]
|
||||
})
|
||||
|
||||
// Or enable Intelligent-Tiering for hands-off optimization
|
||||
await storage.enableIntelligentTiering('entities/', 'auto-optimize')
|
||||
```
|
||||
|
||||
**Cost Impact (500TB dataset):**
|
||||
| Storage Class | Cost/GB/Month | 500TB/Year | Savings |
|
||||
|---------------|---------------|------------|---------|
|
||||
| Standard | $0.023 | $138,000 | Baseline |
|
||||
| Intelligent-Tiering | Variable | $6,900 | **95%** |
|
||||
| With lifecycle policy | Variable | $5,940 | **96%** |
|
||||
|
||||
### Enable Batch Operations
|
||||
|
||||
**Efficient bulk deletions:**
|
||||
```javascript
|
||||
// Batch delete (1000 objects per request)
|
||||
const idsToDelete = [/* array of entity IDs */]
|
||||
const paths = idsToDelete.flatMap(id => [
|
||||
`entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
|
||||
`entities/nouns/metadata/${id.substring(0, 2)}/${id}.json`
|
||||
])
|
||||
|
||||
await storage.batchDelete(paths) // Much faster than individual deletes
|
||||
```
|
||||
|
||||
### Enable Compression (FileSystem)
|
||||
|
||||
**For local/server deployments:**
|
||||
```javascript
|
||||
const storage = new FileSystemStorage({
|
||||
path: './data',
|
||||
compression: true // 60-80% space savings
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Cache the brain instance** - Initialize once and reuse across requests
|
||||
2. **Use S3CompatibleStorage** for cloud deployments (better scalability)
|
||||
3. **Enable the cache augmentation** for frequently accessed data
|
||||
4. **Configure appropriate memory limits** for your runtime
|
||||
5. Enable lifecycle policies to reduce storage costs by 96%
|
||||
6. Use batch operations for cleanup tasks
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **"Brainy not initialized"** - Make sure to call `await brain.init()` before use
|
||||
2. **"augmentationRegistry.getAugmentation is not a function"** - The augmentation wasn't loaded properly
|
||||
3. **S3 Access Denied** - Check your IAM permissions and credentials
|
||||
4. **CORS errors** - Configure the CORS settings in the API Server augmentation
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable debug logging by setting:
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage,
|
||||
debug: true,
|
||||
augmentations: [{
|
||||
name: 'api-server',
|
||||
config: {
|
||||
enabled: true,
|
||||
verbose: true
|
||||
}
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
- Documentation: https://github.com/soulcraft/brainy/docs
|
||||
- Issues: https://github.com/soulcraft/brainy/issues
|
||||
- NPM Package: https://www.npmjs.com/package/@soulcraft/brainy
|
||||
|
||||
---
|
||||
|
||||
This guide contains only verified, working code from the actual Brainy 3.0 codebase. No mock implementations, no stub methods, only production-ready code.
|
||||
|
|
@ -1,505 +0,0 @@
|
|||
# AWS Deployment Guide for Brainy
|
||||
|
||||
## Overview
|
||||
Deploy Brainy on AWS with automatic scaling, high availability, and zero-config dynamic adaptation. Brainy automatically adapts to your AWS environment without manual configuration.
|
||||
|
||||
## Quick Start (Zero-Config)
|
||||
|
||||
### Option 1: AWS Lambda (Serverless)
|
||||
|
||||
```bash
|
||||
# Install Brainy
|
||||
npm install @soulcraft/brainy
|
||||
|
||||
# Create handler.js
|
||||
cat > handler.js << 'EOF'
|
||||
const { Brainy } = require('@soulcraft/brainy')
|
||||
|
||||
let brain
|
||||
|
||||
exports.handler = async (event) => {
|
||||
// Brainy auto-detects Lambda environment and configures accordingly
|
||||
if (!brain) {
|
||||
brain = new Brainy() // Zero config - auto-adapts to Lambda
|
||||
await brain.init()
|
||||
}
|
||||
|
||||
const { method, ...params } = JSON.parse(event.body)
|
||||
|
||||
switch(method) {
|
||||
case 'add':
|
||||
const id = await brain.add(params)
|
||||
return { statusCode: 200, body: JSON.stringify({ id }) }
|
||||
case 'find':
|
||||
const results = await brain.find(params)
|
||||
return { statusCode: 200, body: JSON.stringify({ results }) }
|
||||
default:
|
||||
return { statusCode: 400, body: 'Unknown method' }
|
||||
}
|
||||
}
|
||||
EOF
|
||||
|
||||
# Deploy with AWS SAM
|
||||
sam init --runtime nodejs20.x --name brainy-app
|
||||
sam deploy --guided
|
||||
```
|
||||
|
||||
### Option 2: ECS Fargate (Container)
|
||||
|
||||
```bash
|
||||
# Build and push Docker image
|
||||
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $ECR_URI
|
||||
docker build -t brainy .
|
||||
docker tag brainy:latest $ECR_URI/brainy:latest
|
||||
docker push $ECR_URI/brainy:latest
|
||||
|
||||
# Deploy with minimal ECS task definition
|
||||
cat > task-definition.json << 'EOF'
|
||||
{
|
||||
"family": "brainy",
|
||||
"networkMode": "awsvpc",
|
||||
"requiresCompatibilities": ["FARGATE"],
|
||||
"cpu": "256",
|
||||
"memory": "512",
|
||||
"containerDefinitions": [{
|
||||
"name": "brainy",
|
||||
"image": "$ECR_URI/brainy:latest",
|
||||
"environment": [
|
||||
{"name": "NODE_ENV", "value": "production"}
|
||||
],
|
||||
"logConfiguration": {
|
||||
"logDriver": "awslogs",
|
||||
"options": {
|
||||
"awslogs-group": "/ecs/brainy",
|
||||
"awslogs-region": "us-east-1",
|
||||
"awslogs-stream-prefix": "ecs"
|
||||
}
|
||||
}
|
||||
}]
|
||||
}
|
||||
EOF
|
||||
|
||||
# Brainy auto-detects ECS environment and uses S3 for storage
|
||||
aws ecs register-task-definition --cli-input-json file://task-definition.json
|
||||
aws ecs create-service --cluster default --service-name brainy --task-definition brainy --desired-count 2
|
||||
```
|
||||
|
||||
### Option 3: EC2 Auto-Scaling
|
||||
|
||||
```bash
|
||||
# User data script for EC2 instances
|
||||
#!/bin/bash
|
||||
curl -fsSL https://rpm.nodesource.com/setup_20.x | sudo bash -
|
||||
sudo yum install -y nodejs git
|
||||
|
||||
# Clone and setup (or use your deployment method)
|
||||
git clone https://github.com/yourorg/brainy-app.git /app
|
||||
cd /app
|
||||
npm install --production
|
||||
|
||||
# Create systemd service
|
||||
cat > /etc/systemd/system/brainy.service << 'EOF'
|
||||
[Unit]
|
||||
Description=Brainy
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=ec2-user
|
||||
WorkingDirectory=/app
|
||||
ExecStart=/usr/bin/node index.js
|
||||
Restart=on-failure
|
||||
Environment=NODE_ENV=production
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
systemctl start brainy
|
||||
systemctl enable brainy
|
||||
```
|
||||
|
||||
## Zero-Config Storage (Automatic)
|
||||
|
||||
Brainy automatically detects and uses the best available storage:
|
||||
|
||||
```javascript
|
||||
// No configuration needed - Brainy auto-detects:
|
||||
const brain = new Brainy()
|
||||
|
||||
// Auto-detection priority:
|
||||
// 1. S3 (if IAM role has permissions)
|
||||
// 2. EFS (if mounted at /mnt/efs)
|
||||
// 3. EBS volume (if available)
|
||||
// 4. Instance storage (fallback)
|
||||
```
|
||||
|
||||
### Manual S3 Configuration (Optional)
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: {
|
||||
bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket
|
||||
region: process.env.AWS_REGION || 'auto', // 'auto' detects region
|
||||
// IAM role provides credentials automatically
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Scaling Strategies
|
||||
|
||||
### 1. Horizontal Scaling (Recommended)
|
||||
|
||||
```yaml
|
||||
# Auto-scaling policy
|
||||
Resources:
|
||||
AutoScalingTarget:
|
||||
Type: AWS::ApplicationAutoScaling::ScalableTarget
|
||||
Properties:
|
||||
ServiceNamespace: ecs
|
||||
ResourceId: service/default/brainy
|
||||
ScalableDimension: ecs:service:DesiredCount
|
||||
MinCapacity: 2
|
||||
MaxCapacity: 100
|
||||
|
||||
AutoScalingPolicy:
|
||||
Type: AWS::ApplicationAutoScaling::ScalingPolicy
|
||||
Properties:
|
||||
PolicyType: TargetTrackingScaling
|
||||
TargetTrackingScalingPolicyConfiguration:
|
||||
TargetValue: 70.0
|
||||
PredefinedMetricSpecification:
|
||||
PredefinedMetricType: ECSServiceAverageCPUUtilization
|
||||
```
|
||||
|
||||
### 2. Vertical Scaling
|
||||
|
||||
Brainy automatically adapts to available memory:
|
||||
- **256MB**: Minimal mode, optimized caching
|
||||
- **512MB**: Standard mode, balanced performance
|
||||
- **1GB+**: Full mode, maximum performance
|
||||
|
||||
## High Availability Setup
|
||||
|
||||
### Multi-AZ Deployment
|
||||
|
||||
```javascript
|
||||
// Brainy automatically handles multi-AZ with S3
|
||||
const brain = new Brainy({
|
||||
distributed: {
|
||||
enabled: true, // Auto-enables with S3 storage
|
||||
coordinationMethod: 'auto' // Uses S3 for coordination
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Load Balancing
|
||||
|
||||
```bash
|
||||
# Application Load Balancer with health checks
|
||||
aws elbv2 create-load-balancer \
|
||||
--name brainy-alb \
|
||||
--subnets subnet-xxx subnet-yyy \
|
||||
--security-groups sg-xxx
|
||||
|
||||
aws elbv2 create-target-group \
|
||||
--name brainy-targets \
|
||||
--protocol HTTP \
|
||||
--port 3000 \
|
||||
--vpc-id vpc-xxx \
|
||||
--health-check-path /health \
|
||||
--health-check-interval-seconds 30
|
||||
```
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### CloudWatch Integration
|
||||
|
||||
Brainy automatically sends metrics when running on AWS:
|
||||
|
||||
```javascript
|
||||
// Automatic CloudWatch metrics (no config needed)
|
||||
// - Request count
|
||||
// - Response time
|
||||
// - Error rate
|
||||
// - Storage usage
|
||||
// - Memory usage
|
||||
```
|
||||
|
||||
### Custom Metrics
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
monitoring: {
|
||||
enabled: true,
|
||||
customMetrics: {
|
||||
namespace: 'Brainy/Production',
|
||||
dimensions: [
|
||||
{ Name: 'Environment', Value: 'production' },
|
||||
{ Name: 'Service', Value: 'api' }
|
||||
]
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. IAM Role (Recommended)
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:PutObject",
|
||||
"s3:DeleteObject",
|
||||
"s3:ListBucket"
|
||||
],
|
||||
"Resource": [
|
||||
"arn:aws:s3:::brainy-*/*",
|
||||
"arn:aws:s3:::brainy-*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 2. VPC Configuration
|
||||
|
||||
```bash
|
||||
# Private subnets with NAT Gateway
|
||||
aws ec2 create-vpc --cidr-block 10.0.0.0/16
|
||||
aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.1.0/24 --availability-zone us-east-1a
|
||||
aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.2.0/24 --availability-zone us-east-1b
|
||||
```
|
||||
|
||||
### 3. Encryption
|
||||
|
||||
```javascript
|
||||
// Automatic encryption with S3
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: {
|
||||
encryption: 'auto' // Uses S3 SSE-S3 by default
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
### 1. Spot Instances (70% savings)
|
||||
|
||||
```bash
|
||||
aws ec2 request-spot-fleet --spot-fleet-request-config '{
|
||||
"IamFleetRole": "arn:aws:iam::xxx:role/fleet-role",
|
||||
"TargetCapacity": 2,
|
||||
"SpotPrice": "0.05",
|
||||
"LaunchSpecifications": [{
|
||||
"ImageId": "ami-xxx",
|
||||
"InstanceType": "t3.medium",
|
||||
"UserData": "BASE64_ENCODED_STARTUP_SCRIPT"
|
||||
}]
|
||||
}'
|
||||
```
|
||||
|
||||
### 2. S3 Intelligent-Tiering
|
||||
|
||||
```javascript
|
||||
// Brainy automatically uses S3 Intelligent-Tiering
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: {
|
||||
storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Lambda Reserved Concurrency
|
||||
|
||||
```bash
|
||||
aws lambda put-function-concurrency \
|
||||
--function-name brainy-handler \
|
||||
--reserved-concurrent-executions 10
|
||||
```
|
||||
|
||||
## Deployment Automation
|
||||
|
||||
### GitHub Actions CI/CD
|
||||
|
||||
```yaml
|
||||
name: Deploy to AWS
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Configure AWS credentials
|
||||
uses: aws-actions/configure-aws-credentials@v1
|
||||
with:
|
||||
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
aws-region: us-east-1
|
||||
|
||||
- name: Deploy to ECS
|
||||
run: |
|
||||
docker build -t brainy .
|
||||
docker tag brainy:latest $ECR_URI/brainy:latest
|
||||
docker push $ECR_URI/brainy:latest
|
||||
aws ecs update-service --cluster default --service brainy --force-new-deployment
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Storage Auto-Detection Fails**
|
||||
```javascript
|
||||
// Explicitly specify storage
|
||||
const brain = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'my-bucket' } }
|
||||
})
|
||||
```
|
||||
|
||||
2. **Memory Issues**
|
||||
```javascript
|
||||
// Optimize for low memory
|
||||
const brain = new Brainy({
|
||||
cache: { maxSize: 100 }, // Reduce cache size
|
||||
index: { M: 8 } // Reduce HNSW connections
|
||||
})
|
||||
```
|
||||
|
||||
3. **Cold Starts (Lambda)**
|
||||
|
||||
**Progressive Initialization (Zero-Config)**
|
||||
|
||||
Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME)
|
||||
and uses progressive initialization for <200ms cold starts:
|
||||
|
||||
```javascript
|
||||
// Zero-config - Brainy auto-detects Lambda
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
s3Storage: {
|
||||
bucketName: 'my-bucket',
|
||||
region: 'us-east-1',
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
})
|
||||
await brain.init() // Returns in <200ms
|
||||
|
||||
// First write validates bucket (lazy validation)
|
||||
await brain.add('noun', { name: 'test' }) // Validates here
|
||||
```
|
||||
|
||||
**Manual Override (if needed)**
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
s3Storage: {
|
||||
bucketName: 'my-bucket',
|
||||
// Force specific mode
|
||||
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
| Mode | Cold Start | Best For |
|
||||
|------|------------|----------|
|
||||
| `auto` (default) | <200ms in Lambda | Zero-config, auto-detects |
|
||||
| `progressive` | <200ms always | Force fast init everywhere |
|
||||
| `strict` | 100-500ms+ | Local dev, tests, debugging |
|
||||
|
||||
**Warm-up (Alternative)**
|
||||
```javascript
|
||||
exports.warmup = async () => {
|
||||
if (!brain) {
|
||||
brain = new Brainy({ warmup: true })
|
||||
await brain.init()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Readiness Detection**
|
||||
|
||||
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
|
||||
|
||||
```javascript
|
||||
let brain
|
||||
|
||||
exports.handler = async (event) => {
|
||||
if (!brain) {
|
||||
brain = new Brainy({ storage: { type: 's3', ... } })
|
||||
brain.init() // Fire and forget
|
||||
}
|
||||
|
||||
// Wait for initialization to complete
|
||||
await brain.ready
|
||||
|
||||
// Now safe to use brain methods
|
||||
const results = await brain.find({ query: event.queryStringParameters.q })
|
||||
return { statusCode: 200, body: JSON.stringify(results) }
|
||||
}
|
||||
```
|
||||
|
||||
For health checks, use `isFullyInitialized()` to verify all background tasks are complete:
|
||||
|
||||
```javascript
|
||||
exports.healthCheck = async () => {
|
||||
try {
|
||||
await brain.ready
|
||||
return {
|
||||
statusCode: 200,
|
||||
body: JSON.stringify({
|
||||
status: 'ready',
|
||||
fullyInitialized: brain.isFullyInitialized()
|
||||
})
|
||||
}
|
||||
} catch (error) {
|
||||
return {
|
||||
statusCode: 503,
|
||||
body: JSON.stringify({ status: 'initializing' })
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Production Checklist
|
||||
|
||||
- [ ] IAM roles configured with minimal permissions
|
||||
- [ ] VPC with private subnets
|
||||
- [ ] Auto-scaling configured
|
||||
- [ ] CloudWatch alarms set up
|
||||
- [ ] Backup strategy (S3 versioning enabled)
|
||||
- [ ] SSL/TLS certificates configured
|
||||
- [ ] Rate limiting enabled
|
||||
- [ ] Health checks configured
|
||||
- [ ] Monitoring dashboard created
|
||||
- [ ] Cost alerts configured
|
||||
|
||||
## Support
|
||||
|
||||
- Documentation: https://brainy.soulcraft.ai/docs
|
||||
- Issues: https://github.com/soulcraft/brainy/issues
|
||||
- Community: https://discord.gg/brainy
|
||||
|
|
@ -1,627 +0,0 @@
|
|||
# Google Cloud Platform Deployment Guide for Brainy
|
||||
|
||||
## Overview
|
||||
Deploy Brainy on GCP with automatic scaling, global distribution, and zero-config dynamic adaptation. Brainy automatically detects and optimizes for GCP services.
|
||||
|
||||
## Quick Start (Zero-Config)
|
||||
|
||||
### Option 1: Cloud Run (Serverless Containers)
|
||||
|
||||
```bash
|
||||
# Build and deploy with one command
|
||||
gcloud run deploy brainy \
|
||||
--source . \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--allow-unauthenticated
|
||||
|
||||
# Brainy auto-detects Cloud Run and configures:
|
||||
# - Memory-optimized caching
|
||||
# - GCS for storage (if available)
|
||||
# - Cloud SQL for metadata (if available)
|
||||
```
|
||||
|
||||
### Option 2: Cloud Functions (Serverless)
|
||||
|
||||
```javascript
|
||||
// index.js
|
||||
const { Brainy } = require('@soulcraft/brainy')
|
||||
|
||||
let brain
|
||||
|
||||
exports.brainyHandler = async (req, res) => {
|
||||
// Zero-config - auto-adapts to Cloud Functions
|
||||
if (!brain) {
|
||||
brain = new Brainy() // Detects GCP environment automatically
|
||||
await brain.init()
|
||||
}
|
||||
|
||||
const { method, ...params } = req.body
|
||||
|
||||
try {
|
||||
let result
|
||||
switch(method) {
|
||||
case 'add':
|
||||
result = await brain.add(params)
|
||||
break
|
||||
case 'find':
|
||||
result = await brain.find(params)
|
||||
break
|
||||
case 'relate':
|
||||
result = await brain.relate(params)
|
||||
break
|
||||
default:
|
||||
return res.status(400).json({ error: 'Unknown method' })
|
||||
}
|
||||
res.json({ result })
|
||||
} catch (error) {
|
||||
res.status(500).json({ error: error.message })
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Deploy:
|
||||
```bash
|
||||
gcloud functions deploy brainy \
|
||||
--runtime nodejs20 \
|
||||
--trigger-http \
|
||||
--entry-point brainyHandler \
|
||||
--memory 512MB \
|
||||
--timeout 60s
|
||||
```
|
||||
|
||||
### Option 3: Google Kubernetes Engine (GKE)
|
||||
|
||||
```bash
|
||||
# Create autopilot cluster (fully managed, zero-config)
|
||||
gcloud container clusters create-auto brainy-cluster \
|
||||
--region us-central1
|
||||
|
||||
# Deploy using Cloud Build
|
||||
gcloud builds submit --tag gcr.io/$PROJECT_ID/brainy
|
||||
|
||||
# Apply Kubernetes manifest
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brainy
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: brainy
|
||||
spec:
|
||||
containers:
|
||||
- name: brainy
|
||||
image: gcr.io/$PROJECT_ID/brainy
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "250m"
|
||||
env:
|
||||
- name: NODE_ENV
|
||||
value: production
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: brainy-service
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
selector:
|
||||
app: brainy
|
||||
ports:
|
||||
- port: 80
|
||||
targetPort: 3000
|
||||
EOF
|
||||
```
|
||||
|
||||
## Zero-Config Storage (Automatic)
|
||||
|
||||
Brainy automatically detects and uses the best GCP storage:
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy()
|
||||
// Auto-detection priority:
|
||||
// 1. Firestore (if available)
|
||||
// 2. Cloud Storage (GCS)
|
||||
// 3. Cloud SQL
|
||||
// 4. Persistent Disk
|
||||
// 5. Memory (fallback)
|
||||
```
|
||||
|
||||
### Cloud Storage Configuration (Optional)
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3', // GCS is S3-compatible
|
||||
options: {
|
||||
endpoint: 'https://storage.googleapis.com',
|
||||
bucket: process.env.GCS_BUCKET || 'auto', // Auto-creates bucket
|
||||
// Uses Application Default Credentials automatically
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Firestore Integration (Optional)
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'firestore',
|
||||
options: {
|
||||
projectId: process.env.GCP_PROJECT || 'auto',
|
||||
collection: 'brainy-data'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Scaling Strategies
|
||||
|
||||
### 1. Cloud Run Auto-scaling
|
||||
|
||||
```yaml
|
||||
# service.yaml
|
||||
apiVersion: serving.knative.dev/v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: brainy
|
||||
annotations:
|
||||
run.googleapis.com/execution-environment: gen2
|
||||
spec:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
autoscaling.knative.dev/minScale: "1"
|
||||
autoscaling.knative.dev/maxScale: "1000"
|
||||
autoscaling.knative.dev/target: "80"
|
||||
spec:
|
||||
containerConcurrency: 100
|
||||
containers:
|
||||
- image: gcr.io/PROJECT_ID/brainy
|
||||
resources:
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: "2Gi"
|
||||
```
|
||||
|
||||
### 2. GKE Horizontal Pod Autoscaling
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: brainy-hpa
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: brainy
|
||||
minReplicas: 3
|
||||
maxReplicas: 100
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 80
|
||||
```
|
||||
|
||||
## Global Distribution
|
||||
|
||||
### Multi-Region Setup
|
||||
|
||||
```javascript
|
||||
// Brainy automatically handles multi-region with GCS
|
||||
const brain = new Brainy({
|
||||
distributed: {
|
||||
enabled: true,
|
||||
regions: ['us-central1', 'europe-west1', 'asia-northeast1'],
|
||||
replication: 'auto' // Automatic cross-region replication
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Traffic Director Configuration
|
||||
|
||||
```bash
|
||||
# Global load balancing with Traffic Director
|
||||
gcloud compute backend-services create brainy-global \
|
||||
--global \
|
||||
--load-balancing-scheme=INTERNAL_SELF_MANAGED \
|
||||
--protocol=HTTP2
|
||||
|
||||
gcloud compute backend-services add-backend brainy-global \
|
||||
--global \
|
||||
--network-endpoint-group=brainy-neg \
|
||||
--network-endpoint-group-region=us-central1
|
||||
```
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### Cloud Monitoring (Automatic)
|
||||
|
||||
Brainy automatically sends metrics to Cloud Monitoring:
|
||||
|
||||
```javascript
|
||||
// No configuration needed - automatic when running on GCP
|
||||
const brain = new Brainy()
|
||||
|
||||
// Automatic metrics:
|
||||
// - Request latency
|
||||
// - Error rate
|
||||
// - Storage operations
|
||||
// - Cache hit rate
|
||||
// - Memory usage
|
||||
```
|
||||
|
||||
### Custom Metrics
|
||||
|
||||
```javascript
|
||||
const { Monitoring } = require('@google-cloud/monitoring')
|
||||
const monitoring = new Monitoring.MetricServiceClient()
|
||||
|
||||
const brain = new Brainy({
|
||||
onMetric: async (metric) => {
|
||||
// Send custom metrics to Cloud Monitoring
|
||||
await monitoring.createTimeSeries({
|
||||
name: monitoring.projectPath(projectId),
|
||||
timeSeries: [{
|
||||
metric: {
|
||||
type: `custom.googleapis.com/brainy/${metric.name}`,
|
||||
labels: metric.labels
|
||||
},
|
||||
points: [{
|
||||
interval: { endTime: { seconds: Date.now() / 1000 } },
|
||||
value: { doubleValue: metric.value }
|
||||
}]
|
||||
}]
|
||||
})
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Cloud Trace Integration
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
tracing: {
|
||||
enabled: true,
|
||||
sampleRate: 0.1 // Sample 10% of requests
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Workload Identity (GKE)
|
||||
|
||||
```yaml
|
||||
# Enable Workload Identity
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: brainy-sa
|
||||
annotations:
|
||||
iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com
|
||||
```
|
||||
|
||||
### 2. Binary Authorization
|
||||
|
||||
```yaml
|
||||
# Ensure only signed container images
|
||||
apiVersion: binaryauthorization.grafeas.io/v1beta1
|
||||
kind: Policy
|
||||
metadata:
|
||||
name: brainy-policy
|
||||
spec:
|
||||
defaultAdmissionRule:
|
||||
requireAttestationsBy:
|
||||
- projects/PROJECT_ID/attestors/prod-attestor
|
||||
```
|
||||
|
||||
### 3. VPC Service Controls
|
||||
|
||||
```bash
|
||||
# Create VPC Service Perimeter
|
||||
gcloud access-context-manager perimeters create brainy_perimeter \
|
||||
--resources=projects/PROJECT_NUMBER \
|
||||
--restricted-services=storage.googleapis.com \
|
||||
--title="Brainy Security Perimeter"
|
||||
```
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
### 1. Preemptible VMs (80% savings)
|
||||
|
||||
```yaml
|
||||
# GKE node pool with preemptible VMs
|
||||
apiVersion: container.cnrm.cloud.google.com/v1beta1
|
||||
kind: ContainerNodePool
|
||||
metadata:
|
||||
name: brainy-preemptible-pool
|
||||
spec:
|
||||
clusterRef:
|
||||
name: brainy-cluster
|
||||
config:
|
||||
preemptible: true
|
||||
machineType: n2-standard-2
|
||||
autoscaling:
|
||||
minNodeCount: 1
|
||||
maxNodeCount: 10
|
||||
```
|
||||
|
||||
### 2. Cloud CDN for Static Assets
|
||||
|
||||
```bash
|
||||
# Enable Cloud CDN for frequently accessed data
|
||||
gcloud compute backend-buckets create brainy-assets \
|
||||
--gcs-bucket-name=brainy-static
|
||||
|
||||
gcloud compute backend-buckets update brainy-assets \
|
||||
--enable-cdn \
|
||||
--cache-mode=CACHE_ALL_STATIC
|
||||
```
|
||||
|
||||
### 3. Committed Use Discounts
|
||||
|
||||
```bash
|
||||
# Purchase committed use for predictable workloads
|
||||
gcloud compute commitments create brainy-commitment \
|
||||
--plan=TWELVE_MONTH \
|
||||
--resources=vcpu=100,memory=400
|
||||
```
|
||||
|
||||
## Deployment Automation
|
||||
|
||||
### Cloud Build CI/CD
|
||||
|
||||
```yaml
|
||||
# cloudbuild.yaml
|
||||
steps:
|
||||
# Build container
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.']
|
||||
|
||||
# Push to registry
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA']
|
||||
|
||||
# Deploy to Cloud Run
|
||||
- name: 'gcr.io/cloud-builders/gcloud'
|
||||
args:
|
||||
- 'run'
|
||||
- 'deploy'
|
||||
- 'brainy'
|
||||
- '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'
|
||||
- '--region=us-central1'
|
||||
- '--platform=managed'
|
||||
|
||||
# Trigger on push to main
|
||||
trigger:
|
||||
branch:
|
||||
name: main
|
||||
```
|
||||
|
||||
### Terraform Infrastructure
|
||||
|
||||
```hcl
|
||||
# main.tf
|
||||
resource "google_cloud_run_service" "brainy" {
|
||||
name = "brainy"
|
||||
location = "us-central1"
|
||||
|
||||
template {
|
||||
spec {
|
||||
containers {
|
||||
image = "gcr.io/${var.project_id}/brainy"
|
||||
|
||||
resources {
|
||||
limits = {
|
||||
cpu = "2"
|
||||
memory = "2Gi"
|
||||
}
|
||||
}
|
||||
|
||||
env {
|
||||
name = "NODE_ENV"
|
||||
value = "production"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
traffic {
|
||||
percent = 100
|
||||
latest_revision = true
|
||||
}
|
||||
}
|
||||
|
||||
resource "google_cloud_run_service_iam_member" "public" {
|
||||
service = google_cloud_run_service.brainy.name
|
||||
location = google_cloud_run_service.brainy.location
|
||||
role = "roles/run.invoker"
|
||||
member = "allUsers"
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### 1. Memory Store (Redis Compatible)
|
||||
|
||||
```javascript
|
||||
// Brainy can use Memorystore for caching
|
||||
const brain = new Brainy({
|
||||
cache: {
|
||||
type: 'redis',
|
||||
options: {
|
||||
host: process.env.REDIS_HOST || 'auto-detect',
|
||||
port: 6379
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### 2. Cloud Spanner for Global Consistency
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
metadata: {
|
||||
type: 'spanner',
|
||||
options: {
|
||||
instance: 'brainy-instance',
|
||||
database: 'brainy-db'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Quota Exceeded**
|
||||
```bash
|
||||
# Check quotas
|
||||
gcloud compute project-info describe --project=$PROJECT_ID
|
||||
|
||||
# Request increase
|
||||
gcloud compute project-info add-metadata \
|
||||
--metadata google-compute-default-region=us-central1
|
||||
```
|
||||
|
||||
2. **Cold Starts**
|
||||
|
||||
**Progressive Initialization (Zero-Config)**
|
||||
|
||||
Brainy automatically detects Cloud Run and Cloud Functions environments
|
||||
and uses progressive initialization for <200ms cold starts:
|
||||
|
||||
```javascript
|
||||
// Zero-config - Brainy auto-detects Cloud Run (K_SERVICE env var)
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
gcsNativeStorage: { bucketName: 'my-bucket' }
|
||||
}
|
||||
})
|
||||
await brain.init() // Returns in <200ms
|
||||
|
||||
// First write validates bucket (lazy validation)
|
||||
await brain.add('noun', { name: 'test' }) // Validates here
|
||||
```
|
||||
|
||||
**Manual Override (if needed)**
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
gcsNativeStorage: {
|
||||
bucketName: 'my-bucket',
|
||||
// Force specific mode
|
||||
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
| Mode | Cold Start | Best For |
|
||||
|------|------------|----------|
|
||||
| `auto` (default) | <200ms in cloud | Zero-config, auto-detects |
|
||||
| `progressive` | <200ms always | Force fast init everywhere |
|
||||
| `strict` | 100-500ms+ | Local dev, tests, debugging |
|
||||
|
||||
**Keep Warm (Alternative)**
|
||||
```javascript
|
||||
// Keep minimum instances warm
|
||||
const brain = new Brainy({
|
||||
warmup: {
|
||||
enabled: true,
|
||||
interval: 60000 // Ping every minute
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Readiness Detection**
|
||||
|
||||
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
|
||||
|
||||
```javascript
|
||||
let brain
|
||||
|
||||
async function handleRequest(req, res) {
|
||||
if (!brain) {
|
||||
brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||
brain.init() // Fire and forget
|
||||
}
|
||||
|
||||
// Wait for initialization to complete
|
||||
await brain.ready
|
||||
|
||||
// Now safe to use brain methods
|
||||
const results = await brain.find({ query: req.query.q })
|
||||
res.json(results)
|
||||
}
|
||||
```
|
||||
|
||||
For Cloud Run health checks, use `isFullyInitialized()` to verify all background tasks are complete:
|
||||
|
||||
```javascript
|
||||
// Health check endpoint for Cloud Run
|
||||
app.get('/health', async (req, res) => {
|
||||
try {
|
||||
await brain.ready
|
||||
res.json({
|
||||
status: 'ready',
|
||||
fullyInitialized: brain.isFullyInitialized()
|
||||
})
|
||||
} catch (error) {
|
||||
res.status(503).json({ status: 'initializing' })
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
3. **Memory Pressure**
|
||||
```javascript
|
||||
// Optimize for GCP memory constraints
|
||||
const brain = new Brainy({
|
||||
memory: {
|
||||
mode: 'aggressive', // Aggressive garbage collection
|
||||
maxHeap: 0.8 // Use 80% of available memory
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Production Checklist
|
||||
|
||||
- [ ] Enable Workload Identity for secure access
|
||||
- [ ] Configure Cloud Armor for DDoS protection
|
||||
- [ ] Set up Cloud KMS for encryption keys
|
||||
- [ ] Enable VPC Service Controls
|
||||
- [ ] Configure Cloud IAP for authentication
|
||||
- [ ] Set up Cloud Monitoring dashboards
|
||||
- [ ] Configure Error Reporting
|
||||
- [ ] Enable Cloud Trace
|
||||
- [ ] Set up budget alerts
|
||||
- [ ] Configure backup and disaster recovery
|
||||
|
||||
## Support
|
||||
|
||||
- Documentation: https://brainy.soulcraft.ai/docs
|
||||
- Issues: https://github.com/soulcraft/brainy/issues
|
||||
- GCP Marketplace: https://console.cloud.google.com/marketplace/product/brainy
|
||||
|
|
@ -1,727 +0,0 @@
|
|||
# Kubernetes Deployment Guide for Brainy
|
||||
|
||||
## Overview
|
||||
Deploy Brainy on Kubernetes with automatic scaling, high availability, and zero-config dynamic adaptation. Works with any Kubernetes distribution (vanilla, EKS, GKE, AKS, OpenShift, etc.).
|
||||
|
||||
## Quick Start (Zero-Config)
|
||||
|
||||
### Basic Deployment
|
||||
|
||||
```yaml
|
||||
# brainy-deployment.yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brainy
|
||||
labels:
|
||||
app: brainy
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: brainy
|
||||
spec:
|
||||
containers:
|
||||
- name: brainy
|
||||
image: soulcraft/brainy:latest
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
env:
|
||||
- name: NODE_ENV
|
||||
value: production
|
||||
# Brainy auto-detects Kubernetes and configures accordingly
|
||||
resources:
|
||||
requests:
|
||||
memory: "256Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "500m"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: brainy-service
|
||||
spec:
|
||||
selector:
|
||||
app: brainy
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 80
|
||||
targetPort: 3000
|
||||
type: LoadBalancer
|
||||
```
|
||||
|
||||
Deploy:
|
||||
```bash
|
||||
kubectl apply -f brainy-deployment.yaml
|
||||
```
|
||||
|
||||
## Production-Grade Setup
|
||||
|
||||
### 1. StatefulSet with Persistent Storage
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: brainy-storage
|
||||
provisioner: kubernetes.io/aws-ebs # Or your cloud provider
|
||||
parameters:
|
||||
type: gp3
|
||||
iopsPerGB: "10"
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: brainy
|
||||
spec:
|
||||
serviceName: brainy-headless
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: brainy
|
||||
spec:
|
||||
containers:
|
||||
- name: brainy
|
||||
image: soulcraft/brainy:latest
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
env:
|
||||
- name: NODE_ENV
|
||||
value: production
|
||||
- name: BRAINY_STORAGE_TYPE
|
||||
value: filesystem
|
||||
- name: BRAINY_STORAGE_PATH
|
||||
value: /data
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 3000
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: 3000
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: brainy-storage
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
```
|
||||
|
||||
### 2. Horizontal Pod Autoscaler
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: brainy-hpa
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: brainy
|
||||
minReplicas: 2
|
||||
maxReplicas: 100
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 70
|
||||
- type: Resource
|
||||
resource:
|
||||
name: memory
|
||||
target:
|
||||
type: Utilization
|
||||
averageUtilization: 80
|
||||
behavior:
|
||||
scaleUp:
|
||||
stabilizationWindowSeconds: 60
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 100
|
||||
periodSeconds: 60
|
||||
scaleDown:
|
||||
stabilizationWindowSeconds: 300
|
||||
policies:
|
||||
- type: Percent
|
||||
value: 10
|
||||
periodSeconds: 60
|
||||
```
|
||||
|
||||
### 3. Ingress Configuration
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: brainy-ingress
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: nginx
|
||||
cert-manager.io/cluster-issuer: letsencrypt-prod
|
||||
nginx.ingress.kubernetes.io/rate-limit: "100"
|
||||
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
|
||||
spec:
|
||||
tls:
|
||||
- hosts:
|
||||
- api.brainy.example.com
|
||||
secretName: brainy-tls
|
||||
rules:
|
||||
- host: api.brainy.example.com
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: brainy-service
|
||||
port:
|
||||
number: 80
|
||||
```
|
||||
|
||||
## Zero-Config Storage Options
|
||||
|
||||
### Option 1: S3-Compatible Storage (Recommended)
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: brainy-s3-credentials
|
||||
type: Opaque
|
||||
data:
|
||||
access-key: <base64-encoded-key>
|
||||
secret-key: <base64-encoded-secret>
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brainy
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: brainy
|
||||
env:
|
||||
- name: BRAINY_STORAGE_TYPE
|
||||
value: s3
|
||||
- name: AWS_ACCESS_KEY_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: brainy-s3-credentials
|
||||
key: access-key
|
||||
- name: AWS_SECRET_ACCESS_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: brainy-s3-credentials
|
||||
key: secret-key
|
||||
- name: S3_BUCKET
|
||||
value: brainy-data
|
||||
- name: AWS_REGION
|
||||
value: us-east-1
|
||||
```
|
||||
|
||||
### Option 2: MinIO (Self-Hosted S3)
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: minio
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: minio
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: minio
|
||||
spec:
|
||||
containers:
|
||||
- name: minio
|
||||
image: minio/minio:latest
|
||||
args:
|
||||
- server
|
||||
- /data
|
||||
env:
|
||||
- name: MINIO_ROOT_USER
|
||||
value: brainy
|
||||
- name: MINIO_ROOT_PASSWORD
|
||||
value: brainy123456
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: minio-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: minio-service
|
||||
spec:
|
||||
selector:
|
||||
app: minio
|
||||
ports:
|
||||
- port: 9000
|
||||
targetPort: 9000
|
||||
```
|
||||
|
||||
### Option 3: Shared NFS Storage
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: brainy-nfs-pv
|
||||
spec:
|
||||
capacity:
|
||||
storage: 100Gi
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
nfs:
|
||||
server: nfs-server.example.com
|
||||
path: /export/brainy
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: brainy-nfs-pvc
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
resources:
|
||||
requests:
|
||||
storage: 100Gi
|
||||
```
|
||||
|
||||
## High Availability Configuration
|
||||
|
||||
### 1. Pod Anti-Affinity
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brainy
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
- labelSelector:
|
||||
matchExpressions:
|
||||
- key: app
|
||||
operator: In
|
||||
values:
|
||||
- brainy
|
||||
topologyKey: kubernetes.io/hostname
|
||||
```
|
||||
|
||||
### 2. Pod Disruption Budget
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: brainy-pdb
|
||||
spec:
|
||||
minAvailable: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
```
|
||||
|
||||
### 3. Multi-Zone Deployment
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: brainy
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: topology.kubernetes.io/zone
|
||||
whenUnsatisfiable: DoNotSchedule
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
```
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
### 1. Prometheus Metrics
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: brainy-metrics
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9090"
|
||||
prometheus.io/path: "/metrics"
|
||||
spec:
|
||||
selector:
|
||||
app: brainy
|
||||
ports:
|
||||
- name: metrics
|
||||
port: 9090
|
||||
targetPort: 9090
|
||||
```
|
||||
|
||||
### 2. Grafana Dashboard
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: brainy-dashboard
|
||||
data:
|
||||
dashboard.json: |
|
||||
{
|
||||
"dashboard": {
|
||||
"title": "Brainy Metrics",
|
||||
"panels": [
|
||||
{
|
||||
"title": "Request Rate",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(brainy_requests_total[5m])"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Response Time",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, brainy_response_time)"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Logging with Fluentd
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: fluentd-config
|
||||
data:
|
||||
fluent.conf: |
|
||||
<source>
|
||||
@type tail
|
||||
path /var/log/containers/brainy*.log
|
||||
pos_file /var/log/fluentd-brainy.log.pos
|
||||
tag brainy.*
|
||||
<parse>
|
||||
@type json
|
||||
</parse>
|
||||
</source>
|
||||
|
||||
<match brainy.**>
|
||||
@type elasticsearch
|
||||
host elasticsearch.logging.svc.cluster.local
|
||||
port 9200
|
||||
logstash_format true
|
||||
logstash_prefix brainy
|
||||
</match>
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Network Policies
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: brainy-netpol
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: brainy
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
name: ingress-nginx
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3000
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector: {}
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443 # HTTPS
|
||||
- protocol: TCP
|
||||
port: 9000 # MinIO/S3
|
||||
```
|
||||
|
||||
### 2. Pod Security Policy
|
||||
|
||||
```yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: brainy-psp
|
||||
spec:
|
||||
privileged: false
|
||||
allowPrivilegeEscalation: false
|
||||
requiredDropCapabilities:
|
||||
- ALL
|
||||
volumes:
|
||||
- 'configMap'
|
||||
- 'emptyDir'
|
||||
- 'projected'
|
||||
- 'secret'
|
||||
- 'persistentVolumeClaim'
|
||||
runAsUser:
|
||||
rule: 'MustRunAsNonRoot'
|
||||
seLinux:
|
||||
rule: 'RunAsAny'
|
||||
fsGroup:
|
||||
rule: 'RunAsAny'
|
||||
```
|
||||
|
||||
### 3. RBAC Configuration
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: brainy-sa
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: brainy-role
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["configmaps"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["get"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: brainy-rolebinding
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: brainy-sa
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: brainy-role
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
## GitOps with ArgoCD
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: brainy
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: default
|
||||
source:
|
||||
repoURL: https://github.com/yourorg/brainy-k8s
|
||||
targetRevision: HEAD
|
||||
path: manifests
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: brainy
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
```
|
||||
|
||||
## Helm Chart Installation
|
||||
|
||||
```bash
|
||||
# Add Brainy Helm repository
|
||||
helm repo add brainy https://charts.brainy.io
|
||||
helm repo update
|
||||
|
||||
# Install with custom values
|
||||
cat > values.yaml << EOF
|
||||
replicaCount: 3
|
||||
image:
|
||||
repository: soulcraft/brainy
|
||||
tag: latest
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
service:
|
||||
type: LoadBalancer
|
||||
port: 80
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
hosts:
|
||||
- host: api.brainy.example.com
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 1Gi
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
|
||||
autoscaling:
|
||||
enabled: true
|
||||
minReplicas: 2
|
||||
maxReplicas: 100
|
||||
targetCPUUtilizationPercentage: 70
|
||||
|
||||
storage:
|
||||
type: s3
|
||||
s3:
|
||||
bucket: brainy-data
|
||||
region: us-east-1
|
||||
EOF
|
||||
|
||||
helm install brainy brainy/brainy -f values.yaml
|
||||
```
|
||||
|
||||
## Cost Optimization
|
||||
|
||||
### 1. Spot/Preemptible Nodes
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: NodePool
|
||||
metadata:
|
||||
name: brainy-spot-pool
|
||||
spec:
|
||||
nodeSelector:
|
||||
node.kubernetes.io/lifecycle: spot
|
||||
taints:
|
||||
- key: spot
|
||||
value: "true"
|
||||
effect: NoSchedule
|
||||
tolerations:
|
||||
- key: spot
|
||||
operator: Equal
|
||||
value: "true"
|
||||
effect: NoSchedule
|
||||
```
|
||||
|
||||
### 2. Vertical Pod Autoscaler
|
||||
|
||||
```yaml
|
||||
apiVersion: autoscaling.k8s.io/v1
|
||||
kind: VerticalPodAutoscaler
|
||||
metadata:
|
||||
name: brainy-vpa
|
||||
spec:
|
||||
targetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: brainy
|
||||
updatePolicy:
|
||||
updateMode: "Auto"
|
||||
resourcePolicy:
|
||||
containerPolicies:
|
||||
- containerName: brainy
|
||||
minAllowed:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
maxAllowed:
|
||||
cpu: 2
|
||||
memory: 2Gi
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Pod CrashLoopBackOff**
|
||||
```bash
|
||||
kubectl logs -f pod/brainy-xxx
|
||||
kubectl describe pod brainy-xxx
|
||||
```
|
||||
|
||||
2. **Storage Issues**
|
||||
```bash
|
||||
kubectl get pv,pvc
|
||||
kubectl describe pvc brainy-data
|
||||
```
|
||||
|
||||
3. **Network Connectivity**
|
||||
```bash
|
||||
kubectl exec -it pod/brainy-xxx -- curl http://brainy-service/health
|
||||
kubectl get endpoints brainy-service
|
||||
```
|
||||
|
||||
4. **Memory Pressure**
|
||||
```bash
|
||||
kubectl top pods -l app=brainy
|
||||
kubectl describe node
|
||||
```
|
||||
|
||||
## Production Checklist
|
||||
|
||||
- [ ] High availability with multiple replicas
|
||||
- [ ] Pod disruption budgets configured
|
||||
- [ ] Resource limits and requests set
|
||||
- [ ] Horizontal and vertical autoscaling enabled
|
||||
- [ ] Persistent storage configured
|
||||
- [ ] Network policies in place
|
||||
- [ ] RBAC properly configured
|
||||
- [ ] Monitoring and alerting setup
|
||||
- [ ] Backup and disaster recovery plan
|
||||
- [ ] Security scanning enabled
|
||||
- [ ] GitOps deployment pipeline
|
||||
|
||||
## Support
|
||||
|
||||
- Documentation: https://brainy.soulcraft.ai/docs
|
||||
- Helm Charts: https://github.com/soulcraft/brainy-charts
|
||||
- Issues: https://github.com/soulcraft/brainy/issues
|
||||
- Slack: https://brainy-community.slack.com
|
||||
|
|
@ -5,197 +5,144 @@ public: true
|
|||
category: guides
|
||||
template: guide
|
||||
order: 2
|
||||
description: "Six adapters for every deployment: in-memory, OPFS (browser), filesystem, S3, Google Cloud Storage, Azure Blob, and Cloudflare R2. All support copy-on-write branching."
|
||||
description: "Two adapters cover every deployment: in-memory for tests + ephemeral workloads, filesystem for everything that needs to persist. Both support copy-on-write branching. Cloud backup is operator tooling, not a built-in adapter."
|
||||
next:
|
||||
- guides/plugins
|
||||
- concepts/zero-config
|
||||
---
|
||||
|
||||
# Storage Adapters Guide
|
||||
# Storage Adapters
|
||||
|
||||
## Overview
|
||||
Brainy supports 6 storage adapters for different deployment environments. All adapters implement the same StorageAdapter interface and support copy-on-write branching.
|
||||
Brainy 8.0 ships **two storage adapters**:
|
||||
|
||||
## Adapters
|
||||
- **`FileSystemStorage`** — persistent on-disk storage. The default for any
|
||||
deployment that needs to survive a restart. Runs on Node.js, Bun, and Deno.
|
||||
- **`MemoryStorage`** — in-memory only. The right choice for tests, ephemeral
|
||||
workloads, and short-lived demos.
|
||||
|
||||
### 1. MemoryStorage
|
||||
- **File**: `src/storage/adapters/memoryStorage.ts`
|
||||
- **Use case**: Development, testing, prototyping
|
||||
- **Configuration**: None required
|
||||
- **Persistence**: None (data lost on restart)
|
||||
- **Batch config**: 1000 batch size, 0ms delay, 1000 concurrent ops, 100k ops/sec
|
||||
Both implement the same `StorageAdapter` interface, support copy-on-write
|
||||
branching, and use the same on-disk layout (memory's "disk" is a JS Map).
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: { type: 'memory' } })
|
||||
## Quick start
|
||||
|
||||
```ts
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// Filesystem (recommended for any persistent workload):
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: './brainy-data' }
|
||||
})
|
||||
|
||||
// Memory (tests, ephemeral):
|
||||
const brainMem = new Brainy({ storage: { type: 'memory' } })
|
||||
|
||||
// Auto-detect (filesystem on Node-like runtimes, memory in browsers):
|
||||
const brainAuto = new Brainy({ storage: { type: 'auto' } })
|
||||
```
|
||||
|
||||
### 2. FileSystemStorage
|
||||
- **File**: `src/storage/adapters/fileSystemStorage.ts`
|
||||
- **Use case**: Node.js local persistence, single-server deployments
|
||||
- **Configuration**: `basePath` (required), `readOnly` (optional)
|
||||
- **Features**: zlib compression, atomic writes via temp files, UUID-based sharding
|
||||
## When to use which
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './brainy-data' }
|
||||
})
|
||||
| Use case | Adapter | Why |
|
||||
|---|---|---|
|
||||
| Production app | `filesystem` | Durable, branchable, mmap-able |
|
||||
| Tests, CI | `memory` | No disk teardown; fast |
|
||||
| Short-lived data pipeline | `memory` | No persistence needed |
|
||||
| In-browser demo | `memory` | Filesystem unavailable in browsers |
|
||||
| Cloud deployment | `filesystem` on local disk + operator backup | See "Cloud backup" below |
|
||||
|
||||
## Cloud backup — operator tooling, not a built-in
|
||||
|
||||
Brainy 8.0 deliberately ships **no cloud storage adapters**. Cloud backup is
|
||||
handled at the operator layer with standard tooling, the same pattern every
|
||||
production database uses (Postgres, SQLite, Redis):
|
||||
|
||||
```bash
|
||||
# After a brainy flush, sync the on-disk artefact to your cloud of choice.
|
||||
gsutil rsync -r /var/lib/brainy gs://my-backup-bucket/brainy/
|
||||
# or:
|
||||
aws s3 sync /var/lib/brainy s3://my-backup-bucket/brainy/
|
||||
# or:
|
||||
rclone sync /var/lib/brainy remote:brainy-backups/
|
||||
# or:
|
||||
azcopy sync /var/lib/brainy "https://account.blob.core.windows.net/brainy?sv=..."
|
||||
```
|
||||
|
||||
### 3. S3CompatibleStorage
|
||||
- **File**: `src/storage/adapters/s3CompatibleStorage.ts`
|
||||
- **Use case**: AWS S3, MinIO, DigitalOcean Spaces, any S3-compatible service
|
||||
- **Configuration**: `bucketName`, `region`, `accessKeyId`, `secretAccessKey`, `endpoint?` (for custom endpoints), `s3ForcePathStyle?`
|
||||
- **Features**: Write buffering, request coalescing, throttling detection (429/503), progressive initialization
|
||||
- **Batch config**: 1000 batch size, 150 concurrent ops, 5000 ops/sec burst
|
||||
Brainy's filesystem layout is sync-friendly:
|
||||
- Atomic writes (temp + rename) — readers never see torn files
|
||||
- Per-shard files — `rsync`-style incremental sync works well
|
||||
- Content-addressed blobs (`_blobs/`) — immutable, cache-friendly
|
||||
- Branch refs live under `_cow/` — pick up branches automatically
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
s3Storage: {
|
||||
bucketName: 'my-brainy-data',
|
||||
region: 'us-east-1',
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
})
|
||||
For point-in-time backups, take a filesystem snapshot (ZFS, btrfs, LVM, EBS,
|
||||
etc.) or use `brain.persist(path)` to write a self-contained snapshot you
|
||||
can sync independently of the live brain.
|
||||
|
||||
// For MinIO or other S3-compatible services:
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
s3Storage: {
|
||||
bucketName: 'my-data',
|
||||
region: 'us-east-1',
|
||||
endpoint: 'http://localhost:9000',
|
||||
s3ForcePathStyle: true,
|
||||
accessKeyId: 'minio-key',
|
||||
secretAccessKey: 'minio-secret'
|
||||
}
|
||||
}
|
||||
})
|
||||
## Why no cloud adapters in 8.0?
|
||||
|
||||
Cloud storage adapters lived in Brainy 4.x-7.x. They were dropped in 8.0
|
||||
per **BR-BRAINY-80-STORAGE-SIMPLIFY** because:
|
||||
|
||||
- Zero production consumers used them at scale. Every Soulcraft consumer
|
||||
ran on local filesystem.
|
||||
- Cloud-storage HNSW / DiskANN doesn't perform — vector indexes need
|
||||
low-latency random reads that S3 / GCS / R2 / Azure can't provide
|
||||
consistently.
|
||||
- Bundling cloud SDKs into the library cost ~3000-5000 LOC + 4-7 transitive
|
||||
dependencies for a feature nobody used.
|
||||
- Cloud backup via operator tooling is strictly more reliable than in-app
|
||||
upload (better retry semantics, better observability, better cost control).
|
||||
|
||||
Brainy 8.0 is smaller, faster to install, and clearer about what it does.
|
||||
|
||||
## Configuration
|
||||
|
||||
```ts
|
||||
interface StorageOptions {
|
||||
// The adapter type. Defaults to 'auto'.
|
||||
type?: 'auto' | 'memory' | 'filesystem'
|
||||
|
||||
// Force a specific adapter regardless of type.
|
||||
forceMemoryStorage?: boolean
|
||||
forceFileSystemStorage?: boolean
|
||||
|
||||
// Filesystem only.
|
||||
rootDirectory?: string
|
||||
|
||||
// COW branch to open. Defaults to 'main'.
|
||||
branch?: string
|
||||
|
||||
// COW compression toggle. Defaults to true.
|
||||
enableCompression?: boolean
|
||||
}
|
||||
```
|
||||
|
||||
### 4. R2Storage (Cloudflare)
|
||||
- **File**: `src/storage/adapters/r2Storage.ts`
|
||||
- **Use case**: Zero egress fees, cost-effective cloud storage
|
||||
- **Configuration**: `bucketName`, `accountId`, `accessKeyId`, `secretAccessKey`, `cacheConfig?`
|
||||
- **Features**: Zero egress fees, aggressive caching, write buffering, request coalescing
|
||||
- **Batch config**: 1000 batch size, 150 concurrent ops, 6000 ops/sec
|
||||
## Direct construction
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'r2',
|
||||
r2Storage: {
|
||||
accountId: process.env.CF_ACCOUNT_ID,
|
||||
bucketName: 'my-brainy-data',
|
||||
accessKeyId: process.env.CF_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.CF_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
})
|
||||
If you want to skip the factory:
|
||||
|
||||
```ts
|
||||
import { FileSystemStorage, MemoryStorage } from '@soulcraft/brainy'
|
||||
|
||||
const fsStorage = new FileSystemStorage('./brainy-data')
|
||||
const memStorage = new MemoryStorage()
|
||||
|
||||
const brain = new Brainy({ storage: fsStorage })
|
||||
```
|
||||
|
||||
### 5. GcsStorage (Google Cloud)
|
||||
- **File**: `src/storage/adapters/gcsStorage.ts`
|
||||
- **Use case**: Google Cloud ecosystem, Cloud Run deployments
|
||||
- **Authentication priority**:
|
||||
1. Service Account Key File (`keyFilename`)
|
||||
2. Credentials Object (`credentials`)
|
||||
3. HMAC Keys (backward compat)
|
||||
4. Application Default Credentials (automatic in Cloud Run)
|
||||
- **Features**: Progressive initialization (fast cold starts <200ms), Autoclass lifecycle, bucket validation
|
||||
- **Batch config**: 1000 batch size, 100 concurrent ops, 1000 ops/sec
|
||||
## Migration from 7.x cloud adapters
|
||||
|
||||
```typescript
|
||||
// With explicit credentials
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
gcsStorage: {
|
||||
bucketName: 'my-brainy-data',
|
||||
keyFilename: './service-account.json'
|
||||
}
|
||||
}
|
||||
})
|
||||
7.x consumers of `OPFSStorage`, `GcsStorage`, `R2Storage`, `S3CompatibleStorage`,
|
||||
or `AzureBlobStorage` need to migrate to `FileSystemStorage` plus operator
|
||||
backup tooling. The recipe:
|
||||
|
||||
// In Cloud Run (uses Application Default Credentials automatically)
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
gcsStorage: {
|
||||
bucketName: 'my-brainy-data'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
1. On the host running Brainy, mount a local disk (NVMe recommended). Cloud
|
||||
providers all expose persistent local disks: GCP Persistent Disk, AWS EBS,
|
||||
Azure Managed Disks.
|
||||
2. Set `storage: { type: 'filesystem', rootDirectory: '/mnt/brainy-data' }`.
|
||||
3. Run your existing data import once into the new local store.
|
||||
4. Set up an operator backup job using `gsutil` / `aws s3` / `rclone` /
|
||||
`azcopy` on a cron — hourly or whatever your RPO requires. Point it at
|
||||
the brainy data dir.
|
||||
5. For point-in-time backups, use filesystem snapshots or `brain.persist()`.
|
||||
|
||||
**Progressive Initialization Modes:**
|
||||
- `'strict'` (default locally): Full validation during init (100-500ms+)
|
||||
- `'progressive'` (default in Cloud Run/Lambda): Fast init <200ms, background validation
|
||||
- `'auto'`: Auto-detects environment
|
||||
|
||||
### 6. AzureBlobStorage
|
||||
- **File**: `src/storage/adapters/azureBlobStorage.ts`
|
||||
- **Use case**: Azure ecosystem, enterprise deployments
|
||||
- **Authentication priority**:
|
||||
1. DefaultAzureCredential (Managed Identity - automatic in Azure)
|
||||
2. Connection String
|
||||
3. Account Name + Account Key
|
||||
4. SAS Token
|
||||
- **Features**: Progressive initialization, lifecycle management, container validation
|
||||
- **Batch config**: 1000 batch size, 100 concurrent ops, 3000 ops/sec
|
||||
|
||||
```typescript
|
||||
// With connection string
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'azure',
|
||||
azureStorage: {
|
||||
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
|
||||
containerName: 'brainy-data'
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// With Managed Identity (automatic in Azure App Service/Functions)
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'azure',
|
||||
azureStorage: {
|
||||
containerName: 'brainy-data'
|
||||
// DefaultAzureCredential used automatically
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Choosing an Adapter
|
||||
|
||||
| Scenario | Recommended Adapter |
|
||||
|----------|-------------------|
|
||||
| Development/Testing | MemoryStorage |
|
||||
| Local persistence | FileSystemStorage |
|
||||
| AWS deployment | S3CompatibleStorage |
|
||||
| Google Cloud / Cloud Run | GcsStorage |
|
||||
| Azure deployment | AzureBlobStorage |
|
||||
| Cost-sensitive (high egress) | R2Storage |
|
||||
| Self-hosted (MinIO) | S3CompatibleStorage |
|
||||
|
||||
## Common Configuration
|
||||
|
||||
All cloud adapters support:
|
||||
- **Progressive initialization** for fast serverless cold starts
|
||||
- **Read-only mode** (`readOnly: true`)
|
||||
- **Cache configuration** (`cacheConfig: { hotCacheMaxSize, warmCacheTTL }`)
|
||||
- **Copy-on-write branching** for git-style data management
|
||||
|
||||
## See Also
|
||||
|
||||
- [Cloud Deployment Guide](../deployment/CLOUD_DEPLOYMENT_GUIDE.md)
|
||||
- [Cost Optimization: AWS S3](../operations/cost-optimization-aws-s3.md)
|
||||
- [Cost Optimization: GCS](../operations/cost-optimization-gcs.md)
|
||||
- [Cost Optimization: Azure](../operations/cost-optimization-azure.md)
|
||||
- [Cost Optimization: R2](../operations/cost-optimization-cloudflare-r2.md)
|
||||
Same data, same APIs, no library-side cloud code.
|
||||
|
|
|
|||
|
|
@ -1,138 +0,0 @@
|
|||
# Cloud Run + Filestore (NFS) Deployment Guide
|
||||
|
||||
> Eliminate cloud storage rate limiting by using brainy's FileSystem adapter with a Filestore NFS mount on Cloud Run.
|
||||
|
||||
## Why Filestore?
|
||||
|
||||
Brainy's metadata index generates 26-40 file writes per `brain.add()` call. On GCS, each write is a cloud API call subject to rate limiting (~1 write/sec per object), causing HTTP 429 errors and 61-second latency for a single add operation.
|
||||
|
||||
With Filestore:
|
||||
- **Local disk speed**: Each write is ~1ms (vs 50-300ms for cloud API)
|
||||
- **No rate limits**: NFS has no per-object write throttling
|
||||
- **Zero code changes**: Use brainy's existing `FileSystemStorage` adapter
|
||||
- **Same Cloud Run deployment**: Just add a volume mount
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Google Cloud project with Filestore API enabled
|
||||
- Cloud Run service (gen2 execution environment required for NFS)
|
||||
- VPC connector or Direct VPC egress configured
|
||||
|
||||
## Step 1: Create a Filestore Instance
|
||||
|
||||
```bash
|
||||
gcloud filestore instances create brainy-store \
|
||||
--zone=us-central1-b \
|
||||
--tier=BASIC_SSD \
|
||||
--file-share=name=brainy_data,capacity=1TB \
|
||||
--network=name=default
|
||||
```
|
||||
|
||||
**Tier recommendations:**
|
||||
- **BASIC_SSD** (~$370/month for 1 TiB): Good balance of performance and cost
|
||||
- **BASIC_HDD** (~$204/month for 1 TiB): Lower cost, sufficient for moderate workloads
|
||||
- **ENTERPRISE** or **ZONAL**: Higher IOPS for heavy workloads
|
||||
|
||||
Note the IP address from the output — you'll need it for the Cloud Run mount.
|
||||
|
||||
## Step 2: Configure Cloud Run NFS Volume Mount
|
||||
|
||||
```yaml
|
||||
# service.yaml
|
||||
apiVersion: serving.knative.dev/v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: my-brainy-service
|
||||
spec:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
run.googleapis.com/execution-environment: gen2
|
||||
run.googleapis.com/vpc-access-connector: projects/PROJECT/locations/REGION/connectors/CONNECTOR
|
||||
spec:
|
||||
containers:
|
||||
- image: gcr.io/PROJECT/my-brainy-app
|
||||
volumeMounts:
|
||||
- name: brainy-nfs
|
||||
mountPath: /mnt/brainy
|
||||
volumes:
|
||||
- name: brainy-nfs
|
||||
nfs:
|
||||
server: FILESTORE_IP # e.g., 10.0.0.2
|
||||
path: /brainy_data
|
||||
```
|
||||
|
||||
Deploy:
|
||||
|
||||
```bash
|
||||
gcloud run services replace service.yaml --region=us-central1
|
||||
```
|
||||
|
||||
## Step 3: Configure Brainy to Use FileSystem Adapter
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { FileSystemStorage } from '@soulcraft/brainy/storage'
|
||||
|
||||
const storage = new FileSystemStorage({
|
||||
basePath: '/mnt/brainy/brain-data'
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
That's it. No GCS adapter, no rate limits, no retry logic needed.
|
||||
|
||||
## Caveats
|
||||
|
||||
### Single-Writer Recommendation
|
||||
|
||||
NFS does not provide strong file locking guarantees on Cloud Run. For best results:
|
||||
|
||||
- Use a **single Cloud Run instance** (max-instances=1) for write operations
|
||||
- Multiple read-only instances can safely read from the same Filestore mount
|
||||
- For multi-writer scenarios, use the GCS adapter with the write buffer (rate limit protection built in)
|
||||
|
||||
### Filestore Permissions
|
||||
|
||||
Cloud Run gen2 instances run as a specific service account. Ensure the Filestore instance allows access from your VPC network. No additional IAM permissions are needed — Filestore uses NFS network-level access.
|
||||
|
||||
### Cost Comparison
|
||||
|
||||
| Solution | Monthly Cost (1 TiB) | Write Latency | Rate Limits |
|
||||
|----------|---------------------|---------------|-------------|
|
||||
| GCS Standard | ~$20 storage + API ops | 50-300ms/write | 1 write/sec/object |
|
||||
| GCS HNS | ~$20 storage + API ops | 50-300ms/write | 8,000 writes/sec |
|
||||
| Filestore SSD | ~$370 fixed | ~1ms/write | None |
|
||||
| Filestore HDD | ~$204 fixed | ~5ms/write | None |
|
||||
|
||||
Filestore costs more for storage but eliminates all rate limiting issues and provides significantly lower write latency.
|
||||
|
||||
### When to Choose Filestore vs GCS
|
||||
|
||||
**Choose Filestore when:**
|
||||
- Write-heavy workloads (frequent `brain.add()`, chat applications)
|
||||
- Low-latency requirements
|
||||
- Single-writer architecture is acceptable
|
||||
|
||||
**Choose GCS (with write buffer) when:**
|
||||
- Multi-instance deployments need shared storage
|
||||
- Cost optimization for large datasets (lifecycle policies, Autoclass)
|
||||
- Read-heavy workloads with infrequent writes
|
||||
|
||||
## Verification
|
||||
|
||||
After deploying, verify the mount works:
|
||||
|
||||
```bash
|
||||
# In Cloud Run container
|
||||
ls -la /mnt/brainy/
|
||||
# Should show the Filestore share contents
|
||||
|
||||
# Test write performance
|
||||
dd if=/dev/zero of=/mnt/brainy/test bs=1M count=100
|
||||
# Expected: ~100MB/s+ for SSD tier
|
||||
```
|
||||
|
||||
Then run your brainy application and confirm `brain.add()` completes without rate limit errors.
|
||||
|
|
@ -1,399 +0,0 @@
|
|||
# AWS S3 Cost Optimization Guide for Brainy
|
||||
> **Cost Impact**: Reduce storage costs from $138k/year to $5.9k/year at 500TB scale (**96% savings**)
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance.
|
||||
|
||||
## Cost Breakdown (Before Optimization)
|
||||
|
||||
### Standard S3 Storage Costs (500TB Dataset)
|
||||
|
||||
```
|
||||
Storage: 500TB × $0.023/GB/month × 12 months = $138,000/year
|
||||
Operations: ~$5,000/year (PUT, GET, LIST requests)
|
||||
Total: $143,000/year
|
||||
```
|
||||
|
||||
## S3 Storage Tiers
|
||||
|
||||
| Tier | Cost/GB/Month | Retrieval Fee | Use Case |
|
||||
|------|---------------|---------------|----------|
|
||||
| **Standard** | $0.023 | None | Frequently accessed |
|
||||
| **Standard-IA** | $0.0125 | $0.01/GB | Infrequently accessed (30+ days) |
|
||||
| **Intelligent-Tiering** | $0.023-0.00099 | None | Automatic optimization |
|
||||
| **Glacier Instant** | $0.004 | $0.03/GB | Rare access, instant retrieval |
|
||||
| **Glacier Flexible** | $0.0036 | $0.01/GB + time | Archive (minutes-hours retrieval) |
|
||||
| **Glacier Deep Archive** | $0.00099 | $0.02/GB + time | Long-term archive (12 hours retrieval) |
|
||||
|
||||
## Strategy 1: Lifecycle Policies (Recommended)
|
||||
|
||||
### Setup: Automatic Tier Transitions
|
||||
|
||||
**Best for**: Predictable access patterns, batch workloads, archival data
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
|
||||
|
||||
// Initialize Brainy with S3 storage
|
||||
const storage = new S3CompatibleStorage({
|
||||
bucket: 'my-brainy-data',
|
||||
region: 'us-east-1',
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
|
||||
// Set lifecycle policy for automatic archival
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'optimize-vectors',
|
||||
prefix: 'entities/nouns/vectors/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' }, // Infrequent Access after 30 days
|
||||
{ days: 90, storageClass: 'GLACIER' }, // Glacier after 90 days
|
||||
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 1 year
|
||||
]
|
||||
}, {
|
||||
id: 'optimize-metadata',
|
||||
prefix: 'entities/nouns/metadata/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' },
|
||||
{ days: 180, storageClass: 'GLACIER' }
|
||||
]
|
||||
}]
|
||||
})
|
||||
|
||||
// Verify lifecycle policy
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Lifecycle policy active:', policy.rules.length, 'rules')
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB with Lifecycle Policy)
|
||||
|
||||
**Assumptions:**
|
||||
- 40% of data accessed in last 30 days (Standard)
|
||||
- 30% of data 30-90 days old (Standard-IA)
|
||||
- 20% of data 90-365 days old (Glacier)
|
||||
- 10% of data 365+ days old (Deep Archive)
|
||||
|
||||
```
|
||||
Standard (200TB): 200TB × $0.023/GB × 12 = $55,200/year
|
||||
Standard-IA (150TB): 150TB × $0.0125/GB × 12 = $22,500/year
|
||||
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
|
||||
Deep Archive (50TB): 50TB × $0.00099/GB × 12 = $594/year
|
||||
|
||||
Total Storage Cost: $83,094/year (instead of $138,000)
|
||||
Total with Operations: ~$88,000/year
|
||||
Savings: $55,000/year (40%)
|
||||
```
|
||||
|
||||
**But we can do better with Intelligent-Tiering...**
|
||||
|
||||
## Strategy 2: Intelligent-Tiering (Most Recommended)
|
||||
|
||||
### Setup: Automatic Access-Based Optimization
|
||||
|
||||
**Best for**: Unpredictable access patterns, mixed workloads, maximum automation
|
||||
|
||||
```typescript
|
||||
// Enable Intelligent-Tiering for automatic optimization
|
||||
await storage.enableIntelligentTiering('entities/', 'brainy-auto-optimize')
|
||||
|
||||
// Benefits:
|
||||
// - Automatically moves objects between tiers based on access patterns
|
||||
// - No retrieval fees (unlike Glacier)
|
||||
// - Transitions happen within 24-48 hours of last access
|
||||
// - Supports Archive Access tier (90+ days) and Deep Archive Access tier (180+ days)
|
||||
```
|
||||
|
||||
### Intelligent-Tiering Tiers
|
||||
|
||||
Intelligent-Tiering automatically moves objects between:
|
||||
|
||||
1. **Frequent Access**: $0.023/GB/month (0-30 days)
|
||||
2. **Infrequent Access**: $0.0125/GB/month (30-90 days)
|
||||
3. **Archive Access**: $0.004/GB/month (90-180 days)
|
||||
4. **Deep Archive Access**: $0.00099/GB/month (180+ days)
|
||||
|
||||
**Monitoring Fee**: $0.0025 per 1000 objects (minimal)
|
||||
|
||||
### Cost Calculation (500TB with Intelligent-Tiering)
|
||||
|
||||
**Realistic distribution after 1 year:**
|
||||
- 15% Frequent Access (hot data)
|
||||
- 20% Infrequent Access
|
||||
- 35% Archive Access
|
||||
- 30% Deep Archive Access
|
||||
|
||||
```
|
||||
Frequent (75TB): 75TB × $0.023/GB × 12 = $20,700/year
|
||||
Infrequent (100TB): 100TB × $0.0125/GB × 12 = $15,000/year
|
||||
Archive (175TB): 175TB × $0.004/GB × 12 = $8,400/year
|
||||
Deep Archive (150TB): 150TB × $0.00099/GB × 12 = $1,782/year
|
||||
Monitoring: ~$300/year (minimal)
|
||||
|
||||
Total Storage Cost: $46,182/year
|
||||
Total with Operations: ~$51,000/year
|
||||
Savings vs Standard: $92,000/year (67%)
|
||||
```
|
||||
|
||||
## Strategy 3: Hybrid Approach (Maximum Savings)
|
||||
|
||||
### Setup: Lifecycle + Intelligent-Tiering
|
||||
|
||||
**Best for**: Maximum cost optimization with fine-grained control
|
||||
|
||||
```typescript
|
||||
// Enable Intelligent-Tiering for vectors (frequently searched)
|
||||
await storage.enableIntelligentTiering('entities/nouns/vectors/', 'vectors-auto')
|
||||
await storage.enableIntelligentTiering('entities/verbs/vectors/', 'verbs-auto')
|
||||
|
||||
// Set lifecycle policy for metadata (less frequently accessed)
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'archive-old-metadata',
|
||||
prefix: 'entities/nouns/metadata/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' },
|
||||
{ days: 60, storageClass: 'GLACIER' },
|
||||
{ days: 180, storageClass: 'DEEP_ARCHIVE' }
|
||||
]
|
||||
}, {
|
||||
id: 'cleanup-old-system-data',
|
||||
prefix: '_system/',
|
||||
status: 'Enabled',
|
||||
expiration: { days: 365 } // Delete old statistics
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB Hybrid Approach)
|
||||
|
||||
**Vectors (300TB with Intelligent-Tiering):**
|
||||
```
|
||||
Frequent (45TB): 45TB × $0.023/GB × 12 = $12,420/year
|
||||
Infrequent (60TB): 60TB × $0.0125/GB × 12 = $9,000/year
|
||||
Archive (105TB): 105TB × $0.004/GB × 12 = $5,040/year
|
||||
Deep Archive (90TB): 90TB × $0.00099/GB × 12 = $1,069/year
|
||||
Subtotal: $27,529/year
|
||||
```
|
||||
|
||||
**Metadata (200TB with Lifecycle Policy):**
|
||||
```
|
||||
Standard (60TB): 60TB × $0.023/GB × 12 = $16,560/year
|
||||
Standard-IA (40TB): 40TB × $0.0125/GB × 12 = $6,000/year
|
||||
Glacier (60TB): 60TB × $0.004/GB × 12 = $2,880/year
|
||||
Deep Archive (40TB): 40TB × $0.00099/GB × 12 = $475/year
|
||||
Subtotal: $25,915/year
|
||||
```
|
||||
|
||||
**Total Cost: $53,444/year + operations (~$58,500/year total)**
|
||||
**Savings vs Standard: $84,500/year (61%)**
|
||||
|
||||
## Strategy 4: Aggressive Archival (Maximum Savings)
|
||||
|
||||
### Setup: Fast Archival for Cold Data
|
||||
|
||||
**Best for**: Archival workloads, historical data, compliance
|
||||
|
||||
```typescript
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'aggressive-archival',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 14, storageClass: 'STANDARD_IA' }, // IA after 2 weeks
|
||||
{ days: 30, storageClass: 'GLACIER' }, // Glacier after 1 month
|
||||
{ days: 90, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 3 months
|
||||
]
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB Aggressive Archival)
|
||||
|
||||
**After 1 year:**
|
||||
```
|
||||
Standard (50TB): 50TB × $0.023/GB × 12 = $13,800/year
|
||||
Standard-IA (50TB): 50TB × $0.0125/GB × 12 = $7,500/year
|
||||
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
|
||||
Deep Archive (300TB): 300TB × $0.00099/GB × 12 = $3,564/year
|
||||
|
||||
Total Storage Cost: $29,664/year
|
||||
Total with Operations: ~$34,000/year
|
||||
Savings: $109,000/year (76%)
|
||||
|
||||
Note: Retrieval costs may be significant if archived data is accessed frequently
|
||||
```
|
||||
|
||||
## Comparison Table: All Strategies
|
||||
|
||||
| Strategy | Annual Cost | Savings | Retrieval Speed | Best For |
|
||||
|----------|-------------|---------|-----------------|----------|
|
||||
| **No Optimization** | $143,000 | 0% | Instant | N/A |
|
||||
| **Lifecycle Policy** | $88,000 | 38% | Varies | Predictable patterns |
|
||||
| **Intelligent-Tiering** | $51,000 | 64% | Instant (no retrieval fees) | **Recommended** |
|
||||
| **Hybrid Approach** | $58,500 | 59% | Instant for vectors | Fine-grained control |
|
||||
| **Aggressive Archival** | $34,000 | 76% | Hours to 12 hours | Cold data, compliance |
|
||||
|
||||
## Batch Delete Operations
|
||||
|
||||
### Efficient Cleanup
|
||||
|
||||
```typescript
|
||||
// Batch delete (1000 objects per request)
|
||||
const idsToDelete = [/* array of entity IDs */]
|
||||
|
||||
// Generate paths for both vector and metadata files
|
||||
const paths = idsToDelete.flatMap(id => {
|
||||
const shard = id.substring(0, 2)
|
||||
return [
|
||||
`entities/nouns/vectors/${shard}/${id}.json`,
|
||||
`entities/nouns/metadata/${shard}/${id}.json`
|
||||
]
|
||||
})
|
||||
|
||||
// Batch delete (much faster and cheaper than individual deletes)
|
||||
await storage.batchDelete(paths)
|
||||
|
||||
// Cost impact:
|
||||
// - Individual deletes: 1M objects × $0.005 per 1000 = $5,000
|
||||
// - Batch deletes: 1M/1000 × $0.005 = $5 (1000x cheaper!)
|
||||
```
|
||||
|
||||
## Monitoring and Optimization
|
||||
|
||||
### Get Current Lifecycle Policy
|
||||
|
||||
```typescript
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Active rules:', policy.rules)
|
||||
|
||||
// Example output:
|
||||
// {
|
||||
// rules: [
|
||||
// {
|
||||
// id: 'optimize-vectors',
|
||||
// prefix: 'entities/nouns/vectors/',
|
||||
// status: 'Enabled',
|
||||
// transitions: [...]
|
||||
// }
|
||||
// ]
|
||||
// }
|
||||
```
|
||||
|
||||
### Remove Lifecycle Policy (if needed)
|
||||
|
||||
```typescript
|
||||
// Remove all lifecycle rules
|
||||
await storage.removeLifecyclePolicy()
|
||||
```
|
||||
|
||||
### Check Intelligent-Tiering Configurations
|
||||
|
||||
```typescript
|
||||
const configs = await storage.getIntelligentTieringConfigs()
|
||||
console.log('Active configurations:', configs)
|
||||
```
|
||||
|
||||
### Disable Intelligent-Tiering
|
||||
|
||||
```typescript
|
||||
await storage.disableIntelligentTiering('brainy-auto-optimize')
|
||||
```
|
||||
|
||||
## AWS Cost Explorer Analysis
|
||||
|
||||
### Track Your Savings
|
||||
|
||||
1. **Enable Cost Explorer** in AWS Console
|
||||
2. **Group by Storage Class** to see tier distribution
|
||||
3. **Set up Cost Anomaly Detection** for unexpected spikes
|
||||
4. **Create Budget Alerts** for monthly storage costs
|
||||
|
||||
### Expected Metrics After 6 Months
|
||||
|
||||
```
|
||||
Standard storage: 15-20% of total data
|
||||
Standard-IA: 20-25%
|
||||
Archive tiers: 55-65%
|
||||
|
||||
Monthly cost trend: Decreasing 5-10% per month as data ages into cheaper tiers
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. ✅ **Start with Intelligent-Tiering** - No retrieval fees, automatic optimization
|
||||
2. ✅ **Use batch operations** for deletions - 1000x cheaper than individual deletes
|
||||
3. ✅ **Monitor storage class distribution** monthly via Cost Explorer
|
||||
4. ✅ **Set lifecycle policies** for predictable archival (metadata, logs)
|
||||
5. ✅ **Enable S3 Storage Lens** for detailed storage analytics
|
||||
6. ✅ **Use S3 Select** for querying archived data without full retrieval
|
||||
7. ✅ **Consider S3 Batch Operations** for large-scale tier changes
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Data not transitioning to cheaper tiers
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Check if lifecycle policy is active
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Policy status:', policy.rules.map(r => r.status))
|
||||
|
||||
// Ensure objects are old enough
|
||||
// S3 requires objects to be at least 30 days old for IA transition
|
||||
```
|
||||
|
||||
### Issue: High retrieval costs from Glacier
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Switch to Intelligent-Tiering (no retrieval fees)
|
||||
await storage.disableIntelligentTiering('old-config')
|
||||
await storage.enableIntelligentTiering('entities/', 'new-config')
|
||||
|
||||
// Or use Glacier Instant Retrieval instead of Glacier Flexible
|
||||
```
|
||||
|
||||
### Issue: Unexpected monitoring fees
|
||||
|
||||
**Solution:**
|
||||
- Intelligent-Tiering has $0.0025 per 1000 objects monitoring fee
|
||||
- For 1 billion objects: $2,500/month monitoring
|
||||
- If cost is high, use lifecycle policies instead (no monitoring fee)
|
||||
|
||||
## Summary
|
||||
|
||||
**Recommended Strategy for Most Use Cases:**
|
||||
- **Intelligent-Tiering** for vectors and frequently queried data
|
||||
- **Lifecycle policies** for metadata and system files
|
||||
- **Batch operations** for efficient cleanup
|
||||
|
||||
**Expected Savings:**
|
||||
- **Year 1**: 40-50% reduction in storage costs
|
||||
- **Year 2+**: 60-70% reduction as more data ages into archive tiers
|
||||
- **Long-term**: 75-85% reduction for mature datasets
|
||||
|
||||
**500TB Example (Intelligent-Tiering):**
|
||||
- Before: $143,000/year
|
||||
- After: $51,000/year
|
||||
- **Savings: $92,000/year (64%)**
|
||||
|
||||
**1PB Example (Intelligent-Tiering):**
|
||||
- Before: $286,000/year
|
||||
- After: $102,000/year
|
||||
- **Savings: $184,000/year (64%)**
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-10-17
|
||||
**Cloud Provider**: AWS S3
|
||||
|
|
@ -1,552 +0,0 @@
|
|||
# Azure Blob Storage Cost Optimization Guide for Brainy
|
||||
> **Cost Impact**: Reduce storage costs from $107k/year to $5k/year at 500TB scale (**95% savings**)
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy provides enterprise-grade cost optimization features for Azure Blob Storage, including manual tier management, lifecycle policies, and batch operations.
|
||||
|
||||
## Cost Breakdown (Before Optimization)
|
||||
|
||||
### Hot Tier Azure Storage Costs (500TB Dataset)
|
||||
|
||||
```
|
||||
Storage: 500TB × $0.0184/GB/month × 12 months = $107,520/year
|
||||
Operations: ~$5,000/year (write/read operations)
|
||||
Total: $112,520/year
|
||||
```
|
||||
|
||||
## Azure Blob Storage Tiers
|
||||
|
||||
| Tier | Cost/GB/Month | Retrieval | Early Deletion | Use Case |
|
||||
|------|---------------|-----------|----------------|----------|
|
||||
| **Hot** | $0.0184 | None | None | Frequent access |
|
||||
| **Cool** | $0.0115 | $0.01/GB | 30 days | Infrequent access |
|
||||
| **Archive** | $0.00099 | $0.02/GB + rehydration time | 180 days | Long-term archive |
|
||||
|
||||
**Key Difference from AWS/GCS:**
|
||||
- Azure has only 3 tiers (vs AWS 6 tiers, GCS 4 classes)
|
||||
- Archive tier requires rehydration (hours to 15 hours) before access
|
||||
- No "Intelligent-Tiering" equivalent - must use lifecycle policies or manual management
|
||||
|
||||
## Strategy 1: Manual Tier Management (Immediate Savings)
|
||||
|
||||
### Setup: Change Blob Tiers Manually
|
||||
|
||||
**Best for**: Quick wins, specific files, immediate cost reduction
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { AzureBlobStorage } from '@soulcraft/brainy/storage'
|
||||
|
||||
// Initialize Brainy with Azure storage
|
||||
const storage = new AzureBlobStorage({
|
||||
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
|
||||
containerName: 'brainy-data'
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
|
||||
// Change tier for a single blob
|
||||
await storage.changeBlobTier(
|
||||
'entities/nouns/vectors/00/00123456-uuid.json',
|
||||
'Cool'
|
||||
)
|
||||
|
||||
// Batch tier changes (efficient for thousands of blobs)
|
||||
const blobsToMove = [
|
||||
'entities/nouns/vectors/01/...',
|
||||
'entities/nouns/vectors/02/...',
|
||||
// ... up to thousands of blobs
|
||||
]
|
||||
|
||||
await storage.batchChangeTier(blobsToMove, 'Cool') // Hot → Cool
|
||||
await storage.batchChangeTier(oldBlobs, 'Archive') // Cool → Archive
|
||||
```
|
||||
|
||||
### Immediate Cost Impact
|
||||
|
||||
Moving 400TB from Hot to Cool:
|
||||
```
|
||||
Before (Hot): 400TB × $0.0184/GB × 12 = $86,016/year
|
||||
After (Cool): 400TB × $0.0115/GB × 12 = $53,760/year
|
||||
Savings: $32,256/year (37% savings on moved data)
|
||||
```
|
||||
|
||||
Moving 100TB from Hot to Archive:
|
||||
```
|
||||
Before (Hot): 100TB × $0.0184/GB × 12 = $21,504/year
|
||||
After (Archive): 100TB × $0.00099/GB × 12 = $1,158/year
|
||||
Savings: $20,346/year (95% savings on moved data)
|
||||
```
|
||||
|
||||
## Strategy 2: Lifecycle Policies (Automated)
|
||||
|
||||
### Setup: Automatic Tier Transitions
|
||||
|
||||
**Best for**: Predictable patterns, automatic management
|
||||
|
||||
```typescript
|
||||
// Set lifecycle policy for automatic tier management
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
name: 'optimizeVectors',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: {
|
||||
blobTypes: ['blockBlob'],
|
||||
prefixMatch: ['entities/nouns/vectors/']
|
||||
},
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 30 },
|
||||
tierToArchive: { daysAfterModificationGreaterThan: 90 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}, {
|
||||
name: 'optimizeMetadata',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: {
|
||||
blobTypes: ['blockBlob'],
|
||||
prefixMatch: ['entities/nouns/metadata/']
|
||||
},
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 30 },
|
||||
tierToArchive: { daysAfterModificationGreaterThan: 180 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}, {
|
||||
name: 'cleanupOldSystemFiles',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: {
|
||||
blobTypes: ['blockBlob'],
|
||||
prefixMatch: ['_system/']
|
||||
},
|
||||
actions: {
|
||||
baseBlob: {
|
||||
delete: { daysAfterModificationGreaterThan: 365 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
|
||||
// Verify lifecycle policy
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Lifecycle policy active:', policy.rules.length, 'rules')
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB with Lifecycle Policy)
|
||||
|
||||
**Assumptions:**
|
||||
- 30% of data in Hot tier (< 30 days old)
|
||||
- 40% of data in Cool tier (30-90 days old)
|
||||
- 30% of data in Archive tier (90+ days old)
|
||||
|
||||
```
|
||||
Hot (150TB): 150TB × $0.0184/GB × 12 = $32,256/year
|
||||
Cool (200TB): 200TB × $0.0115/GB × 12 = $26,880/year
|
||||
Archive (150TB): 150TB × $0.00099/GB × 12 = $1,732/year
|
||||
|
||||
Total Storage Cost: $60,868/year
|
||||
Total with Operations: ~$65,500/year
|
||||
Savings: $47,000/year (42%)
|
||||
```
|
||||
|
||||
## Strategy 3: Aggressive Archival (Maximum Savings)
|
||||
|
||||
### Setup: Fast Archival for Cold Data
|
||||
|
||||
**Best for**: Archival workloads, compliance, historical data
|
||||
|
||||
```typescript
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
name: 'aggressiveArchival',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: {
|
||||
blobTypes: ['blockBlob'],
|
||||
prefixMatch: ['entities/']
|
||||
},
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 14 },
|
||||
tierToArchive: { daysAfterModificationGreaterThan: 30 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB Aggressive Archival)
|
||||
|
||||
**After 6 months:**
|
||||
```
|
||||
Hot (50TB): 50TB × $0.0184/GB × 12 = $10,752/year
|
||||
Cool (100TB): 100TB × $0.0115/GB × 12 = $13,440/year
|
||||
Archive (350TB): 350TB × $0.00099/GB × 12 = $4,039/year
|
||||
|
||||
Total Storage Cost: $28,231/year
|
||||
Total with Operations: ~$33,000/year
|
||||
Savings: $79,500/year (71%)
|
||||
|
||||
Warning: Archive rehydration takes 1-15 hours
|
||||
```
|
||||
|
||||
## Strategy 4: Hybrid Approach (Balanced)
|
||||
|
||||
### Setup: Different Policies for Different Data Types
|
||||
|
||||
```typescript
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
// Vectors: Keep in Hot/Cool for search performance
|
||||
name: 'vectors-moderate',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: {
|
||||
blobTypes: ['blockBlob'],
|
||||
prefixMatch: ['entities/nouns/vectors/', 'entities/verbs/vectors/']
|
||||
},
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 60 }
|
||||
// Don't archive vectors - keep searchable
|
||||
}
|
||||
}
|
||||
}
|
||||
}, {
|
||||
// Metadata: Aggressive archival
|
||||
name: 'metadata-aggressive',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: {
|
||||
blobTypes: ['blockBlob'],
|
||||
prefixMatch: ['entities/nouns/metadata/', 'entities/verbs/metadata/']
|
||||
},
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 30 },
|
||||
tierToArchive: { daysAfterModificationGreaterThan: 90 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB Hybrid Approach)
|
||||
|
||||
**Vectors (300TB):**
|
||||
```
|
||||
Hot (90TB): 90TB × $0.0184/GB × 12 = $19,354/year
|
||||
Cool (210TB): 210TB × $0.0115/GB × 12 = $28,980/year
|
||||
Subtotal: $48,334/year
|
||||
```
|
||||
|
||||
**Metadata (200TB):**
|
||||
```
|
||||
Hot (30TB): 30TB × $0.0184/GB × 12 = $6,451/year
|
||||
Cool (70TB): 70TB × $0.0115/GB × 12 = $9,660/year
|
||||
Archive (100TB): 100TB × $0.00099/GB × 12 = $1,158/year
|
||||
Subtotal: $17,269/year
|
||||
```
|
||||
|
||||
**Total Cost: $65,603/year + operations (~$70,500/year total)**
|
||||
**Savings vs Hot: $42,000/year (37%)**
|
||||
|
||||
## Comparison Table: All Strategies
|
||||
|
||||
| Strategy | Annual Cost | Savings | Archive % | Best For |
|
||||
|----------|-------------|---------|-----------|----------|
|
||||
| **No Optimization** | $112,500 | 0% | 0% | N/A |
|
||||
| **Manual Tier Mgmt** | $75,000 | 33% | 20% | Immediate savings |
|
||||
| **Lifecycle Policy** | $65,500 | 42% | 30% | Automated management |
|
||||
| **Hybrid Approach** | $70,500 | 37% | 20% | Balance performance/cost |
|
||||
| **Aggressive Archival** | $33,000 | **71%** | 70% | Cold data, compliance |
|
||||
|
||||
## Archive Rehydration (Important!)
|
||||
|
||||
### Rehydrate from Archive Tier
|
||||
|
||||
**Required before accessing archived blobs:**
|
||||
|
||||
```typescript
|
||||
// Rehydrate blob from Archive to Hot (high priority)
|
||||
await storage.rehydrateBlob(
|
||||
'entities/nouns/vectors/00/00123456-uuid.json',
|
||||
'High' // 'Standard' or 'High' priority
|
||||
)
|
||||
|
||||
// Rehydration time:
|
||||
// - High priority: 1 hour
|
||||
// - Standard priority: up to 15 hours
|
||||
|
||||
// Check rehydration status
|
||||
const metadata = await storage.getBlobMetadata(blobPath)
|
||||
console.log('Archive status:', metadata.archiveStatus)
|
||||
// Output: 'rehydrate-pending-to-hot', 'rehydrate-pending-to-cool', or undefined (done)
|
||||
```
|
||||
|
||||
### Batch Rehydration
|
||||
|
||||
```typescript
|
||||
// Rehydrate multiple blobs (useful for planned access)
|
||||
const blobsToRehydrate = [/* array of blob paths */]
|
||||
|
||||
for (const blobPath of blobsToRehydrate) {
|
||||
await storage.rehydrateBlob(blobPath, 'High')
|
||||
}
|
||||
|
||||
// Wait for rehydration to complete (1-15 hours)
|
||||
// Then access blobs normally
|
||||
```
|
||||
|
||||
### Cost Impact of Rehydration
|
||||
|
||||
```
|
||||
Retrieval fee: $0.02/GB
|
||||
High priority: Additional $0.10/GB
|
||||
|
||||
Examples:
|
||||
- 1GB blob (standard): $0.02 + wait 15 hours
|
||||
- 1GB blob (high priority): $0.12 + wait 1 hour
|
||||
- 1TB batch (high priority): $122.88 + wait 1 hour
|
||||
```
|
||||
|
||||
## Batch Operations
|
||||
|
||||
### Efficient Bulk Deletions
|
||||
|
||||
```typescript
|
||||
// Batch delete (256 blobs per request)
|
||||
const idsToDelete = [/* array of entity IDs */]
|
||||
|
||||
const paths = idsToDelete.flatMap(id => {
|
||||
const shard = id.substring(0, 2)
|
||||
return [
|
||||
`entities/nouns/vectors/${shard}/${id}.json`,
|
||||
`entities/nouns/metadata/${shard}/${id}.json`
|
||||
]
|
||||
})
|
||||
|
||||
// Batch delete via BlobBatchClient
|
||||
await storage.batchDelete(paths)
|
||||
|
||||
// Cost impact:
|
||||
// - Individual deletes: 1M operations × $0.0005 per 10k = $50
|
||||
// - Batch deletes: 4k batches × $0.0005 = $2 (25x cheaper!)
|
||||
```
|
||||
|
||||
### Batch Tier Changes
|
||||
|
||||
```typescript
|
||||
// Change tier for thousands of blobs efficiently
|
||||
const vectorPaths = [/* 10,000 blob paths */]
|
||||
|
||||
// Batch operation (256 blobs per batch)
|
||||
await storage.batchChangeTier(vectorPaths, 'Cool')
|
||||
|
||||
// Much faster than individual changeBlobTier() calls
|
||||
// Azure automatically batches internally for efficiency
|
||||
```
|
||||
|
||||
## Monitoring and Management
|
||||
|
||||
### Get Current Lifecycle Policy
|
||||
|
||||
```typescript
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Active rules:', policy.rules)
|
||||
|
||||
// Example output:
|
||||
// {
|
||||
// rules: [
|
||||
// {
|
||||
// name: 'optimizeVectors',
|
||||
// enabled: true,
|
||||
// type: 'Lifecycle',
|
||||
// definition: {...}
|
||||
// }
|
||||
// ]
|
||||
// }
|
||||
```
|
||||
|
||||
### Remove Lifecycle Policy
|
||||
|
||||
```typescript
|
||||
await storage.removeLifecyclePolicy()
|
||||
```
|
||||
|
||||
### Get Storage Status
|
||||
|
||||
```typescript
|
||||
const status = await storage.getStorageStatus()
|
||||
console.log('Storage type:', status.type)
|
||||
console.log('Container:', status.details.container)
|
||||
console.log('Account:', status.details.account)
|
||||
```
|
||||
|
||||
## Azure Portal Monitoring
|
||||
|
||||
### Track Your Savings
|
||||
|
||||
1. **Storage Account** → **Insights** → View tier distribution
|
||||
2. **Cost Management** → **Cost Analysis** → Filter by storage account
|
||||
3. **Monitoring** → **Metrics** → Track blob count by tier
|
||||
4. **Lifecycle Management** → View policy execution logs
|
||||
|
||||
### Expected Metrics After 6 Months
|
||||
|
||||
```
|
||||
Hot tier: 20-30% of total data
|
||||
Cool tier: 40-50%
|
||||
Archive tier: 20-40%
|
||||
|
||||
Monthly cost trend: Decreasing 5-8% per month as data transitions
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. ✅ **Use lifecycle policies** for automatic management
|
||||
2. ✅ **Archive cold data** aggressively (95% cost savings!)
|
||||
3. ✅ **Use batch operations** for tier changes and deletions
|
||||
4. ✅ **Plan rehydration** ahead of time (1-15 hour delay)
|
||||
5. ✅ **Monitor early deletion charges** (Cool: 30 days, Archive: 180 days)
|
||||
6. ✅ **Use Cool tier for infrequent access** (no rehydration needed)
|
||||
7. ✅ **Consider ZRS or GRS** for critical data redundancy
|
||||
|
||||
## Storage Redundancy Options
|
||||
|
||||
| Option | Cost Multiplier | Copies | Availability | Use Case |
|
||||
|--------|-----------------|--------|--------------|----------|
|
||||
| **LRS** | 1x | 3 (same datacenter) | 11 nines | Cost-optimized |
|
||||
| **ZRS** | 1.25x | 3 (different zones) | 12 nines | High availability |
|
||||
| **GRS** | 2x | 6 (secondary region) | 16 nines | Disaster recovery |
|
||||
| **RA-GRS** | 2.5x | 6 (read access) | 16 nines | Global read access |
|
||||
|
||||
**Recommendation for Brainy:**
|
||||
- **Production**: GRS (geo-redundancy for disaster recovery)
|
||||
- **Cost-optimized**: LRS (lowest cost, still very reliable)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Data not transitioning tiers
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Check lifecycle policy status
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Policy rules:', policy.rules.map(r => ({
|
||||
name: r.name,
|
||||
enabled: r.enabled
|
||||
})))
|
||||
|
||||
// Azure lifecycle policies run once per day
|
||||
// Transitions may take 24-48 hours to execute
|
||||
```
|
||||
|
||||
### Issue: Access denied on archived blob
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Archived blobs cannot be accessed directly
|
||||
// Must rehydrate first:
|
||||
await storage.rehydrateBlob(blobPath, 'High')
|
||||
|
||||
// Wait for rehydration (check status)
|
||||
let status
|
||||
do {
|
||||
await new Promise(resolve => setTimeout(resolve, 60000)) // Wait 1 minute
|
||||
const metadata = await storage.getBlobMetadata(blobPath)
|
||||
status = metadata.archiveStatus
|
||||
} while (status && status.includes('pending'))
|
||||
|
||||
// Now access the blob
|
||||
const data = await storage.get(blobPath)
|
||||
```
|
||||
|
||||
### Issue: High early deletion charges
|
||||
|
||||
**Solution:**
|
||||
- Cool tier: 30-day minimum storage
|
||||
- Archive tier: 180-day minimum storage
|
||||
- Early deletion incurs pro-rated charges
|
||||
- Use lifecycle policies instead of manual changes to avoid early deletion fees
|
||||
|
||||
## Authentication
|
||||
|
||||
### Connection String (Simple)
|
||||
|
||||
```typescript
|
||||
const storage = new AzureBlobStorage({
|
||||
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
|
||||
containerName: 'brainy-data'
|
||||
})
|
||||
```
|
||||
|
||||
### Account Key (Explicit)
|
||||
|
||||
```typescript
|
||||
const storage = new AzureBlobStorage({
|
||||
accountName: process.env.AZURE_STORAGE_ACCOUNT,
|
||||
accountKey: process.env.AZURE_STORAGE_KEY,
|
||||
containerName: 'brainy-data'
|
||||
})
|
||||
```
|
||||
|
||||
### SAS Token (Granular Access)
|
||||
|
||||
```typescript
|
||||
const storage = new AzureBlobStorage({
|
||||
sasToken: process.env.AZURE_STORAGE_SAS_TOKEN,
|
||||
accountName: process.env.AZURE_STORAGE_ACCOUNT,
|
||||
containerName: 'brainy-data'
|
||||
})
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
**Recommended Strategy for Most Use Cases:**
|
||||
- **Lifecycle policies** for automatic tier management
|
||||
- **Batch operations** for efficient tier changes and deletions
|
||||
- **Cool tier** for infrequently accessed data (no rehydration delay)
|
||||
- **Archive tier** for long-term storage (1-15 hour rehydration)
|
||||
|
||||
**Expected Savings:**
|
||||
- **Year 1**: 40-50% reduction in storage costs
|
||||
- **Year 2+**: 60-70% reduction as more data ages into Archive
|
||||
- **Long-term**: 75-85% reduction for mature datasets
|
||||
|
||||
**500TB Example (Lifecycle Policy):**
|
||||
- Before: $112,500/year
|
||||
- After: $65,500/year
|
||||
- **Savings: $47,000/year (42%)**
|
||||
|
||||
**500TB Example (Aggressive Archival):**
|
||||
- Before: $112,500/year
|
||||
- After: $33,000/year
|
||||
- **Savings: $79,500/year (71%)**
|
||||
|
||||
**1PB Example (Lifecycle Policy):**
|
||||
- Before: $225,000/year
|
||||
- After: $131,000/year
|
||||
- **Savings: $94,000/year (42%)**
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-10-17
|
||||
**Cloud Provider**: Azure Blob Storage
|
||||
|
|
@ -1,450 +0,0 @@
|
|||
# Cloudflare R2 Cost Optimization Guide for Brainy
|
||||
> **Cost Impact**: $0 egress fees + 96% storage savings = Lowest cloud storage costs
|
||||
|
||||
## Overview
|
||||
|
||||
Cloudflare R2 is an S3-compatible object storage with **zero egress fees**, making it ideal for high-traffic applications. Brainy fully supports R2 with lifecycle policies and batch operations.
|
||||
|
||||
## Cost Breakdown
|
||||
|
||||
### R2 Pricing (As of 2025)
|
||||
|
||||
```
|
||||
Storage: $0.015/GB/month
|
||||
Class A Operations (write): $4.50 per million
|
||||
Class B Operations (read): $0.36 per million
|
||||
Egress: $0.00 (FREE!)
|
||||
```
|
||||
|
||||
### Comparison to AWS S3 (500TB Dataset)
|
||||
|
||||
**AWS S3 Standard:**
|
||||
```
|
||||
Storage: 500TB × $0.023/GB × 12 = $138,000/year
|
||||
Egress (assume 100TB/month): 1.2PB × $0.09/GB = $108,000/year
|
||||
Operations: $5,000/year
|
||||
Total: $251,000/year
|
||||
```
|
||||
|
||||
**Cloudflare R2 Standard:**
|
||||
```
|
||||
Storage: 500TB × $0.015/GB × 12 = $90,000/year
|
||||
Egress: $0 (FREE!)
|
||||
Operations: $5,000/year
|
||||
Total: $95,000/year
|
||||
Savings vs AWS: $156,000/year (62%)
|
||||
```
|
||||
|
||||
**R2's Zero Egress Advantage:**
|
||||
- **High-traffic apps**: Save $100k-$1M/year in egress fees
|
||||
- **Video/media delivery**: No CDN egress costs
|
||||
- **API responses**: Unlimited reads at no extra cost
|
||||
|
||||
## R2 Storage Classes (Coming Soon)
|
||||
|
||||
**Current State (2025):**
|
||||
- R2 currently has only **one storage class** (Standard)
|
||||
- No lifecycle policies or tier transitions yet
|
||||
- Cloudflare plans to add infrequent access tiers
|
||||
|
||||
**When lifecycle features arrive:**
|
||||
- Brainy is already prepared with `setLifecyclePolicy()` support
|
||||
- Will work seamlessly once Cloudflare enables lifecycle management
|
||||
|
||||
## Strategy 1: Use R2 Standard (Current Best Practice)
|
||||
|
||||
### Setup: S3-Compatible API
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
|
||||
|
||||
// R2 uses S3-compatible API
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
|
||||
bucket: 'my-brainy-data',
|
||||
region: 'auto', // R2 uses 'auto' region
|
||||
accessKeyId: process.env.R2_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB on R2)
|
||||
|
||||
```
|
||||
Storage: 500TB × $0.015/GB × 12 = $90,000/year
|
||||
Class A ops (10M writes): $45/year
|
||||
Class B ops (100M reads): $36/year
|
||||
Egress: $0
|
||||
|
||||
Total: $90,081/year
|
||||
```
|
||||
|
||||
**Compared to AWS S3 (with egress):**
|
||||
- AWS: $251,000/year
|
||||
- R2: $90,081/year
|
||||
- **Savings: $160,919/year (64%)**
|
||||
|
||||
## Strategy 2: R2 + Workers (Edge Computing)
|
||||
|
||||
### Setup: Compute at the Edge
|
||||
|
||||
```typescript
|
||||
// Cloudflare Worker (runs at edge)
|
||||
export default {
|
||||
async fetch(request, env) {
|
||||
// Initialize Brainy with R2
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: env.R2_ENDPOINT,
|
||||
bucket: env.R2_BUCKET,
|
||||
accessKeyId: env.R2_ACCESS_KEY,
|
||||
secretAccessKey: env.R2_SECRET_KEY,
|
||||
region: 'auto'
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
|
||||
// Process at edge (no origin server needed)
|
||||
const results = await brain.search(request.query)
|
||||
return new Response(JSON.stringify(results))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cost Calculation (Workers + R2)
|
||||
|
||||
```
|
||||
R2 Storage (500TB): $90,000/year
|
||||
Workers (10M requests/day):
|
||||
- First 100k requests/day: FREE
|
||||
- Additional 350M requests/month: $1,750/year
|
||||
- CPU time (50ms avg): $5,000/year
|
||||
|
||||
Total: $96,750/year
|
||||
|
||||
vs Traditional Setup (AWS S3 + EC2 + CloudFront):
|
||||
- S3: $138,000/year
|
||||
- EC2 (t3.xlarge × 4): $24,000/year
|
||||
- CloudFront egress: $50,000/year
|
||||
- Load balancer: $3,000/year
|
||||
Total: $215,000/year
|
||||
|
||||
Savings: $118,250/year (55%)
|
||||
```
|
||||
|
||||
## Strategy 3: Hybrid Multi-Cloud (R2 + S3)
|
||||
|
||||
### Setup: R2 for Hot Data, S3 for Archives
|
||||
|
||||
```typescript
|
||||
// Use R2 for frequently accessed data (zero egress)
|
||||
const hotStorage = new S3CompatibleStorage({
|
||||
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
|
||||
bucket: 'brainy-hot',
|
||||
region: 'auto',
|
||||
accessKeyId: process.env.R2_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
|
||||
})
|
||||
|
||||
// Use AWS S3 with Intelligent-Tiering for cold data
|
||||
const coldStorage = new S3CompatibleStorage({
|
||||
endpoint: 's3.amazonaws.com',
|
||||
bucket: 'brainy-archive',
|
||||
region: 'us-east-1',
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
})
|
||||
|
||||
// Initialize separate Brainy instances or implement tiering logic
|
||||
```
|
||||
|
||||
### Cost Calculation (300TB R2 + 200TB S3 Archive)
|
||||
|
||||
```
|
||||
R2 Hot Data (300TB):
|
||||
Storage: 300TB × $0.015/GB × 12 = $54,000/year
|
||||
Egress: $0
|
||||
|
||||
S3 Cold Data (200TB with lifecycle → Deep Archive):
|
||||
Storage: 200TB × $0.00099/GB × 12 = $2,376/year
|
||||
Ops: $1,000/year
|
||||
|
||||
Total: $57,376/year
|
||||
|
||||
vs All AWS S3:
|
||||
- S3 storage + egress: $251,000/year
|
||||
|
||||
Savings: $193,624/year (77%)
|
||||
```
|
||||
|
||||
## Batch Operations
|
||||
|
||||
### Efficient Bulk Deletions
|
||||
|
||||
```typescript
|
||||
// R2 supports S3 batch delete API
|
||||
const idsToDelete = [/* array of entity IDs */]
|
||||
|
||||
const paths = idsToDelete.flatMap(id => {
|
||||
const shard = id.substring(0, 2)
|
||||
return [
|
||||
`entities/nouns/vectors/${shard}/${id}.json`,
|
||||
`entities/nouns/metadata/${shard}/${id}.json`
|
||||
]
|
||||
})
|
||||
|
||||
// Batch delete (1000 objects per request)
|
||||
await storage.batchDelete(paths)
|
||||
|
||||
// Cost impact:
|
||||
// - Individual deletes: 1M × $4.50/1M = $4.50
|
||||
// - Batch deletes: 1k batches × $4.50/1M = $0.0045 (1000x cheaper!)
|
||||
```
|
||||
|
||||
## R2 Advanced Features
|
||||
|
||||
### 1. R2 Custom Domains
|
||||
|
||||
**Free custom domains for R2 buckets:**
|
||||
|
||||
```bash
|
||||
# Configure custom domain in Cloudflare dashboard
|
||||
# Then access via your domain
|
||||
https://storage.yourdomain.com/entities/nouns/vectors/...
|
||||
|
||||
# Benefits:
|
||||
# - No additional cost
|
||||
# - Automatic SSL/TLS
|
||||
# - Global CDN included
|
||||
# - DDoS protection
|
||||
```
|
||||
|
||||
### 2. R2 Event Notifications
|
||||
|
||||
**Trigger Workers on object events:**
|
||||
|
||||
```typescript
|
||||
// Worker triggered on R2 object upload
|
||||
export default {
|
||||
async fetch(request, env) {
|
||||
// Process new objects automatically
|
||||
// E.g., index new entities, generate thumbnails, etc.
|
||||
}
|
||||
}
|
||||
|
||||
// Cost: Only pay for Worker execution (no polling needed)
|
||||
```
|
||||
|
||||
### 3. R2 Presigned URLs
|
||||
|
||||
```typescript
|
||||
// Generate presigned URL for direct browser uploads
|
||||
const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry
|
||||
|
||||
// Client uploads directly to R2 (no server bandwidth used)
|
||||
```
|
||||
|
||||
## Monitoring and Management
|
||||
|
||||
### Get Storage Status
|
||||
|
||||
```typescript
|
||||
const status = await storage.getStorageStatus()
|
||||
console.log('Storage type:', status.type) // 's3-compatible'
|
||||
console.log('Bucket:', status.details.bucket)
|
||||
console.log('Endpoint:', status.details.endpoint)
|
||||
```
|
||||
|
||||
### Cloudflare Dashboard Monitoring
|
||||
|
||||
1. **R2 Dashboard** → View bucket metrics
|
||||
2. **Analytics** → Track requests, storage, and bandwidth
|
||||
3. **Workers Analytics** → Monitor edge compute usage
|
||||
4. **Logs** → Real-time logs with Logpush
|
||||
|
||||
### Expected Metrics
|
||||
|
||||
```
|
||||
Storage: Growing with your data
|
||||
Class A ops: Writes (higher cost)
|
||||
Class B ops: Reads (minimal cost)
|
||||
Egress: Always $0 (R2's advantage)
|
||||
```
|
||||
|
||||
## Comparison Table: R2 vs Other Providers
|
||||
|
||||
| Feature | R2 | AWS S3 | GCS | Azure |
|
||||
|---------|-----|--------|-----|-------|
|
||||
| **Storage** | $0.015/GB | $0.023/GB | $0.020/GB | $0.0184/GB |
|
||||
| **Egress** | **$0** | $0.09/GB | $0.12/GB | $0.087/GB |
|
||||
| **Lifecycle Tiers** | Coming soon | 6 tiers | 4 classes | 3 tiers |
|
||||
| **S3 API Compatible** | ✅ Yes | ✅ Native | ⚠️ Via interop | ⚠️ Via SDK |
|
||||
| **CDN Included** | ✅ Yes | ❌ Extra cost | ❌ Extra cost | ❌ Extra cost |
|
||||
| **Edge Compute** | ✅ Workers | ❌ Lambda@Edge | ❌ Cloud Functions | ❌ Functions |
|
||||
|
||||
## R2 Free Tier
|
||||
|
||||
**Generous free tier:**
|
||||
```
|
||||
Storage: 10 GB free per month
|
||||
Class A ops: 1 million free per month
|
||||
Class B ops: 10 million free per month
|
||||
Egress: Unlimited (always free)
|
||||
```
|
||||
|
||||
**Perfect for:**
|
||||
- Development and testing
|
||||
- Small applications (<10GB)
|
||||
- Prototypes
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. ✅ **Use R2 for high-egress workloads** - Zero egress fees
|
||||
2. ✅ **Combine with Workers** - Edge compute included
|
||||
3. ✅ **Use custom domains** - Free branded URLs
|
||||
4. ✅ **Batch operations** for deletions - 1000x cheaper
|
||||
5. ✅ **Use presigned URLs** - Direct client uploads
|
||||
6. ✅ **Monitor with Analytics** - Built-in dashboarding
|
||||
7. ✅ **Consider hybrid approach** - R2 hot + S3 archive cold
|
||||
|
||||
## Migration from S3 to R2
|
||||
|
||||
### Using rclone
|
||||
|
||||
```bash
|
||||
# Install rclone
|
||||
brew install rclone # or apt-get install rclone
|
||||
|
||||
# Configure S3 source
|
||||
rclone config create s3-source s3 \
|
||||
access_key_id=$AWS_ACCESS_KEY \
|
||||
secret_access_key=$AWS_SECRET_KEY \
|
||||
region=us-east-1
|
||||
|
||||
# Configure R2 destination
|
||||
rclone config create r2-dest s3 \
|
||||
access_key_id=$R2_ACCESS_KEY \
|
||||
secret_access_key=$R2_SECRET_KEY \
|
||||
endpoint=https://$R2_ACCOUNT_ID.r2.cloudflarestorage.com \
|
||||
region=auto
|
||||
|
||||
# Copy data
|
||||
rclone copy s3-source:my-bucket r2-dest:my-bucket --progress
|
||||
|
||||
# Verify
|
||||
rclone check s3-source:my-bucket r2-dest:my-bucket
|
||||
```
|
||||
|
||||
### Cost of Migration
|
||||
|
||||
```
|
||||
Data transfer out from S3: 500TB × $0.09/GB = $45,000
|
||||
Data transfer into R2: $0 (ingress is free)
|
||||
|
||||
One-time migration cost: $45,000
|
||||
|
||||
Monthly savings after migration:
|
||||
S3 storage + egress: $20,833/month
|
||||
R2 storage: $7,500/month
|
||||
Savings: $13,333/month
|
||||
|
||||
ROI: 3.4 months
|
||||
```
|
||||
|
||||
## Future: R2 Lifecycle Policies (When Available)
|
||||
|
||||
### Prepared for Future Features
|
||||
|
||||
```typescript
|
||||
// Brainy is ready for R2 lifecycle features
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'archive-old-data',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'INFREQUENT_ACCESS' }, // When available
|
||||
{ days: 90, storageClass: 'ARCHIVE' }
|
||||
]
|
||||
}]
|
||||
})
|
||||
|
||||
// Expected cost impact (when lifecycle is available):
|
||||
// Standard: $0.015/GB
|
||||
// Infrequent: ~$0.008/GB (estimated)
|
||||
// Archive: ~$0.002/GB (estimated)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Connection errors
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Ensure correct endpoint format
|
||||
const storage = new S3CompatibleStorage({
|
||||
endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
|
||||
// NOT: `https://r2.cloudflarestorage.com/${accountId}`
|
||||
|
||||
region: 'auto', // R2 requires 'auto'
|
||||
forcePathStyle: false // R2 uses virtual-hosted-style
|
||||
})
|
||||
```
|
||||
|
||||
### Issue: High Class A operation costs
|
||||
|
||||
**Solution:**
|
||||
- Use batch operations (writes are most expensive)
|
||||
- Cache frequently written data
|
||||
- Consolidate small writes into larger batches
|
||||
- Consider Workers KV for high-frequency writes
|
||||
|
||||
### Issue: Need lifecycle management now
|
||||
|
||||
**Solution:**
|
||||
- Manually move old data to S3 Deep Archive
|
||||
- Use hybrid approach: R2 for hot, S3 for cold
|
||||
- Wait for R2 lifecycle features (planned)
|
||||
|
||||
## Summary
|
||||
|
||||
**R2 Advantages:**
|
||||
- ✅ **Zero egress fees** - Unlimited reads at no cost
|
||||
- ✅ **Lower storage costs** - $0.015/GB vs $0.023/GB (AWS)
|
||||
- ✅ **S3-compatible API** - Drop-in replacement for S3
|
||||
- ✅ **Global CDN included** - No additional CDN costs
|
||||
- ✅ **Edge Workers** - Compute at the edge
|
||||
- ✅ **Free custom domains** - Branded URLs
|
||||
- ✅ **No minimums** - No minimum storage duration
|
||||
|
||||
**R2 Limitations (Current):**
|
||||
- ⚠️ Single storage class (for now)
|
||||
- ⚠️ No lifecycle policies yet (coming soon)
|
||||
- ⚠️ Less mature than S3/GCS/Azure
|
||||
|
||||
**Recommended Use Cases:**
|
||||
- 🎯 High-traffic APIs (zero egress fees!)
|
||||
- 🎯 Video/media delivery (massive savings)
|
||||
- 🎯 User-generated content
|
||||
- 🎯 Web application assets
|
||||
- 🎯 Hot data storage
|
||||
|
||||
**500TB Example (R2 vs AWS S3 with 100TB/month egress):**
|
||||
- AWS S3: $251,000/year
|
||||
- Cloudflare R2: $90,000/year
|
||||
- **Savings: $161,000/year (64%)**
|
||||
|
||||
**1PB Example (R2 vs AWS S3 with 200TB/month egress):**
|
||||
- AWS S3: $686,000/year
|
||||
- Cloudflare R2: $180,000/year
|
||||
- **Savings: $506,000/year (74%)**
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-10-17
|
||||
**Cloud Provider**: Cloudflare R2
|
||||
**Key Advantage**: **$0 egress fees forever**
|
||||
|
|
@ -1,465 +0,0 @@
|
|||
# Google Cloud Storage Cost Optimization Guide for Brainy
|
||||
> **Cost Impact**: Reduce storage costs from $138k/year to $8.3k/year at 500TB scale (**94% savings**)
|
||||
|
||||
> **Disclaimer**: Cost savings percentages are PROJECTED based on published cloud provider pricing at 500TB scale. Actual savings depend on access patterns, data lifecycle, and workload characteristics. These are not measured benchmarks.
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management.
|
||||
|
||||
## Cost Breakdown (Before Optimization)
|
||||
|
||||
### Standard GCS Storage Costs (500TB Dataset)
|
||||
|
||||
```
|
||||
Storage: 500TB × $0.023/GB/month × 12 months = $138,000/year
|
||||
Operations: ~$5,000/year (Class A/B operations)
|
||||
Total: $143,000/year
|
||||
```
|
||||
|
||||
## GCS Storage Classes
|
||||
|
||||
| Class | Cost/GB/Month | Retrieval Fee | Minimum Storage | Use Case |
|
||||
|-------|---------------|---------------|-----------------|----------|
|
||||
| **Standard** | $0.020 | None | None | Frequent access |
|
||||
| **Nearline** | $0.010 | $0.01/GB | 30 days | Once per month |
|
||||
| **Coldline** | $0.004 | $0.02/GB | 90 days | Once per quarter |
|
||||
| **Archive** | $0.0012 | $0.05/GB | 365 days | Long-term archive |
|
||||
|
||||
## Strategy 1: Lifecycle Policies (Manual Control)
|
||||
|
||||
### Setup: Automatic Tier Transitions
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { GcsStorage } from '@soulcraft/brainy/storage'
|
||||
|
||||
// Initialize Brainy with GCS storage
|
||||
const storage = new GcsStorage({
|
||||
bucketName: 'my-brainy-data',
|
||||
keyFilename: './service-account.json' // Or use ADC
|
||||
})
|
||||
|
||||
const brain = new Brainy({ storage })
|
||||
await brain.init()
|
||||
|
||||
// Set lifecycle policy for automatic archival
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
condition: { age: 30 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||||
}, {
|
||||
condition: { age: 90 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||||
}, {
|
||||
condition: { age: 365 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||||
}]
|
||||
})
|
||||
|
||||
// Verify lifecycle policy
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Lifecycle policy active:', policy.rules.length, 'rules')
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB with Lifecycle Policy)
|
||||
|
||||
**Assumptions:**
|
||||
- 40% of data accessed in last 30 days (Standard)
|
||||
- 30% of data 30-90 days old (Nearline)
|
||||
- 20% of data 90-365 days old (Coldline)
|
||||
- 10% of data 365+ days old (Archive)
|
||||
|
||||
```
|
||||
Standard (200TB): 200TB × $0.020/GB × 12 = $48,000/year
|
||||
Nearline (150TB): 150TB × $0.010/GB × 12 = $18,000/year
|
||||
Coldline (100TB): 100TB × $0.004/GB × 12 = $4,800/year
|
||||
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
|
||||
|
||||
Total Storage Cost: $71,520/year
|
||||
Total with Operations: ~$76,500/year
|
||||
Savings: $66,500/year (46%)
|
||||
```
|
||||
|
||||
## Strategy 2: Autoclass (Recommended)
|
||||
|
||||
### Setup: Automatic Class Optimization
|
||||
|
||||
**Best for**: Maximum automation, unpredictable access patterns
|
||||
|
||||
```typescript
|
||||
// Enable Autoclass for automatic tier management
|
||||
await storage.enableAutoclass({
|
||||
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
|
||||
})
|
||||
|
||||
// Benefits:
|
||||
// - Automatically moves objects between classes based on access patterns
|
||||
// - No data retrieval delays (unlike AWS Glacier)
|
||||
// - Transparent tier transitions within 24 hours
|
||||
// - Supports all storage classes including Archive
|
||||
// - No extra monitoring fees
|
||||
```
|
||||
|
||||
### How Autoclass Works
|
||||
|
||||
1. **Initial Placement**: New objects start in Standard class
|
||||
2. **Automatic Demotion**: Objects move to Nearline (30 days) → Coldline (90 days) → Archive (365 days)
|
||||
3. **Automatic Promotion**: Accessed objects move back to Standard class
|
||||
4. **Access-Pattern Learning**: Uses 90-day access history for optimization
|
||||
|
||||
### Cost Calculation (500TB with Autoclass)
|
||||
|
||||
**Realistic distribution after 1 year:**
|
||||
- 10% Standard (hot data, frequently accessed)
|
||||
- 15% Nearline (warm data)
|
||||
- 35% Coldline (cool data)
|
||||
- 40% Archive (cold data)
|
||||
|
||||
```
|
||||
Standard (50TB): 50TB × $0.020/GB × 12 = $12,000/year
|
||||
Nearline (75TB): 75TB × $0.010/GB × 12 = $9,000/year
|
||||
Coldline (175TB): 175TB × $0.004/GB × 12 = $8,400/year
|
||||
Archive (200TB): 200TB × $0.0012/GB × 12 = $2,880/year
|
||||
|
||||
Total Storage Cost: $32,280/year
|
||||
Total with Operations: ~$37,000/year
|
||||
Savings vs Standard: $106,000/year (74%)
|
||||
```
|
||||
|
||||
## Strategy 3: Hybrid Approach (Maximum Savings)
|
||||
|
||||
### Setup: Autoclass + Lifecycle Policies
|
||||
|
||||
```typescript
|
||||
// Enable Autoclass for vectors (frequently searched)
|
||||
await storage.enableAutoclass({
|
||||
terminalStorageClass: 'COLDLINE' // Don't archive vectors deeply
|
||||
})
|
||||
|
||||
// Set lifecycle policy for metadata (less frequently accessed)
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
condition: { age: 30, matchesPrefix: ['entities/nouns/metadata/'] },
|
||||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||||
}, {
|
||||
condition: { age: 60, matchesPrefix: ['entities/nouns/metadata/'] },
|
||||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||||
}, {
|
||||
condition: { age: 180, matchesPrefix: ['entities/nouns/metadata/'] },
|
||||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||||
}, {
|
||||
condition: { age: 730, matchesPrefix: ['_system/'] },
|
||||
action: { type: 'Delete' } // Delete old system files after 2 years
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB Hybrid Approach)
|
||||
|
||||
**Vectors (300TB with Autoclass):**
|
||||
```
|
||||
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
|
||||
Nearline (45TB): 45TB × $0.010/GB × 12 = $5,400/year
|
||||
Coldline (225TB): 225TB × $0.004/GB × 12 = $10,800/year
|
||||
Subtotal: $23,400/year
|
||||
```
|
||||
|
||||
**Metadata (200TB with Lifecycle Policy):**
|
||||
```
|
||||
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
|
||||
Nearline (40TB): 40TB × $0.010/GB × 12 = $4,800/year
|
||||
Coldline (80TB): 80TB × $0.004/GB × 12 = $3,840/year
|
||||
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
|
||||
Subtotal: $16,560/year
|
||||
```
|
||||
|
||||
**Total Cost: $39,960/year + operations (~$45,000/year total)**
|
||||
**Savings vs Standard: $98,000/year (69%)**
|
||||
|
||||
## Strategy 4: Aggressive Archival (Maximum Savings)
|
||||
|
||||
### Setup: Fast Archival for Cold Data
|
||||
|
||||
```typescript
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
condition: { age: 14 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||||
}, {
|
||||
condition: { age: 30 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||||
}, {
|
||||
condition: { age: 90 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||||
}]
|
||||
})
|
||||
|
||||
// Note: Archive class has 365-day minimum storage duration
|
||||
// Early deletion incurs pro-rated charges for remaining days
|
||||
```
|
||||
|
||||
### Cost Calculation (500TB Aggressive Archival)
|
||||
|
||||
**After 1 year:**
|
||||
```
|
||||
Standard (25TB): 25TB × $0.020/GB × 12 = $6,000/year
|
||||
Nearline (50TB): 50TB × $0.010/GB × 12 = $6,000/year
|
||||
Coldline (75TB): 75TB × $0.004/GB × 12 = $3,600/year
|
||||
Archive (350TB): 350TB × $0.0012/GB × 12 = $5,040/year
|
||||
|
||||
Total Storage Cost: $20,640/year
|
||||
Total with Operations: ~$25,500/year
|
||||
Savings: $117,500/year (82%)
|
||||
|
||||
Warning: High retrieval costs if archived data is accessed frequently
|
||||
```
|
||||
|
||||
## Comparison Table: All Strategies
|
||||
|
||||
| Strategy | Annual Cost | Savings | Best For |
|
||||
|----------|-------------|---------|----------|
|
||||
| **No Optimization** | $143,000 | 0% | N/A |
|
||||
| **Lifecycle Policy** | $76,500 | 46% | Predictable patterns |
|
||||
| **Autoclass** | $37,000 | **74%** | **Recommended** |
|
||||
| **Hybrid Approach** | $45,000 | 69% | Fine-grained control |
|
||||
| **Aggressive Archival** | $25,500 | 82% | Cold data, compliance |
|
||||
|
||||
## Autoclass vs Lifecycle Policies
|
||||
|
||||
| Feature | Autoclass | Lifecycle Policies |
|
||||
|---------|-----------|-------------------|
|
||||
| **Automation** | Fully automatic | Rule-based |
|
||||
| **Access-pattern learning** | Yes (90-day history) | No |
|
||||
| **Promotion to Standard** | Automatic on access | Manual only |
|
||||
| **Terminal class** | Configurable | Fixed by rules |
|
||||
| **Complexity** | Single command | Multiple rules |
|
||||
| **Cost** | Lower (smarter) | Moderate |
|
||||
| **Best for** | Unpredictable patterns | Predictable patterns |
|
||||
|
||||
## Batch Delete Operations
|
||||
|
||||
### Efficient Cleanup
|
||||
|
||||
```typescript
|
||||
// Batch delete (100 objects per request for GCS)
|
||||
const idsToDelete = [/* array of entity IDs */]
|
||||
|
||||
const paths = idsToDelete.flatMap(id => {
|
||||
const shard = id.substring(0, 2)
|
||||
return [
|
||||
`entities/nouns/vectors/${shard}/${id}.json`,
|
||||
`entities/nouns/metadata/${shard}/${id}.json`
|
||||
]
|
||||
})
|
||||
|
||||
// Batch delete
|
||||
await storage.batchDelete(paths)
|
||||
|
||||
// Cost impact:
|
||||
// - Individual deletes: 1M operations × $0.005 per 10k = $500
|
||||
// - Batch deletes: 10k batches × $0.005 = $5 (100x cheaper!)
|
||||
```
|
||||
|
||||
## Monitoring and Management
|
||||
|
||||
### Check Autoclass Status
|
||||
|
||||
```typescript
|
||||
const status = await storage.getAutoclassStatus()
|
||||
console.log('Autoclass enabled:', status.enabled)
|
||||
console.log('Terminal class:', status.terminalStorageClass)
|
||||
|
||||
// Example output:
|
||||
// {
|
||||
// enabled: true,
|
||||
// terminalStorageClass: 'ARCHIVE',
|
||||
// toggleTime: '2025-01-15T10:30:00Z'
|
||||
// }
|
||||
```
|
||||
|
||||
### Disable Autoclass
|
||||
|
||||
```typescript
|
||||
// Disable Autoclass (objects remain in current class)
|
||||
await storage.disableAutoclass()
|
||||
```
|
||||
|
||||
### Get Current Lifecycle Policy
|
||||
|
||||
```typescript
|
||||
const policy = await storage.getLifecyclePolicy()
|
||||
console.log('Active rules:', policy.rules)
|
||||
```
|
||||
|
||||
### Remove Lifecycle Policy
|
||||
|
||||
```typescript
|
||||
await storage.removeLifecyclePolicy()
|
||||
```
|
||||
|
||||
## GCS Cloud Console Monitoring
|
||||
|
||||
### Track Your Savings
|
||||
|
||||
1. **Storage Browser** → View storage class distribution
|
||||
2. **Monitoring** → Create custom dashboards for storage metrics
|
||||
3. **Cloud Logging** → Track class transition events
|
||||
4. **Cloud Billing Reports** → Compare storage costs month-over-month
|
||||
|
||||
### Expected Metrics After 6 Months (Autoclass)
|
||||
|
||||
```
|
||||
Standard: 10-15% of total data
|
||||
Nearline: 15-20%
|
||||
Coldline: 30-40%
|
||||
Archive: 30-45%
|
||||
|
||||
Monthly cost trend: Decreasing 8-12% per month as data ages into cheaper classes
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. ✅ **Start with Autoclass** - Simplest and most effective
|
||||
2. ✅ **Set terminal class to ARCHIVE** for maximum savings
|
||||
3. ✅ **Use lifecycle policies for system files** - Predictable archival
|
||||
4. ✅ **Monitor class distribution** monthly in Cloud Console
|
||||
5. ✅ **Use batch operations** for deletions - 100x cheaper
|
||||
6. ✅ **Enable Object Lifecycle Management logging** for auditing
|
||||
7. ✅ **Consider Turbo Replication** for multi-region redundancy
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Data not transitioning to cheaper classes
|
||||
|
||||
**Solution:**
|
||||
```typescript
|
||||
// Check Autoclass status
|
||||
const status = await storage.getAutoclassStatus()
|
||||
if (!status.enabled) {
|
||||
await storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' })
|
||||
}
|
||||
|
||||
// Autoclass requires 24-48 hours for initial transitions
|
||||
```
|
||||
|
||||
### Issue: High retrieval costs
|
||||
|
||||
**Solution:**
|
||||
- GCS has lower retrieval fees than AWS Glacier ($0.01-0.05/GB vs $0.01-0.20/GB)
|
||||
- Autoclass automatically promotes frequently accessed objects to Standard
|
||||
- Use Coldline for occasional access (better than Archive)
|
||||
|
||||
### Issue: Minimum storage duration charges
|
||||
|
||||
**Solution:**
|
||||
- Nearline: 30-day minimum
|
||||
- Coldline: 90-day minimum
|
||||
- Archive: 365-day minimum
|
||||
- Early deletion incurs pro-rated charges
|
||||
- Use Autoclass to avoid manual class changes that might trigger early deletion fees
|
||||
|
||||
## ADC (Application Default Credentials) Setup
|
||||
|
||||
### Production Best Practice
|
||||
|
||||
```typescript
|
||||
// Use ADC instead of service account key file
|
||||
const storage = new GcsStorage({
|
||||
bucketName: 'my-brainy-data'
|
||||
// No keyFilename needed - uses ADC automatically
|
||||
})
|
||||
|
||||
// ADC authentication order:
|
||||
// 1. GOOGLE_APPLICATION_CREDENTIALS environment variable
|
||||
// 2. gcloud CLI credentials
|
||||
// 3. Compute Engine/Cloud Run service account
|
||||
```
|
||||
|
||||
### Set up ADC
|
||||
|
||||
```bash
|
||||
# For local development
|
||||
gcloud auth application-default login
|
||||
|
||||
# For production (use service account)
|
||||
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
|
||||
|
||||
# For Cloud Run/GKE/Compute Engine (automatic)
|
||||
# Service account is automatically available
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
**Recommended Strategy for Most Use Cases:**
|
||||
- **Autoclass** for automatic optimization (simplest, most effective)
|
||||
- **Lifecycle policies** for predictable archival (system files, logs)
|
||||
- **Batch operations** for efficient cleanup
|
||||
|
||||
**Expected Savings:**
|
||||
- **Year 1**: 50-60% reduction in storage costs
|
||||
- **Year 2+**: 70-80% reduction as more data ages into archive classes
|
||||
- **Long-term**: 85-90% reduction for mature datasets
|
||||
|
||||
**500TB Example (Autoclass):**
|
||||
- Before: $143,000/year
|
||||
- After: $37,000/year
|
||||
- **Savings: $106,000/year (74%)**
|
||||
|
||||
**1PB Example (Autoclass):**
|
||||
- Before: $286,000/year
|
||||
- After: $74,000/year
|
||||
- **Savings: $212,000/year (74%)**
|
||||
|
||||
**10PB Example (Autoclass):**
|
||||
- Before: $2,860,000/year
|
||||
- After: $740,000/year
|
||||
- **Savings: $2,120,000/year (74%)**
|
||||
|
||||
## Strategy 5: HNS Buckets for Write-Heavy Workloads
|
||||
|
||||
### When to Use HNS (Hierarchical Namespace)
|
||||
|
||||
If you experience HTTP 429 rate limit errors during `brain.add()` operations (especially with many metadata fields), GCS Hierarchical Namespace buckets provide significantly higher write throughput with zero code changes.
|
||||
|
||||
### Why It Helps
|
||||
|
||||
Standard GCS buckets enforce ~1 write/sec per object path. Brainy's metadata index writes multiple chunk and sparse index files per `brain.add()` call, which can exceed these limits during burst writes.
|
||||
|
||||
HNS buckets provide:
|
||||
- **8x higher initial QPS** (8,000 writes/sec vs 1,000 for standard buckets)
|
||||
- **Better handling of hierarchical key patterns** (which brainy uses for sharded storage)
|
||||
- **No code changes required** — create a new HNS-enabled bucket and point brainy at it
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
# Create HNS-enabled bucket
|
||||
gcloud storage buckets create gs://my-brainy-hns \
|
||||
--location=us-central1 \
|
||||
--uniform-bucket-level-access \
|
||||
--enable-hierarchical-namespace
|
||||
```
|
||||
|
||||
Then configure brainy to use the new bucket:
|
||||
|
||||
```typescript
|
||||
const storage = new GcsStorage({
|
||||
bucketName: 'my-brainy-hns'
|
||||
})
|
||||
```
|
||||
|
||||
### Trade-offs
|
||||
|
||||
- HNS buckets do not support object versioning (irrelevant for brainy — uses its own COW versioning)
|
||||
- HNS is available in most GCS regions
|
||||
- Pricing is the same as standard buckets
|
||||
|
||||
### Alternative: Cloud Run with Filestore/NFS
|
||||
|
||||
For the highest write throughput with zero rate limiting, see the [Cloud Run Filestore Guide](cloud-run-filestore-guide.md) — mount a Filestore NFS volume and use brainy's FileSystem adapter instead of GCS.
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2025-10-17
|
||||
**Cloud Provider**: Google Cloud Storage
|
||||
|
|
@ -628,105 +628,43 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
// Note: LSM-trees will be recreated from storage via their own initialization
|
||||
// Verb data will be loaded on-demand via UnifiedCache
|
||||
|
||||
// Adaptive loading strategy based on storage type
|
||||
// Brainy 8.0: storage is always local (filesystem or memory) per
|
||||
// BR-BRAINY-80-STORAGE-SIMPLIFY. Load all verbs at once.
|
||||
const storageType = this.storage?.constructor.name || ''
|
||||
const isLocalStorage =
|
||||
storageType === 'FileSystemStorage' ||
|
||||
storageType === 'MemoryStorage' ||
|
||||
storageType === 'OPFSStorage'
|
||||
|
||||
let totalVerbs = 0
|
||||
|
||||
if (isLocalStorage) {
|
||||
// Local storage: Load all verbs at once to avoid repeated getAllShardedFiles() calls
|
||||
prodLog.info(
|
||||
`GraphAdjacencyIndex: Using optimized strategy - load all verbs at once (${storageType})`
|
||||
)
|
||||
prodLog.info(`GraphAdjacencyIndex: Load all verbs at once (${storageType})`)
|
||||
|
||||
const result = await this.storage.getVerbs({
|
||||
pagination: { limit: 10000000 } // Effectively unlimited for local development
|
||||
})
|
||||
const result = await this.storage.getVerbs({
|
||||
pagination: { limit: 10000000 } // Effectively unlimited for local storage
|
||||
})
|
||||
|
||||
// Add each verb to index
|
||||
for (const verb of result.items) {
|
||||
// Convert HNSWVerbWithMetadata to GraphVerb format
|
||||
const graphVerb: GraphVerb = {
|
||||
id: verb.id,
|
||||
sourceId: verb.sourceId,
|
||||
targetId: verb.targetId,
|
||||
vector: verb.vector,
|
||||
source: verb.sourceId,
|
||||
target: verb.targetId,
|
||||
verb: verb.verb,
|
||||
createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 },
|
||||
updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 },
|
||||
createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' },
|
||||
service: verb.service,
|
||||
data: verb.data,
|
||||
embedding: verb.vector,
|
||||
confidence: verb.confidence,
|
||||
weight: verb.weight
|
||||
}
|
||||
await this.addVerb(graphVerb)
|
||||
totalVerbs++
|
||||
for (const verb of result.items) {
|
||||
const graphVerb: GraphVerb = {
|
||||
id: verb.id,
|
||||
sourceId: verb.sourceId,
|
||||
targetId: verb.targetId,
|
||||
vector: verb.vector,
|
||||
source: verb.sourceId,
|
||||
target: verb.targetId,
|
||||
verb: verb.verb,
|
||||
createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 },
|
||||
updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 },
|
||||
createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' },
|
||||
service: verb.service,
|
||||
data: verb.data,
|
||||
embedding: verb.vector,
|
||||
confidence: verb.confidence,
|
||||
weight: verb.weight
|
||||
}
|
||||
|
||||
prodLog.info(
|
||||
`GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs at once (local storage)`
|
||||
)
|
||||
} else {
|
||||
// Cloud storage: Use pagination with native cloud APIs (efficient)
|
||||
prodLog.info(
|
||||
`GraphAdjacencyIndex: Using cloud pagination strategy (${storageType})`
|
||||
)
|
||||
|
||||
let hasMore = true
|
||||
let cursor: string | undefined = undefined
|
||||
const batchSize = 1000
|
||||
|
||||
while (hasMore) {
|
||||
const result = await this.storage.getVerbs({
|
||||
pagination: { limit: batchSize, cursor }
|
||||
})
|
||||
|
||||
// Add each verb to index
|
||||
for (const verb of result.items) {
|
||||
// Convert HNSWVerbWithMetadata to GraphVerb format
|
||||
const graphVerb: GraphVerb = {
|
||||
id: verb.id,
|
||||
sourceId: verb.sourceId,
|
||||
targetId: verb.targetId,
|
||||
vector: verb.vector,
|
||||
source: verb.sourceId,
|
||||
target: verb.targetId,
|
||||
verb: verb.verb,
|
||||
createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 },
|
||||
updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 },
|
||||
createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' },
|
||||
service: verb.service,
|
||||
data: verb.data,
|
||||
embedding: verb.vector,
|
||||
confidence: verb.confidence,
|
||||
weight: verb.weight
|
||||
}
|
||||
await this.addVerb(graphVerb)
|
||||
totalVerbs++
|
||||
}
|
||||
|
||||
hasMore = result.hasMore
|
||||
cursor = result.nextCursor
|
||||
|
||||
// Progress logging
|
||||
if (totalVerbs % 10000 === 0) {
|
||||
prodLog.info(`GraphAdjacencyIndex: Indexed ${totalVerbs} verbs...`)
|
||||
}
|
||||
}
|
||||
|
||||
prodLog.info(
|
||||
`GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs via pagination (cloud storage)`
|
||||
)
|
||||
await this.addVerb(graphVerb)
|
||||
totalVerbs++
|
||||
}
|
||||
|
||||
prodLog.info(
|
||||
`GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs (${storageType})`
|
||||
)
|
||||
|
||||
const rebuildTime = Date.now() - this.rebuildStartTime
|
||||
const memoryUsage = this.calculateMemoryUsage()
|
||||
|
||||
|
|
|
|||
|
|
@ -1470,20 +1470,15 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
)
|
||||
}
|
||||
|
||||
// Step 4: Adaptive loading strategy based on storage type
|
||||
// FileSystem/Memory/OPFS: Load all at once (avoids repeated getAllShardedFiles() calls)
|
||||
// Cloud (GCS/S3/R2): Use pagination (efficient native cloud APIs)
|
||||
// Step 4: Load all HNSW nodes at once. Brainy 8.0 ships filesystem +
|
||||
// memory storage only (per BR-BRAINY-80-STORAGE-SIMPLIFY); the
|
||||
// cloud-pagination rebuild path was deleted alongside the cloud adapters.
|
||||
const storageType = this.storage?.constructor.name || ''
|
||||
const isLocalStorage = storageType === 'FileSystemStorage' ||
|
||||
storageType === 'MemoryStorage' ||
|
||||
storageType === 'OPFSStorage'
|
||||
|
||||
let loadedCount = 0
|
||||
let totalCount: number | undefined = undefined
|
||||
|
||||
if (isLocalStorage) {
|
||||
// Local storage: Load all nouns at once
|
||||
prodLog.info(`HNSW: Using optimized strategy - load all nodes at once (${storageType})`)
|
||||
{
|
||||
prodLog.info(`HNSW: Load all nodes at once (${storageType})`)
|
||||
|
||||
const result: {
|
||||
items: HNSWNoun[]
|
||||
|
|
@ -1554,94 +1549,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
options.onProgress(loadedCount, totalCount)
|
||||
}
|
||||
|
||||
prodLog.info(`HNSW: Loaded ${loadedCount.toLocaleString()} nodes at once (local storage)`)
|
||||
|
||||
} else {
|
||||
// Cloud storage: Use pagination with native cloud APIs
|
||||
prodLog.info(`HNSW: Using cloud pagination strategy (${storageType})`)
|
||||
|
||||
let hasMore = true
|
||||
let offset = 0 // Use offset-based pagination instead of cursor (bug fix for infinite loop)
|
||||
|
||||
while (hasMore) {
|
||||
// Fetch batch of nouns from storage (cast needed as method is not in base interface)
|
||||
const result: {
|
||||
items: HNSWNoun[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
} = await (this.storage as any).getNounsWithPagination({
|
||||
limit: batchSize,
|
||||
offset // Pass offset for proper pagination (previously passed cursor which was ignored)
|
||||
})
|
||||
|
||||
// Set total count on first batch
|
||||
if (totalCount === undefined && result.totalCount !== undefined) {
|
||||
totalCount = result.totalCount
|
||||
}
|
||||
|
||||
// Process each noun in the batch
|
||||
for (const nounData of result.items) {
|
||||
try {
|
||||
// Load HNSW graph data for this entity
|
||||
const hnswData = await (this.storage as any).getVectorIndexData(nounData.id)
|
||||
|
||||
if (!hnswData) {
|
||||
// No HNSW data - skip (might be entity added before persistence)
|
||||
continue
|
||||
}
|
||||
|
||||
// Determine if vector should be kept in memory
|
||||
const keepVector = shouldPreload && this.vectorStorageMode !== 'lazy'
|
||||
|
||||
// Create noun object with restored connections
|
||||
const noun: HNSWNoun = {
|
||||
id: nounData.id,
|
||||
vector: keepVector ? nounData.vector : [], // Preload if dataset is small and not lazy
|
||||
connections: new Map(),
|
||||
level: hnswData.level
|
||||
}
|
||||
|
||||
// Restore SQ8 quantized data if quantization is enabled
|
||||
if (this.quantizationEnabled && nounData.vector.length > 0) {
|
||||
const sq8 = quantizeSQ8(nounData.vector)
|
||||
noun.quantizedVector = sq8.quantized
|
||||
noun.codebookMin = sq8.min
|
||||
noun.codebookMax = sq8.max
|
||||
}
|
||||
|
||||
// Restore connections from persisted data — compressed blob path
|
||||
// first, legacy JSON-array fallback for indexes written before
|
||||
// graph link compression landed.
|
||||
await this.restoreNodeConnections(nounData.id, hnswData, noun)
|
||||
|
||||
// Add to in-memory index
|
||||
this.nouns.set(nounData.id, noun)
|
||||
|
||||
// Track high-level nodes for O(1) entry point selection
|
||||
if (noun.level >= 2 && noun.level <= this.MAX_TRACKED_LEVELS) {
|
||||
if (!this.highLevelNodes.has(noun.level)) {
|
||||
this.highLevelNodes.set(noun.level, new Set())
|
||||
}
|
||||
this.highLevelNodes.get(noun.level)!.add(nounData.id)
|
||||
}
|
||||
|
||||
loadedCount++
|
||||
} catch (error) {
|
||||
// Log error but continue (robust error recovery)
|
||||
console.error(`Failed to rebuild HNSW data for ${nounData.id}:`, error)
|
||||
}
|
||||
}
|
||||
|
||||
// Report progress
|
||||
if (options.onProgress && totalCount !== undefined) {
|
||||
options.onProgress(loadedCount, totalCount)
|
||||
}
|
||||
|
||||
// Check for more data
|
||||
hasMore = result.hasMore
|
||||
offset += batchSize // Increment offset for next page
|
||||
}
|
||||
prodLog.info(`HNSW: Loaded ${loadedCount.toLocaleString()} nodes (${storageType})`)
|
||||
}
|
||||
|
||||
// Step 5: CRITICAL - Recover entry point if missing)
|
||||
|
|
|
|||
|
|
@ -3056,21 +3056,13 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
// Clear chunk manager cache
|
||||
this.chunkManager.clearCache()
|
||||
|
||||
// Adaptive rebuild strategy based on storage adapter
|
||||
// FileSystem/Memory/OPFS: Load all at once (avoids getAllShardedFiles() overhead on every batch)
|
||||
// Cloud (GCS/S3/R2): Use pagination with small batches (prevent socket exhaustion)
|
||||
const storageType = this.storage.constructor.name
|
||||
const isLocalStorage = storageType === 'FileSystemStorage' ||
|
||||
storageType === 'MemoryStorage' ||
|
||||
storageType === 'OPFSStorage'
|
||||
|
||||
let nounLimit: number
|
||||
// Brainy 8.0 ships filesystem + memory storage only. Load all nouns
|
||||
// at once — the cloud-storage paginated branch was deleted alongside
|
||||
// the cloud adapters in step 7.
|
||||
let totalNounsProcessed = 0
|
||||
|
||||
if (isLocalStorage) {
|
||||
// Load all nouns at once for local storage
|
||||
// Avoids repeated directory scans in getAllShardedFiles()
|
||||
prodLog.info(`⚡ Using optimized strategy: load all nouns at once (local storage)`)
|
||||
{
|
||||
prodLog.info(`⚡ Loading all nouns at once (local storage)`)
|
||||
const result = await this.storage.getNouns({
|
||||
pagination: { offset: 0, limit: 1000000 } // Effectively unlimited
|
||||
})
|
||||
|
|
@ -3085,7 +3077,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
metadataBatch = await this.storage.getMetadataBatch(nounIds)
|
||||
prodLog.info(`✅ Loaded ${metadataBatch.size}/${nounIds.length} metadata objects`)
|
||||
} else {
|
||||
// Fallback to individual calls
|
||||
metadataBatch = new Map()
|
||||
for (const id of nounIds) {
|
||||
try {
|
||||
|
|
@ -3097,131 +3088,21 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
}
|
||||
}
|
||||
|
||||
// Process all nouns
|
||||
let localCount = 0
|
||||
for (const noun of result.items) {
|
||||
const metadata = metadataBatch.get(noun.id)
|
||||
if (metadata) {
|
||||
await this.addToIndex(noun.id, metadata, true, true)
|
||||
localCount++
|
||||
// Periodic safety flush every 5000 entities to cap memory during rebuild
|
||||
// (Periodic safety flush removed in 7.22.0 — the underlying
|
||||
// flushDirtyMetadata was a no-op since the 7.20.0 column-store
|
||||
// refactor. Column store handles its own flushing.)
|
||||
}
|
||||
}
|
||||
|
||||
totalNounsProcessed = result.items.length
|
||||
prodLog.info(`✅ Indexed ${totalNounsProcessed} nouns`)
|
||||
|
||||
} else {
|
||||
// Cloud storage: use conservative batching
|
||||
nounLimit = 25
|
||||
prodLog.info(`⚡ Using conservative batch size: ${nounLimit} items/batch (cloud storage)`)
|
||||
|
||||
let nounOffset = 0
|
||||
let hasMoreNouns = true
|
||||
let consecutiveEmptyBatches = 0
|
||||
const MAX_ITERATIONS = 10000
|
||||
let iterations = 0
|
||||
|
||||
while (hasMoreNouns && iterations < MAX_ITERATIONS) {
|
||||
iterations++
|
||||
const result = await this.storage.getNouns({
|
||||
pagination: { offset: nounOffset, limit: nounLimit }
|
||||
})
|
||||
|
||||
// CRITICAL SAFETY CHECK: Prevent infinite loop on empty results
|
||||
if (result.items.length === 0) {
|
||||
consecutiveEmptyBatches++
|
||||
if (consecutiveEmptyBatches >= 3) {
|
||||
prodLog.warn('⚠️ Breaking metadata rebuild loop: received 3 consecutive empty batches')
|
||||
break
|
||||
}
|
||||
// If hasMore is true but items are empty, it's likely a bug
|
||||
if (result.hasMore) {
|
||||
prodLog.warn(`⚠️ Storage returned empty items but hasMore=true at offset ${nounOffset}`)
|
||||
hasMoreNouns = false // Force exit
|
||||
break
|
||||
}
|
||||
} else {
|
||||
consecutiveEmptyBatches = 0 // Reset counter on non-empty batch
|
||||
}
|
||||
|
||||
// CRITICAL FIX: Use batch metadata reading to prevent socket exhaustion
|
||||
const nounIds = result.items.map(noun => noun.id)
|
||||
|
||||
let metadataBatch: Map<string, any>
|
||||
if (this.storage.getMetadataBatch) {
|
||||
// Use batch reading if available (prevents socket exhaustion)
|
||||
prodLog.info(`📦 Processing metadata batch ${Math.floor(totalNounsProcessed / nounLimit) + 1} (${nounIds.length} items)...`)
|
||||
metadataBatch = await this.storage.getMetadataBatch(nounIds)
|
||||
const successRate = ((metadataBatch.size / nounIds.length) * 100).toFixed(1)
|
||||
prodLog.info(`✅ Batch loaded ${metadataBatch.size}/${nounIds.length} metadata objects (${successRate}% success)`)
|
||||
} else {
|
||||
// Fallback to individual calls with strict concurrency control
|
||||
prodLog.warn(`⚠️ FALLBACK: Storage adapter missing getMetadataBatch - using individual calls with concurrency limit`)
|
||||
metadataBatch = new Map()
|
||||
const CONCURRENCY_LIMIT = 3 // Very conservative limit
|
||||
|
||||
for (let i = 0; i < nounIds.length; i += CONCURRENCY_LIMIT) {
|
||||
const batch = nounIds.slice(i, i + CONCURRENCY_LIMIT)
|
||||
const batchPromises = batch.map(async (id) => {
|
||||
try {
|
||||
const metadata = await this.storage.getNounMetadata(id)
|
||||
return { id, metadata }
|
||||
} catch (error) {
|
||||
prodLog.debug(`Failed to read metadata for ${id}:`, error)
|
||||
return { id, metadata: null }
|
||||
}
|
||||
})
|
||||
|
||||
const batchResults = await Promise.all(batchPromises)
|
||||
for (const { id, metadata } of batchResults) {
|
||||
if (metadata) {
|
||||
metadataBatch.set(id, metadata)
|
||||
}
|
||||
}
|
||||
|
||||
// Yield between batches to prevent socket exhaustion
|
||||
await this.yieldToEventLoop()
|
||||
}
|
||||
}
|
||||
|
||||
// Process the metadata batch
|
||||
for (const noun of result.items) {
|
||||
const metadata = metadataBatch.get(noun.id)
|
||||
if (metadata) {
|
||||
// Skip flush during rebuild for performance
|
||||
await this.addToIndex(noun.id, metadata, true, true)
|
||||
}
|
||||
}
|
||||
|
||||
// Yield after processing the entire batch
|
||||
await this.yieldToEventLoop()
|
||||
|
||||
totalNounsProcessed += result.items.length
|
||||
hasMoreNouns = result.hasMore
|
||||
nounOffset += nounLimit
|
||||
|
||||
// Progress logging and event loop yield after each batch
|
||||
if (totalNounsProcessed % 100 === 0 || !hasMoreNouns) {
|
||||
prodLog.debug(`📊 Indexed ${totalNounsProcessed} nouns...`)
|
||||
}
|
||||
await this.yieldToEventLoop()
|
||||
}
|
||||
|
||||
// Check iteration limits for cloud storage
|
||||
if (iterations >= MAX_ITERATIONS) {
|
||||
prodLog.error(`❌ Metadata noun rebuild hit maximum iteration limit (${MAX_ITERATIONS}). This indicates a bug in storage pagination.`)
|
||||
}
|
||||
}
|
||||
|
||||
// Rebuild verb metadata indexes - same strategy as nouns
|
||||
// Rebuild verb metadata indexes — same single-pass local strategy.
|
||||
let totalVerbsProcessed = 0
|
||||
|
||||
if (isLocalStorage) {
|
||||
// Load all verbs at once for local storage
|
||||
{
|
||||
prodLog.info(`⚡ Loading all verbs at once (local storage)`)
|
||||
const result = await this.storage.getVerbs({
|
||||
pagination: { offset: 0, limit: 1000000 } // Effectively unlimited
|
||||
|
|
@ -3229,7 +3110,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
|
||||
prodLog.info(`📦 Loading ${result.items.length} verbs with metadata...`)
|
||||
|
||||
// Get all verb metadata at once
|
||||
const verbIds = result.items.map(verb => verb.id)
|
||||
let verbMetadataBatch: Map<string, any>
|
||||
|
||||
|
|
@ -3248,126 +3128,23 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
|||
}
|
||||
}
|
||||
|
||||
// Process all verbs
|
||||
let verbLocalCount = 0
|
||||
for (const verb of result.items) {
|
||||
const metadata = verbMetadataBatch.get(verb.id)
|
||||
if (metadata) {
|
||||
await this.addToIndex(verb.id, metadata, true, true)
|
||||
verbLocalCount++
|
||||
// (Periodic safety flush removed in 7.22.0 — see noun loop above.)
|
||||
}
|
||||
}
|
||||
|
||||
totalVerbsProcessed = result.items.length
|
||||
prodLog.info(`✅ Indexed ${totalVerbsProcessed} verbs`)
|
||||
|
||||
} else {
|
||||
// Cloud storage: use conservative batching
|
||||
let verbOffset = 0
|
||||
const verbLimit = 25
|
||||
let hasMoreVerbs = true
|
||||
let consecutiveEmptyVerbBatches = 0
|
||||
let verbIterations = 0
|
||||
const MAX_ITERATIONS = 10000
|
||||
|
||||
while (hasMoreVerbs && verbIterations < MAX_ITERATIONS) {
|
||||
verbIterations++
|
||||
const result = await this.storage.getVerbs({
|
||||
pagination: { offset: verbOffset, limit: verbLimit }
|
||||
})
|
||||
|
||||
// CRITICAL SAFETY CHECK: Prevent infinite loop on empty results
|
||||
if (result.items.length === 0) {
|
||||
consecutiveEmptyVerbBatches++
|
||||
if (consecutiveEmptyVerbBatches >= 3) {
|
||||
prodLog.warn('⚠️ Breaking verb metadata rebuild loop: received 3 consecutive empty batches')
|
||||
break
|
||||
}
|
||||
// If hasMore is true but items are empty, it's likely a bug
|
||||
if (result.hasMore) {
|
||||
prodLog.warn(`⚠️ Storage returned empty verb items but hasMore=true at offset ${verbOffset}`)
|
||||
hasMoreVerbs = false // Force exit
|
||||
break
|
||||
}
|
||||
} else {
|
||||
consecutiveEmptyVerbBatches = 0 // Reset counter on non-empty batch
|
||||
}
|
||||
|
||||
// CRITICAL FIX: Use batch verb metadata reading to prevent socket exhaustion
|
||||
const verbIds = result.items.map(verb => verb.id)
|
||||
|
||||
let verbMetadataBatch: Map<string, any>
|
||||
if ((this.storage as any).getVerbMetadataBatch) {
|
||||
// Use batch reading if available (prevents socket exhaustion)
|
||||
verbMetadataBatch = await (this.storage as any).getVerbMetadataBatch(verbIds)
|
||||
prodLog.debug(`📦 Batch loaded ${verbMetadataBatch.size}/${verbIds.length} verb metadata objects`)
|
||||
} else {
|
||||
// Fallback to individual calls with strict concurrency control
|
||||
verbMetadataBatch = new Map()
|
||||
const CONCURRENCY_LIMIT = 3 // Very conservative limit to prevent socket exhaustion
|
||||
|
||||
for (let i = 0; i < verbIds.length; i += CONCURRENCY_LIMIT) {
|
||||
const batch = verbIds.slice(i, i + CONCURRENCY_LIMIT)
|
||||
const batchPromises = batch.map(async (id) => {
|
||||
try {
|
||||
const metadata = await this.storage.getVerbMetadata(id)
|
||||
return { id, metadata }
|
||||
} catch (error) {
|
||||
prodLog.debug(`Failed to read verb metadata for ${id}:`, error)
|
||||
return { id, metadata: null }
|
||||
}
|
||||
})
|
||||
|
||||
const batchResults = await Promise.all(batchPromises)
|
||||
for (const { id, metadata } of batchResults) {
|
||||
if (metadata) {
|
||||
verbMetadataBatch.set(id, metadata)
|
||||
}
|
||||
}
|
||||
|
||||
// Yield between batches to prevent socket exhaustion
|
||||
await this.yieldToEventLoop()
|
||||
}
|
||||
}
|
||||
|
||||
// Process the verb metadata batch
|
||||
for (const verb of result.items) {
|
||||
const metadata = verbMetadataBatch.get(verb.id)
|
||||
if (metadata) {
|
||||
// Skip flush during rebuild for performance
|
||||
await this.addToIndex(verb.id, metadata, true, true)
|
||||
}
|
||||
}
|
||||
|
||||
// Yield after processing the entire batch
|
||||
await this.yieldToEventLoop()
|
||||
|
||||
totalVerbsProcessed += result.items.length
|
||||
hasMoreVerbs = result.hasMore
|
||||
verbOffset += verbLimit
|
||||
|
||||
// Progress logging and event loop yield after each batch
|
||||
if (totalVerbsProcessed % 100 === 0 || !hasMoreVerbs) {
|
||||
prodLog.debug(`🔗 Indexed ${totalVerbsProcessed} verbs...`)
|
||||
}
|
||||
await this.yieldToEventLoop()
|
||||
}
|
||||
|
||||
// Check iteration limits for cloud storage
|
||||
if (verbIterations >= MAX_ITERATIONS) {
|
||||
prodLog.error(`❌ Metadata verb rebuild hit maximum iteration limit (${MAX_ITERATIONS}). This indicates a bug in storage pagination.`)
|
||||
}
|
||||
}
|
||||
|
||||
// Flush to storage with final yield. The column store's flush() handles
|
||||
// tail-buffer-to-segment promotion + manifest persistence.
|
||||
// Flush to storage. The column store's flush() handles tail-buffer-to-
|
||||
// segment promotion + manifest persistence.
|
||||
prodLog.debug('💾 Flushing metadata index to storage...')
|
||||
await this.flush()
|
||||
await this.yieldToEventLoop()
|
||||
|
||||
prodLog.info(`✅ Metadata index rebuild completed! Processed ${totalNounsProcessed} nouns and ${totalVerbsProcessed} verbs`)
|
||||
prodLog.info(`🎯 Initial indexing may show minor socket timeouts - this is expected and doesn't affect data processing`)
|
||||
|
||||
} finally {
|
||||
this.isRebuilding = false
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue