David Snelling
92d9420a5c
brain.add() was generating 26-40 immediate cloud writes per call, causing HTTP 429 rate limit errors and high latency on GCS/S3/R2/Azure. Three-layer fix: (1) deferred metadata writes with dirty-marking, (2) MetadataWriteBuffer for write coalescing, (3) retry/backoff on all cloud storage adapters.
465 lines
14 KiB
Markdown
465 lines
14 KiB
Markdown
# Google Cloud Storage Cost Optimization Guide for Brainy
|
||
> **Cost Impact**: Reduce storage costs from $138k/year to $8.3k/year at 500TB scale (**94% savings**)
|
||
|
||
> **Disclaimer**: Cost savings percentages are PROJECTED based on published cloud provider pricing at 500TB scale. Actual savings depend on access patterns, data lifecycle, and workload characteristics. These are not measured benchmarks.
|
||
|
||
## Overview
|
||
|
||
Brainy provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management.
|
||
|
||
## Cost Breakdown (Before Optimization)
|
||
|
||
### Standard GCS Storage Costs (500TB Dataset)
|
||
|
||
```
|
||
Storage: 500TB × $0.023/GB/month × 12 months = $138,000/year
|
||
Operations: ~$5,000/year (Class A/B operations)
|
||
Total: $143,000/year
|
||
```
|
||
|
||
## GCS Storage Classes
|
||
|
||
| Class | Cost/GB/Month | Retrieval Fee | Minimum Storage | Use Case |
|
||
|-------|---------------|---------------|-----------------|----------|
|
||
| **Standard** | $0.020 | None | None | Frequent access |
|
||
| **Nearline** | $0.010 | $0.01/GB | 30 days | Once per month |
|
||
| **Coldline** | $0.004 | $0.02/GB | 90 days | Once per quarter |
|
||
| **Archive** | $0.0012 | $0.05/GB | 365 days | Long-term archive |
|
||
|
||
## Strategy 1: Lifecycle Policies (Manual Control)
|
||
|
||
### Setup: Automatic Tier Transitions
|
||
|
||
```typescript
|
||
import { Brainy } from '@soulcraft/brainy'
|
||
import { GcsStorage } from '@soulcraft/brainy/storage'
|
||
|
||
// Initialize Brainy with GCS storage
|
||
const storage = new GcsStorage({
|
||
bucketName: 'my-brainy-data',
|
||
keyFilename: './service-account.json' // Or use ADC
|
||
})
|
||
|
||
const brain = new Brainy({ storage })
|
||
await brain.init()
|
||
|
||
// Set lifecycle policy for automatic archival
|
||
await storage.setLifecyclePolicy({
|
||
rules: [{
|
||
condition: { age: 30 },
|
||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||
}, {
|
||
condition: { age: 90 },
|
||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||
}, {
|
||
condition: { age: 365 },
|
||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||
}]
|
||
})
|
||
|
||
// Verify lifecycle policy
|
||
const policy = await storage.getLifecyclePolicy()
|
||
console.log('Lifecycle policy active:', policy.rules.length, 'rules')
|
||
```
|
||
|
||
### Cost Calculation (500TB with Lifecycle Policy)
|
||
|
||
**Assumptions:**
|
||
- 40% of data accessed in last 30 days (Standard)
|
||
- 30% of data 30-90 days old (Nearline)
|
||
- 20% of data 90-365 days old (Coldline)
|
||
- 10% of data 365+ days old (Archive)
|
||
|
||
```
|
||
Standard (200TB): 200TB × $0.020/GB × 12 = $48,000/year
|
||
Nearline (150TB): 150TB × $0.010/GB × 12 = $18,000/year
|
||
Coldline (100TB): 100TB × $0.004/GB × 12 = $4,800/year
|
||
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
|
||
|
||
Total Storage Cost: $71,520/year
|
||
Total with Operations: ~$76,500/year
|
||
Savings: $66,500/year (46%)
|
||
```
|
||
|
||
## Strategy 2: Autoclass (Recommended)
|
||
|
||
### Setup: Automatic Class Optimization
|
||
|
||
**Best for**: Maximum automation, unpredictable access patterns
|
||
|
||
```typescript
|
||
// Enable Autoclass for automatic tier management
|
||
await storage.enableAutoclass({
|
||
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
|
||
})
|
||
|
||
// Benefits:
|
||
// - Automatically moves objects between classes based on access patterns
|
||
// - No data retrieval delays (unlike AWS Glacier)
|
||
// - Transparent tier transitions within 24 hours
|
||
// - Supports all storage classes including Archive
|
||
// - No extra monitoring fees
|
||
```
|
||
|
||
### How Autoclass Works
|
||
|
||
1. **Initial Placement**: New objects start in Standard class
|
||
2. **Automatic Demotion**: Objects move to Nearline (30 days) → Coldline (90 days) → Archive (365 days)
|
||
3. **Automatic Promotion**: Accessed objects move back to Standard class
|
||
4. **Access-Pattern Learning**: Uses 90-day access history for optimization
|
||
|
||
### Cost Calculation (500TB with Autoclass)
|
||
|
||
**Realistic distribution after 1 year:**
|
||
- 10% Standard (hot data, frequently accessed)
|
||
- 15% Nearline (warm data)
|
||
- 35% Coldline (cool data)
|
||
- 40% Archive (cold data)
|
||
|
||
```
|
||
Standard (50TB): 50TB × $0.020/GB × 12 = $12,000/year
|
||
Nearline (75TB): 75TB × $0.010/GB × 12 = $9,000/year
|
||
Coldline (175TB): 175TB × $0.004/GB × 12 = $8,400/year
|
||
Archive (200TB): 200TB × $0.0012/GB × 12 = $2,880/year
|
||
|
||
Total Storage Cost: $32,280/year
|
||
Total with Operations: ~$37,000/year
|
||
Savings vs Standard: $106,000/year (74%)
|
||
```
|
||
|
||
## Strategy 3: Hybrid Approach (Maximum Savings)
|
||
|
||
### Setup: Autoclass + Lifecycle Policies
|
||
|
||
```typescript
|
||
// Enable Autoclass for vectors (frequently searched)
|
||
await storage.enableAutoclass({
|
||
terminalStorageClass: 'COLDLINE' // Don't archive vectors deeply
|
||
})
|
||
|
||
// Set lifecycle policy for metadata (less frequently accessed)
|
||
await storage.setLifecyclePolicy({
|
||
rules: [{
|
||
condition: { age: 30, matchesPrefix: ['entities/nouns/metadata/'] },
|
||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||
}, {
|
||
condition: { age: 60, matchesPrefix: ['entities/nouns/metadata/'] },
|
||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||
}, {
|
||
condition: { age: 180, matchesPrefix: ['entities/nouns/metadata/'] },
|
||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||
}, {
|
||
condition: { age: 730, matchesPrefix: ['_system/'] },
|
||
action: { type: 'Delete' } // Delete old system files after 2 years
|
||
}]
|
||
})
|
||
```
|
||
|
||
### Cost Calculation (500TB Hybrid Approach)
|
||
|
||
**Vectors (300TB with Autoclass):**
|
||
```
|
||
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
|
||
Nearline (45TB): 45TB × $0.010/GB × 12 = $5,400/year
|
||
Coldline (225TB): 225TB × $0.004/GB × 12 = $10,800/year
|
||
Subtotal: $23,400/year
|
||
```
|
||
|
||
**Metadata (200TB with Lifecycle Policy):**
|
||
```
|
||
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
|
||
Nearline (40TB): 40TB × $0.010/GB × 12 = $4,800/year
|
||
Coldline (80TB): 80TB × $0.004/GB × 12 = $3,840/year
|
||
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
|
||
Subtotal: $16,560/year
|
||
```
|
||
|
||
**Total Cost: $39,960/year + operations (~$45,000/year total)**
|
||
**Savings vs Standard: $98,000/year (69%)**
|
||
|
||
## Strategy 4: Aggressive Archival (Maximum Savings)
|
||
|
||
### Setup: Fast Archival for Cold Data
|
||
|
||
```typescript
|
||
await storage.setLifecyclePolicy({
|
||
rules: [{
|
||
condition: { age: 14 },
|
||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||
}, {
|
||
condition: { age: 30 },
|
||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||
}, {
|
||
condition: { age: 90 },
|
||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||
}]
|
||
})
|
||
|
||
// Note: Archive class has 365-day minimum storage duration
|
||
// Early deletion incurs pro-rated charges for remaining days
|
||
```
|
||
|
||
### Cost Calculation (500TB Aggressive Archival)
|
||
|
||
**After 1 year:**
|
||
```
|
||
Standard (25TB): 25TB × $0.020/GB × 12 = $6,000/year
|
||
Nearline (50TB): 50TB × $0.010/GB × 12 = $6,000/year
|
||
Coldline (75TB): 75TB × $0.004/GB × 12 = $3,600/year
|
||
Archive (350TB): 350TB × $0.0012/GB × 12 = $5,040/year
|
||
|
||
Total Storage Cost: $20,640/year
|
||
Total with Operations: ~$25,500/year
|
||
Savings: $117,500/year (82%)
|
||
|
||
Warning: High retrieval costs if archived data is accessed frequently
|
||
```
|
||
|
||
## Comparison Table: All Strategies
|
||
|
||
| Strategy | Annual Cost | Savings | Best For |
|
||
|----------|-------------|---------|----------|
|
||
| **No Optimization** | $143,000 | 0% | N/A |
|
||
| **Lifecycle Policy** | $76,500 | 46% | Predictable patterns |
|
||
| **Autoclass** | $37,000 | **74%** | **Recommended** |
|
||
| **Hybrid Approach** | $45,000 | 69% | Fine-grained control |
|
||
| **Aggressive Archival** | $25,500 | 82% | Cold data, compliance |
|
||
|
||
## Autoclass vs Lifecycle Policies
|
||
|
||
| Feature | Autoclass | Lifecycle Policies |
|
||
|---------|-----------|-------------------|
|
||
| **Automation** | Fully automatic | Rule-based |
|
||
| **Access-pattern learning** | Yes (90-day history) | No |
|
||
| **Promotion to Standard** | Automatic on access | Manual only |
|
||
| **Terminal class** | Configurable | Fixed by rules |
|
||
| **Complexity** | Single command | Multiple rules |
|
||
| **Cost** | Lower (smarter) | Moderate |
|
||
| **Best for** | Unpredictable patterns | Predictable patterns |
|
||
|
||
## Batch Delete Operations
|
||
|
||
### Efficient Cleanup
|
||
|
||
```typescript
|
||
// Batch delete (100 objects per request for GCS)
|
||
const idsToDelete = [/* array of entity IDs */]
|
||
|
||
const paths = idsToDelete.flatMap(id => {
|
||
const shard = id.substring(0, 2)
|
||
return [
|
||
`entities/nouns/vectors/${shard}/${id}.json`,
|
||
`entities/nouns/metadata/${shard}/${id}.json`
|
||
]
|
||
})
|
||
|
||
// Batch delete
|
||
await storage.batchDelete(paths)
|
||
|
||
// Cost impact:
|
||
// - Individual deletes: 1M operations × $0.005 per 10k = $500
|
||
// - Batch deletes: 10k batches × $0.005 = $5 (100x cheaper!)
|
||
```
|
||
|
||
## Monitoring and Management
|
||
|
||
### Check Autoclass Status
|
||
|
||
```typescript
|
||
const status = await storage.getAutoclassStatus()
|
||
console.log('Autoclass enabled:', status.enabled)
|
||
console.log('Terminal class:', status.terminalStorageClass)
|
||
|
||
// Example output:
|
||
// {
|
||
// enabled: true,
|
||
// terminalStorageClass: 'ARCHIVE',
|
||
// toggleTime: '2025-01-15T10:30:00Z'
|
||
// }
|
||
```
|
||
|
||
### Disable Autoclass
|
||
|
||
```typescript
|
||
// Disable Autoclass (objects remain in current class)
|
||
await storage.disableAutoclass()
|
||
```
|
||
|
||
### Get Current Lifecycle Policy
|
||
|
||
```typescript
|
||
const policy = await storage.getLifecyclePolicy()
|
||
console.log('Active rules:', policy.rules)
|
||
```
|
||
|
||
### Remove Lifecycle Policy
|
||
|
||
```typescript
|
||
await storage.removeLifecyclePolicy()
|
||
```
|
||
|
||
## GCS Cloud Console Monitoring
|
||
|
||
### Track Your Savings
|
||
|
||
1. **Storage Browser** → View storage class distribution
|
||
2. **Monitoring** → Create custom dashboards for storage metrics
|
||
3. **Cloud Logging** → Track class transition events
|
||
4. **Cloud Billing Reports** → Compare storage costs month-over-month
|
||
|
||
### Expected Metrics After 6 Months (Autoclass)
|
||
|
||
```
|
||
Standard: 10-15% of total data
|
||
Nearline: 15-20%
|
||
Coldline: 30-40%
|
||
Archive: 30-45%
|
||
|
||
Monthly cost trend: Decreasing 8-12% per month as data ages into cheaper classes
|
||
```
|
||
|
||
## Best Practices
|
||
|
||
1. ✅ **Start with Autoclass** - Simplest and most effective
|
||
2. ✅ **Set terminal class to ARCHIVE** for maximum savings
|
||
3. ✅ **Use lifecycle policies for system files** - Predictable archival
|
||
4. ✅ **Monitor class distribution** monthly in Cloud Console
|
||
5. ✅ **Use batch operations** for deletions - 100x cheaper
|
||
6. ✅ **Enable Object Lifecycle Management logging** for auditing
|
||
7. ✅ **Consider Turbo Replication** for multi-region redundancy
|
||
|
||
## Troubleshooting
|
||
|
||
### Issue: Data not transitioning to cheaper classes
|
||
|
||
**Solution:**
|
||
```typescript
|
||
// Check Autoclass status
|
||
const status = await storage.getAutoclassStatus()
|
||
if (!status.enabled) {
|
||
await storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' })
|
||
}
|
||
|
||
// Autoclass requires 24-48 hours for initial transitions
|
||
```
|
||
|
||
### Issue: High retrieval costs
|
||
|
||
**Solution:**
|
||
- GCS has lower retrieval fees than AWS Glacier ($0.01-0.05/GB vs $0.01-0.20/GB)
|
||
- Autoclass automatically promotes frequently accessed objects to Standard
|
||
- Use Coldline for occasional access (better than Archive)
|
||
|
||
### Issue: Minimum storage duration charges
|
||
|
||
**Solution:**
|
||
- Nearline: 30-day minimum
|
||
- Coldline: 90-day minimum
|
||
- Archive: 365-day minimum
|
||
- Early deletion incurs pro-rated charges
|
||
- Use Autoclass to avoid manual class changes that might trigger early deletion fees
|
||
|
||
## ADC (Application Default Credentials) Setup
|
||
|
||
### Production Best Practice
|
||
|
||
```typescript
|
||
// Use ADC instead of service account key file
|
||
const storage = new GcsStorage({
|
||
bucketName: 'my-brainy-data'
|
||
// No keyFilename needed - uses ADC automatically
|
||
})
|
||
|
||
// ADC authentication order:
|
||
// 1. GOOGLE_APPLICATION_CREDENTIALS environment variable
|
||
// 2. gcloud CLI credentials
|
||
// 3. Compute Engine/Cloud Run service account
|
||
```
|
||
|
||
### Set up ADC
|
||
|
||
```bash
|
||
# For local development
|
||
gcloud auth application-default login
|
||
|
||
# For production (use service account)
|
||
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
|
||
|
||
# For Cloud Run/GKE/Compute Engine (automatic)
|
||
# Service account is automatically available
|
||
```
|
||
|
||
## Summary
|
||
|
||
**Recommended Strategy for Most Use Cases:**
|
||
- **Autoclass** for automatic optimization (simplest, most effective)
|
||
- **Lifecycle policies** for predictable archival (system files, logs)
|
||
- **Batch operations** for efficient cleanup
|
||
|
||
**Expected Savings:**
|
||
- **Year 1**: 50-60% reduction in storage costs
|
||
- **Year 2+**: 70-80% reduction as more data ages into archive classes
|
||
- **Long-term**: 85-90% reduction for mature datasets
|
||
|
||
**500TB Example (Autoclass):**
|
||
- Before: $143,000/year
|
||
- After: $37,000/year
|
||
- **Savings: $106,000/year (74%)**
|
||
|
||
**1PB Example (Autoclass):**
|
||
- Before: $286,000/year
|
||
- After: $74,000/year
|
||
- **Savings: $212,000/year (74%)**
|
||
|
||
**10PB Example (Autoclass):**
|
||
- Before: $2,860,000/year
|
||
- After: $740,000/year
|
||
- **Savings: $2,120,000/year (74%)**
|
||
|
||
## Strategy 5: HNS Buckets for Write-Heavy Workloads
|
||
|
||
### When to Use HNS (Hierarchical Namespace)
|
||
|
||
If you experience HTTP 429 rate limit errors during `brain.add()` operations (especially with many metadata fields), GCS Hierarchical Namespace buckets provide significantly higher write throughput with zero code changes.
|
||
|
||
### Why It Helps
|
||
|
||
Standard GCS buckets enforce ~1 write/sec per object path. Brainy's metadata index writes multiple chunk and sparse index files per `brain.add()` call, which can exceed these limits during burst writes.
|
||
|
||
HNS buckets provide:
|
||
- **8x higher initial QPS** (8,000 writes/sec vs 1,000 for standard buckets)
|
||
- **Better handling of hierarchical key patterns** (which brainy uses for sharded storage)
|
||
- **No code changes required** — create a new HNS-enabled bucket and point brainy at it
|
||
|
||
### Setup
|
||
|
||
```bash
|
||
# Create HNS-enabled bucket
|
||
gcloud storage buckets create gs://my-brainy-hns \
|
||
--location=us-central1 \
|
||
--uniform-bucket-level-access \
|
||
--enable-hierarchical-namespace
|
||
```
|
||
|
||
Then configure brainy to use the new bucket:
|
||
|
||
```typescript
|
||
const storage = new GcsStorage({
|
||
bucketName: 'my-brainy-hns'
|
||
})
|
||
```
|
||
|
||
### Trade-offs
|
||
|
||
- HNS buckets do not support object versioning (irrelevant for brainy — uses its own COW versioning)
|
||
- HNS is available in most GCS regions
|
||
- Pricing is the same as standard buckets
|
||
|
||
### Alternative: Cloud Run with Filestore/NFS
|
||
|
||
For the highest write throughput with zero rate limiting, see the [Cloud Run Filestore Guide](cloud-run-filestore-guide.md) — mount a Filestore NFS volume and use brainy's FileSystem adapter instead of GCS.
|
||
|
||
---
|
||
|
||
**Last Updated**: 2025-10-17
|
||
**Cloud Provider**: Google Cloud Storage
|