docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
parent
2626ab8d62
commit
adda1570f3
22 changed files with 570 additions and 2658 deletions
|
|
@ -852,7 +852,7 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
|
|||
## Level 5: Production Scale (90 minutes)
|
||||
|
||||
### What You'll Learn
|
||||
- Cloud storage (GCS, S3, R2)
|
||||
- Production filesystem storage and off-site backup
|
||||
- Performance optimization
|
||||
- Batch imports (CSV, Excel, PDF)
|
||||
- Metadata query optimization
|
||||
|
|
@ -863,18 +863,13 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
|
|||
```typescript
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
// 1. PRODUCTION STORAGE - Google Cloud Storage (Native SDK)
|
||||
console.log('☁️ Initializing production storage...\n')
|
||||
// 1. PRODUCTION STORAGE - Filesystem with off-site snapshots
|
||||
console.log('Initializing production storage...\n')
|
||||
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs-native', // Native GCS SDK (recommended)
|
||||
gcsNativeStorage: {
|
||||
bucketName: 'my-brainy-production',
|
||||
// ADC (Application Default Credentials) - zero config in Cloud Run/GCE!
|
||||
// Or provide credentials:
|
||||
// keyFilename: '/path/to/service-account.json'
|
||||
}
|
||||
type: 'filesystem',
|
||||
rootDirectory: '/var/lib/brainy'
|
||||
},
|
||||
|
||||
// Performance tuning
|
||||
|
|
@ -888,7 +883,8 @@ const brain = new Brainy({
|
|||
})
|
||||
|
||||
await brain.init()
|
||||
console.log('✅ Brainy initialized with GCS Native storage\n')
|
||||
console.log('Brainy initialized with filesystem storage')
|
||||
console.log('Snapshot /var/lib/brainy off-site via cron with `gsutil rsync` / `aws s3 sync` / `rclone`.\n')
|
||||
|
||||
// 2. BATCH IMPORT - CSV File
|
||||
console.log('📊 Importing CSV data...\n')
|
||||
|
|
@ -1044,7 +1040,7 @@ console.log('✅ Brain closed cleanly')
|
|||
|
||||
console.log('\n\n🎓 Production Deployment Complete!')
|
||||
console.log('\n📚 Key Production Learnings:')
|
||||
console.log(' 1. Use native cloud storage (GCS, S3, R2) for persistence')
|
||||
console.log(' 1. Use filesystem storage and snapshot rootDirectory off-site from your scheduler')
|
||||
console.log(' 2. Batch operations = 100x faster than individual ops')
|
||||
console.log(' 3. Metadata query optimization for complex filters')
|
||||
console.log(' 4. Monitor query performance with explain: true')
|
||||
|
|
@ -1057,41 +1053,22 @@ console.log(' 7. Stream large imports with progress callbacks')
|
|||
|
||||
#### 1. **Storage Options Comparison**
|
||||
|
||||
| Storage | Use Case | Performance | Cost | Setup |
|
||||
|---------|----------|-------------|------|-------|
|
||||
| Memory | Dev/testing | Fastest | Free | Zero config |
|
||||
| Filesystem | Local prod | Fast | Free | Local path |
|
||||
| GCS Native | GCP prod | Fast | $$$ | Service account |
|
||||
| S3 | AWS prod | Fast | $$$ | Access keys |
|
||||
| R2 | Cloudflare | Fast | $ | Access keys |
|
||||
| Storage | Use Case | Performance | Setup |
|
||||
|---------|----------|-------------|-------|
|
||||
| Memory | Dev/testing | Fastest | Zero config |
|
||||
| Filesystem | Production | Fast | Local path + scheduled off-site backup |
|
||||
|
||||
#### 2. **GCS Native vs S3-Compatible**
|
||||
#### 2. **Off-Site Backup**
|
||||
|
||||
```typescript
|
||||
// ✅ RECOMMENDED: GCS Native SDK
|
||||
{
|
||||
storage: {
|
||||
type: 'gcs-native',
|
||||
gcsNativeStorage: {
|
||||
bucketName: 'my-bucket'
|
||||
// ADC handles auth automatically in Cloud Run/GCE!
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ⚠️ LEGACY: S3-compatible mode
|
||||
{
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
gcsStorage: {
|
||||
bucketName: 'my-bucket',
|
||||
accessKeyId: process.env.GCS_ACCESS_KEY,
|
||||
secretAccessKey: process.env.GCS_SECRET_KEY
|
||||
}
|
||||
}
|
||||
}
|
||||
```bash
|
||||
# Cron / systemd timer / k8s CronJob — pick whatever you already operate
|
||||
*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup
|
||||
*/15 * * * * aws s3 sync /var/lib/brainy s3://my-bucket/brainy-backup
|
||||
*/15 * * * * gsutil rsync -r /var/lib/brainy gs://my-bucket/brainy-backup
|
||||
```
|
||||
|
||||
Brainy itself never reaches out to an object store. Snapshot `rootDirectory` from your scheduler.
|
||||
|
||||
#### 3. **Performance Optimization**
|
||||
|
||||
```typescript
|
||||
|
|
@ -1164,8 +1141,8 @@ const brain = new Brainy({ verbose: true })
|
|||
|
||||
#### Before Deployment
|
||||
|
||||
- [ ] Choose cloud storage (GCS/S3/R2)
|
||||
- [ ] Set up authentication (service account/access keys)
|
||||
- [ ] Provision a writable `rootDirectory` on the host
|
||||
- [ ] Set up an off-site snapshot job (`rclone` / `aws s3 sync` / `gsutil rsync`)
|
||||
- [ ] Configure caching
|
||||
- [ ] Test batch operations
|
||||
- [ ] Benchmark query performance
|
||||
|
|
@ -1189,12 +1166,12 @@ const brain = new Brainy({ verbose: true })
|
|||
|
||||
### Practice Exercises
|
||||
|
||||
1. Deploy Brainy with GCS Native storage
|
||||
1. Deploy Brainy with filesystem storage + scheduled off-site backup
|
||||
2. Import a 10,000 row CSV file
|
||||
3. Measure query performance for different filters
|
||||
4. Optimize a slow query using getOptimalQueryPlan()
|
||||
5. Set up monitoring dashboard
|
||||
6. Create backup/restore scripts
|
||||
6. Test restore from off-site snapshot
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -1212,7 +1189,7 @@ const brain = new Brainy({ verbose: true })
|
|||
|
||||
#### Advanced Topics
|
||||
|
||||
- **Distributed Systems**: Sharding, replication, coordination
|
||||
- **Multi-instance Deployments**: Run multiple Brainy processes behind your own routing layer
|
||||
- **Custom Augmentations**: Extend Brainy with plugins
|
||||
- **Streaming Pipelines**: Real-time data ingestion
|
||||
- **Security**: Encryption, access control, audit logs
|
||||
|
|
@ -1223,7 +1200,6 @@ const brain = new Brainy({ verbose: true })
|
|||
- 📚 [API Reference](../api/README.md) - Complete API documentation
|
||||
- 📁 [VFS Guide](../vfs/VFS_API_GUIDE.md) - Virtual Filesystem deep dive
|
||||
- 🤖 [Neural API](../guides/neural-api.md) - Advanced neural operations
|
||||
- 🌐 [Distributed Guide](../guides/distributed-system.md) - Planet-scale architecture
|
||||
- 💬 [Discord Community](https://discord.gg/brainy) - Get help, share projects
|
||||
|
||||
#### Share Your Success
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue