docs(8.0): Phase F — deep clean across 21 docs

Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.

Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
  cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
  HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
  sections replaced with single-node vector tuning + filesystem framing.

Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md

Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md

MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
David Snelling 2026-06-09 16:13:35 -07:00
parent 2626ab8d62
commit adda1570f3
22 changed files with 570 additions and 2658 deletions

View file

@ -852,7 +852,7 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
## Level 5: Production Scale (90 minutes)
### What You'll Learn
- Cloud storage (GCS, S3, R2)
- Production filesystem storage and off-site backup
- Performance optimization
- Batch imports (CSV, Excel, PDF)
- Metadata query optimization
@ -863,18 +863,13 @@ Ready for production deployment? Level 5 covers **planet-scale architecture**.
```typescript
import { Brainy, NounType } from '@soulcraft/brainy'
// 1. PRODUCTION STORAGE - Google Cloud Storage (Native SDK)
console.log('☁️ Initializing production storage...\n')
// 1. PRODUCTION STORAGE - Filesystem with off-site snapshots
console.log('Initializing production storage...\n')
const brain = new Brainy({
storage: {
type: 'gcs-native', // Native GCS SDK (recommended)
gcsNativeStorage: {
bucketName: 'my-brainy-production',
// ADC (Application Default Credentials) - zero config in Cloud Run/GCE!
// Or provide credentials:
// keyFilename: '/path/to/service-account.json'
}
type: 'filesystem',
rootDirectory: '/var/lib/brainy'
},
// Performance tuning
@ -888,7 +883,8 @@ const brain = new Brainy({
})
await brain.init()
console.log('✅ Brainy initialized with GCS Native storage\n')
console.log('Brainy initialized with filesystem storage')
console.log('Snapshot /var/lib/brainy off-site via cron with `gsutil rsync` / `aws s3 sync` / `rclone`.\n')
// 2. BATCH IMPORT - CSV File
console.log('📊 Importing CSV data...\n')
@ -1044,7 +1040,7 @@ console.log('✅ Brain closed cleanly')
console.log('\n\n🎓 Production Deployment Complete!')
console.log('\n📚 Key Production Learnings:')
console.log(' 1. Use native cloud storage (GCS, S3, R2) for persistence')
console.log(' 1. Use filesystem storage and snapshot rootDirectory off-site from your scheduler')
console.log(' 2. Batch operations = 100x faster than individual ops')
console.log(' 3. Metadata query optimization for complex filters')
console.log(' 4. Monitor query performance with explain: true')
@ -1057,41 +1053,22 @@ console.log(' 7. Stream large imports with progress callbacks')
#### 1. **Storage Options Comparison**
| Storage | Use Case | Performance | Cost | Setup |
|---------|----------|-------------|------|-------|
| Memory | Dev/testing | Fastest | Free | Zero config |
| Filesystem | Local prod | Fast | Free | Local path |
| GCS Native | GCP prod | Fast | $$$ | Service account |
| S3 | AWS prod | Fast | $$$ | Access keys |
| R2 | Cloudflare | Fast | $ | Access keys |
| Storage | Use Case | Performance | Setup |
|---------|----------|-------------|-------|
| Memory | Dev/testing | Fastest | Zero config |
| Filesystem | Production | Fast | Local path + scheduled off-site backup |
#### 2. **GCS Native vs S3-Compatible**
#### 2. **Off-Site Backup**
```typescript
// ✅ RECOMMENDED: GCS Native SDK
{
storage: {
type: 'gcs-native',
gcsNativeStorage: {
bucketName: 'my-bucket'
// ADC handles auth automatically in Cloud Run/GCE!
}
}
}
// ⚠️ LEGACY: S3-compatible mode
{
storage: {
type: 'gcs',
gcsStorage: {
bucketName: 'my-bucket',
accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY
}
}
}
```bash
# Cron / systemd timer / k8s CronJob — pick whatever you already operate
*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup
*/15 * * * * aws s3 sync /var/lib/brainy s3://my-bucket/brainy-backup
*/15 * * * * gsutil rsync -r /var/lib/brainy gs://my-bucket/brainy-backup
```
Brainy itself never reaches out to an object store. Snapshot `rootDirectory` from your scheduler.
#### 3. **Performance Optimization**
```typescript
@ -1164,8 +1141,8 @@ const brain = new Brainy({ verbose: true })
#### Before Deployment
- [ ] Choose cloud storage (GCS/S3/R2)
- [ ] Set up authentication (service account/access keys)
- [ ] Provision a writable `rootDirectory` on the host
- [ ] Set up an off-site snapshot job (`rclone` / `aws s3 sync` / `gsutil rsync`)
- [ ] Configure caching
- [ ] Test batch operations
- [ ] Benchmark query performance
@ -1189,12 +1166,12 @@ const brain = new Brainy({ verbose: true })
### Practice Exercises
1. Deploy Brainy with GCS Native storage
1. Deploy Brainy with filesystem storage + scheduled off-site backup
2. Import a 10,000 row CSV file
3. Measure query performance for different filters
4. Optimize a slow query using getOptimalQueryPlan()
5. Set up monitoring dashboard
6. Create backup/restore scripts
6. Test restore from off-site snapshot
---
@ -1212,7 +1189,7 @@ const brain = new Brainy({ verbose: true })
#### Advanced Topics
- **Distributed Systems**: Sharding, replication, coordination
- **Multi-instance Deployments**: Run multiple Brainy processes behind your own routing layer
- **Custom Augmentations**: Extend Brainy with plugins
- **Streaming Pipelines**: Real-time data ingestion
- **Security**: Encryption, access control, audit logs
@ -1223,7 +1200,6 @@ const brain = new Brainy({ verbose: true })
- 📚 [API Reference](../api/README.md) - Complete API documentation
- 📁 [VFS Guide](../vfs/VFS_API_GUIDE.md) - Virtual Filesystem deep dive
- 🤖 [Neural API](../guides/neural-api.md) - Advanced neural operations
- 🌐 [Distributed Guide](../guides/distributed-system.md) - Planet-scale architecture
- 💬 [Discord Community](https://discord.gg/brainy) - Get help, share projects
#### Share Your Success