The distributed-clustering subsystem never ran in production: it was inert, orphaned dead code (faked consensus, stub replication, no live wiring, and it did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process library. Scale is single-process + the optional native provider (@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools + horizontal read scaling (many reader processes, one writer). Removed: - src/distributed/ entirely (coordinator, shardManager, cacheSync, readWriteSeparation, queryPlanner, healthMonitor, configManager, hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts (slimmed to the live surface). - src/types/distributedTypes.ts; config.distributed field + JSDoc; coreTypes distributedConfig; memoryStorage distributedConfig persistence. - DistributedRole enum + src/config/distributedPresets.ts and the orphaned src/config/extensibleConfig.ts (config/augmentation registry built on removed cloud adapters + distributed presets), plus their src/index.ts re-exports. - 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook; enableDistributedSearch (dead config flag); the metadata partition field; the distributed_ reserved key prefix. - Orphaned src/storage/readOnlyOptimizations.ts (zero importers). - Tests targeting the subsystem: distributed-demo, distributed-cluster helper, distributed-transactions, sharding-transactions. - Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/ shard-manager/multi-node prose from v3-features, enterprise-for-everyone, augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP, vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4, storage-architecture; reframed scale prose to the 8.0 model. Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via getShardIdFromUuid — used live by baseStorage, unrelated to clustering); the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering. RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0 scale model.
230 lines
5.7 KiB
Markdown
230 lines
5.7 KiB
Markdown
# VFS Roadmap - Planned Features
|
|
|
|
**Status:** These features are planned but not yet implemented.
|
|
**Current Version:** See main [VFS README](./README.md) for implemented features.
|
|
|
|
---
|
|
|
|
## v1.1 (Next Release)
|
|
|
|
### Enhanced Streaming Support
|
|
Improve VFS streaming to support true chunk-by-chunk reads for large files without loading entire content into memory.
|
|
|
|
**Status:** Planned
|
|
**Effort:** 3-4 weeks
|
|
|
|
### VFS Security Integration
|
|
Integrate Brainy's SecurityAPI with VFS file operations.
|
|
|
|
```typescript
|
|
// Planned API (not yet implemented)
|
|
await vfs.encrypt('/sensitive-data.json', { algorithm: 'AES-256' })
|
|
await vfs.setACL('/project', { user: 'alice', permissions: 'rw-' })
|
|
const auditLog = await vfs.getAuditLog('/project', { since: lastWeek })
|
|
```
|
|
|
|
**Status:** Planned
|
|
**Note:** SecurityAPI exists (`src/api/SecurityAPI.ts`) but not integrated with VFS operations.
|
|
|
|
### Atomic Operations Layer
|
|
Add transaction support for compound VFS operations.
|
|
|
|
```typescript
|
|
// Planned API (not yet implemented)
|
|
await vfs.transaction(async (tx) => {
|
|
await tx.move('/src/file.txt', '/dest/file.txt')
|
|
await tx.writeFile('/dest/metadata.json', metadata)
|
|
// Both succeed or both fail
|
|
})
|
|
```
|
|
|
|
**Status:** Planned
|
|
**Effort:** 1-2 weeks
|
|
|
|
---
|
|
|
|
## v1.2
|
|
|
|
### Version History
|
|
Track file versions with history and restoration capabilities.
|
|
|
|
```typescript
|
|
// Planned API (not yet implemented)
|
|
await vfs.enableVersioning('/important-doc.md')
|
|
const versions = await vfs.getVersions('/important-doc.md')
|
|
await vfs.restoreVersion('/important-doc.md', versions[2].id)
|
|
const diff = await vfs.diffVersions('/important-doc.md', v1, v2)
|
|
```
|
|
|
|
**Features:**
|
|
- Automatic versioning on file changes
|
|
- Version history with metadata
|
|
- Point-in-time restoration
|
|
- Version comparison/diff
|
|
|
|
**Status:** Planned
|
|
**Effort:** 4-5 weeks
|
|
|
|
### Backup & Recovery API
|
|
Built-in backup and recovery operations.
|
|
|
|
```typescript
|
|
// Planned API (not yet implemented)
|
|
const changes = await vfs.getChangesSince(lastBackupTime)
|
|
await vfs.createSnapshot('/backup-snapshot-2024')
|
|
await vfs.restoreToTime('/project', timestamp)
|
|
await vfs.verifyIntegrity('/project')
|
|
const issues = await vfs.repair('/project')
|
|
```
|
|
|
|
**Status:** Planned
|
|
**Effort:** 3-4 weeks
|
|
|
|
---
|
|
|
|
## v2.0
|
|
|
|
### AI-Powered Auto-Organization
|
|
Intelligent file organization based on content and usage patterns.
|
|
|
|
```typescript
|
|
// Planned API (not yet implemented)
|
|
await vfs.autoOrganize('/downloads', {
|
|
strategy: 'by-content-type',
|
|
createFolders: true
|
|
})
|
|
|
|
const duplicates = await vfs.detectDuplicates('/photos')
|
|
await vfs.deduplicateFiles(duplicates, { keep: 'highest-quality' })
|
|
|
|
await vfs.optimizeStorage('/project', {
|
|
compress: ['*.log', '*.txt'],
|
|
archive: { olderThan: '90d' }
|
|
})
|
|
```
|
|
|
|
**Features:**
|
|
- Content-based organization
|
|
- Smart deduplication
|
|
- Content-aware compression
|
|
- Automatic archival
|
|
|
|
**Status:** Planned - Research phase
|
|
**Effort:** 8-10 weeks
|
|
|
|
### Smart Collections / Virtual Directories
|
|
Persistent query-based virtual directories that auto-update.
|
|
|
|
```typescript
|
|
// Planned API (not yet implemented)
|
|
await vfs.createVirtualDirectory('/auth-related', {
|
|
query: { concepts: { contains: 'authentication' }},
|
|
autoUpdate: true
|
|
})
|
|
|
|
// Directory automatically updates as matching files are added/modified
|
|
```
|
|
|
|
**Note:** Current VFS supports query-based *access* (e.g., `/by-concept/auth`) but not persistent virtual directories.
|
|
|
|
**Status:** Planned
|
|
**Effort:** 4-5 weeks
|
|
|
|
### FUSE Driver
|
|
Mount VFS as a native filesystem on Linux/Mac/Windows.
|
|
|
|
```typescript
|
|
// Planned (research phase)
|
|
import { mountVFS } from '@soulcraft/brainy/vfs/fuse'
|
|
|
|
await mountVFS(vfs, {
|
|
mountPoint: '/mnt/brainy',
|
|
options: { allowOther: true }
|
|
})
|
|
```
|
|
|
|
**Challenges:**
|
|
- FUSE requires synchronous operations (VFS is async-first)
|
|
- Kernel-level integration complexity
|
|
- Cross-platform support (FUSE/Dokan/WinFsp)
|
|
|
|
**Status:** Research phase
|
|
**Effort:** 10-12 weeks + significant testing
|
|
|
|
---
|
|
|
|
## Community Contributions Wanted
|
|
|
|
These features would benefit from community contributions. If you're interested in building any of these, please open an issue!
|
|
|
|
### Express.js Static Middleware
|
|
```typescript
|
|
// Wanted: Community contribution
|
|
import { createStaticMiddleware } from '@soulcraft/brainy/vfs/express'
|
|
|
|
app.use('/files', createStaticMiddleware(vfs, {
|
|
index: ['index.html', 'index.md'],
|
|
etag: true,
|
|
cacheControl: 'max-age=3600'
|
|
}))
|
|
```
|
|
|
|
### VSCode Extension
|
|
```typescript
|
|
// Wanted: Community contribution
|
|
import { VFSProvider } from '@soulcraft/brainy/vfs/vscode'
|
|
|
|
const provider = new VFSProvider(vfs)
|
|
vscode.workspace.registerFileSystemProvider('brainy', provider)
|
|
```
|
|
|
|
**Features:**
|
|
- Browse VFS in VSCode explorer
|
|
- Semantic search from command palette
|
|
- Concept highlighting
|
|
- Relationship visualization
|
|
|
|
### Webpack Plugin
|
|
Semantic-aware webpack builds with dependency graph from VFS relationships.
|
|
|
|
### Vite Plugin
|
|
Vite integration with VFS for semantic module resolution.
|
|
|
|
---
|
|
|
|
## Far Future (v5.0+)
|
|
|
|
### Cloud Provider Auto-Detection
|
|
```typescript
|
|
// Far future concept
|
|
const brain = new Brainy({
|
|
storage: 'cloud://brainy-data' // Auto-detects AWS/GCP/Azure
|
|
})
|
|
```
|
|
|
|
### Hot/Cold Storage Tiering
|
|
Automatic data movement between hot (SSD/local) and cold (S3/archive) storage based on access patterns.
|
|
|
|
---
|
|
|
|
## How to Track Progress
|
|
|
|
- **GitHub Issues:** Track feature development
|
|
- **Discussions:** Discuss feature design and requirements
|
|
- **Pull Requests:** Submit contributions
|
|
|
|
---
|
|
|
|
## Contributing
|
|
|
|
Interested in implementing a planned feature? Here's how to get started:
|
|
|
|
1. **Open an issue** discussing the feature you want to implement
|
|
2. **Review the design** - we'll help with architecture decisions
|
|
3. **Submit a PR** with implementation + tests
|
|
4. **Celebrate!** Your contribution helps everyone 🎉
|
|
|
|
---
|
|
|
|
**Last Updated:** 2025-10-29
|
|
**VFS Version:** 4.9.0
|