brainy/docs/vfs/ROADMAP.md
David Snelling 00d3203d68 refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider
The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).

Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
  readWriteSeparation, queryPlanner, healthMonitor, configManager,
  hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
  transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
  (slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
  coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
  src/config/extensibleConfig.ts (config/augmentation registry built on removed
  cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
  enableDistributedSearch (dead config flag); the metadata partition field;
  the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
  distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
  shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
  augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
  vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
  storage-architecture; reframed scale prose to the 8.0 model.

Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.

RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
2026-06-15 10:37:39 -07:00

230 lines
5.7 KiB
Markdown

# VFS Roadmap - Planned Features
**Status:** These features are planned but not yet implemented.
**Current Version:** See main [VFS README](./README.md) for implemented features.
---
## v1.1 (Next Release)
### Enhanced Streaming Support
Improve VFS streaming to support true chunk-by-chunk reads for large files without loading entire content into memory.
**Status:** Planned
**Effort:** 3-4 weeks
### VFS Security Integration
Integrate Brainy's SecurityAPI with VFS file operations.
```typescript
// Planned API (not yet implemented)
await vfs.encrypt('/sensitive-data.json', { algorithm: 'AES-256' })
await vfs.setACL('/project', { user: 'alice', permissions: 'rw-' })
const auditLog = await vfs.getAuditLog('/project', { since: lastWeek })
```
**Status:** Planned
**Note:** SecurityAPI exists (`src/api/SecurityAPI.ts`) but not integrated with VFS operations.
### Atomic Operations Layer
Add transaction support for compound VFS operations.
```typescript
// Planned API (not yet implemented)
await vfs.transaction(async (tx) => {
await tx.move('/src/file.txt', '/dest/file.txt')
await tx.writeFile('/dest/metadata.json', metadata)
// Both succeed or both fail
})
```
**Status:** Planned
**Effort:** 1-2 weeks
---
## v1.2
### Version History
Track file versions with history and restoration capabilities.
```typescript
// Planned API (not yet implemented)
await vfs.enableVersioning('/important-doc.md')
const versions = await vfs.getVersions('/important-doc.md')
await vfs.restoreVersion('/important-doc.md', versions[2].id)
const diff = await vfs.diffVersions('/important-doc.md', v1, v2)
```
**Features:**
- Automatic versioning on file changes
- Version history with metadata
- Point-in-time restoration
- Version comparison/diff
**Status:** Planned
**Effort:** 4-5 weeks
### Backup & Recovery API
Built-in backup and recovery operations.
```typescript
// Planned API (not yet implemented)
const changes = await vfs.getChangesSince(lastBackupTime)
await vfs.createSnapshot('/backup-snapshot-2024')
await vfs.restoreToTime('/project', timestamp)
await vfs.verifyIntegrity('/project')
const issues = await vfs.repair('/project')
```
**Status:** Planned
**Effort:** 3-4 weeks
---
## v2.0
### AI-Powered Auto-Organization
Intelligent file organization based on content and usage patterns.
```typescript
// Planned API (not yet implemented)
await vfs.autoOrganize('/downloads', {
strategy: 'by-content-type',
createFolders: true
})
const duplicates = await vfs.detectDuplicates('/photos')
await vfs.deduplicateFiles(duplicates, { keep: 'highest-quality' })
await vfs.optimizeStorage('/project', {
compress: ['*.log', '*.txt'],
archive: { olderThan: '90d' }
})
```
**Features:**
- Content-based organization
- Smart deduplication
- Content-aware compression
- Automatic archival
**Status:** Planned - Research phase
**Effort:** 8-10 weeks
### Smart Collections / Virtual Directories
Persistent query-based virtual directories that auto-update.
```typescript
// Planned API (not yet implemented)
await vfs.createVirtualDirectory('/auth-related', {
query: { concepts: { contains: 'authentication' }},
autoUpdate: true
})
// Directory automatically updates as matching files are added/modified
```
**Note:** Current VFS supports query-based *access* (e.g., `/by-concept/auth`) but not persistent virtual directories.
**Status:** Planned
**Effort:** 4-5 weeks
### FUSE Driver
Mount VFS as a native filesystem on Linux/Mac/Windows.
```typescript
// Planned (research phase)
import { mountVFS } from '@soulcraft/brainy/vfs/fuse'
await mountVFS(vfs, {
mountPoint: '/mnt/brainy',
options: { allowOther: true }
})
```
**Challenges:**
- FUSE requires synchronous operations (VFS is async-first)
- Kernel-level integration complexity
- Cross-platform support (FUSE/Dokan/WinFsp)
**Status:** Research phase
**Effort:** 10-12 weeks + significant testing
---
## Community Contributions Wanted
These features would benefit from community contributions. If you're interested in building any of these, please open an issue!
### Express.js Static Middleware
```typescript
// Wanted: Community contribution
import { createStaticMiddleware } from '@soulcraft/brainy/vfs/express'
app.use('/files', createStaticMiddleware(vfs, {
index: ['index.html', 'index.md'],
etag: true,
cacheControl: 'max-age=3600'
}))
```
### VSCode Extension
```typescript
// Wanted: Community contribution
import { VFSProvider } from '@soulcraft/brainy/vfs/vscode'
const provider = new VFSProvider(vfs)
vscode.workspace.registerFileSystemProvider('brainy', provider)
```
**Features:**
- Browse VFS in VSCode explorer
- Semantic search from command palette
- Concept highlighting
- Relationship visualization
### Webpack Plugin
Semantic-aware webpack builds with dependency graph from VFS relationships.
### Vite Plugin
Vite integration with VFS for semantic module resolution.
---
## Far Future (v5.0+)
### Cloud Provider Auto-Detection
```typescript
// Far future concept
const brain = new Brainy({
storage: 'cloud://brainy-data' // Auto-detects AWS/GCP/Azure
})
```
### Hot/Cold Storage Tiering
Automatic data movement between hot (SSD/local) and cold (S3/archive) storage based on access patterns.
---
## How to Track Progress
- **GitHub Issues:** Track feature development
- **Discussions:** Discuss feature design and requirements
- **Pull Requests:** Submit contributions
---
## Contributing
Interested in implementing a planned feature? Here's how to get started:
1. **Open an issue** discussing the feature you want to implement
2. **Review the design** - we'll help with architecture decisions
3. **Submit a PR** with implementation + tests
4. **Celebrate!** Your contribution helps everyone 🎉
---
**Last Updated:** 2025-10-29
**VFS Version:** 4.9.0