brainy/docs/vfs/ROADMAP.md
David Snelling 00d3203d68 refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider
The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).

Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
  readWriteSeparation, queryPlanner, healthMonitor, configManager,
  hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
  transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
  (slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
  coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
  src/config/extensibleConfig.ts (config/augmentation registry built on removed
  cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
  enableDistributedSearch (dead config flag); the metadata partition field;
  the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
  distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
  shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
  augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
  vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
  storage-architecture; reframed scale prose to the 8.0 model.

Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.

RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
2026-06-15 10:37:39 -07:00

5.7 KiB

VFS Roadmap - Planned Features

Status: These features are planned but not yet implemented. Current Version: See main VFS README for implemented features.


v1.1 (Next Release)

Enhanced Streaming Support

Improve VFS streaming to support true chunk-by-chunk reads for large files without loading entire content into memory.

Status: Planned Effort: 3-4 weeks

VFS Security Integration

Integrate Brainy's SecurityAPI with VFS file operations.

// Planned API (not yet implemented)
await vfs.encrypt('/sensitive-data.json', { algorithm: 'AES-256' })
await vfs.setACL('/project', { user: 'alice', permissions: 'rw-' })
const auditLog = await vfs.getAuditLog('/project', { since: lastWeek })

Status: Planned Note: SecurityAPI exists (src/api/SecurityAPI.ts) but not integrated with VFS operations.

Atomic Operations Layer

Add transaction support for compound VFS operations.

// Planned API (not yet implemented)
await vfs.transaction(async (tx) => {
  await tx.move('/src/file.txt', '/dest/file.txt')
  await tx.writeFile('/dest/metadata.json', metadata)
  // Both succeed or both fail
})

Status: Planned Effort: 1-2 weeks


v1.2

Version History

Track file versions with history and restoration capabilities.

// Planned API (not yet implemented)
await vfs.enableVersioning('/important-doc.md')
const versions = await vfs.getVersions('/important-doc.md')
await vfs.restoreVersion('/important-doc.md', versions[2].id)
const diff = await vfs.diffVersions('/important-doc.md', v1, v2)

Features:

  • Automatic versioning on file changes
  • Version history with metadata
  • Point-in-time restoration
  • Version comparison/diff

Status: Planned Effort: 4-5 weeks

Backup & Recovery API

Built-in backup and recovery operations.

// Planned API (not yet implemented)
const changes = await vfs.getChangesSince(lastBackupTime)
await vfs.createSnapshot('/backup-snapshot-2024')
await vfs.restoreToTime('/project', timestamp)
await vfs.verifyIntegrity('/project')
const issues = await vfs.repair('/project')

Status: Planned Effort: 3-4 weeks


v2.0

AI-Powered Auto-Organization

Intelligent file organization based on content and usage patterns.

// Planned API (not yet implemented)
await vfs.autoOrganize('/downloads', {
  strategy: 'by-content-type',
  createFolders: true
})

const duplicates = await vfs.detectDuplicates('/photos')
await vfs.deduplicateFiles(duplicates, { keep: 'highest-quality' })

await vfs.optimizeStorage('/project', {
  compress: ['*.log', '*.txt'],
  archive: { olderThan: '90d' }
})

Features:

  • Content-based organization
  • Smart deduplication
  • Content-aware compression
  • Automatic archival

Status: Planned - Research phase Effort: 8-10 weeks

Smart Collections / Virtual Directories

Persistent query-based virtual directories that auto-update.

// Planned API (not yet implemented)
await vfs.createVirtualDirectory('/auth-related', {
  query: { concepts: { contains: 'authentication' }},
  autoUpdate: true
})

// Directory automatically updates as matching files are added/modified

Note: Current VFS supports query-based access (e.g., /by-concept/auth) but not persistent virtual directories.

Status: Planned Effort: 4-5 weeks

FUSE Driver

Mount VFS as a native filesystem on Linux/Mac/Windows.

// Planned (research phase)
import { mountVFS } from '@soulcraft/brainy/vfs/fuse'

await mountVFS(vfs, {
  mountPoint: '/mnt/brainy',
  options: { allowOther: true }
})

Challenges:

  • FUSE requires synchronous operations (VFS is async-first)
  • Kernel-level integration complexity
  • Cross-platform support (FUSE/Dokan/WinFsp)

Status: Research phase Effort: 10-12 weeks + significant testing


Community Contributions Wanted

These features would benefit from community contributions. If you're interested in building any of these, please open an issue!

Express.js Static Middleware

// Wanted: Community contribution
import { createStaticMiddleware } from '@soulcraft/brainy/vfs/express'

app.use('/files', createStaticMiddleware(vfs, {
  index: ['index.html', 'index.md'],
  etag: true,
  cacheControl: 'max-age=3600'
}))

VSCode Extension

// Wanted: Community contribution
import { VFSProvider } from '@soulcraft/brainy/vfs/vscode'

const provider = new VFSProvider(vfs)
vscode.workspace.registerFileSystemProvider('brainy', provider)

Features:

  • Browse VFS in VSCode explorer
  • Semantic search from command palette
  • Concept highlighting
  • Relationship visualization

Webpack Plugin

Semantic-aware webpack builds with dependency graph from VFS relationships.

Vite Plugin

Vite integration with VFS for semantic module resolution.


Far Future (v5.0+)

Cloud Provider Auto-Detection

// Far future concept
const brain = new Brainy({
  storage: 'cloud://brainy-data' // Auto-detects AWS/GCP/Azure
})

Hot/Cold Storage Tiering

Automatic data movement between hot (SSD/local) and cold (S3/archive) storage based on access patterns.


How to Track Progress

  • GitHub Issues: Track feature development
  • Discussions: Discuss feature design and requirements
  • Pull Requests: Submit contributions

Contributing

Interested in implementing a planned feature? Here's how to get started:

  1. Open an issue discussing the feature you want to implement
  2. Review the design - we'll help with architecture decisions
  3. Submit a PR with implementation + tests
  4. Celebrate! Your contribution helps everyone 🎉

Last Updated: 2025-10-29 VFS Version: 4.9.0