diff --git a/docs/vfs/README.md b/docs/vfs/README.md index b7195ddc..9c3b095a 100644 --- a/docs/vfs/README.md +++ b/docs/vfs/README.md @@ -163,28 +163,32 @@ const parisPhotos = await vfs.search('', { }) ``` -### ๐ŸŽฏ Smart Collections +### ๐ŸŽฏ Semantic Path Access -Virtual directories based on queries: +Access files through semantic dimensions (see [Semantic VFS](./SEMANTIC_VFS.md)): ```javascript -// Create a smart folder that auto-updates -await vfs.createVirtualDirectory('/smart/recent-docs', { - query: 'type:document modified:last-7-days' +// Query-based path access (current functionality) +const authFiles = await vfs.search('', { + where: { concepts: { contains: 'authentication' }} }) -// Create a collection based on similarity -await vfs.createVirtualDirectory('/smart/like-this', { - similar: '/examples/good-code.js', +// Find files by custom metadata +const recent = await vfs.search('', { + where: { + type: 'document', + modified: { greaterThan: Date.now() - 7*24*60*60*1000 } + } +}) + +// Find similar files +const similar = await vfs.findSimilar('/examples/good-code.js', { threshold: 0.7 }) - -// Tag-based collections -await vfs.createVirtualDirectory('/smart/important', { - where: { tags: 'important' } -}) ``` +> **Note:** Virtual directories (persistent query-based folders) are planned for v2.0. See [ROADMAP](./ROADMAP.md). + ## Real-World Examples ### ๐Ÿ“š Knowledge Management @@ -314,150 +318,28 @@ const urgent = await vfs.search('', { ## Advanced Features -### ๐Ÿ”„ Version History +> **Note:** See [VFS ROADMAP](./ROADMAP.md) for planned advanced features like version history, distributed filesystem, and more. -```javascript -// Enable versioning for a file -await vfs.enableVersioning('/important/contract.pdf') +## Integration Possibilities -// Write updates - automatically creates versions -await vfs.writeFile('/important/contract.pdf', newVersion) +VFS can be integrated with existing applications. See [VFS ROADMAP](./ROADMAP.md) for planned integrations like Express.js middleware, VSCode extensions, and more. -// Get version history -const versions = await vfs.getVersions('/important/contract.pdf') -// Returns: [{version: 1, date: ..., size: ...}, {version: 2, ...}] +**Current approach:** Use VFS directly via API for custom integrations. -// Restore a previous version -await vfs.restoreVersion('/important/contract.pdf', 1) - -// Compare versions -const diff = await vfs.diffVersions('/important/contract.pdf', 1, 2) -``` - -### ๐ŸŒ Distributed Filesystem - -```javascript -// Mount remote Brainy instances -await vfs.mount('/remote/server2', { - host: 'brainy.server2.com', - credentials: { ... } -}) - -// Federated search across all mounted systems -const results = await vfs.search('project documentation', { - distributed: true -}) - -// Sync directories across instances -await vfs.sync('/projects', '/remote/server2/backup/projects') -``` - -### ๐Ÿค– AI-Powered Automation - -```javascript -// Auto-organize downloads folder -await vfs.autoOrganize('/downloads', { - rules: [ - { pattern: '*.pdf', destination: '/documents' }, - { pattern: '*.{jpg,png}', destination: '/images' }, - { semantic: 'code files', destination: '/code' } - ] -}) - -// Smart deduplication -const duplicates = await vfs.detectDuplicates('/photos') -await vfs.deduplicateFiles(duplicates, { - strategy: 'keep-highest-quality' -}) - -// Content-aware compression -await vfs.optimizeStorage('/archives', { - compress: true, - deduplicate: true, - indexContent: true -}) -``` - -### ๐Ÿ” Security & Permissions - -```javascript -// Set access control -await vfs.setACL('/private', { - owner: 'user123', - permissions: { - owner: 'rwx', - group: 'r-x', - others: '---' - } -}) - -// Encryption at rest -await vfs.encrypt('/sensitive', { - algorithm: 'AES-256', - key: encryptionKey -}) - -// Audit trail -const audit = await vfs.getAuditLog('/financial/reports') -// Returns: who accessed what and when -``` - -## Integration Examples - -### Node.js fs Compatibility - -```javascript -// Drop-in replacement for fs module -import { promises as fs } from '@soulcraft/brainy/vfs/fs' - -// Works with existing code! -const data = await fs.readFile('/config.json', 'utf8') -const config = JSON.parse(data) - -await fs.writeFile('/output.txt', 'Hello VFS!') -const stats = await fs.stat('/output.txt') -``` - -### Express.js Static Files - -```javascript -import express from 'express' -import { createStaticMiddleware } from '@soulcraft/brainy/vfs/express' - -const app = express() - -// Serve files from VFS -app.use('/static', createStaticMiddleware('/public', { - intelligentCaching: true, // Cache based on access patterns - autoCompress: true // Compress on the fly -})) -``` - -### VSCode Extension - -```javascript -// Open VFS in VSCode -import { workspace } from 'vscode' -import { VFSProvider } from '@soulcraft/brainy/vfs/vscode' - -// Register VFS as a filesystem provider -workspace.registerFileSystemProvider('brainy', new VFSProvider(), { - isCaseSensitive: true, - isReadonly: false -}) - -// Now you can open: brainy:///projects/my-app -``` - -## Performance +## Performance Characteristics Brainy VFS is designed for speed and scale: -- **Sub-10ms latency** for basic operations -- **Intelligent caching** reduces repeated reads to <1ms -- **Vector search** returns results in <100ms for millions of files -- **Streaming support** for files of any size -- **Distributed sharding** for billions of files +**Tested at 1K-10K file scale:** +- **Sub-10ms latency** for basic operations (measured) +- **Intelligent caching** reduces repeated reads to <5ms (measured) + +**PROJECTED at larger scales (not yet tested):** +- **Vector search** <100ms for millions of files (projected) +- **Streaming support** for files of any size (architecture supports, see [limitations in ROADMAP](./ROADMAP.md)) +- **Distributed sharding** for billions of files (architecture supports, not tested at scale) + +See tests in `tests/vfs/` for actual measured performance. ## Triple Intelligence Power ๐Ÿง โšก @@ -736,54 +618,11 @@ const assigned = await vfs.search('', { }) ``` -### Monitoring & Operations +### Monitoring & Operations (Planned) -#### **Production Metrics** +> **Note:** Production monitoring features are planned for v1.1. See [ROADMAP](./ROADMAP.md). -```javascript -// Get VFS statistics -const stats = vfs.getStatistics() -console.log(stats) -// { -// totalFiles: 1234567, -// totalDirectories: 45678, -// totalSize: 123456789000, -// cacheHitRate: 0.95, -// avgResponseTime: 12, -// activeConnections: 234 -// } - -// Monitor hot paths -const hotPaths = vfs.getHotPaths() -// Paths accessed >100 times/minute - -// Check health -const health = await vfs.healthCheck() -// { -// status: 'healthy', -// latency: { p50: 10, p99: 100 }, -// errors: { rate: 0.001 } -// } -``` - -#### **Backup & Recovery** - -```javascript -// Incremental backup -const changes = await vfs.getChangesSince(lastBackupTime) -for (const change of changes) { - await backupSystem.store(change) -} - -// Point-in-time recovery -await vfs.restoreToTime(timestamp) - -// Verify integrity -const corrupted = await vfs.verifyIntegrity() -if (corrupted.length > 0) { - await vfs.repair(corrupted) -} -``` +> **Note:** Backup & Recovery features are planned for v1.2. See [ROADMAP](./ROADMAP.md). ### Deployment Options @@ -827,12 +666,15 @@ await vfs.init({ }) ``` -## Coming Soon +## Roadmap & Future Features -- **Version 1.1**: Full streaming support, FUSE driver -- **Version 1.2**: Distributed transactions, global replication -- **Version 1.3**: Time-travel queries, branching -- **Version 2.0**: Quantum-resistant encryption, neural interfaces +See [VFS ROADMAP](./ROADMAP.md) for planned features including: +- Enhanced streaming support (v1.1) +- Version history (v1.2) +- Distributed filesystem (v1.2) +- AI-powered automation (v2.0) +- FUSE driver (v2.0 research) +- And more community-requested features ## Contributing diff --git a/docs/vfs/ROADMAP.md b/docs/vfs/ROADMAP.md new file mode 100644 index 00000000..14833def --- /dev/null +++ b/docs/vfs/ROADMAP.md @@ -0,0 +1,257 @@ +# VFS Roadmap - Planned Features + +**Status:** These features are planned but not yet implemented. +**Current Version:** See main [VFS README](./README.md) for implemented features. + +--- + +## v1.1 (Next Release) + +### Enhanced Streaming Support +Improve VFS streaming to support true chunk-by-chunk reads for large files without loading entire content into memory. + +**Status:** Planned +**Effort:** 3-4 weeks + +### VFS Security Integration +Integrate Brainy's SecurityAPI with VFS file operations. + +```typescript +// Planned API (not yet implemented) +await vfs.encrypt('/sensitive-data.json', { algorithm: 'AES-256' }) +await vfs.setACL('/project', { user: 'alice', permissions: 'rw-' }) +const auditLog = await vfs.getAuditLog('/project', { since: lastWeek }) +``` + +**Status:** Planned +**Note:** SecurityAPI exists (`src/api/SecurityAPI.ts`) but not integrated with VFS operations. + +### Atomic Operations Layer +Add transaction support for compound VFS operations. + +```typescript +// Planned API (not yet implemented) +await vfs.transaction(async (tx) => { + await tx.move('/src/file.txt', '/dest/file.txt') + await tx.writeFile('/dest/metadata.json', metadata) + // Both succeed or both fail +}) +``` + +**Status:** Planned +**Effort:** 1-2 weeks + +--- + +## v1.2 + +### Version History +Track file versions with history and restoration capabilities. + +```typescript +// Planned API (not yet implemented) +await vfs.enableVersioning('/important-doc.md') +const versions = await vfs.getVersions('/important-doc.md') +await vfs.restoreVersion('/important-doc.md', versions[2].id) +const diff = await vfs.diffVersions('/important-doc.md', v1, v2) +``` + +**Features:** +- Automatic versioning on file changes +- Version history with metadata +- Point-in-time restoration +- Version comparison/diff + +**Status:** Planned +**Effort:** 4-5 weeks + +### Distributed Filesystem +Mount remote Brainy instances for federated file access. + +```typescript +// Planned API (not yet implemented) +await vfs.mount('/remote-team', { + type: 'brainy-remote', + url: 'https://team-brainy.example.com', + credentials: {...} +}) + +// Federated search across mounted instances +const results = await vfs.search('project docs', { includeMounted: true }) + +// Sync directories +await vfs.sync('/local/docs', '/remote-team/docs') +``` + +**Status:** Planned +**Effort:** 6-8 weeks + +### Backup & Recovery API +Built-in backup and recovery operations. + +```typescript +// Planned API (not yet implemented) +const changes = await vfs.getChangesSince(lastBackupTime) +await vfs.createSnapshot('/backup-snapshot-2024') +await vfs.restoreToTime('/project', timestamp) +await vfs.verifyIntegrity('/project') +const issues = await vfs.repair('/project') +``` + +**Status:** Planned +**Effort:** 3-4 weeks + +--- + +## v2.0 + +### AI-Powered Auto-Organization +Intelligent file organization based on content and usage patterns. + +```typescript +// Planned API (not yet implemented) +await vfs.autoOrganize('/downloads', { + strategy: 'by-content-type', + createFolders: true +}) + +const duplicates = await vfs.detectDuplicates('/photos') +await vfs.deduplicateFiles(duplicates, { keep: 'highest-quality' }) + +await vfs.optimizeStorage('/project', { + compress: ['*.log', '*.txt'], + archive: { olderThan: '90d' } +}) +``` + +**Features:** +- Content-based organization +- Smart deduplication +- Content-aware compression +- Automatic archival + +**Status:** Planned - Research phase +**Effort:** 8-10 weeks + +### Smart Collections / Virtual Directories +Persistent query-based virtual directories that auto-update. + +```typescript +// Planned API (not yet implemented) +await vfs.createVirtualDirectory('/auth-related', { + query: { concepts: { contains: 'authentication' }}, + autoUpdate: true +}) + +// Directory automatically updates as matching files are added/modified +``` + +**Note:** Current VFS supports query-based *access* (e.g., `/by-concept/auth`) but not persistent virtual directories. + +**Status:** Planned +**Effort:** 4-5 weeks + +### FUSE Driver +Mount VFS as a native filesystem on Linux/Mac/Windows. + +```typescript +// Planned (research phase) +import { mountVFS } from '@soulcraft/brainy/vfs/fuse' + +await mountVFS(vfs, { + mountPoint: '/mnt/brainy', + options: { allowOther: true } +}) +``` + +**Challenges:** +- FUSE requires synchronous operations (VFS is async-first) +- Kernel-level integration complexity +- Cross-platform support (FUSE/Dokan/WinFsp) + +**Status:** Research phase +**Effort:** 10-12 weeks + significant testing + +--- + +## Community Contributions Wanted + +These features would benefit from community contributions. If you're interested in building any of these, please open an issue! + +### Express.js Static Middleware +```typescript +// Wanted: Community contribution +import { createStaticMiddleware } from '@soulcraft/brainy/vfs/express' + +app.use('/files', createStaticMiddleware(vfs, { + index: ['index.html', 'index.md'], + etag: true, + cacheControl: 'max-age=3600' +})) +``` + +### VSCode Extension +```typescript +// Wanted: Community contribution +import { VFSProvider } from '@soulcraft/brainy/vfs/vscode' + +const provider = new VFSProvider(vfs) +vscode.workspace.registerFileSystemProvider('brainy', provider) +``` + +**Features:** +- Browse VFS in VSCode explorer +- Semantic search from command palette +- Concept highlighting +- Relationship visualization + +### Webpack Plugin +Semantic-aware webpack builds with dependency graph from VFS relationships. + +### Vite Plugin +Vite integration with VFS for semantic module resolution. + +--- + +## Far Future (v5.0+) + +### Automatic Node Discovery +Zero-config multi-node setup with automatic discovery via UDP broadcast, Kubernetes DNS, or cloud provider APIs. + +### Automatic Failover +Health monitoring and automatic failover in distributed deployments. + +### Cloud Provider Auto-Detection +```typescript +// Far future concept +const brain = new Brainy({ + storage: 'cloud://brainy-data' // Auto-detects AWS/GCP/Azure +}) +``` + +### Hot/Cold Storage Tiering +Automatic data movement between hot (SSD/local) and cold (S3/archive) storage based on access patterns. + +--- + +## How to Track Progress + +- **GitHub Issues:** Track feature development +- **Discussions:** Discuss feature design and requirements +- **Pull Requests:** Submit contributions + +--- + +## Contributing + +Interested in implementing a planned feature? Here's how to get started: + +1. **Open an issue** discussing the feature you want to implement +2. **Review the design** - we'll help with architecture decisions +3. **Submit a PR** with implementation + tests +4. **Celebrate!** Your contribution helps everyone ๐ŸŽ‰ + +--- + +**Last Updated:** 2025-10-29 +**VFS Version:** 4.9.0 diff --git a/docs/vfs/SEMANTIC_VFS.md b/docs/vfs/SEMANTIC_VFS.md index ca1709b9..03639cc8 100644 --- a/docs/vfs/SEMANTIC_VFS.md +++ b/docs/vfs/SEMANTIC_VFS.md @@ -81,13 +81,15 @@ const experiments = await vfs.readdir('/by-tag/experimental') ## Supported Semantic Dimensions -### 1. Traditional Path (Hierarchical) +### 1. Traditional Path (Hierarchical) โœ… **Production** ```typescript await vfs.readFile('/src/auth/login.ts') // Works exactly like a normal filesystem ``` -### 2. By Concept (Semantic) +**Status:** โœ… Fully implemented and tested + +### 2. By Concept (Semantic) โš ๏ธ **Beta** ```typescript await vfs.readdir('/by-concept/authentication') // Returns all files about authentication @@ -98,7 +100,9 @@ await vfs.readFile('/by-concept/authentication/login.ts') **How it works:** Uses `brain.extractConcepts()` with NeuralEntityExtractor to extract concepts from file content using embeddings and the NounType taxonomy. Indexes concept names for O(log n) queries. See [Neural Extraction API](./NEURAL_EXTRACTION.md) for details. -### 3. By Author (Ownership) +**Status:** โš ๏ธ Beta - Requires NeuralEntityExtractor setup, tested at <1K file scale + +### 3. By Author (Ownership) โœ… **Production** ```typescript await vfs.readdir('/by-author/alice') // All files owned/modified by alice @@ -109,7 +113,9 @@ await vfs.stat('/by-author/alice/config.ts') **How it works:** Tracks owner metadata on every file. Indexed by MetadataIndexManager. -### 4. By Time (Temporal) +**Status:** โœ… Fully implemented and tested at 10K file scale + +### 4. By Time (Temporal) โœ… **Production** ```typescript await vfs.readdir('/as-of/2024-03-15') // Files modified on March 15, 2024 @@ -120,7 +126,9 @@ await vfs.readFile('/as-of/2024-03-15/src/auth.ts') **How it works:** Tracks `modified` timestamp. Uses B-tree range queries (`greaterEqual`/`lessEqual`) for O(log n) performance. -### 5. By Relationship (Graph) +**Status:** โœ… Fully implemented and tested at 10K file scale + +### 5. By Relationship (Graph) โœ… **Production** ```typescript await vfs.readdir('/related-to/src/auth.ts/depth-2') // Files within 2 relationship hops @@ -131,7 +139,9 @@ await vfs.readdir('/related-to/src/auth.ts/depth-2/types-contains,references') **How it works:** Uses GraphAdjacencyIndex for O(1) graph traversal. Supports depth limits and relationship type filtering. -### 6. By Similarity (Vector) +**Status:** โœ… Fully implemented and tested at 10K node scale + +### 6. By Similarity (Vector) โœ… **Production** ```typescript await vfs.readdir('/similar-to/src/auth.ts/threshold-0.8') // Files with 80%+ similarity to auth.ts @@ -142,7 +152,9 @@ await vfs.similar('/src/auth.ts', { threshold: 0.9, limit: 10 }) **How it works:** Uses HNSW vector index for O(log n) nearest neighbor search. Based on content embeddings. -### 7. By Tag (Classification) +**Status:** โœ… Fully implemented and tested at 100K vector scale + +### 7. By Tag (Classification) โœ… **Production** ```typescript await vfs.readdir('/by-tag/security') // All security-tagged files @@ -155,25 +167,27 @@ await vfs.writeFile('/src/admin.ts', code, { **How it works:** Stores tags in metadata. Indexed for fast queries. +**Status:** โœ… Fully implemented and tested at 10K file scale + --- ## Performance Characteristics -### Scalability to Millions of Files +### Tested Performance at Scale All semantic paths use **indexed data structures** for optimal performance: -| Dimension | Data Structure | Time Complexity | Million-Scale Ready | -|-----------|---------------|-----------------|---------------------| -| Traditional | PathCache + Graph | O(path depth) | โœ… Yes | -| Concept | MetadataIndex (B-tree) | O(log n) | โœ… Yes* | -| Author | MetadataIndex (B-tree) | O(log n) | โœ… Yes | -| Time | MetadataIndex (B-tree) | O(log n) | โœ… Yes | -| Relationship | GraphAdjacency | O(depth) | โœ… Yes | -| Similarity | HNSW Index | O(log n) | โœ… Yes | -| Tag | MetadataIndex (B-tree) | O(log n) | โœ… Yes* | +| Dimension | Data Structure | Time Complexity | Tested Scale | Production Ready | +|-----------|---------------|-----------------|--------------|------------------| +| Traditional | PathCache + Graph | O(path depth) | Up to 10K files | โœ… Yes | +| Concept | MetadataIndex (B-tree) | O(log n) | Up to 1K files | โš ๏ธ Beta | +| Author | MetadataIndex (B-tree) | O(log n) | Up to 10K files | โœ… Yes | +| Time | MetadataIndex (B-tree) | O(log n) | Up to 10K files | โœ… Yes | +| Relationship | GraphAdjacency | O(depth) | Up to 10K nodes | โœ… Yes | +| Similarity | HNSW Index | O(log n) | Up to 100K vectors | โœ… Yes | +| Tag | MetadataIndex (B-tree) | O(log n) | Up to 10K files | โœ… Yes | -\* *Requires concept/tag flattening (automatic in future versions)* +**Note:** Million-scale performance is PROJECTED based on underlying index complexity. VFS-specific testing conducted at 1K-100K scale. See `tests/vfs/` for measured performance. ### Cache Strategy @@ -301,7 +315,11 @@ console.log(id1 === id2 && id2 === id3) // true --- -## Extending with Custom Projections +## Extending with Custom Projections ๐Ÿงช **Experimental** + +**Status:** ๐Ÿงช Experimental - API subject to change + +**Warning:** This API uses internal VFS interfaces that are not yet officially exposed. The registration mechanism will change in a future release to provide a stable public API. Create your own semantic dimensions: @@ -334,12 +352,12 @@ class PriorityProjection extends BaseProjectionStrategy { } } -// Register custom projection +// Register custom projection (experimental - uses internal API) const brain = new Brainy() await brain.init() const vfs = brain.vfs() -// Access via VFS internals (will be exposed in future API) +// โš ๏ธ Internal API - will be replaced with public registration method vfs.projectionRegistry.register(new PriorityProjection()) // Now use it! @@ -348,6 +366,8 @@ const highPriority = await vfs.readdir('/by-priority/high') See [PROJECTION_STRATEGY_API.md](./PROJECTION_STRATEGY_API.md) for full guide. +**Roadmap:** Public projection registration API coming in v1.2 (see [VFS ROADMAP](./ROADMAP.md)) + --- ## Architecture diff --git a/docs/vfs/VFS_API_GUIDE.md b/docs/vfs/VFS_API_GUIDE.md index 722ab8e6..5a358575 100644 --- a/docs/vfs/VFS_API_GUIDE.md +++ b/docs/vfs/VFS_API_GUIDE.md @@ -626,40 +626,68 @@ try { ### Storage Compatibility -VFS works with **all** Brainy storage adapters: +VFS works with all built-in Brainy storage adapters: ```typescript -// Memory (testing) +// Memory (testing/development) const brain = new Brainy({ storage: { type: 'memory' } }) -// Redis (development) +// Filesystem (local development) const brain = new Brainy({ storage: { - type: 'redis', - url: 'redis://localhost:6379' + type: 'filesystem', + path: './brainy-data' } }) -// PostgreSQL (production) +// S3-compatible storage (production) const brain = new Brainy({ storage: { - type: 'postgresql', - connectionString: 'postgresql://user:pass@localhost/db' + type: 's3', + bucket: 'my-bucket', + region: 'us-east-1' } }) -// ChromaDB (vector-optimized) +// Cloudflare R2 (production) const brain = new Brainy({ storage: { - type: 'chroma', - url: 'http://localhost:8000' + type: 'r2', + accountId: 'your-account-id', + bucket: 'my-bucket' } }) +// Google Cloud Storage (production) +const brain = new Brainy({ + storage: { + type: 'gcs', + bucket: 'my-bucket', + projectId: 'my-project' + } +}) + +// Azure Blob Storage (production) +const brain = new Brainy({ + storage: { + type: 'azure', + accountName: 'myaccount', + containerName: 'my-container' + } +}) + +// OPFS (browser-native persistence) +const brain = new Brainy({ storage: { type: 'opfs' } }) + +// TypeAware storage (optimized for entity types) +const brain = new Brainy({ storage: { type: 'typeaware' } }) + // All work identically with VFS const vfs = new VirtualFileSystem(brain) ``` +**Custom Storage Adapters:** Redis, PostgreSQL, and other databases can be added via the [extension system](../api/EXTENSIBILITY.md). See `src/config/extensibleConfig.ts` for examples. + ### Best Practices #### File Organization