soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions
brainy/docs/vfs/SEMANTIC_VFS.md

15 KiB
Raw Blame History

Semantic VFS - Revolutionary File System

What is Semantic VFS?

Semantic VFS transforms traditional hierarchical file systems into multi-dimensional knowledge graphs. The same file can be accessed through multiple semantic dimensions simultaneously.

Traditional vs Semantic

Traditional File Systems:

/src/auth/login.ts          # One path, one location
/src/users/profile.ts       # Separate location

Semantic VFS:

# Traditional path (still works!)
/src/auth/login.ts

# By concept
/by-concept/authentication/login.ts
/by-concept/security/login.ts

# By author
/by-author/alice/login.ts

# By time
/as-of/2024-03-15/login.ts

# By relationship
/related-to/src/users/profile.ts/depth-2

The same file, accessible 6+ different ways! This is polymorphic file access.

Why Semantic VFS?

1. Natural Organization

Developers think in concepts, not directories:

// Find all authentication-related files
const authFiles = await vfs.readdir('/by-concept/authentication')

// Find all files Alice worked on
const aliceFiles = await vfs.readdir('/by-author/alice')

2. Change Tracking by Date

List the files that were modified on any given day:

// Files that changed on March 15th
const changed = await vfs.readdir('/as-of/2024-03-15')

// Everything under /src right now
const current = await vfs.readdir('/src')

/as-of/<date> selects by modification date — it reads the files' current content, not historical versions. For true point-in-time queries over entity state, use the Db API (brain.asOf(generation)).

3. Knowledge Graph Navigation

Navigate by semantic relationships:

// Files related to auth system (within 2 hops)
const related = await vfs.readdir('/related-to/src/auth.ts/depth-2')

// Files similar to this implementation
const similar = await vfs.readdir('/similar-to/src/auth.ts/threshold-0.8')

4. Tag-Based Organization

Organize by purpose, not location:

// All security-critical files
const security = await vfs.readdir('/by-tag/security')

// All experimental features
const experiments = await vfs.readdir('/by-tag/experimental')

Supported Semantic Dimensions

1. Traditional Path (Hierarchical) ✅ Production

await vfs.readFile('/src/auth/login.ts')
// Works exactly like a normal filesystem

Status: ✅ Fully implemented and tested

2. By Concept (Semantic) ⚠️ Beta

await vfs.readdir('/by-concept/authentication')
// Returns all files about authentication

await vfs.readFile('/by-concept/authentication/login.ts')
// Find specific file within concept

How it works: Uses brain.extractConcepts() with NeuralEntityExtractor to extract concepts from file content using embeddings and the NounType taxonomy. Indexes concept names for O(log n) queries. See Neural Extraction API for details.

Status: ⚠️ Beta - Requires NeuralEntityExtractor setup, tested at <1K file scale

3. By Author (Ownership) ✅ Production

await vfs.readdir('/by-author/alice')
// All files owned/modified by alice

await vfs.stat('/by-author/alice/config.ts')
// Check specific file

How it works: Tracks owner metadata on every file. Indexed by MetadataIndexManager.

Status: ✅ Fully implemented and tested at 10K file scale

4. By Time (Temporal) ✅ Production

await vfs.readdir('/as-of/2024-03-15')
// Files modified on March 15, 2024 (24-hour window)

await vfs.readFile('/as-of/2024-03-15/auth.ts')
// Current content of auth.ts, addressed by modification date —
// the path only resolves if auth.ts was modified that day

How it works: Tracks the modified timestamp on every file and runs a range query (greaterEqual/lessEqual) over one 24-hour window for O(log n) performance. The VFS does not store historical file contents — /as-of/ filters by when a file last changed; reads return the current bytes. For point-in-time state, use the Db API (brain.asOf(generation)).

Status: ✅ Fully implemented and tested at 10K file scale

5. By Relationship (Graph) ✅ Production

await vfs.readdir('/related-to/src/auth.ts/depth-2')
// Files within 2 relationship hops

await vfs.readdir('/related-to/src/auth.ts/depth-2/types-contains,references')
// Only follow 'contains' and 'references' relationships

How it works: Uses GraphAdjacencyIndex for O(1) graph traversal. Supports depth limits and relationship type filtering.

Status: ✅ Fully implemented and tested at 10K node scale

6. By Similarity (Vector) ✅ Production

await vfs.readdir('/similar-to/src/auth.ts/threshold-0.8')
// Files with 80%+ similarity to auth.ts

await vfs.similar('/src/auth.ts', { threshold: 0.9, limit: 10 })
// Top 10 most similar files (90%+ match)

How it works: Uses HNSW vector index for O(log n) nearest neighbor search. Based on content embeddings.

Status: ✅ Fully implemented and tested at 100K vector scale

7. By Tag (Classification) ✅ Production

await vfs.readdir('/by-tag/security')
// All security-tagged files

await vfs.writeFile('/src/admin.ts', code, {
  metadata: { tags: ['security', 'admin'] }
})
// Tag files on write

How it works: Stores tags in metadata. Indexed for fast queries.

Status: ✅ Fully implemented and tested at 10K file scale

Performance Characteristics

Tested Performance at Scale

All semantic paths use indexed data structures for optimal performance:

Dimension Data Structure Time Complexity Tested Scale Production Ready
Traditional PathCache + Graph O(path depth) Up to 10K files ✅ Yes
Concept MetadataIndex (B-tree) O(log n) Up to 1K files ⚠️ Beta
Author MetadataIndex (B-tree) O(log n) Up to 10K files ✅ Yes
Time MetadataIndex (B-tree) O(log n) Up to 10K files ✅ Yes
Relationship GraphAdjacency O(depth) Up to 10K nodes ✅ Yes
Similarity HNSW Index O(log n) Up to 100K vectors ✅ Yes
Tag MetadataIndex (B-tree) O(log n) Up to 10K files ✅ Yes

Note: Million-scale performance is PROJECTED based on underlying index complexity. VFS-specific testing conducted at 1K-100K scale. See tests/vfs/ for measured performance.

Cache Strategy

Multi-layer caching ensures hot paths are O(1):

Request → Hot Path Cache (O(1))
       → Semantic Cache (5 min TTL)
       → Index Lookup (O(log n))

Usage Examples

Example 1: Find All Files by Concept

const brain = new Brainy()
await brain.init()
const vfs = brain.vfs()
await vfs.init()

// Write files (concepts extracted automatically)
await vfs.writeFile('/src/auth/login.ts', `
  export function authenticate(user, password) {
    // Authentication logic
  }
`)

// Access by concept
const authFiles = await vfs.readdir('/by-concept/authentication')
console.log(authFiles)
// ['login.ts', 'signup.ts', 'oauth.ts']

Example 2: Changes by Day

// See what changed today
const today = new Date().toISOString().split('T')[0]
const todaysFiles = await vfs.readdir(`/as-of/${today}`)

// Compare with yesterday
const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0]
const yesterdaysFiles = await vfs.readdir(`/as-of/${yesterday}`)

const onlyToday = todaysFiles.filter(f => !yesterdaysFiles.includes(f))
console.log('Changed today (untouched yesterday):', onlyToday)

Example 3: Graph Navigation

// Find all files related to auth
const authId = await vfs.resolvePath('/src/auth.ts')
const related = await vfs.readdir('/related-to/src/auth.ts/depth-2')

// Get relationship details
for (const file of related) {
  const rels = await vfs.getRelationships(file.path)
  console.log(`${file.name}: ${rels.length} relationships`)
}

Example 4: Semantic Search

// Find similar implementations
const similar = await vfs.similar('/src/auth.ts', {
  threshold: 0.8,
  limit: 10
})

for (const result of similar) {
  console.log(`${result.entity.name}: ${result.similarity.toFixed(2)}`)
}

API Reference

Reading Semantic Paths

All standard VFS methods work with semantic paths:

// Read directory
await vfs.readdir('/by-concept/authentication')

// Read file
await vfs.readFile('/by-concept/authentication/login.ts')

// Get stats
await vfs.stat('/by-author/alice/config.ts')

// Check existence
await vfs.exists('/as-of/2024-03-15/src/auth.ts')

Writing Files

Files are automatically indexed for semantic access:

await vfs.writeFile('/src/auth.ts', content, {
  metadata: {
    tags: ['security', 'authentication'],
    owner: 'alice'
  },
  extractConcepts: true,  // default: true
  extractEntities: true,  // default: true
  recordEvent: true       // default: true
})

Polymorphic Access

The same file is accessible through multiple paths:

// All these resolve to the SAME file entity:
const id1 = await vfs.resolvePath('/src/auth/login.ts')
const id2 = await vfs.resolvePath('/by-concept/authentication/login.ts')
const id3 = await vfs.resolvePath('/by-author/alice/login.ts')

console.log(id1 === id2 && id2 === id3)  // true

Extending with Custom Projections 🧪 Experimental

Status: 🧪 Experimental - API subject to change

Warning: This API uses internal VFS interfaces that are not yet officially exposed. The registration mechanism will change in a future release to provide a stable public API.

Create your own semantic dimensions:

import { BaseProjectionStrategy } from '@soulcraft/brainy/vfs/semantic'

class PriorityProjection extends BaseProjectionStrategy {
  readonly name = 'priority'

  async resolve(brain, vfs, priority) {
    return await brain.find({
      where: {
        vfsType: 'file',
        priority: priority  // Custom metadata field
      },
      limit: 1000
    })
    .then(results => results.map(r => r.id))
  }

  async list(brain, vfs, limit = 100) {
    const results = await brain.find({
      where: {
        vfsType: 'file',
        priority: { exists: true }
      },
      limit
    })
    return results.map(r => r.entity)
  }
}

// Register custom projection (experimental - uses internal API)
const brain = new Brainy()
await brain.init()
const vfs = brain.vfs()

// ⚠️ Internal API - will be replaced with public registration method
vfs.projectionRegistry.register(new PriorityProjection())

// Now use it!
const highPriority = await vfs.readdir('/by-priority/high')

See PROJECTION_STRATEGY_API.md for full guide.

Roadmap: Public projection registration API coming in v1.2 (see VFS ROADMAP)

Architecture

Triple Intelligence™ Foundation

Semantic VFS is built on Brainy's Triple Intelligence™:

┌─────────────────────────────────────────┐
│         Semantic VFS Layer              │
├─────────────────────────────────────────┤
│    ProjectionRegistry + Strategies      │
├─────────────────────────────────────────┤
│        SemanticPathResolver             │
├─────────────────────────────────────────┤
│                                         │
│    Triple Intelligence™ (Brainy)        │
│  ┌─────────┬─────────┬─────────────┐  │
│  │ Vector  │  Graph  │  Metadata   │  │
│  │  HNSW   │  Adj    │   B-tree    │  │
│  │ O(log n)│  O(1)   │  O(log n)   │  │
│  └─────────┴─────────┴─────────────┘  │
└─────────────────────────────────────────┘

Real Implementations, Zero Mocks

Every component uses production Brainy APIs:

  • brain.find() - Real metadata queries
  • brain.similar() - Real HNSW search
  • brain.related() - Real graph traversal
  • MetadataIndexManager - Real B-tree indexes
  • GraphAdjacencyIndex - Real graph storage
  • HNSW Index - Real vector search

No mocks. No stubs. No fake code.

Best Practices

1. Use Semantic Paths for Discovery

// ❌ Don't hardcode paths
const files = ['/src/auth.ts', '/src/login.ts', '/src/oauth.ts']

// ✅ Discover by concept
const authFiles = await vfs.readdir('/by-concept/authentication')

2. Tag Strategically

// ✅ Good: Clear, actionable tags
await vfs.writeFile(path, code, {
  metadata: { tags: ['security', 'requires-review', 'public-api'] }
})

// ❌ Bad: Vague, redundant tags
await vfs.writeFile(path, code, {
  metadata: { tags: ['code', 'file', 'important'] }
})

3. Combine Dimensions

// Find security files Alice changed on a given day
// (each /as-of/<date> path covers exactly that one day)
const aliceFiles = await vfs.readdir('/by-author/alice')
const securityFiles = await vfs.readdir('/by-tag/security')
const changedThatDay = await vfs.readdir('/as-of/2024-03-15')

const intersection = aliceFiles
  .filter(f => securityFiles.includes(f))
  .filter(f => changedThatDay.includes(f))

Troubleshooting

Concepts Not Being Extracted

// Check if concepts are enabled (default: true)
await vfs.writeFile(path, code, { extractConcepts: true })

// Verify concept extraction works
const entity = await vfs.getEntity(path)
console.log(entity.metadata.concepts)

Slow Queries on Large Datasets

// Check if indexes are built and populated
const stats = await brain.getIndexStats()
console.log(stats)

If an index looks empty or inconsistent, rebuild from raw storage with the CLI (stop the live writer first): brainy inspect repair <data-dir>.

Semantic Path Returns Empty

// Check if metadata exists
const files = await vfs.readdir('/src')
for (const file of files) {
  const entity = await vfs.getEntity(file.path)
  console.log(entity.metadata)
}

What's Next?

  • Natural Language Paths: /find "authentication logic"
  • Intent-Based Access: /to-review, /to-deploy
  • Temporal Queries: /changed-since/2024-03-01
  • Custom Dimensions: Plugin system for domain-specific projections

See Also