brainy/docs/vfs/VFS_KNOWLEDGE_LAYER.md

VFS + Knowledge Layer Integration

Overview

The Knowledge Layer is an optional augmentation that transforms VFS from a filesystem into an intelligent knowledge management system. When enabled, it adds event recording, semantic versioning, persistent entities, universal concepts, and Git integration.

Enabling Knowledge Layer

const brain = new Brainy({
  storage: { type: 'memory' },
  silent: true
})
await brain.init()

const vfs = brain.vfs()
await vfs.init()

// Enable Knowledge Layer augmentation
await vfs.enableKnowledgeLayer()

// Now VFS has additional intelligent features

Architecture

The Knowledge Layer consists of five integrated systems:

1. EventRecorder

Tracks all filesystem operations as searchable events with embeddings.

2. SemanticVersioning

Creates versions based on semantic meaning changes, not just byte differences.

3. PersistentEntitySystem

Tracks evolving entities (characters, concepts, systems) across files.

4. ConceptSystem

Manages universal concepts that span multiple files and projects.

5. GitBridge

Enables import/export between VFS and Git repositories.

Event Recording

Every filesystem operation is recorded as an event:

// All operations are automatically recorded
await vfs.writeFile('/doc.txt', 'Initial content')
await vfs.appendFile('/doc.txt', '\nMore content')
await vfs.rename('/doc.txt', '/document.txt')

// Query events
const history = await vfs.getHistory('/document.txt')
for (const event of history) {
  console.log(event.type, event.timestamp, event.user)
  // 'create' 2025-01-20T10:00:00Z 'alice'
  // 'write' 2025-01-20T10:01:00Z 'alice'
  // 'rename' 2025-01-20T10:02:00Z 'alice'
}

// Get timeline of events
const events = await vfs.getTimeline({ from: '2025-01-01' })

Event Types

• create - File/directory created
• write - Content written
• append - Content appended
• delete - File/directory deleted
• rename - Path changed
• move - File relocated
• metadata - Metadata updated
• relationship - Relationship added/removed

Event Schema

{
  id: 'uuid',
  type: 'write',
  path: '/document.txt',
  oldPath: null,              // For renames/moves
  timestamp: Date.now(),
  user: 'current-user',
  size: 1024,                 // Bytes affected
  contentHash: 'sha256...',   // Content fingerprint
  vector: [0.1, 0.2, ...],    // Semantic embedding
  metadata: {
    mimeType: 'text/plain',
    encoding: 'utf8'
  }
}

Semantic Versioning

Versions are created when content meaning changes significantly:

// Initial version
await vfs.writeFile('/story.txt', 'Once upon a time...')

// Minor change - no new version (typo fix)
await vfs.writeFile('/story.txt', 'Once upon a time...')

// Major change - creates new version (plot development)
await vfs.writeFile('/story.txt', 'Once upon a time, the kingdom fell...')

// Get versions
const versions = await vfs.getVersions('/story.txt')
for (const version of versions) {
  console.log(version.id, version.timestamp, version.semanticHash)
  // Compare semantic similarity between versions
  console.log(version.similarity) // 0.45 (significantly different)
}

// Restore version
await vfs.restoreVersion('/story.txt', versions[0].id)

// Compare versions by restoring
const v1Content = await vfs.getVersion('/story.txt', v1.id)
const v2Content = await vfs.getVersion('/story.txt', v2.id)
// Compare the content as needed

Version Triggers

• Semantic similarity < 0.7 threshold
• New concepts introduced
• Major structural changes
• Explicit version creation

Persistent Entities

Track characters, systems, and entities across files:

// Create persistent entity
const character = await vfs.createEntity({
  name: 'Alice',
  type: 'character',
  description: 'Main protagonist, a curious explorer',
  attributes: {
    age: 25,
    occupation: 'Archaeologist',
    traits: ['brave', 'intelligent', 'curious']
  }
})

// Entity appears across multiple files
await vfs.writeFile('/chapter1.txt', 'Alice entered the ancient tomb...')
await vfs.writeFile('/chapter2.txt', 'Alice decoded the hieroglyphs...')

// Track entity across files
const occurrences = await vfs.findEntityOccurrences('Alice')
// Returns all files mentioning Alice with context

// Update entity globally
await vfs.updateEntity(character.id, {
  attributes: {
    age: 26,  // Birthday happened in the story
    newTrait: 'experienced'
  }
})

// Entity types
const entities = await vfs.listEntities({ type: 'character' })
// Supports: character, location, object, system, concept, etc.

Entity Relationships

// Link entities
await vfs.linkEntities('Alice', 'Ancient Tomb', 'explores')
await vfs.linkEntities('Alice', 'Bob', 'mentored_by')

// Query entity graph
const graph = await vfs.getEntityGraph('Alice', { depth: 2 })
// Returns connected entities and their relationships

Concept System

Universal concepts that transcend individual files:

// Create concept
const authConcept = await vfs.createConcept({
  name: 'Authentication',
  type: 'technical',
  domain: 'security',
  description: 'User identity verification system',
  keywords: ['login', 'password', 'token', 'session'],
  relatedConcepts: ['Authorization', 'Security']
})

// Concepts are automatically detected in files
await vfs.writeFile('/auth.js', 'function authenticate(user, password) {...}')
await vfs.writeFile('/login.tsx', 'const LoginForm = () => {...}')

// Find files by concept
const authFiles = await vfs.findByConcept('Authentication')
// Returns all files related to authentication concept

// Find files by concept
const authFiles = await vfs.findByConcept('Authentication')
// Returns all files related to the authentication concept

Working with Concepts

// Concepts can reference each other through their descriptions
// and keywords, creating an implicit network of related ideas.
// The findByConcept method searches across these relationships.

GitBridge Integration

Seamlessly work with Git repositories:

// Import from Git repo
await vfs.importFromGit('/local/git/repo', '/vfs/project')

// Imports:
// - All files and directories
// - Git history as VFS events
// - Commit messages as event metadata
// - Branch structure as relationships

// Export to Git format
await vfs.exportToGit('/vfs/project', '/local/git/repo')

// Exports:
// - Files to working directory
// - VFS events as git commits
// - Relationships as .brainy/relationships.json
// - Entities as .brainy/entities.json
// - Concepts as .brainy/concepts.json

// Export/import operations are available
// For remote sync, use git commands after export:
// await vfs.exportToGit('/vfs/project', '/local/repo')
// Then use git push/pull as normal

Git Integration

// Import from Git preserves history as events
// Export to Git creates .brainy/ metadata directory
// Use standard git commands for remote operations

Knowledge Queries

Powerful queries across all Knowledge Layer data:

// Timeline query
const timeline = await vfs.getTimeline({
  from: '2025-01-01',
  to: '2025-01-31',
  types: ['write', 'create']
})

// Timeline queries
const timeline = await vfs.getTimeline({
  from: '2025-01-01',
  to: '2025-01-31',
  types: ['write', 'create']
})

// Project statistics
const stats = await vfs.getProjectStats('/project')
console.log('Total files:', stats.fileCount)
console.log('Total size:', stats.totalSize)
console.log('Todo count:', stats.todoCount)

// Search with Triple Intelligence
const results = await vfs.search('authentication', {
  path: '/src',
  type: 'file',
  limit: 20
})

Background Processing

Knowledge Layer operations run in the background:

// Operations are non-blocking
await vfs.writeFile('/large-doc.txt', hugeContent)
// Returns immediately

// Knowledge processing happens asynchronously:
// 1. Event recording (immediate)
// 2. Embedding generation (100ms)
// 3. Version checking (200ms)
// 4. Entity extraction (500ms)
// 5. Concept detection (1s)

// Background processing happens automatically
// Events are recorded immediately
// Embeddings and versions are processed asynchronously

Search and Analysis

The Knowledge Layer provides powerful search and analysis capabilities:

// Find files by concept
const authFiles = await vfs.findByConcept('Authentication')
// Returns all files related to the authentication concept

// Get timeline of changes
const timeline = await vfs.getTimeline({
  from: '2025-01-01',
  to: '2025-01-31',
  types: ['write', 'create']
})
// Returns chronological list of events

// Get project statistics
const stats = await vfs.getProjectStats('/project')
console.log(stats.fileCount)       // Number of files
console.log(stats.totalSize)       // Total size in bytes
console.log(stats.todoCount)       // Number of todos
console.log(stats.largestFile)     // Largest file info

// Export directory to markdown
const markdown = await vfs.exportToMarkdown('/docs')
// Returns formatted markdown of entire directory structure

Collaboration Features

Knowledge Layer enables multi-user collaboration:

// Track user actions
vfs.setUser('alice')
await vfs.writeFile('/shared.txt', 'Alice\'s content')

vfs.setUser('bob')
await vfs.appendFile('/shared.txt', 'Bob\'s addition')

// Get collaboration history
const collabHistory = await vfs.getCollaborationHistory('/shared.txt')
// Returns who edited the file and when:
// [
//   { user: 'alice', timestamp: Date, action: 'write', size: 15 },
//   { user: 'bob', timestamp: Date, action: 'append', size: 14 }
// ]

// Get all todos across project
const allTodos = await vfs.getAllTodos('/project')
// Returns todos from all files recursively

Performance Impact

Knowledge Layer overhead:

• Write operations: +50-200ms for event recording
• Read operations: No impact (cached)
• Search operations: 10x faster (pre-computed embeddings)
• Storage: ~20% additional for events and embeddings
• Memory: +100MB for caches and indexes

Configuration

Fine-tune Knowledge Layer behavior:

await vfs.enableKnowledgeLayer({
  eventRecording: true,        // Track all operations
  semanticVersioning: true,    // Smart versioning
  versionThreshold: 0.7,       // Similarity threshold
  persistentEntities: true,    // Track entities
  entityTypes: ['character', 'location', 'system'],
  concepts: true,              // Universal concepts
  conceptDomains: ['technical', 'narrative', 'business'],
  gitBridge: true,            // Git integration
  backgroundProcessing: true,  // Non-blocking
  processingDelay: 100,       // Ms before processing
  cacheSizes: {
    events: 10000,
    versions: 1000,
    entities: 5000,
    concepts: 2000
  }
})

Complete Example

import { Brainy } from '@soulcraft/brainy'

async function knowledgeExample() {
  // Initialize with Knowledge Layer
  const brain = new Brainy({
    storage: { type: 'memory' }
  })
  await brain.init()

  const vfs = brain.vfs()
  await vfs.init()
  await vfs.enableKnowledgeLayer()

  // Create a story with tracked entities
  const alice = await vfs.createEntity({
    name: 'Alice',
    type: 'character',
    description: 'Protagonist'
  })

  await vfs.writeFile('/chapter1.md', `
    # Chapter 1
    Alice discovered the ancient artifact...
  `)

  // File automatically:
  // - Records write event
  // - Generates embedding
  // - Links to Alice entity
  // - Detects "ancient artifact" concept

  // Create technical documentation
  await vfs.createConcept({
    name: 'API Design',
    type: 'technical',
    domain: 'software'
  })

  await vfs.writeFile('/api-guide.md', `
    # API Design Guide
    RESTful principles...
  `)

  // Check Knowledge Layer insights
  const insights = await vfs.getInsights('/')
  console.log('Entities:', insights.entities)
  console.log('Concepts:', insights.concepts)
  console.log('Relationships:', insights.relationships)

  // Query across knowledge
  const results = await vfs.knowledgeSearch({
    query: 'Alice artifact',
    includeEvents: true,
    includeEntities: true
  })

  await vfs.close()
  await brain.close()
}

The Knowledge Layer transforms VFS from a filesystem into an intelligent knowledge management system that understands content, tracks evolution, and enables semantic collaboration.