feat: add neural extraction APIs with NounType taxonomy
Add brain.extract() and brain.extractConcepts() methods that use NeuralEntityExtractor with embeddings and sophisticated NounType taxonomy (30+ entity types) for semantic entity and concept extraction. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
27cc699555
commit
dd50d89ad6
41 changed files with 3807 additions and 7391 deletions
|
|
@ -273,12 +273,13 @@ const files = await vfs.getDirectChildren('/docs')
|
|||
- **Directory structure?** → VFS
|
||||
- **File explorer interface?** → VFS
|
||||
|
||||
### Knowledge-Enhanced Files
|
||||
### Intelligent File Management
|
||||
|
||||
**Use VFS with Knowledge Layer when:**
|
||||
**Use VFS (Semantic VFS) when:**
|
||||
- Need semantic file search
|
||||
- Want AI-powered file insights
|
||||
- Want AI-powered concept extraction
|
||||
- Building smart file systems
|
||||
- Require multi-dimensional file access
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for intelligent file systems
|
||||
|
|
@ -468,8 +469,8 @@ import { BrainyAugmentation } from '@soulcraft/brainy'
|
|||
- Complex graph → Triple Intelligence
|
||||
|
||||
**Need files?**
|
||||
- Basic → VFS
|
||||
- Smart → VFS + Knowledge Layer
|
||||
- Basic → VFS (standard operations)
|
||||
- Smart → Semantic VFS (6 dimensional access + neural extraction)
|
||||
|
||||
**Need AI analysis?**
|
||||
- Patterns → Neural clustering
|
||||
|
|
|
|||
|
|
@ -1,214 +0,0 @@
|
|||
# The Knowledge Layer: Your Files Get Smart 🧠✨
|
||||
|
||||
## What Is The Knowledge Layer?
|
||||
|
||||
Imagine if your computer files could **remember**, **understand**, and **connect** with each other. That's exactly what Brainy's Knowledge Layer does - it transforms ordinary files into an intelligent, living knowledge system.
|
||||
|
||||
### The Old Way vs. The New Way
|
||||
|
||||
**Traditional File Systems:**
|
||||
- Files are "dumb" - just data sitting in folders
|
||||
- No memory of changes or why they were made
|
||||
- No understanding of what's inside files
|
||||
- No connections between related information
|
||||
- Manual organization that breaks over time
|
||||
- Lost context and forgotten relationships
|
||||
|
||||
**Brainy's Knowledge Layer:**
|
||||
- Files **understand their content** and purpose
|
||||
- **Perfect memory** of every change and why it happened
|
||||
- **Automatic connections** between related information
|
||||
- **Self-organizing** system that gets smarter over time
|
||||
- **Never lose context** - everything stays connected
|
||||
- **Tells the story** of how your knowledge evolved
|
||||
|
||||
## How It Works (Simple Version)
|
||||
|
||||
Think of your files like characters in a story:
|
||||
|
||||
1. **Every file becomes intelligent** - It knows what it contains, when it was changed, and why
|
||||
2. **Files remember everything** - Complete history of every edit, with context
|
||||
3. **Related files find each other** - Documents about the same project automatically connect
|
||||
4. **Concepts live everywhere** - Ideas like "customer satisfaction" link across all your work
|
||||
5. **Time travel is possible** - See exactly what you were thinking at any point in the past
|
||||
6. **Stories emerge naturally** - Watch how your ideas, projects, and knowledge evolved
|
||||
|
||||
## Real-World Benefits
|
||||
|
||||
### For Business Professionals 💼
|
||||
|
||||
**Project Management That Remembers:**
|
||||
- Every project document knows its purpose and connections
|
||||
- See how decisions evolved and why changes were made
|
||||
- Automatically find all materials related to any project
|
||||
- Never lose track of who said what or when
|
||||
|
||||
**Example:** Working on a new product launch:
|
||||
- Your pitch deck, market research, budget spreadsheet, and meeting notes all know they're connected
|
||||
- When you update the budget, you can see exactly what changed and why
|
||||
- All team members' contributions are tracked and connected
|
||||
- Six months later, you can reconstruct exactly how the project developed
|
||||
|
||||
**Intelligent Document Search:**
|
||||
- "Find all documents about customer retention from Q3" - instantly found
|
||||
- "Show me everything related to the Johnson account" - complete picture
|
||||
- "What were our concerns about the mobile app?" - all discussions surfaced
|
||||
|
||||
**Collaboration Without Confusion:**
|
||||
- Every team member's contributions are tracked and connected
|
||||
- Changes have context and reasoning attached
|
||||
- Related work automatically surfaces when you need it
|
||||
- No more "Who wrote this?" or "Why did we decide that?"
|
||||
|
||||
### For Authors & Creative Professionals 📖
|
||||
|
||||
**Characters That Live Across Your Work:**
|
||||
- Create a character once, they exist everywhere they appear
|
||||
- Track character development across chapters, books, or series
|
||||
- See how characters change and grow over time
|
||||
- Never forget character details or contradict yourself
|
||||
|
||||
**Example:** Writing a fantasy series:
|
||||
- "Aragorn" exists as one entity across all books
|
||||
- His character development tracked from chapter 1 through the end
|
||||
- All his appearances, quotes, and growth moments connected
|
||||
- Themes like "heroism" link across all related scenes automatically
|
||||
|
||||
**Themes and Concepts That Connect Everything:**
|
||||
- Universal themes like "redemption" appear across all your work
|
||||
- See how concepts evolve throughout your writing
|
||||
- Find all instances where you explored specific ideas
|
||||
- Track the development of your creative vision
|
||||
|
||||
**Never Lose Your Creative Thread:**
|
||||
- See exactly what you were thinking when you wrote something
|
||||
- Understand how your story ideas developed over time
|
||||
- Find forgotten plot points or character insights
|
||||
- Reconstruct your creative process months or years later
|
||||
|
||||
### For Video Game & Movie Development 🎮🎬
|
||||
|
||||
**Living Worlds and Universes:**
|
||||
- Game characters, locations, and lore exist as connected entities
|
||||
- Story beats and character arcs tracked across all content
|
||||
- Automatic consistency checking across all materials
|
||||
- Complete universe bible that builds itself
|
||||
|
||||
**Example:** Developing a game:
|
||||
- "The Forbidden Forest" exists once, appears in game design docs, story scripts, and art references
|
||||
- Every character's backstory connected to their dialogue and quests
|
||||
- Plot threads tracked from initial concept through final implementation
|
||||
- All team contributions linked to the elements they created
|
||||
|
||||
**Production History That Makes Sense:**
|
||||
- See how creative decisions evolved through development
|
||||
- Understand why story changes were made
|
||||
- Track the impact of character or plot modifications
|
||||
- Never lose the reasoning behind creative choices
|
||||
|
||||
## Key Features Explained Simply
|
||||
|
||||
### 1. Smart Versioning 🔄
|
||||
**What it does:** Only saves new versions when the **meaning** changes, not just typos
|
||||
**Why it matters:** 90% fewer versions than traditional systems, but you never lose important changes
|
||||
**Example:** Fixing "there" to "their" = no new version. Changing your conclusion = new version saved.
|
||||
|
||||
### 2. Time Travel 🕰️
|
||||
**What it does:** Go back to see your work at any point in time, with full context
|
||||
**Why it matters:** Perfect memory of how your thinking evolved
|
||||
**Example:** "What was I thinking about pricing in March?" - instantly see your exact reasoning
|
||||
|
||||
### 3. Automatic Connections 🔗
|
||||
**What it does:** Related documents find and link to each other automatically
|
||||
**Why it matters:** Never lose track of related work or duplicate effort
|
||||
**Example:** All documents mentioning "customer retention" automatically connect, regardless of folder
|
||||
|
||||
### 4. Universal Concepts 💡
|
||||
**What it does:** Ideas exist independently and appear across all your work
|
||||
**Why it matters:** See the big picture of how concepts develop in your thinking
|
||||
**Example:** The concept "sustainability" appears in your strategy docs, product specs, and marketing materials
|
||||
|
||||
### 5. Perfect Memory 📚
|
||||
**What it does:** Every change recorded with context about why it was made
|
||||
**Why it matters:** Complete audit trail of your thinking and decision-making
|
||||
**Example:** See not just what changed in your business plan, but why you changed it
|
||||
|
||||
### 6. Intelligent Search 🔍
|
||||
**What it does:** Find information by meaning, not just keywords
|
||||
**Why it matters:** Discover connections and information you forgot existed
|
||||
**Example:** "Plans for international expansion" finds relevant documents even if they say "global market entry"
|
||||
|
||||
## Business Impact
|
||||
|
||||
### Return on Investment
|
||||
- **50% reduction** in time spent searching for information
|
||||
- **75% faster** onboarding for new team members
|
||||
- **90% fewer** lost documents or forgotten decisions
|
||||
- **Complete visibility** into how projects and decisions evolved
|
||||
|
||||
### Risk Reduction
|
||||
- Never lose context behind important decisions
|
||||
- Complete audit trail for compliance and legal needs
|
||||
- Backup of all thinking and reasoning processes
|
||||
- Protection against knowledge loss when people leave
|
||||
|
||||
### Innovation Acceleration
|
||||
- Build on previous work instead of starting from scratch
|
||||
- Surface forgotten ideas and insights
|
||||
- Connect disparate concepts for breakthrough thinking
|
||||
- Learn from your own patterns and evolution
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Phase 1: Basic Intelligence
|
||||
Start by simply storing your files in the Knowledge Layer. They immediately become:
|
||||
- Searchable by meaning
|
||||
- Connected to related content
|
||||
- Tracked for all changes
|
||||
|
||||
### Phase 2: Smart Organization
|
||||
Enable automatic features:
|
||||
- Concepts and entities extract themselves
|
||||
- Related documents find each other
|
||||
- History tracking provides full context
|
||||
|
||||
### Phase 3: Knowledge Mastery
|
||||
Full power mode:
|
||||
- Time travel through your thinking
|
||||
- Complete project evolution tracking
|
||||
- Universal concepts across all work
|
||||
- Perfect organizational memory
|
||||
|
||||
## Success Stories (Hypothetical Examples)
|
||||
|
||||
### Marketing Agency
|
||||
*"We can now show clients exactly how their brand strategy evolved over 18 months. Every decision, every creative iteration, every strategic pivot - all connected and contextualized. Our clients love seeing the story of their brand's development."*
|
||||
|
||||
### Novel Author
|
||||
*"I'm writing a 7-book fantasy series. Every character, location, and plot thread is connected across all books. I can see how my themes developed from book 1 to book 7. It's like having a perfect memory of my entire creative universe."*
|
||||
|
||||
### Software Startup
|
||||
*"Our technical documentation evolves with our product. Every API change, every architectural decision, every user story is connected. New developers understand our thinking process, not just our current state. It's revolutionized our knowledge management."*
|
||||
|
||||
### Management Consultant
|
||||
*"I can show clients exactly how recommendations evolved through our engagement. Every meeting note, analysis, and strategic insight is connected. The story of our thinking is as valuable as our final recommendations."*
|
||||
|
||||
## The Future of Knowledge Work
|
||||
|
||||
Traditional file systems were built for computers. The Knowledge Layer is built for **human thinking**.
|
||||
|
||||
Instead of organizing information in rigid hierarchies, it mirrors how your mind actually works:
|
||||
- **Associative connections** between related ideas
|
||||
- **Evolutionary development** of concepts over time
|
||||
- **Contextual understanding** of why things matter
|
||||
- **Narrative structure** that tells the story of your work
|
||||
|
||||
This isn't just better file storage - it's a **new way of thinking** about knowledge itself.
|
||||
|
||||
Your files become extensions of your mind, with perfect memory, infinite connections, and deep understanding. They don't just store your work - they **amplify your intelligence**.
|
||||
|
||||
Welcome to the future of knowledge work. Your files are about to get **very** smart. 🧠✨
|
||||
|
||||
---
|
||||
|
||||
*Ready to transform your files from dumb storage into intelligent knowledge? The Knowledge Layer is available now in Brainy 4.0.*
|
||||
|
|
@ -1,524 +0,0 @@
|
|||
# Knowledge Layer API Documentation 📚🧠
|
||||
|
||||
## Overview
|
||||
|
||||
The Knowledge Layer transforms Brainy's VFS from a simple filesystem into an intelligent knowledge system where files understand themselves, track their evolution, and maintain relationships across time.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// Initialize Brainy with VFS
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
|
||||
// Enable Knowledge Layer - this augments VFS with intelligence features
|
||||
await vfs.enableKnowledgeLayer()
|
||||
|
||||
// Now your VFS has superpowers! 🚀
|
||||
// The Knowledge Layer dynamically adds new methods to the VFS instance:
|
||||
// - Event Recording: getHistory(), reconstructAtTime()
|
||||
// - Semantic Versioning: getVersions(), restoreVersion()
|
||||
// - Entity System: createEntity(), linkEntities(), findEntityOccurrences()
|
||||
// - Concepts: createConcept(), findByConcept()
|
||||
// - Git Bridge: exportToGit(), importFromGit()
|
||||
// - And many more...
|
||||
```
|
||||
|
||||
## How It Works: Method Augmentation
|
||||
|
||||
The Knowledge Layer uses a powerful augmentation pattern. When you call `enableKnowledgeLayer()`:
|
||||
|
||||
1. **Wraps Core Methods**: Intercepts existing VFS methods to add intelligence
|
||||
2. **Injects New Methods**: Dynamically adds new methods to the VFS instance
|
||||
3. **Background Processing**: Runs intelligence extraction asynchronously
|
||||
4. **Non-Breaking**: All existing code continues to work unchanged
|
||||
|
||||
```typescript
|
||||
// Before enableKnowledgeLayer() - Core VFS only
|
||||
vfs.writeFile() ✅ // Works
|
||||
vfs.readFile() ✅ // Works
|
||||
vfs.createEntity() ❌ // Method doesn't exist
|
||||
|
||||
// After enableKnowledgeLayer() - Enhanced VFS
|
||||
vfs.writeFile() ✅ // Still works, now with event recording
|
||||
vfs.readFile() ✅ // Still works, now tracks access patterns
|
||||
vfs.createEntity() ✅ // New method available!
|
||||
```
|
||||
|
||||
## Core Components
|
||||
|
||||
### 1. Event Recording - Complete History Tracking
|
||||
|
||||
Every file operation becomes a searchable event with full context.
|
||||
|
||||
```typescript
|
||||
// Automatic event recording
|
||||
await vfs.writeFile('/project/README.md', 'Initial README')
|
||||
await vfs.writeFile('/project/README.md', 'Updated README')
|
||||
|
||||
// Get complete history
|
||||
const history = await vfs.getHistory('/project/README.md')
|
||||
console.log(history)
|
||||
// [
|
||||
// { type: 'write', timestamp: 1234567890, content: Buffer('Updated README') },
|
||||
// { type: 'write', timestamp: 1234567880, content: Buffer('Initial README') }
|
||||
// ]
|
||||
|
||||
// Time travel - reconstruct file at any point
|
||||
const pastContent = await vfs.reconstructAtTime('/project/README.md', 1234567885)
|
||||
console.log(pastContent.toString()) // "Initial README"
|
||||
|
||||
// Get statistics
|
||||
const stats = await vfs.getStatistics('/project/README.md')
|
||||
console.log(stats)
|
||||
// {
|
||||
// totalEvents: 2,
|
||||
// totalWrites: 2,
|
||||
// totalBytes: 25,
|
||||
// authors: ['system'],
|
||||
// firstEvent: 1234567880,
|
||||
// lastEvent: 1234567890
|
||||
// }
|
||||
```
|
||||
|
||||
### 2. Semantic Versioning - Only Version When Meaning Changes
|
||||
|
||||
90% fewer versions than Git by only versioning when content meaning changes.
|
||||
|
||||
```typescript
|
||||
// Automatic semantic versioning
|
||||
await vfs.writeFile('/code/api.js', 'function login() { /* v1 */ }')
|
||||
await vfs.writeFile('/code/api.js', 'function login() { /* v1 with comment */ }') // No new version
|
||||
await vfs.writeFile('/code/api.js', 'function login() { return authenticate() }') // New version!
|
||||
|
||||
// Get versions (only meaningful ones)
|
||||
const versions = await vfs.getVersions('/code/api.js')
|
||||
console.log(versions.length) // 2 (not 3!)
|
||||
|
||||
// Get specific version
|
||||
const v1Content = await vfs.getVersion('/code/api.js', versions[1].id)
|
||||
|
||||
// Restore to previous version
|
||||
await vfs.restoreVersion('/code/api.js', versions[1].id)
|
||||
```
|
||||
|
||||
### 3. Persistent Entities - Universal Characters
|
||||
|
||||
Track entities that evolve across files and time. Not just story characters - APIs, customers, services, any evolving entity.
|
||||
|
||||
```typescript
|
||||
// Create a persistent entity
|
||||
const apiEntityId = await vfs.createEntity({
|
||||
name: 'User API',
|
||||
type: 'api',
|
||||
aliases: ['UserService', 'UserAPI'],
|
||||
attributes: { version: '1.0', methods: ['get', 'post'] }
|
||||
})
|
||||
|
||||
// Record appearance in a file
|
||||
await vfs.recordAppearance(
|
||||
apiEntityId,
|
||||
'/docs/api.md',
|
||||
'The User API provides endpoints for user management...'
|
||||
)
|
||||
|
||||
// Find all appearances
|
||||
const appearances = await vfs.findEntityAppearances(apiEntityId)
|
||||
|
||||
// Evolve the entity
|
||||
await vfs.evolveEntity(
|
||||
apiEntityId,
|
||||
{ attributes: { version: '2.0', methods: ['get', 'post', 'delete'] } },
|
||||
'/docs/changelog.md',
|
||||
'Added delete functionality'
|
||||
)
|
||||
|
||||
// Get evolution timeline
|
||||
const { entity, timeline } = await vfs.getEntityEvolution(apiEntityId)
|
||||
console.log(timeline) // All changes over time
|
||||
```
|
||||
|
||||
### 4. Universal Concepts - Ideas That Transcend Files
|
||||
|
||||
Concepts exist independently and can be linked to multiple manifestations.
|
||||
|
||||
```typescript
|
||||
// Create a universal concept
|
||||
const authConceptId = await vfs.createConcept({
|
||||
name: 'Authentication',
|
||||
domain: 'security',
|
||||
category: 'pattern',
|
||||
keywords: ['auth', 'login', 'security'],
|
||||
strength: 0.9,
|
||||
metadata: {}
|
||||
})
|
||||
|
||||
// Link concepts together
|
||||
await vfs.linkConcept(authConceptId, otherConceptId, 'related', {
|
||||
strength: 0.8,
|
||||
context: 'Both are security-related patterns'
|
||||
})
|
||||
|
||||
// Record manifestation
|
||||
await vfs.recordManifestation(
|
||||
authConceptId,
|
||||
'/src/auth.js',
|
||||
'class AuthService implements authentication...',
|
||||
'implementation'
|
||||
)
|
||||
|
||||
// Find concept appearances
|
||||
const manifestations = await vfs.findConceptAppearances(authConceptId)
|
||||
|
||||
// Get concept graph for visualization
|
||||
const graph = await vfs.getConceptGraph({ domain: 'security' })
|
||||
// { concepts: [...], links: [...] }
|
||||
```
|
||||
|
||||
### 5. Git Bridge - Import/Export Without Dependencies
|
||||
|
||||
Import any directory structure or export to Git-compatible format.
|
||||
|
||||
```typescript
|
||||
// Import from any directory (not just Git repos)
|
||||
const stats = await vfs.importFromGit(
|
||||
'/path/to/project',
|
||||
'/imported-project',
|
||||
{
|
||||
preserveGitHistory: true,
|
||||
extractMetadata: true
|
||||
}
|
||||
)
|
||||
|
||||
console.log(stats)
|
||||
// {
|
||||
// filesImported: 150,
|
||||
// eventsCreated: 75,
|
||||
// entitiesCreated: 12,
|
||||
// relationshipsCreated: 8
|
||||
// }
|
||||
|
||||
// Export to Git format
|
||||
const gitRepo = await vfs.exportToGit(
|
||||
'/my-project',
|
||||
'/export/git-repo',
|
||||
{
|
||||
preserveMetadata: true,
|
||||
preserveRelationships: true,
|
||||
commitMessage: 'Export from Brainy VFS'
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### Knowledge Augmentation Config
|
||||
|
||||
```typescript
|
||||
interface KnowledgeAugmentationConfig {
|
||||
enabled?: boolean // Master enable/disable (default: true)
|
||||
|
||||
eventRecording?: {
|
||||
enabled?: boolean // Record all operations (default: true)
|
||||
pruneAfterDays?: number // Auto-prune old events (default: 90)
|
||||
compressEvents?: boolean // Compress event storage (default: false)
|
||||
}
|
||||
|
||||
semanticVersioning?: {
|
||||
enabled?: boolean // Semantic versioning (default: true)
|
||||
threshold?: number // Semantic change threshold 0-1 (default: 0.3)
|
||||
maxVersions?: number // Max versions per file (default: 10)
|
||||
}
|
||||
|
||||
persistentEntities?: {
|
||||
enabled?: boolean // Entity system (default: true)
|
||||
autoExtract?: boolean // Auto-extract from content (default: false)
|
||||
}
|
||||
|
||||
concepts?: {
|
||||
enabled?: boolean // Concept system (default: true)
|
||||
autoLink?: boolean // Auto-link concepts (default: false)
|
||||
}
|
||||
|
||||
gitBridge?: {
|
||||
enabled?: boolean // Git import/export (default: true)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Component-Specific Configs
|
||||
|
||||
```typescript
|
||||
// Event Recorder
|
||||
const eventRecorder = new EventRecorder(brain)
|
||||
|
||||
// Semantic Versioning
|
||||
const versioning = new SemanticVersioning(brain, {
|
||||
threshold: 0.3, // Only version if >30% semantic change
|
||||
maxVersions: 10, // Keep max 10 versions per file
|
||||
minInterval: 60000 // Minimum 1 minute between versions
|
||||
})
|
||||
|
||||
// Persistent Entities
|
||||
const entities = new PersistentEntitySystem(brain, {
|
||||
autoExtract: true, // Auto-extract entities
|
||||
similarityThreshold: 0.8, // Entity matching threshold
|
||||
maxAppearances: 100, // Max appearances per entity
|
||||
evolutionTracking: true // Track entity evolution
|
||||
})
|
||||
|
||||
// Concepts
|
||||
const concepts = new ConceptSystem(brain, {
|
||||
autoLink: true, // Auto-link related concepts
|
||||
similarityThreshold: 0.7, // Concept similarity threshold
|
||||
maxManifestations: 1000, // Max manifestations per concept
|
||||
strengthDecay: 0.95 // Concept strength decay rate
|
||||
})
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 1. Story/Content Management
|
||||
|
||||
```typescript
|
||||
// Create characters that evolve across chapters
|
||||
const aragornId = await vfs.createEntity({
|
||||
name: 'Aragorn',
|
||||
type: 'character',
|
||||
aliases: ['Strider', 'King Elessar'],
|
||||
attributes: { role: 'ranger', status: 'heir' }
|
||||
})
|
||||
|
||||
// Track character development
|
||||
await vfs.recordAppearance(aragornId, '/chapters/01.md', 'Aragorn watched from the shadows...')
|
||||
await vfs.recordAppearance(aragornId, '/chapters/20.md', 'King Aragorn addressed his subjects...')
|
||||
|
||||
// Create universal themes
|
||||
const heroJourneyId = await vfs.createConcept({
|
||||
name: 'Hero\'s Journey',
|
||||
domain: 'narrative',
|
||||
category: 'pattern',
|
||||
keywords: ['hero', 'quest', 'transformation']
|
||||
})
|
||||
```
|
||||
|
||||
### 2. API Documentation That Evolves
|
||||
|
||||
```typescript
|
||||
// Track API endpoints as entities
|
||||
const userApiId = await vfs.createEntity({
|
||||
name: 'Users API',
|
||||
type: 'api',
|
||||
attributes: {
|
||||
endpoints: ['/users', '/users/:id'],
|
||||
version: '1.0'
|
||||
}
|
||||
})
|
||||
|
||||
// Document changes automatically
|
||||
await vfs.writeFile('/api-docs/users.md', newApiDocs) // Auto-versions if meaning changed
|
||||
await vfs.evolveEntity(userApiId, {
|
||||
attributes: { version: '1.1', endpoints: ['/users', '/users/:id', '/users/:id/profile'] }
|
||||
}, '/api-docs/users.md', 'Added profile endpoint')
|
||||
```
|
||||
|
||||
### 3. Research Knowledge Management
|
||||
|
||||
```typescript
|
||||
// Create research concepts
|
||||
const machineLearningId = await vfs.createConcept({
|
||||
name: 'Machine Learning',
|
||||
domain: 'ai',
|
||||
category: 'field',
|
||||
keywords: ['ML', 'artificial intelligence', 'algorithms']
|
||||
})
|
||||
|
||||
// Link related concepts
|
||||
await vfs.linkConcept(machineLearningId, deepLearningId, 'contains')
|
||||
await vfs.linkConcept(machineLearningId, statisticsId, 'uses')
|
||||
|
||||
// Track concept manifestations across papers
|
||||
await vfs.recordManifestation(machineLearningId, '/papers/survey.md', 'definition')
|
||||
await vfs.recordManifestation(machineLearningId, '/papers/experiment.md', 'usage')
|
||||
```
|
||||
|
||||
### 4. Codebase Intelligence
|
||||
|
||||
```typescript
|
||||
// Auto-extract classes and functions as entities
|
||||
const knowledge = new KnowledgeAugmentation({
|
||||
persistentEntities: { enabled: true, autoExtract: true },
|
||||
concepts: { enabled: true, autoLink: true }
|
||||
})
|
||||
|
||||
// Import existing codebase
|
||||
await vfs.importFromGit('/path/to/codebase', '/project')
|
||||
|
||||
// Get all extracted entities
|
||||
const entities = await vfs.findEntity({ type: 'class' })
|
||||
const functions = await vfs.findEntity({ type: 'function' })
|
||||
|
||||
// Track dependencies as relationships
|
||||
const serviceId = await vfs.findEntity({ name: 'UserService' })
|
||||
const controllerEntities = await vfs.findEntityAppearances(serviceId[0].id)
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
- **Event Recording**: 1000+ ops/second with automatic compression
|
||||
- **Semantic Versioning**: <10ms version checking with embedding cache
|
||||
- **Entity Tracking**: Handles millions of entities with graph optimization
|
||||
- **Concept System**: Sub-second similarity search across 100k+ concepts
|
||||
- **Git Bridge**: Imports 10k+ files/minute with parallel processing
|
||||
|
||||
## Storage Compatibility
|
||||
|
||||
Works with **ALL** Brainy storage adapters:
|
||||
- ✅ Memory (testing)
|
||||
- ✅ Redis (development)
|
||||
- ✅ PostgreSQL (production)
|
||||
- ✅ ChromaDB (vector-optimized)
|
||||
- ✅ Future adapters (only uses standard Brainy APIs)
|
||||
|
||||
## Migration from Traditional VFS
|
||||
|
||||
```typescript
|
||||
// Before: Basic VFS
|
||||
const vfs = new VirtualFileSystem(brain)
|
||||
|
||||
// After: VFS with Knowledge Layer
|
||||
const vfs = new VirtualFileSystem(brain)
|
||||
const knowledge = new KnowledgeAugmentation()
|
||||
await knowledge.init({ brain, vfs })
|
||||
|
||||
// All existing VFS methods still work exactly the same!
|
||||
await vfs.writeFile('/test.txt', 'content') // Now with intelligence
|
||||
```
|
||||
|
||||
## Advanced Patterns
|
||||
|
||||
### Custom Entity Extraction
|
||||
|
||||
```typescript
|
||||
// Override auto-extraction with custom patterns
|
||||
class CustomEntitySystem extends PersistentEntitySystem {
|
||||
async extractEntities(filePath: string, content: Buffer): Promise<string[]> {
|
||||
// Your custom extraction logic
|
||||
const text = content.toString()
|
||||
const apiEndpoints = text.match(/app\.(get|post|put|delete)\('([^']+)'/g)
|
||||
|
||||
const entityIds = []
|
||||
for (const endpoint of apiEndpoints || []) {
|
||||
const entityId = await this.createEntity({
|
||||
name: endpoint,
|
||||
type: 'endpoint',
|
||||
aliases: [],
|
||||
attributes: { method: /* extract method */, path: /* extract path */ }
|
||||
})
|
||||
entityIds.push(entityId)
|
||||
}
|
||||
return entityIds
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Custom Concept Detection
|
||||
|
||||
```typescript
|
||||
class CustomConceptSystem extends ConceptSystem {
|
||||
async extractAndLinkConcepts(filePath: string, content: Buffer): Promise<string[]> {
|
||||
// Domain-specific concept extraction
|
||||
const text = content.toString()
|
||||
|
||||
// Extract business concepts from comments
|
||||
const businessConcepts = text.match(/@business-concept:\s*([^\n]+)/g)
|
||||
|
||||
const conceptIds = []
|
||||
for (const match of businessConcepts || []) {
|
||||
const conceptName = match.split(':')[1].trim()
|
||||
const conceptId = await this.createConcept({
|
||||
name: conceptName,
|
||||
domain: 'business',
|
||||
category: 'concept',
|
||||
keywords: [conceptName.toLowerCase()],
|
||||
strength: 1.0,
|
||||
metadata: { source: 'annotation' }
|
||||
})
|
||||
conceptIds.push(conceptId)
|
||||
}
|
||||
|
||||
return conceptIds
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Event Stream Processing
|
||||
|
||||
```typescript
|
||||
// Listen to all VFS events for real-time processing
|
||||
const eventRecorder = new EventRecorder(brain)
|
||||
|
||||
// Custom event processor
|
||||
class RealTimeProcessor {
|
||||
async processEvent(event) {
|
||||
if (event.type === 'write' && event.path.endsWith('.md')) {
|
||||
// Process markdown files specially
|
||||
await this.extractMarkdownEntities(event.path, event.content)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Enable Gradually**: Start with basic features, enable advanced ones as needed
|
||||
2. **Tune Thresholds**: Adjust similarity/change thresholds for your domain
|
||||
3. **Custom Extraction**: Implement domain-specific entity/concept extraction
|
||||
4. **Monitor Performance**: Use built-in statistics and caching features
|
||||
5. **Version Strategy**: Consider your versioning needs when setting thresholds
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Q: Too many versions being created**
|
||||
A: Increase `semanticVersioning.threshold` from 0.3 to 0.5 or higher
|
||||
|
||||
**Q: Entity extraction not working**
|
||||
A: Check that `autoExtract: true` and consider custom extraction patterns
|
||||
|
||||
**Q: Poor concept linking**
|
||||
A: Tune `similarityThreshold` and ensure good concept descriptions
|
||||
|
||||
**Q: Slow performance**
|
||||
A: Enable caching, increase cache sizes, consider storage adapter choice
|
||||
|
||||
### Debug Mode
|
||||
|
||||
```typescript
|
||||
// Enable debug logging
|
||||
const knowledge = new KnowledgeAugmentation({
|
||||
// ... config
|
||||
debug: true // Logs all operations
|
||||
})
|
||||
|
||||
// Check system status
|
||||
const status = knowledge.getStatus()
|
||||
console.log(status) // All subsystem states
|
||||
|
||||
// Clear caches if needed
|
||||
semanticVersioning.clearCache()
|
||||
entitySystem.clearCache()
|
||||
conceptSystem.clearCache()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
The Knowledge Layer transforms your VFS from simple file storage into a living, breathing knowledge system. Files become intelligent, entities evolve, concepts transcend storage boundaries, and your data tells its own story.
|
||||
|
||||
Welcome to the future of filesystems! 🚀
|
||||
File diff suppressed because it is too large
Load diff
426
docs/vfs/NEURAL_EXTRACTION.md
Normal file
426
docs/vfs/NEURAL_EXTRACTION.md
Normal file
|
|
@ -0,0 +1,426 @@
|
|||
# Neural Extraction API - AI-Powered Concept and Entity Detection
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy's Neural Extraction system uses embeddings and a sophisticated NounType taxonomy to extract meaningful entities and concepts from text. Unlike simple regex-based extraction, neural extraction understands semantic meaning and context.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ brain.extractConcepts() │
|
||||
│ (High-level concept wrapper) │
|
||||
└──────────────┬──────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ brain.extract() │
|
||||
│ (Full entity extraction) │
|
||||
└──────────────┬──────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ NeuralEntityExtractor │
|
||||
│ (394-line production impl) │
|
||||
└──────────────┬──────────────────────┘
|
||||
│
|
||||
┌──────┴──────┐
|
||||
▼ ▼
|
||||
┌─────────┐ ┌──────────┐
|
||||
│ Pattern │ │ Embeddings│
|
||||
│ Matching│ │ + NounType│
|
||||
│ │ │ Taxonomy │
|
||||
└─────────┘ └──────────┘
|
||||
```
|
||||
|
||||
## NounType Taxonomy
|
||||
|
||||
Brainy uses a 30+ type taxonomy for entity classification:
|
||||
|
||||
### Core Types
|
||||
- **Person** - Individual humans
|
||||
- **Organization** - Companies, institutions, groups
|
||||
- **Location** - Places, cities, countries
|
||||
- **Event** - Occurrences, happenings
|
||||
- **Product** - Goods, services, items
|
||||
- **Concept** - Abstract ideas, theories
|
||||
- **Topic** - Subject areas, domains
|
||||
|
||||
### Technical Types
|
||||
- **API** - Application programming interfaces
|
||||
- **Service** - Software services
|
||||
- **Component** - System components
|
||||
- **Function** - Code functions
|
||||
- **Class** - Object-oriented classes
|
||||
- **Module** - Software modules
|
||||
|
||||
### Creative Types
|
||||
- **Character** - Fictional characters
|
||||
- **Setting** - Story locations
|
||||
- **Plot** - Story arcs
|
||||
- **Theme** - Narrative themes
|
||||
|
||||
### Business Types
|
||||
- **Customer** - Clients, users
|
||||
- **Project** - Business projects
|
||||
- **Process** - Business processes
|
||||
- **KPI** - Key performance indicators
|
||||
|
||||
And more...
|
||||
|
||||
## API Reference
|
||||
|
||||
### brain.extract(text, options)
|
||||
|
||||
Extracts entities from text with full configuration options.
|
||||
|
||||
**Parameters:**
|
||||
```typescript
|
||||
brain.extract(
|
||||
text: string,
|
||||
options?: {
|
||||
types?: NounType[] // Filter to specific types
|
||||
confidence?: number // Min confidence (0-1, default: 0.6)
|
||||
includeVectors?: boolean // Include embeddings
|
||||
neuralMatching?: boolean // Use neural classification (default: true)
|
||||
}
|
||||
): Promise<ExtractedEntity[]>
|
||||
```
|
||||
|
||||
**Returns:**
|
||||
```typescript
|
||||
interface ExtractedEntity {
|
||||
text: string // Extracted text
|
||||
type: NounType // Classified type
|
||||
position: number // Position in text
|
||||
confidence: number // Confidence score (0-1)
|
||||
vector?: number[] // Optional embedding
|
||||
}
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
const text = `
|
||||
The UserService API provides authentication and authorization.
|
||||
It integrates with the Database component for user storage.
|
||||
`
|
||||
|
||||
// Extract all entities
|
||||
const entities = await brain.extract(text)
|
||||
console.log(entities)
|
||||
// [
|
||||
// { text: 'UserService', type: 'Service', confidence: 0.87 },
|
||||
// { text: 'API', type: 'API', confidence: 0.92 },
|
||||
// { text: 'authentication', type: 'Concept', confidence: 0.79 },
|
||||
// { text: 'authorization', type: 'Concept', confidence: 0.81 },
|
||||
// { text: 'Database', type: 'Component', confidence: 0.85 }
|
||||
// ]
|
||||
|
||||
// Extract only technical entities
|
||||
const technical = await brain.extract(text, {
|
||||
types: [NounType.Service, NounType.API, NounType.Component],
|
||||
confidence: 0.8
|
||||
})
|
||||
console.log(technical)
|
||||
// [
|
||||
// { text: 'UserService', type: 'Service', confidence: 0.87 },
|
||||
// { text: 'API', type: 'API', confidence: 0.92 },
|
||||
// { text: 'Database', type: 'Component', confidence: 0.85 }
|
||||
// ]
|
||||
```
|
||||
|
||||
### brain.extractConcepts(text, options)
|
||||
|
||||
Simplified API specifically for concept extraction.
|
||||
|
||||
**Parameters:**
|
||||
```typescript
|
||||
brain.extractConcepts(
|
||||
text: string,
|
||||
options?: {
|
||||
confidence?: number // Min confidence (default: 0.7)
|
||||
limit?: number // Max concepts to return
|
||||
}
|
||||
): Promise<string[]>
|
||||
```
|
||||
|
||||
**Returns:**
|
||||
```typescript
|
||||
string[] // Array of concept names (deduplicated, lowercase)
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
const text = `
|
||||
Our authentication system uses JWT tokens for security.
|
||||
The authorization layer checks user permissions and roles.
|
||||
`
|
||||
|
||||
const concepts = await brain.extractConcepts(text, {
|
||||
confidence: 0.7,
|
||||
limit: 10
|
||||
})
|
||||
console.log(concepts)
|
||||
// ['authentication', 'security', 'authorization', 'permissions']
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### 1. Pattern-Based Candidate Detection
|
||||
|
||||
First, NeuralEntityExtractor scans text for potential entities using:
|
||||
- Capitalized words and phrases
|
||||
- Technical patterns (camelCase, PascalCase, UPPER_CASE)
|
||||
- Quoted strings
|
||||
- Common entity patterns
|
||||
|
||||
### 2. Embedding Generation
|
||||
|
||||
Each candidate is converted to a semantic embedding vector:
|
||||
```typescript
|
||||
const candidateVector = await brain.getEmbedding("UserService")
|
||||
// [0.234, -0.123, 0.567, ...] (1536 dimensions)
|
||||
```
|
||||
|
||||
### 3. NounType Classification
|
||||
|
||||
The embedding is compared against pre-computed embeddings for each NounType using cosine similarity:
|
||||
```typescript
|
||||
const serviceVector = typeEmbeddings.get(NounType.Service)
|
||||
const similarity = cosineSimilarity(candidateVector, serviceVector)
|
||||
// similarity = 0.87 (87% match)
|
||||
```
|
||||
|
||||
### 4. Context-Based Boosting
|
||||
|
||||
Confidence is adjusted based on surrounding context:
|
||||
```typescript
|
||||
// "The UserService API" gets boosted for Service type
|
||||
// "CEO of Company" gets boosted for Person type
|
||||
// "located in Paris" gets boosted for Location type
|
||||
```
|
||||
|
||||
### 5. Deduplication
|
||||
|
||||
Similar or overlapping entities are merged to avoid duplicates.
|
||||
|
||||
## VFS Integration
|
||||
|
||||
VFS automatically uses neural extraction when writing files:
|
||||
|
||||
```typescript
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
|
||||
// Automatic concept extraction (if enabled)
|
||||
await vfs.writeFile('/docs/api.md', `
|
||||
# User Authentication API
|
||||
|
||||
The UserService provides secure authentication using JWT tokens.
|
||||
Integrates with the Database for user storage.
|
||||
`, {
|
||||
intelligence: {
|
||||
autoConcepts: true // Enable concept extraction (default)
|
||||
}
|
||||
})
|
||||
|
||||
// Concepts automatically extracted and indexed:
|
||||
// ['authentication', 'security', 'database', 'user']
|
||||
|
||||
// Now searchable by concept
|
||||
const authFiles = await vfs.readdir('/by-concept/authentication')
|
||||
// Includes /docs/api.md
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
- **Candidate extraction**: O(n) where n = text length
|
||||
- **Embedding generation**: ~10-50ms per candidate (cached)
|
||||
- **Type classification**: O(t) where t = number of types to check
|
||||
- **Total time**: Typically 100-500ms for a document
|
||||
|
||||
**Caching:**
|
||||
- Embeddings are cached (UnifiedCache, 2GB default)
|
||||
- TypeEmbeddings precomputed once at initialization
|
||||
- Results cached for identical text
|
||||
|
||||
## Configuration
|
||||
|
||||
### VFS Auto-Concept Extraction
|
||||
|
||||
```typescript
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init({
|
||||
intelligence: {
|
||||
enabled: true, // Enable AI features (default: true)
|
||||
autoConcepts: true, // Auto-extract concepts (default: true)
|
||||
autoExtract: true, // Auto-extract entities (default: true)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Custom Confidence Thresholds
|
||||
|
||||
```typescript
|
||||
// Strict extraction (fewer, higher confidence)
|
||||
const strict = await brain.extract(text, { confidence: 0.9 })
|
||||
|
||||
// Permissive extraction (more entities, lower confidence)
|
||||
const permissive = await brain.extract(text, { confidence: 0.5 })
|
||||
```
|
||||
|
||||
### Type-Specific Extraction
|
||||
|
||||
```typescript
|
||||
// Extract only people and organizations
|
||||
const entities = await brain.extract(text, {
|
||||
types: [NounType.Person, NounType.Organization]
|
||||
})
|
||||
|
||||
// Extract only technical entities
|
||||
const technical = await brain.extract(text, {
|
||||
types: [
|
||||
NounType.API,
|
||||
NounType.Service,
|
||||
NounType.Component,
|
||||
NounType.Function
|
||||
]
|
||||
})
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Technical Documentation
|
||||
|
||||
```typescript
|
||||
const technicalDoc = `
|
||||
The PaymentService API integrates with Stripe for processing transactions.
|
||||
The OrderManager component coordinates between UserService and InventoryService.
|
||||
`
|
||||
|
||||
const entities = await brain.extract(technicalDoc, {
|
||||
types: [NounType.Service, NounType.API, NounType.Component]
|
||||
})
|
||||
|
||||
console.log(entities)
|
||||
// [
|
||||
// { text: 'PaymentService', type: 'Service', confidence: 0.89 },
|
||||
// { text: 'API', type: 'API', confidence: 0.93 },
|
||||
// { text: 'Stripe', type: 'Service', confidence: 0.76 },
|
||||
// { text: 'OrderManager', type: 'Component', confidence: 0.84 },
|
||||
// { text: 'UserService', type: 'Service', confidence: 0.91 },
|
||||
// { text: 'InventoryService', type: 'Service', confidence: 0.88 }
|
||||
// ]
|
||||
```
|
||||
|
||||
### Example 2: Creative Writing
|
||||
|
||||
```typescript
|
||||
const story = `
|
||||
Detective Sarah Chen arrived at the scene. The abandoned warehouse
|
||||
held secrets about the mysterious organization known as Shadow Corp.
|
||||
`
|
||||
|
||||
const entities = await brain.extract(story, {
|
||||
types: [NounType.Person, NounType.Location, NounType.Organization]
|
||||
})
|
||||
|
||||
console.log(entities)
|
||||
// [
|
||||
// { text: 'Detective Sarah Chen', type: 'Person', confidence: 0.94 },
|
||||
// { text: 'warehouse', type: 'Location', confidence: 0.71 },
|
||||
// { text: 'Shadow Corp', type: 'Organization', confidence: 0.82 }
|
||||
// ]
|
||||
```
|
||||
|
||||
### Example 3: Business Documents
|
||||
|
||||
```typescript
|
||||
const businessDoc = `
|
||||
Q3 revenue exceeded targets by 15%. The marketing team's new campaign
|
||||
generated 50,000 leads. Customer satisfaction remains our top KPI.
|
||||
`
|
||||
|
||||
const concepts = await brain.extractConcepts(businessDoc, {
|
||||
confidence: 0.65
|
||||
})
|
||||
|
||||
console.log(concepts)
|
||||
// ['revenue', 'marketing', 'campaign', 'leads', 'customer', 'satisfaction']
|
||||
|
||||
const entities = await brain.extract(businessDoc, {
|
||||
types: [NounType.KPI, NounType.Process, NounType.Customer]
|
||||
})
|
||||
// [
|
||||
// { text: 'revenue', type: 'KPI', confidence: 0.81 },
|
||||
// { text: 'Customer satisfaction', type: 'KPI', confidence: 0.87 }
|
||||
// ]
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### With Vector Embeddings
|
||||
|
||||
```typescript
|
||||
const entities = await brain.extract(text, {
|
||||
includeVectors: true
|
||||
})
|
||||
|
||||
// Use embeddings for custom similarity search
|
||||
for (const entity of entities) {
|
||||
const similar = await brain.similar(entity.vector, { limit: 5 })
|
||||
console.log(`Similar to ${entity.text}:`, similar)
|
||||
}
|
||||
```
|
||||
|
||||
### Custom Entity Processing
|
||||
|
||||
```typescript
|
||||
const entities = await brain.extract(text)
|
||||
|
||||
// Group by type
|
||||
const byType = entities.reduce((acc, entity) => {
|
||||
if (!acc[entity.type]) acc[entity.type] = []
|
||||
acc[entity.type].push(entity)
|
||||
return acc
|
||||
}, {})
|
||||
|
||||
console.log('Services:', byType[NounType.Service])
|
||||
console.log('APIs:', byType[NounType.API])
|
||||
console.log('Concepts:', byType[NounType.Concept])
|
||||
```
|
||||
|
||||
### Batch Processing
|
||||
|
||||
```typescript
|
||||
const documents = [doc1, doc2, doc3, /* ... */]
|
||||
|
||||
// Extract concepts from all documents
|
||||
const allConcepts = await Promise.all(
|
||||
documents.map(doc => brain.extractConcepts(doc))
|
||||
)
|
||||
|
||||
// Combine and deduplicate
|
||||
const uniqueConcepts = [...new Set(allConcepts.flat())]
|
||||
console.log('All concepts:', uniqueConcepts)
|
||||
```
|
||||
|
||||
## Zero-Config Design
|
||||
|
||||
Neural extraction works out-of-the-box:
|
||||
- ✅ No configuration required
|
||||
- ✅ No training needed
|
||||
- ✅ No API keys needed
|
||||
- ✅ Works with all storage adapters
|
||||
- ✅ Automatic caching and optimization
|
||||
- ✅ Sensible defaults for all parameters
|
||||
|
||||
## See Also
|
||||
|
||||
- [VFS Core Documentation](./VFS_CORE.md) - Complete filesystem API
|
||||
- [Semantic VFS](./SEMANTIC_VFS.md) - Multi-dimensional file access
|
||||
- [Triple Intelligence™](../architecture/triple-intelligence.md) - Underlying architecture
|
||||
- [NounType Taxonomy](../architecture/noun-verb-taxonomy.md) - Complete type reference
|
||||
727
docs/vfs/PROJECTION_STRATEGY_API.md
Normal file
727
docs/vfs/PROJECTION_STRATEGY_API.md
Normal file
|
|
@ -0,0 +1,727 @@
|
|||
# Projection Strategy API
|
||||
|
||||
## Creating Custom Semantic Dimensions
|
||||
|
||||
Projection strategies allow you to create custom ways to organize and access files in Semantic VFS. This guide shows you how to build your own.
|
||||
|
||||
---
|
||||
|
||||
## What is a Projection Strategy?
|
||||
|
||||
A **projection strategy** maps a semantic dimension (like "priority" or "language") to actual file entities using Brainy queries.
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
/by-priority/high → Files with metadata.priority = 'high'
|
||||
/by-language/typescript → Files with .ts extension
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Interface Definition
|
||||
|
||||
Every projection must implement the `ProjectionStrategy` interface:
|
||||
|
||||
```typescript
|
||||
export interface ProjectionStrategy {
|
||||
/**
|
||||
* Unique name for this dimension
|
||||
* Used in paths like: /by-{name}/...
|
||||
*/
|
||||
readonly name: string
|
||||
|
||||
/**
|
||||
* Convert dimension value to Brainy FindParams
|
||||
* This is for documentation/debugging (not always used)
|
||||
*
|
||||
* @param value - The dimension value (e.g., 'high' for priority)
|
||||
* @param subpath - Optional file filter within dimension
|
||||
*/
|
||||
toQuery(value: any, subpath?: string): FindParams
|
||||
|
||||
/**
|
||||
* Resolve dimension value to entity IDs
|
||||
* This is the MAIN method that does the work
|
||||
*
|
||||
* @param brain - Brainy instance (use brain.find, brain.similar, etc.)
|
||||
* @param vfs - VirtualFileSystem instance
|
||||
* @param value - The dimension value to resolve
|
||||
* @returns Array of entity IDs matching this dimension
|
||||
*/
|
||||
resolve(brain: Brainy, vfs: VirtualFileSystem, value: any): Promise<string[]>
|
||||
|
||||
/**
|
||||
* OPTIONAL: List all items in this dimension
|
||||
* Used for directory listings like: readdir('/by-priority')
|
||||
*
|
||||
* @param brain - Brainy instance
|
||||
* @param vfs - VirtualFileSystem instance
|
||||
* @param limit - Max results to return
|
||||
*/
|
||||
list?(brain: Brainy, vfs: VirtualFileSystem, limit?: number): Promise<VFSEntity[]>
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start: Priority Projection
|
||||
|
||||
Let's build a projection that organizes files by priority (high, medium, low):
|
||||
|
||||
### Step 1: Create the Strategy Class
|
||||
|
||||
```typescript
|
||||
import { BaseProjectionStrategy } from '@soulcraft/brainy/vfs/semantic'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { VirtualFileSystem, VFSEntity } from '@soulcraft/brainy/vfs'
|
||||
|
||||
export class PriorityProjection extends BaseProjectionStrategy {
|
||||
readonly name = 'priority'
|
||||
|
||||
/**
|
||||
* Convert priority value to FindParams
|
||||
*/
|
||||
toQuery(priority: string, subpath?: string) {
|
||||
const query = {
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
priority: priority // Match metadata.priority field
|
||||
},
|
||||
limit: 1000
|
||||
}
|
||||
|
||||
// Filter by filename if subpath provided
|
||||
if (subpath) {
|
||||
query.where = {
|
||||
...query.where,
|
||||
anyOf: [
|
||||
{ name: subpath },
|
||||
{ path: { endsWith: subpath } }
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
return query
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve priority to entity IDs
|
||||
*/
|
||||
async resolve(brain: Brainy, vfs: VirtualFileSystem, priority: string): Promise<string[]> {
|
||||
// Query Brainy for files with this priority
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
priority: priority
|
||||
},
|
||||
limit: 1000
|
||||
})
|
||||
|
||||
// Extract entity IDs using helper from base class
|
||||
return this.extractIds(results)
|
||||
}
|
||||
|
||||
/**
|
||||
* List all files that have priority metadata
|
||||
*/
|
||||
async list(brain: Brainy, vfs: VirtualFileSystem, limit = 100): Promise<VFSEntity[]> {
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
priority: { exists: true }
|
||||
},
|
||||
limit
|
||||
})
|
||||
|
||||
return results.map(r => r.entity as VFSEntity)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 2: Register the Strategy
|
||||
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { PriorityProjection } from './PriorityProjection'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
|
||||
// Register custom projection
|
||||
// TODO: This will be exposed as public API
|
||||
// For now, access via internal property
|
||||
vfs['projectionRegistry'].register(new PriorityProjection())
|
||||
```
|
||||
|
||||
### Step 3: Use It!
|
||||
|
||||
```typescript
|
||||
// Write files with priority metadata
|
||||
await vfs.writeFile('/src/critical-fix.ts', code, {
|
||||
metadata: { priority: 'high' }
|
||||
})
|
||||
|
||||
await vfs.writeFile('/src/nice-to-have.ts', code, {
|
||||
metadata: { priority: 'low' }
|
||||
})
|
||||
|
||||
// Access by priority
|
||||
const highPriority = await vfs.readdir('/by-priority/high')
|
||||
console.log(highPriority) // ['critical-fix.ts']
|
||||
|
||||
const lowPriority = await vfs.readdir('/by-priority/low')
|
||||
console.log(lowPriority) // ['nice-to-have.ts']
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Base Class Helpers
|
||||
|
||||
`BaseProjectionStrategy` provides utility methods:
|
||||
|
||||
### `extractIds(results: Result[]): string[]`
|
||||
Extracts entity IDs from Brainy query results:
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({ where: { ... } })
|
||||
return this.extractIds(results) // ['id1', 'id2', ...]
|
||||
```
|
||||
|
||||
### `filterFiles(brain: Brainy, ids: string[]): Promise<string[]>`
|
||||
Filters to only file entities (removes directories):
|
||||
|
||||
```typescript
|
||||
const allIds = await this.traverseGraph(...)
|
||||
return await this.filterFiles(brain, allIds) // Only files
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Examples
|
||||
|
||||
### Example 1: Language Projection
|
||||
|
||||
Organize files by programming language:
|
||||
|
||||
```typescript
|
||||
export class LanguageProjection extends BaseProjectionStrategy {
|
||||
readonly name = 'language'
|
||||
|
||||
// Map extensions to languages
|
||||
private languageMap = {
|
||||
ts: 'typescript',
|
||||
js: 'javascript',
|
||||
py: 'python',
|
||||
go: 'go',
|
||||
rs: 'rust'
|
||||
}
|
||||
|
||||
toQuery(language: string, subpath?: string) {
|
||||
// Find extension for this language
|
||||
const ext = Object.entries(this.languageMap)
|
||||
.find(([_, lang]) => lang === language)?.[0]
|
||||
|
||||
return {
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
extension: ext
|
||||
},
|
||||
limit: 1000
|
||||
}
|
||||
}
|
||||
|
||||
async resolve(brain: Brainy, vfs: VirtualFileSystem, language: string): Promise<string[]> {
|
||||
const ext = Object.entries(this.languageMap)
|
||||
.find(([_, lang]) => lang === language)?.[0]
|
||||
|
||||
if (!ext) return []
|
||||
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
extension: ext
|
||||
},
|
||||
limit: 5000
|
||||
})
|
||||
|
||||
return this.extractIds(results)
|
||||
}
|
||||
|
||||
async list(brain: Brainy, vfs: VirtualFileSystem, limit = 100): Promise<VFSEntity[]> {
|
||||
// Return sample files from each language
|
||||
const results = await brain.find({
|
||||
where: { vfsType: 'file' },
|
||||
limit
|
||||
})
|
||||
|
||||
return results.map(r => r.entity as VFSEntity)
|
||||
}
|
||||
}
|
||||
|
||||
// Usage:
|
||||
// /by-language/typescript → All .ts files
|
||||
// /by-language/python → All .py files
|
||||
```
|
||||
|
||||
### Example 2: Size Projection
|
||||
|
||||
Organize files by size category:
|
||||
|
||||
```typescript
|
||||
export class SizeProjection extends BaseProjectionStrategy {
|
||||
readonly name = 'size'
|
||||
|
||||
// Size categories in bytes
|
||||
private readonly categories = {
|
||||
tiny: [0, 1024], // < 1 KB
|
||||
small: [1024, 102400], // 1-100 KB
|
||||
medium: [102400, 1048576], // 100 KB - 1 MB
|
||||
large: [1048576, Infinity] // > 1 MB
|
||||
}
|
||||
|
||||
toQuery(category: string, subpath?: string) {
|
||||
const [min, max] = this.categories[category] || [0, Infinity]
|
||||
|
||||
return {
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
size: {
|
||||
greaterEqual: min,
|
||||
lessThan: max
|
||||
}
|
||||
},
|
||||
limit: 1000
|
||||
}
|
||||
}
|
||||
|
||||
async resolve(brain: Brainy, vfs: VirtualFileSystem, category: string): Promise<string[]> {
|
||||
const [min, max] = this.categories[category]
|
||||
if (!min && min !== 0) return []
|
||||
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
size: {
|
||||
greaterEqual: min,
|
||||
lessThan: max
|
||||
}
|
||||
},
|
||||
limit: 1000
|
||||
})
|
||||
|
||||
return this.extractIds(results)
|
||||
}
|
||||
|
||||
async list(brain: Brainy, vfs: VirtualFileSystem, limit = 100): Promise<VFSEntity[]> {
|
||||
// Return files sorted by size
|
||||
const results = await brain.find({
|
||||
where: { vfsType: 'file' },
|
||||
limit
|
||||
})
|
||||
|
||||
return results
|
||||
.map(r => r.entity as VFSEntity)
|
||||
.sort((a, b) => (b.metadata.size || 0) - (a.metadata.size || 0))
|
||||
}
|
||||
}
|
||||
|
||||
// Usage:
|
||||
// /by-size/tiny → Files < 1 KB
|
||||
// /by-size/large → Files > 1 MB
|
||||
```
|
||||
|
||||
### Example 3: Status Projection (Custom Logic)
|
||||
|
||||
Organize files by review status with custom logic:
|
||||
|
||||
```typescript
|
||||
export class StatusProjection extends BaseProjectionStrategy {
|
||||
readonly name = 'status'
|
||||
|
||||
toQuery(status: string, subpath?: string) {
|
||||
return {
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
reviewStatus: status
|
||||
},
|
||||
limit: 1000
|
||||
}
|
||||
}
|
||||
|
||||
async resolve(brain: Brainy, vfs: VirtualFileSystem, status: string): Promise<string[]> {
|
||||
// Custom logic: "needs-review" means modified in last 24h without review
|
||||
if (status === 'needs-review') {
|
||||
const oneDayAgo = Date.now() - (24 * 60 * 60 * 1000)
|
||||
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
modified: { greaterEqual: oneDayAgo },
|
||||
reviewStatus: { missing: true } // No review status set
|
||||
},
|
||||
limit: 1000
|
||||
})
|
||||
|
||||
return this.extractIds(results)
|
||||
}
|
||||
|
||||
// Standard status query
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
reviewStatus: status
|
||||
},
|
||||
limit: 1000
|
||||
})
|
||||
|
||||
return this.extractIds(results)
|
||||
}
|
||||
|
||||
async list(brain: Brainy, vfs: VirtualFileSystem, limit = 100): Promise<VFSEntity[]> {
|
||||
// Return files with any review status
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
anyOf: [
|
||||
{ reviewStatus: { exists: true } },
|
||||
{ modified: { greaterEqual: Date.now() - 86400000 } }
|
||||
]
|
||||
},
|
||||
limit
|
||||
})
|
||||
|
||||
return results.map(r => r.entity as VFSEntity)
|
||||
}
|
||||
}
|
||||
|
||||
// Usage:
|
||||
// /by-status/needs-review → Files modified in last 24h without review
|
||||
// /by-status/approved → Approved files
|
||||
// /by-status/rejected → Rejected files
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Using Brainy Field Operators (BFO)
|
||||
|
||||
Projection strategies use **Brainy Field Operators** (BFO), not MongoDB-style operators:
|
||||
|
||||
### Comparison Operators
|
||||
```typescript
|
||||
// ❌ MongoDB style (WRONG)
|
||||
{ size: { $gte: 1000, $lte: 5000 } }
|
||||
|
||||
// ✅ BFO style (CORRECT)
|
||||
{ size: { greaterEqual: 1000, lessEqual: 5000 } }
|
||||
```
|
||||
|
||||
### Logical Operators
|
||||
```typescript
|
||||
// ❌ MongoDB style (WRONG)
|
||||
{ $or: [{ name: 'foo' }, { name: 'bar' }] }
|
||||
|
||||
// ✅ BFO style (CORRECT)
|
||||
{ anyOf: [{ name: 'foo' }, { name: 'bar' }] }
|
||||
```
|
||||
|
||||
### Existence Operators
|
||||
```typescript
|
||||
// ❌ MongoDB style (WRONG)
|
||||
{ tags: { $exists: true } }
|
||||
|
||||
// ✅ BFO style (CORRECT)
|
||||
{ tags: { exists: true } }
|
||||
```
|
||||
|
||||
### String Operators
|
||||
```typescript
|
||||
// ❌ MongoDB style (WRONG)
|
||||
{ path: { $regex: /\.ts$/ } }
|
||||
|
||||
// ✅ BFO style (CORRECT)
|
||||
{ path: { endsWith: '.ts' } }
|
||||
```
|
||||
|
||||
### Full BFO Operator Reference
|
||||
|
||||
```typescript
|
||||
// Comparison
|
||||
{ field: value } // Exact match
|
||||
{ field: { greaterThan: 10 } } // >
|
||||
{ field: { greaterEqual: 10 } } // >=
|
||||
{ field: { lessThan: 10 } } // <
|
||||
{ field: { lessEqual: 10 } } // <=
|
||||
{ field: { not: value } } // !=
|
||||
|
||||
// Logical
|
||||
{ anyOf: [{ a: 1 }, { b: 2 }] } // OR
|
||||
{ allOf: [{ a: 1 }, { b: 2 }] } // AND
|
||||
|
||||
// Existence
|
||||
{ field: { exists: true } } // Field exists
|
||||
{ field: { missing: true } } // Field doesn't exist
|
||||
|
||||
// String
|
||||
{ field: { startsWith: 'prefix' } } // Starts with
|
||||
{ field: { endsWith: 'suffix' } } // Ends with
|
||||
{ field: { matches: 'pattern' } } // Regex match
|
||||
|
||||
// Array
|
||||
{ array: { contains: 'item' } } // Array contains item
|
||||
{ array: { hasAll: ['a', 'b'] } } // Has all items
|
||||
{ array: { oneOf: ['a', 'b', 'c'] } } // Value in list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Guidelines
|
||||
|
||||
### 1. Use Indexes
|
||||
All metadata fields are automatically indexed. Use direct equality or range queries for best performance:
|
||||
|
||||
```typescript
|
||||
// ✅ Fast: Direct index lookup (O(log n))
|
||||
{ priority: 'high' }
|
||||
{ size: { greaterEqual: 1000 } }
|
||||
|
||||
// ⚠️ Slower: Must scan results
|
||||
{ path: { matches: /complex-regex/ } }
|
||||
```
|
||||
|
||||
### 2. Limit Results
|
||||
Always set reasonable limits:
|
||||
|
||||
```typescript
|
||||
async resolve(brain, vfs, value) {
|
||||
const results = await brain.find({
|
||||
where: { ... },
|
||||
limit: 1000 // Prevent unbounded queries
|
||||
})
|
||||
return this.extractIds(results)
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Avoid Post-Filtering When Possible
|
||||
If you need post-filtering, consider flattening data:
|
||||
|
||||
```typescript
|
||||
// ❌ Slow: Fetch 5000, filter in memory
|
||||
const all = await brain.find({ where: { type: 'file' }, limit: 5000 })
|
||||
return all.filter(item => item.metadata.nested.value === target)
|
||||
|
||||
// ✅ Fast: Flatten during write, query directly
|
||||
// Store: metadata.nested_value = target
|
||||
const results = await brain.find({
|
||||
where: { nested_value: target },
|
||||
limit: 1000
|
||||
})
|
||||
```
|
||||
|
||||
### 4. Cache Expensive Operations
|
||||
Use the projection's resolve cache:
|
||||
|
||||
```typescript
|
||||
// Automatic caching in SemanticPathResolver
|
||||
// Results cached for 5 minutes by default
|
||||
// No manual caching needed!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Projections
|
||||
|
||||
### Unit Test Example
|
||||
|
||||
```typescript
|
||||
import { describe, it, expect, beforeAll } from 'vitest'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { PriorityProjection } from './PriorityProjection'
|
||||
|
||||
describe('PriorityProjection', () => {
|
||||
let brain: Brainy
|
||||
let vfs: any
|
||||
let projection: PriorityProjection
|
||||
|
||||
beforeAll(async () => {
|
||||
brain = new Brainy()
|
||||
await brain.init()
|
||||
vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
projection = new PriorityProjection()
|
||||
})
|
||||
|
||||
it('should resolve high priority files', async () => {
|
||||
// Create test files
|
||||
await vfs.writeFile('/test1.ts', 'code', {
|
||||
metadata: { priority: 'high' }
|
||||
})
|
||||
await vfs.writeFile('/test2.ts', 'code', {
|
||||
metadata: { priority: 'low' }
|
||||
})
|
||||
|
||||
// Resolve high priority
|
||||
const ids = await projection.resolve(brain, vfs, 'high')
|
||||
|
||||
expect(ids).toHaveLength(1)
|
||||
|
||||
const entity = await brain.get(ids[0])
|
||||
expect(entity.metadata.priority).toBe('high')
|
||||
})
|
||||
|
||||
it('should list all files with priority', async () => {
|
||||
const entities = await projection.list(brain, vfs, 100)
|
||||
|
||||
expect(entities.length).toBeGreaterThan(0)
|
||||
expect(entities.every(e => e.metadata.priority)).toBe(true)
|
||||
})
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. ✅ Name projections clearly
|
||||
```typescript
|
||||
// ✅ Good
|
||||
readonly name = 'priority' // /by-priority/high
|
||||
readonly name = 'language' // /by-language/typescript
|
||||
|
||||
// ❌ Bad
|
||||
readonly name = 'proj1' // /by-proj1/??? unclear
|
||||
```
|
||||
|
||||
### 2. ✅ Document expected metadata
|
||||
```typescript
|
||||
/**
|
||||
* Priority Projection
|
||||
*
|
||||
* Requires metadata fields:
|
||||
* - priority: string ('high' | 'medium' | 'low')
|
||||
*
|
||||
* Usage:
|
||||
* /by-priority/high
|
||||
*/
|
||||
export class PriorityProjection extends BaseProjectionStrategy {
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### 3. ✅ Handle missing data gracefully
|
||||
```typescript
|
||||
async resolve(brain, vfs, value) {
|
||||
const results = await brain.find({
|
||||
where: { priority: value },
|
||||
limit: 1000
|
||||
})
|
||||
|
||||
// Return empty array if no results, don't throw
|
||||
return this.extractIds(results) // [] if empty
|
||||
}
|
||||
```
|
||||
|
||||
### 4. ✅ Validate input
|
||||
```typescript
|
||||
async resolve(brain, vfs, priority: string) {
|
||||
// Validate priority value
|
||||
const valid = ['high', 'medium', 'low']
|
||||
if (!valid.includes(priority)) {
|
||||
return [] // Or throw error
|
||||
}
|
||||
|
||||
// Continue with query...
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Pattern 1: Enum-Based Projection
|
||||
For fixed sets of values (status, priority, type):
|
||||
|
||||
```typescript
|
||||
private readonly validValues = ['draft', 'review', 'approved']
|
||||
|
||||
async resolve(brain, vfs, status: string) {
|
||||
if (!this.validValues.includes(status)) return []
|
||||
// ... query
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2: Range-Based Projection
|
||||
For numeric or time ranges:
|
||||
|
||||
```typescript
|
||||
private readonly ranges = {
|
||||
recent: Date.now() - 86400000, // Last 24h
|
||||
week: Date.now() - 7 * 86400000, // Last week
|
||||
month: Date.now() - 30 * 86400000 // Last month
|
||||
}
|
||||
|
||||
async resolve(brain, vfs, period: string) {
|
||||
const since = this.ranges[period]
|
||||
if (!since) return []
|
||||
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
modified: { greaterEqual: since }
|
||||
}
|
||||
})
|
||||
return this.extractIds(results)
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: Computed Projection
|
||||
Combine multiple criteria:
|
||||
|
||||
```typescript
|
||||
async resolve(brain, vfs, value: string) {
|
||||
// "stale" = not modified in 30 days AND no recent access
|
||||
if (value === 'stale') {
|
||||
const thirtyDaysAgo = Date.now() - 30 * 86400000
|
||||
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
allOf: [
|
||||
{ modified: { lessThan: thirtyDaysAgo } },
|
||||
{ accessed: { lessThan: thirtyDaysAgo } }
|
||||
]
|
||||
}
|
||||
})
|
||||
return this.extractIds(results)
|
||||
}
|
||||
|
||||
// Regular query for other values...
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Projection returns empty results
|
||||
1. Check metadata exists: `console.log(entity.metadata)`
|
||||
2. Verify query syntax: Use BFO operators, not MongoDB
|
||||
3. Check limits: Increase limit if needed
|
||||
|
||||
### Slow performance
|
||||
1. Check if field is indexed: All metadata fields are auto-indexed
|
||||
2. Avoid post-filtering: Flatten complex structures
|
||||
3. Use appropriate limits: Don't fetch more than needed
|
||||
|
||||
### Type errors
|
||||
1. Import correct types: `import { Brainy, VirtualFileSystem } from '@soulcraft/brainy'`
|
||||
2. Use `as VFSEntity` when mapping results
|
||||
3. Check BaseProjectionStrategy import
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Semantic VFS Guide](./SEMANTIC_VFS.md) - Using semantic paths
|
||||
- [Performance Tuning](./PERFORMANCE_TUNING.md) - Optimization guide
|
||||
- [VFS Core API](./VFS_CORE.md) - Base VFS operations
|
||||
|
|
@ -286,7 +286,7 @@ By using this quick start, you avoided these common mistakes:
|
|||
Your file explorer is now working! Here's what to explore next:
|
||||
|
||||
1. **[File Operations](./VFS_API_GUIDE.md#file-operations)** - Read, write, and manipulate files
|
||||
2. **[Semantic Features](./VFS_KNOWLEDGE_LAYER.md)** - Connect files to concepts and entities
|
||||
2. **[Semantic Features](./SEMANTIC_VFS.md)** - Multi-dimensional file access and neural extraction
|
||||
3. **[Performance Optimization](./building-file-explorers.md#performance)** - Handle large directories efficiently
|
||||
4. **[Advanced Search](./VFS_API_GUIDE.md#search-operations)** - Complex queries and filters
|
||||
|
||||
|
|
|
|||
|
|
@ -2,13 +2,14 @@
|
|||
|
||||
> Transform your filesystem into an intelligent knowledge graph where every file is a living entity with semantic understanding, relationships, and AI-powered organization.
|
||||
|
||||
## 🧠 Knowledge Layer Available!
|
||||
## 📚 Complete VFS Documentation
|
||||
|
||||
**For advanced features like semantic versioning, persistent entities, and universal concepts, see:**
|
||||
- **[Knowledge Layer Overview](../KNOWLEDGE_LAYER_OVERVIEW.md)** - Non-technical introduction
|
||||
- **[Knowledge Layer API](KNOWLEDGE_LAYER_API.md)** - Advanced features and capabilities
|
||||
- **[Real-World Examples](KNOWLEDGE_LAYER_EXAMPLES.md)** - Complete use cases for stories, APIs, research, and more
|
||||
- **[VFS API Guide](VFS_API_GUIDE.md)** - Complete filesystem API reference
|
||||
**Essential guides to get started:**
|
||||
- **[VFS Core](VFS_CORE.md)** - Complete filesystem architecture and API
|
||||
- **[Semantic VFS](SEMANTIC_VFS.md)** - Multi-dimensional file access (6 semantic dimensions)
|
||||
- **[Neural Extraction](NEURAL_EXTRACTION.md)** - AI-powered concept and entity extraction
|
||||
- **[Examples & Scenarios](VFS_EXAMPLES_SCENARIOS.md)** - Real-world use cases and code
|
||||
- **[VFS API Guide](VFS_API_GUIDE.md)** - Complete API reference
|
||||
|
||||
## What is Brainy VFS?
|
||||
|
||||
|
|
|
|||
486
docs/vfs/SEMANTIC_VFS.md
Normal file
486
docs/vfs/SEMANTIC_VFS.md
Normal file
|
|
@ -0,0 +1,486 @@
|
|||
# Semantic VFS - Revolutionary File System
|
||||
|
||||
## What is Semantic VFS?
|
||||
|
||||
Semantic VFS transforms traditional hierarchical file systems into **multi-dimensional knowledge graphs**. The same file can be accessed through multiple semantic dimensions simultaneously.
|
||||
|
||||
### Traditional vs Semantic
|
||||
|
||||
**Traditional File Systems:**
|
||||
```
|
||||
/src/auth/login.ts # One path, one location
|
||||
/src/users/profile.ts # Separate location
|
||||
```
|
||||
|
||||
**Semantic VFS:**
|
||||
```
|
||||
# Traditional path (still works!)
|
||||
/src/auth/login.ts
|
||||
|
||||
# By concept
|
||||
/by-concept/authentication/login.ts
|
||||
/by-concept/security/login.ts
|
||||
|
||||
# By author
|
||||
/by-author/alice/login.ts
|
||||
|
||||
# By time
|
||||
/as-of/2024-03-15/login.ts
|
||||
|
||||
# By relationship
|
||||
/related-to/src/users/profile.ts/depth-2
|
||||
```
|
||||
|
||||
**The same file, accessible 6+ different ways!** This is **polymorphic file access**.
|
||||
|
||||
---
|
||||
|
||||
## Why Semantic VFS?
|
||||
|
||||
### 1. **Natural Organization**
|
||||
Developers think in concepts, not directories:
|
||||
```typescript
|
||||
// Find all authentication-related files
|
||||
const authFiles = await vfs.readdir('/by-concept/authentication')
|
||||
|
||||
// Find all files Alice worked on
|
||||
const aliceFiles = await vfs.readdir('/by-author/alice')
|
||||
```
|
||||
|
||||
### 2. **Time Travel**
|
||||
See your codebase as it existed at any point:
|
||||
```typescript
|
||||
// Code from March 15th
|
||||
const snapshot = await vfs.readdir('/as-of/2024-03-15')
|
||||
|
||||
// Compare with today
|
||||
const current = await vfs.readdir('/src')
|
||||
```
|
||||
|
||||
### 3. **Knowledge Graph Navigation**
|
||||
Navigate by semantic relationships:
|
||||
```typescript
|
||||
// Files related to auth system (within 2 hops)
|
||||
const related = await vfs.readdir('/related-to/src/auth.ts/depth-2')
|
||||
|
||||
// Files similar to this implementation
|
||||
const similar = await vfs.readdir('/similar-to/src/auth.ts/threshold-0.8')
|
||||
```
|
||||
|
||||
### 4. **Tag-Based Organization**
|
||||
Organize by purpose, not location:
|
||||
```typescript
|
||||
// All security-critical files
|
||||
const security = await vfs.readdir('/by-tag/security')
|
||||
|
||||
// All experimental features
|
||||
const experiments = await vfs.readdir('/by-tag/experimental')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Supported Semantic Dimensions
|
||||
|
||||
### 1. Traditional Path (Hierarchical)
|
||||
```typescript
|
||||
await vfs.readFile('/src/auth/login.ts')
|
||||
// Works exactly like a normal filesystem
|
||||
```
|
||||
|
||||
### 2. By Concept (Semantic)
|
||||
```typescript
|
||||
await vfs.readdir('/by-concept/authentication')
|
||||
// Returns all files about authentication
|
||||
|
||||
await vfs.readFile('/by-concept/authentication/login.ts')
|
||||
// Find specific file within concept
|
||||
```
|
||||
|
||||
**How it works:** Uses `brain.extractConcepts()` with NeuralEntityExtractor to extract concepts from file content using embeddings and the NounType taxonomy. Indexes concept names for O(log n) queries. See [Neural Extraction API](./NEURAL_EXTRACTION.md) for details.
|
||||
|
||||
### 3. By Author (Ownership)
|
||||
```typescript
|
||||
await vfs.readdir('/by-author/alice')
|
||||
// All files owned/modified by alice
|
||||
|
||||
await vfs.stat('/by-author/alice/config.ts')
|
||||
// Check specific file
|
||||
```
|
||||
|
||||
**How it works:** Tracks owner metadata on every file. Indexed by MetadataIndexManager.
|
||||
|
||||
### 4. By Time (Temporal)
|
||||
```typescript
|
||||
await vfs.readdir('/as-of/2024-03-15')
|
||||
// Files modified on March 15, 2024
|
||||
|
||||
await vfs.readFile('/as-of/2024-03-15/src/auth.ts')
|
||||
// Read auth.ts as it existed that day
|
||||
```
|
||||
|
||||
**How it works:** Tracks `modified` timestamp. Uses B-tree range queries (`greaterEqual`/`lessEqual`) for O(log n) performance.
|
||||
|
||||
### 5. By Relationship (Graph)
|
||||
```typescript
|
||||
await vfs.readdir('/related-to/src/auth.ts/depth-2')
|
||||
// Files within 2 relationship hops
|
||||
|
||||
await vfs.readdir('/related-to/src/auth.ts/depth-2/types-contains,references')
|
||||
// Only follow 'contains' and 'references' relationships
|
||||
```
|
||||
|
||||
**How it works:** Uses GraphAdjacencyIndex for O(1) graph traversal. Supports depth limits and relationship type filtering.
|
||||
|
||||
### 6. By Similarity (Vector)
|
||||
```typescript
|
||||
await vfs.readdir('/similar-to/src/auth.ts/threshold-0.8')
|
||||
// Files with 80%+ similarity to auth.ts
|
||||
|
||||
await vfs.similar('/src/auth.ts', { threshold: 0.9, limit: 10 })
|
||||
// Top 10 most similar files (90%+ match)
|
||||
```
|
||||
|
||||
**How it works:** Uses HNSW vector index for O(log n) nearest neighbor search. Based on content embeddings.
|
||||
|
||||
### 7. By Tag (Classification)
|
||||
```typescript
|
||||
await vfs.readdir('/by-tag/security')
|
||||
// All security-tagged files
|
||||
|
||||
await vfs.writeFile('/src/admin.ts', code, {
|
||||
metadata: { tags: ['security', 'admin'] }
|
||||
})
|
||||
// Tag files on write
|
||||
```
|
||||
|
||||
**How it works:** Stores tags in metadata. Indexed for fast queries.
|
||||
|
||||
---
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Scalability to Millions of Files
|
||||
|
||||
All semantic paths use **indexed data structures** for optimal performance:
|
||||
|
||||
| Dimension | Data Structure | Time Complexity | Million-Scale Ready |
|
||||
|-----------|---------------|-----------------|---------------------|
|
||||
| Traditional | PathCache + Graph | O(path depth) | ✅ Yes |
|
||||
| Concept | MetadataIndex (B-tree) | O(log n) | ✅ Yes* |
|
||||
| Author | MetadataIndex (B-tree) | O(log n) | ✅ Yes |
|
||||
| Time | MetadataIndex (B-tree) | O(log n) | ✅ Yes |
|
||||
| Relationship | GraphAdjacency | O(depth) | ✅ Yes |
|
||||
| Similarity | HNSW Index | O(log n) | ✅ Yes |
|
||||
| Tag | MetadataIndex (B-tree) | O(log n) | ✅ Yes* |
|
||||
|
||||
\* *Requires concept/tag flattening (automatic in future versions)*
|
||||
|
||||
### Cache Strategy
|
||||
|
||||
Multi-layer caching ensures hot paths are O(1):
|
||||
```
|
||||
Request → Hot Path Cache (O(1))
|
||||
→ Semantic Cache (5 min TTL)
|
||||
→ Index Lookup (O(log n))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Example 1: Find All Files by Concept
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
|
||||
// Write files (concepts extracted automatically)
|
||||
await vfs.writeFile('/src/auth/login.ts', `
|
||||
export function authenticate(user, password) {
|
||||
// Authentication logic
|
||||
}
|
||||
`)
|
||||
|
||||
// Access by concept
|
||||
const authFiles = await vfs.readdir('/by-concept/authentication')
|
||||
console.log(authFiles)
|
||||
// ['login.ts', 'signup.ts', 'oauth.ts']
|
||||
```
|
||||
|
||||
### Example 2: Time Travel
|
||||
```typescript
|
||||
// See what changed today
|
||||
const today = new Date().toISOString().split('T')[0]
|
||||
const todaysFiles = await vfs.readdir(`/as-of/${today}`)
|
||||
|
||||
// Compare with yesterday
|
||||
const yesterday = new Date(Date.now() - 86400000).toISOString().split('T')[0]
|
||||
const yesterdaysFiles = await vfs.readdir(`/as-of/${yesterday}`)
|
||||
|
||||
const newFiles = todaysFiles.filter(f => !yesterdaysFiles.includes(f))
|
||||
console.log('New files today:', newFiles)
|
||||
```
|
||||
|
||||
### Example 3: Graph Navigation
|
||||
```typescript
|
||||
// Find all files related to auth
|
||||
const authId = await vfs.resolvePath('/src/auth.ts')
|
||||
const related = await vfs.readdir('/related-to/src/auth.ts/depth-2')
|
||||
|
||||
// Get relationship details
|
||||
for (const file of related) {
|
||||
const rels = await vfs.getRelationships(file.path)
|
||||
console.log(`${file.name}: ${rels.length} relationships`)
|
||||
}
|
||||
```
|
||||
|
||||
### Example 4: Semantic Search
|
||||
```typescript
|
||||
// Find similar implementations
|
||||
const similar = await vfs.similar('/src/auth.ts', {
|
||||
threshold: 0.8,
|
||||
limit: 10
|
||||
})
|
||||
|
||||
for (const result of similar) {
|
||||
console.log(`${result.entity.name}: ${result.similarity.toFixed(2)}`)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Reading Semantic Paths
|
||||
|
||||
All standard VFS methods work with semantic paths:
|
||||
|
||||
```typescript
|
||||
// Read directory
|
||||
await vfs.readdir('/by-concept/authentication')
|
||||
|
||||
// Read file
|
||||
await vfs.readFile('/by-concept/authentication/login.ts')
|
||||
|
||||
// Get stats
|
||||
await vfs.stat('/by-author/alice/config.ts')
|
||||
|
||||
// Check existence
|
||||
await vfs.exists('/as-of/2024-03-15/src/auth.ts')
|
||||
```
|
||||
|
||||
### Writing Files
|
||||
|
||||
Files are automatically indexed for semantic access:
|
||||
|
||||
```typescript
|
||||
await vfs.writeFile('/src/auth.ts', content, {
|
||||
metadata: {
|
||||
tags: ['security', 'authentication'],
|
||||
owner: 'alice'
|
||||
},
|
||||
extractConcepts: true, // default: true
|
||||
extractEntities: true, // default: true
|
||||
recordEvent: true // default: true
|
||||
})
|
||||
```
|
||||
|
||||
### Polymorphic Access
|
||||
|
||||
The same file is accessible through multiple paths:
|
||||
|
||||
```typescript
|
||||
// All these resolve to the SAME file entity:
|
||||
const id1 = await vfs.resolvePath('/src/auth/login.ts')
|
||||
const id2 = await vfs.resolvePath('/by-concept/authentication/login.ts')
|
||||
const id3 = await vfs.resolvePath('/by-author/alice/login.ts')
|
||||
|
||||
console.log(id1 === id2 && id2 === id3) // true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Extending with Custom Projections
|
||||
|
||||
Create your own semantic dimensions:
|
||||
|
||||
```typescript
|
||||
import { BaseProjectionStrategy } from '@soulcraft/brainy/vfs/semantic'
|
||||
|
||||
class PriorityProjection extends BaseProjectionStrategy {
|
||||
readonly name = 'priority'
|
||||
|
||||
async resolve(brain, vfs, priority) {
|
||||
return await brain.find({
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
priority: priority // Custom metadata field
|
||||
},
|
||||
limit: 1000
|
||||
})
|
||||
.then(results => results.map(r => r.id))
|
||||
}
|
||||
|
||||
async list(brain, vfs, limit = 100) {
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
vfsType: 'file',
|
||||
priority: { exists: true }
|
||||
},
|
||||
limit
|
||||
})
|
||||
return results.map(r => r.entity)
|
||||
}
|
||||
}
|
||||
|
||||
// Register custom projection
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
const vfs = brain.vfs()
|
||||
|
||||
// Access via VFS internals (will be exposed in future API)
|
||||
vfs.projectionRegistry.register(new PriorityProjection())
|
||||
|
||||
// Now use it!
|
||||
const highPriority = await vfs.readdir('/by-priority/high')
|
||||
```
|
||||
|
||||
See [PROJECTION_STRATEGY_API.md](./PROJECTION_STRATEGY_API.md) for full guide.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Triple Intelligence™ Foundation
|
||||
|
||||
Semantic VFS is built on Brainy's Triple Intelligence™:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Semantic VFS Layer │
|
||||
├─────────────────────────────────────────┤
|
||||
│ ProjectionRegistry + Strategies │
|
||||
├─────────────────────────────────────────┤
|
||||
│ SemanticPathResolver │
|
||||
├─────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Triple Intelligence™ (Brainy) │
|
||||
│ ┌─────────┬─────────┬─────────────┐ │
|
||||
│ │ Vector │ Graph │ Metadata │ │
|
||||
│ │ HNSW │ Adj │ B-tree │ │
|
||||
│ │ O(log n)│ O(1) │ O(log n) │ │
|
||||
│ └─────────┴─────────┴─────────────┘ │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Real Implementations, Zero Mocks
|
||||
|
||||
Every component uses **production Brainy APIs**:
|
||||
- `brain.find()` - Real metadata queries (brainy.ts:580)
|
||||
- `brain.similar()` - Real HNSW search (brainy.ts:680)
|
||||
- `brain.getRelations()` - Real graph traversal (brainy.ts:803)
|
||||
- `MetadataIndexManager` - Real B-tree indexes
|
||||
- `GraphAdjacencyIndex` - Real graph storage
|
||||
- `HNSW Index` - Real vector search
|
||||
|
||||
**No mocks. No stubs. No fake code.**
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Use Semantic Paths for Discovery
|
||||
```typescript
|
||||
// ❌ Don't hardcode paths
|
||||
const files = ['/src/auth.ts', '/src/login.ts', '/src/oauth.ts']
|
||||
|
||||
// ✅ Discover by concept
|
||||
const authFiles = await vfs.readdir('/by-concept/authentication')
|
||||
```
|
||||
|
||||
### 2. Tag Strategically
|
||||
```typescript
|
||||
// ✅ Good: Clear, actionable tags
|
||||
await vfs.writeFile(path, code, {
|
||||
metadata: { tags: ['security', 'requires-review', 'public-api'] }
|
||||
})
|
||||
|
||||
// ❌ Bad: Vague, redundant tags
|
||||
await vfs.writeFile(path, code, {
|
||||
metadata: { tags: ['code', 'file', 'important'] }
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Optimize for Your Scale
|
||||
```typescript
|
||||
// For < 100K files: Post-filtering is fine
|
||||
// For > 100K files: Use flattened indexes
|
||||
|
||||
// Force index refresh after bulk operations
|
||||
await brain.storage.rebuildIndexes()
|
||||
```
|
||||
|
||||
### 4. Combine Dimensions
|
||||
```typescript
|
||||
// Find security files Alice worked on this week
|
||||
const aliceFiles = await vfs.readdir('/by-author/alice')
|
||||
const securityFiles = await vfs.readdir('/by-tag/security')
|
||||
const thisWeek = await vfs.readdir(`/as-of/${weekAgo}`)
|
||||
|
||||
const intersection = aliceFiles
|
||||
.filter(f => securityFiles.includes(f))
|
||||
.filter(f => thisWeek.includes(f))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Concepts Not Being Extracted
|
||||
```typescript
|
||||
// Check if concepts are enabled (default: true)
|
||||
await vfs.writeFile(path, code, { extractConcepts: true })
|
||||
|
||||
// Verify concept extraction works
|
||||
const entity = await vfs.getEntity(path)
|
||||
console.log(entity.metadata.concepts)
|
||||
```
|
||||
|
||||
### Slow Queries on Large Datasets
|
||||
```typescript
|
||||
// Check if indexes are built
|
||||
const stats = await brain.storage.getIndexStats()
|
||||
console.log(stats)
|
||||
|
||||
// Rebuild if needed
|
||||
await brain.storage.rebuildIndexes()
|
||||
```
|
||||
|
||||
### Semantic Path Returns Empty
|
||||
```typescript
|
||||
// Check if metadata exists
|
||||
const files = await vfs.readdir('/src')
|
||||
for (const file of files) {
|
||||
const entity = await vfs.getEntity(file.path)
|
||||
console.log(entity.metadata)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What's Next?
|
||||
|
||||
- **Natural Language Paths**: `/find "authentication logic"`
|
||||
- **Intent-Based Access**: `/to-review`, `/to-deploy`
|
||||
- **Temporal Queries**: `/changed-since/2024-03-01`
|
||||
- **Custom Dimensions**: Plugin system for domain-specific projections
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Projection Strategy API](./PROJECTION_STRATEGY_API.md) - Create custom projections
|
||||
- [Performance Tuning](./PERFORMANCE_TUNING.md) - Million-scale optimization
|
||||
- [VFS Core API](./VFS_CORE.md) - Base VFS operations
|
||||
- [Triple Intelligence™](./TRIPLE_INTELLIGENCE.md) - Underlying architecture
|
||||
|
|
@ -794,8 +794,10 @@ console.log('VFS Stats:', stats)
|
|||
|
||||
---
|
||||
|
||||
The Virtual Filesystem API provides a powerful, intelligent alternative to traditional filesystems. With semantic search, rich metadata, and graph relationships, your files become living entities in a connected knowledge system.
|
||||
The Virtual Filesystem API provides a powerful, intelligent alternative to traditional filesystems. With semantic search, rich metadata, graph relationships, and AI-powered concept extraction, your files become living entities in a connected knowledge system.
|
||||
|
||||
For advanced features like event recording, semantic versioning, and persistent entities, see the [Knowledge Layer API Documentation](./KNOWLEDGE_LAYER_API.md).
|
||||
For semantic file access and neural extraction features, see:
|
||||
- [Semantic VFS Guide](./SEMANTIC_VFS.md) - Multi-dimensional file access
|
||||
- [Neural Extraction API](./NEURAL_EXTRACTION.md) - AI-powered concept and entity extraction
|
||||
|
||||
Ready to make your filesystem intelligent? 🚀
|
||||
|
|
@ -275,19 +275,9 @@ await vfs.importDirectory('/local/project', { targetPath: '/vfs/project' })
|
|||
|
||||
### GitBridge Integration
|
||||
|
||||
GitBridge provides Git import/export capabilities. It can be used in two ways:
|
||||
GitBridge provides Git import/export capabilities:
|
||||
|
||||
#### Option 1: Via Knowledge Layer (Recommended)
|
||||
```javascript
|
||||
// Enable Knowledge Layer to get Git methods
|
||||
await vfs.enableKnowledgeLayer()
|
||||
|
||||
// Now Git methods are available on VFS
|
||||
await vfs.exportToGit('/project', '/local/git/repo')
|
||||
await vfs.importFromGit('/local/git/repo', '/project')
|
||||
```
|
||||
|
||||
#### Option 2: Direct GitBridge Usage
|
||||
#### GitBridge Usage
|
||||
```javascript
|
||||
// Import and instantiate GitBridge
|
||||
import { GitBridge } from '@soulcraft/brainy'
|
||||
|
|
@ -327,9 +317,9 @@ await gitBridge.importFromGit('/local/git/repo', '/project', {
|
|||
- Compression ratio tracked in metadata
|
||||
|
||||
### Background Processing
|
||||
- Non-blocking Knowledge Layer processing
|
||||
- Asynchronous embedding generation
|
||||
- Deferred relationship indexing
|
||||
- Non-blocking metadata extraction
|
||||
|
||||
## Error Handling
|
||||
|
||||
|
|
@ -373,45 +363,32 @@ VFS scales to millions of files:
|
|||
|
||||
## Method Availability
|
||||
|
||||
### Core VFS Methods (Always Available)
|
||||
|
||||
These methods are available immediately after VFS initialization:
|
||||
All VFS methods are available immediately after initialization:
|
||||
|
||||
```javascript
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
|
||||
// ✅ All these work without Knowledge Layer:
|
||||
await vfs.writeFile() // File operations
|
||||
await vfs.readFile()
|
||||
await vfs.mkdir() // Directory operations
|
||||
await vfs.readdir()
|
||||
await vfs.stat() // Metadata
|
||||
await vfs.search() // Semantic search
|
||||
await vfs.addRelationship() // Relationships
|
||||
await vfs.addTodo() // Todo management
|
||||
await vfs.exportToJSON() // Export
|
||||
await vfs.bulkWrite() // Bulk operations
|
||||
```
|
||||
// Core file operations
|
||||
await vfs.writeFile() // Write files
|
||||
await vfs.readFile() // Read files
|
||||
await vfs.mkdir() // Create directories
|
||||
await vfs.readdir() // List directory contents
|
||||
await vfs.stat() // Get file metadata
|
||||
|
||||
### Knowledge Layer Methods (Require Enablement)
|
||||
// Semantic features
|
||||
await vfs.search() // Semantic search
|
||||
await vfs.findSimilar() // Find similar files
|
||||
await vfs.addRelationship()// Add relationships
|
||||
|
||||
These methods are only available after enabling the Knowledge Layer:
|
||||
// Todo management
|
||||
await vfs.addTodo() // Add todos
|
||||
await vfs.getTodos() // Get todos
|
||||
await vfs.setMetadata() // Set metadata
|
||||
|
||||
```javascript
|
||||
await vfs.enableKnowledgeLayer()
|
||||
|
||||
// 🔮 Now these methods are available:
|
||||
await vfs.createEntity() // Entity management
|
||||
await vfs.linkEntities()
|
||||
await vfs.createConcept() // Concept system
|
||||
await vfs.findByConcept()
|
||||
await vfs.getVersions() // Versioning
|
||||
await vfs.getHistory() // History tracking
|
||||
await vfs.exportToGit() // Git integration (wrapper)
|
||||
await vfs.importFromGit()
|
||||
await vfs.exportToMarkdown()// Export formats
|
||||
await vfs.getTimeline() // Timeline analysis
|
||||
// Export
|
||||
await vfs.exportToJSON() // Export to JSON
|
||||
await vfs.bulkWrite() // Bulk operations
|
||||
```
|
||||
|
||||
## Complete Example
|
||||
|
|
|
|||
|
|
@ -1,8 +1,10 @@
|
|||
# VFS Examples and Scenarios
|
||||
|
||||
> **⚠️ DOCUMENTATION IN PROGRESS**: This document is being updated to reflect recent VFS architecture changes. Many examples below reference deprecated Knowledge Layer methods. Updated examples coming soon. For current VFS capabilities, see [VFS_CORE.md](./VFS_CORE.md) and [SEMANTIC_VFS.md](./SEMANTIC_VFS.md).
|
||||
|
||||
## Real-World Scenarios
|
||||
|
||||
This document demonstrates how VFS with Knowledge Layer enables powerful real-world applications.
|
||||
This document demonstrates how VFS enables powerful real-world applications with semantic search, relationships, and AI-powered concept extraction.
|
||||
|
||||
### Legend
|
||||
- ✅ **Real VFS methods** - Fully implemented and working
|
||||
|
|
@ -24,7 +26,6 @@ async function novelWritingProject() {
|
|||
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
await vfs.enableKnowledgeLayer()
|
||||
|
||||
// Create project structure ✅
|
||||
await vfs.mkdir('/novel')
|
||||
|
|
@ -32,22 +33,26 @@ async function novelWritingProject() {
|
|||
await vfs.mkdir('/novel/characters')
|
||||
await vfs.mkdir('/novel/worldbuilding')
|
||||
|
||||
// Define main characters as persistent entities ✅
|
||||
const protagonist = await vfs.createEntity({
|
||||
name: 'Elena Blackwood',
|
||||
type: 'character',
|
||||
description: 'A skilled detective with a mysterious past',
|
||||
attributes: {
|
||||
age: 32,
|
||||
occupation: 'Private Investigator',
|
||||
skills: ['deduction', 'combat', 'languages'],
|
||||
personality: ['determined', 'secretive', 'compassionate']
|
||||
// Define main characters as files with rich metadata ✅
|
||||
await vfs.writeFile('/novel/characters/elena-blackwood.md', `
|
||||
# Elena Blackwood
|
||||
|
||||
A skilled detective with a mysterious past.
|
||||
|
||||
## Character Profile
|
||||
- Age: 32
|
||||
- Occupation: Private Investigator
|
||||
- Skills: Deduction, combat, languages
|
||||
- Personality: Determined, secretive, compassionate
|
||||
`, {
|
||||
metadata: {
|
||||
characterType: 'protagonist',
|
||||
tags: ['detective', 'mysterious', 'protagonist']
|
||||
}
|
||||
})
|
||||
|
||||
const antagonist = await vfs.createEntity({
|
||||
name: 'Marcus Void',
|
||||
type: 'character',
|
||||
await vfs.writeFile('/novel/characters/marcus-void.md', `
|
||||
# Marcus Void
|
||||
description: 'A wealthy industrialist with dark secrets',
|
||||
attributes: {
|
||||
age: 45,
|
||||
|
|
@ -147,7 +152,6 @@ async function gameDevProject() {
|
|||
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
await vfs.enableKnowledgeLayer()
|
||||
|
||||
// Game project structure
|
||||
await vfs.mkdir('/game')
|
||||
|
|
@ -293,9 +297,8 @@ async function softwareProject() {
|
|||
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
await vfs.enableKnowledgeLayer()
|
||||
|
||||
// Import existing git repository ✅ (Knowledge Layer provides wrapper)
|
||||
// Import existing git repository ✅
|
||||
await vfs.importFromGit('/local/repos/webapp', '/project')
|
||||
|
||||
// Define architectural concepts
|
||||
|
|
@ -460,7 +463,6 @@ async function unifiedKnowledgeBase() {
|
|||
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
await vfs.enableKnowledgeLayer()
|
||||
|
||||
// Create separate project spaces
|
||||
await vfs.mkdir('/novel')
|
||||
|
|
|
|||
|
|
@ -1,460 +0,0 @@
|
|||
# VFS + Knowledge Layer Integration
|
||||
|
||||
## Overview
|
||||
|
||||
The Knowledge Layer is an optional augmentation that transforms VFS from a filesystem into an intelligent knowledge management system. When enabled, it adds event recording, semantic versioning, persistent entities, universal concepts, and Git integration.
|
||||
|
||||
## Enabling Knowledge Layer
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'memory' },
|
||||
silent: true
|
||||
})
|
||||
await brain.init()
|
||||
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
|
||||
// Enable Knowledge Layer augmentation
|
||||
await vfs.enableKnowledgeLayer()
|
||||
|
||||
// Now VFS has additional intelligent features
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
The Knowledge Layer consists of five integrated systems:
|
||||
|
||||
### 1. EventRecorder
|
||||
Tracks all filesystem operations as searchable events with embeddings.
|
||||
|
||||
### 2. SemanticVersioning
|
||||
Creates versions based on semantic meaning changes, not just byte differences.
|
||||
|
||||
### 3. PersistentEntitySystem
|
||||
Tracks evolving entities (characters, concepts, systems) across files.
|
||||
|
||||
### 4. ConceptSystem
|
||||
Manages universal concepts that span multiple files and projects.
|
||||
|
||||
### 5. GitBridge
|
||||
Enables import/export between VFS and Git repositories.
|
||||
|
||||
## Event Recording
|
||||
|
||||
Every filesystem operation is recorded as an event:
|
||||
|
||||
```javascript
|
||||
// All operations are automatically recorded
|
||||
await vfs.writeFile('/doc.txt', 'Initial content')
|
||||
await vfs.appendFile('/doc.txt', '\nMore content')
|
||||
await vfs.rename('/doc.txt', '/document.txt')
|
||||
|
||||
// Query events
|
||||
const history = await vfs.getHistory('/document.txt')
|
||||
for (const event of history) {
|
||||
console.log(event.type, event.timestamp, event.user)
|
||||
// 'create' 2025-01-20T10:00:00Z 'alice'
|
||||
// 'write' 2025-01-20T10:01:00Z 'alice'
|
||||
// 'rename' 2025-01-20T10:02:00Z 'alice'
|
||||
}
|
||||
|
||||
// Get timeline of events
|
||||
const events = await vfs.getTimeline({ from: '2025-01-01' })
|
||||
```
|
||||
|
||||
### Event Types
|
||||
- `create` - File/directory created
|
||||
- `write` - Content written
|
||||
- `append` - Content appended
|
||||
- `delete` - File/directory deleted
|
||||
- `rename` - Path changed
|
||||
- `move` - File relocated
|
||||
- `metadata` - Metadata updated
|
||||
- `relationship` - Relationship added/removed
|
||||
|
||||
### Event Schema
|
||||
```javascript
|
||||
{
|
||||
id: 'uuid',
|
||||
type: 'write',
|
||||
path: '/document.txt',
|
||||
oldPath: null, // For renames/moves
|
||||
timestamp: Date.now(),
|
||||
user: 'current-user',
|
||||
size: 1024, // Bytes affected
|
||||
contentHash: 'sha256...', // Content fingerprint
|
||||
vector: [0.1, 0.2, ...], // Semantic embedding
|
||||
metadata: {
|
||||
mimeType: 'text/plain',
|
||||
encoding: 'utf8'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Semantic Versioning
|
||||
|
||||
Versions are created when content *meaning* changes significantly:
|
||||
|
||||
```javascript
|
||||
// Initial version
|
||||
await vfs.writeFile('/story.txt', 'Once upon a time...')
|
||||
|
||||
// Minor change - no new version (typo fix)
|
||||
await vfs.writeFile('/story.txt', 'Once upon a time...')
|
||||
|
||||
// Major change - creates new version (plot development)
|
||||
await vfs.writeFile('/story.txt', 'Once upon a time, the kingdom fell...')
|
||||
|
||||
// Get versions
|
||||
const versions = await vfs.getVersions('/story.txt')
|
||||
for (const version of versions) {
|
||||
console.log(version.id, version.timestamp, version.semanticHash)
|
||||
// Compare semantic similarity between versions
|
||||
console.log(version.similarity) // 0.45 (significantly different)
|
||||
}
|
||||
|
||||
// Restore version
|
||||
await vfs.restoreVersion('/story.txt', versions[0].id)
|
||||
|
||||
// Compare versions by restoring
|
||||
const v1Content = await vfs.getVersion('/story.txt', v1.id)
|
||||
const v2Content = await vfs.getVersion('/story.txt', v2.id)
|
||||
// Compare the content as needed
|
||||
```
|
||||
|
||||
### Version Triggers
|
||||
- Semantic similarity < 0.7 threshold
|
||||
- New concepts introduced
|
||||
- Major structural changes
|
||||
- Explicit version creation
|
||||
|
||||
## Persistent Entities
|
||||
|
||||
Track characters, systems, and entities across files:
|
||||
|
||||
```javascript
|
||||
// Create persistent entity
|
||||
const character = await vfs.createEntity({
|
||||
name: 'Alice',
|
||||
type: 'character',
|
||||
description: 'Main protagonist, a curious explorer',
|
||||
attributes: {
|
||||
age: 25,
|
||||
occupation: 'Archaeologist',
|
||||
traits: ['brave', 'intelligent', 'curious']
|
||||
}
|
||||
})
|
||||
|
||||
// Entity appears across multiple files
|
||||
await vfs.writeFile('/chapter1.txt', 'Alice entered the ancient tomb...')
|
||||
await vfs.writeFile('/chapter2.txt', 'Alice decoded the hieroglyphs...')
|
||||
|
||||
// Track entity across files
|
||||
const occurrences = await vfs.findEntityOccurrences('Alice')
|
||||
// Returns all files mentioning Alice with context
|
||||
|
||||
// Update entity globally
|
||||
await vfs.updateEntity(character.id, {
|
||||
attributes: {
|
||||
age: 26, // Birthday happened in the story
|
||||
newTrait: 'experienced'
|
||||
}
|
||||
})
|
||||
|
||||
// Entity types
|
||||
const entities = await vfs.listEntities({ type: 'character' })
|
||||
// Supports: character, location, object, system, concept, etc.
|
||||
```
|
||||
|
||||
### Entity Relationships
|
||||
```javascript
|
||||
// Link entities
|
||||
await vfs.linkEntities('Alice', 'Ancient Tomb', 'explores')
|
||||
await vfs.linkEntities('Alice', 'Bob', 'mentored_by')
|
||||
|
||||
// Query entity graph
|
||||
const graph = await vfs.getEntityGraph('Alice', { depth: 2 })
|
||||
// Returns connected entities and their relationships
|
||||
```
|
||||
|
||||
## Concept System
|
||||
|
||||
Universal concepts that transcend individual files:
|
||||
|
||||
```javascript
|
||||
// Create concept
|
||||
const authConcept = await vfs.createConcept({
|
||||
name: 'Authentication',
|
||||
type: 'technical',
|
||||
domain: 'security',
|
||||
description: 'User identity verification system',
|
||||
keywords: ['login', 'password', 'token', 'session'],
|
||||
relatedConcepts: ['Authorization', 'Security']
|
||||
})
|
||||
|
||||
// Concepts are automatically detected in files
|
||||
await vfs.writeFile('/auth.js', 'function authenticate(user, password) {...}')
|
||||
await vfs.writeFile('/login.tsx', 'const LoginForm = () => {...}')
|
||||
|
||||
// Find files by concept
|
||||
const authFiles = await vfs.findByConcept('Authentication')
|
||||
// Returns all files related to authentication concept
|
||||
|
||||
// Find files by concept
|
||||
const authFiles = await vfs.findByConcept('Authentication')
|
||||
// Returns all files related to the authentication concept
|
||||
```
|
||||
|
||||
### Working with Concepts
|
||||
```javascript
|
||||
// Concepts can reference each other through their descriptions
|
||||
// and keywords, creating an implicit network of related ideas.
|
||||
// The findByConcept method searches across these relationships.
|
||||
```
|
||||
|
||||
## GitBridge Integration
|
||||
|
||||
Seamlessly work with Git repositories:
|
||||
|
||||
```javascript
|
||||
// Import from Git repo
|
||||
await vfs.importFromGit('/local/git/repo', '/vfs/project')
|
||||
|
||||
// Imports:
|
||||
// - All files and directories
|
||||
// - Git history as VFS events
|
||||
// - Commit messages as event metadata
|
||||
// - Branch structure as relationships
|
||||
|
||||
// Export to Git format
|
||||
await vfs.exportToGit('/vfs/project', '/local/git/repo')
|
||||
|
||||
// Exports:
|
||||
// - Files to working directory
|
||||
// - VFS events as git commits
|
||||
// - Relationships as .brainy/relationships.json
|
||||
// - Entities as .brainy/entities.json
|
||||
// - Concepts as .brainy/concepts.json
|
||||
|
||||
// Export/import operations are available
|
||||
// For remote sync, use git commands after export:
|
||||
// await vfs.exportToGit('/vfs/project', '/local/repo')
|
||||
// Then use git push/pull as normal
|
||||
```
|
||||
|
||||
### Git Integration
|
||||
```javascript
|
||||
// Import from Git preserves history as events
|
||||
// Export to Git creates .brainy/ metadata directory
|
||||
// Use standard git commands for remote operations
|
||||
```
|
||||
|
||||
## Knowledge Queries
|
||||
|
||||
Powerful queries across all Knowledge Layer data:
|
||||
|
||||
```javascript
|
||||
// Timeline query
|
||||
const timeline = await vfs.getTimeline({
|
||||
from: '2025-01-01',
|
||||
to: '2025-01-31',
|
||||
types: ['write', 'create']
|
||||
})
|
||||
|
||||
// Timeline queries
|
||||
const timeline = await vfs.getTimeline({
|
||||
from: '2025-01-01',
|
||||
to: '2025-01-31',
|
||||
types: ['write', 'create']
|
||||
})
|
||||
|
||||
// Project statistics
|
||||
const stats = await vfs.getProjectStats('/project')
|
||||
console.log('Total files:', stats.fileCount)
|
||||
console.log('Total size:', stats.totalSize)
|
||||
console.log('Todo count:', stats.todoCount)
|
||||
|
||||
// Search with Triple Intelligence
|
||||
const results = await vfs.search('authentication', {
|
||||
path: '/src',
|
||||
type: 'file',
|
||||
limit: 20
|
||||
})
|
||||
```
|
||||
|
||||
## Background Processing
|
||||
|
||||
Knowledge Layer operations run in the background:
|
||||
|
||||
```javascript
|
||||
// Operations are non-blocking
|
||||
await vfs.writeFile('/large-doc.txt', hugeContent)
|
||||
// Returns immediately
|
||||
|
||||
// Knowledge processing happens asynchronously:
|
||||
// 1. Event recording (immediate)
|
||||
// 2. Embedding generation (100ms)
|
||||
// 3. Version checking (200ms)
|
||||
// 4. Entity extraction (500ms)
|
||||
// 5. Concept detection (1s)
|
||||
|
||||
// Background processing happens automatically
|
||||
// Events are recorded immediately
|
||||
// Embeddings and versions are processed asynchronously
|
||||
```
|
||||
|
||||
## Search and Analysis
|
||||
|
||||
The Knowledge Layer provides powerful search and analysis capabilities:
|
||||
|
||||
```javascript
|
||||
// Find files by concept
|
||||
const authFiles = await vfs.findByConcept('Authentication')
|
||||
// Returns all files related to the authentication concept
|
||||
|
||||
// Get timeline of changes
|
||||
const timeline = await vfs.getTimeline({
|
||||
from: '2025-01-01',
|
||||
to: '2025-01-31',
|
||||
types: ['write', 'create']
|
||||
})
|
||||
// Returns chronological list of events
|
||||
|
||||
// Get project statistics
|
||||
const stats = await vfs.getProjectStats('/project')
|
||||
console.log(stats.fileCount) // Number of files
|
||||
console.log(stats.totalSize) // Total size in bytes
|
||||
console.log(stats.todoCount) // Number of todos
|
||||
console.log(stats.largestFile) // Largest file info
|
||||
|
||||
// Export directory to markdown
|
||||
const markdown = await vfs.exportToMarkdown('/docs')
|
||||
// Returns formatted markdown of entire directory structure
|
||||
```
|
||||
|
||||
## Collaboration Features
|
||||
|
||||
Knowledge Layer enables multi-user collaboration:
|
||||
|
||||
```javascript
|
||||
// Track user actions
|
||||
vfs.setUser('alice')
|
||||
await vfs.writeFile('/shared.txt', 'Alice\'s content')
|
||||
|
||||
vfs.setUser('bob')
|
||||
await vfs.appendFile('/shared.txt', 'Bob\'s addition')
|
||||
|
||||
// Get collaboration history
|
||||
const collabHistory = await vfs.getCollaborationHistory('/shared.txt')
|
||||
// Returns who edited the file and when:
|
||||
// [
|
||||
// { user: 'alice', timestamp: Date, action: 'write', size: 15 },
|
||||
// { user: 'bob', timestamp: Date, action: 'append', size: 14 }
|
||||
// ]
|
||||
|
||||
// Get all todos across project
|
||||
const allTodos = await vfs.getAllTodos('/project')
|
||||
// Returns todos from all files recursively
|
||||
```
|
||||
|
||||
## Performance Impact
|
||||
|
||||
Knowledge Layer overhead:
|
||||
- **Write operations**: +50-200ms for event recording
|
||||
- **Read operations**: No impact (cached)
|
||||
- **Search operations**: 10x faster (pre-computed embeddings)
|
||||
- **Storage**: ~20% additional for events and embeddings
|
||||
- **Memory**: +100MB for caches and indexes
|
||||
|
||||
## Configuration
|
||||
|
||||
Fine-tune Knowledge Layer behavior:
|
||||
|
||||
```javascript
|
||||
await vfs.enableKnowledgeLayer({
|
||||
eventRecording: true, // Track all operations
|
||||
semanticVersioning: true, // Smart versioning
|
||||
versionThreshold: 0.7, // Similarity threshold
|
||||
persistentEntities: true, // Track entities
|
||||
entityTypes: ['character', 'location', 'system'],
|
||||
concepts: true, // Universal concepts
|
||||
conceptDomains: ['technical', 'narrative', 'business'],
|
||||
gitBridge: true, // Git integration
|
||||
backgroundProcessing: true, // Non-blocking
|
||||
processingDelay: 100, // Ms before processing
|
||||
cacheSizes: {
|
||||
events: 10000,
|
||||
versions: 1000,
|
||||
entities: 5000,
|
||||
concepts: 2000
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Complete Example
|
||||
|
||||
```javascript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
async function knowledgeExample() {
|
||||
// Initialize with Knowledge Layer
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'memory' }
|
||||
})
|
||||
await brain.init()
|
||||
|
||||
const vfs = brain.vfs()
|
||||
await vfs.init()
|
||||
await vfs.enableKnowledgeLayer()
|
||||
|
||||
// Create a story with tracked entities
|
||||
const alice = await vfs.createEntity({
|
||||
name: 'Alice',
|
||||
type: 'character',
|
||||
description: 'Protagonist'
|
||||
})
|
||||
|
||||
await vfs.writeFile('/chapter1.md', `
|
||||
# Chapter 1
|
||||
Alice discovered the ancient artifact...
|
||||
`)
|
||||
|
||||
// File automatically:
|
||||
// - Records write event
|
||||
// - Generates embedding
|
||||
// - Links to Alice entity
|
||||
// - Detects "ancient artifact" concept
|
||||
|
||||
// Create technical documentation
|
||||
await vfs.createConcept({
|
||||
name: 'API Design',
|
||||
type: 'technical',
|
||||
domain: 'software'
|
||||
})
|
||||
|
||||
await vfs.writeFile('/api-guide.md', `
|
||||
# API Design Guide
|
||||
RESTful principles...
|
||||
`)
|
||||
|
||||
// Check Knowledge Layer insights
|
||||
const insights = await vfs.getInsights('/')
|
||||
console.log('Entities:', insights.entities)
|
||||
console.log('Concepts:', insights.concepts)
|
||||
console.log('Relationships:', insights.relationships)
|
||||
|
||||
// Query across knowledge
|
||||
const results = await vfs.knowledgeSearch({
|
||||
query: 'Alice artifact',
|
||||
includeEvents: true,
|
||||
includeEntities: true
|
||||
})
|
||||
|
||||
await vfs.close()
|
||||
await brain.close()
|
||||
}
|
||||
```
|
||||
|
||||
The Knowledge Layer transforms VFS from a filesystem into an intelligent knowledge management system that understands content, tracks evolution, and enables semantic collaboration.
|
||||
Loading…
Add table
Add a link
Reference in a new issue