fix: exclude __words__ keyword index from corruption detection and getStats()

The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:

1. validateConsistency() to falsely detect corruption on every startup,
   triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
   and report inflated totalEntries/totalIds stats

Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
This commit is contained in:
David Snelling 2026-01-27 15:38:21 -08:00
parent 32dbdcec61
commit 364360d447
128 changed files with 5637 additions and 5682 deletions

View file

@ -22,13 +22,13 @@ import { Brainy } from '@soulcraft/brainy'
// ✅ CORRECT: Use filesystem storage for production
const brain = new Brainy({
storage: {
type: 'filesystem',
path: './brainy-data' // Your data directory
}
storage: {
type: 'filesystem',
path: './brainy-data' // Your data directory
}
})
await brain.init() // VFS auto-initialized!
await brain.init() // VFS auto-initialized!
console.log('🎉 VFS ready!')
```
@ -47,40 +47,40 @@ const badItems = allNodes.filter(node => node.path.startsWith(dirPath))
```typescript
// ✅ Method 1: Get direct children (recommended for UI)
async function loadDirectoryContents(brain: Brainy, path: string) {
try {
const children = await brain.vfs.getDirectChildren(path)
try {
const children = await brain.vfs.getDirectChildren(path)
// Sort directories first, then files
return children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
} catch (error) {
console.error(`Failed to load ${path}:`, error.message)
return []
}
// Sort directories first, then files
return children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
} catch (error) {
console.error(`Failed to load ${path}:`, error.message)
return []
}
}
// ✅ Method 2: Get complete tree structure (for full trees)
async function loadFullTree(brain: Brainy, path: string) {
const tree = await brain.vfs.getTreeStructure(path, {
maxDepth: 3, // Prevent deep recursion
includeHidden: false, // Skip hidden files
sort: 'name'
})
return tree
const tree = await brain.vfs.getTreeStructure(path, {
maxDepth: 3, // Prevent deep recursion
includeHidden: false, // Skip hidden files
sort: 'name'
})
return tree
}
// ✅ Method 3: Get detailed path info
async function inspectPath(brain: Brainy, path: string) {
const info = await brain.vfs.inspect(path)
return {
isDirectory: info.node.metadata.vfsType === 'directory',
children: info.children,
parent: info.parent,
stats: info.stats
}
const info = await brain.vfs.inspect(path)
return {
isDirectory: info.node.metadata.vfsType === 'directory',
children: info.children,
parent: info.parent,
stats: info.stats
}
}
```
@ -89,19 +89,19 @@ async function inspectPath(brain: Brainy, path: string) {
```typescript
// ✅ Find files by content, not just filename
async function searchFiles(brain: Brainy, query: string, basePath: string = '/') {
const results = await brain.vfs.search(query, {
path: basePath, // Limit search to specific directory
limit: 50, // Reasonable limit
type: 'file' // Only search files, not directories
})
const results = await brain.vfs.search(query, {
path: basePath, // Limit search to specific directory
limit: 50, // Reasonable limit
type: 'file' // Only search files, not directories
})
return results.map(result => ({
path: result.path,
score: result.score,
type: result.type,
size: result.size,
modified: result.modified
}))
return results.map(result => ({
path: result.path,
score: result.score,
type: result.type,
size: result.size,
modified: result.modified
}))
}
// Example usage
@ -118,149 +118,149 @@ import React, { useState, useEffect } from 'react'
import { Brainy } from '@soulcraft/brainy'
export function FileExplorer() {
const [brain, setBrain] = useState(null)
const [currentPath, setCurrentPath] = useState('/')
const [items, setItems] = useState([])
const [loading, setLoading] = useState(true)
const [searchQuery, setSearchQuery] = useState('')
const [brain, setBrain] = useState(null)
const [currentPath, setCurrentPath] = useState('/')
const [items, setItems] = useState([])
const [loading, setLoading] = useState(true)
const [searchQuery, setSearchQuery] = useState('')
// Initialize Brainy (VFS auto-initialized!)
useEffect(() => {
async function initBrainy() {
const brainInstance = new Brainy({
storage: { type: 'filesystem', path: './brainy-data' }
})
await brainInstance.init() // VFS ready after this!
// Initialize Brainy (VFS auto-initialized!)
useEffect(() => {
async function initBrainy() {
const brainInstance = new Brainy({
storage: { type: 'filesystem', path: './brainy-data' }
})
await brainInstance.init() // VFS ready after this!
setBrain(brainInstance)
setLoading(false)
}
initBrainy()
}, [])
setBrain(brainInstance)
setLoading(false)
}
initBrainy()
}, [])
// Load directory contents
const loadDirectory = async (path: string) => {
if (!brain) return
// Load directory contents
const loadDirectory = async (path: string) => {
if (!brain) return
setLoading(true)
try {
// ✅ CORRECT: Use getDirectChildren to prevent recursion
const children = await brain.vfs.getDirectChildren(path)
setLoading(true)
try {
// ✅ CORRECT: Use getDirectChildren to prevent recursion
const children = await brain.vfs.getDirectChildren(path)
// Sort directories first
const sorted = children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
// Sort directories first
const sorted = children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
setItems(sorted)
setCurrentPath(path)
} catch (error) {
console.error('Failed to load directory:', error)
setItems([])
} finally {
setLoading(false)
}
}
setItems(sorted)
setCurrentPath(path)
} catch (error) {
console.error('Failed to load directory:', error)
setItems([])
} finally {
setLoading(false)
}
}
// Search files
const handleSearch = async () => {
if (!brain || !searchQuery.trim()) {
loadDirectory(currentPath)
return
}
// Search files
const handleSearch = async () => {
if (!brain || !searchQuery.trim()) {
loadDirectory(currentPath)
return
}
setLoading(true)
try {
const results = await brain.vfs.search(searchQuery, {
path: currentPath,
limit: 100
})
setItems(results)
} catch (error) {
console.error('Search failed:', error)
} finally {
setLoading(false)
}
}
setLoading(true)
try {
const results = await brain.vfs.search(searchQuery, {
path: currentPath,
limit: 100
})
setItems(results)
} catch (error) {
console.error('Search failed:', error)
} finally {
setLoading(false)
}
}
// Initial load
useEffect(() => {
if (brain) {
loadDirectory('/')
}
}, [brain])
// Initial load
useEffect(() => {
if (brain) {
loadDirectory('/')
}
}, [brain])
if (loading && !brain) {
return <div>Initializing Brainy...</div>
}
if (loading && !brain) {
return <div>Initializing Brainy...</div>
}
return (
<div className="file-explorer">
{/* Search bar */}
<div className="search-bar">
<input
type="text"
value={searchQuery}
onChange={(e) => setSearchQuery(e.target.value)}
placeholder="Search files by content..."
onKeyPress={(e) => e.key === 'Enter' && handleSearch()}
/>
<button onClick={handleSearch}>Search</button>
{searchQuery && (
<button onClick={() => { setSearchQuery(''); loadDirectory(currentPath) }}>
Clear
</button>
)}
</div>
return (
<div className="file-explorer">
{/* Search bar */}
<div className="search-bar">
<input
type="text"
value={searchQuery}
onChange={(e) => setSearchQuery(e.target.value)}
placeholder="Search files by content..."
onKeyPress={(e) => e.key === 'Enter' && handleSearch()}
/>
<button onClick={handleSearch}>Search</button>
{searchQuery && (
<button onClick={() => { setSearchQuery(''); loadDirectory(currentPath) }}>
Clear
</button>
)}
</div>
{/* Current path */}
<div className="current-path">
📁 {currentPath}
{currentPath !== '/' && (
<button onClick={() => loadDirectory(currentPath.split('/').slice(0, -1).join('/') || '/')}>
⬆️ Up
</button>
)}
</div>
{/* Current path */}
<div className="current-path">
📁 {currentPath}
{currentPath !== '/' && (
<button onClick={() => loadDirectory(currentPath.split('/').slice(0, -1).join('/') || '/')}>
⬆️ Up
</button>
)}
</div>
{/* File list */}
{loading ? (
<div>Loading...</div>
) : (
<div className="file-list">
{items.map((item) => (
<div
key={item.id}
className={`file-item ${item.metadata.vfsType}`}
onClick={() => {
if (item.metadata.vfsType === 'directory') {
loadDirectory(item.metadata.path)
} else {
console.log('Open file:', item.metadata.path)
// Add your file opening logic here
}
}}
>
<span className="icon">
{item.metadata.vfsType === 'directory' ? '📁' : '📄'}
</span>
<span className="name">{item.metadata.name}</span>
<span className="size">
{item.metadata.size ? `${Math.round(item.metadata.size / 1024)}KB` : ''}
</span>
</div>
))}
{items.length === 0 && (
<div className="empty">
{searchQuery ? 'No results found' : 'Empty directory'}
</div>
)}
</div>
)}
</div>
)
{/* File list */}
{loading ? (
<div>Loading...</div>
) : (
<div className="file-list">
{items.map((item) => (
<div
key={item.id}
className={`file-item ${item.metadata.vfsType}`}
onClick={() => {
if (item.metadata.vfsType === 'directory') {
loadDirectory(item.metadata.path)
} else {
console.log('Open file:', item.metadata.path)
// Add your file opening logic here
}
}}
>
<span className="icon">
{item.metadata.vfsType === 'directory' ? '📁' : '📄'}
</span>
<span className="name">{item.metadata.name}</span>
<span className="size">
{item.metadata.size ? `${Math.round(item.metadata.size / 1024)}KB` : ''}
</span>
</div>
))}
{items.length === 0 && (
<div className="empty">
{searchQuery ? 'No results found' : 'Empty directory'}
</div>
)}
</div>
)}
</div>
)
}
```
@ -288,13 +288,13 @@ Your file explorer is now working! Here's what to explore next:
### "Module not found" errors
```bash
# Make sure you're using the right import
npm ls @soulcraft/brainy # Check version
npm install @soulcraft/brainy@latest # Update if needed
npm ls @soulcraft/brainy # Check version
npm install @soulcraft/brainy@latest # Update if needed
```
### "VFS not initialized" errors
```typescript
// v5.1.0+: Just await brain.init() - VFS is auto-initialized!
// Just await brain.init() - VFS is auto-initialized!
await brain.init()
// VFS ready - use brain.vfs directly
await brain.vfs.writeFile('/test.txt', 'data')
@ -304,8 +304,8 @@ await brain.vfs.writeFile('/test.txt', 'data')
```typescript
// Add pagination for large directories
const children = await brain.vfs.getDirectChildren(path, {
limit: 100, // Load only first 100 items
offset: 0 // Start from beginning
limit: 100, // Load only first 100 items
offset: 0 // Start from beginning
})
```
@ -313,8 +313,8 @@ const children = await brain.vfs.getDirectChildren(path, {
```typescript
// Make sure files are imported into VFS first
await brain.vfs.importDirectory('./my-files', {
recursive: true,
extractMetadata: true // Enable content understanding
recursive: true,
extractMetadata: true // Enable content understanding
})
```

View file

@ -28,15 +28,15 @@ import { VirtualFileSystem } from '@soulcraft/brainy/vfs'
// Initialize the VFS
const vfs = new VirtualFileSystem({
root: '/my-brain',
intelligent: true // Enable AI features
root: '/my-brain',
intelligent: true // Enable AI features
})
await vfs.init()
// Write a file - it automatically becomes intelligent
await vfs.writeFile('/projects/my-app/index.js',
'console.log("Hello, World!")')
'console.log("Hello, World!")')
// Find similar files using semantic search
const similar = await vfs.findSimilar('/projects/my-app/index.js')
@ -48,11 +48,11 @@ const results = await vfs.search('files about authentication')
await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements')
```
## ⚡ Performance (v5.11.1)
## ⚡ Performance
**75% Faster File Operations!** VFS now automatically benefits from brain.get() metadata-only optimization:
| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup |
| Operation | Before | After | Speedup |
|-----------|------------------|-----------------|---------|
| `readFile()` | 53ms | **~13ms** | **75%** |
| `stat()` | 53ms | **~13ms** | **75%** |
@ -60,7 +60,7 @@ await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements')
**Zero configuration** - automatic optimization for all VFS operations!
VFS operations only need metadata (path, size, timestamps), not 384-dimensional vector embeddings. The v5.11.1 optimization automatically uses metadata-only reads, saving 95% bandwidth and 76-81% time.
VFS operations only need metadata (path, size, timestamps), not 384-dimensional vector embeddings. The optimization automatically uses metadata-only reads, saving 95% bandwidth and 76-81% time.
## Core Features
@ -110,20 +110,20 @@ const docs = await vfs.search('technical documentation for API endpoints')
// Find similar files
const similar = await vfs.findSimilar('/code/auth.js', {
limit: 5,
threshold: 0.8 // 80% similarity
limit: 5,
threshold: 0.8 // 80% similarity
})
// Get related files through the knowledge graph
const related = await vfs.getRelated('/proposal.pdf', {
depth: 2 // Include relationships of relationships
depth: 2 // Include relationships of relationships
})
// Auto-organization suggestions
const suggestions = await vfs.suggestOrganization([
'/downloads/doc1.pdf',
'/downloads/image.jpg',
'/downloads/code.py'
'/downloads/doc1.pdf',
'/downloads/image.jpg',
'/downloads/code.py'
])
// Returns: suggested folders and categorization
```
@ -144,10 +144,10 @@ const connections = await vfs.getConnections('/code/impl.js')
// Traverse the graph
const implementations = await vfs.search('', {
connected: {
to: '/spec.md',
via: 'implements'
}
connected: {
to: '/spec.md',
via: 'implements'
}
})
```
@ -158,8 +158,8 @@ Store anything alongside your files:
```javascript
// Add todos to files
await vfs.setTodos('/projects/app/index.js', [
{ task: 'Add error handling', priority: 'high', due: '2024-01-20' },
{ task: 'Optimize performance', priority: 'medium' }
{ task: 'Add error handling', priority: 'high', due: '2024-01-20' },
{ task: 'Optimize performance', priority: 'medium' }
])
// Set custom attributes
@ -169,11 +169,11 @@ await vfs.setxattr('/video.mp4', 'tags', ['tutorial', 'react', 'hooks'])
// Query by metadata
const urgent = await vfs.search('', {
where: { 'todos.priority': 'high' }
where: { 'todos.priority': 'high' }
})
const parisPhotos = await vfs.search('', {
where: { location: 'Paris, France' }
where: { location: 'Paris, France' }
})
```
@ -184,20 +184,20 @@ Access files through semantic dimensions (see [Semantic VFS](./SEMANTIC_VFS.md))
```javascript
// Query-based path access (current functionality)
const authFiles = await vfs.search('', {
where: { concepts: { contains: 'authentication' }}
where: { concepts: { contains: 'authentication' }}
})
// Find files by custom metadata
const recent = await vfs.search('', {
where: {
type: 'document',
modified: { greaterThan: Date.now() - 7*24*60*60*1000 }
}
where: {
type: 'document',
modified: { greaterThan: Date.now() - 7*24*60*60*1000 }
}
})
// Find similar files
const similar = await vfs.findSimilar('/examples/good-code.js', {
threshold: 0.7
threshold: 0.7
})
```
@ -210,24 +210,24 @@ const similar = await vfs.findSimilar('/examples/good-code.js', {
```javascript
// Store a research paper with automatic analysis
await vfs.writeFile('/research/quantum-computing.pdf', pdfBuffer, {
metadata: {
authors: ['Dr. Alice Smith', 'Dr. Bob Jones'],
year: 2024,
topics: ['quantum', 'computing', 'algorithms'],
citations: 42
}
metadata: {
authors: ['Dr. Alice Smith', 'Dr. Bob Jones'],
year: 2024,
topics: ['quantum', 'computing', 'algorithms'],
citations: 42
}
})
// Find all papers on similar topics
const related = await vfs.search('quantum algorithms', {
type: 'document',
where: { year: { $gte: 2020 } }
type: 'document',
where: { year: { $gte: 2020 } }
})
// Find papers that cite this one
const citations = await vfs.getConnections('/research/quantum-computing.pdf', {
type: 'cites',
direction: 'incoming'
type: 'cites',
direction: 'incoming'
})
```
@ -245,12 +245,12 @@ await vfs.writeFile('/src/utils/auth.js', authCode)
// Find all files that import this module
const importers = await vfs.search('', {
where: { dependencies: 'utils/auth.js' }
where: { dependencies: 'utils/auth.js' }
})
// Find test files for this code
const tests = await vfs.getRelated('/src/utils/auth.js', {
type: 'tests'
type: 'tests'
})
// Find similar implementations
@ -262,12 +262,12 @@ const similar = await vfs.findSimilar('/src/utils/auth.js')
```javascript
// Store media with rich metadata
await vfs.writeFile('/photos/sunset.jpg', imageBuffer, {
metadata: {
camera: 'Canon R5',
location: { lat: 37.7749, lng: -122.4194 },
tags: ['sunset', 'golden-gate', 'landscape'],
album: 'San Francisco 2024'
}
metadata: {
camera: 'Canon R5',
location: { lat: 37.7749, lng: -122.4194 },
tags: ['sunset', 'golden-gate', 'landscape'],
album: 'San Francisco 2024'
}
})
// Find similar images
@ -275,18 +275,18 @@ const similar = await vfs.findSimilar('/photos/sunset.jpg')
// Find photos by location
const nearby = await vfs.search('', {
type: 'image',
where: {
'location.lat': { $between: [37.7, 37.8] },
'location.lng': { $between: [-122.5, -122.3] }
}
type: 'image',
where: {
'location.lat': { $between: [37.7, 37.8] },
'location.lng': { $between: [-122.5, -122.3] }
}
})
// Smart albums
await vfs.createVirtualDirectory('/albums/best-sunsets', {
query: 'sunset',
type: 'image',
where: { rating: { $gte: 4 } }
query: 'sunset',
type: 'image',
where: { rating: { $gte: 4 } }
})
```
@ -308,25 +308,25 @@ await vfs.setxattr(projectPath, 'status', 'in-progress')
// Add todos to specific files
await vfs.setTodos(`${projectPath}/src/index.js`, [
{ task: 'Implement user authentication', assignee: 'Alice' },
{ task: 'Add error handling', assignee: 'Bob' }
{ task: 'Implement user authentication', assignee: 'Alice' },
{ task: 'Add error handling', assignee: 'Bob' }
])
// Find all files with pending todos
const pending = await vfs.search('', {
where: {
path: { $startsWith: projectPath },
'todos.status': 'pending'
}
where: {
path: { $startsWith: projectPath },
'todos.status': 'pending'
}
})
// Find projects nearing deadline
const urgent = await vfs.search('', {
where: {
type: 'directory',
'deadline': { $lte: '2024-02-01' },
'status': 'in-progress'
}
where: {
type: 'directory',
'deadline': { $lte: '2024-02-01' },
'status': 'in-progress'
}
})
```
@ -417,9 +417,9 @@ The VFS is built with production scalability in mind:
#### **Storage Strategy**
```typescript
// Adaptive storage based on file size
< 100KB: Inline storage (entity.data)
< 10MB: External reference (S3/R2 key)
> 10MB: Chunked storage (parallel chunks)
< 100KB: Inline storage (entity.data)
< 10MB: External reference (S3/R2 key)
> 10MB: Chunked storage (parallel chunks)
```
#### **Performance Metrics**
@ -434,24 +434,24 @@ The VFS is built with production scalability in mind:
#### **How It Handles Scale**
1. **Hierarchical Caching**
- 100K+ path cache entries
- Parent-child relationship caching
- Hot path detection and optimization
- 100K+ path cache entries
- Parent-child relationship caching
- Hot path detection and optimization
2. **Distributed Architecture**
- Sharding by path prefix
- Read replicas for hot directories
- CDN integration for static files
- Sharding by path prefix
- Read replicas for hot directories
- CDN integration for static files
3. **Intelligent Indexing**
- Compound indexes on (parent, name)
- Vector indexes for semantic search
- Graph indexes for relationships
- Compound indexes on (parent, name)
- Vector indexes for semantic search
- Graph indexes for relationships
4. **Streaming Everything**
- Large files never fully in memory
- Progressive loading
- Chunked transfers
- Large files never fully in memory
- Progressive loading
- Chunked transfers
### Real Production Scenarios
@ -462,29 +462,29 @@ Store build artifacts with automatic relationships:
```javascript
// Store build output with metadata
await vfs.writeFile('/builds/v1.2.3/app.js', buildOutput, {
metadata: {
commit: 'abc123',
branch: 'main',
timestamp: Date.now(),
tests: 'passing',
coverage: 0.92
}
metadata: {
commit: 'abc123',
branch: 'main',
timestamp: Date.now(),
tests: 'passing',
coverage: 0.92
}
})
// Find all builds for a commit
const builds = await vfs.search('', {
where: { commit: 'abc123' }
where: { commit: 'abc123' }
})
// Get latest passing build
const latest = await vfs.search('', {
where: {
branch: 'main',
tests: 'passing'
},
sort: 'modified',
order: 'desc',
limit: 1
where: {
branch: 'main',
tests: 'passing'
},
sort: 'modified',
order: 'desc',
limit: 1
})
```
@ -495,8 +495,8 @@ Isolate customer data with semantic understanding:
```javascript
// Each tenant gets their own root
const tenantVfs = new VirtualFileSystem({
root: `/tenants/${tenantId}`,
service: tenantId // Isolate at Brainy level too
root: `/tenants/${tenantId}`,
service: tenantId // Isolate at Brainy level too
})
// Tenant uploads document
@ -505,11 +505,11 @@ await tenantVfs.writeFile('/documents/contract.pdf', pdfBuffer)
// Cross-tenant analytics (admin only)
const adminVfs = new VirtualFileSystem({ root: '/tenants' })
const stats = await adminVfs.search('contract', {
recursive: true,
aggregations: {
byTenant: { field: 'service' },
byType: { field: 'mimeType' }
}
recursive: true,
aggregations: {
byTenant: { field: 'service' },
byType: { field: 'mimeType' }
}
})
```
@ -520,38 +520,38 @@ Connect datasets, models, and results:
```javascript
// Store training data
await vfs.writeFile('/datasets/train.csv', csvData, {
metadata: {
samples: 100000,
features: 50,
labels: 10
}
metadata: {
samples: 100000,
features: 50,
labels: 10
}
})
// Store trained model
await vfs.writeFile('/models/v1/model.pkl', modelBuffer, {
metadata: {
algorithm: 'random-forest',
accuracy: 0.95,
trainedOn: '/datasets/train.csv',
hyperparameters: { trees: 100, depth: 10 }
}
metadata: {
algorithm: 'random-forest',
accuracy: 0.95,
trainedOn: '/datasets/train.csv',
hyperparameters: { trees: 100, depth: 10 }
}
})
// Connect model to its training data
await vfs.addRelationship(
'/models/v1/model.pkl',
'/datasets/train.csv',
'trained-on'
'/models/v1/model.pkl',
'/datasets/train.csv',
'trained-on'
)
// Find best model for a dataset
const models = await vfs.search('', {
connected: {
to: '/datasets/train.csv',
via: 'trained-on'
},
sort: 'accuracy',
order: 'desc'
connected: {
to: '/datasets/train.csv',
via: 'trained-on'
},
sort: 'accuracy',
order: 'desc'
})
```
@ -562,30 +562,30 @@ Intelligent content organization:
```javascript
// Auto-organize uploads
vfs.on('file:added', async (path) => {
if (path.startsWith('/uploads/')) {
const file = await vfs.getEntity(path)
if (path.startsWith('/uploads/')) {
const file = await vfs.getEntity(path)
// Auto-categorize by AI
const category = await detectCategory(file)
const newPath = `/content/${category}/${file.metadata.name}`
// Auto-categorize by AI
const category = await detectCategory(file)
const newPath = `/content/${category}/${file.metadata.name}`
await vfs.move(path, newPath)
await vfs.move(path, newPath)
// Auto-tag
const tags = await extractTags(file)
await vfs.setxattr(newPath, 'tags', tags)
// Auto-tag
const tags = await extractTags(file)
await vfs.setxattr(newPath, 'tags', tags)
// Find related content
const related = await vfs.findSimilar(newPath, {
limit: 5,
threshold: 0.8
})
// Find related content
const related = await vfs.findSimilar(newPath, {
limit: 5,
threshold: 0.8
})
// Create relationships
for (const rel of related) {
await vfs.addRelationship(newPath, rel.path, 'related-to')
}
}
// Create relationships
for (const rel of related) {
await vfs.addRelationship(newPath, rel.path, 'related-to')
}
}
})
```
@ -596,39 +596,39 @@ Collaborative file management:
```javascript
// Track file ownership and access
await vfs.writeFile('/projects/alpha/spec.md', content, {
metadata: {
owner: userId,
team: 'engineering',
permissions: {
[userId]: 'rw',
'team:engineering': 'r',
'others': '-'
}
}
metadata: {
owner: userId,
team: 'engineering',
permissions: {
[userId]: 'rw',
'team:engineering': 'r',
'others': '-'
}
}
})
// Add collaborative features
await vfs.addTodo('/projects/alpha/spec.md', {
task: 'Review security section',
assignee: 'alice@company.com',
due: '2024-02-01',
priority: 'high'
task: 'Review security section',
assignee: 'alice@company.com',
due: '2024-02-01',
priority: 'high'
})
// Track who's working on what
await vfs.setxattr('/projects/alpha/spec.md', 'locks', {
section3: {
user: 'bob@company.com',
since: Date.now()
}
section3: {
user: 'bob@company.com',
since: Date.now()
}
})
// Find all files assigned to a user
const assigned = await vfs.search('', {
where: {
'todos.assignee': 'alice@company.com',
'todos.status': 'pending'
}
where: {
'todos.assignee': 'alice@company.com',
'todos.status': 'pending'
}
})
```
@ -643,15 +643,15 @@ const assigned = await vfs.search('', {
#### **Standalone Mode**
```javascript
const vfs = new VirtualFileSystem()
await vfs.init() // Uses in-memory storage
await vfs.init() // Uses in-memory storage
```
#### **Local Persistence**
```javascript
const vfs = new VirtualFileSystem()
await vfs.init({
storage: 'filesystem',
dataDir: '/var/lib/brainy-vfs'
storage: 'filesystem',
dataDir: '/var/lib/brainy-vfs'
})
```
@ -659,9 +659,9 @@ await vfs.init({
```javascript
const vfs = new VirtualFileSystem()
await vfs.init({
storage: 's3',
bucket: 'my-vfs-data',
region: 'us-west-2'
storage: 's3',
bucket: 'my-vfs-data',
region: 'us-west-2'
})
```
@ -669,14 +669,14 @@ await vfs.init({
```javascript
const vfs = new VirtualFileSystem()
await vfs.init({
distributed: true,
nodes: [
'vfs1.internal:8080',
'vfs2.internal:8080',
'vfs3.internal:8080'
],
replication: 3,
consistency: 'eventual'
distributed: true,
nodes: [
'vfs1.internal:8080',
'vfs2.internal:8080',
'vfs3.internal:8080'
],
replication: 3,
consistency: 'eventual'
})
```

View file

@ -12,8 +12,8 @@
**Root Cause:**
The root directory entity exists but doesn't have the proper metadata structure.
**Solution (v3.15.0+):**
This issue has been fixed in v3.15.0. The VFS now:
**Solution:**
This issue has been fixed. The VFS now:
1. Ensures root directory has `vfsType: 'directory'` metadata
2. Adds compatibility layer for entities with malformed metadata
3. Automatically repairs metadata on entity retrieval
@ -21,17 +21,17 @@ This issue has been fixed in v3.15.0. The VFS now:
**Manual Fix (if needed):**
```javascript
// Force re-initialization of root directory
await vfs.init() // Will repair root if needed
await vfs.init() // Will repair root if needed
// Or manually update root entity
const rootId = vfs.rootEntityId
await brain.update({
id: rootId,
metadata: {
path: '/',
vfsType: 'directory',
// ... other metadata
}
id: rootId,
metadata: {
path: '/',
vfsType: 'directory',
// ... other metadata
}
})
```
@ -47,7 +47,7 @@ await brain.update({
**Root Cause:**
Contains relationships are missing between parent directories and files.
**Solution (v3.15.0+):**
**Solution:**
This issue has been fixed. The VFS now:
1. Creates Contains relationships when writing new files
2. Ensures Contains relationships exist when updating files
@ -60,9 +60,9 @@ const parentId = await vfs.resolvePath('/directory')
const fileId = await vfs.resolvePath('/directory/file.txt')
await brain.relate({
from: parentId,
to: fileId,
type: VerbType.Contains
from: parentId,
to: fileId,
type: VerbType.Contains
})
```
@ -80,8 +80,8 @@ The VFS `init()` method wasn't called after getting the VFS instance.
**Solution:**
```javascript
// ✅ CORRECT
const vfs = brain.vfs() // Get instance
await vfs.init() // Initialize (REQUIRED!)
const vfs = brain.vfs() // Get instance
await vfs.init() // Initialize (REQUIRED!)
// ❌ WRONG
const vfs = brain.vfs()
@ -104,15 +104,15 @@ Using in-memory storage instead of persistent storage.
```javascript
// ✅ Use persistent storage
const brain = new Brainy({
storage: {
type: 'filesystem', // Persistent
path: './brainy-data'
}
storage: {
type: 'filesystem', // Persistent
path: './brainy-data'
}
})
// ❌ Don't use memory for production
const brain = new Brainy({
storage: { type: 'memory' } // Data lost on restart!
storage: { type: 'memory' } // Data lost on restart!
})
```
@ -136,7 +136,7 @@ const children = await vfs.getDirectChildren('/directory')
// ❌ WRONG - Path prefix matching causes recursion
const allNodes = await brain.find({})
const children = allNodes.filter(n =>
n.metadata.path.startsWith('/directory/')) // Directory matches itself!
n.metadata.path.startsWith('/directory/')) // Directory matches itself!
```
---
@ -153,13 +153,13 @@ Files aren't being properly embedded or indexed.
**Solution:**
```javascript
// Ensure files have content for embedding
await vfs.writeFile('/doc.txt', 'Actual content here') // Not empty!
await vfs.writeFile('/doc.txt', 'Actual content here') // Not empty!
// Use semantic search correctly
const results = await vfs.search('machine learning', {
path: '/documents', // Search within path
limit: 10,
type: 'file'
path: '/documents', // Search within path
limit: 10,
type: 'file'
})
```
@ -184,8 +184,8 @@ console.log('VFS type:', entity.metadata.vfsType)
```javascript
const parentId = await vfs.resolvePath('/directory')
const relations = await brain.getRelations({
from: parentId,
type: VerbType.Contains
from: parentId,
type: VerbType.Contains
})
console.log('Child count:', relations.length)
```
@ -193,11 +193,11 @@ console.log('Child count:', relations.length)
### 4. Enable Debug Logging
```javascript
const brain = new Brainy({
storage: { type: 'filesystem' },
logger: {
level: 'debug',
enabled: true
}
storage: { type: 'filesystem' },
logger: {
level: 'debug',
enabled: true
}
})
```
@ -209,11 +209,11 @@ const brain = new Brainy({
```javascript
// Enable caching in VFS config
const vfs = brain.vfs({
cache: {
enabled: true,
ttl: 300000, // 5 minutes
maxSize: 1000
}
cache: {
enabled: true,
ttl: 300000, // 5 minutes
maxSize: 1000
}
})
```
@ -221,12 +221,12 @@ const vfs = brain.vfs({
```javascript
// Write multiple files efficiently
const files = [
{ path: '/file1.txt', content: 'content1' },
{ path: '/file2.txt', content: 'content2' }
{ path: '/file1.txt', content: 'content1' },
{ path: '/file2.txt', content: 'content2' }
]
await Promise.all(
files.map(f => vfs.writeFile(f.path, f.content))
files.map(f => vfs.writeFile(f.path, f.content))
)
```
@ -234,8 +234,8 @@ await Promise.all(
```javascript
// Don't traverse too deep
const tree = await vfs.getTreeStructure('/', {
maxDepth: 3, // Limit recursion
includeHidden: false
maxDepth: 3, // Limit recursion
includeHidden: false
})
```

View file

@ -418,7 +418,7 @@ await vfs.importDirectory()// Import directory from filesystem
## Bulk Write Operations
**v6.5.0**: `bulkWrite` efficiently processes multiple VFS operations with automatic ordering to prevent race conditions.
`bulkWrite` efficiently processes multiple VFS operations with automatic ordering to prevent race conditions.
```javascript
const result = await vfs.bulkWrite([
@ -439,7 +439,7 @@ console.log(`Successful: ${result.successful}, Failed: ${result.failed.length}`)
- `delete` - Delete file
- `update` - Update file metadata only
**Operation ordering (v6.5.0):**
**Operation ordering:**
1. `mkdir` operations run **first**, sequentially, sorted by path depth (shallowest first)
2. Other operations (`write`, `delete`, `update`) run **after** in parallel batches of 10

View file

@ -1,4 +1,4 @@
# VFS Initialization Guide (v5.1.0+)
# VFS Initialization Guide
## Quick Start
@ -30,7 +30,7 @@ await vfs.init() // ❌ Separate initialization
await vfs.writeFile(...)
```
### After (v5.1.0+):
### After:
```javascript
const brain = new Brainy(...)
await brain.init() // VFS auto-initialized!
@ -53,7 +53,7 @@ await vfs.init()
await vfs.writeFile('/test.txt', 'data')
```
### New Pattern (v5.1.0+):
### New Pattern:
```javascript
// Just remove the () and init() call
await brain.vfs.writeFile('/test.txt', 'data')
@ -142,7 +142,7 @@ await brain.init() // Required!
await brain.vfs.writeFile(...) // Now this works
```
## Fork Support (v5.0.0+)
## Fork Support
VFS works seamlessly with Brainy's Copy-on-Write (COW) fork feature: