soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase

Browse source
This commit is contained in:
David Snelling 2026-06-11 14:51:00 -07:00
parent 970e08c466
commit 1f7e365a4e
237 changed files with 1951 additions and 49413 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/ADR-001-generational-mvcc.md
View file

@ -28,7 +28,7 @@ path and the native path.
- A **monotonic u64 generation counter** is the store's logical clock. It
advances once per committed `transact()` batch and once per
single-operation write (`add`/`update`/`delete`/`relate`/…), so
single-operation write (`add`/`update`/`remove`/`relate`/…), so
`brain.generation()` is always a meaningful watermark. It is persisted in
`_system/generation.json` and never reissued for anything durable.
- `brain.now()` **pins** the current generation in O(1) and returns a `Db` —

2
docs/BATCHING.md
View file

@ -316,7 +316,7 @@ const entities = Array.from(results.values())
```typescript
const allVerbs = []
for (const sourceId of sourceIds) {
const verbs = await brain.getRelations({ from: sourceId })
const verbs = await brain.related({ from: sourceId })
allVerbs.push(...verbs)
}
```

435
docs/CREATING-AUGMENTATIONS.md
View file

@ -1,435 +0,0 @@
# Creating Augmentations for Brainy
> **Updated** - Includes metadata structure changes and type system improvements
## The BrainyAugmentation Interface
Every augmentation implements this simple yet powerful interface:
```typescript
interface BrainyAugmentation {
// Identification
name: string // Unique name for your augmentation
// Execution control
timing: 'before' | 'after' | 'around' | 'replace' // When to execute
operations: string[] // Which operations to intercept
priority: number // Execution order (higher = first)
// Lifecycle methods
initialize(context: AugmentationContext): Promise<void>
execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T>
shutdown?(): Promise<void> // Optional cleanup
}
```
## Breaking Changes for Augmentation Developers
### 1. Metadata Structure Separation
Brainy introduces strict metadata/vector separation for billion-scale performance:
```typescript
// ✅ Metadata has required type field
interface NounMetadata {
noun: NounType // Required! Must be a valid noun type
[key: string]: any // Your custom metadata
}
interface VerbMetadata {
verb: VerbType // Required! Must be a valid verb type
sourceId: string
targetId: string
[key: string]: any
}
```
### 2. Storage Adapter Return Types
Storage adapters now return different types at different boundaries:
```typescript
// Internal methods: Pure structures (no metadata)
abstract _getNoun(id: string): Promise<HNSWNoun | null>
// Public API: WithMetadata structures
abstract getNoun(id: string): Promise<HNSWNounWithMetadata | null>
```
### 3. Verb Property Renamed
The verb relationship field changed from `type` to `verb`:
```typescript
// ❌ v3.x
verb.type === 'relatedTo'
// ✅ Current
verb.verb === 'relatedTo'
```
## Creating a Storage Augmentation
Storage augmentations are special - they provide the storage backend for Brainy.
### Important: Storage Requirements
Your storage adapter MUST:
1. **Wrap metadata** with required `noun`/`verb` fields
2. **Return pure structures** from internal `_methods`
3. **Return WithMetadata types** from public methods
```typescript
import { StorageAugmentation } from 'brainy/augmentations'
import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy'
export class MyCustomStorage extends BaseStorageAdapter {
// Internal method: Returns pure structure
async _getNoun(id: string): Promise<HNSWNoun | null> {
const data = await this.fetchFromDatabase(id)
return data ? {
id: data.id,
vector: data.vector,
nounType: data.type
} : null
}
// Public method: Returns WithMetadata structure
async getNoun(id: string): Promise<HNSWNounWithMetadata | null> {
const noun = await this._getNoun(id)
if (!noun) return null
// Fetch metadata separately
const metadata = await this.getNounMetadata(id)
return {
...noun,
metadata: metadata || { noun: noun.nounType || 'thing' }
}
}
// CRITICAL: Always save with proper metadata structure
async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> {
// Validate metadata has required 'noun' field
if (!metadata?.noun) {
throw new Error('NounMetadata requires "noun" field')
}
await this.database.save({
id: noun.id,
vector: noun.vector,
nounType: noun.nounType,
metadata: metadata // Stored separately
})
}
}
export class MyStorageAugmentation extends StorageAugmentation {
private config: MyStorageConfig
constructor(config: MyStorageConfig) {
super()
this.name = 'my-custom-storage'
this.config = config
}
// Called during storage resolution phase
async provideStorage(): Promise<StorageAdapter> {
const storage = new MyCustomStorage(this.config)
this.storageAdapter = storage
return storage
}
// Called during augmentation initialization
protected async onInitialize(): Promise<void> {
await this.storageAdapter!.init()
this.log(`Custom storage initialized`)
}
}
```
### Using Your Storage Augmentation
```typescript
// Register before brain.init()
const brain = new Brainy()
brain.augmentations.register(new MyStorageAugmentation({
connectionString: 'redis://localhost:6379'
}))
await brain.init() // Will use your storage!
```
## Creating a Feature Augmentation
Here's a complete example of a caching augmentation:
```typescript
import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations'
export class CachingAugmentation extends BaseAugmentation {
private cache = new Map<string, any>()
constructor() {
super()
this.name = 'smart-cache'
this.timing = 'around' // Wrap operations
this.operations = ['search'] // Only cache searches
this.priority = 50 // Mid-priority
}
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
if (operation === 'search') {
// Check cache
const cacheKey = JSON.stringify(params)
if (this.cache.has(cacheKey)) {
this.log('Cache hit!')
return this.cache.get(cacheKey)
}
// Execute and cache
const result = await next()
this.cache.set(cacheKey, result)
return result
}
// Pass through other operations
return next()
}
protected async onInitialize(): Promise<void> {
this.log('Cache initialized')
}
async shutdown(): Promise<void> {
this.cache.clear()
await super.shutdown()
}
}
```
## The Four Timing Modes
### 1. `before` - Pre-processing
```typescript
timing = 'before'
async execute(op, params, next) {
// Validate/transform input
const validated = await validate(params)
return next(validated) // Pass modified params
}
```
### 2. `after` - Post-processing
```typescript
timing = 'after'
async execute(op, params, next) {
const result = await next()
// Log, analyze, or modify result
console.log(`Operation ${op} returned:`, result)
return result
}
```
### 3. `around` - Wrapping (middleware)
```typescript
timing = 'around'
async execute(op, params, next) {
console.log('Starting', op)
try {
const result = await next()
console.log('Success', op)
return result
} catch (error) {
console.log('Failed', op, error)
throw error
}
}
```
### 4. `replace` - Complete replacement
```typescript
timing = 'replace'
async execute(op, params, next) {
// Don't call next() - replace entirely!
return myCustomImplementation(params)
}
```
## Operations You Can Intercept
Common operations in Brainy:
- `'storage'` - Storage resolution (special)
- `'add'` - Adding data
- `'search'`, `'similar'` - Searching
- `'update'`, `'delete'` - Modifications
- `'saveNoun'`, `'saveVerb'` - Storage operations
- `'all'` - Intercept everything
## Context Available to Augmentations
```typescript
interface AugmentationContext {
brain: Brainy // The brain instance
storage: StorageAdapter // Storage backend
config: BrainyConfig // Configuration
log: (message: string, level?: 'info' | 'warn' | 'error') => void
}
```
## Real-World Examples
### 1. Redis Storage Augmentation
```typescript
export class RedisStorageAugmentation extends StorageAugmentation {
async provideStorage(): Promise<StorageAdapter> {
return new RedisAdapter({
host: 'localhost',
port: 6379,
// Implement full StorageAdapter interface
})
}
}
```
### 2. Audit Trail Augmentation
```typescript
export class AuditAugmentation extends BaseAugmentation {
timing = 'after'
operations = ['add', 'update', 'delete']
async execute(op, params, next) {
const result = await next()
// Log to audit trail
await this.logAudit({
operation: op,
params,
result,
timestamp: new Date(),
user: this.context.config.currentUser
})
return result
}
}
```
### 3. Rate Limiting Augmentation
```typescript
export class RateLimitAugmentation extends BaseAugmentation {
timing = 'before'
operations = ['search']
private limiter = new RateLimiter({ rps: 100 })
async execute(op, params, next) {
await this.limiter.acquire() // Wait if rate limited
return next()
}
}
```
## Publishing to Brain Cloud Marketplace
Future capability for premium augmentations:
```typescript
// package.json
{
"name": "@brain-cloud/redis-storage",
"brainy": {
"type": "augmentation",
"category": "storage",
"premium": true
}
}
// Users can install via:
// brainy augment install redis-storage
```
## Best Practices
### General Practices
1. **Use BaseAugmentation** - Provides common functionality
2. **Set appropriate priority** - Storage (100), System (80-99), Features (10-50)
3. **Be selective with operations** - Don't use 'all' unless necessary
4. **Handle errors gracefully** - Don't break the chain
5. **Clean up in shutdown()** - Release resources
6. **Log appropriately** - Use context.log() for consistent output
7. **Document your augmentation** - Include examples
### Specific Best Practices
8. **Always include `noun` field** when creating/modifying NounMetadata:
```typescript
const metadata: NounMetadata = {
noun: 'thing', // REQUIRED!
yourField: 'value'
}
```
9. **Use `verb` property** not `type` when working with relationships:
```typescript
// ✅ Correct
if (verb.verb === 'relatedTo') { ... }
// ❌ Wrong (v3.x pattern)
if (verb.type === 'relatedTo') { ... }
```
10. **Access metadata correctly** from storage:
```typescript
// ✅ Correct - metadata is already structured
const nounType = noun.metadata.noun
// ⚠️ Fallback pattern for robustness
const nounType = noun.metadata?.noun || 'thing'
```
11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations:
```typescript
// ✅ Good - Separate concerns
await storage.saveNoun(noun)
await storage.saveMetadata(noun.id, metadata)
// ❌ Bad - Mixing concerns
await storage.saveNounWithEverything(combinedData)
```
## Testing Your Augmentation
```typescript
import { Brainy } from 'brainy'
import { MyAugmentation } from './my-augmentation'
describe('MyAugmentation', () => {
let brain: Brainy
beforeEach(async () => {
brain = new Brainy()
brain.augmentations.register(new MyAugmentation())
await brain.init()
})
afterEach(async () => {
await brain.destroy()
})
it('should enhance searches', async () => {
// Test your augmentation's effect
const results = await brain.search('test')
expect(results).toHaveProperty('enhanced', true)
})
})
```
## Summary
Augmentations are Brainy's extension system. They can:
- Replace storage backends
- Add caching layers
- Implement audit trails
- Add rate limiting
- Sync with external systems
- Transform data
- And much more!
The unified BrainyAugmentation interface makes it easy to create powerful extensions while maintaining consistency across the entire system.

2
docs/DATA_MODEL.md
View file

@ -238,7 +238,7 @@ await brain.relate({
Fast-path filter on the verb side:
```typescript
const direct = await brain.getRelations({
const direct = await brain.related({
from: ceoId,
type: VerbType.ReportsTo,
subtype: 'direct'

12
docs/DEVELOPER_LEARNING_PATH.md
View file

@ -193,21 +193,21 @@ console.log('✅ Created relationships')
console.log('\n🔍 Querying relationships...')
// Who works for TechCorp?
const techcorpEmployees = await brain.getRelations({
const techcorpEmployees = await brain.related({
to: techcorpId,
type: VerbType.WorksWith
})
console.log(`TechCorp has ${techcorpEmployees.length} employees`)
// Who works on the AI project?
const projectContributors = await brain.getRelations({
const projectContributors = await brain.related({
to: projectId,
type: [VerbType.WorksOn, VerbType.Manages]
})
console.log(`AI Project has ${projectContributors.length} contributors`)
// Who does John collaborate with?
const johnsCollaborators = await brain.getRelations({
const johnsCollaborators = await brain.related({
from: johnId,
type: VerbType.CollaboratesWith
})
@ -294,7 +294,7 @@ await brain.update({
1. Create an organizational hierarchy (CEO → Managers → Engineers)
2. Build a project dependency graph
3. Model a social network with CollaboratesWith relationships
4. Query "Who reports to Alice?" using getRelations()
4. Query "Who reports to Alice?" using related()
5. Batch update all projects to add a "year: 2024" field
### Next Steps
@ -676,7 +676,7 @@ if (readmeEntity.length > 0) {
}
// Query relationships
const conceptDocs = await brain.getRelations({
const conceptDocs = await brain.related({
from: conceptId,
type: VerbType.DocumentedBy
})
@ -798,7 +798,7 @@ await brain.relate({
})
// Query across boundaries
const conceptDocs = await brain.getRelations({
const conceptDocs = await brain.related({
from: conceptId,
type: VerbType.DocumentedBy
})

2
docs/FIND_SYSTEM.md
View file

@ -1228,7 +1228,7 @@ where: { age: { gt: 18 } }
**No connected entities found**:
```typescript
// Verify relationship exists
const relations = await brain.getRelations({
const relations = await brain.related({
from: 'entity-a',
to: 'entity-b'
})

2
docs/MIGRATION-V3-TO-V4.md
View file

@ -48,7 +48,7 @@ await storage.setLifecyclePolicy({
```typescript
// v3: Delete one at a time (slow, expensive)
for (const id of idsToDelete) {
await brain.delete(id) // 1000 API calls for 1000 entities
await brain.remove(id) // 1000 API calls for 1000 entities
}
// v4.0.0: Batch delete (fast, cheap)

6
docs/QUERY_OPERATORS.md
View file

@ -216,18 +216,18 @@ See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the f
### Filter relationships by subtype (7.30+)
Verbs are first-class peers — `getRelations()` and graph traversal both honor subtype filters on the fast path:
Verbs are first-class peers — `related()` and graph traversal both honor subtype filters on the fast path:
```typescript
// Filter relationships by VerbType subtype
const direct = await brain.getRelations({
const direct = await brain.related({
from: ceoId,
type: VerbType.ReportsTo,
subtype: 'direct'
})
// Set membership on verb subtype
const all = await brain.getRelations({
const all = await brain.related({
from: ceoId,
type: VerbType.ReportsTo,
subtype: ['direct', 'dotted-line']

7
docs/README.md
View file

@ -89,14 +89,11 @@ See [vfs/](./vfs/) for the complete VFS documentation set.
---
## Plugins & Augmentations
## Plugins
| Document | Description |
|----------|-------------|
| [Plugins](./PLUGINS.md) | Plugin system overview |
| [Creating Augmentations](./CREATING-AUGMENTATIONS.md) | Build custom plugins |
| [Augmentations Reference](./augmentations/COMPLETE-REFERENCE.md) | Full augmentation API |
| [Augmentations Developer Guide](./augmentations/DEVELOPER-GUIDE.md) | Plugin development guide |
| [Plugins](./PLUGINS.md) | Plugin system overview — providers, `plugins` config, `brain.use()` |
---

249
docs/api/README.md
View file

@ -185,12 +185,12 @@ await brain.update({
---
### `delete(id)` → `Promise<void>`
### `remove(id)` → `Promise<void>`
Delete a single entity.
Remove a single entity (and every relationship where it is source or target).
```typescript
await brain.delete(id)
await brain.remove(id)
```
**Parameters:**
@ -444,7 +444,7 @@ brain.defineAggregate({
name: 'monthly_spending',
source: {
type: NounType.Event,
where: { domain: 'financial', subtype: 'transaction' }
where: { domain: 'financial' } // matches custom metadata fields
},
groupBy: [
'category',
@ -468,10 +468,10 @@ brain.defineAggregate({
|-------|------|-------------|
| `name` | `string` | Unique identifier for this aggregate |
| `source.type` | `NounType \| NounType[]` | Entity types that feed into this aggregate |
| `source.where` | `Record<string, unknown>` | Metadata filter (same syntax as `find({ where })`) |
| `source.where` | `Record<string, unknown>` | Filter on custom **metadata** fields (matched against the entity's `metadata` bag) |
| `source.service` | `string` | Multi-tenancy filter |
| `groupBy` | `GroupByDimension[]` | Dimensions to group by — plain field names or `{ field, window }` for time bucketing |
| `metrics` | `Record<string, AggregateMetricDef>` | Named metrics with `op` (`sum`, `count`, `avg`, `min`, `max`, `stddev`, `variance`) and optional `field` |
| `groupBy` | `GroupByDimension[]` | Dimensions to group by — plain field names, `{ field, window }` for time bucketing, or `{ field, unnest: true }` for array fields (one contribution per element) |
| `metrics` | `Record<string, AggregateMetricDef>` | Named metrics with `op` (`sum`, `count`, `avg`, `min`, `max`, `stddev`, `variance`, `percentile`, `distinctCount`) and optional `field`. `percentile` additionally requires `p` in `[0, 1]`. |
| `materialize` | `boolean \| object` | Write results as `NounType.Measurement` entities (auto-visible in OData/Sheets/SSE) |
**Time window granularities:** `'hour'`, `'day'`, `'week'`, `'month'`, `'quarter'`, `'year'`, or `{ seconds: number }` for custom intervals.
@ -539,6 +539,20 @@ const recentFood = await brain.find({
}
```
### `queryAggregate(name, params?)` → `Promise<AggregateResult[]>`
The first-class analytics path — returns plain group rows (`{ groupKey, metrics, count }`) instead of `find()`-style `Result` wrappers. Supports `where` (group-key filter), `having` (SQL-HAVING metric filter), `orderBy`, `order`, `limit`, `offset`:
```typescript
const rows = await brain.queryAggregate('monthly_spending', {
having: { total: { greaterThan: 100 } }, // filter by computed metrics
orderBy: 'total',
order: 'desc',
limit: 10
})
// [{ groupKey: { category: 'food', date: '2024-01' }, metrics: { total: 342.5, count: 28 }, count: 28 }, ...]
```
### How It Works
Aggregation hooks run **outside transactions** on every write operation:
@ -557,16 +571,16 @@ Aggregation hooks run **outside transactions** on every write operation:
### Financial Data Modeling
Brainy supports financial analytics through **metadata conventions** on existing NounTypes — no custom types needed:
Brainy supports financial analytics through **subtypes and metadata conventions** on existing NounTypes — no custom types needed:
```typescript
// Transaction = NounType.Event + financial metadata
// Transaction = NounType.Event + 'transaction' subtype + financial metadata
await brain.add({
data: 'Coffee at Blue Bottle',
type: NounType.Event,
subtype: 'transaction', // top-level standard field (reserved — never in metadata)
metadata: {
domain: 'financial',
subtype: 'transaction',
amount: 5.50,
currency: 'USD',
category: 'food',
@ -575,26 +589,26 @@ await brain.add({
}
})
// Account = NounType.Collection + financial metadata
// Account = NounType.Collection + 'account' subtype + financial metadata
await brain.add({
data: 'Checking Account',
type: NounType.Collection,
subtype: 'account',
metadata: {
domain: 'financial',
subtype: 'account',
accountType: 'checking',
currency: 'USD',
institution: 'Chase'
}
})
// Invoice = NounType.Document + financial metadata
// Invoice = NounType.Document + 'invoice' subtype + financial metadata
await brain.add({
data: 'Invoice #1234 from Acme Corp',
type: NounType.Document,
subtype: 'invoice',
metadata: {
domain: 'financial',
subtype: 'invoice',
amount: 15000,
currency: 'USD',
status: 'pending',
@ -672,32 +686,33 @@ await brain.updateRelation({ id: relId, type: VerbType.WorksWith })
---
### `getRelations(params)` → `Promise<Relation[]>`
### `related(params)` → `Promise<Relation[]>`
Get relationships for an entity.
Get relationships for an entity. Same name and surface as `db.related()` on a
pinned `Db` view.
```typescript
// Get all relationships FROM an entity
const outgoing = await brain.getRelations({ from: entityId })
const outgoing = await brain.related({ from: entityId })
// Get all relationships TO an entity
const incoming = await brain.getRelations({ to: entityId })
const incoming = await brain.related({ to: entityId })
// Filter by type
const related = await brain.getRelations({
const contains = await brain.related({
from: entityId,
type: VerbType.Contains
})
// Filter by subtype (fast path, column-store hit)
const direct = await brain.getRelations({
const direct = await brain.related({
from: entityId,
type: VerbType.ReportsTo,
subtype: 'direct'
})
// Set membership on subtype
const all = await brain.getRelations({
const all = await brain.related({
from: entityId,
type: VerbType.ReportsTo,
subtype: ['direct', 'dotted-line']
@ -739,12 +754,12 @@ console.log(result.failed) // Array of errors
---
### `deleteMany(params)` → `Promise<BatchResult<string>>`
### `removeMany(params)` → `Promise<BatchResult<string>>`
Delete multiple entities.
Remove multiple entities.
```typescript
const result = await brain.deleteMany({
const result = await brain.removeMany({
ids: [id1, id2, id3]
})
```
@ -757,7 +772,7 @@ Update multiple entities.
```typescript
const result = await brain.updateMany({
updates: [
items: [
{ id: id1, metadata: { updated: true } },
{ id: id2, data: 'New content' }
]
@ -772,7 +787,7 @@ Create multiple relationships.
```typescript
const ids = await brain.relateMany({
relations: [
items: [
{ from: id1, to: id2, type: VerbType.RelatedTo },
{ from: id1, to: id3, type: VerbType.Contains }
]
@ -1213,39 +1228,16 @@ const todos = await brain.vfs.getTodos('/src/App.tsx')
---
#### `vfs.getAllTodos(path?)` → `Promise<Todo[]>`
#### `vfs.searchEntities(query)` → `Promise<Array<{ id, path, type, metadata }>>`
Get all TODOs from directory tree.
Search for semantic entities tracked by the VFS, filtered by type, name, or metadata.
```typescript
const allTodos = await brain.vfs.getAllTodos('/src')
```
---
### Project Analysis
#### `vfs.getProjectStats(path?)` → `Promise<Stats>`
Get project statistics.
```typescript
const stats = await brain.vfs.getProjectStats('/projects/my-app')
console.log(stats.fileCount)
console.log(stats.totalSize)
console.log(stats.fileTypes) // Breakdown by extension
```
---
#### `vfs.searchEntities(query)` → `Promise<VFSEntity[]>`
Search for VFS entities by metadata.
```typescript
const tsxFiles = await brain.vfs.searchEntities({
type: 'file',
extension: '.tsx'
const people = await brain.vfs.searchEntities({
type: 'person', // entity type filter
name: 'Ada', // semantic name search
where: { role: 'author' }, // metadata filters
limit: 50
})
```
@ -1280,48 +1272,52 @@ console.log(result.explanation)
---
### `neural().clusters(input?, options?)` → `Promise<Cluster[]>`
### `neural().clusters(input?, options?)` → `Promise<SemanticCluster[]>`
Automatic clustering.
Automatic clustering. Accepts a text query, an array of texts, or a `ClusteringOptions` object.
```typescript
const clusters = await brain.neural().clusters({
algorithm: 'kmeans',
k: 5,
minSize: 3
algorithm: 'kmeans', // 'auto' | 'hierarchical' | 'kmeans' | 'dbscan' | ...
maxClusters: 5,
minClusterSize: 3
})
clusters.forEach(cluster => {
console.log(cluster.label)
console.log(cluster.items)
console.log(cluster.centroid)
console.log(cluster.label) // auto-generated label (when available)
console.log(cluster.members) // entity ids in the cluster
console.log(cluster.centroid) // centroid vector
})
```
---
### `neural().neighbors(id, options?)` → `Promise<Neighbor[]>`
### `neural().neighbors(id, options?)` → `Promise<NeighborsResult>`
Find k-nearest neighbors.
Find nearest neighbors.
```typescript
const neighbors = await brain.neural().neighbors(entityId, {
k: 10,
threshold: 0.7
const result = await brain.neural().neighbors(entityId, {
limit: 10,
minSimilarity: 0.7
})
```
**Options:** `limit?`, `radius?`, `minSimilarity?`, `includeMetadata?`, `sortBy?: 'similarity' | 'importance' | 'recency'`
---
### `neural().outliers(threshold?)` → `Promise<string[]>`
### `neural().outliers(options?)` → `Promise<Outlier[]>`
Detect outlier entities.
```typescript
const outliers = await brain.neural().outliers(0.3)
// Returns entity IDs that are outliers
const outliers = await brain.neural().outliers({ threshold: 0.3 })
// Each outlier carries the entity id plus outlier scoring details
```
**Options:** `threshold?`, `method?: 'isolation' | 'statistical' | 'cluster-based'`, `minNeighbors?`, `includeReasons?`
---
### `neural().visualize(options?)` → `Promise<VizData>`
@ -1342,27 +1338,28 @@ const vizData = await brain.neural().visualize({
### Performance Methods
#### `neural().clusterFast(options)` → `Promise<Cluster[]>`
#### `neural().clusterFast(options?)` → `Promise<SemanticCluster[]>`
Fast clustering for large datasets.
```typescript
const clusters = await brain.neural().clusterFast({
k: 10,
maxIterations: 50
maxClusters: 10
})
```
**Options:** `level?` (hierarchy level), `maxClusters?`
---
#### `neural().clusterLarge(options)` → `Promise<Cluster[]>`
#### `neural().clusterLarge(options?)` → `Promise<SemanticCluster[]>`
Streaming clustering for very large datasets.
Sampling-based clustering for very large datasets.
```typescript
const clusters = await brain.neural().clusterLarge({
k: 20,
batchSize: 1000
sampleSize: 1000,
strategy: 'diverse' // 'random' | 'diverse' | 'recent'
})
```
@ -1381,17 +1378,15 @@ await brain.import('data.csv', {
createEntities: true
})
// Excel import
// Excel import (all sheets processed automatically)
await brain.import('sales.xlsx', {
format: 'excel',
sheets: ['Q1', 'Q2']
vfsPath: '/imports/sales', // optional: mirror into the VFS
groupBy: 'sheet'
})
// PDF import
await brain.import('research.pdf', {
format: 'pdf',
extractTables: true
})
// PDF import (tables extracted automatically)
await brain.import('research.pdf', { format: 'pdf' })
// URL import
await brain.import('https://api.example.com/data.json')
@ -1400,10 +1395,17 @@ await brain.import('https://api.example.com/data.json')
**Parameters:**
- `source`: `string | Buffer | object` - File path, URL, buffer, or object
- `options?`: Import configuration
- `format?`: `'csv' | 'excel' | 'pdf' | 'json'` - Auto-detected if omitted
- `format?`: `'excel' | 'pdf' | 'csv' | 'json' | 'markdown' | 'yaml' | 'docx' | 'image'` - Auto-detected if omitted
- `vfsPath?`: `string` - Mirror imported content into the VFS at this path
- `groupBy?`: `'type' | 'sheet' | 'flat' | 'custom'` - VFS grouping strategy
- `createEntities?`: `boolean` - Create entities from rows
- `sheets?`: `string[]` - Excel sheets to import
- `extractTables?`: `boolean` - Extract tables from PDF
- `createRelationships?`: `boolean` - Create relationships between extracted entities
- `preserveSource?`: `boolean` - Save the original file in the VFS
- `enableNeuralExtraction?`: `boolean` - Extract entity names via AI
- `enableRelationshipInference?`: `boolean` - Infer relationships via AI
- `enableConceptExtraction?`: `boolean` - Extract entity types via AI
- `confidenceThreshold?`: `number` - Minimum confidence for extracted entities
- `onProgress?`: `(progress) => void` - Progress callback (stage, counts, throughput, ETA)
**Returns:** `Promise<ImportResult>` - Import statistics
@ -1414,9 +1416,6 @@ await brain.import('https://api.example.com/data.json')
### Export & Snapshots
```typescript
// Export to file
await brain.export('/path/to/backup.brainy')
// Instant hard-link snapshot via the Db API
const pin = brain.now()
await pin.persist('/backups/2026-06-11')
@ -1453,12 +1452,11 @@ const brain = new Brainy({
// Model: all-MiniLM-L6-v2 (384 dimensions)
// Device: CPU via WASM (works everywhere)
// Cache configuration
cache: {
enabled: true,
maxSize: 10000,
ttl: 3600000 // 1 hour in ms
}
// Cache configuration — `true`/`false`, or an options object
cache: {
maxSize: 10000,
ttl: 3600000 // 1 hour in ms
}
})
await brain.init() // Required! VFS auto-initialized
@ -1672,7 +1670,7 @@ When strict mode is on:
- Brainy's own VFS infrastructure writes bypass via the `metadata.isVFSEntity: true` marker.
- Per-type registrations always apply regardless of the brain-wide flag.
Becomes the default in 8.0.0.
The default since 8.0.0 — pass `requireSubtype: false` to opt out while migrating pre-8.0 data.
#### `trackField(name, options?)` → `void`
@ -1718,6 +1716,27 @@ await brain.migrateField({
Returns `{ scanned: number, migrated: number, skipped: number, errors: Array<{id, error}> }`.
#### `fillSubtypes(rules, options?)` → `Promise<FillSubtypesResult>` (8.0+)
Back-fill missing `subtype` values across entities AND relationships in one streaming pass — the migration companion to `audit()`. Keys are `NounType`/`VerbType` values; each rule is a literal subtype string or a function deriving one from the entry (return `undefined` to decline). Idempotent: entries that already carry a subtype are never touched, so a crashed run is resumed safely by re-running.
```typescript
const report = await brain.fillSubtypes({
[NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified', // derived
[NounType.Document]: 'general', // literal default
[VerbType.RelatedTo]: 'unspecified' // relationship rule
})
// → { scanned, filled, skipped, errors, byType }
```
**Parameters:**
- `rules`: `FillSubtypeRules` - Map of NounType/VerbType → literal subtype or `(entry) => string | undefined`
- `options.includeVFS?`: `boolean` - Also fill VFS infrastructure entries (default `false`)
- `options.batchSize?`: `number` - Pagination batch size (default `200`)
- `options.onProgress?`: `(progress: { scanned, filled, skipped }) => void` - Per-batch callback
Returns `{ scanned, filled, skipped, errors, byType }`. After a clean run, `skipped` equals the remaining `audit().total`. See the [migration recipe](../guides/subtypes-and-facets.md).
---
### `embed(data)` → `Promise<number[]>` ✨
@ -1812,7 +1831,7 @@ for (const group of duplicates) {
// Find person duplicates with higher threshold
const personDupes = await brain.findDuplicates({
type: NounType.PERSON,
type: NounType.Person,
threshold: 0.9,
limit: 50
})
@ -1885,15 +1904,19 @@ const docClusters = await brain.cluster({
---
### `getStats()` → `Statistics`
### `getStats(options?)` → `Promise<Statistics>`
Get comprehensive statistics.
Get complete entity/relationship statistics (convenience wrapper over `brain.counts`).
```typescript
const stats = brain.getStats()
console.log(stats.entityCount)
console.log(stats.relationshipCount)
console.log(stats.cacheHitRate)
const stats = await brain.getStats()
console.log(stats.entities.total) // total entity count
console.log(stats.entities.byType) // counts per NounType
console.log(stats.relationships) // relationship stats
console.log(stats.density) // relationships per entity
// Exclude VFS infrastructure entities from the counts
const semanticOnly = await brain.getStats({ excludeVFS: true })
```
---
@ -1914,7 +1937,7 @@ VFS is auto-initialized during `brain.init()` - no separate `vfs.init()` needed!
### Shutdown
```typescript
await brain.shutdown() // Graceful shutdown, flush caches
await brain.close() // Graceful shutdown — flushes pending writes and releases the writer lock
```
---
@ -1940,8 +1963,8 @@ await brain.update({
metadata: { updated: true }
})
// Delete
await brain.delete(id)
// Remove
await brain.remove(id)
```
---
@ -2001,7 +2024,7 @@ const results = await brain.find({
// Speculate: what would this change look like? (nothing touches disk)
const base = brain.now()
const whatIf = await base.with([
{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature' }
{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature', metadata: { draft: true } }
])
await whatIf.find({ where: { draft: true } })
await whatIf.release()
@ -2009,7 +2032,7 @@ await base.release()
// Commit it for real — one atomic generation, with audit metadata
await brain.transact(
[{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature' }],
[{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature', metadata: { draft: true } }],
{ meta: { author: 'dev@example.com', message: 'Add new feature' } }
)
```
@ -2115,7 +2138,8 @@ For the full taxonomy with all 169 types and their descriptions, see:
- **[Stage 3 CANONICAL Taxonomy](../STAGE3-CANONICAL-TAXONOMY.md)** - Complete list with categories
- **[Noun-Verb Taxonomy Architecture](../architecture/noun-verb-taxonomy.md)** - Design rationale
### Migration from
### Migration from pre-Stage-3 taxonomies
**Breaking Changes:**
- `NounType.Content` removed → Use `Document`, `Message`, or `InformationContent`
- `NounType.User` removed → Use `Person` or `Agent`
@ -2161,7 +2185,6 @@ For the full taxonomy with all 169 types and their descriptions, see:
- **[Find System](../FIND_SYSTEM.md)** - Natural language find() details
- **[VFS Quick Start](../vfs/QUICK_START.md)** - Complete VFS documentation
- **[Import Anything Guide](../guides/import-anything.md)** - CSV, Excel, PDF, URL imports
- **[Cloud Deployment](../deployment/CLOUD_DEPLOYMENT_GUIDE.md)** - Production deployment
- **[Consistency Model](../concepts/consistency-model.md)** - The guarantees behind the Db API
- **[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md)** - Backup, restore, what-if, audit recipes

2
docs/architecture/aggregation.md
View file

@ -97,7 +97,7 @@ On `brain.update(entity)`, the engine reverses the old entity's contribution and
### Delete Handling
On `brain.delete(id)`, the engine reverses the entity's contribution:
On `brain.remove(id)`, the engine reverses the entity's contribution:
- `count` and `sum` are decremented
- `min`/`max` may become stale (marked in `staleMinMax` for lazy recompute)

359
docs/architecture/augmentation-system-audit.md
View file

@ -1,359 +0,0 @@
# 🔍 Brainy 2.0 Augmentation System Architecture Audit (REVISED)
**Author**: Senior Architecture Review
**Date**: 2025-08-25
**Status**: 🟡 **WORKING BUT NEEDS MARKETPLACE FEATURES**
## Executive Summary
The augmentation system core execution is WORKING correctly through `AugmentationRegistry`. The system properly executes augmentations before/after operations. However, there's no discovery, installation, or marketplace integration for the brain-cloud registry vision.
---
## 🟢 What's Actually Working
### 1. Execution Mechanism ✅
The `AugmentationRegistry` class properly implements:
```typescript
async execute<T>(operation: string, params: any, mainOperation: () => Promise<T>): Promise<T>
```
- Chains augmentations correctly
- Respects timing (before/after/around)
- Handles operation filtering
- Works with all 27 augmentations
### 2. Registration System ✅
```typescript
brain.augmentations.register(augmentation)
```
- Two-phase initialization works (storage first)
- Context injection works
- Lifecycle management works
### 3. Clean Interface ✅
- 100% of augmentations use `BrainyAugmentation`
- `BaseAugmentation` provides solid foundation
- Proper TypeScript types
### 4. Auto-Configuration ✅
```typescript
new Brainy({
cache: true, // Auto-registers CacheAugmentation
index: true, // Auto-registers IndexAugmentation
storage: { type: 'filesystem', rootDirectory: './data' } // Auto-registers FileSystemStorageAugmentation
})
```
---
## 🟡 Missing for Marketplace Vision
### 1. No Package Discovery
**Current**: Manual registration only
```typescript
// Current (manual)
import { NotionSynapse } from './my-custom-synapse'
brain.augmentations.register(new NotionSynapse())
// Needed
await brain.discover('notion') // Search npm/brain-cloud
await brain.install('@soulcraft/notion-synapse')
```
### 2. No Installation Mechanism
**Current**: Must be bundled at build time
**Needed**: Dynamic installation
```typescript
interface AugmentationMarketplace {
search(query: string): Promise<Package[]>
install(packageId: string): Promise<void>
uninstall(packageId: string): Promise<void>
listInstalled(): Promise<Package[]>
checkUpdates(): Promise<Update[]>
}
```
### 3. No Brain Cloud Registry Client
**Current**: No registry concept
**Needed**: Registry integration
```typescript
class BrainCloudRegistry {
private apiUrl = 'https://api.soulcraft.com/brain-cloud'
async search(query: string): Promise<AugmentationPackage[]> {
const response = await fetch(`${this.apiUrl}/augmentations/search?q=${query}`)
return response.json()
}
async getPackage(id: string): Promise<AugmentationPackage> {
const response = await fetch(`${this.apiUrl}/augmentations/${id}`)
return response.json()
}
}
```
### 4. No License Management
**Current**: All augmentations free/bundled
**Needed**: License verification
```typescript
interface LicenseManager {
verify(packageId: string, licenseKey: string): Promise<boolean>
activate(packageId: string, licenseKey: string): Promise<void>
deactivate(packageId: string): Promise<void>
getStatus(packageId: string): Promise<LicenseStatus>
}
```
### 5. No Version Management
**Current**: No versioning
**Needed**: Semver support
```typescript
interface VersionManager {
checkCompatibility(pkg: Package, brainyVersion: string): boolean
resolveConflicts(packages: Package[]): Package[]
upgrade(packageId: string, toVersion: string): Promise<void>
}
```
---
## 📋 Implementation Plan for Marketplace
### Phase 1: Local Package Discovery (1 week)
```typescript
class LocalPackageDiscovery {
async discover(): Promise<Package[]> {
// 1. Search node_modules for brainy augmentations
const packages = await glob('node_modules/@*/package.json')
// 2. Filter for brainy augmentations
return packages.filter(pkg => pkg.brainy?.type === 'augmentation')
}
async load(packageId: string): Promise<BrainyAugmentation> {
// Dynamic import
const module = await import(packageId)
return new module.default()
}
}
```
### Phase 2: NPM Integration (1 week)
```typescript
class NPMRegistry {
async search(query: string): Promise<Package[]> {
// Search npm for packages with brainy keyword
const response = await fetch(
`https://registry.npmjs.org/-/v1/search?text=${query}+keywords:brainy-augmentation`
)
return response.json()
}
async install(packageId: string): Promise<void> {
// Use npm programmatically
await exec(`npm install ${packageId}`)
// Auto-register after install
const aug = await this.load(packageId)
this.brain.augmentations.register(aug)
}
}
```
### Phase 3: Brain Cloud Registry (2 weeks)
```typescript
class BrainCloudMarketplace {
private registry = new BrainCloudRegistry()
private licenses = new LicenseManager()
private installer = new AugmentationInstaller()
async browse(category?: string): Promise<MarketplaceListing[]> {
const packages = await this.registry.list(category)
return packages.map(pkg => ({
...pkg,
installed: this.isInstalled(pkg.id),
licensed: this.isLicensed(pkg.id),
updates: this.hasUpdates(pkg.id)
}))
}
async purchase(packageId: string): Promise<void> {
// 1. Process payment
const license = await this.processPayment(packageId)
// 2. Activate license
await this.licenses.activate(packageId, license)
// 3. Install package
await this.install(packageId)
}
}
```
### Phase 4: Developer Tools (1 week)
```typescript
// CLI for augmentation development
class AugmentationCLI {
async create(name: string): Promise<void> {
// Scaffold new augmentation project
await this.scaffold(name, 'augmentation-template')
}
async test(path: string): Promise<void> {
// Test augmentation locally
const aug = await this.load(path)
await this.runTests(aug)
}
async publish(path: string): Promise<void> {
// Publish to brain-cloud
const pkg = await this.package(path)
await this.registry.publish(pkg)
}
}
```
---
## 🏗️ Recommended Architecture
### 1. Augmentation Package Structure
```json
{
"name": "@soulcraft/notion-synapse",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"brainy": {
"type": "augmentation",
"class": "NotionSynapse",
"timing": "after",
"operations": ["addNoun", "updateNoun"],
"priority": 20,
"license": "premium",
"price": 9.99,
"compatibility": ">=2.0.0",
"dependencies": []
},
"keywords": ["brainy-augmentation", "notion", "sync"]
}
```
### 2. Installation Flow
```typescript
// User flow
await brain.marketplace.search('notion')
// Returns: [@soulcraft/notion-synapse, @community/notion-sync, ...]
await brain.marketplace.install('@soulcraft/notion-synapse')
// 1. Check license (prompt for purchase if needed)
// 2. Check compatibility
// 3. Install dependencies
// 4. Download package
// 5. Load augmentation
// 6. Register with brain
// 7. Initialize
// Now it's working!
brain.augmentations.list()
// [..., { name: '@soulcraft/notion-synapse', enabled: true }]
```
### 3. Discovery UI
```typescript
// Web UI component
<AugmentationMarketplace>
<SearchBar />
<Categories>
<Category name="Storage" count={12} />
<Category name="Sync" count={8} />
<Category name="AI" count={15} />
</Categories>
<Results>
<AugmentationCard
name="Notion Synapse"
author="Soulcraft"
rating={4.8}
installs={1200}
price={9.99}
onInstall={...}
/>
</Results>
</AugmentationMarketplace>
```
---
## 🎯 Priority for 2.0 Release
### Must Have (Release Blockers)
- ✅ Working execution (DONE)
- ✅ Clean interface (DONE)
- ✅ Documentation (DONE)
- ⏳ Fix augmentationPipeline.ts removal
- ⏳ Test all 27 augmentations work
### Nice to Have (2.0.x)
- Local package discovery
- NPM integration
- Basic CLI tools
### Future (2.1+)
- Brain Cloud Registry
- License management
- Payment processing
- Marketplace UI
- Developer portal
---
## 📊 Current State Assessment
| Component | Status | Notes |
|-----------|--------|-------|
| Core Execution | ✅ Working | AugmentationRegistry.execute() works |
| Registration | ✅ Working | Manual registration works |
| Auto-Config | ✅ Working | Cache, index, storage auto-register |
| Lifecycle | ✅ Working | Init, execute, shutdown work |
| Discovery | ❌ Missing | No package discovery |
| Installation | ❌ Missing | No dynamic installation |
| Marketplace | ❌ Missing | No registry client |
| Licensing | ❌ Missing | No license management |
| Versioning | ❌ Missing | No version checks |
---
## 💡 Recommendations
### For 2.0 Release
1. **Ship with manual registration** - It works!
2. **Document how to create augmentations** - Critical for adoption
3. **Create 2-3 example augmentations** - Show the patterns
4. **Add basic CLI for testing** - Help developers
### For 2.1 (Q1 2025)
1. **Add NPM discovery** - Find installed augmentations
2. **Dynamic loading** - Import augmentations at runtime
3. **Basic marketplace API** - List available augmentations
4. **Version checking** - Ensure compatibility
### For 3.0 (Q2 2025)
1. **Full marketplace** - Browse, search, install
2. **Payment integration** - Premium augmentations
3. **Developer portal** - Publish augmentations
4. **Enterprise features** - Private registries
---
## ✅ Good News Summary
The augmentation system WORKS! The core architecture is solid:
- Execution mechanism is correct
- Registration works
- Lifecycle management works
- All 27 augmentations function properly
What's missing is the marketplace/discovery layer, which can be added incrementally without breaking the core system. The 2.0 release can ship with manual augmentation registration, and the marketplace features can be added in 2.1+.
**Recommendation: Ship 2.0 with current system, add marketplace in 2.1**

2
docs/architecture/data-storage-architecture.md
View file

@ -195,7 +195,7 @@ tx-log line is appended last (advisory). Crash recovery on open discards any
Two write classes share the generation clock:
- **Single-operation writes** (`add`/`update`/`delete`/`relate` outside
- **Single-operation writes** (`add`/`update`/`remove`/`relate` outside
`transact()`) bump `generation.json` so watermarks and `_rev` CAS stay sound,
but write **no** history — they are not visible to `db.since()` and remain
visible through earlier pins.

41
docs/architecture/storage-architecture.md
View file

@ -239,37 +239,30 @@ await brain.storage.withLock('resource-id', async () => {
## Migration and Backup
### Export Data
Backup and restore go through the Db API — see
[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) for the full
recipe book.
### Snapshot (backup)
```typescript
// Export entire database
const backup = await brain.export({
format: 'json',
includeVectors: true,
includeIndexes: false
})
// Instant, self-contained snapshot (hard links on filesystem storage)
const db = brain.now()
await db.persist('/backups/2026-06-11')
await db.release()
```
### Import Data
### Restore
```typescript
// Import from backup
await brain.import(backup, {
mode: 'merge', // or 'replace'
validateSchema: true
})
// Replace the store's entire state from a snapshot (destructive — confirm required)
await brain.restore('/backups/2026-06-11', { confirm: true })
```
### Storage Migration
### Move to a new directory
```typescript
// Migrate between storage types
const oldBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './old' } })
const newBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './new' } })
await oldBrain.init()
await newBrain.init()
// Transfer all data
const data = await oldBrain.export()
await newBrain.import(data)
// A snapshot directory is a complete store: restore it into a fresh brain
const brain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './new' } })
await brain.init()
await brain.restore('/backups/2026-06-11', { confirm: true })
```
## Performance Tuning

12
docs/architecture/zero-config.md
View file

@ -1,11 +1,11 @@
---
title: Zero Configuration
slug: concepts/zero-config
public: true
public: false
category: concepts
template: concept
order: 3
description: Brainy auto-detects storage, initializes embeddings, and builds indexes — no configuration required. Works in Node.js, Bun, OPFS, and cloud environments.
description: Brainy auto-detects storage, initializes embeddings, and builds indexes — no configuration required. Works in Node.js and Bun (server-only since 8.0).
next:
- getting-started/installation
- guides/storage-adapters
@ -13,7 +13,13 @@ next:
# Zero Configuration & Auto-Adaptation
> **Current Status**: Basic zero-config is fully functional. Advanced auto-adaptation features are in development.
> **Status (8.0):** This document predates Brainy 8.0 and needs a rewrite before
> republication. Large parts describe storage backends and environments that 8.0
> removed (browser/OPFS/IndexedDB, edge KV, S3) or features that were never built
> (model auto-selection, workload detection). Brainy 8.0 is server-only (Node.js/Bun)
> with two storage adapters: `memory` and `filesystem` — see
> [Storage Adapters](../concepts/storage-adapters.md) for the accurate story.
> Basic zero-config (`new Brainy()` with auto-selected storage) works as described.
## Overview

396
docs/augmentations/COMPLETE-REFERENCE.md
View file

@ -1,396 +0,0 @@
# 🔌 Brainy Augmentations Complete Reference
> **All augmentations that power Brainy's extensibility - with locations, usage, and examples**
>
> **⚠️ Update**: Updated for metadata structure changes and billion-scale optimizations
## Quick Start
```typescript
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy({
// Augmentations auto-configure based on environment
storage: { type: 'auto', rootDirectory: './brainy-data' }, // Storage augmentation
cache: true, // Cache augmentation
index: true // Index augmentation
})
await brain.init() // Augmentations initialize automatically
```
## Augmentation Architecture
### Key Improvements for Billion-Scale Performance
1. **Metadata/Vector Separation**: Augmentations now work with separated metadata and vectors
- Metadata stored separately from vector data
- 99.2% memory reduction for type tracking
- Two-file storage pattern for optimal I/O
2. **Type System Enforcement**: All metadata requires type fields
- `NounMetadata` requires `noun: NounType`
- `VerbMetadata` requires `verb: VerbType`
- Type inference system available as public API
3. **Storage Adapter Pattern**: Internal vs public method distinction
- `_methods`: Return pure structures (HNSWNoun, HNSWVerb)
- Public methods: Return WithMetadata types
- MetadataEnforcer Proxy ensures proper access
### What This Means for Augmentation Users
**✅ If you use built-in augmentations**: No changes needed! They're all updated.
**⚠️ If you created custom storage augmentations**: Update your storage adapter to:
- Wrap metadata with required `noun`/`verb` fields
- Follow the internal/public method pattern
- Use two-file storage approach
**⚠️ If you access relationship data**: Change `verb.type` to `verb.verb`
## Core Concepts
### What are Augmentations?
Augmentations are modular extensions that add functionality to Brainy without cluttering the core API. They follow a unified interface and can be:
- **Auto-enabled**: Based on configuration (cache, index, storage)
- **Manually registered**: For custom functionality
- **Chained**: Multiple augmentations work together seamlessly
- **Billion-scale ready**: Optimized for datasets with billions of nouns and verbs
### Augmentation Lifecycle
1. **Registration**: Augmentations register before init()
2. **Initialization**: Two-phase init (storage first, then others)
3. **Execution**: Hook into operations (before/after/both)
4. **Shutdown**: Clean teardown on brain.shutdown()
---
## Storage Augmentations
### MemoryStorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Auto-enabled**: When `storage: { type: 'memory' }` or in test environments
**Purpose**: In-memory storage for testing and temporary data
```typescript
const brain = new Brainy({ storage: { type: 'memory' } })
```
### FileSystemStorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Auto-enabled**: When `storage: { type: 'filesystem' }` or Node.js detected
**Purpose**: Persistent file-based storage for Node.js applications
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', rootDirectory: './data' }
})
```
For off-site backup, snapshot `rootDirectory` from your scheduler using `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` — there are no cloud storage augmentations.
### StorageAugmentation (base)
**Location**: `src/augmentations/storageAugmentation.ts`
**Purpose**: Base class for custom storage implementations
### DynamicStorageAugmentation
**Location**: `src/augmentations/storageAugmentation.ts`
**Purpose**: Runtime storage adapter switching
---
## Performance Augmentations (7 total)
### CacheAugmentation
**Location**: `src/augmentations/cacheAugmentation.ts`
**Auto-enabled**: When `cache: true` (default)
**Purpose**: LRU cache for search results and frequent queries
```typescript
brain.clearCache() // Exposed via API
brain.getCacheStats() // Cache hit/miss statistics
```
### IndexAugmentation
**Location**: `src/augmentations/indexAugmentation.ts`
**Auto-enabled**: When `index: true` (default)
**Purpose**: Metadata indexing for O(1) field lookups
```typescript
brain.rebuildMetadataIndex() // Exposed via API
// Enables fast where queries:
brain.find({ where: { category: 'tech' } })
```
### MetricsAugmentation
**Location**: `src/augmentations/metricsAugmentation.ts`
**Auto-enabled**: Always active
**Purpose**: Performance metrics and statistics collection
```typescript
brain.getStats() // Comprehensive metrics
```
### MonitoringAugmentation
**Location**: `src/augmentations/monitoringAugmentation.ts`
**Manual**: Register for detailed monitoring
**Purpose**: Real-time performance monitoring and alerts
### BatchProcessingAugmentation
**Location**: `src/augmentations/batchProcessingAugmentation.ts`
**Auto-enabled**: For batch operations
**Purpose**: Optimizes bulk add/update/delete operations
```typescript
brain.addNouns([...]) // Automatically batched
```
### RequestDeduplicatorAugmentation
**Location**: `src/augmentations/requestDeduplicatorAugmentation.ts`
**Auto-enabled**: Always active
**Purpose**: Prevents duplicate concurrent operations
---
## Data Integrity Augmentations (3 total)
**Auto-enabled**: When `wal: true`
**Purpose**: Write-ahead logging for crash recovery
```typescript
const brain = new Brainy({ wal: true })
// Automatic recovery on restart after crash
```
### EntityRegistryAugmentation
**Location**: `src/augmentations/entityRegistryAugmentation.ts`
**Auto-enabled**: For streaming operations
**Purpose**: High-speed deduplication for real-time data
```typescript
// Prevents duplicate entities in streaming scenarios
brain.add(data) // Automatically deduplicated
```
### AutoRegisterEntitiesAugmentation
**Location**: `src/augmentations/entityRegistryAugmentation.ts`
**Manual**: For automatic entity discovery
**Purpose**: Auto-discovers and registers entities from data
---
## Intelligence Augmentations (2 total)
### NeuralImportAugmentation
**Location**: `src/augmentations/neuralImport.ts`
**Manual**: Via `brain.neuralImport()`
**Purpose**: AI-powered smart data import
```typescript
const result = await brain.neuralImport(data, {
confidenceThreshold: 0.7,
autoApply: true
})
// Automatically detects entities and relationships
```
### IntelligentVerbScoringAugmentation
**Location**: `src/augmentations/intelligentVerbScoringAugmentation.ts`
**Auto-enabled**: When verbs are used
**Purpose**: ML-based relationship strength scoring
```typescript
brain.verbScoring.train(feedback)
brain.verbScoring.getScore(verbId)
```
---
## Communication Augmentations (4 total)
### APIServerAugmentation
**Location**: `src/augmentations/apiServerAugmentation.ts`
**Manual**: For server deployments
**Purpose**: REST/WebSocket/MCP API server
```typescript
const augmentation = new APIServerAugmentation()
await brain.registerAugmentation(augmentation)
// Exposes full Brainy API over network
```
### WebSocketConduitAugmentation
**Location**: `src/augmentations/conduitAugmentations.ts`
**Manual**: For Brainy-to-Brainy sync
**Purpose**: Real-time sync between Brainy instances
```typescript
const conduit = new WebSocketConduitAugmentation()
await conduit.establishConnection('ws://other-brain')
```
### ServerSearchConduitAugmentation
**Location**: `src/augmentations/serverSearchAugmentations.ts`
**Manual**: For client-server search
**Purpose**: Search remote Brainy instance, cache locally
### ServerSearchActivationAugmentation
**Location**: `src/augmentations/serverSearchAugmentations.ts`
**Manual**: Works with ServerSearchConduit
**Purpose**: Triggers and manages server search operations
---
## External Integration (2 total)
### SynapseAugmentation (base)
**Location**: `src/augmentations/synapseAugmentation.ts`
**Purpose**: Base class for external platform integrations
```typescript
// Example: NotionSynapse, SlackSynapse, etc.
class NotionSynapse extends SynapseAugmentation {
async fetchData() { /* Notion API calls */ }
async pushData() { /* Sync to Notion */ }
}
```
### ExampleFileSystemSynapse
**Location**: `src/augmentations/synapseAugmentation.ts`
**Purpose**: Example implementation for file system sync
---
## Augmentation Configuration
### Auto-Configuration
```typescript
const brain = new Brainy({
// These auto-register augmentations:
storage: { type: 'auto', rootDirectory: './brainy-data' }, // Storage augmentation
cache: true, // Cache augmentation
index: true, // Index augmentation
metrics: true // Metrics augmentation
})
```
### Manual Registration
```typescript
const brain = new Brainy()
// Register before init()
const customAug = new MyCustomAugmentation()
await brain.registerAugmentation(customAug)
await brain.init()
```
### Creating Custom Augmentations
```typescript
import { BaseAugmentation } from '@soulcraft/brainy'
class MyAugmentation extends BaseAugmentation {
readonly name = 'my-augmentation'
readonly timing = 'after' // before | after | both
readonly operations = ['addNoun', 'search'] // Which ops to hook
readonly priority = 10 // Execution order (lower = earlier)
protected async onInit(): Promise<void> {
// Initialize your augmentation
}
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'addNoun') {
console.log('Noun added:', params)
}
}
protected async onShutdown(): Promise<void> {
// Cleanup
}
}
```
---
## Augmentation Timing & Priority
### Timing Options
- **`before`**: Runs before the operation (can modify params)
- **`after`**: Runs after the operation (can see results)
- **`both`**: Runs before AND after
### Priority (lower = earlier)
1. Storage augmentations (priority: 0)
2. Cache/Index augmentations (priority: 5-10)
3. Monitoring/Metrics (priority: 15-20)
4. Conduits/Synapses (priority: 20-30)
---
## Key Integration Points
### Where Augmentations Hook In
**Brainy Constructor**:
- Storage augmentations register based on config
- Cache/Index augmentations auto-register if enabled
**brain.init()**:
- Two-phase initialization (storage first, then others)
- Augmentations can access brain instance via context
**Operations** (addNoun, search, etc.):
- Augmentations execute based on timing and operations filter
- Can modify params (before) or see results (after)
**brain.shutdown()**:
- All augmentations cleaned up in reverse order
---
## Performance Impact
Most augmentations have minimal overhead:
- **Cache**: ~1ms per search (saves 10-100ms on hits)
- **Index**: ~1ms per operation (saves 100ms+ on queries)
- **Metrics**: <1ms per operation
- **Storage**: Varies by adapter (memory: 0ms, filesystem: 1-10ms)
---
## Best Practices
1. **Let auto-configuration work**: Most apps need zero manual config
2. **Storage first**: Always configure storage before other augmentations
3. **Use built-in augmentations**: They're optimized and battle-tested
4. **Custom augmentations**: Extend BaseAugmentation for consistency
5. **Respect timing**: Use 'before' to modify, 'after' to observe
6. **Mind priority**: Lower numbers execute first
---
## Troubleshooting
### Augmentation not working?
```typescript
// Check if registered
brain.listAugmentations()
// Check if enabled
brain.isAugmentationEnabled('cache')
// Enable/disable at runtime
brain.enableAugmentation('cache')
brain.disableAugmentation('cache')
```
### Performance issues?
```typescript
// Check augmentation overhead
const stats = brain.getStats()
console.log(stats.augmentations)
// Disable non-critical augmentations
brain.disableAugmentation('monitoring')
```
---
---
*Augmentations make Brainy infinitely extensible while keeping the core API clean and simple!*

620
docs/augmentations/CONFIGURATION.md
View file

@ -1,620 +0,0 @@
# Augmentation Configuration System
**Version**: 2.0.0
**Status**: Production Ready
## Overview
The Brainy Augmentation Configuration System provides a VSCode-style extension architecture with multiple configuration sources, schema validation, and tool discovery. This system maintains Brainy's zero-config philosophy while enabling sophisticated enterprise configuration management.
## Table of Contents
- [Quick Start](#quick-start)
- [Configuration Sources](#configuration-sources)
- [Creating Configurable Augmentations](#creating-configurable-augmentations)
- [Configuration Discovery](#configuration-discovery)
- [Runtime Configuration](#runtime-configuration)
- [Environment Variables](#environment-variables)
- [Configuration Files](#configuration-files)
- [CLI Commands](#cli-commands)
- [Tool Integration](#tool-integration)
- [Migration Guide](#migration-guide)
## Quick Start
### Using an Augmentation with Configuration
```typescript
import { Brainy } from '@soulcraft/brainy'
// Zero-config (uses defaults)
const brain = new Brainy()
// With custom configuration
immediateWrites: true,
checkpointInterval: 300000 // 5 minutes
}))
```
### Configuring via Environment Variables
```bash
export BRAINY_AUG_CACHE_TTL=600000
```
### Configuring via Files
Create a `.brainyrc` file in your project root:
```json
{
"augmentations": {
"wal": {
"enabled": true,
"immediateWrites": true,
"maxSize": 20971520
},
"cache": {
"ttl": 600000,
"maxSize": 2000
}
}
}
```
## Configuration Sources
Configuration is resolved in the following priority order (highest to lowest):
1. **Runtime Updates** - Dynamic configuration changes via API
2. **Constructor Parameters** - Code-time configuration
3. **Environment Variables** - `BRAINY_AUG_<NAME>_<KEY>`
4. **Configuration Files** - `.brainyrc`, `brainy.config.json`
5. **Schema Defaults** - Default values from manifest
### Resolution Example
```typescript
// Schema default
{ maxSize: 10485760 }
// File configuration (.brainyrc)
{ maxSize: 20971520 }
// Environment variable
// Constructor parameter
// Final resolved value: 41943040 (constructor wins)
```
## Creating Configurable Augmentations
### Step 1: Extend ConfigurableAugmentation
```typescript
import { ConfigurableAugmentation, AugmentationManifest } from '@soulcraft/brainy'
export class MyAugmentation extends ConfigurableAugmentation {
name = 'my-augmentation'
timing = 'around' as const
metadata = 'none' as const
operations = ['search', 'add']
priority = 50
constructor(config?: MyConfig) {
super(config) // Handles configuration resolution
}
// Required: Provide manifest for discovery
getManifest(): AugmentationManifest {
return {
id: 'my-augmentation',
name: 'My Augmentation',
version: '1.0.0',
description: 'Does something amazing',
category: 'performance',
configSchema: {
type: 'object',
properties: {
enabled: {
type: 'boolean',
default: true,
description: 'Enable this augmentation'
},
threshold: {
type: 'number',
default: 100,
minimum: 1,
maximum: 1000,
description: 'Processing threshold'
}
}
}
}
}
// Optional: Handle runtime configuration changes
protected async onConfigChange(newConfig: MyConfig, oldConfig: MyConfig): Promise<void> {
if (newConfig.threshold !== oldConfig.threshold) {
// React to threshold change
this.updateThreshold(newConfig.threshold)
}
}
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
if (!this.config.enabled) {
return next()
}
// Your augmentation logic here
return next()
}
}
```
### Step 2: Define Configuration Interface
```typescript
interface MyConfig {
enabled?: boolean
threshold?: number
mode?: 'fast' | 'balanced' | 'thorough'
}
```
### Step 3: Add JSON Schema in Manifest
```typescript
configSchema: {
type: 'object',
properties: {
enabled: {
type: 'boolean',
default: true,
description: 'Enable augmentation'
},
threshold: {
type: 'number',
default: 100,
minimum: 1,
maximum: 1000,
description: 'Processing threshold'
},
mode: {
type: 'string',
default: 'balanced',
enum: ['fast', 'balanced', 'thorough'],
description: 'Processing mode'
}
},
required: [],
additionalProperties: false
}
```
## Configuration Discovery
The Discovery API allows tools to discover and configure augmentations dynamically:
```typescript
import { AugmentationDiscovery } from '@soulcraft/brainy'
const discovery = new AugmentationDiscovery(brain.augmentations)
// Discover all augmentations with manifests
const listings = await discovery.discover({
includeConfig: true,
includeSchema: true
})
// Get configuration schema
const schema = await discovery.getConfigSchema('wal')
// Validate configuration
const validation = await discovery.validateConfig('wal', {
enabled: true,
maxSize: 'invalid' // Will fail validation
})
// Update configuration at runtime
await discovery.updateConfig('wal', {
checkpointInterval: 120000
})
```
## Runtime Configuration
### Update Configuration Dynamically
```typescript
// Get augmentation
const wal = brain.augmentations.get('wal')
// Update configuration
await wal.updateConfig({
checkpointInterval: 300000
})
// Get current configuration
const config = wal.getConfig()
```
### React to Configuration Changes
```typescript
class MyAugmentation extends ConfigurableAugmentation {
protected async onConfigChange(newConfig: any, oldConfig: any): Promise<void> {
// Stop old processes
if (oldConfig.enabled && !newConfig.enabled) {
await this.stop()
}
// Start new processes
if (!oldConfig.enabled && newConfig.enabled) {
await this.start()
}
// Update settings
if (newConfig.interval !== oldConfig.interval) {
this.rescheduleTimer(newConfig.interval)
}
}
}
```
## Environment Variables
### Naming Convention
```bash
BRAINY_AUG_<AUGMENTATION_ID>_<CONFIG_KEY>=value
```
### Examples
```bash
# Cache augmentation
BRAINY_AUG_CACHE_ENABLED=true
BRAINY_AUG_CACHE_MAX_SIZE=2000
BRAINY_AUG_CACHE_TTL=600000
# Complex values (JSON)
BRAINY_AUG_MYAUG_FILTERS='["*.js","*.ts"]'
BRAINY_AUG_MYAUG_OPTIONS='{"deep":true,"follow":false}'
```
### Docker Example
```dockerfile
ENV BRAINY_AUG_CACHE_TTL=600000
```
## Configuration Files
### File Locations (Priority Order)
1. `.brainyrc` (current directory)
2. `.brainyrc.json` (current directory)
3. `brainy.config.json` (current directory)
4. `~/.brainy/config.json` (user home)
5. `~/.brainyrc` (user home)
### File Format
```json
{
"augmentations": {
"wal": {
"enabled": true,
"immediateWrites": true,
"maxSize": 20971520,
"checkpointInterval": 300000
},
"cache": {
"enabled": true,
"maxSize": 2000,
"ttl": 600000
},
"metrics": {
"enabled": false
}
}
}
```
### Per-Environment Configuration
```json
{
"augmentations": {
"wal": {
"development": {
"enabled": true,
"immediateWrites": true,
"maxSize": 5242880
},
"production": {
"enabled": true,
"immediateWrites": false,
"maxSize": 104857600,
"checkpointInterval": 60000
}
}
}
}
```
## CLI Commands
### List Augmentations with Configuration
```bash
# Show all augmentations with config status
brainy augment list --detailed
# Show configuration for specific augmentation
brainy augment config wal
# Set configuration value
brainy augment config wal --set immediateWrites=true
# Show environment variable names
brainy augment config wal --env
# Export configuration schema
brainy augment schema wal > wal-schema.json
# Validate configuration file
brainy augment validate --file config.json
```
### Interactive Configuration
```bash
# Interactive configuration wizard
brainy augment configure wal
? Operation mode?
❯ Performance (immediate writes)
Durability (synchronous writes)
Custom
? Maximum log size? (10MB) 20MB
? Checkpoint interval? (1 minute) 5 minutes
Configuration saved to .brainyrc
```
## Tool Integration
### Brain-Cloud Explorer UI
```typescript
// Auto-generate configuration form from schema
const ConfigurationUI = ({ augmentationId }) => {
const [manifest, setManifest] = useState(null)
const [config, setConfig] = useState({})
useEffect(() => {
// Fetch manifest with schema
fetch(`/api/augmentations/${augmentationId}/manifest`)
.then(res => res.json())
.then(setManifest)
// Get current configuration
discovery.getConfig(augmentationId)
.then(setConfig)
}, [augmentationId])
const handleSave = async (newConfig) => {
// Validate configuration
const validation = await fetch(`/api/augmentations/${augmentationId}/validate`, {
method: 'POST',
body: JSON.stringify(newConfig)
}).then(res => res.json())
if (validation.valid) {
// Apply configuration
await discovery.updateConfig(augmentationId, newConfig)
}
}
// Render form based on schema
return <SchemaForm
schema={manifest?.configSchema}
values={config}
onSubmit={handleSave}
/>
}
```
### VS Code Extension
```json
// package.json contribution points
{
"contributes": {
"configuration": {
"title": "Brainy Augmentations",
"properties": {
"brainy.augmentations.wal.enabled": {
"type": "boolean",
"default": true,
},
"brainy.augmentations.wal.maxSize": {
"type": "number",
"default": 10485760,
}
}
}
}
}
```
## Migration Guide
### Migrating from BaseAugmentation
**Before:**
```typescript
export class MyAugmentation extends BaseAugmentation {
constructor(config: MyConfig = {}) {
super()
this.config = {
enabled: config.enabled ?? true,
threshold: config.threshold ?? 100
}
}
// No manifest
// No config discovery
// No runtime updates
}
```
**After:**
```typescript
export class MyAugmentation extends ConfigurableAugmentation {
constructor(config?: MyConfig) {
super(config) // Config resolution handled automatically
}
getManifest(): AugmentationManifest {
return {
id: 'my-augmentation',
name: 'My Augmentation',
version: '1.0.0',
description: 'Does something amazing',
category: 'performance',
configSchema: {
type: 'object',
properties: {
enabled: { type: 'boolean', default: true },
threshold: { type: 'number', default: 100 }
}
}
}
}
// Optional: Handle config changes
protected async onConfigChange(newConfig: MyConfig, oldConfig: MyConfig): Promise<void> {
// React to changes
}
}
```
### Backwards Compatibility
The system maintains full backwards compatibility:
1. **BaseAugmentation still works** - Existing augmentations continue to function
2. **Constructor config still works** - Existing configuration patterns preserved
3. **Zero-config still works** - Defaults are applied automatically
4. **Progressive enhancement** - Add features as needed
## Best Practices
### 1. Always Provide Defaults
```typescript
configSchema: {
properties: {
enabled: {
type: 'boolean',
default: true, // Always provide defaults
description: 'Enable this feature'
}
}
}
```
### 2. Use Descriptive Configuration Keys
```typescript
// Good
checkpointInterval: 60000
// Bad
ci: 60000
```
### 3. Validate Configuration
```typescript
protected async onConfigChange(newConfig: any, oldConfig: any): Promise<void> {
// Validate before applying
if (newConfig.maxSize < 1048576) {
throw new Error('maxSize must be at least 1MB')
}
// Apply changes
this.maxSize = newConfig.maxSize
}
```
### 4. Document Environment Variables
```typescript
/**
* Environment Variables:
* - BRAINY_AUG_MYAUG_ENABLED: Enable augmentation (boolean)
* - BRAINY_AUG_MYAUG_THRESHOLD: Processing threshold (number)
* - BRAINY_AUG_MYAUG_MODE: Processing mode (fast|balanced|thorough)
*/
```
### 5. Provide Configuration Examples
```typescript
configExamples: [
{
name: 'Production',
description: 'Optimized for production use',
config: {
enabled: true,
mode: 'thorough',
threshold: 500
}
},
{
name: 'Development',
description: 'Lightweight for development',
config: {
enabled: true,
mode: 'fast',
threshold: 10
}
}
]
```
## Troubleshooting
### Configuration Not Loading
1. Check file locations and names
2. Verify JSON syntax in config files
3. Check environment variable names (case-sensitive)
4. Use `brainy augment config <name> --debug` to see resolution
### Validation Errors
1. Check schema requirements
2. Verify data types match schema
3. Check minimum/maximum constraints
4. Use discovery API to validate before applying
### Runtime Updates Not Working
1. Ensure augmentation extends ConfigurableAugmentation
2. Implement onConfigChange if needed
3. Check for validation errors
4. Verify augmentation is initialized
## API Reference
See the [Discovery API Documentation](./discovery-api.md) for complete API details.
## Examples
See the [examples directory](../../examples/augmentation-config/) for complete working examples.

532
docs/augmentations/DEVELOPER-GUIDE.md
View file

@ -1,532 +0,0 @@
# 🛠️ Brainy Augmentation Developer Guide
> **How to create, test, and use augmentations in Brainy**
>
> **⚠️ Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
## Migration Guide
### What Changed?
1. **Metadata Structure**: All metadata now requires type fields (`noun` or `verb`)
2. **Property Rename**: `verb.type` → `verb.verb` for relationships
3. **Two-File Storage**: Vectors and metadata stored separately for performance
4. **Return Types**: Storage methods distinguish between internal (pure) and public (WithMetadata) returns
### Migration Checklist
- [ ] Update metadata creation to include required `noun` field
- [ ] Change `verb.type` to `verb.verb` in all relationship code
- [ ] Update storage adapter methods to follow internal/public pattern
- [ ] Ensure metadata access uses correct structure
### Quick Migration Example
```typescript
// ❌ v3.x
const verb = {
type: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
if (verb.type === 'relatedTo') { ... }
// ✅ Current
const verb = {
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
const metadata: VerbMetadata = {
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
if (verb.verb === 'relatedTo') { ... }
```
## Quick Start: Your First Augmentation
```typescript
import { BaseAugmentation, BrainyAugmentation, AugmentationContext } from '@soulcraft/brainy'
export class MyFirstAugmentation extends BaseAugmentation {
readonly name = 'my-first-augmentation'
readonly timing = 'after' as const // When to run: before | after | both
readonly operations = ['add'] as const // Which operations to hook
readonly priority = 10 // Execution order (lower = first)
protected async onInit(): Promise<void> {
// Initialize your augmentation
console.log('MyFirstAugmentation initialized!')
}
async execute<T = any>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'add') {
console.log('Noun added:', params.noun)
// Access metadata correctly
if (params.noun?.metadata) {
console.log('Noun type:', params.noun.metadata.noun) // Required field
}
// You can access the brain instance
const stats = await context?.brain.getStats()
console.log('Total nouns:', stats.totalNouns)
}
}
protected async onShutdown(): Promise<void> {
// Cleanup
console.log('MyFirstAugmentation shutting down')
}
}
```
## Using Your Augmentation
```typescript
import { Brainy } from '@soulcraft/brainy'
import { MyFirstAugmentation } from './my-first-augmentation'
const brain = new Brainy()
// Register before init()
brain.augmentations.register(new MyFirstAugmentation())
await brain.init()
// Now your augmentation runs automatically!
await brain.add('Hello World')
// Console: "Noun added: { id: '...', vector: [...], metadata: {} }"
```
---
## Augmentation Lifecycle
### 1. Registration Phase
```typescript
const aug = new MyAugmentation()
brain.augmentations.register(aug) // Before brain.init()!
```
### 2. Initialization Phase
```typescript
await brain.init() // Calls aug.initialize() internally
// Your onInit() method runs here
```
### 3. Execution Phase
```typescript
await brain.add('data') // Your execute() method runs
```
### 4. Shutdown Phase
```typescript
await brain.shutdown() // Your onShutdown() method runs
```
---
## Timing Options
### `before` - Modify Input
```typescript
class ValidationAugmentation extends BaseAugmentation {
readonly timing = 'before' as const
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Validate and/or modify params
if (!params.content) {
throw new Error('Content required')
}
// Return modified params
return { ...params, validated: true }
}
}
}
```
### `after` - React to Results
```typescript
class LoggingAugmentation extends BaseAugmentation {
readonly timing = 'after' as const
async execute<T>(operation: string, params: any): Promise<void> {
if (operation === 'search') {
console.log(`Search for "${params.query}" returned ${params.result.length} results`)
}
// Don't return anything - just observe
}
}
```
### `both` - Before AND After
```typescript
class TimingAugmentation extends BaseAugmentation {
readonly timing = 'both' as const
private startTime?: number
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
if (!this.startTime) {
// Before execution
this.startTime = Date.now()
} else {
// After execution
const duration = Date.now() - this.startTime
console.log(`${operation} took ${duration}ms`)
this.startTime = undefined
}
}
}
```
---
## Operation Hooks
### Core Operations You Can Hook
```typescript
readonly operations = [
'add', // Adding data
'update', // Updating data
'delete', // Deleting data
'get', // Retrieving data
'search', // Searching
'find', // Triple Intelligence queries
'relate', // Adding relationships
'unrelate', // Removing relationships
'clear', // Clearing data
'all' // Hook ALL operations
] as const
```
### Example: Multi-Operation Hook
```typescript
class AuditAugmentation extends BaseAugmentation {
readonly operations = ['add', 'update', 'delete'] as const
async execute<T>(operation: string, params: any): Promise<void> {
// Log all data modifications
await this.logToAuditTrail(operation, params)
}
}
```
---
## Accessing Brain Context
```typescript
class ContextAwareAugmentation extends BaseAugmentation {
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<void> {
// Access the brain instance
const brain = context?.brain
if (!brain) return
// Use any brain method
const stats = await brain.getStats()
const size = await brain.size()
const results = await brain.search('query')
// Access other augmentations
const cache = brain.augmentations.get('cache')
if (cache) {
await cache.clear()
}
}
}
```
---
## Real-World Examples
### 1. Backup Augmentation
```typescript
class BackupAugmentation extends BaseAugmentation {
readonly name = 'backup'
readonly timing = 'after' as const
readonly operations = ['add', 'update', 'delete'] as const
readonly priority = 5
private changes = 0
private readonly backupThreshold = 100
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
this.changes++
if (this.changes >= this.backupThreshold) {
await this.performBackup(context?.brain)
this.changes = 0
}
}
private async performBackup(brain?: any): Promise<void> {
if (!brain) return
// Cut an instant hard-link snapshot via the Db API
const snapshotPath = `/backups/auto-${Date.now()}`
const pin = brain.now()
try {
await pin.persist(snapshotPath)
} finally {
await pin.release()
}
console.log(`Automatic snapshot created: ${snapshotPath}`)
}
}
```
### 2. Rate Limiting Augmentation
```typescript
class RateLimitAugmentation extends BaseAugmentation {
readonly name = 'rate-limit'
readonly timing = 'before' as const
readonly operations = ['search', 'find'] as const
readonly priority = 100 // High priority - run first
private requests = new Map<string, number[]>()
private readonly limit = 100 // 100 requests
private readonly window = 60000 // per minute
async execute<T>(operation: string, params: any): Promise<void> {
const now = Date.now()
const key = params.userId || 'anonymous'
// Get request timestamps
const timestamps = this.requests.get(key) || []
// Remove old timestamps
const recent = timestamps.filter(t => now - t < this.window)
// Check limit
if (recent.length >= this.limit) {
throw new Error('Rate limit exceeded')
}
// Add current request
recent.push(now)
this.requests.set(key, recent)
}
}
```
### 3. Encryption Augmentation
```typescript
class EncryptionAugmentation extends BaseAugmentation {
readonly name = 'encryption'
readonly timing = 'both' as const
readonly operations = ['add', 'get'] as const
readonly priority = 90 // Run early
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Encrypt before storing
if (params.metadata?.sensitive) {
params.content = await this.encrypt(params.content)
params.encrypted = true
}
return params
}
if (operation === 'get' && params.result?.encrypted) {
// Decrypt after retrieval
params.result.content = await this.decrypt(params.result.content)
delete params.result.encrypted
return params.result
}
}
}
```
---
## Testing Your Augmentation
```typescript
import { describe, it, expect } from 'vitest'
import { Brainy } from '@soulcraft/brainy'
import { MyAugmentation } from './my-augmentation'
describe('MyAugmentation', () => {
it('should hook into addNoun', async () => {
const brain = new Brainy({ storage: 'memory' })
const aug = new MyAugmentation()
// Spy on the execute method
const executeSpy = vi.spyOn(aug, 'execute')
brain.augmentations.register(aug)
await brain.init()
// Trigger the augmentation
await brain.add('test data')
// Verify it was called
expect(executeSpy).toHaveBeenCalledWith(
'add',
expect.objectContaining({ content: 'test data' }),
expect.any(Object)
)
})
})
```
---
## Best Practices
### 1. Use Proper Timing
- `before`: Validation, modification, rate limiting
- `after`: Logging, metrics, side effects
- `both`: Timing, tracing, wrapping
### 2. Set Appropriate Priority
```typescript
// Priority guidelines
100: Critical (auth, rate limiting)
50: Important (validation, transformation)
10: Normal (logging, metrics)
1: Optional (debugging, tracing)
```
### 3. Handle Errors Gracefully
```typescript
async execute<T>(operation: string, params: any): Promise<void> {
try {
await this.riskyOperation()
} catch (error) {
// Log but don't break the main operation
console.error(`Augmentation error in ${this.name}:`, error)
// Optionally report to monitoring
this.reportError(error)
}
}
```
### 4. Be Performance Conscious
```typescript
class CachedAugmentation extends BaseAugmentation {
private cache = new Map<string, any>()
async execute<T>(operation: string, params: any): Promise<any> {
const key = this.getCacheKey(params)
// Check cache first
if (this.cache.has(key)) {
return this.cache.get(key)
}
// Expensive operation
const result = await this.expensiveOperation(params)
this.cache.set(key, result)
return result
}
}
```
### 5. Clean Up Resources
```typescript
protected async onShutdown(): Promise<void> {
// Close connections
await this.connection?.close()
// Clear intervals
clearInterval(this.interval)
// Flush buffers
await this.flush()
// Clear caches
this.cache.clear()
}
```
---
## Publishing Your Augmentation (Future)
### Package Structure
```
my-augmentation/
├── src/
│ └── index.ts # Your augmentation
├── dist/ # Built output
├── tests/
│ └── augmentation.test.ts
├── package.json
├── tsconfig.json
└── README.md
```
### package.json
```json
{
"name": "@mycompany/brainy-custom-augmentation",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"keywords": ["brainy-augmentation"],
"peerDependencies": {
"@soulcraft/brainy": ">=2.0.0"
},
"brainy": {
"type": "augmentation",
"class": "CustomAugmentation",
"timing": "after",
"operations": ["add"],
"priority": 10
}
}
```
### Future: Brain Cloud Registry
```bash
# Coming in 2.1+
npm run build
npm test
brainy publish # Publishes to brain-cloud registry
```
---
## FAQ
### Q: Can I modify the operation result?
**A**: Yes, if `timing: 'before'`, return modified params. If `timing: 'after'`, you can see but not modify results.
### Q: Can augmentations communicate?
**A**: Yes, through the context: `context.brain.augmentations.get('other-augmentation')`
### Q: What if my augmentation fails?
**A**: Handle errors internally. Don't break the main operation unless critical.
### Q: Can I use async operations?
**A**: Yes, everything is async-friendly.
### Q: How do I access storage directly?
**A**: Through context: `context.brain.storage` (but prefer using brain methods)
---
## Get Help
- **GitHub**: [github.com/soulcraft/brainy](https://github.com/soulcraft/brainy)
- **Discord**: [discord.gg/brainy](https://discord.gg/brainy)
- **Examples**: See `/examples/augmentations/` in the repo
---
*Start building your augmentation today! The marketplace is coming in 2.1 🚀*

669
docs/augmentations/EXAMPLES.md
View file

@ -1,669 +0,0 @@
# Augmentation Examples - Import, Store, Export
This guide shows two complete workflows:
1. **Simple Handler** - Just import a new file type
2. **Full Augmentation** - Import + Store + Export (premium-ready)
---
## Workflow 1: Simple Handler (Import Only)
**Use case:** You want to import a new file type (e.g., CAD files) into Brainy's knowledge graph.
### Step 1: Create the Handler
```typescript
// src/handlers/CADHandler.ts
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
import type { FormatHandlerOptions, ProcessedData } from '@soulcraft/brainy/augmentations/intelligentImport'
import { parseCAD } from 'cad-parser' // Your parsing library
export class CADHandler extends BaseFormatHandler {
readonly format = 'cad'
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
if (typeof data === 'object' && 'filename' in data) {
const mimeType = this.getMimeType(data)
return this.mimeTypeMatches(mimeType, ['image/vnd.dwg', 'image/vnd.dxf'])
}
return false
}
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
// Parse CAD file
const cad = await parseCAD(buffer)
// Extract entities for knowledge graph
const entities = []
// Document entity
entities.push({
type: 'CADDocument',
filename: options.filename,
units: cad.units,
bounds: cad.bounds,
version: cad.version
})
// Layer entities
for (const layer of cad.layers) {
entities.push({
type: 'CADLayer',
name: layer.name,
color: layer.color,
visible: layer.visible
})
}
// Object entities
for (const obj of cad.objects) {
entities.push({
type: 'CADObject',
objectType: obj.type,
layer: obj.layer,
geometry: obj.geometry
})
}
return {
format: 'cad',
data: entities,
metadata: {
rowCount: entities.length,
fields: ['type', 'name', 'geometry'],
processingTime: Date.now() - startTime,
layerCount: cad.layers.length,
objectCount: cad.objects.length
},
filename: options.filename
}
}
}
```
### Step 2: Register the Handler
```typescript
// src/index.ts
import { globalHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
import { CADHandler } from './handlers/CADHandler.js'
// Register handler globally
globalHandlerRegistry.registerHandler({
name: 'cad',
mimeTypes: ['image/vnd.dwg', 'image/vnd.dxf'],
extensions: ['.dwg', '.dxf', '.dwf'],
loader: async () => new CADHandler()
})
```
### Step 3: Use It
```typescript
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy()
await brain.init()
// Import CAD file - automatically routed to CADHandler
const result = await brain.import({
type: 'file',
data: cadFileBuffer,
filename: 'floor-plan.dwg'
})
console.log(`Imported ${result.entities.length} CAD entities`)
// Query the imported data
const layers = await brain.find({ type: 'CADLayer' })
const objects = await brain.find({ type: 'CADObject', layer: 'WALLS' })
```
**That's it!** Simple handlers just import data. Brainy handles storage automatically.
---
## Workflow 2: Full Augmentation (Import + Store + Export)
**Use case:** You want a complete solution that imports project files, stores them with special logic, and exports results (e.g., React project analyzer).
### Step 1: Create the Augmentation
```typescript
// @yourcompany/brainy-react-analyzer
import { BaseAugmentation, type AugmentationContext } from '@soulcraft/brainy'
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
import type { ProcessedData } from '@soulcraft/brainy/augmentations/intelligentImport'
import { parse } from '@babel/parser'
import traverse from '@babel/traverse'
import * as t from '@babel/types'
/**
* React Project Analyzer Augmentation
*
* Features:
* - Import: Parse React components, extract props, hooks, imports
* - Store: Create relationships between components
* - Export: Generate component diagram, dependency graph
*/
export class ReactAnalyzerAugmentation extends BaseAugmentation {
readonly name = 'react-analyzer'
readonly timing = 'before' as const
readonly operations = ['import', 'export'] as any[]
readonly priority = 75
private handler: ReactComponentHandler
constructor(config = {}) {
super(config)
this.handler = new ReactComponentHandler()
}
async execute<T = any>(
operation: string,
params: any,
next: () => Promise<T>
): Promise<T> {
// IMPORT: Parse React files
if (operation === 'import' && this.isReactFile(params)) {
return this.handleImport(params, next)
}
// EXPORT: Generate diagrams/reports
if (operation === 'export' && params.format === 'react-diagram') {
return this.handleExport(params, next)
}
return next()
}
private isReactFile(params: any): boolean {
const filename = params.filename || ''
return (
(filename.endsWith('.tsx') || filename.endsWith('.jsx')) &&
params.data?.includes('React')
)
}
private async handleImport<T>(params: any, next: () => Promise<T>): Promise<T> {
// Parse React component
const processed = await this.handler.process(params.data, params.options)
// Enrich with relationships
params.data = processed.data
params.metadata = {
...params.metadata,
reactAnalysis: processed.metadata
}
// Continue to next augmentation/storage
const result = await next()
// Post-process: Create component relationships
await this.createComponentRelationships(processed, result)
return result
}
private async createComponentRelationships(
processed: ProcessedData,
result: any
): Promise<void> {
const brain = this.getBrain()
if (!brain) return
// Find the component entity that was created
const component = processed.data.find(d => d.type === 'ReactComponent')
if (!component) return
// Create relationships for imports
for (const imp of component.imports || []) {
// Find or create imported component
const imported = await brain.findOne({
type: 'ReactComponent',
name: imp.name
})
if (imported) {
// Create "Imports" relationship
await brain.createRelation({
source: result.entities[0].id,
verb: 'Imports',
target: imported.id,
metadata: {
importPath: imp.path,
importType: imp.type
}
})
}
}
// Create relationships for prop types
for (const prop of component.props || []) {
if (prop.typeRef) {
const typeEntity = await brain.findOne({
type: 'TypeDefinition',
name: prop.typeRef
})
if (typeEntity) {
await brain.createRelation({
source: result.entities[0].id,
verb: 'UsesPropType',
target: typeEntity.id
})
}
}
}
}
private async handleExport<T>(params: any, next: () => Promise<T>): Promise<T> {
const brain = this.getBrain()
if (!brain) return next()
// Query all React components
const components = await brain.find({ type: 'ReactComponent' })
// Build dependency graph
const graph = await this.buildDependencyGraph(components)
// Generate diagram
const diagram = this.generateMermaidDiagram(graph)
return {
format: 'react-diagram',
diagram,
components: components.length,
dependencies: graph.edges.length
} as T
}
private async buildDependencyGraph(components: any[]): Promise<any> {
const brain = this.getBrain()
const nodes = components.map(c => ({
id: c.id,
name: c.name,
props: c.props
}))
const edges = []
for (const component of components) {
const imports = await brain.getRelated(component.id, 'Imports')
for (const imp of imports) {
edges.push({
from: component.id,
to: imp.id,
type: 'imports'
})
}
}
return { nodes, edges }
}
private generateMermaidDiagram(graph: any): string {
let mermaid = 'graph TD\n'
for (const node of graph.nodes) {
mermaid += ` ${node.id}[${node.name}]\n`
}
for (const edge of graph.edges) {
mermaid += ` ${edge.from} --> ${edge.to}\n`
}
return mermaid
}
}
/**
* React Component Handler
*/
class ReactComponentHandler extends BaseFormatHandler {
readonly format = 'react'
canHandle(data: any): boolean {
if (typeof data === 'object' && 'filename' in data) {
return data.filename?.match(/\.(tsx|jsx)$/) !== null
}
return false
}
async process(data: Buffer | string, options: any): Promise<ProcessedData> {
const code = Buffer.isBuffer(data) ? data.toString('utf-8') : data
// Parse with Babel
const ast = parse(code, {
sourceType: 'module',
plugins: ['jsx', 'typescript']
})
// Extract component info
const components: any[] = []
const imports: any[] = []
const exports: any[] = []
traverse(ast, {
// Detect function components
FunctionDeclaration(path) {
if (this.isReactComponent(path.node)) {
components.push({
type: 'ReactComponent',
name: path.node.id?.name,
componentType: 'function',
props: this.extractProps(path),
hooks: this.extractHooks(path),
state: this.extractState(path)
})
}
},
// Detect class components
ClassDeclaration(path) {
if (this.isReactClassComponent(path.node)) {
components.push({
type: 'ReactComponent',
name: path.node.id.name,
componentType: 'class',
props: this.extractClassProps(path),
state: this.extractClassState(path),
lifecycle: this.extractLifecycleMethods(path)
})
}
},
// Extract imports
ImportDeclaration(path) {
imports.push({
type: 'Import',
from: path.node.source.value,
imports: path.node.specifiers.map(s => ({
name: s.local.name,
imported: t.isImportSpecifier(s) ? s.imported.name : null
}))
})
},
// Extract exports
ExportNamedDeclaration(path) {
exports.push({
type: 'Export',
name: path.node.declaration?.id?.name
})
}
})
// Enrich components with import info
for (const component of components) {
component.imports = imports
component.exports = exports.find(e => e.name === component.name)
}
return {
format: 'react',
data: components,
metadata: {
rowCount: components.length,
fields: ['type', 'name', 'props', 'hooks'],
processingTime: Date.now() - startTime,
componentCount: components.length,
importCount: imports.length,
exportCount: exports.length
},
filename: options.filename
}
}
private isReactComponent(node: any): boolean {
// Check if function returns JSX
return node.body?.body?.some(stmt =>
t.isReturnStatement(stmt) && this.isJSX(stmt.argument)
)
}
private isJSX(node: any): boolean {
return t.isJSXElement(node) || t.isJSXFragment(node)
}
private extractProps(path: any): any[] {
const params = path.node.params
if (params.length === 0) return []
const propsParam = params[0]
if (t.isObjectPattern(propsParam)) {
return propsParam.properties.map(p => ({
name: p.key.name,
type: p.typeAnnotation?.typeAnnotation?.type
}))
}
return []
}
private extractHooks(path: any): string[] {
const hooks: string[] = []
path.traverse({
CallExpression(hookPath) {
const callee = hookPath.node.callee
if (t.isIdentifier(callee) && callee.name.startsWith('use')) {
hooks.push(callee.name)
}
}
})
return hooks
}
private extractState(path: any): any[] {
const stateVars: any[] = []
path.traverse({
CallExpression(hookPath) {
if (
t.isIdentifier(hookPath.node.callee) &&
hookPath.node.callee.name === 'useState'
) {
const parent = hookPath.parent
if (t.isVariableDeclarator(parent) && t.isArrayPattern(parent.id)) {
const [stateVar] = parent.id.elements
if (t.isIdentifier(stateVar)) {
stateVars.push({
name: stateVar.name,
initialValue: hookPath.node.arguments[0]
})
}
}
}
}
})
return stateVars
}
}
```
### Step 2: Package as NPM Module
```json
// package.json
{
"name": "@yourcompany/brainy-react-analyzer",
"version": "1.0.0",
"description": "React project analyzer for Brainy",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
".": {
"import": "./dist/index.js",
"require": "./dist/index.cjs"
}
},
"keywords": ["brainy", "react", "analyzer", "augmentation"],
"peerDependencies": {
"@soulcraft/brainy": "^5.2.0"
},
"dependencies": {
"@babel/parser": "^7.23.0",
"@babel/traverse": "^7.23.0",
"@babel/types": "^7.23.0"
}
}
```
### Step 3: Use the Augmentation
```typescript
// Install
// npm install @yourcompany/brainy-react-analyzer
import { Brainy } from '@soulcraft/brainy'
import { ReactAnalyzerAugmentation } from '@yourcompany/brainy-react-analyzer'
const brain = new Brainy()
// Add augmentation
brain.addAugmentation(new ReactAnalyzerAugmentation())
await brain.init()
// Import React project
await brain.import({
type: 'directory',
path: '/path/to/react-project/src',
recursive: true
})
// Query components
const components = await brain.find({ type: 'ReactComponent' })
console.log(`Found ${components.length} React components`)
// Find component dependencies
const appComponent = await brain.findOne({ type: 'ReactComponent', name: 'App' })
const imports = await brain.getRelated(appComponent.id, 'Imports')
console.log(`App component imports:`, imports.map(c => c.name))
// Export diagram
const diagram = await brain.export({
format: 'react-diagram'
})
console.log(diagram.diagram) // Mermaid diagram
```
### Step 4: Premium Licensing (Optional)
```typescript
// Add license checking
export class ReactAnalyzerAugmentation extends BaseAugmentation {
private licenseKey?: string
constructor(config: { licenseKey?: string } = {}) {
super(config)
this.licenseKey = config.licenseKey
}
async onInitialize(): Promise<void> {
if (!this.licenseKey) {
throw new Error('React Analyzer requires a license key. Get one at https://yourcompany.com/brainy-react')
}
// Verify license
const valid = await this.verifyLicense(this.licenseKey)
if (!valid) {
throw new Error('Invalid license key')
}
this.log('React Analyzer initialized successfully')
}
private async verifyLicense(key: string): Promise<boolean> {
// Check with your license server
const response = await fetch('https://api.yourcompany.com/verify', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ key, product: 'brainy-react-analyzer' })
})
const data = await response.json()
return data.valid
}
}
// Usage with license
brain.addAugmentation(new ReactAnalyzerAugmentation({
licenseKey: 'YOUR-LICENSE-KEY'
}))
```
---
## Comparison: Handler vs Augmentation
| Feature | Simple Handler | Full Augmentation |
|---------|---------------|-------------------|
| **Import** | ✅ Yes (automatic) | ✅ Yes (with custom logic) |
| **Storage** | ✅ Automatic (Brainy core) | ✅ Custom logic + relationships |
| **Export** | ❌ No | ✅ Yes (custom formats) |
| **Relationships** | ❌ No | ✅ Yes (create custom relationships) |
| **Premium licensing** | ❌ Difficult | ✅ Easy (augmentation-level) |
| **Custom operations** | ❌ Import only | ✅ Any operation |
| **Complexity** | Low (50-100 lines) | Medium (200-500 lines) |
### When to use Handler:
- Just need to import a new file type
- Don't need custom export
- Don't need special relationships
- Simple use case
### When to use Augmentation:
- Need import + export workflow
- Need custom relationship logic
- Want premium licensing capability
- Complex business logic
- Multiple operations (import + export + query)
---
## More Examples
### Example: Python Project Analyzer
```typescript
class PythonAnalyzerAugmentation extends BaseAugmentation {
// Import Python files, extract classes/functions
// Create relationships between modules
// Export: Dependency diagram, call graph
}
```
### Example: Database Schema Sync
```typescript
class DatabaseSyncAugmentation extends BaseAugmentation {
// Import: Parse SQL schema
// Store: Tables, columns, relationships
// Export: Generate migration scripts
}
```
### Example: API Documentation Generator
```typescript
class APIDocAugmentation extends BaseAugmentation {
// Import: Parse TypeScript types
// Store: Endpoints, parameters, responses
// Export: OpenAPI spec, Markdown docs
}
```
---
## See Also
- [FORMAT_HANDLERS.md](./FORMAT_HANDLERS.md) - Creating format handlers
- [AUGMENTATIONS.md](./AUGMENTATIONS.md) - Augmentation system details
- [ImageHandler source](../../src/augmentations/intelligentImport/handlers/imageHandler.ts) - Handler example
- [IntelligentImportAugmentation source](../../src/augmentations/intelligentImport/IntelligentImportAugmentation.ts) - Augmentation example

687
docs/augmentations/FORMAT_HANDLERS.md
View file

@ -1,687 +0,0 @@
# Creating Custom Format Handlers
**Version:** 5.2.0+
Format handlers enable you to import ANY file type into Brainy as structured knowledge graph data. This guide shows how to create custom format handlers for your specific file formats.
---
## Quick Start
```typescript
import { BaseFormatHandler, globalHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
import type { FormatHandlerOptions, ProcessedData } from '@soulcraft/brainy/augmentations/intelligentImport'
class MyFormatHandler extends BaseFormatHandler {
readonly format = 'myformat'
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
// Option 1: Check by MIME type
if (typeof data === 'object' && 'filename' in data) {
const mimeType = this.getMimeType(data)
return this.mimeTypeMatches(mimeType, ['application/x-myformat'])
}
// Option 2: Check by magic bytes
if (Buffer.isBuffer(data)) {
return data[0] === 0x4D && data[1] === 0x59 // "MY" magic bytes
}
return false
}
async process(
data: Buffer | string,
options: FormatHandlerOptions
): Promise<ProcessedData> {
// Convert to Buffer
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
// Parse your format
const parsed = this.parseMyFormat(buffer)
// Return structured data
return {
format: 'myformat',
data: [
{
type: 'MyEntity',
name: parsed.name,
metadata: parsed.metadata
}
],
metadata: {
rowCount: 1,
fields: ['type', 'name', 'metadata'],
processingTime: Date.now() - startTime
}
}
}
private parseMyFormat(buffer: Buffer): any {
// Your parsing logic here
return { name: 'example', metadata: {} }
}
}
// Register globally
globalHandlerRegistry.registerHandler({
name: 'myformat',
mimeTypes: ['application/x-myformat'],
extensions: ['.myf', '.myfmt'],
loader: async () => new MyFormatHandler()
})
```
Now Brainy automatically handles your format:
```typescript
await brain.import({
type: 'file',
data: myFormatBuffer,
filename: 'data.myf'
})
// Automatically routes to MyFormatHandler!
```
---
## BaseFormatHandler
All format handlers should extend `BaseFormatHandler`, which provides:
### MIME Type Detection
```typescript
protected getMimeType(data: Buffer | string | { filename?: string }): string
```
Detects MIME type from filename or buffer. Uses Brainy's comprehensive MIME detection (2000+ types).
**Example:**
```typescript
const mimeType = this.getMimeType({ filename: 'data.dwg' })
// Returns: 'image/vnd.dwg'
```
### MIME Type Matching
```typescript
protected mimeTypeMatches(mimeType: string, patterns: string[]): boolean
```
Checks if MIME type matches patterns. Supports wildcards (`image/*`).
**Example:**
```typescript
if (this.mimeTypeMatches(mimeType, ['image/*', 'video/*'])) {
// Handle all images and videos
}
```
### Extension Detection
```typescript
protected detectExtension(data: string | Buffer | { filename?: string; ext?: string }): string | null
```
Extracts file extension for fallback detection.
---
## canHandle() Method
The `canHandle()` method determines if your handler can process the given data.
### Strategy 1: MIME Type Detection (Recommended)
```typescript
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
if (typeof data === 'object' && 'filename' in data) {
const mimeType = this.getMimeType(data)
return this.mimeTypeMatches(mimeType, [
'application/x-myformat',
'application/myformat'
])
}
return false
}
```
✅ **Pros:** Automatic, comprehensive, works with 2000+ types
❌ **Cons:** Requires filename
### Strategy 2: Magic Byte Detection
```typescript
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
if (Buffer.isBuffer(data)) {
// Check magic bytes
return (
data[0] === 0x50 && // 'P'
data[1] === 0x4B && // 'K'
data[2] === 0x03 &&
data[3] === 0x04 // ZIP signature
)
}
return false
}
```
✅ **Pros:** Works without filename, robust
❌ **Cons:** Requires knowledge of format structure
### Strategy 3: Combined Approach
```typescript
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
// Try MIME type first
if (typeof data === 'object' && 'filename' in data) {
const mimeType = this.getMimeType(data)
if (this.mimeTypeMatches(mimeType, ['application/x-myformat'])) {
return true
}
}
// Fallback to magic bytes
if (Buffer.isBuffer(data)) {
return this.checkMagicBytes(data)
}
return false
}
```
✅ **Pros:** Robust, works in all scenarios
❌ **Cons:** More complex
---
## process() Method
The `process()` method extracts structured data from the file.
### Return Format
```typescript
interface ProcessedData {
/** Format identifier */
format: string
/** Array of extracted entities */
data: Array<Record<string, any>>
/** Metadata about processing */
metadata: {
rowCount: number
fields: string[]
processingTime: number
[key: string]: any
}
/** Original filename (optional) */
filename?: string
}
```
### Example: CAD File Handler
```typescript
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
const startTime = Date.now()
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
// Parse CAD file
const cad = await this.parseCAD(buffer)
// Extract entities
const entities = []
// Main CAD document
entities.push({
type: 'CADDocument',
filename: options.filename,
units: cad.units,
bounds: cad.bounds
})
// Layers
for (const layer of cad.layers) {
entities.push({
type: 'CADLayer',
name: layer.name,
color: layer.color,
visible: layer.visible,
objectCount: layer.objects.length
})
}
// Objects
for (const obj of cad.objects) {
entities.push({
type: 'CADObject',
objectType: obj.type,
layer: obj.layer,
geometry: obj.geometry,
properties: obj.properties
})
}
return {
format: 'cad',
data: entities,
metadata: {
rowCount: entities.length,
fields: ['type', 'name', 'geometry', 'properties'],
processingTime: Date.now() - startTime,
layerCount: cad.layers.length,
objectCount: cad.objects.length,
units: cad.units
},
filename: options.filename
}
}
```
### Best Practices
1. **Always track processing time:**
```typescript
const startTime = Date.now()
// ... processing ...
metadata.processingTime = Date.now() - startTime
```
2. **Include rich metadata:**
```typescript
metadata: {
rowCount: entities.length,
fields: ['type', 'name', ...],
processingTime: 123,
// Format-specific metadata
layerCount: 5,
objectCount: 150,
version: '2.0'
}
```
3. **Handle errors gracefully:**
```typescript
try {
const parsed = this.parse(buffer)
return { format: 'myformat', data: parsed, ... }
} catch (error) {
throw new Error(
`Failed to parse myformat: ${error instanceof Error ? error.message : String(error)}`
)
}
```
4. **Support progress reporting (optional):**
```typescript
if (options.progressHooks?.onCurrentItem) {
options.progressHooks.onCurrentItem(`Processing layer ${i}/${total}`)
}
```
---
## FormatHandlerRegistry
### Global Registry
Use the global registry for application-wide handlers:
```typescript
import { globalHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
globalHandlerRegistry.registerHandler({
name: 'myformat',
mimeTypes: ['application/x-myformat'],
extensions: ['.myf'],
loader: async () => new MyFormatHandler()
})
```
### Local Registry
Create a local registry for scoped handlers:
```typescript
import { FormatHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
const registry = new FormatHandlerRegistry()
registry.registerHandler({ ... })
```
### Lazy Loading
Handlers are lazy-loaded for performance:
```typescript
globalHandlerRegistry.registerHandler({
name: 'heavy',
mimeTypes: ['application/x-heavy'],
extensions: ['.heavy'],
loader: async () => {
// Only loaded when first needed
const { HeavyHandler } = await import('./HeavyHandler.js')
return new HeavyHandler()
}
})
```
### Getting Handlers
```typescript
// By filename (automatic MIME detection)
const handler = await registry.getHandler('data.myf')
// By MIME type
const handler = await registry.getHandlerByMimeType('application/x-myformat')
// By extension
const handler = await registry.getHandlerByExtension('.myf')
// By name
const handler = await registry.getHandlerByName('myformat')
```
---
## Real-World Examples
### Example 1: Video Metadata Extractor
```typescript
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
import ffmpeg from 'fluent-ffmpeg'
class VideoHandler extends BaseFormatHandler {
readonly format = 'video'
canHandle(data) {
const mimeType = this.getMimeType(data)
return this.mimeTypeMatches(mimeType, ['video/*'])
}
async process(data, options) {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
// Extract video metadata with ffmpeg
const metadata = await this.extractVideoMetadata(buffer)
return {
format: 'video',
data: [{
type: 'Video',
duration: metadata.duration,
codec: metadata.codec,
resolution: metadata.resolution,
frameRate: metadata.frameRate,
bitrate: metadata.bitrate,
audioTracks: metadata.audioTracks,
subtitles: metadata.subtitles
}],
metadata: {
rowCount: 1,
fields: ['type', 'duration', 'codec', 'resolution'],
processingTime: metadata.processingTime
}
}
}
private async extractVideoMetadata(buffer: Buffer) {
// Use ffmpeg to extract metadata
return new Promise((resolve, reject) => {
ffmpeg(buffer)
.ffprobe((err, data) => {
if (err) reject(err)
else resolve(this.parseFFmpegOutput(data))
})
})
}
}
```
### Example 2: Git Repository Parser
```typescript
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
import { simpleGit } from 'simple-git'
class GitRepoHandler extends BaseFormatHandler {
readonly format = 'git-repo'
canHandle(data) {
// Check for .git directory
if (typeof data === 'object' && 'filename' in data) {
return data.filename?.includes('.git') || false
}
return false
}
async process(data, options) {
const repoPath = options.filename || ''
const git = simpleGit(repoPath)
// Extract commits
const log = await git.log()
const commits = log.all.map(commit => ({
type: 'GitCommit',
hash: commit.hash,
message: commit.message,
author: commit.author_name,
date: commit.date
}))
// Extract branches
const branchSummary = await git.branchLocal()
const branches = Object.keys(branchSummary.branches).map(name => ({
type: 'GitBranch',
name,
current: branchSummary.current === name
}))
return {
format: 'git-repo',
data: [...commits, ...branches],
metadata: {
rowCount: commits.length + branches.length,
fields: ['type', 'hash', 'message', 'author'],
processingTime: Date.now() - startTime,
commitCount: commits.length,
branchCount: branches.length
}
}
}
}
```
### Example 3: Database Schema Importer
```typescript
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
import { Client } from 'pg'
class PostgreSQLSchemaHandler extends BaseFormatHandler {
readonly format = 'postgres-schema'
canHandle(data) {
if (typeof data === 'object' && 'filename' in data) {
return data.filename?.endsWith('.sql') || false
}
return false
}
async process(data, options) {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
const sql = buffer.toString('utf-8')
// Parse SQL or connect to database
const schema = await this.parseSchema(sql)
const entities = []
// Tables
for (const table of schema.tables) {
entities.push({
type: 'Table',
name: table.name,
schema: table.schema,
columnCount: table.columns.length
})
// Columns
for (const column of table.columns) {
entities.push({
type: 'Column',
name: column.name,
table: table.name,
dataType: column.dataType,
nullable: column.nullable,
primaryKey: column.primaryKey
})
}
}
// Foreign keys
for (const fk of schema.foreignKeys) {
entities.push({
type: 'ForeignKey',
from: `${fk.fromTable}.${fk.fromColumn}`,
to: `${fk.toTable}.${fk.toColumn}`
})
}
return {
format: 'postgres-schema',
data: entities,
metadata: {
rowCount: entities.length,
fields: ['type', 'name', 'table', 'dataType'],
processingTime: Date.now() - startTime,
tableCount: schema.tables.length,
columnCount: schema.tables.reduce((sum, t) => sum + t.columns.length, 0),
foreignKeyCount: schema.foreignKeys.length
}
}
}
}
```
---
## Creating Premium Augmentations
Package your handler as a premium augmentation:
```typescript
// @yourcompany/brainy-cad-importer
import { BaseAugmentation } from '@soulcraft/brainy'
import { CADHandler } from './CADHandler.js'
export class CADImportAugmentation extends BaseAugmentation {
readonly name = 'cad-import'
readonly timing = 'before'
readonly operations = ['import', 'importFile']
private handler: CADHandler
constructor(config = {}) {
super(config)
this.handler = new CADHandler()
}
async execute(operation, params, next) {
// Check if this is a CAD file
if (this.isCADFile(params)) {
const processed = await this.handler.process(params.data, params.options)
params.data = processed.data
params.metadata = { ...params.metadata, ...processed.metadata }
}
return next()
}
private isCADFile(params: any): boolean {
return this.handler.canHandle(params.data || params)
}
}
// Usage:
// npm install @yourcompany/brainy-cad-importer
// brain.addAugmentation(new CADImportAugmentation())
```
---
## Testing
```typescript
import { describe, it, expect } from 'vitest'
import { MyFormatHandler } from './MyFormatHandler.js'
describe('MyFormatHandler', () => {
let handler: MyFormatHandler
beforeEach(() => {
handler = new MyFormatHandler()
})
describe('canHandle', () => {
it('should handle .myf files', () => {
expect(handler.canHandle({ filename: 'data.myf' })).toBe(true)
})
it('should handle by MIME type', () => {
expect(handler.canHandle({ filename: 'data.myformat' })).toBe(true)
})
it('should reject non-myformat files', () => {
expect(handler.canHandle({ filename: 'data.txt' })).toBe(false)
})
})
describe('process', () => {
it('should extract structured data', async () => {
const testData = Buffer.from('MY format data')
const result = await handler.process(testData)
expect(result.format).toBe('myformat')
expect(result.data).toHaveLength(1)
expect(result.metadata.processingTime).toBeGreaterThan(0)
})
it('should handle errors gracefully', async () => {
const invalidData = Buffer.from('invalid')
await expect(handler.process(invalidData)).rejects.toThrow()
})
})
})
```
---
## Best Practices
1. **Always extend BaseFormatHandler** - provides MIME detection and utilities
2. **Use MIME types for routing** - automatic, comprehensive, maintainable
3. **Lazy load heavy dependencies** - better performance
4. **Extract rich metadata** - make data queryable in knowledge graph
5. **Handle errors gracefully** - fail fast with clear error messages
6. **Test thoroughly** - test canHandle() and process() with real data
7. **Document your format** - explain what data is extracted and how
8. **Follow ProcessedData format** - ensures compatibility with Brainy
---
## See Also
- [ImageHandler source](../../src/augmentations/intelligentImport/handlers/imageHandler.ts) - Reference implementation
- [BaseFormatHandler source](../../src/augmentations/intelligentImport/handlers/base.ts) - Base class
- [FormatHandlerRegistry source](../../src/augmentations/intelligentImport/FormatHandlerRegistry.ts) - Registry implementation
- [Augmentations Guide](./AUGMENTATIONS.md) - Creating augmentations

204
docs/augmentations/README.md
View file

@ -1,204 +0,0 @@
# Brainy Augmentations
Augmentations are the core extensibility mechanism in Brainy. They allow you to modify, enhance, and extend Brainy's behavior without changing the core code.
## Core Principle: One Interface, Infinite Possibilities
Every augmentation implements the same simple `BrainyAugmentation` interface:
```typescript
interface BrainyAugmentation {
name: string
timing: 'before' | 'after' | 'around' | 'replace'
operations: string[]
priority: number
initialize(context): Promise<void>
execute(operation, params, next): Promise<any>
}
```
This single interface can handle EVERYTHING - from adding AI capabilities to exposing APIs to replacing storage backends.
## Available Augmentations
### 🧠 Data Processing
Augmentations that enhance how data is processed and stored.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **NeuralImportAugmentation** | AI-powered entity and relationship extraction | `before` | ✅ Production |
| **EntityRegistryAugmentation** | High-performance entity deduplication | `before` | ✅ Production |
| **BatchProcessingAugmentation** | Optimizes bulk operations | `around` | ✅ Production |
| **IntelligentVerbScoringAugmentation** | Learns relationship importance over time | `after` | ✅ Production |
### 🔌 External Connections (Synapses)
Connect Brainy to external services and data sources.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **NotionSynapse** | Sync with Notion databases | `after` | 📝 Example |
| **SalesforceSynapse** | Connect to Salesforce CRM | `after` | 📝 Example |
| **SlackSynapse** | Import Slack conversations | `after` | 📝 Example |
| **GoogleDriveSynapse** | Sync Google Drive documents | `after` | 📝 Example |
### 🌐 API Exposure
Expose Brainy through various protocols.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **APIServerAugmentation** | REST, WebSocket, and MCP server | `after` | ✅ Production |
| **GraphQLAugmentation** | GraphQL API endpoint | `after` | 🚧 Planned |
| **gRPCAugmentation** | gRPC service | `after` | 🚧 Planned |
### 💾 Storage Backends
Replace or enhance the storage layer.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **RedisAugmentation** | Redis caching layer | `around` | 📝 Example |
| **PostgresAugmentation** | PostgreSQL persistence | `replace` | 📝 Example |
> Brainy 8.0 ships `filesystem` and `memory` adapters out of the box. For off-site backup of the filesystem artifact, snapshot `rootDirectory` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, `tar`).
### 🔄 Real-time & Sync
Handle real-time updates and synchronization.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **WebSocketConduitAugmentation** | WebSocket client connections | `after` | ⚠️ Legacy |
| **ServerSearchAugmentation** | Connect to remote Brainy servers | `after` | ⚠️ Legacy |
| **TeamCoordinationAugmentation** | Multi-agent synchronization | `after` | 📝 Example |
### 🛡️ Infrastructure
Core infrastructure and reliability features.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **RequestDeduplicatorAugmentation** | Prevent duplicate concurrent requests | `before` | ✅ Production |
| **TransactionAugmentation** | ACID transaction support | `around` | 🚧 Planned |
| **CacheAugmentation** | Multi-level caching | `around` | ✅ Production |
### 📊 Monitoring & Analytics
Track and analyze Brainy's behavior.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **MetricsAugmentation** | Prometheus metrics | `after` | 📝 Example |
| **LoggingAugmentation** | Structured logging | `after` | 📝 Example |
| **TracingAugmentation** | Distributed tracing | `around` | 🚧 Planned |
### 🤖 AI & Chat
AI-powered interfaces and chat capabilities.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **ChatInterfaceAugmentation** | Natural language interface | `before` | 📝 Example |
| **MCPAgentMemoryAugmentation** | AI agent memory via MCP | `after` | 📝 Example |
| **LLMQueryAugmentation** | LLM-enhanced queries | `before` | 📝 Example |
### 📈 Visualization
Visual representations of data.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **GraphVisualizationAugmentation** | Real-time graph visualization | `after` | 📝 Example |
| **DashboardAugmentation** | Web-based dashboard | `after` | 🚧 Planned |
## Status Legend
- ✅ **Production**: Fully implemented and tested
- 📝 **Example**: Example implementation available
- 🚧 **Planned**: On the roadmap
- ⚠️ **Legacy**: Being replaced by newer augmentations
## Using Augmentations
### Zero-Config Approach
```typescript
const brain = new Brainy()
// Just register augmentations - they work automatically!
brain.augmentations.register(new EntityRegistryAugmentation())
brain.augmentations.register(new APIServerAugmentation())
await brain.init()
```
### With Configuration
```typescript
const brain = new Brainy()
brain.augmentations.register(
new APIServerAugmentation({
port: 8080,
auth: { required: true }
})
)
await brain.init()
```
## Creating Custom Augmentations
See [Creating Custom Augmentations](./creating-augmentations.md) for a complete guide.
Quick example:
```typescript
class MyAugmentation extends BaseAugmentation {
readonly name = 'my-augmentation'
readonly timing = 'after'
readonly operations = ['add', 'search']
readonly priority = 50
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
console.log(`Before ${operation}`)
const result = await next()
console.log(`After ${operation}`)
return result
}
}
```
## Augmentation Timing
### `before`
Executes before the main operation. Used for:
- Input validation
- Data transformation
- Authentication checks
### `after`
Executes after the main operation. Used for:
- Broadcasting updates
- Syncing to external services
- Logging and metrics
### `around`
Wraps the main operation. Used for:
- Transactions
- Caching
- Error handling
### `replace`
Completely replaces the main operation. Used for:
- Alternative storage backends
- Mock implementations
- Proxy operations
## Priority System
Higher numbers execute first:
- **100**: Critical system operations
- **50**: Performance optimizations
- **10**: Enhancement features
- **1**: Optional features
## Related Documentation
- [API Server Augmentation](./api-server.md) - Complete API server documentation
- [Creating Augmentations](./creating-augmentations.md) - How to build your own
- [Augmentation Examples](../AUGMENTATION-EXAMPLES.md) - Real-world examples
- [Architecture Overview](../COMPLETE-ARCHITECTURE-VISION.md) - System architecture

429
docs/augmentations/api-server.md
View file

@ -1,429 +0,0 @@
# API Server Augmentation
## Overview
The `APIServerAugmentation` is a powerful augmentation that exposes your Brainy instance through REST, WebSocket, and MCP (Model Context Protocol) APIs. It transforms Brainy into a full-featured API server with zero configuration required.
## Features
### 🌐 REST API
Complete CRUD operations and advanced queries through HTTP endpoints.
### 🔌 WebSocket Server
Real-time bidirectional communication with automatic operation broadcasting.
### 🧠 MCP Integration
Built-in Model Context Protocol support for AI agent communication.
### 📊 Operation Broadcasting
Automatically broadcasts all Brainy operations to subscribed WebSocket clients.
### 🔒 Optional Security
Built-in authentication and rate limiting when needed.
## Installation
The APIServerAugmentation is included in Brainy core. No additional installation required.
**Bun** (recommended): No additional dependencies needed - use `Bun.serve()` directly.
**Node.js**: Install optional Express dependencies:
```bash
npm install express cors ws
```
## Zero-Config Usage
```typescript
import { Brainy } from 'brainy'
import { APIServerAugmentation } from 'brainy/augmentations'
const brain = new Brainy()
// Register the API server augmentation
brain.augmentations.register(new APIServerAugmentation())
await brain.init()
// Server is now running at http://localhost:3000
console.log('API Server ready!')
console.log('REST: http://localhost:3000/api/*')
console.log('WebSocket: ws://localhost:3000/ws')
console.log('MCP: http://localhost:3000/api/mcp')
```
## Configuration Options
While zero-config works great, you can customize the server:
```typescript
const apiServer = new APIServerAugmentation({
enabled: true, // Enable/disable the server
port: 3000, // HTTP port
host: '0.0.0.0', // Bind address
cors: {
origin: '*', // CORS allowed origins
credentials: true // Allow credentials
},
auth: {
required: false, // Require authentication
apiKeys: [], // Valid API keys
bearerTokens: [] // Valid bearer tokens
},
rateLimit: {
windowMs: 60000, // Rate limit window (ms)
max: 100 // Max requests per window
}
})
```
## REST API Endpoints
### Health Check
```http
GET /health
```
Returns server status and basic metrics.
### Search
```http
POST /api/search
Content-Type: application/json
{
"query": "search text",
"limit": 10,
"options": {}
}
```
### Add Data
```http
POST /api/add
Content-Type: application/json
{
"content": "data to add",
"metadata": {
"key": "value"
}
}
```
### Get by ID
```http
GET /api/get/:id
```
### Delete
```http
DELETE /api/delete/:id
```
### Create Relationship
```http
POST /api/relate
Content-Type: application/json
{
"source": "id1",
"target": "id2",
"verb": "relates_to",
"metadata": {}
}
```
### Complex Queries
```http
POST /api/find
Content-Type: application/json
{
"where": { "type": "document" },
"like": "machine learning",
"limit": 10
}
```
### Clustering
```http
POST /api/cluster
Content-Type: application/json
{
"algorithm": "kmeans",
"options": {
"k": 5
}
}
```
### Statistics
```http
GET /api/stats
```
### Operation History
```http
GET /api/history
```
## WebSocket API
### Connection
```javascript
const ws = new WebSocket('ws://localhost:3000/ws')
ws.onopen = () => {
console.log('Connected to Brainy WebSocket')
}
ws.onmessage = (event) => {
const msg = JSON.parse(event.data)
console.log('Received:', msg)
}
```
### Subscribe to Operations
```javascript
ws.send(JSON.stringify({
type: 'subscribe',
operations: ['all'] // or specific: ['add', 'search', 'delete']
}))
```
### Search via WebSocket
```javascript
ws.send(JSON.stringify({
type: 'search',
query: 'your search',
limit: 10,
requestId: 'unique-id'
}))
```
### Add Data via WebSocket
```javascript
ws.send(JSON.stringify({
type: 'add',
content: 'data to add',
metadata: {},
requestId: 'unique-id'
}))
```
### Operation Broadcasts
When subscribed, you'll receive real-time updates:
```javascript
{
"type": "operation",
"operation": "add",
"params": { /* sanitized parameters */ },
"timestamp": 1234567890,
"duration": 15
}
```
## MCP (Model Context Protocol)
The MCP endpoint allows AI agents to interact with Brainy:
```http
POST /api/mcp
Content-Type: application/json
{
"method": "search",
"params": {
"query": "find documents about AI"
}
}
```
## Authentication
When authentication is enabled:
### API Key
```http
GET /api/stats
X-API-Key: your-api-key
```
### Bearer Token
```http
GET /api/stats
Authorization: Bearer your-token
```
## Environment Support
### Bun ✅ (Recommended)
Native performance with Bun.serve() - no Express required. Works with `bun --compile` for single-binary deployment.
```typescript
// Simple Bun server with Brainy
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy()
await brain.init()
Bun.serve({
port: 3000,
async fetch(req) {
const url = new URL(req.url)
if (url.pathname === '/api/search') {
const { query } = await req.json()
const results = await brain.search(query)
return Response.json(results)
}
return new Response('Not Found', { status: 404 })
}
})
```
### Node.js ✅
Full support with Express, WebSocket, and all features.
### Deno 🚧
Planned support using Deno.serve() or oak framework.
### Browser/Service Worker 🚧
Planned support for intercepting fetch() calls locally.
## How It Works
The APIServerAugmentation hooks into Brainy's augmentation pipeline:
1. **Timing**: Executes `after` operations complete
2. **Operations**: Monitors `all` operations
3. **Broadcasting**: Sends operation details to subscribed clients
4. **History**: Maintains operation history (last 1000 operations)
## Example: Multi-Client Sync
```typescript
// Server
const brain = new Brainy()
brain.augmentations.register(new APIServerAugmentation())
await brain.init()
// Client 1 - WebSocket subscriber
const ws1 = new WebSocket('ws://localhost:3000/ws')
ws1.onopen = () => {
ws1.send(JSON.stringify({
type: 'subscribe',
operations: ['add', 'delete']
}))
}
ws1.onmessage = (e) => {
console.log('Client 1 received update:', JSON.parse(e.data))
}
// Client 2 - REST API user
fetch('http://localhost:3000/api/add', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
content: 'New data',
metadata: { source: 'client2' }
})
})
// Client 1 automatically receives notification!
```
## Performance Considerations
- **Operation History**: Limited to last 1000 operations
- **WebSocket Heartbeat**: Every 30 seconds
- **Client Timeout**: 60 seconds of inactivity
- **Parameter Sanitization**: Sensitive fields removed, large content truncated
- **Rate Limiting**: In-memory tracking (use Redis in production)
## Security Notes
1. **Default Configuration**: No auth, open CORS - suitable for development
2. **Production**: Enable auth, configure CORS, use HTTPS
3. **Sensitive Data**: Parameters are sanitized before broadcasting
4. **Rate Limiting**: Basic in-memory implementation included
## Comparison with Previous Implementations
The APIServerAugmentation unifies and replaces:
- `BrainyMCPBroadcast` - Node-specific WebSocket/HTTP server
- `WebSocketConduitAugmentation` - WebSocket client functionality
- `ServerSearchAugmentations` - Remote Brainy connections
Benefits of the unified approach:
- Single augmentation for all API needs
- Consistent interface across protocols
- Automatic operation broadcasting
- Environment-aware implementation
- Zero-configuration philosophy
## Advanced Usage
### Custom Operation Filtering
```typescript
class FilteredAPIServer extends APIServerAugmentation {
shouldExecute(operation: string, params: any): boolean {
// Don't broadcast sensitive operations
if (operation === 'delete' && params.sensitive) {
return false
}
return true
}
}
```
### Integration with Other Augmentations
```typescript
const brain = new Brainy()
// Stack augmentations for complete system
brain.augmentations.register(new EntityRegistryAugmentation()) // Dedup
brain.augmentations.register(new APIServerAugmentation()) // API
await brain.init()
// All augmentations work together seamlessly!
```
## Troubleshooting
### Server won't start
- Check if port is already in use
- Verify Node.js dependencies are installed: `npm install express cors ws`
- Check console for error messages
### WebSocket connections drop
- Ensure heartbeat responses are handled
- Check for proxy/firewall issues
- Verify CORS configuration
### Authentication not working
- Ensure `auth.required` is set to `true`
- Verify API keys or bearer tokens are correctly configured
- Check request headers are properly formatted
## Future Enhancements
- [ ] Deno server implementation
- [ ] Service Worker implementation
- [ ] GraphQL endpoint
- [ ] gRPC support
- [ ] Built-in SSL/TLS
- [ ] Redis-based rate limiting
- [ ] Prometheus metrics endpoint
- [ ] OpenAPI/Swagger documentation
## Related Documentation
- [Augmentation System Overview](../AUGMENTATION-SYSTEM.md)
- [BrainyAugmentation Interface](./brainy-augmentation.md)
- [MCP Integration](../mcp/README.md)
- [Zero-Config Philosophy](../ZERO-CONFIG.md)

6
docs/concepts/consistency-model.md
View file

@ -41,7 +41,7 @@ by a dedicated test in `tests/integration/db-mvcc.test.ts`.
A **monotonic generation counter** is the store's logical clock:
- It advances **once per committed `transact()` batch** and once per
single-operation write (`add`/`update`/`delete`/`relate`/…).
single-operation write (`add`/`update`/`remove`/`relate`/…).
- `brain.generation()` reads it; it is persisted in the data directory and
**never reissued** — not across restarts, and not across `restore()`
(the counter is floored at its pre-restore value).
@ -175,7 +175,7 @@ identical on both paths.
### History granularity — the honest limit
Generation *records* are written per `transact()` batch only.
Single-operation writes (`add`/`update`/`delete`/`relate`/… outside
Single-operation writes (`add`/`update`/`remove`/`relate`/… outside
`transact()`) advance the generation counter — so watermarks and CAS stay
sound — but do **not** stage before-images: they remain visible through
earlier pins and are not reported by `db.since()`. Code that needs pinned
@ -284,7 +284,7 @@ enforce the contract:
`update({ metadata: { confidence: 0.9 } })` therefore behaves exactly
like `update({ confidence: 0.9 })`.
3. **Read time** — every read path (`get`, `find`, `search`,
`getRelations`, batch reads, and historical `asOf()` materialization)
`related`, batch reads, and historical `asOf()` materialization)
surfaces reserved fields **only at top level**: `entity.metadata` and
`relation.metadata` contain only your custom fields, always.

2
docs/features/v3-features.md
View file

@ -152,7 +152,7 @@ Optimized bulk processing:
// Parallel processing with automatic batching
await brain.addMany(items) // <10ms per item
await brain.updateMany(updates)
await brain.deleteMany(filters)
await brain.removeMany(filters)
```
### Request Deduplication ✅

8
docs/guides/framework-integration.md
View file

@ -348,7 +348,7 @@ export default function RootLayout({ children }) {
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy({
storage: { type: 'filesystem', path: './data' }
storage: { type: 'filesystem', rootDirectory: './data' }
})
await brain.init()
@ -536,12 +536,12 @@ import { Brainy } from '@soulcraft/brainy'
export async function generateStaticProps() {
const brain = new Brainy({
storage: { type: 'filesystem', path: './content' }
storage: { type: 'filesystem', rootDirectory: './content' }
})
await brain.init()
// Build search index
const allContent = await brain.export()
// Build search index (paginate with { limit, offset } for larger stores)
const allContent = await brain.find({ limit: 1000 })
return {
props: { searchIndex: allContent }

15
docs/guides/import-anything.md
View file

@ -49,28 +49,23 @@ await brain.import(csv, { format: 'csv' })
### 📊 Import Excel - Multi-Sheet Support
```javascript
// Import entire Excel workbook
// Import entire Excel workbook — every sheet is processed automatically
await brain.import('sales-report.xlsx')
// ✨ Processes all sheets, preserves structure, infers types!
// Or specific sheets only
// Mirror the workbook into the VFS, grouped by sheet
await brain.import('data.xlsx', {
excelSheets: ['Customers', 'Orders']
vfsPath: '/imports/data',
groupBy: 'sheet'
})
// ✨ Multi-sheet data becomes interconnected entities!
```
### 📑 Import PDF - Text & Tables
```javascript
// Import PDF documents
// Import PDF documents — text and tables are extracted automatically
await brain.import('research-paper.pdf')
// ✨ Extracts text, detects tables, preserves metadata!
// With table extraction
await brain.import('report.pdf', {
pdfExtractTables: true
})
// ✨ Converts PDF tables to structured data automatically!
```
### 📝 Import YAML - File or String

13
docs/guides/import-flow.md
View file

@ -1173,21 +1173,22 @@ await this.graphIndex.addEdge(
```
**Benefits**:
- **O(1) relationship lookups**: `getRelations(entityId)` is instant
- **O(1) relationship lookups**: `related(entityId)` is instant
- **Bidirectional traversal**: Find incoming and outgoing edges
- **Type filtering**: Get only `CreatedBy` relationships
- **Global statistics**: Count relationships by type
**Query Examples**:
```typescript
// What did Mona Lisa create?
const outgoing = await brain.getRelations('ent_mona_lisa_...', { direction: 'outgoing' })
// What did Mona Lisa create? (outgoing edges)
const outgoing = await brain.related({ from: 'ent_mona_lisa_...' })
// What created Mona Lisa?
const incoming = await brain.getRelations('ent_mona_lisa_...', { direction: 'incoming' })
// What created Mona Lisa? (incoming edges)
const incoming = await brain.related({ to: 'ent_mona_lisa_...' })
// Get only CreatedBy relationships
const createdBy = await brain.getRelations('ent_mona_lisa_...', {
const createdBy = await brain.related({
from: 'ent_mona_lisa_...',
type: VerbType.CreatedBy
})
```

2
docs/guides/import-quick-reference.md
View file

@ -351,7 +351,7 @@ const result = await brain.import(file)
const products = await brain.find({ type: 'Product' })
// Get entity relationships
const relations = await brain.getRelations(products[0].id)
const relations = await brain.related(products[0].id)
// Search VFS
const vfsFiles = await brain.vfs().find(result.vfs.rootPath + '/**/*.json')

2
docs/guides/inspection.md
View file

@ -134,7 +134,7 @@ console.log(health.overall)
await reader.close()
```
Every mutation method (`add`, `update`, `delete`, `relate`, `transact`,
Every mutation method (`add`, `update`, `remove`, `relate`, `transact`,
`restore`, ...) throws on a read-only instance with a clear message.
## Backups

27
docs/guides/installation.md
View file

@ -5,10 +5,10 @@ public: true
category: getting-started
template: guide
order: 1
description: Install Brainy with npm, bun, yarn, or pnpm. Works in Node.js 22+, Bun 1.0+, and browser (OPFS). TypeScript included.
description: Install Brainy with npm, bun, yarn, or pnpm. Works in Node.js 22+ and Bun 1.0+ (server-only since 8.0). TypeScript included.
next:
- getting-started/quick-start
- concepts/zero-config
- guides/storage-adapters
---
# Installation
@ -53,28 +53,18 @@ npm install @soulcraft/cortex
```typescript
import { Brainy } from '@soulcraft/brainy'
import { registerCortex } from '@soulcraft/cortex'
registerCortex() // activates native acceleration globally
const brain = new Brainy()
await brain.init()
const brain = new Brainy({ plugins: ['@soulcraft/cortex'] })
await brain.init() // native providers registered during init
```
Cortex delivers a **5.2x geometric mean speedup** — see [Brainy vs Cortex](/docs/cortex/comparison) for measured benchmarks.
## Browser (OPFS)
## Server-only since 8.0
Brainy works in the browser using the Origin Private File System:
```typescript
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy({ storage: 'opfs' })
await brain.init()
```
No server required. Data persists across page refreshes in the browser's private storage.
Brainy 8.0 runs on Node.js 22+ and Bun 1.0+. Browser support (OPFS storage,
Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line
remains available on npm if you need it.
## TypeScript
@ -96,5 +86,4 @@ const id = await brain.add({
## Next Steps
- [Quick Start](/docs/getting-started/quick-start) — build your first knowledge graph in 60 seconds
- [Zero Configuration](/docs/concepts/zero-config) — understand what Brainy auto-detects
- [Storage Adapters](/docs/guides/storage-adapters) — choose the right storage for your deployment

2
docs/guides/migrating-to-v4.md
View file

@ -252,7 +252,7 @@ const result = await brain.import('./glossary.xlsx', {
// - "part_of"
// - "related_to"
const relations = await brain.getRelations({ limit: 100 })
const relations = await brain.related({ limit: 100 })
const types = new Set(relations.map(r => r.label))
console.log(types)
// Set { 'capital_of', 'guards', 'located_in', 'related_to' }

2
docs/guides/neural-api.md
View file

@ -367,5 +367,5 @@ const brain = new Brainy({
## Next Steps
- Explore [Triple Intelligence](../architecture/triple-intelligence.md) for combined vector + graph + metadata queries
- Learn about [Augmentations](../augmentations/README.md) to extend Neural API
- Learn about [Plugins](../PLUGINS.md) to extend Brainy with native providers
- See [API Reference](../api/README.md) for complete method documentation

2
docs/guides/storage-adapters.md
View file

@ -8,7 +8,7 @@ order: 2
description: "Two adapters cover every deployment: in-memory for tests + ephemeral workloads, filesystem for everything that needs to persist. Both share one on-disk contract, including generational history and snapshots. Cloud backup is operator tooling, not a built-in adapter."
next:
- guides/plugins
- concepts/zero-config
- concepts/consistency-model
---
# Storage Adapters

8
docs/guides/subtypes-and-facets.md
View file

@ -301,14 +301,14 @@ await brain.relate({
```typescript
// Direct reports — fast path filter (column-store hit, not metadata fallback)
const direct = await brain.getRelations({
const direct = await brain.related({
from: ceoId,
type: VerbType.ReportsTo,
subtype: 'direct'
})
// Set membership
const allReports = await brain.getRelations({
const allReports = await brain.related({
from: ceoId,
type: VerbType.ReportsTo,
subtype: ['direct', 'dotted-line']
@ -552,8 +552,8 @@ Brainy 8.0 ships:
- `brain.relate({ ..., subtype: 'value' })` — write the field
- `brain.updateRelation({ id, subtype, type?, weight?, ... })` — change a relationship
- `brain.getRelations({ subtype })` — filter (fast path)
- `brain.getRelations({ subtype: ['a', 'b'] })` — set membership
- `brain.related({ subtype })` — filter (fast path)
- `brain.related({ subtype: ['a', 'b'] })` — set membership
- `brain.find({ connected: { via, subtype, depth } })` — traversal filter
- `brain.counts.byRelationshipSubtype(verb, subtype?)` — O(1) counts
- `brain.counts.topRelationshipSubtypes(verb, n?)` — top N by count

8
docs/transactions.md
View file

@ -23,7 +23,7 @@ Brainy's transaction system provides **atomic operations** with automatic rollba
- **Atomicity**: All operations succeed or all rollback
- **Consistency**: Indexes and storage remain consistent
- **Automatic**: Transparently used by all `brain.add()`, `brain.update()`, `brain.delete()`, and `brain.relate()` operations
- **Automatic**: Transparently used by all `brain.add()`, `brain.update()`, `brain.remove()`, and `brain.relate()` operations
- **Composable**: `brain.transact()` runs a multi-write batch through the same machinery as exactly one atomic commit
## Architecture
@ -296,7 +296,7 @@ await brain.relate({
})
// Delete person (atomic - deletes entity + relationships)
await brain.delete(personId)
await brain.remove(personId)
// If delete fails, both entity and relationships remain
```
@ -363,7 +363,7 @@ for the history-granularity contract.
// ✅ Recommended: Use Brainy's API (transactions automatic)
await brain.add({ data, type })
await brain.update({ id, data })
await brain.delete(id)
await brain.remove(id)
// ❌ Avoid: Direct storage access bypasses transactions
await brain.storage.saveNoun(noun) // No transaction protection
@ -565,4 +565,4 @@ Brainy's transaction system provides **production-ready atomic operations** with
- ✅ **Compatible**: Works with all storage adapters and features
- ✅ **Coordinated**: Per-entity `ifRev` and whole-store `ifAtGeneration` CAS reject conflicting batches before anything is staged
Start using transactions today - they're already built into `brain.add()`, `brain.update()`, `brain.delete()`, and `brain.relate()` — and reach for `brain.transact()` whenever several writes must land together.
Start using transactions today - they're already built into `brain.add()`, `brain.update()`, `brain.remove()`, and `brain.relate()` — and reach for `brain.transact()` whenever several writes must land together.

2
docs/vfs/README.md
View file

@ -8,7 +8,7 @@
- **[VFS Core](VFS_CORE.md)** - Complete filesystem architecture and API
- **[Semantic VFS](SEMANTIC_VFS.md)** - Multi-dimensional file access (6 semantic dimensions)
- **[Neural Extraction](NEURAL_EXTRACTION.md)** - AI-powered concept and entity extraction
- **[Examples & Scenarios](VFS_EXAMPLES_SCENARIOS.md)** - Real-world use cases and code
- **[Common Patterns](COMMON_PATTERNS.md)** - Real-world use cases and code
- **[VFS API Guide](VFS_API_GUIDE.md)** - Complete API reference
## What is Brainy VFS?

6
docs/vfs/SEMANTIC_VFS.md
View file

@ -400,9 +400,9 @@ Semantic VFS is built on Brainy's Triple Intelligence™:
### Real Implementations, Zero Mocks
Every component uses **production Brainy APIs**:
- `brain.find()` - Real metadata queries (brainy.ts:580)
- `brain.similar()` - Real HNSW search (brainy.ts:680)
- `brain.getRelations()` - Real graph traversal (brainy.ts:803)
- `brain.find()` - Real metadata queries
- `brain.similar()` - Real HNSW search
- `brain.related()` - Real graph traversal
- `MetadataIndexManager` - Real B-tree indexes
- `GraphAdjacencyIndex` - Real graph storage
- `HNSW Index` - Real vector search

4
docs/vfs/TROUBLESHOOTING.md
View file

@ -183,7 +183,7 @@ console.log('VFS type:', entity.metadata.vfsType)
### 3. Check Relationships
```javascript
const parentId = await vfs.resolvePath('/directory')
const relations = await brain.getRelations({
const relations = await brain.related({
from: parentId,
type: VerbType.Contains
})
@ -246,7 +246,7 @@ const tree = await vfs.getTreeStructure('/', {
If you encounter issues not covered here:
1. Check the [VFS API Guide](./VFS_API_GUIDE.md)
2. Review [VFS Examples](./VFS_EXAMPLES_SCENARIOS.md)
2. Review [Common Patterns](./COMMON_PATTERNS.md)
3. Look at [test files](../../tests/vfs/) for working examples
4. Report issues at [GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)

728
docs/vfs/USER_FUNCTIONS.md
View file

@ -1,728 +0,0 @@
# VFS User Functions - Templates and Examples
This document provides template functions that you can implement for domain-specific needs. These functions combine VFS primitives to solve common problems.
## Table of Contents
1. [Code Analysis Functions](#code-analysis-functions)
2. [Export Format Functions](#export-format-functions)
3. [Project Management Functions](#project-management-functions)
4. [Creative Writing Functions](#creative-writing-functions)
5. [Game Development Functions](#game-development-functions)
## Code Analysis Functions
### Get Dependency Graph
```javascript
/**
* Build a dependency graph for JavaScript/TypeScript projects
*/
async function getDependencyGraph(vfs, srcPath) {
const files = await vfs.readdir(srcPath, { recursive: true })
const graph = {}
for (const file of files) {
const filePath = `${srcPath}/${file}`
// Only process JS/TS files
if (file.match(/\.(js|ts|jsx|tsx)$/)) {
const content = await vfs.readFile(filePath)
const text = content.toString()
// Parse imports (basic regex, use proper AST parser for production)
const imports = []
const importRegex = /import\s+.*?\s+from\s+['"](.+?)['"]/g
const requireRegex = /require\(['"](.+?)['"]\)/g
let match
while ((match = importRegex.exec(text)) !== null) {
imports.push(match[1])
}
while ((match = requireRegex.exec(text)) !== null) {
imports.push(match[1])
}
graph[filePath] = imports
}
}
return graph
}
// Use it
const deps = await getDependencyGraph(vfs, '/src')
```
### Find Circular Dependencies
```javascript
/**
* Detect circular dependencies in your code
*/
async function findCircularDependencies(vfs, srcPath) {
const graph = await getDependencyGraph(vfs, srcPath)
const cycles = []
function detectCycle(node, visited = new Set(), stack = []) {
if (stack.includes(node)) {
const cycleStart = stack.indexOf(node)
cycles.push(stack.slice(cycleStart))
return
}
if (visited.has(node)) return
visited.add(node)
stack.push(node)
const dependencies = graph[node] || []
for (const dep of dependencies) {
// Resolve relative imports
const resolvedDep = resolvePath(node, dep)
if (graph[resolvedDep]) {
detectCycle(resolvedDep, visited, [...stack])
}
}
}
Object.keys(graph).forEach(node => detectCycle(node))
return cycles
}
```
### Find Untested Code
```javascript
/**
* Find source files without corresponding test files
*/
async function findUntestedCode(vfs, srcPath, testPath = null) {
testPath = testPath || srcPath.replace('/src', '/tests')
const sourceFiles = await vfs.readdir(srcPath, { recursive: true })
const testFiles = await vfs.readdir(testPath, { recursive: true }).catch(() => [])
const untestedFiles = []
for (const sourceFile of sourceFiles) {
if (!sourceFile.match(/\.(js|ts|jsx|tsx)$/)) continue
// Look for corresponding test file
const baseName = sourceFile.replace(/\.(js|ts|jsx|tsx)$/, '')
const hasTest = testFiles.some(testFile =>
testFile.includes(baseName) &&
testFile.match(/\.(test|spec)\.(js|ts|jsx|tsx)$/)
)
if (!hasTest) {
untestedFiles.push(`${srcPath}/${sourceFile}`)
}
}
return untestedFiles
}
```
### Find Similar Code (Duplicate Detection)
```javascript
/**
* Find potentially duplicate code using similarity scoring
*/
async function findSimilarCode(vfs, filePath, options = {}) {
const threshold = options.threshold || 0.8
const searchPath = options.searchPath || '/'
// Get the reference file content
const referenceContent = await vfs.readFile(filePath)
const referenceText = referenceContent.toString()
// Use VFS's semantic search
const similar = await vfs.findSimilar(filePath, {
limit: 10,
threshold
})
// Additionally, do structural comparison
const results = []
for (const match of similar) {
const matchContent = await vfs.readFile(match.path)
const matchText = matchContent.toString()
// Simple line-based similarity (use better algorithms in production)
const similarity = calculateSimilarity(referenceText, matchText)
if (similarity > threshold) {
results.push({
path: match.path,
similarity,
semanticScore: match.score
})
}
}
return results.sort((a, b) => b.similarity - a.similarity)
}
function calculateSimilarity(text1, text2) {
// Simple Jaccard similarity on lines
const lines1 = new Set(text1.split('\n').map(l => l.trim()).filter(l => l))
const lines2 = new Set(text2.split('\n').map(l => l.trim()).filter(l => l))
const intersection = new Set([...lines1].filter(x => lines2.has(x)))
const union = new Set([...lines1, ...lines2])
return intersection.size / union.size
}
```
## Export Format Functions
### Export to EPUB (for novels)
```javascript
/**
* Export a directory of markdown files to EPUB format
*/
async function exportToEpub(vfs, path, metadata = {}) {
// First get the markdown export
const markdown = await vfs.exportToMarkdown(path)
// You'll need an EPUB library like epub-gen
const Epub = require('epub-gen')
// Convert markdown chapters to EPUB format
const chapters = []
const files = await vfs.readdir(path, { recursive: true })
for (const file of files.sort()) {
if (file.endsWith('.md')) {
const content = await vfs.readFile(`${path}/${file}`)
const title = file.replace('.md', '').replace(/-/g, ' ')
chapters.push({
title: title,
data: content.toString()
})
}
}
const options = {
title: metadata.title || 'My Book',
author: metadata.author || 'Author',
chapters: chapters
}
return new Epub(options)
}
```
### Export to Static Site
```javascript
/**
* Export VFS content to static HTML site
*/
async function exportToStaticSite(vfs, sourcePath, options = {}) {
const json = await vfs.exportToJSON(sourcePath)
const html = []
html.push('<!DOCTYPE html>')
html.push('<html><head>')
html.push(`<title>${options.title || 'Documentation'}</title>`)
html.push('<style>/* Add your styles */</style>')
html.push('</head><body>')
function renderNode(node, name, depth = 0) {
const indent = ' '.repeat(depth)
if (node._meta?.type === 'file') {
html.push(`${indent}<article>`)
html.push(`${indent} <h${Math.min(depth + 2, 6)}>${name}</h${Math.min(depth + 2, 6)}>`)
if (typeof node._content === 'string') {
// Convert markdown to HTML if needed
html.push(`${indent} <pre>${escapeHtml(node._content)}</pre>`)
}
html.push(`${indent}</article>`)
} else if (node._meta?.type === 'directory') {
html.push(`${indent}<section>`)
html.push(`${indent} <h${Math.min(depth + 1, 6)}>${name}</h${Math.min(depth + 1, 6)}>`)
for (const [childName, childNode] of Object.entries(node)) {
if (!childName.startsWith('_')) {
renderNode(childNode, childName, depth + 1)
}
}
html.push(`${indent}</section>`)
}
}
renderNode(json, options.title || 'Root')
html.push('</body></html>')
return html.join('\n')
}
```
### Export to GraphQL Schema
```javascript
/**
* Generate GraphQL schema from VFS entities
*/
async function exportToGraphQLSchema(vfs) {
const entities = await vfs.listEntities()
const types = new Map()
// Group entities by type
for (const entity of entities) {
const type = entity.type || 'Unknown'
if (!types.has(type)) {
types.set(type, [])
}
types.get(type).push(entity)
}
// Generate schema
let schema = 'type Query {\n'
for (const [typeName, entities] of types) {
schema += ` get${typeName}(id: ID!): ${typeName}\n`
schema += ` list${typeName}s: [${typeName}!]!\n`
}
schema += '}\n\n'
// Generate types
for (const [typeName, entities] of types) {
schema += `type ${typeName} {\n`
schema += ' id: ID!\n'
// Infer fields from first entity
if (entities.length > 0) {
const sample = entities[0]
for (const [key, value] of Object.entries(sample)) {
if (key !== 'id') {
const fieldType = inferGraphQLType(value)
schema += ` ${key}: ${fieldType}\n`
}
}
}
schema += '}\n\n'
}
return schema
}
```
## Project Management Functions
### Get Project Insights
```javascript
/**
* Analyze project for insights and patterns
*/
async function getProjectInsights(vfs, projectPath) {
const stats = await vfs.getProjectStats(projectPath)
const todos = await vfs.getAllTodos(projectPath)
const timeline = await vfs.getTimeline({ limit: 100 })
// Analyze activity patterns
const activityByDay = {}
const activityByUser = {}
const activityByFile = {}
for (const event of timeline) {
const day = event.timestamp.toISOString().split('T')[0]
activityByDay[day] = (activityByDay[day] || 0) + 1
const user = event.user || 'system'
activityByUser[user] = (activityByUser[user] || 0) + 1
activityByFile[event.path] = (activityByFile[event.path] || 0) + 1
}
// Find hotspots (most edited files)
const hotspots = Object.entries(activityByFile)
.sort(([,a], [,b]) => b - a)
.slice(0, 10)
.map(([path, count]) => ({ path, edits: count }))
// Todo analysis
const todosByPriority = {}
const todosByStatus = {}
for (const todo of todos) {
todosByPriority[todo.priority] = (todosByPriority[todo.priority] || 0) + 1
todosByStatus[todo.status] = (todosByStatus[todo.status] || 0) + 1
}
return {
stats,
activity: {
byDay: activityByDay,
byUser: activityByUser,
hotspots
},
todos: {
total: todos.length,
byPriority: todosByPriority,
byStatus: todosByStatus,
highPriority: todos.filter(t => t.priority === 'high' && t.status === 'pending')
},
recommendations: generateRecommendations(stats, todos, hotspots)
}
}
function generateRecommendations(stats, todos, hotspots) {
const recommendations = []
if (stats.largestFile && stats.largestFile.size > 1024 * 1024) {
recommendations.push({
type: 'refactor',
message: `Consider splitting ${stats.largestFile.path} (${Math.round(stats.largestFile.size / 1024)}KB)`
})
}
if (todos.filter(t => t.priority === 'high' && t.status === 'pending').length > 5) {
recommendations.push({
type: 'priority',
message: 'You have many high-priority pending todos'
})
}
if (hotspots.length > 0 && hotspots[0].edits > 50) {
recommendations.push({
type: 'stability',
message: `${hotspots[0].path} changes frequently, consider stabilizing`
})
}
return recommendations
}
```
### Generate Sprint Report
```javascript
/**
* Generate a report for the current sprint
*/
async function generateSprintReport(vfs, sprintStart, sprintEnd = new Date()) {
const timeline = await vfs.getTimeline({
from: sprintStart,
to: sprintEnd
})
const todos = await vfs.getAllTodos()
// Group work by user
const workByUser = {}
for (const event of timeline) {
const user = event.user || 'system'
if (!workByUser[user]) {
workByUser[user] = {
commits: 0,
filesModified: new Set(),
linesChanged: 0
}
}
workByUser[user].commits++
workByUser[user].filesModified.add(event.path)
}
// Calculate completion rate
const completedTodos = todos.filter(t => t.status === 'completed').length
const totalTodos = todos.length
const completionRate = totalTodos > 0 ? (completedTodos / totalTodos * 100).toFixed(1) : 0
return {
period: {
start: sprintStart,
end: sprintEnd,
days: Math.ceil((sprintEnd - sprintStart) / (1000 * 60 * 60 * 24))
},
team: Object.entries(workByUser).map(([user, work]) => ({
user,
commits: work.commits,
filesModified: work.filesModified.size
})),
todos: {
completed: completedTodos,
total: totalTodos,
completionRate: `${completionRate}%`,
remaining: todos.filter(t => t.status === 'pending')
},
velocity: {
commitsPerDay: (timeline.length / 7).toFixed(1),
todosPerDay: (completedTodos / 7).toFixed(1)
}
}
}
```
## Creative Writing Functions
### Track Character Arcs
```javascript
/**
* Track how characters evolve throughout a story
*/
async function trackCharacterArc(vfs, characterName, storyPath = '/') {
// Find the character entity
const entities = await vfs.searchEntities({
type: 'character',
name: characterName
})
if (entities.length === 0) {
throw new Error(`Character ${characterName} not found`)
}
const character = entities[0]
const occurrences = await vfs.findEntityOccurrences(character.id)
// Analyze each appearance
const arc = []
for (const occurrence of occurrences) {
const content = await vfs.readFile(occurrence.path)
const text = content.toString()
// Find mentions of the character (basic approach)
const mentions = text.split('\n').filter(line =>
line.toLowerCase().includes(characterName.toLowerCase())
)
arc.push({
chapter: occurrence.path,
mentions: mentions.length,
// Analyze emotional tone (simplified)
mood: analyzeMood(mentions),
// Extract key actions
actions: extractActions(mentions, characterName)
})
}
return {
character: character.metadata,
arc: arc,
summary: summarizeArc(arc)
}
}
function analyzeMood(mentions) {
const positiveWords = ['smiled', 'laughed', 'happy', 'joy', 'love', 'success']
const negativeWords = ['cried', 'angry', 'sad', 'fear', 'fail', 'death']
let positive = 0, negative = 0
for (const mention of mentions) {
const lower = mention.toLowerCase()
positive += positiveWords.filter(w => lower.includes(w)).length
negative += negativeWords.filter(w => lower.includes(w)).length
}
if (positive > negative) return 'positive'
if (negative > positive) return 'negative'
return 'neutral'
}
```
### Generate Story Bible
```javascript
/**
* Create a comprehensive reference for your story universe
*/
async function generateStoryBible(vfs, storyPath) {
const characters = await vfs.listEntities({ type: 'character' })
const locations = await vfs.listEntities({ type: 'location' })
const concepts = await vfs.findConcepts({ domain: 'narrative' })
const bible = {
title: 'Story Bible',
generated: new Date(),
characters: {},
locations: {},
plotThreads: {},
timeline: []
}
// Document characters
for (const char of characters) {
const occurrences = await vfs.findEntityOccurrences(char.id)
bible.characters[char.metadata.name] = {
...char.metadata,
appearances: occurrences.map(o => o.path),
relationships: await vfs.getEntityGraph(char.id, { depth: 1 })
}
}
// Document locations
for (const loc of locations) {
bible.locations[loc.metadata.name] = {
...loc.metadata,
scenes: await vfs.findEntityOccurrences(loc.id)
}
}
// Plot threads from concepts
for (const concept of concepts) {
if (concept.type === 'plot') {
bible.plotThreads[concept.name] = {
description: concept.description,
keywords: concept.keywords,
chapters: await vfs.findByConcept(concept.name)
}
}
}
// Generate timeline
const events = await vfs.getTimeline({ limit: 1000 })
bible.timeline = events.map(e => ({
date: e.timestamp,
event: e.description,
chapter: e.path
}))
return bible
}
```
## Game Development Functions
### Validate Game Data
```javascript
/**
* Validate game configuration files for consistency
*/
async function validateGameData(vfs, gamePath) {
const errors = []
const warnings = []
// Load all game data
const gameData = await vfs.exportToJSON(gamePath)
// Check quest references
if (gameData.quests) {
for (const [questName, quest] of Object.entries(gameData.quests)) {
// Check NPC references
if (quest._content?.questGiver) {
const npcPath = `${gamePath}/npcs/${quest._content.questGiver}.json`
const exists = await vfs.exists(npcPath)
if (!exists) {
errors.push(`Quest ${questName} references missing NPC: ${quest._content.questGiver}`)
}
}
// Check item rewards
if (quest._content?.rewards?.items) {
for (const item of quest._content.rewards.items) {
const itemPath = `${gamePath}/items/${item}.json`
const exists = await vfs.exists(itemPath)
if (!exists) {
warnings.push(`Quest ${questName} rewards missing item: ${item}`)
}
}
}
}
}
// Check balance
if (gameData.items) {
const itemPowers = []
for (const [itemName, item] of Object.entries(gameData.items)) {
if (item._content?.stats) {
const totalPower = Object.values(item._content.stats)
.reduce((a, b) => a + b, 0)
itemPowers.push({ name: itemName, power: totalPower })
}
}
// Find outliers
const avgPower = itemPowers.reduce((a, b) => a + b.power, 0) / itemPowers.length
const outliers = itemPowers.filter(i => Math.abs(i.power - avgPower) > avgPower * 2)
for (const outlier of outliers) {
warnings.push(`Item ${outlier.name} may be imbalanced (power: ${outlier.power}, avg: ${avgPower})`)
}
}
return { errors, warnings, valid: errors.length === 0 }
}
```
### Generate Loot Tables
```javascript
/**
* Generate weighted loot tables from item definitions
*/
async function generateLootTables(vfs, itemsPath) {
const items = await vfs.exportToJSON(itemsPath)
const tables = {
common: [],
uncommon: [],
rare: [],
epic: [],
legendary: []
}
for (const [itemName, item] of Object.entries(items)) {
if (item._meta?.type === 'file' && item._content?.rarity) {
const entry = {
name: itemName.replace('.json', ''),
weight: getWeight(item._content.rarity),
data: item._content
}
tables[item._content.rarity.toLowerCase()].push(entry)
}
}
// Normalize weights
for (const table of Object.values(tables)) {
const totalWeight = table.reduce((a, b) => a + b.weight, 0)
for (const entry of table) {
entry.probability = (entry.weight / totalWeight * 100).toFixed(2) + '%'
}
}
return tables
}
function getWeight(rarity) {
const weights = {
common: 100,
uncommon: 50,
rare: 20,
epic: 5,
legendary: 1
}
return weights[rarity.toLowerCase()] || 10
}
```
## Using These Functions
All these functions are templates that you can customize for your specific needs. To use them:
1. Copy the function you need
2. Modify it for your specific requirements
3. Use it with your VFS instance:
```javascript
import { Brainy } from '@soulcraft/brainy'
// Initialize VFS
const brain = new Brainy()
await brain.init()
const vfs = brain.vfs()
await vfs.init()
// Use your custom function
const insights = await getProjectInsights(vfs, '/my-project')
console.log(insights.recommendations)
// Combine multiple functions
const deps = await getDependencyGraph(vfs, '/src')
const cycles = await findCircularDependencies(vfs, '/src')
const untested = await findUntestedCode(vfs, '/src', '/tests')
```
Remember: These are starting points. The power of VFS is that you can combine its primitives to build exactly what you need for your domain!

684
docs/vfs/VFS_EXAMPLES_SCENARIOS.md
View file

@ -1,684 +0,0 @@
# VFS Examples and Scenarios
> **⚠️ DOCUMENTATION IN PROGRESS**: This document is being updated to reflect recent VFS architecture changes. Many examples below reference deprecated Knowledge Layer methods. Updated examples coming soon. For current VFS capabilities, see [VFS_CORE.md](./VFS_CORE.md) and [SEMANTIC_VFS.md](./SEMANTIC_VFS.md).
## Real-World Scenarios
This document demonstrates how VFS enables powerful real-world applications with semantic search, relationships, and AI-powered concept extraction.
### Legend
- ✅ **Real VFS methods** - Fully implemented and working
- 📝 **User functions** - Templates available in [USER_FUNCTIONS.md](./USER_FUNCTIONS.md)
- 🔮 **Future features** - Not yet available (AI augmentations)
**Note:** All ✅ marked methods are production-ready. For 📝 methods, see USER_FUNCTIONS.md for implementation templates.
## Scenario 1: Collaborative Novel Writing
Multiple authors working on a shared universe with recurring characters and locations.
```javascript
import { Brainy } from '@soulcraft/brainy'
async function novelWritingProject() {
const brain = new Brainy({ storage: { type: 'file', path: './novel-data' } })
await brain.init()
const vfs = brain.vfs()
await vfs.init()
// Create project structure ✅
await vfs.mkdir('/novel')
await vfs.mkdir('/novel/chapters')
await vfs.mkdir('/novel/characters')
await vfs.mkdir('/novel/worldbuilding')
// Define main characters as files with rich metadata ✅
await vfs.writeFile('/novel/characters/elena-blackwood.md', `
# Elena Blackwood
A skilled detective with a mysterious past.
## Character Profile
- Age: 32
- Occupation: Private Investigator
- Skills: Deduction, combat, languages
- Personality: Determined, secretive, compassionate
`, {
metadata: {
characterType: 'protagonist',
tags: ['detective', 'mysterious', 'protagonist']
}
})
await vfs.writeFile('/novel/characters/marcus-void.md', `
# Marcus Void
description: 'A wealthy industrialist with dark secrets',
attributes: {
age: 45,
occupation: 'CEO of Void Industries',
traits: ['ruthless', 'charismatic', 'brilliant']
}
})
// Create location entities
const city = await vfs.createEntity({
name: 'Neo Tokyo',
type: 'location',
description: 'A sprawling cyberpunk metropolis',
attributes: {
population: '40 million',
districts: ['Shibuya-5', 'Crypto Quarter', 'Old Town'],
atmosphere: 'neon-lit, rain-soaked, vertical'
}
})
// Link entities ✅
await vfs.linkEntities(protagonist.id, city.id, 'lives_in')
await vfs.linkEntities(protagonist.id, antagonist.id, 'investigates')
// Write chapters with automatic entity tracking
await vfs.writeFile('/novel/chapters/chapter1.md', `
# Chapter 1: Rain in Neo Tokyo
Elena Blackwood stood at the edge of Shibuya-5, watching the rain cascade down
the neon-lit towers. The case file on Marcus Void burned in her pocket.
She had been tracking Void Industries for months, following a trail of
disappeared scientists and mysterious experiments. Tonight, she would finally
infiltrate the Crypto Quarter facility.
`)
// Multiple authors can work simultaneously ✅
vfs.setUser('author-alice')
await vfs.writeFile('/novel/chapters/chapter2.md', `
# Chapter 2: The Void Industries Tower
Marcus Void gazed down at Neo Tokyo from his penthouse office. Somewhere
in those rain-slicked streets, Elena Blackwood was hunting him...
`)
vfs.setUser('author-bob')
await vfs.appendFile('/novel/chapters/chapter2.md', `
He smiled. Let her come. The trap was already set.
`)
// Track character appearances across chapters ✅
const elenaAppearances = await vfs.findEntityOccurrences(protagonist.id)
console.log('Elena appears in:', elenaAppearances.map(f => f.path))
// Find all locations mentioned ✅
const locations = await vfs.listEntities({ type: 'location' })
// Generate character relationship graph ✅
const relationships = await vfs.getEntityGraph(protagonist.id, { depth: 2 })
// Track plot threads using concepts ✅
await vfs.createConcept({
name: 'The Void Conspiracy',
type: 'plot',
domain: 'narrative',
description: 'The main mystery involving disappeared scientists',
keywords: ['scientists', 'experiments', 'void industries', 'conspiracy']
})
// Find all chapters related to the conspiracy ✅
const conspiracyChapters = await vfs.findByConcept('The Void Conspiracy')
// Version control for revisions ✅
const chapterVersions = await vfs.getVersions('/novel/chapters/chapter1.md')
// Collaborative editing history ✅
const history = await vfs.getCollaborationHistory('/novel/chapters/chapter2.md')
console.log('Chapter 2 edited by:', history.map(h => h.user))
// Export for publishing ✅
const manuscript = await vfs.exportToMarkdown('/novel/chapters')
await vfs.close()
await brain.close()
}
```
## Scenario 2: Video Game Development
Game developers creating a complex RPG with quests, items, and NPCs.
```javascript
async function gameDevProject() {
const brain = new Brainy({ storage: { type: 's3', bucket: 'game-assets' } })
await brain.init()
const vfs = brain.vfs()
await vfs.init()
// Game project structure
await vfs.mkdir('/game')
await vfs.mkdir('/game/scripts')
await vfs.mkdir('/game/assets')
await vfs.mkdir('/game/quests')
await vfs.mkdir('/game/npcs')
await vfs.mkdir('/game/items')
// Define game systems as concepts
await vfs.createConcept({
name: 'Combat System',
type: 'system',
domain: 'gameplay',
keywords: ['damage', 'health', 'attacks', 'defense']
})
await vfs.createConcept({
name: 'Quest System',
type: 'system',
domain: 'gameplay',
keywords: ['objectives', 'rewards', 'dialogue', 'progression']
})
// Create NPC entities
const questGiver = await vfs.createEntity({
name: 'Elder Sage',
type: 'npc',
description: 'Wise old man who gives the main quest',
attributes: {
location: 'Village Square',
questsOffered: ['The Ancient Artifact', 'Lost Knowledge'],
dialogue: {
greeting: "Welcome, young adventurer...",
questStart: "I have a task of great importance..."
}
}
})
// Create item entities
const artifact = await vfs.createEntity({
name: 'Crystal of Power',
type: 'item',
description: 'A legendary artifact with immense magical power',
attributes: {
rarity: 'Legendary',
stats: { magic: 100, wisdom: 50 },
questItem: true,
lore: 'Forged in the age of dragons...'
}
})
// Write quest scripts
await vfs.writeFile('/game/quests/main_quest.json', JSON.stringify({
id: 'main_quest_01',
name: 'The Ancient Artifact',
description: 'Retrieve the Crystal of Power from the Dark Tower',
objectives: [
{ id: 'obj_1', description: 'Speak to the Elder Sage' },
{ id: 'obj_2', description: 'Travel to the Dark Tower' },
{ id: 'obj_3', description: 'Defeat the Guardian' },
{ id: 'obj_4', description: 'Retrieve the Crystal of Power' }
],
rewards: {
experience: 5000,
gold: 1000,
items: ['Crystal of Power']
}
}, null, 2))
// Link quest elements
await vfs.linkEntities(questGiver.id, 'main_quest_01', 'gives_quest')
await vfs.linkEntities('main_quest_01', artifact.id, 'rewards_item')
// Combat script with system tracking
await vfs.writeFile('/game/scripts/combat.js', `
class CombatSystem {
calculateDamage(attacker, defender) {
const baseDamage = attacker.stats.attack
const defense = defender.stats.defense
return Math.max(1, baseDamage - defense)
}
executeAttack(attacker, defender) {
const damage = this.calculateDamage(attacker, defender)
defender.health -= damage
return { damage, remaining: defender.health }
}
}
export default CombatSystem
`)
// Asset management ✅
await vfs.writeFile('/game/assets/sprites/elder_sage.png', spriteData)
await vfs.setMetadata('/game/assets/sprites/elder_sage.png', {
dimensions: '64x64',
animations: ['idle', 'talking'],
artist: 'Alice',
license: 'CC-BY-4.0'
})
// Track dependencies ✅
await vfs.addRelationship('/game/quests/main_quest.json', '/game/npcs/elder_sage.json', 'uses')
await vfs.addRelationship('/game/scripts/combat.js', '/game/systems/stats.js', 'imports')
// Find all content related to combat ✅
const combatFiles = await vfs.findByConcept('Combat System')
// Get all NPCs in a specific location ✅
const villageNPCs = await vfs.searchEntities({
type: 'npc',
where: { 'attributes.location': 'Village Square' }
})
// Track game balance changes — read the file as it was a week ago ✅
const lastWeek = await brain.asOf(new Date(Date.now() - 7 * 86_400_000))
const oldBalance = await lastWeek.find({ where: { path: '/game/data/balance.json' }, limit: 1 })
await lastWeek.release()
// Collaborative development tracking ✅
await vfs.addTodo('/game/quests/main_quest.json', {
task: 'Add voice dialogue triggers',
priority: 'medium',
status: 'pending',
assignee: 'audio-team'
})
// Export for build system ✅
const gameData = await vfs.exportToJSON('/game')
await vfs.close()
await brain.close()
}
```
## Scenario 3: Software Development Project
Building a web application with full project management.
```javascript
async function softwareProject() {
const brain = new Brainy({ storage: { type: 'r2', bucket: 'project-files' } })
await brain.init()
const vfs = brain.vfs()
await vfs.init()
// Import existing git repository ✅
await vfs.importFromGit('/local/repos/webapp', '/project')
// Define architectural concepts
await vfs.createConcept({
name: 'Authentication',
type: 'architecture',
domain: 'backend',
keywords: ['jwt', 'login', 'session', 'oauth'],
relatedConcepts: ['Security', 'User Management']
})
await vfs.createConcept({
name: 'State Management',
type: 'architecture',
domain: 'frontend',
keywords: ['redux', 'context', 'store', 'actions']
})
// Write source code
await vfs.writeFile('/project/src/auth/login.ts', `
import { User } from '../models/User'
import { generateJWT } from '../utils/jwt'
export async function login(email: string, password: string): Promise<string> {
const user = await User.findByEmail(email)
if (!user || !user.verifyPassword(password)) {
throw new Error('Invalid credentials')
}
return generateJWT(user.id)
}
`)
// Track imports and dependencies
await vfs.addRelationship('/project/src/auth/login.ts', '/project/src/models/User.ts', 'imports')
await vfs.addRelationship('/project/src/auth/login.ts', '/project/src/utils/jwt.ts', 'imports')
// Add inline TODOs
await vfs.addTodo('/project/src/auth/login.ts', {
task: 'Add rate limiting',
priority: 'high',
status: 'pending',
assignee: 'security-team',
due: '2025-02-01'
})
await vfs.addTodo('/project/src/auth/login.ts', {
task: 'Implement OAuth providers',
priority: 'medium',
status: 'in_progress',
assignee: 'backend-team'
})
// Configuration files
await vfs.writeFile('/project/config/database.json', JSON.stringify({
development: {
host: 'localhost',
port: 5432,
database: 'webapp_dev'
},
production: {
host: '${DB_HOST}',
port: 5432,
database: 'webapp_prod'
}
}, null, 2))
// Set security metadata
await vfs.setMetadata('/project/config/database.json', {
sensitive: true,
environment: 'all',
lastReviewed: '2025-01-15',
reviewer: 'security-team'
})
// Tests with relationship to source
await vfs.writeFile('/project/tests/auth/login.test.ts', `
import { login } from '../../src/auth/login'
describe('Authentication', () => {
test('valid login returns JWT', async () => {
const token = await login('user@example.com', 'password')
expect(token).toBeDefined()
})
})
`)
await vfs.addRelationship('/project/tests/auth/login.test.ts', '/project/src/auth/login.ts', 'tests')
// Documentation
await vfs.writeFile('/project/docs/API.md', `
# API Documentation
## Authentication
### POST /api/login
Authenticates a user and returns a JWT token.
**Request:**
\`\`\`json
{
"email": "user@example.com",
"password": "secret"
}
\`\`\`
`)
// Find all files needing security review ✅
const securityFiles = await vfs.search('authentication password jwt oauth', {
path: '/project/src',
type: 'file'
})
// Get project insights 📝 (see USER_FUNCTIONS.md for getProjectInsights)
const insights = await getProjectInsights(vfs, '/project') // User function
console.log('Most modified files:', insights.hotspots)
console.log('Key concepts:', insights.concepts)
console.log('Team activity:', insights.contributors)
// Find circular dependencies 📝 (see USER_FUNCTIONS.md)
const circularDeps = await findCircularDependencies(vfs, '/project/src') // User function
// Get test coverage relationships 📝 (see USER_FUNCTIONS.md)
const untested = await findUntestedCode(vfs, '/project/src') // User function
// Track technical debt ✅
const todos = await vfs.getAllTodos('/project')
const highPriorityDebt = todos.filter(t => t.priority === 'high' && t.status === 'pending')
// Generate dependency graph 📝 (see USER_FUNCTIONS.md)
const depGraph = await getDependencyGraph(vfs, '/project/src') // User function
// Find similar code (potential refactoring) 📝 (see USER_FUNCTIONS.md)
const similarCode = await findSimilarCode(vfs, '/project/src/auth/login.ts', {
threshold: 0.8,
minLines: 10
}) // User function
// Export for CI/CD ✅ (Knowledge Layer provides wrapper)
await vfs.exportToGit('/project', '/tmp/build-output')
// Collaborative features ✅ / 🔮
vfs.setUser('developer-alice') // ✅ Real method
// const conflicts = await vfs.detectConflicts('/project/src/auth/login.ts') // 🔮 Future feature
await vfs.close()
await brain.close()
}
```
## Scenario 4: Multi-Project Knowledge Base
All projects in one Brainy instance, sharing concepts and cross-referencing.
```javascript
async function unifiedKnowledgeBase() {
const brain = new Brainy({
storage: { type: 'file', path: './knowledge-base' }
})
await brain.init()
const vfs = brain.vfs()
await vfs.init()
// Create separate project spaces
await vfs.mkdir('/novel')
await vfs.mkdir('/game')
await vfs.mkdir('/webapp')
// Shared universe - characters appear in both novel and game
const sharedCharacter = await vfs.createEntity({
name: 'Elena Blackwood',
type: 'character',
description: 'Protagonist appearing in multiple works'
})
// Novel chapter mentioning Elena
await vfs.writeFile('/novel/chapter1.md', `
Elena Blackwood stood at the edge of the digital void...
`)
// Game script featuring Elena as NPC
await vfs.writeFile('/game/npcs/elena.json', JSON.stringify({
name: 'Elena Blackwood',
type: 'ally',
dialogue: ['I\'ve seen this before in my world...']
}))
// Web app has Elena as example user
await vfs.writeFile('/webapp/seeds/users.json', JSON.stringify([{
name: 'Elena Blackwood',
email: 'elena@example.com',
role: 'detective'
}]))
// Cross-project entity tracking ✅
const elenaOccurrences = await vfs.findEntityOccurrences(sharedCharacter.id)
console.log('Elena appears across projects:', elenaOccurrences)
// Shared technical concepts
const authConcept = await vfs.createConcept({
name: 'Authentication',
type: 'technical',
domain: 'software'
})
// Find auth implementations across all projects ✅
const authImplementations = await vfs.findByConcept('Authentication')
// Returns: /webapp/src/auth.js, /game/scripts/player-auth.js, etc.
// Cross-project relationships ✅
await vfs.addRelationship('/novel/chapter1.md', '/game/story/intro.txt', 'inspires')
await vfs.addRelationship('/game/npcs/elena.json', '/novel/characters/elena.md', 'based_on')
// Universal search across all projects ✅
const results = await vfs.search('Elena Blackwood authentication', {
path: '/',
recursive: true
})
// Project statistics 📝 (see USER_FUNCTIONS.md for getProjectStats)
const novelStats = await getProjectStats(vfs, '/novel') // User function
const gameStats = await getProjectStats(vfs, '/game') // User function
const webappStats = await getProjectStats(vfs, '/webapp') // User function
console.log('Total files:', novelStats.fileCount + gameStats.fileCount + webappStats.fileCount)
console.log('Total size:', novelStats.totalSize + gameStats.totalSize + webappStats.totalSize)
// Knowledge graph visualization data 🔮 (future feature)
// const knowledgeGraph = await vfs.getGlobalKnowledgeGraph() // Not yet implemented
// Returns nodes (files, entities, concepts) and edges (relationships)
// Find connections between projects 🔮 (future feature)
// const crossProjectLinks = await vfs.findCrossProjectLinks() // Not yet implemented
// Unified timeline ✅
const timeline = await vfs.getTimeline({
from: '2025-01-01',
to: '2025-12-31'
})
await vfs.close()
await brain.close()
}
```
## Advanced Features
### Semantic Code Analysis 📝
These are user functions - see [USER_FUNCTIONS.md](./USER_FUNCTIONS.md) for implementation templates:
```javascript
// Find security vulnerabilities (user function example)
const vulnerabilities = await findPatterns(vfs, [
'eval(',
'innerHTML =',
'SQL injection',
'hardcoded password'
])
// Find code smells (user function example)
const codeSmells = await analyzeCodeQuality(vfs, '/src', {
checkDuplication: true,
checkComplexity: true,
checkNaming: true
})
```
### AI-Powered Features 🔮
**Note:** These features require AI integration and are not yet available.
```javascript
// Future: Generate documentation
// const docs = await vfs.generateDocumentation('/src/auth/login.ts')
// Future: Suggest refactorings
// const refactorings = await vfs.suggestRefactorings('/src/utils.js')
// Future: Auto-complete code
// const completion = await vfs.completeCode('/src/api.ts', { line: 42, column: 10 })
```
### Migration and Backup 🔮
**Note:** These features are planned but not yet implemented.
```javascript
// Future: Backup with full history
// await vfs.createBackup('/backup/2025-01-20.brainy')
// Future: Migrate between storage backends
// const migration = await vfs.migrate({
// from: { type: 'file', path: './old-data' },
// to: { type: 's3', bucket: 'new-bucket' }
// })
// Future: Incremental sync
// await vfs.sync('/local/path', '/vfs/path', {
// bidirectional: true,
// conflictStrategy: 'newest'
// })
```
### Performance at Scale
```javascript
// Handle millions of files ✅
for (let i = 0; i < 1000000; i++) {
await vfs.writeFile(`/data/file${i}.txt`, `Content ${i}`)
// Uses chunking, compression, and efficient indexing
}
// Fast parallel operations ✅
await Promise.all([
vfs.writeFile('/file1.txt', 'data1'),
vfs.writeFile('/file2.txt', 'data2'),
vfs.writeFile('/file3.txt', 'data3')
])
// Bulk write operations ✅
const files = [
{ path: '/data/file1.txt', content: 'Content 1' },
{ path: '/data/file2.txt', content: 'Content 2' },
// ... more files
]
await vfs.bulkWrite(files)
// Bulk imports 🔮 (future feature)
// await vfs.bulkImport('/massive/dataset', {
// parallel: 10,
// batchSize: 1000,
// progress: (count, total) => console.log(`${count}/${total}`)
// })
```
## Implementation Status
### ✅ Fully Implemented Features
All methods marked with ✅ are production-ready:
1. **Core VFS Operations**: mkdir, writeFile, readFile, appendFile, stat, readdir, etc.
2. **Entity System**: createEntity, linkEntities, findEntityOccurrences, updateEntity, getEntityGraph
3. **Concept System**: createConcept, findByConcept
4. **Knowledge Layer**: Event recording, semantic versioning, collaboration tracking
5. **Search**: Triple Intelligence (vector + field + graph)
6. **Git Integration**: importFromGit, exportToGit
7. **Export Formats**: exportToMarkdown, exportToJSON
8. **Bulk Operations**: bulkWrite for efficient batch processing
9. **Project Management**: todos, metadata, relationships
### 📝 User Functions
Methods marked with 📝 are domain-specific functions that you can implement using VFS primitives. See [USER_FUNCTIONS.md](./USER_FUNCTIONS.md) for ready-to-use templates:
- **Code Analysis**: getDependencyGraph, findCircularDependencies, findUntestedCode, findSimilarCode
- **Creative Writing**: trackCharacterArc, generateStoryBible
- **Game Development**: validateGameData, generateLootTables
- **Project Management**: getProjectInsights, generateSprintReport
- **Export Formats**: exportToEpub, exportToStaticSite
### 🔮 Future Features
Methods marked with 🔮 require AI integration or are planned for future releases:
- **AI-Powered**: generateDocumentation, suggestRefactorings, completeCode
- **Advanced Analysis**: detectConflicts, getGlobalKnowledgeGraph, findCrossProjectLinks
- **Migration Tools**: createBackup, migrate, sync, bulkImport
## Real Implementation Guarantees
- **No Mocks**: Every ✅ method is fully functional
- **Real Storage**: Uses Brainy entities with embeddings
- **Real Search**: Triple Intelligence combining vectors, fields, and graphs
- **Real Relationships**: Graph-based connections via brain.relate()
- **Production Ready**: Complete error handling, async/await, resource cleanup
The VFS + Knowledge Layer combination provides a solid foundation for intelligent applications. Use the ✅ methods directly, implement 📝 functions as needed for your domain, and stay tuned for 🔮 features.

4
docs/vfs/VFS_GRAPH_TYPES.md
View file

@ -118,13 +118,13 @@ const results = await brain.search('machine learning', {
### 2. Graph Traversal
```javascript
// Find all entities contained in a directory
const contained = await brain.getRelations({
const contained = await brain.related({
from: directoryId,
type: VerbType.Contains
})
// Find what contains a file (parent directories)
const parents = await brain.getRelations({
const parents = await brain.related({
to: fileId,
type: VerbType.Contains
})