soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase

Browse source
This commit is contained in:
David Snelling 2026-06-11 14:51:00 -07:00
parent 970e08c466
commit 1f7e365a4e
237 changed files with 1951 additions and 49413 deletions
Download patch file Download diff file Expand all files Collapse all files

2
CHANGELOG.md
View file

@ -1538,7 +1538,7 @@ Expected behavior after upgrade:
### 🐛 Bug Fixes
* **storage**: Fix `clear()` not deleting COW version control data ([#workshop-bug-report](https://github.com/soulcraftlabs/brainy/issues))
* **storage**: Fix `clear()` not deleting COW version control data (consumer-reported)
- Fixed all storage adapters to properly delete `_cow/` directory on clear()
- Fixed in-memory entity counters not being reset after clear()
- Prevents COW reinitialization after clear() by setting `cowEnabled = false`

7
README.md
View file

@ -196,9 +196,10 @@ CSV, Excel, PDF, URLs — auto-detected format, auto-classified entities.
```javascript
await brain.import('customers.csv')
await brain.import('sales-data.xlsx', { excelSheets: ['Q1', 'Q2'] })
await brain.import('research-paper.pdf', { pdfExtractTables: true })
await brain.import('sales-data.xlsx') // all sheets processed
await brain.import('research-paper.pdf') // tables extracted automatically
await brain.import('https://api.example.com/data.json')
await brain.import('./handbook.md', { vfsPath: '/imports/handbook' }) // preserve in VFS
```
**[Import Guide](docs/guides/import-anything.md)**
@ -265,7 +266,7 @@ await brain.relate({
// Filter on the fast path — column-store hit, not metadata fallback:
const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
const directReports = await brain.getRelations({ from: ceoId, subtype: 'direct' })
const directReports = await brain.related({ from: ceoId, subtype: 'direct' })
// O(1) counts via the persisted rollups:
brain.counts.bySubtype(NounType.Person)

29
RELEASES.md
View file

@ -106,6 +106,12 @@ Hard renames, no aliases. Every pair below is verified against the 8.0 source.
| Brainy 7.x | Brainy 8.0 |
|---|---|
| `brain.delete(id)` | `brain.remove(id)` — matches the transact op vocabulary (`add` / `update` / `remove` / `relate` / `unrelate`) |
| `brain.deleteMany(params)` | `brain.removeMany(params)` |
| `DeleteManyParams` (type) | `RemoveManyParams` |
| `brain.getRelations(paramsOrId)` | `brain.related(paramsOrId)` — the same name a pinned `Db` exposes (`db.related()`) |
| `GetRelationsParams` (type) | `RelatedParams` |
| CLI `delete <id>` | CLI `remove <id>` (JSON output `{ id, removed: true }`, matching `unrelate`) |
| `HnswProvider` (from `@soulcraft/brainy/plugin`) | `VectorIndexProvider` |
| `HNSWIndex` (exported class) | `JsHnswVectorIndex` |
| Provider keys `'hnsw'` and `'diskann'` | `'vector'` (the only key Brainy consults for the vector index) |
@ -129,9 +135,17 @@ find . -name '*.ts' -not -path '*/node_modules/*' -print0 | xargs -0 sed -i \
-e "s/registerProvider('hnsw'/registerProvider('vector'/g" \
-e "s/registerProvider('diskann'/registerProvider('vector'/g" \
-e "s/getProvider('hnsw'/getProvider('vector'/g" \
-e "s/getProvider('diskann'/getProvider('vector'/g"
-e "s/getProvider('diskann'/getProvider('vector'/g" \
-e 's/\.getRelations(/.related(/g' \
-e 's/\bGetRelationsParams\b/RelatedParams/g' \
-e 's/\.deleteMany(/.removeMany(/g' \
-e 's/\bDeleteManyParams\b/RemoveManyParams/g'
```
`delete()` → `remove()` is deliberately **not** in the snippet: `.delete(` is
too common (`Map`/`Set`/storage adapters) for a blind sed. Find your
`brain.delete(...)` call sites and rename them by hand.
The config shape changes need a human (they are not 1:1 textual):
```ts
@ -192,7 +206,7 @@ this rename; only the API surface moved.
implements the BFS in the JS graph index and routes to a native provider
when one is registered.
- **Every write advances the generation clock; only `transact()` writes
history.** Single-operation writes (`add` / `update` / `delete` / `relate`
history.** Single-operation writes (`add` / `update` / `remove` / `relate`
outside `transact()`) bump `brain.generation()` so watermarks and CAS stay
sound, but they do not stage historical records — they remain visible
through earlier pins and are not reported by `db.since()`. Writes you want
@ -210,13 +224,20 @@ this rename; only the API surface moved.
no-ops, closing a 7.x trap), system-managed fields drop with a one-shot
warning naming the right path; (3) **split at read time** through one
canonical helper, so `entity.metadata` / `relation.metadata` contain ONLY
custom fields on every read path — `get`, `find`, `getRelations` (several
custom fields on every read path — `get`, `find`, `related` (several
paginated/by-source/by-target paths previously echoed the full stored
record, including the `verb` type key, inside `metadata`), batch reads,
and historical `asOf()` reads. `getRelations()` results now also surface
and historical `asOf()` reads. `related()` results now also surface
`confidence`/`updatedAt` top-level, and `updateRelation()` no longer
erases a relationship's `service`/`createdBy`. See "Reserved fields" in
`docs/concepts/consistency-model.md`.
- **Named errors for not-found contract failures.** `update()`, `relate()`,
`updateRelation()`, `similar({ to: id })`, `transact()` planning, and
speculative `db.with()` planning now throw `EntityNotFoundError` /
`RelationNotFoundError` (both exported from the package root, both carrying
the missing `id` as a field) instead of a generic `Error`. Message texts are
unchanged, so existing `/not found/` matching keeps working — `instanceof`
is now the supported way to branch.
- **Unchanged from 7.31:** per-entity `_rev`, `update({ ifRev })`
(`RevisionConflictError`), and `add({ ifAbsent })` work exactly as before,
and `ifRev` is also accepted on `transact()` update operations (a conflict

2
docs/ADR-001-generational-mvcc.md
View file

@ -28,7 +28,7 @@ path and the native path.
- A **monotonic u64 generation counter** is the store's logical clock. It
advances once per committed `transact()` batch and once per
single-operation write (`add`/`update`/`delete`/`relate`/…), so
single-operation write (`add`/`update`/`remove`/`relate`/…), so
`brain.generation()` is always a meaningful watermark. It is persisted in
`_system/generation.json` and never reissued for anything durable.
- `brain.now()` **pins** the current generation in O(1) and returns a `Db` —

2
docs/BATCHING.md
View file

@ -316,7 +316,7 @@ const entities = Array.from(results.values())
```typescript
const allVerbs = []
for (const sourceId of sourceIds) {
const verbs = await brain.getRelations({ from: sourceId })
const verbs = await brain.related({ from: sourceId })
allVerbs.push(...verbs)
}
```

435
docs/CREATING-AUGMENTATIONS.md
View file

@ -1,435 +0,0 @@
# Creating Augmentations for Brainy
> **Updated** - Includes metadata structure changes and type system improvements
## The BrainyAugmentation Interface
Every augmentation implements this simple yet powerful interface:
```typescript
interface BrainyAugmentation {
// Identification
name: string // Unique name for your augmentation
// Execution control
timing: 'before' | 'after' | 'around' | 'replace' // When to execute
operations: string[] // Which operations to intercept
priority: number // Execution order (higher = first)
// Lifecycle methods
initialize(context: AugmentationContext): Promise<void>
execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T>
shutdown?(): Promise<void> // Optional cleanup
}
```
## Breaking Changes for Augmentation Developers
### 1. Metadata Structure Separation
Brainy introduces strict metadata/vector separation for billion-scale performance:
```typescript
// ✅ Metadata has required type field
interface NounMetadata {
noun: NounType // Required! Must be a valid noun type
[key: string]: any // Your custom metadata
}
interface VerbMetadata {
verb: VerbType // Required! Must be a valid verb type
sourceId: string
targetId: string
[key: string]: any
}
```
### 2. Storage Adapter Return Types
Storage adapters now return different types at different boundaries:
```typescript
// Internal methods: Pure structures (no metadata)
abstract _getNoun(id: string): Promise<HNSWNoun | null>
// Public API: WithMetadata structures
abstract getNoun(id: string): Promise<HNSWNounWithMetadata | null>
```
### 3. Verb Property Renamed
The verb relationship field changed from `type` to `verb`:
```typescript
// ❌ v3.x
verb.type === 'relatedTo'
// ✅ Current
verb.verb === 'relatedTo'
```
## Creating a Storage Augmentation
Storage augmentations are special - they provide the storage backend for Brainy.
### Important: Storage Requirements
Your storage adapter MUST:
1. **Wrap metadata** with required `noun`/`verb` fields
2. **Return pure structures** from internal `_methods`
3. **Return WithMetadata types** from public methods
```typescript
import { StorageAugmentation } from 'brainy/augmentations'
import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy'
export class MyCustomStorage extends BaseStorageAdapter {
// Internal method: Returns pure structure
async _getNoun(id: string): Promise<HNSWNoun | null> {
const data = await this.fetchFromDatabase(id)
return data ? {
id: data.id,
vector: data.vector,
nounType: data.type
} : null
}
// Public method: Returns WithMetadata structure
async getNoun(id: string): Promise<HNSWNounWithMetadata | null> {
const noun = await this._getNoun(id)
if (!noun) return null
// Fetch metadata separately
const metadata = await this.getNounMetadata(id)
return {
...noun,
metadata: metadata || { noun: noun.nounType || 'thing' }
}
}
// CRITICAL: Always save with proper metadata structure
async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> {
// Validate metadata has required 'noun' field
if (!metadata?.noun) {
throw new Error('NounMetadata requires "noun" field')
}
await this.database.save({
id: noun.id,
vector: noun.vector,
nounType: noun.nounType,
metadata: metadata // Stored separately
})
}
}
export class MyStorageAugmentation extends StorageAugmentation {
private config: MyStorageConfig
constructor(config: MyStorageConfig) {
super()
this.name = 'my-custom-storage'
this.config = config
}
// Called during storage resolution phase
async provideStorage(): Promise<StorageAdapter> {
const storage = new MyCustomStorage(this.config)
this.storageAdapter = storage
return storage
}
// Called during augmentation initialization
protected async onInitialize(): Promise<void> {
await this.storageAdapter!.init()
this.log(`Custom storage initialized`)
}
}
```
### Using Your Storage Augmentation
```typescript
// Register before brain.init()
const brain = new Brainy()
brain.augmentations.register(new MyStorageAugmentation({
connectionString: 'redis://localhost:6379'
}))
await brain.init() // Will use your storage!
```
## Creating a Feature Augmentation
Here's a complete example of a caching augmentation:
```typescript
import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations'
export class CachingAugmentation extends BaseAugmentation {
private cache = new Map<string, any>()
constructor() {
super()
this.name = 'smart-cache'
this.timing = 'around' // Wrap operations
this.operations = ['search'] // Only cache searches
this.priority = 50 // Mid-priority
}
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
if (operation === 'search') {
// Check cache
const cacheKey = JSON.stringify(params)
if (this.cache.has(cacheKey)) {
this.log('Cache hit!')
return this.cache.get(cacheKey)
}
// Execute and cache
const result = await next()
this.cache.set(cacheKey, result)
return result
}
// Pass through other operations
return next()
}
protected async onInitialize(): Promise<void> {
this.log('Cache initialized')
}
async shutdown(): Promise<void> {
this.cache.clear()
await super.shutdown()
}
}
```
## The Four Timing Modes
### 1. `before` - Pre-processing
```typescript
timing = 'before'
async execute(op, params, next) {
// Validate/transform input
const validated = await validate(params)
return next(validated) // Pass modified params
}
```
### 2. `after` - Post-processing
```typescript
timing = 'after'
async execute(op, params, next) {
const result = await next()
// Log, analyze, or modify result
console.log(`Operation ${op} returned:`, result)
return result
}
```
### 3. `around` - Wrapping (middleware)
```typescript
timing = 'around'
async execute(op, params, next) {
console.log('Starting', op)
try {
const result = await next()
console.log('Success', op)
return result
} catch (error) {
console.log('Failed', op, error)
throw error
}
}
```
### 4. `replace` - Complete replacement
```typescript
timing = 'replace'
async execute(op, params, next) {
// Don't call next() - replace entirely!
return myCustomImplementation(params)
}
```
## Operations You Can Intercept
Common operations in Brainy:
- `'storage'` - Storage resolution (special)
- `'add'` - Adding data
- `'search'`, `'similar'` - Searching
- `'update'`, `'delete'` - Modifications
- `'saveNoun'`, `'saveVerb'` - Storage operations
- `'all'` - Intercept everything
## Context Available to Augmentations
```typescript
interface AugmentationContext {
brain: Brainy // The brain instance
storage: StorageAdapter // Storage backend
config: BrainyConfig // Configuration
log: (message: string, level?: 'info' | 'warn' | 'error') => void
}
```
## Real-World Examples
### 1. Redis Storage Augmentation
```typescript
export class RedisStorageAugmentation extends StorageAugmentation {
async provideStorage(): Promise<StorageAdapter> {
return new RedisAdapter({
host: 'localhost',
port: 6379,
// Implement full StorageAdapter interface
})
}
}
```
### 2. Audit Trail Augmentation
```typescript
export class AuditAugmentation extends BaseAugmentation {
timing = 'after'
operations = ['add', 'update', 'delete']
async execute(op, params, next) {
const result = await next()
// Log to audit trail
await this.logAudit({
operation: op,
params,
result,
timestamp: new Date(),
user: this.context.config.currentUser
})
return result
}
}
```
### 3. Rate Limiting Augmentation
```typescript
export class RateLimitAugmentation extends BaseAugmentation {
timing = 'before'
operations = ['search']
private limiter = new RateLimiter({ rps: 100 })
async execute(op, params, next) {
await this.limiter.acquire() // Wait if rate limited
return next()
}
}
```
## Publishing to Brain Cloud Marketplace
Future capability for premium augmentations:
```typescript
// package.json
{
"name": "@brain-cloud/redis-storage",
"brainy": {
"type": "augmentation",
"category": "storage",
"premium": true
}
}
// Users can install via:
// brainy augment install redis-storage
```
## Best Practices
### General Practices
1. **Use BaseAugmentation** - Provides common functionality
2. **Set appropriate priority** - Storage (100), System (80-99), Features (10-50)
3. **Be selective with operations** - Don't use 'all' unless necessary
4. **Handle errors gracefully** - Don't break the chain
5. **Clean up in shutdown()** - Release resources
6. **Log appropriately** - Use context.log() for consistent output
7. **Document your augmentation** - Include examples
### Specific Best Practices
8. **Always include `noun` field** when creating/modifying NounMetadata:
```typescript
const metadata: NounMetadata = {
noun: 'thing', // REQUIRED!
yourField: 'value'
}
```
9. **Use `verb` property** not `type` when working with relationships:
```typescript
// ✅ Correct
if (verb.verb === 'relatedTo') { ... }
// ❌ Wrong (v3.x pattern)
if (verb.type === 'relatedTo') { ... }
```
10. **Access metadata correctly** from storage:
```typescript
// ✅ Correct - metadata is already structured
const nounType = noun.metadata.noun
// ⚠️ Fallback pattern for robustness
const nounType = noun.metadata?.noun || 'thing'
```
11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations:
```typescript
// ✅ Good - Separate concerns
await storage.saveNoun(noun)
await storage.saveMetadata(noun.id, metadata)
// ❌ Bad - Mixing concerns
await storage.saveNounWithEverything(combinedData)
```
## Testing Your Augmentation
```typescript
import { Brainy } from 'brainy'
import { MyAugmentation } from './my-augmentation'
describe('MyAugmentation', () => {
let brain: Brainy
beforeEach(async () => {
brain = new Brainy()
brain.augmentations.register(new MyAugmentation())
await brain.init()
})
afterEach(async () => {
await brain.destroy()
})
it('should enhance searches', async () => {
// Test your augmentation's effect
const results = await brain.search('test')
expect(results).toHaveProperty('enhanced', true)
})
})
```
## Summary
Augmentations are Brainy's extension system. They can:
- Replace storage backends
- Add caching layers
- Implement audit trails
- Add rate limiting
- Sync with external systems
- Transform data
- And much more!
The unified BrainyAugmentation interface makes it easy to create powerful extensions while maintaining consistency across the entire system.

2
docs/DATA_MODEL.md
View file

@ -238,7 +238,7 @@ await brain.relate({
Fast-path filter on the verb side:
```typescript
const direct = await brain.getRelations({
const direct = await brain.related({
from: ceoId,
type: VerbType.ReportsTo,
subtype: 'direct'

12
docs/DEVELOPER_LEARNING_PATH.md
View file

@ -193,21 +193,21 @@ console.log('✅ Created relationships')
console.log('\n🔍 Querying relationships...')
// Who works for TechCorp?
const techcorpEmployees = await brain.getRelations({
const techcorpEmployees = await brain.related({
to: techcorpId,
type: VerbType.WorksWith
})
console.log(`TechCorp has ${techcorpEmployees.length} employees`)
// Who works on the AI project?
const projectContributors = await brain.getRelations({
const projectContributors = await brain.related({
to: projectId,
type: [VerbType.WorksOn, VerbType.Manages]
})
console.log(`AI Project has ${projectContributors.length} contributors`)
// Who does John collaborate with?
const johnsCollaborators = await brain.getRelations({
const johnsCollaborators = await brain.related({
from: johnId,
type: VerbType.CollaboratesWith
})
@ -294,7 +294,7 @@ await brain.update({
1. Create an organizational hierarchy (CEO → Managers → Engineers)
2. Build a project dependency graph
3. Model a social network with CollaboratesWith relationships
4. Query "Who reports to Alice?" using getRelations()
4. Query "Who reports to Alice?" using related()
5. Batch update all projects to add a "year: 2024" field
### Next Steps
@ -676,7 +676,7 @@ if (readmeEntity.length > 0) {
}
// Query relationships
const conceptDocs = await brain.getRelations({
const conceptDocs = await brain.related({
from: conceptId,
type: VerbType.DocumentedBy
})
@ -798,7 +798,7 @@ await brain.relate({
})
// Query across boundaries
const conceptDocs = await brain.getRelations({
const conceptDocs = await brain.related({
from: conceptId,
type: VerbType.DocumentedBy
})

2
docs/FIND_SYSTEM.md
View file

@ -1228,7 +1228,7 @@ where: { age: { gt: 18 } }
**No connected entities found**:
```typescript
// Verify relationship exists
const relations = await brain.getRelations({
const relations = await brain.related({
from: 'entity-a',
to: 'entity-b'
})

2
docs/MIGRATION-V3-TO-V4.md
View file

@ -48,7 +48,7 @@ await storage.setLifecyclePolicy({
```typescript
// v3: Delete one at a time (slow, expensive)
for (const id of idsToDelete) {
await brain.delete(id) // 1000 API calls for 1000 entities
await brain.remove(id) // 1000 API calls for 1000 entities
}
// v4.0.0: Batch delete (fast, cheap)

6
docs/QUERY_OPERATORS.md
View file

@ -216,18 +216,18 @@ See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the f
### Filter relationships by subtype (7.30+)
Verbs are first-class peers — `getRelations()` and graph traversal both honor subtype filters on the fast path:
Verbs are first-class peers — `related()` and graph traversal both honor subtype filters on the fast path:
```typescript
// Filter relationships by VerbType subtype
const direct = await brain.getRelations({
const direct = await brain.related({
from: ceoId,
type: VerbType.ReportsTo,
subtype: 'direct'
})
// Set membership on verb subtype
const all = await brain.getRelations({
const all = await brain.related({
from: ceoId,
type: VerbType.ReportsTo,
subtype: ['direct', 'dotted-line']

7
docs/README.md
View file

@ -89,14 +89,11 @@ See [vfs/](./vfs/) for the complete VFS documentation set.
---
## Plugins & Augmentations
## Plugins
| Document | Description |
|----------|-------------|
| [Plugins](./PLUGINS.md) | Plugin system overview |
| [Creating Augmentations](./CREATING-AUGMENTATIONS.md) | Build custom plugins |
| [Augmentations Reference](./augmentations/COMPLETE-REFERENCE.md) | Full augmentation API |
| [Augmentations Developer Guide](./augmentations/DEVELOPER-GUIDE.md) | Plugin development guide |
| [Plugins](./PLUGINS.md) | Plugin system overview — providers, `plugins` config, `brain.use()` |
---

249
docs/api/README.md
View file

@ -185,12 +185,12 @@ await brain.update({
---
### `delete(id)` → `Promise<void>`
### `remove(id)` → `Promise<void>`
Delete a single entity.
Remove a single entity (and every relationship where it is source or target).
```typescript
await brain.delete(id)
await brain.remove(id)
```
**Parameters:**
@ -444,7 +444,7 @@ brain.defineAggregate({
name: 'monthly_spending',
source: {
type: NounType.Event,
where: { domain: 'financial', subtype: 'transaction' }
where: { domain: 'financial' } // matches custom metadata fields
},
groupBy: [
'category',
@ -468,10 +468,10 @@ brain.defineAggregate({
|-------|------|-------------|
| `name` | `string` | Unique identifier for this aggregate |
| `source.type` | `NounType \| NounType[]` | Entity types that feed into this aggregate |
| `source.where` | `Record<string, unknown>` | Metadata filter (same syntax as `find({ where })`) |
| `source.where` | `Record<string, unknown>` | Filter on custom **metadata** fields (matched against the entity's `metadata` bag) |
| `source.service` | `string` | Multi-tenancy filter |
| `groupBy` | `GroupByDimension[]` | Dimensions to group by — plain field names or `{ field, window }` for time bucketing |
| `metrics` | `Record<string, AggregateMetricDef>` | Named metrics with `op` (`sum`, `count`, `avg`, `min`, `max`, `stddev`, `variance`) and optional `field` |
| `groupBy` | `GroupByDimension[]` | Dimensions to group by — plain field names, `{ field, window }` for time bucketing, or `{ field, unnest: true }` for array fields (one contribution per element) |
| `metrics` | `Record<string, AggregateMetricDef>` | Named metrics with `op` (`sum`, `count`, `avg`, `min`, `max`, `stddev`, `variance`, `percentile`, `distinctCount`) and optional `field`. `percentile` additionally requires `p` in `[0, 1]`. |
| `materialize` | `boolean \| object` | Write results as `NounType.Measurement` entities (auto-visible in OData/Sheets/SSE) |
**Time window granularities:** `'hour'`, `'day'`, `'week'`, `'month'`, `'quarter'`, `'year'`, or `{ seconds: number }` for custom intervals.
@ -539,6 +539,20 @@ const recentFood = await brain.find({
}
```
### `queryAggregate(name, params?)` → `Promise<AggregateResult[]>`
The first-class analytics path — returns plain group rows (`{ groupKey, metrics, count }`) instead of `find()`-style `Result` wrappers. Supports `where` (group-key filter), `having` (SQL-HAVING metric filter), `orderBy`, `order`, `limit`, `offset`:
```typescript
const rows = await brain.queryAggregate('monthly_spending', {
having: { total: { greaterThan: 100 } }, // filter by computed metrics
orderBy: 'total',
order: 'desc',
limit: 10
})
// [{ groupKey: { category: 'food', date: '2024-01' }, metrics: { total: 342.5, count: 28 }, count: 28 }, ...]
```
### How It Works
Aggregation hooks run **outside transactions** on every write operation:
@ -557,16 +571,16 @@ Aggregation hooks run **outside transactions** on every write operation:
### Financial Data Modeling
Brainy supports financial analytics through **metadata conventions** on existing NounTypes — no custom types needed:
Brainy supports financial analytics through **subtypes and metadata conventions** on existing NounTypes — no custom types needed:
```typescript
// Transaction = NounType.Event + financial metadata
// Transaction = NounType.Event + 'transaction' subtype + financial metadata
await brain.add({
data: 'Coffee at Blue Bottle',
type: NounType.Event,
subtype: 'transaction', // top-level standard field (reserved — never in metadata)
metadata: {
domain: 'financial',
subtype: 'transaction',
amount: 5.50,
currency: 'USD',
category: 'food',
@ -575,26 +589,26 @@ await brain.add({
}
})
// Account = NounType.Collection + financial metadata
// Account = NounType.Collection + 'account' subtype + financial metadata
await brain.add({
data: 'Checking Account',
type: NounType.Collection,
subtype: 'account',
metadata: {
domain: 'financial',
subtype: 'account',
accountType: 'checking',
currency: 'USD',
institution: 'Chase'
}
})
// Invoice = NounType.Document + financial metadata
// Invoice = NounType.Document + 'invoice' subtype + financial metadata
await brain.add({
data: 'Invoice #1234 from Acme Corp',
type: NounType.Document,
subtype: 'invoice',
metadata: {
domain: 'financial',
subtype: 'invoice',
amount: 15000,
currency: 'USD',
status: 'pending',
@ -672,32 +686,33 @@ await brain.updateRelation({ id: relId, type: VerbType.WorksWith })
---
### `getRelations(params)` → `Promise<Relation[]>`
### `related(params)` → `Promise<Relation[]>`
Get relationships for an entity.
Get relationships for an entity. Same name and surface as `db.related()` on a
pinned `Db` view.
```typescript
// Get all relationships FROM an entity
const outgoing = await brain.getRelations({ from: entityId })
const outgoing = await brain.related({ from: entityId })
// Get all relationships TO an entity
const incoming = await brain.getRelations({ to: entityId })
const incoming = await brain.related({ to: entityId })
// Filter by type
const related = await brain.getRelations({
const contains = await brain.related({
from: entityId,
type: VerbType.Contains
})
// Filter by subtype (fast path, column-store hit)
const direct = await brain.getRelations({
const direct = await brain.related({
from: entityId,
type: VerbType.ReportsTo,
subtype: 'direct'
})
// Set membership on subtype
const all = await brain.getRelations({
const all = await brain.related({
from: entityId,
type: VerbType.ReportsTo,
subtype: ['direct', 'dotted-line']
@ -739,12 +754,12 @@ console.log(result.failed) // Array of errors
---
### `deleteMany(params)` → `Promise<BatchResult<string>>`
### `removeMany(params)` → `Promise<BatchResult<string>>`
Delete multiple entities.
Remove multiple entities.
```typescript
const result = await brain.deleteMany({
const result = await brain.removeMany({
ids: [id1, id2, id3]
})
```
@ -757,7 +772,7 @@ Update multiple entities.
```typescript
const result = await brain.updateMany({
updates: [
items: [
{ id: id1, metadata: { updated: true } },
{ id: id2, data: 'New content' }
]
@ -772,7 +787,7 @@ Create multiple relationships.
```typescript
const ids = await brain.relateMany({
relations: [
items: [
{ from: id1, to: id2, type: VerbType.RelatedTo },
{ from: id1, to: id3, type: VerbType.Contains }
]
@ -1213,39 +1228,16 @@ const todos = await brain.vfs.getTodos('/src/App.tsx')
---
#### `vfs.getAllTodos(path?)` → `Promise<Todo[]>`
#### `vfs.searchEntities(query)` → `Promise<Array<{ id, path, type, metadata }>>`
Get all TODOs from directory tree.
Search for semantic entities tracked by the VFS, filtered by type, name, or metadata.
```typescript
const allTodos = await brain.vfs.getAllTodos('/src')
```
---
### Project Analysis
#### `vfs.getProjectStats(path?)` → `Promise<Stats>`
Get project statistics.
```typescript
const stats = await brain.vfs.getProjectStats('/projects/my-app')
console.log(stats.fileCount)
console.log(stats.totalSize)
console.log(stats.fileTypes) // Breakdown by extension
```
---
#### `vfs.searchEntities(query)` → `Promise<VFSEntity[]>`
Search for VFS entities by metadata.
```typescript
const tsxFiles = await brain.vfs.searchEntities({
type: 'file',
extension: '.tsx'
const people = await brain.vfs.searchEntities({
type: 'person', // entity type filter
name: 'Ada', // semantic name search
where: { role: 'author' }, // metadata filters
limit: 50
})
```
@ -1280,48 +1272,52 @@ console.log(result.explanation)
---
### `neural().clusters(input?, options?)` → `Promise<Cluster[]>`
### `neural().clusters(input?, options?)` → `Promise<SemanticCluster[]>`
Automatic clustering.
Automatic clustering. Accepts a text query, an array of texts, or a `ClusteringOptions` object.
```typescript
const clusters = await brain.neural().clusters({
algorithm: 'kmeans',
k: 5,
minSize: 3
algorithm: 'kmeans', // 'auto' | 'hierarchical' | 'kmeans' | 'dbscan' | ...
maxClusters: 5,
minClusterSize: 3
})
clusters.forEach(cluster => {
console.log(cluster.label)
console.log(cluster.items)
console.log(cluster.centroid)
console.log(cluster.label) // auto-generated label (when available)
console.log(cluster.members) // entity ids in the cluster
console.log(cluster.centroid) // centroid vector
})
```
---
### `neural().neighbors(id, options?)` → `Promise<Neighbor[]>`
### `neural().neighbors(id, options?)` → `Promise<NeighborsResult>`
Find k-nearest neighbors.
Find nearest neighbors.
```typescript
const neighbors = await brain.neural().neighbors(entityId, {
k: 10,
threshold: 0.7
const result = await brain.neural().neighbors(entityId, {
limit: 10,
minSimilarity: 0.7
})
```
**Options:** `limit?`, `radius?`, `minSimilarity?`, `includeMetadata?`, `sortBy?: 'similarity' | 'importance' | 'recency'`
---
### `neural().outliers(threshold?)` → `Promise<string[]>`
### `neural().outliers(options?)` → `Promise<Outlier[]>`
Detect outlier entities.
```typescript
const outliers = await brain.neural().outliers(0.3)
// Returns entity IDs that are outliers
const outliers = await brain.neural().outliers({ threshold: 0.3 })
// Each outlier carries the entity id plus outlier scoring details
```
**Options:** `threshold?`, `method?: 'isolation' | 'statistical' | 'cluster-based'`, `minNeighbors?`, `includeReasons?`
---
### `neural().visualize(options?)` → `Promise<VizData>`
@ -1342,27 +1338,28 @@ const vizData = await brain.neural().visualize({
### Performance Methods
#### `neural().clusterFast(options)` → `Promise<Cluster[]>`
#### `neural().clusterFast(options?)` → `Promise<SemanticCluster[]>`
Fast clustering for large datasets.
```typescript
const clusters = await brain.neural().clusterFast({
k: 10,
maxIterations: 50
maxClusters: 10
})
```
**Options:** `level?` (hierarchy level), `maxClusters?`
---
#### `neural().clusterLarge(options)` → `Promise<Cluster[]>`
#### `neural().clusterLarge(options?)` → `Promise<SemanticCluster[]>`
Streaming clustering for very large datasets.
Sampling-based clustering for very large datasets.
```typescript
const clusters = await brain.neural().clusterLarge({
k: 20,
batchSize: 1000
sampleSize: 1000,
strategy: 'diverse' // 'random' | 'diverse' | 'recent'
})
```
@ -1381,17 +1378,15 @@ await brain.import('data.csv', {
createEntities: true
})
// Excel import
// Excel import (all sheets processed automatically)
await brain.import('sales.xlsx', {
format: 'excel',
sheets: ['Q1', 'Q2']
vfsPath: '/imports/sales', // optional: mirror into the VFS
groupBy: 'sheet'
})
// PDF import
await brain.import('research.pdf', {
format: 'pdf',
extractTables: true
})
// PDF import (tables extracted automatically)
await brain.import('research.pdf', { format: 'pdf' })
// URL import
await brain.import('https://api.example.com/data.json')
@ -1400,10 +1395,17 @@ await brain.import('https://api.example.com/data.json')
**Parameters:**
- `source`: `string | Buffer | object` - File path, URL, buffer, or object
- `options?`: Import configuration
- `format?`: `'csv' | 'excel' | 'pdf' | 'json'` - Auto-detected if omitted
- `format?`: `'excel' | 'pdf' | 'csv' | 'json' | 'markdown' | 'yaml' | 'docx' | 'image'` - Auto-detected if omitted
- `vfsPath?`: `string` - Mirror imported content into the VFS at this path
- `groupBy?`: `'type' | 'sheet' | 'flat' | 'custom'` - VFS grouping strategy
- `createEntities?`: `boolean` - Create entities from rows
- `sheets?`: `string[]` - Excel sheets to import
- `extractTables?`: `boolean` - Extract tables from PDF
- `createRelationships?`: `boolean` - Create relationships between extracted entities
- `preserveSource?`: `boolean` - Save the original file in the VFS
- `enableNeuralExtraction?`: `boolean` - Extract entity names via AI
- `enableRelationshipInference?`: `boolean` - Infer relationships via AI
- `enableConceptExtraction?`: `boolean` - Extract entity types via AI
- `confidenceThreshold?`: `number` - Minimum confidence for extracted entities
- `onProgress?`: `(progress) => void` - Progress callback (stage, counts, throughput, ETA)
**Returns:** `Promise<ImportResult>` - Import statistics
@ -1414,9 +1416,6 @@ await brain.import('https://api.example.com/data.json')
### Export & Snapshots
```typescript
// Export to file
await brain.export('/path/to/backup.brainy')
// Instant hard-link snapshot via the Db API
const pin = brain.now()
await pin.persist('/backups/2026-06-11')
@ -1453,12 +1452,11 @@ const brain = new Brainy({
// Model: all-MiniLM-L6-v2 (384 dimensions)
// Device: CPU via WASM (works everywhere)
// Cache configuration
cache: {
enabled: true,
maxSize: 10000,
ttl: 3600000 // 1 hour in ms
}
// Cache configuration — `true`/`false`, or an options object
cache: {
maxSize: 10000,
ttl: 3600000 // 1 hour in ms
}
})
await brain.init() // Required! VFS auto-initialized
@ -1672,7 +1670,7 @@ When strict mode is on:
- Brainy's own VFS infrastructure writes bypass via the `metadata.isVFSEntity: true` marker.
- Per-type registrations always apply regardless of the brain-wide flag.
Becomes the default in 8.0.0.
The default since 8.0.0 — pass `requireSubtype: false` to opt out while migrating pre-8.0 data.
#### `trackField(name, options?)` → `void`
@ -1718,6 +1716,27 @@ await brain.migrateField({
Returns `{ scanned: number, migrated: number, skipped: number, errors: Array<{id, error}> }`.
#### `fillSubtypes(rules, options?)` → `Promise<FillSubtypesResult>` (8.0+)
Back-fill missing `subtype` values across entities AND relationships in one streaming pass — the migration companion to `audit()`. Keys are `NounType`/`VerbType` values; each rule is a literal subtype string or a function deriving one from the entry (return `undefined` to decline). Idempotent: entries that already carry a subtype are never touched, so a crashed run is resumed safely by re-running.
```typescript
const report = await brain.fillSubtypes({
[NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified', // derived
[NounType.Document]: 'general', // literal default
[VerbType.RelatedTo]: 'unspecified' // relationship rule
})
// → { scanned, filled, skipped, errors, byType }
```
**Parameters:**
- `rules`: `FillSubtypeRules` - Map of NounType/VerbType → literal subtype or `(entry) => string | undefined`
- `options.includeVFS?`: `boolean` - Also fill VFS infrastructure entries (default `false`)
- `options.batchSize?`: `number` - Pagination batch size (default `200`)
- `options.onProgress?`: `(progress: { scanned, filled, skipped }) => void` - Per-batch callback
Returns `{ scanned, filled, skipped, errors, byType }`. After a clean run, `skipped` equals the remaining `audit().total`. See the [migration recipe](../guides/subtypes-and-facets.md).
---
### `embed(data)` → `Promise<number[]>` ✨
@ -1812,7 +1831,7 @@ for (const group of duplicates) {
// Find person duplicates with higher threshold
const personDupes = await brain.findDuplicates({
type: NounType.PERSON,
type: NounType.Person,
threshold: 0.9,
limit: 50
})
@ -1885,15 +1904,19 @@ const docClusters = await brain.cluster({
---
### `getStats()` → `Statistics`
### `getStats(options?)` → `Promise<Statistics>`
Get comprehensive statistics.
Get complete entity/relationship statistics (convenience wrapper over `brain.counts`).
```typescript
const stats = brain.getStats()
console.log(stats.entityCount)
console.log(stats.relationshipCount)
console.log(stats.cacheHitRate)
const stats = await brain.getStats()
console.log(stats.entities.total) // total entity count
console.log(stats.entities.byType) // counts per NounType
console.log(stats.relationships) // relationship stats
console.log(stats.density) // relationships per entity
// Exclude VFS infrastructure entities from the counts
const semanticOnly = await brain.getStats({ excludeVFS: true })
```
---
@ -1914,7 +1937,7 @@ VFS is auto-initialized during `brain.init()` - no separate `vfs.init()` needed!
### Shutdown
```typescript
await brain.shutdown() // Graceful shutdown, flush caches
await brain.close() // Graceful shutdown — flushes pending writes and releases the writer lock
```
---
@ -1940,8 +1963,8 @@ await brain.update({
metadata: { updated: true }
})
// Delete
await brain.delete(id)
// Remove
await brain.remove(id)
```
---
@ -2001,7 +2024,7 @@ const results = await brain.find({
// Speculate: what would this change look like? (nothing touches disk)
const base = brain.now()
const whatIf = await base.with([
{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature' }
{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature', metadata: { draft: true } }
])
await whatIf.find({ where: { draft: true } })
await whatIf.release()
@ -2009,7 +2032,7 @@ await base.release()
// Commit it for real — one atomic generation, with audit metadata
await brain.transact(
[{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature' }],
[{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature', metadata: { draft: true } }],
{ meta: { author: 'dev@example.com', message: 'Add new feature' } }
)
```
@ -2115,7 +2138,8 @@ For the full taxonomy with all 169 types and their descriptions, see:
- **[Stage 3 CANONICAL Taxonomy](../STAGE3-CANONICAL-TAXONOMY.md)** - Complete list with categories
- **[Noun-Verb Taxonomy Architecture](../architecture/noun-verb-taxonomy.md)** - Design rationale
### Migration from
### Migration from pre-Stage-3 taxonomies
**Breaking Changes:**
- `NounType.Content` removed → Use `Document`, `Message`, or `InformationContent`
- `NounType.User` removed → Use `Person` or `Agent`
@ -2161,7 +2185,6 @@ For the full taxonomy with all 169 types and their descriptions, see:
- **[Find System](../FIND_SYSTEM.md)** - Natural language find() details
- **[VFS Quick Start](../vfs/QUICK_START.md)** - Complete VFS documentation
- **[Import Anything Guide](../guides/import-anything.md)** - CSV, Excel, PDF, URL imports
- **[Cloud Deployment](../deployment/CLOUD_DEPLOYMENT_GUIDE.md)** - Production deployment
- **[Consistency Model](../concepts/consistency-model.md)** - The guarantees behind the Db API
- **[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md)** - Backup, restore, what-if, audit recipes

2
docs/architecture/aggregation.md
View file

@ -97,7 +97,7 @@ On `brain.update(entity)`, the engine reverses the old entity's contribution and
### Delete Handling
On `brain.delete(id)`, the engine reverses the entity's contribution:
On `brain.remove(id)`, the engine reverses the entity's contribution:
- `count` and `sum` are decremented
- `min`/`max` may become stale (marked in `staleMinMax` for lazy recompute)

359
docs/architecture/augmentation-system-audit.md
View file

@ -1,359 +0,0 @@
# 🔍 Brainy 2.0 Augmentation System Architecture Audit (REVISED)
**Author**: Senior Architecture Review
**Date**: 2025-08-25
**Status**: 🟡 **WORKING BUT NEEDS MARKETPLACE FEATURES**
## Executive Summary
The augmentation system core execution is WORKING correctly through `AugmentationRegistry`. The system properly executes augmentations before/after operations. However, there's no discovery, installation, or marketplace integration for the brain-cloud registry vision.
---
## 🟢 What's Actually Working
### 1. Execution Mechanism ✅
The `AugmentationRegistry` class properly implements:
```typescript
async execute<T>(operation: string, params: any, mainOperation: () => Promise<T>): Promise<T>
```
- Chains augmentations correctly
- Respects timing (before/after/around)
- Handles operation filtering
- Works with all 27 augmentations
### 2. Registration System ✅
```typescript
brain.augmentations.register(augmentation)
```
- Two-phase initialization works (storage first)
- Context injection works
- Lifecycle management works
### 3. Clean Interface ✅
- 100% of augmentations use `BrainyAugmentation`
- `BaseAugmentation` provides solid foundation
- Proper TypeScript types
### 4. Auto-Configuration ✅
```typescript
new Brainy({
cache: true, // Auto-registers CacheAugmentation
index: true, // Auto-registers IndexAugmentation
storage: { type: 'filesystem', rootDirectory: './data' } // Auto-registers FileSystemStorageAugmentation
})
```
---
## 🟡 Missing for Marketplace Vision
### 1. No Package Discovery
**Current**: Manual registration only
```typescript
// Current (manual)
import { NotionSynapse } from './my-custom-synapse'
brain.augmentations.register(new NotionSynapse())
// Needed
await brain.discover('notion') // Search npm/brain-cloud
await brain.install('@soulcraft/notion-synapse')
```
### 2. No Installation Mechanism
**Current**: Must be bundled at build time
**Needed**: Dynamic installation
```typescript
interface AugmentationMarketplace {
search(query: string): Promise<Package[]>
install(packageId: string): Promise<void>
uninstall(packageId: string): Promise<void>
listInstalled(): Promise<Package[]>
checkUpdates(): Promise<Update[]>
}
```
### 3. No Brain Cloud Registry Client
**Current**: No registry concept
**Needed**: Registry integration
```typescript
class BrainCloudRegistry {
private apiUrl = 'https://api.soulcraft.com/brain-cloud'
async search(query: string): Promise<AugmentationPackage[]> {
const response = await fetch(`${this.apiUrl}/augmentations/search?q=${query}`)
return response.json()
}
async getPackage(id: string): Promise<AugmentationPackage> {
const response = await fetch(`${this.apiUrl}/augmentations/${id}`)
return response.json()
}
}
```
### 4. No License Management
**Current**: All augmentations free/bundled
**Needed**: License verification
```typescript
interface LicenseManager {
verify(packageId: string, licenseKey: string): Promise<boolean>
activate(packageId: string, licenseKey: string): Promise<void>
deactivate(packageId: string): Promise<void>
getStatus(packageId: string): Promise<LicenseStatus>
}
```
### 5. No Version Management
**Current**: No versioning
**Needed**: Semver support
```typescript
interface VersionManager {
checkCompatibility(pkg: Package, brainyVersion: string): boolean
resolveConflicts(packages: Package[]): Package[]
upgrade(packageId: string, toVersion: string): Promise<void>
}
```
---
## 📋 Implementation Plan for Marketplace
### Phase 1: Local Package Discovery (1 week)
```typescript
class LocalPackageDiscovery {
async discover(): Promise<Package[]> {
// 1. Search node_modules for brainy augmentations
const packages = await glob('node_modules/@*/package.json')
// 2. Filter for brainy augmentations
return packages.filter(pkg => pkg.brainy?.type === 'augmentation')
}
async load(packageId: string): Promise<BrainyAugmentation> {
// Dynamic import
const module = await import(packageId)
return new module.default()
}
}
```
### Phase 2: NPM Integration (1 week)
```typescript
class NPMRegistry {
async search(query: string): Promise<Package[]> {
// Search npm for packages with brainy keyword
const response = await fetch(
`https://registry.npmjs.org/-/v1/search?text=${query}+keywords:brainy-augmentation`
)
return response.json()
}
async install(packageId: string): Promise<void> {
// Use npm programmatically
await exec(`npm install ${packageId}`)
// Auto-register after install
const aug = await this.load(packageId)
this.brain.augmentations.register(aug)
}
}
```
### Phase 3: Brain Cloud Registry (2 weeks)
```typescript
class BrainCloudMarketplace {
private registry = new BrainCloudRegistry()
private licenses = new LicenseManager()
private installer = new AugmentationInstaller()
async browse(category?: string): Promise<MarketplaceListing[]> {
const packages = await this.registry.list(category)
return packages.map(pkg => ({
...pkg,
installed: this.isInstalled(pkg.id),
licensed: this.isLicensed(pkg.id),
updates: this.hasUpdates(pkg.id)
}))
}
async purchase(packageId: string): Promise<void> {
// 1. Process payment
const license = await this.processPayment(packageId)
// 2. Activate license
await this.licenses.activate(packageId, license)
// 3. Install package
await this.install(packageId)
}
}
```
### Phase 4: Developer Tools (1 week)
```typescript
// CLI for augmentation development
class AugmentationCLI {
async create(name: string): Promise<void> {
// Scaffold new augmentation project
await this.scaffold(name, 'augmentation-template')
}
async test(path: string): Promise<void> {
// Test augmentation locally
const aug = await this.load(path)
await this.runTests(aug)
}
async publish(path: string): Promise<void> {
// Publish to brain-cloud
const pkg = await this.package(path)
await this.registry.publish(pkg)
}
}
```
---
## 🏗️ Recommended Architecture
### 1. Augmentation Package Structure
```json
{
"name": "@soulcraft/notion-synapse",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"brainy": {
"type": "augmentation",
"class": "NotionSynapse",
"timing": "after",
"operations": ["addNoun", "updateNoun"],
"priority": 20,
"license": "premium",
"price": 9.99,
"compatibility": ">=2.0.0",
"dependencies": []
},
"keywords": ["brainy-augmentation", "notion", "sync"]
}
```
### 2. Installation Flow
```typescript
// User flow
await brain.marketplace.search('notion')
// Returns: [@soulcraft/notion-synapse, @community/notion-sync, ...]
await brain.marketplace.install('@soulcraft/notion-synapse')
// 1. Check license (prompt for purchase if needed)
// 2. Check compatibility
// 3. Install dependencies
// 4. Download package
// 5. Load augmentation
// 6. Register with brain
// 7. Initialize
// Now it's working!
brain.augmentations.list()
// [..., { name: '@soulcraft/notion-synapse', enabled: true }]
```
### 3. Discovery UI
```typescript
// Web UI component
<AugmentationMarketplace>
<SearchBar />
<Categories>
<Category name="Storage" count={12} />
<Category name="Sync" count={8} />
<Category name="AI" count={15} />
</Categories>
<Results>
<AugmentationCard
name="Notion Synapse"
author="Soulcraft"
rating={4.8}
installs={1200}
price={9.99}
onInstall={...}
/>
</Results>
</AugmentationMarketplace>
```
---
## 🎯 Priority for 2.0 Release
### Must Have (Release Blockers)
- ✅ Working execution (DONE)
- ✅ Clean interface (DONE)
- ✅ Documentation (DONE)
- ⏳ Fix augmentationPipeline.ts removal
- ⏳ Test all 27 augmentations work
### Nice to Have (2.0.x)
- Local package discovery
- NPM integration
- Basic CLI tools
### Future (2.1+)
- Brain Cloud Registry
- License management
- Payment processing
- Marketplace UI
- Developer portal
---
## 📊 Current State Assessment
| Component | Status | Notes |
|-----------|--------|-------|
| Core Execution | ✅ Working | AugmentationRegistry.execute() works |
| Registration | ✅ Working | Manual registration works |
| Auto-Config | ✅ Working | Cache, index, storage auto-register |
| Lifecycle | ✅ Working | Init, execute, shutdown work |
| Discovery | ❌ Missing | No package discovery |
| Installation | ❌ Missing | No dynamic installation |
| Marketplace | ❌ Missing | No registry client |
| Licensing | ❌ Missing | No license management |
| Versioning | ❌ Missing | No version checks |
---
## 💡 Recommendations
### For 2.0 Release
1. **Ship with manual registration** - It works!
2. **Document how to create augmentations** - Critical for adoption
3. **Create 2-3 example augmentations** - Show the patterns
4. **Add basic CLI for testing** - Help developers
### For 2.1 (Q1 2025)
1. **Add NPM discovery** - Find installed augmentations
2. **Dynamic loading** - Import augmentations at runtime
3. **Basic marketplace API** - List available augmentations
4. **Version checking** - Ensure compatibility
### For 3.0 (Q2 2025)
1. **Full marketplace** - Browse, search, install
2. **Payment integration** - Premium augmentations
3. **Developer portal** - Publish augmentations
4. **Enterprise features** - Private registries
---
## ✅ Good News Summary
The augmentation system WORKS! The core architecture is solid:
- Execution mechanism is correct
- Registration works
- Lifecycle management works
- All 27 augmentations function properly
What's missing is the marketplace/discovery layer, which can be added incrementally without breaking the core system. The 2.0 release can ship with manual augmentation registration, and the marketplace features can be added in 2.1+.
**Recommendation: Ship 2.0 with current system, add marketplace in 2.1**

2
docs/architecture/data-storage-architecture.md
View file

@ -195,7 +195,7 @@ tx-log line is appended last (advisory). Crash recovery on open discards any
Two write classes share the generation clock:
- **Single-operation writes** (`add`/`update`/`delete`/`relate` outside
- **Single-operation writes** (`add`/`update`/`remove`/`relate` outside
`transact()`) bump `generation.json` so watermarks and `_rev` CAS stay sound,
but write **no** history — they are not visible to `db.since()` and remain
visible through earlier pins.

41
docs/architecture/storage-architecture.md
View file

@ -239,37 +239,30 @@ await brain.storage.withLock('resource-id', async () => {
## Migration and Backup
### Export Data
Backup and restore go through the Db API — see
[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) for the full
recipe book.
### Snapshot (backup)
```typescript
// Export entire database
const backup = await brain.export({
format: 'json',
includeVectors: true,
includeIndexes: false
})
// Instant, self-contained snapshot (hard links on filesystem storage)
const db = brain.now()
await db.persist('/backups/2026-06-11')
await db.release()
```
### Import Data
### Restore
```typescript
// Import from backup
await brain.import(backup, {
mode: 'merge', // or 'replace'
validateSchema: true
})
// Replace the store's entire state from a snapshot (destructive — confirm required)
await brain.restore('/backups/2026-06-11', { confirm: true })
```
### Storage Migration
### Move to a new directory
```typescript
// Migrate between storage types
const oldBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './old' } })
const newBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './new' } })
await oldBrain.init()
await newBrain.init()
// Transfer all data
const data = await oldBrain.export()
await newBrain.import(data)
// A snapshot directory is a complete store: restore it into a fresh brain
const brain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './new' } })
await brain.init()
await brain.restore('/backups/2026-06-11', { confirm: true })
```
## Performance Tuning

12
docs/architecture/zero-config.md
View file

@ -1,11 +1,11 @@
---
title: Zero Configuration
slug: concepts/zero-config
public: true
public: false
category: concepts
template: concept
order: 3
description: Brainy auto-detects storage, initializes embeddings, and builds indexes — no configuration required. Works in Node.js, Bun, OPFS, and cloud environments.
description: Brainy auto-detects storage, initializes embeddings, and builds indexes — no configuration required. Works in Node.js and Bun (server-only since 8.0).
next:
- getting-started/installation
- guides/storage-adapters
@ -13,7 +13,13 @@ next:
# Zero Configuration & Auto-Adaptation
> **Current Status**: Basic zero-config is fully functional. Advanced auto-adaptation features are in development.
> **Status (8.0):** This document predates Brainy 8.0 and needs a rewrite before
> republication. Large parts describe storage backends and environments that 8.0
> removed (browser/OPFS/IndexedDB, edge KV, S3) or features that were never built
> (model auto-selection, workload detection). Brainy 8.0 is server-only (Node.js/Bun)
> with two storage adapters: `memory` and `filesystem` — see
> [Storage Adapters](../concepts/storage-adapters.md) for the accurate story.
> Basic zero-config (`new Brainy()` with auto-selected storage) works as described.
## Overview

396
docs/augmentations/COMPLETE-REFERENCE.md
View file

@ -1,396 +0,0 @@
# 🔌 Brainy Augmentations Complete Reference
> **All augmentations that power Brainy's extensibility - with locations, usage, and examples**
>
> **⚠️ Update**: Updated for metadata structure changes and billion-scale optimizations
## Quick Start
```typescript
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy({
// Augmentations auto-configure based on environment
storage: { type: 'auto', rootDirectory: './brainy-data' }, // Storage augmentation
cache: true, // Cache augmentation
index: true // Index augmentation
})
await brain.init() // Augmentations initialize automatically
```
## Augmentation Architecture
### Key Improvements for Billion-Scale Performance
1. **Metadata/Vector Separation**: Augmentations now work with separated metadata and vectors
- Metadata stored separately from vector data
- 99.2% memory reduction for type tracking
- Two-file storage pattern for optimal I/O
2. **Type System Enforcement**: All metadata requires type fields
- `NounMetadata` requires `noun: NounType`
- `VerbMetadata` requires `verb: VerbType`
- Type inference system available as public API
3. **Storage Adapter Pattern**: Internal vs public method distinction
- `_methods`: Return pure structures (HNSWNoun, HNSWVerb)
- Public methods: Return WithMetadata types
- MetadataEnforcer Proxy ensures proper access
### What This Means for Augmentation Users
**✅ If you use built-in augmentations**: No changes needed! They're all updated.
**⚠️ If you created custom storage augmentations**: Update your storage adapter to:
- Wrap metadata with required `noun`/`verb` fields
- Follow the internal/public method pattern
- Use two-file storage approach
**⚠️ If you access relationship data**: Change `verb.type` to `verb.verb`
## Core Concepts
### What are Augmentations?
Augmentations are modular extensions that add functionality to Brainy without cluttering the core API. They follow a unified interface and can be:
- **Auto-enabled**: Based on configuration (cache, index, storage)
- **Manually registered**: For custom functionality
- **Chained**: Multiple augmentations work together seamlessly
- **Billion-scale ready**: Optimized for datasets with billions of nouns and verbs
### Augmentation Lifecycle
1. **Registration**: Augmentations register before init()
2. **Initialization**: Two-phase init (storage first, then others)
3. **Execution**: Hook into operations (before/after/both)
4. **Shutdown**: Clean teardown on brain.shutdown()
---
## Storage Augmentations
### MemoryStorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Auto-enabled**: When `storage: { type: 'memory' }` or in test environments
**Purpose**: In-memory storage for testing and temporary data
```typescript
const brain = new Brainy({ storage: { type: 'memory' } })
```
### FileSystemStorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Auto-enabled**: When `storage: { type: 'filesystem' }` or Node.js detected
**Purpose**: Persistent file-based storage for Node.js applications
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', rootDirectory: './data' }
})
```
For off-site backup, snapshot `rootDirectory` from your scheduler using `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` — there are no cloud storage augmentations.
### StorageAugmentation (base)
**Location**: `src/augmentations/storageAugmentation.ts`
**Purpose**: Base class for custom storage implementations
### DynamicStorageAugmentation
**Location**: `src/augmentations/storageAugmentation.ts`
**Purpose**: Runtime storage adapter switching
---
## Performance Augmentations (7 total)
### CacheAugmentation
**Location**: `src/augmentations/cacheAugmentation.ts`
**Auto-enabled**: When `cache: true` (default)
**Purpose**: LRU cache for search results and frequent queries
```typescript
brain.clearCache() // Exposed via API
brain.getCacheStats() // Cache hit/miss statistics
```
### IndexAugmentation
**Location**: `src/augmentations/indexAugmentation.ts`
**Auto-enabled**: When `index: true` (default)
**Purpose**: Metadata indexing for O(1) field lookups
```typescript
brain.rebuildMetadataIndex() // Exposed via API
// Enables fast where queries:
brain.find({ where: { category: 'tech' } })
```
### MetricsAugmentation
**Location**: `src/augmentations/metricsAugmentation.ts`
**Auto-enabled**: Always active
**Purpose**: Performance metrics and statistics collection
```typescript
brain.getStats() // Comprehensive metrics
```
### MonitoringAugmentation
**Location**: `src/augmentations/monitoringAugmentation.ts`
**Manual**: Register for detailed monitoring
**Purpose**: Real-time performance monitoring and alerts
### BatchProcessingAugmentation
**Location**: `src/augmentations/batchProcessingAugmentation.ts`
**Auto-enabled**: For batch operations
**Purpose**: Optimizes bulk add/update/delete operations
```typescript
brain.addNouns([...]) // Automatically batched
```
### RequestDeduplicatorAugmentation
**Location**: `src/augmentations/requestDeduplicatorAugmentation.ts`
**Auto-enabled**: Always active
**Purpose**: Prevents duplicate concurrent operations
---
## Data Integrity Augmentations (3 total)
**Auto-enabled**: When `wal: true`
**Purpose**: Write-ahead logging for crash recovery
```typescript
const brain = new Brainy({ wal: true })
// Automatic recovery on restart after crash
```
### EntityRegistryAugmentation
**Location**: `src/augmentations/entityRegistryAugmentation.ts`
**Auto-enabled**: For streaming operations
**Purpose**: High-speed deduplication for real-time data
```typescript
// Prevents duplicate entities in streaming scenarios
brain.add(data) // Automatically deduplicated
```
### AutoRegisterEntitiesAugmentation
**Location**: `src/augmentations/entityRegistryAugmentation.ts`
**Manual**: For automatic entity discovery
**Purpose**: Auto-discovers and registers entities from data
---
## Intelligence Augmentations (2 total)
### NeuralImportAugmentation
**Location**: `src/augmentations/neuralImport.ts`
**Manual**: Via `brain.neuralImport()`
**Purpose**: AI-powered smart data import
```typescript
const result = await brain.neuralImport(data, {
confidenceThreshold: 0.7,
autoApply: true
})
// Automatically detects entities and relationships
```
### IntelligentVerbScoringAugmentation
**Location**: `src/augmentations/intelligentVerbScoringAugmentation.ts`
**Auto-enabled**: When verbs are used
**Purpose**: ML-based relationship strength scoring
```typescript
brain.verbScoring.train(feedback)
brain.verbScoring.getScore(verbId)
```
---
## Communication Augmentations (4 total)
### APIServerAugmentation
**Location**: `src/augmentations/apiServerAugmentation.ts`
**Manual**: For server deployments
**Purpose**: REST/WebSocket/MCP API server
```typescript
const augmentation = new APIServerAugmentation()
await brain.registerAugmentation(augmentation)
// Exposes full Brainy API over network
```
### WebSocketConduitAugmentation
**Location**: `src/augmentations/conduitAugmentations.ts`
**Manual**: For Brainy-to-Brainy sync
**Purpose**: Real-time sync between Brainy instances
```typescript
const conduit = new WebSocketConduitAugmentation()
await conduit.establishConnection('ws://other-brain')
```
### ServerSearchConduitAugmentation
**Location**: `src/augmentations/serverSearchAugmentations.ts`
**Manual**: For client-server search
**Purpose**: Search remote Brainy instance, cache locally
### ServerSearchActivationAugmentation
**Location**: `src/augmentations/serverSearchAugmentations.ts`
**Manual**: Works with ServerSearchConduit
**Purpose**: Triggers and manages server search operations
---
## External Integration (2 total)
### SynapseAugmentation (base)
**Location**: `src/augmentations/synapseAugmentation.ts`
**Purpose**: Base class for external platform integrations
```typescript
// Example: NotionSynapse, SlackSynapse, etc.
class NotionSynapse extends SynapseAugmentation {
async fetchData() { /* Notion API calls */ }
async pushData() { /* Sync to Notion */ }
}
```
### ExampleFileSystemSynapse
**Location**: `src/augmentations/synapseAugmentation.ts`
**Purpose**: Example implementation for file system sync
---
## Augmentation Configuration
### Auto-Configuration
```typescript
const brain = new Brainy({
// These auto-register augmentations:
storage: { type: 'auto', rootDirectory: './brainy-data' }, // Storage augmentation
cache: true, // Cache augmentation
index: true, // Index augmentation
metrics: true // Metrics augmentation
})
```
### Manual Registration
```typescript
const brain = new Brainy()
// Register before init()
const customAug = new MyCustomAugmentation()
await brain.registerAugmentation(customAug)
await brain.init()
```
### Creating Custom Augmentations
```typescript
import { BaseAugmentation } from '@soulcraft/brainy'
class MyAugmentation extends BaseAugmentation {
readonly name = 'my-augmentation'
readonly timing = 'after' // before | after | both
readonly operations = ['addNoun', 'search'] // Which ops to hook
readonly priority = 10 // Execution order (lower = earlier)
protected async onInit(): Promise<void> {
// Initialize your augmentation
}
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'addNoun') {
console.log('Noun added:', params)
}
}
protected async onShutdown(): Promise<void> {
// Cleanup
}
}
```
---
## Augmentation Timing & Priority
### Timing Options
- **`before`**: Runs before the operation (can modify params)
- **`after`**: Runs after the operation (can see results)
- **`both`**: Runs before AND after
### Priority (lower = earlier)
1. Storage augmentations (priority: 0)
2. Cache/Index augmentations (priority: 5-10)
3. Monitoring/Metrics (priority: 15-20)
4. Conduits/Synapses (priority: 20-30)
---
## Key Integration Points
### Where Augmentations Hook In
**Brainy Constructor**:
- Storage augmentations register based on config
- Cache/Index augmentations auto-register if enabled
**brain.init()**:
- Two-phase initialization (storage first, then others)
- Augmentations can access brain instance via context
**Operations** (addNoun, search, etc.):
- Augmentations execute based on timing and operations filter
- Can modify params (before) or see results (after)
**brain.shutdown()**:
- All augmentations cleaned up in reverse order
---
## Performance Impact
Most augmentations have minimal overhead:
- **Cache**: ~1ms per search (saves 10-100ms on hits)
- **Index**: ~1ms per operation (saves 100ms+ on queries)
- **Metrics**: <1ms per operation
- **Storage**: Varies by adapter (memory: 0ms, filesystem: 1-10ms)
---
## Best Practices
1. **Let auto-configuration work**: Most apps need zero manual config
2. **Storage first**: Always configure storage before other augmentations
3. **Use built-in augmentations**: They're optimized and battle-tested
4. **Custom augmentations**: Extend BaseAugmentation for consistency
5. **Respect timing**: Use 'before' to modify, 'after' to observe
6. **Mind priority**: Lower numbers execute first
---
## Troubleshooting
### Augmentation not working?
```typescript
// Check if registered
brain.listAugmentations()
// Check if enabled
brain.isAugmentationEnabled('cache')
// Enable/disable at runtime
brain.enableAugmentation('cache')
brain.disableAugmentation('cache')
```
### Performance issues?
```typescript
// Check augmentation overhead
const stats = brain.getStats()
console.log(stats.augmentations)
// Disable non-critical augmentations
brain.disableAugmentation('monitoring')
```
---
---
*Augmentations make Brainy infinitely extensible while keeping the core API clean and simple!*

620
docs/augmentations/CONFIGURATION.md
View file

@ -1,620 +0,0 @@
# Augmentation Configuration System
**Version**: 2.0.0
**Status**: Production Ready
## Overview
The Brainy Augmentation Configuration System provides a VSCode-style extension architecture with multiple configuration sources, schema validation, and tool discovery. This system maintains Brainy's zero-config philosophy while enabling sophisticated enterprise configuration management.
## Table of Contents
- [Quick Start](#quick-start)
- [Configuration Sources](#configuration-sources)
- [Creating Configurable Augmentations](#creating-configurable-augmentations)
- [Configuration Discovery](#configuration-discovery)
- [Runtime Configuration](#runtime-configuration)
- [Environment Variables](#environment-variables)
- [Configuration Files](#configuration-files)
- [CLI Commands](#cli-commands)
- [Tool Integration](#tool-integration)
- [Migration Guide](#migration-guide)
## Quick Start
### Using an Augmentation with Configuration
```typescript
import { Brainy } from '@soulcraft/brainy'
// Zero-config (uses defaults)
const brain = new Brainy()
// With custom configuration
immediateWrites: true,
checkpointInterval: 300000 // 5 minutes
}))
```
### Configuring via Environment Variables
```bash
export BRAINY_AUG_CACHE_TTL=600000
```
### Configuring via Files
Create a `.brainyrc` file in your project root:
```json
{
"augmentations": {
"wal": {
"enabled": true,
"immediateWrites": true,
"maxSize": 20971520
},
"cache": {
"ttl": 600000,
"maxSize": 2000
}
}
}
```
## Configuration Sources
Configuration is resolved in the following priority order (highest to lowest):
1. **Runtime Updates** - Dynamic configuration changes via API
2. **Constructor Parameters** - Code-time configuration
3. **Environment Variables** - `BRAINY_AUG_<NAME>_<KEY>`
4. **Configuration Files** - `.brainyrc`, `brainy.config.json`
5. **Schema Defaults** - Default values from manifest
### Resolution Example
```typescript
// Schema default
{ maxSize: 10485760 }
// File configuration (.brainyrc)
{ maxSize: 20971520 }
// Environment variable
// Constructor parameter
// Final resolved value: 41943040 (constructor wins)
```
## Creating Configurable Augmentations
### Step 1: Extend ConfigurableAugmentation
```typescript
import { ConfigurableAugmentation, AugmentationManifest } from '@soulcraft/brainy'
export class MyAugmentation extends ConfigurableAugmentation {
name = 'my-augmentation'
timing = 'around' as const
metadata = 'none' as const
operations = ['search', 'add']
priority = 50
constructor(config?: MyConfig) {
super(config) // Handles configuration resolution
}
// Required: Provide manifest for discovery
getManifest(): AugmentationManifest {
return {
id: 'my-augmentation',
name: 'My Augmentation',
version: '1.0.0',
description: 'Does something amazing',
category: 'performance',
configSchema: {
type: 'object',
properties: {
enabled: {
type: 'boolean',
default: true,
description: 'Enable this augmentation'
},
threshold: {
type: 'number',
default: 100,
minimum: 1,
maximum: 1000,
description: 'Processing threshold'
}
}
}
}
}
// Optional: Handle runtime configuration changes
protected async onConfigChange(newConfig: MyConfig, oldConfig: MyConfig): Promise<void> {
if (newConfig.threshold !== oldConfig.threshold) {
// React to threshold change
this.updateThreshold(newConfig.threshold)
}
}
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
if (!this.config.enabled) {
return next()
}
// Your augmentation logic here
return next()
}
}
```
### Step 2: Define Configuration Interface
```typescript
interface MyConfig {
enabled?: boolean
threshold?: number
mode?: 'fast' | 'balanced' | 'thorough'
}
```
### Step 3: Add JSON Schema in Manifest
```typescript
configSchema: {
type: 'object',
properties: {
enabled: {
type: 'boolean',
default: true,
description: 'Enable augmentation'
},
threshold: {
type: 'number',
default: 100,
minimum: 1,
maximum: 1000,
description: 'Processing threshold'
},
mode: {
type: 'string',
default: 'balanced',
enum: ['fast', 'balanced', 'thorough'],
description: 'Processing mode'
}
},
required: [],
additionalProperties: false
}
```
## Configuration Discovery
The Discovery API allows tools to discover and configure augmentations dynamically:
```typescript
import { AugmentationDiscovery } from '@soulcraft/brainy'
const discovery = new AugmentationDiscovery(brain.augmentations)
// Discover all augmentations with manifests
const listings = await discovery.discover({
includeConfig: true,
includeSchema: true
})
// Get configuration schema
const schema = await discovery.getConfigSchema('wal')
// Validate configuration
const validation = await discovery.validateConfig('wal', {
enabled: true,
maxSize: 'invalid' // Will fail validation
})
// Update configuration at runtime
await discovery.updateConfig('wal', {
checkpointInterval: 120000
})
```
## Runtime Configuration
### Update Configuration Dynamically
```typescript
// Get augmentation
const wal = brain.augmentations.get('wal')
// Update configuration
await wal.updateConfig({
checkpointInterval: 300000
})
// Get current configuration
const config = wal.getConfig()
```
### React to Configuration Changes
```typescript
class MyAugmentation extends ConfigurableAugmentation {
protected async onConfigChange(newConfig: any, oldConfig: any): Promise<void> {
// Stop old processes
if (oldConfig.enabled && !newConfig.enabled) {
await this.stop()
}
// Start new processes
if (!oldConfig.enabled && newConfig.enabled) {
await this.start()
}
// Update settings
if (newConfig.interval !== oldConfig.interval) {
this.rescheduleTimer(newConfig.interval)
}
}
}
```
## Environment Variables
### Naming Convention
```bash
BRAINY_AUG_<AUGMENTATION_ID>_<CONFIG_KEY>=value
```
### Examples
```bash
# Cache augmentation
BRAINY_AUG_CACHE_ENABLED=true
BRAINY_AUG_CACHE_MAX_SIZE=2000
BRAINY_AUG_CACHE_TTL=600000
# Complex values (JSON)
BRAINY_AUG_MYAUG_FILTERS='["*.js","*.ts"]'
BRAINY_AUG_MYAUG_OPTIONS='{"deep":true,"follow":false}'
```
### Docker Example
```dockerfile
ENV BRAINY_AUG_CACHE_TTL=600000
```
## Configuration Files
### File Locations (Priority Order)
1. `.brainyrc` (current directory)
2. `.brainyrc.json` (current directory)
3. `brainy.config.json` (current directory)
4. `~/.brainy/config.json` (user home)
5. `~/.brainyrc` (user home)
### File Format
```json
{
"augmentations": {
"wal": {
"enabled": true,
"immediateWrites": true,
"maxSize": 20971520,
"checkpointInterval": 300000
},
"cache": {
"enabled": true,
"maxSize": 2000,
"ttl": 600000
},
"metrics": {
"enabled": false
}
}
}
```
### Per-Environment Configuration
```json
{
"augmentations": {
"wal": {
"development": {
"enabled": true,
"immediateWrites": true,
"maxSize": 5242880
},
"production": {
"enabled": true,
"immediateWrites": false,
"maxSize": 104857600,
"checkpointInterval": 60000
}
}
}
}
```
## CLI Commands
### List Augmentations with Configuration
```bash
# Show all augmentations with config status
brainy augment list --detailed
# Show configuration for specific augmentation
brainy augment config wal
# Set configuration value
brainy augment config wal --set immediateWrites=true
# Show environment variable names
brainy augment config wal --env
# Export configuration schema
brainy augment schema wal > wal-schema.json
# Validate configuration file
brainy augment validate --file config.json
```
### Interactive Configuration
```bash
# Interactive configuration wizard
brainy augment configure wal
? Operation mode?
❯ Performance (immediate writes)
Durability (synchronous writes)
Custom
? Maximum log size? (10MB) 20MB
? Checkpoint interval? (1 minute) 5 minutes
Configuration saved to .brainyrc
```
## Tool Integration
### Brain-Cloud Explorer UI
```typescript
// Auto-generate configuration form from schema
const ConfigurationUI = ({ augmentationId }) => {
const [manifest, setManifest] = useState(null)
const [config, setConfig] = useState({})
useEffect(() => {
// Fetch manifest with schema
fetch(`/api/augmentations/${augmentationId}/manifest`)
.then(res => res.json())
.then(setManifest)
// Get current configuration
discovery.getConfig(augmentationId)
.then(setConfig)
}, [augmentationId])
const handleSave = async (newConfig) => {
// Validate configuration
const validation = await fetch(`/api/augmentations/${augmentationId}/validate`, {
method: 'POST',
body: JSON.stringify(newConfig)
}).then(res => res.json())
if (validation.valid) {
// Apply configuration
await discovery.updateConfig(augmentationId, newConfig)
}
}
// Render form based on schema
return <SchemaForm
schema={manifest?.configSchema}
values={config}
onSubmit={handleSave}
/>
}
```
### VS Code Extension
```json
// package.json contribution points
{
"contributes": {
"configuration": {
"title": "Brainy Augmentations",
"properties": {
"brainy.augmentations.wal.enabled": {
"type": "boolean",
"default": true,
},
"brainy.augmentations.wal.maxSize": {
"type": "number",
"default": 10485760,
}
}
}
}
}
```
## Migration Guide
### Migrating from BaseAugmentation
**Before:**
```typescript
export class MyAugmentation extends BaseAugmentation {
constructor(config: MyConfig = {}) {
super()
this.config = {
enabled: config.enabled ?? true,
threshold: config.threshold ?? 100
}
}
// No manifest
// No config discovery
// No runtime updates
}
```
**After:**
```typescript
export class MyAugmentation extends ConfigurableAugmentation {
constructor(config?: MyConfig) {
super(config) // Config resolution handled automatically
}
getManifest(): AugmentationManifest {
return {
id: 'my-augmentation',
name: 'My Augmentation',
version: '1.0.0',
description: 'Does something amazing',
category: 'performance',
configSchema: {
type: 'object',
properties: {
enabled: { type: 'boolean', default: true },
threshold: { type: 'number', default: 100 }
}
}
}
}
// Optional: Handle config changes
protected async onConfigChange(newConfig: MyConfig, oldConfig: MyConfig): Promise<void> {
// React to changes
}
}
```
### Backwards Compatibility
The system maintains full backwards compatibility:
1. **BaseAugmentation still works** - Existing augmentations continue to function
2. **Constructor config still works** - Existing configuration patterns preserved
3. **Zero-config still works** - Defaults are applied automatically
4. **Progressive enhancement** - Add features as needed
## Best Practices
### 1. Always Provide Defaults
```typescript
configSchema: {
properties: {
enabled: {
type: 'boolean',
default: true, // Always provide defaults
description: 'Enable this feature'
}
}
}
```
### 2. Use Descriptive Configuration Keys
```typescript
// Good
checkpointInterval: 60000
// Bad
ci: 60000
```
### 3. Validate Configuration
```typescript
protected async onConfigChange(newConfig: any, oldConfig: any): Promise<void> {
// Validate before applying
if (newConfig.maxSize < 1048576) {
throw new Error('maxSize must be at least 1MB')
}
// Apply changes
this.maxSize = newConfig.maxSize
}
```
### 4. Document Environment Variables
```typescript
/**
* Environment Variables:
* - BRAINY_AUG_MYAUG_ENABLED: Enable augmentation (boolean)
* - BRAINY_AUG_MYAUG_THRESHOLD: Processing threshold (number)
* - BRAINY_AUG_MYAUG_MODE: Processing mode (fast|balanced|thorough)
*/
```
### 5. Provide Configuration Examples
```typescript
configExamples: [
{
name: 'Production',
description: 'Optimized for production use',
config: {
enabled: true,
mode: 'thorough',
threshold: 500
}
},
{
name: 'Development',
description: 'Lightweight for development',
config: {
enabled: true,
mode: 'fast',
threshold: 10
}
}
]
```
## Troubleshooting
### Configuration Not Loading
1. Check file locations and names
2. Verify JSON syntax in config files
3. Check environment variable names (case-sensitive)
4. Use `brainy augment config <name> --debug` to see resolution
### Validation Errors
1. Check schema requirements
2. Verify data types match schema
3. Check minimum/maximum constraints
4. Use discovery API to validate before applying
### Runtime Updates Not Working
1. Ensure augmentation extends ConfigurableAugmentation
2. Implement onConfigChange if needed
3. Check for validation errors
4. Verify augmentation is initialized
## API Reference
See the [Discovery API Documentation](./discovery-api.md) for complete API details.
## Examples
See the [examples directory](../../examples/augmentation-config/) for complete working examples.

532
docs/augmentations/DEVELOPER-GUIDE.md
View file

@ -1,532 +0,0 @@
# 🛠️ Brainy Augmentation Developer Guide
> **How to create, test, and use augmentations in Brainy**
>
> **⚠️ Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
## Migration Guide
### What Changed?
1. **Metadata Structure**: All metadata now requires type fields (`noun` or `verb`)
2. **Property Rename**: `verb.type` → `verb.verb` for relationships
3. **Two-File Storage**: Vectors and metadata stored separately for performance
4. **Return Types**: Storage methods distinguish between internal (pure) and public (WithMetadata) returns
### Migration Checklist
- [ ] Update metadata creation to include required `noun` field
- [ ] Change `verb.type` to `verb.verb` in all relationship code
- [ ] Update storage adapter methods to follow internal/public pattern
- [ ] Ensure metadata access uses correct structure
### Quick Migration Example
```typescript
// ❌ v3.x
const verb = {
type: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
if (verb.type === 'relatedTo') { ... }
// ✅ Current
const verb = {
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
const metadata: VerbMetadata = {
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
if (verb.verb === 'relatedTo') { ... }
```
## Quick Start: Your First Augmentation
```typescript
import { BaseAugmentation, BrainyAugmentation, AugmentationContext } from '@soulcraft/brainy'
export class MyFirstAugmentation extends BaseAugmentation {
readonly name = 'my-first-augmentation'
readonly timing = 'after' as const // When to run: before | after | both
readonly operations = ['add'] as const // Which operations to hook
readonly priority = 10 // Execution order (lower = first)
protected async onInit(): Promise<void> {
// Initialize your augmentation
console.log('MyFirstAugmentation initialized!')
}
async execute<T = any>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'add') {
console.log('Noun added:', params.noun)
// Access metadata correctly
if (params.noun?.metadata) {
console.log('Noun type:', params.noun.metadata.noun) // Required field
}
// You can access the brain instance
const stats = await context?.brain.getStats()
console.log('Total nouns:', stats.totalNouns)
}
}
protected async onShutdown(): Promise<void> {
// Cleanup
console.log('MyFirstAugmentation shutting down')
}
}
```
## Using Your Augmentation
```typescript
import { Brainy } from '@soulcraft/brainy'
import { MyFirstAugmentation } from './my-first-augmentation'
const brain = new Brainy()
// Register before init()
brain.augmentations.register(new MyFirstAugmentation())
await brain.init()
// Now your augmentation runs automatically!
await brain.add('Hello World')
// Console: "Noun added: { id: '...', vector: [...], metadata: {} }"
```
---
## Augmentation Lifecycle
### 1. Registration Phase
```typescript
const aug = new MyAugmentation()
brain.augmentations.register(aug) // Before brain.init()!
```
### 2. Initialization Phase
```typescript
await brain.init() // Calls aug.initialize() internally
// Your onInit() method runs here
```
### 3. Execution Phase
```typescript
await brain.add('data') // Your execute() method runs
```
### 4. Shutdown Phase
```typescript
await brain.shutdown() // Your onShutdown() method runs
```
---
## Timing Options
### `before` - Modify Input
```typescript
class ValidationAugmentation extends BaseAugmentation {
readonly timing = 'before' as const
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Validate and/or modify params
if (!params.content) {
throw new Error('Content required')
}
// Return modified params
return { ...params, validated: true }
}
}
}
```
### `after` - React to Results
```typescript
class LoggingAugmentation extends BaseAugmentation {
readonly timing = 'after' as const
async execute<T>(operation: string, params: any): Promise<void> {
if (operation === 'search') {
console.log(`Search for "${params.query}" returned ${params.result.length} results`)
}
// Don't return anything - just observe
}
}
```
### `both` - Before AND After
```typescript
class TimingAugmentation extends BaseAugmentation {
readonly timing = 'both' as const
private startTime?: number
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
if (!this.startTime) {
// Before execution
this.startTime = Date.now()
} else {
// After execution
const duration = Date.now() - this.startTime
console.log(`${operation} took ${duration}ms`)
this.startTime = undefined
}
}
}
```
---
## Operation Hooks
### Core Operations You Can Hook
```typescript
readonly operations = [
'add', // Adding data
'update', // Updating data
'delete', // Deleting data
'get', // Retrieving data
'search', // Searching
'find', // Triple Intelligence queries
'relate', // Adding relationships
'unrelate', // Removing relationships
'clear', // Clearing data
'all' // Hook ALL operations
] as const
```
### Example: Multi-Operation Hook
```typescript
class AuditAugmentation extends BaseAugmentation {
readonly operations = ['add', 'update', 'delete'] as const
async execute<T>(operation: string, params: any): Promise<void> {
// Log all data modifications
await this.logToAuditTrail(operation, params)
}
}
```
---
## Accessing Brain Context
```typescript
class ContextAwareAugmentation extends BaseAugmentation {
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<void> {
// Access the brain instance
const brain = context?.brain
if (!brain) return
// Use any brain method
const stats = await brain.getStats()
const size = await brain.size()
const results = await brain.search('query')
// Access other augmentations
const cache = brain.augmentations.get('cache')
if (cache) {
await cache.clear()
}
}
}
```
---
## Real-World Examples
### 1. Backup Augmentation
```typescript
class BackupAugmentation extends BaseAugmentation {
readonly name = 'backup'
readonly timing = 'after' as const
readonly operations = ['add', 'update', 'delete'] as const
readonly priority = 5
private changes = 0
private readonly backupThreshold = 100
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
this.changes++
if (this.changes >= this.backupThreshold) {
await this.performBackup(context?.brain)
this.changes = 0
}
}
private async performBackup(brain?: any): Promise<void> {
if (!brain) return
// Cut an instant hard-link snapshot via the Db API
const snapshotPath = `/backups/auto-${Date.now()}`
const pin = brain.now()
try {
await pin.persist(snapshotPath)
} finally {
await pin.release()
}
console.log(`Automatic snapshot created: ${snapshotPath}`)
}
}
```
### 2. Rate Limiting Augmentation
```typescript
class RateLimitAugmentation extends BaseAugmentation {
readonly name = 'rate-limit'
readonly timing = 'before' as const
readonly operations = ['search', 'find'] as const
readonly priority = 100 // High priority - run first
private requests = new Map<string, number[]>()
private readonly limit = 100 // 100 requests
private readonly window = 60000 // per minute
async execute<T>(operation: string, params: any): Promise<void> {
const now = Date.now()
const key = params.userId || 'anonymous'
// Get request timestamps
const timestamps = this.requests.get(key) || []
// Remove old timestamps
const recent = timestamps.filter(t => now - t < this.window)
// Check limit
if (recent.length >= this.limit) {
throw new Error('Rate limit exceeded')
}
// Add current request
recent.push(now)
this.requests.set(key, recent)
}
}
```
### 3. Encryption Augmentation
```typescript
class EncryptionAugmentation extends BaseAugmentation {
readonly name = 'encryption'
readonly timing = 'both' as const
readonly operations = ['add', 'get'] as const
readonly priority = 90 // Run early
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Encrypt before storing
if (params.metadata?.sensitive) {
params.content = await this.encrypt(params.content)
params.encrypted = true
}
return params
}
if (operation === 'get' && params.result?.encrypted) {
// Decrypt after retrieval
params.result.content = await this.decrypt(params.result.content)
delete params.result.encrypted
return params.result
}
}
}
```
---
## Testing Your Augmentation
```typescript
import { describe, it, expect } from 'vitest'
import { Brainy } from '@soulcraft/brainy'
import { MyAugmentation } from './my-augmentation'
describe('MyAugmentation', () => {
it('should hook into addNoun', async () => {
const brain = new Brainy({ storage: 'memory' })
const aug = new MyAugmentation()
// Spy on the execute method
const executeSpy = vi.spyOn(aug, 'execute')
brain.augmentations.register(aug)
await brain.init()
// Trigger the augmentation
await brain.add('test data')
// Verify it was called
expect(executeSpy).toHaveBeenCalledWith(
'add',
expect.objectContaining({ content: 'test data' }),
expect.any(Object)
)
})
})
```
---
## Best Practices
### 1. Use Proper Timing
- `before`: Validation, modification, rate limiting
- `after`: Logging, metrics, side effects
- `both`: Timing, tracing, wrapping
### 2. Set Appropriate Priority
```typescript
// Priority guidelines
100: Critical (auth, rate limiting)
50: Important (validation, transformation)
10: Normal (logging, metrics)
1: Optional (debugging, tracing)
```
### 3. Handle Errors Gracefully
```typescript
async execute<T>(operation: string, params: any): Promise<void> {
try {
await this.riskyOperation()
} catch (error) {
// Log but don't break the main operation
console.error(`Augmentation error in ${this.name}:`, error)
// Optionally report to monitoring
this.reportError(error)
}
}
```
### 4. Be Performance Conscious
```typescript
class CachedAugmentation extends BaseAugmentation {
private cache = new Map<string, any>()
async execute<T>(operation: string, params: any): Promise<any> {
const key = this.getCacheKey(params)
// Check cache first
if (this.cache.has(key)) {
return this.cache.get(key)
}
// Expensive operation
const result = await this.expensiveOperation(params)
this.cache.set(key, result)
return result
}
}
```
### 5. Clean Up Resources
```typescript
protected async onShutdown(): Promise<void> {
// Close connections
await this.connection?.close()
// Clear intervals
clearInterval(this.interval)
// Flush buffers
await this.flush()
// Clear caches
this.cache.clear()
}
```
---
## Publishing Your Augmentation (Future)
### Package Structure
```
my-augmentation/
├── src/
│ └── index.ts # Your augmentation
├── dist/ # Built output
├── tests/
│ └── augmentation.test.ts
├── package.json
├── tsconfig.json
└── README.md
```
### package.json
```json
{
"name": "@mycompany/brainy-custom-augmentation",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"keywords": ["brainy-augmentation"],
"peerDependencies": {
"@soulcraft/brainy": ">=2.0.0"
},
"brainy": {
"type": "augmentation",
"class": "CustomAugmentation",
"timing": "after",
"operations": ["add"],
"priority": 10
}
}
```
### Future: Brain Cloud Registry
```bash
# Coming in 2.1+
npm run build
npm test
brainy publish # Publishes to brain-cloud registry
```
---
## FAQ
### Q: Can I modify the operation result?
**A**: Yes, if `timing: 'before'`, return modified params. If `timing: 'after'`, you can see but not modify results.
### Q: Can augmentations communicate?
**A**: Yes, through the context: `context.brain.augmentations.get('other-augmentation')`
### Q: What if my augmentation fails?
**A**: Handle errors internally. Don't break the main operation unless critical.
### Q: Can I use async operations?
**A**: Yes, everything is async-friendly.
### Q: How do I access storage directly?
**A**: Through context: `context.brain.storage` (but prefer using brain methods)
---
## Get Help
- **GitHub**: [github.com/soulcraft/brainy](https://github.com/soulcraft/brainy)
- **Discord**: [discord.gg/brainy](https://discord.gg/brainy)
- **Examples**: See `/examples/augmentations/` in the repo
---
*Start building your augmentation today! The marketplace is coming in 2.1 🚀*

669
docs/augmentations/EXAMPLES.md
View file

@ -1,669 +0,0 @@
# Augmentation Examples - Import, Store, Export
This guide shows two complete workflows:
1. **Simple Handler** - Just import a new file type
2. **Full Augmentation** - Import + Store + Export (premium-ready)
---
## Workflow 1: Simple Handler (Import Only)
**Use case:** You want to import a new file type (e.g., CAD files) into Brainy's knowledge graph.
### Step 1: Create the Handler
```typescript
// src/handlers/CADHandler.ts
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
import type { FormatHandlerOptions, ProcessedData } from '@soulcraft/brainy/augmentations/intelligentImport'
import { parseCAD } from 'cad-parser' // Your parsing library
export class CADHandler extends BaseFormatHandler {
readonly format = 'cad'
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
if (typeof data === 'object' && 'filename' in data) {
const mimeType = this.getMimeType(data)
return this.mimeTypeMatches(mimeType, ['image/vnd.dwg', 'image/vnd.dxf'])
}
return false
}
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
// Parse CAD file
const cad = await parseCAD(buffer)
// Extract entities for knowledge graph
const entities = []
// Document entity
entities.push({
type: 'CADDocument',
filename: options.filename,
units: cad.units,
bounds: cad.bounds,
version: cad.version
})
// Layer entities
for (const layer of cad.layers) {
entities.push({
type: 'CADLayer',
name: layer.name,
color: layer.color,
visible: layer.visible
})
}
// Object entities
for (const obj of cad.objects) {
entities.push({
type: 'CADObject',
objectType: obj.type,
layer: obj.layer,
geometry: obj.geometry
})
}
return {
format: 'cad',
data: entities,
metadata: {
rowCount: entities.length,
fields: ['type', 'name', 'geometry'],
processingTime: Date.now() - startTime,
layerCount: cad.layers.length,
objectCount: cad.objects.length
},
filename: options.filename
}
}
}
```
### Step 2: Register the Handler
```typescript
// src/index.ts
import { globalHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
import { CADHandler } from './handlers/CADHandler.js'
// Register handler globally
globalHandlerRegistry.registerHandler({
name: 'cad',
mimeTypes: ['image/vnd.dwg', 'image/vnd.dxf'],
extensions: ['.dwg', '.dxf', '.dwf'],
loader: async () => new CADHandler()
})
```
### Step 3: Use It
```typescript
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy()
await brain.init()
// Import CAD file - automatically routed to CADHandler
const result = await brain.import({
type: 'file',
data: cadFileBuffer,
filename: 'floor-plan.dwg'
})
console.log(`Imported ${result.entities.length} CAD entities`)
// Query the imported data
const layers = await brain.find({ type: 'CADLayer' })
const objects = await brain.find({ type: 'CADObject', layer: 'WALLS' })
```
**That's it!** Simple handlers just import data. Brainy handles storage automatically.
---
## Workflow 2: Full Augmentation (Import + Store + Export)
**Use case:** You want a complete solution that imports project files, stores them with special logic, and exports results (e.g., React project analyzer).
### Step 1: Create the Augmentation
```typescript
// @yourcompany/brainy-react-analyzer
import { BaseAugmentation, type AugmentationContext } from '@soulcraft/brainy'
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
import type { ProcessedData } from '@soulcraft/brainy/augmentations/intelligentImport'
import { parse } from '@babel/parser'
import traverse from '@babel/traverse'
import * as t from '@babel/types'
/**
* React Project Analyzer Augmentation
*
* Features:
* - Import: Parse React components, extract props, hooks, imports
* - Store: Create relationships between components
* - Export: Generate component diagram, dependency graph
*/
export class ReactAnalyzerAugmentation extends BaseAugmentation {
readonly name = 'react-analyzer'
readonly timing = 'before' as const
readonly operations = ['import', 'export'] as any[]
readonly priority = 75
private handler: ReactComponentHandler
constructor(config = {}) {
super(config)
this.handler = new ReactComponentHandler()
}
async execute<T = any>(
operation: string,
params: any,
next: () => Promise<T>
): Promise<T> {
// IMPORT: Parse React files
if (operation === 'import' && this.isReactFile(params)) {
return this.handleImport(params, next)
}
// EXPORT: Generate diagrams/reports
if (operation === 'export' && params.format === 'react-diagram') {
return this.handleExport(params, next)
}
return next()
}
private isReactFile(params: any): boolean {
const filename = params.filename || ''
return (
(filename.endsWith('.tsx') || filename.endsWith('.jsx')) &&
params.data?.includes('React')
)
}
private async handleImport<T>(params: any, next: () => Promise<T>): Promise<T> {
// Parse React component
const processed = await this.handler.process(params.data, params.options)
// Enrich with relationships
params.data = processed.data
params.metadata = {
...params.metadata,
reactAnalysis: processed.metadata
}
// Continue to next augmentation/storage
const result = await next()
// Post-process: Create component relationships
await this.createComponentRelationships(processed, result)
return result
}
private async createComponentRelationships(
processed: ProcessedData,
result: any
): Promise<void> {
const brain = this.getBrain()
if (!brain) return
// Find the component entity that was created
const component = processed.data.find(d => d.type === 'ReactComponent')
if (!component) return
// Create relationships for imports
for (const imp of component.imports || []) {
// Find or create imported component
const imported = await brain.findOne({
type: 'ReactComponent',
name: imp.name
})
if (imported) {
// Create "Imports" relationship
await brain.createRelation({
source: result.entities[0].id,
verb: 'Imports',
target: imported.id,
metadata: {
importPath: imp.path,
importType: imp.type
}
})
}
}
// Create relationships for prop types
for (const prop of component.props || []) {
if (prop.typeRef) {
const typeEntity = await brain.findOne({
type: 'TypeDefinition',
name: prop.typeRef
})
if (typeEntity) {
await brain.createRelation({
source: result.entities[0].id,
verb: 'UsesPropType',
target: typeEntity.id
})
}
}
}
}
private async handleExport<T>(params: any, next: () => Promise<T>): Promise<T> {
const brain = this.getBrain()
if (!brain) return next()
// Query all React components
const components = await brain.find({ type: 'ReactComponent' })
// Build dependency graph
const graph = await this.buildDependencyGraph(components)
// Generate diagram
const diagram = this.generateMermaidDiagram(graph)
return {
format: 'react-diagram',
diagram,
components: components.length,
dependencies: graph.edges.length
} as T
}
private async buildDependencyGraph(components: any[]): Promise<any> {
const brain = this.getBrain()
const nodes = components.map(c => ({
id: c.id,
name: c.name,
props: c.props
}))
const edges = []
for (const component of components) {
const imports = await brain.getRelated(component.id, 'Imports')
for (const imp of imports) {
edges.push({
from: component.id,
to: imp.id,
type: 'imports'
})
}
}
return { nodes, edges }
}
private generateMermaidDiagram(graph: any): string {
let mermaid = 'graph TD\n'
for (const node of graph.nodes) {
mermaid += ` ${node.id}[${node.name}]\n`
}
for (const edge of graph.edges) {
mermaid += ` ${edge.from} --> ${edge.to}\n`
}
return mermaid
}
}
/**
* React Component Handler
*/
class ReactComponentHandler extends BaseFormatHandler {
readonly format = 'react'
canHandle(data: any): boolean {
if (typeof data === 'object' && 'filename' in data) {
return data.filename?.match(/\.(tsx|jsx)$/) !== null
}
return false
}
async process(data: Buffer | string, options: any): Promise<ProcessedData> {
const code = Buffer.isBuffer(data) ? data.toString('utf-8') : data
// Parse with Babel
const ast = parse(code, {
sourceType: 'module',
plugins: ['jsx', 'typescript']
})
// Extract component info
const components: any[] = []
const imports: any[] = []
const exports: any[] = []
traverse(ast, {
// Detect function components
FunctionDeclaration(path) {
if (this.isReactComponent(path.node)) {
components.push({
type: 'ReactComponent',
name: path.node.id?.name,
componentType: 'function',
props: this.extractProps(path),
hooks: this.extractHooks(path),
state: this.extractState(path)
})
}
},
// Detect class components
ClassDeclaration(path) {
if (this.isReactClassComponent(path.node)) {
components.push({
type: 'ReactComponent',
name: path.node.id.name,
componentType: 'class',
props: this.extractClassProps(path),
state: this.extractClassState(path),
lifecycle: this.extractLifecycleMethods(path)
})
}
},
// Extract imports
ImportDeclaration(path) {
imports.push({
type: 'Import',
from: path.node.source.value,
imports: path.node.specifiers.map(s => ({
name: s.local.name,
imported: t.isImportSpecifier(s) ? s.imported.name : null
}))
})
},
// Extract exports
ExportNamedDeclaration(path) {
exports.push({
type: 'Export',
name: path.node.declaration?.id?.name
})
}
})
// Enrich components with import info
for (const component of components) {
component.imports = imports
component.exports = exports.find(e => e.name === component.name)
}
return {
format: 'react',
data: components,
metadata: {
rowCount: components.length,
fields: ['type', 'name', 'props', 'hooks'],
processingTime: Date.now() - startTime,
componentCount: components.length,
importCount: imports.length,
exportCount: exports.length
},
filename: options.filename
}
}
private isReactComponent(node: any): boolean {
// Check if function returns JSX
return node.body?.body?.some(stmt =>
t.isReturnStatement(stmt) && this.isJSX(stmt.argument)
)
}
private isJSX(node: any): boolean {
return t.isJSXElement(node) || t.isJSXFragment(node)
}
private extractProps(path: any): any[] {
const params = path.node.params
if (params.length === 0) return []
const propsParam = params[0]
if (t.isObjectPattern(propsParam)) {
return propsParam.properties.map(p => ({
name: p.key.name,
type: p.typeAnnotation?.typeAnnotation?.type
}))
}
return []
}
private extractHooks(path: any): string[] {
const hooks: string[] = []
path.traverse({
CallExpression(hookPath) {
const callee = hookPath.node.callee
if (t.isIdentifier(callee) && callee.name.startsWith('use')) {
hooks.push(callee.name)
}
}
})
return hooks
}
private extractState(path: any): any[] {
const stateVars: any[] = []
path.traverse({
CallExpression(hookPath) {
if (
t.isIdentifier(hookPath.node.callee) &&
hookPath.node.callee.name === 'useState'
) {
const parent = hookPath.parent
if (t.isVariableDeclarator(parent) && t.isArrayPattern(parent.id)) {
const [stateVar] = parent.id.elements
if (t.isIdentifier(stateVar)) {
stateVars.push({
name: stateVar.name,
initialValue: hookPath.node.arguments[0]
})
}
}
}
}
})
return stateVars
}
}
```
### Step 2: Package as NPM Module
```json
// package.json
{
"name": "@yourcompany/brainy-react-analyzer",
"version": "1.0.0",
"description": "React project analyzer for Brainy",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"exports": {
".": {
"import": "./dist/index.js",
"require": "./dist/index.cjs"
}
},
"keywords": ["brainy", "react", "analyzer", "augmentation"],
"peerDependencies": {
"@soulcraft/brainy": "^5.2.0"
},
"dependencies": {
"@babel/parser": "^7.23.0",
"@babel/traverse": "^7.23.0",
"@babel/types": "^7.23.0"
}
}
```
### Step 3: Use the Augmentation
```typescript
// Install
// npm install @yourcompany/brainy-react-analyzer
import { Brainy } from '@soulcraft/brainy'
import { ReactAnalyzerAugmentation } from '@yourcompany/brainy-react-analyzer'
const brain = new Brainy()
// Add augmentation
brain.addAugmentation(new ReactAnalyzerAugmentation())
await brain.init()
// Import React project
await brain.import({
type: 'directory',
path: '/path/to/react-project/src',
recursive: true
})
// Query components
const components = await brain.find({ type: 'ReactComponent' })
console.log(`Found ${components.length} React components`)
// Find component dependencies
const appComponent = await brain.findOne({ type: 'ReactComponent', name: 'App' })
const imports = await brain.getRelated(appComponent.id, 'Imports')
console.log(`App component imports:`, imports.map(c => c.name))
// Export diagram
const diagram = await brain.export({
format: 'react-diagram'
})
console.log(diagram.diagram) // Mermaid diagram
```
### Step 4: Premium Licensing (Optional)
```typescript
// Add license checking
export class ReactAnalyzerAugmentation extends BaseAugmentation {
private licenseKey?: string
constructor(config: { licenseKey?: string } = {}) {
super(config)
this.licenseKey = config.licenseKey
}
async onInitialize(): Promise<void> {
if (!this.licenseKey) {
throw new Error('React Analyzer requires a license key. Get one at https://yourcompany.com/brainy-react')
}
// Verify license
const valid = await this.verifyLicense(this.licenseKey)
if (!valid) {
throw new Error('Invalid license key')
}
this.log('React Analyzer initialized successfully')
}
private async verifyLicense(key: string): Promise<boolean> {
// Check with your license server
const response = await fetch('https://api.yourcompany.com/verify', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ key, product: 'brainy-react-analyzer' })
})
const data = await response.json()
return data.valid
}
}
// Usage with license
brain.addAugmentation(new ReactAnalyzerAugmentation({
licenseKey: 'YOUR-LICENSE-KEY'
}))
```
---
## Comparison: Handler vs Augmentation
| Feature | Simple Handler | Full Augmentation |
|---------|---------------|-------------------|
| **Import** | ✅ Yes (automatic) | ✅ Yes (with custom logic) |
| **Storage** | ✅ Automatic (Brainy core) | ✅ Custom logic + relationships |
| **Export** | ❌ No | ✅ Yes (custom formats) |
| **Relationships** | ❌ No | ✅ Yes (create custom relationships) |
| **Premium licensing** | ❌ Difficult | ✅ Easy (augmentation-level) |
| **Custom operations** | ❌ Import only | ✅ Any operation |
| **Complexity** | Low (50-100 lines) | Medium (200-500 lines) |
### When to use Handler:
- Just need to import a new file type
- Don't need custom export
- Don't need special relationships
- Simple use case
### When to use Augmentation:
- Need import + export workflow
- Need custom relationship logic
- Want premium licensing capability
- Complex business logic
- Multiple operations (import + export + query)
---
## More Examples
### Example: Python Project Analyzer
```typescript
class PythonAnalyzerAugmentation extends BaseAugmentation {
// Import Python files, extract classes/functions
// Create relationships between modules
// Export: Dependency diagram, call graph
}
```
### Example: Database Schema Sync
```typescript
class DatabaseSyncAugmentation extends BaseAugmentation {
// Import: Parse SQL schema
// Store: Tables, columns, relationships
// Export: Generate migration scripts
}
```
### Example: API Documentation Generator
```typescript
class APIDocAugmentation extends BaseAugmentation {
// Import: Parse TypeScript types
// Store: Endpoints, parameters, responses
// Export: OpenAPI spec, Markdown docs
}
```
---
## See Also
- [FORMAT_HANDLERS.md](./FORMAT_HANDLERS.md) - Creating format handlers
- [AUGMENTATIONS.md](./AUGMENTATIONS.md) - Augmentation system details
- [ImageHandler source](../../src/augmentations/intelligentImport/handlers/imageHandler.ts) - Handler example
- [IntelligentImportAugmentation source](../../src/augmentations/intelligentImport/IntelligentImportAugmentation.ts) - Augmentation example

687
docs/augmentations/FORMAT_HANDLERS.md
View file

@ -1,687 +0,0 @@
# Creating Custom Format Handlers
**Version:** 5.2.0+
Format handlers enable you to import ANY file type into Brainy as structured knowledge graph data. This guide shows how to create custom format handlers for your specific file formats.
---
## Quick Start
```typescript
import { BaseFormatHandler, globalHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
import type { FormatHandlerOptions, ProcessedData } from '@soulcraft/brainy/augmentations/intelligentImport'
class MyFormatHandler extends BaseFormatHandler {
readonly format = 'myformat'
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
// Option 1: Check by MIME type
if (typeof data === 'object' && 'filename' in data) {
const mimeType = this.getMimeType(data)
return this.mimeTypeMatches(mimeType, ['application/x-myformat'])
}
// Option 2: Check by magic bytes
if (Buffer.isBuffer(data)) {
return data[0] === 0x4D && data[1] === 0x59 // "MY" magic bytes
}
return false
}
async process(
data: Buffer | string,
options: FormatHandlerOptions
): Promise<ProcessedData> {
// Convert to Buffer
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
// Parse your format
const parsed = this.parseMyFormat(buffer)
// Return structured data
return {
format: 'myformat',
data: [
{
type: 'MyEntity',
name: parsed.name,
metadata: parsed.metadata
}
],
metadata: {
rowCount: 1,
fields: ['type', 'name', 'metadata'],
processingTime: Date.now() - startTime
}
}
}
private parseMyFormat(buffer: Buffer): any {
// Your parsing logic here
return { name: 'example', metadata: {} }
}
}
// Register globally
globalHandlerRegistry.registerHandler({
name: 'myformat',
mimeTypes: ['application/x-myformat'],
extensions: ['.myf', '.myfmt'],
loader: async () => new MyFormatHandler()
})
```
Now Brainy automatically handles your format:
```typescript
await brain.import({
type: 'file',
data: myFormatBuffer,
filename: 'data.myf'
})
// Automatically routes to MyFormatHandler!
```
---
## BaseFormatHandler
All format handlers should extend `BaseFormatHandler`, which provides:
### MIME Type Detection
```typescript
protected getMimeType(data: Buffer | string | { filename?: string }): string
```
Detects MIME type from filename or buffer. Uses Brainy's comprehensive MIME detection (2000+ types).
**Example:**
```typescript
const mimeType = this.getMimeType({ filename: 'data.dwg' })
// Returns: 'image/vnd.dwg'
```
### MIME Type Matching
```typescript
protected mimeTypeMatches(mimeType: string, patterns: string[]): boolean
```
Checks if MIME type matches patterns. Supports wildcards (`image/*`).
**Example:**
```typescript
if (this.mimeTypeMatches(mimeType, ['image/*', 'video/*'])) {
// Handle all images and videos
}
```
### Extension Detection
```typescript
protected detectExtension(data: string | Buffer | { filename?: string; ext?: string }): string | null
```
Extracts file extension for fallback detection.
---
## canHandle() Method
The `canHandle()` method determines if your handler can process the given data.
### Strategy 1: MIME Type Detection (Recommended)
```typescript
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
if (typeof data === 'object' && 'filename' in data) {
const mimeType = this.getMimeType(data)
return this.mimeTypeMatches(mimeType, [
'application/x-myformat',
'application/myformat'
])
}
return false
}
```
✅ **Pros:** Automatic, comprehensive, works with 2000+ types
❌ **Cons:** Requires filename
### Strategy 2: Magic Byte Detection
```typescript
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
if (Buffer.isBuffer(data)) {
// Check magic bytes
return (
data[0] === 0x50 && // 'P'
data[1] === 0x4B && // 'K'
data[2] === 0x03 &&
data[3] === 0x04 // ZIP signature
)
}
return false
}
```
✅ **Pros:** Works without filename, robust
❌ **Cons:** Requires knowledge of format structure
### Strategy 3: Combined Approach
```typescript
canHandle(data: Buffer | string | { filename?: string; ext?: string }): boolean {
// Try MIME type first
if (typeof data === 'object' && 'filename' in data) {
const mimeType = this.getMimeType(data)
if (this.mimeTypeMatches(mimeType, ['application/x-myformat'])) {
return true
}
}
// Fallback to magic bytes
if (Buffer.isBuffer(data)) {
return this.checkMagicBytes(data)
}
return false
}
```
✅ **Pros:** Robust, works in all scenarios
❌ **Cons:** More complex
---
## process() Method
The `process()` method extracts structured data from the file.
### Return Format
```typescript
interface ProcessedData {
/** Format identifier */
format: string
/** Array of extracted entities */
data: Array<Record<string, any>>
/** Metadata about processing */
metadata: {
rowCount: number
fields: string[]
processingTime: number
[key: string]: any
}
/** Original filename (optional) */
filename?: string
}
```
### Example: CAD File Handler
```typescript
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
const startTime = Date.now()
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
// Parse CAD file
const cad = await this.parseCAD(buffer)
// Extract entities
const entities = []
// Main CAD document
entities.push({
type: 'CADDocument',
filename: options.filename,
units: cad.units,
bounds: cad.bounds
})
// Layers
for (const layer of cad.layers) {
entities.push({
type: 'CADLayer',
name: layer.name,
color: layer.color,
visible: layer.visible,
objectCount: layer.objects.length
})
}
// Objects
for (const obj of cad.objects) {
entities.push({
type: 'CADObject',
objectType: obj.type,
layer: obj.layer,
geometry: obj.geometry,
properties: obj.properties
})
}
return {
format: 'cad',
data: entities,
metadata: {
rowCount: entities.length,
fields: ['type', 'name', 'geometry', 'properties'],
processingTime: Date.now() - startTime,
layerCount: cad.layers.length,
objectCount: cad.objects.length,
units: cad.units
},
filename: options.filename
}
}
```
### Best Practices
1. **Always track processing time:**
```typescript
const startTime = Date.now()
// ... processing ...
metadata.processingTime = Date.now() - startTime
```
2. **Include rich metadata:**
```typescript
metadata: {
rowCount: entities.length,
fields: ['type', 'name', ...],
processingTime: 123,
// Format-specific metadata
layerCount: 5,
objectCount: 150,
version: '2.0'
}
```
3. **Handle errors gracefully:**
```typescript
try {
const parsed = this.parse(buffer)
return { format: 'myformat', data: parsed, ... }
} catch (error) {
throw new Error(
`Failed to parse myformat: ${error instanceof Error ? error.message : String(error)}`
)
}
```
4. **Support progress reporting (optional):**
```typescript
if (options.progressHooks?.onCurrentItem) {
options.progressHooks.onCurrentItem(`Processing layer ${i}/${total}`)
}
```
---
## FormatHandlerRegistry
### Global Registry
Use the global registry for application-wide handlers:
```typescript
import { globalHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
globalHandlerRegistry.registerHandler({
name: 'myformat',
mimeTypes: ['application/x-myformat'],
extensions: ['.myf'],
loader: async () => new MyFormatHandler()
})
```
### Local Registry
Create a local registry for scoped handlers:
```typescript
import { FormatHandlerRegistry } from '@soulcraft/brainy/augmentations/intelligentImport'
const registry = new FormatHandlerRegistry()
registry.registerHandler({ ... })
```
### Lazy Loading
Handlers are lazy-loaded for performance:
```typescript
globalHandlerRegistry.registerHandler({
name: 'heavy',
mimeTypes: ['application/x-heavy'],
extensions: ['.heavy'],
loader: async () => {
// Only loaded when first needed
const { HeavyHandler } = await import('./HeavyHandler.js')
return new HeavyHandler()
}
})
```
### Getting Handlers
```typescript
// By filename (automatic MIME detection)
const handler = await registry.getHandler('data.myf')
// By MIME type
const handler = await registry.getHandlerByMimeType('application/x-myformat')
// By extension
const handler = await registry.getHandlerByExtension('.myf')
// By name
const handler = await registry.getHandlerByName('myformat')
```
---
## Real-World Examples
### Example 1: Video Metadata Extractor
```typescript
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
import ffmpeg from 'fluent-ffmpeg'
class VideoHandler extends BaseFormatHandler {
readonly format = 'video'
canHandle(data) {
const mimeType = this.getMimeType(data)
return this.mimeTypeMatches(mimeType, ['video/*'])
}
async process(data, options) {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
// Extract video metadata with ffmpeg
const metadata = await this.extractVideoMetadata(buffer)
return {
format: 'video',
data: [{
type: 'Video',
duration: metadata.duration,
codec: metadata.codec,
resolution: metadata.resolution,
frameRate: metadata.frameRate,
bitrate: metadata.bitrate,
audioTracks: metadata.audioTracks,
subtitles: metadata.subtitles
}],
metadata: {
rowCount: 1,
fields: ['type', 'duration', 'codec', 'resolution'],
processingTime: metadata.processingTime
}
}
}
private async extractVideoMetadata(buffer: Buffer) {
// Use ffmpeg to extract metadata
return new Promise((resolve, reject) => {
ffmpeg(buffer)
.ffprobe((err, data) => {
if (err) reject(err)
else resolve(this.parseFFmpegOutput(data))
})
})
}
}
```
### Example 2: Git Repository Parser
```typescript
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
import { simpleGit } from 'simple-git'
class GitRepoHandler extends BaseFormatHandler {
readonly format = 'git-repo'
canHandle(data) {
// Check for .git directory
if (typeof data === 'object' && 'filename' in data) {
return data.filename?.includes('.git') || false
}
return false
}
async process(data, options) {
const repoPath = options.filename || ''
const git = simpleGit(repoPath)
// Extract commits
const log = await git.log()
const commits = log.all.map(commit => ({
type: 'GitCommit',
hash: commit.hash,
message: commit.message,
author: commit.author_name,
date: commit.date
}))
// Extract branches
const branchSummary = await git.branchLocal()
const branches = Object.keys(branchSummary.branches).map(name => ({
type: 'GitBranch',
name,
current: branchSummary.current === name
}))
return {
format: 'git-repo',
data: [...commits, ...branches],
metadata: {
rowCount: commits.length + branches.length,
fields: ['type', 'hash', 'message', 'author'],
processingTime: Date.now() - startTime,
commitCount: commits.length,
branchCount: branches.length
}
}
}
}
```
### Example 3: Database Schema Importer
```typescript
import { BaseFormatHandler } from '@soulcraft/brainy/augmentations/intelligentImport'
import { Client } from 'pg'
class PostgreSQLSchemaHandler extends BaseFormatHandler {
readonly format = 'postgres-schema'
canHandle(data) {
if (typeof data === 'object' && 'filename' in data) {
return data.filename?.endsWith('.sql') || false
}
return false
}
async process(data, options) {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
const sql = buffer.toString('utf-8')
// Parse SQL or connect to database
const schema = await this.parseSchema(sql)
const entities = []
// Tables
for (const table of schema.tables) {
entities.push({
type: 'Table',
name: table.name,
schema: table.schema,
columnCount: table.columns.length
})
// Columns
for (const column of table.columns) {
entities.push({
type: 'Column',
name: column.name,
table: table.name,
dataType: column.dataType,
nullable: column.nullable,
primaryKey: column.primaryKey
})
}
}
// Foreign keys
for (const fk of schema.foreignKeys) {
entities.push({
type: 'ForeignKey',
from: `${fk.fromTable}.${fk.fromColumn}`,
to: `${fk.toTable}.${fk.toColumn}`
})
}
return {
format: 'postgres-schema',
data: entities,
metadata: {
rowCount: entities.length,
fields: ['type', 'name', 'table', 'dataType'],
processingTime: Date.now() - startTime,
tableCount: schema.tables.length,
columnCount: schema.tables.reduce((sum, t) => sum + t.columns.length, 0),
foreignKeyCount: schema.foreignKeys.length
}
}
}
}
```
---
## Creating Premium Augmentations
Package your handler as a premium augmentation:
```typescript
// @yourcompany/brainy-cad-importer
import { BaseAugmentation } from '@soulcraft/brainy'
import { CADHandler } from './CADHandler.js'
export class CADImportAugmentation extends BaseAugmentation {
readonly name = 'cad-import'
readonly timing = 'before'
readonly operations = ['import', 'importFile']
private handler: CADHandler
constructor(config = {}) {
super(config)
this.handler = new CADHandler()
}
async execute(operation, params, next) {
// Check if this is a CAD file
if (this.isCADFile(params)) {
const processed = await this.handler.process(params.data, params.options)
params.data = processed.data
params.metadata = { ...params.metadata, ...processed.metadata }
}
return next()
}
private isCADFile(params: any): boolean {
return this.handler.canHandle(params.data || params)
}
}
// Usage:
// npm install @yourcompany/brainy-cad-importer
// brain.addAugmentation(new CADImportAugmentation())
```
---
## Testing
```typescript
import { describe, it, expect } from 'vitest'
import { MyFormatHandler } from './MyFormatHandler.js'
describe('MyFormatHandler', () => {
let handler: MyFormatHandler
beforeEach(() => {
handler = new MyFormatHandler()
})
describe('canHandle', () => {
it('should handle .myf files', () => {
expect(handler.canHandle({ filename: 'data.myf' })).toBe(true)
})
it('should handle by MIME type', () => {
expect(handler.canHandle({ filename: 'data.myformat' })).toBe(true)
})
it('should reject non-myformat files', () => {
expect(handler.canHandle({ filename: 'data.txt' })).toBe(false)
})
})
describe('process', () => {
it('should extract structured data', async () => {
const testData = Buffer.from('MY format data')
const result = await handler.process(testData)
expect(result.format).toBe('myformat')
expect(result.data).toHaveLength(1)
expect(result.metadata.processingTime).toBeGreaterThan(0)
})
it('should handle errors gracefully', async () => {
const invalidData = Buffer.from('invalid')
await expect(handler.process(invalidData)).rejects.toThrow()
})
})
})
```
---
## Best Practices
1. **Always extend BaseFormatHandler** - provides MIME detection and utilities
2. **Use MIME types for routing** - automatic, comprehensive, maintainable
3. **Lazy load heavy dependencies** - better performance
4. **Extract rich metadata** - make data queryable in knowledge graph
5. **Handle errors gracefully** - fail fast with clear error messages
6. **Test thoroughly** - test canHandle() and process() with real data
7. **Document your format** - explain what data is extracted and how
8. **Follow ProcessedData format** - ensures compatibility with Brainy
---
## See Also
- [ImageHandler source](../../src/augmentations/intelligentImport/handlers/imageHandler.ts) - Reference implementation
- [BaseFormatHandler source](../../src/augmentations/intelligentImport/handlers/base.ts) - Base class
- [FormatHandlerRegistry source](../../src/augmentations/intelligentImport/FormatHandlerRegistry.ts) - Registry implementation
- [Augmentations Guide](./AUGMENTATIONS.md) - Creating augmentations

204
docs/augmentations/README.md
View file

@ -1,204 +0,0 @@
# Brainy Augmentations
Augmentations are the core extensibility mechanism in Brainy. They allow you to modify, enhance, and extend Brainy's behavior without changing the core code.
## Core Principle: One Interface, Infinite Possibilities
Every augmentation implements the same simple `BrainyAugmentation` interface:
```typescript
interface BrainyAugmentation {
name: string
timing: 'before' | 'after' | 'around' | 'replace'
operations: string[]
priority: number
initialize(context): Promise<void>
execute(operation, params, next): Promise<any>
}
```
This single interface can handle EVERYTHING - from adding AI capabilities to exposing APIs to replacing storage backends.
## Available Augmentations
### 🧠 Data Processing
Augmentations that enhance how data is processed and stored.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **NeuralImportAugmentation** | AI-powered entity and relationship extraction | `before` | ✅ Production |
| **EntityRegistryAugmentation** | High-performance entity deduplication | `before` | ✅ Production |
| **BatchProcessingAugmentation** | Optimizes bulk operations | `around` | ✅ Production |
| **IntelligentVerbScoringAugmentation** | Learns relationship importance over time | `after` | ✅ Production |
### 🔌 External Connections (Synapses)
Connect Brainy to external services and data sources.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **NotionSynapse** | Sync with Notion databases | `after` | 📝 Example |
| **SalesforceSynapse** | Connect to Salesforce CRM | `after` | 📝 Example |
| **SlackSynapse** | Import Slack conversations | `after` | 📝 Example |
| **GoogleDriveSynapse** | Sync Google Drive documents | `after` | 📝 Example |
### 🌐 API Exposure
Expose Brainy through various protocols.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **APIServerAugmentation** | REST, WebSocket, and MCP server | `after` | ✅ Production |
| **GraphQLAugmentation** | GraphQL API endpoint | `after` | 🚧 Planned |
| **gRPCAugmentation** | gRPC service | `after` | 🚧 Planned |
### 💾 Storage Backends
Replace or enhance the storage layer.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **RedisAugmentation** | Redis caching layer | `around` | 📝 Example |
| **PostgresAugmentation** | PostgreSQL persistence | `replace` | 📝 Example |
> Brainy 8.0 ships `filesystem` and `memory` adapters out of the box. For off-site backup of the filesystem artifact, snapshot `rootDirectory` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, `tar`).
### 🔄 Real-time & Sync
Handle real-time updates and synchronization.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **WebSocketConduitAugmentation** | WebSocket client connections | `after` | ⚠️ Legacy |
| **ServerSearchAugmentation** | Connect to remote Brainy servers | `after` | ⚠️ Legacy |
| **TeamCoordinationAugmentation** | Multi-agent synchronization | `after` | 📝 Example |
### 🛡️ Infrastructure
Core infrastructure and reliability features.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **RequestDeduplicatorAugmentation** | Prevent duplicate concurrent requests | `before` | ✅ Production |
| **TransactionAugmentation** | ACID transaction support | `around` | 🚧 Planned |
| **CacheAugmentation** | Multi-level caching | `around` | ✅ Production |
### 📊 Monitoring & Analytics
Track and analyze Brainy's behavior.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **MetricsAugmentation** | Prometheus metrics | `after` | 📝 Example |
| **LoggingAugmentation** | Structured logging | `after` | 📝 Example |
| **TracingAugmentation** | Distributed tracing | `around` | 🚧 Planned |
### 🤖 AI & Chat
AI-powered interfaces and chat capabilities.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **ChatInterfaceAugmentation** | Natural language interface | `before` | 📝 Example |
| **MCPAgentMemoryAugmentation** | AI agent memory via MCP | `after` | 📝 Example |
| **LLMQueryAugmentation** | LLM-enhanced queries | `before` | 📝 Example |
### 📈 Visualization
Visual representations of data.
| Augmentation | Description | Timing | Status |
|-------------|-------------|--------|--------|
| **GraphVisualizationAugmentation** | Real-time graph visualization | `after` | 📝 Example |
| **DashboardAugmentation** | Web-based dashboard | `after` | 🚧 Planned |
## Status Legend
- ✅ **Production**: Fully implemented and tested
- 📝 **Example**: Example implementation available
- 🚧 **Planned**: On the roadmap
- ⚠️ **Legacy**: Being replaced by newer augmentations
## Using Augmentations
### Zero-Config Approach
```typescript
const brain = new Brainy()
// Just register augmentations - they work automatically!
brain.augmentations.register(new EntityRegistryAugmentation())
brain.augmentations.register(new APIServerAugmentation())
await brain.init()
```
### With Configuration
```typescript
const brain = new Brainy()
brain.augmentations.register(
new APIServerAugmentation({
port: 8080,
auth: { required: true }
})
)
await brain.init()
```
## Creating Custom Augmentations
See [Creating Custom Augmentations](./creating-augmentations.md) for a complete guide.
Quick example:
```typescript
class MyAugmentation extends BaseAugmentation {
readonly name = 'my-augmentation'
readonly timing = 'after'
readonly operations = ['add', 'search']
readonly priority = 50
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
console.log(`Before ${operation}`)
const result = await next()
console.log(`After ${operation}`)
return result
}
}
```
## Augmentation Timing
### `before`
Executes before the main operation. Used for:
- Input validation
- Data transformation
- Authentication checks
### `after`
Executes after the main operation. Used for:
- Broadcasting updates
- Syncing to external services
- Logging and metrics
### `around`
Wraps the main operation. Used for:
- Transactions
- Caching
- Error handling
### `replace`
Completely replaces the main operation. Used for:
- Alternative storage backends
- Mock implementations
- Proxy operations
## Priority System
Higher numbers execute first:
- **100**: Critical system operations
- **50**: Performance optimizations
- **10**: Enhancement features
- **1**: Optional features
## Related Documentation
- [API Server Augmentation](./api-server.md) - Complete API server documentation
- [Creating Augmentations](./creating-augmentations.md) - How to build your own
- [Augmentation Examples](../AUGMENTATION-EXAMPLES.md) - Real-world examples
- [Architecture Overview](../COMPLETE-ARCHITECTURE-VISION.md) - System architecture

429
docs/augmentations/api-server.md
View file

@ -1,429 +0,0 @@
# API Server Augmentation
## Overview
The `APIServerAugmentation` is a powerful augmentation that exposes your Brainy instance through REST, WebSocket, and MCP (Model Context Protocol) APIs. It transforms Brainy into a full-featured API server with zero configuration required.
## Features
### 🌐 REST API
Complete CRUD operations and advanced queries through HTTP endpoints.
### 🔌 WebSocket Server
Real-time bidirectional communication with automatic operation broadcasting.
### 🧠 MCP Integration
Built-in Model Context Protocol support for AI agent communication.
### 📊 Operation Broadcasting
Automatically broadcasts all Brainy operations to subscribed WebSocket clients.
### 🔒 Optional Security
Built-in authentication and rate limiting when needed.
## Installation
The APIServerAugmentation is included in Brainy core. No additional installation required.
**Bun** (recommended): No additional dependencies needed - use `Bun.serve()` directly.
**Node.js**: Install optional Express dependencies:
```bash
npm install express cors ws
```
## Zero-Config Usage
```typescript
import { Brainy } from 'brainy'
import { APIServerAugmentation } from 'brainy/augmentations'
const brain = new Brainy()
// Register the API server augmentation
brain.augmentations.register(new APIServerAugmentation())
await brain.init()
// Server is now running at http://localhost:3000
console.log('API Server ready!')
console.log('REST: http://localhost:3000/api/*')
console.log('WebSocket: ws://localhost:3000/ws')
console.log('MCP: http://localhost:3000/api/mcp')
```
## Configuration Options
While zero-config works great, you can customize the server:
```typescript
const apiServer = new APIServerAugmentation({
enabled: true, // Enable/disable the server
port: 3000, // HTTP port
host: '0.0.0.0', // Bind address
cors: {
origin: '*', // CORS allowed origins
credentials: true // Allow credentials
},
auth: {
required: false, // Require authentication
apiKeys: [], // Valid API keys
bearerTokens: [] // Valid bearer tokens
},
rateLimit: {
windowMs: 60000, // Rate limit window (ms)
max: 100 // Max requests per window
}
})
```
## REST API Endpoints
### Health Check
```http
GET /health
```
Returns server status and basic metrics.
### Search
```http
POST /api/search
Content-Type: application/json
{
"query": "search text",
"limit": 10,
"options": {}
}
```
### Add Data
```http
POST /api/add
Content-Type: application/json
{
"content": "data to add",
"metadata": {
"key": "value"
}
}
```
### Get by ID
```http
GET /api/get/:id
```
### Delete
```http
DELETE /api/delete/:id
```
### Create Relationship
```http
POST /api/relate
Content-Type: application/json
{
"source": "id1",
"target": "id2",
"verb": "relates_to",
"metadata": {}
}
```
### Complex Queries
```http
POST /api/find
Content-Type: application/json
{
"where": { "type": "document" },
"like": "machine learning",
"limit": 10
}
```
### Clustering
```http
POST /api/cluster
Content-Type: application/json
{
"algorithm": "kmeans",
"options": {
"k": 5
}
}
```
### Statistics
```http
GET /api/stats
```
### Operation History
```http
GET /api/history
```
## WebSocket API
### Connection
```javascript
const ws = new WebSocket('ws://localhost:3000/ws')
ws.onopen = () => {
console.log('Connected to Brainy WebSocket')
}
ws.onmessage = (event) => {
const msg = JSON.parse(event.data)
console.log('Received:', msg)
}
```
### Subscribe to Operations
```javascript
ws.send(JSON.stringify({
type: 'subscribe',
operations: ['all'] // or specific: ['add', 'search', 'delete']
}))
```
### Search via WebSocket
```javascript
ws.send(JSON.stringify({
type: 'search',
query: 'your search',
limit: 10,
requestId: 'unique-id'
}))
```
### Add Data via WebSocket
```javascript
ws.send(JSON.stringify({
type: 'add',
content: 'data to add',
metadata: {},
requestId: 'unique-id'
}))
```
### Operation Broadcasts
When subscribed, you'll receive real-time updates:
```javascript
{
"type": "operation",
"operation": "add",
"params": { /* sanitized parameters */ },
"timestamp": 1234567890,
"duration": 15
}
```
## MCP (Model Context Protocol)
The MCP endpoint allows AI agents to interact with Brainy:
```http
POST /api/mcp
Content-Type: application/json
{
"method": "search",
"params": {
"query": "find documents about AI"
}
}
```
## Authentication
When authentication is enabled:
### API Key
```http
GET /api/stats
X-API-Key: your-api-key
```
### Bearer Token
```http
GET /api/stats
Authorization: Bearer your-token
```
## Environment Support
### Bun ✅ (Recommended)
Native performance with Bun.serve() - no Express required. Works with `bun --compile` for single-binary deployment.
```typescript
// Simple Bun server with Brainy
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy()
await brain.init()
Bun.serve({
port: 3000,
async fetch(req) {
const url = new URL(req.url)
if (url.pathname === '/api/search') {
const { query } = await req.json()
const results = await brain.search(query)
return Response.json(results)
}
return new Response('Not Found', { status: 404 })
}
})
```
### Node.js ✅
Full support with Express, WebSocket, and all features.
### Deno 🚧
Planned support using Deno.serve() or oak framework.
### Browser/Service Worker 🚧
Planned support for intercepting fetch() calls locally.
## How It Works
The APIServerAugmentation hooks into Brainy's augmentation pipeline:
1. **Timing**: Executes `after` operations complete
2. **Operations**: Monitors `all` operations
3. **Broadcasting**: Sends operation details to subscribed clients
4. **History**: Maintains operation history (last 1000 operations)
## Example: Multi-Client Sync
```typescript
// Server
const brain = new Brainy()
brain.augmentations.register(new APIServerAugmentation())
await brain.init()
// Client 1 - WebSocket subscriber
const ws1 = new WebSocket('ws://localhost:3000/ws')
ws1.onopen = () => {
ws1.send(JSON.stringify({
type: 'subscribe',
operations: ['add', 'delete']
}))
}
ws1.onmessage = (e) => {
console.log('Client 1 received update:', JSON.parse(e.data))
}
// Client 2 - REST API user
fetch('http://localhost:3000/api/add', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
content: 'New data',
metadata: { source: 'client2' }
})
})
// Client 1 automatically receives notification!
```
## Performance Considerations
- **Operation History**: Limited to last 1000 operations
- **WebSocket Heartbeat**: Every 30 seconds
- **Client Timeout**: 60 seconds of inactivity
- **Parameter Sanitization**: Sensitive fields removed, large content truncated
- **Rate Limiting**: In-memory tracking (use Redis in production)
## Security Notes
1. **Default Configuration**: No auth, open CORS - suitable for development
2. **Production**: Enable auth, configure CORS, use HTTPS
3. **Sensitive Data**: Parameters are sanitized before broadcasting
4. **Rate Limiting**: Basic in-memory implementation included
## Comparison with Previous Implementations
The APIServerAugmentation unifies and replaces:
- `BrainyMCPBroadcast` - Node-specific WebSocket/HTTP server
- `WebSocketConduitAugmentation` - WebSocket client functionality
- `ServerSearchAugmentations` - Remote Brainy connections
Benefits of the unified approach:
- Single augmentation for all API needs
- Consistent interface across protocols
- Automatic operation broadcasting
- Environment-aware implementation
- Zero-configuration philosophy
## Advanced Usage
### Custom Operation Filtering
```typescript
class FilteredAPIServer extends APIServerAugmentation {
shouldExecute(operation: string, params: any): boolean {
// Don't broadcast sensitive operations
if (operation === 'delete' && params.sensitive) {
return false
}
return true
}
}
```
### Integration with Other Augmentations
```typescript
const brain = new Brainy()
// Stack augmentations for complete system
brain.augmentations.register(new EntityRegistryAugmentation()) // Dedup
brain.augmentations.register(new APIServerAugmentation()) // API
await brain.init()
// All augmentations work together seamlessly!
```
## Troubleshooting
### Server won't start
- Check if port is already in use
- Verify Node.js dependencies are installed: `npm install express cors ws`
- Check console for error messages
### WebSocket connections drop
- Ensure heartbeat responses are handled
- Check for proxy/firewall issues
- Verify CORS configuration
### Authentication not working
- Ensure `auth.required` is set to `true`
- Verify API keys or bearer tokens are correctly configured
- Check request headers are properly formatted
## Future Enhancements
- [ ] Deno server implementation
- [ ] Service Worker implementation
- [ ] GraphQL endpoint
- [ ] gRPC support
- [ ] Built-in SSL/TLS
- [ ] Redis-based rate limiting
- [ ] Prometheus metrics endpoint
- [ ] OpenAPI/Swagger documentation
## Related Documentation
- [Augmentation System Overview](../AUGMENTATION-SYSTEM.md)
- [BrainyAugmentation Interface](./brainy-augmentation.md)
- [MCP Integration](../mcp/README.md)
- [Zero-Config Philosophy](../ZERO-CONFIG.md)

6
docs/concepts/consistency-model.md
View file

@ -41,7 +41,7 @@ by a dedicated test in `tests/integration/db-mvcc.test.ts`.
A **monotonic generation counter** is the store's logical clock:
- It advances **once per committed `transact()` batch** and once per
single-operation write (`add`/`update`/`delete`/`relate`/…).
single-operation write (`add`/`update`/`remove`/`relate`/…).
- `brain.generation()` reads it; it is persisted in the data directory and
**never reissued** — not across restarts, and not across `restore()`
(the counter is floored at its pre-restore value).
@ -175,7 +175,7 @@ identical on both paths.
### History granularity — the honest limit
Generation *records* are written per `transact()` batch only.
Single-operation writes (`add`/`update`/`delete`/`relate`/… outside
Single-operation writes (`add`/`update`/`remove`/`relate`/… outside
`transact()`) advance the generation counter — so watermarks and CAS stay
sound — but do **not** stage before-images: they remain visible through
earlier pins and are not reported by `db.since()`. Code that needs pinned
@ -284,7 +284,7 @@ enforce the contract:
`update({ metadata: { confidence: 0.9 } })` therefore behaves exactly
like `update({ confidence: 0.9 })`.
3. **Read time** — every read path (`get`, `find`, `search`,
`getRelations`, batch reads, and historical `asOf()` materialization)
`related`, batch reads, and historical `asOf()` materialization)
surfaces reserved fields **only at top level**: `entity.metadata` and
`relation.metadata` contain only your custom fields, always.

2
docs/features/v3-features.md
View file

@ -152,7 +152,7 @@ Optimized bulk processing:
// Parallel processing with automatic batching
await brain.addMany(items) // <10ms per item
await brain.updateMany(updates)
await brain.deleteMany(filters)
await brain.removeMany(filters)
```
### Request Deduplication ✅

8
docs/guides/framework-integration.md
View file

@ -348,7 +348,7 @@ export default function RootLayout({ children }) {
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy({
storage: { type: 'filesystem', path: './data' }
storage: { type: 'filesystem', rootDirectory: './data' }
})
await brain.init()
@ -536,12 +536,12 @@ import { Brainy } from '@soulcraft/brainy'
export async function generateStaticProps() {
const brain = new Brainy({
storage: { type: 'filesystem', path: './content' }
storage: { type: 'filesystem', rootDirectory: './content' }
})
await brain.init()
// Build search index
const allContent = await brain.export()
// Build search index (paginate with { limit, offset } for larger stores)
const allContent = await brain.find({ limit: 1000 })
return {
props: { searchIndex: allContent }

15
docs/guides/import-anything.md
View file

@ -49,28 +49,23 @@ await brain.import(csv, { format: 'csv' })
### 📊 Import Excel - Multi-Sheet Support
```javascript
// Import entire Excel workbook
// Import entire Excel workbook — every sheet is processed automatically
await brain.import('sales-report.xlsx')
// ✨ Processes all sheets, preserves structure, infers types!
// Or specific sheets only
// Mirror the workbook into the VFS, grouped by sheet
await brain.import('data.xlsx', {
excelSheets: ['Customers', 'Orders']
vfsPath: '/imports/data',
groupBy: 'sheet'
})
// ✨ Multi-sheet data becomes interconnected entities!
```
### 📑 Import PDF - Text & Tables
```javascript
// Import PDF documents
// Import PDF documents — text and tables are extracted automatically
await brain.import('research-paper.pdf')
// ✨ Extracts text, detects tables, preserves metadata!
// With table extraction
await brain.import('report.pdf', {
pdfExtractTables: true
})
// ✨ Converts PDF tables to structured data automatically!
```
### 📝 Import YAML - File or String

13
docs/guides/import-flow.md
View file

@ -1173,21 +1173,22 @@ await this.graphIndex.addEdge(
```
**Benefits**:
- **O(1) relationship lookups**: `getRelations(entityId)` is instant
- **O(1) relationship lookups**: `related(entityId)` is instant
- **Bidirectional traversal**: Find incoming and outgoing edges
- **Type filtering**: Get only `CreatedBy` relationships
- **Global statistics**: Count relationships by type
**Query Examples**:
```typescript
// What did Mona Lisa create?
const outgoing = await brain.getRelations('ent_mona_lisa_...', { direction: 'outgoing' })
// What did Mona Lisa create? (outgoing edges)
const outgoing = await brain.related({ from: 'ent_mona_lisa_...' })
// What created Mona Lisa?
const incoming = await brain.getRelations('ent_mona_lisa_...', { direction: 'incoming' })
// What created Mona Lisa? (incoming edges)
const incoming = await brain.related({ to: 'ent_mona_lisa_...' })
// Get only CreatedBy relationships
const createdBy = await brain.getRelations('ent_mona_lisa_...', {
const createdBy = await brain.related({
from: 'ent_mona_lisa_...',
type: VerbType.CreatedBy
})
```

2
docs/guides/import-quick-reference.md
View file

@ -351,7 +351,7 @@ const result = await brain.import(file)
const products = await brain.find({ type: 'Product' })
// Get entity relationships
const relations = await brain.getRelations(products[0].id)
const relations = await brain.related(products[0].id)
// Search VFS
const vfsFiles = await brain.vfs().find(result.vfs.rootPath + '/**/*.json')

2
docs/guides/inspection.md
View file

@ -134,7 +134,7 @@ console.log(health.overall)
await reader.close()
```
Every mutation method (`add`, `update`, `delete`, `relate`, `transact`,
Every mutation method (`add`, `update`, `remove`, `relate`, `transact`,
`restore`, ...) throws on a read-only instance with a clear message.
## Backups

27
docs/guides/installation.md
View file

@ -5,10 +5,10 @@ public: true
category: getting-started
template: guide
order: 1
description: Install Brainy with npm, bun, yarn, or pnpm. Works in Node.js 22+, Bun 1.0+, and browser (OPFS). TypeScript included.
description: Install Brainy with npm, bun, yarn, or pnpm. Works in Node.js 22+ and Bun 1.0+ (server-only since 8.0). TypeScript included.
next:
- getting-started/quick-start
- concepts/zero-config
- guides/storage-adapters
---
# Installation
@ -53,28 +53,18 @@ npm install @soulcraft/cortex
```typescript
import { Brainy } from '@soulcraft/brainy'
import { registerCortex } from '@soulcraft/cortex'
registerCortex() // activates native acceleration globally
const brain = new Brainy()
await brain.init()
const brain = new Brainy({ plugins: ['@soulcraft/cortex'] })
await brain.init() // native providers registered during init
```
Cortex delivers a **5.2x geometric mean speedup** — see [Brainy vs Cortex](/docs/cortex/comparison) for measured benchmarks.
## Browser (OPFS)
## Server-only since 8.0
Brainy works in the browser using the Origin Private File System:
```typescript
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy({ storage: 'opfs' })
await brain.init()
```
No server required. Data persists across page refreshes in the browser's private storage.
Brainy 8.0 runs on Node.js 22+ and Bun 1.0+. Browser support (OPFS storage,
Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line
remains available on npm if you need it.
## TypeScript
@ -96,5 +86,4 @@ const id = await brain.add({
## Next Steps
- [Quick Start](/docs/getting-started/quick-start) — build your first knowledge graph in 60 seconds
- [Zero Configuration](/docs/concepts/zero-config) — understand what Brainy auto-detects
- [Storage Adapters](/docs/guides/storage-adapters) — choose the right storage for your deployment

2
docs/guides/migrating-to-v4.md
View file

@ -252,7 +252,7 @@ const result = await brain.import('./glossary.xlsx', {
// - "part_of"
// - "related_to"
const relations = await brain.getRelations({ limit: 100 })
const relations = await brain.related({ limit: 100 })
const types = new Set(relations.map(r => r.label))
console.log(types)
// Set { 'capital_of', 'guards', 'located_in', 'related_to' }

2
docs/guides/neural-api.md
View file

@ -367,5 +367,5 @@ const brain = new Brainy({
## Next Steps
- Explore [Triple Intelligence](../architecture/triple-intelligence.md) for combined vector + graph + metadata queries
- Learn about [Augmentations](../augmentations/README.md) to extend Neural API
- Learn about [Plugins](../PLUGINS.md) to extend Brainy with native providers
- See [API Reference](../api/README.md) for complete method documentation

2
docs/guides/storage-adapters.md
View file

@ -8,7 +8,7 @@ order: 2
description: "Two adapters cover every deployment: in-memory for tests + ephemeral workloads, filesystem for everything that needs to persist. Both share one on-disk contract, including generational history and snapshots. Cloud backup is operator tooling, not a built-in adapter."
next:
- guides/plugins
- concepts/zero-config
- concepts/consistency-model
---
# Storage Adapters

8
docs/guides/subtypes-and-facets.md
View file

@ -301,14 +301,14 @@ await brain.relate({
```typescript
// Direct reports — fast path filter (column-store hit, not metadata fallback)
const direct = await brain.getRelations({
const direct = await brain.related({
from: ceoId,
type: VerbType.ReportsTo,
subtype: 'direct'
})
// Set membership
const allReports = await brain.getRelations({
const allReports = await brain.related({
from: ceoId,
type: VerbType.ReportsTo,
subtype: ['direct', 'dotted-line']
@ -552,8 +552,8 @@ Brainy 8.0 ships:
- `brain.relate({ ..., subtype: 'value' })` — write the field
- `brain.updateRelation({ id, subtype, type?, weight?, ... })` — change a relationship
- `brain.getRelations({ subtype })` — filter (fast path)
- `brain.getRelations({ subtype: ['a', 'b'] })` — set membership
- `brain.related({ subtype })` — filter (fast path)
- `brain.related({ subtype: ['a', 'b'] })` — set membership
- `brain.find({ connected: { via, subtype, depth } })` — traversal filter
- `brain.counts.byRelationshipSubtype(verb, subtype?)` — O(1) counts
- `brain.counts.topRelationshipSubtypes(verb, n?)` — top N by count

8
docs/transactions.md
View file

@ -23,7 +23,7 @@ Brainy's transaction system provides **atomic operations** with automatic rollba
- **Atomicity**: All operations succeed or all rollback
- **Consistency**: Indexes and storage remain consistent
- **Automatic**: Transparently used by all `brain.add()`, `brain.update()`, `brain.delete()`, and `brain.relate()` operations
- **Automatic**: Transparently used by all `brain.add()`, `brain.update()`, `brain.remove()`, and `brain.relate()` operations
- **Composable**: `brain.transact()` runs a multi-write batch through the same machinery as exactly one atomic commit
## Architecture
@ -296,7 +296,7 @@ await brain.relate({
})
// Delete person (atomic - deletes entity + relationships)
await brain.delete(personId)
await brain.remove(personId)
// If delete fails, both entity and relationships remain
```
@ -363,7 +363,7 @@ for the history-granularity contract.
// ✅ Recommended: Use Brainy's API (transactions automatic)
await brain.add({ data, type })
await brain.update({ id, data })
await brain.delete(id)
await brain.remove(id)
// ❌ Avoid: Direct storage access bypasses transactions
await brain.storage.saveNoun(noun) // No transaction protection
@ -565,4 +565,4 @@ Brainy's transaction system provides **production-ready atomic operations** with
- ✅ **Compatible**: Works with all storage adapters and features
- ✅ **Coordinated**: Per-entity `ifRev` and whole-store `ifAtGeneration` CAS reject conflicting batches before anything is staged
Start using transactions today - they're already built into `brain.add()`, `brain.update()`, `brain.delete()`, and `brain.relate()` — and reach for `brain.transact()` whenever several writes must land together.
Start using transactions today - they're already built into `brain.add()`, `brain.update()`, `brain.remove()`, and `brain.relate()` — and reach for `brain.transact()` whenever several writes must land together.

2
docs/vfs/README.md
View file

@ -8,7 +8,7 @@
- **[VFS Core](VFS_CORE.md)** - Complete filesystem architecture and API
- **[Semantic VFS](SEMANTIC_VFS.md)** - Multi-dimensional file access (6 semantic dimensions)
- **[Neural Extraction](NEURAL_EXTRACTION.md)** - AI-powered concept and entity extraction
- **[Examples & Scenarios](VFS_EXAMPLES_SCENARIOS.md)** - Real-world use cases and code
- **[Common Patterns](COMMON_PATTERNS.md)** - Real-world use cases and code
- **[VFS API Guide](VFS_API_GUIDE.md)** - Complete API reference
## What is Brainy VFS?

6
docs/vfs/SEMANTIC_VFS.md
View file

@ -400,9 +400,9 @@ Semantic VFS is built on Brainy's Triple Intelligence™:
### Real Implementations, Zero Mocks
Every component uses **production Brainy APIs**:
- `brain.find()` - Real metadata queries (brainy.ts:580)
- `brain.similar()` - Real HNSW search (brainy.ts:680)
- `brain.getRelations()` - Real graph traversal (brainy.ts:803)
- `brain.find()` - Real metadata queries
- `brain.similar()` - Real HNSW search
- `brain.related()` - Real graph traversal
- `MetadataIndexManager` - Real B-tree indexes
- `GraphAdjacencyIndex` - Real graph storage
- `HNSW Index` - Real vector search

4
docs/vfs/TROUBLESHOOTING.md
View file

@ -183,7 +183,7 @@ console.log('VFS type:', entity.metadata.vfsType)
### 3. Check Relationships
```javascript
const parentId = await vfs.resolvePath('/directory')
const relations = await brain.getRelations({
const relations = await brain.related({
from: parentId,
type: VerbType.Contains
})
@ -246,7 +246,7 @@ const tree = await vfs.getTreeStructure('/', {
If you encounter issues not covered here:
1. Check the [VFS API Guide](./VFS_API_GUIDE.md)
2. Review [VFS Examples](./VFS_EXAMPLES_SCENARIOS.md)
2. Review [Common Patterns](./COMMON_PATTERNS.md)
3. Look at [test files](../../tests/vfs/) for working examples
4. Report issues at [GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)

728
docs/vfs/USER_FUNCTIONS.md
View file

@ -1,728 +0,0 @@
# VFS User Functions - Templates and Examples
This document provides template functions that you can implement for domain-specific needs. These functions combine VFS primitives to solve common problems.
## Table of Contents
1. [Code Analysis Functions](#code-analysis-functions)
2. [Export Format Functions](#export-format-functions)
3. [Project Management Functions](#project-management-functions)
4. [Creative Writing Functions](#creative-writing-functions)
5. [Game Development Functions](#game-development-functions)
## Code Analysis Functions
### Get Dependency Graph
```javascript
/**
* Build a dependency graph for JavaScript/TypeScript projects
*/
async function getDependencyGraph(vfs, srcPath) {
const files = await vfs.readdir(srcPath, { recursive: true })
const graph = {}
for (const file of files) {
const filePath = `${srcPath}/${file}`
// Only process JS/TS files
if (file.match(/\.(js|ts|jsx|tsx)$/)) {
const content = await vfs.readFile(filePath)
const text = content.toString()
// Parse imports (basic regex, use proper AST parser for production)
const imports = []
const importRegex = /import\s+.*?\s+from\s+['"](.+?)['"]/g
const requireRegex = /require\(['"](.+?)['"]\)/g
let match
while ((match = importRegex.exec(text)) !== null) {
imports.push(match[1])
}
while ((match = requireRegex.exec(text)) !== null) {
imports.push(match[1])
}
graph[filePath] = imports
}
}
return graph
}
// Use it
const deps = await getDependencyGraph(vfs, '/src')
```
### Find Circular Dependencies
```javascript
/**
* Detect circular dependencies in your code
*/
async function findCircularDependencies(vfs, srcPath) {
const graph = await getDependencyGraph(vfs, srcPath)
const cycles = []
function detectCycle(node, visited = new Set(), stack = []) {
if (stack.includes(node)) {
const cycleStart = stack.indexOf(node)
cycles.push(stack.slice(cycleStart))
return
}
if (visited.has(node)) return
visited.add(node)
stack.push(node)
const dependencies = graph[node] || []
for (const dep of dependencies) {
// Resolve relative imports
const resolvedDep = resolvePath(node, dep)
if (graph[resolvedDep]) {
detectCycle(resolvedDep, visited, [...stack])
}
}
}
Object.keys(graph).forEach(node => detectCycle(node))
return cycles
}
```
### Find Untested Code
```javascript
/**
* Find source files without corresponding test files
*/
async function findUntestedCode(vfs, srcPath, testPath = null) {
testPath = testPath || srcPath.replace('/src', '/tests')
const sourceFiles = await vfs.readdir(srcPath, { recursive: true })
const testFiles = await vfs.readdir(testPath, { recursive: true }).catch(() => [])
const untestedFiles = []
for (const sourceFile of sourceFiles) {
if (!sourceFile.match(/\.(js|ts|jsx|tsx)$/)) continue
// Look for corresponding test file
const baseName = sourceFile.replace(/\.(js|ts|jsx|tsx)$/, '')
const hasTest = testFiles.some(testFile =>
testFile.includes(baseName) &&
testFile.match(/\.(test|spec)\.(js|ts|jsx|tsx)$/)
)
if (!hasTest) {
untestedFiles.push(`${srcPath}/${sourceFile}`)
}
}
return untestedFiles
}
```
### Find Similar Code (Duplicate Detection)
```javascript
/**
* Find potentially duplicate code using similarity scoring
*/
async function findSimilarCode(vfs, filePath, options = {}) {
const threshold = options.threshold || 0.8
const searchPath = options.searchPath || '/'
// Get the reference file content
const referenceContent = await vfs.readFile(filePath)
const referenceText = referenceContent.toString()
// Use VFS's semantic search
const similar = await vfs.findSimilar(filePath, {
limit: 10,
threshold
})
// Additionally, do structural comparison
const results = []
for (const match of similar) {
const matchContent = await vfs.readFile(match.path)
const matchText = matchContent.toString()
// Simple line-based similarity (use better algorithms in production)
const similarity = calculateSimilarity(referenceText, matchText)
if (similarity > threshold) {
results.push({
path: match.path,
similarity,
semanticScore: match.score
})
}
}
return results.sort((a, b) => b.similarity - a.similarity)
}
function calculateSimilarity(text1, text2) {
// Simple Jaccard similarity on lines
const lines1 = new Set(text1.split('\n').map(l => l.trim()).filter(l => l))
const lines2 = new Set(text2.split('\n').map(l => l.trim()).filter(l => l))
const intersection = new Set([...lines1].filter(x => lines2.has(x)))
const union = new Set([...lines1, ...lines2])
return intersection.size / union.size
}
```
## Export Format Functions
### Export to EPUB (for novels)
```javascript
/**
* Export a directory of markdown files to EPUB format
*/
async function exportToEpub(vfs, path, metadata = {}) {
// First get the markdown export
const markdown = await vfs.exportToMarkdown(path)
// You'll need an EPUB library like epub-gen
const Epub = require('epub-gen')
// Convert markdown chapters to EPUB format
const chapters = []
const files = await vfs.readdir(path, { recursive: true })
for (const file of files.sort()) {
if (file.endsWith('.md')) {
const content = await vfs.readFile(`${path}/${file}`)
const title = file.replace('.md', '').replace(/-/g, ' ')
chapters.push({
title: title,
data: content.toString()
})
}
}
const options = {
title: metadata.title || 'My Book',
author: metadata.author || 'Author',
chapters: chapters
}
return new Epub(options)
}
```
### Export to Static Site
```javascript
/**
* Export VFS content to static HTML site
*/
async function exportToStaticSite(vfs, sourcePath, options = {}) {
const json = await vfs.exportToJSON(sourcePath)
const html = []
html.push('<!DOCTYPE html>')
html.push('<html><head>')
html.push(`<title>${options.title || 'Documentation'}</title>`)
html.push('<style>/* Add your styles */</style>')
html.push('</head><body>')
function renderNode(node, name, depth = 0) {
const indent = ' '.repeat(depth)
if (node._meta?.type === 'file') {
html.push(`${indent}<article>`)
html.push(`${indent} <h${Math.min(depth + 2, 6)}>${name}</h${Math.min(depth + 2, 6)}>`)
if (typeof node._content === 'string') {
// Convert markdown to HTML if needed
html.push(`${indent} <pre>${escapeHtml(node._content)}</pre>`)
}
html.push(`${indent}</article>`)
} else if (node._meta?.type === 'directory') {
html.push(`${indent}<section>`)
html.push(`${indent} <h${Math.min(depth + 1, 6)}>${name}</h${Math.min(depth + 1, 6)}>`)
for (const [childName, childNode] of Object.entries(node)) {
if (!childName.startsWith('_')) {
renderNode(childNode, childName, depth + 1)
}
}
html.push(`${indent}</section>`)
}
}
renderNode(json, options.title || 'Root')
html.push('</body></html>')
return html.join('\n')
}
```
### Export to GraphQL Schema
```javascript
/**
* Generate GraphQL schema from VFS entities
*/
async function exportToGraphQLSchema(vfs) {
const entities = await vfs.listEntities()
const types = new Map()
// Group entities by type
for (const entity of entities) {
const type = entity.type || 'Unknown'
if (!types.has(type)) {
types.set(type, [])
}
types.get(type).push(entity)
}
// Generate schema
let schema = 'type Query {\n'
for (const [typeName, entities] of types) {
schema += ` get${typeName}(id: ID!): ${typeName}\n`
schema += ` list${typeName}s: [${typeName}!]!\n`
}
schema += '}\n\n'
// Generate types
for (const [typeName, entities] of types) {
schema += `type ${typeName} {\n`
schema += ' id: ID!\n'
// Infer fields from first entity
if (entities.length > 0) {
const sample = entities[0]
for (const [key, value] of Object.entries(sample)) {
if (key !== 'id') {
const fieldType = inferGraphQLType(value)
schema += ` ${key}: ${fieldType}\n`
}
}
}
schema += '}\n\n'
}
return schema
}
```
## Project Management Functions
### Get Project Insights
```javascript
/**
* Analyze project for insights and patterns
*/
async function getProjectInsights(vfs, projectPath) {
const stats = await vfs.getProjectStats(projectPath)
const todos = await vfs.getAllTodos(projectPath)
const timeline = await vfs.getTimeline({ limit: 100 })
// Analyze activity patterns
const activityByDay = {}
const activityByUser = {}
const activityByFile = {}
for (const event of timeline) {
const day = event.timestamp.toISOString().split('T')[0]
activityByDay[day] = (activityByDay[day] || 0) + 1
const user = event.user || 'system'
activityByUser[user] = (activityByUser[user] || 0) + 1
activityByFile[event.path] = (activityByFile[event.path] || 0) + 1
}
// Find hotspots (most edited files)
const hotspots = Object.entries(activityByFile)
.sort(([,a], [,b]) => b - a)
.slice(0, 10)
.map(([path, count]) => ({ path, edits: count }))
// Todo analysis
const todosByPriority = {}
const todosByStatus = {}
for (const todo of todos) {
todosByPriority[todo.priority] = (todosByPriority[todo.priority] || 0) + 1
todosByStatus[todo.status] = (todosByStatus[todo.status] || 0) + 1
}
return {
stats,
activity: {
byDay: activityByDay,
byUser: activityByUser,
hotspots
},
todos: {
total: todos.length,
byPriority: todosByPriority,
byStatus: todosByStatus,
highPriority: todos.filter(t => t.priority === 'high' && t.status === 'pending')
},
recommendations: generateRecommendations(stats, todos, hotspots)
}
}
function generateRecommendations(stats, todos, hotspots) {
const recommendations = []
if (stats.largestFile && stats.largestFile.size > 1024 * 1024) {
recommendations.push({
type: 'refactor',
message: `Consider splitting ${stats.largestFile.path} (${Math.round(stats.largestFile.size / 1024)}KB)`
})
}
if (todos.filter(t => t.priority === 'high' && t.status === 'pending').length > 5) {
recommendations.push({
type: 'priority',
message: 'You have many high-priority pending todos'
})
}
if (hotspots.length > 0 && hotspots[0].edits > 50) {
recommendations.push({
type: 'stability',
message: `${hotspots[0].path} changes frequently, consider stabilizing`
})
}
return recommendations
}
```
### Generate Sprint Report
```javascript
/**
* Generate a report for the current sprint
*/
async function generateSprintReport(vfs, sprintStart, sprintEnd = new Date()) {
const timeline = await vfs.getTimeline({
from: sprintStart,
to: sprintEnd
})
const todos = await vfs.getAllTodos()
// Group work by user
const workByUser = {}
for (const event of timeline) {
const user = event.user || 'system'
if (!workByUser[user]) {
workByUser[user] = {
commits: 0,
filesModified: new Set(),
linesChanged: 0
}
}
workByUser[user].commits++
workByUser[user].filesModified.add(event.path)
}
// Calculate completion rate
const completedTodos = todos.filter(t => t.status === 'completed').length
const totalTodos = todos.length
const completionRate = totalTodos > 0 ? (completedTodos / totalTodos * 100).toFixed(1) : 0
return {
period: {
start: sprintStart,
end: sprintEnd,
days: Math.ceil((sprintEnd - sprintStart) / (1000 * 60 * 60 * 24))
},
team: Object.entries(workByUser).map(([user, work]) => ({
user,
commits: work.commits,
filesModified: work.filesModified.size
})),
todos: {
completed: completedTodos,
total: totalTodos,
completionRate: `${completionRate}%`,
remaining: todos.filter(t => t.status === 'pending')
},
velocity: {
commitsPerDay: (timeline.length / 7).toFixed(1),
todosPerDay: (completedTodos / 7).toFixed(1)
}
}
}
```
## Creative Writing Functions
### Track Character Arcs
```javascript
/**
* Track how characters evolve throughout a story
*/
async function trackCharacterArc(vfs, characterName, storyPath = '/') {
// Find the character entity
const entities = await vfs.searchEntities({
type: 'character',
name: characterName
})
if (entities.length === 0) {
throw new Error(`Character ${characterName} not found`)
}
const character = entities[0]
const occurrences = await vfs.findEntityOccurrences(character.id)
// Analyze each appearance
const arc = []
for (const occurrence of occurrences) {
const content = await vfs.readFile(occurrence.path)
const text = content.toString()
// Find mentions of the character (basic approach)
const mentions = text.split('\n').filter(line =>
line.toLowerCase().includes(characterName.toLowerCase())
)
arc.push({
chapter: occurrence.path,
mentions: mentions.length,
// Analyze emotional tone (simplified)
mood: analyzeMood(mentions),
// Extract key actions
actions: extractActions(mentions, characterName)
})
}
return {
character: character.metadata,
arc: arc,
summary: summarizeArc(arc)
}
}
function analyzeMood(mentions) {
const positiveWords = ['smiled', 'laughed', 'happy', 'joy', 'love', 'success']
const negativeWords = ['cried', 'angry', 'sad', 'fear', 'fail', 'death']
let positive = 0, negative = 0
for (const mention of mentions) {
const lower = mention.toLowerCase()
positive += positiveWords.filter(w => lower.includes(w)).length
negative += negativeWords.filter(w => lower.includes(w)).length
}
if (positive > negative) return 'positive'
if (negative > positive) return 'negative'
return 'neutral'
}
```
### Generate Story Bible
```javascript
/**
* Create a comprehensive reference for your story universe
*/
async function generateStoryBible(vfs, storyPath) {
const characters = await vfs.listEntities({ type: 'character' })
const locations = await vfs.listEntities({ type: 'location' })
const concepts = await vfs.findConcepts({ domain: 'narrative' })
const bible = {
title: 'Story Bible',
generated: new Date(),
characters: {},
locations: {},
plotThreads: {},
timeline: []
}
// Document characters
for (const char of characters) {
const occurrences = await vfs.findEntityOccurrences(char.id)
bible.characters[char.metadata.name] = {
...char.metadata,
appearances: occurrences.map(o => o.path),
relationships: await vfs.getEntityGraph(char.id, { depth: 1 })
}
}
// Document locations
for (const loc of locations) {
bible.locations[loc.metadata.name] = {
...loc.metadata,
scenes: await vfs.findEntityOccurrences(loc.id)
}
}
// Plot threads from concepts
for (const concept of concepts) {
if (concept.type === 'plot') {
bible.plotThreads[concept.name] = {
description: concept.description,
keywords: concept.keywords,
chapters: await vfs.findByConcept(concept.name)
}
}
}
// Generate timeline
const events = await vfs.getTimeline({ limit: 1000 })
bible.timeline = events.map(e => ({
date: e.timestamp,
event: e.description,
chapter: e.path
}))
return bible
}
```
## Game Development Functions
### Validate Game Data
```javascript
/**
* Validate game configuration files for consistency
*/
async function validateGameData(vfs, gamePath) {
const errors = []
const warnings = []
// Load all game data
const gameData = await vfs.exportToJSON(gamePath)
// Check quest references
if (gameData.quests) {
for (const [questName, quest] of Object.entries(gameData.quests)) {
// Check NPC references
if (quest._content?.questGiver) {
const npcPath = `${gamePath}/npcs/${quest._content.questGiver}.json`
const exists = await vfs.exists(npcPath)
if (!exists) {
errors.push(`Quest ${questName} references missing NPC: ${quest._content.questGiver}`)
}
}
// Check item rewards
if (quest._content?.rewards?.items) {
for (const item of quest._content.rewards.items) {
const itemPath = `${gamePath}/items/${item}.json`
const exists = await vfs.exists(itemPath)
if (!exists) {
warnings.push(`Quest ${questName} rewards missing item: ${item}`)
}
}
}
}
}
// Check balance
if (gameData.items) {
const itemPowers = []
for (const [itemName, item] of Object.entries(gameData.items)) {
if (item._content?.stats) {
const totalPower = Object.values(item._content.stats)
.reduce((a, b) => a + b, 0)
itemPowers.push({ name: itemName, power: totalPower })
}
}
// Find outliers
const avgPower = itemPowers.reduce((a, b) => a + b.power, 0) / itemPowers.length
const outliers = itemPowers.filter(i => Math.abs(i.power - avgPower) > avgPower * 2)
for (const outlier of outliers) {
warnings.push(`Item ${outlier.name} may be imbalanced (power: ${outlier.power}, avg: ${avgPower})`)
}
}
return { errors, warnings, valid: errors.length === 0 }
}
```
### Generate Loot Tables
```javascript
/**
* Generate weighted loot tables from item definitions
*/
async function generateLootTables(vfs, itemsPath) {
const items = await vfs.exportToJSON(itemsPath)
const tables = {
common: [],
uncommon: [],
rare: [],
epic: [],
legendary: []
}
for (const [itemName, item] of Object.entries(items)) {
if (item._meta?.type === 'file' && item._content?.rarity) {
const entry = {
name: itemName.replace('.json', ''),
weight: getWeight(item._content.rarity),
data: item._content
}
tables[item._content.rarity.toLowerCase()].push(entry)
}
}
// Normalize weights
for (const table of Object.values(tables)) {
const totalWeight = table.reduce((a, b) => a + b.weight, 0)
for (const entry of table) {
entry.probability = (entry.weight / totalWeight * 100).toFixed(2) + '%'
}
}
return tables
}
function getWeight(rarity) {
const weights = {
common: 100,
uncommon: 50,
rare: 20,
epic: 5,
legendary: 1
}
return weights[rarity.toLowerCase()] || 10
}
```
## Using These Functions
All these functions are templates that you can customize for your specific needs. To use them:
1. Copy the function you need
2. Modify it for your specific requirements
3. Use it with your VFS instance:
```javascript
import { Brainy } from '@soulcraft/brainy'
// Initialize VFS
const brain = new Brainy()
await brain.init()
const vfs = brain.vfs()
await vfs.init()
// Use your custom function
const insights = await getProjectInsights(vfs, '/my-project')
console.log(insights.recommendations)
// Combine multiple functions
const deps = await getDependencyGraph(vfs, '/src')
const cycles = await findCircularDependencies(vfs, '/src')
const untested = await findUntestedCode(vfs, '/src', '/tests')
```
Remember: These are starting points. The power of VFS is that you can combine its primitives to build exactly what you need for your domain!

684
docs/vfs/VFS_EXAMPLES_SCENARIOS.md
View file

@ -1,684 +0,0 @@
# VFS Examples and Scenarios
> **⚠️ DOCUMENTATION IN PROGRESS**: This document is being updated to reflect recent VFS architecture changes. Many examples below reference deprecated Knowledge Layer methods. Updated examples coming soon. For current VFS capabilities, see [VFS_CORE.md](./VFS_CORE.md) and [SEMANTIC_VFS.md](./SEMANTIC_VFS.md).
## Real-World Scenarios
This document demonstrates how VFS enables powerful real-world applications with semantic search, relationships, and AI-powered concept extraction.
### Legend
- ✅ **Real VFS methods** - Fully implemented and working
- 📝 **User functions** - Templates available in [USER_FUNCTIONS.md](./USER_FUNCTIONS.md)
- 🔮 **Future features** - Not yet available (AI augmentations)
**Note:** All ✅ marked methods are production-ready. For 📝 methods, see USER_FUNCTIONS.md for implementation templates.
## Scenario 1: Collaborative Novel Writing
Multiple authors working on a shared universe with recurring characters and locations.
```javascript
import { Brainy } from '@soulcraft/brainy'
async function novelWritingProject() {
const brain = new Brainy({ storage: { type: 'file', path: './novel-data' } })
await brain.init()
const vfs = brain.vfs()
await vfs.init()
// Create project structure ✅
await vfs.mkdir('/novel')
await vfs.mkdir('/novel/chapters')
await vfs.mkdir('/novel/characters')
await vfs.mkdir('/novel/worldbuilding')
// Define main characters as files with rich metadata ✅
await vfs.writeFile('/novel/characters/elena-blackwood.md', `
# Elena Blackwood
A skilled detective with a mysterious past.
## Character Profile
- Age: 32
- Occupation: Private Investigator
- Skills: Deduction, combat, languages
- Personality: Determined, secretive, compassionate
`, {
metadata: {
characterType: 'protagonist',
tags: ['detective', 'mysterious', 'protagonist']
}
})
await vfs.writeFile('/novel/characters/marcus-void.md', `
# Marcus Void
description: 'A wealthy industrialist with dark secrets',
attributes: {
age: 45,
occupation: 'CEO of Void Industries',
traits: ['ruthless', 'charismatic', 'brilliant']
}
})
// Create location entities
const city = await vfs.createEntity({
name: 'Neo Tokyo',
type: 'location',
description: 'A sprawling cyberpunk metropolis',
attributes: {
population: '40 million',
districts: ['Shibuya-5', 'Crypto Quarter', 'Old Town'],
atmosphere: 'neon-lit, rain-soaked, vertical'
}
})
// Link entities ✅
await vfs.linkEntities(protagonist.id, city.id, 'lives_in')
await vfs.linkEntities(protagonist.id, antagonist.id, 'investigates')
// Write chapters with automatic entity tracking
await vfs.writeFile('/novel/chapters/chapter1.md', `
# Chapter 1: Rain in Neo Tokyo
Elena Blackwood stood at the edge of Shibuya-5, watching the rain cascade down
the neon-lit towers. The case file on Marcus Void burned in her pocket.
She had been tracking Void Industries for months, following a trail of
disappeared scientists and mysterious experiments. Tonight, she would finally
infiltrate the Crypto Quarter facility.
`)
// Multiple authors can work simultaneously ✅
vfs.setUser('author-alice')
await vfs.writeFile('/novel/chapters/chapter2.md', `
# Chapter 2: The Void Industries Tower
Marcus Void gazed down at Neo Tokyo from his penthouse office. Somewhere
in those rain-slicked streets, Elena Blackwood was hunting him...
`)
vfs.setUser('author-bob')
await vfs.appendFile('/novel/chapters/chapter2.md', `
He smiled. Let her come. The trap was already set.
`)
// Track character appearances across chapters ✅
const elenaAppearances = await vfs.findEntityOccurrences(protagonist.id)
console.log('Elena appears in:', elenaAppearances.map(f => f.path))
// Find all locations mentioned ✅
const locations = await vfs.listEntities({ type: 'location' })
// Generate character relationship graph ✅
const relationships = await vfs.getEntityGraph(protagonist.id, { depth: 2 })
// Track plot threads using concepts ✅
await vfs.createConcept({
name: 'The Void Conspiracy',
type: 'plot',
domain: 'narrative',
description: 'The main mystery involving disappeared scientists',
keywords: ['scientists', 'experiments', 'void industries', 'conspiracy']
})
// Find all chapters related to the conspiracy ✅
const conspiracyChapters = await vfs.findByConcept('The Void Conspiracy')
// Version control for revisions ✅
const chapterVersions = await vfs.getVersions('/novel/chapters/chapter1.md')
// Collaborative editing history ✅
const history = await vfs.getCollaborationHistory('/novel/chapters/chapter2.md')
console.log('Chapter 2 edited by:', history.map(h => h.user))
// Export for publishing ✅
const manuscript = await vfs.exportToMarkdown('/novel/chapters')
await vfs.close()
await brain.close()
}
```
## Scenario 2: Video Game Development
Game developers creating a complex RPG with quests, items, and NPCs.
```javascript
async function gameDevProject() {
const brain = new Brainy({ storage: { type: 's3', bucket: 'game-assets' } })
await brain.init()
const vfs = brain.vfs()
await vfs.init()
// Game project structure
await vfs.mkdir('/game')
await vfs.mkdir('/game/scripts')
await vfs.mkdir('/game/assets')
await vfs.mkdir('/game/quests')
await vfs.mkdir('/game/npcs')
await vfs.mkdir('/game/items')
// Define game systems as concepts
await vfs.createConcept({
name: 'Combat System',
type: 'system',
domain: 'gameplay',
keywords: ['damage', 'health', 'attacks', 'defense']
})
await vfs.createConcept({
name: 'Quest System',
type: 'system',
domain: 'gameplay',
keywords: ['objectives', 'rewards', 'dialogue', 'progression']
})
// Create NPC entities
const questGiver = await vfs.createEntity({
name: 'Elder Sage',
type: 'npc',
description: 'Wise old man who gives the main quest',
attributes: {
location: 'Village Square',
questsOffered: ['The Ancient Artifact', 'Lost Knowledge'],
dialogue: {
greeting: "Welcome, young adventurer...",
questStart: "I have a task of great importance..."
}
}
})
// Create item entities
const artifact = await vfs.createEntity({
name: 'Crystal of Power',
type: 'item',
description: 'A legendary artifact with immense magical power',
attributes: {
rarity: 'Legendary',
stats: { magic: 100, wisdom: 50 },
questItem: true,
lore: 'Forged in the age of dragons...'
}
})
// Write quest scripts
await vfs.writeFile('/game/quests/main_quest.json', JSON.stringify({
id: 'main_quest_01',
name: 'The Ancient Artifact',
description: 'Retrieve the Crystal of Power from the Dark Tower',
objectives: [
{ id: 'obj_1', description: 'Speak to the Elder Sage' },
{ id: 'obj_2', description: 'Travel to the Dark Tower' },
{ id: 'obj_3', description: 'Defeat the Guardian' },
{ id: 'obj_4', description: 'Retrieve the Crystal of Power' }
],
rewards: {
experience: 5000,
gold: 1000,
items: ['Crystal of Power']
}
}, null, 2))
// Link quest elements
await vfs.linkEntities(questGiver.id, 'main_quest_01', 'gives_quest')
await vfs.linkEntities('main_quest_01', artifact.id, 'rewards_item')
// Combat script with system tracking
await vfs.writeFile('/game/scripts/combat.js', `
class CombatSystem {
calculateDamage(attacker, defender) {
const baseDamage = attacker.stats.attack
const defense = defender.stats.defense
return Math.max(1, baseDamage - defense)
}
executeAttack(attacker, defender) {
const damage = this.calculateDamage(attacker, defender)
defender.health -= damage
return { damage, remaining: defender.health }
}
}
export default CombatSystem
`)
// Asset management ✅
await vfs.writeFile('/game/assets/sprites/elder_sage.png', spriteData)
await vfs.setMetadata('/game/assets/sprites/elder_sage.png', {
dimensions: '64x64',
animations: ['idle', 'talking'],
artist: 'Alice',
license: 'CC-BY-4.0'
})
// Track dependencies ✅
await vfs.addRelationship('/game/quests/main_quest.json', '/game/npcs/elder_sage.json', 'uses')
await vfs.addRelationship('/game/scripts/combat.js', '/game/systems/stats.js', 'imports')
// Find all content related to combat ✅
const combatFiles = await vfs.findByConcept('Combat System')
// Get all NPCs in a specific location ✅
const villageNPCs = await vfs.searchEntities({
type: 'npc',
where: { 'attributes.location': 'Village Square' }
})
// Track game balance changes — read the file as it was a week ago ✅
const lastWeek = await brain.asOf(new Date(Date.now() - 7 * 86_400_000))
const oldBalance = await lastWeek.find({ where: { path: '/game/data/balance.json' }, limit: 1 })
await lastWeek.release()
// Collaborative development tracking ✅
await vfs.addTodo('/game/quests/main_quest.json', {
task: 'Add voice dialogue triggers',
priority: 'medium',
status: 'pending',
assignee: 'audio-team'
})
// Export for build system ✅
const gameData = await vfs.exportToJSON('/game')
await vfs.close()
await brain.close()
}
```
## Scenario 3: Software Development Project
Building a web application with full project management.
```javascript
async function softwareProject() {
const brain = new Brainy({ storage: { type: 'r2', bucket: 'project-files' } })
await brain.init()
const vfs = brain.vfs()
await vfs.init()
// Import existing git repository ✅
await vfs.importFromGit('/local/repos/webapp', '/project')
// Define architectural concepts
await vfs.createConcept({
name: 'Authentication',
type: 'architecture',
domain: 'backend',
keywords: ['jwt', 'login', 'session', 'oauth'],
relatedConcepts: ['Security', 'User Management']
})
await vfs.createConcept({
name: 'State Management',
type: 'architecture',
domain: 'frontend',
keywords: ['redux', 'context', 'store', 'actions']
})
// Write source code
await vfs.writeFile('/project/src/auth/login.ts', `
import { User } from '../models/User'
import { generateJWT } from '../utils/jwt'
export async function login(email: string, password: string): Promise<string> {
const user = await User.findByEmail(email)
if (!user || !user.verifyPassword(password)) {
throw new Error('Invalid credentials')
}
return generateJWT(user.id)
}
`)
// Track imports and dependencies
await vfs.addRelationship('/project/src/auth/login.ts', '/project/src/models/User.ts', 'imports')
await vfs.addRelationship('/project/src/auth/login.ts', '/project/src/utils/jwt.ts', 'imports')
// Add inline TODOs
await vfs.addTodo('/project/src/auth/login.ts', {
task: 'Add rate limiting',
priority: 'high',
status: 'pending',
assignee: 'security-team',
due: '2025-02-01'
})
await vfs.addTodo('/project/src/auth/login.ts', {
task: 'Implement OAuth providers',
priority: 'medium',
status: 'in_progress',
assignee: 'backend-team'
})
// Configuration files
await vfs.writeFile('/project/config/database.json', JSON.stringify({
development: {
host: 'localhost',
port: 5432,
database: 'webapp_dev'
},
production: {
host: '${DB_HOST}',
port: 5432,
database: 'webapp_prod'
}
}, null, 2))
// Set security metadata
await vfs.setMetadata('/project/config/database.json', {
sensitive: true,
environment: 'all',
lastReviewed: '2025-01-15',
reviewer: 'security-team'
})
// Tests with relationship to source
await vfs.writeFile('/project/tests/auth/login.test.ts', `
import { login } from '../../src/auth/login'
describe('Authentication', () => {
test('valid login returns JWT', async () => {
const token = await login('user@example.com', 'password')
expect(token).toBeDefined()
})
})
`)
await vfs.addRelationship('/project/tests/auth/login.test.ts', '/project/src/auth/login.ts', 'tests')
// Documentation
await vfs.writeFile('/project/docs/API.md', `
# API Documentation
## Authentication
### POST /api/login
Authenticates a user and returns a JWT token.
**Request:**
\`\`\`json
{
"email": "user@example.com",
"password": "secret"
}
\`\`\`
`)
// Find all files needing security review ✅
const securityFiles = await vfs.search('authentication password jwt oauth', {
path: '/project/src',
type: 'file'
})
// Get project insights 📝 (see USER_FUNCTIONS.md for getProjectInsights)
const insights = await getProjectInsights(vfs, '/project') // User function
console.log('Most modified files:', insights.hotspots)
console.log('Key concepts:', insights.concepts)
console.log('Team activity:', insights.contributors)
// Find circular dependencies 📝 (see USER_FUNCTIONS.md)
const circularDeps = await findCircularDependencies(vfs, '/project/src') // User function
// Get test coverage relationships 📝 (see USER_FUNCTIONS.md)
const untested = await findUntestedCode(vfs, '/project/src') // User function
// Track technical debt ✅
const todos = await vfs.getAllTodos('/project')
const highPriorityDebt = todos.filter(t => t.priority === 'high' && t.status === 'pending')
// Generate dependency graph 📝 (see USER_FUNCTIONS.md)
const depGraph = await getDependencyGraph(vfs, '/project/src') // User function
// Find similar code (potential refactoring) 📝 (see USER_FUNCTIONS.md)
const similarCode = await findSimilarCode(vfs, '/project/src/auth/login.ts', {
threshold: 0.8,
minLines: 10
}) // User function
// Export for CI/CD ✅ (Knowledge Layer provides wrapper)
await vfs.exportToGit('/project', '/tmp/build-output')
// Collaborative features ✅ / 🔮
vfs.setUser('developer-alice') // ✅ Real method
// const conflicts = await vfs.detectConflicts('/project/src/auth/login.ts') // 🔮 Future feature
await vfs.close()
await brain.close()
}
```
## Scenario 4: Multi-Project Knowledge Base
All projects in one Brainy instance, sharing concepts and cross-referencing.
```javascript
async function unifiedKnowledgeBase() {
const brain = new Brainy({
storage: { type: 'file', path: './knowledge-base' }
})
await brain.init()
const vfs = brain.vfs()
await vfs.init()
// Create separate project spaces
await vfs.mkdir('/novel')
await vfs.mkdir('/game')
await vfs.mkdir('/webapp')
// Shared universe - characters appear in both novel and game
const sharedCharacter = await vfs.createEntity({
name: 'Elena Blackwood',
type: 'character',
description: 'Protagonist appearing in multiple works'
})
// Novel chapter mentioning Elena
await vfs.writeFile('/novel/chapter1.md', `
Elena Blackwood stood at the edge of the digital void...
`)
// Game script featuring Elena as NPC
await vfs.writeFile('/game/npcs/elena.json', JSON.stringify({
name: 'Elena Blackwood',
type: 'ally',
dialogue: ['I\'ve seen this before in my world...']
}))
// Web app has Elena as example user
await vfs.writeFile('/webapp/seeds/users.json', JSON.stringify([{
name: 'Elena Blackwood',
email: 'elena@example.com',
role: 'detective'
}]))
// Cross-project entity tracking ✅
const elenaOccurrences = await vfs.findEntityOccurrences(sharedCharacter.id)
console.log('Elena appears across projects:', elenaOccurrences)
// Shared technical concepts
const authConcept = await vfs.createConcept({
name: 'Authentication',
type: 'technical',
domain: 'software'
})
// Find auth implementations across all projects ✅
const authImplementations = await vfs.findByConcept('Authentication')
// Returns: /webapp/src/auth.js, /game/scripts/player-auth.js, etc.
// Cross-project relationships ✅
await vfs.addRelationship('/novel/chapter1.md', '/game/story/intro.txt', 'inspires')
await vfs.addRelationship('/game/npcs/elena.json', '/novel/characters/elena.md', 'based_on')
// Universal search across all projects ✅
const results = await vfs.search('Elena Blackwood authentication', {
path: '/',
recursive: true
})
// Project statistics 📝 (see USER_FUNCTIONS.md for getProjectStats)
const novelStats = await getProjectStats(vfs, '/novel') // User function
const gameStats = await getProjectStats(vfs, '/game') // User function
const webappStats = await getProjectStats(vfs, '/webapp') // User function
console.log('Total files:', novelStats.fileCount + gameStats.fileCount + webappStats.fileCount)
console.log('Total size:', novelStats.totalSize + gameStats.totalSize + webappStats.totalSize)
// Knowledge graph visualization data 🔮 (future feature)
// const knowledgeGraph = await vfs.getGlobalKnowledgeGraph() // Not yet implemented
// Returns nodes (files, entities, concepts) and edges (relationships)
// Find connections between projects 🔮 (future feature)
// const crossProjectLinks = await vfs.findCrossProjectLinks() // Not yet implemented
// Unified timeline ✅
const timeline = await vfs.getTimeline({
from: '2025-01-01',
to: '2025-12-31'
})
await vfs.close()
await brain.close()
}
```
## Advanced Features
### Semantic Code Analysis 📝
These are user functions - see [USER_FUNCTIONS.md](./USER_FUNCTIONS.md) for implementation templates:
```javascript
// Find security vulnerabilities (user function example)
const vulnerabilities = await findPatterns(vfs, [
'eval(',
'innerHTML =',
'SQL injection',
'hardcoded password'
])
// Find code smells (user function example)
const codeSmells = await analyzeCodeQuality(vfs, '/src', {
checkDuplication: true,
checkComplexity: true,
checkNaming: true
})
```
### AI-Powered Features 🔮
**Note:** These features require AI integration and are not yet available.
```javascript
// Future: Generate documentation
// const docs = await vfs.generateDocumentation('/src/auth/login.ts')
// Future: Suggest refactorings
// const refactorings = await vfs.suggestRefactorings('/src/utils.js')
// Future: Auto-complete code
// const completion = await vfs.completeCode('/src/api.ts', { line: 42, column: 10 })
```
### Migration and Backup 🔮
**Note:** These features are planned but not yet implemented.
```javascript
// Future: Backup with full history
// await vfs.createBackup('/backup/2025-01-20.brainy')
// Future: Migrate between storage backends
// const migration = await vfs.migrate({
// from: { type: 'file', path: './old-data' },
// to: { type: 's3', bucket: 'new-bucket' }
// })
// Future: Incremental sync
// await vfs.sync('/local/path', '/vfs/path', {
// bidirectional: true,
// conflictStrategy: 'newest'
// })
```
### Performance at Scale
```javascript
// Handle millions of files ✅
for (let i = 0; i < 1000000; i++) {
await vfs.writeFile(`/data/file${i}.txt`, `Content ${i}`)
// Uses chunking, compression, and efficient indexing
}
// Fast parallel operations ✅
await Promise.all([
vfs.writeFile('/file1.txt', 'data1'),
vfs.writeFile('/file2.txt', 'data2'),
vfs.writeFile('/file3.txt', 'data3')
])
// Bulk write operations ✅
const files = [
{ path: '/data/file1.txt', content: 'Content 1' },
{ path: '/data/file2.txt', content: 'Content 2' },
// ... more files
]
await vfs.bulkWrite(files)
// Bulk imports 🔮 (future feature)
// await vfs.bulkImport('/massive/dataset', {
// parallel: 10,
// batchSize: 1000,
// progress: (count, total) => console.log(`${count}/${total}`)
// })
```
## Implementation Status
### ✅ Fully Implemented Features
All methods marked with ✅ are production-ready:
1. **Core VFS Operations**: mkdir, writeFile, readFile, appendFile, stat, readdir, etc.
2. **Entity System**: createEntity, linkEntities, findEntityOccurrences, updateEntity, getEntityGraph
3. **Concept System**: createConcept, findByConcept
4. **Knowledge Layer**: Event recording, semantic versioning, collaboration tracking
5. **Search**: Triple Intelligence (vector + field + graph)
6. **Git Integration**: importFromGit, exportToGit
7. **Export Formats**: exportToMarkdown, exportToJSON
8. **Bulk Operations**: bulkWrite for efficient batch processing
9. **Project Management**: todos, metadata, relationships
### 📝 User Functions
Methods marked with 📝 are domain-specific functions that you can implement using VFS primitives. See [USER_FUNCTIONS.md](./USER_FUNCTIONS.md) for ready-to-use templates:
- **Code Analysis**: getDependencyGraph, findCircularDependencies, findUntestedCode, findSimilarCode
- **Creative Writing**: trackCharacterArc, generateStoryBible
- **Game Development**: validateGameData, generateLootTables
- **Project Management**: getProjectInsights, generateSprintReport
- **Export Formats**: exportToEpub, exportToStaticSite
### 🔮 Future Features
Methods marked with 🔮 require AI integration or are planned for future releases:
- **AI-Powered**: generateDocumentation, suggestRefactorings, completeCode
- **Advanced Analysis**: detectConflicts, getGlobalKnowledgeGraph, findCrossProjectLinks
- **Migration Tools**: createBackup, migrate, sync, bulkImport
## Real Implementation Guarantees
- **No Mocks**: Every ✅ method is fully functional
- **Real Storage**: Uses Brainy entities with embeddings
- **Real Search**: Triple Intelligence combining vectors, fields, and graphs
- **Real Relationships**: Graph-based connections via brain.relate()
- **Production Ready**: Complete error handling, async/await, resource cleanup
The VFS + Knowledge Layer combination provides a solid foundation for intelligent applications. Use the ✅ methods directly, implement 📝 functions as needed for your domain, and stay tuned for 🔮 features.

4
docs/vfs/VFS_GRAPH_TYPES.md
View file

@ -118,13 +118,13 @@ const results = await brain.search('machine learning', {
### 2. Graph Traversal
```javascript
// Find all entities contained in a directory
const contained = await brain.getRelations({
const contained = await brain.related({
from: directoryId,
type: VerbType.Contains
})
// Find what contains a file (parent directories)
const parents = await brain.getRelations({
const parents = await brain.related({
to: fileId,
type: VerbType.Contains
})

25
models-cache/Xenova/all-MiniLM-L6-v2/config.json
View file

@ -1,25 +0,0 @@
{
"_name_or_path": "sentence-transformers/all-MiniLM-L6-v2",
"architectures": [
"BertModel"
],
"attention_probs_dropout_prob": 0.1,
"classifier_dropout": null,
"gradient_checkpointing": false,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 384,
"initializer_range": 0.02,
"intermediate_size": 1536,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 12,
"num_hidden_layers": 6,
"pad_token_id": 0,
"position_embedding_type": "absolute",
"transformers_version": "4.29.2",
"type_vocab_size": 2,
"use_cache": true,
"vocab_size": 30522
}

BIN
models-cache/Xenova/all-MiniLM-L6-v2/onnx/model.onnx
View file

Binary file not shown.

30686
models-cache/Xenova/all-MiniLM-L6-v2/tokenizer.json
View file

File diff suppressed because it is too large Load diff

15
models-cache/Xenova/all-MiniLM-L6-v2/tokenizer_config.json
View file

@ -1,15 +0,0 @@
{
"clean_up_tokenization_spaces": true,
"cls_token": "[CLS]",
"do_basic_tokenize": true,
"do_lower_case": true,
"mask_token": "[MASK]",
"model_max_length": 512,
"never_split": null,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}

115
package-lock.json generated
View file

@ -10,24 +10,20 @@
"license": "MIT",
"dependencies": {
"@msgpack/msgpack": "^3.1.2",
"@types/js-yaml": "^4.0.9",
"boxen": "^8.0.1",
"chalk": "^5.3.0",
"chardet": "^2.0.0",
"cli-table3": "^0.6.5",
"commander": "^11.1.0",
"csv-parse": "^6.1.0",
"exifr": "^7.1.3",
"inquirer": "^12.9.3",
"js-yaml": "^4.1.0",
"mammoth": "^1.11.0",
"mime": "^4.1.0",
"ora": "^8.2.0",
"pdfjs-dist": "^4.0.379",
"probe-image-size": "^7.2.3",
"prompts": "^2.4.2",
"roaring-wasm": "^1.1.0",
"uuid": "^9.0.1",
"ws": "^8.18.3",
"xlsx": "^0.18.5"
},
@ -40,9 +36,9 @@
"@rollup/plugin-replace": "^6.0.2",
"@rollup/plugin-terser": "^0.4.4",
"@testcontainers/redis": "^11.5.1",
"@types/js-yaml": "^4.0.9",
"@types/mime": "^3.0.4",
"@types/node": "^20.11.30",
"@types/probe-image-size": "^7.2.5",
"@types/uuid": "^10.0.0",
"@types/ws": "^8.18.1",
"@typescript-eslint/eslint-plugin": "^8.0.0",
@ -54,6 +50,7 @@
"testcontainers": "^11.5.1",
"tsx": "^4.19.2",
"typescript": "^5.4.5",
"uuid": "^9.0.1",
"vitest": "^3.2.4"
},
"engines": {
@ -2496,6 +2493,7 @@
"version": "4.0.9",
"resolved": "https://registry.npmjs.org/@types/js-yaml/-/js-yaml-4.0.9.tgz",
"integrity": "sha512-k4MGaQl5TGo/iipqb2UDG2UwjXziSWkh0uysQelTlJpX1qGlpUZYm8PnO4DxG1qBomtJUdYJ6qR6xdIah10JLg==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/json-schema": {
@ -2520,16 +2518,6 @@
"dev": true,
"license": "MIT"
},
"node_modules/@types/needle": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/@types/needle/-/needle-3.3.0.tgz",
"integrity": "sha512-UFIuc1gdyzAqeVUYpSL+cliw2MmU/ZUhVZKE7Zo4wPbgc8hbljeKSnn6ls6iG8r5jpegPXLUIhJ+Wb2kLVs8cg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/node": "*"
}
},
"node_modules/@types/node": {
"version": "20.19.27",
"resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.27.tgz",
@ -2554,17 +2542,6 @@
"dev": true,
"license": "MIT"
},
"node_modules/@types/probe-image-size": {
"version": "7.2.5",
"resolved": "https://registry.npmjs.org/@types/probe-image-size/-/probe-image-size-7.2.5.tgz",
"integrity": "sha512-9Bg6d/GNnjmhMMxadDstwrSlquuuLf0jQuPszbU6n3QUfybif3V/ryD3J2i9iaiC5JB/FU/8E41n88SM/UB+Tg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/needle": "*",
"@types/node": "*"
}
},
"node_modules/@types/raf": {
"version": "3.4.3",
"resolved": "https://registry.npmjs.org/@types/raf/-/raf-3.4.3.tgz",
@ -5403,12 +5380,6 @@
"bare-events": "^2.7.0"
}
},
"node_modules/exifr": {
"version": "7.1.3",
"resolved": "https://registry.npmjs.org/exifr/-/exifr-7.1.3.tgz",
"integrity": "sha512-g/aje2noHivrRSLbAUtBPWFbxKdKhgj/xr1vATDdUXPOFYJlQ62Ft0oy+72V6XLIpDJfHs6gXLbBLAolqOXYRw==",
"license": "MIT"
},
"node_modules/expect-type": {
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.3.0.tgz",
@ -6874,7 +6845,9 @@
"version": "4.6.2",
"resolved": "https://registry.npmjs.org/lodash.merge/-/lodash.merge-4.6.2.tgz",
"integrity": "sha512-0KpjqXRVvrYyCsX1swR/XTK0va6VQkQM6MNo7PqW77ByjAhoARA8EfrP1N4+KlKj8YS0ZUCtRT/YUuhyYDujIQ==",
"license": "MIT"
"dev": true,
"license": "MIT",
"peer": true
},
"node_modules/log-symbols": {
"version": "6.0.0",
@ -7447,6 +7420,7 @@
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
"dev": true,
"license": "MIT"
},
"node_modules/mute-stream": {
@ -7492,44 +7466,6 @@
"dev": true,
"license": "MIT"
},
"node_modules/needle": {
"version": "2.9.1",
"resolved": "https://registry.npmjs.org/needle/-/needle-2.9.1.tgz",
"integrity": "sha512-6R9fqJ5Zcmf+uYaFgdIHmLwNldn5HbK8L5ybn7Uz+ylX/rnOsSp1AHcvQSrCaFN+qNM1wpymHqD7mVasEOlHGQ==",
"license": "MIT",
"dependencies": {
"debug": "^3.2.6",
"iconv-lite": "^0.4.4",
"sax": "^1.2.4"
},
"bin": {
"needle": "bin/needle"
},
"engines": {
"node": ">= 4.4.x"
}
},
"node_modules/needle/node_modules/debug": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/debug/-/debug-3.2.7.tgz",
"integrity": "sha512-CFjzYYAi4ThfiQvizrFQevTTXHtnCqWfe7x1AhgEscTz6ZbLbfoLRLPugTQyBth6f8ZERVUSyWHFD/7Wu4t1XQ==",
"license": "MIT",
"dependencies": {
"ms": "^2.1.1"
}
},
"node_modules/needle/node_modules/iconv-lite": {
"version": "0.4.24",
"resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
"integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
"license": "MIT",
"dependencies": {
"safer-buffer": ">= 2.1.2 < 3"
},
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/neo-async": {
"version": "2.6.2",
"resolved": "https://registry.npmjs.org/neo-async/-/neo-async-2.6.2.tgz",
@ -7920,17 +7856,6 @@
"node": ">= 0.8.0"
}
},
"node_modules/probe-image-size": {
"version": "7.2.3",
"resolved": "https://registry.npmjs.org/probe-image-size/-/probe-image-size-7.2.3.tgz",
"integrity": "sha512-HubhG4Rb2UH8YtV4ba0Vp5bQ7L78RTONYu/ujmCu5nBI8wGv24s4E9xSKBi0N1MowRpxk76pFCpJtW0KPzOK0w==",
"license": "MIT",
"dependencies": {
"lodash.merge": "^4.6.2",
"needle": "^2.5.2",
"stream-parser": "~0.3.1"
}
},
"node_modules/process": {
"version": "0.11.10",
"resolved": "https://registry.npmjs.org/process/-/process-0.11.10.tgz",
@ -8504,6 +8429,7 @@
"version": "1.4.3",
"resolved": "https://registry.npmjs.org/sax/-/sax-1.4.3.tgz",
"integrity": "sha512-yqYn1JhPczigF94DMS+shiDMjDowYO6y9+wB/4WgO0Y19jWYk0lQ4tuG5KI7kj4FTp1wxPj5IFfcrz/s1c3jjQ==",
"dev": true,
"license": "BlueOak-1.0.0"
},
"node_modules/semver": {
@ -8934,30 +8860,6 @@
"stream-chain": "^2.2.5"
}
},
"node_modules/stream-parser": {
"version": "0.3.1",
"resolved": "https://registry.npmjs.org/stream-parser/-/stream-parser-0.3.1.tgz",
"integrity": "sha512-bJ/HgKq41nlKvlhccD5kaCr/P+Hu0wPNKPJOH7en+YrJu/9EgqUF+88w5Jb6KNcjOFMhfX4B2asfeAtIGuHObQ==",
"license": "MIT",
"dependencies": {
"debug": "2"
}
},
"node_modules/stream-parser/node_modules/debug": {
"version": "2.6.9",
"resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
"integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
"license": "MIT",
"dependencies": {
"ms": "2.0.0"
}
},
"node_modules/stream-parser/node_modules/ms": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
"license": "MIT"
},
"node_modules/streamx": {
"version": "2.23.0",
"resolved": "https://registry.npmjs.org/streamx/-/streamx-2.23.0.tgz",
@ -9584,6 +9486,7 @@
"version": "9.0.1",
"resolved": "https://registry.npmjs.org/uuid/-/uuid-9.0.1.tgz",
"integrity": "sha512-b+1eJOlsR9K8HJpow9Ok3fiWOWSIcIzXodvv0rQjVoOVNpWMpxf1wZNpt4y9h10odCNrqnYp1OBzRktckBe3sA==",
"dev": true,
"funding": [
"https://github.com/sponsors/broofa",
"https://github.com/sponsors/ctavan"

9
package.json
View file

@ -98,8 +98,6 @@
"lint:fix": "eslint --ext .ts,.js src/ --fix",
"format": "prettier --write \"src/**/*.{ts,js}\"",
"format:check": "prettier --check \"src/**/*.{ts,js}\"",
"migrate:logger": "tsx scripts/migrate-to-structured-logger.ts",
"migrate:logger:dry": "tsx scripts/migrate-to-structured-logger.ts --dry-run",
"release": "./scripts/release.sh patch",
"release:patch": "./scripts/release.sh patch",
"release:minor": "./scripts/release.sh minor",
@ -160,9 +158,9 @@
"@rollup/plugin-replace": "^6.0.2",
"@rollup/plugin-terser": "^0.4.4",
"@testcontainers/redis": "^11.5.1",
"@types/js-yaml": "^4.0.9",
"@types/mime": "^3.0.4",
"@types/node": "^20.11.30",
"@types/probe-image-size": "^7.2.5",
"@types/uuid": "^10.0.0",
"@types/ws": "^8.18.1",
"@typescript-eslint/eslint-plugin": "^8.0.0",
@ -174,28 +172,25 @@
"testcontainers": "^11.5.1",
"tsx": "^4.19.2",
"typescript": "^5.4.5",
"uuid": "^9.0.1",
"vitest": "^3.2.4"
},
"dependencies": {
"@msgpack/msgpack": "^3.1.2",
"@types/js-yaml": "^4.0.9",
"boxen": "^8.0.1",
"chalk": "^5.3.0",
"chardet": "^2.0.0",
"cli-table3": "^0.6.5",
"commander": "^11.1.0",
"csv-parse": "^6.1.0",
"exifr": "^7.1.3",
"inquirer": "^12.9.3",
"js-yaml": "^4.1.0",
"mammoth": "^1.11.0",
"mime": "^4.1.0",
"ora": "^8.2.0",
"pdfjs-dist": "^4.0.379",
"probe-image-size": "^7.2.3",
"prompts": "^2.4.2",
"roaring-wasm": "^1.1.0",
"uuid": "^9.0.1",
"ws": "^8.18.3",
"xlsx": "^0.18.5"
},

60
scripts/create-models-release.sh
View file

@ -1,60 +0,0 @@
#!/bin/bash
# Create GitHub release with unarchived model files for direct serving
# This allows transformers.js to download individual files directly
set -e
echo "🚀 Creating GitHub release with unarchived model files..."
# Create temporary directory
TEMP_DIR=$(mktemp -d)
cd "$TEMP_DIR"
echo "📥 Downloading existing model archive..."
curl -sL https://github.com/soulcraftlabs/brainy/releases/download/models-v1/all-MiniLM-L6-v2.tar.gz -o models.tar.gz
echo "📦 Extracting models..."
tar -xzf models.tar.gz
echo "🔧 Preparing individual model files for release..."
# Create properly structured directory
mkdir -p release-files/Xenova/all-MiniLM-L6-v2/onnx
# Copy files with correct naming for direct serving
cp all-MiniLM-L6-v2/config.json release-files/Xenova/all-MiniLM-L6-v2/
cp all-MiniLM-L6-v2/tokenizer.json release-files/Xenova/all-MiniLM-L6-v2/
cp all-MiniLM-L6-v2/tokenizer_config.json release-files/Xenova/all-MiniLM-L6-v2/
cp all-MiniLM-L6-v2/onnx/model.onnx release-files/Xenova/all-MiniLM-L6-v2/onnx/
# Also keep the tar.gz for backward compatibility
cp models.tar.gz release-files/
echo "📋 Files to upload:"
find release-files -type f -exec ls -lh {} \;
echo ""
echo "✅ Files ready for release!"
echo ""
echo "To create the release with individual files:"
echo ""
echo "1. Go to: https://github.com/soulcraftlabs/brainy/releases/new"
echo "2. Tag: models-v2"
echo "3. Title: Model Files v2 - Unarchived"
echo "4. Upload these files from $TEMP_DIR/release-files/:"
echo " - Xenova/all-MiniLM-L6-v2/config.json"
echo " - Xenova/all-MiniLM-L6-v2/tokenizer.json"
echo " - Xenova/all-MiniLM-L6-v2/tokenizer_config.json"
echo " - Xenova/all-MiniLM-L6-v2/onnx/model.onnx"
echo " - all-MiniLM-L6-v2.tar.gz (for compatibility)"
echo ""
echo "Or use GitHub CLI:"
echo " cd $TEMP_DIR/release-files"
echo " gh release create models-v2 --repo soulcraftlabs/brainy \\"
echo " --title 'Model Files v2 - Direct Serving' \\"
echo " --notes 'Unarchived model files for direct HTTP serving' \\"
echo " Xenova/all-MiniLM-L6-v2/config.json \\"
echo " Xenova/all-MiniLM-L6-v2/tokenizer.json \\"
echo " Xenova/all-MiniLM-L6-v2/tokenizer_config.json \\"
echo " Xenova/all-MiniLM-L6-v2/onnx/model.onnx \\"
echo " models.tar.gz"

194
scripts/diagnose-workshop-issue.ts
View file

@ -1,194 +0,0 @@
/**
* Diagnostic script for Workshop type filtering issue
*
* This script mimics Workshop's exact import pattern to see what's happening
*/
import { Brainy, NounType } from '../src/index.js'
import * as fs from 'fs'
import * as path from 'path'
async function diagnose() {
console.log('\n🔬 Workshop Type Filtering Diagnostic\n')
console.log('='.repeat(70))
// Use temporary directory
const testDir = './test-workshop-data'
if (fs.existsSync(testDir)) {
fs.rmSync(testDir, { recursive: true })
}
const brain = new Brainy({
storage: {
type: 'filesystem',
path: testDir
}
})
await brain.init()
console.log('\n1️⃣ Testing WITHOUT VFS (control group)...\n')
// Add entities without VFS
console.log('Adding 3 person entities directly...')
await brain.add({ data: 'Person 1', type: NounType.Person, metadata: { name: 'Person 1' } })
await brain.add({ data: 'Person 2', type: NounType.Person, metadata: { name: 'Person 2' } })
await brain.add({ data: 'Person 3', type: NounType.Person, metadata: { name: 'Person 3' } })
console.log('Adding 2 location entities directly...')
await brain.add({ data: 'Location 1', type: NounType.Location, metadata: { name: 'Location 1' } })
await brain.add({ data: 'Location 2', type: NounType.Location, metadata: { name: 'Location 2' } })
console.log('\n📊 Testing type filtering on direct entities...\n')
const allDirect = await brain.find({ limit: 100 })
console.log(` Total entities: ${allDirect.length}`)
const peopleDirect = await brain.find({ type: NounType.Person, limit: 100 })
console.log(` Person filter: ${peopleDirect.length} (expected: 3)`)
const locationsDirect = await brain.find({ type: NounType.Location, limit: 100 })
console.log(` Location filter: ${locationsDirect.length} (expected: 2)`)
if (peopleDirect.length === 3 && locationsDirect.length === 2) {
console.log('\n ✅ Type filtering works on direct entities!')
} else {
console.log('\n ❌ Type filtering BROKEN on direct entities!')
return
}
console.log('\n' + '='.repeat(70))
console.log('\n2️⃣ Testing WITH VFS (Workshop pattern)...\n')
// Simulate Workshop's pattern: create VFS file entities
console.log('Creating VFS file wrappers (like import does)...')
const vfs = brain.vfs()
await vfs.init()
// Create VFS directory
await vfs.mkdir('/imports/test', { recursive: true })
// Create VFS files with embedded entity data (mimics import)
console.log('Creating VFS file for Person entity...')
const personEntityData = {
id: 'ent_person_test',
name: 'John Smith',
type: 'person',
metadata: {
source: 'excel',
originalData: {
Name: 'John Smith',
_sheet: 'Characters'
}
}
}
await vfs.writeFile(
'/imports/test/john_smith.json',
Buffer.from(JSON.stringify(personEntityData, null, 2))
)
console.log('Creating VFS file for Location entity...')
const locationEntityData = {
id: 'ent_location_test',
name: 'New York',
type: 'location',
metadata: {
source: 'excel',
originalData: {
Name: 'New York',
_sheet: 'Places'
}
}
}
await vfs.writeFile(
'/imports/test/new_york.json',
Buffer.from(JSON.stringify(locationEntityData, null, 2))
)
console.log('\n📊 Checking what entities exist now...\n')
const allWithVfs = await brain.find({ limit: 100 })
console.log(` Total entities: ${allWithVfs.length}`)
// Analyze entity types
const typeCounts: Record<string, number> = {}
for (const result of allWithVfs) {
const type = result.type || 'unknown'
typeCounts[type] = (typeCounts[type] || 0) + 1
}
console.log('\n Entity types breakdown:')
for (const [type, count] of Object.entries(typeCounts)) {
console.log(` - ${type}: ${count}`)
}
console.log('\n📊 Testing type filtering with VFS entities...\n')
const peopleWithVfs = await brain.find({ type: NounType.Person, limit: 100 })
console.log(` Person filter: ${peopleWithVfs.length} (expected: 3 direct + maybe VFS?)`)
const locationsWithVfs = await brain.find({ type: NounType.Location, limit: 100 })
console.log(` Location filter: ${locationsWithVfs.length} (expected: 2 direct + maybe VFS?)`)
const documents = await brain.find({ type: NounType.Document, limit: 100 })
console.log(` Document filter: ${documents.length} (VFS wrappers?)`)
console.log('\n' + '='.repeat(70))
console.log('\n3️⃣ Analyzing VFS wrapper structure...\n')
// Get a VFS wrapper entity
const vfsWrapper = allWithVfs.find(e => e.metadata?.vfsType === 'file')
if (vfsWrapper) {
console.log('Found VFS wrapper entity:')
console.log(` - ID: ${vfsWrapper.id}`)
console.log(` - Type: ${vfsWrapper.type}`)
console.log(` - VFS Type: ${vfsWrapper.metadata?.vfsType}`)
console.log(` - Has rawData: ${!!vfsWrapper.metadata?.rawData}`)
console.log(` - Path: ${vfsWrapper.metadata?.path}`)
if (vfsWrapper.metadata?.rawData) {
console.log('\n Decoding rawData...')
try {
const decoded = Buffer.from(vfsWrapper.metadata.rawData, 'base64').toString()
const entity = JSON.parse(decoded)
console.log(` - Embedded entity name: ${entity.name}`)
console.log(` - Embedded entity type: ${entity.type}`)
console.log(` - Wrapper type: ${vfsWrapper.type}`)
console.log('\n 🔍 KEY INSIGHT:')
console.log(` Wrapper has type="${vfsWrapper.type}"`)
console.log(` But embedded entity has type="${entity.type}"`)
console.log(' When you filter by person, you get the WRAPPER type, not embedded type!')
} catch (err) {
console.log(' ❌ Failed to decode rawData')
}
}
} else {
console.log('No VFS wrapper entities found')
}
console.log('\n' + '='.repeat(70))
console.log('\n4️⃣ DIAGNOSIS\n')
if (documents.length > 0 && peopleWithVfs.length === 3) {
console.log('❌ FOUND THE BUG!')
console.log('')
console.log('VFS creates document wrappers with type="document".')
console.log('The actual entity data is stored as base64 in metadata.rawData.')
console.log('When you filter by type="person", you\'re filtering the WRAPPER type.')
console.log('Since wrappers are type="document", you get 0 results.')
console.log('')
console.log('This is a DESIGN ISSUE in how VFS import works!')
} else if (peopleWithVfs.length > 3) {
console.log('✅ Type filtering works correctly!')
console.log('VFS entities are being created with proper types.')
} else {
console.log('🤔 Unclear - need more investigation')
}
console.log('\n' + '='.repeat(70) + '\n')
// Cleanup
fs.rmSync(testDir, { recursive: true })
}
diagnose().catch(console.error)

323
scripts/migrate-to-structured-logger.ts
View file

@ -1,323 +0,0 @@
#!/usr/bin/env node
/**
* Migration script to replace console.log statements with structured logger
* Usage: npm run migrate:logger [--dry-run] [--file=path/to/file.ts]
*/
import * as fs from 'fs/promises'
import * as path from 'path'
import { glob } from 'glob'
import { fileURLToPath } from 'url'
interface MigrationOptions {
dryRun: boolean
targetFile?: string
verbose: boolean
}
interface MigrationResult {
file: string
changes: number
errors: string[]
}
class LoggerMigrator {
private results: MigrationResult[] = []
constructor(private options: MigrationOptions) {}
async migrate(): Promise<void> {
console.log('🔄 Starting logger migration...')
const files = await this.getFilesToMigrate()
console.log(`Found ${files.length} TypeScript files to process`)
for (const file of files) {
await this.migrateFile(file)
}
this.printSummary()
}
private async getFilesToMigrate(): Promise<string[]> {
if (this.options.targetFile) {
return [this.options.targetFile]
}
// Get all TypeScript files excluding node_modules, dist, and test files
const pattern = 'src/**/*.ts'
const ignore = [
'**/node_modules/**',
'**/dist/**',
'**/*.test.ts',
'**/*.spec.ts',
'**/logger.ts',
'**/structuredLogger.ts'
]
return glob(pattern, { ignore })
}
private async migrateFile(filePath: string): Promise<void> {
const result: MigrationResult = {
file: filePath,
changes: 0,
errors: []
}
try {
const content = await fs.readFile(filePath, 'utf-8')
const moduleName = this.extractModuleName(filePath)
let modified = content
let changesMade = false
// Check if file already imports logger
const hasLoggerImport = /import.*(?:createModuleLogger|structuredLogger).*from/.test(content)
const hasConsoleUsage = /console\.(log|warn|error|info|debug)/.test(content)
if (!hasConsoleUsage) {
if (this.options.verbose) {
console.log(` ⏭️ ${filePath} - No console statements found`)
}
return
}
// Add import if needed
if (!hasLoggerImport && hasConsoleUsage) {
modified = this.addLoggerImport(modified)
changesMade = true
}
// Replace console statements
const patterns = [
{
// console.log with string literal
pattern: /console\.log\s*\(\s*(['"`])([^'"`]*)\1\s*(?:,\s*(.+?))?\s*\)/g,
replacement: (match: string, quote: string, message: string, args?: string) => {
result.changes++
return args
? `logger.info('${message}', ${args})`
: `logger.info('${message}')`
}
},
{
// console.error with string literal
pattern: /console\.error\s*\(\s*(['"`])([^'"`]*)\1\s*(?:,\s*(.+?))?\s*\)/g,
replacement: (match: string, quote: string, message: string, args?: string) => {
result.changes++
return args
? `logger.error('${message}', ${args})`
: `logger.error('${message}')`
}
},
{
// console.warn with string literal
pattern: /console\.warn\s*\(\s*(['"`])([^'"`]*)\1\s*(?:,\s*(.+?))?\s*\)/g,
replacement: (match: string, quote: string, message: string, args?: string) => {
result.changes++
return args
? `logger.warn('${message}', ${args})`
: `logger.warn('${message}')`
}
},
{
// console.info with string literal
pattern: /console\.info\s*\(\s*(['"`])([^'"`]*)\1\s*(?:,\s*(.+?))?\s*\)/g,
replacement: (match: string, quote: string, message: string, args?: string) => {
result.changes++
return args
? `logger.info('${message}', ${args})`
: `logger.info('${message}')`
}
},
{
// console.debug with string literal
pattern: /console\.debug\s*\(\s*(['"`])([^'"`]*)\1\s*(?:,\s*(.+?))?\s*\)/g,
replacement: (match: string, quote: string, message: string, args?: string) => {
result.changes++
return args
? `logger.debug('${message}', ${args})`
: `logger.debug('${message}')`
}
}
]
// Apply replacements
for (const { pattern, replacement } of patterns) {
const before = modified
modified = modified.replace(pattern, replacement as any)
if (before !== modified) {
changesMade = true
}
}
// Handle complex console statements that need manual review
const complexPatterns = [
/console\.(log|warn|error|info|debug)\s*\([^'"`]/g
]
for (const pattern of complexPatterns) {
const matches = modified.match(pattern)
if (matches) {
for (const match of matches) {
result.errors.push(`Complex console statement needs manual review: ${match.substring(0, 50)}...`)
// Add a TODO comment for manual review
modified = modified.replace(
match,
`// TODO: Migrate to structured logger\n ${match}`
)
}
}
}
// Add logger declaration after imports
if (changesMade && !hasLoggerImport) {
const importEndMatch = modified.match(/^((?:import.*\n)+)/m)
if (importEndMatch) {
const afterImports = importEndMatch.index! + importEndMatch[0].length
modified =
modified.slice(0, afterImports) +
`\nconst logger = createModuleLogger('${moduleName}')\n` +
modified.slice(afterImports)
}
}
// Write changes
if (changesMade && !this.options.dryRun) {
await fs.writeFile(filePath, modified, 'utf-8')
console.log(` ✅ ${filePath} - ${result.changes} changes`)
} else if (changesMade) {
console.log(` 🔍 ${filePath} - ${result.changes} changes (dry run)`)
}
this.results.push(result)
} catch (error) {
result.errors.push(`Failed to process file: ${error}`)
this.results.push(result)
}
}
private addLoggerImport(content: string): string {
// Find the last import statement
const importMatches = [...content.matchAll(/^import.*$/gm)]
if (importMatches.length > 0) {
const lastImport = importMatches[importMatches.length - 1]
const insertPos = lastImport.index! + lastImport[0].length
const relativeImportPath = this.getRelativeImportPath()
const importStatement = `\nimport { createModuleLogger } from '${relativeImportPath}'`
return content.slice(0, insertPos) + importStatement + content.slice(insertPos)
}
// No imports found, add at the beginning
const relativeImportPath = this.getRelativeImportPath()
return `import { createModuleLogger } from '${relativeImportPath}'\n\n${content}`
}
private getRelativeImportPath(): string {
// This will be calculated based on the file being processed
// For now, return a placeholder
return '../utils/structuredLogger.js'
}
private extractModuleName(filePath: string): string {
// Extract module name from file path
const relativePath = path.relative('src', filePath)
const moduleName = relativePath
.replace(/\.ts$/, '')
.replace(/\//g, ':')
.replace(/\\+/g, ':')
return moduleName
}
private printSummary(): void {
console.log('\n📊 Migration Summary:')
console.log('=' .repeat(50))
let totalChanges = 0
let totalErrors = 0
let filesWithChanges = 0
let filesWithErrors = 0
for (const result of this.results) {
if (result.changes > 0) {
filesWithChanges++
totalChanges += result.changes
}
if (result.errors.length > 0) {
filesWithErrors++
totalErrors += result.errors.length
console.log(`\n⚠️ ${result.file}:`)
for (const error of result.errors) {
console.log(` - ${error}`)
}
}
}
console.log('\n📈 Statistics:')
console.log(` Files processed: ${this.results.length}`)
console.log(` Files modified: ${filesWithChanges}`)
console.log(` Total changes: ${totalChanges}`)
console.log(` Files with errors: ${filesWithErrors}`)
console.log(` Total errors: ${totalErrors}`)
if (this.options.dryRun) {
console.log('\n⚠️ This was a dry run. No files were modified.')
console.log('Run without --dry-run to apply changes.')
}
if (totalErrors > 0) {
console.log('\n⚠️ Some console statements need manual review.')
console.log('Search for "TODO: Migrate to structured logger" in the code.')
}
}
}
// Parse command line arguments
function parseArgs(): MigrationOptions {
const args = process.argv.slice(2)
const options: MigrationOptions = {
dryRun: false,
verbose: false
}
for (const arg of args) {
if (arg === '--dry-run') {
options.dryRun = true
} else if (arg === '--verbose' || arg === '-v') {
options.verbose = true
} else if (arg.startsWith('--file=')) {
options.targetFile = arg.split('=')[1]
}
}
return options
}
// Main execution
async function main() {
const options = parseArgs()
const migrator = new LoggerMigrator(options)
try {
await migrator.migrate()
} catch (error) {
console.error('Migration failed:', error)
process.exit(1)
}
}
// Run if executed directly
if (import.meta.url === `file://${process.argv[1]}`) {
main().catch(console.error)
}
export { LoggerMigrator, MigrationOptions }

269
scripts/migrate-vfs-storage.ts
View file

@ -1,269 +0,0 @@
/**
* VFS Storage Migration Script: v5.1.x → v5.2.0
*
* Converts VFS files from 3-tier storage (inline/reference/chunked)
* to unified BlobStorage (content-addressable with deduplication)
*
* Usage:
* import { migrateVFSStorage } from './scripts/migrate-vfs-storage'
* await migrateVFSStorage(brain)
*/
import { Brainy } from '../src/brainy.js'
import { VFSMetadata } from '../src/vfs/types.js'
export interface MigrationStats {
filesProcessed: number
inlineMigrated: number
referenceMigrated: number
chunkedMigrated: number
alreadyMigrated: number
errors: number
deduplicated: number
totalSizeBefore: number
totalSizeAfter: number
durationMs: number
}
export async function migrateVFSStorage(
brain: Brainy,
options: {
dryRun?: boolean
verbose?: boolean
batchSize?: number
} = {}
): Promise<MigrationStats> {
const { dryRun = false, verbose = false, batchSize = 100 } = options
const stats: MigrationStats = {
filesProcessed: 0,
inlineMigrated: 0,
referenceMigrated: 0,
chunkedMigrated: 0,
alreadyMigrated: 0,
errors: 0,
deduplicated: 0,
totalSizeBefore: 0,
totalSizeAfter: 0,
durationMs: 0
}
const startTime = Date.now()
if (verbose) {
console.log('🔄 Starting VFS storage migration (v5.1.x → v5.2.0)')
if (dryRun) console.log(' DRY RUN: No changes will be made')
}
try {
// Find all VFS file entities
const files = await brain.find({
where: { 'metadata.vfsType': 'file' },
limit: 100000 // Large limit to get all files
})
if (verbose) {
console.log(`📁 Found ${files.length} VFS files to process`)
}
// Process in batches
for (let i = 0; i < files.length; i += batchSize) {
const batch = files.slice(i, i + batchSize)
await Promise.all(batch.map(async (file) => {
try {
stats.filesProcessed++
const metadata = file.metadata as VFSMetadata
const storage = metadata.storage
// Already migrated?
if (storage?.type === 'blob') {
stats.alreadyMigrated++
if (verbose && stats.filesProcessed % 100 === 0) {
console.log(` ✓ ${metadata.path} (already migrated)`)
}
return
}
let buffer: Buffer | null = null
let sizeBefore = 0
// Migrate based on old storage type
if (!storage || storage.type === 'inline') {
// Inline storage: content in metadata.rawData
if (metadata.rawData) {
buffer = Buffer.from(metadata.rawData, 'base64')
sizeBefore = buffer.length
stats.inlineMigrated++
if (verbose && stats.filesProcessed % 100 === 0) {
console.log(` → ${metadata.path} (inline, ${sizeBefore} bytes)`)
}
}
} else if (storage.type === 'reference') {
// Reference storage: content stored as separate entity
if (storage.key) {
const contentEntity = await brain.get(storage.key)
if (contentEntity && contentEntity.data) {
buffer = Buffer.isBuffer(contentEntity.data)
? contentEntity.data
: Buffer.from(contentEntity.data as string)
sizeBefore = buffer.length
stats.referenceMigrated++
if (verbose && stats.filesProcessed % 100 === 0) {
console.log(` → ${metadata.path} (reference, ${sizeBefore} bytes)`)
}
// Delete old reference entity (unless dry run)
if (!dryRun) {
await brain.delete(storage.key)
}
}
}
} else if (storage.type === 'chunked') {
// Chunked storage: content split across multiple entities
if (storage.chunks && storage.chunks.length > 0) {
const chunkBuffers = await Promise.all(
storage.chunks.map(async (chunkId) => {
const chunkEntity = await brain.get(chunkId)
if (chunkEntity && chunkEntity.data) {
return Buffer.isBuffer(chunkEntity.data)
? chunkEntity.data
: Buffer.from(chunkEntity.data as string)
}
return Buffer.alloc(0)
})
)
buffer = Buffer.concat(chunkBuffers)
sizeBefore = buffer.length
stats.chunkedMigrated++
if (verbose && stats.filesProcessed % 100 === 0) {
console.log(` → ${metadata.path} (chunked, ${storage.chunks.length} chunks, ${sizeBefore} bytes)`)
}
// Delete old chunk entities (unless dry run)
if (!dryRun) {
await Promise.all(storage.chunks.map(id => brain.delete(id)))
}
}
}
// Store in BlobStorage
if (buffer) {
stats.totalSizeBefore += sizeBefore
if (!dryRun) {
// Write to BlobStorage (content-addressable)
const blobHash = await brain.vfs['blobStorage'].write(buffer)
// Check if deduplicated
const blobMeta = await brain.vfs['blobStorage'].getMetadata(blobHash)
if (blobMeta && blobMeta.refCount > 1) {
stats.deduplicated++
}
// Update VFS metadata
await brain.update(file.id, {
metadata: {
...metadata,
storage: {
type: 'blob',
hash: blobHash,
size: buffer.length,
compressed: blobMeta?.compressed
}
}
})
stats.totalSizeAfter += blobMeta?.compressedSize || buffer.length
} else {
// Dry run: just count sizes
stats.totalSizeAfter += buffer.length
}
}
} catch (error) {
stats.errors++
if (verbose) {
console.error(` ✗ Error migrating ${file.metadata?.path}: ${error.message}`)
}
}
}))
if (verbose && i + batchSize < files.length) {
const progress = Math.round(((i + batchSize) / files.length) * 100)
console.log(` Progress: ${progress}% (${i + batchSize}/${files.length})`)
}
}
stats.durationMs = Date.now() - startTime
if (verbose) {
console.log('\n✅ Migration complete!\n')
console.log('📊 Statistics:')
console.log(` Files processed: ${stats.filesProcessed}`)
console.log(` Inline migrated: ${stats.inlineMigrated}`)
console.log(` Reference migrated: ${stats.referenceMigrated}`)
console.log(` Chunked migrated: ${stats.chunkedMigrated}`)
console.log(` Already migrated: ${stats.alreadyMigrated}`)
console.log(` Deduplicated: ${stats.deduplicated}`)
console.log(` Errors: ${stats.errors}`)
console.log(` Size before: ${formatBytes(stats.totalSizeBefore)}`)
console.log(` Size after: ${formatBytes(stats.totalSizeAfter)}`)
console.log(` Space saved: ${formatBytes(stats.totalSizeBefore - stats.totalSizeAfter)}`)
console.log(` Duration: ${stats.durationMs}ms`)
if (dryRun) {
console.log('\n⚠️ DRY RUN: No changes were made')
}
}
return stats
} catch (error) {
if (verbose) {
console.error('❌ Migration failed:', error)
}
throw error
}
}
function formatBytes(bytes: number): string {
if (bytes === 0) return '0 Bytes'
const k = 1024
const sizes = ['Bytes', 'KB', 'MB', 'GB']
const i = Math.floor(Math.log(bytes) / Math.log(k))
return Math.round((bytes / Math.pow(k, i)) * 100) / 100 + ' ' + sizes[i]
}
/**
* Auto-detect and migrate if needed
* Called automatically on brain.init() if old format detected
*/
export async function autoMigrateIfNeeded(brain: Brainy): Promise<boolean> {
try {
// Check for old format files
const oldFormatFiles = await brain.find({
where: {
'metadata.vfsType': 'file',
'metadata.storage.type': { $in: ['inline', 'reference', 'chunked'] }
},
limit: 1
})
if (oldFormatFiles.length > 0) {
console.log('🔄 Detected v5.1.x VFS storage format. Auto-migrating to v5.2.0...')
await migrateVFSStorage(brain, { verbose: true })
return true
}
return false
} catch (error) {
console.error('❌ Auto-migration failed:', error)
return false
}
}

88
scripts/setup-github-models.sh
View file

@ -1,88 +0,0 @@
#!/bin/bash
# Setup GitHub branch with extracted model files for direct serving
# This creates a 'models' branch with uncompressed files that transformers.js can use directly
set -e
echo "🚀 Setting up GitHub models branch for reliable model serving..."
# Create a temporary directory for our work
TEMP_DIR=$(mktemp -d)
cd "$TEMP_DIR"
echo "📥 Downloading model tar.gz from release..."
curl -sL https://github.com/soulcraftlabs/brainy/releases/download/models-v1/all-MiniLM-L6-v2.tar.gz -o models.tar.gz
echo "📦 Extracting models..."
tar -xzf models.tar.gz
echo "🔧 Setting up models branch..."
git clone https://github.com/soulcraftlabs/brainy.git --depth 1 -b main repo
cd repo
# Create orphan branch for models (no history needed)
git checkout --orphan models
git rm -rf . 2>/dev/null || true
# Create the proper directory structure for transformers.js
mkdir -p Xenova/all-MiniLM-L6-v2/onnx
# Copy model files with correct structure
cp ../all-MiniLM-L6-v2/config.json Xenova/all-MiniLM-L6-v2/
cp ../all-MiniLM-L6-v2/tokenizer.json Xenova/all-MiniLM-L6-v2/
cp ../all-MiniLM-L6-v2/tokenizer_config.json Xenova/all-MiniLM-L6-v2/
cp ../all-MiniLM-L6-v2/onnx/model.onnx Xenova/all-MiniLM-L6-v2/onnx/
cp ../all-MiniLM-L6-v2/onnx/model_quantized.onnx Xenova/all-MiniLM-L6-v2/onnx/ 2>/dev/null || true
# Add README explaining this branch
cat > README.md << 'EOF'
# Brainy Model Files
This branch contains extracted model files for direct serving via raw.githubusercontent.com
## Structure
```
Xenova/
all-MiniLM-L6-v2/
config.json
tokenizer.json
tokenizer_config.json
onnx/
model.onnx
model_quantized.onnx
```
## Usage
These files are automatically loaded by Brainy when models are needed.
The URL pattern is:
`https://raw.githubusercontent.com/soulcraftlabs/brainy/models/Xenova/all-MiniLM-L6-v2/{file}`
## DO NOT EDIT
This branch is managed automatically. Do not edit files directly.
EOF
# Create .gitattributes for LFS if needed (for large model files)
cat > .gitattributes << 'EOF'
*.onnx filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
EOF
echo "📝 Creating commit..."
git add .
git commit -m "feat: extracted model files for direct GitHub serving
- Xenova/all-MiniLM-L6-v2 model files
- Proper directory structure for transformers.js
- Direct serving via raw.githubusercontent.com"
echo "✅ Models branch ready!"
echo ""
echo "To push this branch to GitHub, run:"
echo " cd $TEMP_DIR/repo"
echo " git push origin models"
echo ""
echo "Once pushed, models will be available at:"
echo " https://raw.githubusercontent.com/soulcraftlabs/brainy/models/Xenova/all-MiniLM-L6-v2/config.json"
echo " https://raw.githubusercontent.com/soulcraftlabs/brainy/models/Xenova/all-MiniLM-L6-v2/tokenizer.json"
echo " etc..."

28
scripts/test-with-memory.sh
View file

@ -1,28 +0,0 @@
#!/bin/bash
# Run tests with adequate memory for transformer models
echo "🧠 Running Brainy tests with 8GB heap allocation"
echo "This is required for the transformer model (ONNX runtime)"
echo "================================================"
# Set memory allocation
export NODE_OPTIONS='--max-old-space-size=8192'
# Run tests based on argument
if [ "$1" = "single" ]; then
echo "Running tests sequentially (memory-safe)..."
npx vitest run --pool=forks --poolOptions.forks.maxForks=1 --reporter=dot
elif [ "$1" = "quick" ]; then
echo "Running quick test..."
node test-quick.js
elif [ "$1" = "core" ]; then
echo "Running core tests only..."
npx vitest run tests/core.test.ts --reporter=verbose
else
echo "Running full test suite..."
echo "Note: This requires 8GB+ RAM available"
npm test
fi
echo ""
echo "Test complete. Memory was allocated at 8GB for ONNX runtime."

63
scripts/update-augmentations-metadata.sh
View file

@ -1,63 +0,0 @@
#!/bin/bash
# Script to update all augmentations with metadata declarations
echo "📝 Updating augmentations with metadata declarations..."
# Category 1: No metadata access ('none')
echo "🚫 Updating 'none' metadata augmentations..."
# RequestDeduplicatorAugmentation
sed -i "/export class RequestDeduplicatorAugmentation extends BaseAugmentation {/a\\
readonly metadata = 'none' as const // Doesn't access metadata" \
src/augmentations/requestDeduplicatorAugmentation.ts
# ConnectionPoolAugmentation
sed -i "/export class ConnectionPoolAugmentation extends BaseAugmentation {/a\\
readonly metadata = 'none' as const // Doesn't access metadata" \
src/augmentations/connectionPoolAugmentation.ts
# StorageAugmentation (abstract class)
sed -i "/export abstract class StorageAugmentation extends BaseAugmentation/a\\
readonly metadata = 'none' as const // Storage doesn't directly access metadata" \
src/augmentations/storageAugmentation.ts
# Category 2: Read-only access ('readonly')
echo "👁 Updating 'readonly' metadata augmentations..."
# WALAugmentation
sed -i "/export class WALAugmentation extends BaseAugmentation {/a\\
readonly metadata = 'readonly' as const // Reads metadata for logging" \
src/augmentations/walAugmentation.ts
# IndexAugmentation
sed -i "/export class IndexAugmentation extends BaseAugmentation {/a\\
readonly metadata = 'readonly' as const // Reads metadata to build indexes" \
src/augmentations/indexAugmentation.ts
# MonitoringAugmentation
sed -i "/export class MonitoringAugmentation extends BaseAugmentation {/a\\
readonly metadata = 'readonly' as const // Reads metadata for monitoring" \
src/augmentations/monitoringAugmentation.ts
# MetricsAugmentation
sed -i "/export class MetricsAugmentation extends BaseAugmentation {/a\\
readonly metadata = 'readonly' as const // Reads metadata for metrics" \
src/augmentations/metricsAugmentation.ts
# BatchProcessingAugmentation
sed -i "/export class BatchProcessingAugmentation extends BaseAugmentation {/a\\
readonly metadata = 'readonly' as const // Reads metadata for batching decisions" \
src/augmentations/batchProcessingAugmentation.ts
# EntityRegistryAugmentation
sed -i "/export class EntityRegistryAugmentation extends BaseAugmentation {/a\\
readonly metadata = 'readonly' as const // Reads metadata to register entities" \
src/augmentations/entityRegistryAugmentation.ts
# AutoRegisterEntitiesAugmentation
sed -i "/export class AutoRegisterEntitiesAugmentation extends BaseAugmentation {/a\\
readonly metadata = 'readonly' as const // Reads metadata for auto-registration" \
src/augmentations/entityRegistryAugmentation.ts
echo "✅ Done updating augmentations!"

20
src/aggregation/AggregationIndex.ts
View file

@ -312,7 +312,7 @@ export class AggregationIndex {
*/
async init(): Promise<void> {
// Load persisted definitions
const savedDefs = await this.storage.getMetadata(DEFINITIONS_KEY) as any
const savedDefs = await this.storage.getMetadata(DEFINITIONS_KEY)
if (savedDefs && typeof savedDefs === 'object' && savedDefs.definitions) {
const defs = savedDefs.definitions as Array<AggregateDefinition & { _hash?: string }>
@ -322,7 +322,7 @@ export class AggregationIndex {
const savedHash = def._hash || ''
// Load persisted state
const stateData = await this.storage.getMetadata(`${STATE_KEY_PREFIX}${def.name}__`) as any
const stateData = await this.storage.getMetadata(`${STATE_KEY_PREFIX}${def.name}__`)
if (stateData && stateData.groups && savedHash === currentHash) {
// Definition unchanged — load state
const groupMap = new Map<string, AggregateGroupState>()
@ -349,11 +349,13 @@ export class AggregationIndex {
// Restore native provider state from persistence
if (this.nativeProvider?.restoreState) {
const nativeState = await this.storage.getMetadata('__aggregation_native_state__') as any
const nativeState = await this.storage.getMetadata('__aggregation_native_state__')
if (nativeState && typeof nativeState === 'string') {
this.nativeProvider.restoreState(nativeState)
} else if (nativeState && typeof nativeState === 'object' && nativeState.data) {
this.nativeProvider.restoreState(nativeState.data)
// flush() persists `{ data: serializeState() }`, so `data` is the
// provider's serialized state string.
this.nativeProvider.restoreState(nativeState.data as string)
}
}
}
@ -367,7 +369,7 @@ export class AggregationIndex {
...def,
_hash: this.definitionHashes.get(def.name)
}))
await this.storage.saveMetadata(DEFINITIONS_KEY, { definitions: defsToSave } as any)
await this.storage.saveMetadata(DEFINITIONS_KEY, { definitions: defsToSave })
// Persist dirty states
for (const name of this.dirty) {
@ -376,7 +378,7 @@ export class AggregationIndex {
const groups = Array.from(stateMap.values())
await this.storage.saveMetadata(
`${STATE_KEY_PREFIX}${name}__`,
{ groups } as any
{ groups }
)
}
}
@ -386,7 +388,7 @@ export class AggregationIndex {
const nativeState = this.nativeProvider.serializeState()
await this.storage.saveMetadata(
'__aggregation_native_state__',
{ data: nativeState } as any
{ data: nativeState }
)
}
@ -613,7 +615,7 @@ export class AggregationIndex {
// Apply where filter on group keys
if (params.where && Object.keys(params.where).length > 0) {
if (!matchesMetadataFilter(group.groupKey as any, params.where)) continue
if (!matchesMetadataFilter(group.groupKey, params.where)) continue
}
// Compute result metrics from running state
@ -664,7 +666,7 @@ export class AggregationIndex {
// HAVING: filter groups by computed metric values (post-compute, O(groups), before
// sort/pagination). Reuses the where-operator engine over metrics + `count`.
if (params.having && Object.keys(params.having).length > 0) {
if (!matchesMetadataFilter({ ...metrics, count: totalCount } as any, params.having)) continue
if (!matchesMetadataFilter({ ...metrics, count: totalCount }, params.having)) continue
}
results.push({

262
src/api/ConfigAPI.ts
View file

@ -1,262 +0,0 @@
/**
* Configuration API for Brainy 3.0
* Provides configuration storage with optional encryption
*/
import { SecurityAPI } from './SecurityAPI.js'
import { StorageAdapter } from '../coreTypes.js'
export interface ConfigOptions {
encrypt?: boolean
decrypt?: boolean
}
export interface ConfigEntry {
key: string
value: any
encrypted: boolean
createdAt: number
updatedAt: number
}
export class ConfigAPI {
private security: SecurityAPI
private configCache: Map<string, ConfigEntry> = new Map()
private CONFIG_NOUN_PREFIX = '_config_'
constructor(private storage: StorageAdapter) {
this.security = new SecurityAPI()
}
/**
* Set a configuration value with optional encryption
*/
async set(params: {
key: string
value: any
encrypt?: boolean
}): Promise<void> {
const { key, value, encrypt = false } = params
// Serialize and optionally encrypt the value
let storedValue: any = value
if (typeof value !== 'string') {
storedValue = JSON.stringify(value)
}
if (encrypt) {
storedValue = await this.security.encrypt(storedValue)
}
// Create config entry
const entry: ConfigEntry = {
key,
value: storedValue,
encrypted: encrypt,
createdAt: this.configCache.get(key)?.createdAt || Date.now(),
updatedAt: Date.now()
}
// Store in cache
this.configCache.set(key, entry)
// Persist to storage as NounMetadata
const configId = this.CONFIG_NOUN_PREFIX + key
const nounMetadata = {
noun: 'config' as any,
...entry,
service: 'config',
createdAt: entry.createdAt,
updatedAt: entry.updatedAt
}
await this.storage.saveNounMetadata(configId, nounMetadata)
}
/**
* Get a configuration value with optional decryption
*/
async get(params: {
key: string
decrypt?: boolean
defaultValue?: any
}): Promise<any> {
const { key, decrypt, defaultValue } = params
// Check cache first
let entry = this.configCache.get(key)
// If not in cache, load from storage
if (!entry) {
const configId = this.CONFIG_NOUN_PREFIX + key
const metadata = await this.storage.getNounMetadata(configId)
if (!metadata) {
return defaultValue
}
// Extract ConfigEntry from NounMetadata
const createdAtVal = typeof metadata.createdAt === 'object' && metadata.createdAt !== null && 'seconds' in metadata.createdAt
? metadata.createdAt.seconds * 1000 + Math.floor((metadata.createdAt.nanoseconds || 0) / 1000000)
: ((metadata.createdAt as unknown as number) || Date.now())
const updatedAtVal = typeof metadata.updatedAt === 'object' && metadata.updatedAt !== null && 'seconds' in metadata.updatedAt
? metadata.updatedAt.seconds * 1000 + Math.floor((metadata.updatedAt.nanoseconds || 0) / 1000000)
: ((metadata.updatedAt as unknown as number) || createdAtVal)
entry = {
key: metadata.key as string,
value: metadata.value,
encrypted: metadata.encrypted as boolean,
createdAt: createdAtVal,
updatedAt: updatedAtVal
}
this.configCache.set(key, entry)
}
let value = entry.value
// Decrypt if needed
const shouldDecrypt = decrypt !== undefined ? decrypt : entry.encrypted
if (shouldDecrypt && entry.encrypted && typeof value === 'string') {
value = await this.security.decrypt(value)
}
// Try to parse JSON if it looks like JSON
if (typeof value === 'string' && (value.startsWith('{') || value.startsWith('['))) {
try {
value = JSON.parse(value)
} catch {
// Not JSON, return as string
}
}
return value
}
/**
* Delete a configuration value
*/
async delete(key: string): Promise<void> {
// Remove from cache
this.configCache.delete(key)
// Remove from storage
const configId = this.CONFIG_NOUN_PREFIX + key
await this.storage.deleteNounMetadata(configId)
}
/**
* List all configuration keys
*/
async list(): Promise<string[]> {
// Get all nouns and filter for config entries
const result = await this.storage.getNouns({ pagination: { limit: 10000 } })
const configKeys: string[] = []
for (const noun of result.items) {
if (noun.id.startsWith(this.CONFIG_NOUN_PREFIX)) {
configKeys.push(noun.id.substring(this.CONFIG_NOUN_PREFIX.length))
}
}
return configKeys
}
/**
* Check if a configuration key exists
*/
async has(key: string): Promise<boolean> {
if (this.configCache.has(key)) {
return true
}
const configId = this.CONFIG_NOUN_PREFIX + key
const metadata = await this.storage.getNounMetadata(configId)
return metadata !== null && metadata !== undefined
}
/**
* Clear all configuration
*/
async clear(): Promise<void> {
// Clear cache
this.configCache.clear()
// Clear from storage
const keys = await this.list()
for (const key of keys) {
await this.delete(key)
}
}
/**
* Export all configuration
*/
async export(): Promise<Record<string, ConfigEntry>> {
const keys = await this.list()
const config: Record<string, ConfigEntry> = {}
for (const key of keys) {
const entry = await this.getEntry(key)
if (entry) {
config[key] = entry
}
}
return config
}
/**
* Import configuration
*/
async import(config: Record<string, ConfigEntry>): Promise<void> {
for (const [key, entry] of Object.entries(config)) {
this.configCache.set(key, entry)
const configId = this.CONFIG_NOUN_PREFIX + key
// Convert ConfigEntry to NounMetadata
const nounMetadata = {
noun: 'config' as any,
...entry,
service: 'config',
createdAt: entry.createdAt,
updatedAt: entry.updatedAt
}
await this.storage.saveNounMetadata(configId, nounMetadata)
}
}
/**
* Get raw config entry (without decryption)
*/
private async getEntry(key: string): Promise<ConfigEntry | null> {
if (this.configCache.has(key)) {
return this.configCache.get(key)!
}
const configId = this.CONFIG_NOUN_PREFIX + key
const metadata = await this.storage.getNounMetadata(configId)
if (!metadata) {
return null
}
// Extract ConfigEntry from NounMetadata
const createdAtVal = typeof metadata.createdAt === 'object' && metadata.createdAt !== null && 'seconds' in metadata.createdAt
? metadata.createdAt.seconds * 1000 + Math.floor((metadata.createdAt.nanoseconds || 0) / 1000000)
: ((metadata.createdAt as unknown as number) || Date.now())
const updatedAtVal = typeof metadata.updatedAt === 'object' && metadata.updatedAt !== null && 'seconds' in metadata.updatedAt
? metadata.updatedAt.seconds * 1000 + Math.floor((metadata.updatedAt.nanoseconds || 0) / 1000000)
: ((metadata.updatedAt as unknown as number) || createdAtVal)
const entry: ConfigEntry = {
key: metadata.key as string,
value: metadata.value,
encrypted: metadata.encrypted as boolean,
createdAt: createdAtVal,
updatedAt: updatedAtVal
}
this.configCache.set(key, entry)
return entry
}
}

163
src/api/SecurityAPI.ts
View file

@ -1,163 +0,0 @@
/**
* Security API for Brainy 3.0
* Provides encryption, decryption, hashing, and secure storage
*/
export class SecurityAPI {
private encryptionKey?: Uint8Array
constructor(private config?: { encryptionKey?: string }) {
if (config?.encryptionKey) {
// Use provided key (must be 32 bytes hex string)
this.encryptionKey = this.hexToBytes(config.encryptionKey)
}
}
/**
* Encrypt data using AES-256-CBC
*/
async encrypt(data: string): Promise<string> {
const crypto = await import('../universal/crypto.js')
// Generate or use existing key
const key = this.encryptionKey || crypto.randomBytes(32)
const iv = crypto.randomBytes(16)
const cipher = crypto.createCipheriv('aes-256-cbc', key, iv)
let encrypted = cipher.update(data, 'utf8', 'hex')
encrypted += cipher.final('hex')
// Package encrypted data with metadata
// In production, store keys separately in a key management service
return JSON.stringify({
encrypted,
key: this.bytesToHex(key),
iv: this.bytesToHex(iv),
algorithm: 'aes-256-cbc',
timestamp: Date.now()
})
}
/**
* Decrypt data encrypted with encrypt()
*/
async decrypt(encryptedData: string): Promise<string> {
const crypto = await import('../universal/crypto.js')
try {
const parsed = JSON.parse(encryptedData)
const { encrypted, key: keyHex, iv: ivHex, algorithm } = parsed
if (algorithm && algorithm !== 'aes-256-cbc') {
throw new Error(`Unsupported encryption algorithm: ${algorithm}`)
}
const key = this.hexToBytes(keyHex)
const iv = this.hexToBytes(ivHex)
const decipher = crypto.createDecipheriv('aes-256-cbc', key, iv)
let decrypted = decipher.update(encrypted, 'hex', 'utf8')
decrypted += decipher.final('utf8')
return decrypted
} catch (error) {
throw new Error(`Decryption failed: ${(error as Error).message}`)
}
}
/**
* Create a one-way hash of data (for passwords, etc)
*/
async hash(data: string, algorithm: 'sha256' | 'sha512' = 'sha256'): Promise<string> {
const crypto = await import('../universal/crypto.js')
const hash = crypto.createHash(algorithm)
hash.update(data)
return hash.digest('hex')
}
/**
* Compare data with a hash (for password verification)
*/
async compare(data: string, hash: string, algorithm: 'sha256' | 'sha512' = 'sha256'): Promise<boolean> {
const dataHash = await this.hash(data, algorithm)
return this.constantTimeCompare(dataHash, hash)
}
/**
* Generate a secure random token
*/
async generateToken(bytes: number = 32): Promise<string> {
const crypto = await import('../universal/crypto.js')
const buffer = crypto.randomBytes(bytes)
return this.bytesToHex(buffer)
}
/**
* Derive a key from a password using PBKDF2
* Note: Simplified version using hash instead of PBKDF2 which may not be available
*/
async deriveKey(password: string, salt?: string, iterations: number = 100000): Promise<{
key: string
salt: string
}> {
const crypto = await import('../universal/crypto.js')
const actualSalt = salt || this.bytesToHex(crypto.randomBytes(32))
// Simplified key derivation using repeated hashing
// In production, use a proper PBKDF2 implementation
let derived = password + actualSalt
for (let i = 0; i < Math.min(iterations, 1000); i++) {
const hash = crypto.createHash('sha256')
hash.update(derived)
derived = hash.digest('hex')
}
return {
key: derived,
salt: actualSalt
}
}
/**
* Sign data with HMAC
*/
async sign(data: string, secret?: string): Promise<string> {
const crypto = await import('../universal/crypto.js')
const actualSecret = secret || (await this.generateToken())
const hmac = crypto.createHmac('sha256', actualSecret)
hmac.update(data)
return hmac.digest('hex')
}
/**
* Verify HMAC signature
*/
async verify(data: string, signature: string, secret: string): Promise<boolean> {
const expectedSignature = await this.sign(data, secret)
return this.constantTimeCompare(signature, expectedSignature)
}
// Helper methods
private hexToBytes(hex: string): Uint8Array {
const matches = hex.match(/.{1,2}/g)
if (!matches) throw new Error('Invalid hex string')
return new Uint8Array(matches.map(byte => parseInt(byte, 16)))
}
private bytesToHex(bytes: Uint8Array | Buffer): string {
return Array.from(bytes)
.map(b => b.toString(16).padStart(2, '0'))
.join('')
}
private constantTimeCompare(a: string, b: string): boolean {
if (a.length !== b.length) return false
let result = 0
for (let i = 0; i < a.length; i++) {
result |= a.charCodeAt(i) ^ b.charCodeAt(i)
}
return result === 0
}
}

930
src/api/UniversalImportAPI.ts
View file

@ -1,930 +0,0 @@
/**
* Universal Neural Import API
*
* ALWAYS uses neural matching to map ANY data to our strict NounTypes and VerbTypes
* Never falls back to rules - neural matching is MANDATORY
*
* Handles:
* - Strings (text, JSON, CSV, YAML, Markdown)
* - Files (local paths, any format) - uses MimeTypeDetector for 2000+ types
* - URLs (web pages, APIs, documents)
* - Objects (structured data)
* - Binary data (images, PDFs via extraction)
*/
import { NounType, VerbType } from '../types/graphTypes.js'
import { Vector } from '../coreTypes.js'
import type { Brainy } from '../brainy.js'
import type { Entity, Relation } from '../types/brainy.types.js'
import { NeuralImportAugmentation } from '../neural/neuralImportAugmentation.js'
import { mimeDetector } from '../vfs/MimeTypeDetector.js'
export interface ImportSource {
type: 'string' | 'file' | 'url' | 'object' | 'binary'
data: any
format?: string // Optional hint about format
metadata?: any // Additional context
}
export interface NeuralImportResult {
entities: Array<{
id: string
type: NounType
data: any
vector: Vector
confidence: number
metadata: any
}>
relationships: Array<{
id: string
from: string
to: string
type: VerbType
weight: number
confidence: number
metadata?: any
}>
stats: {
totalProcessed: number
entitiesCreated: number
relationshipsCreated: number
averageConfidence: number
processingTimeMs: number
}
}
export interface NeuralImportProgress {
phase: 'extracting' | 'storing-entities' | 'storing-relationships' | 'complete'
message: string
current: number
total: number
entities?: number
relationships?: number
}
export class UniversalImportAPI {
private brain: Brainy<any>
private neuralImport: NeuralImportAugmentation
private embedCache = new Map<string, Vector>()
constructor(brain: Brainy<any>) {
this.brain = brain
this.neuralImport = new NeuralImportAugmentation({
confidenceThreshold: 0.0, // Accept ALL confidence levels - never reject
enableWeights: true,
skipDuplicates: false // Process everything
})
}
/**
* Initialize the neural import system
*/
async init(): Promise<void> {
// Neural import initializes itself
}
/**
* Universal import - handles ANY data source
* ALWAYS uses neural matching, NEVER falls back
*/
async import(
source: ImportSource | string | any,
options?: { onProgress?: (progress: NeuralImportProgress) => void }
): Promise<NeuralImportResult> {
const startTime = Date.now()
// Normalize source
const normalizedSource = this.normalizeSource(source)
options?.onProgress?.({
phase: 'extracting',
message: 'Extracting data from source...',
current: 0,
total: 0
})
// Extract data based on source type
const extractedData = await this.extractData(normalizedSource)
// Neural processing - MANDATORY
const neuralResults = await this.neuralProcess(extractedData)
// Store in brain
const result = await this.storeInBrain(neuralResults, options?.onProgress)
result.stats.processingTimeMs = Date.now() - startTime
options?.onProgress?.({
phase: 'complete',
message: 'Import complete',
current: result.stats.entitiesCreated + result.stats.relationshipsCreated,
total: result.stats.totalProcessed,
entities: result.stats.entitiesCreated,
relationships: result.stats.relationshipsCreated
})
return result
}
/**
* Import from URL - fetches and processes
*/
async importFromURL(url: string): Promise<NeuralImportResult> {
const response = await fetch(url)
const contentType = response.headers.get('content-type') || 'text/plain'
let data: any
if (contentType.includes('json')) {
data = await response.json()
} else if (contentType.includes('text') || contentType.includes('html')) {
data = await response.text()
} else {
// Binary data
const buffer = await response.arrayBuffer()
data = new Uint8Array(buffer)
}
return this.import({
type: 'url',
data,
format: contentType,
metadata: { url, fetchedAt: Date.now() }
})
}
/**
* Import from file - reads and processes
* Note: In browser environment, use File API instead
*
* Uses MimeTypeDetector for comprehensive format detection (2000+ types)
*/
async importFromFile(filePath: string): Promise<NeuralImportResult> {
// Read the actual file content
const { readFileSync } = await import('node:fs')
// Use MimeTypeDetector for comprehensive format detection
const mimeType = mimeDetector.detectMimeType(filePath)
const ext = filePath.split('.').pop()?.toLowerCase() || 'txt'
try {
const fileContent = readFileSync(filePath, 'utf-8')
return this.import({
type: 'file',
data: fileContent, // Actual file content
format: ext, // Keep ext for backward compatibility
metadata: {
path: filePath,
mimeType, // Add detected MIME type
importedAt: Date.now(),
fileSize: fileContent.length
}
})
} catch (error) {
throw new Error(`Failed to read file ${filePath}: ${(error as Error).message}`)
}
}
/**
* Normalize any input to ImportSource
*/
private normalizeSource(source: any): ImportSource {
// Already normalized
if (source && typeof source === 'object' && 'type' in source && 'data' in source) {
return source as ImportSource
}
// String input
if (typeof source === 'string') {
// Check if it's a URL
if (source.startsWith('http://') || source.startsWith('https://')) {
return { type: 'url', data: source }
}
// Check if it looks like a file path
if (source.includes('/') || source.includes('\\') || source.includes('.')) {
// Assume it's a file path reference
return { type: 'file', data: source }
}
// Treat as raw string data
return { type: 'string', data: source }
}
// Object/Array input
if (typeof source === 'object') {
return { type: 'object', data: source }
}
// Default to string
return { type: 'string', data: String(source) }
}
/**
* Extract structured data from source
*/
private async extractData(source: ImportSource): Promise<any[]> {
switch (source.type) {
case 'url':
// URL is in data field, need to fetch
return this.extractFromURL(source.data)
case 'file':
// File path is in data field, need to read
return this.extractFromFile(source.data)
case 'string':
return this.extractFromString(source.data, source.format)
case 'object':
return Array.isArray(source.data) ? source.data : [source.data]
case 'binary':
return this.extractFromBinary(source.data, source.format)
default:
// Unknown type, treat as object
return [source.data]
}
}
/**
* Extract data from URL
*/
private async extractFromURL(url: string): Promise<any[]> {
const result = await this.importFromURL(url)
return result.entities.map(e => e.data)
}
/**
* Extract data from file
*/
private async extractFromFile(filePath: string): Promise<any[]> {
const result = await this.importFromFile(filePath)
return result.entities.map(e => e.data)
}
/**
* Extract data from string based on format
*/
private extractFromString(data: string, format?: string): any[] {
// Try to detect format if not provided
const detectedFormat = format || this.detectFormat(data)
switch (detectedFormat) {
case 'json':
try {
const parsed = JSON.parse(data)
return Array.isArray(parsed) ? parsed : [parsed]
} catch {
// Not valid JSON, treat as text
return this.extractFromText(data)
}
case 'csv':
return this.parseCSV(data)
case 'yaml':
case 'yml':
return this.parseYAML(data)
case 'markdown':
case 'md':
return this.parseMarkdown(data)
case 'xml':
case 'html':
return this.parseHTML(data)
default:
return this.extractFromText(data)
}
}
/**
* Extract from binary data (images, PDFs, etc)
*/
private async extractFromBinary(data: Uint8Array, format?: string): Promise<any[]> {
// For now, create a single entity representing the binary data
// In production, would use OCR, image recognition, PDF extraction, etc.
return [{
type: 'binary',
format: format || 'unknown',
size: data.length,
hash: await this.hashBinary(data),
extractedAt: Date.now()
}]
}
/**
* Extract entities from plain text
*/
private extractFromText(text: string): any[] {
// Split into meaningful chunks
const chunks: any[] = []
// Split by paragraphs
const paragraphs = text.split(/\n\n+/)
for (const para of paragraphs) {
if (para.trim()) {
chunks.push({
text: para.trim(),
type: 'paragraph',
length: para.length
})
}
}
// If no paragraphs, split by sentences
if (chunks.length === 0) {
const sentences = text.match(/[^.!?]+[.!?]+/g) || [text]
for (const sentence of sentences) {
if (sentence.trim()) {
chunks.push({
text: sentence.trim(),
type: 'sentence',
length: sentence.length
})
}
}
}
return chunks
}
/**
* Neural processing - CORE of the system
* ALWAYS uses embeddings and neural matching
*/
private async neuralProcess(data: any[]): Promise<{
entities: Map<string, any>
relationships: Map<string, any>
}> {
const entities = new Map<string, any>()
const relationships = new Map<string, any>()
for (const item of data) {
// Generate embedding for the item
const embedding = await this.generateEmbedding(item)
// Determine noun type from data
const nounType = this.inferNounType(item)
const entityId = this.generateId(item)
entities.set(entityId, {
id: entityId,
type: nounType,
data: item,
vector: embedding,
confidence: 1.0,
metadata: {
...item,
_importedAt: Date.now()
}
})
// Detect relationships using neural matching
await this.detectNeuralRelationships(item, entityId, entities, relationships)
}
return { entities, relationships }
}
/**
* Generate embedding for any data
*/
private async generateEmbedding(data: any): Promise<Vector> {
// Convert to string for embedding
const text = this.dataToText(data)
// Check cache
if (this.embedCache.has(text)) {
return this.embedCache.get(text)!
}
// Generate new embedding
const embedding = await (this.brain as any).embed(text)
// Cache it
this.embedCache.set(text, embedding)
return embedding
}
/**
* Convert any data to text for embedding
*/
private dataToText(data: any): string {
if (typeof data === 'string') return data
if (typeof data === 'object') {
// Extract meaningful text from object
const parts: string[] = []
// Priority fields
const priorityFields = ['name', 'title', 'description', 'text', 'content', 'label', 'value']
for (const field of priorityFields) {
if (data[field]) {
parts.push(String(data[field]))
}
}
// Add other fields
for (const [key, value] of Object.entries(data)) {
if (!priorityFields.includes(key) && value) {
if (typeof value === 'string' || typeof value === 'number') {
parts.push(`${key}: ${value}`)
}
}
}
return parts.join(' ')
}
return JSON.stringify(data)
}
/**
* Detect relationships using neural matching
*/
private async detectNeuralRelationships(
item: any,
sourceId: string,
entities: Map<string, any>,
relationships: Map<string, any>
): Promise<void> {
if (typeof item !== 'object') return
// Look for references to other entities
for (const [key, value] of Object.entries(item)) {
// Check if this looks like a reference
if (this.looksLikeReference(key, value)) {
const targetId = String(value)
const verbType = this.inferVerbType(key)
const relationId = `${sourceId}_${verbType}_${targetId}`
relationships.set(relationId, {
id: relationId,
from: sourceId,
to: targetId,
type: verbType,
weight: 1.0,
confidence: 1.0,
metadata: {
field: key,
_importedAt: Date.now()
}
})
}
// Handle arrays of references
if (Array.isArray(value)) {
for (const arrayItem of value) {
if (this.looksLikeReference(key, arrayItem)) {
const targetId = String(arrayItem)
const verbType = this.inferVerbType(key)
const relationId = `${sourceId}_${verbType}_${targetId}`
relationships.set(relationId, {
id: relationId,
from: sourceId,
to: targetId,
type: verbType,
weight: 1.0,
confidence: 1.0,
metadata: {
field: key,
array: true,
_importedAt: Date.now()
}
})
}
}
}
}
}
/**
* Check if a field looks like a reference
*/
private looksLikeReference(key: string, value: any): boolean {
// Field name patterns that suggest references
const refPatterns = [
/[Ii]d$/, // ends with Id or id
/_id$/, // ends with _id
/^parent/i, // starts with parent
/^child/i, // starts with child
/^related/i, // starts with related
/^ref/i, // starts with ref
/^link/i, // starts with link
/^target/i, // starts with target
/^source/i, // starts with source
]
// Check if field name matches patterns
const fieldLooksLikeRef = refPatterns.some(pattern => pattern.test(key))
// Check if value looks like an ID
const valueLooksLikeId = (
typeof value === 'string' ||
typeof value === 'number'
) && String(value).length > 0
return fieldLooksLikeRef && valueLooksLikeId
}
/**
* Infer noun type from object structure using field heuristics
*/
private inferNounType(obj: any): NounType {
if (typeof obj !== 'object' || obj === null) return NounType.Thing
// Check for explicit type field
if (obj.type && typeof obj.type === 'string') {
const normalized = obj.type.charAt(0).toUpperCase() + obj.type.slice(1)
if (Object.values(NounType).includes(normalized as NounType)) {
return normalized as NounType
}
}
// Person heuristics
if (obj.email || obj.firstName || obj.lastName || obj.username || obj.age) {
return NounType.Person
}
// Organization heuristics
if (obj.companyName || obj.organizationId || obj.employees || obj.industry) {
return NounType.Organization
}
// Location heuristics
if (obj.latitude || obj.longitude || obj.address || obj.city || obj.country) {
return NounType.Location
}
// Document heuristics
if ((obj.content && (obj.title || obj.author)) || obj.documentType || obj.pages) {
return NounType.Document
}
// Event heuristics
if (obj.startTime || obj.endTime || obj.date || obj.eventType || obj.attendees) {
return NounType.Event
}
// Product heuristics
if (obj.price || obj.sku || obj.inventory || obj.productId) {
return NounType.Product
}
// Task heuristics
if ((obj.status && (obj.assignee || obj.dueDate)) || obj.priority || obj.completed !== undefined) {
return NounType.Task
}
// Dataset heuristics
if (Array.isArray(obj.data) || obj.rows || obj.columns || obj.schema) {
return NounType.Dataset
}
return NounType.Thing
}
/**
* Infer verb type from field name using common patterns
*/
private inferVerbType(fieldName: string): VerbType {
const field = fieldName.toLowerCase()
if (field.includes('parent') || field.includes('child') || field.includes('contain')) {
return VerbType.Contains
}
if (field.includes('owner') || field.includes('created') || field.includes('author')) {
return VerbType.Creates
}
if (field.includes('member') || field.includes('belong')) {
return VerbType.MemberOf
}
if (field.includes('depend') || field.includes('require')) {
return VerbType.DependsOn
}
if (field.includes('ref') || field.includes('link') || field.includes('source')) {
return VerbType.References
}
return VerbType.RelatedTo
}
/**
* Store processed data in brain
*/
private async storeInBrain(
neuralResults: {
entities: Map<string, any>
relationships: Map<string, any>
},
onProgress?: (progress: NeuralImportProgress) => void
): Promise<NeuralImportResult> {
const result: NeuralImportResult = {
entities: [],
relationships: [],
stats: {
totalProcessed: neuralResults.entities.size + neuralResults.relationships.size,
entitiesCreated: 0,
relationshipsCreated: 0,
averageConfidence: 0,
processingTimeMs: 0
}
}
let totalConfidence = 0
// Store entities
onProgress?.({
phase: 'storing-entities',
message: 'Storing entities...',
current: 0,
total: neuralResults.entities.size
})
let entitiesProcessed = 0
for (const entity of neuralResults.entities.values()) {
// Subtype precedence: extractor-set → `'extracted'` (this is the universal
// neural-extraction path, so `'extracted'` is the honest provenance label).
// Added 7.30.1 so enforcement consumers don't get rejected on auto-extraction.
const id = await this.brain.add({
data: entity.data,
type: entity.type,
subtype: (entity as any).subtype ?? 'extracted',
metadata: entity.metadata,
vector: entity.vector
})
// Update entity ID for relationship mapping
entity.id = id
result.entities.push({
...entity,
id
})
result.stats.entitiesCreated++
totalConfidence += entity.confidence
entitiesProcessed++
// Report progress periodically
if (entitiesProcessed % 10 === 0 || entitiesProcessed === neuralResults.entities.size) {
onProgress?.({
phase: 'storing-entities',
message: `Storing entities: ${entitiesProcessed}/${neuralResults.entities.size}`,
current: entitiesProcessed,
total: neuralResults.entities.size,
entities: entitiesProcessed
})
}
}
// Store relationships using batch processing
if (neuralResults.relationships.size > 0) {
onProgress?.({
phase: 'storing-relationships',
message: 'Preparing relationships...',
current: 0,
total: neuralResults.relationships.size
})
// Collect all relationship parameters. Subtype `extracted` matches the
// entity-side label so consumers can query "everything from this neural pass"
// via `(type, subtype: 'extracted')` (added 7.30.1).
const relationshipParams: Array<{from: string; to: string; type: VerbType; subtype?: string; weight?: number; metadata?: any}> = []
for (const relation of neuralResults.relationships.values()) {
// Map to actual entity IDs
const sourceEntity = Array.from(neuralResults.entities.values())
.find(e => e.id === relation.from)
const targetEntity = Array.from(neuralResults.entities.values())
.find(e => e.id === relation.to)
if (sourceEntity && targetEntity) {
relationshipParams.push({
from: sourceEntity.id,
to: targetEntity.id,
type: relation.type,
subtype: (relation as any).subtype ?? 'extracted',
weight: relation.weight,
metadata: relation.metadata
})
totalConfidence += relation.confidence
}
}
// Batch create relationships with progress
if (relationshipParams.length > 0) {
const relationshipIds = await this.brain.relateMany({
items: relationshipParams,
parallel: true,
chunkSize: 100,
continueOnError: true,
onProgress: (done, total) => {
onProgress?.({
phase: 'storing-relationships',
message: `Building relationships: ${done}/${total}`,
current: done,
total: total,
entities: result.stats.entitiesCreated,
relationships: done
})
}
})
// Map results back
relationshipIds.forEach((id, index) => {
if (id && relationshipParams[index]) {
result.relationships.push({
id,
from: relationshipParams[index].from,
to: relationshipParams[index].to,
type: relationshipParams[index].type,
weight: relationshipParams[index].weight || 1,
confidence: 0.5, // Default confidence
metadata: relationshipParams[index].metadata
})
}
})
result.stats.relationshipsCreated = relationshipIds.length
}
}
// Calculate average confidence
const totalItems = result.stats.entitiesCreated + result.stats.relationshipsCreated
result.stats.averageConfidence = totalItems > 0 ? totalConfidence / totalItems : 0
return result
}
// Helper methods for parsing different formats
private detectFormat(data: string): string {
const trimmed = data.trim()
// JSON
if ((trimmed.startsWith('{') && trimmed.endsWith('}')) ||
(trimmed.startsWith('[') && trimmed.endsWith(']'))) {
return 'json'
}
// CSV (has commas and newlines)
if (trimmed.includes(',') && trimmed.includes('\n')) {
return 'csv'
}
// YAML (has colons and indentation)
if (trimmed.includes(':') && (trimmed.includes('\n ') || trimmed.includes('\n\t'))) {
return 'yaml'
}
// Markdown (has headers)
if (trimmed.includes('#') || trimmed.includes('```')) {
return 'markdown'
}
// HTML/XML
if (trimmed.includes('<') && trimmed.includes('>')) {
return trimmed.toLowerCase().includes('<!doctype html') ? 'html' : 'xml'
}
return 'text'
}
private parseCSV(data: string): any[] {
// Reuse the CSV parser from neural import
const lines = data.split('\n').filter(l => l.trim())
if (lines.length === 0) return []
const headers = lines[0].split(',').map(h => h.trim())
const results = []
for (let i = 1; i < lines.length; i++) {
const values = lines[i].split(',').map(v => v.trim())
const obj: any = {}
headers.forEach((header, index) => {
obj[header] = values[index] || ''
})
results.push(obj)
}
return results
}
private parseYAML(data: string): any[] {
// Simple YAML parser
const results = []
const lines = data.split('\n')
let current: any = null
for (const line of lines) {
const trimmed = line.trim()
if (!trimmed || trimmed.startsWith('#')) continue
if (trimmed.startsWith('- ')) {
// Array item
const value = trimmed.substring(2)
if (!current) {
results.push(value)
} else {
if (!current._items) current._items = []
current._items.push(value)
}
} else if (trimmed.includes(':')) {
// Key-value
const [key, ...valueParts] = trimmed.split(':')
const value = valueParts.join(':').trim()
if (!current) {
current = {}
results.push(current)
}
current[key.trim()] = value
}
}
return results.length > 0 ? results : [{ text: data }]
}
private parseMarkdown(data: string): any[] {
const results = []
const lines = data.split('\n')
let current: any = null
let inCodeBlock = false
for (const line of lines) {
if (line.startsWith('```')) {
inCodeBlock = !inCodeBlock
if (inCodeBlock && current) {
current.code = ''
}
continue
}
if (inCodeBlock && current) {
current.code += line + '\n'
} else if (line.startsWith('#')) {
// Header
const level = line.match(/^#+/)?.[0].length || 1
const text = line.replace(/^#+\s*/, '')
current = {
type: 'heading',
level,
text
}
results.push(current)
} else if (line.trim()) {
// Paragraph
if (!current || current.type !== 'paragraph') {
current = {
type: 'paragraph',
text: ''
}
results.push(current)
}
current.text += line + ' '
}
}
return results
}
private parseHTML(data: string): any[] {
// Simple HTML text extraction
const text = data
.replace(/<script\b[^<]*(?:(?!<\/script>)<[^<]*)*<\/script>/gi, '') // Remove scripts
.replace(/<style\b[^<]*(?:(?!<\/style>)<[^<]*)*<\/style>/gi, '') // Remove styles
.replace(/<[^>]+>/g, ' ') // Remove tags
.replace(/\s+/g, ' ') // Normalize whitespace
.trim()
return this.extractFromText(text)
}
private generateId(data: any): string {
// Generate deterministic ID based on content
const text = this.dataToText(data)
const hash = this.simpleHash(text)
return `import_${hash}_${Date.now()}`
}
private simpleHash(text: string): string {
let hash = 0
for (let i = 0; i < text.length; i++) {
const char = text.charCodeAt(i)
hash = ((hash << 5) - hash) + char
hash = hash & hash
}
return Math.abs(hash).toString(36)
}
private async hashBinary(data: Uint8Array): Promise<string> {
// Simple binary hash
let hash = 0
for (let i = 0; i < Math.min(data.length, 1000); i++) {
hash = ((hash << 5) - hash) + data[i]
hash = hash & hash
}
return Math.abs(hash).toString(36)
}
}

513
src/brainy.ts
View file

File diff suppressed because it is too large Load diff

99
src/cli/commands/core.ts
View file

@ -7,10 +7,11 @@
*/
import chalk from 'chalk'
import ora from 'ora'
import ora, { type Ora } from 'ora'
import inquirer from 'inquirer'
import { Brainy } from '../../brainy.js'
import { BrainyTypes, NounType, VerbType } from '../../index.js'
import type { Entity, Result } from '../../types/brainy.types.js'
interface CoreOptions {
verbose?: boolean
@ -87,7 +88,7 @@ export const coreCommands = {
* Add data to the neural database
*/
async add(text: string | undefined, options: AddOptions) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no text provided
if (!text) {
@ -222,7 +223,7 @@ export const coreCommands = {
* Search the neural database with Triple Intelligence™
*/
async search(query: string | undefined, options: SearchOptions) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no query provided
if (!query) {
@ -407,8 +408,13 @@ export const coreCommands = {
if (result.score !== undefined) {
console.log(chalk.green(` Score: ${(result.score * 100).toFixed(1)}%`))
if (options.explain && (result as any).scores) {
const scores = (result as any).scores
// Per-intelligence sub-scores aren't part of the public Result
// contract (the typed breakdown lives on `explanation`) — read
// defensively so --explain degrades to no breakdown when absent.
const scores = (result as Result & {
scores?: { vector?: number; graph?: number; field?: number }
}).scores
if (options.explain && scores) {
if (scores.vector !== undefined) {
console.log(chalk.dim(` Vector: ${(scores.vector * 100).toFixed(1)}%`))
}
@ -422,24 +428,28 @@ export const coreCommands = {
}
// Show type
if ((entity as any).type) {
console.log(chalk.dim(` Type: ${(entity as any).type}`))
if (entity.type) {
console.log(chalk.dim(` Type: ${entity.type}`))
}
// Show content preview
if ((entity as any).content) {
const preview = (entity as any).content.substring(0, 80)
console.log(chalk.dim(` Content: ${preview}${(entity as any).content.length > 80 ? '...' : ''}`))
// Show content preview. `content` is not a standard Entity field —
// surfaced defensively for records that carry flattened text content.
const content = (entity as Entity & { content?: string }).content
if (content) {
const preview = content.substring(0, 80)
console.log(chalk.dim(` Content: ${preview}${content.length > 80 ? '...' : ''}`))
}
// Show metadata
if ((entity as any).metadata && Object.keys((entity as any).metadata).length > 0) {
console.log(chalk.dim(` Metadata: ${JSON.stringify((entity as any).metadata)}`))
if (entity.metadata && Object.keys(entity.metadata).length > 0) {
console.log(chalk.dim(` Metadata: ${JSON.stringify(entity.metadata)}`))
}
// Show relationships
if (options.includeRelations && (result as any).relations) {
const relations = (result as any).relations
// Show relationships. `relations` isn't part of the public Result
// contract — read defensively so --include-relations degrades to
// no output when find() doesn't attach it.
const relations = (result as Result & { relations?: unknown[] }).relations
if (options.includeRelations && relations) {
if (relations.length > 0) {
console.log(chalk.dim(` Relations: ${relations.length} connections`))
}
@ -487,7 +497,7 @@ export const coreCommands = {
* Get item by ID
*/
async get(id: string | undefined, options: GetOptions) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no ID provided
if (!id) {
@ -527,18 +537,25 @@ export const coreCommands = {
if (!options.json) {
console.log(chalk.cyan('\nItem Details:'))
console.log(` ID: ${item.id}`)
console.log(` Content: ${(item as any).content || 'N/A'}`)
// `content` is not a standard Entity field — see the search() note.
console.log(` Content: ${(item as Entity & { content?: string }).content || 'N/A'}`)
if (item.metadata) {
console.log(` Metadata: ${JSON.stringify(item.metadata, null, 2)}`)
}
if (options.withConnections) {
// Get verbs/relationships
// Get connections if method exists
const connections = (brain as any).getConnections ? await (brain as any).getConnections(id) : []
// Brainy's public API has no getConnections() — this optional probe
// predates related() and resolves to an empty list on current
// builds. Typed to match the legacy edge shape it rendered.
const legacyBrain = brain as Brainy & {
getConnections?: (id: string | undefined) => Promise<
Array<{ source: string; type: string; target: string }>
>
}
const connections = legacyBrain.getConnections ? await legacyBrain.getConnections(id) : []
if (connections && connections.length > 0) {
console.log(chalk.cyan('\nConnections:'))
connections.forEach((conn: any) => {
connections.forEach((conn) => {
console.log(` ${conn.source} --[${conn.type}]--> ${conn.target}`)
})
}
@ -561,7 +578,7 @@ export const coreCommands = {
* Create relationship between items
*/
async relate(source: string | undefined, verb: string | undefined, target: string | undefined, options: RelateOptions) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if parameters missing
if (!source || !verb || !target) {
@ -630,10 +647,12 @@ export const coreCommands = {
// `--subtype <value>` → Brainy default `'cli-relate'`. Same rationale
// as `brainy add` — guarantees CLI works under strict-mode brains
// (added 7.30.1).
// The verb arrives as a raw CLI string; relate() validates it against
// the VerbType vocabulary at runtime.
const result = await brain.relate({
from: source,
to: target,
type: verb as any,
type: verb as VerbType,
subtype: options.subtype ?? 'cli-relate',
metadata
})
@ -664,7 +683,7 @@ export const coreCommands = {
* Update an existing entity
*/
async update(id: string | undefined, options: AddOptions & { content?: string }) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no ID provided
if (!id) {
@ -770,10 +789,10 @@ export const coreCommands = {
},
/**
* Delete an entity
* Remove an entity
*/
async deleteEntity(id: string | undefined, options: CoreOptions & { force?: boolean }) {
let spinner: any = null
async removeEntity(id: string | undefined, options: CoreOptions & { force?: boolean }) {
let spinner: Ora | null = null
try {
// Interactive mode if no ID provided
if (!id) {
@ -781,7 +800,7 @@ export const coreCommands = {
{
type: 'input',
name: 'id',
message: 'Entity ID to delete:',
message: 'Entity ID to remove:',
validate: (input: string) => input.trim().length > 0 || 'ID cannot be empty'
},
{
@ -793,7 +812,7 @@ export const coreCommands = {
])
if (!answers.confirm) {
console.log(chalk.yellow('Delete cancelled'))
console.log(chalk.yellow('Operation cancelled'))
return
}
@ -803,36 +822,36 @@ export const coreCommands = {
const answer = await inquirer.prompt([{
type: 'confirm',
name: 'confirm',
message: `Delete entity ${id}? This cannot be undone.`,
message: `Remove entity ${id}? This cannot be undone.`,
default: false
}])
if (!answer.confirm) {
console.log(chalk.yellow('Delete cancelled'))
console.log(chalk.yellow('Operation cancelled'))
return
}
}
spinner = ora('Deleting entity...').start()
spinner = ora('Removing entity...').start()
const brain = getBrainy()
await brain.init()
await brain.delete(id)
await brain.remove(id)
spinner.succeed('Entity deleted successfully')
spinner.succeed('Entity removed successfully')
if (!options.json) {
console.log(chalk.green(`✓ Deleted entity: ${id}`))
console.log(chalk.green(`✓ Removed entity: ${id}`))
} else {
formatOutput({ id, deleted: true }, options)
formatOutput({ id, removed: true }, options)
}
// One-shot command — see add() for why the explicit close + exit.
await brain.close()
process.exit(0)
} catch (error: any) {
if (spinner) spinner.fail('Failed to delete entity')
console.error(chalk.red('Delete failed:', error.message))
if (spinner) spinner.fail('Failed to remove entity')
console.error(chalk.red('Remove failed:', error.message))
process.exit(1)
}
},
@ -841,7 +860,7 @@ export const coreCommands = {
* Remove a relationship
*/
async unrelate(id: string | undefined, options: CoreOptions & { force?: boolean }) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no ID provided
if (!id) {

2
src/cli/commands/data.ts
View file

@ -10,7 +10,7 @@
* The legacy backup/import/export facade that used to live behind this
* command was superseded in 8.0: snapshots are `brainy snapshot` / `brainy
* restore` (the Db API's `persist()`/`restore()`), and data ingestion is
* `brainy import` (UniversalImportAPI).
* `brainy import` (`brain.import()`).
*/
import chalk from 'chalk'

23
src/cli/commands/import.ts
View file

@ -3,7 +3,7 @@
* @description Import commands — neural import and data import.
*
* Complete import system exposing ALL Brainy import capabilities:
* - UniversalImportAPI: Neural import with AI type matching
* - `brain.import()`: universal ingestion with neural type matching
* - DirectoryImporter: VFS directory imports
*
* Supports: files, directories, URLs, all formats. Backup/restore is the
@ -11,19 +11,23 @@
*/
import chalk from 'chalk'
import ora from 'ora'
import ora, { type Ora } from 'ora'
import inquirer from 'inquirer'
import { readFileSync, statSync, existsSync } from 'node:fs'
import { Brainy } from '../../brainy.js'
import { NounType } from '../../types/graphTypes.js'
import type { SupportedFormat } from '../../import/FormatDetector.js'
interface ImportOptions {
verbose?: boolean
json?: boolean
pretty?: boolean
quiet?: boolean
// Format options
format?: 'json' | 'csv' | 'jsonl' | 'yaml' | 'markdown' | 'html' | 'xml' | 'text'
// Format options. Populated by commander straight from `--format` with no
// validation, so the runtime value is whatever the user typed — kept as a
// plain string and narrowed to brain.import()'s SupportedFormat at the call
// boundary (the import extractor rejects unsupported values at runtime).
format?: string
// Import behavior
recursive?: boolean
batchSize?: string
@ -60,11 +64,11 @@ const formatOutput = (data: any, options: ImportOptions): void => {
export const importCommands = {
/**
* Enhanced import using UniversalImportAPI
* Supports files, directories, URLs, all formats
* Enhanced import via `brain.import()`.
* Supports files, directories, URLs, all formats.
*/
async import(source: string | undefined, options: ImportOptions) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no source provided
if (!source) {
@ -260,7 +264,8 @@ export const importCommands = {
} else {
// File import with progress
result = await brain.import(source, {
format: options.format as any,
// Raw CLI string → SupportedFormat (see ImportOptions.format note).
format: options.format as SupportedFormat | undefined,
enableNeuralExtraction: options.extractEntities !== false,
enableRelationshipInference: options.detectRelationships !== false,
enableConceptExtraction: options.extractConcepts || false,
@ -422,7 +427,7 @@ export const importCommands = {
* VFS-specific import (files/directories into VFS)
*/
async vfsImport(source: string | undefined, options: ImportOptions) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no source provided
if (!source) {

6
src/cli/commands/insights.ts
View file

@ -5,7 +5,7 @@
*/
import chalk from 'chalk'
import ora from 'ora'
import ora, { type Ora } from 'ora'
import inquirer from 'inquirer'
import Table from 'cli-table3'
import { Brainy } from '../../brainy.js'
@ -192,7 +192,7 @@ export const insightsCommands = {
* Get field values for a specific field
*/
async fieldValues(field: string | undefined, options: InsightsOptions & { limit?: string }) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no field provided
if (!field) {
@ -274,7 +274,7 @@ export const insightsCommands = {
* Get optimal query plan for filters
*/
async queryPlan(options: InsightsOptions & { filters?: string }) {
let spinner: any = null
let spinner: Ora | null = null
try {
let filters: Record<string, any> = {}

24
src/cli/commands/inspect.ts
View file

@ -15,6 +15,8 @@
import chalk from 'chalk'
import ora from 'ora'
import { Brainy } from '../../brainy.js'
import type { RelatedParams } from '../../types/brainy.types.js'
import type { NounType, VerbType } from '../../types/graphTypes.js'
interface InspectOptions {
fresh?: boolean
@ -109,8 +111,10 @@ export const inspectCommands = {
const where = options.where ? JSON.parse(options.where) : undefined
const limit = options.limit ? parseInt(options.limit, 10) : 20
const offset = options.offset ? parseInt(options.offset, 10) : 0
// --type arrives as a raw CLI string; find() resolves it against the
// NounType vocabulary at runtime.
const results = await brain.find({
type: options.type as any,
type: options.type as NounType | undefined,
where,
limit,
offset
@ -158,12 +162,16 @@ export const inspectCommands = {
if (spinner) spinner.text = 'Walking graph…'
const direction = options.direction ?? 'both'
const limit = options.limit ? parseInt(options.limit, 10) : 50
const rels = await brain.getRelations({
// `direction` is accepted by the CLI surface but RelatedParams has
// no such filter — related() reads from/to/type/limit and ignores
// extras, so the assertion only admits the extra key, it doesn't change
// what the query does. --type arrives as a raw CLI string.
const rels = await brain.related({
from: id,
direction,
type: options.type as any,
type: options.type as VerbType | undefined,
limit
} as any)
} as RelatedParams)
spinner?.succeed(`Found ${rels.length} relationship(s)`)
emit(rels, options)
await brain.close()
@ -185,7 +193,7 @@ export const inspectCommands = {
const brain = await openReader(path, options)
if (spinner) spinner.text = 'Planning query…'
const where = options.where ? JSON.parse(options.where) : {}
const plan = await brain.explain({ type: options.type as any, where })
const plan = await brain.explain({ type: options.type as NounType | undefined, where })
spinner?.succeed('Plan ready')
emit(plan, options)
await brain.close()
@ -228,7 +236,7 @@ export const inspectCommands = {
// dedicated random-sample API in Brainy core for this v1.
const window = Math.min(Math.max(n * 10, 50), 1000)
const pool = await brain.find({
type: options.type as any,
type: options.type as NounType | undefined,
limit: window
})
const sample = pool
@ -275,7 +283,7 @@ export const inspectCommands = {
let offset = 0
while (true) {
const page = await brain.find({
type: options.type as any,
type: options.type as NounType | undefined,
limit: batch,
offset
})
@ -310,7 +318,7 @@ export const inspectCommands = {
try {
const brain = await openReader(path, { ...options, quiet: true })
const page = await brain.find({
type: options.type as any,
type: options.type as NounType | undefined,
limit: 200
})
for (const e of page) {

9
src/cli/commands/neural.ts
View file

@ -10,10 +10,11 @@
import inquirer from 'inquirer'
import chalk from 'chalk'
import ora from 'ora'
import ora, { type Ora } from 'ora'
import fs from 'node:fs'
import path from 'node:path'
import { Brainy } from '../../brainy.js'
import type { NeuralClusterParams } from '../../types/brainy.types.js'
interface CommandArguments {
action?: string;
@ -113,10 +114,12 @@ async function handleSimilarCommand(neural: any, argv: CommandArguments): Promis
}
async function handleClustersCommand(neural: any, argv: CommandArguments): Promise<void> {
let spinner: any = null
let spinner: Ora | null = null
try {
let options: any = {
algorithm: argv.algorithm as any,
// --algorithm arrives as a raw CLI string; the interactive prompt below
// and neural.clusters() constrain it to the supported algorithms.
algorithm: argv.algorithm as NeuralClusterParams['algorithm'],
threshold: argv.threshold,
maxClusters: argv.limit
}

8
src/cli/commands/nlp.ts
View file

@ -5,7 +5,7 @@
*/
import chalk from 'chalk'
import ora from 'ora'
import ora, { type Ora } from 'ora'
import inquirer from 'inquirer'
import Table from 'cli-table3'
import { Brainy } from '../../brainy.js'
@ -37,7 +37,7 @@ export const nlpCommands = {
* Extract entities from text
*/
async extract(text: string | undefined, options: NLPOptions) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no text provided
if (!text) {
@ -126,7 +126,7 @@ export const nlpCommands = {
* Extract concepts from text
*/
async extractConcepts(text: string | undefined, options: NLPOptions & { threshold?: string }) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no text provided
if (!text) {
@ -197,7 +197,7 @@ export const nlpCommands = {
* Analyze text with full NLP pipeline
*/
async analyze(text: string | undefined, options: NLPOptions) {
let spinner: any = null
let spinner: Ora | null = null
try {
// Interactive mode if no text provided
if (!text) {

6
src/cli/commands/storage.ts
View file

@ -6,7 +6,7 @@
* - `storage status` — backend, root directory, entity/relationship counts,
* writer-lock holder, and operational mode (rendered from `brain.stats()`).
* - `storage batch-delete <file>` — delete a newline-separated list of entity
* IDs through the public `brain.delete()` path (vectors, metadata, indexes,
* IDs through the public `brain.remove()` path (vectors, metadata, indexes,
* and statistics all stay consistent), with per-ID retry and an optional
* continue-on-error mode.
*
@ -144,7 +144,7 @@ export const storageCommands = {
/**
* @description Delete a list of entity IDs (one per line in a file) through
* the public `brain.delete()` path so vectors, metadata, graph edges,
* the public `brain.remove()` path so vectors, metadata, graph edges,
* indexes, and statistics all stay consistent. Asks for confirmation before
* deleting; failed IDs are retried up to `--max-retries` times, and
* `--continue-on-error` keeps going past IDs that still fail.
@ -204,7 +204,7 @@ export const storageCommands = {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
await brain.delete(id)
await brain.remove(id)
succeeded = true
break
} catch (error) {

18
src/cli/commands/utility.ts
View file

@ -30,6 +30,15 @@ interface BenchmarkOptions extends UtilityOptions {
iterations?: string
}
/** Per-operation timing summary produced by `brainy benchmark`. */
interface BenchmarkOperationStats {
avg: string
min: number
max: number
median: number
ops: string
}
let brainyInstance: Brainy | null = null
const getBrainy = (): Brainy => {
@ -188,7 +197,10 @@ export const utilityCommands = {
console.log(chalk.cyan(`\n🚀 Running Benchmarks (${iterations} iterations)\n`))
const results: any = {
const results: {
operations: Record<string, BenchmarkOperationStats>
summary: { totalOperations?: number; averageOpsPerSec?: string }
} = {
operations: {},
summary: {}
}
@ -253,7 +265,7 @@ export const utilityCommands = {
}
// Calculate summary
const totalOps: number = (Object.values(results.operations) as any[]).reduce((sum: number, op: any) =>
const totalOps: number = Object.values(results.operations).reduce((sum: number, op) =>
sum + parseFloat(op.ops), 0)
results.summary = {
@ -277,7 +289,7 @@ export const utilityCommands = {
style: { head: [], border: [] }
})
Object.entries(results.operations).forEach(([op, stats]: [string, any]) => {
Object.entries(results.operations).forEach(([op, stats]) => {
table.push([
op,
stats.avg,

29
src/cli/commands/vfs.ts
View file

@ -9,6 +9,7 @@ import ora from 'ora'
import Table from 'cli-table3'
import { readFileSync, writeFileSync } from 'node:fs'
import { Brainy } from '../../brainy.js'
import type { SearchResult, VFSDirent } from '../../vfs/types.js'
interface VFSOptions {
verbose?: boolean
@ -54,8 +55,9 @@ export const vfsCommands = {
try {
const brain = await getBrainy() // Await async getBrainy
// VFS auto-initialized, no need for vfs.init()
// --encoding arrives as a raw CLI string (Node validates it at use time).
const buffer = await brain.vfs.readFile(path, {
encoding: options.encoding as any
encoding: options.encoding as BufferEncoding | undefined
})
spinner.succeed('File read successfully')
@ -103,8 +105,9 @@ export const vfsCommands = {
process.exit(1)
}
// --encoding arrives as a raw CLI string (Node validates it at use time).
await brain.vfs.writeFile(path, data, {
encoding: options.encoding as any
encoding: options.encoding as BufferEncoding | undefined
})
spinner.succeed('File written successfully')
@ -135,7 +138,9 @@ export const vfsCommands = {
try {
const brain = await getBrainy()
const entries = await brain.vfs.readdir(path, { withFileTypes: true })
// withFileTypes: true selects the VFSDirent[] branch of readdir's
// `string[] | VFSDirent[]` union return type.
const entries = await brain.vfs.readdir(path, { withFileTypes: true }) as VFSDirent[]
spinner.succeed(`Found ${Array.isArray(entries) ? entries.length : 0} items`)
@ -154,13 +159,14 @@ export const vfsCommands = {
style: { head: [], border: [] }
})
for (const entry of entries as any[]) {
for (const entry of entries) {
if (!options.all && entry.name.startsWith('.')) continue
const isDirectory = entry.type === 'directory'
const stat = await brain.vfs.stat(`${path}/${entry.name}`)
table.push([
entry.isDirectory() ? chalk.blue('DIR') : 'FILE',
entry.isDirectory() ? '-' : formatBytes(stat.size),
isDirectory ? chalk.blue('DIR') : 'FILE',
isDirectory ? '-' : formatBytes(stat.size),
formatDate(stat.mtime),
entry.name
])
@ -170,10 +176,10 @@ export const vfsCommands = {
} else {
// Simple format
console.log()
for (const entry of entries as any[]) {
for (const entry of entries) {
if (!options.all && entry.name.startsWith('.')) continue
if (entry.isDirectory()) {
if (entry.type === 'directory') {
console.log(chalk.blue(entry.name + '/'))
} else {
console.log(entry.name)
@ -322,8 +328,11 @@ export const vfsCommands = {
if (result.score) {
console.log(chalk.dim(` Score: ${(result.score * 100).toFixed(1)}%`))
}
if ((result as any).excerpt) {
console.log(chalk.dim(` ${(result as any).excerpt}`))
// `excerpt` isn't part of the VFS SearchResult contract — search()
// doesn't attach it today; read defensively.
const excerpt = (result as SearchResult & { excerpt?: string }).excerpt
if (excerpt) {
console.log(chalk.dim(` ${excerpt}`))
}
console.log()
})

10
src/cli/index.ts
View file

@ -46,7 +46,7 @@ ${chalk.cyan('Examples:')}
$ brainy add "React is a JavaScript library"
$ brainy find "JavaScript frameworks"
$ brainy update <id> --content "Updated content"
$ brainy delete <id> ${chalk.dim('# Requires confirmation')}
$ brainy remove <id> ${chalk.dim('# Requires confirmation')}
$ brainy search "react" --type Component --where '{"tested":true}'
${chalk.dim('# Neural API')}
@ -143,10 +143,10 @@ program
.action(coreCommands.update)
program
.command('delete [id]')
.description('Delete an entity (interactive if no ID, requires confirmation)')
.command('remove [id]')
.description('Remove an entity (interactive if no ID, requires confirmation)')
.option('-f, --force', 'Skip confirmation prompt')
.action(coreCommands.deleteEntity)
.action(coreCommands.removeEntity)
program
.command('unrelate [id]')
@ -472,7 +472,7 @@ program
.addCommand(
new Command('batch-delete')
.argument('<file>', 'File containing entity IDs (one per line)')
.description('Delete a list of entities through the public delete path (with retry)')
.description('Delete a list of entities through the public remove() path (with retry)')
.option('--max-retries <n>', 'Maximum retry attempts per ID', '3')
.option('--continue-on-error', 'Continue past IDs that still fail after retries')
.action((file, options) => {

11
src/config/extensibleConfig.ts
View file

@ -126,7 +126,10 @@ class ConfigurationRegistry {
if (await provider.detect()) {
const config = await provider.getConfig()
return {
type: provider.type as any,
// Registered providers deliberately extend the built-in storage
// vocabulary ('redis', 'mongodb', ...) — the registry contract is
// open, so narrow the provider's free-form type string here.
type: provider.type as StorageConfigResult['type'],
config,
reason: `Auto-detected ${provider.name}`,
autoSelected: true
@ -313,7 +316,11 @@ export function registerPresetAugmentation(name: string, config: PresetConfig):
* Example preset for Redis-based caching service
*/
export const redisCachePreset: PresetConfig = {
storage: 'redis' as any, // Extended storage type
// 'redis' is an extended storage type outside the built-in StorageOption
// enum — extension presets resolve it at runtime through the provider
// registry, so this is a genuine typed boundary (hence the double
// assertion: the literal has no overlap with the enum).
storage: 'redis' as unknown as StorageOption,
model: ModelPrecision.Q8,
features: ['core', 'cache', 'search'],
distributed: true,

16
src/config/zeroConfig.ts
View file

@ -149,7 +149,9 @@ export async function processZeroConfig(input?: string | BrainyZeroConfig): Prom
// Handle string shorthand (preset name)
if (typeof input === 'string') {
if (input in PRESETS) {
config = { mode: input as any }
// The `in` guard above proved the string is a preset key, which is
// exactly the BrainyZeroConfig mode union.
config = { mode: input as keyof typeof PRESETS }
} else {
throw new Error(`Unknown preset: ${input}. Valid presets: ${Object.keys(PRESETS).join(', ')}`)
}
@ -263,7 +265,17 @@ export async function processZeroConfig(input?: string | BrainyZeroConfig): Prom
// Apply distributed preset settings if applicable
if (config.mode === 'writer' || config.mode === 'reader') {
const presetSettings = PRESETS[config.mode] as any // Cast to any since we know these presets have additional properties
// The writer/reader presets each carry their own subset of the
// distributed flags beyond the base preset shape — widen to the union of
// those flags so both branches read uniformly.
const presetSettings: {
distributed: boolean
role: 'writer' | 'reader'
readOnly?: boolean
writeOnly?: boolean
allowDirectReads?: boolean
lazyLoadInReadOnlyMode?: boolean
} = PRESETS[config.mode]
// Apply distributed-specific settings
finalConfig.distributed = presetSettings.distributed

4
src/coreTypes.ts
View file

@ -389,7 +389,7 @@ export interface HNSWVerbWithMetadata {
// `ReportsTo` relationship might have subtype 'direct' vs 'dotted-line'; a `RelatedTo`
// edge might carry 'spouse' / 'sibling' / 'colleague'). Flat string (no hierarchy) —
// consumers decide the vocabulary. Indexed and rolled up into per-VerbType statistics
// so it's queryable (`getRelations({ verb, subtype })`) and aggregable
// so it's queryable (`related({ verb, subtype })`) and aggregable
// (`groupBy:['subtype']`) on the standard-field fast path, never falling through to
// the metadata fallback.
subtype?: string
@ -417,7 +417,7 @@ export interface HNSWVerbWithMetadata {
* Verb representing a relationship between nouns
* Stored separately from the HNSW vector index for lightweight performance.
* This is the canonical verb shape — Brainy's internal graph engine, the public
* `relate()` / `getRelations()` surface, and storage adapters all speak it.
* `relate()` / `related()` surface, and storage adapters all speak it.
*/
export interface GraphVerb {
id: string // Unique identifier — always a UUID (8.0 contract: brainy generates every verb id, so providers may key verb-int interning on the raw UUID bytes)

140
src/critical/model-guardian.ts
View file

@ -1,140 +0,0 @@
/**
* MODEL GUARDIAN - CRITICAL PATH
*
* THIS IS THE MOST CRITICAL COMPONENT OF BRAINY
* Without the exact model, users CANNOT access their data
*
* Requirements:
* 1. Model MUST be all-MiniLM-L6-v2-q8 (bundled in package)
* 2. Model MUST be available at runtime (embedded in npm package)
* 3. Model MUST produce consistent 384-dim embeddings
* 4. System MUST fail fast if model unavailable in production
*/
import { WASMEmbeddingEngine } from '../embeddings/wasm/index.js'
// CRITICAL: These values MUST NEVER CHANGE
const CRITICAL_MODEL_CONFIG = {
modelName: 'all-MiniLM-L6-v2-q8',
embeddingDimensions: 384,
// Model is bundled in package - no external downloads needed
bundled: true
}
export class ModelGuardian {
private static instance: ModelGuardian
private isVerified = false
private lastVerification: Date | null = null
private constructor() {
// Model is bundled - no path detection needed
}
static getInstance(): ModelGuardian {
if (!ModelGuardian.instance) {
ModelGuardian.instance = new ModelGuardian()
}
return ModelGuardian.instance
}
/**
* CRITICAL: Verify model availability and integrity
* This MUST be called before any embedding operations
*/
async ensureCriticalModel(): Promise<void> {
// Check if already verified in this session
if (this.isVerified && this.lastVerification) {
const hoursSinceVerification =
(Date.now() - this.lastVerification.getTime()) / (1000 * 60 * 60)
if (hoursSinceVerification < 24) {
return
}
}
// Verify the bundled WASM model works
const modelWorks = await this.verifyBundledModel()
if (modelWorks) {
this.isVerified = true
this.lastVerification = new Date()
return
}
// CRITICAL FAILURE
throw new Error(
'🚨 CRITICAL FAILURE: Bundled transformer model not working!\n' +
'The model is REQUIRED for Brainy to function.\n' +
'Users CANNOT access their data without it.\n' +
'This indicates a package installation issue.'
)
}
/**
* Verify the bundled WASM model works correctly
*/
private async verifyBundledModel(): Promise<boolean> {
try {
const engine = WASMEmbeddingEngine.getInstance()
// Initialize the engine (loads bundled model)
await engine.initialize()
// Test embedding generation
const testEmbedding = await engine.embed('test verification')
// Verify dimensions
if (testEmbedding.length !== CRITICAL_MODEL_CONFIG.embeddingDimensions) {
console.error(
`❌ CRITICAL: Model dimension mismatch!\n` +
`Expected: ${CRITICAL_MODEL_CONFIG.embeddingDimensions}\n` +
`Got: ${testEmbedding.length}`
)
return false
}
// Verify normalization (should be unit length)
const norm = Math.sqrt(testEmbedding.reduce((sum, v) => sum + v * v, 0))
if (Math.abs(norm - 1.0) > 0.01) {
console.error(`❌ CRITICAL: Embeddings not normalized! Norm: ${norm}`)
return false
}
return true
} catch (error) {
console.error('❌ Model verification failed:', error)
return false
}
}
/**
* Get model status for diagnostics
*/
async getStatus(): Promise<{
verified: boolean
lastVerification: Date | null
modelName: string
dimensions: number
bundled: boolean
}> {
return {
verified: this.isVerified,
lastVerification: this.lastVerification,
modelName: CRITICAL_MODEL_CONFIG.modelName,
dimensions: CRITICAL_MODEL_CONFIG.embeddingDimensions,
bundled: CRITICAL_MODEL_CONFIG.bundled
}
}
/**
* Force re-verification (for testing)
*/
async forceReverify(): Promise<void> {
this.isVerified = false
this.lastVerification = null
await this.ensureCriticalModel()
}
}
// Export singleton instance
export const modelGuardian = ModelGuardian.getInstance()

193
src/data/expandedKeywordDictionary.ts
View file

@ -1,193 +0,0 @@
/**
* Expanded Keyword Dictionary for Semantic Type Inference
*
* Comprehensive keyword-to-type mappings including:
* - Canonical keywords (primary terms)
* - Synonyms (alternative terms with slightly lower confidence)
* - Domain-specific variations
* - Common abbreviations
*
* Expanded from 767 → 1500+ keywords for better semantic coverage
*/
import { NounType } from '../types/graphTypes.js'
export interface KeywordDefinition {
keyword: string
type: NounType
confidence: number // 0.7-0.95 (higher = more canonical)
isCanonical: boolean // True for primary terms, false for synonyms
}
/**
* Expanded keyword dictionary (1500+ keywords for 31 NounTypes)
*/
export const EXPANDED_KEYWORD_DICTIONARY: KeywordDefinition[] = [
// ========== Person - Medical Professions ==========
// Canonical
{ keyword: 'doctor', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'physician', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'surgeon', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'nurse', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'cardiologist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'oncologist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'neurologist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'psychiatrist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'psychologist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'radiologist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'pathologist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'anesthesiologist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'dermatologist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'pediatrician', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'obstetrician', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'gynecologist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'ophthalmologist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'dentist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'orthodontist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'pharmacist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'paramedic', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'therapist', type: NounType.Person, confidence: 0.90, isCanonical: true },
// Synonyms
{ keyword: 'medic', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'practitioner', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'clinician', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'medical professional', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'healthcare worker', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'medical doctor', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'registered nurse', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'emt', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'counselor', type: NounType.Person, confidence: 0.85, isCanonical: false },
// ========== Person - Engineering & Tech ==========
// Canonical
{ keyword: 'engineer', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'developer', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'programmer', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'architect', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'designer', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'technician', type: NounType.Person, confidence: 0.90, isCanonical: true },
// Synonyms
{ keyword: 'coder', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'software engineer', type: NounType.Person, confidence: 0.95, isCanonical: false },
{ keyword: 'software developer', type: NounType.Person, confidence: 0.95, isCanonical: false },
{ keyword: 'web developer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'frontend developer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'backend developer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'full stack developer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'devops engineer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'data engineer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'ml engineer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'machine learning engineer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'data scientist', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'ux designer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'ui designer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'graphic designer', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'systems architect', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'solutions architect', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'tech lead', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'techie', type: NounType.Person, confidence: 0.80, isCanonical: false },
// ========== Person - Management & Leadership ==========
// Canonical
{ keyword: 'manager', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'director', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'executive', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'leader', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'ceo', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'cto', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'cfo', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'coo', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'president', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'founder', type: NounType.Person, confidence: 0.95, isCanonical: true },
// Synonyms
{ keyword: 'supervisor', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'coordinator', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'vp', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'vice president', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'owner', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'product manager', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'project manager', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'engineering manager', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'team lead', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'chief executive officer', type: NounType.Person, confidence: 0.95, isCanonical: false },
{ keyword: 'chief technology officer', type: NounType.Person, confidence: 0.95, isCanonical: false },
{ keyword: 'chief financial officer', type: NounType.Person, confidence: 0.95, isCanonical: false },
// ========== Person - Professional Services ==========
// Canonical
{ keyword: 'analyst', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'consultant', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'specialist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'expert', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'professional', type: NounType.Person, confidence: 0.85, isCanonical: true },
{ keyword: 'lawyer', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'attorney', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'accountant', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'auditor', type: NounType.Person, confidence: 0.90, isCanonical: true },
// Synonyms
{ keyword: 'advisor', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'counselor', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'paralegal', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'legal counsel', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'business analyst', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'financial analyst', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'data analyst', type: NounType.Person, confidence: 0.90, isCanonical: false },
// ========== Person - Education & Research ==========
// Canonical
{ keyword: 'teacher', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'professor', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'researcher', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'scientist', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'student', type: NounType.Person, confidence: 0.95, isCanonical: true },
// Synonyms
{ keyword: 'instructor', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'educator', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'tutor', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'scholar', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'academic', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'pupil', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'learner', type: NounType.Person, confidence: 0.80, isCanonical: false },
{ keyword: 'trainee', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'intern', type: NounType.Person, confidence: 0.85, isCanonical: false },
// ========== Person - Creative Professions ==========
// Canonical
{ keyword: 'artist', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'musician', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'writer', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'author', type: NounType.Person, confidence: 0.90, isCanonical: true },
// Synonyms
{ keyword: 'painter', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'sculptor', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'performer', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'journalist', type: NounType.Person, confidence: 0.90, isCanonical: false },
{ keyword: 'editor', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'reporter', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'content creator', type: NounType.Person, confidence: 0.80, isCanonical: false },
{ keyword: 'blogger', type: NounType.Person, confidence: 0.80, isCanonical: false },
// ========== Person - General ==========
{ keyword: 'person', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'people', type: NounType.Person, confidence: 0.95, isCanonical: true },
{ keyword: 'individual', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'human', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'employee', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'worker', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'staff', type: NounType.Person, confidence: 0.90, isCanonical: true },
{ keyword: 'personnel', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'member', type: NounType.Person, confidence: 0.85, isCanonical: false },
{ keyword: 'team', type: NounType.Person, confidence: 0.80, isCanonical: false },
// Continuing with the rest... (this is getting long, so I'll create a comprehensive version)
// Let me structure this better by importing from the existing typeInference and expanding it
]
// Note: This file will be completed with all 1500+ keywords in the actual implementation
// For now, this shows the structure and approach

50
src/db/db.ts
View file

@ -52,7 +52,7 @@ import type {
Entity,
FindParams,
GetOptions,
GetRelationsParams,
RelatedParams,
Relation,
Result
} from '../types/brainy.types.js'
@ -61,6 +61,7 @@ import {
splitVerbMetadataRecord
} from '../types/reservedFields.js'
import { v4 as uuidv4 } from '../universal/uuid.js'
import { EntityNotFoundError } from '../errors/notFound.js'
import { SpeculativeOverlayError } from './errors.js'
import type { GenerationStore } from './generationStore.js'
import type { ChangedIds, TransactReceipt, TxOperation } from './types.js'
@ -76,8 +77,8 @@ import { entityMatchesFind, resolveEntityField, UnsupportedWhereOperatorError }
export interface HistoricalQueryHandle<T = any> {
/** Full `find()` surface (vector/semantic/graph/cursor/aggregate) at the pinned generation. */
find(query: string | FindParams<T>): Promise<Result<T>[]>
/** Full `getRelations()` surface (including cursor pagination) at the pinned generation. */
getRelations(paramsOrId?: string | GetRelationsParams): Promise<Relation<T>[]>
/** Full `related()` surface (including cursor pagination) at the pinned generation. */
related(paramsOrId?: string | RelatedParams): Promise<Relation<T>[]>
/** Free the materialized indexes (closes the ephemeral reader). Idempotent. */
close(): Promise<void>
}
@ -108,8 +109,8 @@ export interface DbHost<T = any> {
get(id: string, options?: GetOptions): Promise<Entity<T> | null>
/** Live `brain.find()` (current-generation fast path). */
find(query: string | FindParams<T>): Promise<Result<T>[]>
/** Live `brain.getRelations()` (current-generation fast path). */
getRelations(paramsOrId?: string | GetRelationsParams): Promise<Relation<T>[]>
/** Live `brain.related()` (current-generation fast path). */
related(paramsOrId?: string | RelatedParams): Promise<Relation<T>[]>
/** Materialize an entity from a generation record's raw stored objects. */
entityFromRecord(
id: string,
@ -410,7 +411,7 @@ export class Db<T = any> {
/**
* @description Relationships **as of this view's generation** — the `Db`
* counterpart of `brain.getRelations()` (same string-id shorthand and
* counterpart of `brain.related()` (same string-id shorthand and
* filter surface). Relations whose edges changed after the pin are
* resolved from generation records; overlay relations (speculative
* `relate`/`unrelate`) and cascade tombstones (relations touching a
@ -420,18 +421,18 @@ export class Db<T = any> {
* speculative overlay it throws {@link SpeculativeOverlayError}.
*
* @param paramsOrId - An entity id (shorthand for `{ from: id }`) or a
* `GetRelationsParams` filter object.
* `RelatedParams` filter object.
* @returns Relations as of this generation.
*/
async related(paramsOrId?: string | GetRelationsParams): Promise<Relation<T>[]> {
async related(paramsOrId?: string | RelatedParams): Promise<Relation<T>[]> {
this.assertUsable('related')
const params: GetRelationsParams =
const params: RelatedParams =
typeof paramsOrId === 'string' ? { from: paramsOrId } : (paramsOrId ?? {})
const historical = this.isHistorical()
if (!historical && !this.overlay) {
return this.host.getRelations(paramsOrId)
return this.host.related(paramsOrId)
}
if (params.cursor !== undefined) {
@ -439,7 +440,7 @@ export class Db<T = any> {
throw new SpeculativeOverlayError('cursor pagination on related()', this.gen)
}
const materialized = await this.materialize()
return materialized.getRelations(params)
return materialized.related(params)
}
const limit = params.limit ?? 100
@ -451,7 +452,7 @@ export class Db<T = any> {
const overlayVerbs = this.overlay?.verbs ?? new Map<string, Relation<T> | null>()
const excluded = new Set<string>([...changedVerbs, ...overlayVerbs.keys()])
const base = await this.host.getRelations({
const base = await this.host.related({
...params,
limit: limit + offset + excluded.size,
offset: 0
@ -480,7 +481,7 @@ export class Db<T = any> {
// Cascade semantics for speculative deletes: a relation whose endpoint
// is tombstoned in the overlay is gone in this view, exactly as
// brain.delete() cascades on the durable path.
// brain.remove() cascades on the durable path.
if (this.overlay) {
const tombstoned = new Set<string>()
for (const [id, entity] of this.overlay.nouns) {
@ -524,8 +525,8 @@ export class Db<T = any> {
*
* @param ops - The same declarative operations `brain.transact()` accepts.
* @returns A new speculative `Db` over this view.
* @throws Error when an operation references a missing entity/relation,
* mirroring the durable path's planning errors.
* @throws EntityNotFoundError when an operation references a missing
* entity, mirroring the durable path's planning errors.
*/
async with(ops: TxOperation<T>[]): Promise<Db<T>> {
this.assertUsable('with')
@ -587,7 +588,10 @@ export class Db<T = any> {
case 'update': {
const base = await speculativeGet(op.id)
if (!base) {
throw new Error(`with(): entity ${op.id} not found at generation ${this.gen}`)
throw new EntityNotFoundError(
op.id,
`with(): entity ${op.id} not found at generation ${this.gen}`
)
}
// Same reserved-field normalization as the committed update path.
const { reserved, custom } = splitNounMetadataRecord(
@ -631,8 +635,12 @@ export class Db<T = any> {
case 'relate': {
const from = await speculativeGet(op.from)
const to = await speculativeGet(op.to)
if (!from) throw new Error(`with(): source entity ${op.from} not found`)
if (!to) throw new Error(`with(): target entity ${op.to} not found`)
if (!from) {
throw new EntityNotFoundError(op.from, `with(): source entity ${op.from} not found`)
}
if (!to) {
throw new EntityNotFoundError(op.to, `with(): target entity ${op.to} not found`)
}
// Dedupe against the view (overlay first, then the edges visible
// at this generation via the record layer) — mirror of relate().
@ -934,11 +942,11 @@ function sortResultsBy<T>(results: Result<T>[], orderBy: string, order: 'asc' |
}
/**
* @description Filter one relation against `GetRelationsParams`, mirroring
* the storage-level filter `brain.getRelations()` builds (`from` → sourceId,
* @description Filter one relation against `RelatedParams`, mirroring
* the storage-level filter `brain.related()` builds (`from` → sourceId,
* `to` → targetId, type/subtype set membership, service equality).
*/
function relationMatchesParams<T>(relation: Relation<T>, params: GetRelationsParams): boolean {
function relationMatchesParams<T>(relation: Relation<T>, params: RelatedParams): boolean {
if (params.from !== undefined && relation.from !== params.from) return false
if (params.to !== undefined && relation.to !== params.to) return false
if (params.type !== undefined) {

4
src/db/types.ts
View file

@ -53,14 +53,14 @@ export interface TxUpdateOperation<T = any> extends UpdateParams<T> {
}
/**
* @description Delete an entity (and, exactly like `brain.delete()`, every
* @description Remove an entity (and, exactly like `brain.remove()`, every
* relationship where it is source or target — the cascade is part of the
* same atomic batch).
*/
export interface TxRemoveOperation {
/** Discriminator. */
op: 'remove'
/** Id of the entity to delete. */
/** Id of the entity to remove. */
id: string
}

17
src/distributed/coordinator.ts
View file

@ -123,12 +123,13 @@ export class DistributedCoordinator extends EventEmitter {
*/
private setupNetworkHandlers(): void {
if (!this.networkTransport) return
// Register message handlers directly on the messageHandlers map
const handlers = (this.networkTransport as any).messageHandlers as Map<string, (msg: NetworkMessage) => Promise<any>>
// Register handlers through the transport's public onMessage(), which
// writes to the same messageHandlers map this code used to reach into.
const transport = this.networkTransport
// Handle vote requests
handlers.set('requestVote', async (msg: NetworkMessage) => {
transport.onMessage('requestVote', async (msg: NetworkMessage) => {
const response = await this.handleVoteRequest(msg)
// Send response back
if (this.networkTransport) {
@ -138,12 +139,12 @@ export class DistributedCoordinator extends EventEmitter {
})
// Handle vote responses
handlers.set('voteResponse', async (msg: NetworkMessage) => {
transport.onMessage('voteResponse', async (msg: NetworkMessage) => {
this.handleVoteResponse(msg)
})
// Handle heartbeats/append entries
handlers.set('appendEntries', async (msg: NetworkMessage) => {
transport.onMessage('appendEntries', async (msg: NetworkMessage) => {
const response = await this.handleAppendEntries(msg)
// Send response back
if (this.networkTransport) {
@ -153,7 +154,7 @@ export class DistributedCoordinator extends EventEmitter {
})
// Handle append responses
handlers.set('appendResponse', async (msg: NetworkMessage) => {
transport.onMessage('appendResponse', async (msg: NetworkMessage) => {
this.handleAppendResponse(msg)
})
}

7
src/distributed/networkTransport.ts
View file

@ -75,7 +75,12 @@ export class NetworkTransport extends EventEmitter {
if (this.isRunning) return
const ws = await import('ws')
WebSocketServer = (ws as any).WebSocketServer || (ws as any).Server || ws.default
// The ws package has exported its server class as `WebSocketServer`
// (v8+), `Server` (older releases) and via the default export — and the
// available type declarations disagree on which of those are value
// exports, so probe all three through an optional-keyed module view.
const wsModule = ws as typeof ws & { WebSocketServer?: unknown; Server?: unknown }
WebSocketServer = wsModule.WebSocketServer || wsModule.Server || ws.default
await this.startHTTPServer()
await this.startWebSocketServer()

8
src/distributed/storageDiscovery.ts
View file

@ -6,7 +6,7 @@
import { EventEmitter } from 'node:events'
import * as os from 'node:os'
import { StorageAdapter } from '../coreTypes.js'
import { NounMetadata, StorageAdapter } from '../coreTypes.js'
export interface NodeInfo {
id: string
@ -357,7 +357,11 @@ export class StorageDiscovery extends EventEmitter {
*/
private async loadNodeRegistry(): Promise<string[]> {
try {
const registry = await this.storage.getMetadata(`${this.CLUSTER_PATH}/registry.json`) as any
// The registry document is written by updateNodeRegistry() below with a
// `nodes` string-ID array on top of the base NounMetadata shape.
const registry = await this.storage.getMetadata(
`${this.CLUSTER_PATH}/registry.json`
) as (NounMetadata & { nodes?: string[] }) | null
return registry?.nodes || []
} catch (err) {
return []

17
src/embeddings/EmbeddingManager.ts
View file

@ -15,6 +15,17 @@
import { Vector, EmbeddingFunction } from '../coreTypes.js'
import { WASMEmbeddingEngine } from './wasm/index.js'
declare global {
/**
* Unit-test escape hatch set by the test harness (tests/setup-unit.ts):
* when truthy, EmbeddingManager serves deterministic mock embeddings
* instead of loading the real model. Guarded against production use in
* init()/embed()/embedBatch(). Ambient `var` is required here — `let`/
* `const` in `declare global` do not attach to `globalThis`.
*/
var __BRAINY_UNIT_TEST__: boolean | undefined
}
// Types
export type ModelPrecision = 'q8' | 'fp32'
@ -68,7 +79,7 @@ export class EmbeddingManager {
// In unit test mode, skip real model initialization
const isTestMode =
process.env.BRAINY_UNIT_TEST === 'true' ||
(globalThis as any).__BRAINY_UNIT_TEST__
globalThis.__BRAINY_UNIT_TEST__
if (isTestMode) {
// Production safeguard
@ -142,7 +153,7 @@ export class EmbeddingManager {
// Check for unit test environment
const isTestMode =
process.env.BRAINY_UNIT_TEST === 'true' ||
(globalThis as any).__BRAINY_UNIT_TEST__
globalThis.__BRAINY_UNIT_TEST__
if (isTestMode) {
if (process.env.NODE_ENV === 'production') {
@ -224,7 +235,7 @@ export class EmbeddingManager {
const isTestMode =
process.env.BRAINY_UNIT_TEST === 'true' ||
(globalThis as any).__BRAINY_UNIT_TEST__
globalThis.__BRAINY_UNIT_TEST__
if (isTestMode) {
if (process.env.NODE_ENV === 'production') {

79
src/errors/notFound.ts Normal file
View file

@ -0,0 +1,79 @@
/**
* @module errors/notFound
* @description Named not-found errors for Brainy's public contract.
*
* Operations that require an existing record — `update()`, `relate()`,
* `updateRelation()`, `similar({ to: id })`, `transact()` planning, and
* speculative `db.with()` planning — throw these instead of a generic
* `Error`, so callers can branch with `instanceof` and read the missing
* `id` programmatically instead of parsing message strings:
*
* - {@link EntityNotFoundError} — a referenced entity (noun) does not exist
* in the store (or, for `db.with()`, is not visible at the pinned
* generation).
* - {@link RelationNotFoundError} — a referenced relationship (verb) does
* not exist.
*
* Both are exported from the package root (`@soulcraft/brainy`).
*/
/**
* @description Thrown when an operation references an entity id that does
* not exist. Carries the missing {@link EntityNotFoundError.id} so callers
* can recover (re-create, skip, or surface it) without string matching.
*
* @example
* try {
* await brain.update({ id, metadata: { status: 'active' } })
* } catch (err) {
* if (err instanceof EntityNotFoundError) {
* console.log(`missing entity: ${err.id}`)
* }
* }
*/
export class EntityNotFoundError extends Error {
/** The entity id that could not be found. */
public readonly id: string
/**
* @param id - The entity id that could not be found.
* @param message - Optional message override for call sites that add
* context (e.g. source/target role, pinned generation). Defaults to
* `Entity <id> not found`.
*/
constructor(id: string, message?: string) {
super(message ?? `Entity ${id} not found`)
this.name = 'EntityNotFoundError'
this.id = id
}
}
/**
* @description Thrown when an operation references a relationship id that
* does not exist. Carries the missing {@link RelationNotFoundError.id} so
* callers can recover without string matching.
*
* @example
* try {
* await brain.updateRelation({ id: relId, subtype: 'dotted-line' })
* } catch (err) {
* if (err instanceof RelationNotFoundError) {
* console.log(`missing relation: ${err.id}`)
* }
* }
*/
export class RelationNotFoundError extends Error {
/** The relationship id that could not be found. */
public readonly id: string
/**
* @param id - The relationship id that could not be found.
* @param message - Optional message override for call sites that add
* context. Defaults to `Relation <id> not found`.
*/
constructor(id: string, message?: string) {
super(message ?? `Relation ${id} not found`)
this.name = 'RelationNotFoundError'
this.id = id
}
}

6
src/graph/graphAdjacencyIndex.ts
View file

@ -570,9 +570,9 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
for (const [verbId, verb] of loadedVerbs.entries()) {
const cacheKey = `graph:verb:${verbId}`
// Cache the loaded verb with metadata
// Note: HNSWVerbWithMetadata is compatible with GraphVerb (both interfaces)
this.unifiedCache.set(cacheKey, verb as any, 'other', 128, 50) // 128 bytes estimated size, 50ms rebuild cost
results.set(verbId, verb as any)
// Note: HNSWVerbWithMetadata is structurally assignable to GraphVerb
this.unifiedCache.set(cacheKey, verb, 'other', 128, 50) // 128 bytes estimated size, 50ms rebuild cost
results.set(verbId, verb)
}
}

32
src/graph/lsm/LSMTree.ts
View file

@ -145,6 +145,26 @@ class MemTable {
/**
* Manifest - Tracks all SSTables and their levels
*/
/**
* Persisted manifest payload as written by `saveManifest()` (the `data` field
* of the manifest metadata record, after a JSON round-trip).
*/
type PersistedManifestData = {
sstables?: Record<string, number>
lastCompaction?: number
totalRelationships?: number
}
/**
* Persisted SSTable payload as written by `flushMemTable()`/`compact()` (the
* `data` field of an SSTable metadata record): serialized bytes stored as a
* plain number array so they survive JSON round-trips.
*/
type PersistedSSTableData = {
type: string
data: number[]
}
interface Manifest {
/**
* Map of SSTable ID to level
@ -512,10 +532,12 @@ export class LSMTree {
const metadata = await this.storage.getMetadata(`${this.config.storagePrefix}-manifest`)
if (metadata && metadata.data) {
const data = metadata.data as any
// Storage boundary: `data` is the JSON manifest payload written by
// saveManifest(); re-typed from the metadata channel's `unknown`.
const data = metadata.data as PersistedManifestData
this.manifest.sstables = new Map(Object.entries(data.sstables || {}))
this.manifest.lastCompaction = (data.lastCompaction as number) || Date.now()
this.manifest.totalRelationships = (data.totalRelationships as number) || 0
this.manifest.lastCompaction = data.lastCompaction || Date.now()
this.manifest.totalRelationships = data.totalRelationships || 0
// Load SSTables from storage
await this.loadSSTables()
@ -538,7 +560,9 @@ export class LSMTree {
const metadata = await this.storage.getMetadata(storageKey)
if (metadata && metadata.data) {
const data = metadata.data as any
// Storage boundary: `data` is the JSON SSTable payload written by
// flushMemTable()/compact(); re-typed from `unknown`.
const data = metadata.data as PersistedSSTableData
if (data.type === 'lsm-sstable') {
// Convert number[] back to Uint8Array
const uint8Data = new Uint8Array(data.data)

14
src/hnsw/hnswIndex.ts
View file

@ -1228,7 +1228,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
this.clear()
// Step 2: Load system data (entry point, max level)
const systemData = await (this.storage as any).getHNSWSystem()
const systemData = await this.storage.getHNSWSystem()
if (systemData) {
this.entryPointId = systemData.entryPointId
this.maxLevel = systemData.maxLevel
@ -1270,13 +1270,9 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
{
prodLog.info(`HNSW: Load all nodes at once (${storageType})`)
const result: {
items: HNSWNoun[]
totalCount?: number
hasMore: boolean
nextCursor?: string
} = await (this.storage as any).getNounsWithPagination({
limit: 10000000 // Effectively unlimited for local development
const result = await this.storage.getNounsWithPagination({
limit: 10000000, // Effectively unlimited for local development
offset: 0
})
totalCount = result.totalCount || result.items.length
@ -1285,7 +1281,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
for (const nounData of result.items) {
try {
// Load HNSW graph data for this entity
const hnswData = await (this.storage as any).getVectorIndexData(nounData.id)
const hnswData = await this.storage.getVectorIndexData(nounData.id)
if (!hnswData) {
// No HNSW data - skip (might be entity added before persistence)

13
src/import/BackgroundDeduplicator.ts
View file

@ -123,7 +123,7 @@ export class BackgroundDeduplicator {
try {
// Get all entities from this import using brain.find()
const results = await this.brain.find({
where: { importId } as any,
where: { importId },
limit: 100000 // Large limit to get all entities from import
})
@ -298,7 +298,7 @@ export class BackgroundDeduplicator {
return this.brain.find({
query,
limit: 5,
where: { type: entity.type } as any // Type-aware search
where: { type: entity.type } // Type-aware search
})
})
@ -318,8 +318,11 @@ export class BackgroundDeduplicator {
// Check if not already merged
const stillExists = await this.brain.get(entity.id)
if (stillExists) {
// Cast match.entity to HNSWNounWithMetadata (it comes from brain.find results)
const matchEntity = match.entity as any as HNSWNounWithMetadata
// Typed boundary: bridge the public Entity<T> result shape from
// brain.find() to the storage-layer HNSWNounWithMetadata record
// (same structural bridge as the results.map above). The merge
// path only reads id/type/metadata, present in both shapes.
const matchEntity = match.entity as unknown as HNSWNounWithMetadata
await this.mergeEntities([entity, matchEntity], 'Similarity')
merged++
break // Only merge with first high-similarity match
@ -388,7 +391,7 @@ export class BackgroundDeduplicator {
for (const entity of entities) {
if (entity.id !== primary.id) {
try {
await this.brain.delete(entity.id)
await this.brain.remove(entity.id)
} catch (error) {
// Entity might already be deleted, continue
prodLog.debug(`[BackgroundDedup] Could not delete ${entity.id}:`, error)

6
src/import/EntityDeduplicator.ts
View file

@ -88,7 +88,7 @@ export class EntityDeduplicator {
const results = await this.brain.find({
query: searchText,
limit: 5,
where: opts.strictTypeMatching ? { type: candidate.type } as any : undefined
where: opts.strictTypeMatching ? { type: candidate.type } : undefined
})
// Check each result for potential duplicates
@ -227,7 +227,9 @@ export class EntityDeduplicator {
const entityId = await this.brain.add({
data: candidate.description || candidate.name,
type: candidate.type,
subtype: (candidate as any).subtype ?? 'imported',
subtype:
(candidate as EntityCandidate & { subtype?: string }).subtype ??
'imported',
confidence: candidate.confidence,
metadata: {
...candidate.metadata,

55
src/import/ImportCoordinator.ts
View file

@ -12,7 +12,7 @@
import { Brainy } from '../brainy.js'
import { FormatDetector, SupportedFormat } from './FormatDetector.js'
import { ImportHistory } from './ImportHistory.js'
import { ImportHistory, type ImportHistoryEntry } from './ImportHistory.js'
import { BackgroundDeduplicator } from './BackgroundDeduplicator.js'
import { SmartExcelImporter } from '../importers/SmartExcelImporter.js'
import { SmartPDFImporter } from '../importers/SmartPDFImporter.js'
@ -489,7 +489,13 @@ export class ImportCoordinator {
await this.history.recordImport(
importId,
{
type: normalizedSource.type === 'path' ? 'file' : normalizedSource.type as any,
// 'url' sources were already converted to 'buffer' by
// normalizeSource() → fetchUrl(), so only history-recordable
// source types remain here after the 'path' → 'file' mapping.
type:
normalizedSource.type === 'path'
? 'file'
: (normalizedSource.type as ImportHistoryEntry['source']['type']),
filename: normalizedSource.filename,
format: detection.format
},
@ -699,11 +705,25 @@ export class ImportCoordinator {
format: SupportedFormat,
options: ImportOptions
): Promise<any> {
// Check if IntelligentImportAugmentation already extracted data
if ((options as any)._intelligentImport && (options as any)._extractedData) {
const extractedData = (options as any)._extractedData
// Check if IntelligentImportAugmentation already extracted data.
// Typed boundary: the intelligent-import pre-processing path smuggles its
// results onto ImportOptions via these underscore-prefixed private fields —
// they are not part of the public ImportOptions contract.
const intelligentOptions = options as ImportOptions & {
_intelligentImport?: boolean
_extractedData?: Array<{
id?: string
name?: string
type?: string
description?: string
metadata?: Record<string, unknown>
}>
_metadata?: { intelligentImport?: Record<string, unknown> }
}
if (intelligentOptions._intelligentImport && intelligentOptions._extractedData) {
const extractedData = intelligentOptions._extractedData
// Convert extracted data to ExtractedRow format
const rows = extractedData.map((item: any) => ({
const rows = extractedData.map((item) => ({
entity: {
id: item.id || `entity-${Date.now()}-${Math.random()}`,
name: item.name || item.type || 'Unnamed',
@ -719,7 +739,7 @@ export class ImportCoordinator {
rows,
entities: extractedData,
relationships: [],
metadata: (options as any)._metadata?.intelligentImport || {},
metadata: intelligentOptions._metadata?.intelligentImport || {},
stats: {
byType: {},
byConfidence: {}
@ -811,7 +831,7 @@ export class ImportCoordinator {
entity: {
id: imageId,
name: imageName,
type: 'media' as any,
type: 'media',
description: '',
confidence: 1.0,
metadata: { subtype: 'image' }
@ -864,7 +884,18 @@ export class ImportCoordinator {
provenanceCount?: number
}> {
const entities: Array<{ id: string; name: string; type: NounType; vfsPath?: string; metadata?: Record<string, any> }> = []
const relationships: Array<{ id: string; from: string; to: string; type: VerbType }> = []
// Wider than the declared return type: confidence/weight/metadata are
// collected per relationship for the relateMany() batch below; callers of
// createGraphEntities() only see the narrower {id, from, to, type} rows.
const relationships: Array<{
id: string
from: string
to: string
type: VerbType
confidence?: number
weight?: number
metadata?: { evidence?: string; [key: string]: unknown }
}> = []
let mergedCount = 0
let newCount = 0
@ -1227,7 +1258,7 @@ export class ImportCoordinator {
...trackingContext.customMetadata
})
}
} as any)
})
} catch (error) {
// Skip relationship collection errors (entity might not exist, etc.)
continue
@ -1314,7 +1345,7 @@ export class ImportCoordinator {
verbType = this.inferRelationshipType(
sourceEntity.type,
targetEntity.type,
(rel as any).metadata?.evidence
rel.metadata?.evidence
)
}
@ -1323,7 +1354,7 @@ export class ImportCoordinator {
to: rel.to,
type: verbType, // Enhanced type
metadata: {
...((rel as any).metadata || {}),
...(rel.metadata || {}),
relationshipType: 'semantic', // Distinguish from VFS/provenance
inferredType: verbType !== rel.type, // Track if type was enhanced
originalType: rel.type

2
src/import/ImportHistory.ts
View file

@ -165,7 +165,7 @@ export class ImportHistory {
// Delete entities
for (const entityId of entry.entities) {
try {
await this.brain.delete(entityId)
await this.brain.remove(entityId)
result.entitiesDeleted++
} catch (error) {
result.errors.push(`Failed to delete entity ${entityId}: ${error instanceof Error ? error.message : String(error)}`)

6
src/importers/SmartCSVImporter.ts
View file

@ -164,7 +164,9 @@ export class SmartCSVImporter {
relatedColumn: 'related|see also|links|references',
csvDelimiter: undefined as string | undefined,
csvHeaders: true,
onProgress: () => {},
// Annotated with the full callback signature so the merged opts type has
// a single callable shape (the no-op default narrowed the union).
onProgress: (() => {}) as NonNullable<SmartCSVOptions['onProgress']>,
...options
}
@ -379,7 +381,7 @@ export class SmartCSVImporter {
throughput: Math.round(rowsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining),
phase: 'extracting'
} as any)
})
}
return {

18
src/importers/SmartExcelImporter.ts
View file

@ -161,15 +161,9 @@ export class SmartExcelImporter {
const opts = {
enableNeuralExtraction: true,
enableRelationshipInference: true,
// CONCEPT EXTRACTION PRODUCTION-READY:
// Type embeddings are now pre-computed at build time - zero runtime cost!
// All 42 noun types + 127 verb types instantly available
//
// Performance profile:
// - Type embeddings: INSTANT (pre-computed at build time, ~100KB in-memory)
// - Model loading: ~2-5 seconds (one-time, cached after first use)
// - Per-row extraction: ~50-200ms depending on definition length
// - 100 rows: ~5-20 seconds total (production ready)
// Type embeddings are pre-computed at build time (~100KB in-memory),
// so concept extraction costs no runtime embedding calls: model loading
// is a one-time ~2-5s, then ~50-200ms per row.
// - 1000 rows: ~50-200 seconds (disable if needed via enableConceptExtraction: false)
//
// Enabled by default for production use.
@ -179,7 +173,9 @@ export class SmartExcelImporter {
definitionColumn: 'definition|description|desc|details',
typeColumn: 'type|category|kind',
relatedColumn: 'related|see also|links',
onProgress: () => {},
// Annotated with the full callback signature so the merged opts type has
// a single callable shape (the no-op default narrowed the union).
onProgress: (() => {}) as NonNullable<SmartExcelOptions['onProgress']>,
...options
}
@ -489,7 +485,7 @@ export class SmartExcelImporter {
throughput: Math.round(rowsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining),
phase: 'extracting'
} as any)
})
}
// Group rows by sheet for VFS extraction

40
src/importers/SmartImportOrchestrator.ts
View file

@ -133,7 +133,10 @@ export class SmartImportOrchestrator {
const startTime = Date.now()
const result: SmartImportResult = {
success: false,
extraction: null as any,
// Typed boundary: populated in the extraction phase below. If extraction
// throws, the error path returns with this still null (pre-existing
// contract — consumers check `success`/`errors` before reading it).
extraction: null as unknown as SmartExcelResult,
entityIds: [],
relationshipIds: [],
stats: {
@ -193,7 +196,9 @@ export class SmartImportOrchestrator {
const entityId = await this.brain.add({
data: extracted.entity.description,
type: extracted.entity.type,
subtype: (extracted.entity as any).subtype ?? options.defaultSubtype ?? 'imported',
subtype:
(extracted.entity as typeof extracted.entity & { subtype?: string })
.subtype ?? options.defaultSubtype ?? 'imported',
confidence: extracted.entity.confidence, // reserved field — dedicated param, not metadata
metadata: {
...extracted.entity.metadata,
@ -280,7 +285,9 @@ export class SmartImportOrchestrator {
from: extracted.entity.id,
to: toEntityId,
type: rel.type,
subtype: (rel as any).subtype ?? options.defaultSubtype ?? 'imported',
subtype:
(rel as typeof rel & { subtype?: string }).subtype ??
options.defaultSubtype ?? 'imported',
metadata: {
confidence: rel.confidence,
evidence: rel.evidence
@ -393,7 +400,8 @@ export class SmartImportOrchestrator {
const startTime = Date.now()
const result: SmartImportResult = {
success: false,
extraction: null as any,
// Typed boundary: populated after extraction (see importExcel).
extraction: null as unknown as SmartExcelResult,
entityIds: [],
relationshipIds: [],
stats: {
@ -413,7 +421,7 @@ export class SmartImportOrchestrator {
const pdfResult = await this.pdfImporter.extract(buffer, options)
// Convert PDF result to Excel-like format for processing
result.extraction = this.convertPDFToExcelFormat(pdfResult) as any
result.extraction = this.convertPDFToExcelFormat(pdfResult)
result.stats.rowsProcessed = pdfResult.sectionsProcessed
// Phase 2 & 3: Create entities and relationships
@ -470,7 +478,8 @@ export class SmartImportOrchestrator {
const startTime = Date.now()
const result: SmartImportResult = {
success: false,
extraction: null as any,
// Typed boundary: populated after extraction (see importExcel).
extraction: null as unknown as SmartExcelResult,
entityIds: [],
relationshipIds: [],
stats: {
@ -488,7 +497,7 @@ export class SmartImportOrchestrator {
const jsonResult = await this.jsonImporter.extract(data, options)
result.extraction = this.convertJSONToExcelFormat(jsonResult) as any
result.extraction = this.convertJSONToExcelFormat(jsonResult)
result.stats.rowsProcessed = jsonResult.nodesProcessed
await this.createEntitiesAndRelationships(result, options, onProgress)
@ -532,7 +541,8 @@ export class SmartImportOrchestrator {
const startTime = Date.now()
const result: SmartImportResult = {
success: false,
extraction: null as any,
// Typed boundary: populated after extraction (see importExcel).
extraction: null as unknown as SmartExcelResult,
entityIds: [],
relationshipIds: [],
stats: {
@ -550,7 +560,7 @@ export class SmartImportOrchestrator {
const mdResult = await this.markdownImporter.extract(markdown, options)
result.extraction = this.convertMarkdownToExcelFormat(mdResult) as any
result.extraction = this.convertMarkdownToExcelFormat(mdResult)
result.stats.rowsProcessed = mdResult.sectionsProcessed
await this.createEntitiesAndRelationships(result, options, onProgress)
@ -601,7 +611,9 @@ export class SmartImportOrchestrator {
const entityId = await this.brain.add({
data: extracted.entity.description,
type: extracted.entity.type,
subtype: (extracted.entity as any).subtype ?? options.defaultSubtype ?? 'imported',
subtype:
(extracted.entity as typeof extracted.entity & { subtype?: string })
.subtype ?? options.defaultSubtype ?? 'imported',
confidence: extracted.entity.confidence, // reserved field — dedicated param, not metadata
metadata: { ...extracted.entity.metadata, name: extracted.entity.name, importedFrom: 'smart-import' }
})
@ -637,7 +649,7 @@ export class SmartImportOrchestrator {
}
// Relationship subtype precedence: extractor → caller default → `'imported'` (7.30.1).
// `confidence` is a reserved top-level field — dedicated relate() param, not metadata
relationshipParams.push({ from: extracted.entity.id, to: toEntityId, type: rel.type, subtype: (rel as any).subtype ?? options.defaultSubtype ?? 'imported', confidence: rel.confidence, metadata: { evidence: rel.evidence } })
relationshipParams.push({ from: extracted.entity.id, to: toEntityId, type: rel.type, subtype: (rel as typeof rel & { subtype?: string }).subtype ?? options.defaultSubtype ?? 'imported', confidence: rel.confidence, metadata: { evidence: rel.evidence } })
} catch (error: any) {
result.errors.push(`Failed to prepare relationship: ${error.message}`)
}
@ -688,7 +700,7 @@ export class SmartImportOrchestrator {
rows,
entityMap: pdfResult.entityMap,
processingTime: pdfResult.processingTime,
stats: pdfResult.stats as any
stats: pdfResult.stats
}
}
@ -710,7 +722,7 @@ export class SmartImportOrchestrator {
rows,
entityMap: jsonResult.entityMap,
processingTime: jsonResult.processingTime,
stats: jsonResult.stats as any
stats: jsonResult.stats
}
}
@ -734,7 +746,7 @@ export class SmartImportOrchestrator {
rows,
entityMap: mdResult.entityMap,
processingTime: mdResult.processingTime,
stats: mdResult.stats as any
stats: mdResult.stats
}
}

10
src/importers/SmartPDFImporter.ts
View file

@ -176,7 +176,9 @@ export class SmartPDFImporter {
minParagraphLength: 50,
extractFromTables: true,
groupBy: 'document' as const,
onProgress: () => {},
// Annotated with the full callback signature so the merged opts type has
// a single callable shape (the no-op default narrowed the union).
onProgress: (() => {}) as NonNullable<SmartPDFOptions['onProgress']>,
...options
}
@ -274,7 +276,7 @@ export class SmartPDFImporter {
throughput: Math.round(sectionsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining),
phase: 'extracting'
} as any)
})
}
const pagesProcessed = new Set(data.map(d => d._page)).size
@ -338,7 +340,7 @@ export class SmartPDFImporter {
* Process a single section
*/
private async processSection(
group: { id: string, type: string, items: Array<Record<string, any>> },
group: { id: string, type: 'page' | 'paragraph' | 'table', items: Array<Record<string, any>> },
options: SmartPDFOptions,
stats: SmartPDFResult['stats'],
entityMap: Map<string, string>
@ -444,7 +446,7 @@ export class SmartPDFImporter {
return {
sectionId: group.id,
sectionType: group.type as any,
sectionType: group.type,
entities,
relationships,
concepts,

7
src/importers/handlers/excelHandler.ts
View file

@ -69,13 +69,14 @@ export class ExcelHandler extends BaseFormatHandler {
const sheet = workbook.Sheets[sheetName]
if (!sheet) continue
// Convert sheet to JSON with headers
const sheetData = XLSX.utils.sheet_to_json(sheet, {
// Convert sheet to JSON with headers. `header: 1` makes xlsx return
// rows as arrays of raw cell values, typed via the generic overload.
const sheetData = XLSX.utils.sheet_to_json<unknown[]>(sheet, {
header: 1, // Get as array of arrays first
defval: null,
blankrows: false,
raw: false // Convert to formatted strings
}) as any[][]
})
if (sheetData.length === 0) continue

28
src/importers/handlers/pdfHandler.ts
View file

@ -70,8 +70,20 @@ export class PDFHandler extends BaseFormatHandler {
const pdfDoc = await loadingTask.promise
// Extract metadata
// Extract metadata. pdfjs-dist types getMetadata().info as the bare
// `Object` type, so the well-known PDF document-information dictionary
// keys (PDF 32000-1:2008 §14.3.3) are surfaced via a structural type at
// this library boundary.
const metadata = await pdfDoc.getMetadata()
const documentInfo = metadata.info as {
Title?: string
Author?: string
Subject?: string
Creator?: string
Producer?: string
CreationDate?: string
ModDate?: string
} | undefined
const numPages = pdfDoc.numPages
// Report document loaded
@ -177,13 +189,13 @@ export class PDFHandler extends BaseFormatHandler {
textLength: totalTextLength,
tableCount: detectedTables,
pdfMetadata: {
title: (metadata.info as any)?.Title || null,
author: (metadata.info as any)?.Author || null,
subject: (metadata.info as any)?.Subject || null,
creator: (metadata.info as any)?.Creator || null,
producer: (metadata.info as any)?.Producer || null,
creationDate: (metadata.info as any)?.CreationDate || null,
modificationDate: (metadata.info as any)?.ModDate || null
title: documentInfo?.Title || null,
author: documentInfo?.Author || null,
subject: documentInfo?.Subject || null,
creator: documentInfo?.Creator || null,
producer: documentInfo?.Producer || null,
creationDate: documentInfo?.CreationDate || null,
modificationDate: documentInfo?.ModDate || null
}
}
),

19
src/index.ts
View file

@ -1,15 +1,16 @@
/**
* Brainy 3.0 - Your AI-Powered Second Brain
* Brainy 8.0 - Your AI-Powered Second Brain
* 🧠⚛️ A multi-dimensional database with vector, graph, and relational storage
*
*
* Core Components:
* - Brainy: The unified database with Triple Intelligence
* - Triple Intelligence: Seamless fusion of vector + graph + field search
* - Db: Immutable, generation-pinned database values (now/transact/asOf/with)
* - Plugins: Extensible plugin system (cortex, storage adapters)
* - Neural API: AI-powered clustering and analysis
*/
// Export main Brainy class - the modern, clean API for Brainy 3.0
// Export main Brainy class
import { Brainy } from './brainy.js'
export { Brainy }
@ -28,6 +29,14 @@ export type {
RelateParams,
UpdateRelationParams,
FindParams,
SimilarParams,
GetOptions,
RelatedParams,
AddManyParams,
UpdateManyParams,
RemoveManyParams,
RelateManyParams,
BatchResult,
SubtypeRegistry,
FillSubtypeRule,
FillSubtypeRules,
@ -152,6 +161,10 @@ export type { Migration, MigrationState, MigrationPreview, MigrationResult, Migr
// Export optimistic-concurrency types (7.31.0)
export { RevisionConflictError } from './transaction/RevisionConflictError.js'
// Named not-found errors — thrown by update/relate/updateRelation/similar/
// transact/with() when a referenced entity or relation does not exist.
export { EntityNotFoundError, RelationNotFoundError } from './errors/notFound.js'
// ============= 8.0 Db API — generational MVCC =============
// Immutable database values: brain.now() / brain.transact() / brain.asOf() /
// db.with() / db.persist() / Brainy.load(). See src/db/ for the record layer.

17
src/indexes/columnStore/ColumnManifest.ts
View file

@ -59,9 +59,15 @@ export class ColumnManifest {
async load(storage: StorageAdapter): Promise<void> {
try {
const path = this.manifestPath()
const data = await (storage as any).readObjectFromPath(path)
// readObjectFromPath is a BaseStorage primitive (protected on the
// class, not part of the StorageAdapter interface) — the column store
// owns everything under its base path, and the field-name check below
// validates the manifest before it is adopted.
const data = await (storage as unknown as {
readObjectFromPath: (path: string) => Promise<ManifestData | null>
}).readObjectFromPath(path)
if (data && data.field === this.fieldName) {
this.data = data as ManifestData
this.data = data
}
} catch {
// Not found — start fresh (data is already initialized in constructor)
@ -76,7 +82,12 @@ export class ColumnManifest {
async save(storage: StorageAdapter): Promise<void> {
this.data.version++
const path = this.manifestPath()
await (storage as any).writeObjectToPath(path, this.data)
// writeObjectToPath is a BaseStorage primitive (protected on the class,
// not part of the StorageAdapter interface) — same typed boundary as
// load() above.
await (storage as unknown as {
writeObjectToPath: (path: string, data: ManifestData) => Promise<void>
}).writeObjectToPath(path, this.data)
}
/**

Some files were not shown because too many files have changed in this diff Show more