soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: exclude __words__ keyword index from corruption detection and getStats()

Browse source 
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:

1. validateConsistency() to falsely detect corruption on every startup,
   triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
   and report inflated totalEntries/totalIds stats

Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
This commit is contained in:
David Snelling 2026-01-27 15:38:21 -08:00
parent 32dbdcec61
commit 364360d447
128 changed files with 5637 additions and 5682 deletions
Download patch file Download diff file Expand all files Collapse all files

756
docs/API_REFERENCE.md
View file

File diff suppressed because it is too large Load diff

143
docs/BATCHING.md
View file

@ -1,23 +1,22 @@
# Batch Operations API v5.12.0
# Batch Operations API
> **Enterprise Production-Ready** | Zero N+1 Query Patterns | 90%+ Performance Improvement
## Overview
Brainy v5.12.0 introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage.
Brainy introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage.
### Problem Solved
**Before v5.12.0:**
**Before optimization:**
- VFS `getTreeStructure()` on cloud storage: **12.7 seconds** for directory with 12 files
- N+1 query pattern: 1 directory query + N individual file queries (22 sequential calls × 580ms latency)
**After v5.12.0:**
**After optimization:**
- VFS `getTreeStructure()`: **<1 second** for 12 files (90%+ improvement)
- 2-3 batched calls instead of 22 sequential calls
- Native cloud storage batch APIs for maximum throughput
**IMPORTANT:** The v5.12.0 batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See v6.0.0 changes for comprehensive storage path optimizations.
**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See changes for comprehensive storage path optimizations.
---
@ -56,7 +55,7 @@ results.size // → 3 (number of found entities)
### 2. `storage.getNounMetadataBatch(ids)`
Batch metadata retrieval with direct O(1) path construction (v6.0.0+).
Batch metadata retrieval with direct O(1) path construction.
```typescript
const storage = brain.storage as BaseStorage
@ -65,8 +64,8 @@ const ids = ['id1', 'id2', 'id3']
const metadataMap: Map<string, NounMetadata> = await storage.getNounMetadataBatch(ids)
for (const [id, metadata] of metadataMap) {
console.log(metadata.noun) // Type: 'document', 'person', etc.
console.log(metadata.data) // Entity data
console.log(metadata.noun) // Type: 'document', 'person', etc.
console.log(metadata.data) // Entity data
}
```
@ -92,22 +91,22 @@ const storage = brain.storage as BaseStorage
// Get all relationships from multiple sources
const results: Map<string, GraphVerb[]> = await storage.getVerbsBySourceBatch([
'person1',
'person2'
'person1',
'person2'
])
// Filter by verb type
const createsResults = await storage.getVerbsBySourceBatch(
['person1', 'person2'],
'creates'
['person1', 'person2'],
'creates'
)
// Process results
for (const [sourceId, verbs] of results) {
console.log(`${sourceId} has ${verbs.length} relationships`)
verbs.forEach(verb => {
console.log(` → ${verb.verb} → ${verb.targetId}`)
})
console.log(`${sourceId} has ${verbs.length} relationships`)
verbs.forEach(verb => {
console.log(` → ${verb.verb} → ${verb.targetId}`)
})
}
```
@ -130,8 +129,8 @@ COW-aware batch path resolution with branch inheritance.
const storage = brain.storage as BaseStorage
const paths = [
'entities/nouns/{shard}/id1/metadata.json',
'entities/nouns/{shard}/id2/metadata.json'
'entities/nouns/{shard}/id1/metadata.json',
'entities/nouns/{shard}/id2/metadata.json'
]
// Resolves to: branches/{branch}/entities/nouns/{shard}/{id}/metadata.json
@ -160,9 +159,9 @@ const results = await gcsStorage.readBatch(paths)
// Configuration
gcsStorage.getBatchConfig() // → {
// maxBatchSize: 1000,
// maxConcurrent: 100,
// operationsPerSecond: 1000
// maxBatchSize: 1000,
// maxConcurrent: 100,
// operationsPerSecond: 1000
// }
```
@ -185,9 +184,9 @@ const results = await s3Storage.readBatch(paths)
// Configuration
s3Storage.getBatchConfig() // → {
// maxBatchSize: 1000,
// maxConcurrent: 150,
// operationsPerSecond: 5000
// maxBatchSize: 1000,
// maxConcurrent: 150,
// operationsPerSecond: 5000
// }
```
@ -208,9 +207,9 @@ const results = await r2Storage.readBatch(paths)
// Configuration
r2Storage.getBatchConfig() // → {
// maxBatchSize: 1000,
// maxConcurrent: 150,
// operationsPerSecond: 6000
// maxBatchSize: 1000,
// maxConcurrent: 150,
// operationsPerSecond: 6000
// }
```
@ -231,9 +230,9 @@ const results = await azureStorage.readBatch(paths)
// Configuration
azureStorage.getBatchConfig() // → {
// maxBatchSize: 1000,
// maxConcurrent: 100,
// operationsPerSecond: 3000
// maxBatchSize: 1000,
// maxConcurrent: 100,
// operationsPerSecond: 3000
// }
```
@ -254,7 +253,7 @@ VFS operations automatically use batch APIs for maximum performance.
// OLD: Sequential N+1 pattern (12.7 seconds for 12 files)
const tree = await brain.vfs.getTreeStructure('/my-dir')
// NEW v5.12.0: Parallel breadth-first with batching (<1 second)
// NEW Parallel breadth-first with batching (<1 second)
// ✅ PathResolver.getChildren() uses brain.batchGet() internally
// ✅ Parallel traversal of directories at same tree level
// ✅ 2-3 batched calls instead of 22 sequential calls
@ -264,16 +263,16 @@ const tree = await brain.vfs.getTreeStructure('/my-dir')
```
VFS.getTreeStructure()
↓ PARALLEL (breadth-first traversal)
→ PathResolver.getChildren() [all dirs at level processed in parallel]
↓ BATCHED
→ brain.batchGet(childIds) [1 call instead of N]
↓ BATCHED
→ storage.getNounMetadataBatch(ids) [1 call instead of N]
↓ ADAPTER-SPECIFIC
→ GCS: readBatch() with 100 concurrent downloads
→ S3: readBatch() with 150 concurrent downloads
→ Memory: Promise.all() parallel reads
↓ PARALLEL (breadth-first traversal)
→ PathResolver.getChildren() [all dirs at level processed in parallel]
↓ BATCHED
→ brain.batchGet(childIds) [1 call instead of N]
↓ BATCHED
→ storage.getNounMetadataBatch(ids) [1 call instead of N]
↓ ADAPTER-SPECIFIC
→ GCS: readBatch() with 100 concurrent downloads
→ S3: readBatch() with 150 concurrent downloads
→ Memory: Promise.all() parallel reads
```
**Performance Gains:**
@ -285,11 +284,11 @@ VFS.getTreeStructure()
## Advanced Features Compatibility
### ✅ ID-First Storage Architecture (v6.0.0+)
### ✅ ID-First Storage Architecture
All batch operations use direct ID-first paths - no type lookup needed!
**NEW v6.0.0 Path Structure:**
**ID-First Path Structure:**
```
entities/nouns/{SHARD}/{ID}/metadata.json
entities/verbs/{SHARD}/{ID}/metadata.json
@ -299,7 +298,7 @@ entities/verbs/{SHARD}/{ID}/metadata.json
```typescript
// Every ID maps directly to exactly ONE path - 40x faster!
const id = 'abc-123'
const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
const path = `entities/nouns/${shard}/${id}/metadata.json`
// No type cache needed!
@ -343,7 +342,7 @@ const fork = await brain.fork('experiment')
// Batch operations are isolated
await brain.batchGet([id1, id2]) // → Reads from: branches/main/...
await fork.batchGet([id1, id2]) // → Reads from: branches/experiment/...
await fork.batchGet([id1, id2]) // → Reads from: branches/experiment/...
```
**Inheritance:**
@ -391,7 +390,7 @@ Historical queries use `HistoricalStorageAdapter` which wraps batch operations t
### VFS Operations (12 Files)
| Storage | Before v5.12.0 | After v5.12.0 | Improvement |
| Storage | Before optimization | After optimization | Improvement |
|---------|---------------|---------------|-------------|
| **GCS** | 12.7s | <1s | **92% faster** |
| **S3** | 13.2s | <1s | **92% faster** |
@ -430,8 +429,8 @@ Batch operations gracefully handle missing or invalid entities:
```typescript
const validId = 'abc-123-...'
const invalidIds = [
'11111111-1111-1111-1111-111111111111',
'22222222-2222-2222-2222-222222222222'
'11111111-1111-1111-1111-111111111111',
'22222222-2222-2222-2222-222222222222'
]
const results = await brain.batchGet([validId, ...invalidIds])
@ -471,8 +470,8 @@ results.size // → 1 (deduplicated automatically)
```typescript
const entities = []
for (const id of ids) {
const entity = await brain.get(id)
if (entity) entities.push(entity)
const entity = await brain.get(id)
if (entity) entities.push(entity)
}
```
@ -492,8 +491,8 @@ const entities = Array.from(results.values())
```typescript
const allVerbs = []
for (const sourceId of sourceIds) {
const verbs = await brain.getRelations({ from: sourceId })
allVerbs.push(...verbs)
const verbs = await brain.getRelations({ from: sourceId })
allVerbs.push(...verbs)
}
```
@ -504,7 +503,7 @@ const results = await storage.getVerbsBySourceBatch(sourceIds)
const allVerbs = []
for (const verbs of results.values()) {
allVerbs.push(...verbs)
allVerbs.push(...verbs)
}
```
@ -522,7 +521,7 @@ const results = await brain.batchGet(ids)
// ❌ BAD: Individual gets in loop
for (const id of ids) {
await brain.get(id)
await brain.get(id)
}
```
@ -556,13 +555,13 @@ const results = await brain.batchGet(ids)
// Check results
for (const id of ids) {
if (results.has(id)) {
// Entity exists
const entity = results.get(id)
} else {
// Entity missing (not an error)
console.log(`Entity ${id} not found`)
}
if (results.has(id)) {
// Entity exists
const entity = results.get(id)
} else {
// Entity missing (not an error)
console.log(`Entity ${id} not found`)
}
}
```
@ -597,15 +596,15 @@ npx vitest run tests/integration/storage-batch-operations.test.ts
```
User Code (brain.batchGet)
↓
↓
High-Level API (src/brainy.ts)
↓
↓
Storage Layer (src/storage/baseStorage.ts)
↓
↓
COW Layer (readBatchWithInheritance)
↓
↓
Adapter Layer (readBatchFromAdapter)
↓
↓
Cloud Adapter (GCS/S3/Azure native batch APIs)
```
@ -616,11 +615,11 @@ If an adapter doesn't implement `readBatch()`, the system automatically falls ba
```typescript
// BaseStorage.readBatchFromAdapter()
if (typeof selfWithBatch.readBatch === 'function') {
// Use native batch API
return await selfWithBatch.readBatch(resolvedPaths)
// Use native batch API
return await selfWithBatch.readBatch(resolvedPaths)
} else {
// Automatic parallel fallback
return await Promise.all(resolvedPaths.map(path => this.read(path)))
// Automatic parallel fallback
return await Promise.all(resolvedPaths.map(path => this.read(path)))
}
```
@ -658,7 +657,7 @@ if (typeof selfWithBatch.readBatch === 'function') {
- Zero N+1 query patterns
**Compatibility:**
- ✅ ID-first storage (v6.0.0+)
- ✅ ID-first storage
- ✅ Sharding (256 shards)
- ✅ COW (branch isolation, inheritance)
- ✅ fork() and checkout()

430
docs/CREATING-AUGMENTATIONS.md
View file

@ -1,6 +1,6 @@
# Creating Augmentations for Brainy
> **Updated for v4.0.0** - Includes metadata structure changes and type system improvements
> **Updated** - Includes metadata structure changes and type system improvements
## The BrainyAugmentation Interface
@ -8,38 +8,38 @@ Every augmentation implements this simple yet powerful interface:
```typescript
interface BrainyAugmentation {
// Identification
name: string // Unique name for your augmentation
// Identification
name: string // Unique name for your augmentation
// Execution control
timing: 'before' | 'after' | 'around' | 'replace' // When to execute
operations: string[] // Which operations to intercept
priority: number // Execution order (higher = first)
// Execution control
timing: 'before' | 'after' | 'around' | 'replace' // When to execute
operations: string[] // Which operations to intercept
priority: number // Execution order (higher = first)
// Lifecycle methods
initialize(context: AugmentationContext): Promise<void>
execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T>
shutdown?(): Promise<void> // Optional cleanup
// Lifecycle methods
initialize(context: AugmentationContext): Promise<void>
execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T>
shutdown?(): Promise<void> // Optional cleanup
}
```
## v4.0.0 Breaking Changes for Augmentation Developers
## Breaking Changes for Augmentation Developers
### 1. Metadata Structure Separation
v4.0.0 introduces strict metadata/vector separation for billion-scale performance:
Brainy introduces strict metadata/vector separation for billion-scale performance:
```typescript
// ✅ v4.0.0: Metadata has required type field
// ✅ Metadata has required type field
interface NounMetadata {
noun: NounType // Required! Must be a valid noun type
[key: string]: any // Your custom metadata
noun: NounType // Required! Must be a valid noun type
[key: string]: any // Your custom metadata
}
interface VerbMetadata {
verb: VerbType // Required! Must be a valid verb type
sourceId: string
targetId: string
[key: string]: any
verb: VerbType // Required! Must be a valid verb type
sourceId: string
targetId: string
[key: string]: any
}
```
@ -61,7 +61,7 @@ The verb relationship field changed from `type` to `verb`:
// ❌ v3.x
verb.type === 'relatedTo'
// ✅ v4.0.0
// ✅ Current
verb.verb === 'relatedTo'
```
@ -69,7 +69,7 @@ verb.verb === 'relatedTo'
Storage augmentations are special - they provide the storage backend for Brainy.
### Important: v4.0.0 Storage Requirements
### Important: Storage Requirements
Your storage adapter MUST:
1. **Wrap metadata** with required `noun`/`verb` fields
@ -81,67 +81,67 @@ import { StorageAugmentation } from 'brainy/augmentations'
import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy'
export class MyCustomStorage extends BaseStorageAdapter {
// Internal method: Returns pure structure
async _getNoun(id: string): Promise<HNSWNoun | null> {
const data = await this.fetchFromDatabase(id)
return data ? {
id: data.id,
vector: data.vector,
nounType: data.type
} : null
}
// Internal method: Returns pure structure
async _getNoun(id: string): Promise<HNSWNoun | null> {
const data = await this.fetchFromDatabase(id)
return data ? {
id: data.id,
vector: data.vector,
nounType: data.type
} : null
}
// Public method: Returns WithMetadata structure
async getNoun(id: string): Promise<HNSWNounWithMetadata | null> {
const noun = await this._getNoun(id)
if (!noun) return null
// Public method: Returns WithMetadata structure
async getNoun(id: string): Promise<HNSWNounWithMetadata | null> {
const noun = await this._getNoun(id)
if (!noun) return null
// Fetch metadata separately (v4.0.0 pattern)
const metadata = await this.getNounMetadata(id)
// Fetch metadata separately
const metadata = await this.getNounMetadata(id)
return {
...noun,
metadata: metadata || { noun: noun.nounType || 'thing' }
}
}
return {
...noun,
metadata: metadata || { noun: noun.nounType || 'thing' }
}
}
// CRITICAL: Always save with proper metadata structure
async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> {
// Validate metadata has required 'noun' field
if (!metadata?.noun) {
throw new Error('v4.0.0: NounMetadata requires "noun" field')
}
// CRITICAL: Always save with proper metadata structure
async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> {
// Validate metadata has required 'noun' field
if (!metadata?.noun) {
throw new Error('NounMetadata requires "noun" field')
}
await this.database.save({
id: noun.id,
vector: noun.vector,
nounType: noun.nounType,
metadata: metadata // Stored separately in v4.0.0
})
}
await this.database.save({
id: noun.id,
vector: noun.vector,
nounType: noun.nounType,
metadata: metadata // Stored separately
})
}
}
export class MyStorageAugmentation extends StorageAugmentation {
private config: MyStorageConfig
private config: MyStorageConfig
constructor(config: MyStorageConfig) {
super()
this.name = 'my-custom-storage'
this.config = config
}
constructor(config: MyStorageConfig) {
super()
this.name = 'my-custom-storage'
this.config = config
}
// Called during storage resolution phase
async provideStorage(): Promise<StorageAdapter> {
const storage = new MyCustomStorage(this.config)
this.storageAdapter = storage
return storage
}
// Called during storage resolution phase
async provideStorage(): Promise<StorageAdapter> {
const storage = new MyCustomStorage(this.config)
this.storageAdapter = storage
return storage
}
// Called during augmentation initialization
protected async onInitialize(): Promise<void> {
await this.storageAdapter!.init()
this.log(`Custom storage initialized`)
}
// Called during augmentation initialization
protected async onInitialize(): Promise<void> {
await this.storageAdapter!.init()
this.log(`Custom storage initialized`)
}
}
```
@ -151,9 +151,9 @@ export class MyStorageAugmentation extends StorageAugmentation {
// Register before brain.init()
const brain = new Brainy()
brain.augmentations.register(new MyStorageAugmentation({
connectionString: 'redis://localhost:6379'
connectionString: 'redis://localhost:6379'
}))
await brain.init() // Will use your storage!
await brain.init() // Will use your storage!
```
## Creating a Feature Augmentation
@ -164,43 +164,43 @@ Here's a complete example of a caching augmentation:
import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations'
export class CachingAugmentation extends BaseAugmentation {
private cache = new Map<string, any>()
constructor() {
super()
this.name = 'smart-cache'
this.timing = 'around' // Wrap operations
this.operations = ['search'] // Only cache searches
this.priority = 50 // Mid-priority
}
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
if (operation === 'search') {
// Check cache
const cacheKey = JSON.stringify(params)
if (this.cache.has(cacheKey)) {
this.log('Cache hit!')
return this.cache.get(cacheKey)
}
// Execute and cache
const result = await next()
this.cache.set(cacheKey, result)
return result
}
// Pass through other operations
return next()
}
protected async onInitialize(): Promise<void> {
this.log('Cache initialized')
}
async shutdown(): Promise<void> {
this.cache.clear()
await super.shutdown()
}
private cache = new Map<string, any>()
constructor() {
super()
this.name = 'smart-cache'
this.timing = 'around' // Wrap operations
this.operations = ['search'] // Only cache searches
this.priority = 50 // Mid-priority
}
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
if (operation === 'search') {
// Check cache
const cacheKey = JSON.stringify(params)
if (this.cache.has(cacheKey)) {
this.log('Cache hit!')
return this.cache.get(cacheKey)
}
// Execute and cache
const result = await next()
this.cache.set(cacheKey, result)
return result
}
// Pass through other operations
return next()
}
protected async onInitialize(): Promise<void> {
this.log('Cache initialized')
}
async shutdown(): Promise<void> {
this.cache.clear()
await super.shutdown()
}
}
```
@ -210,20 +210,20 @@ export class CachingAugmentation extends BaseAugmentation {
```typescript
timing = 'before'
async execute(op, params, next) {
// Validate/transform input
const validated = await validate(params)
return next(validated) // Pass modified params
// Validate/transform input
const validated = await validate(params)
return next(validated) // Pass modified params
}
```
### 2. `after` - Post-processing
### 2. `after` - Post-processing
```typescript
timing = 'after'
async execute(op, params, next) {
const result = await next()
// Log, analyze, or modify result
console.log(`Operation ${op} returned:`, result)
return result
const result = await next()
// Log, analyze, or modify result
console.log(`Operation ${op} returned:`, result)
return result
}
```
@ -231,15 +231,15 @@ async execute(op, params, next) {
```typescript
timing = 'around'
async execute(op, params, next) {
console.log('Starting', op)
try {
const result = await next()
console.log('Success', op)
return result
} catch (error) {
console.log('Failed', op, error)
throw error
}
console.log('Starting', op)
try {
const result = await next()
console.log('Success', op)
return result
} catch (error) {
console.log('Failed', op, error)
throw error
}
}
```
@ -247,8 +247,8 @@ async execute(op, params, next) {
```typescript
timing = 'replace'
async execute(op, params, next) {
// Don't call next() - replace entirely!
return myCustomImplementation(params)
// Don't call next() - replace entirely!
return myCustomImplementation(params)
}
```
@ -266,10 +266,10 @@ Common operations in Brainy:
```typescript
interface AugmentationContext {
brain: Brainy // The brain instance
storage: StorageAdapter // Storage backend
config: BrainyConfig // Configuration
log: (message: string, level?: 'info' | 'warn' | 'error') => void
brain: Brainy // The brain instance
storage: StorageAdapter // Storage backend
config: BrainyConfig // Configuration
log: (message: string, level?: 'info' | 'warn' | 'error') => void
}
```
@ -278,50 +278,50 @@ interface AugmentationContext {
### 1. Redis Storage Augmentation
```typescript
export class RedisStorageAugmentation extends StorageAugmentation {
async provideStorage(): Promise<StorageAdapter> {
return new RedisAdapter({
host: 'localhost',
port: 6379,
// Implement full StorageAdapter interface
})
}
async provideStorage(): Promise<StorageAdapter> {
return new RedisAdapter({
host: 'localhost',
port: 6379,
// Implement full StorageAdapter interface
})
}
}
```
### 2. Audit Trail Augmentation
```typescript
export class AuditAugmentation extends BaseAugmentation {
timing = 'after'
operations = ['add', 'update', 'delete']
async execute(op, params, next) {
const result = await next()
// Log to audit trail
await this.logAudit({
operation: op,
params,
result,
timestamp: new Date(),
user: this.context.config.currentUser
})
return result
}
timing = 'after'
operations = ['add', 'update', 'delete']
async execute(op, params, next) {
const result = await next()
// Log to audit trail
await this.logAudit({
operation: op,
params,
result,
timestamp: new Date(),
user: this.context.config.currentUser
})
return result
}
}
```
### 3. Rate Limiting Augmentation
```typescript
export class RateLimitAugmentation extends BaseAugmentation {
timing = 'before'
operations = ['search']
private limiter = new RateLimiter({ rps: 100 })
async execute(op, params, next) {
await this.limiter.acquire() // Wait if rate limited
return next()
}
timing = 'before'
operations = ['search']
private limiter = new RateLimiter({ rps: 100 })
async execute(op, params, next) {
await this.limiter.acquire() // Wait if rate limited
return next()
}
}
```
@ -332,12 +332,12 @@ Future capability for premium augmentations:
```typescript
// package.json
{
"name": "@brain-cloud/redis-storage",
"brainy": {
"type": "augmentation",
"category": "storage",
"premium": true
}
"name": "@brain-cloud/redis-storage",
"brainy": {
"type": "augmentation",
"category": "storage",
"premium": true
}
}
// Users can install via:
@ -356,43 +356,43 @@ Future capability for premium augmentations:
6. **Log appropriately** - Use context.log() for consistent output
7. **Document your augmentation** - Include examples
### v4.0.0 Specific Best Practices
### Specific Best Practices
8. **Always include `noun` field** when creating/modifying NounMetadata:
```typescript
const metadata: NounMetadata = {
noun: 'thing', // REQUIRED!
yourField: 'value'
}
```
```typescript
const metadata: NounMetadata = {
noun: 'thing', // REQUIRED!
yourField: 'value'
}
```
9. **Use `verb` property** not `type` when working with relationships:
```typescript
// ✅ Correct
if (verb.verb === 'relatedTo') { ... }
```typescript
// ✅ Correct
if (verb.verb === 'relatedTo') { ... }
// ❌ Wrong (v3.x pattern)
if (verb.type === 'relatedTo') { ... }
```
// ❌ Wrong (v3.x pattern)
if (verb.type === 'relatedTo') { ... }
```
10. **Access metadata correctly** from storage:
```typescript
// ✅ Correct - metadata is already structured
const nounType = noun.metadata.noun
```typescript
// ✅ Correct - metadata is already structured
const nounType = noun.metadata.noun
// ⚠️ Fallback pattern for robustness
const nounType = noun.metadata?.noun || 'thing'
```
// ⚠️ Fallback pattern for robustness
const nounType = noun.metadata?.noun || 'thing'
```
11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations:
```typescript
// ✅ Good - Separate concerns
await storage.saveNoun(noun)
await storage.saveMetadata(noun.id, metadata)
```typescript
// ✅ Good - Separate concerns
await storage.saveNoun(noun)
await storage.saveMetadata(noun.id, metadata)
// ❌ Bad - Mixing concerns
await storage.saveNounWithEverything(combinedData)
```
// ❌ Bad - Mixing concerns
await storage.saveNounWithEverything(combinedData)
```
## Testing Your Augmentation
@ -401,23 +401,23 @@ import { Brainy } from 'brainy'
import { MyAugmentation } from './my-augmentation'
describe('MyAugmentation', () => {
let brain: Brainy
beforeEach(async () => {
brain = new Brainy()
brain.augmentations.register(new MyAugmentation())
await brain.init()
})
afterEach(async () => {
await brain.destroy()
})
it('should enhance searches', async () => {
// Test your augmentation's effect
const results = await brain.search('test')
expect(results).toHaveProperty('enhanced', true)
})
let brain: Brainy
beforeEach(async () => {
brain = new Brainy()
brain.augmentations.register(new MyAugmentation())
await brain.init()
})
afterEach(async () => {
await brain.destroy()
})
it('should enhance searches', async () => {
// Test your augmentation's effect
const results = await brain.search('test')
expect(results).toHaveProperty('enhanced', true)
})
})
```

4
docs/DEVELOPER_LEARNING_PATH.md
View file

@ -613,7 +613,7 @@ vfsFiles.forEach(f => {
})
// 3. VFS FILTERING IN KNOWLEDGE QUERIES
console.log('\n🔍 Understanding VFS filtering (v4.4.0)...\n')
console.log('\n🔍 Understanding VFS filtering...\n')
// Create some knowledge entities
const conceptId = await brain.add({
@ -758,7 +758,7 @@ await brain.close()
### Key Concepts
#### 1. **VFS Filtering Architecture (v4.4.0)**
#### 1. **VFS Filtering Architecture**
```typescript
// 🎯 DEFAULT BEHAVIOR: Clean Separation

835
docs/FIND_SYSTEM.md
View file

File diff suppressed because it is too large Load diff

8
docs/PERFORMANCE.md
View file

@ -24,11 +24,11 @@ Where:
- `f` = number of fields for entity type
- `t` = number of types (42 nouns, 127 verbs)
### v5.11.1: brain.get() Metadata-Only Optimization
### brain.get() Metadata-Only Optimization
✨ **Massive Performance Improvement**: `brain.get()` is now **76-81% faster** by default!
| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup | Use Case |
| Operation | Before | After | Speedup | Use Case |
|-----------|------------------|-----------------|---------|----------|
| **brain.get() (metadata-only)** | 43ms, 6KB | **10ms, 300 bytes** | **76-81%** | VFS, existence checks, metadata |
| **brain.get({ includeVectors: true })** | 43ms, 6KB | 43ms, 6KB | 0% | Similarity calculations |
@ -308,7 +308,7 @@ const results = await Promise.all(searchPromises)
- ✅ **Horizontally Scalable**: Stateless operations support clustering
- ✅ **Zero Stubs**: Every line of code is production-ready
## Lazy Loading Performance (v5.7.7+)
## Lazy Loading Performance
Brainy supports two initialization modes for optimal performance across different use cases:
@ -324,7 +324,7 @@ await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities)
- First query: Instant (indexes already loaded)
- Use case: Traditional applications, long-running servers
### Mode 2: Lazy Loading (v5.7.7+)
### Mode 2: Lazy Loading
```javascript
const brain = new Brainy({ disableAutoRebuild: true })

147
docs/README.md
View file

@ -1,8 +1,8 @@
# Brainy Documentation (v6.5.0)
# Brainy Documentation
Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine.
## 🆕 What's New in v4.0.0
## 🆕 What's New in
**Production-Ready Cost Optimization:**
- **Lifecycle Management**: Automatic tier transitions for S3, GCS, Azure (96% cost savings!)
@ -14,7 +14,7 @@ Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI
**Cost Impact Example (500TB dataset):**
- Before: $138,000/year
- After v4.0.0: $5,940/year
- After $5,940/year
- **Savings: $132,060/year (96%)**
## 📊 Implementation Status
@ -85,23 +85,23 @@ await brain.init()
// Add entities (nouns)
const articleId = await brain.add({
data: "Revolutionary AI Breakthrough",
type: NounType.Document,
metadata: { category: "technology", rating: 4.8 }
data: "Revolutionary AI Breakthrough",
type: NounType.Document,
metadata: { category: "technology", rating: 4.8 }
})
const authorId = await brain.add({
data: "Dr. Sarah Chen",
type: NounType.Person,
metadata: { role: "researcher" }
data: "Dr. Sarah Chen",
type: NounType.Person,
metadata: { role: "researcher" }
})
// Create relationships (verbs)
await brain.relate({
from: authorId,
to: articleId,
type: VerbType.CreatedBy,
metadata: { date: "2024-01-15", contribution: "primary" }
from: authorId,
to: articleId,
type: VerbType.CreatedBy,
metadata: { date: "2024-01-15", contribution: "primary" }
})
// Query naturally
@ -117,7 +117,7 @@ const results = await brain.find("highly rated technology articles by researcher
| [API Reference](./api/README.md) | **START HERE** - Complete API documentation with examples |
| [VFS Quick Start](./vfs/QUICK_START.md) | Virtual filesystem in 30 seconds |
### 🆕 v4.0.0 Migration & Optimization
### 🆕 Migration & Optimization
| Document | Description |
|----------|-------------|
@ -135,15 +135,15 @@ const results = await brain.find("highly rated technology articles by researcher
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | Revolutionary data model - entities and relationships |
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Field unified query system |
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
| [Storage Architecture](./architecture/storage-architecture.md) | **v4.0.0** - Storage adapters and optimization |
| [Data Storage Architecture](./architecture/data-storage-architecture.md) | **v4.0.0** - Metadata/vector separation, sharding |
| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
| [Data Storage Architecture](./architecture/data-storage-architecture.md) | Metadata/vector separation, sharding |
| [Index Architecture](./architecture/index-architecture.md) | HNSW, Graph, and Metadata indexing |
### 💾 Storage & Deployment
| Document | Description |
|----------|-------------|
| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | **v4.0.0** - Deploy on AWS, GCP, Azure, Cloudflare |
| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | Deploy on AWS, GCP, Azure, Cloudflare |
| [AWS Deployment](./deployment/aws-deployment.md) | AWS-specific deployment patterns |
| [GCP Deployment](./deployment/gcp-deployment.md) | Google Cloud deployment |
| [Kubernetes Deployment](./deployment/kubernetes-deployment.md) | K8s deployment configurations |
@ -257,75 +257,68 @@ const results = await brain.find("highly rated technology articles by researcher
```
docs/
├── README.md (this file) # Complete documentation index
├── MIGRATION-V3-TO-V4.md # v4.0.0 migration guide
├── README.md (this file) # Complete documentation index
├── MIGRATION-V3-TO-V4.md # Migration guide
│
├── guides/ # User guides
│ ├── natural-language.md
│ ├── neural-api.md
│ ├── import-anything.md
│ ├── framework-integration.md
│ ├── nextjs-integration.md
│ ├── vue-integration.md
│ ├── distributed-system.md
│ ├── model-loading.md
│ └── enterprise-for-everyone.md
├── guides/ # User guides
│ ├── natural-language.md
│ ├── neural-api.md
│ ├── import-anything.md
│ ├── framework-integration.md
│ ├── nextjs-integration.md
│ ├── vue-integration.md
│ ├── distributed-system.md
│ ├── model-loading.md
│ └── enterprise-for-everyone.md
│
├── architecture/ # System architecture
│ ├── overview.md
│ ├── noun-verb-taxonomy.md
│ ├── triple-intelligence.md
│ ├── zero-config.md
│ ├── storage-architecture.md # v4.0.0
│ ├── data-storage-architecture.md # v4.0.0
│ ├── index-architecture.md
│ ├── distributed-storage.md
│ ├── augmentations.md
│ ├── augmentation-system-audit.md
│ ├── augmentations-actual.md
│ ├── finite-type-system.md
│ └── ...
├── architecture/ # System architecture
│ ├── overview.md
│ ├── noun-verb-taxonomy.md
│ ├── triple-intelligence.md
│ ├── zero-config.md
│ ├── storage-architecture.md│ ├── data-storage-architecture.md│ ├── index-architecture.md
│ ├── distributed-storage.md
│ ├── augmentations.md
│ ├── augmentation-system-audit.md
│ ├── augmentations-actual.md
│ ├── finite-type-system.md
│ └── ...
│
├── operations/ # Operations guides
│ ├── cost-optimization-aws-s3.md # v4.0.0
│ ├── cost-optimization-gcs.md # v4.0.0
│ ├── cost-optimization-azure.md # v4.0.0
│ ├── cost-optimization-cloudflare-r2.md # v4.0.0
│ └── capacity-planning.md
├── operations/ # Operations guides
│ ├── cost-optimization-aws-s3.md│ ├── cost-optimization-gcs.md│ ├── cost-optimization-azure.md│ ├── cost-optimization-cloudflare-r2.md│ └── capacity-planning.md
│
├── deployment/ # Deployment guides
│ ├── CLOUD_DEPLOYMENT_GUIDE.md # v4.0.0
│ ├── aws-deployment.md
│ ├── gcp-deployment.md
│ └── kubernetes-deployment.md
├── deployment/ # Deployment guides
│ ├── CLOUD_DEPLOYMENT_GUIDE.md│ ├── aws-deployment.md
│ ├── gcp-deployment.md
│ └── kubernetes-deployment.md
│
├── vfs/ # Virtual Filesystem docs
│ ├── QUICK_START.md
│ ├── VFS_CORE.md
│ ├── SEMANTIC_VFS.md
│ ├── VFS_API_GUIDE.md
│ ├── NEURAL_EXTRACTION.md
│ ├── COMMON_PATTERNS.md
│ └── ...
├── vfs/ # Virtual Filesystem docs
│ ├── QUICK_START.md
│ ├── VFS_CORE.md
│ ├── SEMANTIC_VFS.md
│ ├── VFS_API_GUIDE.md
│ ├── NEURAL_EXTRACTION.md
│ ├── COMMON_PATTERNS.md
│ └── ...
│
├── api/ # API documentation
│ ├── README.md
│ └── COMPREHENSIVE_API_OVERVIEW.md
├── api/ # API documentation
│ ├── README.md
│ └── COMPREHENSIVE_API_OVERVIEW.md
│
├── augmentations/ # Augmentation docs
│ ├── COMPLETE-REFERENCE.md
│ ├── DEVELOPER-GUIDE.md
│ ├── CONFIGURATION.md
│ └── api-server.md
├── augmentations/ # Augmentation docs
│ ├── COMPLETE-REFERENCE.md
│ ├── DEVELOPER-GUIDE.md
│ ├── CONFIGURATION.md
│ └── api-server.md
│
├── features/ # Feature documentation
│ ├── complete-feature-list.md
│ └── v3-features.md
├── features/ # Feature documentation
│ ├── complete-feature-list.md
│ └── v3-features.md
│
└── internal/ # Internal docs
├── AUDIT_REPORT.md
├── CLEANUP_SUMMARY.md
└── HONEST_STATUS.md
└── internal/ # Internal docs
├── AUDIT_REPORT.md
├── CLEANUP_SUMMARY.md
└── HONEST_STATUS.md
```
## Community

2
docs/RELEASE-GUIDE.md
View file

@ -48,7 +48,7 @@ git commit -m "chore: remove unused dependency"
# DON'T use BREAKING CHANGE for internal changes
git commit -m "feat: improve model delivery
BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers v3.0.0
BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers
```
## Release Workflow Checklist

2
docs/STAGE3-CANONICAL-TAXONOMY.md
View file

@ -1,4 +1,4 @@
# Brainy Stage 3: Canonical Taxonomy (v6.0.0)
# Brainy Stage 3: Canonical Taxonomy
**Status:** FINAL - This is the definitive, timeless taxonomy
**Total Types:** 169 (42 nouns + 127 verbs)

234
docs/api/COMPREHENSIVE_API_OVERVIEW.md
View file

@ -1,6 +1,6 @@
# Brainy Complete Public API Reference
> **Accurate API documentation for Brainy v6.5.0+**
> **Accurate API documentation for Brainy**
## Initialization
@ -13,14 +13,14 @@ await brain.init()
// With configuration
const brain = new Brainy({
storage: { type: 'memory' }, // or { path: './my-data' } for filesystem
embeddingModel: 'Q8', // Q4, Q8, F16, F32
silent: true // Suppress logs
storage: { type: 'memory' }, // or { path: './my-data' } for filesystem
embeddingModel: 'Q8', // Q4, Q8, F16, F32
silent: true // Suppress logs
})
await brain.init()
```
## Readiness API (v7.3.0+)
## Readiness API
For reliable initialization detection, especially in cloud environments with progressive initialization:
@ -28,10 +28,10 @@ For reliable initialization detection, especially in cloud environments with pro
```typescript
const brain = new Brainy()
brain.init() // Fire and forget
brain.init() // Fire and forget
// Elsewhere (e.g., API handler)
await brain.ready // Wait until init() completes
await brain.ready // Wait until init() completes
const results = await brain.find({ query: 'test' })
```
@ -39,7 +39,7 @@ const results = await brain.find({ query: 'test' })
```typescript
if (brain.isInitialized) {
// Safe to use brain methods
// Safe to use brain methods
}
```
@ -49,7 +49,7 @@ if (brain.isInitialized) {
// Returns true when ALL initialization is complete, including background tasks
// Useful for cloud storage adapters with progressive initialization
if (brain.isFullyInitialized()) {
console.log('All background tasks complete')
console.log('All background tasks complete')
}
```
@ -57,7 +57,7 @@ if (brain.isFullyInitialized()) {
```typescript
const brain = new Brainy({ storage: { type: 'gcs', ... } })
await brain.init() // Fast return in cloud (<200ms)
await brain.init() // Fast return in cloud (<200ms)
// Optional: wait for all background tasks (bucket validation, count sync)
await brain.awaitBackgroundInit()
@ -68,15 +68,15 @@ console.log('Fully initialized including background tasks')
```typescript
app.get('/health', async (req, res) => {
try {
await brain.ready
res.json({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
} catch (error) {
res.status(503).json({ status: 'initializing', error: error.message })
}
try {
await brain.ready
res.json({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
} catch (error) {
res.status(503).json({ status: 'initializing', error: error.message })
}
})
```
@ -87,16 +87,16 @@ app.get('/health', async (req, res) => {
```typescript
// Add with data (auto-embedded)
const id = await brain.add({
data: 'Machine learning is a subset of AI',
type: NounType.Document,
metadata: { author: 'Alice', tags: ['ml', 'ai'] }
data: 'Machine learning is a subset of AI',
type: NounType.Document,
metadata: { author: 'Alice', tags: ['ml', 'ai'] }
})
// Add with pre-computed vector
const id = await brain.add({
data: 'Content here',
vector: [0.1, 0.2, ...], // 384 dimensions
type: NounType.Concept
data: 'Content here',
vector: [0.1, 0.2, ...], // 384 dimensions
type: NounType.Concept
})
```
@ -120,9 +120,9 @@ const entity = await brain.get('entity-id', { includeVectors: true })
```typescript
await brain.update({
id: 'entity-id',
data: 'Updated content', // Re-embeds if changed
metadata: { reviewed: true } // Merges with existing
id: 'entity-id',
data: 'Updated content', // Re-embeds if changed
metadata: { reviewed: true } // Merges with existing
})
```
@ -144,10 +144,10 @@ await brain.clear()
```typescript
const relationId = await brain.relate({
from: 'source-entity-id',
to: 'target-entity-id',
type: VerbType.RelatedTo,
metadata: { strength: 0.9 }
from: 'source-entity-id',
to: 'target-entity-id',
type: VerbType.RelatedTo,
metadata: { strength: 0.9 }
})
```
@ -168,8 +168,8 @@ const relations = await brain.getRelations({ to: 'entity-id' })
// Filter by type
const relations = await brain.getRelations({
from: 'entity-id',
type: VerbType.Contains
from: 'entity-id',
type: VerbType.Contains
})
```
@ -191,17 +191,17 @@ const results = await brain.find('machine learning algorithms')
// With options
const results = await brain.find({
query: 'machine learning',
limit: 10,
threshold: 0.7,
type: NounType.Document,
where: { author: 'Alice' },
excludeVFS: true // Exclude VFS files from results
query: 'machine learning',
limit: 10,
threshold: 0.7,
type: NounType.Document,
where: { author: 'Alice' },
excludeVFS: true // Exclude VFS files from results
})
// Natural language query (Triple Intelligence)
const results = await brain.find(
'Show me documents about AI written by Alice in 2024'
'Show me documents about AI written by Alice in 2024'
)
```
@ -218,15 +218,15 @@ const results = await brain.find(
```typescript
// Find similar to entity ID
const similar = await brain.similar({
to: 'entity-id',
limit: 5
to: 'entity-id',
limit: 5
})
// Find similar to vector
const similar = await brain.similar({
to: [0.1, 0.2, ...], // Vector
limit: 10,
type: NounType.Document
to: [0.1, 0.2, ...], // Vector
limit: 10,
type: NounType.Document
})
```
@ -236,15 +236,15 @@ const similar = await brain.similar({
```typescript
const result = await brain.addMany({
items: [
{ data: 'First item', type: NounType.Document },
{ data: 'Second item', type: NounType.Concept },
{ data: 'Third item', metadata: { priority: 'high' } }
],
continueOnError: true, // Don't stop on failures
onProgress: (completed, total) => {
console.log(`${completed}/${total} complete`)
}
items: [
{ data: 'First item', type: NounType.Document },
{ data: 'Second item', type: NounType.Concept },
{ data: 'Third item', metadata: { priority: 'high' } }
],
continueOnError: true, // Don't stop on failures
onProgress: (completed, total) => {
console.log(`${completed}/${total} complete`)
}
})
console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}`)
@ -255,18 +255,18 @@ console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}
```typescript
// Delete by IDs
const result = await brain.deleteMany({
ids: ['id1', 'id2', 'id3'],
continueOnError: true
ids: ['id1', 'id2', 'id3'],
continueOnError: true
})
// Delete by type
const result = await brain.deleteMany({
type: NounType.TempData
type: NounType.TempData
})
// Delete by metadata filter
const result = await brain.deleteMany({
where: { status: 'archived' }
where: { status: 'archived' }
})
```
@ -274,12 +274,12 @@ const result = await brain.deleteMany({
```typescript
const ids = await brain.relateMany({
items: [
{ from: 'a', to: 'b', type: VerbType.RelatedTo },
{ from: 'b', to: 'c', type: VerbType.Contains },
{ from: 'c', to: 'd', type: VerbType.References }
],
continueOnError: true
items: [
{ from: 'a', to: 'b', type: VerbType.RelatedTo },
{ from: 'b', to: 'c', type: VerbType.Contains },
{ from: 'c', to: 'd', type: VerbType.References }
],
continueOnError: true
})
```
@ -287,11 +287,11 @@ const ids = await brain.relateMany({
```typescript
const result = await brain.updateMany({
items: [
{ id: 'id1', metadata: { reviewed: true } },
{ id: 'id2', data: 'Updated content' }
],
continueOnError: true
items: [
{ id: 'id1', metadata: { reviewed: true } },
{ id: 'id2', data: 'Updated content' }
],
continueOnError: true
})
```
@ -309,8 +309,8 @@ const detailed = await neural.similar('text1', 'text2', { detailed: true })
// Clustering
const clusters = await neural.clusters()
const clusters = await neural.clusters({
algorithm: 'hierarchical',
maxClusters: 10
algorithm: 'hierarchical',
maxClusters: 10
})
// K-nearest neighbors
@ -327,13 +327,13 @@ const domainClusters = await neural.clusterByDomain('category')
// Temporal clustering
const temporalClusters = await neural.clusterByTime('createdAt', [
{ start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1' },
{ start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2' }
{ start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1' },
{ start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2' }
])
// Streaming clusters (for large datasets)
for await (const batch of neural.clusterStream({ batchSize: 100 })) {
console.log(`Progress: ${batch.progress.percentage}%`)
console.log(`Progress: ${batch.progress.percentage}%`)
}
```
@ -355,11 +355,11 @@ await vfs.mkdir('/project/src', { recursive: true })
const files = await vfs.readdir('/project')
await vfs.rmdir('/project', { recursive: true })
// Bulk operations (v6.5.0+)
// Bulk operations
const result = await vfs.bulkWrite([
{ type: 'mkdir', path: '/data' },
{ type: 'write', path: '/data/config.json', data: '{}' },
{ type: 'write', path: '/data/users.json', data: '[]' }
{ type: 'mkdir', path: '/data' },
{ type: 'write', path: '/data/config.json', data: '{}' },
{ type: 'write', path: '/data/users.json', data: '[]' }
])
// Note: mkdir operations run first (sequentially), then other ops in parallel
@ -414,22 +414,22 @@ const snapshot = await brain.asOf('commit-id')
```typescript
// Stream all entities
for await (const entity of brain.streaming.entities()) {
console.log(entity.id)
console.log(entity.id)
}
// Stream with filters
for await (const entity of brain.streaming.entities({
type: NounType.Document,
where: { status: 'active' }
type: NounType.Document,
where: { status: 'active' }
})) {
// Process each entity
// Process each entity
}
// Stream relationships
for await (const relation of brain.streaming.relations({
from: 'entity-id'
from: 'entity-id'
})) {
console.log(relation)
console.log(relation)
}
```
@ -438,9 +438,9 @@ for await (const relation of brain.streaming.relations({
```typescript
// Paginated queries
const page1 = await brain.pagination.find({
query: 'machine learning',
page: 1,
pageSize: 20
query: 'machine learning',
page: 1,
pageSize: 20
})
console.log(`Page ${page1.page} of ${page1.totalPages}`)
@ -448,9 +448,9 @@ console.log(`Total results: ${page1.total}`)
// Get next page
const page2 = await brain.pagination.find({
query: 'machine learning',
page: 2,
pageSize: 20
query: 'machine learning',
page: 2,
pageSize: 20
})
```
@ -517,20 +517,20 @@ VerbType.ModifiedBy
```typescript
try {
await brain.get('nonexistent-id')
await brain.get('nonexistent-id')
} catch (error) {
if (error.message.includes('not found')) {
// Handle missing entity
}
if (error.message.includes('not found')) {
// Handle missing entity
}
}
// VFS errors use POSIX codes
try {
await vfs.readFile('/nonexistent')
await vfs.readFile('/nonexistent')
} catch (error) {
if (error.code === 'ENOENT') {
// File not found
}
if (error.code === 'ENOENT') {
// File not found
}
}
```
@ -538,26 +538,26 @@ try {
```typescript
const brain = new Brainy({
// Storage
storage: {
type: 'memory' | 'filesystem',
path: './data', // For filesystem
forceMemoryStorage: false // Force memory even if path exists
},
// Storage
storage: {
type: 'memory' | 'filesystem',
path: './data', // For filesystem
forceMemoryStorage: false // Force memory even if path exists
},
// Embeddings
embeddingModel: 'Q8', // Q4, Q8, F16, F32
dimensions: 384, // Auto-detected
// Embeddings
embeddingModel: 'Q8', // Q4, Q8, F16, F32
dimensions: 384, // Auto-detected
// Performance
silent: false, // Suppress console output
verbose: false, // Extra logging
// Performance
silent: false, // Suppress console output
verbose: false, // Extra logging
// Augmentations (auto-enabled by default)
augmentations: {
cache: true,
display: true,
metrics: true
}
// Augmentations (auto-enabled by default)
augmentations: {
cache: true,
display: true,
metrics: true
}
})
```

32
docs/architecture/data-storage-architecture.md
View file

@ -1,4 +1,4 @@
# Brainy Data Storage Architecture (v5.11.0)
# Brainy Data Storage Architecture
**Complete file structure reference for all storage backends**
@ -13,7 +13,7 @@ This document explains how Brainy stores, indexes, and scales data across all st
3. [The 4 Indexes](#3-the-4-indexes)
4. [Sharding Strategy](#4-sharding-strategy)
5. [COW (Copy-on-Write) Architecture](#5-cow-copy-on-write-architecture)
6. [ID-First Storage Architecture (v6.0.0+)](#6-id-first-storage-architecture-v600)
6. [ID-First Storage Architecture](#6-id-first-storage-architecture-v600)
7. [VFS (Virtual File System)](#7-vfs-virtual-file-system)
8. [Storage Backend Mapping](#8-storage-backend-mapping)
9. [Performance Characteristics](#9-performance-characteristics)
@ -438,7 +438,7 @@ Unlike entities and relationships, system metadata consists of **index files** t
Understanding how Brainy constructs storage paths is critical for debugging and optimization.
### Path Construction Steps (v6.0.0+)
### Path Construction Steps
**For an entity (noun)**:
```typescript
@ -498,7 +498,7 @@ const fieldIndexPath = `_system/metadata_indexes/__metadata_field_index__${field
const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
```
### Path Patterns Summary (v6.0.0+)
### Path Patterns Summary
| Data Type | Path Pattern | Sharded? | Branched? |
|-----------|--------------|----------|-----------|
@ -516,7 +516,7 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
| **HNSW node** | `_system/hnsw/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No |
| **Field index** | `_system/metadata_indexes/__metadata_field_index__{field}.json` | ❌ No | ❌ No |
### Key Principles (v6.0.0+)
### Key Principles
1. **Shard Extraction**: Always use first 2 hex characters of UUID/SHA-256
2. **ID-First**: Shard + ID come BEFORE type (type is in metadata)
@ -549,7 +549,7 @@ Brainy uses four complementary index systems for different query patterns.
- Memory (standard): ~200MB per 100K entities
- Memory (lazy): ~15-33MB per 100K entities (5-10x less!)
**Automatic Lazy Mode** (v3.36.0+): Enables automatically when vectors don't fit in UnifiedCache
**Automatic Lazy Mode**: Enables automatically when vectors don't fit in UnifiedCache
---
@ -559,7 +559,7 @@ Brainy uses four complementary index systems for different query patterns.
**Location**: MetadataIndexManager index on `noun` field
**Data Structure**: RoaringBitmap32 per type value
**How It Works** (v6.0.0+):
**How It Works**:
```typescript
// Find all Person entities
const people = await brain.getNouns({ type: 'person' })
@ -721,7 +721,7 @@ COW is Brainy's **git-like versioning system** that enables:
- ✅ **Deduplication** (identical data stored only once)
- ✅ **Version history** (full audit trail of all changes)
**Status**: ALWAYS ENABLED (v5.11.0+) - cannot be disabled
**Status**: ALWAYS ENABLED - cannot be disabled
---
@ -782,19 +782,19 @@ _cow/blobs/ab/abc123...sha256.bin (used by both entities)
---
## 6. ID-First Storage Architecture (v6.0.0+)
## 6. ID-First Storage Architecture
### 6.1 What is ID-First?
**ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused 20-21 second delays on cloud storage.
**Old type-first structure** (v5.4.0-v5.12.0):
**Old type-first structure** (v5.12.0):
```
branches/main/entities/nouns/{TYPE}/metadata/00/001234...uuid.json
# Problem: Requires knowing type OR searching 42 type directories!
```
**NEW ID-first structure** (v6.0.0+):
**NEW ID-first structure**:
```
branches/main/entities/nouns/00/001234...uuid/metadata.json
# Direct O(1) lookup - no type needed!
@ -809,7 +809,7 @@ branches/main/entities/nouns/00/001234...uuid/metadata.json
// v5.x: Had to search 42 types if type unknown
// Result: 21 seconds on GCS (42 types × 500ms)
// v6.0.0: Direct path from ID
// Direct path from ID
const id = '001234...'
const shard = id.substring(0, 2) // '00'
const path = `branches/main/entities/nouns/${shard}/${id}/metadata.json`
@ -1165,7 +1165,7 @@ opfs://root/brainy/
### 10.2 clear() Operation
**What clear() deletes** (v5.11.0+):
**What clear() deletes**:
✅ Deletes:
- `branches/` → ALL entity data (all types, all shards, all branches, all forks)
@ -1184,7 +1184,7 @@ opfs://root/brainy/
**Example**:
```typescript
await brain.storage.clear() // ✅ Deletes ALL data correctly (v5.11.0+)
await brain.storage.clear() // ✅ Deletes ALL data correctly
await brain.add({ data: 'Alice', type: 'person' }) // ✅ COW reinitializes automatically
```
@ -1353,7 +1353,7 @@ const historicalData = await yesterday.getNouns({ type: 'Character' })
await brain.storage.clear()
```
**What happens in storage** (v5.11.0+):
**What happens in storage**:
```
1. Delete all entity data:
→ Remove: branches/ (entire directory)
@ -1452,7 +1452,7 @@ await brain.addBatch([
## 11. Summary
**Complete Storage Structure (v6.0.0)**:
**Complete Storage Structure**:
- **3 storage layers**: branches/ (data), _cow/ (versions), _system/ (indexes)
- **2 files per entity**: metadata.json + vector.json (optimized I/O)
- **4 indexes**: HNSW (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields)

32
docs/architecture/index-architecture.md
View file

@ -25,7 +25,7 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
- **FieldTypeInference** - DuckDB-inspired value-based field type detection
- **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count)
- **Sorted Indexes** - Support orderBy queries (automatically maintained)
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes (v7.7.0)
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes
**GraphAdjacencyIndex contains:**
- **lsmTreeSource** - Source → Targets (outgoing edges)
@ -39,7 +39,7 @@ All indexes share a **UnifiedCache** for coordinated memory management, ensuring
**Purpose**: Enable O(1) field-value lookups and O(log n) range queries on metadata fields using adaptive chunked sparse indexing.
### Internal Architecture (v3.42.0)
### Internal Architecture
```typescript
class MetadataIndexManager {
@ -64,7 +64,7 @@ class MetadataIndexManager {
### Key Data Structures
#### Chunked Sparse Index (NEW in v3.42.0)
#### Chunked Sparse Index
```typescript
// SparseIndex: Directory of chunks for a field
// Example: field="status"
@ -87,7 +87,7 @@ interface ChunkDescriptor {
class ChunkData {
chunkId: number
field: string
entries: Map<value, RoaringBitmap32> // ~50 values per chunk (v3.43.0: roaring bitmaps!)
entries: Map<value, RoaringBitmap32> // ~50 values per chunk (roaring bitmaps!)
}
```
@ -96,7 +96,7 @@ class ChunkData {
- O(log n) range queries with zone maps
- 630x file reduction (560k flat files → 89 chunk files)
#### Roaring Bitmap Optimization (NEW in v3.43.0)
#### Roaring Bitmap Optimization
**Problem Solved**: JavaScript `Set<string>` for storing entity IDs was inefficient:
- Memory overhead: ~40 bytes per UUID string (36 chars + overhead)
@ -150,7 +150,7 @@ class ChunkData {
**Multi-Field Intersection (THE BIG WIN!)**:
```typescript
// Before (v3.42.0): JavaScript array filtering
// Before: JavaScript array filtering
async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string[]> {
// 1. Fetch UUID arrays for each field
const statusIds = await this.getIds('status', 'active') // ["uuid1", "uuid2", ...]
@ -160,7 +160,7 @@ async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string
return statusIds.filter(id => roleIds.includes(id)) // O(n*m) array filtering
}
// After (v3.43.0): Roaring bitmap intersection
// After: Roaring bitmap intersection
async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
// 1. Fetch roaring bitmaps (integers, not UUIDs)
const bitmaps: RoaringBitmap32[] = []
@ -229,7 +229,7 @@ interface ZoneMap {
**Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field
#### Word Index (`__words__`) - v7.7.0
#### Word Index (`__words__`) -
```typescript
// Special field for text/keyword search
// Entity text content is tokenized and indexed as word hashes
@ -253,14 +253,14 @@ interface ZoneMap {
- **Lowercase normalization**: Case-insensitive matching
- **Automatic integration**: Words extracted via `extractIndexableFields()`
**Hybrid Search** (v7.7.0): Text results combined with vector results using Reciprocal Rank Fusion (RRF):
**Hybrid Search**: Text results combined with vector results using Reciprocal Rank Fusion (RRF):
```typescript
// RRF formula: score(d) = sum(1 / (k + rank(d)))
// where k = 60 (standard constant)
// alpha = weight for semantic (0 = text only, 1 = semantic only)
```
### Query Algorithm (v3.42.0)
### Query Algorithm
**Exact Match Query**:
```typescript
@ -317,7 +317,7 @@ async getIdsForRange(field: string, min: any, max: any): Promise<string[]> {
- Adaptive chunking: ~50 values per chunk optimizes I/O
- Immediate flushing: No need for dirty tracking or batch writes
### Temporal Bucketing (v3.41.0)
### Temporal Bucketing
**Problem Solved**: High-cardinality timestamp fields created massive file pollution.
- Example: 575 entities with unique timestamps → 358,407 index files (98.7% pollution!)
@ -396,7 +396,7 @@ const DEFAULT_EXCLUDE_FIELDS = [
]
```
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of v3.41.0 - they are indexed with automatic bucketing.
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
## 2. HNSWIndex - Vector Similarity Search
@ -735,7 +735,7 @@ async stats(): Promise<Statistics> {
}
```
### 5. Index Rebuilding (v5.7.7: Lazy Loading Support)
### 5. Index Rebuilding (Lazy Loading Support)
**Two modes of index loading:**
@ -762,7 +762,7 @@ async init(): Promise<void> {
}
```
#### Mode 2: Lazy Loading on First Query (v5.7.7+)
#### Mode 2: Lazy Loading on First Query
```typescript
// When disableAutoRebuild: true
@ -918,7 +918,7 @@ All indexes scale gracefully:
- [Performance Guide](../PERFORMANCE.md) - Performance tuning
- [Overview](./overview.md) - High-level architecture
## Summary: Index Hierarchy (v5.7.7)
## Summary: Index Hierarchy
### Level 1: Main Indexes (3)
All have rebuild() methods and are covered by lazy loading:
@ -933,7 +933,7 @@ Automatically managed by parent rebuild():
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
### Lazy Loading (v5.7.7)
### Lazy Loading
- **Mode 1**: Auto-rebuild on init() (default)
- **Mode 2**: Lazy rebuild on first query (when `disableAutoRebuild: true`)
- **Concurrency-safe**: Mutex prevents duplicate rebuilds

8
docs/architecture/initialization-and-rebuild.md
View file

@ -15,7 +15,7 @@ This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, Grap
| **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
| **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
#### MetadataIndex Persistence Details (v4.2.1+)
#### MetadataIndex Persistence Details
The MetadataIndex now persists two components:
@ -129,7 +129,7 @@ async init(): Promise<void> {
- 100-2000ms: One-time rebuild to create indices
- Total: ~1-3 seconds (one time only)
#### Mode 2: Lazy Loading on First Query (v5.7.7+)
#### Mode 2: Lazy Loading on First Query
When `disableAutoRebuild: true`, indexes remain empty after init() and rebuild on first query:
@ -350,7 +350,7 @@ public async rebuild(options?: {
**Performance Impact**: 200-600x speedup (5 minutes → 500ms for 10K entities)
**Correct Pattern** (v3.45.0):
**Correct Pattern**:
```typescript
// Load ALL nouns ONCE (not 31 times!)
while (hasMore) {
@ -392,7 +392,7 @@ while (hasMore) {
```typescript
// src/utils/metadataIndex.ts (lines 202-216)
async init(): Promise<void> {
// STEP 1: Load field registry to discover persisted indices (v4.2.1)
// STEP 1: Load field registry to discover persisted indices
// This is THE KEY FIX - O(1) discovery of existing indices
await this.loadFieldRegistry()

382
docs/architecture/storage-architecture.md
View file

@ -1,40 +1,40 @@
# Storage Architecture (v4.0.0)
# Storage Architecture
> **Updated for v4.0.0**: Metadata/vector separation, UUID-based sharding, lifecycle management
> **Updated**: Metadata/vector separation, UUID-based sharding, lifecycle management
## Storage Structure
### v4.0.0 Architecture: Metadata/Vector Separation
### Architecture: Metadata/Vector Separation
In v4.0.0, entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
```
brainy-data/
├── _system/ # System metadata (not sharded)
│ ├── statistics.json # Performance metrics
│ ├── __metadata_field_index__*.json # Field indexes
│ └── __metadata_sorted_index__*.json # Sorted indexes
├── _system/ # System metadata (not sharded)
│ ├── statistics.json # Performance metrics
│ ├── __metadata_field_index__*.json # Field indexes
│ └── __metadata_sorted_index__*.json # Sorted indexes
│
├── entities/
│ ├── nouns/
│ │ ├── vectors/ # HNSW graph data (sharded by UUID)
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
│ │ │ │ ├── 00123456-....json # Vector + HNSW connections
│ │ │ │ └── 00abcdef-....json
│ │ │ ├── 01/ ... ff/ # 256 shards total
│ │ │
│ │ └── metadata/ # Business data (sharded by UUID)
│ │ ├── 00/
│ │ │ ├── 00123456-....json # Entity metadata only
│ │ │ └── 00abcdef-....json
│ │ ├── 01/ ... ff/
│ │
│ └── verbs/
│ ├── vectors/ # Relationship vectors (sharded)
│ │ ├── 00/ ... ff/
│ │
│ └── metadata/ # Relationship data (sharded)
│ ├── 00/ ... ff/
│ ├── nouns/
│ │ ├── vectors/ # HNSW graph data (sharded by UUID)
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
│ │ │ │ ├── 00123456-....json # Vector + HNSW connections
│ │ │ │ └── 00abcdef-....json
│ │ │ ├── 01/ ... ff/ # 256 shards total
│ │ │
│ │ └── metadata/ # Business data (sharded by UUID)
│ │ ├── 00/
│ │ │ ├── 00123456-....json # Entity metadata only
│ │ │ └── 00abcdef-....json
│ │ ├── 01/ ... ff/
│ │
│ └── verbs/
│ ├── vectors/ # Relationship vectors (sharded)
│ │ ├── 00/ ... ff/
│ │
│ └── metadata/ # Relationship data (sharded)
│ ├── 00/ ... ff/
```
### Why Split Metadata and Vectors?
@ -50,9 +50,9 @@ brainy-data/
**How it works:**
```typescript
const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
const shard = uuid.substring(0, 2) // "3f"
const shard = uuid.substring(0, 2) // "3f"
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
```
@ -64,103 +64,103 @@ const shard = uuid.substring(0, 2) // "3f"
## Storage Adapters
Brainy provides multiple storage adapters with identical APIs and v4.0.0 production features:
Brainy provides multiple storage adapters with identical APIs and production features:
### FileSystem Storage (Node.js)
```typescript
const brain = new Brainy({
storage: {
type: 'filesystem',
path: './data',
compression: true // v4.0.0: Gzip compression (60-80% space savings)
}
storage: {
type: 'filesystem',
path: './data',
compression: true // Gzip compression (60-80% space savings)
}
})
```
- **Use case**: Server applications, CLI tools
- **Performance**: Direct file I/O with optional compression
- **Persistence**: Permanent on disk
- **v4.0.0 Features**:
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
- **Batch Delete**: Efficient bulk deletion with retries
- **UUID Sharding**: Automatic 256-shard distribution
- **Features**:
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
- **Batch Delete**: Efficient bulk deletion with retries
- **UUID Sharding**: Automatic 256-shard distribution
### S3 Compatible Storage (AWS, MinIO, R2)
```typescript
const brain = new Brainy({
storage: {
type: 's3',
bucket: 'my-brainy-data',
region: 'us-east-1',
credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}
}
storage: {
type: 's3',
bucket: 'my-brainy-data',
region: 'us-east-1',
credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}
}
})
```
- **Use case**: Distributed applications, cloud deployments
- **Performance**: Network dependent, with intelligent caching
- **Persistence**: Cloud storage durability (99.999999999%)
- **v4.0.0 Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
- **Batch Delete**: Efficient bulk deletion (1000 objects per request)
- **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!)
- **Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
- **Batch Delete**: Efficient bulk deletion (1000 objects per request)
- **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!)
### Google Cloud Storage (GCS)
```typescript
const brain = new Brainy({
storage: {
type: 'gcs',
bucketName: 'my-brainy-data',
keyFilename: './service-account.json' // Or use ADC
}
storage: {
type: 'gcs',
bucketName: 'my-brainy-data',
keyFilename: './service-account.json' // Or use ADC
}
})
```
- **Use case**: Google Cloud deployments
- **Performance**: Global CDN with edge caching
- **Persistence**: 99.999999999% durability
- **v4.0.0 Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
- **Autoclass**: Intelligent automatic tier optimization
- **Batch Delete**: Efficient bulk operations
- **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!)
- **Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
- **Autoclass**: Intelligent automatic tier optimization
- **Batch Delete**: Efficient bulk operations
- **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!)
### Azure Blob Storage
```typescript
const brain = new Brainy({
storage: {
type: 'azure',
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
}
storage: {
type: 'azure',
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
}
})
```
- **Use case**: Azure cloud deployments
- **Performance**: Global replication with CDN
- **Persistence**: LRS, ZRS, GRS, RA-GRS options
- **v4.0.0 Features**:
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
- **Lifecycle Policies**: Automatic tier transitions and deletions
- **Batch Delete**: BlobBatchClient for efficient bulk operations
- **Batch Tier Changes**: Move thousands of blobs efficiently
- **Archive Rehydration**: Smart rehydration with priority options
- **Features**:
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
- **Lifecycle Policies**: Automatic tier transitions and deletions
- **Batch Delete**: BlobBatchClient for efficient bulk operations
- **Batch Tier Changes**: Move thousands of blobs efficiently
- **Archive Rehydration**: Smart rehydration with priority options
### Origin Private File System (Browser)
```typescript
const brain = new Brainy({
storage: {
type: 'opfs'
}
storage: {
type: 'opfs'
}
})
```
- **Use case**: Browser applications, PWAs
- **Performance**: Near-native file system speed
- **Persistence**: Permanent in browser (with quota limits)
- **v4.0.0 Features**:
- **Quota Monitoring**: Real-time quota tracking and warnings
- **Batch Delete**: Efficient bulk deletion
- **Storage Status**: Detailed usage/available reporting
- **Features**:
- **Quota Monitoring**: Real-time quota tracking and warnings
- **Batch Delete**: Efficient bulk deletion
- **Storage Status**: Detailed usage/available reporting
## Metadata Indexing System
@ -170,12 +170,12 @@ Tracks all unique values for each field:
```json
// __metadata_field_index__field_category.json
{
"values": {
"technology": 45,
"science": 32,
"business": 28
},
"lastUpdated": 1699564234567
"values": {
"technology": 45,
"science": 32,
"business": 28
},
"lastUpdated": 1699564234567
}
```
@ -185,11 +185,11 @@ Maps field+value combinations to entity IDs:
```json
// __metadata_index__category_technology_chunk0.json
{
"field": "category",
"value": "technology",
"ids": ["uuid1", "uuid2", "uuid3", ...],
"chunk": 0,
"total": 45
"field": "category",
"value": "technology",
"ids": ["uuid1", "uuid2", "uuid3", ...],
"chunk": 0,
"total": 45
}
```
@ -207,14 +207,14 @@ High-performance deduplication system for streaming data:
```json
// __entity_registry__.json
{
"mappings": {
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
},
"stats": {
"totalMappings": 10000,
"lastSync": 1699564234567
}
"mappings": {
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
},
"stats": {
"totalMappings": 10000,
"lastSync": 1699564234567
}
}
```
@ -229,14 +229,14 @@ Ensures durability and enables recovery:
```json
{
"timestamp": 1699564234567,
"operation": "add",
"data": {
"id": "550e8400-e29b-41d4-a716-446655440000",
"content": "...",
"metadata": {}
},
"checksum": "sha256:..."
"timestamp": 1699564234567,
"operation": "add",
"data": {
"id": "550e8400-e29b-41d4-a716-446655440000",
"content": "...",
"metadata": {}
},
"checksum": "sha256:..."
}
```
@ -244,7 +244,7 @@ Ensures durability and enables recovery:
2. Replay operations from last checkpoint
3. Verify checksums for integrity
## Storage Optimization (v4.0.0)
## Storage Optimization
### 1. Lifecycle Policies (Cloud Storage)
@ -253,48 +253,48 @@ Ensures durability and enables recovery:
```typescript
// S3: Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days
{ days: 90, storageClass: 'GLACIER' }, // Archive after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year
]
}]
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days
{ days: 90, storageClass: 'GLACIER' }, // Archive after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year
]
}]
})
// GCS: Set lifecycle policy
await storage.setLifecyclePolicy({
rules: [{
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 365 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
rules: [{
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 365 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
})
// Azure: Set lifecycle policy
await storage.setLifecyclePolicy({
rules: [{
name: 'archiveOldData',
enabled: true,
type: 'Lifecycle',
definition: {
filters: { blobTypes: ['blockBlob'] },
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}]
rules: [{
name: 'archiveOldData',
enabled: true,
type: 'Lifecycle',
definition: {
filters: { blobTypes: ['blockBlob'] },
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}]
})
```
@ -327,7 +327,7 @@ await storage.enableIntelligentTiering('entities/', 'auto-optimize')
```typescript
// Enable GCS Autoclass
await storage.enableAutoclass({
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
})
// Benefits:
@ -342,11 +342,11 @@ await storage.enableAutoclass({
```typescript
// Enable gzip compression for local storage
const brain = new Brainy({
storage: {
type: 'filesystem',
path: './data',
compression: true // 60-80% space savings
}
storage: {
type: 'filesystem',
path: './data',
compression: true // 60-80% space savings
}
})
// Performance impact:
@ -359,11 +359,11 @@ const brain = new Brainy({
### 5. Batch Operations
```typescript
// v4.0.0: Efficient batch delete
// Efficient batch delete
await storage.batchDelete([
'entities/nouns/vectors/00/00123456-....json',
'entities/nouns/metadata/00/00123456-....json',
// ... up to 1000 objects
'entities/nouns/vectors/00/00123456-....json',
'entities/nouns/metadata/00/00123456-....json',
// ... up to 1000 objects
])
// Benefits:
@ -375,9 +375,9 @@ await storage.batchDelete([
// Batch writes for performance
await brain.addBatch([
{ content: "item1", metadata: {} },
{ content: "item2", metadata: {} },
{ content: "item3", metadata: {} }
{ content: "item1", metadata: {} },
{ content: "item2", metadata: {} },
{ content: "item3", metadata: {} }
])
// Single transaction, optimized I/O
```
@ -390,14 +390,14 @@ const status = await storage.getStorageStatus()
console.log(status)
// {
// type: 'opfs',
// available: true,
// details: {
// usage: 45829120, // 43.7 MB used
// quota: 536870912, // 512 MB available
// usagePercent: 8.5,
// quotaExceeded: false
// }
// type: 'opfs',
// available: true,
// details: {
// usage: 45829120, // 43.7 MB used
// quota: 536870912, // 512 MB available
// usagePercent: 8.5,
// quotaExceeded: false
// }
// }
// Proactive quota management:
@ -410,14 +410,14 @@ console.log(status)
```typescript
// Change blob tier for cost optimization
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings)
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings)
// Batch tier changes (efficient)
await storage.batchChangeTier([blob1, blob2, blob3], 'Cool')
// Rehydrate from Archive when needed
await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
```
### 8. Caching Strategy
@ -425,15 +425,15 @@ await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
```typescript
// Configure caching per storage type
const brain = new Brainy({
storage: {
type: 'filesystem',
cache: {
enabled: true,
maxSize: 1000, // Maximum cached items
ttl: 300000, // 5 minutes
strategy: 'lru' // Least recently used
}
}
storage: {
type: 'filesystem',
cache: {
enabled: true,
maxSize: 1000, // Maximum cached items
ttl: 300000, // 5 minutes
strategy: 'lru' // Least recently used
}
}
})
```
@ -443,8 +443,8 @@ const brain = new Brainy({
```typescript
// Automatic locking for write operations
await brain.storage.withLock('resource-id', async () => {
// Exclusive access to resource
await brain.storage.saveNoun(id, data)
// Exclusive access to resource
await brain.storage.saveNoun(id, data)
})
```
@ -459,9 +459,9 @@ await brain.storage.withLock('resource-id', async () => {
```typescript
// Export entire database
const backup = await brain.export({
format: 'json',
includeVectors: true,
includeIndexes: false
format: 'json',
includeVectors: true,
includeIndexes: false
})
```
@ -469,8 +469,8 @@ const backup = await brain.export({
```typescript
// Import from backup
await brain.import(backup, {
mode: 'merge', // or 'replace'
validateSchema: true
mode: 'merge', // or 'replace'
validateSchema: true
})
```
@ -514,15 +514,15 @@ await newBrain.import(data)
const stats = await brain.storage.getStatistics()
console.log(stats)
// {
// totalSize: 1048576,
// entityCount: 1000,
// indexSize: 204800,
// walSize: 10240,
// cacheHitRate: 0.85
// totalSize: 1048576,
// entityCount: 1000,
// indexSize: 204800,
// walSize: 10240,
// cacheHitRate: 0.85
// }
```
## Best Practices (v4.0.0)
## Best Practices
### Choose the Right Adapter
1. **Development**: FileSystem with compression (local persistence, small storage footprint)
@ -537,7 +537,7 @@ console.log(stats)
4. **Archival**: Cloud storage with lifecycle policies (96% cost savings!)
5. **Large-scale**: Metadata/vector separation + UUID sharding + lifecycle policies
### v4.0.0 Cost Optimization
### Cost Optimization
1. **Enable lifecycle policies** for cloud storage (automated cost reduction)
2. **Use Intelligent-Tiering (S3)** or Autoclass (GCS) for automatic optimization
3. **Enable compression** for FileSystem storage (60-80% space savings)
@ -547,7 +547,7 @@ console.log(stats)
**Example Cost Savings (500TB dataset):**
- Without lifecycle policies: **$138,000/year**
- With v4.0.0 lifecycle policies: **$5,940/year**
- With lifecycle policies: **$5,940/year**
- **Savings: $132,060/year (96%)**
### Monitor and Maintain

248
docs/augmentations/COMPLETE-REFERENCE.md
View file

@ -1,8 +1,8 @@
# 🔌 Brainy v4.0.0 Augmentations Complete Reference
# 🔌 Brainy Augmentations Complete Reference
> **All augmentations that power Brainy's extensibility - with locations, usage, and examples**
>
> **⚠️ v4.0.0 Update**: Updated for metadata structure changes and billion-scale optimizations
> **⚠️ Update**: Updated for metadata structure changes and billion-scale optimizations
## Quick Start
@ -10,37 +10,37 @@
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy({
// Augmentations auto-configure based on environment
storage: 'auto', // Storage augmentation
cache: true, // Cache augmentation
index: true // Index augmentation
// Augmentations auto-configure based on environment
storage: 'auto', // Storage augmentation
cache: true, // Cache augmentation
index: true // Index augmentation
})
await brain.init() // Augmentations initialize automatically
await brain.init() // Augmentations initialize automatically
```
## v4.0.0 Augmentation Architecture
## Augmentation Architecture
### Key Improvements for Billion-Scale Performance
1. **Metadata/Vector Separation**: Augmentations now work with separated metadata and vectors
- Metadata stored separately from vector data
- 99.2% memory reduction for type tracking
- Two-file storage pattern for optimal I/O
- Metadata stored separately from vector data
- 99.2% memory reduction for type tracking
- Two-file storage pattern for optimal I/O
2. **Type System Enforcement**: All metadata requires type fields
- `NounMetadata` requires `noun: NounType`
- `VerbMetadata` requires `verb: VerbType`
- Type inference system available as public API
- `NounMetadata` requires `noun: NounType`
- `VerbMetadata` requires `verb: VerbType`
- Type inference system available as public API
3. **Storage Adapter Pattern**: Internal vs public method distinction
- `_methods`: Return pure structures (HNSWNoun, HNSWVerb)
- Public methods: Return WithMetadata types
- MetadataEnforcer Proxy ensures proper access
- `_methods`: Return pure structures (HNSWNoun, HNSWVerb)
- Public methods: Return WithMetadata types
- MetadataEnforcer Proxy ensures proper access
### What This Means for Augmentation Users
**✅ If you use built-in augmentations**: No changes needed! They're all updated for v4.0.0.
**✅ If you use built-in augmentations**: No changes needed! They're all updated.
**⚠️ If you created custom storage augmentations**: Update your storage adapter to:
- Wrap metadata with required `noun`/`verb` fields
@ -69,81 +69,81 @@ Augmentations are modular extensions that add functionality to Brainy without cl
## Storage Augmentations (8 total)
### MemoryStorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Auto-enabled**: When `storage: 'memory'` or in test environments
**Location**: `src/augmentations/storageAugmentations.ts`
**Auto-enabled**: When `storage: 'memory'` or in test environments
**Purpose**: In-memory storage for testing and temporary data
```typescript
const brain = new Brainy({ storage: 'memory' })
```
### FileSystemStorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Auto-enabled**: When `storage: 'filesystem'` or Node.js detected
### FileSystemStorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Auto-enabled**: When `storage: 'filesystem'` or Node.js detected
**Purpose**: Persistent file-based storage for Node.js applications
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', path: './data' }
const brain = new Brainy({
storage: { type: 'filesystem', path: './data' }
})
```
### OPFSStorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Auto-enabled**: When `storage: 'opfs'` or browser with OPFS support
**Location**: `src/augmentations/storageAugmentations.ts`
**Auto-enabled**: When `storage: 'opfs'` or browser with OPFS support
**Purpose**: Browser-based persistent storage using Origin Private File System
```typescript
const brain = new Brainy({ storage: 'opfs' })
```
### S3StorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Manual**: Requires AWS credentials
**Location**: `src/augmentations/storageAugmentations.ts`
**Manual**: Requires AWS credentials
**Purpose**: AWS S3-compatible cloud storage
```typescript
const brain = new Brainy({
storage: {
type: 's3',
bucket: 'my-bucket',
region: 'us-east-1',
credentials: { accessKeyId, secretAccessKey }
}
const brain = new Brainy({
storage: {
type: 's3',
bucket: 'my-bucket',
region: 'us-east-1',
credentials: { accessKeyId, secretAccessKey }
}
})
```
### R2StorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Manual**: Requires Cloudflare credentials
**Location**: `src/augmentations/storageAugmentations.ts`
**Manual**: Requires Cloudflare credentials
**Purpose**: Cloudflare R2 storage (S3-compatible)
```typescript
const brain = new Brainy({
storage: {
type: 'r2',
accountId: 'xxx',
bucket: 'my-bucket',
credentials: { accessKeyId, secretAccessKey }
}
const brain = new Brainy({
storage: {
type: 'r2',
accountId: 'xxx',
bucket: 'my-bucket',
credentials: { accessKeyId, secretAccessKey }
}
})
```
### GCSStorageAugmentation
**Location**: `src/augmentations/storageAugmentations.ts`
**Manual**: Requires Google Cloud credentials
**Location**: `src/augmentations/storageAugmentations.ts`
**Manual**: Requires Google Cloud credentials
**Purpose**: Google Cloud Storage
```typescript
const brain = new Brainy({
storage: {
type: 'gcs',
bucket: 'my-bucket',
projectId: 'my-project'
}
const brain = new Brainy({
storage: {
type: 'gcs',
bucket: 'my-bucket',
projectId: 'my-project'
}
})
```
### StorageAugmentation (base)
**Location**: `src/augmentations/storageAugmentation.ts`
**Location**: `src/augmentations/storageAugmentation.ts`
**Purpose**: Base class for custom storage implementations
### DynamicStorageAugmentation
**Location**: `src/augmentations/storageAugmentation.ts`
**Location**: `src/augmentations/storageAugmentation.ts`
**Purpose**: Runtime storage adapter switching
---
@ -151,17 +151,17 @@ const brain = new Brainy({
## Performance Augmentations (7 total)
### CacheAugmentation
**Location**: `src/augmentations/cacheAugmentation.ts`
**Auto-enabled**: When `cache: true` (default)
**Location**: `src/augmentations/cacheAugmentation.ts`
**Auto-enabled**: When `cache: true` (default)
**Purpose**: LRU cache for search results and frequent queries
```typescript
brain.clearCache() // Exposed via API
brain.getCacheStats() // Cache hit/miss statistics
brain.clearCache() // Exposed via API
brain.getCacheStats() // Cache hit/miss statistics
```
### IndexAugmentation
**Location**: `src/augmentations/indexAugmentation.ts`
**Auto-enabled**: When `index: true` (default)
**Location**: `src/augmentations/indexAugmentation.ts`
**Auto-enabled**: When `index: true` (default)
**Purpose**: Metadata indexing for O(1) field lookups
```typescript
brain.rebuildMetadataIndex() // Exposed via API
@ -170,41 +170,41 @@ brain.find({ where: { category: 'tech' } })
```
### MetricsAugmentation
**Location**: `src/augmentations/metricsAugmentation.ts`
**Auto-enabled**: Always active
**Location**: `src/augmentations/metricsAugmentation.ts`
**Auto-enabled**: Always active
**Purpose**: Performance metrics and statistics collection
```typescript
brain.getStats() // Comprehensive metrics
brain.getStats() // Comprehensive metrics
```
### MonitoringAugmentation
**Location**: `src/augmentations/monitoringAugmentation.ts`
**Manual**: Register for detailed monitoring
**Location**: `src/augmentations/monitoringAugmentation.ts`
**Manual**: Register for detailed monitoring
**Purpose**: Real-time performance monitoring and alerts
### BatchProcessingAugmentation
**Location**: `src/augmentations/batchProcessingAugmentation.ts`
**Auto-enabled**: For batch operations
**Location**: `src/augmentations/batchProcessingAugmentation.ts`
**Auto-enabled**: For batch operations
**Purpose**: Optimizes bulk add/update/delete operations
```typescript
brain.addNouns([...]) // Automatically batched
brain.addNouns([...]) // Automatically batched
```
### RequestDeduplicatorAugmentation
**Location**: `src/augmentations/requestDeduplicatorAugmentation.ts`
**Auto-enabled**: Always active
**Location**: `src/augmentations/requestDeduplicatorAugmentation.ts`
**Auto-enabled**: Always active
**Purpose**: Prevents duplicate concurrent operations
### ConnectionPoolAugmentation
**Location**: `src/augmentations/connectionPoolAugmentation.ts`
**Auto-enabled**: For network storage
**Location**: `src/augmentations/connectionPoolAugmentation.ts`
**Auto-enabled**: For network storage
**Purpose**: Connection pooling for cloud storage adapters
---
## Data Integrity Augmentations (3 total)
**Auto-enabled**: When `wal: true`
**Auto-enabled**: When `wal: true`
**Purpose**: Write-ahead logging for crash recovery
```typescript
const brain = new Brainy({ wal: true })
@ -212,8 +212,8 @@ const brain = new Brainy({ wal: true })
```
### EntityRegistryAugmentation
**Location**: `src/augmentations/entityRegistryAugmentation.ts`
**Auto-enabled**: For streaming operations
**Location**: `src/augmentations/entityRegistryAugmentation.ts`
**Auto-enabled**: For streaming operations
**Purpose**: High-speed deduplication for real-time data
```typescript
// Prevents duplicate entities in streaming scenarios
@ -221,8 +221,8 @@ brain.add(data) // Automatically deduplicated
```
### AutoRegisterEntitiesAugmentation
**Location**: `src/augmentations/entityRegistryAugmentation.ts`
**Manual**: For automatic entity discovery
**Location**: `src/augmentations/entityRegistryAugmentation.ts`
**Manual**: For automatic entity discovery
**Purpose**: Auto-discovers and registers entities from data
---
@ -230,20 +230,20 @@ brain.add(data) // Automatically deduplicated
## Intelligence Augmentations (2 total)
### NeuralImportAugmentation
**Location**: `src/augmentations/neuralImport.ts`
**Manual**: Via `brain.neuralImport()`
**Location**: `src/augmentations/neuralImport.ts`
**Manual**: Via `brain.neuralImport()`
**Purpose**: AI-powered smart data import
```typescript
const result = await brain.neuralImport(data, {
confidenceThreshold: 0.7,
autoApply: true
confidenceThreshold: 0.7,
autoApply: true
})
// Automatically detects entities and relationships
```
### IntelligentVerbScoringAugmentation
**Location**: `src/augmentations/intelligentVerbScoringAugmentation.ts`
**Auto-enabled**: When verbs are used
**Location**: `src/augmentations/intelligentVerbScoringAugmentation.ts`
**Auto-enabled**: When verbs are used
**Purpose**: ML-based relationship strength scoring
```typescript
brain.verbScoring.train(feedback)
@ -255,8 +255,8 @@ brain.verbScoring.getScore(verbId)
## Communication Augmentations (4 total)
### APIServerAugmentation
**Location**: `src/augmentations/apiServerAugmentation.ts`
**Manual**: For server deployments
**Location**: `src/augmentations/apiServerAugmentation.ts`
**Manual**: For server deployments
**Purpose**: REST/WebSocket/MCP API server
```typescript
const augmentation = new APIServerAugmentation()
@ -265,8 +265,8 @@ await brain.registerAugmentation(augmentation)
```
### WebSocketConduitAugmentation
**Location**: `src/augmentations/conduitAugmentations.ts`
**Manual**: For Brainy-to-Brainy sync
**Location**: `src/augmentations/conduitAugmentations.ts`
**Manual**: For Brainy-to-Brainy sync
**Purpose**: Real-time sync between Brainy instances
```typescript
const conduit = new WebSocketConduitAugmentation()
@ -274,13 +274,13 @@ await conduit.establishConnection('ws://other-brain')
```
### ServerSearchConduitAugmentation
**Location**: `src/augmentations/serverSearchAugmentations.ts`
**Manual**: For client-server search
**Location**: `src/augmentations/serverSearchAugmentations.ts`
**Manual**: For client-server search
**Purpose**: Search remote Brainy instance, cache locally
### ServerSearchActivationAugmentation
**Location**: `src/augmentations/serverSearchAugmentations.ts`
**Manual**: Works with ServerSearchConduit
**Location**: `src/augmentations/serverSearchAugmentations.ts`
**Manual**: Works with ServerSearchConduit
**Purpose**: Triggers and manages server search operations
---
@ -288,18 +288,18 @@ await conduit.establishConnection('ws://other-brain')
## External Integration (2 total)
### SynapseAugmentation (base)
**Location**: `src/augmentations/synapseAugmentation.ts`
**Location**: `src/augmentations/synapseAugmentation.ts`
**Purpose**: Base class for external platform integrations
```typescript
// Example: NotionSynapse, SlackSynapse, etc.
class NotionSynapse extends SynapseAugmentation {
async fetchData() { /* Notion API calls */ }
async pushData() { /* Sync to Notion */ }
async fetchData() { /* Notion API calls */ }
async pushData() { /* Sync to Notion */ }
}
```
### ExampleFileSystemSynapse
**Location**: `src/augmentations/synapseAugmentation.ts`
**Location**: `src/augmentations/synapseAugmentation.ts`
**Purpose**: Example implementation for file system sync
---
@ -309,11 +309,11 @@ class NotionSynapse extends SynapseAugmentation {
### Auto-Configuration
```typescript
const brain = new Brainy({
// These auto-register augmentations:
storage: 'auto', // Storage augmentation
cache: true, // Cache augmentation
index: true, // Index augmentation
metrics: true // Metrics augmentation
// These auto-register augmentations:
storage: 'auto', // Storage augmentation
cache: true, // Cache augmentation
index: true, // Index augmentation
metrics: true // Metrics augmentation
})
```
@ -333,29 +333,29 @@ await brain.init()
import { BaseAugmentation } from '@soulcraft/brainy'
class MyAugmentation extends BaseAugmentation {
readonly name = 'my-augmentation'
readonly timing = 'after' // before | after | both
readonly operations = ['addNoun', 'search'] // Which ops to hook
readonly priority = 10 // Execution order (lower = earlier)
protected async onInit(): Promise<void> {
// Initialize your augmentation
}
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'addNoun') {
console.log('Noun added:', params)
}
}
protected async onShutdown(): Promise<void> {
// Cleanup
}
readonly name = 'my-augmentation'
readonly timing = 'after' // before | after | both
readonly operations = ['addNoun', 'search'] // Which ops to hook
readonly priority = 10 // Execution order (lower = earlier)
protected async onInit(): Promise<void> {
// Initialize your augmentation
}
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'addNoun') {
console.log('Noun added:', params)
}
}
protected async onShutdown(): Promise<void> {
// Cleanup
}
}
```

530
docs/augmentations/DEVELOPER-GUIDE.md
View file

@ -1,10 +1,10 @@
# 🛠️ Brainy Augmentation Developer Guide
> **How to create, test, and use augmentations in Brainy v4.0.0**
> **How to create, test, and use augmentations in Brainy**
>
> **⚠️ v4.0.0 Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
> **⚠️ Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
## v4.0.0 Migration Guide
## Migration Guide
### What Changed?
@ -25,22 +25,22 @@
```typescript
// ❌ v3.x
const verb = {
type: 'relatedTo',
sourceId: 'a',
targetId: 'b'
type: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
if (verb.type === 'relatedTo') { ... }
// ✅ v4.0.0
// ✅ Current
const verb = {
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
const metadata: VerbMetadata = {
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
if (verb.verb === 'relatedTo') { ... }
```
@ -51,40 +51,40 @@ if (verb.verb === 'relatedTo') { ... }
import { BaseAugmentation, BrainyAugmentation, AugmentationContext } from '@soulcraft/brainy'
export class MyFirstAugmentation extends BaseAugmentation {
readonly name = 'my-first-augmentation'
readonly timing = 'after' as const // When to run: before | after | both
readonly operations = ['add'] as const // Which operations to hook
readonly priority = 10 // Execution order (lower = first)
readonly name = 'my-first-augmentation'
readonly timing = 'after' as const // When to run: before | after | both
readonly operations = ['add'] as const // Which operations to hook
readonly priority = 10 // Execution order (lower = first)
protected async onInit(): Promise<void> {
// Initialize your augmentation
console.log('MyFirstAugmentation initialized!')
}
protected async onInit(): Promise<void> {
// Initialize your augmentation
console.log('MyFirstAugmentation initialized!')
}
async execute<T = any>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'add') {
console.log('Noun added:', params.noun)
async execute<T = any>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'add') {
console.log('Noun added:', params.noun)
// v4.0.0: Access metadata correctly
if (params.noun?.metadata) {
console.log('Noun type:', params.noun.metadata.noun) // Required field
}
// Access metadata correctly
if (params.noun?.metadata) {
console.log('Noun type:', params.noun.metadata.noun) // Required field
}
// You can access the brain instance
const stats = await context?.brain.getStats()
console.log('Total nouns:', stats.totalNouns)
}
}
// You can access the brain instance
const stats = await context?.brain.getStats()
console.log('Total nouns:', stats.totalNouns)
}
}
protected async onShutdown(): Promise<void> {
// Cleanup
console.log('MyFirstAugmentation shutting down')
}
protected async onShutdown(): Promise<void> {
// Cleanup
console.log('MyFirstAugmentation shutting down')
}
}
```
@ -113,23 +113,23 @@ await brain.add('Hello World')
### 1. Registration Phase
```typescript
const aug = new MyAugmentation()
brain.augmentations.register(aug) // Before brain.init()!
brain.augmentations.register(aug) // Before brain.init()!
```
### 2. Initialization Phase
```typescript
await brain.init() // Calls aug.initialize() internally
await brain.init() // Calls aug.initialize() internally
// Your onInit() method runs here
```
### 3. Execution Phase
```typescript
await brain.add('data') // Your execute() method runs
await brain.add('data') // Your execute() method runs
```
### 4. Shutdown Phase
```typescript
await brain.shutdown() // Your onShutdown() method runs
await brain.shutdown() // Your onShutdown() method runs
```
---
@ -139,52 +139,52 @@ await brain.shutdown() // Your onShutdown() method runs
### `before` - Modify Input
```typescript
class ValidationAugmentation extends BaseAugmentation {
readonly timing = 'before' as const
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Validate and/or modify params
if (!params.content) {
throw new Error('Content required')
}
// Return modified params
return { ...params, validated: true }
}
}
readonly timing = 'before' as const
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Validate and/or modify params
if (!params.content) {
throw new Error('Content required')
}
// Return modified params
return { ...params, validated: true }
}
}
}
```
### `after` - React to Results
```typescript
class LoggingAugmentation extends BaseAugmentation {
readonly timing = 'after' as const
async execute<T>(operation: string, params: any): Promise<void> {
if (operation === 'search') {
console.log(`Search for "${params.query}" returned ${params.result.length} results`)
}
// Don't return anything - just observe
}
readonly timing = 'after' as const
async execute<T>(operation: string, params: any): Promise<void> {
if (operation === 'search') {
console.log(`Search for "${params.query}" returned ${params.result.length} results`)
}
// Don't return anything - just observe
}
}
```
### `both` - Before AND After
```typescript
class TimingAugmentation extends BaseAugmentation {
readonly timing = 'both' as const
private startTime?: number
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
if (!this.startTime) {
// Before execution
this.startTime = Date.now()
} else {
// After execution
const duration = Date.now() - this.startTime
console.log(`${operation} took ${duration}ms`)
this.startTime = undefined
}
}
readonly timing = 'both' as const
private startTime?: number
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
if (!this.startTime) {
// Before execution
this.startTime = Date.now()
} else {
// After execution
const duration = Date.now() - this.startTime
console.log(`${operation} took ${duration}ms`)
this.startTime = undefined
}
}
}
```
@ -195,28 +195,28 @@ class TimingAugmentation extends BaseAugmentation {
### Core Operations You Can Hook
```typescript
readonly operations = [
'add', // Adding data
'update', // Updating data
'delete', // Deleting data
'get', // Retrieving data
'search', // Searching
'find', // Triple Intelligence queries
'relate', // Adding relationships
'unrelate', // Removing relationships
'clear', // Clearing data
'all' // Hook ALL operations
'add', // Adding data
'update', // Updating data
'delete', // Deleting data
'get', // Retrieving data
'search', // Searching
'find', // Triple Intelligence queries
'relate', // Adding relationships
'unrelate', // Removing relationships
'clear', // Clearing data
'all' // Hook ALL operations
] as const
```
### Example: Multi-Operation Hook
```typescript
class AuditAugmentation extends BaseAugmentation {
readonly operations = ['add', 'update', 'delete'] as const
async execute<T>(operation: string, params: any): Promise<void> {
// Log all data modifications
await this.logToAuditTrail(operation, params)
}
readonly operations = ['add', 'update', 'delete'] as const
async execute<T>(operation: string, params: any): Promise<void> {
// Log all data modifications
await this.logToAuditTrail(operation, params)
}
}
```
@ -226,26 +226,26 @@ class AuditAugmentation extends BaseAugmentation {
```typescript
class ContextAwareAugmentation extends BaseAugmentation {
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<void> {
// Access the brain instance
const brain = context?.brain
if (!brain) return
// Use any brain method
const stats = await brain.getStats()
const size = await brain.size()
const results = await brain.search('query')
// Access other augmentations
const cache = brain.augmentations.get('cache')
if (cache) {
await cache.clear()
}
}
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<void> {
// Access the brain instance
const brain = context?.brain
if (!brain) return
// Use any brain method
const stats = await brain.getStats()
const size = await brain.size()
const results = await brain.search('query')
// Access other augmentations
const cache = brain.augmentations.get('cache')
if (cache) {
await cache.clear()
}
}
}
```
@ -256,92 +256,92 @@ class ContextAwareAugmentation extends BaseAugmentation {
### 1. Backup Augmentation
```typescript
class BackupAugmentation extends BaseAugmentation {
readonly name = 'backup'
readonly timing = 'after' as const
readonly operations = ['add', 'update', 'delete'] as const
readonly priority = 5
private changes = 0
private readonly backupThreshold = 100
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
this.changes++
if (this.changes >= this.backupThreshold) {
await this.performBackup(context?.brain)
this.changes = 0
}
}
private async performBackup(brain?: any): Promise<void> {
if (!brain) return
// Create instant COW snapshot
const snapshotName = `backup-${Date.now()}`
await brain.fork(snapshotName)
console.log(`Automatic snapshot created: ${snapshotName}`)
}
readonly name = 'backup'
readonly timing = 'after' as const
readonly operations = ['add', 'update', 'delete'] as const
readonly priority = 5
private changes = 0
private readonly backupThreshold = 100
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
this.changes++
if (this.changes >= this.backupThreshold) {
await this.performBackup(context?.brain)
this.changes = 0
}
}
private async performBackup(brain?: any): Promise<void> {
if (!brain) return
// Create instant COW snapshot
const snapshotName = `backup-${Date.now()}`
await brain.fork(snapshotName)
console.log(`Automatic snapshot created: ${snapshotName}`)
}
}
```
### 2. Rate Limiting Augmentation
```typescript
class RateLimitAugmentation extends BaseAugmentation {
readonly name = 'rate-limit'
readonly timing = 'before' as const
readonly operations = ['search', 'find'] as const
readonly priority = 100 // High priority - run first
private requests = new Map<string, number[]>()
private readonly limit = 100 // 100 requests
private readonly window = 60000 // per minute
async execute<T>(operation: string, params: any): Promise<void> {
const now = Date.now()
const key = params.userId || 'anonymous'
// Get request timestamps
const timestamps = this.requests.get(key) || []
// Remove old timestamps
const recent = timestamps.filter(t => now - t < this.window)
// Check limit
if (recent.length >= this.limit) {
throw new Error('Rate limit exceeded')
}
// Add current request
recent.push(now)
this.requests.set(key, recent)
}
readonly name = 'rate-limit'
readonly timing = 'before' as const
readonly operations = ['search', 'find'] as const
readonly priority = 100 // High priority - run first
private requests = new Map<string, number[]>()
private readonly limit = 100 // 100 requests
private readonly window = 60000 // per minute
async execute<T>(operation: string, params: any): Promise<void> {
const now = Date.now()
const key = params.userId || 'anonymous'
// Get request timestamps
const timestamps = this.requests.get(key) || []
// Remove old timestamps
const recent = timestamps.filter(t => now - t < this.window)
// Check limit
if (recent.length >= this.limit) {
throw new Error('Rate limit exceeded')
}
// Add current request
recent.push(now)
this.requests.set(key, recent)
}
}
```
### 3. Encryption Augmentation
```typescript
class EncryptionAugmentation extends BaseAugmentation {
readonly name = 'encryption'
readonly timing = 'both' as const
readonly operations = ['add', 'get'] as const
readonly priority = 90 // Run early
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Encrypt before storing
if (params.metadata?.sensitive) {
params.content = await this.encrypt(params.content)
params.encrypted = true
}
return params
}
readonly name = 'encryption'
readonly timing = 'both' as const
readonly operations = ['add', 'get'] as const
readonly priority = 90 // Run early
if (operation === 'get' && params.result?.encrypted) {
// Decrypt after retrieval
params.result.content = await this.decrypt(params.result.content)
delete params.result.encrypted
return params.result
}
}
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Encrypt before storing
if (params.metadata?.sensitive) {
params.content = await this.encrypt(params.content)
params.encrypted = true
}
return params
}
if (operation === 'get' && params.result?.encrypted) {
// Decrypt after retrieval
params.result.content = await this.decrypt(params.result.content)
delete params.result.encrypted
return params.result
}
}
}
```
@ -355,26 +355,26 @@ import { Brainy } from '@soulcraft/brainy'
import { MyAugmentation } from './my-augmentation'
describe('MyAugmentation', () => {
it('should hook into addNoun', async () => {
const brain = new Brainy({ storage: 'memory' })
const aug = new MyAugmentation()
// Spy on the execute method
const executeSpy = vi.spyOn(aug, 'execute')
brain.augmentations.register(aug)
await brain.init()
// Trigger the augmentation
await brain.add('test data')
it('should hook into addNoun', async () => {
const brain = new Brainy({ storage: 'memory' })
const aug = new MyAugmentation()
// Verify it was called
expect(executeSpy).toHaveBeenCalledWith(
'add',
expect.objectContaining({ content: 'test data' }),
expect.any(Object)
)
})
// Spy on the execute method
const executeSpy = vi.spyOn(aug, 'execute')
brain.augmentations.register(aug)
await brain.init()
// Trigger the augmentation
await brain.add('test data')
// Verify it was called
expect(executeSpy).toHaveBeenCalledWith(
'add',
expect.objectContaining({ content: 'test data' }),
expect.any(Object)
)
})
})
```
@ -391,61 +391,61 @@ describe('MyAugmentation', () => {
```typescript
// Priority guidelines
100: Critical (auth, rate limiting)
50: Important (validation, transformation)
10: Normal (logging, metrics)
1: Optional (debugging, tracing)
50: Important (validation, transformation)
10: Normal (logging, metrics)
1: Optional (debugging, tracing)
```
### 3. Handle Errors Gracefully
```typescript
async execute<T>(operation: string, params: any): Promise<void> {
try {
await this.riskyOperation()
} catch (error) {
// Log but don't break the main operation
console.error(`Augmentation error in ${this.name}:`, error)
// Optionally report to monitoring
this.reportError(error)
}
try {
await this.riskyOperation()
} catch (error) {
// Log but don't break the main operation
console.error(`Augmentation error in ${this.name}:`, error)
// Optionally report to monitoring
this.reportError(error)
}
}
```
### 4. Be Performance Conscious
```typescript
class CachedAugmentation extends BaseAugmentation {
private cache = new Map<string, any>()
async execute<T>(operation: string, params: any): Promise<any> {
const key = this.getCacheKey(params)
// Check cache first
if (this.cache.has(key)) {
return this.cache.get(key)
}
// Expensive operation
const result = await this.expensiveOperation(params)
this.cache.set(key, result)
return result
}
private cache = new Map<string, any>()
async execute<T>(operation: string, params: any): Promise<any> {
const key = this.getCacheKey(params)
// Check cache first
if (this.cache.has(key)) {
return this.cache.get(key)
}
// Expensive operation
const result = await this.expensiveOperation(params)
this.cache.set(key, result)
return result
}
}
```
### 5. Clean Up Resources
```typescript
protected async onShutdown(): Promise<void> {
// Close connections
await this.connection?.close()
// Clear intervals
clearInterval(this.interval)
// Flush buffers
await this.flush()
// Clear caches
this.cache.clear()
// Close connections
await this.connection?.close()
// Clear intervals
clearInterval(this.interval)
// Flush buffers
await this.flush()
// Clear caches
this.cache.clear()
}
```
@ -457,10 +457,10 @@ protected async onShutdown(): Promise<void> {
```
my-augmentation/
├── src/
│ └── index.ts # Your augmentation
├── dist/ # Built output
│ └── index.ts # Your augmentation
├── dist/ # Built output
├── tests/
│ └── augmentation.test.ts
│ └── augmentation.test.ts
├── package.json
├── tsconfig.json
└── README.md
@ -469,21 +469,21 @@ my-augmentation/
### package.json
```json
{
"name": "@mycompany/brainy-custom-augmentation",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"keywords": ["brainy-augmentation"],
"peerDependencies": {
"@soulcraft/brainy": ">=2.0.0"
},
"brainy": {
"type": "augmentation",
"class": "CustomAugmentation",
"timing": "after",
"operations": ["add"],
"priority": 10
}
"name": "@mycompany/brainy-custom-augmentation",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"keywords": ["brainy-augmentation"],
"peerDependencies": {
"@soulcraft/brainy": ">=2.0.0"
},
"brainy": {
"type": "augmentation",
"class": "CustomAugmentation",
"timing": "after",
"operations": ["add"],
"priority": 10
}
}
```
@ -492,7 +492,7 @@ my-augmentation/
# Coming in 2.1+
npm run build
npm test
brainy publish # Publishes to brain-cloud registry
brainy publish # Publishes to brain-cloud registry
```
---

782
docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md
View file

@ -1,8 +1,8 @@
# Brainy v4.0.0 Cloud Deployment Guide
# Brainy Cloud Deployment Guide
This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy v4.0.0 codebase.
This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy codebase.
## 🆕 v4.0.0 Production Features
## 🆕 Production Features
**Cost Optimization at Scale:**
- **Lifecycle Policies**: Automatic tier transitions for 96% cost savings
@ -12,7 +12,7 @@ This guide provides production-ready deployment configurations for Brainy using
**Example Impact (500TB dataset):**
- Before: $138,000/year
- With v4.0.0 lifecycle policies: $5,940/year
- With lifecycle policies: $5,940/year
- **Savings: $132,060/year (96%)**
See the [Cost Optimization](#cost-optimization-v40) section below for implementation details.
@ -43,53 +43,53 @@ let brain
let handler
exports.handler = async (event) => {
if (!brain) {
const storage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com',
region: process.env.AWS_REGION,
bucket: process.env.BRAINY_BUCKET,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
prefix: 'brainy-data/'
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
auth: {
required: true,
apiKeys: [process.env.API_KEY]
}
}
}]
})
await brain.init()
// Get the universal handler from the augmentation
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// Convert Lambda event to Request
const url = `https://${event.requestContext.domainName}${event.rawPath}`
const request = new Request(url, {
method: event.requestContext.http.method,
headers: event.headers,
body: event.body
})
// Use the universal handler
const response = await handler(request)
return {
statusCode: response.status,
headers: Object.fromEntries(response.headers),
body: await response.text()
}
if (!brain) {
const storage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com',
region: process.env.AWS_REGION,
bucket: process.env.BRAINY_BUCKET,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
prefix: 'brainy-data/'
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
auth: {
required: true,
apiKeys: [process.env.API_KEY]
}
}
}]
})
await brain.init()
// Get the universal handler from the augmentation
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// Convert Lambda event to Request
const url = `https://${event.requestContext.domainName}${event.rawPath}`
const request = new Request(url, {
method: event.requestContext.http.method,
headers: event.headers,
body: event.body
})
// Use the universal handler
const response = await handler(request)
return {
statusCode: response.status,
headers: Object.fromEntries(response.headers),
body: await response.text()
}
}
```
@ -104,36 +104,36 @@ let brain
let handler
exports.brainyAPI = async (req, res) => {
if (!brain) {
const storage = new S3CompatibleStorage({
endpoint: 'storage.googleapis.com',
bucket: process.env.GCS_BUCKET,
accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY,
prefix: 'brainy/',
forcePathStyle: false,
region: 'US'
})
brain = new Brainy({ storage })
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// Convert Express req/res to Request/Response
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: JSON.stringify(req.body)
})
const response = await handler(request)
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
if (!brain) {
const storage = new S3CompatibleStorage({
endpoint: 'storage.googleapis.com',
bucket: process.env.GCS_BUCKET,
accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY,
prefix: 'brainy/',
forcePathStyle: false,
region: 'US'
})
brain = new Brainy({ storage })
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// Convert Express req/res to Request/Response
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: JSON.stringify(req.body)
})
const response = await handler(request)
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
}
```
@ -156,39 +156,39 @@ let brain
let handler
async function initBrainy() {
// Google Cloud Storage is S3-compatible
const storage = new S3CompatibleStorage({
endpoint: 'storage.googleapis.com',
bucket: process.env.GCS_BUCKET || 'brainy-data',
accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY,
prefix: 'brainy/',
forcePathStyle: false,
region: process.env.GCS_REGION || 'US'
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
port: PORT,
cors: {
origin: process.env.CORS_ORIGIN || '*'
},
auth: {
required: process.env.AUTH_REQUIRED === 'true',
apiKeys: process.env.API_KEY ? [process.env.API_KEY] : []
}
}
}]
})
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
// Google Cloud Storage is S3-compatible
const storage = new S3CompatibleStorage({
endpoint: 'storage.googleapis.com',
bucket: process.env.GCS_BUCKET || 'brainy-data',
accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY,
prefix: 'brainy/',
forcePathStyle: false,
region: process.env.GCS_REGION || 'US'
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
port: PORT,
cors: {
origin: process.env.CORS_ORIGIN || '*'
},
auth: {
required: process.env.AUTH_REQUIRED === 'true',
apiKeys: process.env.API_KEY ? [process.env.API_KEY] : []
}
}
}]
})
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// Initialize on startup
@ -196,26 +196,26 @@ await initBrainy()
// Health check endpoint
app.get('/health', (req, res) => {
res.json({ status: 'healthy', service: 'brainy-api' })
res.json({ status: 'healthy', service: 'brainy-api' })
})
// Universal handler for all API routes
app.use('*', async (req, res) => {
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined
})
const response = await handler(request)
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined
})
const response = await handler(request)
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
})
app.listen(PORT, () => {
console.log(`Brainy API running on port ${PORT}`)
console.log(`Brainy API running on port ${PORT}`)
})
```
@ -237,43 +237,43 @@ CMD ["node", "server.js"]
```yaml
# cloudbuild.yaml
steps:
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.']
# Push to Container Registry
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA']
# Deploy to Cloud Run
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud
args:
- 'run'
- 'deploy'
- 'brainy-api'
- '--image'
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
- '--region'
- 'us-central1'
- '--platform'
- 'managed'
- '--allow-unauthenticated'
- '--set-env-vars'
- 'GCS_BUCKET=brainy-storage,GCS_REGION=US'
- '--set-secrets'
- 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest'
- '--memory'
- '2Gi'
- '--cpu'
- '2'
- '--max-instances'
- '100'
- '--min-instances'
- '0'
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.']
# Push to Container Registry
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA']
# Deploy to Cloud Run
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud
args:
- 'run'
- 'deploy'
- 'brainy-api'
- '--image'
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
- '--region'
- 'us-central1'
- '--platform'
- 'managed'
- '--allow-unauthenticated'
- '--set-env-vars'
- 'GCS_BUCKET=brainy-storage,GCS_REGION=US'
- '--set-secrets'
- 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest'
- '--memory'
- '2Gi'
- '--cpu'
- '2'
- '--max-instances'
- '100'
- '--min-instances'
- '0'
images:
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
```
**Deploy with gcloud CLI:**
@ -283,12 +283,12 @@ gcloud builds submit --config cloudbuild.yaml
# Or deploy directly
gcloud run deploy brainy-api \
--source . \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--set-env-vars GCS_BUCKET=brainy-storage \
--set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest"
--source . \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--set-env-vars GCS_BUCKET=brainy-storage \
--set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest"
```
**Create GCS Bucket with S3-compatible access:**
@ -312,37 +312,37 @@ echo -n "YOUR_SECRET_KEY" | gcloud secrets create gcs-secret-key --data-file=-
```javascript
// index.js
module.exports = async function (context, req) {
const { Brainy } = require('@soulcraft/brainy')
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
const storage = new S3CompatibleStorage({
endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`,
bucket: 'brainy-data',
accessKeyId: process.env.AZURE_STORAGE_ACCOUNT,
secretAccessKey: process.env.AZURE_STORAGE_KEY,
prefix: 'entities/',
forcePathStyle: false
})
const brain = new Brainy({ storage })
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
const handler = apiAugmentation.createUniversalHandler()
const request = new Request(`https://${context.req.headers.host}${context.req.url}`, {
method: context.req.method,
headers: context.req.headers,
body: JSON.stringify(context.req.body)
})
const response = await handler(request)
context.res = {
status: response.status,
headers: Object.fromEntries(response.headers),
body: await response.text()
}
const { Brainy } = require('@soulcraft/brainy')
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
const storage = new S3CompatibleStorage({
endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`,
bucket: 'brainy-data',
accessKeyId: process.env.AZURE_STORAGE_ACCOUNT,
secretAccessKey: process.env.AZURE_STORAGE_KEY,
prefix: 'entities/',
forcePathStyle: false
})
const brain = new Brainy({ storage })
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
const handler = apiAugmentation.createUniversalHandler()
const request = new Request(`https://${context.req.headers.host}${context.req.url}`, {
method: context.req.method,
headers: context.req.headers,
body: JSON.stringify(context.req.body)
})
const response = await handler(request)
context.res = {
status: response.status,
headers: Object.fromEntries(response.headers),
body: await response.text()
}
}
```
@ -356,37 +356,37 @@ import { R2Storage } from '@soulcraft/brainy/storage' // Alias for S3CompatibleS
let handler
export default {
async fetch(request, env, ctx) {
if (!handler) {
const storage = new R2Storage({
endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'brainy-data',
accessKeyId: env.R2_ACCESS_KEY_ID,
secretAccessKey: env.R2_SECRET_ACCESS_KEY,
region: 'auto',
forcePathStyle: true
})
const brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
cors: { origin: '*' }
}
}]
})
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// The handler works directly with Request/Response!
return handler(request)
}
async fetch(request, env, ctx) {
if (!handler) {
const storage = new R2Storage({
endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'brainy-data',
accessKeyId: env.R2_ACCESS_KEY_ID,
secretAccessKey: env.R2_SECRET_ACCESS_KEY,
region: 'auto',
forcePathStyle: true
})
const brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
cors: { origin: '*' }
}
}]
})
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// The handler works directly with Request/Response!
return handler(request)
}
}
```
@ -418,52 +418,52 @@ import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
let handler
export const config = {
runtime: 'edge',
runtime: 'edge',
}
export default async (request) => {
if (!handler) {
const storage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com',
region: 'us-east-1',
bucket: process.env.S3_BUCKET,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
})
const brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: { enabled: true }
}]
})
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
return handler(request)
if (!handler) {
const storage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com',
region: 'us-east-1',
bucket: process.env.S3_BUCKET,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
})
const brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: { enabled: true }
}]
})
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
return handler(request)
}
```
```json
// vercel.json
{
"functions": {
"api/brainy.js": {
"maxDuration": 30,
"memory": 1024
}
},
"rewrites": [
{
"source": "/api/:path*",
"destination": "/api/brainy"
}
]
"functions": {
"api/brainy.js": {
"maxDuration": 30,
"memory": 1024
}
},
"rewrites": [
{
"source": "/api/:path*",
"destination": "/api/brainy"
}
]
}
```
@ -482,51 +482,51 @@ let brain
let handler
async function init() {
const storage = new S3CompatibleStorage({
endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com',
region: process.env.S3_REGION || 'us-east-1',
bucket: process.env.S3_BUCKET,
accessKeyId: process.env.S3_ACCESS_KEY,
secretAccessKey: process.env.S3_SECRET_KEY,
prefix: 'brainy/'
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
port: PORT
}
}]
})
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
const storage = new S3CompatibleStorage({
endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com',
region: process.env.S3_REGION || 'us-east-1',
bucket: process.env.S3_BUCKET,
accessKeyId: process.env.S3_ACCESS_KEY,
secretAccessKey: process.env.S3_SECRET_KEY,
prefix: 'brainy/'
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
port: PORT
}
}]
})
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
await init()
// Universal handler for all routes
app.use('*', async (req, res) => {
const request = new Request(`http://localhost${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: JSON.stringify(req.body)
})
const response = await handler(request)
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
const request = new Request(`http://localhost${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: JSON.stringify(req.body)
})
const response = await handler(request)
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
})
app.listen(PORT, () => {
console.log(`Brainy API running on port ${PORT}`)
console.log(`Brainy API running on port ${PORT}`)
})
```
@ -552,26 +552,26 @@ restartPolicyMaxRetries = 3
```yaml
# render.yaml
services:
- type: web
name: brainy-api
runtime: node
buildCommand: npm install
startCommand: node server.js
envVars:
- key: S3_BUCKET
value: brainy-data
- key: S3_ENDPOINT
value: s3.amazonaws.com
- key: S3_REGION
value: us-east-1
- key: S3_ACCESS_KEY
sync: false
- key: S3_SECRET_KEY
sync: false
- key: API_KEY
generateValue: true
healthCheckPath: /health
autoDeploy: true
- type: web
name: brainy-api
runtime: node
buildCommand: npm install
startCommand: node server.js
envVars:
- key: S3_BUCKET
value: brainy-data
- key: S3_ENDPOINT
value: s3.amazonaws.com
- key: S3_REGION
value: us-east-1
- key: S3_ACCESS_KEY
sync: false
- key: S3_SECRET_KEY
sync: false
- key: API_KEY
generateValue: true
healthCheckPath: /health
autoDeploy: true
```
### Deno Deploy
@ -582,19 +582,19 @@ import { Brainy } from "npm:@soulcraft/brainy"
import { S3CompatibleStorage } from "npm:@soulcraft/brainy/storage"
const storage = new S3CompatibleStorage({
endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com",
bucket: Deno.env.get("S3_BUCKET")!,
accessKeyId: Deno.env.get("S3_ACCESS_KEY")!,
secretAccessKey: Deno.env.get("S3_SECRET_KEY")!,
region: "auto"
endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com",
bucket: Deno.env.get("S3_BUCKET")!,
accessKeyId: Deno.env.get("S3_ACCESS_KEY")!,
secretAccessKey: Deno.env.get("S3_SECRET_KEY")!,
region: "auto"
})
const brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: { enabled: true }
}]
storage,
augmentations: [{
name: 'api-server',
config: { enabled: true }
}]
})
await brain.init()
@ -653,29 +653,29 @@ RATE_LIMIT_MAX=100
```javascript
// REST API Client
const response = await fetch('https://your-api.com/api/brainy/add', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY'
},
body: JSON.stringify({
data: 'Your content here',
metadata: { type: 'document' }
})
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY'
},
body: JSON.stringify({
data: 'Your content here',
metadata: { type: 'document' }
})
})
const { id } = await response.json()
// Search
const searchResponse = await fetch('https://your-api.com/api/brainy/find', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY'
},
body: JSON.stringify({
query: 'neural networks'
})
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY'
},
body: JSON.stringify({
query: 'neural networks'
})
})
const results = await searchResponse.json()
@ -683,14 +683,14 @@ const results = await searchResponse.json()
// WebSocket Client
const ws = new WebSocket('wss://your-api.com/ws')
ws.onopen = () => {
ws.send(JSON.stringify({
type: 'subscribe',
pattern: 'technology'
}))
ws.send(JSON.stringify({
type: 'subscribe',
pattern: 'technology'
}))
}
ws.onmessage = (event) => {
const update = JSON.parse(event.data)
console.log('Real-time update:', update)
const update = JSON.parse(event.data)
console.log('Real-time update:', update)
}
```
@ -700,13 +700,13 @@ S3CompatibleStorage constructor parameters (verified from source):
```javascript
{
endpoint: string, // Required (e.g., 's3.amazonaws.com')
bucket: string, // Required
accessKeyId: string, // Required
secretAccessKey: string, // Required
region?: string, // Optional (default: 'us-east-1')
prefix?: string, // Optional (e.g., 'brainy/')
forcePathStyle?: boolean, // Optional (needed for some S3-compatible services)
endpoint: string, // Required (e.g., 's3.amazonaws.com')
bucket: string, // Required
accessKeyId: string, // Required
secretAccessKey: string, // Required
region?: string, // Optional (default: 'us-east-1')
prefix?: string, // Optional (e.g., 'brainy/')
forcePathStyle?: boolean, // Optional (needed for some S3-compatible services)
}
```
@ -727,7 +727,7 @@ S3CompatibleStorage constructor parameters (verified from source):
4. **Implement rate limiting** to prevent abuse
5. **Configure CORS** appropriately for your use case
## Cost Optimization (v4.0.0)
## Cost Optimization
### Enable Lifecycle Policies
@ -738,16 +738,16 @@ const storage = brain.storage
// Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // $0.0125/GB after 30 days
{ days: 90, storageClass: 'GLACIER' }, // $0.004/GB after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // $0.00099/GB after 1 year
]
}]
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // $0.0125/GB after 30 days
{ days: 90, storageClass: 'GLACIER' }, // $0.004/GB after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // $0.00099/GB after 1 year
]
}]
})
// Or enable Intelligent-Tiering for hands-off optimization
@ -765,14 +765,14 @@ await storage.enableIntelligentTiering('entities/', 'auto-optimize')
**Efficient bulk deletions:**
```javascript
// v4.0.0: Batch delete (1000 objects per request)
// Batch delete (1000 objects per request)
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => [
`entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
`entities/nouns/metadata/${id.substring(0, 2)}/${id}.json`
`entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
`entities/nouns/metadata/${id.substring(0, 2)}/${id}.json`
])
await storage.batchDelete(paths) // Much faster than individual deletes
await storage.batchDelete(paths) // Much faster than individual deletes
```
### Enable Compression (FileSystem)
@ -780,8 +780,8 @@ await storage.batchDelete(paths) // Much faster than individual deletes
**For local/server deployments:**
```javascript
const storage = new FileSystemStorage({
path: './data',
compression: true // 60-80% space savings
path: './data',
compression: true // 60-80% space savings
})
const brain = new Brainy({ storage })
@ -793,8 +793,8 @@ const brain = new Brainy({ storage })
2. **Use S3CompatibleStorage** for cloud deployments (better scalability)
3. **Enable the cache augmentation** for frequently accessed data
4. **Configure appropriate memory limits** for your runtime
5. **v4.0.0**: Enable lifecycle policies to reduce storage costs by 96%
6. **v4.0.0**: Use batch operations for cleanup tasks
5. Enable lifecycle policies to reduce storage costs by 96%
6. Use batch operations for cleanup tasks
## Troubleshooting
@ -810,15 +810,15 @@ const brain = new Brainy({ storage })
Enable debug logging by setting:
```javascript
const brain = new Brainy({
storage,
debug: true,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
verbose: true
}
}]
storage,
debug: true,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
verbose: true
}
}]
})
```

480
docs/deployment/aws-deployment.md
View file

@ -18,24 +18,24 @@ const { Brainy } = require('@soulcraft/brainy')
let brain
exports.handler = async (event) => {
// Brainy auto-detects Lambda environment and configures accordingly
if (!brain) {
brain = new Brainy() // Zero config - auto-adapts to Lambda
await brain.init()
}
const { method, ...params } = JSON.parse(event.body)
switch(method) {
case 'add':
const id = await brain.add(params)
return { statusCode: 200, body: JSON.stringify({ id }) }
case 'find':
const results = await brain.find(params)
return { statusCode: 200, body: JSON.stringify({ results }) }
default:
return { statusCode: 400, body: 'Unknown method' }
}
// Brainy auto-detects Lambda environment and configures accordingly
if (!brain) {
brain = new Brainy() // Zero config - auto-adapts to Lambda
await brain.init()
}
const { method, ...params } = JSON.parse(event.body)
switch(method) {
case 'add':
const id = await brain.add(params)
return { statusCode: 200, body: JSON.stringify({ id }) }
case 'find':
const results = await brain.find(params)
return { statusCode: 200, body: JSON.stringify({ results }) }
default:
return { statusCode: 400, body: 'Unknown method' }
}
}
EOF
@ -56,26 +56,26 @@ docker push $ECR_URI/brainy:latest
# Deploy with minimal ECS task definition
cat > task-definition.json << 'EOF'
{
"family": "brainy",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"cpu": "256",
"memory": "512",
"containerDefinitions": [{
"name": "brainy",
"image": "$ECR_URI/brainy:latest",
"environment": [
{"name": "NODE_ENV", "value": "production"}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/brainy",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
}
}]
"family": "brainy",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"cpu": "256",
"memory": "512",
"containerDefinitions": [{
"name": "brainy",
"image": "$ECR_URI/brainy:latest",
"environment": [
{"name": "NODE_ENV", "value": "production"}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/brainy",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
}
}]
}
EOF
@ -138,14 +138,14 @@ const brain = new Brainy()
```javascript
const brain = new Brainy({
storage: {
type: 's3',
options: {
bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket
region: process.env.AWS_REGION || 'auto', // 'auto' detects region
// IAM role provides credentials automatically
}
}
storage: {
type: 's3',
options: {
bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket
region: process.env.AWS_REGION || 'auto', // 'auto' detects region
// IAM role provides credentials automatically
}
}
})
```
@ -156,23 +156,23 @@ const brain = new Brainy({
```yaml
# Auto-scaling policy
Resources:
AutoScalingTarget:
Type: AWS::ApplicationAutoScaling::ScalableTarget
Properties:
ServiceNamespace: ecs
ResourceId: service/default/brainy
ScalableDimension: ecs:service:DesiredCount
MinCapacity: 2
MaxCapacity: 100
AutoScalingPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyType: TargetTrackingScaling
TargetTrackingScalingPolicyConfiguration:
TargetValue: 70.0
PredefinedMetricSpecification:
PredefinedMetricType: ECSServiceAverageCPUUtilization
AutoScalingTarget:
Type: AWS::ApplicationAutoScaling::ScalableTarget
Properties:
ServiceNamespace: ecs
ResourceId: service/default/brainy
ScalableDimension: ecs:service:DesiredCount
MinCapacity: 2
MaxCapacity: 100
AutoScalingPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyType: TargetTrackingScaling
TargetTrackingScalingPolicyConfiguration:
TargetValue: 70.0
PredefinedMetricSpecification:
PredefinedMetricType: ECSServiceAverageCPUUtilization
```
### 2. Vertical Scaling
@ -189,10 +189,10 @@ Brainy automatically adapts to available memory:
```javascript
// Brainy automatically handles multi-AZ with S3
const brain = new Brainy({
distributed: {
enabled: true, // Auto-enables with S3 storage
coordinationMethod: 'auto' // Uses S3 for coordination
}
distributed: {
enabled: true, // Auto-enables with S3 storage
coordinationMethod: 'auto' // Uses S3 for coordination
}
})
```
@ -201,17 +201,17 @@ const brain = new Brainy({
```bash
# Application Load Balancer with health checks
aws elbv2 create-load-balancer \
--name brainy-alb \
--subnets subnet-xxx subnet-yyy \
--security-groups sg-xxx
--name brainy-alb \
--subnets subnet-xxx subnet-yyy \
--security-groups sg-xxx
aws elbv2 create-target-group \
--name brainy-targets \
--protocol HTTP \
--port 3000 \
--vpc-id vpc-xxx \
--health-check-path /health \
--health-check-interval-seconds 30
--name brainy-targets \
--protocol HTTP \
--port 3000 \
--vpc-id vpc-xxx \
--health-check-path /health \
--health-check-interval-seconds 30
```
## Monitoring & Observability
@ -233,16 +233,16 @@ Brainy automatically sends metrics when running on AWS:
```javascript
const brain = new Brainy({
monitoring: {
enabled: true,
customMetrics: {
namespace: 'Brainy/Production',
dimensions: [
{ Name: 'Environment', Value: 'production' },
{ Name: 'Service', Value: 'api' }
]
}
}
monitoring: {
enabled: true,
customMetrics: {
namespace: 'Brainy/Production',
dimensions: [
{ Name: 'Environment', Value: 'production' },
{ Name: 'Service', Value: 'api' }
]
}
}
})
```
@ -252,22 +252,22 @@ const brain = new Brainy({
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::brainy-*/*",
"arn:aws:s3:::brainy-*"
]
}
]
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::brainy-*/*",
"arn:aws:s3:::brainy-*"
]
}
]
}
```
@ -285,12 +285,12 @@ aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.2.0/24 --availability-z
```javascript
// Automatic encryption with S3
const brain = new Brainy({
storage: {
type: 's3',
options: {
encryption: 'auto' // Uses S3 SSE-S3 by default
}
}
storage: {
type: 's3',
options: {
encryption: 'auto' // Uses S3 SSE-S3 by default
}
}
})
```
@ -300,14 +300,14 @@ const brain = new Brainy({
```bash
aws ec2 request-spot-fleet --spot-fleet-request-config '{
"IamFleetRole": "arn:aws:iam::xxx:role/fleet-role",
"TargetCapacity": 2,
"SpotPrice": "0.05",
"LaunchSpecifications": [{
"ImageId": "ami-xxx",
"InstanceType": "t3.medium",
"UserData": "BASE64_ENCODED_STARTUP_SCRIPT"
}]
"IamFleetRole": "arn:aws:iam::xxx:role/fleet-role",
"TargetCapacity": 2,
"SpotPrice": "0.05",
"LaunchSpecifications": [{
"ImageId": "ami-xxx",
"InstanceType": "t3.medium",
"UserData": "BASE64_ENCODED_STARTUP_SCRIPT"
}]
}'
```
@ -316,12 +316,12 @@ aws ec2 request-spot-fleet --spot-fleet-request-config '{
```javascript
// Brainy automatically uses S3 Intelligent-Tiering
const brain = new Brainy({
storage: {
type: 's3',
options: {
storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization
}
}
storage: {
type: 's3',
options: {
storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization
}
}
})
```
@ -329,8 +329,8 @@ const brain = new Brainy({
```bash
aws lambda put-function-concurrency \
--function-name brainy-handler \
--reserved-concurrent-executions 10
--function-name brainy-handler \
--reserved-concurrent-executions 10
```
## Deployment Automation
@ -340,28 +340,28 @@ aws lambda put-function-concurrency \
```yaml
name: Deploy to AWS
on:
push:
branches: [main]
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Deploy to ECS
run: |
docker build -t brainy .
docker tag brainy:latest $ECR_URI/brainy:latest
docker push $ECR_URI/brainy:latest
aws ecs update-service --cluster default --service brainy --force-new-deployment
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Deploy to ECS
run: |
docker build -t brainy .
docker tag brainy:latest $ECR_URI/brainy:latest
docker push $ECR_URI/brainy:latest
aws ecs update-service --cluster default --service brainy --force-new-deployment
```
## Troubleshooting
@ -369,121 +369,121 @@ jobs:
### Common Issues
1. **Storage Auto-Detection Fails**
```javascript
// Explicitly specify storage
const brain = new Brainy({
storage: { type: 's3', options: { bucket: 'my-bucket' } }
})
```
```javascript
// Explicitly specify storage
const brain = new Brainy({
storage: { type: 's3', options: { bucket: 'my-bucket' } }
})
```
2. **Memory Issues**
```javascript
// Optimize for low memory
const brain = new Brainy({
cache: { maxSize: 100 }, // Reduce cache size
index: { M: 8 } // Reduce HNSW connections
})
```
```javascript
// Optimize for low memory
const brain = new Brainy({
cache: { maxSize: 100 }, // Reduce cache size
index: { M: 8 } // Reduce HNSW connections
})
```
3. **Cold Starts (Lambda)**
**v7.3.0+ Progressive Initialization (Zero-Config)**
**Progressive Initialization (Zero-Config)**
Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME)
and uses progressive initialization for <200ms cold starts:
Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME)
and uses progressive initialization for <200ms cold starts:
```javascript
// Zero-config - Brainy auto-detects Lambda
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'my-bucket',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}
}
})
await brain.init() // Returns in <200ms
```javascript
// Zero-config - Brainy auto-detects Lambda
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'my-bucket',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}
}
})
await brain.init() // Returns in <200ms
// First write validates bucket (lazy validation)
await brain.add('noun', { name: 'test' }) // Validates here
```
// First write validates bucket (lazy validation)
await brain.add('noun', { name: 'test' }) // Validates here
```
**Manual Override (if needed)**
```javascript
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'my-bucket',
// Force specific mode
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
}
}
})
```
**Manual Override (if needed)**
```javascript
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'my-bucket',
// Force specific mode
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
}
}
})
```
| Mode | Cold Start | Best For |
|------|------------|----------|
| `auto` (default) | <200ms in Lambda | Zero-config, auto-detects |
| `progressive` | <200ms always | Force fast init everywhere |
| `strict` | 100-500ms+ | Local dev, tests, debugging |
| Mode | Cold Start | Best For |
|------|------------|----------|
| `auto` (default) | <200ms in Lambda | Zero-config, auto-detects |
| `progressive` | <200ms always | Force fast init everywhere |
| `strict` | 100-500ms+ | Local dev, tests, debugging |
**Warm-up (Alternative)**
```javascript
exports.warmup = async () => {
if (!brain) {
brain = new Brainy({ warmup: true })
await brain.init()
}
}
```
**Warm-up (Alternative)**
```javascript
exports.warmup = async () => {
if (!brain) {
brain = new Brainy({ warmup: true })
await brain.init()
}
}
```
**Readiness Detection (v7.3.0+)**
**Readiness Detection**
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
```javascript
let brain
```javascript
let brain
exports.handler = async (event) => {
if (!brain) {
brain = new Brainy({ storage: { type: 's3', ... } })
brain.init() // Fire and forget
}
exports.handler = async (event) => {
if (!brain) {
brain = new Brainy({ storage: { type: 's3', ... } })
brain.init() // Fire and forget
}
// Wait for initialization to complete
await brain.ready
// Wait for initialization to complete
await brain.ready
// Now safe to use brain methods
const results = await brain.find({ query: event.queryStringParameters.q })
return { statusCode: 200, body: JSON.stringify(results) }
}
```
// Now safe to use brain methods
const results = await brain.find({ query: event.queryStringParameters.q })
return { statusCode: 200, body: JSON.stringify(results) }
}
```
For health checks, use `isFullyInitialized()` to verify all background tasks are complete:
For health checks, use `isFullyInitialized()` to verify all background tasks are complete:
```javascript
exports.healthCheck = async () => {
try {
await brain.ready
return {
statusCode: 200,
body: JSON.stringify({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
}
} catch (error) {
return {
statusCode: 503,
body: JSON.stringify({ status: 'initializing' })
}
}
}
```
```javascript
exports.healthCheck = async () => {
try {
await brain.ready
return {
statusCode: 200,
body: JSON.stringify({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
}
} catch (error) {
return {
statusCode: 503,
body: JSON.stringify({ status: 'initializing' })
}
}
}
```
## Production Checklist

648
docs/deployment/gcp-deployment.md
View file

@ -10,10 +10,10 @@ Deploy Brainy on GCP with automatic scaling, global distribution, and zero-confi
```bash
# Build and deploy with one command
gcloud run deploy brainy \
--source . \
--platform managed \
--region us-central1 \
--allow-unauthenticated
--source . \
--platform managed \
--region us-central1 \
--allow-unauthenticated
# Brainy auto-detects Cloud Run and configures:
# - Memory-optimized caching
@ -30,44 +30,44 @@ const { Brainy } = require('@soulcraft/brainy')
let brain
exports.brainyHandler = async (req, res) => {
// Zero-config - auto-adapts to Cloud Functions
if (!brain) {
brain = new Brainy() // Detects GCP environment automatically
await brain.init()
}
const { method, ...params } = req.body
try {
let result
switch(method) {
case 'add':
result = await brain.add(params)
break
case 'find':
result = await brain.find(params)
break
case 'relate':
result = await brain.relate(params)
break
default:
return res.status(400).json({ error: 'Unknown method' })
}
res.json({ result })
} catch (error) {
res.status(500).json({ error: error.message })
}
// Zero-config - auto-adapts to Cloud Functions
if (!brain) {
brain = new Brainy() // Detects GCP environment automatically
await brain.init()
}
const { method, ...params } = req.body
try {
let result
switch(method) {
case 'add':
result = await brain.add(params)
break
case 'find':
result = await brain.find(params)
break
case 'relate':
result = await brain.relate(params)
break
default:
return res.status(400).json({ error: 'Unknown method' })
}
res.json({ result })
} catch (error) {
res.status(500).json({ error: error.message })
}
}
```
Deploy:
```bash
gcloud functions deploy brainy \
--runtime nodejs20 \
--trigger-http \
--entry-point brainyHandler \
--memory 512MB \
--timeout 60s
--runtime nodejs20 \
--trigger-http \
--entry-point brainyHandler \
--memory 512MB \
--timeout 60s
```
### Option 3: Google Kubernetes Engine (GKE)
@ -75,7 +75,7 @@ gcloud functions deploy brainy \
```bash
# Create autopilot cluster (fully managed, zero-config)
gcloud container clusters create-auto brainy-cluster \
--region us-central1
--region us-central1
# Deploy using Cloud Build
gcloud builds submit --tag gcr.io/$PROJECT_ID/brainy
@ -85,39 +85,39 @@ kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: brainy
name: brainy
spec:
replicas: 3
selector:
matchLabels:
app: brainy
template:
metadata:
labels:
app: brainy
spec:
containers:
- name: brainy
image: gcr.io/$PROJECT_ID/brainy
resources:
requests:
memory: "512Mi"
cpu: "250m"
env:
- name: NODE_ENV
value: production
replicas: 3
selector:
matchLabels:
app: brainy
template:
metadata:
labels:
app: brainy
spec:
containers:
- name: brainy
image: gcr.io/$PROJECT_ID/brainy
resources:
requests:
memory: "512Mi"
cpu: "250m"
env:
- name: NODE_ENV
value: production
---
apiVersion: v1
kind: Service
metadata:
name: brainy-service
name: brainy-service
spec:
type: LoadBalancer
selector:
app: brainy
ports:
- port: 80
targetPort: 3000
type: LoadBalancer
selector:
app: brainy
ports:
- port: 80
targetPort: 3000
EOF
```
@ -139,14 +139,14 @@ const brain = new Brainy()
```javascript
const brain = new Brainy({
storage: {
type: 's3', // GCS is S3-compatible
options: {
endpoint: 'https://storage.googleapis.com',
bucket: process.env.GCS_BUCKET || 'auto', // Auto-creates bucket
// Uses Application Default Credentials automatically
}
}
storage: {
type: 's3', // GCS is S3-compatible
options: {
endpoint: 'https://storage.googleapis.com',
bucket: process.env.GCS_BUCKET || 'auto', // Auto-creates bucket
// Uses Application Default Credentials automatically
}
}
})
```
@ -154,13 +154,13 @@ const brain = new Brainy({
```javascript
const brain = new Brainy({
storage: {
type: 'firestore',
options: {
projectId: process.env.GCP_PROJECT || 'auto',
collection: 'brainy-data'
}
}
storage: {
type: 'firestore',
options: {
projectId: process.env.GCP_PROJECT || 'auto',
collection: 'brainy-data'
}
}
})
```
@ -173,24 +173,24 @@ const brain = new Brainy({
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: brainy
annotations:
run.googleapis.com/execution-environment: gen2
name: brainy
annotations:
run.googleapis.com/execution-environment: gen2
spec:
template:
metadata:
annotations:
autoscaling.knative.dev/minScale: "1"
autoscaling.knative.dev/maxScale: "1000"
autoscaling.knative.dev/target: "80"
spec:
containerConcurrency: 100
containers:
- image: gcr.io/PROJECT_ID/brainy
resources:
limits:
cpu: "2"
memory: "2Gi"
template:
metadata:
annotations:
autoscaling.knative.dev/minScale: "1"
autoscaling.knative.dev/maxScale: "1000"
autoscaling.knative.dev/target: "80"
spec:
containerConcurrency: 100
containers:
- image: gcr.io/PROJECT_ID/brainy
resources:
limits:
cpu: "2"
memory: "2Gi"
```
### 2. GKE Horizontal Pod Autoscaling
@ -199,27 +199,27 @@ spec:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: brainy-hpa
name: brainy-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: brainy
minReplicas: 3
maxReplicas: 100
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: brainy
minReplicas: 3
maxReplicas: 100
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
```
## Global Distribution
@ -229,11 +229,11 @@ spec:
```javascript
// Brainy automatically handles multi-region with GCS
const brain = new Brainy({
distributed: {
enabled: true,
regions: ['us-central1', 'europe-west1', 'asia-northeast1'],
replication: 'auto' // Automatic cross-region replication
}
distributed: {
enabled: true,
regions: ['us-central1', 'europe-west1', 'asia-northeast1'],
replication: 'auto' // Automatic cross-region replication
}
})
```
@ -242,14 +242,14 @@ const brain = new Brainy({
```bash
# Global load balancing with Traffic Director
gcloud compute backend-services create brainy-global \
--global \
--load-balancing-scheme=INTERNAL_SELF_MANAGED \
--protocol=HTTP2
--global \
--load-balancing-scheme=INTERNAL_SELF_MANAGED \
--protocol=HTTP2
gcloud compute backend-services add-backend brainy-global \
--global \
--network-endpoint-group=brainy-neg \
--network-endpoint-group-region=us-central1
--global \
--network-endpoint-group=brainy-neg \
--network-endpoint-group-region=us-central1
```
## Monitoring & Observability
@ -277,22 +277,22 @@ const { Monitoring } = require('@google-cloud/monitoring')
const monitoring = new Monitoring.MetricServiceClient()
const brain = new Brainy({
onMetric: async (metric) => {
// Send custom metrics to Cloud Monitoring
await monitoring.createTimeSeries({
name: monitoring.projectPath(projectId),
timeSeries: [{
metric: {
type: `custom.googleapis.com/brainy/${metric.name}`,
labels: metric.labels
},
points: [{
interval: { endTime: { seconds: Date.now() / 1000 } },
value: { doubleValue: metric.value }
}]
}]
})
}
onMetric: async (metric) => {
// Send custom metrics to Cloud Monitoring
await monitoring.createTimeSeries({
name: monitoring.projectPath(projectId),
timeSeries: [{
metric: {
type: `custom.googleapis.com/brainy/${metric.name}`,
labels: metric.labels
},
points: [{
interval: { endTime: { seconds: Date.now() / 1000 } },
value: { doubleValue: metric.value }
}]
}]
})
}
})
```
@ -300,10 +300,10 @@ const brain = new Brainy({
```javascript
const brain = new Brainy({
tracing: {
enabled: true,
sampleRate: 0.1 // Sample 10% of requests
}
tracing: {
enabled: true,
sampleRate: 0.1 // Sample 10% of requests
}
})
```
@ -316,9 +316,9 @@ const brain = new Brainy({
apiVersion: v1
kind: ServiceAccount
metadata:
name: brainy-sa
annotations:
iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com
name: brainy-sa
annotations:
iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com
```
### 2. Binary Authorization
@ -328,11 +328,11 @@ metadata:
apiVersion: binaryauthorization.grafeas.io/v1beta1
kind: Policy
metadata:
name: brainy-policy
name: brainy-policy
spec:
defaultAdmissionRule:
requireAttestationsBy:
- projects/PROJECT_ID/attestors/prod-attestor
defaultAdmissionRule:
requireAttestationsBy:
- projects/PROJECT_ID/attestors/prod-attestor
```
### 3. VPC Service Controls
@ -340,9 +340,9 @@ spec:
```bash
# Create VPC Service Perimeter
gcloud access-context-manager perimeters create brainy_perimeter \
--resources=projects/PROJECT_NUMBER \
--restricted-services=storage.googleapis.com \
--title="Brainy Security Perimeter"
--resources=projects/PROJECT_NUMBER \
--restricted-services=storage.googleapis.com \
--title="Brainy Security Perimeter"
```
## Cost Optimization
@ -354,16 +354,16 @@ gcloud access-context-manager perimeters create brainy_perimeter \
apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerNodePool
metadata:
name: brainy-preemptible-pool
name: brainy-preemptible-pool
spec:
clusterRef:
name: brainy-cluster
config:
preemptible: true
machineType: n2-standard-2
autoscaling:
minNodeCount: 1
maxNodeCount: 10
clusterRef:
name: brainy-cluster
config:
preemptible: true
machineType: n2-standard-2
autoscaling:
minNodeCount: 1
maxNodeCount: 10
```
### 2. Cloud CDN for Static Assets
@ -371,11 +371,11 @@ spec:
```bash
# Enable Cloud CDN for frequently accessed data
gcloud compute backend-buckets create brainy-assets \
--gcs-bucket-name=brainy-static
--gcs-bucket-name=brainy-static
gcloud compute backend-buckets update brainy-assets \
--enable-cdn \
--cache-mode=CACHE_ALL_STATIC
--enable-cdn \
--cache-mode=CACHE_ALL_STATIC
```
### 3. Committed Use Discounts
@ -383,8 +383,8 @@ gcloud compute backend-buckets update brainy-assets \
```bash
# Purchase committed use for predictable workloads
gcloud compute commitments create brainy-commitment \
--plan=TWELVE_MONTH \
--resources=vcpu=100,memory=400
--plan=TWELVE_MONTH \
--resources=vcpu=100,memory=400
```
## Deployment Automation
@ -394,28 +394,28 @@ gcloud compute commitments create brainy-commitment \
```yaml
# cloudbuild.yaml
steps:
# Build container
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.']
# Push to registry
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA']
# Deploy to Cloud Run
- name: 'gcr.io/cloud-builders/gcloud'
args:
- 'run'
- 'deploy'
- 'brainy'
- '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'
- '--region=us-central1'
- '--platform=managed'
# Build container
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.']
# Push to registry
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA']
# Deploy to Cloud Run
- name: 'gcr.io/cloud-builders/gcloud'
args:
- 'run'
- 'deploy'
- 'brainy'
- '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'
- '--region=us-central1'
- '--platform=managed'
# Trigger on push to main
trigger:
branch:
name: main
branch:
name: main
```
### Terraform Infrastructure
@ -423,40 +423,40 @@ trigger:
```hcl
# main.tf
resource "google_cloud_run_service" "brainy" {
name = "brainy"
location = "us-central1"
name = "brainy"
location = "us-central1"
template {
spec {
containers {
image = "gcr.io/${var.project_id}/brainy"
resources {
limits = {
cpu = "2"
memory = "2Gi"
}
}
env {
name = "NODE_ENV"
value = "production"
}
}
}
}
template {
spec {
containers {
image = "gcr.io/${var.project_id}/brainy"
traffic {
percent = 100
latest_revision = true
}
resources {
limits = {
cpu = "2"
memory = "2Gi"
}
}
env {
name = "NODE_ENV"
value = "production"
}
}
}
}
traffic {
percent = 100
latest_revision = true
}
}
resource "google_cloud_run_service_iam_member" "public" {
service = google_cloud_run_service.brainy.name
location = google_cloud_run_service.brainy.location
role = "roles/run.invoker"
member = "allUsers"
service = google_cloud_run_service.brainy.name
location = google_cloud_run_service.brainy.location
role = "roles/run.invoker"
member = "allUsers"
}
```
@ -467,13 +467,13 @@ resource "google_cloud_run_service_iam_member" "public" {
```javascript
// Brainy can use Memorystore for caching
const brain = new Brainy({
cache: {
type: 'redis',
options: {
host: process.env.REDIS_HOST || 'auto-detect',
port: 6379
}
}
cache: {
type: 'redis',
options: {
host: process.env.REDIS_HOST || 'auto-detect',
port: 6379
}
}
})
```
@ -481,13 +481,13 @@ const brain = new Brainy({
```javascript
const brain = new Brainy({
metadata: {
type: 'spanner',
options: {
instance: 'brainy-instance',
database: 'brainy-db'
}
}
metadata: {
type: 'spanner',
options: {
instance: 'brainy-instance',
database: 'brainy-db'
}
}
})
```
@ -496,116 +496,116 @@ const brain = new Brainy({
### Common Issues
1. **Quota Exceeded**
```bash
# Check quotas
gcloud compute project-info describe --project=$PROJECT_ID
# Request increase
gcloud compute project-info add-metadata \
--metadata google-compute-default-region=us-central1
```
```bash
# Check quotas
gcloud compute project-info describe --project=$PROJECT_ID
# Request increase
gcloud compute project-info add-metadata \
--metadata google-compute-default-region=us-central1
```
2. **Cold Starts**
**v7.3.0+ Progressive Initialization (Zero-Config)**
**Progressive Initialization (Zero-Config)**
Brainy automatically detects Cloud Run and Cloud Functions environments
and uses progressive initialization for <200ms cold starts:
Brainy automatically detects Cloud Run and Cloud Functions environments
and uses progressive initialization for <200ms cold starts:
```javascript
// Zero-config - Brainy auto-detects Cloud Run (K_SERVICE env var)
const brain = new Brainy({
storage: {
type: 'gcs',
gcsNativeStorage: { bucketName: 'my-bucket' }
}
})
await brain.init() // Returns in <200ms
```javascript
// Zero-config - Brainy auto-detects Cloud Run (K_SERVICE env var)
const brain = new Brainy({
storage: {
type: 'gcs',
gcsNativeStorage: { bucketName: 'my-bucket' }
}
})
await brain.init() // Returns in <200ms
// First write validates bucket (lazy validation)
await brain.add('noun', { name: 'test' }) // Validates here
```
// First write validates bucket (lazy validation)
await brain.add('noun', { name: 'test' }) // Validates here
```
**Manual Override (if needed)**
```javascript
const brain = new Brainy({
storage: {
type: 'gcs',
gcsNativeStorage: {
bucketName: 'my-bucket',
// Force specific mode
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
}
}
})
```
**Manual Override (if needed)**
```javascript
const brain = new Brainy({
storage: {
type: 'gcs',
gcsNativeStorage: {
bucketName: 'my-bucket',
// Force specific mode
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
}
}
})
```
| Mode | Cold Start | Best For |
|------|------------|----------|
| `auto` (default) | <200ms in cloud | Zero-config, auto-detects |
| `progressive` | <200ms always | Force fast init everywhere |
| `strict` | 100-500ms+ | Local dev, tests, debugging |
| Mode | Cold Start | Best For |
|------|------------|----------|
| `auto` (default) | <200ms in cloud | Zero-config, auto-detects |
| `progressive` | <200ms always | Force fast init everywhere |
| `strict` | 100-500ms+ | Local dev, tests, debugging |
**Keep Warm (Alternative)**
```javascript
// Keep minimum instances warm
const brain = new Brainy({
warmup: {
enabled: true,
interval: 60000 // Ping every minute
}
})
```
**Keep Warm (Alternative)**
```javascript
// Keep minimum instances warm
const brain = new Brainy({
warmup: {
enabled: true,
interval: 60000 // Ping every minute
}
})
```
**Readiness Detection (v7.3.0+)**
**Readiness Detection**
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
```javascript
let brain
```javascript
let brain
async function handleRequest(req, res) {
if (!brain) {
brain = new Brainy({ storage: { type: 'gcs', ... } })
brain.init() // Fire and forget
}
async function handleRequest(req, res) {
if (!brain) {
brain = new Brainy({ storage: { type: 'gcs', ... } })
brain.init() // Fire and forget
}
// Wait for initialization to complete
await brain.ready
// Wait for initialization to complete
await brain.ready
// Now safe to use brain methods
const results = await brain.find({ query: req.query.q })
res.json(results)
}
```
// Now safe to use brain methods
const results = await brain.find({ query: req.query.q })
res.json(results)
}
```
For Cloud Run health checks, use `isFullyInitialized()` to verify all background tasks are complete:
For Cloud Run health checks, use `isFullyInitialized()` to verify all background tasks are complete:
```javascript
// Health check endpoint for Cloud Run
app.get('/health', async (req, res) => {
try {
await brain.ready
res.json({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
} catch (error) {
res.status(503).json({ status: 'initializing' })
}
})
```
```javascript
// Health check endpoint for Cloud Run
app.get('/health', async (req, res) => {
try {
await brain.ready
res.json({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
} catch (error) {
res.status(503).json({ status: 'initializing' })
}
})
```
3. **Memory Pressure**
```javascript
// Optimize for GCP memory constraints
const brain = new Brainy({
memory: {
mode: 'aggressive', // Aggressive garbage collection
maxHeap: 0.8 // Use 80% of available memory
}
})
```
```javascript
// Optimize for GCP memory constraints
const brain = new Brainy({
memory: {
mode: 'aggressive', // Aggressive garbage collection
maxHeap: 0.8 // Use 80% of available memory
}
})
```
## Production Checklist

161
docs/features/instant-fork.md
View file

@ -42,10 +42,10 @@ await experiment.updateAll(riskyTransformation)
// Works? Great! Use the experimental branch.
// Failed? Just discard.
if (success) {
// Make experiment the new main branch
await brain.checkout('test-migration')
// Make experiment the new main branch
await brain.checkout('test-migration')
} else {
await experiment.destroy() // No harm done
await experiment.destroy() // No harm done
}
```
@ -61,25 +61,25 @@ if (success) {
Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking:
### Architecture (v5.0.0)
### Architecture
1. **HNSW Index COW** (The Performance Bottleneck):
- **Shallow Copy**: O(1) Map reference copying (~10ms for 1M+ nodes)
- **Lazy Deep Copy**: Nodes copied only when modified (`ensureCOW()`)
- **Write Isolation**: Fork modifications don't affect parent
- **Implementation**: `src/hnsw/hnswIndex.ts` lines 2088-2150
- **Shallow Copy**: O(1) Map reference copying (~10ms for 1M+ nodes)
- **Lazy Deep Copy**: Nodes copied only when modified (`ensureCOW()`)
- **Write Isolation**: Fork modifications don't affect parent
- **Implementation**: `src/hnsw/hnswIndex.ts` lines 2088-2150
2. **Metadata & Graph Indexes** (Fast Rebuild):
- **Rebuild from Storage**: < 500ms total for both indexes
- **Shared Storage**: Both indexes read from COW-enabled storage layer
- **Acceptable Overhead**: Fast enough not to need in-memory COW
- **Rebuild from Storage**: < 500ms total for both indexes
- **Shared Storage**: Both indexes read from COW-enabled storage layer
- **Acceptable Overhead**: Fast enough not to need in-memory COW
3. **Storage Layer** (Shared):
- **RefManager**: Manages branch references
- **BlobStorage**: Content-addressable with deduplication
- **All Adapters**: Memory, FileSystem, S3, R2, GCS, Azure, OPFS
- **RefManager**: Manages branch references
- **BlobStorage**: Content-addressable with deduplication
- **All Adapters**: Memory, FileSystem, S3, R2, GCS, Azure, OPFS
**Performance (v5.0.0)**:
**Performance**:
- **Fork time**: < 100ms @ 10K entities (MEASURED in tests)
- **Storage overhead**: 10-20% (shared blobs, only changed data duplicated)
- **Memory overhead**: 10-20% (shared HNSW nodes, only modified nodes duplicated)
@ -87,11 +87,11 @@ Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking:
**Technical Details**:
```typescript
// Shallow copy HNSW (instant)
clone.index.enableCOW(this.index) // O(1) Map reference copy
clone.index.enableCOW(this.index) // O(1) Map reference copy
// Fast rebuild small indexes from shared storage
clone.metadataIndex = new MetadataIndexManager(clone.storage) // <100ms
clone.graphIndex = new GraphAdjacencyIndex(clone.storage) // <500ms
clone.metadataIndex = new MetadataIndexManager(clone.storage) // <100ms
clone.graphIndex = new GraphAdjacencyIndex(clone.storage) // <500ms
```
---
@ -128,11 +128,11 @@ await fork.add({ noun: 'user', data: { name: 'Charlie' } })
// All APIs work
const users = await fork.find({ noun: 'user' })
console.log(users.length) // 3 (Alice, Bob, Charlie)
console.log(users.length) // 3 (Alice, Bob, Charlie)
// Original brain unchanged
const originalUsers = await brain.find({ noun: 'user' })
console.log(originalUsers.length) // 2 (Alice, Bob)
console.log(originalUsers.length) // 2 (Alice, Bob)
```
### 3. Discard When Done
@ -165,30 +165,30 @@ const migration = await brain.fork('migration-test')
// Run migration on fork
const users = await migration.find({ noun: 'user' })
for (const user of users) {
await migration.update(user.id, {
email: user.data.email.toLowerCase(), // Normalize emails
verified: user.data.verified ?? false // Add missing field
})
await migration.update(user.id, {
email: user.data.email.toLowerCase(), // Normalize emails
verified: user.data.verified ?? false // Add missing field
})
}
// Validate migration
const allValid = (await migration.find({ noun: 'user' }))
.every(u => u.data.email === u.data.email.toLowerCase())
.every(u => u.data.email === u.data.email.toLowerCase())
if (allValid) {
console.log('✅ Migration safe! Apply changes to production brain')
// Apply validated migration to main brain
const users = await brain.find({ noun: 'user' })
for (const user of users) {
await brain.update(user.id, {
email: user.data.email.toLowerCase(),
verified: user.data.verified ?? false
})
}
await migration.destroy()
console.log('✅ Migration safe! Apply changes to production brain')
// Apply validated migration to main brain
const users = await brain.find({ noun: 'user' })
for (const user of users) {
await brain.update(user.id, {
email: user.data.email.toLowerCase(),
verified: user.data.verified ?? false
})
}
await migration.destroy()
} else {
console.log('❌ Migration failed! Discarding fork.')
await migration.destroy()
console.log('❌ Migration failed! Discarding fork.')
await migration.destroy()
}
```
@ -203,8 +203,8 @@ const brain = new Brainy({ storage: { adapter: 's3', bucket: 'production' } })
await brain.init()
// Create two variants
const variantA = await brain.fork('variant-a') // Control
const variantB = await brain.fork('variant-b') // Test
const variantA = await brain.fork('variant-a') // Control
const variantB = await brain.fork('variant-b') // Test
// Run different algorithms
await variantA.processWithAlgorithm('current')
@ -219,14 +219,14 @@ console.log('Variant B accuracy:', metricsB.accuracy)
// Choose winner and update main brain
if (metricsB.accuracy > metricsA.accuracy) {
console.log('B wins! Apply algorithm B to production')
await brain.processWithAlgorithm('improved')
await variantA.destroy()
await variantB.destroy()
console.log('B wins! Apply algorithm B to production')
await brain.processWithAlgorithm('improved')
await variantA.destroy()
await variantB.destroy()
} else {
console.log('A wins! Keeping current algorithm.')
await variantA.destroy()
await variantB.destroy()
console.log('A wins! Keeping current algorithm.')
await variantA.destroy()
await variantB.destroy()
}
```
@ -311,12 +311,12 @@ await brain.update(docs[0].id, { version: 2 })
// Test against original state
const originalData = await snapshot.find({ noun: 'doc' })
console.log(originalData[0].data.version) // 1 (original state!)
console.log(originalData[0].data.version) // 1 (original state!)
// Clean up
await snapshot.destroy()
// Note: Time-travel queries (asOf) planned for v5.1.0
// Note: Time-travel queries (asOf) Planned
```
---
@ -330,27 +330,27 @@ await snapshot.destroy()
const fork1 = await brain.fork('my-experiment')
// Auto-generated name (uses timestamp)
const fork2 = await brain.fork() // 'fork-1635789012345'
const fork2 = await brain.fork() // 'fork-1635789012345'
// Fork with metadata (for tracking)
const fork3 = await brain.fork('test', {
author: 'Alice',
message: 'Testing new feature'
author: 'Alice',
message: 'Testing new feature'
})
```
### Branch Management (v5.0.0)
### Branch Management
**NEW in v5.0.0:** Full branch management now available!
Full branch management now available!
```javascript
// List all branches
const branches = await brain.listBranches()
console.log(branches) // ['main', 'experiment', 'test']
console.log(branches) // ['main', 'experiment', 'test']
// Get current branch
const current = await brain.getCurrentBranch()
console.log(current) // 'main'
console.log(current) // 'main'
// Switch between branches
await brain.checkout('experiment')
@ -359,33 +359,33 @@ await brain.checkout('experiment')
await brain.deleteBranch('old-experiment')
```
### Commit Tracking (v5.0.0)
### Commit Tracking
**NEW in v5.0.0:** Git-style commit tracking!
Git-style commit tracking!
```javascript
// Create a commit (snapshot of current state)
await brain.add({ type: 'user', data: { name: 'Alice' } })
const commitHash = await brain.commit({
message: 'Add Alice user',
author: 'dev@example.com'
message: 'Add Alice user',
author: 'dev@example.com'
})
console.log(commitHash) // 'a3f2c1b9...'
console.log(commitHash) // 'a3f2c1b9...'
```
### Commit History (v5.0.0)
### Commit History
**NEW in v5.0.0:** View commit history!
View commit history!
```javascript
// Get commit history for current branch
const history = await brain.getHistory({ limit: 10 })
history.forEach(commit => {
console.log(`${commit.hash}: ${commit.message}`)
console.log(` By: ${commit.author} at ${new Date(commit.timestamp)}`)
console.log(`${commit.hash}: ${commit.message}`)
console.log(` By: ${commit.author} at ${new Date(commit.timestamp)}`)
})
## Performance Characteristics
@ -425,7 +425,7 @@ Savings: 40% less memory
## Zero Configuration
**Fork is enabled by default in v5.0.0+. No setup required.**
**Fork is enabled by default. No setup required.**
```javascript
// This is all you need:
@ -467,9 +467,9 @@ await new Brainy({ storage: { adapter: 's3', bucket: 'my-data' } }).fork()
```javascript
const brain = new Brainy({
storage: { adapter: 'memory' },
vfs: { enabled: true },
intelligence: { enabled: true }
storage: { adapter: 'memory' },
vfs: { enabled: true },
intelligence: { enabled: true }
})
await brain.init()
@ -484,9 +484,9 @@ await brain.add({ noun: 'user', data: { name: 'Alice' } })
const fork = await brain.fork('test')
// All features work on fork:
await fork.vfs.readFile('/project/README.md') // ✅ VFS
await fork.find({ noun: 'user' }) // ✅ find()
await fork.query('users named Alice') // ✅ Triple Intelligence
await fork.vfs.readFile('/project/README.md') // ✅ VFS
await fork.find({ noun: 'user' }) // ✅ find()
await fork.query('users named Alice') // ✅ Triple Intelligence
```
### Works at Billion Scale
@ -494,8 +494,8 @@ await fork.query('users named Alice') // ✅ Triple Intelligence
```javascript
// Tested at 1M entities, extrapolates to 1B
const brain = new Brainy({
storage: { adapter: 'gcs', bucket: 'billion-scale' },
hnsw: { typeAware: true } // 87% memory reduction
storage: { adapter: 'gcs', bucket: 'billion-scale' },
hnsw: { typeAware: true } // 87% memory reduction
})
await brain.init()
@ -563,7 +563,7 @@ ALTER TABLE users_backup RENAME TO users;
```javascript
const fork = await brain.fork('test')
await fork.updateAll({ email: (u) => u.email.toLowerCase() })
if (success) await brain.checkout('test') // Make test branch active
if (success) await brain.checkout('test') // Make test branch active
else await fork.destroy()
```
@ -585,7 +585,7 @@ mongoimport --db mydb --collection users_backup --file users.json
**Brainy:**
```javascript
const fork = await brain.fork() // Done!
const fork = await brain.fork() // Done!
```
**Winner: Brainy** (100x faster, built-in)
@ -612,8 +612,7 @@ const fork = await brain.fork() // Done!
## What's Implemented vs. What's Next
### ✅ Available in v5.0.0:
- ✅ `fork()` - Instant clone in <100ms
### ✅ Available in - ✅ `fork()` - Instant clone in <100ms
- ✅ `listBranches()` - List all forks
- ✅ `getCurrentBranch()` - Get active branch
- ✅ `checkout()` - Switch between branches
@ -621,14 +620,14 @@ const fork = await brain.fork() // Done!
- ✅ `commit()` - Create state snapshots
- ✅ `getHistory()` - View commit history
### 🔮 Planned for v5.1.0+:
### 🔮 Planned for:
**Temporal Features:**
- `asOf(timestamp)` - Query data at specific time (✅ v5.0.0+)
- `asOf(timestamp)` - Query data at specific time (✅ available)
- `rollback(commitHash)` - Restore to previous state
- Full audit trail for all changes
These features require additional temporal infrastructure and are being carefully designed for v5.1.0+
These features require additional temporal infrastructure and are being carefully designed
---
@ -691,4 +690,4 @@ console.log('Fork created in < 2 seconds! 🚀')
---
**Brainy v5.0.0+** | [GitHub](https://github.com/soulcraftlabs/brainy) | [npm](https://npmjs.com/package/@soulcraft/brainy)
**Brainy** | [GitHub](https://github.com/soulcraftlabs/brainy) | [npm](https://npmjs.com/package/@soulcraft/brainy)

162
docs/guides/import-anything.md
View file

@ -24,8 +24,8 @@ await brain.import(anything)
```javascript
// Array of objects? No problem.
const people = [
{ name: 'Alice', role: 'Engineer', company: 'TechCorp' },
{ name: 'Bob', role: 'Designer', company: 'TechCorp' }
{ name: 'Alice', role: 'Engineer', company: 'TechCorp' },
{ name: 'Bob', role: 'Designer', company: 'TechCorp' }
]
await brain.import(people)
@ -55,7 +55,7 @@ await brain.import('sales-report.xlsx')
// Or specific sheets only
await brain.import('data.xlsx', {
excelSheets: ['Customers', 'Orders']
excelSheets: ['Customers', 'Orders']
})
// ✨ Multi-sheet data becomes interconnected entities!
```
@ -68,7 +68,7 @@ await brain.import('research-paper.pdf')
// With table extraction
await brain.import('report.pdf', {
pdfExtractTables: true
pdfExtractTables: true
})
// ✨ Converts PDF tables to structured data automatically!
```
@ -83,16 +83,16 @@ await brain.import('config.yaml')
const yaml = `
project: AI Assistant
team:
- name: Alice
role: Lead
- name: Bob
role: Dev
- name: Alice
role: Lead
- name: Bob
role: Dev
`
await brain.import(yaml, { format: 'yaml' })
// ✨ Hierarchical data becomes a connected graph!
```
### 📄 Import Word Documents (DOCX) - v4.2.0
### 📄 Import Word Documents (DOCX) -
```javascript
// From file path
await brain.import('research-paper.docx')
@ -105,8 +105,8 @@ await brain.import(buffer, { format: 'docx' })
// With neural extraction
await brain.import('report.docx', {
enableNeuralExtraction: true,
enableHierarchicalRelationships: true
enableNeuralExtraction: true,
enableHierarchicalRelationships: true
})
// ✨ Extracts entities from paragraphs and creates relationships within sections!
```
@ -121,25 +121,25 @@ await brain.import('https://api.example.com/data.json')
await brain.import('https://data.gov/census.csv')
// ✨ Fetches CSV from web, parses, imports!
// With authentication (v4.2.0)
// With authentication
await brain.import({
type: 'url',
data: 'https://api.example.com/private/data.xlsx',
auth: {
username: 'user',
password: 'pass'
}
type: 'url',
data: 'https://api.example.com/private/data.xlsx',
auth: {
username: 'user',
password: 'pass'
}
})
// ✨ Supports basic authentication for protected resources!
// With custom headers (v4.2.0)
// With custom headers
await brain.import({
type: 'url',
data: 'https://api.example.com/data.json',
headers: {
'Authorization': 'Bearer TOKEN',
'X-API-Key': 'your-key'
}
type: 'url',
data: 'https://api.example.com/data.json',
headers: {
'Authorization': 'Bearer TOKEN',
'X-API-Key': 'your-key'
}
})
// ✨ Full HTTP header customization support!
```
@ -163,9 +163,9 @@ When you import data, Brainy:
2. **Intelligent parsing** - CSV (encoding/delimiter), Excel (multi-sheet), PDF (text/tables), DOCX (headings/paragraphs)
3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
5. **Scores confidence & weight** - Every entity and relationship gets quality metrics (v4.2.0)
5. **Scores confidence & weight** - Every entity and relationship gets quality metrics
6. **Creates embeddings** - Makes everything semantically searchable
7. **Indexes metadata** - Enables lightning-fast filtering with range queries (v4.2.0)
7. **Indexes metadata** - Enables lightning-fast filtering with range queries
## Intelligent Type Detection
@ -176,7 +176,7 @@ Brainy automatically detects what TYPE of data you're importing:
{ name: 'John', email: 'john@example.com' }
// This becomes an Organization
{ companyName: 'Acme', employees: 500 }
{ companyName: 'Acme', employees: 500 }
// This becomes a Document
{ title: 'Report', content: '...', author: 'Jane' }
@ -193,9 +193,9 @@ Brainy finds connections in your data:
```javascript
const data = [
{ id: 'u1', name: 'Alice', managerId: 'u2' },
{ id: 'u2', name: 'Bob', departmentId: 'd1' },
{ id: 'd1', name: 'Engineering' }
{ id: 'u1', name: 'Alice', managerId: 'u2' },
{ id: 'u2', name: 'Bob', departmentId: 'd1' },
{ id: 'd1', name: 'Engineering' }
]
await brain.import(data)
@ -204,28 +204,27 @@ await brain.import(data)
// - Bob "memberOf" Engineering
```
## Confidence & Weight Scoring - v4.2.0
## Confidence & Weight Scoring -
Every entity and relationship gets confidence and weight scores:
```javascript
// Import with confidence threshold
await brain.import(data, {
confidenceThreshold: 0.8 // Only extract entities with >80% confidence
confidenceThreshold: 0.8 // Only extract entities with >80% confidence
})
// Query high-confidence entities using range queries
const highConfidence = await brain.find({
where: {
confidence: { gte: 0.8 } // Get entities with confidence >= 0.8
}
where: {
confidence: { gte: 0.8 } // Get entities with confidence >= 0.8
}
})
// Range query operators: gt, gte, lt, lte, between
const mediumConfidence = await brain.find({
where: {
confidence: { between: [0.6, 0.8] }
}
where: {
confidence: { between: [0.6, 0.8] }
}
})
```
@ -236,24 +235,23 @@ const mediumConfidence = await brain.find({
**Weights** indicate importance/relevance within the document context.
## Per-Sheet Excel Extraction - v4.2.0
## Per-Sheet Excel Extraction -
Excel files with multiple sheets can be organized by sheet:
```javascript
// Group entities by sheet in VFS
await brain.import('multi-sheet-data.xlsx', {
groupBy: 'sheet' // Creates separate directories for each sheet
groupBy: 'sheet' // Creates separate directories for each sheet
})
// Result VFS structure:
// /imports/data/
// ├── Sheet1/
// │ ├── entity1.json
// │ └── entity2.json
// └── Sheet2/
// ├── entity3.json
// └── entity4.json
// ├── Sheet1/
// │ ├── entity1.json
// │ └── entity2.json
// └── Sheet2/
// ├── entity3.json
// └── entity4.json
// Other groupBy options:
// - 'type': Group by entity type (Person, Place, etc.)
@ -274,9 +272,9 @@ const results = await brain.find('people in engineering who joined this year')
// Graph traversal + filters
const connected = await brain.find({
like: 'Alice',
connected: { depth: 2 },
where: { department: 'Engineering' }
like: 'Alice',
connected: { depth: 2 },
where: { department: 'Engineering' }
})
```
@ -286,37 +284,37 @@ Everything works with zero config, but you can customize:
```javascript
await brain.import(data, {
// Format detection
format: 'excel', // Force specific format (auto-detected if not specified)
// Format detection
format: 'excel', // Force specific format (auto-detected if not specified)
// VFS & Organization
vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified)
groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
preserveSource: true, // Keep original source file in VFS (default: true)
// VFS & Organization
vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified)
groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
preserveSource: true, // Keep original source file in VFS (default: true)
// Entity & Relationship Creation
createEntities: true, // Create entities in knowledge graph (default: true)
createRelationships: true, // Create relationships in knowledge graph (default: true)
// Entity & Relationship Creation
createEntities: true, // Create entities in knowledge graph (default: true)
createRelationships: true, // Create relationships in knowledge graph (default: true)
// Neural Intelligence
enableNeuralExtraction: true, // Use AI to extract entities (default: true)
enableRelationshipInference: true, // Use AI to infer relationships (default: true)
enableConceptExtraction: true, // Extract concepts from text (default: true)
confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6)
// Neural Intelligence
enableNeuralExtraction: true, // Use AI to extract entities (default: true)
enableRelationshipInference: true, // Use AI to infer relationships (default: true)
enableConceptExtraction: true, // Extract concepts from text (default: true)
confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6)
// Deduplication
enableDeduplication: true, // Check for duplicate entities (default: true)
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
// Note: Auto-disabled for imports >100 entities
// Deduplication
enableDeduplication: true, // Check for duplicate entities (default: true)
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
// Note: Auto-disabled for imports >100 entities
// Performance
chunkSize: 100, // Batch size for processing (default: varies by operation)
// Performance
chunkSize: 100, // Batch size for processing (default: varies by operation)
// History & Progress
enableHistory: true, // Track import history (default: true)
onProgress: (progress) => { // Progress callback
console.log(progress.stage, progress.message)
}
// History & Progress
enableHistory: true, // Track import history (default: true)
onProgress: (progress) => { // Progress callback
console.log(progress.stage, progress.message)
}
})
```
@ -343,9 +341,9 @@ const results = await brain.import(problematicData)
### 🏢 Business Data
```javascript
// Import ANY source - ONE method!
await brain.import('customers.csv') // File
await brain.import('https://api.co/orders') // URL
await brain.import(productsArray) // Data
await brain.import('customers.csv') // File
await brain.import('https://api.co/orders') // URL
await brain.import(productsArray) // Data
// Now query across all of it!
await brain.find('customers who bought products in Q4')
@ -389,8 +387,8 @@ await brain.find('posts by users following Alice with >10 comments')
```javascript
// ONE method that understands EVERYTHING:
await brain.import(data) // Objects, arrays, strings
await brain.import('file.csv') // Files (auto-detected)
await brain.import(data) // Objects, arrays, strings
await brain.import('file.csv') // Files (auto-detected)
await brain.import('http://..') // URLs (auto-fetched)
// It ALWAYS knows what to do! ✨

4
docs/guides/import-flow.md
View file

@ -39,7 +39,7 @@ Each phase adds intelligence and structure to your raw data, transforming it int
---
## 🌊 Always-On Streaming Architecture (v4.2.0+)
## 🌊 Always-On Streaming Architecture
All imports use streaming with **progressive flush intervals**:
@ -1062,7 +1062,7 @@ if (!fromEntity || !toEntity) {
}
```
#### 5.3b: Check for Duplicates (v3.43.2 Critical Fix)
#### 5.3b: Check for Duplicates (Critical Fix)
**The Bug**: Without duplicate checking, re-importing would create:
```

278
docs/guides/import-progress-examples.md
View file

@ -2,7 +2,7 @@
**How to Use Progress Tracking in Your Applications**
Brainy v4.5.0+ provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
Brainy provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
> **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation.
@ -20,12 +20,12 @@ const brain = await Brainy.create()
// Import with progress tracking
const result = await brain.import(fs.readFileSync('large-file.xlsx'), {
onProgress: (progress) => {
console.log(`Progress: ${progress.stage}`)
console.log(` Message: ${progress.message}`)
console.log(` Entities: ${progress.entities || 0}`)
console.log(` Relationships: ${progress.relationships || 0}`)
}
onProgress: (progress) => {
console.log(`Progress: ${progress.stage}`)
console.log(` Message: ${progress.message}`)
console.log(` Entities: ${progress.entities || 0}`)
console.log(` Relationships: ${progress.relationships || 0}`)
}
})
console.log(`Import complete: ${result.entities.length} entities created`)
@ -34,30 +34,30 @@ console.log(`Import complete: ${result.entities.length} entities created`)
**Expected Output:**
```
Progress: detecting
Message: Detecting format...
Entities: 0
Relationships: 0
Message: Detecting format...
Entities: 0
Relationships: 0
Progress: extracting
Message: Loading Excel workbook...
Entities: 0
Relationships: 0
Message: Loading Excel workbook...
Entities: 0
Relationships: 0
Progress: extracting
Message: Reading sheet: Sales (1/3)
Entities: 0
Relationships: 0
Message: Reading sheet: Sales (1/3)
Entities: 0
Relationships: 0
Progress: extracting
Message: Parsing Excel (33%)
Entities: 0
Relationships: 0
Message: Parsing Excel (33%)
Entities: 0
Relationships: 0
Progress: extracting
Message: Reading sheet: Products (2/3)
Entities: 0
Relationships: 0
Message: Reading sheet: Products (2/3)
Entities: 0
Relationships: 0
... (more progress updates)
Progress: complete
Message: Import complete
Entities: 1523
Relationships: 892
Message: Import complete
Entities: 1523
Relationships: 892
Import complete: 1523 entities created
```
@ -70,19 +70,19 @@ The examples below show format-specific messages, but **you don't need format-sp
```typescript
// ONE HANDLER FOR ALL FORMATS!
function universalProgressHandler(progress) {
console.log(`[${progress.stage}] ${progress.message}`)
console.log(`[${progress.stage}] ${progress.message}`)
if (progress.processed && progress.total) {
console.log(` Progress: ${progress.processed}/${progress.total}`)
}
if (progress.processed && progress.total) {
console.log(` Progress: ${progress.processed}/${progress.total}`)
}
if (progress.entities || progress.relationships) {
console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
}
if (progress.entities || progress.relationships) {
console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
}
if (progress.throughput && progress.eta) {
console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
}
if (progress.throughput && progress.eta) {
console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
}
}
// Use it for ANY format!
@ -105,20 +105,20 @@ The examples below show **what messages look like** for different formats using
```typescript
await brain.import(csvBuffer, {
format: 'csv',
onProgress: (progress) => {
if (progress.stage === 'extracting') {
// CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
console.log(progress.message)
}
}
format: 'csv',
onProgress: (progress) => {
if (progress.stage === 'extracting') {
// CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
console.log(progress.message)
}
}
})
```
**CSV Progress Messages:**
- ✅ "Detecting CSV encoding and delimiter..."
- ✅ "Parsing CSV rows (delimiter: ",")"
- ✅ "Parsed 75%" (via bytes processed)
- ✅ "Parsed 75%" (via bytes processed)
- ✅ "Extracted 1000 rows"
- ✅ "Converting types: 5000/10000 rows..."
- ✅ "CSV processing complete: 10000 rows"
@ -129,12 +129,12 @@ await brain.import(csvBuffer, {
```typescript
await brain.import(pdfBuffer, {
format: 'pdf',
onProgress: (progress) => {
// PDF reports exact page numbers
console.log(progress.message)
// Example: "Processing page 5 of 23"
}
format: 'pdf',
onProgress: (progress) => {
// PDF reports exact page numbers
console.log(progress.message)
// Example: "Processing page 5 of 23"
}
})
```
@ -152,12 +152,12 @@ await brain.import(pdfBuffer, {
```typescript
await brain.import(excelBuffer, {
format: 'excel',
onProgress: (progress) => {
// Excel reports sheet names
console.log(progress.message)
// Example: "Reading sheet: Q2 Sales (2/5)"
}
format: 'excel',
onProgress: (progress) => {
// Excel reports sheet names
console.log(progress.message)
// Example: "Reading sheet: Q2 Sales (2/5)"
}
})
```
@ -175,11 +175,11 @@ await brain.import(excelBuffer, {
```typescript
await brain.import(jsonBuffer, {
format: 'json',
onProgress: (progress) => {
// JSON reports every 10 nodes
console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
}
format: 'json',
onProgress: (progress) => {
// JSON reports every 10 nodes
console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
}
})
```
@ -189,10 +189,10 @@ await brain.import(jsonBuffer, {
```typescript
await brain.import(markdownString, {
format: 'markdown',
onProgress: (progress) => {
console.log(`Section ${progress.processed}/${progress.total}`)
}
format: 'markdown',
onProgress: (progress) => {
console.log(`Section ${progress.processed}/${progress.total}`)
}
})
```
@ -204,44 +204,44 @@ await brain.import(markdownString, {
```typescript
function ImportProgress({ file }: { file: File }) {
const [progress, setProgress] = useState({
stage: 'idle',
message: '',
percent: 0,
entities: 0,
relationships: 0
})
const [progress, setProgress] = useState({
stage: 'idle',
message: '',
percent: 0,
entities: 0,
relationships: 0
})
const handleImport = async () => {
const buffer = await file.arrayBuffer()
const handleImport = async () => {
const buffer = await file.arrayBuffer()
await brain.import(Buffer.from(buffer), {
onProgress: (p) => {
setProgress({
stage: p.stage,
message: p.message,
// Estimate percentage from stage
percent: {
detecting: 10,
extracting: 50,
'storing-vfs': 80,
'storing-graph': 90,
complete: 100
}[p.stage] || 0,
entities: p.entities || 0,
relationships: p.relationships || 0
})
}
})
}
await brain.import(Buffer.from(buffer), {
onProgress: (p) => {
setProgress({
stage: p.stage,
message: p.message,
// Estimate percentage from stage
percent: {
detecting: 10,
extracting: 50,
'storing-vfs': 80,
'storing-graph': 90,
complete: 100
}[p.stage] || 0,
entities: p.entities || 0,
relationships: p.relationships || 0
})
}
})
}
return (
<div>
<ProgressBar value={progress.percent} />
<p>{progress.message}</p>
<p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
</div>
)
return (
<div>
<ProgressBar value={progress.percent} />
<p>{progress.message}</p>
<p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
</div>
)
}
```
@ -255,13 +255,13 @@ import ora from 'ora'
const spinner = ora('Starting import...').start()
await brain.import(buffer, {
onProgress: (progress) => {
spinner.text = progress.message
onProgress: (progress) => {
spinner.text = progress.message
if (progress.stage === 'complete') {
spinner.succeed(`Import complete: ${progress.entities} entities`)
}
}
if (progress.stage === 'complete') {
spinner.succeed(`Import complete: ${progress.entities} entities`)
}
}
})
```
@ -285,20 +285,20 @@ let startTime = Date.now()
let lastUpdate = startTime
await brain.import(buffer, {
onProgress: (progress) => {
const elapsed = Date.now() - startTime
const rate = progress.entities / (elapsed / 1000) // entities/sec
onProgress: (progress) => {
const elapsed = Date.now() - startTime
const rate = progress.entities / (elapsed / 1000) // entities/sec
console.clear()
console.log('Import Progress Dashboard')
console.log('========================')
console.log(`Stage: ${progress.stage}`)
console.log(`Status: ${progress.message}`)
console.log(`Entities: ${progress.entities}`)
console.log(`Relationships: ${progress.relationships}`)
console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
}
console.clear()
console.log('Import Progress Dashboard')
console.log('========================')
console.log(`Stage: ${progress.stage}`)
console.log(`Status: ${progress.message}`)
console.log(`Entities: ${progress.entities}`)
console.log(`Relationships: ${progress.relationships}`)
console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
}
})
```
@ -310,21 +310,21 @@ await brain.import(buffer, {
```typescript
const formatMessages = {
csv: (p) => `CSV: ${p.message}`,
pdf: (p) => `PDF: ${p.message}`,
excel: (p) => `Excel: ${p.message}`,
json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
yaml: (p) => `YAML: ${p.processed} nodes`,
docx: (p) => `DOCX: ${p.processed} paragraphs`
csv: (p) => `CSV: ${p.message}`,
pdf: (p) => `PDF: ${p.message}`,
excel: (p) => `Excel: ${p.message}`,
json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
yaml: (p) => `YAML: ${p.processed} nodes`,
docx: (p) => `DOCX: ${p.processed} paragraphs`
}
await brain.import(buffer, {
onProgress: (progress) => {
// Format is available in progress.stage metadata
const message = formatMessages[detectedFormat]?.(progress) || progress.message
console.log(message)
}
onProgress: (progress) => {
// Format is available in progress.stage metadata
const message = formatMessages[detectedFormat]?.(progress) || progress.message
console.log(message)
}
})
```
@ -336,18 +336,18 @@ await brain.import(buffer, {
```typescript
let lastUIUpdate = 0
const THROTTLE_MS = 100 // Update UI max once per 100ms
const THROTTLE_MS = 100 // Update UI max once per 100ms
await brain.import(buffer, {
onProgress: (progress) => {
const now = Date.now()
if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
return // Skip this update
}
onProgress: (progress) => {
const now = Date.now()
if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
return // Skip this update
}
lastUIUpdate = now
updateUI(progress) // Only update every 100ms
}
lastUIUpdate = now
updateUI(progress) // Only update every 100ms
}
})
```

664
docs/guides/import-progress-implementation.md
View file

@ -28,17 +28,17 @@
```typescript
// THE PUBLIC API - Same for ALL 7 formats!
brain.import(buffer, {
onProgress: (progress: ImportProgress) => {
// These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
progress.message // Human-readable status (varies by format, always readable)
progress.processed // Items processed (optional)
progress.total // Total items (optional)
progress.entities // Entities extracted (optional)
progress.relationships // Relationships inferred (optional)
progress.throughput // Items/sec (optional, during extraction)
progress.eta // Time remaining in ms (optional)
}
onProgress: (progress: ImportProgress) => {
// These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
progress.message // Human-readable status (varies by format, always readable)
progress.processed // Items processed (optional)
progress.total // Total items (optional)
progress.entities // Entities extracted (optional)
progress.relationships // Relationships inferred (optional)
progress.throughput // Items/sec (optional, during extraction)
progress.eta // Time remaining in ms (optional)
}
})
```
@ -49,14 +49,14 @@ The table below shows how formats implement progress *internally*. Normal develo
```typescript
// Internal: Binary formats use handler hooks (you added these!)
interface FormatHandlerProgressHooks {
onBytesProcessed?: (bytes: number) => void
onCurrentItem?: (message: string) => void
onDataExtracted?: (count: number, total?: number) => void
onBytesProcessed?: (bytes: number) => void
onCurrentItem?: (message: string) => void
onDataExtracted?: (count: number, total?: number) => void
}
// Internal: Text formats use importer callbacks
interface ImporterProgressCallback {
onProgress?: (stats: { processed, total, entities, relationships }) => void
onProgress?: (stats: { processed, total, entities, relationships }) => void
}
// Both are converted to ImportProgress by ImportCoordinator!
@ -74,7 +74,7 @@ interface ImporterProgressCallback {
## 🎯 Overview
As of v4.5.0, Brainy supports comprehensive, multi-dimensional progress tracking for imports:
Brainy supports comprehensive, multi-dimensional progress tracking for imports:
- **Bytes processed** (always available, most deterministic)
- **Entities extracted** (AI extraction phase)
- **Stage-specific metrics** (parsing: MB/s, extraction: entities/s)
@ -91,23 +91,23 @@ All handlers follow a simple, consistent pattern using **progress hooks**.
```typescript
export interface FormatHandlerProgressHooks {
/**
* Report bytes processed
* Call this as you read/parse the file
*/
onBytesProcessed?: (bytes: number) => void
/**
* Report bytes processed
* Call this as you read/parse the file
*/
onBytesProcessed?: (bytes: number) => void
/**
* Set current processing context
* Examples: "Processing page 5", "Reading sheet: Q2 Sales"
*/
onCurrentItem?: (item: string) => void
/**
* Set current processing context
* Examples: "Processing page 5", "Reading sheet: Q2 Sales"
*/
onCurrentItem?: (item: string) => void
/**
* Report structured data extraction progress
* Examples: "Extracted 100 rows", "Parsed 50 paragraphs"
*/
onDataExtracted?: (count: number, total?: number) => void
/**
* Report structured data extraction progress
* Examples: "Extracted 100 rows", "Parsed 50 paragraphs"
*/
onDataExtracted?: (count: number, total?: number) => void
}
```
@ -117,19 +117,19 @@ Progress hooks are automatically passed to your handler via `FormatHandlerOption
```typescript
export interface FormatHandlerOptions {
// ... existing options ...
// ... existing options ...
/**
* Progress hooks (v4.5.0)
* Handlers call these to report progress during processing
*/
progressHooks?: FormatHandlerProgressHooks
/**
* Progress hooks
* Handlers call these to report progress during processing
*/
progressHooks?: FormatHandlerProgressHooks
/**
* Total file size in bytes (v4.5.0)
* Used for progress percentage calculation
*/
totalBytes?: number
/**
* Total file size in bytes
* Used for progress percentage calculation
*/
totalBytes?: number
}
```
@ -141,36 +141,36 @@ Every handler follows these 5 steps:
```typescript
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
const progressHooks = options.progressHooks // Step 1: Get hooks
const progressHooks = options.progressHooks // Step 1: Get hooks
// Step 2: Report initial progress
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Starting import...')
}
// Step 2: Report initial progress
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Starting import...')
}
// Step 3: Report bytes as you process
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0) // Start
}
// Step 3: Report bytes as you process
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0) // Start
}
// ... do parsing ...
// ... do parsing ...
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(buffer.length) // Complete
}
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(buffer.length) // Complete
}
// Step 4: Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length)
}
// Step 4: Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length)
}
// Step 5: Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Complete: ${data.length} items processed`)
}
// Step 5: Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Complete: ${data.length} items processed`)
}
return { format, data, metadata }
return { format, data, metadata }
}
```
@ -182,76 +182,76 @@ Here's the **ACTUAL implementation** from CSV handler showing all the key progre
```typescript
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
const startTime = Date.now()
const progressHooks = options.progressHooks // ✅ Step 1
const startTime = Date.now()
const progressHooks = options.progressHooks // ✅ Step 1
// Convert to buffer if string
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
const totalBytes = buffer.length
// Convert to buffer if string
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
const totalBytes = buffer.length
// ✅ Step 2: Report start
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...')
}
// ✅ Step 2: Report start
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...')
}
// Detect encoding
const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer)
const text = buffer.toString(detectedEncoding as BufferEncoding)
// Detect encoding
const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer)
const text = buffer.toString(detectedEncoding as BufferEncoding)
// Detect delimiter
const delimiter = options.csvDelimiter || this.detectDelimiter(text)
// Detect delimiter
const delimiter = options.csvDelimiter || this.detectDelimiter(text)
// ✅ Progress update: Parsing phase
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
}
// ✅ Progress update: Parsing phase
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
}
// Parse CSV
const records = parse(text, { /* options */ })
// Parse CSV
const records = parse(text, { /* options */ })
// ✅ Step 3: Report bytes processed (entire file parsed)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
// ✅ Step 3: Report bytes processed (entire file parsed)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
const data = Array.isArray(records) ? records : [records]
const data = Array.isArray(records) ? records : [records]
// ✅ Step 4: Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`)
}
// ✅ Step 4: Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`)
}
// Type inference and conversion
const fields = data.length > 0 ? Object.keys(data[0]) : []
const types = this.inferFieldTypes(data)
// Type inference and conversion
const fields = data.length > 0 ? Object.keys(data[0]) : []
const types = this.inferFieldTypes(data)
const convertedData = data.map((row, index) => {
const converted = this.convertRow(row, types)
const convertedData = data.map((row, index) => {
const converted = this.convertRow(row, types)
// ✅ Progress update every 1000 rows (avoid spam)
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
}
// ✅ Progress update every 1000 rows (avoid spam)
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
}
return converted
})
return converted
})
// ✅ Step 5: Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
}
// ✅ Step 5: Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
}
return {
format: this.format,
data: convertedData,
metadata: { /* ... */ }
}
return {
format: this.format,
data: convertedData,
metadata: { /* ... */ }
}
}
```
@ -292,51 +292,51 @@ Brainy supports **7 file formats** with full progress tracking:
```typescript
async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
const progressHooks = options.progressHooks
const totalBytes = data.length
const progressHooks = options.progressHooks
const totalBytes = data.length
// Report start
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Loading PDF document...')
}
// Report start
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Loading PDF document...')
}
const pdfDoc = await loadPDF(data)
const totalPages = pdfDoc.numPages
const pdfDoc = await loadPDF(data)
const totalPages = pdfDoc.numPages
const extractedData: any[] = []
let bytesProcessed = 0
const extractedData: any[] = []
let bytesProcessed = 0
for (let pageNum = 1; pageNum <= totalPages; pageNum++) {
// ✅ Report current page
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`)
}
for (let pageNum = 1; pageNum <= totalPages; pageNum++) {
// ✅ Report current page
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`)
}
const page = await pdfDoc.getPage(pageNum)
const text = await page.getTextContent()
extractedData.push(this.processPageText(text))
const page = await pdfDoc.getPage(pageNum)
const text = await page.getTextContent()
extractedData.push(this.processPageText(text))
// ✅ Estimate bytes processed (pages are sequential)
bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// ✅ Estimate bytes processed (pages are sequential)
bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// ✅ Report extraction progress
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(pageNum, totalPages)
}
}
// ✅ Report extraction progress
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(pageNum, totalPages)
}
}
// Final progress
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`)
}
// Final progress
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`)
}
return { format: 'pdf', data: extractedData, metadata: { /* ... */ } }
return { format: 'pdf', data: extractedData, metadata: { /* ... */ } }
}
```
@ -344,55 +344,55 @@ async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedDat
```typescript
async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
const progressHooks = options.progressHooks
const totalBytes = data.length
const progressHooks = options.progressHooks
const totalBytes = data.length
// Load workbook
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Loading Excel workbook...')
}
// Load workbook
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Loading Excel workbook...')
}
const workbook = XLSX.read(data)
const sheetNames = options.excelSheets === 'all'
? workbook.SheetNames
: (options.excelSheets || [workbook.SheetNames[0]])
const workbook = XLSX.read(data)
const sheetNames = options.excelSheets === 'all'
? workbook.SheetNames
: (options.excelSheets || [workbook.SheetNames[0]])
const allData: any[] = []
let bytesProcessed = 0
const allData: any[] = []
let bytesProcessed = 0
for (let i = 0; i < sheetNames.length; i++) {
const sheetName = sheetNames[i]
for (let i = 0; i < sheetNames.length; i++) {
const sheetName = sheetNames[i]
// ✅ Report current sheet
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`)
}
// ✅ Report current sheet
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`)
}
const sheet = workbook.Sheets[sheetName]
const sheetData = XLSX.utils.sheet_to_json(sheet)
allData.push(...sheetData)
const sheet = workbook.Sheets[sheetName]
const sheetData = XLSX.utils.sheet_to_json(sheet)
allData.push(...sheetData)
// ✅ Estimate bytes processed (sheets processed sequentially)
bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// ✅ Estimate bytes processed (sheets processed sequentially)
bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// ✅ Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done
}
}
// ✅ Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done
}
}
// Final progress
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`)
}
// Final progress
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`)
}
return { format: 'xlsx', data: allData, metadata: { /* ... */ } }
return { format: 'xlsx', data: allData, metadata: { /* ... */ } }
}
```
@ -400,44 +400,44 @@ async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedDat
```typescript
async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResult> {
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Parse JSON if string
let jsonData = typeof data === 'string' ? JSON.parse(data) : data
// Parse JSON if string
let jsonData = typeof data === 'string' ? JSON.parse(data) : data
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Traverse and extract (reports progress every 10 nodes)
const entities: ExtractedJSONEntity[] = []
const relationships: ExtractedJSONRelationship[] = []
let nodesProcessed = 0
// Traverse and extract (reports progress every 10 nodes)
const entities: ExtractedJSONEntity[] = []
const relationships: ExtractedJSONRelationship[] = []
let nodesProcessed = 0
await this.traverseJSON(
jsonData,
entities,
relationships,
() => {
nodesProcessed++
if (nodesProcessed % 10 === 0) {
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
}
}
)
await this.traverseJSON(
jsonData,
entities,
relationships,
() => {
nodesProcessed++
if (nodesProcessed % 10 === 0) {
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
}
}
)
// ✅ Report completion
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
// ✅ Report completion
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
return { nodesProcessed, entitiesExtracted: entities.length, ... }
return { nodesProcessed, entitiesExtracted: entities.length, ... }
}
```
@ -445,38 +445,38 @@ async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResul
```typescript
async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<SmartMarkdownResult> {
// ✅ Report parsing start
options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 })
// ✅ Report parsing start
options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 })
// Parse markdown into sections
const parsedSections = this.parseMarkdown(markdown, options)
// Parse markdown into sections
const parsedSections = this.parseMarkdown(markdown, options)
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 })
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 })
// Process each section (reports progress after each section)
const sections: MarkdownSection[] = []
for (let i = 0; i < parsedSections.length; i++) {
const section = await this.processSection(parsedSections[i], options)
sections.push(section)
// Process each section (reports progress after each section)
const sections: MarkdownSection[] = []
for (let i = 0; i < parsedSections.length; i++) {
const section = await this.processSection(parsedSections[i], options)
sections.push(section)
options.onProgress?.({
processed: i + 1,
total: parsedSections.length,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
})
}
options.onProgress?.({
processed: i + 1,
total: parsedSections.length,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
})
}
// ✅ Report completion
options.onProgress?.({
processed: sections.length,
total: sections.length,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
})
// ✅ Report completion
options.onProgress?.({
processed: sections.length,
total: sections.length,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
})
return { sectionsProcessed: sections.length, ... }
return { sectionsProcessed: sections.length, ... }
}
```
@ -484,27 +484,27 @@ async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<Sma
```typescript
async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Promise<SmartYAMLResult> {
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Parse YAML
const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8')
const data = yaml.load(yamlString)
// Parse YAML
const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8')
const data = yaml.load(yamlString)
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Traverse YAML structure (reports progress every 10 nodes)
// ... similar to JSON traversal ...
// Traverse YAML structure (reports progress every 10 nodes)
// ... similar to JSON traversal ...
// ✅ Report completion (already implemented)
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
// ✅ Report completion (already implemented)
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
return { nodesProcessed, entitiesExtracted: entities.length, ... }
return { nodesProcessed, entitiesExtracted: entities.length, ... }
}
```
@ -512,39 +512,39 @@ async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Pro
```typescript
async extract(buffer: Buffer, options: SmartDOCXOptions = {}): Promise<SmartDOCXResult> {
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Extract text and HTML using Mammoth
const textResult = await mammoth.extractRawText({ buffer })
const htmlResult = await mammoth.convertToHtml({ buffer })
// Extract text and HTML using Mammoth
const textResult = await mammoth.extractRawText({ buffer })
const htmlResult = await mammoth.convertToHtml({ buffer })
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Process paragraphs (reports progress every 10 paragraphs)
const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength)
// Process paragraphs (reports progress every 10 paragraphs)
const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength)
for (let i = 0; i < paragraphs.length; i++) {
await this.processParagraph(paragraphs[i])
for (let i = 0; i < paragraphs.length; i++) {
await this.processParagraph(paragraphs[i])
if (i % 10 === 0) {
options.onProgress?.({
processed: i + 1,
entities: entities.length,
relationships: relationships.length
})
}
}
if (i % 10 === 0) {
options.onProgress?.({
processed: i + 1,
entities: entities.length,
relationships: relationships.length
})
}
}
// ✅ Report completion (already implemented)
options.onProgress?.({
processed: paragraphs.length,
entities: entities.length,
relationships: relationships.length
})
// ✅ Report completion (already implemented)
options.onProgress?.({
processed: paragraphs.length,
entities: entities.length,
relationships: relationships.length
})
return { paragraphsProcessed: paragraphs.length, ... }
return { paragraphsProcessed: paragraphs.length, ... }
}
```
@ -559,20 +559,20 @@ Progress hooks are **optional**. Always check before calling:
```typescript
// ✅ Good - safe
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytes)
progressHooks.onBytesProcessed(bytes)
}
// ❌ Bad - will crash if hooks undefined
progressHooks.onBytesProcessed(bytes) // TypeError!
progressHooks.onBytesProcessed(bytes) // TypeError!
```
### 2. Report Bytes at Start and End
```typescript
// ✅ Good - clear start and end
progressHooks?.onBytesProcessed(0) // Start
progressHooks?.onBytesProcessed(0) // Start
// ... processing ...
progressHooks?.onBytesProcessed(totalBytes) // End
progressHooks?.onBytesProcessed(totalBytes) // End
// ❌ Bad - no clear boundaries
// ... just start processing without reporting start
@ -583,17 +583,17 @@ progressHooks?.onBytesProcessed(totalBytes) // End
```typescript
// ✅ Good - report every 1000 items
for (let i = 0; i < items.length; i++) {
processItem(items[i])
processItem(items[i])
if (i > 0 && i % 1000 === 0) {
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`)
}
if (i > 0 && i % 1000 === 0) {
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`)
}
}
// ❌ Bad - report EVERY item (spam!)
for (let i = 0; i < items.length; i++) {
processItem(items[i])
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks!
processItem(items[i])
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks!
}
```
@ -614,13 +614,13 @@ progressHooks?.onCurrentItem('Working...')
```typescript
// ✅ Good - total known
progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows
progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows
// ✅ Also good - total unknown (streaming)
progressHooks?.onDataExtracted(100, undefined) // 100 rows so far
progressHooks?.onDataExtracted(100, undefined) // 100 rows so far
// ✅ Also good - complete
progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows
progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows
```
---
@ -637,18 +637,18 @@ const handler = new CSVHandler()
const data = fs.readFileSync('./test.csv')
const result = await handler.process(data, {
filename: 'test.csv',
progressHooks: {
onBytesProcessed: (bytes) => {
console.log(`Bytes: ${bytes}`)
},
onCurrentItem: (item) => {
console.log(`Status: ${item}`)
},
onDataExtracted: (count, total) => {
console.log(`Extracted: ${count}${total ? `/${total}` : ''}`)
}
}
filename: 'test.csv',
progressHooks: {
onBytesProcessed: (bytes) => {
console.log(`Bytes: ${bytes}`)
},
onCurrentItem: (item) => {
console.log(`Status: ${item}`)
},
onDataExtracted: (count, total) => {
console.log(`Extracted: ${count}${total ? `/${total}` : ''}`)
}
}
})
console.log(`Complete: ${result.data.length} rows`)
@ -673,24 +673,24 @@ Complete: 1000 rows
```
User Imports File
↓
↓
ImportManager
↓
↓
Creates ProgressTracker
↓
↓
Calls Handler.process() with progressHooks
↓
↓
Handler Reports Progress:
├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated
├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated
├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated
├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated
└─ onCurrentItem("Complete") → ProgressTracker → final progress
↓
├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated
├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated
├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated
├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated
└─ onCurrentItem("Complete") → ProgressTracker → final progress
↓
ProgressTracker emits to callback (throttled 100ms)
↓
↓
User sees:
"Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..."
"Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..."
```
---

4
docs/guides/import-quick-reference.md
View file

@ -91,7 +91,7 @@ await brain.import(file, {
})
```
### Import Tracking (v4.10.0+)
### Import Tracking
Track and organize imports by project:
@ -149,7 +149,7 @@ await brain.import(file, {
})
```
### Always-On Streaming (v4.2.0+)
### Always-On Streaming
All imports use streaming with adaptive flush intervals. Query data as it's imported:

2
docs/guides/standard-import-progress.md
View file

@ -65,7 +65,7 @@ interface ImportProgress {
/** Estimated time remaining (milliseconds) */
eta?: number
/** Whether data is queryable at this point (v4.2.0+) */
/** Whether data is queryable at this point */
queryable?: boolean
}
```

4
docs/guides/streaming-imports.md
View file

@ -300,7 +300,7 @@ interface ImportProgress {
relationships?: number // Relationships inferred so far
/**
* Whether data is queryable (v4.2.0+)
* Whether data is queryable
*
* true = Indexes flushed, queries will be fast and complete
* false/undefined = Data in storage but indexes not flushed yet
@ -376,7 +376,7 @@ No changes required! Streaming is now always enabled with optimal defaults:
// Before (v3.x, v4.0, v4.1): Works the same
await brain.import(file)
// After (v4.2.0+): Streaming always on, zero config
// After: Streaming always on, zero config
await brain.import(file)
```

397
docs/operations/capacity-planning.md
View file

@ -1,6 +1,6 @@
# Capacity Planning & Operations Guide
**Brainy v3.36.0+ Enterprise Operations**
**Brainy Enterprise Operations**
This guide provides production-ready capacity planning formulas, deployment strategies, and operational guidelines for scaling Brainy from development (2GB) to enterprise (128GB+) deployments.
@ -24,13 +24,13 @@ Where:
### Adaptive Caching Strategy
```
estimatedVectorMemory = entityCount × 1536 bytes // 384 dims × 4 bytes per float
hnswCacheBudget = cacheSize × 0.80 // 80% threshold for preloading decision
estimatedVectorMemory = entityCount × 1536 bytes // 384 dims × 4 bytes per float
hnswCacheBudget = cacheSize × 0.80 // 80% threshold for preloading decision
if estimatedVectorMemory < hnswCacheBudget:
cachingStrategy = 'preloaded' // All vectors loaded at init
cachingStrategy = 'preloaded' // All vectors loaded at init
else:
cachingStrategy = 'on-demand' // Vectors loaded adaptively via UnifiedCache
cachingStrategy = 'on-demand' // Vectors loaded adaptively via UnifiedCache
```
---
@ -46,19 +46,19 @@ else:
**Memory Breakdown:**
```
System Memory: 2048 MB
OS Reserved (20%): -410 MB
Available: 1638 MB
Model Memory: -140 MB
├─ WASM + Weights: 90 MB
└─ Workspace: 50 MB
System Memory: 2048 MB
OS Reserved (20%): -410 MB
Available: 1638 MB
Model Memory: -140 MB
├─ WASM + Weights: 90 MB
└─ Workspace: 50 MB
───────────────────────────
Available for Cache: 1488 MB
Available for Cache: 1488 MB
Dev Allocation (25%): 372 MB UnifiedCache
├─ HNSW (30%): 112 MB
├─ Metadata (40%): 149 MB
├─ Search (20%): 74 MB
└─ Shared (10%): 37 MB
├─ HNSW (30%): 112 MB
├─ Metadata (40%): 149 MB
├─ Search (20%): 74 MB
└─ Shared (10%): 37 MB
```
**Capacity:**
@ -75,9 +75,9 @@ Dev Allocation (25%): 372 MB UnifiedCache
**Configuration:**
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', path: './brainy-data' },
model: { precision: 'q8' },
cache: { /* auto-sized to 372MB */ }
storage: { type: 'filesystem', path: './brainy-data' },
model: { precision: 'q8' },
cache: { /* auto-sized to 372MB */ }
})
```
@ -92,17 +92,17 @@ const brain = new Brainy({
**Memory Breakdown:**
```
System Memory: 8192 MB
OS Reserved (20%): -1638 MB
Available: 6554 MB
Model Memory (Q8): -150 MB
System Memory: 8192 MB
OS Reserved (20%): -1638 MB
Available: 6554 MB
Model Memory (Q8): -150 MB
───────────────────────────
Available for Cache: 6404 MB
Available for Cache: 6404 MB
Prod Allocation (50%): 3202 MB UnifiedCache
├─ HNSW (30%): 961 MB
├─ Metadata (40%): 1281 MB
├─ Search (20%): 640 MB
└─ Shared (10%): 320 MB
├─ HNSW (30%): 961 MB
├─ Metadata (40%): 1281 MB
├─ Search (20%): 640 MB
└─ Shared (10%): 320 MB
```
**Capacity:**
@ -119,9 +119,9 @@ Prod Allocation (50%): 3202 MB UnifiedCache
**Configuration:**
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', path: '/var/lib/brainy' },
model: { precision: 'q8' },
// Auto-sized cache: 3202MB
storage: { type: 'filesystem', path: '/var/lib/brainy' },
model: { precision: 'q8' },
// Auto-sized cache: 3202MB
})
// Monitor health
@ -141,17 +141,17 @@ console.log(`Caching strategy: ${stats.cachingStrategy}`)
**Memory Breakdown:**
```
System Memory: 32768 MB
OS Reserved (20%): -6554 MB
Available: 26214 MB
Model Memory (Q8): -150 MB
System Memory: 32768 MB
OS Reserved (20%): -6554 MB
Available: 26214 MB
Model Memory (Q8): -150 MB
───────────────────────────
Available for Cache: 26064 MB
Available for Cache: 26064 MB
Prod Allocation (50%): 13032 MB UnifiedCache
├─ HNSW (30%): 3910 MB
├─ Metadata (40%): 5213 MB
├─ Search (20%): 2606 MB
└─ Shared (10%): 1303 MB
├─ HNSW (30%): 3910 MB
├─ Metadata (40%): 5213 MB
├─ Search (20%): 2606 MB
└─ Shared (10%): 1303 MB
```
**Capacity:**
@ -168,11 +168,11 @@ Prod Allocation (50%): 13032 MB UnifiedCache
**Configuration:**
```typescript
const brain = new Brainy({
storage: {
type: 'gcs-native',
gcsNativeStorage: { bucketName: 'production-data' }
},
model: { precision: 'q8' } // or 'fp32' for +0.5% accuracy
storage: {
type: 'gcs-native',
gcsNativeStorage: { bucketName: 'production-data' }
},
model: { precision: 'q8' } // or 'fp32' for +0.5% accuracy
})
// Verify allocation
@ -192,17 +192,17 @@ console.log(`Environment: ${memoryInfo.memoryInfo.environment}`)
**Memory Breakdown:**
```
System Memory: 131072 MB
OS Reserved (20%): -26214 MB
Available: 104858 MB
Model Memory (FP32): -250 MB
System Memory: 131072 MB
OS Reserved (20%): -26214 MB
Available: 104858 MB
Model Memory (FP32): -250 MB
───────────────────────────────
Available for Cache: 104608 MB
Available for Cache: 104608 MB
Prod Allocation (50%): 52304 MB UnifiedCache (logarithmic scaling applies)
├─ HNSW (30%): 15691 MB
├─ Metadata (40%): 20922 MB
├─ Search (20%): 10461 MB
└─ Shared (10%): 5230 MB
├─ HNSW (30%): 15691 MB
├─ Metadata (40%): 20922 MB
├─ Search (20%): 10461 MB
└─ Shared (10%): 5230 MB
```
**Logarithmic Scaling Applied:**
@ -227,30 +227,30 @@ Actual cache size: ~40GB (prevents waste on 128GB systems)
**Configuration:**
```typescript
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'enterprise-data',
region: 'us-east-1'
}
},
model: { precision: 'fp32' } // Maximum accuracy
storage: {
type: 's3',
s3Storage: {
bucketName: 'enterprise-data',
region: 'us-east-1'
}
},
model: { precision: 'fp32' } // Maximum accuracy
})
// Enterprise monitoring
setInterval(() => {
const stats = brain.hnsw.getCacheStats()
const stats = brain.hnsw.getCacheStats()
if (stats.fairness.fairnessViolation) {
console.warn('FAIRNESS VIOLATION: HNSW using too much cache')
console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access, ${stats.hnswCache.sizePercent}% size`)
}
if (stats.fairness.fairnessViolation) {
console.warn('FAIRNESS VIOLATION: HNSW using too much cache')
console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access, ${stats.hnswCache.sizePercent}% size`)
}
if (stats.unifiedCache.hitRatePercent < 75) {
console.warn(`Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`)
console.warn('Recommendations:', stats.recommendations)
}
}, 60000) // Check every minute
if (stats.unifiedCache.hitRatePercent < 75) {
console.warn(`Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`)
console.warn('Recommendations:', stats.recommendations)
}
}, 60000) // Check every minute
```
---
@ -263,12 +263,12 @@ Brainy auto-detects container memory limits via cgroups v1/v2:
```typescript
// Automatic detection
const brain = new Brainy() // Detects cgroup limits automatically
const brain = new Brainy() // Detects cgroup limits automatically
// Verify detection
const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
console.log(`Container: ${memoryInfo.memoryInfo.isContainer}`)
console.log(`Source: ${memoryInfo.memoryInfo.source}`) // 'cgroup-v2' or 'cgroup-v1'
console.log(`Source: ${memoryInfo.memoryInfo.source}`) // 'cgroup-v2' or 'cgroup-v1'
console.log(`Limit: ${Math.round(memoryInfo.memoryInfo.available / 1024 / 1024)}MB`)
```
@ -294,37 +294,37 @@ CMD ["node", "dist/index.js"]
```bash
docker run \
--memory="2g" \
--memory-reservation="1.5g" \
--cpus="2" \
my-brainy-app
--memory="2g" \
--memory-reservation="1.5g" \
--cpus="2" \
my-brainy-app
```
**Expected allocation:**
```
Container Limit: 2048 MB
Available: 1638 MB (80% usable)
Model Memory: -150 MB
Available for Cache: 1488 MB
Container Limit: 2048 MB
Available: 1638 MB (80% usable)
Model Memory: -150 MB
Available for Cache: 1488 MB
Container Ratio (40%): 595 MB UnifiedCache
```
**Medium Container (8GB)**
```bash
docker run \
--memory="8g" \
--memory-reservation="6g" \
--cpus="4" \
-e NODE_OPTIONS="--max-old-space-size=6144" \
my-brainy-app
--memory="8g" \
--memory-reservation="6g" \
--cpus="4" \
-e NODE_OPTIONS="--max-old-space-size=6144" \
my-brainy-app
```
**Expected allocation:**
```
Container Limit: 8192 MB
Available: 6554 MB
Model Memory: -150 MB
Available for Cache: 6404 MB
Container Limit: 8192 MB
Available: 6554 MB
Model Memory: -150 MB
Available for Cache: 6404 MB
Container Ratio (40%): 2562 MB UnifiedCache
```
@ -335,24 +335,24 @@ Container Ratio (40%): 2562 MB UnifiedCache
apiVersion: apps/v1
kind: Deployment
metadata:
name: brainy-api
name: brainy-api
spec:
replicas: 3
template:
spec:
containers:
- name: brainy
image: my-brainy-app:latest
resources:
requests:
memory: "1.5Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=1536"
replicas: 3
template:
spec:
containers:
- name: brainy
image: my-brainy-app:latest
resources:
requests:
memory: "1.5Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=1536"
```
**Medium Pod (8GB)**
@ -360,24 +360,24 @@ spec:
apiVersion: apps/v1
kind: Deployment
metadata:
name: brainy-api
name: brainy-api
spec:
replicas: 2
template:
spec:
containers:
- name: brainy
image: my-brainy-app:latest
resources:
requests:
memory: "6Gi"
cpu: "2000m"
limits:
memory: "8Gi"
cpu: "4000m"
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=6144"
replicas: 2
template:
spec:
containers:
- name: brainy
image: my-brainy-app:latest
resources:
requests:
memory: "6Gi"
cpu: "2000m"
limits:
memory: "8Gi"
cpu: "4000m"
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=6144"
```
**Best Practices:**
@ -399,15 +399,15 @@ The system automatically chooses the optimal caching strategy:
**Auto-detection logic:**
```typescript
const vectorMemoryNeeded = entityCount × 1536 // bytes
const vectorMemoryNeeded = entityCount × 1536 // bytes
const hnswCacheAvailable = unifiedCache.maxSize × 0.80
if (vectorMemoryNeeded < hnswCacheAvailable) {
// Preload strategy: all vectors loaded at init
console.log('Caching strategy: preloaded (all vectors in memory)')
// Preload strategy: all vectors loaded at init
console.log('Caching strategy: preloaded (all vectors in memory)')
} else {
// On-demand strategy: vectors loaded adaptively
console.log('Caching strategy: on-demand (adaptive loading via UnifiedCache)')
// On-demand strategy: vectors loaded adaptively
console.log('Caching strategy: on-demand (adaptive loading via UnifiedCache)')
}
```
@ -422,12 +422,12 @@ Consider increasing RAM when:
**Decision tree:**
```
If cache hit rate < 70%:
└─> Is working set < 50% of total entities?
├─> YES: Increase cache size (add RAM)
└─> NO: Working set too large, consider:
├─> Application-level caching
├─> Query optimization
└─> Sharding dataset
└─> Is working set < 50% of total entities?
├─> YES: Increase cache size (add RAM)
└─> NO: Working set too large, consider:
├─> Application-level caching
├─> Query optimization
└─> Sharding dataset
```
### When to Shard/Distribute
@ -442,17 +442,17 @@ Consider sharding when:
```typescript
// Example: Geographic sharding
const usEastBrain = new Brainy({
storage: { type: 's3', s3Storage: { bucket: 'us-east-data' } }
storage: { type: 's3', s3Storage: { bucket: 'us-east-data' } }
})
const euWestBrain = new Brainy({
storage: { type: 's3', s3Storage: { bucket: 'eu-west-data' } }
storage: { type: 's3', s3Storage: { bucket: 'eu-west-data' } }
})
// Route queries based on user location
async function search(query, userRegion) {
const brain = userRegion === 'US' ? usEastBrain : euWestBrain
return await brain.search(query)
const brain = userRegion === 'US' ? usEastBrain : euWestBrain
return await brain.search(query)
}
```
@ -482,7 +482,7 @@ console.log(`Pressure: ${memoryInfo.currentPressure.pressure}`)
// Values: 'low', 'moderate', 'high', 'critical'
if (memoryInfo.currentPressure.warnings.length > 0) {
console.warn('Memory warnings:', memoryInfo.currentPressure.warnings)
console.warn('Memory warnings:', memoryInfo.currentPressure.warnings)
}
```
@ -491,9 +491,9 @@ if (memoryInfo.currentPressure.warnings.length > 0) {
const stats = brain.hnsw.getCacheStats()
if (stats.fairness.fairnessViolation) {
console.warn('Cache fairness violation detected')
console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access`)
console.warn(`HNSW: ${stats.hnswCache.sizePercent}% of cache`)
console.warn('Cache fairness violation detected')
console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access`)
console.warn(`HNSW: ${stats.hnswCache.sizePercent}% of cache`)
}
```
@ -502,7 +502,7 @@ if (stats.fairness.fairnessViolation) {
// Track search latency
console.time('search')
const results = await brain.search('query')
console.timeEnd('search') // Target: <10ms for hot queries
console.timeEnd('search') // Target: <10ms for hot queries
```
### Alerting Thresholds
@ -516,26 +516,26 @@ Set up alerts for:
**Example monitoring script:**
```typescript
async function monitorHealth() {
const stats = brain.hnsw.getCacheStats()
const stats = brain.hnsw.getCacheStats()
// Alert on low cache hit rate
if (stats.unifiedCache.hitRatePercent < 70) {
await sendAlert({
severity: 'warning',
message: `Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`,
recommendations: stats.recommendations
})
}
// Alert on low cache hit rate
if (stats.unifiedCache.hitRatePercent < 70) {
await sendAlert({
severity: 'warning',
message: `Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`,
recommendations: stats.recommendations
})
}
// Alert on memory pressure
const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
if (memoryInfo.currentPressure.pressure === 'high') {
await sendAlert({
severity: 'critical',
message: 'High memory pressure detected',
warnings: memoryInfo.currentPressure.warnings
})
}
// Alert on memory pressure
const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
if (memoryInfo.currentPressure.pressure === 'high') {
await sendAlert({
severity: 'critical',
message: 'High memory pressure detected',
warnings: memoryInfo.currentPressure.warnings
})
}
}
// Run every 60 seconds
@ -552,9 +552,9 @@ setInterval(monitorHealth, 60000)
**Sizing:**
```
Products: 500,000
Vector memory needed: 500K × 1536 bytes = 768 MB
HNSW cache available: (16GB × 0.8 - 150MB) × 0.5 × 0.3 = 1,915 MB
Products: 500,000
Vector memory needed: 500K × 1536 bytes = 768 MB
HNSW cache available: (16GB × 0.8 - 150MB) × 0.5 × 0.3 = 1,915 MB
Result: Standard mode (all vectors fit in HNSW cache)
```
@ -562,16 +562,16 @@ Result: Standard mode (all vectors fit in HNSW cache)
**Configuration:**
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', path: '/var/lib/brainy' },
model: { precision: 'q8' }
storage: { type: 'filesystem', path: '/var/lib/brainy' },
model: { precision: 'q8' }
})
await brain.init()
// Verify preloaded strategy (all vectors in memory)
const stats = brain.hnsw.getCacheStats()
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'preloaded'
console.log(`Search latency: ${stats.performance.avgSearchMs}ms`) // ~3ms
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'preloaded'
console.log(`Search latency: ${stats.performance.avgSearchMs}ms`) // ~3ms
```
### Example 2: Document Search (5M documents)
@ -580,9 +580,9 @@ console.log(`Search latency: ${stats.performance.avgSearchMs}ms`) // ~3ms
**Sizing:**
```
Documents: 5,000,000
Vector memory needed: 5M × 1536 bytes = 7,680 MB
HNSW cache available: (32GB × 0.8 - 150MB) × 0.5 × 0.3 = 3,910 MB
Documents: 5,000,000
Vector memory needed: 5M × 1536 bytes = 7,680 MB
HNSW cache available: (32GB × 0.8 - 150MB) × 0.5 × 0.3 = 3,910 MB
Result: On-demand caching (vectors loaded adaptively)
```
@ -590,20 +590,20 @@ Result: On-demand caching (vectors loaded adaptively)
**Configuration:**
```typescript
const brain = new Brainy({
storage: {
type: 'gcs-native',
gcsNativeStorage: { bucketName: 'docs-production' }
},
model: { precision: 'q8' }
storage: {
type: 'gcs-native',
gcsNativeStorage: { bucketName: 'docs-production' }
},
model: { precision: 'q8' }
})
await brain.init()
// Monitor cache performance
const stats = brain.hnsw.getCacheStats()
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand'
console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >80%
console.log(`Cold search latency: ${stats.performance.avgSearchMs}ms`) // ~12ms
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand'
console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >80%
console.log(`Cold search latency: ${stats.performance.avgSearchMs}ms`) // ~12ms
// Recommendations
console.log('Recommendations:', stats.recommendations)
@ -616,9 +616,9 @@ console.log('Recommendations:', stats.recommendations)
**Sizing:**
```
Entities: 20,000,000
Vector memory needed: 20M × 1536 bytes = 30,720 MB
HNSW cache available: ~15,691 MB (after logarithmic scaling)
Entities: 20,000,000
Vector memory needed: 20M × 1536 bytes = 30,720 MB
HNSW cache available: ~15,691 MB (after logarithmic scaling)
Result: On-demand caching with high-performance adaptive loading
```
@ -626,14 +626,14 @@ Result: On-demand caching with high-performance adaptive loading
**Configuration:**
```typescript
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'knowledge-graph-prod',
region: 'us-east-1'
}
},
model: { precision: 'fp32' } // Maximum accuracy
storage: {
type: 's3',
s3Storage: {
bucketName: 'knowledge-graph-prod',
region: 'us-east-1'
}
},
model: { precision: 'fp32' } // Maximum accuracy
})
await brain.init()
@ -641,13 +641,13 @@ await brain.init()
// Enterprise monitoring
const stats = brain.hnsw.getCacheStats()
console.log(`Entities: ${stats.autoDetection.entityCount.toLocaleString()}`)
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand'
console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >85%
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand'
console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >85%
console.log(`HNSW cache: ${stats.hnswCache.estimatedMemoryMB}MB`)
// Fairness check
if (stats.fairness.fairnessViolation) {
console.warn('HNSW dominating cache - consider tuning eviction policies')
console.warn('HNSW dominating cache - consider tuning eviction policies')
}
```
@ -690,8 +690,8 @@ console.log(`Warnings:`, memoryInfo.currentPressure.warnings)
```typescript
const stats = brain.hnsw.getCacheStats()
if (stats.fairness.fairnessViolation) {
console.log(`HNSW access: ${stats.fairness.hnswAccessPercent}%`)
console.log(`HNSW cache: ${stats.hnswCache.sizePercent}%`)
console.log(`HNSW access: ${stats.fairness.hnswAccessPercent}%`)
console.log(`HNSW cache: ${stats.hnswCache.sizePercent}%`)
}
```
@ -704,8 +704,7 @@ if (stats.fairness.fairnessViolation) {
## 📚 Additional Resources
- **[Migration Guide](../guides/migration-3.36.0.md)** - Upgrading to v3.36.0
- **[Architecture Overview](../architecture/data-storage-architecture.md)** - Deep dive into storage and caching
- **[Migration Guide](../guides/migration-3.36.0.md)** - Upgrading to **[Architecture Overview](../architecture/data-storage-architecture.md)** - Deep dive into storage and caching
- **[GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)** - Report problems or ask questions
---

170
docs/operations/cost-optimization-aws-s3.md
View file

@ -1,10 +1,9 @@
# AWS S3 Cost Optimization Guide for Brainy v4.0.0
# AWS S3 Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $138k/year to $5.9k/year at 500TB scale (**96% savings**)
## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance.
Brainy provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance.
## Cost Breakdown (Before Optimization)
@ -39,10 +38,10 @@ import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
// Initialize Brainy with S3 storage
const storage = new S3CompatibleStorage({
bucket: 'my-brainy-data',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
bucket: 'my-brainy-data',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
})
const brain = new Brainy({ storage })
@ -50,24 +49,24 @@ await brain.init()
// Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({
rules: [{
id: 'optimize-vectors',
prefix: 'entities/nouns/vectors/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // Infrequent Access after 30 days
{ days: 90, storageClass: 'GLACIER' }, // Glacier after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 1 year
]
}, {
id: 'optimize-metadata',
prefix: 'entities/nouns/metadata/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' },
{ days: 180, storageClass: 'GLACIER' }
]
}]
rules: [{
id: 'optimize-vectors',
prefix: 'entities/nouns/vectors/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // Infrequent Access after 30 days
{ days: 90, storageClass: 'GLACIER' }, // Glacier after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 1 year
]
}, {
id: 'optimize-metadata',
prefix: 'entities/nouns/metadata/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' },
{ days: 180, storageClass: 'GLACIER' }
]
}]
})
// Verify lifecycle policy
@ -84,10 +83,10 @@ console.log('Lifecycle policy active:', policy.rules.length, 'rules')
- 10% of data 365+ days old (Deep Archive)
```
Standard (200TB): 200TB × $0.023/GB × 12 = $55,200/year
Standard-IA (150TB): 150TB × $0.0125/GB × 12 = $22,500/year
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Deep Archive (50TB): 50TB × $0.00099/GB × 12 = $594/year
Standard (200TB): 200TB × $0.023/GB × 12 = $55,200/year
Standard-IA (150TB): 150TB × $0.0125/GB × 12 = $22,500/year
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Deep Archive (50TB): 50TB × $0.00099/GB × 12 = $594/year
Total Storage Cost: $83,094/year (instead of $138,000)
Total with Operations: ~$88,000/year
@ -133,11 +132,11 @@ Intelligent-Tiering automatically moves objects between:
- 30% Deep Archive Access
```
Frequent (75TB): 75TB × $0.023/GB × 12 = $20,700/year
Infrequent (100TB): 100TB × $0.0125/GB × 12 = $15,000/year
Archive (175TB): 175TB × $0.004/GB × 12 = $8,400/year
Deep Archive (150TB): 150TB × $0.00099/GB × 12 = $1,782/year
Monitoring: ~$300/year (minimal)
Frequent (75TB): 75TB × $0.023/GB × 12 = $20,700/year
Infrequent (100TB): 100TB × $0.0125/GB × 12 = $15,000/year
Archive (175TB): 175TB × $0.004/GB × 12 = $8,400/year
Deep Archive (150TB): 150TB × $0.00099/GB × 12 = $1,782/year
Monitoring: ~$300/year (minimal)
Total Storage Cost: $46,182/year
Total with Operations: ~$51,000/year
@ -157,21 +156,21 @@ await storage.enableIntelligentTiering('entities/verbs/vectors/', 'verbs-auto')
// Set lifecycle policy for metadata (less frequently accessed)
await storage.setLifecyclePolicy({
rules: [{
id: 'archive-old-metadata',
prefix: 'entities/nouns/metadata/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' },
{ days: 60, storageClass: 'GLACIER' },
{ days: 180, storageClass: 'DEEP_ARCHIVE' }
]
}, {
id: 'cleanup-old-system-data',
prefix: '_system/',
status: 'Enabled',
expiration: { days: 365 } // Delete old statistics
}]
rules: [{
id: 'archive-old-metadata',
prefix: 'entities/nouns/metadata/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' },
{ days: 60, storageClass: 'GLACIER' },
{ days: 180, storageClass: 'DEEP_ARCHIVE' }
]
}, {
id: 'cleanup-old-system-data',
prefix: '_system/',
status: 'Enabled',
expiration: { days: 365 } // Delete old statistics
}]
})
```
@ -179,19 +178,19 @@ await storage.setLifecyclePolicy({
**Vectors (300TB with Intelligent-Tiering):**
```
Frequent (45TB): 45TB × $0.023/GB × 12 = $12,420/year
Infrequent (60TB): 60TB × $0.0125/GB × 12 = $9,000/year
Archive (105TB): 105TB × $0.004/GB × 12 = $5,040/year
Deep Archive (90TB): 90TB × $0.00099/GB × 12 = $1,069/year
Frequent (45TB): 45TB × $0.023/GB × 12 = $12,420/year
Infrequent (60TB): 60TB × $0.0125/GB × 12 = $9,000/year
Archive (105TB): 105TB × $0.004/GB × 12 = $5,040/year
Deep Archive (90TB): 90TB × $0.00099/GB × 12 = $1,069/year
Subtotal: $27,529/year
```
**Metadata (200TB with Lifecycle Policy):**
```
Standard (60TB): 60TB × $0.023/GB × 12 = $16,560/year
Standard-IA (40TB): 40TB × $0.0125/GB × 12 = $6,000/year
Glacier (60TB): 60TB × $0.004/GB × 12 = $2,880/year
Deep Archive (40TB): 40TB × $0.00099/GB × 12 = $475/year
Standard (60TB): 60TB × $0.023/GB × 12 = $16,560/year
Standard-IA (40TB): 40TB × $0.0125/GB × 12 = $6,000/year
Glacier (60TB): 60TB × $0.004/GB × 12 = $2,880/year
Deep Archive (40TB): 40TB × $0.00099/GB × 12 = $475/year
Subtotal: $25,915/year
```
@ -206,16 +205,16 @@ Subtotal: $25,915/year
```typescript
await storage.setLifecyclePolicy({
rules: [{
id: 'aggressive-archival',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 14, storageClass: 'STANDARD_IA' }, // IA after 2 weeks
{ days: 30, storageClass: 'GLACIER' }, // Glacier after 1 month
{ days: 90, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 3 months
]
}]
rules: [{
id: 'aggressive-archival',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 14, storageClass: 'STANDARD_IA' }, // IA after 2 weeks
{ days: 30, storageClass: 'GLACIER' }, // Glacier after 1 month
{ days: 90, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 3 months
]
}]
})
```
@ -223,10 +222,10 @@ await storage.setLifecyclePolicy({
**After 1 year:**
```
Standard (50TB): 50TB × $0.023/GB × 12 = $13,800/year
Standard-IA (50TB): 50TB × $0.0125/GB × 12 = $7,500/year
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Deep Archive (300TB): 300TB × $0.00099/GB × 12 = $3,564/year
Standard (50TB): 50TB × $0.023/GB × 12 = $13,800/year
Standard-IA (50TB): 50TB × $0.0125/GB × 12 = $7,500/year
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Deep Archive (300TB): 300TB × $0.00099/GB × 12 = $3,564/year
Total Storage Cost: $29,664/year
Total with Operations: ~$34,000/year
@ -250,16 +249,16 @@ Note: Retrieval costs may be significant if archived data is accessed frequently
### Efficient Cleanup
```typescript
// v4.0.0: Batch delete (1000 objects per request)
// Batch delete (1000 objects per request)
const idsToDelete = [/* array of entity IDs */]
// Generate paths for both vector and metadata files
const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
})
// Batch delete (much faster and cheaper than individual deletes)
@ -280,14 +279,14 @@ console.log('Active rules:', policy.rules)
// Example output:
// {
// rules: [
// {
// id: 'optimize-vectors',
// prefix: 'entities/nouns/vectors/',
// status: 'Enabled',
// transitions: [...]
// }
// ]
// rules: [
// {
// id: 'optimize-vectors',
// prefix: 'entities/nouns/vectors/',
// status: 'Enabled',
// transitions: [...]
// }
// ]
// }
```
@ -396,6 +395,5 @@ await storage.enableIntelligentTiering('entities/', 'new-config')
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: AWS S3

306
docs/operations/cost-optimization-azure.md
View file

@ -1,10 +1,9 @@
# Azure Blob Storage Cost Optimization Guide for Brainy v4.0.0
# Azure Blob Storage Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $107k/year to $5k/year at 500TB scale (**95% savings**)
## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for Azure Blob Storage, including manual tier management, lifecycle policies, and batch operations.
Brainy provides enterprise-grade cost optimization features for Azure Blob Storage, including manual tier management, lifecycle policies, and batch operations.
## Cost Breakdown (Before Optimization)
@ -41,8 +40,8 @@ import { AzureBlobStorage } from '@soulcraft/brainy/storage'
// Initialize Brainy with Azure storage
const storage = new AzureBlobStorage({
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
})
const brain = new Brainy({ storage })
@ -50,19 +49,19 @@ await brain.init()
// Change tier for a single blob
await storage.changeBlobTier(
'entities/nouns/vectors/00/00123456-uuid.json',
'Cool'
'entities/nouns/vectors/00/00123456-uuid.json',
'Cool'
)
// Batch tier changes (efficient for thousands of blobs)
const blobsToMove = [
'entities/nouns/vectors/01/...',
'entities/nouns/vectors/02/...',
// ... up to thousands of blobs
'entities/nouns/vectors/01/...',
'entities/nouns/vectors/02/...',
// ... up to thousands of blobs
]
await storage.batchChangeTier(blobsToMove, 'Cool') // Hot → Cool
await storage.batchChangeTier(oldBlobs, 'Archive') // Cool → Archive
await storage.batchChangeTier(blobsToMove, 'Cool') // Hot → Cool
await storage.batchChangeTier(oldBlobs, 'Archive') // Cool → Archive
```
### Immediate Cost Impact
@ -90,54 +89,54 @@ Savings: $20,346/year (95% savings on moved data)
```typescript
// Set lifecycle policy for automatic tier management
await storage.setLifecyclePolicy({
rules: [{
name: 'optimizeVectors',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/vectors/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}, {
name: 'optimizeMetadata',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/metadata/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 180 }
}
}
}
}, {
name: 'cleanupOldSystemFiles',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['_system/']
},
actions: {
baseBlob: {
delete: { daysAfterModificationGreaterThan: 365 }
}
}
}
}]
rules: [{
name: 'optimizeVectors',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/vectors/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}, {
name: 'optimizeMetadata',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/metadata/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 180 }
}
}
}
}, {
name: 'cleanupOldSystemFiles',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['_system/']
},
actions: {
baseBlob: {
delete: { daysAfterModificationGreaterThan: 365 }
}
}
}
}]
})
// Verify lifecycle policy
@ -153,9 +152,9 @@ console.log('Lifecycle policy active:', policy.rules.length, 'rules')
- 30% of data in Archive tier (90+ days old)
```
Hot (150TB): 150TB × $0.0184/GB × 12 = $32,256/year
Cool (200TB): 200TB × $0.0115/GB × 12 = $26,880/year
Archive (150TB): 150TB × $0.00099/GB × 12 = $1,732/year
Hot (150TB): 150TB × $0.0184/GB × 12 = $32,256/year
Cool (200TB): 200TB × $0.0115/GB × 12 = $26,880/year
Archive (150TB): 150TB × $0.00099/GB × 12 = $1,732/year
Total Storage Cost: $60,868/year
Total with Operations: ~$65,500/year
@ -170,23 +169,23 @@ Savings: $47,000/year (42%)
```typescript
await storage.setLifecyclePolicy({
rules: [{
name: 'aggressiveArchival',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 14 },
tierToArchive: { daysAfterModificationGreaterThan: 30 }
}
}
}
}]
rules: [{
name: 'aggressiveArchival',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 14 },
tierToArchive: { daysAfterModificationGreaterThan: 30 }
}
}
}
}]
})
```
@ -194,9 +193,9 @@ await storage.setLifecyclePolicy({
**After 6 months:**
```
Hot (50TB): 50TB × $0.0184/GB × 12 = $10,752/year
Cool (100TB): 100TB × $0.0115/GB × 12 = $13,440/year
Archive (350TB): 350TB × $0.00099/GB × 12 = $4,039/year
Hot (50TB): 50TB × $0.0184/GB × 12 = $10,752/year
Cool (100TB): 100TB × $0.0115/GB × 12 = $13,440/year
Archive (350TB): 350TB × $0.00099/GB × 12 = $4,039/year
Total Storage Cost: $28,231/year
Total with Operations: ~$33,000/year
@ -211,41 +210,41 @@ Warning: Archive rehydration takes 1-15 hours
```typescript
await storage.setLifecyclePolicy({
rules: [{
// Vectors: Keep in Hot/Cool for search performance
name: 'vectors-moderate',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/vectors/', 'entities/verbs/vectors/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 60 }
// Don't archive vectors - keep searchable
}
}
}
}, {
// Metadata: Aggressive archival
name: 'metadata-aggressive',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/metadata/', 'entities/verbs/metadata/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}]
rules: [{
// Vectors: Keep in Hot/Cool for search performance
name: 'vectors-moderate',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/vectors/', 'entities/verbs/vectors/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 60 }
// Don't archive vectors - keep searchable
}
}
}
}, {
// Metadata: Aggressive archival
name: 'metadata-aggressive',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/metadata/', 'entities/verbs/metadata/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}]
})
```
@ -253,16 +252,16 @@ await storage.setLifecyclePolicy({
**Vectors (300TB):**
```
Hot (90TB): 90TB × $0.0184/GB × 12 = $19,354/year
Cool (210TB): 210TB × $0.0115/GB × 12 = $28,980/year
Hot (90TB): 90TB × $0.0184/GB × 12 = $19,354/year
Cool (210TB): 210TB × $0.0115/GB × 12 = $28,980/year
Subtotal: $48,334/year
```
**Metadata (200TB):**
```
Hot (30TB): 30TB × $0.0184/GB × 12 = $6,451/year
Cool (70TB): 70TB × $0.0115/GB × 12 = $9,660/year
Archive (100TB): 100TB × $0.00099/GB × 12 = $1,158/year
Hot (30TB): 30TB × $0.0184/GB × 12 = $6,451/year
Cool (70TB): 70TB × $0.0115/GB × 12 = $9,660/year
Archive (100TB): 100TB × $0.00099/GB × 12 = $1,158/year
Subtotal: $17,269/year
```
@ -288,8 +287,8 @@ Subtotal: $17,269/year
```typescript
// Rehydrate blob from Archive to Hot (high priority)
await storage.rehydrateBlob(
'entities/nouns/vectors/00/00123456-uuid.json',
'High' // 'Standard' or 'High' priority
'entities/nouns/vectors/00/00123456-uuid.json',
'High' // 'Standard' or 'High' priority
)
// Rehydration time:
@ -309,7 +308,7 @@ console.log('Archive status:', metadata.archiveStatus)
const blobsToRehydrate = [/* array of blob paths */]
for (const blobPath of blobsToRehydrate) {
await storage.rehydrateBlob(blobPath, 'High')
await storage.rehydrateBlob(blobPath, 'High')
}
// Wait for rehydration to complete (1-15 hours)
@ -333,15 +332,15 @@ Examples:
### Efficient Bulk Deletions
```typescript
// v4.0.0: Batch delete (256 blobs per request)
// Batch delete (256 blobs per request)
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
})
// Batch delete via BlobBatchClient
@ -375,14 +374,14 @@ console.log('Active rules:', policy.rules)
// Example output:
// {
// rules: [
// {
// name: 'optimizeVectors',
// enabled: true,
// type: 'Lifecycle',
// definition: {...}
// }
// ]
// rules: [
// {
// name: 'optimizeVectors',
// enabled: true,
// type: 'Lifecycle',
// definition: {...}
// }
// ]
// }
```
@ -452,8 +451,8 @@ Monthly cost trend: Decreasing 5-8% per month as data transitions
// Check lifecycle policy status
const policy = await storage.getLifecyclePolicy()
console.log('Policy rules:', policy.rules.map(r => ({
name: r.name,
enabled: r.enabled
name: r.name,
enabled: r.enabled
})))
// Azure lifecycle policies run once per day
@ -471,9 +470,9 @@ await storage.rehydrateBlob(blobPath, 'High')
// Wait for rehydration (check status)
let status
do {
await new Promise(resolve => setTimeout(resolve, 60000)) // Wait 1 minute
const metadata = await storage.getBlobMetadata(blobPath)
status = metadata.archiveStatus
await new Promise(resolve => setTimeout(resolve, 60000)) // Wait 1 minute
const metadata = await storage.getBlobMetadata(blobPath)
status = metadata.archiveStatus
} while (status && status.includes('pending'))
// Now access the blob
@ -494,8 +493,8 @@ const data = await storage.get(blobPath)
```typescript
const storage = new AzureBlobStorage({
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
})
```
@ -503,9 +502,9 @@ const storage = new AzureBlobStorage({
```typescript
const storage = new AzureBlobStorage({
accountName: process.env.AZURE_STORAGE_ACCOUNT,
accountKey: process.env.AZURE_STORAGE_KEY,
containerName: 'brainy-data'
accountName: process.env.AZURE_STORAGE_ACCOUNT,
accountKey: process.env.AZURE_STORAGE_KEY,
containerName: 'brainy-data'
})
```
@ -513,9 +512,9 @@ const storage = new AzureBlobStorage({
```typescript
const storage = new AzureBlobStorage({
sasToken: process.env.AZURE_STORAGE_SAS_TOKEN,
accountName: process.env.AZURE_STORAGE_ACCOUNT,
containerName: 'brainy-data'
sasToken: process.env.AZURE_STORAGE_SAS_TOKEN,
accountName: process.env.AZURE_STORAGE_ACCOUNT,
containerName: 'brainy-data'
})
```
@ -549,6 +548,5 @@ const storage = new AzureBlobStorage({
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: Azure Blob Storage

162
docs/operations/cost-optimization-cloudflare-r2.md
View file

@ -1,10 +1,9 @@
# Cloudflare R2 Cost Optimization Guide for Brainy v4.0.0
# Cloudflare R2 Cost Optimization Guide for Brainy
> **Cost Impact**: $0 egress fees + 96% storage savings = Lowest cloud storage costs
## Overview
Cloudflare R2 is an S3-compatible object storage with **zero egress fees**, making it ideal for high-traffic applications. Brainy v4.0.0 fully supports R2 with lifecycle policies and batch operations.
Cloudflare R2 is an S3-compatible object storage with **zero egress fees**, making it ideal for high-traffic applications. Brainy fully supports R2 with lifecycle policies and batch operations.
## Cost Breakdown
@ -49,7 +48,7 @@ Savings vs AWS: $156,000/year (62%)
- Cloudflare plans to add infrequent access tiers
**When lifecycle features arrive:**
- Brainy v4.0.0 is already prepared with `setLifecyclePolicy()` support
- Brainy is already prepared with `setLifecyclePolicy()` support
- Will work seamlessly once Cloudflare enables lifecycle management
## Strategy 1: Use R2 Standard (Current Best Practice)
@ -62,11 +61,11 @@ import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
// R2 uses S3-compatible API
const storage = new S3CompatibleStorage({
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'my-brainy-data',
region: 'auto', // R2 uses 'auto' region
accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'my-brainy-data',
region: 'auto', // R2 uses 'auto' region
accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
})
const brain = new Brainy({ storage })
@ -96,23 +95,23 @@ Total: $90,081/year
```typescript
// Cloudflare Worker (runs at edge)
export default {
async fetch(request, env) {
// Initialize Brainy with R2
const storage = new S3CompatibleStorage({
endpoint: env.R2_ENDPOINT,
bucket: env.R2_BUCKET,
accessKeyId: env.R2_ACCESS_KEY,
secretAccessKey: env.R2_SECRET_KEY,
region: 'auto'
})
async fetch(request, env) {
// Initialize Brainy with R2
const storage = new S3CompatibleStorage({
endpoint: env.R2_ENDPOINT,
bucket: env.R2_BUCKET,
accessKeyId: env.R2_ACCESS_KEY,
secretAccessKey: env.R2_SECRET_KEY,
region: 'auto'
})
const brain = new Brainy({ storage })
await brain.init()
const brain = new Brainy({ storage })
await brain.init()
// Process at edge (no origin server needed)
const results = await brain.search(request.query)
return new Response(JSON.stringify(results))
}
// Process at edge (no origin server needed)
const results = await brain.search(request.query)
return new Response(JSON.stringify(results))
}
}
```
@ -121,18 +120,18 @@ export default {
```
R2 Storage (500TB): $90,000/year
Workers (10M requests/day):
- First 100k requests/day: FREE
- Additional 350M requests/month: $1,750/year
- CPU time (50ms avg): $5,000/year
- First 100k requests/day: FREE
- Additional 350M requests/month: $1,750/year
- CPU time (50ms avg): $5,000/year
Total: $96,750/year
vs Traditional Setup (AWS S3 + EC2 + CloudFront):
- S3: $138,000/year
- EC2 (t3.xlarge × 4): $24,000/year
- CloudFront egress: $50,000/year
- Load balancer: $3,000/year
Total: $215,000/year
- S3: $138,000/year
- EC2 (t3.xlarge × 4): $24,000/year
- CloudFront egress: $50,000/year
- Load balancer: $3,000/year
Total: $215,000/year
Savings: $118,250/year (55%)
```
@ -144,20 +143,20 @@ Savings: $118,250/year (55%)
```typescript
// Use R2 for frequently accessed data (zero egress)
const hotStorage = new S3CompatibleStorage({
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'brainy-hot',
region: 'auto',
accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'brainy-hot',
region: 'auto',
accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
})
// Use AWS S3 with Intelligent-Tiering for cold data
const coldStorage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com',
bucket: 'brainy-archive',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
endpoint: 's3.amazonaws.com',
bucket: 'brainy-archive',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
})
// Initialize separate Brainy instances or implement tiering logic
@ -167,17 +166,17 @@ const coldStorage = new S3CompatibleStorage({
```
R2 Hot Data (300TB):
Storage: 300TB × $0.015/GB × 12 = $54,000/year
Egress: $0
Storage: 300TB × $0.015/GB × 12 = $54,000/year
Egress: $0
S3 Cold Data (200TB with lifecycle → Deep Archive):
Storage: 200TB × $0.00099/GB × 12 = $2,376/year
Ops: $1,000/year
Storage: 200TB × $0.00099/GB × 12 = $2,376/year
Ops: $1,000/year
Total: $57,376/year
vs All AWS S3:
- S3 storage + egress: $251,000/year
- S3 storage + egress: $251,000/year
Savings: $193,624/year (77%)
```
@ -187,15 +186,15 @@ Savings: $193,624/year (77%)
### Efficient Bulk Deletions
```typescript
// v4.0.0: R2 supports S3 batch delete API
// R2 supports S3 batch delete API
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
})
// Batch delete (1000 objects per request)
@ -231,10 +230,10 @@ https://storage.yourdomain.com/entities/nouns/vectors/...
```typescript
// Worker triggered on R2 object upload
export default {
async fetch(request, env) {
// Process new objects automatically
// E.g., index new entities, generate thumbnails, etc.
}
async fetch(request, env) {
// Process new objects automatically
// E.g., index new entities, generate thumbnails, etc.
}
}
// Cost: Only pay for Worker execution (no polling needed)
@ -244,7 +243,7 @@ export default {
```typescript
// Generate presigned URL for direct browser uploads
const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry
const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry
// Client uploads directly to R2 (no server bandwidth used)
```
@ -255,7 +254,7 @@ const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry
```typescript
const status = await storage.getStorageStatus()
console.log('Storage type:', status.type) // 's3-compatible'
console.log('Storage type:', status.type) // 's3-compatible'
console.log('Bucket:', status.details.bucket)
console.log('Endpoint:', status.details.endpoint)
```
@ -318,20 +317,20 @@ Egress: Unlimited (always free)
```bash
# Install rclone
brew install rclone # or apt-get install rclone
brew install rclone # or apt-get install rclone
# Configure S3 source
rclone config create s3-source s3 \
access_key_id=$AWS_ACCESS_KEY \
secret_access_key=$AWS_SECRET_KEY \
region=us-east-1
access_key_id=$AWS_ACCESS_KEY \
secret_access_key=$AWS_SECRET_KEY \
region=us-east-1
# Configure R2 destination
rclone config create r2-dest s3 \
access_key_id=$R2_ACCESS_KEY \
secret_access_key=$R2_SECRET_KEY \
endpoint=https://$R2_ACCOUNT_ID.r2.cloudflarestorage.com \
region=auto
access_key_id=$R2_ACCESS_KEY \
secret_access_key=$R2_SECRET_KEY \
endpoint=https://$R2_ACCOUNT_ID.r2.cloudflarestorage.com \
region=auto
# Copy data
rclone copy s3-source:my-bucket r2-dest:my-bucket --progress
@ -361,17 +360,17 @@ ROI: 3.4 months
### Prepared for Future Features
```typescript
// Brainy v4.0.0 is ready for R2 lifecycle features
// Brainy is ready for R2 lifecycle features
await storage.setLifecyclePolicy({
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'INFREQUENT_ACCESS' }, // When available
{ days: 90, storageClass: 'ARCHIVE' }
]
}]
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'INFREQUENT_ACCESS' }, // When available
{ days: 90, storageClass: 'ARCHIVE' }
]
}]
})
// Expected cost impact (when lifecycle is available):
@ -388,11 +387,11 @@ await storage.setLifecyclePolicy({
```typescript
// Ensure correct endpoint format
const storage = new S3CompatibleStorage({
endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
// NOT: `https://r2.cloudflarestorage.com/${accountId}`
endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
// NOT: `https://r2.cloudflarestorage.com/${accountId}`
region: 'auto', // R2 requires 'auto'
forcePathStyle: false // R2 uses virtual-hosted-style
region: 'auto', // R2 requires 'auto'
forcePathStyle: false // R2 uses virtual-hosted-style
})
```
@ -446,7 +445,6 @@ const storage = new S3CompatibleStorage({
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: Cloudflare R2
**Key Advantage**: **$0 egress fees forever**

142
docs/operations/cost-optimization-gcs.md
View file

@ -1,10 +1,9 @@
# Google Cloud Storage Cost Optimization Guide for Brainy v4.0.0
# Google Cloud Storage Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $138k/year to $8.3k/year at 500TB scale (**94% savings**)
## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management.
Brainy provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management.
## Cost Breakdown (Before Optimization)
@ -35,8 +34,8 @@ import { GcsStorage } from '@soulcraft/brainy/storage'
// Initialize Brainy with GCS storage
const storage = new GcsStorage({
bucketName: 'my-brainy-data',
keyFilename: './service-account.json' // Or use ADC
bucketName: 'my-brainy-data',
keyFilename: './service-account.json' // Or use ADC
})
const brain = new Brainy({ storage })
@ -44,16 +43,16 @@ await brain.init()
// Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({
rules: [{
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 365 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
rules: [{
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 365 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
})
// Verify lifecycle policy
@ -70,10 +69,10 @@ console.log('Lifecycle policy active:', policy.rules.length, 'rules')
- 10% of data 365+ days old (Archive)
```
Standard (200TB): 200TB × $0.020/GB × 12 = $48,000/year
Nearline (150TB): 150TB × $0.010/GB × 12 = $18,000/year
Coldline (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
Standard (200TB): 200TB × $0.020/GB × 12 = $48,000/year
Nearline (150TB): 150TB × $0.010/GB × 12 = $18,000/year
Coldline (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
Total Storage Cost: $71,520/year
Total with Operations: ~$76,500/year
@ -89,7 +88,7 @@ Savings: $66,500/year (46%)
```typescript
// Enable Autoclass for automatic tier management
await storage.enableAutoclass({
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
})
// Benefits:
@ -116,10 +115,10 @@ await storage.enableAutoclass({
- 40% Archive (cold data)
```
Standard (50TB): 50TB × $0.020/GB × 12 = $12,000/year
Nearline (75TB): 75TB × $0.010/GB × 12 = $9,000/year
Coldline (175TB): 175TB × $0.004/GB × 12 = $8,400/year
Archive (200TB): 200TB × $0.0012/GB × 12 = $2,880/year
Standard (50TB): 50TB × $0.020/GB × 12 = $12,000/year
Nearline (75TB): 75TB × $0.010/GB × 12 = $9,000/year
Coldline (175TB): 175TB × $0.004/GB × 12 = $8,400/year
Archive (200TB): 200TB × $0.0012/GB × 12 = $2,880/year
Total Storage Cost: $32,280/year
Total with Operations: ~$37,000/year
@ -133,24 +132,24 @@ Savings vs Standard: $106,000/year (74%)
```typescript
// Enable Autoclass for vectors (frequently searched)
await storage.enableAutoclass({
terminalStorageClass: 'COLDLINE' // Don't archive vectors deeply
terminalStorageClass: 'COLDLINE' // Don't archive vectors deeply
})
// Set lifecycle policy for metadata (less frequently accessed)
await storage.setLifecyclePolicy({
rules: [{
condition: { age: 30, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 60, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 180, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}, {
condition: { age: 730, matchesPrefix: ['_system/'] },
action: { type: 'Delete' } // Delete old system files after 2 years
}]
rules: [{
condition: { age: 30, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 60, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 180, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}, {
condition: { age: 730, matchesPrefix: ['_system/'] },
action: { type: 'Delete' } // Delete old system files after 2 years
}]
})
```
@ -158,18 +157,18 @@ await storage.setLifecyclePolicy({
**Vectors (300TB with Autoclass):**
```
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
Nearline (45TB): 45TB × $0.010/GB × 12 = $5,400/year
Coldline (225TB): 225TB × $0.004/GB × 12 = $10,800/year
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
Nearline (45TB): 45TB × $0.010/GB × 12 = $5,400/year
Coldline (225TB): 225TB × $0.004/GB × 12 = $10,800/year
Subtotal: $23,400/year
```
**Metadata (200TB with Lifecycle Policy):**
```
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
Nearline (40TB): 40TB × $0.010/GB × 12 = $4,800/year
Coldline (80TB): 80TB × $0.004/GB × 12 = $3,840/year
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
Nearline (40TB): 40TB × $0.010/GB × 12 = $4,800/year
Coldline (80TB): 80TB × $0.004/GB × 12 = $3,840/year
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
Subtotal: $16,560/year
```
@ -182,16 +181,16 @@ Subtotal: $16,560/year
```typescript
await storage.setLifecyclePolicy({
rules: [{
condition: { age: 14 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
rules: [{
condition: { age: 14 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
})
// Note: Archive class has 365-day minimum storage duration
@ -202,10 +201,10 @@ await storage.setLifecyclePolicy({
**After 1 year:**
```
Standard (25TB): 25TB × $0.020/GB × 12 = $6,000/year
Nearline (50TB): 50TB × $0.010/GB × 12 = $6,000/year
Coldline (75TB): 75TB × $0.004/GB × 12 = $3,600/year
Archive (350TB): 350TB × $0.0012/GB × 12 = $5,040/year
Standard (25TB): 25TB × $0.020/GB × 12 = $6,000/year
Nearline (50TB): 50TB × $0.010/GB × 12 = $6,000/year
Coldline (75TB): 75TB × $0.004/GB × 12 = $3,600/year
Archive (350TB): 350TB × $0.0012/GB × 12 = $5,040/year
Total Storage Cost: $20,640/year
Total with Operations: ~$25,500/year
@ -241,15 +240,15 @@ Warning: High retrieval costs if archived data is accessed frequently
### Efficient Cleanup
```typescript
// v4.0.0: Batch delete (100 objects per request for GCS)
// Batch delete (100 objects per request for GCS)
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
})
// Batch delete
@ -271,9 +270,9 @@ console.log('Terminal class:', status.terminalStorageClass)
// Example output:
// {
// enabled: true,
// terminalStorageClass: 'ARCHIVE',
// toggleTime: '2025-01-15T10:30:00Z'
// enabled: true,
// terminalStorageClass: 'ARCHIVE',
// toggleTime: '2025-01-15T10:30:00Z'
// }
```
@ -336,7 +335,7 @@ Monthly cost trend: Decreasing 8-12% per month as data ages into cheaper classes
// Check Autoclass status
const status = await storage.getAutoclassStatus()
if (!status.enabled) {
await storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' })
await storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' })
}
// Autoclass requires 24-48 hours for initial transitions
@ -365,8 +364,8 @@ if (!status.enabled) {
```typescript
// Use ADC instead of service account key file
const storage = new GcsStorage({
bucketName: 'my-brainy-data'
// No keyFilename needed - uses ADC automatically
bucketName: 'my-brainy-data'
// No keyFilename needed - uses ADC automatically
})
// ADC authentication order:
@ -417,6 +416,5 @@ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: Google Cloud Storage

246
docs/transactions.md
View file

@ -1,6 +1,6 @@
# Transaction System
**Status:** ✅ Production Ready (v5.8.0+)
**Status:** ✅ Production Ready
## Overview
@ -17,11 +17,11 @@ Brainy's transaction system provides **atomic operations** with automatic rollba
```
User Code (brain.add(), brain.update(), etc.)
↓
↓
Transaction Manager (orchestration)
↓
↓
Operations (SaveNounMetadataOperation, SaveNounOperation, etc.)
↓
↓
Storage Adapter (COW, sharding, type-aware routing)
```
@ -32,8 +32,8 @@ Every write operation in Brainy automatically uses transactions:
```typescript
// Internally, this uses a transaction
const id = await brain.add({
data: { name: 'Alice', role: 'Engineer' },
type: NounType.Person
data: { name: 'Alice', role: 'Engineer' },
type: NounType.Person
})
```
@ -51,19 +51,19 @@ Each operation implements both **execute** and **undo**:
```typescript
class SaveNounMetadataOperation {
async execute(): Promise<void> {
// Save new metadata
await this.storage.saveNounMetadata(this.id, this.metadata)
}
async execute(): Promise<void> {
// Save new metadata
await this.storage.saveNounMetadata(this.id, this.metadata)
}
async undo(): Promise<void> {
// Restore previous metadata (or delete if new entity)
if (this.previousMetadata) {
await this.storage.saveNounMetadata(this.id, this.previousMetadata)
} else {
await this.storage.deleteNounMetadata(this.id)
}
}
async undo(): Promise<void> {
// Restore previous metadata (or delete if new entity)
if (this.previousMetadata) {
await this.storage.saveNounMetadata(this.id, this.previousMetadata)
} else {
await this.storage.deleteNounMetadata(this.id)
}
}
}
```
@ -87,8 +87,8 @@ await brain.cow.checkout('feature-branch')
// Add entity (uses transaction on this branch)
const id = await brain.add({
data: { name: 'Feature Entity' },
type: NounType.Thing
data: { name: 'Feature Entity' },
type: NounType.Thing
})
// On rollback: Branch remains clean, no partial commits
@ -123,7 +123,7 @@ await brain.relate({ from: id1, to: id2, type: VerbType.RelatesTo })
- Transaction operations don't need to know about shards
- Rollback works across all shards involved
### ID-First Storage (v6.0.0+)
### ID-First Storage
✅ **Fully Compatible**
@ -132,27 +132,27 @@ Transactions work with direct ID-first paths - no type routing needed!
```typescript
// Entities stored with direct ID-first paths
const personId = await brain.add({
data: { name: 'John Doe' },
type: NounType.Person // → entities/nouns/{shard}/{id}/metadata.json
data: { name: 'John Doe' },
type: NounType.Person // → entities/nouns/{shard}/{id}/metadata.json
})
const orgId = await brain.add({
data: { name: 'Acme Corp' },
type: NounType.Organization // → entities/nouns/{shard}/{id}/metadata.json
data: { name: 'Acme Corp' },
type: NounType.Organization // → entities/nouns/{shard}/{id}/metadata.json
})
// Type changes handled atomically (type is just metadata)
await brain.update({
id: personId,
type: NounType.Organization, // Type change
data: { name: 'Doe Corp' }
id: personId,
type: NounType.Organization, // Type change
data: { name: 'Doe Corp' }
})
```
**How It Works:**
- Type information stored in metadata.noun field
- Storage layer uses O(1) ID-first path construction
- No type cache needed (removed in v6.0.0)
- No type cache needed (removed in a previous version)
- Type counters adjusted on commit/rollback
- 40x faster on cloud storage (eliminates 42-type search)
@ -165,10 +165,10 @@ Transactions work with distributed/remote storage:
```typescript
// Works with S3, Azure, GCS, etc.
const brain = new Brainy({
storage: {
type: 's3Compatible',
config: { /* S3 config */ }
}
storage: {
type: 's3Compatible',
config: { /* S3 config */ }
}
})
// Transactions ensure atomicity at write coordinator level
@ -199,8 +199,8 @@ await brain.init()
// Automatically uses transaction
const id = await brain.add({
data: { name: 'Alice', role: 'Engineer' },
type: NounType.Person
data: { name: 'Alice', role: 'Engineer' },
type: NounType.Person
})
// If add fails, all changes rolled back automatically
@ -211,15 +211,15 @@ const id = await brain.add({
```typescript
// Original entity
const id = await brain.add({
data: { name: 'John Smith', category: 'individual' },
type: NounType.Person
data: { name: 'John Smith', category: 'individual' },
type: NounType.Person
})
// Update with type change (atomic)
await brain.update({
id,
type: NounType.Organization, // Type change
data: { name: 'Smith Corp', category: 'business' }
id,
type: NounType.Organization, // Type change
data: { name: 'Smith Corp', category: 'business' }
})
// If update fails, original type and data restored
@ -229,20 +229,20 @@ await brain.update({
```typescript
const personId = await brain.add({
data: { name: 'Alice' },
type: NounType.Person
data: { name: 'Alice' },
type: NounType.Person
})
const projectId = await brain.add({
data: { name: 'Project X' },
type: NounType.Thing
data: { name: 'Project X' },
type: NounType.Thing
})
// Create relationship (atomic)
await brain.relate({
from: personId,
to: projectId,
type: VerbType.WorksOn
from: personId,
to: projectId,
type: VerbType.WorksOn
})
// If relate fails, no partial relationship created
@ -253,10 +253,10 @@ await brain.relate({
```typescript
// Multiple operations, all atomic
for (let i = 0; i < 100; i++) {
await brain.add({
data: { name: `Entity ${i}`, index: i },
type: NounType.Thing
})
await brain.add({
data: { name: `Entity ${i}`, index: i },
type: NounType.Thing
})
}
// Each add() is a separate transaction
@ -267,19 +267,19 @@ for (let i = 0; i < 100; i++) {
```typescript
const personId = await brain.add({
data: { name: 'Bob' },
type: NounType.Person
data: { name: 'Bob' },
type: NounType.Person
})
const projectId = await brain.add({
data: { name: 'Project Y' },
type: NounType.Thing
data: { name: 'Project Y' },
type: NounType.Thing
})
await brain.relate({
from: personId,
to: projectId,
type: VerbType.WorksOn
from: personId,
to: projectId,
type: VerbType.WorksOn
})
// Delete person (atomic - deletes entity + relationships)
@ -294,15 +294,15 @@ Transactions automatically handle errors and rollback:
```typescript
try {
await brain.add({
data: { name: 'Test Entity' },
type: NounType.Thing,
vector: [1, 2, 3] // Wrong dimension → error
})
await brain.add({
data: { name: 'Test Entity' },
type: NounType.Thing,
vector: [1, 2, 3] // Wrong dimension → error
})
} catch (error) {
// Transaction automatically rolled back
// No partial data in storage or indexes
console.error('Add failed:', error.message)
// Transaction automatically rolled back
// No partial data in storage or indexes
console.error('Add failed:', error.message)
}
```
@ -333,11 +333,11 @@ try {
const stats = brain.transactionManager?.getStats()
console.log(stats)
// {
// totalTransactions: 1234,
// successfulTransactions: 1200,
// failedTransactions: 34,
// rollbacks: 34,
// averageOperationsPerTransaction: 4.2
// totalTransactions: 1234,
// successfulTransactions: 1200,
// failedTransactions: 34,
// rollbacks: 34,
// averageOperationsPerTransaction: 4.2
// }
```
@ -359,7 +359,7 @@ await brain.update({ id, data })
await brain.delete(id)
// ❌ Avoid: Direct storage access bypasses transactions
await brain.storage.saveNoun(noun) // No transaction protection
await brain.storage.saveNoun(noun) // No transaction protection
```
### 2. Handle Errors Gracefully
@ -367,11 +367,11 @@ await brain.storage.saveNoun(noun) // No transaction protection
```typescript
// ✅ Recommended: Catch errors, transaction rolls back automatically
try {
const id = await brain.add({ data, type })
return id
const id = await brain.add({ data, type })
return id
} catch (error) {
console.error('Add failed, rolled back:', error)
// Decide how to handle (retry, log, alert user)
console.error('Add failed, rolled back:', error)
// Decide how to handle (retry, log, alert user)
}
```
@ -380,7 +380,7 @@ try {
```typescript
// ✅ Recommended: Validate early to avoid unnecessary rollbacks
if (!isValidVector(vector, brain.dimension)) {
throw new Error(`Vector must have ${brain.dimension} dimensions`)
throw new Error(`Vector must have ${brain.dimension} dimensions`)
}
await brain.add({ data, type, vector })
@ -391,14 +391,14 @@ await brain.add({ data, type, vector })
```typescript
// ✅ Recommended: Monitor in production
setInterval(() => {
const stats = brain.transactionManager?.getStats()
if (stats) {
const failureRate = stats.failedTransactions / stats.totalTransactions
if (failureRate > 0.05) { // > 5% failure rate
console.warn('High transaction failure rate:', failureRate)
}
}
}, 60000) // Check every minute
const stats = brain.transactionManager?.getStats()
if (stats) {
const failureRate = stats.failedTransactions / stats.totalTransactions
if (failureRate > 0.05) { // > 5% failure rate
console.warn('High transaction failure rate:', failureRate)
}
}
}, 60000) // Check every minute
```
### 5. Understand Atomicity Guarantees
@ -425,25 +425,25 @@ import { describe, it, expect } from 'vitest'
import { Brainy } from '@soulcraft/brainy'
describe('Transaction Tests', () => {
it('should rollback on failure', async () => {
const brain = new Brainy()
await brain.init()
it('should rollback on failure', async () => {
const brain = new Brainy()
await brain.init()
const id1 = await brain.add({ data: { name: 'Entity 1' }, type: NounType.Thing })
const id1 = await brain.add({ data: { name: 'Entity 1' }, type: NounType.Thing })
try {
await brain.add({
data: null as any, // Invalid - will fail
type: NounType.Thing
})
} catch (e) {
// Expected failure
}
try {
await brain.add({
data: null as any, // Invalid - will fail
type: NounType.Thing
})
} catch (e) {
// Expected failure
}
// First entity should still exist (rollback didn't affect it)
const entity1 = await brain.get(id1)
expect(entity1).toBeTruthy()
})
// First entity should still exist (rollback didn't affect it)
const entity1 = await brain.get(id1)
expect(entity1).toBeTruthy()
})
})
```
@ -510,21 +510,21 @@ const stats = brain.transactionManager?.getStats()
```
1. BEGIN
↓
↓
2. ADD OPERATIONS
- SaveNounMetadataOperation
- SaveNounOperation
- UpdateGraphIndexOperation
↓
- SaveNounMetadataOperation
- SaveNounOperation
- UpdateGraphIndexOperation
↓
3. EXECUTE (sequential)
- Execute operation 1 → Success
- Execute operation 2 → Success
- Execute operation 3 → FAILURE
↓
- Execute operation 1 → Success
- Execute operation 2 → Success
- Execute operation 3 → FAILURE
↓
4. ROLLBACK (reverse order)
- Undo operation 2
- Undo operation 1
↓
- Undo operation 2
- Undo operation 1
↓
5. THROW ERROR
```
@ -544,11 +544,11 @@ Transactions use the `StorageAdapter` interface:
```typescript
interface StorageAdapter {
saveNounMetadata(id: string, metadata: NounMetadata): Promise<void>
saveNoun(noun: Noun): Promise<void>
deleteNounMetadata(id: string): Promise<void>
deleteNoun(id: string): Promise<void>
// ... other methods
saveNounMetadata(id: string, metadata: NounMetadata): Promise<void>
saveNoun(noun: Noun): Promise<void>
deleteNounMetadata(id: string): Promise<void>
deleteNoun(id: string): Promise<void>
// ... other methods
}
```
@ -563,11 +563,11 @@ interface StorageAdapter {
## Version History
- **v5.8.0**: Initial transaction system release
- Atomic operations with rollback
- Compatible with COW, sharding, type-aware, distributed
- 36/36 unit tests passing
- 35 integration test scenarios
- Initial transaction system release
- Atomic operations with rollback
- Compatible with COW, sharding, type-aware, distributed
- 36/36 unit tests passing
- 35 integration test scenarios
## Summary

358
docs/vfs/QUICK_START.md
View file

@ -22,13 +22,13 @@ import { Brainy } from '@soulcraft/brainy'
// ✅ CORRECT: Use filesystem storage for production
const brain = new Brainy({
storage: {
type: 'filesystem',
path: './brainy-data' // Your data directory
}
storage: {
type: 'filesystem',
path: './brainy-data' // Your data directory
}
})
await brain.init() // VFS auto-initialized!
await brain.init() // VFS auto-initialized!
console.log('🎉 VFS ready!')
```
@ -47,40 +47,40 @@ const badItems = allNodes.filter(node => node.path.startsWith(dirPath))
```typescript
// ✅ Method 1: Get direct children (recommended for UI)
async function loadDirectoryContents(brain: Brainy, path: string) {
try {
const children = await brain.vfs.getDirectChildren(path)
try {
const children = await brain.vfs.getDirectChildren(path)
// Sort directories first, then files
return children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
} catch (error) {
console.error(`Failed to load ${path}:`, error.message)
return []
}
// Sort directories first, then files
return children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
} catch (error) {
console.error(`Failed to load ${path}:`, error.message)
return []
}
}
// ✅ Method 2: Get complete tree structure (for full trees)
async function loadFullTree(brain: Brainy, path: string) {
const tree = await brain.vfs.getTreeStructure(path, {
maxDepth: 3, // Prevent deep recursion
includeHidden: false, // Skip hidden files
sort: 'name'
})
return tree
const tree = await brain.vfs.getTreeStructure(path, {
maxDepth: 3, // Prevent deep recursion
includeHidden: false, // Skip hidden files
sort: 'name'
})
return tree
}
// ✅ Method 3: Get detailed path info
async function inspectPath(brain: Brainy, path: string) {
const info = await brain.vfs.inspect(path)
return {
isDirectory: info.node.metadata.vfsType === 'directory',
children: info.children,
parent: info.parent,
stats: info.stats
}
const info = await brain.vfs.inspect(path)
return {
isDirectory: info.node.metadata.vfsType === 'directory',
children: info.children,
parent: info.parent,
stats: info.stats
}
}
```
@ -89,19 +89,19 @@ async function inspectPath(brain: Brainy, path: string) {
```typescript
// ✅ Find files by content, not just filename
async function searchFiles(brain: Brainy, query: string, basePath: string = '/') {
const results = await brain.vfs.search(query, {
path: basePath, // Limit search to specific directory
limit: 50, // Reasonable limit
type: 'file' // Only search files, not directories
})
const results = await brain.vfs.search(query, {
path: basePath, // Limit search to specific directory
limit: 50, // Reasonable limit
type: 'file' // Only search files, not directories
})
return results.map(result => ({
path: result.path,
score: result.score,
type: result.type,
size: result.size,
modified: result.modified
}))
return results.map(result => ({
path: result.path,
score: result.score,
type: result.type,
size: result.size,
modified: result.modified
}))
}
// Example usage
@ -118,149 +118,149 @@ import React, { useState, useEffect } from 'react'
import { Brainy } from '@soulcraft/brainy'
export function FileExplorer() {
const [brain, setBrain] = useState(null)
const [currentPath, setCurrentPath] = useState('/')
const [items, setItems] = useState([])
const [loading, setLoading] = useState(true)
const [searchQuery, setSearchQuery] = useState('')
const [brain, setBrain] = useState(null)
const [currentPath, setCurrentPath] = useState('/')
const [items, setItems] = useState([])
const [loading, setLoading] = useState(true)
const [searchQuery, setSearchQuery] = useState('')
// Initialize Brainy (VFS auto-initialized!)
useEffect(() => {
async function initBrainy() {
const brainInstance = new Brainy({
storage: { type: 'filesystem', path: './brainy-data' }
})
await brainInstance.init() // VFS ready after this!
// Initialize Brainy (VFS auto-initialized!)
useEffect(() => {
async function initBrainy() {
const brainInstance = new Brainy({
storage: { type: 'filesystem', path: './brainy-data' }
})
await brainInstance.init() // VFS ready after this!
setBrain(brainInstance)
setLoading(false)
}
initBrainy()
}, [])
setBrain(brainInstance)
setLoading(false)
}
initBrainy()
}, [])
// Load directory contents
const loadDirectory = async (path: string) => {
if (!brain) return
// Load directory contents
const loadDirectory = async (path: string) => {
if (!brain) return
setLoading(true)
try {
// ✅ CORRECT: Use getDirectChildren to prevent recursion
const children = await brain.vfs.getDirectChildren(path)
setLoading(true)
try {
// ✅ CORRECT: Use getDirectChildren to prevent recursion
const children = await brain.vfs.getDirectChildren(path)
// Sort directories first
const sorted = children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
// Sort directories first
const sorted = children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
setItems(sorted)
setCurrentPath(path)
} catch (error) {
console.error('Failed to load directory:', error)
setItems([])
} finally {
setLoading(false)
}
}
setItems(sorted)
setCurrentPath(path)
} catch (error) {
console.error('Failed to load directory:', error)
setItems([])
} finally {
setLoading(false)
}
}
// Search files
const handleSearch = async () => {
if (!brain || !searchQuery.trim()) {
loadDirectory(currentPath)
return
}
// Search files
const handleSearch = async () => {
if (!brain || !searchQuery.trim()) {
loadDirectory(currentPath)
return
}
setLoading(true)
try {
const results = await brain.vfs.search(searchQuery, {
path: currentPath,
limit: 100
})
setItems(results)
} catch (error) {
console.error('Search failed:', error)
} finally {
setLoading(false)
}
}
setLoading(true)
try {
const results = await brain.vfs.search(searchQuery, {
path: currentPath,
limit: 100
})
setItems(results)
} catch (error) {
console.error('Search failed:', error)
} finally {
setLoading(false)
}
}
// Initial load
useEffect(() => {
if (brain) {
loadDirectory('/')
}
}, [brain])
// Initial load
useEffect(() => {
if (brain) {
loadDirectory('/')
}
}, [brain])
if (loading && !brain) {
return <div>Initializing Brainy...</div>
}
if (loading && !brain) {
return <div>Initializing Brainy...</div>
}
return (
<div className="file-explorer">
{/* Search bar */}
<div className="search-bar">
<input
type="text"
value={searchQuery}
onChange={(e) => setSearchQuery(e.target.value)}
placeholder="Search files by content..."
onKeyPress={(e) => e.key === 'Enter' && handleSearch()}
/>
<button onClick={handleSearch}>Search</button>
{searchQuery && (
<button onClick={() => { setSearchQuery(''); loadDirectory(currentPath) }}>
Clear
</button>
)}
</div>
return (
<div className="file-explorer">
{/* Search bar */}
<div className="search-bar">
<input
type="text"
value={searchQuery}
onChange={(e) => setSearchQuery(e.target.value)}
placeholder="Search files by content..."
onKeyPress={(e) => e.key === 'Enter' && handleSearch()}
/>
<button onClick={handleSearch}>Search</button>
{searchQuery && (
<button onClick={() => { setSearchQuery(''); loadDirectory(currentPath) }}>
Clear
</button>
)}
</div>
{/* Current path */}
<div className="current-path">
📁 {currentPath}
{currentPath !== '/' && (
<button onClick={() => loadDirectory(currentPath.split('/').slice(0, -1).join('/') || '/')}>
⬆️ Up
</button>
)}
</div>
{/* Current path */}
<div className="current-path">
📁 {currentPath}
{currentPath !== '/' && (
<button onClick={() => loadDirectory(currentPath.split('/').slice(0, -1).join('/') || '/')}>
⬆️ Up
</button>
)}
</div>
{/* File list */}
{loading ? (
<div>Loading...</div>
) : (
<div className="file-list">
{items.map((item) => (
<div
key={item.id}
className={`file-item ${item.metadata.vfsType}`}
onClick={() => {
if (item.metadata.vfsType === 'directory') {
loadDirectory(item.metadata.path)
} else {
console.log('Open file:', item.metadata.path)
// Add your file opening logic here
}
}}
>
<span className="icon">
{item.metadata.vfsType === 'directory' ? '📁' : '📄'}
</span>
<span className="name">{item.metadata.name}</span>
<span className="size">
{item.metadata.size ? `${Math.round(item.metadata.size / 1024)}KB` : ''}
</span>
</div>
))}
{items.length === 0 && (
<div className="empty">
{searchQuery ? 'No results found' : 'Empty directory'}
</div>
)}
</div>
)}
</div>
)
{/* File list */}
{loading ? (
<div>Loading...</div>
) : (
<div className="file-list">
{items.map((item) => (
<div
key={item.id}
className={`file-item ${item.metadata.vfsType}`}
onClick={() => {
if (item.metadata.vfsType === 'directory') {
loadDirectory(item.metadata.path)
} else {
console.log('Open file:', item.metadata.path)
// Add your file opening logic here
}
}}
>
<span className="icon">
{item.metadata.vfsType === 'directory' ? '📁' : '📄'}
</span>
<span className="name">{item.metadata.name}</span>
<span className="size">
{item.metadata.size ? `${Math.round(item.metadata.size / 1024)}KB` : ''}
</span>
</div>
))}
{items.length === 0 && (
<div className="empty">
{searchQuery ? 'No results found' : 'Empty directory'}
</div>
)}
</div>
)}
</div>
)
}
```
@ -288,13 +288,13 @@ Your file explorer is now working! Here's what to explore next:
### "Module not found" errors
```bash
# Make sure you're using the right import
npm ls @soulcraft/brainy # Check version
npm install @soulcraft/brainy@latest # Update if needed
npm ls @soulcraft/brainy # Check version
npm install @soulcraft/brainy@latest # Update if needed
```
### "VFS not initialized" errors
```typescript
// v5.1.0+: Just await brain.init() - VFS is auto-initialized!
// Just await brain.init() - VFS is auto-initialized!
await brain.init()
// VFS ready - use brain.vfs directly
await brain.vfs.writeFile('/test.txt', 'data')
@ -304,8 +304,8 @@ await brain.vfs.writeFile('/test.txt', 'data')
```typescript
// Add pagination for large directories
const children = await brain.vfs.getDirectChildren(path, {
limit: 100, // Load only first 100 items
offset: 0 // Start from beginning
limit: 100, // Load only first 100 items
offset: 0 // Start from beginning
})
```
@ -313,8 +313,8 @@ const children = await brain.vfs.getDirectChildren(path, {
```typescript
// Make sure files are imported into VFS first
await brain.vfs.importDirectory('./my-files', {
recursive: true,
extractMetadata: true // Enable content understanding
recursive: true,
extractMetadata: true // Enable content understanding
})
```

348
docs/vfs/README.md
View file

@ -28,15 +28,15 @@ import { VirtualFileSystem } from '@soulcraft/brainy/vfs'
// Initialize the VFS
const vfs = new VirtualFileSystem({
root: '/my-brain',
intelligent: true // Enable AI features
root: '/my-brain',
intelligent: true // Enable AI features
})
await vfs.init()
// Write a file - it automatically becomes intelligent
await vfs.writeFile('/projects/my-app/index.js',
'console.log("Hello, World!")')
'console.log("Hello, World!")')
// Find similar files using semantic search
const similar = await vfs.findSimilar('/projects/my-app/index.js')
@ -48,11 +48,11 @@ const results = await vfs.search('files about authentication')
await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements')
```
## ⚡ Performance (v5.11.1)
## ⚡ Performance
**75% Faster File Operations!** VFS now automatically benefits from brain.get() metadata-only optimization:
| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup |
| Operation | Before | After | Speedup |
|-----------|------------------|-----------------|---------|
| `readFile()` | 53ms | **~13ms** | **75%** |
| `stat()` | 53ms | **~13ms** | **75%** |
@ -60,7 +60,7 @@ await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements')
**Zero configuration** - automatic optimization for all VFS operations!
VFS operations only need metadata (path, size, timestamps), not 384-dimensional vector embeddings. The v5.11.1 optimization automatically uses metadata-only reads, saving 95% bandwidth and 76-81% time.
VFS operations only need metadata (path, size, timestamps), not 384-dimensional vector embeddings. The optimization automatically uses metadata-only reads, saving 95% bandwidth and 76-81% time.
## Core Features
@ -110,20 +110,20 @@ const docs = await vfs.search('technical documentation for API endpoints')
// Find similar files
const similar = await vfs.findSimilar('/code/auth.js', {
limit: 5,
threshold: 0.8 // 80% similarity
limit: 5,
threshold: 0.8 // 80% similarity
})
// Get related files through the knowledge graph
const related = await vfs.getRelated('/proposal.pdf', {
depth: 2 // Include relationships of relationships
depth: 2 // Include relationships of relationships
})
// Auto-organization suggestions
const suggestions = await vfs.suggestOrganization([
'/downloads/doc1.pdf',
'/downloads/image.jpg',
'/downloads/code.py'
'/downloads/doc1.pdf',
'/downloads/image.jpg',
'/downloads/code.py'
])
// Returns: suggested folders and categorization
```
@ -144,10 +144,10 @@ const connections = await vfs.getConnections('/code/impl.js')
// Traverse the graph
const implementations = await vfs.search('', {
connected: {
to: '/spec.md',
via: 'implements'
}
connected: {
to: '/spec.md',
via: 'implements'
}
})
```
@ -158,8 +158,8 @@ Store anything alongside your files:
```javascript
// Add todos to files
await vfs.setTodos('/projects/app/index.js', [
{ task: 'Add error handling', priority: 'high', due: '2024-01-20' },
{ task: 'Optimize performance', priority: 'medium' }
{ task: 'Add error handling', priority: 'high', due: '2024-01-20' },
{ task: 'Optimize performance', priority: 'medium' }
])
// Set custom attributes
@ -169,11 +169,11 @@ await vfs.setxattr('/video.mp4', 'tags', ['tutorial', 'react', 'hooks'])
// Query by metadata
const urgent = await vfs.search('', {
where: { 'todos.priority': 'high' }
where: { 'todos.priority': 'high' }
})
const parisPhotos = await vfs.search('', {
where: { location: 'Paris, France' }
where: { location: 'Paris, France' }
})
```
@ -184,20 +184,20 @@ Access files through semantic dimensions (see [Semantic VFS](./SEMANTIC_VFS.md))
```javascript
// Query-based path access (current functionality)
const authFiles = await vfs.search('', {
where: { concepts: { contains: 'authentication' }}
where: { concepts: { contains: 'authentication' }}
})
// Find files by custom metadata
const recent = await vfs.search('', {
where: {
type: 'document',
modified: { greaterThan: Date.now() - 7*24*60*60*1000 }
}
where: {
type: 'document',
modified: { greaterThan: Date.now() - 7*24*60*60*1000 }
}
})
// Find similar files
const similar = await vfs.findSimilar('/examples/good-code.js', {
threshold: 0.7
threshold: 0.7
})
```
@ -210,24 +210,24 @@ const similar = await vfs.findSimilar('/examples/good-code.js', {
```javascript
// Store a research paper with automatic analysis
await vfs.writeFile('/research/quantum-computing.pdf', pdfBuffer, {
metadata: {
authors: ['Dr. Alice Smith', 'Dr. Bob Jones'],
year: 2024,
topics: ['quantum', 'computing', 'algorithms'],
citations: 42
}
metadata: {
authors: ['Dr. Alice Smith', 'Dr. Bob Jones'],
year: 2024,
topics: ['quantum', 'computing', 'algorithms'],
citations: 42
}
})
// Find all papers on similar topics
const related = await vfs.search('quantum algorithms', {
type: 'document',
where: { year: { $gte: 2020 } }
type: 'document',
where: { year: { $gte: 2020 } }
})
// Find papers that cite this one
const citations = await vfs.getConnections('/research/quantum-computing.pdf', {
type: 'cites',
direction: 'incoming'
type: 'cites',
direction: 'incoming'
})
```
@ -245,12 +245,12 @@ await vfs.writeFile('/src/utils/auth.js', authCode)
// Find all files that import this module
const importers = await vfs.search('', {
where: { dependencies: 'utils/auth.js' }
where: { dependencies: 'utils/auth.js' }
})
// Find test files for this code
const tests = await vfs.getRelated('/src/utils/auth.js', {
type: 'tests'
type: 'tests'
})
// Find similar implementations
@ -262,12 +262,12 @@ const similar = await vfs.findSimilar('/src/utils/auth.js')
```javascript
// Store media with rich metadata
await vfs.writeFile('/photos/sunset.jpg', imageBuffer, {
metadata: {
camera: 'Canon R5',
location: { lat: 37.7749, lng: -122.4194 },
tags: ['sunset', 'golden-gate', 'landscape'],
album: 'San Francisco 2024'
}
metadata: {
camera: 'Canon R5',
location: { lat: 37.7749, lng: -122.4194 },
tags: ['sunset', 'golden-gate', 'landscape'],
album: 'San Francisco 2024'
}
})
// Find similar images
@ -275,18 +275,18 @@ const similar = await vfs.findSimilar('/photos/sunset.jpg')
// Find photos by location
const nearby = await vfs.search('', {
type: 'image',
where: {
'location.lat': { $between: [37.7, 37.8] },
'location.lng': { $between: [-122.5, -122.3] }
}
type: 'image',
where: {
'location.lat': { $between: [37.7, 37.8] },
'location.lng': { $between: [-122.5, -122.3] }
}
})
// Smart albums
await vfs.createVirtualDirectory('/albums/best-sunsets', {
query: 'sunset',
type: 'image',
where: { rating: { $gte: 4 } }
query: 'sunset',
type: 'image',
where: { rating: { $gte: 4 } }
})
```
@ -308,25 +308,25 @@ await vfs.setxattr(projectPath, 'status', 'in-progress')
// Add todos to specific files
await vfs.setTodos(`${projectPath}/src/index.js`, [
{ task: 'Implement user authentication', assignee: 'Alice' },
{ task: 'Add error handling', assignee: 'Bob' }
{ task: 'Implement user authentication', assignee: 'Alice' },
{ task: 'Add error handling', assignee: 'Bob' }
])
// Find all files with pending todos
const pending = await vfs.search('', {
where: {
path: { $startsWith: projectPath },
'todos.status': 'pending'
}
where: {
path: { $startsWith: projectPath },
'todos.status': 'pending'
}
})
// Find projects nearing deadline
const urgent = await vfs.search('', {
where: {
type: 'directory',
'deadline': { $lte: '2024-02-01' },
'status': 'in-progress'
}
where: {
type: 'directory',
'deadline': { $lte: '2024-02-01' },
'status': 'in-progress'
}
})
```
@ -417,9 +417,9 @@ The VFS is built with production scalability in mind:
#### **Storage Strategy**
```typescript
// Adaptive storage based on file size
< 100KB: Inline storage (entity.data)
< 10MB: External reference (S3/R2 key)
> 10MB: Chunked storage (parallel chunks)
< 100KB: Inline storage (entity.data)
< 10MB: External reference (S3/R2 key)
> 10MB: Chunked storage (parallel chunks)
```
#### **Performance Metrics**
@ -434,24 +434,24 @@ The VFS is built with production scalability in mind:
#### **How It Handles Scale**
1. **Hierarchical Caching**
- 100K+ path cache entries
- Parent-child relationship caching
- Hot path detection and optimization
- 100K+ path cache entries
- Parent-child relationship caching
- Hot path detection and optimization
2. **Distributed Architecture**
- Sharding by path prefix
- Read replicas for hot directories
- CDN integration for static files
- Sharding by path prefix
- Read replicas for hot directories
- CDN integration for static files
3. **Intelligent Indexing**
- Compound indexes on (parent, name)
- Vector indexes for semantic search
- Graph indexes for relationships
- Compound indexes on (parent, name)
- Vector indexes for semantic search
- Graph indexes for relationships
4. **Streaming Everything**
- Large files never fully in memory
- Progressive loading
- Chunked transfers
- Large files never fully in memory
- Progressive loading
- Chunked transfers
### Real Production Scenarios
@ -462,29 +462,29 @@ Store build artifacts with automatic relationships:
```javascript
// Store build output with metadata
await vfs.writeFile('/builds/v1.2.3/app.js', buildOutput, {
metadata: {
commit: 'abc123',
branch: 'main',
timestamp: Date.now(),
tests: 'passing',
coverage: 0.92
}
metadata: {
commit: 'abc123',
branch: 'main',
timestamp: Date.now(),
tests: 'passing',
coverage: 0.92
}
})
// Find all builds for a commit
const builds = await vfs.search('', {
where: { commit: 'abc123' }
where: { commit: 'abc123' }
})
// Get latest passing build
const latest = await vfs.search('', {
where: {
branch: 'main',
tests: 'passing'
},
sort: 'modified',
order: 'desc',
limit: 1
where: {
branch: 'main',
tests: 'passing'
},
sort: 'modified',
order: 'desc',
limit: 1
})
```
@ -495,8 +495,8 @@ Isolate customer data with semantic understanding:
```javascript
// Each tenant gets their own root
const tenantVfs = new VirtualFileSystem({
root: `/tenants/${tenantId}`,
service: tenantId // Isolate at Brainy level too
root: `/tenants/${tenantId}`,
service: tenantId // Isolate at Brainy level too
})
// Tenant uploads document
@ -505,11 +505,11 @@ await tenantVfs.writeFile('/documents/contract.pdf', pdfBuffer)
// Cross-tenant analytics (admin only)
const adminVfs = new VirtualFileSystem({ root: '/tenants' })
const stats = await adminVfs.search('contract', {
recursive: true,
aggregations: {
byTenant: { field: 'service' },
byType: { field: 'mimeType' }
}
recursive: true,
aggregations: {
byTenant: { field: 'service' },
byType: { field: 'mimeType' }
}
})
```
@ -520,38 +520,38 @@ Connect datasets, models, and results:
```javascript
// Store training data
await vfs.writeFile('/datasets/train.csv', csvData, {
metadata: {
samples: 100000,
features: 50,
labels: 10
}
metadata: {
samples: 100000,
features: 50,
labels: 10
}
})
// Store trained model
await vfs.writeFile('/models/v1/model.pkl', modelBuffer, {
metadata: {
algorithm: 'random-forest',
accuracy: 0.95,
trainedOn: '/datasets/train.csv',
hyperparameters: { trees: 100, depth: 10 }
}
metadata: {
algorithm: 'random-forest',
accuracy: 0.95,
trainedOn: '/datasets/train.csv',
hyperparameters: { trees: 100, depth: 10 }
}
})
// Connect model to its training data
await vfs.addRelationship(
'/models/v1/model.pkl',
'/datasets/train.csv',
'trained-on'
'/models/v1/model.pkl',
'/datasets/train.csv',
'trained-on'
)
// Find best model for a dataset
const models = await vfs.search('', {
connected: {
to: '/datasets/train.csv',
via: 'trained-on'
},
sort: 'accuracy',
order: 'desc'
connected: {
to: '/datasets/train.csv',
via: 'trained-on'
},
sort: 'accuracy',
order: 'desc'
})
```
@ -562,30 +562,30 @@ Intelligent content organization:
```javascript
// Auto-organize uploads
vfs.on('file:added', async (path) => {
if (path.startsWith('/uploads/')) {
const file = await vfs.getEntity(path)
if (path.startsWith('/uploads/')) {
const file = await vfs.getEntity(path)
// Auto-categorize by AI
const category = await detectCategory(file)
const newPath = `/content/${category}/${file.metadata.name}`
// Auto-categorize by AI
const category = await detectCategory(file)
const newPath = `/content/${category}/${file.metadata.name}`
await vfs.move(path, newPath)
await vfs.move(path, newPath)
// Auto-tag
const tags = await extractTags(file)
await vfs.setxattr(newPath, 'tags', tags)
// Auto-tag
const tags = await extractTags(file)
await vfs.setxattr(newPath, 'tags', tags)
// Find related content
const related = await vfs.findSimilar(newPath, {
limit: 5,
threshold: 0.8
})
// Find related content
const related = await vfs.findSimilar(newPath, {
limit: 5,
threshold: 0.8
})
// Create relationships
for (const rel of related) {
await vfs.addRelationship(newPath, rel.path, 'related-to')
}
}
// Create relationships
for (const rel of related) {
await vfs.addRelationship(newPath, rel.path, 'related-to')
}
}
})
```
@ -596,39 +596,39 @@ Collaborative file management:
```javascript
// Track file ownership and access
await vfs.writeFile('/projects/alpha/spec.md', content, {
metadata: {
owner: userId,
team: 'engineering',
permissions: {
[userId]: 'rw',
'team:engineering': 'r',
'others': '-'
}
}
metadata: {
owner: userId,
team: 'engineering',
permissions: {
[userId]: 'rw',
'team:engineering': 'r',
'others': '-'
}
}
})
// Add collaborative features
await vfs.addTodo('/projects/alpha/spec.md', {
task: 'Review security section',
assignee: 'alice@company.com',
due: '2024-02-01',
priority: 'high'
task: 'Review security section',
assignee: 'alice@company.com',
due: '2024-02-01',
priority: 'high'
})
// Track who's working on what
await vfs.setxattr('/projects/alpha/spec.md', 'locks', {
section3: {
user: 'bob@company.com',
since: Date.now()
}
section3: {
user: 'bob@company.com',
since: Date.now()
}
})
// Find all files assigned to a user
const assigned = await vfs.search('', {
where: {
'todos.assignee': 'alice@company.com',
'todos.status': 'pending'
}
where: {
'todos.assignee': 'alice@company.com',
'todos.status': 'pending'
}
})
```
@ -643,15 +643,15 @@ const assigned = await vfs.search('', {
#### **Standalone Mode**
```javascript
const vfs = new VirtualFileSystem()
await vfs.init() // Uses in-memory storage
await vfs.init() // Uses in-memory storage
```
#### **Local Persistence**
```javascript
const vfs = new VirtualFileSystem()
await vfs.init({
storage: 'filesystem',
dataDir: '/var/lib/brainy-vfs'
storage: 'filesystem',
dataDir: '/var/lib/brainy-vfs'
})
```
@ -659,9 +659,9 @@ await vfs.init({
```javascript
const vfs = new VirtualFileSystem()
await vfs.init({
storage: 's3',
bucket: 'my-vfs-data',
region: 'us-west-2'
storage: 's3',
bucket: 'my-vfs-data',
region: 'us-west-2'
})
```
@ -669,14 +669,14 @@ await vfs.init({
```javascript
const vfs = new VirtualFileSystem()
await vfs.init({
distributed: true,
nodes: [
'vfs1.internal:8080',
'vfs2.internal:8080',
'vfs3.internal:8080'
],
replication: 3,
consistency: 'eventual'
distributed: true,
nodes: [
'vfs1.internal:8080',
'vfs2.internal:8080',
'vfs3.internal:8080'
],
replication: 3,
consistency: 'eventual'
})
```

84
docs/vfs/TROUBLESHOOTING.md
View file

@ -12,8 +12,8 @@
**Root Cause:**
The root directory entity exists but doesn't have the proper metadata structure.
**Solution (v3.15.0+):**
This issue has been fixed in v3.15.0. The VFS now:
**Solution:**
This issue has been fixed. The VFS now:
1. Ensures root directory has `vfsType: 'directory'` metadata
2. Adds compatibility layer for entities with malformed metadata
3. Automatically repairs metadata on entity retrieval
@ -21,17 +21,17 @@ This issue has been fixed in v3.15.0. The VFS now:
**Manual Fix (if needed):**
```javascript
// Force re-initialization of root directory
await vfs.init() // Will repair root if needed
await vfs.init() // Will repair root if needed
// Or manually update root entity
const rootId = vfs.rootEntityId
await brain.update({
id: rootId,
metadata: {
path: '/',
vfsType: 'directory',
// ... other metadata
}
id: rootId,
metadata: {
path: '/',
vfsType: 'directory',
// ... other metadata
}
})
```
@ -47,7 +47,7 @@ await brain.update({
**Root Cause:**
Contains relationships are missing between parent directories and files.
**Solution (v3.15.0+):**
**Solution:**
This issue has been fixed. The VFS now:
1. Creates Contains relationships when writing new files
2. Ensures Contains relationships exist when updating files
@ -60,9 +60,9 @@ const parentId = await vfs.resolvePath('/directory')
const fileId = await vfs.resolvePath('/directory/file.txt')
await brain.relate({
from: parentId,
to: fileId,
type: VerbType.Contains
from: parentId,
to: fileId,
type: VerbType.Contains
})
```
@ -80,8 +80,8 @@ The VFS `init()` method wasn't called after getting the VFS instance.
**Solution:**
```javascript
// ✅ CORRECT
const vfs = brain.vfs() // Get instance
await vfs.init() // Initialize (REQUIRED!)
const vfs = brain.vfs() // Get instance
await vfs.init() // Initialize (REQUIRED!)
// ❌ WRONG
const vfs = brain.vfs()
@ -104,15 +104,15 @@ Using in-memory storage instead of persistent storage.
```javascript
// ✅ Use persistent storage
const brain = new Brainy({
storage: {
type: 'filesystem', // Persistent
path: './brainy-data'
}
storage: {
type: 'filesystem', // Persistent
path: './brainy-data'
}
})
// ❌ Don't use memory for production
const brain = new Brainy({
storage: { type: 'memory' } // Data lost on restart!
storage: { type: 'memory' } // Data lost on restart!
})
```
@ -136,7 +136,7 @@ const children = await vfs.getDirectChildren('/directory')
// ❌ WRONG - Path prefix matching causes recursion
const allNodes = await brain.find({})
const children = allNodes.filter(n =>
n.metadata.path.startsWith('/directory/')) // Directory matches itself!
n.metadata.path.startsWith('/directory/')) // Directory matches itself!
```
---
@ -153,13 +153,13 @@ Files aren't being properly embedded or indexed.
**Solution:**
```javascript
// Ensure files have content for embedding
await vfs.writeFile('/doc.txt', 'Actual content here') // Not empty!
await vfs.writeFile('/doc.txt', 'Actual content here') // Not empty!
// Use semantic search correctly
const results = await vfs.search('machine learning', {
path: '/documents', // Search within path
limit: 10,
type: 'file'
path: '/documents', // Search within path
limit: 10,
type: 'file'
})
```
@ -184,8 +184,8 @@ console.log('VFS type:', entity.metadata.vfsType)
```javascript
const parentId = await vfs.resolvePath('/directory')
const relations = await brain.getRelations({
from: parentId,
type: VerbType.Contains
from: parentId,
type: VerbType.Contains
})
console.log('Child count:', relations.length)
```
@ -193,11 +193,11 @@ console.log('Child count:', relations.length)
### 4. Enable Debug Logging
```javascript
const brain = new Brainy({
storage: { type: 'filesystem' },
logger: {
level: 'debug',
enabled: true
}
storage: { type: 'filesystem' },
logger: {
level: 'debug',
enabled: true
}
})
```
@ -209,11 +209,11 @@ const brain = new Brainy({
```javascript
// Enable caching in VFS config
const vfs = brain.vfs({
cache: {
enabled: true,
ttl: 300000, // 5 minutes
maxSize: 1000
}
cache: {
enabled: true,
ttl: 300000, // 5 minutes
maxSize: 1000
}
})
```
@ -221,12 +221,12 @@ const vfs = brain.vfs({
```javascript
// Write multiple files efficiently
const files = [
{ path: '/file1.txt', content: 'content1' },
{ path: '/file2.txt', content: 'content2' }
{ path: '/file1.txt', content: 'content1' },
{ path: '/file2.txt', content: 'content2' }
]
await Promise.all(
files.map(f => vfs.writeFile(f.path, f.content))
files.map(f => vfs.writeFile(f.path, f.content))
)
```
@ -234,8 +234,8 @@ await Promise.all(
```javascript
// Don't traverse too deep
const tree = await vfs.getTreeStructure('/', {
maxDepth: 3, // Limit recursion
includeHidden: false
maxDepth: 3, // Limit recursion
includeHidden: false
})
```

4
docs/vfs/VFS_CORE.md
View file

@ -418,7 +418,7 @@ await vfs.importDirectory()// Import directory from filesystem
## Bulk Write Operations
**v6.5.0**: `bulkWrite` efficiently processes multiple VFS operations with automatic ordering to prevent race conditions.
`bulkWrite` efficiently processes multiple VFS operations with automatic ordering to prevent race conditions.
```javascript
const result = await vfs.bulkWrite([
@ -439,7 +439,7 @@ console.log(`Successful: ${result.successful}, Failed: ${result.failed.length}`)
- `delete` - Delete file
- `update` - Update file metadata only
**Operation ordering (v6.5.0):**
**Operation ordering:**
1. `mkdir` operations run **first**, sequentially, sorted by path depth (shallowest first)
2. Other operations (`write`, `delete`, `update`) run **after** in parallel batches of 10

8
docs/vfs/VFS_INITIALIZATION.md
View file

@ -1,4 +1,4 @@
# VFS Initialization Guide (v5.1.0+)
# VFS Initialization Guide
## Quick Start
@ -30,7 +30,7 @@ await vfs.init() // ❌ Separate initialization
await vfs.writeFile(...)
```
### After (v5.1.0+):
### After:
```javascript
const brain = new Brainy(...)
await brain.init() // VFS auto-initialized!
@ -53,7 +53,7 @@ await vfs.init()
await vfs.writeFile('/test.txt', 'data')
```
### New Pattern (v5.1.0+):
### New Pattern:
```javascript
// Just remove the () and init() call
await brain.vfs.writeFile('/test.txt', 'data')
@ -142,7 +142,7 @@ await brain.init() // Required!
await brain.vfs.writeFile(...) // Now this works
```
## Fork Support (v5.0.0+)
## Fork Support
VFS works seamlessly with Brainy's Copy-on-Write (COW) fork feature: