soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: exclude __words__ keyword index from corruption detection and getStats()

Browse source 
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:

1. validateConsistency() to falsely detect corruption on every startup,
   triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
   and report inflated totalEntries/totalIds stats

Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
This commit is contained in:
David Snelling 2026-01-27 15:38:21 -08:00
parent 32dbdcec61
commit 364360d447
128 changed files with 5637 additions and 5682 deletions
Download patch file Download diff file Expand all files Collapse all files

756
docs/API_REFERENCE.md
View file

File diff suppressed because it is too large Load diff

143
docs/BATCHING.md
View file

@ -1,23 +1,22 @@
# Batch Operations API v5.12.0
# Batch Operations API
> **Enterprise Production-Ready** | Zero N+1 Query Patterns | 90%+ Performance Improvement
## Overview
Brainy v5.12.0 introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage.
Brainy introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage.
### Problem Solved
**Before v5.12.0:**
**Before optimization:**
- VFS `getTreeStructure()` on cloud storage: **12.7 seconds** for directory with 12 files
- N+1 query pattern: 1 directory query + N individual file queries (22 sequential calls × 580ms latency)
**After v5.12.0:**
**After optimization:**
- VFS `getTreeStructure()`: **<1 second** for 12 files (90%+ improvement)
- 2-3 batched calls instead of 22 sequential calls
- Native cloud storage batch APIs for maximum throughput
**IMPORTANT:** The v5.12.0 batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See v6.0.0 changes for comprehensive storage path optimizations.
**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See changes for comprehensive storage path optimizations.
---
@ -56,7 +55,7 @@ results.size // → 3 (number of found entities)
### 2. `storage.getNounMetadataBatch(ids)`
Batch metadata retrieval with direct O(1) path construction (v6.0.0+).
Batch metadata retrieval with direct O(1) path construction.
```typescript
const storage = brain.storage as BaseStorage
@ -65,8 +64,8 @@ const ids = ['id1', 'id2', 'id3']
const metadataMap: Map<string, NounMetadata> = await storage.getNounMetadataBatch(ids)
for (const [id, metadata] of metadataMap) {
console.log(metadata.noun) // Type: 'document', 'person', etc.
console.log(metadata.data) // Entity data
console.log(metadata.noun) // Type: 'document', 'person', etc.
console.log(metadata.data) // Entity data
}
```
@ -92,22 +91,22 @@ const storage = brain.storage as BaseStorage
// Get all relationships from multiple sources
const results: Map<string, GraphVerb[]> = await storage.getVerbsBySourceBatch([
'person1',
'person2'
'person1',
'person2'
])
// Filter by verb type
const createsResults = await storage.getVerbsBySourceBatch(
['person1', 'person2'],
'creates'
['person1', 'person2'],
'creates'
)
// Process results
for (const [sourceId, verbs] of results) {
console.log(`${sourceId} has ${verbs.length} relationships`)
verbs.forEach(verb => {
console.log(` → ${verb.verb} → ${verb.targetId}`)
})
console.log(`${sourceId} has ${verbs.length} relationships`)
verbs.forEach(verb => {
console.log(` → ${verb.verb} → ${verb.targetId}`)
})
}
```
@ -130,8 +129,8 @@ COW-aware batch path resolution with branch inheritance.
const storage = brain.storage as BaseStorage
const paths = [
'entities/nouns/{shard}/id1/metadata.json',
'entities/nouns/{shard}/id2/metadata.json'
'entities/nouns/{shard}/id1/metadata.json',
'entities/nouns/{shard}/id2/metadata.json'
]
// Resolves to: branches/{branch}/entities/nouns/{shard}/{id}/metadata.json
@ -160,9 +159,9 @@ const results = await gcsStorage.readBatch(paths)
// Configuration
gcsStorage.getBatchConfig() // → {
// maxBatchSize: 1000,
// maxConcurrent: 100,
// operationsPerSecond: 1000
// maxBatchSize: 1000,
// maxConcurrent: 100,
// operationsPerSecond: 1000
// }
```
@ -185,9 +184,9 @@ const results = await s3Storage.readBatch(paths)
// Configuration
s3Storage.getBatchConfig() // → {
// maxBatchSize: 1000,
// maxConcurrent: 150,
// operationsPerSecond: 5000
// maxBatchSize: 1000,
// maxConcurrent: 150,
// operationsPerSecond: 5000
// }
```
@ -208,9 +207,9 @@ const results = await r2Storage.readBatch(paths)
// Configuration
r2Storage.getBatchConfig() // → {
// maxBatchSize: 1000,
// maxConcurrent: 150,
// operationsPerSecond: 6000
// maxBatchSize: 1000,
// maxConcurrent: 150,
// operationsPerSecond: 6000
// }
```
@ -231,9 +230,9 @@ const results = await azureStorage.readBatch(paths)
// Configuration
azureStorage.getBatchConfig() // → {
// maxBatchSize: 1000,
// maxConcurrent: 100,
// operationsPerSecond: 3000
// maxBatchSize: 1000,
// maxConcurrent: 100,
// operationsPerSecond: 3000
// }
```
@ -254,7 +253,7 @@ VFS operations automatically use batch APIs for maximum performance.
// OLD: Sequential N+1 pattern (12.7 seconds for 12 files)
const tree = await brain.vfs.getTreeStructure('/my-dir')
// NEW v5.12.0: Parallel breadth-first with batching (<1 second)
// NEW Parallel breadth-first with batching (<1 second)
// ✅ PathResolver.getChildren() uses brain.batchGet() internally
// ✅ Parallel traversal of directories at same tree level
// ✅ 2-3 batched calls instead of 22 sequential calls
@ -264,16 +263,16 @@ const tree = await brain.vfs.getTreeStructure('/my-dir')
```
VFS.getTreeStructure()
↓ PARALLEL (breadth-first traversal)
→ PathResolver.getChildren() [all dirs at level processed in parallel]
↓ BATCHED
→ brain.batchGet(childIds) [1 call instead of N]
↓ BATCHED
→ storage.getNounMetadataBatch(ids) [1 call instead of N]
↓ ADAPTER-SPECIFIC
→ GCS: readBatch() with 100 concurrent downloads
→ S3: readBatch() with 150 concurrent downloads
→ Memory: Promise.all() parallel reads
↓ PARALLEL (breadth-first traversal)
→ PathResolver.getChildren() [all dirs at level processed in parallel]
↓ BATCHED
→ brain.batchGet(childIds) [1 call instead of N]
↓ BATCHED
→ storage.getNounMetadataBatch(ids) [1 call instead of N]
↓ ADAPTER-SPECIFIC
→ GCS: readBatch() with 100 concurrent downloads
→ S3: readBatch() with 150 concurrent downloads
→ Memory: Promise.all() parallel reads
```
**Performance Gains:**
@ -285,11 +284,11 @@ VFS.getTreeStructure()
## Advanced Features Compatibility
### ✅ ID-First Storage Architecture (v6.0.0+)
### ✅ ID-First Storage Architecture
All batch operations use direct ID-first paths - no type lookup needed!
**NEW v6.0.0 Path Structure:**
**ID-First Path Structure:**
```
entities/nouns/{SHARD}/{ID}/metadata.json
entities/verbs/{SHARD}/{ID}/metadata.json
@ -299,7 +298,7 @@ entities/verbs/{SHARD}/{ID}/metadata.json
```typescript
// Every ID maps directly to exactly ONE path - 40x faster!
const id = 'abc-123'
const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
const path = `entities/nouns/${shard}/${id}/metadata.json`
// No type cache needed!
@ -343,7 +342,7 @@ const fork = await brain.fork('experiment')
// Batch operations are isolated
await brain.batchGet([id1, id2]) // → Reads from: branches/main/...
await fork.batchGet([id1, id2]) // → Reads from: branches/experiment/...
await fork.batchGet([id1, id2]) // → Reads from: branches/experiment/...
```
**Inheritance:**
@ -391,7 +390,7 @@ Historical queries use `HistoricalStorageAdapter` which wraps batch operations t
### VFS Operations (12 Files)
| Storage | Before v5.12.0 | After v5.12.0 | Improvement |
| Storage | Before optimization | After optimization | Improvement |
|---------|---------------|---------------|-------------|
| **GCS** | 12.7s | <1s | **92% faster** |
| **S3** | 13.2s | <1s | **92% faster** |
@ -430,8 +429,8 @@ Batch operations gracefully handle missing or invalid entities:
```typescript
const validId = 'abc-123-...'
const invalidIds = [
'11111111-1111-1111-1111-111111111111',
'22222222-2222-2222-2222-222222222222'
'11111111-1111-1111-1111-111111111111',
'22222222-2222-2222-2222-222222222222'
]
const results = await brain.batchGet([validId, ...invalidIds])
@ -471,8 +470,8 @@ results.size // → 1 (deduplicated automatically)
```typescript
const entities = []
for (const id of ids) {
const entity = await brain.get(id)
if (entity) entities.push(entity)
const entity = await brain.get(id)
if (entity) entities.push(entity)
}
```
@ -492,8 +491,8 @@ const entities = Array.from(results.values())
```typescript
const allVerbs = []
for (const sourceId of sourceIds) {
const verbs = await brain.getRelations({ from: sourceId })
allVerbs.push(...verbs)
const verbs = await brain.getRelations({ from: sourceId })
allVerbs.push(...verbs)
}
```
@ -504,7 +503,7 @@ const results = await storage.getVerbsBySourceBatch(sourceIds)
const allVerbs = []
for (const verbs of results.values()) {
allVerbs.push(...verbs)
allVerbs.push(...verbs)
}
```
@ -522,7 +521,7 @@ const results = await brain.batchGet(ids)
// ❌ BAD: Individual gets in loop
for (const id of ids) {
await brain.get(id)
await brain.get(id)
}
```
@ -556,13 +555,13 @@ const results = await brain.batchGet(ids)
// Check results
for (const id of ids) {
if (results.has(id)) {
// Entity exists
const entity = results.get(id)
} else {
// Entity missing (not an error)
console.log(`Entity ${id} not found`)
}
if (results.has(id)) {
// Entity exists
const entity = results.get(id)
} else {
// Entity missing (not an error)
console.log(`Entity ${id} not found`)
}
}
```
@ -597,15 +596,15 @@ npx vitest run tests/integration/storage-batch-operations.test.ts
```
User Code (brain.batchGet)
↓
↓
High-Level API (src/brainy.ts)
↓
↓
Storage Layer (src/storage/baseStorage.ts)
↓
↓
COW Layer (readBatchWithInheritance)
↓
↓
Adapter Layer (readBatchFromAdapter)
↓
↓
Cloud Adapter (GCS/S3/Azure native batch APIs)
```
@ -616,11 +615,11 @@ If an adapter doesn't implement `readBatch()`, the system automatically falls ba
```typescript
// BaseStorage.readBatchFromAdapter()
if (typeof selfWithBatch.readBatch === 'function') {
// Use native batch API
return await selfWithBatch.readBatch(resolvedPaths)
// Use native batch API
return await selfWithBatch.readBatch(resolvedPaths)
} else {
// Automatic parallel fallback
return await Promise.all(resolvedPaths.map(path => this.read(path)))
// Automatic parallel fallback
return await Promise.all(resolvedPaths.map(path => this.read(path)))
}
```
@ -658,7 +657,7 @@ if (typeof selfWithBatch.readBatch === 'function') {
- Zero N+1 query patterns
**Compatibility:**
- ✅ ID-first storage (v6.0.0+)
- ✅ ID-first storage
- ✅ Sharding (256 shards)
- ✅ COW (branch isolation, inheritance)
- ✅ fork() and checkout()

402
docs/CREATING-AUGMENTATIONS.md
View file

@ -1,6 +1,6 @@
# Creating Augmentations for Brainy
> **Updated for v4.0.0** - Includes metadata structure changes and type system improvements
> **Updated** - Includes metadata structure changes and type system improvements
## The BrainyAugmentation Interface
@ -8,38 +8,38 @@ Every augmentation implements this simple yet powerful interface:
```typescript
interface BrainyAugmentation {
// Identification
name: string // Unique name for your augmentation
// Identification
name: string // Unique name for your augmentation
// Execution control
timing: 'before' | 'after' | 'around' | 'replace' // When to execute
operations: string[] // Which operations to intercept
priority: number // Execution order (higher = first)
// Execution control
timing: 'before' | 'after' | 'around' | 'replace' // When to execute
operations: string[] // Which operations to intercept
priority: number // Execution order (higher = first)
// Lifecycle methods
initialize(context: AugmentationContext): Promise<void>
execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T>
shutdown?(): Promise<void> // Optional cleanup
// Lifecycle methods
initialize(context: AugmentationContext): Promise<void>
execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T>
shutdown?(): Promise<void> // Optional cleanup
}
```
## v4.0.0 Breaking Changes for Augmentation Developers
## Breaking Changes for Augmentation Developers
### 1. Metadata Structure Separation
v4.0.0 introduces strict metadata/vector separation for billion-scale performance:
Brainy introduces strict metadata/vector separation for billion-scale performance:
```typescript
// ✅ v4.0.0: Metadata has required type field
// ✅ Metadata has required type field
interface NounMetadata {
noun: NounType // Required! Must be a valid noun type
[key: string]: any // Your custom metadata
noun: NounType // Required! Must be a valid noun type
[key: string]: any // Your custom metadata
}
interface VerbMetadata {
verb: VerbType // Required! Must be a valid verb type
sourceId: string
targetId: string
[key: string]: any
verb: VerbType // Required! Must be a valid verb type
sourceId: string
targetId: string
[key: string]: any
}
```
@ -61,7 +61,7 @@ The verb relationship field changed from `type` to `verb`:
// ❌ v3.x
verb.type === 'relatedTo'
// ✅ v4.0.0
// ✅ Current
verb.verb === 'relatedTo'
```
@ -69,7 +69,7 @@ verb.verb === 'relatedTo'
Storage augmentations are special - they provide the storage backend for Brainy.
### Important: v4.0.0 Storage Requirements
### Important: Storage Requirements
Your storage adapter MUST:
1. **Wrap metadata** with required `noun`/`verb` fields
@ -81,67 +81,67 @@ import { StorageAugmentation } from 'brainy/augmentations'
import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy'
export class MyCustomStorage extends BaseStorageAdapter {
// Internal method: Returns pure structure
async _getNoun(id: string): Promise<HNSWNoun | null> {
const data = await this.fetchFromDatabase(id)
return data ? {
id: data.id,
vector: data.vector,
nounType: data.type
} : null
}
// Internal method: Returns pure structure
async _getNoun(id: string): Promise<HNSWNoun | null> {
const data = await this.fetchFromDatabase(id)
return data ? {
id: data.id,
vector: data.vector,
nounType: data.type
} : null
}
// Public method: Returns WithMetadata structure
async getNoun(id: string): Promise<HNSWNounWithMetadata | null> {
const noun = await this._getNoun(id)
if (!noun) return null
// Public method: Returns WithMetadata structure
async getNoun(id: string): Promise<HNSWNounWithMetadata | null> {
const noun = await this._getNoun(id)
if (!noun) return null
// Fetch metadata separately (v4.0.0 pattern)
const metadata = await this.getNounMetadata(id)
// Fetch metadata separately
const metadata = await this.getNounMetadata(id)
return {
...noun,
metadata: metadata || { noun: noun.nounType || 'thing' }
}
}
return {
...noun,
metadata: metadata || { noun: noun.nounType || 'thing' }
}
}
// CRITICAL: Always save with proper metadata structure
async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> {
// Validate metadata has required 'noun' field
if (!metadata?.noun) {
throw new Error('v4.0.0: NounMetadata requires "noun" field')
}
// CRITICAL: Always save with proper metadata structure
async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> {
// Validate metadata has required 'noun' field
if (!metadata?.noun) {
throw new Error('NounMetadata requires "noun" field')
}
await this.database.save({
id: noun.id,
vector: noun.vector,
nounType: noun.nounType,
metadata: metadata // Stored separately in v4.0.0
})
}
await this.database.save({
id: noun.id,
vector: noun.vector,
nounType: noun.nounType,
metadata: metadata // Stored separately
})
}
}
export class MyStorageAugmentation extends StorageAugmentation {
private config: MyStorageConfig
private config: MyStorageConfig
constructor(config: MyStorageConfig) {
super()
this.name = 'my-custom-storage'
this.config = config
}
constructor(config: MyStorageConfig) {
super()
this.name = 'my-custom-storage'
this.config = config
}
// Called during storage resolution phase
async provideStorage(): Promise<StorageAdapter> {
const storage = new MyCustomStorage(this.config)
this.storageAdapter = storage
return storage
}
// Called during storage resolution phase
async provideStorage(): Promise<StorageAdapter> {
const storage = new MyCustomStorage(this.config)
this.storageAdapter = storage
return storage
}
// Called during augmentation initialization
protected async onInitialize(): Promise<void> {
await this.storageAdapter!.init()
this.log(`Custom storage initialized`)
}
// Called during augmentation initialization
protected async onInitialize(): Promise<void> {
await this.storageAdapter!.init()
this.log(`Custom storage initialized`)
}
}
```
@ -151,9 +151,9 @@ export class MyStorageAugmentation extends StorageAugmentation {
// Register before brain.init()
const brain = new Brainy()
brain.augmentations.register(new MyStorageAugmentation({
connectionString: 'redis://localhost:6379'
connectionString: 'redis://localhost:6379'
}))
await brain.init() // Will use your storage!
await brain.init() // Will use your storage!
```
## Creating a Feature Augmentation
@ -164,43 +164,43 @@ Here's a complete example of a caching augmentation:
import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations'
export class CachingAugmentation extends BaseAugmentation {
private cache = new Map<string, any>()
private cache = new Map<string, any>()
constructor() {
super()
this.name = 'smart-cache'
this.timing = 'around' // Wrap operations
this.operations = ['search'] // Only cache searches
this.priority = 50 // Mid-priority
}
constructor() {
super()
this.name = 'smart-cache'
this.timing = 'around' // Wrap operations
this.operations = ['search'] // Only cache searches
this.priority = 50 // Mid-priority
}
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
if (operation === 'search') {
// Check cache
const cacheKey = JSON.stringify(params)
if (this.cache.has(cacheKey)) {
this.log('Cache hit!')
return this.cache.get(cacheKey)
}
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
if (operation === 'search') {
// Check cache
const cacheKey = JSON.stringify(params)
if (this.cache.has(cacheKey)) {
this.log('Cache hit!')
return this.cache.get(cacheKey)
}
// Execute and cache
const result = await next()
this.cache.set(cacheKey, result)
return result
}
// Execute and cache
const result = await next()
this.cache.set(cacheKey, result)
return result
}
// Pass through other operations
return next()
}
// Pass through other operations
return next()
}
protected async onInitialize(): Promise<void> {
this.log('Cache initialized')
}
protected async onInitialize(): Promise<void> {
this.log('Cache initialized')
}
async shutdown(): Promise<void> {
this.cache.clear()
await super.shutdown()
}
async shutdown(): Promise<void> {
this.cache.clear()
await super.shutdown()
}
}
```
@ -210,9 +210,9 @@ export class CachingAugmentation extends BaseAugmentation {
```typescript
timing = 'before'
async execute(op, params, next) {
// Validate/transform input
const validated = await validate(params)
return next(validated) // Pass modified params
// Validate/transform input
const validated = await validate(params)
return next(validated) // Pass modified params
}
```
@ -220,10 +220,10 @@ async execute(op, params, next) {
```typescript
timing = 'after'
async execute(op, params, next) {
const result = await next()
// Log, analyze, or modify result
console.log(`Operation ${op} returned:`, result)
return result
const result = await next()
// Log, analyze, or modify result
console.log(`Operation ${op} returned:`, result)
return result
}
```
@ -231,15 +231,15 @@ async execute(op, params, next) {
```typescript
timing = 'around'
async execute(op, params, next) {
console.log('Starting', op)
try {
const result = await next()
console.log('Success', op)
return result
} catch (error) {
console.log('Failed', op, error)
throw error
}
console.log('Starting', op)
try {
const result = await next()
console.log('Success', op)
return result
} catch (error) {
console.log('Failed', op, error)
throw error
}
}
```
@ -247,8 +247,8 @@ async execute(op, params, next) {
```typescript
timing = 'replace'
async execute(op, params, next) {
// Don't call next() - replace entirely!
return myCustomImplementation(params)
// Don't call next() - replace entirely!
return myCustomImplementation(params)
}
```
@ -266,10 +266,10 @@ Common operations in Brainy:
```typescript
interface AugmentationContext {
brain: Brainy // The brain instance
storage: StorageAdapter // Storage backend
config: BrainyConfig // Configuration
log: (message: string, level?: 'info' | 'warn' | 'error') => void
brain: Brainy // The brain instance
storage: StorageAdapter // Storage backend
config: BrainyConfig // Configuration
log: (message: string, level?: 'info' | 'warn' | 'error') => void
}
```
@ -278,50 +278,50 @@ interface AugmentationContext {
### 1. Redis Storage Augmentation
```typescript
export class RedisStorageAugmentation extends StorageAugmentation {
async provideStorage(): Promise<StorageAdapter> {
return new RedisAdapter({
host: 'localhost',
port: 6379,
// Implement full StorageAdapter interface
})
}
async provideStorage(): Promise<StorageAdapter> {
return new RedisAdapter({
host: 'localhost',
port: 6379,
// Implement full StorageAdapter interface
})
}
}
```
### 2. Audit Trail Augmentation
```typescript
export class AuditAugmentation extends BaseAugmentation {
timing = 'after'
operations = ['add', 'update', 'delete']
timing = 'after'
operations = ['add', 'update', 'delete']
async execute(op, params, next) {
const result = await next()
async execute(op, params, next) {
const result = await next()
// Log to audit trail
await this.logAudit({
operation: op,
params,
result,
timestamp: new Date(),
user: this.context.config.currentUser
})
// Log to audit trail
await this.logAudit({
operation: op,
params,
result,
timestamp: new Date(),
user: this.context.config.currentUser
})
return result
}
return result
}
}
```
### 3. Rate Limiting Augmentation
```typescript
export class RateLimitAugmentation extends BaseAugmentation {
timing = 'before'
operations = ['search']
private limiter = new RateLimiter({ rps: 100 })
timing = 'before'
operations = ['search']
private limiter = new RateLimiter({ rps: 100 })
async execute(op, params, next) {
await this.limiter.acquire() // Wait if rate limited
return next()
}
async execute(op, params, next) {
await this.limiter.acquire() // Wait if rate limited
return next()
}
}
```
@ -332,12 +332,12 @@ Future capability for premium augmentations:
```typescript
// package.json
{
"name": "@brain-cloud/redis-storage",
"brainy": {
"type": "augmentation",
"category": "storage",
"premium": true
}
"name": "@brain-cloud/redis-storage",
"brainy": {
"type": "augmentation",
"category": "storage",
"premium": true
}
}
// Users can install via:
@ -356,43 +356,43 @@ Future capability for premium augmentations:
6. **Log appropriately** - Use context.log() for consistent output
7. **Document your augmentation** - Include examples
### v4.0.0 Specific Best Practices
### Specific Best Practices
8. **Always include `noun` field** when creating/modifying NounMetadata:
```typescript
const metadata: NounMetadata = {
noun: 'thing', // REQUIRED!
yourField: 'value'
}
```
```typescript
const metadata: NounMetadata = {
noun: 'thing', // REQUIRED!
yourField: 'value'
}
```
9. **Use `verb` property** not `type` when working with relationships:
```typescript
// ✅ Correct
if (verb.verb === 'relatedTo') { ... }
```typescript
// ✅ Correct
if (verb.verb === 'relatedTo') { ... }
// ❌ Wrong (v3.x pattern)
if (verb.type === 'relatedTo') { ... }
```
// ❌ Wrong (v3.x pattern)
if (verb.type === 'relatedTo') { ... }
```
10. **Access metadata correctly** from storage:
```typescript
// ✅ Correct - metadata is already structured
const nounType = noun.metadata.noun
```typescript
// ✅ Correct - metadata is already structured
const nounType = noun.metadata.noun
// ⚠️ Fallback pattern for robustness
const nounType = noun.metadata?.noun || 'thing'
```
// ⚠️ Fallback pattern for robustness
const nounType = noun.metadata?.noun || 'thing'
```
11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations:
```typescript
// ✅ Good - Separate concerns
await storage.saveNoun(noun)
await storage.saveMetadata(noun.id, metadata)
```typescript
// ✅ Good - Separate concerns
await storage.saveNoun(noun)
await storage.saveMetadata(noun.id, metadata)
// ❌ Bad - Mixing concerns
await storage.saveNounWithEverything(combinedData)
```
// ❌ Bad - Mixing concerns
await storage.saveNounWithEverything(combinedData)
```
## Testing Your Augmentation
@ -401,23 +401,23 @@ import { Brainy } from 'brainy'
import { MyAugmentation } from './my-augmentation'
describe('MyAugmentation', () => {
let brain: Brainy
let brain: Brainy
beforeEach(async () => {
brain = new Brainy()
brain.augmentations.register(new MyAugmentation())
await brain.init()
})
beforeEach(async () => {
brain = new Brainy()
brain.augmentations.register(new MyAugmentation())
await brain.init()
})
afterEach(async () => {
await brain.destroy()
})
afterEach(async () => {
await brain.destroy()
})
it('should enhance searches', async () => {
// Test your augmentation's effect
const results = await brain.search('test')
expect(results).toHaveProperty('enhanced', true)
})
it('should enhance searches', async () => {
// Test your augmentation's effect
const results = await brain.search('test')
expect(results).toHaveProperty('enhanced', true)
})
})
```

4
docs/DEVELOPER_LEARNING_PATH.md
View file

@ -613,7 +613,7 @@ vfsFiles.forEach(f => {
})
// 3. VFS FILTERING IN KNOWLEDGE QUERIES
console.log('\n🔍 Understanding VFS filtering (v4.4.0)...\n')
console.log('\n🔍 Understanding VFS filtering...\n')
// Create some knowledge entities
const conceptId = await brain.add({
@ -758,7 +758,7 @@ await brain.close()
### Key Concepts
#### 1. **VFS Filtering Architecture (v4.4.0)**
#### 1. **VFS Filtering Architecture**
```typescript
// 🎯 DEFAULT BEHAVIOR: Clean Separation

793
docs/FIND_SYSTEM.md
View file

File diff suppressed because it is too large Load diff

8
docs/PERFORMANCE.md
View file

@ -24,11 +24,11 @@ Where:
- `f` = number of fields for entity type
- `t` = number of types (42 nouns, 127 verbs)
### v5.11.1: brain.get() Metadata-Only Optimization
### brain.get() Metadata-Only Optimization
✨ **Massive Performance Improvement**: `brain.get()` is now **76-81% faster** by default!
| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup | Use Case |
| Operation | Before | After | Speedup | Use Case |
|-----------|------------------|-----------------|---------|----------|
| **brain.get() (metadata-only)** | 43ms, 6KB | **10ms, 300 bytes** | **76-81%** | VFS, existence checks, metadata |
| **brain.get({ includeVectors: true })** | 43ms, 6KB | 43ms, 6KB | 0% | Similarity calculations |
@ -308,7 +308,7 @@ const results = await Promise.all(searchPromises)
- ✅ **Horizontally Scalable**: Stateless operations support clustering
- ✅ **Zero Stubs**: Every line of code is production-ready
## Lazy Loading Performance (v5.7.7+)
## Lazy Loading Performance
Brainy supports two initialization modes for optimal performance across different use cases:
@ -324,7 +324,7 @@ await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities)
- First query: Instant (indexes already loaded)
- Use case: Traditional applications, long-running servers
### Mode 2: Lazy Loading (v5.7.7+)
### Mode 2: Lazy Loading
```javascript
const brain = new Brainy({ disableAutoRebuild: true })

147
docs/README.md
View file

@ -1,8 +1,8 @@
# Brainy Documentation (v6.5.0)
# Brainy Documentation
Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine.
## 🆕 What's New in v4.0.0
## 🆕 What's New in
**Production-Ready Cost Optimization:**
- **Lifecycle Management**: Automatic tier transitions for S3, GCS, Azure (96% cost savings!)
@ -14,7 +14,7 @@ Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI
**Cost Impact Example (500TB dataset):**
- Before: $138,000/year
- After v4.0.0: $5,940/year
- After $5,940/year
- **Savings: $132,060/year (96%)**
## 📊 Implementation Status
@ -85,23 +85,23 @@ await brain.init()
// Add entities (nouns)
const articleId = await brain.add({
data: "Revolutionary AI Breakthrough",
type: NounType.Document,
metadata: { category: "technology", rating: 4.8 }
data: "Revolutionary AI Breakthrough",
type: NounType.Document,
metadata: { category: "technology", rating: 4.8 }
})
const authorId = await brain.add({
data: "Dr. Sarah Chen",
type: NounType.Person,
metadata: { role: "researcher" }
data: "Dr. Sarah Chen",
type: NounType.Person,
metadata: { role: "researcher" }
})
// Create relationships (verbs)
await brain.relate({
from: authorId,
to: articleId,
type: VerbType.CreatedBy,
metadata: { date: "2024-01-15", contribution: "primary" }
from: authorId,
to: articleId,
type: VerbType.CreatedBy,
metadata: { date: "2024-01-15", contribution: "primary" }
})
// Query naturally
@ -117,7 +117,7 @@ const results = await brain.find("highly rated technology articles by researcher
| [API Reference](./api/README.md) | **START HERE** - Complete API documentation with examples |
| [VFS Quick Start](./vfs/QUICK_START.md) | Virtual filesystem in 30 seconds |
### 🆕 v4.0.0 Migration & Optimization
### 🆕 Migration & Optimization
| Document | Description |
|----------|-------------|
@ -135,15 +135,15 @@ const results = await brain.find("highly rated technology articles by researcher
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | Revolutionary data model - entities and relationships |
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Field unified query system |
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
| [Storage Architecture](./architecture/storage-architecture.md) | **v4.0.0** - Storage adapters and optimization |
| [Data Storage Architecture](./architecture/data-storage-architecture.md) | **v4.0.0** - Metadata/vector separation, sharding |
| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
| [Data Storage Architecture](./architecture/data-storage-architecture.md) | Metadata/vector separation, sharding |
| [Index Architecture](./architecture/index-architecture.md) | HNSW, Graph, and Metadata indexing |
### 💾 Storage & Deployment
| Document | Description |
|----------|-------------|
| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | **v4.0.0** - Deploy on AWS, GCP, Azure, Cloudflare |
| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | Deploy on AWS, GCP, Azure, Cloudflare |
| [AWS Deployment](./deployment/aws-deployment.md) | AWS-specific deployment patterns |
| [GCP Deployment](./deployment/gcp-deployment.md) | Google Cloud deployment |
| [Kubernetes Deployment](./deployment/kubernetes-deployment.md) | K8s deployment configurations |
@ -257,75 +257,68 @@ const results = await brain.find("highly rated technology articles by researcher
```
docs/
├── README.md (this file) # Complete documentation index
├── MIGRATION-V3-TO-V4.md # v4.0.0 migration guide
├── README.md (this file) # Complete documentation index
├── MIGRATION-V3-TO-V4.md # Migration guide
│
├── guides/ # User guides
│ ├── natural-language.md
│ ├── neural-api.md
│ ├── import-anything.md
│ ├── framework-integration.md
│ ├── nextjs-integration.md
│ ├── vue-integration.md
│ ├── distributed-system.md
│ ├── model-loading.md
│ └── enterprise-for-everyone.md
├── guides/ # User guides
│ ├── natural-language.md
│ ├── neural-api.md
│ ├── import-anything.md
│ ├── framework-integration.md
│ ├── nextjs-integration.md
│ ├── vue-integration.md
│ ├── distributed-system.md
│ ├── model-loading.md
│ └── enterprise-for-everyone.md
│
├── architecture/ # System architecture
│ ├── overview.md
│ ├── noun-verb-taxonomy.md
│ ├── triple-intelligence.md
│ ├── zero-config.md
│ ├── storage-architecture.md # v4.0.0
│ ├── data-storage-architecture.md # v4.0.0
│ ├── index-architecture.md
│ ├── distributed-storage.md
│ ├── augmentations.md
│ ├── augmentation-system-audit.md
│ ├── augmentations-actual.md
│ ├── finite-type-system.md
│ └── ...
├── architecture/ # System architecture
│ ├── overview.md
│ ├── noun-verb-taxonomy.md
│ ├── triple-intelligence.md
│ ├── zero-config.md
│ ├── storage-architecture.md│ ├── data-storage-architecture.md│ ├── index-architecture.md
│ ├── distributed-storage.md
│ ├── augmentations.md
│ ├── augmentation-system-audit.md
│ ├── augmentations-actual.md
│ ├── finite-type-system.md
│ └── ...
│
├── operations/ # Operations guides
│ ├── cost-optimization-aws-s3.md # v4.0.0
│ ├── cost-optimization-gcs.md # v4.0.0
│ ├── cost-optimization-azure.md # v4.0.0
│ ├── cost-optimization-cloudflare-r2.md # v4.0.0
│ └── capacity-planning.md
├── operations/ # Operations guides
│ ├── cost-optimization-aws-s3.md│ ├── cost-optimization-gcs.md│ ├── cost-optimization-azure.md│ ├── cost-optimization-cloudflare-r2.md│ └── capacity-planning.md
│
├── deployment/ # Deployment guides
│ ├── CLOUD_DEPLOYMENT_GUIDE.md # v4.0.0
│ ├── aws-deployment.md
│ ├── gcp-deployment.md
│ └── kubernetes-deployment.md
├── deployment/ # Deployment guides
│ ├── CLOUD_DEPLOYMENT_GUIDE.md│ ├── aws-deployment.md
│ ├── gcp-deployment.md
│ └── kubernetes-deployment.md
│
├── vfs/ # Virtual Filesystem docs
│ ├── QUICK_START.md
│ ├── VFS_CORE.md
│ ├── SEMANTIC_VFS.md
│ ├── VFS_API_GUIDE.md
│ ├── NEURAL_EXTRACTION.md
│ ├── COMMON_PATTERNS.md
│ └── ...
├── vfs/ # Virtual Filesystem docs
│ ├── QUICK_START.md
│ ├── VFS_CORE.md
│ ├── SEMANTIC_VFS.md
│ ├── VFS_API_GUIDE.md
│ ├── NEURAL_EXTRACTION.md
│ ├── COMMON_PATTERNS.md
│ └── ...
│
├── api/ # API documentation
│ ├── README.md
│ └── COMPREHENSIVE_API_OVERVIEW.md
├── api/ # API documentation
│ ├── README.md
│ └── COMPREHENSIVE_API_OVERVIEW.md
│
├── augmentations/ # Augmentation docs
│ ├── COMPLETE-REFERENCE.md
│ ├── DEVELOPER-GUIDE.md
│ ├── CONFIGURATION.md
│ └── api-server.md
├── augmentations/ # Augmentation docs
│ ├── COMPLETE-REFERENCE.md
│ ├── DEVELOPER-GUIDE.md
│ ├── CONFIGURATION.md
│ └── api-server.md
│
├── features/ # Feature documentation
│ ├── complete-feature-list.md
│ └── v3-features.md
├── features/ # Feature documentation
│ ├── complete-feature-list.md
│ └── v3-features.md
│
└── internal/ # Internal docs
├── AUDIT_REPORT.md
├── CLEANUP_SUMMARY.md
└── HONEST_STATUS.md
└── internal/ # Internal docs
├── AUDIT_REPORT.md
├── CLEANUP_SUMMARY.md
└── HONEST_STATUS.md
```
## Community

2
docs/RELEASE-GUIDE.md
View file

@ -48,7 +48,7 @@ git commit -m "chore: remove unused dependency"
# DON'T use BREAKING CHANGE for internal changes
git commit -m "feat: improve model delivery
BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers v3.0.0
BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers
```
## Release Workflow Checklist

2
docs/STAGE3-CANONICAL-TAXONOMY.md
View file

@ -1,4 +1,4 @@
# Brainy Stage 3: Canonical Taxonomy (v6.0.0)
# Brainy Stage 3: Canonical Taxonomy
**Status:** FINAL - This is the definitive, timeless taxonomy
**Total Types:** 169 (42 nouns + 127 verbs)

234
docs/api/COMPREHENSIVE_API_OVERVIEW.md
View file

@ -1,6 +1,6 @@
# Brainy Complete Public API Reference
> **Accurate API documentation for Brainy v6.5.0+**
> **Accurate API documentation for Brainy**
## Initialization
@ -13,14 +13,14 @@ await brain.init()
// With configuration
const brain = new Brainy({
storage: { type: 'memory' }, // or { path: './my-data' } for filesystem
embeddingModel: 'Q8', // Q4, Q8, F16, F32
silent: true // Suppress logs
storage: { type: 'memory' }, // or { path: './my-data' } for filesystem
embeddingModel: 'Q8', // Q4, Q8, F16, F32
silent: true // Suppress logs
})
await brain.init()
```
## Readiness API (v7.3.0+)
## Readiness API
For reliable initialization detection, especially in cloud environments with progressive initialization:
@ -28,10 +28,10 @@ For reliable initialization detection, especially in cloud environments with pro
```typescript
const brain = new Brainy()
brain.init() // Fire and forget
brain.init() // Fire and forget
// Elsewhere (e.g., API handler)
await brain.ready // Wait until init() completes
await brain.ready // Wait until init() completes
const results = await brain.find({ query: 'test' })
```
@ -39,7 +39,7 @@ const results = await brain.find({ query: 'test' })
```typescript
if (brain.isInitialized) {
// Safe to use brain methods
// Safe to use brain methods
}
```
@ -49,7 +49,7 @@ if (brain.isInitialized) {
// Returns true when ALL initialization is complete, including background tasks
// Useful for cloud storage adapters with progressive initialization
if (brain.isFullyInitialized()) {
console.log('All background tasks complete')
console.log('All background tasks complete')
}
```
@ -57,7 +57,7 @@ if (brain.isFullyInitialized()) {
```typescript
const brain = new Brainy({ storage: { type: 'gcs', ... } })
await brain.init() // Fast return in cloud (<200ms)
await brain.init() // Fast return in cloud (<200ms)
// Optional: wait for all background tasks (bucket validation, count sync)
await brain.awaitBackgroundInit()
@ -68,15 +68,15 @@ console.log('Fully initialized including background tasks')
```typescript
app.get('/health', async (req, res) => {
try {
await brain.ready
res.json({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
} catch (error) {
res.status(503).json({ status: 'initializing', error: error.message })
}
try {
await brain.ready
res.json({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
} catch (error) {
res.status(503).json({ status: 'initializing', error: error.message })
}
})
```
@ -87,16 +87,16 @@ app.get('/health', async (req, res) => {
```typescript
// Add with data (auto-embedded)
const id = await brain.add({
data: 'Machine learning is a subset of AI',
type: NounType.Document,
metadata: { author: 'Alice', tags: ['ml', 'ai'] }
data: 'Machine learning is a subset of AI',
type: NounType.Document,
metadata: { author: 'Alice', tags: ['ml', 'ai'] }
})
// Add with pre-computed vector
const id = await brain.add({
data: 'Content here',
vector: [0.1, 0.2, ...], // 384 dimensions
type: NounType.Concept
data: 'Content here',
vector: [0.1, 0.2, ...], // 384 dimensions
type: NounType.Concept
})
```
@ -120,9 +120,9 @@ const entity = await brain.get('entity-id', { includeVectors: true })
```typescript
await brain.update({
id: 'entity-id',
data: 'Updated content', // Re-embeds if changed
metadata: { reviewed: true } // Merges with existing
id: 'entity-id',
data: 'Updated content', // Re-embeds if changed
metadata: { reviewed: true } // Merges with existing
})
```
@ -144,10 +144,10 @@ await brain.clear()
```typescript
const relationId = await brain.relate({
from: 'source-entity-id',
to: 'target-entity-id',
type: VerbType.RelatedTo,
metadata: { strength: 0.9 }
from: 'source-entity-id',
to: 'target-entity-id',
type: VerbType.RelatedTo,
metadata: { strength: 0.9 }
})
```
@ -168,8 +168,8 @@ const relations = await brain.getRelations({ to: 'entity-id' })
// Filter by type
const relations = await brain.getRelations({
from: 'entity-id',
type: VerbType.Contains
from: 'entity-id',
type: VerbType.Contains
})
```
@ -191,17 +191,17 @@ const results = await brain.find('machine learning algorithms')
// With options
const results = await brain.find({
query: 'machine learning',
limit: 10,
threshold: 0.7,
type: NounType.Document,
where: { author: 'Alice' },
excludeVFS: true // Exclude VFS files from results
query: 'machine learning',
limit: 10,
threshold: 0.7,
type: NounType.Document,
where: { author: 'Alice' },
excludeVFS: true // Exclude VFS files from results
})
// Natural language query (Triple Intelligence)
const results = await brain.find(
'Show me documents about AI written by Alice in 2024'
'Show me documents about AI written by Alice in 2024'
)
```
@ -218,15 +218,15 @@ const results = await brain.find(
```typescript
// Find similar to entity ID
const similar = await brain.similar({
to: 'entity-id',
limit: 5
to: 'entity-id',
limit: 5
})
// Find similar to vector
const similar = await brain.similar({
to: [0.1, 0.2, ...], // Vector
limit: 10,
type: NounType.Document
to: [0.1, 0.2, ...], // Vector
limit: 10,
type: NounType.Document
})
```
@ -236,15 +236,15 @@ const similar = await brain.similar({
```typescript
const result = await brain.addMany({
items: [
{ data: 'First item', type: NounType.Document },
{ data: 'Second item', type: NounType.Concept },
{ data: 'Third item', metadata: { priority: 'high' } }
],
continueOnError: true, // Don't stop on failures
onProgress: (completed, total) => {
console.log(`${completed}/${total} complete`)
}
items: [
{ data: 'First item', type: NounType.Document },
{ data: 'Second item', type: NounType.Concept },
{ data: 'Third item', metadata: { priority: 'high' } }
],
continueOnError: true, // Don't stop on failures
onProgress: (completed, total) => {
console.log(`${completed}/${total} complete`)
}
})
console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}`)
@ -255,18 +255,18 @@ console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}
```typescript
// Delete by IDs
const result = await brain.deleteMany({
ids: ['id1', 'id2', 'id3'],
continueOnError: true
ids: ['id1', 'id2', 'id3'],
continueOnError: true
})
// Delete by type
const result = await brain.deleteMany({
type: NounType.TempData
type: NounType.TempData
})
// Delete by metadata filter
const result = await brain.deleteMany({
where: { status: 'archived' }
where: { status: 'archived' }
})
```
@ -274,12 +274,12 @@ const result = await brain.deleteMany({
```typescript
const ids = await brain.relateMany({
items: [
{ from: 'a', to: 'b', type: VerbType.RelatedTo },
{ from: 'b', to: 'c', type: VerbType.Contains },
{ from: 'c', to: 'd', type: VerbType.References }
],
continueOnError: true
items: [
{ from: 'a', to: 'b', type: VerbType.RelatedTo },
{ from: 'b', to: 'c', type: VerbType.Contains },
{ from: 'c', to: 'd', type: VerbType.References }
],
continueOnError: true
})
```
@ -287,11 +287,11 @@ const ids = await brain.relateMany({
```typescript
const result = await brain.updateMany({
items: [
{ id: 'id1', metadata: { reviewed: true } },
{ id: 'id2', data: 'Updated content' }
],
continueOnError: true
items: [
{ id: 'id1', metadata: { reviewed: true } },
{ id: 'id2', data: 'Updated content' }
],
continueOnError: true
})
```
@ -309,8 +309,8 @@ const detailed = await neural.similar('text1', 'text2', { detailed: true })
// Clustering
const clusters = await neural.clusters()
const clusters = await neural.clusters({
algorithm: 'hierarchical',
maxClusters: 10
algorithm: 'hierarchical',
maxClusters: 10
})
// K-nearest neighbors
@ -327,13 +327,13 @@ const domainClusters = await neural.clusterByDomain('category')
// Temporal clustering
const temporalClusters = await neural.clusterByTime('createdAt', [
{ start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1' },
{ start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2' }
{ start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1' },
{ start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2' }
])
// Streaming clusters (for large datasets)
for await (const batch of neural.clusterStream({ batchSize: 100 })) {
console.log(`Progress: ${batch.progress.percentage}%`)
console.log(`Progress: ${batch.progress.percentage}%`)
}
```
@ -355,11 +355,11 @@ await vfs.mkdir('/project/src', { recursive: true })
const files = await vfs.readdir('/project')
await vfs.rmdir('/project', { recursive: true })
// Bulk operations (v6.5.0+)
// Bulk operations
const result = await vfs.bulkWrite([
{ type: 'mkdir', path: '/data' },
{ type: 'write', path: '/data/config.json', data: '{}' },
{ type: 'write', path: '/data/users.json', data: '[]' }
{ type: 'mkdir', path: '/data' },
{ type: 'write', path: '/data/config.json', data: '{}' },
{ type: 'write', path: '/data/users.json', data: '[]' }
])
// Note: mkdir operations run first (sequentially), then other ops in parallel
@ -414,22 +414,22 @@ const snapshot = await brain.asOf('commit-id')
```typescript
// Stream all entities
for await (const entity of brain.streaming.entities()) {
console.log(entity.id)
console.log(entity.id)
}
// Stream with filters
for await (const entity of brain.streaming.entities({
type: NounType.Document,
where: { status: 'active' }
type: NounType.Document,
where: { status: 'active' }
})) {
// Process each entity
// Process each entity
}
// Stream relationships
for await (const relation of brain.streaming.relations({
from: 'entity-id'
from: 'entity-id'
})) {
console.log(relation)
console.log(relation)
}
```
@ -438,9 +438,9 @@ for await (const relation of brain.streaming.relations({
```typescript
// Paginated queries
const page1 = await brain.pagination.find({
query: 'machine learning',
page: 1,
pageSize: 20
query: 'machine learning',
page: 1,
pageSize: 20
})
console.log(`Page ${page1.page} of ${page1.totalPages}`)
@ -448,9 +448,9 @@ console.log(`Total results: ${page1.total}`)
// Get next page
const page2 = await brain.pagination.find({
query: 'machine learning',
page: 2,
pageSize: 20
query: 'machine learning',
page: 2,
pageSize: 20
})
```
@ -517,20 +517,20 @@ VerbType.ModifiedBy
```typescript
try {
await brain.get('nonexistent-id')
await brain.get('nonexistent-id')
} catch (error) {
if (error.message.includes('not found')) {
// Handle missing entity
}
if (error.message.includes('not found')) {
// Handle missing entity
}
}
// VFS errors use POSIX codes
try {
await vfs.readFile('/nonexistent')
await vfs.readFile('/nonexistent')
} catch (error) {
if (error.code === 'ENOENT') {
// File not found
}
if (error.code === 'ENOENT') {
// File not found
}
}
```
@ -538,26 +538,26 @@ try {
```typescript
const brain = new Brainy({
// Storage
storage: {
type: 'memory' | 'filesystem',
path: './data', // For filesystem
forceMemoryStorage: false // Force memory even if path exists
},
// Storage
storage: {
type: 'memory' | 'filesystem',
path: './data', // For filesystem
forceMemoryStorage: false // Force memory even if path exists
},
// Embeddings
embeddingModel: 'Q8', // Q4, Q8, F16, F32
dimensions: 384, // Auto-detected
// Embeddings
embeddingModel: 'Q8', // Q4, Q8, F16, F32
dimensions: 384, // Auto-detected
// Performance
silent: false, // Suppress console output
verbose: false, // Extra logging
// Performance
silent: false, // Suppress console output
verbose: false, // Extra logging
// Augmentations (auto-enabled by default)
augmentations: {
cache: true,
display: true,
metrics: true
}
// Augmentations (auto-enabled by default)
augmentations: {
cache: true,
display: true,
metrics: true
}
})
```

32
docs/architecture/data-storage-architecture.md
View file

@ -1,4 +1,4 @@
# Brainy Data Storage Architecture (v5.11.0)
# Brainy Data Storage Architecture
**Complete file structure reference for all storage backends**
@ -13,7 +13,7 @@ This document explains how Brainy stores, indexes, and scales data across all st
3. [The 4 Indexes](#3-the-4-indexes)
4. [Sharding Strategy](#4-sharding-strategy)
5. [COW (Copy-on-Write) Architecture](#5-cow-copy-on-write-architecture)
6. [ID-First Storage Architecture (v6.0.0+)](#6-id-first-storage-architecture-v600)
6. [ID-First Storage Architecture](#6-id-first-storage-architecture-v600)
7. [VFS (Virtual File System)](#7-vfs-virtual-file-system)
8. [Storage Backend Mapping](#8-storage-backend-mapping)
9. [Performance Characteristics](#9-performance-characteristics)
@ -438,7 +438,7 @@ Unlike entities and relationships, system metadata consists of **index files** t
Understanding how Brainy constructs storage paths is critical for debugging and optimization.
### Path Construction Steps (v6.0.0+)
### Path Construction Steps
**For an entity (noun)**:
```typescript
@ -498,7 +498,7 @@ const fieldIndexPath = `_system/metadata_indexes/__metadata_field_index__${field
const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
```
### Path Patterns Summary (v6.0.0+)
### Path Patterns Summary
| Data Type | Path Pattern | Sharded? | Branched? |
|-----------|--------------|----------|-----------|
@ -516,7 +516,7 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
| **HNSW node** | `_system/hnsw/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No |
| **Field index** | `_system/metadata_indexes/__metadata_field_index__{field}.json` | ❌ No | ❌ No |
### Key Principles (v6.0.0+)
### Key Principles
1. **Shard Extraction**: Always use first 2 hex characters of UUID/SHA-256
2. **ID-First**: Shard + ID come BEFORE type (type is in metadata)
@ -549,7 +549,7 @@ Brainy uses four complementary index systems for different query patterns.
- Memory (standard): ~200MB per 100K entities
- Memory (lazy): ~15-33MB per 100K entities (5-10x less!)
**Automatic Lazy Mode** (v3.36.0+): Enables automatically when vectors don't fit in UnifiedCache
**Automatic Lazy Mode**: Enables automatically when vectors don't fit in UnifiedCache
---
@ -559,7 +559,7 @@ Brainy uses four complementary index systems for different query patterns.
**Location**: MetadataIndexManager index on `noun` field
**Data Structure**: RoaringBitmap32 per type value
**How It Works** (v6.0.0+):
**How It Works**:
```typescript
// Find all Person entities
const people = await brain.getNouns({ type: 'person' })
@ -721,7 +721,7 @@ COW is Brainy's **git-like versioning system** that enables:
- ✅ **Deduplication** (identical data stored only once)
- ✅ **Version history** (full audit trail of all changes)
**Status**: ALWAYS ENABLED (v5.11.0+) - cannot be disabled
**Status**: ALWAYS ENABLED - cannot be disabled
---
@ -782,19 +782,19 @@ _cow/blobs/ab/abc123...sha256.bin (used by both entities)
---
## 6. ID-First Storage Architecture (v6.0.0+)
## 6. ID-First Storage Architecture
### 6.1 What is ID-First?
**ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused 20-21 second delays on cloud storage.
**Old type-first structure** (v5.4.0-v5.12.0):
**Old type-first structure** (v5.12.0):
```
branches/main/entities/nouns/{TYPE}/metadata/00/001234...uuid.json
# Problem: Requires knowing type OR searching 42 type directories!
```
**NEW ID-first structure** (v6.0.0+):
**NEW ID-first structure**:
```
branches/main/entities/nouns/00/001234...uuid/metadata.json
# Direct O(1) lookup - no type needed!
@ -809,7 +809,7 @@ branches/main/entities/nouns/00/001234...uuid/metadata.json
// v5.x: Had to search 42 types if type unknown
// Result: 21 seconds on GCS (42 types × 500ms)
// v6.0.0: Direct path from ID
// Direct path from ID
const id = '001234...'
const shard = id.substring(0, 2) // '00'
const path = `branches/main/entities/nouns/${shard}/${id}/metadata.json`
@ -1165,7 +1165,7 @@ opfs://root/brainy/
### 10.2 clear() Operation
**What clear() deletes** (v5.11.0+):
**What clear() deletes**:
✅ Deletes:
- `branches/` → ALL entity data (all types, all shards, all branches, all forks)
@ -1184,7 +1184,7 @@ opfs://root/brainy/
**Example**:
```typescript
await brain.storage.clear() // ✅ Deletes ALL data correctly (v5.11.0+)
await brain.storage.clear() // ✅ Deletes ALL data correctly
await brain.add({ data: 'Alice', type: 'person' }) // ✅ COW reinitializes automatically
```
@ -1353,7 +1353,7 @@ const historicalData = await yesterday.getNouns({ type: 'Character' })
await brain.storage.clear()
```
**What happens in storage** (v5.11.0+):
**What happens in storage**:
```
1. Delete all entity data:
→ Remove: branches/ (entire directory)
@ -1452,7 +1452,7 @@ await brain.addBatch([
## 11. Summary
**Complete Storage Structure (v6.0.0)**:
**Complete Storage Structure**:
- **3 storage layers**: branches/ (data), _cow/ (versions), _system/ (indexes)
- **2 files per entity**: metadata.json + vector.json (optimized I/O)
- **4 indexes**: HNSW (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields)

32
docs/architecture/index-architecture.md
View file

@ -25,7 +25,7 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
- **FieldTypeInference** - DuckDB-inspired value-based field type detection
- **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count)
- **Sorted Indexes** - Support orderBy queries (automatically maintained)
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes (v7.7.0)
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes
**GraphAdjacencyIndex contains:**
- **lsmTreeSource** - Source → Targets (outgoing edges)
@ -39,7 +39,7 @@ All indexes share a **UnifiedCache** for coordinated memory management, ensuring
**Purpose**: Enable O(1) field-value lookups and O(log n) range queries on metadata fields using adaptive chunked sparse indexing.
### Internal Architecture (v3.42.0)
### Internal Architecture
```typescript
class MetadataIndexManager {
@ -64,7 +64,7 @@ class MetadataIndexManager {
### Key Data Structures
#### Chunked Sparse Index (NEW in v3.42.0)
#### Chunked Sparse Index
```typescript
// SparseIndex: Directory of chunks for a field
// Example: field="status"
@ -87,7 +87,7 @@ interface ChunkDescriptor {
class ChunkData {
chunkId: number
field: string
entries: Map<value, RoaringBitmap32> // ~50 values per chunk (v3.43.0: roaring bitmaps!)
entries: Map<value, RoaringBitmap32> // ~50 values per chunk (roaring bitmaps!)
}
```
@ -96,7 +96,7 @@ class ChunkData {
- O(log n) range queries with zone maps
- 630x file reduction (560k flat files → 89 chunk files)
#### Roaring Bitmap Optimization (NEW in v3.43.0)
#### Roaring Bitmap Optimization
**Problem Solved**: JavaScript `Set<string>` for storing entity IDs was inefficient:
- Memory overhead: ~40 bytes per UUID string (36 chars + overhead)
@ -150,7 +150,7 @@ class ChunkData {
**Multi-Field Intersection (THE BIG WIN!)**:
```typescript
// Before (v3.42.0): JavaScript array filtering
// Before: JavaScript array filtering
async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string[]> {
// 1. Fetch UUID arrays for each field
const statusIds = await this.getIds('status', 'active') // ["uuid1", "uuid2", ...]
@ -160,7 +160,7 @@ async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string
return statusIds.filter(id => roleIds.includes(id)) // O(n*m) array filtering
}
// After (v3.43.0): Roaring bitmap intersection
// After: Roaring bitmap intersection
async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
// 1. Fetch roaring bitmaps (integers, not UUIDs)
const bitmaps: RoaringBitmap32[] = []
@ -229,7 +229,7 @@ interface ZoneMap {
**Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field
#### Word Index (`__words__`) - v7.7.0
#### Word Index (`__words__`) -
```typescript
// Special field for text/keyword search
// Entity text content is tokenized and indexed as word hashes
@ -253,14 +253,14 @@ interface ZoneMap {
- **Lowercase normalization**: Case-insensitive matching
- **Automatic integration**: Words extracted via `extractIndexableFields()`
**Hybrid Search** (v7.7.0): Text results combined with vector results using Reciprocal Rank Fusion (RRF):
**Hybrid Search**: Text results combined with vector results using Reciprocal Rank Fusion (RRF):
```typescript
// RRF formula: score(d) = sum(1 / (k + rank(d)))
// where k = 60 (standard constant)
// alpha = weight for semantic (0 = text only, 1 = semantic only)
```
### Query Algorithm (v3.42.0)
### Query Algorithm
**Exact Match Query**:
```typescript
@ -317,7 +317,7 @@ async getIdsForRange(field: string, min: any, max: any): Promise<string[]> {
- Adaptive chunking: ~50 values per chunk optimizes I/O
- Immediate flushing: No need for dirty tracking or batch writes
### Temporal Bucketing (v3.41.0)
### Temporal Bucketing
**Problem Solved**: High-cardinality timestamp fields created massive file pollution.
- Example: 575 entities with unique timestamps → 358,407 index files (98.7% pollution!)
@ -396,7 +396,7 @@ const DEFAULT_EXCLUDE_FIELDS = [
]
```
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of v3.41.0 - they are indexed with automatic bucketing.
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
## 2. HNSWIndex - Vector Similarity Search
@ -735,7 +735,7 @@ async stats(): Promise<Statistics> {
}
```
### 5. Index Rebuilding (v5.7.7: Lazy Loading Support)
### 5. Index Rebuilding (Lazy Loading Support)
**Two modes of index loading:**
@ -762,7 +762,7 @@ async init(): Promise<void> {
}
```
#### Mode 2: Lazy Loading on First Query (v5.7.7+)
#### Mode 2: Lazy Loading on First Query
```typescript
// When disableAutoRebuild: true
@ -918,7 +918,7 @@ All indexes scale gracefully:
- [Performance Guide](../PERFORMANCE.md) - Performance tuning
- [Overview](./overview.md) - High-level architecture
## Summary: Index Hierarchy (v5.7.7)
## Summary: Index Hierarchy
### Level 1: Main Indexes (3)
All have rebuild() methods and are covered by lazy loading:
@ -933,7 +933,7 @@ Automatically managed by parent rebuild():
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
### Lazy Loading (v5.7.7)
### Lazy Loading
- **Mode 1**: Auto-rebuild on init() (default)
- **Mode 2**: Lazy rebuild on first query (when `disableAutoRebuild: true`)
- **Concurrency-safe**: Mutex prevents duplicate rebuilds

8
docs/architecture/initialization-and-rebuild.md
View file

@ -15,7 +15,7 @@ This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, Grap
| **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
| **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
#### MetadataIndex Persistence Details (v4.2.1+)
#### MetadataIndex Persistence Details
The MetadataIndex now persists two components:
@ -129,7 +129,7 @@ async init(): Promise<void> {
- 100-2000ms: One-time rebuild to create indices
- Total: ~1-3 seconds (one time only)
#### Mode 2: Lazy Loading on First Query (v5.7.7+)
#### Mode 2: Lazy Loading on First Query
When `disableAutoRebuild: true`, indexes remain empty after init() and rebuild on first query:
@ -350,7 +350,7 @@ public async rebuild(options?: {
**Performance Impact**: 200-600x speedup (5 minutes → 500ms for 10K entities)
**Correct Pattern** (v3.45.0):
**Correct Pattern**:
```typescript
// Load ALL nouns ONCE (not 31 times!)
while (hasMore) {
@ -392,7 +392,7 @@ while (hasMore) {
```typescript
// src/utils/metadataIndex.ts (lines 202-216)
async init(): Promise<void> {
// STEP 1: Load field registry to discover persisted indices (v4.2.1)
// STEP 1: Load field registry to discover persisted indices
// This is THE KEY FIX - O(1) discovery of existing indices
await this.loadFieldRegistry()

382
docs/architecture/storage-architecture.md
View file

@ -1,40 +1,40 @@
# Storage Architecture (v4.0.0)
# Storage Architecture
> **Updated for v4.0.0**: Metadata/vector separation, UUID-based sharding, lifecycle management
> **Updated**: Metadata/vector separation, UUID-based sharding, lifecycle management
## Storage Structure
### v4.0.0 Architecture: Metadata/Vector Separation
### Architecture: Metadata/Vector Separation
In v4.0.0, entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
```
brainy-data/
├── _system/ # System metadata (not sharded)
│ ├── statistics.json # Performance metrics
│ ├── __metadata_field_index__*.json # Field indexes
│ └── __metadata_sorted_index__*.json # Sorted indexes
├── _system/ # System metadata (not sharded)
│ ├── statistics.json # Performance metrics
│ ├── __metadata_field_index__*.json # Field indexes
│ └── __metadata_sorted_index__*.json # Sorted indexes
│
├── entities/
│ ├── nouns/
│ │ ├── vectors/ # HNSW graph data (sharded by UUID)
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
│ │ │ │ ├── 00123456-....json # Vector + HNSW connections
│ │ │ │ └── 00abcdef-....json
│ │ │ ├── 01/ ... ff/ # 256 shards total
│ │ │
│ │ └── metadata/ # Business data (sharded by UUID)
│ │ ├── 00/
│ │ │ ├── 00123456-....json # Entity metadata only
│ │ │ └── 00abcdef-....json
│ │ ├── 01/ ... ff/
│ │
│ └── verbs/
│ ├── vectors/ # Relationship vectors (sharded)
│ │ ├── 00/ ... ff/
│ │
│ └── metadata/ # Relationship data (sharded)
│ ├── 00/ ... ff/
│ ├── nouns/
│ │ ├── vectors/ # HNSW graph data (sharded by UUID)
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
│ │ │ │ ├── 00123456-....json # Vector + HNSW connections
│ │ │ │ └── 00abcdef-....json
│ │ │ ├── 01/ ... ff/ # 256 shards total
│ │ │
│ │ └── metadata/ # Business data (sharded by UUID)
│ │ ├── 00/
│ │ │ ├── 00123456-....json # Entity metadata only
│ │ │ └── 00abcdef-....json
│ │ ├── 01/ ... ff/
│ │
│ └── verbs/
│ ├── vectors/ # Relationship vectors (sharded)
│ │ ├── 00/ ... ff/
│ │
│ └── metadata/ # Relationship data (sharded)
│ ├── 00/ ... ff/
```
### Why Split Metadata and Vectors?
@ -50,9 +50,9 @@ brainy-data/
**How it works:**
```typescript
const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
const shard = uuid.substring(0, 2) // "3f"
const shard = uuid.substring(0, 2) // "3f"
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
```
@ -64,103 +64,103 @@ const shard = uuid.substring(0, 2) // "3f"
## Storage Adapters
Brainy provides multiple storage adapters with identical APIs and v4.0.0 production features:
Brainy provides multiple storage adapters with identical APIs and production features:
### FileSystem Storage (Node.js)
```typescript
const brain = new Brainy({
storage: {
type: 'filesystem',
path: './data',
compression: true // v4.0.0: Gzip compression (60-80% space savings)
}
storage: {
type: 'filesystem',
path: './data',
compression: true // Gzip compression (60-80% space savings)
}
})
```
- **Use case**: Server applications, CLI tools
- **Performance**: Direct file I/O with optional compression
- **Persistence**: Permanent on disk
- **v4.0.0 Features**:
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
- **Batch Delete**: Efficient bulk deletion with retries
- **UUID Sharding**: Automatic 256-shard distribution
- **Features**:
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
- **Batch Delete**: Efficient bulk deletion with retries
- **UUID Sharding**: Automatic 256-shard distribution
### S3 Compatible Storage (AWS, MinIO, R2)
```typescript
const brain = new Brainy({
storage: {
type: 's3',
bucket: 'my-brainy-data',
region: 'us-east-1',
credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}
}
storage: {
type: 's3',
bucket: 'my-brainy-data',
region: 'us-east-1',
credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}
}
})
```
- **Use case**: Distributed applications, cloud deployments
- **Performance**: Network dependent, with intelligent caching
- **Persistence**: Cloud storage durability (99.999999999%)
- **v4.0.0 Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
- **Batch Delete**: Efficient bulk deletion (1000 objects per request)
- **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!)
- **Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
- **Batch Delete**: Efficient bulk deletion (1000 objects per request)
- **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!)
### Google Cloud Storage (GCS)
```typescript
const brain = new Brainy({
storage: {
type: 'gcs',
bucketName: 'my-brainy-data',
keyFilename: './service-account.json' // Or use ADC
}
storage: {
type: 'gcs',
bucketName: 'my-brainy-data',
keyFilename: './service-account.json' // Or use ADC
}
})
```
- **Use case**: Google Cloud deployments
- **Performance**: Global CDN with edge caching
- **Persistence**: 99.999999999% durability
- **v4.0.0 Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
- **Autoclass**: Intelligent automatic tier optimization
- **Batch Delete**: Efficient bulk operations
- **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!)
- **Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
- **Autoclass**: Intelligent automatic tier optimization
- **Batch Delete**: Efficient bulk operations
- **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!)
### Azure Blob Storage
```typescript
const brain = new Brainy({
storage: {
type: 'azure',
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
}
storage: {
type: 'azure',
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
}
})
```
- **Use case**: Azure cloud deployments
- **Performance**: Global replication with CDN
- **Persistence**: LRS, ZRS, GRS, RA-GRS options
- **v4.0.0 Features**:
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
- **Lifecycle Policies**: Automatic tier transitions and deletions
- **Batch Delete**: BlobBatchClient for efficient bulk operations
- **Batch Tier Changes**: Move thousands of blobs efficiently
- **Archive Rehydration**: Smart rehydration with priority options
- **Features**:
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
- **Lifecycle Policies**: Automatic tier transitions and deletions
- **Batch Delete**: BlobBatchClient for efficient bulk operations
- **Batch Tier Changes**: Move thousands of blobs efficiently
- **Archive Rehydration**: Smart rehydration with priority options
### Origin Private File System (Browser)
```typescript
const brain = new Brainy({
storage: {
type: 'opfs'
}
storage: {
type: 'opfs'
}
})
```
- **Use case**: Browser applications, PWAs
- **Performance**: Near-native file system speed
- **Persistence**: Permanent in browser (with quota limits)
- **v4.0.0 Features**:
- **Quota Monitoring**: Real-time quota tracking and warnings
- **Batch Delete**: Efficient bulk deletion
- **Storage Status**: Detailed usage/available reporting
- **Features**:
- **Quota Monitoring**: Real-time quota tracking and warnings
- **Batch Delete**: Efficient bulk deletion
- **Storage Status**: Detailed usage/available reporting
## Metadata Indexing System
@ -170,12 +170,12 @@ Tracks all unique values for each field:
```json
// __metadata_field_index__field_category.json
{
"values": {
"technology": 45,
"science": 32,
"business": 28
},
"lastUpdated": 1699564234567
"values": {
"technology": 45,
"science": 32,
"business": 28
},
"lastUpdated": 1699564234567
}
```
@ -185,11 +185,11 @@ Maps field+value combinations to entity IDs:
```json
// __metadata_index__category_technology_chunk0.json
{
"field": "category",
"value": "technology",
"ids": ["uuid1", "uuid2", "uuid3", ...],
"chunk": 0,
"total": 45
"field": "category",
"value": "technology",
"ids": ["uuid1", "uuid2", "uuid3", ...],
"chunk": 0,
"total": 45
}
```
@ -207,14 +207,14 @@ High-performance deduplication system for streaming data:
```json
// __entity_registry__.json
{
"mappings": {
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
},
"stats": {
"totalMappings": 10000,
"lastSync": 1699564234567
}
"mappings": {
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
},
"stats": {
"totalMappings": 10000,
"lastSync": 1699564234567
}
}
```
@ -229,14 +229,14 @@ Ensures durability and enables recovery:
```json
{
"timestamp": 1699564234567,
"operation": "add",
"data": {
"id": "550e8400-e29b-41d4-a716-446655440000",
"content": "...",
"metadata": {}
},
"checksum": "sha256:..."
"timestamp": 1699564234567,
"operation": "add",
"data": {
"id": "550e8400-e29b-41d4-a716-446655440000",
"content": "...",
"metadata": {}
},
"checksum": "sha256:..."
}
```
@ -244,7 +244,7 @@ Ensures durability and enables recovery:
2. Replay operations from last checkpoint
3. Verify checksums for integrity
## Storage Optimization (v4.0.0)
## Storage Optimization
### 1. Lifecycle Policies (Cloud Storage)
@ -253,48 +253,48 @@ Ensures durability and enables recovery:
```typescript
// S3: Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days
{ days: 90, storageClass: 'GLACIER' }, // Archive after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year
]
}]
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days
{ days: 90, storageClass: 'GLACIER' }, // Archive after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year
]
}]
})
// GCS: Set lifecycle policy
await storage.setLifecyclePolicy({
rules: [{
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 365 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
rules: [{
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 365 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
})
// Azure: Set lifecycle policy
await storage.setLifecyclePolicy({
rules: [{
name: 'archiveOldData',
enabled: true,
type: 'Lifecycle',
definition: {
filters: { blobTypes: ['blockBlob'] },
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}]
rules: [{
name: 'archiveOldData',
enabled: true,
type: 'Lifecycle',
definition: {
filters: { blobTypes: ['blockBlob'] },
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}]
})
```
@ -327,7 +327,7 @@ await storage.enableIntelligentTiering('entities/', 'auto-optimize')
```typescript
// Enable GCS Autoclass
await storage.enableAutoclass({
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
})
// Benefits:
@ -342,11 +342,11 @@ await storage.enableAutoclass({
```typescript
// Enable gzip compression for local storage
const brain = new Brainy({
storage: {
type: 'filesystem',
path: './data',
compression: true // 60-80% space savings
}
storage: {
type: 'filesystem',
path: './data',
compression: true // 60-80% space savings
}
})
// Performance impact:
@ -359,11 +359,11 @@ const brain = new Brainy({
### 5. Batch Operations
```typescript
// v4.0.0: Efficient batch delete
// Efficient batch delete
await storage.batchDelete([
'entities/nouns/vectors/00/00123456-....json',
'entities/nouns/metadata/00/00123456-....json',
// ... up to 1000 objects
'entities/nouns/vectors/00/00123456-....json',
'entities/nouns/metadata/00/00123456-....json',
// ... up to 1000 objects
])
// Benefits:
@ -375,9 +375,9 @@ await storage.batchDelete([
// Batch writes for performance
await brain.addBatch([
{ content: "item1", metadata: {} },
{ content: "item2", metadata: {} },
{ content: "item3", metadata: {} }
{ content: "item1", metadata: {} },
{ content: "item2", metadata: {} },
{ content: "item3", metadata: {} }
])
// Single transaction, optimized I/O
```
@ -390,14 +390,14 @@ const status = await storage.getStorageStatus()
console.log(status)
// {
// type: 'opfs',
// available: true,
// details: {
// usage: 45829120, // 43.7 MB used
// quota: 536870912, // 512 MB available
// usagePercent: 8.5,
// quotaExceeded: false
// }
// type: 'opfs',
// available: true,
// details: {
// usage: 45829120, // 43.7 MB used
// quota: 536870912, // 512 MB available
// usagePercent: 8.5,
// quotaExceeded: false
// }
// }
// Proactive quota management:
@ -410,14 +410,14 @@ console.log(status)
```typescript
// Change blob tier for cost optimization
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings)
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings)
// Batch tier changes (efficient)
await storage.batchChangeTier([blob1, blob2, blob3], 'Cool')
// Rehydrate from Archive when needed
await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
```
### 8. Caching Strategy
@ -425,15 +425,15 @@ await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
```typescript
// Configure caching per storage type
const brain = new Brainy({
storage: {
type: 'filesystem',
cache: {
enabled: true,
maxSize: 1000, // Maximum cached items
ttl: 300000, // 5 minutes
strategy: 'lru' // Least recently used
}
}
storage: {
type: 'filesystem',
cache: {
enabled: true,
maxSize: 1000, // Maximum cached items
ttl: 300000, // 5 minutes
strategy: 'lru' // Least recently used
}
}
})
```
@ -443,8 +443,8 @@ const brain = new Brainy({
```typescript
// Automatic locking for write operations
await brain.storage.withLock('resource-id', async () => {
// Exclusive access to resource
await brain.storage.saveNoun(id, data)
// Exclusive access to resource
await brain.storage.saveNoun(id, data)
})
```
@ -459,9 +459,9 @@ await brain.storage.withLock('resource-id', async () => {
```typescript
// Export entire database
const backup = await brain.export({
format: 'json',
includeVectors: true,
includeIndexes: false
format: 'json',
includeVectors: true,
includeIndexes: false
})
```
@ -469,8 +469,8 @@ const backup = await brain.export({
```typescript
// Import from backup
await brain.import(backup, {
mode: 'merge', // or 'replace'
validateSchema: true
mode: 'merge', // or 'replace'
validateSchema: true
})
```
@ -514,15 +514,15 @@ await newBrain.import(data)
const stats = await brain.storage.getStatistics()
console.log(stats)
// {
// totalSize: 1048576,
// entityCount: 1000,
// indexSize: 204800,
// walSize: 10240,
// cacheHitRate: 0.85
// totalSize: 1048576,
// entityCount: 1000,
// indexSize: 204800,
// walSize: 10240,
// cacheHitRate: 0.85
// }
```
## Best Practices (v4.0.0)
## Best Practices
### Choose the Right Adapter
1. **Development**: FileSystem with compression (local persistence, small storage footprint)
@ -537,7 +537,7 @@ console.log(stats)
4. **Archival**: Cloud storage with lifecycle policies (96% cost savings!)
5. **Large-scale**: Metadata/vector separation + UUID sharding + lifecycle policies
### v4.0.0 Cost Optimization
### Cost Optimization
1. **Enable lifecycle policies** for cloud storage (automated cost reduction)
2. **Use Intelligent-Tiering (S3)** or Autoclass (GCS) for automatic optimization
3. **Enable compression** for FileSystem storage (60-80% space savings)
@ -547,7 +547,7 @@ console.log(stats)
**Example Cost Savings (500TB dataset):**
- Without lifecycle policies: **$138,000/year**
- With v4.0.0 lifecycle policies: **$5,940/year**
- With lifecycle policies: **$5,940/year**
- **Savings: $132,060/year (96%)**
### Monitor and Maintain

138
docs/augmentations/COMPLETE-REFERENCE.md
View file

@ -1,8 +1,8 @@
# 🔌 Brainy v4.0.0 Augmentations Complete Reference
# 🔌 Brainy Augmentations Complete Reference
> **All augmentations that power Brainy's extensibility - with locations, usage, and examples**
>
> **⚠️ v4.0.0 Update**: Updated for metadata structure changes and billion-scale optimizations
> **⚠️ Update**: Updated for metadata structure changes and billion-scale optimizations
## Quick Start
@ -10,37 +10,37 @@
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy({
// Augmentations auto-configure based on environment
storage: 'auto', // Storage augmentation
cache: true, // Cache augmentation
index: true // Index augmentation
// Augmentations auto-configure based on environment
storage: 'auto', // Storage augmentation
cache: true, // Cache augmentation
index: true // Index augmentation
})
await brain.init() // Augmentations initialize automatically
await brain.init() // Augmentations initialize automatically
```
## v4.0.0 Augmentation Architecture
## Augmentation Architecture
### Key Improvements for Billion-Scale Performance
1. **Metadata/Vector Separation**: Augmentations now work with separated metadata and vectors
- Metadata stored separately from vector data
- 99.2% memory reduction for type tracking
- Two-file storage pattern for optimal I/O
- Metadata stored separately from vector data
- 99.2% memory reduction for type tracking
- Two-file storage pattern for optimal I/O
2. **Type System Enforcement**: All metadata requires type fields
- `NounMetadata` requires `noun: NounType`
- `VerbMetadata` requires `verb: VerbType`
- Type inference system available as public API
- `NounMetadata` requires `noun: NounType`
- `VerbMetadata` requires `verb: VerbType`
- Type inference system available as public API
3. **Storage Adapter Pattern**: Internal vs public method distinction
- `_methods`: Return pure structures (HNSWNoun, HNSWVerb)
- Public methods: Return WithMetadata types
- MetadataEnforcer Proxy ensures proper access
- `_methods`: Return pure structures (HNSWNoun, HNSWVerb)
- Public methods: Return WithMetadata types
- MetadataEnforcer Proxy ensures proper access
### What This Means for Augmentation Users
**✅ If you use built-in augmentations**: No changes needed! They're all updated for v4.0.0.
**✅ If you use built-in augmentations**: No changes needed! They're all updated.
**⚠️ If you created custom storage augmentations**: Update your storage adapter to:
- Wrap metadata with required `noun`/`verb` fields
@ -82,7 +82,7 @@ const brain = new Brainy({ storage: 'memory' })
**Purpose**: Persistent file-based storage for Node.js applications
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', path: './data' }
storage: { type: 'filesystem', path: './data' }
})
```
@ -100,12 +100,12 @@ const brain = new Brainy({ storage: 'opfs' })
**Purpose**: AWS S3-compatible cloud storage
```typescript
const brain = new Brainy({
storage: {
type: 's3',
bucket: 'my-bucket',
region: 'us-east-1',
credentials: { accessKeyId, secretAccessKey }
}
storage: {
type: 's3',
bucket: 'my-bucket',
region: 'us-east-1',
credentials: { accessKeyId, secretAccessKey }
}
})
```
@ -115,12 +115,12 @@ const brain = new Brainy({
**Purpose**: Cloudflare R2 storage (S3-compatible)
```typescript
const brain = new Brainy({
storage: {
type: 'r2',
accountId: 'xxx',
bucket: 'my-bucket',
credentials: { accessKeyId, secretAccessKey }
}
storage: {
type: 'r2',
accountId: 'xxx',
bucket: 'my-bucket',
credentials: { accessKeyId, secretAccessKey }
}
})
```
@ -130,11 +130,11 @@ const brain = new Brainy({
**Purpose**: Google Cloud Storage
```typescript
const brain = new Brainy({
storage: {
type: 'gcs',
bucket: 'my-bucket',
projectId: 'my-project'
}
storage: {
type: 'gcs',
bucket: 'my-bucket',
projectId: 'my-project'
}
})
```
@ -155,8 +155,8 @@ const brain = new Brainy({
**Auto-enabled**: When `cache: true` (default)
**Purpose**: LRU cache for search results and frequent queries
```typescript
brain.clearCache() // Exposed via API
brain.getCacheStats() // Cache hit/miss statistics
brain.clearCache() // Exposed via API
brain.getCacheStats() // Cache hit/miss statistics
```
### IndexAugmentation
@ -174,7 +174,7 @@ brain.find({ where: { category: 'tech' } })
**Auto-enabled**: Always active
**Purpose**: Performance metrics and statistics collection
```typescript
brain.getStats() // Comprehensive metrics
brain.getStats() // Comprehensive metrics
```
### MonitoringAugmentation
@ -187,7 +187,7 @@ brain.getStats() // Comprehensive metrics
**Auto-enabled**: For batch operations
**Purpose**: Optimizes bulk add/update/delete operations
```typescript
brain.addNouns([...]) // Automatically batched
brain.addNouns([...]) // Automatically batched
```
### RequestDeduplicatorAugmentation
@ -235,8 +235,8 @@ brain.add(data) // Automatically deduplicated
**Purpose**: AI-powered smart data import
```typescript
const result = await brain.neuralImport(data, {
confidenceThreshold: 0.7,
autoApply: true
confidenceThreshold: 0.7,
autoApply: true
})
// Automatically detects entities and relationships
```
@ -293,8 +293,8 @@ await conduit.establishConnection('ws://other-brain')
```typescript
// Example: NotionSynapse, SlackSynapse, etc.
class NotionSynapse extends SynapseAugmentation {
async fetchData() { /* Notion API calls */ }
async pushData() { /* Sync to Notion */ }
async fetchData() { /* Notion API calls */ }
async pushData() { /* Sync to Notion */ }
}
```
@ -309,11 +309,11 @@ class NotionSynapse extends SynapseAugmentation {
### Auto-Configuration
```typescript
const brain = new Brainy({
// These auto-register augmentations:
storage: 'auto', // Storage augmentation
cache: true, // Cache augmentation
index: true, // Index augmentation
metrics: true // Metrics augmentation
// These auto-register augmentations:
storage: 'auto', // Storage augmentation
cache: true, // Cache augmentation
index: true, // Index augmentation
metrics: true // Metrics augmentation
})
```
@ -333,29 +333,29 @@ await brain.init()
import { BaseAugmentation } from '@soulcraft/brainy'
class MyAugmentation extends BaseAugmentation {
readonly name = 'my-augmentation'
readonly timing = 'after' // before | after | both
readonly operations = ['addNoun', 'search'] // Which ops to hook
readonly priority = 10 // Execution order (lower = earlier)
readonly name = 'my-augmentation'
readonly timing = 'after' // before | after | both
readonly operations = ['addNoun', 'search'] // Which ops to hook
readonly priority = 10 // Execution order (lower = earlier)
protected async onInit(): Promise<void> {
// Initialize your augmentation
}
protected async onInit(): Promise<void> {
// Initialize your augmentation
}
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'addNoun') {
console.log('Noun added:', params)
}
}
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'addNoun') {
console.log('Noun added:', params)
}
}
protected async onShutdown(): Promise<void> {
// Cleanup
}
protected async onShutdown(): Promise<void> {
// Cleanup
}
}
```

476
docs/augmentations/DEVELOPER-GUIDE.md
View file

@ -1,10 +1,10 @@
# 🛠️ Brainy Augmentation Developer Guide
> **How to create, test, and use augmentations in Brainy v4.0.0**
> **How to create, test, and use augmentations in Brainy**
>
> **⚠️ v4.0.0 Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
> **⚠️ Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
## v4.0.0 Migration Guide
## Migration Guide
### What Changed?
@ -25,22 +25,22 @@
```typescript
// ❌ v3.x
const verb = {
type: 'relatedTo',
sourceId: 'a',
targetId: 'b'
type: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
if (verb.type === 'relatedTo') { ... }
// ✅ v4.0.0
// ✅ Current
const verb = {
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
const metadata: VerbMetadata = {
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
verb: 'relatedTo',
sourceId: 'a',
targetId: 'b'
}
if (verb.verb === 'relatedTo') { ... }
```
@ -51,40 +51,40 @@ if (verb.verb === 'relatedTo') { ... }
import { BaseAugmentation, BrainyAugmentation, AugmentationContext } from '@soulcraft/brainy'
export class MyFirstAugmentation extends BaseAugmentation {
readonly name = 'my-first-augmentation'
readonly timing = 'after' as const // When to run: before | after | both
readonly operations = ['add'] as const // Which operations to hook
readonly priority = 10 // Execution order (lower = first)
readonly name = 'my-first-augmentation'
readonly timing = 'after' as const // When to run: before | after | both
readonly operations = ['add'] as const // Which operations to hook
readonly priority = 10 // Execution order (lower = first)
protected async onInit(): Promise<void> {
// Initialize your augmentation
console.log('MyFirstAugmentation initialized!')
}
protected async onInit(): Promise<void> {
// Initialize your augmentation
console.log('MyFirstAugmentation initialized!')
}
async execute<T = any>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'add') {
console.log('Noun added:', params.noun)
async execute<T = any>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<T | void> {
// Your augmentation logic
if (operation === 'add') {
console.log('Noun added:', params.noun)
// v4.0.0: Access metadata correctly
if (params.noun?.metadata) {
console.log('Noun type:', params.noun.metadata.noun) // Required field
}
// Access metadata correctly
if (params.noun?.metadata) {
console.log('Noun type:', params.noun.metadata.noun) // Required field
}
// You can access the brain instance
const stats = await context?.brain.getStats()
console.log('Total nouns:', stats.totalNouns)
}
}
// You can access the brain instance
const stats = await context?.brain.getStats()
console.log('Total nouns:', stats.totalNouns)
}
}
protected async onShutdown(): Promise<void> {
// Cleanup
console.log('MyFirstAugmentation shutting down')
}
protected async onShutdown(): Promise<void> {
// Cleanup
console.log('MyFirstAugmentation shutting down')
}
}
```
@ -113,23 +113,23 @@ await brain.add('Hello World')
### 1. Registration Phase
```typescript
const aug = new MyAugmentation()
brain.augmentations.register(aug) // Before brain.init()!
brain.augmentations.register(aug) // Before brain.init()!
```
### 2. Initialization Phase
```typescript
await brain.init() // Calls aug.initialize() internally
await brain.init() // Calls aug.initialize() internally
// Your onInit() method runs here
```
### 3. Execution Phase
```typescript
await brain.add('data') // Your execute() method runs
await brain.add('data') // Your execute() method runs
```
### 4. Shutdown Phase
```typescript
await brain.shutdown() // Your onShutdown() method runs
await brain.shutdown() // Your onShutdown() method runs
```
---
@ -139,52 +139,52 @@ await brain.shutdown() // Your onShutdown() method runs
### `before` - Modify Input
```typescript
class ValidationAugmentation extends BaseAugmentation {
readonly timing = 'before' as const
readonly timing = 'before' as const
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Validate and/or modify params
if (!params.content) {
throw new Error('Content required')
}
// Return modified params
return { ...params, validated: true }
}
}
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Validate and/or modify params
if (!params.content) {
throw new Error('Content required')
}
// Return modified params
return { ...params, validated: true }
}
}
}
```
### `after` - React to Results
```typescript
class LoggingAugmentation extends BaseAugmentation {
readonly timing = 'after' as const
readonly timing = 'after' as const
async execute<T>(operation: string, params: any): Promise<void> {
if (operation === 'search') {
console.log(`Search for "${params.query}" returned ${params.result.length} results`)
}
// Don't return anything - just observe
}
async execute<T>(operation: string, params: any): Promise<void> {
if (operation === 'search') {
console.log(`Search for "${params.query}" returned ${params.result.length} results`)
}
// Don't return anything - just observe
}
}
```
### `both` - Before AND After
```typescript
class TimingAugmentation extends BaseAugmentation {
readonly timing = 'both' as const
private startTime?: number
readonly timing = 'both' as const
private startTime?: number
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
if (!this.startTime) {
// Before execution
this.startTime = Date.now()
} else {
// After execution
const duration = Date.now() - this.startTime
console.log(`${operation} took ${duration}ms`)
this.startTime = undefined
}
}
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
if (!this.startTime) {
// Before execution
this.startTime = Date.now()
} else {
// After execution
const duration = Date.now() - this.startTime
console.log(`${operation} took ${duration}ms`)
this.startTime = undefined
}
}
}
```
@ -195,28 +195,28 @@ class TimingAugmentation extends BaseAugmentation {
### Core Operations You Can Hook
```typescript
readonly operations = [
'add', // Adding data
'update', // Updating data
'delete', // Deleting data
'get', // Retrieving data
'search', // Searching
'find', // Triple Intelligence queries
'relate', // Adding relationships
'unrelate', // Removing relationships
'clear', // Clearing data
'all' // Hook ALL operations
'add', // Adding data
'update', // Updating data
'delete', // Deleting data
'get', // Retrieving data
'search', // Searching
'find', // Triple Intelligence queries
'relate', // Adding relationships
'unrelate', // Removing relationships
'clear', // Clearing data
'all' // Hook ALL operations
] as const
```
### Example: Multi-Operation Hook
```typescript
class AuditAugmentation extends BaseAugmentation {
readonly operations = ['add', 'update', 'delete'] as const
readonly operations = ['add', 'update', 'delete'] as const
async execute<T>(operation: string, params: any): Promise<void> {
// Log all data modifications
await this.logToAuditTrail(operation, params)
}
async execute<T>(operation: string, params: any): Promise<void> {
// Log all data modifications
await this.logToAuditTrail(operation, params)
}
}
```
@ -226,26 +226,26 @@ class AuditAugmentation extends BaseAugmentation {
```typescript
class ContextAwareAugmentation extends BaseAugmentation {
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<void> {
// Access the brain instance
const brain = context?.brain
if (!brain) return
async execute<T>(
operation: string,
params: any,
context?: AugmentationContext
): Promise<void> {
// Access the brain instance
const brain = context?.brain
if (!brain) return
// Use any brain method
const stats = await brain.getStats()
const size = await brain.size()
const results = await brain.search('query')
// Use any brain method
const stats = await brain.getStats()
const size = await brain.size()
const results = await brain.search('query')
// Access other augmentations
const cache = brain.augmentations.get('cache')
if (cache) {
await cache.clear()
}
}
// Access other augmentations
const cache = brain.augmentations.get('cache')
if (cache) {
await cache.clear()
}
}
}
```
@ -256,92 +256,92 @@ class ContextAwareAugmentation extends BaseAugmentation {
### 1. Backup Augmentation
```typescript
class BackupAugmentation extends BaseAugmentation {
readonly name = 'backup'
readonly timing = 'after' as const
readonly operations = ['add', 'update', 'delete'] as const
readonly priority = 5
readonly name = 'backup'
readonly timing = 'after' as const
readonly operations = ['add', 'update', 'delete'] as const
readonly priority = 5
private changes = 0
private readonly backupThreshold = 100
private changes = 0
private readonly backupThreshold = 100
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
this.changes++
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
this.changes++
if (this.changes >= this.backupThreshold) {
await this.performBackup(context?.brain)
this.changes = 0
}
}
if (this.changes >= this.backupThreshold) {
await this.performBackup(context?.brain)
this.changes = 0
}
}
private async performBackup(brain?: any): Promise<void> {
if (!brain) return
// Create instant COW snapshot
const snapshotName = `backup-${Date.now()}`
await brain.fork(snapshotName)
console.log(`Automatic snapshot created: ${snapshotName}`)
}
private async performBackup(brain?: any): Promise<void> {
if (!brain) return
// Create instant COW snapshot
const snapshotName = `backup-${Date.now()}`
await brain.fork(snapshotName)
console.log(`Automatic snapshot created: ${snapshotName}`)
}
}
```
### 2. Rate Limiting Augmentation
```typescript
class RateLimitAugmentation extends BaseAugmentation {
readonly name = 'rate-limit'
readonly timing = 'before' as const
readonly operations = ['search', 'find'] as const
readonly priority = 100 // High priority - run first
readonly name = 'rate-limit'
readonly timing = 'before' as const
readonly operations = ['search', 'find'] as const
readonly priority = 100 // High priority - run first
private requests = new Map<string, number[]>()
private readonly limit = 100 // 100 requests
private readonly window = 60000 // per minute
private requests = new Map<string, number[]>()
private readonly limit = 100 // 100 requests
private readonly window = 60000 // per minute
async execute<T>(operation: string, params: any): Promise<void> {
const now = Date.now()
const key = params.userId || 'anonymous'
async execute<T>(operation: string, params: any): Promise<void> {
const now = Date.now()
const key = params.userId || 'anonymous'
// Get request timestamps
const timestamps = this.requests.get(key) || []
// Get request timestamps
const timestamps = this.requests.get(key) || []
// Remove old timestamps
const recent = timestamps.filter(t => now - t < this.window)
// Remove old timestamps
const recent = timestamps.filter(t => now - t < this.window)
// Check limit
if (recent.length >= this.limit) {
throw new Error('Rate limit exceeded')
}
// Check limit
if (recent.length >= this.limit) {
throw new Error('Rate limit exceeded')
}
// Add current request
recent.push(now)
this.requests.set(key, recent)
}
// Add current request
recent.push(now)
this.requests.set(key, recent)
}
}
```
### 3. Encryption Augmentation
```typescript
class EncryptionAugmentation extends BaseAugmentation {
readonly name = 'encryption'
readonly timing = 'both' as const
readonly operations = ['add', 'get'] as const
readonly priority = 90 // Run early
readonly name = 'encryption'
readonly timing = 'both' as const
readonly operations = ['add', 'get'] as const
readonly priority = 90 // Run early
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Encrypt before storing
if (params.metadata?.sensitive) {
params.content = await this.encrypt(params.content)
params.encrypted = true
}
return params
}
async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') {
// Encrypt before storing
if (params.metadata?.sensitive) {
params.content = await this.encrypt(params.content)
params.encrypted = true
}
return params
}
if (operation === 'get' && params.result?.encrypted) {
// Decrypt after retrieval
params.result.content = await this.decrypt(params.result.content)
delete params.result.encrypted
return params.result
}
}
if (operation === 'get' && params.result?.encrypted) {
// Decrypt after retrieval
params.result.content = await this.decrypt(params.result.content)
delete params.result.encrypted
return params.result
}
}
}
```
@ -355,26 +355,26 @@ import { Brainy } from '@soulcraft/brainy'
import { MyAugmentation } from './my-augmentation'
describe('MyAugmentation', () => {
it('should hook into addNoun', async () => {
const brain = new Brainy({ storage: 'memory' })
const aug = new MyAugmentation()
it('should hook into addNoun', async () => {
const brain = new Brainy({ storage: 'memory' })
const aug = new MyAugmentation()
// Spy on the execute method
const executeSpy = vi.spyOn(aug, 'execute')
// Spy on the execute method
const executeSpy = vi.spyOn(aug, 'execute')
brain.augmentations.register(aug)
await brain.init()
brain.augmentations.register(aug)
await brain.init()
// Trigger the augmentation
await brain.add('test data')
// Trigger the augmentation
await brain.add('test data')
// Verify it was called
expect(executeSpy).toHaveBeenCalledWith(
'add',
expect.objectContaining({ content: 'test data' }),
expect.any(Object)
)
})
// Verify it was called
expect(executeSpy).toHaveBeenCalledWith(
'add',
expect.objectContaining({ content: 'test data' }),
expect.any(Object)
)
})
})
```
@ -391,61 +391,61 @@ describe('MyAugmentation', () => {
```typescript
// Priority guidelines
100: Critical (auth, rate limiting)
50: Important (validation, transformation)
10: Normal (logging, metrics)
1: Optional (debugging, tracing)
50: Important (validation, transformation)
10: Normal (logging, metrics)
1: Optional (debugging, tracing)
```
### 3. Handle Errors Gracefully
```typescript
async execute<T>(operation: string, params: any): Promise<void> {
try {
await this.riskyOperation()
} catch (error) {
// Log but don't break the main operation
console.error(`Augmentation error in ${this.name}:`, error)
// Optionally report to monitoring
this.reportError(error)
}
try {
await this.riskyOperation()
} catch (error) {
// Log but don't break the main operation
console.error(`Augmentation error in ${this.name}:`, error)
// Optionally report to monitoring
this.reportError(error)
}
}
```
### 4. Be Performance Conscious
```typescript
class CachedAugmentation extends BaseAugmentation {
private cache = new Map<string, any>()
private cache = new Map<string, any>()
async execute<T>(operation: string, params: any): Promise<any> {
const key = this.getCacheKey(params)
async execute<T>(operation: string, params: any): Promise<any> {
const key = this.getCacheKey(params)
// Check cache first
if (this.cache.has(key)) {
return this.cache.get(key)
}
// Check cache first
if (this.cache.has(key)) {
return this.cache.get(key)
}
// Expensive operation
const result = await this.expensiveOperation(params)
this.cache.set(key, result)
// Expensive operation
const result = await this.expensiveOperation(params)
this.cache.set(key, result)
return result
}
return result
}
}
```
### 5. Clean Up Resources
```typescript
protected async onShutdown(): Promise<void> {
// Close connections
await this.connection?.close()
// Close connections
await this.connection?.close()
// Clear intervals
clearInterval(this.interval)
// Clear intervals
clearInterval(this.interval)
// Flush buffers
await this.flush()
// Flush buffers
await this.flush()
// Clear caches
this.cache.clear()
// Clear caches
this.cache.clear()
}
```
@ -457,10 +457,10 @@ protected async onShutdown(): Promise<void> {
```
my-augmentation/
├── src/
│ └── index.ts # Your augmentation
├── dist/ # Built output
│ └── index.ts # Your augmentation
├── dist/ # Built output
├── tests/
│ └── augmentation.test.ts
│ └── augmentation.test.ts
├── package.json
├── tsconfig.json
└── README.md
@ -469,21 +469,21 @@ my-augmentation/
### package.json
```json
{
"name": "@mycompany/brainy-custom-augmentation",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"keywords": ["brainy-augmentation"],
"peerDependencies": {
"@soulcraft/brainy": ">=2.0.0"
},
"brainy": {
"type": "augmentation",
"class": "CustomAugmentation",
"timing": "after",
"operations": ["add"],
"priority": 10
}
"name": "@mycompany/brainy-custom-augmentation",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"keywords": ["brainy-augmentation"],
"peerDependencies": {
"@soulcraft/brainy": ">=2.0.0"
},
"brainy": {
"type": "augmentation",
"class": "CustomAugmentation",
"timing": "after",
"operations": ["add"],
"priority": 10
}
}
```
@ -492,7 +492,7 @@ my-augmentation/
# Coming in 2.1+
npm run build
npm test
brainy publish # Publishes to brain-cloud registry
brainy publish # Publishes to brain-cloud registry
```
---

708
docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md
View file

@ -1,8 +1,8 @@
# Brainy v4.0.0 Cloud Deployment Guide
# Brainy Cloud Deployment Guide
This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy v4.0.0 codebase.
This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy codebase.
## 🆕 v4.0.0 Production Features
## 🆕 Production Features
**Cost Optimization at Scale:**
- **Lifecycle Policies**: Automatic tier transitions for 96% cost savings
@ -12,7 +12,7 @@ This guide provides production-ready deployment configurations for Brainy using
**Example Impact (500TB dataset):**
- Before: $138,000/year
- With v4.0.0 lifecycle policies: $5,940/year
- With lifecycle policies: $5,940/year
- **Savings: $132,060/year (96%)**
See the [Cost Optimization](#cost-optimization-v40) section below for implementation details.
@ -43,53 +43,53 @@ let brain
let handler
exports.handler = async (event) => {
if (!brain) {
const storage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com',
region: process.env.AWS_REGION,
bucket: process.env.BRAINY_BUCKET,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
prefix: 'brainy-data/'
})
if (!brain) {
const storage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com',
region: process.env.AWS_REGION,
bucket: process.env.BRAINY_BUCKET,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
prefix: 'brainy-data/'
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
auth: {
required: true,
apiKeys: [process.env.API_KEY]
}
}
}]
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
auth: {
required: true,
apiKeys: [process.env.API_KEY]
}
}
}]
})
await brain.init()
await brain.init()
// Get the universal handler from the augmentation
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// Get the universal handler from the augmentation
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// Convert Lambda event to Request
const url = `https://${event.requestContext.domainName}${event.rawPath}`
const request = new Request(url, {
method: event.requestContext.http.method,
headers: event.headers,
body: event.body
})
// Convert Lambda event to Request
const url = `https://${event.requestContext.domainName}${event.rawPath}`
const request = new Request(url, {
method: event.requestContext.http.method,
headers: event.headers,
body: event.body
})
// Use the universal handler
const response = await handler(request)
// Use the universal handler
const response = await handler(request)
return {
statusCode: response.status,
headers: Object.fromEntries(response.headers),
body: await response.text()
}
return {
statusCode: response.status,
headers: Object.fromEntries(response.headers),
body: await response.text()
}
}
```
@ -104,36 +104,36 @@ let brain
let handler
exports.brainyAPI = async (req, res) => {
if (!brain) {
const storage = new S3CompatibleStorage({
endpoint: 'storage.googleapis.com',
bucket: process.env.GCS_BUCKET,
accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY,
prefix: 'brainy/',
forcePathStyle: false,
region: 'US'
})
if (!brain) {
const storage = new S3CompatibleStorage({
endpoint: 'storage.googleapis.com',
bucket: process.env.GCS_BUCKET,
accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY,
prefix: 'brainy/',
forcePathStyle: false,
region: 'US'
})
brain = new Brainy({ storage })
await brain.init()
brain = new Brainy({ storage })
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// Convert Express req/res to Request/Response
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: JSON.stringify(req.body)
})
// Convert Express req/res to Request/Response
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: JSON.stringify(req.body)
})
const response = await handler(request)
const response = await handler(request)
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
}
```
@ -156,39 +156,39 @@ let brain
let handler
async function initBrainy() {
// Google Cloud Storage is S3-compatible
const storage = new S3CompatibleStorage({
endpoint: 'storage.googleapis.com',
bucket: process.env.GCS_BUCKET || 'brainy-data',
accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY,
prefix: 'brainy/',
forcePathStyle: false,
region: process.env.GCS_REGION || 'US'
})
// Google Cloud Storage is S3-compatible
const storage = new S3CompatibleStorage({
endpoint: 'storage.googleapis.com',
bucket: process.env.GCS_BUCKET || 'brainy-data',
accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY,
prefix: 'brainy/',
forcePathStyle: false,
region: process.env.GCS_REGION || 'US'
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
port: PORT,
cors: {
origin: process.env.CORS_ORIGIN || '*'
},
auth: {
required: process.env.AUTH_REQUIRED === 'true',
apiKeys: process.env.API_KEY ? [process.env.API_KEY] : []
}
}
}]
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
port: PORT,
cors: {
origin: process.env.CORS_ORIGIN || '*'
},
auth: {
required: process.env.AUTH_REQUIRED === 'true',
apiKeys: process.env.API_KEY ? [process.env.API_KEY] : []
}
}
}]
})
await brain.init()
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// Initialize on startup
@ -196,26 +196,26 @@ await initBrainy()
// Health check endpoint
app.get('/health', (req, res) => {
res.json({ status: 'healthy', service: 'brainy-api' })
res.json({ status: 'healthy', service: 'brainy-api' })
})
// Universal handler for all API routes
app.use('*', async (req, res) => {
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined
})
const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined
})
const response = await handler(request)
const response = await handler(request)
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
})
app.listen(PORT, () => {
console.log(`Brainy API running on port ${PORT}`)
console.log(`Brainy API running on port ${PORT}`)
})
```
@ -237,43 +237,43 @@ CMD ["node", "server.js"]
```yaml
# cloudbuild.yaml
steps:
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.']
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.']
# Push to Container Registry
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA']
# Push to Container Registry
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA']
# Deploy to Cloud Run
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud
args:
- 'run'
- 'deploy'
- 'brainy-api'
- '--image'
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
- '--region'
- 'us-central1'
- '--platform'
- 'managed'
- '--allow-unauthenticated'
- '--set-env-vars'
- 'GCS_BUCKET=brainy-storage,GCS_REGION=US'
- '--set-secrets'
- 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest'
- '--memory'
- '2Gi'
- '--cpu'
- '2'
- '--max-instances'
- '100'
- '--min-instances'
- '0'
# Deploy to Cloud Run
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud
args:
- 'run'
- 'deploy'
- 'brainy-api'
- '--image'
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
- '--region'
- 'us-central1'
- '--platform'
- 'managed'
- '--allow-unauthenticated'
- '--set-env-vars'
- 'GCS_BUCKET=brainy-storage,GCS_REGION=US'
- '--set-secrets'
- 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest'
- '--memory'
- '2Gi'
- '--cpu'
- '2'
- '--max-instances'
- '100'
- '--min-instances'
- '0'
images:
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
```
**Deploy with gcloud CLI:**
@ -283,12 +283,12 @@ gcloud builds submit --config cloudbuild.yaml
# Or deploy directly
gcloud run deploy brainy-api \
--source . \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--set-env-vars GCS_BUCKET=brainy-storage \
--set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest"
--source . \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--set-env-vars GCS_BUCKET=brainy-storage \
--set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest"
```
**Create GCS Bucket with S3-compatible access:**
@ -312,37 +312,37 @@ echo -n "YOUR_SECRET_KEY" | gcloud secrets create gcs-secret-key --data-file=-
```javascript
// index.js
module.exports = async function (context, req) {
const { Brainy } = require('@soulcraft/brainy')
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
const { Brainy } = require('@soulcraft/brainy')
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
const storage = new S3CompatibleStorage({
endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`,
bucket: 'brainy-data',
accessKeyId: process.env.AZURE_STORAGE_ACCOUNT,
secretAccessKey: process.env.AZURE_STORAGE_KEY,
prefix: 'entities/',
forcePathStyle: false
})
const storage = new S3CompatibleStorage({
endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`,
bucket: 'brainy-data',
accessKeyId: process.env.AZURE_STORAGE_ACCOUNT,
secretAccessKey: process.env.AZURE_STORAGE_KEY,
prefix: 'entities/',
forcePathStyle: false
})
const brain = new Brainy({ storage })
await brain.init()
const brain = new Brainy({ storage })
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
const handler = apiAugmentation.createUniversalHandler()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
const handler = apiAugmentation.createUniversalHandler()
const request = new Request(`https://${context.req.headers.host}${context.req.url}`, {
method: context.req.method,
headers: context.req.headers,
body: JSON.stringify(context.req.body)
})
const request = new Request(`https://${context.req.headers.host}${context.req.url}`, {
method: context.req.method,
headers: context.req.headers,
body: JSON.stringify(context.req.body)
})
const response = await handler(request)
const response = await handler(request)
context.res = {
status: response.status,
headers: Object.fromEntries(response.headers),
body: await response.text()
}
context.res = {
status: response.status,
headers: Object.fromEntries(response.headers),
body: await response.text()
}
}
```
@ -356,37 +356,37 @@ import { R2Storage } from '@soulcraft/brainy/storage' // Alias for S3CompatibleS
let handler
export default {
async fetch(request, env, ctx) {
if (!handler) {
const storage = new R2Storage({
endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'brainy-data',
accessKeyId: env.R2_ACCESS_KEY_ID,
secretAccessKey: env.R2_SECRET_ACCESS_KEY,
region: 'auto',
forcePathStyle: true
})
async fetch(request, env, ctx) {
if (!handler) {
const storage = new R2Storage({
endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'brainy-data',
accessKeyId: env.R2_ACCESS_KEY_ID,
secretAccessKey: env.R2_SECRET_ACCESS_KEY,
region: 'auto',
forcePathStyle: true
})
const brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
cors: { origin: '*' }
}
}]
})
const brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
cors: { origin: '*' }
}
}]
})
await brain.init()
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
// The handler works directly with Request/Response!
return handler(request)
}
// The handler works directly with Request/Response!
return handler(request)
}
}
```
@ -418,52 +418,52 @@ import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
let handler
export const config = {
runtime: 'edge',
runtime: 'edge',
}
export default async (request) => {
if (!handler) {
const storage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com',
region: 'us-east-1',
bucket: process.env.S3_BUCKET,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
})
if (!handler) {
const storage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com',
region: 'us-east-1',
bucket: process.env.S3_BUCKET,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
})
const brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: { enabled: true }
}]
})
const brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: { enabled: true }
}]
})
await brain.init()
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
return handler(request)
return handler(request)
}
```
```json
// vercel.json
{
"functions": {
"api/brainy.js": {
"maxDuration": 30,
"memory": 1024
}
},
"rewrites": [
{
"source": "/api/:path*",
"destination": "/api/brainy"
}
]
"functions": {
"api/brainy.js": {
"maxDuration": 30,
"memory": 1024
}
},
"rewrites": [
{
"source": "/api/:path*",
"destination": "/api/brainy"
}
]
}
```
@ -482,51 +482,51 @@ let brain
let handler
async function init() {
const storage = new S3CompatibleStorage({
endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com',
region: process.env.S3_REGION || 'us-east-1',
bucket: process.env.S3_BUCKET,
accessKeyId: process.env.S3_ACCESS_KEY,
secretAccessKey: process.env.S3_SECRET_KEY,
prefix: 'brainy/'
})
const storage = new S3CompatibleStorage({
endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com',
region: process.env.S3_REGION || 'us-east-1',
bucket: process.env.S3_BUCKET,
accessKeyId: process.env.S3_ACCESS_KEY,
secretAccessKey: process.env.S3_SECRET_KEY,
prefix: 'brainy/'
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
port: PORT
}
}]
})
brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
port: PORT
}
}]
})
await brain.init()
await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler()
}
await init()
// Universal handler for all routes
app.use('*', async (req, res) => {
const request = new Request(`http://localhost${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: JSON.stringify(req.body)
})
const request = new Request(`http://localhost${req.originalUrl}`, {
method: req.method,
headers: req.headers,
body: JSON.stringify(req.body)
})
const response = await handler(request)
const response = await handler(request)
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
res.status(response.status)
res.set(Object.fromEntries(response.headers))
res.send(await response.text())
})
app.listen(PORT, () => {
console.log(`Brainy API running on port ${PORT}`)
console.log(`Brainy API running on port ${PORT}`)
})
```
@ -552,26 +552,26 @@ restartPolicyMaxRetries = 3
```yaml
# render.yaml
services:
- type: web
name: brainy-api
runtime: node
buildCommand: npm install
startCommand: node server.js
envVars:
- key: S3_BUCKET
value: brainy-data
- key: S3_ENDPOINT
value: s3.amazonaws.com
- key: S3_REGION
value: us-east-1
- key: S3_ACCESS_KEY
sync: false
- key: S3_SECRET_KEY
sync: false
- key: API_KEY
generateValue: true
healthCheckPath: /health
autoDeploy: true
- type: web
name: brainy-api
runtime: node
buildCommand: npm install
startCommand: node server.js
envVars:
- key: S3_BUCKET
value: brainy-data
- key: S3_ENDPOINT
value: s3.amazonaws.com
- key: S3_REGION
value: us-east-1
- key: S3_ACCESS_KEY
sync: false
- key: S3_SECRET_KEY
sync: false
- key: API_KEY
generateValue: true
healthCheckPath: /health
autoDeploy: true
```
### Deno Deploy
@ -582,19 +582,19 @@ import { Brainy } from "npm:@soulcraft/brainy"
import { S3CompatibleStorage } from "npm:@soulcraft/brainy/storage"
const storage = new S3CompatibleStorage({
endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com",
bucket: Deno.env.get("S3_BUCKET")!,
accessKeyId: Deno.env.get("S3_ACCESS_KEY")!,
secretAccessKey: Deno.env.get("S3_SECRET_KEY")!,
region: "auto"
endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com",
bucket: Deno.env.get("S3_BUCKET")!,
accessKeyId: Deno.env.get("S3_ACCESS_KEY")!,
secretAccessKey: Deno.env.get("S3_SECRET_KEY")!,
region: "auto"
})
const brain = new Brainy({
storage,
augmentations: [{
name: 'api-server',
config: { enabled: true }
}]
storage,
augmentations: [{
name: 'api-server',
config: { enabled: true }
}]
})
await brain.init()
@ -653,29 +653,29 @@ RATE_LIMIT_MAX=100
```javascript
// REST API Client
const response = await fetch('https://your-api.com/api/brainy/add', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY'
},
body: JSON.stringify({
data: 'Your content here',
metadata: { type: 'document' }
})
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY'
},
body: JSON.stringify({
data: 'Your content here',
metadata: { type: 'document' }
})
})
const { id } = await response.json()
// Search
const searchResponse = await fetch('https://your-api.com/api/brainy/find', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY'
},
body: JSON.stringify({
query: 'neural networks'
})
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY'
},
body: JSON.stringify({
query: 'neural networks'
})
})
const results = await searchResponse.json()
@ -683,14 +683,14 @@ const results = await searchResponse.json()
// WebSocket Client
const ws = new WebSocket('wss://your-api.com/ws')
ws.onopen = () => {
ws.send(JSON.stringify({
type: 'subscribe',
pattern: 'technology'
}))
ws.send(JSON.stringify({
type: 'subscribe',
pattern: 'technology'
}))
}
ws.onmessage = (event) => {
const update = JSON.parse(event.data)
console.log('Real-time update:', update)
const update = JSON.parse(event.data)
console.log('Real-time update:', update)
}
```
@ -700,13 +700,13 @@ S3CompatibleStorage constructor parameters (verified from source):
```javascript
{
endpoint: string, // Required (e.g., 's3.amazonaws.com')
bucket: string, // Required
accessKeyId: string, // Required
secretAccessKey: string, // Required
region?: string, // Optional (default: 'us-east-1')
prefix?: string, // Optional (e.g., 'brainy/')
forcePathStyle?: boolean, // Optional (needed for some S3-compatible services)
endpoint: string, // Required (e.g., 's3.amazonaws.com')
bucket: string, // Required
accessKeyId: string, // Required
secretAccessKey: string, // Required
region?: string, // Optional (default: 'us-east-1')
prefix?: string, // Optional (e.g., 'brainy/')
forcePathStyle?: boolean, // Optional (needed for some S3-compatible services)
}
```
@ -727,7 +727,7 @@ S3CompatibleStorage constructor parameters (verified from source):
4. **Implement rate limiting** to prevent abuse
5. **Configure CORS** appropriately for your use case
## Cost Optimization (v4.0.0)
## Cost Optimization
### Enable Lifecycle Policies
@ -738,16 +738,16 @@ const storage = brain.storage
// Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // $0.0125/GB after 30 days
{ days: 90, storageClass: 'GLACIER' }, // $0.004/GB after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // $0.00099/GB after 1 year
]
}]
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // $0.0125/GB after 30 days
{ days: 90, storageClass: 'GLACIER' }, // $0.004/GB after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // $0.00099/GB after 1 year
]
}]
})
// Or enable Intelligent-Tiering for hands-off optimization
@ -765,14 +765,14 @@ await storage.enableIntelligentTiering('entities/', 'auto-optimize')
**Efficient bulk deletions:**
```javascript
// v4.0.0: Batch delete (1000 objects per request)
// Batch delete (1000 objects per request)
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => [
`entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
`entities/nouns/metadata/${id.substring(0, 2)}/${id}.json`
`entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
`entities/nouns/metadata/${id.substring(0, 2)}/${id}.json`
])
await storage.batchDelete(paths) // Much faster than individual deletes
await storage.batchDelete(paths) // Much faster than individual deletes
```
### Enable Compression (FileSystem)
@ -780,8 +780,8 @@ await storage.batchDelete(paths) // Much faster than individual deletes
**For local/server deployments:**
```javascript
const storage = new FileSystemStorage({
path: './data',
compression: true // 60-80% space savings
path: './data',
compression: true // 60-80% space savings
})
const brain = new Brainy({ storage })
@ -793,8 +793,8 @@ const brain = new Brainy({ storage })
2. **Use S3CompatibleStorage** for cloud deployments (better scalability)
3. **Enable the cache augmentation** for frequently accessed data
4. **Configure appropriate memory limits** for your runtime
5. **v4.0.0**: Enable lifecycle policies to reduce storage costs by 96%
6. **v4.0.0**: Use batch operations for cleanup tasks
5. Enable lifecycle policies to reduce storage costs by 96%
6. Use batch operations for cleanup tasks
## Troubleshooting
@ -810,15 +810,15 @@ const brain = new Brainy({ storage })
Enable debug logging by setting:
```javascript
const brain = new Brainy({
storage,
debug: true,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
verbose: true
}
}]
storage,
debug: true,
augmentations: [{
name: 'api-server',
config: {
enabled: true,
verbose: true
}
}]
})
```

470
docs/deployment/aws-deployment.md
View file

@ -18,24 +18,24 @@ const { Brainy } = require('@soulcraft/brainy')
let brain
exports.handler = async (event) => {
// Brainy auto-detects Lambda environment and configures accordingly
if (!brain) {
brain = new Brainy() // Zero config - auto-adapts to Lambda
await brain.init()
}
// Brainy auto-detects Lambda environment and configures accordingly
if (!brain) {
brain = new Brainy() // Zero config - auto-adapts to Lambda
await brain.init()
}
const { method, ...params } = JSON.parse(event.body)
const { method, ...params } = JSON.parse(event.body)
switch(method) {
case 'add':
const id = await brain.add(params)
return { statusCode: 200, body: JSON.stringify({ id }) }
case 'find':
const results = await brain.find(params)
return { statusCode: 200, body: JSON.stringify({ results }) }
default:
return { statusCode: 400, body: 'Unknown method' }
}
switch(method) {
case 'add':
const id = await brain.add(params)
return { statusCode: 200, body: JSON.stringify({ id }) }
case 'find':
const results = await brain.find(params)
return { statusCode: 200, body: JSON.stringify({ results }) }
default:
return { statusCode: 400, body: 'Unknown method' }
}
}
EOF
@ -56,26 +56,26 @@ docker push $ECR_URI/brainy:latest
# Deploy with minimal ECS task definition
cat > task-definition.json << 'EOF'
{
"family": "brainy",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"cpu": "256",
"memory": "512",
"containerDefinitions": [{
"name": "brainy",
"image": "$ECR_URI/brainy:latest",
"environment": [
{"name": "NODE_ENV", "value": "production"}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/brainy",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
}
}]
"family": "brainy",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"cpu": "256",
"memory": "512",
"containerDefinitions": [{
"name": "brainy",
"image": "$ECR_URI/brainy:latest",
"environment": [
{"name": "NODE_ENV", "value": "production"}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/brainy",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
}
}]
}
EOF
@ -138,14 +138,14 @@ const brain = new Brainy()
```javascript
const brain = new Brainy({
storage: {
type: 's3',
options: {
bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket
region: process.env.AWS_REGION || 'auto', // 'auto' detects region
// IAM role provides credentials automatically
}
}
storage: {
type: 's3',
options: {
bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket
region: process.env.AWS_REGION || 'auto', // 'auto' detects region
// IAM role provides credentials automatically
}
}
})
```
@ -156,23 +156,23 @@ const brain = new Brainy({
```yaml
# Auto-scaling policy
Resources:
AutoScalingTarget:
Type: AWS::ApplicationAutoScaling::ScalableTarget
Properties:
ServiceNamespace: ecs
ResourceId: service/default/brainy
ScalableDimension: ecs:service:DesiredCount
MinCapacity: 2
MaxCapacity: 100
AutoScalingTarget:
Type: AWS::ApplicationAutoScaling::ScalableTarget
Properties:
ServiceNamespace: ecs
ResourceId: service/default/brainy
ScalableDimension: ecs:service:DesiredCount
MinCapacity: 2
MaxCapacity: 100
AutoScalingPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyType: TargetTrackingScaling
TargetTrackingScalingPolicyConfiguration:
TargetValue: 70.0
PredefinedMetricSpecification:
PredefinedMetricType: ECSServiceAverageCPUUtilization
AutoScalingPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyType: TargetTrackingScaling
TargetTrackingScalingPolicyConfiguration:
TargetValue: 70.0
PredefinedMetricSpecification:
PredefinedMetricType: ECSServiceAverageCPUUtilization
```
### 2. Vertical Scaling
@ -189,10 +189,10 @@ Brainy automatically adapts to available memory:
```javascript
// Brainy automatically handles multi-AZ with S3
const brain = new Brainy({
distributed: {
enabled: true, // Auto-enables with S3 storage
coordinationMethod: 'auto' // Uses S3 for coordination
}
distributed: {
enabled: true, // Auto-enables with S3 storage
coordinationMethod: 'auto' // Uses S3 for coordination
}
})
```
@ -201,17 +201,17 @@ const brain = new Brainy({
```bash
# Application Load Balancer with health checks
aws elbv2 create-load-balancer \
--name brainy-alb \
--subnets subnet-xxx subnet-yyy \
--security-groups sg-xxx
--name brainy-alb \
--subnets subnet-xxx subnet-yyy \
--security-groups sg-xxx
aws elbv2 create-target-group \
--name brainy-targets \
--protocol HTTP \
--port 3000 \
--vpc-id vpc-xxx \
--health-check-path /health \
--health-check-interval-seconds 30
--name brainy-targets \
--protocol HTTP \
--port 3000 \
--vpc-id vpc-xxx \
--health-check-path /health \
--health-check-interval-seconds 30
```
## Monitoring & Observability
@ -233,16 +233,16 @@ Brainy automatically sends metrics when running on AWS:
```javascript
const brain = new Brainy({
monitoring: {
enabled: true,
customMetrics: {
namespace: 'Brainy/Production',
dimensions: [
{ Name: 'Environment', Value: 'production' },
{ Name: 'Service', Value: 'api' }
]
}
}
monitoring: {
enabled: true,
customMetrics: {
namespace: 'Brainy/Production',
dimensions: [
{ Name: 'Environment', Value: 'production' },
{ Name: 'Service', Value: 'api' }
]
}
}
})
```
@ -252,22 +252,22 @@ const brain = new Brainy({
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::brainy-*/*",
"arn:aws:s3:::brainy-*"
]
}
]
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::brainy-*/*",
"arn:aws:s3:::brainy-*"
]
}
]
}
```
@ -285,12 +285,12 @@ aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.2.0/24 --availability-z
```javascript
// Automatic encryption with S3
const brain = new Brainy({
storage: {
type: 's3',
options: {
encryption: 'auto' // Uses S3 SSE-S3 by default
}
}
storage: {
type: 's3',
options: {
encryption: 'auto' // Uses S3 SSE-S3 by default
}
}
})
```
@ -300,14 +300,14 @@ const brain = new Brainy({
```bash
aws ec2 request-spot-fleet --spot-fleet-request-config '{
"IamFleetRole": "arn:aws:iam::xxx:role/fleet-role",
"TargetCapacity": 2,
"SpotPrice": "0.05",
"LaunchSpecifications": [{
"ImageId": "ami-xxx",
"InstanceType": "t3.medium",
"UserData": "BASE64_ENCODED_STARTUP_SCRIPT"
}]
"IamFleetRole": "arn:aws:iam::xxx:role/fleet-role",
"TargetCapacity": 2,
"SpotPrice": "0.05",
"LaunchSpecifications": [{
"ImageId": "ami-xxx",
"InstanceType": "t3.medium",
"UserData": "BASE64_ENCODED_STARTUP_SCRIPT"
}]
}'
```
@ -316,12 +316,12 @@ aws ec2 request-spot-fleet --spot-fleet-request-config '{
```javascript
// Brainy automatically uses S3 Intelligent-Tiering
const brain = new Brainy({
storage: {
type: 's3',
options: {
storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization
}
}
storage: {
type: 's3',
options: {
storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization
}
}
})
```
@ -329,8 +329,8 @@ const brain = new Brainy({
```bash
aws lambda put-function-concurrency \
--function-name brainy-handler \
--reserved-concurrent-executions 10
--function-name brainy-handler \
--reserved-concurrent-executions 10
```
## Deployment Automation
@ -340,28 +340,28 @@ aws lambda put-function-concurrency \
```yaml
name: Deploy to AWS
on:
push:
branches: [main]
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Deploy to ECS
run: |
docker build -t brainy .
docker tag brainy:latest $ECR_URI/brainy:latest
docker push $ECR_URI/brainy:latest
aws ecs update-service --cluster default --service brainy --force-new-deployment
- name: Deploy to ECS
run: |
docker build -t brainy .
docker tag brainy:latest $ECR_URI/brainy:latest
docker push $ECR_URI/brainy:latest
aws ecs update-service --cluster default --service brainy --force-new-deployment
```
## Troubleshooting
@ -369,121 +369,121 @@ jobs:
### Common Issues
1. **Storage Auto-Detection Fails**
```javascript
// Explicitly specify storage
const brain = new Brainy({
storage: { type: 's3', options: { bucket: 'my-bucket' } }
})
```
```javascript
// Explicitly specify storage
const brain = new Brainy({
storage: { type: 's3', options: { bucket: 'my-bucket' } }
})
```
2. **Memory Issues**
```javascript
// Optimize for low memory
const brain = new Brainy({
cache: { maxSize: 100 }, // Reduce cache size
index: { M: 8 } // Reduce HNSW connections
})
```
```javascript
// Optimize for low memory
const brain = new Brainy({
cache: { maxSize: 100 }, // Reduce cache size
index: { M: 8 } // Reduce HNSW connections
})
```
3. **Cold Starts (Lambda)**
**v7.3.0+ Progressive Initialization (Zero-Config)**
**Progressive Initialization (Zero-Config)**
Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME)
and uses progressive initialization for <200ms cold starts:
Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME)
and uses progressive initialization for <200ms cold starts:
```javascript
// Zero-config - Brainy auto-detects Lambda
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'my-bucket',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}
}
})
await brain.init() // Returns in <200ms
```javascript
// Zero-config - Brainy auto-detects Lambda
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'my-bucket',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}
}
})
await brain.init() // Returns in <200ms
// First write validates bucket (lazy validation)
await brain.add('noun', { name: 'test' }) // Validates here
```
// First write validates bucket (lazy validation)
await brain.add('noun', { name: 'test' }) // Validates here
```
**Manual Override (if needed)**
```javascript
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'my-bucket',
// Force specific mode
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
}
}
})
```
**Manual Override (if needed)**
```javascript
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'my-bucket',
// Force specific mode
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
}
}
})
```
| Mode | Cold Start | Best For |
|------|------------|----------|
| `auto` (default) | <200ms in Lambda | Zero-config, auto-detects |
| `progressive` | <200ms always | Force fast init everywhere |
| `strict` | 100-500ms+ | Local dev, tests, debugging |
| Mode | Cold Start | Best For |
|------|------------|----------|
| `auto` (default) | <200ms in Lambda | Zero-config, auto-detects |
| `progressive` | <200ms always | Force fast init everywhere |
| `strict` | 100-500ms+ | Local dev, tests, debugging |
**Warm-up (Alternative)**
```javascript
exports.warmup = async () => {
if (!brain) {
brain = new Brainy({ warmup: true })
await brain.init()
}
}
```
**Warm-up (Alternative)**
```javascript
exports.warmup = async () => {
if (!brain) {
brain = new Brainy({ warmup: true })
await brain.init()
}
}
```
**Readiness Detection (v7.3.0+)**
**Readiness Detection**
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
```javascript
let brain
```javascript
let brain
exports.handler = async (event) => {
if (!brain) {
brain = new Brainy({ storage: { type: 's3', ... } })
brain.init() // Fire and forget
}
exports.handler = async (event) => {
if (!brain) {
brain = new Brainy({ storage: { type: 's3', ... } })
brain.init() // Fire and forget
}
// Wait for initialization to complete
await brain.ready
// Wait for initialization to complete
await brain.ready
// Now safe to use brain methods
const results = await brain.find({ query: event.queryStringParameters.q })
return { statusCode: 200, body: JSON.stringify(results) }
}
```
// Now safe to use brain methods
const results = await brain.find({ query: event.queryStringParameters.q })
return { statusCode: 200, body: JSON.stringify(results) }
}
```
For health checks, use `isFullyInitialized()` to verify all background tasks are complete:
For health checks, use `isFullyInitialized()` to verify all background tasks are complete:
```javascript
exports.healthCheck = async () => {
try {
await brain.ready
return {
statusCode: 200,
body: JSON.stringify({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
}
} catch (error) {
return {
statusCode: 503,
body: JSON.stringify({ status: 'initializing' })
}
}
}
```
```javascript
exports.healthCheck = async () => {
try {
await brain.ready
return {
statusCode: 200,
body: JSON.stringify({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
}
} catch (error) {
return {
statusCode: 503,
body: JSON.stringify({ status: 'initializing' })
}
}
}
```
## Production Checklist

634
docs/deployment/gcp-deployment.md
View file

@ -10,10 +10,10 @@ Deploy Brainy on GCP with automatic scaling, global distribution, and zero-confi
```bash
# Build and deploy with one command
gcloud run deploy brainy \
--source . \
--platform managed \
--region us-central1 \
--allow-unauthenticated
--source . \
--platform managed \
--region us-central1 \
--allow-unauthenticated
# Brainy auto-detects Cloud Run and configures:
# - Memory-optimized caching
@ -30,44 +30,44 @@ const { Brainy } = require('@soulcraft/brainy')
let brain
exports.brainyHandler = async (req, res) => {
// Zero-config - auto-adapts to Cloud Functions
if (!brain) {
brain = new Brainy() // Detects GCP environment automatically
await brain.init()
}
// Zero-config - auto-adapts to Cloud Functions
if (!brain) {
brain = new Brainy() // Detects GCP environment automatically
await brain.init()
}
const { method, ...params } = req.body
const { method, ...params } = req.body
try {
let result
switch(method) {
case 'add':
result = await brain.add(params)
break
case 'find':
result = await brain.find(params)
break
case 'relate':
result = await brain.relate(params)
break
default:
return res.status(400).json({ error: 'Unknown method' })
}
res.json({ result })
} catch (error) {
res.status(500).json({ error: error.message })
}
try {
let result
switch(method) {
case 'add':
result = await brain.add(params)
break
case 'find':
result = await brain.find(params)
break
case 'relate':
result = await brain.relate(params)
break
default:
return res.status(400).json({ error: 'Unknown method' })
}
res.json({ result })
} catch (error) {
res.status(500).json({ error: error.message })
}
}
```
Deploy:
```bash
gcloud functions deploy brainy \
--runtime nodejs20 \
--trigger-http \
--entry-point brainyHandler \
--memory 512MB \
--timeout 60s
--runtime nodejs20 \
--trigger-http \
--entry-point brainyHandler \
--memory 512MB \
--timeout 60s
```
### Option 3: Google Kubernetes Engine (GKE)
@ -75,7 +75,7 @@ gcloud functions deploy brainy \
```bash
# Create autopilot cluster (fully managed, zero-config)
gcloud container clusters create-auto brainy-cluster \
--region us-central1
--region us-central1
# Deploy using Cloud Build
gcloud builds submit --tag gcr.io/$PROJECT_ID/brainy
@ -85,39 +85,39 @@ kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: brainy
name: brainy
spec:
replicas: 3
selector:
matchLabels:
app: brainy
template:
metadata:
labels:
app: brainy
spec:
containers:
- name: brainy
image: gcr.io/$PROJECT_ID/brainy
resources:
requests:
memory: "512Mi"
cpu: "250m"
env:
- name: NODE_ENV
value: production
replicas: 3
selector:
matchLabels:
app: brainy
template:
metadata:
labels:
app: brainy
spec:
containers:
- name: brainy
image: gcr.io/$PROJECT_ID/brainy
resources:
requests:
memory: "512Mi"
cpu: "250m"
env:
- name: NODE_ENV
value: production
---
apiVersion: v1
kind: Service
metadata:
name: brainy-service
name: brainy-service
spec:
type: LoadBalancer
selector:
app: brainy
ports:
- port: 80
targetPort: 3000
type: LoadBalancer
selector:
app: brainy
ports:
- port: 80
targetPort: 3000
EOF
```
@ -139,14 +139,14 @@ const brain = new Brainy()
```javascript
const brain = new Brainy({
storage: {
type: 's3', // GCS is S3-compatible
options: {
endpoint: 'https://storage.googleapis.com',
bucket: process.env.GCS_BUCKET || 'auto', // Auto-creates bucket
// Uses Application Default Credentials automatically
}
}
storage: {
type: 's3', // GCS is S3-compatible
options: {
endpoint: 'https://storage.googleapis.com',
bucket: process.env.GCS_BUCKET || 'auto', // Auto-creates bucket
// Uses Application Default Credentials automatically
}
}
})
```
@ -154,13 +154,13 @@ const brain = new Brainy({
```javascript
const brain = new Brainy({
storage: {
type: 'firestore',
options: {
projectId: process.env.GCP_PROJECT || 'auto',
collection: 'brainy-data'
}
}
storage: {
type: 'firestore',
options: {
projectId: process.env.GCP_PROJECT || 'auto',
collection: 'brainy-data'
}
}
})
```
@ -173,24 +173,24 @@ const brain = new Brainy({
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: brainy
annotations:
run.googleapis.com/execution-environment: gen2
name: brainy
annotations:
run.googleapis.com/execution-environment: gen2
spec:
template:
metadata:
annotations:
autoscaling.knative.dev/minScale: "1"
autoscaling.knative.dev/maxScale: "1000"
autoscaling.knative.dev/target: "80"
spec:
containerConcurrency: 100
containers:
- image: gcr.io/PROJECT_ID/brainy
resources:
limits:
cpu: "2"
memory: "2Gi"
template:
metadata:
annotations:
autoscaling.knative.dev/minScale: "1"
autoscaling.knative.dev/maxScale: "1000"
autoscaling.knative.dev/target: "80"
spec:
containerConcurrency: 100
containers:
- image: gcr.io/PROJECT_ID/brainy
resources:
limits:
cpu: "2"
memory: "2Gi"
```
### 2. GKE Horizontal Pod Autoscaling
@ -199,27 +199,27 @@ spec:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: brainy-hpa
name: brainy-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: brainy
minReplicas: 3
maxReplicas: 100
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: brainy
minReplicas: 3
maxReplicas: 100
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
```
## Global Distribution
@ -229,11 +229,11 @@ spec:
```javascript
// Brainy automatically handles multi-region with GCS
const brain = new Brainy({
distributed: {
enabled: true,
regions: ['us-central1', 'europe-west1', 'asia-northeast1'],
replication: 'auto' // Automatic cross-region replication
}
distributed: {
enabled: true,
regions: ['us-central1', 'europe-west1', 'asia-northeast1'],
replication: 'auto' // Automatic cross-region replication
}
})
```
@ -242,14 +242,14 @@ const brain = new Brainy({
```bash
# Global load balancing with Traffic Director
gcloud compute backend-services create brainy-global \
--global \
--load-balancing-scheme=INTERNAL_SELF_MANAGED \
--protocol=HTTP2
--global \
--load-balancing-scheme=INTERNAL_SELF_MANAGED \
--protocol=HTTP2
gcloud compute backend-services add-backend brainy-global \
--global \
--network-endpoint-group=brainy-neg \
--network-endpoint-group-region=us-central1
--global \
--network-endpoint-group=brainy-neg \
--network-endpoint-group-region=us-central1
```
## Monitoring & Observability
@ -277,22 +277,22 @@ const { Monitoring } = require('@google-cloud/monitoring')
const monitoring = new Monitoring.MetricServiceClient()
const brain = new Brainy({
onMetric: async (metric) => {
// Send custom metrics to Cloud Monitoring
await monitoring.createTimeSeries({
name: monitoring.projectPath(projectId),
timeSeries: [{
metric: {
type: `custom.googleapis.com/brainy/${metric.name}`,
labels: metric.labels
},
points: [{
interval: { endTime: { seconds: Date.now() / 1000 } },
value: { doubleValue: metric.value }
}]
}]
})
}
onMetric: async (metric) => {
// Send custom metrics to Cloud Monitoring
await monitoring.createTimeSeries({
name: monitoring.projectPath(projectId),
timeSeries: [{
metric: {
type: `custom.googleapis.com/brainy/${metric.name}`,
labels: metric.labels
},
points: [{
interval: { endTime: { seconds: Date.now() / 1000 } },
value: { doubleValue: metric.value }
}]
}]
})
}
})
```
@ -300,10 +300,10 @@ const brain = new Brainy({
```javascript
const brain = new Brainy({
tracing: {
enabled: true,
sampleRate: 0.1 // Sample 10% of requests
}
tracing: {
enabled: true,
sampleRate: 0.1 // Sample 10% of requests
}
})
```
@ -316,9 +316,9 @@ const brain = new Brainy({
apiVersion: v1
kind: ServiceAccount
metadata:
name: brainy-sa
annotations:
iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com
name: brainy-sa
annotations:
iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com
```
### 2. Binary Authorization
@ -328,11 +328,11 @@ metadata:
apiVersion: binaryauthorization.grafeas.io/v1beta1
kind: Policy
metadata:
name: brainy-policy
name: brainy-policy
spec:
defaultAdmissionRule:
requireAttestationsBy:
- projects/PROJECT_ID/attestors/prod-attestor
defaultAdmissionRule:
requireAttestationsBy:
- projects/PROJECT_ID/attestors/prod-attestor
```
### 3. VPC Service Controls
@ -340,9 +340,9 @@ spec:
```bash
# Create VPC Service Perimeter
gcloud access-context-manager perimeters create brainy_perimeter \
--resources=projects/PROJECT_NUMBER \
--restricted-services=storage.googleapis.com \
--title="Brainy Security Perimeter"
--resources=projects/PROJECT_NUMBER \
--restricted-services=storage.googleapis.com \
--title="Brainy Security Perimeter"
```
## Cost Optimization
@ -354,16 +354,16 @@ gcloud access-context-manager perimeters create brainy_perimeter \
apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerNodePool
metadata:
name: brainy-preemptible-pool
name: brainy-preemptible-pool
spec:
clusterRef:
name: brainy-cluster
config:
preemptible: true
machineType: n2-standard-2
autoscaling:
minNodeCount: 1
maxNodeCount: 10
clusterRef:
name: brainy-cluster
config:
preemptible: true
machineType: n2-standard-2
autoscaling:
minNodeCount: 1
maxNodeCount: 10
```
### 2. Cloud CDN for Static Assets
@ -371,11 +371,11 @@ spec:
```bash
# Enable Cloud CDN for frequently accessed data
gcloud compute backend-buckets create brainy-assets \
--gcs-bucket-name=brainy-static
--gcs-bucket-name=brainy-static
gcloud compute backend-buckets update brainy-assets \
--enable-cdn \
--cache-mode=CACHE_ALL_STATIC
--enable-cdn \
--cache-mode=CACHE_ALL_STATIC
```
### 3. Committed Use Discounts
@ -383,8 +383,8 @@ gcloud compute backend-buckets update brainy-assets \
```bash
# Purchase committed use for predictable workloads
gcloud compute commitments create brainy-commitment \
--plan=TWELVE_MONTH \
--resources=vcpu=100,memory=400
--plan=TWELVE_MONTH \
--resources=vcpu=100,memory=400
```
## Deployment Automation
@ -394,28 +394,28 @@ gcloud compute commitments create brainy-commitment \
```yaml
# cloudbuild.yaml
steps:
# Build container
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.']
# Build container
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.']
# Push to registry
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA']
# Push to registry
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA']
# Deploy to Cloud Run
- name: 'gcr.io/cloud-builders/gcloud'
args:
- 'run'
- 'deploy'
- 'brainy'
- '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'
- '--region=us-central1'
- '--platform=managed'
# Deploy to Cloud Run
- name: 'gcr.io/cloud-builders/gcloud'
args:
- 'run'
- 'deploy'
- 'brainy'
- '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'
- '--region=us-central1'
- '--platform=managed'
# Trigger on push to main
trigger:
branch:
name: main
branch:
name: main
```
### Terraform Infrastructure
@ -423,40 +423,40 @@ trigger:
```hcl
# main.tf
resource "google_cloud_run_service" "brainy" {
name = "brainy"
location = "us-central1"
name = "brainy"
location = "us-central1"
template {
spec {
containers {
image = "gcr.io/${var.project_id}/brainy"
template {
spec {
containers {
image = "gcr.io/${var.project_id}/brainy"
resources {
limits = {
cpu = "2"
memory = "2Gi"
}
}
resources {
limits = {
cpu = "2"
memory = "2Gi"
}
}
env {
name = "NODE_ENV"
value = "production"
}
}
}
}
env {
name = "NODE_ENV"
value = "production"
}
}
}
}
traffic {
percent = 100
latest_revision = true
}
traffic {
percent = 100
latest_revision = true
}
}
resource "google_cloud_run_service_iam_member" "public" {
service = google_cloud_run_service.brainy.name
location = google_cloud_run_service.brainy.location
role = "roles/run.invoker"
member = "allUsers"
service = google_cloud_run_service.brainy.name
location = google_cloud_run_service.brainy.location
role = "roles/run.invoker"
member = "allUsers"
}
```
@ -467,13 +467,13 @@ resource "google_cloud_run_service_iam_member" "public" {
```javascript
// Brainy can use Memorystore for caching
const brain = new Brainy({
cache: {
type: 'redis',
options: {
host: process.env.REDIS_HOST || 'auto-detect',
port: 6379
}
}
cache: {
type: 'redis',
options: {
host: process.env.REDIS_HOST || 'auto-detect',
port: 6379
}
}
})
```
@ -481,13 +481,13 @@ const brain = new Brainy({
```javascript
const brain = new Brainy({
metadata: {
type: 'spanner',
options: {
instance: 'brainy-instance',
database: 'brainy-db'
}
}
metadata: {
type: 'spanner',
options: {
instance: 'brainy-instance',
database: 'brainy-db'
}
}
})
```
@ -496,116 +496,116 @@ const brain = new Brainy({
### Common Issues
1. **Quota Exceeded**
```bash
# Check quotas
gcloud compute project-info describe --project=$PROJECT_ID
```bash
# Check quotas
gcloud compute project-info describe --project=$PROJECT_ID
# Request increase
gcloud compute project-info add-metadata \
--metadata google-compute-default-region=us-central1
```
# Request increase
gcloud compute project-info add-metadata \
--metadata google-compute-default-region=us-central1
```
2. **Cold Starts**
**v7.3.0+ Progressive Initialization (Zero-Config)**
**Progressive Initialization (Zero-Config)**
Brainy automatically detects Cloud Run and Cloud Functions environments
and uses progressive initialization for <200ms cold starts:
Brainy automatically detects Cloud Run and Cloud Functions environments
and uses progressive initialization for <200ms cold starts:
```javascript
// Zero-config - Brainy auto-detects Cloud Run (K_SERVICE env var)
const brain = new Brainy({
storage: {
type: 'gcs',
gcsNativeStorage: { bucketName: 'my-bucket' }
}
})
await brain.init() // Returns in <200ms
```javascript
// Zero-config - Brainy auto-detects Cloud Run (K_SERVICE env var)
const brain = new Brainy({
storage: {
type: 'gcs',
gcsNativeStorage: { bucketName: 'my-bucket' }
}
})
await brain.init() // Returns in <200ms
// First write validates bucket (lazy validation)
await brain.add('noun', { name: 'test' }) // Validates here
```
// First write validates bucket (lazy validation)
await brain.add('noun', { name: 'test' }) // Validates here
```
**Manual Override (if needed)**
```javascript
const brain = new Brainy({
storage: {
type: 'gcs',
gcsNativeStorage: {
bucketName: 'my-bucket',
// Force specific mode
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
}
}
})
```
**Manual Override (if needed)**
```javascript
const brain = new Brainy({
storage: {
type: 'gcs',
gcsNativeStorage: {
bucketName: 'my-bucket',
// Force specific mode
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
}
}
})
```
| Mode | Cold Start | Best For |
|------|------------|----------|
| `auto` (default) | <200ms in cloud | Zero-config, auto-detects |
| `progressive` | <200ms always | Force fast init everywhere |
| `strict` | 100-500ms+ | Local dev, tests, debugging |
| Mode | Cold Start | Best For |
|------|------------|----------|
| `auto` (default) | <200ms in cloud | Zero-config, auto-detects |
| `progressive` | <200ms always | Force fast init everywhere |
| `strict` | 100-500ms+ | Local dev, tests, debugging |
**Keep Warm (Alternative)**
```javascript
// Keep minimum instances warm
const brain = new Brainy({
warmup: {
enabled: true,
interval: 60000 // Ping every minute
}
})
```
**Keep Warm (Alternative)**
```javascript
// Keep minimum instances warm
const brain = new Brainy({
warmup: {
enabled: true,
interval: 60000 // Ping every minute
}
})
```
**Readiness Detection (v7.3.0+)**
**Readiness Detection**
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
```javascript
let brain
```javascript
let brain
async function handleRequest(req, res) {
if (!brain) {
brain = new Brainy({ storage: { type: 'gcs', ... } })
brain.init() // Fire and forget
}
async function handleRequest(req, res) {
if (!brain) {
brain = new Brainy({ storage: { type: 'gcs', ... } })
brain.init() // Fire and forget
}
// Wait for initialization to complete
await brain.ready
// Wait for initialization to complete
await brain.ready
// Now safe to use brain methods
const results = await brain.find({ query: req.query.q })
res.json(results)
}
```
// Now safe to use brain methods
const results = await brain.find({ query: req.query.q })
res.json(results)
}
```
For Cloud Run health checks, use `isFullyInitialized()` to verify all background tasks are complete:
For Cloud Run health checks, use `isFullyInitialized()` to verify all background tasks are complete:
```javascript
// Health check endpoint for Cloud Run
app.get('/health', async (req, res) => {
try {
await brain.ready
res.json({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
} catch (error) {
res.status(503).json({ status: 'initializing' })
}
})
```
```javascript
// Health check endpoint for Cloud Run
app.get('/health', async (req, res) => {
try {
await brain.ready
res.json({
status: 'ready',
fullyInitialized: brain.isFullyInitialized()
})
} catch (error) {
res.status(503).json({ status: 'initializing' })
}
})
```
3. **Memory Pressure**
```javascript
// Optimize for GCP memory constraints
const brain = new Brainy({
memory: {
mode: 'aggressive', // Aggressive garbage collection
maxHeap: 0.8 // Use 80% of available memory
}
})
```
```javascript
// Optimize for GCP memory constraints
const brain = new Brainy({
memory: {
mode: 'aggressive', // Aggressive garbage collection
maxHeap: 0.8 // Use 80% of available memory
}
})
```
## Production Checklist

161
docs/features/instant-fork.md
View file

@ -42,10 +42,10 @@ await experiment.updateAll(riskyTransformation)
// Works? Great! Use the experimental branch.
// Failed? Just discard.
if (success) {
// Make experiment the new main branch
await brain.checkout('test-migration')
// Make experiment the new main branch
await brain.checkout('test-migration')
} else {
await experiment.destroy() // No harm done
await experiment.destroy() // No harm done
}
```
@ -61,25 +61,25 @@ if (success) {
Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking:
### Architecture (v5.0.0)
### Architecture
1. **HNSW Index COW** (The Performance Bottleneck):
- **Shallow Copy**: O(1) Map reference copying (~10ms for 1M+ nodes)
- **Lazy Deep Copy**: Nodes copied only when modified (`ensureCOW()`)
- **Write Isolation**: Fork modifications don't affect parent
- **Implementation**: `src/hnsw/hnswIndex.ts` lines 2088-2150
- **Shallow Copy**: O(1) Map reference copying (~10ms for 1M+ nodes)
- **Lazy Deep Copy**: Nodes copied only when modified (`ensureCOW()`)
- **Write Isolation**: Fork modifications don't affect parent
- **Implementation**: `src/hnsw/hnswIndex.ts` lines 2088-2150
2. **Metadata & Graph Indexes** (Fast Rebuild):
- **Rebuild from Storage**: < 500ms total for both indexes
- **Shared Storage**: Both indexes read from COW-enabled storage layer
- **Acceptable Overhead**: Fast enough not to need in-memory COW
- **Rebuild from Storage**: < 500ms total for both indexes
- **Shared Storage**: Both indexes read from COW-enabled storage layer
- **Acceptable Overhead**: Fast enough not to need in-memory COW
3. **Storage Layer** (Shared):
- **RefManager**: Manages branch references
- **BlobStorage**: Content-addressable with deduplication
- **All Adapters**: Memory, FileSystem, S3, R2, GCS, Azure, OPFS
- **RefManager**: Manages branch references
- **BlobStorage**: Content-addressable with deduplication
- **All Adapters**: Memory, FileSystem, S3, R2, GCS, Azure, OPFS
**Performance (v5.0.0)**:
**Performance**:
- **Fork time**: < 100ms @ 10K entities (MEASURED in tests)
- **Storage overhead**: 10-20% (shared blobs, only changed data duplicated)
- **Memory overhead**: 10-20% (shared HNSW nodes, only modified nodes duplicated)
@ -87,11 +87,11 @@ Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking:
**Technical Details**:
```typescript
// Shallow copy HNSW (instant)
clone.index.enableCOW(this.index) // O(1) Map reference copy
clone.index.enableCOW(this.index) // O(1) Map reference copy
// Fast rebuild small indexes from shared storage
clone.metadataIndex = new MetadataIndexManager(clone.storage) // <100ms
clone.graphIndex = new GraphAdjacencyIndex(clone.storage) // <500ms
clone.metadataIndex = new MetadataIndexManager(clone.storage) // <100ms
clone.graphIndex = new GraphAdjacencyIndex(clone.storage) // <500ms
```
---
@ -128,11 +128,11 @@ await fork.add({ noun: 'user', data: { name: 'Charlie' } })
// All APIs work
const users = await fork.find({ noun: 'user' })
console.log(users.length) // 3 (Alice, Bob, Charlie)
console.log(users.length) // 3 (Alice, Bob, Charlie)
// Original brain unchanged
const originalUsers = await brain.find({ noun: 'user' })
console.log(originalUsers.length) // 2 (Alice, Bob)
console.log(originalUsers.length) // 2 (Alice, Bob)
```
### 3. Discard When Done
@ -165,30 +165,30 @@ const migration = await brain.fork('migration-test')
// Run migration on fork
const users = await migration.find({ noun: 'user' })
for (const user of users) {
await migration.update(user.id, {
email: user.data.email.toLowerCase(), // Normalize emails
verified: user.data.verified ?? false // Add missing field
})
await migration.update(user.id, {
email: user.data.email.toLowerCase(), // Normalize emails
verified: user.data.verified ?? false // Add missing field
})
}
// Validate migration
const allValid = (await migration.find({ noun: 'user' }))
.every(u => u.data.email === u.data.email.toLowerCase())
.every(u => u.data.email === u.data.email.toLowerCase())
if (allValid) {
console.log('✅ Migration safe! Apply changes to production brain')
// Apply validated migration to main brain
const users = await brain.find({ noun: 'user' })
for (const user of users) {
await brain.update(user.id, {
email: user.data.email.toLowerCase(),
verified: user.data.verified ?? false
})
}
await migration.destroy()
console.log('✅ Migration safe! Apply changes to production brain')
// Apply validated migration to main brain
const users = await brain.find({ noun: 'user' })
for (const user of users) {
await brain.update(user.id, {
email: user.data.email.toLowerCase(),
verified: user.data.verified ?? false
})
}
await migration.destroy()
} else {
console.log('❌ Migration failed! Discarding fork.')
await migration.destroy()
console.log('❌ Migration failed! Discarding fork.')
await migration.destroy()
}
```
@ -203,8 +203,8 @@ const brain = new Brainy({ storage: { adapter: 's3', bucket: 'production' } })
await brain.init()
// Create two variants
const variantA = await brain.fork('variant-a') // Control
const variantB = await brain.fork('variant-b') // Test
const variantA = await brain.fork('variant-a') // Control
const variantB = await brain.fork('variant-b') // Test
// Run different algorithms
await variantA.processWithAlgorithm('current')
@ -219,14 +219,14 @@ console.log('Variant B accuracy:', metricsB.accuracy)
// Choose winner and update main brain
if (metricsB.accuracy > metricsA.accuracy) {
console.log('B wins! Apply algorithm B to production')
await brain.processWithAlgorithm('improved')
await variantA.destroy()
await variantB.destroy()
console.log('B wins! Apply algorithm B to production')
await brain.processWithAlgorithm('improved')
await variantA.destroy()
await variantB.destroy()
} else {
console.log('A wins! Keeping current algorithm.')
await variantA.destroy()
await variantB.destroy()
console.log('A wins! Keeping current algorithm.')
await variantA.destroy()
await variantB.destroy()
}
```
@ -311,12 +311,12 @@ await brain.update(docs[0].id, { version: 2 })
// Test against original state
const originalData = await snapshot.find({ noun: 'doc' })
console.log(originalData[0].data.version) // 1 (original state!)
console.log(originalData[0].data.version) // 1 (original state!)
// Clean up
await snapshot.destroy()
// Note: Time-travel queries (asOf) planned for v5.1.0
// Note: Time-travel queries (asOf) Planned
```
---
@ -330,27 +330,27 @@ await snapshot.destroy()
const fork1 = await brain.fork('my-experiment')
// Auto-generated name (uses timestamp)
const fork2 = await brain.fork() // 'fork-1635789012345'
const fork2 = await brain.fork() // 'fork-1635789012345'
// Fork with metadata (for tracking)
const fork3 = await brain.fork('test', {
author: 'Alice',
message: 'Testing new feature'
author: 'Alice',
message: 'Testing new feature'
})
```
### Branch Management (v5.0.0)
### Branch Management
**NEW in v5.0.0:** Full branch management now available!
Full branch management now available!
```javascript
// List all branches
const branches = await brain.listBranches()
console.log(branches) // ['main', 'experiment', 'test']
console.log(branches) // ['main', 'experiment', 'test']
// Get current branch
const current = await brain.getCurrentBranch()
console.log(current) // 'main'
console.log(current) // 'main'
// Switch between branches
await brain.checkout('experiment')
@ -359,33 +359,33 @@ await brain.checkout('experiment')
await brain.deleteBranch('old-experiment')
```
### Commit Tracking (v5.0.0)
### Commit Tracking
**NEW in v5.0.0:** Git-style commit tracking!
Git-style commit tracking!
```javascript
// Create a commit (snapshot of current state)
await brain.add({ type: 'user', data: { name: 'Alice' } })
const commitHash = await brain.commit({
message: 'Add Alice user',
author: 'dev@example.com'
message: 'Add Alice user',
author: 'dev@example.com'
})
console.log(commitHash) // 'a3f2c1b9...'
console.log(commitHash) // 'a3f2c1b9...'
```
### Commit History (v5.0.0)
### Commit History
**NEW in v5.0.0:** View commit history!
View commit history!
```javascript
// Get commit history for current branch
const history = await brain.getHistory({ limit: 10 })
history.forEach(commit => {
console.log(`${commit.hash}: ${commit.message}`)
console.log(` By: ${commit.author} at ${new Date(commit.timestamp)}`)
console.log(`${commit.hash}: ${commit.message}`)
console.log(` By: ${commit.author} at ${new Date(commit.timestamp)}`)
})
## Performance Characteristics
@ -425,7 +425,7 @@ Savings: 40% less memory
## Zero Configuration
**Fork is enabled by default in v5.0.0+. No setup required.**
**Fork is enabled by default. No setup required.**
```javascript
// This is all you need:
@ -467,9 +467,9 @@ await new Brainy({ storage: { adapter: 's3', bucket: 'my-data' } }).fork()
```javascript
const brain = new Brainy({
storage: { adapter: 'memory' },
vfs: { enabled: true },
intelligence: { enabled: true }
storage: { adapter: 'memory' },
vfs: { enabled: true },
intelligence: { enabled: true }
})
await brain.init()
@ -484,9 +484,9 @@ await brain.add({ noun: 'user', data: { name: 'Alice' } })
const fork = await brain.fork('test')
// All features work on fork:
await fork.vfs.readFile('/project/README.md') // ✅ VFS
await fork.find({ noun: 'user' }) // ✅ find()
await fork.query('users named Alice') // ✅ Triple Intelligence
await fork.vfs.readFile('/project/README.md') // ✅ VFS
await fork.find({ noun: 'user' }) // ✅ find()
await fork.query('users named Alice') // ✅ Triple Intelligence
```
### Works at Billion Scale
@ -494,8 +494,8 @@ await fork.query('users named Alice') // ✅ Triple Intelligence
```javascript
// Tested at 1M entities, extrapolates to 1B
const brain = new Brainy({
storage: { adapter: 'gcs', bucket: 'billion-scale' },
hnsw: { typeAware: true } // 87% memory reduction
storage: { adapter: 'gcs', bucket: 'billion-scale' },
hnsw: { typeAware: true } // 87% memory reduction
})
await brain.init()
@ -563,7 +563,7 @@ ALTER TABLE users_backup RENAME TO users;
```javascript
const fork = await brain.fork('test')
await fork.updateAll({ email: (u) => u.email.toLowerCase() })
if (success) await brain.checkout('test') // Make test branch active
if (success) await brain.checkout('test') // Make test branch active
else await fork.destroy()
```
@ -585,7 +585,7 @@ mongoimport --db mydb --collection users_backup --file users.json
**Brainy:**
```javascript
const fork = await brain.fork() // Done!
const fork = await brain.fork() // Done!
```
**Winner: Brainy** (100x faster, built-in)
@ -612,8 +612,7 @@ const fork = await brain.fork() // Done!
## What's Implemented vs. What's Next
### ✅ Available in v5.0.0:
- ✅ `fork()` - Instant clone in <100ms
### ✅ Available in - ✅ `fork()` - Instant clone in <100ms
- ✅ `listBranches()` - List all forks
- ✅ `getCurrentBranch()` - Get active branch
- ✅ `checkout()` - Switch between branches
@ -621,14 +620,14 @@ const fork = await brain.fork() // Done!
- ✅ `commit()` - Create state snapshots
- ✅ `getHistory()` - View commit history
### 🔮 Planned for v5.1.0+:
### 🔮 Planned for:
**Temporal Features:**
- `asOf(timestamp)` - Query data at specific time (✅ v5.0.0+)
- `asOf(timestamp)` - Query data at specific time (✅ available)
- `rollback(commitHash)` - Restore to previous state
- Full audit trail for all changes
These features require additional temporal infrastructure and are being carefully designed for v5.1.0+
These features require additional temporal infrastructure and are being carefully designed
---
@ -691,4 +690,4 @@ console.log('Fork created in < 2 seconds! 🚀')
---
**Brainy v5.0.0+** | [GitHub](https://github.com/soulcraftlabs/brainy) | [npm](https://npmjs.com/package/@soulcraft/brainy)
**Brainy** | [GitHub](https://github.com/soulcraftlabs/brainy) | [npm](https://npmjs.com/package/@soulcraft/brainy)

160
docs/guides/import-anything.md
View file

@ -24,8 +24,8 @@ await brain.import(anything)
```javascript
// Array of objects? No problem.
const people = [
{ name: 'Alice', role: 'Engineer', company: 'TechCorp' },
{ name: 'Bob', role: 'Designer', company: 'TechCorp' }
{ name: 'Alice', role: 'Engineer', company: 'TechCorp' },
{ name: 'Bob', role: 'Designer', company: 'TechCorp' }
]
await brain.import(people)
@ -55,7 +55,7 @@ await brain.import('sales-report.xlsx')
// Or specific sheets only
await brain.import('data.xlsx', {
excelSheets: ['Customers', 'Orders']
excelSheets: ['Customers', 'Orders']
})
// ✨ Multi-sheet data becomes interconnected entities!
```
@ -68,7 +68,7 @@ await brain.import('research-paper.pdf')
// With table extraction
await brain.import('report.pdf', {
pdfExtractTables: true
pdfExtractTables: true
})
// ✨ Converts PDF tables to structured data automatically!
```
@ -83,16 +83,16 @@ await brain.import('config.yaml')
const yaml = `
project: AI Assistant
team:
- name: Alice
role: Lead
- name: Bob
role: Dev
- name: Alice
role: Lead
- name: Bob
role: Dev
`
await brain.import(yaml, { format: 'yaml' })
// ✨ Hierarchical data becomes a connected graph!
```
### 📄 Import Word Documents (DOCX) - v4.2.0
### 📄 Import Word Documents (DOCX) -
```javascript
// From file path
await brain.import('research-paper.docx')
@ -105,8 +105,8 @@ await brain.import(buffer, { format: 'docx' })
// With neural extraction
await brain.import('report.docx', {
enableNeuralExtraction: true,
enableHierarchicalRelationships: true
enableNeuralExtraction: true,
enableHierarchicalRelationships: true
})
// ✨ Extracts entities from paragraphs and creates relationships within sections!
```
@ -121,25 +121,25 @@ await brain.import('https://api.example.com/data.json')
await brain.import('https://data.gov/census.csv')
// ✨ Fetches CSV from web, parses, imports!
// With authentication (v4.2.0)
// With authentication
await brain.import({
type: 'url',
data: 'https://api.example.com/private/data.xlsx',
auth: {
username: 'user',
password: 'pass'
}
type: 'url',
data: 'https://api.example.com/private/data.xlsx',
auth: {
username: 'user',
password: 'pass'
}
})
// ✨ Supports basic authentication for protected resources!
// With custom headers (v4.2.0)
// With custom headers
await brain.import({
type: 'url',
data: 'https://api.example.com/data.json',
headers: {
'Authorization': 'Bearer TOKEN',
'X-API-Key': 'your-key'
}
type: 'url',
data: 'https://api.example.com/data.json',
headers: {
'Authorization': 'Bearer TOKEN',
'X-API-Key': 'your-key'
}
})
// ✨ Full HTTP header customization support!
```
@ -163,9 +163,9 @@ When you import data, Brainy:
2. **Intelligent parsing** - CSV (encoding/delimiter), Excel (multi-sheet), PDF (text/tables), DOCX (headings/paragraphs)
3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
5. **Scores confidence & weight** - Every entity and relationship gets quality metrics (v4.2.0)
5. **Scores confidence & weight** - Every entity and relationship gets quality metrics
6. **Creates embeddings** - Makes everything semantically searchable
7. **Indexes metadata** - Enables lightning-fast filtering with range queries (v4.2.0)
7. **Indexes metadata** - Enables lightning-fast filtering with range queries
## Intelligent Type Detection
@ -193,9 +193,9 @@ Brainy finds connections in your data:
```javascript
const data = [
{ id: 'u1', name: 'Alice', managerId: 'u2' },
{ id: 'u2', name: 'Bob', departmentId: 'd1' },
{ id: 'd1', name: 'Engineering' }
{ id: 'u1', name: 'Alice', managerId: 'u2' },
{ id: 'u2', name: 'Bob', departmentId: 'd1' },
{ id: 'd1', name: 'Engineering' }
]
await brain.import(data)
@ -204,28 +204,27 @@ await brain.import(data)
// - Bob "memberOf" Engineering
```
## Confidence & Weight Scoring - v4.2.0
## Confidence & Weight Scoring -
Every entity and relationship gets confidence and weight scores:
```javascript
// Import with confidence threshold
await brain.import(data, {
confidenceThreshold: 0.8 // Only extract entities with >80% confidence
confidenceThreshold: 0.8 // Only extract entities with >80% confidence
})
// Query high-confidence entities using range queries
const highConfidence = await brain.find({
where: {
confidence: { gte: 0.8 } // Get entities with confidence >= 0.8
}
where: {
confidence: { gte: 0.8 } // Get entities with confidence >= 0.8
}
})
// Range query operators: gt, gte, lt, lte, between
const mediumConfidence = await brain.find({
where: {
confidence: { between: [0.6, 0.8] }
}
where: {
confidence: { between: [0.6, 0.8] }
}
})
```
@ -236,24 +235,23 @@ const mediumConfidence = await brain.find({
**Weights** indicate importance/relevance within the document context.
## Per-Sheet Excel Extraction - v4.2.0
## Per-Sheet Excel Extraction -
Excel files with multiple sheets can be organized by sheet:
```javascript
// Group entities by sheet in VFS
await brain.import('multi-sheet-data.xlsx', {
groupBy: 'sheet' // Creates separate directories for each sheet
groupBy: 'sheet' // Creates separate directories for each sheet
})
// Result VFS structure:
// /imports/data/
// ├── Sheet1/
// │ ├── entity1.json
// │ └── entity2.json
// └── Sheet2/
// ├── entity3.json
// └── entity4.json
// ├── Sheet1/
// │ ├── entity1.json
// │ └── entity2.json
// └── Sheet2/
// ├── entity3.json
// └── entity4.json
// Other groupBy options:
// - 'type': Group by entity type (Person, Place, etc.)
@ -274,9 +272,9 @@ const results = await brain.find('people in engineering who joined this year')
// Graph traversal + filters
const connected = await brain.find({
like: 'Alice',
connected: { depth: 2 },
where: { department: 'Engineering' }
like: 'Alice',
connected: { depth: 2 },
where: { department: 'Engineering' }
})
```
@ -286,37 +284,37 @@ Everything works with zero config, but you can customize:
```javascript
await brain.import(data, {
// Format detection
format: 'excel', // Force specific format (auto-detected if not specified)
// Format detection
format: 'excel', // Force specific format (auto-detected if not specified)
// VFS & Organization
vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified)
groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
preserveSource: true, // Keep original source file in VFS (default: true)
// VFS & Organization
vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified)
groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
preserveSource: true, // Keep original source file in VFS (default: true)
// Entity & Relationship Creation
createEntities: true, // Create entities in knowledge graph (default: true)
createRelationships: true, // Create relationships in knowledge graph (default: true)
// Entity & Relationship Creation
createEntities: true, // Create entities in knowledge graph (default: true)
createRelationships: true, // Create relationships in knowledge graph (default: true)
// Neural Intelligence
enableNeuralExtraction: true, // Use AI to extract entities (default: true)
enableRelationshipInference: true, // Use AI to infer relationships (default: true)
enableConceptExtraction: true, // Extract concepts from text (default: true)
confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6)
// Neural Intelligence
enableNeuralExtraction: true, // Use AI to extract entities (default: true)
enableRelationshipInference: true, // Use AI to infer relationships (default: true)
enableConceptExtraction: true, // Extract concepts from text (default: true)
confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6)
// Deduplication
enableDeduplication: true, // Check for duplicate entities (default: true)
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
// Note: Auto-disabled for imports >100 entities
// Deduplication
enableDeduplication: true, // Check for duplicate entities (default: true)
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
// Note: Auto-disabled for imports >100 entities
// Performance
chunkSize: 100, // Batch size for processing (default: varies by operation)
// Performance
chunkSize: 100, // Batch size for processing (default: varies by operation)
// History & Progress
enableHistory: true, // Track import history (default: true)
onProgress: (progress) => { // Progress callback
console.log(progress.stage, progress.message)
}
// History & Progress
enableHistory: true, // Track import history (default: true)
onProgress: (progress) => { // Progress callback
console.log(progress.stage, progress.message)
}
})
```
@ -343,9 +341,9 @@ const results = await brain.import(problematicData)
### 🏢 Business Data
```javascript
// Import ANY source - ONE method!
await brain.import('customers.csv') // File
await brain.import('https://api.co/orders') // URL
await brain.import(productsArray) // Data
await brain.import('customers.csv') // File
await brain.import('https://api.co/orders') // URL
await brain.import(productsArray) // Data
// Now query across all of it!
await brain.find('customers who bought products in Q4')
@ -389,8 +387,8 @@ await brain.find('posts by users following Alice with >10 comments')
```javascript
// ONE method that understands EVERYTHING:
await brain.import(data) // Objects, arrays, strings
await brain.import('file.csv') // Files (auto-detected)
await brain.import(data) // Objects, arrays, strings
await brain.import('file.csv') // Files (auto-detected)
await brain.import('http://..') // URLs (auto-fetched)
// It ALWAYS knows what to do! ✨

4
docs/guides/import-flow.md
View file

@ -39,7 +39,7 @@ Each phase adds intelligence and structure to your raw data, transforming it int
---
## 🌊 Always-On Streaming Architecture (v4.2.0+)
## 🌊 Always-On Streaming Architecture
All imports use streaming with **progressive flush intervals**:
@ -1062,7 +1062,7 @@ if (!fromEntity || !toEntity) {
}
```
#### 5.3b: Check for Duplicates (v3.43.2 Critical Fix)
#### 5.3b: Check for Duplicates (Critical Fix)
**The Bug**: Without duplicate checking, re-importing would create:
```

278
docs/guides/import-progress-examples.md
View file

@ -2,7 +2,7 @@
**How to Use Progress Tracking in Your Applications**
Brainy v4.5.0+ provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
Brainy provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
> **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation.
@ -20,12 +20,12 @@ const brain = await Brainy.create()
// Import with progress tracking
const result = await brain.import(fs.readFileSync('large-file.xlsx'), {
onProgress: (progress) => {
console.log(`Progress: ${progress.stage}`)
console.log(` Message: ${progress.message}`)
console.log(` Entities: ${progress.entities || 0}`)
console.log(` Relationships: ${progress.relationships || 0}`)
}
onProgress: (progress) => {
console.log(`Progress: ${progress.stage}`)
console.log(` Message: ${progress.message}`)
console.log(` Entities: ${progress.entities || 0}`)
console.log(` Relationships: ${progress.relationships || 0}`)
}
})
console.log(`Import complete: ${result.entities.length} entities created`)
@ -34,30 +34,30 @@ console.log(`Import complete: ${result.entities.length} entities created`)
**Expected Output:**
```
Progress: detecting
Message: Detecting format...
Entities: 0
Relationships: 0
Message: Detecting format...
Entities: 0
Relationships: 0
Progress: extracting
Message: Loading Excel workbook...
Entities: 0
Relationships: 0
Message: Loading Excel workbook...
Entities: 0
Relationships: 0
Progress: extracting
Message: Reading sheet: Sales (1/3)
Entities: 0
Relationships: 0
Message: Reading sheet: Sales (1/3)
Entities: 0
Relationships: 0
Progress: extracting
Message: Parsing Excel (33%)
Entities: 0
Relationships: 0
Message: Parsing Excel (33%)
Entities: 0
Relationships: 0
Progress: extracting
Message: Reading sheet: Products (2/3)
Entities: 0
Relationships: 0
Message: Reading sheet: Products (2/3)
Entities: 0
Relationships: 0
... (more progress updates)
Progress: complete
Message: Import complete
Entities: 1523
Relationships: 892
Message: Import complete
Entities: 1523
Relationships: 892
Import complete: 1523 entities created
```
@ -70,19 +70,19 @@ The examples below show format-specific messages, but **you don't need format-sp
```typescript
// ONE HANDLER FOR ALL FORMATS!
function universalProgressHandler(progress) {
console.log(`[${progress.stage}] ${progress.message}`)
console.log(`[${progress.stage}] ${progress.message}`)
if (progress.processed && progress.total) {
console.log(` Progress: ${progress.processed}/${progress.total}`)
}
if (progress.processed && progress.total) {
console.log(` Progress: ${progress.processed}/${progress.total}`)
}
if (progress.entities || progress.relationships) {
console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
}
if (progress.entities || progress.relationships) {
console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
}
if (progress.throughput && progress.eta) {
console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
}
if (progress.throughput && progress.eta) {
console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
}
}
// Use it for ANY format!
@ -105,20 +105,20 @@ The examples below show **what messages look like** for different formats using
```typescript
await brain.import(csvBuffer, {
format: 'csv',
onProgress: (progress) => {
if (progress.stage === 'extracting') {
// CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
console.log(progress.message)
}
}
format: 'csv',
onProgress: (progress) => {
if (progress.stage === 'extracting') {
// CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
console.log(progress.message)
}
}
})
```
**CSV Progress Messages:**
- ✅ "Detecting CSV encoding and delimiter..."
- ✅ "Parsing CSV rows (delimiter: ",")"
- ✅ "Parsed 75%" (via bytes processed)
- ✅ "Parsed 75%" (via bytes processed)
- ✅ "Extracted 1000 rows"
- ✅ "Converting types: 5000/10000 rows..."
- ✅ "CSV processing complete: 10000 rows"
@ -129,12 +129,12 @@ await brain.import(csvBuffer, {
```typescript
await brain.import(pdfBuffer, {
format: 'pdf',
onProgress: (progress) => {
// PDF reports exact page numbers
console.log(progress.message)
// Example: "Processing page 5 of 23"
}
format: 'pdf',
onProgress: (progress) => {
// PDF reports exact page numbers
console.log(progress.message)
// Example: "Processing page 5 of 23"
}
})
```
@ -152,12 +152,12 @@ await brain.import(pdfBuffer, {
```typescript
await brain.import(excelBuffer, {
format: 'excel',
onProgress: (progress) => {
// Excel reports sheet names
console.log(progress.message)
// Example: "Reading sheet: Q2 Sales (2/5)"
}
format: 'excel',
onProgress: (progress) => {
// Excel reports sheet names
console.log(progress.message)
// Example: "Reading sheet: Q2 Sales (2/5)"
}
})
```
@ -175,11 +175,11 @@ await brain.import(excelBuffer, {
```typescript
await brain.import(jsonBuffer, {
format: 'json',
onProgress: (progress) => {
// JSON reports every 10 nodes
console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
}
format: 'json',
onProgress: (progress) => {
// JSON reports every 10 nodes
console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
}
})
```
@ -189,10 +189,10 @@ await brain.import(jsonBuffer, {
```typescript
await brain.import(markdownString, {
format: 'markdown',
onProgress: (progress) => {
console.log(`Section ${progress.processed}/${progress.total}`)
}
format: 'markdown',
onProgress: (progress) => {
console.log(`Section ${progress.processed}/${progress.total}`)
}
})
```
@ -204,44 +204,44 @@ await brain.import(markdownString, {
```typescript
function ImportProgress({ file }: { file: File }) {
const [progress, setProgress] = useState({
stage: 'idle',
message: '',
percent: 0,
entities: 0,
relationships: 0
})
const [progress, setProgress] = useState({
stage: 'idle',
message: '',
percent: 0,
entities: 0,
relationships: 0
})
const handleImport = async () => {
const buffer = await file.arrayBuffer()
const handleImport = async () => {
const buffer = await file.arrayBuffer()
await brain.import(Buffer.from(buffer), {
onProgress: (p) => {
setProgress({
stage: p.stage,
message: p.message,
// Estimate percentage from stage
percent: {
detecting: 10,
extracting: 50,
'storing-vfs': 80,
'storing-graph': 90,
complete: 100
}[p.stage] || 0,
entities: p.entities || 0,
relationships: p.relationships || 0
})
}
})
}
await brain.import(Buffer.from(buffer), {
onProgress: (p) => {
setProgress({
stage: p.stage,
message: p.message,
// Estimate percentage from stage
percent: {
detecting: 10,
extracting: 50,
'storing-vfs': 80,
'storing-graph': 90,
complete: 100
}[p.stage] || 0,
entities: p.entities || 0,
relationships: p.relationships || 0
})
}
})
}
return (
<div>
<ProgressBar value={progress.percent} />
<p>{progress.message}</p>
<p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
</div>
)
return (
<div>
<ProgressBar value={progress.percent} />
<p>{progress.message}</p>
<p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
</div>
)
}
```
@ -255,13 +255,13 @@ import ora from 'ora'
const spinner = ora('Starting import...').start()
await brain.import(buffer, {
onProgress: (progress) => {
spinner.text = progress.message
onProgress: (progress) => {
spinner.text = progress.message
if (progress.stage === 'complete') {
spinner.succeed(`Import complete: ${progress.entities} entities`)
}
}
if (progress.stage === 'complete') {
spinner.succeed(`Import complete: ${progress.entities} entities`)
}
}
})
```
@ -285,20 +285,20 @@ let startTime = Date.now()
let lastUpdate = startTime
await brain.import(buffer, {
onProgress: (progress) => {
const elapsed = Date.now() - startTime
const rate = progress.entities / (elapsed / 1000) // entities/sec
onProgress: (progress) => {
const elapsed = Date.now() - startTime
const rate = progress.entities / (elapsed / 1000) // entities/sec
console.clear()
console.log('Import Progress Dashboard')
console.log('========================')
console.log(`Stage: ${progress.stage}`)
console.log(`Status: ${progress.message}`)
console.log(`Entities: ${progress.entities}`)
console.log(`Relationships: ${progress.relationships}`)
console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
}
console.clear()
console.log('Import Progress Dashboard')
console.log('========================')
console.log(`Stage: ${progress.stage}`)
console.log(`Status: ${progress.message}`)
console.log(`Entities: ${progress.entities}`)
console.log(`Relationships: ${progress.relationships}`)
console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
}
})
```
@ -310,21 +310,21 @@ await brain.import(buffer, {
```typescript
const formatMessages = {
csv: (p) => `CSV: ${p.message}`,
pdf: (p) => `PDF: ${p.message}`,
excel: (p) => `Excel: ${p.message}`,
json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
yaml: (p) => `YAML: ${p.processed} nodes`,
docx: (p) => `DOCX: ${p.processed} paragraphs`
csv: (p) => `CSV: ${p.message}`,
pdf: (p) => `PDF: ${p.message}`,
excel: (p) => `Excel: ${p.message}`,
json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
yaml: (p) => `YAML: ${p.processed} nodes`,
docx: (p) => `DOCX: ${p.processed} paragraphs`
}
await brain.import(buffer, {
onProgress: (progress) => {
// Format is available in progress.stage metadata
const message = formatMessages[detectedFormat]?.(progress) || progress.message
console.log(message)
}
onProgress: (progress) => {
// Format is available in progress.stage metadata
const message = formatMessages[detectedFormat]?.(progress) || progress.message
console.log(message)
}
})
```
@ -336,18 +336,18 @@ await brain.import(buffer, {
```typescript
let lastUIUpdate = 0
const THROTTLE_MS = 100 // Update UI max once per 100ms
const THROTTLE_MS = 100 // Update UI max once per 100ms
await brain.import(buffer, {
onProgress: (progress) => {
const now = Date.now()
if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
return // Skip this update
}
onProgress: (progress) => {
const now = Date.now()
if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
return // Skip this update
}
lastUIUpdate = now
updateUI(progress) // Only update every 100ms
}
lastUIUpdate = now
updateUI(progress) // Only update every 100ms
}
})
```

664
docs/guides/import-progress-implementation.md
View file

@ -28,17 +28,17 @@
```typescript
// THE PUBLIC API - Same for ALL 7 formats!
brain.import(buffer, {
onProgress: (progress: ImportProgress) => {
// These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
progress.message // Human-readable status (varies by format, always readable)
progress.processed // Items processed (optional)
progress.total // Total items (optional)
progress.entities // Entities extracted (optional)
progress.relationships // Relationships inferred (optional)
progress.throughput // Items/sec (optional, during extraction)
progress.eta // Time remaining in ms (optional)
}
onProgress: (progress: ImportProgress) => {
// These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
progress.message // Human-readable status (varies by format, always readable)
progress.processed // Items processed (optional)
progress.total // Total items (optional)
progress.entities // Entities extracted (optional)
progress.relationships // Relationships inferred (optional)
progress.throughput // Items/sec (optional, during extraction)
progress.eta // Time remaining in ms (optional)
}
})
```
@ -49,14 +49,14 @@ The table below shows how formats implement progress *internally*. Normal develo
```typescript
// Internal: Binary formats use handler hooks (you added these!)
interface FormatHandlerProgressHooks {
onBytesProcessed?: (bytes: number) => void
onCurrentItem?: (message: string) => void
onDataExtracted?: (count: number, total?: number) => void
onBytesProcessed?: (bytes: number) => void
onCurrentItem?: (message: string) => void
onDataExtracted?: (count: number, total?: number) => void
}
// Internal: Text formats use importer callbacks
interface ImporterProgressCallback {
onProgress?: (stats: { processed, total, entities, relationships }) => void
onProgress?: (stats: { processed, total, entities, relationships }) => void
}
// Both are converted to ImportProgress by ImportCoordinator!
@ -74,7 +74,7 @@ interface ImporterProgressCallback {
## 🎯 Overview
As of v4.5.0, Brainy supports comprehensive, multi-dimensional progress tracking for imports:
Brainy supports comprehensive, multi-dimensional progress tracking for imports:
- **Bytes processed** (always available, most deterministic)
- **Entities extracted** (AI extraction phase)
- **Stage-specific metrics** (parsing: MB/s, extraction: entities/s)
@ -91,23 +91,23 @@ All handlers follow a simple, consistent pattern using **progress hooks**.
```typescript
export interface FormatHandlerProgressHooks {
/**
* Report bytes processed
* Call this as you read/parse the file
*/
onBytesProcessed?: (bytes: number) => void
/**
* Report bytes processed
* Call this as you read/parse the file
*/
onBytesProcessed?: (bytes: number) => void
/**
* Set current processing context
* Examples: "Processing page 5", "Reading sheet: Q2 Sales"
*/
onCurrentItem?: (item: string) => void
/**
* Set current processing context
* Examples: "Processing page 5", "Reading sheet: Q2 Sales"
*/
onCurrentItem?: (item: string) => void
/**
* Report structured data extraction progress
* Examples: "Extracted 100 rows", "Parsed 50 paragraphs"
*/
onDataExtracted?: (count: number, total?: number) => void
/**
* Report structured data extraction progress
* Examples: "Extracted 100 rows", "Parsed 50 paragraphs"
*/
onDataExtracted?: (count: number, total?: number) => void
}
```
@ -117,19 +117,19 @@ Progress hooks are automatically passed to your handler via `FormatHandlerOption
```typescript
export interface FormatHandlerOptions {
// ... existing options ...
// ... existing options ...
/**
* Progress hooks (v4.5.0)
* Handlers call these to report progress during processing
*/
progressHooks?: FormatHandlerProgressHooks
/**
* Progress hooks
* Handlers call these to report progress during processing
*/
progressHooks?: FormatHandlerProgressHooks
/**
* Total file size in bytes (v4.5.0)
* Used for progress percentage calculation
*/
totalBytes?: number
/**
* Total file size in bytes
* Used for progress percentage calculation
*/
totalBytes?: number
}
```
@ -141,36 +141,36 @@ Every handler follows these 5 steps:
```typescript
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
const progressHooks = options.progressHooks // Step 1: Get hooks
const progressHooks = options.progressHooks // Step 1: Get hooks
// Step 2: Report initial progress
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Starting import...')
}
// Step 2: Report initial progress
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Starting import...')
}
// Step 3: Report bytes as you process
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0) // Start
}
// Step 3: Report bytes as you process
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0) // Start
}
// ... do parsing ...
// ... do parsing ...
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(buffer.length) // Complete
}
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(buffer.length) // Complete
}
// Step 4: Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length)
}
// Step 4: Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length)
}
// Step 5: Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Complete: ${data.length} items processed`)
}
// Step 5: Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Complete: ${data.length} items processed`)
}
return { format, data, metadata }
return { format, data, metadata }
}
```
@ -182,76 +182,76 @@ Here's the **ACTUAL implementation** from CSV handler showing all the key progre
```typescript
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
const startTime = Date.now()
const progressHooks = options.progressHooks // ✅ Step 1
const startTime = Date.now()
const progressHooks = options.progressHooks // ✅ Step 1
// Convert to buffer if string
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
const totalBytes = buffer.length
// Convert to buffer if string
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
const totalBytes = buffer.length
// ✅ Step 2: Report start
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...')
}
// ✅ Step 2: Report start
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...')
}
// Detect encoding
const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer)
const text = buffer.toString(detectedEncoding as BufferEncoding)
// Detect encoding
const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer)
const text = buffer.toString(detectedEncoding as BufferEncoding)
// Detect delimiter
const delimiter = options.csvDelimiter || this.detectDelimiter(text)
// Detect delimiter
const delimiter = options.csvDelimiter || this.detectDelimiter(text)
// ✅ Progress update: Parsing phase
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
}
// ✅ Progress update: Parsing phase
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
}
// Parse CSV
const records = parse(text, { /* options */ })
// Parse CSV
const records = parse(text, { /* options */ })
// ✅ Step 3: Report bytes processed (entire file parsed)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
// ✅ Step 3: Report bytes processed (entire file parsed)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
const data = Array.isArray(records) ? records : [records]
const data = Array.isArray(records) ? records : [records]
// ✅ Step 4: Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`)
}
// ✅ Step 4: Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`)
}
// Type inference and conversion
const fields = data.length > 0 ? Object.keys(data[0]) : []
const types = this.inferFieldTypes(data)
// Type inference and conversion
const fields = data.length > 0 ? Object.keys(data[0]) : []
const types = this.inferFieldTypes(data)
const convertedData = data.map((row, index) => {
const converted = this.convertRow(row, types)
const convertedData = data.map((row, index) => {
const converted = this.convertRow(row, types)
// ✅ Progress update every 1000 rows (avoid spam)
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
}
// ✅ Progress update every 1000 rows (avoid spam)
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
}
return converted
})
return converted
})
// ✅ Step 5: Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
}
// ✅ Step 5: Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
}
return {
format: this.format,
data: convertedData,
metadata: { /* ... */ }
}
return {
format: this.format,
data: convertedData,
metadata: { /* ... */ }
}
}
```
@ -292,51 +292,51 @@ Brainy supports **7 file formats** with full progress tracking:
```typescript
async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
const progressHooks = options.progressHooks
const totalBytes = data.length
const progressHooks = options.progressHooks
const totalBytes = data.length
// Report start
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Loading PDF document...')
}
// Report start
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Loading PDF document...')
}
const pdfDoc = await loadPDF(data)
const totalPages = pdfDoc.numPages
const pdfDoc = await loadPDF(data)
const totalPages = pdfDoc.numPages
const extractedData: any[] = []
let bytesProcessed = 0
const extractedData: any[] = []
let bytesProcessed = 0
for (let pageNum = 1; pageNum <= totalPages; pageNum++) {
// ✅ Report current page
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`)
}
for (let pageNum = 1; pageNum <= totalPages; pageNum++) {
// ✅ Report current page
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`)
}
const page = await pdfDoc.getPage(pageNum)
const text = await page.getTextContent()
extractedData.push(this.processPageText(text))
const page = await pdfDoc.getPage(pageNum)
const text = await page.getTextContent()
extractedData.push(this.processPageText(text))
// ✅ Estimate bytes processed (pages are sequential)
bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// ✅ Estimate bytes processed (pages are sequential)
bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// ✅ Report extraction progress
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(pageNum, totalPages)
}
}
// ✅ Report extraction progress
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(pageNum, totalPages)
}
}
// Final progress
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`)
}
// Final progress
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`)
}
return { format: 'pdf', data: extractedData, metadata: { /* ... */ } }
return { format: 'pdf', data: extractedData, metadata: { /* ... */ } }
}
```
@ -344,55 +344,55 @@ async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedDat
```typescript
async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
const progressHooks = options.progressHooks
const totalBytes = data.length
const progressHooks = options.progressHooks
const totalBytes = data.length
// Load workbook
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Loading Excel workbook...')
}
// Load workbook
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Loading Excel workbook...')
}
const workbook = XLSX.read(data)
const sheetNames = options.excelSheets === 'all'
? workbook.SheetNames
: (options.excelSheets || [workbook.SheetNames[0]])
const workbook = XLSX.read(data)
const sheetNames = options.excelSheets === 'all'
? workbook.SheetNames
: (options.excelSheets || [workbook.SheetNames[0]])
const allData: any[] = []
let bytesProcessed = 0
const allData: any[] = []
let bytesProcessed = 0
for (let i = 0; i < sheetNames.length; i++) {
const sheetName = sheetNames[i]
for (let i = 0; i < sheetNames.length; i++) {
const sheetName = sheetNames[i]
// ✅ Report current sheet
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`)
}
// ✅ Report current sheet
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`)
}
const sheet = workbook.Sheets[sheetName]
const sheetData = XLSX.utils.sheet_to_json(sheet)
allData.push(...sheetData)
const sheet = workbook.Sheets[sheetName]
const sheetData = XLSX.utils.sheet_to_json(sheet)
allData.push(...sheetData)
// ✅ Estimate bytes processed (sheets processed sequentially)
bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// ✅ Estimate bytes processed (sheets processed sequentially)
bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// ✅ Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done
}
}
// ✅ Report data extraction
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done
}
}
// Final progress
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`)
}
// Final progress
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`)
}
return { format: 'xlsx', data: allData, metadata: { /* ... */ } }
return { format: 'xlsx', data: allData, metadata: { /* ... */ } }
}
```
@ -400,44 +400,44 @@ async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedDat
```typescript
async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResult> {
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Parse JSON if string
let jsonData = typeof data === 'string' ? JSON.parse(data) : data
// Parse JSON if string
let jsonData = typeof data === 'string' ? JSON.parse(data) : data
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Traverse and extract (reports progress every 10 nodes)
const entities: ExtractedJSONEntity[] = []
const relationships: ExtractedJSONRelationship[] = []
let nodesProcessed = 0
// Traverse and extract (reports progress every 10 nodes)
const entities: ExtractedJSONEntity[] = []
const relationships: ExtractedJSONRelationship[] = []
let nodesProcessed = 0
await this.traverseJSON(
jsonData,
entities,
relationships,
() => {
nodesProcessed++
if (nodesProcessed % 10 === 0) {
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
}
}
)
await this.traverseJSON(
jsonData,
entities,
relationships,
() => {
nodesProcessed++
if (nodesProcessed % 10 === 0) {
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
}
}
)
// ✅ Report completion
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
// ✅ Report completion
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
return { nodesProcessed, entitiesExtracted: entities.length, ... }
return { nodesProcessed, entitiesExtracted: entities.length, ... }
}
```
@ -445,38 +445,38 @@ async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResul
```typescript
async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<SmartMarkdownResult> {
// ✅ Report parsing start
options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 })
// ✅ Report parsing start
options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 })
// Parse markdown into sections
const parsedSections = this.parseMarkdown(markdown, options)
// Parse markdown into sections
const parsedSections = this.parseMarkdown(markdown, options)
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 })
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 })
// Process each section (reports progress after each section)
const sections: MarkdownSection[] = []
for (let i = 0; i < parsedSections.length; i++) {
const section = await this.processSection(parsedSections[i], options)
sections.push(section)
// Process each section (reports progress after each section)
const sections: MarkdownSection[] = []
for (let i = 0; i < parsedSections.length; i++) {
const section = await this.processSection(parsedSections[i], options)
sections.push(section)
options.onProgress?.({
processed: i + 1,
total: parsedSections.length,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
})
}
options.onProgress?.({
processed: i + 1,
total: parsedSections.length,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
})
}
// ✅ Report completion
options.onProgress?.({
processed: sections.length,
total: sections.length,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
})
// ✅ Report completion
options.onProgress?.({
processed: sections.length,
total: sections.length,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
})
return { sectionsProcessed: sections.length, ... }
return { sectionsProcessed: sections.length, ... }
}
```
@ -484,27 +484,27 @@ async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<Sma
```typescript
async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Promise<SmartYAMLResult> {
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Parse YAML
const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8')
const data = yaml.load(yamlString)
// Parse YAML
const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8')
const data = yaml.load(yamlString)
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Traverse YAML structure (reports progress every 10 nodes)
// ... similar to JSON traversal ...
// Traverse YAML structure (reports progress every 10 nodes)
// ... similar to JSON traversal ...
// ✅ Report completion (already implemented)
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
// ✅ Report completion (already implemented)
options.onProgress?.({
processed: nodesProcessed,
entities: entities.length,
relationships: relationships.length
})
return { nodesProcessed, entitiesExtracted: entities.length, ... }
return { nodesProcessed, entitiesExtracted: entities.length, ... }
}
```
@ -512,39 +512,39 @@ async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Pro
```typescript
async extract(buffer: Buffer, options: SmartDOCXOptions = {}): Promise<SmartDOCXResult> {
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Extract text and HTML using Mammoth
const textResult = await mammoth.extractRawText({ buffer })
const htmlResult = await mammoth.convertToHtml({ buffer })
// Extract text and HTML using Mammoth
const textResult = await mammoth.extractRawText({ buffer })
const htmlResult = await mammoth.convertToHtml({ buffer })
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Process paragraphs (reports progress every 10 paragraphs)
const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength)
// Process paragraphs (reports progress every 10 paragraphs)
const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength)
for (let i = 0; i < paragraphs.length; i++) {
await this.processParagraph(paragraphs[i])
for (let i = 0; i < paragraphs.length; i++) {
await this.processParagraph(paragraphs[i])
if (i % 10 === 0) {
options.onProgress?.({
processed: i + 1,
entities: entities.length,
relationships: relationships.length
})
}
}
if (i % 10 === 0) {
options.onProgress?.({
processed: i + 1,
entities: entities.length,
relationships: relationships.length
})
}
}
// ✅ Report completion (already implemented)
options.onProgress?.({
processed: paragraphs.length,
entities: entities.length,
relationships: relationships.length
})
// ✅ Report completion (already implemented)
options.onProgress?.({
processed: paragraphs.length,
entities: entities.length,
relationships: relationships.length
})
return { paragraphsProcessed: paragraphs.length, ... }
return { paragraphsProcessed: paragraphs.length, ... }
}
```
@ -559,20 +559,20 @@ Progress hooks are **optional**. Always check before calling:
```typescript
// ✅ Good - safe
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytes)
progressHooks.onBytesProcessed(bytes)
}
// ❌ Bad - will crash if hooks undefined
progressHooks.onBytesProcessed(bytes) // TypeError!
progressHooks.onBytesProcessed(bytes) // TypeError!
```
### 2. Report Bytes at Start and End
```typescript
// ✅ Good - clear start and end
progressHooks?.onBytesProcessed(0) // Start
progressHooks?.onBytesProcessed(0) // Start
// ... processing ...
progressHooks?.onBytesProcessed(totalBytes) // End
progressHooks?.onBytesProcessed(totalBytes) // End
// ❌ Bad - no clear boundaries
// ... just start processing without reporting start
@ -583,17 +583,17 @@ progressHooks?.onBytesProcessed(totalBytes) // End
```typescript
// ✅ Good - report every 1000 items
for (let i = 0; i < items.length; i++) {
processItem(items[i])
processItem(items[i])
if (i > 0 && i % 1000 === 0) {
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`)
}
if (i > 0 && i % 1000 === 0) {
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`)
}
}
// ❌ Bad - report EVERY item (spam!)
for (let i = 0; i < items.length; i++) {
processItem(items[i])
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks!
processItem(items[i])
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks!
}
```
@ -614,13 +614,13 @@ progressHooks?.onCurrentItem('Working...')
```typescript
// ✅ Good - total known
progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows
progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows
// ✅ Also good - total unknown (streaming)
progressHooks?.onDataExtracted(100, undefined) // 100 rows so far
progressHooks?.onDataExtracted(100, undefined) // 100 rows so far
// ✅ Also good - complete
progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows
progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows
```
---
@ -637,18 +637,18 @@ const handler = new CSVHandler()
const data = fs.readFileSync('./test.csv')
const result = await handler.process(data, {
filename: 'test.csv',
progressHooks: {
onBytesProcessed: (bytes) => {
console.log(`Bytes: ${bytes}`)
},
onCurrentItem: (item) => {
console.log(`Status: ${item}`)
},
onDataExtracted: (count, total) => {
console.log(`Extracted: ${count}${total ? `/${total}` : ''}`)
}
}
filename: 'test.csv',
progressHooks: {
onBytesProcessed: (bytes) => {
console.log(`Bytes: ${bytes}`)
},
onCurrentItem: (item) => {
console.log(`Status: ${item}`)
},
onDataExtracted: (count, total) => {
console.log(`Extracted: ${count}${total ? `/${total}` : ''}`)
}
}
})
console.log(`Complete: ${result.data.length} rows`)
@ -673,24 +673,24 @@ Complete: 1000 rows
```
User Imports File
↓
↓
ImportManager
↓
↓
Creates ProgressTracker
↓
↓
Calls Handler.process() with progressHooks
↓
↓
Handler Reports Progress:
├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated
├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated
├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated
├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated
└─ onCurrentItem("Complete") → ProgressTracker → final progress
↓
├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated
├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated
├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated
├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated
└─ onCurrentItem("Complete") → ProgressTracker → final progress
↓
ProgressTracker emits to callback (throttled 100ms)
↓
↓
User sees:
"Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..."
"Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..."
```
---

4
docs/guides/import-quick-reference.md
View file

@ -91,7 +91,7 @@ await brain.import(file, {
})
```
### Import Tracking (v4.10.0+)
### Import Tracking
Track and organize imports by project:
@ -149,7 +149,7 @@ await brain.import(file, {
})
```
### Always-On Streaming (v4.2.0+)
### Always-On Streaming
All imports use streaming with adaptive flush intervals. Query data as it's imported:

2
docs/guides/standard-import-progress.md
View file

@ -65,7 +65,7 @@ interface ImportProgress {
/** Estimated time remaining (milliseconds) */
eta?: number
/** Whether data is queryable at this point (v4.2.0+) */
/** Whether data is queryable at this point */
queryable?: boolean
}
```

4
docs/guides/streaming-imports.md
View file

@ -300,7 +300,7 @@ interface ImportProgress {
relationships?: number // Relationships inferred so far
/**
* Whether data is queryable (v4.2.0+)
* Whether data is queryable
*
* true = Indexes flushed, queries will be fast and complete
* false/undefined = Data in storage but indexes not flushed yet
@ -376,7 +376,7 @@ No changes required! Streaming is now always enabled with optimal defaults:
// Before (v3.x, v4.0, v4.1): Works the same
await brain.import(file)
// After (v4.2.0+): Streaming always on, zero config
// After: Streaming always on, zero config
await brain.import(file)
```

397
docs/operations/capacity-planning.md
View file

@ -1,6 +1,6 @@
# Capacity Planning & Operations Guide
**Brainy v3.36.0+ Enterprise Operations**
**Brainy Enterprise Operations**
This guide provides production-ready capacity planning formulas, deployment strategies, and operational guidelines for scaling Brainy from development (2GB) to enterprise (128GB+) deployments.
@ -24,13 +24,13 @@ Where:
### Adaptive Caching Strategy
```
estimatedVectorMemory = entityCount × 1536 bytes // 384 dims × 4 bytes per float
hnswCacheBudget = cacheSize × 0.80 // 80% threshold for preloading decision
estimatedVectorMemory = entityCount × 1536 bytes // 384 dims × 4 bytes per float
hnswCacheBudget = cacheSize × 0.80 // 80% threshold for preloading decision
if estimatedVectorMemory < hnswCacheBudget:
cachingStrategy = 'preloaded' // All vectors loaded at init
cachingStrategy = 'preloaded' // All vectors loaded at init
else:
cachingStrategy = 'on-demand' // Vectors loaded adaptively via UnifiedCache
cachingStrategy = 'on-demand' // Vectors loaded adaptively via UnifiedCache
```
---
@ -46,19 +46,19 @@ else:
**Memory Breakdown:**
```
System Memory: 2048 MB
OS Reserved (20%): -410 MB
Available: 1638 MB
Model Memory: -140 MB
├─ WASM + Weights: 90 MB
└─ Workspace: 50 MB
System Memory: 2048 MB
OS Reserved (20%): -410 MB
Available: 1638 MB
Model Memory: -140 MB
├─ WASM + Weights: 90 MB
└─ Workspace: 50 MB
───────────────────────────
Available for Cache: 1488 MB
Available for Cache: 1488 MB
Dev Allocation (25%): 372 MB UnifiedCache
├─ HNSW (30%): 112 MB
├─ Metadata (40%): 149 MB
├─ Search (20%): 74 MB
└─ Shared (10%): 37 MB
├─ HNSW (30%): 112 MB
├─ Metadata (40%): 149 MB
├─ Search (20%): 74 MB
└─ Shared (10%): 37 MB
```
**Capacity:**
@ -75,9 +75,9 @@ Dev Allocation (25%): 372 MB UnifiedCache
**Configuration:**
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', path: './brainy-data' },
model: { precision: 'q8' },
cache: { /* auto-sized to 372MB */ }
storage: { type: 'filesystem', path: './brainy-data' },
model: { precision: 'q8' },
cache: { /* auto-sized to 372MB */ }
})
```
@ -92,17 +92,17 @@ const brain = new Brainy({
**Memory Breakdown:**
```
System Memory: 8192 MB
OS Reserved (20%): -1638 MB
Available: 6554 MB
Model Memory (Q8): -150 MB
System Memory: 8192 MB
OS Reserved (20%): -1638 MB
Available: 6554 MB
Model Memory (Q8): -150 MB
───────────────────────────
Available for Cache: 6404 MB
Available for Cache: 6404 MB
Prod Allocation (50%): 3202 MB UnifiedCache
├─ HNSW (30%): 961 MB
├─ Metadata (40%): 1281 MB
├─ Search (20%): 640 MB
└─ Shared (10%): 320 MB
├─ HNSW (30%): 961 MB
├─ Metadata (40%): 1281 MB
├─ Search (20%): 640 MB
└─ Shared (10%): 320 MB
```
**Capacity:**
@ -119,9 +119,9 @@ Prod Allocation (50%): 3202 MB UnifiedCache
**Configuration:**
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', path: '/var/lib/brainy' },
model: { precision: 'q8' },
// Auto-sized cache: 3202MB
storage: { type: 'filesystem', path: '/var/lib/brainy' },
model: { precision: 'q8' },
// Auto-sized cache: 3202MB
})
// Monitor health
@ -141,17 +141,17 @@ console.log(`Caching strategy: ${stats.cachingStrategy}`)
**Memory Breakdown:**
```
System Memory: 32768 MB
OS Reserved (20%): -6554 MB
Available: 26214 MB
Model Memory (Q8): -150 MB
System Memory: 32768 MB
OS Reserved (20%): -6554 MB
Available: 26214 MB
Model Memory (Q8): -150 MB
───────────────────────────
Available for Cache: 26064 MB
Available for Cache: 26064 MB
Prod Allocation (50%): 13032 MB UnifiedCache
├─ HNSW (30%): 3910 MB
├─ Metadata (40%): 5213 MB
├─ Search (20%): 2606 MB
└─ Shared (10%): 1303 MB
├─ HNSW (30%): 3910 MB
├─ Metadata (40%): 5213 MB
├─ Search (20%): 2606 MB
└─ Shared (10%): 1303 MB
```
**Capacity:**
@ -168,11 +168,11 @@ Prod Allocation (50%): 13032 MB UnifiedCache
**Configuration:**
```typescript
const brain = new Brainy({
storage: {
type: 'gcs-native',
gcsNativeStorage: { bucketName: 'production-data' }
},
model: { precision: 'q8' } // or 'fp32' for +0.5% accuracy
storage: {
type: 'gcs-native',
gcsNativeStorage: { bucketName: 'production-data' }
},
model: { precision: 'q8' } // or 'fp32' for +0.5% accuracy
})
// Verify allocation
@ -192,17 +192,17 @@ console.log(`Environment: ${memoryInfo.memoryInfo.environment}`)
**Memory Breakdown:**
```
System Memory: 131072 MB
OS Reserved (20%): -26214 MB
Available: 104858 MB
Model Memory (FP32): -250 MB
System Memory: 131072 MB
OS Reserved (20%): -26214 MB
Available: 104858 MB
Model Memory (FP32): -250 MB
───────────────────────────────
Available for Cache: 104608 MB
Available for Cache: 104608 MB
Prod Allocation (50%): 52304 MB UnifiedCache (logarithmic scaling applies)
├─ HNSW (30%): 15691 MB
├─ Metadata (40%): 20922 MB
├─ Search (20%): 10461 MB
└─ Shared (10%): 5230 MB
├─ HNSW (30%): 15691 MB
├─ Metadata (40%): 20922 MB
├─ Search (20%): 10461 MB
└─ Shared (10%): 5230 MB
```
**Logarithmic Scaling Applied:**
@ -227,30 +227,30 @@ Actual cache size: ~40GB (prevents waste on 128GB systems)
**Configuration:**
```typescript
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'enterprise-data',
region: 'us-east-1'
}
},
model: { precision: 'fp32' } // Maximum accuracy
storage: {
type: 's3',
s3Storage: {
bucketName: 'enterprise-data',
region: 'us-east-1'
}
},
model: { precision: 'fp32' } // Maximum accuracy
})
// Enterprise monitoring
setInterval(() => {
const stats = brain.hnsw.getCacheStats()
const stats = brain.hnsw.getCacheStats()
if (stats.fairness.fairnessViolation) {
console.warn('FAIRNESS VIOLATION: HNSW using too much cache')
console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access, ${stats.hnswCache.sizePercent}% size`)
}
if (stats.fairness.fairnessViolation) {
console.warn('FAIRNESS VIOLATION: HNSW using too much cache')
console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access, ${stats.hnswCache.sizePercent}% size`)
}
if (stats.unifiedCache.hitRatePercent < 75) {
console.warn(`Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`)
console.warn('Recommendations:', stats.recommendations)
}
}, 60000) // Check every minute
if (stats.unifiedCache.hitRatePercent < 75) {
console.warn(`Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`)
console.warn('Recommendations:', stats.recommendations)
}
}, 60000) // Check every minute
```
---
@ -263,12 +263,12 @@ Brainy auto-detects container memory limits via cgroups v1/v2:
```typescript
// Automatic detection
const brain = new Brainy() // Detects cgroup limits automatically
const brain = new Brainy() // Detects cgroup limits automatically
// Verify detection
const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
console.log(`Container: ${memoryInfo.memoryInfo.isContainer}`)
console.log(`Source: ${memoryInfo.memoryInfo.source}`) // 'cgroup-v2' or 'cgroup-v1'
console.log(`Source: ${memoryInfo.memoryInfo.source}`) // 'cgroup-v2' or 'cgroup-v1'
console.log(`Limit: ${Math.round(memoryInfo.memoryInfo.available / 1024 / 1024)}MB`)
```
@ -294,37 +294,37 @@ CMD ["node", "dist/index.js"]
```bash
docker run \
--memory="2g" \
--memory-reservation="1.5g" \
--cpus="2" \
my-brainy-app
--memory="2g" \
--memory-reservation="1.5g" \
--cpus="2" \
my-brainy-app
```
**Expected allocation:**
```
Container Limit: 2048 MB
Available: 1638 MB (80% usable)
Model Memory: -150 MB
Available for Cache: 1488 MB
Container Limit: 2048 MB
Available: 1638 MB (80% usable)
Model Memory: -150 MB
Available for Cache: 1488 MB
Container Ratio (40%): 595 MB UnifiedCache
```
**Medium Container (8GB)**
```bash
docker run \
--memory="8g" \
--memory-reservation="6g" \
--cpus="4" \
-e NODE_OPTIONS="--max-old-space-size=6144" \
my-brainy-app
--memory="8g" \
--memory-reservation="6g" \
--cpus="4" \
-e NODE_OPTIONS="--max-old-space-size=6144" \
my-brainy-app
```
**Expected allocation:**
```
Container Limit: 8192 MB
Available: 6554 MB
Model Memory: -150 MB
Available for Cache: 6404 MB
Container Limit: 8192 MB
Available: 6554 MB
Model Memory: -150 MB
Available for Cache: 6404 MB
Container Ratio (40%): 2562 MB UnifiedCache
```
@ -335,24 +335,24 @@ Container Ratio (40%): 2562 MB UnifiedCache
apiVersion: apps/v1
kind: Deployment
metadata:
name: brainy-api
name: brainy-api
spec:
replicas: 3
template:
spec:
containers:
- name: brainy
image: my-brainy-app:latest
resources:
requests:
memory: "1.5Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=1536"
replicas: 3
template:
spec:
containers:
- name: brainy
image: my-brainy-app:latest
resources:
requests:
memory: "1.5Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=1536"
```
**Medium Pod (8GB)**
@ -360,24 +360,24 @@ spec:
apiVersion: apps/v1
kind: Deployment
metadata:
name: brainy-api
name: brainy-api
spec:
replicas: 2
template:
spec:
containers:
- name: brainy
image: my-brainy-app:latest
resources:
requests:
memory: "6Gi"
cpu: "2000m"
limits:
memory: "8Gi"
cpu: "4000m"
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=6144"
replicas: 2
template:
spec:
containers:
- name: brainy
image: my-brainy-app:latest
resources:
requests:
memory: "6Gi"
cpu: "2000m"
limits:
memory: "8Gi"
cpu: "4000m"
env:
- name: NODE_OPTIONS
value: "--max-old-space-size=6144"
```
**Best Practices:**
@ -399,15 +399,15 @@ The system automatically chooses the optimal caching strategy:
**Auto-detection logic:**
```typescript
const vectorMemoryNeeded = entityCount × 1536 // bytes
const vectorMemoryNeeded = entityCount × 1536 // bytes
const hnswCacheAvailable = unifiedCache.maxSize × 0.80
if (vectorMemoryNeeded < hnswCacheAvailable) {
// Preload strategy: all vectors loaded at init
console.log('Caching strategy: preloaded (all vectors in memory)')
// Preload strategy: all vectors loaded at init
console.log('Caching strategy: preloaded (all vectors in memory)')
} else {
// On-demand strategy: vectors loaded adaptively
console.log('Caching strategy: on-demand (adaptive loading via UnifiedCache)')
// On-demand strategy: vectors loaded adaptively
console.log('Caching strategy: on-demand (adaptive loading via UnifiedCache)')
}
```
@ -422,12 +422,12 @@ Consider increasing RAM when:
**Decision tree:**
```
If cache hit rate < 70%:
└─> Is working set < 50% of total entities?
├─> YES: Increase cache size (add RAM)
└─> NO: Working set too large, consider:
├─> Application-level caching
├─> Query optimization
└─> Sharding dataset
└─> Is working set < 50% of total entities?
├─> YES: Increase cache size (add RAM)
└─> NO: Working set too large, consider:
├─> Application-level caching
├─> Query optimization
└─> Sharding dataset
```
### When to Shard/Distribute
@ -442,17 +442,17 @@ Consider sharding when:
```typescript
// Example: Geographic sharding
const usEastBrain = new Brainy({
storage: { type: 's3', s3Storage: { bucket: 'us-east-data' } }
storage: { type: 's3', s3Storage: { bucket: 'us-east-data' } }
})
const euWestBrain = new Brainy({
storage: { type: 's3', s3Storage: { bucket: 'eu-west-data' } }
storage: { type: 's3', s3Storage: { bucket: 'eu-west-data' } }
})
// Route queries based on user location
async function search(query, userRegion) {
const brain = userRegion === 'US' ? usEastBrain : euWestBrain
return await brain.search(query)
const brain = userRegion === 'US' ? usEastBrain : euWestBrain
return await brain.search(query)
}
```
@ -482,7 +482,7 @@ console.log(`Pressure: ${memoryInfo.currentPressure.pressure}`)
// Values: 'low', 'moderate', 'high', 'critical'
if (memoryInfo.currentPressure.warnings.length > 0) {
console.warn('Memory warnings:', memoryInfo.currentPressure.warnings)
console.warn('Memory warnings:', memoryInfo.currentPressure.warnings)
}
```
@ -491,9 +491,9 @@ if (memoryInfo.currentPressure.warnings.length > 0) {
const stats = brain.hnsw.getCacheStats()
if (stats.fairness.fairnessViolation) {
console.warn('Cache fairness violation detected')
console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access`)
console.warn(`HNSW: ${stats.hnswCache.sizePercent}% of cache`)
console.warn('Cache fairness violation detected')
console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access`)
console.warn(`HNSW: ${stats.hnswCache.sizePercent}% of cache`)
}
```
@ -502,7 +502,7 @@ if (stats.fairness.fairnessViolation) {
// Track search latency
console.time('search')
const results = await brain.search('query')
console.timeEnd('search') // Target: <10ms for hot queries
console.timeEnd('search') // Target: <10ms for hot queries
```
### Alerting Thresholds
@ -516,26 +516,26 @@ Set up alerts for:
**Example monitoring script:**
```typescript
async function monitorHealth() {
const stats = brain.hnsw.getCacheStats()
const stats = brain.hnsw.getCacheStats()
// Alert on low cache hit rate
if (stats.unifiedCache.hitRatePercent < 70) {
await sendAlert({
severity: 'warning',
message: `Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`,
recommendations: stats.recommendations
})
}
// Alert on low cache hit rate
if (stats.unifiedCache.hitRatePercent < 70) {
await sendAlert({
severity: 'warning',
message: `Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`,
recommendations: stats.recommendations
})
}
// Alert on memory pressure
const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
if (memoryInfo.currentPressure.pressure === 'high') {
await sendAlert({
severity: 'critical',
message: 'High memory pressure detected',
warnings: memoryInfo.currentPressure.warnings
})
}
// Alert on memory pressure
const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
if (memoryInfo.currentPressure.pressure === 'high') {
await sendAlert({
severity: 'critical',
message: 'High memory pressure detected',
warnings: memoryInfo.currentPressure.warnings
})
}
}
// Run every 60 seconds
@ -552,9 +552,9 @@ setInterval(monitorHealth, 60000)
**Sizing:**
```
Products: 500,000
Vector memory needed: 500K × 1536 bytes = 768 MB
HNSW cache available: (16GB × 0.8 - 150MB) × 0.5 × 0.3 = 1,915 MB
Products: 500,000
Vector memory needed: 500K × 1536 bytes = 768 MB
HNSW cache available: (16GB × 0.8 - 150MB) × 0.5 × 0.3 = 1,915 MB
Result: Standard mode (all vectors fit in HNSW cache)
```
@ -562,16 +562,16 @@ Result: Standard mode (all vectors fit in HNSW cache)
**Configuration:**
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', path: '/var/lib/brainy' },
model: { precision: 'q8' }
storage: { type: 'filesystem', path: '/var/lib/brainy' },
model: { precision: 'q8' }
})
await brain.init()
// Verify preloaded strategy (all vectors in memory)
const stats = brain.hnsw.getCacheStats()
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'preloaded'
console.log(`Search latency: ${stats.performance.avgSearchMs}ms`) // ~3ms
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'preloaded'
console.log(`Search latency: ${stats.performance.avgSearchMs}ms`) // ~3ms
```
### Example 2: Document Search (5M documents)
@ -580,9 +580,9 @@ console.log(`Search latency: ${stats.performance.avgSearchMs}ms`) // ~3ms
**Sizing:**
```
Documents: 5,000,000
Vector memory needed: 5M × 1536 bytes = 7,680 MB
HNSW cache available: (32GB × 0.8 - 150MB) × 0.5 × 0.3 = 3,910 MB
Documents: 5,000,000
Vector memory needed: 5M × 1536 bytes = 7,680 MB
HNSW cache available: (32GB × 0.8 - 150MB) × 0.5 × 0.3 = 3,910 MB
Result: On-demand caching (vectors loaded adaptively)
```
@ -590,20 +590,20 @@ Result: On-demand caching (vectors loaded adaptively)
**Configuration:**
```typescript
const brain = new Brainy({
storage: {
type: 'gcs-native',
gcsNativeStorage: { bucketName: 'docs-production' }
},
model: { precision: 'q8' }
storage: {
type: 'gcs-native',
gcsNativeStorage: { bucketName: 'docs-production' }
},
model: { precision: 'q8' }
})
await brain.init()
// Monitor cache performance
const stats = brain.hnsw.getCacheStats()
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand'
console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >80%
console.log(`Cold search latency: ${stats.performance.avgSearchMs}ms`) // ~12ms
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand'
console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >80%
console.log(`Cold search latency: ${stats.performance.avgSearchMs}ms`) // ~12ms
// Recommendations
console.log('Recommendations:', stats.recommendations)
@ -616,9 +616,9 @@ console.log('Recommendations:', stats.recommendations)
**Sizing:**
```
Entities: 20,000,000
Vector memory needed: 20M × 1536 bytes = 30,720 MB
HNSW cache available: ~15,691 MB (after logarithmic scaling)
Entities: 20,000,000
Vector memory needed: 20M × 1536 bytes = 30,720 MB
HNSW cache available: ~15,691 MB (after logarithmic scaling)
Result: On-demand caching with high-performance adaptive loading
```
@ -626,14 +626,14 @@ Result: On-demand caching with high-performance adaptive loading
**Configuration:**
```typescript
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'knowledge-graph-prod',
region: 'us-east-1'
}
},
model: { precision: 'fp32' } // Maximum accuracy
storage: {
type: 's3',
s3Storage: {
bucketName: 'knowledge-graph-prod',
region: 'us-east-1'
}
},
model: { precision: 'fp32' } // Maximum accuracy
})
await brain.init()
@ -641,13 +641,13 @@ await brain.init()
// Enterprise monitoring
const stats = brain.hnsw.getCacheStats()
console.log(`Entities: ${stats.autoDetection.entityCount.toLocaleString()}`)
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand'
console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >85%
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand'
console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >85%
console.log(`HNSW cache: ${stats.hnswCache.estimatedMemoryMB}MB`)
// Fairness check
if (stats.fairness.fairnessViolation) {
console.warn('HNSW dominating cache - consider tuning eviction policies')
console.warn('HNSW dominating cache - consider tuning eviction policies')
}
```
@ -690,8 +690,8 @@ console.log(`Warnings:`, memoryInfo.currentPressure.warnings)
```typescript
const stats = brain.hnsw.getCacheStats()
if (stats.fairness.fairnessViolation) {
console.log(`HNSW access: ${stats.fairness.hnswAccessPercent}%`)
console.log(`HNSW cache: ${stats.hnswCache.sizePercent}%`)
console.log(`HNSW access: ${stats.fairness.hnswAccessPercent}%`)
console.log(`HNSW cache: ${stats.hnswCache.sizePercent}%`)
}
```
@ -704,8 +704,7 @@ if (stats.fairness.fairnessViolation) {
## 📚 Additional Resources
- **[Migration Guide](../guides/migration-3.36.0.md)** - Upgrading to v3.36.0
- **[Architecture Overview](../architecture/data-storage-architecture.md)** - Deep dive into storage and caching
- **[Migration Guide](../guides/migration-3.36.0.md)** - Upgrading to **[Architecture Overview](../architecture/data-storage-architecture.md)** - Deep dive into storage and caching
- **[GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)** - Report problems or ask questions
---

170
docs/operations/cost-optimization-aws-s3.md
View file

@ -1,10 +1,9 @@
# AWS S3 Cost Optimization Guide for Brainy v4.0.0
# AWS S3 Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $138k/year to $5.9k/year at 500TB scale (**96% savings**)
## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance.
Brainy provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance.
## Cost Breakdown (Before Optimization)
@ -39,10 +38,10 @@ import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
// Initialize Brainy with S3 storage
const storage = new S3CompatibleStorage({
bucket: 'my-brainy-data',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
bucket: 'my-brainy-data',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
})
const brain = new Brainy({ storage })
@ -50,24 +49,24 @@ await brain.init()
// Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({
rules: [{
id: 'optimize-vectors',
prefix: 'entities/nouns/vectors/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // Infrequent Access after 30 days
{ days: 90, storageClass: 'GLACIER' }, // Glacier after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 1 year
]
}, {
id: 'optimize-metadata',
prefix: 'entities/nouns/metadata/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' },
{ days: 180, storageClass: 'GLACIER' }
]
}]
rules: [{
id: 'optimize-vectors',
prefix: 'entities/nouns/vectors/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // Infrequent Access after 30 days
{ days: 90, storageClass: 'GLACIER' }, // Glacier after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 1 year
]
}, {
id: 'optimize-metadata',
prefix: 'entities/nouns/metadata/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' },
{ days: 180, storageClass: 'GLACIER' }
]
}]
})
// Verify lifecycle policy
@ -84,10 +83,10 @@ console.log('Lifecycle policy active:', policy.rules.length, 'rules')
- 10% of data 365+ days old (Deep Archive)
```
Standard (200TB): 200TB × $0.023/GB × 12 = $55,200/year
Standard-IA (150TB): 150TB × $0.0125/GB × 12 = $22,500/year
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Deep Archive (50TB): 50TB × $0.00099/GB × 12 = $594/year
Standard (200TB): 200TB × $0.023/GB × 12 = $55,200/year
Standard-IA (150TB): 150TB × $0.0125/GB × 12 = $22,500/year
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Deep Archive (50TB): 50TB × $0.00099/GB × 12 = $594/year
Total Storage Cost: $83,094/year (instead of $138,000)
Total with Operations: ~$88,000/year
@ -133,11 +132,11 @@ Intelligent-Tiering automatically moves objects between:
- 30% Deep Archive Access
```
Frequent (75TB): 75TB × $0.023/GB × 12 = $20,700/year
Infrequent (100TB): 100TB × $0.0125/GB × 12 = $15,000/year
Archive (175TB): 175TB × $0.004/GB × 12 = $8,400/year
Deep Archive (150TB): 150TB × $0.00099/GB × 12 = $1,782/year
Monitoring: ~$300/year (minimal)
Frequent (75TB): 75TB × $0.023/GB × 12 = $20,700/year
Infrequent (100TB): 100TB × $0.0125/GB × 12 = $15,000/year
Archive (175TB): 175TB × $0.004/GB × 12 = $8,400/year
Deep Archive (150TB): 150TB × $0.00099/GB × 12 = $1,782/year
Monitoring: ~$300/year (minimal)
Total Storage Cost: $46,182/year
Total with Operations: ~$51,000/year
@ -157,21 +156,21 @@ await storage.enableIntelligentTiering('entities/verbs/vectors/', 'verbs-auto')
// Set lifecycle policy for metadata (less frequently accessed)
await storage.setLifecyclePolicy({
rules: [{
id: 'archive-old-metadata',
prefix: 'entities/nouns/metadata/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' },
{ days: 60, storageClass: 'GLACIER' },
{ days: 180, storageClass: 'DEEP_ARCHIVE' }
]
}, {
id: 'cleanup-old-system-data',
prefix: '_system/',
status: 'Enabled',
expiration: { days: 365 } // Delete old statistics
}]
rules: [{
id: 'archive-old-metadata',
prefix: 'entities/nouns/metadata/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'STANDARD_IA' },
{ days: 60, storageClass: 'GLACIER' },
{ days: 180, storageClass: 'DEEP_ARCHIVE' }
]
}, {
id: 'cleanup-old-system-data',
prefix: '_system/',
status: 'Enabled',
expiration: { days: 365 } // Delete old statistics
}]
})
```
@ -179,19 +178,19 @@ await storage.setLifecyclePolicy({
**Vectors (300TB with Intelligent-Tiering):**
```
Frequent (45TB): 45TB × $0.023/GB × 12 = $12,420/year
Infrequent (60TB): 60TB × $0.0125/GB × 12 = $9,000/year
Archive (105TB): 105TB × $0.004/GB × 12 = $5,040/year
Deep Archive (90TB): 90TB × $0.00099/GB × 12 = $1,069/year
Frequent (45TB): 45TB × $0.023/GB × 12 = $12,420/year
Infrequent (60TB): 60TB × $0.0125/GB × 12 = $9,000/year
Archive (105TB): 105TB × $0.004/GB × 12 = $5,040/year
Deep Archive (90TB): 90TB × $0.00099/GB × 12 = $1,069/year
Subtotal: $27,529/year
```
**Metadata (200TB with Lifecycle Policy):**
```
Standard (60TB): 60TB × $0.023/GB × 12 = $16,560/year
Standard-IA (40TB): 40TB × $0.0125/GB × 12 = $6,000/year
Glacier (60TB): 60TB × $0.004/GB × 12 = $2,880/year
Deep Archive (40TB): 40TB × $0.00099/GB × 12 = $475/year
Standard (60TB): 60TB × $0.023/GB × 12 = $16,560/year
Standard-IA (40TB): 40TB × $0.0125/GB × 12 = $6,000/year
Glacier (60TB): 60TB × $0.004/GB × 12 = $2,880/year
Deep Archive (40TB): 40TB × $0.00099/GB × 12 = $475/year
Subtotal: $25,915/year
```
@ -206,16 +205,16 @@ Subtotal: $25,915/year
```typescript
await storage.setLifecyclePolicy({
rules: [{
id: 'aggressive-archival',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 14, storageClass: 'STANDARD_IA' }, // IA after 2 weeks
{ days: 30, storageClass: 'GLACIER' }, // Glacier after 1 month
{ days: 90, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 3 months
]
}]
rules: [{
id: 'aggressive-archival',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 14, storageClass: 'STANDARD_IA' }, // IA after 2 weeks
{ days: 30, storageClass: 'GLACIER' }, // Glacier after 1 month
{ days: 90, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 3 months
]
}]
})
```
@ -223,10 +222,10 @@ await storage.setLifecyclePolicy({
**After 1 year:**
```
Standard (50TB): 50TB × $0.023/GB × 12 = $13,800/year
Standard-IA (50TB): 50TB × $0.0125/GB × 12 = $7,500/year
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Deep Archive (300TB): 300TB × $0.00099/GB × 12 = $3,564/year
Standard (50TB): 50TB × $0.023/GB × 12 = $13,800/year
Standard-IA (50TB): 50TB × $0.0125/GB × 12 = $7,500/year
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Deep Archive (300TB): 300TB × $0.00099/GB × 12 = $3,564/year
Total Storage Cost: $29,664/year
Total with Operations: ~$34,000/year
@ -250,16 +249,16 @@ Note: Retrieval costs may be significant if archived data is accessed frequently
### Efficient Cleanup
```typescript
// v4.0.0: Batch delete (1000 objects per request)
// Batch delete (1000 objects per request)
const idsToDelete = [/* array of entity IDs */]
// Generate paths for both vector and metadata files
const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
})
// Batch delete (much faster and cheaper than individual deletes)
@ -280,14 +279,14 @@ console.log('Active rules:', policy.rules)
// Example output:
// {
// rules: [
// {
// id: 'optimize-vectors',
// prefix: 'entities/nouns/vectors/',
// status: 'Enabled',
// transitions: [...]
// }
// ]
// rules: [
// {
// id: 'optimize-vectors',
// prefix: 'entities/nouns/vectors/',
// status: 'Enabled',
// transitions: [...]
// }
// ]
// }
```
@ -396,6 +395,5 @@ await storage.enableIntelligentTiering('entities/', 'new-config')
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: AWS S3

306
docs/operations/cost-optimization-azure.md
View file

@ -1,10 +1,9 @@
# Azure Blob Storage Cost Optimization Guide for Brainy v4.0.0
# Azure Blob Storage Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $107k/year to $5k/year at 500TB scale (**95% savings**)
## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for Azure Blob Storage, including manual tier management, lifecycle policies, and batch operations.
Brainy provides enterprise-grade cost optimization features for Azure Blob Storage, including manual tier management, lifecycle policies, and batch operations.
## Cost Breakdown (Before Optimization)
@ -41,8 +40,8 @@ import { AzureBlobStorage } from '@soulcraft/brainy/storage'
// Initialize Brainy with Azure storage
const storage = new AzureBlobStorage({
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
})
const brain = new Brainy({ storage })
@ -50,19 +49,19 @@ await brain.init()
// Change tier for a single blob
await storage.changeBlobTier(
'entities/nouns/vectors/00/00123456-uuid.json',
'Cool'
'entities/nouns/vectors/00/00123456-uuid.json',
'Cool'
)
// Batch tier changes (efficient for thousands of blobs)
const blobsToMove = [
'entities/nouns/vectors/01/...',
'entities/nouns/vectors/02/...',
// ... up to thousands of blobs
'entities/nouns/vectors/01/...',
'entities/nouns/vectors/02/...',
// ... up to thousands of blobs
]
await storage.batchChangeTier(blobsToMove, 'Cool') // Hot → Cool
await storage.batchChangeTier(oldBlobs, 'Archive') // Cool → Archive
await storage.batchChangeTier(blobsToMove, 'Cool') // Hot → Cool
await storage.batchChangeTier(oldBlobs, 'Archive') // Cool → Archive
```
### Immediate Cost Impact
@ -90,54 +89,54 @@ Savings: $20,346/year (95% savings on moved data)
```typescript
// Set lifecycle policy for automatic tier management
await storage.setLifecyclePolicy({
rules: [{
name: 'optimizeVectors',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/vectors/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}, {
name: 'optimizeMetadata',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/metadata/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 180 }
}
}
}
}, {
name: 'cleanupOldSystemFiles',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['_system/']
},
actions: {
baseBlob: {
delete: { daysAfterModificationGreaterThan: 365 }
}
}
}
}]
rules: [{
name: 'optimizeVectors',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/vectors/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}, {
name: 'optimizeMetadata',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/metadata/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 180 }
}
}
}
}, {
name: 'cleanupOldSystemFiles',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['_system/']
},
actions: {
baseBlob: {
delete: { daysAfterModificationGreaterThan: 365 }
}
}
}
}]
})
// Verify lifecycle policy
@ -153,9 +152,9 @@ console.log('Lifecycle policy active:', policy.rules.length, 'rules')
- 30% of data in Archive tier (90+ days old)
```
Hot (150TB): 150TB × $0.0184/GB × 12 = $32,256/year
Cool (200TB): 200TB × $0.0115/GB × 12 = $26,880/year
Archive (150TB): 150TB × $0.00099/GB × 12 = $1,732/year
Hot (150TB): 150TB × $0.0184/GB × 12 = $32,256/year
Cool (200TB): 200TB × $0.0115/GB × 12 = $26,880/year
Archive (150TB): 150TB × $0.00099/GB × 12 = $1,732/year
Total Storage Cost: $60,868/year
Total with Operations: ~$65,500/year
@ -170,23 +169,23 @@ Savings: $47,000/year (42%)
```typescript
await storage.setLifecyclePolicy({
rules: [{
name: 'aggressiveArchival',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 14 },
tierToArchive: { daysAfterModificationGreaterThan: 30 }
}
}
}
}]
rules: [{
name: 'aggressiveArchival',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 14 },
tierToArchive: { daysAfterModificationGreaterThan: 30 }
}
}
}
}]
})
```
@ -194,9 +193,9 @@ await storage.setLifecyclePolicy({
**After 6 months:**
```
Hot (50TB): 50TB × $0.0184/GB × 12 = $10,752/year
Cool (100TB): 100TB × $0.0115/GB × 12 = $13,440/year
Archive (350TB): 350TB × $0.00099/GB × 12 = $4,039/year
Hot (50TB): 50TB × $0.0184/GB × 12 = $10,752/year
Cool (100TB): 100TB × $0.0115/GB × 12 = $13,440/year
Archive (350TB): 350TB × $0.00099/GB × 12 = $4,039/year
Total Storage Cost: $28,231/year
Total with Operations: ~$33,000/year
@ -211,41 +210,41 @@ Warning: Archive rehydration takes 1-15 hours
```typescript
await storage.setLifecyclePolicy({
rules: [{
// Vectors: Keep in Hot/Cool for search performance
name: 'vectors-moderate',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/vectors/', 'entities/verbs/vectors/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 60 }
// Don't archive vectors - keep searchable
}
}
}
}, {
// Metadata: Aggressive archival
name: 'metadata-aggressive',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/metadata/', 'entities/verbs/metadata/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}]
rules: [{
// Vectors: Keep in Hot/Cool for search performance
name: 'vectors-moderate',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/vectors/', 'entities/verbs/vectors/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 60 }
// Don't archive vectors - keep searchable
}
}
}
}, {
// Metadata: Aggressive archival
name: 'metadata-aggressive',
enabled: true,
type: 'Lifecycle',
definition: {
filters: {
blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/metadata/', 'entities/verbs/metadata/']
},
actions: {
baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 }
}
}
}
}]
})
```
@ -253,16 +252,16 @@ await storage.setLifecyclePolicy({
**Vectors (300TB):**
```
Hot (90TB): 90TB × $0.0184/GB × 12 = $19,354/year
Cool (210TB): 210TB × $0.0115/GB × 12 = $28,980/year
Hot (90TB): 90TB × $0.0184/GB × 12 = $19,354/year
Cool (210TB): 210TB × $0.0115/GB × 12 = $28,980/year
Subtotal: $48,334/year
```
**Metadata (200TB):**
```
Hot (30TB): 30TB × $0.0184/GB × 12 = $6,451/year
Cool (70TB): 70TB × $0.0115/GB × 12 = $9,660/year
Archive (100TB): 100TB × $0.00099/GB × 12 = $1,158/year
Hot (30TB): 30TB × $0.0184/GB × 12 = $6,451/year
Cool (70TB): 70TB × $0.0115/GB × 12 = $9,660/year
Archive (100TB): 100TB × $0.00099/GB × 12 = $1,158/year
Subtotal: $17,269/year
```
@ -288,8 +287,8 @@ Subtotal: $17,269/year
```typescript
// Rehydrate blob from Archive to Hot (high priority)
await storage.rehydrateBlob(
'entities/nouns/vectors/00/00123456-uuid.json',
'High' // 'Standard' or 'High' priority
'entities/nouns/vectors/00/00123456-uuid.json',
'High' // 'Standard' or 'High' priority
)
// Rehydration time:
@ -309,7 +308,7 @@ console.log('Archive status:', metadata.archiveStatus)
const blobsToRehydrate = [/* array of blob paths */]
for (const blobPath of blobsToRehydrate) {
await storage.rehydrateBlob(blobPath, 'High')
await storage.rehydrateBlob(blobPath, 'High')
}
// Wait for rehydration to complete (1-15 hours)
@ -333,15 +332,15 @@ Examples:
### Efficient Bulk Deletions
```typescript
// v4.0.0: Batch delete (256 blobs per request)
// Batch delete (256 blobs per request)
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
})
// Batch delete via BlobBatchClient
@ -375,14 +374,14 @@ console.log('Active rules:', policy.rules)
// Example output:
// {
// rules: [
// {
// name: 'optimizeVectors',
// enabled: true,
// type: 'Lifecycle',
// definition: {...}
// }
// ]
// rules: [
// {
// name: 'optimizeVectors',
// enabled: true,
// type: 'Lifecycle',
// definition: {...}
// }
// ]
// }
```
@ -452,8 +451,8 @@ Monthly cost trend: Decreasing 5-8% per month as data transitions
// Check lifecycle policy status
const policy = await storage.getLifecyclePolicy()
console.log('Policy rules:', policy.rules.map(r => ({
name: r.name,
enabled: r.enabled
name: r.name,
enabled: r.enabled
})))
// Azure lifecycle policies run once per day
@ -471,9 +470,9 @@ await storage.rehydrateBlob(blobPath, 'High')
// Wait for rehydration (check status)
let status
do {
await new Promise(resolve => setTimeout(resolve, 60000)) // Wait 1 minute
const metadata = await storage.getBlobMetadata(blobPath)
status = metadata.archiveStatus
await new Promise(resolve => setTimeout(resolve, 60000)) // Wait 1 minute
const metadata = await storage.getBlobMetadata(blobPath)
status = metadata.archiveStatus
} while (status && status.includes('pending'))
// Now access the blob
@ -494,8 +493,8 @@ const data = await storage.get(blobPath)
```typescript
const storage = new AzureBlobStorage({
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data'
})
```
@ -503,9 +502,9 @@ const storage = new AzureBlobStorage({
```typescript
const storage = new AzureBlobStorage({
accountName: process.env.AZURE_STORAGE_ACCOUNT,
accountKey: process.env.AZURE_STORAGE_KEY,
containerName: 'brainy-data'
accountName: process.env.AZURE_STORAGE_ACCOUNT,
accountKey: process.env.AZURE_STORAGE_KEY,
containerName: 'brainy-data'
})
```
@ -513,9 +512,9 @@ const storage = new AzureBlobStorage({
```typescript
const storage = new AzureBlobStorage({
sasToken: process.env.AZURE_STORAGE_SAS_TOKEN,
accountName: process.env.AZURE_STORAGE_ACCOUNT,
containerName: 'brainy-data'
sasToken: process.env.AZURE_STORAGE_SAS_TOKEN,
accountName: process.env.AZURE_STORAGE_ACCOUNT,
containerName: 'brainy-data'
})
```
@ -549,6 +548,5 @@ const storage = new AzureBlobStorage({
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: Azure Blob Storage

162
docs/operations/cost-optimization-cloudflare-r2.md
View file

@ -1,10 +1,9 @@
# Cloudflare R2 Cost Optimization Guide for Brainy v4.0.0
# Cloudflare R2 Cost Optimization Guide for Brainy
> **Cost Impact**: $0 egress fees + 96% storage savings = Lowest cloud storage costs
## Overview
Cloudflare R2 is an S3-compatible object storage with **zero egress fees**, making it ideal for high-traffic applications. Brainy v4.0.0 fully supports R2 with lifecycle policies and batch operations.
Cloudflare R2 is an S3-compatible object storage with **zero egress fees**, making it ideal for high-traffic applications. Brainy fully supports R2 with lifecycle policies and batch operations.
## Cost Breakdown
@ -49,7 +48,7 @@ Savings vs AWS: $156,000/year (62%)
- Cloudflare plans to add infrequent access tiers
**When lifecycle features arrive:**
- Brainy v4.0.0 is already prepared with `setLifecyclePolicy()` support
- Brainy is already prepared with `setLifecyclePolicy()` support
- Will work seamlessly once Cloudflare enables lifecycle management
## Strategy 1: Use R2 Standard (Current Best Practice)
@ -62,11 +61,11 @@ import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
// R2 uses S3-compatible API
const storage = new S3CompatibleStorage({
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'my-brainy-data',
region: 'auto', // R2 uses 'auto' region
accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'my-brainy-data',
region: 'auto', // R2 uses 'auto' region
accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
})
const brain = new Brainy({ storage })
@ -96,23 +95,23 @@ Total: $90,081/year
```typescript
// Cloudflare Worker (runs at edge)
export default {
async fetch(request, env) {
// Initialize Brainy with R2
const storage = new S3CompatibleStorage({
endpoint: env.R2_ENDPOINT,
bucket: env.R2_BUCKET,
accessKeyId: env.R2_ACCESS_KEY,
secretAccessKey: env.R2_SECRET_KEY,
region: 'auto'
})
async fetch(request, env) {
// Initialize Brainy with R2
const storage = new S3CompatibleStorage({
endpoint: env.R2_ENDPOINT,
bucket: env.R2_BUCKET,
accessKeyId: env.R2_ACCESS_KEY,
secretAccessKey: env.R2_SECRET_KEY,
region: 'auto'
})
const brain = new Brainy({ storage })
await brain.init()
const brain = new Brainy({ storage })
await brain.init()
// Process at edge (no origin server needed)
const results = await brain.search(request.query)
return new Response(JSON.stringify(results))
}
// Process at edge (no origin server needed)
const results = await brain.search(request.query)
return new Response(JSON.stringify(results))
}
}
```
@ -121,18 +120,18 @@ export default {
```
R2 Storage (500TB): $90,000/year
Workers (10M requests/day):
- First 100k requests/day: FREE
- Additional 350M requests/month: $1,750/year
- CPU time (50ms avg): $5,000/year
- First 100k requests/day: FREE
- Additional 350M requests/month: $1,750/year
- CPU time (50ms avg): $5,000/year
Total: $96,750/year
vs Traditional Setup (AWS S3 + EC2 + CloudFront):
- S3: $138,000/year
- EC2 (t3.xlarge × 4): $24,000/year
- CloudFront egress: $50,000/year
- Load balancer: $3,000/year
Total: $215,000/year
- S3: $138,000/year
- EC2 (t3.xlarge × 4): $24,000/year
- CloudFront egress: $50,000/year
- Load balancer: $3,000/year
Total: $215,000/year
Savings: $118,250/year (55%)
```
@ -144,20 +143,20 @@ Savings: $118,250/year (55%)
```typescript
// Use R2 for frequently accessed data (zero egress)
const hotStorage = new S3CompatibleStorage({
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'brainy-hot',
region: 'auto',
accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'brainy-hot',
region: 'auto',
accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
})
// Use AWS S3 with Intelligent-Tiering for cold data
const coldStorage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com',
bucket: 'brainy-archive',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
endpoint: 's3.amazonaws.com',
bucket: 'brainy-archive',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
})
// Initialize separate Brainy instances or implement tiering logic
@ -167,17 +166,17 @@ const coldStorage = new S3CompatibleStorage({
```
R2 Hot Data (300TB):
Storage: 300TB × $0.015/GB × 12 = $54,000/year
Egress: $0
Storage: 300TB × $0.015/GB × 12 = $54,000/year
Egress: $0
S3 Cold Data (200TB with lifecycle → Deep Archive):
Storage: 200TB × $0.00099/GB × 12 = $2,376/year
Ops: $1,000/year
Storage: 200TB × $0.00099/GB × 12 = $2,376/year
Ops: $1,000/year
Total: $57,376/year
vs All AWS S3:
- S3 storage + egress: $251,000/year
- S3 storage + egress: $251,000/year
Savings: $193,624/year (77%)
```
@ -187,15 +186,15 @@ Savings: $193,624/year (77%)
### Efficient Bulk Deletions
```typescript
// v4.0.0: R2 supports S3 batch delete API
// R2 supports S3 batch delete API
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
})
// Batch delete (1000 objects per request)
@ -231,10 +230,10 @@ https://storage.yourdomain.com/entities/nouns/vectors/...
```typescript
// Worker triggered on R2 object upload
export default {
async fetch(request, env) {
// Process new objects automatically
// E.g., index new entities, generate thumbnails, etc.
}
async fetch(request, env) {
// Process new objects automatically
// E.g., index new entities, generate thumbnails, etc.
}
}
// Cost: Only pay for Worker execution (no polling needed)
@ -244,7 +243,7 @@ export default {
```typescript
// Generate presigned URL for direct browser uploads
const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry
const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry
// Client uploads directly to R2 (no server bandwidth used)
```
@ -255,7 +254,7 @@ const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry
```typescript
const status = await storage.getStorageStatus()
console.log('Storage type:', status.type) // 's3-compatible'
console.log('Storage type:', status.type) // 's3-compatible'
console.log('Bucket:', status.details.bucket)
console.log('Endpoint:', status.details.endpoint)
```
@ -318,20 +317,20 @@ Egress: Unlimited (always free)
```bash
# Install rclone
brew install rclone # or apt-get install rclone
brew install rclone # or apt-get install rclone
# Configure S3 source
rclone config create s3-source s3 \
access_key_id=$AWS_ACCESS_KEY \
secret_access_key=$AWS_SECRET_KEY \
region=us-east-1
access_key_id=$AWS_ACCESS_KEY \
secret_access_key=$AWS_SECRET_KEY \
region=us-east-1
# Configure R2 destination
rclone config create r2-dest s3 \
access_key_id=$R2_ACCESS_KEY \
secret_access_key=$R2_SECRET_KEY \
endpoint=https://$R2_ACCOUNT_ID.r2.cloudflarestorage.com \
region=auto
access_key_id=$R2_ACCESS_KEY \
secret_access_key=$R2_SECRET_KEY \
endpoint=https://$R2_ACCOUNT_ID.r2.cloudflarestorage.com \
region=auto
# Copy data
rclone copy s3-source:my-bucket r2-dest:my-bucket --progress
@ -361,17 +360,17 @@ ROI: 3.4 months
### Prepared for Future Features
```typescript
// Brainy v4.0.0 is ready for R2 lifecycle features
// Brainy is ready for R2 lifecycle features
await storage.setLifecyclePolicy({
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'INFREQUENT_ACCESS' }, // When available
{ days: 90, storageClass: 'ARCHIVE' }
]
}]
rules: [{
id: 'archive-old-data',
prefix: 'entities/',
status: 'Enabled',
transitions: [
{ days: 30, storageClass: 'INFREQUENT_ACCESS' }, // When available
{ days: 90, storageClass: 'ARCHIVE' }
]
}]
})
// Expected cost impact (when lifecycle is available):
@ -388,11 +387,11 @@ await storage.setLifecyclePolicy({
```typescript
// Ensure correct endpoint format
const storage = new S3CompatibleStorage({
endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
// NOT: `https://r2.cloudflarestorage.com/${accountId}`
endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
// NOT: `https://r2.cloudflarestorage.com/${accountId}`
region: 'auto', // R2 requires 'auto'
forcePathStyle: false // R2 uses virtual-hosted-style
region: 'auto', // R2 requires 'auto'
forcePathStyle: false // R2 uses virtual-hosted-style
})
```
@ -446,7 +445,6 @@ const storage = new S3CompatibleStorage({
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: Cloudflare R2
**Key Advantage**: **$0 egress fees forever**

142
docs/operations/cost-optimization-gcs.md
View file

@ -1,10 +1,9 @@
# Google Cloud Storage Cost Optimization Guide for Brainy v4.0.0
# Google Cloud Storage Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $138k/year to $8.3k/year at 500TB scale (**94% savings**)
## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management.
Brainy provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management.
## Cost Breakdown (Before Optimization)
@ -35,8 +34,8 @@ import { GcsStorage } from '@soulcraft/brainy/storage'
// Initialize Brainy with GCS storage
const storage = new GcsStorage({
bucketName: 'my-brainy-data',
keyFilename: './service-account.json' // Or use ADC
bucketName: 'my-brainy-data',
keyFilename: './service-account.json' // Or use ADC
})
const brain = new Brainy({ storage })
@ -44,16 +43,16 @@ await brain.init()
// Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({
rules: [{
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 365 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
rules: [{
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 365 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
})
// Verify lifecycle policy
@ -70,10 +69,10 @@ console.log('Lifecycle policy active:', policy.rules.length, 'rules')
- 10% of data 365+ days old (Archive)
```
Standard (200TB): 200TB × $0.020/GB × 12 = $48,000/year
Nearline (150TB): 150TB × $0.010/GB × 12 = $18,000/year
Coldline (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
Standard (200TB): 200TB × $0.020/GB × 12 = $48,000/year
Nearline (150TB): 150TB × $0.010/GB × 12 = $18,000/year
Coldline (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
Total Storage Cost: $71,520/year
Total with Operations: ~$76,500/year
@ -89,7 +88,7 @@ Savings: $66,500/year (46%)
```typescript
// Enable Autoclass for automatic tier management
await storage.enableAutoclass({
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
})
// Benefits:
@ -116,10 +115,10 @@ await storage.enableAutoclass({
- 40% Archive (cold data)
```
Standard (50TB): 50TB × $0.020/GB × 12 = $12,000/year
Nearline (75TB): 75TB × $0.010/GB × 12 = $9,000/year
Coldline (175TB): 175TB × $0.004/GB × 12 = $8,400/year
Archive (200TB): 200TB × $0.0012/GB × 12 = $2,880/year
Standard (50TB): 50TB × $0.020/GB × 12 = $12,000/year
Nearline (75TB): 75TB × $0.010/GB × 12 = $9,000/year
Coldline (175TB): 175TB × $0.004/GB × 12 = $8,400/year
Archive (200TB): 200TB × $0.0012/GB × 12 = $2,880/year
Total Storage Cost: $32,280/year
Total with Operations: ~$37,000/year
@ -133,24 +132,24 @@ Savings vs Standard: $106,000/year (74%)
```typescript
// Enable Autoclass for vectors (frequently searched)
await storage.enableAutoclass({
terminalStorageClass: 'COLDLINE' // Don't archive vectors deeply
terminalStorageClass: 'COLDLINE' // Don't archive vectors deeply
})
// Set lifecycle policy for metadata (less frequently accessed)
await storage.setLifecyclePolicy({
rules: [{
condition: { age: 30, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 60, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 180, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}, {
condition: { age: 730, matchesPrefix: ['_system/'] },
action: { type: 'Delete' } // Delete old system files after 2 years
}]
rules: [{
condition: { age: 30, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 60, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 180, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}, {
condition: { age: 730, matchesPrefix: ['_system/'] },
action: { type: 'Delete' } // Delete old system files after 2 years
}]
})
```
@ -158,18 +157,18 @@ await storage.setLifecyclePolicy({
**Vectors (300TB with Autoclass):**
```
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
Nearline (45TB): 45TB × $0.010/GB × 12 = $5,400/year
Coldline (225TB): 225TB × $0.004/GB × 12 = $10,800/year
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
Nearline (45TB): 45TB × $0.010/GB × 12 = $5,400/year
Coldline (225TB): 225TB × $0.004/GB × 12 = $10,800/year
Subtotal: $23,400/year
```
**Metadata (200TB with Lifecycle Policy):**
```
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
Nearline (40TB): 40TB × $0.010/GB × 12 = $4,800/year
Coldline (80TB): 80TB × $0.004/GB × 12 = $3,840/year
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
Nearline (40TB): 40TB × $0.010/GB × 12 = $4,800/year
Coldline (80TB): 80TB × $0.004/GB × 12 = $3,840/year
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
Subtotal: $16,560/year
```
@ -182,16 +181,16 @@ Subtotal: $16,560/year
```typescript
await storage.setLifecyclePolicy({
rules: [{
condition: { age: 14 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
rules: [{
condition: { age: 14 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, {
condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, {
condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}]
})
// Note: Archive class has 365-day minimum storage duration
@ -202,10 +201,10 @@ await storage.setLifecyclePolicy({
**After 1 year:**
```
Standard (25TB): 25TB × $0.020/GB × 12 = $6,000/year
Nearline (50TB): 50TB × $0.010/GB × 12 = $6,000/year
Coldline (75TB): 75TB × $0.004/GB × 12 = $3,600/year
Archive (350TB): 350TB × $0.0012/GB × 12 = $5,040/year
Standard (25TB): 25TB × $0.020/GB × 12 = $6,000/year
Nearline (50TB): 50TB × $0.010/GB × 12 = $6,000/year
Coldline (75TB): 75TB × $0.004/GB × 12 = $3,600/year
Archive (350TB): 350TB × $0.0012/GB × 12 = $5,040/year
Total Storage Cost: $20,640/year
Total with Operations: ~$25,500/year
@ -241,15 +240,15 @@ Warning: High retrieval costs if archived data is accessed frequently
### Efficient Cleanup
```typescript
// v4.0.0: Batch delete (100 objects per request for GCS)
// Batch delete (100 objects per request for GCS)
const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
const shard = id.substring(0, 2)
return [
`entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json`
]
})
// Batch delete
@ -271,9 +270,9 @@ console.log('Terminal class:', status.terminalStorageClass)
// Example output:
// {
// enabled: true,
// terminalStorageClass: 'ARCHIVE',
// toggleTime: '2025-01-15T10:30:00Z'
// enabled: true,
// terminalStorageClass: 'ARCHIVE',
// toggleTime: '2025-01-15T10:30:00Z'
// }
```
@ -336,7 +335,7 @@ Monthly cost trend: Decreasing 8-12% per month as data ages into cheaper classes
// Check Autoclass status
const status = await storage.getAutoclassStatus()
if (!status.enabled) {
await storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' })
await storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' })
}
// Autoclass requires 24-48 hours for initial transitions
@ -365,8 +364,8 @@ if (!status.enabled) {
```typescript
// Use ADC instead of service account key file
const storage = new GcsStorage({
bucketName: 'my-brainy-data'
// No keyFilename needed - uses ADC automatically
bucketName: 'my-brainy-data'
// No keyFilename needed - uses ADC automatically
})
// ADC authentication order:
@ -417,6 +416,5 @@ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
---
**Version**: v4.0.0
**Last Updated**: 2025-10-17
**Cloud Provider**: Google Cloud Storage

246
docs/transactions.md
View file

@ -1,6 +1,6 @@
# Transaction System
**Status:** ✅ Production Ready (v5.8.0+)
**Status:** ✅ Production Ready
## Overview
@ -17,11 +17,11 @@ Brainy's transaction system provides **atomic operations** with automatic rollba
```
User Code (brain.add(), brain.update(), etc.)
↓
↓
Transaction Manager (orchestration)
↓
↓
Operations (SaveNounMetadataOperation, SaveNounOperation, etc.)
↓
↓
Storage Adapter (COW, sharding, type-aware routing)
```
@ -32,8 +32,8 @@ Every write operation in Brainy automatically uses transactions:
```typescript
// Internally, this uses a transaction
const id = await brain.add({
data: { name: 'Alice', role: 'Engineer' },
type: NounType.Person
data: { name: 'Alice', role: 'Engineer' },
type: NounType.Person
})
```
@ -51,19 +51,19 @@ Each operation implements both **execute** and **undo**:
```typescript
class SaveNounMetadataOperation {
async execute(): Promise<void> {
// Save new metadata
await this.storage.saveNounMetadata(this.id, this.metadata)
}
async execute(): Promise<void> {
// Save new metadata
await this.storage.saveNounMetadata(this.id, this.metadata)
}
async undo(): Promise<void> {
// Restore previous metadata (or delete if new entity)
if (this.previousMetadata) {
await this.storage.saveNounMetadata(this.id, this.previousMetadata)
} else {
await this.storage.deleteNounMetadata(this.id)
}
}
async undo(): Promise<void> {
// Restore previous metadata (or delete if new entity)
if (this.previousMetadata) {
await this.storage.saveNounMetadata(this.id, this.previousMetadata)
} else {
await this.storage.deleteNounMetadata(this.id)
}
}
}
```
@ -87,8 +87,8 @@ await brain.cow.checkout('feature-branch')
// Add entity (uses transaction on this branch)
const id = await brain.add({
data: { name: 'Feature Entity' },
type: NounType.Thing
data: { name: 'Feature Entity' },
type: NounType.Thing
})
// On rollback: Branch remains clean, no partial commits
@ -123,7 +123,7 @@ await brain.relate({ from: id1, to: id2, type: VerbType.RelatesTo })
- Transaction operations don't need to know about shards
- Rollback works across all shards involved
### ID-First Storage (v6.0.0+)
### ID-First Storage
✅ **Fully Compatible**
@ -132,27 +132,27 @@ Transactions work with direct ID-first paths - no type routing needed!
```typescript
// Entities stored with direct ID-first paths
const personId = await brain.add({
data: { name: 'John Doe' },
type: NounType.Person // → entities/nouns/{shard}/{id}/metadata.json
data: { name: 'John Doe' },
type: NounType.Person // → entities/nouns/{shard}/{id}/metadata.json
})
const orgId = await brain.add({
data: { name: 'Acme Corp' },
type: NounType.Organization // → entities/nouns/{shard}/{id}/metadata.json
data: { name: 'Acme Corp' },
type: NounType.Organization // → entities/nouns/{shard}/{id}/metadata.json
})
// Type changes handled atomically (type is just metadata)
await brain.update({
id: personId,
type: NounType.Organization, // Type change
data: { name: 'Doe Corp' }
id: personId,
type: NounType.Organization, // Type change
data: { name: 'Doe Corp' }
})
```
**How It Works:**
- Type information stored in metadata.noun field
- Storage layer uses O(1) ID-first path construction
- No type cache needed (removed in v6.0.0)
- No type cache needed (removed in a previous version)
- Type counters adjusted on commit/rollback
- 40x faster on cloud storage (eliminates 42-type search)
@ -165,10 +165,10 @@ Transactions work with distributed/remote storage:
```typescript
// Works with S3, Azure, GCS, etc.
const brain = new Brainy({
storage: {
type: 's3Compatible',
config: { /* S3 config */ }
}
storage: {
type: 's3Compatible',
config: { /* S3 config */ }
}
})
// Transactions ensure atomicity at write coordinator level
@ -199,8 +199,8 @@ await brain.init()
// Automatically uses transaction
const id = await brain.add({
data: { name: 'Alice', role: 'Engineer' },
type: NounType.Person
data: { name: 'Alice', role: 'Engineer' },
type: NounType.Person
})
// If add fails, all changes rolled back automatically
@ -211,15 +211,15 @@ const id = await brain.add({
```typescript
// Original entity
const id = await brain.add({
data: { name: 'John Smith', category: 'individual' },
type: NounType.Person
data: { name: 'John Smith', category: 'individual' },
type: NounType.Person
})
// Update with type change (atomic)
await brain.update({
id,
type: NounType.Organization, // Type change
data: { name: 'Smith Corp', category: 'business' }
id,
type: NounType.Organization, // Type change
data: { name: 'Smith Corp', category: 'business' }
})
// If update fails, original type and data restored
@ -229,20 +229,20 @@ await brain.update({
```typescript
const personId = await brain.add({
data: { name: 'Alice' },
type: NounType.Person
data: { name: 'Alice' },
type: NounType.Person
})
const projectId = await brain.add({
data: { name: 'Project X' },
type: NounType.Thing
data: { name: 'Project X' },
type: NounType.Thing
})
// Create relationship (atomic)
await brain.relate({
from: personId,
to: projectId,
type: VerbType.WorksOn
from: personId,
to: projectId,
type: VerbType.WorksOn
})
// If relate fails, no partial relationship created
@ -253,10 +253,10 @@ await brain.relate({
```typescript
// Multiple operations, all atomic
for (let i = 0; i < 100; i++) {
await brain.add({
data: { name: `Entity ${i}`, index: i },
type: NounType.Thing
})
await brain.add({
data: { name: `Entity ${i}`, index: i },
type: NounType.Thing
})
}
// Each add() is a separate transaction
@ -267,19 +267,19 @@ for (let i = 0; i < 100; i++) {
```typescript
const personId = await brain.add({
data: { name: 'Bob' },
type: NounType.Person
data: { name: 'Bob' },
type: NounType.Person
})
const projectId = await brain.add({
data: { name: 'Project Y' },
type: NounType.Thing
data: { name: 'Project Y' },
type: NounType.Thing
})
await brain.relate({
from: personId,
to: projectId,
type: VerbType.WorksOn
from: personId,
to: projectId,
type: VerbType.WorksOn
})
// Delete person (atomic - deletes entity + relationships)
@ -294,15 +294,15 @@ Transactions automatically handle errors and rollback:
```typescript
try {
await brain.add({
data: { name: 'Test Entity' },
type: NounType.Thing,
vector: [1, 2, 3] // Wrong dimension → error
})
await brain.add({
data: { name: 'Test Entity' },
type: NounType.Thing,
vector: [1, 2, 3] // Wrong dimension → error
})
} catch (error) {
// Transaction automatically rolled back
// No partial data in storage or indexes
console.error('Add failed:', error.message)
// Transaction automatically rolled back
// No partial data in storage or indexes
console.error('Add failed:', error.message)
}
```
@ -333,11 +333,11 @@ try {
const stats = brain.transactionManager?.getStats()
console.log(stats)
// {
// totalTransactions: 1234,
// successfulTransactions: 1200,
// failedTransactions: 34,
// rollbacks: 34,
// averageOperationsPerTransaction: 4.2
// totalTransactions: 1234,
// successfulTransactions: 1200,
// failedTransactions: 34,
// rollbacks: 34,
// averageOperationsPerTransaction: 4.2
// }
```
@ -359,7 +359,7 @@ await brain.update({ id, data })
await brain.delete(id)
// ❌ Avoid: Direct storage access bypasses transactions
await brain.storage.saveNoun(noun) // No transaction protection
await brain.storage.saveNoun(noun) // No transaction protection
```
### 2. Handle Errors Gracefully
@ -367,11 +367,11 @@ await brain.storage.saveNoun(noun) // No transaction protection
```typescript
// ✅ Recommended: Catch errors, transaction rolls back automatically
try {
const id = await brain.add({ data, type })
return id
const id = await brain.add({ data, type })
return id
} catch (error) {
console.error('Add failed, rolled back:', error)
// Decide how to handle (retry, log, alert user)
console.error('Add failed, rolled back:', error)
// Decide how to handle (retry, log, alert user)
}
```
@ -380,7 +380,7 @@ try {
```typescript
// ✅ Recommended: Validate early to avoid unnecessary rollbacks
if (!isValidVector(vector, brain.dimension)) {
throw new Error(`Vector must have ${brain.dimension} dimensions`)
throw new Error(`Vector must have ${brain.dimension} dimensions`)
}
await brain.add({ data, type, vector })
@ -391,14 +391,14 @@ await brain.add({ data, type, vector })
```typescript
// ✅ Recommended: Monitor in production
setInterval(() => {
const stats = brain.transactionManager?.getStats()
if (stats) {
const failureRate = stats.failedTransactions / stats.totalTransactions
if (failureRate > 0.05) { // > 5% failure rate
console.warn('High transaction failure rate:', failureRate)
}
}
}, 60000) // Check every minute
const stats = brain.transactionManager?.getStats()
if (stats) {
const failureRate = stats.failedTransactions / stats.totalTransactions
if (failureRate > 0.05) { // > 5% failure rate
console.warn('High transaction failure rate:', failureRate)
}
}
}, 60000) // Check every minute
```
### 5. Understand Atomicity Guarantees
@ -425,25 +425,25 @@ import { describe, it, expect } from 'vitest'
import { Brainy } from '@soulcraft/brainy'
describe('Transaction Tests', () => {
it('should rollback on failure', async () => {
const brain = new Brainy()
await brain.init()
it('should rollback on failure', async () => {
const brain = new Brainy()
await brain.init()
const id1 = await brain.add({ data: { name: 'Entity 1' }, type: NounType.Thing })
const id1 = await brain.add({ data: { name: 'Entity 1' }, type: NounType.Thing })
try {
await brain.add({
data: null as any, // Invalid - will fail
type: NounType.Thing
})
} catch (e) {
// Expected failure
}
try {
await brain.add({
data: null as any, // Invalid - will fail
type: NounType.Thing
})
} catch (e) {
// Expected failure
}
// First entity should still exist (rollback didn't affect it)
const entity1 = await brain.get(id1)
expect(entity1).toBeTruthy()
})
// First entity should still exist (rollback didn't affect it)
const entity1 = await brain.get(id1)
expect(entity1).toBeTruthy()
})
})
```
@ -510,21 +510,21 @@ const stats = brain.transactionManager?.getStats()
```
1. BEGIN
↓
↓
2. ADD OPERATIONS
- SaveNounMetadataOperation
- SaveNounOperation
- UpdateGraphIndexOperation
↓
- SaveNounMetadataOperation
- SaveNounOperation
- UpdateGraphIndexOperation
↓
3. EXECUTE (sequential)
- Execute operation 1 → Success
- Execute operation 2 → Success
- Execute operation 3 → FAILURE
↓
- Execute operation 1 → Success
- Execute operation 2 → Success
- Execute operation 3 → FAILURE
↓
4. ROLLBACK (reverse order)
- Undo operation 2
- Undo operation 1
↓
- Undo operation 2
- Undo operation 1
↓
5. THROW ERROR
```
@ -544,11 +544,11 @@ Transactions use the `StorageAdapter` interface:
```typescript
interface StorageAdapter {
saveNounMetadata(id: string, metadata: NounMetadata): Promise<void>
saveNoun(noun: Noun): Promise<void>
deleteNounMetadata(id: string): Promise<void>
deleteNoun(id: string): Promise<void>
// ... other methods
saveNounMetadata(id: string, metadata: NounMetadata): Promise<void>
saveNoun(noun: Noun): Promise<void>
deleteNounMetadata(id: string): Promise<void>
deleteNoun(id: string): Promise<void>
// ... other methods
}
```
@ -563,11 +563,11 @@ interface StorageAdapter {
## Version History
- **v5.8.0**: Initial transaction system release
- Atomic operations with rollback
- Compatible with COW, sharding, type-aware, distributed
- 36/36 unit tests passing
- 35 integration test scenarios
- Initial transaction system release
- Atomic operations with rollback
- Compatible with COW, sharding, type-aware, distributed
- 36/36 unit tests passing
- 35 integration test scenarios
## Summary

358
docs/vfs/QUICK_START.md
View file

@ -22,13 +22,13 @@ import { Brainy } from '@soulcraft/brainy'
// ✅ CORRECT: Use filesystem storage for production
const brain = new Brainy({
storage: {
type: 'filesystem',
path: './brainy-data' // Your data directory
}
storage: {
type: 'filesystem',
path: './brainy-data' // Your data directory
}
})
await brain.init() // VFS auto-initialized!
await brain.init() // VFS auto-initialized!
console.log('🎉 VFS ready!')
```
@ -47,40 +47,40 @@ const badItems = allNodes.filter(node => node.path.startsWith(dirPath))
```typescript
// ✅ Method 1: Get direct children (recommended for UI)
async function loadDirectoryContents(brain: Brainy, path: string) {
try {
const children = await brain.vfs.getDirectChildren(path)
try {
const children = await brain.vfs.getDirectChildren(path)
// Sort directories first, then files
return children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
} catch (error) {
console.error(`Failed to load ${path}:`, error.message)
return []
}
// Sort directories first, then files
return children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
} catch (error) {
console.error(`Failed to load ${path}:`, error.message)
return []
}
}
// ✅ Method 2: Get complete tree structure (for full trees)
async function loadFullTree(brain: Brainy, path: string) {
const tree = await brain.vfs.getTreeStructure(path, {
maxDepth: 3, // Prevent deep recursion
includeHidden: false, // Skip hidden files
sort: 'name'
})
return tree
const tree = await brain.vfs.getTreeStructure(path, {
maxDepth: 3, // Prevent deep recursion
includeHidden: false, // Skip hidden files
sort: 'name'
})
return tree
}
// ✅ Method 3: Get detailed path info
async function inspectPath(brain: Brainy, path: string) {
const info = await brain.vfs.inspect(path)
return {
isDirectory: info.node.metadata.vfsType === 'directory',
children: info.children,
parent: info.parent,
stats: info.stats
}
const info = await brain.vfs.inspect(path)
return {
isDirectory: info.node.metadata.vfsType === 'directory',
children: info.children,
parent: info.parent,
stats: info.stats
}
}
```
@ -89,19 +89,19 @@ async function inspectPath(brain: Brainy, path: string) {
```typescript
// ✅ Find files by content, not just filename
async function searchFiles(brain: Brainy, query: string, basePath: string = '/') {
const results = await brain.vfs.search(query, {
path: basePath, // Limit search to specific directory
limit: 50, // Reasonable limit
type: 'file' // Only search files, not directories
})
const results = await brain.vfs.search(query, {
path: basePath, // Limit search to specific directory
limit: 50, // Reasonable limit
type: 'file' // Only search files, not directories
})
return results.map(result => ({
path: result.path,
score: result.score,
type: result.type,
size: result.size,
modified: result.modified
}))
return results.map(result => ({
path: result.path,
score: result.score,
type: result.type,
size: result.size,
modified: result.modified
}))
}
// Example usage
@ -118,149 +118,149 @@ import React, { useState, useEffect } from 'react'
import { Brainy } from '@soulcraft/brainy'
export function FileExplorer() {
const [brain, setBrain] = useState(null)
const [currentPath, setCurrentPath] = useState('/')
const [items, setItems] = useState([])
const [loading, setLoading] = useState(true)
const [searchQuery, setSearchQuery] = useState('')
const [brain, setBrain] = useState(null)
const [currentPath, setCurrentPath] = useState('/')
const [items, setItems] = useState([])
const [loading, setLoading] = useState(true)
const [searchQuery, setSearchQuery] = useState('')
// Initialize Brainy (VFS auto-initialized!)
useEffect(() => {
async function initBrainy() {
const brainInstance = new Brainy({
storage: { type: 'filesystem', path: './brainy-data' }
})
await brainInstance.init() // VFS ready after this!
// Initialize Brainy (VFS auto-initialized!)
useEffect(() => {
async function initBrainy() {
const brainInstance = new Brainy({
storage: { type: 'filesystem', path: './brainy-data' }
})
await brainInstance.init() // VFS ready after this!
setBrain(brainInstance)
setLoading(false)
}
initBrainy()
}, [])
setBrain(brainInstance)
setLoading(false)
}
initBrainy()
}, [])
// Load directory contents
const loadDirectory = async (path: string) => {
if (!brain) return
// Load directory contents
const loadDirectory = async (path: string) => {
if (!brain) return
setLoading(true)
try {
// ✅ CORRECT: Use getDirectChildren to prevent recursion
const children = await brain.vfs.getDirectChildren(path)
setLoading(true)
try {
// ✅ CORRECT: Use getDirectChildren to prevent recursion
const children = await brain.vfs.getDirectChildren(path)
// Sort directories first
const sorted = children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
// Sort directories first
const sorted = children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name)
})
setItems(sorted)
setCurrentPath(path)
} catch (error) {
console.error('Failed to load directory:', error)
setItems([])
} finally {
setLoading(false)
}
}
setItems(sorted)
setCurrentPath(path)
} catch (error) {
console.error('Failed to load directory:', error)
setItems([])
} finally {
setLoading(false)
}
}
// Search files
const handleSearch = async () => {
if (!brain || !searchQuery.trim()) {
loadDirectory(currentPath)
return
}
// Search files
const handleSearch = async () => {
if (!brain || !searchQuery.trim()) {
loadDirectory(currentPath)
return
}
setLoading(true)
try {
const results = await brain.vfs.search(searchQuery, {
path: currentPath,
limit: 100
})
setItems(results)
} catch (error) {
console.error('Search failed:', error)
} finally {
setLoading(false)
}
}
setLoading(true)
try {
const results = await brain.vfs.search(searchQuery, {
path: currentPath,
limit: 100
})
setItems(results)
} catch (error) {
console.error('Search failed:', error)
} finally {
setLoading(false)
}
}
// Initial load
useEffect(() => {
if (brain) {
loadDirectory('/')
}
}, [brain])
// Initial load
useEffect(() => {
if (brain) {
loadDirectory('/')
}
}, [brain])
if (loading && !brain) {
return <div>Initializing Brainy...</div>
}
if (loading && !brain) {
return <div>Initializing Brainy...</div>
}
return (
<div className="file-explorer">
{/* Search bar */}
<div className="search-bar">
<input
type="text"
value={searchQuery}
onChange={(e) => setSearchQuery(e.target.value)}
placeholder="Search files by content..."
onKeyPress={(e) => e.key === 'Enter' && handleSearch()}
/>
<button onClick={handleSearch}>Search</button>
{searchQuery && (
<button onClick={() => { setSearchQuery(''); loadDirectory(currentPath) }}>
Clear
</button>
)}
</div>
return (
<div className="file-explorer">
{/* Search bar */}
<div className="search-bar">
<input
type="text"
value={searchQuery}
onChange={(e) => setSearchQuery(e.target.value)}
placeholder="Search files by content..."
onKeyPress={(e) => e.key === 'Enter' && handleSearch()}
/>
<button onClick={handleSearch}>Search</button>
{searchQuery && (
<button onClick={() => { setSearchQuery(''); loadDirectory(currentPath) }}>
Clear
</button>
)}
</div>
{/* Current path */}
<div className="current-path">
📁 {currentPath}
{currentPath !== '/' && (
<button onClick={() => loadDirectory(currentPath.split('/').slice(0, -1).join('/') || '/')}>
⬆️ Up
</button>
)}
</div>
{/* Current path */}
<div className="current-path">
📁 {currentPath}
{currentPath !== '/' && (
<button onClick={() => loadDirectory(currentPath.split('/').slice(0, -1).join('/') || '/')}>
⬆️ Up
</button>
)}
</div>
{/* File list */}
{loading ? (
<div>Loading...</div>
) : (
<div className="file-list">
{items.map((item) => (
<div
key={item.id}
className={`file-item ${item.metadata.vfsType}`}
onClick={() => {
if (item.metadata.vfsType === 'directory') {
loadDirectory(item.metadata.path)
} else {
console.log('Open file:', item.metadata.path)
// Add your file opening logic here
}
}}
>
<span className="icon">
{item.metadata.vfsType === 'directory' ? '📁' : '📄'}
</span>
<span className="name">{item.metadata.name}</span>
<span className="size">
{item.metadata.size ? `${Math.round(item.metadata.size / 1024)}KB` : ''}
</span>
</div>
))}
{items.length === 0 && (
<div className="empty">
{searchQuery ? 'No results found' : 'Empty directory'}
</div>
)}
</div>
)}
</div>
)
{/* File list */}
{loading ? (
<div>Loading...</div>
) : (
<div className="file-list">
{items.map((item) => (
<div
key={item.id}
className={`file-item ${item.metadata.vfsType}`}
onClick={() => {
if (item.metadata.vfsType === 'directory') {
loadDirectory(item.metadata.path)
} else {
console.log('Open file:', item.metadata.path)
// Add your file opening logic here
}
}}
>
<span className="icon">
{item.metadata.vfsType === 'directory' ? '📁' : '📄'}
</span>
<span className="name">{item.metadata.name}</span>
<span className="size">
{item.metadata.size ? `${Math.round(item.metadata.size / 1024)}KB` : ''}
</span>
</div>
))}
{items.length === 0 && (
<div className="empty">
{searchQuery ? 'No results found' : 'Empty directory'}
</div>
)}
</div>
)}
</div>
)
}
```
@ -288,13 +288,13 @@ Your file explorer is now working! Here's what to explore next:
### "Module not found" errors
```bash
# Make sure you're using the right import
npm ls @soulcraft/brainy # Check version
npm install @soulcraft/brainy@latest # Update if needed
npm ls @soulcraft/brainy # Check version
npm install @soulcraft/brainy@latest # Update if needed
```
### "VFS not initialized" errors
```typescript
// v5.1.0+: Just await brain.init() - VFS is auto-initialized!
// Just await brain.init() - VFS is auto-initialized!
await brain.init()
// VFS ready - use brain.vfs directly
await brain.vfs.writeFile('/test.txt', 'data')
@ -304,8 +304,8 @@ await brain.vfs.writeFile('/test.txt', 'data')
```typescript
// Add pagination for large directories
const children = await brain.vfs.getDirectChildren(path, {
limit: 100, // Load only first 100 items
offset: 0 // Start from beginning
limit: 100, // Load only first 100 items
offset: 0 // Start from beginning
})
```
@ -313,8 +313,8 @@ const children = await brain.vfs.getDirectChildren(path, {
```typescript
// Make sure files are imported into VFS first
await brain.vfs.importDirectory('./my-files', {
recursive: true,
extractMetadata: true // Enable content understanding
recursive: true,
extractMetadata: true // Enable content understanding
})
```

348
docs/vfs/README.md
View file

@ -28,15 +28,15 @@ import { VirtualFileSystem } from '@soulcraft/brainy/vfs'
// Initialize the VFS
const vfs = new VirtualFileSystem({
root: '/my-brain',
intelligent: true // Enable AI features
root: '/my-brain',
intelligent: true // Enable AI features
})
await vfs.init()
// Write a file - it automatically becomes intelligent
await vfs.writeFile('/projects/my-app/index.js',
'console.log("Hello, World!")')
'console.log("Hello, World!")')
// Find similar files using semantic search
const similar = await vfs.findSimilar('/projects/my-app/index.js')
@ -48,11 +48,11 @@ const results = await vfs.search('files about authentication')
await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements')
```
## ⚡ Performance (v5.11.1)
## ⚡ Performance
**75% Faster File Operations!** VFS now automatically benefits from brain.get() metadata-only optimization:
| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup |
| Operation | Before | After | Speedup |
|-----------|------------------|-----------------|---------|
| `readFile()` | 53ms | **~13ms** | **75%** |
| `stat()` | 53ms | **~13ms** | **75%** |
@ -60,7 +60,7 @@ await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements')
**Zero configuration** - automatic optimization for all VFS operations!
VFS operations only need metadata (path, size, timestamps), not 384-dimensional vector embeddings. The v5.11.1 optimization automatically uses metadata-only reads, saving 95% bandwidth and 76-81% time.
VFS operations only need metadata (path, size, timestamps), not 384-dimensional vector embeddings. The optimization automatically uses metadata-only reads, saving 95% bandwidth and 76-81% time.
## Core Features
@ -110,20 +110,20 @@ const docs = await vfs.search('technical documentation for API endpoints')
// Find similar files
const similar = await vfs.findSimilar('/code/auth.js', {
limit: 5,
threshold: 0.8 // 80% similarity
limit: 5,
threshold: 0.8 // 80% similarity
})
// Get related files through the knowledge graph
const related = await vfs.getRelated('/proposal.pdf', {
depth: 2 // Include relationships of relationships
depth: 2 // Include relationships of relationships
})
// Auto-organization suggestions
const suggestions = await vfs.suggestOrganization([
'/downloads/doc1.pdf',
'/downloads/image.jpg',
'/downloads/code.py'
'/downloads/doc1.pdf',
'/downloads/image.jpg',
'/downloads/code.py'
])
// Returns: suggested folders and categorization
```
@ -144,10 +144,10 @@ const connections = await vfs.getConnections('/code/impl.js')
// Traverse the graph
const implementations = await vfs.search('', {
connected: {
to: '/spec.md',
via: 'implements'
}
connected: {
to: '/spec.md',
via: 'implements'
}
})
```
@ -158,8 +158,8 @@ Store anything alongside your files:
```javascript
// Add todos to files
await vfs.setTodos('/projects/app/index.js', [
{ task: 'Add error handling', priority: 'high', due: '2024-01-20' },
{ task: 'Optimize performance', priority: 'medium' }
{ task: 'Add error handling', priority: 'high', due: '2024-01-20' },
{ task: 'Optimize performance', priority: 'medium' }
])
// Set custom attributes
@ -169,11 +169,11 @@ await vfs.setxattr('/video.mp4', 'tags', ['tutorial', 'react', 'hooks'])
// Query by metadata
const urgent = await vfs.search('', {
where: { 'todos.priority': 'high' }
where: { 'todos.priority': 'high' }
})
const parisPhotos = await vfs.search('', {
where: { location: 'Paris, France' }
where: { location: 'Paris, France' }
})
```
@ -184,20 +184,20 @@ Access files through semantic dimensions (see [Semantic VFS](./SEMANTIC_VFS.md))
```javascript
// Query-based path access (current functionality)
const authFiles = await vfs.search('', {
where: { concepts: { contains: 'authentication' }}
where: { concepts: { contains: 'authentication' }}
})
// Find files by custom metadata
const recent = await vfs.search('', {
where: {
type: 'document',
modified: { greaterThan: Date.now() - 7*24*60*60*1000 }
}
where: {
type: 'document',
modified: { greaterThan: Date.now() - 7*24*60*60*1000 }
}
})
// Find similar files
const similar = await vfs.findSimilar('/examples/good-code.js', {
threshold: 0.7
threshold: 0.7
})
```
@ -210,24 +210,24 @@ const similar = await vfs.findSimilar('/examples/good-code.js', {
```javascript
// Store a research paper with automatic analysis
await vfs.writeFile('/research/quantum-computing.pdf', pdfBuffer, {
metadata: {
authors: ['Dr. Alice Smith', 'Dr. Bob Jones'],
year: 2024,
topics: ['quantum', 'computing', 'algorithms'],
citations: 42
}
metadata: {
authors: ['Dr. Alice Smith', 'Dr. Bob Jones'],
year: 2024,
topics: ['quantum', 'computing', 'algorithms'],
citations: 42
}
})
// Find all papers on similar topics
const related = await vfs.search('quantum algorithms', {
type: 'document',
where: { year: { $gte: 2020 } }
type: 'document',
where: { year: { $gte: 2020 } }
})
// Find papers that cite this one
const citations = await vfs.getConnections('/research/quantum-computing.pdf', {
type: 'cites',
direction: 'incoming'
type: 'cites',
direction: 'incoming'
})
```
@ -245,12 +245,12 @@ await vfs.writeFile('/src/utils/auth.js', authCode)
// Find all files that import this module
const importers = await vfs.search('', {
where: { dependencies: 'utils/auth.js' }
where: { dependencies: 'utils/auth.js' }
})
// Find test files for this code
const tests = await vfs.getRelated('/src/utils/auth.js', {
type: 'tests'
type: 'tests'
})
// Find similar implementations
@ -262,12 +262,12 @@ const similar = await vfs.findSimilar('/src/utils/auth.js')
```javascript
// Store media with rich metadata
await vfs.writeFile('/photos/sunset.jpg', imageBuffer, {
metadata: {
camera: 'Canon R5',
location: { lat: 37.7749, lng: -122.4194 },
tags: ['sunset', 'golden-gate', 'landscape'],
album: 'San Francisco 2024'
}
metadata: {
camera: 'Canon R5',
location: { lat: 37.7749, lng: -122.4194 },
tags: ['sunset', 'golden-gate', 'landscape'],
album: 'San Francisco 2024'
}
})
// Find similar images
@ -275,18 +275,18 @@ const similar = await vfs.findSimilar('/photos/sunset.jpg')
// Find photos by location
const nearby = await vfs.search('', {
type: 'image',
where: {
'location.lat': { $between: [37.7, 37.8] },
'location.lng': { $between: [-122.5, -122.3] }
}
type: 'image',
where: {
'location.lat': { $between: [37.7, 37.8] },
'location.lng': { $between: [-122.5, -122.3] }
}
})
// Smart albums
await vfs.createVirtualDirectory('/albums/best-sunsets', {
query: 'sunset',
type: 'image',
where: { rating: { $gte: 4 } }
query: 'sunset',
type: 'image',
where: { rating: { $gte: 4 } }
})
```
@ -308,25 +308,25 @@ await vfs.setxattr(projectPath, 'status', 'in-progress')
// Add todos to specific files
await vfs.setTodos(`${projectPath}/src/index.js`, [
{ task: 'Implement user authentication', assignee: 'Alice' },
{ task: 'Add error handling', assignee: 'Bob' }
{ task: 'Implement user authentication', assignee: 'Alice' },
{ task: 'Add error handling', assignee: 'Bob' }
])
// Find all files with pending todos
const pending = await vfs.search('', {
where: {
path: { $startsWith: projectPath },
'todos.status': 'pending'
}
where: {
path: { $startsWith: projectPath },
'todos.status': 'pending'
}
})
// Find projects nearing deadline
const urgent = await vfs.search('', {
where: {
type: 'directory',
'deadline': { $lte: '2024-02-01' },
'status': 'in-progress'
}
where: {
type: 'directory',
'deadline': { $lte: '2024-02-01' },
'status': 'in-progress'
}
})
```
@ -417,9 +417,9 @@ The VFS is built with production scalability in mind:
#### **Storage Strategy**
```typescript
// Adaptive storage based on file size
< 100KB: Inline storage (entity.data)
< 10MB: External reference (S3/R2 key)
> 10MB: Chunked storage (parallel chunks)
< 100KB: Inline storage (entity.data)
< 10MB: External reference (S3/R2 key)
> 10MB: Chunked storage (parallel chunks)
```
#### **Performance Metrics**
@ -434,24 +434,24 @@ The VFS is built with production scalability in mind:
#### **How It Handles Scale**
1. **Hierarchical Caching**
- 100K+ path cache entries
- Parent-child relationship caching
- Hot path detection and optimization
- 100K+ path cache entries
- Parent-child relationship caching
- Hot path detection and optimization
2. **Distributed Architecture**
- Sharding by path prefix
- Read replicas for hot directories
- CDN integration for static files
- Sharding by path prefix
- Read replicas for hot directories
- CDN integration for static files
3. **Intelligent Indexing**
- Compound indexes on (parent, name)
- Vector indexes for semantic search
- Graph indexes for relationships
- Compound indexes on (parent, name)
- Vector indexes for semantic search
- Graph indexes for relationships
4. **Streaming Everything**
- Large files never fully in memory
- Progressive loading
- Chunked transfers
- Large files never fully in memory
- Progressive loading
- Chunked transfers
### Real Production Scenarios
@ -462,29 +462,29 @@ Store build artifacts with automatic relationships:
```javascript
// Store build output with metadata
await vfs.writeFile('/builds/v1.2.3/app.js', buildOutput, {
metadata: {
commit: 'abc123',
branch: 'main',
timestamp: Date.now(),
tests: 'passing',
coverage: 0.92
}
metadata: {
commit: 'abc123',
branch: 'main',
timestamp: Date.now(),
tests: 'passing',
coverage: 0.92
}
})
// Find all builds for a commit
const builds = await vfs.search('', {
where: { commit: 'abc123' }
where: { commit: 'abc123' }
})
// Get latest passing build
const latest = await vfs.search('', {
where: {
branch: 'main',
tests: 'passing'
},
sort: 'modified',
order: 'desc',
limit: 1
where: {
branch: 'main',
tests: 'passing'
},
sort: 'modified',
order: 'desc',
limit: 1
})
```
@ -495,8 +495,8 @@ Isolate customer data with semantic understanding:
```javascript
// Each tenant gets their own root
const tenantVfs = new VirtualFileSystem({
root: `/tenants/${tenantId}`,
service: tenantId // Isolate at Brainy level too
root: `/tenants/${tenantId}`,
service: tenantId // Isolate at Brainy level too
})
// Tenant uploads document
@ -505,11 +505,11 @@ await tenantVfs.writeFile('/documents/contract.pdf', pdfBuffer)
// Cross-tenant analytics (admin only)
const adminVfs = new VirtualFileSystem({ root: '/tenants' })
const stats = await adminVfs.search('contract', {
recursive: true,
aggregations: {
byTenant: { field: 'service' },
byType: { field: 'mimeType' }
}
recursive: true,
aggregations: {
byTenant: { field: 'service' },
byType: { field: 'mimeType' }
}
})
```
@ -520,38 +520,38 @@ Connect datasets, models, and results:
```javascript
// Store training data
await vfs.writeFile('/datasets/train.csv', csvData, {
metadata: {
samples: 100000,
features: 50,
labels: 10
}
metadata: {
samples: 100000,
features: 50,
labels: 10
}
})
// Store trained model
await vfs.writeFile('/models/v1/model.pkl', modelBuffer, {
metadata: {
algorithm: 'random-forest',
accuracy: 0.95,
trainedOn: '/datasets/train.csv',
hyperparameters: { trees: 100, depth: 10 }
}
metadata: {
algorithm: 'random-forest',
accuracy: 0.95,
trainedOn: '/datasets/train.csv',
hyperparameters: { trees: 100, depth: 10 }
}
})
// Connect model to its training data
await vfs.addRelationship(
'/models/v1/model.pkl',
'/datasets/train.csv',
'trained-on'
'/models/v1/model.pkl',
'/datasets/train.csv',
'trained-on'
)
// Find best model for a dataset
const models = await vfs.search('', {
connected: {
to: '/datasets/train.csv',
via: 'trained-on'
},
sort: 'accuracy',
order: 'desc'
connected: {
to: '/datasets/train.csv',
via: 'trained-on'
},
sort: 'accuracy',
order: 'desc'
})
```
@ -562,30 +562,30 @@ Intelligent content organization:
```javascript
// Auto-organize uploads
vfs.on('file:added', async (path) => {
if (path.startsWith('/uploads/')) {
const file = await vfs.getEntity(path)
if (path.startsWith('/uploads/')) {
const file = await vfs.getEntity(path)
// Auto-categorize by AI
const category = await detectCategory(file)
const newPath = `/content/${category}/${file.metadata.name}`
// Auto-categorize by AI
const category = await detectCategory(file)
const newPath = `/content/${category}/${file.metadata.name}`
await vfs.move(path, newPath)
await vfs.move(path, newPath)
// Auto-tag
const tags = await extractTags(file)
await vfs.setxattr(newPath, 'tags', tags)
// Auto-tag
const tags = await extractTags(file)
await vfs.setxattr(newPath, 'tags', tags)
// Find related content
const related = await vfs.findSimilar(newPath, {
limit: 5,
threshold: 0.8
})
// Find related content
const related = await vfs.findSimilar(newPath, {
limit: 5,
threshold: 0.8
})
// Create relationships
for (const rel of related) {
await vfs.addRelationship(newPath, rel.path, 'related-to')
}
}
// Create relationships
for (const rel of related) {
await vfs.addRelationship(newPath, rel.path, 'related-to')
}
}
})
```
@ -596,39 +596,39 @@ Collaborative file management:
```javascript
// Track file ownership and access
await vfs.writeFile('/projects/alpha/spec.md', content, {
metadata: {
owner: userId,
team: 'engineering',
permissions: {
[userId]: 'rw',
'team:engineering': 'r',
'others': '-'
}
}
metadata: {
owner: userId,
team: 'engineering',
permissions: {
[userId]: 'rw',
'team:engineering': 'r',
'others': '-'
}
}
})
// Add collaborative features
await vfs.addTodo('/projects/alpha/spec.md', {
task: 'Review security section',
assignee: 'alice@company.com',
due: '2024-02-01',
priority: 'high'
task: 'Review security section',
assignee: 'alice@company.com',
due: '2024-02-01',
priority: 'high'
})
// Track who's working on what
await vfs.setxattr('/projects/alpha/spec.md', 'locks', {
section3: {
user: 'bob@company.com',
since: Date.now()
}
section3: {
user: 'bob@company.com',
since: Date.now()
}
})
// Find all files assigned to a user
const assigned = await vfs.search('', {
where: {
'todos.assignee': 'alice@company.com',
'todos.status': 'pending'
}
where: {
'todos.assignee': 'alice@company.com',
'todos.status': 'pending'
}
})
```
@ -643,15 +643,15 @@ const assigned = await vfs.search('', {
#### **Standalone Mode**
```javascript
const vfs = new VirtualFileSystem()
await vfs.init() // Uses in-memory storage
await vfs.init() // Uses in-memory storage
```
#### **Local Persistence**
```javascript
const vfs = new VirtualFileSystem()
await vfs.init({
storage: 'filesystem',
dataDir: '/var/lib/brainy-vfs'
storage: 'filesystem',
dataDir: '/var/lib/brainy-vfs'
})
```
@ -659,9 +659,9 @@ await vfs.init({
```javascript
const vfs = new VirtualFileSystem()
await vfs.init({
storage: 's3',
bucket: 'my-vfs-data',
region: 'us-west-2'
storage: 's3',
bucket: 'my-vfs-data',
region: 'us-west-2'
})
```
@ -669,14 +669,14 @@ await vfs.init({
```javascript
const vfs = new VirtualFileSystem()
await vfs.init({
distributed: true,
nodes: [
'vfs1.internal:8080',
'vfs2.internal:8080',
'vfs3.internal:8080'
],
replication: 3,
consistency: 'eventual'
distributed: true,
nodes: [
'vfs1.internal:8080',
'vfs2.internal:8080',
'vfs3.internal:8080'
],
replication: 3,
consistency: 'eventual'
})
```

84
docs/vfs/TROUBLESHOOTING.md
View file

@ -12,8 +12,8 @@
**Root Cause:**
The root directory entity exists but doesn't have the proper metadata structure.
**Solution (v3.15.0+):**
This issue has been fixed in v3.15.0. The VFS now:
**Solution:**
This issue has been fixed. The VFS now:
1. Ensures root directory has `vfsType: 'directory'` metadata
2. Adds compatibility layer for entities with malformed metadata
3. Automatically repairs metadata on entity retrieval
@ -21,17 +21,17 @@ This issue has been fixed in v3.15.0. The VFS now:
**Manual Fix (if needed):**
```javascript
// Force re-initialization of root directory
await vfs.init() // Will repair root if needed
await vfs.init() // Will repair root if needed
// Or manually update root entity
const rootId = vfs.rootEntityId
await brain.update({
id: rootId,
metadata: {
path: '/',
vfsType: 'directory',
// ... other metadata
}
id: rootId,
metadata: {
path: '/',
vfsType: 'directory',
// ... other metadata
}
})
```
@ -47,7 +47,7 @@ await brain.update({
**Root Cause:**
Contains relationships are missing between parent directories and files.
**Solution (v3.15.0+):**
**Solution:**
This issue has been fixed. The VFS now:
1. Creates Contains relationships when writing new files
2. Ensures Contains relationships exist when updating files
@ -60,9 +60,9 @@ const parentId = await vfs.resolvePath('/directory')
const fileId = await vfs.resolvePath('/directory/file.txt')
await brain.relate({
from: parentId,
to: fileId,
type: VerbType.Contains
from: parentId,
to: fileId,
type: VerbType.Contains
})
```
@ -80,8 +80,8 @@ The VFS `init()` method wasn't called after getting the VFS instance.
**Solution:**
```javascript
// ✅ CORRECT
const vfs = brain.vfs() // Get instance
await vfs.init() // Initialize (REQUIRED!)
const vfs = brain.vfs() // Get instance
await vfs.init() // Initialize (REQUIRED!)
// ❌ WRONG
const vfs = brain.vfs()
@ -104,15 +104,15 @@ Using in-memory storage instead of persistent storage.
```javascript
// ✅ Use persistent storage
const brain = new Brainy({
storage: {
type: 'filesystem', // Persistent
path: './brainy-data'
}
storage: {
type: 'filesystem', // Persistent
path: './brainy-data'
}
})
// ❌ Don't use memory for production
const brain = new Brainy({
storage: { type: 'memory' } // Data lost on restart!
storage: { type: 'memory' } // Data lost on restart!
})
```
@ -136,7 +136,7 @@ const children = await vfs.getDirectChildren('/directory')
// ❌ WRONG - Path prefix matching causes recursion
const allNodes = await brain.find({})
const children = allNodes.filter(n =>
n.metadata.path.startsWith('/directory/')) // Directory matches itself!
n.metadata.path.startsWith('/directory/')) // Directory matches itself!
```
---
@ -153,13 +153,13 @@ Files aren't being properly embedded or indexed.
**Solution:**
```javascript
// Ensure files have content for embedding
await vfs.writeFile('/doc.txt', 'Actual content here') // Not empty!
await vfs.writeFile('/doc.txt', 'Actual content here') // Not empty!
// Use semantic search correctly
const results = await vfs.search('machine learning', {
path: '/documents', // Search within path
limit: 10,
type: 'file'
path: '/documents', // Search within path
limit: 10,
type: 'file'
})
```
@ -184,8 +184,8 @@ console.log('VFS type:', entity.metadata.vfsType)
```javascript
const parentId = await vfs.resolvePath('/directory')
const relations = await brain.getRelations({
from: parentId,
type: VerbType.Contains
from: parentId,
type: VerbType.Contains
})
console.log('Child count:', relations.length)
```
@ -193,11 +193,11 @@ console.log('Child count:', relations.length)
### 4. Enable Debug Logging
```javascript
const brain = new Brainy({
storage: { type: 'filesystem' },
logger: {
level: 'debug',
enabled: true
}
storage: { type: 'filesystem' },
logger: {
level: 'debug',
enabled: true
}
})
```
@ -209,11 +209,11 @@ const brain = new Brainy({
```javascript
// Enable caching in VFS config
const vfs = brain.vfs({
cache: {
enabled: true,
ttl: 300000, // 5 minutes
maxSize: 1000
}
cache: {
enabled: true,
ttl: 300000, // 5 minutes
maxSize: 1000
}
})
```
@ -221,12 +221,12 @@ const vfs = brain.vfs({
```javascript
// Write multiple files efficiently
const files = [
{ path: '/file1.txt', content: 'content1' },
{ path: '/file2.txt', content: 'content2' }
{ path: '/file1.txt', content: 'content1' },
{ path: '/file2.txt', content: 'content2' }
]
await Promise.all(
files.map(f => vfs.writeFile(f.path, f.content))
files.map(f => vfs.writeFile(f.path, f.content))
)
```
@ -234,8 +234,8 @@ await Promise.all(
```javascript
// Don't traverse too deep
const tree = await vfs.getTreeStructure('/', {
maxDepth: 3, // Limit recursion
includeHidden: false
maxDepth: 3, // Limit recursion
includeHidden: false
})
```

4
docs/vfs/VFS_CORE.md
View file

@ -418,7 +418,7 @@ await vfs.importDirectory()// Import directory from filesystem
## Bulk Write Operations
**v6.5.0**: `bulkWrite` efficiently processes multiple VFS operations with automatic ordering to prevent race conditions.
`bulkWrite` efficiently processes multiple VFS operations with automatic ordering to prevent race conditions.
```javascript
const result = await vfs.bulkWrite([
@ -439,7 +439,7 @@ console.log(`Successful: ${result.successful}, Failed: ${result.failed.length}`)
- `delete` - Delete file
- `update` - Update file metadata only
**Operation ordering (v6.5.0):**
**Operation ordering:**
1. `mkdir` operations run **first**, sequentially, sorted by path depth (shallowest first)
2. Other operations (`write`, `delete`, `update`) run **after** in parallel batches of 10

8
docs/vfs/VFS_INITIALIZATION.md
View file

@ -1,4 +1,4 @@
# VFS Initialization Guide (v5.1.0+)
# VFS Initialization Guide
## Quick Start
@ -30,7 +30,7 @@ await vfs.init() // ❌ Separate initialization
await vfs.writeFile(...)
```
### After (v5.1.0+):
### After:
```javascript
const brain = new Brainy(...)
await brain.init() // VFS auto-initialized!
@ -53,7 +53,7 @@ await vfs.init()
await vfs.writeFile('/test.txt', 'data')
```
### New Pattern (v5.1.0+):
### New Pattern:
```javascript
// Just remove the () and init() call
await brain.vfs.writeFile('/test.txt', 'data')
@ -142,7 +142,7 @@ await brain.init() // Required!
await brain.vfs.writeFile(...) // Now this works
```
## Fork Support (v5.0.0+)
## Fork Support
VFS works seamlessly with Brainy's Copy-on-Write (COW) fork feature:

12
src/api/ConfigAPI.ts
View file

@ -60,7 +60,7 @@ export class ConfigAPI {
// Store in cache
this.configCache.set(key, entry)
// v4.0.0: Persist to storage as NounMetadata
// Persist to storage as NounMetadata
const configId = this.CONFIG_NOUN_PREFIX + key
const nounMetadata = {
noun: 'config' as any,
@ -94,7 +94,7 @@ export class ConfigAPI {
return defaultValue
}
// v4.0.0: Extract ConfigEntry from NounMetadata
// Extract ConfigEntry from NounMetadata
const createdAtVal = typeof metadata.createdAt === 'object' && metadata.createdAt !== null && 'seconds' in metadata.createdAt
? metadata.createdAt.seconds * 1000 + Math.floor((metadata.createdAt.nanoseconds || 0) / 1000000)
: ((metadata.createdAt as unknown as number) || Date.now())
@ -140,7 +140,7 @@ export class ConfigAPI {
// Remove from cache
this.configCache.delete(key)
// v4.0.0: Remove from storage
// Remove from storage
const configId = this.CONFIG_NOUN_PREFIX + key
await this.storage.deleteNounMetadata(configId)
}
@ -149,7 +149,7 @@ export class ConfigAPI {
* List all configuration keys
*/
async list(): Promise<string[]> {
// v4.0.0: Get all nouns and filter for config entries
// Get all nouns and filter for config entries
const result = await this.storage.getNouns({ pagination: { limit: 10000 } })
const configKeys: string[] = []
@ -213,7 +213,7 @@ export class ConfigAPI {
for (const [key, entry] of Object.entries(config)) {
this.configCache.set(key, entry)
const configId = this.CONFIG_NOUN_PREFIX + key
// v4.0.0: Convert ConfigEntry to NounMetadata
// Convert ConfigEntry to NounMetadata
const nounMetadata = {
noun: 'config' as any,
...entry,
@ -240,7 +240,7 @@ export class ConfigAPI {
return null
}
// v4.0.0: Extract ConfigEntry from NounMetadata
// Extract ConfigEntry from NounMetadata
const createdAtVal = typeof metadata.createdAt === 'object' && metadata.createdAt !== null && 'seconds' in metadata.createdAt
? metadata.createdAt.seconds * 1000 + Math.floor((metadata.createdAt.nanoseconds || 0) / 1000000)
: ((metadata.createdAt as unknown as number) || Date.now())

2
src/api/DataAPI.ts
View file

@ -210,7 +210,7 @@ export class DataAPI {
this.validateImportItem(mapped)
}
// v4.0.0: Save entity - separate vector and metadata
// Save entity - separate vector and metadata
const id = mapped.id || this.generateId()
const noun: HNSWNoun = {
id,

10
src/augmentations/brainyAugmentation.ts
View file

@ -375,7 +375,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
}
/**
* Get CommitLog for temporal features (v5.0.0+)
* Get CommitLog for temporal features
*
* Provides access to commit history for time-travel queries, audit trails,
* and branch management. Available after initialize() is called.
@ -414,7 +414,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
}
/**
* Get BlobStorage for content-addressable storage (v5.0.0+)
* Get BlobStorage for content-addressable storage
*
* Provides access to the underlying blob storage system for storing
* and retrieving content-addressed data. Available after initialize() is called.
@ -454,7 +454,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
}
/**
* Get RefManager for branch/ref management (v5.0.0+)
* Get RefManager for branch/ref management
*
* Provides access to the reference manager for creating, updating,
* and managing Git-style branches and refs. Available after initialize() is called.
@ -496,7 +496,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
}
/**
* Get current branch name (v5.0.0+)
* Get current branch name
*
* Convenience helper for getting the current branch from the Brainy instance.
* Available after initialize() is called.
@ -525,7 +525,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
if (typeof brain.getCurrentBranch !== 'function') {
throw new Error(
`${this.name}: getCurrentBranch() not available on Brainy instance. ` +
`This method requires Brainy v5.0.0+.`
`This method requires Brainy with VFS support.`
)
}

2
src/augmentations/intelligentImport/FormatHandlerRegistry.ts
View file

@ -1,5 +1,5 @@
/**
* Format Handler Registry (v5.2.0)
* Format Handler Registry
*
* Central registry for format handlers with:
* - MIME type-based routing

2
src/augmentations/intelligentImport/IntelligentImportAugmentation.ts
View file

@ -35,7 +35,7 @@ export class IntelligentImportAugmentation extends BaseAugmentation {
enableCSV: true,
enableExcel: true,
enablePDF: true,
enableImage: true, // v5.2.0: Image handler enabled by default
enableImage: true, // Image handler enabled by default
maxFileSize: 100 * 1024 * 1024, // 100MB default
enableCache: true,
cacheTTL: 24 * 60 * 60 * 1000, // 24 hours

12
src/augmentations/intelligentImport/handlers/csvHandler.ts
View file

@ -40,7 +40,7 @@ export class CSVHandler extends BaseFormatHandler {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
const totalBytes = buffer.length
// v4.5.0: Report total bytes for progress tracking
// Report total bytes for progress tracking
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0)
}
@ -55,7 +55,7 @@ export class CSVHandler extends BaseFormatHandler {
// Detect delimiter if not specified
const delimiter = options.csvDelimiter || this.detectDelimiter(text)
// v4.5.0: Report progress - parsing started
// Report progress - parsing started
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
}
@ -75,7 +75,7 @@ export class CSVHandler extends BaseFormatHandler {
cast: false // We'll do type inference ourselves
})
// v4.5.0: Report bytes processed (entire file parsed)
// Report bytes processed (entire file parsed)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
@ -83,7 +83,7 @@ export class CSVHandler extends BaseFormatHandler {
// Convert to array of objects
const data = Array.isArray(records) ? records : [records]
// v4.5.0: Report data extraction progress
// Report data extraction progress
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length)
}
@ -101,7 +101,7 @@ export class CSVHandler extends BaseFormatHandler {
converted[key] = this.convertValue(value, types[key] || 'string')
}
// v4.5.0: Report progress every 1000 rows
// Report progress every 1000 rows
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
}
@ -111,7 +111,7 @@ export class CSVHandler extends BaseFormatHandler {
const processingTime = Date.now() - startTime
// v4.5.0: Final progress update
// Final progress update
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
}

18
src/augmentations/intelligentImport/handlers/excelHandler.ts
View file

@ -27,7 +27,7 @@ export class ExcelHandler extends BaseFormatHandler {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'binary')
const totalBytes = buffer.length
// v4.5.0: Report start
// Report start
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0)
}
@ -47,7 +47,7 @@ export class ExcelHandler extends BaseFormatHandler {
// Determine which sheets to process
const sheetsToProcess = this.getSheetsToProcess(workbook, options)
// v4.5.0: Report workbook loaded
// Report workbook loaded
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing ${sheetsToProcess.length} sheets...`)
}
@ -59,7 +59,7 @@ export class ExcelHandler extends BaseFormatHandler {
for (let sheetIndex = 0; sheetIndex < sheetsToProcess.length; sheetIndex++) {
const sheetName = sheetsToProcess[sheetIndex]
// v4.5.0: Report current sheet
// Report current sheet
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(
`Reading sheet: ${sheetName} (${sheetIndex + 1}/${sheetsToProcess.length})`
@ -117,19 +117,19 @@ export class ExcelHandler extends BaseFormatHandler {
headers
}
// v4.5.0: Estimate bytes processed (sheets are sequential)
// Estimate bytes processed (sheets are sequential)
const bytesProcessed = Math.floor(((sheetIndex + 1) / sheetsToProcess.length) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// v4.5.0: Report extraction progress
// Report extraction progress
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until complete
}
}
// v4.5.0: Report data extraction complete
// Report data extraction complete
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Extracted ${allData.length} rows, inferring types...`)
}
@ -152,7 +152,7 @@ export class ExcelHandler extends BaseFormatHandler {
}
}
// v4.5.0: Report progress every 1000 rows (avoid spam)
// Report progress every 1000 rows (avoid spam)
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${allData.length} rows...`)
}
@ -160,14 +160,14 @@ export class ExcelHandler extends BaseFormatHandler {
return converted
})
// v4.5.0: Final progress - all bytes processed
// Final progress - all bytes processed
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
const processingTime = Date.now() - startTime
// v4.5.0: Report completion
// Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(
`Excel complete: ${sheetsToProcess.length} sheets, ${convertedData.length} rows`

6
src/augmentations/intelligentImport/handlers/imageHandler.ts
View file

@ -1,5 +1,5 @@
/**
* Image Import Handler (v5.8.0 - Pure JavaScript)
* Image Import Handler
*
* Handles image files with:
* - EXIF metadata extraction (camera, GPS, timestamps) via exifr
@ -7,7 +7,7 @@
* - Support for JPEG, PNG, WebP, GIF, TIFF, BMP, SVG
*
* NO NATIVE DEPENDENCIES - Pure JavaScript implementation
* Replaces Sharp (v5.7.x) with lightweight pure-JS alternatives
* Replaces Sharp with lightweight pure-JS alternatives
*/
import { BaseFormatHandler } from './base.js'
@ -96,7 +96,7 @@ export interface ImageHandlerOptions extends FormatHandlerOptions {
* Enables developers to import images into the knowledge graph with
* full metadata extraction.
*
* v5.8.0: Pure JavaScript implementation (no native dependencies)
* Pure JavaScript implementation (no native dependencies)
*/
export class ImageHandler extends BaseFormatHandler {
readonly format = 'image'

14
src/augmentations/intelligentImport/handlers/pdfHandler.ts
View file

@ -52,7 +52,7 @@ export class PDFHandler extends BaseFormatHandler {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'binary')
const totalBytes = buffer.length
// v4.5.0: Report start
// Report start
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0)
}
@ -74,7 +74,7 @@ export class PDFHandler extends BaseFormatHandler {
const metadata = await pdfDoc.getMetadata()
const numPages = pdfDoc.numPages
// v4.5.0: Report document loaded
// Report document loaded
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing ${numPages} pages...`)
}
@ -85,7 +85,7 @@ export class PDFHandler extends BaseFormatHandler {
let detectedTables = 0
for (let pageNum = 1; pageNum <= numPages; pageNum++) {
// v4.5.0: Report current page
// Report current page
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing page ${pageNum} of ${numPages}`)
}
@ -131,19 +131,19 @@ export class PDFHandler extends BaseFormatHandler {
}
}
// v4.5.0: Estimate bytes processed (pages are sequential)
// Estimate bytes processed (pages are sequential)
const bytesProcessed = Math.floor((pageNum / numPages) * totalBytes)
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed)
}
// v4.5.0: Report extraction progress
// Report extraction progress
if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until complete
}
}
// v4.5.0: Final progress - all bytes processed
// Final progress - all bytes processed
if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes)
}
@ -153,7 +153,7 @@ export class PDFHandler extends BaseFormatHandler {
const processingTime = Date.now() - startTime
// v4.5.0: Report completion
// Report completion
if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(
`PDF complete: ${numPages} pages, ${allData.length} items extracted`

4
src/augmentations/intelligentImport/index.ts
View file

@ -18,14 +18,14 @@ export { ExcelHandler } from './handlers/excelHandler.js'
export { PDFHandler } from './handlers/pdfHandler.js'
export { ImageHandler } from './handlers/imageHandler.js'
// Format Handler Registry (v5.2.0)
// Format Handler Registry
export {
FormatHandlerRegistry,
globalHandlerRegistry
} from './FormatHandlerRegistry.js'
export type { HandlerRegistration } from './FormatHandlerRegistry.js'
// Image Handler Types (v5.2.0)
// Image Handler Types
export type {
ImageMetadata,
EXIFData,

8
src/augmentations/intelligentImport/types.ts
View file

@ -88,13 +88,13 @@ export interface FormatHandlerOptions {
streaming?: boolean
/**
* Progress hooks (v4.5.0)
* Progress hooks
* Handlers call these to report progress during processing
*/
progressHooks?: FormatHandlerProgressHooks
/**
* Total file size in bytes (v4.5.0)
* Total file size in bytes
* Used for progress percentage calculation
*/
totalBytes?: number
@ -168,7 +168,7 @@ export interface IntelligentImportConfig {
/** Enable PDF handler */
enablePDF: boolean
/** Enable Image handler (v5.2.0) */
/** Enable Image handler */
enableImage: boolean
/** Default options for CSV */
@ -180,7 +180,7 @@ export interface IntelligentImportConfig {
/** Default options for PDF */
pdfDefaults?: Partial<FormatHandlerOptions>
/** Default options for Image (v5.2.0) */
/** Default options for Image */
imageDefaults?: Partial<FormatHandlerOptions>
/** Maximum file size to process (bytes) */

2
src/augmentations/typeMatching/brainyTypes.ts
View file

@ -168,7 +168,7 @@ export interface TypeMatchResult {
/**
* BrainyTypes - Intelligent type detection for nouns and verbs
* PRODUCTION OPTIMIZATION (v3.33.0): Uses pre-computed type embeddings
* PRODUCTION OPTIMIZATION: Uses pre-computed type embeddings
* Type embeddings are loaded instantly; only input objects are embedded at runtime
*/
export class BrainyTypes {

4
src/cli/commands/core.ts
View file

@ -172,7 +172,7 @@ export const coreCommands = {
metadata
}
// v4.3.x: Add confidence and weight if provided
// Add confidence and weight if provided
if (options.confidence) {
addParams.confidence = parseFloat(options.confidence)
}
@ -347,7 +347,7 @@ export const coreCommands = {
searchParams.includeRelations = true
}
// v4.7.0: VFS is now part of the knowledge graph (included by default)
// VFS is now part of the knowledge graph (included by default)
// Users can exclude VFS with --where vfsType exists:false if needed
// Triple Intelligence Fusion - custom weighting

4
src/cli/commands/cow.ts
View file

@ -312,7 +312,7 @@ ${chalk.cyan('Fork Statistics:')}
},
/**
* Migrate from v4.x to v5.0.0 (one-time)
* Migrate storage format (one-time)
*/
async migrate(options: MigrateOptions) {
let spinner: any = null
@ -388,7 +388,7 @@ ${chalk.cyan('Fork Statistics:')}
await oldBrain.init()
// Create new brain (v5.0.0)
// Create new brain
const newBrain = new Brainy({
storage: {
type: 'filesystem',

4
src/cli/commands/neural.ts
View file

@ -117,7 +117,7 @@ export const neuralCommand = {
await handleNeighborsCommand(neural, argv)
break
case 'path':
console.log(chalk.yellow('\n⚠️ Semantic path finding coming in v3.21.0'))
console.log(chalk.yellow('\n⚠️ Semantic path finding coming soon'))
console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
console.log(chalk.dim('Use "neighbors" and "hierarchy" commands to explore connections'))
break
@ -148,7 +148,7 @@ async function promptForAction(): Promise<string> {
{ name: '🎯 Find semantic clusters', value: 'clusters' },
{ name: '🌳 Show item hierarchy', value: 'hierarchy' },
{ name: '🕸️ Find semantic neighbors', value: 'neighbors' },
{ name: '🛣️ Find semantic path between items (v3.21.0)', value: 'path', disabled: true },
{ name: '🛣️ Find semantic path between items (coming soon)', value: 'path', disabled: true },
{ name: '🚨 Detect outliers', value: 'outliers' },
{ name: '📊 Generate visualization data', value: 'visualize' }
]

2
src/cli/commands/storage.ts
View file

@ -1,5 +1,5 @@
/**
* 💾 Storage Management Commands - v4.0.0
* 💾 Storage Management Commands
*
* Modern interactive CLI for storage lifecycle, cost optimization, and management
*/

2
src/cli/commands/utility.ts
View file

@ -133,7 +133,7 @@ export const utilityCommands = {
// removeOrphans and rebuildIndex would require new Brainy APIs
if (options.removeOrphans || options.rebuildIndex) {
spinner.warn('Advanced cleanup options not yet implemented')
console.log(chalk.yellow('\n⚠️ Advanced cleanup features coming in v3.21.0:'))
console.log(chalk.yellow('\n⚠️ Advanced cleanup features coming soon:'))
console.log(chalk.dim(' • --remove-orphans: Remove disconnected items'))
console.log(chalk.dim(' • --rebuild-index: Rebuild vector index'))
console.log(chalk.dim('\nUse "brainy clean" without options to clear the database'))

6
src/cli/commands/vfs.ts
View file

@ -21,7 +21,7 @@ let brainyInstance: Brainy | null = null
const getBrainy = async (): Promise<Brainy> => {
if (!brainyInstance) {
brainyInstance = new Brainy()
await brainyInstance.init() // v5.0.1: Initialize brain (VFS auto-initialized here!)
await brainyInstance.init() // Initialize brain (VFS auto-initialized here!)
}
return brainyInstance
}
@ -52,8 +52,8 @@ export const vfsCommands = {
const spinner = ora('Reading file...').start()
try {
const brain = await getBrainy() // v5.0.1: Await async getBrainy
// v5.0.1: VFS auto-initialized, no need for vfs.init()
const brain = await getBrainy() // Await async getBrainy
// VFS auto-initialized, no need for vfs.init()
const buffer = await brain.vfs.readFile(path, {
encoding: options.encoding as any
})

10
src/cli/index.ts
View file

@ -68,7 +68,7 @@ ${chalk.cyan('Examples:')}
$ brainy vfs search "React components"
$ brainy vfs similar /code/Button.tsx
${chalk.dim('# Storage management (v4.0.0)')}
${chalk.dim('# Storage management')}
$ brainy storage status --quota
$ brainy storage lifecycle set ${chalk.dim('# Interactive mode')}
$ brainy storage cost-estimate
@ -217,11 +217,11 @@ program
program
.command('path <from> <to>')
.description('Find semantic path between items (v3.21.0)')
.description('Find semantic path between items')
.option('--steps', 'Show step-by-step path')
.option('--max-hops <number>', 'Maximum path length', '5')
.action(() => {
console.log(chalk.yellow('\n⚠️ Semantic path finding coming in v3.21.0'))
console.log(chalk.yellow('\n⚠️ Semantic path finding coming soon'))
console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
console.log(chalk.dim('Use "brainy neighbors" and "brainy hierarchy" to explore connections'))
})
@ -446,7 +446,7 @@ program
vfsCommands.tree(path, options)
})
// ===== Storage Management Commands (v4.0.0) =====
// ===== Storage Management Commands =====
program
.command('storage')
@ -610,7 +610,7 @@ program
.option('--iterations <n>', 'Number of iterations', '100')
.action(utilityCommands.benchmark)
// ===== COW Commands (v5.0.0) - Instant Fork & Branching =====
// ===== COW Commands - Instant Fork & Branching =====
program
.command('fork [name]')

2
src/cli/interactive.ts
View file

@ -625,7 +625,7 @@ export async function promptCommand(): Promise<string> {
*/
export async function startInteractiveMode() {
console.log(chalk.cyan('\n🧠 Brainy Interactive Mode\n'))
console.log(chalk.yellow('Interactive REPL mode coming in v3.20.0\n'))
console.log(chalk.yellow('Interactive REPL mode coming soon\n'))
console.log(chalk.dim('Use specific commands for now: brainy add, brainy search, etc.'))
process.exit(0)
}

68
src/coreTypes.ts
View file

@ -57,7 +57,7 @@ export type DistanceFunction = (a: Vector, b: Vector) => number
/**
* Embedding function for converting data to vectors
* v4.0.0: Now properly typed - accepts string, string array (batch), or object, no `any`
* Now properly typed - accepts string, string array (batch), or object, no `any`
*/
export type EmbeddingFunction = (data: string | string[] | Record<string, unknown>) => Promise<Vector>
@ -72,7 +72,7 @@ export interface EmbeddingModel {
/**
* Embed data into a vector
* v4.0.0: Now properly typed - accepts string, string array (batch), or object, no `any`
* Now properly typed - accepts string, string array (batch), or object, no `any`
*/
embed(data: string | string[] | Record<string, unknown>): Promise<Vector>
@ -83,9 +83,9 @@ export interface EmbeddingModel {
}
/**
* HNSW graph noun - Pure vector structure (v4.0.0)
* HNSW graph noun - Pure vector structure
*
* v4.0.0 BREAKING CHANGE: metadata field removed
* metadata field removed
* - Stores ONLY vector data for optimal memory usage
* - Metadata stored separately and combined on retrieval
* - 25% memory reduction @ 1B scale (no in-memory metadata)
@ -100,15 +100,15 @@ export interface HNSWNoun {
}
/**
* Lightweight verb for HNSW index storage - Core relational structure (v4.0.0)
* Lightweight verb for HNSW index storage - Core relational structure
*
* Core fields (v3.50.1+): verb/sourceId/targetId are first-class fields
* Core fields: verb/sourceId/targetId are first-class fields
* These are NOT metadata - they're the essence of what a verb IS:
* - verb: The relationship type (creates, contains, etc.) - needed for routing & display
* - sourceId: What entity this verb connects FROM - needed for graph traversal
* - targetId: What entity this verb connects TO - needed for graph traversal
*
* v4.0.0 BREAKING CHANGE: metadata field removed
* metadata field removed
* - Stores ONLY vector + core relational data
* - User metadata (weight, custom fields) stored separately
* - 10x faster metadata-only updates (skip HNSW rebuild)
@ -134,9 +134,9 @@ export interface HNSWVerb {
}
/**
* Noun metadata structure (v4.8.0)
* Noun metadata structure
*
* v4.8.0 BREAKING CHANGE: Now contains ONLY custom user-defined fields
* Now contains ONLY custom user-defined fields
* - Standard fields (confidence, weight, timestamps, etc.) moved to top-level in HNSWNounWithMetadata
* - This interface represents custom metadata stored separately from vector data
* - Storage format unchanged (backward compatible at storage layer)
@ -162,9 +162,9 @@ export interface NounMetadata {
}
/**
* Verb metadata structure (v4.8.0)
* Verb metadata structure
*
* v4.8.0 BREAKING CHANGE: Now contains ONLY custom user-defined fields
* Now contains ONLY custom user-defined fields
* - Standard fields (weight, confidence, timestamps, etc.) moved to top-level in HNSWVerbWithMetadata
* - This interface represents custom metadata stored separately from vector + core relational data
* - Storage format unchanged (backward compatible at storage layer)
@ -190,9 +190,9 @@ export interface VerbMetadata {
}
/**
* Combined noun structure for transport/API boundaries (v4.8.0)
* Combined noun structure for transport/API boundaries
*
* v4.8.0 BREAKING CHANGE: Standard fields moved to top-level
* Standard fields moved to top-level
* - ALL standard fields (confidence, weight, timestamps, etc.) are now at top-level
* - metadata contains ONLY custom user-defined fields
* - Provides clean, predictable API: entity.confidence always works
@ -230,9 +230,9 @@ export interface HNSWNounWithMetadata {
}
/**
* Combined verb structure for transport/API boundaries (v4.8.0)
* Combined verb structure for transport/API boundaries
*
* v4.8.0 BREAKING CHANGE: Standard fields moved to top-level
* Standard fields moved to top-level
* - ALL standard fields (weight, confidence, timestamps, etc.) are now at top-level
* - metadata contains ONLY custom user-defined fields
* - Provides clean, predictable API: verb.weight always works
@ -308,7 +308,7 @@ export interface HNSWConfig {
efSearch: number // Size of the dynamic candidate list during search
ml: number // Maximum level
useDiskBasedIndex?: boolean // Whether to use disk-based index
maxConcurrentNeighborWrites?: number // Maximum concurrent neighbor updates during insert (v4.10.0+). Default: unlimited (full concurrency)
maxConcurrentNeighborWrites?: number // Maximum concurrent neighbor updates during insert. Default: unlimited (full concurrency)
}
/**
@ -548,7 +548,7 @@ export interface StatisticsData {
}
/**
* Change record for getChangesSince (v4.0.0)
* Change record for getChangesSince
* Replaces `any[]` with properly typed structure
*/
export interface Change {
@ -563,27 +563,27 @@ export interface StorageAdapter {
init(): Promise<void>
/**
* Save noun - Pure HNSW vector data only (v4.0.0)
* Save noun - Pure HNSW vector data only
* @param noun Pure HNSW vector data (no metadata)
* Note: Use saveNounMetadata() to save metadata separately
*/
saveNoun(noun: HNSWNoun): Promise<void>
/**
* Save noun metadata separately (v4.0.0)
* Save noun metadata separately
* @param id Noun ID
* @param metadata Noun metadata
*/
saveNounMetadata(id: string, metadata: NounMetadata): Promise<void>
/**
* Delete noun metadata (v4.0.0)
* Delete noun metadata
* @param id Noun ID
*/
deleteNounMetadata(id: string): Promise<void>
/**
* Get noun with metadata combined (v4.0.0)
* Get noun with metadata combined
* @returns Combined HNSWNounWithMetadata or null
*/
getNoun(id: string): Promise<HNSWNounWithMetadata | null>
@ -622,14 +622,14 @@ export interface StorageAdapter {
deleteNoun(id: string): Promise<void>
/**
* Save verb - Pure HNSW verb with core fields only (v4.0.0)
* Save verb - Pure HNSW verb with core fields only
* @param verb Pure HNSW verb data (vector + core fields, no user metadata)
* Note: Use saveVerbMetadata() to save metadata separately
*/
saveVerb(verb: HNSWVerb): Promise<void>
/**
* Get verb with metadata combined (v4.0.0)
* Get verb with metadata combined
* @returns Combined HNSWVerbWithMetadata or null
*/
getVerb(id: string): Promise<HNSWVerbWithMetadata | null>
@ -686,14 +686,14 @@ export interface StorageAdapter {
deleteVerb(id: string): Promise<void>
/**
* Save metadata (v4.0.0: now typed)
* Save metadata
* @param id Entity ID
* @param metadata Typed noun metadata
*/
saveMetadata(id: string, metadata: NounMetadata): Promise<void>
/**
* Get metadata (v4.0.0: now typed)
* Get metadata
* @param id Entity ID
* @returns Typed noun metadata or null
*/
@ -702,26 +702,26 @@ export interface StorageAdapter {
/**
* Get multiple metadata objects in batches (prevents socket exhaustion)
* @param ids Array of IDs to get metadata for
* @returns Promise that resolves to a Map of id -> metadata (v4.0.0: typed)
* @returns Promise that resolves to a Map of id -> metadata
*/
getMetadataBatch?(ids: string[]): Promise<Map<string, NounMetadata>>
/**
* Get noun metadata from storage (v4.0.0: now typed)
* Get noun metadata from storage
* @param id The ID of the noun
* @returns Promise that resolves to the metadata or null if not found
*/
getNounMetadata(id: string): Promise<NounMetadata | null>
/**
* Batch get multiple nouns with vectors (v6.2.0 - N+1 fix)
* Batch get multiple nouns with vectors
* @param ids Array of noun IDs to fetch
* @returns Map of id → HNSWNounWithMetadata (only successful reads included)
*/
getNounBatch?(ids: string[]): Promise<Map<string, HNSWNounWithMetadata>>
/**
* Save verb metadata to storage (v4.0.0: now typed)
* Save verb metadata to storage
* @param id The ID of the verb
* @param metadata The metadata to save
* @returns Promise that resolves when the metadata is saved
@ -729,14 +729,14 @@ export interface StorageAdapter {
saveVerbMetadata(id: string, metadata: VerbMetadata): Promise<void>
/**
* Get verb metadata from storage (v4.0.0: now typed)
* Get verb metadata from storage
* @param id The ID of the verb
* @returns Promise that resolves to the metadata or null if not found
*/
getVerbMetadata(id: string): Promise<VerbMetadata | null>
/**
* Batch get multiple verbs (v6.2.0 - N+1 fix)
* Batch get multiple verbs
* @param ids Array of verb IDs to fetch
* @returns Map of id → HNSWVerbWithMetadata (only successful reads included)
*/
@ -745,7 +745,7 @@ export interface StorageAdapter {
clear(): Promise<void>
/**
* Batch delete multiple objects from storage (v4.0.0)
* Batch delete multiple objects from storage
* Efficient deletion of large numbers of entities using cloud provider batch APIs.
* Significantly faster and cheaper than individual deletes (up to 1000x speedup).
*
@ -853,7 +853,7 @@ export interface StorageAdapter {
flushStatisticsToStorage(): Promise<void>
/**
* Track field names from a JSON document (v4.0.0: now typed)
* Track field names from a JSON document
* @param jsonDocument The JSON document to extract field names from
* @param service The service that inserted the data
*/
@ -872,7 +872,7 @@ export interface StorageAdapter {
getStandardFieldMappings(): Promise<Record<string, Record<string, string[]>>>
/**
* Get changes since a specific timestamp (v4.0.0: now typed)
* Get changes since a specific timestamp
* @param timestamp The timestamp to get changes since
* @param limit Optional limit on the number of changes to return
* @returns Promise that resolves to an array of properly typed changes

4
src/embeddings/candle-wasm/src/lib.rs
View file

@ -26,7 +26,7 @@ use js_sys::{Array, Float32Array};
use tokenizers::Tokenizer;
use wasm_bindgen::prelude::*;
// v7.2.0: Model weights are NO LONGER embedded in WASM
// Model weights are NO LONGER embedded in WASM
//
// Previous design: 90MB WASM with model weights embedded via include_bytes!()
// Problem: WASM parsing/compilation took 139+ seconds on throttled CPU (Cloud Run)
@ -78,7 +78,7 @@ impl EmbeddingEngine {
/// Load the model and tokenizer from bytes
///
/// v7.2.0: This is now the ONLY way to initialize the engine.
/// This is now the ONLY way to initialize the engine.
/// Model weights are no longer embedded in WASM for faster initialization.
///
/// # Arguments

10
src/embeddings/wasm/CandleEmbeddingEngine.ts
View file

@ -5,7 +5,7 @@
* Pure Rust/WASM implementation for sentence embeddings.
* Works with Bun, Node.js, Bun --compile, and browsers.
*
* v7.2.0 Architecture (20x faster initialization):
* Architecture (20x faster initialization):
* - WASM file: ~2.4MB (inference code only)
* - Model files: ~88MB (loaded separately as raw bytes)
* - Init time: ~5-7 seconds (vs 139 seconds with embedded model)
@ -34,7 +34,7 @@ interface CandleWasmModule {
}
interface CandleEngineInstance {
// v7.2.0: load() is the only initialization method (no more embedded model)
// load() is the only initialization method (no more embedded model)
load(modelBytes: Uint8Array, tokenizerBytes: Uint8Array, configBytes: Uint8Array): void
is_ready(): boolean
embed(text: string): Float32Array
@ -54,7 +54,7 @@ let globalInitPromise: Promise<void> | null = null
* Uses the Candle ML framework (Rust/WASM) for inference.
* Supports all-MiniLM-L6-v2 with 384-dimensional embeddings.
*
* v7.2.0: Model weights are loaded separately from WASM for 20x faster init.
* Model weights are loaded separately from WASM for 20x faster init.
* For bun --compile deployments, both WASM and model files are automatically
* embedded in the binary - single file deployment still works.
*/
@ -104,7 +104,7 @@ export class CandleEmbeddingEngine {
/**
* Perform actual initialization
*
* v7.2.0: WASM and model files are loaded separately for 20x faster init.
* WASM and model files are loaded separately for 20x faster init.
* - WASM (~2.4MB): Compiles in ~3-5 seconds
* - Model (~88MB): Loads as raw bytes in ~1-2 seconds
* - Total: ~5-7 seconds (vs 139 seconds with embedded model)
@ -153,7 +153,7 @@ export class CandleEmbeddingEngine {
/**
* Load the WASM module
*
* v7.2.0: WASM is now only ~2.4MB (inference code only, no model weights).
* WASM is now only ~2.4MB (inference code only, no model weights).
* Uses wasmLoader.ts for cross-environment compatibility.
*/
private async loadWasmModule(): Promise<CandleWasmModule> {

2
src/embeddings/wasm/modelLoader.ts
View file

@ -1,7 +1,7 @@
/**
* Universal Model Loader for Candle Embeddings
*
* v7.2.0: Model weights are now loaded separately from WASM for faster initialization.
* Model weights are now loaded separately from WASM for faster initialization.
* This reduces WASM compilation time from 139 seconds to ~3-5 seconds.
*
* Loads model files from appropriate source based on environment:

37
src/graph/graphAdjacencyIndex.ts
View file

@ -45,7 +45,7 @@ export class GraphAdjacencyIndex {
private lsmTreeVerbsBySource: LSMTree // sourceId -> verbIds
private lsmTreeVerbsByTarget: LSMTree // targetId -> verbIds
// v5.7.0: ID-only tracking for billion-scale memory optimization
// ID-only tracking for billion-scale memory optimization
// Previous: Map<string, GraphVerb> stored full objects (128GB @ 1B verbs)
// Now: Set<string> stores only IDs (~100KB @ 1B verbs) = 1,280,000x reduction
private verbIdSet = new Set<string>()
@ -117,7 +117,7 @@ export class GraphAdjacencyIndex {
/**
* Initialize the graph index (lazy initialization)
* v6.3.0: Added defensive auto-rebuild check for verbIdSet consistency
* Added defensive auto-rebuild check for verbIdSet consistency
*/
private async ensureInitialized(): Promise<void> {
if (this.initialized) {
@ -129,7 +129,7 @@ export class GraphAdjacencyIndex {
await this.lsmTreeVerbsBySource.init()
await this.lsmTreeVerbsByTarget.init()
// v6.3.0: Defensive check - if LSM-trees have data but verbIdSet is empty,
// Defensive check - if LSM-trees have data but verbIdSet is empty,
// the index was created without proper rebuild (shouldn't happen with singleton
// pattern but protects against edge cases and future refactoring)
const lsmTreeSize = this.lsmTreeVerbsBySource.size()
@ -151,7 +151,7 @@ export class GraphAdjacencyIndex {
}
/**
* Populate verbIdSet from storage without full rebuild (v6.3.0)
* Populate verbIdSet from storage without full rebuild
* Lighter weight than full rebuild - only loads verb IDs, not all verb data
* @private
*/
@ -192,7 +192,7 @@ export class GraphAdjacencyIndex {
* Core API - Neighbor lookup with LSM-tree storage
*
* O(log n) with bloom filter optimization (90% of queries skip disk I/O)
* v5.8.0: Added pagination support for high-degree nodes
* Added pagination support for high-degree nodes
*
* @param id Entity ID to get neighbors for
* @param optionsOrDirection Optional: direction string OR options object
@ -252,7 +252,7 @@ export class GraphAdjacencyIndex {
// Convert to array for pagination
let result = Array.from(neighbors)
// Apply pagination if requested (v5.8.0)
// Apply pagination if requested
if (options?.limit !== undefined || options?.offset !== undefined) {
const offset = options.offset || 0
const limit = options.limit !== undefined ? options.limit : result.length
@ -273,8 +273,8 @@ export class GraphAdjacencyIndex {
* Get verb IDs by source - Billion-scale optimization for getVerbsBySource
*
* O(log n) LSM-tree lookup with bloom filter optimization
* v5.7.1: Filters out deleted verb IDs (tombstone deletion workaround)
* v5.8.0: Added pagination support for entities with many relationships
* Filters out deleted verb IDs (tombstone deletion workaround)
* Added pagination support for entities with many relationships
*
* @param sourceId Source entity ID
* @param options Optional configuration
@ -318,7 +318,7 @@ export class GraphAdjacencyIndex {
const allIds = verbIds || []
let result = allIds.filter(id => this.verbIdSet.has(id))
// Apply pagination if requested (v5.8.0)
// Apply pagination if requested
if (options?.limit !== undefined || options?.offset !== undefined) {
const offset = options.offset || 0
const limit = options.limit !== undefined ? options.limit : result.length
@ -332,8 +332,8 @@ export class GraphAdjacencyIndex {
* Get verb IDs by target - Billion-scale optimization for getVerbsByTarget
*
* O(log n) LSM-tree lookup with bloom filter optimization
* v5.7.1: Filters out deleted verb IDs (tombstone deletion workaround)
* v5.8.0: Added pagination support for popular target entities
* Filters out deleted verb IDs (tombstone deletion workaround)
* Added pagination support for popular target entities
*
* @param targetId Target entity ID
* @param options Optional configuration
@ -377,7 +377,7 @@ export class GraphAdjacencyIndex {
const allIds = verbIds || []
let result = allIds.filter(id => this.verbIdSet.has(id))
// Apply pagination if requested (v5.8.0)
// Apply pagination if requested
if (options?.limit !== undefined || options?.offset !== undefined) {
const offset = options.offset || 0
const limit = options.limit !== undefined ? options.limit : result.length
@ -414,7 +414,7 @@ export class GraphAdjacencyIndex {
}
/**
* Batch get multiple verbs with caching (v6.2.0 - N+1 fix)
* Batch get multiple verbs with caching
*
* **Performance**: Eliminates N+1 pattern for verb loading
* - Current: N × getVerbCached() = N × 50ms on GCS = 250ms for 5 verbs
@ -433,7 +433,6 @@ export class GraphAdjacencyIndex {
* @param verbIds Array of verb IDs to fetch
* @returns Map of verbId → GraphVerb (only successful reads included)
*
* @since v6.2.0
*/
async getVerbsBatchCached(verbIds: string[]): Promise<Map<string, GraphVerb>> {
const results = new Map<string, GraphVerb>()
@ -514,7 +513,7 @@ export class GraphAdjacencyIndex {
const targetStats = this.lsmTreeTarget.getStats()
// Note: Exact unique node counts would require full LSM-tree scan
// v5.7.0: Using verbIdSet (ID-only tracking) for memory efficiency
// Using verbIdSet (ID-only tracking) for memory efficiency
const uniqueSourceNodes = this.verbIdSet.size
const uniqueTargetNodes = this.verbIdSet.size
const totalNodes = this.verbIdSet.size
@ -622,13 +621,13 @@ export class GraphAdjacencyIndex {
// Clear current index
this.verbIdSet.clear()
this.totalRelationshipsIndexed = 0
// v6.2.4: CRITICAL FIX - Clear relationship counts to prevent accumulation
// CRITICAL FIX - Clear relationship counts to prevent accumulation
this.relationshipCountsByType.clear()
// Note: LSM-trees will be recreated from storage via their own initialization
// Verb data will be loaded on-demand via UnifiedCache
// Adaptive loading strategy based on storage type (v4.2.4)
// Adaptive loading strategy based on storage type
const storageType = this.storage?.constructor.name || ''
const isLocalStorage =
storageType === 'FileSystemStorage' ||
@ -754,7 +753,7 @@ export class GraphAdjacencyIndex {
bytes += targetStats.memTableMemory
// Verb ID set (memory-efficient: IDs only, ~8 bytes per ID pointer)
// v5.7.0: Previous verbIndex Map stored full objects (128 bytes each = 128GB @ 1B verbs)
// Previous verbIndex Map stored full objects (128 bytes each = 128GB @ 1B verbs)
// Now: verbIdSet stores only IDs (~8 bytes each = ~100KB @ 1B verbs) = 1,280,000x reduction
bytes += this.verbIdSet.size * 8
@ -792,7 +791,7 @@ export class GraphAdjacencyIndex {
/**
* Flush LSM-tree MemTables to disk
* CRITICAL FIX (v3.43.2): Now public so it can be called from brain.flush()
* CRITICAL FIX: Now public so it can be called from brain.flush()
*/
async flush(): Promise<void> {
if (!this.initialized) {

61
src/hnsw/hnswIndex.ts
View file

@ -35,18 +35,18 @@ export class HNSWIndex {
private distanceFunction: DistanceFunction
private dimension: number | null = null
private useParallelization: boolean = true // Whether to use parallelization for performance-critical operations
private storage: BaseStorage | null = null // Storage adapter for HNSW persistence (v3.35.0+)
private storage: BaseStorage | null = null // Storage adapter for HNSW persistence
// Universal memory management (v3.36.0+)
// Universal memory management
private unifiedCache: UnifiedCache // Shared cache with Graph and Metadata indexes
// Always-adaptive caching (v3.36.0+) - no "mode" concept, system adapts automatically
// Always-adaptive caching - no "mode" concept, system adapts automatically
// COW (Copy-on-Write) support - v5.0.0
// COW (Copy-on-Write) support
private cowEnabled: boolean = false
private cowModifiedNodes: Set<string> = new Set()
private cowParent: HNSWIndex | null = null
// v6.2.8: Deferred HNSW persistence for cloud storage performance
// Deferred HNSW persistence for cloud storage performance
// In deferred mode, HNSW connections are only persisted on flush/close
// This reduces GCS operations from 70 to 2-3 per add() (30-50× faster)
private persistMode: 'immediate' | 'deferred' = 'immediate'
@ -86,7 +86,7 @@ export class HNSWIndex {
}
/**
* v6.2.8: Flush dirty HNSW data to storage
* Flush dirty HNSW data to storage
*
* In deferred persistence mode, HNSW connections are tracked as dirty but not
* immediately persisted. Call flush() to persist all pending changes.
@ -361,6 +361,19 @@ export class HNSWIndex {
this.entryPointId = id
this.maxLevel = nounLevel
this.nouns.set(id, noun)
// Persist system data for first noun (previously skipped)
if (this.storage && this.persistMode === 'immediate') {
await this.storage.saveHNSWSystem({
entryPointId: this.entryPointId,
maxLevel: this.maxLevel
}).catch(error => {
console.error('Failed to persist initial HNSW system data:', error)
})
} else if (this.persistMode === 'deferred') {
this.dirtySystem = true
}
return id
}
@ -438,7 +451,7 @@ export class HNSWIndex {
)
// Add bidirectional connections
// PERFORMANCE OPTIMIZATION (v4.10.0): Collect all neighbor updates for concurrent execution
// PERFORMANCE OPTIMIZATION: Collect all neighbor updates for concurrent execution
const neighborUpdates: Array<{
neighborId: string
promise: Promise<void>
@ -467,9 +480,9 @@ export class HNSWIndex {
await this.pruneConnections(neighbor, level)
}
// Persist updated neighbor HNSW data (v3.35.0+)
// Persist updated neighbor HNSW data
//
// v6.2.8: Deferred persistence mode for cloud storage performance
// Deferred persistence mode for cloud storage performance
// In deferred mode, we track dirty nodes instead of persisting immediately
// This reduces GCS operations from 70 to 2-3 per add() (30-50× faster)
if (this.storage && this.persistMode === 'immediate') {
@ -561,8 +574,8 @@ export class HNSWIndex {
this.highLevelNodes.get(nounLevel)!.add(id)
}
// Persist HNSW graph data to storage (v3.35.0+)
// v6.2.8: Respect persistMode setting
// Persist HNSW graph data to storage
// Respect persistMode setting
if (this.storage && this.persistMode === 'immediate') {
// IMMEDIATE MODE: Original behavior - persist new entity and system data
const connectionsObj: Record<string, string[]> = {}
@ -594,7 +607,7 @@ export class HNSWIndex {
}
/**
* O(1) entry point recovery using highLevelNodes index (v6.2.3).
* O(1) entry point recovery using highLevelNodes index.
* At any reasonable scale (1000+ nodes), level 2+ nodes are guaranteed to exist.
* For tiny indexes with only level 0-1 nodes, any node works as entry point.
*/
@ -640,7 +653,7 @@ export class HNSWIndex {
}
// Start from the entry point
// If entry point is null but nouns exist, attempt O(1) recovery (v6.2.3)
// If entry point is null but nouns exist, attempt O(1) recovery
if (!this.entryPointId && this.nouns.size > 0) {
const { id: recoveredId, level: recoveredLevel } = this.recoverEntryPointO1()
if (recoveredId) {
@ -656,7 +669,7 @@ export class HNSWIndex {
let entryPoint = this.nouns.get(this.entryPointId)
if (!entryPoint) {
// Entry point ID exists but noun was deleted - O(1) recovery (v6.2.3)
// Entry point ID exists but noun was deleted - O(1) recovery
if (this.nouns.size > 0) {
const { id: recoveredId, level: recoveredLevel } = this.recoverEntryPointO1()
if (recoveredId) {
@ -944,7 +957,7 @@ export class HNSWIndex {
/**
* Get vector safely (always uses adaptive caching via UnifiedCache)
*
* Production-grade adaptive caching (v3.36.0+):
* Production-grade adaptive caching:
* - Vector already loaded: Returns immediately (O(1))
* - Vector in cache: Loads from UnifiedCache (O(1) hash lookup)
* - Vector on disk: Loads from storage → UnifiedCache (O(disk))
@ -990,7 +1003,7 @@ export class HNSWIndex {
}
/**
* Get vector synchronously if available in memory (v3.36.0+)
* Get vector synchronously if available in memory
*
* Sync fast path optimization:
* - Vector in memory: Returns immediately (zero overhead)
@ -1048,7 +1061,7 @@ export class HNSWIndex {
}
/**
* Calculate distance with sync fast path (v3.36.0+)
* Calculate distance with sync fast path
*
* Eliminates async overhead when vectors are in memory:
* - Sync path: Vector in memory → returns number (zero overhead)
@ -1093,7 +1106,7 @@ export class HNSWIndex {
}
/**
* Rebuild HNSW index from persisted graph data (v3.35.0+)
* Rebuild HNSW index from persisted graph data
*
* This is a production-grade O(N) rebuild that restores the pre-computed graph structure
* from storage. Much faster than re-building which is O(N log N).
@ -1157,7 +1170,7 @@ export class HNSWIndex {
)
}
// Step 4: Adaptive loading strategy based on storage type (v4.2.4)
// Step 4: Adaptive loading strategy based on storage type
// FileSystem/Memory/OPFS: Load all at once (avoids repeated getAllShardedFiles() calls)
// Cloud (GCS/S3/R2): Use pagination (efficient native cloud APIs)
const storageType = this.storage?.constructor.name || ''
@ -1238,7 +1251,7 @@ export class HNSWIndex {
prodLog.info(`HNSW: Using cloud pagination strategy (${storageType})`)
let hasMore = true
let offset = 0 // v5.7.11: Use offset-based pagination instead of cursor (bug fix for infinite loop)
let offset = 0 // Use offset-based pagination instead of cursor (bug fix for infinite loop)
while (hasMore) {
// Fetch batch of nouns from storage (cast needed as method is not in base interface)
@ -1249,7 +1262,7 @@ export class HNSWIndex {
nextCursor?: string
} = await (this.storage as any).getNounsWithPagination({
limit: batchSize,
offset // v5.7.11: Pass offset for proper pagination (previously passed cursor which was ignored)
offset // Pass offset for proper pagination (previously passed cursor which was ignored)
})
// Set total count on first batch
@ -1307,11 +1320,11 @@ export class HNSWIndex {
// Check for more data
hasMore = result.hasMore
offset += batchSize // v5.7.11: Increment offset for next page
offset += batchSize // Increment offset for next page
}
}
// Step 5: CRITICAL - Recover entry point if missing (v6.2.3 - O(1))
// Step 5: CRITICAL - Recover entry point if missing)
// This ensures consistency even if getHNSWSystem() returned null
if (this.nouns.size > 0 && this.entryPointId === null) {
prodLog.warn('HNSW rebuild: Entry point was null after loading nouns - recovering with O(1) lookup')
@ -1426,7 +1439,7 @@ export class HNSWIndex {
}
/**
* Get cache performance statistics for monitoring and diagnostics (v3.36.0+)
* Get cache performance statistics for monitoring and diagnostics
*
* Production-grade monitoring:
* - Adaptive caching strategy (preloading vs on-demand)

26
src/hnsw/typeAwareHNSWIndex.ts
View file

@ -118,7 +118,7 @@ export class TypeAwareHNSWIndex {
}
/**
* v6.2.8: Flush dirty HNSW data to storage for all type-specific indexes
* Flush dirty HNSW data to storage for all type-specific indexes
*
* In deferred persistence mode, HNSW connections are tracked as dirty but not
* immediately persisted. Call flush() to persist all pending changes across
@ -502,7 +502,7 @@ export class TypeAwareHNSWIndex {
// Load ALL nouns ONCE and route to correct type indexes
// This is O(N) instead of O(42*N) from the previous parallel approach
let offset = 0 // v5.7.11: Use offset-based pagination instead of cursor (bug fix for infinite loop)
let offset = 0 // Use offset-based pagination instead of cursor (bug fix for infinite loop)
let hasMore = true
let totalLoaded = 0
const loadedByType = new Map<NounType, number>()
@ -515,13 +515,13 @@ export class TypeAwareHNSWIndex {
totalCount?: number
} = await (this.storage as any).getNounsWithPagination({
limit: batchSize,
offset // v5.7.11: Pass offset for proper pagination (previously passed cursor which was ignored)
offset // Pass offset for proper pagination (previously passed cursor which was ignored)
})
// Route each noun to its type index
for (const nounData of result.items) {
try {
// v7.3.0: Use 'type' property from HNSWNounWithMetadata (not 'nounType')
// Use 'type' property from HNSWNounWithMetadata (not 'nounType')
// Previously accessed wrong property, causing N+1 metadata fetches
// getNounsWithPagination already includes type in response
const nounType = nounData.type
@ -578,7 +578,7 @@ export class TypeAwareHNSWIndex {
}
hasMore = result.hasMore
offset += batchSize // v5.7.11: Increment offset for next page
offset += batchSize // Increment offset for next page
// Progress logging
if (totalLoaded % 1000 === 0) {
@ -593,12 +593,26 @@ export class TypeAwareHNSWIndex {
let entryPointId: string | null = null
for (const [id, noun] of (index as any).nouns.entries()) {
if (noun.level > maxLevel) {
if (entryPointId === null || noun.level > maxLevel) {
maxLevel = noun.level
entryPointId = id
}
}
// Recovery: if still null after loop but nouns exist
if ((index as any).nouns.size > 0 && !entryPointId) {
const { id: recoveredId, level: recoveredLevel } = (index as any).recoverEntryPointO1()
entryPointId = recoveredId
maxLevel = recoveredLevel
}
// Validation: if entry point doesn't exist in loaded nouns
if (entryPointId && !(index as any).nouns.has(entryPointId)) {
const { id: recoveredId, level: recoveredLevel } = (index as any).recoverEntryPointO1()
entryPointId = recoveredId
maxLevel = recoveredLevel
}
;(index as any).entryPointId = entryPointId
;(index as any).maxLevel = maxLevel

8
src/import/FormatDetector.ts
View file

@ -111,7 +111,7 @@ export class FormatDetector {
return 'docx'
}
// Images (v5.2.0: ImageHandler support)
// Images (ImageHandler support)
if (mimeType.startsWith('image/')) {
return 'image'
}
@ -134,7 +134,7 @@ export class FormatDetector {
}
}
// YAML detection (v4.2.0)
// YAML detection
if (this.looksLikeYAML(trimmed)) {
return {
format: 'yaml',
@ -206,7 +206,7 @@ export class FormatDetector {
}
}
// Image formats (v5.2.0)
// Image formats
// JPEG: FF D8 FF
if (buffer[0] === 0xFF && buffer[1] === 0xD8 && buffer[2] === 0xFF) {
return {
@ -384,7 +384,7 @@ export class FormatDetector {
/**
* Check if content looks like YAML
* v4.2.0: Added YAML detection
* Added YAML detection
*/
private looksLikeYAML(content: string): boolean {
const lines = content.split('\n').filter(l => l.trim()).slice(0, 20)

110
src/import/ImportCoordinator.ts
View file

@ -37,10 +37,10 @@ export interface ImportSource {
/** Optional filename hint */
filename?: string
/** HTTP headers for URL imports (v4.2.0) */
/** HTTP headers for URL imports */
headers?: Record<string, string>
/** Basic authentication for URL imports (v4.2.0) */
/** Basic authentication for URL imports */
auth?: {
username: string
password: string
@ -93,7 +93,7 @@ export interface ValidImportOptions {
/** Create relationships in knowledge graph */
createRelationships?: boolean
/** Create provenance relationships (document → entity) [v4.9.0] */
/** Create provenance relationships (document → entity) */
createProvenanceLinks?: boolean
/** Preserve source file in VFS */
@ -144,7 +144,7 @@ export interface ValidImportOptions {
customMetadata?: Record<string, any>
/**
* Progress callback for tracking import progress (v4.2.0+)
* Progress callback for tracking import progress
*
* **Streaming Architecture** (always enabled):
* - Indexes are flushed periodically during import (adaptive intervals)
@ -227,21 +227,21 @@ export type ImportOptions = ValidImportOptions & DeprecatedImportOptions
export interface ImportProgress {
stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'relationships' | 'complete'
/** Phase of import - extraction or relationship building (v3.49.0) */
/** Phase of import - extraction or relationship building */
phase?: 'extraction' | 'relationships'
message: string
processed?: number
/** Alias for processed, used in relationship phase (v3.49.0) */
/** Alias for processed, used in relationship phase */
current?: number
total?: number
entities?: number
relationships?: number
/** Rows per second (v3.38.0) */
/** Rows per second */
throughput?: number
/** Estimated time remaining in ms (v3.38.0) */
/** Estimated time remaining in ms */
eta?: number
/**
* Whether data is queryable at this point (v4.2.0+)
* Whether data is queryable at this point
*
* When true, indexes have been flushed and queries will return up-to-date results.
* When false, data exists in storage but indexes may not be current (queries may be slower/incomplete).
@ -357,7 +357,7 @@ export class ImportCoordinator {
/**
* Import from any source with auto-detection
* v4.2.0: Now supports URL imports with authentication
* Now supports URL imports with authentication
*/
async import(
source: Buffer | string | object | ImportSource,
@ -365,10 +365,10 @@ export class ImportCoordinator {
): Promise<ImportResult> {
const startTime = Date.now()
// Validate options (v4.0.0+: Reject deprecated v3.x options)
// Validate options (Reject deprecated options)
this.validateOptions(options)
// Normalize source (v4.2.0: handles URL fetching)
// Normalize source (handles URL fetching)
const normalizedSource = await this.normalizeSource(source, options.format)
// Report detection stage
@ -387,10 +387,10 @@ export class ImportCoordinator {
}
// Set defaults early (needed for tracking context)
// CRITICAL FIX (v4.3.2): Spread options FIRST, then apply defaults
// CRITICAL FIX: Spread options FIRST, then apply defaults
// Previously: ...options at the end overwrote normalized defaults with undefined
// Now: Defaults properly override undefined values
// v4.4.0: Enable AI features by default for smarter imports
// Enable AI features by default for smarter imports
const opts = {
...options, // Spread first to get all options
vfsPath: options.vfsPath || `/imports/${Date.now()}`,
@ -399,13 +399,13 @@ export class ImportCoordinator {
createRelationships: options.createRelationships !== false,
preserveSource: options.preserveSource !== false,
enableDeduplication: options.enableDeduplication !== false,
enableNeuralExtraction: options.enableNeuralExtraction !== false, // v4.4.0: Default true
enableRelationshipInference: options.enableRelationshipInference !== false, // v4.4.0: Default true
enableNeuralExtraction: options.enableNeuralExtraction !== false, // Default true
enableRelationshipInference: options.enableRelationshipInference !== false, // Default true
enableConceptExtraction: options.enableConceptExtraction !== false, // Already defaults to true
deduplicationThreshold: options.deduplicationThreshold || 0.85
}
// Generate tracking context (v4.10.0+: Unified import/project tracking)
// Generate tracking context (Unified import/project tracking)
const importId = options.importId || uuidv4()
const projectId = options.projectId || this.deriveProjectId(opts.vfsPath)
const trackingContext: TrackingContext = {
@ -441,13 +441,13 @@ export class ImportCoordinator {
groupBy: opts.groupBy,
customGrouping: opts.customGrouping,
preserveSource: opts.preserveSource,
// v5.1.2: Fix sourceBuffer for file paths - type is 'path' not 'buffer' from normalizeSource()
// Fix sourceBuffer for file paths - type is 'path' not 'buffer' from normalizeSource()
sourceBuffer: Buffer.isBuffer(normalizedSource.data) ? normalizedSource.data as Buffer : undefined,
sourceFilename: normalizedSource.filename || `import.${detection.format}`,
createRelationshipFile: true,
createMetadataFile: true,
trackingContext, // v4.10.0: Pass tracking metadata to VFS
// v4.11.1: Pass progress callback for VFS creation updates
trackingContext, // Pass tracking metadata to VFS
// Pass progress callback for VFS creation updates
onProgress: (vfsProgress) => {
options.onProgress?.({
stage: 'storing-vfs',
@ -473,7 +473,7 @@ export class ImportCoordinator {
sourceFilename: normalizedSource.filename || `import.${detection.format}`,
format: detection.format
},
trackingContext // v4.10.0: Pass tracking metadata to graph creation
trackingContext // Pass tracking metadata to graph creation
)
// Report complete
@ -520,7 +520,7 @@ export class ImportCoordinator {
)
}
// CRITICAL FIX (v3.43.2): Auto-flush all indexes before returning
// CRITICAL FIX: Auto-flush all indexes before returning
// Ensures imported data survives server restarts
// Bug #5: Import data was only in memory, lost on restart
options.onProgress?.({
@ -535,7 +535,7 @@ export class ImportCoordinator {
/**
* Normalize source to ImportSource
* v4.2.0: Now async to support URL fetching
* Now async to support URL fetching
*/
private async normalizeSource(
source: Buffer | string | object | ImportSource,
@ -616,7 +616,7 @@ export class ImportCoordinator {
/**
* Fetch content from URL
* v4.2.0: Supports authentication and custom headers
* Supports authentication and custom headers
*/
private async fetchUrl(source: ImportSource): Promise<ImportSource> {
const url = typeof source.data === 'string' ? source.data : String(source.data)
@ -722,7 +722,7 @@ export class ImportCoordinator {
format: SupportedFormat,
options: ImportOptions
): Promise<any> {
// v5.2.0: Check if IntelligentImportAugmentation already extracted data
// Check if IntelligentImportAugmentation already extracted data
if ((options as any)._intelligentImport && (options as any)._extractedData) {
const extractedData = (options as any)._extractedData
// Convert extracted data to ExtractedRow format
@ -760,7 +760,7 @@ export class ImportCoordinator {
enableConceptExtraction: options.enableConceptExtraction !== false,
confidenceThreshold: options.confidenceThreshold || 0.6,
onProgress: (stats: any) => {
// Enhanced progress reporting (v3.38.0) with throughput and ETA
// Enhanced progress reporting with throughput and ETA
const message = stats.throughput
? `Extracting entities from ${format} (${stats.throughput} rows/sec, ETA: ${Math.round(stats.eta / 1000)}s)...`
: `Extracting entities from ${format}...`
@ -825,7 +825,7 @@ export class ImportCoordinator {
return await this.docxImporter.extract(docxBuffer, extractOptions)
case 'image':
// v5.2.0: Images are handled by IntelligentImportAugmentation
// Images are handled by IntelligentImportAugmentation
// If we reach here, augmentation didn't process it - return minimal result
const imageName = source.filename || 'image'
const imageId = `image-${Date.now()}`
@ -867,7 +867,7 @@ export class ImportCoordinator {
/**
* Create entities and relationships in knowledge graph
* v4.9.0: Added sourceInfo parameter for document entity creation
* Added sourceInfo parameter for document entity creation
*/
private async createGraphEntities(
extractionResult: any,
@ -877,7 +877,7 @@ export class ImportCoordinator {
sourceFilename: string
format: string
},
trackingContext?: TrackingContext // v4.10.0: Import/project tracking
trackingContext?: TrackingContext // Import/project tracking
): Promise<{
entities: Array<{ id: string; name: string; type: NounType; vfsPath?: string; metadata?: Record<string, any> }>
relationships: Array<{ id: string; from: string; to: string; type: VerbType }>
@ -891,7 +891,7 @@ export class ImportCoordinator {
let mergedCount = 0
let newCount = 0
// CRITICAL FIX (v4.3.2): Default to true when undefined
// CRITICAL FIX: Default to true when undefined
// Previously: if (!options.createEntities) treated undefined as false
// Now: Only skip when explicitly set to false
if (options.createEntities === false) {
@ -908,7 +908,7 @@ export class ImportCoordinator {
// Extract rows/sections/entities from result (unified across formats)
const rows = extractionResult.rows || extractionResult.sections || extractionResult.entities || []
// Progressive flush interval - adjusts based on current count (v4.2.0+)
// Progressive flush interval - adjusts based on current count
// Starts at 100, increases to 1000 at 1K entities, then 5000 at 10K
// This works for both known totals (files) and unknown totals (streaming APIs)
let currentFlushInterval = 100 // Start with frequent updates for better UX
@ -938,7 +938,7 @@ export class ImportCoordinator {
}
// ============================================
// v4.9.0: Create document entity for import source
// Create document entity for import source
// ============================================
let documentEntityId: string | null = null
let provenanceCount = 0
@ -957,7 +957,7 @@ export class ImportCoordinator {
vfsPath: vfsResult.rootPath,
totalRows: rows.length,
byType: this.countByType(rows),
// v4.10.0: Import tracking metadata
// Import tracking metadata
...(trackingContext && {
importIds: [trackingContext.importId],
projectId: trackingContext.projectId,
@ -973,7 +973,7 @@ export class ImportCoordinator {
}
// ============================================
// v4.11.0: Batch entity creation using addMany()
// Batch entity creation using addMany()
// Replaces entity-by-entity loop for 10-100x performance improvement on cloud storage
// ============================================
@ -1038,7 +1038,7 @@ export class ImportCoordinator {
name: entity.name,
type: entity.type,
vfsPath: vfsFile?.path,
metadata: entity.metadata // v5.2.0: Include metadata in return (for ImageHandler, etc)
metadata: entity.metadata // Include metadata in return (for ImageHandler, etc)
})
newCount++
}
@ -1090,7 +1090,7 @@ export class ImportCoordinator {
const importSource = vfsResult.rootPath
let entityId: string
// v5.7.0: No deduplication during import (12-24x speedup)
// No deduplication during import (12-24x speedup)
// Background deduplication runs 5 minutes after import completes
entityId = await this.brain.add({
data: entity.description || entity.name,
@ -1101,7 +1101,7 @@ export class ImportCoordinator {
confidence: entity.confidence,
vfsPath: vfsFile?.path,
importedFrom: 'import-coordinator',
// v4.10.0: Import tracking metadata
// Import tracking metadata
...(trackingContext && {
importId: trackingContext.importId, // Used for background dedup
importIds: [trackingContext.importId],
@ -1126,11 +1126,11 @@ export class ImportCoordinator {
name: entity.name,
type: entity.type,
vfsPath: vfsFile?.path,
metadata: entity.metadata // v5.2.0: Include metadata in return (for ImageHandler, etc)
metadata: entity.metadata // Include metadata in return (for ImageHandler, etc)
})
// ============================================
// v4.9.0: Create provenance relationship (document → entity)
// Create provenance relationship (document → entity)
// ============================================
if (documentEntityId && options.createProvenanceLinks !== false) {
await this.brain.relate({
@ -1144,7 +1144,7 @@ export class ImportCoordinator {
rowNumber: row.rowNumber,
extractedAt: Date.now(),
format: sourceInfo?.format,
// v4.10.0: Import tracking metadata
// Import tracking metadata
...(trackingContext && {
importIds: [trackingContext.importId],
projectId: trackingContext.projectId,
@ -1161,7 +1161,7 @@ export class ImportCoordinator {
if (options.createRelationships && row.relationships) {
for (const rel of row.relationships) {
try {
// CRITICAL FIX (v3.43.2): Prevent infinite placeholder creation loop
// CRITICAL FIX: Prevent infinite placeholder creation loop
// Find or create target entity using EXACT matching only
let targetEntityId: string | undefined
@ -1195,7 +1195,7 @@ export class ImportCoordinator {
name: rel.to,
placeholder: true,
inferredFrom: entity.name,
// v4.10.0: Import tracking metadata
// Import tracking metadata
...(trackingContext && {
importIds: [trackingContext.importId],
projectId: trackingContext.projectId,
@ -1221,11 +1221,11 @@ export class ImportCoordinator {
from: entityId,
to: targetEntityId,
type: rel.type,
confidence: rel.confidence, // v4.2.0: Top-level field
weight: rel.weight || 1.0, // v4.2.0: Top-level field
confidence: rel.confidence, // Top-level field
weight: rel.weight || 1.0, // Top-level field
metadata: {
evidence: rel.evidence,
// v4.10.0: Import tracking metadata (will be merged in batch creation)
// Import tracking metadata (will be merged in batch creation)
...(trackingContext && {
importIds: [trackingContext.importId],
projectId: trackingContext.projectId,
@ -1242,7 +1242,7 @@ export class ImportCoordinator {
}
}
// Streaming import: Progressive flush with dynamic interval adjustment (v4.2.0+)
// Streaming import: Progressive flush with dynamic interval adjustment
entitiesSinceFlush++
if (entitiesSinceFlush >= currentFlushInterval) {
@ -1307,7 +1307,7 @@ export class ImportCoordinator {
}
// Batch create all relationships using brain.relateMany() for performance
// v4.9.0: Enhanced with type-based inference and semantic metadata
// Enhanced with type-based inference and semantic metadata
if (options.createRelationships && relationships.length > 0) {
try {
const relationshipParams = relationships.map(rel => {
@ -1331,7 +1331,7 @@ export class ImportCoordinator {
type: verbType, // Enhanced type
metadata: {
...((rel as any).metadata || {}),
relationshipType: 'semantic', // v4.9.0: Distinguish from VFS/provenance
relationshipType: 'semantic', // Distinguish from VFS/provenance
inferredType: verbType !== rel.type, // Track if type was enhanced
originalType: rel.type
}
@ -1369,7 +1369,7 @@ export class ImportCoordinator {
}
}
// v5.7.0: Schedule background deduplication (debounced 5 minutes)
// Schedule background deduplication (debounced 5 minutes)
if (trackingContext && trackingContext.importId) {
this.backgroundDedup.scheduleDedup(trackingContext.importId)
}
@ -1457,7 +1457,7 @@ export class ImportCoordinator {
}
}
// YAML: entities -> rows (v4.2.0)
// YAML: entities -> rows
if (format === 'yaml') {
const rows = result.entities.map((entity: any) => ({
entity,
@ -1477,7 +1477,7 @@ export class ImportCoordinator {
}
}
// DOCX: entities -> rows (v4.2.0)
// DOCX: entities -> rows
if (format === 'docx') {
const rows = result.entities.map((entity: any) => ({
entity,
@ -1502,7 +1502,7 @@ export class ImportCoordinator {
}
/**
* Validate options and reject deprecated v3.x options (v4.0.0+)
* Validate options and reject deprecated v3.x options
* Throws clear errors with migration guidance
*/
private validateOptions(options: any): void {
@ -1657,7 +1657,7 @@ ${optionDetails}
}
/**
* Get progressive flush interval based on CURRENT entity count (v4.2.0+)
* Get progressive flush interval based on CURRENT entity count
*
* Unlike adaptive intervals (which require knowing total count upfront),
* progressive intervals adjust dynamically as import proceeds.
@ -1699,7 +1699,7 @@ ${optionDetails}
/**
* Infer relationship type based on entity types and context
* v4.9.0: Semantic relationship enhancement
* Semantic relationship enhancement
*
* @param sourceType - Type of source entity
* @param targetType - Type of target entity
@ -1769,7 +1769,7 @@ ${optionDetails}
/**
* Count entities by type for document metadata
* v4.9.0: Used for document entity statistics
* Used for document entity statistics
*
* @param rows - Extracted rows from import
* @returns Record of entity type counts

14
src/importers/SmartCSVImporter.ts
View file

@ -42,17 +42,17 @@ export interface SmartCSVOptions extends FormatHandlerOptions {
csvDelimiter?: string
csvHeaders?: boolean
/** Progress callback (v3.39.0: Enhanced with performance metrics) */
/** Progress callback (Enhanced with performance metrics) */
onProgress?: (stats: {
processed: number
total: number
entities: number
relationships: number
/** Rows per second (v3.39.0) */
/** Rows per second */
throughput?: number
/** Estimated time remaining in ms (v3.39.0) */
/** Estimated time remaining in ms */
eta?: number
/** Current phase (v3.39.0) */
/** Current phase */
phase?: string
}) => void
}
@ -169,7 +169,7 @@ export class SmartCSVImporter {
}
// Parse CSV using existing handler
// v4.5.0: Pass progress hooks to handler for file parsing progress
// Pass progress hooks to handler for file parsing progress
const processedData = await this.csvHandler.process(buffer, {
...options,
csvDelimiter: opts.csvDelimiter,
@ -217,7 +217,7 @@ export class SmartCSVImporter {
// Detect column names
const columns = this.detectColumns(rows[0], opts)
// Process each row with BATCHED PARALLEL PROCESSING (v3.39.0)
// Process each row with BATCHED PARALLEL PROCESSING
const extractedRows: ExtractedRow[] = []
const entityMap = new Map<string, string>()
const stats = {
@ -375,7 +375,7 @@ export class SmartCSVImporter {
total: rows.length,
entities: extractedRows.reduce((sum, row) => sum + 1 + row.relatedEntities.length, 0),
relationships: extractedRows.reduce((sum, row) => sum + row.relationships.length, 0),
// Additional performance metrics (v3.39.0)
// Additional performance metrics
throughput: Math.round(rowsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining),
phase: 'extracting'

6
src/importers/SmartDOCXImporter.ts
View file

@ -9,7 +9,7 @@
* - NaturalLanguageProcessor for relationship inference
* - Hierarchical relationship creation based on heading hierarchy
*
* v4.2.0: New format handler
* New format handler
* NO MOCKS - Production-ready implementation
*/
@ -187,7 +187,7 @@ export class SmartDOCXImporter {
await this.init()
}
// v4.5.0: Report parsing start
// Report parsing start
options.onProgress?.({
processed: 0,
entities: 0,
@ -200,7 +200,7 @@ export class SmartDOCXImporter {
// Extract HTML for structure analysis (headings, tables)
const htmlResult = await mammoth.convertToHtml({ buffer })
// v4.5.0: Report parsing complete
// Report parsing complete
options.onProgress?.({
processed: 0,
entities: 0,

28
src/importers/SmartExcelImporter.ts
View file

@ -36,17 +36,17 @@ export interface SmartExcelOptions extends FormatHandlerOptions {
typeColumn?: string // e.g., "Type", "Category"
relatedColumn?: string // e.g., "Related Terms", "See Also"
/** Progress callback (v3.38.0: Enhanced with performance metrics) */
/** Progress callback (Enhanced with performance metrics) */
onProgress?: (stats: {
processed: number
total: number
entities: number
relationships: number
/** Rows per second (v3.38.0) */
/** Rows per second */
throughput?: number
/** Estimated time remaining in ms (v3.38.0) */
/** Estimated time remaining in ms */
eta?: number
/** Current phase (v3.38.0) */
/** Current phase */
phase?: string
}) => void
}
@ -111,7 +111,7 @@ export interface SmartExcelResult {
}
}
/** Sheet-specific data for VFS extraction (v4.2.0) */
/** Sheet-specific data for VFS extraction */
sheets?: Array<{
name: string
rows: ExtractedRow[]
@ -161,7 +161,7 @@ export class SmartExcelImporter {
const opts = {
enableNeuralExtraction: true,
enableRelationshipInference: true,
// CONCEPT EXTRACTION PRODUCTION-READY (v3.33.0+):
// CONCEPT EXTRACTION PRODUCTION-READY:
// Type embeddings are now pre-computed at build time - zero runtime cost!
// All 42 noun types + 127 verb types instantly available
//
@ -184,7 +184,7 @@ export class SmartExcelImporter {
}
// Parse Excel using existing handler
// v4.5.0: Pass progress hooks to handler for file parsing progress
// Pass progress hooks to handler for file parsing progress
const processedData = await this.excelHandler.process(buffer, {
...options,
totalBytes: buffer.length,
@ -227,7 +227,7 @@ export class SmartExcelImporter {
return this.emptyResult(startTime)
}
// CRITICAL FIX (v4.8.6): Detect columns per-sheet, not globally
// CRITICAL FIX: Detect columns per-sheet, not globally
// Different sheets may have different column structures (Term vs Name, etc.)
// Group rows by sheet and detect columns for each sheet separately
const rowsBySheet = new Map<string, typeof rows>()
@ -247,7 +247,7 @@ export class SmartExcelImporter {
}
}
// Process each row with BATCHED PARALLEL PROCESSING (v3.38.0)
// Process each row with BATCHED PARALLEL PROCESSING
const extractedRows: ExtractedRow[] = []
const entityMap = new Map<string, string>()
const stats = {
@ -269,7 +269,7 @@ export class SmartExcelImporter {
chunk.map(async (row, chunkIndex) => {
const i = chunkStart + chunkIndex
// CRITICAL FIX (v4.8.6): Use sheet-specific column mapping
// CRITICAL FIX: Use sheet-specific column mapping
const sheet = row._sheet || 'default'
const columns = columnsBySheet.get(sheet) || this.detectColumns(row, opts)
@ -353,7 +353,7 @@ export class SmartExcelImporter {
}
// ============================================
// v4.9.0: Enhanced column-based relationship detection
// Enhanced column-based relationship detection
// ============================================
// Parse explicit "Related Terms" column
if (relatedTerms) {
@ -379,7 +379,7 @@ export class SmartExcelImporter {
}
}
// v4.9.0: Check for additional relationship-indicating columns
// Check for additional relationship-indicating columns
// Expanded patterns for various relationship types
const relationshipColumnPatterns = [
{ pattern: /^(location|home|lives in|resides|dwelling|place)$/i, defaultType: VerbType.LocatedAt },
@ -485,14 +485,14 @@ export class SmartExcelImporter {
total: rows.length,
entities: extractedRows.reduce((sum, row) => sum + 1 + row.relatedEntities.length, 0),
relationships: extractedRows.reduce((sum, row) => sum + row.relationships.length, 0),
// Additional performance metrics (v3.38.0)
// Additional performance metrics
throughput: Math.round(rowsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining),
phase: 'extracting'
} as any)
}
// Group rows by sheet for VFS extraction (v4.2.0)
// Group rows by sheet for VFS extraction
const sheetGroups = new Map<string, ExtractedRow[]>()
extractedRows.forEach((extractedRow, index) => {
const originalRow = rows[index]

6
src/importers/SmartJSONImporter.ts
View file

@ -167,7 +167,7 @@ export class SmartJSONImporter {
...options
}
// v4.5.0: Report parsing start
// Report parsing start
opts.onProgress({
processed: 0,
entities: 0,
@ -186,7 +186,7 @@ export class SmartJSONImporter {
jsonData = data
}
// v4.5.0: Report parsing complete, starting traversal
// Report parsing complete, starting traversal
opts.onProgress({
processed: 0,
entities: 0,
@ -228,7 +228,7 @@ export class SmartJSONImporter {
}
)
// v4.5.0: Report completion
// Report completion
opts.onProgress({
processed: nodesProcessed,
entities: entities.length,

6
src/importers/SmartMarkdownImporter.ts
View file

@ -174,7 +174,7 @@ export class SmartMarkdownImporter {
...options
}
// v4.5.0: Report parsing start
// Report parsing start
opts.onProgress({
processed: 0,
total: 0,
@ -185,7 +185,7 @@ export class SmartMarkdownImporter {
// Parse markdown into sections
const parsedSections = this.parseMarkdown(markdown, opts)
// v4.5.0: Report parsing complete
// Report parsing complete
opts.onProgress({
processed: 0,
total: parsedSections.length,
@ -218,7 +218,7 @@ export class SmartMarkdownImporter {
})
}
// v4.5.0: Report completion
// Report completion
const totalEntities = sections.reduce((sum, s) => sum + s.entities.length, 0)
const totalRelationships = sections.reduce((sum, s) => sum + s.relationships.length, 0)
opts.onProgress({

16
src/importers/SmartPDFImporter.ts
View file

@ -40,17 +40,17 @@ export interface SmartPDFOptions extends FormatHandlerOptions {
/** Group by page or full document */
groupBy?: 'page' | 'document'
/** Progress callback (v3.39.0: Enhanced with performance metrics) */
/** Progress callback (Enhanced with performance metrics) */
onProgress?: (stats: {
processed: number
total: number
entities: number
relationships: number
/** Sections per second (v3.39.0) */
/** Sections per second */
throughput?: number
/** Estimated time remaining in ms (v3.39.0) */
/** Estimated time remaining in ms */
eta?: number
/** Current phase (v3.39.0) */
/** Current phase */
phase?: string
}) => void
}
@ -181,7 +181,7 @@ export class SmartPDFImporter {
}
// Parse PDF using existing handler
// v4.5.0: Pass progress hooks to handler for file parsing progress
// Pass progress hooks to handler for file parsing progress
const processedData = await this.pdfHandler.process(buffer, {
...options,
totalBytes: buffer.length,
@ -228,7 +228,7 @@ export class SmartPDFImporter {
// Group data by page or combine into single document
const grouped = this.groupData(data, opts)
// Process each group with BATCHED PARALLEL PROCESSING (v3.39.0)
// Process each group with BATCHED PARALLEL PROCESSING
const sections: ExtractedSection[] = []
const entityMap = new Map<string, string>()
const stats = {
@ -270,7 +270,7 @@ export class SmartPDFImporter {
total: totalGroups,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0),
// Additional performance metrics (v3.39.0)
// Additional performance metrics
throughput: Math.round(sectionsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining),
phase: 'extracting'
@ -367,7 +367,7 @@ export class SmartPDFImporter {
const combinedText = texts.join('\n\n')
// Parallel extraction: entities AND concepts at the same time (v3.39.0)
// Parallel extraction: entities AND concepts at the same time
const [extractedEntities, concepts] = await Promise.all([
// Extract entities if enabled
options.enableNeuralExtraction && combinedText.length > 0

6
src/importers/SmartYAMLImporter.ts
View file

@ -8,7 +8,7 @@
* - NaturalLanguageProcessor for relationship inference
* - Hierarchical relationship creation (parent-child, contains, etc.)
*
* v4.2.0: New format handler
* New format handler
* NO MOCKS - Production-ready implementation
*/
@ -159,7 +159,7 @@ export class SmartYAMLImporter {
): Promise<SmartYAMLResult> {
const startTime = Date.now()
// v4.5.0: Report parsing start
// Report parsing start
options.onProgress?.({
processed: 0,
entities: 0,
@ -178,7 +178,7 @@ export class SmartYAMLImporter {
throw new Error(`Failed to parse YAML: ${error.message}`)
}
// v4.5.0: Report parsing complete
// Report parsing complete
options.onProgress?.({
processed: 0,
entities: 0,

26
src/importers/VFSStructureGenerator.ts
View file

@ -40,10 +40,10 @@ export interface VFSStructureOptions {
/** Create metadata file */
createMetadataFile?: boolean
/** Import tracking context (v4.10.0) */
/** Import tracking context */
trackingContext?: TrackingContext
/** Progress callback (v4.11.1) - Reports VFS creation progress */
/** Progress callback - Reports VFS creation progress */
onProgress?: (progress: {
stage: 'directories' | 'entities' | 'metadata'
message: string
@ -98,7 +98,7 @@ export class VFSStructureGenerator {
// Get brain's cached VFS instance (creates if doesn't exist)
this.vfs = this.brain.vfs
// CRITICAL FIX (v4.10.2): Always call vfs.init() explicitly
// CRITICAL FIX: Always call vfs.init() explicitly
// The previous code tried to check if initialized via stat('/') but this was unreliable
// vfs.init() is idempotent, so calling it multiple times is safe
await this.vfs.init()
@ -123,7 +123,7 @@ export class VFSStructureGenerator {
// Ensure VFS is initialized
await this.init()
// v4.11.1: Calculate total operations for progress tracking
// Calculate total operations for progress tracking
const groups = this.groupEntities(importResult, options)
const totalEntities = Array.from(groups.values()).reduce((sum, entities) => sum + entities.length, 0)
const totalOperations =
@ -161,7 +161,7 @@ export class VFSStructureGenerator {
try {
await this.vfs.mkdir(options.rootPath, {
recursive: true,
metadata: trackingMetadata // v4.10.0: Add tracking metadata
metadata: trackingMetadata // Add tracking metadata
})
result.directories.push(options.rootPath)
result.operations++
@ -179,7 +179,7 @@ export class VFSStructureGenerator {
if (options.preserveSource && options.sourceBuffer && options.sourceFilename) {
const sourcePath = `${options.rootPath}/_source${this.getExtension(options.sourceFilename)}`
await this.vfs.writeFile(sourcePath, options.sourceBuffer, {
metadata: trackingMetadata // v4.10.0: Add tracking metadata
metadata: trackingMetadata // Add tracking metadata
})
result.files.push({
path: sourcePath,
@ -200,7 +200,7 @@ export class VFSStructureGenerator {
try {
await this.vfs.mkdir(groupPath, {
recursive: true,
metadata: trackingMetadata // v4.10.0: Add tracking metadata
metadata: trackingMetadata // Add tracking metadata
})
result.directories.push(groupPath)
result.operations++
@ -242,7 +242,7 @@ export class VFSStructureGenerator {
await this.vfs.writeFile(entityPath, JSON.stringify(entityJson, null, 2), {
metadata: {
...trackingMetadata, // v4.10.0: Add tracking metadata
...trackingMetadata, // Add tracking metadata
entityId: extracted.entity.id
}
})
@ -253,7 +253,7 @@ export class VFSStructureGenerator {
})
result.operations++
// v4.11.1: Report progress every 10 entities (or on last entity)
// Report progress every 10 entities (or on last entity)
if (entityCount % 10 === 0 || entityCount === entities.length) {
reportProgress('entities', `Created ${entityCount}/${entities.length} ${groupName} files`)
}
@ -280,7 +280,7 @@ export class VFSStructureGenerator {
}
await this.vfs.writeFile(relationshipsPath, JSON.stringify(relationshipsJson, null, 2), {
metadata: trackingMetadata // v4.10.0: Add tracking metadata
metadata: trackingMetadata // Add tracking metadata
})
result.files.push({
path: relationshipsPath,
@ -322,7 +322,7 @@ export class VFSStructureGenerator {
}
await this.vfs.writeFile(metadataPath, JSON.stringify(metadataJson, null, 2), {
metadata: trackingMetadata // v4.10.0: Add tracking metadata
metadata: trackingMetadata // Add tracking metadata
})
result.files.push({
path: metadataPath,
@ -332,7 +332,7 @@ export class VFSStructureGenerator {
reportProgress('metadata', 'Created metadata file')
}
// v4.11.1: Final progress update
// Final progress update
if (options.onProgress) {
options.onProgress({
stage: 'metadata',
@ -355,7 +355,7 @@ export class VFSStructureGenerator {
): Map<string, typeof importResult.rows> {
const groups = new Map<string, typeof importResult.rows>()
// Handle sheet-based grouping (v4.2.0)
// Handle sheet-based grouping
if (options.groupBy === 'sheet' && importResult.sheets && importResult.sheets.length > 0) {
for (const sheet of importResult.sheets) {
groups.set(sheet.name, sheet.rows)

6
src/index.ts
View file

@ -70,7 +70,7 @@ export type {
NeuralImportOptions
} from './cortex/neuralImport.js'
// Export Neural Entity Extraction (v5.7.6 - Workshop request)
// Export Neural Entity Extraction
export { NeuralEntityExtractor } from './neural/entityExtractor.js'
export { SmartExtractor } from './neural/SmartExtractor.js'
export { SmartRelationshipExtractor } from './neural/SmartRelationshipExtractor.js'
@ -220,7 +220,7 @@ export {
// FileSystemStorage is exported separately to avoid browser build issues
export { FileSystemStorage } from './storage/adapters/fileSystemStorage.js'
// Export COW (Copy-on-Write) infrastructure for v5.0.0
// Export COW (Copy-on-Write) infrastructure
// Enables premium augmentations to implement temporal features
import { CommitLog } from './storage/cow/CommitLog.js'
import { CommitObject, CommitBuilder } from './storage/cow/CommitObject.js'
@ -543,7 +543,7 @@ export type {
MCPTool
}
// ============= Integration Hub (v7.4.0) =============
// ============= Integration Hub =============
// Connect Brainy to Excel, Power BI, Google Sheets, and more
// Enable with: new Brainy({ integrations: true })

2
src/integrations/index.ts
View file

@ -7,7 +7,7 @@
* - Real-time dashboards (SSE streaming)
* - External webhooks (push notifications)
*
* @example Enable integrations (v7.4.0 - recommended)
* @example Enable integrations (recommended)
* ```typescript
* import { Brainy } from '@soulcraft/brainy'
*

6
src/interfaces/IIndex.ts
View file

@ -1,5 +1,5 @@
/**
* Unified Index Interface (v3.35.0+)
* Unified Index Interface
*
* Standardizes index lifecycle across all index types in Brainy.
* All indexes (HNSW Vector, Graph Adjacency, Metadata Field) implement this interface
@ -57,7 +57,7 @@ export interface RebuildOptions {
* - On-demand: Large datasets loaded adaptively via UnifiedCache
*
* This option is kept for backwards compatibility but is ignored.
* The system always uses adaptive caching (v3.36.0+).
* The system always uses adaptive caching.
*/
lazy?: boolean
@ -104,7 +104,7 @@ export interface IIndex {
* - Provide progress reporting for large datasets
* - Recover gracefully from partial failures
*
* Adaptive Caching (v3.36.0+):
* Adaptive Caching:
* System automatically chooses optimal strategy:
* - Small datasets: Preload all data at init for zero-latency access
* - Large datasets: Load on-demand via UnifiedCache for memory efficiency

20
src/neural/entityExtractor.ts
View file

@ -2,7 +2,7 @@
* Neural Entity Extractor using Brainy's NounTypes
* Uses embeddings and similarity matching for accurate type detection
*
* v4.2.0: Now powered by SmartExtractor for ultra-neural classification
* Now powered by SmartExtractor for ultra-neural classification
* PRODUCTION-READY with caching support
*/
@ -24,7 +24,7 @@ export interface ExtractedEntity {
type: NounType
position: { start: number; end: number }
confidence: number
weight?: number // v4.2.0: Entity importance/salience
weight?: number // Entity importance/salience
vector?: Vector
metadata?: any
}
@ -39,7 +39,7 @@ export class NeuralEntityExtractor {
// Entity extraction cache
private cache: EntityExtractionCache
// Runtime embedding cache for performance (v3.38.0)
// Runtime embedding cache for performance
// Caches candidate embeddings during an extraction session to avoid redundant model calls
private embeddingCache: Map<string, Vector> = new Map()
private embeddingCacheStats = {
@ -48,7 +48,7 @@ export class NeuralEntityExtractor {
size: 0
}
// v4.2.0: SmartExtractor for ultra-neural classification
// SmartExtractor for ultra-neural classification
private smartExtractor: SmartExtractor
constructor(brain: Brainy | Brainy<any>, cacheOptions?: EntityCacheOptions) {
@ -63,7 +63,7 @@ export class NeuralEntityExtractor {
/**
* Initialize type embeddings for neural matching
* PRODUCTION OPTIMIZATION (v3.33.0): Uses pre-computed embeddings from build time
* PRODUCTION OPTIMIZATION: Uses pre-computed embeddings from build time
* Zero runtime cost - embeddings are loaded instantly from embedded data
*/
private async initializeTypeEmbeddings(requestedTypes?: NounType[]): Promise<void> {
@ -109,7 +109,7 @@ export class NeuralEntityExtractor {
}
}
): Promise<ExtractedEntity[]> {
// PRODUCTION OPTIMIZATION (v3.33.0): Load pre-computed type embeddings
// PRODUCTION OPTIMIZATION: Load pre-computed type embeddings
// Zero runtime cost - embeddings were computed at build time
await this.initializeTypeEmbeddings(options?.types)
@ -138,7 +138,7 @@ export class NeuralEntityExtractor {
// Step 1: Extract potential entities using patterns
const candidates = await this.extractCandidates(text)
// Step 2: Classify each candidate using SmartExtractor (v4.2.0)
// Step 2: Classify each candidate using SmartExtractor
for (const candidate of candidates) {
// Use SmartExtractor for unified neural + rule-based classification
const classification = await this.smartExtractor.extract(candidate.text, {
@ -357,7 +357,7 @@ export class NeuralEntityExtractor {
}
/**
* Get embedding for text with caching (v3.38.0)
* Get embedding for text with caching
*
* PERFORMANCE OPTIMIZATION: Caches embeddings during extraction session
* to avoid redundant model calls for repeated text (common in large imports)
@ -509,7 +509,7 @@ export class NeuralEntityExtractor {
}
/**
* Clear embedding cache (v3.38.0)
* Clear embedding cache
*
* Clears the runtime embedding cache. Useful for:
* - Freeing memory after large imports
@ -525,7 +525,7 @@ export class NeuralEntityExtractor {
}
/**
* Get embedding cache statistics (v3.38.0)
* Get embedding cache statistics
*
* Returns performance metrics for the embedding cache:
* - hits: Number of cache hits (avoided model calls)

2
src/neural/naturalLanguageProcessor.ts
View file

@ -99,7 +99,7 @@ export class NaturalLanguageProcessor {
/**
* Initialize embeddings for all NounTypes and VerbTypes
* PRODUCTION OPTIMIZATION (v3.33.0): Uses pre-computed type embeddings
* PRODUCTION OPTIMIZATION: Uses pre-computed type embeddings
* Zero runtime cost - embeddings are loaded instantly from embedded data
*/
private async initializeTypeEmbeddings(): Promise<void> {

4
src/neural/signals/PatternSignal.ts
View file

@ -122,7 +122,7 @@ export class PatternSignal {
])
// Location patterns - MEDIUM PRIORITY (city/country format - requires more context)
// v4.11.2: Lower priority to avoid matching person names with commas
// Lower priority to avoid matching person names with commas
this.addPatterns(NounType.Location, 0.75, [
/\b[A-Z][a-z]+,\s*(?:Japan|China|France|Germany|Italy|Spain|Canada|Mexico|Brazil|India|Australia|Russia|UK|USA)\b/
])
@ -170,7 +170,7 @@ export class PatternSignal {
// Technology patterns (Thing type)
this.addPatterns(NounType.Thing, 0.82, [
/\b(?:JavaScript|TypeScript|Python|Java|Go|Rust|Swift|Kotlin)\b/,
/\bC\+\+(?!\w)/, // v4.11.2: Special handling for C++ (word boundary doesn't work with +)
/\bC\+\+(?!\w)/, // Special handling for C++ (word boundary doesn't work with +)
/\b(?:React|Vue|Angular|Node|Express|Django|Flask|Rails)\b/,
/\b(?:AWS|Azure|GCP|Docker|Kubernetes|Git|GitHub|GitLab)\b/,
/\b(?:API|SDK|CLI|IDE|framework|library|package|module)\b/i,

2
src/setup.ts
View file

@ -1,7 +1,7 @@
/**
* Brainy Setup - Minimal Polyfills
*
* ARCHITECTURE (v7.0.0):
* ARCHITECTURE:
* Brainy uses Candle WASM (Rust-based) for embeddings.
* No transformers.js or ONNX Runtime dependency, no hacks required.
*

108
src/storage/adapters/azureBlobStorage.ts
View file

@ -9,7 +9,7 @@
* 4. SAS Token
* 5. Azure AD (OAuth2) via DefaultAzureCredential
*
* v4.0.0: Fully compatible with metadata/vector separation architecture
* Fully compatible with metadata/vector separation architecture
*/
import {
@ -65,7 +65,7 @@ const MAX_AZURE_PAGE_SIZE = 5000
* 3. Storage Account Key - if accountName + accountKey provided
* 4. SAS Token - if accountName + sasToken provided
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed pagination overrides
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -105,7 +105,7 @@ export class AzureBlobStorage extends BaseStorage {
// Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance
// Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Multi-level cache manager for efficient data access
@ -115,7 +115,7 @@ export class AzureBlobStorage extends BaseStorage {
// Module logger
private logger = createModuleLogger('AzureBlobStorage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races
// HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>()
/**
@ -162,7 +162,7 @@ export class AzureBlobStorage extends BaseStorage {
}
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Azure Functions),
* strict locally. Zero-config optimization.
@ -171,7 +171,6 @@ export class AzureBlobStorage extends BaseStorage {
* - `'strict'`: Traditional blocking init. Validates container and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: InitMode
@ -185,7 +184,7 @@ export class AzureBlobStorage extends BaseStorage {
this.sasToken = options.sasToken
this.readOnly = options.readOnly || false
// v7.3.0: Handle initMode
// Handle initMode
if (options.initMode) {
this.initMode = options.initMode
}
@ -201,7 +200,7 @@ export class AzureBlobStorage extends BaseStorage {
this.nounCacheManager = new CacheManager<HNSWNode>(options.cacheConfig)
this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig)
// v6.2.7: Write buffering always enabled - no env var check needed
// Write buffering always enabled - no env var check needed
}
/**
@ -216,7 +215,7 @@ export class AzureBlobStorage extends BaseStorage {
* Recent Azure improvements make parallel downloads very efficient
*
* @returns Azure Blob-optimized batch configuration
* @since v5.12.0 - Updated for native batch API
* Updated for native batch API
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -241,7 +240,6 @@ export class AzureBlobStorage extends BaseStorage {
*
* @param paths - Array of Azure blob paths to read
* @returns Map of path -> parsed JSON data (only successful reads)
* @since v5.12.0
*/
public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized()
@ -297,7 +295,7 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Initialize the storage adapter
*
* v7.3.0: Supports progressive initialization for fast cold starts
* Supports progressive initialization for fast cold starts
*
* | Mode | Init Time | When |
* |------|-----------|------|
@ -404,7 +402,7 @@ export class AzureBlobStorage extends BaseStorage {
this.verbCacheManager.clear()
prodLog.info('✅ Cache cleared - starting fresh')
// v7.3.0: Progressive vs Strict initialization
// Progressive vs Strict initialization
if (effectiveMode === 'progressive') {
// PROGRESSIVE MODE: Fast init, background validation
// Mark as initialized immediately - ready to accept operations
@ -413,7 +411,7 @@ export class AzureBlobStorage extends BaseStorage {
prodLog.info(`✅ Azure progressive init complete: ${this.containerName} (validation deferred)`)
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Schedule background tasks (non-blocking)
@ -434,7 +432,7 @@ export class AzureBlobStorage extends BaseStorage {
await this.initializeCounts()
this.countsLoaded = true
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Mark background tasks as complete (nothing to do in background)
@ -447,7 +445,7 @@ export class AzureBlobStorage extends BaseStorage {
}
// =============================================
// Progressive Initialization (v7.3.0+)
// Progressive Initialization
// =============================================
/**
@ -461,7 +459,6 @@ export class AzureBlobStorage extends BaseStorage {
*
* @protected
* @override
* @since v7.3.0
*/
protected async runBackgroundInit(): Promise<void> {
const startTime = Date.now()
@ -485,7 +482,6 @@ export class AzureBlobStorage extends BaseStorage {
* bucketValidated/bucketValidationError for lazy use.
*
* @private
* @since v7.3.0
*/
private async validateContainerInBackground(): Promise<void> {
try {
@ -517,7 +513,6 @@ export class AzureBlobStorage extends BaseStorage {
* Load counts from storage in background.
*
* @private
* @since v7.3.0
*/
private async loadCountsInBackground(): Promise<void> {
try {
@ -541,7 +536,6 @@ export class AzureBlobStorage extends BaseStorage {
* @throws Error if container validation fails
* @protected
* @override
* @since v7.3.0
*/
protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do
@ -665,7 +659,7 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage
// Removed checkVolumeMode() - write buffering always enabled for cloud storage
/**
* Flush noun buffer to Azure
@ -697,20 +691,20 @@ export class AzureBlobStorage extends BaseStorage {
await Promise.all(writes)
}
// v5.4.0: Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node)
}
@ -791,7 +785,7 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v5.4.0: Removed getNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed getNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Get a node from storage
@ -889,18 +883,18 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v5.4.0: Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Write an object to a specific path in Azure
* Primitive operation required by base class
*
* v7.3.0: Performs lazy container validation on first write in progressive mode.
* Performs lazy container validation on first write in progressive mode.
* @protected
*/
protected async writeObjectToPath(path: string, data: any): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy container validation for progressive init
// Lazy container validation for progressive init
await this.ensureValidatedForWrite()
try {
@ -954,12 +948,12 @@ export class AzureBlobStorage extends BaseStorage {
* Delete an object from a specific path in Azure
* Primitive operation required by base class
*
* v7.3.0: Performs lazy container validation on first delete in progressive mode.
* Performs lazy container validation on first delete in progressive mode.
* @protected
*/
protected async deleteObjectFromPath(path: string): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy container validation for progressive init
// Lazy container validation for progressive init
await this.ensureValidatedForWrite()
try {
@ -1209,20 +1203,20 @@ export class AzureBlobStorage extends BaseStorage {
})
}
// v5.4.0: Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation
/**
* Save an edge to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.verbWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding verb ${edge.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
this.verbCacheManager.set(edge.id, edge)
await this.verbWriteBuffer.add(edge.id, edge)
@ -1255,7 +1249,7 @@ export class AzureBlobStorage extends BaseStorage {
])
),
// CORE RELATIONAL DATA (v4.0.0)
// CORE RELATIONAL DATA
verb: edge.verb,
sourceId: edge.sourceId,
targetId: edge.targetId,
@ -1280,7 +1274,7 @@ export class AzureBlobStorage extends BaseStorage {
// Update cache
this.verbCacheManager.set(edge.id, edge)
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet
this.logger.trace(`Edge ${edge.id} saved successfully`)
@ -1298,7 +1292,7 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v5.4.0: Removed getVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed getVerb_internal - now inherit from BaseStorage's type-first implementation
/**
* Get an edge from storage
@ -1335,7 +1329,7 @@ export class AzureBlobStorage extends BaseStorage {
connections.set(Number(level), new Set(verbIds as string[]))
}
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
// Return HNSWVerb with core relational fields (NO metadata field)
const edge: Edge = {
id: data.id,
vector: data.vector,
@ -1346,7 +1340,7 @@ export class AzureBlobStorage extends BaseStorage {
sourceId: data.sourceId,
targetId: data.targetId
// ✅ NO metadata field in v4.0.0
// ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata()
}
@ -1375,13 +1369,13 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v5.4.0: Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed getNounsWithPagination - now inherit from BaseStorage's type-first implementation
// Removed getNounsWithPagination - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed getNounsByNounType_internal - now inherit from BaseStorage's type-first implementation
// Removed getNounsByNounType_internal - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed 3 verb query *_internal methods (getVerbsBySource, getVerbsByTarget, getVerbsByType) - now inherit from BaseStorage's type-first implementation
// Removed 3 verb query *_internal methods (getVerbsBySource, getVerbsByTarget, getVerbsByType) - now inherit from BaseStorage's type-first implementation
/**
* Clear all data from storage
@ -1393,7 +1387,7 @@ export class AzureBlobStorage extends BaseStorage {
this.logger.info('🧹 Clearing all data from Azure container...')
// Delete all blobs in container
// v5.6.1: listBlobsFlat() returns ALL blobs including _cow/ prefix
// listBlobsFlat() returns ALL blobs including _cow/ prefix
// This correctly deletes COW version control data (commits, trees, blobs, refs)
for await (const blob of this.containerClient!.listBlobsFlat()) {
if (blob.name) {
@ -1402,7 +1396,7 @@ export class AzureBlobStorage extends BaseStorage {
}
}
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -1461,12 +1455,12 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker blob exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
@ -1643,7 +1637,7 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Get a noun's vector for HNSW rebuild
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
@ -1653,7 +1647,7 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Save HNSW graph data for a noun
*
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths)
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Uses mutex locking to prevent read-modify-write races
*/
public async saveHNSWData(nounId: string, hnswData: {
@ -1662,7 +1656,7 @@ export class AzureBlobStorage extends BaseStorage {
}): Promise<void> {
const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races
// CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3])
@ -1682,7 +1676,7 @@ export class AzureBlobStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise)
try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths)
// Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId)
@ -1704,7 +1698,7 @@ export class AzureBlobStorage extends BaseStorage {
connections: connectionsMap
}
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
// Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
await this.saveNoun(updatedNoun)
} finally {
// Release lock (ALWAYS runs, even if error thrown)
@ -1715,7 +1709,7 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getHNSWData(nounId: string): Promise<{
level: number
@ -1744,7 +1738,7 @@ export class AzureBlobStorage extends BaseStorage {
/**
* Save HNSW system data (entry point, max level)
*
* CRITICAL FIX (v4.10.1): Optimistic locking with ETags to prevent race conditions
* CRITICAL FIX: Optimistic locking with ETags to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null
@ -1840,7 +1834,7 @@ export class AzureBlobStorage extends BaseStorage {
}
/**
* Set the access tier for a specific blob (v4.0.0 cost optimization)
* Set the access tier for a specific blob (cost optimization)
* Azure Blob Storage tiers:
* - Hot: $0.0184/GB/month - Frequently accessed data
* - Cool: $0.01/GB/month - Infrequently accessed data (45% cheaper)
@ -1906,7 +1900,7 @@ export class AzureBlobStorage extends BaseStorage {
}
/**
* Set access tier for multiple blobs in batch (v4.0.0 cost optimization)
* Set access tier for multiple blobs in batch (cost optimization)
* Efficiently move large numbers of blobs between tiers for cost optimization
*
* @param blobs - Array of blob names and their target tiers
@ -2128,7 +2122,7 @@ export class AzureBlobStorage extends BaseStorage {
}
/**
* Set lifecycle management policy for automatic tier transitions and deletions (v4.0.0)
* Set lifecycle management policy for automatic tier transitions and deletions
* Automates cost optimization by moving old data to cheaper tiers or deleting it
*
* Azure Lifecycle Management rules run once per day and apply to the entire container.

43
src/storage/adapters/baseStorageAdapter.ts
View file

@ -19,7 +19,7 @@ import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/field
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
// =============================================================================
// Progressive Initialization Types (v7.3.0+)
// Progressive Initialization Types
// =============================================================================
/**
@ -29,7 +29,6 @@ import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
* fast cold starts in serverless environments while maintaining strict
* validation for local development.
*
* @since v7.3.0
*
* | Mode | Description | Use Case |
* |------|-------------|----------|
@ -96,7 +95,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
abstract getVerbMetadata(id: string): Promise<any | null>
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
// These methods enable HNSW index rebuilding after container restarts
abstract getNounVector(id: string): Promise<number[] | null>
@ -139,7 +138,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* relateMany(), and import operations to automatically adapt to storage capabilities.
*
* @returns Batch configuration optimized for this storage type
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
// Conservative defaults that work safely across all storage types
@ -295,7 +293,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
}> = new Map()
// =============================================
// Progressive Initialization State (v7.3.0+)
// Progressive Initialization State
// =============================================
// These properties enable fast cold starts in cloud environments
// by deferring validation and count loading to background tasks.
@ -308,7 +306,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* - `'strict'`: Always validate during init (traditional behavior)
*
* @protected
* @since v7.3.0
*/
protected initMode: InitMode = 'auto'
@ -319,7 +316,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* after the first successful write operation or background validation.
*
* @protected
* @since v7.3.0
*/
protected bucketValidated = false
@ -329,7 +325,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Stored here so subsequent operations can fail fast with the same error.
*
* @protected
* @since v7.3.0
*/
protected bucketValidationError: Error | null = null
@ -340,7 +335,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Operations work immediately; counts become accurate after background load.
*
* @protected
* @since v7.3.0
*/
protected countsLoaded = false
@ -350,7 +344,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Useful for tests and diagnostics to ensure full initialization.
*
* @protected
* @since v7.3.0
*/
protected backgroundTasksComplete = false
@ -361,7 +354,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* before proceeding (e.g., in tests).
*
* @protected
* @since v7.3.0
*/
protected backgroundInitPromise: Promise<void> | null = null
@ -1057,7 +1049,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL
// =============================================
// Smart Count Batching (v3.32.3+)
// Smart Count Batching
// =============================================
// Count batching state - mirrors statistics batching pattern
@ -1112,7 +1104,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex()
await mutex.runExclusive(`count-entity-${type}`, async () => {
this.incrementEntityCount(type)
// Smart batching (v3.32.3+): Adapts to storage type
// Smart batching: Adapts to storage type
// - Cloud storage (GCS, S3): Batches 10 ops OR 5 seconds
// - Local storage (File, Memory): Persists immediately
await this.scheduleCountPersist()
@ -1147,13 +1139,13 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex()
await mutex.runExclusive(`count-entity-${type}`, async () => {
this.decrementEntityCount(type)
// Smart batching (v3.32.3+): Adapts to storage type
// Smart batching: Adapts to storage type
await this.scheduleCountPersist()
})
}
/**
* Increment verb count - O(1) operation (v4.1.2: now synchronous)
* Increment verb count - O(1) operation (now synchronous)
* Protected by storage-specific mechanisms (mutex, distributed consensus, etc.)
* @param type The verb type
*/
@ -1168,7 +1160,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
}
/**
* Thread-safe increment for verb counts (v4.1.2)
* Thread-safe increment for verb counts
* Uses mutex for single-node, distributed consensus for multi-node
* @param type The verb type
*/
@ -1176,13 +1168,13 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex()
await mutex.runExclusive(`count-verb-${type}`, async () => {
this.incrementVerbCount(type)
// Smart batching (v3.32.3+): Adapts to storage type
// Smart batching: Adapts to storage type
await this.scheduleCountPersist()
})
}
/**
* Decrement verb count - O(1) operation (v4.1.2: now synchronous)
* Decrement verb count - O(1) operation (now synchronous)
* @param type The verb type
*/
protected decrementVerbCount(type: string): void {
@ -1203,20 +1195,20 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
}
/**
* Thread-safe decrement for verb counts (v4.1.2)
* Thread-safe decrement for verb counts
* @param type The verb type
*/
protected async decrementVerbCountSafe(type: string): Promise<void> {
const mutex = getGlobalMutex()
await mutex.runExclusive(`count-verb-${type}`, async () => {
this.decrementVerbCount(type)
// Smart batching (v3.32.3+): Adapts to storage type
// Smart batching: Adapts to storage type
await this.scheduleCountPersist()
})
}
// =============================================
// Smart Batching Methods (v3.32.3+)
// Smart Batching Methods
// =============================================
/**
@ -1235,7 +1227,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
}
// =============================================
// Progressive Initialization Methods (v7.3.0+)
// Progressive Initialization Methods
// =============================================
/**
@ -1254,7 +1246,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
*
* @returns `true` if running in a detected cloud environment
* @protected
* @since v7.3.0
*/
protected detectCloudEnvironment(): boolean {
return !!(
@ -1274,7 +1265,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
*
* @returns The resolved init mode (`'progressive'` or `'strict'`)
* @protected
* @since v7.3.0
*/
protected resolveInitMode(): 'progressive' | 'strict' {
if (this.initMode === 'auto') {
@ -1295,7 +1285,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Always call `super.scheduleBackgroundInit()` first.
*
* @protected
* @since v7.3.0
*/
protected scheduleBackgroundInit(): void {
// Create a promise that tracks all background work
@ -1320,7 +1309,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* The default implementation does nothing (for local storage adapters).
*
* @protected
* @since v7.3.0
*/
protected async runBackgroundInit(): Promise<void> {
// Default implementation: nothing to do for local storage
@ -1348,7 +1336,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* ```
*
* @public
* @since v7.3.0
*/
public async awaitBackgroundInit(): Promise<void> {
if (this.backgroundInitPromise) {
@ -1361,7 +1348,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
*
* @returns `true` if all background tasks are complete
* @public
* @since v7.3.0
*/
public isBackgroundInitComplete(): boolean {
return this.backgroundTasksComplete
@ -1379,7 +1365,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
*
* @throws Error if bucket validation fails
* @protected
* @since v7.3.0
*/
protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do

109
src/storage/adapters/fileSystemStorage.ts
View file

@ -59,7 +59,7 @@ try {
* File system storage adapter for Node.js environments
* Uses the file system to store data in the specified directory structure
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -91,12 +91,12 @@ export class FileSystemStorage extends BaseStorage {
private lockTimers: Map<string, NodeJS.Timeout> = new Map() // Track timers for cleanup
private allTimers: Set<NodeJS.Timeout> = new Set() // Track all timers for cleanup
// CRITICAL FIX (v4.10.1): Mutex locks for HNSW concurrency control
// CRITICAL FIX: Mutex locks for HNSW concurrency control
// Prevents read-modify-write races during concurrent neighbor updates at scale (1000+ ops)
// Matches MemoryStorage and OPFSStorage behavior (tested in production)
private hnswLocks = new Map<string, Promise<void>>()
// Compression configuration (v4.0.0)
// Compression configuration
private compressionEnabled: boolean = true // Enable gzip compression by default for 60-80% disk savings
private compressionLevel: number = 6 // zlib compression level (1-9, default: 6 = balanced)
@ -136,7 +136,6 @@ export class FileSystemStorage extends BaseStorage {
* - Parallel processing supported
*
* @returns FileSystem-optimized batch configuration
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -179,7 +178,7 @@ export class FileSystemStorage extends BaseStorage {
try {
// Initialize directory paths now that path module is loaded
// Clean directory structure (v4.7.2+)
// Clean directory structure
this.nounsDir = path.join(this.rootDir, 'entities/nouns/hnsw')
this.verbsDir = path.join(this.rootDir, 'entities/verbs/hnsw')
this.metadataDir = path.join(this.rootDir, 'entities/nouns/metadata') // Legacy reference
@ -245,7 +244,7 @@ export class FileSystemStorage extends BaseStorage {
// Always use fixed depth after migration/detection
this.cachedShardingDepth = this.SHARDING_DEPTH
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
} catch (error) {
console.error('Error initializing FileSystemStorage:', error)
@ -281,7 +280,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Save a node to storage
* CRITICAL FIX (v4.10.3): Added atomic write pattern to prevent file corruption during concurrent imports
* CRITICAL FIX: Added atomic write pattern to prevent file corruption during concurrent imports
*/
protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized()
@ -303,7 +302,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${filePath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try {
// ATOMIC WRITE SEQUENCE (v4.10.3):
// ATOMIC WRITE SEQUENCE:
// 1. Write to temp file
await this.ensureDirectoryExists(path.dirname(tempPath))
await fs.promises.writeFile(tempPath, JSON.stringify(serializableNode, null, 2))
@ -320,7 +319,7 @@ export class FileSystemStorage extends BaseStorage {
throw error
}
// Count tracking happens in baseStorage.saveNounMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveNounMetadata_internal
// This fixes the race condition where metadata didn't exist yet
}
@ -361,7 +360,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Get all nodes from storage
* CRITICAL FIX (v3.43.2): Now scans sharded subdirectories (depth=1)
* CRITICAL FIX: Now scans sharded subdirectories (depth=1)
* Previously only scanned flat directory, causing rebuild to find 0 entities
*/
protected async getAllNodes(): Promise<HNSWNode[]> {
@ -406,7 +405,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Get nodes by noun type
* CRITICAL FIX (v3.43.2): Now scans sharded subdirectories (depth=1)
* CRITICAL FIX: Now scans sharded subdirectories (depth=1)
* @param nounType The noun type to filter by
* @returns Promise that resolves to an array of nodes of the specified noun type
*/
@ -462,7 +461,7 @@ export class FileSystemStorage extends BaseStorage {
const filePath = this.getNodePath(id)
// Load metadata to get type for count update (v4.0.0: separate storage)
// Load metadata to get type for count update (separate storage)
try {
const metadata = await this.getNounMetadata(id)
if (metadata) {
@ -490,13 +489,13 @@ export class FileSystemStorage extends BaseStorage {
/**
* Save an edge to storage
* CRITICAL FIX (v4.10.3): Added atomic write pattern to prevent file corruption during concurrent imports
* CRITICAL FIX: Added atomic write pattern to prevent file corruption during concurrent imports
*/
protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized()
// Convert connections Map to a serializable format
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields in verb vector file
// ARCHITECTURAL FIX: Include core relational fields in verb vector file
// These fields are essential for 90% of operations - no metadata lookup needed
const serializableEdge = {
id: edge.id,
@ -505,7 +504,7 @@ export class FileSystemStorage extends BaseStorage {
Array.from(set as Set<string>)
),
// CORE RELATIONAL DATA (v3.50.1+)
// CORE RELATIONAL DATA
verb: edge.verb,
sourceId: edge.sourceId,
targetId: edge.targetId,
@ -518,7 +517,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${filePath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try {
// ATOMIC WRITE SEQUENCE (v4.10.3):
// ATOMIC WRITE SEQUENCE:
// 1. Write to temp file
await this.ensureDirectoryExists(path.dirname(tempPath))
await fs.promises.writeFile(tempPath, JSON.stringify(serializableEdge, null, 2))
@ -535,7 +534,7 @@ export class FileSystemStorage extends BaseStorage {
throw error
}
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet
}
@ -556,7 +555,7 @@ export class FileSystemStorage extends BaseStorage {
connections.set(Number(level), new Set(nodeIds as string[]))
}
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
// Return HNSWVerb with core relational fields (NO metadata field)
return {
id: parsedEdge.id,
vector: parsedEdge.vector,
@ -567,7 +566,7 @@ export class FileSystemStorage extends BaseStorage {
sourceId: parsedEdge.sourceId,
targetId: parsedEdge.targetId
// ✅ NO metadata field in v4.0.0
// ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata()
}
} catch (error: any) {
@ -580,7 +579,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Get all edges from storage
* CRITICAL FIX (v3.43.2): Now scans sharded subdirectories (depth=1)
* CRITICAL FIX: Now scans sharded subdirectories (depth=1)
* Previously only scanned flat directory, causing rebuild to find 0 relationships
*/
protected async getAllEdges(): Promise<Edge[]> {
@ -608,7 +607,7 @@ export class FileSystemStorage extends BaseStorage {
connections.set(Number(level), new Set(nodeIds as string[]))
}
// v4.0.0: Include core relational fields (NO metadata field)
// Include core relational fields (NO metadata field)
allEdges.push({
id: parsedEdge.id,
vector: parsedEdge.vector,
@ -619,7 +618,7 @@ export class FileSystemStorage extends BaseStorage {
sourceId: parsedEdge.sourceId,
targetId: parsedEdge.targetId
// ✅ NO metadata field in v4.0.0
// ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata()
})
}
@ -696,8 +695,8 @@ export class FileSystemStorage extends BaseStorage {
/**
* Primitive operation: Write object to path
* All metadata operations use this internally via base class routing
* v4.0.0: Supports gzip compression for 60-80% disk savings
* CRITICAL FIX (v4.10.3): Added atomic write pattern to prevent file corruption during concurrent imports
* Supports gzip compression for 60-80% disk savings
* CRITICAL FIX: Added atomic write pattern to prevent file corruption during concurrent imports
*/
protected async writeObjectToPath(pathStr: string, data: any): Promise<void> {
await this.ensureInitialized()
@ -711,7 +710,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${compressedPath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try {
// ATOMIC WRITE SEQUENCE (v4.10.3):
// ATOMIC WRITE SEQUENCE:
// 1. Compress and write to temp file
const jsonString = JSON.stringify(data, null, 2)
const compressed = await new Promise<Buffer>((resolve, reject) => {
@ -748,7 +747,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${fullPath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try {
// ATOMIC WRITE SEQUENCE (v4.10.3):
// ATOMIC WRITE SEQUENCE:
// 1. Write to temp file
await fs.promises.writeFile(tempPath, JSON.stringify(data, null, 2))
@ -770,7 +769,7 @@ export class FileSystemStorage extends BaseStorage {
* Primitive operation: Read object from path
* All metadata operations use this internally via base class routing
* Enhanced error handling for corrupted metadata files (Bug #3 mitigation)
* v4.0.0: Supports reading both compressed (.gz) and uncompressed files for backward compatibility
* Supports reading both compressed (.gz) and uncompressed files for backward compatibility
*/
protected async readObjectFromPath(pathStr: string): Promise<any | null> {
await this.ensureInitialized()
@ -822,7 +821,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Primitive operation: Delete object from path
* All metadata operations use this internally via base class routing
* v4.0.0: Deletes both compressed and uncompressed versions (for cleanup)
* Deletes both compressed and uncompressed versions (for cleanup)
*/
protected async deleteObjectFromPath(pathStr: string): Promise<void> {
await this.ensureInitialized()
@ -863,7 +862,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Primitive operation: List objects under path prefix
* All metadata operations use this internally via base class routing
* v4.0.0: Handles both .json and .json.gz files, normalizes paths
* Handles both .json and .json.gz files, normalizes paths
*/
protected async listObjectsUnderPath(prefix: string): Promise<string[]> {
await this.ensureInitialized()
@ -877,7 +876,7 @@ export class FileSystemStorage extends BaseStorage {
for (const entry of entries) {
if (entry.isFile()) {
// v5.3.5: Handle multiple compression formats for broad compatibility
// Handle multiple compression formats for broad compatibility
// - .json.gz: Standard entity/metadata files (JSON compressed)
// - .gz: COW files (refs, blobs, commits - raw compressed)
// - .json: Uncompressed JSON files
@ -890,7 +889,7 @@ export class FileSystemStorage extends BaseStorage {
seen.add(normalizedPath)
}
} else if (entry.name.endsWith('.gz')) {
// v5.3.5 fix: COW files stored as .gz (not .json.gz)
// COW files stored as .gz (not .json.gz)
// Strip .gz extension and return path
const normalizedName = entry.name.slice(0, -3) // Remove .gz
const normalizedPath = path.join(prefix, normalizedName)
@ -966,7 +965,7 @@ export class FileSystemStorage extends BaseStorage {
* Get nouns with pagination support
* @param options Pagination options
*/
// v5.4.0: Removed getNounsWithPagination override - now uses BaseStorage's type-first implementation
// Removed getNounsWithPagination override - now uses BaseStorage's type-first implementation
/**
* Clear all data from storage
@ -1002,7 +1001,7 @@ export class FileSystemStorage extends BaseStorage {
}
}
// v5.10.4: Clear the entire branches/ directory (branch-based storage)
// Clear the entire branches/ directory (branch-based storage)
// Bug fix: Data is stored in branches/main/entities/, not just entities/
// The branch-based structure was introduced for COW support
const branchesDir = path.join(this.rootDir, 'branches')
@ -1016,7 +1015,7 @@ export class FileSystemStorage extends BaseStorage {
await removeDirectoryContents(this.indexDir)
}
// v5.6.1: Remove COW (copy-on-write) version control data
// Remove COW (copy-on-write) version control data
// This directory stores all git-like versioning data (commits, trees, blobs, refs)
// Must be deleted to fully clear all data including version history
const cowDir = path.join(this.rootDir, '_cow')
@ -1024,7 +1023,7 @@ export class FileSystemStorage extends BaseStorage {
// Delete the entire _cow/ directory (not just contents)
await fs.promises.rm(cowDir, { recursive: true, force: true })
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -1035,12 +1034,12 @@ export class FileSystemStorage extends BaseStorage {
this.statisticsCache = null
this.statisticsModified = false
// v5.6.1: Reset entity counters (inherited from BaseStorageAdapter)
// Reset entity counters (inherited from BaseStorageAdapter)
// These in-memory counters must be reset to 0 after clearing all data
;(this as any).totalNounCount = 0
;(this as any).totalVerbCount = 0
// v7.3.1: Clear write-through cache (inherited from BaseStorage)
// Clear write-through cache (inherited from BaseStorage)
// Without this, readWithInheritance() would return stale cached data
// after clear(), causing "ghost" entities to appear
this.clearWriteCache()
@ -1074,12 +1073,12 @@ export class FileSystemStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker file exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
@ -1152,7 +1151,7 @@ export class FileSystemStorage extends BaseStorage {
totalSize = nounsDirSize + verbsDirSize + metadataDirSize + indexDirSize
// CRITICAL FIX (v3.43.2): Use persisted counts instead of directory reads
// CRITICAL FIX: Use persisted counts instead of directory reads
// This is O(1) instead of O(n), and handles sharded structure correctly
const nounsCount = this.totalNounCount
const verbsCount = this.totalVerbCount
@ -1210,8 +1209,8 @@ export class FileSystemStorage extends BaseStorage {
}
}
// v5.4.0: Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed 2 pagination methods (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's implementation
// Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
// Removed 2 pagination methods (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's implementation
/**
* Acquire a file-based lock for coordinating operations across multiple processes
@ -1503,14 +1502,14 @@ export class FileSystemStorage extends BaseStorage {
this.totalVerbCount = validVerbFiles.length
// Sample some files to get type distribution (don't read all)
// v4.0.0: Load metadata separately for type information
// Load metadata separately for type information
const sampleSize = Math.min(100, validNounFiles.length)
for (let i = 0; i < sampleSize; i++) {
try {
const file = validNounFiles[i]
const id = file.replace('.json', '')
// v4.0.0: Load metadata from separate storage for type info
// Load metadata from separate storage for type info
const metadata = await this.getNounMetadata(id)
if (metadata) {
const type = metadata.noun || 'default'
@ -2148,7 +2147,7 @@ export class FileSystemStorage extends BaseStorage {
const edge = JSON.parse(data)
const metadata = await this.getVerbMetadata(id)
// v4.8.1: Don't skip verbs without metadata - metadata is optional
// Don't skip verbs without metadata - metadata is optional
// FIX: This was the root cause of the VFS bug (11 versions)
// Verbs can exist without metadata files (e.g., from imports/migrations)
@ -2162,7 +2161,7 @@ export class FileSystemStorage extends BaseStorage {
connections = connectionsMap
}
// v4.8.0: Extract standard fields from metadata to top-level
// Extract standard fields from metadata to top-level
const metadataObj = (metadata || {}) as VerbMetadata
const { createdAt, updatedAt, confidence, weight, service, data: dataField, createdBy, ...customMetadata } = metadataObj
@ -2309,12 +2308,12 @@ export class FileSystemStorage extends BaseStorage {
}
// =============================================
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
// =============================================
/**
* Get vector for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
@ -2324,7 +2323,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Save HNSW graph data for a noun
*
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths)
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Preserves mutex locking to prevent read-modify-write races
*/
public async saveHNSWData(nounId: string, hnswData: {
@ -2333,7 +2332,7 @@ export class FileSystemStorage extends BaseStorage {
}): Promise<void> {
const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races
// CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3])
@ -2353,7 +2352,7 @@ export class FileSystemStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise)
try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths)
// Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId)
@ -2375,7 +2374,7 @@ export class FileSystemStorage extends BaseStorage {
connections: connectionsMap
}
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write)
// Use BaseStorage's saveNoun (type-first paths, atomic write)
await this.saveNoun(updatedNoun)
} finally {
// Release lock (ALWAYS runs, even if error thrown)
@ -2386,7 +2385,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getHNSWData(nounId: string): Promise<{
level: number
@ -2415,7 +2414,7 @@ export class FileSystemStorage extends BaseStorage {
/**
* Save HNSW system data (entry point, max level)
*
* CRITICAL FIX (v4.10.1): Mutex lock + atomic write to prevent race conditions
* CRITICAL FIX: Mutex lock + atomic write to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null
@ -2425,7 +2424,7 @@ export class FileSystemStorage extends BaseStorage {
const lockKey = 'hnsw/system'
// CRITICAL FIX (v4.10.1): Mutex lock to serialize system updates
// CRITICAL FIX: Mutex lock to serialize system updates
// System data (entry point, max level) updated frequently during HNSW construction
// Without mutex, concurrent updates can lose data (same as entity-level problem)

135
src/storage/adapters/gcsStorage.ts
View file

@ -65,7 +65,7 @@ const MAX_GCS_PAGE_SIZE = 5000
* 3. Service Account Credentials Object (if credentials provided)
* 4. HMAC Keys (if accessKeyId/secretAccessKey provided)
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -110,7 +110,7 @@ export class GcsStorage extends BaseStorage {
// Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance
// Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Multi-level cache manager for efficient data access
@ -120,7 +120,7 @@ export class GcsStorage extends BaseStorage {
// Module logger
private logger = createModuleLogger('GcsStorage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races
// HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>()
// Configuration options
@ -166,7 +166,7 @@ export class GcsStorage extends BaseStorage {
secretAccessKey?: string
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Cloud Run, Lambda),
* strict locally. Zero-config optimization.
@ -175,19 +175,18 @@ export class GcsStorage extends BaseStorage {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: InitMode
/**
* @deprecated Use `initMode: 'progressive'` instead.
* Will be removed in v8.0.0.
* Will be removed in a future version.
*/
skipInitialScan?: boolean
/**
* @deprecated Use `initMode: 'progressive'` instead.
* Will be removed in v8.0.0.
* Will be removed in a future version.
*/
skipCountsFile?: boolean
@ -206,7 +205,7 @@ export class GcsStorage extends BaseStorage {
this.accessKeyId = options.accessKeyId
this.secretAccessKey = options.secretAccessKey
// v7.3.0: Handle initMode and deprecated skip* flags
// Handle initMode and deprecated skip* flags
if (options.initMode) {
this.initMode = options.initMode
}
@ -215,14 +214,14 @@ export class GcsStorage extends BaseStorage {
if (options.skipInitialScan) {
console.warn(
'[GcsStorage] DEPRECATION WARNING: skipInitialScan is deprecated. ' +
'Use initMode: "progressive" instead. Will be removed in v8.0.0.'
'Use initMode: "progressive" instead. Will be removed in a future version.'
)
this.skipInitialScan = true
}
if (options.skipCountsFile) {
console.warn(
'[GcsStorage] DEPRECATION WARNING: skipCountsFile is deprecated. ' +
'Use initMode: "progressive" instead. Will be removed in v8.0.0.'
'Use initMode: "progressive" instead. Will be removed in a future version.'
)
this.skipCountsFile = true
}
@ -240,13 +239,13 @@ export class GcsStorage extends BaseStorage {
this.nounCacheManager = new CacheManager<HNSWNode>(options.cacheConfig)
this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig)
// v6.2.7: Write buffering always enabled - no env var check needed
// Write buffering always enabled - no env var check needed
}
/**
* Initialize the storage adapter
*
* v7.3.0: Supports progressive initialization for fast cold starts
* Supports progressive initialization for fast cold starts
*
* | Mode | Init Time | When |
* |------|-----------|------|
@ -337,14 +336,14 @@ export class GcsStorage extends BaseStorage {
}
)
// CRITICAL FIX (v3.37.7): Clear any stale cache entries from previous runs
// CRITICAL FIX: Clear any stale cache entries from previous runs
// This prevents cache poisoning from causing silent failures on container restart
prodLog.info('🧹 Clearing cache from previous run to prevent cache poisoning')
this.nounCacheManager.clear()
this.verbCacheManager.clear()
prodLog.info('✅ Cache cleared - starting fresh')
// v7.3.0: Progressive vs Strict initialization
// Progressive vs Strict initialization
if (effectiveMode === 'progressive') {
// PROGRESSIVE MODE: Fast init, background validation
// Mark as initialized immediately - ready to accept operations
@ -353,7 +352,7 @@ export class GcsStorage extends BaseStorage {
prodLog.info(`✅ GCS progressive init complete: ${this.bucketName} (validation deferred)`)
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Schedule background tasks (non-blocking)
@ -373,7 +372,7 @@ export class GcsStorage extends BaseStorage {
await this.initializeCounts()
this.countsLoaded = true
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Mark background tasks as complete (nothing to do in background)
@ -386,7 +385,7 @@ export class GcsStorage extends BaseStorage {
}
// =============================================
// Progressive Initialization (v7.3.0+)
// Progressive Initialization
// =============================================
/**
@ -400,7 +399,6 @@ export class GcsStorage extends BaseStorage {
*
* @protected
* @override
* @since v7.3.0
*/
protected async runBackgroundInit(): Promise<void> {
const startTime = Date.now()
@ -423,7 +421,6 @@ export class GcsStorage extends BaseStorage {
* Stores result in bucketValidated/bucketValidationError for lazy use.
*
* @private
* @since v7.3.0
*/
private async validateBucketInBackground(): Promise<void> {
try {
@ -451,7 +448,6 @@ export class GcsStorage extends BaseStorage {
* Uses the existing initializeCounts() logic but in background.
*
* @private
* @since v7.3.0
*/
private async loadCountsInBackground(): Promise<void> {
try {
@ -475,7 +471,6 @@ export class GcsStorage extends BaseStorage {
* @throws Error if bucket does not exist or is not accessible
* @protected
* @override
* @since v7.3.0
*/
protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do
@ -562,7 +557,7 @@ export class GcsStorage extends BaseStorage {
}
/**
* Override base class to enable smart batching for cloud storage (v3.32.3+)
* Override base class to enable smart batching for cloud storage
*
* GCS is cloud storage with network latency (~50ms per write).
* Smart batching reduces writes from 1000 ops → 100 batches.
@ -596,7 +591,7 @@ export class GcsStorage extends BaseStorage {
}
}
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage
// Removed checkVolumeMode() - write buffering always enabled for cloud storage
/**
* Flush noun buffer to GCS
@ -628,20 +623,20 @@ export class GcsStorage extends BaseStorage {
await Promise.all(writes)
}
// v5.4.0: Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node)
}
@ -657,10 +652,10 @@ export class GcsStorage extends BaseStorage {
/**
* Save a node directly to GCS (bypass buffer)
*
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
* Performs lazy bucket validation on first write in progressive mode.
*/
private async saveNodeDirect(node: HNSWNode): Promise<void> {
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
// Apply backpressure before starting operation
@ -695,14 +690,14 @@ export class GcsStorage extends BaseStorage {
resumable: false // For small objects, non-resumable is faster
})
// CRITICAL FIX (v3.37.8): Only cache nodes with non-empty vectors
// CRITICAL FIX: Only cache nodes with non-empty vectors
// This prevents cache pollution from HNSW's lazy-loading nodes (vector: [])
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node)
}
// Note: Empty vectors are intentional during HNSW lazy mode - not logged
// Count tracking happens in baseStorage.saveNounMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveNounMetadata_internal
// This fixes the race condition where metadata didn't exist yet
this.logger.trace(`Node ${node.id} saved successfully`)
@ -721,7 +716,7 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.4.0: Removed getNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed getNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Get a node from storage
@ -732,7 +727,7 @@ export class GcsStorage extends BaseStorage {
// Check cache first
const cached: HNSWNode | null = await this.nounCacheManager.get(id)
// Validate cached object before returning (v3.37.8+)
// Validate cached object before returning
if (cached !== undefined && cached !== null) {
// Validate cached object has required fields (including non-empty vector!)
if (!cached.id || !cached.vector || !Array.isArray(cached.vector) || cached.vector.length === 0) {
@ -831,18 +826,18 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.4.0: Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation
// Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation
/**
* Write an object to a specific path in GCS
* Primitive operation required by base class
*
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
* Performs lazy bucket validation on first write in progressive mode.
* @protected
*/
protected async writeObjectToPath(path: string, data: any): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
try {
@ -892,7 +887,7 @@ export class GcsStorage extends BaseStorage {
}
/**
* Batch read multiple objects from GCS (v5.12.0 - Cloud Storage Optimization)
* Batch read multiple objects from GCS
*
* **Performance**: GCS-optimized parallel downloads
* - Uses Promise.all() for concurrent requests
@ -908,7 +903,6 @@ export class GcsStorage extends BaseStorage {
* @returns Map of path → data (only successful reads included)
*
* @public - Called by baseStorage.readBatchFromAdapter()
* @since v5.12.0
*/
public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized()
@ -960,12 +954,11 @@ export class GcsStorage extends BaseStorage {
}
/**
* Get GCS-specific batch configuration (v5.12.0)
* Get GCS-specific batch configuration
*
* GCS performs well with high concurrency due to HTTP/2 multiplexing
*
* @public - Overrides BaseStorage.getBatchConfig()
* @since v5.12.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -984,12 +977,12 @@ export class GcsStorage extends BaseStorage {
* Delete an object from a specific path in GCS
* Primitive operation required by base class
*
* v7.3.0: Performs lazy bucket validation on first delete in progressive mode.
* Performs lazy bucket validation on first delete in progressive mode.
* @protected
*/
protected async deleteObjectFromPath(path: string): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
try {
@ -1034,20 +1027,20 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.4.0: Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation
/**
* Save an edge to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.verbWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding verb ${edge.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
this.verbCacheManager.set(edge.id, edge)
await this.verbWriteBuffer.add(edge.id, edge)
@ -1061,10 +1054,10 @@ export class GcsStorage extends BaseStorage {
/**
* Save an edge directly to GCS (bypass buffer)
*
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
* Performs lazy bucket validation on first write in progressive mode.
*/
private async saveEdgeDirect(edge: Edge): Promise<void> {
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
const requestId = await this.applyBackpressure()
@ -1073,7 +1066,7 @@ export class GcsStorage extends BaseStorage {
this.logger.trace(`Saving edge ${edge.id}`)
// Convert connections Map to serializable format
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields in verb vector file
// ARCHITECTURAL FIX: Include core relational fields in verb vector file
// These fields are essential for 90% of operations - no metadata lookup needed
const serializableEdge = {
id: edge.id,
@ -1085,7 +1078,7 @@ export class GcsStorage extends BaseStorage {
])
),
// CORE RELATIONAL DATA (v3.50.1+)
// CORE RELATIONAL DATA
verb: edge.verb,
sourceId: edge.sourceId,
targetId: edge.targetId,
@ -1107,7 +1100,7 @@ export class GcsStorage extends BaseStorage {
// Update cache
this.verbCacheManager.set(edge.id, edge)
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet
this.logger.trace(`Edge ${edge.id} saved successfully`)
@ -1125,7 +1118,7 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.4.0: Removed getVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed getVerb_internal - now inherit from BaseStorage's type-first implementation
/**
* Get an edge from storage
@ -1161,7 +1154,7 @@ export class GcsStorage extends BaseStorage {
connections.set(Number(level), new Set(verbIds as string[]))
}
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
// Return HNSWVerb with core relational fields (NO metadata field)
const edge: Edge = {
id: data.id,
vector: data.vector,
@ -1172,7 +1165,7 @@ export class GcsStorage extends BaseStorage {
sourceId: data.sourceId,
targetId: data.targetId
// ✅ NO metadata field in v4.0.0
// ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata()
}
@ -1201,13 +1194,13 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.4.0: Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation
// Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed pagination overrides - use BaseStorage's type-first implementation
// Removed pagination overrides - use BaseStorage's type-first implementation
// - getNounsWithPagination, getNodesWithPagination, getVerbsWithPagination
// - getNouns, getVerbs (public wrappers)
// v5.4.0: Removed 4 query *_internal methods - now inherit from BaseStorage's type-first implementation
// Removed 4 query *_internal methods - now inherit from BaseStorage's type-first implementation
// (getNounsByNounType_internal, getVerbsBySource_internal, getVerbsByTarget_internal, getVerbsByType_internal)
/**
@ -1280,7 +1273,7 @@ export class GcsStorage extends BaseStorage {
}
}
// v5.11.0: Clear ALL data using correct paths
// Clear ALL data using correct paths
// Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks)
await deleteObjectsWithPrefix('branches/')
@ -1290,7 +1283,7 @@ export class GcsStorage extends BaseStorage {
// Delete system metadata
await deleteObjectsWithPrefix('_system/')
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -1352,12 +1345,12 @@ export class GcsStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker object exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
@ -1413,7 +1406,7 @@ export class GcsStorage extends BaseStorage {
}
} catch (error: any) {
if (error.code === 404) {
// CRITICAL FIX (v3.37.4): Statistics file doesn't exist yet (first restart)
// CRITICAL FIX: Statistics file doesn't exist yet (first restart)
// Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild
this.logger.trace('Statistics file not found - returning minimal stats with counts')
@ -1607,11 +1600,11 @@ export class GcsStorage extends BaseStorage {
}
}
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
/**
* Get a noun's vector for HNSW rebuild
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
@ -1621,7 +1614,7 @@ export class GcsStorage extends BaseStorage {
/**
* Save HNSW graph data for a noun
*
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths)
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Uses mutex locking to prevent read-modify-write races
*/
public async saveHNSWData(nounId: string, hnswData: {
@ -1630,7 +1623,7 @@ export class GcsStorage extends BaseStorage {
}): Promise<void> {
const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races
// CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3])
@ -1650,7 +1643,7 @@ export class GcsStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise)
try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths)
// Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId)
@ -1672,7 +1665,7 @@ export class GcsStorage extends BaseStorage {
connections: connectionsMap
}
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
// Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
await this.saveNoun(updatedNoun)
} finally {
// Release lock (ALWAYS runs, even if error thrown)
@ -1683,7 +1676,7 @@ export class GcsStorage extends BaseStorage {
/**
* Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getHNSWData(nounId: string): Promise<{
level: number
@ -1713,7 +1706,7 @@ export class GcsStorage extends BaseStorage {
* Save HNSW system data (entry point, max level)
* Storage path: system/hnsw-system.json
*
* CRITICAL FIX (v4.10.1): Optimistic locking with generation numbers to prevent race conditions
* CRITICAL FIX: Optimistic locking with generation numbers to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null
@ -1807,7 +1800,7 @@ export class GcsStorage extends BaseStorage {
}
// ============================================================================
// GCS Lifecycle Management & Autoclass (v4.0.0)
// GCS Lifecycle Management & Autoclass
// Cost optimization through automatic tier transitions and Autoclass
// ============================================================================

10
src/storage/adapters/historicalStorageAdapter.ts
View file

@ -25,7 +25,7 @@
* - Lazy loading: only loads entities when accessed
* - No eager-loading of entire commit state
*
* v5.4.0: Production-ready, billion-scale historical queries
* Production-ready, billion-scale historical queries
*/
import { BaseStorage } from '../baseStorage.js'
@ -144,7 +144,7 @@ export class HistoricalStorageAdapter extends BaseStorage {
*/
public async init(): Promise<void> {
// Get COW components from underlying storage
// v6.2.4: Fixed property names - use public properties without underscore prefix
// Fixed property names - use public properties without underscore prefix
this.commitLog = this.underlyingStorage.commitLog
this.blobStorage = this.underlyingStorage.blobStorage
@ -161,7 +161,7 @@ export class HistoricalStorageAdapter extends BaseStorage {
throw new Error(`Commit not found: ${this.commitId}`)
}
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
}
@ -310,12 +310,12 @@ export class HistoricalStorageAdapter extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: No-op for HistoricalStorageAdapter (read-only, doesn't manage COW)
* No-op for HistoricalStorageAdapter (read-only, doesn't manage COW)
* @returns Always false (read-only adapter doesn't manage COW state)
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/

49
src/storage/adapters/memoryStorage.ts
View file

@ -24,7 +24,7 @@ import { PaginatedResult } from '../../types/paginationTypes.js'
* Uses Maps to store data in memory
*/
export class MemoryStorage extends BaseStorage {
// v5.4.0: Removed redundant Maps (nouns, verbs) - objectStore handles all storage via type-first paths
// Removed redundant Maps (nouns, verbs) - objectStore handles all storage via type-first paths
private statistics: StatisticsData | null = null
// Unified object store for primitive operations (replaces metadata, nounMetadata, verbMetadata)
@ -55,7 +55,6 @@ export class MemoryStorage extends BaseStorage {
* - Parallel processing maximizes throughput
*
* @returns Memory-optimized batch configuration
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -72,27 +71,27 @@ export class MemoryStorage extends BaseStorage {
/**
* Initialize the storage adapter
* v6.0.0: Calls super.init() to initialize GraphAdjacencyIndex and type statistics
* Calls super.init() to initialize GraphAdjacencyIndex and type statistics
*/
public async init(): Promise<void> {
await super.init()
}
// v5.4.0: Removed saveNoun_internal and getNoun_internal - using BaseStorage's type-first implementation
// Removed saveNoun_internal and getNoun_internal - using BaseStorage's type-first implementation
/**
* Get nouns with pagination and filtering
* v4.0.0: Returns HNSWNounWithMetadata[] (includes metadata field)
* Returns HNSWNounWithMetadata[] (includes metadata field)
* @param options Pagination and filtering options
* @returns Promise that resolves to a paginated result of nouns with metadata
*/
// v5.4.0: Removed public method overrides (getNouns, getNounsWithPagination, getVerbs) - using BaseStorage's type-first implementation
// Removed public method overrides (getNouns, getNounsWithPagination, getVerbs) - using BaseStorage's type-first implementation
// v5.4.0: Removed getNounsByNounType_internal and deleteNoun_internal - using BaseStorage's type-first implementation
// Removed getNounsByNounType_internal and deleteNoun_internal - using BaseStorage's type-first implementation
// v5.4.0: Removed saveVerb_internal and getVerb_internal - using BaseStorage's type-first implementation
// Removed saveVerb_internal and getVerb_internal - using BaseStorage's type-first implementation
// v5.4.0: Removed verb *_internal method overrides - using BaseStorage's type-first implementation
// Removed verb *_internal method overrides - using BaseStorage's type-first implementation
/**
* Primitive operation: Write object to path
@ -159,8 +158,8 @@ export class MemoryStorage extends BaseStorage {
/**
* Clear all data from storage
* v5.4.0: Clears objectStore (type-first paths)
* v7.3.1: Also clears writeCache to prevent stale data after clear
* Clears objectStore (type-first paths)
* Also clears writeCache to prevent stale data after clear
*/
public async clear(): Promise<void> {
this.objectStore.clear()
@ -174,7 +173,7 @@ export class MemoryStorage extends BaseStorage {
this.statisticsCache = null
this.statisticsModified = false
// v7.3.1: Clear write-through cache (inherited from BaseStorage)
// Clear write-through cache (inherited from BaseStorage)
// Without this, readWithInheritance() would return stale cached data
// after clear(), causing "ghost" entities to appear
this.clearWriteCache()
@ -182,7 +181,7 @@ export class MemoryStorage extends BaseStorage {
/**
* Get information about storage usage and capacity
* v5.4.0: Uses BaseStorage counts
* Uses BaseStorage counts
*/
public async getStorageStatus(): Promise<{
type: string
@ -204,12 +203,12 @@ export class MemoryStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: No-op for MemoryStorage (doesn't persist)
* No-op for MemoryStorage (doesn't persist)
* @returns Always false (marker doesn't persist in memory)
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
@ -252,7 +251,7 @@ export class MemoryStorage extends BaseStorage {
*/
protected async getStatisticsData(): Promise<StatisticsData | null> {
if (!this.statistics) {
// CRITICAL FIX (v3.37.4): Statistics don't exist yet (first init)
// CRITICAL FIX: Statistics don't exist yet (first init)
// Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild
return {
@ -299,10 +298,10 @@ export class MemoryStorage extends BaseStorage {
}
/**
* Initialize counts from in-memory storage - O(1) operation (v4.0.0)
* Initialize counts from in-memory storage - O(1) operation
*/
protected async initializeCounts(): Promise<void> {
// v6.0.0: Scan objectStore paths (ID-first structure) to count entities
// Scan objectStore paths (ID-first structure) to count entities
this.entityCounts.clear()
this.verbCounts.clear()
@ -314,14 +313,14 @@ export class MemoryStorage extends BaseStorage {
// Count nouns (entities/nouns/{shard}/{id}/vectors.json)
const nounMatch = path.match(/^entities\/nouns\/[0-9a-f]{2}\/[^/]+\/vectors\.json$/)
if (nounMatch) {
// v6.0.0: Type is in metadata, not path - just count total
// Type is in metadata, not path - just count total
totalNouns++
}
// Count verbs (entities/verbs/{shard}/{id}/vectors.json)
const verbMatch = path.match(/^entities\/verbs\/[0-9a-f]{2}\/[^/]+\/vectors\.json$/)
if (verbMatch) {
// v6.0.0: Type is in metadata, not path - just count total
// Type is in metadata, not path - just count total
totalVerbs++
}
}
@ -339,26 +338,26 @@ export class MemoryStorage extends BaseStorage {
}
// =============================================
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
// =============================================
/**
* Get vector for a noun
* v5.4.0: Uses BaseStorage's type-first implementation
* Uses BaseStorage's type-first implementation
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
return noun ? [...noun.vector] : null
}
// CRITICAL FIX (v4.10.1): Mutex locks for HNSW concurrency control
// CRITICAL FIX: Mutex locks for HNSW concurrency control
// Even in-memory operations need serialization to prevent async race conditions
private hnswLocks = new Map<string, Promise<void>>()
/**
* Save HNSW graph data for a noun
*
* CRITICAL FIX (v4.10.1): Mutex locking to prevent race conditions during concurrent HNSW updates
* CRITICAL FIX: Mutex locking to prevent race conditions during concurrent HNSW updates
* Even in-memory operations can race due to async/await interleaving
* Prevents data corruption when multiple entities connect to same neighbor simultaneously
*/
@ -417,7 +416,7 @@ export class MemoryStorage extends BaseStorage {
/**
* Save HNSW system data (entry point, max level)
*
* CRITICAL FIX (v4.10.1): Mutex locking to prevent race conditions
* CRITICAL FIX: Mutex locking to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null

43
src/storage/adapters/opfsStorage.ts
View file

@ -57,7 +57,7 @@ const ROOT_DIR = 'opfs-vector-db'
* OPFS storage adapter for browser environments
* Uses the Origin Private File System API to store data persistently
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -97,7 +97,6 @@ export class OPFSStorage extends BaseStorage {
* - Sequential processing preferred for stability
*
* @returns OPFS-optimized batch configuration
* @since v4.11.0
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -172,7 +171,7 @@ export class OPFSStorage extends BaseStorage {
// Initialize counts from storage
await this.initializeCounts()
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
} catch (error) {
console.error('Failed to initialize OPFS storage:', error)
@ -232,7 +231,7 @@ export class OPFSStorage extends BaseStorage {
}
}
// v5.4.0: Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
// Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
/**
* Delete an edge from storage
@ -482,14 +481,14 @@ export class OPFSStorage extends BaseStorage {
// Remove all files in the index directory
await removeDirectoryContents(this.indexDir!)
// v5.6.1: Remove COW (copy-on-write) version control data
// Remove COW (copy-on-write) version control data
// This directory stores all git-like versioning data (commits, trees, blobs, refs)
// Must be deleted to fully clear all data including version history
try {
// Delete the entire _cow/ directory (not just contents)
await this.rootDir!.removeEntry('_cow', { recursive: true })
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -505,7 +504,7 @@ export class OPFSStorage extends BaseStorage {
this.statisticsCache = null
this.statisticsModified = false
// v5.6.1: Reset entity counters (inherited from BaseStorageAdapter)
// Reset entity counters (inherited from BaseStorageAdapter)
// These in-memory counters must be reset to 0 after clearing all data
;(this as any).totalNounCount = 0
;(this as any).totalVerbCount = 0
@ -517,16 +516,16 @@ export class OPFSStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker file exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
// Quota monitoring configuration (v4.0.0)
// Quota monitoring configuration
private quotaWarningThreshold = 0.8 // Warn at 80% usage
private quotaCriticalThreshold = 0.95 // Critical at 95% usage
private lastQuotaCheck: number = 0
@ -675,7 +674,7 @@ export class OPFSStorage extends BaseStorage {
}
/**
* Get detailed quota status with warnings (v4.0.0)
* Get detailed quota status with warnings
* Monitors storage usage and warns when approaching quota limits
*
* @returns Promise that resolves to quota status with warning levels
@ -769,7 +768,7 @@ export class OPFSStorage extends BaseStorage {
}
/**
* Monitor quota during operations (v4.0.0)
* Monitor quota during operations
* Automatically checks quota at regular intervals and warns if approaching limits
* Call this before write operations to ensure quota is available
*
@ -1185,7 +1184,7 @@ export class OPFSStorage extends BaseStorage {
}
}
} catch (error) {
// CRITICAL FIX (v3.37.4): No statistics files exist (first init)
// CRITICAL FIX: No statistics files exist (first init)
// Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild
return {
@ -1228,7 +1227,7 @@ export class OPFSStorage extends BaseStorage {
* @param options Pagination and filter options
* @returns Promise that resolves to a paginated result of nouns
*/
// v5.4.0: Removed pagination overrides (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's type-first implementation
// Removed pagination overrides (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's type-first implementation
/**
* Initialize counts from OPFS storage
@ -1313,28 +1312,28 @@ export class OPFSStorage extends BaseStorage {
}
}
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
/**
* Get a noun's vector for HNSW rebuild
*/
/**
* Get vector for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
return noun ? noun.vector : null
}
// CRITICAL FIX (v4.10.1): Mutex locks for HNSW concurrency control
// CRITICAL FIX: Mutex locks for HNSW concurrency control
// Browser environments are single-threaded but async operations can still interleave
private hnswLocks = new Map<string, Promise<void>>()
/**
* Save HNSW graph data for a noun
*
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths)
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Preserves mutex locking to prevent read-modify-write races
*/
public async saveHNSWData(nounId: string, hnswData: {
@ -1354,7 +1353,7 @@ export class OPFSStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise)
try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths)
// Use BaseStorage's getNoun (type-first paths)
const existingNoun = await this.getNoun(nounId)
if (!existingNoun) {
@ -1374,7 +1373,7 @@ export class OPFSStorage extends BaseStorage {
connections: connectionsMap
}
// v5.4.0: Use BaseStorage's saveNoun (type-first paths)
// Use BaseStorage's saveNoun (type-first paths)
await this.saveNoun(updatedNoun)
} finally {
// Release lock
@ -1385,7 +1384,7 @@ export class OPFSStorage extends BaseStorage {
/**
* Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getHNSWData(nounId: string): Promise<{
level: number
@ -1415,7 +1414,7 @@ export class OPFSStorage extends BaseStorage {
* Save HNSW system data (entry point, max level)
* Storage path: index/hnsw-system.json
*
* CRITICAL FIX (v4.10.1): Mutex locking to prevent race conditions
* CRITICAL FIX: Mutex locking to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null

4
src/storage/adapters/optimizedS3Search.ts
View file

@ -107,7 +107,7 @@ export class OptimizedS3Search {
}
// Determine if there are more items
const hasMore = listResult.hasMore || nouns.length > limit // v5.7.11: Fixed >= to > (was causing infinite loop)
const hasMore = listResult.hasMore || nouns.length > limit // Fixed >= to > (was causing infinite loop)
// Set next cursor
let nextCursor: string | undefined
@ -190,7 +190,7 @@ export class OptimizedS3Search {
}
// Determine if there are more items
const hasMore = listResult.hasMore || verbs.length > limit // v5.7.11: Fixed >= to > (was causing infinite loop)
const hasMore = listResult.hasMore || verbs.length > limit // Fixed >= to > (was causing infinite loop)
// Set next cursor
let nextCursor: string | undefined

49
src/storage/adapters/r2Storage.ts
View file

@ -58,7 +58,7 @@ const MAX_R2_PAGE_SIZE = 1000
* Dedicated Cloudflare R2 storage adapter
* Optimized for R2's unique characteristics and global edge network
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed getNounsWithPagination override
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -112,7 +112,7 @@ export class R2Storage extends BaseStorage {
// Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance
// Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Multi-level cache manager for efficient data access
@ -122,7 +122,7 @@ export class R2Storage extends BaseStorage {
// Module logger
private logger = createModuleLogger('R2Storage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races
// HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>()
/**
@ -168,7 +168,7 @@ export class R2Storage extends BaseStorage {
})
this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig)
// v6.2.7: Write buffering always enabled - no env var check needed
// Write buffering always enabled - no env var check needed
}
/**
@ -183,7 +183,7 @@ export class R2Storage extends BaseStorage {
* Zero egress fees enable aggressive caching and parallel downloads
*
* @returns R2-optimized batch configuration
* @since v5.12.0 - Updated for native batch API
* Updated for native batch API
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -208,7 +208,6 @@ export class R2Storage extends BaseStorage {
*
* @param paths - Array of R2 object keys to read
* @returns Map of path -> parsed JSON data (only successful reads)
* @since v5.12.0
*/
public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized()
@ -336,7 +335,7 @@ export class R2Storage extends BaseStorage {
this.nounCacheManager.clear()
this.verbCacheManager.clear()
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
} catch (error) {
this.logger.error('Failed to initialize R2 storage:', error)
@ -409,7 +408,7 @@ export class R2Storage extends BaseStorage {
}
}
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage
// Removed checkVolumeMode() - write buffering always enabled for cloud storage
/**
* Flush noun buffer to R2
@ -444,16 +443,16 @@ export class R2Storage extends BaseStorage {
/**
* Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node)
}
@ -728,14 +727,14 @@ export class R2Storage extends BaseStorage {
/**
* Save an edge to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.verbWriteBuffer) {
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
this.verbCacheManager.set(edge.id, edge)
await this.verbWriteBuffer.add(edge.id, edge)
@ -750,7 +749,7 @@ export class R2Storage extends BaseStorage {
const requestId = await this.applyBackpressure()
try {
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields in verb vector file
// ARCHITECTURAL FIX: Include core relational fields in verb vector file
// These fields are essential for 90% of operations - no metadata lookup needed
const serializableEdge = {
id: edge.id,
@ -762,7 +761,7 @@ export class R2Storage extends BaseStorage {
])
),
// CORE RELATIONAL DATA (v3.50.1+)
// CORE RELATIONAL DATA
verb: edge.verb,
sourceId: edge.sourceId,
targetId: edge.targetId,
@ -785,7 +784,7 @@ export class R2Storage extends BaseStorage {
this.verbCacheManager.set(edge.id, edge)
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2)
// Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet
this.releaseBackpressure(true, requestId)
@ -831,7 +830,7 @@ export class R2Storage extends BaseStorage {
connections.set(Number(level), new Set(verbIds as string[]))
}
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
// Return HNSWVerb with core relational fields (NO metadata field)
const edge: Edge = {
id: data.id,
vector: data.vector,
@ -842,7 +841,7 @@ export class R2Storage extends BaseStorage {
sourceId: data.sourceId,
targetId: data.targetId
// ✅ NO metadata field in v4.0.0
// ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata()
}
@ -1081,7 +1080,7 @@ export class R2Storage extends BaseStorage {
prodLog.info('🧹 R2: Clearing all data from bucket...')
// v5.11.0: Clear ALL data using correct paths
// Clear ALL data using correct paths
// Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks)
const branchObjects = await this.listObjectsUnderPath('branches/')
for (const key of branchObjects) {
@ -1100,7 +1099,7 @@ export class R2Storage extends BaseStorage {
await this.deleteObjectFromPath(key)
}
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -1143,17 +1142,17 @@ export class R2Storage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker object exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
// v5.4.0: Removed getNounsWithPagination override - use BaseStorage's type-first implementation
// Removed getNounsWithPagination override - use BaseStorage's type-first implementation
// v5.4.0: Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
// Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
}

98
src/storage/adapters/s3CompatibleStorage.ts
View file

@ -85,7 +85,7 @@ type S3Command = any
* - endpoint: GCS endpoint (e.g., 'https://storage.googleapis.com')
* - bucketName: GCS bucket name
*
* v5.4.0: Type-aware storage now built into BaseStorage
* Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -152,7 +152,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance
// Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Operation executors for timeout and retry handling
@ -165,7 +165,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Module logger
private logger = createModuleLogger('S3Storage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races
// HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>()
/**
@ -210,7 +210,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Lambda, Cloud Run),
* strict locally. Zero-config optimization.
@ -219,7 +219,6 @@ export class S3CompatibleStorage extends BaseStorage {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: InitMode
@ -236,7 +235,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.serviceType = options.serviceType || 's3'
this.readOnly = options.readOnly || false
// v7.3.0: Handle initMode
// Handle initMode
if (options.initMode) {
this.initMode = options.initMode
}
@ -270,7 +269,7 @@ export class S3CompatibleStorage extends BaseStorage {
* S3 supports ~5000 operations/second with burst capacity up to 10,000
*
* @returns S3-optimized batch configuration
* @since v5.12.0 - Updated for native batch API
* Updated for native batch API
*/
public getBatchConfig(): StorageBatchConfig {
return {
@ -295,7 +294,6 @@ export class S3CompatibleStorage extends BaseStorage {
*
* @param paths - Array of S3 object keys to read
* @returns Map of path -> parsed JSON data (only successful reads)
* @since v5.12.0
*/
public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized()
@ -358,7 +356,7 @@ export class S3CompatibleStorage extends BaseStorage {
/**
* Initialize the storage adapter
*
* v7.3.0: Supports progressive initialization for fast cold starts
* Supports progressive initialization for fast cold starts
*
* | Mode | Init Time | When |
* |------|-----------|------|
@ -514,7 +512,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Initialize request coalescer
this.initializeCoalescer()
// CRITICAL FIX (v3.37.7): Clear any stale cache entries from previous runs
// CRITICAL FIX: Clear any stale cache entries from previous runs
// This prevents cache poisoning from causing silent failures on container restart
const nodeCacheSize = this.nodeCache?.size || 0
if (nodeCacheSize > 0) {
@ -524,7 +522,7 @@ export class S3CompatibleStorage extends BaseStorage {
prodLog.info('🧹 Node cache is empty - starting fresh')
}
// v7.3.0: Progressive vs Strict initialization
// Progressive vs Strict initialization
if (effectiveMode === 'progressive') {
// PROGRESSIVE MODE: Fast init, background validation
// Mark as initialized immediately - ready to accept operations
@ -533,7 +531,7 @@ export class S3CompatibleStorage extends BaseStorage {
prodLog.info(`✅ S3 progressive init complete: ${this.bucketName} (validation deferred)`)
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Schedule background tasks (non-blocking)
@ -556,7 +554,7 @@ export class S3CompatibleStorage extends BaseStorage {
await this.initializeCounts()
this.countsLoaded = true
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
// Initialize GraphAdjacencyIndex and type statistics
await super.init()
// Mark background tasks as complete (nothing to do in background)
@ -573,7 +571,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
// =============================================
// Progressive Initialization (v7.3.0+)
// Progressive Initialization
// =============================================
/**
@ -588,7 +586,6 @@ export class S3CompatibleStorage extends BaseStorage {
*
* @protected
* @override
* @since v7.3.0
*/
protected async runBackgroundInit(): Promise<void> {
const startTime = Date.now()
@ -614,7 +611,6 @@ export class S3CompatibleStorage extends BaseStorage {
* Stores result in bucketValidated/bucketValidationError for lazy use.
*
* @private
* @since v7.3.0
*/
private async validateBucketInBackground(): Promise<void> {
try {
@ -638,7 +634,6 @@ export class S3CompatibleStorage extends BaseStorage {
* Load counts from storage in background.
*
* @private
* @since v7.3.0
*/
private async loadCountsInBackground(): Promise<void> {
try {
@ -662,7 +657,6 @@ export class S3CompatibleStorage extends BaseStorage {
* @throws Error if bucket does not exist or is not accessible
* @protected
* @override
* @since v7.3.0
*/
protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do
@ -918,7 +912,7 @@ export class S3CompatibleStorage extends BaseStorage {
)
}
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage
// Removed checkVolumeMode() - write buffering always enabled for cloud storage
/**
* Bulk write nouns to S3
@ -1179,20 +1173,20 @@ export class S3CompatibleStorage extends BaseStorage {
return this.socketManager.getBatchSize()
}
// v5.4.0: Removed 10 *_internal method overrides (lines 984-2069) - now inherit from BaseStorage's type-first implementation
// Removed 10 *_internal method overrides (lines 984-2069) - now inherit from BaseStorage's type-first implementation
/**
* Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance
* Always uses write buffer for consistent performance
*/
protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching
// Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency
// Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node)
}
@ -1242,7 +1236,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.logger.debug(`Node ${node.id} saved successfully`)
// Log the change for efficient synchronization (v4.0.0: no metadata on node)
// Log the change for efficient synchronization (no metadata on node)
await this.appendToChangeLog({
timestamp: Date.now(),
operation: 'add', // Could be 'update' if we track existing nodes
@ -1250,7 +1244,7 @@ export class S3CompatibleStorage extends BaseStorage {
entityId: node.id,
data: {
vector: node.vector
// ✅ NO metadata field in v4.0.0 - stored separately
// ✅ NO metadata field - stored separately
}
})
@ -1296,7 +1290,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
}
// v5.4.0: Removed getNoun_internal override - uses BaseStorage type-first implementation
// Removed getNoun_internal override - uses BaseStorage type-first implementation
/**
* Get a node from storage
@ -1307,7 +1301,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Check cache first
const cached = this.nodeCache.get(id)
// Validate cached object before returning (v3.37.8+)
// Validate cached object before returning
if (cached !== undefined && cached !== null) {
// Validate cached object has required fields (including non-empty vector!)
if (!cached.id || !cached.vector || !Array.isArray(cached.vector) || cached.vector.length === 0) {
@ -1607,7 +1601,7 @@ export class S3CompatibleStorage extends BaseStorage {
return nodes
}
// v5.4.0: Removed 4 *_internal method overrides (getNounsByNounType_internal, deleteNoun_internal, saveVerb_internal, getVerb_internal)
// Removed 4 *_internal method overrides (getNounsByNounType_internal, deleteNoun_internal, saveVerb_internal, getVerb_internal)
// Now inherit from BaseStorage's type-first implementation
@ -1788,23 +1782,23 @@ export class S3CompatibleStorage extends BaseStorage {
return true // Return all edges since filtering requires metadata
}
// v5.4.0: Removed getVerbsWithPagination override - use BaseStorage's type-first implementation
// Removed getVerbsWithPagination override - use BaseStorage's type-first implementation
// v5.4.0: Removed 4 more *_internal method overrides (getVerbsBySource, getVerbsByTarget, getVerbsByType, deleteVerb)
// Removed 4 more *_internal method overrides (getVerbsBySource, getVerbsByTarget, getVerbsByType, deleteVerb)
// Total: 8 *_internal methods removed - all now inherit from BaseStorage's type-first implementation
/**
* Primitive operation: Write object to path
* All metadata operations use this internally via base class routing
*
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
* Performs lazy bucket validation on first write in progressive mode.
*/
protected async writeObjectToPath(path: string, data: any): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
// Apply backpressure before starting operation
@ -1897,11 +1891,11 @@ export class S3CompatibleStorage extends BaseStorage {
* Primitive operation: Delete object from path
* All metadata operations use this internally via base class routing
*
* v7.3.0: Performs lazy bucket validation on first delete in progressive mode.
* Performs lazy bucket validation on first delete in progressive mode.
*/
protected async deleteObjectFromPath(path: string): Promise<void> {
await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init
// Lazy bucket validation for progressive init
await this.ensureValidatedForWrite()
try {
@ -2315,7 +2309,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
}
// v5.11.0: Clear ALL data using correct paths
// Clear ALL data using correct paths
// Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks)
await deleteObjectsWithPrefix('branches/')
@ -2325,7 +2319,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Delete system metadata
await deleteObjectsWithPrefix('_system/')
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled)
// Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use
this.refManager = undefined
this.blobStorage = undefined
@ -2335,7 +2329,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.statisticsCache = null
this.statisticsModified = false
// v5.6.1: Reset entity counters (inherited from BaseStorageAdapter)
// Reset entity counters (inherited from BaseStorageAdapter)
// These in-memory counters must be reset to 0 after clearing all data
;(this as any).totalNounCount = 0
;(this as any).totalVerbCount = 0
@ -2457,12 +2451,12 @@ export class S3CompatibleStorage extends BaseStorage {
/**
* Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts
* Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker object exists, false otherwise
* @protected
*/
/**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods
* Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used
*/
@ -2851,7 +2845,7 @@ export class S3CompatibleStorage extends BaseStorage {
totalEdges: this.totalVerbCount
}
}
// CRITICAL FIX (v3.37.4): Statistics file doesn't exist yet (first restart)
// CRITICAL FIX: Statistics file doesn't exist yet (first restart)
// Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild
return {
@ -3415,7 +3409,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
}
// v5.4.0: Removed getNounsWithPagination override - use BaseStorage's type-first implementation
// Removed getNounsWithPagination override - use BaseStorage's type-first implementation
/**
* Estimate total noun count by listing objects across all shards
@ -3549,7 +3543,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
/**
* Override base class to enable smart batching for cloud storage (v3.32.3+)
* Override base class to enable smart batching for cloud storage
*
* S3 is cloud storage with network latency (~50ms per write).
* Smart batching reduces writes from 1000 ops → 100 batches.
@ -3560,11 +3554,11 @@ export class S3CompatibleStorage extends BaseStorage {
return true // S3 benefits from batching
}
// HNSW Index Persistence (v3.35.0+)
// HNSW Index Persistence
/**
* Get a noun's vector for HNSW rebuild
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id)
@ -3574,7 +3568,7 @@ export class S3CompatibleStorage extends BaseStorage {
/**
* Save HNSW graph data for a noun
*
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths)
* Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Uses mutex locking to prevent read-modify-write races
*/
public async saveHNSWData(nounId: string, hnswData: {
@ -3583,7 +3577,7 @@ export class S3CompatibleStorage extends BaseStorage {
}): Promise<void> {
const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races
// CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3])
@ -3603,7 +3597,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise)
try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths)
// Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId)
@ -3625,7 +3619,7 @@ export class S3CompatibleStorage extends BaseStorage {
connections: connectionsMap
}
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
// Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
await this.saveNoun(updatedNoun)
} finally {
// Release lock (ALWAYS runs, even if error thrown)
@ -3636,7 +3630,7 @@ export class S3CompatibleStorage extends BaseStorage {
/**
* Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths)
* Uses BaseStorage's getNoun (type-first paths)
*/
public async getHNSWData(nounId: string): Promise<{
level: number
@ -3666,7 +3660,7 @@ export class S3CompatibleStorage extends BaseStorage {
* Save HNSW system data (entry point, max level)
* Storage path: system/hnsw-system.json
*
* CRITICAL FIX (v4.10.1): Optimistic locking with ETags to prevent race conditions
* CRITICAL FIX: Optimistic locking with ETags to prevent race conditions
*/
public async saveHNSWSystem(systemData: {
entryPointId: string | null
@ -3781,7 +3775,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
/**
* Set S3 lifecycle policy for automatic tier transitions and deletions (v4.0.0)
* Set S3 lifecycle policy for automatic tier transitions and deletions
* Automates cost optimization by moving old data to cheaper storage classes
*
* S3 Storage Classes:
@ -3976,7 +3970,7 @@ export class S3CompatibleStorage extends BaseStorage {
}
/**
* Enable S3 Intelligent-Tiering for automatic cost optimization (v4.0.0)
* Enable S3 Intelligent-Tiering for automatic cost optimization
* Automatically moves objects between access tiers based on usage patterns
*
* Intelligent-Tiering automatically saves up to 95% on storage costs:

4
src/storage/backwardCompatibility.ts
View file

@ -1,6 +1,6 @@
/**
* DEPRECATED (v4.7.2): Backward compatibility stubs
* TODO: Remove in v4.7.3 after migrating s3CompatibleStorage
* DEPRECATED: Backward compatibility stubs
* TODO: Remove after migrating s3CompatibleStorage
*/
export class StorageCompatibilityLayer {

339
src/storage/baseStorage.ts
View file

File diff suppressed because it is too large Load diff

14
src/storage/cow/BlobStorage.ts
View file

@ -161,7 +161,7 @@ export class BlobStorage {
}
/**
* v5.7.5: Ensure compression is ready before write operations
* Ensure compression is ready before write operations
* Fixes race condition where write happens before async compression init completes
*/
private async ensureCompressionReady(): Promise<void> {
@ -206,7 +206,7 @@ export class BlobStorage {
return hash
}
// v5.7.5: Ensure compression is initialized before writing
// Ensure compression is initialized before writing
// Fixes race condition where write happens before async init completes
await this.ensureCompressionReady()
@ -223,7 +223,7 @@ export class BlobStorage {
}
// Create metadata
// v5.7.5: Store ACTUAL compression state, not intended
// Store ACTUAL compression state, not intended
// Prevents corruption if compression failed to initialize
const actualCompression = finalData === data ? 'none' : compression
const metadata: BlobMetadata = {
@ -277,7 +277,7 @@ export class BlobStorage {
* @returns Blob data
*/
async read(hash: string, options: BlobReadOptions = {}): Promise<Buffer> {
// v5.3.4 fix: Guard against NULL hash (sentinel value)
// Guard against NULL hash (sentinel value)
// NULL_HASH ('0000...0000') is used as a sentinel for "no parent" or "empty tree"
// It should NEVER be read as actual blob data
if (isNullHash(hash)) {
@ -309,7 +309,7 @@ export class BlobStorage {
metadataBuffer = await this.adapter.get(`${tryPrefix}-meta:${hash}`)
if (metadataBuffer) {
prefix = tryPrefix
// v5.10.1: Unwrap metadata before parsing (defense-in-depth)
// Unwrap metadata before parsing (defense-in-depth)
// Metadata should be JSON, but adapter might return wrapped format
const unwrappedMetadata = unwrapBinaryData(metadataBuffer)
metadata = JSON.parse(unwrappedMetadata.toString())
@ -338,8 +338,8 @@ export class BlobStorage {
finalData = await this.zstdDecompress(data)
}
// v5.10.1: Defense-in-depth unwrap (CRITICAL FIX for blob integrity regression)
// Even though COW adapter should unwrap (v5.7.5), verify it happened and re-unwrap if needed
// Defense-in-depth unwrap (CRITICAL FIX for blob integrity regression)
// Even though COW adapter should unwrap, verify it happened and re-unwrap if needed
// This prevents "Blob integrity check failed" errors if adapter returns wrapped data
// Uses shared binaryDataCodec utility (single source of truth for unwrap logic)
const unwrappedData = unwrapBinaryData(finalData)

2
src/storage/cow/CommitLog.ts
View file

@ -43,7 +43,7 @@ export interface CommitLogStats {
/**
* CommitLog: Efficient commit history traversal and querying
*
* Pure v5.0.0 implementation - modern, clean, fast
* Pure implementation - modern, clean, fast
*/
export class CommitLog {
private blobStorage: BlobStorage

2
src/storage/cow/CommitObject.ts
View file

@ -337,7 +337,7 @@ export class CommitObject {
let currentHash: string | null = startHash
let depth = 0
// v5.3.4 fix: Guard against NULL hash (sentinel for "no parent")
// Guard against NULL hash (sentinel for "no parent")
// The initial commit has parent = null or NULL_HASH ('0000...0000')
// We must stop walking when we reach it, not try to read it
while (currentHash && !isNullHash(currentHash)) {

2
src/storage/cow/RefManager.ts
View file

@ -55,7 +55,7 @@ export interface RefUpdateOptions {
/**
* RefManager: Manages branches, tags, and HEAD pointer
*
* Pure implementation for v5.0.0 - no backward compatibility
* Pure implementation - no backward compatibility
*/
export class RefManager {
private adapter: COWStorageAdapter

4
src/storage/cow/binaryDataCodec.ts
View file

@ -86,7 +86,7 @@ export function unwrapBinaryData(data: any): Buffer {
/**
* Wrap binary data for JSON storage
*
* ⚠️ WARNING: DO NOT USE THIS ON WRITE PATH! (v6.2.0)
* ⚠️ WARNING: DO NOT USE THIS ON WRITE PATH!
* ⚠️ Use key-based dispatch in baseStorage.ts COW adapter instead.
* ⚠️ This function exists for legacy/compatibility only.
*
@ -94,7 +94,7 @@ export function unwrapBinaryData(data: any): Buffer {
* This is FRAGILE because compressed binary can accidentally parse as valid JSON,
* causing blob integrity failures.
*
* v6.2.0 SOLUTION: baseStorage.ts COW adapter now uses key naming convention:
* SOLUTION: baseStorage.ts COW adapter now uses key naming convention:
* - Keys with '-meta:' or 'ref:' prefix → Always JSON
* - Keys with 'blob:', 'commit:', 'tree:' prefix → Always binary
* No guessing needed!

23
src/storage/storageFactory.ts
View file

@ -10,7 +10,7 @@ import { S3CompatibleStorage } from './adapters/s3CompatibleStorage.js'
import { R2Storage } from './adapters/r2Storage.js'
import { GcsStorage } from './adapters/gcsStorage.js'
import { AzureBlobStorage } from './adapters/azureBlobStorage.js'
// TypeAwareStorageAdapter removed in v5.4.0 - type-aware now built into BaseStorage
// TypeAwareStorageAdapter removed - type-aware now built into BaseStorage
// FileSystemStorage is dynamically imported to avoid issues in browser environments
import { isBrowser } from '../utils/environment.js'
import { OperationConfig } from '../utils/operationUtils.js'
@ -94,7 +94,7 @@ export interface StorageOptions {
sessionToken?: string
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Lambda),
* strict locally. Zero-config optimization.
@ -103,7 +103,6 @@ export interface StorageOptions {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: 'progressive' | 'strict' | 'auto'
}
@ -215,7 +214,7 @@ export interface StorageOptions {
skipCountsFile?: boolean
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Cloud Run),
* strict locally. Zero-config optimization.
@ -224,7 +223,6 @@ export interface StorageOptions {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: 'progressive' | 'strict' | 'auto'
}
@ -259,7 +257,7 @@ export interface StorageOptions {
sasToken?: string
/**
* Initialization mode for fast cold starts (v7.3.0+)
* Initialization mode for fast cold starts
*
* - `'auto'` (default): Progressive in cloud environments (Azure Functions),
* strict locally. Zero-config optimization.
@ -268,7 +266,6 @@ export interface StorageOptions {
* - `'strict'`: Traditional blocking init. Validates container and loads counts
* before init() returns.
*
* @since v7.3.0
*/
initMode?: 'progressive' | 'strict' | 'auto'
}
@ -388,7 +385,7 @@ export interface StorageOptions {
/**
* COW (Copy-on-Write) configuration for instant fork() capability
* v5.0.1: COW is now always enabled (automatic, zero-config)
* COW is now always enabled (automatic, zero-config)
*/
branch?: string // Current branch name (default: 'main')
enableCompression?: boolean // Enable zstd compression for COW blobs (default: true)
@ -410,14 +407,14 @@ function getFileSystemPath(options: StorageOptions): string {
}
/**
* Configure COW (Copy-on-Write) options on a storage adapter (v5.4.0)
* Configure COW (Copy-on-Write) options on a storage adapter
* TypeAware is now built-in to all adapters, no wrapper needed!
*
* @param storage - The storage adapter
* @param options - Storage options (for COW configuration)
*/
function configureCOW(storage: any, options?: StorageOptions): void {
// v5.0.1: COW will be initialized AFTER storage.init() in Brainy
// COW will be initialized AFTER storage.init() in Brainy
// Store COW options for later initialization
if (typeof storage.initializeCOW === 'function') {
storage._cowOptions = {
@ -672,10 +669,10 @@ export async function createStorage(
}
case 'type-aware':
// v5.0.0: TypeAware is now the default for ALL adapters
// TypeAware is now the default for ALL adapters
// Redirect to the underlying type instead
console.warn(
'⚠️ type-aware is deprecated in v5.0.0 - TypeAware is now always enabled.'
'⚠️ type-aware is deprecated - TypeAware is now always enabled.'
)
console.warn(
' Just use the underlying storage type (e.g., "filesystem", "s3", etc.)'
@ -866,7 +863,7 @@ export async function createStorage(
}
/**
* Export storage adapters (v5.4.0: TypeAware is now built-in, no separate export)
* Export storage adapters (TypeAware is now built-in, no separate export)
*/
export {
MemoryStorage,

Some files were not shown because too many files have changed in this diff Show more