fix: exclude __words__ keyword index from corruption detection and getStats()

The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:

1. validateConsistency() to falsely detect corruption on every startup,
   triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
   and report inflated totalEntries/totalIds stats

Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
This commit is contained in:
David Snelling 2026-01-27 15:38:21 -08:00
parent 32dbdcec61
commit 364360d447
128 changed files with 5637 additions and 5682 deletions

File diff suppressed because it is too large Load diff

View file

@ -1,23 +1,22 @@
# Batch Operations API v5.12.0 # Batch Operations API
> **Enterprise Production-Ready** | Zero N+1 Query Patterns | 90%+ Performance Improvement > **Enterprise Production-Ready** | Zero N+1 Query Patterns | 90%+ Performance Improvement
## Overview ## Overview
Brainy v5.12.0 introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage. Brainy introduces comprehensive batch operations at the storage layer, eliminating N+1 query patterns and dramatically improving performance for VFS operations, relationship queries, and entity retrieval on cloud storage.
### Problem Solved ### Problem Solved
**Before v5.12.0:** **Before optimization:**
- VFS `getTreeStructure()` on cloud storage: **12.7 seconds** for directory with 12 files - VFS `getTreeStructure()` on cloud storage: **12.7 seconds** for directory with 12 files
- N+1 query pattern: 1 directory query + N individual file queries (22 sequential calls × 580ms latency) - N+1 query pattern: 1 directory query + N individual file queries (22 sequential calls × 580ms latency)
**After v5.12.0:** **After optimization:**
- VFS `getTreeStructure()`: **<1 second** for 12 files (90%+ improvement) - VFS `getTreeStructure()`: **<1 second** for 12 files (90%+ improvement)
- 2-3 batched calls instead of 22 sequential calls - 2-3 batched calls instead of 22 sequential calls
- Native cloud storage batch APIs for maximum throughput - Native cloud storage batch APIs for maximum throughput
**IMPORTANT:** The v5.12.0 batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See v6.0.0 changes for comprehensive storage path optimizations. **IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`**, not to `readFile()` or individual `get()` operations. See changes for comprehensive storage path optimizations.
--- ---
@ -56,7 +55,7 @@ results.size // → 3 (number of found entities)
### 2. `storage.getNounMetadataBatch(ids)` ### 2. `storage.getNounMetadataBatch(ids)`
Batch metadata retrieval with direct O(1) path construction (v6.0.0+). Batch metadata retrieval with direct O(1) path construction.
```typescript ```typescript
const storage = brain.storage as BaseStorage const storage = brain.storage as BaseStorage
@ -65,8 +64,8 @@ const ids = ['id1', 'id2', 'id3']
const metadataMap: Map<string, NounMetadata> = await storage.getNounMetadataBatch(ids) const metadataMap: Map<string, NounMetadata> = await storage.getNounMetadataBatch(ids)
for (const [id, metadata] of metadataMap) { for (const [id, metadata] of metadataMap) {
console.log(metadata.noun) // Type: 'document', 'person', etc. console.log(metadata.noun) // Type: 'document', 'person', etc.
console.log(metadata.data) // Entity data console.log(metadata.data) // Entity data
} }
``` ```
@ -92,22 +91,22 @@ const storage = brain.storage as BaseStorage
// Get all relationships from multiple sources // Get all relationships from multiple sources
const results: Map<string, GraphVerb[]> = await storage.getVerbsBySourceBatch([ const results: Map<string, GraphVerb[]> = await storage.getVerbsBySourceBatch([
'person1', 'person1',
'person2' 'person2'
]) ])
// Filter by verb type // Filter by verb type
const createsResults = await storage.getVerbsBySourceBatch( const createsResults = await storage.getVerbsBySourceBatch(
['person1', 'person2'], ['person1', 'person2'],
'creates' 'creates'
) )
// Process results // Process results
for (const [sourceId, verbs] of results) { for (const [sourceId, verbs] of results) {
console.log(`${sourceId} has ${verbs.length} relationships`) console.log(`${sourceId} has ${verbs.length} relationships`)
verbs.forEach(verb => { verbs.forEach(verb => {
console.log(` → ${verb.verb} → ${verb.targetId}`) console.log(` → ${verb.verb} → ${verb.targetId}`)
}) })
} }
``` ```
@ -130,8 +129,8 @@ COW-aware batch path resolution with branch inheritance.
const storage = brain.storage as BaseStorage const storage = brain.storage as BaseStorage
const paths = [ const paths = [
'entities/nouns/{shard}/id1/metadata.json', 'entities/nouns/{shard}/id1/metadata.json',
'entities/nouns/{shard}/id2/metadata.json' 'entities/nouns/{shard}/id2/metadata.json'
] ]
// Resolves to: branches/{branch}/entities/nouns/{shard}/{id}/metadata.json // Resolves to: branches/{branch}/entities/nouns/{shard}/{id}/metadata.json
@ -160,9 +159,9 @@ const results = await gcsStorage.readBatch(paths)
// Configuration // Configuration
gcsStorage.getBatchConfig() // → { gcsStorage.getBatchConfig() // → {
// maxBatchSize: 1000, // maxBatchSize: 1000,
// maxConcurrent: 100, // maxConcurrent: 100,
// operationsPerSecond: 1000 // operationsPerSecond: 1000
// } // }
``` ```
@ -185,9 +184,9 @@ const results = await s3Storage.readBatch(paths)
// Configuration // Configuration
s3Storage.getBatchConfig() // → { s3Storage.getBatchConfig() // → {
// maxBatchSize: 1000, // maxBatchSize: 1000,
// maxConcurrent: 150, // maxConcurrent: 150,
// operationsPerSecond: 5000 // operationsPerSecond: 5000
// } // }
``` ```
@ -208,9 +207,9 @@ const results = await r2Storage.readBatch(paths)
// Configuration // Configuration
r2Storage.getBatchConfig() // → { r2Storage.getBatchConfig() // → {
// maxBatchSize: 1000, // maxBatchSize: 1000,
// maxConcurrent: 150, // maxConcurrent: 150,
// operationsPerSecond: 6000 // operationsPerSecond: 6000
// } // }
``` ```
@ -231,9 +230,9 @@ const results = await azureStorage.readBatch(paths)
// Configuration // Configuration
azureStorage.getBatchConfig() // → { azureStorage.getBatchConfig() // → {
// maxBatchSize: 1000, // maxBatchSize: 1000,
// maxConcurrent: 100, // maxConcurrent: 100,
// operationsPerSecond: 3000 // operationsPerSecond: 3000
// } // }
``` ```
@ -254,7 +253,7 @@ VFS operations automatically use batch APIs for maximum performance.
// OLD: Sequential N+1 pattern (12.7 seconds for 12 files) // OLD: Sequential N+1 pattern (12.7 seconds for 12 files)
const tree = await brain.vfs.getTreeStructure('/my-dir') const tree = await brain.vfs.getTreeStructure('/my-dir')
// NEW v5.12.0: Parallel breadth-first with batching (<1 second) // NEW Parallel breadth-first with batching (<1 second)
// ✅ PathResolver.getChildren() uses brain.batchGet() internally // ✅ PathResolver.getChildren() uses brain.batchGet() internally
// ✅ Parallel traversal of directories at same tree level // ✅ Parallel traversal of directories at same tree level
// ✅ 2-3 batched calls instead of 22 sequential calls // ✅ 2-3 batched calls instead of 22 sequential calls
@ -264,16 +263,16 @@ const tree = await brain.vfs.getTreeStructure('/my-dir')
``` ```
VFS.getTreeStructure() VFS.getTreeStructure()
↓ PARALLEL (breadth-first traversal) ↓ PARALLEL (breadth-first traversal)
→ PathResolver.getChildren() [all dirs at level processed in parallel] → PathResolver.getChildren() [all dirs at level processed in parallel]
↓ BATCHED ↓ BATCHED
→ brain.batchGet(childIds) [1 call instead of N] → brain.batchGet(childIds) [1 call instead of N]
↓ BATCHED ↓ BATCHED
→ storage.getNounMetadataBatch(ids) [1 call instead of N] → storage.getNounMetadataBatch(ids) [1 call instead of N]
↓ ADAPTER-SPECIFIC ↓ ADAPTER-SPECIFIC
→ GCS: readBatch() with 100 concurrent downloads → GCS: readBatch() with 100 concurrent downloads
→ S3: readBatch() with 150 concurrent downloads → S3: readBatch() with 150 concurrent downloads
→ Memory: Promise.all() parallel reads → Memory: Promise.all() parallel reads
``` ```
**Performance Gains:** **Performance Gains:**
@ -285,11 +284,11 @@ VFS.getTreeStructure()
## Advanced Features Compatibility ## Advanced Features Compatibility
### ✅ ID-First Storage Architecture (v6.0.0+) ### ✅ ID-First Storage Architecture
All batch operations use direct ID-first paths - no type lookup needed! All batch operations use direct ID-first paths - no type lookup needed!
**NEW v6.0.0 Path Structure:** **ID-First Path Structure:**
``` ```
entities/nouns/{SHARD}/{ID}/metadata.json entities/nouns/{SHARD}/{ID}/metadata.json
entities/verbs/{SHARD}/{ID}/metadata.json entities/verbs/{SHARD}/{ID}/metadata.json
@ -299,7 +298,7 @@ entities/verbs/{SHARD}/{ID}/metadata.json
```typescript ```typescript
// Every ID maps directly to exactly ONE path - 40x faster! // Every ID maps directly to exactly ONE path - 40x faster!
const id = 'abc-123' const id = 'abc-123'
const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars) const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
const path = `entities/nouns/${shard}/${id}/metadata.json` const path = `entities/nouns/${shard}/${id}/metadata.json`
// No type cache needed! // No type cache needed!
@ -343,7 +342,7 @@ const fork = await brain.fork('experiment')
// Batch operations are isolated // Batch operations are isolated
await brain.batchGet([id1, id2]) // → Reads from: branches/main/... await brain.batchGet([id1, id2]) // → Reads from: branches/main/...
await fork.batchGet([id1, id2]) // → Reads from: branches/experiment/... await fork.batchGet([id1, id2]) // → Reads from: branches/experiment/...
``` ```
**Inheritance:** **Inheritance:**
@ -391,7 +390,7 @@ Historical queries use `HistoricalStorageAdapter` which wraps batch operations t
### VFS Operations (12 Files) ### VFS Operations (12 Files)
| Storage | Before v5.12.0 | After v5.12.0 | Improvement | | Storage | Before optimization | After optimization | Improvement |
|---------|---------------|---------------|-------------| |---------|---------------|---------------|-------------|
| **GCS** | 12.7s | <1s | **92% faster** | | **GCS** | 12.7s | <1s | **92% faster** |
| **S3** | 13.2s | <1s | **92% faster** | | **S3** | 13.2s | <1s | **92% faster** |
@ -430,8 +429,8 @@ Batch operations gracefully handle missing or invalid entities:
```typescript ```typescript
const validId = 'abc-123-...' const validId = 'abc-123-...'
const invalidIds = [ const invalidIds = [
'11111111-1111-1111-1111-111111111111', '11111111-1111-1111-1111-111111111111',
'22222222-2222-2222-2222-222222222222' '22222222-2222-2222-2222-222222222222'
] ]
const results = await brain.batchGet([validId, ...invalidIds]) const results = await brain.batchGet([validId, ...invalidIds])
@ -471,8 +470,8 @@ results.size // → 1 (deduplicated automatically)
```typescript ```typescript
const entities = [] const entities = []
for (const id of ids) { for (const id of ids) {
const entity = await brain.get(id) const entity = await brain.get(id)
if (entity) entities.push(entity) if (entity) entities.push(entity)
} }
``` ```
@ -492,8 +491,8 @@ const entities = Array.from(results.values())
```typescript ```typescript
const allVerbs = [] const allVerbs = []
for (const sourceId of sourceIds) { for (const sourceId of sourceIds) {
const verbs = await brain.getRelations({ from: sourceId }) const verbs = await brain.getRelations({ from: sourceId })
allVerbs.push(...verbs) allVerbs.push(...verbs)
} }
``` ```
@ -504,7 +503,7 @@ const results = await storage.getVerbsBySourceBatch(sourceIds)
const allVerbs = [] const allVerbs = []
for (const verbs of results.values()) { for (const verbs of results.values()) {
allVerbs.push(...verbs) allVerbs.push(...verbs)
} }
``` ```
@ -522,7 +521,7 @@ const results = await brain.batchGet(ids)
// ❌ BAD: Individual gets in loop // ❌ BAD: Individual gets in loop
for (const id of ids) { for (const id of ids) {
await brain.get(id) await brain.get(id)
} }
``` ```
@ -556,13 +555,13 @@ const results = await brain.batchGet(ids)
// Check results // Check results
for (const id of ids) { for (const id of ids) {
if (results.has(id)) { if (results.has(id)) {
// Entity exists // Entity exists
const entity = results.get(id) const entity = results.get(id)
} else { } else {
// Entity missing (not an error) // Entity missing (not an error)
console.log(`Entity ${id} not found`) console.log(`Entity ${id} not found`)
} }
} }
``` ```
@ -597,15 +596,15 @@ npx vitest run tests/integration/storage-batch-operations.test.ts
``` ```
User Code (brain.batchGet) User Code (brain.batchGet)
High-Level API (src/brainy.ts) High-Level API (src/brainy.ts)
Storage Layer (src/storage/baseStorage.ts) Storage Layer (src/storage/baseStorage.ts)
COW Layer (readBatchWithInheritance) COW Layer (readBatchWithInheritance)
Adapter Layer (readBatchFromAdapter) Adapter Layer (readBatchFromAdapter)
Cloud Adapter (GCS/S3/Azure native batch APIs) Cloud Adapter (GCS/S3/Azure native batch APIs)
``` ```
@ -616,11 +615,11 @@ If an adapter doesn't implement `readBatch()`, the system automatically falls ba
```typescript ```typescript
// BaseStorage.readBatchFromAdapter() // BaseStorage.readBatchFromAdapter()
if (typeof selfWithBatch.readBatch === 'function') { if (typeof selfWithBatch.readBatch === 'function') {
// Use native batch API // Use native batch API
return await selfWithBatch.readBatch(resolvedPaths) return await selfWithBatch.readBatch(resolvedPaths)
} else { } else {
// Automatic parallel fallback // Automatic parallel fallback
return await Promise.all(resolvedPaths.map(path => this.read(path))) return await Promise.all(resolvedPaths.map(path => this.read(path)))
} }
``` ```
@ -658,7 +657,7 @@ if (typeof selfWithBatch.readBatch === 'function') {
- Zero N+1 query patterns - Zero N+1 query patterns
**Compatibility:** **Compatibility:**
- ✅ ID-first storage (v6.0.0+) - ✅ ID-first storage
- ✅ Sharding (256 shards) - ✅ Sharding (256 shards)
- ✅ COW (branch isolation, inheritance) - ✅ COW (branch isolation, inheritance)
- ✅ fork() and checkout() - ✅ fork() and checkout()

View file

@ -1,6 +1,6 @@
# Creating Augmentations for Brainy # Creating Augmentations for Brainy
> **Updated for v4.0.0** - Includes metadata structure changes and type system improvements > **Updated** - Includes metadata structure changes and type system improvements
## The BrainyAugmentation Interface ## The BrainyAugmentation Interface
@ -8,38 +8,38 @@ Every augmentation implements this simple yet powerful interface:
```typescript ```typescript
interface BrainyAugmentation { interface BrainyAugmentation {
// Identification // Identification
name: string // Unique name for your augmentation name: string // Unique name for your augmentation
// Execution control // Execution control
timing: 'before' | 'after' | 'around' | 'replace' // When to execute timing: 'before' | 'after' | 'around' | 'replace' // When to execute
operations: string[] // Which operations to intercept operations: string[] // Which operations to intercept
priority: number // Execution order (higher = first) priority: number // Execution order (higher = first)
// Lifecycle methods // Lifecycle methods
initialize(context: AugmentationContext): Promise<void> initialize(context: AugmentationContext): Promise<void>
execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T>
shutdown?(): Promise<void> // Optional cleanup shutdown?(): Promise<void> // Optional cleanup
} }
``` ```
## v4.0.0 Breaking Changes for Augmentation Developers ## Breaking Changes for Augmentation Developers
### 1. Metadata Structure Separation ### 1. Metadata Structure Separation
v4.0.0 introduces strict metadata/vector separation for billion-scale performance: Brainy introduces strict metadata/vector separation for billion-scale performance:
```typescript ```typescript
// ✅ v4.0.0: Metadata has required type field // ✅ Metadata has required type field
interface NounMetadata { interface NounMetadata {
noun: NounType // Required! Must be a valid noun type noun: NounType // Required! Must be a valid noun type
[key: string]: any // Your custom metadata [key: string]: any // Your custom metadata
} }
interface VerbMetadata { interface VerbMetadata {
verb: VerbType // Required! Must be a valid verb type verb: VerbType // Required! Must be a valid verb type
sourceId: string sourceId: string
targetId: string targetId: string
[key: string]: any [key: string]: any
} }
``` ```
@ -61,7 +61,7 @@ The verb relationship field changed from `type` to `verb`:
// ❌ v3.x // ❌ v3.x
verb.type === 'relatedTo' verb.type === 'relatedTo'
// ✅ v4.0.0 // ✅ Current
verb.verb === 'relatedTo' verb.verb === 'relatedTo'
``` ```
@ -69,7 +69,7 @@ verb.verb === 'relatedTo'
Storage augmentations are special - they provide the storage backend for Brainy. Storage augmentations are special - they provide the storage backend for Brainy.
### Important: v4.0.0 Storage Requirements ### Important: Storage Requirements
Your storage adapter MUST: Your storage adapter MUST:
1. **Wrap metadata** with required `noun`/`verb` fields 1. **Wrap metadata** with required `noun`/`verb` fields
@ -81,67 +81,67 @@ import { StorageAugmentation } from 'brainy/augmentations'
import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy' import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy'
export class MyCustomStorage extends BaseStorageAdapter { export class MyCustomStorage extends BaseStorageAdapter {
// Internal method: Returns pure structure // Internal method: Returns pure structure
async _getNoun(id: string): Promise<HNSWNoun | null> { async _getNoun(id: string): Promise<HNSWNoun | null> {
const data = await this.fetchFromDatabase(id) const data = await this.fetchFromDatabase(id)
return data ? { return data ? {
id: data.id, id: data.id,
vector: data.vector, vector: data.vector,
nounType: data.type nounType: data.type
} : null } : null
} }
// Public method: Returns WithMetadata structure // Public method: Returns WithMetadata structure
async getNoun(id: string): Promise<HNSWNounWithMetadata | null> { async getNoun(id: string): Promise<HNSWNounWithMetadata | null> {
const noun = await this._getNoun(id) const noun = await this._getNoun(id)
if (!noun) return null if (!noun) return null
// Fetch metadata separately (v4.0.0 pattern) // Fetch metadata separately
const metadata = await this.getNounMetadata(id) const metadata = await this.getNounMetadata(id)
return { return {
...noun, ...noun,
metadata: metadata || { noun: noun.nounType || 'thing' } metadata: metadata || { noun: noun.nounType || 'thing' }
} }
} }
// CRITICAL: Always save with proper metadata structure // CRITICAL: Always save with proper metadata structure
async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> { async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise<void> {
// Validate metadata has required 'noun' field // Validate metadata has required 'noun' field
if (!metadata?.noun) { if (!metadata?.noun) {
throw new Error('v4.0.0: NounMetadata requires "noun" field') throw new Error('NounMetadata requires "noun" field')
} }
await this.database.save({ await this.database.save({
id: noun.id, id: noun.id,
vector: noun.vector, vector: noun.vector,
nounType: noun.nounType, nounType: noun.nounType,
metadata: metadata // Stored separately in v4.0.0 metadata: metadata // Stored separately
}) })
} }
} }
export class MyStorageAugmentation extends StorageAugmentation { export class MyStorageAugmentation extends StorageAugmentation {
private config: MyStorageConfig private config: MyStorageConfig
constructor(config: MyStorageConfig) { constructor(config: MyStorageConfig) {
super() super()
this.name = 'my-custom-storage' this.name = 'my-custom-storage'
this.config = config this.config = config
} }
// Called during storage resolution phase // Called during storage resolution phase
async provideStorage(): Promise<StorageAdapter> { async provideStorage(): Promise<StorageAdapter> {
const storage = new MyCustomStorage(this.config) const storage = new MyCustomStorage(this.config)
this.storageAdapter = storage this.storageAdapter = storage
return storage return storage
} }
// Called during augmentation initialization // Called during augmentation initialization
protected async onInitialize(): Promise<void> { protected async onInitialize(): Promise<void> {
await this.storageAdapter!.init() await this.storageAdapter!.init()
this.log(`Custom storage initialized`) this.log(`Custom storage initialized`)
} }
} }
``` ```
@ -151,9 +151,9 @@ export class MyStorageAugmentation extends StorageAugmentation {
// Register before brain.init() // Register before brain.init()
const brain = new Brainy() const brain = new Brainy()
brain.augmentations.register(new MyStorageAugmentation({ brain.augmentations.register(new MyStorageAugmentation({
connectionString: 'redis://localhost:6379' connectionString: 'redis://localhost:6379'
})) }))
await brain.init() // Will use your storage! await brain.init() // Will use your storage!
``` ```
## Creating a Feature Augmentation ## Creating a Feature Augmentation
@ -164,43 +164,43 @@ Here's a complete example of a caching augmentation:
import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations' import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations'
export class CachingAugmentation extends BaseAugmentation { export class CachingAugmentation extends BaseAugmentation {
private cache = new Map<string, any>() private cache = new Map<string, any>()
constructor() { constructor() {
super() super()
this.name = 'smart-cache' this.name = 'smart-cache'
this.timing = 'around' // Wrap operations this.timing = 'around' // Wrap operations
this.operations = ['search'] // Only cache searches this.operations = ['search'] // Only cache searches
this.priority = 50 // Mid-priority this.priority = 50 // Mid-priority
} }
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> { async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
if (operation === 'search') { if (operation === 'search') {
// Check cache // Check cache
const cacheKey = JSON.stringify(params) const cacheKey = JSON.stringify(params)
if (this.cache.has(cacheKey)) { if (this.cache.has(cacheKey)) {
this.log('Cache hit!') this.log('Cache hit!')
return this.cache.get(cacheKey) return this.cache.get(cacheKey)
} }
// Execute and cache // Execute and cache
const result = await next() const result = await next()
this.cache.set(cacheKey, result) this.cache.set(cacheKey, result)
return result return result
} }
// Pass through other operations // Pass through other operations
return next() return next()
} }
protected async onInitialize(): Promise<void> { protected async onInitialize(): Promise<void> {
this.log('Cache initialized') this.log('Cache initialized')
} }
async shutdown(): Promise<void> { async shutdown(): Promise<void> {
this.cache.clear() this.cache.clear()
await super.shutdown() await super.shutdown()
} }
} }
``` ```
@ -210,9 +210,9 @@ export class CachingAugmentation extends BaseAugmentation {
```typescript ```typescript
timing = 'before' timing = 'before'
async execute(op, params, next) { async execute(op, params, next) {
// Validate/transform input // Validate/transform input
const validated = await validate(params) const validated = await validate(params)
return next(validated) // Pass modified params return next(validated) // Pass modified params
} }
``` ```
@ -220,10 +220,10 @@ async execute(op, params, next) {
```typescript ```typescript
timing = 'after' timing = 'after'
async execute(op, params, next) { async execute(op, params, next) {
const result = await next() const result = await next()
// Log, analyze, or modify result // Log, analyze, or modify result
console.log(`Operation ${op} returned:`, result) console.log(`Operation ${op} returned:`, result)
return result return result
} }
``` ```
@ -231,15 +231,15 @@ async execute(op, params, next) {
```typescript ```typescript
timing = 'around' timing = 'around'
async execute(op, params, next) { async execute(op, params, next) {
console.log('Starting', op) console.log('Starting', op)
try { try {
const result = await next() const result = await next()
console.log('Success', op) console.log('Success', op)
return result return result
} catch (error) { } catch (error) {
console.log('Failed', op, error) console.log('Failed', op, error)
throw error throw error
} }
} }
``` ```
@ -247,8 +247,8 @@ async execute(op, params, next) {
```typescript ```typescript
timing = 'replace' timing = 'replace'
async execute(op, params, next) { async execute(op, params, next) {
// Don't call next() - replace entirely! // Don't call next() - replace entirely!
return myCustomImplementation(params) return myCustomImplementation(params)
} }
``` ```
@ -266,10 +266,10 @@ Common operations in Brainy:
```typescript ```typescript
interface AugmentationContext { interface AugmentationContext {
brain: Brainy // The brain instance brain: Brainy // The brain instance
storage: StorageAdapter // Storage backend storage: StorageAdapter // Storage backend
config: BrainyConfig // Configuration config: BrainyConfig // Configuration
log: (message: string, level?: 'info' | 'warn' | 'error') => void log: (message: string, level?: 'info' | 'warn' | 'error') => void
} }
``` ```
@ -278,50 +278,50 @@ interface AugmentationContext {
### 1. Redis Storage Augmentation ### 1. Redis Storage Augmentation
```typescript ```typescript
export class RedisStorageAugmentation extends StorageAugmentation { export class RedisStorageAugmentation extends StorageAugmentation {
async provideStorage(): Promise<StorageAdapter> { async provideStorage(): Promise<StorageAdapter> {
return new RedisAdapter({ return new RedisAdapter({
host: 'localhost', host: 'localhost',
port: 6379, port: 6379,
// Implement full StorageAdapter interface // Implement full StorageAdapter interface
}) })
} }
} }
``` ```
### 2. Audit Trail Augmentation ### 2. Audit Trail Augmentation
```typescript ```typescript
export class AuditAugmentation extends BaseAugmentation { export class AuditAugmentation extends BaseAugmentation {
timing = 'after' timing = 'after'
operations = ['add', 'update', 'delete'] operations = ['add', 'update', 'delete']
async execute(op, params, next) { async execute(op, params, next) {
const result = await next() const result = await next()
// Log to audit trail // Log to audit trail
await this.logAudit({ await this.logAudit({
operation: op, operation: op,
params, params,
result, result,
timestamp: new Date(), timestamp: new Date(),
user: this.context.config.currentUser user: this.context.config.currentUser
}) })
return result return result
} }
} }
``` ```
### 3. Rate Limiting Augmentation ### 3. Rate Limiting Augmentation
```typescript ```typescript
export class RateLimitAugmentation extends BaseAugmentation { export class RateLimitAugmentation extends BaseAugmentation {
timing = 'before' timing = 'before'
operations = ['search'] operations = ['search']
private limiter = new RateLimiter({ rps: 100 }) private limiter = new RateLimiter({ rps: 100 })
async execute(op, params, next) { async execute(op, params, next) {
await this.limiter.acquire() // Wait if rate limited await this.limiter.acquire() // Wait if rate limited
return next() return next()
} }
} }
``` ```
@ -332,12 +332,12 @@ Future capability for premium augmentations:
```typescript ```typescript
// package.json // package.json
{ {
"name": "@brain-cloud/redis-storage", "name": "@brain-cloud/redis-storage",
"brainy": { "brainy": {
"type": "augmentation", "type": "augmentation",
"category": "storage", "category": "storage",
"premium": true "premium": true
} }
} }
// Users can install via: // Users can install via:
@ -356,43 +356,43 @@ Future capability for premium augmentations:
6. **Log appropriately** - Use context.log() for consistent output 6. **Log appropriately** - Use context.log() for consistent output
7. **Document your augmentation** - Include examples 7. **Document your augmentation** - Include examples
### v4.0.0 Specific Best Practices ### Specific Best Practices
8. **Always include `noun` field** when creating/modifying NounMetadata: 8. **Always include `noun` field** when creating/modifying NounMetadata:
```typescript ```typescript
const metadata: NounMetadata = { const metadata: NounMetadata = {
noun: 'thing', // REQUIRED! noun: 'thing', // REQUIRED!
yourField: 'value' yourField: 'value'
} }
``` ```
9. **Use `verb` property** not `type` when working with relationships: 9. **Use `verb` property** not `type` when working with relationships:
```typescript ```typescript
// ✅ Correct // ✅ Correct
if (verb.verb === 'relatedTo') { ... } if (verb.verb === 'relatedTo') { ... }
// ❌ Wrong (v3.x pattern) // ❌ Wrong (v3.x pattern)
if (verb.type === 'relatedTo') { ... } if (verb.type === 'relatedTo') { ... }
``` ```
10. **Access metadata correctly** from storage: 10. **Access metadata correctly** from storage:
```typescript ```typescript
// ✅ Correct - metadata is already structured // ✅ Correct - metadata is already structured
const nounType = noun.metadata.noun const nounType = noun.metadata.noun
// ⚠️ Fallback pattern for robustness // ⚠️ Fallback pattern for robustness
const nounType = noun.metadata?.noun || 'thing' const nounType = noun.metadata?.noun || 'thing'
``` ```
11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations: 11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations:
```typescript ```typescript
// ✅ Good - Separate concerns // ✅ Good - Separate concerns
await storage.saveNoun(noun) await storage.saveNoun(noun)
await storage.saveMetadata(noun.id, metadata) await storage.saveMetadata(noun.id, metadata)
// ❌ Bad - Mixing concerns // ❌ Bad - Mixing concerns
await storage.saveNounWithEverything(combinedData) await storage.saveNounWithEverything(combinedData)
``` ```
## Testing Your Augmentation ## Testing Your Augmentation
@ -401,23 +401,23 @@ import { Brainy } from 'brainy'
import { MyAugmentation } from './my-augmentation' import { MyAugmentation } from './my-augmentation'
describe('MyAugmentation', () => { describe('MyAugmentation', () => {
let brain: Brainy let brain: Brainy
beforeEach(async () => { beforeEach(async () => {
brain = new Brainy() brain = new Brainy()
brain.augmentations.register(new MyAugmentation()) brain.augmentations.register(new MyAugmentation())
await brain.init() await brain.init()
}) })
afterEach(async () => { afterEach(async () => {
await brain.destroy() await brain.destroy()
}) })
it('should enhance searches', async () => { it('should enhance searches', async () => {
// Test your augmentation's effect // Test your augmentation's effect
const results = await brain.search('test') const results = await brain.search('test')
expect(results).toHaveProperty('enhanced', true) expect(results).toHaveProperty('enhanced', true)
}) })
}) })
``` ```

View file

@ -613,7 +613,7 @@ vfsFiles.forEach(f => {
}) })
// 3. VFS FILTERING IN KNOWLEDGE QUERIES // 3. VFS FILTERING IN KNOWLEDGE QUERIES
console.log('\n🔍 Understanding VFS filtering (v4.4.0)...\n') console.log('\n🔍 Understanding VFS filtering...\n')
// Create some knowledge entities // Create some knowledge entities
const conceptId = await brain.add({ const conceptId = await brain.add({
@ -758,7 +758,7 @@ await brain.close()
### Key Concepts ### Key Concepts
#### 1. **VFS Filtering Architecture (v4.4.0)** #### 1. **VFS Filtering Architecture**
```typescript ```typescript
// 🎯 DEFAULT BEHAVIOR: Clean Separation // 🎯 DEFAULT BEHAVIOR: Clean Separation

File diff suppressed because it is too large Load diff

View file

@ -24,11 +24,11 @@ Where:
- `f` = number of fields for entity type - `f` = number of fields for entity type
- `t` = number of types (42 nouns, 127 verbs) - `t` = number of types (42 nouns, 127 verbs)
### v5.11.1: brain.get() Metadata-Only Optimization ### brain.get() Metadata-Only Optimization
**Massive Performance Improvement**: `brain.get()` is now **76-81% faster** by default! **Massive Performance Improvement**: `brain.get()` is now **76-81% faster** by default!
| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup | Use Case | | Operation | Before | After | Speedup | Use Case |
|-----------|------------------|-----------------|---------|----------| |-----------|------------------|-----------------|---------|----------|
| **brain.get() (metadata-only)** | 43ms, 6KB | **10ms, 300 bytes** | **76-81%** | VFS, existence checks, metadata | | **brain.get() (metadata-only)** | 43ms, 6KB | **10ms, 300 bytes** | **76-81%** | VFS, existence checks, metadata |
| **brain.get({ includeVectors: true })** | 43ms, 6KB | 43ms, 6KB | 0% | Similarity calculations | | **brain.get({ includeVectors: true })** | 43ms, 6KB | 43ms, 6KB | 0% | Similarity calculations |
@ -308,7 +308,7 @@ const results = await Promise.all(searchPromises)
- ✅ **Horizontally Scalable**: Stateless operations support clustering - ✅ **Horizontally Scalable**: Stateless operations support clustering
- ✅ **Zero Stubs**: Every line of code is production-ready - ✅ **Zero Stubs**: Every line of code is production-ready
## Lazy Loading Performance (v5.7.7+) ## Lazy Loading Performance
Brainy supports two initialization modes for optimal performance across different use cases: Brainy supports two initialization modes for optimal performance across different use cases:
@ -324,7 +324,7 @@ await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities)
- First query: Instant (indexes already loaded) - First query: Instant (indexes already loaded)
- Use case: Traditional applications, long-running servers - Use case: Traditional applications, long-running servers
### Mode 2: Lazy Loading (v5.7.7+) ### Mode 2: Lazy Loading
```javascript ```javascript
const brain = new Brainy({ disableAutoRebuild: true }) const brain = new Brainy({ disableAutoRebuild: true })

View file

@ -1,8 +1,8 @@
# Brainy Documentation (v6.5.0) # Brainy Documentation
Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine. Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine.
## 🆕 What's New in v4.0.0 ## 🆕 What's New in
**Production-Ready Cost Optimization:** **Production-Ready Cost Optimization:**
- **Lifecycle Management**: Automatic tier transitions for S3, GCS, Azure (96% cost savings!) - **Lifecycle Management**: Automatic tier transitions for S3, GCS, Azure (96% cost savings!)
@ -14,7 +14,7 @@ Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI
**Cost Impact Example (500TB dataset):** **Cost Impact Example (500TB dataset):**
- Before: $138,000/year - Before: $138,000/year
- After v4.0.0: $5,940/year - After $5,940/year
- **Savings: $132,060/year (96%)** - **Savings: $132,060/year (96%)**
## 📊 Implementation Status ## 📊 Implementation Status
@ -85,23 +85,23 @@ await brain.init()
// Add entities (nouns) // Add entities (nouns)
const articleId = await brain.add({ const articleId = await brain.add({
data: "Revolutionary AI Breakthrough", data: "Revolutionary AI Breakthrough",
type: NounType.Document, type: NounType.Document,
metadata: { category: "technology", rating: 4.8 } metadata: { category: "technology", rating: 4.8 }
}) })
const authorId = await brain.add({ const authorId = await brain.add({
data: "Dr. Sarah Chen", data: "Dr. Sarah Chen",
type: NounType.Person, type: NounType.Person,
metadata: { role: "researcher" } metadata: { role: "researcher" }
}) })
// Create relationships (verbs) // Create relationships (verbs)
await brain.relate({ await brain.relate({
from: authorId, from: authorId,
to: articleId, to: articleId,
type: VerbType.CreatedBy, type: VerbType.CreatedBy,
metadata: { date: "2024-01-15", contribution: "primary" } metadata: { date: "2024-01-15", contribution: "primary" }
}) })
// Query naturally // Query naturally
@ -117,7 +117,7 @@ const results = await brain.find("highly rated technology articles by researcher
| [API Reference](./api/README.md) | **START HERE** - Complete API documentation with examples | | [API Reference](./api/README.md) | **START HERE** - Complete API documentation with examples |
| [VFS Quick Start](./vfs/QUICK_START.md) | Virtual filesystem in 30 seconds | | [VFS Quick Start](./vfs/QUICK_START.md) | Virtual filesystem in 30 seconds |
### 🆕 v4.0.0 Migration & Optimization ### 🆕 Migration & Optimization
| Document | Description | | Document | Description |
|----------|-------------| |----------|-------------|
@ -135,15 +135,15 @@ const results = await brain.find("highly rated technology articles by researcher
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | Revolutionary data model - entities and relationships | | [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | Revolutionary data model - entities and relationships |
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Field unified query system | | [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Field unified query system |
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment | | [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
| [Storage Architecture](./architecture/storage-architecture.md) | **v4.0.0** - Storage adapters and optimization | | [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
| [Data Storage Architecture](./architecture/data-storage-architecture.md) | **v4.0.0** - Metadata/vector separation, sharding | | [Data Storage Architecture](./architecture/data-storage-architecture.md) | Metadata/vector separation, sharding |
| [Index Architecture](./architecture/index-architecture.md) | HNSW, Graph, and Metadata indexing | | [Index Architecture](./architecture/index-architecture.md) | HNSW, Graph, and Metadata indexing |
### 💾 Storage & Deployment ### 💾 Storage & Deployment
| Document | Description | | Document | Description |
|----------|-------------| |----------|-------------|
| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | **v4.0.0** - Deploy on AWS, GCP, Azure, Cloudflare | | [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | Deploy on AWS, GCP, Azure, Cloudflare |
| [AWS Deployment](./deployment/aws-deployment.md) | AWS-specific deployment patterns | | [AWS Deployment](./deployment/aws-deployment.md) | AWS-specific deployment patterns |
| [GCP Deployment](./deployment/gcp-deployment.md) | Google Cloud deployment | | [GCP Deployment](./deployment/gcp-deployment.md) | Google Cloud deployment |
| [Kubernetes Deployment](./deployment/kubernetes-deployment.md) | K8s deployment configurations | | [Kubernetes Deployment](./deployment/kubernetes-deployment.md) | K8s deployment configurations |
@ -257,75 +257,68 @@ const results = await brain.find("highly rated technology articles by researcher
``` ```
docs/ docs/
├── README.md (this file) # Complete documentation index ├── README.md (this file) # Complete documentation index
├── MIGRATION-V3-TO-V4.md # v4.0.0 migration guide ├── MIGRATION-V3-TO-V4.md # Migration guide
├── guides/ # User guides ├── guides/ # User guides
├── natural-language.md │ ├── natural-language.md
├── neural-api.md │ ├── neural-api.md
├── import-anything.md │ ├── import-anything.md
├── framework-integration.md │ ├── framework-integration.md
├── nextjs-integration.md │ ├── nextjs-integration.md
├── vue-integration.md │ ├── vue-integration.md
├── distributed-system.md │ ├── distributed-system.md
├── model-loading.md │ ├── model-loading.md
└── enterprise-for-everyone.md │ └── enterprise-for-everyone.md
├── architecture/ # System architecture ├── architecture/ # System architecture
│ ├── overview.md │ ├── overview.md
│ ├── noun-verb-taxonomy.md │ ├── noun-verb-taxonomy.md
│ ├── triple-intelligence.md │ ├── triple-intelligence.md
│ ├── zero-config.md │ ├── zero-config.md
│ ├── storage-architecture.md # v4.0.0 │ ├── storage-architecture.md│ ├── data-storage-architecture.md│ ├── index-architecture.md
│ ├── data-storage-architecture.md # v4.0.0 │ ├── distributed-storage.md
│ ├── index-architecture.md │ ├── augmentations.md
│ ├── distributed-storage.md │ ├── augmentation-system-audit.md
│ ├── augmentations.md │ ├── augmentations-actual.md
│ ├── augmentation-system-audit.md │ ├── finite-type-system.md
│ ├── augmentations-actual.md │ └── ...
│ ├── finite-type-system.md
│ └── ...
├── operations/ # Operations guides ├── operations/ # Operations guides
│ ├── cost-optimization-aws-s3.md # v4.0.0 │ ├── cost-optimization-aws-s3.md│ ├── cost-optimization-gcs.md│ ├── cost-optimization-azure.md│ ├── cost-optimization-cloudflare-r2.md│ └── capacity-planning.md
│ ├── cost-optimization-gcs.md # v4.0.0
│ ├── cost-optimization-azure.md # v4.0.0
│ ├── cost-optimization-cloudflare-r2.md # v4.0.0
│ └── capacity-planning.md
├── deployment/ # Deployment guides ├── deployment/ # Deployment guides
│ ├── CLOUD_DEPLOYMENT_GUIDE.md # v4.0.0 │ ├── CLOUD_DEPLOYMENT_GUIDE.md│ ├── aws-deployment.md
│ ├── aws-deployment.md │ ├── gcp-deployment.md
│ ├── gcp-deployment.md │ └── kubernetes-deployment.md
│ └── kubernetes-deployment.md
├── vfs/ # Virtual Filesystem docs ├── vfs/ # Virtual Filesystem docs
├── QUICK_START.md │ ├── QUICK_START.md
├── VFS_CORE.md │ ├── VFS_CORE.md
├── SEMANTIC_VFS.md │ ├── SEMANTIC_VFS.md
├── VFS_API_GUIDE.md │ ├── VFS_API_GUIDE.md
├── NEURAL_EXTRACTION.md │ ├── NEURAL_EXTRACTION.md
├── COMMON_PATTERNS.md │ ├── COMMON_PATTERNS.md
└── ... │ └── ...
├── api/ # API documentation ├── api/ # API documentation
├── README.md │ ├── README.md
└── COMPREHENSIVE_API_OVERVIEW.md │ └── COMPREHENSIVE_API_OVERVIEW.md
├── augmentations/ # Augmentation docs ├── augmentations/ # Augmentation docs
├── COMPLETE-REFERENCE.md │ ├── COMPLETE-REFERENCE.md
├── DEVELOPER-GUIDE.md │ ├── DEVELOPER-GUIDE.md
├── CONFIGURATION.md │ ├── CONFIGURATION.md
└── api-server.md │ └── api-server.md
├── features/ # Feature documentation ├── features/ # Feature documentation
├── complete-feature-list.md │ ├── complete-feature-list.md
└── v3-features.md │ └── v3-features.md
└── internal/ # Internal docs └── internal/ # Internal docs
├── AUDIT_REPORT.md ├── AUDIT_REPORT.md
├── CLEANUP_SUMMARY.md ├── CLEANUP_SUMMARY.md
└── HONEST_STATUS.md └── HONEST_STATUS.md
``` ```
## Community ## Community

View file

@ -48,7 +48,7 @@ git commit -m "chore: remove unused dependency"
# DON'T use BREAKING CHANGE for internal changes # DON'T use BREAKING CHANGE for internal changes
git commit -m "feat: improve model delivery git commit -m "feat: improve model delivery
BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers v3.0.0 BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers
``` ```
## Release Workflow Checklist ## Release Workflow Checklist

View file

@ -1,4 +1,4 @@
# Brainy Stage 3: Canonical Taxonomy (v6.0.0) # Brainy Stage 3: Canonical Taxonomy
**Status:** FINAL - This is the definitive, timeless taxonomy **Status:** FINAL - This is the definitive, timeless taxonomy
**Total Types:** 169 (42 nouns + 127 verbs) **Total Types:** 169 (42 nouns + 127 verbs)

View file

@ -1,6 +1,6 @@
# Brainy Complete Public API Reference # Brainy Complete Public API Reference
> **Accurate API documentation for Brainy v6.5.0+** > **Accurate API documentation for Brainy**
## Initialization ## Initialization
@ -13,14 +13,14 @@ await brain.init()
// With configuration // With configuration
const brain = new Brainy({ const brain = new Brainy({
storage: { type: 'memory' }, // or { path: './my-data' } for filesystem storage: { type: 'memory' }, // or { path: './my-data' } for filesystem
embeddingModel: 'Q8', // Q4, Q8, F16, F32 embeddingModel: 'Q8', // Q4, Q8, F16, F32
silent: true // Suppress logs silent: true // Suppress logs
}) })
await brain.init() await brain.init()
``` ```
## Readiness API (v7.3.0+) ## Readiness API
For reliable initialization detection, especially in cloud environments with progressive initialization: For reliable initialization detection, especially in cloud environments with progressive initialization:
@ -28,10 +28,10 @@ For reliable initialization detection, especially in cloud environments with pro
```typescript ```typescript
const brain = new Brainy() const brain = new Brainy()
brain.init() // Fire and forget brain.init() // Fire and forget
// Elsewhere (e.g., API handler) // Elsewhere (e.g., API handler)
await brain.ready // Wait until init() completes await brain.ready // Wait until init() completes
const results = await brain.find({ query: 'test' }) const results = await brain.find({ query: 'test' })
``` ```
@ -39,7 +39,7 @@ const results = await brain.find({ query: 'test' })
```typescript ```typescript
if (brain.isInitialized) { if (brain.isInitialized) {
// Safe to use brain methods // Safe to use brain methods
} }
``` ```
@ -49,7 +49,7 @@ if (brain.isInitialized) {
// Returns true when ALL initialization is complete, including background tasks // Returns true when ALL initialization is complete, including background tasks
// Useful for cloud storage adapters with progressive initialization // Useful for cloud storage adapters with progressive initialization
if (brain.isFullyInitialized()) { if (brain.isFullyInitialized()) {
console.log('All background tasks complete') console.log('All background tasks complete')
} }
``` ```
@ -57,7 +57,7 @@ if (brain.isFullyInitialized()) {
```typescript ```typescript
const brain = new Brainy({ storage: { type: 'gcs', ... } }) const brain = new Brainy({ storage: { type: 'gcs', ... } })
await brain.init() // Fast return in cloud (<200ms) await brain.init() // Fast return in cloud (<200ms)
// Optional: wait for all background tasks (bucket validation, count sync) // Optional: wait for all background tasks (bucket validation, count sync)
await brain.awaitBackgroundInit() await brain.awaitBackgroundInit()
@ -68,15 +68,15 @@ console.log('Fully initialized including background tasks')
```typescript ```typescript
app.get('/health', async (req, res) => { app.get('/health', async (req, res) => {
try { try {
await brain.ready await brain.ready
res.json({ res.json({
status: 'ready', status: 'ready',
fullyInitialized: brain.isFullyInitialized() fullyInitialized: brain.isFullyInitialized()
}) })
} catch (error) { } catch (error) {
res.status(503).json({ status: 'initializing', error: error.message }) res.status(503).json({ status: 'initializing', error: error.message })
} }
}) })
``` ```
@ -87,16 +87,16 @@ app.get('/health', async (req, res) => {
```typescript ```typescript
// Add with data (auto-embedded) // Add with data (auto-embedded)
const id = await brain.add({ const id = await brain.add({
data: 'Machine learning is a subset of AI', data: 'Machine learning is a subset of AI',
type: NounType.Document, type: NounType.Document,
metadata: { author: 'Alice', tags: ['ml', 'ai'] } metadata: { author: 'Alice', tags: ['ml', 'ai'] }
}) })
// Add with pre-computed vector // Add with pre-computed vector
const id = await brain.add({ const id = await brain.add({
data: 'Content here', data: 'Content here',
vector: [0.1, 0.2, ...], // 384 dimensions vector: [0.1, 0.2, ...], // 384 dimensions
type: NounType.Concept type: NounType.Concept
}) })
``` ```
@ -120,9 +120,9 @@ const entity = await brain.get('entity-id', { includeVectors: true })
```typescript ```typescript
await brain.update({ await brain.update({
id: 'entity-id', id: 'entity-id',
data: 'Updated content', // Re-embeds if changed data: 'Updated content', // Re-embeds if changed
metadata: { reviewed: true } // Merges with existing metadata: { reviewed: true } // Merges with existing
}) })
``` ```
@ -144,10 +144,10 @@ await brain.clear()
```typescript ```typescript
const relationId = await brain.relate({ const relationId = await brain.relate({
from: 'source-entity-id', from: 'source-entity-id',
to: 'target-entity-id', to: 'target-entity-id',
type: VerbType.RelatedTo, type: VerbType.RelatedTo,
metadata: { strength: 0.9 } metadata: { strength: 0.9 }
}) })
``` ```
@ -168,8 +168,8 @@ const relations = await brain.getRelations({ to: 'entity-id' })
// Filter by type // Filter by type
const relations = await brain.getRelations({ const relations = await brain.getRelations({
from: 'entity-id', from: 'entity-id',
type: VerbType.Contains type: VerbType.Contains
}) })
``` ```
@ -191,17 +191,17 @@ const results = await brain.find('machine learning algorithms')
// With options // With options
const results = await brain.find({ const results = await brain.find({
query: 'machine learning', query: 'machine learning',
limit: 10, limit: 10,
threshold: 0.7, threshold: 0.7,
type: NounType.Document, type: NounType.Document,
where: { author: 'Alice' }, where: { author: 'Alice' },
excludeVFS: true // Exclude VFS files from results excludeVFS: true // Exclude VFS files from results
}) })
// Natural language query (Triple Intelligence) // Natural language query (Triple Intelligence)
const results = await brain.find( const results = await brain.find(
'Show me documents about AI written by Alice in 2024' 'Show me documents about AI written by Alice in 2024'
) )
``` ```
@ -218,15 +218,15 @@ const results = await brain.find(
```typescript ```typescript
// Find similar to entity ID // Find similar to entity ID
const similar = await brain.similar({ const similar = await brain.similar({
to: 'entity-id', to: 'entity-id',
limit: 5 limit: 5
}) })
// Find similar to vector // Find similar to vector
const similar = await brain.similar({ const similar = await brain.similar({
to: [0.1, 0.2, ...], // Vector to: [0.1, 0.2, ...], // Vector
limit: 10, limit: 10,
type: NounType.Document type: NounType.Document
}) })
``` ```
@ -236,15 +236,15 @@ const similar = await brain.similar({
```typescript ```typescript
const result = await brain.addMany({ const result = await brain.addMany({
items: [ items: [
{ data: 'First item', type: NounType.Document }, { data: 'First item', type: NounType.Document },
{ data: 'Second item', type: NounType.Concept }, { data: 'Second item', type: NounType.Concept },
{ data: 'Third item', metadata: { priority: 'high' } } { data: 'Third item', metadata: { priority: 'high' } }
], ],
continueOnError: true, // Don't stop on failures continueOnError: true, // Don't stop on failures
onProgress: (completed, total) => { onProgress: (completed, total) => {
console.log(`${completed}/${total} complete`) console.log(`${completed}/${total} complete`)
} }
}) })
console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}`) console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}`)
@ -255,18 +255,18 @@ console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}
```typescript ```typescript
// Delete by IDs // Delete by IDs
const result = await brain.deleteMany({ const result = await brain.deleteMany({
ids: ['id1', 'id2', 'id3'], ids: ['id1', 'id2', 'id3'],
continueOnError: true continueOnError: true
}) })
// Delete by type // Delete by type
const result = await brain.deleteMany({ const result = await brain.deleteMany({
type: NounType.TempData type: NounType.TempData
}) })
// Delete by metadata filter // Delete by metadata filter
const result = await brain.deleteMany({ const result = await brain.deleteMany({
where: { status: 'archived' } where: { status: 'archived' }
}) })
``` ```
@ -274,12 +274,12 @@ const result = await brain.deleteMany({
```typescript ```typescript
const ids = await brain.relateMany({ const ids = await brain.relateMany({
items: [ items: [
{ from: 'a', to: 'b', type: VerbType.RelatedTo }, { from: 'a', to: 'b', type: VerbType.RelatedTo },
{ from: 'b', to: 'c', type: VerbType.Contains }, { from: 'b', to: 'c', type: VerbType.Contains },
{ from: 'c', to: 'd', type: VerbType.References } { from: 'c', to: 'd', type: VerbType.References }
], ],
continueOnError: true continueOnError: true
}) })
``` ```
@ -287,11 +287,11 @@ const ids = await brain.relateMany({
```typescript ```typescript
const result = await brain.updateMany({ const result = await brain.updateMany({
items: [ items: [
{ id: 'id1', metadata: { reviewed: true } }, { id: 'id1', metadata: { reviewed: true } },
{ id: 'id2', data: 'Updated content' } { id: 'id2', data: 'Updated content' }
], ],
continueOnError: true continueOnError: true
}) })
``` ```
@ -309,8 +309,8 @@ const detailed = await neural.similar('text1', 'text2', { detailed: true })
// Clustering // Clustering
const clusters = await neural.clusters() const clusters = await neural.clusters()
const clusters = await neural.clusters({ const clusters = await neural.clusters({
algorithm: 'hierarchical', algorithm: 'hierarchical',
maxClusters: 10 maxClusters: 10
}) })
// K-nearest neighbors // K-nearest neighbors
@ -327,13 +327,13 @@ const domainClusters = await neural.clusterByDomain('category')
// Temporal clustering // Temporal clustering
const temporalClusters = await neural.clusterByTime('createdAt', [ const temporalClusters = await neural.clusterByTime('createdAt', [
{ start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1' }, { start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1' },
{ start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2' } { start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2' }
]) ])
// Streaming clusters (for large datasets) // Streaming clusters (for large datasets)
for await (const batch of neural.clusterStream({ batchSize: 100 })) { for await (const batch of neural.clusterStream({ batchSize: 100 })) {
console.log(`Progress: ${batch.progress.percentage}%`) console.log(`Progress: ${batch.progress.percentage}%`)
} }
``` ```
@ -355,11 +355,11 @@ await vfs.mkdir('/project/src', { recursive: true })
const files = await vfs.readdir('/project') const files = await vfs.readdir('/project')
await vfs.rmdir('/project', { recursive: true }) await vfs.rmdir('/project', { recursive: true })
// Bulk operations (v6.5.0+) // Bulk operations
const result = await vfs.bulkWrite([ const result = await vfs.bulkWrite([
{ type: 'mkdir', path: '/data' }, { type: 'mkdir', path: '/data' },
{ type: 'write', path: '/data/config.json', data: '{}' }, { type: 'write', path: '/data/config.json', data: '{}' },
{ type: 'write', path: '/data/users.json', data: '[]' } { type: 'write', path: '/data/users.json', data: '[]' }
]) ])
// Note: mkdir operations run first (sequentially), then other ops in parallel // Note: mkdir operations run first (sequentially), then other ops in parallel
@ -414,22 +414,22 @@ const snapshot = await brain.asOf('commit-id')
```typescript ```typescript
// Stream all entities // Stream all entities
for await (const entity of brain.streaming.entities()) { for await (const entity of brain.streaming.entities()) {
console.log(entity.id) console.log(entity.id)
} }
// Stream with filters // Stream with filters
for await (const entity of brain.streaming.entities({ for await (const entity of brain.streaming.entities({
type: NounType.Document, type: NounType.Document,
where: { status: 'active' } where: { status: 'active' }
})) { })) {
// Process each entity // Process each entity
} }
// Stream relationships // Stream relationships
for await (const relation of brain.streaming.relations({ for await (const relation of brain.streaming.relations({
from: 'entity-id' from: 'entity-id'
})) { })) {
console.log(relation) console.log(relation)
} }
``` ```
@ -438,9 +438,9 @@ for await (const relation of brain.streaming.relations({
```typescript ```typescript
// Paginated queries // Paginated queries
const page1 = await brain.pagination.find({ const page1 = await brain.pagination.find({
query: 'machine learning', query: 'machine learning',
page: 1, page: 1,
pageSize: 20 pageSize: 20
}) })
console.log(`Page ${page1.page} of ${page1.totalPages}`) console.log(`Page ${page1.page} of ${page1.totalPages}`)
@ -448,9 +448,9 @@ console.log(`Total results: ${page1.total}`)
// Get next page // Get next page
const page2 = await brain.pagination.find({ const page2 = await brain.pagination.find({
query: 'machine learning', query: 'machine learning',
page: 2, page: 2,
pageSize: 20 pageSize: 20
}) })
``` ```
@ -517,20 +517,20 @@ VerbType.ModifiedBy
```typescript ```typescript
try { try {
await brain.get('nonexistent-id') await brain.get('nonexistent-id')
} catch (error) { } catch (error) {
if (error.message.includes('not found')) { if (error.message.includes('not found')) {
// Handle missing entity // Handle missing entity
} }
} }
// VFS errors use POSIX codes // VFS errors use POSIX codes
try { try {
await vfs.readFile('/nonexistent') await vfs.readFile('/nonexistent')
} catch (error) { } catch (error) {
if (error.code === 'ENOENT') { if (error.code === 'ENOENT') {
// File not found // File not found
} }
} }
``` ```
@ -538,26 +538,26 @@ try {
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
// Storage // Storage
storage: { storage: {
type: 'memory' | 'filesystem', type: 'memory' | 'filesystem',
path: './data', // For filesystem path: './data', // For filesystem
forceMemoryStorage: false // Force memory even if path exists forceMemoryStorage: false // Force memory even if path exists
}, },
// Embeddings // Embeddings
embeddingModel: 'Q8', // Q4, Q8, F16, F32 embeddingModel: 'Q8', // Q4, Q8, F16, F32
dimensions: 384, // Auto-detected dimensions: 384, // Auto-detected
// Performance // Performance
silent: false, // Suppress console output silent: false, // Suppress console output
verbose: false, // Extra logging verbose: false, // Extra logging
// Augmentations (auto-enabled by default) // Augmentations (auto-enabled by default)
augmentations: { augmentations: {
cache: true, cache: true,
display: true, display: true,
metrics: true metrics: true
} }
}) })
``` ```

View file

@ -1,4 +1,4 @@
# Brainy Data Storage Architecture (v5.11.0) # Brainy Data Storage Architecture
**Complete file structure reference for all storage backends** **Complete file structure reference for all storage backends**
@ -13,7 +13,7 @@ This document explains how Brainy stores, indexes, and scales data across all st
3. [The 4 Indexes](#3-the-4-indexes) 3. [The 4 Indexes](#3-the-4-indexes)
4. [Sharding Strategy](#4-sharding-strategy) 4. [Sharding Strategy](#4-sharding-strategy)
5. [COW (Copy-on-Write) Architecture](#5-cow-copy-on-write-architecture) 5. [COW (Copy-on-Write) Architecture](#5-cow-copy-on-write-architecture)
6. [ID-First Storage Architecture (v6.0.0+)](#6-id-first-storage-architecture-v600) 6. [ID-First Storage Architecture](#6-id-first-storage-architecture-v600)
7. [VFS (Virtual File System)](#7-vfs-virtual-file-system) 7. [VFS (Virtual File System)](#7-vfs-virtual-file-system)
8. [Storage Backend Mapping](#8-storage-backend-mapping) 8. [Storage Backend Mapping](#8-storage-backend-mapping)
9. [Performance Characteristics](#9-performance-characteristics) 9. [Performance Characteristics](#9-performance-characteristics)
@ -438,7 +438,7 @@ Unlike entities and relationships, system metadata consists of **index files** t
Understanding how Brainy constructs storage paths is critical for debugging and optimization. Understanding how Brainy constructs storage paths is critical for debugging and optimization.
### Path Construction Steps (v6.0.0+) ### Path Construction Steps
**For an entity (noun)**: **For an entity (noun)**:
```typescript ```typescript
@ -498,7 +498,7 @@ const fieldIndexPath = `_system/metadata_indexes/__metadata_field_index__${field
const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json` const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
``` ```
### Path Patterns Summary (v6.0.0+) ### Path Patterns Summary
| Data Type | Path Pattern | Sharded? | Branched? | | Data Type | Path Pattern | Sharded? | Branched? |
|-----------|--------------|----------|-----------| |-----------|--------------|----------|-----------|
@ -516,7 +516,7 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
| **HNSW node** | `_system/hnsw/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No | | **HNSW node** | `_system/hnsw/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No |
| **Field index** | `_system/metadata_indexes/__metadata_field_index__{field}.json` | ❌ No | ❌ No | | **Field index** | `_system/metadata_indexes/__metadata_field_index__{field}.json` | ❌ No | ❌ No |
### Key Principles (v6.0.0+) ### Key Principles
1. **Shard Extraction**: Always use first 2 hex characters of UUID/SHA-256 1. **Shard Extraction**: Always use first 2 hex characters of UUID/SHA-256
2. **ID-First**: Shard + ID come BEFORE type (type is in metadata) 2. **ID-First**: Shard + ID come BEFORE type (type is in metadata)
@ -549,7 +549,7 @@ Brainy uses four complementary index systems for different query patterns.
- Memory (standard): ~200MB per 100K entities - Memory (standard): ~200MB per 100K entities
- Memory (lazy): ~15-33MB per 100K entities (5-10x less!) - Memory (lazy): ~15-33MB per 100K entities (5-10x less!)
**Automatic Lazy Mode** (v3.36.0+): Enables automatically when vectors don't fit in UnifiedCache **Automatic Lazy Mode**: Enables automatically when vectors don't fit in UnifiedCache
--- ---
@ -559,7 +559,7 @@ Brainy uses four complementary index systems for different query patterns.
**Location**: MetadataIndexManager index on `noun` field **Location**: MetadataIndexManager index on `noun` field
**Data Structure**: RoaringBitmap32 per type value **Data Structure**: RoaringBitmap32 per type value
**How It Works** (v6.0.0+): **How It Works**:
```typescript ```typescript
// Find all Person entities // Find all Person entities
const people = await brain.getNouns({ type: 'person' }) const people = await brain.getNouns({ type: 'person' })
@ -721,7 +721,7 @@ COW is Brainy's **git-like versioning system** that enables:
- ✅ **Deduplication** (identical data stored only once) - ✅ **Deduplication** (identical data stored only once)
- ✅ **Version history** (full audit trail of all changes) - ✅ **Version history** (full audit trail of all changes)
**Status**: ALWAYS ENABLED (v5.11.0+) - cannot be disabled **Status**: ALWAYS ENABLED - cannot be disabled
--- ---
@ -782,19 +782,19 @@ _cow/blobs/ab/abc123...sha256.bin (used by both entities)
--- ---
## 6. ID-First Storage Architecture (v6.0.0+) ## 6. ID-First Storage Architecture
### 6.1 What is ID-First? ### 6.1 What is ID-First?
**ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused 20-21 second delays on cloud storage. **ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused 20-21 second delays on cloud storage.
**Old type-first structure** (v5.4.0-v5.12.0): **Old type-first structure** (v5.12.0):
``` ```
branches/main/entities/nouns/{TYPE}/metadata/00/001234...uuid.json branches/main/entities/nouns/{TYPE}/metadata/00/001234...uuid.json
# Problem: Requires knowing type OR searching 42 type directories! # Problem: Requires knowing type OR searching 42 type directories!
``` ```
**NEW ID-first structure** (v6.0.0+): **NEW ID-first structure**:
``` ```
branches/main/entities/nouns/00/001234...uuid/metadata.json branches/main/entities/nouns/00/001234...uuid/metadata.json
# Direct O(1) lookup - no type needed! # Direct O(1) lookup - no type needed!
@ -809,7 +809,7 @@ branches/main/entities/nouns/00/001234...uuid/metadata.json
// v5.x: Had to search 42 types if type unknown // v5.x: Had to search 42 types if type unknown
// Result: 21 seconds on GCS (42 types × 500ms) // Result: 21 seconds on GCS (42 types × 500ms)
// v6.0.0: Direct path from ID // Direct path from ID
const id = '001234...' const id = '001234...'
const shard = id.substring(0, 2) // '00' const shard = id.substring(0, 2) // '00'
const path = `branches/main/entities/nouns/${shard}/${id}/metadata.json` const path = `branches/main/entities/nouns/${shard}/${id}/metadata.json`
@ -1165,7 +1165,7 @@ opfs://root/brainy/
### 10.2 clear() Operation ### 10.2 clear() Operation
**What clear() deletes** (v5.11.0+): **What clear() deletes**:
✅ Deletes: ✅ Deletes:
- `branches/` → ALL entity data (all types, all shards, all branches, all forks) - `branches/` → ALL entity data (all types, all shards, all branches, all forks)
@ -1184,7 +1184,7 @@ opfs://root/brainy/
**Example**: **Example**:
```typescript ```typescript
await brain.storage.clear() // ✅ Deletes ALL data correctly (v5.11.0+) await brain.storage.clear() // ✅ Deletes ALL data correctly
await brain.add({ data: 'Alice', type: 'person' }) // ✅ COW reinitializes automatically await brain.add({ data: 'Alice', type: 'person' }) // ✅ COW reinitializes automatically
``` ```
@ -1353,7 +1353,7 @@ const historicalData = await yesterday.getNouns({ type: 'Character' })
await brain.storage.clear() await brain.storage.clear()
``` ```
**What happens in storage** (v5.11.0+): **What happens in storage**:
``` ```
1. Delete all entity data: 1. Delete all entity data:
→ Remove: branches/ (entire directory) → Remove: branches/ (entire directory)
@ -1452,7 +1452,7 @@ await brain.addBatch([
## 11. Summary ## 11. Summary
**Complete Storage Structure (v6.0.0)**: **Complete Storage Structure**:
- **3 storage layers**: branches/ (data), _cow/ (versions), _system/ (indexes) - **3 storage layers**: branches/ (data), _cow/ (versions), _system/ (indexes)
- **2 files per entity**: metadata.json + vector.json (optimized I/O) - **2 files per entity**: metadata.json + vector.json (optimized I/O)
- **4 indexes**: HNSW (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields) - **4 indexes**: HNSW (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields)

View file

@ -25,7 +25,7 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
- **FieldTypeInference** - DuckDB-inspired value-based field type detection - **FieldTypeInference** - DuckDB-inspired value-based field type detection
- **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count) - **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count)
- **Sorted Indexes** - Support orderBy queries (automatically maintained) - **Sorted Indexes** - Support orderBy queries (automatically maintained)
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes (v7.7.0) - **Word Index (`__words__`)** - Text search via FNV-1a word hashes
**GraphAdjacencyIndex contains:** **GraphAdjacencyIndex contains:**
- **lsmTreeSource** - Source → Targets (outgoing edges) - **lsmTreeSource** - Source → Targets (outgoing edges)
@ -39,7 +39,7 @@ All indexes share a **UnifiedCache** for coordinated memory management, ensuring
**Purpose**: Enable O(1) field-value lookups and O(log n) range queries on metadata fields using adaptive chunked sparse indexing. **Purpose**: Enable O(1) field-value lookups and O(log n) range queries on metadata fields using adaptive chunked sparse indexing.
### Internal Architecture (v3.42.0) ### Internal Architecture
```typescript ```typescript
class MetadataIndexManager { class MetadataIndexManager {
@ -64,7 +64,7 @@ class MetadataIndexManager {
### Key Data Structures ### Key Data Structures
#### Chunked Sparse Index (NEW in v3.42.0) #### Chunked Sparse Index
```typescript ```typescript
// SparseIndex: Directory of chunks for a field // SparseIndex: Directory of chunks for a field
// Example: field="status" // Example: field="status"
@ -87,7 +87,7 @@ interface ChunkDescriptor {
class ChunkData { class ChunkData {
chunkId: number chunkId: number
field: string field: string
entries: Map<value, RoaringBitmap32> // ~50 values per chunk (v3.43.0: roaring bitmaps!) entries: Map<value, RoaringBitmap32> // ~50 values per chunk (roaring bitmaps!)
} }
``` ```
@ -96,7 +96,7 @@ class ChunkData {
- O(log n) range queries with zone maps - O(log n) range queries with zone maps
- 630x file reduction (560k flat files → 89 chunk files) - 630x file reduction (560k flat files → 89 chunk files)
#### Roaring Bitmap Optimization (NEW in v3.43.0) #### Roaring Bitmap Optimization
**Problem Solved**: JavaScript `Set<string>` for storing entity IDs was inefficient: **Problem Solved**: JavaScript `Set<string>` for storing entity IDs was inefficient:
- Memory overhead: ~40 bytes per UUID string (36 chars + overhead) - Memory overhead: ~40 bytes per UUID string (36 chars + overhead)
@ -150,7 +150,7 @@ class ChunkData {
**Multi-Field Intersection (THE BIG WIN!)**: **Multi-Field Intersection (THE BIG WIN!)**:
```typescript ```typescript
// Before (v3.42.0): JavaScript array filtering // Before: JavaScript array filtering
async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string[]> { async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string[]> {
// 1. Fetch UUID arrays for each field // 1. Fetch UUID arrays for each field
const statusIds = await this.getIds('status', 'active') // ["uuid1", "uuid2", ...] const statusIds = await this.getIds('status', 'active') // ["uuid1", "uuid2", ...]
@ -160,7 +160,7 @@ async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string
return statusIds.filter(id => roleIds.includes(id)) // O(n*m) array filtering return statusIds.filter(id => roleIds.includes(id)) // O(n*m) array filtering
} }
// After (v3.43.0): Roaring bitmap intersection // After: Roaring bitmap intersection
async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> { async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
// 1. Fetch roaring bitmaps (integers, not UUIDs) // 1. Fetch roaring bitmaps (integers, not UUIDs)
const bitmaps: RoaringBitmap32[] = [] const bitmaps: RoaringBitmap32[] = []
@ -229,7 +229,7 @@ interface ZoneMap {
**Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field **Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field
#### Word Index (`__words__`) - v7.7.0 #### Word Index (`__words__`) -
```typescript ```typescript
// Special field for text/keyword search // Special field for text/keyword search
// Entity text content is tokenized and indexed as word hashes // Entity text content is tokenized and indexed as word hashes
@ -253,14 +253,14 @@ interface ZoneMap {
- **Lowercase normalization**: Case-insensitive matching - **Lowercase normalization**: Case-insensitive matching
- **Automatic integration**: Words extracted via `extractIndexableFields()` - **Automatic integration**: Words extracted via `extractIndexableFields()`
**Hybrid Search** (v7.7.0): Text results combined with vector results using Reciprocal Rank Fusion (RRF): **Hybrid Search**: Text results combined with vector results using Reciprocal Rank Fusion (RRF):
```typescript ```typescript
// RRF formula: score(d) = sum(1 / (k + rank(d))) // RRF formula: score(d) = sum(1 / (k + rank(d)))
// where k = 60 (standard constant) // where k = 60 (standard constant)
// alpha = weight for semantic (0 = text only, 1 = semantic only) // alpha = weight for semantic (0 = text only, 1 = semantic only)
``` ```
### Query Algorithm (v3.42.0) ### Query Algorithm
**Exact Match Query**: **Exact Match Query**:
```typescript ```typescript
@ -317,7 +317,7 @@ async getIdsForRange(field: string, min: any, max: any): Promise<string[]> {
- Adaptive chunking: ~50 values per chunk optimizes I/O - Adaptive chunking: ~50 values per chunk optimizes I/O
- Immediate flushing: No need for dirty tracking or batch writes - Immediate flushing: No need for dirty tracking or batch writes
### Temporal Bucketing (v3.41.0) ### Temporal Bucketing
**Problem Solved**: High-cardinality timestamp fields created massive file pollution. **Problem Solved**: High-cardinality timestamp fields created massive file pollution.
- Example: 575 entities with unique timestamps → 358,407 index files (98.7% pollution!) - Example: 575 entities with unique timestamps → 358,407 index files (98.7% pollution!)
@ -396,7 +396,7 @@ const DEFAULT_EXCLUDE_FIELDS = [
] ]
``` ```
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of v3.41.0 - they are indexed with automatic bucketing. **Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
## 2. HNSWIndex - Vector Similarity Search ## 2. HNSWIndex - Vector Similarity Search
@ -735,7 +735,7 @@ async stats(): Promise<Statistics> {
} }
``` ```
### 5. Index Rebuilding (v5.7.7: Lazy Loading Support) ### 5. Index Rebuilding (Lazy Loading Support)
**Two modes of index loading:** **Two modes of index loading:**
@ -762,7 +762,7 @@ async init(): Promise<void> {
} }
``` ```
#### Mode 2: Lazy Loading on First Query (v5.7.7+) #### Mode 2: Lazy Loading on First Query
```typescript ```typescript
// When disableAutoRebuild: true // When disableAutoRebuild: true
@ -918,7 +918,7 @@ All indexes scale gracefully:
- [Performance Guide](../PERFORMANCE.md) - Performance tuning - [Performance Guide](../PERFORMANCE.md) - Performance tuning
- [Overview](./overview.md) - High-level architecture - [Overview](./overview.md) - High-level architecture
## Summary: Index Hierarchy (v5.7.7) ## Summary: Index Hierarchy
### Level 1: Main Indexes (3) ### Level 1: Main Indexes (3)
All have rebuild() methods and are covered by lazy loading: All have rebuild() methods and are covered by lazy loading:
@ -933,7 +933,7 @@ Automatically managed by parent rebuild():
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget) - **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex) - **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
### Lazy Loading (v5.7.7) ### Lazy Loading
- **Mode 1**: Auto-rebuild on init() (default) - **Mode 1**: Auto-rebuild on init() (default)
- **Mode 2**: Lazy rebuild on first query (when `disableAutoRebuild: true`) - **Mode 2**: Lazy rebuild on first query (when `disableAutoRebuild: true`)
- **Concurrency-safe**: Mutex prevents duplicate rebuilds - **Concurrency-safe**: Mutex prevents duplicate rebuilds

View file

@ -15,7 +15,7 @@ This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, Grap
| **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 | | **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
| **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 | | **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
#### MetadataIndex Persistence Details (v4.2.1+) #### MetadataIndex Persistence Details
The MetadataIndex now persists two components: The MetadataIndex now persists two components:
@ -129,7 +129,7 @@ async init(): Promise<void> {
- 100-2000ms: One-time rebuild to create indices - 100-2000ms: One-time rebuild to create indices
- Total: ~1-3 seconds (one time only) - Total: ~1-3 seconds (one time only)
#### Mode 2: Lazy Loading on First Query (v5.7.7+) #### Mode 2: Lazy Loading on First Query
When `disableAutoRebuild: true`, indexes remain empty after init() and rebuild on first query: When `disableAutoRebuild: true`, indexes remain empty after init() and rebuild on first query:
@ -350,7 +350,7 @@ public async rebuild(options?: {
**Performance Impact**: 200-600x speedup (5 minutes → 500ms for 10K entities) **Performance Impact**: 200-600x speedup (5 minutes → 500ms for 10K entities)
**Correct Pattern** (v3.45.0): **Correct Pattern**:
```typescript ```typescript
// Load ALL nouns ONCE (not 31 times!) // Load ALL nouns ONCE (not 31 times!)
while (hasMore) { while (hasMore) {
@ -392,7 +392,7 @@ while (hasMore) {
```typescript ```typescript
// src/utils/metadataIndex.ts (lines 202-216) // src/utils/metadataIndex.ts (lines 202-216)
async init(): Promise<void> { async init(): Promise<void> {
// STEP 1: Load field registry to discover persisted indices (v4.2.1) // STEP 1: Load field registry to discover persisted indices
// This is THE KEY FIX - O(1) discovery of existing indices // This is THE KEY FIX - O(1) discovery of existing indices
await this.loadFieldRegistry() await this.loadFieldRegistry()

View file

@ -1,40 +1,40 @@
# Storage Architecture (v4.0.0) # Storage Architecture
> **Updated for v4.0.0**: Metadata/vector separation, UUID-based sharding, lifecycle management > **Updated**: Metadata/vector separation, UUID-based sharding, lifecycle management
## Storage Structure ## Storage Structure
### v4.0.0 Architecture: Metadata/Vector Separation ### Architecture: Metadata/Vector Separation
In v4.0.0, entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale: entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
``` ```
brainy-data/ brainy-data/
├── _system/ # System metadata (not sharded) ├── _system/ # System metadata (not sharded)
├── statistics.json # Performance metrics │ ├── statistics.json # Performance metrics
├── __metadata_field_index__*.json # Field indexes │ ├── __metadata_field_index__*.json # Field indexes
└── __metadata_sorted_index__*.json # Sorted indexes │ └── __metadata_sorted_index__*.json # Sorted indexes
├── entities/ ├── entities/
├── nouns/ │ ├── nouns/
│ ├── vectors/ # HNSW graph data (sharded by UUID) │ ├── vectors/ # HNSW graph data (sharded by UUID)
│ │ ├── 00/ # Shard 00 (first 2 hex digits) │ │ ├── 00/ # Shard 00 (first 2 hex digits)
│ │ │ ├── 00123456-....json # Vector + HNSW connections │ │ │ ├── 00123456-....json # Vector + HNSW connections
│ │ │ └── 00abcdef-....json │ │ │ └── 00abcdef-....json
│ │ ├── 01/ ... ff/ # 256 shards total │ │ ├── 01/ ... ff/ # 256 shards total
│ └── metadata/ # Business data (sharded by UUID) │ └── metadata/ # Business data (sharded by UUID)
├── 00/ ├── 00/
│ │ ├── 00123456-....json # Entity metadata only │ │ ├── 00123456-....json # Entity metadata only
│ │ └── 00abcdef-....json │ │ └── 00abcdef-....json
├── 01/ ... ff/ ├── 01/ ... ff/
│ │
└── verbs/ │ └── verbs/
├── vectors/ # Relationship vectors (sharded) │ ├── vectors/ # Relationship vectors (sharded)
├── 00/ ... ff/ ├── 00/ ... ff/
│ │
└── metadata/ # Relationship data (sharded) │ └── metadata/ # Relationship data (sharded)
├── 00/ ... ff/ │ ├── 00/ ... ff/
``` ```
### Why Split Metadata and Vectors? ### Why Split Metadata and Vectors?
@ -50,9 +50,9 @@ brainy-data/
**How it works:** **How it works:**
```typescript ```typescript
const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6" const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
const shard = uuid.substring(0, 2) // "3f" const shard = uuid.substring(0, 2) // "3f"
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json // Vector path: entities/nouns/vectors/3f/3fa85f64-....json
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json // Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
``` ```
@ -64,103 +64,103 @@ const shard = uuid.substring(0, 2) // "3f"
## Storage Adapters ## Storage Adapters
Brainy provides multiple storage adapters with identical APIs and v4.0.0 production features: Brainy provides multiple storage adapters with identical APIs and production features:
### FileSystem Storage (Node.js) ### FileSystem Storage (Node.js)
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'filesystem', type: 'filesystem',
path: './data', path: './data',
compression: true // v4.0.0: Gzip compression (60-80% space savings) compression: true // Gzip compression (60-80% space savings)
} }
}) })
``` ```
- **Use case**: Server applications, CLI tools - **Use case**: Server applications, CLI tools
- **Performance**: Direct file I/O with optional compression - **Performance**: Direct file I/O with optional compression
- **Persistence**: Permanent on disk - **Persistence**: Permanent on disk
- **v4.0.0 Features**: - **Features**:
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead - **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
- **Batch Delete**: Efficient bulk deletion with retries - **Batch Delete**: Efficient bulk deletion with retries
- **UUID Sharding**: Automatic 256-shard distribution - **UUID Sharding**: Automatic 256-shard distribution
### S3 Compatible Storage (AWS, MinIO, R2) ### S3 Compatible Storage (AWS, MinIO, R2)
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3', type: 's3',
bucket: 'my-brainy-data', bucket: 'my-brainy-data',
region: 'us-east-1', region: 'us-east-1',
credentials: { credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID, accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
} }
} }
}) })
``` ```
- **Use case**: Distributed applications, cloud deployments - **Use case**: Distributed applications, cloud deployments
- **Performance**: Network dependent, with intelligent caching - **Performance**: Network dependent, with intelligent caching
- **Persistence**: Cloud storage durability (99.999999999%) - **Persistence**: Cloud storage durability (99.999999999%)
- **v4.0.0 Features**: - **Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive) - **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings) - **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
- **Batch Delete**: Efficient bulk deletion (1000 objects per request) - **Batch Delete**: Efficient bulk deletion (1000 objects per request)
- **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!) - **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!)
### Google Cloud Storage (GCS) ### Google Cloud Storage (GCS)
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'gcs', type: 'gcs',
bucketName: 'my-brainy-data', bucketName: 'my-brainy-data',
keyFilename: './service-account.json' // Or use ADC keyFilename: './service-account.json' // Or use ADC
} }
}) })
``` ```
- **Use case**: Google Cloud deployments - **Use case**: Google Cloud deployments
- **Performance**: Global CDN with edge caching - **Performance**: Global CDN with edge caching
- **Persistence**: 99.999999999% durability - **Persistence**: 99.999999999% durability
- **v4.0.0 Features**: - **Features**:
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive) - **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
- **Autoclass**: Intelligent automatic tier optimization - **Autoclass**: Intelligent automatic tier optimization
- **Batch Delete**: Efficient bulk operations - **Batch Delete**: Efficient bulk operations
- **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!) - **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!)
### Azure Blob Storage ### Azure Blob Storage
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'azure', type: 'azure',
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING, connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data' containerName: 'brainy-data'
} }
}) })
``` ```
- **Use case**: Azure cloud deployments - **Use case**: Azure cloud deployments
- **Performance**: Global replication with CDN - **Performance**: Global replication with CDN
- **Persistence**: LRS, ZRS, GRS, RA-GRS options - **Persistence**: LRS, ZRS, GRS, RA-GRS options
- **v4.0.0 Features**: - **Features**:
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings) - **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
- **Lifecycle Policies**: Automatic tier transitions and deletions - **Lifecycle Policies**: Automatic tier transitions and deletions
- **Batch Delete**: BlobBatchClient for efficient bulk operations - **Batch Delete**: BlobBatchClient for efficient bulk operations
- **Batch Tier Changes**: Move thousands of blobs efficiently - **Batch Tier Changes**: Move thousands of blobs efficiently
- **Archive Rehydration**: Smart rehydration with priority options - **Archive Rehydration**: Smart rehydration with priority options
### Origin Private File System (Browser) ### Origin Private File System (Browser)
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'opfs' type: 'opfs'
} }
}) })
``` ```
- **Use case**: Browser applications, PWAs - **Use case**: Browser applications, PWAs
- **Performance**: Near-native file system speed - **Performance**: Near-native file system speed
- **Persistence**: Permanent in browser (with quota limits) - **Persistence**: Permanent in browser (with quota limits)
- **v4.0.0 Features**: - **Features**:
- **Quota Monitoring**: Real-time quota tracking and warnings - **Quota Monitoring**: Real-time quota tracking and warnings
- **Batch Delete**: Efficient bulk deletion - **Batch Delete**: Efficient bulk deletion
- **Storage Status**: Detailed usage/available reporting - **Storage Status**: Detailed usage/available reporting
## Metadata Indexing System ## Metadata Indexing System
@ -170,12 +170,12 @@ Tracks all unique values for each field:
```json ```json
// __metadata_field_index__field_category.json // __metadata_field_index__field_category.json
{ {
"values": { "values": {
"technology": 45, "technology": 45,
"science": 32, "science": 32,
"business": 28 "business": 28
}, },
"lastUpdated": 1699564234567 "lastUpdated": 1699564234567
} }
``` ```
@ -185,11 +185,11 @@ Maps field+value combinations to entity IDs:
```json ```json
// __metadata_index__category_technology_chunk0.json // __metadata_index__category_technology_chunk0.json
{ {
"field": "category", "field": "category",
"value": "technology", "value": "technology",
"ids": ["uuid1", "uuid2", "uuid3", ...], "ids": ["uuid1", "uuid2", "uuid3", ...],
"chunk": 0, "chunk": 0,
"total": 45 "total": 45
} }
``` ```
@ -207,14 +207,14 @@ High-performance deduplication system for streaming data:
```json ```json
// __entity_registry__.json // __entity_registry__.json
{ {
"mappings": { "mappings": {
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000", "did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000" "handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
}, },
"stats": { "stats": {
"totalMappings": 10000, "totalMappings": 10000,
"lastSync": 1699564234567 "lastSync": 1699564234567
} }
} }
``` ```
@ -229,14 +229,14 @@ Ensures durability and enables recovery:
```json ```json
{ {
"timestamp": 1699564234567, "timestamp": 1699564234567,
"operation": "add", "operation": "add",
"data": { "data": {
"id": "550e8400-e29b-41d4-a716-446655440000", "id": "550e8400-e29b-41d4-a716-446655440000",
"content": "...", "content": "...",
"metadata": {} "metadata": {}
}, },
"checksum": "sha256:..." "checksum": "sha256:..."
} }
``` ```
@ -244,7 +244,7 @@ Ensures durability and enables recovery:
2. Replay operations from last checkpoint 2. Replay operations from last checkpoint
3. Verify checksums for integrity 3. Verify checksums for integrity
## Storage Optimization (v4.0.0) ## Storage Optimization
### 1. Lifecycle Policies (Cloud Storage) ### 1. Lifecycle Policies (Cloud Storage)
@ -253,48 +253,48 @@ Ensures durability and enables recovery:
```typescript ```typescript
// S3: Set lifecycle policy for automatic archival // S3: Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
id: 'archive-old-data', id: 'archive-old-data',
prefix: 'entities/', prefix: 'entities/',
status: 'Enabled', status: 'Enabled',
transitions: [ transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days { days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days
{ days: 90, storageClass: 'GLACIER' }, // Archive after 90 days { days: 90, storageClass: 'GLACIER' }, // Archive after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year { days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year
] ]
}] }]
}) })
// GCS: Set lifecycle policy // GCS: Set lifecycle policy
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
condition: { age: 30 }, condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' } action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, { }, {
condition: { age: 90 }, condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' } action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, { }, {
condition: { age: 365 }, condition: { age: 365 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' } action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}] }]
}) })
// Azure: Set lifecycle policy // Azure: Set lifecycle policy
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
name: 'archiveOldData', name: 'archiveOldData',
enabled: true, enabled: true,
type: 'Lifecycle', type: 'Lifecycle',
definition: { definition: {
filters: { blobTypes: ['blockBlob'] }, filters: { blobTypes: ['blockBlob'] },
actions: { actions: {
baseBlob: { baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 }, tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 } tierToArchive: { daysAfterModificationGreaterThan: 90 }
} }
} }
} }
}] }]
}) })
``` ```
@ -327,7 +327,7 @@ await storage.enableIntelligentTiering('entities/', 'auto-optimize')
```typescript ```typescript
// Enable GCS Autoclass // Enable GCS Autoclass
await storage.enableAutoclass({ await storage.enableAutoclass({
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
}) })
// Benefits: // Benefits:
@ -342,11 +342,11 @@ await storage.enableAutoclass({
```typescript ```typescript
// Enable gzip compression for local storage // Enable gzip compression for local storage
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'filesystem', type: 'filesystem',
path: './data', path: './data',
compression: true // 60-80% space savings compression: true // 60-80% space savings
} }
}) })
// Performance impact: // Performance impact:
@ -359,11 +359,11 @@ const brain = new Brainy({
### 5. Batch Operations ### 5. Batch Operations
```typescript ```typescript
// v4.0.0: Efficient batch delete // Efficient batch delete
await storage.batchDelete([ await storage.batchDelete([
'entities/nouns/vectors/00/00123456-....json', 'entities/nouns/vectors/00/00123456-....json',
'entities/nouns/metadata/00/00123456-....json', 'entities/nouns/metadata/00/00123456-....json',
// ... up to 1000 objects // ... up to 1000 objects
]) ])
// Benefits: // Benefits:
@ -375,9 +375,9 @@ await storage.batchDelete([
// Batch writes for performance // Batch writes for performance
await brain.addBatch([ await brain.addBatch([
{ content: "item1", metadata: {} }, { content: "item1", metadata: {} },
{ content: "item2", metadata: {} }, { content: "item2", metadata: {} },
{ content: "item3", metadata: {} } { content: "item3", metadata: {} }
]) ])
// Single transaction, optimized I/O // Single transaction, optimized I/O
``` ```
@ -390,14 +390,14 @@ const status = await storage.getStorageStatus()
console.log(status) console.log(status)
// { // {
// type: 'opfs', // type: 'opfs',
// available: true, // available: true,
// details: { // details: {
// usage: 45829120, // 43.7 MB used // usage: 45829120, // 43.7 MB used
// quota: 536870912, // 512 MB available // quota: 536870912, // 512 MB available
// usagePercent: 8.5, // usagePercent: 8.5,
// quotaExceeded: false // quotaExceeded: false
// } // }
// } // }
// Proactive quota management: // Proactive quota management:
@ -410,14 +410,14 @@ console.log(status)
```typescript ```typescript
// Change blob tier for cost optimization // Change blob tier for cost optimization
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings) await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings) await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings)
// Batch tier changes (efficient) // Batch tier changes (efficient)
await storage.batchChangeTier([blob1, blob2, blob3], 'Cool') await storage.batchChangeTier([blob1, blob2, blob3], 'Cool')
// Rehydrate from Archive when needed // Rehydrate from Archive when needed
await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
``` ```
### 8. Caching Strategy ### 8. Caching Strategy
@ -425,15 +425,15 @@ await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
```typescript ```typescript
// Configure caching per storage type // Configure caching per storage type
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'filesystem', type: 'filesystem',
cache: { cache: {
enabled: true, enabled: true,
maxSize: 1000, // Maximum cached items maxSize: 1000, // Maximum cached items
ttl: 300000, // 5 minutes ttl: 300000, // 5 minutes
strategy: 'lru' // Least recently used strategy: 'lru' // Least recently used
} }
} }
}) })
``` ```
@ -443,8 +443,8 @@ const brain = new Brainy({
```typescript ```typescript
// Automatic locking for write operations // Automatic locking for write operations
await brain.storage.withLock('resource-id', async () => { await brain.storage.withLock('resource-id', async () => {
// Exclusive access to resource // Exclusive access to resource
await brain.storage.saveNoun(id, data) await brain.storage.saveNoun(id, data)
}) })
``` ```
@ -459,9 +459,9 @@ await brain.storage.withLock('resource-id', async () => {
```typescript ```typescript
// Export entire database // Export entire database
const backup = await brain.export({ const backup = await brain.export({
format: 'json', format: 'json',
includeVectors: true, includeVectors: true,
includeIndexes: false includeIndexes: false
}) })
``` ```
@ -469,8 +469,8 @@ const backup = await brain.export({
```typescript ```typescript
// Import from backup // Import from backup
await brain.import(backup, { await brain.import(backup, {
mode: 'merge', // or 'replace' mode: 'merge', // or 'replace'
validateSchema: true validateSchema: true
}) })
``` ```
@ -514,15 +514,15 @@ await newBrain.import(data)
const stats = await brain.storage.getStatistics() const stats = await brain.storage.getStatistics()
console.log(stats) console.log(stats)
// { // {
// totalSize: 1048576, // totalSize: 1048576,
// entityCount: 1000, // entityCount: 1000,
// indexSize: 204800, // indexSize: 204800,
// walSize: 10240, // walSize: 10240,
// cacheHitRate: 0.85 // cacheHitRate: 0.85
// } // }
``` ```
## Best Practices (v4.0.0) ## Best Practices
### Choose the Right Adapter ### Choose the Right Adapter
1. **Development**: FileSystem with compression (local persistence, small storage footprint) 1. **Development**: FileSystem with compression (local persistence, small storage footprint)
@ -537,7 +537,7 @@ console.log(stats)
4. **Archival**: Cloud storage with lifecycle policies (96% cost savings!) 4. **Archival**: Cloud storage with lifecycle policies (96% cost savings!)
5. **Large-scale**: Metadata/vector separation + UUID sharding + lifecycle policies 5. **Large-scale**: Metadata/vector separation + UUID sharding + lifecycle policies
### v4.0.0 Cost Optimization ### Cost Optimization
1. **Enable lifecycle policies** for cloud storage (automated cost reduction) 1. **Enable lifecycle policies** for cloud storage (automated cost reduction)
2. **Use Intelligent-Tiering (S3)** or Autoclass (GCS) for automatic optimization 2. **Use Intelligent-Tiering (S3)** or Autoclass (GCS) for automatic optimization
3. **Enable compression** for FileSystem storage (60-80% space savings) 3. **Enable compression** for FileSystem storage (60-80% space savings)
@ -547,7 +547,7 @@ console.log(stats)
**Example Cost Savings (500TB dataset):** **Example Cost Savings (500TB dataset):**
- Without lifecycle policies: **$138,000/year** - Without lifecycle policies: **$138,000/year**
- With v4.0.0 lifecycle policies: **$5,940/year** - With lifecycle policies: **$5,940/year**
- **Savings: $132,060/year (96%)** - **Savings: $132,060/year (96%)**
### Monitor and Maintain ### Monitor and Maintain

View file

@ -1,8 +1,8 @@
# 🔌 Brainy v4.0.0 Augmentations Complete Reference # 🔌 Brainy Augmentations Complete Reference
> **All augmentations that power Brainy's extensibility - with locations, usage, and examples** > **All augmentations that power Brainy's extensibility - with locations, usage, and examples**
> >
> **⚠️ v4.0.0 Update**: Updated for metadata structure changes and billion-scale optimizations > **⚠️ Update**: Updated for metadata structure changes and billion-scale optimizations
## Quick Start ## Quick Start
@ -10,37 +10,37 @@
import { Brainy } from '@soulcraft/brainy' import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy({ const brain = new Brainy({
// Augmentations auto-configure based on environment // Augmentations auto-configure based on environment
storage: 'auto', // Storage augmentation storage: 'auto', // Storage augmentation
cache: true, // Cache augmentation cache: true, // Cache augmentation
index: true // Index augmentation index: true // Index augmentation
}) })
await brain.init() // Augmentations initialize automatically await brain.init() // Augmentations initialize automatically
``` ```
## v4.0.0 Augmentation Architecture ## Augmentation Architecture
### Key Improvements for Billion-Scale Performance ### Key Improvements for Billion-Scale Performance
1. **Metadata/Vector Separation**: Augmentations now work with separated metadata and vectors 1. **Metadata/Vector Separation**: Augmentations now work with separated metadata and vectors
- Metadata stored separately from vector data - Metadata stored separately from vector data
- 99.2% memory reduction for type tracking - 99.2% memory reduction for type tracking
- Two-file storage pattern for optimal I/O - Two-file storage pattern for optimal I/O
2. **Type System Enforcement**: All metadata requires type fields 2. **Type System Enforcement**: All metadata requires type fields
- `NounMetadata` requires `noun: NounType` - `NounMetadata` requires `noun: NounType`
- `VerbMetadata` requires `verb: VerbType` - `VerbMetadata` requires `verb: VerbType`
- Type inference system available as public API - Type inference system available as public API
3. **Storage Adapter Pattern**: Internal vs public method distinction 3. **Storage Adapter Pattern**: Internal vs public method distinction
- `_methods`: Return pure structures (HNSWNoun, HNSWVerb) - `_methods`: Return pure structures (HNSWNoun, HNSWVerb)
- Public methods: Return WithMetadata types - Public methods: Return WithMetadata types
- MetadataEnforcer Proxy ensures proper access - MetadataEnforcer Proxy ensures proper access
### What This Means for Augmentation Users ### What This Means for Augmentation Users
**✅ If you use built-in augmentations**: No changes needed! They're all updated for v4.0.0. **✅ If you use built-in augmentations**: No changes needed! They're all updated.
**⚠️ If you created custom storage augmentations**: Update your storage adapter to: **⚠️ If you created custom storage augmentations**: Update your storage adapter to:
- Wrap metadata with required `noun`/`verb` fields - Wrap metadata with required `noun`/`verb` fields
@ -82,7 +82,7 @@ const brain = new Brainy({ storage: 'memory' })
**Purpose**: Persistent file-based storage for Node.js applications **Purpose**: Persistent file-based storage for Node.js applications
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { type: 'filesystem', path: './data' } storage: { type: 'filesystem', path: './data' }
}) })
``` ```
@ -100,12 +100,12 @@ const brain = new Brainy({ storage: 'opfs' })
**Purpose**: AWS S3-compatible cloud storage **Purpose**: AWS S3-compatible cloud storage
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3', type: 's3',
bucket: 'my-bucket', bucket: 'my-bucket',
region: 'us-east-1', region: 'us-east-1',
credentials: { accessKeyId, secretAccessKey } credentials: { accessKeyId, secretAccessKey }
} }
}) })
``` ```
@ -115,12 +115,12 @@ const brain = new Brainy({
**Purpose**: Cloudflare R2 storage (S3-compatible) **Purpose**: Cloudflare R2 storage (S3-compatible)
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'r2', type: 'r2',
accountId: 'xxx', accountId: 'xxx',
bucket: 'my-bucket', bucket: 'my-bucket',
credentials: { accessKeyId, secretAccessKey } credentials: { accessKeyId, secretAccessKey }
} }
}) })
``` ```
@ -130,11 +130,11 @@ const brain = new Brainy({
**Purpose**: Google Cloud Storage **Purpose**: Google Cloud Storage
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'gcs', type: 'gcs',
bucket: 'my-bucket', bucket: 'my-bucket',
projectId: 'my-project' projectId: 'my-project'
} }
}) })
``` ```
@ -155,8 +155,8 @@ const brain = new Brainy({
**Auto-enabled**: When `cache: true` (default) **Auto-enabled**: When `cache: true` (default)
**Purpose**: LRU cache for search results and frequent queries **Purpose**: LRU cache for search results and frequent queries
```typescript ```typescript
brain.clearCache() // Exposed via API brain.clearCache() // Exposed via API
brain.getCacheStats() // Cache hit/miss statistics brain.getCacheStats() // Cache hit/miss statistics
``` ```
### IndexAugmentation ### IndexAugmentation
@ -174,7 +174,7 @@ brain.find({ where: { category: 'tech' } })
**Auto-enabled**: Always active **Auto-enabled**: Always active
**Purpose**: Performance metrics and statistics collection **Purpose**: Performance metrics and statistics collection
```typescript ```typescript
brain.getStats() // Comprehensive metrics brain.getStats() // Comprehensive metrics
``` ```
### MonitoringAugmentation ### MonitoringAugmentation
@ -187,7 +187,7 @@ brain.getStats() // Comprehensive metrics
**Auto-enabled**: For batch operations **Auto-enabled**: For batch operations
**Purpose**: Optimizes bulk add/update/delete operations **Purpose**: Optimizes bulk add/update/delete operations
```typescript ```typescript
brain.addNouns([...]) // Automatically batched brain.addNouns([...]) // Automatically batched
``` ```
### RequestDeduplicatorAugmentation ### RequestDeduplicatorAugmentation
@ -235,8 +235,8 @@ brain.add(data) // Automatically deduplicated
**Purpose**: AI-powered smart data import **Purpose**: AI-powered smart data import
```typescript ```typescript
const result = await brain.neuralImport(data, { const result = await brain.neuralImport(data, {
confidenceThreshold: 0.7, confidenceThreshold: 0.7,
autoApply: true autoApply: true
}) })
// Automatically detects entities and relationships // Automatically detects entities and relationships
``` ```
@ -293,8 +293,8 @@ await conduit.establishConnection('ws://other-brain')
```typescript ```typescript
// Example: NotionSynapse, SlackSynapse, etc. // Example: NotionSynapse, SlackSynapse, etc.
class NotionSynapse extends SynapseAugmentation { class NotionSynapse extends SynapseAugmentation {
async fetchData() { /* Notion API calls */ } async fetchData() { /* Notion API calls */ }
async pushData() { /* Sync to Notion */ } async pushData() { /* Sync to Notion */ }
} }
``` ```
@ -309,11 +309,11 @@ class NotionSynapse extends SynapseAugmentation {
### Auto-Configuration ### Auto-Configuration
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
// These auto-register augmentations: // These auto-register augmentations:
storage: 'auto', // Storage augmentation storage: 'auto', // Storage augmentation
cache: true, // Cache augmentation cache: true, // Cache augmentation
index: true, // Index augmentation index: true, // Index augmentation
metrics: true // Metrics augmentation metrics: true // Metrics augmentation
}) })
``` ```
@ -333,29 +333,29 @@ await brain.init()
import { BaseAugmentation } from '@soulcraft/brainy' import { BaseAugmentation } from '@soulcraft/brainy'
class MyAugmentation extends BaseAugmentation { class MyAugmentation extends BaseAugmentation {
readonly name = 'my-augmentation' readonly name = 'my-augmentation'
readonly timing = 'after' // before | after | both readonly timing = 'after' // before | after | both
readonly operations = ['addNoun', 'search'] // Which ops to hook readonly operations = ['addNoun', 'search'] // Which ops to hook
readonly priority = 10 // Execution order (lower = earlier) readonly priority = 10 // Execution order (lower = earlier)
protected async onInit(): Promise<void> { protected async onInit(): Promise<void> {
// Initialize your augmentation // Initialize your augmentation
} }
async execute<T>( async execute<T>(
operation: string, operation: string,
params: any, params: any,
context?: AugmentationContext context?: AugmentationContext
): Promise<T | void> { ): Promise<T | void> {
// Your augmentation logic // Your augmentation logic
if (operation === 'addNoun') { if (operation === 'addNoun') {
console.log('Noun added:', params) console.log('Noun added:', params)
} }
} }
protected async onShutdown(): Promise<void> { protected async onShutdown(): Promise<void> {
// Cleanup // Cleanup
} }
} }
``` ```

View file

@ -1,10 +1,10 @@
# 🛠️ Brainy Augmentation Developer Guide # 🛠️ Brainy Augmentation Developer Guide
> **How to create, test, and use augmentations in Brainy v4.0.0** > **How to create, test, and use augmentations in Brainy**
> >
> **⚠️ v4.0.0 Update**: This guide has been updated with breaking changes for metadata structure and type system improvements. > **⚠️ Update**: This guide has been updated with breaking changes for metadata structure and type system improvements.
## v4.0.0 Migration Guide ## Migration Guide
### What Changed? ### What Changed?
@ -25,22 +25,22 @@
```typescript ```typescript
// ❌ v3.x // ❌ v3.x
const verb = { const verb = {
type: 'relatedTo', type: 'relatedTo',
sourceId: 'a', sourceId: 'a',
targetId: 'b' targetId: 'b'
} }
if (verb.type === 'relatedTo') { ... } if (verb.type === 'relatedTo') { ... }
// ✅ v4.0.0 // ✅ Current
const verb = { const verb = {
verb: 'relatedTo', verb: 'relatedTo',
sourceId: 'a', sourceId: 'a',
targetId: 'b' targetId: 'b'
} }
const metadata: VerbMetadata = { const metadata: VerbMetadata = {
verb: 'relatedTo', verb: 'relatedTo',
sourceId: 'a', sourceId: 'a',
targetId: 'b' targetId: 'b'
} }
if (verb.verb === 'relatedTo') { ... } if (verb.verb === 'relatedTo') { ... }
``` ```
@ -51,40 +51,40 @@ if (verb.verb === 'relatedTo') { ... }
import { BaseAugmentation, BrainyAugmentation, AugmentationContext } from '@soulcraft/brainy' import { BaseAugmentation, BrainyAugmentation, AugmentationContext } from '@soulcraft/brainy'
export class MyFirstAugmentation extends BaseAugmentation { export class MyFirstAugmentation extends BaseAugmentation {
readonly name = 'my-first-augmentation' readonly name = 'my-first-augmentation'
readonly timing = 'after' as const // When to run: before | after | both readonly timing = 'after' as const // When to run: before | after | both
readonly operations = ['add'] as const // Which operations to hook readonly operations = ['add'] as const // Which operations to hook
readonly priority = 10 // Execution order (lower = first) readonly priority = 10 // Execution order (lower = first)
protected async onInit(): Promise<void> { protected async onInit(): Promise<void> {
// Initialize your augmentation // Initialize your augmentation
console.log('MyFirstAugmentation initialized!') console.log('MyFirstAugmentation initialized!')
} }
async execute<T = any>( async execute<T = any>(
operation: string, operation: string,
params: any, params: any,
context?: AugmentationContext context?: AugmentationContext
): Promise<T | void> { ): Promise<T | void> {
// Your augmentation logic // Your augmentation logic
if (operation === 'add') { if (operation === 'add') {
console.log('Noun added:', params.noun) console.log('Noun added:', params.noun)
// v4.0.0: Access metadata correctly // Access metadata correctly
if (params.noun?.metadata) { if (params.noun?.metadata) {
console.log('Noun type:', params.noun.metadata.noun) // Required field console.log('Noun type:', params.noun.metadata.noun) // Required field
} }
// You can access the brain instance // You can access the brain instance
const stats = await context?.brain.getStats() const stats = await context?.brain.getStats()
console.log('Total nouns:', stats.totalNouns) console.log('Total nouns:', stats.totalNouns)
} }
} }
protected async onShutdown(): Promise<void> { protected async onShutdown(): Promise<void> {
// Cleanup // Cleanup
console.log('MyFirstAugmentation shutting down') console.log('MyFirstAugmentation shutting down')
} }
} }
``` ```
@ -113,23 +113,23 @@ await brain.add('Hello World')
### 1. Registration Phase ### 1. Registration Phase
```typescript ```typescript
const aug = new MyAugmentation() const aug = new MyAugmentation()
brain.augmentations.register(aug) // Before brain.init()! brain.augmentations.register(aug) // Before brain.init()!
``` ```
### 2. Initialization Phase ### 2. Initialization Phase
```typescript ```typescript
await brain.init() // Calls aug.initialize() internally await brain.init() // Calls aug.initialize() internally
// Your onInit() method runs here // Your onInit() method runs here
``` ```
### 3. Execution Phase ### 3. Execution Phase
```typescript ```typescript
await brain.add('data') // Your execute() method runs await brain.add('data') // Your execute() method runs
``` ```
### 4. Shutdown Phase ### 4. Shutdown Phase
```typescript ```typescript
await brain.shutdown() // Your onShutdown() method runs await brain.shutdown() // Your onShutdown() method runs
``` ```
--- ---
@ -139,52 +139,52 @@ await brain.shutdown() // Your onShutdown() method runs
### `before` - Modify Input ### `before` - Modify Input
```typescript ```typescript
class ValidationAugmentation extends BaseAugmentation { class ValidationAugmentation extends BaseAugmentation {
readonly timing = 'before' as const readonly timing = 'before' as const
async execute<T>(operation: string, params: any): Promise<any> { async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') { if (operation === 'add') {
// Validate and/or modify params // Validate and/or modify params
if (!params.content) { if (!params.content) {
throw new Error('Content required') throw new Error('Content required')
} }
// Return modified params // Return modified params
return { ...params, validated: true } return { ...params, validated: true }
} }
} }
} }
``` ```
### `after` - React to Results ### `after` - React to Results
```typescript ```typescript
class LoggingAugmentation extends BaseAugmentation { class LoggingAugmentation extends BaseAugmentation {
readonly timing = 'after' as const readonly timing = 'after' as const
async execute<T>(operation: string, params: any): Promise<void> { async execute<T>(operation: string, params: any): Promise<void> {
if (operation === 'search') { if (operation === 'search') {
console.log(`Search for "${params.query}" returned ${params.result.length} results`) console.log(`Search for "${params.query}" returned ${params.result.length} results`)
} }
// Don't return anything - just observe // Don't return anything - just observe
} }
} }
``` ```
### `both` - Before AND After ### `both` - Before AND After
```typescript ```typescript
class TimingAugmentation extends BaseAugmentation { class TimingAugmentation extends BaseAugmentation {
readonly timing = 'both' as const readonly timing = 'both' as const
private startTime?: number private startTime?: number
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> { async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
if (!this.startTime) { if (!this.startTime) {
// Before execution // Before execution
this.startTime = Date.now() this.startTime = Date.now()
} else { } else {
// After execution // After execution
const duration = Date.now() - this.startTime const duration = Date.now() - this.startTime
console.log(`${operation} took ${duration}ms`) console.log(`${operation} took ${duration}ms`)
this.startTime = undefined this.startTime = undefined
} }
} }
} }
``` ```
@ -195,28 +195,28 @@ class TimingAugmentation extends BaseAugmentation {
### Core Operations You Can Hook ### Core Operations You Can Hook
```typescript ```typescript
readonly operations = [ readonly operations = [
'add', // Adding data 'add', // Adding data
'update', // Updating data 'update', // Updating data
'delete', // Deleting data 'delete', // Deleting data
'get', // Retrieving data 'get', // Retrieving data
'search', // Searching 'search', // Searching
'find', // Triple Intelligence queries 'find', // Triple Intelligence queries
'relate', // Adding relationships 'relate', // Adding relationships
'unrelate', // Removing relationships 'unrelate', // Removing relationships
'clear', // Clearing data 'clear', // Clearing data
'all' // Hook ALL operations 'all' // Hook ALL operations
] as const ] as const
``` ```
### Example: Multi-Operation Hook ### Example: Multi-Operation Hook
```typescript ```typescript
class AuditAugmentation extends BaseAugmentation { class AuditAugmentation extends BaseAugmentation {
readonly operations = ['add', 'update', 'delete'] as const readonly operations = ['add', 'update', 'delete'] as const
async execute<T>(operation: string, params: any): Promise<void> { async execute<T>(operation: string, params: any): Promise<void> {
// Log all data modifications // Log all data modifications
await this.logToAuditTrail(operation, params) await this.logToAuditTrail(operation, params)
} }
} }
``` ```
@ -226,26 +226,26 @@ class AuditAugmentation extends BaseAugmentation {
```typescript ```typescript
class ContextAwareAugmentation extends BaseAugmentation { class ContextAwareAugmentation extends BaseAugmentation {
async execute<T>( async execute<T>(
operation: string, operation: string,
params: any, params: any,
context?: AugmentationContext context?: AugmentationContext
): Promise<void> { ): Promise<void> {
// Access the brain instance // Access the brain instance
const brain = context?.brain const brain = context?.brain
if (!brain) return if (!brain) return
// Use any brain method // Use any brain method
const stats = await brain.getStats() const stats = await brain.getStats()
const size = await brain.size() const size = await brain.size()
const results = await brain.search('query') const results = await brain.search('query')
// Access other augmentations // Access other augmentations
const cache = brain.augmentations.get('cache') const cache = brain.augmentations.get('cache')
if (cache) { if (cache) {
await cache.clear() await cache.clear()
} }
} }
} }
``` ```
@ -256,92 +256,92 @@ class ContextAwareAugmentation extends BaseAugmentation {
### 1. Backup Augmentation ### 1. Backup Augmentation
```typescript ```typescript
class BackupAugmentation extends BaseAugmentation { class BackupAugmentation extends BaseAugmentation {
readonly name = 'backup' readonly name = 'backup'
readonly timing = 'after' as const readonly timing = 'after' as const
readonly operations = ['add', 'update', 'delete'] as const readonly operations = ['add', 'update', 'delete'] as const
readonly priority = 5 readonly priority = 5
private changes = 0 private changes = 0
private readonly backupThreshold = 100 private readonly backupThreshold = 100
async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> { async execute<T>(operation: string, params: any, context?: AugmentationContext): Promise<void> {
this.changes++ this.changes++
if (this.changes >= this.backupThreshold) { if (this.changes >= this.backupThreshold) {
await this.performBackup(context?.brain) await this.performBackup(context?.brain)
this.changes = 0 this.changes = 0
} }
} }
private async performBackup(brain?: any): Promise<void> { private async performBackup(brain?: any): Promise<void> {
if (!brain) return if (!brain) return
// Create instant COW snapshot // Create instant COW snapshot
const snapshotName = `backup-${Date.now()}` const snapshotName = `backup-${Date.now()}`
await brain.fork(snapshotName) await brain.fork(snapshotName)
console.log(`Automatic snapshot created: ${snapshotName}`) console.log(`Automatic snapshot created: ${snapshotName}`)
} }
} }
``` ```
### 2. Rate Limiting Augmentation ### 2. Rate Limiting Augmentation
```typescript ```typescript
class RateLimitAugmentation extends BaseAugmentation { class RateLimitAugmentation extends BaseAugmentation {
readonly name = 'rate-limit' readonly name = 'rate-limit'
readonly timing = 'before' as const readonly timing = 'before' as const
readonly operations = ['search', 'find'] as const readonly operations = ['search', 'find'] as const
readonly priority = 100 // High priority - run first readonly priority = 100 // High priority - run first
private requests = new Map<string, number[]>() private requests = new Map<string, number[]>()
private readonly limit = 100 // 100 requests private readonly limit = 100 // 100 requests
private readonly window = 60000 // per minute private readonly window = 60000 // per minute
async execute<T>(operation: string, params: any): Promise<void> { async execute<T>(operation: string, params: any): Promise<void> {
const now = Date.now() const now = Date.now()
const key = params.userId || 'anonymous' const key = params.userId || 'anonymous'
// Get request timestamps // Get request timestamps
const timestamps = this.requests.get(key) || [] const timestamps = this.requests.get(key) || []
// Remove old timestamps // Remove old timestamps
const recent = timestamps.filter(t => now - t < this.window) const recent = timestamps.filter(t => now - t < this.window)
// Check limit // Check limit
if (recent.length >= this.limit) { if (recent.length >= this.limit) {
throw new Error('Rate limit exceeded') throw new Error('Rate limit exceeded')
} }
// Add current request // Add current request
recent.push(now) recent.push(now)
this.requests.set(key, recent) this.requests.set(key, recent)
} }
} }
``` ```
### 3. Encryption Augmentation ### 3. Encryption Augmentation
```typescript ```typescript
class EncryptionAugmentation extends BaseAugmentation { class EncryptionAugmentation extends BaseAugmentation {
readonly name = 'encryption' readonly name = 'encryption'
readonly timing = 'both' as const readonly timing = 'both' as const
readonly operations = ['add', 'get'] as const readonly operations = ['add', 'get'] as const
readonly priority = 90 // Run early readonly priority = 90 // Run early
async execute<T>(operation: string, params: any): Promise<any> { async execute<T>(operation: string, params: any): Promise<any> {
if (operation === 'add') { if (operation === 'add') {
// Encrypt before storing // Encrypt before storing
if (params.metadata?.sensitive) { if (params.metadata?.sensitive) {
params.content = await this.encrypt(params.content) params.content = await this.encrypt(params.content)
params.encrypted = true params.encrypted = true
} }
return params return params
} }
if (operation === 'get' && params.result?.encrypted) { if (operation === 'get' && params.result?.encrypted) {
// Decrypt after retrieval // Decrypt after retrieval
params.result.content = await this.decrypt(params.result.content) params.result.content = await this.decrypt(params.result.content)
delete params.result.encrypted delete params.result.encrypted
return params.result return params.result
} }
} }
} }
``` ```
@ -355,26 +355,26 @@ import { Brainy } from '@soulcraft/brainy'
import { MyAugmentation } from './my-augmentation' import { MyAugmentation } from './my-augmentation'
describe('MyAugmentation', () => { describe('MyAugmentation', () => {
it('should hook into addNoun', async () => { it('should hook into addNoun', async () => {
const brain = new Brainy({ storage: 'memory' }) const brain = new Brainy({ storage: 'memory' })
const aug = new MyAugmentation() const aug = new MyAugmentation()
// Spy on the execute method // Spy on the execute method
const executeSpy = vi.spyOn(aug, 'execute') const executeSpy = vi.spyOn(aug, 'execute')
brain.augmentations.register(aug) brain.augmentations.register(aug)
await brain.init() await brain.init()
// Trigger the augmentation // Trigger the augmentation
await brain.add('test data') await brain.add('test data')
// Verify it was called // Verify it was called
expect(executeSpy).toHaveBeenCalledWith( expect(executeSpy).toHaveBeenCalledWith(
'add', 'add',
expect.objectContaining({ content: 'test data' }), expect.objectContaining({ content: 'test data' }),
expect.any(Object) expect.any(Object)
) )
}) })
}) })
``` ```
@ -391,61 +391,61 @@ describe('MyAugmentation', () => {
```typescript ```typescript
// Priority guidelines // Priority guidelines
100: Critical (auth, rate limiting) 100: Critical (auth, rate limiting)
50: Important (validation, transformation) 50: Important (validation, transformation)
10: Normal (logging, metrics) 10: Normal (logging, metrics)
1: Optional (debugging, tracing) 1: Optional (debugging, tracing)
``` ```
### 3. Handle Errors Gracefully ### 3. Handle Errors Gracefully
```typescript ```typescript
async execute<T>(operation: string, params: any): Promise<void> { async execute<T>(operation: string, params: any): Promise<void> {
try { try {
await this.riskyOperation() await this.riskyOperation()
} catch (error) { } catch (error) {
// Log but don't break the main operation // Log but don't break the main operation
console.error(`Augmentation error in ${this.name}:`, error) console.error(`Augmentation error in ${this.name}:`, error)
// Optionally report to monitoring // Optionally report to monitoring
this.reportError(error) this.reportError(error)
} }
} }
``` ```
### 4. Be Performance Conscious ### 4. Be Performance Conscious
```typescript ```typescript
class CachedAugmentation extends BaseAugmentation { class CachedAugmentation extends BaseAugmentation {
private cache = new Map<string, any>() private cache = new Map<string, any>()
async execute<T>(operation: string, params: any): Promise<any> { async execute<T>(operation: string, params: any): Promise<any> {
const key = this.getCacheKey(params) const key = this.getCacheKey(params)
// Check cache first // Check cache first
if (this.cache.has(key)) { if (this.cache.has(key)) {
return this.cache.get(key) return this.cache.get(key)
} }
// Expensive operation // Expensive operation
const result = await this.expensiveOperation(params) const result = await this.expensiveOperation(params)
this.cache.set(key, result) this.cache.set(key, result)
return result return result
} }
} }
``` ```
### 5. Clean Up Resources ### 5. Clean Up Resources
```typescript ```typescript
protected async onShutdown(): Promise<void> { protected async onShutdown(): Promise<void> {
// Close connections // Close connections
await this.connection?.close() await this.connection?.close()
// Clear intervals // Clear intervals
clearInterval(this.interval) clearInterval(this.interval)
// Flush buffers // Flush buffers
await this.flush() await this.flush()
// Clear caches // Clear caches
this.cache.clear() this.cache.clear()
} }
``` ```
@ -457,10 +457,10 @@ protected async onShutdown(): Promise<void> {
``` ```
my-augmentation/ my-augmentation/
├── src/ ├── src/
└── index.ts # Your augmentation │ └── index.ts # Your augmentation
├── dist/ # Built output ├── dist/ # Built output
├── tests/ ├── tests/
└── augmentation.test.ts │ └── augmentation.test.ts
├── package.json ├── package.json
├── tsconfig.json ├── tsconfig.json
└── README.md └── README.md
@ -469,21 +469,21 @@ my-augmentation/
### package.json ### package.json
```json ```json
{ {
"name": "@mycompany/brainy-custom-augmentation", "name": "@mycompany/brainy-custom-augmentation",
"version": "1.0.0", "version": "1.0.0",
"main": "dist/index.js", "main": "dist/index.js",
"types": "dist/index.d.ts", "types": "dist/index.d.ts",
"keywords": ["brainy-augmentation"], "keywords": ["brainy-augmentation"],
"peerDependencies": { "peerDependencies": {
"@soulcraft/brainy": ">=2.0.0" "@soulcraft/brainy": ">=2.0.0"
}, },
"brainy": { "brainy": {
"type": "augmentation", "type": "augmentation",
"class": "CustomAugmentation", "class": "CustomAugmentation",
"timing": "after", "timing": "after",
"operations": ["add"], "operations": ["add"],
"priority": 10 "priority": 10
} }
} }
``` ```
@ -492,7 +492,7 @@ my-augmentation/
# Coming in 2.1+ # Coming in 2.1+
npm run build npm run build
npm test npm test
brainy publish # Publishes to brain-cloud registry brainy publish # Publishes to brain-cloud registry
``` ```
--- ---

View file

@ -1,8 +1,8 @@
# Brainy v4.0.0 Cloud Deployment Guide # Brainy Cloud Deployment Guide
This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy v4.0.0 codebase. This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy codebase.
## 🆕 v4.0.0 Production Features ## 🆕 Production Features
**Cost Optimization at Scale:** **Cost Optimization at Scale:**
- **Lifecycle Policies**: Automatic tier transitions for 96% cost savings - **Lifecycle Policies**: Automatic tier transitions for 96% cost savings
@ -12,7 +12,7 @@ This guide provides production-ready deployment configurations for Brainy using
**Example Impact (500TB dataset):** **Example Impact (500TB dataset):**
- Before: $138,000/year - Before: $138,000/year
- With v4.0.0 lifecycle policies: $5,940/year - With lifecycle policies: $5,940/year
- **Savings: $132,060/year (96%)** - **Savings: $132,060/year (96%)**
See the [Cost Optimization](#cost-optimization-v40) section below for implementation details. See the [Cost Optimization](#cost-optimization-v40) section below for implementation details.
@ -43,53 +43,53 @@ let brain
let handler let handler
exports.handler = async (event) => { exports.handler = async (event) => {
if (!brain) { if (!brain) {
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com', endpoint: 's3.amazonaws.com',
region: process.env.AWS_REGION, region: process.env.AWS_REGION,
bucket: process.env.BRAINY_BUCKET, bucket: process.env.BRAINY_BUCKET,
accessKeyId: process.env.AWS_ACCESS_KEY_ID, accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
prefix: 'brainy-data/' prefix: 'brainy-data/'
}) })
brain = new Brainy({ brain = new Brainy({
storage, storage,
augmentations: [{ augmentations: [{
name: 'api-server', name: 'api-server',
config: { config: {
enabled: true, enabled: true,
auth: { auth: {
required: true, required: true,
apiKeys: [process.env.API_KEY] apiKeys: [process.env.API_KEY]
} }
} }
}] }]
}) })
await brain.init() await brain.init()
// Get the universal handler from the augmentation // Get the universal handler from the augmentation
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler() handler = apiAugmentation.createUniversalHandler()
} }
// Convert Lambda event to Request // Convert Lambda event to Request
const url = `https://${event.requestContext.domainName}${event.rawPath}` const url = `https://${event.requestContext.domainName}${event.rawPath}`
const request = new Request(url, { const request = new Request(url, {
method: event.requestContext.http.method, method: event.requestContext.http.method,
headers: event.headers, headers: event.headers,
body: event.body body: event.body
}) })
// Use the universal handler // Use the universal handler
const response = await handler(request) const response = await handler(request)
return { return {
statusCode: response.status, statusCode: response.status,
headers: Object.fromEntries(response.headers), headers: Object.fromEntries(response.headers),
body: await response.text() body: await response.text()
} }
} }
``` ```
@ -104,36 +104,36 @@ let brain
let handler let handler
exports.brainyAPI = async (req, res) => { exports.brainyAPI = async (req, res) => {
if (!brain) { if (!brain) {
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
endpoint: 'storage.googleapis.com', endpoint: 'storage.googleapis.com',
bucket: process.env.GCS_BUCKET, bucket: process.env.GCS_BUCKET,
accessKeyId: process.env.GCS_ACCESS_KEY, accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY, secretAccessKey: process.env.GCS_SECRET_KEY,
prefix: 'brainy/', prefix: 'brainy/',
forcePathStyle: false, forcePathStyle: false,
region: 'US' region: 'US'
}) })
brain = new Brainy({ storage }) brain = new Brainy({ storage })
await brain.init() await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler() handler = apiAugmentation.createUniversalHandler()
} }
// Convert Express req/res to Request/Response // Convert Express req/res to Request/Response
const request = new Request(`https://${req.hostname}${req.originalUrl}`, { const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
method: req.method, method: req.method,
headers: req.headers, headers: req.headers,
body: JSON.stringify(req.body) body: JSON.stringify(req.body)
}) })
const response = await handler(request) const response = await handler(request)
res.status(response.status) res.status(response.status)
res.set(Object.fromEntries(response.headers)) res.set(Object.fromEntries(response.headers))
res.send(await response.text()) res.send(await response.text())
} }
``` ```
@ -156,39 +156,39 @@ let brain
let handler let handler
async function initBrainy() { async function initBrainy() {
// Google Cloud Storage is S3-compatible // Google Cloud Storage is S3-compatible
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
endpoint: 'storage.googleapis.com', endpoint: 'storage.googleapis.com',
bucket: process.env.GCS_BUCKET || 'brainy-data', bucket: process.env.GCS_BUCKET || 'brainy-data',
accessKeyId: process.env.GCS_ACCESS_KEY, accessKeyId: process.env.GCS_ACCESS_KEY,
secretAccessKey: process.env.GCS_SECRET_KEY, secretAccessKey: process.env.GCS_SECRET_KEY,
prefix: 'brainy/', prefix: 'brainy/',
forcePathStyle: false, forcePathStyle: false,
region: process.env.GCS_REGION || 'US' region: process.env.GCS_REGION || 'US'
}) })
brain = new Brainy({ brain = new Brainy({
storage, storage,
augmentations: [{ augmentations: [{
name: 'api-server', name: 'api-server',
config: { config: {
enabled: true, enabled: true,
port: PORT, port: PORT,
cors: { cors: {
origin: process.env.CORS_ORIGIN || '*' origin: process.env.CORS_ORIGIN || '*'
}, },
auth: { auth: {
required: process.env.AUTH_REQUIRED === 'true', required: process.env.AUTH_REQUIRED === 'true',
apiKeys: process.env.API_KEY ? [process.env.API_KEY] : [] apiKeys: process.env.API_KEY ? [process.env.API_KEY] : []
} }
} }
}] }]
}) })
await brain.init() await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler() handler = apiAugmentation.createUniversalHandler()
} }
// Initialize on startup // Initialize on startup
@ -196,26 +196,26 @@ await initBrainy()
// Health check endpoint // Health check endpoint
app.get('/health', (req, res) => { app.get('/health', (req, res) => {
res.json({ status: 'healthy', service: 'brainy-api' }) res.json({ status: 'healthy', service: 'brainy-api' })
}) })
// Universal handler for all API routes // Universal handler for all API routes
app.use('*', async (req, res) => { app.use('*', async (req, res) => {
const request = new Request(`https://${req.hostname}${req.originalUrl}`, { const request = new Request(`https://${req.hostname}${req.originalUrl}`, {
method: req.method, method: req.method,
headers: req.headers, headers: req.headers,
body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined
}) })
const response = await handler(request) const response = await handler(request)
res.status(response.status) res.status(response.status)
res.set(Object.fromEntries(response.headers)) res.set(Object.fromEntries(response.headers))
res.send(await response.text()) res.send(await response.text())
}) })
app.listen(PORT, () => { app.listen(PORT, () => {
console.log(`Brainy API running on port ${PORT}`) console.log(`Brainy API running on port ${PORT}`)
}) })
``` ```
@ -237,43 +237,43 @@ CMD ["node", "server.js"]
```yaml ```yaml
# cloudbuild.yaml # cloudbuild.yaml
steps: steps:
# Build the container image # Build the container image
- name: 'gcr.io/cloud-builders/docker' - name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.'] args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.']
# Push to Container Registry # Push to Container Registry
- name: 'gcr.io/cloud-builders/docker' - name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'] args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA']
# Deploy to Cloud Run # Deploy to Cloud Run
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk' - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud entrypoint: gcloud
args: args:
- 'run' - 'run'
- 'deploy' - 'deploy'
- 'brainy-api' - 'brainy-api'
- '--image' - '--image'
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA' - 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
- '--region' - '--region'
- 'us-central1' - 'us-central1'
- '--platform' - '--platform'
- 'managed' - 'managed'
- '--allow-unauthenticated' - '--allow-unauthenticated'
- '--set-env-vars' - '--set-env-vars'
- 'GCS_BUCKET=brainy-storage,GCS_REGION=US' - 'GCS_BUCKET=brainy-storage,GCS_REGION=US'
- '--set-secrets' - '--set-secrets'
- 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest' - 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest'
- '--memory' - '--memory'
- '2Gi' - '2Gi'
- '--cpu' - '--cpu'
- '2' - '2'
- '--max-instances' - '--max-instances'
- '100' - '100'
- '--min-instances' - '--min-instances'
- '0' - '0'
images: images:
- 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA' - 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'
``` ```
**Deploy with gcloud CLI:** **Deploy with gcloud CLI:**
@ -283,12 +283,12 @@ gcloud builds submit --config cloudbuild.yaml
# Or deploy directly # Or deploy directly
gcloud run deploy brainy-api \ gcloud run deploy brainy-api \
--source . \ --source . \
--platform managed \ --platform managed \
--region us-central1 \ --region us-central1 \
--allow-unauthenticated \ --allow-unauthenticated \
--set-env-vars GCS_BUCKET=brainy-storage \ --set-env-vars GCS_BUCKET=brainy-storage \
--set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest" --set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest"
``` ```
**Create GCS Bucket with S3-compatible access:** **Create GCS Bucket with S3-compatible access:**
@ -312,37 +312,37 @@ echo -n "YOUR_SECRET_KEY" | gcloud secrets create gcs-secret-key --data-file=-
```javascript ```javascript
// index.js // index.js
module.exports = async function (context, req) { module.exports = async function (context, req) {
const { Brainy } = require('@soulcraft/brainy') const { Brainy } = require('@soulcraft/brainy')
const { S3CompatibleStorage } = require('@soulcraft/brainy/storage') const { S3CompatibleStorage } = require('@soulcraft/brainy/storage')
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`, endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`,
bucket: 'brainy-data', bucket: 'brainy-data',
accessKeyId: process.env.AZURE_STORAGE_ACCOUNT, accessKeyId: process.env.AZURE_STORAGE_ACCOUNT,
secretAccessKey: process.env.AZURE_STORAGE_KEY, secretAccessKey: process.env.AZURE_STORAGE_KEY,
prefix: 'entities/', prefix: 'entities/',
forcePathStyle: false forcePathStyle: false
}) })
const brain = new Brainy({ storage }) const brain = new Brainy({ storage })
await brain.init() await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
const handler = apiAugmentation.createUniversalHandler() const handler = apiAugmentation.createUniversalHandler()
const request = new Request(`https://${context.req.headers.host}${context.req.url}`, { const request = new Request(`https://${context.req.headers.host}${context.req.url}`, {
method: context.req.method, method: context.req.method,
headers: context.req.headers, headers: context.req.headers,
body: JSON.stringify(context.req.body) body: JSON.stringify(context.req.body)
}) })
const response = await handler(request) const response = await handler(request)
context.res = { context.res = {
status: response.status, status: response.status,
headers: Object.fromEntries(response.headers), headers: Object.fromEntries(response.headers),
body: await response.text() body: await response.text()
} }
} }
``` ```
@ -356,37 +356,37 @@ import { R2Storage } from '@soulcraft/brainy/storage' // Alias for S3CompatibleS
let handler let handler
export default { export default {
async fetch(request, env, ctx) { async fetch(request, env, ctx) {
if (!handler) { if (!handler) {
const storage = new R2Storage({ const storage = new R2Storage({
endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`, endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'brainy-data', bucket: 'brainy-data',
accessKeyId: env.R2_ACCESS_KEY_ID, accessKeyId: env.R2_ACCESS_KEY_ID,
secretAccessKey: env.R2_SECRET_ACCESS_KEY, secretAccessKey: env.R2_SECRET_ACCESS_KEY,
region: 'auto', region: 'auto',
forcePathStyle: true forcePathStyle: true
}) })
const brain = new Brainy({ const brain = new Brainy({
storage, storage,
augmentations: [{ augmentations: [{
name: 'api-server', name: 'api-server',
config: { config: {
enabled: true, enabled: true,
cors: { origin: '*' } cors: { origin: '*' }
} }
}] }]
}) })
await brain.init() await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler() handler = apiAugmentation.createUniversalHandler()
} }
// The handler works directly with Request/Response! // The handler works directly with Request/Response!
return handler(request) return handler(request)
} }
} }
``` ```
@ -418,52 +418,52 @@ import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
let handler let handler
export const config = { export const config = {
runtime: 'edge', runtime: 'edge',
} }
export default async (request) => { export default async (request) => {
if (!handler) { if (!handler) {
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com', endpoint: 's3.amazonaws.com',
region: 'us-east-1', region: 'us-east-1',
bucket: process.env.S3_BUCKET, bucket: process.env.S3_BUCKET,
accessKeyId: process.env.AWS_ACCESS_KEY_ID, accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}) })
const brain = new Brainy({ const brain = new Brainy({
storage, storage,
augmentations: [{ augmentations: [{
name: 'api-server', name: 'api-server',
config: { enabled: true } config: { enabled: true }
}] }]
}) })
await brain.init() await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler() handler = apiAugmentation.createUniversalHandler()
} }
return handler(request) return handler(request)
} }
``` ```
```json ```json
// vercel.json // vercel.json
{ {
"functions": { "functions": {
"api/brainy.js": { "api/brainy.js": {
"maxDuration": 30, "maxDuration": 30,
"memory": 1024 "memory": 1024
} }
}, },
"rewrites": [ "rewrites": [
{ {
"source": "/api/:path*", "source": "/api/:path*",
"destination": "/api/brainy" "destination": "/api/brainy"
} }
] ]
} }
``` ```
@ -482,51 +482,51 @@ let brain
let handler let handler
async function init() { async function init() {
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com', endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com',
region: process.env.S3_REGION || 'us-east-1', region: process.env.S3_REGION || 'us-east-1',
bucket: process.env.S3_BUCKET, bucket: process.env.S3_BUCKET,
accessKeyId: process.env.S3_ACCESS_KEY, accessKeyId: process.env.S3_ACCESS_KEY,
secretAccessKey: process.env.S3_SECRET_KEY, secretAccessKey: process.env.S3_SECRET_KEY,
prefix: 'brainy/' prefix: 'brainy/'
}) })
brain = new Brainy({ brain = new Brainy({
storage, storage,
augmentations: [{ augmentations: [{
name: 'api-server', name: 'api-server',
config: { config: {
enabled: true, enabled: true,
port: PORT port: PORT
} }
}] }]
}) })
await brain.init() await brain.init()
const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server')
handler = apiAugmentation.createUniversalHandler() handler = apiAugmentation.createUniversalHandler()
} }
await init() await init()
// Universal handler for all routes // Universal handler for all routes
app.use('*', async (req, res) => { app.use('*', async (req, res) => {
const request = new Request(`http://localhost${req.originalUrl}`, { const request = new Request(`http://localhost${req.originalUrl}`, {
method: req.method, method: req.method,
headers: req.headers, headers: req.headers,
body: JSON.stringify(req.body) body: JSON.stringify(req.body)
}) })
const response = await handler(request) const response = await handler(request)
res.status(response.status) res.status(response.status)
res.set(Object.fromEntries(response.headers)) res.set(Object.fromEntries(response.headers))
res.send(await response.text()) res.send(await response.text())
}) })
app.listen(PORT, () => { app.listen(PORT, () => {
console.log(`Brainy API running on port ${PORT}`) console.log(`Brainy API running on port ${PORT}`)
}) })
``` ```
@ -552,26 +552,26 @@ restartPolicyMaxRetries = 3
```yaml ```yaml
# render.yaml # render.yaml
services: services:
- type: web - type: web
name: brainy-api name: brainy-api
runtime: node runtime: node
buildCommand: npm install buildCommand: npm install
startCommand: node server.js startCommand: node server.js
envVars: envVars:
- key: S3_BUCKET - key: S3_BUCKET
value: brainy-data value: brainy-data
- key: S3_ENDPOINT - key: S3_ENDPOINT
value: s3.amazonaws.com value: s3.amazonaws.com
- key: S3_REGION - key: S3_REGION
value: us-east-1 value: us-east-1
- key: S3_ACCESS_KEY - key: S3_ACCESS_KEY
sync: false sync: false
- key: S3_SECRET_KEY - key: S3_SECRET_KEY
sync: false sync: false
- key: API_KEY - key: API_KEY
generateValue: true generateValue: true
healthCheckPath: /health healthCheckPath: /health
autoDeploy: true autoDeploy: true
``` ```
### Deno Deploy ### Deno Deploy
@ -582,19 +582,19 @@ import { Brainy } from "npm:@soulcraft/brainy"
import { S3CompatibleStorage } from "npm:@soulcraft/brainy/storage" import { S3CompatibleStorage } from "npm:@soulcraft/brainy/storage"
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com", endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com",
bucket: Deno.env.get("S3_BUCKET")!, bucket: Deno.env.get("S3_BUCKET")!,
accessKeyId: Deno.env.get("S3_ACCESS_KEY")!, accessKeyId: Deno.env.get("S3_ACCESS_KEY")!,
secretAccessKey: Deno.env.get("S3_SECRET_KEY")!, secretAccessKey: Deno.env.get("S3_SECRET_KEY")!,
region: "auto" region: "auto"
}) })
const brain = new Brainy({ const brain = new Brainy({
storage, storage,
augmentations: [{ augmentations: [{
name: 'api-server', name: 'api-server',
config: { enabled: true } config: { enabled: true }
}] }]
}) })
await brain.init() await brain.init()
@ -653,29 +653,29 @@ RATE_LIMIT_MAX=100
```javascript ```javascript
// REST API Client // REST API Client
const response = await fetch('https://your-api.com/api/brainy/add', { const response = await fetch('https://your-api.com/api/brainy/add', {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY' 'Authorization': 'Bearer YOUR_API_KEY'
}, },
body: JSON.stringify({ body: JSON.stringify({
data: 'Your content here', data: 'Your content here',
metadata: { type: 'document' } metadata: { type: 'document' }
}) })
}) })
const { id } = await response.json() const { id } = await response.json()
// Search // Search
const searchResponse = await fetch('https://your-api.com/api/brainy/find', { const searchResponse = await fetch('https://your-api.com/api/brainy/find', {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
'Authorization': 'Bearer YOUR_API_KEY' 'Authorization': 'Bearer YOUR_API_KEY'
}, },
body: JSON.stringify({ body: JSON.stringify({
query: 'neural networks' query: 'neural networks'
}) })
}) })
const results = await searchResponse.json() const results = await searchResponse.json()
@ -683,14 +683,14 @@ const results = await searchResponse.json()
// WebSocket Client // WebSocket Client
const ws = new WebSocket('wss://your-api.com/ws') const ws = new WebSocket('wss://your-api.com/ws')
ws.onopen = () => { ws.onopen = () => {
ws.send(JSON.stringify({ ws.send(JSON.stringify({
type: 'subscribe', type: 'subscribe',
pattern: 'technology' pattern: 'technology'
})) }))
} }
ws.onmessage = (event) => { ws.onmessage = (event) => {
const update = JSON.parse(event.data) const update = JSON.parse(event.data)
console.log('Real-time update:', update) console.log('Real-time update:', update)
} }
``` ```
@ -700,13 +700,13 @@ S3CompatibleStorage constructor parameters (verified from source):
```javascript ```javascript
{ {
endpoint: string, // Required (e.g., 's3.amazonaws.com') endpoint: string, // Required (e.g., 's3.amazonaws.com')
bucket: string, // Required bucket: string, // Required
accessKeyId: string, // Required accessKeyId: string, // Required
secretAccessKey: string, // Required secretAccessKey: string, // Required
region?: string, // Optional (default: 'us-east-1') region?: string, // Optional (default: 'us-east-1')
prefix?: string, // Optional (e.g., 'brainy/') prefix?: string, // Optional (e.g., 'brainy/')
forcePathStyle?: boolean, // Optional (needed for some S3-compatible services) forcePathStyle?: boolean, // Optional (needed for some S3-compatible services)
} }
``` ```
@ -727,7 +727,7 @@ S3CompatibleStorage constructor parameters (verified from source):
4. **Implement rate limiting** to prevent abuse 4. **Implement rate limiting** to prevent abuse
5. **Configure CORS** appropriately for your use case 5. **Configure CORS** appropriately for your use case
## Cost Optimization (v4.0.0) ## Cost Optimization
### Enable Lifecycle Policies ### Enable Lifecycle Policies
@ -738,16 +738,16 @@ const storage = brain.storage
// Set lifecycle policy for automatic archival // Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
id: 'archive-old-data', id: 'archive-old-data',
prefix: 'entities/', prefix: 'entities/',
status: 'Enabled', status: 'Enabled',
transitions: [ transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // $0.0125/GB after 30 days { days: 30, storageClass: 'STANDARD_IA' }, // $0.0125/GB after 30 days
{ days: 90, storageClass: 'GLACIER' }, // $0.004/GB after 90 days { days: 90, storageClass: 'GLACIER' }, // $0.004/GB after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // $0.00099/GB after 1 year { days: 365, storageClass: 'DEEP_ARCHIVE' } // $0.00099/GB after 1 year
] ]
}] }]
}) })
// Or enable Intelligent-Tiering for hands-off optimization // Or enable Intelligent-Tiering for hands-off optimization
@ -765,14 +765,14 @@ await storage.enableIntelligentTiering('entities/', 'auto-optimize')
**Efficient bulk deletions:** **Efficient bulk deletions:**
```javascript ```javascript
// v4.0.0: Batch delete (1000 objects per request) // Batch delete (1000 objects per request)
const idsToDelete = [/* array of entity IDs */] const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => [ const paths = idsToDelete.flatMap(id => [
`entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`, `entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`,
`entities/nouns/metadata/${id.substring(0, 2)}/${id}.json` `entities/nouns/metadata/${id.substring(0, 2)}/${id}.json`
]) ])
await storage.batchDelete(paths) // Much faster than individual deletes await storage.batchDelete(paths) // Much faster than individual deletes
``` ```
### Enable Compression (FileSystem) ### Enable Compression (FileSystem)
@ -780,8 +780,8 @@ await storage.batchDelete(paths) // Much faster than individual deletes
**For local/server deployments:** **For local/server deployments:**
```javascript ```javascript
const storage = new FileSystemStorage({ const storage = new FileSystemStorage({
path: './data', path: './data',
compression: true // 60-80% space savings compression: true // 60-80% space savings
}) })
const brain = new Brainy({ storage }) const brain = new Brainy({ storage })
@ -793,8 +793,8 @@ const brain = new Brainy({ storage })
2. **Use S3CompatibleStorage** for cloud deployments (better scalability) 2. **Use S3CompatibleStorage** for cloud deployments (better scalability)
3. **Enable the cache augmentation** for frequently accessed data 3. **Enable the cache augmentation** for frequently accessed data
4. **Configure appropriate memory limits** for your runtime 4. **Configure appropriate memory limits** for your runtime
5. **v4.0.0**: Enable lifecycle policies to reduce storage costs by 96% 5. Enable lifecycle policies to reduce storage costs by 96%
6. **v4.0.0**: Use batch operations for cleanup tasks 6. Use batch operations for cleanup tasks
## Troubleshooting ## Troubleshooting
@ -810,15 +810,15 @@ const brain = new Brainy({ storage })
Enable debug logging by setting: Enable debug logging by setting:
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
storage, storage,
debug: true, debug: true,
augmentations: [{ augmentations: [{
name: 'api-server', name: 'api-server',
config: { config: {
enabled: true, enabled: true,
verbose: true verbose: true
} }
}] }]
}) })
``` ```

View file

@ -18,24 +18,24 @@ const { Brainy } = require('@soulcraft/brainy')
let brain let brain
exports.handler = async (event) => { exports.handler = async (event) => {
// Brainy auto-detects Lambda environment and configures accordingly // Brainy auto-detects Lambda environment and configures accordingly
if (!brain) { if (!brain) {
brain = new Brainy() // Zero config - auto-adapts to Lambda brain = new Brainy() // Zero config - auto-adapts to Lambda
await brain.init() await brain.init()
} }
const { method, ...params } = JSON.parse(event.body) const { method, ...params } = JSON.parse(event.body)
switch(method) { switch(method) {
case 'add': case 'add':
const id = await brain.add(params) const id = await brain.add(params)
return { statusCode: 200, body: JSON.stringify({ id }) } return { statusCode: 200, body: JSON.stringify({ id }) }
case 'find': case 'find':
const results = await brain.find(params) const results = await brain.find(params)
return { statusCode: 200, body: JSON.stringify({ results }) } return { statusCode: 200, body: JSON.stringify({ results }) }
default: default:
return { statusCode: 400, body: 'Unknown method' } return { statusCode: 400, body: 'Unknown method' }
} }
} }
EOF EOF
@ -56,26 +56,26 @@ docker push $ECR_URI/brainy:latest
# Deploy with minimal ECS task definition # Deploy with minimal ECS task definition
cat > task-definition.json << 'EOF' cat > task-definition.json << 'EOF'
{ {
"family": "brainy", "family": "brainy",
"networkMode": "awsvpc", "networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"], "requiresCompatibilities": ["FARGATE"],
"cpu": "256", "cpu": "256",
"memory": "512", "memory": "512",
"containerDefinitions": [{ "containerDefinitions": [{
"name": "brainy", "name": "brainy",
"image": "$ECR_URI/brainy:latest", "image": "$ECR_URI/brainy:latest",
"environment": [ "environment": [
{"name": "NODE_ENV", "value": "production"} {"name": "NODE_ENV", "value": "production"}
], ],
"logConfiguration": { "logConfiguration": {
"logDriver": "awslogs", "logDriver": "awslogs",
"options": { "options": {
"awslogs-group": "/ecs/brainy", "awslogs-group": "/ecs/brainy",
"awslogs-region": "us-east-1", "awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs" "awslogs-stream-prefix": "ecs"
} }
} }
}] }]
} }
EOF EOF
@ -138,14 +138,14 @@ const brain = new Brainy()
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3', type: 's3',
options: { options: {
bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket
region: process.env.AWS_REGION || 'auto', // 'auto' detects region region: process.env.AWS_REGION || 'auto', // 'auto' detects region
// IAM role provides credentials automatically // IAM role provides credentials automatically
} }
} }
}) })
``` ```
@ -156,23 +156,23 @@ const brain = new Brainy({
```yaml ```yaml
# Auto-scaling policy # Auto-scaling policy
Resources: Resources:
AutoScalingTarget: AutoScalingTarget:
Type: AWS::ApplicationAutoScaling::ScalableTarget Type: AWS::ApplicationAutoScaling::ScalableTarget
Properties: Properties:
ServiceNamespace: ecs ServiceNamespace: ecs
ResourceId: service/default/brainy ResourceId: service/default/brainy
ScalableDimension: ecs:service:DesiredCount ScalableDimension: ecs:service:DesiredCount
MinCapacity: 2 MinCapacity: 2
MaxCapacity: 100 MaxCapacity: 100
AutoScalingPolicy: AutoScalingPolicy:
Type: AWS::ApplicationAutoScaling::ScalingPolicy Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties: Properties:
PolicyType: TargetTrackingScaling PolicyType: TargetTrackingScaling
TargetTrackingScalingPolicyConfiguration: TargetTrackingScalingPolicyConfiguration:
TargetValue: 70.0 TargetValue: 70.0
PredefinedMetricSpecification: PredefinedMetricSpecification:
PredefinedMetricType: ECSServiceAverageCPUUtilization PredefinedMetricType: ECSServiceAverageCPUUtilization
``` ```
### 2. Vertical Scaling ### 2. Vertical Scaling
@ -189,10 +189,10 @@ Brainy automatically adapts to available memory:
```javascript ```javascript
// Brainy automatically handles multi-AZ with S3 // Brainy automatically handles multi-AZ with S3
const brain = new Brainy({ const brain = new Brainy({
distributed: { distributed: {
enabled: true, // Auto-enables with S3 storage enabled: true, // Auto-enables with S3 storage
coordinationMethod: 'auto' // Uses S3 for coordination coordinationMethod: 'auto' // Uses S3 for coordination
} }
}) })
``` ```
@ -201,17 +201,17 @@ const brain = new Brainy({
```bash ```bash
# Application Load Balancer with health checks # Application Load Balancer with health checks
aws elbv2 create-load-balancer \ aws elbv2 create-load-balancer \
--name brainy-alb \ --name brainy-alb \
--subnets subnet-xxx subnet-yyy \ --subnets subnet-xxx subnet-yyy \
--security-groups sg-xxx --security-groups sg-xxx
aws elbv2 create-target-group \ aws elbv2 create-target-group \
--name brainy-targets \ --name brainy-targets \
--protocol HTTP \ --protocol HTTP \
--port 3000 \ --port 3000 \
--vpc-id vpc-xxx \ --vpc-id vpc-xxx \
--health-check-path /health \ --health-check-path /health \
--health-check-interval-seconds 30 --health-check-interval-seconds 30
``` ```
## Monitoring & Observability ## Monitoring & Observability
@ -233,16 +233,16 @@ Brainy automatically sends metrics when running on AWS:
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
monitoring: { monitoring: {
enabled: true, enabled: true,
customMetrics: { customMetrics: {
namespace: 'Brainy/Production', namespace: 'Brainy/Production',
dimensions: [ dimensions: [
{ Name: 'Environment', Value: 'production' }, { Name: 'Environment', Value: 'production' },
{ Name: 'Service', Value: 'api' } { Name: 'Service', Value: 'api' }
] ]
} }
} }
}) })
``` ```
@ -252,22 +252,22 @@ const brain = new Brainy({
```json ```json
{ {
"Version": "2012-10-17", "Version": "2012-10-17",
"Statement": [ "Statement": [
{ {
"Effect": "Allow", "Effect": "Allow",
"Action": [ "Action": [
"s3:GetObject", "s3:GetObject",
"s3:PutObject", "s3:PutObject",
"s3:DeleteObject", "s3:DeleteObject",
"s3:ListBucket" "s3:ListBucket"
], ],
"Resource": [ "Resource": [
"arn:aws:s3:::brainy-*/*", "arn:aws:s3:::brainy-*/*",
"arn:aws:s3:::brainy-*" "arn:aws:s3:::brainy-*"
] ]
} }
] ]
} }
``` ```
@ -285,12 +285,12 @@ aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.2.0/24 --availability-z
```javascript ```javascript
// Automatic encryption with S3 // Automatic encryption with S3
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3', type: 's3',
options: { options: {
encryption: 'auto' // Uses S3 SSE-S3 by default encryption: 'auto' // Uses S3 SSE-S3 by default
} }
} }
}) })
``` ```
@ -300,14 +300,14 @@ const brain = new Brainy({
```bash ```bash
aws ec2 request-spot-fleet --spot-fleet-request-config '{ aws ec2 request-spot-fleet --spot-fleet-request-config '{
"IamFleetRole": "arn:aws:iam::xxx:role/fleet-role", "IamFleetRole": "arn:aws:iam::xxx:role/fleet-role",
"TargetCapacity": 2, "TargetCapacity": 2,
"SpotPrice": "0.05", "SpotPrice": "0.05",
"LaunchSpecifications": [{ "LaunchSpecifications": [{
"ImageId": "ami-xxx", "ImageId": "ami-xxx",
"InstanceType": "t3.medium", "InstanceType": "t3.medium",
"UserData": "BASE64_ENCODED_STARTUP_SCRIPT" "UserData": "BASE64_ENCODED_STARTUP_SCRIPT"
}] }]
}' }'
``` ```
@ -316,12 +316,12 @@ aws ec2 request-spot-fleet --spot-fleet-request-config '{
```javascript ```javascript
// Brainy automatically uses S3 Intelligent-Tiering // Brainy automatically uses S3 Intelligent-Tiering
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3', type: 's3',
options: { options: {
storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization
} }
} }
}) })
``` ```
@ -329,8 +329,8 @@ const brain = new Brainy({
```bash ```bash
aws lambda put-function-concurrency \ aws lambda put-function-concurrency \
--function-name brainy-handler \ --function-name brainy-handler \
--reserved-concurrent-executions 10 --reserved-concurrent-executions 10
``` ```
## Deployment Automation ## Deployment Automation
@ -340,28 +340,28 @@ aws lambda put-function-concurrency \
```yaml ```yaml
name: Deploy to AWS name: Deploy to AWS
on: on:
push: push:
branches: [main] branches: [main]
jobs: jobs:
deploy: deploy:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v2 - uses: actions/checkout@v2
- name: Configure AWS credentials - name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1 uses: aws-actions/configure-aws-credentials@v1
with: with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1 aws-region: us-east-1
- name: Deploy to ECS - name: Deploy to ECS
run: | run: |
docker build -t brainy . docker build -t brainy .
docker tag brainy:latest $ECR_URI/brainy:latest docker tag brainy:latest $ECR_URI/brainy:latest
docker push $ECR_URI/brainy:latest docker push $ECR_URI/brainy:latest
aws ecs update-service --cluster default --service brainy --force-new-deployment aws ecs update-service --cluster default --service brainy --force-new-deployment
``` ```
## Troubleshooting ## Troubleshooting
@ -369,121 +369,121 @@ jobs:
### Common Issues ### Common Issues
1. **Storage Auto-Detection Fails** 1. **Storage Auto-Detection Fails**
```javascript ```javascript
// Explicitly specify storage // Explicitly specify storage
const brain = new Brainy({ const brain = new Brainy({
storage: { type: 's3', options: { bucket: 'my-bucket' } } storage: { type: 's3', options: { bucket: 'my-bucket' } }
}) })
``` ```
2. **Memory Issues** 2. **Memory Issues**
```javascript ```javascript
// Optimize for low memory // Optimize for low memory
const brain = new Brainy({ const brain = new Brainy({
cache: { maxSize: 100 }, // Reduce cache size cache: { maxSize: 100 }, // Reduce cache size
index: { M: 8 } // Reduce HNSW connections index: { M: 8 } // Reduce HNSW connections
}) })
``` ```
3. **Cold Starts (Lambda)** 3. **Cold Starts (Lambda)**
**v7.3.0+ Progressive Initialization (Zero-Config)** **Progressive Initialization (Zero-Config)**
Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME) Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME)
and uses progressive initialization for <200ms cold starts: and uses progressive initialization for <200ms cold starts:
```javascript ```javascript
// Zero-config - Brainy auto-detects Lambda // Zero-config - Brainy auto-detects Lambda
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3', type: 's3',
s3Storage: { s3Storage: {
bucketName: 'my-bucket', bucketName: 'my-bucket',
region: 'us-east-1', region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID, accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
} }
} }
}) })
await brain.init() // Returns in <200ms await brain.init() // Returns in <200ms
// First write validates bucket (lazy validation) // First write validates bucket (lazy validation)
await brain.add('noun', { name: 'test' }) // Validates here await brain.add('noun', { name: 'test' }) // Validates here
``` ```
**Manual Override (if needed)** **Manual Override (if needed)**
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3', type: 's3',
s3Storage: { s3Storage: {
bucketName: 'my-bucket', bucketName: 'my-bucket',
// Force specific mode // Force specific mode
initMode: 'progressive' // 'auto' | 'progressive' | 'strict' initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
} }
} }
}) })
``` ```
| Mode | Cold Start | Best For | | Mode | Cold Start | Best For |
|------|------------|----------| |------|------------|----------|
| `auto` (default) | <200ms in Lambda | Zero-config, auto-detects | | `auto` (default) | <200ms in Lambda | Zero-config, auto-detects |
| `progressive` | <200ms always | Force fast init everywhere | | `progressive` | <200ms always | Force fast init everywhere |
| `strict` | 100-500ms+ | Local dev, tests, debugging | | `strict` | 100-500ms+ | Local dev, tests, debugging |
**Warm-up (Alternative)** **Warm-up (Alternative)**
```javascript ```javascript
exports.warmup = async () => { exports.warmup = async () => {
if (!brain) { if (!brain) {
brain = new Brainy({ warmup: true }) brain = new Brainy({ warmup: true })
await brain.init() await brain.init()
} }
} }
``` ```
**Readiness Detection (v7.3.0+)** **Readiness Detection**
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests: Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
```javascript ```javascript
let brain let brain
exports.handler = async (event) => { exports.handler = async (event) => {
if (!brain) { if (!brain) {
brain = new Brainy({ storage: { type: 's3', ... } }) brain = new Brainy({ storage: { type: 's3', ... } })
brain.init() // Fire and forget brain.init() // Fire and forget
} }
// Wait for initialization to complete // Wait for initialization to complete
await brain.ready await brain.ready
// Now safe to use brain methods // Now safe to use brain methods
const results = await brain.find({ query: event.queryStringParameters.q }) const results = await brain.find({ query: event.queryStringParameters.q })
return { statusCode: 200, body: JSON.stringify(results) } return { statusCode: 200, body: JSON.stringify(results) }
} }
``` ```
For health checks, use `isFullyInitialized()` to verify all background tasks are complete: For health checks, use `isFullyInitialized()` to verify all background tasks are complete:
```javascript ```javascript
exports.healthCheck = async () => { exports.healthCheck = async () => {
try { try {
await brain.ready await brain.ready
return { return {
statusCode: 200, statusCode: 200,
body: JSON.stringify({ body: JSON.stringify({
status: 'ready', status: 'ready',
fullyInitialized: brain.isFullyInitialized() fullyInitialized: brain.isFullyInitialized()
}) })
} }
} catch (error) { } catch (error) {
return { return {
statusCode: 503, statusCode: 503,
body: JSON.stringify({ status: 'initializing' }) body: JSON.stringify({ status: 'initializing' })
} }
} }
} }
``` ```
## Production Checklist ## Production Checklist

View file

@ -10,10 +10,10 @@ Deploy Brainy on GCP with automatic scaling, global distribution, and zero-confi
```bash ```bash
# Build and deploy with one command # Build and deploy with one command
gcloud run deploy brainy \ gcloud run deploy brainy \
--source . \ --source . \
--platform managed \ --platform managed \
--region us-central1 \ --region us-central1 \
--allow-unauthenticated --allow-unauthenticated
# Brainy auto-detects Cloud Run and configures: # Brainy auto-detects Cloud Run and configures:
# - Memory-optimized caching # - Memory-optimized caching
@ -30,44 +30,44 @@ const { Brainy } = require('@soulcraft/brainy')
let brain let brain
exports.brainyHandler = async (req, res) => { exports.brainyHandler = async (req, res) => {
// Zero-config - auto-adapts to Cloud Functions // Zero-config - auto-adapts to Cloud Functions
if (!brain) { if (!brain) {
brain = new Brainy() // Detects GCP environment automatically brain = new Brainy() // Detects GCP environment automatically
await brain.init() await brain.init()
} }
const { method, ...params } = req.body const { method, ...params } = req.body
try { try {
let result let result
switch(method) { switch(method) {
case 'add': case 'add':
result = await brain.add(params) result = await brain.add(params)
break break
case 'find': case 'find':
result = await brain.find(params) result = await brain.find(params)
break break
case 'relate': case 'relate':
result = await brain.relate(params) result = await brain.relate(params)
break break
default: default:
return res.status(400).json({ error: 'Unknown method' }) return res.status(400).json({ error: 'Unknown method' })
} }
res.json({ result }) res.json({ result })
} catch (error) { } catch (error) {
res.status(500).json({ error: error.message }) res.status(500).json({ error: error.message })
} }
} }
``` ```
Deploy: Deploy:
```bash ```bash
gcloud functions deploy brainy \ gcloud functions deploy brainy \
--runtime nodejs20 \ --runtime nodejs20 \
--trigger-http \ --trigger-http \
--entry-point brainyHandler \ --entry-point brainyHandler \
--memory 512MB \ --memory 512MB \
--timeout 60s --timeout 60s
``` ```
### Option 3: Google Kubernetes Engine (GKE) ### Option 3: Google Kubernetes Engine (GKE)
@ -75,7 +75,7 @@ gcloud functions deploy brainy \
```bash ```bash
# Create autopilot cluster (fully managed, zero-config) # Create autopilot cluster (fully managed, zero-config)
gcloud container clusters create-auto brainy-cluster \ gcloud container clusters create-auto brainy-cluster \
--region us-central1 --region us-central1
# Deploy using Cloud Build # Deploy using Cloud Build
gcloud builds submit --tag gcr.io/$PROJECT_ID/brainy gcloud builds submit --tag gcr.io/$PROJECT_ID/brainy
@ -85,39 +85,39 @@ kubectl apply -f - <<EOF
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
metadata: metadata:
name: brainy name: brainy
spec: spec:
replicas: 3 replicas: 3
selector: selector:
matchLabels: matchLabels:
app: brainy app: brainy
template: template:
metadata: metadata:
labels: labels:
app: brainy app: brainy
spec: spec:
containers: containers:
- name: brainy - name: brainy
image: gcr.io/$PROJECT_ID/brainy image: gcr.io/$PROJECT_ID/brainy
resources: resources:
requests: requests:
memory: "512Mi" memory: "512Mi"
cpu: "250m" cpu: "250m"
env: env:
- name: NODE_ENV - name: NODE_ENV
value: production value: production
--- ---
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
metadata: metadata:
name: brainy-service name: brainy-service
spec: spec:
type: LoadBalancer type: LoadBalancer
selector: selector:
app: brainy app: brainy
ports: ports:
- port: 80 - port: 80
targetPort: 3000 targetPort: 3000
EOF EOF
``` ```
@ -139,14 +139,14 @@ const brain = new Brainy()
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3', // GCS is S3-compatible type: 's3', // GCS is S3-compatible
options: { options: {
endpoint: 'https://storage.googleapis.com', endpoint: 'https://storage.googleapis.com',
bucket: process.env.GCS_BUCKET || 'auto', // Auto-creates bucket bucket: process.env.GCS_BUCKET || 'auto', // Auto-creates bucket
// Uses Application Default Credentials automatically // Uses Application Default Credentials automatically
} }
} }
}) })
``` ```
@ -154,13 +154,13 @@ const brain = new Brainy({
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'firestore', type: 'firestore',
options: { options: {
projectId: process.env.GCP_PROJECT || 'auto', projectId: process.env.GCP_PROJECT || 'auto',
collection: 'brainy-data' collection: 'brainy-data'
} }
} }
}) })
``` ```
@ -173,24 +173,24 @@ const brain = new Brainy({
apiVersion: serving.knative.dev/v1 apiVersion: serving.knative.dev/v1
kind: Service kind: Service
metadata: metadata:
name: brainy name: brainy
annotations: annotations:
run.googleapis.com/execution-environment: gen2 run.googleapis.com/execution-environment: gen2
spec: spec:
template: template:
metadata: metadata:
annotations: annotations:
autoscaling.knative.dev/minScale: "1" autoscaling.knative.dev/minScale: "1"
autoscaling.knative.dev/maxScale: "1000" autoscaling.knative.dev/maxScale: "1000"
autoscaling.knative.dev/target: "80" autoscaling.knative.dev/target: "80"
spec: spec:
containerConcurrency: 100 containerConcurrency: 100
containers: containers:
- image: gcr.io/PROJECT_ID/brainy - image: gcr.io/PROJECT_ID/brainy
resources: resources:
limits: limits:
cpu: "2" cpu: "2"
memory: "2Gi" memory: "2Gi"
``` ```
### 2. GKE Horizontal Pod Autoscaling ### 2. GKE Horizontal Pod Autoscaling
@ -199,27 +199,27 @@ spec:
apiVersion: autoscaling/v2 apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler kind: HorizontalPodAutoscaler
metadata: metadata:
name: brainy-hpa name: brainy-hpa
spec: spec:
scaleTargetRef: scaleTargetRef:
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
name: brainy name: brainy
minReplicas: 3 minReplicas: 3
maxReplicas: 100 maxReplicas: 100
metrics: metrics:
- type: Resource - type: Resource
resource: resource:
name: cpu name: cpu
target: target:
type: Utilization type: Utilization
averageUtilization: 70 averageUtilization: 70
- type: Resource - type: Resource
resource: resource:
name: memory name: memory
target: target:
type: Utilization type: Utilization
averageUtilization: 80 averageUtilization: 80
``` ```
## Global Distribution ## Global Distribution
@ -229,11 +229,11 @@ spec:
```javascript ```javascript
// Brainy automatically handles multi-region with GCS // Brainy automatically handles multi-region with GCS
const brain = new Brainy({ const brain = new Brainy({
distributed: { distributed: {
enabled: true, enabled: true,
regions: ['us-central1', 'europe-west1', 'asia-northeast1'], regions: ['us-central1', 'europe-west1', 'asia-northeast1'],
replication: 'auto' // Automatic cross-region replication replication: 'auto' // Automatic cross-region replication
} }
}) })
``` ```
@ -242,14 +242,14 @@ const brain = new Brainy({
```bash ```bash
# Global load balancing with Traffic Director # Global load balancing with Traffic Director
gcloud compute backend-services create brainy-global \ gcloud compute backend-services create brainy-global \
--global \ --global \
--load-balancing-scheme=INTERNAL_SELF_MANAGED \ --load-balancing-scheme=INTERNAL_SELF_MANAGED \
--protocol=HTTP2 --protocol=HTTP2
gcloud compute backend-services add-backend brainy-global \ gcloud compute backend-services add-backend brainy-global \
--global \ --global \
--network-endpoint-group=brainy-neg \ --network-endpoint-group=brainy-neg \
--network-endpoint-group-region=us-central1 --network-endpoint-group-region=us-central1
``` ```
## Monitoring & Observability ## Monitoring & Observability
@ -277,22 +277,22 @@ const { Monitoring } = require('@google-cloud/monitoring')
const monitoring = new Monitoring.MetricServiceClient() const monitoring = new Monitoring.MetricServiceClient()
const brain = new Brainy({ const brain = new Brainy({
onMetric: async (metric) => { onMetric: async (metric) => {
// Send custom metrics to Cloud Monitoring // Send custom metrics to Cloud Monitoring
await monitoring.createTimeSeries({ await monitoring.createTimeSeries({
name: monitoring.projectPath(projectId), name: monitoring.projectPath(projectId),
timeSeries: [{ timeSeries: [{
metric: { metric: {
type: `custom.googleapis.com/brainy/${metric.name}`, type: `custom.googleapis.com/brainy/${metric.name}`,
labels: metric.labels labels: metric.labels
}, },
points: [{ points: [{
interval: { endTime: { seconds: Date.now() / 1000 } }, interval: { endTime: { seconds: Date.now() / 1000 } },
value: { doubleValue: metric.value } value: { doubleValue: metric.value }
}] }]
}] }]
}) })
} }
}) })
``` ```
@ -300,10 +300,10 @@ const brain = new Brainy({
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
tracing: { tracing: {
enabled: true, enabled: true,
sampleRate: 0.1 // Sample 10% of requests sampleRate: 0.1 // Sample 10% of requests
} }
}) })
``` ```
@ -316,9 +316,9 @@ const brain = new Brainy({
apiVersion: v1 apiVersion: v1
kind: ServiceAccount kind: ServiceAccount
metadata: metadata:
name: brainy-sa name: brainy-sa
annotations: annotations:
iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com
``` ```
### 2. Binary Authorization ### 2. Binary Authorization
@ -328,11 +328,11 @@ metadata:
apiVersion: binaryauthorization.grafeas.io/v1beta1 apiVersion: binaryauthorization.grafeas.io/v1beta1
kind: Policy kind: Policy
metadata: metadata:
name: brainy-policy name: brainy-policy
spec: spec:
defaultAdmissionRule: defaultAdmissionRule:
requireAttestationsBy: requireAttestationsBy:
- projects/PROJECT_ID/attestors/prod-attestor - projects/PROJECT_ID/attestors/prod-attestor
``` ```
### 3. VPC Service Controls ### 3. VPC Service Controls
@ -340,9 +340,9 @@ spec:
```bash ```bash
# Create VPC Service Perimeter # Create VPC Service Perimeter
gcloud access-context-manager perimeters create brainy_perimeter \ gcloud access-context-manager perimeters create brainy_perimeter \
--resources=projects/PROJECT_NUMBER \ --resources=projects/PROJECT_NUMBER \
--restricted-services=storage.googleapis.com \ --restricted-services=storage.googleapis.com \
--title="Brainy Security Perimeter" --title="Brainy Security Perimeter"
``` ```
## Cost Optimization ## Cost Optimization
@ -354,16 +354,16 @@ gcloud access-context-manager perimeters create brainy_perimeter \
apiVersion: container.cnrm.cloud.google.com/v1beta1 apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerNodePool kind: ContainerNodePool
metadata: metadata:
name: brainy-preemptible-pool name: brainy-preemptible-pool
spec: spec:
clusterRef: clusterRef:
name: brainy-cluster name: brainy-cluster
config: config:
preemptible: true preemptible: true
machineType: n2-standard-2 machineType: n2-standard-2
autoscaling: autoscaling:
minNodeCount: 1 minNodeCount: 1
maxNodeCount: 10 maxNodeCount: 10
``` ```
### 2. Cloud CDN for Static Assets ### 2. Cloud CDN for Static Assets
@ -371,11 +371,11 @@ spec:
```bash ```bash
# Enable Cloud CDN for frequently accessed data # Enable Cloud CDN for frequently accessed data
gcloud compute backend-buckets create brainy-assets \ gcloud compute backend-buckets create brainy-assets \
--gcs-bucket-name=brainy-static --gcs-bucket-name=brainy-static
gcloud compute backend-buckets update brainy-assets \ gcloud compute backend-buckets update brainy-assets \
--enable-cdn \ --enable-cdn \
--cache-mode=CACHE_ALL_STATIC --cache-mode=CACHE_ALL_STATIC
``` ```
### 3. Committed Use Discounts ### 3. Committed Use Discounts
@ -383,8 +383,8 @@ gcloud compute backend-buckets update brainy-assets \
```bash ```bash
# Purchase committed use for predictable workloads # Purchase committed use for predictable workloads
gcloud compute commitments create brainy-commitment \ gcloud compute commitments create brainy-commitment \
--plan=TWELVE_MONTH \ --plan=TWELVE_MONTH \
--resources=vcpu=100,memory=400 --resources=vcpu=100,memory=400
``` ```
## Deployment Automation ## Deployment Automation
@ -394,28 +394,28 @@ gcloud compute commitments create brainy-commitment \
```yaml ```yaml
# cloudbuild.yaml # cloudbuild.yaml
steps: steps:
# Build container # Build container
- name: 'gcr.io/cloud-builders/docker' - name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.'] args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.']
# Push to registry # Push to registry
- name: 'gcr.io/cloud-builders/docker' - name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'] args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA']
# Deploy to Cloud Run # Deploy to Cloud Run
- name: 'gcr.io/cloud-builders/gcloud' - name: 'gcr.io/cloud-builders/gcloud'
args: args:
- 'run' - 'run'
- 'deploy' - 'deploy'
- 'brainy' - 'brainy'
- '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA' - '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'
- '--region=us-central1' - '--region=us-central1'
- '--platform=managed' - '--platform=managed'
# Trigger on push to main # Trigger on push to main
trigger: trigger:
branch: branch:
name: main name: main
``` ```
### Terraform Infrastructure ### Terraform Infrastructure
@ -423,40 +423,40 @@ trigger:
```hcl ```hcl
# main.tf # main.tf
resource "google_cloud_run_service" "brainy" { resource "google_cloud_run_service" "brainy" {
name = "brainy" name = "brainy"
location = "us-central1" location = "us-central1"
template { template {
spec { spec {
containers { containers {
image = "gcr.io/${var.project_id}/brainy" image = "gcr.io/${var.project_id}/brainy"
resources { resources {
limits = { limits = {
cpu = "2" cpu = "2"
memory = "2Gi" memory = "2Gi"
} }
} }
env { env {
name = "NODE_ENV" name = "NODE_ENV"
value = "production" value = "production"
} }
} }
} }
} }
traffic { traffic {
percent = 100 percent = 100
latest_revision = true latest_revision = true
} }
} }
resource "google_cloud_run_service_iam_member" "public" { resource "google_cloud_run_service_iam_member" "public" {
service = google_cloud_run_service.brainy.name service = google_cloud_run_service.brainy.name
location = google_cloud_run_service.brainy.location location = google_cloud_run_service.brainy.location
role = "roles/run.invoker" role = "roles/run.invoker"
member = "allUsers" member = "allUsers"
} }
``` ```
@ -467,13 +467,13 @@ resource "google_cloud_run_service_iam_member" "public" {
```javascript ```javascript
// Brainy can use Memorystore for caching // Brainy can use Memorystore for caching
const brain = new Brainy({ const brain = new Brainy({
cache: { cache: {
type: 'redis', type: 'redis',
options: { options: {
host: process.env.REDIS_HOST || 'auto-detect', host: process.env.REDIS_HOST || 'auto-detect',
port: 6379 port: 6379
} }
} }
}) })
``` ```
@ -481,13 +481,13 @@ const brain = new Brainy({
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
metadata: { metadata: {
type: 'spanner', type: 'spanner',
options: { options: {
instance: 'brainy-instance', instance: 'brainy-instance',
database: 'brainy-db' database: 'brainy-db'
} }
} }
}) })
``` ```
@ -496,116 +496,116 @@ const brain = new Brainy({
### Common Issues ### Common Issues
1. **Quota Exceeded** 1. **Quota Exceeded**
```bash ```bash
# Check quotas # Check quotas
gcloud compute project-info describe --project=$PROJECT_ID gcloud compute project-info describe --project=$PROJECT_ID
# Request increase # Request increase
gcloud compute project-info add-metadata \ gcloud compute project-info add-metadata \
--metadata google-compute-default-region=us-central1 --metadata google-compute-default-region=us-central1
``` ```
2. **Cold Starts** 2. **Cold Starts**
**v7.3.0+ Progressive Initialization (Zero-Config)** **Progressive Initialization (Zero-Config)**
Brainy automatically detects Cloud Run and Cloud Functions environments Brainy automatically detects Cloud Run and Cloud Functions environments
and uses progressive initialization for <200ms cold starts: and uses progressive initialization for <200ms cold starts:
```javascript ```javascript
// Zero-config - Brainy auto-detects Cloud Run (K_SERVICE env var) // Zero-config - Brainy auto-detects Cloud Run (K_SERVICE env var)
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'gcs', type: 'gcs',
gcsNativeStorage: { bucketName: 'my-bucket' } gcsNativeStorage: { bucketName: 'my-bucket' }
} }
}) })
await brain.init() // Returns in <200ms await brain.init() // Returns in <200ms
// First write validates bucket (lazy validation) // First write validates bucket (lazy validation)
await brain.add('noun', { name: 'test' }) // Validates here await brain.add('noun', { name: 'test' }) // Validates here
``` ```
**Manual Override (if needed)** **Manual Override (if needed)**
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'gcs', type: 'gcs',
gcsNativeStorage: { gcsNativeStorage: {
bucketName: 'my-bucket', bucketName: 'my-bucket',
// Force specific mode // Force specific mode
initMode: 'progressive' // 'auto' | 'progressive' | 'strict' initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
} }
} }
}) })
``` ```
| Mode | Cold Start | Best For | | Mode | Cold Start | Best For |
|------|------------|----------| |------|------------|----------|
| `auto` (default) | <200ms in cloud | Zero-config, auto-detects | | `auto` (default) | <200ms in cloud | Zero-config, auto-detects |
| `progressive` | <200ms always | Force fast init everywhere | | `progressive` | <200ms always | Force fast init everywhere |
| `strict` | 100-500ms+ | Local dev, tests, debugging | | `strict` | 100-500ms+ | Local dev, tests, debugging |
**Keep Warm (Alternative)** **Keep Warm (Alternative)**
```javascript ```javascript
// Keep minimum instances warm // Keep minimum instances warm
const brain = new Brainy({ const brain = new Brainy({
warmup: { warmup: {
enabled: true, enabled: true,
interval: 60000 // Ping every minute interval: 60000 // Ping every minute
} }
}) })
``` ```
**Readiness Detection (v7.3.0+)** **Readiness Detection**
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests: Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
```javascript ```javascript
let brain let brain
async function handleRequest(req, res) { async function handleRequest(req, res) {
if (!brain) { if (!brain) {
brain = new Brainy({ storage: { type: 'gcs', ... } }) brain = new Brainy({ storage: { type: 'gcs', ... } })
brain.init() // Fire and forget brain.init() // Fire and forget
} }
// Wait for initialization to complete // Wait for initialization to complete
await brain.ready await brain.ready
// Now safe to use brain methods // Now safe to use brain methods
const results = await brain.find({ query: req.query.q }) const results = await brain.find({ query: req.query.q })
res.json(results) res.json(results)
} }
``` ```
For Cloud Run health checks, use `isFullyInitialized()` to verify all background tasks are complete: For Cloud Run health checks, use `isFullyInitialized()` to verify all background tasks are complete:
```javascript ```javascript
// Health check endpoint for Cloud Run // Health check endpoint for Cloud Run
app.get('/health', async (req, res) => { app.get('/health', async (req, res) => {
try { try {
await brain.ready await brain.ready
res.json({ res.json({
status: 'ready', status: 'ready',
fullyInitialized: brain.isFullyInitialized() fullyInitialized: brain.isFullyInitialized()
}) })
} catch (error) { } catch (error) {
res.status(503).json({ status: 'initializing' }) res.status(503).json({ status: 'initializing' })
} }
}) })
``` ```
3. **Memory Pressure** 3. **Memory Pressure**
```javascript ```javascript
// Optimize for GCP memory constraints // Optimize for GCP memory constraints
const brain = new Brainy({ const brain = new Brainy({
memory: { memory: {
mode: 'aggressive', // Aggressive garbage collection mode: 'aggressive', // Aggressive garbage collection
maxHeap: 0.8 // Use 80% of available memory maxHeap: 0.8 // Use 80% of available memory
} }
}) })
``` ```
## Production Checklist ## Production Checklist

View file

@ -42,10 +42,10 @@ await experiment.updateAll(riskyTransformation)
// Works? Great! Use the experimental branch. // Works? Great! Use the experimental branch.
// Failed? Just discard. // Failed? Just discard.
if (success) { if (success) {
// Make experiment the new main branch // Make experiment the new main branch
await brain.checkout('test-migration') await brain.checkout('test-migration')
} else { } else {
await experiment.destroy() // No harm done await experiment.destroy() // No harm done
} }
``` ```
@ -61,25 +61,25 @@ if (success) {
Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking: Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking:
### Architecture (v5.0.0) ### Architecture
1. **HNSW Index COW** (The Performance Bottleneck): 1. **HNSW Index COW** (The Performance Bottleneck):
- **Shallow Copy**: O(1) Map reference copying (~10ms for 1M+ nodes) - **Shallow Copy**: O(1) Map reference copying (~10ms for 1M+ nodes)
- **Lazy Deep Copy**: Nodes copied only when modified (`ensureCOW()`) - **Lazy Deep Copy**: Nodes copied only when modified (`ensureCOW()`)
- **Write Isolation**: Fork modifications don't affect parent - **Write Isolation**: Fork modifications don't affect parent
- **Implementation**: `src/hnsw/hnswIndex.ts` lines 2088-2150 - **Implementation**: `src/hnsw/hnswIndex.ts` lines 2088-2150
2. **Metadata & Graph Indexes** (Fast Rebuild): 2. **Metadata & Graph Indexes** (Fast Rebuild):
- **Rebuild from Storage**: < 500ms total for both indexes - **Rebuild from Storage**: < 500ms total for both indexes
- **Shared Storage**: Both indexes read from COW-enabled storage layer - **Shared Storage**: Both indexes read from COW-enabled storage layer
- **Acceptable Overhead**: Fast enough not to need in-memory COW - **Acceptable Overhead**: Fast enough not to need in-memory COW
3. **Storage Layer** (Shared): 3. **Storage Layer** (Shared):
- **RefManager**: Manages branch references - **RefManager**: Manages branch references
- **BlobStorage**: Content-addressable with deduplication - **BlobStorage**: Content-addressable with deduplication
- **All Adapters**: Memory, FileSystem, S3, R2, GCS, Azure, OPFS - **All Adapters**: Memory, FileSystem, S3, R2, GCS, Azure, OPFS
**Performance (v5.0.0)**: **Performance**:
- **Fork time**: < 100ms @ 10K entities (MEASURED in tests) - **Fork time**: < 100ms @ 10K entities (MEASURED in tests)
- **Storage overhead**: 10-20% (shared blobs, only changed data duplicated) - **Storage overhead**: 10-20% (shared blobs, only changed data duplicated)
- **Memory overhead**: 10-20% (shared HNSW nodes, only modified nodes duplicated) - **Memory overhead**: 10-20% (shared HNSW nodes, only modified nodes duplicated)
@ -87,11 +87,11 @@ Brainy uses **Snowflake-style Copy-on-Write (COW)** for instant forking:
**Technical Details**: **Technical Details**:
```typescript ```typescript
// Shallow copy HNSW (instant) // Shallow copy HNSW (instant)
clone.index.enableCOW(this.index) // O(1) Map reference copy clone.index.enableCOW(this.index) // O(1) Map reference copy
// Fast rebuild small indexes from shared storage // Fast rebuild small indexes from shared storage
clone.metadataIndex = new MetadataIndexManager(clone.storage) // <100ms clone.metadataIndex = new MetadataIndexManager(clone.storage) // <100ms
clone.graphIndex = new GraphAdjacencyIndex(clone.storage) // <500ms clone.graphIndex = new GraphAdjacencyIndex(clone.storage) // <500ms
``` ```
--- ---
@ -128,11 +128,11 @@ await fork.add({ noun: 'user', data: { name: 'Charlie' } })
// All APIs work // All APIs work
const users = await fork.find({ noun: 'user' }) const users = await fork.find({ noun: 'user' })
console.log(users.length) // 3 (Alice, Bob, Charlie) console.log(users.length) // 3 (Alice, Bob, Charlie)
// Original brain unchanged // Original brain unchanged
const originalUsers = await brain.find({ noun: 'user' }) const originalUsers = await brain.find({ noun: 'user' })
console.log(originalUsers.length) // 2 (Alice, Bob) console.log(originalUsers.length) // 2 (Alice, Bob)
``` ```
### 3. Discard When Done ### 3. Discard When Done
@ -165,30 +165,30 @@ const migration = await brain.fork('migration-test')
// Run migration on fork // Run migration on fork
const users = await migration.find({ noun: 'user' }) const users = await migration.find({ noun: 'user' })
for (const user of users) { for (const user of users) {
await migration.update(user.id, { await migration.update(user.id, {
email: user.data.email.toLowerCase(), // Normalize emails email: user.data.email.toLowerCase(), // Normalize emails
verified: user.data.verified ?? false // Add missing field verified: user.data.verified ?? false // Add missing field
}) })
} }
// Validate migration // Validate migration
const allValid = (await migration.find({ noun: 'user' })) const allValid = (await migration.find({ noun: 'user' }))
.every(u => u.data.email === u.data.email.toLowerCase()) .every(u => u.data.email === u.data.email.toLowerCase())
if (allValid) { if (allValid) {
console.log('✅ Migration safe! Apply changes to production brain') console.log('✅ Migration safe! Apply changes to production brain')
// Apply validated migration to main brain // Apply validated migration to main brain
const users = await brain.find({ noun: 'user' }) const users = await brain.find({ noun: 'user' })
for (const user of users) { for (const user of users) {
await brain.update(user.id, { await brain.update(user.id, {
email: user.data.email.toLowerCase(), email: user.data.email.toLowerCase(),
verified: user.data.verified ?? false verified: user.data.verified ?? false
}) })
} }
await migration.destroy() await migration.destroy()
} else { } else {
console.log('❌ Migration failed! Discarding fork.') console.log('❌ Migration failed! Discarding fork.')
await migration.destroy() await migration.destroy()
} }
``` ```
@ -203,8 +203,8 @@ const brain = new Brainy({ storage: { adapter: 's3', bucket: 'production' } })
await brain.init() await brain.init()
// Create two variants // Create two variants
const variantA = await brain.fork('variant-a') // Control const variantA = await brain.fork('variant-a') // Control
const variantB = await brain.fork('variant-b') // Test const variantB = await brain.fork('variant-b') // Test
// Run different algorithms // Run different algorithms
await variantA.processWithAlgorithm('current') await variantA.processWithAlgorithm('current')
@ -219,14 +219,14 @@ console.log('Variant B accuracy:', metricsB.accuracy)
// Choose winner and update main brain // Choose winner and update main brain
if (metricsB.accuracy > metricsA.accuracy) { if (metricsB.accuracy > metricsA.accuracy) {
console.log('B wins! Apply algorithm B to production') console.log('B wins! Apply algorithm B to production')
await brain.processWithAlgorithm('improved') await brain.processWithAlgorithm('improved')
await variantA.destroy() await variantA.destroy()
await variantB.destroy() await variantB.destroy()
} else { } else {
console.log('A wins! Keeping current algorithm.') console.log('A wins! Keeping current algorithm.')
await variantA.destroy() await variantA.destroy()
await variantB.destroy() await variantB.destroy()
} }
``` ```
@ -311,12 +311,12 @@ await brain.update(docs[0].id, { version: 2 })
// Test against original state // Test against original state
const originalData = await snapshot.find({ noun: 'doc' }) const originalData = await snapshot.find({ noun: 'doc' })
console.log(originalData[0].data.version) // 1 (original state!) console.log(originalData[0].data.version) // 1 (original state!)
// Clean up // Clean up
await snapshot.destroy() await snapshot.destroy()
// Note: Time-travel queries (asOf) planned for v5.1.0 // Note: Time-travel queries (asOf) Planned
``` ```
--- ---
@ -330,27 +330,27 @@ await snapshot.destroy()
const fork1 = await brain.fork('my-experiment') const fork1 = await brain.fork('my-experiment')
// Auto-generated name (uses timestamp) // Auto-generated name (uses timestamp)
const fork2 = await brain.fork() // 'fork-1635789012345' const fork2 = await brain.fork() // 'fork-1635789012345'
// Fork with metadata (for tracking) // Fork with metadata (for tracking)
const fork3 = await brain.fork('test', { const fork3 = await brain.fork('test', {
author: 'Alice', author: 'Alice',
message: 'Testing new feature' message: 'Testing new feature'
}) })
``` ```
### Branch Management (v5.0.0) ### Branch Management
**NEW in v5.0.0:** Full branch management now available! Full branch management now available!
```javascript ```javascript
// List all branches // List all branches
const branches = await brain.listBranches() const branches = await brain.listBranches()
console.log(branches) // ['main', 'experiment', 'test'] console.log(branches) // ['main', 'experiment', 'test']
// Get current branch // Get current branch
const current = await brain.getCurrentBranch() const current = await brain.getCurrentBranch()
console.log(current) // 'main' console.log(current) // 'main'
// Switch between branches // Switch between branches
await brain.checkout('experiment') await brain.checkout('experiment')
@ -359,33 +359,33 @@ await brain.checkout('experiment')
await brain.deleteBranch('old-experiment') await brain.deleteBranch('old-experiment')
``` ```
### Commit Tracking (v5.0.0) ### Commit Tracking
**NEW in v5.0.0:** Git-style commit tracking! Git-style commit tracking!
```javascript ```javascript
// Create a commit (snapshot of current state) // Create a commit (snapshot of current state)
await brain.add({ type: 'user', data: { name: 'Alice' } }) await brain.add({ type: 'user', data: { name: 'Alice' } })
const commitHash = await brain.commit({ const commitHash = await brain.commit({
message: 'Add Alice user', message: 'Add Alice user',
author: 'dev@example.com' author: 'dev@example.com'
}) })
console.log(commitHash) // 'a3f2c1b9...' console.log(commitHash) // 'a3f2c1b9...'
``` ```
### Commit History (v5.0.0) ### Commit History
**NEW in v5.0.0:** View commit history! View commit history!
```javascript ```javascript
// Get commit history for current branch // Get commit history for current branch
const history = await brain.getHistory({ limit: 10 }) const history = await brain.getHistory({ limit: 10 })
history.forEach(commit => { history.forEach(commit => {
console.log(`${commit.hash}: ${commit.message}`) console.log(`${commit.hash}: ${commit.message}`)
console.log(` By: ${commit.author} at ${new Date(commit.timestamp)}`) console.log(` By: ${commit.author} at ${new Date(commit.timestamp)}`)
}) })
## Performance Characteristics ## Performance Characteristics
@ -425,7 +425,7 @@ Savings: 40% less memory
## Zero Configuration ## Zero Configuration
**Fork is enabled by default in v5.0.0+. No setup required.** **Fork is enabled by default. No setup required.**
```javascript ```javascript
// This is all you need: // This is all you need:
@ -467,9 +467,9 @@ await new Brainy({ storage: { adapter: 's3', bucket: 'my-data' } }).fork()
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
storage: { adapter: 'memory' }, storage: { adapter: 'memory' },
vfs: { enabled: true }, vfs: { enabled: true },
intelligence: { enabled: true } intelligence: { enabled: true }
}) })
await brain.init() await brain.init()
@ -484,9 +484,9 @@ await brain.add({ noun: 'user', data: { name: 'Alice' } })
const fork = await brain.fork('test') const fork = await brain.fork('test')
// All features work on fork: // All features work on fork:
await fork.vfs.readFile('/project/README.md') // ✅ VFS await fork.vfs.readFile('/project/README.md') // ✅ VFS
await fork.find({ noun: 'user' }) // ✅ find() await fork.find({ noun: 'user' }) // ✅ find()
await fork.query('users named Alice') // ✅ Triple Intelligence await fork.query('users named Alice') // ✅ Triple Intelligence
``` ```
### Works at Billion Scale ### Works at Billion Scale
@ -494,8 +494,8 @@ await fork.query('users named Alice') // ✅ Triple Intelligence
```javascript ```javascript
// Tested at 1M entities, extrapolates to 1B // Tested at 1M entities, extrapolates to 1B
const brain = new Brainy({ const brain = new Brainy({
storage: { adapter: 'gcs', bucket: 'billion-scale' }, storage: { adapter: 'gcs', bucket: 'billion-scale' },
hnsw: { typeAware: true } // 87% memory reduction hnsw: { typeAware: true } // 87% memory reduction
}) })
await brain.init() await brain.init()
@ -563,7 +563,7 @@ ALTER TABLE users_backup RENAME TO users;
```javascript ```javascript
const fork = await brain.fork('test') const fork = await brain.fork('test')
await fork.updateAll({ email: (u) => u.email.toLowerCase() }) await fork.updateAll({ email: (u) => u.email.toLowerCase() })
if (success) await brain.checkout('test') // Make test branch active if (success) await brain.checkout('test') // Make test branch active
else await fork.destroy() else await fork.destroy()
``` ```
@ -585,7 +585,7 @@ mongoimport --db mydb --collection users_backup --file users.json
**Brainy:** **Brainy:**
```javascript ```javascript
const fork = await brain.fork() // Done! const fork = await brain.fork() // Done!
``` ```
**Winner: Brainy** (100x faster, built-in) **Winner: Brainy** (100x faster, built-in)
@ -612,8 +612,7 @@ const fork = await brain.fork() // Done!
## What's Implemented vs. What's Next ## What's Implemented vs. What's Next
### ✅ Available in v5.0.0: ### ✅ Available in - ✅ `fork()` - Instant clone in <100ms
- ✅ `fork()` - Instant clone in <100ms
- ✅ `listBranches()` - List all forks - ✅ `listBranches()` - List all forks
- ✅ `getCurrentBranch()` - Get active branch - ✅ `getCurrentBranch()` - Get active branch
- ✅ `checkout()` - Switch between branches - ✅ `checkout()` - Switch between branches
@ -621,14 +620,14 @@ const fork = await brain.fork() // Done!
- ✅ `commit()` - Create state snapshots - ✅ `commit()` - Create state snapshots
- ✅ `getHistory()` - View commit history - ✅ `getHistory()` - View commit history
### 🔮 Planned for v5.1.0+: ### 🔮 Planned for:
**Temporal Features:** **Temporal Features:**
- `asOf(timestamp)` - Query data at specific time (✅ v5.0.0+) - `asOf(timestamp)` - Query data at specific time (✅ available)
- `rollback(commitHash)` - Restore to previous state - `rollback(commitHash)` - Restore to previous state
- Full audit trail for all changes - Full audit trail for all changes
These features require additional temporal infrastructure and are being carefully designed for v5.1.0+ These features require additional temporal infrastructure and are being carefully designed
--- ---
@ -691,4 +690,4 @@ console.log('Fork created in < 2 seconds! 🚀')
--- ---
**Brainy v5.0.0+** | [GitHub](https://github.com/soulcraftlabs/brainy) | [npm](https://npmjs.com/package/@soulcraft/brainy) **Brainy** | [GitHub](https://github.com/soulcraftlabs/brainy) | [npm](https://npmjs.com/package/@soulcraft/brainy)

View file

@ -24,8 +24,8 @@ await brain.import(anything)
```javascript ```javascript
// Array of objects? No problem. // Array of objects? No problem.
const people = [ const people = [
{ name: 'Alice', role: 'Engineer', company: 'TechCorp' }, { name: 'Alice', role: 'Engineer', company: 'TechCorp' },
{ name: 'Bob', role: 'Designer', company: 'TechCorp' } { name: 'Bob', role: 'Designer', company: 'TechCorp' }
] ]
await brain.import(people) await brain.import(people)
@ -55,7 +55,7 @@ await brain.import('sales-report.xlsx')
// Or specific sheets only // Or specific sheets only
await brain.import('data.xlsx', { await brain.import('data.xlsx', {
excelSheets: ['Customers', 'Orders'] excelSheets: ['Customers', 'Orders']
}) })
// ✨ Multi-sheet data becomes interconnected entities! // ✨ Multi-sheet data becomes interconnected entities!
``` ```
@ -68,7 +68,7 @@ await brain.import('research-paper.pdf')
// With table extraction // With table extraction
await brain.import('report.pdf', { await brain.import('report.pdf', {
pdfExtractTables: true pdfExtractTables: true
}) })
// ✨ Converts PDF tables to structured data automatically! // ✨ Converts PDF tables to structured data automatically!
``` ```
@ -83,16 +83,16 @@ await brain.import('config.yaml')
const yaml = ` const yaml = `
project: AI Assistant project: AI Assistant
team: team:
- name: Alice - name: Alice
role: Lead role: Lead
- name: Bob - name: Bob
role: Dev role: Dev
` `
await brain.import(yaml, { format: 'yaml' }) await brain.import(yaml, { format: 'yaml' })
// ✨ Hierarchical data becomes a connected graph! // ✨ Hierarchical data becomes a connected graph!
``` ```
### 📄 Import Word Documents (DOCX) - v4.2.0 ### 📄 Import Word Documents (DOCX) -
```javascript ```javascript
// From file path // From file path
await brain.import('research-paper.docx') await brain.import('research-paper.docx')
@ -105,8 +105,8 @@ await brain.import(buffer, { format: 'docx' })
// With neural extraction // With neural extraction
await brain.import('report.docx', { await brain.import('report.docx', {
enableNeuralExtraction: true, enableNeuralExtraction: true,
enableHierarchicalRelationships: true enableHierarchicalRelationships: true
}) })
// ✨ Extracts entities from paragraphs and creates relationships within sections! // ✨ Extracts entities from paragraphs and creates relationships within sections!
``` ```
@ -121,25 +121,25 @@ await brain.import('https://api.example.com/data.json')
await brain.import('https://data.gov/census.csv') await brain.import('https://data.gov/census.csv')
// ✨ Fetches CSV from web, parses, imports! // ✨ Fetches CSV from web, parses, imports!
// With authentication (v4.2.0) // With authentication
await brain.import({ await brain.import({
type: 'url', type: 'url',
data: 'https://api.example.com/private/data.xlsx', data: 'https://api.example.com/private/data.xlsx',
auth: { auth: {
username: 'user', username: 'user',
password: 'pass' password: 'pass'
} }
}) })
// ✨ Supports basic authentication for protected resources! // ✨ Supports basic authentication for protected resources!
// With custom headers (v4.2.0) // With custom headers
await brain.import({ await brain.import({
type: 'url', type: 'url',
data: 'https://api.example.com/data.json', data: 'https://api.example.com/data.json',
headers: { headers: {
'Authorization': 'Bearer TOKEN', 'Authorization': 'Bearer TOKEN',
'X-API-Key': 'your-key' 'X-API-Key': 'your-key'
} }
}) })
// ✨ Full HTTP header customization support! // ✨ Full HTTP header customization support!
``` ```
@ -163,9 +163,9 @@ When you import data, Brainy:
2. **Intelligent parsing** - CSV (encoding/delimiter), Excel (multi-sheet), PDF (text/tables), DOCX (headings/paragraphs) 2. **Intelligent parsing** - CSV (encoding/delimiter), Excel (multi-sheet), PDF (text/tables), DOCX (headings/paragraphs)
3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!) 3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!) 4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
5. **Scores confidence & weight** - Every entity and relationship gets quality metrics (v4.2.0) 5. **Scores confidence & weight** - Every entity and relationship gets quality metrics
6. **Creates embeddings** - Makes everything semantically searchable 6. **Creates embeddings** - Makes everything semantically searchable
7. **Indexes metadata** - Enables lightning-fast filtering with range queries (v4.2.0) 7. **Indexes metadata** - Enables lightning-fast filtering with range queries
## Intelligent Type Detection ## Intelligent Type Detection
@ -193,9 +193,9 @@ Brainy finds connections in your data:
```javascript ```javascript
const data = [ const data = [
{ id: 'u1', name: 'Alice', managerId: 'u2' }, { id: 'u1', name: 'Alice', managerId: 'u2' },
{ id: 'u2', name: 'Bob', departmentId: 'd1' }, { id: 'u2', name: 'Bob', departmentId: 'd1' },
{ id: 'd1', name: 'Engineering' } { id: 'd1', name: 'Engineering' }
] ]
await brain.import(data) await brain.import(data)
@ -204,28 +204,27 @@ await brain.import(data)
// - Bob "memberOf" Engineering // - Bob "memberOf" Engineering
``` ```
## Confidence & Weight Scoring - v4.2.0 ## Confidence & Weight Scoring -
Every entity and relationship gets confidence and weight scores: Every entity and relationship gets confidence and weight scores:
```javascript ```javascript
// Import with confidence threshold // Import with confidence threshold
await brain.import(data, { await brain.import(data, {
confidenceThreshold: 0.8 // Only extract entities with >80% confidence confidenceThreshold: 0.8 // Only extract entities with >80% confidence
}) })
// Query high-confidence entities using range queries // Query high-confidence entities using range queries
const highConfidence = await brain.find({ const highConfidence = await brain.find({
where: { where: {
confidence: { gte: 0.8 } // Get entities with confidence >= 0.8 confidence: { gte: 0.8 } // Get entities with confidence >= 0.8
} }
}) })
// Range query operators: gt, gte, lt, lte, between // Range query operators: gt, gte, lt, lte, between
const mediumConfidence = await brain.find({ const mediumConfidence = await brain.find({
where: { where: {
confidence: { between: [0.6, 0.8] } confidence: { between: [0.6, 0.8] }
} }
}) })
``` ```
@ -236,24 +235,23 @@ const mediumConfidence = await brain.find({
**Weights** indicate importance/relevance within the document context. **Weights** indicate importance/relevance within the document context.
## Per-Sheet Excel Extraction - v4.2.0 ## Per-Sheet Excel Extraction -
Excel files with multiple sheets can be organized by sheet: Excel files with multiple sheets can be organized by sheet:
```javascript ```javascript
// Group entities by sheet in VFS // Group entities by sheet in VFS
await brain.import('multi-sheet-data.xlsx', { await brain.import('multi-sheet-data.xlsx', {
groupBy: 'sheet' // Creates separate directories for each sheet groupBy: 'sheet' // Creates separate directories for each sheet
}) })
// Result VFS structure: // Result VFS structure:
// /imports/data/ // /imports/data/
// ├── Sheet1/ // ├── Sheet1/
// ├── entity1.json // ├── entity1.json
// └── entity2.json // └── entity2.json
// └── Sheet2/ // └── Sheet2/
// ├── entity3.json // ├── entity3.json
// └── entity4.json // └── entity4.json
// Other groupBy options: // Other groupBy options:
// - 'type': Group by entity type (Person, Place, etc.) // - 'type': Group by entity type (Person, Place, etc.)
@ -274,9 +272,9 @@ const results = await brain.find('people in engineering who joined this year')
// Graph traversal + filters // Graph traversal + filters
const connected = await brain.find({ const connected = await brain.find({
like: 'Alice', like: 'Alice',
connected: { depth: 2 }, connected: { depth: 2 },
where: { department: 'Engineering' } where: { department: 'Engineering' }
}) })
``` ```
@ -286,37 +284,37 @@ Everything works with zero config, but you can customize:
```javascript ```javascript
await brain.import(data, { await brain.import(data, {
// Format detection // Format detection
format: 'excel', // Force specific format (auto-detected if not specified) format: 'excel', // Force specific format (auto-detected if not specified)
// VFS & Organization // VFS & Organization
vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified) vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified)
groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom' groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom'
preserveSource: true, // Keep original source file in VFS (default: true) preserveSource: true, // Keep original source file in VFS (default: true)
// Entity & Relationship Creation // Entity & Relationship Creation
createEntities: true, // Create entities in knowledge graph (default: true) createEntities: true, // Create entities in knowledge graph (default: true)
createRelationships: true, // Create relationships in knowledge graph (default: true) createRelationships: true, // Create relationships in knowledge graph (default: true)
// Neural Intelligence // Neural Intelligence
enableNeuralExtraction: true, // Use AI to extract entities (default: true) enableNeuralExtraction: true, // Use AI to extract entities (default: true)
enableRelationshipInference: true, // Use AI to infer relationships (default: true) enableRelationshipInference: true, // Use AI to infer relationships (default: true)
enableConceptExtraction: true, // Extract concepts from text (default: true) enableConceptExtraction: true, // Extract concepts from text (default: true)
confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6) confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6)
// Deduplication // Deduplication
enableDeduplication: true, // Check for duplicate entities (default: true) enableDeduplication: true, // Check for duplicate entities (default: true)
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85) deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
// Note: Auto-disabled for imports >100 entities // Note: Auto-disabled for imports >100 entities
// Performance // Performance
chunkSize: 100, // Batch size for processing (default: varies by operation) chunkSize: 100, // Batch size for processing (default: varies by operation)
// History & Progress // History & Progress
enableHistory: true, // Track import history (default: true) enableHistory: true, // Track import history (default: true)
onProgress: (progress) => { // Progress callback onProgress: (progress) => { // Progress callback
console.log(progress.stage, progress.message) console.log(progress.stage, progress.message)
} }
}) })
``` ```
@ -343,9 +341,9 @@ const results = await brain.import(problematicData)
### 🏢 Business Data ### 🏢 Business Data
```javascript ```javascript
// Import ANY source - ONE method! // Import ANY source - ONE method!
await brain.import('customers.csv') // File await brain.import('customers.csv') // File
await brain.import('https://api.co/orders') // URL await brain.import('https://api.co/orders') // URL
await brain.import(productsArray) // Data await brain.import(productsArray) // Data
// Now query across all of it! // Now query across all of it!
await brain.find('customers who bought products in Q4') await brain.find('customers who bought products in Q4')
@ -389,8 +387,8 @@ await brain.find('posts by users following Alice with >10 comments')
```javascript ```javascript
// ONE method that understands EVERYTHING: // ONE method that understands EVERYTHING:
await brain.import(data) // Objects, arrays, strings await brain.import(data) // Objects, arrays, strings
await brain.import('file.csv') // Files (auto-detected) await brain.import('file.csv') // Files (auto-detected)
await brain.import('http://..') // URLs (auto-fetched) await brain.import('http://..') // URLs (auto-fetched)
// It ALWAYS knows what to do! ✨ // It ALWAYS knows what to do! ✨

View file

@ -39,7 +39,7 @@ Each phase adds intelligence and structure to your raw data, transforming it int
--- ---
## 🌊 Always-On Streaming Architecture (v4.2.0+) ## 🌊 Always-On Streaming Architecture
All imports use streaming with **progressive flush intervals**: All imports use streaming with **progressive flush intervals**:
@ -1062,7 +1062,7 @@ if (!fromEntity || !toEntity) {
} }
``` ```
#### 5.3b: Check for Duplicates (v3.43.2 Critical Fix) #### 5.3b: Check for Duplicates (Critical Fix)
**The Bug**: Without duplicate checking, re-importing would create: **The Bug**: Without duplicate checking, re-importing would create:
``` ```

View file

@ -2,7 +2,7 @@
**How to Use Progress Tracking in Your Applications** **How to Use Progress Tracking in Your Applications**
Brainy v4.5.0+ provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX). Brainy provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX).
> **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation. > **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation.
@ -20,12 +20,12 @@ const brain = await Brainy.create()
// Import with progress tracking // Import with progress tracking
const result = await brain.import(fs.readFileSync('large-file.xlsx'), { const result = await brain.import(fs.readFileSync('large-file.xlsx'), {
onProgress: (progress) => { onProgress: (progress) => {
console.log(`Progress: ${progress.stage}`) console.log(`Progress: ${progress.stage}`)
console.log(` Message: ${progress.message}`) console.log(` Message: ${progress.message}`)
console.log(` Entities: ${progress.entities || 0}`) console.log(` Entities: ${progress.entities || 0}`)
console.log(` Relationships: ${progress.relationships || 0}`) console.log(` Relationships: ${progress.relationships || 0}`)
} }
}) })
console.log(`Import complete: ${result.entities.length} entities created`) console.log(`Import complete: ${result.entities.length} entities created`)
@ -34,30 +34,30 @@ console.log(`Import complete: ${result.entities.length} entities created`)
**Expected Output:** **Expected Output:**
``` ```
Progress: detecting Progress: detecting
Message: Detecting format... Message: Detecting format...
Entities: 0 Entities: 0
Relationships: 0 Relationships: 0
Progress: extracting Progress: extracting
Message: Loading Excel workbook... Message: Loading Excel workbook...
Entities: 0 Entities: 0
Relationships: 0 Relationships: 0
Progress: extracting Progress: extracting
Message: Reading sheet: Sales (1/3) Message: Reading sheet: Sales (1/3)
Entities: 0 Entities: 0
Relationships: 0 Relationships: 0
Progress: extracting Progress: extracting
Message: Parsing Excel (33%) Message: Parsing Excel (33%)
Entities: 0 Entities: 0
Relationships: 0 Relationships: 0
Progress: extracting Progress: extracting
Message: Reading sheet: Products (2/3) Message: Reading sheet: Products (2/3)
Entities: 0 Entities: 0
Relationships: 0 Relationships: 0
... (more progress updates) ... (more progress updates)
Progress: complete Progress: complete
Message: Import complete Message: Import complete
Entities: 1523 Entities: 1523
Relationships: 892 Relationships: 892
Import complete: 1523 entities created Import complete: 1523 entities created
``` ```
@ -70,19 +70,19 @@ The examples below show format-specific messages, but **you don't need format-sp
```typescript ```typescript
// ONE HANDLER FOR ALL FORMATS! // ONE HANDLER FOR ALL FORMATS!
function universalProgressHandler(progress) { function universalProgressHandler(progress) {
console.log(`[${progress.stage}] ${progress.message}`) console.log(`[${progress.stage}] ${progress.message}`)
if (progress.processed && progress.total) { if (progress.processed && progress.total) {
console.log(` Progress: ${progress.processed}/${progress.total}`) console.log(` Progress: ${progress.processed}/${progress.total}`)
} }
if (progress.entities || progress.relationships) { if (progress.entities || progress.relationships) {
console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`) console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`)
} }
if (progress.throughput && progress.eta) { if (progress.throughput && progress.eta) {
console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`) console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`)
} }
} }
// Use it for ANY format! // Use it for ANY format!
@ -105,20 +105,20 @@ The examples below show **what messages look like** for different formats using
```typescript ```typescript
await brain.import(csvBuffer, { await brain.import(csvBuffer, {
format: 'csv', format: 'csv',
onProgress: (progress) => { onProgress: (progress) => {
if (progress.stage === 'extracting') { if (progress.stage === 'extracting') {
// CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc. // CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc.
console.log(progress.message) console.log(progress.message)
} }
} }
}) })
``` ```
**CSV Progress Messages:** **CSV Progress Messages:**
- ✅ "Detecting CSV encoding and delimiter..." - ✅ "Detecting CSV encoding and delimiter..."
- ✅ "Parsing CSV rows (delimiter: ",")" - ✅ "Parsing CSV rows (delimiter: ",")"
- ✅ "Parsed 75%" (via bytes processed) - ✅ "Parsed 75%" (via bytes processed)
- ✅ "Extracted 1000 rows" - ✅ "Extracted 1000 rows"
- ✅ "Converting types: 5000/10000 rows..." - ✅ "Converting types: 5000/10000 rows..."
- ✅ "CSV processing complete: 10000 rows" - ✅ "CSV processing complete: 10000 rows"
@ -129,12 +129,12 @@ await brain.import(csvBuffer, {
```typescript ```typescript
await brain.import(pdfBuffer, { await brain.import(pdfBuffer, {
format: 'pdf', format: 'pdf',
onProgress: (progress) => { onProgress: (progress) => {
// PDF reports exact page numbers // PDF reports exact page numbers
console.log(progress.message) console.log(progress.message)
// Example: "Processing page 5 of 23" // Example: "Processing page 5 of 23"
} }
}) })
``` ```
@ -152,12 +152,12 @@ await brain.import(pdfBuffer, {
```typescript ```typescript
await brain.import(excelBuffer, { await brain.import(excelBuffer, {
format: 'excel', format: 'excel',
onProgress: (progress) => { onProgress: (progress) => {
// Excel reports sheet names // Excel reports sheet names
console.log(progress.message) console.log(progress.message)
// Example: "Reading sheet: Q2 Sales (2/5)" // Example: "Reading sheet: Q2 Sales (2/5)"
} }
}) })
``` ```
@ -175,11 +175,11 @@ await brain.import(excelBuffer, {
```typescript ```typescript
await brain.import(jsonBuffer, { await brain.import(jsonBuffer, {
format: 'json', format: 'json',
onProgress: (progress) => { onProgress: (progress) => {
// JSON reports every 10 nodes // JSON reports every 10 nodes
console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`) console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`)
} }
}) })
``` ```
@ -189,10 +189,10 @@ await brain.import(jsonBuffer, {
```typescript ```typescript
await brain.import(markdownString, { await brain.import(markdownString, {
format: 'markdown', format: 'markdown',
onProgress: (progress) => { onProgress: (progress) => {
console.log(`Section ${progress.processed}/${progress.total}`) console.log(`Section ${progress.processed}/${progress.total}`)
} }
}) })
``` ```
@ -204,44 +204,44 @@ await brain.import(markdownString, {
```typescript ```typescript
function ImportProgress({ file }: { file: File }) { function ImportProgress({ file }: { file: File }) {
const [progress, setProgress] = useState({ const [progress, setProgress] = useState({
stage: 'idle', stage: 'idle',
message: '', message: '',
percent: 0, percent: 0,
entities: 0, entities: 0,
relationships: 0 relationships: 0
}) })
const handleImport = async () => { const handleImport = async () => {
const buffer = await file.arrayBuffer() const buffer = await file.arrayBuffer()
await brain.import(Buffer.from(buffer), { await brain.import(Buffer.from(buffer), {
onProgress: (p) => { onProgress: (p) => {
setProgress({ setProgress({
stage: p.stage, stage: p.stage,
message: p.message, message: p.message,
// Estimate percentage from stage // Estimate percentage from stage
percent: { percent: {
detecting: 10, detecting: 10,
extracting: 50, extracting: 50,
'storing-vfs': 80, 'storing-vfs': 80,
'storing-graph': 90, 'storing-graph': 90,
complete: 100 complete: 100
}[p.stage] || 0, }[p.stage] || 0,
entities: p.entities || 0, entities: p.entities || 0,
relationships: p.relationships || 0 relationships: p.relationships || 0
}) })
} }
}) })
} }
return ( return (
<div> <div>
<ProgressBar value={progress.percent} /> <ProgressBar value={progress.percent} />
<p>{progress.message}</p> <p>{progress.message}</p>
<p>Entities: {progress.entities} | Relationships: {progress.relationships}</p> <p>Entities: {progress.entities} | Relationships: {progress.relationships}</p>
</div> </div>
) )
} }
``` ```
@ -255,13 +255,13 @@ import ora from 'ora'
const spinner = ora('Starting import...').start() const spinner = ora('Starting import...').start()
await brain.import(buffer, { await brain.import(buffer, {
onProgress: (progress) => { onProgress: (progress) => {
spinner.text = progress.message spinner.text = progress.message
if (progress.stage === 'complete') { if (progress.stage === 'complete') {
spinner.succeed(`Import complete: ${progress.entities} entities`) spinner.succeed(`Import complete: ${progress.entities} entities`)
} }
} }
}) })
``` ```
@ -285,20 +285,20 @@ let startTime = Date.now()
let lastUpdate = startTime let lastUpdate = startTime
await brain.import(buffer, { await brain.import(buffer, {
onProgress: (progress) => { onProgress: (progress) => {
const elapsed = Date.now() - startTime const elapsed = Date.now() - startTime
const rate = progress.entities / (elapsed / 1000) // entities/sec const rate = progress.entities / (elapsed / 1000) // entities/sec
console.clear() console.clear()
console.log('Import Progress Dashboard') console.log('Import Progress Dashboard')
console.log('========================') console.log('========================')
console.log(`Stage: ${progress.stage}`) console.log(`Stage: ${progress.stage}`)
console.log(`Status: ${progress.message}`) console.log(`Status: ${progress.message}`)
console.log(`Entities: ${progress.entities}`) console.log(`Entities: ${progress.entities}`)
console.log(`Relationships: ${progress.relationships}`) console.log(`Relationships: ${progress.relationships}`)
console.log(`Rate: ${rate.toFixed(1)} entities/sec`) console.log(`Rate: ${rate.toFixed(1)} entities/sec`)
console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`) console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`)
} }
}) })
``` ```
@ -310,21 +310,21 @@ await brain.import(buffer, {
```typescript ```typescript
const formatMessages = { const formatMessages = {
csv: (p) => `CSV: ${p.message}`, csv: (p) => `CSV: ${p.message}`,
pdf: (p) => `PDF: ${p.message}`, pdf: (p) => `PDF: ${p.message}`,
excel: (p) => `Excel: ${p.message}`, excel: (p) => `Excel: ${p.message}`,
json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`, json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`,
markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`, markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`,
yaml: (p) => `YAML: ${p.processed} nodes`, yaml: (p) => `YAML: ${p.processed} nodes`,
docx: (p) => `DOCX: ${p.processed} paragraphs` docx: (p) => `DOCX: ${p.processed} paragraphs`
} }
await brain.import(buffer, { await brain.import(buffer, {
onProgress: (progress) => { onProgress: (progress) => {
// Format is available in progress.stage metadata // Format is available in progress.stage metadata
const message = formatMessages[detectedFormat]?.(progress) || progress.message const message = formatMessages[detectedFormat]?.(progress) || progress.message
console.log(message) console.log(message)
} }
}) })
``` ```
@ -336,18 +336,18 @@ await brain.import(buffer, {
```typescript ```typescript
let lastUIUpdate = 0 let lastUIUpdate = 0
const THROTTLE_MS = 100 // Update UI max once per 100ms const THROTTLE_MS = 100 // Update UI max once per 100ms
await brain.import(buffer, { await brain.import(buffer, {
onProgress: (progress) => { onProgress: (progress) => {
const now = Date.now() const now = Date.now()
if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') { if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') {
return // Skip this update return // Skip this update
} }
lastUIUpdate = now lastUIUpdate = now
updateUI(progress) // Only update every 100ms updateUI(progress) // Only update every 100ms
} }
}) })
``` ```

View file

@ -28,17 +28,17 @@
```typescript ```typescript
// THE PUBLIC API - Same for ALL 7 formats! // THE PUBLIC API - Same for ALL 7 formats!
brain.import(buffer, { brain.import(buffer, {
onProgress: (progress: ImportProgress) => { onProgress: (progress: ImportProgress) => {
// These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX // These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX
progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete' progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
progress.message // Human-readable status (varies by format, always readable) progress.message // Human-readable status (varies by format, always readable)
progress.processed // Items processed (optional) progress.processed // Items processed (optional)
progress.total // Total items (optional) progress.total // Total items (optional)
progress.entities // Entities extracted (optional) progress.entities // Entities extracted (optional)
progress.relationships // Relationships inferred (optional) progress.relationships // Relationships inferred (optional)
progress.throughput // Items/sec (optional, during extraction) progress.throughput // Items/sec (optional, during extraction)
progress.eta // Time remaining in ms (optional) progress.eta // Time remaining in ms (optional)
} }
}) })
``` ```
@ -49,14 +49,14 @@ The table below shows how formats implement progress *internally*. Normal develo
```typescript ```typescript
// Internal: Binary formats use handler hooks (you added these!) // Internal: Binary formats use handler hooks (you added these!)
interface FormatHandlerProgressHooks { interface FormatHandlerProgressHooks {
onBytesProcessed?: (bytes: number) => void onBytesProcessed?: (bytes: number) => void
onCurrentItem?: (message: string) => void onCurrentItem?: (message: string) => void
onDataExtracted?: (count: number, total?: number) => void onDataExtracted?: (count: number, total?: number) => void
} }
// Internal: Text formats use importer callbacks // Internal: Text formats use importer callbacks
interface ImporterProgressCallback { interface ImporterProgressCallback {
onProgress?: (stats: { processed, total, entities, relationships }) => void onProgress?: (stats: { processed, total, entities, relationships }) => void
} }
// Both are converted to ImportProgress by ImportCoordinator! // Both are converted to ImportProgress by ImportCoordinator!
@ -74,7 +74,7 @@ interface ImporterProgressCallback {
## 🎯 Overview ## 🎯 Overview
As of v4.5.0, Brainy supports comprehensive, multi-dimensional progress tracking for imports: Brainy supports comprehensive, multi-dimensional progress tracking for imports:
- **Bytes processed** (always available, most deterministic) - **Bytes processed** (always available, most deterministic)
- **Entities extracted** (AI extraction phase) - **Entities extracted** (AI extraction phase)
- **Stage-specific metrics** (parsing: MB/s, extraction: entities/s) - **Stage-specific metrics** (parsing: MB/s, extraction: entities/s)
@ -91,23 +91,23 @@ All handlers follow a simple, consistent pattern using **progress hooks**.
```typescript ```typescript
export interface FormatHandlerProgressHooks { export interface FormatHandlerProgressHooks {
/** /**
* Report bytes processed * Report bytes processed
* Call this as you read/parse the file * Call this as you read/parse the file
*/ */
onBytesProcessed?: (bytes: number) => void onBytesProcessed?: (bytes: number) => void
/** /**
* Set current processing context * Set current processing context
* Examples: "Processing page 5", "Reading sheet: Q2 Sales" * Examples: "Processing page 5", "Reading sheet: Q2 Sales"
*/ */
onCurrentItem?: (item: string) => void onCurrentItem?: (item: string) => void
/** /**
* Report structured data extraction progress * Report structured data extraction progress
* Examples: "Extracted 100 rows", "Parsed 50 paragraphs" * Examples: "Extracted 100 rows", "Parsed 50 paragraphs"
*/ */
onDataExtracted?: (count: number, total?: number) => void onDataExtracted?: (count: number, total?: number) => void
} }
``` ```
@ -117,19 +117,19 @@ Progress hooks are automatically passed to your handler via `FormatHandlerOption
```typescript ```typescript
export interface FormatHandlerOptions { export interface FormatHandlerOptions {
// ... existing options ... // ... existing options ...
/** /**
* Progress hooks (v4.5.0) * Progress hooks
* Handlers call these to report progress during processing * Handlers call these to report progress during processing
*/ */
progressHooks?: FormatHandlerProgressHooks progressHooks?: FormatHandlerProgressHooks
/** /**
* Total file size in bytes (v4.5.0) * Total file size in bytes
* Used for progress percentage calculation * Used for progress percentage calculation
*/ */
totalBytes?: number totalBytes?: number
} }
``` ```
@ -141,36 +141,36 @@ Every handler follows these 5 steps:
```typescript ```typescript
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> { async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
const progressHooks = options.progressHooks // Step 1: Get hooks const progressHooks = options.progressHooks // Step 1: Get hooks
// Step 2: Report initial progress // Step 2: Report initial progress
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Starting import...') progressHooks.onCurrentItem('Starting import...')
} }
// Step 3: Report bytes as you process // Step 3: Report bytes as you process
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data) const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data)
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0) // Start progressHooks.onBytesProcessed(0) // Start
} }
// ... do parsing ... // ... do parsing ...
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(buffer.length) // Complete progressHooks.onBytesProcessed(buffer.length) // Complete
} }
// Step 4: Report data extraction // Step 4: Report data extraction
if (progressHooks?.onDataExtracted) { if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length) progressHooks.onDataExtracted(data.length, data.length)
} }
// Step 5: Report completion // Step 5: Report completion
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Complete: ${data.length} items processed`) progressHooks.onCurrentItem(`Complete: ${data.length} items processed`)
} }
return { format, data, metadata } return { format, data, metadata }
} }
``` ```
@ -182,76 +182,76 @@ Here's the **ACTUAL implementation** from CSV handler showing all the key progre
```typescript ```typescript
async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> { async process(data: Buffer | string, options: FormatHandlerOptions): Promise<ProcessedData> {
const startTime = Date.now() const startTime = Date.now()
const progressHooks = options.progressHooks // ✅ Step 1 const progressHooks = options.progressHooks // ✅ Step 1
// Convert to buffer if string // Convert to buffer if string
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8') const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
const totalBytes = buffer.length const totalBytes = buffer.length
// ✅ Step 2: Report start // ✅ Step 2: Report start
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0) progressHooks.onBytesProcessed(0)
} }
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...') progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...')
} }
// Detect encoding // Detect encoding
const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer) const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer)
const text = buffer.toString(detectedEncoding as BufferEncoding) const text = buffer.toString(detectedEncoding as BufferEncoding)
// Detect delimiter // Detect delimiter
const delimiter = options.csvDelimiter || this.detectDelimiter(text) const delimiter = options.csvDelimiter || this.detectDelimiter(text)
// ✅ Progress update: Parsing phase // ✅ Progress update: Parsing phase
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`) progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
} }
// Parse CSV // Parse CSV
const records = parse(text, { /* options */ }) const records = parse(text, { /* options */ })
// ✅ Step 3: Report bytes processed (entire file parsed) // ✅ Step 3: Report bytes processed (entire file parsed)
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes) progressHooks.onBytesProcessed(totalBytes)
} }
const data = Array.isArray(records) ? records : [records] const data = Array.isArray(records) ? records : [records]
// ✅ Step 4: Report data extraction // ✅ Step 4: Report data extraction
if (progressHooks?.onDataExtracted) { if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length) progressHooks.onDataExtracted(data.length, data.length)
} }
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`) progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`)
} }
// Type inference and conversion // Type inference and conversion
const fields = data.length > 0 ? Object.keys(data[0]) : [] const fields = data.length > 0 ? Object.keys(data[0]) : []
const types = this.inferFieldTypes(data) const types = this.inferFieldTypes(data)
const convertedData = data.map((row, index) => { const convertedData = data.map((row, index) => {
const converted = this.convertRow(row, types) const converted = this.convertRow(row, types)
// ✅ Progress update every 1000 rows (avoid spam) // ✅ Progress update every 1000 rows (avoid spam)
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) { if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`) progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
} }
return converted return converted
}) })
// ✅ Step 5: Report completion // ✅ Step 5: Report completion
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`) progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
} }
return { return {
format: this.format, format: this.format,
data: convertedData, data: convertedData,
metadata: { /* ... */ } metadata: { /* ... */ }
} }
} }
``` ```
@ -292,51 +292,51 @@ Brainy supports **7 file formats** with full progress tracking:
```typescript ```typescript
async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> { async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
const progressHooks = options.progressHooks const progressHooks = options.progressHooks
const totalBytes = data.length const totalBytes = data.length
// Report start // Report start
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Loading PDF document...') progressHooks.onCurrentItem('Loading PDF document...')
} }
const pdfDoc = await loadPDF(data) const pdfDoc = await loadPDF(data)
const totalPages = pdfDoc.numPages const totalPages = pdfDoc.numPages
const extractedData: any[] = [] const extractedData: any[] = []
let bytesProcessed = 0 let bytesProcessed = 0
for (let pageNum = 1; pageNum <= totalPages; pageNum++) { for (let pageNum = 1; pageNum <= totalPages; pageNum++) {
// ✅ Report current page // ✅ Report current page
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`) progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`)
} }
const page = await pdfDoc.getPage(pageNum) const page = await pdfDoc.getPage(pageNum)
const text = await page.getTextContent() const text = await page.getTextContent()
extractedData.push(this.processPageText(text)) extractedData.push(this.processPageText(text))
// ✅ Estimate bytes processed (pages are sequential) // ✅ Estimate bytes processed (pages are sequential)
bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes) bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes)
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed) progressHooks.onBytesProcessed(bytesProcessed)
} }
// ✅ Report extraction progress // ✅ Report extraction progress
if (progressHooks?.onDataExtracted) { if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(pageNum, totalPages) progressHooks.onDataExtracted(pageNum, totalPages)
} }
} }
// Final progress // Final progress
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes) progressHooks.onBytesProcessed(totalBytes)
} }
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`) progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`)
} }
return { format: 'pdf', data: extractedData, metadata: { /* ... */ } } return { format: 'pdf', data: extractedData, metadata: { /* ... */ } }
} }
``` ```
@ -344,55 +344,55 @@ async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedDat
```typescript ```typescript
async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> { async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedData> {
const progressHooks = options.progressHooks const progressHooks = options.progressHooks
const totalBytes = data.length const totalBytes = data.length
// Load workbook // Load workbook
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem('Loading Excel workbook...') progressHooks.onCurrentItem('Loading Excel workbook...')
} }
const workbook = XLSX.read(data) const workbook = XLSX.read(data)
const sheetNames = options.excelSheets === 'all' const sheetNames = options.excelSheets === 'all'
? workbook.SheetNames ? workbook.SheetNames
: (options.excelSheets || [workbook.SheetNames[0]]) : (options.excelSheets || [workbook.SheetNames[0]])
const allData: any[] = [] const allData: any[] = []
let bytesProcessed = 0 let bytesProcessed = 0
for (let i = 0; i < sheetNames.length; i++) { for (let i = 0; i < sheetNames.length; i++) {
const sheetName = sheetNames[i] const sheetName = sheetNames[i]
// ✅ Report current sheet // ✅ Report current sheet
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`) progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`)
} }
const sheet = workbook.Sheets[sheetName] const sheet = workbook.Sheets[sheetName]
const sheetData = XLSX.utils.sheet_to_json(sheet) const sheetData = XLSX.utils.sheet_to_json(sheet)
allData.push(...sheetData) allData.push(...sheetData)
// ✅ Estimate bytes processed (sheets processed sequentially) // ✅ Estimate bytes processed (sheets processed sequentially)
bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes) bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes)
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed) progressHooks.onBytesProcessed(bytesProcessed)
} }
// ✅ Report data extraction // ✅ Report data extraction
if (progressHooks?.onDataExtracted) { if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done
} }
} }
// Final progress // Final progress
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes) progressHooks.onBytesProcessed(totalBytes)
} }
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`) progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`)
} }
return { format: 'xlsx', data: allData, metadata: { /* ... */ } } return { format: 'xlsx', data: allData, metadata: { /* ... */ } }
} }
``` ```
@ -400,44 +400,44 @@ async process(data: Buffer, options: FormatHandlerOptions): Promise<ProcessedDat
```typescript ```typescript
async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResult> { async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResult> {
// ✅ Report parsing start // ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Parse JSON if string // Parse JSON if string
let jsonData = typeof data === 'string' ? JSON.parse(data) : data let jsonData = typeof data === 'string' ? JSON.parse(data) : data
// ✅ Report parsing complete // ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Traverse and extract (reports progress every 10 nodes) // Traverse and extract (reports progress every 10 nodes)
const entities: ExtractedJSONEntity[] = [] const entities: ExtractedJSONEntity[] = []
const relationships: ExtractedJSONRelationship[] = [] const relationships: ExtractedJSONRelationship[] = []
let nodesProcessed = 0 let nodesProcessed = 0
await this.traverseJSON( await this.traverseJSON(
jsonData, jsonData,
entities, entities,
relationships, relationships,
() => { () => {
nodesProcessed++ nodesProcessed++
if (nodesProcessed % 10 === 0) { if (nodesProcessed % 10 === 0) {
options.onProgress?.({ options.onProgress?.({
processed: nodesProcessed, processed: nodesProcessed,
entities: entities.length, entities: entities.length,
relationships: relationships.length relationships: relationships.length
}) })
} }
} }
) )
// ✅ Report completion // ✅ Report completion
options.onProgress?.({ options.onProgress?.({
processed: nodesProcessed, processed: nodesProcessed,
entities: entities.length, entities: entities.length,
relationships: relationships.length relationships: relationships.length
}) })
return { nodesProcessed, entitiesExtracted: entities.length, ... } return { nodesProcessed, entitiesExtracted: entities.length, ... }
} }
``` ```
@ -445,38 +445,38 @@ async extract(data: any, options: SmartJSONOptions = {}): Promise<SmartJSONResul
```typescript ```typescript
async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<SmartMarkdownResult> { async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<SmartMarkdownResult> {
// ✅ Report parsing start // ✅ Report parsing start
options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 }) options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 })
// Parse markdown into sections // Parse markdown into sections
const parsedSections = this.parseMarkdown(markdown, options) const parsedSections = this.parseMarkdown(markdown, options)
// ✅ Report parsing complete // ✅ Report parsing complete
options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 }) options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 })
// Process each section (reports progress after each section) // Process each section (reports progress after each section)
const sections: MarkdownSection[] = [] const sections: MarkdownSection[] = []
for (let i = 0; i < parsedSections.length; i++) { for (let i = 0; i < parsedSections.length; i++) {
const section = await this.processSection(parsedSections[i], options) const section = await this.processSection(parsedSections[i], options)
sections.push(section) sections.push(section)
options.onProgress?.({ options.onProgress?.({
processed: i + 1, processed: i + 1,
total: parsedSections.length, total: parsedSections.length,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0), entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0) relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
}) })
} }
// ✅ Report completion // ✅ Report completion
options.onProgress?.({ options.onProgress?.({
processed: sections.length, processed: sections.length,
total: sections.length, total: sections.length,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0), entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0) relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0)
}) })
return { sectionsProcessed: sections.length, ... } return { sectionsProcessed: sections.length, ... }
} }
``` ```
@ -484,27 +484,27 @@ async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise<Sma
```typescript ```typescript
async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Promise<SmartYAMLResult> { async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Promise<SmartYAMLResult> {
// ✅ Report parsing start // ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Parse YAML // Parse YAML
const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8') const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8')
const data = yaml.load(yamlString) const data = yaml.load(yamlString)
// ✅ Report parsing complete // ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Traverse YAML structure (reports progress every 10 nodes) // Traverse YAML structure (reports progress every 10 nodes)
// ... similar to JSON traversal ... // ... similar to JSON traversal ...
// ✅ Report completion (already implemented) // ✅ Report completion (already implemented)
options.onProgress?.({ options.onProgress?.({
processed: nodesProcessed, processed: nodesProcessed,
entities: entities.length, entities: entities.length,
relationships: relationships.length relationships: relationships.length
}) })
return { nodesProcessed, entitiesExtracted: entities.length, ... } return { nodesProcessed, entitiesExtracted: entities.length, ... }
} }
``` ```
@ -512,39 +512,39 @@ async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Pro
```typescript ```typescript
async extract(buffer: Buffer, options: SmartDOCXOptions = {}): Promise<SmartDOCXResult> { async extract(buffer: Buffer, options: SmartDOCXOptions = {}): Promise<SmartDOCXResult> {
// ✅ Report parsing start // ✅ Report parsing start
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Extract text and HTML using Mammoth // Extract text and HTML using Mammoth
const textResult = await mammoth.extractRawText({ buffer }) const textResult = await mammoth.extractRawText({ buffer })
const htmlResult = await mammoth.convertToHtml({ buffer }) const htmlResult = await mammoth.convertToHtml({ buffer })
// ✅ Report parsing complete // ✅ Report parsing complete
options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) options.onProgress?.({ processed: 0, entities: 0, relationships: 0 })
// Process paragraphs (reports progress every 10 paragraphs) // Process paragraphs (reports progress every 10 paragraphs)
const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength) const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength)
for (let i = 0; i < paragraphs.length; i++) { for (let i = 0; i < paragraphs.length; i++) {
await this.processParagraph(paragraphs[i]) await this.processParagraph(paragraphs[i])
if (i % 10 === 0) { if (i % 10 === 0) {
options.onProgress?.({ options.onProgress?.({
processed: i + 1, processed: i + 1,
entities: entities.length, entities: entities.length,
relationships: relationships.length relationships: relationships.length
}) })
} }
} }
// ✅ Report completion (already implemented) // ✅ Report completion (already implemented)
options.onProgress?.({ options.onProgress?.({
processed: paragraphs.length, processed: paragraphs.length,
entities: entities.length, entities: entities.length,
relationships: relationships.length relationships: relationships.length
}) })
return { paragraphsProcessed: paragraphs.length, ... } return { paragraphsProcessed: paragraphs.length, ... }
} }
``` ```
@ -559,20 +559,20 @@ Progress hooks are **optional**. Always check before calling:
```typescript ```typescript
// ✅ Good - safe // ✅ Good - safe
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytes) progressHooks.onBytesProcessed(bytes)
} }
// ❌ Bad - will crash if hooks undefined // ❌ Bad - will crash if hooks undefined
progressHooks.onBytesProcessed(bytes) // TypeError! progressHooks.onBytesProcessed(bytes) // TypeError!
``` ```
### 2. Report Bytes at Start and End ### 2. Report Bytes at Start and End
```typescript ```typescript
// ✅ Good - clear start and end // ✅ Good - clear start and end
progressHooks?.onBytesProcessed(0) // Start progressHooks?.onBytesProcessed(0) // Start
// ... processing ... // ... processing ...
progressHooks?.onBytesProcessed(totalBytes) // End progressHooks?.onBytesProcessed(totalBytes) // End
// ❌ Bad - no clear boundaries // ❌ Bad - no clear boundaries
// ... just start processing without reporting start // ... just start processing without reporting start
@ -583,17 +583,17 @@ progressHooks?.onBytesProcessed(totalBytes) // End
```typescript ```typescript
// ✅ Good - report every 1000 items // ✅ Good - report every 1000 items
for (let i = 0; i < items.length; i++) { for (let i = 0; i < items.length; i++) {
processItem(items[i]) processItem(items[i])
if (i > 0 && i % 1000 === 0) { if (i > 0 && i % 1000 === 0) {
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`)
} }
} }
// ❌ Bad - report EVERY item (spam!) // ❌ Bad - report EVERY item (spam!)
for (let i = 0; i < items.length; i++) { for (let i = 0; i < items.length; i++) {
processItem(items[i]) processItem(items[i])
progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks! progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks!
} }
``` ```
@ -614,13 +614,13 @@ progressHooks?.onCurrentItem('Working...')
```typescript ```typescript
// ✅ Good - total known // ✅ Good - total known
progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows
// ✅ Also good - total unknown (streaming) // ✅ Also good - total unknown (streaming)
progressHooks?.onDataExtracted(100, undefined) // 100 rows so far progressHooks?.onDataExtracted(100, undefined) // 100 rows so far
// ✅ Also good - complete // ✅ Also good - complete
progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows
``` ```
--- ---
@ -637,18 +637,18 @@ const handler = new CSVHandler()
const data = fs.readFileSync('./test.csv') const data = fs.readFileSync('./test.csv')
const result = await handler.process(data, { const result = await handler.process(data, {
filename: 'test.csv', filename: 'test.csv',
progressHooks: { progressHooks: {
onBytesProcessed: (bytes) => { onBytesProcessed: (bytes) => {
console.log(`Bytes: ${bytes}`) console.log(`Bytes: ${bytes}`)
}, },
onCurrentItem: (item) => { onCurrentItem: (item) => {
console.log(`Status: ${item}`) console.log(`Status: ${item}`)
}, },
onDataExtracted: (count, total) => { onDataExtracted: (count, total) => {
console.log(`Extracted: ${count}${total ? `/${total}` : ''}`) console.log(`Extracted: ${count}${total ? `/${total}` : ''}`)
} }
} }
}) })
console.log(`Complete: ${result.data.length} rows`) console.log(`Complete: ${result.data.length} rows`)
@ -673,24 +673,24 @@ Complete: 1000 rows
``` ```
User Imports File User Imports File
ImportManager ImportManager
Creates ProgressTracker Creates ProgressTracker
Calls Handler.process() with progressHooks Calls Handler.process() with progressHooks
Handler Reports Progress: Handler Reports Progress:
├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated ├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated
├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated ├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated
├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated ├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated
├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated ├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated
└─ onCurrentItem("Complete") → ProgressTracker → final progress └─ onCurrentItem("Complete") → ProgressTracker → final progress
ProgressTracker emits to callback (throttled 100ms) ProgressTracker emits to callback (throttled 100ms)
User sees: User sees:
"Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..." "Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..."
``` ```
--- ---

View file

@ -91,7 +91,7 @@ await brain.import(file, {
}) })
``` ```
### Import Tracking (v4.10.0+) ### Import Tracking
Track and organize imports by project: Track and organize imports by project:
@ -149,7 +149,7 @@ await brain.import(file, {
}) })
``` ```
### Always-On Streaming (v4.2.0+) ### Always-On Streaming
All imports use streaming with adaptive flush intervals. Query data as it's imported: All imports use streaming with adaptive flush intervals. Query data as it's imported:

View file

@ -65,7 +65,7 @@ interface ImportProgress {
/** Estimated time remaining (milliseconds) */ /** Estimated time remaining (milliseconds) */
eta?: number eta?: number
/** Whether data is queryable at this point (v4.2.0+) */ /** Whether data is queryable at this point */
queryable?: boolean queryable?: boolean
} }
``` ```

View file

@ -300,7 +300,7 @@ interface ImportProgress {
relationships?: number // Relationships inferred so far relationships?: number // Relationships inferred so far
/** /**
* Whether data is queryable (v4.2.0+) * Whether data is queryable
* *
* true = Indexes flushed, queries will be fast and complete * true = Indexes flushed, queries will be fast and complete
* false/undefined = Data in storage but indexes not flushed yet * false/undefined = Data in storage but indexes not flushed yet
@ -376,7 +376,7 @@ No changes required! Streaming is now always enabled with optimal defaults:
// Before (v3.x, v4.0, v4.1): Works the same // Before (v3.x, v4.0, v4.1): Works the same
await brain.import(file) await brain.import(file)
// After (v4.2.0+): Streaming always on, zero config // After: Streaming always on, zero config
await brain.import(file) await brain.import(file)
``` ```

View file

@ -1,6 +1,6 @@
# Capacity Planning & Operations Guide # Capacity Planning & Operations Guide
**Brainy v3.36.0+ Enterprise Operations** **Brainy Enterprise Operations**
This guide provides production-ready capacity planning formulas, deployment strategies, and operational guidelines for scaling Brainy from development (2GB) to enterprise (128GB+) deployments. This guide provides production-ready capacity planning formulas, deployment strategies, and operational guidelines for scaling Brainy from development (2GB) to enterprise (128GB+) deployments.
@ -24,13 +24,13 @@ Where:
### Adaptive Caching Strategy ### Adaptive Caching Strategy
``` ```
estimatedVectorMemory = entityCount × 1536 bytes // 384 dims × 4 bytes per float estimatedVectorMemory = entityCount × 1536 bytes // 384 dims × 4 bytes per float
hnswCacheBudget = cacheSize × 0.80 // 80% threshold for preloading decision hnswCacheBudget = cacheSize × 0.80 // 80% threshold for preloading decision
if estimatedVectorMemory < hnswCacheBudget: if estimatedVectorMemory < hnswCacheBudget:
cachingStrategy = 'preloaded' // All vectors loaded at init cachingStrategy = 'preloaded' // All vectors loaded at init
else: else:
cachingStrategy = 'on-demand' // Vectors loaded adaptively via UnifiedCache cachingStrategy = 'on-demand' // Vectors loaded adaptively via UnifiedCache
``` ```
--- ---
@ -46,19 +46,19 @@ else:
**Memory Breakdown:** **Memory Breakdown:**
``` ```
System Memory: 2048 MB System Memory: 2048 MB
OS Reserved (20%): -410 MB OS Reserved (20%): -410 MB
Available: 1638 MB Available: 1638 MB
Model Memory: -140 MB Model Memory: -140 MB
├─ WASM + Weights: 90 MB ├─ WASM + Weights: 90 MB
└─ Workspace: 50 MB └─ Workspace: 50 MB
─────────────────────────── ───────────────────────────
Available for Cache: 1488 MB Available for Cache: 1488 MB
Dev Allocation (25%): 372 MB UnifiedCache Dev Allocation (25%): 372 MB UnifiedCache
├─ HNSW (30%): 112 MB ├─ HNSW (30%): 112 MB
├─ Metadata (40%): 149 MB ├─ Metadata (40%): 149 MB
├─ Search (20%): 74 MB ├─ Search (20%): 74 MB
└─ Shared (10%): 37 MB └─ Shared (10%): 37 MB
``` ```
**Capacity:** **Capacity:**
@ -75,9 +75,9 @@ Dev Allocation (25%): 372 MB UnifiedCache
**Configuration:** **Configuration:**
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { type: 'filesystem', path: './brainy-data' }, storage: { type: 'filesystem', path: './brainy-data' },
model: { precision: 'q8' }, model: { precision: 'q8' },
cache: { /* auto-sized to 372MB */ } cache: { /* auto-sized to 372MB */ }
}) })
``` ```
@ -92,17 +92,17 @@ const brain = new Brainy({
**Memory Breakdown:** **Memory Breakdown:**
``` ```
System Memory: 8192 MB System Memory: 8192 MB
OS Reserved (20%): -1638 MB OS Reserved (20%): -1638 MB
Available: 6554 MB Available: 6554 MB
Model Memory (Q8): -150 MB Model Memory (Q8): -150 MB
─────────────────────────── ───────────────────────────
Available for Cache: 6404 MB Available for Cache: 6404 MB
Prod Allocation (50%): 3202 MB UnifiedCache Prod Allocation (50%): 3202 MB UnifiedCache
├─ HNSW (30%): 961 MB ├─ HNSW (30%): 961 MB
├─ Metadata (40%): 1281 MB ├─ Metadata (40%): 1281 MB
├─ Search (20%): 640 MB ├─ Search (20%): 640 MB
└─ Shared (10%): 320 MB └─ Shared (10%): 320 MB
``` ```
**Capacity:** **Capacity:**
@ -119,9 +119,9 @@ Prod Allocation (50%): 3202 MB UnifiedCache
**Configuration:** **Configuration:**
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { type: 'filesystem', path: '/var/lib/brainy' }, storage: { type: 'filesystem', path: '/var/lib/brainy' },
model: { precision: 'q8' }, model: { precision: 'q8' },
// Auto-sized cache: 3202MB // Auto-sized cache: 3202MB
}) })
// Monitor health // Monitor health
@ -141,17 +141,17 @@ console.log(`Caching strategy: ${stats.cachingStrategy}`)
**Memory Breakdown:** **Memory Breakdown:**
``` ```
System Memory: 32768 MB System Memory: 32768 MB
OS Reserved (20%): -6554 MB OS Reserved (20%): -6554 MB
Available: 26214 MB Available: 26214 MB
Model Memory (Q8): -150 MB Model Memory (Q8): -150 MB
─────────────────────────── ───────────────────────────
Available for Cache: 26064 MB Available for Cache: 26064 MB
Prod Allocation (50%): 13032 MB UnifiedCache Prod Allocation (50%): 13032 MB UnifiedCache
├─ HNSW (30%): 3910 MB ├─ HNSW (30%): 3910 MB
├─ Metadata (40%): 5213 MB ├─ Metadata (40%): 5213 MB
├─ Search (20%): 2606 MB ├─ Search (20%): 2606 MB
└─ Shared (10%): 1303 MB └─ Shared (10%): 1303 MB
``` ```
**Capacity:** **Capacity:**
@ -168,11 +168,11 @@ Prod Allocation (50%): 13032 MB UnifiedCache
**Configuration:** **Configuration:**
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'gcs-native', type: 'gcs-native',
gcsNativeStorage: { bucketName: 'production-data' } gcsNativeStorage: { bucketName: 'production-data' }
}, },
model: { precision: 'q8' } // or 'fp32' for +0.5% accuracy model: { precision: 'q8' } // or 'fp32' for +0.5% accuracy
}) })
// Verify allocation // Verify allocation
@ -192,17 +192,17 @@ console.log(`Environment: ${memoryInfo.memoryInfo.environment}`)
**Memory Breakdown:** **Memory Breakdown:**
``` ```
System Memory: 131072 MB System Memory: 131072 MB
OS Reserved (20%): -26214 MB OS Reserved (20%): -26214 MB
Available: 104858 MB Available: 104858 MB
Model Memory (FP32): -250 MB Model Memory (FP32): -250 MB
─────────────────────────────── ───────────────────────────────
Available for Cache: 104608 MB Available for Cache: 104608 MB
Prod Allocation (50%): 52304 MB UnifiedCache (logarithmic scaling applies) Prod Allocation (50%): 52304 MB UnifiedCache (logarithmic scaling applies)
├─ HNSW (30%): 15691 MB ├─ HNSW (30%): 15691 MB
├─ Metadata (40%): 20922 MB ├─ Metadata (40%): 20922 MB
├─ Search (20%): 10461 MB ├─ Search (20%): 10461 MB
└─ Shared (10%): 5230 MB └─ Shared (10%): 5230 MB
``` ```
**Logarithmic Scaling Applied:** **Logarithmic Scaling Applied:**
@ -227,30 +227,30 @@ Actual cache size: ~40GB (prevents waste on 128GB systems)
**Configuration:** **Configuration:**
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3', type: 's3',
s3Storage: { s3Storage: {
bucketName: 'enterprise-data', bucketName: 'enterprise-data',
region: 'us-east-1' region: 'us-east-1'
} }
}, },
model: { precision: 'fp32' } // Maximum accuracy model: { precision: 'fp32' } // Maximum accuracy
}) })
// Enterprise monitoring // Enterprise monitoring
setInterval(() => { setInterval(() => {
const stats = brain.hnsw.getCacheStats() const stats = brain.hnsw.getCacheStats()
if (stats.fairness.fairnessViolation) { if (stats.fairness.fairnessViolation) {
console.warn('FAIRNESS VIOLATION: HNSW using too much cache') console.warn('FAIRNESS VIOLATION: HNSW using too much cache')
console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access, ${stats.hnswCache.sizePercent}% size`) console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access, ${stats.hnswCache.sizePercent}% size`)
} }
if (stats.unifiedCache.hitRatePercent < 75) { if (stats.unifiedCache.hitRatePercent < 75) {
console.warn(`Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) console.warn(`Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`)
console.warn('Recommendations:', stats.recommendations) console.warn('Recommendations:', stats.recommendations)
} }
}, 60000) // Check every minute }, 60000) // Check every minute
``` ```
--- ---
@ -263,12 +263,12 @@ Brainy auto-detects container memory limits via cgroups v1/v2:
```typescript ```typescript
// Automatic detection // Automatic detection
const brain = new Brainy() // Detects cgroup limits automatically const brain = new Brainy() // Detects cgroup limits automatically
// Verify detection // Verify detection
const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo() const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
console.log(`Container: ${memoryInfo.memoryInfo.isContainer}`) console.log(`Container: ${memoryInfo.memoryInfo.isContainer}`)
console.log(`Source: ${memoryInfo.memoryInfo.source}`) // 'cgroup-v2' or 'cgroup-v1' console.log(`Source: ${memoryInfo.memoryInfo.source}`) // 'cgroup-v2' or 'cgroup-v1'
console.log(`Limit: ${Math.round(memoryInfo.memoryInfo.available / 1024 / 1024)}MB`) console.log(`Limit: ${Math.round(memoryInfo.memoryInfo.available / 1024 / 1024)}MB`)
``` ```
@ -294,37 +294,37 @@ CMD ["node", "dist/index.js"]
```bash ```bash
docker run \ docker run \
--memory="2g" \ --memory="2g" \
--memory-reservation="1.5g" \ --memory-reservation="1.5g" \
--cpus="2" \ --cpus="2" \
my-brainy-app my-brainy-app
``` ```
**Expected allocation:** **Expected allocation:**
``` ```
Container Limit: 2048 MB Container Limit: 2048 MB
Available: 1638 MB (80% usable) Available: 1638 MB (80% usable)
Model Memory: -150 MB Model Memory: -150 MB
Available for Cache: 1488 MB Available for Cache: 1488 MB
Container Ratio (40%): 595 MB UnifiedCache Container Ratio (40%): 595 MB UnifiedCache
``` ```
**Medium Container (8GB)** **Medium Container (8GB)**
```bash ```bash
docker run \ docker run \
--memory="8g" \ --memory="8g" \
--memory-reservation="6g" \ --memory-reservation="6g" \
--cpus="4" \ --cpus="4" \
-e NODE_OPTIONS="--max-old-space-size=6144" \ -e NODE_OPTIONS="--max-old-space-size=6144" \
my-brainy-app my-brainy-app
``` ```
**Expected allocation:** **Expected allocation:**
``` ```
Container Limit: 8192 MB Container Limit: 8192 MB
Available: 6554 MB Available: 6554 MB
Model Memory: -150 MB Model Memory: -150 MB
Available for Cache: 6404 MB Available for Cache: 6404 MB
Container Ratio (40%): 2562 MB UnifiedCache Container Ratio (40%): 2562 MB UnifiedCache
``` ```
@ -335,24 +335,24 @@ Container Ratio (40%): 2562 MB UnifiedCache
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
metadata: metadata:
name: brainy-api name: brainy-api
spec: spec:
replicas: 3 replicas: 3
template: template:
spec: spec:
containers: containers:
- name: brainy - name: brainy
image: my-brainy-app:latest image: my-brainy-app:latest
resources: resources:
requests: requests:
memory: "1.5Gi" memory: "1.5Gi"
cpu: "500m" cpu: "500m"
limits: limits:
memory: "2Gi" memory: "2Gi"
cpu: "1000m" cpu: "1000m"
env: env:
- name: NODE_OPTIONS - name: NODE_OPTIONS
value: "--max-old-space-size=1536" value: "--max-old-space-size=1536"
``` ```
**Medium Pod (8GB)** **Medium Pod (8GB)**
@ -360,24 +360,24 @@ spec:
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
metadata: metadata:
name: brainy-api name: brainy-api
spec: spec:
replicas: 2 replicas: 2
template: template:
spec: spec:
containers: containers:
- name: brainy - name: brainy
image: my-brainy-app:latest image: my-brainy-app:latest
resources: resources:
requests: requests:
memory: "6Gi" memory: "6Gi"
cpu: "2000m" cpu: "2000m"
limits: limits:
memory: "8Gi" memory: "8Gi"
cpu: "4000m" cpu: "4000m"
env: env:
- name: NODE_OPTIONS - name: NODE_OPTIONS
value: "--max-old-space-size=6144" value: "--max-old-space-size=6144"
``` ```
**Best Practices:** **Best Practices:**
@ -399,15 +399,15 @@ The system automatically chooses the optimal caching strategy:
**Auto-detection logic:** **Auto-detection logic:**
```typescript ```typescript
const vectorMemoryNeeded = entityCount × 1536 // bytes const vectorMemoryNeeded = entityCount × 1536 // bytes
const hnswCacheAvailable = unifiedCache.maxSize × 0.80 const hnswCacheAvailable = unifiedCache.maxSize × 0.80
if (vectorMemoryNeeded < hnswCacheAvailable) { if (vectorMemoryNeeded < hnswCacheAvailable) {
// Preload strategy: all vectors loaded at init // Preload strategy: all vectors loaded at init
console.log('Caching strategy: preloaded (all vectors in memory)') console.log('Caching strategy: preloaded (all vectors in memory)')
} else { } else {
// On-demand strategy: vectors loaded adaptively // On-demand strategy: vectors loaded adaptively
console.log('Caching strategy: on-demand (adaptive loading via UnifiedCache)') console.log('Caching strategy: on-demand (adaptive loading via UnifiedCache)')
} }
``` ```
@ -422,12 +422,12 @@ Consider increasing RAM when:
**Decision tree:** **Decision tree:**
``` ```
If cache hit rate < 70%: If cache hit rate < 70%:
└─> Is working set < 50% of total entities? └─> Is working set < 50% of total entities?
├─> YES: Increase cache size (add RAM) ├─> YES: Increase cache size (add RAM)
└─> NO: Working set too large, consider: └─> NO: Working set too large, consider:
├─> Application-level caching ├─> Application-level caching
├─> Query optimization ├─> Query optimization
└─> Sharding dataset └─> Sharding dataset
``` ```
### When to Shard/Distribute ### When to Shard/Distribute
@ -442,17 +442,17 @@ Consider sharding when:
```typescript ```typescript
// Example: Geographic sharding // Example: Geographic sharding
const usEastBrain = new Brainy({ const usEastBrain = new Brainy({
storage: { type: 's3', s3Storage: { bucket: 'us-east-data' } } storage: { type: 's3', s3Storage: { bucket: 'us-east-data' } }
}) })
const euWestBrain = new Brainy({ const euWestBrain = new Brainy({
storage: { type: 's3', s3Storage: { bucket: 'eu-west-data' } } storage: { type: 's3', s3Storage: { bucket: 'eu-west-data' } }
}) })
// Route queries based on user location // Route queries based on user location
async function search(query, userRegion) { async function search(query, userRegion) {
const brain = userRegion === 'US' ? usEastBrain : euWestBrain const brain = userRegion === 'US' ? usEastBrain : euWestBrain
return await brain.search(query) return await brain.search(query)
} }
``` ```
@ -482,7 +482,7 @@ console.log(`Pressure: ${memoryInfo.currentPressure.pressure}`)
// Values: 'low', 'moderate', 'high', 'critical' // Values: 'low', 'moderate', 'high', 'critical'
if (memoryInfo.currentPressure.warnings.length > 0) { if (memoryInfo.currentPressure.warnings.length > 0) {
console.warn('Memory warnings:', memoryInfo.currentPressure.warnings) console.warn('Memory warnings:', memoryInfo.currentPressure.warnings)
} }
``` ```
@ -491,9 +491,9 @@ if (memoryInfo.currentPressure.warnings.length > 0) {
const stats = brain.hnsw.getCacheStats() const stats = brain.hnsw.getCacheStats()
if (stats.fairness.fairnessViolation) { if (stats.fairness.fairnessViolation) {
console.warn('Cache fairness violation detected') console.warn('Cache fairness violation detected')
console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access`) console.warn(`HNSW: ${stats.fairness.hnswAccessPercent}% access`)
console.warn(`HNSW: ${stats.hnswCache.sizePercent}% of cache`) console.warn(`HNSW: ${stats.hnswCache.sizePercent}% of cache`)
} }
``` ```
@ -502,7 +502,7 @@ if (stats.fairness.fairnessViolation) {
// Track search latency // Track search latency
console.time('search') console.time('search')
const results = await brain.search('query') const results = await brain.search('query')
console.timeEnd('search') // Target: <10ms for hot queries console.timeEnd('search') // Target: <10ms for hot queries
``` ```
### Alerting Thresholds ### Alerting Thresholds
@ -516,26 +516,26 @@ Set up alerts for:
**Example monitoring script:** **Example monitoring script:**
```typescript ```typescript
async function monitorHealth() { async function monitorHealth() {
const stats = brain.hnsw.getCacheStats() const stats = brain.hnsw.getCacheStats()
// Alert on low cache hit rate // Alert on low cache hit rate
if (stats.unifiedCache.hitRatePercent < 70) { if (stats.unifiedCache.hitRatePercent < 70) {
await sendAlert({ await sendAlert({
severity: 'warning', severity: 'warning',
message: `Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`, message: `Low cache hit rate: ${stats.unifiedCache.hitRatePercent}%`,
recommendations: stats.recommendations recommendations: stats.recommendations
}) })
} }
// Alert on memory pressure // Alert on memory pressure
const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo() const memoryInfo = brain.hnsw.unifiedCache.getMemoryInfo()
if (memoryInfo.currentPressure.pressure === 'high') { if (memoryInfo.currentPressure.pressure === 'high') {
await sendAlert({ await sendAlert({
severity: 'critical', severity: 'critical',
message: 'High memory pressure detected', message: 'High memory pressure detected',
warnings: memoryInfo.currentPressure.warnings warnings: memoryInfo.currentPressure.warnings
}) })
} }
} }
// Run every 60 seconds // Run every 60 seconds
@ -552,9 +552,9 @@ setInterval(monitorHealth, 60000)
**Sizing:** **Sizing:**
``` ```
Products: 500,000 Products: 500,000
Vector memory needed: 500K × 1536 bytes = 768 MB Vector memory needed: 500K × 1536 bytes = 768 MB
HNSW cache available: (16GB × 0.8 - 150MB) × 0.5 × 0.3 = 1,915 MB HNSW cache available: (16GB × 0.8 - 150MB) × 0.5 × 0.3 = 1,915 MB
Result: Standard mode (all vectors fit in HNSW cache) Result: Standard mode (all vectors fit in HNSW cache)
``` ```
@ -562,16 +562,16 @@ Result: Standard mode (all vectors fit in HNSW cache)
**Configuration:** **Configuration:**
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { type: 'filesystem', path: '/var/lib/brainy' }, storage: { type: 'filesystem', path: '/var/lib/brainy' },
model: { precision: 'q8' } model: { precision: 'q8' }
}) })
await brain.init() await brain.init()
// Verify preloaded strategy (all vectors in memory) // Verify preloaded strategy (all vectors in memory)
const stats = brain.hnsw.getCacheStats() const stats = brain.hnsw.getCacheStats()
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'preloaded' console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'preloaded'
console.log(`Search latency: ${stats.performance.avgSearchMs}ms`) // ~3ms console.log(`Search latency: ${stats.performance.avgSearchMs}ms`) // ~3ms
``` ```
### Example 2: Document Search (5M documents) ### Example 2: Document Search (5M documents)
@ -580,9 +580,9 @@ console.log(`Search latency: ${stats.performance.avgSearchMs}ms`) // ~3ms
**Sizing:** **Sizing:**
``` ```
Documents: 5,000,000 Documents: 5,000,000
Vector memory needed: 5M × 1536 bytes = 7,680 MB Vector memory needed: 5M × 1536 bytes = 7,680 MB
HNSW cache available: (32GB × 0.8 - 150MB) × 0.5 × 0.3 = 3,910 MB HNSW cache available: (32GB × 0.8 - 150MB) × 0.5 × 0.3 = 3,910 MB
Result: On-demand caching (vectors loaded adaptively) Result: On-demand caching (vectors loaded adaptively)
``` ```
@ -590,20 +590,20 @@ Result: On-demand caching (vectors loaded adaptively)
**Configuration:** **Configuration:**
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'gcs-native', type: 'gcs-native',
gcsNativeStorage: { bucketName: 'docs-production' } gcsNativeStorage: { bucketName: 'docs-production' }
}, },
model: { precision: 'q8' } model: { precision: 'q8' }
}) })
await brain.init() await brain.init()
// Monitor cache performance // Monitor cache performance
const stats = brain.hnsw.getCacheStats() const stats = brain.hnsw.getCacheStats()
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand' console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand'
console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >80% console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >80%
console.log(`Cold search latency: ${stats.performance.avgSearchMs}ms`) // ~12ms console.log(`Cold search latency: ${stats.performance.avgSearchMs}ms`) // ~12ms
// Recommendations // Recommendations
console.log('Recommendations:', stats.recommendations) console.log('Recommendations:', stats.recommendations)
@ -616,9 +616,9 @@ console.log('Recommendations:', stats.recommendations)
**Sizing:** **Sizing:**
``` ```
Entities: 20,000,000 Entities: 20,000,000
Vector memory needed: 20M × 1536 bytes = 30,720 MB Vector memory needed: 20M × 1536 bytes = 30,720 MB
HNSW cache available: ~15,691 MB (after logarithmic scaling) HNSW cache available: ~15,691 MB (after logarithmic scaling)
Result: On-demand caching with high-performance adaptive loading Result: On-demand caching with high-performance adaptive loading
``` ```
@ -626,14 +626,14 @@ Result: On-demand caching with high-performance adaptive loading
**Configuration:** **Configuration:**
```typescript ```typescript
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3', type: 's3',
s3Storage: { s3Storage: {
bucketName: 'knowledge-graph-prod', bucketName: 'knowledge-graph-prod',
region: 'us-east-1' region: 'us-east-1'
} }
}, },
model: { precision: 'fp32' } // Maximum accuracy model: { precision: 'fp32' } // Maximum accuracy
}) })
await brain.init() await brain.init()
@ -641,13 +641,13 @@ await brain.init()
// Enterprise monitoring // Enterprise monitoring
const stats = brain.hnsw.getCacheStats() const stats = brain.hnsw.getCacheStats()
console.log(`Entities: ${stats.autoDetection.entityCount.toLocaleString()}`) console.log(`Entities: ${stats.autoDetection.entityCount.toLocaleString()}`)
console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand' console.log(`Caching strategy: ${stats.cachingStrategy}`) // 'on-demand'
console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >85% console.log(`Cache hit rate: ${stats.unifiedCache.hitRatePercent}%`) // Target >85%
console.log(`HNSW cache: ${stats.hnswCache.estimatedMemoryMB}MB`) console.log(`HNSW cache: ${stats.hnswCache.estimatedMemoryMB}MB`)
// Fairness check // Fairness check
if (stats.fairness.fairnessViolation) { if (stats.fairness.fairnessViolation) {
console.warn('HNSW dominating cache - consider tuning eviction policies') console.warn('HNSW dominating cache - consider tuning eviction policies')
} }
``` ```
@ -690,8 +690,8 @@ console.log(`Warnings:`, memoryInfo.currentPressure.warnings)
```typescript ```typescript
const stats = brain.hnsw.getCacheStats() const stats = brain.hnsw.getCacheStats()
if (stats.fairness.fairnessViolation) { if (stats.fairness.fairnessViolation) {
console.log(`HNSW access: ${stats.fairness.hnswAccessPercent}%`) console.log(`HNSW access: ${stats.fairness.hnswAccessPercent}%`)
console.log(`HNSW cache: ${stats.hnswCache.sizePercent}%`) console.log(`HNSW cache: ${stats.hnswCache.sizePercent}%`)
} }
``` ```
@ -704,8 +704,7 @@ if (stats.fairness.fairnessViolation) {
## 📚 Additional Resources ## 📚 Additional Resources
- **[Migration Guide](../guides/migration-3.36.0.md)** - Upgrading to v3.36.0 - **[Migration Guide](../guides/migration-3.36.0.md)** - Upgrading to **[Architecture Overview](../architecture/data-storage-architecture.md)** - Deep dive into storage and caching
- **[Architecture Overview](../architecture/data-storage-architecture.md)** - Deep dive into storage and caching
- **[GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)** - Report problems or ask questions - **[GitHub Issues](https://github.com/soulcraftlabs/brainy/issues)** - Report problems or ask questions
--- ---

View file

@ -1,10 +1,9 @@
# AWS S3 Cost Optimization Guide for Brainy v4.0.0 # AWS S3 Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $138k/year to $5.9k/year at 500TB scale (**96% savings**) > **Cost Impact**: Reduce storage costs from $138k/year to $5.9k/year at 500TB scale (**96% savings**)
## Overview ## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance. Brainy provides enterprise-grade cost optimization features for AWS S3 storage, enabling automatic tier transitions that dramatically reduce storage costs while maintaining performance.
## Cost Breakdown (Before Optimization) ## Cost Breakdown (Before Optimization)
@ -39,10 +38,10 @@ import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
// Initialize Brainy with S3 storage // Initialize Brainy with S3 storage
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
bucket: 'my-brainy-data', bucket: 'my-brainy-data',
region: 'us-east-1', region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID, accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}) })
const brain = new Brainy({ storage }) const brain = new Brainy({ storage })
@ -50,24 +49,24 @@ await brain.init()
// Set lifecycle policy for automatic archival // Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
id: 'optimize-vectors', id: 'optimize-vectors',
prefix: 'entities/nouns/vectors/', prefix: 'entities/nouns/vectors/',
status: 'Enabled', status: 'Enabled',
transitions: [ transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, // Infrequent Access after 30 days { days: 30, storageClass: 'STANDARD_IA' }, // Infrequent Access after 30 days
{ days: 90, storageClass: 'GLACIER' }, // Glacier after 90 days { days: 90, storageClass: 'GLACIER' }, // Glacier after 90 days
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 1 year { days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 1 year
] ]
}, { }, {
id: 'optimize-metadata', id: 'optimize-metadata',
prefix: 'entities/nouns/metadata/', prefix: 'entities/nouns/metadata/',
status: 'Enabled', status: 'Enabled',
transitions: [ transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, { days: 30, storageClass: 'STANDARD_IA' },
{ days: 180, storageClass: 'GLACIER' } { days: 180, storageClass: 'GLACIER' }
] ]
}] }]
}) })
// Verify lifecycle policy // Verify lifecycle policy
@ -84,10 +83,10 @@ console.log('Lifecycle policy active:', policy.rules.length, 'rules')
- 10% of data 365+ days old (Deep Archive) - 10% of data 365+ days old (Deep Archive)
``` ```
Standard (200TB): 200TB × $0.023/GB × 12 = $55,200/year Standard (200TB): 200TB × $0.023/GB × 12 = $55,200/year
Standard-IA (150TB): 150TB × $0.0125/GB × 12 = $22,500/year Standard-IA (150TB): 150TB × $0.0125/GB × 12 = $22,500/year
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Deep Archive (50TB): 50TB × $0.00099/GB × 12 = $594/year Deep Archive (50TB): 50TB × $0.00099/GB × 12 = $594/year
Total Storage Cost: $83,094/year (instead of $138,000) Total Storage Cost: $83,094/year (instead of $138,000)
Total with Operations: ~$88,000/year Total with Operations: ~$88,000/year
@ -133,11 +132,11 @@ Intelligent-Tiering automatically moves objects between:
- 30% Deep Archive Access - 30% Deep Archive Access
``` ```
Frequent (75TB): 75TB × $0.023/GB × 12 = $20,700/year Frequent (75TB): 75TB × $0.023/GB × 12 = $20,700/year
Infrequent (100TB): 100TB × $0.0125/GB × 12 = $15,000/year Infrequent (100TB): 100TB × $0.0125/GB × 12 = $15,000/year
Archive (175TB): 175TB × $0.004/GB × 12 = $8,400/year Archive (175TB): 175TB × $0.004/GB × 12 = $8,400/year
Deep Archive (150TB): 150TB × $0.00099/GB × 12 = $1,782/year Deep Archive (150TB): 150TB × $0.00099/GB × 12 = $1,782/year
Monitoring: ~$300/year (minimal) Monitoring: ~$300/year (minimal)
Total Storage Cost: $46,182/year Total Storage Cost: $46,182/year
Total with Operations: ~$51,000/year Total with Operations: ~$51,000/year
@ -157,21 +156,21 @@ await storage.enableIntelligentTiering('entities/verbs/vectors/', 'verbs-auto')
// Set lifecycle policy for metadata (less frequently accessed) // Set lifecycle policy for metadata (less frequently accessed)
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
id: 'archive-old-metadata', id: 'archive-old-metadata',
prefix: 'entities/nouns/metadata/', prefix: 'entities/nouns/metadata/',
status: 'Enabled', status: 'Enabled',
transitions: [ transitions: [
{ days: 30, storageClass: 'STANDARD_IA' }, { days: 30, storageClass: 'STANDARD_IA' },
{ days: 60, storageClass: 'GLACIER' }, { days: 60, storageClass: 'GLACIER' },
{ days: 180, storageClass: 'DEEP_ARCHIVE' } { days: 180, storageClass: 'DEEP_ARCHIVE' }
] ]
}, { }, {
id: 'cleanup-old-system-data', id: 'cleanup-old-system-data',
prefix: '_system/', prefix: '_system/',
status: 'Enabled', status: 'Enabled',
expiration: { days: 365 } // Delete old statistics expiration: { days: 365 } // Delete old statistics
}] }]
}) })
``` ```
@ -179,19 +178,19 @@ await storage.setLifecyclePolicy({
**Vectors (300TB with Intelligent-Tiering):** **Vectors (300TB with Intelligent-Tiering):**
``` ```
Frequent (45TB): 45TB × $0.023/GB × 12 = $12,420/year Frequent (45TB): 45TB × $0.023/GB × 12 = $12,420/year
Infrequent (60TB): 60TB × $0.0125/GB × 12 = $9,000/year Infrequent (60TB): 60TB × $0.0125/GB × 12 = $9,000/year
Archive (105TB): 105TB × $0.004/GB × 12 = $5,040/year Archive (105TB): 105TB × $0.004/GB × 12 = $5,040/year
Deep Archive (90TB): 90TB × $0.00099/GB × 12 = $1,069/year Deep Archive (90TB): 90TB × $0.00099/GB × 12 = $1,069/year
Subtotal: $27,529/year Subtotal: $27,529/year
``` ```
**Metadata (200TB with Lifecycle Policy):** **Metadata (200TB with Lifecycle Policy):**
``` ```
Standard (60TB): 60TB × $0.023/GB × 12 = $16,560/year Standard (60TB): 60TB × $0.023/GB × 12 = $16,560/year
Standard-IA (40TB): 40TB × $0.0125/GB × 12 = $6,000/year Standard-IA (40TB): 40TB × $0.0125/GB × 12 = $6,000/year
Glacier (60TB): 60TB × $0.004/GB × 12 = $2,880/year Glacier (60TB): 60TB × $0.004/GB × 12 = $2,880/year
Deep Archive (40TB): 40TB × $0.00099/GB × 12 = $475/year Deep Archive (40TB): 40TB × $0.00099/GB × 12 = $475/year
Subtotal: $25,915/year Subtotal: $25,915/year
``` ```
@ -206,16 +205,16 @@ Subtotal: $25,915/year
```typescript ```typescript
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
id: 'aggressive-archival', id: 'aggressive-archival',
prefix: 'entities/', prefix: 'entities/',
status: 'Enabled', status: 'Enabled',
transitions: [ transitions: [
{ days: 14, storageClass: 'STANDARD_IA' }, // IA after 2 weeks { days: 14, storageClass: 'STANDARD_IA' }, // IA after 2 weeks
{ days: 30, storageClass: 'GLACIER' }, // Glacier after 1 month { days: 30, storageClass: 'GLACIER' }, // Glacier after 1 month
{ days: 90, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 3 months { days: 90, storageClass: 'DEEP_ARCHIVE' } // Deep Archive after 3 months
] ]
}] }]
}) })
``` ```
@ -223,10 +222,10 @@ await storage.setLifecyclePolicy({
**After 1 year:** **After 1 year:**
``` ```
Standard (50TB): 50TB × $0.023/GB × 12 = $13,800/year Standard (50TB): 50TB × $0.023/GB × 12 = $13,800/year
Standard-IA (50TB): 50TB × $0.0125/GB × 12 = $7,500/year Standard-IA (50TB): 50TB × $0.0125/GB × 12 = $7,500/year
Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year Glacier (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Deep Archive (300TB): 300TB × $0.00099/GB × 12 = $3,564/year Deep Archive (300TB): 300TB × $0.00099/GB × 12 = $3,564/year
Total Storage Cost: $29,664/year Total Storage Cost: $29,664/year
Total with Operations: ~$34,000/year Total with Operations: ~$34,000/year
@ -250,16 +249,16 @@ Note: Retrieval costs may be significant if archived data is accessed frequently
### Efficient Cleanup ### Efficient Cleanup
```typescript ```typescript
// v4.0.0: Batch delete (1000 objects per request) // Batch delete (1000 objects per request)
const idsToDelete = [/* array of entity IDs */] const idsToDelete = [/* array of entity IDs */]
// Generate paths for both vector and metadata files // Generate paths for both vector and metadata files
const paths = idsToDelete.flatMap(id => { const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2) const shard = id.substring(0, 2)
return [ return [
`entities/nouns/vectors/${shard}/${id}.json`, `entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json` `entities/nouns/metadata/${shard}/${id}.json`
] ]
}) })
// Batch delete (much faster and cheaper than individual deletes) // Batch delete (much faster and cheaper than individual deletes)
@ -280,14 +279,14 @@ console.log('Active rules:', policy.rules)
// Example output: // Example output:
// { // {
// rules: [ // rules: [
// { // {
// id: 'optimize-vectors', // id: 'optimize-vectors',
// prefix: 'entities/nouns/vectors/', // prefix: 'entities/nouns/vectors/',
// status: 'Enabled', // status: 'Enabled',
// transitions: [...] // transitions: [...]
// } // }
// ] // ]
// } // }
``` ```
@ -396,6 +395,5 @@ await storage.enableIntelligentTiering('entities/', 'new-config')
--- ---
**Version**: v4.0.0
**Last Updated**: 2025-10-17 **Last Updated**: 2025-10-17
**Cloud Provider**: AWS S3 **Cloud Provider**: AWS S3

View file

@ -1,10 +1,9 @@
# Azure Blob Storage Cost Optimization Guide for Brainy v4.0.0 # Azure Blob Storage Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $107k/year to $5k/year at 500TB scale (**95% savings**) > **Cost Impact**: Reduce storage costs from $107k/year to $5k/year at 500TB scale (**95% savings**)
## Overview ## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for Azure Blob Storage, including manual tier management, lifecycle policies, and batch operations. Brainy provides enterprise-grade cost optimization features for Azure Blob Storage, including manual tier management, lifecycle policies, and batch operations.
## Cost Breakdown (Before Optimization) ## Cost Breakdown (Before Optimization)
@ -41,8 +40,8 @@ import { AzureBlobStorage } from '@soulcraft/brainy/storage'
// Initialize Brainy with Azure storage // Initialize Brainy with Azure storage
const storage = new AzureBlobStorage({ const storage = new AzureBlobStorage({
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING, connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data' containerName: 'brainy-data'
}) })
const brain = new Brainy({ storage }) const brain = new Brainy({ storage })
@ -50,19 +49,19 @@ await brain.init()
// Change tier for a single blob // Change tier for a single blob
await storage.changeBlobTier( await storage.changeBlobTier(
'entities/nouns/vectors/00/00123456-uuid.json', 'entities/nouns/vectors/00/00123456-uuid.json',
'Cool' 'Cool'
) )
// Batch tier changes (efficient for thousands of blobs) // Batch tier changes (efficient for thousands of blobs)
const blobsToMove = [ const blobsToMove = [
'entities/nouns/vectors/01/...', 'entities/nouns/vectors/01/...',
'entities/nouns/vectors/02/...', 'entities/nouns/vectors/02/...',
// ... up to thousands of blobs // ... up to thousands of blobs
] ]
await storage.batchChangeTier(blobsToMove, 'Cool') // Hot → Cool await storage.batchChangeTier(blobsToMove, 'Cool') // Hot → Cool
await storage.batchChangeTier(oldBlobs, 'Archive') // Cool → Archive await storage.batchChangeTier(oldBlobs, 'Archive') // Cool → Archive
``` ```
### Immediate Cost Impact ### Immediate Cost Impact
@ -90,54 +89,54 @@ Savings: $20,346/year (95% savings on moved data)
```typescript ```typescript
// Set lifecycle policy for automatic tier management // Set lifecycle policy for automatic tier management
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
name: 'optimizeVectors', name: 'optimizeVectors',
enabled: true, enabled: true,
type: 'Lifecycle', type: 'Lifecycle',
definition: { definition: {
filters: { filters: {
blobTypes: ['blockBlob'], blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/vectors/'] prefixMatch: ['entities/nouns/vectors/']
}, },
actions: { actions: {
baseBlob: { baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 }, tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 } tierToArchive: { daysAfterModificationGreaterThan: 90 }
} }
} }
} }
}, { }, {
name: 'optimizeMetadata', name: 'optimizeMetadata',
enabled: true, enabled: true,
type: 'Lifecycle', type: 'Lifecycle',
definition: { definition: {
filters: { filters: {
blobTypes: ['blockBlob'], blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/metadata/'] prefixMatch: ['entities/nouns/metadata/']
}, },
actions: { actions: {
baseBlob: { baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 }, tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 180 } tierToArchive: { daysAfterModificationGreaterThan: 180 }
} }
} }
} }
}, { }, {
name: 'cleanupOldSystemFiles', name: 'cleanupOldSystemFiles',
enabled: true, enabled: true,
type: 'Lifecycle', type: 'Lifecycle',
definition: { definition: {
filters: { filters: {
blobTypes: ['blockBlob'], blobTypes: ['blockBlob'],
prefixMatch: ['_system/'] prefixMatch: ['_system/']
}, },
actions: { actions: {
baseBlob: { baseBlob: {
delete: { daysAfterModificationGreaterThan: 365 } delete: { daysAfterModificationGreaterThan: 365 }
} }
} }
} }
}] }]
}) })
// Verify lifecycle policy // Verify lifecycle policy
@ -153,9 +152,9 @@ console.log('Lifecycle policy active:', policy.rules.length, 'rules')
- 30% of data in Archive tier (90+ days old) - 30% of data in Archive tier (90+ days old)
``` ```
Hot (150TB): 150TB × $0.0184/GB × 12 = $32,256/year Hot (150TB): 150TB × $0.0184/GB × 12 = $32,256/year
Cool (200TB): 200TB × $0.0115/GB × 12 = $26,880/year Cool (200TB): 200TB × $0.0115/GB × 12 = $26,880/year
Archive (150TB): 150TB × $0.00099/GB × 12 = $1,732/year Archive (150TB): 150TB × $0.00099/GB × 12 = $1,732/year
Total Storage Cost: $60,868/year Total Storage Cost: $60,868/year
Total with Operations: ~$65,500/year Total with Operations: ~$65,500/year
@ -170,23 +169,23 @@ Savings: $47,000/year (42%)
```typescript ```typescript
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
name: 'aggressiveArchival', name: 'aggressiveArchival',
enabled: true, enabled: true,
type: 'Lifecycle', type: 'Lifecycle',
definition: { definition: {
filters: { filters: {
blobTypes: ['blockBlob'], blobTypes: ['blockBlob'],
prefixMatch: ['entities/'] prefixMatch: ['entities/']
}, },
actions: { actions: {
baseBlob: { baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 14 }, tierToCool: { daysAfterModificationGreaterThan: 14 },
tierToArchive: { daysAfterModificationGreaterThan: 30 } tierToArchive: { daysAfterModificationGreaterThan: 30 }
} }
} }
} }
}] }]
}) })
``` ```
@ -194,9 +193,9 @@ await storage.setLifecyclePolicy({
**After 6 months:** **After 6 months:**
``` ```
Hot (50TB): 50TB × $0.0184/GB × 12 = $10,752/year Hot (50TB): 50TB × $0.0184/GB × 12 = $10,752/year
Cool (100TB): 100TB × $0.0115/GB × 12 = $13,440/year Cool (100TB): 100TB × $0.0115/GB × 12 = $13,440/year
Archive (350TB): 350TB × $0.00099/GB × 12 = $4,039/year Archive (350TB): 350TB × $0.00099/GB × 12 = $4,039/year
Total Storage Cost: $28,231/year Total Storage Cost: $28,231/year
Total with Operations: ~$33,000/year Total with Operations: ~$33,000/year
@ -211,41 +210,41 @@ Warning: Archive rehydration takes 1-15 hours
```typescript ```typescript
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
// Vectors: Keep in Hot/Cool for search performance // Vectors: Keep in Hot/Cool for search performance
name: 'vectors-moderate', name: 'vectors-moderate',
enabled: true, enabled: true,
type: 'Lifecycle', type: 'Lifecycle',
definition: { definition: {
filters: { filters: {
blobTypes: ['blockBlob'], blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/vectors/', 'entities/verbs/vectors/'] prefixMatch: ['entities/nouns/vectors/', 'entities/verbs/vectors/']
}, },
actions: { actions: {
baseBlob: { baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 60 } tierToCool: { daysAfterModificationGreaterThan: 60 }
// Don't archive vectors - keep searchable // Don't archive vectors - keep searchable
} }
} }
} }
}, { }, {
// Metadata: Aggressive archival // Metadata: Aggressive archival
name: 'metadata-aggressive', name: 'metadata-aggressive',
enabled: true, enabled: true,
type: 'Lifecycle', type: 'Lifecycle',
definition: { definition: {
filters: { filters: {
blobTypes: ['blockBlob'], blobTypes: ['blockBlob'],
prefixMatch: ['entities/nouns/metadata/', 'entities/verbs/metadata/'] prefixMatch: ['entities/nouns/metadata/', 'entities/verbs/metadata/']
}, },
actions: { actions: {
baseBlob: { baseBlob: {
tierToCool: { daysAfterModificationGreaterThan: 30 }, tierToCool: { daysAfterModificationGreaterThan: 30 },
tierToArchive: { daysAfterModificationGreaterThan: 90 } tierToArchive: { daysAfterModificationGreaterThan: 90 }
} }
} }
} }
}] }]
}) })
``` ```
@ -253,16 +252,16 @@ await storage.setLifecyclePolicy({
**Vectors (300TB):** **Vectors (300TB):**
``` ```
Hot (90TB): 90TB × $0.0184/GB × 12 = $19,354/year Hot (90TB): 90TB × $0.0184/GB × 12 = $19,354/year
Cool (210TB): 210TB × $0.0115/GB × 12 = $28,980/year Cool (210TB): 210TB × $0.0115/GB × 12 = $28,980/year
Subtotal: $48,334/year Subtotal: $48,334/year
``` ```
**Metadata (200TB):** **Metadata (200TB):**
``` ```
Hot (30TB): 30TB × $0.0184/GB × 12 = $6,451/year Hot (30TB): 30TB × $0.0184/GB × 12 = $6,451/year
Cool (70TB): 70TB × $0.0115/GB × 12 = $9,660/year Cool (70TB): 70TB × $0.0115/GB × 12 = $9,660/year
Archive (100TB): 100TB × $0.00099/GB × 12 = $1,158/year Archive (100TB): 100TB × $0.00099/GB × 12 = $1,158/year
Subtotal: $17,269/year Subtotal: $17,269/year
``` ```
@ -288,8 +287,8 @@ Subtotal: $17,269/year
```typescript ```typescript
// Rehydrate blob from Archive to Hot (high priority) // Rehydrate blob from Archive to Hot (high priority)
await storage.rehydrateBlob( await storage.rehydrateBlob(
'entities/nouns/vectors/00/00123456-uuid.json', 'entities/nouns/vectors/00/00123456-uuid.json',
'High' // 'Standard' or 'High' priority 'High' // 'Standard' or 'High' priority
) )
// Rehydration time: // Rehydration time:
@ -309,7 +308,7 @@ console.log('Archive status:', metadata.archiveStatus)
const blobsToRehydrate = [/* array of blob paths */] const blobsToRehydrate = [/* array of blob paths */]
for (const blobPath of blobsToRehydrate) { for (const blobPath of blobsToRehydrate) {
await storage.rehydrateBlob(blobPath, 'High') await storage.rehydrateBlob(blobPath, 'High')
} }
// Wait for rehydration to complete (1-15 hours) // Wait for rehydration to complete (1-15 hours)
@ -333,15 +332,15 @@ Examples:
### Efficient Bulk Deletions ### Efficient Bulk Deletions
```typescript ```typescript
// v4.0.0: Batch delete (256 blobs per request) // Batch delete (256 blobs per request)
const idsToDelete = [/* array of entity IDs */] const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => { const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2) const shard = id.substring(0, 2)
return [ return [
`entities/nouns/vectors/${shard}/${id}.json`, `entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json` `entities/nouns/metadata/${shard}/${id}.json`
] ]
}) })
// Batch delete via BlobBatchClient // Batch delete via BlobBatchClient
@ -375,14 +374,14 @@ console.log('Active rules:', policy.rules)
// Example output: // Example output:
// { // {
// rules: [ // rules: [
// { // {
// name: 'optimizeVectors', // name: 'optimizeVectors',
// enabled: true, // enabled: true,
// type: 'Lifecycle', // type: 'Lifecycle',
// definition: {...} // definition: {...}
// } // }
// ] // ]
// } // }
``` ```
@ -452,8 +451,8 @@ Monthly cost trend: Decreasing 5-8% per month as data transitions
// Check lifecycle policy status // Check lifecycle policy status
const policy = await storage.getLifecyclePolicy() const policy = await storage.getLifecyclePolicy()
console.log('Policy rules:', policy.rules.map(r => ({ console.log('Policy rules:', policy.rules.map(r => ({
name: r.name, name: r.name,
enabled: r.enabled enabled: r.enabled
}))) })))
// Azure lifecycle policies run once per day // Azure lifecycle policies run once per day
@ -471,9 +470,9 @@ await storage.rehydrateBlob(blobPath, 'High')
// Wait for rehydration (check status) // Wait for rehydration (check status)
let status let status
do { do {
await new Promise(resolve => setTimeout(resolve, 60000)) // Wait 1 minute await new Promise(resolve => setTimeout(resolve, 60000)) // Wait 1 minute
const metadata = await storage.getBlobMetadata(blobPath) const metadata = await storage.getBlobMetadata(blobPath)
status = metadata.archiveStatus status = metadata.archiveStatus
} while (status && status.includes('pending')) } while (status && status.includes('pending'))
// Now access the blob // Now access the blob
@ -494,8 +493,8 @@ const data = await storage.get(blobPath)
```typescript ```typescript
const storage = new AzureBlobStorage({ const storage = new AzureBlobStorage({
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING, connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
containerName: 'brainy-data' containerName: 'brainy-data'
}) })
``` ```
@ -503,9 +502,9 @@ const storage = new AzureBlobStorage({
```typescript ```typescript
const storage = new AzureBlobStorage({ const storage = new AzureBlobStorage({
accountName: process.env.AZURE_STORAGE_ACCOUNT, accountName: process.env.AZURE_STORAGE_ACCOUNT,
accountKey: process.env.AZURE_STORAGE_KEY, accountKey: process.env.AZURE_STORAGE_KEY,
containerName: 'brainy-data' containerName: 'brainy-data'
}) })
``` ```
@ -513,9 +512,9 @@ const storage = new AzureBlobStorage({
```typescript ```typescript
const storage = new AzureBlobStorage({ const storage = new AzureBlobStorage({
sasToken: process.env.AZURE_STORAGE_SAS_TOKEN, sasToken: process.env.AZURE_STORAGE_SAS_TOKEN,
accountName: process.env.AZURE_STORAGE_ACCOUNT, accountName: process.env.AZURE_STORAGE_ACCOUNT,
containerName: 'brainy-data' containerName: 'brainy-data'
}) })
``` ```
@ -549,6 +548,5 @@ const storage = new AzureBlobStorage({
--- ---
**Version**: v4.0.0
**Last Updated**: 2025-10-17 **Last Updated**: 2025-10-17
**Cloud Provider**: Azure Blob Storage **Cloud Provider**: Azure Blob Storage

View file

@ -1,10 +1,9 @@
# Cloudflare R2 Cost Optimization Guide for Brainy v4.0.0 # Cloudflare R2 Cost Optimization Guide for Brainy
> **Cost Impact**: $0 egress fees + 96% storage savings = Lowest cloud storage costs > **Cost Impact**: $0 egress fees + 96% storage savings = Lowest cloud storage costs
## Overview ## Overview
Cloudflare R2 is an S3-compatible object storage with **zero egress fees**, making it ideal for high-traffic applications. Brainy v4.0.0 fully supports R2 with lifecycle policies and batch operations. Cloudflare R2 is an S3-compatible object storage with **zero egress fees**, making it ideal for high-traffic applications. Brainy fully supports R2 with lifecycle policies and batch operations.
## Cost Breakdown ## Cost Breakdown
@ -49,7 +48,7 @@ Savings vs AWS: $156,000/year (62%)
- Cloudflare plans to add infrequent access tiers - Cloudflare plans to add infrequent access tiers
**When lifecycle features arrive:** **When lifecycle features arrive:**
- Brainy v4.0.0 is already prepared with `setLifecyclePolicy()` support - Brainy is already prepared with `setLifecyclePolicy()` support
- Will work seamlessly once Cloudflare enables lifecycle management - Will work seamlessly once Cloudflare enables lifecycle management
## Strategy 1: Use R2 Standard (Current Best Practice) ## Strategy 1: Use R2 Standard (Current Best Practice)
@ -62,11 +61,11 @@ import { S3CompatibleStorage } from '@soulcraft/brainy/storage'
// R2 uses S3-compatible API // R2 uses S3-compatible API
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`, endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'my-brainy-data', bucket: 'my-brainy-data',
region: 'auto', // R2 uses 'auto' region region: 'auto', // R2 uses 'auto' region
accessKeyId: process.env.R2_ACCESS_KEY_ID, accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
}) })
const brain = new Brainy({ storage }) const brain = new Brainy({ storage })
@ -96,23 +95,23 @@ Total: $90,081/year
```typescript ```typescript
// Cloudflare Worker (runs at edge) // Cloudflare Worker (runs at edge)
export default { export default {
async fetch(request, env) { async fetch(request, env) {
// Initialize Brainy with R2 // Initialize Brainy with R2
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
endpoint: env.R2_ENDPOINT, endpoint: env.R2_ENDPOINT,
bucket: env.R2_BUCKET, bucket: env.R2_BUCKET,
accessKeyId: env.R2_ACCESS_KEY, accessKeyId: env.R2_ACCESS_KEY,
secretAccessKey: env.R2_SECRET_KEY, secretAccessKey: env.R2_SECRET_KEY,
region: 'auto' region: 'auto'
}) })
const brain = new Brainy({ storage }) const brain = new Brainy({ storage })
await brain.init() await brain.init()
// Process at edge (no origin server needed) // Process at edge (no origin server needed)
const results = await brain.search(request.query) const results = await brain.search(request.query)
return new Response(JSON.stringify(results)) return new Response(JSON.stringify(results))
} }
} }
``` ```
@ -121,18 +120,18 @@ export default {
``` ```
R2 Storage (500TB): $90,000/year R2 Storage (500TB): $90,000/year
Workers (10M requests/day): Workers (10M requests/day):
- First 100k requests/day: FREE - First 100k requests/day: FREE
- Additional 350M requests/month: $1,750/year - Additional 350M requests/month: $1,750/year
- CPU time (50ms avg): $5,000/year - CPU time (50ms avg): $5,000/year
Total: $96,750/year Total: $96,750/year
vs Traditional Setup (AWS S3 + EC2 + CloudFront): vs Traditional Setup (AWS S3 + EC2 + CloudFront):
- S3: $138,000/year - S3: $138,000/year
- EC2 (t3.xlarge × 4): $24,000/year - EC2 (t3.xlarge × 4): $24,000/year
- CloudFront egress: $50,000/year - CloudFront egress: $50,000/year
- Load balancer: $3,000/year - Load balancer: $3,000/year
Total: $215,000/year Total: $215,000/year
Savings: $118,250/year (55%) Savings: $118,250/year (55%)
``` ```
@ -144,20 +143,20 @@ Savings: $118,250/year (55%)
```typescript ```typescript
// Use R2 for frequently accessed data (zero egress) // Use R2 for frequently accessed data (zero egress)
const hotStorage = new S3CompatibleStorage({ const hotStorage = new S3CompatibleStorage({
endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`, endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
bucket: 'brainy-hot', bucket: 'brainy-hot',
region: 'auto', region: 'auto',
accessKeyId: process.env.R2_ACCESS_KEY_ID, accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
}) })
// Use AWS S3 with Intelligent-Tiering for cold data // Use AWS S3 with Intelligent-Tiering for cold data
const coldStorage = new S3CompatibleStorage({ const coldStorage = new S3CompatibleStorage({
endpoint: 's3.amazonaws.com', endpoint: 's3.amazonaws.com',
bucket: 'brainy-archive', bucket: 'brainy-archive',
region: 'us-east-1', region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID, accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}) })
// Initialize separate Brainy instances or implement tiering logic // Initialize separate Brainy instances or implement tiering logic
@ -167,17 +166,17 @@ const coldStorage = new S3CompatibleStorage({
``` ```
R2 Hot Data (300TB): R2 Hot Data (300TB):
Storage: 300TB × $0.015/GB × 12 = $54,000/year Storage: 300TB × $0.015/GB × 12 = $54,000/year
Egress: $0 Egress: $0
S3 Cold Data (200TB with lifecycle → Deep Archive): S3 Cold Data (200TB with lifecycle → Deep Archive):
Storage: 200TB × $0.00099/GB × 12 = $2,376/year Storage: 200TB × $0.00099/GB × 12 = $2,376/year
Ops: $1,000/year Ops: $1,000/year
Total: $57,376/year Total: $57,376/year
vs All AWS S3: vs All AWS S3:
- S3 storage + egress: $251,000/year - S3 storage + egress: $251,000/year
Savings: $193,624/year (77%) Savings: $193,624/year (77%)
``` ```
@ -187,15 +186,15 @@ Savings: $193,624/year (77%)
### Efficient Bulk Deletions ### Efficient Bulk Deletions
```typescript ```typescript
// v4.0.0: R2 supports S3 batch delete API // R2 supports S3 batch delete API
const idsToDelete = [/* array of entity IDs */] const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => { const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2) const shard = id.substring(0, 2)
return [ return [
`entities/nouns/vectors/${shard}/${id}.json`, `entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json` `entities/nouns/metadata/${shard}/${id}.json`
] ]
}) })
// Batch delete (1000 objects per request) // Batch delete (1000 objects per request)
@ -231,10 +230,10 @@ https://storage.yourdomain.com/entities/nouns/vectors/...
```typescript ```typescript
// Worker triggered on R2 object upload // Worker triggered on R2 object upload
export default { export default {
async fetch(request, env) { async fetch(request, env) {
// Process new objects automatically // Process new objects automatically
// E.g., index new entities, generate thumbnails, etc. // E.g., index new entities, generate thumbnails, etc.
} }
} }
// Cost: Only pay for Worker execution (no polling needed) // Cost: Only pay for Worker execution (no polling needed)
@ -244,7 +243,7 @@ export default {
```typescript ```typescript
// Generate presigned URL for direct browser uploads // Generate presigned URL for direct browser uploads
const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry
// Client uploads directly to R2 (no server bandwidth used) // Client uploads directly to R2 (no server bandwidth used)
``` ```
@ -255,7 +254,7 @@ const url = await storage.getPresignedUrl('upload-path', 3600) // 1 hour expiry
```typescript ```typescript
const status = await storage.getStorageStatus() const status = await storage.getStorageStatus()
console.log('Storage type:', status.type) // 's3-compatible' console.log('Storage type:', status.type) // 's3-compatible'
console.log('Bucket:', status.details.bucket) console.log('Bucket:', status.details.bucket)
console.log('Endpoint:', status.details.endpoint) console.log('Endpoint:', status.details.endpoint)
``` ```
@ -318,20 +317,20 @@ Egress: Unlimited (always free)
```bash ```bash
# Install rclone # Install rclone
brew install rclone # or apt-get install rclone brew install rclone # or apt-get install rclone
# Configure S3 source # Configure S3 source
rclone config create s3-source s3 \ rclone config create s3-source s3 \
access_key_id=$AWS_ACCESS_KEY \ access_key_id=$AWS_ACCESS_KEY \
secret_access_key=$AWS_SECRET_KEY \ secret_access_key=$AWS_SECRET_KEY \
region=us-east-1 region=us-east-1
# Configure R2 destination # Configure R2 destination
rclone config create r2-dest s3 \ rclone config create r2-dest s3 \
access_key_id=$R2_ACCESS_KEY \ access_key_id=$R2_ACCESS_KEY \
secret_access_key=$R2_SECRET_KEY \ secret_access_key=$R2_SECRET_KEY \
endpoint=https://$R2_ACCOUNT_ID.r2.cloudflarestorage.com \ endpoint=https://$R2_ACCOUNT_ID.r2.cloudflarestorage.com \
region=auto region=auto
# Copy data # Copy data
rclone copy s3-source:my-bucket r2-dest:my-bucket --progress rclone copy s3-source:my-bucket r2-dest:my-bucket --progress
@ -361,17 +360,17 @@ ROI: 3.4 months
### Prepared for Future Features ### Prepared for Future Features
```typescript ```typescript
// Brainy v4.0.0 is ready for R2 lifecycle features // Brainy is ready for R2 lifecycle features
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
id: 'archive-old-data', id: 'archive-old-data',
prefix: 'entities/', prefix: 'entities/',
status: 'Enabled', status: 'Enabled',
transitions: [ transitions: [
{ days: 30, storageClass: 'INFREQUENT_ACCESS' }, // When available { days: 30, storageClass: 'INFREQUENT_ACCESS' }, // When available
{ days: 90, storageClass: 'ARCHIVE' } { days: 90, storageClass: 'ARCHIVE' }
] ]
}] }]
}) })
// Expected cost impact (when lifecycle is available): // Expected cost impact (when lifecycle is available):
@ -388,11 +387,11 @@ await storage.setLifecyclePolicy({
```typescript ```typescript
// Ensure correct endpoint format // Ensure correct endpoint format
const storage = new S3CompatibleStorage({ const storage = new S3CompatibleStorage({
endpoint: `https://${accountId}.r2.cloudflarestorage.com`, endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
// NOT: `https://r2.cloudflarestorage.com/${accountId}` // NOT: `https://r2.cloudflarestorage.com/${accountId}`
region: 'auto', // R2 requires 'auto' region: 'auto', // R2 requires 'auto'
forcePathStyle: false // R2 uses virtual-hosted-style forcePathStyle: false // R2 uses virtual-hosted-style
}) })
``` ```
@ -446,7 +445,6 @@ const storage = new S3CompatibleStorage({
--- ---
**Version**: v4.0.0
**Last Updated**: 2025-10-17 **Last Updated**: 2025-10-17
**Cloud Provider**: Cloudflare R2 **Cloud Provider**: Cloudflare R2
**Key Advantage**: **$0 egress fees forever** **Key Advantage**: **$0 egress fees forever**

View file

@ -1,10 +1,9 @@
# Google Cloud Storage Cost Optimization Guide for Brainy v4.0.0 # Google Cloud Storage Cost Optimization Guide for Brainy
> **Cost Impact**: Reduce storage costs from $138k/year to $8.3k/year at 500TB scale (**94% savings**) > **Cost Impact**: Reduce storage costs from $138k/year to $8.3k/year at 500TB scale (**94% savings**)
## Overview ## Overview
Brainy v4.0.0 provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management. Brainy provides enterprise-grade cost optimization features for Google Cloud Storage, including lifecycle policies and Autoclass for automatic tier management.
## Cost Breakdown (Before Optimization) ## Cost Breakdown (Before Optimization)
@ -35,8 +34,8 @@ import { GcsStorage } from '@soulcraft/brainy/storage'
// Initialize Brainy with GCS storage // Initialize Brainy with GCS storage
const storage = new GcsStorage({ const storage = new GcsStorage({
bucketName: 'my-brainy-data', bucketName: 'my-brainy-data',
keyFilename: './service-account.json' // Or use ADC keyFilename: './service-account.json' // Or use ADC
}) })
const brain = new Brainy({ storage }) const brain = new Brainy({ storage })
@ -44,16 +43,16 @@ await brain.init()
// Set lifecycle policy for automatic archival // Set lifecycle policy for automatic archival
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
condition: { age: 30 }, condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' } action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, { }, {
condition: { age: 90 }, condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' } action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, { }, {
condition: { age: 365 }, condition: { age: 365 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' } action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}] }]
}) })
// Verify lifecycle policy // Verify lifecycle policy
@ -70,10 +69,10 @@ console.log('Lifecycle policy active:', policy.rules.length, 'rules')
- 10% of data 365+ days old (Archive) - 10% of data 365+ days old (Archive)
``` ```
Standard (200TB): 200TB × $0.020/GB × 12 = $48,000/year Standard (200TB): 200TB × $0.020/GB × 12 = $48,000/year
Nearline (150TB): 150TB × $0.010/GB × 12 = $18,000/year Nearline (150TB): 150TB × $0.010/GB × 12 = $18,000/year
Coldline (100TB): 100TB × $0.004/GB × 12 = $4,800/year Coldline (100TB): 100TB × $0.004/GB × 12 = $4,800/year
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
Total Storage Cost: $71,520/year Total Storage Cost: $71,520/year
Total with Operations: ~$76,500/year Total with Operations: ~$76,500/year
@ -89,7 +88,7 @@ Savings: $66,500/year (46%)
```typescript ```typescript
// Enable Autoclass for automatic tier management // Enable Autoclass for automatic tier management
await storage.enableAutoclass({ await storage.enableAutoclass({
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
}) })
// Benefits: // Benefits:
@ -116,10 +115,10 @@ await storage.enableAutoclass({
- 40% Archive (cold data) - 40% Archive (cold data)
``` ```
Standard (50TB): 50TB × $0.020/GB × 12 = $12,000/year Standard (50TB): 50TB × $0.020/GB × 12 = $12,000/year
Nearline (75TB): 75TB × $0.010/GB × 12 = $9,000/year Nearline (75TB): 75TB × $0.010/GB × 12 = $9,000/year
Coldline (175TB): 175TB × $0.004/GB × 12 = $8,400/year Coldline (175TB): 175TB × $0.004/GB × 12 = $8,400/year
Archive (200TB): 200TB × $0.0012/GB × 12 = $2,880/year Archive (200TB): 200TB × $0.0012/GB × 12 = $2,880/year
Total Storage Cost: $32,280/year Total Storage Cost: $32,280/year
Total with Operations: ~$37,000/year Total with Operations: ~$37,000/year
@ -133,24 +132,24 @@ Savings vs Standard: $106,000/year (74%)
```typescript ```typescript
// Enable Autoclass for vectors (frequently searched) // Enable Autoclass for vectors (frequently searched)
await storage.enableAutoclass({ await storage.enableAutoclass({
terminalStorageClass: 'COLDLINE' // Don't archive vectors deeply terminalStorageClass: 'COLDLINE' // Don't archive vectors deeply
}) })
// Set lifecycle policy for metadata (less frequently accessed) // Set lifecycle policy for metadata (less frequently accessed)
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
condition: { age: 30, matchesPrefix: ['entities/nouns/metadata/'] }, condition: { age: 30, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' } action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, { }, {
condition: { age: 60, matchesPrefix: ['entities/nouns/metadata/'] }, condition: { age: 60, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' } action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, { }, {
condition: { age: 180, matchesPrefix: ['entities/nouns/metadata/'] }, condition: { age: 180, matchesPrefix: ['entities/nouns/metadata/'] },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' } action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}, { }, {
condition: { age: 730, matchesPrefix: ['_system/'] }, condition: { age: 730, matchesPrefix: ['_system/'] },
action: { type: 'Delete' } // Delete old system files after 2 years action: { type: 'Delete' } // Delete old system files after 2 years
}] }]
}) })
``` ```
@ -158,18 +157,18 @@ await storage.setLifecyclePolicy({
**Vectors (300TB with Autoclass):** **Vectors (300TB with Autoclass):**
``` ```
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
Nearline (45TB): 45TB × $0.010/GB × 12 = $5,400/year Nearline (45TB): 45TB × $0.010/GB × 12 = $5,400/year
Coldline (225TB): 225TB × $0.004/GB × 12 = $10,800/year Coldline (225TB): 225TB × $0.004/GB × 12 = $10,800/year
Subtotal: $23,400/year Subtotal: $23,400/year
``` ```
**Metadata (200TB with Lifecycle Policy):** **Metadata (200TB with Lifecycle Policy):**
``` ```
Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year Standard (30TB): 30TB × $0.020/GB × 12 = $7,200/year
Nearline (40TB): 40TB × $0.010/GB × 12 = $4,800/year Nearline (40TB): 40TB × $0.010/GB × 12 = $4,800/year
Coldline (80TB): 80TB × $0.004/GB × 12 = $3,840/year Coldline (80TB): 80TB × $0.004/GB × 12 = $3,840/year
Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year Archive (50TB): 50TB × $0.0012/GB × 12 = $720/year
Subtotal: $16,560/year Subtotal: $16,560/year
``` ```
@ -182,16 +181,16 @@ Subtotal: $16,560/year
```typescript ```typescript
await storage.setLifecyclePolicy({ await storage.setLifecyclePolicy({
rules: [{ rules: [{
condition: { age: 14 }, condition: { age: 14 },
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' } action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
}, { }, {
condition: { age: 30 }, condition: { age: 30 },
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' } action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
}, { }, {
condition: { age: 90 }, condition: { age: 90 },
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' } action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
}] }]
}) })
// Note: Archive class has 365-day minimum storage duration // Note: Archive class has 365-day minimum storage duration
@ -202,10 +201,10 @@ await storage.setLifecyclePolicy({
**After 1 year:** **After 1 year:**
``` ```
Standard (25TB): 25TB × $0.020/GB × 12 = $6,000/year Standard (25TB): 25TB × $0.020/GB × 12 = $6,000/year
Nearline (50TB): 50TB × $0.010/GB × 12 = $6,000/year Nearline (50TB): 50TB × $0.010/GB × 12 = $6,000/year
Coldline (75TB): 75TB × $0.004/GB × 12 = $3,600/year Coldline (75TB): 75TB × $0.004/GB × 12 = $3,600/year
Archive (350TB): 350TB × $0.0012/GB × 12 = $5,040/year Archive (350TB): 350TB × $0.0012/GB × 12 = $5,040/year
Total Storage Cost: $20,640/year Total Storage Cost: $20,640/year
Total with Operations: ~$25,500/year Total with Operations: ~$25,500/year
@ -241,15 +240,15 @@ Warning: High retrieval costs if archived data is accessed frequently
### Efficient Cleanup ### Efficient Cleanup
```typescript ```typescript
// v4.0.0: Batch delete (100 objects per request for GCS) // Batch delete (100 objects per request for GCS)
const idsToDelete = [/* array of entity IDs */] const idsToDelete = [/* array of entity IDs */]
const paths = idsToDelete.flatMap(id => { const paths = idsToDelete.flatMap(id => {
const shard = id.substring(0, 2) const shard = id.substring(0, 2)
return [ return [
`entities/nouns/vectors/${shard}/${id}.json`, `entities/nouns/vectors/${shard}/${id}.json`,
`entities/nouns/metadata/${shard}/${id}.json` `entities/nouns/metadata/${shard}/${id}.json`
] ]
}) })
// Batch delete // Batch delete
@ -271,9 +270,9 @@ console.log('Terminal class:', status.terminalStorageClass)
// Example output: // Example output:
// { // {
// enabled: true, // enabled: true,
// terminalStorageClass: 'ARCHIVE', // terminalStorageClass: 'ARCHIVE',
// toggleTime: '2025-01-15T10:30:00Z' // toggleTime: '2025-01-15T10:30:00Z'
// } // }
``` ```
@ -336,7 +335,7 @@ Monthly cost trend: Decreasing 8-12% per month as data ages into cheaper classes
// Check Autoclass status // Check Autoclass status
const status = await storage.getAutoclassStatus() const status = await storage.getAutoclassStatus()
if (!status.enabled) { if (!status.enabled) {
await storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' }) await storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' })
} }
// Autoclass requires 24-48 hours for initial transitions // Autoclass requires 24-48 hours for initial transitions
@ -365,8 +364,8 @@ if (!status.enabled) {
```typescript ```typescript
// Use ADC instead of service account key file // Use ADC instead of service account key file
const storage = new GcsStorage({ const storage = new GcsStorage({
bucketName: 'my-brainy-data' bucketName: 'my-brainy-data'
// No keyFilename needed - uses ADC automatically // No keyFilename needed - uses ADC automatically
}) })
// ADC authentication order: // ADC authentication order:
@ -417,6 +416,5 @@ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
--- ---
**Version**: v4.0.0
**Last Updated**: 2025-10-17 **Last Updated**: 2025-10-17
**Cloud Provider**: Google Cloud Storage **Cloud Provider**: Google Cloud Storage

View file

@ -1,6 +1,6 @@
# Transaction System # Transaction System
**Status:** ✅ Production Ready (v5.8.0+) **Status:** ✅ Production Ready
## Overview ## Overview
@ -17,11 +17,11 @@ Brainy's transaction system provides **atomic operations** with automatic rollba
``` ```
User Code (brain.add(), brain.update(), etc.) User Code (brain.add(), brain.update(), etc.)
Transaction Manager (orchestration) Transaction Manager (orchestration)
Operations (SaveNounMetadataOperation, SaveNounOperation, etc.) Operations (SaveNounMetadataOperation, SaveNounOperation, etc.)
Storage Adapter (COW, sharding, type-aware routing) Storage Adapter (COW, sharding, type-aware routing)
``` ```
@ -32,8 +32,8 @@ Every write operation in Brainy automatically uses transactions:
```typescript ```typescript
// Internally, this uses a transaction // Internally, this uses a transaction
const id = await brain.add({ const id = await brain.add({
data: { name: 'Alice', role: 'Engineer' }, data: { name: 'Alice', role: 'Engineer' },
type: NounType.Person type: NounType.Person
}) })
``` ```
@ -51,19 +51,19 @@ Each operation implements both **execute** and **undo**:
```typescript ```typescript
class SaveNounMetadataOperation { class SaveNounMetadataOperation {
async execute(): Promise<void> { async execute(): Promise<void> {
// Save new metadata // Save new metadata
await this.storage.saveNounMetadata(this.id, this.metadata) await this.storage.saveNounMetadata(this.id, this.metadata)
} }
async undo(): Promise<void> { async undo(): Promise<void> {
// Restore previous metadata (or delete if new entity) // Restore previous metadata (or delete if new entity)
if (this.previousMetadata) { if (this.previousMetadata) {
await this.storage.saveNounMetadata(this.id, this.previousMetadata) await this.storage.saveNounMetadata(this.id, this.previousMetadata)
} else { } else {
await this.storage.deleteNounMetadata(this.id) await this.storage.deleteNounMetadata(this.id)
} }
} }
} }
``` ```
@ -87,8 +87,8 @@ await brain.cow.checkout('feature-branch')
// Add entity (uses transaction on this branch) // Add entity (uses transaction on this branch)
const id = await brain.add({ const id = await brain.add({
data: { name: 'Feature Entity' }, data: { name: 'Feature Entity' },
type: NounType.Thing type: NounType.Thing
}) })
// On rollback: Branch remains clean, no partial commits // On rollback: Branch remains clean, no partial commits
@ -123,7 +123,7 @@ await brain.relate({ from: id1, to: id2, type: VerbType.RelatesTo })
- Transaction operations don't need to know about shards - Transaction operations don't need to know about shards
- Rollback works across all shards involved - Rollback works across all shards involved
### ID-First Storage (v6.0.0+) ### ID-First Storage
✅ **Fully Compatible** ✅ **Fully Compatible**
@ -132,27 +132,27 @@ Transactions work with direct ID-first paths - no type routing needed!
```typescript ```typescript
// Entities stored with direct ID-first paths // Entities stored with direct ID-first paths
const personId = await brain.add({ const personId = await brain.add({
data: { name: 'John Doe' }, data: { name: 'John Doe' },
type: NounType.Person // → entities/nouns/{shard}/{id}/metadata.json type: NounType.Person // → entities/nouns/{shard}/{id}/metadata.json
}) })
const orgId = await brain.add({ const orgId = await brain.add({
data: { name: 'Acme Corp' }, data: { name: 'Acme Corp' },
type: NounType.Organization // → entities/nouns/{shard}/{id}/metadata.json type: NounType.Organization // → entities/nouns/{shard}/{id}/metadata.json
}) })
// Type changes handled atomically (type is just metadata) // Type changes handled atomically (type is just metadata)
await brain.update({ await brain.update({
id: personId, id: personId,
type: NounType.Organization, // Type change type: NounType.Organization, // Type change
data: { name: 'Doe Corp' } data: { name: 'Doe Corp' }
}) })
``` ```
**How It Works:** **How It Works:**
- Type information stored in metadata.noun field - Type information stored in metadata.noun field
- Storage layer uses O(1) ID-first path construction - Storage layer uses O(1) ID-first path construction
- No type cache needed (removed in v6.0.0) - No type cache needed (removed in a previous version)
- Type counters adjusted on commit/rollback - Type counters adjusted on commit/rollback
- 40x faster on cloud storage (eliminates 42-type search) - 40x faster on cloud storage (eliminates 42-type search)
@ -165,10 +165,10 @@ Transactions work with distributed/remote storage:
```typescript ```typescript
// Works with S3, Azure, GCS, etc. // Works with S3, Azure, GCS, etc.
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 's3Compatible', type: 's3Compatible',
config: { /* S3 config */ } config: { /* S3 config */ }
} }
}) })
// Transactions ensure atomicity at write coordinator level // Transactions ensure atomicity at write coordinator level
@ -199,8 +199,8 @@ await brain.init()
// Automatically uses transaction // Automatically uses transaction
const id = await brain.add({ const id = await brain.add({
data: { name: 'Alice', role: 'Engineer' }, data: { name: 'Alice', role: 'Engineer' },
type: NounType.Person type: NounType.Person
}) })
// If add fails, all changes rolled back automatically // If add fails, all changes rolled back automatically
@ -211,15 +211,15 @@ const id = await brain.add({
```typescript ```typescript
// Original entity // Original entity
const id = await brain.add({ const id = await brain.add({
data: { name: 'John Smith', category: 'individual' }, data: { name: 'John Smith', category: 'individual' },
type: NounType.Person type: NounType.Person
}) })
// Update with type change (atomic) // Update with type change (atomic)
await brain.update({ await brain.update({
id, id,
type: NounType.Organization, // Type change type: NounType.Organization, // Type change
data: { name: 'Smith Corp', category: 'business' } data: { name: 'Smith Corp', category: 'business' }
}) })
// If update fails, original type and data restored // If update fails, original type and data restored
@ -229,20 +229,20 @@ await brain.update({
```typescript ```typescript
const personId = await brain.add({ const personId = await brain.add({
data: { name: 'Alice' }, data: { name: 'Alice' },
type: NounType.Person type: NounType.Person
}) })
const projectId = await brain.add({ const projectId = await brain.add({
data: { name: 'Project X' }, data: { name: 'Project X' },
type: NounType.Thing type: NounType.Thing
}) })
// Create relationship (atomic) // Create relationship (atomic)
await brain.relate({ await brain.relate({
from: personId, from: personId,
to: projectId, to: projectId,
type: VerbType.WorksOn type: VerbType.WorksOn
}) })
// If relate fails, no partial relationship created // If relate fails, no partial relationship created
@ -253,10 +253,10 @@ await brain.relate({
```typescript ```typescript
// Multiple operations, all atomic // Multiple operations, all atomic
for (let i = 0; i < 100; i++) { for (let i = 0; i < 100; i++) {
await brain.add({ await brain.add({
data: { name: `Entity ${i}`, index: i }, data: { name: `Entity ${i}`, index: i },
type: NounType.Thing type: NounType.Thing
}) })
} }
// Each add() is a separate transaction // Each add() is a separate transaction
@ -267,19 +267,19 @@ for (let i = 0; i < 100; i++) {
```typescript ```typescript
const personId = await brain.add({ const personId = await brain.add({
data: { name: 'Bob' }, data: { name: 'Bob' },
type: NounType.Person type: NounType.Person
}) })
const projectId = await brain.add({ const projectId = await brain.add({
data: { name: 'Project Y' }, data: { name: 'Project Y' },
type: NounType.Thing type: NounType.Thing
}) })
await brain.relate({ await brain.relate({
from: personId, from: personId,
to: projectId, to: projectId,
type: VerbType.WorksOn type: VerbType.WorksOn
}) })
// Delete person (atomic - deletes entity + relationships) // Delete person (atomic - deletes entity + relationships)
@ -294,15 +294,15 @@ Transactions automatically handle errors and rollback:
```typescript ```typescript
try { try {
await brain.add({ await brain.add({
data: { name: 'Test Entity' }, data: { name: 'Test Entity' },
type: NounType.Thing, type: NounType.Thing,
vector: [1, 2, 3] // Wrong dimension → error vector: [1, 2, 3] // Wrong dimension → error
}) })
} catch (error) { } catch (error) {
// Transaction automatically rolled back // Transaction automatically rolled back
// No partial data in storage or indexes // No partial data in storage or indexes
console.error('Add failed:', error.message) console.error('Add failed:', error.message)
} }
``` ```
@ -333,11 +333,11 @@ try {
const stats = brain.transactionManager?.getStats() const stats = brain.transactionManager?.getStats()
console.log(stats) console.log(stats)
// { // {
// totalTransactions: 1234, // totalTransactions: 1234,
// successfulTransactions: 1200, // successfulTransactions: 1200,
// failedTransactions: 34, // failedTransactions: 34,
// rollbacks: 34, // rollbacks: 34,
// averageOperationsPerTransaction: 4.2 // averageOperationsPerTransaction: 4.2
// } // }
``` ```
@ -359,7 +359,7 @@ await brain.update({ id, data })
await brain.delete(id) await brain.delete(id)
// ❌ Avoid: Direct storage access bypasses transactions // ❌ Avoid: Direct storage access bypasses transactions
await brain.storage.saveNoun(noun) // No transaction protection await brain.storage.saveNoun(noun) // No transaction protection
``` ```
### 2. Handle Errors Gracefully ### 2. Handle Errors Gracefully
@ -367,11 +367,11 @@ await brain.storage.saveNoun(noun) // No transaction protection
```typescript ```typescript
// ✅ Recommended: Catch errors, transaction rolls back automatically // ✅ Recommended: Catch errors, transaction rolls back automatically
try { try {
const id = await brain.add({ data, type }) const id = await brain.add({ data, type })
return id return id
} catch (error) { } catch (error) {
console.error('Add failed, rolled back:', error) console.error('Add failed, rolled back:', error)
// Decide how to handle (retry, log, alert user) // Decide how to handle (retry, log, alert user)
} }
``` ```
@ -380,7 +380,7 @@ try {
```typescript ```typescript
// ✅ Recommended: Validate early to avoid unnecessary rollbacks // ✅ Recommended: Validate early to avoid unnecessary rollbacks
if (!isValidVector(vector, brain.dimension)) { if (!isValidVector(vector, brain.dimension)) {
throw new Error(`Vector must have ${brain.dimension} dimensions`) throw new Error(`Vector must have ${brain.dimension} dimensions`)
} }
await brain.add({ data, type, vector }) await brain.add({ data, type, vector })
@ -391,14 +391,14 @@ await brain.add({ data, type, vector })
```typescript ```typescript
// ✅ Recommended: Monitor in production // ✅ Recommended: Monitor in production
setInterval(() => { setInterval(() => {
const stats = brain.transactionManager?.getStats() const stats = brain.transactionManager?.getStats()
if (stats) { if (stats) {
const failureRate = stats.failedTransactions / stats.totalTransactions const failureRate = stats.failedTransactions / stats.totalTransactions
if (failureRate > 0.05) { // > 5% failure rate if (failureRate > 0.05) { // > 5% failure rate
console.warn('High transaction failure rate:', failureRate) console.warn('High transaction failure rate:', failureRate)
} }
} }
}, 60000) // Check every minute }, 60000) // Check every minute
``` ```
### 5. Understand Atomicity Guarantees ### 5. Understand Atomicity Guarantees
@ -425,25 +425,25 @@ import { describe, it, expect } from 'vitest'
import { Brainy } from '@soulcraft/brainy' import { Brainy } from '@soulcraft/brainy'
describe('Transaction Tests', () => { describe('Transaction Tests', () => {
it('should rollback on failure', async () => { it('should rollback on failure', async () => {
const brain = new Brainy() const brain = new Brainy()
await brain.init() await brain.init()
const id1 = await brain.add({ data: { name: 'Entity 1' }, type: NounType.Thing }) const id1 = await brain.add({ data: { name: 'Entity 1' }, type: NounType.Thing })
try { try {
await brain.add({ await brain.add({
data: null as any, // Invalid - will fail data: null as any, // Invalid - will fail
type: NounType.Thing type: NounType.Thing
}) })
} catch (e) { } catch (e) {
// Expected failure // Expected failure
} }
// First entity should still exist (rollback didn't affect it) // First entity should still exist (rollback didn't affect it)
const entity1 = await brain.get(id1) const entity1 = await brain.get(id1)
expect(entity1).toBeTruthy() expect(entity1).toBeTruthy()
}) })
}) })
``` ```
@ -510,21 +510,21 @@ const stats = brain.transactionManager?.getStats()
``` ```
1. BEGIN 1. BEGIN
2. ADD OPERATIONS 2. ADD OPERATIONS
- SaveNounMetadataOperation - SaveNounMetadataOperation
- SaveNounOperation - SaveNounOperation
- UpdateGraphIndexOperation - UpdateGraphIndexOperation
3. EXECUTE (sequential) 3. EXECUTE (sequential)
- Execute operation 1 → Success - Execute operation 1 → Success
- Execute operation 2 → Success - Execute operation 2 → Success
- Execute operation 3 → FAILURE - Execute operation 3 → FAILURE
4. ROLLBACK (reverse order) 4. ROLLBACK (reverse order)
- Undo operation 2 - Undo operation 2
- Undo operation 1 - Undo operation 1
5. THROW ERROR 5. THROW ERROR
``` ```
@ -544,11 +544,11 @@ Transactions use the `StorageAdapter` interface:
```typescript ```typescript
interface StorageAdapter { interface StorageAdapter {
saveNounMetadata(id: string, metadata: NounMetadata): Promise<void> saveNounMetadata(id: string, metadata: NounMetadata): Promise<void>
saveNoun(noun: Noun): Promise<void> saveNoun(noun: Noun): Promise<void>
deleteNounMetadata(id: string): Promise<void> deleteNounMetadata(id: string): Promise<void>
deleteNoun(id: string): Promise<void> deleteNoun(id: string): Promise<void>
// ... other methods // ... other methods
} }
``` ```
@ -563,11 +563,11 @@ interface StorageAdapter {
## Version History ## Version History
- **v5.8.0**: Initial transaction system release - Initial transaction system release
- Atomic operations with rollback - Atomic operations with rollback
- Compatible with COW, sharding, type-aware, distributed - Compatible with COW, sharding, type-aware, distributed
- 36/36 unit tests passing - 36/36 unit tests passing
- 35 integration test scenarios - 35 integration test scenarios
## Summary ## Summary

View file

@ -22,13 +22,13 @@ import { Brainy } from '@soulcraft/brainy'
// ✅ CORRECT: Use filesystem storage for production // ✅ CORRECT: Use filesystem storage for production
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'filesystem', type: 'filesystem',
path: './brainy-data' // Your data directory path: './brainy-data' // Your data directory
} }
}) })
await brain.init() // VFS auto-initialized! await brain.init() // VFS auto-initialized!
console.log('🎉 VFS ready!') console.log('🎉 VFS ready!')
``` ```
@ -47,40 +47,40 @@ const badItems = allNodes.filter(node => node.path.startsWith(dirPath))
```typescript ```typescript
// ✅ Method 1: Get direct children (recommended for UI) // ✅ Method 1: Get direct children (recommended for UI)
async function loadDirectoryContents(brain: Brainy, path: string) { async function loadDirectoryContents(brain: Brainy, path: string) {
try { try {
const children = await brain.vfs.getDirectChildren(path) const children = await brain.vfs.getDirectChildren(path)
// Sort directories first, then files // Sort directories first, then files
return children.sort((a, b) => { return children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1 if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1 if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name) return a.metadata.name.localeCompare(b.metadata.name)
}) })
} catch (error) { } catch (error) {
console.error(`Failed to load ${path}:`, error.message) console.error(`Failed to load ${path}:`, error.message)
return [] return []
} }
} }
// ✅ Method 2: Get complete tree structure (for full trees) // ✅ Method 2: Get complete tree structure (for full trees)
async function loadFullTree(brain: Brainy, path: string) { async function loadFullTree(brain: Brainy, path: string) {
const tree = await brain.vfs.getTreeStructure(path, { const tree = await brain.vfs.getTreeStructure(path, {
maxDepth: 3, // Prevent deep recursion maxDepth: 3, // Prevent deep recursion
includeHidden: false, // Skip hidden files includeHidden: false, // Skip hidden files
sort: 'name' sort: 'name'
}) })
return tree return tree
} }
// ✅ Method 3: Get detailed path info // ✅ Method 3: Get detailed path info
async function inspectPath(brain: Brainy, path: string) { async function inspectPath(brain: Brainy, path: string) {
const info = await brain.vfs.inspect(path) const info = await brain.vfs.inspect(path)
return { return {
isDirectory: info.node.metadata.vfsType === 'directory', isDirectory: info.node.metadata.vfsType === 'directory',
children: info.children, children: info.children,
parent: info.parent, parent: info.parent,
stats: info.stats stats: info.stats
} }
} }
``` ```
@ -89,19 +89,19 @@ async function inspectPath(brain: Brainy, path: string) {
```typescript ```typescript
// ✅ Find files by content, not just filename // ✅ Find files by content, not just filename
async function searchFiles(brain: Brainy, query: string, basePath: string = '/') { async function searchFiles(brain: Brainy, query: string, basePath: string = '/') {
const results = await brain.vfs.search(query, { const results = await brain.vfs.search(query, {
path: basePath, // Limit search to specific directory path: basePath, // Limit search to specific directory
limit: 50, // Reasonable limit limit: 50, // Reasonable limit
type: 'file' // Only search files, not directories type: 'file' // Only search files, not directories
}) })
return results.map(result => ({ return results.map(result => ({
path: result.path, path: result.path,
score: result.score, score: result.score,
type: result.type, type: result.type,
size: result.size, size: result.size,
modified: result.modified modified: result.modified
})) }))
} }
// Example usage // Example usage
@ -118,149 +118,149 @@ import React, { useState, useEffect } from 'react'
import { Brainy } from '@soulcraft/brainy' import { Brainy } from '@soulcraft/brainy'
export function FileExplorer() { export function FileExplorer() {
const [brain, setBrain] = useState(null) const [brain, setBrain] = useState(null)
const [currentPath, setCurrentPath] = useState('/') const [currentPath, setCurrentPath] = useState('/')
const [items, setItems] = useState([]) const [items, setItems] = useState([])
const [loading, setLoading] = useState(true) const [loading, setLoading] = useState(true)
const [searchQuery, setSearchQuery] = useState('') const [searchQuery, setSearchQuery] = useState('')
// Initialize Brainy (VFS auto-initialized!) // Initialize Brainy (VFS auto-initialized!)
useEffect(() => { useEffect(() => {
async function initBrainy() { async function initBrainy() {
const brainInstance = new Brainy({ const brainInstance = new Brainy({
storage: { type: 'filesystem', path: './brainy-data' } storage: { type: 'filesystem', path: './brainy-data' }
}) })
await brainInstance.init() // VFS ready after this! await brainInstance.init() // VFS ready after this!
setBrain(brainInstance) setBrain(brainInstance)
setLoading(false) setLoading(false)
} }
initBrainy() initBrainy()
}, []) }, [])
// Load directory contents // Load directory contents
const loadDirectory = async (path: string) => { const loadDirectory = async (path: string) => {
if (!brain) return if (!brain) return
setLoading(true) setLoading(true)
try { try {
// ✅ CORRECT: Use getDirectChildren to prevent recursion // ✅ CORRECT: Use getDirectChildren to prevent recursion
const children = await brain.vfs.getDirectChildren(path) const children = await brain.vfs.getDirectChildren(path)
// Sort directories first // Sort directories first
const sorted = children.sort((a, b) => { const sorted = children.sort((a, b) => {
if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1 if (a.metadata.vfsType === 'directory' && b.metadata.vfsType === 'file') return -1
if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1 if (a.metadata.vfsType === 'file' && b.metadata.vfsType === 'directory') return 1
return a.metadata.name.localeCompare(b.metadata.name) return a.metadata.name.localeCompare(b.metadata.name)
}) })
setItems(sorted) setItems(sorted)
setCurrentPath(path) setCurrentPath(path)
} catch (error) { } catch (error) {
console.error('Failed to load directory:', error) console.error('Failed to load directory:', error)
setItems([]) setItems([])
} finally { } finally {
setLoading(false) setLoading(false)
} }
} }
// Search files // Search files
const handleSearch = async () => { const handleSearch = async () => {
if (!brain || !searchQuery.trim()) { if (!brain || !searchQuery.trim()) {
loadDirectory(currentPath) loadDirectory(currentPath)
return return
} }
setLoading(true) setLoading(true)
try { try {
const results = await brain.vfs.search(searchQuery, { const results = await brain.vfs.search(searchQuery, {
path: currentPath, path: currentPath,
limit: 100 limit: 100
}) })
setItems(results) setItems(results)
} catch (error) { } catch (error) {
console.error('Search failed:', error) console.error('Search failed:', error)
} finally { } finally {
setLoading(false) setLoading(false)
} }
} }
// Initial load // Initial load
useEffect(() => { useEffect(() => {
if (brain) { if (brain) {
loadDirectory('/') loadDirectory('/')
} }
}, [brain]) }, [brain])
if (loading && !brain) { if (loading && !brain) {
return <div>Initializing Brainy...</div> return <div>Initializing Brainy...</div>
} }
return ( return (
<div className="file-explorer"> <div className="file-explorer">
{/* Search bar */} {/* Search bar */}
<div className="search-bar"> <div className="search-bar">
<input <input
type="text" type="text"
value={searchQuery} value={searchQuery}
onChange={(e) => setSearchQuery(e.target.value)} onChange={(e) => setSearchQuery(e.target.value)}
placeholder="Search files by content..." placeholder="Search files by content..."
onKeyPress={(e) => e.key === 'Enter' && handleSearch()} onKeyPress={(e) => e.key === 'Enter' && handleSearch()}
/> />
<button onClick={handleSearch}>Search</button> <button onClick={handleSearch}>Search</button>
{searchQuery && ( {searchQuery && (
<button onClick={() => { setSearchQuery(''); loadDirectory(currentPath) }}> <button onClick={() => { setSearchQuery(''); loadDirectory(currentPath) }}>
Clear Clear
</button> </button>
)} )}
</div> </div>
{/* Current path */} {/* Current path */}
<div className="current-path"> <div className="current-path">
📁 {currentPath} 📁 {currentPath}
{currentPath !== '/' && ( {currentPath !== '/' && (
<button onClick={() => loadDirectory(currentPath.split('/').slice(0, -1).join('/') || '/')}> <button onClick={() => loadDirectory(currentPath.split('/').slice(0, -1).join('/') || '/')}>
⬆️ Up ⬆️ Up
</button> </button>
)} )}
</div> </div>
{/* File list */} {/* File list */}
{loading ? ( {loading ? (
<div>Loading...</div> <div>Loading...</div>
) : ( ) : (
<div className="file-list"> <div className="file-list">
{items.map((item) => ( {items.map((item) => (
<div <div
key={item.id} key={item.id}
className={`file-item ${item.metadata.vfsType}`} className={`file-item ${item.metadata.vfsType}`}
onClick={() => { onClick={() => {
if (item.metadata.vfsType === 'directory') { if (item.metadata.vfsType === 'directory') {
loadDirectory(item.metadata.path) loadDirectory(item.metadata.path)
} else { } else {
console.log('Open file:', item.metadata.path) console.log('Open file:', item.metadata.path)
// Add your file opening logic here // Add your file opening logic here
} }
}} }}
> >
<span className="icon"> <span className="icon">
{item.metadata.vfsType === 'directory' ? '📁' : '📄'} {item.metadata.vfsType === 'directory' ? '📁' : '📄'}
</span> </span>
<span className="name">{item.metadata.name}</span> <span className="name">{item.metadata.name}</span>
<span className="size"> <span className="size">
{item.metadata.size ? `${Math.round(item.metadata.size / 1024)}KB` : ''} {item.metadata.size ? `${Math.round(item.metadata.size / 1024)}KB` : ''}
</span> </span>
</div> </div>
))} ))}
{items.length === 0 && ( {items.length === 0 && (
<div className="empty"> <div className="empty">
{searchQuery ? 'No results found' : 'Empty directory'} {searchQuery ? 'No results found' : 'Empty directory'}
</div> </div>
)} )}
</div> </div>
)} )}
</div> </div>
) )
} }
``` ```
@ -288,13 +288,13 @@ Your file explorer is now working! Here's what to explore next:
### "Module not found" errors ### "Module not found" errors
```bash ```bash
# Make sure you're using the right import # Make sure you're using the right import
npm ls @soulcraft/brainy # Check version npm ls @soulcraft/brainy # Check version
npm install @soulcraft/brainy@latest # Update if needed npm install @soulcraft/brainy@latest # Update if needed
``` ```
### "VFS not initialized" errors ### "VFS not initialized" errors
```typescript ```typescript
// v5.1.0+: Just await brain.init() - VFS is auto-initialized! // Just await brain.init() - VFS is auto-initialized!
await brain.init() await brain.init()
// VFS ready - use brain.vfs directly // VFS ready - use brain.vfs directly
await brain.vfs.writeFile('/test.txt', 'data') await brain.vfs.writeFile('/test.txt', 'data')
@ -304,8 +304,8 @@ await brain.vfs.writeFile('/test.txt', 'data')
```typescript ```typescript
// Add pagination for large directories // Add pagination for large directories
const children = await brain.vfs.getDirectChildren(path, { const children = await brain.vfs.getDirectChildren(path, {
limit: 100, // Load only first 100 items limit: 100, // Load only first 100 items
offset: 0 // Start from beginning offset: 0 // Start from beginning
}) })
``` ```
@ -313,8 +313,8 @@ const children = await brain.vfs.getDirectChildren(path, {
```typescript ```typescript
// Make sure files are imported into VFS first // Make sure files are imported into VFS first
await brain.vfs.importDirectory('./my-files', { await brain.vfs.importDirectory('./my-files', {
recursive: true, recursive: true,
extractMetadata: true // Enable content understanding extractMetadata: true // Enable content understanding
}) })
``` ```

View file

@ -28,15 +28,15 @@ import { VirtualFileSystem } from '@soulcraft/brainy/vfs'
// Initialize the VFS // Initialize the VFS
const vfs = new VirtualFileSystem({ const vfs = new VirtualFileSystem({
root: '/my-brain', root: '/my-brain',
intelligent: true // Enable AI features intelligent: true // Enable AI features
}) })
await vfs.init() await vfs.init()
// Write a file - it automatically becomes intelligent // Write a file - it automatically becomes intelligent
await vfs.writeFile('/projects/my-app/index.js', await vfs.writeFile('/projects/my-app/index.js',
'console.log("Hello, World!")') 'console.log("Hello, World!")')
// Find similar files using semantic search // Find similar files using semantic search
const similar = await vfs.findSimilar('/projects/my-app/index.js') const similar = await vfs.findSimilar('/projects/my-app/index.js')
@ -48,11 +48,11 @@ const results = await vfs.search('files about authentication')
await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements') await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements')
``` ```
## ⚡ Performance (v5.11.1) ## ⚡ Performance
**75% Faster File Operations!** VFS now automatically benefits from brain.get() metadata-only optimization: **75% Faster File Operations!** VFS now automatically benefits from brain.get() metadata-only optimization:
| Operation | Before (v5.11.0) | After (v5.11.1) | Speedup | | Operation | Before | After | Speedup |
|-----------|------------------|-----------------|---------| |-----------|------------------|-----------------|---------|
| `readFile()` | 53ms | **~13ms** | **75%** | | `readFile()` | 53ms | **~13ms** | **75%** |
| `stat()` | 53ms | **~13ms** | **75%** | | `stat()` | 53ms | **~13ms** | **75%** |
@ -60,7 +60,7 @@ await vfs.addRelationship('/docs/spec.md', '/projects/my-app/', 'implements')
**Zero configuration** - automatic optimization for all VFS operations! **Zero configuration** - automatic optimization for all VFS operations!
VFS operations only need metadata (path, size, timestamps), not 384-dimensional vector embeddings. The v5.11.1 optimization automatically uses metadata-only reads, saving 95% bandwidth and 76-81% time. VFS operations only need metadata (path, size, timestamps), not 384-dimensional vector embeddings. The optimization automatically uses metadata-only reads, saving 95% bandwidth and 76-81% time.
## Core Features ## Core Features
@ -110,20 +110,20 @@ const docs = await vfs.search('technical documentation for API endpoints')
// Find similar files // Find similar files
const similar = await vfs.findSimilar('/code/auth.js', { const similar = await vfs.findSimilar('/code/auth.js', {
limit: 5, limit: 5,
threshold: 0.8 // 80% similarity threshold: 0.8 // 80% similarity
}) })
// Get related files through the knowledge graph // Get related files through the knowledge graph
const related = await vfs.getRelated('/proposal.pdf', { const related = await vfs.getRelated('/proposal.pdf', {
depth: 2 // Include relationships of relationships depth: 2 // Include relationships of relationships
}) })
// Auto-organization suggestions // Auto-organization suggestions
const suggestions = await vfs.suggestOrganization([ const suggestions = await vfs.suggestOrganization([
'/downloads/doc1.pdf', '/downloads/doc1.pdf',
'/downloads/image.jpg', '/downloads/image.jpg',
'/downloads/code.py' '/downloads/code.py'
]) ])
// Returns: suggested folders and categorization // Returns: suggested folders and categorization
``` ```
@ -144,10 +144,10 @@ const connections = await vfs.getConnections('/code/impl.js')
// Traverse the graph // Traverse the graph
const implementations = await vfs.search('', { const implementations = await vfs.search('', {
connected: { connected: {
to: '/spec.md', to: '/spec.md',
via: 'implements' via: 'implements'
} }
}) })
``` ```
@ -158,8 +158,8 @@ Store anything alongside your files:
```javascript ```javascript
// Add todos to files // Add todos to files
await vfs.setTodos('/projects/app/index.js', [ await vfs.setTodos('/projects/app/index.js', [
{ task: 'Add error handling', priority: 'high', due: '2024-01-20' }, { task: 'Add error handling', priority: 'high', due: '2024-01-20' },
{ task: 'Optimize performance', priority: 'medium' } { task: 'Optimize performance', priority: 'medium' }
]) ])
// Set custom attributes // Set custom attributes
@ -169,11 +169,11 @@ await vfs.setxattr('/video.mp4', 'tags', ['tutorial', 'react', 'hooks'])
// Query by metadata // Query by metadata
const urgent = await vfs.search('', { const urgent = await vfs.search('', {
where: { 'todos.priority': 'high' } where: { 'todos.priority': 'high' }
}) })
const parisPhotos = await vfs.search('', { const parisPhotos = await vfs.search('', {
where: { location: 'Paris, France' } where: { location: 'Paris, France' }
}) })
``` ```
@ -184,20 +184,20 @@ Access files through semantic dimensions (see [Semantic VFS](./SEMANTIC_VFS.md))
```javascript ```javascript
// Query-based path access (current functionality) // Query-based path access (current functionality)
const authFiles = await vfs.search('', { const authFiles = await vfs.search('', {
where: { concepts: { contains: 'authentication' }} where: { concepts: { contains: 'authentication' }}
}) })
// Find files by custom metadata // Find files by custom metadata
const recent = await vfs.search('', { const recent = await vfs.search('', {
where: { where: {
type: 'document', type: 'document',
modified: { greaterThan: Date.now() - 7*24*60*60*1000 } modified: { greaterThan: Date.now() - 7*24*60*60*1000 }
} }
}) })
// Find similar files // Find similar files
const similar = await vfs.findSimilar('/examples/good-code.js', { const similar = await vfs.findSimilar('/examples/good-code.js', {
threshold: 0.7 threshold: 0.7
}) })
``` ```
@ -210,24 +210,24 @@ const similar = await vfs.findSimilar('/examples/good-code.js', {
```javascript ```javascript
// Store a research paper with automatic analysis // Store a research paper with automatic analysis
await vfs.writeFile('/research/quantum-computing.pdf', pdfBuffer, { await vfs.writeFile('/research/quantum-computing.pdf', pdfBuffer, {
metadata: { metadata: {
authors: ['Dr. Alice Smith', 'Dr. Bob Jones'], authors: ['Dr. Alice Smith', 'Dr. Bob Jones'],
year: 2024, year: 2024,
topics: ['quantum', 'computing', 'algorithms'], topics: ['quantum', 'computing', 'algorithms'],
citations: 42 citations: 42
} }
}) })
// Find all papers on similar topics // Find all papers on similar topics
const related = await vfs.search('quantum algorithms', { const related = await vfs.search('quantum algorithms', {
type: 'document', type: 'document',
where: { year: { $gte: 2020 } } where: { year: { $gte: 2020 } }
}) })
// Find papers that cite this one // Find papers that cite this one
const citations = await vfs.getConnections('/research/quantum-computing.pdf', { const citations = await vfs.getConnections('/research/quantum-computing.pdf', {
type: 'cites', type: 'cites',
direction: 'incoming' direction: 'incoming'
}) })
``` ```
@ -245,12 +245,12 @@ await vfs.writeFile('/src/utils/auth.js', authCode)
// Find all files that import this module // Find all files that import this module
const importers = await vfs.search('', { const importers = await vfs.search('', {
where: { dependencies: 'utils/auth.js' } where: { dependencies: 'utils/auth.js' }
}) })
// Find test files for this code // Find test files for this code
const tests = await vfs.getRelated('/src/utils/auth.js', { const tests = await vfs.getRelated('/src/utils/auth.js', {
type: 'tests' type: 'tests'
}) })
// Find similar implementations // Find similar implementations
@ -262,12 +262,12 @@ const similar = await vfs.findSimilar('/src/utils/auth.js')
```javascript ```javascript
// Store media with rich metadata // Store media with rich metadata
await vfs.writeFile('/photos/sunset.jpg', imageBuffer, { await vfs.writeFile('/photos/sunset.jpg', imageBuffer, {
metadata: { metadata: {
camera: 'Canon R5', camera: 'Canon R5',
location: { lat: 37.7749, lng: -122.4194 }, location: { lat: 37.7749, lng: -122.4194 },
tags: ['sunset', 'golden-gate', 'landscape'], tags: ['sunset', 'golden-gate', 'landscape'],
album: 'San Francisco 2024' album: 'San Francisco 2024'
} }
}) })
// Find similar images // Find similar images
@ -275,18 +275,18 @@ const similar = await vfs.findSimilar('/photos/sunset.jpg')
// Find photos by location // Find photos by location
const nearby = await vfs.search('', { const nearby = await vfs.search('', {
type: 'image', type: 'image',
where: { where: {
'location.lat': { $between: [37.7, 37.8] }, 'location.lat': { $between: [37.7, 37.8] },
'location.lng': { $between: [-122.5, -122.3] } 'location.lng': { $between: [-122.5, -122.3] }
} }
}) })
// Smart albums // Smart albums
await vfs.createVirtualDirectory('/albums/best-sunsets', { await vfs.createVirtualDirectory('/albums/best-sunsets', {
query: 'sunset', query: 'sunset',
type: 'image', type: 'image',
where: { rating: { $gte: 4 } } where: { rating: { $gte: 4 } }
}) })
``` ```
@ -308,25 +308,25 @@ await vfs.setxattr(projectPath, 'status', 'in-progress')
// Add todos to specific files // Add todos to specific files
await vfs.setTodos(`${projectPath}/src/index.js`, [ await vfs.setTodos(`${projectPath}/src/index.js`, [
{ task: 'Implement user authentication', assignee: 'Alice' }, { task: 'Implement user authentication', assignee: 'Alice' },
{ task: 'Add error handling', assignee: 'Bob' } { task: 'Add error handling', assignee: 'Bob' }
]) ])
// Find all files with pending todos // Find all files with pending todos
const pending = await vfs.search('', { const pending = await vfs.search('', {
where: { where: {
path: { $startsWith: projectPath }, path: { $startsWith: projectPath },
'todos.status': 'pending' 'todos.status': 'pending'
} }
}) })
// Find projects nearing deadline // Find projects nearing deadline
const urgent = await vfs.search('', { const urgent = await vfs.search('', {
where: { where: {
type: 'directory', type: 'directory',
'deadline': { $lte: '2024-02-01' }, 'deadline': { $lte: '2024-02-01' },
'status': 'in-progress' 'status': 'in-progress'
} }
}) })
``` ```
@ -417,9 +417,9 @@ The VFS is built with production scalability in mind:
#### **Storage Strategy** #### **Storage Strategy**
```typescript ```typescript
// Adaptive storage based on file size // Adaptive storage based on file size
< 100KB: Inline storage (entity.data) < 100KB: Inline storage (entity.data)
< 10MB: External reference (S3/R2 key) < 10MB: External reference (S3/R2 key)
> 10MB: Chunked storage (parallel chunks) > 10MB: Chunked storage (parallel chunks)
``` ```
#### **Performance Metrics** #### **Performance Metrics**
@ -434,24 +434,24 @@ The VFS is built with production scalability in mind:
#### **How It Handles Scale** #### **How It Handles Scale**
1. **Hierarchical Caching** 1. **Hierarchical Caching**
- 100K+ path cache entries - 100K+ path cache entries
- Parent-child relationship caching - Parent-child relationship caching
- Hot path detection and optimization - Hot path detection and optimization
2. **Distributed Architecture** 2. **Distributed Architecture**
- Sharding by path prefix - Sharding by path prefix
- Read replicas for hot directories - Read replicas for hot directories
- CDN integration for static files - CDN integration for static files
3. **Intelligent Indexing** 3. **Intelligent Indexing**
- Compound indexes on (parent, name) - Compound indexes on (parent, name)
- Vector indexes for semantic search - Vector indexes for semantic search
- Graph indexes for relationships - Graph indexes for relationships
4. **Streaming Everything** 4. **Streaming Everything**
- Large files never fully in memory - Large files never fully in memory
- Progressive loading - Progressive loading
- Chunked transfers - Chunked transfers
### Real Production Scenarios ### Real Production Scenarios
@ -462,29 +462,29 @@ Store build artifacts with automatic relationships:
```javascript ```javascript
// Store build output with metadata // Store build output with metadata
await vfs.writeFile('/builds/v1.2.3/app.js', buildOutput, { await vfs.writeFile('/builds/v1.2.3/app.js', buildOutput, {
metadata: { metadata: {
commit: 'abc123', commit: 'abc123',
branch: 'main', branch: 'main',
timestamp: Date.now(), timestamp: Date.now(),
tests: 'passing', tests: 'passing',
coverage: 0.92 coverage: 0.92
} }
}) })
// Find all builds for a commit // Find all builds for a commit
const builds = await vfs.search('', { const builds = await vfs.search('', {
where: { commit: 'abc123' } where: { commit: 'abc123' }
}) })
// Get latest passing build // Get latest passing build
const latest = await vfs.search('', { const latest = await vfs.search('', {
where: { where: {
branch: 'main', branch: 'main',
tests: 'passing' tests: 'passing'
}, },
sort: 'modified', sort: 'modified',
order: 'desc', order: 'desc',
limit: 1 limit: 1
}) })
``` ```
@ -495,8 +495,8 @@ Isolate customer data with semantic understanding:
```javascript ```javascript
// Each tenant gets their own root // Each tenant gets their own root
const tenantVfs = new VirtualFileSystem({ const tenantVfs = new VirtualFileSystem({
root: `/tenants/${tenantId}`, root: `/tenants/${tenantId}`,
service: tenantId // Isolate at Brainy level too service: tenantId // Isolate at Brainy level too
}) })
// Tenant uploads document // Tenant uploads document
@ -505,11 +505,11 @@ await tenantVfs.writeFile('/documents/contract.pdf', pdfBuffer)
// Cross-tenant analytics (admin only) // Cross-tenant analytics (admin only)
const adminVfs = new VirtualFileSystem({ root: '/tenants' }) const adminVfs = new VirtualFileSystem({ root: '/tenants' })
const stats = await adminVfs.search('contract', { const stats = await adminVfs.search('contract', {
recursive: true, recursive: true,
aggregations: { aggregations: {
byTenant: { field: 'service' }, byTenant: { field: 'service' },
byType: { field: 'mimeType' } byType: { field: 'mimeType' }
} }
}) })
``` ```
@ -520,38 +520,38 @@ Connect datasets, models, and results:
```javascript ```javascript
// Store training data // Store training data
await vfs.writeFile('/datasets/train.csv', csvData, { await vfs.writeFile('/datasets/train.csv', csvData, {
metadata: { metadata: {
samples: 100000, samples: 100000,
features: 50, features: 50,
labels: 10 labels: 10
} }
}) })
// Store trained model // Store trained model
await vfs.writeFile('/models/v1/model.pkl', modelBuffer, { await vfs.writeFile('/models/v1/model.pkl', modelBuffer, {
metadata: { metadata: {
algorithm: 'random-forest', algorithm: 'random-forest',
accuracy: 0.95, accuracy: 0.95,
trainedOn: '/datasets/train.csv', trainedOn: '/datasets/train.csv',
hyperparameters: { trees: 100, depth: 10 } hyperparameters: { trees: 100, depth: 10 }
} }
}) })
// Connect model to its training data // Connect model to its training data
await vfs.addRelationship( await vfs.addRelationship(
'/models/v1/model.pkl', '/models/v1/model.pkl',
'/datasets/train.csv', '/datasets/train.csv',
'trained-on' 'trained-on'
) )
// Find best model for a dataset // Find best model for a dataset
const models = await vfs.search('', { const models = await vfs.search('', {
connected: { connected: {
to: '/datasets/train.csv', to: '/datasets/train.csv',
via: 'trained-on' via: 'trained-on'
}, },
sort: 'accuracy', sort: 'accuracy',
order: 'desc' order: 'desc'
}) })
``` ```
@ -562,30 +562,30 @@ Intelligent content organization:
```javascript ```javascript
// Auto-organize uploads // Auto-organize uploads
vfs.on('file:added', async (path) => { vfs.on('file:added', async (path) => {
if (path.startsWith('/uploads/')) { if (path.startsWith('/uploads/')) {
const file = await vfs.getEntity(path) const file = await vfs.getEntity(path)
// Auto-categorize by AI // Auto-categorize by AI
const category = await detectCategory(file) const category = await detectCategory(file)
const newPath = `/content/${category}/${file.metadata.name}` const newPath = `/content/${category}/${file.metadata.name}`
await vfs.move(path, newPath) await vfs.move(path, newPath)
// Auto-tag // Auto-tag
const tags = await extractTags(file) const tags = await extractTags(file)
await vfs.setxattr(newPath, 'tags', tags) await vfs.setxattr(newPath, 'tags', tags)
// Find related content // Find related content
const related = await vfs.findSimilar(newPath, { const related = await vfs.findSimilar(newPath, {
limit: 5, limit: 5,
threshold: 0.8 threshold: 0.8
}) })
// Create relationships // Create relationships
for (const rel of related) { for (const rel of related) {
await vfs.addRelationship(newPath, rel.path, 'related-to') await vfs.addRelationship(newPath, rel.path, 'related-to')
} }
} }
}) })
``` ```
@ -596,39 +596,39 @@ Collaborative file management:
```javascript ```javascript
// Track file ownership and access // Track file ownership and access
await vfs.writeFile('/projects/alpha/spec.md', content, { await vfs.writeFile('/projects/alpha/spec.md', content, {
metadata: { metadata: {
owner: userId, owner: userId,
team: 'engineering', team: 'engineering',
permissions: { permissions: {
[userId]: 'rw', [userId]: 'rw',
'team:engineering': 'r', 'team:engineering': 'r',
'others': '-' 'others': '-'
} }
} }
}) })
// Add collaborative features // Add collaborative features
await vfs.addTodo('/projects/alpha/spec.md', { await vfs.addTodo('/projects/alpha/spec.md', {
task: 'Review security section', task: 'Review security section',
assignee: 'alice@company.com', assignee: 'alice@company.com',
due: '2024-02-01', due: '2024-02-01',
priority: 'high' priority: 'high'
}) })
// Track who's working on what // Track who's working on what
await vfs.setxattr('/projects/alpha/spec.md', 'locks', { await vfs.setxattr('/projects/alpha/spec.md', 'locks', {
section3: { section3: {
user: 'bob@company.com', user: 'bob@company.com',
since: Date.now() since: Date.now()
} }
}) })
// Find all files assigned to a user // Find all files assigned to a user
const assigned = await vfs.search('', { const assigned = await vfs.search('', {
where: { where: {
'todos.assignee': 'alice@company.com', 'todos.assignee': 'alice@company.com',
'todos.status': 'pending' 'todos.status': 'pending'
} }
}) })
``` ```
@ -643,15 +643,15 @@ const assigned = await vfs.search('', {
#### **Standalone Mode** #### **Standalone Mode**
```javascript ```javascript
const vfs = new VirtualFileSystem() const vfs = new VirtualFileSystem()
await vfs.init() // Uses in-memory storage await vfs.init() // Uses in-memory storage
``` ```
#### **Local Persistence** #### **Local Persistence**
```javascript ```javascript
const vfs = new VirtualFileSystem() const vfs = new VirtualFileSystem()
await vfs.init({ await vfs.init({
storage: 'filesystem', storage: 'filesystem',
dataDir: '/var/lib/brainy-vfs' dataDir: '/var/lib/brainy-vfs'
}) })
``` ```
@ -659,9 +659,9 @@ await vfs.init({
```javascript ```javascript
const vfs = new VirtualFileSystem() const vfs = new VirtualFileSystem()
await vfs.init({ await vfs.init({
storage: 's3', storage: 's3',
bucket: 'my-vfs-data', bucket: 'my-vfs-data',
region: 'us-west-2' region: 'us-west-2'
}) })
``` ```
@ -669,14 +669,14 @@ await vfs.init({
```javascript ```javascript
const vfs = new VirtualFileSystem() const vfs = new VirtualFileSystem()
await vfs.init({ await vfs.init({
distributed: true, distributed: true,
nodes: [ nodes: [
'vfs1.internal:8080', 'vfs1.internal:8080',
'vfs2.internal:8080', 'vfs2.internal:8080',
'vfs3.internal:8080' 'vfs3.internal:8080'
], ],
replication: 3, replication: 3,
consistency: 'eventual' consistency: 'eventual'
}) })
``` ```

View file

@ -12,8 +12,8 @@
**Root Cause:** **Root Cause:**
The root directory entity exists but doesn't have the proper metadata structure. The root directory entity exists but doesn't have the proper metadata structure.
**Solution (v3.15.0+):** **Solution:**
This issue has been fixed in v3.15.0. The VFS now: This issue has been fixed. The VFS now:
1. Ensures root directory has `vfsType: 'directory'` metadata 1. Ensures root directory has `vfsType: 'directory'` metadata
2. Adds compatibility layer for entities with malformed metadata 2. Adds compatibility layer for entities with malformed metadata
3. Automatically repairs metadata on entity retrieval 3. Automatically repairs metadata on entity retrieval
@ -21,17 +21,17 @@ This issue has been fixed in v3.15.0. The VFS now:
**Manual Fix (if needed):** **Manual Fix (if needed):**
```javascript ```javascript
// Force re-initialization of root directory // Force re-initialization of root directory
await vfs.init() // Will repair root if needed await vfs.init() // Will repair root if needed
// Or manually update root entity // Or manually update root entity
const rootId = vfs.rootEntityId const rootId = vfs.rootEntityId
await brain.update({ await brain.update({
id: rootId, id: rootId,
metadata: { metadata: {
path: '/', path: '/',
vfsType: 'directory', vfsType: 'directory',
// ... other metadata // ... other metadata
} }
}) })
``` ```
@ -47,7 +47,7 @@ await brain.update({
**Root Cause:** **Root Cause:**
Contains relationships are missing between parent directories and files. Contains relationships are missing between parent directories and files.
**Solution (v3.15.0+):** **Solution:**
This issue has been fixed. The VFS now: This issue has been fixed. The VFS now:
1. Creates Contains relationships when writing new files 1. Creates Contains relationships when writing new files
2. Ensures Contains relationships exist when updating files 2. Ensures Contains relationships exist when updating files
@ -60,9 +60,9 @@ const parentId = await vfs.resolvePath('/directory')
const fileId = await vfs.resolvePath('/directory/file.txt') const fileId = await vfs.resolvePath('/directory/file.txt')
await brain.relate({ await brain.relate({
from: parentId, from: parentId,
to: fileId, to: fileId,
type: VerbType.Contains type: VerbType.Contains
}) })
``` ```
@ -80,8 +80,8 @@ The VFS `init()` method wasn't called after getting the VFS instance.
**Solution:** **Solution:**
```javascript ```javascript
// ✅ CORRECT // ✅ CORRECT
const vfs = brain.vfs() // Get instance const vfs = brain.vfs() // Get instance
await vfs.init() // Initialize (REQUIRED!) await vfs.init() // Initialize (REQUIRED!)
// ❌ WRONG // ❌ WRONG
const vfs = brain.vfs() const vfs = brain.vfs()
@ -104,15 +104,15 @@ Using in-memory storage instead of persistent storage.
```javascript ```javascript
// ✅ Use persistent storage // ✅ Use persistent storage
const brain = new Brainy({ const brain = new Brainy({
storage: { storage: {
type: 'filesystem', // Persistent type: 'filesystem', // Persistent
path: './brainy-data' path: './brainy-data'
} }
}) })
// ❌ Don't use memory for production // ❌ Don't use memory for production
const brain = new Brainy({ const brain = new Brainy({
storage: { type: 'memory' } // Data lost on restart! storage: { type: 'memory' } // Data lost on restart!
}) })
``` ```
@ -136,7 +136,7 @@ const children = await vfs.getDirectChildren('/directory')
// ❌ WRONG - Path prefix matching causes recursion // ❌ WRONG - Path prefix matching causes recursion
const allNodes = await brain.find({}) const allNodes = await brain.find({})
const children = allNodes.filter(n => const children = allNodes.filter(n =>
n.metadata.path.startsWith('/directory/')) // Directory matches itself! n.metadata.path.startsWith('/directory/')) // Directory matches itself!
``` ```
--- ---
@ -153,13 +153,13 @@ Files aren't being properly embedded or indexed.
**Solution:** **Solution:**
```javascript ```javascript
// Ensure files have content for embedding // Ensure files have content for embedding
await vfs.writeFile('/doc.txt', 'Actual content here') // Not empty! await vfs.writeFile('/doc.txt', 'Actual content here') // Not empty!
// Use semantic search correctly // Use semantic search correctly
const results = await vfs.search('machine learning', { const results = await vfs.search('machine learning', {
path: '/documents', // Search within path path: '/documents', // Search within path
limit: 10, limit: 10,
type: 'file' type: 'file'
}) })
``` ```
@ -184,8 +184,8 @@ console.log('VFS type:', entity.metadata.vfsType)
```javascript ```javascript
const parentId = await vfs.resolvePath('/directory') const parentId = await vfs.resolvePath('/directory')
const relations = await brain.getRelations({ const relations = await brain.getRelations({
from: parentId, from: parentId,
type: VerbType.Contains type: VerbType.Contains
}) })
console.log('Child count:', relations.length) console.log('Child count:', relations.length)
``` ```
@ -193,11 +193,11 @@ console.log('Child count:', relations.length)
### 4. Enable Debug Logging ### 4. Enable Debug Logging
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
storage: { type: 'filesystem' }, storage: { type: 'filesystem' },
logger: { logger: {
level: 'debug', level: 'debug',
enabled: true enabled: true
} }
}) })
``` ```
@ -209,11 +209,11 @@ const brain = new Brainy({
```javascript ```javascript
// Enable caching in VFS config // Enable caching in VFS config
const vfs = brain.vfs({ const vfs = brain.vfs({
cache: { cache: {
enabled: true, enabled: true,
ttl: 300000, // 5 minutes ttl: 300000, // 5 minutes
maxSize: 1000 maxSize: 1000
} }
}) })
``` ```
@ -221,12 +221,12 @@ const vfs = brain.vfs({
```javascript ```javascript
// Write multiple files efficiently // Write multiple files efficiently
const files = [ const files = [
{ path: '/file1.txt', content: 'content1' }, { path: '/file1.txt', content: 'content1' },
{ path: '/file2.txt', content: 'content2' } { path: '/file2.txt', content: 'content2' }
] ]
await Promise.all( await Promise.all(
files.map(f => vfs.writeFile(f.path, f.content)) files.map(f => vfs.writeFile(f.path, f.content))
) )
``` ```
@ -234,8 +234,8 @@ await Promise.all(
```javascript ```javascript
// Don't traverse too deep // Don't traverse too deep
const tree = await vfs.getTreeStructure('/', { const tree = await vfs.getTreeStructure('/', {
maxDepth: 3, // Limit recursion maxDepth: 3, // Limit recursion
includeHidden: false includeHidden: false
}) })
``` ```

View file

@ -418,7 +418,7 @@ await vfs.importDirectory()// Import directory from filesystem
## Bulk Write Operations ## Bulk Write Operations
**v6.5.0**: `bulkWrite` efficiently processes multiple VFS operations with automatic ordering to prevent race conditions. `bulkWrite` efficiently processes multiple VFS operations with automatic ordering to prevent race conditions.
```javascript ```javascript
const result = await vfs.bulkWrite([ const result = await vfs.bulkWrite([
@ -439,7 +439,7 @@ console.log(`Successful: ${result.successful}, Failed: ${result.failed.length}`)
- `delete` - Delete file - `delete` - Delete file
- `update` - Update file metadata only - `update` - Update file metadata only
**Operation ordering (v6.5.0):** **Operation ordering:**
1. `mkdir` operations run **first**, sequentially, sorted by path depth (shallowest first) 1. `mkdir` operations run **first**, sequentially, sorted by path depth (shallowest first)
2. Other operations (`write`, `delete`, `update`) run **after** in parallel batches of 10 2. Other operations (`write`, `delete`, `update`) run **after** in parallel batches of 10

View file

@ -1,4 +1,4 @@
# VFS Initialization Guide (v5.1.0+) # VFS Initialization Guide
## Quick Start ## Quick Start
@ -30,7 +30,7 @@ await vfs.init() // ❌ Separate initialization
await vfs.writeFile(...) await vfs.writeFile(...)
``` ```
### After (v5.1.0+): ### After:
```javascript ```javascript
const brain = new Brainy(...) const brain = new Brainy(...)
await brain.init() // VFS auto-initialized! await brain.init() // VFS auto-initialized!
@ -53,7 +53,7 @@ await vfs.init()
await vfs.writeFile('/test.txt', 'data') await vfs.writeFile('/test.txt', 'data')
``` ```
### New Pattern (v5.1.0+): ### New Pattern:
```javascript ```javascript
// Just remove the () and init() call // Just remove the () and init() call
await brain.vfs.writeFile('/test.txt', 'data') await brain.vfs.writeFile('/test.txt', 'data')
@ -142,7 +142,7 @@ await brain.init() // Required!
await brain.vfs.writeFile(...) // Now this works await brain.vfs.writeFile(...) // Now this works
``` ```
## Fork Support (v5.0.0+) ## Fork Support
VFS works seamlessly with Brainy's Copy-on-Write (COW) fork feature: VFS works seamlessly with Brainy's Copy-on-Write (COW) fork feature:

View file

@ -60,7 +60,7 @@ export class ConfigAPI {
// Store in cache // Store in cache
this.configCache.set(key, entry) this.configCache.set(key, entry)
// v4.0.0: Persist to storage as NounMetadata // Persist to storage as NounMetadata
const configId = this.CONFIG_NOUN_PREFIX + key const configId = this.CONFIG_NOUN_PREFIX + key
const nounMetadata = { const nounMetadata = {
noun: 'config' as any, noun: 'config' as any,
@ -94,7 +94,7 @@ export class ConfigAPI {
return defaultValue return defaultValue
} }
// v4.0.0: Extract ConfigEntry from NounMetadata // Extract ConfigEntry from NounMetadata
const createdAtVal = typeof metadata.createdAt === 'object' && metadata.createdAt !== null && 'seconds' in metadata.createdAt const createdAtVal = typeof metadata.createdAt === 'object' && metadata.createdAt !== null && 'seconds' in metadata.createdAt
? metadata.createdAt.seconds * 1000 + Math.floor((metadata.createdAt.nanoseconds || 0) / 1000000) ? metadata.createdAt.seconds * 1000 + Math.floor((metadata.createdAt.nanoseconds || 0) / 1000000)
: ((metadata.createdAt as unknown as number) || Date.now()) : ((metadata.createdAt as unknown as number) || Date.now())
@ -140,7 +140,7 @@ export class ConfigAPI {
// Remove from cache // Remove from cache
this.configCache.delete(key) this.configCache.delete(key)
// v4.0.0: Remove from storage // Remove from storage
const configId = this.CONFIG_NOUN_PREFIX + key const configId = this.CONFIG_NOUN_PREFIX + key
await this.storage.deleteNounMetadata(configId) await this.storage.deleteNounMetadata(configId)
} }
@ -149,7 +149,7 @@ export class ConfigAPI {
* List all configuration keys * List all configuration keys
*/ */
async list(): Promise<string[]> { async list(): Promise<string[]> {
// v4.0.0: Get all nouns and filter for config entries // Get all nouns and filter for config entries
const result = await this.storage.getNouns({ pagination: { limit: 10000 } }) const result = await this.storage.getNouns({ pagination: { limit: 10000 } })
const configKeys: string[] = [] const configKeys: string[] = []
@ -213,7 +213,7 @@ export class ConfigAPI {
for (const [key, entry] of Object.entries(config)) { for (const [key, entry] of Object.entries(config)) {
this.configCache.set(key, entry) this.configCache.set(key, entry)
const configId = this.CONFIG_NOUN_PREFIX + key const configId = this.CONFIG_NOUN_PREFIX + key
// v4.0.0: Convert ConfigEntry to NounMetadata // Convert ConfigEntry to NounMetadata
const nounMetadata = { const nounMetadata = {
noun: 'config' as any, noun: 'config' as any,
...entry, ...entry,
@ -240,7 +240,7 @@ export class ConfigAPI {
return null return null
} }
// v4.0.0: Extract ConfigEntry from NounMetadata // Extract ConfigEntry from NounMetadata
const createdAtVal = typeof metadata.createdAt === 'object' && metadata.createdAt !== null && 'seconds' in metadata.createdAt const createdAtVal = typeof metadata.createdAt === 'object' && metadata.createdAt !== null && 'seconds' in metadata.createdAt
? metadata.createdAt.seconds * 1000 + Math.floor((metadata.createdAt.nanoseconds || 0) / 1000000) ? metadata.createdAt.seconds * 1000 + Math.floor((metadata.createdAt.nanoseconds || 0) / 1000000)
: ((metadata.createdAt as unknown as number) || Date.now()) : ((metadata.createdAt as unknown as number) || Date.now())

View file

@ -210,7 +210,7 @@ export class DataAPI {
this.validateImportItem(mapped) this.validateImportItem(mapped)
} }
// v4.0.0: Save entity - separate vector and metadata // Save entity - separate vector and metadata
const id = mapped.id || this.generateId() const id = mapped.id || this.generateId()
const noun: HNSWNoun = { const noun: HNSWNoun = {
id, id,

View file

@ -375,7 +375,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
} }
/** /**
* Get CommitLog for temporal features (v5.0.0+) * Get CommitLog for temporal features
* *
* Provides access to commit history for time-travel queries, audit trails, * Provides access to commit history for time-travel queries, audit trails,
* and branch management. Available after initialize() is called. * and branch management. Available after initialize() is called.
@ -414,7 +414,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
} }
/** /**
* Get BlobStorage for content-addressable storage (v5.0.0+) * Get BlobStorage for content-addressable storage
* *
* Provides access to the underlying blob storage system for storing * Provides access to the underlying blob storage system for storing
* and retrieving content-addressed data. Available after initialize() is called. * and retrieving content-addressed data. Available after initialize() is called.
@ -454,7 +454,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
} }
/** /**
* Get RefManager for branch/ref management (v5.0.0+) * Get RefManager for branch/ref management
* *
* Provides access to the reference manager for creating, updating, * Provides access to the reference manager for creating, updating,
* and managing Git-style branches and refs. Available after initialize() is called. * and managing Git-style branches and refs. Available after initialize() is called.
@ -496,7 +496,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
} }
/** /**
* Get current branch name (v5.0.0+) * Get current branch name
* *
* Convenience helper for getting the current branch from the Brainy instance. * Convenience helper for getting the current branch from the Brainy instance.
* Available after initialize() is called. * Available after initialize() is called.
@ -525,7 +525,7 @@ export abstract class BaseAugmentation implements BrainyAugmentation {
if (typeof brain.getCurrentBranch !== 'function') { if (typeof brain.getCurrentBranch !== 'function') {
throw new Error( throw new Error(
`${this.name}: getCurrentBranch() not available on Brainy instance. ` + `${this.name}: getCurrentBranch() not available on Brainy instance. ` +
`This method requires Brainy v5.0.0+.` `This method requires Brainy with VFS support.`
) )
} }

View file

@ -1,5 +1,5 @@
/** /**
* Format Handler Registry (v5.2.0) * Format Handler Registry
* *
* Central registry for format handlers with: * Central registry for format handlers with:
* - MIME type-based routing * - MIME type-based routing

View file

@ -35,7 +35,7 @@ export class IntelligentImportAugmentation extends BaseAugmentation {
enableCSV: true, enableCSV: true,
enableExcel: true, enableExcel: true,
enablePDF: true, enablePDF: true,
enableImage: true, // v5.2.0: Image handler enabled by default enableImage: true, // Image handler enabled by default
maxFileSize: 100 * 1024 * 1024, // 100MB default maxFileSize: 100 * 1024 * 1024, // 100MB default
enableCache: true, enableCache: true,
cacheTTL: 24 * 60 * 60 * 1000, // 24 hours cacheTTL: 24 * 60 * 60 * 1000, // 24 hours

View file

@ -40,7 +40,7 @@ export class CSVHandler extends BaseFormatHandler {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8') const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8')
const totalBytes = buffer.length const totalBytes = buffer.length
// v4.5.0: Report total bytes for progress tracking // Report total bytes for progress tracking
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0) progressHooks.onBytesProcessed(0)
} }
@ -55,7 +55,7 @@ export class CSVHandler extends BaseFormatHandler {
// Detect delimiter if not specified // Detect delimiter if not specified
const delimiter = options.csvDelimiter || this.detectDelimiter(text) const delimiter = options.csvDelimiter || this.detectDelimiter(text)
// v4.5.0: Report progress - parsing started // Report progress - parsing started
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`) progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`)
} }
@ -75,7 +75,7 @@ export class CSVHandler extends BaseFormatHandler {
cast: false // We'll do type inference ourselves cast: false // We'll do type inference ourselves
}) })
// v4.5.0: Report bytes processed (entire file parsed) // Report bytes processed (entire file parsed)
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes) progressHooks.onBytesProcessed(totalBytes)
} }
@ -83,7 +83,7 @@ export class CSVHandler extends BaseFormatHandler {
// Convert to array of objects // Convert to array of objects
const data = Array.isArray(records) ? records : [records] const data = Array.isArray(records) ? records : [records]
// v4.5.0: Report data extraction progress // Report data extraction progress
if (progressHooks?.onDataExtracted) { if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(data.length, data.length) progressHooks.onDataExtracted(data.length, data.length)
} }
@ -101,7 +101,7 @@ export class CSVHandler extends BaseFormatHandler {
converted[key] = this.convertValue(value, types[key] || 'string') converted[key] = this.convertValue(value, types[key] || 'string')
} }
// v4.5.0: Report progress every 1000 rows // Report progress every 1000 rows
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) { if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`) progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`)
} }
@ -111,7 +111,7 @@ export class CSVHandler extends BaseFormatHandler {
const processingTime = Date.now() - startTime const processingTime = Date.now() - startTime
// v4.5.0: Final progress update // Final progress update
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`) progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`)
} }

View file

@ -27,7 +27,7 @@ export class ExcelHandler extends BaseFormatHandler {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'binary') const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'binary')
const totalBytes = buffer.length const totalBytes = buffer.length
// v4.5.0: Report start // Report start
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0) progressHooks.onBytesProcessed(0)
} }
@ -47,7 +47,7 @@ export class ExcelHandler extends BaseFormatHandler {
// Determine which sheets to process // Determine which sheets to process
const sheetsToProcess = this.getSheetsToProcess(workbook, options) const sheetsToProcess = this.getSheetsToProcess(workbook, options)
// v4.5.0: Report workbook loaded // Report workbook loaded
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing ${sheetsToProcess.length} sheets...`) progressHooks.onCurrentItem(`Processing ${sheetsToProcess.length} sheets...`)
} }
@ -59,7 +59,7 @@ export class ExcelHandler extends BaseFormatHandler {
for (let sheetIndex = 0; sheetIndex < sheetsToProcess.length; sheetIndex++) { for (let sheetIndex = 0; sheetIndex < sheetsToProcess.length; sheetIndex++) {
const sheetName = sheetsToProcess[sheetIndex] const sheetName = sheetsToProcess[sheetIndex]
// v4.5.0: Report current sheet // Report current sheet
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem( progressHooks.onCurrentItem(
`Reading sheet: ${sheetName} (${sheetIndex + 1}/${sheetsToProcess.length})` `Reading sheet: ${sheetName} (${sheetIndex + 1}/${sheetsToProcess.length})`
@ -117,19 +117,19 @@ export class ExcelHandler extends BaseFormatHandler {
headers headers
} }
// v4.5.0: Estimate bytes processed (sheets are sequential) // Estimate bytes processed (sheets are sequential)
const bytesProcessed = Math.floor(((sheetIndex + 1) / sheetsToProcess.length) * totalBytes) const bytesProcessed = Math.floor(((sheetIndex + 1) / sheetsToProcess.length) * totalBytes)
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed) progressHooks.onBytesProcessed(bytesProcessed)
} }
// v4.5.0: Report extraction progress // Report extraction progress
if (progressHooks?.onDataExtracted) { if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until complete progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until complete
} }
} }
// v4.5.0: Report data extraction complete // Report data extraction complete
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Extracted ${allData.length} rows, inferring types...`) progressHooks.onCurrentItem(`Extracted ${allData.length} rows, inferring types...`)
} }
@ -152,7 +152,7 @@ export class ExcelHandler extends BaseFormatHandler {
} }
} }
// v4.5.0: Report progress every 1000 rows (avoid spam) // Report progress every 1000 rows (avoid spam)
if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) { if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) {
progressHooks.onCurrentItem(`Converting types: ${index}/${allData.length} rows...`) progressHooks.onCurrentItem(`Converting types: ${index}/${allData.length} rows...`)
} }
@ -160,14 +160,14 @@ export class ExcelHandler extends BaseFormatHandler {
return converted return converted
}) })
// v4.5.0: Final progress - all bytes processed // Final progress - all bytes processed
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes) progressHooks.onBytesProcessed(totalBytes)
} }
const processingTime = Date.now() - startTime const processingTime = Date.now() - startTime
// v4.5.0: Report completion // Report completion
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem( progressHooks.onCurrentItem(
`Excel complete: ${sheetsToProcess.length} sheets, ${convertedData.length} rows` `Excel complete: ${sheetsToProcess.length} sheets, ${convertedData.length} rows`

View file

@ -1,5 +1,5 @@
/** /**
* Image Import Handler (v5.8.0 - Pure JavaScript) * Image Import Handler
* *
* Handles image files with: * Handles image files with:
* - EXIF metadata extraction (camera, GPS, timestamps) via exifr * - EXIF metadata extraction (camera, GPS, timestamps) via exifr
@ -7,7 +7,7 @@
* - Support for JPEG, PNG, WebP, GIF, TIFF, BMP, SVG * - Support for JPEG, PNG, WebP, GIF, TIFF, BMP, SVG
* *
* NO NATIVE DEPENDENCIES - Pure JavaScript implementation * NO NATIVE DEPENDENCIES - Pure JavaScript implementation
* Replaces Sharp (v5.7.x) with lightweight pure-JS alternatives * Replaces Sharp with lightweight pure-JS alternatives
*/ */
import { BaseFormatHandler } from './base.js' import { BaseFormatHandler } from './base.js'
@ -96,7 +96,7 @@ export interface ImageHandlerOptions extends FormatHandlerOptions {
* Enables developers to import images into the knowledge graph with * Enables developers to import images into the knowledge graph with
* full metadata extraction. * full metadata extraction.
* *
* v5.8.0: Pure JavaScript implementation (no native dependencies) * Pure JavaScript implementation (no native dependencies)
*/ */
export class ImageHandler extends BaseFormatHandler { export class ImageHandler extends BaseFormatHandler {
readonly format = 'image' readonly format = 'image'

View file

@ -52,7 +52,7 @@ export class PDFHandler extends BaseFormatHandler {
const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'binary') const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'binary')
const totalBytes = buffer.length const totalBytes = buffer.length
// v4.5.0: Report start // Report start
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(0) progressHooks.onBytesProcessed(0)
} }
@ -74,7 +74,7 @@ export class PDFHandler extends BaseFormatHandler {
const metadata = await pdfDoc.getMetadata() const metadata = await pdfDoc.getMetadata()
const numPages = pdfDoc.numPages const numPages = pdfDoc.numPages
// v4.5.0: Report document loaded // Report document loaded
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing ${numPages} pages...`) progressHooks.onCurrentItem(`Processing ${numPages} pages...`)
} }
@ -85,7 +85,7 @@ export class PDFHandler extends BaseFormatHandler {
let detectedTables = 0 let detectedTables = 0
for (let pageNum = 1; pageNum <= numPages; pageNum++) { for (let pageNum = 1; pageNum <= numPages; pageNum++) {
// v4.5.0: Report current page // Report current page
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem(`Processing page ${pageNum} of ${numPages}`) progressHooks.onCurrentItem(`Processing page ${pageNum} of ${numPages}`)
} }
@ -131,19 +131,19 @@ export class PDFHandler extends BaseFormatHandler {
} }
} }
// v4.5.0: Estimate bytes processed (pages are sequential) // Estimate bytes processed (pages are sequential)
const bytesProcessed = Math.floor((pageNum / numPages) * totalBytes) const bytesProcessed = Math.floor((pageNum / numPages) * totalBytes)
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(bytesProcessed) progressHooks.onBytesProcessed(bytesProcessed)
} }
// v4.5.0: Report extraction progress // Report extraction progress
if (progressHooks?.onDataExtracted) { if (progressHooks?.onDataExtracted) {
progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until complete progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until complete
} }
} }
// v4.5.0: Final progress - all bytes processed // Final progress - all bytes processed
if (progressHooks?.onBytesProcessed) { if (progressHooks?.onBytesProcessed) {
progressHooks.onBytesProcessed(totalBytes) progressHooks.onBytesProcessed(totalBytes)
} }
@ -153,7 +153,7 @@ export class PDFHandler extends BaseFormatHandler {
const processingTime = Date.now() - startTime const processingTime = Date.now() - startTime
// v4.5.0: Report completion // Report completion
if (progressHooks?.onCurrentItem) { if (progressHooks?.onCurrentItem) {
progressHooks.onCurrentItem( progressHooks.onCurrentItem(
`PDF complete: ${numPages} pages, ${allData.length} items extracted` `PDF complete: ${numPages} pages, ${allData.length} items extracted`

View file

@ -18,14 +18,14 @@ export { ExcelHandler } from './handlers/excelHandler.js'
export { PDFHandler } from './handlers/pdfHandler.js' export { PDFHandler } from './handlers/pdfHandler.js'
export { ImageHandler } from './handlers/imageHandler.js' export { ImageHandler } from './handlers/imageHandler.js'
// Format Handler Registry (v5.2.0) // Format Handler Registry
export { export {
FormatHandlerRegistry, FormatHandlerRegistry,
globalHandlerRegistry globalHandlerRegistry
} from './FormatHandlerRegistry.js' } from './FormatHandlerRegistry.js'
export type { HandlerRegistration } from './FormatHandlerRegistry.js' export type { HandlerRegistration } from './FormatHandlerRegistry.js'
// Image Handler Types (v5.2.0) // Image Handler Types
export type { export type {
ImageMetadata, ImageMetadata,
EXIFData, EXIFData,

View file

@ -88,13 +88,13 @@ export interface FormatHandlerOptions {
streaming?: boolean streaming?: boolean
/** /**
* Progress hooks (v4.5.0) * Progress hooks
* Handlers call these to report progress during processing * Handlers call these to report progress during processing
*/ */
progressHooks?: FormatHandlerProgressHooks progressHooks?: FormatHandlerProgressHooks
/** /**
* Total file size in bytes (v4.5.0) * Total file size in bytes
* Used for progress percentage calculation * Used for progress percentage calculation
*/ */
totalBytes?: number totalBytes?: number
@ -168,7 +168,7 @@ export interface IntelligentImportConfig {
/** Enable PDF handler */ /** Enable PDF handler */
enablePDF: boolean enablePDF: boolean
/** Enable Image handler (v5.2.0) */ /** Enable Image handler */
enableImage: boolean enableImage: boolean
/** Default options for CSV */ /** Default options for CSV */
@ -180,7 +180,7 @@ export interface IntelligentImportConfig {
/** Default options for PDF */ /** Default options for PDF */
pdfDefaults?: Partial<FormatHandlerOptions> pdfDefaults?: Partial<FormatHandlerOptions>
/** Default options for Image (v5.2.0) */ /** Default options for Image */
imageDefaults?: Partial<FormatHandlerOptions> imageDefaults?: Partial<FormatHandlerOptions>
/** Maximum file size to process (bytes) */ /** Maximum file size to process (bytes) */

View file

@ -168,7 +168,7 @@ export interface TypeMatchResult {
/** /**
* BrainyTypes - Intelligent type detection for nouns and verbs * BrainyTypes - Intelligent type detection for nouns and verbs
* PRODUCTION OPTIMIZATION (v3.33.0): Uses pre-computed type embeddings * PRODUCTION OPTIMIZATION: Uses pre-computed type embeddings
* Type embeddings are loaded instantly; only input objects are embedded at runtime * Type embeddings are loaded instantly; only input objects are embedded at runtime
*/ */
export class BrainyTypes { export class BrainyTypes {

View file

@ -172,7 +172,7 @@ export const coreCommands = {
metadata metadata
} }
// v4.3.x: Add confidence and weight if provided // Add confidence and weight if provided
if (options.confidence) { if (options.confidence) {
addParams.confidence = parseFloat(options.confidence) addParams.confidence = parseFloat(options.confidence)
} }
@ -347,7 +347,7 @@ export const coreCommands = {
searchParams.includeRelations = true searchParams.includeRelations = true
} }
// v4.7.0: VFS is now part of the knowledge graph (included by default) // VFS is now part of the knowledge graph (included by default)
// Users can exclude VFS with --where vfsType exists:false if needed // Users can exclude VFS with --where vfsType exists:false if needed
// Triple Intelligence Fusion - custom weighting // Triple Intelligence Fusion - custom weighting

View file

@ -312,7 +312,7 @@ ${chalk.cyan('Fork Statistics:')}
}, },
/** /**
* Migrate from v4.x to v5.0.0 (one-time) * Migrate storage format (one-time)
*/ */
async migrate(options: MigrateOptions) { async migrate(options: MigrateOptions) {
let spinner: any = null let spinner: any = null
@ -388,7 +388,7 @@ ${chalk.cyan('Fork Statistics:')}
await oldBrain.init() await oldBrain.init()
// Create new brain (v5.0.0) // Create new brain
const newBrain = new Brainy({ const newBrain = new Brainy({
storage: { storage: {
type: 'filesystem', type: 'filesystem',

View file

@ -117,7 +117,7 @@ export const neuralCommand = {
await handleNeighborsCommand(neural, argv) await handleNeighborsCommand(neural, argv)
break break
case 'path': case 'path':
console.log(chalk.yellow('\n⚠ Semantic path finding coming in v3.21.0')) console.log(chalk.yellow('\n⚠ Semantic path finding coming soon'))
console.log(chalk.dim('This feature requires implementing graph traversal algorithms')) console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
console.log(chalk.dim('Use "neighbors" and "hierarchy" commands to explore connections')) console.log(chalk.dim('Use "neighbors" and "hierarchy" commands to explore connections'))
break break
@ -148,7 +148,7 @@ async function promptForAction(): Promise<string> {
{ name: '🎯 Find semantic clusters', value: 'clusters' }, { name: '🎯 Find semantic clusters', value: 'clusters' },
{ name: '🌳 Show item hierarchy', value: 'hierarchy' }, { name: '🌳 Show item hierarchy', value: 'hierarchy' },
{ name: '🕸️ Find semantic neighbors', value: 'neighbors' }, { name: '🕸️ Find semantic neighbors', value: 'neighbors' },
{ name: '🛣️ Find semantic path between items (v3.21.0)', value: 'path', disabled: true }, { name: '🛣️ Find semantic path between items (coming soon)', value: 'path', disabled: true },
{ name: '🚨 Detect outliers', value: 'outliers' }, { name: '🚨 Detect outliers', value: 'outliers' },
{ name: '📊 Generate visualization data', value: 'visualize' } { name: '📊 Generate visualization data', value: 'visualize' }
] ]

View file

@ -1,5 +1,5 @@
/** /**
* 💾 Storage Management Commands - v4.0.0 * 💾 Storage Management Commands
* *
* Modern interactive CLI for storage lifecycle, cost optimization, and management * Modern interactive CLI for storage lifecycle, cost optimization, and management
*/ */

View file

@ -133,7 +133,7 @@ export const utilityCommands = {
// removeOrphans and rebuildIndex would require new Brainy APIs // removeOrphans and rebuildIndex would require new Brainy APIs
if (options.removeOrphans || options.rebuildIndex) { if (options.removeOrphans || options.rebuildIndex) {
spinner.warn('Advanced cleanup options not yet implemented') spinner.warn('Advanced cleanup options not yet implemented')
console.log(chalk.yellow('\n⚠ Advanced cleanup features coming in v3.21.0:')) console.log(chalk.yellow('\n⚠ Advanced cleanup features coming soon:'))
console.log(chalk.dim(' • --remove-orphans: Remove disconnected items')) console.log(chalk.dim(' • --remove-orphans: Remove disconnected items'))
console.log(chalk.dim(' • --rebuild-index: Rebuild vector index')) console.log(chalk.dim(' • --rebuild-index: Rebuild vector index'))
console.log(chalk.dim('\nUse "brainy clean" without options to clear the database')) console.log(chalk.dim('\nUse "brainy clean" without options to clear the database'))

View file

@ -21,7 +21,7 @@ let brainyInstance: Brainy | null = null
const getBrainy = async (): Promise<Brainy> => { const getBrainy = async (): Promise<Brainy> => {
if (!brainyInstance) { if (!brainyInstance) {
brainyInstance = new Brainy() brainyInstance = new Brainy()
await brainyInstance.init() // v5.0.1: Initialize brain (VFS auto-initialized here!) await brainyInstance.init() // Initialize brain (VFS auto-initialized here!)
} }
return brainyInstance return brainyInstance
} }
@ -52,8 +52,8 @@ export const vfsCommands = {
const spinner = ora('Reading file...').start() const spinner = ora('Reading file...').start()
try { try {
const brain = await getBrainy() // v5.0.1: Await async getBrainy const brain = await getBrainy() // Await async getBrainy
// v5.0.1: VFS auto-initialized, no need for vfs.init() // VFS auto-initialized, no need for vfs.init()
const buffer = await brain.vfs.readFile(path, { const buffer = await brain.vfs.readFile(path, {
encoding: options.encoding as any encoding: options.encoding as any
}) })

View file

@ -68,7 +68,7 @@ ${chalk.cyan('Examples:')}
$ brainy vfs search "React components" $ brainy vfs search "React components"
$ brainy vfs similar /code/Button.tsx $ brainy vfs similar /code/Button.tsx
${chalk.dim('# Storage management (v4.0.0)')} ${chalk.dim('# Storage management')}
$ brainy storage status --quota $ brainy storage status --quota
$ brainy storage lifecycle set ${chalk.dim('# Interactive mode')} $ brainy storage lifecycle set ${chalk.dim('# Interactive mode')}
$ brainy storage cost-estimate $ brainy storage cost-estimate
@ -217,11 +217,11 @@ program
program program
.command('path <from> <to>') .command('path <from> <to>')
.description('Find semantic path between items (v3.21.0)') .description('Find semantic path between items')
.option('--steps', 'Show step-by-step path') .option('--steps', 'Show step-by-step path')
.option('--max-hops <number>', 'Maximum path length', '5') .option('--max-hops <number>', 'Maximum path length', '5')
.action(() => { .action(() => {
console.log(chalk.yellow('\n⚠ Semantic path finding coming in v3.21.0')) console.log(chalk.yellow('\n⚠ Semantic path finding coming soon'))
console.log(chalk.dim('This feature requires implementing graph traversal algorithms')) console.log(chalk.dim('This feature requires implementing graph traversal algorithms'))
console.log(chalk.dim('Use "brainy neighbors" and "brainy hierarchy" to explore connections')) console.log(chalk.dim('Use "brainy neighbors" and "brainy hierarchy" to explore connections'))
}) })
@ -446,7 +446,7 @@ program
vfsCommands.tree(path, options) vfsCommands.tree(path, options)
}) })
// ===== Storage Management Commands (v4.0.0) ===== // ===== Storage Management Commands =====
program program
.command('storage') .command('storage')
@ -610,7 +610,7 @@ program
.option('--iterations <n>', 'Number of iterations', '100') .option('--iterations <n>', 'Number of iterations', '100')
.action(utilityCommands.benchmark) .action(utilityCommands.benchmark)
// ===== COW Commands (v5.0.0) - Instant Fork & Branching ===== // ===== COW Commands - Instant Fork & Branching =====
program program
.command('fork [name]') .command('fork [name]')

View file

@ -625,7 +625,7 @@ export async function promptCommand(): Promise<string> {
*/ */
export async function startInteractiveMode() { export async function startInteractiveMode() {
console.log(chalk.cyan('\n🧠 Brainy Interactive Mode\n')) console.log(chalk.cyan('\n🧠 Brainy Interactive Mode\n'))
console.log(chalk.yellow('Interactive REPL mode coming in v3.20.0\n')) console.log(chalk.yellow('Interactive REPL mode coming soon\n'))
console.log(chalk.dim('Use specific commands for now: brainy add, brainy search, etc.')) console.log(chalk.dim('Use specific commands for now: brainy add, brainy search, etc.'))
process.exit(0) process.exit(0)
} }

View file

@ -57,7 +57,7 @@ export type DistanceFunction = (a: Vector, b: Vector) => number
/** /**
* Embedding function for converting data to vectors * Embedding function for converting data to vectors
* v4.0.0: Now properly typed - accepts string, string array (batch), or object, no `any` * Now properly typed - accepts string, string array (batch), or object, no `any`
*/ */
export type EmbeddingFunction = (data: string | string[] | Record<string, unknown>) => Promise<Vector> export type EmbeddingFunction = (data: string | string[] | Record<string, unknown>) => Promise<Vector>
@ -72,7 +72,7 @@ export interface EmbeddingModel {
/** /**
* Embed data into a vector * Embed data into a vector
* v4.0.0: Now properly typed - accepts string, string array (batch), or object, no `any` * Now properly typed - accepts string, string array (batch), or object, no `any`
*/ */
embed(data: string | string[] | Record<string, unknown>): Promise<Vector> embed(data: string | string[] | Record<string, unknown>): Promise<Vector>
@ -83,9 +83,9 @@ export interface EmbeddingModel {
} }
/** /**
* HNSW graph noun - Pure vector structure (v4.0.0) * HNSW graph noun - Pure vector structure
* *
* v4.0.0 BREAKING CHANGE: metadata field removed * metadata field removed
* - Stores ONLY vector data for optimal memory usage * - Stores ONLY vector data for optimal memory usage
* - Metadata stored separately and combined on retrieval * - Metadata stored separately and combined on retrieval
* - 25% memory reduction @ 1B scale (no in-memory metadata) * - 25% memory reduction @ 1B scale (no in-memory metadata)
@ -100,15 +100,15 @@ export interface HNSWNoun {
} }
/** /**
* Lightweight verb for HNSW index storage - Core relational structure (v4.0.0) * Lightweight verb for HNSW index storage - Core relational structure
* *
* Core fields (v3.50.1+): verb/sourceId/targetId are first-class fields * Core fields: verb/sourceId/targetId are first-class fields
* These are NOT metadata - they're the essence of what a verb IS: * These are NOT metadata - they're the essence of what a verb IS:
* - verb: The relationship type (creates, contains, etc.) - needed for routing & display * - verb: The relationship type (creates, contains, etc.) - needed for routing & display
* - sourceId: What entity this verb connects FROM - needed for graph traversal * - sourceId: What entity this verb connects FROM - needed for graph traversal
* - targetId: What entity this verb connects TO - needed for graph traversal * - targetId: What entity this verb connects TO - needed for graph traversal
* *
* v4.0.0 BREAKING CHANGE: metadata field removed * metadata field removed
* - Stores ONLY vector + core relational data * - Stores ONLY vector + core relational data
* - User metadata (weight, custom fields) stored separately * - User metadata (weight, custom fields) stored separately
* - 10x faster metadata-only updates (skip HNSW rebuild) * - 10x faster metadata-only updates (skip HNSW rebuild)
@ -134,9 +134,9 @@ export interface HNSWVerb {
} }
/** /**
* Noun metadata structure (v4.8.0) * Noun metadata structure
* *
* v4.8.0 BREAKING CHANGE: Now contains ONLY custom user-defined fields * Now contains ONLY custom user-defined fields
* - Standard fields (confidence, weight, timestamps, etc.) moved to top-level in HNSWNounWithMetadata * - Standard fields (confidence, weight, timestamps, etc.) moved to top-level in HNSWNounWithMetadata
* - This interface represents custom metadata stored separately from vector data * - This interface represents custom metadata stored separately from vector data
* - Storage format unchanged (backward compatible at storage layer) * - Storage format unchanged (backward compatible at storage layer)
@ -162,9 +162,9 @@ export interface NounMetadata {
} }
/** /**
* Verb metadata structure (v4.8.0) * Verb metadata structure
* *
* v4.8.0 BREAKING CHANGE: Now contains ONLY custom user-defined fields * Now contains ONLY custom user-defined fields
* - Standard fields (weight, confidence, timestamps, etc.) moved to top-level in HNSWVerbWithMetadata * - Standard fields (weight, confidence, timestamps, etc.) moved to top-level in HNSWVerbWithMetadata
* - This interface represents custom metadata stored separately from vector + core relational data * - This interface represents custom metadata stored separately from vector + core relational data
* - Storage format unchanged (backward compatible at storage layer) * - Storage format unchanged (backward compatible at storage layer)
@ -190,9 +190,9 @@ export interface VerbMetadata {
} }
/** /**
* Combined noun structure for transport/API boundaries (v4.8.0) * Combined noun structure for transport/API boundaries
* *
* v4.8.0 BREAKING CHANGE: Standard fields moved to top-level * Standard fields moved to top-level
* - ALL standard fields (confidence, weight, timestamps, etc.) are now at top-level * - ALL standard fields (confidence, weight, timestamps, etc.) are now at top-level
* - metadata contains ONLY custom user-defined fields * - metadata contains ONLY custom user-defined fields
* - Provides clean, predictable API: entity.confidence always works * - Provides clean, predictable API: entity.confidence always works
@ -230,9 +230,9 @@ export interface HNSWNounWithMetadata {
} }
/** /**
* Combined verb structure for transport/API boundaries (v4.8.0) * Combined verb structure for transport/API boundaries
* *
* v4.8.0 BREAKING CHANGE: Standard fields moved to top-level * Standard fields moved to top-level
* - ALL standard fields (weight, confidence, timestamps, etc.) are now at top-level * - ALL standard fields (weight, confidence, timestamps, etc.) are now at top-level
* - metadata contains ONLY custom user-defined fields * - metadata contains ONLY custom user-defined fields
* - Provides clean, predictable API: verb.weight always works * - Provides clean, predictable API: verb.weight always works
@ -308,7 +308,7 @@ export interface HNSWConfig {
efSearch: number // Size of the dynamic candidate list during search efSearch: number // Size of the dynamic candidate list during search
ml: number // Maximum level ml: number // Maximum level
useDiskBasedIndex?: boolean // Whether to use disk-based index useDiskBasedIndex?: boolean // Whether to use disk-based index
maxConcurrentNeighborWrites?: number // Maximum concurrent neighbor updates during insert (v4.10.0+). Default: unlimited (full concurrency) maxConcurrentNeighborWrites?: number // Maximum concurrent neighbor updates during insert. Default: unlimited (full concurrency)
} }
/** /**
@ -548,7 +548,7 @@ export interface StatisticsData {
} }
/** /**
* Change record for getChangesSince (v4.0.0) * Change record for getChangesSince
* Replaces `any[]` with properly typed structure * Replaces `any[]` with properly typed structure
*/ */
export interface Change { export interface Change {
@ -563,27 +563,27 @@ export interface StorageAdapter {
init(): Promise<void> init(): Promise<void>
/** /**
* Save noun - Pure HNSW vector data only (v4.0.0) * Save noun - Pure HNSW vector data only
* @param noun Pure HNSW vector data (no metadata) * @param noun Pure HNSW vector data (no metadata)
* Note: Use saveNounMetadata() to save metadata separately * Note: Use saveNounMetadata() to save metadata separately
*/ */
saveNoun(noun: HNSWNoun): Promise<void> saveNoun(noun: HNSWNoun): Promise<void>
/** /**
* Save noun metadata separately (v4.0.0) * Save noun metadata separately
* @param id Noun ID * @param id Noun ID
* @param metadata Noun metadata * @param metadata Noun metadata
*/ */
saveNounMetadata(id: string, metadata: NounMetadata): Promise<void> saveNounMetadata(id: string, metadata: NounMetadata): Promise<void>
/** /**
* Delete noun metadata (v4.0.0) * Delete noun metadata
* @param id Noun ID * @param id Noun ID
*/ */
deleteNounMetadata(id: string): Promise<void> deleteNounMetadata(id: string): Promise<void>
/** /**
* Get noun with metadata combined (v4.0.0) * Get noun with metadata combined
* @returns Combined HNSWNounWithMetadata or null * @returns Combined HNSWNounWithMetadata or null
*/ */
getNoun(id: string): Promise<HNSWNounWithMetadata | null> getNoun(id: string): Promise<HNSWNounWithMetadata | null>
@ -622,14 +622,14 @@ export interface StorageAdapter {
deleteNoun(id: string): Promise<void> deleteNoun(id: string): Promise<void>
/** /**
* Save verb - Pure HNSW verb with core fields only (v4.0.0) * Save verb - Pure HNSW verb with core fields only
* @param verb Pure HNSW verb data (vector + core fields, no user metadata) * @param verb Pure HNSW verb data (vector + core fields, no user metadata)
* Note: Use saveVerbMetadata() to save metadata separately * Note: Use saveVerbMetadata() to save metadata separately
*/ */
saveVerb(verb: HNSWVerb): Promise<void> saveVerb(verb: HNSWVerb): Promise<void>
/** /**
* Get verb with metadata combined (v4.0.0) * Get verb with metadata combined
* @returns Combined HNSWVerbWithMetadata or null * @returns Combined HNSWVerbWithMetadata or null
*/ */
getVerb(id: string): Promise<HNSWVerbWithMetadata | null> getVerb(id: string): Promise<HNSWVerbWithMetadata | null>
@ -686,14 +686,14 @@ export interface StorageAdapter {
deleteVerb(id: string): Promise<void> deleteVerb(id: string): Promise<void>
/** /**
* Save metadata (v4.0.0: now typed) * Save metadata
* @param id Entity ID * @param id Entity ID
* @param metadata Typed noun metadata * @param metadata Typed noun metadata
*/ */
saveMetadata(id: string, metadata: NounMetadata): Promise<void> saveMetadata(id: string, metadata: NounMetadata): Promise<void>
/** /**
* Get metadata (v4.0.0: now typed) * Get metadata
* @param id Entity ID * @param id Entity ID
* @returns Typed noun metadata or null * @returns Typed noun metadata or null
*/ */
@ -702,26 +702,26 @@ export interface StorageAdapter {
/** /**
* Get multiple metadata objects in batches (prevents socket exhaustion) * Get multiple metadata objects in batches (prevents socket exhaustion)
* @param ids Array of IDs to get metadata for * @param ids Array of IDs to get metadata for
* @returns Promise that resolves to a Map of id -> metadata (v4.0.0: typed) * @returns Promise that resolves to a Map of id -> metadata
*/ */
getMetadataBatch?(ids: string[]): Promise<Map<string, NounMetadata>> getMetadataBatch?(ids: string[]): Promise<Map<string, NounMetadata>>
/** /**
* Get noun metadata from storage (v4.0.0: now typed) * Get noun metadata from storage
* @param id The ID of the noun * @param id The ID of the noun
* @returns Promise that resolves to the metadata or null if not found * @returns Promise that resolves to the metadata or null if not found
*/ */
getNounMetadata(id: string): Promise<NounMetadata | null> getNounMetadata(id: string): Promise<NounMetadata | null>
/** /**
* Batch get multiple nouns with vectors (v6.2.0 - N+1 fix) * Batch get multiple nouns with vectors
* @param ids Array of noun IDs to fetch * @param ids Array of noun IDs to fetch
* @returns Map of id HNSWNounWithMetadata (only successful reads included) * @returns Map of id HNSWNounWithMetadata (only successful reads included)
*/ */
getNounBatch?(ids: string[]): Promise<Map<string, HNSWNounWithMetadata>> getNounBatch?(ids: string[]): Promise<Map<string, HNSWNounWithMetadata>>
/** /**
* Save verb metadata to storage (v4.0.0: now typed) * Save verb metadata to storage
* @param id The ID of the verb * @param id The ID of the verb
* @param metadata The metadata to save * @param metadata The metadata to save
* @returns Promise that resolves when the metadata is saved * @returns Promise that resolves when the metadata is saved
@ -729,14 +729,14 @@ export interface StorageAdapter {
saveVerbMetadata(id: string, metadata: VerbMetadata): Promise<void> saveVerbMetadata(id: string, metadata: VerbMetadata): Promise<void>
/** /**
* Get verb metadata from storage (v4.0.0: now typed) * Get verb metadata from storage
* @param id The ID of the verb * @param id The ID of the verb
* @returns Promise that resolves to the metadata or null if not found * @returns Promise that resolves to the metadata or null if not found
*/ */
getVerbMetadata(id: string): Promise<VerbMetadata | null> getVerbMetadata(id: string): Promise<VerbMetadata | null>
/** /**
* Batch get multiple verbs (v6.2.0 - N+1 fix) * Batch get multiple verbs
* @param ids Array of verb IDs to fetch * @param ids Array of verb IDs to fetch
* @returns Map of id HNSWVerbWithMetadata (only successful reads included) * @returns Map of id HNSWVerbWithMetadata (only successful reads included)
*/ */
@ -745,7 +745,7 @@ export interface StorageAdapter {
clear(): Promise<void> clear(): Promise<void>
/** /**
* Batch delete multiple objects from storage (v4.0.0) * Batch delete multiple objects from storage
* Efficient deletion of large numbers of entities using cloud provider batch APIs. * Efficient deletion of large numbers of entities using cloud provider batch APIs.
* Significantly faster and cheaper than individual deletes (up to 1000x speedup). * Significantly faster and cheaper than individual deletes (up to 1000x speedup).
* *
@ -853,7 +853,7 @@ export interface StorageAdapter {
flushStatisticsToStorage(): Promise<void> flushStatisticsToStorage(): Promise<void>
/** /**
* Track field names from a JSON document (v4.0.0: now typed) * Track field names from a JSON document
* @param jsonDocument The JSON document to extract field names from * @param jsonDocument The JSON document to extract field names from
* @param service The service that inserted the data * @param service The service that inserted the data
*/ */
@ -872,7 +872,7 @@ export interface StorageAdapter {
getStandardFieldMappings(): Promise<Record<string, Record<string, string[]>>> getStandardFieldMappings(): Promise<Record<string, Record<string, string[]>>>
/** /**
* Get changes since a specific timestamp (v4.0.0: now typed) * Get changes since a specific timestamp
* @param timestamp The timestamp to get changes since * @param timestamp The timestamp to get changes since
* @param limit Optional limit on the number of changes to return * @param limit Optional limit on the number of changes to return
* @returns Promise that resolves to an array of properly typed changes * @returns Promise that resolves to an array of properly typed changes

View file

@ -26,7 +26,7 @@ use js_sys::{Array, Float32Array};
use tokenizers::Tokenizer; use tokenizers::Tokenizer;
use wasm_bindgen::prelude::*; use wasm_bindgen::prelude::*;
// v7.2.0: Model weights are NO LONGER embedded in WASM // Model weights are NO LONGER embedded in WASM
// //
// Previous design: 90MB WASM with model weights embedded via include_bytes!() // Previous design: 90MB WASM with model weights embedded via include_bytes!()
// Problem: WASM parsing/compilation took 139+ seconds on throttled CPU (Cloud Run) // Problem: WASM parsing/compilation took 139+ seconds on throttled CPU (Cloud Run)
@ -78,7 +78,7 @@ impl EmbeddingEngine {
/// Load the model and tokenizer from bytes /// Load the model and tokenizer from bytes
/// ///
/// v7.2.0: This is now the ONLY way to initialize the engine. /// This is now the ONLY way to initialize the engine.
/// Model weights are no longer embedded in WASM for faster initialization. /// Model weights are no longer embedded in WASM for faster initialization.
/// ///
/// # Arguments /// # Arguments

View file

@ -5,7 +5,7 @@
* Pure Rust/WASM implementation for sentence embeddings. * Pure Rust/WASM implementation for sentence embeddings.
* Works with Bun, Node.js, Bun --compile, and browsers. * Works with Bun, Node.js, Bun --compile, and browsers.
* *
* v7.2.0 Architecture (20x faster initialization): * Architecture (20x faster initialization):
* - WASM file: ~2.4MB (inference code only) * - WASM file: ~2.4MB (inference code only)
* - Model files: ~88MB (loaded separately as raw bytes) * - Model files: ~88MB (loaded separately as raw bytes)
* - Init time: ~5-7 seconds (vs 139 seconds with embedded model) * - Init time: ~5-7 seconds (vs 139 seconds with embedded model)
@ -34,7 +34,7 @@ interface CandleWasmModule {
} }
interface CandleEngineInstance { interface CandleEngineInstance {
// v7.2.0: load() is the only initialization method (no more embedded model) // load() is the only initialization method (no more embedded model)
load(modelBytes: Uint8Array, tokenizerBytes: Uint8Array, configBytes: Uint8Array): void load(modelBytes: Uint8Array, tokenizerBytes: Uint8Array, configBytes: Uint8Array): void
is_ready(): boolean is_ready(): boolean
embed(text: string): Float32Array embed(text: string): Float32Array
@ -54,7 +54,7 @@ let globalInitPromise: Promise<void> | null = null
* Uses the Candle ML framework (Rust/WASM) for inference. * Uses the Candle ML framework (Rust/WASM) for inference.
* Supports all-MiniLM-L6-v2 with 384-dimensional embeddings. * Supports all-MiniLM-L6-v2 with 384-dimensional embeddings.
* *
* v7.2.0: Model weights are loaded separately from WASM for 20x faster init. * Model weights are loaded separately from WASM for 20x faster init.
* For bun --compile deployments, both WASM and model files are automatically * For bun --compile deployments, both WASM and model files are automatically
* embedded in the binary - single file deployment still works. * embedded in the binary - single file deployment still works.
*/ */
@ -104,7 +104,7 @@ export class CandleEmbeddingEngine {
/** /**
* Perform actual initialization * Perform actual initialization
* *
* v7.2.0: WASM and model files are loaded separately for 20x faster init. * WASM and model files are loaded separately for 20x faster init.
* - WASM (~2.4MB): Compiles in ~3-5 seconds * - WASM (~2.4MB): Compiles in ~3-5 seconds
* - Model (~88MB): Loads as raw bytes in ~1-2 seconds * - Model (~88MB): Loads as raw bytes in ~1-2 seconds
* - Total: ~5-7 seconds (vs 139 seconds with embedded model) * - Total: ~5-7 seconds (vs 139 seconds with embedded model)
@ -153,7 +153,7 @@ export class CandleEmbeddingEngine {
/** /**
* Load the WASM module * Load the WASM module
* *
* v7.2.0: WASM is now only ~2.4MB (inference code only, no model weights). * WASM is now only ~2.4MB (inference code only, no model weights).
* Uses wasmLoader.ts for cross-environment compatibility. * Uses wasmLoader.ts for cross-environment compatibility.
*/ */
private async loadWasmModule(): Promise<CandleWasmModule> { private async loadWasmModule(): Promise<CandleWasmModule> {

View file

@ -1,7 +1,7 @@
/** /**
* Universal Model Loader for Candle Embeddings * Universal Model Loader for Candle Embeddings
* *
* v7.2.0: Model weights are now loaded separately from WASM for faster initialization. * Model weights are now loaded separately from WASM for faster initialization.
* This reduces WASM compilation time from 139 seconds to ~3-5 seconds. * This reduces WASM compilation time from 139 seconds to ~3-5 seconds.
* *
* Loads model files from appropriate source based on environment: * Loads model files from appropriate source based on environment:

View file

@ -45,7 +45,7 @@ export class GraphAdjacencyIndex {
private lsmTreeVerbsBySource: LSMTree // sourceId -> verbIds private lsmTreeVerbsBySource: LSMTree // sourceId -> verbIds
private lsmTreeVerbsByTarget: LSMTree // targetId -> verbIds private lsmTreeVerbsByTarget: LSMTree // targetId -> verbIds
// v5.7.0: ID-only tracking for billion-scale memory optimization // ID-only tracking for billion-scale memory optimization
// Previous: Map<string, GraphVerb> stored full objects (128GB @ 1B verbs) // Previous: Map<string, GraphVerb> stored full objects (128GB @ 1B verbs)
// Now: Set<string> stores only IDs (~100KB @ 1B verbs) = 1,280,000x reduction // Now: Set<string> stores only IDs (~100KB @ 1B verbs) = 1,280,000x reduction
private verbIdSet = new Set<string>() private verbIdSet = new Set<string>()
@ -117,7 +117,7 @@ export class GraphAdjacencyIndex {
/** /**
* Initialize the graph index (lazy initialization) * Initialize the graph index (lazy initialization)
* v6.3.0: Added defensive auto-rebuild check for verbIdSet consistency * Added defensive auto-rebuild check for verbIdSet consistency
*/ */
private async ensureInitialized(): Promise<void> { private async ensureInitialized(): Promise<void> {
if (this.initialized) { if (this.initialized) {
@ -129,7 +129,7 @@ export class GraphAdjacencyIndex {
await this.lsmTreeVerbsBySource.init() await this.lsmTreeVerbsBySource.init()
await this.lsmTreeVerbsByTarget.init() await this.lsmTreeVerbsByTarget.init()
// v6.3.0: Defensive check - if LSM-trees have data but verbIdSet is empty, // Defensive check - if LSM-trees have data but verbIdSet is empty,
// the index was created without proper rebuild (shouldn't happen with singleton // the index was created without proper rebuild (shouldn't happen with singleton
// pattern but protects against edge cases and future refactoring) // pattern but protects against edge cases and future refactoring)
const lsmTreeSize = this.lsmTreeVerbsBySource.size() const lsmTreeSize = this.lsmTreeVerbsBySource.size()
@ -151,7 +151,7 @@ export class GraphAdjacencyIndex {
} }
/** /**
* Populate verbIdSet from storage without full rebuild (v6.3.0) * Populate verbIdSet from storage without full rebuild
* Lighter weight than full rebuild - only loads verb IDs, not all verb data * Lighter weight than full rebuild - only loads verb IDs, not all verb data
* @private * @private
*/ */
@ -192,7 +192,7 @@ export class GraphAdjacencyIndex {
* Core API - Neighbor lookup with LSM-tree storage * Core API - Neighbor lookup with LSM-tree storage
* *
* O(log n) with bloom filter optimization (90% of queries skip disk I/O) * O(log n) with bloom filter optimization (90% of queries skip disk I/O)
* v5.8.0: Added pagination support for high-degree nodes * Added pagination support for high-degree nodes
* *
* @param id Entity ID to get neighbors for * @param id Entity ID to get neighbors for
* @param optionsOrDirection Optional: direction string OR options object * @param optionsOrDirection Optional: direction string OR options object
@ -252,7 +252,7 @@ export class GraphAdjacencyIndex {
// Convert to array for pagination // Convert to array for pagination
let result = Array.from(neighbors) let result = Array.from(neighbors)
// Apply pagination if requested (v5.8.0) // Apply pagination if requested
if (options?.limit !== undefined || options?.offset !== undefined) { if (options?.limit !== undefined || options?.offset !== undefined) {
const offset = options.offset || 0 const offset = options.offset || 0
const limit = options.limit !== undefined ? options.limit : result.length const limit = options.limit !== undefined ? options.limit : result.length
@ -273,8 +273,8 @@ export class GraphAdjacencyIndex {
* Get verb IDs by source - Billion-scale optimization for getVerbsBySource * Get verb IDs by source - Billion-scale optimization for getVerbsBySource
* *
* O(log n) LSM-tree lookup with bloom filter optimization * O(log n) LSM-tree lookup with bloom filter optimization
* v5.7.1: Filters out deleted verb IDs (tombstone deletion workaround) * Filters out deleted verb IDs (tombstone deletion workaround)
* v5.8.0: Added pagination support for entities with many relationships * Added pagination support for entities with many relationships
* *
* @param sourceId Source entity ID * @param sourceId Source entity ID
* @param options Optional configuration * @param options Optional configuration
@ -318,7 +318,7 @@ export class GraphAdjacencyIndex {
const allIds = verbIds || [] const allIds = verbIds || []
let result = allIds.filter(id => this.verbIdSet.has(id)) let result = allIds.filter(id => this.verbIdSet.has(id))
// Apply pagination if requested (v5.8.0) // Apply pagination if requested
if (options?.limit !== undefined || options?.offset !== undefined) { if (options?.limit !== undefined || options?.offset !== undefined) {
const offset = options.offset || 0 const offset = options.offset || 0
const limit = options.limit !== undefined ? options.limit : result.length const limit = options.limit !== undefined ? options.limit : result.length
@ -332,8 +332,8 @@ export class GraphAdjacencyIndex {
* Get verb IDs by target - Billion-scale optimization for getVerbsByTarget * Get verb IDs by target - Billion-scale optimization for getVerbsByTarget
* *
* O(log n) LSM-tree lookup with bloom filter optimization * O(log n) LSM-tree lookup with bloom filter optimization
* v5.7.1: Filters out deleted verb IDs (tombstone deletion workaround) * Filters out deleted verb IDs (tombstone deletion workaround)
* v5.8.0: Added pagination support for popular target entities * Added pagination support for popular target entities
* *
* @param targetId Target entity ID * @param targetId Target entity ID
* @param options Optional configuration * @param options Optional configuration
@ -377,7 +377,7 @@ export class GraphAdjacencyIndex {
const allIds = verbIds || [] const allIds = verbIds || []
let result = allIds.filter(id => this.verbIdSet.has(id)) let result = allIds.filter(id => this.verbIdSet.has(id))
// Apply pagination if requested (v5.8.0) // Apply pagination if requested
if (options?.limit !== undefined || options?.offset !== undefined) { if (options?.limit !== undefined || options?.offset !== undefined) {
const offset = options.offset || 0 const offset = options.offset || 0
const limit = options.limit !== undefined ? options.limit : result.length const limit = options.limit !== undefined ? options.limit : result.length
@ -414,7 +414,7 @@ export class GraphAdjacencyIndex {
} }
/** /**
* Batch get multiple verbs with caching (v6.2.0 - N+1 fix) * Batch get multiple verbs with caching
* *
* **Performance**: Eliminates N+1 pattern for verb loading * **Performance**: Eliminates N+1 pattern for verb loading
* - Current: N × getVerbCached() = N × 50ms on GCS = 250ms for 5 verbs * - Current: N × getVerbCached() = N × 50ms on GCS = 250ms for 5 verbs
@ -433,7 +433,6 @@ export class GraphAdjacencyIndex {
* @param verbIds Array of verb IDs to fetch * @param verbIds Array of verb IDs to fetch
* @returns Map of verbId GraphVerb (only successful reads included) * @returns Map of verbId GraphVerb (only successful reads included)
* *
* @since v6.2.0
*/ */
async getVerbsBatchCached(verbIds: string[]): Promise<Map<string, GraphVerb>> { async getVerbsBatchCached(verbIds: string[]): Promise<Map<string, GraphVerb>> {
const results = new Map<string, GraphVerb>() const results = new Map<string, GraphVerb>()
@ -514,7 +513,7 @@ export class GraphAdjacencyIndex {
const targetStats = this.lsmTreeTarget.getStats() const targetStats = this.lsmTreeTarget.getStats()
// Note: Exact unique node counts would require full LSM-tree scan // Note: Exact unique node counts would require full LSM-tree scan
// v5.7.0: Using verbIdSet (ID-only tracking) for memory efficiency // Using verbIdSet (ID-only tracking) for memory efficiency
const uniqueSourceNodes = this.verbIdSet.size const uniqueSourceNodes = this.verbIdSet.size
const uniqueTargetNodes = this.verbIdSet.size const uniqueTargetNodes = this.verbIdSet.size
const totalNodes = this.verbIdSet.size const totalNodes = this.verbIdSet.size
@ -622,13 +621,13 @@ export class GraphAdjacencyIndex {
// Clear current index // Clear current index
this.verbIdSet.clear() this.verbIdSet.clear()
this.totalRelationshipsIndexed = 0 this.totalRelationshipsIndexed = 0
// v6.2.4: CRITICAL FIX - Clear relationship counts to prevent accumulation // CRITICAL FIX - Clear relationship counts to prevent accumulation
this.relationshipCountsByType.clear() this.relationshipCountsByType.clear()
// Note: LSM-trees will be recreated from storage via their own initialization // Note: LSM-trees will be recreated from storage via their own initialization
// Verb data will be loaded on-demand via UnifiedCache // Verb data will be loaded on-demand via UnifiedCache
// Adaptive loading strategy based on storage type (v4.2.4) // Adaptive loading strategy based on storage type
const storageType = this.storage?.constructor.name || '' const storageType = this.storage?.constructor.name || ''
const isLocalStorage = const isLocalStorage =
storageType === 'FileSystemStorage' || storageType === 'FileSystemStorage' ||
@ -754,7 +753,7 @@ export class GraphAdjacencyIndex {
bytes += targetStats.memTableMemory bytes += targetStats.memTableMemory
// Verb ID set (memory-efficient: IDs only, ~8 bytes per ID pointer) // Verb ID set (memory-efficient: IDs only, ~8 bytes per ID pointer)
// v5.7.0: Previous verbIndex Map stored full objects (128 bytes each = 128GB @ 1B verbs) // Previous verbIndex Map stored full objects (128 bytes each = 128GB @ 1B verbs)
// Now: verbIdSet stores only IDs (~8 bytes each = ~100KB @ 1B verbs) = 1,280,000x reduction // Now: verbIdSet stores only IDs (~8 bytes each = ~100KB @ 1B verbs) = 1,280,000x reduction
bytes += this.verbIdSet.size * 8 bytes += this.verbIdSet.size * 8
@ -792,7 +791,7 @@ export class GraphAdjacencyIndex {
/** /**
* Flush LSM-tree MemTables to disk * Flush LSM-tree MemTables to disk
* CRITICAL FIX (v3.43.2): Now public so it can be called from brain.flush() * CRITICAL FIX: Now public so it can be called from brain.flush()
*/ */
async flush(): Promise<void> { async flush(): Promise<void> {
if (!this.initialized) { if (!this.initialized) {

View file

@ -35,18 +35,18 @@ export class HNSWIndex {
private distanceFunction: DistanceFunction private distanceFunction: DistanceFunction
private dimension: number | null = null private dimension: number | null = null
private useParallelization: boolean = true // Whether to use parallelization for performance-critical operations private useParallelization: boolean = true // Whether to use parallelization for performance-critical operations
private storage: BaseStorage | null = null // Storage adapter for HNSW persistence (v3.35.0+) private storage: BaseStorage | null = null // Storage adapter for HNSW persistence
// Universal memory management (v3.36.0+) // Universal memory management
private unifiedCache: UnifiedCache // Shared cache with Graph and Metadata indexes private unifiedCache: UnifiedCache // Shared cache with Graph and Metadata indexes
// Always-adaptive caching (v3.36.0+) - no "mode" concept, system adapts automatically // Always-adaptive caching - no "mode" concept, system adapts automatically
// COW (Copy-on-Write) support - v5.0.0 // COW (Copy-on-Write) support
private cowEnabled: boolean = false private cowEnabled: boolean = false
private cowModifiedNodes: Set<string> = new Set() private cowModifiedNodes: Set<string> = new Set()
private cowParent: HNSWIndex | null = null private cowParent: HNSWIndex | null = null
// v6.2.8: Deferred HNSW persistence for cloud storage performance // Deferred HNSW persistence for cloud storage performance
// In deferred mode, HNSW connections are only persisted on flush/close // In deferred mode, HNSW connections are only persisted on flush/close
// This reduces GCS operations from 70 to 2-3 per add() (30-50× faster) // This reduces GCS operations from 70 to 2-3 per add() (30-50× faster)
private persistMode: 'immediate' | 'deferred' = 'immediate' private persistMode: 'immediate' | 'deferred' = 'immediate'
@ -86,7 +86,7 @@ export class HNSWIndex {
} }
/** /**
* v6.2.8: Flush dirty HNSW data to storage * Flush dirty HNSW data to storage
* *
* In deferred persistence mode, HNSW connections are tracked as dirty but not * In deferred persistence mode, HNSW connections are tracked as dirty but not
* immediately persisted. Call flush() to persist all pending changes. * immediately persisted. Call flush() to persist all pending changes.
@ -361,6 +361,19 @@ export class HNSWIndex {
this.entryPointId = id this.entryPointId = id
this.maxLevel = nounLevel this.maxLevel = nounLevel
this.nouns.set(id, noun) this.nouns.set(id, noun)
// Persist system data for first noun (previously skipped)
if (this.storage && this.persistMode === 'immediate') {
await this.storage.saveHNSWSystem({
entryPointId: this.entryPointId,
maxLevel: this.maxLevel
}).catch(error => {
console.error('Failed to persist initial HNSW system data:', error)
})
} else if (this.persistMode === 'deferred') {
this.dirtySystem = true
}
return id return id
} }
@ -438,7 +451,7 @@ export class HNSWIndex {
) )
// Add bidirectional connections // Add bidirectional connections
// PERFORMANCE OPTIMIZATION (v4.10.0): Collect all neighbor updates for concurrent execution // PERFORMANCE OPTIMIZATION: Collect all neighbor updates for concurrent execution
const neighborUpdates: Array<{ const neighborUpdates: Array<{
neighborId: string neighborId: string
promise: Promise<void> promise: Promise<void>
@ -467,9 +480,9 @@ export class HNSWIndex {
await this.pruneConnections(neighbor, level) await this.pruneConnections(neighbor, level)
} }
// Persist updated neighbor HNSW data (v3.35.0+) // Persist updated neighbor HNSW data
// //
// v6.2.8: Deferred persistence mode for cloud storage performance // Deferred persistence mode for cloud storage performance
// In deferred mode, we track dirty nodes instead of persisting immediately // In deferred mode, we track dirty nodes instead of persisting immediately
// This reduces GCS operations from 70 to 2-3 per add() (30-50× faster) // This reduces GCS operations from 70 to 2-3 per add() (30-50× faster)
if (this.storage && this.persistMode === 'immediate') { if (this.storage && this.persistMode === 'immediate') {
@ -561,8 +574,8 @@ export class HNSWIndex {
this.highLevelNodes.get(nounLevel)!.add(id) this.highLevelNodes.get(nounLevel)!.add(id)
} }
// Persist HNSW graph data to storage (v3.35.0+) // Persist HNSW graph data to storage
// v6.2.8: Respect persistMode setting // Respect persistMode setting
if (this.storage && this.persistMode === 'immediate') { if (this.storage && this.persistMode === 'immediate') {
// IMMEDIATE MODE: Original behavior - persist new entity and system data // IMMEDIATE MODE: Original behavior - persist new entity and system data
const connectionsObj: Record<string, string[]> = {} const connectionsObj: Record<string, string[]> = {}
@ -594,7 +607,7 @@ export class HNSWIndex {
} }
/** /**
* O(1) entry point recovery using highLevelNodes index (v6.2.3). * O(1) entry point recovery using highLevelNodes index.
* At any reasonable scale (1000+ nodes), level 2+ nodes are guaranteed to exist. * At any reasonable scale (1000+ nodes), level 2+ nodes are guaranteed to exist.
* For tiny indexes with only level 0-1 nodes, any node works as entry point. * For tiny indexes with only level 0-1 nodes, any node works as entry point.
*/ */
@ -640,7 +653,7 @@ export class HNSWIndex {
} }
// Start from the entry point // Start from the entry point
// If entry point is null but nouns exist, attempt O(1) recovery (v6.2.3) // If entry point is null but nouns exist, attempt O(1) recovery
if (!this.entryPointId && this.nouns.size > 0) { if (!this.entryPointId && this.nouns.size > 0) {
const { id: recoveredId, level: recoveredLevel } = this.recoverEntryPointO1() const { id: recoveredId, level: recoveredLevel } = this.recoverEntryPointO1()
if (recoveredId) { if (recoveredId) {
@ -656,7 +669,7 @@ export class HNSWIndex {
let entryPoint = this.nouns.get(this.entryPointId) let entryPoint = this.nouns.get(this.entryPointId)
if (!entryPoint) { if (!entryPoint) {
// Entry point ID exists but noun was deleted - O(1) recovery (v6.2.3) // Entry point ID exists but noun was deleted - O(1) recovery
if (this.nouns.size > 0) { if (this.nouns.size > 0) {
const { id: recoveredId, level: recoveredLevel } = this.recoverEntryPointO1() const { id: recoveredId, level: recoveredLevel } = this.recoverEntryPointO1()
if (recoveredId) { if (recoveredId) {
@ -944,7 +957,7 @@ export class HNSWIndex {
/** /**
* Get vector safely (always uses adaptive caching via UnifiedCache) * Get vector safely (always uses adaptive caching via UnifiedCache)
* *
* Production-grade adaptive caching (v3.36.0+): * Production-grade adaptive caching:
* - Vector already loaded: Returns immediately (O(1)) * - Vector already loaded: Returns immediately (O(1))
* - Vector in cache: Loads from UnifiedCache (O(1) hash lookup) * - Vector in cache: Loads from UnifiedCache (O(1) hash lookup)
* - Vector on disk: Loads from storage UnifiedCache (O(disk)) * - Vector on disk: Loads from storage UnifiedCache (O(disk))
@ -990,7 +1003,7 @@ export class HNSWIndex {
} }
/** /**
* Get vector synchronously if available in memory (v3.36.0+) * Get vector synchronously if available in memory
* *
* Sync fast path optimization: * Sync fast path optimization:
* - Vector in memory: Returns immediately (zero overhead) * - Vector in memory: Returns immediately (zero overhead)
@ -1048,7 +1061,7 @@ export class HNSWIndex {
} }
/** /**
* Calculate distance with sync fast path (v3.36.0+) * Calculate distance with sync fast path
* *
* Eliminates async overhead when vectors are in memory: * Eliminates async overhead when vectors are in memory:
* - Sync path: Vector in memory returns number (zero overhead) * - Sync path: Vector in memory returns number (zero overhead)
@ -1093,7 +1106,7 @@ export class HNSWIndex {
} }
/** /**
* Rebuild HNSW index from persisted graph data (v3.35.0+) * Rebuild HNSW index from persisted graph data
* *
* This is a production-grade O(N) rebuild that restores the pre-computed graph structure * This is a production-grade O(N) rebuild that restores the pre-computed graph structure
* from storage. Much faster than re-building which is O(N log N). * from storage. Much faster than re-building which is O(N log N).
@ -1157,7 +1170,7 @@ export class HNSWIndex {
) )
} }
// Step 4: Adaptive loading strategy based on storage type (v4.2.4) // Step 4: Adaptive loading strategy based on storage type
// FileSystem/Memory/OPFS: Load all at once (avoids repeated getAllShardedFiles() calls) // FileSystem/Memory/OPFS: Load all at once (avoids repeated getAllShardedFiles() calls)
// Cloud (GCS/S3/R2): Use pagination (efficient native cloud APIs) // Cloud (GCS/S3/R2): Use pagination (efficient native cloud APIs)
const storageType = this.storage?.constructor.name || '' const storageType = this.storage?.constructor.name || ''
@ -1238,7 +1251,7 @@ export class HNSWIndex {
prodLog.info(`HNSW: Using cloud pagination strategy (${storageType})`) prodLog.info(`HNSW: Using cloud pagination strategy (${storageType})`)
let hasMore = true let hasMore = true
let offset = 0 // v5.7.11: Use offset-based pagination instead of cursor (bug fix for infinite loop) let offset = 0 // Use offset-based pagination instead of cursor (bug fix for infinite loop)
while (hasMore) { while (hasMore) {
// Fetch batch of nouns from storage (cast needed as method is not in base interface) // Fetch batch of nouns from storage (cast needed as method is not in base interface)
@ -1249,7 +1262,7 @@ export class HNSWIndex {
nextCursor?: string nextCursor?: string
} = await (this.storage as any).getNounsWithPagination({ } = await (this.storage as any).getNounsWithPagination({
limit: batchSize, limit: batchSize,
offset // v5.7.11: Pass offset for proper pagination (previously passed cursor which was ignored) offset // Pass offset for proper pagination (previously passed cursor which was ignored)
}) })
// Set total count on first batch // Set total count on first batch
@ -1307,11 +1320,11 @@ export class HNSWIndex {
// Check for more data // Check for more data
hasMore = result.hasMore hasMore = result.hasMore
offset += batchSize // v5.7.11: Increment offset for next page offset += batchSize // Increment offset for next page
} }
} }
// Step 5: CRITICAL - Recover entry point if missing (v6.2.3 - O(1)) // Step 5: CRITICAL - Recover entry point if missing)
// This ensures consistency even if getHNSWSystem() returned null // This ensures consistency even if getHNSWSystem() returned null
if (this.nouns.size > 0 && this.entryPointId === null) { if (this.nouns.size > 0 && this.entryPointId === null) {
prodLog.warn('HNSW rebuild: Entry point was null after loading nouns - recovering with O(1) lookup') prodLog.warn('HNSW rebuild: Entry point was null after loading nouns - recovering with O(1) lookup')
@ -1426,7 +1439,7 @@ export class HNSWIndex {
} }
/** /**
* Get cache performance statistics for monitoring and diagnostics (v3.36.0+) * Get cache performance statistics for monitoring and diagnostics
* *
* Production-grade monitoring: * Production-grade monitoring:
* - Adaptive caching strategy (preloading vs on-demand) * - Adaptive caching strategy (preloading vs on-demand)

View file

@ -118,7 +118,7 @@ export class TypeAwareHNSWIndex {
} }
/** /**
* v6.2.8: Flush dirty HNSW data to storage for all type-specific indexes * Flush dirty HNSW data to storage for all type-specific indexes
* *
* In deferred persistence mode, HNSW connections are tracked as dirty but not * In deferred persistence mode, HNSW connections are tracked as dirty but not
* immediately persisted. Call flush() to persist all pending changes across * immediately persisted. Call flush() to persist all pending changes across
@ -502,7 +502,7 @@ export class TypeAwareHNSWIndex {
// Load ALL nouns ONCE and route to correct type indexes // Load ALL nouns ONCE and route to correct type indexes
// This is O(N) instead of O(42*N) from the previous parallel approach // This is O(N) instead of O(42*N) from the previous parallel approach
let offset = 0 // v5.7.11: Use offset-based pagination instead of cursor (bug fix for infinite loop) let offset = 0 // Use offset-based pagination instead of cursor (bug fix for infinite loop)
let hasMore = true let hasMore = true
let totalLoaded = 0 let totalLoaded = 0
const loadedByType = new Map<NounType, number>() const loadedByType = new Map<NounType, number>()
@ -515,13 +515,13 @@ export class TypeAwareHNSWIndex {
totalCount?: number totalCount?: number
} = await (this.storage as any).getNounsWithPagination({ } = await (this.storage as any).getNounsWithPagination({
limit: batchSize, limit: batchSize,
offset // v5.7.11: Pass offset for proper pagination (previously passed cursor which was ignored) offset // Pass offset for proper pagination (previously passed cursor which was ignored)
}) })
// Route each noun to its type index // Route each noun to its type index
for (const nounData of result.items) { for (const nounData of result.items) {
try { try {
// v7.3.0: Use 'type' property from HNSWNounWithMetadata (not 'nounType') // Use 'type' property from HNSWNounWithMetadata (not 'nounType')
// Previously accessed wrong property, causing N+1 metadata fetches // Previously accessed wrong property, causing N+1 metadata fetches
// getNounsWithPagination already includes type in response // getNounsWithPagination already includes type in response
const nounType = nounData.type const nounType = nounData.type
@ -578,7 +578,7 @@ export class TypeAwareHNSWIndex {
} }
hasMore = result.hasMore hasMore = result.hasMore
offset += batchSize // v5.7.11: Increment offset for next page offset += batchSize // Increment offset for next page
// Progress logging // Progress logging
if (totalLoaded % 1000 === 0) { if (totalLoaded % 1000 === 0) {
@ -593,12 +593,26 @@ export class TypeAwareHNSWIndex {
let entryPointId: string | null = null let entryPointId: string | null = null
for (const [id, noun] of (index as any).nouns.entries()) { for (const [id, noun] of (index as any).nouns.entries()) {
if (noun.level > maxLevel) { if (entryPointId === null || noun.level > maxLevel) {
maxLevel = noun.level maxLevel = noun.level
entryPointId = id entryPointId = id
} }
} }
// Recovery: if still null after loop but nouns exist
if ((index as any).nouns.size > 0 && !entryPointId) {
const { id: recoveredId, level: recoveredLevel } = (index as any).recoverEntryPointO1()
entryPointId = recoveredId
maxLevel = recoveredLevel
}
// Validation: if entry point doesn't exist in loaded nouns
if (entryPointId && !(index as any).nouns.has(entryPointId)) {
const { id: recoveredId, level: recoveredLevel } = (index as any).recoverEntryPointO1()
entryPointId = recoveredId
maxLevel = recoveredLevel
}
;(index as any).entryPointId = entryPointId ;(index as any).entryPointId = entryPointId
;(index as any).maxLevel = maxLevel ;(index as any).maxLevel = maxLevel

View file

@ -111,7 +111,7 @@ export class FormatDetector {
return 'docx' return 'docx'
} }
// Images (v5.2.0: ImageHandler support) // Images (ImageHandler support)
if (mimeType.startsWith('image/')) { if (mimeType.startsWith('image/')) {
return 'image' return 'image'
} }
@ -134,7 +134,7 @@ export class FormatDetector {
} }
} }
// YAML detection (v4.2.0) // YAML detection
if (this.looksLikeYAML(trimmed)) { if (this.looksLikeYAML(trimmed)) {
return { return {
format: 'yaml', format: 'yaml',
@ -206,7 +206,7 @@ export class FormatDetector {
} }
} }
// Image formats (v5.2.0) // Image formats
// JPEG: FF D8 FF // JPEG: FF D8 FF
if (buffer[0] === 0xFF && buffer[1] === 0xD8 && buffer[2] === 0xFF) { if (buffer[0] === 0xFF && buffer[1] === 0xD8 && buffer[2] === 0xFF) {
return { return {
@ -384,7 +384,7 @@ export class FormatDetector {
/** /**
* Check if content looks like YAML * Check if content looks like YAML
* v4.2.0: Added YAML detection * Added YAML detection
*/ */
private looksLikeYAML(content: string): boolean { private looksLikeYAML(content: string): boolean {
const lines = content.split('\n').filter(l => l.trim()).slice(0, 20) const lines = content.split('\n').filter(l => l.trim()).slice(0, 20)

View file

@ -37,10 +37,10 @@ export interface ImportSource {
/** Optional filename hint */ /** Optional filename hint */
filename?: string filename?: string
/** HTTP headers for URL imports (v4.2.0) */ /** HTTP headers for URL imports */
headers?: Record<string, string> headers?: Record<string, string>
/** Basic authentication for URL imports (v4.2.0) */ /** Basic authentication for URL imports */
auth?: { auth?: {
username: string username: string
password: string password: string
@ -93,7 +93,7 @@ export interface ValidImportOptions {
/** Create relationships in knowledge graph */ /** Create relationships in knowledge graph */
createRelationships?: boolean createRelationships?: boolean
/** Create provenance relationships (document → entity) [v4.9.0] */ /** Create provenance relationships (document → entity) */
createProvenanceLinks?: boolean createProvenanceLinks?: boolean
/** Preserve source file in VFS */ /** Preserve source file in VFS */
@ -144,7 +144,7 @@ export interface ValidImportOptions {
customMetadata?: Record<string, any> customMetadata?: Record<string, any>
/** /**
* Progress callback for tracking import progress (v4.2.0+) * Progress callback for tracking import progress
* *
* **Streaming Architecture** (always enabled): * **Streaming Architecture** (always enabled):
* - Indexes are flushed periodically during import (adaptive intervals) * - Indexes are flushed periodically during import (adaptive intervals)
@ -227,21 +227,21 @@ export type ImportOptions = ValidImportOptions & DeprecatedImportOptions
export interface ImportProgress { export interface ImportProgress {
stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'relationships' | 'complete' stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'relationships' | 'complete'
/** Phase of import - extraction or relationship building (v3.49.0) */ /** Phase of import - extraction or relationship building */
phase?: 'extraction' | 'relationships' phase?: 'extraction' | 'relationships'
message: string message: string
processed?: number processed?: number
/** Alias for processed, used in relationship phase (v3.49.0) */ /** Alias for processed, used in relationship phase */
current?: number current?: number
total?: number total?: number
entities?: number entities?: number
relationships?: number relationships?: number
/** Rows per second (v3.38.0) */ /** Rows per second */
throughput?: number throughput?: number
/** Estimated time remaining in ms (v3.38.0) */ /** Estimated time remaining in ms */
eta?: number eta?: number
/** /**
* Whether data is queryable at this point (v4.2.0+) * Whether data is queryable at this point
* *
* When true, indexes have been flushed and queries will return up-to-date results. * When true, indexes have been flushed and queries will return up-to-date results.
* When false, data exists in storage but indexes may not be current (queries may be slower/incomplete). * When false, data exists in storage but indexes may not be current (queries may be slower/incomplete).
@ -357,7 +357,7 @@ export class ImportCoordinator {
/** /**
* Import from any source with auto-detection * Import from any source with auto-detection
* v4.2.0: Now supports URL imports with authentication * Now supports URL imports with authentication
*/ */
async import( async import(
source: Buffer | string | object | ImportSource, source: Buffer | string | object | ImportSource,
@ -365,10 +365,10 @@ export class ImportCoordinator {
): Promise<ImportResult> { ): Promise<ImportResult> {
const startTime = Date.now() const startTime = Date.now()
// Validate options (v4.0.0+: Reject deprecated v3.x options) // Validate options (Reject deprecated options)
this.validateOptions(options) this.validateOptions(options)
// Normalize source (v4.2.0: handles URL fetching) // Normalize source (handles URL fetching)
const normalizedSource = await this.normalizeSource(source, options.format) const normalizedSource = await this.normalizeSource(source, options.format)
// Report detection stage // Report detection stage
@ -387,10 +387,10 @@ export class ImportCoordinator {
} }
// Set defaults early (needed for tracking context) // Set defaults early (needed for tracking context)
// CRITICAL FIX (v4.3.2): Spread options FIRST, then apply defaults // CRITICAL FIX: Spread options FIRST, then apply defaults
// Previously: ...options at the end overwrote normalized defaults with undefined // Previously: ...options at the end overwrote normalized defaults with undefined
// Now: Defaults properly override undefined values // Now: Defaults properly override undefined values
// v4.4.0: Enable AI features by default for smarter imports // Enable AI features by default for smarter imports
const opts = { const opts = {
...options, // Spread first to get all options ...options, // Spread first to get all options
vfsPath: options.vfsPath || `/imports/${Date.now()}`, vfsPath: options.vfsPath || `/imports/${Date.now()}`,
@ -399,13 +399,13 @@ export class ImportCoordinator {
createRelationships: options.createRelationships !== false, createRelationships: options.createRelationships !== false,
preserveSource: options.preserveSource !== false, preserveSource: options.preserveSource !== false,
enableDeduplication: options.enableDeduplication !== false, enableDeduplication: options.enableDeduplication !== false,
enableNeuralExtraction: options.enableNeuralExtraction !== false, // v4.4.0: Default true enableNeuralExtraction: options.enableNeuralExtraction !== false, // Default true
enableRelationshipInference: options.enableRelationshipInference !== false, // v4.4.0: Default true enableRelationshipInference: options.enableRelationshipInference !== false, // Default true
enableConceptExtraction: options.enableConceptExtraction !== false, // Already defaults to true enableConceptExtraction: options.enableConceptExtraction !== false, // Already defaults to true
deduplicationThreshold: options.deduplicationThreshold || 0.85 deduplicationThreshold: options.deduplicationThreshold || 0.85
} }
// Generate tracking context (v4.10.0+: Unified import/project tracking) // Generate tracking context (Unified import/project tracking)
const importId = options.importId || uuidv4() const importId = options.importId || uuidv4()
const projectId = options.projectId || this.deriveProjectId(opts.vfsPath) const projectId = options.projectId || this.deriveProjectId(opts.vfsPath)
const trackingContext: TrackingContext = { const trackingContext: TrackingContext = {
@ -441,13 +441,13 @@ export class ImportCoordinator {
groupBy: opts.groupBy, groupBy: opts.groupBy,
customGrouping: opts.customGrouping, customGrouping: opts.customGrouping,
preserveSource: opts.preserveSource, preserveSource: opts.preserveSource,
// v5.1.2: Fix sourceBuffer for file paths - type is 'path' not 'buffer' from normalizeSource() // Fix sourceBuffer for file paths - type is 'path' not 'buffer' from normalizeSource()
sourceBuffer: Buffer.isBuffer(normalizedSource.data) ? normalizedSource.data as Buffer : undefined, sourceBuffer: Buffer.isBuffer(normalizedSource.data) ? normalizedSource.data as Buffer : undefined,
sourceFilename: normalizedSource.filename || `import.${detection.format}`, sourceFilename: normalizedSource.filename || `import.${detection.format}`,
createRelationshipFile: true, createRelationshipFile: true,
createMetadataFile: true, createMetadataFile: true,
trackingContext, // v4.10.0: Pass tracking metadata to VFS trackingContext, // Pass tracking metadata to VFS
// v4.11.1: Pass progress callback for VFS creation updates // Pass progress callback for VFS creation updates
onProgress: (vfsProgress) => { onProgress: (vfsProgress) => {
options.onProgress?.({ options.onProgress?.({
stage: 'storing-vfs', stage: 'storing-vfs',
@ -473,7 +473,7 @@ export class ImportCoordinator {
sourceFilename: normalizedSource.filename || `import.${detection.format}`, sourceFilename: normalizedSource.filename || `import.${detection.format}`,
format: detection.format format: detection.format
}, },
trackingContext // v4.10.0: Pass tracking metadata to graph creation trackingContext // Pass tracking metadata to graph creation
) )
// Report complete // Report complete
@ -520,7 +520,7 @@ export class ImportCoordinator {
) )
} }
// CRITICAL FIX (v3.43.2): Auto-flush all indexes before returning // CRITICAL FIX: Auto-flush all indexes before returning
// Ensures imported data survives server restarts // Ensures imported data survives server restarts
// Bug #5: Import data was only in memory, lost on restart // Bug #5: Import data was only in memory, lost on restart
options.onProgress?.({ options.onProgress?.({
@ -535,7 +535,7 @@ export class ImportCoordinator {
/** /**
* Normalize source to ImportSource * Normalize source to ImportSource
* v4.2.0: Now async to support URL fetching * Now async to support URL fetching
*/ */
private async normalizeSource( private async normalizeSource(
source: Buffer | string | object | ImportSource, source: Buffer | string | object | ImportSource,
@ -616,7 +616,7 @@ export class ImportCoordinator {
/** /**
* Fetch content from URL * Fetch content from URL
* v4.2.0: Supports authentication and custom headers * Supports authentication and custom headers
*/ */
private async fetchUrl(source: ImportSource): Promise<ImportSource> { private async fetchUrl(source: ImportSource): Promise<ImportSource> {
const url = typeof source.data === 'string' ? source.data : String(source.data) const url = typeof source.data === 'string' ? source.data : String(source.data)
@ -722,7 +722,7 @@ export class ImportCoordinator {
format: SupportedFormat, format: SupportedFormat,
options: ImportOptions options: ImportOptions
): Promise<any> { ): Promise<any> {
// v5.2.0: Check if IntelligentImportAugmentation already extracted data // Check if IntelligentImportAugmentation already extracted data
if ((options as any)._intelligentImport && (options as any)._extractedData) { if ((options as any)._intelligentImport && (options as any)._extractedData) {
const extractedData = (options as any)._extractedData const extractedData = (options as any)._extractedData
// Convert extracted data to ExtractedRow format // Convert extracted data to ExtractedRow format
@ -760,7 +760,7 @@ export class ImportCoordinator {
enableConceptExtraction: options.enableConceptExtraction !== false, enableConceptExtraction: options.enableConceptExtraction !== false,
confidenceThreshold: options.confidenceThreshold || 0.6, confidenceThreshold: options.confidenceThreshold || 0.6,
onProgress: (stats: any) => { onProgress: (stats: any) => {
// Enhanced progress reporting (v3.38.0) with throughput and ETA // Enhanced progress reporting with throughput and ETA
const message = stats.throughput const message = stats.throughput
? `Extracting entities from ${format} (${stats.throughput} rows/sec, ETA: ${Math.round(stats.eta / 1000)}s)...` ? `Extracting entities from ${format} (${stats.throughput} rows/sec, ETA: ${Math.round(stats.eta / 1000)}s)...`
: `Extracting entities from ${format}...` : `Extracting entities from ${format}...`
@ -825,7 +825,7 @@ export class ImportCoordinator {
return await this.docxImporter.extract(docxBuffer, extractOptions) return await this.docxImporter.extract(docxBuffer, extractOptions)
case 'image': case 'image':
// v5.2.0: Images are handled by IntelligentImportAugmentation // Images are handled by IntelligentImportAugmentation
// If we reach here, augmentation didn't process it - return minimal result // If we reach here, augmentation didn't process it - return minimal result
const imageName = source.filename || 'image' const imageName = source.filename || 'image'
const imageId = `image-${Date.now()}` const imageId = `image-${Date.now()}`
@ -867,7 +867,7 @@ export class ImportCoordinator {
/** /**
* Create entities and relationships in knowledge graph * Create entities and relationships in knowledge graph
* v4.9.0: Added sourceInfo parameter for document entity creation * Added sourceInfo parameter for document entity creation
*/ */
private async createGraphEntities( private async createGraphEntities(
extractionResult: any, extractionResult: any,
@ -877,7 +877,7 @@ export class ImportCoordinator {
sourceFilename: string sourceFilename: string
format: string format: string
}, },
trackingContext?: TrackingContext // v4.10.0: Import/project tracking trackingContext?: TrackingContext // Import/project tracking
): Promise<{ ): Promise<{
entities: Array<{ id: string; name: string; type: NounType; vfsPath?: string; metadata?: Record<string, any> }> entities: Array<{ id: string; name: string; type: NounType; vfsPath?: string; metadata?: Record<string, any> }>
relationships: Array<{ id: string; from: string; to: string; type: VerbType }> relationships: Array<{ id: string; from: string; to: string; type: VerbType }>
@ -891,7 +891,7 @@ export class ImportCoordinator {
let mergedCount = 0 let mergedCount = 0
let newCount = 0 let newCount = 0
// CRITICAL FIX (v4.3.2): Default to true when undefined // CRITICAL FIX: Default to true when undefined
// Previously: if (!options.createEntities) treated undefined as false // Previously: if (!options.createEntities) treated undefined as false
// Now: Only skip when explicitly set to false // Now: Only skip when explicitly set to false
if (options.createEntities === false) { if (options.createEntities === false) {
@ -908,7 +908,7 @@ export class ImportCoordinator {
// Extract rows/sections/entities from result (unified across formats) // Extract rows/sections/entities from result (unified across formats)
const rows = extractionResult.rows || extractionResult.sections || extractionResult.entities || [] const rows = extractionResult.rows || extractionResult.sections || extractionResult.entities || []
// Progressive flush interval - adjusts based on current count (v4.2.0+) // Progressive flush interval - adjusts based on current count
// Starts at 100, increases to 1000 at 1K entities, then 5000 at 10K // Starts at 100, increases to 1000 at 1K entities, then 5000 at 10K
// This works for both known totals (files) and unknown totals (streaming APIs) // This works for both known totals (files) and unknown totals (streaming APIs)
let currentFlushInterval = 100 // Start with frequent updates for better UX let currentFlushInterval = 100 // Start with frequent updates for better UX
@ -938,7 +938,7 @@ export class ImportCoordinator {
} }
// ============================================ // ============================================
// v4.9.0: Create document entity for import source // Create document entity for import source
// ============================================ // ============================================
let documentEntityId: string | null = null let documentEntityId: string | null = null
let provenanceCount = 0 let provenanceCount = 0
@ -957,7 +957,7 @@ export class ImportCoordinator {
vfsPath: vfsResult.rootPath, vfsPath: vfsResult.rootPath,
totalRows: rows.length, totalRows: rows.length,
byType: this.countByType(rows), byType: this.countByType(rows),
// v4.10.0: Import tracking metadata // Import tracking metadata
...(trackingContext && { ...(trackingContext && {
importIds: [trackingContext.importId], importIds: [trackingContext.importId],
projectId: trackingContext.projectId, projectId: trackingContext.projectId,
@ -973,7 +973,7 @@ export class ImportCoordinator {
} }
// ============================================ // ============================================
// v4.11.0: Batch entity creation using addMany() // Batch entity creation using addMany()
// Replaces entity-by-entity loop for 10-100x performance improvement on cloud storage // Replaces entity-by-entity loop for 10-100x performance improvement on cloud storage
// ============================================ // ============================================
@ -1038,7 +1038,7 @@ export class ImportCoordinator {
name: entity.name, name: entity.name,
type: entity.type, type: entity.type,
vfsPath: vfsFile?.path, vfsPath: vfsFile?.path,
metadata: entity.metadata // v5.2.0: Include metadata in return (for ImageHandler, etc) metadata: entity.metadata // Include metadata in return (for ImageHandler, etc)
}) })
newCount++ newCount++
} }
@ -1090,7 +1090,7 @@ export class ImportCoordinator {
const importSource = vfsResult.rootPath const importSource = vfsResult.rootPath
let entityId: string let entityId: string
// v5.7.0: No deduplication during import (12-24x speedup) // No deduplication during import (12-24x speedup)
// Background deduplication runs 5 minutes after import completes // Background deduplication runs 5 minutes after import completes
entityId = await this.brain.add({ entityId = await this.brain.add({
data: entity.description || entity.name, data: entity.description || entity.name,
@ -1101,7 +1101,7 @@ export class ImportCoordinator {
confidence: entity.confidence, confidence: entity.confidence,
vfsPath: vfsFile?.path, vfsPath: vfsFile?.path,
importedFrom: 'import-coordinator', importedFrom: 'import-coordinator',
// v4.10.0: Import tracking metadata // Import tracking metadata
...(trackingContext && { ...(trackingContext && {
importId: trackingContext.importId, // Used for background dedup importId: trackingContext.importId, // Used for background dedup
importIds: [trackingContext.importId], importIds: [trackingContext.importId],
@ -1126,11 +1126,11 @@ export class ImportCoordinator {
name: entity.name, name: entity.name,
type: entity.type, type: entity.type,
vfsPath: vfsFile?.path, vfsPath: vfsFile?.path,
metadata: entity.metadata // v5.2.0: Include metadata in return (for ImageHandler, etc) metadata: entity.metadata // Include metadata in return (for ImageHandler, etc)
}) })
// ============================================ // ============================================
// v4.9.0: Create provenance relationship (document → entity) // Create provenance relationship (document → entity)
// ============================================ // ============================================
if (documentEntityId && options.createProvenanceLinks !== false) { if (documentEntityId && options.createProvenanceLinks !== false) {
await this.brain.relate({ await this.brain.relate({
@ -1144,7 +1144,7 @@ export class ImportCoordinator {
rowNumber: row.rowNumber, rowNumber: row.rowNumber,
extractedAt: Date.now(), extractedAt: Date.now(),
format: sourceInfo?.format, format: sourceInfo?.format,
// v4.10.0: Import tracking metadata // Import tracking metadata
...(trackingContext && { ...(trackingContext && {
importIds: [trackingContext.importId], importIds: [trackingContext.importId],
projectId: trackingContext.projectId, projectId: trackingContext.projectId,
@ -1161,7 +1161,7 @@ export class ImportCoordinator {
if (options.createRelationships && row.relationships) { if (options.createRelationships && row.relationships) {
for (const rel of row.relationships) { for (const rel of row.relationships) {
try { try {
// CRITICAL FIX (v3.43.2): Prevent infinite placeholder creation loop // CRITICAL FIX: Prevent infinite placeholder creation loop
// Find or create target entity using EXACT matching only // Find or create target entity using EXACT matching only
let targetEntityId: string | undefined let targetEntityId: string | undefined
@ -1195,7 +1195,7 @@ export class ImportCoordinator {
name: rel.to, name: rel.to,
placeholder: true, placeholder: true,
inferredFrom: entity.name, inferredFrom: entity.name,
// v4.10.0: Import tracking metadata // Import tracking metadata
...(trackingContext && { ...(trackingContext && {
importIds: [trackingContext.importId], importIds: [trackingContext.importId],
projectId: trackingContext.projectId, projectId: trackingContext.projectId,
@ -1221,11 +1221,11 @@ export class ImportCoordinator {
from: entityId, from: entityId,
to: targetEntityId, to: targetEntityId,
type: rel.type, type: rel.type,
confidence: rel.confidence, // v4.2.0: Top-level field confidence: rel.confidence, // Top-level field
weight: rel.weight || 1.0, // v4.2.0: Top-level field weight: rel.weight || 1.0, // Top-level field
metadata: { metadata: {
evidence: rel.evidence, evidence: rel.evidence,
// v4.10.0: Import tracking metadata (will be merged in batch creation) // Import tracking metadata (will be merged in batch creation)
...(trackingContext && { ...(trackingContext && {
importIds: [trackingContext.importId], importIds: [trackingContext.importId],
projectId: trackingContext.projectId, projectId: trackingContext.projectId,
@ -1242,7 +1242,7 @@ export class ImportCoordinator {
} }
} }
// Streaming import: Progressive flush with dynamic interval adjustment (v4.2.0+) // Streaming import: Progressive flush with dynamic interval adjustment
entitiesSinceFlush++ entitiesSinceFlush++
if (entitiesSinceFlush >= currentFlushInterval) { if (entitiesSinceFlush >= currentFlushInterval) {
@ -1307,7 +1307,7 @@ export class ImportCoordinator {
} }
// Batch create all relationships using brain.relateMany() for performance // Batch create all relationships using brain.relateMany() for performance
// v4.9.0: Enhanced with type-based inference and semantic metadata // Enhanced with type-based inference and semantic metadata
if (options.createRelationships && relationships.length > 0) { if (options.createRelationships && relationships.length > 0) {
try { try {
const relationshipParams = relationships.map(rel => { const relationshipParams = relationships.map(rel => {
@ -1331,7 +1331,7 @@ export class ImportCoordinator {
type: verbType, // Enhanced type type: verbType, // Enhanced type
metadata: { metadata: {
...((rel as any).metadata || {}), ...((rel as any).metadata || {}),
relationshipType: 'semantic', // v4.9.0: Distinguish from VFS/provenance relationshipType: 'semantic', // Distinguish from VFS/provenance
inferredType: verbType !== rel.type, // Track if type was enhanced inferredType: verbType !== rel.type, // Track if type was enhanced
originalType: rel.type originalType: rel.type
} }
@ -1369,7 +1369,7 @@ export class ImportCoordinator {
} }
} }
// v5.7.0: Schedule background deduplication (debounced 5 minutes) // Schedule background deduplication (debounced 5 minutes)
if (trackingContext && trackingContext.importId) { if (trackingContext && trackingContext.importId) {
this.backgroundDedup.scheduleDedup(trackingContext.importId) this.backgroundDedup.scheduleDedup(trackingContext.importId)
} }
@ -1457,7 +1457,7 @@ export class ImportCoordinator {
} }
} }
// YAML: entities -> rows (v4.2.0) // YAML: entities -> rows
if (format === 'yaml') { if (format === 'yaml') {
const rows = result.entities.map((entity: any) => ({ const rows = result.entities.map((entity: any) => ({
entity, entity,
@ -1477,7 +1477,7 @@ export class ImportCoordinator {
} }
} }
// DOCX: entities -> rows (v4.2.0) // DOCX: entities -> rows
if (format === 'docx') { if (format === 'docx') {
const rows = result.entities.map((entity: any) => ({ const rows = result.entities.map((entity: any) => ({
entity, entity,
@ -1502,7 +1502,7 @@ export class ImportCoordinator {
} }
/** /**
* Validate options and reject deprecated v3.x options (v4.0.0+) * Validate options and reject deprecated v3.x options
* Throws clear errors with migration guidance * Throws clear errors with migration guidance
*/ */
private validateOptions(options: any): void { private validateOptions(options: any): void {
@ -1657,7 +1657,7 @@ ${optionDetails}
} }
/** /**
* Get progressive flush interval based on CURRENT entity count (v4.2.0+) * Get progressive flush interval based on CURRENT entity count
* *
* Unlike adaptive intervals (which require knowing total count upfront), * Unlike adaptive intervals (which require knowing total count upfront),
* progressive intervals adjust dynamically as import proceeds. * progressive intervals adjust dynamically as import proceeds.
@ -1699,7 +1699,7 @@ ${optionDetails}
/** /**
* Infer relationship type based on entity types and context * Infer relationship type based on entity types and context
* v4.9.0: Semantic relationship enhancement * Semantic relationship enhancement
* *
* @param sourceType - Type of source entity * @param sourceType - Type of source entity
* @param targetType - Type of target entity * @param targetType - Type of target entity
@ -1769,7 +1769,7 @@ ${optionDetails}
/** /**
* Count entities by type for document metadata * Count entities by type for document metadata
* v4.9.0: Used for document entity statistics * Used for document entity statistics
* *
* @param rows - Extracted rows from import * @param rows - Extracted rows from import
* @returns Record of entity type counts * @returns Record of entity type counts

View file

@ -42,17 +42,17 @@ export interface SmartCSVOptions extends FormatHandlerOptions {
csvDelimiter?: string csvDelimiter?: string
csvHeaders?: boolean csvHeaders?: boolean
/** Progress callback (v3.39.0: Enhanced with performance metrics) */ /** Progress callback (Enhanced with performance metrics) */
onProgress?: (stats: { onProgress?: (stats: {
processed: number processed: number
total: number total: number
entities: number entities: number
relationships: number relationships: number
/** Rows per second (v3.39.0) */ /** Rows per second */
throughput?: number throughput?: number
/** Estimated time remaining in ms (v3.39.0) */ /** Estimated time remaining in ms */
eta?: number eta?: number
/** Current phase (v3.39.0) */ /** Current phase */
phase?: string phase?: string
}) => void }) => void
} }
@ -169,7 +169,7 @@ export class SmartCSVImporter {
} }
// Parse CSV using existing handler // Parse CSV using existing handler
// v4.5.0: Pass progress hooks to handler for file parsing progress // Pass progress hooks to handler for file parsing progress
const processedData = await this.csvHandler.process(buffer, { const processedData = await this.csvHandler.process(buffer, {
...options, ...options,
csvDelimiter: opts.csvDelimiter, csvDelimiter: opts.csvDelimiter,
@ -217,7 +217,7 @@ export class SmartCSVImporter {
// Detect column names // Detect column names
const columns = this.detectColumns(rows[0], opts) const columns = this.detectColumns(rows[0], opts)
// Process each row with BATCHED PARALLEL PROCESSING (v3.39.0) // Process each row with BATCHED PARALLEL PROCESSING
const extractedRows: ExtractedRow[] = [] const extractedRows: ExtractedRow[] = []
const entityMap = new Map<string, string>() const entityMap = new Map<string, string>()
const stats = { const stats = {
@ -375,7 +375,7 @@ export class SmartCSVImporter {
total: rows.length, total: rows.length,
entities: extractedRows.reduce((sum, row) => sum + 1 + row.relatedEntities.length, 0), entities: extractedRows.reduce((sum, row) => sum + 1 + row.relatedEntities.length, 0),
relationships: extractedRows.reduce((sum, row) => sum + row.relationships.length, 0), relationships: extractedRows.reduce((sum, row) => sum + row.relationships.length, 0),
// Additional performance metrics (v3.39.0) // Additional performance metrics
throughput: Math.round(rowsPerSecond * 10) / 10, throughput: Math.round(rowsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining), eta: Math.round(estimatedTimeRemaining),
phase: 'extracting' phase: 'extracting'

View file

@ -9,7 +9,7 @@
* - NaturalLanguageProcessor for relationship inference * - NaturalLanguageProcessor for relationship inference
* - Hierarchical relationship creation based on heading hierarchy * - Hierarchical relationship creation based on heading hierarchy
* *
* v4.2.0: New format handler * New format handler
* NO MOCKS - Production-ready implementation * NO MOCKS - Production-ready implementation
*/ */
@ -187,7 +187,7 @@ export class SmartDOCXImporter {
await this.init() await this.init()
} }
// v4.5.0: Report parsing start // Report parsing start
options.onProgress?.({ options.onProgress?.({
processed: 0, processed: 0,
entities: 0, entities: 0,
@ -200,7 +200,7 @@ export class SmartDOCXImporter {
// Extract HTML for structure analysis (headings, tables) // Extract HTML for structure analysis (headings, tables)
const htmlResult = await mammoth.convertToHtml({ buffer }) const htmlResult = await mammoth.convertToHtml({ buffer })
// v4.5.0: Report parsing complete // Report parsing complete
options.onProgress?.({ options.onProgress?.({
processed: 0, processed: 0,
entities: 0, entities: 0,

View file

@ -36,17 +36,17 @@ export interface SmartExcelOptions extends FormatHandlerOptions {
typeColumn?: string // e.g., "Type", "Category" typeColumn?: string // e.g., "Type", "Category"
relatedColumn?: string // e.g., "Related Terms", "See Also" relatedColumn?: string // e.g., "Related Terms", "See Also"
/** Progress callback (v3.38.0: Enhanced with performance metrics) */ /** Progress callback (Enhanced with performance metrics) */
onProgress?: (stats: { onProgress?: (stats: {
processed: number processed: number
total: number total: number
entities: number entities: number
relationships: number relationships: number
/** Rows per second (v3.38.0) */ /** Rows per second */
throughput?: number throughput?: number
/** Estimated time remaining in ms (v3.38.0) */ /** Estimated time remaining in ms */
eta?: number eta?: number
/** Current phase (v3.38.0) */ /** Current phase */
phase?: string phase?: string
}) => void }) => void
} }
@ -111,7 +111,7 @@ export interface SmartExcelResult {
} }
} }
/** Sheet-specific data for VFS extraction (v4.2.0) */ /** Sheet-specific data for VFS extraction */
sheets?: Array<{ sheets?: Array<{
name: string name: string
rows: ExtractedRow[] rows: ExtractedRow[]
@ -161,7 +161,7 @@ export class SmartExcelImporter {
const opts = { const opts = {
enableNeuralExtraction: true, enableNeuralExtraction: true,
enableRelationshipInference: true, enableRelationshipInference: true,
// CONCEPT EXTRACTION PRODUCTION-READY (v3.33.0+): // CONCEPT EXTRACTION PRODUCTION-READY:
// Type embeddings are now pre-computed at build time - zero runtime cost! // Type embeddings are now pre-computed at build time - zero runtime cost!
// All 42 noun types + 127 verb types instantly available // All 42 noun types + 127 verb types instantly available
// //
@ -184,7 +184,7 @@ export class SmartExcelImporter {
} }
// Parse Excel using existing handler // Parse Excel using existing handler
// v4.5.0: Pass progress hooks to handler for file parsing progress // Pass progress hooks to handler for file parsing progress
const processedData = await this.excelHandler.process(buffer, { const processedData = await this.excelHandler.process(buffer, {
...options, ...options,
totalBytes: buffer.length, totalBytes: buffer.length,
@ -227,7 +227,7 @@ export class SmartExcelImporter {
return this.emptyResult(startTime) return this.emptyResult(startTime)
} }
// CRITICAL FIX (v4.8.6): Detect columns per-sheet, not globally // CRITICAL FIX: Detect columns per-sheet, not globally
// Different sheets may have different column structures (Term vs Name, etc.) // Different sheets may have different column structures (Term vs Name, etc.)
// Group rows by sheet and detect columns for each sheet separately // Group rows by sheet and detect columns for each sheet separately
const rowsBySheet = new Map<string, typeof rows>() const rowsBySheet = new Map<string, typeof rows>()
@ -247,7 +247,7 @@ export class SmartExcelImporter {
} }
} }
// Process each row with BATCHED PARALLEL PROCESSING (v3.38.0) // Process each row with BATCHED PARALLEL PROCESSING
const extractedRows: ExtractedRow[] = [] const extractedRows: ExtractedRow[] = []
const entityMap = new Map<string, string>() const entityMap = new Map<string, string>()
const stats = { const stats = {
@ -269,7 +269,7 @@ export class SmartExcelImporter {
chunk.map(async (row, chunkIndex) => { chunk.map(async (row, chunkIndex) => {
const i = chunkStart + chunkIndex const i = chunkStart + chunkIndex
// CRITICAL FIX (v4.8.6): Use sheet-specific column mapping // CRITICAL FIX: Use sheet-specific column mapping
const sheet = row._sheet || 'default' const sheet = row._sheet || 'default'
const columns = columnsBySheet.get(sheet) || this.detectColumns(row, opts) const columns = columnsBySheet.get(sheet) || this.detectColumns(row, opts)
@ -353,7 +353,7 @@ export class SmartExcelImporter {
} }
// ============================================ // ============================================
// v4.9.0: Enhanced column-based relationship detection // Enhanced column-based relationship detection
// ============================================ // ============================================
// Parse explicit "Related Terms" column // Parse explicit "Related Terms" column
if (relatedTerms) { if (relatedTerms) {
@ -379,7 +379,7 @@ export class SmartExcelImporter {
} }
} }
// v4.9.0: Check for additional relationship-indicating columns // Check for additional relationship-indicating columns
// Expanded patterns for various relationship types // Expanded patterns for various relationship types
const relationshipColumnPatterns = [ const relationshipColumnPatterns = [
{ pattern: /^(location|home|lives in|resides|dwelling|place)$/i, defaultType: VerbType.LocatedAt }, { pattern: /^(location|home|lives in|resides|dwelling|place)$/i, defaultType: VerbType.LocatedAt },
@ -485,14 +485,14 @@ export class SmartExcelImporter {
total: rows.length, total: rows.length,
entities: extractedRows.reduce((sum, row) => sum + 1 + row.relatedEntities.length, 0), entities: extractedRows.reduce((sum, row) => sum + 1 + row.relatedEntities.length, 0),
relationships: extractedRows.reduce((sum, row) => sum + row.relationships.length, 0), relationships: extractedRows.reduce((sum, row) => sum + row.relationships.length, 0),
// Additional performance metrics (v3.38.0) // Additional performance metrics
throughput: Math.round(rowsPerSecond * 10) / 10, throughput: Math.round(rowsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining), eta: Math.round(estimatedTimeRemaining),
phase: 'extracting' phase: 'extracting'
} as any) } as any)
} }
// Group rows by sheet for VFS extraction (v4.2.0) // Group rows by sheet for VFS extraction
const sheetGroups = new Map<string, ExtractedRow[]>() const sheetGroups = new Map<string, ExtractedRow[]>()
extractedRows.forEach((extractedRow, index) => { extractedRows.forEach((extractedRow, index) => {
const originalRow = rows[index] const originalRow = rows[index]

View file

@ -167,7 +167,7 @@ export class SmartJSONImporter {
...options ...options
} }
// v4.5.0: Report parsing start // Report parsing start
opts.onProgress({ opts.onProgress({
processed: 0, processed: 0,
entities: 0, entities: 0,
@ -186,7 +186,7 @@ export class SmartJSONImporter {
jsonData = data jsonData = data
} }
// v4.5.0: Report parsing complete, starting traversal // Report parsing complete, starting traversal
opts.onProgress({ opts.onProgress({
processed: 0, processed: 0,
entities: 0, entities: 0,
@ -228,7 +228,7 @@ export class SmartJSONImporter {
} }
) )
// v4.5.0: Report completion // Report completion
opts.onProgress({ opts.onProgress({
processed: nodesProcessed, processed: nodesProcessed,
entities: entities.length, entities: entities.length,

View file

@ -174,7 +174,7 @@ export class SmartMarkdownImporter {
...options ...options
} }
// v4.5.0: Report parsing start // Report parsing start
opts.onProgress({ opts.onProgress({
processed: 0, processed: 0,
total: 0, total: 0,
@ -185,7 +185,7 @@ export class SmartMarkdownImporter {
// Parse markdown into sections // Parse markdown into sections
const parsedSections = this.parseMarkdown(markdown, opts) const parsedSections = this.parseMarkdown(markdown, opts)
// v4.5.0: Report parsing complete // Report parsing complete
opts.onProgress({ opts.onProgress({
processed: 0, processed: 0,
total: parsedSections.length, total: parsedSections.length,
@ -218,7 +218,7 @@ export class SmartMarkdownImporter {
}) })
} }
// v4.5.0: Report completion // Report completion
const totalEntities = sections.reduce((sum, s) => sum + s.entities.length, 0) const totalEntities = sections.reduce((sum, s) => sum + s.entities.length, 0)
const totalRelationships = sections.reduce((sum, s) => sum + s.relationships.length, 0) const totalRelationships = sections.reduce((sum, s) => sum + s.relationships.length, 0)
opts.onProgress({ opts.onProgress({

View file

@ -40,17 +40,17 @@ export interface SmartPDFOptions extends FormatHandlerOptions {
/** Group by page or full document */ /** Group by page or full document */
groupBy?: 'page' | 'document' groupBy?: 'page' | 'document'
/** Progress callback (v3.39.0: Enhanced with performance metrics) */ /** Progress callback (Enhanced with performance metrics) */
onProgress?: (stats: { onProgress?: (stats: {
processed: number processed: number
total: number total: number
entities: number entities: number
relationships: number relationships: number
/** Sections per second (v3.39.0) */ /** Sections per second */
throughput?: number throughput?: number
/** Estimated time remaining in ms (v3.39.0) */ /** Estimated time remaining in ms */
eta?: number eta?: number
/** Current phase (v3.39.0) */ /** Current phase */
phase?: string phase?: string
}) => void }) => void
} }
@ -181,7 +181,7 @@ export class SmartPDFImporter {
} }
// Parse PDF using existing handler // Parse PDF using existing handler
// v4.5.0: Pass progress hooks to handler for file parsing progress // Pass progress hooks to handler for file parsing progress
const processedData = await this.pdfHandler.process(buffer, { const processedData = await this.pdfHandler.process(buffer, {
...options, ...options,
totalBytes: buffer.length, totalBytes: buffer.length,
@ -228,7 +228,7 @@ export class SmartPDFImporter {
// Group data by page or combine into single document // Group data by page or combine into single document
const grouped = this.groupData(data, opts) const grouped = this.groupData(data, opts)
// Process each group with BATCHED PARALLEL PROCESSING (v3.39.0) // Process each group with BATCHED PARALLEL PROCESSING
const sections: ExtractedSection[] = [] const sections: ExtractedSection[] = []
const entityMap = new Map<string, string>() const entityMap = new Map<string, string>()
const stats = { const stats = {
@ -270,7 +270,7 @@ export class SmartPDFImporter {
total: totalGroups, total: totalGroups,
entities: sections.reduce((sum, s) => sum + s.entities.length, 0), entities: sections.reduce((sum, s) => sum + s.entities.length, 0),
relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0), relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0),
// Additional performance metrics (v3.39.0) // Additional performance metrics
throughput: Math.round(sectionsPerSecond * 10) / 10, throughput: Math.round(sectionsPerSecond * 10) / 10,
eta: Math.round(estimatedTimeRemaining), eta: Math.round(estimatedTimeRemaining),
phase: 'extracting' phase: 'extracting'
@ -367,7 +367,7 @@ export class SmartPDFImporter {
const combinedText = texts.join('\n\n') const combinedText = texts.join('\n\n')
// Parallel extraction: entities AND concepts at the same time (v3.39.0) // Parallel extraction: entities AND concepts at the same time
const [extractedEntities, concepts] = await Promise.all([ const [extractedEntities, concepts] = await Promise.all([
// Extract entities if enabled // Extract entities if enabled
options.enableNeuralExtraction && combinedText.length > 0 options.enableNeuralExtraction && combinedText.length > 0

View file

@ -8,7 +8,7 @@
* - NaturalLanguageProcessor for relationship inference * - NaturalLanguageProcessor for relationship inference
* - Hierarchical relationship creation (parent-child, contains, etc.) * - Hierarchical relationship creation (parent-child, contains, etc.)
* *
* v4.2.0: New format handler * New format handler
* NO MOCKS - Production-ready implementation * NO MOCKS - Production-ready implementation
*/ */
@ -159,7 +159,7 @@ export class SmartYAMLImporter {
): Promise<SmartYAMLResult> { ): Promise<SmartYAMLResult> {
const startTime = Date.now() const startTime = Date.now()
// v4.5.0: Report parsing start // Report parsing start
options.onProgress?.({ options.onProgress?.({
processed: 0, processed: 0,
entities: 0, entities: 0,
@ -178,7 +178,7 @@ export class SmartYAMLImporter {
throw new Error(`Failed to parse YAML: ${error.message}`) throw new Error(`Failed to parse YAML: ${error.message}`)
} }
// v4.5.0: Report parsing complete // Report parsing complete
options.onProgress?.({ options.onProgress?.({
processed: 0, processed: 0,
entities: 0, entities: 0,

View file

@ -40,10 +40,10 @@ export interface VFSStructureOptions {
/** Create metadata file */ /** Create metadata file */
createMetadataFile?: boolean createMetadataFile?: boolean
/** Import tracking context (v4.10.0) */ /** Import tracking context */
trackingContext?: TrackingContext trackingContext?: TrackingContext
/** Progress callback (v4.11.1) - Reports VFS creation progress */ /** Progress callback - Reports VFS creation progress */
onProgress?: (progress: { onProgress?: (progress: {
stage: 'directories' | 'entities' | 'metadata' stage: 'directories' | 'entities' | 'metadata'
message: string message: string
@ -98,7 +98,7 @@ export class VFSStructureGenerator {
// Get brain's cached VFS instance (creates if doesn't exist) // Get brain's cached VFS instance (creates if doesn't exist)
this.vfs = this.brain.vfs this.vfs = this.brain.vfs
// CRITICAL FIX (v4.10.2): Always call vfs.init() explicitly // CRITICAL FIX: Always call vfs.init() explicitly
// The previous code tried to check if initialized via stat('/') but this was unreliable // The previous code tried to check if initialized via stat('/') but this was unreliable
// vfs.init() is idempotent, so calling it multiple times is safe // vfs.init() is idempotent, so calling it multiple times is safe
await this.vfs.init() await this.vfs.init()
@ -123,7 +123,7 @@ export class VFSStructureGenerator {
// Ensure VFS is initialized // Ensure VFS is initialized
await this.init() await this.init()
// v4.11.1: Calculate total operations for progress tracking // Calculate total operations for progress tracking
const groups = this.groupEntities(importResult, options) const groups = this.groupEntities(importResult, options)
const totalEntities = Array.from(groups.values()).reduce((sum, entities) => sum + entities.length, 0) const totalEntities = Array.from(groups.values()).reduce((sum, entities) => sum + entities.length, 0)
const totalOperations = const totalOperations =
@ -161,7 +161,7 @@ export class VFSStructureGenerator {
try { try {
await this.vfs.mkdir(options.rootPath, { await this.vfs.mkdir(options.rootPath, {
recursive: true, recursive: true,
metadata: trackingMetadata // v4.10.0: Add tracking metadata metadata: trackingMetadata // Add tracking metadata
}) })
result.directories.push(options.rootPath) result.directories.push(options.rootPath)
result.operations++ result.operations++
@ -179,7 +179,7 @@ export class VFSStructureGenerator {
if (options.preserveSource && options.sourceBuffer && options.sourceFilename) { if (options.preserveSource && options.sourceBuffer && options.sourceFilename) {
const sourcePath = `${options.rootPath}/_source${this.getExtension(options.sourceFilename)}` const sourcePath = `${options.rootPath}/_source${this.getExtension(options.sourceFilename)}`
await this.vfs.writeFile(sourcePath, options.sourceBuffer, { await this.vfs.writeFile(sourcePath, options.sourceBuffer, {
metadata: trackingMetadata // v4.10.0: Add tracking metadata metadata: trackingMetadata // Add tracking metadata
}) })
result.files.push({ result.files.push({
path: sourcePath, path: sourcePath,
@ -200,7 +200,7 @@ export class VFSStructureGenerator {
try { try {
await this.vfs.mkdir(groupPath, { await this.vfs.mkdir(groupPath, {
recursive: true, recursive: true,
metadata: trackingMetadata // v4.10.0: Add tracking metadata metadata: trackingMetadata // Add tracking metadata
}) })
result.directories.push(groupPath) result.directories.push(groupPath)
result.operations++ result.operations++
@ -242,7 +242,7 @@ export class VFSStructureGenerator {
await this.vfs.writeFile(entityPath, JSON.stringify(entityJson, null, 2), { await this.vfs.writeFile(entityPath, JSON.stringify(entityJson, null, 2), {
metadata: { metadata: {
...trackingMetadata, // v4.10.0: Add tracking metadata ...trackingMetadata, // Add tracking metadata
entityId: extracted.entity.id entityId: extracted.entity.id
} }
}) })
@ -253,7 +253,7 @@ export class VFSStructureGenerator {
}) })
result.operations++ result.operations++
// v4.11.1: Report progress every 10 entities (or on last entity) // Report progress every 10 entities (or on last entity)
if (entityCount % 10 === 0 || entityCount === entities.length) { if (entityCount % 10 === 0 || entityCount === entities.length) {
reportProgress('entities', `Created ${entityCount}/${entities.length} ${groupName} files`) reportProgress('entities', `Created ${entityCount}/${entities.length} ${groupName} files`)
} }
@ -280,7 +280,7 @@ export class VFSStructureGenerator {
} }
await this.vfs.writeFile(relationshipsPath, JSON.stringify(relationshipsJson, null, 2), { await this.vfs.writeFile(relationshipsPath, JSON.stringify(relationshipsJson, null, 2), {
metadata: trackingMetadata // v4.10.0: Add tracking metadata metadata: trackingMetadata // Add tracking metadata
}) })
result.files.push({ result.files.push({
path: relationshipsPath, path: relationshipsPath,
@ -322,7 +322,7 @@ export class VFSStructureGenerator {
} }
await this.vfs.writeFile(metadataPath, JSON.stringify(metadataJson, null, 2), { await this.vfs.writeFile(metadataPath, JSON.stringify(metadataJson, null, 2), {
metadata: trackingMetadata // v4.10.0: Add tracking metadata metadata: trackingMetadata // Add tracking metadata
}) })
result.files.push({ result.files.push({
path: metadataPath, path: metadataPath,
@ -332,7 +332,7 @@ export class VFSStructureGenerator {
reportProgress('metadata', 'Created metadata file') reportProgress('metadata', 'Created metadata file')
} }
// v4.11.1: Final progress update // Final progress update
if (options.onProgress) { if (options.onProgress) {
options.onProgress({ options.onProgress({
stage: 'metadata', stage: 'metadata',
@ -355,7 +355,7 @@ export class VFSStructureGenerator {
): Map<string, typeof importResult.rows> { ): Map<string, typeof importResult.rows> {
const groups = new Map<string, typeof importResult.rows>() const groups = new Map<string, typeof importResult.rows>()
// Handle sheet-based grouping (v4.2.0) // Handle sheet-based grouping
if (options.groupBy === 'sheet' && importResult.sheets && importResult.sheets.length > 0) { if (options.groupBy === 'sheet' && importResult.sheets && importResult.sheets.length > 0) {
for (const sheet of importResult.sheets) { for (const sheet of importResult.sheets) {
groups.set(sheet.name, sheet.rows) groups.set(sheet.name, sheet.rows)

View file

@ -70,7 +70,7 @@ export type {
NeuralImportOptions NeuralImportOptions
} from './cortex/neuralImport.js' } from './cortex/neuralImport.js'
// Export Neural Entity Extraction (v5.7.6 - Workshop request) // Export Neural Entity Extraction
export { NeuralEntityExtractor } from './neural/entityExtractor.js' export { NeuralEntityExtractor } from './neural/entityExtractor.js'
export { SmartExtractor } from './neural/SmartExtractor.js' export { SmartExtractor } from './neural/SmartExtractor.js'
export { SmartRelationshipExtractor } from './neural/SmartRelationshipExtractor.js' export { SmartRelationshipExtractor } from './neural/SmartRelationshipExtractor.js'
@ -220,7 +220,7 @@ export {
// FileSystemStorage is exported separately to avoid browser build issues // FileSystemStorage is exported separately to avoid browser build issues
export { FileSystemStorage } from './storage/adapters/fileSystemStorage.js' export { FileSystemStorage } from './storage/adapters/fileSystemStorage.js'
// Export COW (Copy-on-Write) infrastructure for v5.0.0 // Export COW (Copy-on-Write) infrastructure
// Enables premium augmentations to implement temporal features // Enables premium augmentations to implement temporal features
import { CommitLog } from './storage/cow/CommitLog.js' import { CommitLog } from './storage/cow/CommitLog.js'
import { CommitObject, CommitBuilder } from './storage/cow/CommitObject.js' import { CommitObject, CommitBuilder } from './storage/cow/CommitObject.js'
@ -543,7 +543,7 @@ export type {
MCPTool MCPTool
} }
// ============= Integration Hub (v7.4.0) ============= // ============= Integration Hub =============
// Connect Brainy to Excel, Power BI, Google Sheets, and more // Connect Brainy to Excel, Power BI, Google Sheets, and more
// Enable with: new Brainy({ integrations: true }) // Enable with: new Brainy({ integrations: true })

View file

@ -7,7 +7,7 @@
* - Real-time dashboards (SSE streaming) * - Real-time dashboards (SSE streaming)
* - External webhooks (push notifications) * - External webhooks (push notifications)
* *
* @example Enable integrations (v7.4.0 - recommended) * @example Enable integrations (recommended)
* ```typescript * ```typescript
* import { Brainy } from '@soulcraft/brainy' * import { Brainy } from '@soulcraft/brainy'
* *

View file

@ -1,5 +1,5 @@
/** /**
* Unified Index Interface (v3.35.0+) * Unified Index Interface
* *
* Standardizes index lifecycle across all index types in Brainy. * Standardizes index lifecycle across all index types in Brainy.
* All indexes (HNSW Vector, Graph Adjacency, Metadata Field) implement this interface * All indexes (HNSW Vector, Graph Adjacency, Metadata Field) implement this interface
@ -57,7 +57,7 @@ export interface RebuildOptions {
* - On-demand: Large datasets loaded adaptively via UnifiedCache * - On-demand: Large datasets loaded adaptively via UnifiedCache
* *
* This option is kept for backwards compatibility but is ignored. * This option is kept for backwards compatibility but is ignored.
* The system always uses adaptive caching (v3.36.0+). * The system always uses adaptive caching.
*/ */
lazy?: boolean lazy?: boolean
@ -104,7 +104,7 @@ export interface IIndex {
* - Provide progress reporting for large datasets * - Provide progress reporting for large datasets
* - Recover gracefully from partial failures * - Recover gracefully from partial failures
* *
* Adaptive Caching (v3.36.0+): * Adaptive Caching:
* System automatically chooses optimal strategy: * System automatically chooses optimal strategy:
* - Small datasets: Preload all data at init for zero-latency access * - Small datasets: Preload all data at init for zero-latency access
* - Large datasets: Load on-demand via UnifiedCache for memory efficiency * - Large datasets: Load on-demand via UnifiedCache for memory efficiency

View file

@ -2,7 +2,7 @@
* Neural Entity Extractor using Brainy's NounTypes * Neural Entity Extractor using Brainy's NounTypes
* Uses embeddings and similarity matching for accurate type detection * Uses embeddings and similarity matching for accurate type detection
* *
* v4.2.0: Now powered by SmartExtractor for ultra-neural classification * Now powered by SmartExtractor for ultra-neural classification
* PRODUCTION-READY with caching support * PRODUCTION-READY with caching support
*/ */
@ -24,7 +24,7 @@ export interface ExtractedEntity {
type: NounType type: NounType
position: { start: number; end: number } position: { start: number; end: number }
confidence: number confidence: number
weight?: number // v4.2.0: Entity importance/salience weight?: number // Entity importance/salience
vector?: Vector vector?: Vector
metadata?: any metadata?: any
} }
@ -39,7 +39,7 @@ export class NeuralEntityExtractor {
// Entity extraction cache // Entity extraction cache
private cache: EntityExtractionCache private cache: EntityExtractionCache
// Runtime embedding cache for performance (v3.38.0) // Runtime embedding cache for performance
// Caches candidate embeddings during an extraction session to avoid redundant model calls // Caches candidate embeddings during an extraction session to avoid redundant model calls
private embeddingCache: Map<string, Vector> = new Map() private embeddingCache: Map<string, Vector> = new Map()
private embeddingCacheStats = { private embeddingCacheStats = {
@ -48,7 +48,7 @@ export class NeuralEntityExtractor {
size: 0 size: 0
} }
// v4.2.0: SmartExtractor for ultra-neural classification // SmartExtractor for ultra-neural classification
private smartExtractor: SmartExtractor private smartExtractor: SmartExtractor
constructor(brain: Brainy | Brainy<any>, cacheOptions?: EntityCacheOptions) { constructor(brain: Brainy | Brainy<any>, cacheOptions?: EntityCacheOptions) {
@ -63,7 +63,7 @@ export class NeuralEntityExtractor {
/** /**
* Initialize type embeddings for neural matching * Initialize type embeddings for neural matching
* PRODUCTION OPTIMIZATION (v3.33.0): Uses pre-computed embeddings from build time * PRODUCTION OPTIMIZATION: Uses pre-computed embeddings from build time
* Zero runtime cost - embeddings are loaded instantly from embedded data * Zero runtime cost - embeddings are loaded instantly from embedded data
*/ */
private async initializeTypeEmbeddings(requestedTypes?: NounType[]): Promise<void> { private async initializeTypeEmbeddings(requestedTypes?: NounType[]): Promise<void> {
@ -109,7 +109,7 @@ export class NeuralEntityExtractor {
} }
} }
): Promise<ExtractedEntity[]> { ): Promise<ExtractedEntity[]> {
// PRODUCTION OPTIMIZATION (v3.33.0): Load pre-computed type embeddings // PRODUCTION OPTIMIZATION: Load pre-computed type embeddings
// Zero runtime cost - embeddings were computed at build time // Zero runtime cost - embeddings were computed at build time
await this.initializeTypeEmbeddings(options?.types) await this.initializeTypeEmbeddings(options?.types)
@ -138,7 +138,7 @@ export class NeuralEntityExtractor {
// Step 1: Extract potential entities using patterns // Step 1: Extract potential entities using patterns
const candidates = await this.extractCandidates(text) const candidates = await this.extractCandidates(text)
// Step 2: Classify each candidate using SmartExtractor (v4.2.0) // Step 2: Classify each candidate using SmartExtractor
for (const candidate of candidates) { for (const candidate of candidates) {
// Use SmartExtractor for unified neural + rule-based classification // Use SmartExtractor for unified neural + rule-based classification
const classification = await this.smartExtractor.extract(candidate.text, { const classification = await this.smartExtractor.extract(candidate.text, {
@ -357,7 +357,7 @@ export class NeuralEntityExtractor {
} }
/** /**
* Get embedding for text with caching (v3.38.0) * Get embedding for text with caching
* *
* PERFORMANCE OPTIMIZATION: Caches embeddings during extraction session * PERFORMANCE OPTIMIZATION: Caches embeddings during extraction session
* to avoid redundant model calls for repeated text (common in large imports) * to avoid redundant model calls for repeated text (common in large imports)
@ -509,7 +509,7 @@ export class NeuralEntityExtractor {
} }
/** /**
* Clear embedding cache (v3.38.0) * Clear embedding cache
* *
* Clears the runtime embedding cache. Useful for: * Clears the runtime embedding cache. Useful for:
* - Freeing memory after large imports * - Freeing memory after large imports
@ -525,7 +525,7 @@ export class NeuralEntityExtractor {
} }
/** /**
* Get embedding cache statistics (v3.38.0) * Get embedding cache statistics
* *
* Returns performance metrics for the embedding cache: * Returns performance metrics for the embedding cache:
* - hits: Number of cache hits (avoided model calls) * - hits: Number of cache hits (avoided model calls)

View file

@ -99,7 +99,7 @@ export class NaturalLanguageProcessor {
/** /**
* Initialize embeddings for all NounTypes and VerbTypes * Initialize embeddings for all NounTypes and VerbTypes
* PRODUCTION OPTIMIZATION (v3.33.0): Uses pre-computed type embeddings * PRODUCTION OPTIMIZATION: Uses pre-computed type embeddings
* Zero runtime cost - embeddings are loaded instantly from embedded data * Zero runtime cost - embeddings are loaded instantly from embedded data
*/ */
private async initializeTypeEmbeddings(): Promise<void> { private async initializeTypeEmbeddings(): Promise<void> {

View file

@ -122,7 +122,7 @@ export class PatternSignal {
]) ])
// Location patterns - MEDIUM PRIORITY (city/country format - requires more context) // Location patterns - MEDIUM PRIORITY (city/country format - requires more context)
// v4.11.2: Lower priority to avoid matching person names with commas // Lower priority to avoid matching person names with commas
this.addPatterns(NounType.Location, 0.75, [ this.addPatterns(NounType.Location, 0.75, [
/\b[A-Z][a-z]+,\s*(?:Japan|China|France|Germany|Italy|Spain|Canada|Mexico|Brazil|India|Australia|Russia|UK|USA)\b/ /\b[A-Z][a-z]+,\s*(?:Japan|China|France|Germany|Italy|Spain|Canada|Mexico|Brazil|India|Australia|Russia|UK|USA)\b/
]) ])
@ -170,7 +170,7 @@ export class PatternSignal {
// Technology patterns (Thing type) // Technology patterns (Thing type)
this.addPatterns(NounType.Thing, 0.82, [ this.addPatterns(NounType.Thing, 0.82, [
/\b(?:JavaScript|TypeScript|Python|Java|Go|Rust|Swift|Kotlin)\b/, /\b(?:JavaScript|TypeScript|Python|Java|Go|Rust|Swift|Kotlin)\b/,
/\bC\+\+(?!\w)/, // v4.11.2: Special handling for C++ (word boundary doesn't work with +) /\bC\+\+(?!\w)/, // Special handling for C++ (word boundary doesn't work with +)
/\b(?:React|Vue|Angular|Node|Express|Django|Flask|Rails)\b/, /\b(?:React|Vue|Angular|Node|Express|Django|Flask|Rails)\b/,
/\b(?:AWS|Azure|GCP|Docker|Kubernetes|Git|GitHub|GitLab)\b/, /\b(?:AWS|Azure|GCP|Docker|Kubernetes|Git|GitHub|GitLab)\b/,
/\b(?:API|SDK|CLI|IDE|framework|library|package|module)\b/i, /\b(?:API|SDK|CLI|IDE|framework|library|package|module)\b/i,

View file

@ -1,7 +1,7 @@
/** /**
* Brainy Setup - Minimal Polyfills * Brainy Setup - Minimal Polyfills
* *
* ARCHITECTURE (v7.0.0): * ARCHITECTURE:
* Brainy uses Candle WASM (Rust-based) for embeddings. * Brainy uses Candle WASM (Rust-based) for embeddings.
* No transformers.js or ONNX Runtime dependency, no hacks required. * No transformers.js or ONNX Runtime dependency, no hacks required.
* *

View file

@ -9,7 +9,7 @@
* 4. SAS Token * 4. SAS Token
* 5. Azure AD (OAuth2) via DefaultAzureCredential * 5. Azure AD (OAuth2) via DefaultAzureCredential
* *
* v4.0.0: Fully compatible with metadata/vector separation architecture * Fully compatible with metadata/vector separation architecture
*/ */
import { import {
@ -65,7 +65,7 @@ const MAX_AZURE_PAGE_SIZE = 5000
* 3. Storage Account Key - if accountName + accountKey provided * 3. Storage Account Key - if accountName + accountKey provided
* 4. SAS Token - if accountName + sasToken provided * 4. SAS Token - if accountName + sasToken provided
* *
* v5.4.0: Type-aware storage now built into BaseStorage * Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation) * - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed pagination overrides * - Removed pagination overrides
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths) * - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -105,7 +105,7 @@ export class AzureBlobStorage extends BaseStorage {
// Request coalescer for deduplication // Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance // Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching // Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Multi-level cache manager for efficient data access // Multi-level cache manager for efficient data access
@ -115,7 +115,7 @@ export class AzureBlobStorage extends BaseStorage {
// Module logger // Module logger
private logger = createModuleLogger('AzureBlobStorage') private logger = createModuleLogger('AzureBlobStorage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races // HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>() private hnswLocks = new Map<string, Promise<void>>()
/** /**
@ -162,7 +162,7 @@ export class AzureBlobStorage extends BaseStorage {
} }
/** /**
* Initialization mode for fast cold starts (v7.3.0+) * Initialization mode for fast cold starts
* *
* - `'auto'` (default): Progressive in cloud environments (Azure Functions), * - `'auto'` (default): Progressive in cloud environments (Azure Functions),
* strict locally. Zero-config optimization. * strict locally. Zero-config optimization.
@ -171,7 +171,6 @@ export class AzureBlobStorage extends BaseStorage {
* - `'strict'`: Traditional blocking init. Validates container and loads counts * - `'strict'`: Traditional blocking init. Validates container and loads counts
* before init() returns. * before init() returns.
* *
* @since v7.3.0
*/ */
initMode?: InitMode initMode?: InitMode
@ -185,7 +184,7 @@ export class AzureBlobStorage extends BaseStorage {
this.sasToken = options.sasToken this.sasToken = options.sasToken
this.readOnly = options.readOnly || false this.readOnly = options.readOnly || false
// v7.3.0: Handle initMode // Handle initMode
if (options.initMode) { if (options.initMode) {
this.initMode = options.initMode this.initMode = options.initMode
} }
@ -201,7 +200,7 @@ export class AzureBlobStorage extends BaseStorage {
this.nounCacheManager = new CacheManager<HNSWNode>(options.cacheConfig) this.nounCacheManager = new CacheManager<HNSWNode>(options.cacheConfig)
this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig) this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig)
// v6.2.7: Write buffering always enabled - no env var check needed // Write buffering always enabled - no env var check needed
} }
/** /**
@ -216,7 +215,7 @@ export class AzureBlobStorage extends BaseStorage {
* Recent Azure improvements make parallel downloads very efficient * Recent Azure improvements make parallel downloads very efficient
* *
* @returns Azure Blob-optimized batch configuration * @returns Azure Blob-optimized batch configuration
* @since v5.12.0 - Updated for native batch API * Updated for native batch API
*/ */
public getBatchConfig(): StorageBatchConfig { public getBatchConfig(): StorageBatchConfig {
return { return {
@ -241,7 +240,6 @@ export class AzureBlobStorage extends BaseStorage {
* *
* @param paths - Array of Azure blob paths to read * @param paths - Array of Azure blob paths to read
* @returns Map of path -> parsed JSON data (only successful reads) * @returns Map of path -> parsed JSON data (only successful reads)
* @since v5.12.0
*/ */
public async readBatch(paths: string[]): Promise<Map<string, any>> { public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized() await this.ensureInitialized()
@ -297,7 +295,7 @@ export class AzureBlobStorage extends BaseStorage {
/** /**
* Initialize the storage adapter * Initialize the storage adapter
* *
* v7.3.0: Supports progressive initialization for fast cold starts * Supports progressive initialization for fast cold starts
* *
* | Mode | Init Time | When | * | Mode | Init Time | When |
* |------|-----------|------| * |------|-----------|------|
@ -404,7 +402,7 @@ export class AzureBlobStorage extends BaseStorage {
this.verbCacheManager.clear() this.verbCacheManager.clear()
prodLog.info('✅ Cache cleared - starting fresh') prodLog.info('✅ Cache cleared - starting fresh')
// v7.3.0: Progressive vs Strict initialization // Progressive vs Strict initialization
if (effectiveMode === 'progressive') { if (effectiveMode === 'progressive') {
// PROGRESSIVE MODE: Fast init, background validation // PROGRESSIVE MODE: Fast init, background validation
// Mark as initialized immediately - ready to accept operations // Mark as initialized immediately - ready to accept operations
@ -413,7 +411,7 @@ export class AzureBlobStorage extends BaseStorage {
prodLog.info(`✅ Azure progressive init complete: ${this.containerName} (validation deferred)`) prodLog.info(`✅ Azure progressive init complete: ${this.containerName} (validation deferred)`)
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics // Initialize GraphAdjacencyIndex and type statistics
await super.init() await super.init()
// Schedule background tasks (non-blocking) // Schedule background tasks (non-blocking)
@ -434,7 +432,7 @@ export class AzureBlobStorage extends BaseStorage {
await this.initializeCounts() await this.initializeCounts()
this.countsLoaded = true this.countsLoaded = true
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics // Initialize GraphAdjacencyIndex and type statistics
await super.init() await super.init()
// Mark background tasks as complete (nothing to do in background) // Mark background tasks as complete (nothing to do in background)
@ -447,7 +445,7 @@ export class AzureBlobStorage extends BaseStorage {
} }
// ============================================= // =============================================
// Progressive Initialization (v7.3.0+) // Progressive Initialization
// ============================================= // =============================================
/** /**
@ -461,7 +459,6 @@ export class AzureBlobStorage extends BaseStorage {
* *
* @protected * @protected
* @override * @override
* @since v7.3.0
*/ */
protected async runBackgroundInit(): Promise<void> { protected async runBackgroundInit(): Promise<void> {
const startTime = Date.now() const startTime = Date.now()
@ -485,7 +482,6 @@ export class AzureBlobStorage extends BaseStorage {
* bucketValidated/bucketValidationError for lazy use. * bucketValidated/bucketValidationError for lazy use.
* *
* @private * @private
* @since v7.3.0
*/ */
private async validateContainerInBackground(): Promise<void> { private async validateContainerInBackground(): Promise<void> {
try { try {
@ -517,7 +513,6 @@ export class AzureBlobStorage extends BaseStorage {
* Load counts from storage in background. * Load counts from storage in background.
* *
* @private * @private
* @since v7.3.0
*/ */
private async loadCountsInBackground(): Promise<void> { private async loadCountsInBackground(): Promise<void> {
try { try {
@ -541,7 +536,6 @@ export class AzureBlobStorage extends BaseStorage {
* @throws Error if container validation fails * @throws Error if container validation fails
* @protected * @protected
* @override * @override
* @since v7.3.0
*/ */
protected async ensureValidatedForWrite(): Promise<void> { protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do // If already validated, nothing to do
@ -665,7 +659,7 @@ export class AzureBlobStorage extends BaseStorage {
} }
} }
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage // Removed checkVolumeMode() - write buffering always enabled for cloud storage
/** /**
* Flush noun buffer to Azure * Flush noun buffer to Azure
@ -697,20 +691,20 @@ export class AzureBlobStorage extends BaseStorage {
await Promise.all(writes) await Promise.all(writes)
} }
// v5.4.0: Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation // Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation
/** /**
* Save a node to storage * Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance * Always uses write buffer for consistent performance
*/ */
protected async saveNode(node: HNSWNode): Promise<void> { protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching // Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) { if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`) this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency // Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) { if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node) this.nounCacheManager.set(node.id, node)
} }
@ -791,7 +785,7 @@ export class AzureBlobStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed getNoun_internal - now inherit from BaseStorage's type-first implementation // Removed getNoun_internal - now inherit from BaseStorage's type-first implementation
/** /**
* Get a node from storage * Get a node from storage
@ -889,18 +883,18 @@ export class AzureBlobStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation // Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation
/** /**
* Write an object to a specific path in Azure * Write an object to a specific path in Azure
* Primitive operation required by base class * Primitive operation required by base class
* *
* v7.3.0: Performs lazy container validation on first write in progressive mode. * Performs lazy container validation on first write in progressive mode.
* @protected * @protected
*/ */
protected async writeObjectToPath(path: string, data: any): Promise<void> { protected async writeObjectToPath(path: string, data: any): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v7.3.0: Lazy container validation for progressive init // Lazy container validation for progressive init
await this.ensureValidatedForWrite() await this.ensureValidatedForWrite()
try { try {
@ -954,12 +948,12 @@ export class AzureBlobStorage extends BaseStorage {
* Delete an object from a specific path in Azure * Delete an object from a specific path in Azure
* Primitive operation required by base class * Primitive operation required by base class
* *
* v7.3.0: Performs lazy container validation on first delete in progressive mode. * Performs lazy container validation on first delete in progressive mode.
* @protected * @protected
*/ */
protected async deleteObjectFromPath(path: string): Promise<void> { protected async deleteObjectFromPath(path: string): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v7.3.0: Lazy container validation for progressive init // Lazy container validation for progressive init
await this.ensureValidatedForWrite() await this.ensureValidatedForWrite()
try { try {
@ -1209,20 +1203,20 @@ export class AzureBlobStorage extends BaseStorage {
}) })
} }
// v5.4.0: Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation // Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation
/** /**
* Save an edge to storage * Save an edge to storage
* v6.2.7: Always uses write buffer for consistent performance * Always uses write buffer for consistent performance
*/ */
protected async saveEdge(edge: Edge): Promise<void> { protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching // Always use write buffer - cloud storage benefits from batching
if (this.verbWriteBuffer) { if (this.verbWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding verb ${edge.id} to write buffer`) this.logger.trace(`📝 BUFFERING: Adding verb ${edge.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency // Populate cache BEFORE buffering for read-after-write consistency
this.verbCacheManager.set(edge.id, edge) this.verbCacheManager.set(edge.id, edge)
await this.verbWriteBuffer.add(edge.id, edge) await this.verbWriteBuffer.add(edge.id, edge)
@ -1255,7 +1249,7 @@ export class AzureBlobStorage extends BaseStorage {
]) ])
), ),
// CORE RELATIONAL DATA (v4.0.0) // CORE RELATIONAL DATA
verb: edge.verb, verb: edge.verb,
sourceId: edge.sourceId, sourceId: edge.sourceId,
targetId: edge.targetId, targetId: edge.targetId,
@ -1280,7 +1274,7 @@ export class AzureBlobStorage extends BaseStorage {
// Update cache // Update cache
this.verbCacheManager.set(edge.id, edge) this.verbCacheManager.set(edge.id, edge)
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2) // Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet // This fixes the race condition where metadata didn't exist yet
this.logger.trace(`Edge ${edge.id} saved successfully`) this.logger.trace(`Edge ${edge.id} saved successfully`)
@ -1298,7 +1292,7 @@ export class AzureBlobStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed getVerb_internal - now inherit from BaseStorage's type-first implementation // Removed getVerb_internal - now inherit from BaseStorage's type-first implementation
/** /**
* Get an edge from storage * Get an edge from storage
@ -1335,7 +1329,7 @@ export class AzureBlobStorage extends BaseStorage {
connections.set(Number(level), new Set(verbIds as string[])) connections.set(Number(level), new Set(verbIds as string[]))
} }
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field) // Return HNSWVerb with core relational fields (NO metadata field)
const edge: Edge = { const edge: Edge = {
id: data.id, id: data.id,
vector: data.vector, vector: data.vector,
@ -1346,7 +1340,7 @@ export class AzureBlobStorage extends BaseStorage {
sourceId: data.sourceId, sourceId: data.sourceId,
targetId: data.targetId targetId: data.targetId
// ✅ NO metadata field in v4.0.0 // ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata() // User metadata retrieved separately via getVerbMetadata()
} }
@ -1375,13 +1369,13 @@ export class AzureBlobStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation // Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed getNounsWithPagination - now inherit from BaseStorage's type-first implementation // Removed getNounsWithPagination - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed getNounsByNounType_internal - now inherit from BaseStorage's type-first implementation // Removed getNounsByNounType_internal - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed 3 verb query *_internal methods (getVerbsBySource, getVerbsByTarget, getVerbsByType) - now inherit from BaseStorage's type-first implementation // Removed 3 verb query *_internal methods (getVerbsBySource, getVerbsByTarget, getVerbsByType) - now inherit from BaseStorage's type-first implementation
/** /**
* Clear all data from storage * Clear all data from storage
@ -1393,7 +1387,7 @@ export class AzureBlobStorage extends BaseStorage {
this.logger.info('🧹 Clearing all data from Azure container...') this.logger.info('🧹 Clearing all data from Azure container...')
// Delete all blobs in container // Delete all blobs in container
// v5.6.1: listBlobsFlat() returns ALL blobs including _cow/ prefix // listBlobsFlat() returns ALL blobs including _cow/ prefix
// This correctly deletes COW version control data (commits, trees, blobs, refs) // This correctly deletes COW version control data (commits, trees, blobs, refs)
for await (const blob of this.containerClient!.listBlobsFlat()) { for await (const blob of this.containerClient!.listBlobsFlat()) {
if (blob.name) { if (blob.name) {
@ -1402,7 +1396,7 @@ export class AzureBlobStorage extends BaseStorage {
} }
} }
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled) // Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use // COW will re-initialize automatically on next use
this.refManager = undefined this.refManager = undefined
this.blobStorage = undefined this.blobStorage = undefined
@ -1461,12 +1455,12 @@ export class AzureBlobStorage extends BaseStorage {
/** /**
* Check if COW has been explicitly disabled via clear() * Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts * Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker blob exists, false otherwise * @returns true if marker blob exists, false otherwise
* @protected * @protected
*/ */
/** /**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods * Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used * COW is now always enabled - marker files are no longer used
*/ */
@ -1643,7 +1637,7 @@ export class AzureBlobStorage extends BaseStorage {
/** /**
* Get a noun's vector for HNSW rebuild * Get a noun's vector for HNSW rebuild
* v5.4.0: Uses BaseStorage's getNoun (type-first paths) * Uses BaseStorage's getNoun (type-first paths)
*/ */
public async getNounVector(id: string): Promise<number[] | null> { public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id) const noun = await this.getNoun(id)
@ -1653,7 +1647,7 @@ export class AzureBlobStorage extends BaseStorage {
/** /**
* Save HNSW graph data for a noun * Save HNSW graph data for a noun
* *
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths) * Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Uses mutex locking to prevent read-modify-write races * CRITICAL: Uses mutex locking to prevent read-modify-write races
*/ */
public async saveHNSWData(nounId: string, hnswData: { public async saveHNSWData(nounId: string, hnswData: {
@ -1662,7 +1656,7 @@ export class AzureBlobStorage extends BaseStorage {
}): Promise<void> { }): Promise<void> {
const lockKey = `hnsw/${nounId}` const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races // CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can: // Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3]) // 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3]) // 2. Thread B reads noun (connections: [1,2,3])
@ -1682,7 +1676,7 @@ export class AzureBlobStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise) this.hnswLocks.set(lockKey, lockPromise)
try { try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths) // Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists) // Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId) const existingNoun = await this.getNoun(nounId)
@ -1704,7 +1698,7 @@ export class AzureBlobStorage extends BaseStorage {
connections: connectionsMap connections: connectionsMap
} }
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch) // Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
await this.saveNoun(updatedNoun) await this.saveNoun(updatedNoun)
} finally { } finally {
// Release lock (ALWAYS runs, even if error thrown) // Release lock (ALWAYS runs, even if error thrown)
@ -1715,7 +1709,7 @@ export class AzureBlobStorage extends BaseStorage {
/** /**
* Get HNSW graph data for a noun * Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths) * Uses BaseStorage's getNoun (type-first paths)
*/ */
public async getHNSWData(nounId: string): Promise<{ public async getHNSWData(nounId: string): Promise<{
level: number level: number
@ -1744,7 +1738,7 @@ export class AzureBlobStorage extends BaseStorage {
/** /**
* Save HNSW system data (entry point, max level) * Save HNSW system data (entry point, max level)
* *
* CRITICAL FIX (v4.10.1): Optimistic locking with ETags to prevent race conditions * CRITICAL FIX: Optimistic locking with ETags to prevent race conditions
*/ */
public async saveHNSWSystem(systemData: { public async saveHNSWSystem(systemData: {
entryPointId: string | null entryPointId: string | null
@ -1840,7 +1834,7 @@ export class AzureBlobStorage extends BaseStorage {
} }
/** /**
* Set the access tier for a specific blob (v4.0.0 cost optimization) * Set the access tier for a specific blob (cost optimization)
* Azure Blob Storage tiers: * Azure Blob Storage tiers:
* - Hot: $0.0184/GB/month - Frequently accessed data * - Hot: $0.0184/GB/month - Frequently accessed data
* - Cool: $0.01/GB/month - Infrequently accessed data (45% cheaper) * - Cool: $0.01/GB/month - Infrequently accessed data (45% cheaper)
@ -1906,7 +1900,7 @@ export class AzureBlobStorage extends BaseStorage {
} }
/** /**
* Set access tier for multiple blobs in batch (v4.0.0 cost optimization) * Set access tier for multiple blobs in batch (cost optimization)
* Efficiently move large numbers of blobs between tiers for cost optimization * Efficiently move large numbers of blobs between tiers for cost optimization
* *
* @param blobs - Array of blob names and their target tiers * @param blobs - Array of blob names and their target tiers
@ -2128,7 +2122,7 @@ export class AzureBlobStorage extends BaseStorage {
} }
/** /**
* Set lifecycle management policy for automatic tier transitions and deletions (v4.0.0) * Set lifecycle management policy for automatic tier transitions and deletions
* Automates cost optimization by moving old data to cheaper tiers or deleting it * Automates cost optimization by moving old data to cheaper tiers or deleting it
* *
* Azure Lifecycle Management rules run once per day and apply to the entire container. * Azure Lifecycle Management rules run once per day and apply to the entire container.

View file

@ -19,7 +19,7 @@ import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/field
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js' import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
// ============================================================================= // =============================================================================
// Progressive Initialization Types (v7.3.0+) // Progressive Initialization Types
// ============================================================================= // =============================================================================
/** /**
@ -29,7 +29,6 @@ import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
* fast cold starts in serverless environments while maintaining strict * fast cold starts in serverless environments while maintaining strict
* validation for local development. * validation for local development.
* *
* @since v7.3.0
* *
* | Mode | Description | Use Case | * | Mode | Description | Use Case |
* |------|-------------|----------| * |------|-------------|----------|
@ -96,7 +95,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
abstract getVerbMetadata(id: string): Promise<any | null> abstract getVerbMetadata(id: string): Promise<any | null>
// HNSW Index Persistence (v3.35.0+) // HNSW Index Persistence
// These methods enable HNSW index rebuilding after container restarts // These methods enable HNSW index rebuilding after container restarts
abstract getNounVector(id: string): Promise<number[] | null> abstract getNounVector(id: string): Promise<number[] | null>
@ -139,7 +138,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* relateMany(), and import operations to automatically adapt to storage capabilities. * relateMany(), and import operations to automatically adapt to storage capabilities.
* *
* @returns Batch configuration optimized for this storage type * @returns Batch configuration optimized for this storage type
* @since v4.11.0
*/ */
public getBatchConfig(): StorageBatchConfig { public getBatchConfig(): StorageBatchConfig {
// Conservative defaults that work safely across all storage types // Conservative defaults that work safely across all storage types
@ -295,7 +293,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
}> = new Map() }> = new Map()
// ============================================= // =============================================
// Progressive Initialization State (v7.3.0+) // Progressive Initialization State
// ============================================= // =============================================
// These properties enable fast cold starts in cloud environments // These properties enable fast cold starts in cloud environments
// by deferring validation and count loading to background tasks. // by deferring validation and count loading to background tasks.
@ -308,7 +306,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* - `'strict'`: Always validate during init (traditional behavior) * - `'strict'`: Always validate during init (traditional behavior)
* *
* @protected * @protected
* @since v7.3.0
*/ */
protected initMode: InitMode = 'auto' protected initMode: InitMode = 'auto'
@ -319,7 +316,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* after the first successful write operation or background validation. * after the first successful write operation or background validation.
* *
* @protected * @protected
* @since v7.3.0
*/ */
protected bucketValidated = false protected bucketValidated = false
@ -329,7 +325,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Stored here so subsequent operations can fail fast with the same error. * Stored here so subsequent operations can fail fast with the same error.
* *
* @protected * @protected
* @since v7.3.0
*/ */
protected bucketValidationError: Error | null = null protected bucketValidationError: Error | null = null
@ -340,7 +335,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Operations work immediately; counts become accurate after background load. * Operations work immediately; counts become accurate after background load.
* *
* @protected * @protected
* @since v7.3.0
*/ */
protected countsLoaded = false protected countsLoaded = false
@ -350,7 +344,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Useful for tests and diagnostics to ensure full initialization. * Useful for tests and diagnostics to ensure full initialization.
* *
* @protected * @protected
* @since v7.3.0
*/ */
protected backgroundTasksComplete = false protected backgroundTasksComplete = false
@ -361,7 +354,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* before proceeding (e.g., in tests). * before proceeding (e.g., in tests).
* *
* @protected * @protected
* @since v7.3.0
*/ */
protected backgroundInitPromise: Promise<void> | null = null protected backgroundInitPromise: Promise<void> | null = null
@ -1057,7 +1049,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL
// ============================================= // =============================================
// Smart Count Batching (v3.32.3+) // Smart Count Batching
// ============================================= // =============================================
// Count batching state - mirrors statistics batching pattern // Count batching state - mirrors statistics batching pattern
@ -1112,7 +1104,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex() const mutex = getGlobalMutex()
await mutex.runExclusive(`count-entity-${type}`, async () => { await mutex.runExclusive(`count-entity-${type}`, async () => {
this.incrementEntityCount(type) this.incrementEntityCount(type)
// Smart batching (v3.32.3+): Adapts to storage type // Smart batching: Adapts to storage type
// - Cloud storage (GCS, S3): Batches 10 ops OR 5 seconds // - Cloud storage (GCS, S3): Batches 10 ops OR 5 seconds
// - Local storage (File, Memory): Persists immediately // - Local storage (File, Memory): Persists immediately
await this.scheduleCountPersist() await this.scheduleCountPersist()
@ -1147,13 +1139,13 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex() const mutex = getGlobalMutex()
await mutex.runExclusive(`count-entity-${type}`, async () => { await mutex.runExclusive(`count-entity-${type}`, async () => {
this.decrementEntityCount(type) this.decrementEntityCount(type)
// Smart batching (v3.32.3+): Adapts to storage type // Smart batching: Adapts to storage type
await this.scheduleCountPersist() await this.scheduleCountPersist()
}) })
} }
/** /**
* Increment verb count - O(1) operation (v4.1.2: now synchronous) * Increment verb count - O(1) operation (now synchronous)
* Protected by storage-specific mechanisms (mutex, distributed consensus, etc.) * Protected by storage-specific mechanisms (mutex, distributed consensus, etc.)
* @param type The verb type * @param type The verb type
*/ */
@ -1168,7 +1160,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
} }
/** /**
* Thread-safe increment for verb counts (v4.1.2) * Thread-safe increment for verb counts
* Uses mutex for single-node, distributed consensus for multi-node * Uses mutex for single-node, distributed consensus for multi-node
* @param type The verb type * @param type The verb type
*/ */
@ -1176,13 +1168,13 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
const mutex = getGlobalMutex() const mutex = getGlobalMutex()
await mutex.runExclusive(`count-verb-${type}`, async () => { await mutex.runExclusive(`count-verb-${type}`, async () => {
this.incrementVerbCount(type) this.incrementVerbCount(type)
// Smart batching (v3.32.3+): Adapts to storage type // Smart batching: Adapts to storage type
await this.scheduleCountPersist() await this.scheduleCountPersist()
}) })
} }
/** /**
* Decrement verb count - O(1) operation (v4.1.2: now synchronous) * Decrement verb count - O(1) operation (now synchronous)
* @param type The verb type * @param type The verb type
*/ */
protected decrementVerbCount(type: string): void { protected decrementVerbCount(type: string): void {
@ -1203,20 +1195,20 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
} }
/** /**
* Thread-safe decrement for verb counts (v4.1.2) * Thread-safe decrement for verb counts
* @param type The verb type * @param type The verb type
*/ */
protected async decrementVerbCountSafe(type: string): Promise<void> { protected async decrementVerbCountSafe(type: string): Promise<void> {
const mutex = getGlobalMutex() const mutex = getGlobalMutex()
await mutex.runExclusive(`count-verb-${type}`, async () => { await mutex.runExclusive(`count-verb-${type}`, async () => {
this.decrementVerbCount(type) this.decrementVerbCount(type)
// Smart batching (v3.32.3+): Adapts to storage type // Smart batching: Adapts to storage type
await this.scheduleCountPersist() await this.scheduleCountPersist()
}) })
} }
// ============================================= // =============================================
// Smart Batching Methods (v3.32.3+) // Smart Batching Methods
// ============================================= // =============================================
/** /**
@ -1235,7 +1227,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
} }
// ============================================= // =============================================
// Progressive Initialization Methods (v7.3.0+) // Progressive Initialization Methods
// ============================================= // =============================================
/** /**
@ -1254,7 +1246,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* *
* @returns `true` if running in a detected cloud environment * @returns `true` if running in a detected cloud environment
* @protected * @protected
* @since v7.3.0
*/ */
protected detectCloudEnvironment(): boolean { protected detectCloudEnvironment(): boolean {
return !!( return !!(
@ -1274,7 +1265,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* *
* @returns The resolved init mode (`'progressive'` or `'strict'`) * @returns The resolved init mode (`'progressive'` or `'strict'`)
* @protected * @protected
* @since v7.3.0
*/ */
protected resolveInitMode(): 'progressive' | 'strict' { protected resolveInitMode(): 'progressive' | 'strict' {
if (this.initMode === 'auto') { if (this.initMode === 'auto') {
@ -1295,7 +1285,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* Always call `super.scheduleBackgroundInit()` first. * Always call `super.scheduleBackgroundInit()` first.
* *
* @protected * @protected
* @since v7.3.0
*/ */
protected scheduleBackgroundInit(): void { protected scheduleBackgroundInit(): void {
// Create a promise that tracks all background work // Create a promise that tracks all background work
@ -1320,7 +1309,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* The default implementation does nothing (for local storage adapters). * The default implementation does nothing (for local storage adapters).
* *
* @protected * @protected
* @since v7.3.0
*/ */
protected async runBackgroundInit(): Promise<void> { protected async runBackgroundInit(): Promise<void> {
// Default implementation: nothing to do for local storage // Default implementation: nothing to do for local storage
@ -1348,7 +1336,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* ``` * ```
* *
* @public * @public
* @since v7.3.0
*/ */
public async awaitBackgroundInit(): Promise<void> { public async awaitBackgroundInit(): Promise<void> {
if (this.backgroundInitPromise) { if (this.backgroundInitPromise) {
@ -1361,7 +1348,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* *
* @returns `true` if all background tasks are complete * @returns `true` if all background tasks are complete
* @public * @public
* @since v7.3.0
*/ */
public isBackgroundInitComplete(): boolean { public isBackgroundInitComplete(): boolean {
return this.backgroundTasksComplete return this.backgroundTasksComplete
@ -1379,7 +1365,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
* *
* @throws Error if bucket validation fails * @throws Error if bucket validation fails
* @protected * @protected
* @since v7.3.0
*/ */
protected async ensureValidatedForWrite(): Promise<void> { protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do // If already validated, nothing to do

View file

@ -59,7 +59,7 @@ try {
* File system storage adapter for Node.js environments * File system storage adapter for Node.js environments
* Uses the file system to store data in the specified directory structure * Uses the file system to store data in the specified directory structure
* *
* v5.4.0: Type-aware storage now built into BaseStorage * Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation) * - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination) * - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths) * - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -91,12 +91,12 @@ export class FileSystemStorage extends BaseStorage {
private lockTimers: Map<string, NodeJS.Timeout> = new Map() // Track timers for cleanup private lockTimers: Map<string, NodeJS.Timeout> = new Map() // Track timers for cleanup
private allTimers: Set<NodeJS.Timeout> = new Set() // Track all timers for cleanup private allTimers: Set<NodeJS.Timeout> = new Set() // Track all timers for cleanup
// CRITICAL FIX (v4.10.1): Mutex locks for HNSW concurrency control // CRITICAL FIX: Mutex locks for HNSW concurrency control
// Prevents read-modify-write races during concurrent neighbor updates at scale (1000+ ops) // Prevents read-modify-write races during concurrent neighbor updates at scale (1000+ ops)
// Matches MemoryStorage and OPFSStorage behavior (tested in production) // Matches MemoryStorage and OPFSStorage behavior (tested in production)
private hnswLocks = new Map<string, Promise<void>>() private hnswLocks = new Map<string, Promise<void>>()
// Compression configuration (v4.0.0) // Compression configuration
private compressionEnabled: boolean = true // Enable gzip compression by default for 60-80% disk savings private compressionEnabled: boolean = true // Enable gzip compression by default for 60-80% disk savings
private compressionLevel: number = 6 // zlib compression level (1-9, default: 6 = balanced) private compressionLevel: number = 6 // zlib compression level (1-9, default: 6 = balanced)
@ -136,7 +136,6 @@ export class FileSystemStorage extends BaseStorage {
* - Parallel processing supported * - Parallel processing supported
* *
* @returns FileSystem-optimized batch configuration * @returns FileSystem-optimized batch configuration
* @since v4.11.0
*/ */
public getBatchConfig(): StorageBatchConfig { public getBatchConfig(): StorageBatchConfig {
return { return {
@ -179,7 +178,7 @@ export class FileSystemStorage extends BaseStorage {
try { try {
// Initialize directory paths now that path module is loaded // Initialize directory paths now that path module is loaded
// Clean directory structure (v4.7.2+) // Clean directory structure
this.nounsDir = path.join(this.rootDir, 'entities/nouns/hnsw') this.nounsDir = path.join(this.rootDir, 'entities/nouns/hnsw')
this.verbsDir = path.join(this.rootDir, 'entities/verbs/hnsw') this.verbsDir = path.join(this.rootDir, 'entities/verbs/hnsw')
this.metadataDir = path.join(this.rootDir, 'entities/nouns/metadata') // Legacy reference this.metadataDir = path.join(this.rootDir, 'entities/nouns/metadata') // Legacy reference
@ -245,7 +244,7 @@ export class FileSystemStorage extends BaseStorage {
// Always use fixed depth after migration/detection // Always use fixed depth after migration/detection
this.cachedShardingDepth = this.SHARDING_DEPTH this.cachedShardingDepth = this.SHARDING_DEPTH
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics // Initialize GraphAdjacencyIndex and type statistics
await super.init() await super.init()
} catch (error) { } catch (error) {
console.error('Error initializing FileSystemStorage:', error) console.error('Error initializing FileSystemStorage:', error)
@ -281,7 +280,7 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Save a node to storage * Save a node to storage
* CRITICAL FIX (v4.10.3): Added atomic write pattern to prevent file corruption during concurrent imports * CRITICAL FIX: Added atomic write pattern to prevent file corruption during concurrent imports
*/ */
protected async saveNode(node: HNSWNode): Promise<void> { protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
@ -303,7 +302,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${filePath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}` const tempPath = `${filePath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try { try {
// ATOMIC WRITE SEQUENCE (v4.10.3): // ATOMIC WRITE SEQUENCE:
// 1. Write to temp file // 1. Write to temp file
await this.ensureDirectoryExists(path.dirname(tempPath)) await this.ensureDirectoryExists(path.dirname(tempPath))
await fs.promises.writeFile(tempPath, JSON.stringify(serializableNode, null, 2)) await fs.promises.writeFile(tempPath, JSON.stringify(serializableNode, null, 2))
@ -320,7 +319,7 @@ export class FileSystemStorage extends BaseStorage {
throw error throw error
} }
// Count tracking happens in baseStorage.saveNounMetadata_internal (v4.1.2) // Count tracking happens in baseStorage.saveNounMetadata_internal
// This fixes the race condition where metadata didn't exist yet // This fixes the race condition where metadata didn't exist yet
} }
@ -361,7 +360,7 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Get all nodes from storage * Get all nodes from storage
* CRITICAL FIX (v3.43.2): Now scans sharded subdirectories (depth=1) * CRITICAL FIX: Now scans sharded subdirectories (depth=1)
* Previously only scanned flat directory, causing rebuild to find 0 entities * Previously only scanned flat directory, causing rebuild to find 0 entities
*/ */
protected async getAllNodes(): Promise<HNSWNode[]> { protected async getAllNodes(): Promise<HNSWNode[]> {
@ -406,7 +405,7 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Get nodes by noun type * Get nodes by noun type
* CRITICAL FIX (v3.43.2): Now scans sharded subdirectories (depth=1) * CRITICAL FIX: Now scans sharded subdirectories (depth=1)
* @param nounType The noun type to filter by * @param nounType The noun type to filter by
* @returns Promise that resolves to an array of nodes of the specified noun type * @returns Promise that resolves to an array of nodes of the specified noun type
*/ */
@ -462,7 +461,7 @@ export class FileSystemStorage extends BaseStorage {
const filePath = this.getNodePath(id) const filePath = this.getNodePath(id)
// Load metadata to get type for count update (v4.0.0: separate storage) // Load metadata to get type for count update (separate storage)
try { try {
const metadata = await this.getNounMetadata(id) const metadata = await this.getNounMetadata(id)
if (metadata) { if (metadata) {
@ -490,13 +489,13 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Save an edge to storage * Save an edge to storage
* CRITICAL FIX (v4.10.3): Added atomic write pattern to prevent file corruption during concurrent imports * CRITICAL FIX: Added atomic write pattern to prevent file corruption during concurrent imports
*/ */
protected async saveEdge(edge: Edge): Promise<void> { protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// Convert connections Map to a serializable format // Convert connections Map to a serializable format
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields in verb vector file // ARCHITECTURAL FIX: Include core relational fields in verb vector file
// These fields are essential for 90% of operations - no metadata lookup needed // These fields are essential for 90% of operations - no metadata lookup needed
const serializableEdge = { const serializableEdge = {
id: edge.id, id: edge.id,
@ -505,7 +504,7 @@ export class FileSystemStorage extends BaseStorage {
Array.from(set as Set<string>) Array.from(set as Set<string>)
), ),
// CORE RELATIONAL DATA (v3.50.1+) // CORE RELATIONAL DATA
verb: edge.verb, verb: edge.verb,
sourceId: edge.sourceId, sourceId: edge.sourceId,
targetId: edge.targetId, targetId: edge.targetId,
@ -518,7 +517,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${filePath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}` const tempPath = `${filePath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try { try {
// ATOMIC WRITE SEQUENCE (v4.10.3): // ATOMIC WRITE SEQUENCE:
// 1. Write to temp file // 1. Write to temp file
await this.ensureDirectoryExists(path.dirname(tempPath)) await this.ensureDirectoryExists(path.dirname(tempPath))
await fs.promises.writeFile(tempPath, JSON.stringify(serializableEdge, null, 2)) await fs.promises.writeFile(tempPath, JSON.stringify(serializableEdge, null, 2))
@ -535,7 +534,7 @@ export class FileSystemStorage extends BaseStorage {
throw error throw error
} }
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2) // Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet // This fixes the race condition where metadata didn't exist yet
} }
@ -556,7 +555,7 @@ export class FileSystemStorage extends BaseStorage {
connections.set(Number(level), new Set(nodeIds as string[])) connections.set(Number(level), new Set(nodeIds as string[]))
} }
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field) // Return HNSWVerb with core relational fields (NO metadata field)
return { return {
id: parsedEdge.id, id: parsedEdge.id,
vector: parsedEdge.vector, vector: parsedEdge.vector,
@ -567,7 +566,7 @@ export class FileSystemStorage extends BaseStorage {
sourceId: parsedEdge.sourceId, sourceId: parsedEdge.sourceId,
targetId: parsedEdge.targetId targetId: parsedEdge.targetId
// ✅ NO metadata field in v4.0.0 // ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata() // User metadata retrieved separately via getVerbMetadata()
} }
} catch (error: any) { } catch (error: any) {
@ -580,7 +579,7 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Get all edges from storage * Get all edges from storage
* CRITICAL FIX (v3.43.2): Now scans sharded subdirectories (depth=1) * CRITICAL FIX: Now scans sharded subdirectories (depth=1)
* Previously only scanned flat directory, causing rebuild to find 0 relationships * Previously only scanned flat directory, causing rebuild to find 0 relationships
*/ */
protected async getAllEdges(): Promise<Edge[]> { protected async getAllEdges(): Promise<Edge[]> {
@ -608,7 +607,7 @@ export class FileSystemStorage extends BaseStorage {
connections.set(Number(level), new Set(nodeIds as string[])) connections.set(Number(level), new Set(nodeIds as string[]))
} }
// v4.0.0: Include core relational fields (NO metadata field) // Include core relational fields (NO metadata field)
allEdges.push({ allEdges.push({
id: parsedEdge.id, id: parsedEdge.id,
vector: parsedEdge.vector, vector: parsedEdge.vector,
@ -619,7 +618,7 @@ export class FileSystemStorage extends BaseStorage {
sourceId: parsedEdge.sourceId, sourceId: parsedEdge.sourceId,
targetId: parsedEdge.targetId targetId: parsedEdge.targetId
// ✅ NO metadata field in v4.0.0 // ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata() // User metadata retrieved separately via getVerbMetadata()
}) })
} }
@ -696,8 +695,8 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Primitive operation: Write object to path * Primitive operation: Write object to path
* All metadata operations use this internally via base class routing * All metadata operations use this internally via base class routing
* v4.0.0: Supports gzip compression for 60-80% disk savings * Supports gzip compression for 60-80% disk savings
* CRITICAL FIX (v4.10.3): Added atomic write pattern to prevent file corruption during concurrent imports * CRITICAL FIX: Added atomic write pattern to prevent file corruption during concurrent imports
*/ */
protected async writeObjectToPath(pathStr: string, data: any): Promise<void> { protected async writeObjectToPath(pathStr: string, data: any): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
@ -711,7 +710,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${compressedPath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}` const tempPath = `${compressedPath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try { try {
// ATOMIC WRITE SEQUENCE (v4.10.3): // ATOMIC WRITE SEQUENCE:
// 1. Compress and write to temp file // 1. Compress and write to temp file
const jsonString = JSON.stringify(data, null, 2) const jsonString = JSON.stringify(data, null, 2)
const compressed = await new Promise<Buffer>((resolve, reject) => { const compressed = await new Promise<Buffer>((resolve, reject) => {
@ -748,7 +747,7 @@ export class FileSystemStorage extends BaseStorage {
const tempPath = `${fullPath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}` const tempPath = `${fullPath}.tmp.${Date.now()}.${Math.random().toString(36).substring(2)}`
try { try {
// ATOMIC WRITE SEQUENCE (v4.10.3): // ATOMIC WRITE SEQUENCE:
// 1. Write to temp file // 1. Write to temp file
await fs.promises.writeFile(tempPath, JSON.stringify(data, null, 2)) await fs.promises.writeFile(tempPath, JSON.stringify(data, null, 2))
@ -770,7 +769,7 @@ export class FileSystemStorage extends BaseStorage {
* Primitive operation: Read object from path * Primitive operation: Read object from path
* All metadata operations use this internally via base class routing * All metadata operations use this internally via base class routing
* Enhanced error handling for corrupted metadata files (Bug #3 mitigation) * Enhanced error handling for corrupted metadata files (Bug #3 mitigation)
* v4.0.0: Supports reading both compressed (.gz) and uncompressed files for backward compatibility * Supports reading both compressed (.gz) and uncompressed files for backward compatibility
*/ */
protected async readObjectFromPath(pathStr: string): Promise<any | null> { protected async readObjectFromPath(pathStr: string): Promise<any | null> {
await this.ensureInitialized() await this.ensureInitialized()
@ -822,7 +821,7 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Primitive operation: Delete object from path * Primitive operation: Delete object from path
* All metadata operations use this internally via base class routing * All metadata operations use this internally via base class routing
* v4.0.0: Deletes both compressed and uncompressed versions (for cleanup) * Deletes both compressed and uncompressed versions (for cleanup)
*/ */
protected async deleteObjectFromPath(pathStr: string): Promise<void> { protected async deleteObjectFromPath(pathStr: string): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
@ -863,7 +862,7 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Primitive operation: List objects under path prefix * Primitive operation: List objects under path prefix
* All metadata operations use this internally via base class routing * All metadata operations use this internally via base class routing
* v4.0.0: Handles both .json and .json.gz files, normalizes paths * Handles both .json and .json.gz files, normalizes paths
*/ */
protected async listObjectsUnderPath(prefix: string): Promise<string[]> { protected async listObjectsUnderPath(prefix: string): Promise<string[]> {
await this.ensureInitialized() await this.ensureInitialized()
@ -877,7 +876,7 @@ export class FileSystemStorage extends BaseStorage {
for (const entry of entries) { for (const entry of entries) {
if (entry.isFile()) { if (entry.isFile()) {
// v5.3.5: Handle multiple compression formats for broad compatibility // Handle multiple compression formats for broad compatibility
// - .json.gz: Standard entity/metadata files (JSON compressed) // - .json.gz: Standard entity/metadata files (JSON compressed)
// - .gz: COW files (refs, blobs, commits - raw compressed) // - .gz: COW files (refs, blobs, commits - raw compressed)
// - .json: Uncompressed JSON files // - .json: Uncompressed JSON files
@ -890,7 +889,7 @@ export class FileSystemStorage extends BaseStorage {
seen.add(normalizedPath) seen.add(normalizedPath)
} }
} else if (entry.name.endsWith('.gz')) { } else if (entry.name.endsWith('.gz')) {
// v5.3.5 fix: COW files stored as .gz (not .json.gz) // COW files stored as .gz (not .json.gz)
// Strip .gz extension and return path // Strip .gz extension and return path
const normalizedName = entry.name.slice(0, -3) // Remove .gz const normalizedName = entry.name.slice(0, -3) // Remove .gz
const normalizedPath = path.join(prefix, normalizedName) const normalizedPath = path.join(prefix, normalizedName)
@ -966,7 +965,7 @@ export class FileSystemStorage extends BaseStorage {
* Get nouns with pagination support * Get nouns with pagination support
* @param options Pagination options * @param options Pagination options
*/ */
// v5.4.0: Removed getNounsWithPagination override - now uses BaseStorage's type-first implementation // Removed getNounsWithPagination override - now uses BaseStorage's type-first implementation
/** /**
* Clear all data from storage * Clear all data from storage
@ -1002,7 +1001,7 @@ export class FileSystemStorage extends BaseStorage {
} }
} }
// v5.10.4: Clear the entire branches/ directory (branch-based storage) // Clear the entire branches/ directory (branch-based storage)
// Bug fix: Data is stored in branches/main/entities/, not just entities/ // Bug fix: Data is stored in branches/main/entities/, not just entities/
// The branch-based structure was introduced for COW support // The branch-based structure was introduced for COW support
const branchesDir = path.join(this.rootDir, 'branches') const branchesDir = path.join(this.rootDir, 'branches')
@ -1016,7 +1015,7 @@ export class FileSystemStorage extends BaseStorage {
await removeDirectoryContents(this.indexDir) await removeDirectoryContents(this.indexDir)
} }
// v5.6.1: Remove COW (copy-on-write) version control data // Remove COW (copy-on-write) version control data
// This directory stores all git-like versioning data (commits, trees, blobs, refs) // This directory stores all git-like versioning data (commits, trees, blobs, refs)
// Must be deleted to fully clear all data including version history // Must be deleted to fully clear all data including version history
const cowDir = path.join(this.rootDir, '_cow') const cowDir = path.join(this.rootDir, '_cow')
@ -1024,7 +1023,7 @@ export class FileSystemStorage extends BaseStorage {
// Delete the entire _cow/ directory (not just contents) // Delete the entire _cow/ directory (not just contents)
await fs.promises.rm(cowDir, { recursive: true, force: true }) await fs.promises.rm(cowDir, { recursive: true, force: true })
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled) // Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use // COW will re-initialize automatically on next use
this.refManager = undefined this.refManager = undefined
this.blobStorage = undefined this.blobStorage = undefined
@ -1035,12 +1034,12 @@ export class FileSystemStorage extends BaseStorage {
this.statisticsCache = null this.statisticsCache = null
this.statisticsModified = false this.statisticsModified = false
// v5.6.1: Reset entity counters (inherited from BaseStorageAdapter) // Reset entity counters (inherited from BaseStorageAdapter)
// These in-memory counters must be reset to 0 after clearing all data // These in-memory counters must be reset to 0 after clearing all data
;(this as any).totalNounCount = 0 ;(this as any).totalNounCount = 0
;(this as any).totalVerbCount = 0 ;(this as any).totalVerbCount = 0
// v7.3.1: Clear write-through cache (inherited from BaseStorage) // Clear write-through cache (inherited from BaseStorage)
// Without this, readWithInheritance() would return stale cached data // Without this, readWithInheritance() would return stale cached data
// after clear(), causing "ghost" entities to appear // after clear(), causing "ghost" entities to appear
this.clearWriteCache() this.clearWriteCache()
@ -1074,12 +1073,12 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Check if COW has been explicitly disabled via clear() * Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts * Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker file exists, false otherwise * @returns true if marker file exists, false otherwise
* @protected * @protected
*/ */
/** /**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods * Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used * COW is now always enabled - marker files are no longer used
*/ */
@ -1152,7 +1151,7 @@ export class FileSystemStorage extends BaseStorage {
totalSize = nounsDirSize + verbsDirSize + metadataDirSize + indexDirSize totalSize = nounsDirSize + verbsDirSize + metadataDirSize + indexDirSize
// CRITICAL FIX (v3.43.2): Use persisted counts instead of directory reads // CRITICAL FIX: Use persisted counts instead of directory reads
// This is O(1) instead of O(n), and handles sharded structure correctly // This is O(1) instead of O(n), and handles sharded structure correctly
const nounsCount = this.totalNounCount const nounsCount = this.totalNounCount
const verbsCount = this.totalVerbCount const verbsCount = this.totalVerbCount
@ -1210,8 +1209,8 @@ export class FileSystemStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation // Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed 2 pagination methods (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's implementation // Removed 2 pagination methods (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's implementation
/** /**
* Acquire a file-based lock for coordinating operations across multiple processes * Acquire a file-based lock for coordinating operations across multiple processes
@ -1503,14 +1502,14 @@ export class FileSystemStorage extends BaseStorage {
this.totalVerbCount = validVerbFiles.length this.totalVerbCount = validVerbFiles.length
// Sample some files to get type distribution (don't read all) // Sample some files to get type distribution (don't read all)
// v4.0.0: Load metadata separately for type information // Load metadata separately for type information
const sampleSize = Math.min(100, validNounFiles.length) const sampleSize = Math.min(100, validNounFiles.length)
for (let i = 0; i < sampleSize; i++) { for (let i = 0; i < sampleSize; i++) {
try { try {
const file = validNounFiles[i] const file = validNounFiles[i]
const id = file.replace('.json', '') const id = file.replace('.json', '')
// v4.0.0: Load metadata from separate storage for type info // Load metadata from separate storage for type info
const metadata = await this.getNounMetadata(id) const metadata = await this.getNounMetadata(id)
if (metadata) { if (metadata) {
const type = metadata.noun || 'default' const type = metadata.noun || 'default'
@ -2148,7 +2147,7 @@ export class FileSystemStorage extends BaseStorage {
const edge = JSON.parse(data) const edge = JSON.parse(data)
const metadata = await this.getVerbMetadata(id) const metadata = await this.getVerbMetadata(id)
// v4.8.1: Don't skip verbs without metadata - metadata is optional // Don't skip verbs without metadata - metadata is optional
// FIX: This was the root cause of the VFS bug (11 versions) // FIX: This was the root cause of the VFS bug (11 versions)
// Verbs can exist without metadata files (e.g., from imports/migrations) // Verbs can exist without metadata files (e.g., from imports/migrations)
@ -2162,7 +2161,7 @@ export class FileSystemStorage extends BaseStorage {
connections = connectionsMap connections = connectionsMap
} }
// v4.8.0: Extract standard fields from metadata to top-level // Extract standard fields from metadata to top-level
const metadataObj = (metadata || {}) as VerbMetadata const metadataObj = (metadata || {}) as VerbMetadata
const { createdAt, updatedAt, confidence, weight, service, data: dataField, createdBy, ...customMetadata } = metadataObj const { createdAt, updatedAt, confidence, weight, service, data: dataField, createdBy, ...customMetadata } = metadataObj
@ -2309,12 +2308,12 @@ export class FileSystemStorage extends BaseStorage {
} }
// ============================================= // =============================================
// HNSW Index Persistence (v3.35.0+) // HNSW Index Persistence
// ============================================= // =============================================
/** /**
* Get vector for a noun * Get vector for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths) * Uses BaseStorage's getNoun (type-first paths)
*/ */
public async getNounVector(id: string): Promise<number[] | null> { public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id) const noun = await this.getNoun(id)
@ -2324,7 +2323,7 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Save HNSW graph data for a noun * Save HNSW graph data for a noun
* *
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths) * Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Preserves mutex locking to prevent read-modify-write races * CRITICAL: Preserves mutex locking to prevent read-modify-write races
*/ */
public async saveHNSWData(nounId: string, hnswData: { public async saveHNSWData(nounId: string, hnswData: {
@ -2333,7 +2332,7 @@ export class FileSystemStorage extends BaseStorage {
}): Promise<void> { }): Promise<void> {
const lockKey = `hnsw/${nounId}` const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races // CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can: // Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3]) // 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3]) // 2. Thread B reads noun (connections: [1,2,3])
@ -2353,7 +2352,7 @@ export class FileSystemStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise) this.hnswLocks.set(lockKey, lockPromise)
try { try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths) // Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists) // Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId) const existingNoun = await this.getNoun(nounId)
@ -2375,7 +2374,7 @@ export class FileSystemStorage extends BaseStorage {
connections: connectionsMap connections: connectionsMap
} }
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write) // Use BaseStorage's saveNoun (type-first paths, atomic write)
await this.saveNoun(updatedNoun) await this.saveNoun(updatedNoun)
} finally { } finally {
// Release lock (ALWAYS runs, even if error thrown) // Release lock (ALWAYS runs, even if error thrown)
@ -2386,7 +2385,7 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Get HNSW graph data for a noun * Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths) * Uses BaseStorage's getNoun (type-first paths)
*/ */
public async getHNSWData(nounId: string): Promise<{ public async getHNSWData(nounId: string): Promise<{
level: number level: number
@ -2415,7 +2414,7 @@ export class FileSystemStorage extends BaseStorage {
/** /**
* Save HNSW system data (entry point, max level) * Save HNSW system data (entry point, max level)
* *
* CRITICAL FIX (v4.10.1): Mutex lock + atomic write to prevent race conditions * CRITICAL FIX: Mutex lock + atomic write to prevent race conditions
*/ */
public async saveHNSWSystem(systemData: { public async saveHNSWSystem(systemData: {
entryPointId: string | null entryPointId: string | null
@ -2425,7 +2424,7 @@ export class FileSystemStorage extends BaseStorage {
const lockKey = 'hnsw/system' const lockKey = 'hnsw/system'
// CRITICAL FIX (v4.10.1): Mutex lock to serialize system updates // CRITICAL FIX: Mutex lock to serialize system updates
// System data (entry point, max level) updated frequently during HNSW construction // System data (entry point, max level) updated frequently during HNSW construction
// Without mutex, concurrent updates can lose data (same as entity-level problem) // Without mutex, concurrent updates can lose data (same as entity-level problem)

View file

@ -65,7 +65,7 @@ const MAX_GCS_PAGE_SIZE = 5000
* 3. Service Account Credentials Object (if credentials provided) * 3. Service Account Credentials Object (if credentials provided)
* 4. HMAC Keys (if accessKeyId/secretAccessKey provided) * 4. HMAC Keys (if accessKeyId/secretAccessKey provided)
* *
* v5.4.0: Type-aware storage now built into BaseStorage * Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation) * - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination) * - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths) * - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -110,7 +110,7 @@ export class GcsStorage extends BaseStorage {
// Request coalescer for deduplication // Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance // Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching // Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Multi-level cache manager for efficient data access // Multi-level cache manager for efficient data access
@ -120,7 +120,7 @@ export class GcsStorage extends BaseStorage {
// Module logger // Module logger
private logger = createModuleLogger('GcsStorage') private logger = createModuleLogger('GcsStorage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races // HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>() private hnswLocks = new Map<string, Promise<void>>()
// Configuration options // Configuration options
@ -166,7 +166,7 @@ export class GcsStorage extends BaseStorage {
secretAccessKey?: string secretAccessKey?: string
/** /**
* Initialization mode for fast cold starts (v7.3.0+) * Initialization mode for fast cold starts
* *
* - `'auto'` (default): Progressive in cloud environments (Cloud Run, Lambda), * - `'auto'` (default): Progressive in cloud environments (Cloud Run, Lambda),
* strict locally. Zero-config optimization. * strict locally. Zero-config optimization.
@ -175,19 +175,18 @@ export class GcsStorage extends BaseStorage {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts * - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns. * before init() returns.
* *
* @since v7.3.0
*/ */
initMode?: InitMode initMode?: InitMode
/** /**
* @deprecated Use `initMode: 'progressive'` instead. * @deprecated Use `initMode: 'progressive'` instead.
* Will be removed in v8.0.0. * Will be removed in a future version.
*/ */
skipInitialScan?: boolean skipInitialScan?: boolean
/** /**
* @deprecated Use `initMode: 'progressive'` instead. * @deprecated Use `initMode: 'progressive'` instead.
* Will be removed in v8.0.0. * Will be removed in a future version.
*/ */
skipCountsFile?: boolean skipCountsFile?: boolean
@ -206,7 +205,7 @@ export class GcsStorage extends BaseStorage {
this.accessKeyId = options.accessKeyId this.accessKeyId = options.accessKeyId
this.secretAccessKey = options.secretAccessKey this.secretAccessKey = options.secretAccessKey
// v7.3.0: Handle initMode and deprecated skip* flags // Handle initMode and deprecated skip* flags
if (options.initMode) { if (options.initMode) {
this.initMode = options.initMode this.initMode = options.initMode
} }
@ -215,14 +214,14 @@ export class GcsStorage extends BaseStorage {
if (options.skipInitialScan) { if (options.skipInitialScan) {
console.warn( console.warn(
'[GcsStorage] DEPRECATION WARNING: skipInitialScan is deprecated. ' + '[GcsStorage] DEPRECATION WARNING: skipInitialScan is deprecated. ' +
'Use initMode: "progressive" instead. Will be removed in v8.0.0.' 'Use initMode: "progressive" instead. Will be removed in a future version.'
) )
this.skipInitialScan = true this.skipInitialScan = true
} }
if (options.skipCountsFile) { if (options.skipCountsFile) {
console.warn( console.warn(
'[GcsStorage] DEPRECATION WARNING: skipCountsFile is deprecated. ' + '[GcsStorage] DEPRECATION WARNING: skipCountsFile is deprecated. ' +
'Use initMode: "progressive" instead. Will be removed in v8.0.0.' 'Use initMode: "progressive" instead. Will be removed in a future version.'
) )
this.skipCountsFile = true this.skipCountsFile = true
} }
@ -240,13 +239,13 @@ export class GcsStorage extends BaseStorage {
this.nounCacheManager = new CacheManager<HNSWNode>(options.cacheConfig) this.nounCacheManager = new CacheManager<HNSWNode>(options.cacheConfig)
this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig) this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig)
// v6.2.7: Write buffering always enabled - no env var check needed // Write buffering always enabled - no env var check needed
} }
/** /**
* Initialize the storage adapter * Initialize the storage adapter
* *
* v7.3.0: Supports progressive initialization for fast cold starts * Supports progressive initialization for fast cold starts
* *
* | Mode | Init Time | When | * | Mode | Init Time | When |
* |------|-----------|------| * |------|-----------|------|
@ -337,14 +336,14 @@ export class GcsStorage extends BaseStorage {
} }
) )
// CRITICAL FIX (v3.37.7): Clear any stale cache entries from previous runs // CRITICAL FIX: Clear any stale cache entries from previous runs
// This prevents cache poisoning from causing silent failures on container restart // This prevents cache poisoning from causing silent failures on container restart
prodLog.info('🧹 Clearing cache from previous run to prevent cache poisoning') prodLog.info('🧹 Clearing cache from previous run to prevent cache poisoning')
this.nounCacheManager.clear() this.nounCacheManager.clear()
this.verbCacheManager.clear() this.verbCacheManager.clear()
prodLog.info('✅ Cache cleared - starting fresh') prodLog.info('✅ Cache cleared - starting fresh')
// v7.3.0: Progressive vs Strict initialization // Progressive vs Strict initialization
if (effectiveMode === 'progressive') { if (effectiveMode === 'progressive') {
// PROGRESSIVE MODE: Fast init, background validation // PROGRESSIVE MODE: Fast init, background validation
// Mark as initialized immediately - ready to accept operations // Mark as initialized immediately - ready to accept operations
@ -353,7 +352,7 @@ export class GcsStorage extends BaseStorage {
prodLog.info(`✅ GCS progressive init complete: ${this.bucketName} (validation deferred)`) prodLog.info(`✅ GCS progressive init complete: ${this.bucketName} (validation deferred)`)
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics // Initialize GraphAdjacencyIndex and type statistics
await super.init() await super.init()
// Schedule background tasks (non-blocking) // Schedule background tasks (non-blocking)
@ -373,7 +372,7 @@ export class GcsStorage extends BaseStorage {
await this.initializeCounts() await this.initializeCounts()
this.countsLoaded = true this.countsLoaded = true
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics // Initialize GraphAdjacencyIndex and type statistics
await super.init() await super.init()
// Mark background tasks as complete (nothing to do in background) // Mark background tasks as complete (nothing to do in background)
@ -386,7 +385,7 @@ export class GcsStorage extends BaseStorage {
} }
// ============================================= // =============================================
// Progressive Initialization (v7.3.0+) // Progressive Initialization
// ============================================= // =============================================
/** /**
@ -400,7 +399,6 @@ export class GcsStorage extends BaseStorage {
* *
* @protected * @protected
* @override * @override
* @since v7.3.0
*/ */
protected async runBackgroundInit(): Promise<void> { protected async runBackgroundInit(): Promise<void> {
const startTime = Date.now() const startTime = Date.now()
@ -423,7 +421,6 @@ export class GcsStorage extends BaseStorage {
* Stores result in bucketValidated/bucketValidationError for lazy use. * Stores result in bucketValidated/bucketValidationError for lazy use.
* *
* @private * @private
* @since v7.3.0
*/ */
private async validateBucketInBackground(): Promise<void> { private async validateBucketInBackground(): Promise<void> {
try { try {
@ -451,7 +448,6 @@ export class GcsStorage extends BaseStorage {
* Uses the existing initializeCounts() logic but in background. * Uses the existing initializeCounts() logic but in background.
* *
* @private * @private
* @since v7.3.0
*/ */
private async loadCountsInBackground(): Promise<void> { private async loadCountsInBackground(): Promise<void> {
try { try {
@ -475,7 +471,6 @@ export class GcsStorage extends BaseStorage {
* @throws Error if bucket does not exist or is not accessible * @throws Error if bucket does not exist or is not accessible
* @protected * @protected
* @override * @override
* @since v7.3.0
*/ */
protected async ensureValidatedForWrite(): Promise<void> { protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do // If already validated, nothing to do
@ -562,7 +557,7 @@ export class GcsStorage extends BaseStorage {
} }
/** /**
* Override base class to enable smart batching for cloud storage (v3.32.3+) * Override base class to enable smart batching for cloud storage
* *
* GCS is cloud storage with network latency (~50ms per write). * GCS is cloud storage with network latency (~50ms per write).
* Smart batching reduces writes from 1000 ops 100 batches. * Smart batching reduces writes from 1000 ops 100 batches.
@ -596,7 +591,7 @@ export class GcsStorage extends BaseStorage {
} }
} }
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage // Removed checkVolumeMode() - write buffering always enabled for cloud storage
/** /**
* Flush noun buffer to GCS * Flush noun buffer to GCS
@ -628,20 +623,20 @@ export class GcsStorage extends BaseStorage {
await Promise.all(writes) await Promise.all(writes)
} }
// v5.4.0: Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation // Removed saveNoun_internal - now inherit from BaseStorage's type-first implementation
/** /**
* Save a node to storage * Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance * Always uses write buffer for consistent performance
*/ */
protected async saveNode(node: HNSWNode): Promise<void> { protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching // Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) { if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`) this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency // Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) { if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node) this.nounCacheManager.set(node.id, node)
} }
@ -657,10 +652,10 @@ export class GcsStorage extends BaseStorage {
/** /**
* Save a node directly to GCS (bypass buffer) * Save a node directly to GCS (bypass buffer)
* *
* v7.3.0: Performs lazy bucket validation on first write in progressive mode. * Performs lazy bucket validation on first write in progressive mode.
*/ */
private async saveNodeDirect(node: HNSWNode): Promise<void> { private async saveNodeDirect(node: HNSWNode): Promise<void> {
// v7.3.0: Lazy bucket validation for progressive init // Lazy bucket validation for progressive init
await this.ensureValidatedForWrite() await this.ensureValidatedForWrite()
// Apply backpressure before starting operation // Apply backpressure before starting operation
@ -695,14 +690,14 @@ export class GcsStorage extends BaseStorage {
resumable: false // For small objects, non-resumable is faster resumable: false // For small objects, non-resumable is faster
}) })
// CRITICAL FIX (v3.37.8): Only cache nodes with non-empty vectors // CRITICAL FIX: Only cache nodes with non-empty vectors
// This prevents cache pollution from HNSW's lazy-loading nodes (vector: []) // This prevents cache pollution from HNSW's lazy-loading nodes (vector: [])
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) { if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node) this.nounCacheManager.set(node.id, node)
} }
// Note: Empty vectors are intentional during HNSW lazy mode - not logged // Note: Empty vectors are intentional during HNSW lazy mode - not logged
// Count tracking happens in baseStorage.saveNounMetadata_internal (v4.1.2) // Count tracking happens in baseStorage.saveNounMetadata_internal
// This fixes the race condition where metadata didn't exist yet // This fixes the race condition where metadata didn't exist yet
this.logger.trace(`Node ${node.id} saved successfully`) this.logger.trace(`Node ${node.id} saved successfully`)
@ -721,7 +716,7 @@ export class GcsStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed getNoun_internal - now inherit from BaseStorage's type-first implementation // Removed getNoun_internal - now inherit from BaseStorage's type-first implementation
/** /**
* Get a node from storage * Get a node from storage
@ -732,7 +727,7 @@ export class GcsStorage extends BaseStorage {
// Check cache first // Check cache first
const cached: HNSWNode | null = await this.nounCacheManager.get(id) const cached: HNSWNode | null = await this.nounCacheManager.get(id)
// Validate cached object before returning (v3.37.8+) // Validate cached object before returning
if (cached !== undefined && cached !== null) { if (cached !== undefined && cached !== null) {
// Validate cached object has required fields (including non-empty vector!) // Validate cached object has required fields (including non-empty vector!)
if (!cached.id || !cached.vector || !Array.isArray(cached.vector) || cached.vector.length === 0) { if (!cached.id || !cached.vector || !Array.isArray(cached.vector) || cached.vector.length === 0) {
@ -831,18 +826,18 @@ export class GcsStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation // Removed deleteNoun_internal - now inherit from BaseStorage's type-first implementation
/** /**
* Write an object to a specific path in GCS * Write an object to a specific path in GCS
* Primitive operation required by base class * Primitive operation required by base class
* *
* v7.3.0: Performs lazy bucket validation on first write in progressive mode. * Performs lazy bucket validation on first write in progressive mode.
* @protected * @protected
*/ */
protected async writeObjectToPath(path: string, data: any): Promise<void> { protected async writeObjectToPath(path: string, data: any): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init // Lazy bucket validation for progressive init
await this.ensureValidatedForWrite() await this.ensureValidatedForWrite()
try { try {
@ -892,7 +887,7 @@ export class GcsStorage extends BaseStorage {
} }
/** /**
* Batch read multiple objects from GCS (v5.12.0 - Cloud Storage Optimization) * Batch read multiple objects from GCS
* *
* **Performance**: GCS-optimized parallel downloads * **Performance**: GCS-optimized parallel downloads
* - Uses Promise.all() for concurrent requests * - Uses Promise.all() for concurrent requests
@ -908,7 +903,6 @@ export class GcsStorage extends BaseStorage {
* @returns Map of path data (only successful reads included) * @returns Map of path data (only successful reads included)
* *
* @public - Called by baseStorage.readBatchFromAdapter() * @public - Called by baseStorage.readBatchFromAdapter()
* @since v5.12.0
*/ */
public async readBatch(paths: string[]): Promise<Map<string, any>> { public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized() await this.ensureInitialized()
@ -960,12 +954,11 @@ export class GcsStorage extends BaseStorage {
} }
/** /**
* Get GCS-specific batch configuration (v5.12.0) * Get GCS-specific batch configuration
* *
* GCS performs well with high concurrency due to HTTP/2 multiplexing * GCS performs well with high concurrency due to HTTP/2 multiplexing
* *
* @public - Overrides BaseStorage.getBatchConfig() * @public - Overrides BaseStorage.getBatchConfig()
* @since v5.12.0
*/ */
public getBatchConfig(): StorageBatchConfig { public getBatchConfig(): StorageBatchConfig {
return { return {
@ -984,12 +977,12 @@ export class GcsStorage extends BaseStorage {
* Delete an object from a specific path in GCS * Delete an object from a specific path in GCS
* Primitive operation required by base class * Primitive operation required by base class
* *
* v7.3.0: Performs lazy bucket validation on first delete in progressive mode. * Performs lazy bucket validation on first delete in progressive mode.
* @protected * @protected
*/ */
protected async deleteObjectFromPath(path: string): Promise<void> { protected async deleteObjectFromPath(path: string): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init // Lazy bucket validation for progressive init
await this.ensureValidatedForWrite() await this.ensureValidatedForWrite()
try { try {
@ -1034,20 +1027,20 @@ export class GcsStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation // Removed saveVerb_internal - now inherit from BaseStorage's type-first implementation
/** /**
* Save an edge to storage * Save an edge to storage
* v6.2.7: Always uses write buffer for consistent performance * Always uses write buffer for consistent performance
*/ */
protected async saveEdge(edge: Edge): Promise<void> { protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching // Always use write buffer - cloud storage benefits from batching
if (this.verbWriteBuffer) { if (this.verbWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding verb ${edge.id} to write buffer`) this.logger.trace(`📝 BUFFERING: Adding verb ${edge.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency // Populate cache BEFORE buffering for read-after-write consistency
this.verbCacheManager.set(edge.id, edge) this.verbCacheManager.set(edge.id, edge)
await this.verbWriteBuffer.add(edge.id, edge) await this.verbWriteBuffer.add(edge.id, edge)
@ -1061,10 +1054,10 @@ export class GcsStorage extends BaseStorage {
/** /**
* Save an edge directly to GCS (bypass buffer) * Save an edge directly to GCS (bypass buffer)
* *
* v7.3.0: Performs lazy bucket validation on first write in progressive mode. * Performs lazy bucket validation on first write in progressive mode.
*/ */
private async saveEdgeDirect(edge: Edge): Promise<void> { private async saveEdgeDirect(edge: Edge): Promise<void> {
// v7.3.0: Lazy bucket validation for progressive init // Lazy bucket validation for progressive init
await this.ensureValidatedForWrite() await this.ensureValidatedForWrite()
const requestId = await this.applyBackpressure() const requestId = await this.applyBackpressure()
@ -1073,7 +1066,7 @@ export class GcsStorage extends BaseStorage {
this.logger.trace(`Saving edge ${edge.id}`) this.logger.trace(`Saving edge ${edge.id}`)
// Convert connections Map to serializable format // Convert connections Map to serializable format
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields in verb vector file // ARCHITECTURAL FIX: Include core relational fields in verb vector file
// These fields are essential for 90% of operations - no metadata lookup needed // These fields are essential for 90% of operations - no metadata lookup needed
const serializableEdge = { const serializableEdge = {
id: edge.id, id: edge.id,
@ -1085,7 +1078,7 @@ export class GcsStorage extends BaseStorage {
]) ])
), ),
// CORE RELATIONAL DATA (v3.50.1+) // CORE RELATIONAL DATA
verb: edge.verb, verb: edge.verb,
sourceId: edge.sourceId, sourceId: edge.sourceId,
targetId: edge.targetId, targetId: edge.targetId,
@ -1107,7 +1100,7 @@ export class GcsStorage extends BaseStorage {
// Update cache // Update cache
this.verbCacheManager.set(edge.id, edge) this.verbCacheManager.set(edge.id, edge)
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2) // Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet // This fixes the race condition where metadata didn't exist yet
this.logger.trace(`Edge ${edge.id} saved successfully`) this.logger.trace(`Edge ${edge.id} saved successfully`)
@ -1125,7 +1118,7 @@ export class GcsStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed getVerb_internal - now inherit from BaseStorage's type-first implementation // Removed getVerb_internal - now inherit from BaseStorage's type-first implementation
/** /**
* Get an edge from storage * Get an edge from storage
@ -1161,7 +1154,7 @@ export class GcsStorage extends BaseStorage {
connections.set(Number(level), new Set(verbIds as string[])) connections.set(Number(level), new Set(verbIds as string[]))
} }
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field) // Return HNSWVerb with core relational fields (NO metadata field)
const edge: Edge = { const edge: Edge = {
id: data.id, id: data.id,
vector: data.vector, vector: data.vector,
@ -1172,7 +1165,7 @@ export class GcsStorage extends BaseStorage {
sourceId: data.sourceId, sourceId: data.sourceId,
targetId: data.targetId targetId: data.targetId
// ✅ NO metadata field in v4.0.0 // ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata() // User metadata retrieved separately via getVerbMetadata()
} }
@ -1201,13 +1194,13 @@ export class GcsStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation // Removed deleteVerb_internal - now inherit from BaseStorage's type-first implementation
// v5.4.0: Removed pagination overrides - use BaseStorage's type-first implementation // Removed pagination overrides - use BaseStorage's type-first implementation
// - getNounsWithPagination, getNodesWithPagination, getVerbsWithPagination // - getNounsWithPagination, getNodesWithPagination, getVerbsWithPagination
// - getNouns, getVerbs (public wrappers) // - getNouns, getVerbs (public wrappers)
// v5.4.0: Removed 4 query *_internal methods - now inherit from BaseStorage's type-first implementation // Removed 4 query *_internal methods - now inherit from BaseStorage's type-first implementation
// (getNounsByNounType_internal, getVerbsBySource_internal, getVerbsByTarget_internal, getVerbsByType_internal) // (getNounsByNounType_internal, getVerbsBySource_internal, getVerbsByTarget_internal, getVerbsByType_internal)
/** /**
@ -1280,7 +1273,7 @@ export class GcsStorage extends BaseStorage {
} }
} }
// v5.11.0: Clear ALL data using correct paths // Clear ALL data using correct paths
// Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks) // Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks)
await deleteObjectsWithPrefix('branches/') await deleteObjectsWithPrefix('branches/')
@ -1290,7 +1283,7 @@ export class GcsStorage extends BaseStorage {
// Delete system metadata // Delete system metadata
await deleteObjectsWithPrefix('_system/') await deleteObjectsWithPrefix('_system/')
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled) // Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use // COW will re-initialize automatically on next use
this.refManager = undefined this.refManager = undefined
this.blobStorage = undefined this.blobStorage = undefined
@ -1352,12 +1345,12 @@ export class GcsStorage extends BaseStorage {
/** /**
* Check if COW has been explicitly disabled via clear() * Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts * Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker object exists, false otherwise * @returns true if marker object exists, false otherwise
* @protected * @protected
*/ */
/** /**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods * Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used * COW is now always enabled - marker files are no longer used
*/ */
@ -1413,7 +1406,7 @@ export class GcsStorage extends BaseStorage {
} }
} catch (error: any) { } catch (error: any) {
if (error.code === 404) { if (error.code === 404) {
// CRITICAL FIX (v3.37.4): Statistics file doesn't exist yet (first restart) // CRITICAL FIX: Statistics file doesn't exist yet (first restart)
// Return minimal stats with counts instead of null // Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild // This prevents HNSW from seeing entityCount=0 during index rebuild
this.logger.trace('Statistics file not found - returning minimal stats with counts') this.logger.trace('Statistics file not found - returning minimal stats with counts')
@ -1607,11 +1600,11 @@ export class GcsStorage extends BaseStorage {
} }
} }
// HNSW Index Persistence (v3.35.0+) // HNSW Index Persistence
/** /**
* Get a noun's vector for HNSW rebuild * Get a noun's vector for HNSW rebuild
* v5.4.0: Uses BaseStorage's getNoun (type-first paths) * Uses BaseStorage's getNoun (type-first paths)
*/ */
public async getNounVector(id: string): Promise<number[] | null> { public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id) const noun = await this.getNoun(id)
@ -1621,7 +1614,7 @@ export class GcsStorage extends BaseStorage {
/** /**
* Save HNSW graph data for a noun * Save HNSW graph data for a noun
* *
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths) * Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Uses mutex locking to prevent read-modify-write races * CRITICAL: Uses mutex locking to prevent read-modify-write races
*/ */
public async saveHNSWData(nounId: string, hnswData: { public async saveHNSWData(nounId: string, hnswData: {
@ -1630,7 +1623,7 @@ export class GcsStorage extends BaseStorage {
}): Promise<void> { }): Promise<void> {
const lockKey = `hnsw/${nounId}` const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races // CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can: // Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3]) // 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3]) // 2. Thread B reads noun (connections: [1,2,3])
@ -1650,7 +1643,7 @@ export class GcsStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise) this.hnswLocks.set(lockKey, lockPromise)
try { try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths) // Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists) // Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId) const existingNoun = await this.getNoun(nounId)
@ -1672,7 +1665,7 @@ export class GcsStorage extends BaseStorage {
connections: connectionsMap connections: connectionsMap
} }
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch) // Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
await this.saveNoun(updatedNoun) await this.saveNoun(updatedNoun)
} finally { } finally {
// Release lock (ALWAYS runs, even if error thrown) // Release lock (ALWAYS runs, even if error thrown)
@ -1683,7 +1676,7 @@ export class GcsStorage extends BaseStorage {
/** /**
* Get HNSW graph data for a noun * Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths) * Uses BaseStorage's getNoun (type-first paths)
*/ */
public async getHNSWData(nounId: string): Promise<{ public async getHNSWData(nounId: string): Promise<{
level: number level: number
@ -1713,7 +1706,7 @@ export class GcsStorage extends BaseStorage {
* Save HNSW system data (entry point, max level) * Save HNSW system data (entry point, max level)
* Storage path: system/hnsw-system.json * Storage path: system/hnsw-system.json
* *
* CRITICAL FIX (v4.10.1): Optimistic locking with generation numbers to prevent race conditions * CRITICAL FIX: Optimistic locking with generation numbers to prevent race conditions
*/ */
public async saveHNSWSystem(systemData: { public async saveHNSWSystem(systemData: {
entryPointId: string | null entryPointId: string | null
@ -1807,7 +1800,7 @@ export class GcsStorage extends BaseStorage {
} }
// ============================================================================ // ============================================================================
// GCS Lifecycle Management & Autoclass (v4.0.0) // GCS Lifecycle Management & Autoclass
// Cost optimization through automatic tier transitions and Autoclass // Cost optimization through automatic tier transitions and Autoclass
// ============================================================================ // ============================================================================

View file

@ -25,7 +25,7 @@
* - Lazy loading: only loads entities when accessed * - Lazy loading: only loads entities when accessed
* - No eager-loading of entire commit state * - No eager-loading of entire commit state
* *
* v5.4.0: Production-ready, billion-scale historical queries * Production-ready, billion-scale historical queries
*/ */
import { BaseStorage } from '../baseStorage.js' import { BaseStorage } from '../baseStorage.js'
@ -144,7 +144,7 @@ export class HistoricalStorageAdapter extends BaseStorage {
*/ */
public async init(): Promise<void> { public async init(): Promise<void> {
// Get COW components from underlying storage // Get COW components from underlying storage
// v6.2.4: Fixed property names - use public properties without underscore prefix // Fixed property names - use public properties without underscore prefix
this.commitLog = this.underlyingStorage.commitLog this.commitLog = this.underlyingStorage.commitLog
this.blobStorage = this.underlyingStorage.blobStorage this.blobStorage = this.underlyingStorage.blobStorage
@ -161,7 +161,7 @@ export class HistoricalStorageAdapter extends BaseStorage {
throw new Error(`Commit not found: ${this.commitId}`) throw new Error(`Commit not found: ${this.commitId}`)
} }
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics // Initialize GraphAdjacencyIndex and type statistics
await super.init() await super.init()
} }
@ -310,12 +310,12 @@ export class HistoricalStorageAdapter extends BaseStorage {
/** /**
* Check if COW has been explicitly disabled via clear() * Check if COW has been explicitly disabled via clear()
* v5.10.4: No-op for HistoricalStorageAdapter (read-only, doesn't manage COW) * No-op for HistoricalStorageAdapter (read-only, doesn't manage COW)
* @returns Always false (read-only adapter doesn't manage COW state) * @returns Always false (read-only adapter doesn't manage COW state)
* @protected * @protected
*/ */
/** /**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods * Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used * COW is now always enabled - marker files are no longer used
*/ */

View file

@ -24,7 +24,7 @@ import { PaginatedResult } from '../../types/paginationTypes.js'
* Uses Maps to store data in memory * Uses Maps to store data in memory
*/ */
export class MemoryStorage extends BaseStorage { export class MemoryStorage extends BaseStorage {
// v5.4.0: Removed redundant Maps (nouns, verbs) - objectStore handles all storage via type-first paths // Removed redundant Maps (nouns, verbs) - objectStore handles all storage via type-first paths
private statistics: StatisticsData | null = null private statistics: StatisticsData | null = null
// Unified object store for primitive operations (replaces metadata, nounMetadata, verbMetadata) // Unified object store for primitive operations (replaces metadata, nounMetadata, verbMetadata)
@ -55,7 +55,6 @@ export class MemoryStorage extends BaseStorage {
* - Parallel processing maximizes throughput * - Parallel processing maximizes throughput
* *
* @returns Memory-optimized batch configuration * @returns Memory-optimized batch configuration
* @since v4.11.0
*/ */
public getBatchConfig(): StorageBatchConfig { public getBatchConfig(): StorageBatchConfig {
return { return {
@ -72,27 +71,27 @@ export class MemoryStorage extends BaseStorage {
/** /**
* Initialize the storage adapter * Initialize the storage adapter
* v6.0.0: Calls super.init() to initialize GraphAdjacencyIndex and type statistics * Calls super.init() to initialize GraphAdjacencyIndex and type statistics
*/ */
public async init(): Promise<void> { public async init(): Promise<void> {
await super.init() await super.init()
} }
// v5.4.0: Removed saveNoun_internal and getNoun_internal - using BaseStorage's type-first implementation // Removed saveNoun_internal and getNoun_internal - using BaseStorage's type-first implementation
/** /**
* Get nouns with pagination and filtering * Get nouns with pagination and filtering
* v4.0.0: Returns HNSWNounWithMetadata[] (includes metadata field) * Returns HNSWNounWithMetadata[] (includes metadata field)
* @param options Pagination and filtering options * @param options Pagination and filtering options
* @returns Promise that resolves to a paginated result of nouns with metadata * @returns Promise that resolves to a paginated result of nouns with metadata
*/ */
// v5.4.0: Removed public method overrides (getNouns, getNounsWithPagination, getVerbs) - using BaseStorage's type-first implementation // Removed public method overrides (getNouns, getNounsWithPagination, getVerbs) - using BaseStorage's type-first implementation
// v5.4.0: Removed getNounsByNounType_internal and deleteNoun_internal - using BaseStorage's type-first implementation // Removed getNounsByNounType_internal and deleteNoun_internal - using BaseStorage's type-first implementation
// v5.4.0: Removed saveVerb_internal and getVerb_internal - using BaseStorage's type-first implementation // Removed saveVerb_internal and getVerb_internal - using BaseStorage's type-first implementation
// v5.4.0: Removed verb *_internal method overrides - using BaseStorage's type-first implementation // Removed verb *_internal method overrides - using BaseStorage's type-first implementation
/** /**
* Primitive operation: Write object to path * Primitive operation: Write object to path
@ -159,8 +158,8 @@ export class MemoryStorage extends BaseStorage {
/** /**
* Clear all data from storage * Clear all data from storage
* v5.4.0: Clears objectStore (type-first paths) * Clears objectStore (type-first paths)
* v7.3.1: Also clears writeCache to prevent stale data after clear * Also clears writeCache to prevent stale data after clear
*/ */
public async clear(): Promise<void> { public async clear(): Promise<void> {
this.objectStore.clear() this.objectStore.clear()
@ -174,7 +173,7 @@ export class MemoryStorage extends BaseStorage {
this.statisticsCache = null this.statisticsCache = null
this.statisticsModified = false this.statisticsModified = false
// v7.3.1: Clear write-through cache (inherited from BaseStorage) // Clear write-through cache (inherited from BaseStorage)
// Without this, readWithInheritance() would return stale cached data // Without this, readWithInheritance() would return stale cached data
// after clear(), causing "ghost" entities to appear // after clear(), causing "ghost" entities to appear
this.clearWriteCache() this.clearWriteCache()
@ -182,7 +181,7 @@ export class MemoryStorage extends BaseStorage {
/** /**
* Get information about storage usage and capacity * Get information about storage usage and capacity
* v5.4.0: Uses BaseStorage counts * Uses BaseStorage counts
*/ */
public async getStorageStatus(): Promise<{ public async getStorageStatus(): Promise<{
type: string type: string
@ -204,12 +203,12 @@ export class MemoryStorage extends BaseStorage {
/** /**
* Check if COW has been explicitly disabled via clear() * Check if COW has been explicitly disabled via clear()
* v5.10.4: No-op for MemoryStorage (doesn't persist) * No-op for MemoryStorage (doesn't persist)
* @returns Always false (marker doesn't persist in memory) * @returns Always false (marker doesn't persist in memory)
* @protected * @protected
*/ */
/** /**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods * Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used * COW is now always enabled - marker files are no longer used
*/ */
@ -252,7 +251,7 @@ export class MemoryStorage extends BaseStorage {
*/ */
protected async getStatisticsData(): Promise<StatisticsData | null> { protected async getStatisticsData(): Promise<StatisticsData | null> {
if (!this.statistics) { if (!this.statistics) {
// CRITICAL FIX (v3.37.4): Statistics don't exist yet (first init) // CRITICAL FIX: Statistics don't exist yet (first init)
// Return minimal stats with counts instead of null // Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild // This prevents HNSW from seeing entityCount=0 during index rebuild
return { return {
@ -299,10 +298,10 @@ export class MemoryStorage extends BaseStorage {
} }
/** /**
* Initialize counts from in-memory storage - O(1) operation (v4.0.0) * Initialize counts from in-memory storage - O(1) operation
*/ */
protected async initializeCounts(): Promise<void> { protected async initializeCounts(): Promise<void> {
// v6.0.0: Scan objectStore paths (ID-first structure) to count entities // Scan objectStore paths (ID-first structure) to count entities
this.entityCounts.clear() this.entityCounts.clear()
this.verbCounts.clear() this.verbCounts.clear()
@ -314,14 +313,14 @@ export class MemoryStorage extends BaseStorage {
// Count nouns (entities/nouns/{shard}/{id}/vectors.json) // Count nouns (entities/nouns/{shard}/{id}/vectors.json)
const nounMatch = path.match(/^entities\/nouns\/[0-9a-f]{2}\/[^/]+\/vectors\.json$/) const nounMatch = path.match(/^entities\/nouns\/[0-9a-f]{2}\/[^/]+\/vectors\.json$/)
if (nounMatch) { if (nounMatch) {
// v6.0.0: Type is in metadata, not path - just count total // Type is in metadata, not path - just count total
totalNouns++ totalNouns++
} }
// Count verbs (entities/verbs/{shard}/{id}/vectors.json) // Count verbs (entities/verbs/{shard}/{id}/vectors.json)
const verbMatch = path.match(/^entities\/verbs\/[0-9a-f]{2}\/[^/]+\/vectors\.json$/) const verbMatch = path.match(/^entities\/verbs\/[0-9a-f]{2}\/[^/]+\/vectors\.json$/)
if (verbMatch) { if (verbMatch) {
// v6.0.0: Type is in metadata, not path - just count total // Type is in metadata, not path - just count total
totalVerbs++ totalVerbs++
} }
} }
@ -339,26 +338,26 @@ export class MemoryStorage extends BaseStorage {
} }
// ============================================= // =============================================
// HNSW Index Persistence (v3.35.0+) // HNSW Index Persistence
// ============================================= // =============================================
/** /**
* Get vector for a noun * Get vector for a noun
* v5.4.0: Uses BaseStorage's type-first implementation * Uses BaseStorage's type-first implementation
*/ */
public async getNounVector(id: string): Promise<number[] | null> { public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id) const noun = await this.getNoun(id)
return noun ? [...noun.vector] : null return noun ? [...noun.vector] : null
} }
// CRITICAL FIX (v4.10.1): Mutex locks for HNSW concurrency control // CRITICAL FIX: Mutex locks for HNSW concurrency control
// Even in-memory operations need serialization to prevent async race conditions // Even in-memory operations need serialization to prevent async race conditions
private hnswLocks = new Map<string, Promise<void>>() private hnswLocks = new Map<string, Promise<void>>()
/** /**
* Save HNSW graph data for a noun * Save HNSW graph data for a noun
* *
* CRITICAL FIX (v4.10.1): Mutex locking to prevent race conditions during concurrent HNSW updates * CRITICAL FIX: Mutex locking to prevent race conditions during concurrent HNSW updates
* Even in-memory operations can race due to async/await interleaving * Even in-memory operations can race due to async/await interleaving
* Prevents data corruption when multiple entities connect to same neighbor simultaneously * Prevents data corruption when multiple entities connect to same neighbor simultaneously
*/ */
@ -417,7 +416,7 @@ export class MemoryStorage extends BaseStorage {
/** /**
* Save HNSW system data (entry point, max level) * Save HNSW system data (entry point, max level)
* *
* CRITICAL FIX (v4.10.1): Mutex locking to prevent race conditions * CRITICAL FIX: Mutex locking to prevent race conditions
*/ */
public async saveHNSWSystem(systemData: { public async saveHNSWSystem(systemData: {
entryPointId: string | null entryPointId: string | null

View file

@ -57,7 +57,7 @@ const ROOT_DIR = 'opfs-vector-db'
* OPFS storage adapter for browser environments * OPFS storage adapter for browser environments
* Uses the Origin Private File System API to store data persistently * Uses the Origin Private File System API to store data persistently
* *
* v5.4.0: Type-aware storage now built into BaseStorage * Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation) * - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination) * - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths) * - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -97,7 +97,6 @@ export class OPFSStorage extends BaseStorage {
* - Sequential processing preferred for stability * - Sequential processing preferred for stability
* *
* @returns OPFS-optimized batch configuration * @returns OPFS-optimized batch configuration
* @since v4.11.0
*/ */
public getBatchConfig(): StorageBatchConfig { public getBatchConfig(): StorageBatchConfig {
return { return {
@ -172,7 +171,7 @@ export class OPFSStorage extends BaseStorage {
// Initialize counts from storage // Initialize counts from storage
await this.initializeCounts() await this.initializeCounts()
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics // Initialize GraphAdjacencyIndex and type statistics
await super.init() await super.init()
} catch (error) { } catch (error) {
console.error('Failed to initialize OPFS storage:', error) console.error('Failed to initialize OPFS storage:', error)
@ -232,7 +231,7 @@ export class OPFSStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation // Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
/** /**
* Delete an edge from storage * Delete an edge from storage
@ -482,14 +481,14 @@ export class OPFSStorage extends BaseStorage {
// Remove all files in the index directory // Remove all files in the index directory
await removeDirectoryContents(this.indexDir!) await removeDirectoryContents(this.indexDir!)
// v5.6.1: Remove COW (copy-on-write) version control data // Remove COW (copy-on-write) version control data
// This directory stores all git-like versioning data (commits, trees, blobs, refs) // This directory stores all git-like versioning data (commits, trees, blobs, refs)
// Must be deleted to fully clear all data including version history // Must be deleted to fully clear all data including version history
try { try {
// Delete the entire _cow/ directory (not just contents) // Delete the entire _cow/ directory (not just contents)
await this.rootDir!.removeEntry('_cow', { recursive: true }) await this.rootDir!.removeEntry('_cow', { recursive: true })
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled) // Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use // COW will re-initialize automatically on next use
this.refManager = undefined this.refManager = undefined
this.blobStorage = undefined this.blobStorage = undefined
@ -505,7 +504,7 @@ export class OPFSStorage extends BaseStorage {
this.statisticsCache = null this.statisticsCache = null
this.statisticsModified = false this.statisticsModified = false
// v5.6.1: Reset entity counters (inherited from BaseStorageAdapter) // Reset entity counters (inherited from BaseStorageAdapter)
// These in-memory counters must be reset to 0 after clearing all data // These in-memory counters must be reset to 0 after clearing all data
;(this as any).totalNounCount = 0 ;(this as any).totalNounCount = 0
;(this as any).totalVerbCount = 0 ;(this as any).totalVerbCount = 0
@ -517,16 +516,16 @@ export class OPFSStorage extends BaseStorage {
/** /**
* Check if COW has been explicitly disabled via clear() * Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts * Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker file exists, false otherwise * @returns true if marker file exists, false otherwise
* @protected * @protected
*/ */
/** /**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods * Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used * COW is now always enabled - marker files are no longer used
*/ */
// Quota monitoring configuration (v4.0.0) // Quota monitoring configuration
private quotaWarningThreshold = 0.8 // Warn at 80% usage private quotaWarningThreshold = 0.8 // Warn at 80% usage
private quotaCriticalThreshold = 0.95 // Critical at 95% usage private quotaCriticalThreshold = 0.95 // Critical at 95% usage
private lastQuotaCheck: number = 0 private lastQuotaCheck: number = 0
@ -675,7 +674,7 @@ export class OPFSStorage extends BaseStorage {
} }
/** /**
* Get detailed quota status with warnings (v4.0.0) * Get detailed quota status with warnings
* Monitors storage usage and warns when approaching quota limits * Monitors storage usage and warns when approaching quota limits
* *
* @returns Promise that resolves to quota status with warning levels * @returns Promise that resolves to quota status with warning levels
@ -769,7 +768,7 @@ export class OPFSStorage extends BaseStorage {
} }
/** /**
* Monitor quota during operations (v4.0.0) * Monitor quota during operations
* Automatically checks quota at regular intervals and warns if approaching limits * Automatically checks quota at regular intervals and warns if approaching limits
* Call this before write operations to ensure quota is available * Call this before write operations to ensure quota is available
* *
@ -1185,7 +1184,7 @@ export class OPFSStorage extends BaseStorage {
} }
} }
} catch (error) { } catch (error) {
// CRITICAL FIX (v3.37.4): No statistics files exist (first init) // CRITICAL FIX: No statistics files exist (first init)
// Return minimal stats with counts instead of null // Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild // This prevents HNSW from seeing entityCount=0 during index rebuild
return { return {
@ -1228,7 +1227,7 @@ export class OPFSStorage extends BaseStorage {
* @param options Pagination and filter options * @param options Pagination and filter options
* @returns Promise that resolves to a paginated result of nouns * @returns Promise that resolves to a paginated result of nouns
*/ */
// v5.4.0: Removed pagination overrides (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's type-first implementation // Removed pagination overrides (getNounsWithPagination, getVerbsWithPagination) - use BaseStorage's type-first implementation
/** /**
* Initialize counts from OPFS storage * Initialize counts from OPFS storage
@ -1313,28 +1312,28 @@ export class OPFSStorage extends BaseStorage {
} }
} }
// HNSW Index Persistence (v3.35.0+) // HNSW Index Persistence
/** /**
* Get a noun's vector for HNSW rebuild * Get a noun's vector for HNSW rebuild
*/ */
/** /**
* Get vector for a noun * Get vector for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths) * Uses BaseStorage's getNoun (type-first paths)
*/ */
public async getNounVector(id: string): Promise<number[] | null> { public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id) const noun = await this.getNoun(id)
return noun ? noun.vector : null return noun ? noun.vector : null
} }
// CRITICAL FIX (v4.10.1): Mutex locks for HNSW concurrency control // CRITICAL FIX: Mutex locks for HNSW concurrency control
// Browser environments are single-threaded but async operations can still interleave // Browser environments are single-threaded but async operations can still interleave
private hnswLocks = new Map<string, Promise<void>>() private hnswLocks = new Map<string, Promise<void>>()
/** /**
* Save HNSW graph data for a noun * Save HNSW graph data for a noun
* *
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths) * Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Preserves mutex locking to prevent read-modify-write races * CRITICAL: Preserves mutex locking to prevent read-modify-write races
*/ */
public async saveHNSWData(nounId: string, hnswData: { public async saveHNSWData(nounId: string, hnswData: {
@ -1354,7 +1353,7 @@ export class OPFSStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise) this.hnswLocks.set(lockKey, lockPromise)
try { try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths) // Use BaseStorage's getNoun (type-first paths)
const existingNoun = await this.getNoun(nounId) const existingNoun = await this.getNoun(nounId)
if (!existingNoun) { if (!existingNoun) {
@ -1374,7 +1373,7 @@ export class OPFSStorage extends BaseStorage {
connections: connectionsMap connections: connectionsMap
} }
// v5.4.0: Use BaseStorage's saveNoun (type-first paths) // Use BaseStorage's saveNoun (type-first paths)
await this.saveNoun(updatedNoun) await this.saveNoun(updatedNoun)
} finally { } finally {
// Release lock // Release lock
@ -1385,7 +1384,7 @@ export class OPFSStorage extends BaseStorage {
/** /**
* Get HNSW graph data for a noun * Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths) * Uses BaseStorage's getNoun (type-first paths)
*/ */
public async getHNSWData(nounId: string): Promise<{ public async getHNSWData(nounId: string): Promise<{
level: number level: number
@ -1415,7 +1414,7 @@ export class OPFSStorage extends BaseStorage {
* Save HNSW system data (entry point, max level) * Save HNSW system data (entry point, max level)
* Storage path: index/hnsw-system.json * Storage path: index/hnsw-system.json
* *
* CRITICAL FIX (v4.10.1): Mutex locking to prevent race conditions * CRITICAL FIX: Mutex locking to prevent race conditions
*/ */
public async saveHNSWSystem(systemData: { public async saveHNSWSystem(systemData: {
entryPointId: string | null entryPointId: string | null

View file

@ -107,7 +107,7 @@ export class OptimizedS3Search {
} }
// Determine if there are more items // Determine if there are more items
const hasMore = listResult.hasMore || nouns.length > limit // v5.7.11: Fixed >= to > (was causing infinite loop) const hasMore = listResult.hasMore || nouns.length > limit // Fixed >= to > (was causing infinite loop)
// Set next cursor // Set next cursor
let nextCursor: string | undefined let nextCursor: string | undefined
@ -190,7 +190,7 @@ export class OptimizedS3Search {
} }
// Determine if there are more items // Determine if there are more items
const hasMore = listResult.hasMore || verbs.length > limit // v5.7.11: Fixed >= to > (was causing infinite loop) const hasMore = listResult.hasMore || verbs.length > limit // Fixed >= to > (was causing infinite loop)
// Set next cursor // Set next cursor
let nextCursor: string | undefined let nextCursor: string | undefined

View file

@ -58,7 +58,7 @@ const MAX_R2_PAGE_SIZE = 1000
* Dedicated Cloudflare R2 storage adapter * Dedicated Cloudflare R2 storage adapter
* Optimized for R2's unique characteristics and global edge network * Optimized for R2's unique characteristics and global edge network
* *
* v5.4.0: Type-aware storage now built into BaseStorage * Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation) * - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed getNounsWithPagination override * - Removed getNounsWithPagination override
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths) * - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -112,7 +112,7 @@ export class R2Storage extends BaseStorage {
// Request coalescer for deduplication // Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance // Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching // Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Multi-level cache manager for efficient data access // Multi-level cache manager for efficient data access
@ -122,7 +122,7 @@ export class R2Storage extends BaseStorage {
// Module logger // Module logger
private logger = createModuleLogger('R2Storage') private logger = createModuleLogger('R2Storage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races // HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>() private hnswLocks = new Map<string, Promise<void>>()
/** /**
@ -168,7 +168,7 @@ export class R2Storage extends BaseStorage {
}) })
this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig) this.verbCacheManager = new CacheManager<Edge>(options.cacheConfig)
// v6.2.7: Write buffering always enabled - no env var check needed // Write buffering always enabled - no env var check needed
} }
/** /**
@ -183,7 +183,7 @@ export class R2Storage extends BaseStorage {
* Zero egress fees enable aggressive caching and parallel downloads * Zero egress fees enable aggressive caching and parallel downloads
* *
* @returns R2-optimized batch configuration * @returns R2-optimized batch configuration
* @since v5.12.0 - Updated for native batch API * Updated for native batch API
*/ */
public getBatchConfig(): StorageBatchConfig { public getBatchConfig(): StorageBatchConfig {
return { return {
@ -208,7 +208,6 @@ export class R2Storage extends BaseStorage {
* *
* @param paths - Array of R2 object keys to read * @param paths - Array of R2 object keys to read
* @returns Map of path -> parsed JSON data (only successful reads) * @returns Map of path -> parsed JSON data (only successful reads)
* @since v5.12.0
*/ */
public async readBatch(paths: string[]): Promise<Map<string, any>> { public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized() await this.ensureInitialized()
@ -336,7 +335,7 @@ export class R2Storage extends BaseStorage {
this.nounCacheManager.clear() this.nounCacheManager.clear()
this.verbCacheManager.clear() this.verbCacheManager.clear()
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics // Initialize GraphAdjacencyIndex and type statistics
await super.init() await super.init()
} catch (error) { } catch (error) {
this.logger.error('Failed to initialize R2 storage:', error) this.logger.error('Failed to initialize R2 storage:', error)
@ -409,7 +408,7 @@ export class R2Storage extends BaseStorage {
} }
} }
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage // Removed checkVolumeMode() - write buffering always enabled for cloud storage
/** /**
* Flush noun buffer to R2 * Flush noun buffer to R2
@ -444,16 +443,16 @@ export class R2Storage extends BaseStorage {
/** /**
* Save a node to storage * Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance * Always uses write buffer for consistent performance
*/ */
protected async saveNode(node: HNSWNode): Promise<void> { protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching // Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) { if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`) this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency // Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) { if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node) this.nounCacheManager.set(node.id, node)
} }
@ -728,14 +727,14 @@ export class R2Storage extends BaseStorage {
/** /**
* Save an edge to storage * Save an edge to storage
* v6.2.7: Always uses write buffer for consistent performance * Always uses write buffer for consistent performance
*/ */
protected async saveEdge(edge: Edge): Promise<void> { protected async saveEdge(edge: Edge): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching // Always use write buffer - cloud storage benefits from batching
if (this.verbWriteBuffer) { if (this.verbWriteBuffer) {
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency // Populate cache BEFORE buffering for read-after-write consistency
this.verbCacheManager.set(edge.id, edge) this.verbCacheManager.set(edge.id, edge)
await this.verbWriteBuffer.add(edge.id, edge) await this.verbWriteBuffer.add(edge.id, edge)
@ -750,7 +749,7 @@ export class R2Storage extends BaseStorage {
const requestId = await this.applyBackpressure() const requestId = await this.applyBackpressure()
try { try {
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields in verb vector file // ARCHITECTURAL FIX: Include core relational fields in verb vector file
// These fields are essential for 90% of operations - no metadata lookup needed // These fields are essential for 90% of operations - no metadata lookup needed
const serializableEdge = { const serializableEdge = {
id: edge.id, id: edge.id,
@ -762,7 +761,7 @@ export class R2Storage extends BaseStorage {
]) ])
), ),
// CORE RELATIONAL DATA (v3.50.1+) // CORE RELATIONAL DATA
verb: edge.verb, verb: edge.verb,
sourceId: edge.sourceId, sourceId: edge.sourceId,
targetId: edge.targetId, targetId: edge.targetId,
@ -785,7 +784,7 @@ export class R2Storage extends BaseStorage {
this.verbCacheManager.set(edge.id, edge) this.verbCacheManager.set(edge.id, edge)
// Count tracking happens in baseStorage.saveVerbMetadata_internal (v4.1.2) // Count tracking happens in baseStorage.saveVerbMetadata_internal
// This fixes the race condition where metadata didn't exist yet // This fixes the race condition where metadata didn't exist yet
this.releaseBackpressure(true, requestId) this.releaseBackpressure(true, requestId)
@ -831,7 +830,7 @@ export class R2Storage extends BaseStorage {
connections.set(Number(level), new Set(verbIds as string[])) connections.set(Number(level), new Set(verbIds as string[]))
} }
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field) // Return HNSWVerb with core relational fields (NO metadata field)
const edge: Edge = { const edge: Edge = {
id: data.id, id: data.id,
vector: data.vector, vector: data.vector,
@ -842,7 +841,7 @@ export class R2Storage extends BaseStorage {
sourceId: data.sourceId, sourceId: data.sourceId,
targetId: data.targetId targetId: data.targetId
// ✅ NO metadata field in v4.0.0 // ✅ NO metadata field
// User metadata retrieved separately via getVerbMetadata() // User metadata retrieved separately via getVerbMetadata()
} }
@ -1081,7 +1080,7 @@ export class R2Storage extends BaseStorage {
prodLog.info('🧹 R2: Clearing all data from bucket...') prodLog.info('🧹 R2: Clearing all data from bucket...')
// v5.11.0: Clear ALL data using correct paths // Clear ALL data using correct paths
// Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks) // Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks)
const branchObjects = await this.listObjectsUnderPath('branches/') const branchObjects = await this.listObjectsUnderPath('branches/')
for (const key of branchObjects) { for (const key of branchObjects) {
@ -1100,7 +1099,7 @@ export class R2Storage extends BaseStorage {
await this.deleteObjectFromPath(key) await this.deleteObjectFromPath(key)
} }
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled) // Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use // COW will re-initialize automatically on next use
this.refManager = undefined this.refManager = undefined
this.blobStorage = undefined this.blobStorage = undefined
@ -1143,17 +1142,17 @@ export class R2Storage extends BaseStorage {
/** /**
* Check if COW has been explicitly disabled via clear() * Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts * Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker object exists, false otherwise * @returns true if marker object exists, false otherwise
* @protected * @protected
*/ */
/** /**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods * Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used * COW is now always enabled - marker files are no longer used
*/ */
// v5.4.0: Removed getNounsWithPagination override - use BaseStorage's type-first implementation // Removed getNounsWithPagination override - use BaseStorage's type-first implementation
// v5.4.0: Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation // Removed 10 *_internal method overrides - now inherit from BaseStorage's type-first implementation
} }

View file

@ -85,7 +85,7 @@ type S3Command = any
* - endpoint: GCS endpoint (e.g., 'https://storage.googleapis.com') * - endpoint: GCS endpoint (e.g., 'https://storage.googleapis.com')
* - bucketName: GCS bucket name * - bucketName: GCS bucket name
* *
* v5.4.0: Type-aware storage now built into BaseStorage * Type-aware storage now built into BaseStorage
* - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation) * - Removed 10 *_internal method overrides (now inherit from BaseStorage's type-first implementation)
* - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination) * - Removed 2 pagination method overrides (getNounsWithPagination, getVerbsWithPagination)
* - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths) * - Updated HNSW methods to use BaseStorage's getNoun/saveNoun (type-first paths)
@ -152,7 +152,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Request coalescer for deduplication // Request coalescer for deduplication
private requestCoalescer: RequestCoalescer | null = null private requestCoalescer: RequestCoalescer | null = null
// v6.2.7: Write buffering always enabled for consistent performance // Write buffering always enabled for consistent performance
// Removes dynamic mode switching complexity - cloud storage always benefits from batching // Removes dynamic mode switching complexity - cloud storage always benefits from batching
// Operation executors for timeout and retry handling // Operation executors for timeout and retry handling
@ -165,7 +165,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Module logger // Module logger
private logger = createModuleLogger('S3Storage') private logger = createModuleLogger('S3Storage')
// v5.4.0: HNSW mutex locks to prevent read-modify-write races // HNSW mutex locks to prevent read-modify-write races
private hnswLocks = new Map<string, Promise<void>>() private hnswLocks = new Map<string, Promise<void>>()
/** /**
@ -210,7 +210,7 @@ export class S3CompatibleStorage extends BaseStorage {
} }
/** /**
* Initialization mode for fast cold starts (v7.3.0+) * Initialization mode for fast cold starts
* *
* - `'auto'` (default): Progressive in cloud environments (Lambda, Cloud Run), * - `'auto'` (default): Progressive in cloud environments (Lambda, Cloud Run),
* strict locally. Zero-config optimization. * strict locally. Zero-config optimization.
@ -219,7 +219,6 @@ export class S3CompatibleStorage extends BaseStorage {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts * - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns. * before init() returns.
* *
* @since v7.3.0
*/ */
initMode?: InitMode initMode?: InitMode
@ -236,7 +235,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.serviceType = options.serviceType || 's3' this.serviceType = options.serviceType || 's3'
this.readOnly = options.readOnly || false this.readOnly = options.readOnly || false
// v7.3.0: Handle initMode // Handle initMode
if (options.initMode) { if (options.initMode) {
this.initMode = options.initMode this.initMode = options.initMode
} }
@ -270,7 +269,7 @@ export class S3CompatibleStorage extends BaseStorage {
* S3 supports ~5000 operations/second with burst capacity up to 10,000 * S3 supports ~5000 operations/second with burst capacity up to 10,000
* *
* @returns S3-optimized batch configuration * @returns S3-optimized batch configuration
* @since v5.12.0 - Updated for native batch API * Updated for native batch API
*/ */
public getBatchConfig(): StorageBatchConfig { public getBatchConfig(): StorageBatchConfig {
return { return {
@ -295,7 +294,6 @@ export class S3CompatibleStorage extends BaseStorage {
* *
* @param paths - Array of S3 object keys to read * @param paths - Array of S3 object keys to read
* @returns Map of path -> parsed JSON data (only successful reads) * @returns Map of path -> parsed JSON data (only successful reads)
* @since v5.12.0
*/ */
public async readBatch(paths: string[]): Promise<Map<string, any>> { public async readBatch(paths: string[]): Promise<Map<string, any>> {
await this.ensureInitialized() await this.ensureInitialized()
@ -358,7 +356,7 @@ export class S3CompatibleStorage extends BaseStorage {
/** /**
* Initialize the storage adapter * Initialize the storage adapter
* *
* v7.3.0: Supports progressive initialization for fast cold starts * Supports progressive initialization for fast cold starts
* *
* | Mode | Init Time | When | * | Mode | Init Time | When |
* |------|-----------|------| * |------|-----------|------|
@ -514,7 +512,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Initialize request coalescer // Initialize request coalescer
this.initializeCoalescer() this.initializeCoalescer()
// CRITICAL FIX (v3.37.7): Clear any stale cache entries from previous runs // CRITICAL FIX: Clear any stale cache entries from previous runs
// This prevents cache poisoning from causing silent failures on container restart // This prevents cache poisoning from causing silent failures on container restart
const nodeCacheSize = this.nodeCache?.size || 0 const nodeCacheSize = this.nodeCache?.size || 0
if (nodeCacheSize > 0) { if (nodeCacheSize > 0) {
@ -524,7 +522,7 @@ export class S3CompatibleStorage extends BaseStorage {
prodLog.info('🧹 Node cache is empty - starting fresh') prodLog.info('🧹 Node cache is empty - starting fresh')
} }
// v7.3.0: Progressive vs Strict initialization // Progressive vs Strict initialization
if (effectiveMode === 'progressive') { if (effectiveMode === 'progressive') {
// PROGRESSIVE MODE: Fast init, background validation // PROGRESSIVE MODE: Fast init, background validation
// Mark as initialized immediately - ready to accept operations // Mark as initialized immediately - ready to accept operations
@ -533,7 +531,7 @@ export class S3CompatibleStorage extends BaseStorage {
prodLog.info(`✅ S3 progressive init complete: ${this.bucketName} (validation deferred)`) prodLog.info(`✅ S3 progressive init complete: ${this.bucketName} (validation deferred)`)
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics // Initialize GraphAdjacencyIndex and type statistics
await super.init() await super.init()
// Schedule background tasks (non-blocking) // Schedule background tasks (non-blocking)
@ -556,7 +554,7 @@ export class S3CompatibleStorage extends BaseStorage {
await this.initializeCounts() await this.initializeCounts()
this.countsLoaded = true this.countsLoaded = true
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics // Initialize GraphAdjacencyIndex and type statistics
await super.init() await super.init()
// Mark background tasks as complete (nothing to do in background) // Mark background tasks as complete (nothing to do in background)
@ -573,7 +571,7 @@ export class S3CompatibleStorage extends BaseStorage {
} }
// ============================================= // =============================================
// Progressive Initialization (v7.3.0+) // Progressive Initialization
// ============================================= // =============================================
/** /**
@ -588,7 +586,6 @@ export class S3CompatibleStorage extends BaseStorage {
* *
* @protected * @protected
* @override * @override
* @since v7.3.0
*/ */
protected async runBackgroundInit(): Promise<void> { protected async runBackgroundInit(): Promise<void> {
const startTime = Date.now() const startTime = Date.now()
@ -614,7 +611,6 @@ export class S3CompatibleStorage extends BaseStorage {
* Stores result in bucketValidated/bucketValidationError for lazy use. * Stores result in bucketValidated/bucketValidationError for lazy use.
* *
* @private * @private
* @since v7.3.0
*/ */
private async validateBucketInBackground(): Promise<void> { private async validateBucketInBackground(): Promise<void> {
try { try {
@ -638,7 +634,6 @@ export class S3CompatibleStorage extends BaseStorage {
* Load counts from storage in background. * Load counts from storage in background.
* *
* @private * @private
* @since v7.3.0
*/ */
private async loadCountsInBackground(): Promise<void> { private async loadCountsInBackground(): Promise<void> {
try { try {
@ -662,7 +657,6 @@ export class S3CompatibleStorage extends BaseStorage {
* @throws Error if bucket does not exist or is not accessible * @throws Error if bucket does not exist or is not accessible
* @protected * @protected
* @override * @override
* @since v7.3.0
*/ */
protected async ensureValidatedForWrite(): Promise<void> { protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do // If already validated, nothing to do
@ -918,7 +912,7 @@ export class S3CompatibleStorage extends BaseStorage {
) )
} }
// v6.2.7: Removed checkVolumeMode() - write buffering always enabled for cloud storage // Removed checkVolumeMode() - write buffering always enabled for cloud storage
/** /**
* Bulk write nouns to S3 * Bulk write nouns to S3
@ -1179,20 +1173,20 @@ export class S3CompatibleStorage extends BaseStorage {
return this.socketManager.getBatchSize() return this.socketManager.getBatchSize()
} }
// v5.4.0: Removed 10 *_internal method overrides (lines 984-2069) - now inherit from BaseStorage's type-first implementation // Removed 10 *_internal method overrides (lines 984-2069) - now inherit from BaseStorage's type-first implementation
/** /**
* Save a node to storage * Save a node to storage
* v6.2.7: Always uses write buffer for consistent performance * Always uses write buffer for consistent performance
*/ */
protected async saveNode(node: HNSWNode): Promise<void> { protected async saveNode(node: HNSWNode): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v6.2.7: Always use write buffer - cloud storage benefits from batching // Always use write buffer - cloud storage benefits from batching
if (this.nounWriteBuffer) { if (this.nounWriteBuffer) {
this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`) this.logger.trace(`📝 BUFFERING: Adding noun ${node.id} to write buffer`)
// v6.2.6: Populate cache BEFORE buffering for read-after-write consistency // Populate cache BEFORE buffering for read-after-write consistency
if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) { if (node.vector && Array.isArray(node.vector) && node.vector.length > 0) {
this.nounCacheManager.set(node.id, node) this.nounCacheManager.set(node.id, node)
} }
@ -1242,7 +1236,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.logger.debug(`Node ${node.id} saved successfully`) this.logger.debug(`Node ${node.id} saved successfully`)
// Log the change for efficient synchronization (v4.0.0: no metadata on node) // Log the change for efficient synchronization (no metadata on node)
await this.appendToChangeLog({ await this.appendToChangeLog({
timestamp: Date.now(), timestamp: Date.now(),
operation: 'add', // Could be 'update' if we track existing nodes operation: 'add', // Could be 'update' if we track existing nodes
@ -1250,7 +1244,7 @@ export class S3CompatibleStorage extends BaseStorage {
entityId: node.id, entityId: node.id,
data: { data: {
vector: node.vector vector: node.vector
// ✅ NO metadata field in v4.0.0 - stored separately // ✅ NO metadata field - stored separately
} }
}) })
@ -1296,7 +1290,7 @@ export class S3CompatibleStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed getNoun_internal override - uses BaseStorage type-first implementation // Removed getNoun_internal override - uses BaseStorage type-first implementation
/** /**
* Get a node from storage * Get a node from storage
@ -1307,7 +1301,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Check cache first // Check cache first
const cached = this.nodeCache.get(id) const cached = this.nodeCache.get(id)
// Validate cached object before returning (v3.37.8+) // Validate cached object before returning
if (cached !== undefined && cached !== null) { if (cached !== undefined && cached !== null) {
// Validate cached object has required fields (including non-empty vector!) // Validate cached object has required fields (including non-empty vector!)
if (!cached.id || !cached.vector || !Array.isArray(cached.vector) || cached.vector.length === 0) { if (!cached.id || !cached.vector || !Array.isArray(cached.vector) || cached.vector.length === 0) {
@ -1607,7 +1601,7 @@ export class S3CompatibleStorage extends BaseStorage {
return nodes return nodes
} }
// v5.4.0: Removed 4 *_internal method overrides (getNounsByNounType_internal, deleteNoun_internal, saveVerb_internal, getVerb_internal) // Removed 4 *_internal method overrides (getNounsByNounType_internal, deleteNoun_internal, saveVerb_internal, getVerb_internal)
// Now inherit from BaseStorage's type-first implementation // Now inherit from BaseStorage's type-first implementation
@ -1788,23 +1782,23 @@ export class S3CompatibleStorage extends BaseStorage {
return true // Return all edges since filtering requires metadata return true // Return all edges since filtering requires metadata
} }
// v5.4.0: Removed getVerbsWithPagination override - use BaseStorage's type-first implementation // Removed getVerbsWithPagination override - use BaseStorage's type-first implementation
// v5.4.0: Removed 4 more *_internal method overrides (getVerbsBySource, getVerbsByTarget, getVerbsByType, deleteVerb) // Removed 4 more *_internal method overrides (getVerbsBySource, getVerbsByTarget, getVerbsByType, deleteVerb)
// Total: 8 *_internal methods removed - all now inherit from BaseStorage's type-first implementation // Total: 8 *_internal methods removed - all now inherit from BaseStorage's type-first implementation
/** /**
* Primitive operation: Write object to path * Primitive operation: Write object to path
* All metadata operations use this internally via base class routing * All metadata operations use this internally via base class routing
* *
* v7.3.0: Performs lazy bucket validation on first write in progressive mode. * Performs lazy bucket validation on first write in progressive mode.
*/ */
protected async writeObjectToPath(path: string, data: any): Promise<void> { protected async writeObjectToPath(path: string, data: any): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init // Lazy bucket validation for progressive init
await this.ensureValidatedForWrite() await this.ensureValidatedForWrite()
// Apply backpressure before starting operation // Apply backpressure before starting operation
@ -1897,11 +1891,11 @@ export class S3CompatibleStorage extends BaseStorage {
* Primitive operation: Delete object from path * Primitive operation: Delete object from path
* All metadata operations use this internally via base class routing * All metadata operations use this internally via base class routing
* *
* v7.3.0: Performs lazy bucket validation on first delete in progressive mode. * Performs lazy bucket validation on first delete in progressive mode.
*/ */
protected async deleteObjectFromPath(path: string): Promise<void> { protected async deleteObjectFromPath(path: string): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// v7.3.0: Lazy bucket validation for progressive init // Lazy bucket validation for progressive init
await this.ensureValidatedForWrite() await this.ensureValidatedForWrite()
try { try {
@ -2315,7 +2309,7 @@ export class S3CompatibleStorage extends BaseStorage {
} }
} }
// v5.11.0: Clear ALL data using correct paths // Clear ALL data using correct paths
// Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks) // Delete entire branches/ directory (includes ALL entities, ALL types, ALL VFS data, ALL forks)
await deleteObjectsWithPrefix('branches/') await deleteObjectsWithPrefix('branches/')
@ -2325,7 +2319,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Delete system metadata // Delete system metadata
await deleteObjectsWithPrefix('_system/') await deleteObjectsWithPrefix('_system/')
// v5.11.0: Reset COW managers (but don't disable COW - it's always enabled) // Reset COW managers (but don't disable COW - it's always enabled)
// COW will re-initialize automatically on next use // COW will re-initialize automatically on next use
this.refManager = undefined this.refManager = undefined
this.blobStorage = undefined this.blobStorage = undefined
@ -2335,7 +2329,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.statisticsCache = null this.statisticsCache = null
this.statisticsModified = false this.statisticsModified = false
// v5.6.1: Reset entity counters (inherited from BaseStorageAdapter) // Reset entity counters (inherited from BaseStorageAdapter)
// These in-memory counters must be reset to 0 after clearing all data // These in-memory counters must be reset to 0 after clearing all data
;(this as any).totalNounCount = 0 ;(this as any).totalNounCount = 0
;(this as any).totalVerbCount = 0 ;(this as any).totalVerbCount = 0
@ -2457,12 +2451,12 @@ export class S3CompatibleStorage extends BaseStorage {
/** /**
* Check if COW has been explicitly disabled via clear() * Check if COW has been explicitly disabled via clear()
* v5.10.4: Fixes bug where clear() doesn't persist across instance restarts * Fixes bug where clear() doesn't persist across instance restarts
* @returns true if marker object exists, false otherwise * @returns true if marker object exists, false otherwise
* @protected * @protected
*/ */
/** /**
* v5.11.0: Removed checkClearMarker() and createClearMarker() methods * Removed checkClearMarker() and createClearMarker() methods
* COW is now always enabled - marker files are no longer used * COW is now always enabled - marker files are no longer used
*/ */
@ -2851,7 +2845,7 @@ export class S3CompatibleStorage extends BaseStorage {
totalEdges: this.totalVerbCount totalEdges: this.totalVerbCount
} }
} }
// CRITICAL FIX (v3.37.4): Statistics file doesn't exist yet (first restart) // CRITICAL FIX: Statistics file doesn't exist yet (first restart)
// Return minimal stats with counts instead of null // Return minimal stats with counts instead of null
// This prevents HNSW from seeing entityCount=0 during index rebuild // This prevents HNSW from seeing entityCount=0 during index rebuild
return { return {
@ -3415,7 +3409,7 @@ export class S3CompatibleStorage extends BaseStorage {
} }
} }
// v5.4.0: Removed getNounsWithPagination override - use BaseStorage's type-first implementation // Removed getNounsWithPagination override - use BaseStorage's type-first implementation
/** /**
* Estimate total noun count by listing objects across all shards * Estimate total noun count by listing objects across all shards
@ -3549,7 +3543,7 @@ export class S3CompatibleStorage extends BaseStorage {
} }
/** /**
* Override base class to enable smart batching for cloud storage (v3.32.3+) * Override base class to enable smart batching for cloud storage
* *
* S3 is cloud storage with network latency (~50ms per write). * S3 is cloud storage with network latency (~50ms per write).
* Smart batching reduces writes from 1000 ops 100 batches. * Smart batching reduces writes from 1000 ops 100 batches.
@ -3560,11 +3554,11 @@ export class S3CompatibleStorage extends BaseStorage {
return true // S3 benefits from batching return true // S3 benefits from batching
} }
// HNSW Index Persistence (v3.35.0+) // HNSW Index Persistence
/** /**
* Get a noun's vector for HNSW rebuild * Get a noun's vector for HNSW rebuild
* v5.4.0: Uses BaseStorage's getNoun (type-first paths) * Uses BaseStorage's getNoun (type-first paths)
*/ */
public async getNounVector(id: string): Promise<number[] | null> { public async getNounVector(id: string): Promise<number[] | null> {
const noun = await this.getNoun(id) const noun = await this.getNoun(id)
@ -3574,7 +3568,7 @@ export class S3CompatibleStorage extends BaseStorage {
/** /**
* Save HNSW graph data for a noun * Save HNSW graph data for a noun
* *
* v5.4.0: Uses BaseStorage's getNoun/saveNoun (type-first paths) * Uses BaseStorage's getNoun/saveNoun (type-first paths)
* CRITICAL: Uses mutex locking to prevent read-modify-write races * CRITICAL: Uses mutex locking to prevent read-modify-write races
*/ */
public async saveHNSWData(nounId: string, hnswData: { public async saveHNSWData(nounId: string, hnswData: {
@ -3583,7 +3577,7 @@ export class S3CompatibleStorage extends BaseStorage {
}): Promise<void> { }): Promise<void> {
const lockKey = `hnsw/${nounId}` const lockKey = `hnsw/${nounId}`
// CRITICAL FIX (v4.10.1): Mutex lock to prevent read-modify-write races // CRITICAL FIX: Mutex lock to prevent read-modify-write races
// Problem: Without mutex, concurrent operations can: // Problem: Without mutex, concurrent operations can:
// 1. Thread A reads noun (connections: [1,2,3]) // 1. Thread A reads noun (connections: [1,2,3])
// 2. Thread B reads noun (connections: [1,2,3]) // 2. Thread B reads noun (connections: [1,2,3])
@ -3603,7 +3597,7 @@ export class S3CompatibleStorage extends BaseStorage {
this.hnswLocks.set(lockKey, lockPromise) this.hnswLocks.set(lockKey, lockPromise)
try { try {
// v5.4.0: Use BaseStorage's getNoun (type-first paths) // Use BaseStorage's getNoun (type-first paths)
// Read existing noun data (if exists) // Read existing noun data (if exists)
const existingNoun = await this.getNoun(nounId) const existingNoun = await this.getNoun(nounId)
@ -3625,7 +3619,7 @@ export class S3CompatibleStorage extends BaseStorage {
connections: connectionsMap connections: connectionsMap
} }
// v5.4.0: Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch) // Use BaseStorage's saveNoun (type-first paths, atomic write via writeObjectToBranch)
await this.saveNoun(updatedNoun) await this.saveNoun(updatedNoun)
} finally { } finally {
// Release lock (ALWAYS runs, even if error thrown) // Release lock (ALWAYS runs, even if error thrown)
@ -3636,7 +3630,7 @@ export class S3CompatibleStorage extends BaseStorage {
/** /**
* Get HNSW graph data for a noun * Get HNSW graph data for a noun
* v5.4.0: Uses BaseStorage's getNoun (type-first paths) * Uses BaseStorage's getNoun (type-first paths)
*/ */
public async getHNSWData(nounId: string): Promise<{ public async getHNSWData(nounId: string): Promise<{
level: number level: number
@ -3666,7 +3660,7 @@ export class S3CompatibleStorage extends BaseStorage {
* Save HNSW system data (entry point, max level) * Save HNSW system data (entry point, max level)
* Storage path: system/hnsw-system.json * Storage path: system/hnsw-system.json
* *
* CRITICAL FIX (v4.10.1): Optimistic locking with ETags to prevent race conditions * CRITICAL FIX: Optimistic locking with ETags to prevent race conditions
*/ */
public async saveHNSWSystem(systemData: { public async saveHNSWSystem(systemData: {
entryPointId: string | null entryPointId: string | null
@ -3781,7 +3775,7 @@ export class S3CompatibleStorage extends BaseStorage {
} }
/** /**
* Set S3 lifecycle policy for automatic tier transitions and deletions (v4.0.0) * Set S3 lifecycle policy for automatic tier transitions and deletions
* Automates cost optimization by moving old data to cheaper storage classes * Automates cost optimization by moving old data to cheaper storage classes
* *
* S3 Storage Classes: * S3 Storage Classes:
@ -3976,7 +3970,7 @@ export class S3CompatibleStorage extends BaseStorage {
} }
/** /**
* Enable S3 Intelligent-Tiering for automatic cost optimization (v4.0.0) * Enable S3 Intelligent-Tiering for automatic cost optimization
* Automatically moves objects between access tiers based on usage patterns * Automatically moves objects between access tiers based on usage patterns
* *
* Intelligent-Tiering automatically saves up to 95% on storage costs: * Intelligent-Tiering automatically saves up to 95% on storage costs:

View file

@ -1,6 +1,6 @@
/** /**
* DEPRECATED (v4.7.2): Backward compatibility stubs * DEPRECATED: Backward compatibility stubs
* TODO: Remove in v4.7.3 after migrating s3CompatibleStorage * TODO: Remove after migrating s3CompatibleStorage
*/ */
export class StorageCompatibilityLayer { export class StorageCompatibilityLayer {

File diff suppressed because it is too large Load diff

View file

@ -161,7 +161,7 @@ export class BlobStorage {
} }
/** /**
* v5.7.5: Ensure compression is ready before write operations * Ensure compression is ready before write operations
* Fixes race condition where write happens before async compression init completes * Fixes race condition where write happens before async compression init completes
*/ */
private async ensureCompressionReady(): Promise<void> { private async ensureCompressionReady(): Promise<void> {
@ -206,7 +206,7 @@ export class BlobStorage {
return hash return hash
} }
// v5.7.5: Ensure compression is initialized before writing // Ensure compression is initialized before writing
// Fixes race condition where write happens before async init completes // Fixes race condition where write happens before async init completes
await this.ensureCompressionReady() await this.ensureCompressionReady()
@ -223,7 +223,7 @@ export class BlobStorage {
} }
// Create metadata // Create metadata
// v5.7.5: Store ACTUAL compression state, not intended // Store ACTUAL compression state, not intended
// Prevents corruption if compression failed to initialize // Prevents corruption if compression failed to initialize
const actualCompression = finalData === data ? 'none' : compression const actualCompression = finalData === data ? 'none' : compression
const metadata: BlobMetadata = { const metadata: BlobMetadata = {
@ -277,7 +277,7 @@ export class BlobStorage {
* @returns Blob data * @returns Blob data
*/ */
async read(hash: string, options: BlobReadOptions = {}): Promise<Buffer> { async read(hash: string, options: BlobReadOptions = {}): Promise<Buffer> {
// v5.3.4 fix: Guard against NULL hash (sentinel value) // Guard against NULL hash (sentinel value)
// NULL_HASH ('0000...0000') is used as a sentinel for "no parent" or "empty tree" // NULL_HASH ('0000...0000') is used as a sentinel for "no parent" or "empty tree"
// It should NEVER be read as actual blob data // It should NEVER be read as actual blob data
if (isNullHash(hash)) { if (isNullHash(hash)) {
@ -309,7 +309,7 @@ export class BlobStorage {
metadataBuffer = await this.adapter.get(`${tryPrefix}-meta:${hash}`) metadataBuffer = await this.adapter.get(`${tryPrefix}-meta:${hash}`)
if (metadataBuffer) { if (metadataBuffer) {
prefix = tryPrefix prefix = tryPrefix
// v5.10.1: Unwrap metadata before parsing (defense-in-depth) // Unwrap metadata before parsing (defense-in-depth)
// Metadata should be JSON, but adapter might return wrapped format // Metadata should be JSON, but adapter might return wrapped format
const unwrappedMetadata = unwrapBinaryData(metadataBuffer) const unwrappedMetadata = unwrapBinaryData(metadataBuffer)
metadata = JSON.parse(unwrappedMetadata.toString()) metadata = JSON.parse(unwrappedMetadata.toString())
@ -338,8 +338,8 @@ export class BlobStorage {
finalData = await this.zstdDecompress(data) finalData = await this.zstdDecompress(data)
} }
// v5.10.1: Defense-in-depth unwrap (CRITICAL FIX for blob integrity regression) // Defense-in-depth unwrap (CRITICAL FIX for blob integrity regression)
// Even though COW adapter should unwrap (v5.7.5), verify it happened and re-unwrap if needed // Even though COW adapter should unwrap, verify it happened and re-unwrap if needed
// This prevents "Blob integrity check failed" errors if adapter returns wrapped data // This prevents "Blob integrity check failed" errors if adapter returns wrapped data
// Uses shared binaryDataCodec utility (single source of truth for unwrap logic) // Uses shared binaryDataCodec utility (single source of truth for unwrap logic)
const unwrappedData = unwrapBinaryData(finalData) const unwrappedData = unwrapBinaryData(finalData)

View file

@ -43,7 +43,7 @@ export interface CommitLogStats {
/** /**
* CommitLog: Efficient commit history traversal and querying * CommitLog: Efficient commit history traversal and querying
* *
* Pure v5.0.0 implementation - modern, clean, fast * Pure implementation - modern, clean, fast
*/ */
export class CommitLog { export class CommitLog {
private blobStorage: BlobStorage private blobStorage: BlobStorage

View file

@ -337,7 +337,7 @@ export class CommitObject {
let currentHash: string | null = startHash let currentHash: string | null = startHash
let depth = 0 let depth = 0
// v5.3.4 fix: Guard against NULL hash (sentinel for "no parent") // Guard against NULL hash (sentinel for "no parent")
// The initial commit has parent = null or NULL_HASH ('0000...0000') // The initial commit has parent = null or NULL_HASH ('0000...0000')
// We must stop walking when we reach it, not try to read it // We must stop walking when we reach it, not try to read it
while (currentHash && !isNullHash(currentHash)) { while (currentHash && !isNullHash(currentHash)) {

View file

@ -55,7 +55,7 @@ export interface RefUpdateOptions {
/** /**
* RefManager: Manages branches, tags, and HEAD pointer * RefManager: Manages branches, tags, and HEAD pointer
* *
* Pure implementation for v5.0.0 - no backward compatibility * Pure implementation - no backward compatibility
*/ */
export class RefManager { export class RefManager {
private adapter: COWStorageAdapter private adapter: COWStorageAdapter

View file

@ -86,7 +86,7 @@ export function unwrapBinaryData(data: any): Buffer {
/** /**
* Wrap binary data for JSON storage * Wrap binary data for JSON storage
* *
* WARNING: DO NOT USE THIS ON WRITE PATH! (v6.2.0) * WARNING: DO NOT USE THIS ON WRITE PATH!
* Use key-based dispatch in baseStorage.ts COW adapter instead. * Use key-based dispatch in baseStorage.ts COW adapter instead.
* This function exists for legacy/compatibility only. * This function exists for legacy/compatibility only.
* *
@ -94,7 +94,7 @@ export function unwrapBinaryData(data: any): Buffer {
* This is FRAGILE because compressed binary can accidentally parse as valid JSON, * This is FRAGILE because compressed binary can accidentally parse as valid JSON,
* causing blob integrity failures. * causing blob integrity failures.
* *
* v6.2.0 SOLUTION: baseStorage.ts COW adapter now uses key naming convention: * SOLUTION: baseStorage.ts COW adapter now uses key naming convention:
* - Keys with '-meta:' or 'ref:' prefix Always JSON * - Keys with '-meta:' or 'ref:' prefix Always JSON
* - Keys with 'blob:', 'commit:', 'tree:' prefix Always binary * - Keys with 'blob:', 'commit:', 'tree:' prefix Always binary
* No guessing needed! * No guessing needed!

View file

@ -10,7 +10,7 @@ import { S3CompatibleStorage } from './adapters/s3CompatibleStorage.js'
import { R2Storage } from './adapters/r2Storage.js' import { R2Storage } from './adapters/r2Storage.js'
import { GcsStorage } from './adapters/gcsStorage.js' import { GcsStorage } from './adapters/gcsStorage.js'
import { AzureBlobStorage } from './adapters/azureBlobStorage.js' import { AzureBlobStorage } from './adapters/azureBlobStorage.js'
// TypeAwareStorageAdapter removed in v5.4.0 - type-aware now built into BaseStorage // TypeAwareStorageAdapter removed - type-aware now built into BaseStorage
// FileSystemStorage is dynamically imported to avoid issues in browser environments // FileSystemStorage is dynamically imported to avoid issues in browser environments
import { isBrowser } from '../utils/environment.js' import { isBrowser } from '../utils/environment.js'
import { OperationConfig } from '../utils/operationUtils.js' import { OperationConfig } from '../utils/operationUtils.js'
@ -94,7 +94,7 @@ export interface StorageOptions {
sessionToken?: string sessionToken?: string
/** /**
* Initialization mode for fast cold starts (v7.3.0+) * Initialization mode for fast cold starts
* *
* - `'auto'` (default): Progressive in cloud environments (Lambda), * - `'auto'` (default): Progressive in cloud environments (Lambda),
* strict locally. Zero-config optimization. * strict locally. Zero-config optimization.
@ -103,7 +103,6 @@ export interface StorageOptions {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts * - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns. * before init() returns.
* *
* @since v7.3.0
*/ */
initMode?: 'progressive' | 'strict' | 'auto' initMode?: 'progressive' | 'strict' | 'auto'
} }
@ -215,7 +214,7 @@ export interface StorageOptions {
skipCountsFile?: boolean skipCountsFile?: boolean
/** /**
* Initialization mode for fast cold starts (v7.3.0+) * Initialization mode for fast cold starts
* *
* - `'auto'` (default): Progressive in cloud environments (Cloud Run), * - `'auto'` (default): Progressive in cloud environments (Cloud Run),
* strict locally. Zero-config optimization. * strict locally. Zero-config optimization.
@ -224,7 +223,6 @@ export interface StorageOptions {
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts * - `'strict'`: Traditional blocking init. Validates bucket and loads counts
* before init() returns. * before init() returns.
* *
* @since v7.3.0
*/ */
initMode?: 'progressive' | 'strict' | 'auto' initMode?: 'progressive' | 'strict' | 'auto'
} }
@ -259,7 +257,7 @@ export interface StorageOptions {
sasToken?: string sasToken?: string
/** /**
* Initialization mode for fast cold starts (v7.3.0+) * Initialization mode for fast cold starts
* *
* - `'auto'` (default): Progressive in cloud environments (Azure Functions), * - `'auto'` (default): Progressive in cloud environments (Azure Functions),
* strict locally. Zero-config optimization. * strict locally. Zero-config optimization.
@ -268,7 +266,6 @@ export interface StorageOptions {
* - `'strict'`: Traditional blocking init. Validates container and loads counts * - `'strict'`: Traditional blocking init. Validates container and loads counts
* before init() returns. * before init() returns.
* *
* @since v7.3.0
*/ */
initMode?: 'progressive' | 'strict' | 'auto' initMode?: 'progressive' | 'strict' | 'auto'
} }
@ -388,7 +385,7 @@ export interface StorageOptions {
/** /**
* COW (Copy-on-Write) configuration for instant fork() capability * COW (Copy-on-Write) configuration for instant fork() capability
* v5.0.1: COW is now always enabled (automatic, zero-config) * COW is now always enabled (automatic, zero-config)
*/ */
branch?: string // Current branch name (default: 'main') branch?: string // Current branch name (default: 'main')
enableCompression?: boolean // Enable zstd compression for COW blobs (default: true) enableCompression?: boolean // Enable zstd compression for COW blobs (default: true)
@ -410,14 +407,14 @@ function getFileSystemPath(options: StorageOptions): string {
} }
/** /**
* Configure COW (Copy-on-Write) options on a storage adapter (v5.4.0) * Configure COW (Copy-on-Write) options on a storage adapter
* TypeAware is now built-in to all adapters, no wrapper needed! * TypeAware is now built-in to all adapters, no wrapper needed!
* *
* @param storage - The storage adapter * @param storage - The storage adapter
* @param options - Storage options (for COW configuration) * @param options - Storage options (for COW configuration)
*/ */
function configureCOW(storage: any, options?: StorageOptions): void { function configureCOW(storage: any, options?: StorageOptions): void {
// v5.0.1: COW will be initialized AFTER storage.init() in Brainy // COW will be initialized AFTER storage.init() in Brainy
// Store COW options for later initialization // Store COW options for later initialization
if (typeof storage.initializeCOW === 'function') { if (typeof storage.initializeCOW === 'function') {
storage._cowOptions = { storage._cowOptions = {
@ -672,10 +669,10 @@ export async function createStorage(
} }
case 'type-aware': case 'type-aware':
// v5.0.0: TypeAware is now the default for ALL adapters // TypeAware is now the default for ALL adapters
// Redirect to the underlying type instead // Redirect to the underlying type instead
console.warn( console.warn(
'⚠️ type-aware is deprecated in v5.0.0 - TypeAware is now always enabled.' '⚠️ type-aware is deprecated - TypeAware is now always enabled.'
) )
console.warn( console.warn(
' Just use the underlying storage type (e.g., "filesystem", "s3", etc.)' ' Just use the underlying storage type (e.g., "filesystem", "s3", etc.)'
@ -866,7 +863,7 @@ export async function createStorage(
} }
/** /**
* Export storage adapters (v5.4.0: TypeAware is now built-in, no separate export) * Export storage adapters (TypeAware is now built-in, no separate export)
*/ */
export { export {
MemoryStorage, MemoryStorage,

Some files were not shown because too many files have changed in this diff Show more