brainy/docs/augmentations
David Snelling 92c96246fb feat(v4.0.0): Complete metadata/vector separation architecture with Azure support
This commit completes the core v4.0.0 architecture changes for billion-scale
performance with metadata/vector separation. NO RELEASE YET - remaining optimizations
and testing required before production release.

## Core v4.0.0 Architecture Changes

### Type System Updates
- Fixed all TypeScript compilation errors (zero errors achieved)
- Updated HNSWNoun/HNSWVerb to separate core fields from metadata
- Implemented HNSWNounWithMetadata/HNSWVerbWithMetadata for API boundaries
- Added required 'noun' field to NounMetadata for semantic structure
- Renamed verb.type to verb.verb for consistency

### Storage Adapter Updates
**All adapters updated for v4.0.0 two-file storage pattern:**
- memoryStorage: Proper metadata/vector separation
- fileSystemStorage: Two-file pattern with sharding
- opfsStorage: Browser persistent storage updated
- s3CompatibleStorage: AWS/MinIO/DigitalOcean support
- r2Storage: Cloudflare R2 optimization
- gcsStorage: Google Cloud with ADC support
- **azureBlobStorage: NEW - Full Azure Blob Storage support**

### Storage Features
- BaseStorage: Internal vs public method separation (_getNoun vs getNoun)
- Two-file storage: Vectors in one file, metadata in another
- Change tracking: getChangesSince return type updated
- Pagination: getNounsWithPagination returns WithMetadata types

### Azure Blob Storage Integration (NEW)
- Native @azure/storage-blob SDK integration
- Four authentication methods:
  * DefaultAzureCredential (Managed Identity) - recommended
  * Connection String - simplest setup
  * Account Name + Key - traditional auth
  * SAS Token - delegated access
- High-volume mode with write buffering
- Adaptive backpressure for throttling
- UUID-based sharding for billion-scale
- Full HNSW support with graph persistence

### Utility Updates
- EmbeddingManager: Updated to accept Record<string, unknown>
- LSMTree: Wrapped data in NounMetadata structure with 'noun' field
- EntityIdMapper: Fixed nested metadata.data structure access
- MetadataIndex: Fixed field type inference integration
- PeriodicCleanup: Updated for new metadata structure

### Core API Updates
- Brainy: Updated verb property access from v.type to v.verb
- ConfigAPI: Fixed NounMetadata access patterns
- DataAPI: Updated metadata handling

### Documentation Updates
- CREATING-AUGMENTATIONS.md: v4.0.0 breaking changes guide
- DEVELOPER-GUIDE.md: Migration checklist and examples
- COMPLETE-REFERENCE.md: v4.0.0 architecture improvements
- **finite-type-system.md: NEW - Revolutionary type system benefits**

### Build & Dependencies
- Zero TypeScript compilation errors
- Added @azure/storage-blob and @azure/identity
- 591 tests passing (23 timeout in long-running neural tests)

## What's NOT in This Release
This is a work-in-progress commit. Before v4.0.0 release we need:
- Storage adapter optimizations (batch operations, compression)
- Azure blob tier management (Hot/Cool/Archive)
- Cost optimization implementations
- Additional performance testing at billion-scale
- Migration guides for v3.x users

## Testing
- Clean build: 
- Type checking:  (zero errors)
- Test suite:  (591/614 passing, timeouts in neural tests only)

🔐 Generated with Claude Code
https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-17 12:29:27 -07:00
..
api-server.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
COMPLETE-REFERENCE.md feat(v4.0.0): Complete metadata/vector separation architecture with Azure support 2025-10-17 12:29:27 -07:00
CONFIGURATION.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00
DEVELOPER-GUIDE.md feat(v4.0.0): Complete metadata/vector separation architecture with Azure support 2025-10-17 12:29:27 -07:00
README.md fix: update all imports and references from BrainyData to Brainy 2025-09-30 17:09:15 -07:00

Brainy Augmentations

Augmentations are the core extensibility mechanism in Brainy. They allow you to modify, enhance, and extend Brainy's behavior without changing the core code.

Core Principle: One Interface, Infinite Possibilities

Every augmentation implements the same simple BrainyAugmentation interface:

interface BrainyAugmentation {
  name: string
  timing: 'before' | 'after' | 'around' | 'replace'
  operations: string[]
  priority: number
  initialize(context): Promise<void>
  execute(operation, params, next): Promise<any>
}

This single interface can handle EVERYTHING - from adding AI capabilities to exposing APIs to replacing storage backends.

Available Augmentations

🧠 Data Processing

Augmentations that enhance how data is processed and stored.

Augmentation Description Timing Status
NeuralImportAugmentation AI-powered entity and relationship extraction before Production
EntityRegistryAugmentation High-performance entity deduplication before Production
BatchProcessingAugmentation Optimizes bulk operations around Production
IntelligentVerbScoringAugmentation Learns relationship importance over time after Production

🔌 External Connections (Synapses)

Connect Brainy to external services and data sources.

Augmentation Description Timing Status
NotionSynapse Sync with Notion databases after 📝 Example
SalesforceSynapse Connect to Salesforce CRM after 📝 Example
SlackSynapse Import Slack conversations after 📝 Example
GoogleDriveSynapse Sync Google Drive documents after 📝 Example

🌐 API Exposure

Expose Brainy through various protocols.

Augmentation Description Timing Status
APIServerAugmentation REST, WebSocket, and MCP server after Production
GraphQLAugmentation GraphQL API endpoint after 🚧 Planned
gRPCAugmentation gRPC service after 🚧 Planned

💾 Storage Backends

Replace or enhance the storage layer.

Augmentation Description Timing Status
S3StorageAugmentation Use S3 as storage backend replace 📝 Example
RedisAugmentation Redis caching layer around 📝 Example
PostgresAugmentation PostgreSQL persistence replace 📝 Example

🔄 Real-time & Sync

Handle real-time updates and synchronization.

Augmentation Description Timing Status
WebSocketConduitAugmentation WebSocket client connections after ⚠️ Legacy
ServerSearchAugmentation Connect to remote Brainy servers after ⚠️ Legacy
TeamCoordinationAugmentation Multi-agent synchronization after 📝 Example

🛡️ Infrastructure

Core infrastructure and reliability features.

Augmentation Description Timing Status
ConnectionPoolAugmentation Optimize cloud storage connections before Production
RequestDeduplicatorAugmentation Prevent duplicate concurrent requests before Production
TransactionAugmentation ACID transaction support around 🚧 Planned
CacheAugmentation Multi-level caching around Production

📊 Monitoring & Analytics

Track and analyze Brainy's behavior.

Augmentation Description Timing Status
MetricsAugmentation Prometheus metrics after 📝 Example
LoggingAugmentation Structured logging after 📝 Example
TracingAugmentation Distributed tracing around 🚧 Planned

🤖 AI & Chat

AI-powered interfaces and chat capabilities.

Augmentation Description Timing Status
ChatInterfaceAugmentation Natural language interface before 📝 Example
MCPAgentMemoryAugmentation AI agent memory via MCP after 📝 Example
LLMQueryAugmentation LLM-enhanced queries before 📝 Example

📈 Visualization

Visual representations of data.

Augmentation Description Timing Status
GraphVisualizationAugmentation Real-time graph visualization after 📝 Example
DashboardAugmentation Web-based dashboard after 🚧 Planned

Status Legend

  • Production: Fully implemented and tested
  • 📝 Example: Example implementation available
  • 🚧 Planned: On the roadmap
  • ⚠️ Legacy: Being replaced by newer augmentations

Using Augmentations

Zero-Config Approach

const brain = new Brainy()

// Just register augmentations - they work automatically!
brain.augmentations.register(new EntityRegistryAugmentation())
brain.augmentations.register(new APIServerAugmentation())

await brain.init()

With Configuration

const brain = new Brainy()

brain.augmentations.register(
  new APIServerAugmentation({
    port: 8080,
    auth: { required: true }
  })
)

await brain.init()

Creating Custom Augmentations

See Creating Custom Augmentations for a complete guide.

Quick example:

class MyAugmentation extends BaseAugmentation {
  readonly name = 'my-augmentation'
  readonly timing = 'after'
  readonly operations = ['add', 'search']
  readonly priority = 50
  
  async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
    console.log(`Before ${operation}`)
    const result = await next()
    console.log(`After ${operation}`)
    return result
  }
}

Augmentation Timing

before

Executes before the main operation. Used for:

  • Input validation
  • Data transformation
  • Authentication checks

after

Executes after the main operation. Used for:

  • Broadcasting updates
  • Syncing to external services
  • Logging and metrics

around

Wraps the main operation. Used for:

  • Transactions
  • Caching
  • Error handling

replace

Completely replaces the main operation. Used for:

  • Alternative storage backends
  • Mock implementations
  • Proxy operations

Priority System

Higher numbers execute first:

  • 100: Critical system operations
  • 50: Performance optimizations
  • 10: Enhancement features
  • 1: Optional features