This commit completes the core v4.0.0 architecture changes for billion-scale performance with metadata/vector separation. NO RELEASE YET - remaining optimizations and testing required before production release. ## Core v4.0.0 Architecture Changes ### Type System Updates - Fixed all TypeScript compilation errors (zero errors achieved) - Updated HNSWNoun/HNSWVerb to separate core fields from metadata - Implemented HNSWNounWithMetadata/HNSWVerbWithMetadata for API boundaries - Added required 'noun' field to NounMetadata for semantic structure - Renamed verb.type to verb.verb for consistency ### Storage Adapter Updates **All adapters updated for v4.0.0 two-file storage pattern:** - memoryStorage: Proper metadata/vector separation - fileSystemStorage: Two-file pattern with sharding - opfsStorage: Browser persistent storage updated - s3CompatibleStorage: AWS/MinIO/DigitalOcean support - r2Storage: Cloudflare R2 optimization - gcsStorage: Google Cloud with ADC support - **azureBlobStorage: NEW - Full Azure Blob Storage support** ### Storage Features - BaseStorage: Internal vs public method separation (_getNoun vs getNoun) - Two-file storage: Vectors in one file, metadata in another - Change tracking: getChangesSince return type updated - Pagination: getNounsWithPagination returns WithMetadata types ### Azure Blob Storage Integration (NEW) - Native @azure/storage-blob SDK integration - Four authentication methods: * DefaultAzureCredential (Managed Identity) - recommended * Connection String - simplest setup * Account Name + Key - traditional auth * SAS Token - delegated access - High-volume mode with write buffering - Adaptive backpressure for throttling - UUID-based sharding for billion-scale - Full HNSW support with graph persistence ### Utility Updates - EmbeddingManager: Updated to accept Record<string, unknown> - LSMTree: Wrapped data in NounMetadata structure with 'noun' field - EntityIdMapper: Fixed nested metadata.data structure access - MetadataIndex: Fixed field type inference integration - PeriodicCleanup: Updated for new metadata structure ### Core API Updates - Brainy: Updated verb property access from v.type to v.verb - ConfigAPI: Fixed NounMetadata access patterns - DataAPI: Updated metadata handling ### Documentation Updates - CREATING-AUGMENTATIONS.md: v4.0.0 breaking changes guide - DEVELOPER-GUIDE.md: Migration checklist and examples - COMPLETE-REFERENCE.md: v4.0.0 architecture improvements - **finite-type-system.md: NEW - Revolutionary type system benefits** ### Build & Dependencies - Zero TypeScript compilation errors - Added @azure/storage-blob and @azure/identity - 591 tests passing (23 timeout in long-running neural tests) ## What's NOT in This Release This is a work-in-progress commit. Before v4.0.0 release we need: - Storage adapter optimizations (batch operations, compression) - Azure blob tier management (Hot/Cool/Archive) - Cost optimization implementations - Additional performance testing at billion-scale - Migration guides for v3.x users ## Testing - Clean build: ✅ - Type checking: ✅ (zero errors) - Test suite: ✅ (591/614 passing, timeouts in neural tests only) 🔐 Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| api-server.md | ||
| COMPLETE-REFERENCE.md | ||
| CONFIGURATION.md | ||
| DEVELOPER-GUIDE.md | ||
| README.md | ||
Brainy Augmentations
Augmentations are the core extensibility mechanism in Brainy. They allow you to modify, enhance, and extend Brainy's behavior without changing the core code.
Core Principle: One Interface, Infinite Possibilities
Every augmentation implements the same simple BrainyAugmentation interface:
interface BrainyAugmentation {
name: string
timing: 'before' | 'after' | 'around' | 'replace'
operations: string[]
priority: number
initialize(context): Promise<void>
execute(operation, params, next): Promise<any>
}
This single interface can handle EVERYTHING - from adding AI capabilities to exposing APIs to replacing storage backends.
Available Augmentations
🧠 Data Processing
Augmentations that enhance how data is processed and stored.
| Augmentation | Description | Timing | Status |
|---|---|---|---|
| NeuralImportAugmentation | AI-powered entity and relationship extraction | before |
✅ Production |
| EntityRegistryAugmentation | High-performance entity deduplication | before |
✅ Production |
| BatchProcessingAugmentation | Optimizes bulk operations | around |
✅ Production |
| IntelligentVerbScoringAugmentation | Learns relationship importance over time | after |
✅ Production |
🔌 External Connections (Synapses)
Connect Brainy to external services and data sources.
| Augmentation | Description | Timing | Status |
|---|---|---|---|
| NotionSynapse | Sync with Notion databases | after |
📝 Example |
| SalesforceSynapse | Connect to Salesforce CRM | after |
📝 Example |
| SlackSynapse | Import Slack conversations | after |
📝 Example |
| GoogleDriveSynapse | Sync Google Drive documents | after |
📝 Example |
🌐 API Exposure
Expose Brainy through various protocols.
| Augmentation | Description | Timing | Status |
|---|---|---|---|
| APIServerAugmentation | REST, WebSocket, and MCP server | after |
✅ Production |
| GraphQLAugmentation | GraphQL API endpoint | after |
🚧 Planned |
| gRPCAugmentation | gRPC service | after |
🚧 Planned |
💾 Storage Backends
Replace or enhance the storage layer.
| Augmentation | Description | Timing | Status |
|---|---|---|---|
| S3StorageAugmentation | Use S3 as storage backend | replace |
📝 Example |
| RedisAugmentation | Redis caching layer | around |
📝 Example |
| PostgresAugmentation | PostgreSQL persistence | replace |
📝 Example |
🔄 Real-time & Sync
Handle real-time updates and synchronization.
| Augmentation | Description | Timing | Status |
|---|---|---|---|
| WebSocketConduitAugmentation | WebSocket client connections | after |
⚠️ Legacy |
| ServerSearchAugmentation | Connect to remote Brainy servers | after |
⚠️ Legacy |
| TeamCoordinationAugmentation | Multi-agent synchronization | after |
📝 Example |
🛡️ Infrastructure
Core infrastructure and reliability features.
| Augmentation | Description | Timing | Status |
|---|---|---|---|
| ConnectionPoolAugmentation | Optimize cloud storage connections | before |
✅ Production |
| RequestDeduplicatorAugmentation | Prevent duplicate concurrent requests | before |
✅ Production |
| TransactionAugmentation | ACID transaction support | around |
🚧 Planned |
| CacheAugmentation | Multi-level caching | around |
✅ Production |
📊 Monitoring & Analytics
Track and analyze Brainy's behavior.
| Augmentation | Description | Timing | Status |
|---|---|---|---|
| MetricsAugmentation | Prometheus metrics | after |
📝 Example |
| LoggingAugmentation | Structured logging | after |
📝 Example |
| TracingAugmentation | Distributed tracing | around |
🚧 Planned |
🤖 AI & Chat
AI-powered interfaces and chat capabilities.
| Augmentation | Description | Timing | Status |
|---|---|---|---|
| ChatInterfaceAugmentation | Natural language interface | before |
📝 Example |
| MCPAgentMemoryAugmentation | AI agent memory via MCP | after |
📝 Example |
| LLMQueryAugmentation | LLM-enhanced queries | before |
📝 Example |
📈 Visualization
Visual representations of data.
| Augmentation | Description | Timing | Status |
|---|---|---|---|
| GraphVisualizationAugmentation | Real-time graph visualization | after |
📝 Example |
| DashboardAugmentation | Web-based dashboard | after |
🚧 Planned |
Status Legend
- ✅ Production: Fully implemented and tested
- 📝 Example: Example implementation available
- 🚧 Planned: On the roadmap
- ⚠️ Legacy: Being replaced by newer augmentations
Using Augmentations
Zero-Config Approach
const brain = new Brainy()
// Just register augmentations - they work automatically!
brain.augmentations.register(new EntityRegistryAugmentation())
brain.augmentations.register(new APIServerAugmentation())
await brain.init()
With Configuration
const brain = new Brainy()
brain.augmentations.register(
new APIServerAugmentation({
port: 8080,
auth: { required: true }
})
)
await brain.init()
Creating Custom Augmentations
See Creating Custom Augmentations for a complete guide.
Quick example:
class MyAugmentation extends BaseAugmentation {
readonly name = 'my-augmentation'
readonly timing = 'after'
readonly operations = ['add', 'search']
readonly priority = 50
async execute<T>(operation: string, params: any, next: () => Promise<T>): Promise<T> {
console.log(`Before ${operation}`)
const result = await next()
console.log(`After ${operation}`)
return result
}
}
Augmentation Timing
before
Executes before the main operation. Used for:
- Input validation
- Data transformation
- Authentication checks
after
Executes after the main operation. Used for:
- Broadcasting updates
- Syncing to external services
- Logging and metrics
around
Wraps the main operation. Used for:
- Transactions
- Caching
- Error handling
replace
Completely replaces the main operation. Used for:
- Alternative storage backends
- Mock implementations
- Proxy operations
Priority System
Higher numbers execute first:
- 100: Critical system operations
- 50: Performance optimizations
- 10: Enhancement features
- 1: Optional features
Related Documentation
- API Server Augmentation - Complete API server documentation
- Creating Augmentations - How to build your own
- Augmentation Examples - Real-world examples
- Architecture Overview - System architecture