brainy/docs/architecture/overview.md

Architecture Overview

Brainy is a multi-dimensional AI database that combines vector similarity, graph relationships, and metadata filtering into a unified query system. This document provides a comprehensive overview of the system architecture.

Core Components

BrainyData (Main Entry Point)

The central orchestrator that manages all subsystems:

• HNSW Index: O(log n) vector similarity search
• Storage System: Universal storage adapters (FileSystem, S3, OPFS, Memory)
• Metadata Index: O(1) field lookups with inverted indexing
• Augmentation System: Extensible plugin architecture
• Triple Intelligence: Unified query engine

Triple Intelligence Engine

Brainy's revolutionary feature that unifies three types of search:

• Vector Search: Semantic similarity using HNSW indexing
• Graph Traversal: Relationship-based queries
• Field Filtering: Precise metadata filtering with O(1) performance

// Single query combining all three intelligence types
const results = await brain.find({
  like: "machine learning papers",              // Vector similarity
  connected: { to: "research-team", depth: 2 }, // Graph traversal  
  where: { published: { $gte: "2024-01-01" } }  // Metadata filtering
})

Storage Architecture

brainy-data/
├── _system/           # System management
│   └── statistics.json
├── nouns/            # Entity data storage
│   └── {uuid}.json
├── metadata/         # Metadata and indexing
│   ├── {uuid}.json
│   ├── __entity_registry__.json
│   └── __metadata_index__*.json
├── verbs/            # Relationship storage
├── wal/              # Write-Ahead Logging
└── locks/            # Concurrent access control

HNSW Index

Hierarchical Navigable Small World index for efficient vector search:

• Performance: O(log n) search complexity
• Memory Efficient: Product quantization support
• Scalable: Handles millions of vectors
• Persistent: Serializable to storage

Metadata Index Manager

High-performance field indexing system:

• O(1) Lookups: Inverted index for field→value→IDs mapping
• Query Support: equals, anyOf, allOf, range queries
• Chunked Storage: Supports massive datasets
• Auto-indexing: Automatically maintains indexes on updates

Performance Characteristics

Operation Complexity

• Vector Search: O(log n) via HNSW
• Field Filtering: O(1) via inverted indexes
• Graph Traversal: O(V + E) for breadth-first search
• Add Operation: O(log n) for index insertion
• Update Operation: O(1) for metadata updates

Memory Usage

• Base Memory: ~50MB for core system
• Per Vector: ~1KB (384 dimensions × 4 bytes)
• Index Overhead: ~20% of vector data
• Cache Size: Configurable (default 1000 entries)

Throughput

• Writes: 1000+ ops/second (with batching)
• Reads: 10,000+ ops/second
• Search: 100+ queries/second (varies by complexity)

Augmentation System

Brainy's extensible plugin architecture allows for powerful enhancements:

Core Augmentations

• WAL (Write-Ahead Logging): Durability and crash recovery
• Entity Registry: High-speed deduplication for streaming data
• Batch Processing: Optimized bulk operations
• Connection Pool: Efficient resource management
• Request Deduplicator: Prevents duplicate processing

Creating Custom Augmentations

class CustomAugmentation extends BrainyAugmentation {
  async onInit(brain: BrainyData): Promise<void> {
    // Initialize augmentation
  }
  
  async onAdd(item: any, brain: BrainyData): Promise<any> {
    // Process item before adding
    return item
  }
}

Caching Strategy

Multi-layered caching for optimal performance:

• Search Cache: LRU cache for query results
• Metadata Cache: Field index caching
• Pattern Cache: NLP pattern matching cache
• Entity Cache: In-memory entity registry

Integration Points

Key Objects for Extensions

• brain.index: Access HNSW vector index
• brain.metadataIndex: Access field indexing
• brain.storage: Access storage layer
• brain.augmentations: Access augmentation manager

Event System

brain.on('add', (item) => console.log('Item added:', item))
brain.on('search', (query) => console.log('Search performed:', query))
brain.on('error', (error) => console.error('Error:', error))

Best Practices

When Adding Features

1. Check if similar functionality exists
2. Consider if it should be an augmentation
3. Use existing indexes and caches
4. Avoid duplicating functionality
5. Follow the established patterns

Performance Optimization

1. Use batch operations for bulk data
2. Enable appropriate caching
3. Choose the right storage adapter
4. Configure index parameters for your use case
5. Monitor statistics for bottlenecks

Next Steps

• Storage Architecture - Deep dive into storage system
• Triple Intelligence - Advanced query system
• API Reference - Complete API documentation