diff --git a/README.md b/README.md index ca47764e..c9803005 100644 --- a/README.md +++ b/README.md @@ -13,10 +13,14 @@ ## โœจ What is Brainy? -Imagine a database that thinks like you do - connecting ideas, finding patterns, and getting smarter over time. Brainy is the **AI-native database** that brings vector search and knowledge graphs together in one powerful, ridiculously easy-to-use package. +Imagine a database that thinks like you do - connecting ideas, finding patterns, and getting smarter over time. Brainy +is the **AI-native database** that brings vector search and knowledge graphs together in one powerful, ridiculously +easy-to-use package. ### ๐Ÿ†• NEW: Distributed Mode (v0.38+) + **Scale horizontally with zero configuration!** Brainy now supports distributed deployments with automatic coordination: + - **๐ŸŒ Multi-Instance Coordination** - Multiple readers and writers working in harmony - **๐Ÿท๏ธ Smart Domain Detection** - Automatically categorizes data (medical, legal, product, etc.) - **๐Ÿ“Š Real-Time Health Monitoring** - Track performance across all instances @@ -25,43 +29,78 @@ Imagine a database that thinks like you do - connecting ideas, finding patterns, ### ๐Ÿš€ Why Developers Love Brainy -- **๐Ÿง  It Just Worksโ„ข** - No config files, no tuning parameters, no DevOps headaches. Brainy auto-detects your environment and optimizes itself -- **๐ŸŒ True Write-Once, Run-Anywhere** - Same code runs in React, Angular, Vue, Node.js, Deno, Bun, serverless, edge workers, and even vanilla HTML +- **๐Ÿง  Zero-to-Smartโ„ข** - No config files, no tuning parameters, no DevOps headaches. Brainy auto-detects your + environment and optimizes itself +- **๐ŸŒ True Write-Once, Run-Anywhere** - Same code runs in React, Angular, Vue, Node.js, Deno, Bun, serverless, edge + workers, and even vanilla HTML - **โšก Scary Fast** - Handles millions of vectors with sub-millisecond search. Built-in GPU acceleration when available - **๐ŸŽฏ Self-Learning** - Like having a database that goes to the gym. Gets faster and smarter the more you use it - **๐Ÿ”ฎ AI-First Design** - Built for the age of embeddings, RAG, and semantic search. Your LLMs will thank you - **๐ŸŽฎ Actually Fun to Use** - Clean API, great DX, and it does the heavy lifting so you can build cool stuff +### ๐Ÿš€ NEW: Ultra-Fast Search Performance + Auto-Configuration + +**Your searches just got 100x faster AND Brainy now configures itself!** Advanced performance with zero setup: + +- **๐Ÿค– Intelligent Auto-Configuration** - Detects environment and usage patterns, optimizes automatically +- **โšก Smart Result Caching** - Repeated queries return in <1ms with automatic cache invalidation +- **๐Ÿ“„ Cursor-Based Pagination** - Navigate millions of results with constant O(k) performance +- **๐Ÿ”„ Real-Time Data Sync** - Cache automatically updates when data changes, even in distributed scenarios +- **๐Ÿ“Š Performance Monitoring** - Built-in hit rate and memory usage tracking with adaptive optimization +- **๐ŸŽฏ Zero Breaking Changes** - All existing code works unchanged, just faster and smarter + +```javascript +// Zero configuration - everything optimized automatically! +const brainy = new BrainyData() // Auto-detects environment & optimizes +await brainy.init() + +// Caching happens automatically - no setup needed! +const results1 = await brainy.search('query', 10) // ~50ms first time +const results2 = await brainy.search('query', 10) // <1ms cached hit! + +// Advanced pagination works instantly +const page1 = await brainy.searchWithCursor('query', 100) +const page2 = await brainy.searchWithCursor('query', 100, { + cursor: page1.cursor // Constant time, no matter how deep! +}) + +// Monitor auto-optimized performance +const stats = brainy.getCacheStats() +console.log(`Auto-tuned cache hit rate: ${(stats.search.hitRate * 100).toFixed(1)}%`) +``` + ## ๐Ÿš€ Quick Start (30 seconds!) ### Node.js TLDR + ```bash # Install npm install brainy # Use it ``` + ```javascript import { createAutoBrainy, NounType, VerbType } from 'brainy' const brainy = createAutoBrainy() // Add data with Nouns (entities) -const catId = await brainy.add("Siamese cats are elegant and vocal", { +const catId = await brainy.add("Siamese cats are elegant and vocal", { noun: NounType.Thing, breed: "Siamese", - category: "animal" + category: "animal" }) -const ownerId = await brainy.add("John loves his pets", { +const ownerId = await brainy.add("John loves his pets", { noun: NounType.Person, - name: "John Smith" + name: "John Smith" }) // Connect with Verbs (relationships) -await brainy.addVerb(ownerId, catId, { +await brainy.addVerb(ownerId, catId, { verb: VerbType.Owns, - since: "2020-01-01" + since: "2020-01-01" }) // Search by meaning @@ -78,24 +117,25 @@ const docs = await brainy.searchDocuments("Siamese", { const johnsPets = await brainy.getVerbsBySource(ownerId, VerbType.Owns) ``` -That's it! No config, no setup, it just worksโ„ข +That's it! No config, no setup, Zero-to-Smartโ„ข ### ๐ŸŒ Distributed Mode Example (NEW!) + ```javascript // Writer Instance - Ingests data from multiple sources const writer = createAutoBrainy({ - storage: { type: 's3', bucket: 'my-bucket' }, + storage: { s3Storage: { bucketName: 'my-bucket' } }, distributed: { role: 'writer' } // Explicit role for safety }) // Reader Instance - Optimized for search queries const reader = createAutoBrainy({ - storage: { type: 's3', bucket: 'my-bucket' }, + storage: { s3Storage: { bucketName: 'my-bucket' } }, distributed: { role: 'reader' } // 80% memory for cache }) // Data automatically gets domain tags -await writer.add("Patient shows symptoms of...", { +await writer.add("Patient shows symptoms of...", { diagnosis: "flu" // Auto-tagged as 'medical' domain }) @@ -112,6 +152,7 @@ console.log(`Instance ${health.instanceId}: ${health.status}`) ## ๐ŸŽญ Key Features ### Core Capabilities + - **Vector Search** - Find semantically similar content using embeddings - **Graph Relationships** - Connect data with meaningful relationships - **JSON Document Search** - Search within specific fields with prioritization @@ -121,15 +162,20 @@ console.log(`Instance ${health.instanceId}: ${health.status}`) - **Model Control Protocol** - Let AI models access your data ### Smart Optimizations -- **Auto-Configuration** - Detects environment and optimizes automatically -- **Adaptive Learning** - Gets smarter with usage, optimizes itself over time -- **Intelligent Partitioning** - Hash-based partitioning for perfect load distribution -- **Role-Based Optimization** - Readers maximize cache, writers optimize throughput -- **Domain-Aware Indexing** - Automatic categorization improves search relevance -- **Multi-Level Caching** - Hot/warm/cold caching with predictive prefetching -- **Memory Optimization** - 75% reduction with compression for large datasets + +- **๐Ÿค– Intelligent Auto-Configuration** - Detects environment, usage patterns, and optimizes everything automatically +- **โšก Runtime Performance Adaptation** - Continuously monitors and self-tunes based on real usage +- **๐ŸŒ Distributed Mode Detection** - Automatically enables real-time updates for shared storage scenarios +- **๐Ÿ“Š Workload-Aware Optimization** - Adapts cache size and TTL based on read/write patterns +- **๐Ÿง  Adaptive Learning** - Gets smarter with usage, learns from your data access patterns +- **#๏ธโƒฃ Intelligent Partitioning** - Hash-based partitioning for perfect load distribution +- **๐ŸŽฏ Role-Based Optimization** - Readers maximize cache, writers optimize throughput +- **๐Ÿท๏ธ Domain-Aware Indexing** - Automatic categorization improves search relevance +- **๐Ÿ—‚๏ธ Multi-Level Caching** - Hot/warm/cold caching with predictive prefetching +- **๐Ÿ’พ Memory Optimization** - 75% reduction with compression for large datasets ### Developer Experience + - **TypeScript Support** - Fully typed API with generics - **Extensible Augmentations** - Customize and extend functionality - **REST API** - Web service wrapper for HTTP endpoints @@ -138,16 +184,20 @@ console.log(`Instance ${health.instanceId}: ${health.status}`) ## ๐Ÿ“ฆ Installation ### Main Package + ```bash npm install brainy ``` ### Optional: Offline Models Package + ```bash npm install @soulcraft/brainy-models ``` -The `@soulcraft/brainy-models` package provides **offline access** to the Universal Sentence Encoder model, eliminating network dependencies and ensuring consistent performance. Perfect for: +The `@soulcraft/brainy-models` package provides **offline access** to the Universal Sentence Encoder model, eliminating +network dependencies and ensuring consistent performance. Perfect for: + - **Air-gapped environments** - No internet? No problem - **Consistent performance** - No network latency or throttling - **Privacy-focused apps** - Keep everything local @@ -166,7 +216,8 @@ const brainy = createAutoBrainy({ ## ๐ŸŽจ Build Amazing Things **๐Ÿค– AI Chat Applications** - Build ChatGPT-like apps with long-term memory and context awareness -**๐Ÿ” Semantic Search Engines** - Search by meaning, not keywords. Find "that thing that's like a cat but bigger" โ†’ returns "tiger" +**๐Ÿ” Semantic Search Engines** - Search by meaning, not keywords. Find "that thing that's like a cat but bigger" โ†’ +returns "tiger" **๐ŸŽฏ Recommendation Engines** - "Users who liked this also liked..." but actually good **๐Ÿงฌ Knowledge Graphs** - Connect everything to everything. Wikipedia meets Neo4j meets magic **๐Ÿ‘๏ธ Computer Vision Apps** - Store and search image embeddings. "Find all photos with dogs wearing hats" @@ -178,11 +229,13 @@ const brainy = createAutoBrainy({ ## ๐Ÿงฌ The Power of Nouns & Verbs -Brainy uses a **graph-based data model** that mirrors how humans think - with **Nouns** (entities) connected by **Verbs** (relationships). This isn't just vectors in a void; it's structured, meaningful data. +Brainy uses a **graph-based data model** that mirrors how humans think - with **Nouns** (entities) connected by **Verbs +** (relationships). This isn't just vectors in a void; it's structured, meaningful data. ### ๐Ÿ“ Nouns (What Things Are) Nouns are your entities - the "things" in your data. Each noun has: + - A unique ID - A vector representation (for similarity search) - A type (Person, Document, Concept, etc.) @@ -190,28 +243,29 @@ Nouns are your entities - the "things" in your data. Each noun has: **Available Noun Types:** -| Category | Types | Use For | -|----------|-------|---------| -| **Core Entities** | `Person`, `Organization`, `Location`, `Thing`, `Concept`, `Event` | People, companies, places, objects, ideas, happenings | -| **Digital Content** | `Document`, `Media`, `File`, `Message`, `Content` | PDFs, images, videos, emails, posts, generic content | -| **Collections** | `Collection`, `Dataset` | Groups of items, structured data sets | -| **Business** | `Product`, `Service`, `User`, `Task`, `Project` | E-commerce, SaaS, project management | -| **Descriptive** | `Process`, `State`, `Role` | Workflows, conditions, responsibilities | +| Category | Types | Use For | +|---------------------|-------------------------------------------------------------------|-------------------------------------------------------| +| **Core Entities** | `Person`, `Organization`, `Location`, `Thing`, `Concept`, `Event` | People, companies, places, objects, ideas, happenings | +| **Digital Content** | `Document`, `Media`, `File`, `Message`, `Content` | PDFs, images, videos, emails, posts, generic content | +| **Collections** | `Collection`, `Dataset` | Groups of items, structured data sets | +| **Business** | `Product`, `Service`, `User`, `Task`, `Project` | E-commerce, SaaS, project management | +| **Descriptive** | `Process`, `State`, `Role` | Workflows, conditions, responsibilities | ### ๐Ÿ”— Verbs (How Things Connect) -Verbs are your relationships - they give meaning to connections. Not just "these vectors are similar" but "this OWNS that" or "this CAUSES that". +Verbs are your relationships - they give meaning to connections. Not just "these vectors are similar" but "this OWNS +that" or "this CAUSES that". **Available Verb Types:** -| Category | Types | Examples | -|----------|-------|----------| -| **Core** | `RelatedTo`, `Contains`, `PartOf`, `LocatedAt`, `References` | Generic relations, containment, location | -| **Temporal** | `Precedes`, `Succeeds`, `Causes`, `DependsOn`, `Requires` | Time sequences, causality, dependencies | -| **Creation** | `Creates`, `Transforms`, `Becomes`, `Modifies`, `Consumes` | Creation, change, consumption | -| **Ownership** | `Owns`, `AttributedTo`, `CreatedBy`, `BelongsTo` | Ownership, authorship, belonging | -| **Social** | `MemberOf`, `WorksWith`, `FriendOf`, `Follows`, `Likes`, `ReportsTo` | Social networks, organizations | -| **Functional** | `Describes`, `Implements`, `Validates`, `Triggers`, `Serves` | Functions, implementations, services | +| Category | Types | Examples | +|----------------|----------------------------------------------------------------------|------------------------------------------| +| **Core** | `RelatedTo`, `Contains`, `PartOf`, `LocatedAt`, `References` | Generic relations, containment, location | +| **Temporal** | `Precedes`, `Succeeds`, `Causes`, `DependsOn`, `Requires` | Time sequences, causality, dependencies | +| **Creation** | `Creates`, `Transforms`, `Becomes`, `Modifies`, `Consumes` | Creation, change, consumption | +| **Ownership** | `Owns`, `AttributedTo`, `CreatedBy`, `BelongsTo` | Ownership, authorship, belonging | +| **Social** | `MemberOf`, `WorksWith`, `FriendOf`, `Follows`, `Likes`, `ReportsTo` | Social networks, organizations | +| **Functional** | `Describes`, `Implements`, `Validates`, `Triggers`, `Serves` | Functions, implementations, services | ### ๐Ÿ’ก Why This Matters @@ -221,16 +275,16 @@ const similar = await vectorDB.search(embedding, 10) // Result: [vector1, vector2, ...] - What do these mean? ๐Ÿคท // Brainy: Similarity + Meaning + Relationships -const catId = await brainy.add("Siamese cat", { +const catId = await brainy.add("Siamese cat", { noun: NounType.Thing, - breed: "Siamese" + breed: "Siamese" }) -const ownerId = await brainy.add("John Smith", { - noun: NounType.Person +const ownerId = await brainy.add("John Smith", { + noun: NounType.Person }) -await brainy.addVerb(ownerId, catId, { +await brainy.addVerb(ownerId, catId, { verb: VerbType.Owns, - since: "2020-01-01" + since: "2020-01-01" }) // Now you can search with context! @@ -240,44 +294,45 @@ const catOwners = await brainy.getVerbsByTarget(catId, VerbType.Owns) ## ๐ŸŒ Distributed Mode (New!) -Brainy now supports **distributed deployments** with multiple specialized instances sharing the same data. Perfect for scaling your AI applications across multiple servers. +Brainy now supports **distributed deployments** with multiple specialized instances sharing the same data. Perfect for +scaling your AI applications across multiple servers. ### Distributed Setup ```javascript // Single instance (no change needed!) const brainy = createAutoBrainy({ - storage: { type: 's3', bucket: 'my-bucket' } + storage: { s3Storage: { bucketName: 'my-bucket' } } }) // Distributed mode requires explicit role configuration // Option 1: Via environment variable process.env.BRAINY_ROLE = 'writer' // or 'reader' or 'hybrid' const brainy = createAutoBrainy({ - storage: { type: 's3', bucket: 'my-bucket' }, + storage: { s3Storage: { bucketName: 'my-bucket' } }, distributed: true }) // Option 2: Via configuration const writer = createAutoBrainy({ - storage: { type: 's3', bucket: 'my-bucket' }, + storage: { s3Storage: { bucketName: 'my-bucket' } }, distributed: { role: 'writer' } // Handles data ingestion }) const reader = createAutoBrainy({ - storage: { type: 's3', bucket: 'my-bucket' }, + storage: { s3Storage: { bucketName: 'my-bucket' } }, distributed: { role: 'reader' } // Optimized for queries }) // Option 3: Via read/write mode (role auto-inferred) const writer = createAutoBrainy({ - storage: { type: 's3', bucket: 'my-bucket' }, + storage: { s3Storage: { bucketName: 'my-bucket' } }, writeOnly: true, // Automatically becomes 'writer' role distributed: true }) const reader = createAutoBrainy({ - storage: { type: 's3', bucket: 'my-bucket' }, + storage: { s3Storage: { bucketName: 'my-bucket' } }, readOnly: true, // Automatically becomes 'reader' role distributed: true }) @@ -286,25 +341,28 @@ const reader = createAutoBrainy({ ### Key Distributed Features **๐ŸŽฏ Explicit Role Configuration** + - Roles must be explicitly set (no dangerous auto-assignment) - Can use environment variables, config, or read/write modes - Clear separation between writers and readers **#๏ธโƒฃ Hash-Based Partitioning** + - Handles multiple writers with different data types - Even distribution across partitions - No semantic conflicts with mixed data **๐Ÿท๏ธ Domain Tagging** + - Automatic domain detection (medical, legal, product, etc.) - Filter searches by domain - Logical separation without complexity ```javascript // Data is automatically tagged with domains -await brainy.add({ - symptoms: "fever", - diagnosis: "flu" +await brainy.add({ + symptoms: "fever", + diagnosis: "flu" }, metadata) // Auto-tagged as 'medical' // Search within specific domains @@ -314,6 +372,7 @@ const medicalResults = await brainy.search(query, 10, { ``` **๐Ÿ“Š Health Monitoring** + - Real-time health metrics - Automatic dead instance cleanup - Performance tracking @@ -331,6 +390,7 @@ const health = brainy.getHealthStatus() ``` **โšก Role-Optimized Performance** + - **Readers**: 80% memory for cache, aggressive prefetching - **Writers**: Optimized write batching, minimal cache - **Hybrid**: Adaptive based on workload @@ -338,13 +398,14 @@ const health = brainy.getHealthStatus() ### Deployment Examples **Docker Compose** + ```yaml services: writer: image: myapp environment: BRAINY_ROLE: writer # Optional - auto-detects - + reader: image: myapp environment: @@ -353,6 +414,7 @@ services: ``` **Kubernetes** + ```yaml # Automatically detects role from deployment type apiVersion: apps/v1 @@ -364,12 +426,13 @@ spec: template: spec: containers: - - name: app - image: myapp - # Role auto-detected as 'reader' (multiple replicas) + - name: app + image: myapp + # Role auto-detected as 'reader' (multiple replicas) ``` **Benefits** + - โœ… **50-70% faster searches** with parallel readers - โœ… **No coordination complexity** - Shared JSON config in S3 - โœ… **Zero downtime scaling** - Add/remove instances anytime @@ -378,18 +441,22 @@ spec: ## ๐Ÿค” Why Choose Brainy? ### vs. Traditional Databases + โŒ **PostgreSQL with pgvector** - Requires complex setup, tuning, and DevOps expertise โœ… **Brainy** - Zero config, auto-optimizes, works everywhere from browser to cloud ### vs. Vector Databases + โŒ **Pinecone/Weaviate/Qdrant** - Cloud-only, expensive, vendor lock-in โœ… **Brainy** - Run locally, in browser, or cloud. Your choice, your data ### vs. Graph Databases + โŒ **Neo4j** - Great for graphs, no vector support โœ… **Brainy** - Vectors + graphs in one. Best of both worlds ### vs. DIY Solutions + โŒ **Building your own** - Months of work, optimization nightmares โœ… **Brainy** - Production-ready in 30 seconds @@ -411,7 +478,7 @@ function SemanticSearch() { } return ( - search(e.target.value)} + search(e.target.value)} placeholder="Search by meaning..." /> ) } @@ -446,20 +513,21 @@ export class SearchComponent implements OnInit { ### Vue 3 ```vue +