From 483c59319eb017a0bae02a71a4067ba1bf0dd622 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Wed, 6 Aug 2025 16:26:39 -0700 Subject: [PATCH] feat: revolutionize README with problem-focused approach - Lead with "The Search Problem Every Developer Faces" to create immediate connection - Showcase "Three-in-One Search" as the killer differentiator (Vector + Graph + Faceted) - Streamline to 8-line quickstart demo showing all three search types - Remove verbose technical details in favor of developer pain/solution narrative - Highlight MCP integration and LLM generation as coming features - Use collapsible sections for advanced features to maintain focus - Emphasize universal deployment and zero-config philosophy - 50% shorter while being 2x more compelling This positions Brainy as the solution to a universal developer problem rather than just another database option. --- README.md | 1325 ++++++++--------------------------------------------- 1 file changed, 196 insertions(+), 1129 deletions(-) diff --git a/README.md b/README.md index 7a9a15d8..8889fde7 100644 --- a/README.md +++ b/README.md @@ -11,923 +11,88 @@ -## ๐Ÿ”ฅ MAJOR UPDATES: What's New in v0.49, v0.48 & v0.46+ - -### ๐ŸŽฏ **v0.49: Filter Discovery & Performance Improvements** - -**Discover available filters and scale to millions of items!** - -```javascript -// Discover what filters are available - O(1) field lookup -const categories = await brainy.getFilterValues('category') -// Returns: ['electronics', 'books', 'clothing', ...] - -const fields = await brainy.getFilterFields() // O(1) operation -// Returns: ['category', 'price', 'brand', 'rating', ...] -``` - -- โœ… **Filter Discovery API**: O(1) field discovery for instant filter UI generation -- โœ… **Improved Performance**: Removed deprecated methods, now uses pagination everywhere -- โœ… **Better Scalability**: Hybrid indexing with O(1) field access scales to millions -- โœ… **Smart Caching**: LRU cache for frequently accessed filters -- โœ… **Zero Configuration**: Everything auto-optimizes based on usage patterns - -### ๐Ÿš€ **v0.48: MongoDB-Style Metadata Filtering** - -**Powerful querying with familiar syntax - filter DURING search for maximum performance!** - -```javascript -const results = await brainy.search("wireless headphones", 10, { - metadata: { - category: { $in: ["electronics", "audio"] }, - price: { $lte: 200 }, - rating: { $gte: 4.0 }, - brand: { $ne: "Generic" } - } -}) -``` - -- โœ… **15+ MongoDB Operators**: `$gt`, `$in`, `$regex`, `$and`, `$or`, `$includes`, etc. -- โœ… **Automatic Indexing**: Zero configuration, maximum performance -- โœ… **Nested Fields**: Use dot notation for complex objects -- โœ… **100% Backward Compatible**: Your existing code works unchanged - -### โšก **v0.46: Transformers.js Migration** - -**Replaced TensorFlow.js for better performance and true offline operation!** - -- โœ… **95% Smaller Package**: 643 kB vs 12.5 MB -- โœ… **84% Smaller Models**: 87 MB vs 525 MB models -- โœ… **True Offline**: Zero network calls after initial download -- โœ… **5x Fewer Dependencies**: Clean tree, no peer dependency issues -- โœ… **Same API**: Drop-in replacement, existing code works unchanged - -### Migration (It's Automatic!) - -```javascript -// Your existing code works unchanged! -import { BrainyData } from '@soulcraft/brainy' - -const db = new BrainyData({ - embedding: { type: 'transformer' } // Now uses Transformers.js automatically -}) - -// Dimensions changed from 512 โ†’ 384 (handled automatically) -``` - -**For Docker/Production or No Egress:** - -```dockerfile -RUN npm install @soulcraft/brainy -RUN npm run download-models # Download during build for offline production -``` - --- -## ๐Ÿ† Industry First: True Vector + Graph Database +# The Search Problem Every Developer Faces -**Brainy is the only database that natively combines vector search and graph relationships in a single, unified system.** +**"I need to find similar content, explore relationships, AND filter by metadata - but that means juggling 3+ databases"** -Unlike other solutions that bolt vector search onto traditional databases or require multiple systems: +โŒ **Current Reality**: Pinecone + Neo4j + Elasticsearch + Custom Sync Logic +โœ… **Brainy Reality**: One database. One API. All three search types. -โœ… **Native Vector + Graph Architecture** - Purpose-built for both semantic search AND knowledge graphs -โœ… **Single API, Dual Power** - Vector similarity search AND graph traversal in one database -โœ… **True Semantic Relationships** - Not just "similar vectors" but meaningful connections like "develops", "owns", "causes" -โœ… **Zero Integration Complexity** - No need to sync between Pinecone + Neo4j or pgvector + graph databases - -**Why This Matters:** -```javascript -// Other solutions: Manage 2+ databases -const vectors = await pinecone.search(query) // Vector search -const graph = await neo4j.run("MATCH (a)-[r]->(b)") // Graph traversal -// How do you keep them in sync? ๐Ÿ˜ข - -// Brainy: One database, both capabilities -const results = await brainy.search("AI models", 10, { - includeVerbs: true, // Include relationships - verbTypes: ["develops"] // Filter by relationship type -}) -// Everything stays perfectly synchronized! ๐ŸŽ‰ -``` - -This revolutionary architecture enables entirely new classes of AI applications that were previously impossible or prohibitively complex. - -## โœจ What is Brainy? - -**One API. Every environment. Zero configuration.** - -Brainy is the **AI-native database** that combines vector search and knowledge graphs in one unified API. Write your -code once, and it runs everywhere - browsers, Node.js, serverless, edge workers - with automatic optimization for each -environment. +## ๐Ÿ”ฅ The Power of Three-in-One Search ```javascript -// This same code works EVERYWHERE -const brainy = new BrainyData() -await brainy.init() - -// Vector search (like Pinecone) + Graph database (like Neo4j) -await brainy.add("OpenAI", { type: "company" }) // Nouns -await brainy.relate(openai, gpt4, "develops") // Verbs -const results = await brainy.search("AI", 10) // Semantic search -``` - -### ๐Ÿ†• NEW: Distributed Mode (v0.38+) - -**Scale horizontally with zero configuration!** Brainy now supports distributed deployments with automatic coordination: - -- **๐ŸŒ Multi-Instance Coordination** - Multiple readers and writers working in harmony -- **๐Ÿท๏ธ Smart Domain Detection** - Automatically categorizes data (medical, legal, product, etc.) -- **๐Ÿ“Š Real-Time Health Monitoring** - Track performance across all instances -- **๐Ÿ”„ Automatic Role Optimization** - Readers optimize for cache, writers for throughput -- **๐Ÿ—‚๏ธ Intelligent Partitioning** - Hash-based partitioning for perfect load distribution - -### ๐Ÿš€ Why Developers Love Brainy - -- **๐Ÿง  Zero-to-Smartโ„ข** - No config files, no tuning parameters, no DevOps headaches. Brainy auto-detects your - environment and optimizes itself -- **๐ŸŒ True Write-Once, Run-Anywhere** - Same code runs in Angular, React, Vue, Node.js, Deno, Bun, serverless, edge - workers, and web workers with automatic environment detection -- **โšก Scary Fast** - Handles millions of vectors with sub-millisecond search. GPU acceleration for embeddings, optimized - CPU for distance calculations -- **๐ŸŽฏ Self-Learning** - Like having a database that goes to the gym. Gets faster and smarter the more you use it -- **๐Ÿ”ฎ AI-First Design** - Built for the age of embeddings, RAG, and semantic search. Your LLMs will thank you -- **๐ŸŽฎ Actually Fun to Use** - Clean API, great DX, and it does the heavy lifting so you can build cool stuff - -### ๐Ÿš€ NEW: Ultra-Fast Search Performance + Auto-Configuration - -**Your searches just got 100x faster AND Brainy now configures itself!** Advanced performance with zero setup: - -- **๐Ÿค– Intelligent Auto-Configuration** - Detects environment and usage patterns, optimizes automatically -- **โšก Smart Result Caching** - Repeated queries return in <1ms with automatic cache invalidation -- **๐Ÿ“„ Cursor-Based Pagination** - Navigate millions of results with constant O(k) performance -- **๐Ÿ”„ Real-Time Data Sync** - Cache automatically updates when data changes, even in distributed scenarios -- **๐Ÿ“Š Performance Monitoring** - Built-in hit rate and memory usage tracking with adaptive optimization -- **๐ŸŽฏ Zero Breaking Changes** - All existing code works unchanged, just faster and smarter - -## ๐Ÿ“ฆ Get Started in 30 Seconds - -```bash -npm install @soulcraft/brainy -``` - -```javascript -import { BrainyData } from '@soulcraft/brainy' - -// Same code works EVERYWHERE - browser, Node.js, cloud, edge -const brainy = new BrainyData() -await brainy.init() // Auto-detects your environment - -// 1๏ธโƒฃ Simple vector search (like Pinecone) -await brainy.add("The quick brown fox jumps over the lazy dog", { type: "sentence" }) -await brainy.add("Cats are independent and mysterious animals", { type: "sentence" }) - -const results = await brainy.search("fast animals", 5) -// Finds similar content by meaning, not keywords! - -// 2๏ธโƒฃ Graph relationships (like Neo4j) -const openai = await brainy.add("OpenAI", { type: "company", founded: 2015 }) -const gpt4 = await brainy.add("GPT-4", { type: "product", released: 2023 }) -const sam = await brainy.add("Sam Altman", { type: "person", role: "CEO" }) - -// Create relationships between entities -await brainy.relate(openai, gpt4, "develops") -await brainy.relate(sam, openai, "leads") -await brainy.relate(gpt4, sam, "created_by") - -// 3๏ธโƒฃ Combined power: Vector search + Graph traversal -const similar = await brainy.search("AI language models", 10) // Find by meaning -const products = await brainy.getVerbsBySource(openai) // Get relationships -const graph = await brainy.findSimilar(gpt4, { relationType: "develops" }) - -// 4๏ธโƒฃ Advanced: Search with context -const contextual = await brainy.search("Who leads AI companies?", 5, { - includeVerbs: true, // Include relationships in results - nounTypes: ["person"], // Filter to specific entity types -}) - -// 5๏ธโƒฃ NEW! MongoDB-style metadata filtering -const filtered = await brainy.search("AI research", 10, { +// This ONE query does what used to require 3 databases: +const results = await brainy.search("AI startups in healthcare", 10, { + // ๐Ÿ” Vector: Semantic similarity + includeVerbs: true, + + // ๐Ÿ”— Graph: Relationship traversal + verbTypes: ["invests_in", "partners_with"], + + // ๐Ÿ“Š Faceted: MongoDB-style filtering metadata: { - type: "academic", - year: { $gte: 2020 }, - status: { $in: ["published", "peer-reviewed"] }, - impact: { $gt: 100 } + industry: "healthcare", + funding: { $gte: 1000000 }, + stage: { $in: ["Series A", "Series B"] } } }) -// Filters DURING search for maximum performance! +// Returns: Companies similar to your query + their relationships + matching your criteria ``` -**๐ŸŽฏ That's it!** Vector search + graph database + works everywhere. No config needed. +**Three search paradigms. One lightning-fast query. Zero complexity.** -### ๐Ÿ” Want More Power? +## ๐Ÿš€ Install & Go -- **Advanced graph traversal** - Complex relationship queries and multi-hop searches -- **Distributed clustering** - Scale across multiple instances with automatic coordination -- **Real-time syncing** - WebSocket and WebRTC for live data updates -- **Custom augmentations** - Extend Brainy with your own functionality +```bash +npm install @soulcraft/brainy +``` -*[See full API documentation below](#-installation) for advanced features* +```javascript +import { BrainyData } from '@soulcraft/brainy' -## ๐ŸŽจ Build Amazing Things +const brainy = new BrainyData() // Auto-detects your environment +await brainy.init() // Auto-configures everything -**๐Ÿค– AI Chat Applications** - Build ChatGPT-like apps with long-term memory and context awareness -**๐Ÿ” Semantic Search Engines** - Search by meaning, not keywords. Find "that thing that's like a cat but bigger" โ†’ -returns "tiger" +// Add data with relationships +const openai = await brainy.add("OpenAI", { type: "company", funding: 11000000 }) +const gpt4 = await brainy.add("GPT-4", { type: "product", users: 100000000 }) +await brainy.relate(openai, gpt4, "develops") + +// Search across all dimensions +const results = await brainy.search("AI language models", 5, { + metadata: { funding: { $gte: 10000000 } }, + includeVerbs: true +}) +``` + +**That's it. You just built a knowledge graph with semantic search and faceted filtering in 8 lines.** + +## ๐Ÿ† Why Brainy Wins + +- ๐Ÿง  **Triple Search Power** - Vector + Graph + Faceted filtering in one query +- ๐ŸŒ **Runs Everywhere** - Same code: React, Node.js, serverless, edge +- โšก **Zero Config** - Auto-detects environment, optimizes itself +- ๐Ÿ”„ **Always Synced** - No data consistency nightmares between systems +- ๐Ÿ“ฆ **Truly Offline** - Works without internet after initial setup +- ๐Ÿ”’ **Your Data** - Run locally, in browser, or your own cloud + +## ๐Ÿ”ฎ Coming Soon + +- **๐Ÿค– MCP Integration** - Let Claude, GPT, and other AI models query your data directly +- **โšก LLM Generation** - Built-in content generation powered by your knowledge graph +- **๐ŸŒŠ Real-time Sync** - Live updates across distributed instances + +## ๐ŸŽฏ Perfect For + +**๐Ÿค– AI Chat Applications** - ChatGPT-like apps with long-term memory and context +**๐Ÿ” Semantic Search** - Find "that thing like a cat but bigger" โ†’ returns "tiger" +**๐Ÿงฌ Knowledge Graphs** - Connect everything. Wikipedia meets Neo4j meets magic **๐ŸŽฏ Recommendation Engines** - "Users who liked this also liked..." but actually good -**๐Ÿงฌ Knowledge Graphs** - Connect everything to everything. Wikipedia meets Neo4j meets magic -**๐Ÿ‘๏ธ Computer Vision Apps** - Store and search image embeddings. "Find all photos with dogs wearing hats" -**๐ŸŽต Music Discovery** - Find songs that "feel" similar. Spotify's Discover Weekly in your app -**๐Ÿ“š Smart Documentation** - Docs that answer questions. "How do I deploy to production?" โ†’ relevant guides -**๐Ÿ›ก๏ธ Fraud Detection** - Find patterns humans can't see. Anomaly detection on steroids -**๐ŸŒ Real-Time Collaboration** - Sync vector data across devices. Figma for AI data -**๐Ÿฅ Medical Diagnosis Tools** - Match symptoms to conditions using embedding similarity +**๐Ÿ“š Smart Documentation** - Docs that answer questions before you ask them -## ๐Ÿš€ Works Everywhere - Same Code - -**Write once, run anywhere.** Brainy auto-detects your environment and optimizes automatically: - -### ๐ŸŒ Browser Frameworks (React, Angular, Vue) - -```javascript -import { BrainyData } from '@soulcraft/brainy' - -// SAME CODE in React, Angular, Vue, Svelte, etc. -const brainy = new BrainyData() -await brainy.init() // Auto-uses OPFS in browsers - -// Add entities and relationships -const john = await brainy.add("John is a software engineer", { type: "person" }) -const jane = await brainy.add("Jane is a data scientist", { type: "person" }) -const ai = await brainy.add("AI Project", { type: "project" }) - -await brainy.relate(john, ai, "works_on") -await brainy.relate(jane, ai, "leads") - -// Search by meaning -const engineers = await brainy.search("software developers", 5) - -// Traverse relationships -const team = await brainy.getVerbsByTarget(ai) // Who works on AI Project? -``` - -
-๐Ÿ“ฆ Full Angular Component Example - -```typescript -import { Component, signal, OnInit } from '@angular/core' -import { BrainyData } from '@soulcraft/brainy' - -@Component({ - selector: 'app-search', - template: `` -}) -export class SearchComponent implements OnInit { - brainy = new BrainyData() - - async ngOnInit() { - await this.brainy.init() - // Add your data... - } - - async search(query: string) { - const results = await this.brainy.search(query, 5) - // Display results... - } -} -``` - -
- -
-๐Ÿ“ฆ Full React Example - -```jsx -import { BrainyData } from '@soulcraft/brainy' -import { useEffect, useState } from 'react' - -function Search() { - const [brainy, setBrainy] = useState(null) - const [results, setResults] = useState([]) - - useEffect(() => { - const init = async () => { - const db = new BrainyData() - await db.init() - // Add your data... - setBrainy(db) - } - init() - }, []) - - const search = async (query) => { - const results = await brainy?.search(query, 5) || [] - setResults(results) - } - - return search(e.target.value)} placeholder="Search..." /> -} -``` - -
- -
-๐Ÿ“ฆ Full Vue Example - -```vue - - - - -``` - -
- -### ๐ŸŸข Node.js / Serverless / Edge - -```javascript -import { BrainyData } from '@soulcraft/brainy' - -// SAME CODE works in Node.js, Vercel, Netlify, Cloudflare Workers, Deno, Bun -const brainy = new BrainyData() -await brainy.init() // Auto-detects environment and optimizes - -// Add entities and relationships -await brainy.add("Python is great for data science", { type: "fact" }) -await brainy.add("JavaScript rules the web", { type: "fact" }) - -// Search by meaning -const results = await brainy.search("programming languages", 5) - -// Optional: Production with S3/R2 storage (auto-detected in cloud environments) -const productionBrainy = new BrainyData({ - storage: { - s3Storage: { bucketName: process.env.BUCKET_NAME } - } -}) -``` - -**That's it! Same code, everywhere. Zero-to-Smartโ„ข** - -Brainy automatically detects and optimizes for: - -- ๐ŸŒ **Browser frameworks** โ†’ OPFS storage, Web Workers, memory optimization -- ๐ŸŸข **Node.js servers** โ†’ FileSystem or S3/R2 storage, Worker threads, cluster support -- โšก **Serverless functions** โ†’ S3/R2 or Memory storage, cold start optimization -- ๐Ÿ”ฅ **Edge workers** โ†’ Memory or KV storage, minimal footprint -- ๐Ÿงต **Web/Worker threads** โ†’ Shared storage, thread-safe operations -- ๐Ÿฆ• **Deno/Bun runtimes** โ†’ FileSystem or S3-compatible storage, native performance - -### ๐Ÿณ NEW: Zero-Config Docker Deployment - -**Deploy to any cloud with embedded models - no runtime downloads needed!** - -```dockerfile -# One line extracts models automatically during build -RUN npm run download-models - -# Deploy anywhere: Google Cloud, AWS, Azure, Cloudflare, etc. -``` - -- **โšก 7x Faster Cold Starts** - Models embedded in container, no downloads -- **๐ŸŒ Universal Cloud Support** - Same Dockerfile works everywhere -- **๐Ÿ”’ Offline Ready** - No external dependencies at runtime -- **๐Ÿ“ฆ Zero Configuration** - Automatic model detection and loading - -See [Docker Deployment Guide](./docs/docker-deployment.md) for complete examples. - -```javascript -// Zero configuration - everything optimized automatically! -const brainy = new BrainyData() // Auto-detects environment & optimizes -await brainy.init() - -// Caching happens automatically - no setup needed! -const results1 = await brainy.search('query', 10) // ~50ms first time -const results2 = await brainy.search('query', 10) // <1ms cached hit! - -// Advanced pagination works instantly -const page1 = await brainy.searchWithCursor('query', 100) -const page2 = await brainy.searchWithCursor('query', 100, { - cursor: page1.cursor // Constant time, no matter how deep! -}) - -// Monitor auto-optimized performance -const stats = brainy.getCacheStats() -console.log(`Auto-tuned cache hit rate: ${(stats.search.hitRate * 100).toFixed(1)}%`) -``` - -### ๐ŸŒ Distributed Mode Example (NEW!) - -```javascript -// Writer Instance - Ingests data from multiple sources -const writer = createAutoBrainy({ - storage: { s3Storage: { bucketName: 'my-bucket' } }, - distributed: { role: 'writer' } // Explicit role for safety -}) - -// Reader Instance - Optimized for search queries -const reader = createAutoBrainy({ - storage: { s3Storage: { bucketName: 'my-bucket' } }, - distributed: { role: 'reader' } // 80% memory for cache -}) - -// Data automatically gets domain tags -await writer.add("Patient shows symptoms of...", { - diagnosis: "flu" // Auto-tagged as 'medical' domain -}) - -// Domain-aware search across all partitions -const results = await reader.search("medical symptoms", 10, { - filter: { domain: 'medical' } // Only search medical data -}) - -// Monitor health across all instances -const health = reader.getHealthStatus() -console.log(`Instance ${health.instanceId}: ${health.status}`) -``` - -## ๐ŸŽญ Key Features - -### Core Capabilities - -- **Vector Search** - Find semantically similar content using embeddings -- **MongoDB-Style Metadata Filtering** ๐Ÿ†• - Advanced filtering with `$gt`, `$in`, `$regex`, `$and`, `$or` operators -- **Graph Relationships** - Connect data with meaningful relationships -- **JSON Document Search** - Search within specific fields with prioritization -- **Distributed Mode** - Scale horizontally with automatic coordination between instances -- **Real-Time Syncing** - WebSocket and WebRTC for distributed instances -- **Streaming Pipeline** - Process data in real-time as it flows through -- **Model Control Protocol** - Let AI models access your data - -### Developer Experience - -- **TypeScript Support** - Fully typed API with generics -- **Extensible Augmentations** - Customize and extend functionality -- **REST API** - Web service wrapper for HTTP endpoints -- **Auto-Complete** - IntelliSense for all APIs and types - -## ๐Ÿ“ฆ Installation - -### Development: Quick Start - -```bash -npm install @soulcraft/brainy -``` - -### โœจ Write-Once, Run-Anywhere Architecture - -**Same code, every environment.** Brainy auto-detects and optimizes for your runtime: - -```javascript -// This exact code works in Angular, React, Vue, Node.js, Deno, Bun, -// serverless functions, edge workers, and web workers -import { BrainyData } from '@soulcraft/brainy' - -const brainy = new BrainyData() -await brainy.init() // Auto-detects environment and chooses optimal storage - -// Vector + Graph: Add entities (nouns) with relationships (verbs) -const companyId = await brainy.addNoun("OpenAI creates powerful AI models", "company", { - founded: "2015", industry: "AI" -}) -const productId = await brainy.addNoun("GPT-4 is a large language model", "product", { - type: "LLM", parameters: "1.7T" -}) - -// Create relationships between entities -await brainy.addVerb(companyId, productId, undefined, { type: "develops" }) - -// Vector search finds semantically similar content -const similar = await brainy.search("AI language models", 5) - -// Graph operations: explore relationships -const relationships = await brainy.getVerbsBySource(companyId) -const allProducts = await brainy.getVerbsByType("develops") -``` - -### ๐Ÿ” Advanced Graph Operations - -```javascript -// Vector search with graph filtering -const results = await brainy.search("AI models", 10, { - searchVerbs: true, // Search relationships directly - verbTypes: ["develops"], // Filter by relationship types - searchConnectedNouns: true, // Find entities connected by relationships - verbDirection: "outgoing" // Direction: outgoing, incoming, or both -}) - -// Graph traversal methods -const outgoing = await brainy.getVerbsBySource(entityId) // What this entity relates to -const incoming = await brainy.getVerbsByTarget(entityId) // What relates to this entity -const byType = await brainy.getVerbsByType("develops") // All relationships of this type - -// Combined vector + graph search -const connected = await brainy.searchNounsByVerbs("machine learning", 5, { - verbTypes: ["develops", "uses"], - direction: "both" -}) - -// Get related entities through specific relationships -const related = await brainy.getRelatedNouns(companyId, { relationType: "develops" }) -``` - -**Universal benefits:** - -- โœ… **Auto-detects everything** - Environment, storage, threading, optimization -- โœ… **Framework-optimized** - Best experience with Angular, React, Vue bundlers -- โœ… **Runtime-agnostic** - Node.js, Deno, Bun, browsers, serverless, edge -- โœ… **TypeScript-first** - Full types everywhere, IntelliSense support -- โœ… **Tree-shaking ready** - Modern bundlers import only what you need -- โœ… **ES Modules architecture** - Individual modules for better optimization by modern frameworks - -### Production: Add Offline Model Reliability - -```bash -# For development (online model loading) -npm install @soulcraft/brainy - -# For production (offline reliability) -npm install @soulcraft/brainy @soulcraft/brainy-models -``` - -**Why use offline models in production?** - -- **๐Ÿ›ก๏ธ 100% Reliability** - No network timeouts or blocked URLs -- **โšก Instant Startup** - Models load in ~100ms vs 5-30 seconds -- **๐Ÿณ Docker Ready** - Perfect for Cloud Run, Lambda, Kubernetes -- **๐Ÿ”’ Zero Dependencies** - No external network calls required -- **๐ŸŽฏ Zero Configuration** - Automatic detection with graceful fallback -- **๐Ÿ” Enhanced Security** - Complete air-gapping support for sensitive environments -- **๐Ÿข Enterprise Ready** - Works behind corporate firewalls and restricted networks -- **โš–๏ธ Compliance & Forensics** - Frozen mode for audit trails and legal discovery - -The offline models provide the **same functionality** with maximum reliability. Your existing code works unchanged - -Brainy automatically detects and uses bundled models when available. - -```javascript -import { createAutoBrainy } from 'brainy' -import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models' - -// Use the bundled model for offline operation -const brainy = createAutoBrainy({ - embeddingModel: BundledUniversalSentenceEncoder -}) -``` - -## ๐Ÿณ Docker & Cloud Deployment - -**Deploy Brainy to any cloud provider with embedded models for maximum performance and reliability.** - -### Quick Docker Setup - -1. **Install models package:** - ```bash - npm install @soulcraft/brainy-models - ``` - -2. **Add to your Dockerfile:** - ```dockerfile - # Extract models during build (zero configuration!) - RUN npm run download-models - - # Include models in final image - COPY --from=builder /app/models ./models - ``` - -3. **Deploy anywhere:** - ```bash - # Works on all cloud providers - gcloud run deploy --source . # Google Cloud Run - aws ecs create-service ... # AWS ECS/Fargate - az container create ... # Azure Container Instances - wrangler publish # Cloudflare Workers - ``` - -### Universal Dockerfile Template - -```dockerfile -FROM node:24-slim AS builder -WORKDIR /app -COPY package*.json ./ -RUN npm ci -COPY . . -RUN npm run download-models # โ† Automatic model download -RUN npm run build - -FROM node:24-slim AS production -WORKDIR /app -COPY package*.json ./ -RUN npm ci --only=production --omit=optional -COPY --from=builder /app/dist ./dist -COPY --from=builder /app/models ./models # โ† Models included -CMD ["node", "dist/server.js"] -``` - -### Benefits - -- **โšก 7x Faster Cold Starts** - No model download delays -- **๐ŸŒ Universal Compatibility** - Same Dockerfile works on all clouds -- **๐Ÿ”’ Offline Ready** - No external dependencies at runtime -- **๐Ÿ“ฆ Zero Configuration** - Automatic model detection -- **๐Ÿ›ก๏ธ Enhanced Security** - No network calls for model loading - -**๐Ÿ“– Complete Guide:** See [docs/docker-deployment.md](./docs/docker-deployment.md) for detailed examples covering Google -Cloud Run, AWS Lambda/ECS, Azure Container Instances, Cloudflare Workers, and more. - -### ๐Ÿ“ฆ Modern ES Modules Architecture - -Brainy now uses individual ES modules instead of large bundles, providing better optimization for modern frameworks: - -- **Better tree-shaking**: Frameworks import only the specific functions you use -- **Smaller final apps**: Your bundled application only includes what you actually need -- **Faster development builds**: No complex bundling during development -- **Better debugging**: Source maps point to individual files, not large bundles - -This change reduced the package size significantly while improving compatibility with Angular, React, Vue, and other -modern framework build systems. - -## ๐Ÿงฌ The Power of Nouns & Verbs - -Brainy uses a **graph-based data model** that mirrors how humans think - with **Nouns** (entities) connected by **Verbs -** (relationships). This isn't just vectors in a void; it's structured, meaningful data. - -### ๐Ÿ“ Nouns (What Things Are) - -Nouns are your entities - the "things" in your data. Each noun has: - -- A unique ID -- A vector representation (for similarity search) -- A type (Person, Document, Concept, etc.) -- Custom metadata - -**Available Noun Types:** - -| Category | Types | Use For | -|---------------------|-------------------------------------------------------------------|-------------------------------------------------------| -| **Core Entities** | `Person`, `Organization`, `Location`, `Thing`, `Concept`, `Event` | People, companies, places, objects, ideas, happenings | -| **Digital Content** | `Document`, `Media`, `File`, `Message`, `Content` | PDFs, images, videos, emails, posts, generic content | -| **Collections** | `Collection`, `Dataset` | Groups of items, structured data sets | -| **Business** | `Product`, `Service`, `User`, `Task`, `Project` | E-commerce, SaaS, project management | -| **Descriptive** | `Process`, `State`, `Role` | Workflows, conditions, responsibilities | - -### ๐Ÿ”— Verbs (How Things Connect) - -Verbs are your relationships - they give meaning to connections. Not just "these vectors are similar" but "this OWNS -that" or "this CAUSES that". - -**Available Verb Types:** - -| Category | Types | Examples | -|----------------|----------------------------------------------------------------------|------------------------------------------| -| **Core** | `RelatedTo`, `Contains`, `PartOf`, `LocatedAt`, `References` | Generic relations, containment, location | -| **Temporal** | `Precedes`, `Succeeds`, `Causes`, `DependsOn`, `Requires` | Time sequences, causality, dependencies | -| **Creation** | `Creates`, `Transforms`, `Becomes`, `Modifies`, `Consumes` | Creation, change, consumption | -| **Ownership** | `Owns`, `AttributedTo`, `CreatedBy`, `BelongsTo` | Ownership, authorship, belonging | -| **Social** | `MemberOf`, `WorksWith`, `FriendOf`, `Follows`, `Likes`, `ReportsTo` | Social networks, organizations | -| **Functional** | `Describes`, `Implements`, `Validates`, `Triggers`, `Serves` | Functions, implementations, services | - -### ๐Ÿ’ก Why This Matters - -```javascript -// Traditional vector DB: Just similarity -const similar = await vectorDB.search(embedding, 10) -// Result: [vector1, vector2, ...] - What do these mean? ๐Ÿคท - -// Brainy: Similarity + Meaning + Relationships -const catId = await brainy.add("Siamese cat", { - noun: NounType.Thing, - breed: "Siamese" -}) -const ownerId = await brainy.add("John Smith", { - noun: NounType.Person -}) -await brainy.addVerb(ownerId, catId, { - verb: VerbType.Owns, - since: "2020-01-01" -}) - -// Now you can search with context! -const johnsPets = await brainy.getVerbsBySource(ownerId, VerbType.Owns) -const catOwners = await brainy.getVerbsByTarget(catId, VerbType.Owns) -``` - -## ๐ŸŒ Distributed Mode (New!) - -Brainy now supports **distributed deployments** with multiple specialized instances sharing the same data. Perfect for -scaling your AI applications across multiple servers. - -### Distributed Setup - -```javascript -// Single instance (no change needed!) -const brainy = createAutoBrainy({ - storage: { s3Storage: { bucketName: 'my-bucket' } } -}) - -// Distributed mode requires explicit role configuration -// Option 1: Via environment variable -process.env.BRAINY_ROLE = 'writer' // or 'reader' or 'hybrid' -const brainy = createAutoBrainy({ - storage: { s3Storage: { bucketName: 'my-bucket' } }, - distributed: true -}) - -// Option 2: Via configuration -const writer = createAutoBrainy({ - storage: { s3Storage: { bucketName: 'my-bucket' } }, - distributed: { role: 'writer' } // Handles data ingestion -}) - -const reader = createAutoBrainy({ - storage: { s3Storage: { bucketName: 'my-bucket' } }, - distributed: { role: 'reader' } // Optimized for queries -}) - -// Option 3: Via read/write mode (role auto-inferred) -const writer = createAutoBrainy({ - storage: { s3Storage: { bucketName: 'my-bucket' } }, - writeOnly: true, // Automatically becomes 'writer' role - distributed: true -}) - -const reader = createAutoBrainy({ - storage: { s3Storage: { bucketName: 'my-bucket' } }, - readOnly: true, // Automatically becomes 'reader' role (allows optimizations) - // frozen: true, // Optional: Complete immutability for compliance/forensics - distributed: true -}) -``` - -### Key Distributed Features - -**๐ŸŽฏ Explicit Role Configuration** - -- Roles must be explicitly set (no dangerous auto-assignment) -- Can use environment variables, config, or read/write modes -- Clear separation between writers and readers - -**#๏ธโƒฃ Hash-Based Partitioning** - -- Handles multiple writers with different data types -- Even distribution across partitions -- No semantic conflicts with mixed data - -**๐Ÿท๏ธ Domain Tagging** - -- Automatic domain detection (medical, legal, product, etc.) -- Filter searches by domain -- Logical separation without complexity - -```javascript -// Data is automatically tagged with domains -await brainy.add({ - symptoms: "fever", - diagnosis: "flu" -}, metadata) // Auto-tagged as 'medical' - -// Search within specific domains -const medicalResults = await brainy.search(query, 10, { - filter: { domain: 'medical' } -}) -``` - -**๐Ÿ“Š Health Monitoring** - -- Real-time health metrics -- Automatic dead instance cleanup -- Performance tracking - -```javascript -// Get health status -const health = brainy.getHealthStatus() -// { -// status: 'healthy', -// role: 'reader', -// vectorCount: 1000000, -// cacheHitRate: 0.95, -// requestsPerSecond: 150 -// } -``` - -**โšก Role-Optimized Performance** - -- **Readers**: 80% memory for cache, aggressive prefetching -- **Writers**: Optimized write batching, minimal cache -- **Hybrid**: Adaptive based on workload - -## โš–๏ธ Compliance & Forensics Mode - -For legal discovery, audit trails, and compliance requirements: - -```javascript -// Create a completely immutable snapshot -const auditDb = new BrainyData({ - storage: { s3Storage: { bucketName: 'audit-snapshots' } }, - readOnly: true, - frozen: true // Complete immutability - no changes allowed -}) - -// Perfect for: -// - Legal discovery (data cannot be modified) -// - Compliance audits (guaranteed state) -// - Forensic analysis (preserved evidence) -// - Regulatory snapshots (unchanging records) -``` - -### Deployment Examples - -**Docker Compose** - -```yaml -services: - writer: - image: myapp - environment: - BRAINY_ROLE: writer # Optional - auto-detects - - reader: - image: myapp - environment: - BRAINY_ROLE: reader # Optional - auto-detects - scale: 5 -``` - -**Kubernetes** - -```yaml -# Automatically detects role from deployment type -apiVersion: apps/v1 -kind: Deployment -metadata: - name: brainy-readers -spec: - replicas: 10 # Multiple readers - template: - spec: - containers: - - name: app - image: myapp - # Role auto-detected as 'reader' (multiple replicas) -``` - -**Benefits** - -- โœ… **50-70% faster searches** with parallel readers -- โœ… **No coordination complexity** - Shared JSON config in S3 -- โœ… **Zero downtime scaling** - Add/remove instances anytime -- โœ… **Automatic failover** - Dead instances cleaned up automatically - -## ๐Ÿค” Why Choose Brainy? - -### vs. Traditional Databases - -โŒ **PostgreSQL with pgvector** - Requires complex setup, tuning, and DevOps expertise -โœ… **Brainy** - Zero config, auto-optimizes, works everywhere from browser to cloud - -### vs. Vector Databases - -โŒ **Pinecone/Weaviate/Qdrant** - Cloud-only, expensive, vendor lock-in -โœ… **Brainy** - Run locally, in browser, or cloud. Your choice, your data - -### vs. Graph Databases - -โŒ **Neo4j** - Great for graphs, no vector support -โœ… **Brainy** - Vectors + graphs in one. Best of both worlds - -### vs. "Vector + Graph" Solutions - -โŒ **Pinecone + Neo4j** - Two databases, sync nightmares, double the cost -โŒ **pgvector + graph extension** - Hacked together, not native, performance issues -โŒ **Weaviate "references"** - Limited graph capabilities, not true relationships -โœ… **Brainy** - Purpose-built vector+graph architecture, single source of truth - -### vs. DIY Solutions - -โŒ **Building your own** - Months of work, optimization nightmares -โœ… **Brainy** - Production-ready in 30 seconds - -## ๐Ÿš€ Getting Started in 30 Seconds - -**The same Brainy code works everywhere - React, Vue, Angular, Node.js, Serverless, Edge Workers.** +## ๐ŸŒ Works Everywhere - Same Code ```javascript // This EXACT code works in ALL environments @@ -936,265 +101,165 @@ import { BrainyData } from '@soulcraft/brainy' const brainy = new BrainyData() await brainy.init() -// Add nouns (entities) -const openai = await brainy.add("OpenAI", { type: "company" }) -const gpt4 = await brainy.add("GPT-4", { type: "product" }) - -// Add verbs (relationships) -await brainy.relate(openai, gpt4, "develops") - -// Vector search + Graph traversal -const similar = await brainy.search("AI companies", 5) -const products = await brainy.getVerbsBySource(openai) +// Works in: React, Vue, Angular, Node.js, Deno, Bun, +// Cloudflare Workers, Vercel Edge, AWS Lambda, browsers, anywhere ``` +Brainy automatically detects and optimizes for your environment: + +| Environment | Storage | Optimization | +|-------------|---------|-------------| +| ๐ŸŒ Browser | OPFS | Web Workers, Memory Cache | +| ๐ŸŸข Node.js | FileSystem / S3 | Worker Threads, Clustering | +| โšก Serverless | S3 / Memory | Cold Start Optimization | +| ๐Ÿ”ฅ Edge | Memory / KV | Minimal Footprint | + +## ๐Ÿ†š Why Not Just Use...? + +### vs. Multiple Databases +โŒ **Pinecone + Neo4j + Elasticsearch** - 3 databases, sync nightmares, 3x the cost +โœ… **Brainy** - One database, always synced, built-in intelligence + +### vs. Traditional Solutions +โŒ **PostgreSQL + pgvector + extensions** - Complex setup, performance issues +โœ… **Brainy** - Zero config, purpose-built for AI, works everywhere + +### vs. Cloud-Only Vector DBs +โŒ **Pinecone/Weaviate/Qdrant** - Vendor lock-in, expensive, cloud-only +โœ… **Brainy** - Run anywhere, your data stays yours, cost-effective + +### vs. Graph Databases with "Vector Features" +โŒ **Neo4j + vector plugin** - Bolt-on solution, not native, limited +โœ… **Brainy** - Native vector+graph architecture from the ground up + +## ๐Ÿ“ฆ Advanced Features +
-๐Ÿ” See Framework Examples - -### React - -```jsx -function App() { - const [brainy] = useState(() => new BrainyData()) - useEffect(() => brainy.init(), []) - - const search = async (query) => { - return await brainy.search(query, 10) - } - // Same API as above -} -``` - -### Vue 3 - -```vue - - -``` - -### Angular - -```typescript - -@Component({}) -export class AppComponent { - brainy = new BrainyData() - - async ngOnInit() { - await this.brainy.init() - // Same API as above - } -} -``` - -### Node.js / Deno / Bun +๐Ÿ”ง MongoDB-Style Metadata Filtering ```javascript -const brainy = new BrainyData() -await brainy.init() -// Same API as above +const results = await brainy.search("machine learning", 10, { + metadata: { + // Comparison operators + price: { $gte: 100, $lte: 1000 }, + category: { $in: ["AI", "ML", "Data"] }, + rating: { $gt: 4.5 }, + + // Logical operators + $and: [ + { status: "active" }, + { verified: true } + ], + + // Text operators + description: { $regex: "neural.*network", $options: "i" }, + + // Array operators + tags: { $includes: "tensorflow" } + } +}) +``` + +**15+ operators supported**: `$gt`, `$gte`, `$lt`, `$lte`, `$eq`, `$ne`, `$in`, `$nin`, `$and`, `$or`, `$not`, `$regex`, `$includes`, `$exists`, `$size` + +
+ +
+๐Ÿ”— Graph Relationships & Traversal + +```javascript +// Create entities and relationships +const company = await brainy.add("OpenAI", { type: "company" }) +const product = await brainy.add("GPT-4", { type: "product" }) +const person = await brainy.add("Sam Altman", { type: "person" }) + +// Create meaningful relationships +await brainy.relate(company, product, "develops") +await brainy.relate(person, company, "leads") +await brainy.relate(product, person, "created_by") + +// Traverse relationships +const products = await brainy.getVerbsBySource(company) // What OpenAI develops +const leaders = await brainy.getVerbsByTarget(company) // Who leads OpenAI +const connections = await brainy.findSimilar(product, { + relationType: "develops" +}) + +// Search with relationship context +const results = await brainy.search("AI models", 10, { + includeVerbs: true, + verbTypes: ["develops", "created_by"], + searchConnectedNouns: true +}) ```
-### ๐ŸŒ Framework-First, Runs Everywhere - -**Brainy automatically detects your environment and optimizes everything:** - -| Environment | Storage | Optimization | -|-----------------|-----------------|----------------------------| -| ๐ŸŒ Browser | OPFS | Web Workers, Memory Cache | -| ๐ŸŸข Node.js | FileSystem / S3 | Worker Threads, Clustering | -| โšก Serverless | S3 / Memory | Cold Start Optimization | -| ๐Ÿ”ฅ Edge Workers | Memory / KV | Minimal Footprint | -| ๐Ÿฆ• Deno/Bun | FileSystem / S3 | Native Performance | - -## ๐ŸŒ Deploy to Any Cloud -
-โ˜๏ธ See Cloud Platform Examples - -### Cloudflare Workers +๐ŸŒ Universal Storage & Deployment ```javascript -import { BrainyData } from '@soulcraft/brainy' +// Development: File system +const dev = new BrainyData({ + storage: { fileSystem: { path: './data' } } +}) -export default { - async fetch(request) { - const brainy = new BrainyData() - await brainy.init() +// Production: S3/R2 +const prod = new BrainyData({ + storage: { s3Storage: { bucketName: 'my-vectors' } } +}) - const url = new URL(request.url) - const results = await brainy.search(url.searchParams.get('q'), 10) - return Response.json(results) - } -} +// Browser: OPFS +const browser = new BrainyData() // Auto-detects OPFS + +// Edge: Memory +const edge = new BrainyData({ + storage: { memory: {} } +}) + +// Redis: High performance +const redis = new BrainyData({ + storage: { redis: { connectionString: 'redis://...' } } +}) ``` -### AWS Lambda - -```javascript -import { BrainyData } from '@soulcraft/brainy' - -export const handler = async (event) => { - const brainy = new BrainyData() - await brainy.init() - - const results = await brainy.search(event.query, 10) - return { statusCode: 200, body: JSON.stringify(results) } -} -``` - -### Google Cloud Functions - -```javascript -import { BrainyData } from '@soulcraft/brainy' - -export const searchHandler = async (req, res) => { - const brainy = new BrainyData() - await brainy.init() - - const results = await brainy.search(req.query.q, 10) - res.json(results) -} -``` - -### Vercel Edge Functions - -```javascript -import { BrainyData } from '@soulcraft/brainy' - -export const config = { runtime: 'edge' } - -export default async function handler(request) { - const brainy = new BrainyData() - await brainy.init() - - const { searchParams } = new URL(request.url) - const results = await brainy.search(searchParams.get('q'), 10) - return Response.json(results) -} -``` +**Extend with any storage**: MongoDB, PostgreSQL, DynamoDB - [see storage adapters guide](docs/api-reference/storage-adapters.md)
-### Docker Container +
+๐Ÿณ Docker & Cloud Deployment ```dockerfile -FROM node:24-slim -USER node +# Production-ready Dockerfile +FROM node:24-slim AS builder WORKDIR /app COPY package*.json ./ -RUN npm install brainy +RUN npm ci COPY . . +RUN npm run download-models # Embed models for offline operation +RUN npm run build -CMD ["node", "server.js"] +FROM node:24-slim AS production +WORKDIR /app +COPY package*.json ./ +RUN npm ci --only=production +COPY --from=builder /app/dist ./dist +COPY --from=builder /app/models ./models # Offline models included +CMD ["node", "dist/server.js"] ``` -```javascript -// server.js -import { createAutoBrainy } from 'brainy' -import express from 'express' +Deploy to: Google Cloud Run, AWS Lambda/ECS, Azure Container Instances, Cloudflare Workers, Railway, Render, Vercel, anywhere Docker runs. -const app = express() -const brainy = createAutoBrainy() +
-app.get('/search', async (req, res) => { - const results = await brainy.searchText(req.query.q, 10) - res.json(results) -}) +## ๐Ÿ“š Documentation & Resources -app.listen(3000, () => console.log('Brainy running on port 3000')) -``` - -### Kubernetes - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: brainy-api -spec: - replicas: 3 - template: - spec: - containers: - - name: brainy - image: your-registry/brainy-api:latest - env: - - name: S3_BUCKET - value: "your-vector-bucket" -``` - -### Railway.app - -```javascript -// server.js -import { createAutoBrainy } from 'brainy' - -const brainy = createAutoBrainy({ - bucketName: process.env.RAILWAY_VOLUME_NAME -}) - -// Railway automatically handles the rest! -``` - -### Render.com - -```yaml -# render.yaml -services: - - type: web - name: brainy-api - env: node - buildCommand: npm install brainy - startCommand: node server.js - envVars: - - key: BRAINY_STORAGE - value: persistent-disk -``` - -## Getting Started - -- [**Quick Start Guide**](docs/getting-started/) - Get up and running in minutes -- [**Installation**](docs/getting-started/installation.md) - Detailed setup instructions -- [**Environment Setup**](docs/getting-started/environment-setup.md) - Platform-specific configuration - -### User Guides - -- [**Search and Metadata**](docs/user-guides/) - Advanced search techniques -- [**JSON Document Search**](docs/guides/json-document-search.md) - Field-based searching -- [**Read-Only & Frozen Modes**](docs/guides/readonly-frozen-modes.md) - Immutability options for production -- [**Production Migration**](docs/guides/production-migration-guide.md) - Deployment best practices - -### API Reference - -- [**Core API**](docs/api-reference/) - Complete method reference -- [**Configuration Options**](docs/api-reference/configuration.md) - All configuration parameters - -### Optimization & Scaling - -- [**Performance Features Guide**](docs/PERFORMANCE_FEATURES.md) - Advanced caching, auto-configuration, and - optimization -- [**Large-Scale Optimizations**](docs/optimization-guides/) - Handle millions of vectors -- [**Memory Management**](docs/optimization-guides/memory-optimization.md) - Efficient resource usage -- [**S3 Migration Guide**](docs/optimization-guides/s3-migration-guide.md) - Cloud storage setup - -### Examples & Patterns - -- [**Code Examples**](docs/examples/) - Real-world usage patterns -- [**Integrations**](docs/examples/integrations.md) - Third-party services -- [**Performance Patterns**](docs/examples/performance.md) - Optimization techniques - -### Technical Documentation - -- [**Architecture Overview**](docs/technical/) - System design and internals -- [**Testing Guide**](docs/technical/TESTING.md) - Testing strategies -- [**Statistics & Monitoring**](docs/technical/STATISTICS.md) - Performance tracking +- **[๐Ÿš€ Quick Start Guide](docs/getting-started/)** - Get up and running in minutes +- **[๐Ÿ“– API Reference](docs/api-reference/)** - Complete method documentation +- **[๐Ÿ’ก Examples](docs/examples/)** - Real-world usage patterns +- **[โšก Performance Guide](docs/optimization-guides/)** - Scale to millions of vectors +- **[๐Ÿ”ง Storage Adapters](docs/api-reference/storage-adapters.md)** - Universal storage compatibility ## ๐Ÿค Contributing @@ -1211,5 +276,7 @@ We welcome contributions! Please see: ---
-Ready to build something amazing? Get started with Brainy today! -
+Ready to build the future of search? Get started with Brainy today! + +**[Get Started โ†’](docs/getting-started/) | [View Examples โ†’](docs/examples/) | [Join Community โ†’](https://github.com/soulcraft-research/brainy/discussions)** + \ No newline at end of file