diff --git a/README.md b/README.md index f69ecd9f..7e9f12c2 100644 --- a/README.md +++ b/README.md @@ -9,714 +9,619 @@ [![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) [![TypeScript](https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg)](https://www.typescriptlang.org/) -**๐Ÿง  Brainy - The Knowledge Operating System** +## The Knowledge Operating System -**The world's first Knowledge Operating System** where every piece of knowledge - files, concepts, entities, ideas - exists as living information that understands itself, evolves over time, and connects to everything related. +**Every piece of knowledge in your application โ€” living, connected, and intelligent.** -**Why Brainy Changes Everything**: Traditional systems trap knowledge in files or database rows. Brainy liberates it. Your characters exist across stories. Your concepts span projects. Your APIs remember their evolution. Every piece of knowledge - whether it's code, prose, or pure ideas - lives, breathes, and connects in a unified intelligence layer where everything understands its meaning, remembers its history, and relates to everything else. - -Built on revolutionary **Triple Intelligenceโ„ข** that unifies vector similarity, graph relationships, and document filtering in one magical API. **Framework-first design.** Zero configuration. O(log n) performance, <10ms search latency. **Production-ready for billion-scale deployments.** - ---- - -## ๐ŸŽ‰ NEW in v4.0.0 - Enterprise-Scale Cost Optimization - -**Major Release: Production cost optimization and enterprise-scale features** - -### ๐Ÿ’ฐ **Up to 96% Storage Cost Savings** - -Automatic cloud storage lifecycle management for **AWS S3**, **Google Cloud Storage**, and **Azure Blob Storage**: - -- **GCS Autoclass**: Fully automatic tier optimization (94% savings!) -- **AWS Intelligent-Tiering**: Smart archival with instant retrieval -- **Azure Lifecycle Policies**: Automatic tier transitions -- **Cost Impact**: $138,000/year โ†’ $5,940/year @ 500TB scale - -### โšก **Performance at Billion-Scale** - -- **1000x faster batch deletions** (533 entities/sec vs 0.5/sec) -- **60-80% FileSystem compression** with gzip -- **OPFS quota monitoring** for browser storage -- **Enhanced CLI** with 47 commands including 9 storage management tools - -### ๐Ÿ›ก๏ธ **Zero Breaking Changes** - -**100% backward compatible.** No migration required. All new features are opt-in. - -**[๐Ÿ“– Read the full v4.0.0 Changelog โ†’](CHANGELOG.md)** | **[Migration Guide โ†’](docs/MIGRATION-V3-TO-V4.md)** - ---- - -## ๐ŸŽฏ What Makes Brainy Revolutionary? - -### ๐Ÿง  **Triple Intelligenceโ„ข - The Impossible Made Possible** - -**The world's first to unify three database paradigms in ONE API:** - -- **Vector Search** ๐Ÿ” Semantic similarity like Pinecone/Weaviate -- **Graph Relationships** ๐Ÿ•ธ๏ธ Navigate connections like Neo4j/ArangoDB -- **Document Filtering** ๐Ÿ“Š MongoDB-style queries with O(log n) performance - -**Others make you choose.** Vector OR graph OR document. **Brainy does ALL THREE together.** This is what enables The Knowledge Operating System. - -### ๐Ÿš€ **Zero Configuration - Just Worksโ„ข** +Stop fighting with vector databases, graph databases, and document stores. Stop stitching together Pinecone + Neo4j + MongoDB. **Brainy does all three, in one elegant API, from prototype to planet-scale.** ```javascript -import { Brainy } from '@soulcraft/brainy' +const brain = new Brainy() +await brain.init() + +// That's it. You now have semantic search, graph relationships, +// and document filtering. Zero configuration. Just works. +``` + +**Built by developers who were tired of:** +- Spending weeks configuring embeddings, indexes, and schemas +- Choosing between vector similarity OR graph relationships OR metadata filtering +- Rewriting everything when you need to scale from 1,000 to 1,000,000,000 entities + +**Brainy makes the impossible simple: All three paradigms. One API. Any scale.** + +--- + +## See It In Action + +**30 seconds to understand why Brainy is different:** + +```javascript +import { Brainy, NounType, VerbType } from '@soulcraft/brainy' const brain = new Brainy() await brain.init() -// That's it! Auto-detects storage, optimizes memory, configures everything. -``` - -No configuration files. No environment variables. No complex setup. **It just works.** - -### โšก **Production Performance at Any Scale** - -- **<10ms search** across millions of entities -- **87% memory reduction** @ billion scale (384GB โ†’ 50GB) -- **10x faster queries** with type-aware indexing -- **99% storage cost savings** with intelligent archival -- **Container-aware** memory allocation (Docker/K8s) - -### ๐ŸŽฏ **31 Noun Types ร— 40 Verb Types = Infinite Expressiveness** - -Model **ANY domain** with 1,240 base type combinations + unlimited metadata: - -- Healthcare: Patient โ†’ diagnoses โ†’ Condition -- Finance: Account โ†’ transfers โ†’ Transaction -- Manufacturing: Product โ†’ assembles โ†’ Component -- Education: Student โ†’ completes โ†’ Course -- **Your domain**: Your types + Your relationships = Your knowledge graph - -[โ†’ See the Mathematical Proof](docs/architecture/noun-verb-taxonomy.md) - ---- - -## ๐Ÿ’ก What Can You Build? - -**Brainy unlocks entirely new categories of applications.** Here's what developers are building: - -### ๐Ÿ“š **Intelligent Documentation Systems** -- **Living knowledge bases** where docs understand each other and evolve -- **Smart wikis** that auto-link related concepts and detect outdated information -- **Research assistants** that remember every paper you've read and find connections - -### ๐Ÿค– **AI Agents with Perfect Memory** -- **Conversational AI** that remembers context across months, not just messages -- **Code assistants** that understand your entire codebase as a knowledge graph -- **Personal assistants** with unlimited memory that never forgets important details - -### ๐ŸŽฎ **Rich Interactive Experiences** -- **Game worlds** where NPCs remember every interaction and relationships evolve -- **Story engines** where characters persist across multiple narratives -- **Educational platforms** that build personalized knowledge graphs as you learn - -### ๐Ÿ” **Next-Gen Search & Discovery** -- **Semantic code search** across millions of repositories -- **Smart file explorers** that understand code relationships, not just folders -- **Research platforms** that find papers by meaning, not keywords - -### ๐Ÿข **Enterprise Knowledge Management** -- **Corporate memory systems** where institutional knowledge never gets lost -- **Customer intelligence** platforms that understand every interaction -- **Product catalogs** with semantic search and relationship-based recommendations - -### ๐ŸŽจ **Creative Tools & Content Platforms** -- **Writing assistants** that track characters, plotlines, and themes across stories -- **Content management** where every asset knows its relationships and history -- **Media libraries** with intelligent tagging and similarity-based discovery - -**The Pattern**: If your app needs to **remember**, **understand**, or **connect** information, Brainy makes it trivial. - ---- - -## โšก Quick Start - Zero Configuration - -```bash -npm install @soulcraft/brainy -``` - -### ๐ŸŽฏ **Your First Knowledge Graph in 30 Seconds** - -```javascript -import { Brainy, NounType } from '@soulcraft/brainy' - -// Just this - auto-detects everything! -const brain = new Brainy() -await brain.init() - -// Add entities with automatic embedding -const jsId = await brain.add({ - data: "JavaScript is a programming language", - nounType: NounType.Concept, - metadata: { - type: "language", - year: 1995, - paradigm: "multi-paradigm" - } +// Add knowledge with context +const reactId = await brain.add({ + data: "React is a JavaScript library for building user interfaces", + type: NounType.Concept, + metadata: { category: "frontend", year: 2013 } }) -const nodeId = await brain.add({ - data: "Node.js runtime environment", - nounType: NounType.Concept, - metadata: { - type: "runtime", - year: 2009, - platform: "server-side" - } +const nextId = await brain.add({ + data: "Next.js framework for React with server-side rendering", + type: NounType.Concept, + metadata: { category: "framework", year: 2016 } }) -// Create relationships between entities -await brain.relate({ - from: nodeId, - to: jsId, - type: "executes", - metadata: { - since: 2009, - performance: "high" - } -}) +// Create relationships +await brain.relate({ from: nextId, to: reactId, type: VerbType.BuiltOn }) -// Natural language search with graph relationships +// NOW THE MAGIC: Query with natural language const results = await brain.find({ - query: "programming languages used by server runtimes" + query: "modern frontend frameworks", // ๐Ÿ” Vector similarity + where: { year: { greaterThan: 2015 } }, // ๐Ÿ“Š Document filtering + connected: { to: reactId, depth: 2 } // ๐Ÿ•ธ๏ธ Graph traversal }) -// Triple Intelligence: vector + metadata + relationships -const filtered = await brain.find({ - query: "JavaScript", // Vector similarity - where: {type: "language"}, // Metadata filtering - connected: {from: nodeId, depth: 1} // Graph relationships -}) +// ALL THREE PARADIGMS. ONE QUERY. 10ms response time. ``` -**That's it!** You just created a knowledge graph with semantic search, relationship traversal, and metadata filtering. **No configuration. No complexity. Just works.** +**This is impossible with traditional databases.** Brainy makes it trivial. --- -## ๐ŸŒŸ Core Features +## From Prototype to Planet Scale -### ๐Ÿง  **Natural Language Understanding** +**The same API. Zero rewrites. Any scale.** + +### ๐Ÿ‘ค **Individual Developer** โ†’ Weekend Prototype ```javascript -// Ask questions naturally - Brainy understands -await brain.find("Show me recent React components with tests") -await brain.find("Popular JavaScript libraries similar to Vue") -await brain.find("Documentation about authentication from last month") +// Zero configuration - starts in memory +const brain = new Brainy() +await brain.init() -// Structured queries with Triple Intelligence -await brain.find({ - like: "React", // Vector similarity - where: { // Document filtering - type: "library", - year: {greaterThan: 2020} - }, - connected: {to: "JavaScript", depth: 2} // Graph relationships -}) +// Build your prototype in minutes +// Change nothing when ready to scale ``` -### ๐ŸŒ **Virtual Filesystem - Intelligent File Management** +**Perfect for:** Hackathons, side projects, rapid prototyping, learning AI concepts -Build file explorers, IDEs, and knowledge systems that **never crash**: +### ๐Ÿ‘ฅ **Small Team** โ†’ Production MVP (Thousands of Entities) ```javascript -const vfs = brain.vfs() -await vfs.init() - -// โœ… Safe file operations -await vfs.writeFile('/projects/app/index.js', 'console.log("Hello")') -await vfs.mkdir('/docs') - -// โœ… NEVER crashes: Tree-aware directory listing -const children = await vfs.getDirectChildren('/projects') - -// โœ… Build file explorers safely -const tree = await vfs.getTreeStructure('/projects', { - maxDepth: 3, // Prevent deep recursion - sort: 'name' -}) - -// โœ… Semantic file search -const reactFiles = await vfs.search('React components with hooks') -``` - -**Prevents infinite recursion** that crashes traditional file systems. Tree-aware operations ensure your file explorer never hangs. - -**[๐Ÿ“– VFS Quick Start โ†’](docs/vfs/QUICK_START.md)** | **[๐ŸŽฏ Common Patterns โ†’](docs/vfs/COMMON_PATTERNS.md)** - -### ๐Ÿš€ **Import Anything - CSV, Excel, PDF, URLs** - -```javascript -// Import CSV with auto-detection -await brain.import('customers.csv') -// โœจ Auto-detects: encoding, delimiter, types, creates entities! - -// Import Excel workbooks with multi-sheet support -await brain.import('sales-data.xlsx', { - excelSheets: ['Q1', 'Q2'] -}) - -// Import PDF documents with table extraction -await brain.import('research-paper.pdf', { - pdfExtractTables: true -}) - -// Import from URLs (auto-fetched) -await brain.import('https://api.example.com/data.json') -``` - -**[๐Ÿ“– Complete Import Guide โ†’](docs/guides/import-anything.md)** - -### ๐Ÿง  **Neural API - Advanced Semantic Analysis** - -```javascript -const neural = brain.neural - -// Automatic semantic clustering -const clusters = await neural.clusters({ - algorithm: 'kmeans', - maxClusters: 5, - threshold: 0.8 -}) - -// Calculate similarity between any items -const similarity = await neural.similar('item1', 'item2') - -// Find nearest neighbors -const neighbors = await neural.neighbors('item-id', 10) - -// Detect outliers -const outliers = await neural.outliers(0.3) - -// Generate visualization data for D3/Cytoscape -const vizData = await neural.visualize({ - maxNodes: 100, - dimensions: 3, - algorithm: 'force' -}) -``` - ---- - -## ๐ŸŒ Framework Integration - -**Brainy is framework-first!** Works with **any** modern framework: - -### โš›๏ธ React & Next.js -```javascript -function SearchComponent() { - const [brain] = useState(() => new Brainy()) - useEffect(() => { brain.init() }, []) - - const handleSearch = async (query) => { - const results = await brain.find(query) - setResults(results) - } -} -``` - -### ๐ŸŸข Vue.js & Nuxt.js -```javascript -export default { - async mounted() { - this.brain = new Brainy() - await this.brain.init() - }, - methods: { - async search(query) { - return await this.brain.find(query) - } - } -} -``` - -### ๐Ÿ…ฐ๏ธ Angular -```typescript -@Injectable({ providedIn: 'root' }) -export class BrainyService { - private brain = new Brainy() - async search(query: string) { - return await this.brain.find(query) - } -} -``` - -**Works with:** Svelte, Solid.js, Qwik, Fresh, and more! All bundlers (Webpack, Vite, Rollup). SSR/SSG. Edge runtimes. - ---- - -## ๐Ÿ’พ Storage - From Development to Production - -### ๐Ÿš€ **Development: Just Works** -```javascript -const brain = new Brainy() // Memory storage, auto-configured -``` - -### โšก **Production: FileSystem with Compression** -```javascript +// Add persistence - one line const brain = new Brainy({ storage: { type: 'filesystem', path: './brainy-data', - compression: true // 60-80% space savings! + compression: true // 60-80% space savings } }) ``` -### โ˜๏ธ **Production: Cloud Storage** (NEW in v4.0.0) +**Perfect for:** Startups, MVPs, internal tools, team knowledge bases +**Scale:** Thousands to hundreds of thousands of entities +**Performance:** <5ms queries, sub-second imports -Choose your cloud provider - all support **automatic cost optimization:** +### ๐Ÿข **Growing Company** โ†’ Multi-Million Entity Scale -#### **AWS S3 / Cloudflare R2 / DigitalOcean Spaces** ```javascript +// Scale to cloud - same API const brain = new Brainy({ storage: { type: 's3', s3Storage: { bucketName: 'my-knowledge-base', - region: 'us-east-1', - accessKeyId: process.env.AWS_ACCESS_KEY_ID, - secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY + region: 'us-east-1' } - } + }, + hnsw: { typeAware: true } // 87% memory reduction }) - -// Enable Intelligent-Tiering for automatic 96% cost savings -await brain.storage.enableIntelligentTiering('entities/', 'brainy-auto-tier') ``` -#### **Google Cloud Storage** +**Perfect for:** SaaS products, e-commerce, content platforms, enterprise apps +**Scale:** Millions of entities +**Performance:** <10ms queries, 12GB memory @ 10M entities +**Features:** Auto-scaling, distributed storage, cost optimization (96% savings) + +### ๐ŸŒ **Enterprise / Planet Scale** โ†’ Billion+ Entities + ```javascript +// Billion-scale - STILL the same API const brain = new Brainy({ storage: { type: 'gcs', - gcsStorage: { - bucketName: 'my-knowledge-base', - keyFilename: '/path/to/service-account.json' - } + gcsStorage: { bucketName: 'global-knowledge' } + }, + hnsw: { + typeAware: true, + M: 32, + efConstruction: 400 } }) -// Enable Autoclass for automatic 94% cost savings -await brain.storage.enableAutoclass({ terminalStorageClass: 'ARCHIVE' }) -``` - -#### **Azure Blob Storage** -```javascript -const brain = new Brainy({ - storage: { - type: 'azure', - azureStorage: { - containerName: 'knowledge-base', - connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING - } - } +// Enable intelligent archival +await brain.storage.enableAutoclass({ + terminalStorageClass: 'ARCHIVE' }) - -// Batch tier changes for 99% cost savings -await brain.storage.setBlobTierBatch( - oldBlobs.map(name => ({ blobName: name, tier: 'Archive' })) -) ``` -### ๐Ÿ’ฐ **Cost Optimization Impact** +**Perfect for:** Fortune 500, global platforms, research institutions, government +**Scale:** Billions of entities (tested at 1B+) +**Performance:** 18ms queries @ 1B scale, 50GB memory (87% reduction) +**Cost:** $138k/year โ†’ $6k/year with intelligent tiering (96% savings) +**Features:** Sharding, replication, monitoring, enterprise SLAs -| Scale | Before | After (Archive) | Savings/Year | -|-------|--------|-----------------|--------------| -| 5TB | $1,380/year | $59/year | **$1,321 (96%)** | -| 50TB | $13,800/year | $594/year | **$13,206 (96%)** | -| 500TB | $138,000/year | $5,940/year | **$132,060 (96%)** | +### ๐ŸŽฏ **The Point** -**[๐Ÿ“– AWS S3 Cost Guide โ†’](docs/operations/cost-optimization-aws-s3.md)** | **[GCS โ†’](docs/operations/cost-optimization-gcs.md)** | **[Azure โ†’](docs/operations/cost-optimization-azure.md)** | **[R2 โ†’](docs/operations/cost-optimization-cloudflare-r2.md)** +**Start simple. Scale infinitely. Never rewrite.** + +Most systems force you to choose: +- Simple but doesn't scale (SQLite, Redis) +- Scales but complex (Kubernetes + 7 databases) + +**Brainy gives you both:** Starts simple as SQLite. Scales like Google. --- -## ๐Ÿš€ Production Scale Features (NEW in v4.0.0) +## Why Brainy Is Revolutionary -### ๐ŸŽฏ **Type-Aware HNSW - 87% Memory Reduction** +### ๐Ÿง  **Triple Intelligenceโ„ข** โ€” The Impossible Made Possible -**Billion-scale deployments made affordable:** +**The world's first to unify three database paradigms in ONE API:** -- **Memory @ 1B entities**: 384GB โ†’ 50GB (-87%) -- **Single-type queries**: 10x faster (search 100M instead of 1B) -- **Multi-type queries**: 5-8x faster -- **Optimized rebuilds**: 31x faster with type filtering +| What You Get | Like Having | But Unified | +|-------------|-------------|-------------| +| ๐Ÿ” **Vector Search** | Pinecone, Weaviate | Find by meaning | +| ๐Ÿ•ธ๏ธ **Graph Relationships** | Neo4j, ArangoDB | Navigate connections | +| ๐Ÿ“Š **Document Filtering** | MongoDB, Elasticsearch | Query metadata | +**Every other system makes you choose.** Brainy does all three together. + +**Why this matters:** Your data isn't just vectors or just documents or just graphs. It's all three at once. A research paper is semantically similar to other papers (vector), written by an author (graph), and published in 2023 (document). **Brainy is the only system that understands this.** + +### ๐ŸŽฏ **31 Noun Types ร— 40 Verb Types = Universal Protocol** + +Model **any domain** with mathematical completeness: + +``` +31 Nouns ร— 40 Verbs ร— โˆž Metadata = 1,240+ base combinations +``` + +**Real-world expressiveness:** +- Healthcare: `Patient โ†’ diagnoses โ†’ Condition` +- Finance: `Account โ†’ transfers โ†’ Transaction` +- Manufacturing: `Product โ†’ assembles โ†’ Component` +- Education: `Student โ†’ completes โ†’ Course` +- **YOUR domain** โ†’ Your types + relationships = Your knowledge graph + +[โ†’ See the Mathematical Proof](docs/architecture/noun-verb-taxonomy.md) + +### โšก **Zero Configuration Philosophy** + +**We hate configuration files. So we eliminated them.** + +```javascript +const brain = new Brainy() // Auto-detects everything +await brain.init() // Optimizes for your environment +``` + +Brainy automatically: +- Detects optimal storage (memory/filesystem/cloud) +- Configures memory based on available RAM +- Optimizes for containers (Docker/K8s) +- Tunes indexes for your data patterns +- Manages embedding models and caching + +**You write business logic. Brainy handles infrastructure.** + +--- + +## What Can You Build? + +**If your app needs to remember, understand, or connect information โ€” Brainy makes it trivial.** + +### ๐Ÿค– **AI Agents with Perfect Memory** +Give your AI unlimited context that persists forever. Not just chat history โ€” true understanding of relationships, evolution, and meaning over time. + +**Examples:** Personal assistants, code assistants, conversational AI, research agents + +### ๐Ÿ“š **Living Documentation & Knowledge Bases** +Documentation that understands itself. Auto-links related concepts, detects outdated information, finds connections across your entire knowledge base. + +**Examples:** Internal wikis, research platforms, smart documentation, learning systems + +### ๐Ÿ” **Semantic Search at Any Scale** +Find by meaning, not keywords. Search codebases, research papers, customer data, or media libraries with natural language. + +**Examples:** Code search, research platforms, content discovery, recommendation engines + +### ๐Ÿข **Enterprise Knowledge Management** +Corporate memory that never forgets. Track every customer interaction, product evolution, and business relationship. + +**Examples:** CRM systems, product catalogs, customer intelligence, institutional knowledge + +### ๐ŸŽฎ **Rich Interactive Experiences** +NPCs that remember. Characters that persist across stories. Worlds that evolve based on real relationships. + +**Examples:** Game worlds, interactive fiction, educational platforms, creative tools + +### ๐ŸŽจ **Content & Media Platforms** +Every asset knows its relationships. Intelligent tagging, similarity-based discovery, and relationship-aware management. + +**Examples:** DAM systems, media libraries, writing assistants, content management + +**The pattern:** Knowledge that needs to live, connect, and evolve. That's what Brainy was built for. + +--- + +## Quick Start + +```bash +npm install @soulcraft/brainy +``` + +### Your First Knowledge Graph (60 seconds) + +```javascript +import { Brainy, NounType, VerbType } from '@soulcraft/brainy' + +const brain = new Brainy() +await brain.init() + +// Add knowledge +const jsId = await brain.add({ + data: "JavaScript is a programming language", + type: NounType.Concept, + metadata: { category: "language", year: 1995 } +}) + +const nodeId = await brain.add({ + data: "Node.js runtime environment", + type: NounType.Concept, + metadata: { category: "runtime", year: 2009 } +}) + +// Create relationships +await brain.relate({ from: nodeId, to: jsId, type: VerbType.Executes }) + +// Query with Triple Intelligence +const results = await brain.find({ + query: "JavaScript", // ๐Ÿ” Vector + where: { category: "language" }, // ๐Ÿ“Š Document + connected: { from: nodeId, depth: 1 } // ๐Ÿ•ธ๏ธ Graph +}) +``` + +**Done.** No configuration. No complexity. Production-ready from day one. + +--- + +## Core Features + +### ๐Ÿง  **Natural Language Queries** + +```javascript +// Ask naturally - Brainy understands +await brain.find("recent React components with tests") +await brain.find("JavaScript libraries similar to Vue") + +// Or use structured Triple Intelligence queries +await brain.find({ + query: "React", + where: { type: "library", year: { greaterThan: 2020 } }, + connected: { to: "JavaScript", depth: 2 } +}) +``` + +### ๐ŸŒ **Virtual Filesystem** โ€” Intelligent File Management + +Build file explorers and IDEs that never crash: + +```javascript +const vfs = brain.vfs() + +// Tree-aware operations prevent infinite recursion +const tree = await vfs.getTreeStructure('/projects', { maxDepth: 3 }) + +// Semantic file search +const reactFiles = await vfs.search('React components with hooks') +``` + +**[๐Ÿ“– VFS Quick Start โ†’](docs/vfs/QUICK_START.md)** | **[Common Patterns โ†’](docs/vfs/COMMON_PATTERNS.md)** | **[Neural Extraction โ†’](docs/vfs/NEURAL_EXTRACTION.md)** + +### ๐Ÿš€ **Import Anything** โ€” CSV, Excel, PDF, URLs + +```javascript +await brain.import('customers.csv') // Auto-detects everything +await brain.import('sales-data.xlsx', { excelSheets: ['Q1', 'Q2'] }) +await brain.import('research-paper.pdf', { pdfExtractTables: true }) +await brain.import('https://api.example.com/data.json') +``` + +**[๐Ÿ“– Complete Import Guide โ†’](docs/guides/import-anything.md)** + +### ๐Ÿง  **Neural API** โ€” Advanced Semantic Analysis + +```javascript +// Clustering, similarity, outlier detection, visualization +const clusters = await brain.neural.clusters({ algorithm: 'kmeans' }) +const similarity = await brain.neural.similar('item1', 'item2') +const outliers = await brain.neural.outliers(0.3) +const vizData = await brain.neural.visualize({ maxNodes: 100 }) +``` + +--- + +## Framework Integration + +**Works with any modern framework.** React, Vue, Angular, Svelte, Solid.js โ€” your choice. + +```javascript +// React +const [brain] = useState(() => new Brainy()) +useEffect(() => { brain.init() }, []) + +// Vue +async mounted() { this.brain = await new Brainy().init() } + +// Angular +@Injectable() export class BrainyService { brain = new Brainy() } +``` + +**Supports:** All bundlers (Webpack, Vite, Rollup) โ€ข SSR/SSG โ€ข Edge runtimes โ€ข Browser/Node.js + +**[๐Ÿ“– Framework Integration Guide โ†’](docs/guides/framework-integration.md)** | **[Next.js โ†’](docs/guides/nextjs-integration.md)** | **[Vue โ†’](docs/guides/vue-integration.md)** + +--- + +## Storage โ€” From Memory to Planet-Scale + +### Development โ†’ Just Works +```javascript +const brain = new Brainy() // Memory storage, zero config +``` + +### Production โ†’ Persistence with Compression ```javascript const brain = new Brainy({ - hnsw: { - typeAware: true // Enable type-aware indexing + storage: { type: 'filesystem', path: './data', compression: true } +}) +// 60-80% space savings with gzip +``` + +### Cloud โ†’ AWS, GCS, Azure, Cloudflare R2 +```javascript +// AWS S3 / Cloudflare R2 +const brain = new Brainy({ + storage: { + type: 's3', + s3Storage: { + bucketName: 'my-knowledge-base', + region: 'us-east-1' + } } }) + +// Enable Intelligent-Tiering: 96% cost savings +await brain.storage.enableIntelligentTiering('entities/', 'auto-tier') ``` -### โšก **Production-Ready Storage** +**Cost optimization at scale:** -**NEW v4.0.0 enterprise features:** +| Scale | Standard | With Intelligent Tiering | Annual Savings | +|-------|----------|--------------------------|----------------| +| 5TB | $1,380 | $59 | $1,321 (96%) | +| 50TB | $13,800 | $594 | $13,206 (96%) | +| 500TB | $138,000 | $5,940 | $132,060 (96%) | -- **๐Ÿ—‘๏ธ Batch Operations**: Delete thousands of entities with retry logic -- **๐Ÿ“ฆ Gzip Compression**: 60-80% space savings (FileSystem) -- **๐Ÿ’ฝ OPFS Quota Monitoring**: Real-time quota tracking (Browser) -- **๐Ÿ”„ Metadata/Vector Separation**: Billion-entity scalability -- **๐Ÿ›ก๏ธ Enterprise Reliability**: Backpressure, circuit breakers, retries +**[๐Ÿ“– Cloud Storage Guide โ†’](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** | **[AWS Cost Optimization โ†’](docs/operations/cost-optimization-aws-s3.md)** | **[GCS โ†’](docs/operations/cost-optimization-gcs.md)** | **[Azure โ†’](docs/operations/cost-optimization-azure.md)** + +--- + +## Production Features + +### ๐ŸŽฏ Type-Aware HNSW Indexing โ€” 87% Memory Reduction + +Scale to billions affordably: + +- **1B entities:** 384GB โ†’ 50GB memory (-87%) +- **Single-type queries:** 10x faster +- **Multi-type queries:** 5-8x faster ```javascript -// Batch delete with retry logic -await brain.storage.batchDelete(keys, { - maxRetries: 3, - continueOnError: true -}) +const brain = new Brainy({ hnsw: { typeAware: true } }) +``` -// Get storage status +**[๐Ÿ“– How Type-Aware Indexing Works โ†’](docs/architecture/data-storage-architecture.md)** + +### โšก Enterprise-Ready Operations (v4.0.0) + +- **Batch operations** with retry logic (1000x faster deletes) +- **Gzip compression** (60-80% space savings) +- **OPFS quota monitoring** (browser storage) +- **Metadata/Vector separation** (billion-entity scalability) +- **Circuit breakers & backpressure** (enterprise reliability) + +```javascript +// Batch operations +await brain.storage.batchDelete(keys, { maxRetries: 3 }) + +// Monitor storage const status = await brain.storage.getStorageStatus() -console.log(`Used: ${status.used}, Quota: ${status.quota}`) ``` -### ๐Ÿ“Š **Adaptive Memory Management** +### ๐Ÿ“Š Adaptive Memory Management -**Auto-scales from 2GB to 128GB+ based on resources:** +Auto-scales 2GB โ†’ 128GB+ based on environment: -- Container-aware (Docker/K8s cgroups v1/v2) -- Environment-smart (25% dev, 40% container, 50% production) -- Model memory accounting (150MB Q8, 250MB FP32) -- Built-in cache monitoring with recommendations +- Container-aware (Docker/K8s cgroups) +- Environment-optimized (dev/staging/production) +- Built-in cache monitoring with tuning recommendations ```javascript -// Get cache statistics and recommendations -const stats = brain.getCacheStats() -console.log(`Hit rate: ${stats.hitRate * 100}%`) -// Actionable tuning recommendations included +const stats = brain.getCacheStats() // Performance insights ``` **[๐Ÿ“– Capacity Planning Guide โ†’](docs/operations/capacity-planning.md)** --- -## ๐Ÿข Enterprise Features - No Paywalls +## Benchmarks -Brainy includes **enterprise-grade capabilities at no extra cost**: - -- โœ… **Scales to billions of entities** with 18ms search latency @ 1B scale -- โœ… **Distributed architecture** with sharding and replication -- โœ… **Read/write separation** for horizontal scaling -- โœ… **Connection pooling** and request deduplication -- โœ… **Built-in monitoring** with metrics and health checks -- โœ… **Production ready** with circuit breakers and backpressure - -**No premium tiers. No feature gates. Everyone gets the same powerful system.** +| Operation | Performance | Memory | +|-----------|-------------|--------| +| Initialize | 450ms | 24MB | +| Add entity | 12ms | +0.1MB | +| Vector search (1K) | 3ms | - | +| Metadata filter (10K) | 0.8ms | - | +| Bulk import (1K) | 2.3s | +8MB | +| **10M entities** | **5.8ms** | **12GB** | +| **1B entities** | **18ms** | **50GB** | --- -## ๐Ÿ“Š Benchmarks +## ๐Ÿง  Deep Dive: How Brainy Actually Works -| Operation | Performance | Memory | -|----------------------------------|-------------|----------| -| Initialize | 450ms | 24MB | -| Add Item | 12ms | +0.1MB | -| Vector Search (1k items) | 3ms | - | -| Metadata Filter (10k items) | 0.8ms | - | -| Natural Language Query | 15ms | - | -| Bulk Import (1000 items) | 2.3s | +8MB | -| **Production Scale (10M items)** | **5.8ms** | **12GB** | -| **Billion Scale (1B items)** | **18ms** | **50GB** | +**Want to understand the magic under the hood?** + +### ๐Ÿ” Triple Intelligence & find() API +Understand how vector search, graph relationships, and document filtering work together in one unified query: + +**[๐Ÿ“– Triple Intelligence Architecture โ†’](docs/architecture/triple-intelligence.md)** +**[๐Ÿ“– Natural Language Guide โ†’](docs/guides/natural-language.md)** +**[๐Ÿ“– API Reference: find() โ†’](docs/api/README.md)** + +### ๐Ÿ—‚๏ธ Type-Aware Indexing & HNSW +Learn how we achieve 87% memory reduction and 10x query speedups at billion-scale: + +**[๐Ÿ“– Data Storage Architecture โ†’](docs/architecture/data-storage-architecture.md)** +**[๐Ÿ“– Architecture Overview โ†’](docs/architecture/overview.md)** + +### ๐Ÿ“ˆ Scaling: Individual โ†’ Planet +Understand how the same code scales from prototype to billions of entities: + +**[๐Ÿ“– Capacity Planning โ†’](docs/operations/capacity-planning.md)** +**[๐Ÿ“– Cloud Deployment Guide โ†’](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** + +### ๐ŸŽฏ The Universal Type System +Explore the mathematical foundation: 31 nouns ร— 40 verbs = any domain: + +**[๐Ÿ“– Noun-Verb Taxonomy โ†’](docs/architecture/noun-verb-taxonomy.md)** --- -## ๐ŸŽฏ Use Cases - -### Knowledge Management -```javascript -// Create knowledge graph with relationships -const apiGuide = await brain.add("REST API Guide", { - nounType: NounType.Document, - category: "documentation" -}) - -const author = await brain.add("Jane Developer", { - nounType: NounType.Person, - role: "tech-lead" -}) - -await brain.relate(author, apiGuide, "authored") - -// Query naturally -const docs = await brain.find( - "documentation authored by tech leads for active projects" -) -``` - -### AI Memory Layer -```javascript -// Store conversation context with relationships -const userId = await brain.add("User 123", { - nounType: NounType.User, - tier: "premium" -}) - -const messageId = await brain.add(userMessage, { - nounType: NounType.Message, - timestamp: Date.now() -}) - -await brain.relate(userId, messageId, "sent") - -// Retrieve context -const context = await brain.find({ - where: {type: "message"}, - connected: {from: userId, type: "sent"}, - like: "previous product issues" -}) -``` - -### Semantic Search -```javascript -// Find similar content -const similar = await brain.search(existingContent, { - limit: 5, - threshold: 0.8 -}) -``` - ---- - -## ๐Ÿ› ๏ธ CLI +## CLI Tools ```bash npm install -g brainy -# Add data brainy add "JavaScript is awesome" --metadata '{"type":"opinion"}' - -# Search -brainy search "programming" - -# Natural language find brainy find "awesome programming languages" - -# Interactive mode -brainy chat +brainy search "programming" ``` ---- - -## ๐Ÿ“– Documentation - -### Getting Started -- [Getting Started Guide](docs/guides/getting-started.md) -- [v4.0.0 Migration Guide](docs/MIGRATION-V3-TO-V4.md) **โ† NEW** -- [AWS S3 Cost Guide](docs/operations/cost-optimization-aws-s3.md) | [GCS](docs/operations/cost-optimization-gcs.md) | [Azure](docs/operations/cost-optimization-azure.md) | [R2](docs/operations/cost-optimization-cloudflare-r2.md) **โ† NEW** - -### Framework Integration -- [Framework Integration Guide](docs/guides/framework-integration.md) -- [Next.js Integration](docs/guides/nextjs-integration.md) -- [Vue.js Integration](docs/guides/vue-integration.md) - -### Virtual Filesystem -- [VFS Core Documentation](docs/vfs/VFS_CORE.md) -- [Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md) -- [Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md) - -### Core Documentation -- [API Reference](docs/api/README.md) -- [Architecture Overview](docs/architecture/overview.md) -- [Data Storage Architecture](docs/architecture/data-storage-architecture.md) -- [Natural Language Guide](docs/guides/natural-language.md) -- [Triple Intelligence](docs/architecture/triple-intelligence.md) -- [Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md) - -### Operations & Production -- [Capacity Planning](docs/operations/capacity-planning.md) -- [Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md) **โ† NEW** -- [AWS S3 Cost Guide](docs/operations/cost-optimization-aws-s3.md) | [GCS](docs/operations/cost-optimization-gcs.md) | [Azure](docs/operations/cost-optimization-azure.md) | [R2](docs/operations/cost-optimization-cloudflare-r2.md) **โ† NEW** +47 commands available, including storage management, imports, and neural operations. --- -## ๐Ÿ’– Support Brainy +## Documentation -Brainy is **free and open source** - no paywalls, no premium tiers, no feature gates. If Brainy helps your project, consider supporting development: +### ๐Ÿš€ Getting Started +- **[Getting Started Guide](docs/guides/getting-started.md)** โ€” Your first steps with Brainy +- **[v4.0.0 Migration Guide](docs/MIGRATION-V3-TO-V4.md)** โ€” Upgrade from v3 (backward compatible) -- โญ **Star us on [GitHub](https://github.com/soulcraftlabs/brainy)** -- ๐Ÿ’ **Sponsor via [GitHub Sponsors](https://github.com/sponsors/soulcraftlabs)** -- ๐Ÿ› **Report issues** and contribute code -- ๐Ÿ“ฃ **Share** with your team and community +### ๐Ÿง  Core Concepts +- **[Triple Intelligence Architecture](docs/architecture/triple-intelligence.md)** โ€” How vector + graph + document work together +- **[Natural Language Queries](docs/guides/natural-language.md)** โ€” Using find() effectively +- **[API Reference](docs/api/README.md)** โ€” Complete API documentation +- **[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** โ€” The universal type system -Your support keeps Brainy free for everyone and enables continued development of enterprise features at no cost. +### ๐Ÿ—๏ธ Architecture & Scaling +- **[Architecture Overview](docs/architecture/overview.md)** โ€” System design and components +- **[Data Storage Architecture](docs/architecture/data-storage-architecture.md)** โ€” Type-aware indexing and HNSW +- **[Capacity Planning](docs/operations/capacity-planning.md)** โ€” Memory, storage, and scaling guidelines + +### โ˜๏ธ Production & Operations +- **[Cloud Deployment Guide](docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md)** โ€” Deploy to AWS, GCS, Azure +- **[AWS Cost Optimization](docs/operations/cost-optimization-aws-s3.md)** | **[GCS](docs/operations/cost-optimization-gcs.md)** | **[Azure](docs/operations/cost-optimization-azure.md)** | **[Cloudflare R2](docs/operations/cost-optimization-cloudflare-r2.md)** + +### ๐ŸŒ Framework Integration +- **[Framework Integration Guide](docs/guides/framework-integration.md)** โ€” React, Vue, Angular, Svelte +- **[Next.js Integration](docs/guides/nextjs-integration.md)** +- **[Vue.js Integration](docs/guides/vue-integration.md)** + +### ๐ŸŒณ Virtual Filesystem +- **[VFS Quick Start](docs/vfs/QUICK_START.md)** โ€” Build file explorers that never crash +- **[VFS Core Documentation](docs/vfs/VFS_CORE.md)** +- **[Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md)** +- **[Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md)** + +### ๐Ÿ“ฆ Data Import +- **[Import Anything Guide](docs/guides/import-anything.md)** โ€” CSV, Excel, PDF, URLs --- -## ๐Ÿ“‹ System Requirements +## What's New in v4.0.0 -**Node.js Version:** 22 LTS (recommended) +**Enterprise-scale cost optimization and performance improvements:** -- โœ… **Node.js 22 LTS** - Fully supported (recommended for production) -- โœ… **Node.js 20 LTS** - Compatible (maintenance mode) -- โŒ **Node.js 24** - Not supported (ONNX compatibility issues) +- ๐ŸŽฏ **96% cloud storage cost savings** with intelligent tiering (AWS, GCS, Azure) +- โšก **1000x faster batch deletions** (533 entities/sec vs 0.5/sec) +- ๐Ÿ“ฆ **60-80% compression** with gzip (FileSystem storage) +- ๐Ÿ”„ **Enhanced metadata/vector separation** for billion-scale deployments -If using nvm: `nvm use` (we provide a `.nvmrc` file) +**[๐Ÿ“– Full v4.0.0 Changelog โ†’](CHANGELOG.md)** | **[Migration Guide โ†’](docs/MIGRATION-V3-TO-V4.md)** (100% backward compatible) --- -## ๐Ÿง  The Knowledge Operating System Explained +## Requirements -### How We Achieved The Impossible - -**Triple Intelligenceโ„ข** unifies three database paradigms that were previously incompatible: - -1. **Vector databases** (Pinecone, Weaviate) - semantic similarity -2. **Graph databases** (Neo4j, ArangoDB) - relationships -3. **Document databases** (MongoDB, Elasticsearch) - metadata filtering - -**Others make you choose. Brainy does all three together.** - -### The Math of Infinite Expressiveness - -``` -31 Nouns ร— 40 Verbs ร— โˆž Metadata ร— Triple Intelligence = Universal Protocol -``` - -- **1,240 base combinations** from standardized types -- **โˆž domain specificity** via unlimited metadata -- **โˆž relationship depth** via graph traversal -- **= Model ANYTHING**: From quantum physics to social networks - -### Why This Changes Everything - -**Like HTTP for the web, Brainy for knowledge:** - -- All augmentations compose perfectly - same noun-verb language -- All AI models share knowledge - GPT, Claude, Llama all understand -- All tools integrate seamlessly - no translation layers -- All data flows freely - perfect portability - -**The Vision**: One protocol. All knowledge. Every tool. Any AI. - -**Proven across industries**: Healthcare, Finance, Manufacturing, Education, Legal, Retail, Government, and beyond. - -[โ†’ See the Mathematical Proof & Full Taxonomy](docs/architecture/noun-verb-taxonomy.md) - ---- - -## ๐Ÿข Enterprise & Cloud - -**Brain Cloud** - Managed Brainy with team sync, persistent memory, and enterprise connectors. +**Node.js 22 LTS** (recommended) or **Node.js 20 LTS** ```bash -brainy cloud setup +nvm use # We provide .nvmrc ``` -Visit [soulcraft.com](https://soulcraft.com) for more information. +--- + +## Why Brainy Exists + +**The Vision:** Traditional systems force you to choose between vector databases, graph databases, and document stores. You need all three, but combining them is complex and fragile. + +**Brainy solved the impossible:** One API. All three paradigms. Any scale. + +Like HTTP standardized web communication, **Brainy standardizes knowledge representation.** One protocol that any AI model understands. One system that scales from prototype to planet. + +**[๐Ÿ“– Read the Mathematical Proof โ†’](docs/architecture/noun-verb-taxonomy.md)** --- -## ๐Ÿค Contributing +## Enterprise & Support -We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. +**๐Ÿข Brain Cloud** โ€” Managed Brainy with team sync, persistent memory, and enterprise connectors. +Visit **[soulcraft.com](https://soulcraft.com)** for more information. + +**๐Ÿ’– Support Development:** +- โญ Star us on **[GitHub](https://github.com/soulcraftlabs/brainy)** +- ๐Ÿ’ Sponsor via **[GitHub Sponsors](https://github.com/sponsors/soulcraftlabs)** +- ๐Ÿ› Report issues and contribute code +- ๐Ÿ“ฃ Share with your team and community + +**Brainy is 100% free and open source.** No paywalls, no premium tiers, no feature gates. --- -## ๐Ÿ“„ License +## Contributing + +We welcome contributions! See **[CONTRIBUTING.md](CONTRIBUTING.md)** for guidelines. + +--- + +## License MIT ยฉ Brainy Contributors @@ -724,6 +629,6 @@ MIT ยฉ Brainy Contributors

Built with โค๏ธ by the Brainy community
- Zero-Configuration AI Database with Triple Intelligenceโ„ข
- v4.0.0 - Production-Scale Storage with 99% Cost Savings + The Knowledge Operating System
+ From prototype to planet-scale โ€ข Zero configuration โ€ข Triple Intelligenceโ„ข