From d2cef55644864698e45327c9a11b4e953760696b Mon Sep 17 00:00:00 2001 From: David Snelling Date: Mon, 4 Aug 2025 10:14:33 -0700 Subject: [PATCH] feat(tools): add claude-commit AI-powered git commit tool Implemented a comprehensive AI-powered commit message generator using Claude that: - Automatically generates Conventional Commit formatted messages - Analyzes git diff to create context-aware commit messages - Works globally across all git repositories with 'git cc' command - Supports multi-computer setup through portable dotfiles Major changes: - Added global claude-commit script with git aliases (git cc, git smart-commit) - Created organized documentation in docs/tools/claude-commit/ - Included portable dotfiles structure for easy multi-machine deployment - Updated README with TLDR Node.js quickstart section featuring all Brainy capabilities - Moved documentation section to bottom of README for better flow - Added CLAUDE.md with project-specific instructions for Claude Code The tool eliminates manual commit message writing by leveraging AI to understand code changes and generate properly formatted, meaningful commit messages that follow the Conventional Commits specification. --- .github/git-commit-instructions.md | 3 - .gitignore | 1 - README.md | 2457 +++++------------ docs/tools/claude-commit/README.md | 128 + docs/tools/claude-commit/dotfiles/README.md | 121 + .../claude-commit/dotfiles/bin/claude-commit | 131 + docs/tools/claude-commit/dotfiles/install.sh | 67 + .../dotfiles/setup-claude-commit.sh | 237 ++ docs/tools/claude-commit/setup.sh | 202 ++ scripts/claude-commit.sh | 15 + 10 files changed, 1565 insertions(+), 1797 deletions(-) delete mode 100644 .github/git-commit-instructions.md create mode 100644 docs/tools/claude-commit/README.md create mode 100644 docs/tools/claude-commit/dotfiles/README.md create mode 100755 docs/tools/claude-commit/dotfiles/bin/claude-commit create mode 100755 docs/tools/claude-commit/dotfiles/install.sh create mode 100755 docs/tools/claude-commit/dotfiles/setup-claude-commit.sh create mode 100755 docs/tools/claude-commit/setup.sh create mode 100755 scripts/claude-commit.sh diff --git a/.github/git-commit-instructions.md b/.github/git-commit-instructions.md deleted file mode 100644 index ba724198..00000000 --- a/.github/git-commit-instructions.md +++ /dev/null @@ -1,3 +0,0 @@ -Use Conventional Commit format -Specify the changes in a structured format -Add information about the purpose of the commit diff --git a/.gitignore b/.gitignore index 344ed849..e08106b2 100644 --- a/.gitignore +++ b/.gitignore @@ -70,4 +70,3 @@ debug*.ts /temp/ /temp-tests/ /brainy-models-package/node_modules/ -/CLAUDE.md diff --git a/README.md b/README.md index f7736fec..f4bced4b 100644 --- a/README.md +++ b/README.md @@ -7,1879 +7,750 @@ [![TypeScript](https://img.shields.io/badge/TypeScript-5.4.5-blue.svg)](https://www.typescriptlang.org/) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md) -[//]: # ([![Cartographer](https://img.shields.io/badge/Cartographer-Official%20Standard-brightgreen)](https://github.com/sodal-project/cartographer)) - **A powerful graph & vector data platform for AI applications across any environment** -## โœจ Overview +## โœจ What is Brainy? -Brainy combines the power of vector search with graph relationships in a lightweight, cross-platform database. Whether -you're building AI applications, recommendation systems, or knowledge graphs, Brainy provides the tools you need to -store, connect, and retrieve your data intelligently. +Imagine a database that thinks like you do - connecting ideas, finding patterns, and getting smarter over time. Brainy is the **AI-native database** that brings vector search and knowledge graphs together in one powerful, ridiculously easy-to-use package. -What makes Brainy special? It intelligently adapts to your environment! Brainy automatically detects your platform, -adjusts its storage strategy, and optimizes performance based on your usage patterns. The more you use it, the smarter -it gets - learning from your data to provide increasingly relevant results and connections. +### ๐Ÿš€ Why Developers Love Brainy -### ๐Ÿš€ Key Features +- **๐Ÿง  It Just Worksโ„ข** - No config files, no tuning parameters, no DevOps headaches. Brainy auto-detects your environment and optimizes itself +- **๐ŸŒ True Write-Once, Run-Anywhere** - Same code runs in React, Angular, Vue, Node.js, Deno, Bun, serverless, edge workers, and even vanilla HTML +- **โšก Scary Fast** - Handles millions of vectors with sub-millisecond search. Built-in GPU acceleration when available +- **๐ŸŽฏ Self-Learning** - Like having a database that goes to the gym. Gets faster and smarter the more you use it +- **๐Ÿ”ฎ AI-First Design** - Built for the age of embeddings, RAG, and semantic search. Your LLMs will thank you +- **๐ŸŽฎ Actually Fun to Use** - Clean API, great DX, and it does the heavy lifting so you can build cool stuff -- **๐Ÿง  Zero Configuration** - Auto-detects environment and optimizes automatically -- **โšก Production-Scale Performance** - Handles millions of vectors with sub-second search -- **๐ŸŽฏ Intelligent Partitioning** - Semantic clustering with auto-tuning -- **๐Ÿ“Š Adaptive Learning** - Gets smarter with usage, optimizes itself over time -- **๐Ÿ—„๏ธ Smart Storage** - OPFS, FileSystem, S3 auto-selection based on environment -- **๐Ÿ’พ Massive Memory Optimization** - 75% reduction with compression, intelligent caching -- **๐Ÿš€ Distributed Search** - Parallel processing with load balancing -- **๐Ÿ”„ Real-Time Adaptation** - Automatically adjusts to your data patterns -- **Run Everywhere** - Works in browsers, Node.js, serverless functions, and containers -- **Vector Search** - Find semantically similar content using embeddings -- **Advanced JSON Document Search** - Search within specific fields of JSON documents with field prioritization and - service-based field standardization -- **Graph Relationships** - Connect data with meaningful relationships -- **Streaming Pipeline** - Process data in real-time as it flows through the system -- **Extensible Augmentations** - Customize and extend functionality with pluggable components -- **Built-in Conduits** - Sync and scale across instances with WebSocket and WebRTC -- **TensorFlow Integration** - Use TensorFlow.js for high-quality embeddings -- **Persistent Storage** - Data persists across sessions and scales to any size -- **TypeScript Support** - Fully typed API with generics -- **CLI Tools & Web Service** - Command-line interface and REST API web service for data management -- **Model Control Protocol (MCP)** - Allow external AI models to access Brainy data and use augmentation pipeline as - tools - -## โšก Large-Scale Performance Optimizations - -**New in v0.36.0**: Brainy now includes 6 core optimizations that transform it from a prototype into a production-ready -system capable of handling millions of vectors: - -### ๐ŸŽฏ Performance Benchmarks - -| Dataset Size | Search Time | Memory Usage | API Calls Reduction | -|------------------|-------------|---------------|---------------------| -| **10k vectors** | ~50ms | Standard | N/A | -| **100k vectors** | ~200ms | 30% reduction | 50-70% fewer | -| **1M+ vectors** | ~500ms | 75% reduction | 50-90% fewer | - -### ๐Ÿง  6 Core Optimization Systems - -1. **๐ŸŽ›๏ธ Auto-Configuration System** - Detects environment, resources, and data patterns -2. **๐Ÿ”€ Semantic Partitioning** - Intelligent clustering with auto-tuning (4-32 clusters) -3. **๐Ÿš€ Distributed Search** - Parallel processing across partitions with load balancing -4. **๐Ÿง  Multi-Level Caching** - Hot/Warm/Cold caching with predictive prefetching -5. **๐Ÿ“ฆ Batch S3 Operations** - Reduces cloud storage API calls by 50-90% -6. **๐Ÿ’พ Advanced Compression** - Vector quantization and memory-mapping for large datasets - -### ๐ŸŽฏ Automatic Environment Detection - -| Environment | Auto-Configured | Performance Focus | -|----------------|-----------------------------|----------------------------------------| -| **Browser** | OPFS + Web Workers | Memory efficiency, 512MB-1GB limits | -| **Node.js** | FileSystem + Worker Threads | High performance, 4GB-8GB+ usage | -| **Serverless** | S3 + Memory cache | Cold start optimization, latency focus | - -### ๐Ÿ“Š Intelligent Scaling Strategy - -The system automatically adapts based on your dataset size: - -- **< 25k vectors**: Single optimized index, no partitioning needed -- **25k - 100k**: Semantic clustering (4-8 clusters), balanced performance -- **100k - 1M**: Advanced partitioning (8-16 clusters), scale-optimized -- **1M+ vectors**: Maximum optimization (16-32 clusters), enterprise-grade - -### ๐Ÿง  Adaptive Learning Features - -- **Performance Monitoring**: Tracks latency, cache hits, memory usage -- **Dynamic Tuning**: Adjusts parameters every 50 searches based on performance -- **Pattern Recognition**: Learns from access patterns to improve predictions -- **Self-Optimization**: Automatically enables/disables features based on workload - -> **๐Ÿ“– Full Documentation**: See the -> complete [Large-Scale Optimizations Guide](docs/optimization-guides/large-scale-optimizations.md) for detailed -> configuration options and advanced usage. - -## ๐Ÿš€ Live Demo - -**[Try the live demo](https://soulcraft-research.github.io/brainy/demo/index.html)** - Check out the interactive demo on -GitHub Pages that showcases Brainy's main features. - -## ๐Ÿ“Š What Can You Build? - -- **Semantic Search Engines** - Find content based on meaning, not just keywords -- **Recommendation Systems** - Suggest similar items based on vector similarity -- **Knowledge Graphs** - Build connected data structures with relationships -- **AI Applications** - Store and retrieve embeddings for machine learning models -- **AI-Enhanced Applications** - Build applications that leverage vector embeddings for intelligent data processing -- **Data Organization Tools** - Automatically categorize and connect related information -- **Adaptive Experiences** - Create applications that learn and evolve with your users -- **Model-Integrated Systems** - Connect external AI models to Brainy data and tools using MCP - -## ๐Ÿ”ง Installation +## ๐Ÿš€ Quick Start (30 seconds!) +### Node.js TLDR ```bash -npm install @soulcraft/brainy +# Install +npm install brainy + +# Use it ``` +```javascript +import { createAutoBrainy, NounType, VerbType } from 'brainy' -TensorFlow.js packages are included as bundled dependencies and will be automatically installed without any additional -configuration. - -### Additional Packages - -Brainy offers specialized packages for different use cases: - -#### CLI Package - -```bash -npm install -g @soulcraft/brainy-cli -``` - -Command-line interface for data management, bulk operations, and database administration. - -#### Web Service Package - -```bash -npm install @soulcraft/brainy-web-service -``` - -REST API web service wrapper that provides HTTP endpoints for search operations and database queries. - -## ๐Ÿš€ Quick Setup - Zero Configuration! - -**New in v0.36.0**: Brainy now automatically detects your environment and optimizes itself! Choose your scenario: - -### โœจ Instant Setup (Auto-Everything) - -```typescript -import { createAutoBrainy } from '@soulcraft/brainy' - -// That's it! Everything is auto-configured const brainy = createAutoBrainy() -// Add data and search - all optimizations enabled automatically -await brainy.addVector({ id: '1', vector: [0.1, 0.2, 0.3], text: 'Hello world' }) -const results = await brainy.search([0.1, 0.2, 0.3], 10) +// Add data with Nouns (entities) +const catId = await brainy.add("Siamese cats are elegant and vocal", { + noun: NounType.Thing, + breed: "Siamese", + category: "animal" +}) + +const ownerId = await brainy.add("John loves his pets", { + noun: NounType.Person, + name: "John Smith" +}) + +// Connect with Verbs (relationships) +await brainy.addVerb(ownerId, catId, { + verb: VerbType.Owns, + since: "2020-01-01" +}) + +// Search by meaning +const results = await brainy.searchText("feline companions", 5) + +// Search JSON documents by specific fields +const docs = await brainy.searchDocuments("Siamese", { + fields: ['breed', 'category'], // Search these fields + weights: { breed: 2.0 }, // Prioritize breed matches + limit: 10 +}) + +// Find relationships +const johnsPets = await brainy.getVerbsBySource(ownerId, VerbType.Owns) ``` -### ๐Ÿ“ฆ With S3 Storage (Still Auto-Configured) +That's it! No config, no setup, it just worksโ„ข -```typescript -import { createAutoBrainy } from '@soulcraft/brainy' +## ๐ŸŽญ Key Features -// Auto-detects AWS credentials from environment variables +### Core Capabilities +- **Vector Search** - Find semantically similar content using embeddings +- **Graph Relationships** - Connect data with meaningful relationships +- **JSON Document Search** - Search within specific fields with prioritization +- **Real-Time Syncing** - WebSocket and WebRTC for distributed instances +- **Streaming Pipeline** - Process data in real-time as it flows through +- **Model Control Protocol** - Let AI models access your data + +### Smart Optimizations +- **Auto-Configuration** - Detects environment and optimizes automatically +- **Adaptive Learning** - Gets smarter with usage, optimizes itself over time +- **Intelligent Partitioning** - Semantic clustering with auto-tuning +- **Multi-Level Caching** - Hot/warm/cold caching with predictive prefetching +- **Memory Optimization** - 75% reduction with compression for large datasets + +### Developer Experience +- **TypeScript Support** - Fully typed API with generics +- **Extensible Augmentations** - Customize and extend functionality +- **REST API** - Web service wrapper for HTTP endpoints +- **Auto-Complete** - IntelliSense for all APIs and types + +## ๐Ÿ“ฆ Installation + +### Main Package +```bash +npm install brainy +``` + +### Optional: Offline Models Package +```bash +npm install @soulcraft/brainy-models +``` + +The `@soulcraft/brainy-models` package provides **offline access** to the Universal Sentence Encoder model, eliminating network dependencies and ensuring consistent performance. Perfect for: +- **Air-gapped environments** - No internet? No problem +- **Consistent performance** - No network latency or throttling +- **Privacy-focused apps** - Keep everything local +- **High-reliability systems** - No external dependencies + +```javascript +import { createAutoBrainy } from 'brainy' +import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models' + +// Use the bundled model for offline operation const brainy = createAutoBrainy({ - bucketName: 'my-vector-storage' - // region: 'us-east-1' (default) - // AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from env + embeddingModel: BundledUniversalSentenceEncoder }) ``` -### ๐ŸŽฏ Scenario-Based Setup +## ๐ŸŽจ Build Amazing Things + +**๐Ÿค– AI Chat Applications** - Build ChatGPT-like apps with long-term memory and context awareness +**๐Ÿ” Semantic Search Engines** - Search by meaning, not keywords. Find "that thing that's like a cat but bigger" โ†’ returns "tiger" +**๐ŸŽฏ Recommendation Engines** - "Users who liked this also liked..." but actually good +**๐Ÿงฌ Knowledge Graphs** - Connect everything to everything. Wikipedia meets Neo4j meets magic +**๐Ÿ‘๏ธ Computer Vision Apps** - Store and search image embeddings. "Find all photos with dogs wearing hats" +**๐ŸŽต Music Discovery** - Find songs that "feel" similar. Spotify's Discover Weekly in your app +**๐Ÿ“š Smart Documentation** - Docs that answer questions. "How do I deploy to production?" โ†’ relevant guides +**๐Ÿ›ก๏ธ Fraud Detection** - Find patterns humans can't see. Anomaly detection on steroids +**๐ŸŒ Real-Time Collaboration** - Sync vector data across devices. Figma for AI data +**๐Ÿฅ Medical Diagnosis Tools** - Match symptoms to conditions using embedding similarity + +## ๐Ÿงฌ The Power of Nouns & Verbs + +Brainy uses a **graph-based data model** that mirrors how humans think - with **Nouns** (entities) connected by **Verbs** (relationships). This isn't just vectors in a void; it's structured, meaningful data. + +### ๐Ÿ“ Nouns (What Things Are) + +Nouns are your entities - the "things" in your data. Each noun has: +- A unique ID +- A vector representation (for similarity search) +- A type (Person, Document, Concept, etc.) +- Custom metadata + +**Available Noun Types:** + +| Category | Types | Use For | +|----------|-------|---------| +| **Core Entities** | `Person`, `Organization`, `Location`, `Thing`, `Concept`, `Event` | People, companies, places, objects, ideas, happenings | +| **Digital Content** | `Document`, `Media`, `File`, `Message`, `Content` | PDFs, images, videos, emails, posts, generic content | +| **Collections** | `Collection`, `Dataset` | Groups of items, structured data sets | +| **Business** | `Product`, `Service`, `User`, `Task`, `Project` | E-commerce, SaaS, project management | +| **Descriptive** | `Process`, `State`, `Role` | Workflows, conditions, responsibilities | + +### ๐Ÿ”— Verbs (How Things Connect) + +Verbs are your relationships - they give meaning to connections. Not just "these vectors are similar" but "this OWNS that" or "this CAUSES that". + +**Available Verb Types:** + +| Category | Types | Examples | +|----------|-------|----------| +| **Core** | `RelatedTo`, `Contains`, `PartOf`, `LocatedAt`, `References` | Generic relations, containment, location | +| **Temporal** | `Precedes`, `Succeeds`, `Causes`, `DependsOn`, `Requires` | Time sequences, causality, dependencies | +| **Creation** | `Creates`, `Transforms`, `Becomes`, `Modifies`, `Consumes` | Creation, change, consumption | +| **Ownership** | `Owns`, `AttributedTo`, `CreatedBy`, `BelongsTo` | Ownership, authorship, belonging | +| **Social** | `MemberOf`, `WorksWith`, `FriendOf`, `Follows`, `Likes`, `ReportsTo` | Social networks, organizations | +| **Functional** | `Describes`, `Implements`, `Validates`, `Triggers`, `Serves` | Functions, implementations, services | + +### ๐Ÿ’ก Why This Matters + +```javascript +// Traditional vector DB: Just similarity +const similar = await vectorDB.search(embedding, 10) +// Result: [vector1, vector2, ...] - What do these mean? ๐Ÿคท + +// Brainy: Similarity + Meaning + Relationships +const catId = await brainy.add("Siamese cat", { + noun: NounType.Thing, + breed: "Siamese" +}) +const ownerId = await brainy.add("John Smith", { + noun: NounType.Person +}) +await brainy.addVerb(ownerId, catId, { + verb: VerbType.Owns, + since: "2020-01-01" +}) + +// Now you can search with context! +const johnsPets = await brainy.getVerbsBySource(ownerId, VerbType.Owns) +const catOwners = await brainy.getVerbsByTarget(catId, VerbType.Owns) +``` + +## ๐Ÿค” Why Choose Brainy? + +### vs. Traditional Databases +โŒ **PostgreSQL with pgvector** - Requires complex setup, tuning, and DevOps expertise +โœ… **Brainy** - Zero config, auto-optimizes, works everywhere from browser to cloud + +### vs. Vector Databases +โŒ **Pinecone/Weaviate/Qdrant** - Cloud-only, expensive, vendor lock-in +โœ… **Brainy** - Run locally, in browser, or cloud. Your choice, your data + +### vs. Graph Databases +โŒ **Neo4j** - Great for graphs, no vector support +โœ… **Brainy** - Vectors + graphs in one. Best of both worlds + +### vs. DIY Solutions +โŒ **Building your own** - Months of work, optimization nightmares +โœ… **Brainy** - Production-ready in 30 seconds + +## ๐Ÿš€ Getting Started in 30 Seconds + +### React + +```jsx +import { createAutoBrainy } from 'brainy' +import { useEffect, useState } from 'react' + +function SemanticSearch() { + const [brainy] = useState(() => createAutoBrainy()) + const [results, setResults] = useState([]) + + const search = async (query) => { + const items = await brainy.searchText(query, 10) + setResults(items) + } + + return ( + search(e.target.value)} + placeholder="Search by meaning..." /> + ) +} +``` + +### Angular ```typescript -import { createQuickBrainy } from '@soulcraft/brainy' +import { Component, OnInit } from '@angular/core' +import { createAutoBrainy } from 'brainy' -// Choose your scale: 'small', 'medium', 'large', 'enterprise' -const brainy = await createQuickBrainy('large', { - bucketName: 'my-big-vector-db' +@Component({ + selector: 'app-search', + template: ` + +
+ {{ result.text }} +
+ ` +}) +export class SearchComponent implements OnInit { + brainy = createAutoBrainy() + results = [] + + async search(query: string) { + this.results = await this.brainy.searchText(query, 10) + } +} +``` + +### Vue 3 + +```vue + + + +``` + +### Svelte + +```svelte + + + +{#each results as result} +
{result.text}
+{/each} +``` + +### Next.js (App Router) + +```jsx +// app/search/page.js +import { createAutoBrainy } from 'brainy' + +export default function SearchPage() { + async function search(formData) { + 'use server' + const brainy = createAutoBrainy({ bucketName: 'vectors' }) + const query = formData.get('query') + return await brainy.searchText(query, 10) + } + + return ( +
+ + +
+ ) +} +``` + +### Node.js / Bun / Deno + +```javascript +import { createAutoBrainy } from 'brainy' + +const brainy = createAutoBrainy() + +// Add some data +await brainy.add("TypeScript is a typed superset of JavaScript", { + category: 'programming' +}) + +// Search for similar content +const results = await brainy.searchText("JavaScript with types", 5) +console.log(results) +``` + +### Vanilla JavaScript + +```html + + + + + + + +
+ + +``` + +### Cloudflare Workers + +```javascript +import { createAutoBrainy } from 'brainy' + +export default { + async fetch(request, env) { + const brainy = createAutoBrainy({ + bucketName: env.R2_BUCKET + }) + + const url = new URL(request.url) + const query = url.searchParams.get('q') + + const results = await brainy.searchText(query, 10) + return Response.json(results) + } +} +``` + +### AWS Lambda + +```javascript +import { createAutoBrainy } from 'brainy' + +export const handler = async (event) => { + const brainy = createAutoBrainy({ + bucketName: process.env.S3_BUCKET + }) + + const results = await brainy.searchText(event.query, 10) + + return { + statusCode: 200, + body: JSON.stringify(results) + } +} +``` + +### Azure Functions + +```javascript +import { createAutoBrainy } from 'brainy' + +module.exports = async function (context, req) { + const brainy = createAutoBrainy({ + bucketName: process.env.AZURE_STORAGE_CONTAINER + }) + + const results = await brainy.searchText(req.query.q, 10) + + context.res = { + body: results + } +} +``` + +### Google Cloud Functions + +```javascript +import { createAutoBrainy } from 'brainy' + +export const searchHandler = async (req, res) => { + const brainy = createAutoBrainy({ + bucketName: process.env.GCS_BUCKET + }) + + const results = await brainy.searchText(req.query.q, 10) + res.json(results) +} +``` + +### Google Cloud Run + +```dockerfile +# Dockerfile +FROM node:20-alpine +USER node +WORKDIR /app +COPY package*.json ./ +RUN npm install brainy +COPY . . +CMD ["node", "server.js"] +``` + +```javascript +// server.js +import { createAutoBrainy } from 'brainy' +import express from 'express' + +const app = express() +const brainy = createAutoBrainy({ + bucketName: process.env.GCS_BUCKET +}) + +app.get('/search', async (req, res) => { + const results = await brainy.searchText(req.query.q, 10) + res.json(results) +}) + +const port = process.env.PORT || 8080 +app.listen(port, () => console.log(`Brainy on Cloud Run: ${port}`)) +``` + +```bash +# Deploy to Cloud Run +gcloud run deploy brainy-api \ + --source . \ + --platform managed \ + --region us-central1 \ + --allow-unauthenticated +``` + +### Vercel Edge Functions + +```javascript +import { createAutoBrainy } from 'brainy' + +export const config = { + runtime: 'edge' +} + +export default async function handler(request) { + const brainy = createAutoBrainy() + const { searchParams } = new URL(request.url) + const query = searchParams.get('q') + + const results = await brainy.searchText(query, 10) + return Response.json(results) +} +``` + +### Netlify Functions + +```javascript +import { createAutoBrainy } from 'brainy' + +export async function handler(event, context) { + const brainy = createAutoBrainy() + const query = event.queryStringParameters.q + + const results = await brainy.searchText(query, 10) + + return { + statusCode: 200, + body: JSON.stringify(results) + } +} +``` + +### Supabase Edge Functions + +```typescript +import { createAutoBrainy } from 'brainy' +import { serve } from 'https://deno.land/std@0.168.0/http/server.ts' + +serve(async (req) => { + const brainy = createAutoBrainy() + const url = new URL(req.url) + const query = url.searchParams.get('q') + + const results = await brainy.searchText(query, 10) + + return new Response(JSON.stringify(results), { + headers: { 'Content-Type': 'application/json' } + }) }) ``` -| Scenario | Dataset Size | Memory Usage | S3 Required | Best For | -|--------------|---------------|--------------|-----------------|----------------------| -| `small` | โ‰ค10k vectors | โ‰ค1GB | No | Development, testing | -| `medium` | โ‰ค100k vectors | โ‰ค4GB | Serverless only | Production apps | -| `large` | โ‰ค1M vectors | โ‰ค8GB | Yes | Large applications | -| `enterprise` | โ‰ค10M vectors | โ‰ค32GB | Yes | Enterprise systems | +### Docker Container -### ๐Ÿง  What Auto-Configuration Does +```dockerfile +FROM node:20-alpine +USER node +WORKDIR /app +COPY package*.json ./ +RUN npm install brainy +COPY . . -- **๐ŸŽฏ Environment Detection**: Browser, Node.js, or Serverless -- **๐Ÿ’พ Smart Memory Management**: Uses available RAM optimally -- **๐Ÿ—„๏ธ Storage Selection**: OPFS, FileSystem, S3, or Memory -- **โšก Performance Tuning**: Threading, caching, compression -- **๐Ÿ“Š Adaptive Learning**: Improves performance over time -- **๐Ÿ” Semantic Partitioning**: Auto-clusters similar vectors +CMD ["node", "server.js"] +``` -## ๐Ÿ Traditional Setup (Manual Configuration) +```javascript +// server.js +import { createAutoBrainy } from 'brainy' +import express from 'express' -If you prefer manual control: +const app = express() +const brainy = createAutoBrainy() -```typescript -import { BrainyData, NounType, VerbType } from '@soulcraft/brainy' +app.get('/search', async (req, res) => { + const results = await brainy.searchText(req.query.q, 10) + res.json(results) +}) -// Create and initialize the database +app.listen(3000, () => console.log('Brainy running on port 3000')) +``` + +### Kubernetes + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: brainy-api +spec: + replicas: 3 + template: + spec: + containers: + - name: brainy + image: your-registry/brainy-api:latest + env: + - name: S3_BUCKET + value: "your-vector-bucket" +``` + +### Railway.app + +```javascript +// server.js +import { createAutoBrainy } from 'brainy' + +const brainy = createAutoBrainy({ + bucketName: process.env.RAILWAY_VOLUME_NAME +}) + +// Railway automatically handles the rest! +``` + +### Render.com + +```yaml +# render.yaml +services: + - type: web + name: brainy-api + env: node + buildCommand: npm install brainy + startCommand: node server.js + envVars: + - key: BRAINY_STORAGE + value: persistent-disk +``` + +## ๐Ÿš€ Quick Examples + +### Basic Usage + +```javascript +import { BrainyData, NounType, VerbType } from 'brainy' + +// Initialize const db = new BrainyData() await db.init() -// Add data (automatically converted to vectors) +// Add data (automatically vectorized) const catId = await db.add("Cats are independent pets", { noun: NounType.Thing, category: 'animal' }) -const dogId = await db.add("Dogs are loyal companions", { - noun: NounType.Thing, - category: 'animal' -}) - // Search for similar items -const results = await db.searchText("feline pets", 2) -console.log(results) +const results = await db.searchText("feline pets", 5) -// Add a relationship between items +// Add relationships await db.addVerb(catId, dogId, { verb: VerbType.RelatedTo, - description: 'Both are common household pets' + description: 'Both are pets' }) ``` -### Import Options +### AutoBrainy (Recommended) -```typescript -// Standard import - automatically adapts to any environment -import { BrainyData } from '@soulcraft/brainy' +```javascript +import { createAutoBrainy } from 'brainy' -// Minified version for production -import { BrainyData } from '@soulcraft/brainy/min' +// Everything auto-configured! +const brainy = createAutoBrainy() + +// Just start using it +await brainy.addVector({ id: '1', vector: [0.1, 0.2, 0.3], text: 'Hello' }) +const results = await brainy.search([0.1, 0.2, 0.3], 10) ``` -> **Note**: The CLI functionality is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size -> of the main package. Install it globally with `npm install -g @soulcraft/brainy-cli` to use the command-line -> interface. +### Scenario-Based Setup -### Browser Usage +```javascript +import { createQuickBrainy } from 'brainy' -```html - - +// Choose your scale: 'small', 'medium', 'large', 'enterprise' +const brainy = await createQuickBrainy('large', { + bucketName: 'my-vector-db' +}) ``` -Modern bundlers like Webpack, Rollup, and Vite will automatically use the unified build which adapts to any environment. +### With Offline Models -## ๐Ÿงฉ How It Works - -Brainy combines **six advanced optimization systems** with core vector database technologies to create a -production-ready, self-optimizing system: - -### ๐Ÿ”ง Core Technologies - -1. **Vector Embeddings** - Converts data (text, images, etc.) into numerical vectors using TensorFlow.js -2. **Optimized HNSW Algorithm** - Fast similarity search with semantic partitioning and distributed processing -3. **๐Ÿง  Auto-Configuration Engine** - Detects environment, resources, and data patterns to optimize automatically -4. **๐ŸŽฏ Intelligent Storage System** - Multi-level caching with predictive prefetching and batch operations - -### โšก Advanced Optimization Layer - -5. **Semantic Partitioning** - Auto-clusters similar vectors for faster search (4-32 clusters based on scale) -6. **Distributed Search** - Parallel processing across partitions with intelligent load balancing -7. **Multi-Level Caching** - Hot (RAM) โ†’ Warm (Fast Storage) โ†’ Cold (S3/Disk) with 70-90% hit rates -8. **Batch Operations** - Reduces S3 API calls by 50-90% through intelligent batching -9. **Adaptive Learning** - Continuously learns from usage patterns and optimizes performance -10. **Advanced Compression** - Vector quantization achieves 75% memory reduction for large datasets - -### ๐ŸŽฏ Environment-Specific Optimizations - -| Environment | Storage | Threading | Memory | Focus | -|----------------|-----------------|----------------|-----------|------------------| -| **Browser** | OPFS + Cache | Web Workers | 512MB-1GB | Responsiveness | -| **Node.js** | FileSystem + S3 | Worker Threads | 4GB-8GB+ | Throughput | -| **Serverless** | S3 + Memory | Limited | 1GB-2GB | Cold Start Speed | - -### ๐Ÿ”„ Adaptive Intelligence Flow - -``` -Data Input โ†’ Auto-Detection โ†’ Environment Optimization โ†’ Semantic Partitioning โ†’ -Distributed Search โ†’ Multi-Level Caching โ†’ Performance Learning โ†’ Self-Tuning -``` - -The system **automatically adapts** to your environment, learns from your usage patterns, and **continuously optimizes -itself** for better performance over time. - -## ๐Ÿš€ The Brainy Pipeline - -Brainy's data processing pipeline transforms raw data into searchable, connected knowledge that gets smarter over time: - -``` -Raw Data โ†’ Embedding โ†’ Vector Storage โ†’ Graph Connections โ†’ Adaptive Learning โ†’ Query & Retrieval -``` - -Each time data flows through this pipeline, Brainy learns more about your usage patterns and environment, making future -operations faster and more relevant. - -### Pipeline Stages - -1. **Data Ingestion** - - Raw text or pre-computed vectors enter the pipeline - - Data is validated and prepared for processing - -2. **Embedding Generation** - - Text is transformed into numerical vectors using embedding models - - Uses TensorFlow Universal Sentence Encoder for high-quality text embeddings - - Custom embedding functions can be plugged in for specialized domains - -3. **Vector Indexing** - - Vectors are indexed using the HNSW algorithm - - Hierarchical structure enables fast similarity search - - Configurable parameters for precision vs. performance tradeoffs - -4. **Graph Construction** - - Nouns (entities) become nodes in the knowledge graph - - Verbs (relationships) connect related entities - - Typed relationships add semantic meaning to connections - -5. **Adaptive Learning** - - Analyzes usage patterns to optimize future operations - - Tunes performance parameters based on your environment - - Adjusts search strategies based on query history - - Becomes more efficient and relevant the more you use it - -6. **Intelligent Storage** - - Data is saved using the optimal storage for your environment - - Automatic selection between OPFS, filesystem, S3, or memory - - Migrates between storage types as your application's needs evolve - - Scales from tiny datasets to massive data collections - - Configurable storage adapters for custom persistence needs - -### Augmentation Types - -Brainy uses a powerful augmentation system to extend functionality. Augmentations are processed in the following order: - -1. **SENSE** - - Ingests and processes raw, unstructured data into nouns and verbs - - Handles text, images, audio streams, and other input formats - - Example: Converting raw text into structured entities - -2. **MEMORY** - - Provides storage capabilities for data in different formats - - Manages persistence across sessions - - Example: Storing vectors in OPFS or filesystem - -3. **COGNITION** - - Enables advanced reasoning, inference, and logical operations - - Analyzes relationships between entities - - Examples: - - Inferring new connections between existing data - - Deriving insights from graph relationships - -4. **CONDUIT** - - Establishes channels for structured data exchange - - Connects with external systems and syncs between Brainy instances - - Two built-in iConduit augmentations for scaling out and syncing: - - **WebSocket iConduit** - Syncs data between browsers and servers - - **WebRTC iConduit** - Direct peer-to-peer syncing between browsers - - Examples: - - Integrating with third-party APIs - - Syncing Brainy instances between browsers using WebSockets - - Peer-to-peer syncing between browsers using WebRTC - -5. **ACTIVATION** - - Initiates actions, responses, or data manipulations - - Triggers events based on data changes - - Example: Sending notifications when new data is processed - -6. **PERCEPTION** - - Interprets, contextualizes, and visualizes identified nouns and verbs - - Creates meaningful representations of data - - Example: Generating visualizations of graph relationships - -7. **DIALOG** - - Facilitates natural language understanding and generation - - Enables conversational interactions - - Example: Processing user queries and generating responses - -8. **WEBSOCKET** - - Enables real-time communication via WebSockets - - Can be combined with other augmentation types - - Example: Streaming data processing in real-time - -### Streaming Data Support - -Brainy's pipeline is designed to handle streaming data efficiently: - -1. **WebSocket Integration** - - Built-in support for WebSocket connections - - Process data as it arrives without blocking - - Example: `setupWebSocketPipeline(url, dataType, options)` - -2. **Asynchronous Processing** - - Non-blocking architecture for real-time data handling - - Parallel processing of incoming streams - - Example: `createWebSocketHandler(connection, dataType, options)` - -3. **Event-Based Architecture** - - Augmentations can listen to data feeds and streams - - Real-time updates propagate through the pipeline - - Example: `listenToFeed(feedUrl, callback)` - -4. **Threaded Execution** - - Comprehensive multi-threading for high-performance operations - - Parallel processing for batch operations, vector calculations, and embedding generation - - Configurable execution modes (SEQUENTIAL, PARALLEL, THREADED) - - Automatic thread management based on environment capabilities - - Example: `executeTypedPipeline(augmentations, method, args, { mode: ExecutionMode.THREADED })` - -### Running the Pipeline - -The pipeline runs automatically when you: - -```typescript -// Add data (runs embedding โ†’ indexing โ†’ storage) -const id = await db.add("Your text data here", { metadata }) - -// Search (runs embedding โ†’ similarity search) -const results = await db.searchText("Your query here", 5) - -// Connect entities (runs graph construction โ†’ storage) -await db.addVerb(sourceId, targetId, { verb: VerbType.RelatedTo }) -``` - -Using the CLI: - -```bash -# Add data through the CLI pipeline -brainy add "Your text data here" '{"noun":"Thing"}' - -# Search through the CLI pipeline -brainy search "Your query here" --limit 5 - -# Connect entities through the CLI -brainy addVerb RelatedTo -``` - -### Extending the Pipeline - -Brainy's pipeline is designed for extensibility at every stage: - -1. **Custom Embedding** - ```typescript - // Create your own embedding function - const myEmbedder = async (text) => { - // Your custom embedding logic here - return [0.1, 0.2, 0.3, ...] // Return a vector - } - - // Use it in Brainy - const db = new BrainyData({ - embeddingFunction: myEmbedder - }) - ``` - -2. **Custom Distance Functions** - ```typescript - // Define your own distance function - const myDistance = (a, b) => { - // Your custom distance calculation - return Math.sqrt(a.reduce((sum, val, i) => sum + Math.pow(val - b[i], 2), 0)) - } - - // Use it in Brainy - const db = new BrainyData({ - distanceFunction: myDistance +```javascript +import { createAutoBrainy } from 'brainy' +import { BundledUniversalSentenceEncoder } from '@soulcraft/brainy-models' +// Use bundled model for offline operation +const brainy = createAutoBrainy({ + embeddingModel: BundledUniversalSentenceEncoder, + // Model loads from local files, no network needed! }) - ``` - -3. **Custom Storage Adapters** - ```typescript - // Implement the StorageAdapter interface - class MyStorage implements StorageAdapter { - // Your storage implementation - } - - // Use it in Brainy - const db = new BrainyData({ - storageAdapter: new MyStorage() - }) - ``` - -4. **Augmentations System** - ```typescript - // Create custom augmentations to extend functionality - const myAugmentation = { - type: 'memory', - name: 'my-custom-storage', - // Implementation details - } - - // Register with Brainy - db.registerAugmentation(myAugmentation) - +// Works exactly the same, but 100% offline +await brainy.add("This works without internet!", { + noun: NounType.Content +}) ``` -## Data Model +## ๐ŸŒ Live Demo -Brainy uses a graph-based data model with two primary concepts: +**[Try the interactive demo](https://soulcraft-research.github.io/brainy/demo/index.html)** - See Brainy in action with animations and examples. -### Nouns (Entities) +## ๐Ÿ”ง Environment Support -The main entities in your data (nodes in the graph): - -- Each noun has a unique ID, vector representation, and metadata -- Nouns can be categorized by type (Person, Place, Thing, Event, Concept, etc.) -- Nouns are automatically vectorized for similarity search - -### Verbs (Relationships) - -Connections between nouns (edges in the graph): - -- Each verb connects a source noun to a target noun -- Verbs have types that define the relationship (RelatedTo, Controls, Contains, etc.) -- Verbs can have their own metadata to describe the relationship - -### Type Utilities - -Brainy provides utility functions to access lists of noun and verb types: - -```typescript -import { - NounType, - VerbType, - getNounTypes, - getVerbTypes, - getNounTypeMap, - getVerbTypeMap -} from '@soulcraft/brainy' - -// At development time: -// Access specific types directly from the NounType and VerbType objects -console.log(NounType.Person) // 'person' -console.log(VerbType.Contains) // 'contains' - -// At runtime: -// Get a list of all noun types -const nounTypes = getNounTypes() // ['person', 'organization', 'location', ...] - -// Get a list of all verb types -const verbTypes = getVerbTypes() // ['relatedTo', 'contains', 'partOf', ...] - -// Get a map of noun type keys to values -const nounTypeMap = getNounTypeMap() // { Person: 'person', Organization: 'organization', ... } - -// Get a map of verb type keys to values -const verbTypeMap = getVerbTypeMap() // { RelatedTo: 'relatedTo', Contains: 'contains', ... } -``` - -These utility functions make it easy to: - -- Get a complete list of available noun and verb types -- Validate user input against valid types -- Create dynamic UI components that display or select from available types -- Map between type keys and their string values - -## Command Line Interface - -Brainy includes a powerful CLI for managing your data. The CLI is available as a separate package -`@soulcraft/brainy-cli` to reduce the bundle size of the main package. - -### Installing and Using the CLI - -```bash -# Install the CLI globally -npm install -g @soulcraft/brainy-cli - -# Initialize a database -brainy init - -# Add some data -brainy add "Cats are independent pets" '{"noun":"Thing","category":"animal"}' -brainy add "Dogs are loyal companions" '{"noun":"Thing","category":"animal"}' - -# Search for similar items -brainy search "feline pets" 5 - -# Add relationships between items -brainy addVerb RelatedTo '{"description":"Both are pets"}' - -# Visualize the graph structure -brainy visualize -brainy visualize --root --depth 3 -``` - -### Using the CLI in Your Code - -The CLI functionality is available as a separate package `@soulcraft/brainy-cli`. If you need CLI functionality in your -application, install the CLI package: - -```bash -npm install @soulcraft/brainy-cli -``` - -Then you can use the CLI commands programmatically or through the command line interface. - -### Available Commands - -#### Basic Database Operations: - -- `init` - Initialize a new database -- `add [metadata]` - Add a new noun with text and optional metadata -- `search [limit]` - Search for nouns similar to the query -- `get ` - Get a noun by ID -- `delete ` - Delete a noun by ID -- `addVerb [metadata]` - Add a relationship -- `getVerbs ` - Get all relationships for a noun -- `status` - Show database status -- `clear` - Clear all data from the database -- `generate-random-graph` - Generate test data -- `visualize` - Visualize the graph structure -- `completion-setup` - Setup shell autocomplete - -#### Pipeline and Augmentation Commands: - -- `list-augmentations` - List all available augmentation types and registered augmentations -- `augmentation-info ` - Get detailed information about a specific augmentation type -- `test-pipeline [text]` - Test the sequential pipeline with sample data - - `-t, --data-type ` - Type of data to process (default: 'text') - - `-m, --mode ` - Execution mode: sequential, parallel, threaded (default: 'sequential') - - `-s, --stop-on-error` - Stop execution if an error occurs - - `-v, --verbose` - Show detailed output -- `stream-test` - Test streaming data through the pipeline (simulated) - - `-c, --count ` - Number of data items to stream (default: 5) - - `-i, --interval ` - Interval between data items in milliseconds (default: 1000) - - `-t, --data-type ` - Type of data to process (default: 'text') - - `-v, --verbose` - Show detailed output +| Environment | Storage | Threading | Auto-Configured | +|-------------|---------|-----------|-----------------| +| Browser | OPFS | Web Workers | โœ… | +| Node.js | FileSystem/S3 | Worker Threads | โœ… | +| Serverless | Memory/S3 | Limited | โœ… | +| Edge Functions | Memory/KV | Limited | โœ… | ## ๐Ÿ“š Documentation -### ๐Ÿš€ [Getting Started](docs/getting-started/) +### Getting Started +- [**Quick Start Guide**](docs/getting-started/) - Get up and running in minutes +- [**Installation**](docs/getting-started/installation.md) - Detailed setup instructions +- [**Environment Setup**](docs/getting-started/environment-setup.md) - Platform-specific configuration -Quick setup guides and first steps with Brainy. +### User Guides +- [**Search and Metadata**](docs/user-guides/) - Advanced search techniques +- [**JSON Document Search**](docs/guides/json-document-search.md) - Field-based searching +- [**Production Migration**](docs/guides/production-migration-guide.md) - Deployment best practices -- **[Installation](docs/getting-started/installation.md)** - Installation and setup -- **[Quick Start](docs/getting-started/quick-start.md)** - Get running in 2 minutes -- **[First Steps](docs/getting-started/first-steps.md)** - Core concepts and features -- **[Environment Setup](docs/getting-started/environment-setup.md)** - Environment-specific configuration +### API Reference +- [**Core API**](docs/api-reference/) - Complete method reference +- [**Configuration Options**](docs/api-reference/configuration.md) - All configuration parameters +- [**Auto-Configuration API**](docs/api-reference/auto-configuration-api.md) - Intelligent setup -### ๐Ÿ“– [User Guides](docs/user-guides/) +### Optimization & Scaling +- [**Large-Scale Optimizations**](docs/optimization-guides/) - Handle millions of vectors +- [**Memory Management**](docs/optimization-guides/memory-optimization.md) - Efficient resource usage +- [**S3 Migration Guide**](docs/optimization-guides/s3-migration-guide.md) - Cloud storage setup -Comprehensive guides for using Brainy effectively. +### Examples & Patterns +- [**Code Examples**](docs/examples/) - Real-world usage patterns +- [**Integrations**](docs/examples/integrations.md) - Third-party services +- [**Performance Patterns**](docs/examples/performance.md) - Optimization techniques -- **[Search and Metadata](docs/user-guides/SEARCH_AND_METADATA_GUIDE.md)** - Advanced search techniques -- **[Write-Only Mode](docs/user-guides/WRITEONLY_MODE_IMPLEMENTATION.md)** - High-throughput data loading -- **[JSON Document Search](docs/guides/json-document-search.md)** - Search within JSON fields -- **[Production Migration](docs/guides/production-migration-guide.md)** - Deployment best practices +### Technical Documentation +- [**Architecture Overview**](docs/technical/) - System design and internals +- [**Testing Guide**](docs/technical/TESTING.md) - Testing strategies +- [**Statistics & Monitoring**](docs/technical/STATISTICS.md) - Performance tracking -### โšก [Optimization Guides](docs/optimization-guides/) +## ๐Ÿค Contributing -Transform Brainy from prototype to production-ready system. +We welcome contributions! Please see: +- [Contributing Guidelines](CONTRIBUTING.md) +- [Developer Documentation](docs/development/DEVELOPERS.md) +- [Code of Conduct](CODE_OF_CONDUCT.md) -- **[Large-Scale Optimizations](docs/optimization-guides/large-scale-optimizations.md)** - Complete v0.36.0 optimization - system -- **[Auto-Configuration](docs/optimization-guides/auto-configuration.md)** - Intelligent environment detection -- **[Memory Optimization](docs/optimization-guides/memory-optimization.md)** - Advanced memory management -- **[Storage Optimization](docs/optimization-guides/storage-optimization.md)** - S3 and storage optimization +## ๐Ÿ“„ License -### ๐Ÿ”ง [API Reference](docs/api-reference/) +[MIT](LICENSE) -Complete API documentation and method references. +## ๐Ÿ”— Related Projects -- **[Core API](docs/api-reference/core-api.md)** - Main BrainyData class methods -- **[Vector Operations](docs/api-reference/vector-operations.md)** - Vector storage and search -- **[Configuration](docs/api-reference/configuration.md)** - System configuration -- **[Auto-Configuration API](docs/api-reference/auto-configuration-api.md)** - Intelligent configuration - -### ๐Ÿ’ก [Examples](docs/examples/) - -Practical code examples and real-world applications. - -- **[Basic Usage](docs/examples/basic-usage.md)** - Simple examples to get started -- **[Advanced Patterns](docs/examples/advanced-patterns.md)** - Complex use cases -- **[Integrations](docs/examples/integrations.md)** - Third-party service integrations -- **[Performance Examples](docs/examples/performance.md)** - Optimization and scaling - -### ๐Ÿ”ฌ Technical Documentation - -- **[Testing Guide](docs/technical/TESTING.md)** - Testing strategies and best practices -- **[Statistics Guide](STATISTICS.md)** - Database statistics and monitoring -- **[Technical Guides](TECHNICAL_GUIDES.md)** - Advanced technical topics - -## API Reference - -### Database Management - -```typescript -// Initialize the database -await db.init() - -// Clear all data -await db.clear() - -// Get database status -const status = await db.status() - -// Backup all data from the database -const backupData = await db.backup() - -// Restore data into the database -const restoreResult = await db.restore(backupData, { clearExisting: true }) -``` - -### Database Statistics - -Brainy provides a way to get statistics about the current state of the database. For detailed information about the -statistics system, including implementation details, scalability improvements, and usage examples, see -our [Statistics Guide](STATISTICS.md). - -```typescript -import { BrainyData, getStatistics } from '@soulcraft/brainy' - -// Create and initialize the database -const db = new BrainyData() -await db.init() - -// Get statistics using the instance method -const stats = await db.getStatistics() -console.log(stats) -// Output: { nounCount: 0, verbCount: 0, metadataCount: 0, hnswIndexSize: 0, serviceBreakdown: {...} } -``` - -### Working with Nouns (Entities) - -```typescript -// Add a noun (automatically vectorized) -const id = await db.add(textOrVector, { - noun: NounType.Thing, - // other metadata... -}) - -// Add multiple nouns in parallel (with multithreading and batch embedding) -const ids = await db.addBatch([ - { - vectorOrData: "First item to add", - metadata: { noun: NounType.Thing, category: 'example' } - }, - { - vectorOrData: "Second item to add", - metadata: { noun: NounType.Thing, category: 'example' } - }, - // More items... -], { - forceEmbed: false, - concurrency: 4, // Control the level of parallelism (default: 4) - batchSize: 50 // Control the number of items to process in a single batch (default: 50) -}) - -// Retrieve a noun -const noun = await db.get(id) - -// Update noun metadata -await db.updateMetadata(id, { - noun: NounType.Thing, - // updated metadata... -}) - -// Delete a noun -await db.delete(id) - -// Search for similar nouns -const results = await db.search(vectorOrText, numResults) -const textResults = await db.searchText("query text", numResults) - -// Search by noun type -const thingNouns = await db.searchByNounTypes([NounType.Thing], numResults) - -// Search within specific fields of JSON documents -const fieldResults = await db.search("Acme Corporation", 10, { - searchField: "company" -}) - -// Search using standard field names across different services -const titleResults = await db.searchByStandardField("title", "climate change", 10) -const authorResults = await db.searchByStandardField("author", "johndoe", 10, { - services: ["github", "reddit"] -}) -``` - -### Field Standardization and Service Tracking - -Brainy automatically tracks field names from JSON documents and associates them with the service that inserted the data. -This enables powerful cross-service search capabilities: - -```typescript -// Get all available field names organized by service -const fieldNames = await db.getAvailableFieldNames() -// Example output: { "github": ["repository.name", "issue.title"], "reddit": ["title", "selftext"] } - -// Get standard field mappings -const standardMappings = await db.getStandardFieldMappings() -// Example output: { "title": { "github": ["repository.name"], "reddit": ["title"] } } -``` - -When adding data, specify the service name to ensure proper field tracking: - -```typescript -// Add data with service name -await db.add(jsonData, metadata, { service: "github" }) -``` - -### Working with Verbs (Relationships) - -```typescript -// Add a relationship between nouns -await db.addVerb(sourceId, targetId, { - verb: VerbType.RelatedTo, - // other metadata... -}) - -// Add a relationship with auto-creation of missing nouns -// This is useful when the target noun might not exist yet -await db.addVerb(sourceId, targetId, { - verb: VerbType.RelatedTo, - // Enable auto-creation of missing nouns - autoCreateMissingNouns: true, - // Optional metadata for auto-created nouns - missingNounMetadata: { - noun: NounType.Concept, - description: 'Auto-created noun' - } -}) - -// Get all relationships -const verbs = await db.getAllVerbs() - -// Get relationships by source noun -const outgoingVerbs = await db.getVerbsBySource(sourceId) - -// Get relationships by target noun -const incomingVerbs = await db.getVerbsByTarget(targetId) - -// Get relationships by type -const containsVerbs = await db.getVerbsByType(VerbType.Contains) - -// Get a specific relationship -const verb = await db.getVerb(verbId) - -// Delete a relationship -await db.deleteVerb(verbId) -``` - -## Advanced Configuration - -### Database Modes - -Brainy supports special operational modes that restrict certain operations: - -```typescript -import { BrainyData } from '@soulcraft/brainy' - -// Create and initialize the database -const db = new BrainyData() -await db.init() - -// Set the database to read-only mode (prevents write operations) -db.setReadOnly(true) - -// Check if the database is in read-only mode -const isReadOnly = db.isReadOnly() // Returns true - -// Set the database to write-only mode (prevents search operations) -db.setWriteOnly(true) - -// Check if the database is in write-only mode -const isWriteOnly = db.isWriteOnly() // Returns true - -// Reset to normal mode (allows both read and write operations) -db.setReadOnly(false) -db.setWriteOnly(false) -``` - -- **Read-Only Mode**: When enabled, prevents all write operations (add, update, delete). Useful for deployment scenarios - where you want to prevent modifications to the database. -- **Write-Only Mode**: When enabled, prevents all search operations. Useful for initial data loading or when you want to - optimize for write performance. - -### Embedding - -```typescript -import { - BrainyData, - createTensorFlowEmbeddingFunction, - createThreadedEmbeddingFunction -} from '@soulcraft/brainy' - -// Use the standard TensorFlow Universal Sentence Encoder embedding function -const db = new BrainyData({ - embeddingFunction: createTensorFlowEmbeddingFunction() -}) -await db.init() - -// Or use the threaded embedding function for better performance -const threadedDb = new BrainyData({ - embeddingFunction: createThreadedEmbeddingFunction() -}) -await threadedDb.init() - -// Directly embed text to vectors -const vector = await db.embed("Some text to convert to a vector") - -// Calculate similarity between two texts or vectors -const similarity = await db.calculateSimilarity( - "Cats are furry pets", - "Felines make good companions" -) -console.log(`Similarity score: ${similarity}`) // Higher value means more similar - -// Calculate similarity with custom options -const vectorA = await db.embed("First text") -const vectorB = await db.embed("Second text") -const customSimilarity = await db.calculateSimilarity( - vectorA, // Can use pre-computed vectors - vectorB, - { - forceEmbed: false, // Skip embedding if inputs are already vectors - distanceFunction: cosineDistance // Optional custom distance function - } -) -``` - -The threaded embedding function runs in a separate thread (Web Worker in browsers, Worker Thread in Node.js) to improve -performance, especially for embedding operations. It uses GPU acceleration when available (via WebGL in browsers) and -falls back to CPU processing for compatibility. Universal Sentence Encoder is always used for embeddings. The -implementation includes worker reuse and model caching for optimal performance. - -### Performance Tuning - -Brainy includes comprehensive performance optimizations that work across all environments (browser, CLI, Node.js, -container, server): - -#### GPU and CPU Optimization - -Brainy uses GPU and CPU optimization for compute-intensive operations: - -1. **GPU-Accelerated Embeddings**: Generate text embeddings using TensorFlow.js with WebGL backend when available -2. **Automatic Fallback**: Falls back to CPU backend when GPU is not available -3. **Optimized Distance Calculations**: Perform vector similarity calculations with optimized algorithms -4. **Cross-Environment Support**: Works consistently across browsers and Node.js environments -5. **Memory Management**: Properly disposes of tensors to prevent memory leaks - -#### Multithreading Support - -Brainy includes comprehensive multithreading support to improve performance across all environments: - -1. **Parallel Batch Processing**: Add multiple items concurrently with controlled parallelism -2. **Multithreaded Vector Search**: Perform distance calculations in parallel for faster search operations -3. **Threaded Embedding Generation**: Generate embeddings in separate threads to avoid blocking the main thread -4. **Worker Reuse**: Maintains a pool of workers to avoid the overhead of creating and terminating workers -5. **Model Caching**: Initializes the embedding model once per worker and reuses it for multiple operations -6. **Batch Embedding**: Processes multiple items in a single embedding operation for better performance -7. **Automatic Environment Detection**: Adapts to browser (Web Workers) and Node.js (Worker Threads) environments - -```typescript -import { BrainyData, euclideanDistance } from '@soulcraft/brainy' - -// Configure with custom options -const db = new BrainyData({ - // Use Euclidean distance instead of default cosine distance - distanceFunction: euclideanDistance, - - // HNSW index configuration for search performance - hnsw: { - M: 16, // Max connections per noun - efConstruction: 200, // Construction candidate list size - efSearch: 50, // Search candidate list size - }, - - // Performance optimization options - performance: { - useParallelization: true, // Enable multithreaded search operations - }, - - // Noun and Verb type validation - typeValidation: { - enforceNounTypes: true, // Validate noun types against NounType enum - enforceVerbTypes: true, // Validate verb types against VerbType enum - }, - - // Storage configuration - storage: { - requestPersistentStorage: true, - // Example configuration for cloud storage (replace with your own values): - // s3Storage: { - // bucketName: 'your-s3-bucket-name', - // region: 'your-aws-region' - // // Credentials should be provided via environment variables - // // AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY - // } - } -}) -``` - -### Optimized HNSW for Large Datasets - -Brainy includes an optimized HNSW index implementation for large datasets that may not fit entirely in memory, using a -hybrid approach: - -1. **Product Quantization** - Reduces vector dimensionality while preserving similarity relationships -2. **Disk-Based Storage** - Offloads vectors to disk when memory usage exceeds a threshold -3. **Memory-Efficient Indexing** - Optimizes memory usage for large-scale vector collections - -```typescript -import { BrainyData } from '@soulcraft/brainy' - -// Configure with optimized HNSW index for large datasets -const db = new BrainyData({ - hnswOptimized: { - // Standard HNSW parameters - M: 16, // Max connections per noun - efConstruction: 200, // Construction candidate list size - efSearch: 50, // Search candidate list size - - // Memory threshold in bytes - when exceeded, will use disk-based approach - memoryThreshold: 1024 * 1024 * 1024, // 1GB default threshold - - // Product quantization settings for dimensionality reduction - productQuantization: { - enabled: true, // Enable product quantization - numSubvectors: 16, // Number of subvectors to split the vector into - numCentroids: 256 // Number of centroids per subvector - }, - - // Whether to use disk-based storage for the index - useDiskBasedIndex: true // Enable disk-based storage - }, - - // Storage configuration (required for disk-based index) - storage: { - requestPersistentStorage: true - } -}) - -// The optimized index automatically adapts based on dataset size: -// 1. For small datasets: Uses standard in-memory approach -// 2. For medium datasets: Applies product quantization to reduce memory usage -// 3. For large datasets: Combines product quantization with disk-based storage - -// Check status to see memory usage and optimization details -const status = await db.status() -console.log(status.details.index) -``` - -## Distance Functions - -Brainy provides several distance functions for vector similarity calculations: - -- `cosineDistance` (default): Measures the cosine of the angle between vectors (1 - cosine similarity) -- `euclideanDistance`: Measures the straight-line distance between vectors -- `manhattanDistance`: Measures the sum of absolute differences between vector components -- `dotProductDistance`: Measures the negative dot product between vectors - -All distance functions are optimized for performance and automatically use the most efficient implementation based on -the dataset size and available resources. For large datasets and high-dimensional vectors, Brainy uses batch processing -and multithreading when available to improve performance. - -## Backup and Restore - -Brainy provides backup and restore capabilities that allow you to: - -- Back up your data -- Transfer data between Brainy instances -- Restore existing data into Brainy for vectorization and indexing -- Backup data for analysis or visualization in other tools - -### Backing Up Data - -```typescript -// Backup all data from the database -const backupData = await db.backup() - -// The backup data includes: -// - All nouns (entities) with their vectors and metadata -// - All verbs (relationships) between nouns -// - Noun types and verb types -// - HNSW index data for fast similarity search -// - Version information - -// Save the backup data to a file (Node.js environment) -import fs from 'fs' - -fs.writeFileSync('brainy-backup.json', JSON.stringify(backupData, null, 2)) -``` - -### Restoring Data - -Brainy's restore functionality can handle: - -1. Complete backups with vectors and index data -2. Sparse data without vectors (vectors will be created during restore) -3. Data without HNSW index (index will be reconstructed if needed) - -```typescript -// Restore data with all options -const restoreResult = await db.restore(backupData, { - clearExisting: true // Whether to clear existing data before restore -}) - -// Import sparse data (without vectors) -// Vectors will be automatically created using the embedding function -const sparseData = { - nouns: [ - { - id: '123', - // No vector field - will be created during import - metadata: { - noun: 'Thing', - text: 'This text will be used to generate a vector' - } - } - ], - verbs: [], - version: '1.0.0' -} - -const sparseImportResult = await db.importSparseData(sparseData) -``` - -### CLI Backup/Restore - -```bash -# Backup data to a file -brainy backup --output brainy-backup.json - -# Restore data from a file -brainy restore --input brainy-backup.json --clear-existing - -# Import sparse data (without vectors) -brainy import-sparse --input sparse-data.json -``` - -## Embedding - -Brainy uses the following embedding approach: - -- TensorFlow Universal Sentence Encoder (high-quality text embeddings) -- GPU acceleration when available (via WebGL in browsers) -- Batch embedding for processing multiple items efficiently -- Worker reuse and model caching for optimal performance -- Custom embedding functions can be plugged in for specialized domains - -## Extensions - -Brainy includes an augmentation system for extending functionality: - -- **Memory Augmentations**: Different storage backends -- **Sense Augmentations**: Process raw data -- **Cognition Augmentations**: Reasoning and inference -- **Dialog Augmentations**: Text processing and interaction -- **Perception Augmentations**: Data interpretation and visualization -- **Activation Augmentations**: Trigger actions - -### Simplified Augmentation System - -Brainy provides a simplified factory system for creating, importing, and executing augmentations with minimal -boilerplate: - -```typescript -import { - createMemoryAugmentation, - createConduitAugmentation, - createSenseAugmentation, - addWebSocketSupport, - executeStreamlined, - processStaticData, - processStreamingData, - createPipeline -} from '@soulcraft/brainy' - -// Create a memory augmentation with minimal code -const memoryAug = createMemoryAugmentation({ - name: 'simple-memory', - description: 'A simple in-memory storage augmentation', - autoRegister: true, - autoInitialize: true, - - // Implement only the methods you need - storeData: async (key, data) => { - // Your implementation here - return { - success: true, - data: true - } - }, - - retrieveData: async (key) => { - // Your implementation here - return { - success: true, - data: { example: 'data', key } - } - } -}) - -// Add WebSocket support to any augmentation -const wsAugmentation = addWebSocketSupport(memoryAug, { - connectWebSocket: async (url) => { - // Your implementation here - return { - connectionId: 'ws-1', - url, - status: 'connected' - } - } -}) - -// Process static data through a pipeline -const result = await processStaticData( - 'Input data', - [ - { - augmentation: senseAug, - method: 'processRawData', - transformArgs: (data) => [data, 'text'] - }, - { - augmentation: memoryAug, - method: 'storeData', - transformArgs: (data) => ['processed-data', data] - } - ] -) - -// Create a reusable pipeline -const pipeline = createPipeline([ - { - augmentation: senseAug, - method: 'processRawData', - transformArgs: (data) => [data, 'text'] - }, - { - augmentation: memoryAug, - method: 'storeData', - transformArgs: (data) => ['processed-data', data] - } -]) - -// Use the pipeline -const result = await pipeline('New input data') - -// Dynamically load augmentations at runtime -const loadedAugmentations = await loadAugmentationModule( - import('./my-augmentations.js'), - { - autoRegister: true, - autoInitialize: true - } -) -``` - -The simplified augmentation system provides: - -1. **Factory Functions** - Create augmentations with minimal boilerplate -2. **WebSocket Support** - Add WebSocket capabilities to any augmentation -3. **Streamlined Pipeline** - Process data through augmentations more efficiently -4. **Dynamic Loading** - Load augmentations at runtime when needed -5. **Static & Streaming Data** - Handle both static and streaming data with the same API - -#### WebSocket Augmentation Types - -Brainy exports several WebSocket augmentation types that can be used by augmentation creators to add WebSocket -capabilities to their augmentations: - -```typescript -import { - // Base WebSocket support interface - IWebSocketSupport, - - // Combined WebSocket augmentation types - IWebSocketSenseAugmentation, - IWebSocketConduitAugmentation, - IWebSocketCognitionAugmentation, - IWebSocketMemoryAugmentation, - IWebSocketPerceptionAugmentation, - IWebSocketDialogAugmentation, - IWebSocketActivationAugmentation, - - // Function to add WebSocket support to any augmentation - addWebSocketSupport -} from '@soulcraft/brainy' - -// Example: Creating a typed WebSocket-enabled sense augmentation -const mySenseAug = createSenseAugmentation({ - name: 'my-sense', - processRawData: async (data, dataType) => { - // Implementation - return { - success: true, - data: { nouns: [], verbs: [] } - } - } -}) as IWebSocketSenseAugmentation - -// Add WebSocket support -addWebSocketSupport(mySenseAug, { - connectWebSocket: async (url) => { - // WebSocket implementation - return { - connectionId: 'ws-1', - url, - status: 'connected' - } - }, - sendWebSocketMessage: async (connectionId, data) => { - // Send message implementation - }, - onWebSocketMessage: async (connectionId, callback) => { - // Register callback implementation - }, - offWebSocketMessage: async (connectionId, callback) => { - // Remove callback implementation - }, - closeWebSocket: async (connectionId, code, reason) => { - // Close connection implementation - } -}) - -// Now mySenseAug has both sense augmentation methods and WebSocket methods -await mySenseAug.processRawData('data', 'text') -await mySenseAug.connectWebSocket('wss://example.com') -``` - -These WebSocket augmentation types combine the base augmentation interfaces with the `IWebSocketSupport` interface, -providing type safety and autocompletion for augmentations with WebSocket capabilities. - -### Model Control Protocol (MCP) - -Brainy includes a Model Control Protocol (MCP) implementation that allows external models to access Brainy data and use -the augmentation pipeline as tools: - -- **BrainyMCPAdapter**: Provides access to Brainy data through MCP -- **MCPAugmentationToolset**: Exposes the augmentation pipeline as tools -- **BrainyMCPService**: Integrates the adapter and toolset, providing WebSocket and REST server implementations - -Environment compatibility: - -- **BrainyMCPAdapter** and **MCPAugmentationToolset** can run in any environment (browser, Node.js, server) -- **BrainyMCPService** core functionality works in any environment - -For detailed documentation and usage examples, see the [MCP documentation](src/mcp/README.md). - -## Cross-Environment Compatibility - -Brainy is designed to run seamlessly in any environment, from browsers to Node.js to serverless functions and -containers. All Brainy data, functions, and augmentations are environment-agnostic, allowing you to use the same code -everywhere. - -### Environment Detection - -Brainy automatically detects the environment it's running in: - -```typescript -import { environment } from '@soulcraft/brainy' - -// Check which environment we're running in -console.log(`Running in ${ - environment.isBrowser ? 'browser' : - environment.isNode ? 'Node.js' : - 'serverless/unknown' -} environment`) -``` - -### Adaptive Storage - -Storage adapters are automatically selected based on the environment: - -- **Browser**: Uses Origin Private File System (OPFS) when available, falls back to in-memory storage -- **Node.js**: Uses file system storage by default, with options for S3-compatible cloud storage -- **Serverless**: Uses in-memory storage with options for cloud persistence -- **Container**: Automatically detects and uses the appropriate storage based on available capabilities - -### Dynamic Imports - -Brainy uses dynamic imports to load environment-specific dependencies only when needed, keeping the bundle size small -and ensuring compatibility across environments. - -### Browser Support - -Works in all modern browsers: - -- Chrome 86+ -- Edge 86+ -- Opera 72+ -- Chrome for Android 86+ - -For browsers without OPFS support, falls back to in-memory storage. - -## Related Projects - -- **[Cartographer](https://github.com/sodal-project/cartographer)** - A companion project that provides standardized - interfaces for interacting with Brainy - -## Demo - -The repository includes a comprehensive demo that showcases Brainy's main features: - -- `demo/index.html` - A single demo page with animations demonstrating Brainy's features. - - **[Try the live demo](https://soulcraft-research.github.io/brainy/demo/index.html)** - Check out the - interactive demo on - GitHub Pages - - Or run it locally with `npm run demo` (see [demo instructions](demo.md) for details) - - To deploy your own version to GitHub Pages, use the GitHub Actions workflow in - `.github/workflows/deploy-demo.yml`, - which automatically deploys when pushing to the main branch or can be manually triggered - - To use a custom domain (like www.soulcraft.com): - 1. A CNAME file is already included in the demo directory - 2. In your GitHub repository settings, go to Pages > Custom domain and enter your domain - 3. Configure your domain's DNS settings to point to GitHub Pages: - - - Add a CNAME record for www pointing to `.github.io` (e.g., `soulcraft-research.github.io`) - - Or for an apex domain (soulcraft.com), add A records pointing to GitHub Pages IP addresses - -The demo showcases: - -- How Brainy runs in different environments (browser, Node.js, server, cloud) -- How the noun-verb data model works -- How HNSW search works - -## Syncing Brainy Instances - -You can use the conduit augmentations to sync Brainy instances: - -- **WebSocket iConduit**: For syncing between browsers and servers, or between servers. WebSockets cannot be used for - direct browser-to-browser communication without a server in the middle. -- **WebRTC iConduit**: For direct peer-to-peer syncing between browsers. This is the recommended approach for - browser-to-browser communication. - -#### WebSocket Sync Example - -```typescript -import { - BrainyData, - pipeline, - createConduitAugmentation -} from '@soulcraft/brainy' - -// Create and initialize the database -const db = new BrainyData() -await db.init() - -// Create a WebSocket conduit augmentation -const wsConduit = await createConduitAugmentation('websocket', 'my-websocket-sync') - -// Register the augmentation with the pipeline -pipeline.register(wsConduit) - -// Connect to another Brainy instance (server or browser) -// Replace the example URL below with your actual WebSocket server URL -const connectionResult = await pipeline.executeConduitPipeline( - 'establishConnection', - ['wss://example-websocket-server.com/brainy-sync', { protocols: 'brainy-sync' }] -) - -if (connectionResult[0] && (await connectionResult[0]).success) { - const connection = (await connectionResult[0]).data - - // Read data from the remote instance - const readResult = await pipeline.executeConduitPipeline( - 'readData', - [{ connectionId: connection.connectionId, query: { type: 'getAllNouns' } }] - ) - - // Process and add the received data to the local instance - if (readResult[0] && (await readResult[0]).success) { - const remoteNouns = (await readResult[0]).data - for (const noun of remoteNouns) { - await db.add(noun.vector, noun.metadata) - } - } - - // Set up real-time sync by monitoring the stream - await wsConduit.monitorStream(connection.connectionId, async (data) => { - // Handle incoming data (e.g., new nouns, verbs, updates) - if (data.type === 'newNoun') { - await db.add(data.vector, data.metadata) - } else if (data.type === 'newVerb') { - await db.addVerb(data.sourceId, data.targetId, data.vector, data.options) - } - }) -} -``` - -#### WebRTC Peer-to-Peer Sync Example - -```typescript -import { - BrainyData, - pipeline, - createConduitAugmentation -} from '@soulcraft/brainy' - -// Create and initialize the database -const db = new BrainyData() -await db.init() - -// Create a WebRTC conduit augmentation -const webrtcConduit = await createConduitAugmentation('webrtc', 'my-webrtc-sync') - -// Register the augmentation with the pipeline -pipeline.register(webrtcConduit) - -// Connect to a peer using a signaling server -// Replace the example values below with your actual configuration -const connectionResult = await pipeline.executeConduitPipeline( - 'establishConnection', - [ - 'peer-id-to-connect-to', // Replace with actual peer ID - { - signalServerUrl: 'wss://example-signal-server.com', // Replace with your signal server - localPeerId: 'my-local-peer-id', // Replace with your local peer ID - iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] // Public STUN server - } - ] -) - -if (connectionResult[0] && (await connectionResult[0]).success) { - const connection = (await connectionResult[0]).data - - // Set up real-time sync by monitoring the stream - await webrtcConduit.monitorStream(connection.connectionId, async (data) => { - // Handle incoming data (e.g., new nouns, verbs, updates) - if (data.type === 'newNoun') { - await db.add(data.vector, data.metadata) - } else if (data.type === 'newVerb') { - await db.addVerb(data.sourceId, data.targetId, data.vector, data.options) - } - }) - - // When adding new data locally, also send to the peer - const nounId = await db.add("New data to sync", { noun: "Thing" }) - - // Send the new noun to the peer - await pipeline.executeConduitPipeline( - 'writeData', - [ - { - connectionId: connection.connectionId, - data: { - type: 'newNoun', - id: nounId, - vector: (await db.get(nounId)).vector, - metadata: (await db.get(nounId)).metadata - } - } - ] - ) -} -``` - -#### Browser-Server Search Example - -Brainy supports searching a server-hosted instance from a browser, storing results locally, and performing further -searches against the local instance: - -```typescript -import { BrainyData } from '@soulcraft/brainy' - -// Create and initialize the database with remote server configuration -// Replace the example URL below with your actual Brainy server URL -const db = new BrainyData({ - remoteServer: { - url: 'wss://example-brainy-server.com/ws', // Replace with your server URL - protocols: 'brainy-sync', - autoConnect: true // Connect automatically during initialization - } -}) -await db.init() - -// Or connect manually after initialization -if (!db.isConnectedToRemoteServer()) { - // Replace the example URL below with your actual Brainy server URL - await db.connectToRemoteServer('wss://example-brainy-server.com/ws', 'brainy-sync') -} - -// Search the remote server (results are stored locally) -const remoteResults = await db.searchText('machine learning', 5, { searchMode: 'remote' }) - -// Search the local database (includes previously stored results) -const localResults = await db.searchText('machine learning', 5, { searchMode: 'local' }) - -// Perform a combined search (local first, then remote if needed) -const combinedResults = await db.searchText('neural networks', 5, { searchMode: 'combined' }) - -// Add data to both local and remote instances -const id = await db.addToBoth('Deep learning is a subset of machine learning', { - noun: 'Concept', - category: 'AI', - tags: ['deep learning', 'neural networks'] -}) - -// Clean up when done (this also cleans up worker pools) -await db.shutDown() -``` +- [**Cartographer**](https://github.com/sodal-project/cartographer) - Standardized interfaces for Brainy --- -## ๐Ÿ“ˆ Scaling Strategy - -Brainy is designed to handle datasets of various sizes, from small collections to large-scale deployments. For -terabyte-scale data that can't fit entirely in memory, we provide several approaches: - -- **Disk-Based HNSW**: Modified implementations using intelligent caching and partial loading -- **Distributed HNSW**: Sharding and partitioning across multiple machines -- **Hybrid Solutions**: Combining quantization techniques with multi-tier architectures - -For detailed information on how to scale Brainy for large datasets, vector dimension standardization, threading -implementation, storage testing, and other technical topics, see our -comprehensive [Technical Guides](TECHNICAL_GUIDES.md). - -## Recent Changes and Performance Improvements - -### Enhanced Memory Management and Scalability - -Brainy has been significantly improved to handle larger datasets more efficiently: - -- **Pagination Support**: All data retrieval methods now support pagination to avoid loading entire datasets into memory - at once. The deprecated `getAllNouns()` and `getAllVerbs()` methods have been replaced with `getNouns()` and - `getVerbs()` methods that support pagination, filtering, and cursor-based navigation. - -- **Multi-level Caching**: A sophisticated three-level caching strategy has been implemented: - - **Level 1**: Hot cache (most accessed nodes) - RAM (automatically detecting and adjusting in each environment) - - **Level 2**: Warm cache (recent nodes) - OPFS, Filesystem or S3 depending on environment - - **Level 3**: Cold storage (all nodes) - OPFS, Filesystem or S3 depending on environment - -- **Adaptive Memory Usage**: The system automatically detects available memory and adjusts cache sizes accordingly: - - In Node.js: Uses 10% of free memory (minimum 1000 entries) - - In browsers: Scales based on device memory (500 entries per GB, minimum 1000) - -- **Intelligent Cache Eviction**: Implements a Least Recently Used (LRU) policy that evicts the oldest 20% of items when - the cache reaches the configured threshold. - -- **Prefetching Strategy**: Implements batch prefetching to improve performance while avoiding overwhelming system - resources. - -### S3-Compatible Storage Improvements - -- **Enhanced Cloud Storage**: Improved support for S3-compatible storage services including AWS S3, Cloudflare R2, and - others. - -- **Optimized Data Access**: Batch operations and error handling for efficient cloud storage access. - -- **Change Log Management**: Efficient synchronization through change logs to track updates. - -### Data Compatibility - -Yes, you can use existing data indexed from an old version. Brainy includes robust data migration capabilities: - -- **Vector Regeneration**: If vectors are missing in imported data, they will be automatically created using the - embedding function. - -- **HNSW Index Reconstruction**: The system can reconstruct the HNSW index from backup data, ensuring compatibility with - previous versions. - -- **Sparse Data Import**: Support for importing sparse data (without vectors) through the `importSparseData()` method. - -### System Requirements - -#### Default Mode - -- **Memory**: - - Minimum: 512MB RAM - - Recommended: 2GB+ RAM for medium datasets, 8GB+ for large datasets - -- **CPU**: - - Minimum: 2 cores - - Recommended: 4+ cores for better performance with parallel operations - -- **Storage**: - - Minimum: 1GB available storage - - Recommended: Storage space at least 3x the size of your dataset - -#### Read-Only Mode - -Read-only mode prevents all write operations (add, update, delete) and is optimized for search operations. - -- **Memory**: - - Minimum: 256MB RAM - - Recommended: 1GB+ RAM - -- **CPU**: - - Minimum: 1 core - - Recommended: 2+ cores - -- **Storage**: - - Minimum: Storage space equal to the size of your dataset - - Recommended: 2x the size of your dataset for caching - -- **New Feature**: Lazy loading support in read-only mode for improved performance with large datasets. - -#### Write-Only Mode - -Write-only mode prevents all search operations and is optimized for initial data loading or when you want to optimize -for write performance. - -- **Memory**: - - Minimum: 512MB RAM - - Recommended: 2GB+ RAM - -- **CPU**: - - Minimum: 2 cores - - Recommended: 4+ cores for faster data ingestion - -- **Storage**: - - Minimum: Storage space at least 2x the size of your dataset - - Recommended: 4x the size of your dataset for optimal performance - -### Performance Tuning Parameters - -Brainy offers comprehensive configuration options for performance tuning, with enhanced support for large datasets in S3 -or other remote storage. **All configuration is optional** - the system automatically detects the optimal settings based -on your environment, dataset size, and usage patterns. - -#### Intelligent Defaults - -Brainy uses intelligent defaults that automatically adapt to your environment: - -- **Environment Detection**: Automatically detects whether you're running in Node.js, browser, or worker environment -- **Memory-Aware Caching**: Adjusts cache sizes based on available system memory -- **Dataset Size Adaptation**: Tunes parameters based on the size of your dataset -- **Usage Pattern Optimization**: Adjusts to read-heavy vs. write-heavy workloads -- **Storage Type Awareness**: Optimizes for local vs. remote storage (S3, R2, etc.) -- **Operating Mode Specialization**: Special optimizations for read-only and write-only modes - -#### Cache Configuration (Optional) - -You can override any of these automatically tuned parameters if needed: - -- **Hot Cache Size**: Control the maximum number of items to keep in memory. - - For large datasets (>100K items), consider values between 5,000-50,000 depending on available memory. - - In read-only mode, larger values (10,000-100,000) can be used for better performance. - -- **Eviction Threshold**: Set the threshold at which cache eviction begins (default: 0.8 or 80% of max size). - - For write-heavy workloads, lower values (0.6-0.7) may improve performance. - - For read-heavy workloads, higher values (0.8-0.9) are recommended. - -- **Warm Cache TTL**: Set the time-to-live for items in the warm cache (default: 3600000 ms or 1 hour). - - For frequently changing data, shorter TTLs are recommended. - - For relatively static data, longer TTLs improve performance. - -- **Batch Size**: Control the number of items to process in a single batch for operations like prefetching. - - For S3 or remote storage with large datasets, larger values (50-200) significantly improve throughput. - - In read-only mode with remote storage, even larger values (100-300) can be used. - -#### Auto-Tuning (Enabled by Default) - -- **Auto-Tune**: Enable or disable automatic tuning of cache parameters based on usage patterns (default: true). -- **Auto-Tune Interval**: Set how frequently the system adjusts cache parameters (default: 60000 ms or 1 minute). - -#### Read-Only Mode Optimizations (Automatic) - -Read-only mode includes special optimizations for search performance that are automatically applied: - -- **Larger Cache Sizes**: Automatically uses more memory for caching (up to 40% of free memory for large datasets). -- **Aggressive Prefetching**: Loads more data in each batch to reduce the number of storage requests. -- **Prefetch Strategy**: Defaults to 'aggressive' prefetching strategy in read-only mode. - -#### Example Configuration for Large S3 Datasets - -```javascript -const brainy = new BrainyData({ - readOnly: true, - lazyLoadInReadOnlyMode: true, - storage: { - type: 's3', - s3Storage: { - bucketName: 'your-bucket', - accessKeyId: 'your-access-key', - secretAccessKey: 'your-secret-key', - region: 'your-region' - } - }, - cache: { - hotCacheMaxSize: 20000, - hotCacheEvictionThreshold: 0.85, - batchSize: 100, - readOnlyMode: { - hotCacheMaxSize: 50000, - batchSize: 200, - prefetchStrategy: 'aggressive' - } - } -}); -``` - -These configuration options make Brainy more efficient, scalable, and adaptable to different environments and usage -patterns, especially for large datasets in cloud storage. - -## Testing - -Brainy uses Vitest for testing. For detailed information about testing in Brainy, including test configuration, scripts, -reporting tools, and best practices, see our [Testing Guide](docs/technical/TESTING.md). - -Here are some common test commands: - -```bash -# Run all tests -npm test - -# Run tests with comprehensive reporting -npm run test:report - -# Run tests with coverage -npm run test:coverage -``` - -## Contributing - -For detailed contribution guidelines, please see [CONTRIBUTING.md](CONTRIBUTING.md). - -For developer documentation, including building, testing, and publishing instructions, please -see [DEVELOPERS.md](DEVELOPERS.md). - -We have a [Code of Conduct](CODE_OF_CONDUCT.md) that all contributors are expected to follow. - -### Commit Message Format - -For best results with automatic changelog generation, follow -the [Conventional Commits](https://www.conventionalcommits.org/) specification for your commit messages: - -``` -AI Template for automated commit messages: - -Use Conventional Commit format -Specify the changes in a structured format -Add information about the purpose of the commit -``` - -``` -(): - -[optional body] - -[optional footer(s)] -``` - -Where `` is one of: - -- `feat`: A new feature (maps to **Added** section) -- `fix`: A bug fix (maps to **Fixed** section) -- `chore`: Regular maintenance tasks (maps to **Changed** section) -- `docs`: Documentation changes (maps to **Documentation** section) -- `refactor`: Code changes that neither fix bugs nor add features (maps to **Changed** section) -- `perf`: Performance improvements (maps to **Changed** section) - -### Manual Release Process - -If you need more control over the release process, you can use the individual commands: - -```bash -# Update version and generate changelog -npm run _release:patch # or _release:minor, _release:major - -# Create GitHub release -npm run _github-release - -# Publish to NPM -npm publish -``` - -## License - -[MIT](LICENSE) +
+Ready to build something amazing? Get started with Brainy today! +
diff --git a/docs/tools/claude-commit/README.md b/docs/tools/claude-commit/README.md new file mode 100644 index 00000000..06f5ef05 --- /dev/null +++ b/docs/tools/claude-commit/README.md @@ -0,0 +1,128 @@ +# Claude Commit - AI-Powered Git Commits + +Automatically generate Conventional Commit messages using Claude AI by analyzing your git diff. + +## ๐Ÿš€ Quick Setup + +### One-Line Install (Simplest) + +On any computer, just run: + +```bash +# Download and run the setup script +curl -sSL https://raw.githubusercontent.com/soulcraft-research/brainy/main/docs/tools/claude-commit/setup.sh | bash +``` + +### Manual Install + +If you have the brainy repo cloned: + +```bash +cd ~/Projects/brainy/docs/tools/claude-commit +./setup.sh +``` + +### Using Dotfiles (For Multiple Machines) + +1. **First time:** Create your dotfiles repository + ```bash + # Copy the dotfiles folder from brainy to a new repo + cp -r ~/Projects/brainy/docs/tools/claude-commit/dotfiles ~/Projects/my-dotfiles + cd ~/Projects/my-dotfiles + git init + git add . + git commit -m "feat: add claude-commit dotfiles" + # Create a GitHub repo and push + ``` + +2. **On any new computer:** + ```bash + # One-line install from your dotfiles + curl -sSL https://raw.githubusercontent.com/yourusername/my-dotfiles/main/install.sh | bash + ``` + +## ๐Ÿ“ What Gets Installed + +- `~/.local/bin/claude-commit` - The main script +- Git aliases: `git cc` and `git smart-commit` +- PATH update (if needed) + +## ๐Ÿ”„ Keeping Computers in Sync + +### Manual Sync + +On any computer, update to latest version: + +```bash +# From brainy repo +curl -sSL https://raw.githubusercontent.com/yourusername/brainy/main/dotfiles/bin/claude-commit > ~/.local/bin/claude-commit +chmod +x ~/.local/bin/claude-commit +``` + +### Automatic Sync with Dotfiles + +If you set up a dotfiles repo: + +```bash +cd ~/Projects/my-dotfiles +git pull +./install.sh +``` + +## ๐Ÿ’ป Per-Computer Setup Checklist + +When setting up a new computer: + +- [ ] Install Claude CLI (`claude` command must work) +- [ ] Run the setup script +- [ ] Restart terminal or `source ~/.bashrc` +- [ ] Test with `git cc` in any repo + +## ๐Ÿงช Testing + +After setup, test in any git repository: + +```bash +# Make a change to any file +echo "test" >> README.md + +# Use claude to commit +git cc +``` + +## ๐Ÿ”ง Troubleshooting + +### "command not found: claude" +- Install Claude CLI from https://claude.ai/code + +### "command not found: git cc" +- Run: `source ~/.bashrc` (or `~/.zshrc` for Zsh) +- Verify: `ls -la ~/.local/bin/claude-commit` + +### Script not generating commits +- Check Claude CLI works: `claude --version` +- Ensure you have changes: `git status` + +## ๐Ÿ“ Files in This Setup + +``` +brainy/ +โ”œโ”€โ”€ setup-claude-commit.sh # Standalone installer +โ”œโ”€โ”€ CLAUDE_COMMIT_SETUP.md # This file +โ””โ”€โ”€ dotfiles/ # Portable dotfiles + โ”œโ”€โ”€ README.md # Dotfiles documentation + โ”œโ”€โ”€ install.sh # Master installer + โ”œโ”€โ”€ setup-claude-commit.sh # Claude commit installer + โ””โ”€โ”€ bin/ + โ””โ”€โ”€ claude-commit # The actual script +``` + +## ๐ŸŽฏ Summary + +**For your laptop**, you have three options: + +1. **Quickest:** Run the setup script from brainy repo +2. **Portable:** Create a dotfiles repo and install from there +3. **Manual:** Copy the script directly to `~/.local/bin/` + +All methods give you the same `git cc` command that works in any git repository! \ No newline at end of file diff --git a/docs/tools/claude-commit/dotfiles/README.md b/docs/tools/claude-commit/dotfiles/README.md new file mode 100644 index 00000000..4dcbbdad --- /dev/null +++ b/docs/tools/claude-commit/dotfiles/README.md @@ -0,0 +1,121 @@ +# Dotfiles - Claude Commit Setup + +This repository contains my development environment configuration, including the Claude AI-powered git commit message generator. + +## Quick Setup + +### Option 1: One-Line Install (Recommended) + +```bash +curl -sSL https://raw.githubusercontent.com/yourusername/dotfiles/main/install.sh | bash +``` + +### Option 2: Manual Install + +```bash +# Clone the repository +git clone https://github.com/yourusername/dotfiles.git ~/dotfiles + +# Run the setup script +cd ~/dotfiles +./install.sh +``` + +### Option 3: Just Claude Commit + +If you only want the `git cc` command: + +```bash +# Download and run the setup script +curl -sSL https://raw.githubusercontent.com/yourusername/dotfiles/main/setup-claude-commit.sh | bash +``` + +## What's Included + +### Claude Commit (`git cc`) + +An AI-powered git commit message generator that: +- Analyzes your git diff +- Generates Conventional Commit formatted messages +- Works in any git repository +- No configuration needed + +**Usage:** +```bash +# Make your changes +git cc # Claude generates the commit message +``` + +## Syncing Between Computers + +### First Time Setup (on new computer) + +1. Run the install script (see Quick Setup above) +2. That's it! The `git cc` command is now available + +### Keeping in Sync + +To update to the latest version on any computer: + +```bash +sync-claude-commit +``` + +Or manually: +```bash +curl -sSL https://raw.githubusercontent.com/yourusername/dotfiles/main/bin/claude-commit > ~/.local/bin/claude-commit +chmod +x ~/.local/bin/claude-commit +``` + +## File Structure + +``` +dotfiles/ +โ”œโ”€โ”€ README.md # This file +โ”œโ”€โ”€ install.sh # Main installation script +โ”œโ”€โ”€ setup-claude-commit.sh # Standalone claude-commit installer +โ””โ”€โ”€ bin/ + โ””โ”€โ”€ claude-commit # The actual claude-commit script +``` + +## Requirements + +- Git +- Claude CLI (`claude` command) +- Bash or Zsh + +## Conventional Commit Format + +The tool generates messages following this format: + +``` +(): + + + +