brainy/README.md

282 lines
8.9 KiB
Markdown
Raw Normal View History

<div align="center>
2025-06-24 11:41:30 -07:00
<img src="./brainy.png" alt="Brainy Logo" width="200"/>
<br/><br/>
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Node.js](https://img.shields.io/badge/node-%3E%3D24.4.1-brightgreen.svg)](https://nodejs.org/)
[![TypeScript](https://img.shields.io/badge/TypeScript-5.4.5-blue.svg)](https://www.typescriptlang.org/)
2025-06-24 11:41:30 -07:00
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
**The world's only true Vector + Graph database - unified semantic search and knowledge graphs**
2025-06-24 11:41:30 -07:00
</div>
---
# The Search Problem Every Developer Faces
**"I need to find similar content, explore relationships, AND filter by metadata - but that means juggling 3+ databases"**
feat\!: migrate from TensorFlow.js to Transformers.js with ONNX Runtime BREAKING CHANGE: Complete migration from TensorFlow.js to Transformers.js for embedding generation This is a major architectural change that replaces TensorFlow.js (USE model) with Transformers.js (all-MiniLM-L6-v2) for significantly improved performance and reduced complexity. Key Changes: - Replace TensorFlow.js Universal Sentence Encoder with Transformers.js all-MiniLM-L6-v2 - Reduce model size from 525MB to 87MB (83% reduction) - Reduce embedding dimensions from 512 to 384 (faster distance calculations) - Remove TensorFlow.js Float32Array patching (caused ONNX conflicts) - Implement smart bundled model detection for offline operation - Add explicit model download script for Docker deployments - Remove complex environment variables in favor of simple configuration - Update all distance functions to use optimized pure JavaScript - Remove TensorFlow-specific utilities and type definitions Performance Improvements: - Model loading: 5x faster (87MB vs 525MB) - Memory usage: 75% reduction (~200-400MB vs ~1.5GB) - Distance calculations: Faster pure JS vs GPU overhead for small vectors - Cold start performance: Significantly improved Files Changed: - Updated package.json: New dependencies, simplified scripts - Rewrote src/utils/embedding.ts: Complete Transformers.js implementation - Updated src/utils/distance.ts: Optimized JavaScript distance functions - Simplified src/setup.ts: Removed TensorFlow-specific patching - Simplified src/utils/textEncoding.ts: Only Node.js TextEncoder/Decoder patches - Deleted src/utils/robustModelLoader.ts: TensorFlow-specific loader - Deleted src/types/tensorflowTypes.ts: TensorFlow type definitions - Added scripts/download-models.cjs: Docker-compatible model downloader - Added comprehensive documentation: README.md, OFFLINE_MODELS.md, analysis docs Testing: - All 19 tests passing - Removed test mocking in favor of real implementation testing - Updated test environment for Transformers.js compatibility - Performance tests validate improved efficiency This migration resolves production issues with Docker egress limitations and provides a more robust, performant foundation for vector operations.
2025-08-05 19:29:59 -07:00
**Current Reality**: Pinecone + Neo4j + Elasticsearch + Custom Sync Logic
**Brainy Reality**: One database. One API. All three search types.
feat\!: migrate from TensorFlow.js to Transformers.js with ONNX Runtime BREAKING CHANGE: Complete migration from TensorFlow.js to Transformers.js for embedding generation This is a major architectural change that replaces TensorFlow.js (USE model) with Transformers.js (all-MiniLM-L6-v2) for significantly improved performance and reduced complexity. Key Changes: - Replace TensorFlow.js Universal Sentence Encoder with Transformers.js all-MiniLM-L6-v2 - Reduce model size from 525MB to 87MB (83% reduction) - Reduce embedding dimensions from 512 to 384 (faster distance calculations) - Remove TensorFlow.js Float32Array patching (caused ONNX conflicts) - Implement smart bundled model detection for offline operation - Add explicit model download script for Docker deployments - Remove complex environment variables in favor of simple configuration - Update all distance functions to use optimized pure JavaScript - Remove TensorFlow-specific utilities and type definitions Performance Improvements: - Model loading: 5x faster (87MB vs 525MB) - Memory usage: 75% reduction (~200-400MB vs ~1.5GB) - Distance calculations: Faster pure JS vs GPU overhead for small vectors - Cold start performance: Significantly improved Files Changed: - Updated package.json: New dependencies, simplified scripts - Rewrote src/utils/embedding.ts: Complete Transformers.js implementation - Updated src/utils/distance.ts: Optimized JavaScript distance functions - Simplified src/setup.ts: Removed TensorFlow-specific patching - Simplified src/utils/textEncoding.ts: Only Node.js TextEncoder/Decoder patches - Deleted src/utils/robustModelLoader.ts: TensorFlow-specific loader - Deleted src/types/tensorflowTypes.ts: TensorFlow type definitions - Added scripts/download-models.cjs: Docker-compatible model downloader - Added comprehensive documentation: README.md, OFFLINE_MODELS.md, analysis docs Testing: - All 19 tests passing - Removed test mocking in favor of real implementation testing - Updated test environment for Transformers.js compatibility - Performance tests validate improved efficiency This migration resolves production issues with Docker egress limitations and provides a more robust, performant foundation for vector operations.
2025-08-05 19:29:59 -07:00
## 🔥 The Power of Three-in-One Search
feat\!: migrate from TensorFlow.js to Transformers.js with ONNX Runtime BREAKING CHANGE: Complete migration from TensorFlow.js to Transformers.js for embedding generation This is a major architectural change that replaces TensorFlow.js (USE model) with Transformers.js (all-MiniLM-L6-v2) for significantly improved performance and reduced complexity. Key Changes: - Replace TensorFlow.js Universal Sentence Encoder with Transformers.js all-MiniLM-L6-v2 - Reduce model size from 525MB to 87MB (83% reduction) - Reduce embedding dimensions from 512 to 384 (faster distance calculations) - Remove TensorFlow.js Float32Array patching (caused ONNX conflicts) - Implement smart bundled model detection for offline operation - Add explicit model download script for Docker deployments - Remove complex environment variables in favor of simple configuration - Update all distance functions to use optimized pure JavaScript - Remove TensorFlow-specific utilities and type definitions Performance Improvements: - Model loading: 5x faster (87MB vs 525MB) - Memory usage: 75% reduction (~200-400MB vs ~1.5GB) - Distance calculations: Faster pure JS vs GPU overhead for small vectors - Cold start performance: Significantly improved Files Changed: - Updated package.json: New dependencies, simplified scripts - Rewrote src/utils/embedding.ts: Complete Transformers.js implementation - Updated src/utils/distance.ts: Optimized JavaScript distance functions - Simplified src/setup.ts: Removed TensorFlow-specific patching - Simplified src/utils/textEncoding.ts: Only Node.js TextEncoder/Decoder patches - Deleted src/utils/robustModelLoader.ts: TensorFlow-specific loader - Deleted src/types/tensorflowTypes.ts: TensorFlow type definitions - Added scripts/download-models.cjs: Docker-compatible model downloader - Added comprehensive documentation: README.md, OFFLINE_MODELS.md, analysis docs Testing: - All 19 tests passing - Removed test mocking in favor of real implementation testing - Updated test environment for Transformers.js compatibility - Performance tests validate improved efficiency This migration resolves production issues with Docker egress limitations and provides a more robust, performant foundation for vector operations.
2025-08-05 19:29:59 -07:00
```javascript
// This ONE query does what used to require 3 databases:
const results = await brainy.search("AI startups in healthcare", 10, {
// 🔍 Vector: Semantic similarity
includeVerbs: true,
// 🔗 Graph: Relationship traversal
verbTypes: ["invests_in", "partners_with"],
// 📊 Faceted: MongoDB-style filtering
metadata: {
industry: "healthcare",
funding: { $gte: 1000000 },
stage: { $in: ["Series A", "Series B"] }
}
})
// Returns: Companies similar to your query + their relationships + matching your criteria
```
**Three search paradigms. One lightning-fast query. Zero complexity.**
## 🚀 Install & Go
```bash
npm install @soulcraft/brainy
```
```javascript
import { BrainyData } from '@soulcraft/brainy'
const brainy = new BrainyData() // Auto-detects your environment
await brainy.init() // Auto-configures everything
// Add data with relationships
const openai = await brainy.add("OpenAI", { type: "company", funding: 11000000 })
const gpt4 = await brainy.add("GPT-4", { type: "product", users: 100000000 })
await brainy.relate(openai, gpt4, "develops")
// Search across all dimensions
const results = await brainy.search("AI language models", 5, {
metadata: { funding: { $gte: 10000000 } },
includeVerbs: true
})
2025-08-04 08:56:05 -07:00
```
**That's it. You just built a knowledge graph with semantic search and faceted filtering in 8 lines.**
2025-08-04 08:56:05 -07:00
## 🏆 Why Brainy Wins
- 🧠 **Triple Search Power** - Vector + Graph + Faceted filtering in one query
- 🌍 **Runs Everywhere** - Same code: React, Node.js, serverless, edge
-**Zero Config** - Auto-detects environment, optimizes itself
- 🔄 **Always Synced** - No data consistency nightmares between systems
- 📦 **Truly Offline** - Works without internet after initial setup
- 🔒 **Your Data** - Run locally, in browser, or your own cloud
## 🔮 Coming Soon
- **🤖 MCP Integration** - Let Claude, GPT, and other AI models query your data directly
- **⚡ LLM Generation** - Built-in content generation powered by your knowledge graph
- **🌊 Real-time Sync** - Live updates across distributed instances
## 🎯 Perfect For
**🤖 AI Chat Applications** - ChatGPT-like apps with long-term memory and context
**🔍 Semantic Search** - Find "that thing like a cat but bigger" → returns "tiger"
**🧬 Knowledge Graphs** - Connect everything. Wikipedia meets Neo4j meets magic
**🎯 Recommendation Engines** - "Users who liked this also liked..." but actually good
**📚 Smart Documentation** - Docs that answer questions before you ask them
## 🌍 Works Everywhere - Same Code
```javascript
// This EXACT code works in ALL environments
import { BrainyData } from '@soulcraft/brainy'
const brainy = new BrainyData()
await brainy.init()
// Works in: React, Vue, Angular, Node.js, Deno, Bun,
// Cloudflare Workers, Vercel Edge, AWS Lambda, browsers, anywhere
```
Brainy automatically detects and optimizes for your environment:
| Environment | Storage | Optimization |
|-------------|---------|-------------|
| 🌐 Browser | OPFS | Web Workers, Memory Cache |
| 🟢 Node.js | FileSystem / S3 | Worker Threads, Clustering |
| ⚡ Serverless | S3 / Memory | Cold Start Optimization |
| 🔥 Edge | Memory / KV | Minimal Footprint |
## 🆚 Why Not Just Use...?
### vs. Multiple Databases
**Pinecone + Neo4j + Elasticsearch** - 3 databases, sync nightmares, 3x the cost
**Brainy** - One database, always synced, built-in intelligence
### vs. Traditional Solutions
**PostgreSQL + pgvector + extensions** - Complex setup, performance issues
**Brainy** - Zero config, purpose-built for AI, works everywhere
### vs. Cloud-Only Vector DBs
**Pinecone/Weaviate/Qdrant** - Vendor lock-in, expensive, cloud-only
**Brainy** - Run anywhere, your data stays yours, cost-effective
### vs. Graph Databases with "Vector Features"
**Neo4j + vector plugin** - Bolt-on solution, not native, limited
**Brainy** - Native vector+graph architecture from the ground up
## 📦 Advanced Features
<details>
<summary>🔧 <strong>MongoDB-Style Metadata Filtering</strong></summary>
```javascript
const results = await brainy.search("machine learning", 10, {
metadata: {
// Comparison operators
price: { $gte: 100, $lte: 1000 },
category: { $in: ["AI", "ML", "Data"] },
rating: { $gt: 4.5 },
// Logical operators
$and: [
{ status: "active" },
{ verified: true }
],
// Text operators
description: { $regex: "neural.*network", $options: "i" },
// Array operators
tags: { $includes: "tensorflow" }
}
})
```
**15+ operators supported**: `$gt`, `$gte`, `$lt`, `$lte`, `$eq`, `$ne`, `$in`, `$nin`, `$and`, `$or`, `$not`, `$regex`, `$includes`, `$exists`, `$size`
</details>
2025-08-04 08:56:05 -07:00
<details>
<summary>🔗 <strong>Graph Relationships & Traversal</strong></summary>
feat(distributed): add distributed mode with multi-instance coordination Implements Phase 1 and Phase 2 of distributed enhancements for horizontal scaling: Phase 1 - Zero-Config Distributed Mode: - Add DistributedConfigManager for shared S3 configuration coordination - Implement explicit role configuration (reader/writer/hybrid) for safety - Add instance registration with heartbeat and health monitoring - Create hash-based partitioner for deterministic data distribution Phase 2 - Intelligent Data Management: - Add DomainDetector for automatic data categorization (medical, legal, product, etc.) - Implement domain-aware search filtering for improved relevance - Create role-based operational modes with specific optimizations - Add HealthMonitor for comprehensive metrics tracking Key Features: - Multi-writer support with consistent hash partitioning - Reader instances optimize for 80% cache utilization - Writer instances optimize for batched writes - Automatic domain detection and tagging - Real-time health monitoring across all instances - Cross-platform crypto utilities for browser compatibility Safety Improvements: - Require explicit role configuration (no automatic assignment) - Validate role compatibility on startup - Track instance health and performance metrics Testing: - Add comprehensive test suite for distributed features - All 25 distributed tests passing - Fixed domain filtering in search functionality Documentation: - Update README with distributed mode highlights - Add examples showing reader/writer setup - Document new capabilities and benefits 🤖 Generated with Claude Code https://claude.ai/code Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-04 12:18:58 -07:00
```javascript
// Create entities and relationships
const company = await brainy.add("OpenAI", { type: "company" })
const product = await brainy.add("GPT-4", { type: "product" })
const person = await brainy.add("Sam Altman", { type: "person" })
feat(distributed): add distributed mode with multi-instance coordination Implements Phase 1 and Phase 2 of distributed enhancements for horizontal scaling: Phase 1 - Zero-Config Distributed Mode: - Add DistributedConfigManager for shared S3 configuration coordination - Implement explicit role configuration (reader/writer/hybrid) for safety - Add instance registration with heartbeat and health monitoring - Create hash-based partitioner for deterministic data distribution Phase 2 - Intelligent Data Management: - Add DomainDetector for automatic data categorization (medical, legal, product, etc.) - Implement domain-aware search filtering for improved relevance - Create role-based operational modes with specific optimizations - Add HealthMonitor for comprehensive metrics tracking Key Features: - Multi-writer support with consistent hash partitioning - Reader instances optimize for 80% cache utilization - Writer instances optimize for batched writes - Automatic domain detection and tagging - Real-time health monitoring across all instances - Cross-platform crypto utilities for browser compatibility Safety Improvements: - Require explicit role configuration (no automatic assignment) - Validate role compatibility on startup - Track instance health and performance metrics Testing: - Add comprehensive test suite for distributed features - All 25 distributed tests passing - Fixed domain filtering in search functionality Documentation: - Update README with distributed mode highlights - Add examples showing reader/writer setup - Document new capabilities and benefits 🤖 Generated with Claude Code https://claude.ai/code Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-04 12:18:58 -07:00
// Create meaningful relationships
await brainy.relate(company, product, "develops")
await brainy.relate(person, company, "leads")
await brainy.relate(product, person, "created_by")
feat(distributed): add distributed mode with multi-instance coordination Implements Phase 1 and Phase 2 of distributed enhancements for horizontal scaling: Phase 1 - Zero-Config Distributed Mode: - Add DistributedConfigManager for shared S3 configuration coordination - Implement explicit role configuration (reader/writer/hybrid) for safety - Add instance registration with heartbeat and health monitoring - Create hash-based partitioner for deterministic data distribution Phase 2 - Intelligent Data Management: - Add DomainDetector for automatic data categorization (medical, legal, product, etc.) - Implement domain-aware search filtering for improved relevance - Create role-based operational modes with specific optimizations - Add HealthMonitor for comprehensive metrics tracking Key Features: - Multi-writer support with consistent hash partitioning - Reader instances optimize for 80% cache utilization - Writer instances optimize for batched writes - Automatic domain detection and tagging - Real-time health monitoring across all instances - Cross-platform crypto utilities for browser compatibility Safety Improvements: - Require explicit role configuration (no automatic assignment) - Validate role compatibility on startup - Track instance health and performance metrics Testing: - Add comprehensive test suite for distributed features - All 25 distributed tests passing - Fixed domain filtering in search functionality Documentation: - Update README with distributed mode highlights - Add examples showing reader/writer setup - Document new capabilities and benefits 🤖 Generated with Claude Code https://claude.ai/code Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-04 12:18:58 -07:00
// Traverse relationships
const products = await brainy.getVerbsBySource(company) // What OpenAI develops
const leaders = await brainy.getVerbsByTarget(company) // Who leads OpenAI
const connections = await brainy.findSimilar(product, {
relationType: "develops"
feat(distributed): add distributed mode with multi-instance coordination Implements Phase 1 and Phase 2 of distributed enhancements for horizontal scaling: Phase 1 - Zero-Config Distributed Mode: - Add DistributedConfigManager for shared S3 configuration coordination - Implement explicit role configuration (reader/writer/hybrid) for safety - Add instance registration with heartbeat and health monitoring - Create hash-based partitioner for deterministic data distribution Phase 2 - Intelligent Data Management: - Add DomainDetector for automatic data categorization (medical, legal, product, etc.) - Implement domain-aware search filtering for improved relevance - Create role-based operational modes with specific optimizations - Add HealthMonitor for comprehensive metrics tracking Key Features: - Multi-writer support with consistent hash partitioning - Reader instances optimize for 80% cache utilization - Writer instances optimize for batched writes - Automatic domain detection and tagging - Real-time health monitoring across all instances - Cross-platform crypto utilities for browser compatibility Safety Improvements: - Require explicit role configuration (no automatic assignment) - Validate role compatibility on startup - Track instance health and performance metrics Testing: - Add comprehensive test suite for distributed features - All 25 distributed tests passing - Fixed domain filtering in search functionality Documentation: - Update README with distributed mode highlights - Add examples showing reader/writer setup - Document new capabilities and benefits 🤖 Generated with Claude Code https://claude.ai/code Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-04 12:18:58 -07:00
})
// Search with relationship context
const results = await brainy.search("AI models", 10, {
includeVerbs: true,
verbTypes: ["develops", "created_by"],
searchConnectedNouns: true
feat(distributed): add distributed mode with multi-instance coordination Implements Phase 1 and Phase 2 of distributed enhancements for horizontal scaling: Phase 1 - Zero-Config Distributed Mode: - Add DistributedConfigManager for shared S3 configuration coordination - Implement explicit role configuration (reader/writer/hybrid) for safety - Add instance registration with heartbeat and health monitoring - Create hash-based partitioner for deterministic data distribution Phase 2 - Intelligent Data Management: - Add DomainDetector for automatic data categorization (medical, legal, product, etc.) - Implement domain-aware search filtering for improved relevance - Create role-based operational modes with specific optimizations - Add HealthMonitor for comprehensive metrics tracking Key Features: - Multi-writer support with consistent hash partitioning - Reader instances optimize for 80% cache utilization - Writer instances optimize for batched writes - Automatic domain detection and tagging - Real-time health monitoring across all instances - Cross-platform crypto utilities for browser compatibility Safety Improvements: - Require explicit role configuration (no automatic assignment) - Validate role compatibility on startup - Track instance health and performance metrics Testing: - Add comprehensive test suite for distributed features - All 25 distributed tests passing - Fixed domain filtering in search functionality Documentation: - Update README with distributed mode highlights - Add examples showing reader/writer setup - Document new capabilities and benefits 🤖 Generated with Claude Code https://claude.ai/code Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-04 12:18:58 -07:00
})
```
</details>
<details>
<summary>🌐 <strong>Universal Storage & Deployment</strong></summary>
```javascript
// Development: File system
const dev = new BrainyData({
storage: { fileSystem: { path: './data' } }
})
// Production: S3/R2
const prod = new BrainyData({
storage: { s3Storage: { bucketName: 'my-vectors' } }
})
// Browser: OPFS
const browser = new BrainyData() // Auto-detects OPFS
// Edge: Memory
const edge = new BrainyData({
storage: { memory: {} }
})
// Redis: High performance
const redis = new BrainyData({
storage: { redis: { connectionString: 'redis://...' } }
})
```
2025-08-04 08:56:05 -07:00
**Extend with any storage**: MongoDB, PostgreSQL, DynamoDB - [see storage adapters guide](docs/api-reference/storage-adapters.md)
</details>
<details>
<summary>🐳 <strong>Docker & Cloud Deployment</strong></summary>
```dockerfile
# Production-ready Dockerfile
FROM node:24-slim AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run download-models # Embed models for offline operation
RUN npm run build
FROM node:24-slim AS production
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/models ./models # Offline models included
CMD ["node", "dist/server.js"]
```
2025-08-04 08:56:05 -07:00
Deploy to: Google Cloud Run, AWS Lambda/ECS, Azure Container Instances, Cloudflare Workers, Railway, Render, Vercel, anywhere Docker runs.
2025-08-04 08:56:05 -07:00
</details>
2025-08-04 08:56:05 -07:00
## 📚 Documentation & Resources
2025-08-04 08:56:05 -07:00
- **[🚀 Quick Start Guide](docs/getting-started/)** - Get up and running in minutes
- **[📖 API Reference](docs/api-reference/)** - Complete method documentation
- **[💡 Examples](docs/examples/)** - Real-world usage patterns
- **[⚡ Performance Guide](docs/optimization-guides/)** - Scale to millions of vectors
- **[🔧 Storage Adapters](docs/api-reference/storage-adapters.md)** - Universal storage compatibility
2025-08-04 08:56:05 -07:00
## 🤝 Contributing
2025-08-04 08:56:05 -07:00
We welcome contributions! Please see:
- [Contributing Guidelines](CONTRIBUTING.md)
- [Developer Documentation](docs/development/DEVELOPERS.md)
- [Code of Conduct](CODE_OF_CONDUCT.md)
2025-08-04 08:56:05 -07:00
## 📄 License
2025-08-04 08:56:05 -07:00
[MIT](LICENSE)
2025-08-04 08:56:05 -07:00
---
2025-08-04 08:56:05 -07:00
<div align="center">
<strong>Ready to build the future of search? Get started with Brainy today!</strong>
**[Get Started →](docs/getting-started/) | [View Examples →](docs/examples/) | [Join Community →](https://github.com/soulcraft-research/brainy/discussions)**
</div>