soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat: enforce data/metadata separation, numeric range queries, improved docs

Browse source 
- Store data opaquely in add() and update() instead of spreading object
  properties into top-level metadata. data is for semantic search (HNSW),
  metadata is for structured where-filter queries (MetadataIndex).
- Fix numeric range queries in MetadataIndex — use numeric-aware comparison
  instead of lexicographic string comparison for normalized values.
- Add data field to RelateParams and Relation types for relationship content.
- Add where.type → where.noun alias in metadata-only find() path.
- Rewrite README: focused ~350 lines from 791, quick start first, feature
  showcase with mini-snippets, organized doc links, no version callouts.
- Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs.
- Remove 10 outdated/redundant doc files consolidated into API reference.
- Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods.
- Fix tests asserting data properties appear in metadata (data model violation).
- Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
This commit is contained in:
David Snelling 2026-02-09 12:06:59 -08:00
parent edb5ec4696
commit 0ddc05a5bb
30 changed files with 1356 additions and 7001 deletions
Download patch file Download diff file Expand all files Collapse all files

354
docs/README.md
View file

@ -1,332 +1,136 @@
# Brainy Documentation
Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine.
> The multi-dimensional AI database with Triple Intelligence — vector search, graph traversal, and metadata filtering in one unified API.
## 🆕 What's New in
**Production-Ready Cost Optimization:**
- **Lifecycle Management**: Automatic tier transitions for S3, GCS, Azure (96% cost savings!)
- **Intelligent-Tiering**: S3 Intelligent-Tiering and GCS Autoclass support
- **Batch Operations**: Efficient bulk delete operations (1000 objects per request)
- **Compression**: Gzip compression for FileSystem storage (60-80% space savings)
- **Quota Monitoring**: Real-time OPFS quota tracking for browser apps
- **Tier Management**: Azure Hot/Cool/Archive tier management
**Cost Impact Example (500TB dataset):**
- Before: $138,000/year
- After $5,940/year
- **Savings: $132,060/year (96%)**
## 📊 Implementation Status
- ✅ **Production Ready**: Core features working today
- 🚧 **In Development**: Features coming soon
- 📅 **Roadmap**: See [ROADMAP.md](../ROADMAP.md)
## Quick Links
### Getting Started
- [API Reference](./api/README.md) - Complete API documentation (start here!)
- [Enterprise for Everyone](./guides/enterprise-for-everyone.md) - **No limits, no tiers, everything free**
- [Natural Language Queries](./guides/natural-language.md) - Query with plain English
### Core Concepts
- [Zero Configuration](./architecture/zero-config.md) - **Auto-adapts to any environment**
- [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) - **Revolutionary data model**
- [Triple Intelligence](./architecture/triple-intelligence.md) - Unified query system
- [Architecture Overview](./architecture/overview.md) - System design
### API Documentation
- [API Reference](./api/README.md) - Complete API documentation
- [TypeScript Types](./api/types.md) - Type definitions
### Advanced Topics
- [Augmentations System](./architecture/augmentations.md) - **Enterprise plugins & neural import**
- [Storage Architecture](./architecture/storage.md) - Storage adapter system
- [Performance Tuning](./guides/performance.md) - Optimization guide
- [Migration Guide](../MIGRATION.md) - Upgrading from 1.x
## What is Brainy?
Brainy is a next-generation AI database that combines:
- **Vector Search**: Semantic similarity using HNSW indexing
- **Graph Relationships**: Complex relationship mapping and traversal
- **Field Filtering**: Precise metadata filtering with O(1) lookups
- **Natural Language**: Query in plain English
## Key Features
### 🧠 Triple Intelligence Engine
All three intelligence types (vector, graph, field) work together in every query for optimal results.
### 📝 Noun-Verb Taxonomy
Model your data naturally as entities (nouns) and relationships (verbs) - no complex schemas needed.
### 🌍 Natural Language Queries
Ask questions in plain English and Brainy understands your intent:
```typescript
await brain.find("recent articles about AI with high ratings")
```
### ⚡ Production Ready
- Universal storage (FileSystem, S3, OPFS, Memory)
- Zero configuration with intelligent defaults
- Full TypeScript support
- Cross-platform compatibility
## Quick Example
## Quick Start
```typescript
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
// Initialize
const brain = new Brainy()
await brain.init()
// Add entities (nouns)
const articleId = await brain.add({
data: "Revolutionary AI Breakthrough",
type: NounType.Document,
metadata: { category: "technology", rating: 4.8 }
// Add entities — data is embedded for semantic search, metadata is indexed for filtering
const id = await brain.add({
data: 'Revolutionary AI Breakthrough',
type: NounType.Document,
metadata: { category: 'technology', rating: 4.8 }
})
const authorId = await brain.add({
data: "Dr. Sarah Chen",
type: NounType.Person,
metadata: { role: "researcher" }
// Search with Triple Intelligence
const results = await brain.find({
query: 'artificial intelligence', // Semantic search (on data)
where: { rating: { greaterThan: 4.0 } }, // Metadata filter
connected: { from: authorId, depth: 2 } // Graph traversal
})
// Create relationships (verbs)
await brain.relate({
from: authorId,
to: articleId,
type: VerbType.CreatedBy,
metadata: { date: "2024-01-15", contribution: "primary" }
})
// Query naturally
const results = await brain.find("highly rated technology articles by researchers")
```
## 📚 Complete Documentation Index
---
### 🚀 Quick Start Guides
## Core Documentation
| Document | Description |
|----------|-------------|
| [API Reference](./api/README.md) | **START HERE** - Complete API documentation with examples |
| [VFS Quick Start](./vfs/QUICK_START.md) | Virtual filesystem in 30 seconds |
| **[API Reference](./api/README.md)** | Complete API documentation — **start here** |
| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields |
| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix |
| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details |
### 🆕 Migration & Optimization
---
## Architecture
| Document | Description |
|----------|-------------|
| [v3→v4 Migration Guide](./MIGRATION-V3-TO-V4.md) | **NEW** - Upgrade guide with zero breaking changes |
| [AWS S3 Cost Optimization](./operations/cost-optimization-aws-s3.md) | **NEW** - 96% cost savings with lifecycle policies |
| [GCS Cost Optimization](./operations/cost-optimization-gcs.md) | **NEW** - 94% savings with Autoclass |
| [Azure Cost Optimization](./operations/cost-optimization-azure.md) | **NEW** - 95% savings with tier management |
| [Cloudflare R2 Cost Guide](./operations/cost-optimization-cloudflare-r2.md) | **NEW** - Zero egress fees + S3-compatible API |
### 🎯 Core Concepts
| Document | Description |
|----------|-------------|
| [Architecture Overview](./architecture/overview.md) | High-level system design and components |
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | Revolutionary data model - entities and relationships |
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Field unified query system |
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
| [Architecture Overview](./architecture/overview.md) | High-level system design |
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Metadata unified query |
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | 42 nouns + 127 verbs type system |
| [Stage 3 Canonical Taxonomy](./STAGE3-CANONICAL-TAXONOMY.md) | Complete type reference |
| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
| [Data Storage Architecture](./architecture/data-storage-architecture.md) | Metadata/vector separation, sharding |
| [Index Architecture](./architecture/index-architecture.md) | HNSW, Graph, and Metadata indexing |
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
### 💾 Storage & Deployment
---
## Virtual Filesystem (VFS)
| Document | Description |
|----------|-------------|
| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | Deploy on AWS, GCP, Azure, Cloudflare |
| [AWS Deployment](./deployment/aws-deployment.md) | AWS-specific deployment patterns |
| [GCP Deployment](./deployment/gcp-deployment.md) | Google Cloud deployment |
| [Kubernetes Deployment](./deployment/kubernetes-deployment.md) | K8s deployment configurations |
| [Distributed Storage](./architecture/distributed-storage.md) | Multi-node storage coordination |
| [VFS Quick Start](./vfs/QUICK_START.md) | Get started in 30 seconds |
| [VFS Core](./vfs/VFS_CORE.md) | Core concepts and architecture |
| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
See [vfs/](./vfs/) for the complete VFS documentation set.
---
## Guides
| Document | Description |
|----------|-------------|
| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports |
| [Natural Language](./guides/natural-language.md) | Query in plain English |
| [Neural API](./guides/neural-api.md) | AI-powered features |
| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers |
| [Framework Integration](./guides/framework-integration.md) | React, Vue, Angular, Svelte |
---
## Storage & Deployment
| Document | Description |
|----------|-------------|
| [Cloud Deployment](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | Deploy on AWS, GCP, Azure, Cloudflare |
| [Extending Storage](./EXTENDING_STORAGE.md) | Create custom storage adapters |
| [AWS S3 Cost Optimization](./operations/cost-optimization-aws-s3.md) | 96% cost savings |
| [GCS Cost Optimization](./operations/cost-optimization-gcs.md) | 94% savings with Autoclass |
| [Azure Cost Optimization](./operations/cost-optimization-azure.md) | 95% savings |
| [R2 Cost Optimization](./operations/cost-optimization-cloudflare-r2.md) | Zero egress fees |
| [Capacity Planning](./operations/capacity-planning.md) | Scale to millions of entities |
### 📊 API Documentation
| Document | Description |
|----------|-------------|
| [API Reference](./API_REFERENCE.md) | Complete API documentation |
| [Comprehensive API Overview](./api/COMPREHENSIVE_API_OVERVIEW.md) | All APIs with examples |
| [API Decision Tree](./API_DECISION_TREE.md) | Choose the right API for your use case |
| [Core API Patterns](./CORE_API_PATTERNS.md) | Common patterns and best practices |
| [Neural API Patterns](./NEURAL_API_PATTERNS.md) | AI-powered query patterns |
| [Find System](./FIND_SYSTEM.md) | Natural language find() API |
| [API Surface Design](./architecture/API_SURFACE_DESIGN.md) | API design principles |
| [API Returns](./api-returns.md) | Return types and structures |
### 🔧 Framework Integration
| Document | Description |
|----------|-------------|
| [Framework Integration Guide](./guides/framework-integration.md) | React, Vue, Angular, Svelte, etc. |
| [Next.js Integration](./guides/nextjs-integration.md) | Server-side rendering with Brainy |
| [Vue.js Integration](./guides/vue-integration.md) | Vue 3 integration patterns |
### 📁 Virtual Filesystem (VFS)
| Document | Description |
|----------|-------------|
| [VFS Core Documentation](./vfs/VFS_CORE.md) | Core VFS concepts and architecture |
| [Semantic VFS Guide](./vfs/SEMANTIC_VFS.md) | Semantic filesystem projections |
| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
| [Neural Extraction](./vfs/NEURAL_EXTRACTION.md) | AI-powered file analysis |
| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
| [User Functions](./vfs/USER_FUNCTIONS.md) | Custom VFS functions |
| [VFS Graph Types](./vfs/VFS_GRAPH_TYPES.md) | Graph-based VFS projections |
| [VFS Examples & Scenarios](./vfs/VFS_EXAMPLES_SCENARIOS.md) | Real-world VFS examples |
| [VFS Initialization](./vfs/VFS_INITIALIZATION.md) | Setup and configuration |
| [Projection Strategy API](./vfs/PROJECTION_STRATEGY_API.md) | Custom projection strategies |
| [Building File Explorers](./vfs/building-file-explorers.md) | Build custom file browsers |
| [VFS Troubleshooting](./vfs/TROUBLESHOOTING.md) | Common issues and solutions |
### 🧠 Advanced Topics
| Document | Description |
|----------|-------------|
| [Natural Language Queries](./guides/natural-language.md) | Query in plain English |
| [Neural API](./guides/neural-api.md) | AI-powered features |
| [Import Anything](./guides/import-anything.md) | Import data from any source |
| [Distributed Systems](./guides/distributed-system.md) | Multi-node deployments |
| [Model Loading](./guides/model-loading.md) | Load and manage AI models |
| [Model Loading Quick Reference](./MODEL_LOADING_QUICK_REFERENCE.md) | Model loading cheat sheet |
| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No paywalls, no tiers |
### 🔌 Augmentations (Plugins)
---
## Plugins & Augmentations
| Document | Description |
|----------|-------------|
| [Plugins](./PLUGINS.md) | Plugin system overview |
| [Creating Augmentations](./CREATING-AUGMENTATIONS.md) | Build custom plugins |
| [Augmentations Complete Reference](./augmentations/COMPLETE-REFERENCE.md) | Full augmentation API |
| [Augmentations Reference](./augmentations/COMPLETE-REFERENCE.md) | Full augmentation API |
| [Augmentations Developer Guide](./augmentations/DEVELOPER-GUIDE.md) | Plugin development guide |
| [Augmentation Configuration](./augmentations/CONFIGURATION.md) | Configure augmentations |
| [Augmentation System](./architecture/augmentations.md) | System architecture |
| [Augmentation System Audit](./architecture/augmentation-system-audit.md) | Actual vs planned features |
| [Augmentations Actual](./architecture/augmentations-actual.md) | Currently implemented augmentations |
| [API Server Augmentation](./augmentations/api-server.md) | REST API server plugin |
### ⚡ Performance & Scaling
---
## Performance & Scaling
| Document | Description |
|----------|-------------|
| [Performance Guide](./PERFORMANCE.md) | Optimization techniques |
| [Performance Analysis](./architecture/PERFORMANCE_ANALYSIS.md) | Benchmarks and analysis |
| [Scaling Guide](./SCALING.md) | Scale to billions of entities |
| [Clustering Algorithms Analysis](./architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md) | HNSW algorithm details |
| [Initialization & Rebuild](./architecture/initialization-and-rebuild.md) | Index management |
| [Performance](./PERFORMANCE.md) | Optimization techniques |
| [Scaling](./SCALING.md) | Scale to billions of entities |
| [Batching](./BATCHING.md) | Batch operations guide |
### 📋 Reference
---
## Migration & Reference
| Document | Description |
|----------|-------------|
| [Complete Feature List](./features/complete-feature-list.md) | All Brainy features |
| [v3 Features](./features/v3-features.md) | v3.x feature list |
| [Metadata Architecture](./architecture/METADATA_ARCHITECTURE.md) | Metadata namespacing |
| [Metadata Contract Implementation](./METADATA_CONTRACT_IMPLEMENTATION.md) | Metadata API contracts |
| [Finite Type System](./architecture/finite-type-system.md) | Type-safe noun/verb taxonomy |
| [Validation](./VALIDATION.md) | Data validation rules |
| [Universal Display Augmentation](./universal-display-augmentation.md) | Display system |
### 🛠️ Development
| Document | Description |
|----------|-------------|
| [Troubleshooting](./troubleshooting.md) | Common issues and solutions |
| [v3 to v4 Migration](./MIGRATION-V3-TO-V4.md) | Upgrade guide |
| [Release Guide](./RELEASE-GUIDE.md) | How to release new versions |
| [Production Architecture](./PRODUCTION_SERVICE_ARCHITECTURE.md) | Ops reference |
### 🔍 Internal Documentation
---
## Internal
| Document | Description |
|----------|-------------|
| [Audit Report](./internal/AUDIT_REPORT.md) | Feature audit |
| [Cleanup Summary](./internal/CLEANUP_SUMMARY.md) | Codebase cleanup notes |
| [Honest Status](./internal/HONEST_STATUS.md) | Actual implementation status |
---
## Documentation Structure
```
docs/
├── README.md (this file) # Complete documentation index
├── MIGRATION-V3-TO-V4.md # Migration guide
│
├── guides/ # User guides
│ ├── natural-language.md
│ ├── neural-api.md
│ ├── import-anything.md
│ ├── framework-integration.md
│ ├── nextjs-integration.md
│ ├── vue-integration.md
│ ├── distributed-system.md
│ ├── model-loading.md
│ └── enterprise-for-everyone.md
│
├── architecture/ # System architecture
│ ├── overview.md
│ ├── noun-verb-taxonomy.md
│ ├── triple-intelligence.md
│ ├── zero-config.md
│ ├── storage-architecture.md│ ├── data-storage-architecture.md│ ├── index-architecture.md
│ ├── distributed-storage.md
│ ├── augmentations.md
│ ├── augmentation-system-audit.md
│ ├── augmentations-actual.md
│ ├── finite-type-system.md
│ └── ...
│
├── operations/ # Operations guides
│ ├── cost-optimization-aws-s3.md│ ├── cost-optimization-gcs.md│ ├── cost-optimization-azure.md│ ├── cost-optimization-cloudflare-r2.md│ └── capacity-planning.md
│
├── deployment/ # Deployment guides
│ ├── CLOUD_DEPLOYMENT_GUIDE.md│ ├── aws-deployment.md
│ ├── gcp-deployment.md
│ └── kubernetes-deployment.md
│
├── vfs/ # Virtual Filesystem docs
│ ├── QUICK_START.md
│ ├── VFS_CORE.md
│ ├── SEMANTIC_VFS.md
│ ├── VFS_API_GUIDE.md
│ ├── NEURAL_EXTRACTION.md
│ ├── COMMON_PATTERNS.md
│ └── ...
│
├── api/ # API documentation
│ ├── README.md
│ └── COMPREHENSIVE_API_OVERVIEW.md
│
├── augmentations/ # Augmentation docs
│ ├── COMPLETE-REFERENCE.md
│ ├── DEVELOPER-GUIDE.md
│ ├── CONFIGURATION.md
│ └── api-server.md
│
├── features/ # Feature documentation
│ ├── complete-feature-list.md
│ └── v3-features.md
│
└── internal/ # Internal docs
├── AUDIT_REPORT.md
├── CLEANUP_SUMMARY.md
└── HONEST_STATUS.md
```
## Community
- **GitHub**: [github.com/brainy-org/brainy](https://github.com/brainy-org/brainy)
- **Issues**: [Report bugs or request features](https://github.com/brainy-org/brainy/issues)
- **Discussions**: [Join the conversation](https://github.com/brainy-org/brainy/discussions)
## License
Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.