docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
parent
2626ab8d62
commit
adda1570f3
22 changed files with 570 additions and 2658 deletions
|
|
@ -1,406 +0,0 @@
|
|||
# Distributed Brainy System Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy introduces a groundbreaking **zero-configuration distributed system** that transforms how vector databases scale. Unlike traditional distributed databases that require complex setup (Consul, etcd, Zookeeper), Brainy uses your existing storage (S3, GCS, R2) as the coordination layer.
|
||||
|
||||
## Key Innovation: Storage-Based Coordination
|
||||
|
||||
Instead of requiring separate infrastructure for coordination, Brainy leverages your storage backend:
|
||||
|
||||
```typescript
|
||||
// Traditional distributed database setup:
|
||||
// ❌ Setup Consul/etcd
|
||||
// ❌ Configure node discovery
|
||||
// ❌ Setup health checks
|
||||
// ❌ Configure sharding
|
||||
// ❌ Setup replication
|
||||
|
||||
// Brainy distributed setup:
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: { bucket: 'my-data' }
|
||||
},
|
||||
distributed: true // ✅ That's it!
|
||||
})
|
||||
```
|
||||
|
||||
## Real-World Scenarios
|
||||
|
||||
### 1. Processing Multiple Streaming Data Sources (Bluesky, Twitter, etc.)
|
||||
|
||||
**Problem**: You need to ingest millions of posts from multiple social media firehoses simultaneously while maintaining search performance.
|
||||
|
||||
**Traditional Approach**:
|
||||
- Separate ingestion and search clusters
|
||||
- Complex queue systems (Kafka, RabbitMQ)
|
||||
- Manual sharding configuration
|
||||
- Complicated backpressure handling
|
||||
|
||||
**Brainy Solution**:
|
||||
```typescript
|
||||
// Node 1: Bluesky Ingestion
|
||||
const ingestionNode1 = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'social-data' }},
|
||||
distributed: true,
|
||||
writeOnly: true // Optimized for writes
|
||||
})
|
||||
|
||||
// Continuously ingest Bluesky firehose
|
||||
blueskyStream.on('post', async (post) => {
|
||||
await ingestionNode1.add({
|
||||
content: post.text,
|
||||
author: post.author,
|
||||
timestamp: post.createdAt,
|
||||
platform: 'bluesky'
|
||||
}, 'social-post')
|
||||
})
|
||||
|
||||
// Node 2: Twitter Ingestion (separate machine)
|
||||
const ingestionNode2 = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'social-data' }},
|
||||
distributed: true,
|
||||
writeOnly: true
|
||||
})
|
||||
|
||||
// Node 3-5: Search nodes (auto-balanced)
|
||||
const searchNode = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'social-data' }},
|
||||
distributed: true,
|
||||
readOnly: true // Optimized for queries
|
||||
})
|
||||
|
||||
// Search across ALL data from ALL sources
|
||||
const results = await searchNode.find('AI trends', 100)
|
||||
// Automatically queries all shards across all nodes!
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- **Auto-sharding**: Data automatically distributed by content hash
|
||||
- **No bottlenecks**: Each ingestion node writes directly to storage
|
||||
- **Live search**: Search nodes see new data immediately
|
||||
- **Auto-scaling**: Add nodes anytime, data rebalances automatically
|
||||
|
||||
### 2. Multi-Tenant SaaS Application
|
||||
|
||||
**Problem**: Each customer needs isolated, fast search across their documents, with ability to scale per customer.
|
||||
|
||||
**Brainy Solution**:
|
||||
```typescript
|
||||
// Spin up dedicated nodes per large customer
|
||||
const enterpriseNode = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: {
|
||||
bucket: 'customer-data',
|
||||
prefix: 'customer-123/' // Isolated data
|
||||
}
|
||||
},
|
||||
distributed: true
|
||||
})
|
||||
|
||||
// Shared nodes for smaller customers
|
||||
const sharedNode = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
options: { bucket: 'shared-customers' }
|
||||
},
|
||||
distributed: true
|
||||
})
|
||||
|
||||
// Domain-based sharding ensures customer data stays together
|
||||
await sharedNode.add(document, 'document', {
|
||||
customerId: 'cust-456', // Used for shard assignment
|
||||
domain: 'customer-456' // Keeps related data together
|
||||
})
|
||||
```
|
||||
|
||||
### 3. Global Knowledge Graph with Local Inference
|
||||
|
||||
**Problem**: Building a knowledge graph that needs both global connectivity and local LLM inference.
|
||||
|
||||
**Brainy Solution**:
|
||||
```typescript
|
||||
// Edge nodes near users (with GPU)
|
||||
const edgeNode = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'knowledge-graph' }},
|
||||
distributed: true,
|
||||
models: {
|
||||
embed: { model: 'BAAI/bge-base-en-v1.5' },
|
||||
chat: { model: 'meta-llama/Llama-3.2-3B-Instruct' }
|
||||
}
|
||||
})
|
||||
|
||||
// Central nodes for graph operations
|
||||
const graphNode = new Brainy({
|
||||
storage: { type: 's3', options: { bucket: 'knowledge-graph' }},
|
||||
distributed: true,
|
||||
augmentations: ['graph', 'triple-intelligence']
|
||||
})
|
||||
|
||||
// Local inference with global knowledge
|
||||
const context = await edgeNode.find(userQuery, 10)
|
||||
const response = await edgeNode.chat(userQuery, { context })
|
||||
|
||||
// Graph traversal across all nodes
|
||||
const connections = await graphNode.traverse({
|
||||
start: 'concept:AI',
|
||||
depth: 3,
|
||||
relationship: 'related-to'
|
||||
})
|
||||
```
|
||||
|
||||
## Competitive Advantages
|
||||
|
||||
### vs. Pinecone/Weaviate/Qdrant
|
||||
|
||||
| Feature | Traditional Vector DBs | Brainy Distributed |
|
||||
|---------|----------------------|-------------------|
|
||||
| Setup Complexity | High (k8s, operators) | Zero (just storage) |
|
||||
| Minimum Nodes | 3-5 for HA | 1 (scale as needed) |
|
||||
| Coordination | External (etcd, Consul) | Built-in (via storage) |
|
||||
| Data Locality | Random sharding | Domain-aware sharding |
|
||||
| Query Planning | Basic | Triple Intelligence |
|
||||
| Cost at Scale | High (always-on clusters) | Low (scale to zero) |
|
||||
|
||||
### vs. Neo4j/ArangoDB (Graph Databases)
|
||||
|
||||
| Feature | Graph Databases | Brainy Distributed |
|
||||
|---------|----------------|-------------------|
|
||||
| Vector Search | Bolt-on/Limited | Native HNSW |
|
||||
| Embedding Generation | External | Built-in (30+ models) |
|
||||
| Distributed Transactions | Complex/Slow | Eventually consistent |
|
||||
| Natural Language | No | Native (Triple Intelligence) |
|
||||
| Setup | Very Complex | Zero config |
|
||||
|
||||
### vs. Elasticsearch/OpenSearch
|
||||
|
||||
| Feature | Elasticsearch | Brainy Distributed |
|
||||
|---------|--------------|-------------------|
|
||||
| Vector Support | Added later (slow) | Native (fast HNSW) |
|
||||
| Cluster Management | Complex (master nodes) | Automatic (via storage) |
|
||||
| Shard Rebalancing | Manual/Risky | Automatic/Safe |
|
||||
| Memory Usage | Very High | Efficient |
|
||||
| Query Language | Complex DSL | Natural language |
|
||||
|
||||
## Innovative Features
|
||||
|
||||
### 1. Domain-Aware Sharding
|
||||
|
||||
Unlike hash-based sharding, Brainy understands data relationships:
|
||||
|
||||
```typescript
|
||||
// Documents about the same topic stay on the same shard
|
||||
await brain.add(doc1, 'document', { domain: 'physics' })
|
||||
await brain.add(doc2, 'document', { domain: 'physics' })
|
||||
// Both documents on same shard = faster related queries
|
||||
|
||||
// Customer data stays together
|
||||
await brain.add(order, 'order', { customerId: 'cust-123' })
|
||||
await brain.add(invoice, 'invoice', { customerId: 'cust-123' })
|
||||
// Same customer = same shard = better locality
|
||||
```
|
||||
|
||||
### 2. Streaming Shard Migration
|
||||
|
||||
Zero-downtime data movement between nodes:
|
||||
|
||||
```typescript
|
||||
// Automatically triggered when nodes join/leave
|
||||
// Uses HTTP streaming for efficiency
|
||||
// Validates data integrity
|
||||
// Atomic ownership transfer
|
||||
// No query downtime!
|
||||
```
|
||||
|
||||
### 3. Storage-Based Consensus
|
||||
|
||||
No Raft/Paxos complexity:
|
||||
|
||||
```typescript
|
||||
// Leader election via storage atomic operations
|
||||
// Health monitoring via storage heartbeats
|
||||
// Configuration consensus via storage CAS
|
||||
// No split-brain issues!
|
||||
```
|
||||
|
||||
### 4. Intelligent Query Planning
|
||||
|
||||
The distributed query planner understands:
|
||||
- Which shards contain relevant data
|
||||
- Node health and latency
|
||||
- Data locality and caching
|
||||
- Triple Intelligence scoring
|
||||
|
||||
```typescript
|
||||
// Automatically optimizes query execution
|
||||
const results = await brain.find('quantum physics')
|
||||
// Planner knows:
|
||||
// - Physics domain → shard-3
|
||||
// - Node-2 has shard-3 cached
|
||||
// - Route query to node-2
|
||||
// - Merge results with Triple Intelligence
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
### Scalability
|
||||
- **Horizontal**: Add nodes anytime
|
||||
- **Vertical**: Nodes can differ in size
|
||||
- **Geographic**: Nodes can be globally distributed
|
||||
- **Elastic**: Scale to zero when idle
|
||||
|
||||
### Throughput
|
||||
- **Writes**: Linear scaling with nodes
|
||||
- **Reads**: Sub-linear (due to caching)
|
||||
- **Mixed**: Read/write optimized nodes
|
||||
|
||||
### Latency
|
||||
- **Local queries**: ~10ms p50
|
||||
- **Distributed queries**: ~50ms p50
|
||||
- **Shard migration**: Streaming (no bulk pause)
|
||||
|
||||
## Use Cases Where Brainy Excels
|
||||
|
||||
### ✅ Perfect For:
|
||||
|
||||
1. **Multi-source data ingestion** (social media, logs, events)
|
||||
2. **Global search applications** (distributed teams)
|
||||
3. **Multi-tenant SaaS** (customer isolation)
|
||||
4. **Knowledge graphs** (with vector search)
|
||||
5. **Edge AI applications** (local inference, global knowledge)
|
||||
6. **Document intelligence** (contracts, research papers)
|
||||
7. **Real-time analytics** (streaming + search)
|
||||
|
||||
### ⚠️ Consider Alternatives For:
|
||||
|
||||
1. **Strong consistency requirements** (use PostgreSQL)
|
||||
2. **Sub-millisecond latency** (use Redis)
|
||||
3. **Complex transactions** (use traditional RDBMS)
|
||||
4. **Purely structured data** (use columnar stores)
|
||||
|
||||
## Migration from Other Systems
|
||||
|
||||
### From Pinecone/Weaviate:
|
||||
```typescript
|
||||
// Your existing vector search still works
|
||||
const results = await brain.search(embedding, 10)
|
||||
|
||||
// But now you can scale horizontally!
|
||||
// And add graph relationships!
|
||||
// And use natural language!
|
||||
```
|
||||
|
||||
### From Elasticsearch:
|
||||
```typescript
|
||||
// Import your documents
|
||||
await brain.import('./elasticsearch-export.json')
|
||||
|
||||
// Queries are simpler
|
||||
const results = await brain.find('user query')
|
||||
// No complex DSL needed!
|
||||
```
|
||||
|
||||
### From Neo4j:
|
||||
```typescript
|
||||
// Import your graph
|
||||
await brain.importGraph('./neo4j-export.cypher')
|
||||
|
||||
// Now with vector search!
|
||||
const similar = await brain.find('concepts like quantum computing')
|
||||
```
|
||||
|
||||
## Deployment Patterns
|
||||
|
||||
### 1. Start Simple, Scale Later
|
||||
```typescript
|
||||
// Day 1: Single node
|
||||
const brain = new Brainy({ storage: 's3' })
|
||||
|
||||
// Month 2: Growing data, add distribution
|
||||
const brain = new Brainy({
|
||||
storage: 's3',
|
||||
distributed: true // Just add this!
|
||||
})
|
||||
|
||||
// Month 6: Multiple nodes auto-balance
|
||||
// No migration needed!
|
||||
```
|
||||
|
||||
### 2. Geographic Distribution
|
||||
```typescript
|
||||
// US Node
|
||||
const usNode = new Brainy({
|
||||
storage: { region: 'us-east-1' },
|
||||
distributed: true
|
||||
})
|
||||
|
||||
// EU Node
|
||||
const euNode = new Brainy({
|
||||
storage: { region: 'eu-west-1' },
|
||||
distributed: true
|
||||
})
|
||||
|
||||
// They automatically coordinate!
|
||||
```
|
||||
|
||||
### 3. Specialized Nodes
|
||||
```typescript
|
||||
// GPU nodes for embedding
|
||||
const embedNode = new Brainy({
|
||||
distributed: true,
|
||||
writeOnly: true,
|
||||
models: { embed: 'large-model' }
|
||||
})
|
||||
|
||||
// CPU nodes for search
|
||||
const searchNode = new Brainy({
|
||||
distributed: true,
|
||||
readOnly: true
|
||||
})
|
||||
```
|
||||
|
||||
## Monitoring & Operations
|
||||
|
||||
### Health Checks
|
||||
```typescript
|
||||
const health = await brain.getClusterHealth()
|
||||
// {
|
||||
// nodes: 5,
|
||||
// healthy: 5,
|
||||
// shards: 16,
|
||||
// status: 'green'
|
||||
// }
|
||||
```
|
||||
|
||||
### Shard Distribution
|
||||
```typescript
|
||||
const shards = await brain.getShardDistribution()
|
||||
// Shows which nodes own which shards
|
||||
```
|
||||
|
||||
### Migration Status
|
||||
```typescript
|
||||
const migrations = await brain.getActiveMigrations()
|
||||
// Shows ongoing shard movements
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
Brainy's distributed system is **production-ready** and offers:
|
||||
|
||||
1. **True zero-configuration** - Just add `distributed: true`
|
||||
2. **Storage-based coordination** - No external dependencies
|
||||
3. **Intelligent sharding** - Domain-aware data placement
|
||||
4. **Automatic operations** - Rebalancing, failover, scaling
|
||||
5. **Unified interface** - Vector + Graph + Document + LLM
|
||||
|
||||
This is not just another distributed database. It's a fundamental rethinking of how distributed systems should work in the cloud era.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. [Try the distributed quick start](./distributed-quickstart.md)
|
||||
2. [Read the architecture deep dive](../architecture/distributed-storage.md)
|
||||
3. [View benchmarks](../benchmarks/distributed-performance.md)
|
||||
4. [Deploy to production](./production-deployment.md)
|
||||
|
|
@ -1687,11 +1687,11 @@ verbCounts['CreatedBy']
|
|||
|
||||
---
|
||||
|
||||
### 8. Storage by Cloud Provider
|
||||
### 8. Storage Layout
|
||||
|
||||
Brainy supports multiple storage adapters:
|
||||
Brainy 8.0 ships two adapters: filesystem and memory.
|
||||
|
||||
#### File System (Local)
|
||||
#### Filesystem (Default)
|
||||
```
|
||||
.brainy/
|
||||
├── nouns/
|
||||
|
|
@ -1701,43 +1701,18 @@ Brainy supports multiple storage adapters:
|
|||
└── index.json
|
||||
```
|
||||
|
||||
#### Google Cloud Storage (GCS)
|
||||
```
|
||||
gs://my-bucket/brainy/
|
||||
├── nouns/
|
||||
│ └── ent_mona_lisa_1730000000000.json
|
||||
├── nouns-metadata/
|
||||
│ └── ent_mona_lisa_1730000000000.json
|
||||
└── ...
|
||||
```
|
||||
|
||||
#### Amazon S3
|
||||
```
|
||||
s3://my-bucket/brainy/
|
||||
├── nouns/
|
||||
│ └── ent_mona_lisa_1730000000000.json
|
||||
└── ...
|
||||
```
|
||||
|
||||
#### Cloudflare R2
|
||||
```
|
||||
r2://my-bucket/brainy/
|
||||
├── nouns/
|
||||
│ └── ent_mona_lisa_1730000000000.json
|
||||
└── ...
|
||||
```
|
||||
|
||||
**Configuration**:
|
||||
```typescript
|
||||
const brain = await Brainy.create({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
bucket: 'my-bucket',
|
||||
prefix: 'brainy/'
|
||||
type: 'filesystem',
|
||||
rootDirectory: './.brainy'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
For off-site backup, snapshot `rootDirectory` from your scheduler with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`. Brainy itself doesn't reach out to object storage.
|
||||
|
||||
---
|
||||
|
||||
## Performance & Scale
|
||||
|
|
|
|||
|
|
@ -17,10 +17,7 @@ store. This guide covers the safe ways to query a running Brainy directory.
|
|||
|
||||
## The cardinal rule
|
||||
|
||||
**Never open a second writer on the same directory.** It will throw on
|
||||
filesystem storage; on cloud storage it'll silently overwrite the live
|
||||
writer's state. Use `Brainy.openReadOnly()` or the `brainy inspect` CLI
|
||||
instead.
|
||||
**Never open a second writer on the same directory.** Filesystem storage will throw, and any other write path will corrupt the live writer's state. Use `Brainy.openReadOnly()` or the `brainy inspect` CLI instead.
|
||||
|
||||
## The CLI is the fastest path
|
||||
|
||||
|
|
@ -97,7 +94,7 @@ $ brainy inspect health /data/brain
|
|||
{
|
||||
"overall": "warn",
|
||||
"checks": [
|
||||
{ "name": "index-parity", "status": "pass", "message": "HNSW (1851) and metadata (1851) agree." },
|
||||
{ "name": "index-parity", "status": "pass", "message": "Vector (1851) and metadata (1851) agree." },
|
||||
{ "name": "field-registry", "status": "pass", "message": "23 fields registered for 1851 entities." },
|
||||
{ "name": "seeded-records", "status": "warn", "message": "15 entities tagged _seeded:true." },
|
||||
{ "name": "writer-heartbeat", "status": "pass", "message": "Writer healthy (PID 1774431...)." }
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue