feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object properties into top-level metadata. data is for semantic search (HNSW), metadata is for structured where-filter queries (MetadataIndex). - Fix numeric range queries in MetadataIndex — use numeric-aware comparison instead of lexicographic string comparison for normalized values. - Add data field to RelateParams and Relation types for relationship content. - Add where.type → where.noun alias in metadata-only find() path. - Rewrite README: focused ~350 lines from 791, quick start first, feature showcase with mini-snippets, organized doc links, no version callouts. - Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs. - Remove 10 outdated/redundant doc files consolidated into API reference. - Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods. - Fix tests asserting data properties appear in metadata (data model violation). - Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
This commit is contained in:
parent
edb5ec4696
commit
0ddc05a5bb
30 changed files with 1356 additions and 7001 deletions
|
|
@ -1,481 +0,0 @@
|
|||
# 🧠 Brainy API Decision Tree
|
||||
|
||||
*Choose the right API for your use case with confidence*
|
||||
|
||||
This guide helps you navigate Brainy's comprehensive API surface by asking the right questions to find the perfect method for your specific needs.
|
||||
|
||||
## 🎯 Quick Start: What do you want to do?
|
||||
|
||||
### 📝 **Adding Data**
|
||||
- **Single entity** → [`brainy.add()`](#adding-single-entities)
|
||||
- **Multiple entities** → [`brainy.addMany()`](#adding-multiple-entities)
|
||||
- **Streaming/real-time data** → [Streaming Pipeline](#streaming-data)
|
||||
|
||||
### 🔍 **Finding Data**
|
||||
- **Natural language search** → [`brainy.find("search query")`](#natural-language-search)
|
||||
- **Structured/filtered search** → [`brainy.find({ query, where, type })`](#structured-search)
|
||||
- **Similar entities** → [`brainy.similar()`](#similarity-search)
|
||||
- **Get by ID** → [`brainy.get()`](#retrieval-by-id)
|
||||
|
||||
### 🔗 **Relationships**
|
||||
- **Create relationships** → [`brainy.relate()`](#creating-relationships)
|
||||
- **Query relationships** → [`brainy.getRelations()`](#querying-relationships)
|
||||
- **Graph traversal** → [Graph Navigation](#graph-operations)
|
||||
|
||||
### 📊 **Advanced Features**
|
||||
- **File management** → [VFS (Virtual File System)](#file-operations)
|
||||
- **AI-powered analysis** → [Neural API](#neural-analysis)
|
||||
- **Clustering/insights** → [Intelligence Systems](#intelligence-systems)
|
||||
|
||||
---
|
||||
|
||||
## 🔀 Decision Tree Flow
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[What are you trying to do?] --> B[Store Data]
|
||||
A --> C[Find Data]
|
||||
A --> D[Manage Relationships]
|
||||
A --> E[Work with Files]
|
||||
A --> F[AI Analysis]
|
||||
|
||||
B --> B1[Single Item]
|
||||
B --> B2[Multiple Items]
|
||||
B --> B3[Real-time Stream]
|
||||
|
||||
C --> C1[I know the ID]
|
||||
C --> C2[Natural language query]
|
||||
C --> C3[Complex filters]
|
||||
C --> C4[Find similar items]
|
||||
|
||||
D --> D1[Create relationship]
|
||||
D --> D2[Query relationships]
|
||||
D --> D3[Graph traversal]
|
||||
|
||||
E --> E1[File operations]
|
||||
E --> E2[Knowledge-enhanced files]
|
||||
|
||||
F --> F1[Clustering]
|
||||
F --> F2[Similarity analysis]
|
||||
F --> F3[Insights generation]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Adding Data
|
||||
|
||||
### Adding Single Entities
|
||||
|
||||
**Use `brainy.add()` when:**
|
||||
- Adding one entity at a time
|
||||
- You need the ID immediately for further operations
|
||||
- Working with user input or real-time data
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for single entities
|
||||
const id = await brainy.add({
|
||||
data: "New research paper on quantum computing",
|
||||
type: NounType.Document,
|
||||
metadata: { category: "research", priority: "high" }
|
||||
})
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **Single item?** → `add()`
|
||||
- **Need immediate ID?** → `add()`
|
||||
- **Interactive application?** → `add()`
|
||||
|
||||
### Adding Multiple Entities
|
||||
|
||||
**Use `brainy.addMany()` when:**
|
||||
- Bulk importing data
|
||||
- Processing batches (>10 items)
|
||||
- Performance is critical
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for bulk operations
|
||||
const result = await brainy.addMany({
|
||||
items: documents.map(doc => ({
|
||||
data: doc.content,
|
||||
type: NounType.Document,
|
||||
metadata: doc.metadata
|
||||
})),
|
||||
chunkSize: 100,
|
||||
parallel: true
|
||||
})
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **Multiple items (>10)?** → `addMany()`
|
||||
- **Batch processing?** → `addMany()`
|
||||
- **Can tolerate some failures?** → `addMany()` with `continueOnError: true`
|
||||
|
||||
### Streaming Data
|
||||
|
||||
**Use Streaming Pipeline when:**
|
||||
- Real-time data ingestion
|
||||
- Processing large datasets that don't fit in memory
|
||||
- Need transformation during ingestion
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for streaming
|
||||
const pipeline = brainy.streaming.pipeline()
|
||||
.transform(data => ({ ...data, processed: true }))
|
||||
.batch(50)
|
||||
.into(brainy)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Finding Data
|
||||
|
||||
### Natural Language Search
|
||||
|
||||
**Use `brainy.find("query string")` when:**
|
||||
- User is typing search queries
|
||||
- You want semantic understanding
|
||||
- Building search interfaces
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for user searches
|
||||
const results = await brainy.find("documents about machine learning")
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **User-generated query?** → Natural language `find()`
|
||||
- **Semantic understanding needed?** → Natural language `find()`
|
||||
- **Search interface?** → Natural language `find()`
|
||||
|
||||
### Structured Search
|
||||
|
||||
**Use `brainy.find({ query, where, type })` when:**
|
||||
- Complex filtering requirements
|
||||
- Combining text search with metadata filters
|
||||
- Performance-critical searches
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for complex queries
|
||||
const results = await brainy.find({
|
||||
query: "neural networks",
|
||||
type: NounType.Document,
|
||||
where: {
|
||||
status: "published",
|
||||
year: { $gte: 2020 }
|
||||
},
|
||||
limit: 20
|
||||
})
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **Need metadata filtering?** → Structured `find()`
|
||||
- **Performance critical?** → Structured `find()`
|
||||
- **Complex criteria?** → Structured `find()`
|
||||
|
||||
### Similarity Search
|
||||
|
||||
**Use `brainy.similar()` when:**
|
||||
- Finding "more like this" content
|
||||
- Recommendation systems
|
||||
- Duplicate detection
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for recommendations
|
||||
const similar = await brainy.similar({
|
||||
to: "document-id-123",
|
||||
limit: 10,
|
||||
type: NounType.Document
|
||||
})
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **"More like this" feature?** → `similar()`
|
||||
- **Recommendations?** → `similar()`
|
||||
- **Duplicate detection?** → `similar()`
|
||||
|
||||
### Retrieval by ID
|
||||
|
||||
**Use `brainy.get()` when:**
|
||||
- You know the exact ID
|
||||
- Loading specific entities
|
||||
- Following relationships
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for direct access
|
||||
const entity = await brainy.get("known-id-123")
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **Known ID?** → `get()`
|
||||
- **Direct access needed?** → `get()`
|
||||
- **Following relationships?** → `get()`
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Relationships
|
||||
|
||||
### Creating Relationships
|
||||
|
||||
**Use `brainy.relate()` when:**
|
||||
- Connecting two entities
|
||||
- Building knowledge graphs
|
||||
- Modeling real-world relationships
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for connections
|
||||
await brainy.relate({
|
||||
from: "user-123",
|
||||
to: "project-456",
|
||||
type: VerbType.WorksOn,
|
||||
metadata: { role: "lead", since: "2024-01-01" }
|
||||
})
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **Connecting entities?** → `relate()`
|
||||
- **Need relationship metadata?** → `relate()`
|
||||
- **Building graphs?** → `relate()`
|
||||
|
||||
### Querying Relationships
|
||||
|
||||
**Use `brainy.getRelations()` when:**
|
||||
- Finding all connections for an entity
|
||||
- Exploring relationship patterns
|
||||
- Building relationship views
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for relationship queries
|
||||
const relations = await brainy.getRelations({
|
||||
from: "user-123",
|
||||
type: VerbType.WorksOn
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 File Operations
|
||||
|
||||
### Basic File Operations
|
||||
|
||||
**Use VFS when:**
|
||||
- Managing files and directories
|
||||
- Need hierarchical structure
|
||||
- Building file explorers
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for file management
|
||||
const vfs = brainy.vfs({ storage: 'filesystem' })
|
||||
await vfs.writeFile('/docs/readme.md', 'content')
|
||||
const files = await vfs.getDirectChildren('/docs')
|
||||
```
|
||||
|
||||
**Decision factors:**
|
||||
- **File management?** → VFS
|
||||
- **Directory structure?** → VFS
|
||||
- **File explorer interface?** → VFS
|
||||
|
||||
### Intelligent File Management
|
||||
|
||||
**Use VFS (Semantic VFS) when:**
|
||||
- Need semantic file search
|
||||
- Want AI-powered concept extraction
|
||||
- Building smart file systems
|
||||
- Require multi-dimensional file access
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for intelligent file systems
|
||||
const knowledgeVFS = await vfs.withKnowledge(brainy)
|
||||
const insights = await knowledgeVFS.getFileInsights('/project')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧠 AI Analysis
|
||||
|
||||
### Clustering
|
||||
|
||||
**Use Neural API clustering when:**
|
||||
- Discovering data patterns
|
||||
- Organizing large datasets
|
||||
- Creating automatic categories
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for pattern discovery
|
||||
const neural = brainy.neural()
|
||||
const clusters = await neural.cluster({
|
||||
entities: entityIds,
|
||||
k: 5,
|
||||
method: 'hierarchical'
|
||||
})
|
||||
```
|
||||
|
||||
### Intelligence Systems
|
||||
|
||||
**Use Triple Intelligence when:**
|
||||
- Complex multi-criteria searches
|
||||
- Advanced relationship queries
|
||||
- Performance-critical operations
|
||||
|
||||
```typescript
|
||||
// ✅ Perfect for complex queries
|
||||
const intelligence = brainy.getTripleIntelligence()
|
||||
const results = await intelligence.query({
|
||||
vector: queryVector,
|
||||
metadata: { category: 'research' },
|
||||
graph: { connected: 'user-123' }
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Performance Optimization Guide
|
||||
|
||||
### When Performance Matters
|
||||
|
||||
| Scenario | Best Choice | Why |
|
||||
|----------|-------------|-----|
|
||||
| **Bulk Import** | `addMany()` | Batched operations, parallel processing |
|
||||
| **Metadata-only Search** | `find({ where: {...} })` | Skips vector computation |
|
||||
| **Known ID Access** | `get()` | Direct index lookup |
|
||||
| **Large Result Sets** | Pagination with `offset`/`limit` | Memory efficient |
|
||||
| **Real-time Streams** | Streaming Pipeline | Memory efficient, scalable |
|
||||
|
||||
### Memory Usage Optimization
|
||||
|
||||
```typescript
|
||||
// ❌ Memory intensive
|
||||
const allResults = await brainy.find({ limit: 10000 })
|
||||
|
||||
// ✅ Memory efficient
|
||||
for (let offset = 0; offset < total; offset += 100) {
|
||||
const batch = await brainy.find({
|
||||
query: "...",
|
||||
limit: 100,
|
||||
offset
|
||||
})
|
||||
await processBatch(batch)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Common Use Case Patterns
|
||||
|
||||
### Building a Search Interface
|
||||
|
||||
```typescript
|
||||
// User types query → Natural language search
|
||||
const searchResults = await brainy.find(userQuery)
|
||||
|
||||
// User applies filters → Structured search
|
||||
const filteredResults = await brainy.find({
|
||||
query: userQuery,
|
||||
where: selectedFilters,
|
||||
type: selectedTypes
|
||||
})
|
||||
|
||||
// User clicks "more like this" → Similarity search
|
||||
const similar = await brainy.similar({ to: selectedId })
|
||||
```
|
||||
|
||||
### Building a Recommendation System
|
||||
|
||||
```typescript
|
||||
// 1. Get user's interaction history
|
||||
const user = await brainy.get(userId)
|
||||
|
||||
// 2. Find similar users
|
||||
const similarUsers = await brainy.similar({ to: userId, type: NounType.Person })
|
||||
|
||||
// 3. Get their liked content
|
||||
const recommendations = []
|
||||
for (const similarUser of similarUsers) {
|
||||
const relations = await brainy.getRelations({
|
||||
from: similarUser.id,
|
||||
type: VerbType.Likes
|
||||
})
|
||||
recommendations.push(...relations)
|
||||
}
|
||||
```
|
||||
|
||||
### Building a Knowledge Graph
|
||||
|
||||
```typescript
|
||||
// 1. Add entities
|
||||
const entities = await Promise.all([
|
||||
brainy.add({ data: "Person: Alice", type: NounType.Person }),
|
||||
brainy.add({ data: "Company: TechCorp", type: NounType.Organization }),
|
||||
brainy.add({ data: "Project: AI Assistant", type: NounType.Thing })
|
||||
])
|
||||
|
||||
// 2. Create relationships
|
||||
await brainy.relate({
|
||||
from: entities[0], // Alice
|
||||
to: entities[1], // TechCorp
|
||||
type: VerbType.WorksFor
|
||||
})
|
||||
|
||||
await brainy.relate({
|
||||
from: entities[0], // Alice
|
||||
to: entities[2], // AI Assistant
|
||||
type: VerbType.WorksOn
|
||||
})
|
||||
|
||||
// 3. Query the graph
|
||||
const aliceConnections = await brainy.getRelations({ from: entities[0] })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Migration Guide
|
||||
|
||||
### From v2.x to v3.x APIs
|
||||
|
||||
| v2.x (Deprecated) | v3.x (Current) | When to Use |
|
||||
|-------------------|----------------|-------------|
|
||||
| `brain.store()` | `brainy.add()` | Adding entities |
|
||||
| `brain.search()` | `brainy.find()` | Searching content |
|
||||
| `brain.query()` | `brainy.find({ ... })` | Complex queries |
|
||||
| `brain.similar()` | `brainy.similar()` | ✅ Same API |
|
||||
| `brain.connect()` | `brainy.relate()` | Creating relationships |
|
||||
|
||||
### Legacy Type Migration
|
||||
|
||||
```typescript
|
||||
// ❌ v2.x way
|
||||
import { ISenseAugmentation } from '@soulcraft/brainy/types/augmentations'
|
||||
|
||||
// ✅ v3.x way
|
||||
import { BrainyAugmentation } from '@soulcraft/brainy'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎪 Decision Quick Reference
|
||||
|
||||
**Need to add data?**
|
||||
- 1 item → `add()`
|
||||
- Many items → `addMany()`
|
||||
- Streaming → Pipeline
|
||||
|
||||
**Need to find data?**
|
||||
- Know ID → `get()`
|
||||
- Natural search → `find("query")`
|
||||
- Complex filters → `find({ query, where })`
|
||||
- Similar items → `similar()`
|
||||
|
||||
**Need relationships?**
|
||||
- Create → `relate()`
|
||||
- Query → `getRelations()`
|
||||
- Complex graph → Triple Intelligence
|
||||
|
||||
**Need files?**
|
||||
- Basic → VFS (standard operations)
|
||||
- Smart → Semantic VFS (6 dimensional access + neural extraction)
|
||||
|
||||
**Need AI analysis?**
|
||||
- Patterns → Neural clustering
|
||||
- Complex queries → Triple Intelligence
|
||||
|
||||
---
|
||||
|
||||
*This guide covers 95% of use cases. For edge cases or custom requirements, check the [Core API Patterns](./CORE_API_PATTERNS.md) and [Neural API Patterns](./NEURAL_API_PATTERNS.md) guides.*
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,643 +0,0 @@
|
|||
# 🧠 Core API Patterns: Modern Brainy v3.x
|
||||
|
||||
> Learn the correct patterns for Brainy's core operations. Avoid v2.x confusion and use modern, efficient APIs.
|
||||
|
||||
## 🚨 Critical: Use v3.x APIs Only
|
||||
|
||||
### ❌ **WRONG - Deprecated v2.x APIs**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - These methods don't exist in v3.x!
|
||||
await brain.addNoun(text, type, metadata) // ❌ Removed
|
||||
await brain.getNouns({ pagination }) // ❌ Removed
|
||||
await brain.addVerb(source, target, type) // ❌ Removed
|
||||
await brain.getVerbs() // ❌ Removed
|
||||
await brain.deleteNoun(id) // ❌ Removed
|
||||
await brain.deleteVerb(id) // ❌ Removed
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Modern v3.x APIs**
|
||||
|
||||
```typescript
|
||||
// ✅ Use these modern methods instead
|
||||
await brain.add({ data, type, metadata }) // Modern unified add
|
||||
await brain.find({ limit: 100 }) // Natural language search
|
||||
await brain.relate({ from, to, type }) // Clean relationship creation
|
||||
await brain.getRelations() // Modern relationship queries
|
||||
await brain.delete(id) // Unified deletion
|
||||
// Relationships auto-cascade when entities are deleted
|
||||
```
|
||||
|
||||
## 📋 Entity Management Patterns
|
||||
|
||||
### ❌ **WRONG - v2.x Style**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Old API patterns
|
||||
import { Brainy } from 'old-brainy' // ❌ Wrong import
|
||||
|
||||
const brain = new Brainy({ // ❌ Old class name
|
||||
complexConfig: true
|
||||
})
|
||||
|
||||
const id = await brain.addNoun( // ❌ Deprecated method
|
||||
"John Smith is a developer",
|
||||
"Person",
|
||||
{ role: "engineer" }
|
||||
)
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Modern Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Basic entity creation
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy() // ✅ Zero config
|
||||
await brain.init()
|
||||
|
||||
const id = await brain.add({
|
||||
data: "John Smith is a developer",
|
||||
type: NounType.Person,
|
||||
metadata: { role: "engineer", team: "backend" }
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Bulk entity creation
|
||||
const entities = [
|
||||
{ data: "React framework", type: NounType.Technology },
|
||||
{ data: "Vue.js framework", type: NounType.Technology },
|
||||
{ data: "Angular framework", type: NounType.Technology }
|
||||
]
|
||||
|
||||
const ids = await Promise.all(
|
||||
entities.map(entity => brain.add(entity))
|
||||
)
|
||||
|
||||
// ✅ Pattern 3: Entity with pre-computed vector
|
||||
const customVector = await brain.embed("Custom text")
|
||||
const vectorId = await brain.add({
|
||||
data: "Optimized content",
|
||||
type: NounType.Document,
|
||||
vector: customVector, // Skip re-embedding
|
||||
metadata: { source: "api", optimized: true }
|
||||
})
|
||||
```
|
||||
|
||||
## 🔍 Search & Discovery Patterns
|
||||
|
||||
### ❌ **WRONG - Confusing Old Patterns**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Mixing old and new APIs
|
||||
const results1 = await brain.searchText("query") // ❌ Old method
|
||||
const results2 = await brain.getNouns({ filter }) // ❌ Doesn't exist
|
||||
const results3 = await brain.findSimilar(text) // ❌ Unclear naming
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Clean Search Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Natural language search
|
||||
const results = await brain.find("React developers working on authentication")
|
||||
|
||||
// ✅ Pattern 2: Structured search with filters
|
||||
const filteredResults = await brain.find({
|
||||
like: "machine learning", // Vector similarity
|
||||
where: { // Metadata filtering
|
||||
type: NounType.Document,
|
||||
year: { $gte: 2020 },
|
||||
status: "published"
|
||||
},
|
||||
limit: 50,
|
||||
orderBy: 'relevance'
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Similarity search
|
||||
const similarItems = await brain.similar({
|
||||
to: existingEntityId, // Find items similar to this
|
||||
threshold: 0.8, // Minimum similarity
|
||||
limit: 10,
|
||||
exclude: [existingEntityId] // Don't include the source
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Advanced search with relationships
|
||||
const connectedResults = await brain.find({
|
||||
like: "frontend frameworks",
|
||||
connected: {
|
||||
to: reactId, // Connected to React
|
||||
via: "related-to", // Through this relationship
|
||||
depth: 2 // Up to 2 hops away
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## 🔗 Relationship Patterns
|
||||
|
||||
### ❌ **WRONG - Old Relationship APIs**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Old relationship patterns
|
||||
await brain.addVerb(sourceId, targetId, "uses", { strength: 0.9 }) // ❌ Old API
|
||||
const verbs = await brain.getVerbsBySource(sourceId) // ❌ Removed
|
||||
await brain.deleteVerb(verbId) // ❌ Old pattern
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Modern Relationship Management**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Create relationships
|
||||
const relationId = await brain.relate({
|
||||
from: developerId,
|
||||
to: frameworkId,
|
||||
type: VerbType.Uses,
|
||||
metadata: {
|
||||
since: "2023-01-01",
|
||||
proficiency: "expert",
|
||||
hours_per_week: 40
|
||||
}
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Query relationships
|
||||
const relationships = await brain.getRelations({
|
||||
from: developerId, // Relationships from this entity
|
||||
type: VerbType.Uses, // Of this type
|
||||
limit: 100
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Bidirectional relationships
|
||||
await brain.relate({
|
||||
from: projectId,
|
||||
to: developerId,
|
||||
type: VerbType.AssignedTo,
|
||||
bidirectional: true, // Creates reverse relationship
|
||||
metadata: { role: "lead", start_date: "2024-01-01" }
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Relationship-based discovery
|
||||
const collaborators = await brain.find({
|
||||
connected: {
|
||||
to: currentProjectId,
|
||||
via: VerbType.WorksOn,
|
||||
direction: "incoming" // Who works on this project
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## 🗃️ Data Retrieval Patterns
|
||||
|
||||
### ❌ **WRONG - Inefficient Patterns**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Loading everything
|
||||
const everything = await brain.getNouns({ limit: 1000000 }) // ❌ Crashes
|
||||
const allData = await brain.exportAll() // ❌ Memory explosion
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Efficient Data Access**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Paginated retrieval
|
||||
async function getAllEntitiesPaginated() {
|
||||
const pageSize = 100
|
||||
let offset = 0
|
||||
let allEntities = []
|
||||
|
||||
while (true) {
|
||||
const page = await brain.find({
|
||||
limit: pageSize,
|
||||
offset: offset
|
||||
})
|
||||
|
||||
if (page.length === 0) break
|
||||
|
||||
allEntities.push(...page)
|
||||
offset += pageSize
|
||||
|
||||
// Optional: Progress reporting
|
||||
console.log(`Loaded ${allEntities.length} entities...`)
|
||||
}
|
||||
|
||||
return allEntities
|
||||
}
|
||||
|
||||
// ✅ Pattern 2: Streaming large datasets
|
||||
async function* streamEntities() {
|
||||
const pageSize = 50
|
||||
let offset = 0
|
||||
|
||||
while (true) {
|
||||
const page = await brain.find({
|
||||
limit: pageSize,
|
||||
offset: offset
|
||||
})
|
||||
|
||||
if (page.length === 0) break
|
||||
|
||||
for (const entity of page) {
|
||||
yield entity
|
||||
}
|
||||
|
||||
offset += pageSize
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
for await (const entity of streamEntities()) {
|
||||
await processEntity(entity)
|
||||
}
|
||||
|
||||
// ✅ Pattern 3: Specific entity retrieval
|
||||
const entity = await brain.get(entityId)
|
||||
if (entity) {
|
||||
console.log('Entity data:', entity.data)
|
||||
console.log('Metadata:', entity.metadata)
|
||||
} else {
|
||||
console.log('Entity not found')
|
||||
}
|
||||
```
|
||||
|
||||
## 🔄 Update & Delete Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Update Patterns**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Recreating entities
|
||||
await brain.delete(oldId)
|
||||
const newId = await brain.add(updatedData) // ❌ Loses relationships
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Update Operations**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Update entity data
|
||||
await brain.update(entityId, {
|
||||
data: "Updated content here",
|
||||
metadata: {
|
||||
lastModified: Date.now(),
|
||||
version: "2.0"
|
||||
}
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Partial metadata updates
|
||||
await brain.updateMetadata(entityId, {
|
||||
status: "published",
|
||||
tags: ["important", "featured"]
|
||||
// Merges with existing metadata
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Safe deletion with cascade options
|
||||
await brain.delete(entityId, {
|
||||
cascade: true, // Delete related relationships
|
||||
backup: true // Create backup before deletion
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Bulk operations
|
||||
const updateOperations = entities.map(entity => ({
|
||||
id: entity.id,
|
||||
changes: { status: "processed" }
|
||||
}))
|
||||
|
||||
await brain.updateMany(updateOperations)
|
||||
```
|
||||
|
||||
## 🧮 Vector & Embedding Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Vector Handling**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Manual embedding without understanding
|
||||
const vector = await brain.embed(text)
|
||||
// Store vector somewhere manually // ❌ Missing integration
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Smart Vector Operations**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Automatic embedding (recommended)
|
||||
const id = await brain.add({
|
||||
data: "Content to be embedded",
|
||||
type: NounType.Document
|
||||
// Vector computed automatically
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Pre-computed vectors for optimization
|
||||
const texts = ["Text 1", "Text 2", "Text 3"]
|
||||
const vectors = await Promise.all(
|
||||
texts.map(text => brain.embed(text))
|
||||
)
|
||||
|
||||
const entities = await Promise.all(
|
||||
texts.map((text, i) => brain.add({
|
||||
data: text,
|
||||
type: NounType.Document,
|
||||
vector: vectors[i] // Skip re-embedding
|
||||
}))
|
||||
)
|
||||
|
||||
// ✅ Pattern 3: Vector similarity search
|
||||
const queryVector = await brain.embed("search query")
|
||||
const similar = await brain.similar({
|
||||
vector: queryVector, // Use vector directly
|
||||
threshold: 0.75,
|
||||
limit: 20
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Compare vectors directly
|
||||
const vector1 = await brain.embed("First text")
|
||||
const vector2 = await brain.embed("Second text")
|
||||
const similarity = brain.computeSimilarity(vector1, vector2)
|
||||
console.log(`Similarity: ${similarity}`)
|
||||
```
|
||||
|
||||
## 🏗️ Configuration Patterns
|
||||
|
||||
### ❌ **WRONG - Over-Configuration**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Complex configurations that break
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'complex',
|
||||
options: {
|
||||
nested: {
|
||||
configuration: true,
|
||||
that: "breaks"
|
||||
}
|
||||
}
|
||||
},
|
||||
embedding: {
|
||||
customModel: "broken-model",
|
||||
dimensions: 999999
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Smart Configuration**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Zero configuration (recommended)
|
||||
const brain = new Brainy() // Auto-detects everything
|
||||
await brain.init()
|
||||
|
||||
// ✅ Pattern 2: Simple storage selection
|
||||
const fsBrain = new Brainy({
|
||||
storage: { type: 'filesystem', path: './data' }
|
||||
})
|
||||
|
||||
const cloudBrain = new Brainy({
|
||||
storage: { type: 's3', bucket: 'my-data' }
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Production configuration
|
||||
const prodBrain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
bucket: process.env.BRAINY_BUCKET,
|
||||
region: process.env.AWS_REGION
|
||||
},
|
||||
silent: true, // No console output
|
||||
distributed: true, // Enable clustering
|
||||
cache: { maxSize: 10000 } // Larger cache
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Development vs production
|
||||
const isDev = process.env.NODE_ENV === 'development'
|
||||
|
||||
const brain = new Brainy({
|
||||
storage: isDev
|
||||
? { type: 'memory' } // Fast for dev
|
||||
: { type: 'filesystem', path: './brainy-data' }, // Persistent for prod
|
||||
silent: !isDev, // Verbose in dev, quiet in prod
|
||||
cache: { maxSize: isDev ? 100 : 5000 }
|
||||
})
|
||||
```
|
||||
|
||||
## 🔄 Migration from v2.x
|
||||
|
||||
### ✅ **Migration Patterns**
|
||||
|
||||
```typescript
|
||||
// If you have old v2.x code, here's how to migrate:
|
||||
|
||||
// OLD v2.x:
|
||||
// await brain.addNoun(text, type, metadata)
|
||||
// NEW v3.x:
|
||||
await brain.add({ data: text, type, metadata })
|
||||
|
||||
// OLD v2.x:
|
||||
// await brain.getNouns({ pagination: { limit: 100 } })
|
||||
// NEW v3.x:
|
||||
await brain.find({ limit: 100 })
|
||||
|
||||
// OLD v2.x:
|
||||
// await brain.addVerb(sourceId, targetId, verbType, metadata)
|
||||
// NEW v3.x:
|
||||
await brain.relate({ from: sourceId, to: targetId, type: verbType, metadata })
|
||||
|
||||
// OLD v2.x:
|
||||
// await brain.searchText(query)
|
||||
// NEW v3.x:
|
||||
await brain.find(query) // More powerful natural language search
|
||||
```
|
||||
|
||||
## 🚀 Performance Patterns
|
||||
|
||||
### ✅ **High-Performance Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Batch operations
|
||||
const entities = [/* large array */]
|
||||
const batchSize = 100
|
||||
|
||||
for (let i = 0; i < entities.length; i += batchSize) {
|
||||
const batch = entities.slice(i, i + batchSize)
|
||||
await Promise.all(
|
||||
batch.map(entity => brain.add(entity))
|
||||
)
|
||||
|
||||
// Optional: Rate limiting
|
||||
await new Promise(resolve => setTimeout(resolve, 100))
|
||||
}
|
||||
|
||||
// ✅ Pattern 2: Connection pooling for distributed
|
||||
const brain = new Brainy({
|
||||
distributed: true,
|
||||
connectionPool: {
|
||||
min: 5,
|
||||
max: 50,
|
||||
acquireTimeoutMillis: 30000
|
||||
}
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Efficient caching
|
||||
const brain = new Brainy({
|
||||
cache: {
|
||||
maxSize: 10000, // Number of items
|
||||
ttl: 300000, // 5 minutes
|
||||
updateAgeOnGet: true // LRU behavior
|
||||
}
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Memory-conscious operations
|
||||
const results = await brain.find({
|
||||
query: "large dataset query",
|
||||
limit: 1000, // Reasonable limit
|
||||
includeVectors: false // Exclude vectors if not needed
|
||||
})
|
||||
```
|
||||
|
||||
## 🛡️ Error Handling Patterns
|
||||
|
||||
### ✅ **Robust Error Handling**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Specific error handling
|
||||
try {
|
||||
const result = await brain.add({ data, type, metadata })
|
||||
return result
|
||||
} catch (error) {
|
||||
if (error.code === 'DUPLICATE_ENTITY') {
|
||||
console.log('Entity already exists, updating instead...')
|
||||
return await brain.update(error.existingId, { data, metadata })
|
||||
} else if (error.code === 'STORAGE_FULL') {
|
||||
throw new Error('Storage capacity exceeded')
|
||||
} else if (error.code === 'EMBEDDING_FAILED') {
|
||||
console.warn('Embedding failed, retrying with simpler text...')
|
||||
return await brain.add({
|
||||
data: data.substring(0, 1000), // Truncate
|
||||
type,
|
||||
metadata
|
||||
})
|
||||
}
|
||||
throw error
|
||||
}
|
||||
|
||||
// ✅ Pattern 2: Retry with exponential backoff
|
||||
async function resilientAdd(data: any, maxRetries = 3) {
|
||||
for (let attempt = 1; attempt <= maxRetries; attempt++) {
|
||||
try {
|
||||
return await brain.add(data)
|
||||
} catch (error) {
|
||||
if (attempt === maxRetries) throw error
|
||||
|
||||
const delay = Math.pow(2, attempt) * 1000
|
||||
console.warn(`Attempt ${attempt} failed, retrying in ${delay}ms...`)
|
||||
await new Promise(resolve => setTimeout(resolve, delay))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ✅ Pattern 3: Graceful degradation
|
||||
async function robustSearch(query: string) {
|
||||
try {
|
||||
// Try advanced semantic search first
|
||||
return await brain.find({
|
||||
like: query,
|
||||
threshold: 0.8,
|
||||
limit: 50
|
||||
})
|
||||
} catch (error) {
|
||||
console.warn('Semantic search failed, falling back to basic search:', error.message)
|
||||
|
||||
try {
|
||||
// Fallback to simple text search
|
||||
return await brain.find(query)
|
||||
} catch (fallbackError) {
|
||||
console.error('All search methods failed:', fallbackError.message)
|
||||
return [] // Return empty results rather than crash
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 Monitoring Patterns
|
||||
|
||||
### ✅ **Production Monitoring**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Performance monitoring
|
||||
const startTime = Date.now()
|
||||
const result = await brain.add(data)
|
||||
const duration = Date.now() - startTime
|
||||
|
||||
if (duration > 1000) {
|
||||
console.warn(`Slow add operation: ${duration}ms`)
|
||||
}
|
||||
|
||||
// ✅ Pattern 2: Health checks
|
||||
async function healthCheck() {
|
||||
try {
|
||||
// Test basic operations
|
||||
const testId = await brain.add({
|
||||
data: "health check",
|
||||
type: NounType.System,
|
||||
metadata: { test: true }
|
||||
})
|
||||
|
||||
await brain.get(testId)
|
||||
await brain.delete(testId)
|
||||
|
||||
return { status: 'healthy', timestamp: Date.now() }
|
||||
} catch (error) {
|
||||
return {
|
||||
status: 'unhealthy',
|
||||
error: error.message,
|
||||
timestamp: Date.now()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ✅ Pattern 3: Metrics collection
|
||||
class BrainyMetrics {
|
||||
private metrics = {
|
||||
operations: 0,
|
||||
errors: 0,
|
||||
totalTime: 0
|
||||
}
|
||||
|
||||
async timedOperation<T>(operation: () => Promise<T>): Promise<T> {
|
||||
const start = Date.now()
|
||||
try {
|
||||
const result = await operation()
|
||||
this.metrics.operations++
|
||||
this.metrics.totalTime += Date.now() - start
|
||||
return result
|
||||
} catch (error) {
|
||||
this.metrics.errors++
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
getStats() {
|
||||
return {
|
||||
...this.metrics,
|
||||
avgTime: this.metrics.totalTime / this.metrics.operations || 0,
|
||||
errorRate: this.metrics.errors / this.metrics.operations || 0
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🎯 Summary: Modern Brainy v3.x Best Practices
|
||||
|
||||
| ❌ **Avoid v2.x** | ✅ **Use v3.x** |
|
||||
|------------------|----------------|
|
||||
| `addNoun()` | `add()` |
|
||||
| `getNouns()` | `find()` |
|
||||
| `addVerb()` | `relate()` |
|
||||
| `getVerbs()` | `getRelations()` |
|
||||
| `deleteNoun()` | `delete()` |
|
||||
| Complex configs | Zero-config with `new Brainy()` |
|
||||
| Manual pagination | Built-in smart pagination |
|
||||
| String-based search | Natural language queries |
|
||||
|
||||
---
|
||||
|
||||
**🎉 Following these patterns gives you:**
|
||||
- 🚀 **Modern APIs** that are actively maintained
|
||||
- ⚡ **Better performance** with intelligent defaults
|
||||
- 🛡️ **Robust error handling** with specific error types
|
||||
- 📈 **Scalable patterns** for production applications
|
||||
- 🧠 **Natural language** search capabilities
|
||||
|
||||
**Next:** [Neural API Patterns →](./NEURAL_API_PATTERNS.md) | [VFS Patterns →](./vfs/COMMON_PATTERNS.md)
|
||||
205
docs/DATA_MODEL.md
Normal file
205
docs/DATA_MODEL.md
Normal file
|
|
@ -0,0 +1,205 @@
|
|||
# Data Model
|
||||
|
||||
> How Brainy stores entities and relationships, and the critical distinction between `data` and `metadata`.
|
||||
|
||||
---
|
||||
|
||||
## Entity (Noun)
|
||||
|
||||
An entity is the fundamental data unit in Brainy. Every entity has:
|
||||
|
||||
| Field | Type | Indexed | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `id` | `string` | Primary key | UUID v4 (auto-generated or custom) |
|
||||
| `data` | `any` | **HNSW vector index** | Content used for semantic/hybrid search. Strings auto-embed. |
|
||||
| `metadata` | `object` | **MetadataIndex** | Structured queryable fields (tags, dates, flags, etc.) |
|
||||
| `type` | `NounType` | MetadataIndex (as `noun`) | Entity type classification |
|
||||
| `vector` | `number[]` | HNSW | 384-dim embedding (auto-computed from `data` or user-provided) |
|
||||
| `confidence` | `number` | MetadataIndex | Type classification confidence (0-1) |
|
||||
| `weight` | `number` | MetadataIndex | Entity importance/salience (0-1) |
|
||||
| `service` | `string` | MetadataIndex | Multi-tenancy identifier |
|
||||
| `createdAt` | `number` | MetadataIndex | Creation timestamp (ms since epoch) |
|
||||
| `updatedAt` | `number` | MetadataIndex | Last update timestamp (ms since epoch) |
|
||||
| `createdBy` | `object` | MetadataIndex | Source augmentation info |
|
||||
|
||||
### Example
|
||||
|
||||
```typescript
|
||||
const id = await brain.add({
|
||||
data: 'John Smith is a software engineer at Acme Corp', // → embedded into vector
|
||||
type: NounType.Person,
|
||||
metadata: { // → indexed, queryable via where filters
|
||||
role: 'engineer',
|
||||
department: 'backend',
|
||||
yearsExperience: 8
|
||||
},
|
||||
confidence: 0.95,
|
||||
weight: 0.7
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Relationship (Verb)
|
||||
|
||||
A relationship is a typed, directed edge connecting two entities.
|
||||
|
||||
| Field | Type | Indexed | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `id` | `string` | Primary key | UUID v4 (auto-generated) |
|
||||
| `from` | `string` | **GraphAdjacencyIndex** | Source entity ID |
|
||||
| `to` | `string` | **GraphAdjacencyIndex** | Target entity ID |
|
||||
| `type` | `VerbType` | GraphAdjacencyIndex (as `verb`) | Relationship type classification |
|
||||
| `data` | `any` | — | Opaque content (overrides auto-computed vector if provided) |
|
||||
| `metadata` | `object` | — | Structured fields on the edge |
|
||||
| `weight` | `number` | — | Connection strength (0-1, default: 1.0) |
|
||||
| `confidence` | `number` | — | Relationship certainty (0-1) |
|
||||
| `evidence` | `RelationEvidence` | — | Why this relationship was detected |
|
||||
| `createdAt` | `number` | — | Creation timestamp (ms since epoch) |
|
||||
| `updatedAt` | `number` | — | Last update timestamp (ms since epoch) |
|
||||
| `service` | `string` | — | Multi-tenancy identifier |
|
||||
|
||||
### Example
|
||||
|
||||
```typescript
|
||||
const relId = await brain.relate({
|
||||
from: personId,
|
||||
to: projectId,
|
||||
type: VerbType.WorksOn,
|
||||
data: 'Lead engineer on the AI module', // Optional: content for this edge
|
||||
metadata: { // Optional: queryable edge fields
|
||||
role: 'lead',
|
||||
startDate: '2024-01-15'
|
||||
},
|
||||
weight: 0.9
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data vs Metadata
|
||||
|
||||
This is the most important concept in Brainy's storage model:
|
||||
|
||||
### `data` — Content for Semantic Search
|
||||
|
||||
- Embedded into a 384-dimensional vector via the WASM embedding engine
|
||||
- Searchable via **semantic similarity** (HNSW vector index) and **hybrid text+semantic** search
|
||||
- Queried by passing `query` to `find()`:
|
||||
```typescript
|
||||
brain.find({ query: 'machine learning algorithms' })
|
||||
```
|
||||
- **NOT** indexed by MetadataIndex — you cannot use `where` filters on `data`
|
||||
- Stored opaquely: strings, objects, numbers — anything goes
|
||||
|
||||
### `metadata` — Structured Queryable Fields
|
||||
|
||||
- Indexed by MetadataIndex with O(1) lookups per field
|
||||
- Queryable via `where` filters using [BFO operators](./QUERY_OPERATORS.md):
|
||||
```typescript
|
||||
brain.find({
|
||||
where: {
|
||||
department: 'engineering',
|
||||
yearsExperience: { greaterThan: 5 },
|
||||
tags: { contains: 'senior' }
|
||||
}
|
||||
})
|
||||
```
|
||||
- **NOT** used for vector/semantic search
|
||||
- Must be a flat or lightly nested object
|
||||
|
||||
### Quick Reference
|
||||
|
||||
| | `data` | `metadata` |
|
||||
|---|---|---|
|
||||
| **Purpose** | Content for embedding / semantic search | Structured fields for filtering |
|
||||
| **Searched by** | `find({ query })` — vector similarity, hybrid text+semantic | `find({ where })` — exact, range, set operators |
|
||||
| **Indexed by** | HNSW vector index | MetadataIndex |
|
||||
| **Queryable with operators?** | No | Yes (`equals`, `greaterThan`, `oneOf`, etc.) |
|
||||
| **Auto-embedded?** | Yes (strings → 384-dim vectors) | No |
|
||||
| **Typical content** | Text descriptions, document content | Tags, dates, status flags, categories, numeric fields |
|
||||
|
||||
### Common Pattern
|
||||
|
||||
```typescript
|
||||
// Add an article
|
||||
await brain.add({
|
||||
data: 'A deep dive into transformer architectures and attention mechanisms',
|
||||
type: NounType.Document,
|
||||
metadata: {
|
||||
title: 'Transformer Deep Dive',
|
||||
author: 'Dr. Chen',
|
||||
publishedYear: 2024,
|
||||
tags: ['AI', 'transformers', 'NLP'],
|
||||
status: 'published'
|
||||
}
|
||||
})
|
||||
|
||||
// Search by content (semantic — searches data)
|
||||
const results = await brain.find({ query: 'neural network attention' })
|
||||
|
||||
// Filter by fields (exact — queries metadata)
|
||||
const recent = await brain.find({
|
||||
where: {
|
||||
publishedYear: { greaterThan: 2023 },
|
||||
status: 'published'
|
||||
}
|
||||
})
|
||||
|
||||
// Combine both (Triple Intelligence)
|
||||
const precise = await brain.find({
|
||||
query: 'attention mechanisms', // Semantic search on data
|
||||
where: { author: 'Dr. Chen' }, // Metadata filter
|
||||
connected: { from: authorId, depth: 1 } // Graph traversal
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Storage Field Naming
|
||||
|
||||
Internally, Brainy uses different field names in storage vs the public API:
|
||||
|
||||
| Public API (Entity/Relation) | Storage (metadata object) | Notes |
|
||||
|------------------------------|--------------------------|-------|
|
||||
| `type` | `noun` | Entity type stored as `noun` |
|
||||
| `from` | `sourceId` | Relationship source |
|
||||
| `to` | `targetId` | Relationship target |
|
||||
| `type` (on Relation) | `verb` | Relationship type stored as `verb` |
|
||||
|
||||
When querying with `find()`, you can use:
|
||||
- `type` parameter (convenience alias, equivalent to `where.noun`)
|
||||
- `where.noun` directly
|
||||
|
||||
```typescript
|
||||
// These are equivalent:
|
||||
brain.find({ type: NounType.Person })
|
||||
brain.find({ where: { noun: NounType.Person } })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Standard Metadata Fields
|
||||
|
||||
When you add an entity, Brainy stores these standard fields in the metadata object alongside your custom fields:
|
||||
|
||||
| Field | Set By | Description |
|
||||
|-------|--------|-------------|
|
||||
| `noun` | System | Entity type (NounType enum value) |
|
||||
| `data` | System | The raw `data` value (stored opaquely) |
|
||||
| `createdAt` | System | Creation timestamp |
|
||||
| `updatedAt` | System | Last update timestamp |
|
||||
| `confidence` | User | Type classification confidence |
|
||||
| `weight` | User | Entity importance |
|
||||
| `service` | User | Multi-tenancy identifier |
|
||||
| `createdBy` | User/System | Source augmentation |
|
||||
|
||||
On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [API Reference](./api/README.md) — Complete API documentation
|
||||
- [Query Operators](./QUERY_OPERATORS.md) — All BFO operators with examples
|
||||
- [Find System](./FIND_SYSTEM.md) — Natural language find() details
|
||||
|
|
@ -1220,7 +1220,7 @@ const brain = new Brainy({ verbose: true })
|
|||
|
||||
#### Resources
|
||||
|
||||
- 📚 [API Reference](../API_REFERENCE.md) - Complete API documentation
|
||||
- 📚 [API Reference](../api/README.md) - Complete API documentation
|
||||
- 📁 [VFS Guide](../vfs/VFS_API_GUIDE.md) - Virtual Filesystem deep dive
|
||||
- 🤖 [Neural API](../guides/neural-api.md) - Advanced neural operations
|
||||
- 🌐 [Distributed Guide](../guides/distributed-system.md) - Planet-scale architecture
|
||||
|
|
|
|||
|
|
@ -1,177 +0,0 @@
|
|||
# Metadata Contract Implementation Plan
|
||||
|
||||
## New Required Interface
|
||||
|
||||
```typescript
|
||||
export interface BrainyAugmentation {
|
||||
// Identity
|
||||
name: string
|
||||
timing: 'before' | 'after' | 'around' | 'replace'
|
||||
operations: string[]
|
||||
priority: number
|
||||
|
||||
// REQUIRED metadata contract
|
||||
metadata: 'none' | 'readonly' | MetadataAccess
|
||||
|
||||
// Methods
|
||||
initialize(context: AugmentationContext): Promise<void>
|
||||
execute<T = any>(operation: string, params: any, next: () => Promise<T>): Promise<T>
|
||||
shouldExecute?(operation: string, params: any): boolean
|
||||
shutdown?(): Promise<void>
|
||||
}
|
||||
|
||||
interface MetadataAccess {
|
||||
reads?: string[] | '*' // Fields to read, or '*' for all
|
||||
writes?: string[] | '*' // Fields to write, or '*' for all
|
||||
namespace?: string // Optional: custom namespace like '_myAug'
|
||||
}
|
||||
```
|
||||
|
||||
## Augmentation Analysis & Classification
|
||||
|
||||
### Category 1: No Metadata Access ('none')
|
||||
These augmentations don't read or write metadata at all:
|
||||
|
||||
1. **CacheAugmentation** - Only caches search results
|
||||
2. **RequestDeduplicatorAugmentation** - Only deduplicates requests
|
||||
3. **ConnectionPoolAugmentation** - Only manages storage connections
|
||||
4. **StorageAugmentation** - Base storage layer, metadata handled by Brainy
|
||||
|
||||
### Category 2: Read-Only Access ('readonly')
|
||||
These augmentations read metadata but never modify it:
|
||||
|
||||
6. **IndexAugmentation** - Reads metadata to build indexes
|
||||
7. **MonitoringAugmentation** - Reads metadata for monitoring
|
||||
8. **MetricsAugmentation** - Reads metadata for metrics collection
|
||||
9. **BatchProcessingAugmentation** - Reads metadata to check for external IDs
|
||||
10. **EntityRegistryAugmentation** - Reads metadata to register entities
|
||||
11. **AutoRegisterEntitiesAugmentation** - Reads metadata for auto-registration
|
||||
12. **ConduitAugmentation** - Reads metadata to pass through operations
|
||||
|
||||
### Category 3: Metadata Writers (needs specific access)
|
||||
These augmentations modify metadata and need specific field declarations:
|
||||
|
||||
13. **SynapseAugmentation** - Writes to '_synapse' field
|
||||
```typescript
|
||||
metadata: {
|
||||
reads: '*',
|
||||
writes: ['_synapse', '_synapseTimestamp'],
|
||||
namespace: '_synapse' // Uses its own namespace
|
||||
}
|
||||
```
|
||||
|
||||
14. **IntelligentVerbScoringAugmentation** - Adds scoring to verbs
|
||||
```typescript
|
||||
metadata: {
|
||||
reads: ['type', 'verb', 'source', 'target'],
|
||||
writes: ['weight', 'confidence', 'intelligentScoring']
|
||||
}
|
||||
```
|
||||
|
||||
15. **ServerSearchAugmentation** - Might add server metadata
|
||||
```typescript
|
||||
metadata: {
|
||||
reads: '*',
|
||||
writes: ['_server', '_syncedAt']
|
||||
}
|
||||
```
|
||||
|
||||
16. **NeuralImportAugmentation** - Enriches imported data
|
||||
```typescript
|
||||
metadata: {
|
||||
reads: '*',
|
||||
writes: ['nounType', 'verbType', '_importedAt', '_enriched']
|
||||
}
|
||||
```
|
||||
|
||||
### Category 4: API/Server (needs analysis)
|
||||
17. **ApiServerAugmentation** - Likely read-only for serving data
|
||||
18. **StorageAugmentations** (plural) - Collection of storage implementations
|
||||
19. **ConduitAugmentations** (plural) - Collection of conduit types
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Update Base Interface
|
||||
1. Update `BrainyAugmentation` interface to require `metadata` field
|
||||
2. Update `BaseAugmentation` class to have abstract `metadata` property
|
||||
3. Add runtime enforcement in augmentation executor
|
||||
|
||||
### Phase 2: Update Each Augmentation
|
||||
For each augmentation, add the appropriate metadata declaration:
|
||||
|
||||
#### Example Updates:
|
||||
|
||||
**CacheAugmentation:**
|
||||
```typescript
|
||||
export class CacheAugmentation extends BaseAugmentation {
|
||||
readonly name = 'cache'
|
||||
readonly metadata = 'none' as const // ✅ No metadata access
|
||||
// ... rest unchanged
|
||||
}
|
||||
```
|
||||
|
||||
```typescript
|
||||
readonly metadata = 'readonly' as const // ✅ Only reads for logging
|
||||
// ... rest unchanged
|
||||
}
|
||||
```
|
||||
|
||||
**SynapseAugmentation:**
|
||||
```typescript
|
||||
export abstract class SynapseAugmentation extends BaseAugmentation {
|
||||
readonly name = 'synapse'
|
||||
readonly metadata = {
|
||||
reads: '*',
|
||||
writes: ['_synapse', '_synapseTimestamp'],
|
||||
namespace: '_synapse'
|
||||
} as const
|
||||
// ... rest unchanged
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 3: Runtime Enforcement
|
||||
Add a metadata access enforcer that:
|
||||
1. Wraps metadata objects based on declared access
|
||||
2. Throws errors if augmentation violates its contract
|
||||
3. Logs warnings in development mode
|
||||
|
||||
```typescript
|
||||
class MetadataEnforcer {
|
||||
enforce(augmentation: BrainyAugmentation, metadata: any): any {
|
||||
if (augmentation.metadata === 'none') {
|
||||
return null // No access at all
|
||||
}
|
||||
|
||||
if (augmentation.metadata === 'readonly') {
|
||||
return Object.freeze(deepClone(metadata)) // Read-only copy
|
||||
}
|
||||
|
||||
// For specific access, create proxy that validates
|
||||
return new Proxy(metadata, {
|
||||
set(target, prop, value) {
|
||||
const access = augmentation.metadata as MetadataAccess
|
||||
if (!access.writes?.includes(String(prop)) && access.writes !== '*') {
|
||||
throw new Error(`Augmentation '${augmentation.name}' cannot write to field '${String(prop)}'`)
|
||||
}
|
||||
target[prop] = value
|
||||
return true
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Benefits
|
||||
1. **Type Safety** - TypeScript enforces metadata declaration
|
||||
2. **Runtime Safety** - Violations caught immediately
|
||||
3. **Documentation** - Contract shows exactly what each augmentation does
|
||||
4. **Brain-cloud Ready** - Registry can validate augmentations
|
||||
5. **Developer Friendly** - Most use simple 'none' or 'readonly'
|
||||
|
||||
## Migration Checklist
|
||||
- [ ] Update BrainyAugmentation interface
|
||||
- [ ] Update BaseAugmentation class
|
||||
- [ ] Add MetadataEnforcer
|
||||
- [ ] Update all 19 augmentations with metadata declarations
|
||||
- [ ] Add tests for metadata enforcement
|
||||
- [ ] Update documentation
|
||||
|
|
@ -1,736 +0,0 @@
|
|||
# 🧠 Neural API Patterns: AI-Powered Intelligence
|
||||
|
||||
> Learn the correct patterns for Brainy's Neural API. Avoid performance pitfalls and use AI features effectively.
|
||||
|
||||
## 🚨 Critical: Access Neural APIs Correctly
|
||||
|
||||
### ❌ **WRONG - Outdated Access Patterns**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Outdated documentation patterns
|
||||
import { Brainy } from '@soulcraft/brainy' // ❌ Wrong import
|
||||
const brain = new Brainy() // ❌ Old class name
|
||||
|
||||
// These may not work as expected:
|
||||
const neural = brain.neural // ❌ May be undefined
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Modern Neural Access**
|
||||
|
||||
```typescript
|
||||
// ✅ Use modern Brainy class
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// ✅ Neural API is available after initialization
|
||||
const clusters = await brain.neural.clusters()
|
||||
const similarity = await brain.neural.similar('item1', 'item2')
|
||||
```
|
||||
|
||||
## 🔍 Similarity Analysis Patterns
|
||||
|
||||
### ❌ **WRONG - Inefficient Similarity Checks**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - N² comparisons
|
||||
const items = await brain.find({ limit: 1000 })
|
||||
const similarities = []
|
||||
|
||||
for (const item1 of items) {
|
||||
for (const item2 of items) {
|
||||
if (item1.id !== item2.id) {
|
||||
const sim = await brain.neural.similar(item1.id, item2.id) // ❌ Millions of calls
|
||||
similarities.push({ from: item1.id, to: item2.id, score: sim })
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Efficient Similarity Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Find neighbors (much more efficient)
|
||||
const item = await brain.get('target-item-id')
|
||||
const neighbors = await brain.neural.neighbors(item.id, {
|
||||
limit: 10, // Top 10 most similar
|
||||
threshold: 0.7, // Minimum similarity
|
||||
includeScores: true // Include similarity scores
|
||||
})
|
||||
|
||||
console.log(`Found ${neighbors.length} similar items`)
|
||||
|
||||
// ✅ Pattern 2: Batch similarity for specific pairs
|
||||
const itemPairs = [
|
||||
['item1', 'item2'],
|
||||
['item1', 'item3'],
|
||||
['item2', 'item3']
|
||||
]
|
||||
|
||||
const similarities = await Promise.all(
|
||||
itemPairs.map(async ([a, b]) => ({
|
||||
from: a,
|
||||
to: b,
|
||||
score: await brain.neural.similar(a, b)
|
||||
}))
|
||||
)
|
||||
|
||||
// ✅ Pattern 3: Text-to-text similarity (no need for IDs)
|
||||
const textSimilarity = await brain.neural.similar(
|
||||
"Machine learning is fascinating",
|
||||
"AI and deep learning are interesting",
|
||||
{ detailed: true } // Get explanation of similarity
|
||||
)
|
||||
|
||||
console.log(`Similarity: ${textSimilarity.score}`)
|
||||
console.log(`Explanation: ${textSimilarity.explanation}`)
|
||||
|
||||
// ✅ Pattern 4: Vector-level similarity for optimization
|
||||
const vector1 = await brain.embed("First concept")
|
||||
const vector2 = await brain.embed("Second concept")
|
||||
const vectorSimilarity = await brain.neural.similar(vector1, vector2)
|
||||
```
|
||||
|
||||
## 🎯 Clustering Patterns
|
||||
|
||||
### ❌ **WRONG - Uncontrolled Clustering**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Clustering everything without limits
|
||||
const everything = await brain.find({ limit: 100000 }) // ❌ Too much data
|
||||
const clusters = await brain.neural.clusters() // ❌ May crash or timeout
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Smart Clustering Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Controlled clustering with limits
|
||||
const recentItems = await brain.find({
|
||||
where: {
|
||||
createdAt: { $gte: Date.now() - 30 * 24 * 60 * 60 * 1000 } // Last 30 days
|
||||
},
|
||||
limit: 1000 // Reasonable limit
|
||||
})
|
||||
|
||||
const clusters = await brain.neural.clusters(
|
||||
recentItems.map(item => item.id),
|
||||
{
|
||||
algorithm: 'kmeans', // Reliable algorithm
|
||||
maxClusters: 10, // Reasonable number
|
||||
threshold: 0.75, // High similarity required
|
||||
iterations: 50 // Convergence limit
|
||||
}
|
||||
)
|
||||
|
||||
// ✅ Pattern 2: Domain-specific clustering
|
||||
const techDocs = await brain.find({
|
||||
where: { category: 'technology', type: 'document' },
|
||||
limit: 500
|
||||
})
|
||||
|
||||
const techClusters = await brain.neural.clusterByDomain(
|
||||
'category', // Group by this field
|
||||
{
|
||||
items: techDocs.map(doc => doc.id),
|
||||
minClusterSize: 3, // Minimum items per cluster
|
||||
maxClusters: 8
|
||||
}
|
||||
)
|
||||
|
||||
// ✅ Pattern 3: Temporal clustering for time-series data
|
||||
const timebasedClusters = await brain.neural.clusterByTime(
|
||||
'createdAt', // Time field
|
||||
'week', // Time window (hour, day, week, month)
|
||||
{
|
||||
items: recentItems.map(item => item.id),
|
||||
overlap: 0.2, // 20% overlap between windows
|
||||
minPerWindow: 5 // Minimum items per time window
|
||||
}
|
||||
)
|
||||
|
||||
// ✅ Pattern 4: Streaming clustering for large datasets
|
||||
async function clusterLargeDataset() {
|
||||
const clusterStream = brain.neural.clusterStream({
|
||||
batchSize: 100, // Process 100 items at a time
|
||||
updateInterval: 1000, // Update clusters every 1000 items
|
||||
maxMemory: 512 * 1024 * 1024 // 512MB memory limit
|
||||
})
|
||||
|
||||
const allClusters = []
|
||||
for await (const batch of clusterStream) {
|
||||
console.log(`Processed ${batch.processed} items, found ${batch.clusters.length} clusters`)
|
||||
allClusters.push(...batch.clusters)
|
||||
}
|
||||
|
||||
return allClusters
|
||||
}
|
||||
```
|
||||
|
||||
## 🔍 Neighbor Discovery Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Similarity Searches**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Reinventing neighbor search
|
||||
async function findSimilarManually(targetId: string) {
|
||||
const allItems = await brain.find({ limit: 10000 }) // ❌ Load everything
|
||||
const similarities = []
|
||||
|
||||
for (const item of allItems) {
|
||||
if (item.id !== targetId) {
|
||||
const score = await brain.neural.similar(targetId, item.id) // ❌ Slow
|
||||
if (score > 0.7) {
|
||||
similarities.push({ id: item.id, score })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return similarities.sort((a, b) => b.score - a.score).slice(0, 10) // ❌ Inefficient
|
||||
}
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Optimized Neighbor Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Basic neighbor search
|
||||
const neighbors = await brain.neural.neighbors('target-item-id', {
|
||||
limit: 20, // Top 20 neighbors
|
||||
threshold: 0.6, // Minimum similarity
|
||||
includeMetadata: true, // Include item metadata
|
||||
includeDistances: true // Include exact similarity scores
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Filtered neighbor search
|
||||
const filteredNeighbors = await brain.neural.neighbors('article-id', {
|
||||
limit: 10,
|
||||
filter: {
|
||||
type: 'document', // Only find similar documents
|
||||
status: 'published', // Only published content
|
||||
language: 'en' // Only English content
|
||||
},
|
||||
excludeIds: ['self-id', 'duplicate-id'] // Exclude specific items
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Multi-level neighbor discovery
|
||||
async function discoverNeighborNetwork(rootId: string, maxDepth = 2) {
|
||||
const network = new Map()
|
||||
const visited = new Set()
|
||||
const queue = [{ id: rootId, depth: 0 }]
|
||||
|
||||
while (queue.length > 0) {
|
||||
const { id, depth } = queue.shift()!
|
||||
|
||||
if (visited.has(id) || depth >= maxDepth) continue
|
||||
visited.add(id)
|
||||
|
||||
const neighbors = await brain.neural.neighbors(id, {
|
||||
limit: 5,
|
||||
threshold: 0.8
|
||||
})
|
||||
|
||||
network.set(id, neighbors)
|
||||
|
||||
// Add neighbors to queue for next depth level
|
||||
if (depth < maxDepth - 1) {
|
||||
neighbors.forEach(neighbor => {
|
||||
queue.push({ id: neighbor.id, depth: depth + 1 })
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return network
|
||||
}
|
||||
|
||||
// ✅ Pattern 4: Recommendation engine
|
||||
async function getRecommendations(userId: string) {
|
||||
// Get user's liked items
|
||||
const userItems = await brain.find({
|
||||
connected: { to: userId, via: 'liked-by' }
|
||||
})
|
||||
|
||||
// Find neighbors for each liked item
|
||||
const allNeighbors = await Promise.all(
|
||||
userItems.map(item =>
|
||||
brain.neural.neighbors(item.id, {
|
||||
limit: 10,
|
||||
threshold: 0.7,
|
||||
excludeConnected: { to: userId, via: 'liked-by' } // Exclude already liked
|
||||
})
|
||||
)
|
||||
)
|
||||
|
||||
// Aggregate and rank recommendations
|
||||
const recommendations = new Map()
|
||||
allNeighbors.flat().forEach(neighbor => {
|
||||
const current = recommendations.get(neighbor.id) || { score: 0, count: 0 }
|
||||
current.score += neighbor.score
|
||||
current.count += 1
|
||||
recommendations.set(neighbor.id, current)
|
||||
})
|
||||
|
||||
// Return top recommendations by average score
|
||||
return Array.from(recommendations.entries())
|
||||
.map(([id, stats]) => ({
|
||||
id,
|
||||
avgScore: stats.score / stats.count,
|
||||
mentions: stats.count
|
||||
}))
|
||||
.sort((a, b) => b.avgScore - a.avgScore)
|
||||
.slice(0, 10)
|
||||
}
|
||||
```
|
||||
|
||||
## 🏗️ Hierarchy & Structure Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Hierarchy Building**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Building hierarchies manually
|
||||
async function buildHierarchyManually(rootId: string) {
|
||||
const root = await brain.get(rootId)
|
||||
const allItems = await brain.find({ limit: 1000 }) // ❌ Load everything
|
||||
|
||||
// Manual tree building with nested loops
|
||||
const hierarchy = { root, children: [] }
|
||||
// ... complex manual logic
|
||||
}
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - Semantic Hierarchy Patterns**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Automatic semantic hierarchy
|
||||
const hierarchy = await brain.neural.hierarchy('root-concept-id', {
|
||||
maxDepth: 4, // Maximum tree depth
|
||||
minSimilarity: 0.6, // Minimum similarity for inclusion
|
||||
branchingFactor: 5, // Maximum children per node
|
||||
algorithm: 'semantic' // Use semantic clustering
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Domain-specific hierarchy
|
||||
const techHierarchy = await brain.neural.hierarchy('technology-id', {
|
||||
filter: { category: 'technology' },
|
||||
weights: {
|
||||
semantic: 0.7, // 70% based on content similarity
|
||||
metadata: 0.3 // 30% based on metadata similarity
|
||||
},
|
||||
includeMetrics: true // Include hierarchy quality metrics
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Multi-root hierarchy for complex domains
|
||||
async function buildMultiRootHierarchy(rootIds: string[]) {
|
||||
const hierarchies = await Promise.all(
|
||||
rootIds.map(rootId =>
|
||||
brain.neural.hierarchy(rootId, {
|
||||
maxDepth: 3,
|
||||
crossReference: true // Allow cross-hierarchy connections
|
||||
})
|
||||
)
|
||||
)
|
||||
|
||||
// Merge hierarchies and find connections
|
||||
const merged = {
|
||||
roots: hierarchies,
|
||||
connections: await findCrossHierarchyConnections(hierarchies)
|
||||
}
|
||||
|
||||
return merged
|
||||
}
|
||||
|
||||
async function findCrossHierarchyConnections(hierarchies: any[]) {
|
||||
const connections = []
|
||||
|
||||
for (let i = 0; i < hierarchies.length; i++) {
|
||||
for (let j = i + 1; j < hierarchies.length; j++) {
|
||||
const leafNodes1 = extractLeafNodes(hierarchies[i])
|
||||
const leafNodes2 = extractLeafNodes(hierarchies[j])
|
||||
|
||||
// Find connections between leaf nodes of different hierarchies
|
||||
for (const leaf1 of leafNodes1) {
|
||||
const neighbors = await brain.neural.neighbors(leaf1.id, {
|
||||
limit: 5,
|
||||
threshold: 0.8,
|
||||
filter: { id: { $in: leafNodes2.map(l => l.id) } }
|
||||
})
|
||||
|
||||
connections.push(...neighbors.map(n => ({
|
||||
from: leaf1.id,
|
||||
to: n.id,
|
||||
hierarchyPair: [i, j],
|
||||
similarity: n.score
|
||||
})))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return connections
|
||||
}
|
||||
```
|
||||
|
||||
## 🚨 Outlier Detection Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Outlier Detection**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Manual statistical outlier detection
|
||||
async function findOutliersManually() {
|
||||
const items = await brain.find({ limit: 1000 })
|
||||
const similarities = []
|
||||
|
||||
// Calculate average similarity for each item (expensive)
|
||||
for (const item of items) {
|
||||
let totalSim = 0
|
||||
let count = 0
|
||||
for (const other of items) {
|
||||
if (item.id !== other.id) {
|
||||
totalSim += await brain.neural.similar(item.id, other.id) // ❌ N² operations
|
||||
count++
|
||||
}
|
||||
}
|
||||
similarities.push({ id: item.id, avgSimilarity: totalSim / count })
|
||||
}
|
||||
|
||||
// Manual outlier calculation
|
||||
const threshold = calculateManualThreshold(similarities) // ❌ Complex statistics
|
||||
return similarities.filter(s => s.avgSimilarity < threshold)
|
||||
}
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - AI-Powered Outlier Detection**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Automatic outlier detection
|
||||
const outliers = await brain.neural.outliers({
|
||||
threshold: 0.3, // Items with < 30% avg similarity to others
|
||||
method: 'isolation-forest', // AI-based outlier detection
|
||||
contamination: 0.1, // Expect ~10% outliers
|
||||
includeReasons: true // Explain why each item is an outlier
|
||||
})
|
||||
|
||||
console.log(`Found ${outliers.length} outliers`)
|
||||
outliers.forEach(outlier => {
|
||||
console.log(`Outlier: ${outlier.id}, Score: ${outlier.score}`)
|
||||
console.log(`Reason: ${outlier.reason}`)
|
||||
})
|
||||
|
||||
// ✅ Pattern 2: Domain-specific outlier detection
|
||||
const techOutliers = await brain.neural.outliers({
|
||||
filter: { category: 'technology' },
|
||||
features: ['content', 'metadata.tags', 'metadata.complexity'],
|
||||
method: 'local-outlier-factor',
|
||||
neighbors: 20 // Consider 20 nearest neighbors
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Temporal outlier detection
|
||||
const recentOutliers = await brain.neural.outliers({
|
||||
timeWindow: '7days', // Look at last 7 days
|
||||
baseline: '30days', // Compare to 30-day baseline
|
||||
method: 'statistical', // Use statistical methods
|
||||
autoThreshold: true // Automatically determine threshold
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Streaming outlier detection
|
||||
async function detectOutliersInStream() {
|
||||
const outlierStream = brain.neural.outlierStream({
|
||||
batchSize: 50,
|
||||
updateInterval: 100, // Check every 100 new items
|
||||
adaptiveThreshold: true // Threshold adapts as data changes
|
||||
})
|
||||
|
||||
for await (const batch of outlierStream) {
|
||||
console.log(`Batch ${batch.batchNumber}: ${batch.outliers.length} outliers detected`)
|
||||
|
||||
// Process outliers immediately
|
||||
for (const outlier of batch.outliers) {
|
||||
await handleOutlier(outlier)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function handleOutlier(outlier: any) {
|
||||
// Flag for manual review
|
||||
await brain.update(outlier.id, {
|
||||
metadata: {
|
||||
flagged: true,
|
||||
outlierScore: outlier.score,
|
||||
outlierReason: outlier.reason,
|
||||
flaggedAt: Date.now()
|
||||
}
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 Visualization Patterns
|
||||
|
||||
### ❌ **WRONG - Manual Visualization Data Preparation**
|
||||
|
||||
```typescript
|
||||
// DON'T DO THIS - Manual coordinate calculation
|
||||
async function prepareVisualizationManually() {
|
||||
const items = await brain.find({ limit: 500 })
|
||||
const coordinates = []
|
||||
|
||||
// Manual dimensionality reduction (complex math)
|
||||
for (const item of items) {
|
||||
const vector = await brain.embed(item.data)
|
||||
// Complex PCA/t-SNE calculations manually
|
||||
const x = complexMathFunction(vector) // ❌ Error-prone
|
||||
const y = anotherComplexFunction(vector)
|
||||
coordinates.push({ id: item.id, x, y })
|
||||
}
|
||||
|
||||
return coordinates
|
||||
}
|
||||
```
|
||||
|
||||
### ✅ **CORRECT - AI-Powered Visualization**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Automatic 2D visualization
|
||||
const visualization = await brain.neural.visualize({
|
||||
dimensions: 2, // 2D plot
|
||||
algorithm: 'umap', // UMAP for better clustering preservation
|
||||
maxItems: 1000, // Performance limit
|
||||
includeMetadata: true, // Include item metadata in output
|
||||
colorBy: 'cluster' // Color points by cluster membership
|
||||
})
|
||||
|
||||
// Result format:
|
||||
// {
|
||||
// points: [{ id, x, y, cluster, metadata }, ...],
|
||||
// clusters: [{ id, centroid: [x, y], members: [...] }, ...],
|
||||
// stats: { stress, kruskalStress, trustworthiness }
|
||||
// }
|
||||
|
||||
// ✅ Pattern 2: 3D visualization for complex data
|
||||
const viz3D = await brain.neural.visualize({
|
||||
dimensions: 3,
|
||||
algorithm: 'tsne',
|
||||
perplexity: 30, // t-SNE parameter
|
||||
learningRate: 200, // t-SNE learning rate
|
||||
iterations: 1000 // Number of optimization steps
|
||||
})
|
||||
|
||||
// ✅ Pattern 3: Interactive visualization with filtering
|
||||
const interactiveViz = await brain.neural.visualize({
|
||||
filter: {
|
||||
type: 'document',
|
||||
createdAt: { $gte: Date.now() - 7 * 24 * 60 * 60 * 1000 }
|
||||
},
|
||||
groupBy: 'category', // Group points by metadata field
|
||||
showLabels: true, // Include text labels
|
||||
labelField: 'title', // Field to use for labels
|
||||
includeEdges: true, // Show connections between similar items
|
||||
edgeThreshold: 0.8 // Only show high-similarity connections
|
||||
})
|
||||
|
||||
// ✅ Pattern 4: Real-time visualization updates
|
||||
class LiveVisualization {
|
||||
private visualization: any = null
|
||||
private updateInterval: NodeJS.Timeout | null = null
|
||||
|
||||
async start() {
|
||||
// Initial visualization
|
||||
this.visualization = await brain.neural.visualize({
|
||||
dimensions: 2,
|
||||
algorithm: 'umap',
|
||||
maxItems: 500,
|
||||
includeMetadata: true
|
||||
})
|
||||
|
||||
// Update every 30 seconds
|
||||
this.updateInterval = setInterval(async () => {
|
||||
await this.update()
|
||||
}, 30000)
|
||||
}
|
||||
|
||||
async update() {
|
||||
// Get recent items
|
||||
const recentItems = await brain.find({
|
||||
where: {
|
||||
createdAt: { $gte: Date.now() - 30000 } // Last 30 seconds
|
||||
},
|
||||
limit: 50
|
||||
})
|
||||
|
||||
if (recentItems.length > 0) {
|
||||
// Incrementally update visualization
|
||||
const updates = await brain.neural.updateVisualization(
|
||||
this.visualization.id,
|
||||
{
|
||||
newItems: recentItems.map(item => item.id),
|
||||
algorithm: 'incremental' // Faster incremental updates
|
||||
}
|
||||
)
|
||||
|
||||
this.visualization = { ...this.visualization, ...updates }
|
||||
this.onUpdate(updates)
|
||||
}
|
||||
}
|
||||
|
||||
onUpdate(updates: any) {
|
||||
// Emit updates to frontend
|
||||
console.log(`Visualization updated: ${updates.newPoints.length} new points`)
|
||||
}
|
||||
|
||||
stop() {
|
||||
if (this.updateInterval) {
|
||||
clearInterval(this.updateInterval)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🚀 Performance Optimization Patterns
|
||||
|
||||
### ✅ **High-Performance Neural Operations**
|
||||
|
||||
```typescript
|
||||
// ✅ Pattern 1: Batch processing for similarity
|
||||
async function batchSimilarityCalculation(itemPairs: Array<[string, string]>) {
|
||||
const batchSize = 100
|
||||
const results = []
|
||||
|
||||
for (let i = 0; i < itemPairs.length; i += batchSize) {
|
||||
const batch = itemPairs.slice(i, i + batchSize)
|
||||
|
||||
const batchResults = await Promise.all(
|
||||
batch.map(async ([a, b]) => ({
|
||||
from: a,
|
||||
to: b,
|
||||
similarity: await brain.neural.similar(a, b)
|
||||
}))
|
||||
)
|
||||
|
||||
results.push(...batchResults)
|
||||
|
||||
// Progress reporting
|
||||
console.log(`Processed ${Math.min(i + batchSize, itemPairs.length)}/${itemPairs.length} pairs`)
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
// ✅ Pattern 2: Caching expensive operations
|
||||
class NeuralCache {
|
||||
private clusterCache = new Map()
|
||||
private similarityCache = new Map()
|
||||
private readonly TTL = 5 * 60 * 1000 // 5 minutes
|
||||
|
||||
async getClusters(options: any) {
|
||||
const key = JSON.stringify(options)
|
||||
const cached = this.clusterCache.get(key)
|
||||
|
||||
if (cached && Date.now() - cached.timestamp < this.TTL) {
|
||||
return cached.data
|
||||
}
|
||||
|
||||
const clusters = await brain.neural.clusters(undefined, options)
|
||||
this.clusterCache.set(key, {
|
||||
data: clusters,
|
||||
timestamp: Date.now()
|
||||
})
|
||||
|
||||
return clusters
|
||||
}
|
||||
|
||||
async getSimilarity(id1: string, id2: string) {
|
||||
// Create consistent cache key regardless of order
|
||||
const key = [id1, id2].sort().join('-')
|
||||
const cached = this.similarityCache.get(key)
|
||||
|
||||
if (cached && Date.now() - cached.timestamp < this.TTL) {
|
||||
return cached.data
|
||||
}
|
||||
|
||||
const similarity = await brain.neural.similar(id1, id2)
|
||||
this.similarityCache.set(key, {
|
||||
data: similarity,
|
||||
timestamp: Date.now()
|
||||
})
|
||||
|
||||
return similarity
|
||||
}
|
||||
}
|
||||
|
||||
// ✅ Pattern 3: Memory-efficient streaming
|
||||
async function processLargeDatasetEfficiently() {
|
||||
const stream = brain.neural.clusterStream({
|
||||
batchSize: 50, // Small batches for memory efficiency
|
||||
maxMemoryMB: 256, // Memory limit
|
||||
diskCache: true, // Use disk for temporary storage
|
||||
compression: true // Compress cached data
|
||||
})
|
||||
|
||||
const results = []
|
||||
let totalProcessed = 0
|
||||
|
||||
for await (const batch of stream) {
|
||||
// Process batch immediately, don't accumulate in memory
|
||||
const processedBatch = await processBatch(batch)
|
||||
|
||||
// Save to disk or send to another service
|
||||
await saveBatchToDisk(processedBatch)
|
||||
|
||||
totalProcessed += batch.items.length
|
||||
console.log(`Processed ${totalProcessed} items`)
|
||||
|
||||
// Clear memory
|
||||
batch.items = null
|
||||
}
|
||||
|
||||
return { totalProcessed }
|
||||
}
|
||||
|
||||
// ✅ Pattern 4: Parallel processing with worker threads
|
||||
async function parallelNeuralProcessing(items: string[]) {
|
||||
const numWorkers = require('os').cpus().length
|
||||
const batchSize = Math.ceil(items.length / numWorkers)
|
||||
|
||||
const workers = []
|
||||
for (let i = 0; i < numWorkers; i++) {
|
||||
const batch = items.slice(i * batchSize, (i + 1) * batchSize)
|
||||
if (batch.length > 0) {
|
||||
workers.push(processWorkerBatch(batch))
|
||||
}
|
||||
}
|
||||
|
||||
const results = await Promise.all(workers)
|
||||
return results.flat()
|
||||
}
|
||||
|
||||
async function processWorkerBatch(batch: string[]) {
|
||||
// This would run in a worker thread in real implementation
|
||||
return Promise.all(
|
||||
batch.map(async itemId => {
|
||||
const neighbors = await brain.neural.neighbors(itemId, { limit: 5 })
|
||||
return { itemId, neighbors }
|
||||
})
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
## 🎯 Summary: Neural API Best Practices
|
||||
|
||||
| ❌ **Avoid These Patterns** | ✅ **Use These Instead** |
|
||||
|---------------------------|------------------------|
|
||||
| Manual similarity loops | `brain.neural.neighbors()` |
|
||||
| Uncontrolled clustering | Limit items and set maxClusters |
|
||||
| Manual outlier detection | `brain.neural.outliers()` |
|
||||
| Manual visualization prep | `brain.neural.visualize()` |
|
||||
| Loading entire datasets | Streaming and batch processing |
|
||||
| No caching | Cache expensive operations |
|
||||
| Blocking operations | Parallel and async patterns |
|
||||
|
||||
---
|
||||
|
||||
**🎉 Following these patterns gives you:**
|
||||
- 🚀 **Optimized performance** with intelligent algorithms
|
||||
- 🧠 **AI-powered insights** instead of manual statistics
|
||||
- 📊 **Rich visualizations** for data exploration
|
||||
- 🎯 **Accurate clustering** with semantic understanding
|
||||
- 🚨 **Smart outlier detection** for quality control
|
||||
- ⚡ **Scalable processing** for large datasets
|
||||
|
||||
**Next:** [Augmentation Patterns →](./AUGMENTATION_PATTERNS.md) | [Core API Patterns →](./CORE_API_PATTERNS.md)
|
||||
247
docs/QUERY_OPERATORS.md
Normal file
247
docs/QUERY_OPERATORS.md
Normal file
|
|
@ -0,0 +1,247 @@
|
|||
# Query Operators (BFO)
|
||||
|
||||
> Brainy Field Operators — the complete reference for `where` filters in `find()`.
|
||||
|
||||
All operators work with `find({ where: { ... } })` and filter on **metadata fields** (not `data`).
|
||||
|
||||
---
|
||||
|
||||
## Equality
|
||||
|
||||
| Operator | Alias | Description | Example |
|
||||
|----------|-------|-------------|---------|
|
||||
| `equals` | `eq`, `is` | Exact match | `{ status: { equals: 'active' } }` |
|
||||
| `notEquals` | `ne`, `isNot` | Not equal | `{ status: { notEquals: 'deleted' } }` |
|
||||
|
||||
**Shorthand:** A bare value is treated as `equals`:
|
||||
|
||||
```typescript
|
||||
// These are equivalent:
|
||||
brain.find({ where: { status: 'active' } })
|
||||
brain.find({ where: { status: { equals: 'active' } } })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Comparison
|
||||
|
||||
| Operator | Alias | Description | Example |
|
||||
|----------|-------|-------------|---------|
|
||||
| `greaterThan` | `gt` | Greater than | `{ age: { greaterThan: 18 } }` |
|
||||
| `greaterEqual` | `gte` | Greater or equal | `{ score: { greaterEqual: 90 } }` |
|
||||
| `lessThan` | `lt` | Less than | `{ price: { lessThan: 100 } }` |
|
||||
| `lessEqual` | `lte` | Less or equal | `{ rating: { lessEqual: 3 } }` |
|
||||
| `between` | — | Inclusive range `[min, max]` | `{ year: { between: [2020, 2025] } }` |
|
||||
|
||||
```typescript
|
||||
// Range query
|
||||
const recent = await brain.find({
|
||||
where: {
|
||||
createdAt: { between: [Date.now() - 86400000, Date.now()] }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Array / Set
|
||||
|
||||
| Operator | Alias | Description | Example |
|
||||
|----------|-------|-------------|---------|
|
||||
| `oneOf` | `in` | Value is one of the given options | `{ color: { oneOf: ['red', 'blue'] } }` |
|
||||
| `noneOf` | — | Value is NOT one of the given options | `{ status: { noneOf: ['deleted', 'archived'] } }` |
|
||||
| `contains` | — | Array field contains value | `{ tags: { contains: 'ai' } }` |
|
||||
| `excludes` | — | Array field does NOT contain value | `{ tags: { excludes: 'spam' } }` |
|
||||
| `hasAll` | — | Array field contains ALL listed values | `{ skills: { hasAll: ['js', 'ts'] } }` |
|
||||
|
||||
```typescript
|
||||
// Find entities tagged with 'ai'
|
||||
const aiEntities = await brain.find({
|
||||
where: { tags: { contains: 'ai' } }
|
||||
})
|
||||
|
||||
// Find entities of specific types
|
||||
const people = await brain.find({
|
||||
where: { noun: { oneOf: ['Person', 'Agent'] } }
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Existence
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `exists: true` | Field exists (has any value) | `{ email: { exists: true } }` |
|
||||
| `exists: false` | Field does NOT exist | `{ email: { exists: false } }` |
|
||||
| `missing: true` | Field does NOT exist (alias for `exists: false`) | `{ email: { missing: true } }` |
|
||||
| `missing: false` | Field exists (alias for `exists: true`) | `{ email: { missing: false } }` |
|
||||
|
||||
```typescript
|
||||
// Find entities that have an email field
|
||||
const withEmail = await brain.find({
|
||||
where: { email: { exists: true } }
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pattern (In-Memory Only)
|
||||
|
||||
These operators work via the in-memory filter path. They are applied **after** the indexed query, so use them with other indexed operators for best performance.
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `matches` | Regex or string pattern match | `{ name: { matches: /^Dr\./ } }` |
|
||||
| `startsWith` | String prefix | `{ name: { startsWith: 'John' } }` |
|
||||
| `endsWith` | String suffix | `{ email: { endsWith: '@gmail.com' } }` |
|
||||
|
||||
```typescript
|
||||
const doctors = await brain.find({
|
||||
where: {
|
||||
type: NounType.Person, // Indexed — fast
|
||||
name: { startsWith: 'Dr.' } // In-memory — applied after
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Logical
|
||||
|
||||
Combine multiple conditions:
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `allOf` | ALL sub-filters must match (AND) | `{ allOf: [{ status: 'active' }, { role: 'admin' }] }` |
|
||||
| `anyOf` | ANY sub-filter must match (OR) | `{ anyOf: [{ role: 'admin' }, { role: 'owner' }] }` |
|
||||
| `not` | Invert a filter | `{ not: { status: 'deleted' } }` |
|
||||
|
||||
```typescript
|
||||
// Complex OR query
|
||||
const adminsOrOwners = await brain.find({
|
||||
where: {
|
||||
anyOf: [
|
||||
{ role: 'admin' },
|
||||
{ role: 'owner' }
|
||||
]
|
||||
}
|
||||
})
|
||||
|
||||
// NOT query
|
||||
const notDeleted = await brain.find({
|
||||
where: {
|
||||
not: { status: 'deleted' }
|
||||
}
|
||||
})
|
||||
|
||||
// Combined AND + OR
|
||||
const results = await brain.find({
|
||||
where: {
|
||||
allOf: [
|
||||
{ department: 'engineering' },
|
||||
{ anyOf: [
|
||||
{ level: 'senior' },
|
||||
{ yearsExperience: { greaterThan: 5 } }
|
||||
]}
|
||||
]
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Indexed vs In-Memory Operators
|
||||
|
||||
Brainy's MetadataIndex supports a subset of operators natively for O(1) field lookups. Other operators fall back to in-memory filtering.
|
||||
|
||||
| Operator | MetadataIndex (Indexed) | In-Memory Fallback |
|
||||
|----------|:-----------------------:|:------------------:|
|
||||
| `equals` / `eq` | Yes | Yes |
|
||||
| `notEquals` / `ne` | — | Yes |
|
||||
| `greaterThan` / `gt` | Yes | Yes |
|
||||
| `greaterEqual` / `gte` | Yes | Yes |
|
||||
| `lessThan` / `lt` | Yes | Yes |
|
||||
| `lessEqual` / `lte` | Yes | Yes |
|
||||
| `between` | Yes | Yes |
|
||||
| `oneOf` / `in` | Yes | Yes |
|
||||
| `noneOf` | — | Yes |
|
||||
| `contains` | Yes | Yes |
|
||||
| `exists` / `missing` | Yes | Yes |
|
||||
| `matches` | — | Yes |
|
||||
| `startsWith` | — | Yes |
|
||||
| `endsWith` | — | Yes |
|
||||
| `allOf` | Partial | Yes |
|
||||
| `anyOf` | Partial | Yes |
|
||||
| `not` | — | Yes |
|
||||
|
||||
**Performance tip:** Combine indexed operators (equals, greaterThan, oneOf, between, contains, exists) with pattern operators for optimal speed — the index narrows results first, then patterns filter in memory.
|
||||
|
||||
---
|
||||
|
||||
## Practical Examples
|
||||
|
||||
### Filter by entity type
|
||||
|
||||
```typescript
|
||||
// Using the type shorthand (recommended)
|
||||
brain.find({ type: NounType.Person })
|
||||
|
||||
// Using where.noun directly
|
||||
brain.find({ where: { noun: NounType.Person } })
|
||||
|
||||
// Multiple types
|
||||
brain.find({ type: [NounType.Person, NounType.Agent] })
|
||||
```
|
||||
|
||||
### Combine semantic search with filters
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
query: 'machine learning engineer', // Semantic search (on data)
|
||||
type: NounType.Person, // Type filter (indexed)
|
||||
where: {
|
||||
department: 'engineering', // Exact match (indexed)
|
||||
yearsExperience: { greaterThan: 3 } // Range filter (indexed)
|
||||
},
|
||||
limit: 10
|
||||
})
|
||||
```
|
||||
|
||||
### Temporal queries
|
||||
|
||||
```typescript
|
||||
const lastWeek = Date.now() - 7 * 24 * 60 * 60 * 1000
|
||||
const recentEntities = await brain.find({
|
||||
where: {
|
||||
createdAt: { greaterThan: lastWeek }
|
||||
},
|
||||
orderBy: 'createdAt',
|
||||
order: 'desc',
|
||||
limit: 50
|
||||
})
|
||||
```
|
||||
|
||||
### Graph + metadata combination
|
||||
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
connected: {
|
||||
from: teamLeadId,
|
||||
via: VerbType.WorksWith,
|
||||
depth: 2
|
||||
},
|
||||
where: {
|
||||
role: { oneOf: ['engineer', 'designer'] },
|
||||
active: true
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Data Model](./DATA_MODEL.md) — Entity structure, data vs metadata
|
||||
- [API Reference](./api/README.md) — Complete API documentation
|
||||
- [Find System](./FIND_SYSTEM.md) — Natural language find() details
|
||||
354
docs/README.md
354
docs/README.md
|
|
@ -1,332 +1,136 @@
|
|||
# Brainy Documentation
|
||||
|
||||
Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine.
|
||||
> The multi-dimensional AI database with Triple Intelligence — vector search, graph traversal, and metadata filtering in one unified API.
|
||||
|
||||
## 🆕 What's New in
|
||||
|
||||
**Production-Ready Cost Optimization:**
|
||||
- **Lifecycle Management**: Automatic tier transitions for S3, GCS, Azure (96% cost savings!)
|
||||
- **Intelligent-Tiering**: S3 Intelligent-Tiering and GCS Autoclass support
|
||||
- **Batch Operations**: Efficient bulk delete operations (1000 objects per request)
|
||||
- **Compression**: Gzip compression for FileSystem storage (60-80% space savings)
|
||||
- **Quota Monitoring**: Real-time OPFS quota tracking for browser apps
|
||||
- **Tier Management**: Azure Hot/Cool/Archive tier management
|
||||
|
||||
**Cost Impact Example (500TB dataset):**
|
||||
- Before: $138,000/year
|
||||
- After $5,940/year
|
||||
- **Savings: $132,060/year (96%)**
|
||||
|
||||
## 📊 Implementation Status
|
||||
|
||||
- ✅ **Production Ready**: Core features working today
|
||||
- 🚧 **In Development**: Features coming soon
|
||||
- 📅 **Roadmap**: See [ROADMAP.md](../ROADMAP.md)
|
||||
|
||||
## Quick Links
|
||||
|
||||
### Getting Started
|
||||
- [API Reference](./api/README.md) - Complete API documentation (start here!)
|
||||
- [Enterprise for Everyone](./guides/enterprise-for-everyone.md) - **No limits, no tiers, everything free**
|
||||
- [Natural Language Queries](./guides/natural-language.md) - Query with plain English
|
||||
|
||||
### Core Concepts
|
||||
- [Zero Configuration](./architecture/zero-config.md) - **Auto-adapts to any environment**
|
||||
- [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) - **Revolutionary data model**
|
||||
- [Triple Intelligence](./architecture/triple-intelligence.md) - Unified query system
|
||||
- [Architecture Overview](./architecture/overview.md) - System design
|
||||
|
||||
### API Documentation
|
||||
- [API Reference](./api/README.md) - Complete API documentation
|
||||
- [TypeScript Types](./api/types.md) - Type definitions
|
||||
|
||||
### Advanced Topics
|
||||
- [Augmentations System](./architecture/augmentations.md) - **Enterprise plugins & neural import**
|
||||
- [Storage Architecture](./architecture/storage.md) - Storage adapter system
|
||||
- [Performance Tuning](./guides/performance.md) - Optimization guide
|
||||
- [Migration Guide](../MIGRATION.md) - Upgrading from 1.x
|
||||
|
||||
## What is Brainy?
|
||||
|
||||
Brainy is a next-generation AI database that combines:
|
||||
- **Vector Search**: Semantic similarity using HNSW indexing
|
||||
- **Graph Relationships**: Complex relationship mapping and traversal
|
||||
- **Field Filtering**: Precise metadata filtering with O(1) lookups
|
||||
- **Natural Language**: Query in plain English
|
||||
|
||||
## Key Features
|
||||
|
||||
### 🧠 Triple Intelligence Engine
|
||||
All three intelligence types (vector, graph, field) work together in every query for optimal results.
|
||||
|
||||
### 📝 Noun-Verb Taxonomy
|
||||
Model your data naturally as entities (nouns) and relationships (verbs) - no complex schemas needed.
|
||||
|
||||
### 🌍 Natural Language Queries
|
||||
Ask questions in plain English and Brainy understands your intent:
|
||||
```typescript
|
||||
await brain.find("recent articles about AI with high ratings")
|
||||
```
|
||||
|
||||
### ⚡ Production Ready
|
||||
- Universal storage (FileSystem, S3, OPFS, Memory)
|
||||
- Zero configuration with intelligent defaults
|
||||
- Full TypeScript support
|
||||
- Cross-platform compatibility
|
||||
|
||||
## Quick Example
|
||||
## Quick Start
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
// Initialize
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Add entities (nouns)
|
||||
const articleId = await brain.add({
|
||||
data: "Revolutionary AI Breakthrough",
|
||||
type: NounType.Document,
|
||||
metadata: { category: "technology", rating: 4.8 }
|
||||
// Add entities — data is embedded for semantic search, metadata is indexed for filtering
|
||||
const id = await brain.add({
|
||||
data: 'Revolutionary AI Breakthrough',
|
||||
type: NounType.Document,
|
||||
metadata: { category: 'technology', rating: 4.8 }
|
||||
})
|
||||
|
||||
const authorId = await brain.add({
|
||||
data: "Dr. Sarah Chen",
|
||||
type: NounType.Person,
|
||||
metadata: { role: "researcher" }
|
||||
// Search with Triple Intelligence
|
||||
const results = await brain.find({
|
||||
query: 'artificial intelligence', // Semantic search (on data)
|
||||
where: { rating: { greaterThan: 4.0 } }, // Metadata filter
|
||||
connected: { from: authorId, depth: 2 } // Graph traversal
|
||||
})
|
||||
|
||||
// Create relationships (verbs)
|
||||
await brain.relate({
|
||||
from: authorId,
|
||||
to: articleId,
|
||||
type: VerbType.CreatedBy,
|
||||
metadata: { date: "2024-01-15", contribution: "primary" }
|
||||
})
|
||||
|
||||
// Query naturally
|
||||
const results = await brain.find("highly rated technology articles by researchers")
|
||||
```
|
||||
|
||||
## 📚 Complete Documentation Index
|
||||
---
|
||||
|
||||
### 🚀 Quick Start Guides
|
||||
## Core Documentation
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [API Reference](./api/README.md) | **START HERE** - Complete API documentation with examples |
|
||||
| [VFS Quick Start](./vfs/QUICK_START.md) | Virtual filesystem in 30 seconds |
|
||||
| **[API Reference](./api/README.md)** | Complete API documentation — **start here** |
|
||||
| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields |
|
||||
| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix |
|
||||
| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details |
|
||||
|
||||
### 🆕 Migration & Optimization
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [v3→v4 Migration Guide](./MIGRATION-V3-TO-V4.md) | **NEW** - Upgrade guide with zero breaking changes |
|
||||
| [AWS S3 Cost Optimization](./operations/cost-optimization-aws-s3.md) | **NEW** - 96% cost savings with lifecycle policies |
|
||||
| [GCS Cost Optimization](./operations/cost-optimization-gcs.md) | **NEW** - 94% savings with Autoclass |
|
||||
| [Azure Cost Optimization](./operations/cost-optimization-azure.md) | **NEW** - 95% savings with tier management |
|
||||
| [Cloudflare R2 Cost Guide](./operations/cost-optimization-cloudflare-r2.md) | **NEW** - Zero egress fees + S3-compatible API |
|
||||
|
||||
### 🎯 Core Concepts
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Architecture Overview](./architecture/overview.md) | High-level system design and components |
|
||||
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | Revolutionary data model - entities and relationships |
|
||||
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Field unified query system |
|
||||
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
|
||||
| [Architecture Overview](./architecture/overview.md) | High-level system design |
|
||||
| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Metadata unified query |
|
||||
| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | 42 nouns + 127 verbs type system |
|
||||
| [Stage 3 Canonical Taxonomy](./STAGE3-CANONICAL-TAXONOMY.md) | Complete type reference |
|
||||
| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization |
|
||||
| [Data Storage Architecture](./architecture/data-storage-architecture.md) | Metadata/vector separation, sharding |
|
||||
| [Index Architecture](./architecture/index-architecture.md) | HNSW, Graph, and Metadata indexing |
|
||||
| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment |
|
||||
|
||||
### 💾 Storage & Deployment
|
||||
---
|
||||
|
||||
## Virtual Filesystem (VFS)
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Cloud Deployment Guide](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | Deploy on AWS, GCP, Azure, Cloudflare |
|
||||
| [AWS Deployment](./deployment/aws-deployment.md) | AWS-specific deployment patterns |
|
||||
| [GCP Deployment](./deployment/gcp-deployment.md) | Google Cloud deployment |
|
||||
| [Kubernetes Deployment](./deployment/kubernetes-deployment.md) | K8s deployment configurations |
|
||||
| [Distributed Storage](./architecture/distributed-storage.md) | Multi-node storage coordination |
|
||||
| [VFS Quick Start](./vfs/QUICK_START.md) | Get started in 30 seconds |
|
||||
| [VFS Core](./vfs/VFS_CORE.md) | Core concepts and architecture |
|
||||
| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
|
||||
| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
|
||||
|
||||
See [vfs/](./vfs/) for the complete VFS documentation set.
|
||||
|
||||
---
|
||||
|
||||
## Guides
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports |
|
||||
| [Natural Language](./guides/natural-language.md) | Query in plain English |
|
||||
| [Neural API](./guides/neural-api.md) | AI-powered features |
|
||||
| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers |
|
||||
| [Framework Integration](./guides/framework-integration.md) | React, Vue, Angular, Svelte |
|
||||
|
||||
---
|
||||
|
||||
## Storage & Deployment
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Cloud Deployment](./deployment/CLOUD_DEPLOYMENT_GUIDE.md) | Deploy on AWS, GCP, Azure, Cloudflare |
|
||||
| [Extending Storage](./EXTENDING_STORAGE.md) | Create custom storage adapters |
|
||||
| [AWS S3 Cost Optimization](./operations/cost-optimization-aws-s3.md) | 96% cost savings |
|
||||
| [GCS Cost Optimization](./operations/cost-optimization-gcs.md) | 94% savings with Autoclass |
|
||||
| [Azure Cost Optimization](./operations/cost-optimization-azure.md) | 95% savings |
|
||||
| [R2 Cost Optimization](./operations/cost-optimization-cloudflare-r2.md) | Zero egress fees |
|
||||
| [Capacity Planning](./operations/capacity-planning.md) | Scale to millions of entities |
|
||||
|
||||
### 📊 API Documentation
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [API Reference](./API_REFERENCE.md) | Complete API documentation |
|
||||
| [Comprehensive API Overview](./api/COMPREHENSIVE_API_OVERVIEW.md) | All APIs with examples |
|
||||
| [API Decision Tree](./API_DECISION_TREE.md) | Choose the right API for your use case |
|
||||
| [Core API Patterns](./CORE_API_PATTERNS.md) | Common patterns and best practices |
|
||||
| [Neural API Patterns](./NEURAL_API_PATTERNS.md) | AI-powered query patterns |
|
||||
| [Find System](./FIND_SYSTEM.md) | Natural language find() API |
|
||||
| [API Surface Design](./architecture/API_SURFACE_DESIGN.md) | API design principles |
|
||||
| [API Returns](./api-returns.md) | Return types and structures |
|
||||
|
||||
### 🔧 Framework Integration
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Framework Integration Guide](./guides/framework-integration.md) | React, Vue, Angular, Svelte, etc. |
|
||||
| [Next.js Integration](./guides/nextjs-integration.md) | Server-side rendering with Brainy |
|
||||
| [Vue.js Integration](./guides/vue-integration.md) | Vue 3 integration patterns |
|
||||
|
||||
### 📁 Virtual Filesystem (VFS)
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [VFS Core Documentation](./vfs/VFS_CORE.md) | Core VFS concepts and architecture |
|
||||
| [Semantic VFS Guide](./vfs/SEMANTIC_VFS.md) | Semantic filesystem projections |
|
||||
| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference |
|
||||
| [Neural Extraction](./vfs/NEURAL_EXTRACTION.md) | AI-powered file analysis |
|
||||
| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns |
|
||||
| [User Functions](./vfs/USER_FUNCTIONS.md) | Custom VFS functions |
|
||||
| [VFS Graph Types](./vfs/VFS_GRAPH_TYPES.md) | Graph-based VFS projections |
|
||||
| [VFS Examples & Scenarios](./vfs/VFS_EXAMPLES_SCENARIOS.md) | Real-world VFS examples |
|
||||
| [VFS Initialization](./vfs/VFS_INITIALIZATION.md) | Setup and configuration |
|
||||
| [Projection Strategy API](./vfs/PROJECTION_STRATEGY_API.md) | Custom projection strategies |
|
||||
| [Building File Explorers](./vfs/building-file-explorers.md) | Build custom file browsers |
|
||||
| [VFS Troubleshooting](./vfs/TROUBLESHOOTING.md) | Common issues and solutions |
|
||||
|
||||
### 🧠 Advanced Topics
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Natural Language Queries](./guides/natural-language.md) | Query in plain English |
|
||||
| [Neural API](./guides/neural-api.md) | AI-powered features |
|
||||
| [Import Anything](./guides/import-anything.md) | Import data from any source |
|
||||
| [Distributed Systems](./guides/distributed-system.md) | Multi-node deployments |
|
||||
| [Model Loading](./guides/model-loading.md) | Load and manage AI models |
|
||||
| [Model Loading Quick Reference](./MODEL_LOADING_QUICK_REFERENCE.md) | Model loading cheat sheet |
|
||||
| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No paywalls, no tiers |
|
||||
|
||||
### 🔌 Augmentations (Plugins)
|
||||
---
|
||||
|
||||
## Plugins & Augmentations
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Plugins](./PLUGINS.md) | Plugin system overview |
|
||||
| [Creating Augmentations](./CREATING-AUGMENTATIONS.md) | Build custom plugins |
|
||||
| [Augmentations Complete Reference](./augmentations/COMPLETE-REFERENCE.md) | Full augmentation API |
|
||||
| [Augmentations Reference](./augmentations/COMPLETE-REFERENCE.md) | Full augmentation API |
|
||||
| [Augmentations Developer Guide](./augmentations/DEVELOPER-GUIDE.md) | Plugin development guide |
|
||||
| [Augmentation Configuration](./augmentations/CONFIGURATION.md) | Configure augmentations |
|
||||
| [Augmentation System](./architecture/augmentations.md) | System architecture |
|
||||
| [Augmentation System Audit](./architecture/augmentation-system-audit.md) | Actual vs planned features |
|
||||
| [Augmentations Actual](./architecture/augmentations-actual.md) | Currently implemented augmentations |
|
||||
| [API Server Augmentation](./augmentations/api-server.md) | REST API server plugin |
|
||||
|
||||
### ⚡ Performance & Scaling
|
||||
---
|
||||
|
||||
## Performance & Scaling
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Performance Guide](./PERFORMANCE.md) | Optimization techniques |
|
||||
| [Performance Analysis](./architecture/PERFORMANCE_ANALYSIS.md) | Benchmarks and analysis |
|
||||
| [Scaling Guide](./SCALING.md) | Scale to billions of entities |
|
||||
| [Clustering Algorithms Analysis](./architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md) | HNSW algorithm details |
|
||||
| [Initialization & Rebuild](./architecture/initialization-and-rebuild.md) | Index management |
|
||||
| [Performance](./PERFORMANCE.md) | Optimization techniques |
|
||||
| [Scaling](./SCALING.md) | Scale to billions of entities |
|
||||
| [Batching](./BATCHING.md) | Batch operations guide |
|
||||
|
||||
### 📋 Reference
|
||||
---
|
||||
|
||||
## Migration & Reference
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Complete Feature List](./features/complete-feature-list.md) | All Brainy features |
|
||||
| [v3 Features](./features/v3-features.md) | v3.x feature list |
|
||||
| [Metadata Architecture](./architecture/METADATA_ARCHITECTURE.md) | Metadata namespacing |
|
||||
| [Metadata Contract Implementation](./METADATA_CONTRACT_IMPLEMENTATION.md) | Metadata API contracts |
|
||||
| [Finite Type System](./architecture/finite-type-system.md) | Type-safe noun/verb taxonomy |
|
||||
| [Validation](./VALIDATION.md) | Data validation rules |
|
||||
| [Universal Display Augmentation](./universal-display-augmentation.md) | Display system |
|
||||
|
||||
### 🛠️ Development
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Troubleshooting](./troubleshooting.md) | Common issues and solutions |
|
||||
| [v3 to v4 Migration](./MIGRATION-V3-TO-V4.md) | Upgrade guide |
|
||||
| [Release Guide](./RELEASE-GUIDE.md) | How to release new versions |
|
||||
| [Production Architecture](./PRODUCTION_SERVICE_ARCHITECTURE.md) | Ops reference |
|
||||
|
||||
### 🔍 Internal Documentation
|
||||
---
|
||||
|
||||
## Internal
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Audit Report](./internal/AUDIT_REPORT.md) | Feature audit |
|
||||
| [Cleanup Summary](./internal/CLEANUP_SUMMARY.md) | Codebase cleanup notes |
|
||||
| [Honest Status](./internal/HONEST_STATUS.md) | Actual implementation status |
|
||||
|
||||
---
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
```
|
||||
docs/
|
||||
├── README.md (this file) # Complete documentation index
|
||||
├── MIGRATION-V3-TO-V4.md # Migration guide
|
||||
│
|
||||
├── guides/ # User guides
|
||||
│ ├── natural-language.md
|
||||
│ ├── neural-api.md
|
||||
│ ├── import-anything.md
|
||||
│ ├── framework-integration.md
|
||||
│ ├── nextjs-integration.md
|
||||
│ ├── vue-integration.md
|
||||
│ ├── distributed-system.md
|
||||
│ ├── model-loading.md
|
||||
│ └── enterprise-for-everyone.md
|
||||
│
|
||||
├── architecture/ # System architecture
|
||||
│ ├── overview.md
|
||||
│ ├── noun-verb-taxonomy.md
|
||||
│ ├── triple-intelligence.md
|
||||
│ ├── zero-config.md
|
||||
│ ├── storage-architecture.md│ ├── data-storage-architecture.md│ ├── index-architecture.md
|
||||
│ ├── distributed-storage.md
|
||||
│ ├── augmentations.md
|
||||
│ ├── augmentation-system-audit.md
|
||||
│ ├── augmentations-actual.md
|
||||
│ ├── finite-type-system.md
|
||||
│ └── ...
|
||||
│
|
||||
├── operations/ # Operations guides
|
||||
│ ├── cost-optimization-aws-s3.md│ ├── cost-optimization-gcs.md│ ├── cost-optimization-azure.md│ ├── cost-optimization-cloudflare-r2.md│ └── capacity-planning.md
|
||||
│
|
||||
├── deployment/ # Deployment guides
|
||||
│ ├── CLOUD_DEPLOYMENT_GUIDE.md│ ├── aws-deployment.md
|
||||
│ ├── gcp-deployment.md
|
||||
│ └── kubernetes-deployment.md
|
||||
│
|
||||
├── vfs/ # Virtual Filesystem docs
|
||||
│ ├── QUICK_START.md
|
||||
│ ├── VFS_CORE.md
|
||||
│ ├── SEMANTIC_VFS.md
|
||||
│ ├── VFS_API_GUIDE.md
|
||||
│ ├── NEURAL_EXTRACTION.md
|
||||
│ ├── COMMON_PATTERNS.md
|
||||
│ └── ...
|
||||
│
|
||||
├── api/ # API documentation
|
||||
│ ├── README.md
|
||||
│ └── COMPREHENSIVE_API_OVERVIEW.md
|
||||
│
|
||||
├── augmentations/ # Augmentation docs
|
||||
│ ├── COMPLETE-REFERENCE.md
|
||||
│ ├── DEVELOPER-GUIDE.md
|
||||
│ ├── CONFIGURATION.md
|
||||
│ └── api-server.md
|
||||
│
|
||||
├── features/ # Feature documentation
|
||||
│ ├── complete-feature-list.md
|
||||
│ └── v3-features.md
|
||||
│
|
||||
└── internal/ # Internal docs
|
||||
├── AUDIT_REPORT.md
|
||||
├── CLEANUP_SUMMARY.md
|
||||
└── HONEST_STATUS.md
|
||||
```
|
||||
|
||||
## Community
|
||||
|
||||
- **GitHub**: [github.com/brainy-org/brainy](https://github.com/brainy-org/brainy)
|
||||
- **Issues**: [Report bugs or request features](https://github.com/brainy-org/brainy/issues)
|
||||
- **Discussions**: [Join the conversation](https://github.com/brainy-org/brainy/discussions)
|
||||
|
||||
## License
|
||||
|
||||
Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
|
||||
Brainy is MIT licensed. See [LICENSE](../LICENSE) for details.
|
||||
|
|
|
|||
|
|
@ -1,340 +0,0 @@
|
|||
# Brainy Validation System
|
||||
|
||||
## Zero-Config Philosophy
|
||||
|
||||
Brainy's validation system automatically adapts to your system resources without any configuration. It enforces universal truths while dynamically adjusting limits based on available memory and observed performance.
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. Universal Truths Only
|
||||
We only validate things that are mathematically or logically impossible:
|
||||
- Negative pagination values (there's no page -1)
|
||||
- Probabilities outside 0-1 range
|
||||
- Self-referential relationships
|
||||
- Invalid enum values
|
||||
|
||||
### 2. Auto-Configuration
|
||||
The system automatically configures based on:
|
||||
- **Available Memory**: More RAM = higher limits
|
||||
- **System Performance**: Adjusts based on query response times
|
||||
- **Usage Patterns**: Learns from your actual workload
|
||||
|
||||
### 3. Performance Monitoring
|
||||
Every query is monitored to tune future limits:
|
||||
```typescript
|
||||
// Automatic adjustment based on performance
|
||||
if (avgQueryTime < 100ms && resultCount > 80% of limit) {
|
||||
// Increase limits - system can handle more
|
||||
maxLimit *= 1.5
|
||||
} else if (avgQueryTime > 1000ms) {
|
||||
// Reduce limits - system is struggling
|
||||
maxLimit *= 0.8
|
||||
}
|
||||
```
|
||||
|
||||
## Validation Rules by Method
|
||||
|
||||
### `add(params: AddParams)`
|
||||
|
||||
**Required:**
|
||||
- Either `data` or `vector` must be provided
|
||||
- `type` must be a valid NounType enum
|
||||
|
||||
**Constraints:**
|
||||
- `vector` must have exactly 384 dimensions (for all-MiniLM-L6-v2)
|
||||
- Custom `id` must be unique
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
// ✅ Valid
|
||||
await brain.add({
|
||||
data: "Hello world",
|
||||
type: NounType.Document
|
||||
})
|
||||
|
||||
// ✅ Valid - pre-computed vector
|
||||
await brain.add({
|
||||
vector: new Array(384).fill(0),
|
||||
type: NounType.Document
|
||||
})
|
||||
|
||||
// ❌ Invalid - missing both data and vector
|
||||
await brain.add({
|
||||
type: NounType.Document
|
||||
})
|
||||
// Error: "must provide either data or vector"
|
||||
|
||||
// ❌ Invalid - wrong vector dimensions
|
||||
await brain.add({
|
||||
vector: new Array(100).fill(0),
|
||||
type: NounType.Document
|
||||
})
|
||||
// Error: "vector must have exactly 384 dimensions"
|
||||
```
|
||||
|
||||
### `update(params: UpdateParams)`
|
||||
|
||||
**Required:**
|
||||
- `id` must be provided
|
||||
- At least one field must be updated
|
||||
|
||||
**Important Metadata Behavior:**
|
||||
- `metadata: null` with `merge: false` → **Keeps existing metadata** (does nothing)
|
||||
- `metadata: {}` with `merge: false` → **Clears metadata** ✅
|
||||
- `metadata: undefined` → No change to metadata
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
// ✅ Valid - update metadata
|
||||
await brain.update({
|
||||
id: "xyz",
|
||||
metadata: { status: "published" }
|
||||
})
|
||||
|
||||
// ✅ Valid - clear metadata properly
|
||||
await brain.update({
|
||||
id: "xyz",
|
||||
metadata: {},
|
||||
merge: false
|
||||
})
|
||||
|
||||
// ❌ Invalid - null doesn't clear metadata
|
||||
await brain.update({
|
||||
id: "xyz",
|
||||
metadata: null,
|
||||
merge: false
|
||||
})
|
||||
// Error: "must specify at least one field to update"
|
||||
// (because null metadata doesn't actually update anything)
|
||||
|
||||
// ❌ Invalid - no fields to update
|
||||
await brain.update({
|
||||
id: "xyz"
|
||||
})
|
||||
// Error: "must specify at least one field to update"
|
||||
```
|
||||
|
||||
### `relate(params: RelateParams)`
|
||||
|
||||
**Required:**
|
||||
- `from` entity ID
|
||||
- `to` entity ID
|
||||
- `type` must be valid VerbType enum
|
||||
|
||||
**Constraints:**
|
||||
- `from` and `to` must be different (no self-loops)
|
||||
- `weight` must be between 0 and 1
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
// ✅ Valid
|
||||
await brain.relate({
|
||||
from: "entity1",
|
||||
to: "entity2",
|
||||
type: VerbType.RelatedTo
|
||||
})
|
||||
|
||||
// ❌ Invalid - self-referential
|
||||
await brain.relate({
|
||||
from: "entity1",
|
||||
to: "entity1",
|
||||
type: VerbType.RelatedTo
|
||||
})
|
||||
// Error: "cannot create self-referential relationship"
|
||||
|
||||
// ❌ Invalid - weight out of range
|
||||
await brain.relate({
|
||||
from: "entity1",
|
||||
to: "entity2",
|
||||
type: VerbType.RelatedTo,
|
||||
weight: 1.5
|
||||
})
|
||||
// Error: "weight must be between 0 and 1"
|
||||
```
|
||||
|
||||
### `find(params: FindParams)`
|
||||
|
||||
**Constraints:**
|
||||
- `limit` must be non-negative and below auto-configured maximum
|
||||
- `offset` must be non-negative
|
||||
- Cannot specify both `query` and `vector` (mutually exclusive)
|
||||
- Cannot use both `cursor` and `offset` pagination
|
||||
- `threshold` must be between 0 and 1
|
||||
|
||||
**Auto-Configured Limits:**
|
||||
```typescript
|
||||
// Based on available memory
|
||||
// 1GB RAM → max limit: 10,000
|
||||
// 8GB RAM → max limit: 80,000
|
||||
// 16GB RAM → max limit: 100,000 (capped)
|
||||
|
||||
// Query length also scales with memory
|
||||
// 1GB RAM → max query: 5,000 characters
|
||||
// 8GB RAM → max query: 40,000 characters
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```typescript
|
||||
// ✅ Valid
|
||||
await brain.find({
|
||||
query: "machine learning",
|
||||
limit: 50
|
||||
})
|
||||
|
||||
// ❌ Invalid - negative limit
|
||||
await brain.find({
|
||||
query: "test",
|
||||
limit: -1
|
||||
})
|
||||
// Error: "limit must be non-negative"
|
||||
|
||||
// ❌ Invalid - both query and vector
|
||||
await brain.find({
|
||||
query: "test",
|
||||
vector: new Array(384).fill(0)
|
||||
})
|
||||
// Error: "cannot specify both query and vector - they are mutually exclusive"
|
||||
|
||||
// ❌ Invalid - exceeds auto-configured limit
|
||||
await brain.find({
|
||||
limit: 1000000
|
||||
})
|
||||
// Error: "limit exceeds auto-configured maximum of 80000 (based on available memory)"
|
||||
```
|
||||
|
||||
## Auto-Configuration Details
|
||||
|
||||
### Memory-Based Scaling
|
||||
|
||||
The validation system checks available memory on initialization:
|
||||
|
||||
```typescript
|
||||
const availableMemory = os.freemem()
|
||||
|
||||
// Scale limits based on available memory
|
||||
maxLimit = Math.min(
|
||||
100000, // Absolute maximum for safety
|
||||
Math.floor(availableMemory / (1024 * 1024 * 100)) * 1000
|
||||
)
|
||||
|
||||
// Scale query length similarly
|
||||
maxQueryLength = Math.min(
|
||||
50000,
|
||||
Math.floor(availableMemory / (1024 * 1024 * 10)) * 1000
|
||||
)
|
||||
```
|
||||
|
||||
### Performance-Based Tuning
|
||||
|
||||
The system continuously monitors and adjusts:
|
||||
|
||||
1. **After each query**, performance is recorded
|
||||
2. **Limits adjust** based on response times
|
||||
3. **Gradual optimization** towards optimal throughput
|
||||
|
||||
### Checking Current Configuration
|
||||
|
||||
You can inspect the current validation configuration:
|
||||
|
||||
```typescript
|
||||
import { getValidationConfig } from '@soulcraft/brainy/validation'
|
||||
|
||||
const config = getValidationConfig()
|
||||
console.log(config)
|
||||
// {
|
||||
// maxLimit: 80000,
|
||||
// maxQueryLength: 40000,
|
||||
// maxVectorDimensions: 384,
|
||||
// systemMemory: 17179869184,
|
||||
// availableMemory: 8589934592
|
||||
// }
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Clearing Metadata
|
||||
```typescript
|
||||
// ❌ Wrong - doesn't clear
|
||||
await brain.update({ id, metadata: null, merge: false })
|
||||
|
||||
// ✅ Correct - actually clears
|
||||
await brain.update({ id, metadata: {}, merge: false })
|
||||
```
|
||||
|
||||
### 2. Type Safety
|
||||
```typescript
|
||||
// ❌ Wrong - string type
|
||||
await brain.add({ data: "test", type: "document" })
|
||||
|
||||
// ✅ Correct - enum type
|
||||
import { NounType } from '@soulcraft/brainy'
|
||||
await brain.add({ data: "test", type: NounType.Document })
|
||||
```
|
||||
|
||||
### 3. Pagination
|
||||
```typescript
|
||||
// ✅ Let the system auto-configure limits
|
||||
const results = await brain.find({
|
||||
query: "test",
|
||||
limit: 100 // Will be capped at system maximum
|
||||
})
|
||||
|
||||
// ✅ For large datasets, use pagination
|
||||
let offset = 0
|
||||
const pageSize = 1000
|
||||
while (true) {
|
||||
const results = await brain.find({
|
||||
query: "test",
|
||||
limit: pageSize,
|
||||
offset
|
||||
})
|
||||
if (results.length === 0) break
|
||||
offset += pageSize
|
||||
}
|
||||
```
|
||||
|
||||
## Error Messages
|
||||
|
||||
All validation errors are descriptive and actionable:
|
||||
|
||||
| Error | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| `"must provide either data or vector"` | Missing content in add() | Provide either data to embed or pre-computed vector |
|
||||
| `"limit must be non-negative"` | Negative pagination | Use positive limit value |
|
||||
| `"invalid NounType: xyz"` | Invalid enum value | Use valid NounType enum |
|
||||
| `"cannot create self-referential relationship"` | from === to | Use different entity IDs |
|
||||
| `"must specify at least one field to update"` | Empty update | Provide at least one field to change |
|
||||
| `"vector must have exactly 384 dimensions"` | Wrong vector size | Use 384-dimensional vectors |
|
||||
|
||||
## Performance Impact
|
||||
|
||||
The validation system adds minimal overhead:
|
||||
- **Validation time**: <1ms per operation
|
||||
- **Memory usage**: ~1KB for configuration tracking
|
||||
- **Auto-tuning**: Happens asynchronously, no blocking
|
||||
|
||||
## FAQ
|
||||
|
||||
**Q: Why can't I set metadata to null?**
|
||||
A: Setting metadata to `null` with `merge: false` doesn't actually clear it - it falls back to existing metadata. Use `{}` to clear.
|
||||
|
||||
**Q: Why are my limits being reduced?**
|
||||
A: If queries are taking >1 second, the system automatically reduces limits to maintain performance.
|
||||
|
||||
**Q: Can I override the auto-configured limits?**
|
||||
A: No, this is by design. The system knows better than static configuration what your hardware can handle.
|
||||
|
||||
**Q: Why exactly 384 dimensions for vectors?**
|
||||
A: Brainy uses the all-MiniLM-L6-v2 model which produces 384-dimensional embeddings. This ensures consistency.
|
||||
|
||||
## Summary
|
||||
|
||||
Brainy's validation system:
|
||||
- ✅ **Zero configuration** - adapts to your system
|
||||
- ✅ **Universal truths** - only prevents impossible operations
|
||||
- ✅ **Performance aware** - adjusts based on actual performance
|
||||
- ✅ **Type safe** - enforces enum types
|
||||
- ✅ **Minimal overhead** - <1ms validation time
|
||||
- ✅ **Clear errors** - actionable error messages
|
||||
|
||||
The philosophy is simple: prevent impossible operations, adapt to reality, and get out of the way.
|
||||
|
|
@ -1,406 +0,0 @@
|
|||
# 🚀 Brainy Zero-Configuration Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Starting with v2.10, Brainy introduces a **Zero-Configuration System** that automatically configures everything based on your environment. No more environment variables, no more complex configuration objects - just create and use.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### True Zero Config
|
||||
```typescript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// That's it! Everything auto-configures
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
### Using Strongly-Typed Presets
|
||||
|
||||
```typescript
|
||||
import { Brainy, PresetName } from '@soulcraft/brainy'
|
||||
|
||||
// Type-safe preset selection
|
||||
const brain = new Brainy(PresetName.PRODUCTION)
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
### Common Scenarios
|
||||
|
||||
#### Development
|
||||
```typescript
|
||||
const brain = new Brainy(PresetName.DEVELOPMENT)
|
||||
// ✅ Filesystem storage for persistence
|
||||
// ✅ FP32 models for best quality
|
||||
// ✅ Verbose logging
|
||||
// ✅ All features enabled
|
||||
```
|
||||
|
||||
#### Production
|
||||
```typescript
|
||||
const brain = new Brainy(PresetName.PRODUCTION)
|
||||
// ✅ Disk storage for persistence
|
||||
// ✅ Auto-selected model precision
|
||||
// ✅ Silent logging
|
||||
// ✅ Optimized features
|
||||
```
|
||||
|
||||
#### Minimal
|
||||
```typescript
|
||||
const brain = new Brainy(PresetName.MINIMAL)
|
||||
// ✅ Filesystem storage
|
||||
// ✅ Q8 models for small size
|
||||
// ✅ Core features only
|
||||
// ✅ Minimal resource usage
|
||||
```
|
||||
|
||||
## Distributed Architecture Presets
|
||||
|
||||
Brainy includes specialized presets for distributed and microservice architectures:
|
||||
|
||||
### Basic Distributed Roles
|
||||
```typescript
|
||||
import { Brainy, PresetName } from '@soulcraft/brainy'
|
||||
|
||||
// Write-only instance (data ingestion)
|
||||
const writer = new Brainy(PresetName.WRITER)
|
||||
// ✅ Optimized for writes
|
||||
// ✅ No search index loading
|
||||
// ✅ Minimal memory usage
|
||||
|
||||
// Read-only instance (search API)
|
||||
const reader = new Brainy(PresetName.READER)
|
||||
// ✅ Optimized for search
|
||||
// ✅ Lazy index loading
|
||||
// ✅ Large cache
|
||||
```
|
||||
|
||||
### Service-Specific Presets
|
||||
```typescript
|
||||
// High-throughput data ingestion
|
||||
const ingestion = new Brainy(PresetName.INGESTION_SERVICE)
|
||||
|
||||
// Low-latency search API
|
||||
const searchApi = new Brainy(PresetName.SEARCH_API)
|
||||
|
||||
// Analytics processing
|
||||
const analytics = new Brainy(PresetName.ANALYTICS_SERVICE)
|
||||
|
||||
// Edge location cache
|
||||
const edge = new Brainy(PresetName.EDGE_CACHE)
|
||||
|
||||
// Batch processing
|
||||
const batch = new Brainy(PresetName.BATCH_PROCESSOR)
|
||||
|
||||
// Real-time streaming
|
||||
const streaming = new Brainy(PresetName.STREAMING_SERVICE)
|
||||
|
||||
// ML training
|
||||
const training = new Brainy(PresetName.ML_TRAINING)
|
||||
|
||||
// Lightweight sidecar
|
||||
const sidecar = new Brainy(PresetName.SIDECAR)
|
||||
```
|
||||
|
||||
## Model Precision Control
|
||||
|
||||
You can **explicitly specify** model precision when needed:
|
||||
|
||||
```typescript
|
||||
import { ModelPrecision } from '@soulcraft/brainy'
|
||||
|
||||
// Force FP32 (full precision)
|
||||
const brain = new Brainy({ model: ModelPrecision.FP32 })
|
||||
|
||||
// Force Q8 (quantized, smaller)
|
||||
const brain = new Brainy({ model: ModelPrecision.Q8 })
|
||||
|
||||
// Use presets
|
||||
const brain = new Brainy({ model: ModelPrecision.FAST }) // Maps to fp32
|
||||
const brain = new Brainy({ model: ModelPrecision.SMALL }) // Maps to q8
|
||||
|
||||
// Auto-detection (default)
|
||||
const brain = new Brainy({ model: ModelPrecision.AUTO })
|
||||
```
|
||||
|
||||
### Auto-Detection Logic
|
||||
|
||||
When not specified, Brainy automatically selects the best model:
|
||||
|
||||
- **Browser**: Q8 (smaller download)
|
||||
- **Serverless**: Q8 (faster cold starts)
|
||||
- **Low Memory (<512MB)**: Q8
|
||||
- **Development**: FP32 (best quality)
|
||||
- **Production (>2GB RAM)**: FP32
|
||||
- **Default**: Q8 (balanced)
|
||||
|
||||
## Storage Configuration
|
||||
|
||||
### Automatic Storage Detection
|
||||
|
||||
Brainy automatically detects the best storage option:
|
||||
|
||||
1. **Cloud Storage** (if credentials found)
|
||||
- AWS S3 (checks AWS_ACCESS_KEY_ID, AWS_PROFILE)
|
||||
- Google Cloud Storage (checks GOOGLE_APPLICATION_CREDENTIALS)
|
||||
- Cloudflare R2 (checks R2_ACCESS_KEY_ID)
|
||||
|
||||
2. **Browser Storage**
|
||||
- OPFS (if supported)
|
||||
- Filesystem fallback
|
||||
|
||||
3. **Node.js Storage**
|
||||
- Filesystem (`./brainy-data` or `~/.brainy/data`)
|
||||
|
||||
### Manual Storage Control
|
||||
|
||||
```typescript
|
||||
import { StorageOption } from '@soulcraft/brainy'
|
||||
|
||||
// Force specific storage with enum
|
||||
const brain = new Brainy({ storage: StorageOption.DISK })
|
||||
const brain = new Brainy({ storage: StorageOption.CLOUD })
|
||||
const brain = new Brainy({ storage: StorageOption.AUTO })
|
||||
|
||||
// Custom storage configuration
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
s3Storage: {
|
||||
bucket: 'my-bucket',
|
||||
region: 'us-east-1'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Feature Sets
|
||||
|
||||
Control which features are enabled:
|
||||
|
||||
```typescript
|
||||
import { FeatureSet } from '@soulcraft/brainy'
|
||||
|
||||
// Preset feature sets with enum
|
||||
const brain = new Brainy({ features: FeatureSet.MINIMAL }) // Core only
|
||||
const brain = new Brainy({ features: FeatureSet.DEFAULT }) // Balanced
|
||||
const brain = new Brainy({ features: FeatureSet.FULL }) // Everything
|
||||
|
||||
// Custom features
|
||||
const brain = new Brainy({
|
||||
features: ['core', 'search', 'cache', 'triple-intelligence']
|
||||
})
|
||||
```
|
||||
|
||||
## Simplified Configuration Interface
|
||||
|
||||
The new configuration is dramatically simpler:
|
||||
|
||||
```typescript
|
||||
interface BrainyZeroConfig {
|
||||
// Mode preset - now with distributed options
|
||||
mode?: PresetName // All strongly typed presets
|
||||
|
||||
// Model configuration with enum
|
||||
model?: ModelPrecision
|
||||
|
||||
// Storage configuration with enum
|
||||
storage?: StorageOption | StorageConfig
|
||||
|
||||
// Feature set with enum
|
||||
features?: FeatureSet | string[]
|
||||
|
||||
// Logging
|
||||
verbose?: boolean
|
||||
|
||||
// Escape hatch for advanced users
|
||||
advanced?: any
|
||||
}
|
||||
```
|
||||
|
||||
### Available Enums
|
||||
|
||||
```typescript
|
||||
enum PresetName {
|
||||
// Basic
|
||||
PRODUCTION = 'production',
|
||||
DEVELOPMENT = 'development',
|
||||
MINIMAL = 'minimal',
|
||||
ZERO = 'zero',
|
||||
|
||||
// Distributed
|
||||
WRITER = 'writer',
|
||||
READER = 'reader',
|
||||
|
||||
// Services
|
||||
INGESTION_SERVICE = 'ingestion-service',
|
||||
SEARCH_API = 'search-api',
|
||||
ANALYTICS_SERVICE = 'analytics-service',
|
||||
EDGE_CACHE = 'edge-cache',
|
||||
BATCH_PROCESSOR = 'batch-processor',
|
||||
STREAMING_SERVICE = 'streaming-service',
|
||||
ML_TRAINING = 'ml-training',
|
||||
SIDECAR = 'sidecar'
|
||||
}
|
||||
|
||||
enum ModelPrecision {
|
||||
FP32 = 'fp32',
|
||||
Q8 = 'q8',
|
||||
AUTO = 'auto',
|
||||
FAST = 'fast', // Maps to fp32
|
||||
SMALL = 'small' // Maps to q8
|
||||
}
|
||||
|
||||
enum StorageOption {
|
||||
AUTO = 'auto',
|
||||
DISK = 'disk',
|
||||
CLOUD = 'cloud'
|
||||
}
|
||||
|
||||
enum FeatureSet {
|
||||
MINIMAL = 'minimal',
|
||||
DEFAULT = 'default',
|
||||
FULL = 'full'
|
||||
}
|
||||
```
|
||||
|
||||
## Multi-Instance with Shared Storage
|
||||
|
||||
When multiple Brainy instances connect to the same storage (like S3), you **must ensure they use compatible configurations**:
|
||||
|
||||
```typescript
|
||||
import { ModelPrecision } from '@soulcraft/brainy'
|
||||
|
||||
// Container A - Writer
|
||||
const writer = new Brainy({
|
||||
mode: PresetName.WRITER,
|
||||
model: ModelPrecision.FP32, // ⚠️ MUST match across instances!
|
||||
storage: { s3Storage: { bucket: 'shared-data' }}
|
||||
})
|
||||
|
||||
// Container B - Reader
|
||||
const reader = new Brainy({
|
||||
mode: PresetName.READER,
|
||||
model: ModelPrecision.FP32, // ✅ Matches Container A
|
||||
storage: { s3Storage: { bucket: 'shared-data' }}
|
||||
})
|
||||
```
|
||||
|
||||
### Distributed Architecture Example
|
||||
|
||||
```typescript
|
||||
// Ingestion Service (Writer)
|
||||
const ingestion = new Brainy({
|
||||
mode: PresetName.INGESTION_SERVICE,
|
||||
model: ModelPrecision.Q8, // All instances must use Q8
|
||||
storage: { s3Storage: { bucket: 'production-data' }}
|
||||
})
|
||||
|
||||
// Search API (Reader)
|
||||
const search = new Brainy({
|
||||
mode: PresetName.SEARCH_API,
|
||||
model: ModelPrecision.Q8, // Matches ingestion service
|
||||
storage: { s3Storage: { bucket: 'production-data' }}
|
||||
})
|
||||
|
||||
// Analytics (Hybrid)
|
||||
const analytics = new Brainy({
|
||||
mode: PresetName.ANALYTICS_SERVICE,
|
||||
model: ModelPrecision.Q8, // Matches other services
|
||||
storage: { s3Storage: { bucket: 'production-data' }}
|
||||
})
|
||||
```
|
||||
|
||||
## Migration from Old Configuration
|
||||
|
||||
### Before (Complex)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
hnsw: {
|
||||
M: 16,
|
||||
efConstruction: 200,
|
||||
seed: 42
|
||||
},
|
||||
storage: {
|
||||
s3Storage: {
|
||||
bucketName: 'my-bucket',
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
|
||||
region: 'us-east-1'
|
||||
},
|
||||
cacheConfig: {
|
||||
hotCacheMaxSize: 5000,
|
||||
hotCacheEvictionThreshold: 0.8,
|
||||
warmCacheTTL: 3600000,
|
||||
batchSize: 100
|
||||
}
|
||||
},
|
||||
cache: {
|
||||
autoTune: true,
|
||||
autoTuneInterval: 60000,
|
||||
hotCacheMaxSize: 10000
|
||||
},
|
||||
embeddingFunction: customFunction,
|
||||
readOnly: false,
|
||||
logging: { verbose: true }
|
||||
})
|
||||
```
|
||||
|
||||
### After (Simple)
|
||||
```typescript
|
||||
const brain = new Brainy('production')
|
||||
// Everything above is auto-configured!
|
||||
```
|
||||
|
||||
## Environment Variables (No Longer Needed!)
|
||||
|
||||
These environment variables are **no longer required**:
|
||||
|
||||
- ❌ `BRAINY_ALLOW_REMOTE_MODELS` - Models auto-download when needed
|
||||
- ❌ `BRAINY_MODELS_PATH` - Path auto-selected based on environment
|
||||
- ❌ `BRAINY_Q8_CONFIRMED` - Warnings auto-suppressed in production
|
||||
- ❌ `BRAINY_LOG_LEVEL` - Auto-set based on NODE_ENV
|
||||
- ❌ AWS credentials - Use AWS SDK credential chain
|
||||
|
||||
## Performance Impact
|
||||
|
||||
The zero-config system has **zero performance overhead**:
|
||||
|
||||
- Configuration happens once during initialization
|
||||
- Auto-detected values are cached
|
||||
- Same optimized code paths as manual configuration
|
||||
- Actually **faster** startup due to reduced parsing
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Models Not Downloading
|
||||
- Check internet connection
|
||||
- Ensure firewall allows HTTPS to Hugging Face / CDN
|
||||
- Run `npm run download-models` to pre-download
|
||||
|
||||
### Wrong Model Precision
|
||||
- Explicitly specify: `{ model: 'fp32' }` or `{ model: 'q8' }`
|
||||
- Check shared storage compatibility
|
||||
|
||||
### Storage Detection Issues
|
||||
- Check cloud credentials are properly configured
|
||||
- Verify write permissions for filesystem paths
|
||||
- Use explicit storage configuration if needed
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use zero-config for single instances** - Let Brainy handle everything
|
||||
2. **Specify precision for shared storage** - Ensure compatibility
|
||||
3. **Use presets for common scenarios** - 'development', 'production', 'minimal'
|
||||
4. **Override only what you need** - Start simple, add complexity only if required
|
||||
|
||||
## Summary
|
||||
|
||||
The new zero-config system reduces configuration from **100+ parameters** to **0-3 decisions**:
|
||||
|
||||
| Scenario | Old Config Lines | New Config Lines |
|
||||
|----------|-----------------|------------------|
|
||||
| Development | 50+ | 1 |
|
||||
| Production | 100+ | 1 |
|
||||
| Custom | 200+ | 3-5 |
|
||||
|
||||
**Result**: 95% less configuration, 100% of the power! 🚀
|
||||
|
|
@ -1,152 +0,0 @@
|
|||
# Brainy API Return Values
|
||||
|
||||
## Core Operations
|
||||
|
||||
### `brain.add()` - Adding Entities
|
||||
|
||||
**Returns:** `Promise<string>` - The ID of the created entity
|
||||
|
||||
```typescript
|
||||
// ✅ Correct usage - add() returns the ID string directly
|
||||
const id = await brain.add({
|
||||
type: 'document',
|
||||
data: 'My document content'
|
||||
})
|
||||
|
||||
console.log('Created entity with ID:', id)
|
||||
|
||||
// Use the ID to create relationships
|
||||
await brain.relate({
|
||||
from: id,
|
||||
to: anotherId,
|
||||
type: 'references'
|
||||
})
|
||||
```
|
||||
|
||||
```typescript
|
||||
// ❌ Incorrect - trying to access .id property
|
||||
const result = await brain.add({
|
||||
type: 'document',
|
||||
data: 'My content'
|
||||
})
|
||||
|
||||
console.log(result.id) // ❌ undefined - result IS the ID, not an object!
|
||||
```
|
||||
|
||||
### Getting the Full Entity
|
||||
|
||||
If you need the full entity object after creation, use `brain.get()`:
|
||||
|
||||
```typescript
|
||||
// Add entity and get its ID
|
||||
const id = await brain.add({
|
||||
type: 'document',
|
||||
data: 'My content',
|
||||
metadata: { label: 'Important Doc' }
|
||||
})
|
||||
|
||||
// Get the full entity
|
||||
const entity = await brain.get(id)
|
||||
console.log(entity.id) // The ID
|
||||
console.log(entity.type) // 'document'
|
||||
console.log(entity.data) // 'My content'
|
||||
console.log(entity.metadata) // { label: 'Important Doc', ... }
|
||||
console.log(entity.vector) // The embedding vector
|
||||
console.log(entity.createdAt) // Timestamp
|
||||
```
|
||||
|
||||
### `brain.find()` - Finding Entities
|
||||
|
||||
**Returns:** `Promise<Entity[]>` - Array of full entity objects
|
||||
|
||||
```typescript
|
||||
const entities = await brain.find({
|
||||
query: 'machine learning',
|
||||
limit: 10
|
||||
})
|
||||
|
||||
// Each entity has full information
|
||||
for (const entity of entities) {
|
||||
console.log(entity.id)
|
||||
console.log(entity.type)
|
||||
console.log(entity.data)
|
||||
console.log(entity.metadata)
|
||||
}
|
||||
```
|
||||
|
||||
### `brain.relate()` - Creating Relationships
|
||||
|
||||
**Returns:** `Promise<string>` - The ID of the created relationship
|
||||
|
||||
```typescript
|
||||
const relationId = await brain.relate({
|
||||
from: entityId1,
|
||||
to: entityId2,
|
||||
type: 'references'
|
||||
})
|
||||
|
||||
console.log('Created relationship with ID:', relationId)
|
||||
```
|
||||
|
||||
## Data Field Behavior
|
||||
|
||||
### String Data - Used for Embeddings
|
||||
|
||||
When `data` is a string, it's used to generate embeddings for semantic search:
|
||||
|
||||
```typescript
|
||||
await brain.add({
|
||||
type: 'document',
|
||||
data: 'This text will be converted to an embedding vector',
|
||||
metadata: {
|
||||
title: 'My Document',
|
||||
year: 2024
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Object Data - Structured Information
|
||||
|
||||
When `data` is an object, it's treated as structured data:
|
||||
|
||||
```typescript
|
||||
await brain.add({
|
||||
type: 'product',
|
||||
data: {
|
||||
name: 'Widget',
|
||||
price: 29.99,
|
||||
category: 'Tools'
|
||||
},
|
||||
vector: precomputedVector // Must provide vector when using object data
|
||||
})
|
||||
```
|
||||
|
||||
**Important:** If you provide object data without a `vector`, you must include a string somewhere for embedding generation, or the operation will fail.
|
||||
|
||||
### Metadata vs Data
|
||||
|
||||
- **`data`**: Primary content - used for embeddings (if string) or stored as structured data (if object)
|
||||
- **`metadata`**: Auxiliary information - always stored as structured data, used for filtering
|
||||
|
||||
**Best practice for labels:**
|
||||
```typescript
|
||||
await brain.add({
|
||||
type: 'document',
|
||||
data: 'The full text content of the document...', // For semantic search
|
||||
metadata: {
|
||||
label: 'Quick Reference Label', // For display
|
||||
author: 'John Doe',
|
||||
category: 'Technical'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
| Method | Returns | Contains |
|
||||
|--------|---------|----------|
|
||||
| `brain.add()` | `string` | The ID of the created entity |
|
||||
| `brain.get()` | `Entity \| null` | Full entity object with all fields |
|
||||
| `brain.find()` | `Entity[]` | Array of full entity objects |
|
||||
| `brain.relate()` | `string` | The ID of the created relationship |
|
||||
| `brain.getRelations()` | `Relation[]` | Array of relationship objects |
|
||||
|
|
@ -1,563 +0,0 @@
|
|||
# Brainy Complete Public API Reference
|
||||
|
||||
> **Accurate API documentation for Brainy**
|
||||
|
||||
## Initialization
|
||||
|
||||
```typescript
|
||||
import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
// Zero-config (just works)
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// With configuration
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'memory' }, // or { path: './my-data' } for filesystem
|
||||
embeddingModel: 'Q8', // Q4, Q8, F16, F32
|
||||
silent: true // Suppress logs
|
||||
})
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
## Readiness API
|
||||
|
||||
For reliable initialization detection, especially in cloud environments with progressive initialization:
|
||||
|
||||
### `brain.ready` - Await Initialization
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
brain.init() // Fire and forget
|
||||
|
||||
// Elsewhere (e.g., API handler)
|
||||
await brain.ready // Wait until init() completes
|
||||
const results = await brain.find({ query: 'test' })
|
||||
```
|
||||
|
||||
### `brain.isInitialized` - Check Basic Readiness
|
||||
|
||||
```typescript
|
||||
if (brain.isInitialized) {
|
||||
// Safe to use brain methods
|
||||
}
|
||||
```
|
||||
|
||||
### `brain.isFullyInitialized()` - Check Background Tasks
|
||||
|
||||
```typescript
|
||||
// Returns true when ALL initialization is complete, including background tasks
|
||||
// Useful for cloud storage adapters with progressive initialization
|
||||
if (brain.isFullyInitialized()) {
|
||||
console.log('All background tasks complete')
|
||||
}
|
||||
```
|
||||
|
||||
### `brain.awaitBackgroundInit()` - Wait for Background Tasks
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||
await brain.init() // Fast return in cloud (<200ms)
|
||||
|
||||
// Optional: wait for all background tasks (bucket validation, count sync)
|
||||
await brain.awaitBackgroundInit()
|
||||
console.log('Fully initialized including background tasks')
|
||||
```
|
||||
|
||||
### Health Check Pattern
|
||||
|
||||
```typescript
|
||||
app.get('/health', async (req, res) => {
|
||||
try {
|
||||
await brain.ready
|
||||
res.json({
|
||||
status: 'ready',
|
||||
fullyInitialized: brain.isFullyInitialized()
|
||||
})
|
||||
} catch (error) {
|
||||
res.status(503).json({ status: 'initializing', error: error.message })
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Core CRUD Operations
|
||||
|
||||
### `brain.add(params)` - Add Entity
|
||||
|
||||
```typescript
|
||||
// Add with data (auto-embedded)
|
||||
const id = await brain.add({
|
||||
data: 'Machine learning is a subset of AI',
|
||||
type: NounType.Document,
|
||||
metadata: { author: 'Alice', tags: ['ml', 'ai'] }
|
||||
})
|
||||
|
||||
// Add with pre-computed vector
|
||||
const id = await brain.add({
|
||||
data: 'Content here',
|
||||
vector: [0.1, 0.2, ...], // 384 dimensions
|
||||
type: NounType.Concept
|
||||
})
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `data` (required): Content to embed (string, object, or any serializable data)
|
||||
- `type?`: NounType enum value
|
||||
- `metadata?`: Custom key-value pairs
|
||||
- `vector?`: Pre-computed embedding vector
|
||||
- `id?`: Custom ID (auto-generated if not provided)
|
||||
|
||||
### `brain.get(id, options?)` - Get Entity
|
||||
|
||||
```typescript
|
||||
const entity = await brain.get('entity-id')
|
||||
|
||||
// Include vector embeddings (not loaded by default for performance)
|
||||
const entity = await brain.get('entity-id', { includeVectors: true })
|
||||
```
|
||||
|
||||
### `brain.update(params)` - Update Entity
|
||||
|
||||
```typescript
|
||||
await brain.update({
|
||||
id: 'entity-id',
|
||||
data: 'Updated content', // Re-embeds if changed
|
||||
metadata: { reviewed: true } // Merges with existing
|
||||
})
|
||||
```
|
||||
|
||||
### `brain.delete(id)` - Delete Entity
|
||||
|
||||
```typescript
|
||||
await brain.delete('entity-id')
|
||||
```
|
||||
|
||||
### `brain.clear()` - Clear All Data
|
||||
|
||||
```typescript
|
||||
await brain.clear()
|
||||
```
|
||||
|
||||
## Relationships
|
||||
|
||||
### `brain.relate(params)` - Create Relationship
|
||||
|
||||
```typescript
|
||||
const relationId = await brain.relate({
|
||||
from: 'source-entity-id',
|
||||
to: 'target-entity-id',
|
||||
type: VerbType.RelatedTo,
|
||||
metadata: { strength: 0.9 }
|
||||
})
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `from` (required): Source entity ID
|
||||
- `to` (required): Target entity ID
|
||||
- `type` (required): VerbType enum value
|
||||
- `metadata?`: Custom relationship metadata
|
||||
|
||||
### `brain.getRelations(params)` - Query Relationships
|
||||
|
||||
```typescript
|
||||
// Get all relationships from an entity
|
||||
const relations = await brain.getRelations({ from: 'entity-id' })
|
||||
|
||||
// Get relationships to an entity
|
||||
const relations = await brain.getRelations({ to: 'entity-id' })
|
||||
|
||||
// Filter by type
|
||||
const relations = await brain.getRelations({
|
||||
from: 'entity-id',
|
||||
type: VerbType.Contains
|
||||
})
|
||||
```
|
||||
|
||||
### `brain.unrelate(id)` - Delete Relationship
|
||||
|
||||
```typescript
|
||||
await brain.unrelate('relationship-id')
|
||||
```
|
||||
|
||||
## Search & Query
|
||||
|
||||
### `brain.find(query)` - Semantic Search
|
||||
|
||||
The primary search method with Triple Intelligence (semantic + graph + metadata).
|
||||
|
||||
```typescript
|
||||
// Simple text search
|
||||
const results = await brain.find('machine learning algorithms')
|
||||
|
||||
// With options
|
||||
const results = await brain.find({
|
||||
query: 'machine learning',
|
||||
limit: 10,
|
||||
threshold: 0.7,
|
||||
type: NounType.Document,
|
||||
where: { author: 'Alice' },
|
||||
excludeVFS: true // Exclude VFS files from results
|
||||
})
|
||||
|
||||
// Natural language query (Triple Intelligence)
|
||||
const results = await brain.find(
|
||||
'Show me documents about AI written by Alice in 2024'
|
||||
)
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `query`: Search text (required)
|
||||
- `limit?`: Max results (default: 10)
|
||||
- `threshold?`: Minimum similarity (0-1)
|
||||
- `type?`: Filter by NounType
|
||||
- `where?`: Metadata filters
|
||||
- `excludeVFS?`: Exclude VFS entities
|
||||
|
||||
### `brain.similar(params)` - Find Similar Entities
|
||||
|
||||
```typescript
|
||||
// Find similar to entity ID
|
||||
const similar = await brain.similar({
|
||||
to: 'entity-id',
|
||||
limit: 5
|
||||
})
|
||||
|
||||
// Find similar to vector
|
||||
const similar = await brain.similar({
|
||||
to: [0.1, 0.2, ...], // Vector
|
||||
limit: 10,
|
||||
type: NounType.Document
|
||||
})
|
||||
```
|
||||
|
||||
## Batch Operations
|
||||
|
||||
### `brain.addMany(params)` - Batch Add
|
||||
|
||||
```typescript
|
||||
const result = await brain.addMany({
|
||||
items: [
|
||||
{ data: 'First item', type: NounType.Document },
|
||||
{ data: 'Second item', type: NounType.Concept },
|
||||
{ data: 'Third item', metadata: { priority: 'high' } }
|
||||
],
|
||||
continueOnError: true, // Don't stop on failures
|
||||
onProgress: (completed, total) => {
|
||||
console.log(`${completed}/${total} complete`)
|
||||
}
|
||||
})
|
||||
|
||||
console.log(`Added: ${result.successful.length}, Failed: ${result.failed.length}`)
|
||||
```
|
||||
|
||||
### `brain.deleteMany(params)` - Batch Delete
|
||||
|
||||
```typescript
|
||||
// Delete by IDs
|
||||
const result = await brain.deleteMany({
|
||||
ids: ['id1', 'id2', 'id3'],
|
||||
continueOnError: true
|
||||
})
|
||||
|
||||
// Delete by type
|
||||
const result = await brain.deleteMany({
|
||||
type: NounType.TempData
|
||||
})
|
||||
|
||||
// Delete by metadata filter
|
||||
const result = await brain.deleteMany({
|
||||
where: { status: 'archived' }
|
||||
})
|
||||
```
|
||||
|
||||
### `brain.relateMany(params)` - Batch Relate
|
||||
|
||||
```typescript
|
||||
const ids = await brain.relateMany({
|
||||
items: [
|
||||
{ from: 'a', to: 'b', type: VerbType.RelatedTo },
|
||||
{ from: 'b', to: 'c', type: VerbType.Contains },
|
||||
{ from: 'c', to: 'd', type: VerbType.References }
|
||||
],
|
||||
continueOnError: true
|
||||
})
|
||||
```
|
||||
|
||||
### `brain.updateMany(params)` - Batch Update
|
||||
|
||||
```typescript
|
||||
const result = await brain.updateMany({
|
||||
items: [
|
||||
{ id: 'id1', metadata: { reviewed: true } },
|
||||
{ id: 'id2', data: 'Updated content' }
|
||||
],
|
||||
continueOnError: true
|
||||
})
|
||||
```
|
||||
|
||||
## Neural API
|
||||
|
||||
Access advanced AI/ML features via `brain.neural()`:
|
||||
|
||||
```typescript
|
||||
const neural = brain.neural()
|
||||
|
||||
// Similarity calculation
|
||||
const similarity = await neural.similar('text1', 'text2')
|
||||
const detailed = await neural.similar('text1', 'text2', { detailed: true })
|
||||
|
||||
// Clustering
|
||||
const clusters = await neural.clusters()
|
||||
const clusters = await neural.clusters({
|
||||
algorithm: 'hierarchical',
|
||||
maxClusters: 10
|
||||
})
|
||||
|
||||
// K-nearest neighbors
|
||||
const neighbors = await neural.neighbors('entity-id', { k: 5 })
|
||||
|
||||
// Semantic hierarchy
|
||||
const hierarchy = await neural.hierarchy('entity-id')
|
||||
|
||||
// Outlier/anomaly detection
|
||||
const outliers = await neural.outliers({ threshold: 2.0 })
|
||||
|
||||
// Domain-aware clustering
|
||||
const domainClusters = await neural.clusterByDomain('category')
|
||||
|
||||
// Temporal clustering
|
||||
const temporalClusters = await neural.clusterByTime('createdAt', [
|
||||
{ start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1' },
|
||||
{ start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2' }
|
||||
])
|
||||
|
||||
// Streaming clusters (for large datasets)
|
||||
for await (const batch of neural.clusterStream({ batchSize: 100 })) {
|
||||
console.log(`Progress: ${batch.progress.percentage}%`)
|
||||
}
|
||||
```
|
||||
|
||||
## Virtual File System (VFS)
|
||||
|
||||
Access via `brain.vfs`:
|
||||
|
||||
```typescript
|
||||
const vfs = brain.vfs
|
||||
await vfs.init()
|
||||
|
||||
// File operations
|
||||
await vfs.writeFile('/docs/readme.md', '# Hello')
|
||||
const content = await vfs.readFile('/docs/readme.md')
|
||||
await vfs.unlink('/docs/readme.md')
|
||||
|
||||
// Directory operations
|
||||
await vfs.mkdir('/project/src', { recursive: true })
|
||||
const files = await vfs.readdir('/project')
|
||||
await vfs.rmdir('/project', { recursive: true })
|
||||
|
||||
// Bulk operations
|
||||
const result = await vfs.bulkWrite([
|
||||
{ type: 'mkdir', path: '/data' },
|
||||
{ type: 'write', path: '/data/config.json', data: '{}' },
|
||||
{ type: 'write', path: '/data/users.json', data: '[]' }
|
||||
])
|
||||
// Note: mkdir operations run first (sequentially), then other ops in parallel
|
||||
|
||||
// Semantic search
|
||||
const results = await vfs.search('authentication code', { path: '/src' })
|
||||
|
||||
// File metadata
|
||||
const stats = await vfs.stat('/file.txt')
|
||||
await vfs.setMetadata('/file.txt', { author: 'Alice' })
|
||||
```
|
||||
|
||||
## Counts (O(1) Performance)
|
||||
|
||||
```typescript
|
||||
// Entity counts
|
||||
const total = brain.counts.entities()
|
||||
const byType = await brain.counts.byType(NounType.Document)
|
||||
const nonVFS = await brain.counts.byType({ excludeVFS: true })
|
||||
|
||||
// Relationship counts
|
||||
const relations = brain.counts.relationships()
|
||||
const byVerb = await brain.counts.byVerbType(VerbType.Contains)
|
||||
```
|
||||
|
||||
## Versioning & Branching
|
||||
|
||||
```typescript
|
||||
// Create branch
|
||||
await brain.fork('feature-branch')
|
||||
|
||||
// List branches
|
||||
const branches = await brain.listBranches()
|
||||
|
||||
// Switch branch
|
||||
await brain.checkout('feature-branch')
|
||||
|
||||
// Get current branch
|
||||
const current = await brain.getCurrentBranch()
|
||||
|
||||
// Commit changes
|
||||
await brain.commit({ message: 'Added new features' })
|
||||
|
||||
// View history
|
||||
const history = await brain.getHistory({ limit: 10 })
|
||||
|
||||
// Time travel (read-only snapshot)
|
||||
const snapshot = await brain.asOf('commit-id')
|
||||
```
|
||||
|
||||
## Streaming API
|
||||
|
||||
```typescript
|
||||
// Stream all entities
|
||||
for await (const entity of brain.streaming.entities()) {
|
||||
console.log(entity.id)
|
||||
}
|
||||
|
||||
// Stream with filters
|
||||
for await (const entity of brain.streaming.entities({
|
||||
type: NounType.Document,
|
||||
where: { status: 'active' }
|
||||
})) {
|
||||
// Process each entity
|
||||
}
|
||||
|
||||
// Stream relationships
|
||||
for await (const relation of brain.streaming.relations({
|
||||
from: 'entity-id'
|
||||
})) {
|
||||
console.log(relation)
|
||||
}
|
||||
```
|
||||
|
||||
## Pagination API
|
||||
|
||||
```typescript
|
||||
// Paginated queries
|
||||
const page1 = await brain.pagination.find({
|
||||
query: 'machine learning',
|
||||
page: 1,
|
||||
pageSize: 20
|
||||
})
|
||||
|
||||
console.log(`Page ${page1.page} of ${page1.totalPages}`)
|
||||
console.log(`Total results: ${page1.total}`)
|
||||
|
||||
// Get next page
|
||||
const page2 = await brain.pagination.find({
|
||||
query: 'machine learning',
|
||||
page: 2,
|
||||
pageSize: 20
|
||||
})
|
||||
```
|
||||
|
||||
## Augmentations
|
||||
|
||||
```typescript
|
||||
// List active augmentations
|
||||
const augmentations = brain.augmentations.list()
|
||||
|
||||
// Get specific augmentation
|
||||
const cache = brain.augmentations.get('cache')
|
||||
|
||||
// Augmentations are auto-loaded based on config
|
||||
// Common augmentations: cache, display, metrics, intelligent-import
|
||||
```
|
||||
|
||||
## Utilities
|
||||
|
||||
```typescript
|
||||
// Manual embedding
|
||||
const vector = await brain.embed('text to embed')
|
||||
|
||||
// Flush pending writes
|
||||
await brain.flush()
|
||||
|
||||
// Get statistics
|
||||
const stats = await brain.getStats()
|
||||
const statsNoVFS = await brain.getStats({ excludeVFS: true })
|
||||
|
||||
// Close (cleanup)
|
||||
await brain.close()
|
||||
```
|
||||
|
||||
## Type Enums
|
||||
|
||||
```typescript
|
||||
import { NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
// NounType - Entity types
|
||||
NounType.Document
|
||||
NounType.Person
|
||||
NounType.Concept
|
||||
NounType.Event
|
||||
NounType.Location
|
||||
NounType.Organization
|
||||
NounType.Product
|
||||
NounType.Content
|
||||
NounType.Collection
|
||||
// ... and more
|
||||
|
||||
// VerbType - Relationship types
|
||||
VerbType.RelatedTo
|
||||
VerbType.Contains
|
||||
VerbType.References
|
||||
VerbType.DependsOn
|
||||
VerbType.Precedes
|
||||
VerbType.Follows
|
||||
VerbType.CreatedBy
|
||||
VerbType.ModifiedBy
|
||||
// ... and more
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```typescript
|
||||
try {
|
||||
await brain.get('nonexistent-id')
|
||||
} catch (error) {
|
||||
if (error.message.includes('not found')) {
|
||||
// Handle missing entity
|
||||
}
|
||||
}
|
||||
|
||||
// VFS errors use POSIX codes
|
||||
try {
|
||||
await vfs.readFile('/nonexistent')
|
||||
} catch (error) {
|
||||
if (error.code === 'ENOENT') {
|
||||
// File not found
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
// Storage
|
||||
storage: {
|
||||
type: 'memory' | 'filesystem',
|
||||
path: './data', // For filesystem
|
||||
forceMemoryStorage: false // Force memory even if path exists
|
||||
},
|
||||
|
||||
// Embeddings
|
||||
embeddingModel: 'Q8', // Q4, Q8, F16, F32
|
||||
dimensions: 384, // Auto-detected
|
||||
|
||||
// Performance
|
||||
silent: false, // Suppress console output
|
||||
verbose: false, // Extra logging
|
||||
|
||||
// Augmentations (auto-enabled by default)
|
||||
augmentations: {
|
||||
cache: true,
|
||||
display: true,
|
||||
metrics: true
|
||||
}
|
||||
})
|
||||
```
|
||||
|
|
@ -49,7 +49,13 @@ await brain.versions.save(id, { tag: 'v2.0' })
|
|||
Semantic vectors with metadata and relationships - the fundamental data unit in Brainy.
|
||||
|
||||
### 🔗 Relationships (Verbs)
|
||||
Typed connections between entities - building knowledge graphs.
|
||||
Typed connections between entities with optional `data` and `metadata` - building knowledge graphs.
|
||||
|
||||
### 📊 Data vs Metadata
|
||||
- **`data`**: Content embedded into vectors. Searchable via **semantic similarity** (HNSW) and **hybrid text+semantic** search. NOT queryable via `where` filters.
|
||||
- **`metadata`**: Structured fields indexed by MetadataIndex. Queryable via `where` filters in `find()`.
|
||||
|
||||
See **[Data Model](../DATA_MODEL.md)** for the full explanation.
|
||||
|
||||
### 🧠 Triple Intelligence
|
||||
Vector search + Graph traversal + Metadata filtering in one unified query.
|
||||
|
|
@ -99,9 +105,15 @@ const id = await brain.add({
|
|||
```
|
||||
|
||||
**Parameters:**
|
||||
- `data`: `string | number[]` - Text (auto-embeds) or vector
|
||||
- `data`: `string | number[]` - Content to embed (text auto-embeds) or pre-computed vector
|
||||
- `type`: `NounType` - Entity type (required)
|
||||
- `metadata?`: `object` - Additional metadata
|
||||
- `metadata?`: `object` - Structured queryable fields (indexed by MetadataIndex, used in `where` filters)
|
||||
- `id?`: `string` - Custom ID (auto-generated UUID if not provided)
|
||||
- `vector?`: `number[]` - Pre-computed vector (skips auto-embedding)
|
||||
- `confidence?`: `number` - Type classification confidence (0-1)
|
||||
- `weight?`: `number` - Entity importance/salience (0-1)
|
||||
|
||||
> **`data`** is embedded into vectors for semantic search. **`metadata`** is indexed for `where` filters. See [Data Model](../DATA_MODEL.md).
|
||||
|
||||
**Returns:** `Promise<string>` - Entity ID
|
||||
|
||||
|
|
@ -193,21 +205,27 @@ const results = await brain.find({
|
|||
- **Advanced:** Object with vector + graph + metadata filters
|
||||
|
||||
**FindParams:**
|
||||
- `query?`: `string` - Text for vector similarity
|
||||
- `where?`: `object` - Metadata filters (see [Query Operators](#query-operators))
|
||||
- `query?`: `string` - Text for semantic + hybrid search (searches `data` via HNSW + text index)
|
||||
- `type?`: `NounType | NounType[]` - Filter by entity type(s). Alias for `where.noun`.
|
||||
- `where?`: `object` - Metadata filters. See **[Query Operators](../QUERY_OPERATORS.md)** for all operators.
|
||||
- `connected?`: `object` - Graph traversal options
|
||||
- `to?`: `string` - Target entity ID
|
||||
- `from?`: `string` - Source entity ID
|
||||
- `type?`: `VerbType` - Relationship type
|
||||
- `depth?`: `number` - Traversal depth
|
||||
- `via?`: `VerbType | VerbType[]` - Relationship type(s) to traverse
|
||||
- `type?`: `VerbType | VerbType[]` - Alias for `via`
|
||||
- `depth?`: `number` - Traversal depth (default: 1)
|
||||
- `direction?`: `'in' | 'out' | 'both'` - Traversal direction (default: 'both')
|
||||
- `limit?`: `number` - Max results (default: 10)
|
||||
- `offset?`: `number` - Skip results
|
||||
- `orderBy?`: `string` - Field to sort by (e.g., 'createdAt', 'metadata.priority')
|
||||
- `order?`: `'asc' | 'desc'` - Sort direction (default: 'asc')
|
||||
- `searchMode?`: `'auto' | 'text' | 'semantic' | 'hybrid'` - Search strategy:
|
||||
- `'auto'` (default): Zero-config hybrid combining text + semantic search
|
||||
- `'text'`: Pure keyword/text matching
|
||||
- `'semantic'`: Pure vector similarity
|
||||
- `'semantic'`/`'vector'`: Pure vector similarity
|
||||
- `'hybrid'`: Explicit hybrid mode
|
||||
- `hybridAlpha?`: `number` - Balance between text (0.0) and semantic (1.0) search. Auto-detected by query length if not specified.
|
||||
- `excludeVFS?`: `boolean` - Exclude VFS entities from results (default: false)
|
||||
|
||||
**Returns:** `Promise<Result[]>` - Matching entities with scores
|
||||
|
||||
|
|
@ -358,22 +376,28 @@ highlights.forEach(h => {
|
|||
|
||||
### Query Operators
|
||||
|
||||
Brainy uses clean, readable operators:
|
||||
Brainy uses clean, readable operators (BFO — Brainy Field Operators):
|
||||
|
||||
| Operator | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `equals` | Exact match | `{age: {equals: 25}}` |
|
||||
| `greaterThan` | Greater than | `{age: {greaterThan: 18}}` |
|
||||
| `lessThan` | Less than | `{price: {lessThan: 100}}` |
|
||||
| `greaterEqual` | Greater or equal | `{score: {greaterEqual: 90}}` |
|
||||
| `lessEqual` | Less or equal | `{rating: {lessEqual: 3}}` |
|
||||
| `oneOf` | In array | `{color: {oneOf: ['red', 'blue']}}` |
|
||||
| `notOneOf` | Not in array | `{status: {notOneOf: ['deleted']}}` |
|
||||
| `contains` | Contains value | `{tags: {contains: 'ai'}}` |
|
||||
| `equals` / `eq` | Exact match | `{age: {equals: 25}}` |
|
||||
| `notEquals` / `ne` | Not equal | `{status: {notEquals: 'deleted'}}` |
|
||||
| `greaterThan` / `gt` | Greater than | `{age: {greaterThan: 18}}` |
|
||||
| `greaterEqual` / `gte` | Greater or equal | `{score: {greaterEqual: 90}}` |
|
||||
| `lessThan` / `lt` | Less than | `{price: {lessThan: 100}}` |
|
||||
| `lessEqual` / `lte` | Less or equal | `{rating: {lessEqual: 3}}` |
|
||||
| `between` | Inclusive range | `{year: {between: [2020, 2025]}}` |
|
||||
| `oneOf` / `in` | In array | `{color: {oneOf: ['red', 'blue']}}` |
|
||||
| `noneOf` | Not in array | `{status: {noneOf: ['deleted']}}` |
|
||||
| `contains` | Array contains value | `{tags: {contains: 'ai'}}` |
|
||||
| `exists` / `missing` | Field existence | `{email: {exists: true}}` |
|
||||
| `startsWith` | String prefix | `{name: {startsWith: 'John'}}` |
|
||||
| `endsWith` | String suffix | `{email: {endsWith: '@gmail.com'}}` |
|
||||
| `matches` | Pattern match | `{text: {matches: /^[A-Z]/}}` |
|
||||
| `between` | Range | `{year: {between: [2020, 2024]}}` |
|
||||
| `allOf` | AND combinator | `{allOf: [{active: true}, {role: 'admin'}]}` |
|
||||
| `anyOf` | OR combinator | `{anyOf: [{role: 'admin'}, {role: 'owner'}]}` |
|
||||
|
||||
**[Complete Operator Reference →](../QUERY_OPERATORS.md)** — all operators, aliases, indexed vs in-memory support matrix, and practical examples.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -388,18 +412,23 @@ const relId = await brain.relate({
|
|||
from: sourceId,
|
||||
to: targetId,
|
||||
type: VerbType.RelatedTo,
|
||||
metadata: { // Optional
|
||||
strength: 0.9,
|
||||
confidence: 0.85
|
||||
data: 'Collaborated on the research paper', // Optional: content for this edge
|
||||
metadata: { // Optional: structured edge fields
|
||||
strength: 0.9,
|
||||
role: 'primary author'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Parameters:**
|
||||
- `from`: `string` - Source entity ID
|
||||
- `to`: `string` - Target entity ID
|
||||
- `from`: `string` - Source entity ID (must exist)
|
||||
- `to`: `string` - Target entity ID (must exist)
|
||||
- `type`: `VerbType` - Relationship type
|
||||
- `metadata?`: `object` - Optional metadata
|
||||
- `data?`: `any` - Content for the relationship (overrides auto-computed vector)
|
||||
- `metadata?`: `object` - Structured edge fields
|
||||
- `weight?`: `number` - Connection strength (0-1, default: 1.0)
|
||||
- `bidirectional?`: `boolean` - Create reverse edge too (default: false)
|
||||
- `confidence?`: `number` - Relationship certainty (0-1)
|
||||
|
||||
**Returns:** `Promise<string>` - Relationship ID
|
||||
|
||||
|
|
@ -2153,7 +2182,10 @@ For the full taxonomy with all 169 types and their descriptions, see:
|
|||
|
||||
## See Also
|
||||
|
||||
- **[Data Model](../DATA_MODEL.md)** - Entity structure, data vs metadata, storage fields
|
||||
- **[Query Operators](../QUERY_OPERATORS.md)** - All BFO operators with examples and indexed vs in-memory matrix
|
||||
- **[Triple Intelligence Architecture](../architecture/triple-intelligence.md)** - How vector + graph + document work together
|
||||
- **[Find System](../FIND_SYSTEM.md)** - Natural language find() details
|
||||
- **[VFS Quick Start](../vfs/QUICK_START.md)** - Complete VFS documentation
|
||||
- **[Import Anything Guide](../guides/import-anything.md)** - CSV, Excel, PDF, URL imports
|
||||
- **[Cloud Deployment](../deployment/CLOUD_DEPLOYMENT_GUIDE.md)** - Production deployment
|
||||
|
|
|
|||
|
|
@ -1,200 +0,0 @@
|
|||
# Brainy Neural API Surface Design
|
||||
|
||||
## 🎯 **Clean API Hierarchy**
|
||||
|
||||
### **Main Class Shortcuts (Simple & Common)**
|
||||
```typescript
|
||||
// High-level shortcuts on Brainy - most common operations
|
||||
brain.similar(a, b, options?) // ✅ Keep - very common
|
||||
brain.clusters(items?, options?) // ✅ Keep - very common
|
||||
brain.related(id, options?) // ✅ Keep - common (like neighbors but simpler name)
|
||||
|
||||
// Remove/deprecate confusing shortcuts
|
||||
brain.visualize(options?) // ❌ Remove - too specialized for main class
|
||||
```
|
||||
|
||||
### **Neural Namespace (Full Featured)**
|
||||
```typescript
|
||||
// Core semantic operations
|
||||
brain.neural.similar(a, b, options?) // Comprehensive similarity
|
||||
brain.neural.clusters(items?, options?) // Smart clustering with auto-routing
|
||||
brain.neural.neighbors(id, options?) // K-nearest neighbors (full featured)
|
||||
brain.neural.hierarchy(id, options?) // Semantic hierarchy building
|
||||
brain.neural.outliers(options?) // Anomaly detection
|
||||
brain.neural.visualize(options?) // Visualization data generation
|
||||
|
||||
// Advanced clustering methods
|
||||
brain.neural.clusterByDomain(field, options?) // Domain-aware clustering
|
||||
brain.neural.clusterByTime(field, windows, options?) // Temporal clustering
|
||||
brain.neural.clusterStream(options?) // Streaming/real-time clustering
|
||||
brain.neural.updateClusters(items, options?) // Incremental clustering
|
||||
|
||||
// Utility methods
|
||||
brain.neural.getPerformanceMetrics(operation?) // Performance monitoring
|
||||
brain.neural.clearCaches() // Cache management
|
||||
brain.neural.getCacheStats() // Cache statistics
|
||||
```
|
||||
|
||||
## 🔒 **Private Methods (Internal Implementation)**
|
||||
|
||||
### **Should NOT be exposed publicly:**
|
||||
```typescript
|
||||
// These are implementation details:
|
||||
brain.neural._clusterFast() // ❌ Private - use clusters() with algorithm: 'hierarchical'
|
||||
brain.neural._clusterLarge() // ❌ Private - use clusters() with algorithm: 'sample'
|
||||
brain.neural._performHierarchicalClustering() // ❌ Private - internal routing
|
||||
brain.neural._performKMeansClustering() // ❌ Private - internal routing
|
||||
brain.neural._performDBSCANClustering() // ❌ Private - internal routing
|
||||
brain.neural._performSampledClustering() // ❌ Private - internal routing
|
||||
brain.neural._routeClusteringAlgorithm() // ❌ Private - internal routing
|
||||
brain.neural._similarityById() // ❌ Private - internal routing
|
||||
brain.neural._similarityByVector() // ❌ Private - internal routing
|
||||
brain.neural._similarityByText() // ❌ Private - internal routing
|
||||
brain.neural._isId() // ❌ Private - utility
|
||||
brain.neural._isVector() // ❌ Private - utility
|
||||
brain.neural._convertToVector() // ❌ Private - utility
|
||||
brain.neural._cacheResult() // ❌ Private - caching
|
||||
brain.neural._trackPerformance() // ❌ Private - monitoring
|
||||
```
|
||||
|
||||
### **Current Issues in brain-cloud explorer:**
|
||||
```typescript
|
||||
// ❌ BAD: Accessing private implementation details
|
||||
brain.neural.clusterFast({ maxClusters: count, level: 2 })
|
||||
|
||||
// ✅ GOOD: Use public API with proper options
|
||||
brain.neural.clusters({ algorithm: 'hierarchical', maxClusters: count, level: 2 })
|
||||
```
|
||||
|
||||
## 📊 **API Consistency Fixes**
|
||||
|
||||
### **Method Naming Standardization:**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: Pick one naming pattern and stick to it
|
||||
brain.neural.similar() // Main method name
|
||||
brain.similar() // Shortcut matches
|
||||
|
||||
// ❌ INCONSISTENT: Don't mix these
|
||||
brain.neural.similarity() // Different from shortcut
|
||||
brain.similar()
|
||||
```
|
||||
|
||||
### **Parameter Patterns:**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: Always (data, options?) pattern
|
||||
brain.neural.similar(a, b, options?)
|
||||
brain.neural.clusters(items?, options?)
|
||||
brain.neural.neighbors(id, options?)
|
||||
brain.neural.hierarchy(id, options?)
|
||||
|
||||
// Options should be objects with clear properties
|
||||
interface ClusteringOptions {
|
||||
algorithm?: 'auto' | 'hierarchical' | 'kmeans' | 'dbscan'
|
||||
maxClusters?: number
|
||||
threshold?: number
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### **Return Type Consistency:**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: All clustering methods return SemanticCluster[]
|
||||
brain.neural.clusters() → Promise<SemanticCluster[]>
|
||||
brain.neural.clusterByDomain() → Promise<DomainCluster[]> // extends SemanticCluster
|
||||
brain.neural.clusterByTime() → Promise<TemporalCluster[]> // extends SemanticCluster
|
||||
|
||||
// ✅ CONSISTENT: All similarity methods return number or SimilarityResult
|
||||
brain.neural.similar() → Promise<number | SimilarityResult>
|
||||
brain.similar() → Promise<number> // Shortcut always returns simple number
|
||||
```
|
||||
|
||||
## 🚀 **Performance & Intelligence Routing**
|
||||
|
||||
### **Auto-Algorithm Selection:**
|
||||
```typescript
|
||||
// Smart routing based on data size and characteristics
|
||||
brain.neural.clusters() // Auto-selects:
|
||||
// < 100 items → hierarchical (fast, accurate)
|
||||
// < 1000 items → k-means (balanced)
|
||||
// > 1000 items → sampling (scalable)
|
||||
|
||||
// Manual override available
|
||||
brain.neural.clusters({ algorithm: 'hierarchical' }) // Force specific algorithm
|
||||
```
|
||||
|
||||
### **Caching Strategy:**
|
||||
```typescript
|
||||
// Intelligent caching with LRU eviction
|
||||
brain.neural.similar('id1', 'id2') // First call: compute & cache
|
||||
brain.neural.similar('id1', 'id2') // Second call: instant cache hit
|
||||
|
||||
// Cache management
|
||||
brain.neural.clearCaches() // Manual cache clear
|
||||
brain.neural.getCacheStats() // Monitor cache performance
|
||||
```
|
||||
|
||||
## 📚 **Documentation Structure**
|
||||
|
||||
### **Main Documentation Sections:**
|
||||
1. **Quick Start**: Simple examples using shortcuts (`brain.similar()`, `brain.clusters()`)
|
||||
2. **Neural API Guide**: Comprehensive examples using `brain.neural.*`
|
||||
3. **Advanced Clustering**: Domain, temporal, streaming clustering
|
||||
4. **Performance**: Caching, algorithm selection, monitoring
|
||||
5. **API Reference**: Complete method documentation
|
||||
|
||||
### **Example Progression:**
|
||||
```typescript
|
||||
// 1. BEGINNER: Simple shortcuts
|
||||
const similarity = await brain.similar('text1', 'text2')
|
||||
const clusters = await brain.clusters()
|
||||
|
||||
// 2. INTERMEDIATE: Neural API with options
|
||||
const detailed = await brain.neural.similar('id1', 'id2', { detailed: true })
|
||||
const customClusters = await brain.neural.clusters({ algorithm: 'hierarchical', maxClusters: 5 })
|
||||
|
||||
// 3. ADVANCED: Specialized clustering
|
||||
const domainClusters = await brain.neural.clusterByDomain('category')
|
||||
const streamingClusters = brain.neural.clusterStream({ batchSize: 50 })
|
||||
```
|
||||
|
||||
## ✅ **Implementation Checklist**
|
||||
|
||||
### **High Priority:**
|
||||
- [x] Create comprehensive type definitions
|
||||
- [x] Implement improved NeuralAPI class with proper public/private separation
|
||||
- [ ] Update Brainy integration to use improved API
|
||||
- [ ] Fix brain-cloud explorer to use public APIs
|
||||
- [ ] Update test files to use consistent method names
|
||||
- [ ] Update documentation with new API structure
|
||||
|
||||
### **Medium Priority:**
|
||||
- [ ] Implement placeholder algorithm implementations with real clustering logic
|
||||
- [ ] Add comprehensive error handling and validation
|
||||
- [ ] Add performance monitoring and metrics collection
|
||||
- [ ] Create migration guide for users of deprecated methods
|
||||
|
||||
### **Nice to Have:**
|
||||
- [ ] Add interactive clustering refinement based on user feedback
|
||||
- [ ] Implement explainable clustering with reasoning
|
||||
- [ ] Add multi-modal clustering (text + metadata + relationships)
|
||||
- [ ] Create visualization examples for different graph libraries
|
||||
|
||||
## 🎯 **API Surface Summary**
|
||||
|
||||
### **✅ PUBLIC API** (What users should use):
|
||||
- **Main shortcuts**: `brain.similar()`, `brain.clusters()`, `brain.related()`
|
||||
- **Neural namespace**: `brain.neural.similar()`, `brain.neural.clusters()`, etc.
|
||||
- **Advanced features**: Domain clustering, temporal clustering, streaming
|
||||
- **Utilities**: Performance metrics, cache management
|
||||
|
||||
### **❌ PRIVATE IMPLEMENTATION** (Internal only):
|
||||
- **Algorithm implementations**: `_performKMeansClustering()`, etc.
|
||||
- **Routing logic**: `_routeClusteringAlgorithm()`, etc.
|
||||
- **Utility methods**: `_isId()`, `_convertToVector()`, etc.
|
||||
- **Caching internals**: `_cacheResult()`, `_trackPerformance()`, etc.
|
||||
|
||||
### **⚠️ DEPRECATED** (Should be removed/hidden):
|
||||
- **brain.neural.clusterFast()** → Use `brain.neural.clusters({ algorithm: 'hierarchical' })`
|
||||
- **brain.neural.clusterLarge()** → Use `brain.neural.clusters({ algorithm: 'sample' })`
|
||||
- **brain.neural.similarity()** → Use `brain.neural.similar()` (pick one name)
|
||||
- **brain.visualize()** → Too specialized for main class, use `brain.neural.visualize()`
|
||||
|
|
@ -1,220 +0,0 @@
|
|||
# Brainy Metadata Architecture & Namespacing
|
||||
|
||||
## The Problem 🚨
|
||||
We're mixing internal Brainy fields with user metadata, causing:
|
||||
1. **Namespace collisions** - User's `deleted` field conflicts with our soft-delete
|
||||
2. **API confusion** - Users see internal fields they shouldn't care about
|
||||
3. **Security issues** - Users could manipulate internal fields
|
||||
4. **Augmentation conflicts** - 3rd party augmentations might overwrite our fields
|
||||
|
||||
## Current Internal Fields Being Added
|
||||
|
||||
### Core System Fields
|
||||
```javascript
|
||||
metadata = {
|
||||
// USER DATA
|
||||
name: "Django",
|
||||
type: "framework",
|
||||
|
||||
// OUR INTERNAL FIELDS - COLLISION RISK!
|
||||
deleted: false, // Soft delete status
|
||||
domain: "tech", // Distributed mode domain
|
||||
domainMetadata: {}, // Domain-specific metadata
|
||||
partition: 0, // Partition for sharding
|
||||
createdAt: {...}, // GraphNoun timestamp
|
||||
updatedAt: {...}, // GraphNoun timestamp
|
||||
createdBy: {...}, // Who created this
|
||||
noun: "Concept", // NounType
|
||||
verb: "RELATES_TO", // VerbType (for relationships)
|
||||
isPlaceholder: true, // Write-only mode marker
|
||||
autoCreated: true, // Auto-created noun marker
|
||||
writeOnlyMode: true // High-speed streaming marker
|
||||
}
|
||||
```
|
||||
|
||||
### Augmentation Fields
|
||||
```javascript
|
||||
// Good - neuralImport already uses underscore prefix!
|
||||
metadata._neuralProcessed = true
|
||||
metadata._neuralConfidence = 0.95
|
||||
metadata._detectedEntities = 5
|
||||
metadata._detectedRelationships = 3
|
||||
metadata._neuralInsights = [...]
|
||||
|
||||
// Bad - direct modification
|
||||
metadata.importance = 0.8 // IntelligentVerbScoring
|
||||
```
|
||||
|
||||
## Proposed Solution: Three-Tier Metadata
|
||||
|
||||
### 1. User Metadata (Public)
|
||||
```javascript
|
||||
metadata = {
|
||||
// User's fields - completely untouched
|
||||
name: "Django",
|
||||
type: "framework",
|
||||
deleted: "2024-01-01", // User's own deleted field - no conflict!
|
||||
domain: "web", // User's domain field - no conflict!
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Internal Metadata (Protected)
|
||||
```javascript
|
||||
metadata._brainy = {
|
||||
// Core system fields - O(1) indexed
|
||||
deleted: false, // Our soft delete flag
|
||||
version: 2, // Metadata schema version
|
||||
|
||||
// Distributed mode
|
||||
partition: 0,
|
||||
distributedDomain: "tech",
|
||||
|
||||
// GraphNoun compliance
|
||||
nounType: "Concept",
|
||||
verbType: "RELATES_TO",
|
||||
createdAt: 1704067200000,
|
||||
updatedAt: 1704067200000,
|
||||
createdBy: "user:123",
|
||||
|
||||
// Performance flags
|
||||
indexed: true,
|
||||
searchable: true,
|
||||
placeholder: false,
|
||||
|
||||
// Storage optimization
|
||||
compressed: false,
|
||||
encrypted: false
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Augmentation Metadata (Semi-Protected)
|
||||
```javascript
|
||||
metadata._augmentations = {
|
||||
// Each augmentation gets its own namespace
|
||||
neuralImport: {
|
||||
processed: true,
|
||||
confidence: 0.95,
|
||||
entities: 5,
|
||||
relationships: 3
|
||||
},
|
||||
|
||||
verbScoring: {
|
||||
contextScore: 0.7,
|
||||
importance: 0.8
|
||||
},
|
||||
|
||||
// 3rd party augmentations
|
||||
customAug: {
|
||||
// Their fields isolated here
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
### Phase 1: Core Fields ✅
|
||||
```javascript
|
||||
// Already done with _brainy.deleted
|
||||
const BRAINY_NAMESPACE = '_brainy'
|
||||
const AUGMENTATION_NAMESPACE = '_augmentations'
|
||||
```
|
||||
|
||||
### Phase 2: Migrate All Internal Fields
|
||||
```javascript
|
||||
// Before
|
||||
metadata.domain = "tech"
|
||||
metadata.partition = 0
|
||||
|
||||
// After
|
||||
metadata._brainy.distributedDomain = "tech"
|
||||
metadata._brainy.partition = 0
|
||||
```
|
||||
|
||||
### Phase 3: Augmentation API
|
||||
```javascript
|
||||
class Augmentation {
|
||||
// Read user metadata (read-only)
|
||||
getUserMetadata(metadata) {
|
||||
const { _brainy, _augmentations, ...userMeta } = metadata
|
||||
return userMeta // Clean user data only
|
||||
}
|
||||
|
||||
// Write augmentation data (isolated)
|
||||
setAugmentationData(metadata, augName, data) {
|
||||
if (!metadata._augmentations) metadata._augmentations = {}
|
||||
metadata._augmentations[augName] = data
|
||||
}
|
||||
|
||||
// Read internal fields (for special augmentations only)
|
||||
getInternalField(metadata, field) {
|
||||
return metadata._brainy?.[field]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **No Collisions** - User can have any field names
|
||||
2. **O(1) Performance** - Internal fields still indexed
|
||||
3. **Clean API** - Users only see their data
|
||||
4. **Secure** - Internal fields protected
|
||||
5. **Extensible** - Augmentations isolated
|
||||
6. **Backward Compatible** - Migration path available
|
||||
|
||||
## Query Impact
|
||||
|
||||
### Before (Collision Risk)
|
||||
```javascript
|
||||
where: {
|
||||
deleted: false, // Ambiguous - ours or user's?
|
||||
type: "framework"
|
||||
}
|
||||
```
|
||||
|
||||
### After (Clear Separation)
|
||||
```javascript
|
||||
where: {
|
||||
'_brainy.deleted': false, // Our soft delete
|
||||
type: "framework" // User's field
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Index on `_brainy.deleted`**: O(1) hash lookup ✅
|
||||
- **Index on `_brainy.partition`**: O(1) for sharding ✅
|
||||
- **Nested field access**: Modern DBs handle this efficiently ✅
|
||||
- **Storage overhead**: ~100 bytes per item (acceptable) ✅
|
||||
|
||||
## Migration Path
|
||||
|
||||
1. **New items**: Automatically use namespaced fields
|
||||
2. **Existing items**: Lazy migration on update
|
||||
3. **Queries**: Support both formats temporarily
|
||||
4. **Deprecation**: Remove old format in v3.0
|
||||
|
||||
## Augmentation Guidelines
|
||||
|
||||
### For Core Augmentations
|
||||
- Use `_brainy.*` for system fields
|
||||
- Use `_augmentations.{name}.*` for augmentation data
|
||||
- Never modify user fields directly
|
||||
|
||||
### For 3rd Party Augmentations
|
||||
- Read user metadata via `getUserMetadata()`
|
||||
- Write only to `_augmentations.{yourName}.*`
|
||||
- Request permission for internal field access
|
||||
|
||||
## Critical Fields to Namespace
|
||||
|
||||
| Field | Current Location | New Location | Priority |
|
||||
|-------|-----------------|--------------|----------|
|
||||
| deleted | metadata.deleted | metadata._brainy.deleted | HIGH ✅ |
|
||||
| partition | metadata.partition | metadata._brainy.partition | HIGH |
|
||||
| domain | metadata.domain | metadata._brainy.distributedDomain | HIGH |
|
||||
| createdAt | metadata.createdAt | metadata._brainy.createdAt | MEDIUM |
|
||||
| updatedAt | metadata.updatedAt | metadata._brainy.updatedAt | MEDIUM |
|
||||
| noun | metadata.noun | metadata._brainy.nounType | MEDIUM |
|
||||
| verb | metadata.verb | metadata._brainy.verbType | MEDIUM |
|
||||
| isPlaceholder | metadata.isPlaceholder | metadata._brainy.placeholder | LOW |
|
||||
| autoCreated | metadata.autoCreated | metadata._brainy.autoCreated | LOW |
|
||||
|
|
@ -221,7 +221,7 @@ if (entity.vector.length > 0) {
|
|||
If you encounter migration issues:
|
||||
|
||||
1. Check the [VFS Performance Guide](../vfs/VFS_PERFORMANCE.md)
|
||||
2. Review [API Reference](../API_REFERENCE.md)
|
||||
2. Review [API Reference](../api/README.md)
|
||||
3. See [Performance Documentation](../PERFORMANCE.md)
|
||||
4. File an issue: https://github.com/soulcraft/brainy/issues
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue