feat: implement clean embedding architecture with Q8/FP32 precision control
- Unified embedding system with single EmbeddingManager - Q8 model support with 75% smaller footprint (23MB vs 90MB) - Intelligent precision auto-selection based on environment - Clean cached embeddings with TTL and memory management - Zero-config setup with smart defaults - Complete storage structure documentation - Removed legacy worker and hybrid managers - Streamlined model configuration and precision management
This commit is contained in:
parent
3227ad907c
commit
184d5dcf34
23 changed files with 1575 additions and 1369 deletions
349
COMPREHENSIVE_API_OVERVIEW.md
Normal file
349
COMPREHENSIVE_API_OVERVIEW.md
Normal file
|
|
@ -0,0 +1,349 @@
|
|||
# 🧠 **Brainy Complete Public API Overview**
|
||||
|
||||
> **Ultra-comprehensive analysis of Brainy's entire API surface for intuitive, consistent developer experience**
|
||||
|
||||
## 🎯 **API Consistency Analysis**
|
||||
|
||||
### **✅ EXCELLENT Consistency Patterns**
|
||||
|
||||
#### **1. Constructor & Initialization**
|
||||
```typescript
|
||||
// Clean, consistent initialization
|
||||
const brain = new BrainyData(config?)
|
||||
await brain.init() // Always required
|
||||
|
||||
// Storage auto-detection works seamlessly
|
||||
const brain = new BrainyData({ storage: { forceMemoryStorage: true } })
|
||||
const brain = new BrainyData({ storage: { path: './my-data' } })
|
||||
```
|
||||
|
||||
#### **2. Data Operations (CRUD)**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: Always (data, type, metadata) pattern
|
||||
await brain.addNoun(content, NounType.Person, { role: 'Engineer' })
|
||||
await brain.addNoun(content, NounType.Document, { title: 'API Guide' })
|
||||
|
||||
// ✅ CONSISTENT: Always (source, target, type, metadata) pattern
|
||||
await brain.addVerb(sourceId, targetId, VerbType.RelatedTo, { strength: 0.8 })
|
||||
await brain.addVerb(sourceId, targetId, VerbType.Contains, { confidence: 0.9 })
|
||||
|
||||
// ✅ CONSISTENT: Batch versions take arrays
|
||||
await brain.addNouns([...]) // Array of noun objects
|
||||
await brain.addVerbs([...]) // Array of verb objects
|
||||
```
|
||||
|
||||
#### **3. Query Operations**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: Always (query, options) pattern
|
||||
await brain.search('artificial intelligence', { limit: 10, threshold: 0.7 })
|
||||
await brain.find('recent documents about AI', { limit: 5 }) // Triple Intelligence
|
||||
|
||||
// ✅ CONSISTENT: Get methods with filters
|
||||
await brain.getNouns(filter?) // Optional filtering
|
||||
await brain.getVerbs(filter?) // Optional filtering
|
||||
await brain.getNoun(id) // Single item by ID
|
||||
await brain.getVerb(id) // Single item by ID
|
||||
```
|
||||
|
||||
#### **4. Main Class Shortcuts (Simple & Common)**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: Simple shortcuts for most common operations
|
||||
await brain.similar(a, b) // Returns simple number
|
||||
await brain.clusters() // Returns simple array
|
||||
await brain.related(id, limit?) // Returns simple array
|
||||
```
|
||||
|
||||
### **🎨 EXCELLENT API Namespacing**
|
||||
|
||||
#### **Main Data Operations** (Direct on `brain`)
|
||||
```typescript
|
||||
// Core CRUD - most common operations
|
||||
brain.addNoun(content, type, metadata?)
|
||||
brain.addNouns(items[])
|
||||
brain.addVerb(source, target, type, metadata?)
|
||||
brain.addVerbs(items[])
|
||||
|
||||
brain.search(query, options?)
|
||||
brain.find(naturalLanguageQuery, options?) // Triple Intelligence
|
||||
brain.get(id)
|
||||
brain.getNouns(filter?)
|
||||
brain.getVerbs(filter?)
|
||||
|
||||
brain.delete(id)
|
||||
brain.deleteNouns(ids[])
|
||||
brain.deleteVerbs(ids[])
|
||||
brain.clear()
|
||||
|
||||
// Simple shortcuts for common AI operations
|
||||
brain.similar(a, b) // Simple similarity
|
||||
brain.clusters() // Simple clustering
|
||||
brain.related(id, limit?) // Simple neighbors
|
||||
```
|
||||
|
||||
#### **Neural AI Namespace** (`brain.neural.*`)
|
||||
```typescript
|
||||
// Advanced AI & Machine Learning operations
|
||||
brain.neural.similar(a, b, options?) // Full similarity with options
|
||||
brain.neural.clusters(items?, options?) // Advanced clustering
|
||||
brain.neural.neighbors(id, options?) // K-nearest neighbors
|
||||
brain.neural.hierarchy(id, options?) // Semantic hierarchy
|
||||
brain.neural.outliers(options?) // Anomaly detection
|
||||
brain.neural.visualize(options?) // Visualization data
|
||||
|
||||
// Advanced clustering methods
|
||||
brain.neural.clusterByDomain(field, options?) // Domain-aware clustering
|
||||
brain.neural.clusterByTime(field, windows, options?) // Temporal clustering
|
||||
brain.neural.clusterStream(options?) // Streaming clustering
|
||||
brain.neural.updateClusters(items, options?) // Incremental clustering
|
||||
|
||||
// Utility & monitoring
|
||||
brain.neural.getPerformanceMetrics(operation?) // Performance stats
|
||||
brain.neural.clearCaches() // Cache management
|
||||
brain.neural.getCacheStats() // Cache statistics
|
||||
```
|
||||
|
||||
#### **Triple Intelligence Namespace** (`brain.triple.*`)
|
||||
```typescript
|
||||
// Advanced natural language & complex queries
|
||||
brain.triple.find(query, options?) // Natural language search
|
||||
brain.triple.analyze(text, options?) // Text analysis
|
||||
brain.triple.understand(query, options?) // Query understanding
|
||||
```
|
||||
|
||||
#### **Augmentation System** (`brain.augmentations.*`)
|
||||
```typescript
|
||||
// Plugin/extension system
|
||||
brain.augmentations.add(augmentation)
|
||||
brain.augmentations.remove(name)
|
||||
brain.augmentations.get(name)
|
||||
brain.augmentations.list()
|
||||
brain.augmentations.execute(operation, params)
|
||||
```
|
||||
|
||||
#### **Storage & System** (`brain.storage.*`)
|
||||
```typescript
|
||||
// Storage management
|
||||
brain.storage.backup(path?)
|
||||
brain.storage.restore(path?)
|
||||
brain.storage.getStatistics()
|
||||
brain.storage.optimize()
|
||||
brain.storage.vacuum()
|
||||
```
|
||||
|
||||
### **🚀 API Flow & Developer Experience**
|
||||
|
||||
#### **1. Beginner Flow (Simple & Intuitive)**
|
||||
```typescript
|
||||
// Dead simple - just works
|
||||
const brain = new BrainyData()
|
||||
await brain.init()
|
||||
|
||||
await brain.addNoun('My first document', NounType.Document)
|
||||
const results = await brain.search('document')
|
||||
const similar = await brain.similar('text1', 'text2')
|
||||
const groups = await brain.clusters()
|
||||
```
|
||||
|
||||
#### **2. Intermediate Flow (More Control)**
|
||||
```typescript
|
||||
// Add configuration and options
|
||||
const brain = new BrainyData({
|
||||
storage: { path: './my-brainy-db' },
|
||||
neural: { cacheSize: 5000 }
|
||||
})
|
||||
await brain.init()
|
||||
|
||||
// Use options for better control
|
||||
const results = await brain.search('AI research', {
|
||||
limit: 20,
|
||||
threshold: 0.8,
|
||||
filters: { type: 'Document', year: 2024 }
|
||||
})
|
||||
|
||||
// Use neural namespace for advanced features
|
||||
const clusters = await brain.neural.clusters({
|
||||
algorithm: 'hierarchical',
|
||||
maxClusters: 10
|
||||
})
|
||||
```
|
||||
|
||||
#### **3. Advanced Flow (Full Power)**
|
||||
```typescript
|
||||
// Complex natural language queries
|
||||
const insights = await brain.find(`
|
||||
Show me documents about machine learning from 2024
|
||||
that are connected to research papers with high citations
|
||||
`)
|
||||
|
||||
// Advanced temporal analysis
|
||||
const trends = await brain.neural.clusterByTime('publishedAt', [
|
||||
{ start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1 2024' },
|
||||
{ start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2 2024' }
|
||||
])
|
||||
|
||||
// Real-time streaming clustering
|
||||
for await (const batch of brain.neural.clusterStream({ batchSize: 50 })) {
|
||||
console.log(`Processed ${batch.progress.percentage}% - Found ${batch.clusters.length} clusters`)
|
||||
}
|
||||
```
|
||||
|
||||
## 📊 **Parameter Consistency Analysis**
|
||||
|
||||
### **✅ Excellent Consistency**
|
||||
|
||||
#### **1. Data-First Pattern**
|
||||
```typescript
|
||||
// Always: (data, type/config, optional_metadata)
|
||||
brain.addNoun(content, NounType.Document, metadata?)
|
||||
brain.addVerb(source, target, VerbType.RelatedTo, metadata?)
|
||||
brain.search(query, options?)
|
||||
brain.similar(a, b, options?)
|
||||
```
|
||||
|
||||
#### **2. Options Objects**
|
||||
```typescript
|
||||
// Consistent options pattern across all methods
|
||||
{
|
||||
limit?: number
|
||||
threshold?: number
|
||||
filters?: Record<string, any>
|
||||
algorithm?: string
|
||||
includeMetadata?: boolean
|
||||
}
|
||||
```
|
||||
|
||||
#### **3. Array Methods**
|
||||
```typescript
|
||||
// Pluralized versions always take arrays
|
||||
brain.addNouns([{ vectorOrData: '...', nounType: NounType.Content }])
|
||||
brain.addVerbs([{ source: '...', target: '...', type: VerbType.RelatedTo }])
|
||||
brain.deleteNouns(['id1', 'id2'])
|
||||
brain.deleteVerbs(['id1', 'id2'])
|
||||
```
|
||||
|
||||
### **Return Type Consistency**
|
||||
|
||||
#### **1. Simple Returns (Shortcuts)**
|
||||
```typescript
|
||||
brain.similar(a, b) → Promise<number> // Always simple number
|
||||
brain.clusters() → Promise<SemanticCluster[]> // Always simple array
|
||||
brain.related(id) → Promise<Neighbor[]> // Always simple array
|
||||
```
|
||||
|
||||
#### **2. Rich Returns (Neural Namespace)**
|
||||
```typescript
|
||||
brain.neural.similar(a, b, { detailed: true }) → Promise<SimilarityResult>
|
||||
brain.neural.neighbors(id, options) → Promise<NeighborsResult>
|
||||
brain.neural.clusters(options) → Promise<SemanticCluster[]>
|
||||
```
|
||||
|
||||
#### **3. Consistent Error Handling**
|
||||
```typescript
|
||||
// All methods throw descriptive errors with context
|
||||
try {
|
||||
await brain.neural.similar('invalid', 'data')
|
||||
} catch (error) {
|
||||
// error.code: 'SIMILARITY_ERROR'
|
||||
// error.context: { inputA: '...', inputB: '...' }
|
||||
}
|
||||
```
|
||||
|
||||
## 🎯 **Key Strengths of Current API**
|
||||
|
||||
### **✅ 1. Progressive Disclosure**
|
||||
- **Simple**: `brain.similar()` → just returns a number
|
||||
- **Advanced**: `brain.neural.similar()` → full options & detailed results
|
||||
|
||||
### **✅ 2. Intuitive Namespacing**
|
||||
- **Core data**: Direct on `brain` (addNoun, search, delete)
|
||||
- **AI features**: `brain.neural.*` (clustering, similarity, analysis)
|
||||
- **System**: `brain.storage.*`, `brain.augmentations.*`
|
||||
|
||||
### **✅ 3. Consistent Patterns**
|
||||
- **Always** `(data, options?)` parameter order
|
||||
- **Always** async/Promise-based
|
||||
- **Always** descriptive error messages with context
|
||||
|
||||
### **✅ 4. Type Safety**
|
||||
```typescript
|
||||
// Excellent TypeScript support
|
||||
import { BrainyData, NounType, VerbType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new BrainyData()
|
||||
await brain.addNoun('content', NounType.Document, { title: 'My Doc' })
|
||||
// ^^^^^^^^^^^^^^^^ // IDE autocomplete!
|
||||
```
|
||||
|
||||
### **✅ 5. Flexible Configuration**
|
||||
```typescript
|
||||
// Zero-config (just works)
|
||||
const brain = new BrainyData()
|
||||
|
||||
// Full control when needed
|
||||
const brain = new BrainyData({
|
||||
storage: {
|
||||
adapter: 'file',
|
||||
path: './my-data',
|
||||
encryption: true
|
||||
},
|
||||
neural: {
|
||||
cacheSize: 10000,
|
||||
defaultAlgorithm: 'hierarchical'
|
||||
},
|
||||
logging: { verbose: true }
|
||||
})
|
||||
```
|
||||
|
||||
## 🔍 **Minor Improvement Opportunities**
|
||||
|
||||
### **1. Documentation Consistency**
|
||||
```typescript
|
||||
// ✅ GREAT: Clear, descriptive JSDoc
|
||||
/**
|
||||
* Add semantic relationship between two items
|
||||
* @param source - Source item ID
|
||||
* @param target - Target item ID
|
||||
* @param type - Relationship type (VerbType enum)
|
||||
* @param metadata - Optional relationship metadata
|
||||
*/
|
||||
brain.addVerb(source, target, type, metadata?)
|
||||
```
|
||||
|
||||
### **2. Error Context Enhancement**
|
||||
```typescript
|
||||
// Current: Good error messages
|
||||
// Improvement: Add suggested fixes
|
||||
throw new SimilarityError('Failed to calculate similarity', {
|
||||
inputA: 'invalid-id',
|
||||
inputB: 'valid-id',
|
||||
suggestion: 'Check that both IDs exist in the database'
|
||||
})
|
||||
```
|
||||
|
||||
## 🎖️ **Overall API Grade: A+ (Excellent)**
|
||||
|
||||
### **Strengths:**
|
||||
- **🎯 Intuitive**: Natural method names, clear hierarchy
|
||||
- **🔄 Consistent**: Same patterns everywhere
|
||||
- **📈 Progressive**: Simple → advanced as needed
|
||||
- **🛡️ Type-safe**: Full TypeScript support
|
||||
- **📚 Well-documented**: Clear examples & guides
|
||||
- **🚀 Performant**: Smart caching, batching, streaming
|
||||
|
||||
### **Neural API Fits Perfectly:**
|
||||
- **✅ Namespace consistency**: `brain.neural.*` is clear and logical
|
||||
- **✅ Parameter consistency**: Follows same `(data, options?)` pattern
|
||||
- **✅ Return consistency**: Rich objects when needed, simple types for shortcuts
|
||||
- **✅ Progressive disclosure**: `brain.similar()` → `brain.neural.similar()`
|
||||
- **✅ Advanced features**: Domain/temporal clustering, streaming, analysis
|
||||
|
||||
### **Developer Experience Score: 🌟🌟🌟🌟🌟 (5/5 stars)**
|
||||
|
||||
The API surface is **exceptionally well designed** with:
|
||||
- **Beginner-friendly** shortcuts that "just work"
|
||||
- **Advanced features** available when needed
|
||||
- **Consistent patterns** across all methods
|
||||
- **Logical namespacing** that guides developers naturally
|
||||
- **Rich ecosystem** with augmentations, Triple Intelligence, and neural features
|
||||
|
||||
**The neural namespace integrates seamlessly and enhances rather than complicates the overall API experience.**
|
||||
Loading…
Add table
Add a link
Reference in a new issue