feat: enforce data/metadata separation, numeric range queries, improved docs
- Store data opaquely in add() and update() instead of spreading object properties into top-level metadata. data is for semantic search (HNSW), metadata is for structured where-filter queries (MetadataIndex). - Fix numeric range queries in MetadataIndex — use numeric-aware comparison instead of lexicographic string comparison for normalized values. - Add data field to RelateParams and Relation types for relationship content. - Add where.type → where.noun alias in metadata-only find() path. - Rewrite README: focused ~350 lines from 791, quick start first, feature showcase with mini-snippets, organized doc links, no version callouts. - Add DATA_MODEL.md and QUERY_OPERATORS.md reference docs. - Remove 10 outdated/redundant doc files consolidated into API reference. - Improve JSDoc on Entity, Relation, AddParams, FindParams, and core methods. - Fix tests asserting data properties appear in metadata (data model violation). - Deprecate verb.source/target in favor of from/to (public) and sourceId/targetId (storage).
This commit is contained in:
parent
edb5ec4696
commit
0ddc05a5bb
30 changed files with 1356 additions and 7001 deletions
|
|
@ -1,200 +0,0 @@
|
|||
# Brainy Neural API Surface Design
|
||||
|
||||
## 🎯 **Clean API Hierarchy**
|
||||
|
||||
### **Main Class Shortcuts (Simple & Common)**
|
||||
```typescript
|
||||
// High-level shortcuts on Brainy - most common operations
|
||||
brain.similar(a, b, options?) // ✅ Keep - very common
|
||||
brain.clusters(items?, options?) // ✅ Keep - very common
|
||||
brain.related(id, options?) // ✅ Keep - common (like neighbors but simpler name)
|
||||
|
||||
// Remove/deprecate confusing shortcuts
|
||||
brain.visualize(options?) // ❌ Remove - too specialized for main class
|
||||
```
|
||||
|
||||
### **Neural Namespace (Full Featured)**
|
||||
```typescript
|
||||
// Core semantic operations
|
||||
brain.neural.similar(a, b, options?) // Comprehensive similarity
|
||||
brain.neural.clusters(items?, options?) // Smart clustering with auto-routing
|
||||
brain.neural.neighbors(id, options?) // K-nearest neighbors (full featured)
|
||||
brain.neural.hierarchy(id, options?) // Semantic hierarchy building
|
||||
brain.neural.outliers(options?) // Anomaly detection
|
||||
brain.neural.visualize(options?) // Visualization data generation
|
||||
|
||||
// Advanced clustering methods
|
||||
brain.neural.clusterByDomain(field, options?) // Domain-aware clustering
|
||||
brain.neural.clusterByTime(field, windows, options?) // Temporal clustering
|
||||
brain.neural.clusterStream(options?) // Streaming/real-time clustering
|
||||
brain.neural.updateClusters(items, options?) // Incremental clustering
|
||||
|
||||
// Utility methods
|
||||
brain.neural.getPerformanceMetrics(operation?) // Performance monitoring
|
||||
brain.neural.clearCaches() // Cache management
|
||||
brain.neural.getCacheStats() // Cache statistics
|
||||
```
|
||||
|
||||
## 🔒 **Private Methods (Internal Implementation)**
|
||||
|
||||
### **Should NOT be exposed publicly:**
|
||||
```typescript
|
||||
// These are implementation details:
|
||||
brain.neural._clusterFast() // ❌ Private - use clusters() with algorithm: 'hierarchical'
|
||||
brain.neural._clusterLarge() // ❌ Private - use clusters() with algorithm: 'sample'
|
||||
brain.neural._performHierarchicalClustering() // ❌ Private - internal routing
|
||||
brain.neural._performKMeansClustering() // ❌ Private - internal routing
|
||||
brain.neural._performDBSCANClustering() // ❌ Private - internal routing
|
||||
brain.neural._performSampledClustering() // ❌ Private - internal routing
|
||||
brain.neural._routeClusteringAlgorithm() // ❌ Private - internal routing
|
||||
brain.neural._similarityById() // ❌ Private - internal routing
|
||||
brain.neural._similarityByVector() // ❌ Private - internal routing
|
||||
brain.neural._similarityByText() // ❌ Private - internal routing
|
||||
brain.neural._isId() // ❌ Private - utility
|
||||
brain.neural._isVector() // ❌ Private - utility
|
||||
brain.neural._convertToVector() // ❌ Private - utility
|
||||
brain.neural._cacheResult() // ❌ Private - caching
|
||||
brain.neural._trackPerformance() // ❌ Private - monitoring
|
||||
```
|
||||
|
||||
### **Current Issues in brain-cloud explorer:**
|
||||
```typescript
|
||||
// ❌ BAD: Accessing private implementation details
|
||||
brain.neural.clusterFast({ maxClusters: count, level: 2 })
|
||||
|
||||
// ✅ GOOD: Use public API with proper options
|
||||
brain.neural.clusters({ algorithm: 'hierarchical', maxClusters: count, level: 2 })
|
||||
```
|
||||
|
||||
## 📊 **API Consistency Fixes**
|
||||
|
||||
### **Method Naming Standardization:**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: Pick one naming pattern and stick to it
|
||||
brain.neural.similar() // Main method name
|
||||
brain.similar() // Shortcut matches
|
||||
|
||||
// ❌ INCONSISTENT: Don't mix these
|
||||
brain.neural.similarity() // Different from shortcut
|
||||
brain.similar()
|
||||
```
|
||||
|
||||
### **Parameter Patterns:**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: Always (data, options?) pattern
|
||||
brain.neural.similar(a, b, options?)
|
||||
brain.neural.clusters(items?, options?)
|
||||
brain.neural.neighbors(id, options?)
|
||||
brain.neural.hierarchy(id, options?)
|
||||
|
||||
// Options should be objects with clear properties
|
||||
interface ClusteringOptions {
|
||||
algorithm?: 'auto' | 'hierarchical' | 'kmeans' | 'dbscan'
|
||||
maxClusters?: number
|
||||
threshold?: number
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### **Return Type Consistency:**
|
||||
```typescript
|
||||
// ✅ CONSISTENT: All clustering methods return SemanticCluster[]
|
||||
brain.neural.clusters() → Promise<SemanticCluster[]>
|
||||
brain.neural.clusterByDomain() → Promise<DomainCluster[]> // extends SemanticCluster
|
||||
brain.neural.clusterByTime() → Promise<TemporalCluster[]> // extends SemanticCluster
|
||||
|
||||
// ✅ CONSISTENT: All similarity methods return number or SimilarityResult
|
||||
brain.neural.similar() → Promise<number | SimilarityResult>
|
||||
brain.similar() → Promise<number> // Shortcut always returns simple number
|
||||
```
|
||||
|
||||
## 🚀 **Performance & Intelligence Routing**
|
||||
|
||||
### **Auto-Algorithm Selection:**
|
||||
```typescript
|
||||
// Smart routing based on data size and characteristics
|
||||
brain.neural.clusters() // Auto-selects:
|
||||
// < 100 items → hierarchical (fast, accurate)
|
||||
// < 1000 items → k-means (balanced)
|
||||
// > 1000 items → sampling (scalable)
|
||||
|
||||
// Manual override available
|
||||
brain.neural.clusters({ algorithm: 'hierarchical' }) // Force specific algorithm
|
||||
```
|
||||
|
||||
### **Caching Strategy:**
|
||||
```typescript
|
||||
// Intelligent caching with LRU eviction
|
||||
brain.neural.similar('id1', 'id2') // First call: compute & cache
|
||||
brain.neural.similar('id1', 'id2') // Second call: instant cache hit
|
||||
|
||||
// Cache management
|
||||
brain.neural.clearCaches() // Manual cache clear
|
||||
brain.neural.getCacheStats() // Monitor cache performance
|
||||
```
|
||||
|
||||
## 📚 **Documentation Structure**
|
||||
|
||||
### **Main Documentation Sections:**
|
||||
1. **Quick Start**: Simple examples using shortcuts (`brain.similar()`, `brain.clusters()`)
|
||||
2. **Neural API Guide**: Comprehensive examples using `brain.neural.*`
|
||||
3. **Advanced Clustering**: Domain, temporal, streaming clustering
|
||||
4. **Performance**: Caching, algorithm selection, monitoring
|
||||
5. **API Reference**: Complete method documentation
|
||||
|
||||
### **Example Progression:**
|
||||
```typescript
|
||||
// 1. BEGINNER: Simple shortcuts
|
||||
const similarity = await brain.similar('text1', 'text2')
|
||||
const clusters = await brain.clusters()
|
||||
|
||||
// 2. INTERMEDIATE: Neural API with options
|
||||
const detailed = await brain.neural.similar('id1', 'id2', { detailed: true })
|
||||
const customClusters = await brain.neural.clusters({ algorithm: 'hierarchical', maxClusters: 5 })
|
||||
|
||||
// 3. ADVANCED: Specialized clustering
|
||||
const domainClusters = await brain.neural.clusterByDomain('category')
|
||||
const streamingClusters = brain.neural.clusterStream({ batchSize: 50 })
|
||||
```
|
||||
|
||||
## ✅ **Implementation Checklist**
|
||||
|
||||
### **High Priority:**
|
||||
- [x] Create comprehensive type definitions
|
||||
- [x] Implement improved NeuralAPI class with proper public/private separation
|
||||
- [ ] Update Brainy integration to use improved API
|
||||
- [ ] Fix brain-cloud explorer to use public APIs
|
||||
- [ ] Update test files to use consistent method names
|
||||
- [ ] Update documentation with new API structure
|
||||
|
||||
### **Medium Priority:**
|
||||
- [ ] Implement placeholder algorithm implementations with real clustering logic
|
||||
- [ ] Add comprehensive error handling and validation
|
||||
- [ ] Add performance monitoring and metrics collection
|
||||
- [ ] Create migration guide for users of deprecated methods
|
||||
|
||||
### **Nice to Have:**
|
||||
- [ ] Add interactive clustering refinement based on user feedback
|
||||
- [ ] Implement explainable clustering with reasoning
|
||||
- [ ] Add multi-modal clustering (text + metadata + relationships)
|
||||
- [ ] Create visualization examples for different graph libraries
|
||||
|
||||
## 🎯 **API Surface Summary**
|
||||
|
||||
### **✅ PUBLIC API** (What users should use):
|
||||
- **Main shortcuts**: `brain.similar()`, `brain.clusters()`, `brain.related()`
|
||||
- **Neural namespace**: `brain.neural.similar()`, `brain.neural.clusters()`, etc.
|
||||
- **Advanced features**: Domain clustering, temporal clustering, streaming
|
||||
- **Utilities**: Performance metrics, cache management
|
||||
|
||||
### **❌ PRIVATE IMPLEMENTATION** (Internal only):
|
||||
- **Algorithm implementations**: `_performKMeansClustering()`, etc.
|
||||
- **Routing logic**: `_routeClusteringAlgorithm()`, etc.
|
||||
- **Utility methods**: `_isId()`, `_convertToVector()`, etc.
|
||||
- **Caching internals**: `_cacheResult()`, `_trackPerformance()`, etc.
|
||||
|
||||
### **⚠️ DEPRECATED** (Should be removed/hidden):
|
||||
- **brain.neural.clusterFast()** → Use `brain.neural.clusters({ algorithm: 'hierarchical' })`
|
||||
- **brain.neural.clusterLarge()** → Use `brain.neural.clusters({ algorithm: 'sample' })`
|
||||
- **brain.neural.similarity()** → Use `brain.neural.similar()` (pick one name)
|
||||
- **brain.visualize()** → Too specialized for main class, use `brain.neural.visualize()`
|
||||
|
|
@ -1,220 +0,0 @@
|
|||
# Brainy Metadata Architecture & Namespacing
|
||||
|
||||
## The Problem 🚨
|
||||
We're mixing internal Brainy fields with user metadata, causing:
|
||||
1. **Namespace collisions** - User's `deleted` field conflicts with our soft-delete
|
||||
2. **API confusion** - Users see internal fields they shouldn't care about
|
||||
3. **Security issues** - Users could manipulate internal fields
|
||||
4. **Augmentation conflicts** - 3rd party augmentations might overwrite our fields
|
||||
|
||||
## Current Internal Fields Being Added
|
||||
|
||||
### Core System Fields
|
||||
```javascript
|
||||
metadata = {
|
||||
// USER DATA
|
||||
name: "Django",
|
||||
type: "framework",
|
||||
|
||||
// OUR INTERNAL FIELDS - COLLISION RISK!
|
||||
deleted: false, // Soft delete status
|
||||
domain: "tech", // Distributed mode domain
|
||||
domainMetadata: {}, // Domain-specific metadata
|
||||
partition: 0, // Partition for sharding
|
||||
createdAt: {...}, // GraphNoun timestamp
|
||||
updatedAt: {...}, // GraphNoun timestamp
|
||||
createdBy: {...}, // Who created this
|
||||
noun: "Concept", // NounType
|
||||
verb: "RELATES_TO", // VerbType (for relationships)
|
||||
isPlaceholder: true, // Write-only mode marker
|
||||
autoCreated: true, // Auto-created noun marker
|
||||
writeOnlyMode: true // High-speed streaming marker
|
||||
}
|
||||
```
|
||||
|
||||
### Augmentation Fields
|
||||
```javascript
|
||||
// Good - neuralImport already uses underscore prefix!
|
||||
metadata._neuralProcessed = true
|
||||
metadata._neuralConfidence = 0.95
|
||||
metadata._detectedEntities = 5
|
||||
metadata._detectedRelationships = 3
|
||||
metadata._neuralInsights = [...]
|
||||
|
||||
// Bad - direct modification
|
||||
metadata.importance = 0.8 // IntelligentVerbScoring
|
||||
```
|
||||
|
||||
## Proposed Solution: Three-Tier Metadata
|
||||
|
||||
### 1. User Metadata (Public)
|
||||
```javascript
|
||||
metadata = {
|
||||
// User's fields - completely untouched
|
||||
name: "Django",
|
||||
type: "framework",
|
||||
deleted: "2024-01-01", // User's own deleted field - no conflict!
|
||||
domain: "web", // User's domain field - no conflict!
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Internal Metadata (Protected)
|
||||
```javascript
|
||||
metadata._brainy = {
|
||||
// Core system fields - O(1) indexed
|
||||
deleted: false, // Our soft delete flag
|
||||
version: 2, // Metadata schema version
|
||||
|
||||
// Distributed mode
|
||||
partition: 0,
|
||||
distributedDomain: "tech",
|
||||
|
||||
// GraphNoun compliance
|
||||
nounType: "Concept",
|
||||
verbType: "RELATES_TO",
|
||||
createdAt: 1704067200000,
|
||||
updatedAt: 1704067200000,
|
||||
createdBy: "user:123",
|
||||
|
||||
// Performance flags
|
||||
indexed: true,
|
||||
searchable: true,
|
||||
placeholder: false,
|
||||
|
||||
// Storage optimization
|
||||
compressed: false,
|
||||
encrypted: false
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Augmentation Metadata (Semi-Protected)
|
||||
```javascript
|
||||
metadata._augmentations = {
|
||||
// Each augmentation gets its own namespace
|
||||
neuralImport: {
|
||||
processed: true,
|
||||
confidence: 0.95,
|
||||
entities: 5,
|
||||
relationships: 3
|
||||
},
|
||||
|
||||
verbScoring: {
|
||||
contextScore: 0.7,
|
||||
importance: 0.8
|
||||
},
|
||||
|
||||
// 3rd party augmentations
|
||||
customAug: {
|
||||
// Their fields isolated here
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
### Phase 1: Core Fields ✅
|
||||
```javascript
|
||||
// Already done with _brainy.deleted
|
||||
const BRAINY_NAMESPACE = '_brainy'
|
||||
const AUGMENTATION_NAMESPACE = '_augmentations'
|
||||
```
|
||||
|
||||
### Phase 2: Migrate All Internal Fields
|
||||
```javascript
|
||||
// Before
|
||||
metadata.domain = "tech"
|
||||
metadata.partition = 0
|
||||
|
||||
// After
|
||||
metadata._brainy.distributedDomain = "tech"
|
||||
metadata._brainy.partition = 0
|
||||
```
|
||||
|
||||
### Phase 3: Augmentation API
|
||||
```javascript
|
||||
class Augmentation {
|
||||
// Read user metadata (read-only)
|
||||
getUserMetadata(metadata) {
|
||||
const { _brainy, _augmentations, ...userMeta } = metadata
|
||||
return userMeta // Clean user data only
|
||||
}
|
||||
|
||||
// Write augmentation data (isolated)
|
||||
setAugmentationData(metadata, augName, data) {
|
||||
if (!metadata._augmentations) metadata._augmentations = {}
|
||||
metadata._augmentations[augName] = data
|
||||
}
|
||||
|
||||
// Read internal fields (for special augmentations only)
|
||||
getInternalField(metadata, field) {
|
||||
return metadata._brainy?.[field]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **No Collisions** - User can have any field names
|
||||
2. **O(1) Performance** - Internal fields still indexed
|
||||
3. **Clean API** - Users only see their data
|
||||
4. **Secure** - Internal fields protected
|
||||
5. **Extensible** - Augmentations isolated
|
||||
6. **Backward Compatible** - Migration path available
|
||||
|
||||
## Query Impact
|
||||
|
||||
### Before (Collision Risk)
|
||||
```javascript
|
||||
where: {
|
||||
deleted: false, // Ambiguous - ours or user's?
|
||||
type: "framework"
|
||||
}
|
||||
```
|
||||
|
||||
### After (Clear Separation)
|
||||
```javascript
|
||||
where: {
|
||||
'_brainy.deleted': false, // Our soft delete
|
||||
type: "framework" // User's field
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Index on `_brainy.deleted`**: O(1) hash lookup ✅
|
||||
- **Index on `_brainy.partition`**: O(1) for sharding ✅
|
||||
- **Nested field access**: Modern DBs handle this efficiently ✅
|
||||
- **Storage overhead**: ~100 bytes per item (acceptable) ✅
|
||||
|
||||
## Migration Path
|
||||
|
||||
1. **New items**: Automatically use namespaced fields
|
||||
2. **Existing items**: Lazy migration on update
|
||||
3. **Queries**: Support both formats temporarily
|
||||
4. **Deprecation**: Remove old format in v3.0
|
||||
|
||||
## Augmentation Guidelines
|
||||
|
||||
### For Core Augmentations
|
||||
- Use `_brainy.*` for system fields
|
||||
- Use `_augmentations.{name}.*` for augmentation data
|
||||
- Never modify user fields directly
|
||||
|
||||
### For 3rd Party Augmentations
|
||||
- Read user metadata via `getUserMetadata()`
|
||||
- Write only to `_augmentations.{yourName}.*`
|
||||
- Request permission for internal field access
|
||||
|
||||
## Critical Fields to Namespace
|
||||
|
||||
| Field | Current Location | New Location | Priority |
|
||||
|-------|-----------------|--------------|----------|
|
||||
| deleted | metadata.deleted | metadata._brainy.deleted | HIGH ✅ |
|
||||
| partition | metadata.partition | metadata._brainy.partition | HIGH |
|
||||
| domain | metadata.domain | metadata._brainy.distributedDomain | HIGH |
|
||||
| createdAt | metadata.createdAt | metadata._brainy.createdAt | MEDIUM |
|
||||
| updatedAt | metadata.updatedAt | metadata._brainy.updatedAt | MEDIUM |
|
||||
| noun | metadata.noun | metadata._brainy.nounType | MEDIUM |
|
||||
| verb | metadata.verb | metadata._brainy.verbType | MEDIUM |
|
||||
| isPlaceholder | metadata.isPlaceholder | metadata._brainy.placeholder | LOW |
|
||||
| autoCreated | metadata.autoCreated | metadata._brainy.autoCreated | LOW |
|
||||
Loading…
Add table
Add a link
Reference in a new issue