feat(8.0): API simplification — remove neural()/Db.search, one storage path key, integration→0
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
This commit is contained in:
parent
0c4a51c24e
commit
606445cd61
74 changed files with 712 additions and 7470 deletions
|
|
@ -429,23 +429,7 @@ fusionResults.forEach(r => {
|
|||
}
|
||||
})
|
||||
|
||||
// 5. NEURAL API: Automatic clustering
|
||||
console.log('\n\n🤖 NEURAL API: Automatic Clustering')
|
||||
const neural = brain.neural()
|
||||
|
||||
const clusters = await neural.clusters({
|
||||
maxClusters: 3,
|
||||
minClusterSize: 1
|
||||
})
|
||||
|
||||
console.log(`Found ${clusters.length} semantic clusters:`)
|
||||
clusters.forEach((cluster, i) => {
|
||||
console.log(`\n Cluster ${i + 1}: ${cluster.label || cluster.id}`)
|
||||
console.log(` Members: ${cluster.members.length}`)
|
||||
console.log(` Centroid topics: ${cluster.metadata?.topics?.join(', ') || 'N/A'}`)
|
||||
})
|
||||
|
||||
// 6. SIMILARITY: Find similar documents
|
||||
// 5. SIMILARITY: Find similar documents
|
||||
console.log('\n\n🔍 SIMILARITY: Find Similar Documents')
|
||||
const similarTo = await brain.similar({
|
||||
to: paper1, // Entity ID of first AI paper
|
||||
|
|
@ -459,19 +443,6 @@ similarTo.forEach(r => {
|
|||
console.log(` [${r.score.toFixed(3)}] ${r.entity.data?.substring(0, 50)}...`)
|
||||
})
|
||||
|
||||
// 7. OUTLIER DETECTION
|
||||
console.log('\n\n🚨 OUTLIER DETECTION')
|
||||
const outliers = await neural.outliers({
|
||||
method: 'statistical',
|
||||
threshold: 2.0 // 2 standard deviations
|
||||
})
|
||||
|
||||
console.log(`Found ${outliers.length} outlier documents:`)
|
||||
outliers.forEach(o => {
|
||||
const entity = await brain.get(o.id)
|
||||
console.log(` [Anomaly score: ${o.score.toFixed(3)}] ${entity?.data?.substring(0, 50)}...`)
|
||||
})
|
||||
|
||||
await brain.close()
|
||||
```
|
||||
|
||||
|
|
@ -869,7 +840,7 @@ console.log('Initializing production storage...\n')
|
|||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: '/var/lib/brainy'
|
||||
path: '/var/lib/brainy'
|
||||
},
|
||||
|
||||
// Performance tuning
|
||||
|
|
@ -963,26 +934,6 @@ console.log(` Failed: ${batchResult.failed.length}`)
|
|||
console.log(` Duration: ${duration}ms`)
|
||||
console.log(` Throughput: ${(batchResult.successful.length / (duration / 1000)).toFixed(0)} entities/sec`)
|
||||
|
||||
// 5. ADVANCED CLUSTERING (Large Dataset)
|
||||
console.log('\n\n🤖 Clustering 1000 entities...\n')
|
||||
|
||||
const neural = brain.neural()
|
||||
|
||||
// Use fast clustering for large datasets
|
||||
const clusters = await neural.clusterFast({
|
||||
maxClusters: 5
|
||||
})
|
||||
|
||||
console.log(`Found ${clusters.length} clusters:`)
|
||||
clusters.forEach((cluster, i) => {
|
||||
console.log(`\n Cluster ${i + 1}: ${cluster.label || cluster.id}`)
|
||||
console.log(` Size: ${cluster.members.length} members`)
|
||||
console.log(` Density: ${(cluster.density || 0).toFixed(3)}`)
|
||||
if (cluster.metadata?.keywords) {
|
||||
console.log(` Keywords: ${cluster.metadata.keywords.slice(0, 5).join(', ')}`)
|
||||
}
|
||||
})
|
||||
|
||||
// 6. PRODUCTION STATISTICS
|
||||
console.log('\n\n📊 Production Statistics:\n')
|
||||
|
||||
|
|
@ -1040,7 +991,7 @@ console.log('✅ Brain closed cleanly')
|
|||
|
||||
console.log('\n\n🎓 Production Deployment Complete!')
|
||||
console.log('\n📚 Key Production Learnings:')
|
||||
console.log(' 1. Use filesystem storage and snapshot rootDirectory off-site from your scheduler')
|
||||
console.log(' 1. Use filesystem storage and snapshot the data directory off-site from your scheduler')
|
||||
console.log(' 2. Batch operations = 100x faster than individual ops')
|
||||
console.log(' 3. Metadata query optimization for complex filters')
|
||||
console.log(' 4. Monitor query performance with explain: true')
|
||||
|
|
@ -1067,7 +1018,7 @@ console.log(' 7. Stream large imports with progress callbacks')
|
|||
*/15 * * * * gsutil rsync -r /var/lib/brainy gs://my-bucket/brainy-backup
|
||||
```
|
||||
|
||||
Brainy itself never reaches out to an object store. Snapshot `rootDirectory` from your scheduler.
|
||||
Brainy itself never reaches out to an object store. Snapshot `path` from your scheduler.
|
||||
|
||||
#### 3. **Performance Optimization**
|
||||
|
||||
|
|
@ -1141,7 +1092,7 @@ const brain = new Brainy({ verbose: true })
|
|||
|
||||
#### Before Deployment
|
||||
|
||||
- [ ] Provision a writable `rootDirectory` on the host
|
||||
- [ ] Provision a writable `path` on the host
|
||||
- [ ] Set up an off-site snapshot job (`rclone` / `aws s3 sync` / `gsutil rsync`)
|
||||
- [ ] Configure caching
|
||||
- [ ] Test batch operations
|
||||
|
|
|
|||
|
|
@ -305,7 +305,7 @@ const results = await Promise.all(searchPromises)
|
|||
- ✅ **No Network Calls**: Everything runs locally, including embeddings
|
||||
- ✅ **Thread-Safe**: Immutable data structures where possible
|
||||
- ✅ **Memory Bounded**: Configurable cache sizes and automatic cleanup
|
||||
- ✅ **Single-Node by Design**: One process owns one `rootDirectory`; scale out at the service layer
|
||||
- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
|
||||
- ✅ **Zero Stubs**: Every line of code is production-ready
|
||||
|
||||
## Lazy Loading Performance
|
||||
|
|
@ -439,7 +439,7 @@ For >10M entities, run multiple Brainy processes behind your own routing layer
|
|||
└──────────┴──────────┴──────────────────┘
|
||||
```
|
||||
|
||||
For off-site replication, snapshot `rootDirectory` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`).
|
||||
For off-site replication, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`).
|
||||
|
||||
### Performance at Scale
|
||||
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ const brain = new Brainy({ storage: { type: 'memory' } })
|
|||
### On-Disk (Default for Node)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: './brainy-data' }
|
||||
storage: { type: 'filesystem', path: './brainy-data' }
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -88,11 +88,11 @@ match-set size and is the path to move onto the native provider first._
|
|||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: '/var/lib/brainy'
|
||||
path: '/var/lib/brainy'
|
||||
}
|
||||
})
|
||||
```
|
||||
- Stores everything in a sharded JSON tree under `rootDirectory`
|
||||
- Stores everything in a sharded JSON tree under `path`
|
||||
- Atomic writes via rename
|
||||
- Survives process restarts
|
||||
- Snapshot it off-site with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` from your scheduler
|
||||
|
|
@ -108,10 +108,10 @@ const brain = new Brainy({ storage: { type: 'memory' } })
|
|||
### Auto
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'auto', rootDirectory: './data' }
|
||||
storage: { type: 'auto', path: './data' }
|
||||
})
|
||||
```
|
||||
- Picks `filesystem` when running on Node with a writable `rootDirectory`
|
||||
- Picks `filesystem` when running on Node with a writable `path`
|
||||
- Falls back to `memory` otherwise
|
||||
|
||||
## Scaling Patterns
|
||||
|
|
@ -125,7 +125,7 @@ const brain = new Brainy({ storage: { type: 'memory' } })
|
|||
### Stage 2: Production (Filesystem)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' }
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' }
|
||||
})
|
||||
// Most production workloads up to ~10M entities on a single host
|
||||
```
|
||||
|
|
@ -133,7 +133,7 @@ const brain = new Brainy({
|
|||
### Stage 3: Higher Throughput (Tune the Vector Index)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'fast', // Trade recall for latency
|
||||
persistMode: 'deferred' // Batch persistence
|
||||
|
|
@ -142,14 +142,14 @@ const brain = new Brainy({
|
|||
```
|
||||
|
||||
### Stage 4: Multi-Instance (Operator-Layer)
|
||||
Run multiple Brainy processes behind your own routing/service layer. Each process owns its own `rootDirectory`. Sync each artifact off-site independently. Brainy itself does not coordinate between processes.
|
||||
Run multiple Brainy processes behind your own routing/service layer. Each process owns its own `path`. Sync each artifact off-site independently. Brainy itself does not coordinate between processes.
|
||||
|
||||
## Real World Examples
|
||||
|
||||
### Example 1: Single-Node App With Backup
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' }
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' }
|
||||
})
|
||||
```
|
||||
Schedule (cron / systemd timer):
|
||||
|
|
@ -170,7 +170,7 @@ function brainForTenant(tenantId: string) {
|
|||
return new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: `/var/lib/brainy/${tenantId}`
|
||||
path: `/var/lib/brainy/${tenantId}`
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
@ -180,7 +180,7 @@ Your service layer handles routing and isolation; Brainy stays simple.
|
|||
### Example 4: Higher Recall at Scale
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'accurate'
|
||||
}
|
||||
|
|
@ -226,7 +226,7 @@ const stats = await brain.stats()
|
|||
|
||||
## Best Practices
|
||||
|
||||
1. **One process = one `rootDirectory`** — never share a directory between processes
|
||||
1. **One process = one `path`** — never share a directory between processes
|
||||
2. **Snapshot from your scheduler** — Brainy doesn't ship cloud SDKs; use `rclone` / `aws s3 sync` / `gsutil`
|
||||
3. **Profile before tuning** — `recall: 'balanced'` is right for most workloads
|
||||
4. **Install the native vector provider only when measured profiling shows it pays off**
|
||||
|
|
@ -236,4 +236,4 @@ const stats = await brain.stats()
|
|||
- Brainy 8.0 is a **library**, not a cluster
|
||||
- Storage adapters: `filesystem`, `memory`, `auto`
|
||||
- Vector tuning: `recall`, `persistMode`
|
||||
- Backup is an operator-layer concern — snapshot `rootDirectory`
|
||||
- Backup is an operator-layer concern — snapshot `path`
|
||||
|
|
|
|||
|
|
@ -956,7 +956,7 @@ closes the underlying instance.
|
|||
|
||||
```typescript
|
||||
const db = await Brainy.load('/backups/2026-06-01')
|
||||
const hits = await db.search('quarterly invoices')
|
||||
const hits = await db.find({ query: 'quarterly invoices' })
|
||||
await db.release()
|
||||
```
|
||||
|
||||
|
|
@ -982,7 +982,7 @@ exactly that state.
|
|||
```typescript
|
||||
await db.get(id) // entity as of this generation
|
||||
await db.find({ where: { status: 'open' } }) // full find() surface
|
||||
await db.search('unpaid invoices') // semantic search as of this generation
|
||||
await db.find({ query: 'unpaid invoices' }) // semantic search as of this generation
|
||||
await db.related(entityId) // relationships as of this generation
|
||||
await db.since(olderDb) // ids changed between two views
|
||||
const whatIf = await db.with(ops) // speculative in-memory overlay
|
||||
|
|
@ -1247,124 +1247,6 @@ const people = await brain.vfs.searchEntities({
|
|||
|
||||
---
|
||||
|
||||
## Neural API
|
||||
|
||||
Access advanced AI features via `brain.neural()` (method that returns NeuralAPI instance).
|
||||
|
||||
### `neural().similar(a, b, options?)` → `Promise<number | SimilarityResult>`
|
||||
|
||||
Calculate semantic similarity.
|
||||
|
||||
```typescript
|
||||
// Simple similarity score
|
||||
const score = await brain.neural().similar(
|
||||
'renewable energy',
|
||||
'sustainable power'
|
||||
) // 0.87
|
||||
|
||||
// Detailed result
|
||||
const result = await brain.neural().similar('text1', 'text2', {
|
||||
detailed: true
|
||||
})
|
||||
console.log(result.score)
|
||||
console.log(result.explanation)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `neural().clusters(input?, options?)` → `Promise<SemanticCluster[]>`
|
||||
|
||||
Automatic clustering. Accepts a text query, an array of texts, or a `ClusteringOptions` object.
|
||||
|
||||
```typescript
|
||||
const clusters = await brain.neural().clusters({
|
||||
algorithm: 'kmeans', // 'auto' | 'hierarchical' | 'kmeans' | 'dbscan' | ...
|
||||
maxClusters: 5,
|
||||
minClusterSize: 3
|
||||
})
|
||||
|
||||
clusters.forEach(cluster => {
|
||||
console.log(cluster.label) // auto-generated label (when available)
|
||||
console.log(cluster.members) // entity ids in the cluster
|
||||
console.log(cluster.centroid) // centroid vector
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `neural().neighbors(id, options?)` → `Promise<NeighborsResult>`
|
||||
|
||||
Find nearest neighbors.
|
||||
|
||||
```typescript
|
||||
const result = await brain.neural().neighbors(entityId, {
|
||||
limit: 10,
|
||||
minSimilarity: 0.7
|
||||
})
|
||||
```
|
||||
|
||||
**Options:** `limit?`, `radius?`, `minSimilarity?`, `includeMetadata?`, `sortBy?: 'similarity' | 'importance' | 'recency'`
|
||||
|
||||
---
|
||||
|
||||
### `neural().outliers(options?)` → `Promise<Outlier[]>`
|
||||
|
||||
Detect outlier entities.
|
||||
|
||||
```typescript
|
||||
const outliers = await brain.neural().outliers({ threshold: 0.3 })
|
||||
// Each outlier carries the entity id plus outlier scoring details
|
||||
```
|
||||
|
||||
**Options:** `threshold?`, `method?: 'isolation' | 'statistical' | 'cluster-based'`, `minNeighbors?`, `includeReasons?`
|
||||
|
||||
---
|
||||
|
||||
### `neural().visualize(options?)` → `Promise<VizData>`
|
||||
|
||||
Generate visualization data.
|
||||
|
||||
```typescript
|
||||
const vizData = await brain.neural().visualize({
|
||||
maxNodes: 100,
|
||||
dimensions: 3,
|
||||
algorithm: 'force',
|
||||
includeEdges: true
|
||||
})
|
||||
// Use with D3.js, Cytoscape, GraphML tools
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Performance Methods
|
||||
|
||||
#### `neural().clusterFast(options?)` → `Promise<SemanticCluster[]>`
|
||||
|
||||
Fast clustering for large datasets.
|
||||
|
||||
```typescript
|
||||
const clusters = await brain.neural().clusterFast({
|
||||
maxClusters: 10
|
||||
})
|
||||
```
|
||||
|
||||
**Options:** `level?` (hierarchy level), `maxClusters?`
|
||||
|
||||
---
|
||||
|
||||
#### `neural().clusterLarge(options?)` → `Promise<SemanticCluster[]>`
|
||||
|
||||
Sampling-based clustering for very large datasets.
|
||||
|
||||
```typescript
|
||||
const clusters = await brain.neural().clusterLarge({
|
||||
sampleSize: 1000,
|
||||
strategy: 'diverse' // 'random' | 'diverse' | 'recent'
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Import & Export
|
||||
|
||||
### `import(source, options?)` → `Promise<ImportResult>`
|
||||
|
|
@ -1463,7 +1345,7 @@ const brain = new Brainy({
|
|||
// Storage configuration
|
||||
storage: {
|
||||
type: 'filesystem', // 'memory' | 'filesystem' | 'auto'
|
||||
rootDirectory: './brainy-data'
|
||||
path: './brainy-data'
|
||||
},
|
||||
|
||||
// Vector index configuration (2 knobs)
|
||||
|
|
@ -1510,14 +1392,14 @@ const brain = new Brainy({
|
|||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: './brainy-data'
|
||||
path: './brainy-data'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Use case:** Node.js applications, single-node production deployments
|
||||
|
||||
For off-site backup, snapshot `rootDirectory` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`) — Brainy itself doesn't reach out to cloud object stores.
|
||||
For off-site backup, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`) — Brainy itself doesn't reach out to cloud object stores.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -1525,11 +1407,11 @@ For off-site backup, snapshot `rootDirectory` from your scheduler (`gsutil rsync
|
|||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'auto', rootDirectory: './brainy-data' }
|
||||
storage: { type: 'auto', path: './brainy-data' }
|
||||
})
|
||||
```
|
||||
|
||||
Picks `'filesystem'` on Node with a writable `rootDirectory`, falls back to `'memory'` otherwise.
|
||||
Picks `'filesystem'` on Node with a writable `path`, falls back to `'memory'` otherwise.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,306 +0,0 @@
|
|||
# 🧠 **Brainy Clustering Algorithms - Complete Analysis**
|
||||
|
||||
## 🎯 **Current State & Capabilities**
|
||||
|
||||
### **✅ Existing Infrastructure (Excellent Foundation)**
|
||||
|
||||
#### **1. HNSW Hierarchical Clustering**
|
||||
```typescript
|
||||
// ALREADY IMPLEMENTED & OPTIMIZED
|
||||
brain.neural.clusters({ algorithm: 'hierarchical', level: 2 })
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
- **Leverages HNSW natural hierarchy**: Uses existing index levels as natural cluster boundaries
|
||||
- **O(n) performance**: Much faster than O(n²) traditional clustering
|
||||
- **Multi-level granularity**: Higher levels = fewer, broader clusters; Lower levels = more, specific clusters
|
||||
- **Representative sampling**: Uses HNSW level nodes as natural cluster centers
|
||||
|
||||
**Performance characteristics:**
|
||||
- **Excellent for large datasets** (millions of items)
|
||||
- **Preserves semantic relationships** from vector space
|
||||
- **Automatic granularity control** via level parameter
|
||||
|
||||
#### **2. Distance-Based Algorithms**
|
||||
```typescript
|
||||
// COMPREHENSIVE DISTANCE FUNCTIONS AVAILABLE
|
||||
euclideanDistance, cosineDistance, manhattanDistance, dotProductDistance
|
||||
```
|
||||
|
||||
**Optimized implementations:**
|
||||
- **Batch processing**: `calculateDistancesBatch()` with parallelization
|
||||
- **Multiple metrics**: Choose optimal distance function per use case
|
||||
- **Performance optimized**: Faster than GPU for small vectors due to no transfer overhead
|
||||
|
||||
#### **3. Rich Semantic Taxonomy**
|
||||
```typescript
|
||||
// 25+ NOUN TYPES & 35+ VERB TYPES
|
||||
NounType: Person, Organization, Document, Concept, Event, Media, etc.
|
||||
VerbType: RelatedTo, Contains, PartOf, Causes, CreatedBy, etc.
|
||||
```
|
||||
|
||||
**Semantic clustering capabilities:**
|
||||
- **Type-based clustering**: Group by semantic categories
|
||||
- **Cross-type relationships**: Use verb types to find semantic bridges
|
||||
- **Hierarchical taxonomies**: Natural clustering within and across types
|
||||
|
||||
#### **4. Graph Structure**
|
||||
```typescript
|
||||
// VERB RELATIONSHIPS CREATE RICH GRAPH
|
||||
await brain.relate(sourceId, targetId, VerbType.Causes, { strength: 0.8 })
|
||||
```
|
||||
|
||||
**Graph-based clustering potential:**
|
||||
- **Connected components**: Find strongly connected groups
|
||||
- **Community detection**: Use relationship strength for clustering
|
||||
- **Multi-modal clustering**: Combine graph + vector + taxonomy
|
||||
|
||||
## 🚀 **Advanced Clustering Algorithms We Can Implement**
|
||||
|
||||
### **1. ✅ Already Implemented: HNSW Hierarchical**
|
||||
|
||||
```typescript
|
||||
// PRODUCTION READY - Uses existing HNSW levels
|
||||
const clusters = await brain.neural.clusters({
|
||||
algorithm: 'hierarchical',
|
||||
level: 2, // Control granularity
|
||||
maxClusters: 15
|
||||
})
|
||||
```
|
||||
|
||||
**Performance:** **A+** - O(n) leveraging existing index structure
|
||||
|
||||
### **2. 🔥 Semantic Taxonomy Clustering**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Fast type-based clustering with cross-type bridges
|
||||
const clusters = await brain.neural.clusterByDomain('nounType', {
|
||||
preserveTypeBoundaries: false, // Allow cross-type clusters
|
||||
bridgeStrength: 0.8, // Minimum relationship strength for bridges
|
||||
hybridWeighting: {
|
||||
taxonomy: 0.4, // 40% weight to type similarity
|
||||
vector: 0.4, // 40% weight to vector similarity
|
||||
graph: 0.2 // 20% weight to relationship strength
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Algorithm approach:**
|
||||
1. **Primary clustering by taxonomy**: Group by NounType/VerbType first
|
||||
2. **Vector refinement**: Sub-cluster within types using vector similarity
|
||||
3. **Cross-type bridging**: Find relationships that bridge type boundaries
|
||||
4. **Weighted fusion**: Combine taxonomy + vector + graph signals
|
||||
|
||||
**Performance:** **A+** - O(n log n) - taxonomy grouping is O(n), refinement is HNSW-accelerated
|
||||
|
||||
### **3. 🔥 Graph Community Detection**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Relationship-based clustering
|
||||
const clusters = await brain.neural.clusterByConnections({
|
||||
algorithm: 'modularity', // or 'louvain', 'leiden'
|
||||
minCommunitySize: 3,
|
||||
relationshipWeights: {
|
||||
[VerbType.Creates]: 1.0,
|
||||
[VerbType.PartOf]: 0.8,
|
||||
[VerbType.RelatedTo]: 0.5
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Algorithm approach:**
|
||||
1. **Build weighted graph**: Use verbs as edges, weights from relationship types + metadata
|
||||
2. **Community detection**: Apply Louvain or Leiden algorithm for modularity optimization
|
||||
3. **Semantic enhancement**: Use vector similarity to refine community boundaries
|
||||
|
||||
**Performance:** **A** - O(n log n) for sparse graphs, handles millions of relationships efficiently
|
||||
|
||||
### **4. 🔥 Multi-Modal Fusion Clustering**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Best of all worlds
|
||||
const clusters = await brain.neural.clusters({
|
||||
algorithm: 'multimodal',
|
||||
signals: {
|
||||
vector: { weight: 0.5, metric: 'cosine' },
|
||||
graph: { weight: 0.3, algorithm: 'modularity' },
|
||||
taxonomy: { weight: 0.2, crossTypeThreshold: 0.8 }
|
||||
},
|
||||
fusion: 'weighted_ensemble' // or 'consensus', 'hierarchical'
|
||||
})
|
||||
```
|
||||
|
||||
**Algorithm approach:**
|
||||
1. **Independent clustering**: Run HNSW, graph, and taxonomy clustering separately
|
||||
2. **Consensus building**: Find agreement between different clustering results
|
||||
3. **Conflict resolution**: Use weighted voting or hierarchical merging for disagreements
|
||||
4. **Quality optimization**: Iteratively refine based on silhouette scores
|
||||
|
||||
**Performance:** **A** - O(n log n) - parallel execution of component algorithms
|
||||
|
||||
### **5. 💎 Temporal Pattern Clustering**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Time-aware clustering using existing infrastructure
|
||||
const clusters = await brain.neural.clusterByTime('createdAt', [
|
||||
{ start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1 2024' },
|
||||
{ start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2 2024' }
|
||||
], {
|
||||
evolution: 'track', // Track how clusters evolve over time
|
||||
stability: 0.7, // Minimum stability threshold
|
||||
trendAnalysis: true // Include trend detection
|
||||
})
|
||||
```
|
||||
|
||||
**Algorithm approach:**
|
||||
1. **Time window clustering**: Apply HNSW clustering within each time window
|
||||
2. **Cluster evolution tracking**: Match clusters across time windows using vector similarity
|
||||
3. **Trend analysis**: Detect growing, shrinking, merging, splitting patterns
|
||||
4. **Stability scoring**: Measure cluster consistency over time
|
||||
|
||||
**Performance:** **A+** - O(k*n log n) where k = number of time windows
|
||||
|
||||
### **6. 💎 DBSCAN with Adaptive Parameters**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Density-based clustering with smart parameter selection
|
||||
const clusters = await brain.neural.clusters({
|
||||
algorithm: 'dbscan',
|
||||
autoParams: true, // Automatically select eps and minPts
|
||||
distanceMetric: 'cosine',
|
||||
outlierHandling: 'soft' // Soft assignment instead of hard outliers
|
||||
})
|
||||
```
|
||||
|
||||
**Algorithm approach:**
|
||||
1. **Adaptive parameter selection**: Use HNSW k-NN distances to estimate optimal eps
|
||||
2. **Multi-scale analysis**: Run DBSCAN at multiple scales and merge results
|
||||
3. **Soft outlier assignment**: Assign outliers to nearest clusters with confidence scores
|
||||
|
||||
**Performance:** **A** - O(n log n) using HNSW for neighbor queries
|
||||
|
||||
## 📊 **Performance Comparison Matrix**
|
||||
|
||||
| Algorithm | Time Complexity | Space | Large Scale | Semantic Quality | Graph Aware |
|
||||
|-----------|----------------|-------|-------------|------------------|-------------|
|
||||
| **HNSW Hierarchical** | O(n) | O(n) | ✅ Excellent | ✅ Very Good | ❌ No |
|
||||
| **Taxonomy Fusion** | O(n log n) | O(n) | ✅ Excellent | 🔥 Exceptional | ⚡ Partial |
|
||||
| **Graph Communities** | O(n log n) | O(e) | ✅ Very Good | ✅ Very Good | 🔥 Exceptional |
|
||||
| **Multi-Modal** | O(n log n) | O(n) | ✅ Very Good | 🔥 Exceptional | 🔥 Exceptional |
|
||||
| **Temporal Patterns** | O(k*n log n) | O(n) | ⚡ Good | ✅ Very Good | ⚡ Partial |
|
||||
| **Adaptive DBSCAN** | O(n log n) | O(n) | ✅ Very Good | ✅ Very Good | ❌ No |
|
||||
|
||||
## 🎯 **Specific Improvements Using Existing Capabilities**
|
||||
|
||||
### **1. Enhanced HNSW Clustering (Easy Win)**
|
||||
|
||||
```typescript
|
||||
// IMPROVE EXISTING: Add semantic post-processing
|
||||
private async enhanceHNSWClusters(clusters: SemanticCluster[]): Promise<SemanticCluster[]> {
|
||||
return Promise.all(clusters.map(async cluster => {
|
||||
// Get actual metadata for cluster members
|
||||
const members = await this.brain.getNouns(cluster.members.map(id => ({ id })))
|
||||
|
||||
// Analyze semantic characteristics
|
||||
const semanticProfile = this.analyzeSemanticProfile(members)
|
||||
|
||||
// Generate meaningful cluster labels
|
||||
const label = await this.generateClusterLabel(members, semanticProfile)
|
||||
|
||||
// Calculate cluster coherence using multiple signals
|
||||
const coherence = this.calculateMultiModalCoherence(members)
|
||||
|
||||
return {
|
||||
...cluster,
|
||||
label,
|
||||
semanticProfile,
|
||||
coherence,
|
||||
quality: coherence.overall
|
||||
}
|
||||
}))
|
||||
}
|
||||
```
|
||||
|
||||
### **2. Intelligent Algorithm Selection**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Smart routing based on data characteristics
|
||||
private selectOptimalAlgorithm(dataCharacteristics: {
|
||||
size: number,
|
||||
dimensionality: number,
|
||||
graphDensity: number,
|
||||
typeDistribution: Record<string, number>
|
||||
}): string {
|
||||
if (dataCharacteristics.size > 100000) {
|
||||
return 'hierarchical' // HNSW scales best
|
||||
}
|
||||
|
||||
if (dataCharacteristics.graphDensity > 0.1) {
|
||||
return 'multimodal' // Rich graph structure
|
||||
}
|
||||
|
||||
if (Object.keys(dataCharacteristics.typeDistribution).length > 10) {
|
||||
return 'taxonomy' // Diverse semantic types
|
||||
}
|
||||
|
||||
return 'hierarchical' // Safe default
|
||||
}
|
||||
```
|
||||
|
||||
### **3. Streaming Cluster Updates**
|
||||
|
||||
```typescript
|
||||
// IMPLEMENT: Incremental clustering using existing infrastructure
|
||||
public async updateClusters(newItems: string[]): Promise<SemanticCluster[]> {
|
||||
// Use HNSW nearest neighbor for fast cluster assignment
|
||||
const assignments = await Promise.all(
|
||||
newItems.map(async itemId => {
|
||||
const neighbors = await this.brain.neural.neighbors(itemId, { limit: 5 })
|
||||
return this.assignToNearestCluster(itemId, neighbors, this.existingClusters)
|
||||
})
|
||||
)
|
||||
|
||||
// Incrementally update cluster centroids and boundaries
|
||||
return this.updateClusterBoundaries(assignments)
|
||||
}
|
||||
```
|
||||
|
||||
## 🏆 **Recommended Implementation Priority**
|
||||
|
||||
### **🔥 Phase 1: High Impact, Easy Implementation**
|
||||
1. **Enhanced HNSW Clustering**: Add semantic post-processing to existing algorithm
|
||||
2. **Taxonomy-Aware Clustering**: Leverage existing NounType/VerbType enums
|
||||
3. **Intelligent Algorithm Selection**: Route based on data characteristics
|
||||
|
||||
### **⚡ Phase 2: Advanced Features**
|
||||
4. **Graph Community Detection**: Use existing verb relationships
|
||||
5. **Multi-Modal Fusion**: Combine all signals intelligently
|
||||
6. **Streaming Updates**: Incremental cluster maintenance
|
||||
|
||||
### **💎 Phase 3: Cutting Edge**
|
||||
7. **Temporal Pattern Analysis**: Track cluster evolution over time
|
||||
8. **Adaptive DBSCAN**: Dynamic parameter selection
|
||||
9. **Explainable Clustering**: Generate cluster explanations and reasoning
|
||||
|
||||
## 🎯 **Key Advantages of Our Approach**
|
||||
|
||||
### **✅ Leverages Existing Infrastructure**
|
||||
- **HNSW index**: Already optimized for large-scale vector operations
|
||||
- **Distance functions**: Battle-tested and performance-optimized
|
||||
- **Semantic taxonomy**: Rich type system with 60+ semantic categories
|
||||
- **Graph structure**: Relationship network from verb connections
|
||||
|
||||
### **✅ Multiple Clustering Paradigms**
|
||||
- **Vector similarity**: Traditional embedding-based clustering
|
||||
- **Graph structure**: Relationship-based community detection
|
||||
- **Semantic taxonomy**: Type-aware intelligent grouping
|
||||
- **Temporal patterns**: Time-aware cluster evolution
|
||||
- **Multi-modal fusion**: Best of all worlds
|
||||
|
||||
### **✅ Scalability & Performance**
|
||||
- **O(n) hierarchical clustering**: Leveraging HNSW levels
|
||||
- **Parallel processing**: Batch distance calculations optimized
|
||||
- **Streaming support**: Real-time cluster updates
|
||||
- **Memory efficient**: Existing index structures reused
|
||||
|
||||
**Our clustering algorithms are not just competitive - they're architecturally superior by leveraging Brainy's unique multi-modal semantic infrastructure.**
|
||||
|
|
@ -19,7 +19,7 @@ storage keys its internal map by the identical path strings).
|
|||
|
||||
## 1. Directory Tree
|
||||
|
||||
A real 8.0 filesystem store (`rootDirectory`, default `./brainy-data`):
|
||||
A real 8.0 filesystem store (`path`, default `./brainy-data`):
|
||||
|
||||
```
|
||||
brainy-data/
|
||||
|
|
|
|||
|
|
@ -71,7 +71,7 @@ Brainy 8.0 ships two adapters, both implementing the same `StorageAdapter` inter
|
|||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: './data'
|
||||
path: './data'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
|
@ -99,15 +99,15 @@ const brain = new Brainy({
|
|||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'auto',
|
||||
rootDirectory: './data'
|
||||
path: './data'
|
||||
}
|
||||
})
|
||||
```
|
||||
`'auto'` picks `'filesystem'` when running on Node.js with a writable `rootDirectory`, and falls back to `'memory'` otherwise.
|
||||
`'auto'` picks `'filesystem'` when running on Node.js with a writable `path`, and falls back to `'memory'` otherwise.
|
||||
|
||||
## Backup and Off-Site Replication
|
||||
|
||||
Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `rootDirectory` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:
|
||||
Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `path` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:
|
||||
|
||||
- `gsutil rsync -r ./data gs://my-bucket/brainy-data`
|
||||
- `aws s3 sync ./data s3://my-bucket/brainy-data`
|
||||
|
|
@ -180,7 +180,7 @@ High-performance deduplication system for streaming data:
|
|||
|
||||
## Durability
|
||||
|
||||
Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `rootDirectory` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)).
|
||||
Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `path` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)).
|
||||
|
||||
## Storage Optimization
|
||||
|
||||
|
|
@ -210,7 +210,7 @@ await brain.addBatch([
|
|||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: './data',
|
||||
path: './data',
|
||||
cache: {
|
||||
enabled: true,
|
||||
maxSize: 1000, // Maximum cached items
|
||||
|
|
@ -260,7 +260,7 @@ await brain.restore('/backups/2026-06-11', { confirm: true })
|
|||
### Move to a new directory
|
||||
```typescript
|
||||
// A snapshot directory is a complete store: restore it into a fresh brain
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './new' } })
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', path: './new' } })
|
||||
await brain.init()
|
||||
await brain.restore('/backups/2026-06-11', { confirm: true })
|
||||
```
|
||||
|
|
@ -298,7 +298,7 @@ console.log(stats)
|
|||
1. **Read-heavy**: Enable caching and let the OS page cache do its job
|
||||
2. **Write-heavy**: Batch operations and tune the cache `maxSize`
|
||||
3. **Real-time**: FileSystem with periodic snapshots
|
||||
4. **Archival**: Snapshot `rootDirectory` to cold object storage on a schedule
|
||||
4. **Archival**: Snapshot `path` to cold object storage on a schedule
|
||||
5. **Large-scale**: Rely on metadata/vector separation + UUID sharding
|
||||
|
||||
### Monitor and Maintain
|
||||
|
|
|
|||
|
|
@ -68,7 +68,7 @@ const articles = await brain.find("verified articles by John Smith about machine
|
|||
|
||||
#### Simple Vector Search
|
||||
```typescript
|
||||
const results = await brain.search("machine learning concepts")
|
||||
const results = await brain.find("machine learning concepts")
|
||||
```
|
||||
|
||||
#### Combined Intelligence Query
|
||||
|
|
@ -161,7 +161,7 @@ Brainy includes 220+ embedded patterns for natural language understanding:
|
|||
|
||||
```typescript
|
||||
// Natural language automatically parsed
|
||||
const results = await brain.search(
|
||||
const results = await brain.find(
|
||||
"show me recent AI papers from Stanford published this year"
|
||||
)
|
||||
// Automatically converts to:
|
||||
|
|
@ -189,10 +189,10 @@ The NLP processor identifies query intent:
|
|||
Successful execution plans are cached:
|
||||
```typescript
|
||||
// First query: 50ms (plan generation + execution)
|
||||
await brain.search("machine learning papers")
|
||||
await brain.find("machine learning papers")
|
||||
|
||||
// Subsequent similar queries: 10ms (cached plan)
|
||||
await brain.search("deep learning papers")
|
||||
await brain.find("deep learning papers")
|
||||
```
|
||||
|
||||
### Self-Optimization
|
||||
|
|
|
|||
|
|
@ -65,7 +65,7 @@ With no `storage` option, Brainy uses `type: 'auto'`:
|
|||
```typescript
|
||||
// Explicit override when you want a specific root
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: './brainy-data' }
|
||||
storage: { type: 'filesystem', path: './brainy-data' }
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -135,7 +135,7 @@ explicit constructor option:
|
|||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
||||
storage: { type: 'filesystem', path: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'accurate',
|
||||
persistMode: 'immediate'
|
||||
|
|
|
|||
|
|
@ -105,7 +105,7 @@ coexists with whatever the writer is doing:
|
|||
|
||||
```typescript
|
||||
const reader = await Brainy.openReadOnly({
|
||||
storage: { type: 'filesystem', rootDirectory: '/data/brain' }
|
||||
storage: { type: 'filesystem', path: '/data/brain' }
|
||||
})
|
||||
|
||||
const stats = await reader.stats()
|
||||
|
|
@ -119,7 +119,7 @@ you need fresher state, ask the writer to flush before opening:
|
|||
|
||||
```typescript
|
||||
const reader = await Brainy.openReadOnly({
|
||||
storage: { type: 'filesystem', rootDirectory: '/data/brain' }
|
||||
storage: { type: 'filesystem', path: '/data/brain' }
|
||||
})
|
||||
|
||||
const acked = await reader.requestFlush({ timeoutMs: 5000 })
|
||||
|
|
|
|||
|
|
@ -74,7 +74,7 @@ When the auto-config is wrong for your workload (e.g. you know your entities are
|
|||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', options: { path: './data' } },
|
||||
storage: { type: 'filesystem', path: './data' },
|
||||
maxQueryLimit: 50_000 // raises the cap; still hard-clamped at 100 000
|
||||
})
|
||||
```
|
||||
|
|
|
|||
|
|
@ -274,7 +274,7 @@ export function getBrain() {
|
|||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: './data' }
|
||||
storage: { type: 'filesystem', path: './data' }
|
||||
})
|
||||
await brain.init()
|
||||
return brain
|
||||
|
|
@ -470,7 +470,7 @@ import { Brainy } from '@soulcraft/brainy'
|
|||
|
||||
export async function generateStaticProps() {
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: './content' }
|
||||
storage: { type: 'filesystem', path: './content' }
|
||||
})
|
||||
await brain.init()
|
||||
|
||||
|
|
|
|||
|
|
@ -260,7 +260,7 @@ Once imported, use Triple Intelligence to query:
|
|||
|
||||
```javascript
|
||||
// Vector search
|
||||
const similar = await brain.search('engineers')
|
||||
const similar = await brain.find('engineers')
|
||||
|
||||
// Natural language
|
||||
const results = await brain.find('people in engineering who joined this year')
|
||||
|
|
|
|||
|
|
@ -1707,12 +1707,12 @@ Brainy 8.0 ships two adapters: filesystem and memory.
|
|||
const brain = await Brainy.create({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: './.brainy'
|
||||
path: './.brainy'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
For off-site backup, snapshot `rootDirectory` from your scheduler with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`. Brainy itself doesn't reach out to object storage.
|
||||
For off-site backup, snapshot `path` from your scheduler with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`. Brainy itself doesn't reach out to object storage.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -1743,7 +1743,7 @@ For off-site backup, snapshot `rootDirectory` from your scheduler with `gsutil r
|
|||
- HNSW Index: O(log n) search (1B entities = ~30 hops)
|
||||
- Metadata Index: O(1) filtering
|
||||
- Graph Adjacency: O(1) relationship lookups
|
||||
- Storage: Bounded by the filesystem volume backing `rootDirectory`
|
||||
- Storage: Bounded by the filesystem volume backing `path`
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
|
|
|
|||
|
|
@ -111,7 +111,7 @@ check fails — useful for piping into monitoring or CI.
|
|||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const reader = await Brainy.openReadOnly({
|
||||
storage: { type: 'filesystem', rootDirectory: '/data/brain' }
|
||||
storage: { type: 'filesystem', path: '/data/brain' }
|
||||
})
|
||||
|
||||
// Force the writer to flush before reading
|
||||
|
|
|
|||
|
|
@ -1,371 +0,0 @@
|
|||
# Neural API Guide
|
||||
|
||||
> Semantic intelligence features for clustering, similarity, and analysis
|
||||
|
||||
## Overview
|
||||
|
||||
The Neural API provides advanced AI-powered features for understanding relationships and patterns in your data. Access it through `brain.neural()` (method call) after initializing Brainy.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```javascript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Access Neural API (note: neural() is a method call)
|
||||
const neural = brain.neural()
|
||||
|
||||
// Find similar items
|
||||
const similarity = await neural.similar('text1', 'text2')
|
||||
|
||||
// Auto-cluster your data
|
||||
const clusters = await neural.clusters()
|
||||
```
|
||||
|
||||
## Core Features
|
||||
|
||||
### 1. Semantic Clustering
|
||||
|
||||
Automatically group related items based on their meaning:
|
||||
|
||||
```javascript
|
||||
// Simple clustering - let Brainy decide
|
||||
const clusters = await neural.clusters()
|
||||
|
||||
// Each cluster contains:
|
||||
// - id: Unique identifier
|
||||
// - members: Array of item IDs in this cluster
|
||||
// - centroid: The "center" of the cluster
|
||||
// - label: Optional descriptive label
|
||||
// - confidence: How confident the clustering is
|
||||
|
||||
// Example: Organize customer feedback
|
||||
const feedback = [
|
||||
await brain.add("The app crashes when I upload photos"),
|
||||
await brain.add("Photo upload feature is broken"),
|
||||
await brain.add("Great customer service!"),
|
||||
await brain.add("Support team was very helpful"),
|
||||
await brain.add("Pricing is too high"),
|
||||
await brain.add("Too expensive for what it offers")
|
||||
]
|
||||
|
||||
const themes = await neural.clusters()
|
||||
// Results in 3 clusters: bugs, support, pricing
|
||||
```
|
||||
|
||||
#### Advanced Clustering Options
|
||||
|
||||
```javascript
|
||||
// Control clustering behavior
|
||||
const clusters = await neural.clusters({
|
||||
algorithm: 'kmeans', // Algorithm to use
|
||||
maxClusters: 5, // Maximum clusters to create
|
||||
threshold: 0.7 // Minimum similarity within clusters
|
||||
})
|
||||
|
||||
// Cluster specific items only
|
||||
const techItems = ['id1', 'id2', 'id3', 'id4']
|
||||
const techClusters = await neural.clusters(techItems)
|
||||
|
||||
// Find clusters near a specific item
|
||||
const relatedClusters = await neural.clusters('central-item-id')
|
||||
```
|
||||
|
||||
### 2. Similarity Calculation
|
||||
|
||||
Compare any two items to see how similar they are:
|
||||
|
||||
```javascript
|
||||
// Compare by ID
|
||||
const score = await neural.similar('item1-id', 'item2-id')
|
||||
// Returns 0-1 (0 = completely different, 1 = identical)
|
||||
|
||||
// Compare text directly
|
||||
const score = await neural.similar(
|
||||
"Machine learning is fascinating",
|
||||
"AI and deep learning are interesting"
|
||||
)
|
||||
// Returns ~0.75 (pretty similar)
|
||||
|
||||
// Compare vectors
|
||||
const v1 = await brain.embed("concept 1")
|
||||
const v2 = await brain.embed("concept 2")
|
||||
const score = await neural.similar(v1, v2)
|
||||
|
||||
// Get detailed similarity analysis
|
||||
const detailed = await neural.similar('id1', 'id2', {
|
||||
detailed: true
|
||||
})
|
||||
// Returns: {
|
||||
// score: 0.85,
|
||||
// confidence: 0.92,
|
||||
// explanation: "High semantic overlap in technology domain"
|
||||
// }
|
||||
```
|
||||
|
||||
### 3. Finding Neighbors
|
||||
|
||||
Discover items similar to a given item:
|
||||
|
||||
```javascript
|
||||
// Find 5 most similar items
|
||||
const neighbors = await neural.neighbors('item-id', 5)
|
||||
|
||||
// Each neighbor has:
|
||||
// - id: The neighbor's ID
|
||||
// - similarity: How similar (0-1)
|
||||
// - data: The actual content
|
||||
|
||||
// Example: Recommend similar articles
|
||||
const articleId = await brain.add("Guide to React Hooks")
|
||||
const similar = await neural.neighbors(articleId, 3)
|
||||
|
||||
for (const article of similar) {
|
||||
console.log(`${article.similarity * 100}% similar: ${article.data}`)
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Semantic Hierarchy
|
||||
|
||||
Build a hierarchy showing relationships between items:
|
||||
|
||||
```javascript
|
||||
const hierarchy = await neural.hierarchy('item-id')
|
||||
|
||||
// Returns structure like:
|
||||
// {
|
||||
// self: { id: 'item-id', type: 'article' },
|
||||
// parent: { id: 'parent-id', similarity: 0.8 },
|
||||
// siblings: [
|
||||
// { id: 'sibling1', similarity: 0.75 },
|
||||
// { id: 'sibling2', similarity: 0.72 }
|
||||
// ],
|
||||
// children: [
|
||||
// { id: 'child1', similarity: 0.85 }
|
||||
// ]
|
||||
// }
|
||||
|
||||
// Use for navigation or breadcrumbs
|
||||
const hier = await neural.hierarchy(currentDoc)
|
||||
console.log(`You are here: ${hier.self.id}`)
|
||||
if (hier.parent) {
|
||||
console.log(`Parent topic: ${hier.parent.id}`)
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Outlier Detection
|
||||
|
||||
Find unusual or anomalous items in your data:
|
||||
|
||||
```javascript
|
||||
// Find items that don't fit patterns
|
||||
const outliers = await neural.outliers(0.3)
|
||||
// Returns array of IDs that are > 0.3 distance from others
|
||||
|
||||
// Example: Detect spam or unusual content
|
||||
const messages = [
|
||||
await brain.add("Meeting at 3pm"),
|
||||
await brain.add("Lunch plans for tomorrow"),
|
||||
await brain.add("BUY NOW!!! AMAZING DEALS!!!"),
|
||||
await brain.add("Project deadline next week")
|
||||
]
|
||||
|
||||
const suspicious = await neural.outliers(0.4)
|
||||
// Returns the spam message ID
|
||||
```
|
||||
|
||||
### 6. Visualization Support
|
||||
|
||||
Generate data for visualization libraries:
|
||||
|
||||
```javascript
|
||||
// Create force-directed graph data
|
||||
const vizData = await neural.visualize({
|
||||
maxNodes: 100, // Limit nodes for performance
|
||||
dimensions: 2, // 2D or 3D
|
||||
algorithm: 'force' // Layout algorithm
|
||||
})
|
||||
|
||||
// Returns:
|
||||
// {
|
||||
// nodes: [
|
||||
// { id: 'n1', x: 10, y: 20, cluster: 'c1' },
|
||||
// { id: 'n2', x: 30, y: 40, cluster: 'c1' }
|
||||
// ],
|
||||
// edges: [
|
||||
// { source: 'n1', target: 'n2', weight: 0.8 }
|
||||
// ],
|
||||
// clusters: [
|
||||
// { id: 'c1', color: '#ff6b6b', size: 15 }
|
||||
// ]
|
||||
// }
|
||||
|
||||
// Use with D3.js, Cytoscape, or other viz libraries
|
||||
const data = await neural.visualize({ dimensions: 3 })
|
||||
// Now feed to Three.js for 3D visualization
|
||||
```
|
||||
|
||||
## Practical Examples
|
||||
|
||||
### Content Recommendation System
|
||||
|
||||
```javascript
|
||||
// User reads an article
|
||||
const currentArticle = 'article-123'
|
||||
|
||||
// Find similar content
|
||||
const recommendations = await neural.neighbors(currentArticle, 5)
|
||||
|
||||
// Group all content into topics
|
||||
const topics = await neural.clusters()
|
||||
|
||||
// Find which topic this article belongs to
|
||||
const currentTopic = topics.find(t =>
|
||||
t.members.includes(currentArticle)
|
||||
)
|
||||
|
||||
// Recommend from same topic first, then similar items
|
||||
const sameTopicArticles = currentTopic.members
|
||||
.filter(id => id !== currentArticle)
|
||||
.slice(0, 3)
|
||||
```
|
||||
|
||||
### Customer Feedback Analysis
|
||||
|
||||
```javascript
|
||||
import { NounType } from '@soulcraft/brainy'
|
||||
|
||||
// Add feedback with metadata
|
||||
const feedbackIds = []
|
||||
for (const feedback of customerFeedback) {
|
||||
const id = await brain.add({
|
||||
data: feedback.text,
|
||||
type: NounType.Document,
|
||||
metadata: {
|
||||
rating: feedback.rating,
|
||||
date: feedback.date,
|
||||
product: feedback.product
|
||||
}
|
||||
})
|
||||
feedbackIds.push(id)
|
||||
}
|
||||
|
||||
// Cluster to find themes
|
||||
const neural = brain.neural()
|
||||
const themes = await neural.clusters(feedbackIds)
|
||||
|
||||
// Analyze each theme
|
||||
for (const theme of themes) {
|
||||
// Get items using Promise.all with brain.get()
|
||||
const items = await Promise.all(
|
||||
theme.members.map(id => brain.get(id))
|
||||
)
|
||||
|
||||
const avgRating = items.reduce((sum, item) =>
|
||||
sum + (item?.metadata?.rating || 0), 0) / items.length
|
||||
|
||||
console.log(`Theme with ${theme.members.length} items`)
|
||||
console.log(`Average rating: ${avgRating}`)
|
||||
|
||||
// Find representative feedback for this theme
|
||||
const centroidId = theme.members[0] // Closest to center
|
||||
const example = await brain.get(centroidId)
|
||||
console.log(`Example: "${example?.data}"`)
|
||||
}
|
||||
```
|
||||
|
||||
### Knowledge Base Organization
|
||||
|
||||
```javascript
|
||||
import { NounType } from '@soulcraft/brainy'
|
||||
|
||||
// Analyze existing knowledge base - use find() to get documents
|
||||
const allDocs = await brain.find({ type: NounType.Document, limit: 1000 })
|
||||
|
||||
// Access neural API
|
||||
const neural = brain.neural()
|
||||
|
||||
// Find duplicate or highly similar content
|
||||
const duplicates = []
|
||||
for (let i = 0; i < allDocs.length; i++) {
|
||||
for (let j = i + 1; j < allDocs.length; j++) {
|
||||
const similarity = await neural.similar(
|
||||
allDocs[i].id,
|
||||
allDocs[j].id
|
||||
)
|
||||
if (similarity > 0.95) {
|
||||
duplicates.push([allDocs[i].id, allDocs[j].id])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Build topic hierarchy
|
||||
const mainTopics = await neural.clusters({
|
||||
maxClusters: 10,
|
||||
algorithm: 'hierarchical'
|
||||
})
|
||||
|
||||
// For each main topic, find subtopics
|
||||
for (const topic of mainTopics) {
|
||||
const subtopics = await neural.clusters(topic.members)
|
||||
console.log(`Topic has ${subtopics.length} subtopics`)
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Caching**: Neural API automatically caches results. Repeated calls with same parameters are instant.
|
||||
|
||||
2. **Batch Operations**: Process multiple items together rather than one at a time.
|
||||
|
||||
3. **Sampling**: For large datasets, use sampling:
|
||||
```javascript
|
||||
const clusters = await neural.clusters({
|
||||
algorithm: 'sample',
|
||||
sampleSize: 1000 // Only analyze 1000 items
|
||||
})
|
||||
```
|
||||
|
||||
4. **Async Processing**: All neural operations are async and non-blocking.
|
||||
|
||||
## Error Handling
|
||||
|
||||
```javascript
|
||||
try {
|
||||
const similarity = await neural.similar('id1', 'id2')
|
||||
} catch (error) {
|
||||
// Handle errors
|
||||
if (error.message.includes('not found')) {
|
||||
console.log('One of the items does not exist')
|
||||
}
|
||||
}
|
||||
|
||||
// Safe clustering with empty data
|
||||
const clusters = await neural.clusters([])
|
||||
// Returns empty array, doesn't throw
|
||||
|
||||
// Non-existent IDs return 0 similarity
|
||||
const sim = await neural.similar('fake-id-1', 'fake-id-2')
|
||||
// Returns 0
|
||||
```
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
```javascript
|
||||
// Configure neural behavior at initialization
|
||||
const brain = new Brainy({
|
||||
neural: {
|
||||
cacheSize: 1000, // Cache up to 1000 results
|
||||
defaultAlgorithm: 'kmeans',
|
||||
similarityMetric: 'cosine'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Explore [Triple Intelligence](../architecture/triple-intelligence.md) for combined vector + graph + metadata queries
|
||||
- Learn about [Plugins](../PLUGINS.md) to extend Brainy with native providers
|
||||
- See [API Reference](../api/README.md) for complete method documentation
|
||||
|
|
@ -86,7 +86,7 @@ including vector search:
|
|||
```typescript
|
||||
const db = await Brainy.load('/backups/2026-06-11')
|
||||
|
||||
const hits = await db.search('unpaid invoices from the spring campaign')
|
||||
const hits = await db.find({ query: 'unpaid invoices from the spring campaign' })
|
||||
const orders = await db.find({ type: NounType.Document, subtype: 'order' })
|
||||
|
||||
await db.release() // closes the underlying read-only instance
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ import { Brainy } from '@soulcraft/brainy'
|
|||
|
||||
// Filesystem (recommended for any persistent workload):
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: './brainy-data' }
|
||||
storage: { type: 'filesystem', path: './brainy-data' }
|
||||
})
|
||||
|
||||
// Memory (tests, ephemeral):
|
||||
|
|
@ -102,20 +102,33 @@ Brainy 8.0 is smaller, faster to install, and clearer about what it does.
|
|||
// BrainyConfig['storage'] — either a config object or a pre-constructed adapter:
|
||||
storage?:
|
||||
| {
|
||||
// The adapter type. Defaults to 'auto'
|
||||
// The adapter type. Optional — a top-level `path` implies 'filesystem',
|
||||
// so { path: '/data' } works without it. Defaults to 'auto'
|
||||
// (filesystem on Node-like runtimes, memory otherwise).
|
||||
type: 'auto' | 'memory' | 'filesystem'
|
||||
type?: 'auto' | 'memory' | 'filesystem'
|
||||
|
||||
// Root directory for filesystem storage. Passed through to storage
|
||||
// factories, including plugin-provided ones.
|
||||
// CANONICAL directory for filesystem storage. The rest of the API already
|
||||
// speaks `path` (persist(path), Brainy.load(path), asOf(path),
|
||||
// restore(path)). Specifying it implies type: 'filesystem'. Passed through
|
||||
// to storage factories, including plugin-provided ones.
|
||||
path?: string
|
||||
|
||||
// REMOVED in 8.0 — passing `rootDirectory` THROWS. Use `path`.
|
||||
rootDirectory?: string
|
||||
|
||||
// Adapter-specific options.
|
||||
// REMOVED in 8.0 — a nested `options.{path,rootDirectory}` THROWS. Use `path`.
|
||||
options?: any
|
||||
}
|
||||
| StorageAdapter // e.g. storage: new MemoryStorage()
|
||||
```
|
||||
|
||||
The canonical — and only — key is the top-level **`path`**. The pre-8.0 aliases
|
||||
`rootDirectory`, `options.{path,rootDirectory}`, and
|
||||
`fileSystemStorage.{path,rootDirectory}` were **removed in 8.0**: passing one now
|
||||
throws with the exact rename (never a silent default that would misplace data on
|
||||
upgrade). Because `path` implies filesystem, `{ path: '/data' }` is a complete
|
||||
config; the `type` is optional.
|
||||
|
||||
## Direct construction
|
||||
|
||||
If you want to skip the factory:
|
||||
|
|
@ -138,7 +151,7 @@ backup tooling. The recipe:
|
|||
1. On the host running Brainy, mount a local disk (NVMe recommended). Cloud
|
||||
providers all expose persistent local disks: GCP Persistent Disk, AWS EBS,
|
||||
Azure Managed Disks.
|
||||
2. Set `storage: { type: 'filesystem', rootDirectory: '/mnt/brainy-data' }`.
|
||||
2. Set `storage: { type: 'filesystem', path: '/mnt/brainy-data' }`.
|
||||
3. Run your existing data import once into the new local store.
|
||||
4. Set up an operator backup job using `gsutil` / `aws s3` / `rclone` /
|
||||
`azcopy` on a cron — hourly or whatever your RPO requires. Point it at
|
||||
|
|
|
|||
|
|
@ -242,7 +242,7 @@ A realistic adoption sequence for a brain that started without these primitives:
|
|||
```typescript
|
||||
import { Brainy, NounType } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', options: { path: './brain-data' } } })
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', path: './brain-data' } })
|
||||
await brain.init()
|
||||
|
||||
// 1. Migrate any existing metadata.kind convention to the new top-level subtype
|
||||
|
|
|
|||
|
|
@ -451,7 +451,7 @@ const euWestBrain = new Brainy({
|
|||
// Route queries based on user location
|
||||
async function search(query, userRegion) {
|
||||
const brain = userRegion === 'US' ? usEastBrain : euWestBrain
|
||||
return await brain.search(query)
|
||||
return await brain.find(query)
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -500,7 +500,7 @@ if (stats.fairness.fairnessViolation) {
|
|||
```typescript
|
||||
// Track search latency
|
||||
console.time('search')
|
||||
const results = await brain.search('query')
|
||||
const results = await brain.find('query')
|
||||
console.timeEnd('search') // Target: <10ms for hot queries
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -187,7 +187,7 @@ Transactions go through the `StorageAdapter` interface, so both shipped adapters
|
|||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: './data' }
|
||||
storage: { type: 'filesystem', path: './data' }
|
||||
})
|
||||
|
||||
await brain.add({ data: { name: 'Entity' }, type: NounType.Thing })
|
||||
|
|
|
|||
|
|
@ -162,7 +162,7 @@ const brain = new Brainy({
|
|||
|
||||
3. **Test with exact match**
|
||||
```typescript
|
||||
const results = await brain.search("test content") // Exact text
|
||||
const results = await brain.find("test content") // Exact text
|
||||
console.log('Exact match results:', results)
|
||||
```
|
||||
|
||||
|
|
@ -175,12 +175,13 @@ const brain = new Brainy({
|
|||
1. **Add more context to queries**
|
||||
```typescript
|
||||
// Instead of: "cat"
|
||||
const results = await brain.search("domestic cat animal pet")
|
||||
const results = await brain.find("domestic cat animal pet")
|
||||
```
|
||||
|
||||
2. **Use metadata filtering**
|
||||
```typescript
|
||||
const results = await brain.search("animals", {
|
||||
const results = await brain.find({
|
||||
query: "animals",
|
||||
where: { category: "pets" },
|
||||
limit: 10
|
||||
})
|
||||
|
|
@ -219,13 +220,14 @@ const brain = new Brainy({
|
|||
2. **Use appropriate limits**
|
||||
```typescript
|
||||
// Don't fetch more than needed
|
||||
const results = await brain.search("query", { limit: 10 })
|
||||
const results = await brain.find({ query: "query", limit: 10 })
|
||||
```
|
||||
|
||||
3. **Consider metadata filtering first**
|
||||
```typescript
|
||||
// Filter by metadata first, then semantic search
|
||||
const results = await brain.search("query", {
|
||||
const results = await brain.find({
|
||||
query: "query",
|
||||
where: { category: "specific" }, // Reduces search space
|
||||
limit: 10
|
||||
})
|
||||
|
|
@ -364,7 +366,7 @@ try {
|
|||
await brain.init()
|
||||
|
||||
const id = await brain.add("health check", { nounType: 'content' })
|
||||
const results = await brain.search("health")
|
||||
const results = await brain.find("health")
|
||||
|
||||
console.log('✅ Brainy is working correctly')
|
||||
console.log(`Added item: ${id}`)
|
||||
|
|
|
|||
|
|
@ -380,7 +380,7 @@ displayAug.configure({
|
|||
|
||||
```typescript
|
||||
// Precompute display fields for better performance
|
||||
const entities = await brainy.search('*', { limit: 100 })
|
||||
const entities = await brainy.find({ limit: 100 })
|
||||
await displayAug.precomputeBatch(
|
||||
entities.map(e => ({ id: e.id, data: e.metadata }))
|
||||
)
|
||||
|
|
|
|||
|
|
@ -84,7 +84,7 @@ const brain = new Brainy({
|
|||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: '/var/lib/brainy'
|
||||
path: '/var/lib/brainy'
|
||||
}
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -636,7 +636,7 @@ const brain = new Brainy({ storage: { type: 'memory' } })
|
|||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: './brainy-data'
|
||||
path: './brainy-data'
|
||||
}
|
||||
})
|
||||
|
||||
|
|
@ -644,7 +644,7 @@ const brain = new Brainy({
|
|||
const vfs = new VirtualFileSystem(brain)
|
||||
```
|
||||
|
||||
For off-site backup of the filesystem artifact, snapshot `rootDirectory` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, `tar`).
|
||||
For off-site backup of the filesystem artifact, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, `tar`).
|
||||
|
||||
**Custom Storage Adapters:** Redis, PostgreSQL, and other databases can be added via the [extension system](../api/EXTENSIBILITY.md). See `src/config/extensibleConfig.ts` for examples.
|
||||
|
||||
|
|
|
|||
|
|
@ -110,7 +110,8 @@ While using standard graph types, VFS adds domain-specific metadata for efficien
|
|||
### 1. Cross-Domain Queries
|
||||
```javascript
|
||||
// Find all documents (including VFS files) about "machine learning"
|
||||
const results = await brain.search('machine learning', {
|
||||
const results = await brain.find({
|
||||
query: 'machine learning',
|
||||
where: { type: NounType.Document }
|
||||
})
|
||||
```
|
||||
|
|
|
|||
|
|
@ -149,7 +149,7 @@ API: a persisted snapshot contains every VFS file, and a snapshot opened
|
|||
with `Brainy.load()` serves VFS reads at the snapshot's state:
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './data' } })
|
||||
const brain = new Brainy({ storage: { type: 'filesystem', path: './data' } })
|
||||
await brain.init()
|
||||
|
||||
// Create files
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue