soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat(8.0): API simplification — remove neural()/Db.search, one storage path key, integration→0

Browse source 
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":

- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
  legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
  / `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
  entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
  NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
  Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
  now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
  (`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
  exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
  feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
  storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
  applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
  and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
  shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
  in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
  Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
  flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.

Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
This commit is contained in:
David Snelling 2026-06-20 13:31:11 -07:00
parent 0c4a51c24e
commit 606445cd61
74 changed files with 712 additions and 7470 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/guides/find-limits.md
View file

@ -74,7 +74,7 @@ When the auto-config is wrong for your workload (e.g. you know your entities are
```typescript
const brain = new Brainy({
storage: { type: 'filesystem', options: { path: './data' } },
storage: { type: 'filesystem', path: './data' },
maxQueryLimit: 50_000 // raises the cap; still hard-clamped at 100 000
})
```

4
docs/guides/framework-integration.md
View file

@ -274,7 +274,7 @@ export function getBrain() {
if (!brainPromise) {
brainPromise = (async () => {
const brain = new Brainy({
storage: { type: 'filesystem', rootDirectory: './data' }
storage: { type: 'filesystem', path: './data' }
})
await brain.init()
return brain
@ -470,7 +470,7 @@ import { Brainy } from '@soulcraft/brainy'
export async function generateStaticProps() {
const brain = new Brainy({
storage: { type: 'filesystem', rootDirectory: './content' }
storage: { type: 'filesystem', path: './content' }
})
await brain.init()

2
docs/guides/import-anything.md
View file

@ -260,7 +260,7 @@ Once imported, use Triple Intelligence to query:
```javascript
// Vector search
const similar = await brain.search('engineers')
const similar = await brain.find('engineers')
// Natural language
const results = await brain.find('people in engineering who joined this year')

6
docs/guides/import-flow.md
View file

@ -1707,12 +1707,12 @@ Brainy 8.0 ships two adapters: filesystem and memory.
const brain = await Brainy.create({
storage: {
type: 'filesystem',
rootDirectory: './.brainy'
path: './.brainy'
}
})
```
For off-site backup, snapshot `rootDirectory` from your scheduler with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`. Brainy itself doesn't reach out to object storage.
For off-site backup, snapshot `path` from your scheduler with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`. Brainy itself doesn't reach out to object storage.
---
@ -1743,7 +1743,7 @@ For off-site backup, snapshot `rootDirectory` from your scheduler with `gsutil r
- HNSW Index: O(log n) search (1B entities = ~30 hops)
- Metadata Index: O(1) filtering
- Graph Adjacency: O(1) relationship lookups
- Storage: Bounded by the filesystem volume backing `rootDirectory`
- Storage: Bounded by the filesystem volume backing `path`
### Optimization Tips

2
docs/guides/inspection.md
View file

@ -111,7 +111,7 @@ check fails — useful for piping into monitoring or CI.
import { Brainy } from '@soulcraft/brainy'
const reader = await Brainy.openReadOnly({
storage: { type: 'filesystem', rootDirectory: '/data/brain' }
storage: { type: 'filesystem', path: '/data/brain' }
})
// Force the writer to flush before reading

371
docs/guides/neural-api.md
View file

@ -1,371 +0,0 @@
# Neural API Guide
> Semantic intelligence features for clustering, similarity, and analysis
## Overview
The Neural API provides advanced AI-powered features for understanding relationships and patterns in your data. Access it through `brain.neural()` (method call) after initializing Brainy.
## Quick Start
```javascript
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy()
await brain.init()
// Access Neural API (note: neural() is a method call)
const neural = brain.neural()
// Find similar items
const similarity = await neural.similar('text1', 'text2')
// Auto-cluster your data
const clusters = await neural.clusters()
```
## Core Features
### 1. Semantic Clustering
Automatically group related items based on their meaning:
```javascript
// Simple clustering - let Brainy decide
const clusters = await neural.clusters()
// Each cluster contains:
// - id: Unique identifier
// - members: Array of item IDs in this cluster
// - centroid: The "center" of the cluster
// - label: Optional descriptive label
// - confidence: How confident the clustering is
// Example: Organize customer feedback
const feedback = [
await brain.add("The app crashes when I upload photos"),
await brain.add("Photo upload feature is broken"),
await brain.add("Great customer service!"),
await brain.add("Support team was very helpful"),
await brain.add("Pricing is too high"),
await brain.add("Too expensive for what it offers")
]
const themes = await neural.clusters()
// Results in 3 clusters: bugs, support, pricing
```
#### Advanced Clustering Options
```javascript
// Control clustering behavior
const clusters = await neural.clusters({
algorithm: 'kmeans', // Algorithm to use
maxClusters: 5, // Maximum clusters to create
threshold: 0.7 // Minimum similarity within clusters
})
// Cluster specific items only
const techItems = ['id1', 'id2', 'id3', 'id4']
const techClusters = await neural.clusters(techItems)
// Find clusters near a specific item
const relatedClusters = await neural.clusters('central-item-id')
```
### 2. Similarity Calculation
Compare any two items to see how similar they are:
```javascript
// Compare by ID
const score = await neural.similar('item1-id', 'item2-id')
// Returns 0-1 (0 = completely different, 1 = identical)
// Compare text directly
const score = await neural.similar(
"Machine learning is fascinating",
"AI and deep learning are interesting"
)
// Returns ~0.75 (pretty similar)
// Compare vectors
const v1 = await brain.embed("concept 1")
const v2 = await brain.embed("concept 2")
const score = await neural.similar(v1, v2)
// Get detailed similarity analysis
const detailed = await neural.similar('id1', 'id2', {
detailed: true
})
// Returns: {
// score: 0.85,
// confidence: 0.92,
// explanation: "High semantic overlap in technology domain"
// }
```
### 3. Finding Neighbors
Discover items similar to a given item:
```javascript
// Find 5 most similar items
const neighbors = await neural.neighbors('item-id', 5)
// Each neighbor has:
// - id: The neighbor's ID
// - similarity: How similar (0-1)
// - data: The actual content
// Example: Recommend similar articles
const articleId = await brain.add("Guide to React Hooks")
const similar = await neural.neighbors(articleId, 3)
for (const article of similar) {
console.log(`${article.similarity * 100}% similar: ${article.data}`)
}
```
### 4. Semantic Hierarchy
Build a hierarchy showing relationships between items:
```javascript
const hierarchy = await neural.hierarchy('item-id')
// Returns structure like:
// {
// self: { id: 'item-id', type: 'article' },
// parent: { id: 'parent-id', similarity: 0.8 },
// siblings: [
// { id: 'sibling1', similarity: 0.75 },
// { id: 'sibling2', similarity: 0.72 }
// ],
// children: [
// { id: 'child1', similarity: 0.85 }
// ]
// }
// Use for navigation or breadcrumbs
const hier = await neural.hierarchy(currentDoc)
console.log(`You are here: ${hier.self.id}`)
if (hier.parent) {
console.log(`Parent topic: ${hier.parent.id}`)
}
```
### 5. Outlier Detection
Find unusual or anomalous items in your data:
```javascript
// Find items that don't fit patterns
const outliers = await neural.outliers(0.3)
// Returns array of IDs that are > 0.3 distance from others
// Example: Detect spam or unusual content
const messages = [
await brain.add("Meeting at 3pm"),
await brain.add("Lunch plans for tomorrow"),
await brain.add("BUY NOW!!! AMAZING DEALS!!!"),
await brain.add("Project deadline next week")
]
const suspicious = await neural.outliers(0.4)
// Returns the spam message ID
```
### 6. Visualization Support
Generate data for visualization libraries:
```javascript
// Create force-directed graph data
const vizData = await neural.visualize({
maxNodes: 100, // Limit nodes for performance
dimensions: 2, // 2D or 3D
algorithm: 'force' // Layout algorithm
})
// Returns:
// {
// nodes: [
// { id: 'n1', x: 10, y: 20, cluster: 'c1' },
// { id: 'n2', x: 30, y: 40, cluster: 'c1' }
// ],
// edges: [
// { source: 'n1', target: 'n2', weight: 0.8 }
// ],
// clusters: [
// { id: 'c1', color: '#ff6b6b', size: 15 }
// ]
// }
// Use with D3.js, Cytoscape, or other viz libraries
const data = await neural.visualize({ dimensions: 3 })
// Now feed to Three.js for 3D visualization
```
## Practical Examples
### Content Recommendation System
```javascript
// User reads an article
const currentArticle = 'article-123'
// Find similar content
const recommendations = await neural.neighbors(currentArticle, 5)
// Group all content into topics
const topics = await neural.clusters()
// Find which topic this article belongs to
const currentTopic = topics.find(t =>
t.members.includes(currentArticle)
)
// Recommend from same topic first, then similar items
const sameTopicArticles = currentTopic.members
.filter(id => id !== currentArticle)
.slice(0, 3)
```
### Customer Feedback Analysis
```javascript
import { NounType } from '@soulcraft/brainy'
// Add feedback with metadata
const feedbackIds = []
for (const feedback of customerFeedback) {
const id = await brain.add({
data: feedback.text,
type: NounType.Document,
metadata: {
rating: feedback.rating,
date: feedback.date,
product: feedback.product
}
})
feedbackIds.push(id)
}
// Cluster to find themes
const neural = brain.neural()
const themes = await neural.clusters(feedbackIds)
// Analyze each theme
for (const theme of themes) {
// Get items using Promise.all with brain.get()
const items = await Promise.all(
theme.members.map(id => brain.get(id))
)
const avgRating = items.reduce((sum, item) =>
sum + (item?.metadata?.rating || 0), 0) / items.length
console.log(`Theme with ${theme.members.length} items`)
console.log(`Average rating: ${avgRating}`)
// Find representative feedback for this theme
const centroidId = theme.members[0] // Closest to center
const example = await brain.get(centroidId)
console.log(`Example: "${example?.data}"`)
}
```
### Knowledge Base Organization
```javascript
import { NounType } from '@soulcraft/brainy'
// Analyze existing knowledge base - use find() to get documents
const allDocs = await brain.find({ type: NounType.Document, limit: 1000 })
// Access neural API
const neural = brain.neural()
// Find duplicate or highly similar content
const duplicates = []
for (let i = 0; i < allDocs.length; i++) {
for (let j = i + 1; j < allDocs.length; j++) {
const similarity = await neural.similar(
allDocs[i].id,
allDocs[j].id
)
if (similarity > 0.95) {
duplicates.push([allDocs[i].id, allDocs[j].id])
}
}
}
// Build topic hierarchy
const mainTopics = await neural.clusters({
maxClusters: 10,
algorithm: 'hierarchical'
})
// For each main topic, find subtopics
for (const topic of mainTopics) {
const subtopics = await neural.clusters(topic.members)
console.log(`Topic has ${subtopics.length} subtopics`)
}
```
## Performance Tips
1. **Caching**: Neural API automatically caches results. Repeated calls with same parameters are instant.
2. **Batch Operations**: Process multiple items together rather than one at a time.
3. **Sampling**: For large datasets, use sampling:
```javascript
const clusters = await neural.clusters({
algorithm: 'sample',
sampleSize: 1000 // Only analyze 1000 items
})
```
4. **Async Processing**: All neural operations are async and non-blocking.
## Error Handling
```javascript
try {
const similarity = await neural.similar('id1', 'id2')
} catch (error) {
// Handle errors
if (error.message.includes('not found')) {
console.log('One of the items does not exist')
}
}
// Safe clustering with empty data
const clusters = await neural.clusters([])
// Returns empty array, doesn't throw
// Non-existent IDs return 0 similarity
const sim = await neural.similar('fake-id-1', 'fake-id-2')
// Returns 0
```
## Advanced Configuration
```javascript
// Configure neural behavior at initialization
const brain = new Brainy({
neural: {
cacheSize: 1000, // Cache up to 1000 results
defaultAlgorithm: 'kmeans',
similarityMetric: 'cosine'
}
})
```
## Next Steps
- Explore [Triple Intelligence](../architecture/triple-intelligence.md) for combined vector + graph + metadata queries
- Learn about [Plugins](../PLUGINS.md) to extend Brainy with native providers
- See [API Reference](../api/README.md) for complete method documentation

2
docs/guides/snapshots-and-time-travel.md
View file

@ -86,7 +86,7 @@ including vector search:
```typescript
const db = await Brainy.load('/backups/2026-06-11')
const hits = await db.search('unpaid invoices from the spring campaign')
const hits = await db.find({ query: 'unpaid invoices from the spring campaign' })
const orders = await db.find({ type: NounType.Document, subtype: 'order' })
await db.release() // closes the underlying read-only instance

27
docs/guides/storage-adapters.md
View file

@ -32,7 +32,7 @@ import { Brainy } from '@soulcraft/brainy'
// Filesystem (recommended for any persistent workload):
const brain = new Brainy({
storage: { type: 'filesystem', rootDirectory: './brainy-data' }
storage: { type: 'filesystem', path: './brainy-data' }
})
// Memory (tests, ephemeral):
@ -102,20 +102,33 @@ Brainy 8.0 is smaller, faster to install, and clearer about what it does.
// BrainyConfig['storage'] — either a config object or a pre-constructed adapter:
storage?:
| {
// The adapter type. Defaults to 'auto'
// The adapter type. Optional — a top-level `path` implies 'filesystem',
// so { path: '/data' } works without it. Defaults to 'auto'
// (filesystem on Node-like runtimes, memory otherwise).
type: 'auto' | 'memory' | 'filesystem'
type?: 'auto' | 'memory' | 'filesystem'
// Root directory for filesystem storage. Passed through to storage
// factories, including plugin-provided ones.
// CANONICAL directory for filesystem storage. The rest of the API already
// speaks `path` (persist(path), Brainy.load(path), asOf(path),
// restore(path)). Specifying it implies type: 'filesystem'. Passed through
// to storage factories, including plugin-provided ones.
path?: string
// REMOVED in 8.0 — passing `rootDirectory` THROWS. Use `path`.
rootDirectory?: string
// Adapter-specific options.
// REMOVED in 8.0 — a nested `options.{path,rootDirectory}` THROWS. Use `path`.
options?: any
}
| StorageAdapter // e.g. storage: new MemoryStorage()
```
The canonical — and only — key is the top-level **`path`**. The pre-8.0 aliases
`rootDirectory`, `options.{path,rootDirectory}`, and
`fileSystemStorage.{path,rootDirectory}` were **removed in 8.0**: passing one now
throws with the exact rename (never a silent default that would misplace data on
upgrade). Because `path` implies filesystem, `{ path: '/data' }` is a complete
config; the `type` is optional.
## Direct construction
If you want to skip the factory:
@ -138,7 +151,7 @@ backup tooling. The recipe:
1. On the host running Brainy, mount a local disk (NVMe recommended). Cloud
providers all expose persistent local disks: GCP Persistent Disk, AWS EBS,
Azure Managed Disks.
2. Set `storage: { type: 'filesystem', rootDirectory: '/mnt/brainy-data' }`.
2. Set `storage: { type: 'filesystem', path: '/mnt/brainy-data' }`.
3. Run your existing data import once into the new local store.
4. Set up an operator backup job using `gsutil` / `aws s3` / `rclone` /
`azcopy` on a cron — hourly or whatever your RPO requires. Point it at

2
docs/guides/subtypes-and-facets.md
View file

@ -242,7 +242,7 @@ A realistic adoption sequence for a brain that started without these primitives:
```typescript
import { Brainy, NounType } from '@soulcraft/brainy'
const brain = new Brainy({ storage: { type: 'filesystem', options: { path: './brain-data' } } })
const brain = new Brainy({ storage: { type: 'filesystem', path: './brain-data' } })
await brain.init()
// 1. Migrate any existing metadata.kind convention to the new top-level subtype