feat: add new embedding and analysis APIs for v7.1.0
New public APIs: - embedBatch(texts): Batch embed multiple texts efficiently - similarity(textA, textB): Calculate semantic similarity (0-1 score) - indexStats(): Get comprehensive index statistics with memory usage - neighbors(entityId, options): Get graph neighbors with direction/depth/filter - findDuplicates(options): Find semantic duplicates by embedding similarity - cluster(options): Cluster entities by semantic similarity with centroids All APIs: - Added to BrainyInterface for type safety - Documented in docs/API_REFERENCE.md and docs/api/README.md - Include JSDoc examples and parameter descriptions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
8e21e71980
commit
d8514ab209
4 changed files with 855 additions and 10 deletions
|
|
@ -491,6 +491,195 @@ for (const result of similar) {
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
### `async embed(data: any): Promise<Vector>` ✨ *v7.1.0*
|
||||||
|
Generates an embedding vector from data.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
- `data` - Text, array, or object to embed
|
||||||
|
|
||||||
|
**Returns:** 384-dimensional embedding vector
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```typescript
|
||||||
|
const vector = await brain.embed('Machine learning concepts')
|
||||||
|
console.log(vector.length) // 384
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `async embedBatch(texts: string[]): Promise<number[][]>` ✨ *v7.1.0*
|
||||||
|
Batch embed multiple texts efficiently.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
- `texts` - Array of strings to embed
|
||||||
|
|
||||||
|
**Returns:** Array of 384-dimensional vectors
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```typescript
|
||||||
|
const embeddings = await brain.embedBatch([
|
||||||
|
'Machine learning is fascinating',
|
||||||
|
'Deep neural networks',
|
||||||
|
'Natural language processing'
|
||||||
|
])
|
||||||
|
console.log(embeddings.length) // 3
|
||||||
|
console.log(embeddings[0].length) // 384
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `async similarity(textA: string, textB: string): Promise<number>` ✨ *v7.1.0*
|
||||||
|
Calculate semantic similarity between two texts.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
- `textA` - First text
|
||||||
|
- `textB` - Second text
|
||||||
|
|
||||||
|
**Returns:** Similarity score between 0 (different) and 1 (identical)
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```typescript
|
||||||
|
const score = await brain.similarity(
|
||||||
|
'The cat sat on the mat',
|
||||||
|
'A feline was resting on the rug'
|
||||||
|
)
|
||||||
|
console.log(score) // ~0.85 (high semantic similarity)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `async neighbors(entityId: string, options?): Promise<string[]>` ✨ *v7.1.0*
|
||||||
|
Get graph neighbors of an entity.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
- `entityId` - Entity to get neighbors for
|
||||||
|
- `options.direction` - 'outgoing', 'incoming', or 'both' (default: 'both')
|
||||||
|
- `options.depth` - Traversal depth (default: 1)
|
||||||
|
- `options.verbType` - Filter by relationship type
|
||||||
|
- `options.limit` - Maximum neighbors to return
|
||||||
|
|
||||||
|
**Returns:** Array of neighbor entity IDs
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```typescript
|
||||||
|
// Get all connected entities
|
||||||
|
const neighbors = await brain.neighbors(entityId)
|
||||||
|
|
||||||
|
// Get outgoing connections only
|
||||||
|
const outgoing = await brain.neighbors(entityId, {
|
||||||
|
direction: 'outgoing',
|
||||||
|
limit: 10
|
||||||
|
})
|
||||||
|
|
||||||
|
// Multi-hop traversal
|
||||||
|
const extended = await brain.neighbors(entityId, {
|
||||||
|
depth: 2,
|
||||||
|
direction: 'both'
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `async findDuplicates(options?): Promise<DuplicateResult[]>` ✨ *v7.1.0*
|
||||||
|
Find semantic duplicates in the database.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
- `options.threshold` - Minimum similarity (default: 0.85)
|
||||||
|
- `options.type` - Filter by NounType
|
||||||
|
- `options.limit` - Maximum duplicate groups (default: 100)
|
||||||
|
|
||||||
|
**Returns:** Array of duplicate groups with similarity scores
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```typescript
|
||||||
|
// Find all duplicates
|
||||||
|
const duplicates = await brain.findDuplicates()
|
||||||
|
|
||||||
|
for (const group of duplicates) {
|
||||||
|
console.log('Original:', group.entity.id)
|
||||||
|
for (const dup of group.duplicates) {
|
||||||
|
console.log(` Duplicate: ${dup.entity.id} (${dup.similarity.toFixed(2)})`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Find person duplicates with higher threshold
|
||||||
|
const personDupes = await brain.findDuplicates({
|
||||||
|
type: NounType.PERSON,
|
||||||
|
threshold: 0.9,
|
||||||
|
limit: 50
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `async indexStats(): Promise<IndexStats>` ✨ *v7.1.0*
|
||||||
|
Get comprehensive index statistics.
|
||||||
|
|
||||||
|
**Returns:**
|
||||||
|
- `entities` - Total entity count
|
||||||
|
- `vectors` - Total vectors in HNSW index
|
||||||
|
- `relationships` - Total relationships in graph
|
||||||
|
- `metadataFields` - Indexed metadata fields
|
||||||
|
- `memoryUsage.vectors` - Vector memory in bytes
|
||||||
|
- `memoryUsage.graph` - Graph memory in bytes
|
||||||
|
- `memoryUsage.metadata` - Metadata index memory in bytes
|
||||||
|
- `memoryUsage.total` - Total memory usage
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```typescript
|
||||||
|
const stats = await brain.indexStats()
|
||||||
|
console.log(`Entities: ${stats.entities}`)
|
||||||
|
console.log(`Vectors: ${stats.vectors}`)
|
||||||
|
console.log(`Relationships: ${stats.relationships}`)
|
||||||
|
console.log(`Memory: ${(stats.memoryUsage.total / 1024 / 1024).toFixed(1)}MB`)
|
||||||
|
console.log(`Fields: ${stats.metadataFields.join(', ')}`)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `async cluster(options?): Promise<ClusterResult[]>` ✨ *v7.1.0*
|
||||||
|
Cluster entities by semantic similarity.
|
||||||
|
|
||||||
|
Groups entities into clusters based on their embedding similarity using
|
||||||
|
a greedy algorithm with HNSW-based neighbor lookup.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
- `options.threshold` - Similarity threshold (default: 0.8)
|
||||||
|
- `options.type` - Filter by NounType
|
||||||
|
- `options.minClusterSize` - Minimum entities per cluster (default: 2)
|
||||||
|
- `options.limit` - Maximum clusters to return (default: 100)
|
||||||
|
- `options.includeCentroid` - Calculate cluster centroids (default: false)
|
||||||
|
|
||||||
|
**Returns:** Array of clusters with entities
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```typescript
|
||||||
|
// Find all clusters
|
||||||
|
const clusters = await brain.cluster()
|
||||||
|
|
||||||
|
for (const cluster of clusters) {
|
||||||
|
console.log(`${cluster.clusterId}: ${cluster.entities.length} entities`)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Find document clusters with centroids
|
||||||
|
const docClusters = await brain.cluster({
|
||||||
|
type: NounType.Document,
|
||||||
|
threshold: 0.85,
|
||||||
|
minClusterSize: 3,
|
||||||
|
includeCentroid: true
|
||||||
|
})
|
||||||
|
|
||||||
|
// Use centroids for cluster comparison
|
||||||
|
for (const cluster of docClusters) {
|
||||||
|
console.log(`${cluster.clusterId}: ${cluster.entities.length} entities`)
|
||||||
|
if (cluster.centroid) {
|
||||||
|
console.log(` Centroid dimensions: ${cluster.centroid.length}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### `async insights(): Promise<InsightsResult>`
|
### `async insights(): Promise<InsightsResult>`
|
||||||
Gets statistics and insights about the data.
|
Gets statistics and insights about the data.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,9 @@
|
||||||
# 🧠 Brainy v6.5.0 API Reference
|
# 🧠 Brainy v7.1.0 API Reference
|
||||||
|
|
||||||
> **Complete API documentation for Brainy v6.5.0**
|
> **Complete API documentation for Brainy v7.1.0**
|
||||||
> Zero Configuration • Triple Intelligence • Git-Style Branching • Entity Versioning
|
> Zero Configuration • Triple Intelligence • Git-Style Branching • Entity Versioning • Candle WASM Embeddings
|
||||||
|
|
||||||
**Updated:** 2025-12-11 for v6.5.0
|
**Updated:** 2026-01-06 for v7.1.0
|
||||||
**All APIs verified against actual code**
|
**All APIs verified against actual code**
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -76,6 +76,7 @@ Time-travel and history tracking for individual entities - Git-like version cont
|
||||||
- [Configuration](#configuration)
|
- [Configuration](#configuration)
|
||||||
- [Storage Adapters](#storage-adapters)
|
- [Storage Adapters](#storage-adapters)
|
||||||
- [Utility Methods](#utility-methods)
|
- [Utility Methods](#utility-methods)
|
||||||
|
- [Embedding & Analysis APIs (v7.1.0)](#embedding--analysis-apis-v710)
|
||||||
- [Type System Reference](#type-system-reference)
|
- [Type System Reference](#type-system-reference)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -1560,17 +1561,169 @@ const count = await brain.getVerbCount()
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### `embed(data)` → `Promise<number[]>`
|
### `embed(data)` → `Promise<number[]>` ✨ *Enhanced v7.1.0*
|
||||||
|
|
||||||
Generate embedding vector from text.
|
Generate embedding vector from text or data.
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
const vector = await brain.embed('Hello world')
|
const vector = await brain.embed('Hello world')
|
||||||
// [0.1, -0.3, 0.8, ...]
|
// 384-dimensional vector
|
||||||
|
console.log(vector.length) // 384
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
### `embedBatch(texts)` → `Promise<number[][]>` ✨ *New v7.1.0*
|
||||||
|
|
||||||
|
Batch embed multiple texts efficiently.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const embeddings = await brain.embedBatch([
|
||||||
|
'Machine learning is fascinating',
|
||||||
|
'Deep neural networks',
|
||||||
|
'Natural language processing'
|
||||||
|
])
|
||||||
|
console.log(embeddings.length) // 3
|
||||||
|
console.log(embeddings[0].length) // 384
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `similarity(textA, textB)` → `Promise<number>` ✨ *New v7.1.0*
|
||||||
|
|
||||||
|
Calculate semantic similarity between two texts.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const score = await brain.similarity(
|
||||||
|
'The cat sat on the mat',
|
||||||
|
'A feline was resting on the rug'
|
||||||
|
)
|
||||||
|
console.log(score) // ~0.85 (high semantic similarity)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Returns:** Score from 0 (different) to 1 (identical meaning)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `neighbors(entityId, options?)` → `Promise<string[]>` ✨ *New v7.1.0*
|
||||||
|
|
||||||
|
Get graph neighbors of an entity.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Get all connected entities
|
||||||
|
const neighbors = await brain.neighbors(entityId)
|
||||||
|
|
||||||
|
// Get outgoing connections only
|
||||||
|
const outgoing = await brain.neighbors(entityId, {
|
||||||
|
direction: 'outgoing',
|
||||||
|
limit: 10
|
||||||
|
})
|
||||||
|
|
||||||
|
// Multi-hop traversal
|
||||||
|
const extended = await brain.neighbors(entityId, {
|
||||||
|
depth: 2,
|
||||||
|
direction: 'both'
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `direction`: `'outgoing' | 'incoming' | 'both'` (default: 'both')
|
||||||
|
- `depth`: `number` - Traversal depth (default: 1)
|
||||||
|
- `verbType`: `VerbType` - Filter by relationship type
|
||||||
|
- `limit`: `number` - Maximum neighbors to return
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `findDuplicates(options?)` → `Promise<DuplicateResult[]>` ✨ *New v7.1.0*
|
||||||
|
|
||||||
|
Find semantic duplicates in the database.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Find all duplicates
|
||||||
|
const duplicates = await brain.findDuplicates()
|
||||||
|
|
||||||
|
for (const group of duplicates) {
|
||||||
|
console.log('Original:', group.entity.id)
|
||||||
|
for (const dup of group.duplicates) {
|
||||||
|
console.log(` Duplicate: ${dup.entity.id} (${dup.similarity.toFixed(2)})`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Find person duplicates with higher threshold
|
||||||
|
const personDupes = await brain.findDuplicates({
|
||||||
|
type: NounType.PERSON,
|
||||||
|
threshold: 0.9,
|
||||||
|
limit: 50
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `threshold`: `number` - Minimum similarity (default: 0.85)
|
||||||
|
- `type`: `NounType` - Filter by entity type
|
||||||
|
- `limit`: `number` - Maximum duplicate groups (default: 100)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `indexStats()` → `Promise<IndexStats>` ✨ *New v7.1.0*
|
||||||
|
|
||||||
|
Get comprehensive index statistics.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const stats = await brain.indexStats()
|
||||||
|
console.log(`Entities: ${stats.entities}`)
|
||||||
|
console.log(`Vectors: ${stats.vectors}`)
|
||||||
|
console.log(`Relationships: ${stats.relationships}`)
|
||||||
|
console.log(`Memory: ${(stats.memoryUsage.total / 1024 / 1024).toFixed(1)}MB`)
|
||||||
|
console.log(`Fields: ${stats.metadataFields.join(', ')}`)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Returns:**
|
||||||
|
- `entities` - Total entity count
|
||||||
|
- `vectors` - Total vectors in HNSW index
|
||||||
|
- `relationships` - Total relationships in graph
|
||||||
|
- `metadataFields` - Indexed metadata fields
|
||||||
|
- `memoryUsage.vectors` - Vector memory (bytes)
|
||||||
|
- `memoryUsage.graph` - Graph memory (bytes)
|
||||||
|
- `memoryUsage.metadata` - Metadata index memory (bytes)
|
||||||
|
- `memoryUsage.total` - Total memory usage
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `cluster(options?)` → `Promise<ClusterResult[]>` ✨ *New v7.1.0*
|
||||||
|
|
||||||
|
Cluster entities by semantic similarity.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Find all clusters
|
||||||
|
const clusters = await brain.cluster()
|
||||||
|
|
||||||
|
for (const cluster of clusters) {
|
||||||
|
console.log(`${cluster.clusterId}: ${cluster.entities.length} entities`)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Find document clusters with centroids
|
||||||
|
const docClusters = await brain.cluster({
|
||||||
|
type: NounType.Document,
|
||||||
|
threshold: 0.85,
|
||||||
|
minClusterSize: 3,
|
||||||
|
includeCentroid: true
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `threshold`: `number` - Similarity threshold (default: 0.8)
|
||||||
|
- `type`: `NounType` - Filter by entity type
|
||||||
|
- `minClusterSize`: `number` - Minimum cluster size (default: 2)
|
||||||
|
- `limit`: `number` - Maximum clusters to return (default: 100)
|
||||||
|
- `includeCentroid`: `boolean` - Calculate cluster centroids (default: false)
|
||||||
|
|
||||||
|
**Returns:**
|
||||||
|
- `clusterId` - Unique cluster identifier
|
||||||
|
- `entities` - Array of entities in the cluster
|
||||||
|
- `centroid` - Average embedding vector (if includeCentroid is true)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### `getStats()` → `Statistics`
|
### `getStats()` → `Statistics`
|
||||||
|
|
||||||
Get comprehensive statistics.
|
Get comprehensive statistics.
|
||||||
|
|
|
||||||
423
src/brainy.ts
423
src/brainy.ts
|
|
@ -4311,6 +4311,429 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
return this.counts.getStats(options)
|
return this.counts.getStats(options)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ============= NEW EMBEDDING & ANALYSIS APIs (v7.1.0) =============
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Batch embed multiple texts at once
|
||||||
|
*
|
||||||
|
* More efficient than calling embed() multiple times due to
|
||||||
|
* WASM batch processing optimizations.
|
||||||
|
*
|
||||||
|
* @param texts Array of texts to embed
|
||||||
|
* @returns Array of embedding vectors (384 dimensions each)
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* const embeddings = await brain.embedBatch([
|
||||||
|
* 'Machine learning is fascinating',
|
||||||
|
* 'Deep neural networks',
|
||||||
|
* 'Natural language processing'
|
||||||
|
* ])
|
||||||
|
* // embeddings.length === 3
|
||||||
|
* // embeddings[0].length === 384
|
||||||
|
*/
|
||||||
|
async embedBatch(texts: string[]): Promise<number[][]> {
|
||||||
|
await this.ensureInitialized()
|
||||||
|
|
||||||
|
if (texts.length === 0) {
|
||||||
|
return []
|
||||||
|
}
|
||||||
|
|
||||||
|
// Use the underlying embedder for each text
|
||||||
|
// The WASM engine handles batching internally
|
||||||
|
const embeddings = await Promise.all(
|
||||||
|
texts.map(text => this.embedder(text))
|
||||||
|
)
|
||||||
|
|
||||||
|
return embeddings
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Calculate semantic similarity between two texts
|
||||||
|
*
|
||||||
|
* Returns a score from 0 (completely different) to 1 (identical meaning).
|
||||||
|
* Uses cosine similarity on embedding vectors.
|
||||||
|
*
|
||||||
|
* @param textA First text
|
||||||
|
* @param textB Second text
|
||||||
|
* @returns Similarity score between 0 and 1
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* const score = await brain.similarity(
|
||||||
|
* 'The cat sat on the mat',
|
||||||
|
* 'A feline was resting on the rug'
|
||||||
|
* )
|
||||||
|
* // score ≈ 0.85 (high semantic similarity)
|
||||||
|
*/
|
||||||
|
async similarity(textA: string, textB: string): Promise<number> {
|
||||||
|
await this.ensureInitialized()
|
||||||
|
|
||||||
|
// Embed both texts
|
||||||
|
const [vectorA, vectorB] = await Promise.all([
|
||||||
|
this.embedder(textA),
|
||||||
|
this.embedder(textB)
|
||||||
|
])
|
||||||
|
|
||||||
|
// Calculate cosine similarity (convert from distance)
|
||||||
|
// cosineDistance returns 1 - similarity, so similarity = 1 - distance
|
||||||
|
const distance = this.distance(vectorA, vectorB)
|
||||||
|
return 1 - distance
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get comprehensive index statistics
|
||||||
|
*
|
||||||
|
* Returns detailed stats about all internal indexes including
|
||||||
|
* entity counts, vector index size, graph relationships, and
|
||||||
|
* estimated memory usage.
|
||||||
|
*
|
||||||
|
* @returns Index statistics object
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* const stats = await brain.indexStats()
|
||||||
|
* console.log(`Entities: ${stats.entities}`)
|
||||||
|
* console.log(`Vectors: ${stats.vectors}`)
|
||||||
|
* console.log(`Relationships: ${stats.relationships}`)
|
||||||
|
*/
|
||||||
|
async indexStats(): Promise<{
|
||||||
|
entities: number
|
||||||
|
vectors: number
|
||||||
|
relationships: number
|
||||||
|
metadataFields: string[]
|
||||||
|
memoryUsage: {
|
||||||
|
vectors: number
|
||||||
|
graph: number
|
||||||
|
metadata: number
|
||||||
|
total: number
|
||||||
|
}
|
||||||
|
}> {
|
||||||
|
await this.ensureInitialized()
|
||||||
|
|
||||||
|
const metadataStats = await this.metadataIndex.getStats()
|
||||||
|
const graphStats = this.graphIndex.getStats()
|
||||||
|
const vectorCount = this.index.size()
|
||||||
|
|
||||||
|
// Get unique metadata field names
|
||||||
|
const metadataFields = metadataStats.fieldsIndexed || []
|
||||||
|
|
||||||
|
return {
|
||||||
|
entities: metadataStats.totalEntries,
|
||||||
|
vectors: vectorCount,
|
||||||
|
relationships: graphStats.totalRelationships,
|
||||||
|
metadataFields,
|
||||||
|
memoryUsage: {
|
||||||
|
vectors: vectorCount * 384 * 4, // 384 dimensions * 4 bytes per float32
|
||||||
|
graph: graphStats.memoryUsage,
|
||||||
|
metadata: metadataStats.indexSize || 0,
|
||||||
|
total: (vectorCount * 384 * 4) + graphStats.memoryUsage + (metadataStats.indexSize || 0)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get graph neighbors of an entity
|
||||||
|
*
|
||||||
|
* Traverses the relationship graph to find connected entities.
|
||||||
|
* Supports filtering by direction and relationship type.
|
||||||
|
*
|
||||||
|
* @param entityId The entity to get neighbors for
|
||||||
|
* @param options Optional traversal options
|
||||||
|
* @returns Array of neighbor entity IDs
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* // Get all connected entities
|
||||||
|
* const allNeighbors = await brain.neighbors(entityId)
|
||||||
|
*
|
||||||
|
* // Get only outgoing connections
|
||||||
|
* const outgoing = await brain.neighbors(entityId, { direction: 'outgoing' })
|
||||||
|
*
|
||||||
|
* // Get incoming connections with specific verb type
|
||||||
|
* const incoming = await brain.neighbors(entityId, {
|
||||||
|
* direction: 'incoming',
|
||||||
|
* verbType: VerbType.RELATES_TO
|
||||||
|
* })
|
||||||
|
*/
|
||||||
|
async neighbors(
|
||||||
|
entityId: string,
|
||||||
|
options?: {
|
||||||
|
direction?: 'outgoing' | 'incoming' | 'both'
|
||||||
|
depth?: number
|
||||||
|
verbType?: VerbType
|
||||||
|
limit?: number
|
||||||
|
}
|
||||||
|
): Promise<string[]> {
|
||||||
|
await this.ensureInitialized()
|
||||||
|
|
||||||
|
const direction = options?.direction || 'both'
|
||||||
|
const limit = options?.limit
|
||||||
|
|
||||||
|
// Map our API direction to graphIndex direction
|
||||||
|
const graphDirection = direction === 'outgoing' ? 'out' :
|
||||||
|
direction === 'incoming' ? 'in' : 'both'
|
||||||
|
|
||||||
|
// Get neighbors from graph index
|
||||||
|
let neighbors = await this.graphIndex.getNeighbors(entityId, {
|
||||||
|
direction: graphDirection,
|
||||||
|
limit
|
||||||
|
})
|
||||||
|
|
||||||
|
// Filter by verb type if specified
|
||||||
|
if (options?.verbType) {
|
||||||
|
const filteredNeighbors: string[] = []
|
||||||
|
const verbIds = direction !== 'incoming'
|
||||||
|
? await this.graphIndex.getVerbIdsBySource(entityId)
|
||||||
|
: await this.graphIndex.getVerbIdsByTarget(entityId)
|
||||||
|
|
||||||
|
// Load verbs to check their types
|
||||||
|
const verbs = await this.graphIndex.getVerbsBatchCached(verbIds)
|
||||||
|
|
||||||
|
for (const [, verb] of verbs) {
|
||||||
|
if (verb.type === options.verbType || verb.verb === options.verbType) {
|
||||||
|
const neighborId = verb.sourceId === entityId ? verb.targetId : verb.sourceId
|
||||||
|
if (neighbors.includes(neighborId)) {
|
||||||
|
filteredNeighbors.push(neighborId)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
neighbors = filteredNeighbors
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle depth > 1 (multi-hop traversal)
|
||||||
|
if (options?.depth && options.depth > 1) {
|
||||||
|
const visited = new Set<string>([entityId, ...neighbors])
|
||||||
|
let currentLevel = neighbors
|
||||||
|
|
||||||
|
for (let d = 1; d < options.depth; d++) {
|
||||||
|
const nextLevel: string[] = []
|
||||||
|
|
||||||
|
for (const nodeId of currentLevel) {
|
||||||
|
const nodeNeighbors = await this.graphIndex.getNeighbors(nodeId, {
|
||||||
|
direction: graphDirection
|
||||||
|
})
|
||||||
|
|
||||||
|
for (const neighbor of nodeNeighbors) {
|
||||||
|
if (!visited.has(neighbor)) {
|
||||||
|
visited.add(neighbor)
|
||||||
|
nextLevel.push(neighbor)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (nextLevel.length === 0) break
|
||||||
|
currentLevel = nextLevel
|
||||||
|
neighbors.push(...nextLevel)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Apply limit after multi-hop traversal
|
||||||
|
if (limit && neighbors.length > limit) {
|
||||||
|
neighbors = neighbors.slice(0, limit)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return neighbors
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Find semantic duplicates in the database
|
||||||
|
*
|
||||||
|
* Uses embedding similarity to identify entities that may be
|
||||||
|
* duplicates or near-duplicates based on their content.
|
||||||
|
*
|
||||||
|
* @param options Optional search options
|
||||||
|
* @returns Array of duplicate groups with similarity scores
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* // Find all duplicates with default threshold (0.85)
|
||||||
|
* const duplicates = await brain.findDuplicates()
|
||||||
|
*
|
||||||
|
* // Find duplicates of a specific type with custom threshold
|
||||||
|
* const personDupes = await brain.findDuplicates({
|
||||||
|
* type: NounType.PERSON,
|
||||||
|
* threshold: 0.9,
|
||||||
|
* limit: 100
|
||||||
|
* })
|
||||||
|
*/
|
||||||
|
async findDuplicates(options?: {
|
||||||
|
threshold?: number
|
||||||
|
type?: NounType
|
||||||
|
limit?: number
|
||||||
|
}): Promise<Array<{
|
||||||
|
entity: Entity<any>
|
||||||
|
duplicates: Array<{ entity: Entity<any>; similarity: number }>
|
||||||
|
}>> {
|
||||||
|
await this.ensureInitialized()
|
||||||
|
|
||||||
|
const threshold = options?.threshold ?? 0.85
|
||||||
|
const limit = options?.limit ?? 100
|
||||||
|
|
||||||
|
// Get entities to check
|
||||||
|
const findParams: FindParams<any> = {
|
||||||
|
limit: Math.min(limit * 10, 1000), // Get more entities to find duplicates within
|
||||||
|
type: options?.type
|
||||||
|
}
|
||||||
|
|
||||||
|
const entities = await this.find(findParams)
|
||||||
|
const results: Array<{
|
||||||
|
entity: Entity<any>
|
||||||
|
duplicates: Array<{ entity: Entity<any>; similarity: number }>
|
||||||
|
}> = []
|
||||||
|
|
||||||
|
const processedIds = new Set<string>()
|
||||||
|
|
||||||
|
for (const result of entities) {
|
||||||
|
if (processedIds.has(result.id)) continue
|
||||||
|
|
||||||
|
// Find similar entities
|
||||||
|
const similar = await this.similar({
|
||||||
|
to: result.id,
|
||||||
|
limit: 20, // Check top 20 similar entities
|
||||||
|
type: options?.type
|
||||||
|
})
|
||||||
|
|
||||||
|
// Filter to those above threshold (excluding self)
|
||||||
|
const duplicates = similar
|
||||||
|
.filter(s => s.id !== result.id && s.score >= threshold)
|
||||||
|
.map(s => ({
|
||||||
|
entity: s.entity,
|
||||||
|
similarity: s.score
|
||||||
|
}))
|
||||||
|
|
||||||
|
if (duplicates.length > 0) {
|
||||||
|
results.push({
|
||||||
|
entity: result.entity,
|
||||||
|
duplicates
|
||||||
|
})
|
||||||
|
|
||||||
|
// Mark all duplicates as processed to avoid reverse matches
|
||||||
|
duplicates.forEach(d => processedIds.add(d.entity.id))
|
||||||
|
}
|
||||||
|
|
||||||
|
processedIds.add(result.id)
|
||||||
|
|
||||||
|
// Stop if we have enough results
|
||||||
|
if (results.length >= limit) break
|
||||||
|
}
|
||||||
|
|
||||||
|
return results
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Cluster entities by semantic similarity
|
||||||
|
*
|
||||||
|
* Groups entities into clusters based on their embedding similarity.
|
||||||
|
* Uses a greedy algorithm that finds densely connected components
|
||||||
|
* using the HNSW index for efficient neighbor lookup.
|
||||||
|
*
|
||||||
|
* @param options Optional clustering options
|
||||||
|
* @returns Array of clusters with entities and optional centroids
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* // Find all clusters with default threshold
|
||||||
|
* const clusters = await brain.cluster()
|
||||||
|
*
|
||||||
|
* // Find document clusters with higher threshold
|
||||||
|
* const docClusters = await brain.cluster({
|
||||||
|
* type: NounType.Document,
|
||||||
|
* threshold: 0.85,
|
||||||
|
* minClusterSize: 3
|
||||||
|
* })
|
||||||
|
*
|
||||||
|
* for (const cluster of docClusters) {
|
||||||
|
* console.log(`Cluster ${cluster.clusterId}: ${cluster.entities.length} entities`)
|
||||||
|
* }
|
||||||
|
*/
|
||||||
|
async cluster(options?: {
|
||||||
|
threshold?: number
|
||||||
|
type?: NounType
|
||||||
|
minClusterSize?: number
|
||||||
|
limit?: number
|
||||||
|
includeCentroid?: boolean
|
||||||
|
}): Promise<Array<{
|
||||||
|
clusterId: string
|
||||||
|
entities: Entity<any>[]
|
||||||
|
centroid?: number[]
|
||||||
|
}>> {
|
||||||
|
await this.ensureInitialized()
|
||||||
|
|
||||||
|
const threshold = options?.threshold ?? 0.8
|
||||||
|
const minClusterSize = options?.minClusterSize ?? 2
|
||||||
|
const limit = options?.limit ?? 100
|
||||||
|
const includeCentroid = options?.includeCentroid ?? false
|
||||||
|
|
||||||
|
// Get entities to cluster
|
||||||
|
const findParams: FindParams<any> = {
|
||||||
|
limit: 1000, // Process up to 1000 entities
|
||||||
|
type: options?.type
|
||||||
|
}
|
||||||
|
|
||||||
|
const allEntities = await this.find(findParams)
|
||||||
|
const clustered = new Set<string>()
|
||||||
|
const clusters: Array<{
|
||||||
|
clusterId: string
|
||||||
|
entities: Entity<any>[]
|
||||||
|
centroid?: number[]
|
||||||
|
}> = []
|
||||||
|
|
||||||
|
// Greedy clustering: for each unclustered entity, find its similar neighbors
|
||||||
|
for (const result of allEntities) {
|
||||||
|
if (clustered.has(result.id)) continue
|
||||||
|
|
||||||
|
// Find similar entities to this one
|
||||||
|
const similar = await this.similar({
|
||||||
|
to: result.id,
|
||||||
|
limit: 50,
|
||||||
|
threshold,
|
||||||
|
type: options?.type
|
||||||
|
})
|
||||||
|
|
||||||
|
// Filter to unclustered entities (including self)
|
||||||
|
const clusterMembers = similar.filter(s => !clustered.has(s.id))
|
||||||
|
|
||||||
|
// Only create cluster if it meets minimum size
|
||||||
|
if (clusterMembers.length >= minClusterSize) {
|
||||||
|
const entities = clusterMembers.map(s => s.entity)
|
||||||
|
|
||||||
|
// Mark all as clustered
|
||||||
|
clusterMembers.forEach(s => clustered.add(s.id))
|
||||||
|
|
||||||
|
// Calculate centroid if requested
|
||||||
|
let centroid: number[] | undefined
|
||||||
|
if (includeCentroid && entities.length > 0) {
|
||||||
|
const vectors = entities
|
||||||
|
.filter(e => e.vector && e.vector.length > 0)
|
||||||
|
.map(e => e.vector as number[])
|
||||||
|
|
||||||
|
if (vectors.length > 0) {
|
||||||
|
// Average all vectors to get centroid
|
||||||
|
const dim = vectors[0].length
|
||||||
|
centroid = new Array(dim).fill(0)
|
||||||
|
for (const vec of vectors) {
|
||||||
|
for (let i = 0; i < dim; i++) {
|
||||||
|
centroid[i] += vec[i]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (let i = 0; i < dim; i++) {
|
||||||
|
centroid[i] /= vectors.length
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
clusters.push({
|
||||||
|
clusterId: `cluster-${clusters.length + 1}`,
|
||||||
|
entities,
|
||||||
|
centroid
|
||||||
|
})
|
||||||
|
|
||||||
|
if (clusters.length >= limit) break
|
||||||
|
} else {
|
||||||
|
// Mark single entity as processed (not in a cluster)
|
||||||
|
clustered.add(result.id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return clusters
|
||||||
|
}
|
||||||
|
|
||||||
// ============= INTERNAL VERSIONING API =============
|
// ============= INTERNAL VERSIONING API =============
|
||||||
// These methods are used by the versioning system (brain.versions.*)
|
// These methods are used by the versioning system (brain.versions.*)
|
||||||
// They expose internal storage and index operations needed for entity versioning
|
// They expose internal storage and index operations needed for entity versioning
|
||||||
|
|
|
||||||
|
|
@ -9,6 +9,7 @@
|
||||||
|
|
||||||
import { Vector } from '../coreTypes.js'
|
import { Vector } from '../coreTypes.js'
|
||||||
import { AddParams, RelateParams, Result, Entity, FindParams, SimilarParams } from './brainy.types.js'
|
import { AddParams, RelateParams, Result, Entity, FindParams, SimilarParams } from './brainy.types.js'
|
||||||
|
import { NounType, VerbType } from './graphTypes.js'
|
||||||
|
|
||||||
export interface BrainyInterface<T = unknown> {
|
export interface BrainyInterface<T = unknown> {
|
||||||
/**
|
/**
|
||||||
|
|
@ -53,8 +54,87 @@ export interface BrainyInterface<T = unknown> {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Generate embedding vector from text
|
* Generate embedding vector from text
|
||||||
* @param text The text to embed
|
* @param data The data to embed (text, array, or object)
|
||||||
* @returns Vector representation of the text
|
* @returns Vector representation of the data
|
||||||
*/
|
*/
|
||||||
embed(text: string): Promise<Vector>
|
embed(data: any): Promise<Vector>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Batch embed multiple texts at once
|
||||||
|
* @param texts Array of texts to embed
|
||||||
|
* @returns Array of embedding vectors (384 dimensions each)
|
||||||
|
*/
|
||||||
|
embedBatch(texts: string[]): Promise<number[][]>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Calculate semantic similarity between two texts
|
||||||
|
* @param textA First text
|
||||||
|
* @param textB Second text
|
||||||
|
* @returns Similarity score between 0 and 1
|
||||||
|
*/
|
||||||
|
similarity(textA: string, textB: string): Promise<number>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get comprehensive index statistics
|
||||||
|
* @returns Index statistics object
|
||||||
|
*/
|
||||||
|
indexStats(): Promise<{
|
||||||
|
entities: number
|
||||||
|
vectors: number
|
||||||
|
relationships: number
|
||||||
|
metadataFields: string[]
|
||||||
|
memoryUsage: {
|
||||||
|
vectors: number
|
||||||
|
graph: number
|
||||||
|
metadata: number
|
||||||
|
total: number
|
||||||
|
}
|
||||||
|
}>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get graph neighbors of an entity
|
||||||
|
* @param entityId The entity to get neighbors for
|
||||||
|
* @param options Optional traversal options
|
||||||
|
* @returns Array of neighbor entity IDs
|
||||||
|
*/
|
||||||
|
neighbors(
|
||||||
|
entityId: string,
|
||||||
|
options?: {
|
||||||
|
direction?: 'outgoing' | 'incoming' | 'both'
|
||||||
|
depth?: number
|
||||||
|
verbType?: VerbType
|
||||||
|
limit?: number
|
||||||
|
}
|
||||||
|
): Promise<string[]>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Find semantic duplicates in the database
|
||||||
|
* @param options Optional search options
|
||||||
|
* @returns Array of duplicate groups with similarity scores
|
||||||
|
*/
|
||||||
|
findDuplicates(options?: {
|
||||||
|
threshold?: number
|
||||||
|
type?: NounType
|
||||||
|
limit?: number
|
||||||
|
}): Promise<Array<{
|
||||||
|
entity: Entity<any>
|
||||||
|
duplicates: Array<{ entity: Entity<any>; similarity: number }>
|
||||||
|
}>>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Cluster entities by semantic similarity
|
||||||
|
* @param options Optional clustering options
|
||||||
|
* @returns Array of clusters with entities and optional centroids
|
||||||
|
*/
|
||||||
|
cluster(options?: {
|
||||||
|
threshold?: number
|
||||||
|
type?: NounType
|
||||||
|
minClusterSize?: number
|
||||||
|
limit?: number
|
||||||
|
includeCentroid?: boolean
|
||||||
|
}): Promise<Array<{
|
||||||
|
clusterId: string
|
||||||
|
entities: Entity<any>[]
|
||||||
|
centroid?: number[]
|
||||||
|
}>>
|
||||||
}
|
}
|
||||||
Loading…
Add table
Add a link
Reference in a new issue