soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat: introduce CLI for Brainy and enhance type validation

Browse source 
Added a comprehensive command-line interface (CLI) for interacting with the Brainy vector database. The CLI supports various operations, including database initialization, adding/searching nouns, managing relationships, and querying database status. Enhanced type validation logic for nouns and verbs to ensure consistency and enforce default types for invalid inputs. Updated test scripts to verify type validation and edge cases.
This commit is contained in:
David Snelling 2025-06-05 11:18:20 -07:00
parent a0ce5b0ca9
commit cdbd2a9db4
14 changed files with 1196 additions and 373 deletions
Download patch file Download diff file Expand all files Collapse all files

192
README.md
View file

@ -25,6 +25,20 @@ Brainy combines three key technologies:
- Server: S3-compatible storage (optional) - Server: S3-compatible storage (optional)
- Fallback: In-memory storage - Fallback: In-memory storage
## Data Model
Brainy uses a graph-based data model with two primary concepts:
1. **Nouns**: The main entities in your data (nodes in the graph)
- Each noun has a unique ID, vector representation, and metadata
- Nouns can be categorized by type (Person, Place, Thing, Event, Concept, etc.)
- Nouns are automatically vectorized for similarity search
2. **Verbs**: Relationships between nouns (edges in the graph)
- Each verb connects a source noun to a target noun
- Verbs have types that define the relationship (RelatedTo, Controls, Contains, etc.)
- Verbs can have their own metadata to describe the relationship
## What Can You Use It For? ## What Can You Use It For?
- **Semantic Search**: Find content based on meaning, not just keywords - **Semantic Search**: Find content based on meaning, not just keywords
@ -39,35 +53,103 @@ Brainy combines three key technologies:
npm install @soulcraft/brainy npm install @soulcraft/brainy
``` ```
## Command Line Interface
Brainy includes a command-line interface (CLI) that allows you to experiment with the API and data directly from the terminal.
### Using the CLI during development
```bash
# Run the CLI directly from the source
npm run cli help
# Initialize a database
npm run cli init
# Add some data
npm run cli add "Cats are independent pets" '{"noun":"Thing","category":"animal"}'
npm run cli add "Dogs are loyal companions" '{"noun":"Thing","category":"animal"}'
# Search for similar items
npm run cli search "feline pets" 5
# Add relationships between items
npm run cli addVerb <sourceId> <targetId> RelatedTo '{"description":"Both are pets"}'
# View database status
npm run cli status
```
### Installing the CLI globally
```bash
# Install the package globally
npm install -g @soulcraft/brainy
# Now you can use the 'brainy' command directly
brainy help
brainy init
brainy add "Some text" '{"noun":"Thing"}'
```
### Available Commands
- `init` - Initialize a new database
- `add <text> [metadata]` - Add a new noun with the given text and optional metadata
- `search <query> [limit]` - Search for nouns similar to the query
- `get <id>` - Get a noun by ID
- `delete <id>` - Delete a noun by ID
- `addVerb <sourceId> <targetId> <verbType> [metadata]` - Add a relationship between nouns
- `getVerbs <id>` - Get all relationships for a noun
- `status` - Show database status
## Basic Usage ## Basic Usage
```typescript ```typescript
import {BrainyData} from '@soulcraft/brainy'; import {BrainyData, NounType, VerbType} from '@soulcraft/brainy';
// Create and initialize the database // Create and initialize the database
const db = new BrainyData(); const db = new BrainyData();
await db.init(); await db.init();
// Add text data (automatically converted to vectors) // Add nouns (automatically converted to vectors)
const catId = await db.add("Cats are independent pets", {type: 'animal'}); const catId = await db.add("Cats are independent pets", {
const dogId = await db.add("Dogs are loyal companions", {type: 'animal'}); noun: NounType.Thing,
category: 'animal'
});
// Search for similar content const dogId = await db.add("Dogs are loyal companions", {
noun: NounType.Thing,
category: 'animal'
});
// Search for similar nouns
const results = await db.searchText("feline pets", 2); const results = await db.searchText("feline pets", 2);
console.log(results); console.log(results);
// Returns items similar to "feline pets" with their similarity scores // Returns nouns similar to "feline pets" with their similarity scores
// Add relationships between items // Add a verb (relationship) between nouns
await db.addEdge(catId, dogId, {type: 'related_to'}); await db.addVerb(catId, dogId, {
verb: VerbType.RelatedTo,
description: 'Both are common household pets'
});
// Retrieve data // Retrieve a noun
const cat = await db.get(catId); const cat = await db.get(catId);
console.log(cat); console.log(cat);
// Update metadata // Get all relationships for a noun
await db.updateMetadata(catId, {type: 'animal', size: 'small'}); const catRelationships = await db.getVerbsBySource(catId);
console.log(catRelationships);
// Delete data // Update noun metadata
await db.updateMetadata(catId, {
noun: NounType.Thing,
category: 'animal',
size: 'small'
});
// Delete a noun
await db.delete(dogId); await db.delete(dogId);
``` ```
@ -75,9 +157,10 @@ await db.delete(dogId);
- **Automatic Vectorization**: Converts text and data to vector embeddings - **Automatic Vectorization**: Converts text and data to vector embeddings
- **Fast Similarity Search**: Uses HNSW algorithm for efficient retrieval - **Fast Similarity Search**: Uses HNSW algorithm for efficient retrieval
- **Noun-Based Data Model**: Organize entities with typed categorization (Person, Place, Thing, etc.)
- **Verb-Based Relationships**: Create meaningful connections between nouns with typed relationships
- **Cross-Platform**: Works in browsers, Node.js, and server environments - **Cross-Platform**: Works in browsers, Node.js, and server environments
- **Persistent Storage**: Multiple storage options (OPFS, filesystem, cloud) - **Persistent Storage**: Multiple storage options (OPFS, filesystem, cloud)
- **Graph Capabilities**: Create relationships between data points
- **TypeScript Support**: Fully typed API with generics - **TypeScript Support**: Fully typed API with generics
- **Flexible Configuration**: Customize distance functions, embedding models, and more - **Flexible Configuration**: Customize distance functions, embedding models, and more
@ -110,11 +193,17 @@ const db = new BrainyData({
// HNSW index configuration for search performance // HNSW index configuration for search performance
hnsw: { hnsw: {
M: 16, // Max connections per node M: 16, // Max connections per noun
efConstruction: 200, // Construction candidate list size efConstruction: 200, // Construction candidate list size
efSearch: 50, // Search candidate list size efSearch: 50, // Search candidate list size
}, },
// Noun and Verb type validation
typeValidation: {
enforceNounTypes: true, // Validate noun types against NounType enum
enforceVerbTypes: true, // Validate verb types against VerbType enum
},
// Storage configuration // Storage configuration
storage: { storage: {
requestPersistentStorage: true, requestPersistentStorage: true,
@ -131,34 +220,75 @@ const db = new BrainyData({
## API Reference ## API Reference
### Core Methods ### Database Methods
```typescript ```typescript
// Initialize the database // Initialize the database
await db.init(); await db.init();
// Add data (automatically vectorized)
const id = await db.add(textOrVector, metadata);
// Search by vector or text
const results = await db.search(vectorOrText, numResults);
const textResults = await db.searchText("query text", numResults);
// Manage data
const item = await db.get(id);
await db.updateMetadata(id, newMetadata);
await db.delete(id);
// Graph relationships
await db.addEdge(sourceId, targetId, metadata);
const edges = await db.getAllEdges();
// Database management // Database management
await db.clear(); await db.clear();
const size = db.size(); const size = db.size();
const status = await db.status(); const status = await db.status();
``` ```
### Noun Methods (Entities)
```typescript
// Add a noun (automatically vectorized)
const id = await db.add(textOrVector, {
noun: NounType.Thing,
// other metadata...
});
// Retrieve a noun
const noun = await db.get(id);
// Update noun metadata
await db.updateMetadata(id, {
noun: NounType.Thing,
// updated metadata...
});
// Delete a noun
await db.delete(id);
// Search for similar nouns
const results = await db.search(vectorOrText, numResults);
const textResults = await db.searchText("query text", numResults);
// Search by noun type
const thingNouns = await db.searchByNounTypes([NounType.Thing], numResults);
```
### Verb Methods (Relationships)
```typescript
// Add a relationship between nouns
await db.addVerb(sourceId, targetId, {
verb: VerbType.RelatedTo,
// other metadata...
});
// Get all relationships
const verbs = await db.getAllVerbs();
// Get relationships by source noun
const outgoingVerbs = await db.getVerbsBySource(sourceId);
// Get relationships by target noun
const incomingVerbs = await db.getVerbsByTarget(targetId);
// Get relationships by type
const containsVerbs = await db.getVerbsByType(VerbType.Contains);
// Get a specific relationship
const verb = await db.getVerb(verbId);
// Delete a relationship
await db.deleteVerb(verbId);
```
### Distance Functions ### Distance Functions
- `cosineDistance` (default) - `cosineDistance` (default)

145
examples/dataInspectionExample.js Normal file
View file

@ -0,0 +1,145 @@
/**
* Data Inspection Example
*
* This example demonstrates how to view and inspect the data stored in Brainy
* to verify it's working correctly.
*/
import { BrainyData, createSimpleEmbeddingFunction } from '../dist/index.js';
async function runExample() {
try {
console.log('Brainy Data Inspection Example');
console.log('==============================\n');
// Create a new Brainy database with a simple embedding function
console.log('Creating and initializing Brainy database...');
const simpleEmbedding = createSimpleEmbeddingFunction();
const db = new BrainyData({
embeddingFunction: simpleEmbedding
});
await db.init();
console.log('Database initialized successfully!\n');
// Add sample data - using text that will be automatically embedded to vectors
console.log('Adding sample data to the database...');
const catId = await db.add("Cat is a small domesticated carnivorous mammal", { type: 'mammal', name: 'cat' });
const dogId = await db.add("Dog is a domesticated carnivore of the family Canidae", { type: 'mammal', name: 'dog' });
const fishId = await db.add("Fish are aquatic animals that live in water", { type: 'fish', name: 'fish' });
// Add more text data
const lionDescId = await db.add("Lions are large cats with a golden mane", { type: 'mammal', name: 'lion' });
const tigerDescId = await db.add("Tigers are large cats with striped fur", { type: 'mammal', name: 'tiger' });
// Add an edge between cat and lion (they're related)
const edgeId = await db.addEdge(catId, lionDescId, undefined, {
type: 'related',
weight: 0.8,
metadata: { relationship: 'same family' }
});
console.log('Sample data added successfully!\n');
// Method 1: Check database status
console.log('Method 1: Check Database Status');
console.log('-------------------------------');
const status = await db.status();
console.log('Storage Type:', status.type);
console.log('Used Space:', status.used, 'bytes');
console.log('Storage Quota:', status.quota, 'bytes');
console.log('Number of Items:', db.size());
console.log('Additional Details:', JSON.stringify(status.details, null, 2));
console.log();
// Method 2: Retrieve specific items by ID
console.log('Method 2: Retrieve Specific Items by ID');
console.log('--------------------------------------');
const cat = await db.get(catId);
console.log('Cat Item:');
console.log('- ID:', cat.id);
console.log('- Vector:', cat.vector);
console.log('- Metadata:', JSON.stringify(cat.metadata, null, 2));
console.log();
// Method 3: Search for similar items
console.log('Method 3: Search for Similar Items');
console.log('----------------------------------');
const searchResults = await db.search("cat", 3);
console.log('Search Results:');
searchResults.forEach((result, index) => {
console.log(`Result ${index + 1}:`);
console.log('- ID:', result.id);
console.log('- Score:', result.score);
console.log('- Metadata:', JSON.stringify(result.metadata, null, 2));
});
console.log();
// Method 4: Text search
console.log('Method 4: Text Search');
console.log('--------------------');
const textResults = await db.searchText('cat', 2);
console.log('Text Search Results:');
textResults.forEach((result, index) => {
console.log(`Result ${index + 1}:`);
console.log('- ID:', result.id);
console.log('- Score:', result.score);
console.log('- Metadata:', JSON.stringify(result.metadata, null, 2));
});
console.log();
// Method 5: Get all edges
console.log('Method 5: Get All Edges');
console.log('----------------------');
const allEdges = await db.getAllEdges();
console.log('All Edges:');
allEdges.forEach((edge, index) => {
console.log(`Edge ${index + 1}:`);
console.log('- ID:', edge.id);
console.log('- Source ID:', edge.sourceId);
console.log('- Target ID:', edge.targetId);
console.log('- Type:', edge.type);
console.log('- Weight:', edge.weight);
console.log('- Metadata:', JSON.stringify(edge.metadata, null, 2));
});
console.log();
// Method 6: Get edges by source
console.log('Method 6: Get Edges by Source');
console.log('----------------------------');
const catEdges = await db.getEdgesBySource(catId);
console.log(`Edges from Cat (${catId}):`);
catEdges.forEach((edge, index) => {
console.log(`Edge ${index + 1}:`);
console.log('- ID:', edge.id);
console.log('- Target ID:', edge.targetId);
console.log('- Type:', edge.type);
});
console.log();
// Method 7: Advanced search with noun types and verbs
console.log('Method 7: Advanced Search with Noun Types and Verbs');
console.log('--------------------------------------------------');
const advancedResults = await db.search("cat", 3, {
nounTypes: ['mammal'],
includeVerbs: true
});
console.log('Advanced Search Results:');
advancedResults.forEach((result, index) => {
console.log(`Result ${index + 1}:`);
console.log('- ID:', result.id);
console.log('- Score:', result.score);
console.log('- Metadata:', JSON.stringify(result.metadata, null, 2));
if (result.metadata && result.metadata.associatedVerbs) {
console.log('- Associated Verbs:', result.metadata.associatedVerbs.length);
}
});
console.log();
console.log('Example completed successfully!');
} catch (error) {
console.error('Error running example:', error);
}
}
// Run the example
runExample();

23
package-lock.json generated
View file

@ -15,8 +15,12 @@
"@tensorflow/tfjs-backend-cpu": "^4.22.0", "@tensorflow/tfjs-backend-cpu": "^4.22.0",
"@tensorflow/tfjs-core": "^4.22.0", "@tensorflow/tfjs-core": "^4.22.0",
"@tensorflow/tfjs-layers": "^4.22.0", "@tensorflow/tfjs-layers": "^4.22.0",
"commander": "^14.0.0",
"uuid": "^9.0.0" "uuid": "^9.0.0"
}, },
"bin": {
"brainy": "dist/cli.js"
},
"devDependencies": { "devDependencies": {
"@types/jest": "^29.5.3", "@types/jest": "^29.5.3",
"@types/node": "^20.4.5", "@types/node": "^20.4.5",
@ -2868,16 +2872,6 @@
"@tensorflow/tfjs-core": "4.22.0" "@tensorflow/tfjs-core": "4.22.0"
} }
}, },
"node_modules/@tensorflow/tfjs-converter": {
"version": "3.21.0",
"resolved": "https://registry.npmjs.org/@tensorflow/tfjs-converter/-/tfjs-converter-3.21.0.tgz",
"integrity": "sha512-12Y4zVDq3yW+wSjSDpSv4HnpL2sDZrNiGSg8XNiDE4HQBdjdA+a+Q3sZF/8NV9y2yoBhL5L7V4mMLDdbZBd9/Q==",
"license": "Apache-2.0",
"peer": true,
"peerDependencies": {
"@tensorflow/tfjs-core": "3.21.0"
}
},
"node_modules/@tensorflow/tfjs-core": { "node_modules/@tensorflow/tfjs-core": {
"version": "4.22.0", "version": "4.22.0",
"resolved": "https://registry.npmjs.org/@tensorflow/tfjs-core/-/tfjs-core-4.22.0.tgz", "resolved": "https://registry.npmjs.org/@tensorflow/tfjs-core/-/tfjs-core-4.22.0.tgz",
@ -3847,6 +3841,15 @@
"node": ">= 0.8" "node": ">= 0.8"
} }
}, },
"node_modules/commander": {
"version": "14.0.0",
"resolved": "https://registry.npmjs.org/commander/-/commander-14.0.0.tgz",
"integrity": "sha512-2uM9rYjPvyq39NwLRqaiLtWHyDC1FvryJDa2ATTVims5YAS4PupsEQsDvP14FqhFr0P49CYDugi59xaxJlTXRA==",
"license": "MIT",
"engines": {
"node": ">=20"
}
},
"node_modules/concat-map": { "node_modules/concat-map": {
"version": "0.0.1", "version": "0.0.1",
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz", "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",

9
package.json
View file

@ -28,11 +28,15 @@
"build": "tsc", "build": "tsc",
"test": "jest", "test": "jest",
"start": "node dist/index.js", "start": "node dist/index.js",
"cli": "node dist/cli.js",
"version:patch": "npm version patch", "version:patch": "npm version patch",
"version:minor": "npm version minor", "version:minor": "npm version minor",
"version:major": "npm version major", "version:major": "npm version major",
"deploy": "npm run build && npm publish" "deploy": "npm run build && npm publish"
}, },
"bin": {
"brainy": "dist/cli.js"
},
"keywords": [ "keywords": [
"vector-database", "vector-database",
"hnsw", "hnsw",
@ -60,13 +64,14 @@
"typescript": "^5.1.6" "typescript": "^5.1.6"
}, },
"dependencies": { "dependencies": {
"uuid": "^9.0.0", "@aws-sdk/client-s3": "^3.427.0",
"@tensorflow-models/universal-sentence-encoder": "^1.3.3", "@tensorflow-models/universal-sentence-encoder": "^1.3.3",
"@tensorflow/tfjs": "^4.22.0", "@tensorflow/tfjs": "^4.22.0",
"@tensorflow/tfjs-backend-cpu": "^4.22.0", "@tensorflow/tfjs-backend-cpu": "^4.22.0",
"@tensorflow/tfjs-core": "^4.22.0", "@tensorflow/tfjs-core": "^4.22.0",
"@tensorflow/tfjs-layers": "^4.22.0", "@tensorflow/tfjs-layers": "^4.22.0",
"@aws-sdk/client-s3": "^3.427.0" "commander": "^14.0.0",
"uuid": "^9.0.0"
}, },
"prettier": { "prettier": {
"arrowParens": "always", "arrowParens": "always",

2
src/augmentations/memoryAugmentations.ts
View file

@ -176,7 +176,7 @@ abstract class BaseMemoryAugmentation implements IMemoryAugmentation {
} }
// Get all nodes from storage // Get all nodes from storage
const nodes = await this.storage.getAllNodes() const nodes = await this.storage.getAllNouns()
// Calculate distances and prepare results // Calculate distances and prepare results
const results: Array<{ const results: Array<{

258
src/brainyData.ts
View file

@ -8,15 +8,16 @@ import { HNSWIndex } from './hnsw/hnswIndex.js'
import { createStorage } from './storage/opfsStorage.js' import { createStorage } from './storage/opfsStorage.js'
import { import {
DistanceFunction, DistanceFunction,
Edge, GraphVerb,
EmbeddingFunction, EmbeddingFunction,
HNSWConfig, HNSWNode, HNSWConfig, HNSWNoun,
SearchResult, SearchResult,
StorageAdapter, StorageAdapter,
Vector, Vector,
VectorDocument VectorDocument
} from './coreTypes.js' } from './coreTypes.js'
import { cosineDistance, defaultEmbeddingFunction, euclideanDistance } from './utils/index.js' import { cosineDistance, defaultEmbeddingFunction, euclideanDistance } from './utils/index.js'
import { NounType, VerbType, GraphNoun } from './types/graphTypes.js'
export interface BrainyDataConfig { export interface BrainyDataConfig {
/** /**
@ -160,16 +161,16 @@ export class BrainyData<T = any> {
// Initialize storage // Initialize storage
await this.storage!.init() await this.storage!.init()
// Load all nodes from storage // Load all nouns from storage
const nodes: HNSWNode[] = await this.storage!.getAllNodes() const nouns: HNSWNoun[] = await this.storage!.getAllNouns()
// Clear the index and add all nodes // Clear the index and add all nouns
this.index.clear() this.index.clear()
for (const node of nodes) { for (const noun of nouns) {
// Add to index // Add to index
this.index.addItem({ this.index.addItem({
id: node.id, id: noun.id,
vector: node.vector vector: noun.vector
}) })
} }
@ -231,18 +232,32 @@ export class BrainyData<T = any> {
// Add to index // Add to index
this.index.addItem({ id, vector }) this.index.addItem({ id, vector })
// Get the node from the index // Get the noun from the index
const node = this.index.getNodes().get(id) const noun = this.index.getNodes().get(id)
if (!node) { if (!noun) {
throw new Error(`Failed to retrieve newly created node with ID ${id}`) throw new Error(`Failed to retrieve newly created noun with ID ${id}`)
} }
// Save node to storage // Save noun to storage
await this.storage!.saveNode(node) await this.storage!.saveNoun(noun)
// Save metadata if provided // Save metadata if provided
if (metadata !== undefined) { if (metadata !== undefined) {
// Validate noun type if metadata is for a GraphNoun
if (metadata && typeof metadata === 'object' && 'noun' in metadata) {
const nounType = (metadata as any).noun;
// Check if the noun type is valid
const isValidNounType = Object.values(NounType).includes(nounType);
if (!isValidNounType) {
console.warn(`Invalid noun type: ${nounType}. Falling back to GraphNoun.`);
// Set a default noun type
(metadata as any).noun = NounType.Concept;
}
}
await this.storage!.saveMetadata(id, metadata) await this.storage!.saveMetadata(id, metadata)
} }
@ -331,7 +346,7 @@ export class BrainyData<T = any> {
throw new Error('Query vector is undefined or null') throw new Error('Query vector is undefined or null')
} }
// If no noun types specified, search all nodes // If no noun types specified, search all nouns
if (!nounTypes || nounTypes.length === 0) { if (!nounTypes || nounTypes.length === 0) {
// Search in the index // Search in the index
const results = this.index.search(queryVector, k) const results = this.index.search(queryVector, k)
@ -340,8 +355,8 @@ export class BrainyData<T = any> {
const searchResults: SearchResult<T>[] = [] const searchResults: SearchResult<T>[] = []
for (const [id, score] of results) { for (const [id, score] of results) {
const node = this.index.getNodes().get(id) const noun = this.index.getNodes().get(id)
if (!node) { if (!noun) {
continue continue
} }
@ -350,28 +365,28 @@ export class BrainyData<T = any> {
searchResults.push({ searchResults.push({
id, id,
score, score,
vector: node.vector, vector: noun.vector,
metadata metadata
}) })
} }
return searchResults return searchResults
} else { } else {
// Get nodes for each noun type in parallel // Get nouns for each noun type in parallel
const nodePromises = nounTypes.map(nounType => this.storage!.getNodesByNounType(nounType)) const nounPromises = nounTypes.map(nounType => this.storage!.getNounsByNounType(nounType))
const nodeArrays = await Promise.all(nodePromises) const nounArrays = await Promise.all(nounPromises)
// Combine all nodes // Combine all nouns
const nodes: HNSWNode[] = [] const nouns: HNSWNoun[] = []
for (const nodeArray of nodeArrays) { for (const nounArray of nounArrays) {
nodes.push(...nodeArray) nouns.push(...nounArray)
} }
// Calculate distances for each node // Calculate distances for each noun
const results: Array<[string, number]> = [] const results: Array<[string, number]> = []
for (const node of nodes) { for (const noun of nouns) {
const distance = this.index.getDistanceFunction()(queryVector, node.vector) const distance = this.index.getDistanceFunction()(queryVector, noun.vector)
results.push([node.id, distance]) results.push([noun.id, distance])
} }
// Sort by distance (ascending) // Sort by distance (ascending)
@ -384,8 +399,8 @@ export class BrainyData<T = any> {
const searchResults: SearchResult<T>[] = [] const searchResults: SearchResult<T>[] = []
for (const [id, score] of topResults) { for (const [id, score] of topResults) {
const node = nodes.find(n => n.id === id) const noun = nouns.find(n => n.id === id)
if (!node) { if (!noun) {
continue continue
} }
@ -394,7 +409,7 @@ export class BrainyData<T = any> {
searchResults.push({ searchResults.push({
id, id,
score, score,
vector: node.vector, vector: noun.vector,
metadata metadata
}) })
} }
@ -447,22 +462,22 @@ export class BrainyData<T = any> {
if (options.includeVerbs && this.storage) { if (options.includeVerbs && this.storage) {
for (const result of searchResults) { for (const result of searchResults) {
try { try {
// Get outgoing edges (verbs) for this noun // Get outgoing verbs for this noun
const outgoingEdges = await this.storage.getEdgesBySource(result.id); const outgoingVerbs = await this.storage.getVerbsBySource(result.id);
// Get incoming edges (verbs) for this noun // Get incoming verbs for this noun
const incomingEdges = await this.storage.getEdgesByTarget(result.id); const incomingVerbs = await this.storage.getVerbsByTarget(result.id);
// Combine all edges // Combine all verbs
const allEdges = [...outgoingEdges, ...incomingEdges]; const allVerbs = [...outgoingVerbs, ...incomingVerbs];
// Add edges to the result metadata // Add verbs to the result metadata
if (!result.metadata) { if (!result.metadata) {
result.metadata = {} as T; result.metadata = {} as T;
} }
// Add the edges to the metadata // Add the verbs to the metadata
(result.metadata as any).associatedVerbs = allEdges; (result.metadata as any).associatedVerbs = allVerbs;
} catch (error) { } catch (error) {
console.warn(`Failed to retrieve verbs for noun ${result.id}:`, error); console.warn(`Failed to retrieve verbs for noun ${result.id}:`, error);
} }
@ -479,9 +494,9 @@ export class BrainyData<T = any> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
// Get node from index // Get noun from index
const node = this.index.getNodes().get(id) const noun = this.index.getNodes().get(id)
if (!node) { if (!noun) {
return null return null
} }
@ -490,7 +505,7 @@ export class BrainyData<T = any> {
return { return {
id, id,
vector: node.vector, vector: noun.vector,
metadata metadata
} }
} catch (error) { } catch (error) {
@ -516,7 +531,7 @@ export class BrainyData<T = any> {
} }
// Remove from storage // Remove from storage
await this.storage!.deleteNode(id) await this.storage!.deleteNoun(id)
// Try to remove metadata (ignore errors) // Try to remove metadata (ignore errors)
try { try {
@ -543,11 +558,25 @@ export class BrainyData<T = any> {
try { try {
// Check if a vector exists // Check if a vector exists
const node = this.index.getNodes().get(id) const noun = this.index.getNodes().get(id)
if (!node) { if (!noun) {
return false return false
} }
// Validate noun type if metadata is for a GraphNoun
if (metadata && typeof metadata === 'object' && 'noun' in metadata) {
const nounType = (metadata as any).noun;
// Check if the noun type is valid
const isValidNounType = Object.values(NounType).includes(nounType);
if (!isValidNounType) {
console.warn(`Invalid noun type: ${nounType}. Falling back to GraphNoun.`);
// Set a default noun type
(metadata as any).noun = NounType.Concept;
}
}
// Update metadata // Update metadata
await this.storage!.saveMetadata(id, metadata) await this.storage!.saveMetadata(id, metadata)
@ -559,10 +588,10 @@ export class BrainyData<T = any> {
} }
/** /**
* Add an edge between two nodes * Add a verb between two nouns
* If metadata is provided and vector is not, the metadata will be vectorized using the embedding function * If metadata is provided and vector is not, the metadata will be vectorized using the embedding function
*/ */
public async addEdge( public async addVerb(
sourceId: string, sourceId: string,
targetId: string, targetId: string,
vector?: Vector, vector?: Vector,
@ -579,148 +608,161 @@ export class BrainyData<T = any> {
this.checkReadOnly() this.checkReadOnly()
try { try {
// Check if source and target nodes exist // Check if source and target nouns exist
const sourceNode = this.index.getNodes().get(sourceId) const sourceNoun = this.index.getNodes().get(sourceId)
const targetNode = this.index.getNodes().get(targetId) const targetNoun = this.index.getNodes().get(targetId)
if (!sourceNode) { if (!sourceNoun) {
throw new Error(`Source node with ID ${sourceId} not found`) throw new Error(`Source noun with ID ${sourceId} not found`)
} }
if (!targetNode) { if (!targetNoun) {
throw new Error(`Target node with ID ${targetId} not found`) throw new Error(`Target noun with ID ${targetId} not found`)
} }
// Generate ID for the edge // Generate ID for the verb
const id = uuidv4() const id = uuidv4()
let edgeVector: Vector let verbVector: Vector
// If metadata is provided and no vector is provided or forceEmbed is true, vectorize the metadata // If metadata is provided and no vector is provided or forceEmbed is true, vectorize the metadata
if (options.metadata && (!vector || options.forceEmbed)) { if (options.metadata && (!vector || options.forceEmbed)) {
try { try {
edgeVector = await this.embeddingFunction(options.metadata) verbVector = await this.embeddingFunction(options.metadata)
} catch (embedError) { } catch (embedError) {
throw new Error(`Failed to vectorize edge metadata: ${embedError}`) throw new Error(`Failed to vectorize verb metadata: ${embedError}`)
} }
} else { } else {
// Use a provided vector or average of source and target vectors // Use a provided vector or average of source and target vectors
edgeVector = verbVector =
vector || vector ||
sourceNode.vector.map((val, i) => (val + targetNode.vector[i]) / 2) sourceNoun.vector.map((val, i) => (val + targetNoun.vector[i]) / 2)
} }
// Create edge // Validate verb type if provided
const edge: Edge = { let verbType = options.type;
if (verbType) {
// Check if the verb type is valid
const isValidVerbType = Object.values(VerbType).includes(verbType as any);
if (!isValidVerbType) {
console.warn(`Invalid verb type: ${verbType}. Using RelatedTo as default.`);
// Set a default verb type
verbType = VerbType.RelatedTo;
}
}
// Create verb
const verb: GraphVerb = {
id, id,
vector: edgeVector, vector: verbVector,
connections: new Map(), connections: new Map(),
sourceId, sourceId,
targetId, targetId,
type: options.type, type: verbType,
weight: options.weight, weight: options.weight,
metadata: options.metadata metadata: options.metadata
} }
// Add to index // Add to index
this.index.addItem({ id, vector: edgeVector }) this.index.addItem({ id, vector: verbVector })
// Get the node from the index // Get the noun from the index
const indexNode = this.index.getNodes().get(id) const indexNoun = this.index.getNodes().get(id)
if (!indexNode) { if (!indexNoun) {
throw new Error( throw new Error(
`Failed to retrieve newly created edge node with ID ${id}` `Failed to retrieve newly created verb noun with ID ${id}`
) )
} }
// Update edge connections from index // Update verb connections from index
edge.connections = indexNode.connections verb.connections = indexNoun.connections
// Save edge to storage // Save verb to storage
await this.storage!.saveEdge(edge) await this.storage!.saveVerb(verb)
return id return id
} catch (error) { } catch (error) {
console.error('Failed to add edge:', error) console.error('Failed to add verb:', error)
throw new Error(`Failed to add edge: ${error}`) throw new Error(`Failed to add verb: ${error}`)
} }
} }
/** /**
* Get an edge by ID * Get a verb by ID
*/ */
public async getEdge(id: string): Promise<Edge | null> { public async getVerb(id: string): Promise<GraphVerb | null> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
return await this.storage!.getEdge(id) return await this.storage!.getVerb(id)
} catch (error) { } catch (error) {
console.error(`Failed to get edge ${id}:`, error) console.error(`Failed to get verb ${id}:`, error)
throw new Error(`Failed to get edge ${id}: ${error}`) throw new Error(`Failed to get verb ${id}: ${error}`)
} }
} }
/** /**
* Get all edges * Get all verbs
*/ */
public async getAllEdges(): Promise<Edge[]> { public async getAllVerbs(): Promise<GraphVerb[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
return await this.storage!.getAllEdges() return await this.storage!.getAllVerbs()
} catch (error) { } catch (error) {
console.error('Failed to get all edges:', error) console.error('Failed to get all verbs:', error)
throw new Error(`Failed to get all edges: ${error}`) throw new Error(`Failed to get all verbs: ${error}`)
} }
} }
/** /**
* Get edges by source node ID * Get verbs by source noun ID
*/ */
public async getEdgesBySource(sourceId: string): Promise<Edge[]> { public async getVerbsBySource(sourceId: string): Promise<GraphVerb[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
return await this.storage!.getEdgesBySource(sourceId) return await this.storage!.getVerbsBySource(sourceId)
} catch (error) { } catch (error) {
console.error(`Failed to get edges by source ${sourceId}:`, error) console.error(`Failed to get verbs by source ${sourceId}:`, error)
throw new Error(`Failed to get edges by source ${sourceId}: ${error}`) throw new Error(`Failed to get verbs by source ${sourceId}: ${error}`)
} }
} }
/** /**
* Get edges by target node ID * Get verbs by target noun ID
*/ */
public async getEdgesByTarget(targetId: string): Promise<Edge[]> { public async getVerbsByTarget(targetId: string): Promise<GraphVerb[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
return await this.storage!.getEdgesByTarget(targetId) return await this.storage!.getVerbsByTarget(targetId)
} catch (error) { } catch (error) {
console.error(`Failed to get edges by target ${targetId}:`, error) console.error(`Failed to get verbs by target ${targetId}:`, error)
throw new Error(`Failed to get edges by target ${targetId}: ${error}`) throw new Error(`Failed to get verbs by target ${targetId}: ${error}`)
} }
} }
/** /**
* Get edges by type * Get verbs by type
*/ */
public async getEdgesByType(type: string): Promise<Edge[]> { public async getVerbsByType(type: string): Promise<GraphVerb[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
return await this.storage!.getEdgesByType(type) return await this.storage!.getVerbsByType(type)
} catch (error) { } catch (error) {
console.error(`Failed to get edges by type ${type}:`, error) console.error(`Failed to get verbs by type ${type}:`, error)
throw new Error(`Failed to get edges by type ${type}: ${error}`) throw new Error(`Failed to get verbs by type ${type}: ${error}`)
} }
} }
/** /**
* Delete an edge * Delete a verb
*/ */
public async deleteEdge(id: string): Promise<boolean> { public async deleteVerb(id: string): Promise<boolean> {
await this.ensureInitialized() await this.ensureInitialized()
// Check if database is in read-only mode // Check if database is in read-only mode
@ -734,12 +776,12 @@ export class BrainyData<T = any> {
} }
// Remove from storage // Remove from storage
await this.storage!.deleteEdge(id) await this.storage!.deleteVerb(id)
return true return true
} catch (error) { } catch (error) {
console.error(`Failed to delete edge ${id}:`, error) console.error(`Failed to delete verb ${id}:`, error)
throw new Error(`Failed to delete edge ${id}: ${error}`) throw new Error(`Failed to delete verb ${id}: ${error}`)
} }
} }

288
src/cli.ts Normal file
View file

@ -0,0 +1,288 @@
#!/usr/bin/env node
/**
* Brainy CLI
* A command-line interface for interacting with the Brainy vector database
*/
import { BrainyData, NounType, VerbType, FileSystemStorage } from './index.js';
import { fileURLToPath } from 'url';
import { dirname, join } from 'path';
import fs from 'fs';
import { Command } from 'commander';
// Get the directory of the current module
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
// Import package.json for version info
const packageJson = {
name: '@soulcraft/brainy',
version: '0.6.0',
description: 'A vector database using HNSW indexing with Origin Private File System storage'
};
// Helper function to parse JSON safely
function parseJSON(str: string): any {
try {
return JSON.parse(str);
} catch (e) {
console.error('Error parsing JSON:', (e as Error).message);
return {};
}
}
// Helper function to resolve noun type
function resolveNounType(type: string | number | undefined): NounType {
if (!type) return NounType.Thing;
// If it's a string, try to match it to a NounType
if (typeof type === 'string') {
const nounTypeKey = Object.keys(NounType).find(
key => key.toLowerCase() === type.toLowerCase()
);
return nounTypeKey ? NounType[nounTypeKey as keyof typeof NounType] : NounType.Thing;
}
// Convert number to string type for safety
return Object.values(NounType)[type as number] || NounType.Thing;
}
// Helper function to resolve verb type
function resolveVerbType(type: string | number | undefined): VerbType {
if (!type) return VerbType.RelatedTo;
// If it's a string, try to match it to a VerbType
if (typeof type === 'string') {
const verbTypeKey = Object.keys(VerbType).find(
key => key.toLowerCase() === type.toLowerCase()
);
return verbTypeKey ? VerbType[verbTypeKey as keyof typeof VerbType] : VerbType.RelatedTo;
}
// Convert number to string type for safety
return Object.values(VerbType)[type as number] || VerbType.RelatedTo;
}
// Create a new Command instance
const program = new Command();
// Configure the program
program
.name(packageJson.name)
.description(packageJson.description)
.version(packageJson.version);
// Create data directory if it doesn't exist
const dataDir = join(__dirname, '..', 'data');
if (!fs.existsSync(dataDir)) {
fs.mkdirSync(dataDir, { recursive: true });
}
// Create a database instance with file system storage
const createDb = () => {
return new BrainyData({
storageAdapter: new FileSystemStorage(dataDir)
});
};
// Define commands
program
.command('init')
.description('Initialize a new database')
.action(async () => {
try {
const db = createDb();
await db.init();
console.log('Database initialized successfully');
} catch (error) {
console.error('Error:', (error as Error).message);
process.exit(1);
}
});
program
.command('add')
.description('Add a new noun with the given text and optional metadata')
.argument('<text>', 'Text to add as a noun')
.argument('[metadata]', 'Optional metadata as JSON string')
.action(async (text, metadataStr) => {
try {
const db = createDb();
await db.init();
const metadata = metadataStr ? parseJSON(metadataStr) : {};
// Process metadata to handle noun type
if (metadata.noun) {
metadata.noun = resolveNounType(metadata.noun);
}
const id = await db.add(text, metadata);
console.log(`Added noun with ID: ${id}`);
} catch (error) {
console.error('Error:', (error as Error).message);
process.exit(1);
}
});
program
.command('search')
.description('Search for nouns similar to the query')
.argument('<query>', 'Search query text')
.option('-l, --limit <number>', 'Maximum number of results to return', '5')
.action(async (query, options) => {
try {
const db = createDb();
await db.init();
const limit = parseInt(options.limit, 10);
const results = await db.searchText(query, limit);
console.log(`Search results for "${query}":`);
results.forEach((result, index) => {
console.log(`${index + 1}. ID: ${result.id}`);
console.log(` Score: ${result.score.toFixed(4)}`);
console.log(` Metadata: ${JSON.stringify(result.metadata)}`);
console.log(` Vector: [${result.vector.slice(0, 3).map(v => v.toFixed(2)).join(', ')}...]`);
console.log();
});
} catch (error) {
console.error('Error:', (error as Error).message);
process.exit(1);
}
});
program
.command('get')
.description('Get a noun by ID')
.argument('<id>', 'ID of the noun to get')
.action(async (id) => {
try {
const db = createDb();
await db.init();
const noun = await db.get(id);
if (noun) {
console.log(`Noun ID: ${noun.id}`);
console.log(`Metadata: ${JSON.stringify(noun.metadata)}`);
console.log(`Vector: [${noun.vector.slice(0, 5).map(v => v.toFixed(2)).join(', ')}...]`);
} else {
console.log(`No noun found with ID: ${id}`);
}
} catch (error) {
console.error('Error:', (error as Error).message);
process.exit(1);
}
});
program
.command('delete')
.description('Delete a noun by ID')
.argument('<id>', 'ID of the noun to delete')
.action(async (id) => {
try {
const db = createDb();
await db.init();
await db.delete(id);
console.log(`Deleted noun with ID: ${id}`);
} catch (error) {
console.error('Error:', (error as Error).message);
process.exit(1);
}
});
program
.command('addVerb')
.description('Add a relationship between nouns')
.argument('<sourceId>', 'ID of the source noun')
.argument('<targetId>', 'ID of the target noun')
.argument('<verbType>', 'Type of relationship')
.argument('[metadata]', 'Optional metadata as JSON string')
.action(async (sourceId, targetId, verbTypeStr, metadataStr) => {
try {
const db = createDb();
await db.init();
const verbType = resolveVerbType(verbTypeStr);
const verbMetadata = metadataStr ? parseJSON(metadataStr) : {};
// Add verb type to metadata
verbMetadata.verb = verbType;
const verbId = await db.addVerb(sourceId, targetId, verbMetadata);
console.log(`Added verb with ID: ${verbId}`);
} catch (error) {
console.error('Error:', (error as Error).message);
process.exit(1);
}
});
program
.command('getVerbs')
.description('Get all relationships for a noun')
.argument('<id>', 'ID of the noun to get relationships for')
.action(async (id) => {
try {
const db = createDb();
await db.init();
const verbs = await db.getVerbsBySource(id);
console.log(`Relationships for noun ${id}:`);
if (verbs.length === 0) {
console.log('No relationships found');
} else {
verbs.forEach((verb, index) => {
console.log(`${index + 1}. ID: ${verb.id}`);
console.log(` Type: ${Object.keys(VerbType).find(key => VerbType[key as keyof typeof VerbType] === verb.metadata.verb) || verb.metadata.verb}`);
console.log(` Target: ${verb.targetId}`);
console.log(` Metadata: ${JSON.stringify(verb.metadata)}`);
console.log();
});
}
} catch (error) {
console.error('Error:', (error as Error).message);
process.exit(1);
}
});
program
.command('status')
.description('Show database status')
.action(async () => {
try {
const db = createDb();
await db.init();
const status = await db.status();
console.log('Database Status:');
console.log(`Storage type: ${status.type}`);
console.log(`Storage used: ${status.used} bytes`);
console.log(`Storage quota: ${status.quota !== null ? `${status.quota} bytes` : 'unlimited'}`);
// Display additional details if available
if (status.details) {
console.log('Additional details:');
Object.entries(status.details).forEach(([key, value]) => {
console.log(` ${key}: ${JSON.stringify(value)}`);
});
}
} catch (error) {
console.error('Error:', (error as Error).message);
process.exit(1);
}
});
// Add examples to help text
program.addHelpText('after', `
Examples:
$ brainy init
$ brainy add "Cats are independent pets" '{"noun":"Thing","category":"animal"}'
$ brainy search "feline pets" --limit 5
$ brainy addVerb id1 id2 RelatedTo '{"description":"Both are pets"}'
`);
// Parse command line arguments
program.parse();

48
src/coreTypes.ts
View file

@ -57,31 +57,31 @@ export interface EmbeddingModel {
} }
/** /**
* HNSW graph node * HNSW graph noun
*/ */
export interface HNSWNode { export interface HNSWNoun {
id: string; id: string;
vector: Vector; vector: Vector;
connections: Map<number, Set<string>>; // level -> set of connected node ids connections: Map<number, Set<string>>; // level -> set of connected noun ids
} }
/** /**
* Edge representing a relationship between nodes * Verb representing a relationship between nouns
* Extends HNSWNode to allow edges to be first-class entities in the data model * Extends HNSWNoun to allow verbs to be first-class entities in the data model
*/ */
export interface Edge extends HNSWNode { export interface GraphVerb extends HNSWNoun {
sourceId: string; // ID of the source node sourceId: string; // ID of the source noun
targetId: string; // ID of the target node targetId: string; // ID of the target noun
type?: string; // Optional type of the relationship type?: string; // Optional type of the relationship
weight?: number; // Optional weight of the relationship weight?: number; // Optional weight of the relationship
metadata?: any; // Optional metadata for the edge metadata?: any; // Optional metadata for the verb
} }
/** /**
* HNSW index configuration * HNSW index configuration
*/ */
export interface HNSWConfig { export interface HNSWConfig {
M: number; // Maximum number of connections per node M: number; // Maximum number of connections per noun
efConstruction: number; // Size of the dynamic candidate list during construction efConstruction: number; // Size of the dynamic candidate list during construction
efSearch: number; // Size of the dynamic candidate list during search efSearch: number; // Size of the dynamic candidate list during search
ml: number; // Maximum level ml: number; // Maximum level
@ -93,34 +93,34 @@ export interface HNSWConfig {
export interface StorageAdapter { export interface StorageAdapter {
init(): Promise<void>; init(): Promise<void>;
saveNode(node: HNSWNode): Promise<void>; saveNoun(noun: HNSWNoun): Promise<void>;
getNode(id: string): Promise<HNSWNode | null>; getNoun(id: string): Promise<HNSWNoun | null>;
getAllNodes(): Promise<HNSWNode[]>; getAllNouns(): Promise<HNSWNoun[]>;
/** /**
* Get nodes by noun type * Get nouns by noun type
* @param nounType The noun type to filter by * @param nounType The noun type to filter by
* @returns Promise that resolves to an array of nodes of the specified noun type * @returns Promise that resolves to an array of nouns of the specified noun type
*/ */
getNodesByNounType(nounType: string): Promise<HNSWNode[]>; getNounsByNounType(nounType: string): Promise<HNSWNoun[]>;
deleteNode(id: string): Promise<void>; deleteNoun(id: string): Promise<void>;
saveEdge(edge: Edge): Promise<void>; saveVerb(verb: GraphVerb): Promise<void>;
getEdge(id: string): Promise<Edge | null>; getVerb(id: string): Promise<GraphVerb | null>;
getAllEdges(): Promise<Edge[]>; getAllVerbs(): Promise<GraphVerb[]>;
getEdgesBySource(sourceId: string): Promise<Edge[]>; getVerbsBySource(sourceId: string): Promise<GraphVerb[]>;
getEdgesByTarget(targetId: string): Promise<Edge[]>; getVerbsByTarget(targetId: string): Promise<GraphVerb[]>;
getEdgesByType(type: string): Promise<Edge[]>; getVerbsByType(type: string): Promise<GraphVerb[]>;
deleteEdge(id: string): Promise<void>; deleteVerb(id: string): Promise<void>;
saveMetadata(id: string, metadata: any): Promise<void>; saveMetadata(id: string, metadata: any): Promise<void>;

184
src/hnsw/hnswIndex.ts
View file

@ -3,19 +3,19 @@
* Based on the paper: "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs" * Based on the paper: "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs"
*/ */
import { DistanceFunction, HNSWConfig, HNSWNode, Vector, VectorDocument } from '../coreTypes.js' import { DistanceFunction, HNSWConfig, HNSWNoun, Vector, VectorDocument } from '../coreTypes.js'
import { euclideanDistance } from '../utils/index.js' import { euclideanDistance } from '../utils/index.js'
// Default HNSW parameters // Default HNSW parameters
const DEFAULT_CONFIG: HNSWConfig = { const DEFAULT_CONFIG: HNSWConfig = {
M: 16, // Max number of connections per node M: 16, // Max number of connections per noun
efConstruction: 200, // Size of a dynamic candidate list during construction efConstruction: 200, // Size of a dynamic candidate list during construction
efSearch: 50, // Size of a dynamic candidate list during search efSearch: 50, // Size of a dynamic candidate list during search
ml: 16 // Max level ml: 16 // Max level
} }
export class HNSWIndex { export class HNSWIndex {
private nodes: Map<string, HNSWNode> = new Map() private nouns: Map<string, HNSWNoun> = new Map()
private entryPointId: string | null = null private entryPointId: string | null = null
private maxLevel = 0 private maxLevel = 0
private config: HNSWConfig private config: HNSWConfig
@ -55,55 +55,55 @@ export class HNSWIndex {
) )
} }
// Generate random level for this node // Generate random level for this noun
const nodeLevel = this.getRandomLevel() const nounLevel = this.getRandomLevel()
// Create new node // Create new noun
const node: HNSWNode = { const noun: HNSWNoun = {
id, id,
vector, vector,
connections: new Map() connections: new Map()
} }
// Initialize empty connection sets for each level // Initialize empty connection sets for each level
for (let level = 0; level <= nodeLevel; level++) { for (let level = 0; level <= nounLevel; level++) {
node.connections.set(level, new Set<string>()) noun.connections.set(level, new Set<string>())
} }
// If this is the first node, make it the entry point // If this is the first noun, make it the entry point
if (this.nodes.size === 0) { if (this.nouns.size === 0) {
this.entryPointId = id this.entryPointId = id
this.maxLevel = nodeLevel this.maxLevel = nounLevel
this.nodes.set(id, node) this.nouns.set(id, noun)
return id return id
} }
// Find entry point // Find entry point
if (!this.entryPointId) { if (!this.entryPointId) {
console.error('Entry point ID is null') console.error('Entry point ID is null')
// If there's no entry point, this is the first node, so we should have returned earlier // If there's no entry point, this is the first noun, so we should have returned earlier
// This is a safety check // This is a safety check
this.entryPointId = id this.entryPointId = id
this.maxLevel = nodeLevel this.maxLevel = nounLevel
this.nodes.set(id, node) this.nouns.set(id, noun)
return id return id
} }
const entryPoint = this.nodes.get(this.entryPointId) const entryPoint = this.nouns.get(this.entryPointId)
if (!entryPoint) { if (!entryPoint) {
console.error(`Entry point with ID ${this.entryPointId} not found`) console.error(`Entry point with ID ${this.entryPointId} not found`)
// If the entry point doesn't exist, treat this as the first node // If the entry point doesn't exist, treat this as the first noun
this.entryPointId = id this.entryPointId = id
this.maxLevel = nodeLevel this.maxLevel = nounLevel
this.nodes.set(id, node) this.nouns.set(id, noun)
return id return id
} }
let currObj = entryPoint let currObj = entryPoint
let currDist = this.distanceFunction(vector, entryPoint.vector) let currDist = this.distanceFunction(vector, entryPoint.vector)
// Traverse the graph from top to bottom to find the closest node // Traverse the graph from top to bottom to find the closest noun
for (let level = this.maxLevel; level > nodeLevel; level--) { for (let level = this.maxLevel; level > nounLevel; level--) {
let changed = true let changed = true
while (changed) { while (changed) {
changed = false changed = false
@ -112,7 +112,7 @@ export class HNSWIndex {
const connections = currObj.connections.get(level) || new Set<string>() const connections = currObj.connections.get(level) || new Set<string>()
for (const neighborId of connections) { for (const neighborId of connections) {
const neighbor = this.nodes.get(neighborId) const neighbor = this.nouns.get(neighborId)
if (!neighbor) { if (!neighbor) {
console.error(`Neighbor with ID ${neighborId} not found in addItem traversal`) console.error(`Neighbor with ID ${neighborId} not found in addItem traversal`)
continue continue
@ -128,10 +128,10 @@ export class HNSWIndex {
} }
} }
// For each level from nodeLevel down to 0 // For each level from nounLevel down to 0
for (let level = Math.min(nodeLevel, this.maxLevel); level >= 0; level--) { for (let level = Math.min(nounLevel, this.maxLevel); level >= 0; level--) {
// Find ef nearest elements using greedy search // Find ef nearest elements using greedy search
const nearestNodes = this.searchLayer( const nearestNouns = this.searchLayer(
vector, vector,
currObj, currObj,
this.config.efConstruction, this.config.efConstruction,
@ -141,19 +141,19 @@ export class HNSWIndex {
// Select M nearest neighbors // Select M nearest neighbors
const neighbors = this.selectNeighbors( const neighbors = this.selectNeighbors(
vector, vector,
nearestNodes, nearestNouns,
this.config.M this.config.M
) )
// Add bidirectional connections // Add bidirectional connections
for (const [neighborId, _] of neighbors) { for (const [neighborId, _] of neighbors) {
const neighbor = this.nodes.get(neighborId) const neighbor = this.nouns.get(neighborId)
if (!neighbor) { if (!neighbor) {
console.error(`Neighbor with ID ${neighborId} not found`) console.error(`Neighbor with ID ${neighborId} not found`)
continue continue
} }
node.connections.get(level)!.add(neighborId) noun.connections.get(level)!.add(neighborId)
// Add reverse connection // Add reverse connection
if (!neighbor.connections.has(level)) { if (!neighbor.connections.has(level)) {
@ -168,29 +168,29 @@ export class HNSWIndex {
} }
// Update entry point for the next level // Update entry point for the next level
if (nearestNodes.size > 0) { if (nearestNouns.size > 0) {
const [nearestId, nearestDist] = [...nearestNodes][0] const [nearestId, nearestDist] = [...nearestNouns][0]
if (nearestDist < currDist) { if (nearestDist < currDist) {
currDist = nearestDist currDist = nearestDist
const nearestNode = this.nodes.get(nearestId) const nearestNoun = this.nouns.get(nearestId)
if (!nearestNode) { if (!nearestNoun) {
console.error(`Nearest node with ID ${nearestId} not found in addItem`) console.error(`Nearest noun with ID ${nearestId} not found in addItem`)
// Keep the current object as is // Keep the current object as is
} else { } else {
currObj = nearestNode currObj = nearestNoun
} }
} }
} }
} }
// Update max level and entry point if needed // Update max level and entry point if needed
if (nodeLevel > this.maxLevel) { if (nounLevel > this.maxLevel) {
this.maxLevel = nodeLevel this.maxLevel = nounLevel
this.entryPointId = id this.entryPointId = id
} }
// Add node to the index // Add noun to the index
this.nodes.set(id, node) this.nouns.set(id, noun)
return id return id
} }
@ -198,7 +198,7 @@ export class HNSWIndex {
* Search for nearest neighbors * Search for nearest neighbors
*/ */
public search(queryVector: Vector, k: number = 10): Array<[string, number]> { public search(queryVector: Vector, k: number = 10): Array<[string, number]> {
if (this.nodes.size === 0) { if (this.nouns.size === 0) {
return [] return []
} }
@ -219,7 +219,7 @@ export class HNSWIndex {
return [] return []
} }
const entryPoint = this.nodes.get(this.entryPointId) const entryPoint = this.nouns.get(this.entryPointId)
if (!entryPoint) { if (!entryPoint) {
console.error(`Entry point with ID ${this.entryPointId} not found`) console.error(`Entry point with ID ${this.entryPointId} not found`)
return [] return []
@ -228,7 +228,7 @@ export class HNSWIndex {
let currObj = entryPoint let currObj = entryPoint
let currDist = this.distanceFunction(queryVector, currObj.vector) let currDist = this.distanceFunction(queryVector, currObj.vector)
// Traverse the graph from top to bottom to find the closest node // Traverse the graph from top to bottom to find the closest noun
for (let level = this.maxLevel; level > 0; level--) { for (let level = this.maxLevel; level > 0; level--) {
let changed = true let changed = true
while (changed) { while (changed) {
@ -238,7 +238,7 @@ export class HNSWIndex {
const connections = currObj.connections.get(level) || new Set<string>() const connections = currObj.connections.get(level) || new Set<string>()
for (const neighborId of connections) { for (const neighborId of connections) {
const neighbor = this.nodes.get(neighborId) const neighbor = this.nouns.get(neighborId)
if (!neighbor) { if (!neighbor) {
console.error(`Neighbor with ID ${neighborId} not found in search`) console.error(`Neighbor with ID ${neighborId} not found in search`)
continue continue
@ -258,7 +258,7 @@ export class HNSWIndex {
} }
// Search at level 0 with ef = k // Search at level 0 with ef = k
const nearestNodes = this.searchLayer( const nearestNouns = this.searchLayer(
queryVector, queryVector,
currObj, currObj,
Math.max(this.config.efSearch, k), Math.max(this.config.efSearch, k),
@ -266,23 +266,23 @@ export class HNSWIndex {
) )
// Convert to array and sort by distance // Convert to array and sort by distance
return [...nearestNodes].slice(0, k) return [...nearestNouns].slice(0, k)
} }
/** /**
* Remove an item from the index * Remove an item from the index
*/ */
public removeItem(id: string): boolean { public removeItem(id: string): boolean {
if (!this.nodes.has(id)) { if (!this.nouns.has(id)) {
return false return false
} }
const node = this.nodes.get(id)! const noun = this.nouns.get(id)!
// Remove connections to this node from all neighbors // Remove connections to this noun from all neighbors
for (const [level, connections] of node.connections.entries()) { for (const [level, connections] of noun.connections.entries()) {
for (const neighborId of connections) { for (const neighborId of connections) {
const neighbor = this.nodes.get(neighborId) const neighbor = this.nouns.get(neighborId)
if (!neighbor) { if (!neighbor) {
console.error(`Neighbor with ID ${neighborId} not found in removeItem`) console.error(`Neighbor with ID ${neighborId} not found in removeItem`)
continue continue
@ -290,46 +290,46 @@ export class HNSWIndex {
if (neighbor.connections.has(level)) { if (neighbor.connections.has(level)) {
neighbor.connections.get(level)!.delete(id) neighbor.connections.get(level)!.delete(id)
// Prune connections after removing this node to ensure consistency // Prune connections after removing this noun to ensure consistency
this.pruneConnections(neighbor, level) this.pruneConnections(neighbor, level)
} }
} }
} }
// Also check all other nodes for references to this node and remove them // Also check all other nouns for references to this noun and remove them
for (const [nodeId, otherNode] of this.nodes.entries()) { for (const [nounId, otherNoun] of this.nouns.entries()) {
if (nodeId === id) continue // Skip the node being removed if (nounId === id) continue // Skip the noun being removed
for (const [level, connections] of otherNode.connections.entries()) { for (const [level, connections] of otherNoun.connections.entries()) {
if (connections.has(id)) { if (connections.has(id)) {
connections.delete(id) connections.delete(id)
// Prune connections after removing this reference // Prune connections after removing this reference
this.pruneConnections(otherNode, level) this.pruneConnections(otherNoun, level)
} }
} }
} }
// Remove the node // Remove the noun
this.nodes.delete(id) this.nouns.delete(id)
// If we removed the entry point, find a new one // If we removed the entry point, find a new one
if (this.entryPointId === id) { if (this.entryPointId === id) {
if (this.nodes.size === 0) { if (this.nouns.size === 0) {
this.entryPointId = null this.entryPointId = null
this.maxLevel = 0 this.maxLevel = 0
} else { } else {
// Find the node with the highest level // Find the noun with the highest level
let maxLevel = 0 let maxLevel = 0
let newEntryPointId = null let newEntryPointId = null
for (const [nodeId, node] of this.nodes.entries()) { for (const [nounId, noun] of this.nouns.entries()) {
if (node.connections.size === 0) continue // Skip nodes with no connections if (noun.connections.size === 0) continue // Skip nouns with no connections
const nodeLevel = Math.max(...node.connections.keys()) const nounLevel = Math.max(...noun.connections.keys())
if (nodeLevel >= maxLevel) { if (nounLevel >= maxLevel) {
maxLevel = nodeLevel maxLevel = nounLevel
newEntryPointId = nodeId newEntryPointId = nounId
} }
} }
@ -342,17 +342,25 @@ export class HNSWIndex {
} }
/** /**
* Get all nodes in the index * Get all nouns in the index
*/ */
public getNodes(): Map<string, HNSWNode> { public getNouns(): Map<string, HNSWNoun> {
return new Map(this.nodes) return new Map(this.nouns)
}
/**
* Get all nodes in the index (alias for getNouns for backward compatibility)
* @deprecated Use getNouns() instead
*/
public getNodes(): Map<string, HNSWNoun> {
return this.getNouns()
} }
/** /**
* Clear the index * Clear the index
*/ */
public clear(): void { public clear(): void {
this.nodes.clear() this.nouns.clear()
this.entryPointId = null this.entryPointId = null
this.maxLevel = 0 this.maxLevel = 0
} }
@ -361,7 +369,7 @@ export class HNSWIndex {
* Get the size of the index * Get the size of the index
*/ */
public size(): number { public size(): number {
return this.nodes.size return this.nouns.size
} }
/** /**
@ -373,15 +381,15 @@ export class HNSWIndex {
/** /**
* Search within a specific layer * Search within a specific layer
* Returns a map of node IDs to distances, sorted by distance * Returns a map of noun IDs to distances, sorted by distance
*/ */
private searchLayer( private searchLayer(
queryVector: Vector, queryVector: Vector,
entryPoint: HNSWNode, entryPoint: HNSWNoun,
ef: number, ef: number,
level: number level: number
): Map<string, number> { ): Map<string, number> {
// Set of visited nodes // Set of visited nouns
const visited = new Set<string>([entryPoint.id]) const visited = new Set<string>([entryPoint.id])
// Priority queue of candidates (closest first) // Priority queue of candidates (closest first)
@ -411,18 +419,18 @@ export class HNSWIndex {
} }
// Explore neighbors of the closest candidate // Explore neighbors of the closest candidate
const node = this.nodes.get(closestId) const noun = this.nouns.get(closestId)
if (!node) { if (!noun) {
console.error(`Node with ID ${closestId} not found in searchLayer`) console.error(`Noun with ID ${closestId} not found in searchLayer`)
continue continue
} }
const connections = node.connections.get(level) || new Set<string>() const connections = noun.connections.get(level) || new Set<string>()
for (const neighborId of connections) { for (const neighborId of connections) {
if (!visited.has(neighborId)) { if (!visited.has(neighborId)) {
visited.add(neighborId) visited.add(neighborId)
const neighbor = this.nodes.get(neighborId) const neighbor = this.nouns.get(neighborId)
if (!neighbor) { if (!neighbor) {
console.error(`Neighbor with ID ${neighborId} not found in searchLayer`) console.error(`Neighbor with ID ${neighborId} not found in searchLayer`)
continue continue
@ -478,10 +486,10 @@ export class HNSWIndex {
} }
/** /**
* Ensure a node doesn't have too many connections at a given level * Ensure a noun doesn't have too many connections at a given level
*/ */
private pruneConnections(node: HNSWNode, level: number): void { private pruneConnections(noun: HNSWNoun, level: number): void {
const connections = node.connections.get(level)! const connections = noun.connections.get(level)!
if (connections.size <= this.config.M) { if (connections.size <= this.config.M) {
return return
} }
@ -491,7 +499,7 @@ export class HNSWIndex {
const validNeighborIds = new Set<string>() const validNeighborIds = new Set<string>()
for (const neighborId of connections) { for (const neighborId of connections) {
const neighbor = this.nodes.get(neighborId) const neighbor = this.nouns.get(neighborId)
if (!neighbor) { if (!neighbor) {
console.error(`Neighbor with ID ${neighborId} not found in pruneConnections`) console.error(`Neighbor with ID ${neighborId} not found in pruneConnections`)
continue continue
@ -500,7 +508,7 @@ export class HNSWIndex {
// Only add valid neighbors to the distances map // Only add valid neighbors to the distances map
distances.set( distances.set(
neighborId, neighborId,
this.distanceFunction(node.vector, neighbor.vector) this.distanceFunction(noun.vector, neighbor.vector)
) )
validNeighborIds.add(neighborId) validNeighborIds.add(neighborId)
} }
@ -508,23 +516,23 @@ export class HNSWIndex {
// Only proceed if we have valid neighbors // Only proceed if we have valid neighbors
if (distances.size === 0) { if (distances.size === 0) {
// If no valid neighbors, clear connections at this level // If no valid neighbors, clear connections at this level
node.connections.set(level, new Set()) noun.connections.set(level, new Set())
return return
} }
// Select M closest neighbors from valid ones // Select M closest neighbors from valid ones
const selectedNeighbors = this.selectNeighbors( const selectedNeighbors = this.selectNeighbors(
node.vector, noun.vector,
distances, distances,
this.config.M this.config.M
) )
// Update connections with only valid neighbors // Update connections with only valid neighbors
node.connections.set(level, new Set(selectedNeighbors.keys())) noun.connections.set(level, new Set(selectedNeighbors.keys()))
} }
/** /**
* Generate a random level for a new node * Generate a random level for a new noun
* Uses the same distribution as in the original HNSW paper * Uses the same distribution as in the original HNSW paper
*/ */
private getRandomLevel(): number { private getRandomLevel(): number {

10
src/index.ts
View file

@ -148,8 +148,8 @@ import type {
DistanceFunction, DistanceFunction,
EmbeddingFunction, EmbeddingFunction,
EmbeddingModel, EmbeddingModel,
HNSWNode, HNSWNoun,
Edge, GraphVerb,
HNSWConfig, HNSWConfig,
StorageAdapter StorageAdapter
} from './coreTypes.js' } from './coreTypes.js'
@ -161,8 +161,8 @@ export type {
DistanceFunction, DistanceFunction,
EmbeddingFunction, EmbeddingFunction,
EmbeddingModel, EmbeddingModel,
HNSWNode, HNSWNoun,
Edge, GraphVerb,
HNSWConfig, HNSWConfig,
StorageAdapter StorageAdapter
} }
@ -213,7 +213,6 @@ export type {
// Export graph types // Export graph types
import type { import type {
GraphNoun, GraphNoun,
GraphVerb,
EmbeddedGraphVerb, EmbeddedGraphVerb,
Person, Person,
Place, Place,
@ -226,7 +225,6 @@ import { NounType, VerbType } from './types/graphTypes.js'
export type { export type {
GraphNoun, GraphNoun,
GraphVerb,
EmbeddedGraphVerb, EmbeddedGraphVerb,
Person, Person,
Place, Place,

68
src/storage/fileSystemStorage.ts
View file

@ -1,4 +1,4 @@
import { Edge, HNSWNode, StorageAdapter } from '../coreTypes.js' import { GraphVerb, HNSWNoun, StorageAdapter } from '../coreTypes.js'
// We'll dynamically import Node.js built-in modules // We'll dynamically import Node.js built-in modules
let fs: any let fs: any
@ -114,35 +114,35 @@ export class FileSystemStorage implements StorageAdapter {
/** /**
* Save a node to storage * Save a node to storage
*/ */
public async saveNode(node: HNSWNode): Promise<void> { public async saveNoun(noun: HNSWNoun): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
// Convert connections Map to a serializable format // Convert connections Map to a serializable format
const serializableNode = { const serializableNode = {
...node, ...noun,
connections: this.mapToObject(node.connections, (set) => Array.from(set)) connections: this.mapToObject(noun.connections, (set) => Array.from(set as Set<string>))
} }
// Get the appropriate directory based on the node's metadata // Get the appropriate directory based on the node's metadata
const nodeDir = await this.getNodeDirectory(node.id) const nodeDir = await this.getNodeDirectory(noun.id)
const filePath = path.join(nodeDir, `${node.id}.json`) const filePath = path.join(nodeDir, `${noun.id}.json`)
await fs.promises.writeFile( await fs.promises.writeFile(
filePath, filePath,
JSON.stringify(serializableNode, null, 2), JSON.stringify(serializableNode, null, 2),
'utf8' 'utf8'
) )
} catch (error) { } catch (error) {
console.error(`Failed to save node ${node.id}:`, error) console.error(`Failed to save node ${noun.id}:`, error)
throw new Error(`Failed to save node ${node.id}: ${error}`) throw new Error(`Failed to save node ${noun.id}: ${error}`)
} }
} }
/** /**
* Get a node from storage * Get a node from storage
*/ */
public async getNode(id: string): Promise<HNSWNode | null> { public async getNoun(id: string): Promise<HNSWNoun | null> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
@ -243,7 +243,7 @@ export class FileSystemStorage implements StorageAdapter {
* @param nounType The noun type to filter by * @param nounType The noun type to filter by
* @returns Promise that resolves to an array of nodes of the specified noun type * @returns Promise that resolves to an array of nodes of the specified noun type
*/ */
public async getNodesByNounType(nounType: string): Promise<HNSWNode[]> { public async getNounsByNounType(nounType: string): Promise<HNSWNoun[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
@ -272,7 +272,7 @@ export class FileSystemStorage implements StorageAdapter {
dir = this.defaultDir dir = this.defaultDir
} }
const nodes: HNSWNode[] = [] const nodes: HNSWNoun[] = []
try { try {
const files = await fs.promises.readdir(dir) const files = await fs.promises.readdir(dir)
@ -284,7 +284,7 @@ export class FileSystemStorage implements StorageAdapter {
}) })
const dirNodes = await Promise.all(nodePromises) const dirNodes = await Promise.all(nodePromises)
nodes.push(...dirNodes.filter((node): node is HNSWNode => node !== null)) nodes.push(...dirNodes.filter((node): node is HNSWNoun => node !== null))
} catch (dirError) { } catch (dirError) {
// If directory doesn't exist or can't be read, log a warning // If directory doesn't exist or can't be read, log a warning
console.warn(`Could not read directory for noun type ${nounType}:`, dirError) console.warn(`Could not read directory for noun type ${nounType}:`, dirError)
@ -300,7 +300,7 @@ export class FileSystemStorage implements StorageAdapter {
/** /**
* Get all nodes from storage * Get all nodes from storage
*/ */
public async getAllNodes(): Promise<HNSWNode[]> { public async getAllNouns(): Promise<HNSWNoun[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
@ -316,11 +316,11 @@ export class FileSystemStorage implements StorageAdapter {
] ]
// Run searches in parallel for all noun types // Run searches in parallel for all noun types
const nodePromises = nounTypes.map(nounType => this.getNodesByNounType(nounType)) const nodePromises = nounTypes.map(nounType => this.getNounsByNounType(nounType))
const nodeArrays = await Promise.all(nodePromises) const nodeArrays = await Promise.all(nodePromises)
// Combine all results // Combine all results
const allNodes: HNSWNode[] = [] const allNodes: HNSWNoun[] = []
for (const nodes of nodeArrays) { for (const nodes of nodeArrays) {
allNodes.push(...nodes) allNodes.push(...nodes)
} }
@ -335,7 +335,7 @@ export class FileSystemStorage implements StorageAdapter {
/** /**
* Read a node from a file * Read a node from a file
*/ */
private async readNodeFromFile(filePath: string): Promise<HNSWNode | null> { private async readNodeFromFile(filePath: string): Promise<HNSWNoun | null> {
try { try {
const data = await fs.promises.readFile(filePath, 'utf8') const data = await fs.promises.readFile(filePath, 'utf8')
const parsedNode = JSON.parse(data) const parsedNode = JSON.parse(data)
@ -360,7 +360,7 @@ export class FileSystemStorage implements StorageAdapter {
/** /**
* Delete a node from storage * Delete a node from storage
*/ */
public async deleteNode(id: string): Promise<void> { public async deleteNoun(id: string): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
@ -420,32 +420,32 @@ export class FileSystemStorage implements StorageAdapter {
/** /**
* Save an edge to storage * Save an edge to storage
*/ */
public async saveEdge(edge: Edge): Promise<void> { public async saveVerb(verb: GraphVerb): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
// Convert connections Map to a serializable format // Convert connections Map to a serializable format
const serializableEdge = { const serializableEdge = {
...edge, ...verb,
connections: this.mapToObject(edge.connections, (set) => Array.from(set)) connections: this.mapToObject(verb.connections, (set) => Array.from(set as Set<string>))
} }
const filePath = path.join(this.edgesDir, `${edge.id}.json`) const filePath = path.join(this.edgesDir, `${verb.id}.json`)
await fs.promises.writeFile( await fs.promises.writeFile(
filePath, filePath,
JSON.stringify(serializableEdge, null, 2), JSON.stringify(serializableEdge, null, 2),
'utf8' 'utf8'
) )
} catch (error) { } catch (error) {
console.error(`Failed to save edge ${edge.id}:`, error) console.error(`Failed to save edge ${verb.id}:`, error)
throw new Error(`Failed to save edge ${edge.id}: ${error}`) throw new Error(`Failed to save edge ${verb.id}: ${error}`)
} }
} }
/** /**
* Get an edge from storage * Get an edge from storage
*/ */
public async getEdge(id: string): Promise<Edge | null> { public async getVerb(id: string): Promise<GraphVerb | null> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
@ -486,7 +486,7 @@ export class FileSystemStorage implements StorageAdapter {
/** /**
* Get all edges from storage * Get all edges from storage
*/ */
public async getAllEdges(): Promise<Edge[]> { public async getAllVerbs(): Promise<GraphVerb[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
@ -495,11 +495,11 @@ export class FileSystemStorage implements StorageAdapter {
.filter((file: string) => file.endsWith('.json')) .filter((file: string) => file.endsWith('.json'))
.map((file: string) => { .map((file: string) => {
const id = path.basename(file, '.json') const id = path.basename(file, '.json')
return this.getEdge(id) return this.getVerb(id)
}) })
const edges = await Promise.all(edgePromises) const edges = await Promise.all(edgePromises)
return edges.filter((edge): edge is Edge => edge !== null) return edges.filter((edge): edge is GraphVerb => edge !== null)
} catch (error) { } catch (error) {
console.error('Failed to get all edges:', error) console.error('Failed to get all edges:', error)
throw new Error(`Failed to get all edges: ${error}`) throw new Error(`Failed to get all edges: ${error}`)
@ -509,11 +509,11 @@ export class FileSystemStorage implements StorageAdapter {
/** /**
* Get edges by source node ID * Get edges by source node ID
*/ */
public async getEdgesBySource(sourceId: string): Promise<Edge[]> { public async getVerbsBySource(sourceId: string): Promise<GraphVerb[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
const allEdges = await this.getAllEdges() const allEdges = await this.getAllVerbs()
return allEdges.filter(edge => edge.sourceId === sourceId) return allEdges.filter(edge => edge.sourceId === sourceId)
} catch (error) { } catch (error) {
console.error(`Failed to get edges by source ${sourceId}:`, error) console.error(`Failed to get edges by source ${sourceId}:`, error)
@ -524,11 +524,11 @@ export class FileSystemStorage implements StorageAdapter {
/** /**
* Get edges by target node ID * Get edges by target node ID
*/ */
public async getEdgesByTarget(targetId: string): Promise<Edge[]> { public async getVerbsByTarget(targetId: string): Promise<GraphVerb[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
const allEdges = await this.getAllEdges() const allEdges = await this.getAllVerbs()
return allEdges.filter(edge => edge.targetId === targetId) return allEdges.filter(edge => edge.targetId === targetId)
} catch (error) { } catch (error) {
console.error(`Failed to get edges by target ${targetId}:`, error) console.error(`Failed to get edges by target ${targetId}:`, error)
@ -539,11 +539,11 @@ export class FileSystemStorage implements StorageAdapter {
/** /**
* Get edges by type * Get edges by type
*/ */
public async getEdgesByType(type: string): Promise<Edge[]> { public async getVerbsByType(type: string): Promise<GraphVerb[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
const allEdges = await this.getAllEdges() const allEdges = await this.getAllVerbs()
return allEdges.filter(edge => edge.type === type) return allEdges.filter(edge => edge.type === type)
} catch (error) { } catch (error) {
console.error(`Failed to get edges by type ${type}:`, error) console.error(`Failed to get edges by type ${type}:`, error)
@ -554,7 +554,7 @@ export class FileSystemStorage implements StorageAdapter {
/** /**
* Delete an edge from storage * Delete an edge from storage
*/ */
public async deleteEdge(id: string): Promise<void> { public async deleteVerb(id: string): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {

220
src/storage/opfsStorage.ts
View file

@ -3,7 +3,11 @@
* Provides persistent storage for the vector database using the Origin Private File System API * Provides persistent storage for the vector database using the Origin Private File System API
*/ */
import { Edge, HNSWNode, StorageAdapter } from '../coreTypes.js' import { GraphVerb, HNSWNoun, StorageAdapter } from '../coreTypes.js'
// Type aliases for compatibility
type HNSWNode = HNSWNoun;
type Edge = GraphVerb;
// Directory and file names // Directory and file names
const ROOT_DIR = 'opfs-vector-db' const ROOT_DIR = 'opfs-vector-db'
@ -165,42 +169,42 @@ export class OPFSStorage implements StorageAdapter {
} }
/** /**
* Save a node to storage * Save a noun to storage
*/ */
public async saveNode(node: HNSWNode): Promise<void> { public async saveNoun(noun: HNSWNoun): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
// Convert connections Map to a serializable format // Convert connections Map to a serializable format
const serializableNode = { const serializableNoun = {
...node, ...noun,
connections: this.mapToObject(node.connections, (set) => connections: this.mapToObject(noun.connections, (set) =>
Array.from(set) Array.from(set as Set<string>)
) )
} }
// Get the appropriate directory based on the node's metadata // Get the appropriate directory based on the noun's metadata
const nodeDir = await this.getNodeDirectory(node.id) const nounDir = await this.getNodeDirectory(noun.id)
// Create or get the file for this node // Create or get the file for this noun
const fileHandle = await nodeDir.getFileHandle(node.id, { const fileHandle = await nounDir.getFileHandle(noun.id, {
create: true create: true
}) })
// Write the node data to the file // Write the noun data to the file
const writable = await fileHandle.createWritable() const writable = await fileHandle.createWritable()
await writable.write(JSON.stringify(serializableNode)) await writable.write(JSON.stringify(serializableNoun))
await writable.close() await writable.close()
} catch (error) { } catch (error) {
console.error(`Failed to save node ${node.id}:`, error) console.error(`Failed to save noun ${noun.id}:`, error)
throw new Error(`Failed to save node ${node.id}: ${error}`) throw new Error(`Failed to save noun ${noun.id}: ${error}`)
} }
} }
/** /**
* Get a node from storage * Get a noun from storage
*/ */
public async getNode(id: string): Promise<HNSWNode | null> { public async getNoun(id: string): Promise<HNSWNoun | null> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
@ -301,15 +305,15 @@ export class OPFSStorage implements StorageAdapter {
} }
/** /**
* Get nodes by noun type * Get nouns by noun type
* @param nounType The noun type to filter by * @param nounType The noun type to filter by
* @returns Promise that resolves to an array of nodes of the specified noun type * @returns Promise that resolves to an array of nouns of the specified noun type
*/ */
public async getNodesByNounType(nounType: string): Promise<HNSWNode[]> { public async getNounsByNounType(nounType: string): Promise<HNSWNoun[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
const nodes: HNSWNode[] = [] const nouns: HNSWNoun[] = []
// Determine the directory based on the noun type // Determine the directory based on the noun type
let dir: FileSystemDirectoryHandle let dir: FileSystemDirectoryHandle
@ -358,7 +362,7 @@ export class OPFSStorage implements StorageAdapter {
connections.set(Number(level), new Set(nodeIds as string[])) connections.set(Number(level), new Set(nodeIds as string[]))
} }
nodes.push({ nouns.push({
id: data.id, id: data.id,
vector: data.vector, vector: data.vector,
connections connections
@ -372,17 +376,17 @@ export class OPFSStorage implements StorageAdapter {
console.warn(`Failed to read directory for noun type ${nounType}:`, dirError) console.warn(`Failed to read directory for noun type ${nounType}:`, dirError)
} }
return nodes return nouns
} catch (error) { } catch (error) {
console.error(`Failed to get nodes for noun type ${nounType}:`, error) console.error(`Failed to get nouns for noun type ${nounType}:`, error)
throw new Error(`Failed to get nodes for noun type ${nounType}: ${error}`) throw new Error(`Failed to get nouns for noun type ${nounType}: ${error}`)
} }
} }
/** /**
* Get all nodes from storage * Get all nouns from storage
*/ */
public async getAllNodes(): Promise<HNSWNode[]> { public async getAllNouns(): Promise<HNSWNoun[]> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
@ -398,42 +402,42 @@ export class OPFSStorage implements StorageAdapter {
] ]
// Run searches in parallel for all noun types // Run searches in parallel for all noun types
const nodePromises = nounTypes.map(nounType => this.getNodesByNounType(nounType)) const nounPromises = nounTypes.map(nounType => this.getNounsByNounType(nounType))
const nodeArrays = await Promise.all(nodePromises) const nounArrays = await Promise.all(nounPromises)
// Combine all results // Combine all results
const allNodes: HNSWNode[] = [] const allNouns: HNSWNoun[] = []
for (const nodes of nodeArrays) { for (const nouns of nounArrays) {
allNodes.push(...nodes) allNouns.push(...nouns)
} }
return allNodes return allNouns
} catch (error) { } catch (error) {
console.error('Failed to get all nodes:', error) console.error('Failed to get all nouns:', error)
throw new Error(`Failed to get all nodes: ${error}`) throw new Error(`Failed to get all nouns: ${error}`)
} }
} }
/** /**
* Delete a node from storage * Delete a noun from storage
*/ */
public async deleteNode(id: string): Promise<void> { public async deleteNoun(id: string): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
// Get the appropriate directory based on the node's metadata // Get the appropriate directory based on the noun's metadata
const nodeDir = await this.getNodeDirectory(id) const nounDir = await this.getNodeDirectory(id)
try { try {
// Try to delete the node from the appropriate directory // Try to delete the noun from the appropriate directory
await nodeDir.removeEntry(id) await nounDir.removeEntry(id)
return // Node deleted successfully return // Noun deleted successfully
} catch (dirError) { } catch (dirError) {
// If the file doesn't exist in the expected directory, try the default directory // If the file doesn't exist in the expected directory, try the default directory
if (nodeDir !== this.defaultDir) { if (nounDir !== this.defaultDir) {
try { try {
await this.defaultDir!.removeEntry(id) await this.defaultDir!.removeEntry(id)
return // Node deleted successfully return // Noun deleted successfully
} catch (defaultDirError) { } catch (defaultDirError) {
// If not found in default directory either, try all noun type directories // If not found in default directory either, try all noun type directories
const directories = [ const directories = [
@ -446,7 +450,7 @@ export class OPFSStorage implements StorageAdapter {
] ]
for (const dir of directories) { for (const dir of directories) {
if (dir === nodeDir) continue // Skip the already checked directory if (dir === nounDir) continue // Skip the already checked directory
try { try {
await dir.removeEntry(id) await dir.removeEntry(id)
@ -464,52 +468,52 @@ export class OPFSStorage implements StorageAdapter {
return return
} }
} catch (error) { } catch (error) {
console.error(`Failed to delete node ${id}:`, error) console.error(`Failed to delete noun ${id}:`, error)
throw new Error(`Failed to delete node ${id}: ${error}`) throw new Error(`Failed to delete noun ${id}: ${error}`)
} }
} }
/** /**
* Save an edge to storage * Save a verb to storage
*/ */
public async saveEdge(edge: Edge): Promise<void> { public async saveVerb(verb: GraphVerb): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
// Convert connections Map to a serializable format // Convert connections Map to a serializable format
const serializableEdge = { const serializableVerb = {
...edge, ...verb,
connections: this.mapToObject(edge.connections, (set) => connections: this.mapToObject(verb.connections, (set) =>
Array.from(set) Array.from(set as Set<string>)
) )
} }
// Create or get the file for this edge // Create or get the file for this verb
const fileHandle = await this.edgesDir!.getFileHandle(edge.id, { const fileHandle = await this.edgesDir!.getFileHandle(verb.id, {
create: true create: true
}) })
// Write the edge data to the file // Write the verb data to the file
const writable = await fileHandle.createWritable() const writable = await fileHandle.createWritable()
await writable.write(JSON.stringify(serializableEdge)) await writable.write(JSON.stringify(serializableVerb))
await writable.close() await writable.close()
} catch (error) { } catch (error) {
console.error(`Failed to save edge ${edge.id}:`, error) console.error(`Failed to save verb ${verb.id}:`, error)
throw new Error(`Failed to save edge ${edge.id}: ${error}`) throw new Error(`Failed to save verb ${verb.id}: ${error}`)
} }
} }
/** /**
* Get an edge from storage * Get a verb from storage
*/ */
public async getEdge(id: string): Promise<Edge | null> { public async getVerb(id: string): Promise<GraphVerb | null> {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
// Get the file handle for this edge // Get the file handle for this verb
const fileHandle = await this.edgesDir!.getFileHandle(id) const fileHandle = await this.edgesDir!.getFileHandle(id)
// Read the edge data from the file // Read the verb data from the file
const file = await fileHandle.getFile() const file = await fileHandle.getFile()
const text = await file.text() const text = await file.text()
const data = JSON.parse(text) const data = JSON.parse(text)
@ -536,8 +540,8 @@ export class OPFSStorage implements StorageAdapter {
return null return null
} }
console.error(`Failed to get edge ${id}:`, error) console.error(`Failed to get verb ${id}:`, error)
throw new Error(`Failed to get edge ${id}: ${error}`) throw new Error(`Failed to get verb ${id}: ${error}`)
} }
} }
@ -556,7 +560,7 @@ export class OPFSStorage implements StorageAdapter {
// Iterate through all keys and get the corresponding edges // Iterate through all keys and get the corresponding edges
for await (const name of keys) { for await (const name of keys) {
const edge = await this.getEdge(name) const edge = await this.getVerb(name)
if (edge) { if (edge) {
edges.push(edge) edges.push(edge)
} }
@ -569,6 +573,13 @@ export class OPFSStorage implements StorageAdapter {
} }
} }
/**
* Get all verbs from storage (alias for getAllEdges)
*/
public async getAllVerbs(): Promise<GraphVerb[]> {
return this.getAllEdges();
}
/** /**
* Delete an edge from storage * Delete an edge from storage
*/ */
@ -586,6 +597,13 @@ export class OPFSStorage implements StorageAdapter {
} }
} }
/**
* Delete a verb from storage (alias for deleteEdge)
*/
public async deleteVerb(id: string): Promise<void> {
return this.deleteEdge(id);
}
/** /**
* Get edges by source node ID * Get edges by source node ID
*/ */
@ -601,6 +619,13 @@ export class OPFSStorage implements StorageAdapter {
} }
} }
/**
* Get verbs by source node ID (alias for getEdgesBySource)
*/
public async getVerbsBySource(sourceId: string): Promise<GraphVerb[]> {
return this.getEdgesBySource(sourceId);
}
/** /**
* Get edges by target node ID * Get edges by target node ID
*/ */
@ -616,6 +641,13 @@ export class OPFSStorage implements StorageAdapter {
} }
} }
/**
* Get verbs by target node ID (alias for getEdgesByTarget)
*/
public async getVerbsByTarget(targetId: string): Promise<GraphVerb[]> {
return this.getEdgesByTarget(targetId);
}
/** /**
* Get edges by type * Get edges by type
*/ */
@ -631,6 +663,13 @@ export class OPFSStorage implements StorageAdapter {
} }
} }
/**
* Get verbs by type (alias for getEdgesByType)
*/
public async getVerbsByType(type: string): Promise<GraphVerb[]> {
return this.getEdgesByType(type);
}
/** /**
* Save metadata for a node * Save metadata for a node
*/ */
@ -949,6 +988,55 @@ export class MemoryStorage implements StorageAdapter {
this.nodes.set(DEFAULT_DIR, new Map()) this.nodes.set(DEFAULT_DIR, new Map())
} }
// Alias methods to match StorageAdapter interface
public async saveNoun(noun: HNSWNoun): Promise<void> {
return this.saveNode(noun);
}
public async getNoun(id: string): Promise<HNSWNoun | null> {
return this.getNode(id);
}
public async getAllNouns(): Promise<HNSWNoun[]> {
return this.getAllNodes();
}
public async getNounsByNounType(nounType: string): Promise<HNSWNoun[]> {
return this.getNodesByNounType(nounType);
}
public async deleteNoun(id: string): Promise<void> {
return this.deleteNode(id);
}
public async saveVerb(verb: GraphVerb): Promise<void> {
return this.saveEdge(verb);
}
public async getVerb(id: string): Promise<GraphVerb | null> {
return this.getEdge(id);
}
public async getAllVerbs(): Promise<GraphVerb[]> {
return this.getAllEdges();
}
public async getVerbsBySource(sourceId: string): Promise<GraphVerb[]> {
return this.getEdgesBySource(sourceId);
}
public async getVerbsByTarget(targetId: string): Promise<GraphVerb[]> {
return this.getEdgesByTarget(targetId);
}
public async getVerbsByType(type: string): Promise<GraphVerb[]> {
return this.getEdgesByType(type);
}
public async deleteVerb(id: string): Promise<void> {
return this.deleteEdge(id);
}
public async init(): Promise<void> { public async init(): Promise<void> {
// Nothing to initialize for in-memory storage // Nothing to initialize for in-memory storage
} }

59
src/storage/s3CompatibleStorage.ts
View file

@ -1,4 +1,8 @@
import { Edge, HNSWNode, StorageAdapter } from '../coreTypes.js' import { GraphVerb, HNSWNoun, StorageAdapter } from '../coreTypes.js'
// Type aliases for compatibility
type HNSWNode = HNSWNoun;
type Edge = GraphVerb;
// Constants for S3 bucket prefixes // Constants for S3 bucket prefixes
const NODES_PREFIX = 'nodes/' const NODES_PREFIX = 'nodes/'
@ -59,6 +63,55 @@ export class S3CompatibleStorage implements StorageAdapter {
private s3Client: any // Will be initialized in init() private s3Client: any // Will be initialized in init()
private isInitialized = false private isInitialized = false
// Alias methods to match StorageAdapter interface
public async saveNoun(noun: HNSWNoun): Promise<void> {
return this.saveNode(noun);
}
public async getNoun(id: string): Promise<HNSWNoun | null> {
return this.getNode(id);
}
public async getAllNouns(): Promise<HNSWNoun[]> {
return this.getAllNodes();
}
public async getNounsByNounType(nounType: string): Promise<HNSWNoun[]> {
return this.getNodesByNounType(nounType);
}
public async deleteNoun(id: string): Promise<void> {
return this.deleteNode(id);
}
public async saveVerb(verb: GraphVerb): Promise<void> {
return this.saveEdge(verb);
}
public async getVerb(id: string): Promise<GraphVerb | null> {
return this.getEdge(id);
}
public async getAllVerbs(): Promise<GraphVerb[]> {
return this.getAllEdges();
}
public async getVerbsBySource(sourceId: string): Promise<GraphVerb[]> {
return this.getEdgesBySource(sourceId);
}
public async getVerbsByTarget(targetId: string): Promise<GraphVerb[]> {
return this.getEdgesByTarget(targetId);
}
public async getVerbsByType(type: string): Promise<GraphVerb[]> {
return this.getEdgesByType(type);
}
public async deleteVerb(id: string): Promise<void> {
return this.deleteEdge(id);
}
constructor(options: { constructor(options: {
bucketName: string bucketName: string
accessKeyId: string accessKeyId: string
@ -161,7 +214,7 @@ export class S3CompatibleStorage implements StorageAdapter {
// Convert connections Map to a serializable format // Convert connections Map to a serializable format
const serializableNode = { const serializableNode = {
...node, ...node,
connections: this.mapToObject(node.connections, (set) => Array.from(set)) connections: this.mapToObject(node.connections, (set) => Array.from(set as Set<string>))
} }
// Get the appropriate prefix based on the node's metadata // Get the appropriate prefix based on the node's metadata
@ -529,7 +582,7 @@ export class S3CompatibleStorage implements StorageAdapter {
// Convert connections Map to a serializable format // Convert connections Map to a serializable format
const serializableEdge = { const serializableEdge = {
...edge, ...edge,
connections: this.mapToObject(edge.connections, (set) => Array.from(set)) connections: this.mapToObject(edge.connections, (set) => Array.from(set as Set<string>))
} }
// Import the PutObjectCommand only when needed // Import the PutObjectCommand only when needed

63
test/validateTypes.js Normal file
View file

@ -0,0 +1,63 @@
// Test script to verify type validation in BrainyData
import { BrainyData } from '../src/brainyData.js';
import { NounType, VerbType } from '../src/types/graphTypes.js';
async function testTypeValidation() {
console.log('Testing type validation in BrainyData...');
// Create a new BrainyData instance
const brainy = new BrainyData();
await brainy.init();
console.log('Testing node with valid noun type...');
const validNodeId = await brainy.add([0.1, 0.2, 0.3], {
noun: NounType.Person,
label: 'Test Person'
});
console.log(`Added node with valid noun type: ${validNodeId}`);
console.log('Testing node with invalid noun type...');
const invalidNodeId = await brainy.add([0.4, 0.5, 0.6], {
noun: 'invalid_type',
label: 'Test Invalid'
});
console.log(`Added node with invalid noun type (should be converted to default): ${invalidNodeId}`);
// Get the metadata to verify it was corrected
const invalidNodeMetadata = await brainy.get(invalidNodeId);
console.log('Metadata for node with invalid type:', invalidNodeMetadata);
console.log('Testing edge with valid verb type...');
const validEdgeId = await brainy.addEdge(validNodeId, invalidNodeId, undefined, {
type: VerbType.RelatedTo,
metadata: { label: 'Test Relation' }
});
console.log(`Added edge with valid verb type: ${validEdgeId}`);
console.log('Testing edge with invalid verb type...');
const invalidEdgeId = await brainy.addEdge(validNodeId, invalidNodeId, undefined, {
type: 'invalid_relation',
metadata: { label: 'Test Invalid Relation' }
});
console.log(`Added edge with invalid verb type (should be converted to default): ${invalidEdgeId}`);
// Get the edge to verify it was corrected
const invalidEdge = await brainy.getEdge(invalidEdgeId);
console.log('Edge with invalid type:', invalidEdge);
console.log('Testing updateMetadata with invalid noun type...');
await brainy.updateMetadata(validNodeId, {
noun: 'another_invalid_type',
label: 'Updated Test'
});
// Get the metadata to verify it was corrected
const updatedMetadata = await brainy.get(validNodeId);
console.log('Updated metadata (should have corrected noun type):', updatedMetadata);
console.log('All tests completed.');
}
testTypeValidation().catch(error => {
console.error('Test failed:', error);
});