brainy/README.md

Soulcraft Brainy

A combined Graph and Vector database that runs in browsers and Node.js environments. Brainy provides efficient vector search capabilities with persistent storage options.

What is Brainy?

Brainy is a lightweight database that combines:

• Vector search (for similarity-based retrieval)
• Graph relationships (for structured data connections)
• Persistent storage (works across sessions)

It's designed to work seamlessly in both browser and server environments.

How It Works

Brainy combines three key technologies:

1. Vector Embeddings: Converts data (text, images, etc.) into numerical vectors that capture semantic meaning
2. HNSW Algorithm: Enables fast similarity search through a hierarchical graph structure
3. Persistent Storage: Uses the best available storage option for your environment:
   • Browser: Origin Private File System (OPFS)
   • Node.js: File system
   • Server: S3-compatible storage (optional)
   • Fallback: In-memory storage

What Can You Use It For?

• Semantic Search: Find content based on meaning, not just keywords
• Recommendation Systems: Suggest similar items based on vector similarity
• Knowledge Graphs: Build connected data structures with relationships
• Data Organization: Automatically categorize and connect related information
• AI Applications: Store and retrieve embeddings for machine learning models

Installation

npm install @soulcraft/brainy

Basic Usage

import {BrainyData} from '@soulcraft/brainy';

// Create and initialize the database
const db = new BrainyData();
await db.init();

// Add text data (automatically converted to vectors)
const catId = await db.add("Cats are independent pets", {type: 'animal'});
const dogId = await db.add("Dogs are loyal companions", {type: 'animal'});

// Search for similar content
const results = await db.searchText("feline pets", 2);
console.log(results);
// Returns items similar to "feline pets" with their similarity scores

// Add relationships between items
await db.addEdge(catId, dogId, {type: 'related_to'});

// Retrieve data
const cat = await db.get(catId);
console.log(cat);

// Update metadata
await db.updateMetadata(catId, {type: 'animal', size: 'small'});

// Delete data
await db.delete(dogId);

Key Features

• Automatic Vectorization: Converts text and data to vector embeddings
• Fast Similarity Search: Uses HNSW algorithm for efficient retrieval
• Cross-Platform: Works in browsers, Node.js, and server environments
• Persistent Storage: Multiple storage options (OPFS, filesystem, cloud)
• Graph Capabilities: Create relationships between data points
• TypeScript Support: Fully typed API with generics
• Flexible Configuration: Customize distance functions, embedding models, and more

Advanced Usage

Custom Embedding

import {BrainyData, createSimpleEmbeddingFunction} from '@soulcraft/brainy';

// Use a custom embedding function (faster but less accurate)
const db = new BrainyData({
    embeddingFunction: createSimpleEmbeddingFunction()
});
await db.init();

// Directly embed text to vectors
const vector = await db.embed("Some text to convert to a vector");

Configuration Options

import {BrainyData, euclideanDistance} from '@soulcraft/brainy';

// Configure with custom options
const db = new BrainyData({
    // Use Euclidean distance instead of default cosine distance
    distanceFunction: euclideanDistance,

    // HNSW index configuration for search performance
    hnsw: {
        M: 16,              // Max connections per node
        efConstruction: 200, // Construction candidate list size
        efSearch: 50,       // Search candidate list size
    },

    // Storage configuration
    storage: {
        requestPersistentStorage: true,
        // Uncomment to use cloud storage:
        // s3Storage: {
        //   bucketName: 'your-bucket',
        //   accessKeyId: 'your-key',
        //   secretAccessKey: 'your-secret',
        //   region: 'us-east-1'
        // }
    }
});

API Reference

Core Methods

// Initialize the database
await db.init();

// Add data (automatically vectorized)
const id = await db.add(textOrVector, metadata);

// Search by vector or text
const results = await db.search(vectorOrText, numResults);
const textResults = await db.searchText("query text", numResults);

// Manage data
const item = await db.get(id);
await db.updateMetadata(id, newMetadata);
await db.delete(id);

// Graph relationships
await db.addEdge(sourceId, targetId, metadata);
const edges = await db.getAllEdges();

// Database management
await db.clear();
const size = db.size();
const status = await db.status();

Distance Functions

• cosineDistance (default)
• euclideanDistance
• manhattanDistance
• dotProductDistance

Embedding Options

• Default: TensorFlow Universal Sentence Encoder (high quality)
• Alternative: Simple character-based embedding (faster)

Extensions

Brainy includes an augmentation system for extending functionality:

• Memory Augmentations: Different storage backends
• Sense Augmentations: Process raw data
• Cognition Augmentations: Reasoning and inference
• Dialog Augmentations: Natural language processing
• Perception Augmentations: Data interpretation and/or visualization
• Activation Augmentations: Trigger actions

For detailed documentation on extensions, see the API docs.

Examples

The repository includes several examples:

• Web demo: examples/demo.html
• Basic usage: examples/basicUsage.js
• Custom storage: examples/customStorage.js
• Memory augmentations: examples/memoryAugmentationExample.js

Browser Compatibility

Works in all modern browsers:

• Chrome 86+
• Edge 86+
• Opera 72+
• Chrome for Android 86+

For browsers without OPFS support, falls back to in-memory storage.

Requirements

• Node.js >= 18.0.0

License

MIT