soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

docs: simplify README.md and rebrand to "Brainy"

Browse source 
Rebranded from "Soulcraft Brainy" to "Brainy" throughout `README.md`. Improved readability by reformatting long paragraphs and restructuring lists for better clarity.
This commit is contained in:
David Snelling 2025-06-06 08:10:55 -07:00
parent 4c0922cb94
commit 783b40083f
1 changed files with 109 additions and 89 deletions
Download patch file Download diff file Expand all files Collapse all files

198
README.md
View file

@ -1,6 +1,6 @@
<div align="center"> <div align="center">
# 🧠 Soulcraft Brainy # 🧠 Brainy
[![Version](https://img.shields.io/badge/version-0.7.4-blue.svg)](https://www.npmjs.com/package/@soulcraft/brainy) [![Version](https://img.shields.io/badge/version-0.7.4-blue.svg)](https://www.npmjs.com/package/@soulcraft/brainy)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
@ -12,29 +12,39 @@
## ✨ Overview ## ✨ Overview
Say hello to Brainy, your new favorite data sidekick! 🎉 Brainy combines the power of vector search with graph relationships in a lightweight, cross-platform database that's as smart as it is fun to use. Whether you're building AI applications, recommendation systems, or knowledge graphs, Brainy provides the tools you need to store, connect, and retrieve your data intelligently. Say hello to Brainy, your new favorite data sidekick! 🎉 Brainy combines the power of vector search with graph
relationships in a lightweight, cross-platform database that's as smart as it is fun to use. Whether you're building AI
applications, recommendation systems, or knowledge graphs, Brainy provides the tools you need to store, connect, and
retrieve your data intelligently.
What makes Brainy special? It intelligently adapts to you and your environment! Like a chameleon with a PhD, Brainy automatically detects your platform, adjusts its storage strategy, and optimizes performance based on your usage patterns. The more you use it, the smarter it gets - learning from your data to provide increasingly relevant results and connections. What makes Brainy special? It intelligently adapts to you and your environment! Like a chameleon with a PhD, Brainy
automatically detects your platform, adjusts its storage strategy, and optimizes performance based on your usage
patterns. The more you use it, the smarter it gets - learning from your data to provide increasingly relevant results
and connections.
### 🚀 Key Features ### 🚀 Key Features
- **Vector Search** - Find semantically similar content using embeddings (like having ESP for your data!) - **Vector Search** - Find semantically similar content using embeddings (like having ESP for your data!)
- **Graph Relationships** - Connect data with meaningful relationships (your data's social network) - **Graph Relationships** - Connect data with meaningful relationships (your data's social network)
- **Streaming Pipeline** - Process data in real-time as it flows through the system (like a data waterslide!) - **Streaming Pipeline** - Process data in real-time as it flows through the system (like a data waterslide!)
- **Extensible Augmentations** - Customize and extend functionality with pluggable components (LEGO blocks for your data!) - **Extensible Augmentations** - Customize and extend functionality with pluggable components (LEGO blocks for your
data!)
- **Adaptive Intelligence** - Automatically optimizes for your environment and usage patterns - **Adaptive Intelligence** - Automatically optimizes for your environment and usage patterns
- **Cross-Platform** - Works everywhere you do: browsers, Node.js, and server environments - **Cross-Platform** - Works everywhere you do: browsers, Node.js, and server environments
- **Persistent Storage** - Data persists across sessions and scales to any size (no memory loss here, even for elephant-sized data!) - **Persistent Storage** - Data persists across sessions and scales to any size (no memory loss here, even for
elephant-sized data!)
- **TypeScript Support** - Fully typed API with generics (for those who like their code tidy) - **TypeScript Support** - Fully typed API with generics (for those who like their code tidy)
- **CLI Tools** - Powerful command-line interface for data management (command line wizardry) - **CLI Tools** - Powerful command-line interface for data management (command line wizardry)
## 📊 What Can You Build? (The Fun Stuff!) ## 📊 What Can You Build? (The Fun Stuff!)
- **Semantic Search Engines** - Find content based on meaning, not just keywords (mind-reading for your data!) - **Semantic Search Engines** - Find content based on meaning, not just keywords (mind-reading for your data!)
- **Recommendation Systems** - Suggest similar items based on vector similarity (like a friend who really gets your taste) - **Recommendation Systems** - Suggest similar items based on vector similarity (like a friend who really gets your
taste)
- **Knowledge Graphs** - Build connected data structures with relationships (your data's family tree) - **Knowledge Graphs** - Build connected data structures with relationships (your data's family tree)
- **AI Applications** - Store and retrieve embeddings for machine learning models (brain food for your AI) - **AI Applications** - Store and retrieve embeddings for machine learning models (brain food for your AI)
- **Data Organization Tools** - Automatically categorize and connect related information (like having a librarian in your code) - **Data Organization Tools** - Automatically categorize and connect related information (like having a librarian in
your code)
- **Adaptive Experiences** - Create applications that learn and evolve with your users (digital chameleons!) - **Adaptive Experiences** - Create applications that learn and evolve with your users (digital chameleons!)
## 🔧 Installation ## 🔧 Installation
@ -48,7 +58,7 @@ npm install @soulcraft/brainy --legacy-peer-deps
## 🏁 Quick Start ## 🏁 Quick Start
```typescript ```typescript
import { BrainyData, NounType, VerbType } from '@soulcraft/brainy'; import {BrainyData, NounType, VerbType} from '@soulcraft/brainy';
// Create and initialize the database // Create and initialize the database
const db = new BrainyData(); const db = new BrainyData();
@ -81,130 +91,136 @@ await db.addVerb(catId, dogId, {
Brainy combines four key technologies to create its adaptive intelligence: Brainy combines four key technologies to create its adaptive intelligence:
1. **Vector Embeddings** - Converts data (text, images, etc.) into numerical vectors that capture semantic meaning (translating your data into brain-speak!) 1. **Vector Embeddings** - Converts data (text, images, etc.) into numerical vectors that capture semantic meaning (
2. **HNSW Algorithm** - Enables fast similarity search through a hierarchical graph structure (like a super-efficient treasure map for your data) translating your data into brain-speak!)
2. **HNSW Algorithm** - Enables fast similarity search through a hierarchical graph structure (like a super-efficient
treasure map for your data)
3. **Adaptive Environment Detection** - Automatically senses your platform and optimizes accordingly: 3. **Adaptive Environment Detection** - Automatically senses your platform and optimizes accordingly:
- Adjusts performance parameters based on available resources - Adjusts performance parameters based on available resources
- Learns from query patterns to optimize future searches - Learns from query patterns to optimize future searches
- Tunes itself for your specific use cases the more you use it - Tunes itself for your specific use cases the more you use it
4. **Intelligent Storage Selection** - Uses the best available storage option for your environment, scaling effortlessly to any data size (from bytes to petabytes!): 4. **Intelligent Storage Selection** - Uses the best available storage option for your environment, scaling effortlessly
- Browser: Origin Private File System (OPFS) to any data size (from bytes to petabytes!):
- Node.js: File system - Browser: Origin Private File System (OPFS)
- Server: S3-compatible storage (optional) - Node.js: File system
- Fallback: In-memory storage - Server: S3-compatible storage (optional)
- Automatically migrates between storage types as needed! - Fallback: In-memory storage
- Automatically migrates between storage types as needed!
## 🚀 The Brainy Pipeline (Data's Wild Ride!) ## 🚀 The Brainy Pipeline (Data's Wild Ride!)
Brainy's data processing pipeline transforms raw data into searchable, connected knowledge that gets smarter over time. Here's how the magic happens: Brainy's data processing pipeline transforms raw data into searchable, connected knowledge that gets smarter over time.
Here's how the magic happens:
``` ```
Raw Data → Embedding → Vector Storage → Graph Connections → Adaptive Learning → Query & Retrieval Raw Data → Embedding → Vector Storage → Graph Connections → Adaptive Learning → Query & Retrieval
``` ```
Each time data flows through this pipeline, Brainy learns a little more about your usage patterns and environment, making future operations even faster and more relevant! Each time data flows through this pipeline, Brainy learns a little more about your usage patterns and environment,
making future operations even faster and more relevant!
### 🔄 Pipeline Stages (The Journey of Your Data) ### 🔄 Pipeline Stages (The Journey of Your Data)
1. **Data Ingestion** 🍽️ 1. **Data Ingestion** 🍽️
- Raw text or pre-computed vectors enter the pipeline (dinner time for data!) - Raw text or pre-computed vectors enter the pipeline (dinner time for data!)
- Data is validated and prepared for processing (washing hands before eating) - Data is validated and prepared for processing (washing hands before eating)
2. **Embedding Generation** 🧠 2. **Embedding Generation** 🧠
- Text is transformed into numerical vectors using embedding models (language → math magic) - Text is transformed into numerical vectors using embedding models (language → math magic)
- Choose between TensorFlow Universal Sentence Encoder (high quality) or Simple Embedding (faster) - Choose between TensorFlow Universal Sentence Encoder (high quality) or Simple Embedding (faster)
- Custom embedding functions can be plugged in for specialized domains (bring your own secret sauce) - Custom embedding functions can be plugged in for specialized domains (bring your own secret sauce)
3. **Vector Indexing** 🔍 3. **Vector Indexing** 🔍
- Vectors are indexed using the HNSW algorithm (filing your data in the brain cabinet) - Vectors are indexed using the HNSW algorithm (filing your data in the brain cabinet)
- Hierarchical structure enables lightning-fast similarity search (express lanes for your queries) - Hierarchical structure enables lightning-fast similarity search (express lanes for your queries)
- Configurable parameters for precision vs. performance tradeoffs (dial in your perfect balance) - Configurable parameters for precision vs. performance tradeoffs (dial in your perfect balance)
4. **Graph Construction** 🕸️ 4. **Graph Construction** 🕸️
- Nouns (entities) become nodes in the knowledge graph (data gets its own social network) - Nouns (entities) become nodes in the knowledge graph (data gets its own social network)
- Verbs (relationships) connect related entities (making friends and connections) - Verbs (relationships) connect related entities (making friends and connections)
- Typed relationships add semantic meaning to connections (not just friends, but BFFs) - Typed relationships add semantic meaning to connections (not just friends, but BFFs)
5. **Adaptive Learning** 🌱 5. **Adaptive Learning** 🌱
- Analyzes usage patterns to optimize future operations (gets to know your habits) - Analyzes usage patterns to optimize future operations (gets to know your habits)
- Tunes performance parameters based on your environment (adapts to your digital home) - Tunes performance parameters based on your environment (adapts to your digital home)
- Adjusts search strategies based on query history (learns what you're really looking for) - Adjusts search strategies based on query history (learns what you're really looking for)
- Becomes more efficient and relevant the more you use it (like a good friendship) - Becomes more efficient and relevant the more you use it (like a good friendship)
6. **Intelligent Storage** 💾 6. **Intelligent Storage** 💾
- Data is saved using the optimal storage for your environment (finds the coziest home for your data) - Data is saved using the optimal storage for your environment (finds the coziest home for your data)
- Automatic selection between OPFS, filesystem, S3, or memory (no manual configuration needed!) - Automatic selection between OPFS, filesystem, S3, or memory (no manual configuration needed!)
- Migrates between storage types as your application's needs evolve (moves houses without you noticing) - Migrates between storage types as your application's needs evolve (moves houses without you noticing)
- Scales effortlessly from tiny datasets to massive data collections (from ant-sized to elephant-sized data, no problem!) - Scales effortlessly from tiny datasets to massive data collections (from ant-sized to elephant-sized data, no
- Configurable storage adapters for custom persistence needs (design your own dream data home) problem!)
- Configurable storage adapters for custom persistence needs (design your own dream data home)
### 🧩 Augmentation Types ### 🧩 Augmentation Types
Brainy uses a powerful augmentation system to extend functionality. Augmentations are processed in the following order: Brainy uses a powerful augmentation system to extend functionality. Augmentations are processed in the following order:
1. **SENSE** 👁️ 1. **SENSE** 👁️
- Ingests and processes raw, unstructured data into nouns and verbs - Ingests and processes raw, unstructured data into nouns and verbs
- Handles text, images, audio streams, and other input formats - Handles text, images, audio streams, and other input formats
- Example: Converting raw text into structured entities - Example: Converting raw text into structured entities
2. **MEMORY** 💾 2. **MEMORY** 💾
- Provides storage capabilities for data in different formats - Provides storage capabilities for data in different formats
- Manages persistence across sessions - Manages persistence across sessions
- Example: Storing vectors in OPFS or filesystem - Example: Storing vectors in OPFS or filesystem
3. **COGNITION** 🧠 3. **COGNITION** 🧠
- Enables advanced reasoning, inference, and logical operations - Enables advanced reasoning, inference, and logical operations
- Analyzes relationships between entities - Analyzes relationships between entities
- Example: Inferring new connections between existing data - Example: Inferring new connections between existing data
4. **CONDUIT** 🔌 4. **CONDUIT** 🔌
- Establishes high-bandwidth channels for structured data exchange - Establishes high-bandwidth channels for structured data exchange
- Connects with external systems - Connects with external systems
- Example: Integrating with third-party APIs - Example: Integrating with third-party APIs
5. **ACTIVATION** ⚡ 5. **ACTIVATION** ⚡
- Initiates actions, responses, or data manipulations - Initiates actions, responses, or data manipulations
- Triggers events based on data changes - Triggers events based on data changes
- Example: Sending notifications when new data is processed - Example: Sending notifications when new data is processed
6. **PERCEPTION** 🔍 6. **PERCEPTION** 🔍
- Interprets, contextualizes, and visualizes identified nouns and verbs - Interprets, contextualizes, and visualizes identified nouns and verbs
- Creates meaningful representations of data - Creates meaningful representations of data
- Example: Generating visualizations of graph relationships - Example: Generating visualizations of graph relationships
7. **DIALOG** 💬 7. **DIALOG** 💬
- Facilitates natural language understanding and generation - Facilitates natural language understanding and generation
- Enables conversational interactions - Enables conversational interactions
- Example: Processing user queries and generating responses - Example: Processing user queries and generating responses
8. **WEBSOCKET** 🌐 8. **WEBSOCKET** 🌐
- Enables real-time communication via WebSockets - Enables real-time communication via WebSockets
- Can be combined with other augmentation types - Can be combined with other augmentation types
- Example: Streaming data processing in real-time - Example: Streaming data processing in real-time
### 🌊 Streaming Data Support ### 🌊 Streaming Data Support
Brainy's pipeline is designed to handle streaming data efficiently: Brainy's pipeline is designed to handle streaming data efficiently:
1. **WebSocket Integration** 🔄 1. **WebSocket Integration** 🔄
- Built-in support for WebSocket connections - Built-in support for WebSocket connections
- Process data as it arrives without blocking - Process data as it arrives without blocking
- Example: `setupWebSocketPipeline(url, dataType, options)` - Example: `setupWebSocketPipeline(url, dataType, options)`
2. **Asynchronous Processing** ⚡ 2. **Asynchronous Processing** ⚡
- Non-blocking architecture for real-time data handling - Non-blocking architecture for real-time data handling
- Parallel processing of incoming streams - Parallel processing of incoming streams
- Example: `createWebSocketHandler(connection, dataType, options)` - Example: `createWebSocketHandler(connection, dataType, options)`
3. **Event-Based Architecture** 📡 3. **Event-Based Architecture** 📡
- Augmentations can listen to data feeds and streams - Augmentations can listen to data feeds and streams
- Real-time updates propagate through the pipeline - Real-time updates propagate through the pipeline
- Example: `listenToFeed(feedUrl, callback)` - Example: `listenToFeed(feedUrl, callback)`
4. **Threaded Execution** 🧵 4. **Threaded Execution** 🧵
- Optional multi-threading for high-performance streaming - Optional multi-threading for high-performance streaming
- Configurable execution modes (SEQUENTIAL, PARALLEL, THREADED) - Configurable execution modes (SEQUENTIAL, PARALLEL, THREADED)
- Example: `executeTypedPipeline(augmentations, method, args, { mode: ExecutionMode.THREADED })` - Example: `executeTypedPipeline(augmentations, method, args, { mode: ExecutionMode.THREADED })`
### 🏃‍♀️ Running the Pipeline ### 🏃‍♀️ Running the Pipeline
@ -212,13 +228,13 @@ The pipeline runs automatically when you:
```typescript ```typescript
// Add data (runs embedding → indexing → storage) // Add data (runs embedding → indexing → storage)
const id = await db.add("Your text data here", { metadata }); const id = await db.add("Your text data here", {metadata});
// Search (runs embedding → similarity search) // Search (runs embedding → similarity search)
const results = await db.searchText("Your query here", 5); const results = await db.searchText("Your query here", 5);
// Connect entities (runs graph construction → storage) // Connect entities (runs graph construction → storage)
await db.addVerb(sourceId, targetId, { verb: VerbType.RelatedTo }); await db.addVerb(sourceId, targetId, {verb: VerbType.RelatedTo});
``` ```
Using the CLI: Using the CLI:
@ -299,6 +315,7 @@ Brainy uses a graph-based data model with two primary concepts:
### Nouns (Entities) ### Nouns (Entities)
The main entities in your data (nodes in the graph): The main entities in your data (nodes in the graph):
- Each noun has a unique ID, vector representation, and metadata - Each noun has a unique ID, vector representation, and metadata
- Nouns can be categorized by type (Person, Place, Thing, Event, Concept, etc.) - Nouns can be categorized by type (Person, Place, Thing, Event, Concept, etc.)
- Nouns are automatically vectorized for similarity search - Nouns are automatically vectorized for similarity search
@ -306,6 +323,7 @@ The main entities in your data (nodes in the graph):
### Verbs (Relationships) ### Verbs (Relationships)
Connections between nouns (edges in the graph): Connections between nouns (edges in the graph):
- Each verb connects a source noun to a target noun - Each verb connects a source noun to a target noun
- Verbs have types that define the relationship (RelatedTo, Controls, Contains, etc.) - Verbs have types that define the relationship (RelatedTo, Controls, Contains, etc.)
- Verbs can have their own metadata to describe the relationship - Verbs can have their own metadata to describe the relationship
@ -349,6 +367,7 @@ npm run cli generate-random-graph --noun-count 20 --verb-count 40
### 🔍 Available Commands ### 🔍 Available Commands
#### Basic Database Operations: #### Basic Database Operations:
- `init` - Initialize a new database - `init` - Initialize a new database
- `add <text> [metadata]` - Add a new noun with text and optional metadata - `add <text> [metadata]` - Add a new noun with text and optional metadata
- `search <query> [limit]` - Search for nouns similar to the query - `search <query> [limit]` - Search for nouns similar to the query
@ -363,18 +382,19 @@ npm run cli generate-random-graph --noun-count 20 --verb-count 40
- `completion-setup` - Setup shell autocomplete - `completion-setup` - Setup shell autocomplete
#### Pipeline and Augmentation Commands: #### Pipeline and Augmentation Commands:
- `list-augmentations` - List all available augmentation types and registered augmentations - `list-augmentations` - List all available augmentation types and registered augmentations
- `augmentation-info <type>` - Get detailed information about a specific augmentation type - `augmentation-info <type>` - Get detailed information about a specific augmentation type
- `test-pipeline [text]` - Test the sequential pipeline with sample data - `test-pipeline [text]` - Test the sequential pipeline with sample data
- `-t, --data-type <type>` - Type of data to process (default: 'text') - `-t, --data-type <type>` - Type of data to process (default: 'text')
- `-m, --mode <mode>` - Execution mode: sequential, parallel, threaded (default: 'sequential') - `-m, --mode <mode>` - Execution mode: sequential, parallel, threaded (default: 'sequential')
- `-s, --stop-on-error` - Stop execution if an error occurs - `-s, --stop-on-error` - Stop execution if an error occurs
- `-v, --verbose` - Show detailed output - `-v, --verbose` - Show detailed output
- `stream-test` - Test streaming data through the pipeline (simulated) - `stream-test` - Test streaming data through the pipeline (simulated)
- `-c, --count <number>` - Number of data items to stream (default: 5) - `-c, --count <number>` - Number of data items to stream (default: 5)
- `-i, --interval <ms>` - Interval between data items in milliseconds (default: 1000) - `-i, --interval <ms>` - Interval between data items in milliseconds (default: 1000)
- `-t, --data-type <type>` - Type of data to process (default: 'text') - `-t, --data-type <type>` - Type of data to process (default: 'text')
- `-v, --verbose` - Show detailed output - `-v, --verbose` - Show detailed output
## 🔌 API Reference ## 🔌 API Reference
@ -453,7 +473,7 @@ await db.deleteVerb(verbId);
### Custom Embedding ### Custom Embedding
```typescript ```typescript
import { BrainyData, createSimpleEmbeddingFunction } from '@soulcraft/brainy'; import {BrainyData, createSimpleEmbeddingFunction} from '@soulcraft/brainy';
// Use a custom embedding function (faster but less accurate) // Use a custom embedding function (faster but less accurate)
const db = new BrainyData({ const db = new BrainyData({
@ -468,7 +488,7 @@ const vector = await db.embed("Some text to convert to a vector");
### Performance Tuning ### Performance Tuning
```typescript ```typescript
import { BrainyData, euclideanDistance } from '@soulcraft/brainy'; import {BrainyData, euclideanDistance} from '@soulcraft/brainy';
// Configure with custom options // Configure with custom options
const db = new BrainyData({ const db = new BrainyData({