soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat(core): enhance addVerb functionality with auto-creation of missing nouns

Browse source 
- Added `autoCreateMissingNouns` and `missingNounMetadata` options to the `addVerb` method in `brainyData.ts`, enabling automatic creation of missing source or target nouns.
- Improved error handling and logging for auto-creation failures, ensuring better feedback during runtime.
- Reformatted existing code for storage options validation to improve readability and maintain consistency.

**Purpose**: Simplify the addition of relationships by automating the process for non-existing nouns and enhance developer experience with better error handling and logging.
This commit is contained in:
David Snelling 2025-07-22 16:01:48 -07:00
parent 4abf8070f2
commit c666c6780b
2 changed files with 388 additions and 301 deletions
Download patch file Download diff file Expand all files Collapse all files

606
README.md
View file

@ -62,14 +62,15 @@ GitHub Pages that showcases Brainy's main features.
npm install @soulcraft/brainy npm install @soulcraft/brainy
``` ```
TensorFlow.js packages are included as bundled dependencies and will be automatically installed without any additional configuration. TensorFlow.js packages are included as bundled dependencies and will be automatically installed without any additional
configuration.
## 🏁 Quick Start ## 🏁 Quick Start
Brainy uses a unified build that automatically adapts to your environment (Node.js, browser, or serverless): Brainy uses a unified build that automatically adapts to your environment (Node.js, browser, or serverless):
```typescript ```typescript
import { BrainyData, NounType, VerbType } from '@soulcraft/brainy' import {BrainyData, NounType, VerbType} from '@soulcraft/brainy'
// Create and initialize the database // Create and initialize the database
const db = new BrainyData() const db = new BrainyData()
@ -77,13 +78,13 @@ await db.init()
// Add data (automatically converted to vectors) // Add data (automatically converted to vectors)
const catId = await db.add("Cats are independent pets", { const catId = await db.add("Cats are independent pets", {
noun: NounType.Thing, noun: NounType.Thing,
category: 'animal' category: 'animal'
}) })
const dogId = await db.add("Dogs are loyal companions", { const dogId = await db.add("Dogs are loyal companions", {
noun: NounType.Thing, noun: NounType.Thing,
category: 'animal' category: 'animal'
}) })
// Search for similar items // Search for similar items
@ -92,8 +93,8 @@ console.log(results)
// Add a relationship between items // Add a relationship between items
await db.addVerb(catId, dogId, { await db.addVerb(catId, dogId, {
verb: VerbType.RelatedTo, verb: VerbType.RelatedTo,
description: 'Both are common household pets' description: 'Both are common household pets'
}) })
``` ```
@ -101,28 +102,30 @@ await db.addVerb(catId, dogId, {
```typescript ```typescript
// Standard import - automatically adapts to any environment // Standard import - automatically adapts to any environment
import { BrainyData } from '@soulcraft/brainy' import {BrainyData} from '@soulcraft/brainy'
// Minified version for production // Minified version for production
import { BrainyData } from '@soulcraft/brainy/min' import {BrainyData} from '@soulcraft/brainy/min'
``` ```
> **Note**: The CLI functionality is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size of the main package. Install it globally with `npm install -g @soulcraft/brainy-cli` to use the command-line interface. > **Note**: The CLI functionality is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size
> of the main package. Install it globally with `npm install -g @soulcraft/brainy-cli` to use the command-line
> interface.
### Browser Usage ### Browser Usage
```html ```html
<script type="module"> <script type="module">
// Use local files instead of CDN // Use local files instead of CDN
import { BrainyData } from './dist/unified.js' import {BrainyData} from './dist/unified.js'
// Or minified version // Or minified version
// import { BrainyData } from './dist/unified.min.js' // import { BrainyData } from './dist/unified.min.js'
const db = new BrainyData() const db = new BrainyData()
await db.init() await db.init()
// ... // ...
</script> </script>
``` ```
@ -277,13 +280,13 @@ The pipeline runs automatically when you:
```typescript ```typescript
// Add data (runs embedding → indexing → storage) // Add data (runs embedding → indexing → storage)
const id = await db.add("Your text data here", { metadata }) const id = await db.add("Your text data here", {metadata})
// Search (runs embedding → similarity search) // Search (runs embedding → similarity search)
const results = await db.searchText("Your query here", 5) const results = await db.searchText("Your query here", 5)
// Connect entities (runs graph construction → storage) // Connect entities (runs graph construction → storage)
await db.addVerb(sourceId, targetId, { verb: VerbType.RelatedTo }) await db.addVerb(sourceId, targetId, {verb: VerbType.RelatedTo})
``` ```
Using the CLI: Using the CLI:
@ -408,7 +411,8 @@ brainy visualize --root <id> --depth 3
### Using the CLI in Your Code ### Using the CLI in Your Code
The CLI functionality is available as a separate package `@soulcraft/brainy-cli`. If you need CLI functionality in your application, install the CLI package: The CLI functionality is available as a separate package `@soulcraft/brainy-cli`. If you need CLI functionality in your
application, install the CLI package:
```bash ```bash
npm install @soulcraft/brainy-cli npm install @soulcraft/brainy-cli
@ -466,7 +470,7 @@ const status = await db.status()
const backupData = await db.backup() const backupData = await db.backup()
// Restore data into the database // Restore data into the database
const restoreResult = await db.restore(backupData, { clearExisting: true }) const restoreResult = await db.restore(backupData, {clearExisting: true})
``` ```
### Working with Nouns (Entities) ### Working with Nouns (Entities)
@ -474,25 +478,25 @@ const restoreResult = await db.restore(backupData, { clearExisting: true })
```typescript ```typescript
// Add a noun (automatically vectorized) // Add a noun (automatically vectorized)
const id = await db.add(textOrVector, { const id = await db.add(textOrVector, {
noun: NounType.Thing, noun: NounType.Thing,
// other metadata... // other metadata...
}) })
// Add multiple nouns in parallel (with multithreading and batch embedding) // Add multiple nouns in parallel (with multithreading and batch embedding)
const ids = await db.addBatch([ const ids = await db.addBatch([
{ {
vectorOrData: "First item to add", vectorOrData: "First item to add",
metadata: { noun: NounType.Thing, category: 'example' } metadata: {noun: NounType.Thing, category: 'example'}
}, },
{ {
vectorOrData: "Second item to add", vectorOrData: "Second item to add",
metadata: { noun: NounType.Thing, category: 'example' } metadata: {noun: NounType.Thing, category: 'example'}
}, },
// More items... // More items...
], { ], {
forceEmbed: false, forceEmbed: false,
concurrency: 4, // Control the level of parallelism (default: 4) concurrency: 4, // Control the level of parallelism (default: 4)
batchSize: 50 // Control the number of items to process in a single batch (default: 50) batchSize: 50 // Control the number of items to process in a single batch (default: 50)
}) })
// Retrieve a noun // Retrieve a noun
@ -500,8 +504,8 @@ const noun = await db.get(id)
// Update noun metadata // Update noun metadata
await db.updateMetadata(id, { await db.updateMetadata(id, {
noun: NounType.Thing, noun: NounType.Thing,
// updated metadata... // updated metadata...
}) })
// Delete a noun // Delete a noun
@ -520,8 +524,21 @@ const thingNouns = await db.searchByNounTypes([NounType.Thing], numResults)
```typescript ```typescript
// Add a relationship between nouns // Add a relationship between nouns
await db.addVerb(sourceId, targetId, { await db.addVerb(sourceId, targetId, {
verb: VerbType.RelatedTo, verb: VerbType.RelatedTo,
// other metadata... // other metadata...
})
// Add a relationship with auto-creation of missing nouns
// This is useful when the target noun might not exist yet
await db.addVerb(sourceId, targetId, {
verb: VerbType.RelatedTo,
// Enable auto-creation of missing nouns
autoCreateMissingNouns: true,
// Optional metadata for auto-created nouns
missingNounMetadata: {
noun: NounType.Concept,
description: 'Auto-created noun'
}
}) })
// Get all relationships // Get all relationships
@ -549,20 +566,20 @@ await db.deleteVerb(verbId)
```typescript ```typescript
import { import {
BrainyData, BrainyData,
createTensorFlowEmbeddingFunction, createTensorFlowEmbeddingFunction,
createThreadedEmbeddingFunction createThreadedEmbeddingFunction
} from '@soulcraft/brainy' } from '@soulcraft/brainy'
// Use the standard TensorFlow Universal Sentence Encoder embedding function // Use the standard TensorFlow Universal Sentence Encoder embedding function
const db = new BrainyData({ const db = new BrainyData({
embeddingFunction: createTensorFlowEmbeddingFunction() embeddingFunction: createTensorFlowEmbeddingFunction()
}) })
await db.init() await db.init()
// Or use the threaded embedding function for better performance // Or use the threaded embedding function for better performance
const threadedDb = new BrainyData({ const threadedDb = new BrainyData({
embeddingFunction: createThreadedEmbeddingFunction() embeddingFunction: createThreadedEmbeddingFunction()
}) })
await threadedDb.init() await threadedDb.init()
@ -603,42 +620,42 @@ Brainy includes comprehensive multithreading support to improve performance acro
7. **Automatic Environment Detection**: Adapts to browser (Web Workers) and Node.js (Worker Threads) environments 7. **Automatic Environment Detection**: Adapts to browser (Web Workers) and Node.js (Worker Threads) environments
```typescript ```typescript
import { BrainyData, euclideanDistance } from '@soulcraft/brainy' import {BrainyData, euclideanDistance} from '@soulcraft/brainy'
// Configure with custom options // Configure with custom options
const db = new BrainyData({ const db = new BrainyData({
// Use Euclidean distance instead of default cosine distance // Use Euclidean distance instead of default cosine distance
distanceFunction: euclideanDistance, distanceFunction: euclideanDistance,
// HNSW index configuration for search performance // HNSW index configuration for search performance
hnsw: { hnsw: {
M: 16, // Max connections per noun M: 16, // Max connections per noun
efConstruction: 200, // Construction candidate list size efConstruction: 200, // Construction candidate list size
efSearch: 50, // Search candidate list size efSearch: 50, // Search candidate list size
}, },
// Performance optimization options // Performance optimization options
performance: { performance: {
useParallelization: true, // Enable multithreaded search operations useParallelization: true, // Enable multithreaded search operations
}, },
// Noun and Verb type validation // Noun and Verb type validation
typeValidation: { typeValidation: {
enforceNounTypes: true, // Validate noun types against NounType enum enforceNounTypes: true, // Validate noun types against NounType enum
enforceVerbTypes: true, // Validate verb types against VerbType enum enforceVerbTypes: true, // Validate verb types against VerbType enum
}, },
// Storage configuration // Storage configuration
storage: { storage: {
requestPersistentStorage: true, requestPersistentStorage: true,
// Example configuration for cloud storage (replace with your own values): // Example configuration for cloud storage (replace with your own values):
// s3Storage: { // s3Storage: {
// bucketName: 'your-s3-bucket-name', // bucketName: 'your-s3-bucket-name',
// region: 'your-aws-region' // region: 'your-aws-region'
// // Credentials should be provided via environment variables // // Credentials should be provided via environment variables
// // AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY // // AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
// } // }
} }
}) })
``` ```
@ -652,34 +669,34 @@ hybrid approach:
3. **Memory-Efficient Indexing** - Optimizes memory usage for large-scale vector collections 3. **Memory-Efficient Indexing** - Optimizes memory usage for large-scale vector collections
```typescript ```typescript
import { BrainyData } from '@soulcraft/brainy' import {BrainyData} from '@soulcraft/brainy'
// Configure with optimized HNSW index for large datasets // Configure with optimized HNSW index for large datasets
const db = new BrainyData({ const db = new BrainyData({
hnswOptimized: { hnswOptimized: {
// Standard HNSW parameters // Standard HNSW parameters
M: 16, // Max connections per noun M: 16, // Max connections per noun
efConstruction: 200, // Construction candidate list size efConstruction: 200, // Construction candidate list size
efSearch: 50, // Search candidate list size efSearch: 50, // Search candidate list size
// Memory threshold in bytes - when exceeded, will use disk-based approach // Memory threshold in bytes - when exceeded, will use disk-based approach
memoryThreshold: 1024 * 1024 * 1024, // 1GB default threshold memoryThreshold: 1024 * 1024 * 1024, // 1GB default threshold
// Product quantization settings for dimensionality reduction // Product quantization settings for dimensionality reduction
productQuantization: { productQuantization: {
enabled: true, // Enable product quantization enabled: true, // Enable product quantization
numSubvectors: 16, // Number of subvectors to split the vector into numSubvectors: 16, // Number of subvectors to split the vector into
numCentroids: 256 // Number of centroids per subvector numCentroids: 256 // Number of centroids per subvector
},
// Whether to use disk-based storage for the index
useDiskBasedIndex: true // Enable disk-based storage
}, },
// Whether to use disk-based storage for the index // Storage configuration (required for disk-based index)
useDiskBasedIndex: true // Enable disk-based storage storage: {
}, requestPersistentStorage: true
}
// Storage configuration (required for disk-based index)
storage: {
requestPersistentStorage: true
}
}) })
// The optimized index automatically adapts based on dataset size: // The optimized index automatically adapts based on dataset size:
@ -744,24 +761,24 @@ Brainy's restore functionality can handle:
```typescript ```typescript
// Restore data with all options // Restore data with all options
const restoreResult = await db.restore(backupData, { const restoreResult = await db.restore(backupData, {
clearExisting: true // Whether to clear existing data before restore clearExisting: true // Whether to clear existing data before restore
}) })
// Import sparse data (without vectors) // Import sparse data (without vectors)
// Vectors will be automatically created using the embedding function // Vectors will be automatically created using the embedding function
const sparseData = { const sparseData = {
nouns: [ nouns: [
{ {
id: '123', id: '123',
// No vector field - will be created during import // No vector field - will be created during import
metadata: { metadata: {
noun: 'Thing', noun: 'Thing',
text: 'This text will be used to generate a vector' text: 'This text will be used to generate a vector'
} }
} }
], ],
verbs: [], verbs: [],
version: '1.0.0' version: '1.0.0'
} }
const sparseImportResult = await db.importSparseData(sparseData) const sparseImportResult = await db.importSparseData(sparseData)
@ -808,82 +825,82 @@ boilerplate:
```typescript ```typescript
import { import {
createMemoryAugmentation, createMemoryAugmentation,
createConduitAugmentation, createConduitAugmentation,
createSenseAugmentation, createSenseAugmentation,
addWebSocketSupport, addWebSocketSupport,
executeStreamlined, executeStreamlined,
processStaticData, processStaticData,
processStreamingData, processStreamingData,
createPipeline createPipeline
} from '@soulcraft/brainy' } from '@soulcraft/brainy'
// Create a memory augmentation with minimal code // Create a memory augmentation with minimal code
const memoryAug = createMemoryAugmentation({ const memoryAug = createMemoryAugmentation({
name: 'simple-memory', name: 'simple-memory',
description: 'A simple in-memory storage augmentation', description: 'A simple in-memory storage augmentation',
autoRegister: true, autoRegister: true,
autoInitialize: true, autoInitialize: true,
// Implement only the methods you need // Implement only the methods you need
storeData: async (key, data) => { storeData: async (key, data) => {
// Your implementation here // Your implementation here
return { return {
success: true, success: true,
data: true data: true
} }
}, },
retrieveData: async (key) => { retrieveData: async (key) => {
// Your implementation here // Your implementation here
return { return {
success: true, success: true,
data: { example: 'data', key } data: {example: 'data', key}
}
} }
}
}) })
// Add WebSocket support to any augmentation // Add WebSocket support to any augmentation
const wsAugmentation = addWebSocketSupport(memoryAug, { const wsAugmentation = addWebSocketSupport(memoryAug, {
connectWebSocket: async (url) => { connectWebSocket: async (url) => {
// Your implementation here // Your implementation here
return { return {
connectionId: 'ws-1', connectionId: 'ws-1',
url, url,
status: 'connected' status: 'connected'
}
} }
}
}) })
// Process static data through a pipeline // Process static data through a pipeline
const result = await processStaticData( const result = await processStaticData(
'Input data', 'Input data',
[ [
{ {
augmentation: senseAug, augmentation: senseAug,
method: 'processRawData', method: 'processRawData',
transformArgs: (data) => [data, 'text'] transformArgs: (data) => [data, 'text']
}, },
{ {
augmentation: memoryAug, augmentation: memoryAug,
method: 'storeData', method: 'storeData',
transformArgs: (data) => ['processed-data', data] transformArgs: (data) => ['processed-data', data]
} }
] ]
) )
// Create a reusable pipeline // Create a reusable pipeline
const pipeline = createPipeline([ const pipeline = createPipeline([
{ {
augmentation: senseAug, augmentation: senseAug,
method: 'processRawData', method: 'processRawData',
transformArgs: (data) => [data, 'text'] transformArgs: (data) => [data, 'text']
}, },
{ {
augmentation: memoryAug, augmentation: memoryAug,
method: 'storeData', method: 'storeData',
transformArgs: (data) => ['processed-data', data] transformArgs: (data) => ['processed-data', data]
} }
]) ])
// Use the pipeline // Use the pipeline
@ -891,11 +908,11 @@ const result = await pipeline('New input data')
// Dynamically load augmentations at runtime // Dynamically load augmentations at runtime
const loadedAugmentations = await loadAugmentationModule( const loadedAugmentations = await loadAugmentationModule(
import('./my-augmentations.js'), import('./my-augmentations.js'),
{ {
autoRegister: true, autoRegister: true,
autoInitialize: true autoInitialize: true
} }
) )
``` ```
@ -909,60 +926,61 @@ The simplified augmentation system provides:
#### WebSocket Augmentation Types #### WebSocket Augmentation Types
Brainy exports several WebSocket augmentation types that can be used by augmentation creators to add WebSocket capabilities to their augmentations: Brainy exports several WebSocket augmentation types that can be used by augmentation creators to add WebSocket
capabilities to their augmentations:
```typescript ```typescript
import { import {
// Base WebSocket support interface // Base WebSocket support interface
IWebSocketSupport, IWebSocketSupport,
// Combined WebSocket augmentation types // Combined WebSocket augmentation types
IWebSocketSenseAugmentation, IWebSocketSenseAugmentation,
IWebSocketConduitAugmentation, IWebSocketConduitAugmentation,
IWebSocketCognitionAugmentation, IWebSocketCognitionAugmentation,
IWebSocketMemoryAugmentation, IWebSocketMemoryAugmentation,
IWebSocketPerceptionAugmentation, IWebSocketPerceptionAugmentation,
IWebSocketDialogAugmentation, IWebSocketDialogAugmentation,
IWebSocketActivationAugmentation, IWebSocketActivationAugmentation,
// Function to add WebSocket support to any augmentation // Function to add WebSocket support to any augmentation
addWebSocketSupport addWebSocketSupport
} from '@soulcraft/brainy' } from '@soulcraft/brainy'
// Example: Creating a typed WebSocket-enabled sense augmentation // Example: Creating a typed WebSocket-enabled sense augmentation
const mySenseAug = createSenseAugmentation({ const mySenseAug = createSenseAugmentation({
name: 'my-sense', name: 'my-sense',
processRawData: async (data, dataType) => { processRawData: async (data, dataType) => {
// Implementation // Implementation
return { return {
success: true, success: true,
data: { nouns: [], verbs: [] } data: {nouns: [], verbs: []}
}
} }
}
}) as IWebSocketSenseAugmentation }) as IWebSocketSenseAugmentation
// Add WebSocket support // Add WebSocket support
addWebSocketSupport(mySenseAug, { addWebSocketSupport(mySenseAug, {
connectWebSocket: async (url) => { connectWebSocket: async (url) => {
// WebSocket implementation // WebSocket implementation
return { return {
connectionId: 'ws-1', connectionId: 'ws-1',
url, url,
status: 'connected' status: 'connected'
}
},
sendWebSocketMessage: async (connectionId, data) => {
// Send message implementation
},
onWebSocketMessage: async (connectionId, callback) => {
// Register callback implementation
},
offWebSocketMessage: async (connectionId, callback) => {
// Remove callback implementation
},
closeWebSocket: async (connectionId, code, reason) => {
// Close connection implementation
} }
},
sendWebSocketMessage: async (connectionId, data) => {
// Send message implementation
},
onWebSocketMessage: async (connectionId, callback) => {
// Register callback implementation
},
offWebSocketMessage: async (connectionId, callback) => {
// Remove callback implementation
},
closeWebSocket: async (connectionId, code, reason) => {
// Close connection implementation
}
}) })
// Now mySenseAug has both sense augmentation methods and WebSocket methods // Now mySenseAug has both sense augmentation methods and WebSocket methods
@ -970,7 +988,8 @@ await mySenseAug.processRawData('data', 'text')
await mySenseAug.connectWebSocket('wss://example.com') await mySenseAug.connectWebSocket('wss://example.com')
``` ```
These WebSocket augmentation types combine the base augmentation interfaces with the `IWebSocketSupport` interface, providing type safety and autocompletion for augmentations with WebSocket capabilities. These WebSocket augmentation types combine the base augmentation interfaces with the `IWebSocketSupport` interface,
providing type safety and autocompletion for augmentations with WebSocket capabilities.
### Model Control Protocol (MCP) ### Model Control Protocol (MCP)
@ -999,13 +1018,13 @@ everywhere.
Brainy automatically detects the environment it's running in: Brainy automatically detects the environment it's running in:
```typescript ```typescript
import { environment } from '@soulcraft/brainy' import {environment} from '@soulcraft/brainy'
// Check which environment we're running in // Check which environment we're running in
console.log(`Running in ${ console.log(`Running in ${
environment.isBrowser ? 'browser' : environment.isBrowser ? 'browser' :
environment.isNode ? 'Node.js' : environment.isNode ? 'Node.js' :
'serverless/unknown' 'serverless/unknown'
} environment`) } environment`)
``` ```
@ -1034,7 +1053,6 @@ Works in all modern browsers:
For browsers without OPFS support, falls back to in-memory storage. For browsers without OPFS support, falls back to in-memory storage.
## Related Projects ## Related Projects
- **[Cartographer](https://github.com/sodal-project/cartographer)** - A companion project that provides standardized - **[Cartographer](https://github.com/sodal-project/cartographer)** - A companion project that provides standardized
@ -1079,9 +1097,9 @@ You can use the conduit augmentations to sync Brainy instances:
```typescript ```typescript
import { import {
BrainyData, BrainyData,
pipeline, pipeline,
createConduitAugmentation createConduitAugmentation
} from '@soulcraft/brainy' } from '@soulcraft/brainy'
// Create and initialize the database // Create and initialize the database
@ -1097,36 +1115,36 @@ pipeline.register(wsConduit)
// Connect to another Brainy instance (server or browser) // Connect to another Brainy instance (server or browser)
// Replace the example URL below with your actual WebSocket server URL // Replace the example URL below with your actual WebSocket server URL
const connectionResult = await pipeline.executeConduitPipeline( const connectionResult = await pipeline.executeConduitPipeline(
'establishConnection', 'establishConnection',
['wss://example-websocket-server.com/brainy-sync', { protocols: 'brainy-sync' }] ['wss://example-websocket-server.com/brainy-sync', {protocols: 'brainy-sync'}]
) )
if (connectionResult[0] && (await connectionResult[0]).success) { if (connectionResult[0] && (await connectionResult[0]).success) {
const connection = (await connectionResult[0]).data const connection = (await connectionResult[0]).data
// Read data from the remote instance // Read data from the remote instance
const readResult = await pipeline.executeConduitPipeline( const readResult = await pipeline.executeConduitPipeline(
'readData', 'readData',
[{ connectionId: connection.connectionId, query: { type: 'getAllNouns' } }] [{connectionId: connection.connectionId, query: {type: 'getAllNouns'}}]
) )
// Process and add the received data to the local instance // Process and add the received data to the local instance
if (readResult[0] && (await readResult[0]).success) { if (readResult[0] && (await readResult[0]).success) {
const remoteNouns = (await readResult[0]).data const remoteNouns = (await readResult[0]).data
for (const noun of remoteNouns) { for (const noun of remoteNouns) {
await db.add(noun.vector, noun.metadata) await db.add(noun.vector, noun.metadata)
}
} }
}
// Set up real-time sync by monitoring the stream // Set up real-time sync by monitoring the stream
await wsConduit.monitorStream(connection.connectionId, async (data) => { await wsConduit.monitorStream(connection.connectionId, async (data) => {
// Handle incoming data (e.g., new nouns, verbs, updates) // Handle incoming data (e.g., new nouns, verbs, updates)
if (data.type === 'newNoun') { if (data.type === 'newNoun') {
await db.add(data.vector, data.metadata) await db.add(data.vector, data.metadata)
} else if (data.type === 'newVerb') { } else if (data.type === 'newVerb') {
await db.addVerb(data.sourceId, data.targetId, data.vector, data.options) await db.addVerb(data.sourceId, data.targetId, data.vector, data.options)
} }
}) })
} }
``` ```
@ -1134,9 +1152,9 @@ if (connectionResult[0] && (await connectionResult[0]).success) {
```typescript ```typescript
import { import {
BrainyData, BrainyData,
pipeline, pipeline,
createConduitAugmentation createConduitAugmentation
} from '@soulcraft/brainy' } from '@soulcraft/brainy'
// Create and initialize the database // Create and initialize the database
@ -1152,48 +1170,48 @@ pipeline.register(webrtcConduit)
// Connect to a peer using a signaling server // Connect to a peer using a signaling server
// Replace the example values below with your actual configuration // Replace the example values below with your actual configuration
const connectionResult = await pipeline.executeConduitPipeline( const connectionResult = await pipeline.executeConduitPipeline(
'establishConnection', 'establishConnection',
[ [
'peer-id-to-connect-to', // Replace with actual peer ID 'peer-id-to-connect-to', // Replace with actual peer ID
{ {
signalServerUrl: 'wss://example-signal-server.com', // Replace with your signal server signalServerUrl: 'wss://example-signal-server.com', // Replace with your signal server
localPeerId: 'my-local-peer-id', // Replace with your local peer ID localPeerId: 'my-local-peer-id', // Replace with your local peer ID
iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] // Public STUN server iceServers: [{urls: 'stun:stun.l.google.com:19302'}] // Public STUN server
} }
] ]
) )
if (connectionResult[0] && (await connectionResult[0]).success) { if (connectionResult[0] && (await connectionResult[0]).success) {
const connection = (await connectionResult[0]).data const connection = (await connectionResult[0]).data
// Set up real-time sync by monitoring the stream // Set up real-time sync by monitoring the stream
await webrtcConduit.monitorStream(connection.connectionId, async (data) => { await webrtcConduit.monitorStream(connection.connectionId, async (data) => {
// Handle incoming data (e.g., new nouns, verbs, updates) // Handle incoming data (e.g., new nouns, verbs, updates)
if (data.type === 'newNoun') { if (data.type === 'newNoun') {
await db.add(data.vector, data.metadata) await db.add(data.vector, data.metadata)
} else if (data.type === 'newVerb') { } else if (data.type === 'newVerb') {
await db.addVerb(data.sourceId, data.targetId, data.vector, data.options) await db.addVerb(data.sourceId, data.targetId, data.vector, data.options)
}
})
// When adding new data locally, also send to the peer
const nounId = await db.add("New data to sync", { noun: "Thing" })
// Send the new noun to the peer
await pipeline.executeConduitPipeline(
'writeData',
[
{
connectionId: connection.connectionId,
data: {
type: 'newNoun',
id: nounId,
vector: (await db.get(nounId)).vector,
metadata: (await db.get(nounId)).metadata
} }
} })
]
) // When adding new data locally, also send to the peer
const nounId = await db.add("New data to sync", {noun: "Thing"})
// Send the new noun to the peer
await pipeline.executeConduitPipeline(
'writeData',
[
{
connectionId: connection.connectionId,
data: {
type: 'newNoun',
id: nounId,
vector: (await db.get(nounId)).vector,
metadata: (await db.get(nounId)).metadata
}
}
]
)
} }
``` ```
@ -1203,39 +1221,39 @@ Brainy supports searching a server-hosted instance from a browser, storing resul
searches against the local instance: searches against the local instance:
```typescript ```typescript
import { BrainyData } from '@soulcraft/brainy' import {BrainyData} from '@soulcraft/brainy'
// Create and initialize the database with remote server configuration // Create and initialize the database with remote server configuration
// Replace the example URL below with your actual Brainy server URL // Replace the example URL below with your actual Brainy server URL
const db = new BrainyData({ const db = new BrainyData({
remoteServer: { remoteServer: {
url: 'wss://example-brainy-server.com/ws', // Replace with your server URL url: 'wss://example-brainy-server.com/ws', // Replace with your server URL
protocols: 'brainy-sync', protocols: 'brainy-sync',
autoConnect: true // Connect automatically during initialization autoConnect: true // Connect automatically during initialization
} }
}) })
await db.init() await db.init()
// Or connect manually after initialization // Or connect manually after initialization
if (!db.isConnectedToRemoteServer()) { if (!db.isConnectedToRemoteServer()) {
// Replace the example URL below with your actual Brainy server URL // Replace the example URL below with your actual Brainy server URL
await db.connectToRemoteServer('wss://example-brainy-server.com/ws', 'brainy-sync') await db.connectToRemoteServer('wss://example-brainy-server.com/ws', 'brainy-sync')
} }
// Search the remote server (results are stored locally) // Search the remote server (results are stored locally)
const remoteResults = await db.searchText('machine learning', 5, { searchMode: 'remote' }) const remoteResults = await db.searchText('machine learning', 5, {searchMode: 'remote'})
// Search the local database (includes previously stored results) // Search the local database (includes previously stored results)
const localResults = await db.searchText('machine learning', 5, { searchMode: 'local' }) const localResults = await db.searchText('machine learning', 5, {searchMode: 'local'})
// Perform a combined search (local first, then remote if needed) // Perform a combined search (local first, then remote if needed)
const combinedResults = await db.searchText('neural networks', 5, { searchMode: 'combined' }) const combinedResults = await db.searchText('neural networks', 5, {searchMode: 'combined'})
// Add data to both local and remote instances // Add data to both local and remote instances
const id = await db.addToBoth('Deep learning is a subset of machine learning', { const id = await db.addToBoth('Deep learning is a subset of machine learning', {
noun: 'Concept', noun: 'Concept',
category: 'AI', category: 'AI',
tags: ['deep learning', 'neural networks'] tags: ['deep learning', 'neural networks']
}) })
// Clean up when done (this also cleans up worker pools) // Clean up when done (this also cleans up worker pools)

75
src/brainyData.ts
View file

@ -341,7 +341,7 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
// All required fields are present, keep s3Storage as is // All required fields are present, keep s3Storage as is
} else { } else {
// Missing required fields, remove s3Storage to avoid type errors // Missing required fields, remove s3Storage to avoid type errors
const { s3Storage, ...rest } = storageOptions const {s3Storage, ...rest} = storageOptions
storageOptions = rest storageOptions = rest
console.warn('Ignoring s3Storage configuration due to missing required fields') console.warn('Ignoring s3Storage configuration due to missing required fields')
} }
@ -1253,6 +1253,22 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
/** /**
* Add a verb between two nouns * Add a verb between two nouns
* If metadata is provided and vector is not, the metadata will be vectorized using the embedding function * If metadata is provided and vector is not, the metadata will be vectorized using the embedding function
*
* @param sourceId ID of the source noun
* @param targetId ID of the target noun
* @param vector Optional vector for the verb
* @param options Additional options:
* - type: Type of the verb
* - weight: Weight of the verb
* - metadata: Metadata for the verb
* - forceEmbed: Force using the embedding function for metadata even if vector is provided
* - id: Optional ID to use instead of generating a new one
* - autoCreateMissingNouns: Automatically create missing nouns if they don't exist
* - missingNounMetadata: Metadata to use when auto-creating missing nouns
*
* @returns The ID of the added verb
*
* @throws Error if source or target nouns don't exist and autoCreateMissingNouns is false or auto-creation fails
*/ */
public async addVerb( public async addVerb(
sourceId: string, sourceId: string,
@ -1264,6 +1280,8 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
metadata?: any metadata?: any
forceEmbed?: boolean // Force using the embedding function for metadata even if vector is provided forceEmbed?: boolean // Force using the embedding function for metadata even if vector is provided
id?: string // Optional ID to use instead of generating a new one id?: string // Optional ID to use instead of generating a new one
autoCreateMissingNouns?: boolean // Automatically create missing nouns
missingNounMetadata?: any // Metadata to use when auto-creating missing nouns
} = {} } = {}
): Promise<string> { ): Promise<string> {
await this.ensureInitialized() await this.ensureInitialized()
@ -1273,8 +1291,59 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
try { try {
// Check if source and target nouns exist // Check if source and target nouns exist
const sourceNoun = this.index.getNouns().get(sourceId) let sourceNoun = this.index.getNouns().get(sourceId)
const targetNoun = this.index.getNouns().get(targetId) let targetNoun = this.index.getNouns().get(targetId)
// Auto-create missing nouns if option is enabled
if (!sourceNoun && options.autoCreateMissingNouns) {
try {
// Create a placeholder vector for the missing noun
const placeholderVector = new Array(this._dimensions).fill(0)
// Add metadata if provided
const metadata = options.missingNounMetadata || {
autoCreated: true,
createdAt: new Date().toISOString(),
noun: NounType.Concept
}
// Add the missing noun
await this.add(placeholderVector, metadata, {id: sourceId})
// Get the newly created noun
sourceNoun = this.index.getNouns().get(sourceId)
console.warn(`Auto-created missing source noun with ID ${sourceId}`)
} catch (createError) {
console.error(`Failed to auto-create source noun with ID ${sourceId}:`, createError)
throw new Error(`Failed to auto-create source noun with ID ${sourceId}: ${createError}`)
}
}
if (!targetNoun && options.autoCreateMissingNouns) {
try {
// Create a placeholder vector for the missing noun
const placeholderVector = new Array(this._dimensions).fill(0)
// Add metadata if provided
const metadata = options.missingNounMetadata || {
autoCreated: true,
createdAt: new Date().toISOString(),
noun: NounType.Concept
}
// Add the missing noun
await this.add(placeholderVector, metadata, {id: targetId})
// Get the newly created noun
targetNoun = this.index.getNouns().get(targetId)
console.warn(`Auto-created missing target noun with ID ${targetId}`)
} catch (createError) {
console.error(`Failed to auto-create target noun with ID ${targetId}:`, createError)
throw new Error(`Failed to auto-create target noun with ID ${targetId}: ${createError}`)
}
}
if (!sourceNoun) { if (!sourceNoun) {
throw new Error(`Source noun with ID ${sourceId} not found`) throw new Error(`Source noun with ID ${sourceId} not found`)