soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

feat(core): enhance addVerb functionality with auto-creation of missing nouns

Browse source 
- Added `autoCreateMissingNouns` and `missingNounMetadata` options to the `addVerb` method in `brainyData.ts`, enabling automatic creation of missing source or target nouns.
- Improved error handling and logging for auto-creation failures, ensuring better feedback during runtime.
- Reformatted existing code for storage options validation to improve readability and maintain consistency.

**Purpose**: Simplify the addition of relationships by automating the process for non-existing nouns and enhance developer experience with better error handling and logging.
This commit is contained in:
David Snelling 2025-07-22 16:01:48 -07:00
parent e1865e7808
commit 97db2daa50
2 changed files with 388 additions and 301 deletions
Download patch file Download diff file Expand all files Collapse all files

74
README.md
View file

@ -62,14 +62,15 @@ GitHub Pages that showcases Brainy's main features.
npm install @soulcraft/brainy npm install @soulcraft/brainy
``` ```
TensorFlow.js packages are included as bundled dependencies and will be automatically installed without any additional configuration. TensorFlow.js packages are included as bundled dependencies and will be automatically installed without any additional
configuration.
## 🏁 Quick Start ## 🏁 Quick Start
Brainy uses a unified build that automatically adapts to your environment (Node.js, browser, or serverless): Brainy uses a unified build that automatically adapts to your environment (Node.js, browser, or serverless):
```typescript ```typescript
import { BrainyData, NounType, VerbType } from '@soulcraft/brainy' import {BrainyData, NounType, VerbType} from '@soulcraft/brainy'
// Create and initialize the database // Create and initialize the database
const db = new BrainyData() const db = new BrainyData()
@ -101,13 +102,15 @@ await db.addVerb(catId, dogId, {
```typescript ```typescript
// Standard import - automatically adapts to any environment // Standard import - automatically adapts to any environment
import { BrainyData } from '@soulcraft/brainy' import {BrainyData} from '@soulcraft/brainy'
// Minified version for production // Minified version for production
import { BrainyData } from '@soulcraft/brainy/min' import {BrainyData} from '@soulcraft/brainy/min'
``` ```
> **Note**: The CLI functionality is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size of the main package. Install it globally with `npm install -g @soulcraft/brainy-cli` to use the command-line interface. > **Note**: The CLI functionality is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size
> of the main package. Install it globally with `npm install -g @soulcraft/brainy-cli` to use the command-line
> interface.
### Browser Usage ### Browser Usage
@ -115,7 +118,7 @@ import { BrainyData } from '@soulcraft/brainy/min'
<script type="module"> <script type="module">
// Use local files instead of CDN // Use local files instead of CDN
import { BrainyData } from './dist/unified.js' import {BrainyData} from './dist/unified.js'
// Or minified version // Or minified version
// import { BrainyData } from './dist/unified.min.js' // import { BrainyData } from './dist/unified.min.js'
@ -277,13 +280,13 @@ The pipeline runs automatically when you:
```typescript ```typescript
// Add data (runs embedding → indexing → storage) // Add data (runs embedding → indexing → storage)
const id = await db.add("Your text data here", { metadata }) const id = await db.add("Your text data here", {metadata})
// Search (runs embedding → similarity search) // Search (runs embedding → similarity search)
const results = await db.searchText("Your query here", 5) const results = await db.searchText("Your query here", 5)
// Connect entities (runs graph construction → storage) // Connect entities (runs graph construction → storage)
await db.addVerb(sourceId, targetId, { verb: VerbType.RelatedTo }) await db.addVerb(sourceId, targetId, {verb: VerbType.RelatedTo})
``` ```
Using the CLI: Using the CLI:
@ -408,7 +411,8 @@ brainy visualize --root <id> --depth 3
### Using the CLI in Your Code ### Using the CLI in Your Code
The CLI functionality is available as a separate package `@soulcraft/brainy-cli`. If you need CLI functionality in your application, install the CLI package: The CLI functionality is available as a separate package `@soulcraft/brainy-cli`. If you need CLI functionality in your
application, install the CLI package:
```bash ```bash
npm install @soulcraft/brainy-cli npm install @soulcraft/brainy-cli
@ -466,7 +470,7 @@ const status = await db.status()
const backupData = await db.backup() const backupData = await db.backup()
// Restore data into the database // Restore data into the database
const restoreResult = await db.restore(backupData, { clearExisting: true }) const restoreResult = await db.restore(backupData, {clearExisting: true})
``` ```
### Working with Nouns (Entities) ### Working with Nouns (Entities)
@ -482,11 +486,11 @@ const id = await db.add(textOrVector, {
const ids = await db.addBatch([ const ids = await db.addBatch([
{ {
vectorOrData: "First item to add", vectorOrData: "First item to add",
metadata: { noun: NounType.Thing, category: 'example' } metadata: {noun: NounType.Thing, category: 'example'}
}, },
{ {
vectorOrData: "Second item to add", vectorOrData: "Second item to add",
metadata: { noun: NounType.Thing, category: 'example' } metadata: {noun: NounType.Thing, category: 'example'}
}, },
// More items... // More items...
], { ], {
@ -524,6 +528,19 @@ await db.addVerb(sourceId, targetId, {
// other metadata... // other metadata...
}) })
// Add a relationship with auto-creation of missing nouns
// This is useful when the target noun might not exist yet
await db.addVerb(sourceId, targetId, {
verb: VerbType.RelatedTo,
// Enable auto-creation of missing nouns
autoCreateMissingNouns: true,
// Optional metadata for auto-created nouns
missingNounMetadata: {
noun: NounType.Concept,
description: 'Auto-created noun'
}
})
// Get all relationships // Get all relationships
const verbs = await db.getAllVerbs() const verbs = await db.getAllVerbs()
@ -603,7 +620,7 @@ Brainy includes comprehensive multithreading support to improve performance acro
7. **Automatic Environment Detection**: Adapts to browser (Web Workers) and Node.js (Worker Threads) environments 7. **Automatic Environment Detection**: Adapts to browser (Web Workers) and Node.js (Worker Threads) environments
```typescript ```typescript
import { BrainyData, euclideanDistance } from '@soulcraft/brainy' import {BrainyData, euclideanDistance} from '@soulcraft/brainy'
// Configure with custom options // Configure with custom options
const db = new BrainyData({ const db = new BrainyData({
@ -652,7 +669,7 @@ hybrid approach:
3. **Memory-Efficient Indexing** - Optimizes memory usage for large-scale vector collections 3. **Memory-Efficient Indexing** - Optimizes memory usage for large-scale vector collections
```typescript ```typescript
import { BrainyData } from '@soulcraft/brainy' import {BrainyData} from '@soulcraft/brainy'
// Configure with optimized HNSW index for large datasets // Configure with optimized HNSW index for large datasets
const db = new BrainyData({ const db = new BrainyData({
@ -838,7 +855,7 @@ const memoryAug = createMemoryAugmentation({
// Your implementation here // Your implementation here
return { return {
success: true, success: true,
data: { example: 'data', key } data: {example: 'data', key}
} }
} }
}) })
@ -909,7 +926,8 @@ The simplified augmentation system provides:
#### WebSocket Augmentation Types #### WebSocket Augmentation Types
Brainy exports several WebSocket augmentation types that can be used by augmentation creators to add WebSocket capabilities to their augmentations: Brainy exports several WebSocket augmentation types that can be used by augmentation creators to add WebSocket
capabilities to their augmentations:
```typescript ```typescript
import { import {
@ -936,7 +954,7 @@ const mySenseAug = createSenseAugmentation({
// Implementation // Implementation
return { return {
success: true, success: true,
data: { nouns: [], verbs: [] } data: {nouns: [], verbs: []}
} }
} }
}) as IWebSocketSenseAugmentation }) as IWebSocketSenseAugmentation
@ -970,7 +988,8 @@ await mySenseAug.processRawData('data', 'text')
await mySenseAug.connectWebSocket('wss://example.com') await mySenseAug.connectWebSocket('wss://example.com')
``` ```
These WebSocket augmentation types combine the base augmentation interfaces with the `IWebSocketSupport` interface, providing type safety and autocompletion for augmentations with WebSocket capabilities. These WebSocket augmentation types combine the base augmentation interfaces with the `IWebSocketSupport` interface,
providing type safety and autocompletion for augmentations with WebSocket capabilities.
### Model Control Protocol (MCP) ### Model Control Protocol (MCP)
@ -999,7 +1018,7 @@ everywhere.
Brainy automatically detects the environment it's running in: Brainy automatically detects the environment it's running in:
```typescript ```typescript
import { environment } from '@soulcraft/brainy' import {environment} from '@soulcraft/brainy'
// Check which environment we're running in // Check which environment we're running in
console.log(`Running in ${ console.log(`Running in ${
@ -1034,7 +1053,6 @@ Works in all modern browsers:
For browsers without OPFS support, falls back to in-memory storage. For browsers without OPFS support, falls back to in-memory storage.
## Related Projects ## Related Projects
- **[Cartographer](https://github.com/sodal-project/cartographer)** - A companion project that provides standardized - **[Cartographer](https://github.com/sodal-project/cartographer)** - A companion project that provides standardized
@ -1098,7 +1116,7 @@ pipeline.register(wsConduit)
// Replace the example URL below with your actual WebSocket server URL // Replace the example URL below with your actual WebSocket server URL
const connectionResult = await pipeline.executeConduitPipeline( const connectionResult = await pipeline.executeConduitPipeline(
'establishConnection', 'establishConnection',
['wss://example-websocket-server.com/brainy-sync', { protocols: 'brainy-sync' }] ['wss://example-websocket-server.com/brainy-sync', {protocols: 'brainy-sync'}]
) )
if (connectionResult[0] && (await connectionResult[0]).success) { if (connectionResult[0] && (await connectionResult[0]).success) {
@ -1107,7 +1125,7 @@ if (connectionResult[0] && (await connectionResult[0]).success) {
// Read data from the remote instance // Read data from the remote instance
const readResult = await pipeline.executeConduitPipeline( const readResult = await pipeline.executeConduitPipeline(
'readData', 'readData',
[{ connectionId: connection.connectionId, query: { type: 'getAllNouns' } }] [{connectionId: connection.connectionId, query: {type: 'getAllNouns'}}]
) )
// Process and add the received data to the local instance // Process and add the received data to the local instance
@ -1158,7 +1176,7 @@ const connectionResult = await pipeline.executeConduitPipeline(
{ {
signalServerUrl: 'wss://example-signal-server.com', // Replace with your signal server signalServerUrl: 'wss://example-signal-server.com', // Replace with your signal server
localPeerId: 'my-local-peer-id', // Replace with your local peer ID localPeerId: 'my-local-peer-id', // Replace with your local peer ID
iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] // Public STUN server iceServers: [{urls: 'stun:stun.l.google.com:19302'}] // Public STUN server
} }
] ]
) )
@ -1177,7 +1195,7 @@ if (connectionResult[0] && (await connectionResult[0]).success) {
}) })
// When adding new data locally, also send to the peer // When adding new data locally, also send to the peer
const nounId = await db.add("New data to sync", { noun: "Thing" }) const nounId = await db.add("New data to sync", {noun: "Thing"})
// Send the new noun to the peer // Send the new noun to the peer
await pipeline.executeConduitPipeline( await pipeline.executeConduitPipeline(
@ -1203,7 +1221,7 @@ Brainy supports searching a server-hosted instance from a browser, storing resul
searches against the local instance: searches against the local instance:
```typescript ```typescript
import { BrainyData } from '@soulcraft/brainy' import {BrainyData} from '@soulcraft/brainy'
// Create and initialize the database with remote server configuration // Create and initialize the database with remote server configuration
// Replace the example URL below with your actual Brainy server URL // Replace the example URL below with your actual Brainy server URL
@ -1223,13 +1241,13 @@ if (!db.isConnectedToRemoteServer()) {
} }
// Search the remote server (results are stored locally) // Search the remote server (results are stored locally)
const remoteResults = await db.searchText('machine learning', 5, { searchMode: 'remote' }) const remoteResults = await db.searchText('machine learning', 5, {searchMode: 'remote'})
// Search the local database (includes previously stored results) // Search the local database (includes previously stored results)
const localResults = await db.searchText('machine learning', 5, { searchMode: 'local' }) const localResults = await db.searchText('machine learning', 5, {searchMode: 'local'})
// Perform a combined search (local first, then remote if needed) // Perform a combined search (local first, then remote if needed)
const combinedResults = await db.searchText('neural networks', 5, { searchMode: 'combined' }) const combinedResults = await db.searchText('neural networks', 5, {searchMode: 'combined'})
// Add data to both local and remote instances // Add data to both local and remote instances
const id = await db.addToBoth('Deep learning is a subset of machine learning', { const id = await db.addToBoth('Deep learning is a subset of machine learning', {

75
src/brainyData.ts
View file

@ -341,7 +341,7 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
// All required fields are present, keep s3Storage as is // All required fields are present, keep s3Storage as is
} else { } else {
// Missing required fields, remove s3Storage to avoid type errors // Missing required fields, remove s3Storage to avoid type errors
const { s3Storage, ...rest } = storageOptions const {s3Storage, ...rest} = storageOptions
storageOptions = rest storageOptions = rest
console.warn('Ignoring s3Storage configuration due to missing required fields') console.warn('Ignoring s3Storage configuration due to missing required fields')
} }
@ -1253,6 +1253,22 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
/** /**
* Add a verb between two nouns * Add a verb between two nouns
* If metadata is provided and vector is not, the metadata will be vectorized using the embedding function * If metadata is provided and vector is not, the metadata will be vectorized using the embedding function
*
* @param sourceId ID of the source noun
* @param targetId ID of the target noun
* @param vector Optional vector for the verb
* @param options Additional options:
* - type: Type of the verb
* - weight: Weight of the verb
* - metadata: Metadata for the verb
* - forceEmbed: Force using the embedding function for metadata even if vector is provided
* - id: Optional ID to use instead of generating a new one
* - autoCreateMissingNouns: Automatically create missing nouns if they don't exist
* - missingNounMetadata: Metadata to use when auto-creating missing nouns
*
* @returns The ID of the added verb
*
* @throws Error if source or target nouns don't exist and autoCreateMissingNouns is false or auto-creation fails
*/ */
public async addVerb( public async addVerb(
sourceId: string, sourceId: string,
@ -1264,6 +1280,8 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
metadata?: any metadata?: any
forceEmbed?: boolean // Force using the embedding function for metadata even if vector is provided forceEmbed?: boolean // Force using the embedding function for metadata even if vector is provided
id?: string // Optional ID to use instead of generating a new one id?: string // Optional ID to use instead of generating a new one
autoCreateMissingNouns?: boolean // Automatically create missing nouns
missingNounMetadata?: any // Metadata to use when auto-creating missing nouns
} = {} } = {}
): Promise<string> { ): Promise<string> {
await this.ensureInitialized() await this.ensureInitialized()
@ -1273,8 +1291,59 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
try { try {
// Check if source and target nouns exist // Check if source and target nouns exist
const sourceNoun = this.index.getNouns().get(sourceId) let sourceNoun = this.index.getNouns().get(sourceId)
const targetNoun = this.index.getNouns().get(targetId) let targetNoun = this.index.getNouns().get(targetId)
// Auto-create missing nouns if option is enabled
if (!sourceNoun && options.autoCreateMissingNouns) {
try {
// Create a placeholder vector for the missing noun
const placeholderVector = new Array(this._dimensions).fill(0)
// Add metadata if provided
const metadata = options.missingNounMetadata || {
autoCreated: true,
createdAt: new Date().toISOString(),
noun: NounType.Concept
}
// Add the missing noun
await this.add(placeholderVector, metadata, {id: sourceId})
// Get the newly created noun
sourceNoun = this.index.getNouns().get(sourceId)
console.warn(`Auto-created missing source noun with ID ${sourceId}`)
} catch (createError) {
console.error(`Failed to auto-create source noun with ID ${sourceId}:`, createError)
throw new Error(`Failed to auto-create source noun with ID ${sourceId}: ${createError}`)
}
}
if (!targetNoun && options.autoCreateMissingNouns) {
try {
// Create a placeholder vector for the missing noun
const placeholderVector = new Array(this._dimensions).fill(0)
// Add metadata if provided
const metadata = options.missingNounMetadata || {
autoCreated: true,
createdAt: new Date().toISOString(),
noun: NounType.Concept
}
// Add the missing noun
await this.add(placeholderVector, metadata, {id: targetId})
// Get the newly created noun
targetNoun = this.index.getNouns().get(targetId)
console.warn(`Auto-created missing target noun with ID ${targetId}`)
} catch (createError) {
console.error(`Failed to auto-create target noun with ID ${targetId}:`, createError)
throw new Error(`Failed to auto-create target noun with ID ${targetId}: ${createError}`)
}
}
if (!sourceNoun) { if (!sourceNoun) {
throw new Error(`Source noun with ID ${sourceId} not found`) throw new Error(`Source noun with ID ${sourceId} not found`)