From 6ee0881d8628af48359810ec47e3b20cedb83112 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Fri, 1 Aug 2025 10:16:18 -0700 Subject: [PATCH] **docs(guides): add service identification guide** - Created `service-identification.md` in `docs/guides`: - Detailed guidelines on how services should identify themselves within Brainy. - Documented two identification methods: default service initialization and operation-specific service naming. - Included service name conventions and common examples (`github`, `reddit`, `default`). - Described benefits of proper service identification: - Enhanced statistics tracking and JSON field discovery by service. - Provided best practices for consistent and descriptive service naming. - Explained internal implementation details, such as `getServiceName` usage and statistic tracking. **Purpose**: Help users properly identify services to enable statistics tracking, field discovery, and improved data management in Brainy. --- docs/guides/service-identification.md | 151 ++++++++++++++++++++++++++ src/brainyData.ts | 101 +++++++++++++++-- 2 files changed, 245 insertions(+), 7 deletions(-) create mode 100644 docs/guides/service-identification.md diff --git a/docs/guides/service-identification.md b/docs/guides/service-identification.md new file mode 100644 index 00000000..c8d62587 --- /dev/null +++ b/docs/guides/service-identification.md @@ -0,0 +1,151 @@ +# Service Identification Guide + +## Overview + +This guide explains how services should identify themselves when adding, updating, or deleting data in Brainy. Proper service identification is important for: + +1. **Statistics tracking**: Brainy tracks statistics by service, allowing you to see how much data each service has contributed. +2. **JSON field discovery**: Brainy tracks field names in JSON documents by service, allowing you to discover what fields are available from different services. + +## How to Identify Your Service + +### Option 1: Specify a Default Service During Initialization + +The simplest approach is to specify a default service name when initializing Brainy: + +```javascript +// Initialize Brainy with a default service name +const brainyData = new BrainyData({ + defaultService: 'my-service-name' +}); + +// All operations will automatically use the default service name +await brainyData.add(data, metadata); +await brainyData.updateMetadata(id, newMetadata); +await brainyData.delete(id); +``` + +This approach ensures that all operations use the same service name without having to specify it for each operation. + +### Option 2: Specify Service Name for Each Operation + +Alternatively, you can specify the service name for each operation: + +```javascript +// Adding data with service identification +const id = await brainyData.add(data, metadata, { + service: 'my-service-name' +}); + +// Updating metadata with service identification +await brainyData.updateMetadata(id, newMetadata, { + service: 'my-service-name' +}); + +// Deleting data with service identification +await brainyData.delete(id, { + service: 'my-service-name' +}); +``` + +This approach gives you more flexibility if you need to use different service names for different operations. + +## Service Name Conventions + +Service names should be: + +- Lowercase +- Hyphen-separated (e.g., `github-api`, `bluesky-feed`) +- Descriptive of the data source or service +- Consistent across all operations from the same service + +Common service names used in Brainy include: + +- `github` +- `bluesky` +- `reddit` +- `twitter` +- `google` +- `default` (used when no service is specified) + +## How Service Identification Works + +Internally, Brainy uses the `getServiceName` helper method to determine the service name: + +1. If a `service` parameter is provided in the options object, that value is used. +2. If no `service` parameter is provided, Brainy uses the default service name specified during initialization. +3. If no default service was specified, `'default'` is used. + +## Benefits of Proper Service Identification + +### Statistics Tracking + +With proper service identification, you can see statistics broken down by service: + +```javascript +const stats = await brainyData.getStatistics(); +console.log('Noun count by service:'); +console.log(stats.nounCount); +console.log('Metadata count by service:'); +console.log(stats.metadataCount); +``` + +Example output: +``` +Noun count by service: +{ + "github": 1250, + "bluesky": 843, + "reddit": 567, + "default": 122 +} +``` + +### Field Name Discovery + +You can discover what fields are available from different services: + +```javascript +const fieldNames = await brainyData.getAvailableFieldNames(); +console.log(fieldNames); +``` + +Example output: +``` +{ + "github": ["repository.name", "repository.description", "user.login", "issue.title", ...], + "bluesky": ["post.text", "user.handle", "user.displayName", ...], + "reddit": ["title", "selftext", "author.name", "subreddit.name", ...] +} +``` + +### Standard Field Mappings + +You can use standard field mappings to search consistently across different services: + +```javascript +// Get standard field mappings +const standardFieldMappings = await brainyData.getStandardFieldMappings(); +console.log(standardFieldMappings); + +// Search using standard fields +const results = await brainyData.searchByStandardField("title", "climate change", 10); +``` + +## Best Practices + +1. **Specify a default service name during initialization** for simplicity and consistency. +2. **Override the default service name** only when necessary for specific operations. +3. **Use consistent service names** across all operations from the same service. +4. **Use descriptive service names** that identify the data source. +5. **Document the field names** used by your service to help users understand what fields are available for searching. + +## Implementation Details + +Service identification is implemented through: + +1. The `getServiceName` helper method in `BrainyData` class +2. The `trackFieldNames` method in `StorageAdapter` interface +3. The `incrementStatistic` and `decrementStatistic` methods in `StorageAdapter` interface + +These methods ensure that statistics and field names are properly tracked by service. diff --git a/src/brainyData.ts b/src/brainyData.ts index d2665800..29c7fff4 100644 --- a/src/brainyData.ts +++ b/src/brainyData.ts @@ -55,6 +55,13 @@ export interface BrainyDataConfig { */ hnsw?: Partial + /** + * Default service name to use for all operations + * When specified, this service name will be used for all operations + * that don't explicitly provide a service name + */ + defaultService?: string + /** * Distance function to use for similarity calculations */ @@ -344,6 +351,7 @@ export class BrainyData implements BrainyDataInterface { private useOptimizedIndex: boolean = false private _dimensions: number private loggingConfig: BrainyDataConfig['logging'] = { verbose: true } + private defaultService: string = 'default' // Timeout and retry configuration private timeoutConfig: BrainyDataConfig['timeouts'] = {} @@ -455,6 +463,11 @@ export class BrainyData implements BrainyDataInterface { throw new Error('Database cannot be both read-only and write-only') } + // Set default service name if provided + if (config.defaultService) { + this.defaultService = config.defaultService + } + // Store storage configuration for later use in init() this.storageConfig = config.storage || {} @@ -873,7 +886,7 @@ export class BrainyData implements BrainyDataInterface { } /** - * Get the service name from options or fallback to current augmentation + * Get the service name from options or fallback to default service * This provides a consistent way to handle service names across all methods * @param options Options object that may contain a service property * @returns The service name to use for operations @@ -882,7 +895,9 @@ export class BrainyData implements BrainyDataInterface { if (options?.service) { return options.service } - return this.getCurrentAugmentation() + // Use the default service name specified during initialization + // This simplifies service identification by allowing it to be specified once + return this.defaultService } /** @@ -2377,7 +2392,7 @@ export class BrainyData implements BrainyDataInterface { } // Get the service that's updating the metadata - const service = options.service || this.getCurrentAugmentation() + const service = this.getServiceName(options) const graphNoun = metadata as unknown as GraphNoun // Preserve existing createdBy and createdAt if they exist @@ -2424,7 +2439,7 @@ export class BrainyData implements BrainyDataInterface { await this.storage!.saveMetadata(id, metadata) // Track metadata statistics - const service = options.service || this.getCurrentAugmentation() + const service = this.getServiceName(options) await this.storage!.incrementStatistic('metadata', service) return true @@ -2534,7 +2549,7 @@ export class BrainyData implements BrainyDataInterface { const placeholderVector = new Array(this._dimensions).fill(0) // Add metadata if provided - const service = options.service || this.getCurrentAugmentation() + const service = this.getServiceName(options) const now = new Date() const timestamp = { seconds: Math.floor(now.getTime() / 1000), @@ -2576,7 +2591,7 @@ export class BrainyData implements BrainyDataInterface { const placeholderVector = new Array(this._dimensions).fill(0) // Add metadata if provided - const service = options.service || this.getCurrentAugmentation() + const service = this.getServiceName(options) const now = new Date() const timestamp = { seconds: Math.floor(now.getTime() / 1000), @@ -2685,7 +2700,7 @@ export class BrainyData implements BrainyDataInterface { // Note: We're no longer validating against VerbType enum to allow custom relationship types // Get service name from options or current augmentation - const service = options.service || this.getCurrentAugmentation() + const service = this.getServiceName(options) // Create timestamp for creation/update time const now = new Date() @@ -3271,6 +3286,78 @@ export class BrainyData implements BrainyDataInterface { } } + /** + * Calculate similarity between two vectors or between two pieces of text/data + * This method allows clients to directly calculate similarity scores between items + * without needing to add them to the database + * + * @param a First vector or text/data to compare + * @param b Second vector or text/data to compare + * @param options Additional options + * @returns A promise that resolves to the similarity score (higher means more similar) + */ + public async calculateSimilarity( + a: Vector | string | string[], + b: Vector | string | string[], + options: { + forceEmbed?: boolean // Force using the embedding function even if input is a vector + distanceFunction?: DistanceFunction // Optional custom distance function + } = {} + ): Promise { + await this.ensureInitialized() + + try { + // Convert inputs to vectors if needed + let vectorA: Vector + let vectorB: Vector + + // Process first input + if ( + Array.isArray(a) && + a.every((item) => typeof item === 'number') && + !options.forceEmbed + ) { + // Input is already a vector + vectorA = a + } else { + // Input needs to be vectorized + try { + vectorA = await this.embeddingFunction(a) + } catch (embedError) { + throw new Error(`Failed to vectorize first input: ${embedError}`) + } + } + + // Process second input + if ( + Array.isArray(b) && + b.every((item) => typeof item === 'number') && + !options.forceEmbed + ) { + // Input is already a vector + vectorB = b + } else { + // Input needs to be vectorized + try { + vectorB = await this.embeddingFunction(b) + } catch (embedError) { + throw new Error(`Failed to vectorize second input: ${embedError}`) + } + } + + // Calculate distance using the specified or default distance function + const distanceFunction = options.distanceFunction || this.distanceFunction + const distance = distanceFunction(vectorA, vectorB) + + // Convert distance to similarity score (1 - distance for cosine) + // Higher value means more similar + return 1 - distance + } catch (error) { + console.error('Failed to calculate similarity:', error) + throw new Error(`Failed to calculate similarity: ${error}`) + } + } + /** * Search for verbs by type and/or vector similarity * @param queryVectorOrData Query vector or data to search for