feat(core, tests): add standalone getStatistics function and improve storage configuration
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
2025-07-23 16:26:59 -07:00
/ * *
* Statistics Functionality Tests
* Tests the getStatistics function as a consumer would use it
* /
import { describe , it , expect , beforeAll } from 'vitest'
**feat(core, migration, docs): introduce dimension mismatch resolution tools and migration guide**
- **Core**:
- Added `check-database.js` to verify database status and validate search functionality.
- Created `fix-dimension-mismatch.js` to handle re-embedding of existing data to resolve dimension mismatch from 3 to 512.
- Improved test cases by updating vector operations to support 512 dimensions, replacing previously hardcoded dimensions.
- **Migration**:
- Developed `DIMENSION_MISMATCH_SUMMARY.md`, detailing the root cause, solution, and preventive strategies for dimension mismatch issues.
- Added `production-migration-guide.md` for structured production migration with detailed steps on re-embedding strategies, batching, and error handling.
- **Tests**:
- Enhanced test coverage with 512-dimensional vector validation.
- Introduced helper functions for consistent vector testing behavior and streamlined search test cases.
- **Documentation**:
- Updated project documentation to highlight the resolution process for dimension mismatches, emphasizing preventive mechanisms such as auto-migration and version tracking.
**Purpose**: Address critical dimension mismatch issues caused by embedding changes, restore functionality, and provide a roadmap for robust prevention strategies and migration processes.
2025-07-25 13:38:56 -07:00
/ * *
* Helper function to create a 512 - dimensional vector for testing
* @param primaryIndex The index to set to 1.0 , all other indices will be 0.0
* @returns A 512 - dimensional vector with a single 1.0 value at the specified index
* /
function createTestVector ( primaryIndex : number = 0 ) : number [ ] {
const vector = new Array ( 512 ) . fill ( 0 )
vector [ primaryIndex % 512 ] = 1.0
return vector
}
feat(core, tests): add standalone getStatistics function and improve storage configuration
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
2025-07-23 16:26:59 -07:00
describe ( 'Brainy Statistics Functionality' , ( ) = > {
let brainy : any
beforeAll ( async ( ) = > {
// Load brainy library as a consumer would
brainy = await import ( '../dist/unified.js' )
} )
describe ( 'Library Exports' , ( ) = > {
it ( 'should export getStatistics function at the root level' , ( ) = > {
expect ( brainy . getStatistics ) . toBeDefined ( )
expect ( typeof brainy . getStatistics ) . toBe ( 'function' )
} )
} )
describe ( 'getStatistics Functionality' , ( ) = > {
it ( 'should retrieve statistics from a BrainyData instance' , async ( ) = > {
// Create a BrainyData instance
const data = new brainy . BrainyData ( {
metric : 'euclidean'
} )
await data . init ( )
await data . clear ( ) // Clear any existing data
// Add some test data
**feat(core, migration, docs): introduce dimension mismatch resolution tools and migration guide**
- **Core**:
- Added `check-database.js` to verify database status and validate search functionality.
- Created `fix-dimension-mismatch.js` to handle re-embedding of existing data to resolve dimension mismatch from 3 to 512.
- Improved test cases by updating vector operations to support 512 dimensions, replacing previously hardcoded dimensions.
- **Migration**:
- Developed `DIMENSION_MISMATCH_SUMMARY.md`, detailing the root cause, solution, and preventive strategies for dimension mismatch issues.
- Added `production-migration-guide.md` for structured production migration with detailed steps on re-embedding strategies, batching, and error handling.
- **Tests**:
- Enhanced test coverage with 512-dimensional vector validation.
- Introduced helper functions for consistent vector testing behavior and streamlined search test cases.
- **Documentation**:
- Updated project documentation to highlight the resolution process for dimension mismatches, emphasizing preventive mechanisms such as auto-migration and version tracking.
**Purpose**: Address critical dimension mismatch issues caused by embedding changes, restore functionality, and provide a roadmap for robust prevention strategies and migration processes.
2025-07-25 13:38:56 -07:00
await data . add ( createTestVector ( 0 ) , { id : 'v1' , label : 'x-axis' } )
await data . add ( createTestVector ( 1 ) , { id : 'v2' , label : 'y-axis' } )
await data . add ( createTestVector ( 2 ) , { id : 'v3' , label : 'z-axis' } )
feat(core, tests): add standalone getStatistics function and improve storage configuration
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
2025-07-23 16:26:59 -07:00
// Add a verb
**feat(core, migration, docs): introduce dimension mismatch resolution tools and migration guide**
- **Core**:
- Added `check-database.js` to verify database status and validate search functionality.
- Created `fix-dimension-mismatch.js` to handle re-embedding of existing data to resolve dimension mismatch from 3 to 512.
- Improved test cases by updating vector operations to support 512 dimensions, replacing previously hardcoded dimensions.
- **Migration**:
- Developed `DIMENSION_MISMATCH_SUMMARY.md`, detailing the root cause, solution, and preventive strategies for dimension mismatch issues.
- Added `production-migration-guide.md` for structured production migration with detailed steps on re-embedding strategies, batching, and error handling.
- **Tests**:
- Enhanced test coverage with 512-dimensional vector validation.
- Introduced helper functions for consistent vector testing behavior and streamlined search test cases.
- **Documentation**:
- Updated project documentation to highlight the resolution process for dimension mismatches, emphasizing preventive mechanisms such as auto-migration and version tracking.
**Purpose**: Address critical dimension mismatch issues caused by embedding changes, restore functionality, and provide a roadmap for robust prevention strategies and migration processes.
2025-07-25 13:38:56 -07:00
await data . addVerb ( 'v1' , 'v2' , createTestVector ( 3 ) , { type : 'connected_to' } )
feat(core, tests): add standalone getStatistics function and improve storage configuration
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
2025-07-23 16:26:59 -07:00
// Get statistics using the standalone function
const stats = await brainy . getStatistics ( data )
// Verify statistics
expect ( stats ) . toBeDefined ( )
expect ( stats . nounCount ) . toBe ( 3 )
expect ( stats . verbCount ) . toBe ( 1 )
expect ( stats . metadataCount ) . toBe ( 3 ) // Each noun has metadata
2025-08-04 20:00:38 -07:00
expect ( stats . hnswIndexSize ) . toBe ( 4 ) // 3 nouns + 1 verb (verbs are also added to HNSW index)
feat(core, tests): add standalone getStatistics function and improve storage configuration
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
2025-07-23 16:26:59 -07:00
} )
it ( 'should throw an error when no instance is provided' , async ( ) = > {
await expect ( brainy . getStatistics ( ) ) . rejects . toThrow ( 'BrainyData instance must be provided' )
} )
it ( 'should match the instance method results' , async ( ) = > {
// Create a BrainyData instance
**feat(core, migration, docs): introduce dimension mismatch resolution tools and migration guide**
- **Core**:
- Added `check-database.js` to verify database status and validate search functionality.
- Created `fix-dimension-mismatch.js` to handle re-embedding of existing data to resolve dimension mismatch from 3 to 512.
- Improved test cases by updating vector operations to support 512 dimensions, replacing previously hardcoded dimensions.
- **Migration**:
- Developed `DIMENSION_MISMATCH_SUMMARY.md`, detailing the root cause, solution, and preventive strategies for dimension mismatch issues.
- Added `production-migration-guide.md` for structured production migration with detailed steps on re-embedding strategies, batching, and error handling.
- **Tests**:
- Enhanced test coverage with 512-dimensional vector validation.
- Introduced helper functions for consistent vector testing behavior and streamlined search test cases.
- **Documentation**:
- Updated project documentation to highlight the resolution process for dimension mismatches, emphasizing preventive mechanisms such as auto-migration and version tracking.
**Purpose**: Address critical dimension mismatch issues caused by embedding changes, restore functionality, and provide a roadmap for robust prevention strategies and migration processes.
2025-07-25 13:38:56 -07:00
const data = new brainy . BrainyData ( { } )
feat(core, tests): add standalone getStatistics function and improve storage configuration
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
2025-07-23 16:26:59 -07:00
await data . init ( )
// Add some test data
**feat(core, migration, docs): introduce dimension mismatch resolution tools and migration guide**
- **Core**:
- Added `check-database.js` to verify database status and validate search functionality.
- Created `fix-dimension-mismatch.js` to handle re-embedding of existing data to resolve dimension mismatch from 3 to 512.
- Improved test cases by updating vector operations to support 512 dimensions, replacing previously hardcoded dimensions.
- **Migration**:
- Developed `DIMENSION_MISMATCH_SUMMARY.md`, detailing the root cause, solution, and preventive strategies for dimension mismatch issues.
- Added `production-migration-guide.md` for structured production migration with detailed steps on re-embedding strategies, batching, and error handling.
- **Tests**:
- Enhanced test coverage with 512-dimensional vector validation.
- Introduced helper functions for consistent vector testing behavior and streamlined search test cases.
- **Documentation**:
- Updated project documentation to highlight the resolution process for dimension mismatches, emphasizing preventive mechanisms such as auto-migration and version tracking.
**Purpose**: Address critical dimension mismatch issues caused by embedding changes, restore functionality, and provide a roadmap for robust prevention strategies and migration processes.
2025-07-25 13:38:56 -07:00
await data . add ( createTestVector ( 5 ) , { id : 'test1' } )
feat(core, tests): add standalone getStatistics function and improve storage configuration
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
2025-07-23 16:26:59 -07:00
// Get statistics using both methods
const instanceStats = await data . getStatistics ( )
const functionStats = await brainy . getStatistics ( data )
2025-08-04 20:00:38 -07:00
// Verify core statistics match (ignoring volatile fields like memoryUsage and timestamps)
expect ( functionStats . nounCount ) . toBe ( instanceStats . nounCount )
expect ( functionStats . verbCount ) . toBe ( instanceStats . verbCount )
expect ( functionStats . metadataCount ) . toBe ( instanceStats . metadataCount )
expect ( functionStats . hnswIndexSize ) . toBe ( instanceStats . hnswIndexSize )
// If serviceBreakdown exists, verify it matches
if ( instanceStats . serviceBreakdown ) {
expect ( functionStats . serviceBreakdown ) . toEqual ( instanceStats . serviceBreakdown )
}
feat(core, tests): add standalone getStatistics function and improve storage configuration
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
2025-07-23 16:26:59 -07:00
} )
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
it ( 'should track statistics by service' , async ( ) = > {
// Create a BrainyData instance
const data = new brainy . BrainyData ( {
metric : 'euclidean'
} )
await data . init ( )
await data . clear ( ) // Clear any existing data
// Add data from different services
**feat(core, migration, docs): introduce dimension mismatch resolution tools and migration guide**
- **Core**:
- Added `check-database.js` to verify database status and validate search functionality.
- Created `fix-dimension-mismatch.js` to handle re-embedding of existing data to resolve dimension mismatch from 3 to 512.
- Improved test cases by updating vector operations to support 512 dimensions, replacing previously hardcoded dimensions.
- **Migration**:
- Developed `DIMENSION_MISMATCH_SUMMARY.md`, detailing the root cause, solution, and preventive strategies for dimension mismatch issues.
- Added `production-migration-guide.md` for structured production migration with detailed steps on re-embedding strategies, batching, and error handling.
- **Tests**:
- Enhanced test coverage with 512-dimensional vector validation.
- Introduced helper functions for consistent vector testing behavior and streamlined search test cases.
- **Documentation**:
- Updated project documentation to highlight the resolution process for dimension mismatches, emphasizing preventive mechanisms such as auto-migration and version tracking.
**Purpose**: Address critical dimension mismatch issues caused by embedding changes, restore functionality, and provide a roadmap for robust prevention strategies and migration processes.
2025-07-25 13:38:56 -07:00
await data . add ( createTestVector ( 10 ) , { id : 'v1' , label : 'service1-item' } , { service : 'service1' } )
await data . add ( createTestVector ( 20 ) , { id : 'v2' , label : 'service1-item' } , { service : 'service1' } )
await data . add ( createTestVector ( 30 ) , { id : 'v3' , label : 'service2-item' } , { service : 'service2' } )
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
// Add verbs from different services
await data . addVerb ( 'v1' , 'v2' , undefined , { type : 'related_to' , service : 'service1' } )
await data . addVerb ( 'v2' , 'v3' , undefined , { type : 'related_to' , service : 'service2' } )
// Get statistics for all services
const allStats = await data . getStatistics ( )
// Verify total counts
expect ( allStats . nounCount ) . toBe ( 3 )
expect ( allStats . verbCount ) . toBe ( 2 )
expect ( allStats . metadataCount ) . toBe ( 3 )
// Verify service breakdown exists
expect ( allStats . serviceBreakdown ) . toBeDefined ( )
// Verify service1 statistics
const service1Stats = await data . getStatistics ( { service : 'service1' } )
expect ( service1Stats . nounCount ) . toBe ( 2 )
expect ( service1Stats . verbCount ) . toBe ( 1 )
expect ( service1Stats . metadataCount ) . toBe ( 2 )
// Verify service2 statistics
const service2Stats = await data . getStatistics ( { service : 'service2' } )
expect ( service2Stats . nounCount ) . toBe ( 1 )
expect ( service2Stats . verbCount ) . toBe ( 1 )
expect ( service2Stats . metadataCount ) . toBe ( 1 )
// Verify multiple services filter
const combinedStats = await data . getStatistics ( { service : [ 'service1' , 'service2' ] } )
expect ( combinedStats . nounCount ) . toBe ( 3 )
expect ( combinedStats . verbCount ) . toBe ( 2 )
expect ( combinedStats . metadataCount ) . toBe ( 3 )
} )
feat(core, tests): add standalone getStatistics function and improve storage configuration
- **Core**: Introduced a new `getStatistics` utility function in `statistics.ts` for fetching database statistics at the root level of the library. Enhanced `BrainyData` methods to ensure metadata includes `id` field and refined statistics calculations, excluding verbs from the noun count.
- **Tests**: Added comprehensive test coverage in `statistics.test.ts` for the new utility function, validating proper error handling, statistics accuracy, and consistent results between instance methods and standalone function.
- **Storage Config**: Enabled dynamic support for AWS S3, Cloudflare R2, and Google Cloud Storage in web service configuration, utilizing environment variables for adapter setup. Addressed a race condition in `FileSystemStorage` initialization by deferring path module imports.
**Purpose**: Enhance database analytics by introducing a reusable `getStatistics` function, improve flexibility in storage configuration, and ensure robust testing for reliability and accuracy.
2025-07-23 16:26:59 -07:00
} )
2025-07-31 13:40:28 -07:00
} )