**feat(examples, core, docs): add flushStatistics example, implementation, and documentation**
- **Examples**: Added a new `flush-statistics-example.js` script to demonstrate the usage of the `flushStatistics` method for ensuring updated statistics after data insertion. - **Core**: - Implemented `flushStatistics` in the `BrainyData` class to allow immediate flushing of statistics to storage. - Updated `BaseStorageAdapter` with `flushStatisticsToStorage` to support flushing cached statistics. - Modified the `shutDown` method to ensure statistics are flushed before database shutdown. - **Documentation**: Added `statistics-flush-solution.md` to explain the batch update mechanism, the issue with delayed statistics updates, and how to manually flush statistics in storage. **Purpose**: Provide users with the ability to manually flush statistics for real-time accuracy, particularly useful for systems relying on immediate updates. Improved documentation and examples to guide developers in implementing and using this functionality effectively.
This commit is contained in:
parent
7c493632ad
commit
b81129276c
5 changed files with 251 additions and 0 deletions
98
examples/flush-statistics-example.js
Normal file
98
examples/flush-statistics-example.js
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
/**
|
||||
* Example script demonstrating how to use the flushStatistics method
|
||||
* to ensure statistics are up-to-date after inserting data.
|
||||
*/
|
||||
|
||||
import { BrainyData } from '@soulcraft/brainy';
|
||||
|
||||
// Create a new BrainyData instance
|
||||
const brainyDb = new BrainyData({
|
||||
dimensions: 384,
|
||||
storage: 'memory' // Use memory storage for this example
|
||||
});
|
||||
|
||||
// Initialize the database
|
||||
await brainyDb.init();
|
||||
|
||||
// Function to display statistics
|
||||
async function displayStats() {
|
||||
const stats = await brainyDb.getStatistics();
|
||||
console.log('Statistics:');
|
||||
console.log(`- Noun count: ${stats.nounCount}`);
|
||||
console.log(`- Verb count: ${stats.verbCount}`);
|
||||
console.log(`- Metadata count: ${stats.metadataCount}`);
|
||||
console.log(`- HNSW index size: ${stats.hnswIndexSize}`);
|
||||
console.log('');
|
||||
}
|
||||
|
||||
// Display initial statistics
|
||||
console.log('Initial statistics:');
|
||||
await displayStats();
|
||||
|
||||
// Insert some data
|
||||
console.log('Inserting data...');
|
||||
const vectors = [];
|
||||
for (let i = 0; i < 100; i++) {
|
||||
// Create a random vector
|
||||
const vector = Array.from({ length: 384 }, () => Math.random());
|
||||
vectors.push({
|
||||
vectorOrData: vector,
|
||||
metadata: { id: `item-${i}`, name: `Item ${i}` }
|
||||
});
|
||||
}
|
||||
|
||||
// Add the vectors in batch
|
||||
await brainyDb.addBatch(vectors);
|
||||
console.log('Data inserted.');
|
||||
|
||||
// Display statistics without flushing
|
||||
console.log('Statistics after insertion (without flushing):');
|
||||
await displayStats();
|
||||
|
||||
// Flush statistics to ensure they're up-to-date
|
||||
console.log('Flushing statistics...');
|
||||
await brainyDb.flushStatistics();
|
||||
console.log('Statistics flushed.');
|
||||
|
||||
// Display statistics after flushing
|
||||
console.log('Statistics after flushing:');
|
||||
await displayStats();
|
||||
|
||||
// Shut down the database (this will also flush statistics)
|
||||
console.log('Shutting down database...');
|
||||
await brainyDb.shutDown();
|
||||
console.log('Database shut down.');
|
||||
|
||||
/**
|
||||
* Expected output:
|
||||
*
|
||||
* Initial statistics:
|
||||
* Statistics:
|
||||
* - Noun count: 0
|
||||
* - Verb count: 0
|
||||
* - Metadata count: 0
|
||||
* - HNSW index size: 0
|
||||
*
|
||||
* Inserting data...
|
||||
* Data inserted.
|
||||
*
|
||||
* Statistics after insertion (without flushing):
|
||||
* Statistics:
|
||||
* - Noun count: 100
|
||||
* - Verb count: 0
|
||||
* - Metadata count: 100
|
||||
* - HNSW index size: 100
|
||||
*
|
||||
* Flushing statistics...
|
||||
* Statistics flushed.
|
||||
*
|
||||
* Statistics after flushing:
|
||||
* Statistics:
|
||||
* - Noun count: 100
|
||||
* - Verb count: 0
|
||||
* - Metadata count: 100
|
||||
* - HNSW index size: 100
|
||||
*
|
||||
* Shutting down database...
|
||||
* Database shut down.
|
||||
*/
|
||||
|
|
@ -1880,6 +1880,22 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
|
|||
return nounCount
|
||||
}
|
||||
|
||||
/**
|
||||
* Force an immediate flush of statistics to storage
|
||||
* This ensures that any pending statistics updates are written to persistent storage
|
||||
* @returns Promise that resolves when the statistics have been flushed
|
||||
*/
|
||||
public async flushStatistics(): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
if (!this.storage) {
|
||||
throw new Error('Storage not initialized')
|
||||
}
|
||||
|
||||
// Call the flushStatisticsToStorage method on the storage adapter
|
||||
await this.storage.flushStatisticsToStorage()
|
||||
}
|
||||
|
||||
/**
|
||||
* Get statistics about the current state of the database
|
||||
* @param options Additional options for retrieving statistics
|
||||
|
|
@ -2649,6 +2665,16 @@ export class BrainyData<T = any> implements BrainyDataInterface<T> {
|
|||
*/
|
||||
public async shutDown(): Promise<void> {
|
||||
try {
|
||||
// Flush statistics to ensure they're saved before shutting down
|
||||
if (this.storage && this.isInitialized) {
|
||||
try {
|
||||
await this.flushStatistics()
|
||||
} catch (statsError) {
|
||||
console.warn('Failed to flush statistics during shutdown:', statsError)
|
||||
// Continue with shutdown even if statistics flush fails
|
||||
}
|
||||
}
|
||||
|
||||
// Disconnect from remote server if connected
|
||||
if (this.isConnectedToRemoteServer()) {
|
||||
await this.disconnectFromRemoteServer()
|
||||
|
|
|
|||
|
|
@ -229,4 +229,10 @@ export interface StorageAdapter {
|
|||
* @param size The new size of the HNSW index
|
||||
*/
|
||||
updateHnswIndexSize(size: number): Promise<void>
|
||||
|
||||
/**
|
||||
* Force an immediate flush of statistics to storage
|
||||
* This ensures that any pending statistics updates are written to persistent storage
|
||||
*/
|
||||
flushStatisticsToStorage(): Promise<void>
|
||||
}
|
||||
|
|
|
|||
|
|
@ -290,6 +290,20 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
this.scheduleBatchUpdate()
|
||||
}
|
||||
|
||||
/**
|
||||
* Force an immediate flush of statistics to storage
|
||||
* This ensures that any pending statistics updates are written to persistent storage
|
||||
*/
|
||||
async flushStatisticsToStorage(): Promise<void> {
|
||||
// If there are no statistics in cache or they haven't been modified, nothing to flush
|
||||
if (!this.statisticsCache || !this.statisticsModified) {
|
||||
return
|
||||
}
|
||||
|
||||
// Call the protected flushStatistics method to immediately write to storage
|
||||
await this.flushStatistics()
|
||||
}
|
||||
|
||||
/**
|
||||
* Create default statistics data
|
||||
* @returns Default statistics data
|
||||
|
|
|
|||
107
statistics-flush-solution.md
Normal file
107
statistics-flush-solution.md
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
# Statistics Flush Solution
|
||||
|
||||
## Issue Description
|
||||
|
||||
When inserting lots of data into Brainy, the statistics do not seem to be changing. This is because statistics are updated in memory but might not be immediately flushed to storage due to the batch update mechanism.
|
||||
|
||||
## Root Cause
|
||||
|
||||
The Brainy database uses a batch update mechanism for statistics to optimize performance. When data is inserted, statistics are updated in memory and a batch update is scheduled to flush the statistics to storage. However, this batch update might be delayed by up to 30 seconds (as defined by `MAX_FLUSH_DELAY_MS` in `baseStorageAdapter.ts`).
|
||||
|
||||
If the user checks statistics shortly after inserting data, or if the database is shut down before the batch update occurs, the statistics might not reflect the recent changes.
|
||||
|
||||
## Solution
|
||||
|
||||
The solution is to provide a way to force an immediate flush of statistics to storage, and to ensure that statistics are flushed before the database is shut down. The following changes were made:
|
||||
|
||||
1. Added a new method `flushStatisticsToStorage()` to the `StorageAdapter` interface in `coreTypes.ts`:
|
||||
```typescript
|
||||
/**
|
||||
* Force an immediate flush of statistics to storage
|
||||
* This ensures that any pending statistics updates are written to persistent storage
|
||||
*/
|
||||
flushStatisticsToStorage(): Promise<void>
|
||||
```
|
||||
|
||||
2. Implemented this method in the `BaseStorageAdapter` class in `baseStorageAdapter.ts`:
|
||||
```typescript
|
||||
/**
|
||||
* Force an immediate flush of statistics to storage
|
||||
* This ensures that any pending statistics updates are written to persistent storage
|
||||
*/
|
||||
async flushStatisticsToStorage(): Promise<void> {
|
||||
// If there are no statistics in cache or they haven't been modified, nothing to flush
|
||||
if (!this.statisticsCache || !this.statisticsModified) {
|
||||
return
|
||||
}
|
||||
|
||||
// Call the protected flushStatistics method to immediately write to storage
|
||||
await this.flushStatistics()
|
||||
}
|
||||
```
|
||||
|
||||
3. Added a public method `flushStatistics()` to the `BrainyData` class in `brainyData.ts`:
|
||||
```typescript
|
||||
/**
|
||||
* Force an immediate flush of statistics to storage
|
||||
* This ensures that any pending statistics updates are written to persistent storage
|
||||
* @returns Promise that resolves when the statistics have been flushed
|
||||
*/
|
||||
public async flushStatistics(): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
if (!this.storage) {
|
||||
throw new Error('Storage not initialized')
|
||||
}
|
||||
|
||||
// Call the flushStatisticsToStorage method on the storage adapter
|
||||
await this.storage.flushStatisticsToStorage()
|
||||
}
|
||||
```
|
||||
|
||||
4. Modified the `shutDown()` method in `BrainyData` to flush statistics before shutting down:
|
||||
```typescript
|
||||
/**
|
||||
* Shut down the database and clean up resources
|
||||
* This should be called when the database is no longer needed
|
||||
*/
|
||||
public async shutDown(): Promise<void> {
|
||||
try {
|
||||
// Flush statistics to ensure they're saved before shutting down
|
||||
if (this.storage && this.isInitialized) {
|
||||
try {
|
||||
await this.flushStatistics()
|
||||
} catch (statsError) {
|
||||
console.warn('Failed to flush statistics during shutdown:', statsError)
|
||||
// Continue with shutdown even if statistics flush fails
|
||||
}
|
||||
}
|
||||
|
||||
// Rest of the shutdown process...
|
||||
} catch (error) {
|
||||
console.error('Failed to shut down BrainyData:', error)
|
||||
throw new Error(`Failed to shut down BrainyData: ${error}`)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
To ensure statistics are up-to-date after inserting data, you can now call the `flushStatistics()` method on the `BrainyData` instance:
|
||||
|
||||
```typescript
|
||||
// Insert data
|
||||
await brainyDb.add(vectorOrData, metadata)
|
||||
|
||||
// Force a flush of statistics to ensure they're up-to-date
|
||||
await brainyDb.flushStatistics()
|
||||
|
||||
// Get statistics
|
||||
const stats = await brainyDb.getStatistics()
|
||||
```
|
||||
|
||||
Statistics will also be automatically flushed when the database is shut down, ensuring that no statistics updates are lost.
|
||||
|
||||
## Note on "bluesky-package"
|
||||
|
||||
The issue description mentioned a "bluesky-package" being used to insert data into Brainy. This package was not found in the project, so it might be a third-party package or a typo. The solution implemented here should work regardless of how data is inserted into Brainy, as long as the `flushStatistics()` method is called after inserting data.
|
||||
Loading…
Add table
Add a link
Reference in a new issue