2025-07-21 12:46:41 -07:00
|
|
|
/**
|
|
|
|
|
* Base Storage Adapter
|
|
|
|
|
* Provides common functionality for all storage adapters
|
|
|
|
|
*/
|
|
|
|
|
|
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
|
|
|
import { GraphVerb, HNSWNoun, StatisticsData } from '../coreTypes.js'
|
|
|
|
|
import { BaseStorageAdapter } from './adapters/baseStorageAdapter.js'
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
// Common directory/prefix names
|
|
|
|
|
export const NOUNS_DIR = 'nouns'
|
|
|
|
|
export const VERBS_DIR = 'verbs'
|
|
|
|
|
export const METADATA_DIR = 'metadata'
|
|
|
|
|
export const INDEX_DIR = 'index'
|
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
|
|
|
export const STATISTICS_KEY = 'statistics'
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Base storage adapter that implements common functionality
|
|
|
|
|
* This is an abstract class that should be extended by specific storage adapters
|
|
|
|
|
*/
|
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
|
|
|
export abstract class BaseStorage extends BaseStorageAdapter {
|
2025-07-21 12:46:41 -07:00
|
|
|
protected isInitialized = false
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Initialize the storage adapter
|
|
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
|
|
|
|
public abstract init(): Promise<void>
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Ensure the storage adapter is initialized
|
|
|
|
|
*/
|
|
|
|
|
protected async ensureInitialized(): Promise<void> {
|
|
|
|
|
if (!this.isInitialized) {
|
|
|
|
|
await this.init()
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Save a noun to storage
|
|
|
|
|
*/
|
|
|
|
|
public async saveNoun(noun: HNSWNoun): Promise<void> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.saveNoun_internal(noun)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get a noun from storage
|
|
|
|
|
*/
|
|
|
|
|
public async getNoun(id: string): Promise<HNSWNoun | null> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.getNoun_internal(id)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get all nouns from storage
|
|
|
|
|
*/
|
|
|
|
|
public async getAllNouns(): Promise<HNSWNoun[]> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.getAllNouns_internal()
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get nouns by noun type
|
|
|
|
|
* @param nounType The noun type to filter by
|
|
|
|
|
* @returns Promise that resolves to an array of nouns of the specified noun type
|
|
|
|
|
*/
|
|
|
|
|
public async getNounsByNounType(nounType: string): Promise<HNSWNoun[]> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.getNounsByNounType_internal(nounType)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Delete a noun from storage
|
|
|
|
|
*/
|
|
|
|
|
public async deleteNoun(id: string): Promise<void> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.deleteNoun_internal(id)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Save a verb to storage
|
|
|
|
|
*/
|
|
|
|
|
public async saveVerb(verb: GraphVerb): Promise<void> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.saveVerb_internal(verb)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get a verb from storage
|
|
|
|
|
*/
|
|
|
|
|
public async getVerb(id: string): Promise<GraphVerb | null> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.getVerb_internal(id)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get all verbs from storage
|
|
|
|
|
*/
|
|
|
|
|
public async getAllVerbs(): Promise<GraphVerb[]> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.getAllVerbs_internal()
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get verbs by source
|
|
|
|
|
*/
|
|
|
|
|
public async getVerbsBySource(sourceId: string): Promise<GraphVerb[]> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.getVerbsBySource_internal(sourceId)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get verbs by target
|
|
|
|
|
*/
|
|
|
|
|
public async getVerbsByTarget(targetId: string): Promise<GraphVerb[]> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.getVerbsByTarget_internal(targetId)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get verbs by type
|
|
|
|
|
*/
|
|
|
|
|
public async getVerbsByType(type: string): Promise<GraphVerb[]> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.getVerbsByType_internal(type)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Delete a verb from storage
|
|
|
|
|
*/
|
|
|
|
|
public async deleteVerb(id: string): Promise<void> {
|
|
|
|
|
await this.ensureInitialized()
|
2025-07-25 09:44:10 -07:00
|
|
|
return this.deleteVerb_internal(id)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Clear all data from storage
|
|
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
|
|
|
|
public abstract clear(): Promise<void>
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get information about storage usage and capacity
|
|
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
|
|
|
|
public abstract getStorageStatus(): Promise<{
|
|
|
|
|
type: string
|
|
|
|
|
used: number
|
|
|
|
|
quota: number | null
|
|
|
|
|
details?: Record<string, any>
|
|
|
|
|
}>
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Save metadata to storage
|
|
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
|
|
|
|
public abstract saveMetadata(id: string, metadata: any): Promise<void>
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get metadata from storage
|
|
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
|
|
|
|
public abstract getMetadata(id: string): Promise<any | null>
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Save a noun to storage
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract saveNoun_internal(noun: HNSWNoun): Promise<void>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get a noun from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract getNoun_internal(id: string): Promise<HNSWNoun | null>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get all nouns from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract getAllNouns_internal(): Promise<HNSWNoun[]>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get nouns by noun type
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract getNounsByNounType_internal(nounType: string): Promise<HNSWNoun[]>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Delete a noun from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract deleteNoun_internal(id: string): Promise<void>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Save a verb to storage
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract saveVerb_internal(verb: GraphVerb): Promise<void>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get a verb from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract getVerb_internal(id: string): Promise<GraphVerb | null>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get all verbs from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract getAllVerbs_internal(): Promise<GraphVerb[]>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get verbs by source
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract getVerbsBySource_internal(sourceId: string): Promise<GraphVerb[]>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get verbs by target
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract getVerbsByTarget_internal(targetId: string): Promise<GraphVerb[]>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get verbs by type
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract getVerbsByType_internal(type: string): Promise<GraphVerb[]>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Delete a verb from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected abstract deleteVerb_internal(id: string): Promise<void>
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Helper method to convert a Map to a plain object for serialization
|
|
|
|
|
*/
|
|
|
|
|
protected mapToObject<K extends string | number, V>(
|
|
|
|
|
map: Map<K, V>,
|
|
|
|
|
valueTransformer: (value: V) => any = (v) => v
|
|
|
|
|
): Record<string, any> {
|
|
|
|
|
const obj: Record<string, any> = {}
|
|
|
|
|
for (const [key, value] of map.entries()) {
|
|
|
|
|
obj[key.toString()] = valueTransformer(value)
|
|
|
|
|
}
|
|
|
|
|
return obj
|
|
|
|
|
}
|
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Save statistics data to storage
|
|
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
* @param statistics The statistics data to save
|
|
|
|
|
*/
|
|
|
|
|
protected abstract saveStatisticsData(statistics: StatisticsData): Promise<void>
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get statistics data from storage
|
|
|
|
|
* This method should be implemented by each specific adapter
|
|
|
|
|
* @returns Promise that resolves to the statistics data or null if not found
|
|
|
|
|
*/
|
|
|
|
|
protected abstract getStatisticsData(): Promise<StatisticsData | null>
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|