2025-07-21 12:46:41 -07:00
|
|
|
/**
|
|
|
|
|
* Memory Storage Adapter
|
|
|
|
|
* In-memory storage adapter for environments where persistent storage is not available or needed
|
|
|
|
|
*/
|
|
|
|
|
|
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
|
|
|
import { GraphVerb, HNSWNoun, StatisticsData } from '../../coreTypes.js'
|
|
|
|
|
import { BaseStorage, STATISTICS_KEY } from '../baseStorage.js'
|
2025-07-21 12:46:41 -07:00
|
|
|
|
2025-07-25 11:03:28 -07:00
|
|
|
// No type aliases needed - using the original types directly
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* In-memory storage adapter
|
|
|
|
|
* Uses Maps to store data in memory
|
|
|
|
|
*/
|
|
|
|
|
export class MemoryStorage extends BaseStorage {
|
|
|
|
|
// Single map of noun ID to noun
|
2025-07-25 11:03:28 -07:00
|
|
|
private nouns: Map<string, HNSWNoun> = new Map()
|
|
|
|
|
private verbs: Map<string, GraphVerb> = new Map()
|
2025-07-21 12:46:41 -07:00
|
|
|
private metadata: Map<string, any> = new Map()
|
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
|
|
|
private statistics: StatisticsData | null = null
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
constructor() {
|
|
|
|
|
super()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Initialize the storage adapter
|
|
|
|
|
* Nothing to initialize for in-memory storage
|
|
|
|
|
*/
|
|
|
|
|
public async init(): Promise<void> {
|
|
|
|
|
this.isInitialized = true
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Save a noun to storage
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 11:03:28 -07:00
|
|
|
protected async saveNoun_internal(noun: HNSWNoun): Promise<void> {
|
2025-07-21 12:46:41 -07:00
|
|
|
// Create a deep copy to avoid reference issues
|
2025-07-25 11:03:28 -07:00
|
|
|
const nounCopy: HNSWNoun = {
|
2025-07-25 09:44:10 -07:00
|
|
|
id: noun.id,
|
|
|
|
|
vector: [...noun.vector],
|
2025-07-21 12:46:41 -07:00
|
|
|
connections: new Map()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Copy connections
|
2025-07-25 09:44:10 -07:00
|
|
|
for (const [level, connections] of noun.connections.entries()) {
|
|
|
|
|
nounCopy.connections.set(level, new Set(connections))
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
// Save the noun directly in the nouns map
|
|
|
|
|
this.nouns.set(noun.id, nounCopy)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get a noun from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 11:03:28 -07:00
|
|
|
protected async getNoun_internal(id: string): Promise<HNSWNoun | null> {
|
2025-07-25 09:44:10 -07:00
|
|
|
// Get the noun directly from the nouns map
|
|
|
|
|
const noun = this.nouns.get(id)
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
// If not found, return null
|
2025-07-25 09:44:10 -07:00
|
|
|
if (!noun) {
|
2025-07-21 12:46:41 -07:00
|
|
|
return null
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Return a deep copy to avoid reference issues
|
2025-07-25 11:03:28 -07:00
|
|
|
const nounCopy: HNSWNoun = {
|
2025-07-25 09:44:10 -07:00
|
|
|
id: noun.id,
|
|
|
|
|
vector: [...noun.vector],
|
2025-07-21 12:46:41 -07:00
|
|
|
connections: new Map()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Copy connections
|
2025-07-25 09:44:10 -07:00
|
|
|
for (const [level, connections] of noun.connections.entries()) {
|
|
|
|
|
nounCopy.connections.set(level, new Set(connections))
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
return nounCopy
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get all nouns from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 11:03:28 -07:00
|
|
|
protected async getAllNouns_internal(): Promise<HNSWNoun[]> {
|
|
|
|
|
const allNouns: HNSWNoun[] = []
|
2025-07-21 12:46:41 -07:00
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
// Iterate through all nouns in the nouns map
|
|
|
|
|
for (const [nounId, noun] of this.nouns.entries()) {
|
2025-07-21 12:46:41 -07:00
|
|
|
// Return a deep copy to avoid reference issues
|
2025-07-25 11:03:28 -07:00
|
|
|
const nounCopy: HNSWNoun = {
|
2025-07-25 09:44:10 -07:00
|
|
|
id: noun.id,
|
|
|
|
|
vector: [...noun.vector],
|
2025-07-21 12:46:41 -07:00
|
|
|
connections: new Map()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Copy connections
|
2025-07-25 09:44:10 -07:00
|
|
|
for (const [level, connections] of noun.connections.entries()) {
|
|
|
|
|
nounCopy.connections.set(level, new Set(connections))
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
allNouns.push(nounCopy)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
return allNouns
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get nouns by noun type
|
2025-07-21 12:46:41 -07:00
|
|
|
* @param nounType The noun type to filter by
|
2025-07-25 09:44:10 -07:00
|
|
|
* @returns Promise that resolves to an array of nouns of the specified noun type
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 11:03:28 -07:00
|
|
|
protected async getNounsByNounType_internal(nounType: string): Promise<HNSWNoun[]> {
|
|
|
|
|
const nouns: HNSWNoun[] = []
|
2025-07-21 12:46:41 -07:00
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
// Iterate through all nouns and filter by noun type using metadata
|
|
|
|
|
for (const [nounId, noun] of this.nouns.entries()) {
|
2025-07-21 12:46:41 -07:00
|
|
|
// Get the metadata to check the noun type
|
2025-07-25 09:44:10 -07:00
|
|
|
const metadata = await this.getMetadata(nounId)
|
2025-07-21 12:46:41 -07:00
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
// Include the noun if its noun type matches the requested type
|
2025-07-21 12:46:41 -07:00
|
|
|
if (metadata && metadata.noun === nounType) {
|
|
|
|
|
// Return a deep copy to avoid reference issues
|
2025-07-25 11:03:28 -07:00
|
|
|
const nounCopy: HNSWNoun = {
|
2025-07-25 09:44:10 -07:00
|
|
|
id: noun.id,
|
|
|
|
|
vector: [...noun.vector],
|
2025-07-21 12:46:41 -07:00
|
|
|
connections: new Map()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Copy connections
|
2025-07-25 09:44:10 -07:00
|
|
|
for (const [level, connections] of noun.connections.entries()) {
|
|
|
|
|
nounCopy.connections.set(level, new Set(connections))
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
nouns.push(nounCopy)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
return nouns
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Delete a noun from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected async deleteNoun_internal(id: string): Promise<void> {
|
2025-07-21 12:46:41 -07:00
|
|
|
this.nouns.delete(id)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Save a verb to storage
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 11:03:28 -07:00
|
|
|
protected async saveVerb_internal(verb: GraphVerb): Promise<void> {
|
2025-07-21 12:46:41 -07:00
|
|
|
// Create a deep copy to avoid reference issues
|
2025-07-25 11:03:28 -07:00
|
|
|
const verbCopy: GraphVerb = {
|
2025-07-25 09:44:10 -07:00
|
|
|
id: verb.id,
|
|
|
|
|
vector: [...verb.vector],
|
2025-07-21 12:46:41 -07:00
|
|
|
connections: new Map(),
|
2025-07-25 09:44:10 -07:00
|
|
|
sourceId: verb.sourceId,
|
|
|
|
|
targetId: verb.targetId,
|
|
|
|
|
type: verb.type,
|
|
|
|
|
weight: verb.weight,
|
|
|
|
|
metadata: verb.metadata
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Copy connections
|
2025-07-25 09:44:10 -07:00
|
|
|
for (const [level, connections] of verb.connections.entries()) {
|
|
|
|
|
verbCopy.connections.set(level, new Set(connections))
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
// Save the verb directly in the verbs map
|
|
|
|
|
this.verbs.set(verb.id, verbCopy)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get a verb from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 11:03:28 -07:00
|
|
|
protected async getVerb_internal(id: string): Promise<GraphVerb | null> {
|
2025-07-25 09:44:10 -07:00
|
|
|
// Get the verb directly from the verbs map
|
|
|
|
|
const verb = this.verbs.get(id)
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
// If not found, return null
|
2025-07-25 09:44:10 -07:00
|
|
|
if (!verb) {
|
2025-07-21 12:46:41 -07:00
|
|
|
return null
|
|
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
// Create default timestamp if not present
|
|
|
|
|
const defaultTimestamp = {
|
|
|
|
|
seconds: Math.floor(Date.now() / 1000),
|
|
|
|
|
nanoseconds: (Date.now() % 1000) * 1000000
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Create default createdBy if not present
|
|
|
|
|
const defaultCreatedBy = {
|
|
|
|
|
augmentation: 'unknown',
|
|
|
|
|
version: '1.0'
|
|
|
|
|
}
|
|
|
|
|
|
2025-07-21 12:46:41 -07:00
|
|
|
// Return a deep copy to avoid reference issues
|
2025-07-25 11:03:28 -07:00
|
|
|
const verbCopy: GraphVerb = {
|
2025-07-25 09:44:10 -07:00
|
|
|
id: verb.id,
|
|
|
|
|
vector: [...verb.vector],
|
2025-07-21 12:46:41 -07:00
|
|
|
connections: new Map(),
|
2025-07-25 09:44:10 -07:00
|
|
|
sourceId: (verb.sourceId || verb.source || ""),
|
|
|
|
|
targetId: (verb.targetId || verb.target || ""),
|
|
|
|
|
source: (verb.sourceId || verb.source || ""),
|
|
|
|
|
target: (verb.targetId || verb.target || ""),
|
|
|
|
|
verb: verb.type || verb.verb,
|
|
|
|
|
weight: verb.weight,
|
|
|
|
|
metadata: verb.metadata,
|
|
|
|
|
createdAt: verb.createdAt || defaultTimestamp,
|
|
|
|
|
updatedAt: verb.updatedAt || defaultTimestamp,
|
|
|
|
|
createdBy: verb.createdBy || defaultCreatedBy
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Copy connections
|
2025-07-25 09:44:10 -07:00
|
|
|
for (const [level, connections] of verb.connections.entries()) {
|
|
|
|
|
verbCopy.connections.set(level, new Set(connections))
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
return verbCopy
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get all verbs from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 11:03:28 -07:00
|
|
|
protected async getAllVerbs_internal(): Promise<GraphVerb[]> {
|
|
|
|
|
const allVerbs: GraphVerb[] = []
|
2025-07-25 09:44:10 -07:00
|
|
|
|
|
|
|
|
// Iterate through all verbs in the verbs map
|
|
|
|
|
for (const [verbId, verb] of this.verbs.entries()) {
|
|
|
|
|
// Create default timestamp if not present
|
|
|
|
|
const defaultTimestamp = {
|
|
|
|
|
seconds: Math.floor(Date.now() / 1000),
|
|
|
|
|
nanoseconds: (Date.now() % 1000) * 1000000
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Create default createdBy if not present
|
|
|
|
|
const defaultCreatedBy = {
|
|
|
|
|
augmentation: 'unknown',
|
|
|
|
|
version: '1.0'
|
|
|
|
|
}
|
2025-07-21 12:46:41 -07:00
|
|
|
|
|
|
|
|
// Return a deep copy to avoid reference issues
|
2025-07-25 11:03:28 -07:00
|
|
|
const verbCopy: GraphVerb = {
|
2025-07-25 09:44:10 -07:00
|
|
|
id: verb.id,
|
|
|
|
|
vector: [...verb.vector],
|
2025-07-21 12:46:41 -07:00
|
|
|
connections: new Map(),
|
2025-07-25 09:44:10 -07:00
|
|
|
sourceId: (verb.sourceId || verb.source || ""),
|
|
|
|
|
targetId: (verb.targetId || verb.target || ""),
|
|
|
|
|
source: (verb.sourceId || verb.source || ""),
|
|
|
|
|
target: (verb.targetId || verb.target || ""),
|
|
|
|
|
verb: verb.type || verb.verb,
|
|
|
|
|
weight: verb.weight,
|
|
|
|
|
metadata: verb.metadata,
|
|
|
|
|
createdAt: verb.createdAt || defaultTimestamp,
|
|
|
|
|
updatedAt: verb.updatedAt || defaultTimestamp,
|
|
|
|
|
createdBy: verb.createdBy || defaultCreatedBy
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Copy connections
|
2025-07-25 09:44:10 -07:00
|
|
|
for (const [level, connections] of verb.connections.entries()) {
|
|
|
|
|
verbCopy.connections.set(level, new Set(connections))
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
allVerbs.push(verbCopy)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
2025-07-25 09:44:10 -07:00
|
|
|
return allVerbs
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get verbs by source
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 11:03:28 -07:00
|
|
|
protected async getVerbsBySource_internal(sourceId: string): Promise<GraphVerb[]> {
|
2025-07-25 09:44:10 -07:00
|
|
|
const allVerbs = await this.getAllVerbs_internal()
|
2025-07-25 11:03:28 -07:00
|
|
|
return allVerbs.filter((verb: GraphVerb) => (verb.sourceId || verb.source) === sourceId)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get verbs by target
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 11:03:28 -07:00
|
|
|
protected async getVerbsByTarget_internal(targetId: string): Promise<GraphVerb[]> {
|
2025-07-25 09:44:10 -07:00
|
|
|
const allVerbs = await this.getAllVerbs_internal()
|
2025-07-25 11:03:28 -07:00
|
|
|
return allVerbs.filter((verb: GraphVerb) => (verb.targetId || verb.target) === targetId)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Get verbs by type
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 11:03:28 -07:00
|
|
|
protected async getVerbsByType_internal(type: string): Promise<GraphVerb[]> {
|
2025-07-25 09:44:10 -07:00
|
|
|
const allVerbs = await this.getAllVerbs_internal()
|
2025-07-25 11:03:28 -07:00
|
|
|
return allVerbs.filter((verb: GraphVerb) => (verb.type || verb.verb) === type)
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
2025-07-25 09:44:10 -07:00
|
|
|
* Delete a verb from storage
|
2025-07-21 12:46:41 -07:00
|
|
|
*/
|
2025-07-25 09:44:10 -07:00
|
|
|
protected async deleteVerb_internal(id: string): Promise<void> {
|
|
|
|
|
// Delete the verb directly from the verbs map
|
2025-07-21 12:46:41 -07:00
|
|
|
this.verbs.delete(id)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Save metadata to storage
|
|
|
|
|
*/
|
|
|
|
|
public async saveMetadata(id: string, metadata: any): Promise<void> {
|
|
|
|
|
this.metadata.set(id, JSON.parse(JSON.stringify(metadata)))
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get metadata from storage
|
|
|
|
|
*/
|
|
|
|
|
public async getMetadata(id: string): Promise<any | null> {
|
|
|
|
|
const metadata = this.metadata.get(id)
|
|
|
|
|
if (!metadata) {
|
|
|
|
|
return null
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return JSON.parse(JSON.stringify(metadata))
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Clear all data from storage
|
|
|
|
|
*/
|
|
|
|
|
public async clear(): Promise<void> {
|
|
|
|
|
this.nouns.clear()
|
|
|
|
|
this.verbs.clear()
|
|
|
|
|
this.metadata.clear()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get information about storage usage and capacity
|
|
|
|
|
*/
|
|
|
|
|
public async getStorageStatus(): Promise<{
|
|
|
|
|
type: string
|
|
|
|
|
used: number
|
|
|
|
|
quota: number | null
|
|
|
|
|
details?: Record<string, any>
|
|
|
|
|
}> {
|
|
|
|
|
return {
|
|
|
|
|
type: 'memory',
|
|
|
|
|
used: 0, // In-memory storage doesn't have a meaningful size
|
|
|
|
|
quota: null, // In-memory storage doesn't have a quota
|
|
|
|
|
details: {
|
|
|
|
|
nodeCount: this.nouns.size,
|
|
|
|
|
edgeCount: this.verbs.size,
|
|
|
|
|
metadataCount: this.metadata.size
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Save statistics data to storage
|
|
|
|
|
* @param statistics The statistics data to save
|
|
|
|
|
*/
|
|
|
|
|
protected async saveStatisticsData(statistics: StatisticsData): Promise<void> {
|
**feat(tests, docs, storage): add statistics storage tests and enhance documentation**
- **Tests**: Added new `statistics-storage.test.ts` to validate statistics storage functionality across scenarios including saving, retrieving, time-based partitioning, and backward compatibility. Ensured tests dynamically handle missing environment variables by skipping S3-related tests when credentials are unavailable.
- **Docs**: Enhanced `statistics.md` with detailed explanations of scalability improvements, including adaptive flush timing, batched updates, and time-based partitioning. Improved readability and structure.
- **Storage**: Updated all storage adapters to integrate time-based partitioning and maintain backward compatibility with legacy statistics storage formats.
- **Dependencies**: Added `dotenv` to support environmental variable management for storage adapter tests.
**Purpose**: Strengthen system reliability by adding comprehensive test coverage for statistics storage, improve scalability documentation, and ensure consistency across storage adapters with robust implementations.
2025-07-24 16:24:02 -07:00
|
|
|
// For memory storage, we just need to store the statistics in memory
|
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
|
|
|
// Create a deep copy to avoid reference issues
|
|
|
|
|
this.statistics = {
|
|
|
|
|
nounCount: {...statistics.nounCount},
|
|
|
|
|
verbCount: {...statistics.verbCount},
|
|
|
|
|
metadataCount: {...statistics.metadataCount},
|
|
|
|
|
hnswIndexSize: statistics.hnswIndexSize,
|
|
|
|
|
lastUpdated: statistics.lastUpdated
|
|
|
|
|
}
|
**feat(tests, docs, storage): add statistics storage tests and enhance documentation**
- **Tests**: Added new `statistics-storage.test.ts` to validate statistics storage functionality across scenarios including saving, retrieving, time-based partitioning, and backward compatibility. Ensured tests dynamically handle missing environment variables by skipping S3-related tests when credentials are unavailable.
- **Docs**: Enhanced `statistics.md` with detailed explanations of scalability improvements, including adaptive flush timing, batched updates, and time-based partitioning. Improved readability and structure.
- **Storage**: Updated all storage adapters to integrate time-based partitioning and maintain backward compatibility with legacy statistics storage formats.
- **Dependencies**: Added `dotenv` to support environmental variable management for storage adapter tests.
**Purpose**: Strengthen system reliability by adding comprehensive test coverage for statistics storage, improve scalability documentation, and ensure consistency across storage adapters with robust implementations.
2025-07-24 16:24:02 -07:00
|
|
|
|
|
|
|
|
// Since this is in-memory, there's no need for time-based partitioning
|
|
|
|
|
// or legacy file handling
|
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Get statistics data from storage
|
|
|
|
|
* @returns Promise that resolves to the statistics data or null if not found
|
|
|
|
|
*/
|
|
|
|
|
protected async getStatisticsData(): Promise<StatisticsData | null> {
|
|
|
|
|
if (!this.statistics) {
|
|
|
|
|
return null
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Return a deep copy to avoid reference issues
|
|
|
|
|
return {
|
|
|
|
|
nounCount: {...this.statistics.nounCount},
|
|
|
|
|
verbCount: {...this.statistics.verbCount},
|
|
|
|
|
metadataCount: {...this.statistics.metadataCount},
|
|
|
|
|
hnswIndexSize: this.statistics.hnswIndexSize,
|
|
|
|
|
lastUpdated: this.statistics.lastUpdated
|
|
|
|
|
}
|
**feat(tests, docs, storage): add statistics storage tests and enhance documentation**
- **Tests**: Added new `statistics-storage.test.ts` to validate statistics storage functionality across scenarios including saving, retrieving, time-based partitioning, and backward compatibility. Ensured tests dynamically handle missing environment variables by skipping S3-related tests when credentials are unavailable.
- **Docs**: Enhanced `statistics.md` with detailed explanations of scalability improvements, including adaptive flush timing, batched updates, and time-based partitioning. Improved readability and structure.
- **Storage**: Updated all storage adapters to integrate time-based partitioning and maintain backward compatibility with legacy statistics storage formats.
- **Dependencies**: Added `dotenv` to support environmental variable management for storage adapter tests.
**Purpose**: Strengthen system reliability by adding comprehensive test coverage for statistics storage, improve scalability documentation, and ensure consistency across storage adapters with robust implementations.
2025-07-24 16:24:02 -07:00
|
|
|
|
|
|
|
|
// Since this is in-memory, there's no need for fallback mechanisms
|
|
|
|
|
// to check multiple storage locations
|
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
|
|
|
}
|
2025-07-21 12:46:41 -07:00
|
|
|
}
|