2025-07-21 12:46:41 -07:00
/ * *
* Base Storage Adapter
* Provides common functionality for all storage adapters
* /
2025-08-03 10:47:47 -07:00
import { GraphVerb , HNSWNoun , HNSWVerb , StatisticsData } from '../coreTypes.js'
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
import { BaseStorageAdapter } from './adapters/baseStorageAdapter.js'
2025-07-21 12:46:41 -07:00
// Common directory/prefix names
export const NOUNS_DIR = 'nouns'
export const VERBS_DIR = 'verbs'
export const METADATA_DIR = 'metadata'
2025-08-02 16:41:30 -07:00
export const NOUN_METADATA_DIR = 'noun-metadata'
export const VERB_METADATA_DIR = 'verb-metadata'
2025-07-21 12:46:41 -07:00
export const INDEX_DIR = 'index'
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
export const STATISTICS_KEY = 'statistics'
2025-07-21 12:46:41 -07:00
/ * *
* Base storage adapter that implements common functionality
* This is an abstract class that should be extended by specific storage adapters
* /
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
export abstract class BaseStorage extends BaseStorageAdapter {
2025-07-21 12:46:41 -07:00
protected isInitialized = false
2025-08-05 07:22:05 -07:00
protected readOnly = false
2025-07-21 12:46:41 -07:00
/ * *
* Initialize the storage adapter
* This method should be implemented by each specific adapter
* /
public abstract init ( ) : Promise < void >
/ * *
* Ensure the storage adapter is initialized
* /
protected async ensureInitialized ( ) : Promise < void > {
if ( ! this . isInitialized ) {
await this . init ( )
}
}
/ * *
* Save a noun to storage
* /
public async saveNoun ( noun : HNSWNoun ) : Promise < void > {
await this . ensureInitialized ( )
2025-07-25 09:44:10 -07:00
return this . saveNoun_internal ( noun )
2025-07-21 12:46:41 -07:00
}
/ * *
* Get a noun from storage
* /
public async getNoun ( id : string ) : Promise < HNSWNoun | null > {
await this . ensureInitialized ( )
2025-07-25 09:44:10 -07:00
return this . getNoun_internal ( id )
2025-07-21 12:46:41 -07:00
}
/ * *
* Get all nouns from storage
2025-07-31 17:57:14 -07:00
* @deprecated This method is deprecated and will be removed in a future version .
* It can cause memory issues with large datasets . Use getNouns ( ) with pagination instead .
2025-07-21 12:46:41 -07:00
* /
public async getAllNouns ( ) : Promise < HNSWNoun [ ] > {
await this . ensureInitialized ( )
2025-07-31 17:57:14 -07:00
console . warn ( 'WARNING: getAllNouns() is deprecated and will be removed in a future version. Use getNouns() with pagination instead.' )
2025-07-25 09:44:10 -07:00
return this . getAllNouns_internal ( )
2025-07-21 12:46:41 -07:00
}
/ * *
* Get nouns by noun type
* @param nounType The noun type to filter by
* @returns Promise that resolves to an array of nouns of the specified noun type
* /
public async getNounsByNounType ( nounType : string ) : Promise < HNSWNoun [ ] > {
await this . ensureInitialized ( )
2025-07-25 09:44:10 -07:00
return this . getNounsByNounType_internal ( nounType )
2025-07-21 12:46:41 -07:00
}
/ * *
* Delete a noun from storage
* /
public async deleteNoun ( id : string ) : Promise < void > {
await this . ensureInitialized ( )
2025-07-25 09:44:10 -07:00
return this . deleteNoun_internal ( id )
2025-07-21 12:46:41 -07:00
}
/ * *
* Save a verb to storage
* /
public async saveVerb ( verb : GraphVerb ) : Promise < void > {
await this . ensureInitialized ( )
2025-08-03 10:47:47 -07:00
// Extract the lightweight HNSWVerb data
const hnswVerb : HNSWVerb = {
id : verb.id ,
vector : verb.vector ,
connections : verb.connections || new Map ( )
}
// Extract and save the metadata separately
const metadata = {
sourceId : verb.sourceId || verb . source ,
targetId : verb.targetId || verb . target ,
source : verb.source || verb . sourceId ,
target : verb.target || verb . targetId ,
type : verb . type || verb . verb ,
verb : verb.verb || verb . type ,
weight : verb.weight ,
metadata : verb.metadata ,
data : verb.data ,
createdAt : verb.createdAt ,
updatedAt : verb.updatedAt ,
createdBy : verb.createdBy ,
embedding : verb.embedding
}
// Save both the HNSWVerb and metadata
await this . saveVerb_internal ( hnswVerb )
await this . saveVerbMetadata ( verb . id , metadata )
2025-07-21 12:46:41 -07:00
}
/ * *
* Get a verb from storage
* /
public async getVerb ( id : string ) : Promise < GraphVerb | null > {
await this . ensureInitialized ( )
2025-08-03 10:47:47 -07:00
const hnswVerb = await this . getVerb_internal ( id )
if ( ! hnswVerb ) {
return null
}
return this . convertHNSWVerbToGraphVerb ( hnswVerb )
}
/ * *
* Convert HNSWVerb to GraphVerb by combining with metadata
* /
protected async convertHNSWVerbToGraphVerb ( hnswVerb : HNSWVerb ) : Promise < GraphVerb | null > {
try {
const metadata = await this . getVerbMetadata ( hnswVerb . id )
if ( ! metadata ) {
return null
}
// Create default timestamp if not present
const defaultTimestamp = {
seconds : Math.floor ( Date . now ( ) / 1000 ) ,
nanoseconds : ( Date . now ( ) % 1000 ) * 1000000
}
// Create default createdBy if not present
const defaultCreatedBy = {
augmentation : 'unknown' ,
version : '1.0'
}
return {
id : hnswVerb.id ,
vector : hnswVerb.vector ,
sourceId : metadata.sourceId ,
targetId : metadata.targetId ,
source : metadata.source ,
target : metadata.target ,
verb : metadata.verb ,
type : metadata . type ,
weight : metadata.weight || 1.0 ,
metadata : metadata.metadata || { } ,
createdAt : metadata.createdAt || defaultTimestamp ,
updatedAt : metadata.updatedAt || defaultTimestamp ,
createdBy : metadata.createdBy || defaultCreatedBy ,
data : metadata.data ,
embedding : hnswVerb.vector
}
} catch ( error ) {
console . error ( ` Failed to convert HNSWVerb to GraphVerb for ${ hnswVerb . id } : ` , error )
return null
}
2025-07-21 12:46:41 -07:00
}
/ * *
* Get all verbs from storage
2025-07-31 17:57:14 -07:00
* @deprecated This method is deprecated and will be removed in a future version .
* It can cause memory issues with large datasets . Use getVerbs ( ) with pagination instead .
2025-07-21 12:46:41 -07:00
* /
public async getAllVerbs ( ) : Promise < GraphVerb [ ] > {
await this . ensureInitialized ( )
2025-07-31 17:57:14 -07:00
console . warn ( 'WARNING: getAllVerbs() is deprecated and will be removed in a future version. Use getVerbs() with pagination instead.' )
2025-08-03 10:47:47 -07:00
const hnswVerbs = await this . getAllVerbs_internal ( )
const graphVerbs : GraphVerb [ ] = [ ]
for ( const hnswVerb of hnswVerbs ) {
const graphVerb = await this . convertHNSWVerbToGraphVerb ( hnswVerb )
if ( graphVerb ) {
graphVerbs . push ( graphVerb )
}
}
return graphVerbs
2025-07-21 12:46:41 -07:00
}
/ * *
* Get verbs by source
* /
public async getVerbsBySource ( sourceId : string ) : Promise < GraphVerb [ ] > {
await this . ensureInitialized ( )
2025-08-03 10:47:47 -07:00
// Get all verbs and filter by source
const allVerbs = await this . getAllVerbs ( )
return allVerbs . filter ( verb = >
verb . sourceId === sourceId || verb . source === sourceId
)
2025-07-21 12:46:41 -07:00
}
/ * *
* Get verbs by target
* /
public async getVerbsByTarget ( targetId : string ) : Promise < GraphVerb [ ] > {
await this . ensureInitialized ( )
2025-08-03 10:47:47 -07:00
// Get all verbs and filter by target
const allVerbs = await this . getAllVerbs ( )
return allVerbs . filter ( verb = >
verb . targetId === targetId || verb . target === targetId
)
2025-07-21 12:46:41 -07:00
}
/ * *
* Get verbs by type
* /
public async getVerbsByType ( type : string ) : Promise < GraphVerb [ ] > {
await this . ensureInitialized ( )
2025-08-03 10:47:47 -07:00
// Get all verbs and filter by type
const allVerbs = await this . getAllVerbs ( )
return allVerbs . filter ( verb = >
verb . type === type || verb . verb === type
)
2025-07-21 12:46:41 -07:00
}
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
/ * *
* Get nouns with pagination and filtering
* @param options Pagination and filtering options
* @returns Promise that resolves to a paginated result of nouns
* /
public async getNouns ( options ? : {
pagination ? : {
offset? : number
limit? : number
cursor? : string
}
filter ? : {
nounType? : string | string [ ]
service? : string | string [ ]
metadata? : Record < string , any >
}
} ) : Promise < {
items : HNSWNoun [ ]
totalCount? : number
hasMore : boolean
nextCursor? : string
} > {
await this . ensureInitialized ( )
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Set default pagination values
const pagination = options ? . pagination || { }
const limit = pagination . limit || 100
const offset = pagination . offset || 0
2025-07-31 17:57:14 -07:00
const cursor = pagination . cursor
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Optimize for common filter cases to avoid loading all nouns
if ( options ? . filter ) {
// If filtering by nounType only, use the optimized method
2025-07-31 17:57:14 -07:00
if (
options . filter . nounType &&
! options . filter . service &&
! options . filter . metadata
) {
const nounType = Array . isArray ( options . filter . nounType )
? options . filter . nounType [ 0 ]
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
: options . filter . nounType
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Get nouns by type directly
const nounsByType = await this . getNounsByNounType_internal ( nounType )
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Apply pagination
const paginatedNouns = nounsByType . slice ( offset , offset + limit )
const hasMore = offset + limit < nounsByType . length
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Set next cursor if there are more items
let nextCursor : string | undefined = undefined
if ( hasMore && paginatedNouns . length > 0 ) {
const lastItem = paginatedNouns [ paginatedNouns . length - 1 ]
nextCursor = lastItem . id
}
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
return {
items : paginatedNouns ,
totalCount : nounsByType.length ,
hasMore ,
nextCursor
}
}
}
2025-07-31 17:57:14 -07:00
// For more complex filtering or no filtering, use a paginated approach
// that avoids loading all nouns into memory at once
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
try {
2025-07-31 17:57:14 -07:00
// First, try to get a count of total nouns (if the adapter supports it)
let totalCount : number | undefined = undefined
try {
// This is an optional method that adapters may implement
if ( typeof ( this as any ) . countNouns === 'function' ) {
totalCount = await ( this as any ) . countNouns ( options ? . filter )
}
} catch ( countError ) {
// Ignore errors from count method, it's optional
console . warn ( 'Error getting noun count:' , countError )
}
// Check if the adapter has a paginated method for getting nouns
if ( typeof ( this as any ) . getNounsWithPagination === 'function' ) {
// Use the adapter's paginated method
const result = await ( this as any ) . getNounsWithPagination ( {
limit ,
cursor ,
filter : options?.filter
} )
// Apply offset if needed (some adapters might not support offset)
const items = result . items . slice ( offset )
return {
items ,
totalCount : result.totalCount || totalCount ,
hasMore : result.hasMore ,
nextCursor : result.nextCursor
}
}
// If the adapter doesn't have a paginated method, fall back to the old approach
// but with a warning and a reasonable limit
console . warn (
'Storage adapter does not support pagination, falling back to loading all nouns. This may cause performance issues with large datasets.'
)
// Get nouns with a reasonable limit to avoid memory issues
const maxNouns = Math . min ( offset + limit + 100 , 1000 ) // Reasonable limit
let allNouns : HNSWNoun [ ] = [ ]
try {
// Try to get only the nouns we need
allNouns = await this . getAllNouns_internal ( )
// If we have too many nouns, truncate the array to avoid memory issues
if ( allNouns . length > maxNouns ) {
console . warn (
` Large number of nouns ( ${ allNouns . length } ), truncating to ${ maxNouns } for filtering `
)
allNouns = allNouns . slice ( 0 , maxNouns )
}
} catch ( error ) {
console . error ( 'Error getting all nouns:' , error )
// Return empty result on error
return {
items : [ ] ,
totalCount : 0 ,
hasMore : false
}
}
// Apply filtering if needed
let filteredNouns = allNouns
if ( options ? . filter ) {
// Filter by noun type
if ( options . filter . nounType ) {
const nounTypes = Array . isArray ( options . filter . nounType )
? options . filter . nounType
: [ options . filter . nounType ]
filteredNouns = filteredNouns . filter ( ( noun ) = > {
// HNSWNoun doesn't have a type property directly, check metadata
const nounType = noun . metadata ? . type
return typeof nounType === 'string' && nounTypes . includes ( nounType )
} )
}
// Filter by service
if ( options . filter . service ) {
const services = Array . isArray ( options . filter . service )
? options . filter . service
: [ options . filter . service ]
filteredNouns = filteredNouns . filter ( ( noun ) = > {
// HNSWNoun doesn't have a service property directly, check metadata
const service = noun . metadata ? . service
return typeof service === 'string' && services . includes ( service )
} )
}
// Filter by metadata
if ( options . filter . metadata ) {
const metadataFilter = options . filter . metadata
filteredNouns = filteredNouns . filter ( ( noun ) = > {
if ( ! noun . metadata ) return false
// Check if all metadata keys match
return Object . entries ( metadataFilter ) . every (
( [ key , value ] ) = > noun . metadata && noun . metadata [ key ] === value
)
} )
}
}
// Get total count before pagination
totalCount = totalCount || filteredNouns . length
// Apply pagination
const paginatedNouns = filteredNouns . slice ( offset , offset + limit )
const hasMore = offset + limit < filteredNouns . length || filteredNouns . length >= maxNouns
// Set next cursor if there are more items
let nextCursor : string | undefined = undefined
if ( hasMore && paginatedNouns . length > 0 ) {
const lastItem = paginatedNouns [ paginatedNouns . length - 1 ]
nextCursor = lastItem . id
}
return {
items : paginatedNouns ,
totalCount ,
hasMore ,
nextCursor
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
}
} catch ( error ) {
2025-07-31 17:57:14 -07:00
console . error ( 'Error getting nouns with pagination:' , error )
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
return {
items : [ ] ,
totalCount : 0 ,
hasMore : false
}
}
}
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
/ * *
* Get verbs with pagination and filtering
* @param options Pagination and filtering options
* @returns Promise that resolves to a paginated result of verbs
* /
public async getVerbs ( options ? : {
pagination ? : {
offset? : number
limit? : number
cursor? : string
}
filter ? : {
verbType? : string | string [ ]
sourceId? : string | string [ ]
targetId? : string | string [ ]
service? : string | string [ ]
metadata? : Record < string , any >
}
} ) : Promise < {
items : GraphVerb [ ]
totalCount? : number
hasMore : boolean
nextCursor? : string
} > {
await this . ensureInitialized ( )
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Set default pagination values
const pagination = options ? . pagination || { }
const limit = pagination . limit || 100
const offset = pagination . offset || 0
2025-07-31 17:57:14 -07:00
const cursor = pagination . cursor
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Optimize for common filter cases to avoid loading all verbs
if ( options ? . filter ) {
// If filtering by sourceId only, use the optimized method
2025-07-31 17:57:14 -07:00
if (
options . filter . sourceId &&
! options . filter . verbType &&
! options . filter . targetId &&
! options . filter . service &&
! options . filter . metadata
) {
const sourceId = Array . isArray ( options . filter . sourceId )
? options . filter . sourceId [ 0 ]
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
: options . filter . sourceId
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Get verbs by source directly
const verbsBySource = await this . getVerbsBySource_internal ( sourceId )
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Apply pagination
const paginatedVerbs = verbsBySource . slice ( offset , offset + limit )
const hasMore = offset + limit < verbsBySource . length
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Set next cursor if there are more items
let nextCursor : string | undefined = undefined
if ( hasMore && paginatedVerbs . length > 0 ) {
const lastItem = paginatedVerbs [ paginatedVerbs . length - 1 ]
nextCursor = lastItem . id
}
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
return {
items : paginatedVerbs ,
totalCount : verbsBySource.length ,
hasMore ,
nextCursor
}
}
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// If filtering by targetId only, use the optimized method
2025-07-31 17:57:14 -07:00
if (
options . filter . targetId &&
! options . filter . verbType &&
! options . filter . sourceId &&
! options . filter . service &&
! options . filter . metadata
) {
const targetId = Array . isArray ( options . filter . targetId )
? options . filter . targetId [ 0 ]
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
: options . filter . targetId
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Get verbs by target directly
const verbsByTarget = await this . getVerbsByTarget_internal ( targetId )
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Apply pagination
const paginatedVerbs = verbsByTarget . slice ( offset , offset + limit )
const hasMore = offset + limit < verbsByTarget . length
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Set next cursor if there are more items
let nextCursor : string | undefined = undefined
if ( hasMore && paginatedVerbs . length > 0 ) {
const lastItem = paginatedVerbs [ paginatedVerbs . length - 1 ]
nextCursor = lastItem . id
}
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
return {
items : paginatedVerbs ,
totalCount : verbsByTarget.length ,
hasMore ,
nextCursor
}
}
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// If filtering by verbType only, use the optimized method
2025-07-31 17:57:14 -07:00
if (
options . filter . verbType &&
! options . filter . sourceId &&
! options . filter . targetId &&
! options . filter . service &&
! options . filter . metadata
) {
const verbType = Array . isArray ( options . filter . verbType )
? options . filter . verbType [ 0 ]
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
: options . filter . verbType
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Get verbs by type directly
const verbsByType = await this . getVerbsByType_internal ( verbType )
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Apply pagination
const paginatedVerbs = verbsByType . slice ( offset , offset + limit )
const hasMore = offset + limit < verbsByType . length
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
// Set next cursor if there are more items
let nextCursor : string | undefined = undefined
if ( hasMore && paginatedVerbs . length > 0 ) {
const lastItem = paginatedVerbs [ paginatedVerbs . length - 1 ]
nextCursor = lastItem . id
}
2025-07-31 17:57:14 -07:00
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
return {
items : paginatedVerbs ,
totalCount : verbsByType.length ,
hasMore ,
nextCursor
}
}
}
2025-07-31 17:57:14 -07:00
// For more complex filtering or no filtering, use a paginated approach
// that avoids loading all verbs into memory at once
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
try {
2025-07-31 17:57:14 -07:00
// First, try to get a count of total verbs (if the adapter supports it)
let totalCount : number | undefined = undefined
try {
// This is an optional method that adapters may implement
if ( typeof ( this as any ) . countVerbs === 'function' ) {
totalCount = await ( this as any ) . countVerbs ( options ? . filter )
}
} catch ( countError ) {
// Ignore errors from count method, it's optional
console . warn ( 'Error getting verb count:' , countError )
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
}
2025-07-31 17:57:14 -07:00
// Check if the adapter has a paginated method for getting verbs
if ( typeof ( this as any ) . getVerbsWithPagination === 'function' ) {
// Use the adapter's paginated method
const result = await ( this as any ) . getVerbsWithPagination ( {
limit ,
cursor ,
filter : options?.filter
} )
// Apply offset if needed (some adapters might not support offset)
const items = result . items . slice ( offset )
return {
items ,
totalCount : result.totalCount || totalCount ,
hasMore : result.hasMore ,
nextCursor : result.nextCursor
}
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
}
2025-07-31 17:57:14 -07:00
// If the adapter doesn't have a paginated method, fall back to the old approach
// but with a warning and a reasonable limit
console . warn (
'Storage adapter does not support pagination, falling back to loading all verbs. This may cause performance issues with large datasets.'
)
// Get verbs with a reasonable limit to avoid memory issues
const maxVerbs = Math . min ( offset + limit + 100 , 1000 ) // Reasonable limit
let allVerbs : GraphVerb [ ] = [ ]
try {
// Try to get only the verbs we need
2025-08-03 10:47:47 -07:00
allVerbs = await this . getAllVerbs ( )
2025-07-31 17:57:14 -07:00
// If we have too many verbs, truncate the array to avoid memory issues
if ( allVerbs . length > maxVerbs ) {
console . warn (
` Large number of verbs ( ${ allVerbs . length } ), truncating to ${ maxVerbs } for filtering `
)
allVerbs = allVerbs . slice ( 0 , maxVerbs )
}
} catch ( error ) {
console . error ( 'Error getting all verbs:' , error )
// Return empty result on error
return {
items : [ ] ,
totalCount : 0 ,
hasMore : false
}
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
}
2025-07-31 17:57:14 -07:00
// Apply filtering if needed
let filteredVerbs = allVerbs
if ( options ? . filter ) {
// Filter by verb type
if ( options . filter . verbType ) {
const verbTypes = Array . isArray ( options . filter . verbType )
? options . filter . verbType
: [ options . filter . verbType ]
filteredVerbs = filteredVerbs . filter (
( verb ) = > verb . type !== undefined && verbTypes . includes ( verb . type )
)
}
// Filter by source ID
if ( options . filter . sourceId ) {
const sourceIds = Array . isArray ( options . filter . sourceId )
? options . filter . sourceId
: [ options . filter . sourceId ]
filteredVerbs = filteredVerbs . filter (
( verb ) = >
verb . sourceId !== undefined && sourceIds . includes ( verb . sourceId )
)
}
// Filter by target ID
if ( options . filter . targetId ) {
const targetIds = Array . isArray ( options . filter . targetId )
? options . filter . targetId
: [ options . filter . targetId ]
filteredVerbs = filteredVerbs . filter (
( verb ) = >
verb . targetId !== undefined && targetIds . includes ( verb . targetId )
)
}
// Filter by service
if ( options . filter . service ) {
const services = Array . isArray ( options . filter . service )
? options . filter . service
: [ options . filter . service ]
filteredVerbs = filteredVerbs . filter ( ( verb ) = > {
// GraphVerb doesn't have a service property directly, check metadata
const service = verb . metadata ? . service
return typeof service === 'string' && services . includes ( service )
} )
}
// Filter by metadata
if ( options . filter . metadata ) {
const metadataFilter = options . filter . metadata
filteredVerbs = filteredVerbs . filter ( ( verb ) = > {
if ( ! verb . metadata ) return false
// Check if all metadata keys match
return Object . entries ( metadataFilter ) . every (
( [ key , value ] ) = > verb . metadata && verb . metadata [ key ] === value
)
} )
}
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
}
2025-07-31 17:57:14 -07:00
// Get total count before pagination
totalCount = totalCount || filteredVerbs . length
// Apply pagination
const paginatedVerbs = filteredVerbs . slice ( offset , offset + limit )
const hasMore = offset + limit < filteredVerbs . length || filteredVerbs . length >= maxVerbs
// Set next cursor if there are more items
let nextCursor : string | undefined = undefined
if ( hasMore && paginatedVerbs . length > 0 ) {
const lastItem = paginatedVerbs [ paginatedVerbs . length - 1 ]
nextCursor = lastItem . id
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
}
2025-07-31 17:57:14 -07:00
return {
items : paginatedVerbs ,
totalCount ,
hasMore ,
nextCursor
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
}
2025-07-31 17:57:14 -07:00
} catch ( error ) {
console . error ( 'Error getting verbs with pagination:' , error )
return {
items : [ ] ,
totalCount : 0 ,
hasMore : false
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
}
}
}
2025-07-21 12:46:41 -07:00
/ * *
* Delete a verb from storage
* /
public async deleteVerb ( id : string ) : Promise < void > {
await this . ensureInitialized ( )
2025-07-25 09:44:10 -07:00
return this . deleteVerb_internal ( id )
2025-07-21 12:46:41 -07:00
}
/ * *
* Clear all data from storage
* This method should be implemented by each specific adapter
* /
public abstract clear ( ) : Promise < void >
/ * *
* Get information about storage usage and capacity
* This method should be implemented by each specific adapter
* /
public abstract getStorageStatus ( ) : Promise < {
type : string
used : number
quota : number | null
details? : Record < string , any >
} >
/ * *
* Save metadata to storage
* This method should be implemented by each specific adapter
* /
public abstract saveMetadata ( id : string , metadata : any ) : Promise < void >
/ * *
* Get metadata from storage
* This method should be implemented by each specific adapter
* /
public abstract getMetadata ( id : string ) : Promise < any | null >
2025-08-02 16:41:30 -07:00
/ * *
* Save noun metadata to storage
* This method should be implemented by each specific adapter
* /
public abstract saveNounMetadata ( id : string , metadata : any ) : Promise < void >
/ * *
* Get noun metadata from storage
* This method should be implemented by each specific adapter
* /
public abstract getNounMetadata ( id : string ) : Promise < any | null >
/ * *
* Save verb metadata to storage
* This method should be implemented by each specific adapter
* /
public abstract saveVerbMetadata ( id : string , metadata : any ) : Promise < void >
/ * *
* Get verb metadata from storage
* This method should be implemented by each specific adapter
* /
public abstract getVerbMetadata ( id : string ) : Promise < any | null >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Save a noun to storage
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-07-25 09:44:10 -07:00
protected abstract saveNoun_internal ( noun : HNSWNoun ) : Promise < void >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Get a noun from storage
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-07-25 09:44:10 -07:00
protected abstract getNoun_internal ( id : string ) : Promise < HNSWNoun | null >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Get all nouns from storage
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-07-25 09:44:10 -07:00
protected abstract getAllNouns_internal ( ) : Promise < HNSWNoun [ ] >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Get nouns by noun type
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-07-31 17:57:14 -07:00
protected abstract getNounsByNounType_internal (
nounType : string
) : Promise < HNSWNoun [ ] >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Delete a noun from storage
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-07-25 09:44:10 -07:00
protected abstract deleteNoun_internal ( id : string ) : Promise < void >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Save a verb to storage
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-08-03 10:47:47 -07:00
protected abstract saveVerb_internal ( verb : HNSWVerb ) : Promise < void >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Get a verb from storage
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-08-03 10:47:47 -07:00
protected abstract getVerb_internal ( id : string ) : Promise < HNSWVerb | null >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Get all verbs from storage
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-08-03 10:47:47 -07:00
protected abstract getAllVerbs_internal ( ) : Promise < HNSWVerb [ ] >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Get verbs by source
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-07-31 17:57:14 -07:00
protected abstract getVerbsBySource_internal (
sourceId : string
) : Promise < GraphVerb [ ] >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Get verbs by target
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-07-31 17:57:14 -07:00
protected abstract getVerbsByTarget_internal (
targetId : string
) : Promise < GraphVerb [ ] >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Get verbs by type
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-07-25 09:44:10 -07:00
protected abstract getVerbsByType_internal ( type : string ) : Promise < GraphVerb [ ] >
2025-07-21 12:46:41 -07:00
/ * *
2025-07-25 09:44:10 -07:00
* Delete a verb from storage
2025-07-21 12:46:41 -07:00
* This method should be implemented by each specific adapter
* /
2025-07-25 09:44:10 -07:00
protected abstract deleteVerb_internal ( id : string ) : Promise < void >
2025-07-21 12:46:41 -07:00
/ * *
* Helper method to convert a Map to a plain object for serialization
* /
protected mapToObject < K extends string | number , V > (
map : Map < K , V > ,
valueTransformer : ( value : V ) = > any = ( v ) = > v
) : Record < string , any > {
const obj : Record < string , any > = { }
for ( const [ key , value ] of map . entries ( ) ) {
obj [ key . toString ( ) ] = valueTransformer ( value )
}
return obj
}
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
/ * *
* Save statistics data to storage
* This method should be implemented by each specific adapter
* @param statistics The statistics data to save
* /
2025-07-31 17:57:14 -07:00
protected abstract saveStatisticsData (
statistics : StatisticsData
) : Promise < void >
**feat(core, storage, tests): add service-level statistics tracking and storage adapter enhancements**
- **Core**: Enhanced `getStatistics` function to support `service` and `service[]` filters, enabling statistics breakdown by service. Modified return structure to include `serviceBreakdown` for detailed insights.
- **Storage**: Implemented a new `BaseStorageAdapter` abstract class to centralize statistics-related functionality, such as incrementing/decrementing counters and updating HNSW index size. Refactored all storage adapters (`FileSystemStorage`, `S3CompatibleStorage`, `MemoryStorage`, `OPFSStorage`) to extend `BaseStorageAdapter`, ensuring consistent statistics tracking.
- **Tests**: Added new test cases in `statistics.test.ts` to validate service-level statistics tracking, breakdown accuracy, and multi-service filtering.
**Purpose**: Improve insight into data trends by tracking service-specific usage in statistics. Enhance maintainability and consistency through storage adapter centralization and robust testing.
2025-07-24 11:35:52 -07:00
/ * *
* Get statistics data from storage
* This method should be implemented by each specific adapter
* @returns Promise that resolves to the statistics data or null if not found
* /
protected abstract getStatisticsData ( ) : Promise < StatisticsData | null >
**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.
**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
2025-07-31 13:13:15 -07:00
}