**feat(storage): add pagination and filtering support for nouns and verbs**

- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations.
- Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability.
- Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods.
- Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling.
- Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required.

**Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
This commit is contained in:
David Snelling 2025-07-31 13:13:15 -07:00
parent 2b0f830d89
commit cfa43a4160
8 changed files with 1309 additions and 143 deletions

View file

@ -456,11 +456,57 @@ export class HNSWIndex {
/**
* Get all nouns in the index
* @deprecated Use getNounsPaginated() instead for better scalability
*/
public getNouns(): Map<string, HNSWNoun> {
return new Map(this.nouns)
}
/**
* Get nouns with pagination
* @param options Pagination options
* @returns Object containing paginated nouns and pagination info
*/
public getNounsPaginated(
options: {
offset?: number
limit?: number
filter?: (noun: HNSWNoun) => boolean
} = {}
): {
items: Map<string, HNSWNoun>
totalCount: number
hasMore: boolean
} {
const offset = options.offset || 0
const limit = options.limit || 100
const filter = options.filter || (() => true)
// Get all noun entries
const entries = [...this.nouns.entries()]
// Apply filter if provided
const filteredEntries = entries.filter(([_, noun]) => filter(noun))
// Get total count after filtering
const totalCount = filteredEntries.length
// Apply pagination
const paginatedEntries = filteredEntries.slice(offset, offset + limit)
// Check if there are more items
const hasMore = offset + limit < totalCount
// Create a new map with the paginated entries
const items = new Map(paginatedEntries)
return {
items,
totalCount,
hasMore
}
}
/**
* Clear the index
*/