**feat(storage): add pagination and filtering support for nouns and verbs**
- Introduced `PaginationOptions`, `NounFilterOptions`, and `VerbFilterOptions` types for improved query flexibility in data retrieval operations. - Added `getNouns` and `getVerbs` methods with pagination and filtering capabilities, replacing existing methods for broader use cases and scalability. - Marked legacy methods (`getAllNouns`, `getAllVerbs`, `getVerbsBySource`, `getVerbsByTarget`, `getVerbsByType`) as deprecated, directing users to use new methods. - Updated `coreTypes`, `memoryStorage`, and related modules to support new functionality, including cursor and offset-based pagination handling. - Updated fallback logic for storage adapters, ensuring compatibility with non-paginated operations when required. **Purpose**: Enhance scalability and query precision by implementing paginated and filtered retrieval of nouns and verbs, aligning query methods with modern requirements.
This commit is contained in:
parent
2b0f830d89
commit
cfa43a4160
8 changed files with 1309 additions and 143 deletions
|
|
@ -456,11 +456,57 @@ export class HNSWIndex {
|
|||
|
||||
/**
|
||||
* Get all nouns in the index
|
||||
* @deprecated Use getNounsPaginated() instead for better scalability
|
||||
*/
|
||||
public getNouns(): Map<string, HNSWNoun> {
|
||||
return new Map(this.nouns)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get nouns with pagination
|
||||
* @param options Pagination options
|
||||
* @returns Object containing paginated nouns and pagination info
|
||||
*/
|
||||
public getNounsPaginated(
|
||||
options: {
|
||||
offset?: number
|
||||
limit?: number
|
||||
filter?: (noun: HNSWNoun) => boolean
|
||||
} = {}
|
||||
): {
|
||||
items: Map<string, HNSWNoun>
|
||||
totalCount: number
|
||||
hasMore: boolean
|
||||
} {
|
||||
const offset = options.offset || 0
|
||||
const limit = options.limit || 100
|
||||
const filter = options.filter || (() => true)
|
||||
|
||||
// Get all noun entries
|
||||
const entries = [...this.nouns.entries()]
|
||||
|
||||
// Apply filter if provided
|
||||
const filteredEntries = entries.filter(([_, noun]) => filter(noun))
|
||||
|
||||
// Get total count after filtering
|
||||
const totalCount = filteredEntries.length
|
||||
|
||||
// Apply pagination
|
||||
const paginatedEntries = filteredEntries.slice(offset, offset + limit)
|
||||
|
||||
// Check if there are more items
|
||||
const hasMore = offset + limit < totalCount
|
||||
|
||||
// Create a new map with the paginated entries
|
||||
const items = new Map(paginatedEntries)
|
||||
|
||||
return {
|
||||
items,
|
||||
totalCount,
|
||||
hasMore
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Clear the index
|
||||
*/
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue