feat(v4.0.0): Complete metadata/vector separation architecture with Azure support
This commit completes the core v4.0.0 architecture changes for billion-scale performance with metadata/vector separation. NO RELEASE YET - remaining optimizations and testing required before production release. ## Core v4.0.0 Architecture Changes ### Type System Updates - Fixed all TypeScript compilation errors (zero errors achieved) - Updated HNSWNoun/HNSWVerb to separate core fields from metadata - Implemented HNSWNounWithMetadata/HNSWVerbWithMetadata for API boundaries - Added required 'noun' field to NounMetadata for semantic structure - Renamed verb.type to verb.verb for consistency ### Storage Adapter Updates **All adapters updated for v4.0.0 two-file storage pattern:** - memoryStorage: Proper metadata/vector separation - fileSystemStorage: Two-file pattern with sharding - opfsStorage: Browser persistent storage updated - s3CompatibleStorage: AWS/MinIO/DigitalOcean support - r2Storage: Cloudflare R2 optimization - gcsStorage: Google Cloud with ADC support - **azureBlobStorage: NEW - Full Azure Blob Storage support** ### Storage Features - BaseStorage: Internal vs public method separation (_getNoun vs getNoun) - Two-file storage: Vectors in one file, metadata in another - Change tracking: getChangesSince return type updated - Pagination: getNounsWithPagination returns WithMetadata types ### Azure Blob Storage Integration (NEW) - Native @azure/storage-blob SDK integration - Four authentication methods: * DefaultAzureCredential (Managed Identity) - recommended * Connection String - simplest setup * Account Name + Key - traditional auth * SAS Token - delegated access - High-volume mode with write buffering - Adaptive backpressure for throttling - UUID-based sharding for billion-scale - Full HNSW support with graph persistence ### Utility Updates - EmbeddingManager: Updated to accept Record<string, unknown> - LSMTree: Wrapped data in NounMetadata structure with 'noun' field - EntityIdMapper: Fixed nested metadata.data structure access - MetadataIndex: Fixed field type inference integration - PeriodicCleanup: Updated for new metadata structure ### Core API Updates - Brainy: Updated verb property access from v.type to v.verb - ConfigAPI: Fixed NounMetadata access patterns - DataAPI: Updated metadata handling ### Documentation Updates - CREATING-AUGMENTATIONS.md: v4.0.0 breaking changes guide - DEVELOPER-GUIDE.md: Migration checklist and examples - COMPLETE-REFERENCE.md: v4.0.0 architecture improvements - **finite-type-system.md: NEW - Revolutionary type system benefits** ### Build & Dependencies - Zero TypeScript compilation errors - Added @azure/storage-blob and @azure/identity - 591 tests passing (23 timeout in long-running neural tests) ## What's NOT in This Release This is a work-in-progress commit. Before v4.0.0 release we need: - Storage adapter optimizations (batch operations, compression) - Azure blob tier management (Hot/Cool/Archive) - Cost optimization implementations - Additional performance testing at billion-scale - Migration guides for v3.x users ## Testing - Clean build: ✅ - Type checking: ✅ (zero errors) - Test suite: ✅ (591/614 passing, timeouts in neural tests only) 🔐 Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
8d6dd07e1d
commit
92c96246fb
35 changed files with 4524 additions and 1026 deletions
1553
src/storage/adapters/azureBlobStorage.ts
Normal file
1553
src/storage/adapters/azureBlobStorage.ts
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -3,7 +3,17 @@
|
|||
* Provides common functionality for all storage adapters, including statistics tracking
|
||||
*/
|
||||
|
||||
import { StatisticsData, StorageAdapter, HNSWNoun, GraphVerb } from '../../coreTypes.js'
|
||||
import {
|
||||
StatisticsData,
|
||||
StorageAdapter,
|
||||
HNSWNoun,
|
||||
HNSWVerb,
|
||||
GraphVerb,
|
||||
HNSWNounWithMetadata,
|
||||
HNSWVerbWithMetadata,
|
||||
NounMetadata,
|
||||
VerbMetadata
|
||||
} from '../../coreTypes.js'
|
||||
import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js'
|
||||
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
|
||||
|
||||
|
|
@ -15,22 +25,26 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
abstract init(): Promise<void>
|
||||
|
||||
abstract saveNoun(noun: HNSWNoun): Promise<void>
|
||||
abstract saveNounMetadata(id: string, metadata: NounMetadata): Promise<void>
|
||||
abstract deleteNounMetadata(id: string): Promise<void>
|
||||
|
||||
abstract getNoun(id: string): Promise<HNSWNoun | null>
|
||||
abstract getNoun(id: string): Promise<HNSWNounWithMetadata | null>
|
||||
|
||||
abstract getNounsByNounType(nounType: string): Promise<HNSWNoun[]>
|
||||
abstract getNounsByNounType(nounType: string): Promise<HNSWNounWithMetadata[]>
|
||||
|
||||
abstract deleteNoun(id: string): Promise<void>
|
||||
|
||||
abstract saveVerb(verb: GraphVerb): Promise<void>
|
||||
abstract saveVerb(verb: HNSWVerb): Promise<void>
|
||||
abstract saveVerbMetadata(id: string, metadata: VerbMetadata): Promise<void>
|
||||
abstract deleteVerbMetadata(id: string): Promise<void>
|
||||
|
||||
abstract getVerb(id: string): Promise<GraphVerb | null>
|
||||
abstract getVerb(id: string): Promise<HNSWVerbWithMetadata | null>
|
||||
|
||||
abstract getVerbsBySource(sourceId: string): Promise<GraphVerb[]>
|
||||
abstract getVerbsBySource(sourceId: string): Promise<HNSWVerbWithMetadata[]>
|
||||
|
||||
abstract getVerbsByTarget(targetId: string): Promise<GraphVerb[]>
|
||||
abstract getVerbsByTarget(targetId: string): Promise<HNSWVerbWithMetadata[]>
|
||||
|
||||
abstract getVerbsByType(type: string): Promise<GraphVerb[]>
|
||||
abstract getVerbsByType(type: string): Promise<HNSWVerbWithMetadata[]>
|
||||
|
||||
abstract deleteVerb(id: string): Promise<void>
|
||||
|
||||
|
|
@ -40,8 +54,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
|
||||
abstract getNounMetadata(id: string): Promise<any | null>
|
||||
|
||||
abstract saveVerbMetadata(id: string, metadata: any): Promise<void>
|
||||
|
||||
abstract getVerbMetadata(id: string): Promise<any | null>
|
||||
|
||||
// HNSW Index Persistence (v3.35.0+)
|
||||
|
|
@ -98,7 +110,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
}): Promise<{
|
||||
items: HNSWNoun[]
|
||||
items: HNSWNounWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -123,7 +135,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
}): Promise<{
|
||||
items: GraphVerb[]
|
||||
items: HNSWVerbWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -144,7 +156,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
}): Promise<{
|
||||
items: HNSWNoun[]
|
||||
items: HNSWNounWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -167,7 +179,7 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
}): Promise<{
|
||||
items: GraphVerb[]
|
||||
items: HNSWVerbWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
|
|||
|
|
@ -3,7 +3,16 @@
|
|||
* File system storage adapter for Node.js environments
|
||||
*/
|
||||
|
||||
import { GraphVerb, HNSWNoun, HNSWVerb, StatisticsData } from '../../coreTypes.js'
|
||||
import {
|
||||
GraphVerb,
|
||||
HNSWNoun,
|
||||
HNSWVerb,
|
||||
NounMetadata,
|
||||
VerbMetadata,
|
||||
HNSWNounWithMetadata,
|
||||
HNSWVerbWithMetadata,
|
||||
StatisticsData
|
||||
} from '../../coreTypes.js'
|
||||
import {
|
||||
BaseStorage,
|
||||
NOUNS_DIR,
|
||||
|
|
@ -244,9 +253,11 @@ export class FileSystemStorage extends BaseStorage {
|
|||
JSON.stringify(serializableNode, null, 2)
|
||||
)
|
||||
|
||||
// Update counts for new nodes (intelligent type detection)
|
||||
// Update counts for new nodes (v4.0.0: load metadata separately)
|
||||
if (isNew) {
|
||||
const type = node.metadata?.type || node.metadata?.nounType || 'default'
|
||||
// v4.0.0: Get type from separate metadata storage
|
||||
const metadata = await this.getNounMetadata(node.id)
|
||||
const type = metadata?.noun || 'default'
|
||||
this.incrementEntityCount(type)
|
||||
|
||||
// Persist counts periodically (every 10 operations for efficiency)
|
||||
|
|
@ -394,15 +405,15 @@ export class FileSystemStorage extends BaseStorage {
|
|||
|
||||
const filePath = this.getNodePath(id)
|
||||
|
||||
// Load node to get type for count update
|
||||
// Load metadata to get type for count update (v4.0.0: separate storage)
|
||||
try {
|
||||
const node = await this.getNode(id)
|
||||
if (node) {
|
||||
const type = node.metadata?.type || node.metadata?.nounType || 'default'
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
if (metadata) {
|
||||
const type = metadata.noun || 'default'
|
||||
this.decrementEntityCount(type)
|
||||
}
|
||||
} catch {
|
||||
// Node might not exist, that's ok
|
||||
// Metadata might not exist, that's ok
|
||||
}
|
||||
|
||||
try {
|
||||
|
|
@ -483,7 +494,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
connections.set(Number(level), new Set(nodeIds as string[]))
|
||||
}
|
||||
|
||||
// ARCHITECTURAL FIX (v3.50.1): Return HNSWVerb with core relational fields
|
||||
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
|
||||
return {
|
||||
id: parsedEdge.id,
|
||||
vector: parsedEdge.vector,
|
||||
|
|
@ -492,10 +503,10 @@ export class FileSystemStorage extends BaseStorage {
|
|||
// CORE RELATIONAL DATA (read from vector file)
|
||||
verb: parsedEdge.verb,
|
||||
sourceId: parsedEdge.sourceId,
|
||||
targetId: parsedEdge.targetId,
|
||||
targetId: parsedEdge.targetId
|
||||
|
||||
// User metadata (retrieved separately via getVerbMetadata())
|
||||
metadata: parsedEdge.metadata
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
// User metadata retrieved separately via getVerbMetadata()
|
||||
}
|
||||
} catch (error: any) {
|
||||
if (error.code !== 'ENOENT') {
|
||||
|
|
@ -535,7 +546,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
connections.set(Number(level), new Set(nodeIds as string[]))
|
||||
}
|
||||
|
||||
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields
|
||||
// v4.0.0: Include core relational fields (NO metadata field)
|
||||
allEdges.push({
|
||||
id: parsedEdge.id,
|
||||
vector: parsedEdge.vector,
|
||||
|
|
@ -544,10 +555,10 @@ export class FileSystemStorage extends BaseStorage {
|
|||
// CORE RELATIONAL DATA
|
||||
verb: parsedEdge.verb,
|
||||
sourceId: parsedEdge.sourceId,
|
||||
targetId: parsedEdge.targetId,
|
||||
targetId: parsedEdge.targetId
|
||||
|
||||
// User metadata
|
||||
metadata: parsedEdge.metadata
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
// User metadata retrieved separately via getVerbMetadata()
|
||||
})
|
||||
}
|
||||
} catch (error: any) {
|
||||
|
|
@ -610,7 +621,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
try {
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
if (metadata) {
|
||||
const verbType = metadata.verb || metadata.type || 'default'
|
||||
const verbType = (metadata.verb || metadata.type || 'default') as string
|
||||
this.decrementVerbCount(verbType)
|
||||
await this.deleteVerbMetadata(id)
|
||||
}
|
||||
|
|
@ -763,7 +774,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
cursor?: string
|
||||
filter?: any
|
||||
} = {}): Promise<{
|
||||
items: HNSWNoun[]
|
||||
items: HNSWNounWithMetadata[]
|
||||
totalCount: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -795,8 +806,8 @@ export class FileSystemStorage extends BaseStorage {
|
|||
// Get page of files
|
||||
const pageFiles = nounFiles.slice(startIndex, startIndex + limit)
|
||||
|
||||
// Load nouns - count actual successfully loaded items
|
||||
const items: HNSWNoun[] = []
|
||||
// v4.0.0: Load nouns and combine with metadata
|
||||
const items: HNSWNounWithMetadata[] = []
|
||||
let successfullyLoaded = 0
|
||||
let totalValidFiles = 0
|
||||
|
||||
|
|
@ -806,7 +817,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
// No need to count files anymore - we maintain accurate counts
|
||||
// This eliminates the O(n) operation completely
|
||||
|
||||
// Second pass: load the current page
|
||||
// Second pass: load the current page with metadata
|
||||
for (const file of pageFiles) {
|
||||
try {
|
||||
const id = file.replace('.json', '')
|
||||
|
|
@ -814,14 +825,17 @@ export class FileSystemStorage extends BaseStorage {
|
|||
this.getNodePath(id),
|
||||
'utf-8'
|
||||
)
|
||||
const noun = JSON.parse(data)
|
||||
const parsedNoun = JSON.parse(data)
|
||||
|
||||
// v4.0.0: Load metadata from separate storage
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
if (!metadata) continue
|
||||
|
||||
// Apply filter if provided
|
||||
if (options.filter) {
|
||||
// Simple filter implementation
|
||||
let matches = true
|
||||
for (const [key, value] of Object.entries(options.filter)) {
|
||||
if (noun.metadata && noun.metadata[key] !== value) {
|
||||
if (metadata[key] !== value) {
|
||||
matches = false
|
||||
break
|
||||
}
|
||||
|
|
@ -829,7 +843,26 @@ export class FileSystemStorage extends BaseStorage {
|
|||
if (!matches) continue
|
||||
}
|
||||
|
||||
items.push(noun)
|
||||
// Convert connections if needed
|
||||
let connections = parsedNoun.connections
|
||||
if (connections && typeof connections === 'object' && !(connections instanceof Map)) {
|
||||
const connectionsMap = new Map<number, Set<string>>()
|
||||
for (const [level, nodeIds] of Object.entries(connections)) {
|
||||
connectionsMap.set(Number(level), new Set(nodeIds as string[]))
|
||||
}
|
||||
connections = connectionsMap
|
||||
}
|
||||
|
||||
// v4.0.0: Create HNSWNounWithMetadata by combining noun with metadata
|
||||
const nounWithMetadata: HNSWNounWithMetadata = {
|
||||
id: parsedNoun.id,
|
||||
vector: parsedNoun.vector,
|
||||
connections: connections,
|
||||
level: parsedNoun.level || 0,
|
||||
metadata: metadata
|
||||
}
|
||||
|
||||
items.push(nounWithMetadata)
|
||||
successfullyLoaded++
|
||||
} catch (error) {
|
||||
console.warn(`Failed to read noun file ${file}:`, error)
|
||||
|
|
@ -1085,23 +1118,18 @@ export class FileSystemStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get a noun from storage (internal implementation)
|
||||
* Combines vector data from getNode() with metadata from getNounMetadata()
|
||||
* v4.0.0: Returns ONLY vector data (no metadata field)
|
||||
* Base class combines with metadata via getNoun() -> HNSWNounWithMetadata
|
||||
*/
|
||||
protected async getNoun_internal(id: string): Promise<HNSWNoun | null> {
|
||||
// Get vector data (lightweight)
|
||||
// v4.0.0: Return ONLY vector data (no metadata field)
|
||||
const node = await this.getNode(id)
|
||||
if (!node) {
|
||||
return null
|
||||
}
|
||||
|
||||
// Get metadata (entity data in 2-file system)
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
|
||||
// Combine into complete noun object
|
||||
return {
|
||||
...node,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
// Return pure vector structure
|
||||
return node
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -1130,23 +1158,18 @@ export class FileSystemStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get a verb from storage (internal implementation)
|
||||
* Combines vector data from getEdge() with metadata from getVerbMetadata()
|
||||
* v4.0.0: Returns ONLY vector + core relational fields (no metadata field)
|
||||
* Base class combines with metadata via getVerb() -> HNSWVerbWithMetadata
|
||||
*/
|
||||
protected async getVerb_internal(id: string): Promise<HNSWVerb | null> {
|
||||
// Get vector data (lightweight)
|
||||
// v4.0.0: Return ONLY vector + core relational data (no metadata field)
|
||||
const edge = await this.getEdge(id)
|
||||
if (!edge) {
|
||||
return null
|
||||
}
|
||||
|
||||
// Get metadata (relationship data in 2-file system)
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
|
||||
// Combine into complete verb object
|
||||
return {
|
||||
...edge,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
// Return pure vector + core fields structure
|
||||
return edge
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -1155,15 +1178,15 @@ export class FileSystemStorage extends BaseStorage {
|
|||
*/
|
||||
protected async getVerbsBySource_internal(
|
||||
sourceId: string
|
||||
): Promise<GraphVerb[]> {
|
||||
): Promise<HNSWVerbWithMetadata[]> {
|
||||
console.log(`[DEBUG] getVerbsBySource_internal called for sourceId: ${sourceId}`)
|
||||
|
||||
|
||||
// Use the working pagination method with source filter
|
||||
const result = await this.getVerbsWithPagination({
|
||||
limit: 10000,
|
||||
filter: { sourceId: [sourceId] }
|
||||
})
|
||||
|
||||
|
||||
console.log(`[DEBUG] Found ${result.items.length} verbs for source ${sourceId}`)
|
||||
return result.items
|
||||
}
|
||||
|
|
@ -1173,7 +1196,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
*/
|
||||
protected async getVerbsByTarget_internal(
|
||||
targetId: string
|
||||
): Promise<GraphVerb[]> {
|
||||
): Promise<HNSWVerbWithMetadata[]> {
|
||||
console.log(`[DEBUG] getVerbsByTarget_internal called for targetId: ${targetId}`)
|
||||
|
||||
// Use the working pagination method with target filter
|
||||
|
|
@ -1189,15 +1212,15 @@ export class FileSystemStorage extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by type
|
||||
*/
|
||||
protected async getVerbsByType_internal(type: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByType_internal(type: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
console.log(`[DEBUG] getVerbsByType_internal called for type: ${type}`)
|
||||
|
||||
|
||||
// Use the working pagination method with type filter
|
||||
const result = await this.getVerbsWithPagination({
|
||||
limit: 10000,
|
||||
filter: { verbType: [type] }
|
||||
})
|
||||
|
||||
|
||||
console.log(`[DEBUG] Found ${result.items.length} verbs for type ${type}`)
|
||||
return result.items
|
||||
}
|
||||
|
|
@ -1217,7 +1240,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<{
|
||||
items: GraphVerb[]
|
||||
items: HNSWVerbWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -1250,7 +1273,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
const endIndex = Math.min(startIndex + limit, actualFileCount)
|
||||
|
||||
// Load the requested page of verbs
|
||||
const verbs: GraphVerb[] = []
|
||||
const verbs: HNSWVerbWithMetadata[] = []
|
||||
let successfullyLoaded = 0
|
||||
|
||||
for (let i = startIndex; i < endIndex; i++) {
|
||||
|
|
@ -1272,28 +1295,13 @@ export class FileSystemStorage extends BaseStorage {
|
|||
|
||||
// Get metadata which contains the actual verb information
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
|
||||
// If no metadata exists, try to reconstruct basic metadata from filename
|
||||
|
||||
// v4.0.0: No fallbacks - skip verbs without metadata
|
||||
if (!metadata) {
|
||||
console.warn(`Verb ${id} has no metadata, trying to create minimal verb`)
|
||||
|
||||
// Create minimal GraphVerb without full metadata
|
||||
const minimalVerb: GraphVerb = {
|
||||
id: edge.id,
|
||||
vector: edge.vector,
|
||||
connections: edge.connections || new Map(),
|
||||
sourceId: 'unknown',
|
||||
targetId: 'unknown',
|
||||
source: 'unknown',
|
||||
target: 'unknown',
|
||||
type: 'relationship',
|
||||
verb: 'relatedTo'
|
||||
}
|
||||
|
||||
verbs.push(minimalVerb)
|
||||
console.warn(`Verb ${id} has no metadata, skipping`)
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
// Convert connections Map to proper format if needed
|
||||
let connections = edge.connections
|
||||
if (connections && typeof connections === 'object' && !(connections instanceof Map)) {
|
||||
|
|
@ -1303,25 +1311,16 @@ export class FileSystemStorage extends BaseStorage {
|
|||
}
|
||||
connections = connectionsMap
|
||||
}
|
||||
|
||||
// Properly reconstruct GraphVerb from HNSWVerb + metadata
|
||||
const verb: GraphVerb = {
|
||||
|
||||
// v4.0.0: Clean HNSWVerbWithMetadata construction
|
||||
const verbWithMetadata: HNSWVerbWithMetadata = {
|
||||
id: edge.id,
|
||||
vector: edge.vector, // Include the vector field!
|
||||
vector: edge.vector,
|
||||
connections: connections,
|
||||
sourceId: metadata.sourceId || metadata.source,
|
||||
targetId: metadata.targetId || metadata.target,
|
||||
source: metadata.source || metadata.sourceId,
|
||||
target: metadata.target || metadata.targetId,
|
||||
verb: metadata.verb || metadata.type,
|
||||
type: metadata.type || metadata.verb,
|
||||
weight: metadata.weight,
|
||||
metadata: metadata.metadata || metadata,
|
||||
data: metadata.data,
|
||||
createdAt: metadata.createdAt,
|
||||
updatedAt: metadata.updatedAt,
|
||||
createdBy: metadata.createdBy,
|
||||
embedding: metadata.embedding || edge.vector
|
||||
verb: edge.verb,
|
||||
sourceId: edge.sourceId,
|
||||
targetId: edge.targetId,
|
||||
metadata: metadata
|
||||
}
|
||||
|
||||
// Apply filters if provided
|
||||
|
|
@ -1331,22 +1330,19 @@ export class FileSystemStorage extends BaseStorage {
|
|||
// Check verbType filter
|
||||
if (filter.verbType) {
|
||||
const types = Array.isArray(filter.verbType) ? filter.verbType : [filter.verbType]
|
||||
const verbType = verb.type || verb.verb
|
||||
if (verbType && !types.includes(verbType)) continue
|
||||
if (!types.includes(verbWithMetadata.verb)) continue
|
||||
}
|
||||
|
||||
// Check sourceId filter
|
||||
if (filter.sourceId) {
|
||||
const sources = Array.isArray(filter.sourceId) ? filter.sourceId : [filter.sourceId]
|
||||
const sourceId = verb.sourceId || verb.source
|
||||
if (!sourceId || !sources.includes(sourceId)) continue
|
||||
if (!sources.includes(verbWithMetadata.sourceId)) continue
|
||||
}
|
||||
|
||||
// Check targetId filter
|
||||
if (filter.targetId) {
|
||||
const targets = Array.isArray(filter.targetId) ? filter.targetId : [filter.targetId]
|
||||
const targetId = verb.targetId || verb.target
|
||||
if (!targetId || !targets.includes(targetId)) continue
|
||||
if (!targets.includes(verbWithMetadata.targetId)) continue
|
||||
}
|
||||
|
||||
// Check service filter
|
||||
|
|
@ -1356,7 +1352,7 @@ export class FileSystemStorage extends BaseStorage {
|
|||
}
|
||||
}
|
||||
|
||||
verbs.push(verb)
|
||||
verbs.push(verbWithMetadata)
|
||||
successfullyLoaded++
|
||||
} catch (error) {
|
||||
console.warn(`Failed to read verb ${id}:`, error)
|
||||
|
|
@ -1798,34 +1794,21 @@ export class FileSystemStorage extends BaseStorage {
|
|||
this.totalVerbCount = validVerbFiles.length
|
||||
|
||||
// Sample some files to get type distribution (don't read all)
|
||||
// v4.0.0: Load metadata separately for type information
|
||||
const sampleSize = Math.min(100, validNounFiles.length)
|
||||
for (let i = 0; i < sampleSize; i++) {
|
||||
try {
|
||||
const file = validNounFiles[i]
|
||||
const id = file.replace('.json', '')
|
||||
|
||||
// Construct path using detected depth (not cached depth which may be wrong)
|
||||
let filePath: string
|
||||
switch (depthToUse) {
|
||||
case 0:
|
||||
filePath = path.join(this.nounsDir, `${id}.json`)
|
||||
break
|
||||
case 1:
|
||||
filePath = path.join(this.nounsDir, id.substring(0, 2), `${id}.json`)
|
||||
break
|
||||
case 2:
|
||||
filePath = path.join(this.nounsDir, id.substring(0, 2), id.substring(2, 4), `${id}.json`)
|
||||
break
|
||||
default:
|
||||
throw new Error(`Unsupported depth: ${depthToUse}`)
|
||||
// v4.0.0: Load metadata from separate storage for type info
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
if (metadata) {
|
||||
const type = metadata.noun || 'default'
|
||||
this.entityCounts.set(type, (this.entityCounts.get(type) || 0) + 1)
|
||||
}
|
||||
|
||||
const data = await fs.promises.readFile(filePath, 'utf-8')
|
||||
const noun = JSON.parse(data)
|
||||
const type = noun.metadata?.type || noun.metadata?.nounType || 'default'
|
||||
this.entityCounts.set(type, (this.entityCounts.get(type) || 0) + 1)
|
||||
} catch {
|
||||
// Skip invalid files
|
||||
// Skip invalid files or missing metadata
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -2418,12 +2401,12 @@ export class FileSystemStorage extends BaseStorage {
|
|||
startIndex: number,
|
||||
limit: number
|
||||
): Promise<{
|
||||
items: GraphVerb[]
|
||||
items: HNSWVerbWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
}> {
|
||||
const verbs: GraphVerb[] = []
|
||||
const verbs: HNSWVerbWithMetadata[] = []
|
||||
let processedCount = 0
|
||||
let skippedCount = 0
|
||||
let resultCount = 0
|
||||
|
|
@ -2456,29 +2439,31 @@ export class FileSystemStorage extends BaseStorage {
|
|||
const edge = JSON.parse(data)
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
|
||||
// v4.0.0: No fallbacks - skip verbs without metadata
|
||||
if (!metadata) {
|
||||
processedCount++
|
||||
return true // continue, skip this verb
|
||||
}
|
||||
|
||||
// Reconstruct GraphVerb
|
||||
const verb: GraphVerb = {
|
||||
// Convert connections if needed
|
||||
let connections = edge.connections
|
||||
if (connections && typeof connections === 'object' && !(connections instanceof Map)) {
|
||||
const connectionsMap = new Map<number, Set<string>>()
|
||||
for (const [level, nodeIds] of Object.entries(connections)) {
|
||||
connectionsMap.set(Number(level), new Set(nodeIds as string[]))
|
||||
}
|
||||
connections = connectionsMap
|
||||
}
|
||||
|
||||
// v4.0.0: Clean HNSWVerbWithMetadata construction
|
||||
const verbWithMetadata: HNSWVerbWithMetadata = {
|
||||
id: edge.id,
|
||||
vector: edge.vector,
|
||||
connections: edge.connections || new Map(),
|
||||
sourceId: metadata.sourceId || metadata.source,
|
||||
targetId: metadata.targetId || metadata.target,
|
||||
source: metadata.source || metadata.sourceId,
|
||||
target: metadata.target || metadata.targetId,
|
||||
verb: metadata.verb || metadata.type,
|
||||
type: metadata.type || metadata.verb,
|
||||
weight: metadata.weight,
|
||||
metadata: metadata.metadata || metadata,
|
||||
data: metadata.data,
|
||||
createdAt: metadata.createdAt,
|
||||
updatedAt: metadata.updatedAt,
|
||||
createdBy: metadata.createdBy,
|
||||
embedding: metadata.embedding || edge.vector
|
||||
connections: connections || new Map(),
|
||||
verb: edge.verb,
|
||||
sourceId: edge.sourceId,
|
||||
targetId: edge.targetId,
|
||||
metadata: metadata
|
||||
}
|
||||
|
||||
// Apply filters
|
||||
|
|
@ -2487,24 +2472,21 @@ export class FileSystemStorage extends BaseStorage {
|
|||
|
||||
if (filter.verbType) {
|
||||
const types = Array.isArray(filter.verbType) ? filter.verbType : [filter.verbType]
|
||||
const verbType = verb.type || verb.verb
|
||||
if (verbType && !types.includes(verbType)) return true // continue
|
||||
if (!types.includes(verbWithMetadata.verb)) return true // continue
|
||||
}
|
||||
|
||||
if (filter.sourceId) {
|
||||
const sources = Array.isArray(filter.sourceId) ? filter.sourceId : [filter.sourceId]
|
||||
const sourceId = verb.sourceId || verb.source
|
||||
if (!sourceId || !sources.includes(sourceId)) return true // continue
|
||||
if (!sources.includes(verbWithMetadata.sourceId)) return true // continue
|
||||
}
|
||||
|
||||
if (filter.targetId) {
|
||||
const targets = Array.isArray(filter.targetId) ? filter.targetId : [filter.targetId]
|
||||
const targetId = verb.targetId || verb.target
|
||||
if (!targetId || !targets.includes(targetId)) return true // continue
|
||||
if (!targets.includes(verbWithMetadata.targetId)) return true // continue
|
||||
}
|
||||
}
|
||||
|
||||
verbs.push(verb)
|
||||
verbs.push(verbWithMetadata)
|
||||
resultCount++
|
||||
processedCount++
|
||||
return true // continue
|
||||
|
|
|
|||
|
|
@ -9,7 +9,16 @@
|
|||
* 4. HMAC Keys (fallback for backward compatibility)
|
||||
*/
|
||||
|
||||
import { GraphVerb, HNSWNoun, HNSWVerb, StatisticsData } from '../../coreTypes.js'
|
||||
import {
|
||||
GraphVerb,
|
||||
HNSWNoun,
|
||||
HNSWVerb,
|
||||
NounMetadata,
|
||||
VerbMetadata,
|
||||
HNSWNounWithMetadata,
|
||||
HNSWVerbWithMetadata,
|
||||
StatisticsData
|
||||
} from '../../coreTypes.js'
|
||||
import {
|
||||
BaseStorage,
|
||||
NOUNS_DIR,
|
||||
|
|
@ -472,7 +481,7 @@ export class GcsStorage extends BaseStorage {
|
|||
// Increment noun count
|
||||
const metadata = await this.getNounMetadata(node.id)
|
||||
if (metadata && metadata.type) {
|
||||
await this.incrementEntityCountSafe(metadata.type)
|
||||
await this.incrementEntityCountSafe(metadata.type as string)
|
||||
}
|
||||
|
||||
this.logger.trace(`Node ${node.id} saved successfully`)
|
||||
|
|
@ -493,23 +502,18 @@ export class GcsStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get a noun from storage (internal implementation)
|
||||
* Combines vector data from getNode() with metadata from getNounMetadata()
|
||||
* v4.0.0: Returns ONLY vector data (no metadata field)
|
||||
* Base class combines with metadata via getNoun() -> HNSWNounWithMetadata
|
||||
*/
|
||||
protected async getNoun_internal(id: string): Promise<HNSWNoun | null> {
|
||||
// Get vector data (lightweight)
|
||||
// v4.0.0: Return ONLY vector data (no metadata field)
|
||||
const node = await this.getNode(id)
|
||||
if (!node) {
|
||||
return null
|
||||
}
|
||||
|
||||
// Get metadata (entity data in 2-file system)
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
|
||||
// Combine into complete noun object
|
||||
return {
|
||||
...node,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
// Return pure vector structure
|
||||
return node
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -644,7 +648,7 @@ export class GcsStorage extends BaseStorage {
|
|||
// Decrement noun count
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
if (metadata && metadata.type) {
|
||||
await this.decrementEntityCountSafe(metadata.type)
|
||||
await this.decrementEntityCountSafe(metadata.type as string)
|
||||
}
|
||||
|
||||
this.logger.trace(`Noun ${id} deleted successfully`)
|
||||
|
|
@ -847,7 +851,7 @@ export class GcsStorage extends BaseStorage {
|
|||
// Increment verb count
|
||||
const metadata = await this.getVerbMetadata(edge.id)
|
||||
if (metadata && metadata.type) {
|
||||
await this.incrementVerbCount(metadata.type)
|
||||
await this.incrementVerbCount(metadata.type as string)
|
||||
}
|
||||
|
||||
this.logger.trace(`Edge ${edge.id} saved successfully`)
|
||||
|
|
@ -867,23 +871,18 @@ export class GcsStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get a verb from storage (internal implementation)
|
||||
* Combines vector data from getEdge() with metadata from getVerbMetadata()
|
||||
* v4.0.0: Returns ONLY vector + core relational fields (no metadata field)
|
||||
* Base class combines with metadata via getVerb() -> HNSWVerbWithMetadata
|
||||
*/
|
||||
protected async getVerb_internal(id: string): Promise<HNSWVerb | null> {
|
||||
// Get vector data (lightweight)
|
||||
// v4.0.0: Return ONLY vector + core relational data (no metadata field)
|
||||
const edge = await this.getEdge(id)
|
||||
if (!edge) {
|
||||
return null
|
||||
}
|
||||
|
||||
// Get metadata (relationship data in 2-file system)
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
|
||||
// Combine into complete verb object
|
||||
return {
|
||||
...edge,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
// Return pure vector + core fields structure
|
||||
return edge
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -920,7 +919,7 @@ export class GcsStorage extends BaseStorage {
|
|||
connections.set(Number(level), new Set(verbIds as string[]))
|
||||
}
|
||||
|
||||
// ARCHITECTURAL FIX (v3.50.1): Return HNSWVerb with core relational fields
|
||||
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
|
||||
const edge: Edge = {
|
||||
id: data.id,
|
||||
vector: data.vector,
|
||||
|
|
@ -929,10 +928,10 @@ export class GcsStorage extends BaseStorage {
|
|||
// CORE RELATIONAL DATA (read from vector file)
|
||||
verb: data.verb,
|
||||
sourceId: data.sourceId,
|
||||
targetId: data.targetId,
|
||||
targetId: data.targetId
|
||||
|
||||
// User metadata (retrieved separately via getVerbMetadata())
|
||||
metadata: data.metadata
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
// User metadata retrieved separately via getVerbMetadata()
|
||||
}
|
||||
|
||||
// Update cache
|
||||
|
|
@ -984,7 +983,7 @@ export class GcsStorage extends BaseStorage {
|
|||
// Decrement verb count
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
if (metadata && metadata.type) {
|
||||
await this.decrementVerbCount(metadata.type)
|
||||
await this.decrementVerbCount(metadata.type as string)
|
||||
}
|
||||
|
||||
this.logger.trace(`Verb ${id} deleted successfully`)
|
||||
|
|
@ -1010,6 +1009,7 @@ export class GcsStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get nouns with pagination
|
||||
* v4.0.0: Returns HNSWNounWithMetadata[] (includes metadata field)
|
||||
* Iterates through all UUID-based shards (00-ff) for consistent pagination
|
||||
*/
|
||||
public async getNounsWithPagination(options: {
|
||||
|
|
@ -1021,7 +1021,7 @@ export class GcsStorage extends BaseStorage {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<{
|
||||
items: HNSWNoun[]
|
||||
items: HNSWNounWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -1038,31 +1038,54 @@ export class GcsStorage extends BaseStorage {
|
|||
useCache: true
|
||||
})
|
||||
|
||||
// Apply filters if provided
|
||||
let filteredNodes = result.nodes
|
||||
// v4.0.0: Combine nodes with metadata to create HNSWNounWithMetadata[]
|
||||
const items: HNSWNounWithMetadata[] = []
|
||||
|
||||
if (options.filter) {
|
||||
// Filter by noun type
|
||||
if (options.filter.nounType) {
|
||||
const nounTypes = Array.isArray(options.filter.nounType)
|
||||
? options.filter.nounType
|
||||
: [options.filter.nounType]
|
||||
for (const node of result.nodes) {
|
||||
const metadata = await this.getNounMetadata(node.id)
|
||||
if (!metadata) continue
|
||||
|
||||
const filteredByType: HNSWNoun[] = []
|
||||
for (const node of filteredNodes) {
|
||||
const metadata = await this.getNounMetadata(node.id)
|
||||
if (metadata && nounTypes.includes(metadata.type || metadata.noun)) {
|
||||
filteredByType.push(node)
|
||||
// Apply filters if provided
|
||||
if (options.filter) {
|
||||
// Filter by noun type
|
||||
if (options.filter.nounType) {
|
||||
const nounTypes = Array.isArray(options.filter.nounType)
|
||||
? options.filter.nounType
|
||||
: [options.filter.nounType]
|
||||
|
||||
const nounType = (metadata as any).type || (metadata as any).noun
|
||||
if (!nounType || !nounTypes.includes(nounType)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
filteredNodes = filteredByType
|
||||
|
||||
// Filter by metadata fields if specified
|
||||
if (options.filter.metadata) {
|
||||
let metadataMatch = true
|
||||
for (const [key, value] of Object.entries(options.filter.metadata)) {
|
||||
const metadataValue = (metadata as any)[key]
|
||||
if (metadataValue !== value) {
|
||||
metadataMatch = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if (!metadataMatch) continue
|
||||
}
|
||||
}
|
||||
|
||||
// Additional filter logic can be added here
|
||||
// Combine node with metadata
|
||||
const nounWithMetadata: HNSWNounWithMetadata = {
|
||||
id: node.id,
|
||||
vector: [...node.vector],
|
||||
connections: new Map(node.connections),
|
||||
level: node.level || 0,
|
||||
metadata: metadata
|
||||
}
|
||||
items.push(nounWithMetadata)
|
||||
}
|
||||
|
||||
return {
|
||||
items: filteredNodes,
|
||||
items,
|
||||
totalCount: result.totalCount,
|
||||
hasMore: result.hasMore,
|
||||
nextCursor: result.nextCursor
|
||||
|
|
@ -1203,7 +1226,7 @@ export class GcsStorage extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by source ID (internal implementation)
|
||||
*/
|
||||
protected async getVerbsBySource_internal(sourceId: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsBySource_internal(sourceId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Use the paginated approach to properly handle HNSWVerb to GraphVerb conversion
|
||||
const result = await this.getVerbsWithPagination({
|
||||
limit: Number.MAX_SAFE_INTEGER,
|
||||
|
|
@ -1216,7 +1239,7 @@ export class GcsStorage extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by target ID (internal implementation)
|
||||
*/
|
||||
protected async getVerbsByTarget_internal(targetId: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByTarget_internal(targetId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Use the paginated approach to properly handle HNSWVerb to GraphVerb conversion
|
||||
const result = await this.getVerbsWithPagination({
|
||||
limit: Number.MAX_SAFE_INTEGER,
|
||||
|
|
@ -1229,7 +1252,7 @@ export class GcsStorage extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by type (internal implementation)
|
||||
*/
|
||||
protected async getVerbsByType_internal(type: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByType_internal(type: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Use the paginated approach to properly handle HNSWVerb to GraphVerb conversion
|
||||
const result = await this.getVerbsWithPagination({
|
||||
limit: Number.MAX_SAFE_INTEGER,
|
||||
|
|
@ -1241,6 +1264,7 @@ export class GcsStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get verbs with pagination
|
||||
* v4.0.0: Returns HNSWVerbWithMetadata[] (includes metadata field)
|
||||
*/
|
||||
public async getVerbsWithPagination(options: {
|
||||
limit?: number
|
||||
|
|
@ -1253,7 +1277,7 @@ export class GcsStorage extends BaseStorage {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<{
|
||||
items: GraphVerb[]
|
||||
items: HNSWVerbWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -1303,56 +1327,70 @@ export class GcsStorage extends BaseStorage {
|
|||
}
|
||||
}
|
||||
|
||||
// Convert HNSWVerbs to GraphVerbs by combining with metadata
|
||||
const graphVerbs: GraphVerb[] = []
|
||||
// v4.0.0: Combine HNSWVerbs with metadata to create HNSWVerbWithMetadata[]
|
||||
const items: HNSWVerbWithMetadata[] = []
|
||||
for (const hnswVerb of hnswVerbs) {
|
||||
const graphVerb = await this.convertHNSWVerbToGraphVerb(hnswVerb)
|
||||
if (graphVerb) {
|
||||
graphVerbs.push(graphVerb)
|
||||
const metadata = await this.getVerbMetadata(hnswVerb.id)
|
||||
|
||||
// Apply filters
|
||||
if (options.filter) {
|
||||
// v4.0.0: Core fields (verb, sourceId, targetId) are in HNSWVerb structure
|
||||
if (options.filter.sourceId) {
|
||||
const sourceIds = Array.isArray(options.filter.sourceId)
|
||||
? options.filter.sourceId
|
||||
: [options.filter.sourceId]
|
||||
if (!hnswVerb.sourceId || !sourceIds.includes(hnswVerb.sourceId)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
if (options.filter.targetId) {
|
||||
const targetIds = Array.isArray(options.filter.targetId)
|
||||
? options.filter.targetId
|
||||
: [options.filter.targetId]
|
||||
if (!hnswVerb.targetId || !targetIds.includes(hnswVerb.targetId)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
if (options.filter.verbType) {
|
||||
const verbTypes = Array.isArray(options.filter.verbType)
|
||||
? options.filter.verbType
|
||||
: [options.filter.verbType]
|
||||
if (!hnswVerb.verb || !verbTypes.includes(hnswVerb.verb)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by metadata fields if specified
|
||||
if (options.filter.metadata && metadata) {
|
||||
let metadataMatch = true
|
||||
for (const [key, value] of Object.entries(options.filter.metadata)) {
|
||||
const metadataValue = (metadata as any)[key]
|
||||
if (metadataValue !== value) {
|
||||
metadataMatch = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if (!metadataMatch) continue
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Apply filters
|
||||
let filteredVerbs = graphVerbs
|
||||
if (options.filter) {
|
||||
filteredVerbs = graphVerbs.filter((graphVerb) => {
|
||||
// Filter by sourceId
|
||||
if (options.filter!.sourceId) {
|
||||
const sourceIds = Array.isArray(options.filter!.sourceId)
|
||||
? options.filter!.sourceId
|
||||
: [options.filter!.sourceId]
|
||||
if (!sourceIds.includes(graphVerb.sourceId)) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by targetId
|
||||
if (options.filter!.targetId) {
|
||||
const targetIds = Array.isArray(options.filter!.targetId)
|
||||
? options.filter!.targetId
|
||||
: [options.filter!.targetId]
|
||||
if (!targetIds.includes(graphVerb.targetId)) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by verbType
|
||||
if (options.filter!.verbType) {
|
||||
const verbTypes = Array.isArray(options.filter!.verbType)
|
||||
? options.filter!.verbType
|
||||
: [options.filter!.verbType]
|
||||
const verbType = graphVerb.verb || graphVerb.type || ''
|
||||
if (!verbTypes.includes(verbType)) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
return true
|
||||
})
|
||||
// Combine verb with metadata
|
||||
const verbWithMetadata: HNSWVerbWithMetadata = {
|
||||
id: hnswVerb.id,
|
||||
vector: [...hnswVerb.vector],
|
||||
connections: new Map(hnswVerb.connections),
|
||||
verb: hnswVerb.verb,
|
||||
sourceId: hnswVerb.sourceId,
|
||||
targetId: hnswVerb.targetId,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
items.push(verbWithMetadata)
|
||||
}
|
||||
|
||||
return {
|
||||
items: filteredVerbs,
|
||||
items,
|
||||
totalCount: this.totalVerbCount,
|
||||
hasMore: !!response?.nextPageToken,
|
||||
nextCursor: response?.nextPageToken
|
||||
|
|
@ -1395,6 +1433,7 @@ export class GcsStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get verbs with filtering and pagination (public API)
|
||||
* v4.0.0: Returns HNSWVerbWithMetadata[] (includes metadata field)
|
||||
*/
|
||||
public async getVerbs(options?: {
|
||||
pagination?: {
|
||||
|
|
@ -1410,7 +1449,7 @@ export class GcsStorage extends BaseStorage {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
}): Promise<{
|
||||
items: GraphVerb[]
|
||||
items: HNSWVerbWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
|
|||
|
|
@ -3,7 +3,16 @@
|
|||
* In-memory storage adapter for environments where persistent storage is not available or needed
|
||||
*/
|
||||
|
||||
import { GraphVerb, HNSWNoun, HNSWVerb, StatisticsData } from '../../coreTypes.js'
|
||||
import {
|
||||
GraphVerb,
|
||||
HNSWNoun,
|
||||
HNSWVerb,
|
||||
NounMetadata,
|
||||
VerbMetadata,
|
||||
HNSWNounWithMetadata,
|
||||
HNSWVerbWithMetadata,
|
||||
StatisticsData
|
||||
} from '../../coreTypes.js'
|
||||
import { BaseStorage, STATISTICS_KEY } from '../baseStorage.js'
|
||||
import { PaginatedResult } from '../../types/paginationTypes.js'
|
||||
|
||||
|
|
@ -46,20 +55,20 @@ export class MemoryStorage extends BaseStorage {
|
|||
}
|
||||
|
||||
/**
|
||||
* Save a noun to storage
|
||||
* Save a noun to storage (v4.0.0: pure vector only, no metadata)
|
||||
*/
|
||||
protected async saveNoun_internal(noun: HNSWNoun): Promise<void> {
|
||||
const isNew = !this.nouns.has(noun.id)
|
||||
|
||||
// Create a deep copy to avoid reference issues
|
||||
// CRITICAL: Only save lightweight vector data (no metadata)
|
||||
// Metadata is saved separately via saveNounMetadata() (2-file system)
|
||||
// v4.0.0: Store ONLY vector data (no metadata field)
|
||||
// Metadata is saved separately via saveNounMetadata() by base class
|
||||
const nounCopy: HNSWNoun = {
|
||||
id: noun.id,
|
||||
vector: [...noun.vector],
|
||||
connections: new Map(),
|
||||
level: noun.level || 0
|
||||
// NO metadata field - saved separately for scalability
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
}
|
||||
|
||||
// Copy connections
|
||||
|
|
@ -70,16 +79,12 @@ export class MemoryStorage extends BaseStorage {
|
|||
// Save the noun directly in the nouns map
|
||||
this.nouns.set(noun.id, nounCopy)
|
||||
|
||||
// Update counts for new entities
|
||||
if (isNew) {
|
||||
const type = noun.metadata?.type || noun.metadata?.nounType || 'default'
|
||||
this.incrementEntityCount(type)
|
||||
}
|
||||
// Note: Count tracking happens in saveNounMetadata since type info is in metadata now
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a noun from storage (internal implementation)
|
||||
* Combines vector data from nouns map with metadata from getNounMetadata()
|
||||
* Get a noun from storage (v4.0.0: returns pure vector only)
|
||||
* Base class handles combining with metadata
|
||||
*/
|
||||
protected async getNoun_internal(id: string): Promise<HNSWNoun | null> {
|
||||
// Get the noun directly from the nouns map
|
||||
|
|
@ -91,11 +96,13 @@ export class MemoryStorage extends BaseStorage {
|
|||
}
|
||||
|
||||
// Return a deep copy to avoid reference issues
|
||||
// v4.0.0: Return ONLY vector data (no metadata field)
|
||||
const nounCopy: HNSWNoun = {
|
||||
id: noun.id,
|
||||
vector: [...noun.vector],
|
||||
connections: new Map(),
|
||||
level: noun.level || 0
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
}
|
||||
|
||||
// Copy connections
|
||||
|
|
@ -103,20 +110,14 @@ export class MemoryStorage extends BaseStorage {
|
|||
nounCopy.connections.set(level, new Set(connections))
|
||||
}
|
||||
|
||||
// Get metadata (entity data in 2-file system)
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
|
||||
// Combine into complete noun object
|
||||
return {
|
||||
...nounCopy,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
return nounCopy
|
||||
}
|
||||
|
||||
/**
|
||||
* Get nouns with pagination and filtering
|
||||
* v4.0.0: Returns HNSWNounWithMetadata[] (includes metadata field)
|
||||
* @param options Pagination and filtering options
|
||||
* @returns Promise that resolves to a paginated result of nouns
|
||||
* @returns Promise that resolves to a paginated result of nouns with metadata
|
||||
*/
|
||||
public async getNouns(options: {
|
||||
pagination?: {
|
||||
|
|
@ -129,7 +130,7 @@ export class MemoryStorage extends BaseStorage {
|
|||
service?: string | string[]
|
||||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<PaginatedResult<HNSWNoun>> {
|
||||
} = {}): Promise<{ items: HNSWNounWithMetadata[]; totalCount?: number; hasMore: boolean; nextCursor?: string }> {
|
||||
const pagination = options.pagination || {}
|
||||
const filter = options.filter || {}
|
||||
|
||||
|
|
@ -150,26 +151,26 @@ export class MemoryStorage extends BaseStorage {
|
|||
const matchingIds: string[] = []
|
||||
|
||||
// Iterate through all nouns to find matches
|
||||
// v4.0.0: Load metadata from separate storage (no embedded metadata field)
|
||||
for (const [nounId, noun] of this.nouns.entries()) {
|
||||
// Check the noun's embedded metadata field
|
||||
const nounMetadata = noun.metadata || {}
|
||||
|
||||
// Also check separate metadata store for backward compatibility
|
||||
const separateMetadata = await this.getMetadata(nounId)
|
||||
|
||||
// Merge both metadata sources (noun.metadata takes precedence)
|
||||
const metadata = { ...separateMetadata, ...nounMetadata }
|
||||
|
||||
// Get metadata from separate storage
|
||||
const metadata = await this.getNounMetadata(nounId)
|
||||
|
||||
// Skip if no metadata (shouldn't happen in v4.0.0 but be defensive)
|
||||
if (!metadata) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Filter by noun type if specified
|
||||
if (nounTypes && metadata.noun && !nounTypes.includes(metadata.noun)) {
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
// Filter by service if specified
|
||||
if (services && metadata.service && !services.includes(metadata.service)) {
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
// Filter by metadata fields if specified
|
||||
if (filter.metadata) {
|
||||
let metadataMatch = true
|
||||
|
|
@ -181,7 +182,7 @@ export class MemoryStorage extends BaseStorage {
|
|||
}
|
||||
if (!metadataMatch) continue
|
||||
}
|
||||
|
||||
|
||||
// If we got here, the noun matches all filters
|
||||
matchingIds.push(nounId)
|
||||
}
|
||||
|
|
@ -195,26 +196,31 @@ export class MemoryStorage extends BaseStorage {
|
|||
const nextCursor = hasMore ? `${offset + limit}` : undefined
|
||||
|
||||
// Fetch the actual nouns for the current page
|
||||
const items: HNSWNoun[] = []
|
||||
// v4.0.0: Return HNSWNounWithMetadata (includes metadata field)
|
||||
const items: HNSWNounWithMetadata[] = []
|
||||
for (const id of paginatedIds) {
|
||||
const noun = this.nouns.get(id)
|
||||
if (!noun) continue
|
||||
|
||||
// Create a deep copy to avoid reference issues
|
||||
const nounCopy: HNSWNoun = {
|
||||
id: noun.id,
|
||||
vector: [...noun.vector],
|
||||
connections: new Map(),
|
||||
level: noun.level || 0,
|
||||
metadata: noun.metadata
|
||||
}
|
||||
|
||||
|
||||
// Get metadata from separate storage
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
if (!metadata) continue // Skip if no metadata
|
||||
|
||||
// v4.0.0: Create HNSWNounWithMetadata with metadata field
|
||||
const nounWithMetadata: HNSWNounWithMetadata = {
|
||||
id: noun.id,
|
||||
vector: [...noun.vector],
|
||||
connections: new Map(),
|
||||
level: noun.level || 0,
|
||||
metadata: metadata // Include metadata field
|
||||
}
|
||||
|
||||
// Copy connections
|
||||
for (const [level, connections] of noun.connections.entries()) {
|
||||
nounCopy.connections.set(level, new Set(connections))
|
||||
nounWithMetadata.connections.set(level, new Set(connections))
|
||||
}
|
||||
|
||||
items.push(nounCopy)
|
||||
|
||||
items.push(nounWithMetadata)
|
||||
}
|
||||
|
||||
return {
|
||||
|
|
@ -227,13 +233,14 @@ export class MemoryStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get nouns with pagination - simplified interface for compatibility
|
||||
* v4.0.0: Returns HNSWNounWithMetadata[] (includes metadata field)
|
||||
*/
|
||||
public async getNounsWithPagination(options: {
|
||||
limit?: number
|
||||
cursor?: string
|
||||
filter?: any
|
||||
} = {}): Promise<{
|
||||
items: HNSWNoun[]
|
||||
items: HNSWNounWithMetadata[]
|
||||
totalCount: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -271,37 +278,36 @@ export class MemoryStorage extends BaseStorage {
|
|||
}
|
||||
|
||||
/**
|
||||
* Delete a noun from storage
|
||||
* Delete a noun from storage (v4.0.0)
|
||||
*/
|
||||
protected async deleteNoun_internal(id: string): Promise<void> {
|
||||
const noun = this.nouns.get(id)
|
||||
if (noun) {
|
||||
const type = noun.metadata?.type || noun.metadata?.nounType || 'default'
|
||||
// v4.0.0: Get type from separate metadata storage
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
if (metadata) {
|
||||
const type = metadata.noun || 'default'
|
||||
this.decrementEntityCount(type)
|
||||
}
|
||||
this.nouns.delete(id)
|
||||
}
|
||||
|
||||
/**
|
||||
* Save a verb to storage
|
||||
* Save a verb to storage (v4.0.0: pure vector + core fields, no metadata)
|
||||
*/
|
||||
protected async saveVerb_internal(verb: HNSWVerb): Promise<void> {
|
||||
const isNew = !this.verbs.has(verb.id)
|
||||
|
||||
// Create a deep copy to avoid reference issues
|
||||
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields
|
||||
// v4.0.0: Include core relational fields but NO metadata field
|
||||
const verbCopy: HNSWVerb = {
|
||||
id: verb.id,
|
||||
vector: [...verb.vector],
|
||||
connections: new Map(),
|
||||
|
||||
// CORE RELATIONAL DATA
|
||||
// CORE RELATIONAL DATA (part of HNSWVerb in v4.0.0)
|
||||
verb: verb.verb,
|
||||
sourceId: verb.sourceId,
|
||||
targetId: verb.targetId,
|
||||
|
||||
// User metadata (if any)
|
||||
metadata: verb.metadata
|
||||
targetId: verb.targetId
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
}
|
||||
|
||||
// Copy connections
|
||||
|
|
@ -312,13 +318,12 @@ export class MemoryStorage extends BaseStorage {
|
|||
// Save the verb directly in the verbs map
|
||||
this.verbs.set(verb.id, verbCopy)
|
||||
|
||||
// Count tracking will be handled in saveVerbMetadata_internal
|
||||
// since HNSWVerb doesn't contain type information
|
||||
// Note: Count tracking happens in saveVerbMetadata since metadata is separate
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a verb from storage (internal implementation)
|
||||
* Combines vector data from verbs map with metadata from getVerbMetadata()
|
||||
* Get a verb from storage (v4.0.0: returns pure vector + core fields)
|
||||
* Base class handles combining with metadata
|
||||
*/
|
||||
protected async getVerb_internal(id: string): Promise<HNSWVerb | null> {
|
||||
// Get the verb directly from the verbs map
|
||||
|
|
@ -330,19 +335,17 @@ export class MemoryStorage extends BaseStorage {
|
|||
}
|
||||
|
||||
// Return a deep copy of the HNSWVerb
|
||||
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields
|
||||
// v4.0.0: Include core relational fields but NO metadata field
|
||||
const verbCopy: HNSWVerb = {
|
||||
id: verb.id,
|
||||
vector: [...verb.vector],
|
||||
connections: new Map(),
|
||||
|
||||
// CORE RELATIONAL DATA
|
||||
// CORE RELATIONAL DATA (part of HNSWVerb in v4.0.0)
|
||||
verb: verb.verb,
|
||||
sourceId: verb.sourceId,
|
||||
targetId: verb.targetId,
|
||||
|
||||
// User metadata
|
||||
metadata: verb.metadata
|
||||
targetId: verb.targetId
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
}
|
||||
|
||||
// Copy connections
|
||||
|
|
@ -355,8 +358,9 @@ export class MemoryStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get verbs with pagination and filtering
|
||||
* v4.0.0: Returns HNSWVerbWithMetadata[] (includes metadata field)
|
||||
* @param options Pagination and filtering options
|
||||
* @returns Promise that resolves to a paginated result of verbs
|
||||
* @returns Promise that resolves to a paginated result of verbs with metadata
|
||||
*/
|
||||
public async getVerbs(options: {
|
||||
pagination?: {
|
||||
|
|
@ -371,7 +375,7 @@ export class MemoryStorage extends BaseStorage {
|
|||
service?: string | string[]
|
||||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<PaginatedResult<GraphVerb>> {
|
||||
} = {}): Promise<{ items: HNSWVerbWithMetadata[]; totalCount?: number; hasMore: boolean; nextCursor?: string }> {
|
||||
const pagination = options.pagination || {}
|
||||
const filter = options.filter || {}
|
||||
|
||||
|
|
@ -400,43 +404,47 @@ export class MemoryStorage extends BaseStorage {
|
|||
const matchingIds: string[] = []
|
||||
|
||||
// Iterate through all verbs to find matches
|
||||
// v4.0.0: Core fields (verb, sourceId, targetId) are in HNSWVerb, not metadata
|
||||
for (const [verbId, hnswVerb] of this.verbs.entries()) {
|
||||
// Get the metadata for this verb to do filtering
|
||||
// Get the metadata for service/data filtering
|
||||
const metadata = await this.getVerbMetadata(verbId)
|
||||
|
||||
|
||||
// Filter by verb type if specified
|
||||
if (verbTypes && metadata && !verbTypes.includes(metadata.type || metadata.verb || '')) {
|
||||
// v4.0.0: verb type is in HNSWVerb.verb
|
||||
if (verbTypes && !verbTypes.includes(hnswVerb.verb || '')) {
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
// Filter by source ID if specified
|
||||
if (sourceIds && metadata && !sourceIds.includes(metadata.sourceId || metadata.source || '')) {
|
||||
// v4.0.0: sourceId is in HNSWVerb.sourceId
|
||||
if (sourceIds && !sourceIds.includes(hnswVerb.sourceId || '')) {
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
// Filter by target ID if specified
|
||||
if (targetIds && metadata && !targetIds.includes(metadata.targetId || metadata.target || '')) {
|
||||
// v4.0.0: targetId is in HNSWVerb.targetId
|
||||
if (targetIds && !targetIds.includes(hnswVerb.targetId || '')) {
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
// Filter by metadata fields if specified
|
||||
if (filter.metadata && metadata && metadata.data) {
|
||||
if (filter.metadata && metadata) {
|
||||
let metadataMatch = true
|
||||
for (const [key, value] of Object.entries(filter.metadata)) {
|
||||
if (metadata.data[key] !== value) {
|
||||
const metadataValue = (metadata as any)[key]
|
||||
if (metadataValue !== value) {
|
||||
metadataMatch = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if (!metadataMatch) continue
|
||||
}
|
||||
|
||||
|
||||
// Filter by service if specified
|
||||
if (services && metadata && metadata.createdBy && metadata.createdBy.augmentation &&
|
||||
!services.includes(metadata.createdBy.augmentation)) {
|
||||
if (services && metadata && metadata.service && !services.includes(metadata.service)) {
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
// If we got here, the verb matches all filters
|
||||
matchingIds.push(verbId)
|
||||
}
|
||||
|
|
@ -450,44 +458,37 @@ export class MemoryStorage extends BaseStorage {
|
|||
const nextCursor = hasMore ? `${offset + limit}` : undefined
|
||||
|
||||
// Fetch the actual verbs for the current page
|
||||
const items: GraphVerb[] = []
|
||||
// v4.0.0: Return HNSWVerbWithMetadata (includes metadata field)
|
||||
const items: HNSWVerbWithMetadata[] = []
|
||||
for (const id of paginatedIds) {
|
||||
const hnswVerb = this.verbs.get(id)
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
|
||||
if (!hnswVerb) continue
|
||||
|
||||
if (!metadata) {
|
||||
console.warn(`Verb ${id} found but no metadata - creating minimal GraphVerb`)
|
||||
// Return minimal GraphVerb if metadata is missing
|
||||
items.push({
|
||||
id: hnswVerb.id,
|
||||
vector: hnswVerb.vector,
|
||||
sourceId: '',
|
||||
targetId: ''
|
||||
})
|
||||
continue
|
||||
}
|
||||
|
||||
// Create a complete GraphVerb by combining HNSWVerb with metadata
|
||||
const graphVerb: GraphVerb = {
|
||||
|
||||
// Get metadata from separate storage
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
if (!metadata) continue // Skip if no metadata
|
||||
|
||||
// v4.0.0: Create HNSWVerbWithMetadata with metadata field
|
||||
const verbWithMetadata: HNSWVerbWithMetadata = {
|
||||
id: hnswVerb.id,
|
||||
vector: [...hnswVerb.vector],
|
||||
sourceId: metadata.sourceId,
|
||||
targetId: metadata.targetId,
|
||||
source: metadata.source,
|
||||
target: metadata.target,
|
||||
verb: metadata.verb,
|
||||
type: metadata.type,
|
||||
weight: metadata.weight,
|
||||
createdAt: metadata.createdAt,
|
||||
updatedAt: metadata.updatedAt,
|
||||
createdBy: metadata.createdBy,
|
||||
data: metadata.data,
|
||||
metadata: metadata.metadata || metadata.data // Use metadata.metadata (user's custom metadata)
|
||||
connections: new Map(),
|
||||
|
||||
// Core relational fields (part of HNSWVerb)
|
||||
verb: hnswVerb.verb,
|
||||
sourceId: hnswVerb.sourceId,
|
||||
targetId: hnswVerb.targetId,
|
||||
|
||||
// Metadata field
|
||||
metadata: metadata
|
||||
}
|
||||
|
||||
items.push(graphVerb)
|
||||
|
||||
// Copy connections
|
||||
for (const [level, connections] of hnswVerb.connections.entries()) {
|
||||
verbWithMetadata.connections.set(level, new Set(connections))
|
||||
}
|
||||
|
||||
items.push(verbWithMetadata)
|
||||
}
|
||||
|
||||
return {
|
||||
|
|
@ -502,7 +503,7 @@ export class MemoryStorage extends BaseStorage {
|
|||
* Get verbs by source
|
||||
* @deprecated Use getVerbs() with filter.sourceId instead
|
||||
*/
|
||||
protected async getVerbsBySource_internal(sourceId: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsBySource_internal(sourceId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
const result = await this.getVerbs({
|
||||
filter: {
|
||||
sourceId
|
||||
|
|
@ -515,7 +516,7 @@ export class MemoryStorage extends BaseStorage {
|
|||
* Get verbs by target
|
||||
* @deprecated Use getVerbs() with filter.targetId instead
|
||||
*/
|
||||
protected async getVerbsByTarget_internal(targetId: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByTarget_internal(targetId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
const result = await this.getVerbs({
|
||||
filter: {
|
||||
targetId
|
||||
|
|
@ -528,7 +529,7 @@ export class MemoryStorage extends BaseStorage {
|
|||
* Get verbs by type
|
||||
* @deprecated Use getVerbs() with filter.verbType instead
|
||||
*/
|
||||
protected async getVerbsByType_internal(type: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByType_internal(type: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
const result = await this.getVerbs({
|
||||
filter: {
|
||||
verbType: type
|
||||
|
|
@ -549,7 +550,7 @@ export class MemoryStorage extends BaseStorage {
|
|||
const metadata = await this.getVerbMetadata(id)
|
||||
if (metadata) {
|
||||
const verbType = metadata.verb || metadata.type || 'default'
|
||||
this.decrementVerbCount(verbType)
|
||||
this.decrementVerbCount(verbType as string)
|
||||
|
||||
// Delete the metadata using the base storage method
|
||||
await this.deleteVerbMetadata(id)
|
||||
|
|
@ -740,25 +741,35 @@ export class MemoryStorage extends BaseStorage {
|
|||
}
|
||||
|
||||
/**
|
||||
* Initialize counts from in-memory storage - O(1) operation
|
||||
* Initialize counts from in-memory storage - O(1) operation (v4.0.0)
|
||||
*/
|
||||
protected async initializeCounts(): Promise<void> {
|
||||
// For memory storage, initialize counts from current in-memory state
|
||||
this.totalNounCount = this.nouns.size
|
||||
this.totalVerbCount = this.verbMetadata.size
|
||||
this.totalVerbCount = this.verbs.size
|
||||
|
||||
// Initialize type-based counts by scanning current data
|
||||
// Initialize type-based counts by scanning metadata storage (v4.0.0)
|
||||
this.entityCounts.clear()
|
||||
this.verbCounts.clear()
|
||||
|
||||
for (const noun of this.nouns.values()) {
|
||||
const type = noun.metadata?.type || noun.metadata?.nounType || 'default'
|
||||
this.entityCounts.set(type, (this.entityCounts.get(type) || 0) + 1)
|
||||
// Count nouns by loading metadata for each
|
||||
for (const [nounId, noun] of this.nouns.entries()) {
|
||||
const metadata = await this.getNounMetadata(nounId)
|
||||
if (metadata) {
|
||||
const type = metadata.noun || 'default'
|
||||
this.entityCounts.set(type, (this.entityCounts.get(type) || 0) + 1)
|
||||
}
|
||||
}
|
||||
|
||||
for (const verbMetadata of this.verbMetadata.values()) {
|
||||
const type = verbMetadata?.verb || verbMetadata?.type || 'default'
|
||||
this.verbCounts.set(type, (this.verbCounts.get(type) || 0) + 1)
|
||||
// Count verbs by loading metadata for each
|
||||
for (const [verbId, verb] of this.verbs.entries()) {
|
||||
const metadata = await this.getVerbMetadata(verbId)
|
||||
if (metadata) {
|
||||
// VerbMetadata doesn't have verb type - that's in HNSWVerb now
|
||||
// Use the verb's type from the HNSWVerb itself
|
||||
const type = verb.verb || 'default'
|
||||
this.verbCounts.set(type, (this.verbCounts.get(type) || 0) + 1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -7,6 +7,10 @@ import {
|
|||
GraphVerb,
|
||||
HNSWNoun,
|
||||
HNSWVerb,
|
||||
NounMetadata,
|
||||
VerbMetadata,
|
||||
HNSWNounWithMetadata,
|
||||
HNSWVerbWithMetadata,
|
||||
StatisticsData
|
||||
} from '../../coreTypes.js'
|
||||
import {
|
||||
|
|
@ -266,21 +270,16 @@ export class OPFSStorage extends BaseStorage {
|
|||
connections.set(Number(level), new Set(nounIds as string[]))
|
||||
}
|
||||
|
||||
const node = {
|
||||
// v4.0.0: Return ONLY vector data (no metadata field)
|
||||
const node: HNSWNode = {
|
||||
id: data.id,
|
||||
vector: data.vector,
|
||||
connections,
|
||||
level: data.level || 0
|
||||
}
|
||||
|
||||
// Get metadata (entity data in 2-file system)
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
|
||||
// Combine into complete noun object
|
||||
return {
|
||||
...node,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
// Return pure vector structure
|
||||
return node
|
||||
} catch (error) {
|
||||
// Noun not found or other error
|
||||
return null
|
||||
|
|
@ -444,23 +443,18 @@ export class OPFSStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get a verb from storage (internal implementation)
|
||||
* Combines vector data from getEdge() with metadata from getVerbMetadata()
|
||||
* v4.0.0: Returns ONLY vector + core relational fields (no metadata field)
|
||||
* Base class combines with metadata via getVerb() -> HNSWVerbWithMetadata
|
||||
*/
|
||||
protected async getVerb_internal(id: string): Promise<HNSWVerb | null> {
|
||||
// Get vector data (lightweight)
|
||||
// v4.0.0: Return ONLY vector + core relational data (no metadata field)
|
||||
const edge = await this.getEdge(id)
|
||||
if (!edge) {
|
||||
return null
|
||||
}
|
||||
|
||||
// Get metadata (relationship data in 2-file system)
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
|
||||
// Combine into complete verb object
|
||||
return {
|
||||
...edge,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
// Return pure vector + core fields structure
|
||||
return edge
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -502,7 +496,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
version: '1.0'
|
||||
}
|
||||
|
||||
// ARCHITECTURAL FIX (v3.50.1): Return HNSWVerb with core relational fields
|
||||
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
|
||||
return {
|
||||
id: data.id,
|
||||
vector: data.vector,
|
||||
|
|
@ -511,10 +505,10 @@ export class OPFSStorage extends BaseStorage {
|
|||
// CORE RELATIONAL DATA (read from vector file)
|
||||
verb: data.verb,
|
||||
sourceId: data.sourceId,
|
||||
targetId: data.targetId,
|
||||
targetId: data.targetId
|
||||
|
||||
// User metadata (retrieved separately via getVerbMetadata())
|
||||
metadata: data.metadata
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
// User metadata retrieved separately via getVerbMetadata()
|
||||
}
|
||||
} catch (error) {
|
||||
// Edge not found or other error
|
||||
|
|
@ -563,7 +557,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
version: '1.0'
|
||||
}
|
||||
|
||||
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields
|
||||
// v4.0.0: Include core relational fields (NO metadata field)
|
||||
allEdges.push({
|
||||
id: data.id,
|
||||
vector: data.vector,
|
||||
|
|
@ -572,10 +566,10 @@ export class OPFSStorage extends BaseStorage {
|
|||
// CORE RELATIONAL DATA
|
||||
verb: data.verb,
|
||||
sourceId: data.sourceId,
|
||||
targetId: data.targetId,
|
||||
targetId: data.targetId
|
||||
|
||||
// User metadata
|
||||
metadata: data.metadata
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
// User metadata retrieved separately via getVerbMetadata()
|
||||
})
|
||||
} catch (error) {
|
||||
console.error(`Error reading edge file ${shardName}/${fileName}:`, error)
|
||||
|
|
@ -596,7 +590,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
*/
|
||||
protected async getVerbsBySource_internal(
|
||||
sourceId: string
|
||||
): Promise<GraphVerb[]> {
|
||||
): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Use the paginated approach to properly handle HNSWVerb to GraphVerb conversion
|
||||
const result = await this.getVerbsWithPagination({
|
||||
filter: { sourceId: [sourceId] },
|
||||
|
|
@ -608,7 +602,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
/**
|
||||
* Get edges by source
|
||||
*/
|
||||
protected async getEdgesBySource(sourceId: string): Promise<GraphVerb[]> {
|
||||
protected async getEdgesBySource(sourceId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// This method is deprecated and would require loading metadata for each edge
|
||||
// For now, return empty array since this is not efficiently implementable with new storage pattern
|
||||
console.warn(
|
||||
|
|
@ -622,7 +616,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
*/
|
||||
protected async getVerbsByTarget_internal(
|
||||
targetId: string
|
||||
): Promise<GraphVerb[]> {
|
||||
): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Use the paginated approach to properly handle HNSWVerb to GraphVerb conversion
|
||||
const result = await this.getVerbsWithPagination({
|
||||
filter: { targetId: [targetId] },
|
||||
|
|
@ -634,7 +628,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
/**
|
||||
* Get edges by target
|
||||
*/
|
||||
protected async getEdgesByTarget(targetId: string): Promise<GraphVerb[]> {
|
||||
protected async getEdgesByTarget(targetId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// This method is deprecated and would require loading metadata for each edge
|
||||
// For now, return empty array since this is not efficiently implementable with new storage pattern
|
||||
console.warn(
|
||||
|
|
@ -646,7 +640,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by type (internal implementation)
|
||||
*/
|
||||
protected async getVerbsByType_internal(type: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByType_internal(type: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Use the paginated approach to properly handle HNSWVerb to GraphVerb conversion
|
||||
const result = await this.getVerbsWithPagination({
|
||||
filter: { verbType: [type] },
|
||||
|
|
@ -658,7 +652,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
/**
|
||||
* Get edges by type
|
||||
*/
|
||||
protected async getEdgesByType(type: string): Promise<GraphVerb[]> {
|
||||
protected async getEdgesByType(type: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// This method is deprecated and would require loading metadata for each edge
|
||||
// For now, return empty array since this is not efficiently implementable with new storage pattern
|
||||
console.warn(
|
||||
|
|
@ -1515,7 +1509,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<{
|
||||
items: HNSWNoun[]
|
||||
items: HNSWNounWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -1557,40 +1551,41 @@ export class OPFSStorage extends BaseStorage {
|
|||
// Get the subset of files for this page
|
||||
const pageFiles = nounFiles.slice(startIndex, startIndex + limit)
|
||||
|
||||
// Load nouns from files
|
||||
const items: HNSWNoun[] = []
|
||||
// v4.0.0: Load nouns from files and combine with metadata
|
||||
const items: HNSWNounWithMetadata[] = []
|
||||
for (const fileName of pageFiles) {
|
||||
// fileName is in format "shard/uuid.json", extract just the UUID
|
||||
const id = fileName.split('/')[1].replace('.json', '')
|
||||
const noun = await this.getNoun_internal(id)
|
||||
if (noun) {
|
||||
// Load metadata for filtering and combining
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
if (!metadata) continue
|
||||
|
||||
// Apply filters if provided
|
||||
if (options.filter) {
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
|
||||
// Filter by noun type
|
||||
if (options.filter.nounType) {
|
||||
const nounTypes = Array.isArray(options.filter.nounType)
|
||||
? options.filter.nounType
|
||||
: [options.filter.nounType]
|
||||
if (metadata && !nounTypes.includes(metadata.type || metadata.noun)) {
|
||||
if (!nounTypes.includes((metadata.type || metadata.noun) as string)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// Filter by service
|
||||
if (options.filter.service) {
|
||||
const services = Array.isArray(options.filter.service)
|
||||
? options.filter.service
|
||||
: [options.filter.service]
|
||||
if (metadata && !services.includes(metadata.createdBy?.augmentation)) {
|
||||
if (!metadata.createdBy?.augmentation || !services.includes(metadata.createdBy.augmentation as string)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// Filter by metadata
|
||||
if (options.filter.metadata) {
|
||||
if (!metadata) continue
|
||||
let matches = true
|
||||
for (const [key, value] of Object.entries(options.filter.metadata)) {
|
||||
if (metadata[key] !== value) {
|
||||
|
|
@ -1601,8 +1596,17 @@ export class OPFSStorage extends BaseStorage {
|
|||
if (!matches) continue
|
||||
}
|
||||
}
|
||||
|
||||
items.push(noun)
|
||||
|
||||
// v4.0.0: Create HNSWNounWithMetadata by combining noun with metadata
|
||||
const nounWithMetadata: HNSWNounWithMetadata = {
|
||||
id: noun.id,
|
||||
vector: [...noun.vector],
|
||||
connections: new Map(noun.connections),
|
||||
level: noun.level || 0,
|
||||
metadata: metadata
|
||||
}
|
||||
|
||||
items.push(nounWithMetadata)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1638,7 +1642,7 @@ export class OPFSStorage extends BaseStorage {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<{
|
||||
items: GraphVerb[]
|
||||
items: HNSWVerbWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -1680,73 +1684,87 @@ export class OPFSStorage extends BaseStorage {
|
|||
// Get the subset of files for this page
|
||||
const pageFiles = verbFiles.slice(startIndex, startIndex + limit)
|
||||
|
||||
// Load verbs from files and convert to GraphVerb
|
||||
const items: GraphVerb[] = []
|
||||
// v4.0.0: Load verbs from files and combine with metadata
|
||||
const items: HNSWVerbWithMetadata[] = []
|
||||
for (const fileName of pageFiles) {
|
||||
// fileName is in format "shard/uuid.json", extract just the UUID
|
||||
const id = fileName.split('/')[1].replace('.json', '')
|
||||
const hnswVerb = await this.getVerb_internal(id)
|
||||
if (hnswVerb) {
|
||||
// Convert HNSWVerb to GraphVerb
|
||||
const graphVerb = await this.convertHNSWVerbToGraphVerb(hnswVerb)
|
||||
if (graphVerb) {
|
||||
// Apply filters if provided
|
||||
if (options.filter) {
|
||||
// Filter by verb type
|
||||
if (options.filter.verbType) {
|
||||
const verbTypes = Array.isArray(options.filter.verbType)
|
||||
? options.filter.verbType
|
||||
: [options.filter.verbType]
|
||||
if (graphVerb.verb && !verbTypes.includes(graphVerb.verb)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by source ID
|
||||
if (options.filter.sourceId) {
|
||||
const sourceIds = Array.isArray(options.filter.sourceId)
|
||||
? options.filter.sourceId
|
||||
: [options.filter.sourceId]
|
||||
if (graphVerb.source && !sourceIds.includes(graphVerb.source)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by target ID
|
||||
if (options.filter.targetId) {
|
||||
const targetIds = Array.isArray(options.filter.targetId)
|
||||
? options.filter.targetId
|
||||
: [options.filter.targetId]
|
||||
if (graphVerb.target && !targetIds.includes(graphVerb.target)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by service
|
||||
if (options.filter.service) {
|
||||
const services = Array.isArray(options.filter.service)
|
||||
? options.filter.service
|
||||
: [options.filter.service]
|
||||
if (graphVerb.createdBy?.augmentation && !services.includes(graphVerb.createdBy.augmentation)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by metadata
|
||||
if (options.filter.metadata && graphVerb.metadata) {
|
||||
let matches = true
|
||||
for (const [key, value] of Object.entries(options.filter.metadata)) {
|
||||
if (graphVerb.metadata[key] !== value) {
|
||||
matches = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if (!matches) continue
|
||||
// Load metadata for filtering and combining
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
if (!metadata) continue
|
||||
|
||||
// Apply filters if provided
|
||||
if (options.filter) {
|
||||
// Filter by verb type
|
||||
// v4.0.0: verb field is in HNSWVerb structure (NOT in metadata)
|
||||
if (options.filter.verbType) {
|
||||
const verbTypes = Array.isArray(options.filter.verbType)
|
||||
? options.filter.verbType
|
||||
: [options.filter.verbType]
|
||||
if (!hnswVerb.verb || !verbTypes.includes(hnswVerb.verb)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
items.push(graphVerb)
|
||||
|
||||
// Filter by source ID
|
||||
// v4.0.0: sourceId field is in HNSWVerb structure (NOT in metadata)
|
||||
if (options.filter.sourceId) {
|
||||
const sourceIds = Array.isArray(options.filter.sourceId)
|
||||
? options.filter.sourceId
|
||||
: [options.filter.sourceId]
|
||||
if (!hnswVerb.sourceId || !sourceIds.includes(hnswVerb.sourceId)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by target ID
|
||||
// v4.0.0: targetId field is in HNSWVerb structure (NOT in metadata)
|
||||
if (options.filter.targetId) {
|
||||
const targetIds = Array.isArray(options.filter.targetId)
|
||||
? options.filter.targetId
|
||||
: [options.filter.targetId]
|
||||
if (!hnswVerb.targetId || !targetIds.includes(hnswVerb.targetId)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by service
|
||||
if (options.filter.service) {
|
||||
const services = Array.isArray(options.filter.service)
|
||||
? options.filter.service
|
||||
: [options.filter.service]
|
||||
if (!metadata.createdBy?.augmentation || !services.includes(metadata.createdBy.augmentation as string)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by metadata
|
||||
if (options.filter.metadata) {
|
||||
let matches = true
|
||||
for (const [key, value] of Object.entries(options.filter.metadata)) {
|
||||
if (metadata[key] !== value) {
|
||||
matches = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if (!matches) continue
|
||||
}
|
||||
}
|
||||
|
||||
// v4.0.0: Create HNSWVerbWithMetadata by combining verb with metadata
|
||||
const verbWithMetadata: HNSWVerbWithMetadata = {
|
||||
id: hnswVerb.id,
|
||||
vector: [...hnswVerb.vector],
|
||||
connections: new Map(hnswVerb.connections),
|
||||
verb: hnswVerb.verb,
|
||||
sourceId: hnswVerb.sourceId,
|
||||
targetId: hnswVerb.targetId,
|
||||
metadata: metadata
|
||||
}
|
||||
|
||||
items.push(verbWithMetadata)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,16 @@
|
|||
* Based on latest GCS and S3 implementations with R2-specific enhancements
|
||||
*/
|
||||
|
||||
import { GraphVerb, HNSWNoun, HNSWVerb, StatisticsData } from '../../coreTypes.js'
|
||||
import {
|
||||
GraphVerb,
|
||||
HNSWNoun,
|
||||
HNSWVerb,
|
||||
NounMetadata,
|
||||
VerbMetadata,
|
||||
HNSWNounWithMetadata,
|
||||
HNSWVerbWithMetadata,
|
||||
StatisticsData
|
||||
} from '../../coreTypes.js'
|
||||
import {
|
||||
BaseStorage,
|
||||
NOUNS_DIR,
|
||||
|
|
@ -426,7 +435,7 @@ export class R2Storage extends BaseStorage {
|
|||
// Increment noun count
|
||||
const metadata = await this.getNounMetadata(node.id)
|
||||
if (metadata && metadata.type) {
|
||||
await this.incrementEntityCountSafe(metadata.type)
|
||||
await this.incrementEntityCountSafe(metadata.type as string)
|
||||
}
|
||||
|
||||
this.logger.trace(`Node ${node.id} saved successfully`)
|
||||
|
|
@ -446,19 +455,18 @@ export class R2Storage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get a noun from storage (internal implementation)
|
||||
* v4.0.0: Returns ONLY vector data (no metadata field)
|
||||
* Base class combines with metadata via getNoun() -> HNSWNounWithMetadata
|
||||
*/
|
||||
protected async getNoun_internal(id: string): Promise<HNSWNoun | null> {
|
||||
// v4.0.0: Return ONLY vector data (no metadata field)
|
||||
const node = await this.getNode(id)
|
||||
if (!node) {
|
||||
return null
|
||||
}
|
||||
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
|
||||
return {
|
||||
...node,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
// Return pure vector structure
|
||||
return node
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -565,7 +573,7 @@ export class R2Storage extends BaseStorage {
|
|||
// Decrement noun count
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
if (metadata && metadata.type) {
|
||||
await this.decrementEntityCountSafe(metadata.type)
|
||||
await this.decrementEntityCountSafe(metadata.type as string)
|
||||
}
|
||||
|
||||
this.logger.trace(`Noun ${id} deleted successfully`)
|
||||
|
|
@ -765,7 +773,7 @@ export class R2Storage extends BaseStorage {
|
|||
|
||||
const metadata = await this.getVerbMetadata(edge.id)
|
||||
if (metadata && metadata.type) {
|
||||
await this.incrementVerbCount(metadata.type)
|
||||
await this.incrementVerbCount(metadata.type as string)
|
||||
}
|
||||
|
||||
this.releaseBackpressure(true, requestId)
|
||||
|
|
@ -781,18 +789,20 @@ export class R2Storage extends BaseStorage {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a verb from storage (internal implementation)
|
||||
* v4.0.0: Returns ONLY vector + core relational fields (no metadata field)
|
||||
* Base class combines with metadata via getVerb() -> HNSWVerbWithMetadata
|
||||
*/
|
||||
protected async getVerb_internal(id: string): Promise<HNSWVerb | null> {
|
||||
// v4.0.0: Return ONLY vector + core relational data (no metadata field)
|
||||
const edge = await this.getEdge(id)
|
||||
if (!edge) {
|
||||
return null
|
||||
}
|
||||
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
|
||||
return {
|
||||
...edge,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
// Return pure vector + core fields structure
|
||||
return edge
|
||||
}
|
||||
|
||||
protected async getEdge(id: string): Promise<Edge | null> {
|
||||
|
|
@ -824,7 +834,7 @@ export class R2Storage extends BaseStorage {
|
|||
connections.set(Number(level), new Set(verbIds as string[]))
|
||||
}
|
||||
|
||||
// ARCHITECTURAL FIX (v3.50.1): Return HNSWVerb with core relational fields
|
||||
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
|
||||
const edge: Edge = {
|
||||
id: data.id,
|
||||
vector: data.vector,
|
||||
|
|
@ -833,10 +843,10 @@ export class R2Storage extends BaseStorage {
|
|||
// CORE RELATIONAL DATA (read from vector file)
|
||||
verb: data.verb,
|
||||
sourceId: data.sourceId,
|
||||
targetId: data.targetId,
|
||||
targetId: data.targetId
|
||||
|
||||
// User metadata (retrieved separately via getVerbMetadata())
|
||||
metadata: data.metadata
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
// User metadata retrieved separately via getVerbMetadata()
|
||||
}
|
||||
|
||||
this.verbCacheManager.set(id, edge)
|
||||
|
|
@ -878,7 +888,7 @@ export class R2Storage extends BaseStorage {
|
|||
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
if (metadata && metadata.type) {
|
||||
await this.decrementVerbCount(metadata.type)
|
||||
await this.decrementVerbCount(metadata.type as string)
|
||||
}
|
||||
|
||||
this.releaseBackpressure(true, requestId)
|
||||
|
|
@ -1130,7 +1140,7 @@ export class R2Storage extends BaseStorage {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<{
|
||||
items: HNSWNoun[]
|
||||
items: HNSWNounWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -1150,7 +1160,7 @@ export class R2Storage extends BaseStorage {
|
|||
})
|
||||
)
|
||||
|
||||
const items: HNSWNoun[] = []
|
||||
const items: HNSWNounWithMetadata[] = []
|
||||
const contents = response.Contents || []
|
||||
|
||||
for (const obj of contents) {
|
||||
|
|
@ -1161,7 +1171,55 @@ export class R2Storage extends BaseStorage {
|
|||
|
||||
const noun = await this.getNoun_internal(id)
|
||||
if (noun) {
|
||||
items.push(noun)
|
||||
// v4.0.0: Load metadata and combine with noun to create HNSWNounWithMetadata
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
if (!metadata) continue
|
||||
|
||||
// Apply filters if provided
|
||||
if (options.filter) {
|
||||
// Filter by noun type
|
||||
if (options.filter.nounType) {
|
||||
const nounTypes = Array.isArray(options.filter.nounType)
|
||||
? options.filter.nounType
|
||||
: [options.filter.nounType]
|
||||
if (!nounTypes.includes((metadata.type || metadata.noun) as string)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by service
|
||||
if (options.filter.service) {
|
||||
const services = Array.isArray(options.filter.service)
|
||||
? options.filter.service
|
||||
: [options.filter.service]
|
||||
if (!metadata.createdBy?.augmentation || !services.includes(metadata.createdBy.augmentation as string)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by metadata
|
||||
if (options.filter.metadata) {
|
||||
let matches = true
|
||||
for (const [key, value] of Object.entries(options.filter.metadata)) {
|
||||
if (metadata[key] !== value) {
|
||||
matches = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if (!matches) continue
|
||||
}
|
||||
}
|
||||
|
||||
// v4.0.0: Create HNSWNounWithMetadata by combining noun with metadata
|
||||
const nounWithMetadata: HNSWNounWithMetadata = {
|
||||
id: noun.id,
|
||||
vector: [...noun.vector],
|
||||
connections: new Map(noun.connections),
|
||||
level: noun.level || 0,
|
||||
metadata: metadata
|
||||
}
|
||||
|
||||
items.push(nounWithMetadata)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1182,16 +1240,16 @@ export class R2Storage extends BaseStorage {
|
|||
return result.items
|
||||
}
|
||||
|
||||
protected async getVerbsBySource_internal(sourceId: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsBySource_internal(sourceId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Simplified - full implementation would include proper filtering
|
||||
return []
|
||||
}
|
||||
|
||||
protected async getVerbsByTarget_internal(targetId: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByTarget_internal(targetId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
return []
|
||||
}
|
||||
|
||||
protected async getVerbsByType_internal(type: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByType_internal(type: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
return []
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -4,7 +4,17 @@
|
|||
* including Amazon S3, Cloudflare R2, and Google Cloud Storage
|
||||
*/
|
||||
|
||||
import { GraphVerb, HNSWNoun, HNSWVerb, StatisticsData } from '../../coreTypes.js'
|
||||
import {
|
||||
Change,
|
||||
GraphVerb,
|
||||
HNSWNoun,
|
||||
HNSWVerb,
|
||||
NounMetadata,
|
||||
VerbMetadata,
|
||||
HNSWNounWithMetadata,
|
||||
HNSWVerbWithMetadata,
|
||||
StatisticsData
|
||||
} from '../../coreTypes.js'
|
||||
import {
|
||||
BaseStorage,
|
||||
NOUNS_DIR,
|
||||
|
|
@ -1002,15 +1012,15 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
|
||||
this.logger.debug(`Node ${node.id} saved successfully`)
|
||||
|
||||
// Log the change for efficient synchronization
|
||||
// Log the change for efficient synchronization (v4.0.0: no metadata on node)
|
||||
await this.appendToChangeLog({
|
||||
timestamp: Date.now(),
|
||||
operation: 'add', // Could be 'update' if we track existing nodes
|
||||
entityType: 'noun',
|
||||
entityId: node.id,
|
||||
data: {
|
||||
vector: node.vector,
|
||||
metadata: node.metadata
|
||||
vector: node.vector
|
||||
// ✅ NO metadata field in v4.0.0 - stored separately
|
||||
}
|
||||
})
|
||||
|
||||
|
|
@ -1042,8 +1052,8 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
this.totalNounCount++
|
||||
const metadata = await this.getNounMetadata(node.id)
|
||||
if (metadata && metadata.type) {
|
||||
const currentCount = this.entityCounts.get(metadata.type) || 0
|
||||
this.entityCounts.set(metadata.type, currentCount + 1)
|
||||
const currentCount = this.entityCounts.get(metadata.type as string) || 0
|
||||
this.entityCounts.set(metadata.type as string, currentCount + 1)
|
||||
}
|
||||
|
||||
// Release backpressure on success
|
||||
|
|
@ -1058,23 +1068,18 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get a noun from storage (internal implementation)
|
||||
* Combines vector data from getNode() with metadata from getNounMetadata()
|
||||
* v4.0.0: Returns ONLY vector data (no metadata field)
|
||||
* Base class combines with metadata via getNoun() -> HNSWNounWithMetadata
|
||||
*/
|
||||
protected async getNoun_internal(id: string): Promise<HNSWNoun | null> {
|
||||
// Get vector data (lightweight)
|
||||
// v4.0.0: Return ONLY vector data (no metadata field)
|
||||
const node = await this.getNode(id)
|
||||
if (!node) {
|
||||
return null
|
||||
}
|
||||
|
||||
// Get metadata (entity data in 2-file system)
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
|
||||
// Combine into complete noun object
|
||||
return {
|
||||
...node,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
// Return pure vector structure
|
||||
return node
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -1559,8 +1564,8 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
this.totalVerbCount++
|
||||
const metadata = await this.getVerbMetadata(edge.id)
|
||||
if (metadata && metadata.type) {
|
||||
const currentCount = this.verbCounts.get(metadata.type) || 0
|
||||
this.verbCounts.set(metadata.type, currentCount + 1)
|
||||
const currentCount = this.verbCounts.get(metadata.type as string) || 0
|
||||
this.verbCounts.set(metadata.type as string, currentCount + 1)
|
||||
}
|
||||
|
||||
// Release backpressure on success
|
||||
|
|
@ -1575,23 +1580,18 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
|
||||
/**
|
||||
* Get a verb from storage (internal implementation)
|
||||
* Combines vector data from getEdge() with metadata from getVerbMetadata()
|
||||
* v4.0.0: Returns ONLY vector + core relational fields (no metadata field)
|
||||
* Base class combines with metadata via getVerb() -> HNSWVerbWithMetadata
|
||||
*/
|
||||
protected async getVerb_internal(id: string): Promise<HNSWVerb | null> {
|
||||
// Get vector data (lightweight)
|
||||
// v4.0.0: Return ONLY vector + core relational data (no metadata field)
|
||||
const edge = await this.getEdge(id)
|
||||
if (!edge) {
|
||||
return null
|
||||
}
|
||||
|
||||
// Get metadata (relationship data in 2-file system)
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
|
||||
// Combine into complete verb object
|
||||
return {
|
||||
...edge,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
// Return pure vector + core fields structure
|
||||
return edge
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -1647,7 +1647,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
connections.set(Number(level), new Set(nodeIds as string[]))
|
||||
}
|
||||
|
||||
// ARCHITECTURAL FIX (v3.50.1): Return HNSWVerb with core relational fields
|
||||
// v4.0.0: Return HNSWVerb with core relational fields (NO metadata field)
|
||||
const edge = {
|
||||
id: parsedEdge.id,
|
||||
vector: parsedEdge.vector,
|
||||
|
|
@ -1656,10 +1656,10 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
// CORE RELATIONAL DATA (read from vector file)
|
||||
verb: parsedEdge.verb,
|
||||
sourceId: parsedEdge.sourceId,
|
||||
targetId: parsedEdge.targetId,
|
||||
targetId: parsedEdge.targetId
|
||||
|
||||
// User metadata (retrieved separately via getVerbMetadata())
|
||||
metadata: parsedEdge.metadata
|
||||
// ✅ NO metadata field in v4.0.0
|
||||
// User metadata retrieved separately via getVerbMetadata()
|
||||
}
|
||||
|
||||
this.logger.trace(`Successfully retrieved edge ${id}`)
|
||||
|
|
@ -1869,7 +1869,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<{
|
||||
items: GraphVerb[]
|
||||
items: HNSWVerbWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -1914,55 +1914,63 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
filter: edgeFilter
|
||||
})
|
||||
|
||||
// Convert HNSWVerbs to GraphVerbs by combining with metadata
|
||||
const graphVerbs: GraphVerb[] = []
|
||||
// v4.0.0: Convert HNSWVerbs to HNSWVerbWithMetadata by combining with metadata
|
||||
const verbsWithMetadata: HNSWVerbWithMetadata[] = []
|
||||
for (const hnswVerb of result.edges) {
|
||||
const graphVerb = await this.convertHNSWVerbToGraphVerb(hnswVerb)
|
||||
if (graphVerb) {
|
||||
graphVerbs.push(graphVerb)
|
||||
const metadata = await this.getVerbMetadata(hnswVerb.id)
|
||||
const verbWithMetadata: HNSWVerbWithMetadata = {
|
||||
id: hnswVerb.id,
|
||||
vector: [...hnswVerb.vector],
|
||||
connections: new Map(hnswVerb.connections),
|
||||
verb: hnswVerb.verb,
|
||||
sourceId: hnswVerb.sourceId,
|
||||
targetId: hnswVerb.targetId,
|
||||
metadata: metadata || {}
|
||||
}
|
||||
verbsWithMetadata.push(verbWithMetadata)
|
||||
}
|
||||
|
||||
// Apply filtering at GraphVerb level since HNSWVerb filtering is not supported
|
||||
let filteredGraphVerbs = graphVerbs
|
||||
|
||||
// Apply filtering at HNSWVerbWithMetadata level
|
||||
// v4.0.0: Core fields (verb, sourceId, targetId) are in HNSWVerb, not metadata
|
||||
let filteredVerbs = verbsWithMetadata
|
||||
if (options.filter) {
|
||||
filteredGraphVerbs = graphVerbs.filter((graphVerb) => {
|
||||
filteredVerbs = verbsWithMetadata.filter((verbWithMetadata) => {
|
||||
// Filter by sourceId
|
||||
if (options.filter!.sourceId) {
|
||||
const sourceIds = Array.isArray(options.filter!.sourceId)
|
||||
? options.filter!.sourceId
|
||||
: [options.filter!.sourceId]
|
||||
if (!sourceIds.includes(graphVerb.sourceId)) {
|
||||
if (!verbWithMetadata.sourceId || !sourceIds.includes(verbWithMetadata.sourceId)) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// Filter by targetId
|
||||
if (options.filter!.targetId) {
|
||||
const targetIds = Array.isArray(options.filter!.targetId)
|
||||
? options.filter!.targetId
|
||||
: [options.filter!.targetId]
|
||||
if (!targetIds.includes(graphVerb.targetId)) {
|
||||
if (!verbWithMetadata.targetId || !targetIds.includes(verbWithMetadata.targetId)) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// Filter by verbType (maps to type field)
|
||||
|
||||
// Filter by verbType
|
||||
if (options.filter!.verbType) {
|
||||
const verbTypes = Array.isArray(options.filter!.verbType)
|
||||
? options.filter!.verbType
|
||||
: [options.filter!.verbType]
|
||||
if (graphVerb.type && !verbTypes.includes(graphVerb.type)) {
|
||||
if (!verbWithMetadata.verb || !verbTypes.includes(verbWithMetadata.verb)) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
return true
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
return {
|
||||
items: filteredGraphVerbs,
|
||||
items: filteredVerbs,
|
||||
totalCount: this.totalVerbCount, // Use pre-calculated count from init()
|
||||
hasMore: result.hasMore,
|
||||
nextCursor: result.nextCursor
|
||||
|
|
@ -1975,7 +1983,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by source (internal implementation)
|
||||
*/
|
||||
protected async getVerbsBySource_internal(sourceId: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsBySource_internal(sourceId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Use the paginated approach to properly handle HNSWVerb to GraphVerb conversion
|
||||
const result = await this.getVerbsWithPagination({
|
||||
filter: { sourceId: [sourceId] },
|
||||
|
|
@ -1987,7 +1995,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by target (internal implementation)
|
||||
*/
|
||||
protected async getVerbsByTarget_internal(targetId: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByTarget_internal(targetId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Use the paginated approach to properly handle HNSWVerb to GraphVerb conversion
|
||||
const result = await this.getVerbsWithPagination({
|
||||
filter: { targetId: [targetId] },
|
||||
|
|
@ -1999,7 +2007,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by type (internal implementation)
|
||||
*/
|
||||
protected async getVerbsByType_internal(type: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByType_internal(type: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Use the paginated approach to properly handle HNSWVerb to GraphVerb conversion
|
||||
const result = await this.getVerbsWithPagination({
|
||||
filter: { verbType: [type] },
|
||||
|
|
@ -3025,7 +3033,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
public async getChangesSince(
|
||||
sinceTimestamp: number,
|
||||
maxEntries: number = 1000
|
||||
): Promise<ChangeLogEntry[]> {
|
||||
): Promise<Change[]> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
try {
|
||||
|
|
@ -3047,7 +3055,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
return []
|
||||
}
|
||||
|
||||
const changes: ChangeLogEntry[] = []
|
||||
const changes: Change[] = []
|
||||
|
||||
// Process each change log entry
|
||||
for (const object of response.Contents) {
|
||||
|
|
@ -3068,7 +3076,15 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
|
||||
// Only include entries newer than the specified timestamp
|
||||
if (entry.timestamp > sinceTimestamp) {
|
||||
changes.push(entry)
|
||||
// Convert ChangeLogEntry to Change
|
||||
const change: Change = {
|
||||
id: entry.entityId,
|
||||
type: entry.entityType === 'metadata' ? 'noun' : (entry.entityType as 'noun' | 'verb'),
|
||||
operation: entry.operation === 'add' ? 'create' : entry.operation as 'create' | 'update' | 'delete',
|
||||
timestamp: entry.timestamp,
|
||||
data: entry.data
|
||||
}
|
||||
changes.push(change)
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
|
|
@ -3464,7 +3480,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
} = {}): Promise<{
|
||||
items: HNSWNoun[]
|
||||
items: HNSWNounWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -3480,64 +3496,64 @@ export class S3CompatibleStorage extends BaseStorage {
|
|||
cursor,
|
||||
useCache: true
|
||||
})
|
||||
|
||||
// Apply filters if provided
|
||||
let filteredNodes = result.nodes
|
||||
|
||||
if (options.filter) {
|
||||
// Filter by noun type
|
||||
if (options.filter.nounType) {
|
||||
const nounTypes = Array.isArray(options.filter.nounType)
|
||||
? options.filter.nounType
|
||||
: [options.filter.nounType]
|
||||
|
||||
const filteredByType: HNSWNoun[] = []
|
||||
for (const node of filteredNodes) {
|
||||
const metadata = await this.getNounMetadata(node.id)
|
||||
if (metadata && nounTypes.includes(metadata.type || metadata.noun)) {
|
||||
filteredByType.push(node)
|
||||
|
||||
// v4.0.0: Combine nodes with metadata to create HNSWNounWithMetadata[]
|
||||
const nounsWithMetadata: HNSWNounWithMetadata[] = []
|
||||
|
||||
for (const node of result.nodes) {
|
||||
const metadata = await this.getNounMetadata(node.id)
|
||||
if (!metadata) continue
|
||||
|
||||
// Apply filters if provided
|
||||
if (options.filter) {
|
||||
// Filter by noun type
|
||||
if (options.filter.nounType) {
|
||||
const nounTypes = Array.isArray(options.filter.nounType)
|
||||
? options.filter.nounType
|
||||
: [options.filter.nounType]
|
||||
|
||||
const nounType = (metadata.type || metadata.noun) as string
|
||||
if (!nounType || !nounTypes.includes(nounType)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
filteredNodes = filteredByType
|
||||
}
|
||||
|
||||
// Filter by service
|
||||
if (options.filter.service) {
|
||||
const services = Array.isArray(options.filter.service)
|
||||
? options.filter.service
|
||||
: [options.filter.service]
|
||||
|
||||
const filteredByService: HNSWNoun[] = []
|
||||
for (const node of filteredNodes) {
|
||||
const metadata = await this.getNounMetadata(node.id)
|
||||
if (metadata && services.includes(metadata.service)) {
|
||||
filteredByService.push(node)
|
||||
|
||||
// Filter by service
|
||||
if (options.filter.service) {
|
||||
const services = Array.isArray(options.filter.service)
|
||||
? options.filter.service
|
||||
: [options.filter.service]
|
||||
|
||||
if (!metadata.service || !services.includes(metadata.service as string)) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
filteredNodes = filteredByService
|
||||
}
|
||||
|
||||
// Filter by metadata
|
||||
if (options.filter.metadata) {
|
||||
const metadataFilter = options.filter.metadata
|
||||
const filteredByMetadata: HNSWNoun[] = []
|
||||
for (const node of filteredNodes) {
|
||||
const metadata = await this.getNounMetadata(node.id)
|
||||
if (metadata) {
|
||||
const matches = Object.entries(metadataFilter).every(
|
||||
([key, value]) => metadata[key] === value
|
||||
)
|
||||
if (matches) {
|
||||
filteredByMetadata.push(node)
|
||||
}
|
||||
|
||||
// Filter by metadata fields
|
||||
if (options.filter.metadata) {
|
||||
const metadataFilter = options.filter.metadata
|
||||
const matches = Object.entries(metadataFilter).every(
|
||||
([key, value]) => metadata[key] === value
|
||||
)
|
||||
if (!matches) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
filteredNodes = filteredByMetadata
|
||||
}
|
||||
|
||||
// Create HNSWNounWithMetadata
|
||||
const nounWithMetadata: HNSWNounWithMetadata = {
|
||||
id: node.id,
|
||||
vector: [...node.vector],
|
||||
connections: new Map(node.connections),
|
||||
level: node.level || 0,
|
||||
metadata: metadata
|
||||
}
|
||||
nounsWithMetadata.push(nounWithMetadata)
|
||||
}
|
||||
|
||||
|
||||
return {
|
||||
items: filteredNodes,
|
||||
items: nounsWithMetadata,
|
||||
totalCount: this.totalNounCount, // Use pre-calculated count from init()
|
||||
hasMore: result.hasMore,
|
||||
nextCursor: result.nextCursor
|
||||
|
|
|
|||
|
|
@ -23,6 +23,9 @@ import {
|
|||
GraphVerb,
|
||||
HNSWNoun,
|
||||
HNSWVerb,
|
||||
HNSWVerbWithMetadata,
|
||||
NounMetadata,
|
||||
VerbMetadata,
|
||||
StatisticsData
|
||||
} from '../../coreTypes.js'
|
||||
import {
|
||||
|
|
@ -186,21 +189,20 @@ export class TypeAwareStorageAdapter extends BaseStorage {
|
|||
}
|
||||
|
||||
/**
|
||||
* Get noun type from noun object or cache
|
||||
* Get noun type from cache
|
||||
*
|
||||
* v4.0.0: Metadata is stored separately, so we rely on the cache
|
||||
* which is populated when saveNounMetadata is called
|
||||
*/
|
||||
private getNounType(noun: HNSWNoun): NounType {
|
||||
// Try metadata first (most reliable)
|
||||
if (noun.metadata?.noun) {
|
||||
return noun.metadata.noun as NounType
|
||||
}
|
||||
|
||||
// Try cache
|
||||
// Check cache (populated when metadata is saved)
|
||||
const cached = this.nounTypeCache.get(noun.id)
|
||||
if (cached) {
|
||||
return cached
|
||||
}
|
||||
|
||||
// Default to 'thing' if unknown
|
||||
// This should only happen if saveNoun_internal is called before saveNounMetadata
|
||||
console.warn(`[TypeAwareStorage] Unknown noun type for ${noun.id}, defaulting to 'thing'`)
|
||||
return 'thing'
|
||||
}
|
||||
|
|
@ -412,29 +414,44 @@ export class TypeAwareStorageAdapter extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by source
|
||||
*/
|
||||
protected async getVerbsBySource_internal(sourceId: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsBySource_internal(sourceId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Need to search across all verb types
|
||||
// TODO: Optimize with metadata index in Phase 1b
|
||||
const verbs: GraphVerb[] = []
|
||||
const verbs: HNSWVerbWithMetadata[] = []
|
||||
|
||||
for (let i = 0; i < VERB_TYPE_COUNT; i++) {
|
||||
const type = TypeUtils.getVerbFromIndex(i)
|
||||
const prefix = `entities/verbs/${type}/metadata/`
|
||||
const prefix = `entities/verbs/${type}/vectors/`
|
||||
const paths = await this.u.listObjectsUnderPath(prefix)
|
||||
|
||||
for (const path of paths) {
|
||||
try {
|
||||
const metadata = await this.u.readObjectFromPath(path)
|
||||
if (metadata && metadata.sourceId === sourceId) {
|
||||
// Load the full GraphVerb
|
||||
const id = path.split('/').pop()?.replace('.json', '')
|
||||
if (id) {
|
||||
const verb = await this.getVerb(id)
|
||||
if (verb) {
|
||||
verbs.push(verb)
|
||||
}
|
||||
}
|
||||
const id = path.split('/').pop()?.replace('.json', '')
|
||||
if (!id) continue
|
||||
|
||||
// Load the HNSWVerb
|
||||
const hnswVerb = await this.u.readObjectFromPath(path)
|
||||
if (!hnswVerb) continue
|
||||
|
||||
// Check sourceId from HNSWVerb (v4.0.0: core fields are in HNSWVerb)
|
||||
if (hnswVerb.sourceId !== sourceId) continue
|
||||
|
||||
// Load metadata separately
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
if (!metadata) continue
|
||||
|
||||
// Create HNSWVerbWithMetadata (verbs don't have level field)
|
||||
const verbWithMetadata: HNSWVerbWithMetadata = {
|
||||
id: hnswVerb.id,
|
||||
vector: [...hnswVerb.vector],
|
||||
connections: new Map(hnswVerb.connections),
|
||||
verb: hnswVerb.verb,
|
||||
sourceId: hnswVerb.sourceId,
|
||||
targetId: hnswVerb.targetId,
|
||||
metadata: metadata
|
||||
}
|
||||
|
||||
verbs.push(verbWithMetadata)
|
||||
} catch (error) {
|
||||
// Continue searching
|
||||
}
|
||||
|
|
@ -447,27 +464,43 @@ export class TypeAwareStorageAdapter extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by target
|
||||
*/
|
||||
protected async getVerbsByTarget_internal(targetId: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByTarget_internal(targetId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
// Similar to getVerbsBySource_internal
|
||||
const verbs: GraphVerb[] = []
|
||||
const verbs: HNSWVerbWithMetadata[] = []
|
||||
|
||||
for (let i = 0; i < VERB_TYPE_COUNT; i++) {
|
||||
const type = TypeUtils.getVerbFromIndex(i)
|
||||
const prefix = `entities/verbs/${type}/metadata/`
|
||||
const prefix = `entities/verbs/${type}/vectors/`
|
||||
const paths = await this.u.listObjectsUnderPath(prefix)
|
||||
|
||||
for (const path of paths) {
|
||||
try {
|
||||
const metadata = await this.u.readObjectFromPath(path)
|
||||
if (metadata && metadata.targetId === targetId) {
|
||||
const id = path.split('/').pop()?.replace('.json', '')
|
||||
if (id) {
|
||||
const verb = await this.getVerb(id)
|
||||
if (verb) {
|
||||
verbs.push(verb)
|
||||
}
|
||||
}
|
||||
const id = path.split('/').pop()?.replace('.json', '')
|
||||
if (!id) continue
|
||||
|
||||
// Load the HNSWVerb
|
||||
const hnswVerb = await this.u.readObjectFromPath(path)
|
||||
if (!hnswVerb) continue
|
||||
|
||||
// Check targetId from HNSWVerb (v4.0.0: core fields are in HNSWVerb)
|
||||
if (hnswVerb.targetId !== targetId) continue
|
||||
|
||||
// Load metadata separately
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
if (!metadata) continue
|
||||
|
||||
// Create HNSWVerbWithMetadata (verbs don't have level field)
|
||||
const verbWithMetadata: HNSWVerbWithMetadata = {
|
||||
id: hnswVerb.id,
|
||||
vector: [...hnswVerb.vector],
|
||||
connections: new Map(hnswVerb.connections),
|
||||
verb: hnswVerb.verb,
|
||||
sourceId: hnswVerb.sourceId,
|
||||
targetId: hnswVerb.targetId,
|
||||
metadata: metadata
|
||||
}
|
||||
|
||||
verbs.push(verbWithMetadata)
|
||||
} catch (error) {
|
||||
// Continue
|
||||
}
|
||||
|
|
@ -480,28 +513,39 @@ export class TypeAwareStorageAdapter extends BaseStorage {
|
|||
/**
|
||||
* Get verbs by type (O(1) with type-first paths!)
|
||||
*
|
||||
* ARCHITECTURAL FIX (v3.50.1): Type is now in HNSWVerb, cached on read
|
||||
* v4.0.0: Load verbs and combine with metadata
|
||||
*/
|
||||
protected async getVerbsByType_internal(verbType: string): Promise<GraphVerb[]> {
|
||||
protected async getVerbsByType_internal(verbType: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
const type = verbType as VerbType
|
||||
const prefix = `entities/verbs/${type}/vectors/`
|
||||
|
||||
const paths = await this.u.listObjectsUnderPath(prefix)
|
||||
const verbs: GraphVerb[] = []
|
||||
const verbs: HNSWVerbWithMetadata[] = []
|
||||
|
||||
for (const path of paths) {
|
||||
try {
|
||||
const hnswVerb = await this.u.readObjectFromPath(path)
|
||||
if (hnswVerb) {
|
||||
// Cache type from HNSWVerb for future O(1) retrievals
|
||||
this.verbTypeCache.set(hnswVerb.id, hnswVerb.verb as VerbType)
|
||||
if (!hnswVerb) continue
|
||||
|
||||
// Convert to GraphVerb
|
||||
const graphVerb = await this.convertHNSWVerbToGraphVerb(hnswVerb)
|
||||
if (graphVerb) {
|
||||
verbs.push(graphVerb)
|
||||
}
|
||||
// Cache type from HNSWVerb for future O(1) retrievals
|
||||
this.verbTypeCache.set(hnswVerb.id, hnswVerb.verb as VerbType)
|
||||
|
||||
// Load metadata separately
|
||||
const metadata = await this.getVerbMetadata(hnswVerb.id)
|
||||
if (!metadata) continue
|
||||
|
||||
// Create HNSWVerbWithMetadata (verbs don't have level field)
|
||||
const verbWithMetadata: HNSWVerbWithMetadata = {
|
||||
id: hnswVerb.id,
|
||||
vector: [...hnswVerb.vector],
|
||||
connections: new Map(hnswVerb.connections),
|
||||
verb: hnswVerb.verb,
|
||||
sourceId: hnswVerb.sourceId,
|
||||
targetId: hnswVerb.targetId,
|
||||
metadata: metadata
|
||||
}
|
||||
|
||||
verbs.push(verbWithMetadata)
|
||||
} catch (error) {
|
||||
console.warn(`[TypeAwareStorage] Failed to load verb from ${path}:`, error)
|
||||
}
|
||||
|
|
@ -547,6 +591,160 @@ export class TypeAwareStorageAdapter extends BaseStorage {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Save noun metadata (override to cache type for type-aware routing)
|
||||
*
|
||||
* v4.0.0: Extract and cache noun type when metadata is saved
|
||||
*/
|
||||
async saveNounMetadata(id: string, metadata: NounMetadata): Promise<void> {
|
||||
// Extract and cache the type
|
||||
const type = (metadata.noun || 'thing') as NounType
|
||||
this.nounTypeCache.set(id, type)
|
||||
|
||||
// Save to type-aware path
|
||||
const path = getNounMetadataPath(type, id)
|
||||
await this.u.writeObjectToPath(path, metadata)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get noun metadata (override to use type-aware paths)
|
||||
*/
|
||||
async getNounMetadata(id: string): Promise<NounMetadata | null> {
|
||||
// Try cache first
|
||||
const cachedType = this.nounTypeCache.get(id)
|
||||
if (cachedType) {
|
||||
const path = getNounMetadataPath(cachedType, id)
|
||||
return await this.u.readObjectFromPath(path)
|
||||
}
|
||||
|
||||
// Search across all types
|
||||
for (let i = 0; i < NOUN_TYPE_COUNT; i++) {
|
||||
const type = TypeUtils.getNounFromIndex(i)
|
||||
const path = getNounMetadataPath(type, id)
|
||||
|
||||
try {
|
||||
const metadata = await this.u.readObjectFromPath(path)
|
||||
if (metadata) {
|
||||
// Cache the type for next time
|
||||
const metadataType = (metadata.noun || 'thing') as NounType
|
||||
this.nounTypeCache.set(id, metadataType)
|
||||
return metadata
|
||||
}
|
||||
} catch (error) {
|
||||
// Not in this type, continue searching
|
||||
}
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete noun metadata (override to use type-aware paths)
|
||||
*/
|
||||
async deleteNounMetadata(id: string): Promise<void> {
|
||||
const cachedType = this.nounTypeCache.get(id)
|
||||
if (cachedType) {
|
||||
const path = getNounMetadataPath(cachedType, id)
|
||||
await this.u.deleteObjectFromPath(path)
|
||||
return
|
||||
}
|
||||
|
||||
// Search across all types
|
||||
for (let i = 0; i < NOUN_TYPE_COUNT; i++) {
|
||||
const type = TypeUtils.getNounFromIndex(i)
|
||||
const path = getNounMetadataPath(type, id)
|
||||
|
||||
try {
|
||||
await this.u.deleteObjectFromPath(path)
|
||||
return
|
||||
} catch (error) {
|
||||
// Not in this type, continue
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Save verb metadata (override to use type-aware paths)
|
||||
*
|
||||
* Note: Verb type comes from HNSWVerb.verb field, not metadata
|
||||
* We need to read the verb to get the type for path routing
|
||||
*/
|
||||
async saveVerbMetadata(id: string, metadata: VerbMetadata): Promise<void> {
|
||||
// Get verb type from cache or by reading the verb
|
||||
let type = this.verbTypeCache.get(id)
|
||||
|
||||
if (!type) {
|
||||
// Need to read the verb to get its type
|
||||
const verb = await this.getVerb_internal(id)
|
||||
if (verb) {
|
||||
type = verb.verb as VerbType
|
||||
this.verbTypeCache.set(id, type)
|
||||
} else {
|
||||
type = 'relatedTo' as VerbType
|
||||
}
|
||||
}
|
||||
|
||||
// Save to type-aware path
|
||||
const path = getVerbMetadataPath(type, id)
|
||||
await this.u.writeObjectToPath(path, metadata)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get verb metadata (override to use type-aware paths)
|
||||
*/
|
||||
async getVerbMetadata(id: string): Promise<VerbMetadata | null> {
|
||||
// Try cache first
|
||||
const cachedType = this.verbTypeCache.get(id)
|
||||
if (cachedType) {
|
||||
const path = getVerbMetadataPath(cachedType, id)
|
||||
return await this.u.readObjectFromPath(path)
|
||||
}
|
||||
|
||||
// Search across all types
|
||||
for (let i = 0; i < VERB_TYPE_COUNT; i++) {
|
||||
const type = TypeUtils.getVerbFromIndex(i)
|
||||
const path = getVerbMetadataPath(type, id)
|
||||
|
||||
try {
|
||||
const metadata = await this.u.readObjectFromPath(path)
|
||||
if (metadata) {
|
||||
// Cache the type for next time
|
||||
this.verbTypeCache.set(id, type)
|
||||
return metadata
|
||||
}
|
||||
} catch (error) {
|
||||
// Not in this type, continue
|
||||
}
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete verb metadata (override to use type-aware paths)
|
||||
*/
|
||||
async deleteVerbMetadata(id: string): Promise<void> {
|
||||
const cachedType = this.verbTypeCache.get(id)
|
||||
if (cachedType) {
|
||||
const path = getVerbMetadataPath(cachedType, id)
|
||||
await this.u.deleteObjectFromPath(path)
|
||||
return
|
||||
}
|
||||
|
||||
// Search across all types
|
||||
for (let i = 0; i < VERB_TYPE_COUNT; i++) {
|
||||
const type = TypeUtils.getVerbFromIndex(i)
|
||||
const path = getVerbMetadataPath(type, id)
|
||||
|
||||
try {
|
||||
await this.u.deleteObjectFromPath(path)
|
||||
return
|
||||
} catch (error) {
|
||||
// Not in this type, continue
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Write object to path (delegate to underlying storage)
|
||||
*/
|
||||
|
|
|
|||
|
|
@ -5,7 +5,16 @@
|
|||
|
||||
import { GraphAdjacencyIndex } from '../graph/graphAdjacencyIndex.js'
|
||||
|
||||
import { GraphVerb, HNSWNoun, HNSWVerb, StatisticsData } from '../coreTypes.js'
|
||||
import {
|
||||
GraphVerb,
|
||||
HNSWNoun,
|
||||
HNSWVerb,
|
||||
NounMetadata,
|
||||
VerbMetadata,
|
||||
HNSWNounWithMetadata,
|
||||
HNSWVerbWithMetadata,
|
||||
StatisticsData
|
||||
} from '../coreTypes.js'
|
||||
import { BaseStorageAdapter } from './adapters/baseStorageAdapter.js'
|
||||
import { validateNounType, validateVerbType } from '../utils/typeValidation.js'
|
||||
import { NounType, VerbType } from '../types/graphTypes.js'
|
||||
|
|
@ -167,49 +176,46 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
}
|
||||
|
||||
/**
|
||||
* Save a noun to storage
|
||||
* Save a noun to storage (v4.0.0: vector only, metadata saved separately)
|
||||
* @param noun Pure HNSW vector data (no metadata)
|
||||
*/
|
||||
public async saveNoun(noun: HNSWNoun): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
// Validate noun type before saving - storage boundary protection
|
||||
if (noun.metadata?.noun) {
|
||||
validateNounType(noun.metadata.noun)
|
||||
}
|
||||
|
||||
// Save both the HNSWNoun vector data and metadata separately (2-file system)
|
||||
try {
|
||||
// Save the lightweight HNSWNoun vector file first
|
||||
await this.saveNoun_internal(noun)
|
||||
|
||||
// Then save the metadata to separate file (if present)
|
||||
if (noun.metadata) {
|
||||
await this.saveNounMetadata(noun.id, noun.metadata)
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(`[ERROR] Failed to save noun ${noun.id}:`, error)
|
||||
|
||||
// Attempt cleanup - remove noun file if metadata failed
|
||||
try {
|
||||
const nounExists = await this.getNoun_internal(noun.id)
|
||||
if (nounExists) {
|
||||
console.log(`[CLEANUP] Attempting to remove orphaned noun file ${noun.id}`)
|
||||
await this.deleteNoun_internal(noun.id)
|
||||
}
|
||||
} catch (cleanupError) {
|
||||
console.error(`[ERROR] Failed to cleanup orphaned noun ${noun.id}:`, cleanupError)
|
||||
}
|
||||
|
||||
throw new Error(`Failed to save noun ${noun.id}: ${error instanceof Error ? error.message : String(error)}`)
|
||||
}
|
||||
// Save the HNSWNoun vector data only
|
||||
// Metadata must be saved separately via saveNounMetadata()
|
||||
await this.saveNoun_internal(noun)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a noun from storage
|
||||
* Get a noun from storage (v4.0.0: returns combined HNSWNounWithMetadata)
|
||||
* @param id Entity ID
|
||||
* @returns Combined vector + metadata or null
|
||||
*/
|
||||
public async getNoun(id: string): Promise<HNSWNoun | null> {
|
||||
public async getNoun(id: string): Promise<HNSWNounWithMetadata | null> {
|
||||
await this.ensureInitialized()
|
||||
return this.getNoun_internal(id)
|
||||
|
||||
// Load vector and metadata separately
|
||||
const vector = await this.getNoun_internal(id)
|
||||
if (!vector) {
|
||||
return null
|
||||
}
|
||||
|
||||
// Load metadata
|
||||
const metadata = await this.getNounMetadata(id)
|
||||
if (!metadata) {
|
||||
console.warn(`[Storage] Noun ${id} has vector but no metadata - this should not happen in v4.0.0`)
|
||||
return null
|
||||
}
|
||||
|
||||
// Combine into HNSWNounWithMetadata
|
||||
return {
|
||||
id: vector.id,
|
||||
vector: vector.vector,
|
||||
connections: vector.connections,
|
||||
level: vector.level,
|
||||
metadata
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -217,9 +223,25 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
* @param nounType The noun type to filter by
|
||||
* @returns Promise that resolves to an array of nouns of the specified noun type
|
||||
*/
|
||||
public async getNounsByNounType(nounType: string): Promise<HNSWNoun[]> {
|
||||
public async getNounsByNounType(nounType: string): Promise<HNSWNounWithMetadata[]> {
|
||||
await this.ensureInitialized()
|
||||
return this.getNounsByNounType_internal(nounType)
|
||||
|
||||
// Internal method returns HNSWNoun[], need to combine with metadata
|
||||
const nouns = await this.getNounsByNounType_internal(nounType)
|
||||
|
||||
// Combine each noun with its metadata
|
||||
const nounsWithMetadata: HNSWNounWithMetadata[] = []
|
||||
for (const noun of nouns) {
|
||||
const metadata = await this.getNounMetadata(noun.id)
|
||||
if (metadata) {
|
||||
nounsWithMetadata.push({
|
||||
...noun,
|
||||
metadata
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return nounsWithMetadata
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -241,103 +263,66 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
}
|
||||
|
||||
/**
|
||||
* Save a verb to storage
|
||||
* Save a verb to storage (v4.0.0: verb only, metadata saved separately)
|
||||
*
|
||||
* ARCHITECTURAL FIX (v3.50.1): HNSWVerb now includes verb/sourceId/targetId
|
||||
* These are core relational fields, not metadata. They're stored in the vector
|
||||
* file for fast access and to align with actual usage patterns.
|
||||
* @param verb Pure HNSW verb with core relational fields (verb, sourceId, targetId)
|
||||
*/
|
||||
public async saveVerb(verb: GraphVerb): Promise<void> {
|
||||
public async saveVerb(verb: HNSWVerb): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
// Validate verb type before saving - storage boundary protection
|
||||
if (verb.verb) {
|
||||
validateVerbType(verb.verb)
|
||||
}
|
||||
validateVerbType(verb.verb)
|
||||
|
||||
// Extract HNSWVerb with CORE relational fields included
|
||||
const hnswVerb: HNSWVerb = {
|
||||
id: verb.id,
|
||||
vector: verb.vector,
|
||||
connections: verb.connections || new Map(),
|
||||
|
||||
// CORE RELATIONAL DATA (v3.50.1+)
|
||||
verb: (verb.verb || verb.type || 'relatedTo') as VerbType,
|
||||
sourceId: verb.sourceId || verb.source || '',
|
||||
targetId: verb.targetId || verb.target || '',
|
||||
|
||||
// User metadata (if any)
|
||||
metadata: verb.metadata
|
||||
}
|
||||
|
||||
// Extract lightweight metadata for separate file (optional fields only)
|
||||
const metadata = {
|
||||
weight: verb.weight,
|
||||
data: verb.data,
|
||||
createdAt: verb.createdAt,
|
||||
updatedAt: verb.updatedAt,
|
||||
createdBy: verb.createdBy,
|
||||
|
||||
// Legacy aliases for backward compatibility
|
||||
source: verb.source || verb.sourceId,
|
||||
target: verb.target || verb.targetId,
|
||||
type: verb.type || verb.verb
|
||||
}
|
||||
|
||||
// Save both the HNSWVerb and metadata atomically
|
||||
try {
|
||||
console.log(`[DEBUG] Saving verb ${verb.id}: sourceId=${verb.sourceId}, targetId=${verb.targetId}`)
|
||||
|
||||
// Save the HNSWVerb first
|
||||
await this.saveVerb_internal(hnswVerb)
|
||||
console.log(`[DEBUG] Successfully saved HNSWVerb file for ${verb.id}`)
|
||||
|
||||
// Then save the metadata
|
||||
await this.saveVerbMetadata(verb.id, metadata)
|
||||
console.log(`[DEBUG] Successfully saved metadata file for ${verb.id}`)
|
||||
|
||||
} catch (error) {
|
||||
console.error(`[ERROR] Failed to save verb ${verb.id}:`, error)
|
||||
|
||||
// Attempt cleanup - remove verb file if metadata failed
|
||||
try {
|
||||
const verbExists = await this.getVerb_internal(verb.id)
|
||||
if (verbExists) {
|
||||
console.log(`[CLEANUP] Attempting to remove orphaned verb file ${verb.id}`)
|
||||
await this.deleteVerb_internal(verb.id)
|
||||
}
|
||||
} catch (cleanupError) {
|
||||
console.error(`[ERROR] Failed to cleanup orphaned verb ${verb.id}:`, cleanupError)
|
||||
}
|
||||
|
||||
throw new Error(`Failed to save verb ${verb.id}: ${error instanceof Error ? error.message : String(error)}`)
|
||||
}
|
||||
// Save the HNSWVerb vector and core fields only
|
||||
// Metadata must be saved separately via saveVerbMetadata()
|
||||
await this.saveVerb_internal(verb)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a verb from storage
|
||||
* Get a verb from storage (v4.0.0: returns combined HNSWVerbWithMetadata)
|
||||
* @param id Entity ID
|
||||
* @returns Combined verb + metadata or null
|
||||
*/
|
||||
public async getVerb(id: string): Promise<GraphVerb | null> {
|
||||
public async getVerb(id: string): Promise<HNSWVerbWithMetadata | null> {
|
||||
await this.ensureInitialized()
|
||||
const hnswVerb = await this.getVerb_internal(id)
|
||||
if (!hnswVerb) {
|
||||
|
||||
// Load verb vector and core fields
|
||||
const verb = await this.getVerb_internal(id)
|
||||
if (!verb) {
|
||||
return null
|
||||
}
|
||||
return this.convertHNSWVerbToGraphVerb(hnswVerb)
|
||||
|
||||
// Load metadata
|
||||
const metadata = await this.getVerbMetadata(id)
|
||||
if (!metadata) {
|
||||
console.warn(`[Storage] Verb ${id} has vector but no metadata - this should not happen in v4.0.0`)
|
||||
return null
|
||||
}
|
||||
|
||||
// Combine into HNSWVerbWithMetadata
|
||||
return {
|
||||
id: verb.id,
|
||||
vector: verb.vector,
|
||||
connections: verb.connections,
|
||||
verb: verb.verb,
|
||||
sourceId: verb.sourceId,
|
||||
targetId: verb.targetId,
|
||||
metadata
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert HNSWVerb to GraphVerb by combining with metadata
|
||||
* DEPRECATED: For backward compatibility only. Use getVerb() which returns HNSWVerbWithMetadata.
|
||||
*
|
||||
* ARCHITECTURAL FIX (v3.50.1): Core fields (verb/sourceId/targetId) are now in HNSWVerb
|
||||
* Only optional fields (weight, timestamps, etc.) come from metadata file
|
||||
* @deprecated Use getVerb() instead which returns HNSWVerbWithMetadata
|
||||
*/
|
||||
protected async convertHNSWVerbToGraphVerb(hnswVerb: HNSWVerb): Promise<GraphVerb | null> {
|
||||
try {
|
||||
// Metadata file is now optional - contains only weight, timestamps, etc.
|
||||
// Load metadata
|
||||
const metadata = await this.getVerbMetadata(hnswVerb.id)
|
||||
|
||||
// Create default timestamp if not present
|
||||
// Create default timestamp in Firestore format
|
||||
const defaultTimestamp = {
|
||||
seconds: Math.floor(Date.now() / 1000),
|
||||
nanoseconds: (Date.now() % 1000) * 1000000
|
||||
|
|
@ -349,11 +334,23 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
version: '1.0'
|
||||
}
|
||||
|
||||
// Convert flexible timestamp to Firestore format for GraphVerb
|
||||
const normalizeTimestamp = (ts: any) => {
|
||||
if (!ts) return defaultTimestamp
|
||||
if (typeof ts === 'number') {
|
||||
return {
|
||||
seconds: Math.floor(ts / 1000),
|
||||
nanoseconds: (ts % 1000) * 1000000
|
||||
}
|
||||
}
|
||||
return ts
|
||||
}
|
||||
|
||||
return {
|
||||
id: hnswVerb.id,
|
||||
vector: hnswVerb.vector,
|
||||
|
||||
// CORE FIELDS from HNSWVerb (v3.50.1+)
|
||||
// CORE FIELDS from HNSWVerb
|
||||
verb: hnswVerb.verb,
|
||||
sourceId: hnswVerb.sourceId,
|
||||
targetId: hnswVerb.targetId,
|
||||
|
|
@ -365,11 +362,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
|
||||
// Optional fields from metadata file
|
||||
weight: metadata?.weight || 1.0,
|
||||
metadata: hnswVerb.metadata || {},
|
||||
createdAt: metadata?.createdAt || defaultTimestamp,
|
||||
updatedAt: metadata?.updatedAt || defaultTimestamp,
|
||||
metadata: metadata as any || {},
|
||||
createdAt: normalizeTimestamp(metadata?.createdAt),
|
||||
updatedAt: normalizeTimestamp(metadata?.updatedAt),
|
||||
createdBy: metadata?.createdBy || defaultCreatedBy,
|
||||
data: metadata?.data,
|
||||
data: metadata?.data as Record<string, any> | undefined,
|
||||
embedding: hnswVerb.vector
|
||||
}
|
||||
} catch (error) {
|
||||
|
|
@ -390,25 +387,15 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
pagination: { limit: Number.MAX_SAFE_INTEGER }
|
||||
})
|
||||
|
||||
// Convert GraphVerbs back to HNSWVerbs for internal use
|
||||
// ARCHITECTURAL FIX (v3.50.1): Include core relational fields
|
||||
const hnswVerbs: HNSWVerb[] = []
|
||||
for (const graphVerb of result.items) {
|
||||
const hnswVerb: HNSWVerb = {
|
||||
id: graphVerb.id,
|
||||
vector: graphVerb.vector,
|
||||
connections: new Map(),
|
||||
|
||||
// CORE RELATIONAL DATA
|
||||
verb: (graphVerb.verb || graphVerb.type || 'relatedTo') as VerbType,
|
||||
sourceId: graphVerb.sourceId || graphVerb.source || '',
|
||||
targetId: graphVerb.targetId || graphVerb.target || '',
|
||||
|
||||
// User metadata
|
||||
metadata: graphVerb.metadata
|
||||
}
|
||||
hnswVerbs.push(hnswVerb)
|
||||
}
|
||||
// v4.0.0: Convert HNSWVerbWithMetadata to HNSWVerb (strip metadata)
|
||||
const hnswVerbs: HNSWVerb[] = result.items.map(verbWithMetadata => ({
|
||||
id: verbWithMetadata.id,
|
||||
vector: verbWithMetadata.vector,
|
||||
connections: verbWithMetadata.connections,
|
||||
verb: verbWithMetadata.verb,
|
||||
sourceId: verbWithMetadata.sourceId,
|
||||
targetId: verbWithMetadata.targetId
|
||||
}))
|
||||
|
||||
return hnswVerbs
|
||||
}
|
||||
|
|
@ -416,7 +403,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
/**
|
||||
* Get verbs by source
|
||||
*/
|
||||
public async getVerbsBySource(sourceId: string): Promise<GraphVerb[]> {
|
||||
public async getVerbsBySource(sourceId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
// CRITICAL: Fetch ALL verbs for this source, not just first page
|
||||
|
|
@ -431,7 +418,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
/**
|
||||
* Get verbs by target
|
||||
*/
|
||||
public async getVerbsByTarget(targetId: string): Promise<GraphVerb[]> {
|
||||
public async getVerbsByTarget(targetId: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
// CRITICAL: Fetch ALL verbs for this target, not just first page
|
||||
|
|
@ -446,7 +433,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
/**
|
||||
* Get verbs by type
|
||||
*/
|
||||
public async getVerbsByType(type: string): Promise<GraphVerb[]> {
|
||||
public async getVerbsByType(type: string): Promise<HNSWVerbWithMetadata[]> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
// Fetch ALL verbs of this type (no pagination limit)
|
||||
|
|
@ -489,7 +476,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
}): Promise<{
|
||||
items: HNSWNoun[]
|
||||
items: HNSWNounWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -514,8 +501,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
? options.filter.nounType[0]
|
||||
: options.filter.nounType
|
||||
|
||||
// Get nouns by type directly
|
||||
const nounsByType = await this.getNounsByNounType_internal(nounType)
|
||||
// Get nouns by type directly (already combines with metadata)
|
||||
const nounsByType = await this.getNounsByNounType(nounType)
|
||||
|
||||
// Apply pagination
|
||||
const paginatedNouns = nounsByType.slice(offset, offset + limit)
|
||||
|
|
@ -629,7 +616,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
metadata?: Record<string, any>
|
||||
}
|
||||
}): Promise<{
|
||||
items: GraphVerb[]
|
||||
items: HNSWVerbWithMetadata[]
|
||||
totalCount?: number
|
||||
hasMore: boolean
|
||||
nextCursor?: string
|
||||
|
|
@ -906,53 +893,51 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
protected abstract listObjectsUnderPath(prefix: string): Promise<string[]>
|
||||
|
||||
/**
|
||||
* Save metadata to storage
|
||||
* Save metadata to storage (v4.0.0: now typed)
|
||||
* Routes to correct location (system or entity) based on key format
|
||||
*/
|
||||
public async saveMetadata(id: string, metadata: any): Promise<void> {
|
||||
public async saveMetadata(id: string, metadata: NounMetadata): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
const keyInfo = this.analyzeKey(id, 'system')
|
||||
return this.writeObjectToPath(keyInfo.fullPath, metadata)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get metadata from storage
|
||||
* Get metadata from storage (v4.0.0: now typed)
|
||||
* Routes to correct location (system or entity) based on key format
|
||||
*/
|
||||
public async getMetadata(id: string): Promise<any | null> {
|
||||
public async getMetadata(id: string): Promise<NounMetadata | null> {
|
||||
await this.ensureInitialized()
|
||||
const keyInfo = this.analyzeKey(id, 'system')
|
||||
return this.readObjectFromPath(keyInfo.fullPath)
|
||||
}
|
||||
|
||||
/**
|
||||
* Save noun metadata to storage
|
||||
* Save noun metadata to storage (v4.0.0: now typed)
|
||||
* Routes to correct sharded location based on UUID
|
||||
*/
|
||||
public async saveNounMetadata(id: string, metadata: any): Promise<void> {
|
||||
public async saveNounMetadata(id: string, metadata: NounMetadata): Promise<void> {
|
||||
// Validate noun type in metadata - storage boundary protection
|
||||
if (metadata?.noun) {
|
||||
validateNounType(metadata.noun)
|
||||
}
|
||||
validateNounType(metadata.noun)
|
||||
return this.saveNounMetadata_internal(id, metadata)
|
||||
}
|
||||
|
||||
/**
|
||||
* Internal method for saving noun metadata
|
||||
* Internal method for saving noun metadata (v4.0.0: now typed)
|
||||
* Uses routing logic to handle both UUIDs (sharded) and system keys (unsharded)
|
||||
* @protected
|
||||
*/
|
||||
protected async saveNounMetadata_internal(id: string, metadata: any): Promise<void> {
|
||||
protected async saveNounMetadata_internal(id: string, metadata: NounMetadata): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
const keyInfo = this.analyzeKey(id, 'noun-metadata')
|
||||
return this.writeObjectToPath(keyInfo.fullPath, metadata)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get noun metadata from storage
|
||||
* Get noun metadata from storage (v4.0.0: now typed)
|
||||
* Uses routing logic to handle both UUIDs (sharded) and system keys (unsharded)
|
||||
*/
|
||||
public async getNounMetadata(id: string): Promise<any | null> {
|
||||
public async getNounMetadata(id: string): Promise<NounMetadata | null> {
|
||||
await this.ensureInitialized()
|
||||
const keyInfo = this.analyzeKey(id, 'noun-metadata')
|
||||
return this.readObjectFromPath(keyInfo.fullPath)
|
||||
|
|
@ -969,33 +954,30 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
}
|
||||
|
||||
/**
|
||||
* Save verb metadata to storage
|
||||
* Save verb metadata to storage (v4.0.0: now typed)
|
||||
* Routes to correct sharded location based on UUID
|
||||
*/
|
||||
public async saveVerbMetadata(id: string, metadata: any): Promise<void> {
|
||||
// Validate verb type in metadata - storage boundary protection
|
||||
if (metadata?.verb) {
|
||||
validateVerbType(metadata.verb)
|
||||
}
|
||||
public async saveVerbMetadata(id: string, metadata: VerbMetadata): Promise<void> {
|
||||
// Note: verb type is in HNSWVerb, not metadata
|
||||
return this.saveVerbMetadata_internal(id, metadata)
|
||||
}
|
||||
|
||||
/**
|
||||
* Internal method for saving verb metadata
|
||||
* Internal method for saving verb metadata (v4.0.0: now typed)
|
||||
* Uses routing logic to handle both UUIDs (sharded) and system keys (unsharded)
|
||||
* @protected
|
||||
*/
|
||||
protected async saveVerbMetadata_internal(id: string, metadata: any): Promise<void> {
|
||||
protected async saveVerbMetadata_internal(id: string, metadata: VerbMetadata): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
const keyInfo = this.analyzeKey(id, 'verb-metadata')
|
||||
return this.writeObjectToPath(keyInfo.fullPath, metadata)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get verb metadata from storage
|
||||
* Get verb metadata from storage (v4.0.0: now typed)
|
||||
* Uses routing logic to handle both UUIDs (sharded) and system keys (unsharded)
|
||||
*/
|
||||
public async getVerbMetadata(id: string): Promise<any | null> {
|
||||
public async getVerbMetadata(id: string): Promise<VerbMetadata | null> {
|
||||
await this.ensureInitialized()
|
||||
const keyInfo = this.analyzeKey(id, 'verb-metadata')
|
||||
return this.readObjectFromPath(keyInfo.fullPath)
|
||||
|
|
@ -1055,7 +1037,7 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
*/
|
||||
protected abstract getVerbsBySource_internal(
|
||||
sourceId: string
|
||||
): Promise<GraphVerb[]>
|
||||
): Promise<HNSWVerbWithMetadata[]>
|
||||
|
||||
/**
|
||||
* Get verbs by target
|
||||
|
|
@ -1063,13 +1045,13 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
*/
|
||||
protected abstract getVerbsByTarget_internal(
|
||||
targetId: string
|
||||
): Promise<GraphVerb[]>
|
||||
): Promise<HNSWVerbWithMetadata[]>
|
||||
|
||||
/**
|
||||
* Get verbs by type
|
||||
* This method should be implemented by each specific adapter
|
||||
*/
|
||||
protected abstract getVerbsByType_internal(type: string): Promise<GraphVerb[]>
|
||||
protected abstract getVerbsByType_internal(type: string): Promise<HNSWVerbWithMetadata[]>
|
||||
|
||||
/**
|
||||
* Delete a verb from storage
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import { OPFSStorage } from './adapters/opfsStorage.js'
|
|||
import { S3CompatibleStorage } from './adapters/s3CompatibleStorage.js'
|
||||
import { R2Storage } from './adapters/r2Storage.js'
|
||||
import { GcsStorage } from './adapters/gcsStorage.js'
|
||||
import { AzureBlobStorage } from './adapters/azureBlobStorage.js'
|
||||
import { TypeAwareStorageAdapter } from './adapters/typeAwareStorageAdapter.js'
|
||||
// FileSystemStorage is dynamically imported to avoid issues in browser environments
|
||||
import { isBrowser } from '../utils/environment.js'
|
||||
|
|
@ -28,9 +29,10 @@ export interface StorageOptions {
|
|||
* - 'r2': Use Cloudflare R2 storage
|
||||
* - 'gcs': Use Google Cloud Storage (S3-compatible with HMAC keys)
|
||||
* - 'gcs-native': Use Google Cloud Storage (native SDK with ADC)
|
||||
* - 'azure': Use Azure Blob Storage (native SDK with Managed Identity)
|
||||
* - 'type-aware': Use type-first storage adapter (wraps another adapter)
|
||||
*/
|
||||
type?: 'auto' | 'memory' | 'opfs' | 'filesystem' | 's3' | 'r2' | 'gcs' | 'gcs-native' | 'type-aware'
|
||||
type?: 'auto' | 'memory' | 'opfs' | 'filesystem' | 's3' | 'r2' | 'gcs' | 'gcs-native' | 'azure' | 'type-aware'
|
||||
|
||||
/**
|
||||
* Force the use of memory storage even if other storage types are available
|
||||
|
|
@ -177,6 +179,36 @@ export interface StorageOptions {
|
|||
secretAccessKey?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Configuration for Azure Blob Storage (native SDK with Managed Identity)
|
||||
*/
|
||||
azureStorage?: {
|
||||
/**
|
||||
* Azure container name
|
||||
*/
|
||||
containerName: string
|
||||
|
||||
/**
|
||||
* Azure Storage account name (for Managed Identity or SAS)
|
||||
*/
|
||||
accountName?: string
|
||||
|
||||
/**
|
||||
* Azure Storage account key (optional, uses Managed Identity if not provided)
|
||||
*/
|
||||
accountKey?: string
|
||||
|
||||
/**
|
||||
* Azure connection string (highest priority if provided)
|
||||
*/
|
||||
connectionString?: string
|
||||
|
||||
/**
|
||||
* SAS token (optional, alternative to account key)
|
||||
*/
|
||||
sasToken?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Configuration for Type-Aware Storage (type-first architecture)
|
||||
* Wraps another storage adapter and adds type-first routing
|
||||
|
|
@ -473,6 +505,24 @@ export async function createStorage(
|
|||
return new MemoryStorage()
|
||||
}
|
||||
|
||||
case 'azure':
|
||||
if (options.azureStorage) {
|
||||
console.log('Using Azure Blob Storage (native SDK)')
|
||||
return new AzureBlobStorage({
|
||||
containerName: options.azureStorage.containerName,
|
||||
accountName: options.azureStorage.accountName,
|
||||
accountKey: options.azureStorage.accountKey,
|
||||
connectionString: options.azureStorage.connectionString,
|
||||
sasToken: options.azureStorage.sasToken,
|
||||
cacheConfig: options.cacheConfig
|
||||
})
|
||||
} else {
|
||||
console.warn(
|
||||
'Azure storage configuration is missing, falling back to memory storage'
|
||||
)
|
||||
return new MemoryStorage()
|
||||
}
|
||||
|
||||
case 'type-aware': {
|
||||
console.log('Using Type-Aware Storage (type-first architecture)')
|
||||
|
||||
|
|
@ -571,6 +621,19 @@ export async function createStorage(
|
|||
})
|
||||
}
|
||||
|
||||
// If Azure storage is specified, use it
|
||||
if (options.azureStorage) {
|
||||
console.log('Using Azure Blob Storage (native SDK)')
|
||||
return new AzureBlobStorage({
|
||||
containerName: options.azureStorage.containerName,
|
||||
accountName: options.azureStorage.accountName,
|
||||
accountKey: options.azureStorage.accountKey,
|
||||
connectionString: options.azureStorage.connectionString,
|
||||
sasToken: options.azureStorage.sasToken,
|
||||
cacheConfig: options.cacheConfig
|
||||
})
|
||||
}
|
||||
|
||||
// Auto-detect the best storage adapter based on the environment
|
||||
// First, check if we're in Node.js (prioritize for test environments)
|
||||
if (!isBrowser()) {
|
||||
|
|
@ -632,6 +695,7 @@ export {
|
|||
S3CompatibleStorage,
|
||||
R2Storage,
|
||||
GcsStorage,
|
||||
AzureBlobStorage,
|
||||
TypeAwareStorageAdapter
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue