diff --git a/VFS_DEBUG_INSTRUCTIONS.md b/VFS_DEBUG_INSTRUCTIONS.md deleted file mode 100644 index 92351c44..00000000 --- a/VFS_DEBUG_INSTRUCTIONS.md +++ /dev/null @@ -1,188 +0,0 @@ -# šŸ” VFS Root Debugging Instructions - -## Problem Summary - -Despite 7 attempted fixes (v4.5.1 through v4.7.1), vfs.readdir('/') still returns empty even though: -- āœ… 600 document entities exist -- āœ… 608 Contains relationships exist -- āœ… 9 collection entities (directories) exist - -## Root Cause Hypothesis - -The VFS instance is likely using a **different root entity ID** than the actual root directory where files were imported. This would explain why: -- Import succeeds (creates files under the REAL root) -- getRelations() succeeds (relationships exist in database) -- But readdir('/') returns empty (VFS querying the WRONG root) - -## Debug Script - -I've created `debug-vfs-root.js` which will: -1. Show the root entity ID that VFS is using -2. List ALL collection entities (directories) in the database -3. For each directory, count outgoing Contains relationships -4. Identify which directory is the VFS root and whether it has children - -## How to Run - -```bash -# Make sure you're using v4.7.1 -$ cd /path/to/brainy/project - -# Copy the debug script to your Brainy installation -$ cp /media/dpsifr/storage/home/Projects/brainy/debug-vfs-root.js . - -# Run it with your FileSystemStorage database -$ node debug-vfs-root.js -``` - -## Expected Output - -The script will show: -``` -šŸ” VFS Root Debugging Script -============================================================ -āœ… Brainy initialized - -āœ… VFS instance created - -šŸ“ VFS Root Entity ID: abc123-xyz... - -āœ… Root entity EXISTS in database: - Type: collection - Path: / - VFS Type: directory - Created: 2025-10-27T16:45:00.000Z - -šŸ” Finding ALL collection entities (directories)... - Found 9 collection entities: - - šŸ‘‘ ID: abc123-xyz... - Path: / - VFS Type: directory - Created: 2025-10-27T16:45:00.000Z - āœ… THIS IS THE VFS ROOT - - ID: def456-uvw... - Path: /Characters - VFS Type: directory - Created: 2025-10-27T16:46:00.000Z - - [... 7 more directories ...] - -šŸ” Checking Contains relationships FROM each collection... - - Collection abc123-xyz...: - Path: / - Outgoing Contains: 8 - āœ… THIS IS THE VFS ROOT - should return 8 items for readdir('/') - - Collection def456-uvw...: - Path: /Characters - Outgoing Contains: 127 - - [... more directories ...] - -šŸ” Counting ALL Contains relationships in database... - Total Contains relationships: 608 - -šŸ“‹ Sample Contains relationships (first 10): - 1. abc123-xyz... -> def456-uvw... - From: / (directory) - To: /Characters (directory) - - 2. def456-uvw... -> ghi789-rst... - From: /Characters (directory) - To: /Characters/entity-001.json (file) - - [... more samples ...] - -============================================================ -šŸ Debug Complete -``` - -## What to Look For - -### āœ… Good Case (VFS Working) -``` -šŸ“ VFS Root Entity ID: abc123-xyz... -āœ… Root entity EXISTS in database - Collection abc123-xyz...: - Path: / - Outgoing Contains: 8 šŸ‘ˆ NON-ZERO! - āœ… THIS IS THE VFS ROOT -``` - -### āŒ Bad Case #1: VFS Using Wrong Root -``` -šŸ“ VFS Root Entity ID: xxx000-yyy... -āœ… Root entity EXISTS in database - Collection xxx000-yyy...: - Path: / - Outgoing Contains: 0 šŸ‘ˆ ZERO! This is the bug! - āœ… THIS IS THE VFS ROOT - - Collection abc123-zzz...: - Path: / - Outgoing Contains: 8 šŸ‘ˆ Files are under THIS root instead! -``` - -### āŒ Bad Case #2: VFS Root Doesn't Exist -``` -šŸ“ VFS Root Entity ID: xxx000-yyy... -āŒ Root entity DOES NOT EXIST in database! - This is the bug! VFS is using a root ID that doesn't exist. -``` - -### āŒ Bad Case #3: Root Has No Children -``` -šŸ“ VFS Root Entity ID: abc123-xyz... -āœ… Root entity EXISTS in database - Collection abc123-xyz...: - Path: / - Outgoing Contains: 0 šŸ‘ˆ ZERO! Files aren't under root! - āœ… THIS IS THE VFS ROOT - - [All other collections have Contains relationships, but NOT from root] -``` - -## What This Tells Us - -Based on the output, we can determine: - -1. **If "Outgoing Contains: 0" for VFS root**: The VFS is querying the correct root, but that root has no children. This means files were created under a DIFFERENT root or weren't properly linked. - -2. **If multiple "/" paths exist**: There are duplicate root entities. VFS is using one root, but files are under a different root. - -3. **If "Root entity DOES NOT EXIST"**: VFS is using a stale/wrong entity ID that was deleted or never existed. - -4. **If "Outgoing Contains: 8"**: VFS root HAS children! This means my v4.7.1 fix has another bug - the optimization isn't being triggered or has a logic error. - -## Next Steps - -Please run this script and send me the full output. Based on what we see, I'll know exactly how to fix the real bug! - -## Architecture Notes - -The VFS query flow is: -``` -vfs.readdir('/') - -> PathResolver.resolve('/') - -> returns this.rootEntityId - -> PathResolver.getChildren(rootEntityId) - -> brain.getRelations({ from: rootEntityId, type: VerbType.Contains }) - -> brainy.ts builds filter - -> storage.getVerbs({ filter: { sourceId: rootEntityId, verbType } }) - -> MY v4.7.1 FIX: Check if sourceId + verbType filters exist - -> If YES: Call getVerbsBySource_internal(rootEntityId) - -> Get verbs from graph adjacency index - -> Filter by verbType - -> Return Contains relationships -``` - -If the debug shows "Outgoing Contains: 0" for the VFS root, then either: -- A. getVerbsBySource_internal() is broken -- B. The graph adjacency index isn't being populated -- C. My v4.7.1 optimization has a bug and isn't being called -- D. The rootEntityId is wrong - -This script will tell us which one! diff --git a/debug-vfs-root.js b/debug-vfs-root.js deleted file mode 100644 index 326c35ee..00000000 --- a/debug-vfs-root.js +++ /dev/null @@ -1,124 +0,0 @@ -/** - * VFS Root Debugging Script - * - * This script debugs why vfs.readdir('/') returns empty despite - * 608 Contains relationships existing in the database. - */ - -import { Brainy } from './dist/brainy.js' -import { VerbType, NounType } from './dist/types/graphTypes.js' - -async function debugVFSRoot() { - console.log('\nšŸ” VFS Root Debugging Script\n') - console.log('=' .repeat(60)) - - // Initialize Brainy with FileSystemStorage - const brain = new Brainy({ - storage: { - type: 'filesystem', - config: { - path: './brainy-data' - } - } - }) - - await brain.ready() - console.log('āœ… Brainy initialized\n') - - // Get VFS instance - const vfs = brain.vfs() - console.log('āœ… VFS instance created\n') - - // Get the root entity ID that VFS is using - const vfsRootId = (vfs as any).rootEntityId - console.log(`šŸ“ VFS Root Entity ID: ${vfsRootId}\n`) - - // Try to get the root entity - const rootEntity = await brain.get(vfsRootId) - if (rootEntity) { - console.log('āœ… Root entity EXISTS in database:') - console.log(` Type: ${rootEntity.type}`) - console.log(` Path: ${rootEntity.metadata?.path}`) - console.log(` VFS Type: ${rootEntity.metadata?.vfsType}`) - console.log(` Created: ${new Date(rootEntity.createdAt || 0).toISOString()}`) - console.log() - } else { - console.log('āŒ Root entity DOES NOT EXIST in database!') - console.log(' This is the bug! VFS is using a root ID that doesn\'t exist.\n') - } - - // Find ALL collection entities (directories) - console.log('šŸ” Finding ALL collection entities (directories)...') - const allCollections = await brain.find({ - type: NounType.Collection, - limit: 100 - }) - console.log(` Found ${allCollections.length} collection entities:\n`) - - for (const coll of allCollections) { - const isRoot = coll.metadata?.path === '/' && coll.metadata?.vfsType === 'directory' - console.log(` ${isRoot ? 'šŸ‘‘' : ' '} ID: ${coll.id}`) - console.log(` Path: ${coll.metadata?.path}`) - console.log(` VFS Type: ${coll.metadata?.vfsType}`) - console.log(` Created: ${new Date(coll.entity?.createdAt || 0).toISOString()}`) - - // Check if this matches VFS root - if (coll.id === vfsRootId) { - console.log(` āœ… THIS IS THE VFS ROOT`) - } - console.log() - } - - // For each potential root, count outgoing Contains relationships - console.log('šŸ” Checking Contains relationships FROM each collection...\n') - - for (const coll of allCollections) { - const relations = await brain.getRelations({ - from: coll.id, - type: VerbType.Contains - }) - - console.log(` Collection ${coll.id}:`) - console.log(` Path: ${coll.metadata?.path}`) - console.log(` Outgoing Contains: ${relations.length}`) - - if (coll.id === vfsRootId) { - console.log(` āœ… THIS IS THE VFS ROOT - should return ${relations.length} items for readdir('/')`) - - if (relations.length === 0) { - console.log(` āŒ BUT IT HAS ZERO CHILDREN! This is why readdir() returns empty!`) - } - } - console.log() - } - - // Count total Contains relationships in database - console.log('šŸ” Counting ALL Contains relationships in database...') - const allContains = await brain.getRelations({ - type: VerbType.Contains, - limit: 1000 - }) - console.log(` Total Contains relationships: ${allContains.length}\n`) - - // Sample some Contains relationships - console.log('šŸ“‹ Sample Contains relationships (first 10):') - for (let i = 0; i < Math.min(10, allContains.length); i++) { - const rel = allContains[i] - const fromEntity = await brain.get(rel.from) - const toEntity = await brain.get(rel.to) - - console.log(` ${i + 1}. ${rel.from} -> ${rel.to}`) - console.log(` From: ${fromEntity?.metadata?.path || 'unknown'} (${fromEntity?.metadata?.vfsType || 'unknown'})`) - console.log(` To: ${toEntity?.metadata?.path || 'unknown'} (${toEntity?.metadata?.vfsType || 'unknown'})`) - } - - console.log('\n' + '='.repeat(60)) - console.log('šŸ Debug Complete\n') - - process.exit(0) -} - -debugVFSRoot().catch(err => { - console.error('āŒ Error:', err) - process.exit(1) -}) diff --git a/src/storage/adapters/azureBlobStorage.ts b/src/storage/adapters/azureBlobStorage.ts index 6c5a708b..19f9ec2f 100644 --- a/src/storage/adapters/azureBlobStorage.ts +++ b/src/storage/adapters/azureBlobStorage.ts @@ -1642,14 +1642,39 @@ export class AzureBlobStorage extends BaseStorage { await this.ensureInitialized() try { + // CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata const shard = getShardIdFromUuid(nounId) const key = `entities/nouns/hnsw/${shard}/${nounId}.json` - const blockBlobClient = this.containerClient!.getBlockBlobClient(key) - const content = JSON.stringify(hnswData, null, 2) - await blockBlobClient.upload(content, content.length, { - blobHTTPHeaders: { blobContentType: 'application/json' } - }) + + try { + // Read existing node data + const downloadResponse = await blockBlobClient.download(0) + const existingData = await this.streamToBuffer(downloadResponse.readableStreamBody!) + const existingNode = JSON.parse(existingData.toString()) + + // Preserve id and vector, update only HNSW graph metadata + const updatedNode = { + ...existingNode, + level: hnswData.level, + connections: hnswData.connections + } + + const content = JSON.stringify(updatedNode, null, 2) + await blockBlobClient.upload(content, content.length, { + blobHTTPHeaders: { blobContentType: 'application/json' } + }) + } catch (error: any) { + // If node doesn't exist yet, create it with just HNSW data + if (error.statusCode === 404 || error.code === 'BlobNotFound') { + const content = JSON.stringify(hnswData, null, 2) + await blockBlobClient.upload(content, content.length, { + blobHTTPHeaders: { blobContentType: 'application/json' } + }) + } else { + throw error + } + } } catch (error) { this.logger.error(`Failed to save HNSW data for ${nounId}:`, error) throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`) diff --git a/src/storage/adapters/fileSystemStorage.ts b/src/storage/adapters/fileSystemStorage.ts index a3b0684c..88f0f906 100644 --- a/src/storage/adapters/fileSystemStorage.ts +++ b/src/storage/adapters/fileSystemStorage.ts @@ -2579,13 +2579,36 @@ export class FileSystemStorage extends BaseStorage { }): Promise { await this.ensureInitialized() - // Use sharded path for HNSW data - const shard = nounId.substring(0, 2).toLowerCase() - const hnswDir = path.join(this.rootDir, 'entities', 'nouns', 'hnsw', shard) - await this.ensureDirectoryExists(hnswDir) + // CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata + // Previous implementation overwrote the entire file, destroying vector data + // Now we READ the existing node, UPDATE only connections/level, then WRITE back the complete node - const filePath = path.join(hnswDir, `${nounId}.json`) - await fs.promises.writeFile(filePath, JSON.stringify(hnswData, null, 2)) + const filePath = this.getNodePath(nounId) + + try { + // Read existing node data + const existingData = await fs.promises.readFile(filePath, 'utf-8') + const existingNode = JSON.parse(existingData) + + // Preserve id and vector, update only HNSW graph metadata + const updatedNode = { + ...existingNode, // Preserve all existing fields (id, vector, etc.) + level: hnswData.level, + connections: hnswData.connections + } + + // Write back the COMPLETE node with updated HNSW data + await fs.promises.writeFile(filePath, JSON.stringify(updatedNode, null, 2)) + } catch (error: any) { + // If node doesn't exist yet, create it with just HNSW data + // This should only happen during initial node creation + if (error.code === 'ENOENT') { + await this.ensureDirectoryExists(path.dirname(filePath)) + await fs.promises.writeFile(filePath, JSON.stringify(hnswData, null, 2)) + } else { + throw error + } + } } /** diff --git a/src/storage/adapters/gcsStorage.ts b/src/storage/adapters/gcsStorage.ts index e8b59d15..273d726f 100644 --- a/src/storage/adapters/gcsStorage.ts +++ b/src/storage/adapters/gcsStorage.ts @@ -1867,15 +1867,43 @@ export class GcsStorage extends BaseStorage { await this.ensureInitialized() try { - // Use sharded path for HNSW data + // CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata + // Previous implementation overwrote the entire file, destroying vector data + // Now we READ the existing node, UPDATE only connections/level, then WRITE back the complete node + const shard = getShardIdFromUuid(nounId) const key = `entities/nouns/hnsw/${shard}/${nounId}.json` - const file = this.bucket!.file(key) - await file.save(JSON.stringify(hnswData, null, 2), { - contentType: 'application/json', - resumable: false - }) + + try { + // Read existing node data + const [existingData] = await file.download() + const existingNode = JSON.parse(existingData.toString()) + + // Preserve id and vector, update only HNSW graph metadata + const updatedNode = { + ...existingNode, // Preserve all existing fields (id, vector, etc.) + level: hnswData.level, + connections: hnswData.connections + } + + // Write back the COMPLETE node with updated HNSW data + await file.save(JSON.stringify(updatedNode, null, 2), { + contentType: 'application/json', + resumable: false + }) + } catch (error: any) { + // If node doesn't exist yet, create it with just HNSW data + // This should only happen during initial node creation + if (error.code === 404) { + await file.save(JSON.stringify(hnswData, null, 2), { + contentType: 'application/json', + resumable: false + }) + } else { + throw error + } + } } catch (error) { this.logger.error(`Failed to save HNSW data for ${nounId}:`, error) throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`) diff --git a/src/storage/adapters/opfsStorage.ts b/src/storage/adapters/opfsStorage.ts index c834317a..6adbe63f 100644 --- a/src/storage/adapters/opfsStorage.ts +++ b/src/storage/adapters/opfsStorage.ts @@ -2016,20 +2016,34 @@ export class OPFSStorage extends BaseStorage { await this.ensureInitialized() try { - // Get or create the hnsw directory under nouns + // CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata const hnswDir = await this.nounsDir!.getDirectoryHandle('hnsw', { create: true }) - - // Use sharded path for HNSW data const shard = getShardIdFromUuid(nounId) const shardDir = await hnswDir.getDirectoryHandle(shard, { create: true }) - - // Create or get the file in the shard directory const fileHandle = await shardDir.getFileHandle(`${nounId}.json`, { create: true }) - // Write the HNSW data to the file - const writable = await fileHandle.createWritable() - await writable.write(JSON.stringify(hnswData, null, 2)) - await writable.close() + try { + // Read existing node data + const file = await fileHandle.getFile() + const existingData = await file.text() + const existingNode = JSON.parse(existingData) + + // Preserve id and vector, update only HNSW graph metadata + const updatedNode = { + ...existingNode, + level: hnswData.level, + connections: hnswData.connections + } + + const writable = await fileHandle.createWritable() + await writable.write(JSON.stringify(updatedNode, null, 2)) + await writable.close() + } catch (error) { + // If node doesn't exist or read fails, create with just HNSW data + const writable = await fileHandle.createWritable() + await writable.write(JSON.stringify(hnswData, null, 2)) + await writable.close() + } } catch (error) { console.error(`Failed to save HNSW data for ${nounId}:`, error) throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`) diff --git a/src/storage/adapters/r2Storage.ts b/src/storage/adapters/r2Storage.ts index 8e6a94c3..181d1d32 100644 --- a/src/storage/adapters/r2Storage.ts +++ b/src/storage/adapters/r2Storage.ts @@ -1002,10 +1002,30 @@ export class R2Storage extends BaseStorage { }): Promise { await this.ensureInitialized() + // CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata const shard = getShardIdFromUuid(nounId) const key = `entities/nouns/hnsw/${shard}/${nounId}.json` - await this.writeObjectToPath(key, hnswData) + try { + // Read existing node data + const existingNode = await this.readObjectFromPath(key) + + if (existingNode) { + // Preserve id and vector, update only HNSW graph metadata + const updatedNode = { + ...existingNode, + level: hnswData.level, + connections: hnswData.connections + } + await this.writeObjectToPath(key, updatedNode) + } else { + // Node doesn't exist yet, create with just HNSW data + await this.writeObjectToPath(key, hnswData) + } + } catch (error) { + // If read fails, create with just HNSW data + await this.writeObjectToPath(key, hnswData) + } } public async getHNSWData(nounId: string): Promise<{ diff --git a/src/storage/adapters/s3CompatibleStorage.ts b/src/storage/adapters/s3CompatibleStorage.ts index 2405c9b2..ef023630 100644 --- a/src/storage/adapters/s3CompatibleStorage.ts +++ b/src/storage/adapters/s3CompatibleStorage.ts @@ -3907,20 +3907,53 @@ export class S3CompatibleStorage extends BaseStorage { await this.ensureInitialized() try { - const { PutObjectCommand } = await import('@aws-sdk/client-s3') + const { PutObjectCommand, GetObjectCommand } = await import('@aws-sdk/client-s3') - // Use sharded path for HNSW data + // CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata const shard = getShardIdFromUuid(nounId) const key = `entities/nouns/hnsw/${shard}/${nounId}.json` - await this.s3Client!.send( - new PutObjectCommand({ - Bucket: this.bucketName, - Key: key, - Body: JSON.stringify(hnswData, null, 2), - ContentType: 'application/json' - }) - ) + try { + // Read existing node data + const getResponse = await this.s3Client!.send( + new GetObjectCommand({ + Bucket: this.bucketName, + Key: key + }) + ) + const existingData = await getResponse.Body!.transformToString() + const existingNode = JSON.parse(existingData) + + // Preserve id and vector, update only HNSW graph metadata + const updatedNode = { + ...existingNode, + level: hnswData.level, + connections: hnswData.connections + } + + await this.s3Client!.send( + new PutObjectCommand({ + Bucket: this.bucketName, + Key: key, + Body: JSON.stringify(updatedNode, null, 2), + ContentType: 'application/json' + }) + ) + } catch (error: any) { + // If node doesn't exist yet, create it with just HNSW data + if (error.name === 'NoSuchKey' || error.Code === 'NoSuchKey') { + await this.s3Client!.send( + new PutObjectCommand({ + Bucket: this.bucketName, + Key: key, + Body: JSON.stringify(hnswData, null, 2), + ContentType: 'application/json' + }) + ) + } else { + throw error + } + } } catch (error) { this.logger.error(`Failed to save HNSW data for ${nounId}:`, error) throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`)