soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix(storage): CRITICAL - preserve vectors when updating HNSW connections (v4.7.3)

Browse source 
CRITICAL DATA CORRUPTION FIX affecting ALL storage adapters

## Root Cause
When HNSW index updated node connections (adding new neighbors), saveHNSWData()
overwrote the entire node file with ONLY {level, connections}, destroying vector data.

## Impact
- v4.7.2: Broke ALL imports - relate() crashed with "Cannot read properties of undefined"
- Affected ALL storage adapters: FileSystem, GCS, Azure, R2, OPFS, S3Compatible
- VFS imports completely non-functional
- Any multi-entity operation would corrupt existing entity vectors

## The Bug
```typescript
// OLD CODE (v4.7.2) - DESTROYED VECTORS:
async saveHNSWData(id, hnswData) {
  await writeFile(path, JSON.stringify(hnswData))  // Only {level, connections}!
}
```

When entity2 was added to HNSW:
1. HNSW found entity1 as neighbor
2. Updated entity1's connections
3. Called saveHNSWData(entity1.id, {level, connections})
4. Overwrote entity1.json with ONLY {level, connections}
5. **entity1.vector and entity1.id were DESTROYED**

## The Fix
```typescript
// NEW CODE (v4.7.3) - PRESERVES ALL DATA:
async saveHNSWData(id, hnswData) {
  const existing = await readFile(path)
  const updated = {...existing, level: hnswData.level, connections: hnswData.connections}
  await writeFile(path, JSON.stringify(updated))  // Preserves id, vector, etc.
}
```

Now READ existing node, UPDATE only HNSW fields, WRITE complete node.

## Files Changed
- src/storage/adapters/fileSystemStorage.ts (line 2590-2626)
- src/storage/adapters/gcsStorage.ts (line 1863-1911)
- src/storage/adapters/azureBlobStorage.ts (line 1638-1682)
- src/storage/adapters/r2Storage.ts (line 999-1029)
- src/storage/adapters/opfsStorage.ts (line 2012-2051)
- src/storage/adapters/s3CompatibleStorage.ts (line 3903-3961)

## Testing
✅ FileSystemStorage: Verified with test-relate-crash.js
✅ All adapters: Compilation successful
✅ Imports: VFS directory creation and relate() working

## Breaking Changes
NONE - This is a critical bug fix

## Migration
Workshop team: Delete brainy-data and reimport with v4.7.3

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
David Snelling 2025-10-27 13:07:00 -07:00
parent fc307bc215
commit 46e74827c4
8 changed files with 180 additions and 349 deletions
Download patch file Download diff file Expand all files Collapse all files

188
VFS_DEBUG_INSTRUCTIONS.md
View file

@ -1,188 +0,0 @@
# 🔍 VFS Root Debugging Instructions
## Problem Summary
Despite 7 attempted fixes (v4.5.1 through v4.7.1), vfs.readdir('/') still returns empty even though:
- ✅ 600 document entities exist
- ✅ 608 Contains relationships exist
- ✅ 9 collection entities (directories) exist
## Root Cause Hypothesis
The VFS instance is likely using a **different root entity ID** than the actual root directory where files were imported. This would explain why:
- Import succeeds (creates files under the REAL root)
- getRelations() succeeds (relationships exist in database)
- But readdir('/') returns empty (VFS querying the WRONG root)
## Debug Script
I've created `debug-vfs-root.js` which will:
1. Show the root entity ID that VFS is using
2. List ALL collection entities (directories) in the database
3. For each directory, count outgoing Contains relationships
4. Identify which directory is the VFS root and whether it has children
## How to Run
```bash
# Make sure you're using v4.7.1
$ cd /path/to/brainy/project
# Copy the debug script to your Brainy installation
$ cp /media/dpsifr/storage/home/Projects/brainy/debug-vfs-root.js .
# Run it with your FileSystemStorage database
$ node debug-vfs-root.js
```
## Expected Output
The script will show:
```
🔍 VFS Root Debugging Script
============================================================
✅ Brainy initialized
✅ VFS instance created
📍 VFS Root Entity ID: abc123-xyz...
✅ Root entity EXISTS in database:
Type: collection
Path: /
VFS Type: directory
Created: 2025-10-27T16:45:00.000Z
🔍 Finding ALL collection entities (directories)...
Found 9 collection entities:
👑 ID: abc123-xyz...
Path: /
VFS Type: directory
Created: 2025-10-27T16:45:00.000Z
✅ THIS IS THE VFS ROOT
ID: def456-uvw...
Path: /Characters
VFS Type: directory
Created: 2025-10-27T16:46:00.000Z
[... 7 more directories ...]
🔍 Checking Contains relationships FROM each collection...
Collection abc123-xyz...:
Path: /
Outgoing Contains: 8
✅ THIS IS THE VFS ROOT - should return 8 items for readdir('/')
Collection def456-uvw...:
Path: /Characters
Outgoing Contains: 127
[... more directories ...]
🔍 Counting ALL Contains relationships in database...
Total Contains relationships: 608
📋 Sample Contains relationships (first 10):
1. abc123-xyz... -> def456-uvw...
From: / (directory)
To: /Characters (directory)
2. def456-uvw... -> ghi789-rst...
From: /Characters (directory)
To: /Characters/entity-001.json (file)
[... more samples ...]
============================================================
🏁 Debug Complete
```
## What to Look For
### ✅ Good Case (VFS Working)
```
📍 VFS Root Entity ID: abc123-xyz...
✅ Root entity EXISTS in database
Collection abc123-xyz...:
Path: /
Outgoing Contains: 8 👈 NON-ZERO!
✅ THIS IS THE VFS ROOT
```
### ❌ Bad Case #1: VFS Using Wrong Root
```
📍 VFS Root Entity ID: xxx000-yyy...
✅ Root entity EXISTS in database
Collection xxx000-yyy...:
Path: /
Outgoing Contains: 0 👈 ZERO! This is the bug!
✅ THIS IS THE VFS ROOT
Collection abc123-zzz...:
Path: /
Outgoing Contains: 8 👈 Files are under THIS root instead!
```
### ❌ Bad Case #2: VFS Root Doesn't Exist
```
📍 VFS Root Entity ID: xxx000-yyy...
❌ Root entity DOES NOT EXIST in database!
This is the bug! VFS is using a root ID that doesn't exist.
```
### ❌ Bad Case #3: Root Has No Children
```
📍 VFS Root Entity ID: abc123-xyz...
✅ Root entity EXISTS in database
Collection abc123-xyz...:
Path: /
Outgoing Contains: 0 👈 ZERO! Files aren't under root!
✅ THIS IS THE VFS ROOT
[All other collections have Contains relationships, but NOT from root]
```
## What This Tells Us
Based on the output, we can determine:
1. **If "Outgoing Contains: 0" for VFS root**: The VFS is querying the correct root, but that root has no children. This means files were created under a DIFFERENT root or weren't properly linked.
2. **If multiple "/" paths exist**: There are duplicate root entities. VFS is using one root, but files are under a different root.
3. **If "Root entity DOES NOT EXIST"**: VFS is using a stale/wrong entity ID that was deleted or never existed.
4. **If "Outgoing Contains: 8"**: VFS root HAS children! This means my v4.7.1 fix has another bug - the optimization isn't being triggered or has a logic error.
## Next Steps
Please run this script and send me the full output. Based on what we see, I'll know exactly how to fix the real bug!
## Architecture Notes
The VFS query flow is:
```
vfs.readdir('/')
-> PathResolver.resolve('/')
-> returns this.rootEntityId
-> PathResolver.getChildren(rootEntityId)
-> brain.getRelations({ from: rootEntityId, type: VerbType.Contains })
-> brainy.ts builds filter
-> storage.getVerbs({ filter: { sourceId: rootEntityId, verbType } })
-> MY v4.7.1 FIX: Check if sourceId + verbType filters exist
-> If YES: Call getVerbsBySource_internal(rootEntityId)
-> Get verbs from graph adjacency index
-> Filter by verbType
-> Return Contains relationships
```
If the debug shows "Outgoing Contains: 0" for the VFS root, then either:
- A. getVerbsBySource_internal() is broken
- B. The graph adjacency index isn't being populated
- C. My v4.7.1 optimization has a bug and isn't being called
- D. The rootEntityId is wrong
This script will tell us which one!

124
debug-vfs-root.js
View file

@ -1,124 +0,0 @@
/**
* VFS Root Debugging Script
*
* This script debugs why vfs.readdir('/') returns empty despite
* 608 Contains relationships existing in the database.
*/
import { Brainy } from './dist/brainy.js'
import { VerbType, NounType } from './dist/types/graphTypes.js'
async function debugVFSRoot() {
console.log('\n🔍 VFS Root Debugging Script\n')
console.log('=' .repeat(60))
// Initialize Brainy with FileSystemStorage
const brain = new Brainy({
storage: {
type: 'filesystem',
config: {
path: './brainy-data'
}
}
})
await brain.ready()
console.log('✅ Brainy initialized\n')
// Get VFS instance
const vfs = brain.vfs()
console.log('✅ VFS instance created\n')
// Get the root entity ID that VFS is using
const vfsRootId = (vfs as any).rootEntityId
console.log(`📍 VFS Root Entity ID: ${vfsRootId}\n`)
// Try to get the root entity
const rootEntity = await brain.get(vfsRootId)
if (rootEntity) {
console.log('✅ Root entity EXISTS in database:')
console.log(` Type: ${rootEntity.type}`)
console.log(` Path: ${rootEntity.metadata?.path}`)
console.log(` VFS Type: ${rootEntity.metadata?.vfsType}`)
console.log(` Created: ${new Date(rootEntity.createdAt || 0).toISOString()}`)
console.log()
} else {
console.log('❌ Root entity DOES NOT EXIST in database!')
console.log(' This is the bug! VFS is using a root ID that doesn\'t exist.\n')
}
// Find ALL collection entities (directories)
console.log('🔍 Finding ALL collection entities (directories)...')
const allCollections = await brain.find({
type: NounType.Collection,
limit: 100
})
console.log(` Found ${allCollections.length} collection entities:\n`)
for (const coll of allCollections) {
const isRoot = coll.metadata?.path === '/' && coll.metadata?.vfsType === 'directory'
console.log(` ${isRoot ? '👑' : ' '} ID: ${coll.id}`)
console.log(` Path: ${coll.metadata?.path}`)
console.log(` VFS Type: ${coll.metadata?.vfsType}`)
console.log(` Created: ${new Date(coll.entity?.createdAt || 0).toISOString()}`)
// Check if this matches VFS root
if (coll.id === vfsRootId) {
console.log(` ✅ THIS IS THE VFS ROOT`)
}
console.log()
}
// For each potential root, count outgoing Contains relationships
console.log('🔍 Checking Contains relationships FROM each collection...\n')
for (const coll of allCollections) {
const relations = await brain.getRelations({
from: coll.id,
type: VerbType.Contains
})
console.log(` Collection ${coll.id}:`)
console.log(` Path: ${coll.metadata?.path}`)
console.log(` Outgoing Contains: ${relations.length}`)
if (coll.id === vfsRootId) {
console.log(` ✅ THIS IS THE VFS ROOT - should return ${relations.length} items for readdir('/')`)
if (relations.length === 0) {
console.log(` ❌ BUT IT HAS ZERO CHILDREN! This is why readdir() returns empty!`)
}
}
console.log()
}
// Count total Contains relationships in database
console.log('🔍 Counting ALL Contains relationships in database...')
const allContains = await brain.getRelations({
type: VerbType.Contains,
limit: 1000
})
console.log(` Total Contains relationships: ${allContains.length}\n`)
// Sample some Contains relationships
console.log('📋 Sample Contains relationships (first 10):')
for (let i = 0; i < Math.min(10, allContains.length); i++) {
const rel = allContains[i]
const fromEntity = await brain.get(rel.from)
const toEntity = await brain.get(rel.to)
console.log(` ${i + 1}. ${rel.from} -> ${rel.to}`)
console.log(` From: ${fromEntity?.metadata?.path || 'unknown'} (${fromEntity?.metadata?.vfsType || 'unknown'})`)
console.log(` To: ${toEntity?.metadata?.path || 'unknown'} (${toEntity?.metadata?.vfsType || 'unknown'})`)
}
console.log('\n' + '='.repeat(60))
console.log('🏁 Debug Complete\n')
process.exit(0)
}
debugVFSRoot().catch(err => {
console.error('❌ Error:', err)
process.exit(1)
})

27
src/storage/adapters/azureBlobStorage.ts
View file

@ -1642,14 +1642,39 @@ export class AzureBlobStorage extends BaseStorage {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
// CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata
const shard = getShardIdFromUuid(nounId) const shard = getShardIdFromUuid(nounId)
const key = `entities/nouns/hnsw/${shard}/${nounId}.json` const key = `entities/nouns/hnsw/${shard}/${nounId}.json`
const blockBlobClient = this.containerClient!.getBlockBlobClient(key) const blockBlobClient = this.containerClient!.getBlockBlobClient(key)
try {
// Read existing node data
const downloadResponse = await blockBlobClient.download(0)
const existingData = await this.streamToBuffer(downloadResponse.readableStreamBody!)
const existingNode = JSON.parse(existingData.toString())
// Preserve id and vector, update only HNSW graph metadata
const updatedNode = {
...existingNode,
level: hnswData.level,
connections: hnswData.connections
}
const content = JSON.stringify(updatedNode, null, 2)
await blockBlobClient.upload(content, content.length, {
blobHTTPHeaders: { blobContentType: 'application/json' }
})
} catch (error: any) {
// If node doesn't exist yet, create it with just HNSW data
if (error.statusCode === 404 || error.code === 'BlobNotFound') {
const content = JSON.stringify(hnswData, null, 2) const content = JSON.stringify(hnswData, null, 2)
await blockBlobClient.upload(content, content.length, { await blockBlobClient.upload(content, content.length, {
blobHTTPHeaders: { blobContentType: 'application/json' } blobHTTPHeaders: { blobContentType: 'application/json' }
}) })
} else {
throw error
}
}
} catch (error) { } catch (error) {
this.logger.error(`Failed to save HNSW data for ${nounId}:`, error) this.logger.error(`Failed to save HNSW data for ${nounId}:`, error)
throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`) throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`)

33
src/storage/adapters/fileSystemStorage.ts
View file

@ -2579,13 +2579,36 @@ export class FileSystemStorage extends BaseStorage {
}): Promise<void> { }): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// Use sharded path for HNSW data // CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata
const shard = nounId.substring(0, 2).toLowerCase() // Previous implementation overwrote the entire file, destroying vector data
const hnswDir = path.join(this.rootDir, 'entities', 'nouns', 'hnsw', shard) // Now we READ the existing node, UPDATE only connections/level, then WRITE back the complete node
await this.ensureDirectoryExists(hnswDir)
const filePath = path.join(hnswDir, `${nounId}.json`) const filePath = this.getNodePath(nounId)
try {
// Read existing node data
const existingData = await fs.promises.readFile(filePath, 'utf-8')
const existingNode = JSON.parse(existingData)
// Preserve id and vector, update only HNSW graph metadata
const updatedNode = {
...existingNode, // Preserve all existing fields (id, vector, etc.)
level: hnswData.level,
connections: hnswData.connections
}
// Write back the COMPLETE node with updated HNSW data
await fs.promises.writeFile(filePath, JSON.stringify(updatedNode, null, 2))
} catch (error: any) {
// If node doesn't exist yet, create it with just HNSW data
// This should only happen during initial node creation
if (error.code === 'ENOENT') {
await this.ensureDirectoryExists(path.dirname(filePath))
await fs.promises.writeFile(filePath, JSON.stringify(hnswData, null, 2)) await fs.promises.writeFile(filePath, JSON.stringify(hnswData, null, 2))
} else {
throw error
}
}
} }
/** /**

32
src/storage/adapters/gcsStorage.ts
View file

@ -1867,15 +1867,43 @@ export class GcsStorage extends BaseStorage {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
// Use sharded path for HNSW data // CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata
// Previous implementation overwrote the entire file, destroying vector data
// Now we READ the existing node, UPDATE only connections/level, then WRITE back the complete node
const shard = getShardIdFromUuid(nounId) const shard = getShardIdFromUuid(nounId)
const key = `entities/nouns/hnsw/${shard}/${nounId}.json` const key = `entities/nouns/hnsw/${shard}/${nounId}.json`
const file = this.bucket!.file(key) const file = this.bucket!.file(key)
try {
// Read existing node data
const [existingData] = await file.download()
const existingNode = JSON.parse(existingData.toString())
// Preserve id and vector, update only HNSW graph metadata
const updatedNode = {
...existingNode, // Preserve all existing fields (id, vector, etc.)
level: hnswData.level,
connections: hnswData.connections
}
// Write back the COMPLETE node with updated HNSW data
await file.save(JSON.stringify(updatedNode, null, 2), {
contentType: 'application/json',
resumable: false
})
} catch (error: any) {
// If node doesn't exist yet, create it with just HNSW data
// This should only happen during initial node creation
if (error.code === 404) {
await file.save(JSON.stringify(hnswData, null, 2), { await file.save(JSON.stringify(hnswData, null, 2), {
contentType: 'application/json', contentType: 'application/json',
resumable: false resumable: false
}) })
} else {
throw error
}
}
} catch (error) { } catch (error) {
this.logger.error(`Failed to save HNSW data for ${nounId}:`, error) this.logger.error(`Failed to save HNSW data for ${nounId}:`, error)
throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`) throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`)

26
src/storage/adapters/opfsStorage.ts
View file

@ -2016,20 +2016,34 @@ export class OPFSStorage extends BaseStorage {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
// Get or create the hnsw directory under nouns // CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata
const hnswDir = await this.nounsDir!.getDirectoryHandle('hnsw', { create: true }) const hnswDir = await this.nounsDir!.getDirectoryHandle('hnsw', { create: true })
// Use sharded path for HNSW data
const shard = getShardIdFromUuid(nounId) const shard = getShardIdFromUuid(nounId)
const shardDir = await hnswDir.getDirectoryHandle(shard, { create: true }) const shardDir = await hnswDir.getDirectoryHandle(shard, { create: true })
// Create or get the file in the shard directory
const fileHandle = await shardDir.getFileHandle(`${nounId}.json`, { create: true }) const fileHandle = await shardDir.getFileHandle(`${nounId}.json`, { create: true })
// Write the HNSW data to the file try {
// Read existing node data
const file = await fileHandle.getFile()
const existingData = await file.text()
const existingNode = JSON.parse(existingData)
// Preserve id and vector, update only HNSW graph metadata
const updatedNode = {
...existingNode,
level: hnswData.level,
connections: hnswData.connections
}
const writable = await fileHandle.createWritable()
await writable.write(JSON.stringify(updatedNode, null, 2))
await writable.close()
} catch (error) {
// If node doesn't exist or read fails, create with just HNSW data
const writable = await fileHandle.createWritable() const writable = await fileHandle.createWritable()
await writable.write(JSON.stringify(hnswData, null, 2)) await writable.write(JSON.stringify(hnswData, null, 2))
await writable.close() await writable.close()
}
} catch (error) { } catch (error) {
console.error(`Failed to save HNSW data for ${nounId}:`, error) console.error(`Failed to save HNSW data for ${nounId}:`, error)
throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`) throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`)

20
src/storage/adapters/r2Storage.ts
View file

@ -1002,11 +1002,31 @@ export class R2Storage extends BaseStorage {
}): Promise<void> { }): Promise<void> {
await this.ensureInitialized() await this.ensureInitialized()
// CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata
const shard = getShardIdFromUuid(nounId) const shard = getShardIdFromUuid(nounId)
const key = `entities/nouns/hnsw/${shard}/${nounId}.json` const key = `entities/nouns/hnsw/${shard}/${nounId}.json`
try {
// Read existing node data
const existingNode = await this.readObjectFromPath(key)
if (existingNode) {
// Preserve id and vector, update only HNSW graph metadata
const updatedNode = {
...existingNode,
level: hnswData.level,
connections: hnswData.connections
}
await this.writeObjectToPath(key, updatedNode)
} else {
// Node doesn't exist yet, create with just HNSW data
await this.writeObjectToPath(key, hnswData) await this.writeObjectToPath(key, hnswData)
} }
} catch (error) {
// If read fails, create with just HNSW data
await this.writeObjectToPath(key, hnswData)
}
}
public async getHNSWData(nounId: string): Promise<{ public async getHNSWData(nounId: string): Promise<{
level: number level: number

37
src/storage/adapters/s3CompatibleStorage.ts
View file

@ -3907,12 +3907,41 @@ export class S3CompatibleStorage extends BaseStorage {
await this.ensureInitialized() await this.ensureInitialized()
try { try {
const { PutObjectCommand } = await import('@aws-sdk/client-s3') const { PutObjectCommand, GetObjectCommand } = await import('@aws-sdk/client-s3')
// Use sharded path for HNSW data // CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata
const shard = getShardIdFromUuid(nounId) const shard = getShardIdFromUuid(nounId)
const key = `entities/nouns/hnsw/${shard}/${nounId}.json` const key = `entities/nouns/hnsw/${shard}/${nounId}.json`
try {
// Read existing node data
const getResponse = await this.s3Client!.send(
new GetObjectCommand({
Bucket: this.bucketName,
Key: key
})
)
const existingData = await getResponse.Body!.transformToString()
const existingNode = JSON.parse(existingData)
// Preserve id and vector, update only HNSW graph metadata
const updatedNode = {
...existingNode,
level: hnswData.level,
connections: hnswData.connections
}
await this.s3Client!.send(
new PutObjectCommand({
Bucket: this.bucketName,
Key: key,
Body: JSON.stringify(updatedNode, null, 2),
ContentType: 'application/json'
})
)
} catch (error: any) {
// If node doesn't exist yet, create it with just HNSW data
if (error.name === 'NoSuchKey' || error.Code === 'NoSuchKey') {
await this.s3Client!.send( await this.s3Client!.send(
new PutObjectCommand({ new PutObjectCommand({
Bucket: this.bucketName, Bucket: this.bucketName,
@ -3921,6 +3950,10 @@ export class S3CompatibleStorage extends BaseStorage {
ContentType: 'application/json' ContentType: 'application/json'
}) })
) )
} else {
throw error
}
}
} catch (error) { } catch (error) {
this.logger.error(`Failed to save HNSW data for ${nounId}:`, error) this.logger.error(`Failed to save HNSW data for ${nounId}:`, error)
throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`) throw new Error(`Failed to save HNSW data for ${nounId}: ${error}`)