fix(storage): CRITICAL - preserve vectors when updating HNSW connections (v4.7.3)
CRITICAL DATA CORRUPTION FIX affecting ALL storage adapters
## Root Cause
When HNSW index updated node connections (adding new neighbors), saveHNSWData()
overwrote the entire node file with ONLY {level, connections}, destroying vector data.
## Impact
- v4.7.2: Broke ALL imports - relate() crashed with "Cannot read properties of undefined"
- Affected ALL storage adapters: FileSystem, GCS, Azure, R2, OPFS, S3Compatible
- VFS imports completely non-functional
- Any multi-entity operation would corrupt existing entity vectors
## The Bug
```typescript
// OLD CODE (v4.7.2) - DESTROYED VECTORS:
async saveHNSWData(id, hnswData) {
await writeFile(path, JSON.stringify(hnswData)) // Only {level, connections}!
}
```
When entity2 was added to HNSW:
1. HNSW found entity1 as neighbor
2. Updated entity1's connections
3. Called saveHNSWData(entity1.id, {level, connections})
4. Overwrote entity1.json with ONLY {level, connections}
5. **entity1.vector and entity1.id were DESTROYED**
## The Fix
```typescript
// NEW CODE (v4.7.3) - PRESERVES ALL DATA:
async saveHNSWData(id, hnswData) {
const existing = await readFile(path)
const updated = {...existing, level: hnswData.level, connections: hnswData.connections}
await writeFile(path, JSON.stringify(updated)) // Preserves id, vector, etc.
}
```
Now READ existing node, UPDATE only HNSW fields, WRITE complete node.
## Files Changed
- src/storage/adapters/fileSystemStorage.ts (line 2590-2626)
- src/storage/adapters/gcsStorage.ts (line 1863-1911)
- src/storage/adapters/azureBlobStorage.ts (line 1638-1682)
- src/storage/adapters/r2Storage.ts (line 999-1029)
- src/storage/adapters/opfsStorage.ts (line 2012-2051)
- src/storage/adapters/s3CompatibleStorage.ts (line 3903-3961)
## Testing
✅ FileSystemStorage: Verified with test-relate-crash.js
✅ All adapters: Compilation successful
✅ Imports: VFS directory creation and relate() working
## Breaking Changes
NONE - This is a critical bug fix
## Migration
Workshop team: Delete brainy-data and reimport with v4.7.3
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
fc307bc215
commit
46e74827c4
8 changed files with 180 additions and 349 deletions
|
|
@ -2579,13 +2579,36 @@ export class FileSystemStorage extends BaseStorage {
|
|||
}): Promise<void> {
|
||||
await this.ensureInitialized()
|
||||
|
||||
// Use sharded path for HNSW data
|
||||
const shard = nounId.substring(0, 2).toLowerCase()
|
||||
const hnswDir = path.join(this.rootDir, 'entities', 'nouns', 'hnsw', shard)
|
||||
await this.ensureDirectoryExists(hnswDir)
|
||||
// CRITICAL FIX (v4.7.3): Must preserve existing node data (id, vector) when updating HNSW metadata
|
||||
// Previous implementation overwrote the entire file, destroying vector data
|
||||
// Now we READ the existing node, UPDATE only connections/level, then WRITE back the complete node
|
||||
|
||||
const filePath = path.join(hnswDir, `${nounId}.json`)
|
||||
await fs.promises.writeFile(filePath, JSON.stringify(hnswData, null, 2))
|
||||
const filePath = this.getNodePath(nounId)
|
||||
|
||||
try {
|
||||
// Read existing node data
|
||||
const existingData = await fs.promises.readFile(filePath, 'utf-8')
|
||||
const existingNode = JSON.parse(existingData)
|
||||
|
||||
// Preserve id and vector, update only HNSW graph metadata
|
||||
const updatedNode = {
|
||||
...existingNode, // Preserve all existing fields (id, vector, etc.)
|
||||
level: hnswData.level,
|
||||
connections: hnswData.connections
|
||||
}
|
||||
|
||||
// Write back the COMPLETE node with updated HNSW data
|
||||
await fs.promises.writeFile(filePath, JSON.stringify(updatedNode, null, 2))
|
||||
} catch (error: any) {
|
||||
// If node doesn't exist yet, create it with just HNSW data
|
||||
// This should only happen during initial node creation
|
||||
if (error.code === 'ENOENT') {
|
||||
await this.ensureDirectoryExists(path.dirname(filePath))
|
||||
await fs.promises.writeFile(filePath, JSON.stringify(hnswData, null, 2))
|
||||
} else {
|
||||
throw error
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue