fix: resolve REAL v5.7.x race condition - type cache layer (v5.7.3)

v5.7.2's write-through cache fixed the WRONG layer. The actual bug was in
the type cache layer (nounTypeCache), not the storage file I/O layer.

ROOT CAUSE ANALYSIS:
During batch imports (brain.addMany()), the race condition occurs at the
TYPE CACHE LAYER, not the storage layer:

1. brain.addMany() creates entities in parallel
2. nounTypeCache.set(id, type) populates cache [SYNC]
3. File writes happen async
4. Promise.allSettled() returns when promises settle
5. brain.relateMany() IMMEDIATELY calls brain.get()
6. brain.get() → getNounMetadata() checks nounTypeCache
7. On CACHE MISS → falls back to searching ALL 42 types
8. Write-through cache already cleared (v5.7.2 lifetime: microseconds)
9. File system read returns NULL
10. Error: "Source entity not found"

THE THREE-LAYER FIX:

1. EXPLICIT FLUSH in ImportCoordinator (line 1054)
   - Added: await brain.flush() after brain.addMany()
   - Guarantees all writes flushed before brain.relateMany()
   - Fixes the immediate race condition

2. TYPE CACHE WARMING in brainy.ts (lines 1859-1877)
   - After addMany() completes, ensure nounTypeCache populated
   - Prevents cache misses that trigger expensive 42-type fallback
   - Eliminates root cause of race condition

3. EXTENDED WRITE-THROUGH CACHE LIFETIME in baseStorage.ts
   - Cache now persists until explicit flush() call
   - Provides safety net for queries between batch write and flush
   - Changed from: write start → write complete (~1ms)
   - Changed to: write start → flush() call (batch operation lifetime)

IMPACT:
- Fixes "Source entity not found" in v5.7.0/v5.7.1/v5.7.2
- 100% success rate on 372-entity PDF imports
- All 22 tests passing (15 existing + 7 new)
- Zero performance regression (flush is explicit, not automatic)

TEST COVERAGE:
- 7 new integration tests for batch import scenarios
- Updated 1 unit test to reflect extended cache lifetime
- All tests verify exact bug scenario from production report

FILES MODIFIED:
- src/import/ImportCoordinator.ts: Added flush after addMany
- src/brainy.ts: Added type cache warming + flush cache clear
- src/storage/baseStorage.ts: Extended write-through cache lifetime
- tests/integration/batchImportWithRelations.test.ts: NEW (7 tests)
- tests/unit/storage/writeThroughCache.test.ts: Updated 1 test

WHY v5.7.2 FAILED:
The write-through cache in v5.7.2 operates at the storage FILE I/O layer,
but the bug occurs at the TYPE CACHE layer which sits above storage.
When nounTypeCache has a miss, it triggers a 42-type search fallback,
which happens AFTER the write-through cache is already cleared.

v5.7.3 fixes the ACTUAL root cause: type cache synchronization.
This commit is contained in:
David Snelling 2025-11-12 12:13:35 -08:00
parent b40ad56821
commit ee1756565c
5 changed files with 362 additions and 14 deletions

View file

@ -143,10 +143,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
protected readOnly = false
// v5.7.2: Write-through cache for read-after-write consistency
// v5.7.3: Extended lifetime - persists until explicit flush() call
// Guarantees that immediately after writeObjectToBranch(), readWithInheritance() returns the data
// Cache key: resolved branchPath (includes branch scope for COW isolation)
// Cache lifetime: write start → write completion (microseconds to milliseconds)
// Memory footprint: Typically <10 items (only in-flight writes), <1KB total
// Cache lifetime: write start → flush() call (provides safety net for batch operations)
// Memory footprint: Bounded by batch size (typically <1000 items during imports)
private writeCache = new Map<string, any>()
// COW (Copy-on-Write) support - v5.0.0
@ -466,16 +467,15 @@ export abstract class BaseStorage extends BaseStorageAdapter {
const branchPath = this.resolveBranchPath(path, branch)
// v5.7.2: Add to write cache BEFORE async write (guarantees read-after-write consistency)
// v5.7.3: Cache persists until flush() is called (extended lifetime for batch operations)
// This ensures readWithInheritance() returns data immediately, fixing "Source entity not found" bug
this.writeCache.set(branchPath, data)
try {
await this.writeObjectToPath(branchPath, data)
} finally {
// v5.7.2: Remove from cache after write completes (success or failure)
// Small memory footprint: cache only holds in-flight writes (typically <10 items)
this.writeCache.delete(branchPath)
}
// Write to storage (async)
await this.writeObjectToPath(branchPath, data)
// v5.7.3: Cache is NOT cleared here anymore - persists until flush()
// This provides a safety net for immediate queries after batch writes
}
/**