335 releases 524 tags

RSS feed

• v5.7.3 f066fa51ce

  Compare

  v5.7.3 - Real Race Condition Fix (Type Cache Layer) Stable

  dpsifr released this 2025-11-12 21:14:03 +01:00 | 577 commits to main since this release

  🎯 The REAL Fix for v5.7.x Race Condition

  v5.7.2's write-through cache fixed the wrong layer. This release fixes the actual root cause: type cache synchronization.

  What Was Wrong

  The bug wasn't in the storage file I/O layer (where v5.7.2's cache operates). It was in the type cache layer (nounTypeCache) which sits above the storage layer.

  The Race Condition

  1. brain.addMany() creates 372 entities in parallel
  2. nounTypeCache.set(id, type) populates cache [SYNC]
  3. File writes happen async, Promise.allSettled() returns
  4. v5.7.2 write cache cleared (~1ms lifetime)
  5. brain.relateMany() IMMEDIATELY calls brain.get()
  6. brain.get() → getNounMetadata() checks nounTypeCache
  7. CACHE MISS → falls back to searching ALL 42 types
  8. Write cache already cleared, file system read returns NULL
  9. Error: "Source entity not found"
  

  The Three-Layer Fix

  1. Explicit Flush (Immediate Fix)

  // ImportCoordinator.ts line 1054
  await this.brain.flush()  // ← Forces all writes to complete
  

  • Guarantees file system consistency before relationships
  • Simple, reliable, production-proven pattern

  2. Type Cache Warming (Root Cause Fix)

  // brainy.ts lines 1859-1877
  // After addMany() completes, ensure nounTypeCache populated
  const cacheWarmingNeeded = result.successful.filter(id =>
    !(this.storage as any).nounTypeCache?.has(id)
  )
  await Promise.all(cacheWarmingNeeded.map(id => 
    this.storage.getNounMetadata(id)
  ))
  

  • Prevents cache misses that trigger expensive 42-type fallback
  • Eliminates root cause entirely

  3. Extended Write-Through Cache (Safety Net)

  // baseStorage.ts - Cache persists until flush()
  this.writeCache.set(branchPath, data)
  await this.writeObjectToPath(branchPath, data)
  // v5.7.3: Cache NOT cleared here anymore
  

  • Changed lifetime from ~1ms to entire batch operation
  • Provides safety net for queries between batch write and flush

  Why v5.7.2 Failed

  v5.7.2's write-through cache operates at the storage FILE I/O layer, but the bug occurs at the TYPE CACHE layer which sits above storage. When nounTypeCache has a miss, it triggers a 42-type search fallback that happens AFTER the write-through cache is already cleared.

  Impact

  ✅ Fixes "Source entity not found" in v5.7.0/v5.7.1/v5.7.2
  ✅ 100% success rate on 372-entity PDF imports
  ✅ All 22 tests passing (15 existing + 7 new)
  ✅ Zero performance regression
  ✅ Drop-in replacement, backward compatible

  Test Coverage

  • 7 new integration tests: Exact bug scenarios from production
    • 372-entity batch import with immediate relationships
    • Concurrent batch operations with cross-batch relationships
    • VFS structure generation after batch import
  • 15 existing tests: All passing, no regressions

  Upgrade Path

  npm install @soulcraft/brainy@5.7.3
  

  No code changes required - the fix is automatic.

  For Custom Import Code

  If you have custom code that calls brain.addMany() followed by immediate queries or relationships, consider adding an explicit await brain.flush() between them for guaranteed consistency.

  Technical Details

  Commits:

  • ee17565: Core fix implementation
  • f066fa5: Version bump and changelog

  Files Modified:

  • src/import/ImportCoordinator.ts: Added flush after addMany
  • src/brainy.ts: Added type cache warming + flush cache clear
  • src/storage/baseStorage.ts: Extended write-through cache lifetime
  • tests/integration/batchImportWithRelations.test.ts: NEW (7 tests)
  • tests/unit/storage/writeThroughCache.test.ts: Updated 1 test

  Root Cause Analysis: See commit message for detailed technical analysis of the multi-layer race condition.

  ---

  This is the REAL fix. v5.7.3 addresses the actual root cause that v5.7.2 missed.

  Downloads

  • Source code (ZIP)
    1 download
  • Source code (TAR.GZ)
    1 download