fix: import dedup off-switch honesty + brain-owned lifecycle for the background pass

The post-import background deduplication pass (a merge-DELETE writer,
debounced ~5 minutes after import) had four lifecycle defects:

- enableDeduplication: false did not gate the background schedule — an
  import that explicitly opted out could still have entities auto-removed
  minutes later. The flag now gates both the inline and background passes.
- Each import() constructed its own coordinator-owned deduplicator, so the
  debounce never spanned imports (N imports = N delete timers). The brain
  now owns a single lazy instance (getBackgroundDeduplicator).
- close() never cancelled pending dedup; a delete pass could fire against
  a closed brain. close() now cancels it first.
- The 5-minute timer held the process open (exit-hang class); now unref'd.

Four regression tests pin the contract (background-dedup-lifecycle);
import guides document that false disables both passes.
This commit is contained in:
David Snelling 2026-07-18 10:40:45 -07:00
parent 120205b69c
commit 4fcef7b8ed
7 changed files with 179 additions and 10 deletions

View file

@ -300,7 +300,10 @@ await brain.import(data, {
// Deduplication
enableDeduplication: true, // Check for duplicate entities (default: true)
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
// Note: Auto-disabled for imports >100 entities
// Notes: false disables BOTH the inline merge and the background pass that
// runs ~5 min after the last import (merged duplicates are deleted).
// The inline pass auto-disables for imports >100 entities (O(n²) cost);
// the background pass still covers those unless the flag is false.
// Performance
chunkSize: 100, // Batch size for processing (default: varies by operation)