fix: import dedup off-switch honesty + brain-owned lifecycle for the background pass

The post-import background deduplication pass (a merge-DELETE writer,
debounced ~5 minutes after import) had four lifecycle defects:

- enableDeduplication: false did not gate the background schedule — an
  import that explicitly opted out could still have entities auto-removed
  minutes later. The flag now gates both the inline and background passes.
- Each import() constructed its own coordinator-owned deduplicator, so the
  debounce never spanned imports (N imports = N delete timers). The brain
  now owns a single lazy instance (getBackgroundDeduplicator).
- close() never cancelled pending dedup; a delete pass could fire against
  a closed brain. close() now cancels it first.
- The 5-minute timer held the process open (exit-hang class); now unref'd.

Four regression tests pin the contract (background-dedup-lifecycle);
import guides document that false disables both passes.
This commit is contained in:
David Snelling 2026-07-18 10:40:45 -07:00
parent 120205b69c
commit 4fcef7b8ed
7 changed files with 179 additions and 10 deletions

View file

@ -41,6 +41,14 @@ export interface DeduplicationStats {
* - Import-scoped deduplication (no cross-contamination)
* - 3-tier strategy (ID Name Similarity)
* - Uses existing indexes (EntityIdMapper, MetadataIndexManager, TypeAware HNSW)
*
* Lifecycle: ONE instance per brain, owned by Brainy (getBackgroundDeduplicator)
* so the debounce genuinely spans imports and brain.close() cancels pending
* work via cancelPending() this pass merge-DELETES duplicate entities, so it
* must never fire against a closed brain. The enableDeduplication gate lives
* at the scheduling call site (ImportCoordinator); scheduleDedup itself is
* unconditional. The timer is unref'd a pending pass never holds the
* process open.
*/
export class BackgroundDeduplicator {
private brain: Brainy
@ -67,12 +75,15 @@ export class BackgroundDeduplicator {
clearTimeout(this.debounceTimer)
}
// Schedule for 5 minutes from now
// Schedule for 5 minutes from now. unref'd: a pending dedup pass must
// never hold the process open (exit-hang class) — if the process exits
// first, the pass simply never runs; imports are already durable.
this.debounceTimer = setTimeout(() => {
this.runBatchDedup().catch(error => {
prodLog.error('[BackgroundDedup] Batch dedup failed:', error)
})
}, 5 * 60 * 1000)
this.debounceTimer.unref?.()
}
/**