fix: import dedup off-switch honesty + brain-owned lifecycle for the background pass
The post-import background deduplication pass (a merge-DELETE writer, debounced ~5 minutes after import) had four lifecycle defects: - enableDeduplication: false did not gate the background schedule — an import that explicitly opted out could still have entities auto-removed minutes later. The flag now gates both the inline and background passes. - Each import() constructed its own coordinator-owned deduplicator, so the debounce never spanned imports (N imports = N delete timers). The brain now owns a single lazy instance (getBackgroundDeduplicator). - close() never cancelled pending dedup; a delete pass could fire against a closed brain. close() now cancels it first. - The 5-minute timer held the process open (exit-hang class); now unref'd. Four regression tests pin the contract (background-dedup-lifecycle); import guides document that false disables both passes.
This commit is contained in:
parent
120205b69c
commit
4fcef7b8ed
7 changed files with 179 additions and 10 deletions
|
|
@ -300,7 +300,10 @@ await brain.import(data, {
|
|||
// Deduplication
|
||||
enableDeduplication: true, // Check for duplicate entities (default: true)
|
||||
deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
|
||||
// Note: Auto-disabled for imports >100 entities
|
||||
// Notes: false disables BOTH the inline merge and the background pass that
|
||||
// runs ~5 min after the last import (merged duplicates are deleted).
|
||||
// The inline pass auto-disables for imports >100 entities (O(n²) cost);
|
||||
// the background pass still covers those unless the flag is false.
|
||||
|
||||
// Performance
|
||||
chunkSize: 100, // Batch size for processing (default: varies by operation)
|
||||
|
|
|
|||
|
|
@ -86,11 +86,22 @@ await brain.import(file, {
|
|||
|
||||
```typescript
|
||||
await brain.import(file, {
|
||||
enableDeduplication: true, // Check for duplicates (default: false)
|
||||
enableDeduplication: true, // Check for duplicates (default: true)
|
||||
deduplicationThreshold: 0.85 // Similarity threshold (default: 0.85)
|
||||
})
|
||||
```
|
||||
|
||||
Deduplication merges entities judged duplicates — the non-primary records are
|
||||
**deleted**. Set `enableDeduplication: false` to disable it entirely: the flag
|
||||
gates both the inline merge during import and the background pass that runs
|
||||
about 5 minutes after the last import.
|
||||
|
||||
```typescript
|
||||
await brain.import(file, {
|
||||
enableDeduplication: false // No merging, inline or background
|
||||
})
|
||||
```
|
||||
|
||||
### Import Tracking
|
||||
|
||||
Track and organize imports by project:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue