perf: optimize imports with background deduplication (12-24x speedup)

- Remove O(n²) deduplication from import path for 12-24x faster imports
- Implement BackgroundDeduplicator with 3-tier strategy (ID/Name/Similarity)
- Sequential tier processing reduces entity set after each pass
- Auto-schedules 5 minutes after imports (debounced, zero config)
- Import-scoped deduplication prevents cross-contamination

GraphAdjacencyIndex improvements:
- Fix concurrent rebuild race condition with promise-based locking
- Fix removeVerb() by filtering deleted IDs in query methods
- Replace console.* with prodLog for silent mode compatibility

Performance impact:
- Import speed: O(n²) → O(n) complexity
- 400 entities: 24 min → 2 min (12x faster)
- 1000 entities: >2 hours → 5 min (24x faster)
- Background dedup uses existing indexes (TypeAware HNSW, MetadataIndexManager)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
David Snelling 2025-11-11 14:10:14 -08:00
parent 804319ecaf
commit 02c80a045b
5 changed files with 743 additions and 177 deletions

View file

@ -11,6 +11,7 @@
export { ImportCoordinator } from './ImportCoordinator.js'
export { FormatDetector, SupportedFormat, DetectionResult } from './FormatDetector.js'
export { EntityDeduplicator } from './EntityDeduplicator.js'
export { BackgroundDeduplicator } from './BackgroundDeduplicator.js'
export { ImportHistory } from './ImportHistory.js'
export type {
@ -27,6 +28,10 @@ export type {
MergeResult
} from './EntityDeduplicator.js'
export type {
DeduplicationStats
} from './BackgroundDeduplicator.js'
export type {
ImportHistoryEntry,
RollbackResult