feat: add unified import system with auto-detection and dual storage
Implemented a comprehensive unified import system that revolutionizes how data flows into Brainy:
## Core Features (Phase 1)
- Auto-detection of file formats (Excel, PDF, CSV, JSON, Markdown) via magic bytes and content analysis
- Dual storage architecture: creates both VFS files AND knowledge graph entities
- Single unified API: brain.import() handles all formats automatically
- Format-specific importers for optimal extraction from each file type
- VFS structure generation with configurable grouping (by type, sheet, or flat)
## Entity Deduplication (Phase 2)
- Embedding-based similarity matching to detect duplicate entities across imports
- Intelligent merging with provenance tracking (records which imports contributed)
- Fuzzy name matching using Levenshtein distance
- Confidence score merging with weighted averages
- Cross-import shared knowledge: same entity referenced in multiple datasets gets merged
## Streaming Support (Phase 3)
- Chunked processing for memory-efficient handling of large datasets
- Configurable chunk size for optimal performance
- Progress tracking with real-time callbacks
- Scales to millions of entities without memory issues
## Import History & Rollback (Phase 4)
- Complete tracking of all imports with full metadata
- Rollback capability to undo any import completely
- Statistics and analytics across all imports
- Persistent history stored in VFS
## Architecture
- ImportCoordinator: orchestrates the entire import pipeline
- FormatDetector: auto-detects file formats with high confidence
- EntityDeduplicator: prevents duplicate entities across imports
- ImportHistory: tracks and enables rollback of imports
- Format-specific importers: SmartExcelImporter, SmartPDFImporter, etc.
- VFSStructureGenerator: creates organized file hierarchies
## Usage
```typescript
const result = await brain.import('/path/to/file.xlsx', {
vfsPath: '/imports/data',
groupBy: 'type',
enableDeduplication: true,
onProgress: (progress) => console.log(progress)
})
```
## Production Ready
- 5,500+ lines of production code
- All integration tests passing
- No mocks, stubs, or TODOs
- Full TypeScript type safety
- Comprehensive error handling
- Memory efficient and scalable
Closes requirements for unified data ingestion pipeline.
2025-10-08 16:55:30 -07:00
|
|
|
/**
|
|
|
|
|
* Unified Import System
|
|
|
|
|
*
|
|
|
|
|
* Single entry point for importing any file format into Brainy with:
|
|
|
|
|
* - Auto-detection of formats
|
|
|
|
|
* - Dual storage (VFS + Knowledge Graph)
|
|
|
|
|
* - Shared entities across imports (deduplication)
|
|
|
|
|
* - Simple, powerful API
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
export { ImportCoordinator } from './ImportCoordinator.js'
|
|
|
|
|
export { FormatDetector, SupportedFormat, DetectionResult } from './FormatDetector.js'
|
|
|
|
|
export { EntityDeduplicator } from './EntityDeduplicator.js'
|
2025-11-11 14:10:14 -08:00
|
|
|
export { BackgroundDeduplicator } from './BackgroundDeduplicator.js'
|
feat: add unified import system with auto-detection and dual storage
Implemented a comprehensive unified import system that revolutionizes how data flows into Brainy:
## Core Features (Phase 1)
- Auto-detection of file formats (Excel, PDF, CSV, JSON, Markdown) via magic bytes and content analysis
- Dual storage architecture: creates both VFS files AND knowledge graph entities
- Single unified API: brain.import() handles all formats automatically
- Format-specific importers for optimal extraction from each file type
- VFS structure generation with configurable grouping (by type, sheet, or flat)
## Entity Deduplication (Phase 2)
- Embedding-based similarity matching to detect duplicate entities across imports
- Intelligent merging with provenance tracking (records which imports contributed)
- Fuzzy name matching using Levenshtein distance
- Confidence score merging with weighted averages
- Cross-import shared knowledge: same entity referenced in multiple datasets gets merged
## Streaming Support (Phase 3)
- Chunked processing for memory-efficient handling of large datasets
- Configurable chunk size for optimal performance
- Progress tracking with real-time callbacks
- Scales to millions of entities without memory issues
## Import History & Rollback (Phase 4)
- Complete tracking of all imports with full metadata
- Rollback capability to undo any import completely
- Statistics and analytics across all imports
- Persistent history stored in VFS
## Architecture
- ImportCoordinator: orchestrates the entire import pipeline
- FormatDetector: auto-detects file formats with high confidence
- EntityDeduplicator: prevents duplicate entities across imports
- ImportHistory: tracks and enables rollback of imports
- Format-specific importers: SmartExcelImporter, SmartPDFImporter, etc.
- VFSStructureGenerator: creates organized file hierarchies
## Usage
```typescript
const result = await brain.import('/path/to/file.xlsx', {
vfsPath: '/imports/data',
groupBy: 'type',
enableDeduplication: true,
onProgress: (progress) => console.log(progress)
})
```
## Production Ready
- 5,500+ lines of production code
- All integration tests passing
- No mocks, stubs, or TODOs
- Full TypeScript type safety
- Comprehensive error handling
- Memory efficient and scalable
Closes requirements for unified data ingestion pipeline.
2025-10-08 16:55:30 -07:00
|
|
|
export { ImportHistory } from './ImportHistory.js'
|
|
|
|
|
|
|
|
|
|
export type {
|
|
|
|
|
ImportSource,
|
|
|
|
|
ImportOptions,
|
|
|
|
|
ImportProgress,
|
|
|
|
|
ImportResult
|
|
|
|
|
} from './ImportCoordinator.js'
|
|
|
|
|
|
|
|
|
|
export type {
|
|
|
|
|
EntityCandidate,
|
|
|
|
|
DuplicateMatch,
|
|
|
|
|
EntityDeduplicationOptions,
|
|
|
|
|
MergeResult
|
|
|
|
|
} from './EntityDeduplicator.js'
|
|
|
|
|
|
2025-11-11 14:10:14 -08:00
|
|
|
export type {
|
|
|
|
|
DeduplicationStats
|
|
|
|
|
} from './BackgroundDeduplicator.js'
|
|
|
|
|
|
feat: add unified import system with auto-detection and dual storage
Implemented a comprehensive unified import system that revolutionizes how data flows into Brainy:
## Core Features (Phase 1)
- Auto-detection of file formats (Excel, PDF, CSV, JSON, Markdown) via magic bytes and content analysis
- Dual storage architecture: creates both VFS files AND knowledge graph entities
- Single unified API: brain.import() handles all formats automatically
- Format-specific importers for optimal extraction from each file type
- VFS structure generation with configurable grouping (by type, sheet, or flat)
## Entity Deduplication (Phase 2)
- Embedding-based similarity matching to detect duplicate entities across imports
- Intelligent merging with provenance tracking (records which imports contributed)
- Fuzzy name matching using Levenshtein distance
- Confidence score merging with weighted averages
- Cross-import shared knowledge: same entity referenced in multiple datasets gets merged
## Streaming Support (Phase 3)
- Chunked processing for memory-efficient handling of large datasets
- Configurable chunk size for optimal performance
- Progress tracking with real-time callbacks
- Scales to millions of entities without memory issues
## Import History & Rollback (Phase 4)
- Complete tracking of all imports with full metadata
- Rollback capability to undo any import completely
- Statistics and analytics across all imports
- Persistent history stored in VFS
## Architecture
- ImportCoordinator: orchestrates the entire import pipeline
- FormatDetector: auto-detects file formats with high confidence
- EntityDeduplicator: prevents duplicate entities across imports
- ImportHistory: tracks and enables rollback of imports
- Format-specific importers: SmartExcelImporter, SmartPDFImporter, etc.
- VFSStructureGenerator: creates organized file hierarchies
## Usage
```typescript
const result = await brain.import('/path/to/file.xlsx', {
vfsPath: '/imports/data',
groupBy: 'type',
enableDeduplication: true,
onProgress: (progress) => console.log(progress)
})
```
## Production Ready
- 5,500+ lines of production code
- All integration tests passing
- No mocks, stubs, or TODOs
- Full TypeScript type safety
- Comprehensive error handling
- Memory efficient and scalable
Closes requirements for unified data ingestion pipeline.
2025-10-08 16:55:30 -07:00
|
|
|
export type {
|
|
|
|
|
ImportHistoryEntry,
|
|
|
|
|
RollbackResult
|
|
|
|
|
} from './ImportHistory.js'
|