feat: add unified import system with auto-detection and dual storage

Implemented a comprehensive unified import system that revolutionizes how data flows into Brainy:

## Core Features (Phase 1)
- Auto-detection of file formats (Excel, PDF, CSV, JSON, Markdown) via magic bytes and content analysis
- Dual storage architecture: creates both VFS files AND knowledge graph entities
- Single unified API: brain.import() handles all formats automatically
- Format-specific importers for optimal extraction from each file type
- VFS structure generation with configurable grouping (by type, sheet, or flat)

## Entity Deduplication (Phase 2)
- Embedding-based similarity matching to detect duplicate entities across imports
- Intelligent merging with provenance tracking (records which imports contributed)
- Fuzzy name matching using Levenshtein distance
- Confidence score merging with weighted averages
- Cross-import shared knowledge: same entity referenced in multiple datasets gets merged

## Streaming Support (Phase 3)
- Chunked processing for memory-efficient handling of large datasets
- Configurable chunk size for optimal performance
- Progress tracking with real-time callbacks
- Scales to millions of entities without memory issues

## Import History & Rollback (Phase 4)
- Complete tracking of all imports with full metadata
- Rollback capability to undo any import completely
- Statistics and analytics across all imports
- Persistent history stored in VFS

## Architecture
- ImportCoordinator: orchestrates the entire import pipeline
- FormatDetector: auto-detects file formats with high confidence
- EntityDeduplicator: prevents duplicate entities across imports
- ImportHistory: tracks and enables rollback of imports
- Format-specific importers: SmartExcelImporter, SmartPDFImporter, etc.
- VFSStructureGenerator: creates organized file hierarchies

## Usage
```typescript
const result = await brain.import('/path/to/file.xlsx', {
  vfsPath: '/imports/data',
  groupBy: 'type',
  enableDeduplication: true,
  onProgress: (progress) => console.log(progress)
})
```

## Production Ready
- 5,500+ lines of production code
- All integration tests passing
- No mocks, stubs, or TODOs
- Full TypeScript type safety
- Comprehensive error handling
- Memory efficient and scalable

Closes requirements for unified data ingestion pipeline.
This commit is contained in:
David Snelling 2025-10-08 16:55:30 -07:00
parent 0035701f4a
commit a06e8772f1
21 changed files with 6246 additions and 0 deletions

View file

@ -1660,6 +1660,67 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return options?.limit ? concepts.slice(0, options.limit) : concepts
}
/**
* Import files with auto-detection and dual storage (VFS + Knowledge Graph)
*
* Unified import system that:
* - Auto-detects format (Excel, PDF, CSV, JSON, Markdown)
* - Extracts entities and relationships
* - Stores in both VFS (organized files) and Knowledge Graph (connected entities)
* - Links VFS files to graph entities
*
* @example
* // Import from file path
* const result = await brain.import('/path/to/file.xlsx')
*
* @example
* // Import from buffer
* const result = await brain.import(buffer, { format: 'pdf' })
*
* @example
* // Import JSON object
* const result = await brain.import({ entities: [...] })
*
* @example
* // Custom VFS path and grouping
* const result = await brain.import(buffer, {
* vfsPath: '/my-imports/data',
* groupBy: 'type',
* onProgress: (progress) => console.log(progress.message)
* })
*/
async import(
source: Buffer | string | object,
options?: {
format?: 'excel' | 'pdf' | 'csv' | 'json' | 'markdown'
vfsPath?: string
groupBy?: 'type' | 'sheet' | 'flat' | 'custom'
customGrouping?: (entity: any) => string
createEntities?: boolean
createRelationships?: boolean
preserveSource?: boolean
enableNeuralExtraction?: boolean
enableRelationshipInference?: boolean
enableConceptExtraction?: boolean
confidenceThreshold?: number
onProgress?: (progress: {
stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete'
message: string
processed?: number
total?: number
entities?: number
relationships?: number
}) => void
}
) {
// Lazy load ImportCoordinator
const { ImportCoordinator } = await import('./import/ImportCoordinator.js')
const coordinator = new ImportCoordinator(this)
await coordinator.init()
return await coordinator.import(source, options)
}
/**
* Virtual File System API - Knowledge Operating System
*/