feat: add intelligent import for CSV, Excel, and PDF files
Add IntelligentImportAugmentation with support for: - CSV: auto-detection of encoding, delimiters, and field types - Excel: multi-sheet extraction with metadata preservation - PDF: text extraction, table detection, and metadata extraction Features: - Automatic format detection from file extension or content - Intelligent type inference (string, number, boolean, date) - Seamless integration with neural entity extraction - Production-ready with 69 comprehensive tests Dependencies added: - xlsx@^0.18.5 for Excel parsing - pdfjs-dist@^4.0.379 for PDF parsing - csv-parse@^6.1.0 for CSV parsing - chardet@^2.0.0 for encoding detection Documentation: - Updated README with import examples - Updated API_REFERENCE with comprehensive import() docs - Updated import-anything.md guide - Added working example: import-excel-pdf-csv.ts - Updated augmentations README
This commit is contained in:
parent
aaf8e0f411
commit
814cbb48ee
33 changed files with 4664 additions and 28 deletions
|
|
@ -10,6 +10,47 @@ Brainy's functionality in various ways.
|
|||
|
||||
## Available Augmentations
|
||||
|
||||
### Core Augmentations
|
||||
|
||||
#### IntelligentImportAugmentation
|
||||
|
||||
Automatically detects and processes CSV, Excel, and PDF files with intelligent extraction. This augmentation is enabled by default and provides:
|
||||
|
||||
- **CSV Support**: Auto-detection of encoding, delimiters, and field types
|
||||
- **Excel Support**: Multi-sheet extraction with metadata preservation
|
||||
- **PDF Support**: Text extraction, table detection, and metadata extraction
|
||||
- **Type Inference**: Automatically infers data types (string, number, boolean, date)
|
||||
- **Neural Integration**: Seamlessly integrates with entity extraction and relationship detection
|
||||
|
||||
```javascript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
intelligentImport: {
|
||||
enableCSV: true,
|
||||
enableExcel: true,
|
||||
enablePDF: true,
|
||||
maxFileSize: 100 * 1024 * 1024 // 100MB
|
||||
}
|
||||
})
|
||||
await brain.init()
|
||||
|
||||
// Import CSV with auto-detection
|
||||
await brain.import('customers.csv')
|
||||
|
||||
// Import Excel with specific sheets
|
||||
await brain.import('sales-data.xlsx', {
|
||||
excelSheets: ['Q1', 'Q2']
|
||||
})
|
||||
|
||||
// Import PDF with table extraction
|
||||
await brain.import('report.pdf', {
|
||||
pdfExtractTables: true
|
||||
})
|
||||
```
|
||||
|
||||
**See:** [Import Anything Guide](../../docs/guides/import-anything.md) | [Example](../../examples/import-excel-pdf-csv.ts)
|
||||
|
||||
### Conduit Augmentations
|
||||
|
||||
Conduit augmentations provide data synchronization between Brainy instances.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue