feat: add intelligent import for CSV, Excel, and PDF files
Add IntelligentImportAugmentation with support for: - CSV: auto-detection of encoding, delimiters, and field types - Excel: multi-sheet extraction with metadata preservation - PDF: text extraction, table detection, and metadata extraction Features: - Automatic format detection from file extension or content - Intelligent type inference (string, number, boolean, date) - Seamless integration with neural entity extraction - Production-ready with 69 comprehensive tests Dependencies added: - xlsx@^0.18.5 for Excel parsing - pdfjs-dist@^4.0.379 for PDF parsing - csv-parse@^6.1.0 for CSV parsing - chardet@^2.0.0 for encoding detection Documentation: - Updated README with import examples - Updated API_REFERENCE with comprehensive import() docs - Updated import-anything.md guide - Added working example: import-excel-pdf-csv.ts - Updated augmentations README
This commit is contained in:
parent
aaf8e0f411
commit
814cbb48ee
33 changed files with 4664 additions and 28 deletions
|
|
@ -683,26 +683,75 @@ await data.clear({
|
|||
---
|
||||
|
||||
### `async import(params): Promise<ImportResult>`
|
||||
Imports data from various formats.
|
||||
Imports data from various formats with automatic detection.
|
||||
|
||||
**Parameters:**
|
||||
- `data` (required) - Data to import
|
||||
- `format` (required) - 'json' | 'csv' | 'parquet'
|
||||
- `mapping` - Field mapping
|
||||
- `batchSize` - Import batch size
|
||||
- `validate` - Validate items
|
||||
- `data` (required) - Data to import (Buffer, string, array, or file path)
|
||||
- `format` - 'auto' | 'json' | 'csv' | 'excel' | 'pdf' | 'yaml' | 'text' (default: 'auto')
|
||||
- `mapping` - Field mapping configuration
|
||||
- `batchSize` - Import batch size (default: 50)
|
||||
- `validate` - Validate items before import
|
||||
- `relationships` - Extract relationships automatically (default: true)
|
||||
|
||||
**Example:**
|
||||
**CSV-specific options:**
|
||||
- `csvDelimiter` - Column delimiter (auto-detected if not specified)
|
||||
- `csvHeaders` - First row contains headers (default: true)
|
||||
- `encoding` - Character encoding (auto-detected if not specified)
|
||||
|
||||
**Excel-specific options:**
|
||||
- `excelSheets` - Sheet names array or 'all' for all sheets
|
||||
|
||||
**PDF-specific options:**
|
||||
- `pdfExtractTables` - Extract tables from PDFs (default: true)
|
||||
- `pdfPreserveLayout` - Preserve text layout (default: true)
|
||||
|
||||
**Examples:**
|
||||
```typescript
|
||||
const result = await data.import({
|
||||
data: csvData,
|
||||
format: 'csv',
|
||||
mapping: {
|
||||
'name': 'title',
|
||||
'desc': 'description'
|
||||
},
|
||||
batchSize: 500
|
||||
// Import CSV file with auto-detection
|
||||
const csvResult = await brain.import('customers.csv')
|
||||
// Auto-detects: format, encoding, delimiter, field types
|
||||
|
||||
// Import Excel workbook
|
||||
const excelResult = await brain.import('sales-data.xlsx', {
|
||||
excelSheets: ['Q1', 'Q2'] // Import specific sheets
|
||||
})
|
||||
|
||||
// Import PDF with table extraction
|
||||
const pdfResult = await brain.import('report.pdf', {
|
||||
pdfExtractTables: true
|
||||
})
|
||||
|
||||
// Import data array
|
||||
const dataResult = await brain.import([
|
||||
{ name: 'Alice', role: 'Engineer' },
|
||||
{ name: 'Bob', role: 'Designer' }
|
||||
], {
|
||||
batchSize: 100,
|
||||
relationships: true // Auto-extract relationships
|
||||
})
|
||||
|
||||
// Import with custom CSV delimiter
|
||||
const tsvResult = await brain.import('data.tsv', {
|
||||
format: 'csv',
|
||||
csvDelimiter: '\t'
|
||||
})
|
||||
```
|
||||
|
||||
**Returns:**
|
||||
```typescript
|
||||
{
|
||||
success: boolean
|
||||
imported: number // Number of items successfully imported
|
||||
failed: number // Number of items that failed
|
||||
entityIds: string[] // IDs of created entities
|
||||
metadata: {
|
||||
format: string // Detected format
|
||||
encoding?: string // Detected encoding (CSV)
|
||||
delimiter?: string // Detected delimiter (CSV)
|
||||
sheets?: string[] // Processed sheets (Excel)
|
||||
pageCount?: number // Number of pages (PDF)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue